From python-3000-checkins at python.org Tue Oct 2 09:26:15 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Tue, 2 Oct 2007 09:26:15 +0200 (CEST) Subject: [Python-3000-checkins] r58293 - python/branches/py3k/Doc/library/stdtypes.rst Message-ID: <20071002072615.7BE941E4009@bag.python.org> Author: neal.norwitz Date: Tue Oct 2 09:26:14 2007 New Revision: 58293 Modified: python/branches/py3k/Doc/library/stdtypes.rst Log: Get the doc to build. Modified: python/branches/py3k/Doc/library/stdtypes.rst ============================================================================== --- python/branches/py3k/Doc/library/stdtypes.rst (original) +++ python/branches/py3k/Doc/library/stdtypes.rst Tue Oct 2 09:26:14 2007 @@ -233,7 +233,7 @@ pair: complex number; literals pair: hexadecimal; literals pair: octal; literals - pair: binary: literals + pair: binary; literals Numbers are created by numeric literals or as the result of built-in functions and operators. Unadorned integer literals (including hex, octal and binary @@ -263,7 +263,7 @@ +---------------------+---------------------------------+-------+--------------------+ | Operation | Result | Notes | Full documentation | -+==================== +=================================+=======+====================| ++=====================+=================================+=======+====================+ | ``x + y`` | sum of *x* and *y* | | | +---------------------+---------------------------------+-------+--------------------+ | ``x - y`` | difference of *x* and *y* | | | From python-3000-checkins at python.org Sat Oct 6 10:22:26 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Sat, 6 Oct 2007 10:22:26 +0200 (CEST) Subject: [Python-3000-checkins] r58346 - python/branches/py3k/Modules/_bsddb.c Message-ID: <20071006082226.D59831E4007@bag.python.org> Author: gregory.p.smith Date: Sat Oct 6 10:22:26 2007 New Revision: 58346 Modified: python/branches/py3k/Modules/_bsddb.c Log: Merge 58343: attempt to fix DBSequence.get_key() to not fail or crash. Modified: python/branches/py3k/Modules/_bsddb.c ============================================================================== --- python/branches/py3k/Modules/_bsddb.c (original) +++ python/branches/py3k/Modules/_bsddb.c Sat Oct 6 10:22:26 2007 @@ -5024,14 +5024,20 @@ { int err; DBT key; + PyObject *retval; + key.flags = DB_DBT_MALLOC; CHECK_SEQUENCE_NOT_CLOSED(self) MYDB_BEGIN_ALLOW_THREADS err = self->sequence->get_key(self->sequence, &key); MYDB_END_ALLOW_THREADS + if (!err) + retval = PyBytes_FromStringAndSize(key.data, key.size); + + free_dbt(&key); RETURN_IF_ERR(); - return PyBytes_FromStringAndSize(key.data, key.size); + return retval; } static PyObject* From python-3000-checkins at python.org Sat Oct 6 10:23:11 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Sat, 6 Oct 2007 10:23:11 +0200 (CEST) Subject: [Python-3000-checkins] r58347 - python/branches/py3k/setup.py Message-ID: <20071006082311.3FC551E401C@bag.python.org> Author: gregory.p.smith Date: Sat Oct 6 10:23:10 2007 New Revision: 58347 Modified: python/branches/py3k/setup.py Log: Merge 58344: allow BerkeleyDB 4.6.x >= 4.6.21 for the bsddb module Modified: python/branches/py3k/setup.py ============================================================================== --- python/branches/py3k/setup.py (original) +++ python/branches/py3k/setup.py Sat Oct 6 10:23:10 2007 @@ -630,16 +630,15 @@ # implementation independent wrapper for these; dumbdbm.py provides # similar functionality (but slower of course) implemented in Python. - # Sleepycat^WOracle Berkeley DB interface. http://www.sleepycat.com + # Sleepycat^WOracle Berkeley DB interface. + # http://www.oracle.com/database/berkeley-db/db/index.html # # This requires the Sleepycat^WOracle DB code. The supported versions - # are set below. Visit http://www.sleepycat.com/ to download + # are set below. Visit the URL above to download # a release. Most open source OSes come with one or more # versions of BerkeleyDB already installed. - # XXX(gps) - Do not allow BerkeleyDB 4.6.x until Oracle fixes - # the DB_HASH lockup bug that is present in 4.6.19. - max_db_ver = (4, 5) + max_db_ver = (4, 6) min_db_ver = (3, 3) db_setup_debug = False # verbose debug prints from this script? @@ -710,6 +709,15 @@ db_minor = int(m.group(1)) db_ver = (db_major, db_minor) + # Avoid 4.6 prior to 4.6.21 due to a BerkeleyDB bug + if db_ver == (4, 6): + m = re.search(r"#define\WDB_VERSION_PATCH\W(\d+)", f) + db_patch = int(m.group(1)) + if db_patch < 21: + print("db.h:", db_ver, "patch", db_patch, + "being ignored (4.6.x must be >= 4.6.21)") + continue + if ( (db_ver not in db_ver_inc_map) and (db_ver <= max_db_ver and db_ver >= min_db_ver) ): # save the include directory with the db.h version From python-3000-checkins at python.org Sun Oct 7 21:26:51 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sun, 7 Oct 2007 21:26:51 +0200 (CEST) Subject: [Python-3000-checkins] r58352 - python/branches/py3k/Objects/memoryobject.c Message-ID: <20071007192651.51ED11E4006@bag.python.org> Author: neal.norwitz Date: Sun Oct 7 21:26:50 2007 New Revision: 58352 Modified: python/branches/py3k/Objects/memoryobject.c Log: At least one of the buildbots was complaining about newview being used without being initialized. Also make the code conform to the comment and return the new object. This code needs a test! Modified: python/branches/py3k/Objects/memoryobject.c ============================================================================== --- python/branches/py3k/Objects/memoryobject.c (original) +++ python/branches/py3k/Objects/memoryobject.c Sun Oct 7 21:26:50 2007 @@ -513,12 +513,11 @@ else { /* Return a new memory-view object */ Py_buffer newview; - PyMemoryView_FromMemory(&newview); + memset(&newview, 0, sizeof(newview)); + return PyMemoryView_FromMemory(&newview); } } - - Py_INCREF(Py_NotImplemented); return Py_NotImplemented; } From python-3000-checkins at python.org Mon Oct 8 01:12:42 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 8 Oct 2007 01:12:42 +0200 (CEST) Subject: [Python-3000-checkins] r58353 - python/branches/py3k/Doc/library/stdtypes.rst Message-ID: <20071007231242.22BAC1E4006@bag.python.org> Author: brett.cannon Date: Mon Oct 8 01:12:41 2007 New Revision: 58353 Modified: python/branches/py3k/Doc/library/stdtypes.rst Log: Fix a minor typo. Modified: python/branches/py3k/Doc/library/stdtypes.rst ============================================================================== --- python/branches/py3k/Doc/library/stdtypes.rst (original) +++ python/branches/py3k/Doc/library/stdtypes.rst Mon Oct 8 01:12:41 2007 @@ -218,7 +218,7 @@ There are three distinct numeric types: :dfn:`integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In addition, Booleans are a subtype of -plain integers. Integers have unlimited precision. loating point numbers are +plain integers. Integers have unlimited precision. Floating point numbers are implemented using :ctype:`double` in C. All bets on their precision are off unless you happen to know the machine you are working with. From python-3000-checkins at python.org Mon Oct 8 01:54:41 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 8 Oct 2007 01:54:41 +0200 (CEST) Subject: [Python-3000-checkins] r58355 - in python/branches/py3k-importlib: Doc/library/stdtypes.rst Modules/_bsddb.c Objects/memoryobject.c setup.py Message-ID: <20071007235441.6428F1E4006@bag.python.org> Author: brett.cannon Date: Mon Oct 8 01:54:40 2007 New Revision: 58355 Modified: python/branches/py3k-importlib/ (props changed) python/branches/py3k-importlib/Doc/library/stdtypes.rst python/branches/py3k-importlib/Modules/_bsddb.c python/branches/py3k-importlib/Objects/memoryobject.c python/branches/py3k-importlib/setup.py Log: Merged revisions 58287-58354 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k Modified: python/branches/py3k-importlib/Doc/library/stdtypes.rst ============================================================================== --- python/branches/py3k-importlib/Doc/library/stdtypes.rst (original) +++ python/branches/py3k-importlib/Doc/library/stdtypes.rst Mon Oct 8 01:54:40 2007 @@ -218,7 +218,7 @@ There are three distinct numeric types: :dfn:`integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In addition, Booleans are a subtype of -plain integers. Integers have unlimited precision. loating point numbers are +plain integers. Integers have unlimited precision. Floating point numbers are implemented using :ctype:`double` in C. All bets on their precision are off unless you happen to know the machine you are working with. @@ -233,7 +233,7 @@ pair: complex number; literals pair: hexadecimal; literals pair: octal; literals - pair: binary: literals + pair: binary; literals Numbers are created by numeric literals or as the result of built-in functions and operators. Unadorned integer literals (including hex, octal and binary @@ -263,7 +263,7 @@ +---------------------+---------------------------------+-------+--------------------+ | Operation | Result | Notes | Full documentation | -+==================== +=================================+=======+====================| ++=====================+=================================+=======+====================+ | ``x + y`` | sum of *x* and *y* | | | +---------------------+---------------------------------+-------+--------------------+ | ``x - y`` | difference of *x* and *y* | | | Modified: python/branches/py3k-importlib/Modules/_bsddb.c ============================================================================== --- python/branches/py3k-importlib/Modules/_bsddb.c (original) +++ python/branches/py3k-importlib/Modules/_bsddb.c Mon Oct 8 01:54:40 2007 @@ -5024,14 +5024,20 @@ { int err; DBT key; + PyObject *retval; + key.flags = DB_DBT_MALLOC; CHECK_SEQUENCE_NOT_CLOSED(self) MYDB_BEGIN_ALLOW_THREADS err = self->sequence->get_key(self->sequence, &key); MYDB_END_ALLOW_THREADS + if (!err) + retval = PyBytes_FromStringAndSize(key.data, key.size); + + free_dbt(&key); RETURN_IF_ERR(); - return PyBytes_FromStringAndSize(key.data, key.size); + return retval; } static PyObject* Modified: python/branches/py3k-importlib/Objects/memoryobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/memoryobject.c (original) +++ python/branches/py3k-importlib/Objects/memoryobject.c Mon Oct 8 01:54:40 2007 @@ -513,12 +513,11 @@ else { /* Return a new memory-view object */ Py_buffer newview; - PyMemoryView_FromMemory(&newview); + memset(&newview, 0, sizeof(newview)); + return PyMemoryView_FromMemory(&newview); } } - - Py_INCREF(Py_NotImplemented); return Py_NotImplemented; } Modified: python/branches/py3k-importlib/setup.py ============================================================================== --- python/branches/py3k-importlib/setup.py (original) +++ python/branches/py3k-importlib/setup.py Mon Oct 8 01:54:40 2007 @@ -630,16 +630,15 @@ # implementation independent wrapper for these; dumbdbm.py provides # similar functionality (but slower of course) implemented in Python. - # Sleepycat^WOracle Berkeley DB interface. http://www.sleepycat.com + # Sleepycat^WOracle Berkeley DB interface. + # http://www.oracle.com/database/berkeley-db/db/index.html # # This requires the Sleepycat^WOracle DB code. The supported versions - # are set below. Visit http://www.sleepycat.com/ to download + # are set below. Visit the URL above to download # a release. Most open source OSes come with one or more # versions of BerkeleyDB already installed. - # XXX(gps) - Do not allow BerkeleyDB 4.6.x until Oracle fixes - # the DB_HASH lockup bug that is present in 4.6.19. - max_db_ver = (4, 5) + max_db_ver = (4, 6) min_db_ver = (3, 3) db_setup_debug = False # verbose debug prints from this script? @@ -710,6 +709,15 @@ db_minor = int(m.group(1)) db_ver = (db_major, db_minor) + # Avoid 4.6 prior to 4.6.21 due to a BerkeleyDB bug + if db_ver == (4, 6): + m = re.search(r"#define\WDB_VERSION_PATCH\W(\d+)", f) + db_patch = int(m.group(1)) + if db_patch < 21: + print("db.h:", db_ver, "patch", db_patch, + "being ignored (4.6.x must be >= 4.6.21)") + continue + if ( (db_ver not in db_ver_inc_map) and (db_ver <= max_db_ver and db_ver >= min_db_ver) ): # save the include directory with the db.h version From python-3000-checkins at python.org Mon Oct 8 04:46:16 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 8 Oct 2007 04:46:16 +0200 (CEST) Subject: [Python-3000-checkins] r58360 - in python/branches/py3k: Lib/_abcoll.py Lib/ctypes/test/test_array_in_pointer.py Lib/ctypes/test/test_byteswap.py Lib/ctypes/test/test_strings.py Lib/sqlite3/dbapi2.py Lib/sqlite3/test/dbapi.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Lib/subprocess.py Lib/test/test_array.py Lib/test/test_buffer.py Lib/test/test_bytes.py Lib/test/test_io.py Lib/test/test_marshal.py Lib/test/test_repr.py Lib/test/test_struct.py Lib/test/test_types.py Lib/test/test_unicode.py Lib/types.py Modules/_ctypes/_ctypes.c Modules/_sqlite/connection.c Modules/_sqlite/cursor.c Modules/_sqlite/statement.c Python/bltinmodule.c Message-ID: <20071008024616.EF7851E4006@bag.python.org> Author: guido.van.rossum Date: Mon Oct 8 04:46:15 2007 New Revision: 58360 Removed: python/branches/py3k/Lib/test/test_buffer.py Modified: python/branches/py3k/Lib/_abcoll.py python/branches/py3k/Lib/ctypes/test/test_array_in_pointer.py python/branches/py3k/Lib/ctypes/test/test_byteswap.py python/branches/py3k/Lib/ctypes/test/test_strings.py python/branches/py3k/Lib/sqlite3/dbapi2.py python/branches/py3k/Lib/sqlite3/test/dbapi.py python/branches/py3k/Lib/sqlite3/test/types.py python/branches/py3k/Lib/sqlite3/test/userfunctions.py python/branches/py3k/Lib/subprocess.py python/branches/py3k/Lib/test/test_array.py python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Lib/test/test_io.py python/branches/py3k/Lib/test/test_marshal.py python/branches/py3k/Lib/test/test_repr.py python/branches/py3k/Lib/test/test_struct.py python/branches/py3k/Lib/test/test_types.py python/branches/py3k/Lib/test/test_unicode.py python/branches/py3k/Lib/types.py python/branches/py3k/Modules/_ctypes/_ctypes.c python/branches/py3k/Modules/_sqlite/connection.c python/branches/py3k/Modules/_sqlite/cursor.c python/branches/py3k/Modules/_sqlite/statement.c python/branches/py3k/Python/bltinmodule.c Log: Breaking ground for PEP 3137 implementation: Get rid of buffer(). Use memoryview() in its place where possible. In a few places, do things a bit different, because memoryview() can't slice (yet). Modified: python/branches/py3k/Lib/_abcoll.py ============================================================================== --- python/branches/py3k/Lib/_abcoll.py (original) +++ python/branches/py3k/Lib/_abcoll.py Mon Oct 8 04:46:15 2007 @@ -491,7 +491,7 @@ Sequence.register(tuple) Sequence.register(basestring) -Sequence.register(buffer) +Sequence.register(memoryview) class MutableSequence(Sequence): Modified: python/branches/py3k/Lib/ctypes/test/test_array_in_pointer.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_array_in_pointer.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_array_in_pointer.py Mon Oct 8 04:46:15 2007 @@ -6,7 +6,7 @@ def dump(obj): # helper function to dump memory contents in hex, with a hyphen # between the bytes. - h = str(hexlify(buffer(obj))) + h = str(hexlify(memoryview(obj))) return re.sub(r"(..)", r"\1-", h)[:-1] Modified: python/branches/py3k/Lib/ctypes/test/test_byteswap.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_byteswap.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_byteswap.py Mon Oct 8 04:46:15 2007 @@ -4,7 +4,7 @@ from ctypes import * def bin(s): - return str(hexlify(buffer(s))).upper() + return str(hexlify(memoryview(s))).upper() # Each *simple* type that supports different byte orders has an # __ctype_be__ attribute that specifies the same type in BIG ENDIAN Modified: python/branches/py3k/Lib/ctypes/test/test_strings.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_strings.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_strings.py Mon Oct 8 04:46:15 2007 @@ -30,17 +30,17 @@ buf.value = "Hello, World" self.failUnlessEqual(buf.value, "Hello, World") - self.failUnlessRaises(TypeError, setattr, buf, "value", buffer("Hello, World")) - self.assertRaises(TypeError, setattr, buf, "value", buffer("abc")) - self.assertRaises(ValueError, setattr, buf, "raw", buffer("x" * 100)) + self.failUnlessRaises(TypeError, setattr, buf, "value", memoryview(b"Hello, World")) + self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) + self.assertRaises(ValueError, setattr, buf, "raw", memoryview(b"x" * 100)) def test_c_buffer_raw(self): buf = c_buffer(32) - buf.raw = buffer(b"Hello, World") + buf.raw = memoryview(b"Hello, World") self.failUnlessEqual(buf.value, "Hello, World") - self.assertRaises(TypeError, setattr, buf, "value", buffer("abc")) - self.assertRaises(ValueError, setattr, buf, "raw", buffer("x" * 100)) + self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) + self.assertRaises(ValueError, setattr, buf, "raw", memoryview(b"x" * 100)) def test_param_1(self): BUF = c_char * 4 Modified: python/branches/py3k/Lib/sqlite3/dbapi2.py ============================================================================== --- python/branches/py3k/Lib/sqlite3/dbapi2.py (original) +++ python/branches/py3k/Lib/sqlite3/dbapi2.py Mon Oct 8 04:46:15 2007 @@ -50,7 +50,7 @@ version_info = tuple([int(x) for x in version.split(".")]) sqlite_version_info = tuple([int(x) for x in sqlite_version.split(".")]) -Binary = buffer +Binary = memoryview def register_adapters_and_converters(): def adapt_date(val): Modified: python/branches/py3k/Lib/sqlite3/test/dbapi.py ============================================================================== --- python/branches/py3k/Lib/sqlite3/test/dbapi.py (original) +++ python/branches/py3k/Lib/sqlite3/test/dbapi.py Mon Oct 8 04:46:15 2007 @@ -593,7 +593,7 @@ ts = sqlite.TimestampFromTicks(42) def CheckBinary(self): - b = sqlite.Binary(chr(0) + "'") + b = sqlite.Binary(b"\0'") class ExtensionTests(unittest.TestCase): def CheckScriptStringSql(self): Modified: python/branches/py3k/Lib/sqlite3/test/types.py ============================================================================== --- python/branches/py3k/Lib/sqlite3/test/types.py (original) +++ python/branches/py3k/Lib/sqlite3/test/types.py Mon Oct 8 04:46:15 2007 @@ -62,7 +62,7 @@ self.failUnlessEqual(row[0], val) def CheckBlob(self): - val = buffer(b"Guglhupf") + val = memoryview(b"Guglhupf") self.cur.execute("insert into test(b) values (?)", (val,)) self.cur.execute("select b from test") row = self.cur.fetchone() @@ -203,7 +203,7 @@ def CheckBlob(self): # default - val = buffer(b"Guglhupf") + val = memoryview(b"Guglhupf") self.cur.execute("insert into test(bin) values (?)", (val,)) self.cur.execute("select bin from test") row = self.cur.fetchone() @@ -305,7 +305,7 @@ def CheckBinaryInputForConverter(self): testdata = b"abcdefg" * 10 - result = self.con.execute('select ? as "x [bin]"', (buffer(bz2.compress(testdata)),)).fetchone()[0] + result = self.con.execute('select ? as "x [bin]"', (memoryview(bz2.compress(testdata)),)).fetchone()[0] self.failUnlessEqual(testdata, result) class DateTimeTests(unittest.TestCase): Modified: python/branches/py3k/Lib/sqlite3/test/userfunctions.py ============================================================================== --- python/branches/py3k/Lib/sqlite3/test/userfunctions.py (original) +++ python/branches/py3k/Lib/sqlite3/test/userfunctions.py Mon Oct 8 04:46:15 2007 @@ -36,7 +36,7 @@ def func_returnnull(): return None def func_returnblob(): - return buffer(b"blob") + return b"blob" def func_raiseexception(): 5/0 @@ -49,7 +49,7 @@ def func_isnone(v): return type(v) is type(None) def func_isblob(v): - return type(v) is buffer + return isinstance(v, (bytes, memoryview)) class AggrNoStep: def __init__(self): @@ -100,7 +100,8 @@ self.val = None def step(self, whichType, val): - theType = {"str": str, "int": int, "float": float, "None": type(None), "blob": buffer} + theType = {"str": str, "int": int, "float": float, "None": type(None), + "blob": bytes} self.val = int(theType[whichType] is type(val)) def finalize(self): @@ -196,8 +197,8 @@ cur = self.con.cursor() cur.execute("select returnblob()") val = cur.fetchone()[0] - self.failUnlessEqual(type(val), buffer) - self.failUnlessEqual(val, buffer(b"blob")) + self.failUnlessEqual(type(val), bytes) + self.failUnlessEqual(val, memoryview(b"blob")) def CheckFuncException(self): cur = self.con.cursor() @@ -234,7 +235,7 @@ def CheckParamBlob(self): cur = self.con.cursor() - cur.execute("select isblob(?)", (buffer(b"blob"),)) + cur.execute("select isblob(?)", (memoryview(b"blob"),)) val = cur.fetchone()[0] self.failUnlessEqual(val, 1) @@ -252,7 +253,7 @@ ) """) cur.execute("insert into test(t, i, f, n, b) values (?, ?, ?, ?, ?)", - ("foo", 5, 3.14, None, buffer(b"blob"),)) + ("foo", 5, 3.14, None, memoryview(b"blob"),)) self.con.create_aggregate("nostep", 1, AggrNoStep) self.con.create_aggregate("nofinalize", 1, AggrNoFinalize) @@ -344,7 +345,7 @@ def CheckAggrCheckParamBlob(self): cur = self.con.cursor() - cur.execute("select checkType('blob', ?)", (buffer(b"blob"),)) + cur.execute("select checkType('blob', ?)", (memoryview(b"blob"),)) val = cur.fetchone()[0] self.failUnlessEqual(val, 1) Modified: python/branches/py3k/Lib/subprocess.py ============================================================================== --- python/branches/py3k/Lib/subprocess.py (original) +++ python/branches/py3k/Lib/subprocess.py Mon Oct 8 04:46:15 2007 @@ -1041,8 +1041,11 @@ def _communicate(self, input): - if isinstance(input, str): # Unicode - input = input.encode("utf-8") # XXX What else? + if self.stdin: + if isinstance(input, str): # Unicode + input = input.encode("utf-8") # XXX What else? + if not isinstance(input, (bytes, str8)): + input = bytes(input) read_set = [] write_set = [] stdout = None # Return @@ -1071,7 +1074,8 @@ # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), buffer(input, input_offset, 512)) + chunk = input[input_offset : input_offset + 512] + bytes_written = os.write(self.stdin.fileno(), chunk) input_offset += bytes_written if input_offset >= len(input): self.stdin.close() Modified: python/branches/py3k/Lib/test/test_array.py ============================================================================== --- python/branches/py3k/Lib/test/test_array.py (original) +++ python/branches/py3k/Lib/test/test_array.py Mon Oct 8 04:46:15 2007 @@ -708,7 +708,7 @@ def test_buffer(self): a = array.array(self.typecode, self.example) - b = bytes(buffer(a)) + b = bytes(memoryview(a)) self.assertEqual(b[0], a.tostring()[0]) def test_weakref(self): Deleted: /python/branches/py3k/Lib/test/test_buffer.py ============================================================================== --- /python/branches/py3k/Lib/test/test_buffer.py Mon Oct 8 04:46:15 2007 +++ (empty file) @@ -1,56 +0,0 @@ -"""Unit tests for buffer objects. - -For now, we just test (the brand new) rich comparison. - -""" - -import unittest -from test import test_support - -class BufferTests(unittest.TestCase): - - def test_comparison(self): - a = buffer("a.b.c") - b = buffer("a.b" + ".c") - self.assert_(a == b) - self.assert_(a <= b) - self.assert_(a >= b) - self.assert_(a == "a.b.c") - self.assert_(a <= "a.b.c") - self.assert_(a >= "a.b.c") - b = buffer("a.b.c.d") - self.assert_(a != b) - self.assert_(a <= b) - self.assert_(a < b) - self.assert_(a != "a.b.c.d") - self.assert_(a < "a.b.c.d") - self.assert_(a <= "a.b.c.d") - b = buffer("a.b") - self.assert_(a != b) - self.assert_(a >= b) - self.assert_(a > b) - self.assert_(a != "a.b") - self.assert_(a > "a.b") - self.assert_(a >= "a.b") - b = object() - self.assert_(a != b) - self.failIf(a == b) - self.assertRaises(TypeError, lambda: a < b) - - def test_extended_getslice(self): - # Test extended slicing by comparing with list slicing. - s = bytes(range(255, -1, -1)) - b = buffer(s) - indices = (0, None, 1, 3, 19, 300, -1, -2, -31, -300) - for start in indices: - for stop in indices: - # Skip step 0 (invalid) - for step in indices[1:]: - self.assertEqual(b[start:stop:step], - s[start:stop:step]) - -def test_main(): - test_support.run_unittest(BufferTests) - -if __name__ == "__main__": - test_main() Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Mon Oct 8 04:46:15 2007 @@ -343,7 +343,7 @@ def test_from_buffer(self): sample = str8("Hello world\n\x80\x81\xfe\xff") - buf = buffer(sample) + buf = memoryview(sample) b = bytes(buf) self.assertEqual(b, bytes(map(ord, sample))) @@ -456,8 +456,8 @@ b = bytes([0x1a, 0x2b, 0x30]) self.assertEquals(bytes.fromhex('1a2B30'), b) self.assertEquals(bytes.fromhex(' 1A 2B 30 '), b) - self.assertEquals(bytes.fromhex(buffer(b'')), bytes()) - self.assertEquals(bytes.fromhex(buffer(b'0000')), bytes([0, 0])) + self.assertEquals(bytes.fromhex(memoryview(b'')), bytes()) + self.assertEquals(bytes.fromhex(memoryview(b'0000')), bytes([0, 0])) self.assertRaises(ValueError, bytes.fromhex, 'a') self.assertRaises(ValueError, bytes.fromhex, 'rt') self.assertRaises(ValueError, bytes.fromhex, '1a b cd') @@ -630,7 +630,7 @@ self.assertEqual(b' a bb c '.split(None, 3), [b'a', b'bb', b'c']) def test_split_buffer(self): - self.assertEqual(b'a b'.split(buffer(b' ')), [b'a', b'b']) + self.assertEqual(b'a b'.split(memoryview(b' ')), [b'a', b'b']) def test_split_string_error(self): self.assertRaises(TypeError, b'a b'.split, ' ') @@ -653,7 +653,7 @@ self.assertEqual(b' a bb c '.rsplit(None, 3), [b'a', b'bb', b'c']) def test_rplit_buffer(self): - self.assertEqual(b'a b'.rsplit(buffer(b' ')), [b'a', b'b']) + self.assertEqual(b'a b'.rsplit(memoryview(b' ')), [b'a', b'b']) def test_rplit_string_error(self): self.assertRaises(TypeError, b'a b'.rsplit, ' ') @@ -707,9 +707,9 @@ self.assertEqual(b.rstrip(), b' \t\n\r\f\vabc') def test_strip_buffer(self): - self.assertEqual(b'abc'.strip(buffer(b'ac')), b'b') - self.assertEqual(b'abc'.lstrip(buffer(b'ac')), b'bc') - self.assertEqual(b'abc'.rstrip(buffer(b'ac')), b'ab') + self.assertEqual(b'abc'.strip(memoryview(b'ac')), b'b') + self.assertEqual(b'abc'.lstrip(memoryview(b'ac')), b'bc') + self.assertEqual(b'abc'.rstrip(memoryview(b'ac')), b'ab') def test_strip_string_error(self): self.assertRaises(TypeError, b'abc'.strip, 'b') Modified: python/branches/py3k/Lib/test/test_io.py ============================================================================== --- python/branches/py3k/Lib/test/test_io.py (original) +++ python/branches/py3k/Lib/test/test_io.py Mon Oct 8 04:46:15 2007 @@ -251,7 +251,7 @@ def test_array_writes(self): a = array.array('i', range(10)) - n = len(buffer(a)) + n = len(memoryview(a)) f = io.open(test_support.TESTFN, "wb", 0) self.assertEqual(f.write(a), n) f.close() Modified: python/branches/py3k/Lib/test/test_marshal.py ============================================================================== --- python/branches/py3k/Lib/test/test_marshal.py (original) +++ python/branches/py3k/Lib/test/test_marshal.py Mon Oct 8 04:46:15 2007 @@ -98,9 +98,9 @@ for s in ["", "Andr\xe8 Previn", "abc", " "*10000]: self.helper(s) - def test_buffer(self): + def test_bytes(self): for s in [b"", b"Andr\xe8 Previn", b"abc", b" "*10000]: - self.helper(buffer(s)) + self.helper(s) class ExceptionTestCase(unittest.TestCase): def test_exceptions(self): Modified: python/branches/py3k/Lib/test/test_repr.py ============================================================================== --- python/branches/py3k/Lib/test/test_repr.py (original) +++ python/branches/py3k/Lib/test/test_repr.py Mon Oct 8 04:46:15 2007 @@ -163,12 +163,6 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") - def test_buffer(self): - # XXX doesn't test buffers with no b_base or read-write buffers (see - # bufferobject.c). The test is fairly incomplete too. Sigh. - x = buffer('foo') - self.failUnless(repr(x).startswith('I', data) vereq(value, 0x12345678) -# Test methods to pack and unpack from buffers rather than strings. +# Test methods to pack and unpack from memoryviews rather than strings. test_unpack_from() test_pack_into() test_pack_into_fn() -test_unpack_with_buffer() +test_unpack_with_memoryview() def test_bool(): for prefix in tuple("<>!=")+('',): Modified: python/branches/py3k/Lib/test/test_types.py ============================================================================== --- python/branches/py3k/Lib/test/test_types.py (original) +++ python/branches/py3k/Lib/test/test_types.py Mon Oct 8 04:46:15 2007 @@ -203,54 +203,6 @@ self.assertRaises(TypeError, type, 1, 2) self.assertRaises(TypeError, type, 1, 2, 3, 4) - def test_buffers(self): - self.assertRaises(ValueError, buffer, 'asdf', -1) - self.assertRaises(TypeError, buffer, None) - - a = buffer(b'asdf') - hash(a) - b = a * 5 - if a == b: - self.fail('buffers should not be equal') - if str(b) != ('asdf' * 5): - self.fail('repeated buffer has wrong content') - if str(a * 0) != '': - self.fail('repeated buffer zero times has wrong content') - if str(a + buffer(b'def')) != 'asdfdef': - self.fail('concatenation of buffers yields wrong content') - if str(buffer(a)) != 'asdf': - self.fail('composing buffers failed') - if str(buffer(a, 2)) != 'df': - self.fail('specifying buffer offset failed') - if str(buffer(a, 0, 2)) != 'as': - self.fail('specifying buffer size failed') - if str(buffer(a, 1, 2)) != 'sd': - self.fail('specifying buffer offset and size failed') - self.assertRaises(ValueError, buffer, buffer(b'asdf', 1), -1) - if str(buffer(buffer(b'asdf', 0, 2), 0)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 0, 5000)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 0, -1)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 1, 2)) != 's': - self.fail('composing length-specified buffer failed') - - try: a[1] = 'g' - except TypeError: pass - else: self.fail("buffer assignment should raise TypeError") - - try: a[0:1] = 'g' - except TypeError: pass - else: self.fail("buffer slice assignment should raise TypeError") - - # array.array() returns an object that does not implement a char buffer, - # something which int() uses for conversion. - import array - try: int(buffer(array.array('b'))) - except TypeError: pass - else: self.fail("char buffer (at C level) not working") - def test_main(): run_unittest(TypesTests) Modified: python/branches/py3k/Lib/test/test_unicode.py ============================================================================== --- python/branches/py3k/Lib/test/test_unicode.py (original) +++ python/branches/py3k/Lib/test/test_unicode.py Mon Oct 8 04:46:15 2007 @@ -713,7 +713,7 @@ if not sys.platform.startswith('java'): self.assertEqual( str( - buffer(b'character buffers are decoded to unicode'), + memoryview(b'character buffers are decoded to unicode'), 'utf-8', 'strict' ), Modified: python/branches/py3k/Lib/types.py ============================================================================== --- python/branches/py3k/Lib/types.py (original) +++ python/branches/py3k/Lib/types.py Mon Oct 8 04:46:15 2007 @@ -22,8 +22,6 @@ except NameError: pass -BufferType = buffer - TupleType = tuple ListType = list DictType = DictionaryType = dict Modified: python/branches/py3k/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/_ctypes.c (original) +++ python/branches/py3k/Modules/_ctypes/_ctypes.c Mon Oct 8 04:46:15 2007 @@ -739,18 +739,12 @@ { char *ptr; Py_ssize_t size; - int rel = 0; Py_buffer view; - if (PyBuffer_Check(value)) { - if (PyObject_GetBuffer(value, &view, PyBUF_SIMPLE) < 0) - return -1; - size = view.len; - ptr = view.buf; - rel = 1; - } else if (-1 == PyString_AsStringAndSize(value, &ptr, &size)) { + if (PyObject_GetBuffer(value, &view, PyBUF_SIMPLE) < 0) return -1; - } + size = view.len; + ptr = view.buf; if (size > self->b_size) { PyErr_SetString(PyExc_ValueError, "string too long"); @@ -759,12 +753,10 @@ memcpy(self->b_ptr, ptr, size); - if (rel) - PyObject_ReleaseBuffer(value, &view); + PyObject_ReleaseBuffer(value, &view); return 0; fail: - if (rel) - PyObject_ReleaseBuffer(value, &view); + PyObject_ReleaseBuffer(value, &view); return -1; } Modified: python/branches/py3k/Modules/_sqlite/connection.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/connection.c (original) +++ python/branches/py3k/Modules/_sqlite/connection.c Mon Oct 8 04:46:15 2007 @@ -425,16 +425,16 @@ sqlite3_result_int64(context, (PY_LONG_LONG)longval); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); - } else if (PyBuffer_Check(py_val)) { + } else if (PyString_Check(py_val)) { + sqlite3_result_text(context, PyString_AsString(py_val), -1, SQLITE_TRANSIENT); + } else if (PyUnicode_Check(py_val)) { + sqlite3_result_text(context, PyUnicode_AsString(py_val), -1, SQLITE_TRANSIENT); + } else if (PyObject_CheckBuffer(py_val)) { if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); } else { sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } - } else if (PyString_Check(py_val)) { - sqlite3_result_text(context, PyString_AsString(py_val), -1, SQLITE_TRANSIENT); - } else if (PyUnicode_Check(py_val)) { - sqlite3_result_text(context, PyUnicode_AsString(py_val), -1, SQLITE_TRANSIENT); } else { /* TODO: raise error */ } @@ -478,16 +478,8 @@ break; case SQLITE_BLOB: buflen = sqlite3_value_bytes(cur_value); - cur_py_value = PyBuffer_New(buflen); - if (!cur_py_value) { - break; - } - if (PyObject_AsWriteBuffer(cur_py_value, &raw_buffer, &buflen)) { - Py_DECREF(cur_py_value); - cur_py_value = NULL; - break; - } - memcpy(raw_buffer, sqlite3_value_blob(cur_value), buflen); + cur_py_value = PyBytes_FromStringAndSize( + sqlite3_value_blob(cur_value), buflen); break; case SQLITE_NULL: default: Modified: python/branches/py3k/Modules/_sqlite/cursor.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/cursor.c (original) +++ python/branches/py3k/Modules/_sqlite/cursor.c Mon Oct 8 04:46:15 2007 @@ -380,14 +380,11 @@ } else { /* coltype == SQLITE_BLOB */ nbytes = sqlite3_column_bytes(self->statement->st, i); - buffer = PyBuffer_New(nbytes); + buffer = PyBytes_FromStringAndSize( + sqlite3_column_blob(self->statement->st, i), nbytes); if (!buffer) { break; } - if (PyObject_AsWriteBuffer(buffer, &raw_buffer, &nbytes)) { - break; - } - memcpy(raw_buffer, sqlite3_column_blob(self->statement->st, i), nbytes); converted = buffer; } } Modified: python/branches/py3k/Modules/_sqlite/statement.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/statement.c (original) +++ python/branches/py3k/Modules/_sqlite/statement.c Mon Oct 8 04:46:15 2007 @@ -102,13 +102,6 @@ #endif } else if (PyFloat_Check(parameter)) { rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); - } else if (PyBuffer_Check(parameter)) { - if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { - rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT); - } else { - PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - rc = -1; - } } else if PyString_Check(parameter) { string = PyString_AsString(parameter); rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); @@ -118,6 +111,13 @@ rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); Py_DECREF(stringval); + } else if (PyObject_CheckBuffer(parameter)) { + if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { + rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT); + } else { + PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); + rc = -1; + } } else { rc = -1; } Modified: python/branches/py3k/Python/bltinmodule.c ============================================================================== --- python/branches/py3k/Python/bltinmodule.c (original) +++ python/branches/py3k/Python/bltinmodule.c Mon Oct 8 04:46:15 2007 @@ -1787,8 +1787,7 @@ SETBUILTIN("True", Py_True); SETBUILTIN("basestring", &PyBaseString_Type); SETBUILTIN("bool", &PyBool_Type); - SETBUILTIN("buffer", &PyBuffer_Type); - SETBUILTIN("memoryview", &PyMemoryView_Type); + SETBUILTIN("memoryview", &PyMemoryView_Type); SETBUILTIN("bytes", &PyBytes_Type); SETBUILTIN("classmethod", &PyClassMethod_Type); #ifndef WITHOUT_COMPLEX From python-3000-checkins at python.org Mon Oct 8 05:32:35 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 8 Oct 2007 05:32:35 +0200 (CEST) Subject: [Python-3000-checkins] r58362 - in python/branches/py3k: Include/Python.h Include/bufferobject.h Makefile.pre.in Objects/bufferobject.c Objects/memoryobject.c Objects/unicodeobject.c Message-ID: <20071008033235.99B021E4015@bag.python.org> Author: guido.van.rossum Date: Mon Oct 8 05:32:34 2007 New Revision: 58362 Removed: python/branches/py3k/Include/bufferobject.h python/branches/py3k/Objects/bufferobject.c Modified: python/branches/py3k/Include/Python.h python/branches/py3k/Makefile.pre.in python/branches/py3k/Objects/memoryobject.c python/branches/py3k/Objects/unicodeobject.c Log: Delete bufferobject.[ch]. This will undoubtedly require Windows build file changes too. Modified: python/branches/py3k/Include/Python.h ============================================================================== --- python/branches/py3k/Include/Python.h (original) +++ python/branches/py3k/Include/Python.h Mon Oct 8 05:32:34 2007 @@ -76,7 +76,6 @@ #endif #include "rangeobject.h" #include "stringobject.h" -#include "bufferobject.h" #include "memoryobject.h" #include "tupleobject.h" #include "listobject.h" Deleted: /python/branches/py3k/Include/bufferobject.h ============================================================================== --- /python/branches/py3k/Include/bufferobject.h Mon Oct 8 05:32:34 2007 +++ (empty file) @@ -1,33 +0,0 @@ - -/* Buffer object interface */ - -/* Note: the object's structure is private */ - -#ifndef Py_BUFFEROBJECT_H -#define Py_BUFFEROBJECT_H -#ifdef __cplusplus -extern "C" { -#endif - - -PyAPI_DATA(PyTypeObject) PyBuffer_Type; - -#define PyBuffer_Check(op) (Py_Type(op) == &PyBuffer_Type) - -#define Py_END_OF_BUFFER (-1) - -PyAPI_FUNC(PyObject *) PyBuffer_FromObject(PyObject *base, - Py_ssize_t offset, Py_ssize_t size); -PyAPI_FUNC(PyObject *) PyBuffer_FromReadWriteObject(PyObject *base, - Py_ssize_t offset, - Py_ssize_t size); - -PyAPI_FUNC(PyObject *) PyBuffer_FromMemory(void *ptr, Py_ssize_t size); -PyAPI_FUNC(PyObject *) PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size); - -PyAPI_FUNC(PyObject *) PyBuffer_New(Py_ssize_t size); - -#ifdef __cplusplus -} -#endif -#endif /* !Py_BUFFEROBJECT_H */ Modified: python/branches/py3k/Makefile.pre.in ============================================================================== --- python/branches/py3k/Makefile.pre.in (original) +++ python/branches/py3k/Makefile.pre.in Mon Oct 8 05:32:34 2007 @@ -285,7 +285,6 @@ OBJECT_OBJS= \ Objects/abstract.o \ Objects/boolobject.o \ - Objects/bufferobject.o \ Objects/bytesobject.o \ Objects/cellobject.o \ Objects/classobject.o \ @@ -530,7 +529,6 @@ Include/asdl.h \ Include/abstract.h \ Include/boolobject.h \ - Include/bufferobject.h \ Include/bytesobject.h \ Include/ceval.h \ Include/classobject.h \ Deleted: /python/branches/py3k/Objects/bufferobject.c ============================================================================== --- /python/branches/py3k/Objects/bufferobject.c Mon Oct 8 05:32:34 2007 +++ (empty file) @@ -1,724 +0,0 @@ - -/* Buffer object implementation */ - -#include "Python.h" - - -typedef struct { - PyObject_HEAD - PyObject *b_base; - void *b_ptr; - Py_ssize_t b_size; - Py_ssize_t b_offset; - int b_readonly; - long b_hash; -} PyBufferObject; - - -static int -get_buf(PyBufferObject *self, Py_buffer *view, int flags) -{ - if (self->b_base == NULL) { - view->buf = self->b_ptr; - view->len = self->b_size; - view->readonly = 0; - } - else { - Py_ssize_t count, offset; - PyBufferProcs *bp = self->b_base->ob_type->tp_as_buffer; - if ((*bp->bf_getbuffer)(self->b_base, view, flags) < 0) - return 0; - count = view->len; - /* apply constraints to the start/end */ - if (self->b_offset > count) - offset = count; - else - offset = self->b_offset; - view->buf = (char*)view->buf + offset; - if (self->b_size == Py_END_OF_BUFFER) - view->len = count; - else - view->len = self->b_size; - if (offset + view->len > count) - view->len = count - offset; - } - return 1; -} - - -static int -buffer_getbuf(PyBufferObject *self, Py_buffer *view, int flags) -{ - if (view == NULL) return 0; - if (!get_buf(self, view, flags)) - return -1; - return PyBuffer_FillInfo(view, view->buf, view->len, self->b_readonly, - flags); -} - - -static void -buffer_releasebuf(PyBufferObject *self, Py_buffer *view) -{ - /* No-op if there is no self->b_base */ - if (self->b_base != NULL) { - PyBufferProcs *bp = self->b_base->ob_type->tp_as_buffer; - if (bp->bf_releasebuffer != NULL) { - (*bp->bf_releasebuffer)(self->b_base, view); - } - } -} - -static PyObject * -buffer_from_memory(PyObject *base, Py_ssize_t size, Py_ssize_t offset, - void *ptr, int readonly) -{ - PyBufferObject * b; - - if (size < 0 && size != Py_END_OF_BUFFER) { - PyErr_SetString(PyExc_ValueError, - "size must be zero or positive"); - return NULL; - } - if (offset < 0) { - PyErr_SetString(PyExc_ValueError, - "offset must be zero or positive"); - return NULL; - } - - b = PyObject_NEW(PyBufferObject, &PyBuffer_Type); - if (b == NULL) - return NULL; - - Py_XINCREF(base); - b->b_base = base; - b->b_ptr = ptr; - b->b_size = size; - b->b_offset = offset; - b->b_readonly = readonly; - b->b_hash = -1; - - return (PyObject *) b; -} - -static PyObject * -buffer_from_object(PyObject *base, Py_ssize_t size, Py_ssize_t offset, - int readonly) -{ - if (offset < 0) { - PyErr_SetString(PyExc_ValueError, - "offset must be zero or positive"); - return NULL; - } - if (PyBuffer_Check(base) && (((PyBufferObject *)base)->b_base)) { - /* another buffer, refer to the base object */ - PyBufferObject *b = (PyBufferObject *)base; - if (b->b_size != Py_END_OF_BUFFER) { - Py_ssize_t base_size = b->b_size - offset; - if (base_size < 0) - base_size = 0; - if (size == Py_END_OF_BUFFER || size > base_size) - size = base_size; - } - offset += b->b_offset; - base = b->b_base; - } - return buffer_from_memory(base, size, offset, NULL, readonly); -} - - -PyObject * -PyBuffer_FromObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size) -{ - PyBufferProcs *pb = base->ob_type->tp_as_buffer; - - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_SetString(PyExc_TypeError, "buffer object expected"); - return NULL; - } - - return buffer_from_object(base, size, offset, 1); -} - -PyObject * -PyBuffer_FromReadWriteObject(PyObject *base, Py_ssize_t offset, - Py_ssize_t size) -{ - PyBufferProcs *pb = base->ob_type->tp_as_buffer; - - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_SetString(PyExc_TypeError, "buffer object expected"); - return NULL; - } - - return buffer_from_object(base, size, offset, 0); -} - -PyObject * -PyBuffer_FromMemory(void *ptr, Py_ssize_t size) -{ - return buffer_from_memory(NULL, size, 0, ptr, 1); -} - -PyObject * -PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size) -{ - return buffer_from_memory(NULL, size, 0, ptr, 0); -} - -PyObject * -PyBuffer_New(Py_ssize_t size) -{ - PyObject *o; - PyBufferObject * b; - - if (size < 0) { - PyErr_SetString(PyExc_ValueError, - "size must be zero or positive"); - return NULL; - } - /* XXX: check for overflow in multiply */ - /* Inline PyObject_New */ - o = (PyObject *)PyObject_MALLOC(sizeof(*b) + size); - if (o == NULL) - return PyErr_NoMemory(); - b = (PyBufferObject *) PyObject_INIT(o, &PyBuffer_Type); - - b->b_base = NULL; - b->b_ptr = (void *)(b + 1); - b->b_size = size; - b->b_offset = 0; - b->b_readonly = 0; - b->b_hash = -1; - - return o; -} - -/* Methods */ - -static PyObject * -buffer_new(PyTypeObject *type, PyObject *args, PyObject *kw) -{ - PyObject *ob; - Py_ssize_t offset = 0; - Py_ssize_t size = Py_END_OF_BUFFER; - - if (!_PyArg_NoKeywords("buffer()", kw)) - return NULL; - - if (!PyArg_ParseTuple(args, "O|nn:buffer", &ob, &offset, &size)) - return NULL; - return PyBuffer_FromObject(ob, offset, size); -} - -PyDoc_STRVAR(buffer_doc, -"buffer(object [, offset[, size]])\n\ -\n\ -Create a new buffer object which references the given object.\n\ -The buffer will reference a slice of the target object from the\n\ -start of the object (or at the specified offset). The slice will\n\ -extend to the end of the target object (or with the specified size)."); - - -static void -buffer_dealloc(PyBufferObject *self) -{ - Py_XDECREF(self->b_base); - PyObject_DEL(self); -} - -static int -get_bufx(PyObject *obj, Py_buffer *view, int flags) -{ - PyBufferProcs *bp; - - if (PyBuffer_Check(obj)) { - if (!get_buf((PyBufferObject *)obj, view, flags)) { - PyErr_Clear(); - return 0; - } - else - return 1; - } - bp = obj->ob_type->tp_as_buffer; - if (bp == NULL || - bp->bf_getbuffer == NULL) - return 0; - if ((*bp->bf_getbuffer)(obj, view, PyBUF_SIMPLE) < 0) - return 0; - return 1; -} - -static PyObject * -buffer_richcompare(PyObject *self, PyObject *other, int op) -{ - void *p1, *p2; - Py_ssize_t len1, len2, min_len; - int cmp, ok; - Py_buffer v1, v2; - - ok = 1; - if (!get_bufx(self, &v1, PyBUF_SIMPLE)) - ok = 0; - if (!get_bufx(other, &v2, PyBUF_SIMPLE)) { - if (ok) PyObject_ReleaseBuffer((PyObject *)self, &v1); - ok = 0; - } - if (!ok) { - /* If we can't get the buffers, - == and != are still defined - (and the objects are unequal) */ - PyObject *result; - if (op == Py_EQ) - result = Py_False; - else if (op == Py_NE) - result = Py_True; - else - result = Py_NotImplemented; - Py_INCREF(result); - return result; - } - len1 = v1.len; - len2 = v2.len; - p1 = v1.buf; - p2 = v2.buf; - min_len = (len1 < len2) ? len1 : len2; - cmp = memcmp(p1, p2, min_len); - if (cmp == 0) - cmp = (len1 < len2) ? -1 : - (len1 > len2) ? 1 : 0; - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(other, &v2); - return Py_CmpToRich(op, cmp); -} - -static PyObject * -buffer_repr(PyBufferObject *self) -{ - const char *status = self->b_readonly ? "read-only" : "read-write"; - - if (self->b_base == NULL) - return PyUnicode_FromFormat( - "<%s buffer ptr %p, size %zd at %p>", - status, - self->b_ptr, - self->b_size, - self); - else - return PyUnicode_FromFormat( - "<%s buffer for %p, size %zd, offset %zd at %p>", - status, - self->b_base, - self->b_size, - self->b_offset, - self); -} - -static long -buffer_hash(PyBufferObject *self) -{ - Py_buffer view; - register Py_ssize_t len; - register unsigned char *p; - register long x; - - if (self->b_hash != -1) - return self->b_hash; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - if (!(self->b_readonly)) { - PyErr_SetString(PyExc_TypeError, - "writable buffers are not hashable"); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - p = (unsigned char *) view.buf; - len = view.len; - x = *p << 7; - while (--len >= 0) - x = (1000003*x) ^ *p++; - x ^= view.len; - if (x == -1) - x = -2; - self->b_hash = x; - PyObject_ReleaseBuffer((PyObject *)self, &view); - return x; -} - -static PyObject * -buffer_str(PyBufferObject *self) -{ - Py_buffer view; - PyObject *res; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - res = PyString_FromStringAndSize((const char *)view.buf, view.len); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return res; -} - -/* Sequence methods */ - -static Py_ssize_t -buffer_length(PyBufferObject *self) -{ - Py_buffer view; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - PyObject_ReleaseBuffer((PyObject *)self, &view); - return view.len; -} - -static PyObject * -buffer_concat(PyBufferObject *self, PyObject *other) -{ - PyBufferProcs *pb = other->ob_type->tp_as_buffer; - char *p; - PyObject *ob; - Py_buffer view, view2; - - if (pb == NULL || - pb->bf_getbuffer == NULL) - { - PyErr_BadArgument(); - return NULL; - } - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - - /* optimize special case */ - /* XXX bad idea type-wise */ - if (view.len == 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - Py_INCREF(other); - return other; - } - - if (PyObject_GetBuffer((PyObject *)other, &view2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return NULL; - } - - /* XXX(nnorwitz): need to check for overflow! */ - ob = PyBytes_FromStringAndSize(NULL, view.len+view2.len); - if (ob == NULL) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return NULL; - } - p = PyBytes_AS_STRING(ob); - memcpy(p, view.buf, view.len); - memcpy(p + view.len, view2.buf, view2.len); - - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return ob; -} - -static PyObject * -buffer_repeat(PyBufferObject *self, Py_ssize_t count) -{ - PyObject *ob; - register char *p; - Py_buffer view; - - if (count < 0) - count = 0; - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - /* XXX(nnorwitz): need to check for overflow! */ - ob = PyBytes_FromStringAndSize(NULL, view.len * count); - if (ob == NULL) - return NULL; - - p = PyBytes_AS_STRING(ob); - while (count--) { - memcpy(p, view.buf, view.len); - p += view.len; - } - - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; -} - -static PyObject * -buffer_item(PyBufferObject *self, Py_ssize_t idx) -{ - Py_buffer view; - PyObject *ob; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - if (idx < 0 || idx >= view.len) { - PyErr_SetString(PyExc_IndexError, "buffer index out of range"); - return NULL; - } - ob = PyBytes_FromStringAndSize((char *)view.buf + idx, 1); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; -} - -static PyObject * -buffer_subscript(PyBufferObject *self, PyObject *item) -{ - Py_buffer view; - PyObject *ob; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - if (PyIndex_Check(item)) { - Py_ssize_t idx = PyNumber_AsSsize_t(item, PyExc_IndexError); - - if (idx == -1 && PyErr_Occurred()) - return NULL; - if (idx < 0) - idx += view.len; - if (idx < 0 || idx >= view.len) { - PyErr_SetString(PyExc_IndexError, - "buffer index out of range"); - return NULL; - } - ob = PyBytes_FromStringAndSize((char *)view.buf + idx, 1); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - else if (PySlice_Check(item)) { - Py_ssize_t start, stop, step, slicelength, cur, i; - - if (PySlice_GetIndicesEx((PySliceObject*)item, view.len, - &start, &stop, &step, &slicelength) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return NULL; - } - - if (slicelength <= 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return PyBytes_FromStringAndSize("", 0); - } - else if (step == 1) { - ob = PyBytes_FromStringAndSize((char *)view.buf + - start, stop - start); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - else { - char *source_buf = (char *)view.buf; - char *result_buf = (char *)PyMem_Malloc(slicelength); - - if (result_buf == NULL) - return PyErr_NoMemory(); - - for (cur = start, i = 0; i < slicelength; - cur += step, i++) { - result_buf[i] = source_buf[cur]; - } - - ob = PyBytes_FromStringAndSize(result_buf, - slicelength); - PyMem_Free(result_buf); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - } - else { - PyErr_SetString(PyExc_TypeError, - "sequence index must be integer"); - return NULL; - } -} - -static int -buffer_ass_item(PyBufferObject *self, Py_ssize_t idx, PyObject *other) -{ - PyBufferProcs *pb; - Py_buffer view, view2; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - - if (self->b_readonly || view.readonly) { - PyErr_SetString(PyExc_TypeError, - "buffer is read-only"); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - if (idx < 0 || idx >= view.len) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyErr_SetString(PyExc_IndexError, - "buffer assignment index out of range"); - return -1; - } - - pb = other ? other->ob_type->tp_as_buffer : NULL; - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_BadArgument(); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - if (PyObject_GetBuffer(other, &view2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - if (view.len != 1) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - PyErr_SetString(PyExc_TypeError, - "right operand must be a single byte"); - return -1; - } - - ((char *)(view.buf))[idx] = *((char *)(view2.buf)); - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return 0; -} - -static int -buffer_ass_subscript(PyBufferObject *self, PyObject *item, PyObject *value) -{ - Py_buffer v1; - - if (!get_buf(self, &v1, PyBUF_SIMPLE)) - return -1; - if (self->b_readonly || v1.readonly) { - PyErr_SetString(PyExc_TypeError, - "buffer is read-only"); - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - if (PyIndex_Check(item)) { - Py_ssize_t idx = PyNumber_AsSsize_t(item, PyExc_IndexError); - if (idx == -1 && PyErr_Occurred()) - return -1; - if (idx < 0) - idx += v1.len; - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return buffer_ass_item(self, idx, value); - } - else if (PySlice_Check(item)) { - Py_ssize_t start, stop, step, slicelength; - Py_buffer v2; - PyBufferProcs *pb; - - if (PySlice_GetIndicesEx((PySliceObject *)item, v1.len, - &start, &stop, &step, &slicelength) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - - pb = value ? value->ob_type->tp_as_buffer : NULL; - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyErr_BadArgument(); - return -1; - } - if ((*pb->bf_getbuffer)(value, &v2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - - if (v2.len != slicelength) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(value, &v2); - PyErr_SetString(PyExc_TypeError, "right operand" - " length must match slice length"); - return -1; - } - - if (slicelength == 0) - /* nothing to do */; - else if (step == 1) - memcpy((char *)v1.buf + start, v2.buf, slicelength); - else { - Py_ssize_t cur, i; - - for (cur = start, i = 0; i < slicelength; - cur += step, i++) { - ((char *)v1.buf)[cur] = ((char *)v2.buf)[i]; - } - } - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(value, &v2); - return 0; - } else { - PyErr_SetString(PyExc_TypeError, - "buffer indices must be integers"); - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } -} - -/* Buffer methods */ - -static PySequenceMethods buffer_as_sequence = { - (lenfunc)buffer_length, /*sq_length*/ - (binaryfunc)buffer_concat, /*sq_concat*/ - (ssizeargfunc)buffer_repeat, /*sq_repeat*/ - (ssizeargfunc)buffer_item, /*sq_item*/ - 0, /*sq_slice*/ - (ssizeobjargproc)buffer_ass_item, /*sq_ass_item*/ - 0, /*sq_ass_slice*/ -}; - -static PyMappingMethods buffer_as_mapping = { - (lenfunc)buffer_length, - (binaryfunc)buffer_subscript, - (objobjargproc)buffer_ass_subscript, -}; - -static PyBufferProcs buffer_as_buffer = { - (getbufferproc)buffer_getbuf, - (releasebufferproc)buffer_releasebuf, -}; - -PyTypeObject PyBuffer_Type = { - PyVarObject_HEAD_INIT(&PyType_Type, 0) - "buffer", - sizeof(PyBufferObject), - 0, - (destructor)buffer_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - (reprfunc)buffer_repr, /* tp_repr */ - 0, /* tp_as_number */ - &buffer_as_sequence, /* tp_as_sequence */ - &buffer_as_mapping, /* tp_as_mapping */ - (hashfunc)buffer_hash, /* tp_hash */ - 0, /* tp_call */ - (reprfunc)buffer_str, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - &buffer_as_buffer, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - buffer_doc, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - buffer_richcompare, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - buffer_new, /* tp_new */ -}; Modified: python/branches/py3k/Objects/memoryobject.c ============================================================================== --- python/branches/py3k/Objects/memoryobject.c (original) +++ python/branches/py3k/Objects/memoryobject.c Mon Oct 8 05:32:34 2007 @@ -8,6 +8,8 @@ { if (view != NULL) *view = self->view; + if (self->base == NULL) + return 0; return self->base->ob_type->tp_as_buffer->bf_getbuffer(self->base, NULL, PyBUF_FULL); } @@ -15,7 +17,8 @@ static void memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) { - PyObject_ReleaseBuffer(self->base, NULL); + if (self->base != NULL) + PyObject_ReleaseBuffer(self->base, NULL); } PyDoc_STRVAR(memory_doc, Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Mon Oct 8 05:32:34 2007 @@ -1051,6 +1051,7 @@ const char *errors) { PyObject *buffer = NULL, *unicode; + Py_buffer info; if (encoding == NULL) encoding = PyUnicode_GetDefaultEncoding(); @@ -1068,7 +1069,10 @@ return PyUnicode_DecodeASCII(s, size, errors); /* Decode via the codec registry */ - buffer = PyBuffer_FromMemory((void *)s, size); + buffer = NULL; + if (PyBuffer_FillInfo(&info, (void *)s, size, 1, PyBUF_SIMPLE) < 0) + goto onError; + buffer = PyMemoryView_FromMemory(&info); if (buffer == NULL) goto onError; unicode = PyCodec_Decode(buffer, encoding, errors); From python-3000-checkins at python.org Mon Oct 8 05:39:27 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 8 Oct 2007 05:39:27 +0200 (CEST) Subject: [Python-3000-checkins] r58363 - in python/branches/py3k: PC/VC6/pythoncore.dsp PC/os2emx/Makefile PC/os2emx/python25.def PCbuild/pythoncore.vcproj PCbuild8/pythoncore/pythoncore.vcproj Message-ID: <20071008033927.1D5521E4006@bag.python.org> Author: guido.van.rossum Date: Mon Oct 8 05:39:26 2007 New Revision: 58363 Modified: python/branches/py3k/PC/VC6/pythoncore.dsp python/branches/py3k/PC/os2emx/Makefile python/branches/py3k/PC/os2emx/python25.def python/branches/py3k/PCbuild/pythoncore.vcproj python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj Log: Remove more traces of bufferobject.[ch] from build files etc. Modified: python/branches/py3k/PC/VC6/pythoncore.dsp ============================================================================== --- python/branches/py3k/PC/VC6/pythoncore.dsp (original) +++ python/branches/py3k/PC/VC6/pythoncore.dsp Mon Oct 8 05:39:26 2007 @@ -221,10 +221,6 @@ # End Source File # Begin Source File -SOURCE=..\..\Objects\bufferobject.c -# End Source File -# Begin Source File - SOURCE=..\..\Objects\cellobject.c # End Source File # Begin Source File Modified: python/branches/py3k/PC/os2emx/Makefile ============================================================================== --- python/branches/py3k/PC/os2emx/Makefile (original) +++ python/branches/py3k/PC/os2emx/Makefile Mon Oct 8 05:39:26 2007 @@ -366,7 +366,6 @@ SRC.OBJECT= $(addprefix $(TOP), \ Objects/abstract.c \ Objects/boolobject.c \ - Objects/bufferobject.c \ Objects/cellobject.c \ Objects/classobject.c \ Objects/cobject.c \ Modified: python/branches/py3k/PC/os2emx/python25.def ============================================================================== --- python/branches/py3k/PC/os2emx/python25.def (original) +++ python/branches/py3k/PC/os2emx/python25.def Mon Oct 8 05:39:26 2007 @@ -181,14 +181,6 @@ "_Py_ZeroStruct" "_Py_TrueStruct" -; From python25_s.lib(bufferobject) - "PyBuffer_FromObject" - "PyBuffer_FromReadWriteObject" - "PyBuffer_FromMemory" - "PyBuffer_FromReadWriteMemory" - "PyBuffer_New" - "PyBuffer_Type" - ; From python25_s.lib(cellobject) "PyCell_New" "PyCell_Get" Modified: python/branches/py3k/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/py3k/PCbuild/pythoncore.vcproj (original) +++ python/branches/py3k/PCbuild/pythoncore.vcproj Mon Oct 8 05:39:26 2007 @@ -443,9 +443,6 @@ RelativePath="..\Objects\boolobject.c"> - - - - @@ -978,10 +974,6 @@ > - - From python-3000-checkins at python.org Mon Oct 8 07:28:00 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Mon, 8 Oct 2007 07:28:00 +0200 (CEST) Subject: [Python-3000-checkins] r58366 - python/branches/py3k/Modules/_sqlite/connection.c python/branches/py3k/Modules/_sqlite/cursor.c Message-ID: <20071008052800.BF80F1E4006@bag.python.org> Author: neal.norwitz Date: Mon Oct 8 07:28:00 2007 New Revision: 58366 Modified: python/branches/py3k/Modules/_sqlite/connection.c python/branches/py3k/Modules/_sqlite/cursor.c Log: Remove unused variables Modified: python/branches/py3k/Modules/_sqlite/connection.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/connection.c (original) +++ python/branches/py3k/Modules/_sqlite/connection.c Mon Oct 8 07:28:00 2007 @@ -449,7 +449,6 @@ const char* val_str; PY_LONG_LONG val_int; Py_ssize_t buflen; - void* raw_buffer; args = PyTuple_New(argc); if (!args) { Modified: python/branches/py3k/Modules/_sqlite/cursor.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/cursor.c (original) +++ python/branches/py3k/Modules/_sqlite/cursor.c Mon Oct 8 07:28:00 2007 @@ -296,7 +296,6 @@ PyObject* converted; Py_ssize_t nbytes; PyObject* buffer; - void* raw_buffer; const char* val_str; char buf[200]; const char* colname; From python-3000-checkins at python.org Mon Oct 8 09:00:29 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Mon, 8 Oct 2007 09:00:29 +0200 (CEST) Subject: [Python-3000-checkins] r58367 - python/branches/py3k/Doc/library/functions.rst Message-ID: <20071008070029.80ED21E4006@bag.python.org> Author: georg.brandl Date: Mon Oct 8 09:00:29 2007 New Revision: 58367 Modified: python/branches/py3k/Doc/library/functions.rst Log: Remove buffer() from the docs. Modified: python/branches/py3k/Doc/library/functions.rst ============================================================================== --- python/branches/py3k/Doc/library/functions.rst (original) +++ python/branches/py3k/Doc/library/functions.rst Mon Oct 8 09:00:29 2007 @@ -1079,34 +1079,6 @@ returns an empty iterator. -.. % --------------------------------------------------------------------------- - - -.. _non-essential-built-in-funcs: - -Non-essential Built-in Functions -================================ - -There are several built-in functions that are no longer essential to learn, know -or use in modern Python programming. They have been kept here to maintain -backwards compatibility with programs written for older versions of Python. - -Python programmers, trainers, students and bookwriters should feel free to -bypass these functions without concerns about missing something important. - - -.. XXX does this go away? -.. function:: buffer(object[, offset[, size]]) - - The *object* argument must be an object that supports the buffer call interface - (such as strings, arrays, and buffers). A new buffer object will be created - which references the *object* argument. The buffer object will be a slice from - the beginning of *object* (or from the specified *offset*). The slice will - extend to the end of *object* (or will have a length given by the *size* - argument). - - - .. rubric:: Footnotes .. [#] Specifying a buffer size currently has no effect on systems that don't have From python-3000-checkins at python.org Mon Oct 8 16:08:36 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Mon, 8 Oct 2007 16:08:36 +0200 (CEST) Subject: [Python-3000-checkins] r58375 - python/branches/py3k/Doc/tutorial/datastructures.rst Message-ID: <20071008140836.E23191E4006@bag.python.org> Author: georg.brandl Date: Mon Oct 8 16:08:36 2007 New Revision: 58375 Modified: python/branches/py3k/Doc/tutorial/datastructures.rst Log: #1228: new comparison behavior. Modified: python/branches/py3k/Doc/tutorial/datastructures.rst ============================================================================== --- python/branches/py3k/Doc/tutorial/datastructures.rst (original) +++ python/branches/py3k/Doc/tutorial/datastructures.rst Mon Oct 8 16:08:36 2007 @@ -560,15 +560,8 @@ (1, 2, 3) == (1.0, 2.0, 3.0) (1, 2, ('aa', 'ab')) < (1, 2, ('abc', 'a'), 4) -Note that comparing objects of different types is legal. The outcome is -deterministic but arbitrary: the types are ordered by their name. Thus, a list -is always smaller than a string, a string is always smaller than a tuple, etc. -[#]_ Mixed numeric types are compared according to their numeric value, so 0 -equals 0.0, etc. - - -.. rubric:: Footnotes - -.. [#] The rules for comparing objects of different types should not be relied upon; - they may change in a future version of the language. - +Note that comparing objects of different types with ``<`` or ``>`` is legal +provided that the objects have appropriate comparison methods. For example, +mixed numeric types are compared according to their numeric value, so 0 equals +0.0, etc. Otherwise, rather than providing an arbitrary ordering, the +interpreter will raise a :exc:`TypeError` exception. From python-3000-checkins at python.org Mon Oct 8 21:48:15 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 8 Oct 2007 21:48:15 +0200 (CEST) Subject: [Python-3000-checkins] r58378 - python/branches/py3k/Lib/ftplib.py Message-ID: <20071008194815.C3E0F1E4007@bag.python.org> Author: brett.cannon Date: Mon Oct 8 21:48:15 2007 New Revision: 58378 Modified: python/branches/py3k/Lib/ftplib.py Log: Change the default encoding for ftplib.FTP to latin1 so that bytes can be in the range of 255 instead of only 128. Modified: python/branches/py3k/Lib/ftplib.py ============================================================================== --- python/branches/py3k/Lib/ftplib.py (original) +++ python/branches/py3k/Lib/ftplib.py Mon Oct 8 21:48:15 2007 @@ -103,7 +103,7 @@ file = None welcome = None passiveserver = 1 - encoding = "ASCII" + encoding = "latin1" # Initialization method (called by class instantiation). # Initialize host to localhost, port to standard ftp port From g.brandl at gmx.net Mon Oct 8 21:51:46 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 08 Oct 2007 21:51:46 +0200 Subject: [Python-3000-checkins] r58378 - python/branches/py3k/Lib/ftplib.py In-Reply-To: <20071008194815.C3E0F1E4007@bag.python.org> References: <20071008194815.C3E0F1E4007@bag.python.org> Message-ID: brett.cannon schrieb: > Author: brett.cannon > Date: Mon Oct 8 21:48:15 2007 > New Revision: 58378 > > Modified: > python/branches/py3k/Lib/ftplib.py > Log: > Change the default encoding for ftplib.FTP to latin1 so that bytes can be in > the range of 255 instead of only 128. Shouldn't FTP use bytes in the end? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From brett at python.org Mon Oct 8 22:14:29 2007 From: brett at python.org (Brett Cannon) Date: Mon, 8 Oct 2007 13:14:29 -0700 Subject: [Python-3000-checkins] r58378 - python/branches/py3k/Lib/ftplib.py In-Reply-To: References: <20071008194815.C3E0F1E4007@bag.python.org> Message-ID: On 10/8/07, Georg Brandl wrote: > brett.cannon schrieb: > > Author: brett.cannon > > Date: Mon Oct 8 21:48:15 2007 > > New Revision: 58378 > > > > Modified: > > python/branches/py3k/Lib/ftplib.py > > Log: > > Change the default encoding for ftplib.FTP to latin1 so that bytes can be in > > the range of 255 instead of only 128. > > Shouldn't FTP use bytes in the end? Yes, but I have other Python stuff to do that are more pressing. I filed issue1248 about doing a rewrite. -Brett From python-3000-checkins at python.org Tue Oct 9 05:46:31 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 9 Oct 2007 05:46:31 +0200 (CEST) Subject: [Python-3000-checkins] r58383 - python/branches/py3k/Objects/stringobject.c Message-ID: <20071009034631.134331E4008@bag.python.org> Author: guido.van.rossum Date: Tue Oct 9 05:46:30 2007 New Revision: 58383 Modified: python/branches/py3k/Objects/stringobject.c Log: Replace all (locale-dependent) uses of isupper(), tolower(), etc., by locally-defined macros that assume ASCII and only consider ASCII letters. Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Tue Oct 9 05:46:30 2007 @@ -4,7 +4,256 @@ #include "Python.h" -#include +/* Our own locale-independent ctype.h-like macros */ +/* XXX Move into a header file? */ + +#define FLAG_LOWER 0x01 +#define FLAG_UPPER 0x02 +#define FLAG_ALPHA (FLAG_LOWER|FLAG_UPPER) +#define FLAG_DIGIT 0x04 +#define FLAG_ALNUM (FLAG_ALPHA|FLAG_DIGIT) +#define FLAG_SPACE 0x08 +#define FLAG_XDIGIT 0x10 + +static unsigned int ctype_table[256] = { + 0, /* 0x0 '\x00' */ + 0, /* 0x1 '\x01' */ + 0, /* 0x2 '\x02' */ + 0, /* 0x3 '\x03' */ + 0, /* 0x4 '\x04' */ + 0, /* 0x5 '\x05' */ + 0, /* 0x6 '\x06' */ + 0, /* 0x7 '\x07' */ + 0, /* 0x8 '\x08' */ + FLAG_SPACE, /* 0x9 '\t' */ + FLAG_SPACE, /* 0xa '\n' */ + FLAG_SPACE, /* 0xb '\v' */ + FLAG_SPACE, /* 0xc '\f' */ + FLAG_SPACE, /* 0xd '\r' */ + 0, /* 0xe '\x0e' */ + 0, /* 0xf '\x0f' */ + 0, /* 0x10 '\x10' */ + 0, /* 0x11 '\x11' */ + 0, /* 0x12 '\x12' */ + 0, /* 0x13 '\x13' */ + 0, /* 0x14 '\x14' */ + 0, /* 0x15 '\x15' */ + 0, /* 0x16 '\x16' */ + 0, /* 0x17 '\x17' */ + 0, /* 0x18 '\x18' */ + 0, /* 0x19 '\x19' */ + 0, /* 0x1a '\x1a' */ + 0, /* 0x1b '\x1b' */ + 0, /* 0x1c '\x1c' */ + 0, /* 0x1d '\x1d' */ + 0, /* 0x1e '\x1e' */ + 0, /* 0x1f '\x1f' */ + FLAG_SPACE, /* 0x20 ' ' */ + 0, /* 0x21 '!' */ + 0, /* 0x22 '"' */ + 0, /* 0x23 '#' */ + 0, /* 0x24 '$' */ + 0, /* 0x25 '%' */ + 0, /* 0x26 '&' */ + 0, /* 0x27 "'" */ + 0, /* 0x28 '(' */ + 0, /* 0x29 ')' */ + 0, /* 0x2a '*' */ + 0, /* 0x2b '+' */ + 0, /* 0x2c ',' */ + 0, /* 0x2d '-' */ + 0, /* 0x2e '.' */ + 0, /* 0x2f '/' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x30 '0' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x31 '1' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x32 '2' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x33 '3' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x34 '4' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x35 '5' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x36 '6' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x37 '7' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x38 '8' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x39 '9' */ + 0, /* 0x3a ':' */ + 0, /* 0x3b ';' */ + 0, /* 0x3c '<' */ + 0, /* 0x3d '=' */ + 0, /* 0x3e '>' */ + 0, /* 0x3f '?' */ + 0, /* 0x40 '@' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x41 'A' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x42 'B' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x43 'C' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x44 'D' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x45 'E' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x46 'F' */ + FLAG_UPPER, /* 0x47 'G' */ + FLAG_UPPER, /* 0x48 'H' */ + FLAG_UPPER, /* 0x49 'I' */ + FLAG_UPPER, /* 0x4a 'J' */ + FLAG_UPPER, /* 0x4b 'K' */ + FLAG_UPPER, /* 0x4c 'L' */ + FLAG_UPPER, /* 0x4d 'M' */ + FLAG_UPPER, /* 0x4e 'N' */ + FLAG_UPPER, /* 0x4f 'O' */ + FLAG_UPPER, /* 0x50 'P' */ + FLAG_UPPER, /* 0x51 'Q' */ + FLAG_UPPER, /* 0x52 'R' */ + FLAG_UPPER, /* 0x53 'S' */ + FLAG_UPPER, /* 0x54 'T' */ + FLAG_UPPER, /* 0x55 'U' */ + FLAG_UPPER, /* 0x56 'V' */ + FLAG_UPPER, /* 0x57 'W' */ + FLAG_UPPER, /* 0x58 'X' */ + FLAG_UPPER, /* 0x59 'Y' */ + FLAG_UPPER, /* 0x5a 'Z' */ + 0, /* 0x5b '[' */ + 0, /* 0x5c '\\' */ + 0, /* 0x5d ']' */ + 0, /* 0x5e '^' */ + 0, /* 0x5f '_' */ + 0, /* 0x60 '`' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x61 'a' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x62 'b' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x63 'c' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x64 'd' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x65 'e' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x66 'f' */ + FLAG_LOWER, /* 0x67 'g' */ + FLAG_LOWER, /* 0x68 'h' */ + FLAG_LOWER, /* 0x69 'i' */ + FLAG_LOWER, /* 0x6a 'j' */ + FLAG_LOWER, /* 0x6b 'k' */ + FLAG_LOWER, /* 0x6c 'l' */ + FLAG_LOWER, /* 0x6d 'm' */ + FLAG_LOWER, /* 0x6e 'n' */ + FLAG_LOWER, /* 0x6f 'o' */ + FLAG_LOWER, /* 0x70 'p' */ + FLAG_LOWER, /* 0x71 'q' */ + FLAG_LOWER, /* 0x72 'r' */ + FLAG_LOWER, /* 0x73 's' */ + FLAG_LOWER, /* 0x74 't' */ + FLAG_LOWER, /* 0x75 'u' */ + FLAG_LOWER, /* 0x76 'v' */ + FLAG_LOWER, /* 0x77 'w' */ + FLAG_LOWER, /* 0x78 'x' */ + FLAG_LOWER, /* 0x79 'y' */ + FLAG_LOWER, /* 0x7a 'z' */ + 0, /* 0x7b '{' */ + 0, /* 0x7c '|' */ + 0, /* 0x7d '}' */ + 0, /* 0x7e '~' */ + 0, /* 0x7f '\x7f' */ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; + +#define ISLOWER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_LOWER) +#define ISUPPER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_UPPER) +#define ISALPHA(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALPHA) +#define ISDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_DIGIT) +#define ISXDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_XDIGIT) +#define ISALNUM(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALNUM) +#define ISSPACE(c) (ctype_table[Py_CHARMASK(c)] & FLAG_SPACE) + +#undef islower +#define islower(c) undefined_islower(c) +#undef isupper +#define isupper(c) undefined_isupper(c) +#undef isalpha +#define isalpha(c) undefined_isalpha(c) +#undef isdigit +#define isdigit(c) undefined_isdigit(c) +#undef isxdigit +#define isxdigit(c) undefined_isxdigit(c) +#undef isalnum +#define isalnum(c) undefined_isalnum(c) +#undef isspace +#define isspace(c) undefined_isspace(c) + +static unsigned char ctype_tolower[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + +static unsigned char ctype_toupper[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + +#define TOLOWER(c) (ctype_tolower[Py_CHARMASK(c)]) +#define TOUPPER(c) (ctype_toupper[Py_CHARMASK(c)]) + +#undef tolower +#define tolower(c) undefined_tolower(c) +#undef toupper +#define toupper(c) undefined_toupper(c) #ifdef COUNT_ALLOCS int null_strings, one_strings; @@ -173,7 +422,7 @@ for (f = format; *f; f++) { if (*f == '%') { const char* p = f; - while (*++f && *f != '%' && !isalpha(Py_CHARMASK(*f))) + while (*++f && *f != '%' && !ISALPHA(*f)) ; /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since @@ -242,15 +491,15 @@ /* parse the width.precision part (we're only interested in the precision value, if any) */ n = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) n = (n*10) + *f++ - '0'; if (*f == '.') { f++; n = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) n = (n*10) + *f++ - '0'; } - while (*f && *f != '%' && !isalpha(Py_CHARMASK(*f))) + while (*f && *f != '%' && !ISALPHA(*f)) f++; /* handle the long flag, but only for %ld and %lu. others can be added when necessary. */ @@ -606,23 +855,22 @@ *p++ = c; break; case 'x': - if (isxdigit(Py_CHARMASK(s[0])) - && isxdigit(Py_CHARMASK(s[1]))) { + if (ISXDIGIT(s[0]) && ISXDIGIT(s[1])) { unsigned int x = 0; c = Py_CHARMASK(*s); s++; - if (isdigit(c)) + if (ISDIGIT(c)) x = c - '0'; - else if (islower(c)) + else if (ISLOWER(c)) x = 10 + c - 'a'; else x = 10 + c - 'A'; x = x << 4; c = Py_CHARMASK(*s); s++; - if (isdigit(c)) + if (ISDIGIT(c)) x += c - '0'; - else if (islower(c)) + else if (ISLOWER(c)) x += 10 + c - 'a'; else x += 10 + c - 'A'; @@ -1250,10 +1498,10 @@ /* Always force the list to the expected size. */ #define FIX_PREALLOC_SIZE(list) Py_Size(list) = count -#define SKIP_SPACE(s, i, len) { while (i=0 && isspace(Py_CHARMASK(s[i]))) i--; } -#define RSKIP_NONSPACE(s, i) { while (i>=0 && !isspace(Py_CHARMASK(s[i]))) i--; } +#define SKIP_SPACE(s, i, len) { while (i=0 && ISSPACE(s[i])) i--; } +#define RSKIP_NONSPACE(s, i) { while (i>=0 && !ISSPACE(s[i])) i--; } Py_LOCAL_INLINE(PyObject *) split_whitespace(const char *s, Py_ssize_t len, Py_ssize_t maxsplit) @@ -1869,7 +2117,7 @@ i = 0; if (striptype != RIGHTSTRIP) { - while (i < len && isspace(Py_CHARMASK(s[i]))) { + while (i < len && ISSPACE(s[i])) { i++; } } @@ -1878,7 +2126,7 @@ if (striptype != LEFTSTRIP) { do { j--; - } while (j >= i && isspace(Py_CHARMASK(s[j]))); + } while (j >= i && ISSPACE(s[j])); j++; } @@ -1979,11 +2227,6 @@ \n\ Return a copy of the string S converted to lowercase."); -/* _tolower and _toupper are defined by SUSv2, but they're not ISO C */ -#ifndef _tolower -#define _tolower tolower -#endif - static PyObject * string_lower(PyStringObject *self) { @@ -2001,8 +2244,8 @@ for (i = 0; i < n; i++) { int c = Py_CHARMASK(s[i]); - if (isupper(c)) - s[i] = _tolower(c); + if (ISUPPER(c)) + s[i] = TOLOWER(c); } return newobj; @@ -2013,10 +2256,6 @@ \n\ Return a copy of the string S converted to uppercase."); -#ifndef _toupper -#define _toupper toupper -#endif - static PyObject * string_upper(PyStringObject *self) { @@ -2034,8 +2273,8 @@ for (i = 0; i < n; i++) { int c = Py_CHARMASK(s[i]); - if (islower(c)) - s[i] = _toupper(c); + if (ISLOWER(c)) + s[i] = TOUPPER(c); } return newobj; @@ -2061,13 +2300,13 @@ s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); - if (islower(c)) { + if (ISLOWER(c)) { if (!previous_is_cased) - c = toupper(c); + c = TOUPPER(c); previous_is_cased = 1; - } else if (isupper(c)) { + } else if (ISUPPER(c)) { if (previous_is_cased) - c = tolower(c); + c = TOLOWER(c); previous_is_cased = 1; } else previous_is_cased = 0; @@ -2095,16 +2334,16 @@ s_new = PyString_AsString(newobj); if (0 < n) { int c = Py_CHARMASK(*s++); - if (islower(c)) - *s_new = toupper(c); + if (ISLOWER(c)) + *s_new = TOUPPER(c); else *s_new = c; s_new++; } for (i = 1; i < n; i++) { int c = Py_CHARMASK(*s++); - if (isupper(c)) - *s_new = tolower(c); + if (ISUPPER(c)) + *s_new = TOLOWER(c); else *s_new = c; s_new++; @@ -2173,11 +2412,11 @@ s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); - if (islower(c)) { - *s_new = toupper(c); + if (ISLOWER(c)) { + *s_new = TOUPPER(c); } - else if (isupper(c)) { - *s_new = tolower(c); + else if (ISUPPER(c)) { + *s_new = TOLOWER(c); } else *s_new = c; @@ -3386,7 +3625,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1 && - isspace(*p)) + ISSPACE(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3395,7 +3634,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isspace(*p)) + if (!ISSPACE(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3417,7 +3656,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1 && - isalpha(*p)) + ISALPHA(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3426,7 +3665,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isalpha(*p)) + if (!ISALPHA(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3447,8 +3686,7 @@ register const unsigned char *e; /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - isalnum(*p)) + if (PyString_GET_SIZE(self) == 1 && ISALNUM(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3457,7 +3695,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isalnum(*p)) + if (!ISALNUM(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3478,8 +3716,7 @@ register const unsigned char *e; /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - isdigit(*p)) + if (PyString_GET_SIZE(self) == 1 && ISDIGIT(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3488,7 +3725,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isdigit(*p)) + if (!ISDIGIT(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3511,7 +3748,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(islower(*p) != 0); + return PyBool_FromLong(ISLOWER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3520,9 +3757,9 @@ e = p + PyString_GET_SIZE(self); cased = 0; for (; p < e; p++) { - if (isupper(*p)) + if (ISUPPER(*p)) return PyBool_FromLong(0); - else if (!cased && islower(*p)) + else if (!cased && ISLOWER(*p)) cased = 1; } return PyBool_FromLong(cased); @@ -3545,7 +3782,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(isupper(*p) != 0); + return PyBool_FromLong(ISUPPER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3554,9 +3791,9 @@ e = p + PyString_GET_SIZE(self); cased = 0; for (; p < e; p++) { - if (islower(*p)) + if (ISLOWER(*p)) return PyBool_FromLong(0); - else if (!cased && isupper(*p)) + else if (!cased && ISUPPER(*p)) cased = 1; } return PyBool_FromLong(cased); @@ -3581,7 +3818,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(isupper(*p) != 0); + return PyBool_FromLong(ISUPPER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3593,13 +3830,13 @@ for (; p < e; p++) { register const unsigned char ch = *p; - if (isupper(ch)) { + if (ISUPPER(ch)) { if (previous_is_cased) return PyBool_FromLong(0); previous_is_cased = 1; cased = 1; } - else if (islower(ch)) { + else if (ISLOWER(ch)) { if (!previous_is_cased) return PyBool_FromLong(0); previous_is_cased = 1; @@ -4434,11 +4671,11 @@ if (--fmtcnt >= 0) c = *fmt++; } - else if (c >= 0 && isdigit(c)) { + else if (c >= 0 && ISDIGIT(c)) { width = c - '0'; while (--fmtcnt >= 0) { c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) + if (!ISDIGIT(c)) break; if ((width*10) / 10 != width) { PyErr_SetString( @@ -4471,11 +4708,11 @@ if (--fmtcnt >= 0) c = *fmt++; } - else if (c >= 0 && isdigit(c)) { + else if (c >= 0 && ISDIGIT(c)) { prec = c - '0'; while (--fmtcnt >= 0) { c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) + if (!ISDIGIT(c)) break; if ((prec*10) / 10 != prec) { PyErr_SetString( From python-3000-checkins at python.org Tue Oct 9 19:21:10 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 9 Oct 2007 19:21:10 +0200 (CEST) Subject: [Python-3000-checkins] r58390 - in python/branches/py3k: Lib/test/test_bytes.py Objects/bytesobject.c Message-ID: <20071009172110.7EAD61E401F@bag.python.org> Author: guido.van.rossum Date: Tue Oct 9 19:21:10 2007 New Revision: 58390 Modified: python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Objects/bytesobject.c Log: Patch #1049 by Thomas Lee. Changes comparisons between PyBytes and PyUnicode to return unequal instead of raising TypeError. Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Tue Oct 9 19:21:10 2007 @@ -130,12 +130,14 @@ self.assertEqual(str8("abc") < b"ab", False) self.assertEqual(str8("abc") <= b"ab", False) - # Bytes can't be compared to Unicode! + # Byte comparisons with unicode should always fail! # Test this for all expected byte orders and Unicode character sizes - self.assertRaises(TypeError, lambda: b"\0a\0b\0c" == "abc") - self.assertRaises(TypeError, lambda: b"\0\0\0a\0\0\0b\0\0\0c" == "abc") - self.assertRaises(TypeError, lambda: b"a\0b\0c\0" == "abc") - self.assertRaises(TypeError, lambda: b"a\0\0\0b\0\0\0c\0\0\0" == "abc") + self.assertEqual(b"\0a\0b\0c" == "abc", False) + self.assertEqual(b"\0\0\0a\0\0\0b\0\0\0c" == "abc", False) + self.assertEqual(b"a\0b\0c\0" == "abc", False) + self.assertEqual(b"a\0\0\0b\0\0\0c\0\0\0" == "abc", False) + self.assertEqual(bytes() == str(), False) + self.assertEqual(bytes() != str(), True) def test_nohash(self): self.assertRaises(TypeError, hash, bytes()) Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Tue Oct 9 19:21:10 2007 @@ -964,8 +964,8 @@ error, even if the comparison is for equality. */ if (PyObject_IsInstance(self, (PyObject*)&PyUnicode_Type) || PyObject_IsInstance(other, (PyObject*)&PyUnicode_Type)) { - PyErr_SetString(PyExc_TypeError, "can't compare bytes and str"); - return NULL; + Py_INCREF(Py_NotImplemented); + return Py_NotImplemented; } self_size = _getbuffer(self, &self_bytes); From python-3000-checkins at python.org Tue Oct 9 19:30:03 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 9 Oct 2007 19:30:03 +0200 (CEST) Subject: [Python-3000-checkins] r58392 - python/branches/py3k/Tools/scripts/make_ctype.py Message-ID: <20071009173003.DE7751E400D@bag.python.org> Author: guido.van.rossum Date: Tue Oct 9 19:30:03 2007 New Revision: 58392 Added: python/branches/py3k/Tools/scripts/make_ctype.py (contents, props changed) Log: Silly script I'd rather not throw away. Added: python/branches/py3k/Tools/scripts/make_ctype.py ============================================================================== --- (empty file) +++ python/branches/py3k/Tools/scripts/make_ctype.py Tue Oct 9 19:30:03 2007 @@ -0,0 +1,93 @@ +"""Script that generates the ctype.h-replacement in stringobject.c.""" + +NAMES = ("LOWER", "UPPER", "ALPHA", "DIGIT", "XDIGIT", "ALNUM", "SPACE") + +print(""" +#define FLAG_LOWER 0x01 +#define FLAG_UPPER 0x02 +#define FLAG_ALPHA (FLAG_LOWER|FLAG_UPPER) +#define FLAG_DIGIT 0x04 +#define FLAG_ALNUM (FLAG_ALPHA|FLAG_DIGIT) +#define FLAG_SPACE 0x08 +#define FLAG_XDIGIT 0x10 + +static unsigned int ctype_table[256] = {""") + +for i in range(128): + c = chr(i) + flags = [] + for name in NAMES: + if name in ("ALPHA", "ALNUM"): + continue + if name == "XDIGIT": + method = lambda: c.isdigit() or c.upper() in "ABCDEF" + else: + method = getattr(c, "is" + name.lower()) + if method(): + flags.append("FLAG_" + name) + rc = repr(c) + if c == '\v': + rc = "'\\v'" + elif c == '\f': + rc = "'\\f'" + if not flags: + print(" 0, /* 0x%x %s */" % (i, rc)) + else: + print(" %s, /* 0x%x %s */" % ("|".join(flags), i, rc)) + +for i in range(128, 256, 16): + print(" %s," % ", ".join(16*["0"])) + +print("};") +print("") + +for name in NAMES: + print("#define IS%s(c) (ctype_table[Py_CHARMASK(c)] & FLAG_%s)" % + (name, name)) + +print("") + +for name in NAMES: + name = "is" + name.lower() + print("#undef %s" % name) + print("#define %s(c) undefined_%s(c)" % (name, name)) + +print(""" +static unsigned char ctype_tolower[256] = {""") + +for i in range(0, 256, 8): + values = [] + for i in range(i, i+8): + if i < 128: + c = chr(i) + if c.isupper(): + i = ord(c.lower()) + values.append("0x%02x" % i) + print(" %s," % ", ".join(values)) + +print("};") + +print(""" +static unsigned char ctype_toupper[256] = {""") + +for i in range(0, 256, 8): + values = [] + for i in range(i, i+8): + if i < 128: + c = chr(i) + if c.islower(): + i = ord(c.upper()) + values.append("0x%02x" % i) + print(" %s," % ", ".join(values)) + +print("};") + +print(""" +#define TOLOWER(c) (ctype_tolower[Py_CHARMASK(c)]) +#define TOUPPER(c) (ctype_toupper[Py_CHARMASK(c)]) + +#undef tolower +#define tolower(c) undefined_tolower(c) +#undef toupper +#define toupper(c) undefined_toupper(c) +""") From python-3000-checkins at python.org Tue Oct 9 20:35:17 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Tue, 9 Oct 2007 20:35:17 +0200 (CEST) Subject: [Python-3000-checkins] r58395 - python/branches/py3k/Lib/test/test_socket_ssl.py Message-ID: <20071009183517.885661E4029@bag.python.org> Author: gregory.p.smith Date: Tue Oct 9 20:35:13 2007 New Revision: 58395 Modified: python/branches/py3k/Lib/test/test_socket_ssl.py Log: use a reliable host Modified: python/branches/py3k/Lib/test/test_socket_ssl.py ============================================================================== --- python/branches/py3k/Lib/test/test_socket_ssl.py (original) +++ python/branches/py3k/Lib/test/test_socket_ssl.py Tue Oct 9 20:35:13 2007 @@ -41,10 +41,7 @@ # A service which issues a welcome banner (without need to write # anything). - # XXX ("gmail.org", 995) has been unreliable so far, from time to - # XXX time non-responsive for hours on end (& across all buildbot - # XXX slaves, so that's not just a local thing). - ADDR = "gmail.org", 995 + ADDR = "pop.gmail.com", 995 s = socket.socket() s.settimeout(30.0) From python-3000-checkins at python.org Tue Oct 9 23:55:58 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 9 Oct 2007 23:55:58 +0200 (CEST) Subject: [Python-3000-checkins] r58397 - python/branches/py3k/Objects/fileobject.c Message-ID: <20071009215558.8760F1E401E@bag.python.org> Author: guido.van.rossum Date: Tue Oct 9 23:55:58 2007 New Revision: 58397 Modified: python/branches/py3k/Objects/fileobject.c Log: PyFile_WriteObject() should use PyObject_Repr(), not _ReprStr8(). Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Tue Oct 9 23:55:58 2007 @@ -145,7 +145,7 @@ value = _PyObject_Str(v); } else - value = PyObject_ReprStr8(v); + value = PyObject_Repr(v); if (value == NULL) { Py_DECREF(writer); return -1; From python-3000-checkins at python.org Wed Oct 10 01:12:31 2007 From: python-3000-checkins at python.org (kurt.kaiser) Date: Wed, 10 Oct 2007 01:12:31 +0200 (CEST) Subject: [Python-3000-checkins] r58398 - python/branches/py3k/Lib/idlelib/IOBinding.py python/branches/py3k/Lib/idlelib/NEWS.txt Message-ID: <20071009231231.5CB601E400A@bag.python.org> Author: kurt.kaiser Date: Wed Oct 10 01:12:31 2007 New Revision: 58398 Modified: python/branches/py3k/Lib/idlelib/IOBinding.py python/branches/py3k/Lib/idlelib/NEWS.txt Log: Windows EOL sequence not converted correctly, encoding error. Caused file save to fail. Bug 1130. M idlelib/IOBinding.py M idlelib/NEWS.txt Modified: python/branches/py3k/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/py3k/Lib/idlelib/IOBinding.py (original) +++ python/branches/py3k/Lib/idlelib/IOBinding.py Wed Oct 10 01:12:31 2007 @@ -242,7 +242,6 @@ eol = r"(\r\n)|\n|\r" # \r\n (Windows), \n (UNIX), or \r (Mac) eol_re = re.compile(eol) - eol_convention = os.linesep # Default def loadfile(self, filename): try: @@ -389,9 +388,10 @@ def writefile(self, filename): self.fixlastline() - chars = self.encode(self.text.get("1.0", "end-1c")) + text = self.text.get("1.0", "end-1c") if self.eol_convention != "\n": - chars = chars.replace("\n", self.eol_convention) + text = text.replace("\n", self.eol_convention) + chars = self.encode(self.text.get("1.0", "end-1c")) try: f = open(filename, "wb") f.write(chars) Modified: python/branches/py3k/Lib/idlelib/NEWS.txt ============================================================================== --- python/branches/py3k/Lib/idlelib/NEWS.txt (original) +++ python/branches/py3k/Lib/idlelib/NEWS.txt Wed Oct 10 01:12:31 2007 @@ -1,3 +1,12 @@ +What's New in IDLE 3.0a2? +========================= + +*Release date: XX-XXX-2007* + +- Windows EOL sequence not converted correctly, encoding error. + Caused file save to fail. Bug 1130. + + What's New in IDLE 3.0a1? ========================= @@ -19,7 +28,7 @@ What's New in IDLE 2.6a1? ========================= -*Release date: XX-XXX-200X* UNRELEASED +*Release date: XX-XXX-200X* UNRELEASED, but merged into 3.0a1 - Corrected some bugs in AutoComplete. Also, Page Up/Down in ACW implemented; mouse and cursor selection in ACWindow implemented; double Tab inserts From python-3000-checkins at python.org Wed Oct 10 02:12:46 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 02:12:46 +0200 (CEST) Subject: [Python-3000-checkins] r58400 - python/branches/py3k/Lib/test/test_multibytecodec_support.py Message-ID: <20071010001246.9538E1E400E@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 02:12:46 2007 New Revision: 58400 Modified: python/branches/py3k/Lib/test/test_multibytecodec_support.py Log: Don't depend on str8.splitlines() in test_chunkcoding(). Modified: python/branches/py3k/Lib/test/test_multibytecodec_support.py ============================================================================== --- python/branches/py3k/Lib/test/test_multibytecodec_support.py (original) +++ python/branches/py3k/Lib/test/test_multibytecodec_support.py Wed Oct 10 02:12:46 2007 @@ -31,8 +31,14 @@ self.incrementaldecoder = self.codec.incrementaldecoder def test_chunkcoding(self): - for native, utf8 in zip(*[map(bytes, str8(f).splitlines(1)) - for f in self.tstring]): + tstring_lines = [] + for b in self.tstring: + lines = b.split(b"\n") + last = lines.pop() + assert last == b"" + lines = [line + b"\n" for line in lines] + tstring_lines.append(lines) + for native, utf8 in zip(*tstring_lines): u = self.decode(native)[0] self.assertEqual(u, utf8.decode('utf-8')) if self.roundtriptest: From python-3000-checkins at python.org Wed Oct 10 02:36:38 2007 From: python-3000-checkins at python.org (kurt.kaiser) Date: Wed, 10 Oct 2007 02:36:38 +0200 (CEST) Subject: [Python-3000-checkins] r58402 - python/branches/py3k/Lib/idlelib/IOBinding.py Message-ID: <20071010003638.E3D4F1E4014@bag.python.org> Author: kurt.kaiser Date: Wed Oct 10 02:36:38 2007 New Revision: 58402 Modified: python/branches/py3k/Lib/idlelib/IOBinding.py Log: the default class attr is needed! Modified: python/branches/py3k/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/py3k/Lib/idlelib/IOBinding.py (original) +++ python/branches/py3k/Lib/idlelib/IOBinding.py Wed Oct 10 02:36:38 2007 @@ -242,6 +242,7 @@ eol = r"(\r\n)|\n|\r" # \r\n (Windows), \n (UNIX), or \r (Mac) eol_re = re.compile(eol) + eol_convention = os.linesep # default def loadfile(self, filename): try: From python-3000-checkins at python.org Wed Oct 10 20:00:50 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 20:00:50 +0200 (CEST) Subject: [Python-3000-checkins] r58407 - python/branches/py3k/Lib/pickle.py Message-ID: <20071010180050.5EE971E400E@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 20:00:50 2007 New Revision: 58407 Modified: python/branches/py3k/Lib/pickle.py Log: Random changes having to do with readline() and bytes. Modified: python/branches/py3k/Lib/pickle.py ============================================================================== --- python/branches/py3k/Lib/pickle.py (original) +++ python/branches/py3k/Lib/pickle.py Wed Oct 10 20:00:50 2007 @@ -778,31 +778,17 @@ The protocol version of the pickle is detected automatically, so no proto argument is needed. - The file-like object must have two methods, a read() method that - takes an integer argument, and a readline() method that requires no - arguments. Both methods should return a string. Thus file-like - object can be a file object opened for reading, a StringIO object, - or any other custom object that meets this interface. + The file-like object must have two methods, a read() method + that takes an integer argument, and a readline() method that + requires no arguments. Both methods should return bytes. + Thus file-like object can be a binary file object opened for + reading, a BytesIO object, or any other custom object that + meets this interface. """ - try: - self.readline = file.readline - except AttributeError: - self.file = file + self.readline = file.readline self.read = file.read self.memo = {} - def readline(self): - # XXX Slow but at least correct - b = bytes() - while True: - c = self.file.read(1) - if not c: - break - b += c - if c == b'\n': - break - return b - def load(self): """Read a pickled object representation from the open file. @@ -895,7 +881,8 @@ dispatch[BININT2[0]] = load_binint2 def load_long(self): - self.append(int(str(self.readline()[:-1]), 0)) + val = self.readline()[:-1].decode("ascii") + self.append(int(val, 0)) dispatch[LONG[0]] = load_long def load_long1(self): @@ -1076,8 +1063,10 @@ def find_class(self, module, name): # Subclasses may override this - module = str(module) - name = str(name) + if isinstance(module, bytes): + module = module.decode("utf-8") + if isinstance(name, bytes): + name = name.decode("utf-8") __import__(module) mod = sys.modules[module] klass = getattr(mod, name) @@ -1110,7 +1099,7 @@ dispatch[DUP[0]] = load_dup def load_get(self): - self.append(self.memo[str8(self.readline())[:-1]]) + self.append(self.memo[str(self.readline())[:-1]]) dispatch[GET[0]] = load_get def load_binget(self): From python-3000-checkins at python.org Wed Oct 10 20:44:40 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 20:44:40 +0200 (CEST) Subject: [Python-3000-checkins] r58408 - python/branches/py3k/Parser/tokenizer.c Message-ID: <20071010184440.084761E4002@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 20:44:39 2007 New Revision: 58408 Modified: python/branches/py3k/Parser/tokenizer.c Log: Fix an issue in PyTokenizer_RestoreEncoding() which was treating a PyBytes object with PyString calls and not checking errors. This caused the display of syntax errors to be deformed. Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Wed Oct 10 20:44:39 2007 @@ -1556,7 +1556,10 @@ Py_DECREF(unicode_text); } if (!ret) { - PyErr_Print(); + PyErr_Clear(); + } + else { + assert(PyBytes_Check(ret)); } return ret; } @@ -1569,8 +1572,8 @@ /* convert source to original encondig */ PyObject *lineobj = dec_utf8(tok->encoding, tok->buf, len); if (lineobj != NULL) { - int linelen = PyString_Size(lineobj); - const char *line = PyString_AsString(lineobj); + int linelen = PyBytes_GET_SIZE(lineobj); + const char *line = PyBytes_AS_STRING(lineobj); text = PyObject_MALLOC(linelen + 1); if (text != NULL && line != NULL) { if (linelen) @@ -1582,9 +1585,11 @@ /* adjust error offset */ if (*offset > 1) { PyObject *offsetobj = dec_utf8(tok->encoding, - tok->buf, *offset-1); + tok->buf, + *offset-1); if (offsetobj) { - *offset = PyString_Size(offsetobj) + 1; + *offset = 1 + + PyBytes_GET_SIZE(offsetobj); Py_DECREF(offsetobj); } } From python-3000-checkins at python.org Wed Oct 10 20:49:50 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 20:49:50 +0200 (CEST) Subject: [Python-3000-checkins] r58409 - python/branches/py3k/Parser/tokenizer.c Message-ID: <20071010184950.54BEB1E4013@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 20:49:50 2007 New Revision: 58409 Modified: python/branches/py3k/Parser/tokenizer.c Log: Fix another stray PyString reference that should be PyBytes; and its comment. Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Wed Oct 10 20:49:50 2007 @@ -357,7 +357,7 @@ 1) NULL: need to call tok->decoding_readline to get a new line 2) PyUnicodeObject *: decoding_feof has called tok->decoding_readline and stored the result in tok->decoding_buffer - 3) PyStringObject *: previous call to fp_readl did not have enough room + 3) PyBytesObject *: previous call to fp_readl did not have enough room (in the s buffer) to copy entire contents of the line read by tok->decoding_readline. tok->decoding_buffer has the overflow. In this case, fp_readl is called in a loop (with an expanded buffer) @@ -625,7 +625,7 @@ utf8 = translate_into_utf8(str, tok->enc); if (utf8 == NULL) return error_ret(tok); - str = PyString_AsString(utf8); + str = PyBytes_AsString(utf8); } for (s = str;; s++) { if (*s == '\0') break; From python-3000-checkins at python.org Wed Oct 10 20:53:36 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 20:53:36 +0200 (CEST) Subject: [Python-3000-checkins] r58410 - python/branches/py3k/Python/errors.c Message-ID: <20071010185336.5B3A91E4028@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 20:53:36 2007 New Revision: 58410 Modified: python/branches/py3k/Python/errors.c Log: Minor cleanup of the comment for PyErr_ProgramText() and a tweak to the code to guarantee the claim that it doesn't set an exception. Modified: python/branches/py3k/Python/errors.c ============================================================================== --- python/branches/py3k/Python/errors.c (original) +++ python/branches/py3k/Python/errors.c Wed Oct 10 20:53:36 2007 @@ -800,13 +800,11 @@ PyErr_Restore(exc, v, tb); } -/* com_fetch_program_text will attempt to load the line of text that - the exception refers to. If it fails, it will return NULL but will - not set an exception. +/* Attempt to load the line of text that the exception refers to. If it + fails, it will return NULL but will not set an exception. XXX The functionality of this function is quite similar to the - functionality in tb_displayline() in traceback.c. -*/ + functionality in tb_displayline() in traceback.c. */ PyObject * PyErr_ProgramText(const char *filename, int lineno) @@ -824,7 +822,8 @@ char *pLastChar = &linebuf[sizeof(linebuf) - 2]; do { *pLastChar = '\0'; - if (Py_UniversalNewlineFgets(linebuf, sizeof linebuf, fp, NULL) == NULL) + if (Py_UniversalNewlineFgets(linebuf, sizeof linebuf, + fp, NULL) == NULL) break; /* fgets read *something*; if it didn't get as far as pLastChar, it must have found a newline @@ -836,9 +835,13 @@ fclose(fp); if (i == lineno) { char *p = linebuf; + PyObject *res; while (*p == ' ' || *p == '\t' || *p == '\014') p++; - return PyUnicode_FromString(p); + res = PyUnicode_FromString(p); + if (res == NULL) + PyErr_Clear(); + return res; } return NULL; } From python-3000-checkins at python.org Wed Oct 10 23:38:59 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 10 Oct 2007 23:38:59 +0200 (CEST) Subject: [Python-3000-checkins] r58411 - python/branches/py3k/Python/pythonrun.c Message-ID: <20071010213900.00D9F1E4002@bag.python.org> Author: guido.van.rossum Date: Wed Oct 10 23:38:59 2007 New Revision: 58411 Modified: python/branches/py3k/Python/pythonrun.c Log: get rid of some more PyString uses. Only the filename is still a PyString now. (We'll need to deal with the default filesystem encoding to do it right.) Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Wed Oct 10 23:38:59 2007 @@ -762,19 +762,19 @@ } v = PySys_GetObject("ps1"); if (v != NULL) { - v = PyObject_Str(v); + v = PyObject_Unicode(v); if (v == NULL) PyErr_Clear(); - else if (PyString_Check(v)) - ps1 = PyString_AsString(v); + else if (PyUnicode_Check(v)) + ps1 = PyUnicode_AsString(v); } w = PySys_GetObject("ps2"); if (w != NULL) { - w = PyObject_Str(w); + w = PyObject_Unicode(w); if (w == NULL) PyErr_Clear(); - else if (PyString_Check(w)) - ps2 = PyString_AsString(w); + else if (PyUnicode_Check(w)) + ps2 = PyUnicode_AsString(w); } arena = PyArena_New(); if (arena == NULL) { @@ -979,7 +979,8 @@ goto finally; if (v == Py_None) *text = NULL; - else if (! (*text = PyString_AsString(v))) + else if (!PyUnicode_Check(v) || + !(*text = PyUnicode_AsString(v))) goto finally; Py_DECREF(v); return 1; @@ -1093,7 +1094,7 @@ if (set_sys_last_vars) { PySys_SetObject("last_type", exception); PySys_SetObject("last_value", v); - PySys_SetObject("last_traceback", tb); + PySys_SetObject("last_traceback", tb ? tb : Py_None); } hook = PySys_GetObject("excepthook"); if (hook) { @@ -1195,10 +1196,13 @@ } moduleName = PyObject_GetAttrString(exception, "__module__"); - if (moduleName == NULL) + if (moduleName == NULL || !PyUnicode_Check(moduleName)) + { + Py_DECREF(moduleName); err = PyFile_WriteString("", f); + } else { - char* modstr = PyString_AsString(moduleName); + char* modstr = PyUnicode_AsString(moduleName); if (modstr && strcmp(modstr, "__builtin__")) { err = PyFile_WriteString(modstr, f); @@ -1216,14 +1220,14 @@ else err = PyFile_WriteObject(exception, f, Py_PRINT_RAW); if (err == 0 && (value != Py_None)) { - PyObject *s = PyObject_Str(value); + PyObject *s = PyObject_Unicode(value); /* only print colon if the str() of the object is not the empty string */ if (s == NULL) err = -1; - else if (!PyString_Check(s) || - PyString_GET_SIZE(s) != 0) + else if (!PyUnicode_Check(s) || + PyUnicode_GetSize(s) != 0) err = PyFile_WriteString(": ", f); if (err == 0) err = PyFile_WriteObject(s, f, Py_PRINT_RAW); @@ -1530,9 +1534,9 @@ PyObject *type, *value, *tb; PyErr_Fetch(&type, &value, &tb); if (value != NULL) { - u = PyObject_Str(value); + u = PyObject_Unicode(value); if (u != NULL) { - msg = PyString_AsString(u); + msg = PyUnicode_AsString(u); } } if (msg == NULL) From python-3000-checkins at python.org Thu Oct 11 00:48:24 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Thu, 11 Oct 2007 00:48:24 +0200 (CEST) Subject: [Python-3000-checkins] r58412 - python/branches/py3k/Lib/test/test_xml_etree.py python/branches/py3k/Lib/test/test_xml_etree_c.py Message-ID: <20071010224824.4CBE31E4002@bag.python.org> Author: guido.van.rossum Date: Thu Oct 11 00:48:24 2007 New Revision: 58412 Modified: python/branches/py3k/Lib/test/test_xml_etree.py python/branches/py3k/Lib/test/test_xml_etree_c.py Log: Two changes that are definitely problem-free and avoid calling print() with a bytes instance (as this will soon print something differently). Modified: python/branches/py3k/Lib/test/test_xml_etree.py ============================================================================== --- python/branches/py3k/Lib/test/test_xml_etree.py (original) +++ python/branches/py3k/Lib/test/test_xml_etree.py Thu Oct 11 00:48:24 2007 @@ -183,9 +183,8 @@ text >>> print(ET.tostring(element)) text - >>> print(ET.tostring(element, "ascii")) - - text + >>> print(repr(ET.tostring(element, "ascii"))) + b'\ntext' >>> _, ids = ET.XMLID("text") >>> len(ids) 0 Modified: python/branches/py3k/Lib/test/test_xml_etree_c.py ============================================================================== --- python/branches/py3k/Lib/test/test_xml_etree_c.py (original) +++ python/branches/py3k/Lib/test/test_xml_etree_c.py Thu Oct 11 00:48:24 2007 @@ -175,9 +175,8 @@ text >>> print(ET.tostring(element)) text - >>> print(ET.tostring(element, "ascii")) - - text + >>> print(repr(ET.tostring(element, "ascii"))) + b'\ntext' >>> _, ids = ET.XMLID("text") >>> len(ids) 0 From python-3000-checkins at python.org Thu Oct 11 20:01:44 2007 From: python-3000-checkins at python.org (fred.drake) Date: Thu, 11 Oct 2007 20:01:44 +0200 (CEST) Subject: [Python-3000-checkins] r58413 - in python/branches/py3k: Doc/library/debug.rst Doc/library/hotshot.rst Doc/library/profile.rst Lib/hotshot Lib/test/test_hotshot.py Makefile.pre.in Misc/NEWS Modules/_hotshot.c Tools/scripts/hotshotmain.py setup.py Message-ID: <20071011180144.29BE71E4027@bag.python.org> Author: fred.drake Date: Thu Oct 11 20:01:43 2007 New Revision: 58413 Removed: python/branches/py3k/Doc/library/hotshot.rst python/branches/py3k/Lib/hotshot/ python/branches/py3k/Lib/test/test_hotshot.py python/branches/py3k/Modules/_hotshot.c python/branches/py3k/Tools/scripts/hotshotmain.py Modified: python/branches/py3k/Doc/library/debug.rst python/branches/py3k/Doc/library/profile.rst python/branches/py3k/Makefile.pre.in python/branches/py3k/Misc/NEWS python/branches/py3k/setup.py Log: remove hotshot profiler from Py3k Modified: python/branches/py3k/Doc/library/debug.rst ============================================================================== --- python/branches/py3k/Doc/library/debug.rst (original) +++ python/branches/py3k/Doc/library/debug.rst Thu Oct 11 20:01:43 2007 @@ -12,6 +12,5 @@ bdb.rst pdb.rst profile.rst - hotshot.rst timeit.rst trace.rst \ No newline at end of file Deleted: /python/branches/py3k/Doc/library/hotshot.rst ============================================================================== --- /python/branches/py3k/Doc/library/hotshot.rst Thu Oct 11 20:01:43 2007 +++ (empty file) @@ -1,144 +0,0 @@ - -:mod:`hotshot` --- High performance logging profiler -==================================================== - -.. module:: hotshot - :synopsis: High performance logging profiler, mostly written in C. -.. moduleauthor:: Fred L. Drake, Jr. -.. sectionauthor:: Anthony Baxter - - -This module provides a nicer interface to the :mod:`_hotshot` C module. Hotshot -is a replacement for the existing :mod:`profile` module. As it's written mostly -in C, it should result in a much smaller performance impact than the existing -:mod:`profile` module. - -.. note:: - - The :mod:`hotshot` module focuses on minimizing the overhead while profiling, at - the expense of long data post-processing times. For common usages it is - recommended to use :mod:`cProfile` instead. :mod:`hotshot` is not maintained and - might be removed from the standard library in the future. - -.. warning:: - - The :mod:`hotshot` profiler does not yet work well with threads. It is useful to - use an unthreaded script to run the profiler over the code you're interested in - measuring if at all possible. - - -.. class:: Profile(logfile[, lineevents[, linetimings]]) - - The profiler object. The argument *logfile* is the name of a log file to use for - logged profile data. The argument *lineevents* specifies whether to generate - events for every source line, or just on function call/return. It defaults to - ``0`` (only log function call/return). The argument *linetimings* specifies - whether to record timing information. It defaults to ``1`` (store timing - information). - - -.. _hotshot-objects: - -Profile Objects ---------------- - -Profile objects have the following methods: - - -.. method:: Profile.addinfo(key, value) - - Add an arbitrary labelled value to the profile output. - - -.. method:: Profile.close() - - Close the logfile and terminate the profiler. - - -.. method:: Profile.fileno() - - Return the file descriptor of the profiler's log file. - - -.. method:: Profile.run(cmd) - - Profile an :func:`exec`\ -compatible string in the script environment. The - globals from the :mod:`__main__` module are used as both the globals and locals - for the script. - - -.. method:: Profile.runcall(func, *args, **keywords) - - Profile a single call of a callable. Additional positional and keyword arguments - may be passed along; the result of the call is returned, and exceptions are - allowed to propagate cleanly, while ensuring that profiling is disabled on the - way out. - - -.. method:: Profile.runctx(cmd, globals, locals) - - Profile an :func:`exec`\ -compatible string in a specific environment. The - string is compiled before profiling begins. - - -.. method:: Profile.start() - - Start the profiler. - - -.. method:: Profile.stop() - - Stop the profiler. - - -Using hotshot data ------------------- - -.. module:: hotshot.stats - :synopsis: Statistical analysis for Hotshot - - -This module loads hotshot profiling data into the standard :mod:`pstats` Stats -objects. - - -.. function:: load(filename) - - Load hotshot data from *filename*. Returns an instance of the - :class:`pstats.Stats` class. - - -.. seealso:: - - Module :mod:`profile` - The :mod:`profile` module's :class:`Stats` class - - -.. _hotshot-example: - -Example Usage -------------- - -Note that this example runs the python "benchmark" pystones. It can take some -time to run, and will produce large output files. :: - - >>> import hotshot, hotshot.stats, test.pystone - >>> prof = hotshot.Profile("stones.prof") - >>> benchtime, stones = prof.runcall(test.pystone.pystones) - >>> prof.close() - >>> stats = hotshot.stats.load("stones.prof") - >>> stats.strip_dirs() - >>> stats.sort_stats('time', 'calls') - >>> stats.print_stats(20) - 850004 function calls in 10.090 CPU seconds - - Ordered by: internal time, call count - - ncalls tottime percall cumtime percall filename:lineno(function) - 1 3.295 3.295 10.090 10.090 pystone.py:79(Proc0) - 150000 1.315 0.000 1.315 0.000 pystone.py:203(Proc7) - 50000 1.313 0.000 1.463 0.000 pystone.py:229(Func2) - . - . - . - Modified: python/branches/py3k/Doc/library/profile.rst ============================================================================== --- python/branches/py3k/Doc/library/profile.rst (original) +++ python/branches/py3k/Doc/library/profile.rst Thu Oct 11 20:01:43 2007 @@ -57,7 +57,7 @@ It also provides a series of report generation tools to allow users to rapidly examine the results of a profile operation. -The Python standard library provides three different profilers: +The Python standard library provides two different profilers: #. :mod:`profile`, a pure Python module, described in the sequel. Copyright ?? 1994, by InfoSeek Corporation. @@ -66,15 +66,11 @@ it suitable for profiling long-running programs. Based on :mod:`lsprof`, contributed by Brett Rosen and Ted Czotter. -#. :mod:`hotshot`, a C module focusing on minimizing the overhead while - profiling, at the expense of long data post-processing times. - The :mod:`profile` and :mod:`cProfile` modules export the same interface, so they are mostly interchangeables; :mod:`cProfile` has a much lower overhead but is not so far as well-tested and might not be available on all systems. :mod:`cProfile` is really a compatibility layer on top of the internal -:mod:`_lsprof` module. The :mod:`hotshot` module is reserved to specialized -usages. +:mod:`_lsprof` module. .. % \section{How Is This Profiler Different From The Old Profiler?} .. % \nodename{Profiler Changes} Deleted: /python/branches/py3k/Lib/test/test_hotshot.py ============================================================================== --- /python/branches/py3k/Lib/test/test_hotshot.py Thu Oct 11 20:01:43 2007 +++ (empty file) @@ -1,132 +0,0 @@ -import hotshot -import hotshot.log -import os -import pprint -import unittest - -from test import test_support - -from hotshot.log import ENTER, EXIT, LINE - - -def shortfilename(fn): - # We use a really shortened filename since an exact match is made, - # and the source may be either a Python source file or a - # pre-compiled bytecode file. - if fn: - return os.path.splitext(os.path.basename(fn))[0] - else: - return fn - - -class UnlinkingLogReader(hotshot.log.LogReader): - """Extend the LogReader so the log file is unlinked when we're - done with it.""" - - def __init__(self, logfn): - self.__logfn = logfn - hotshot.log.LogReader.__init__(self, logfn) - - def next(self, index=None): - try: - return hotshot.log.LogReader.next(self) - except StopIteration: - self.close() - os.unlink(self.__logfn) - raise - - -class HotShotTestCase(unittest.TestCase): - def new_profiler(self, lineevents=0, linetimings=1): - self.logfn = test_support.TESTFN - return hotshot.Profile(self.logfn, lineevents, linetimings) - - def get_logreader(self): - return UnlinkingLogReader(self.logfn) - - def get_events_wotime(self): - L = [] - for event in self.get_logreader(): - what, (filename, lineno, funcname), tdelta = event - L.append((what, (shortfilename(filename), lineno, funcname))) - return L - - def check_events(self, expected): - events = self.get_events_wotime() - if events != expected: - self.fail( - "events did not match expectation; got:\n%s\nexpected:\n%s" - % (pprint.pformat(events), pprint.pformat(expected))) - - def run_test(self, callable, events, profiler=None): - if profiler is None: - profiler = self.new_profiler() - self.failUnless(not profiler._prof.closed) - profiler.runcall(callable) - self.failUnless(not profiler._prof.closed) - profiler.close() - self.failUnless(profiler._prof.closed) - self.check_events(events) - - def test_addinfo(self): - def f(p): - p.addinfo("test-key", "test-value") - profiler = self.new_profiler() - profiler.runcall(f, profiler) - profiler.close() - log = self.get_logreader() - info = log._info - list(log) - self.assertEqual(info["test-key"], ["test-value"]) - - def test_line_numbers(self): - def f(): - y = 2 - x = 1 - def g(): - f() - f_lineno = f.__code__.co_firstlineno - g_lineno = g.__code__.co_firstlineno - events = [(ENTER, ("test_hotshot", g_lineno, "g")), - (LINE, ("test_hotshot", g_lineno+1, "g")), - (ENTER, ("test_hotshot", f_lineno, "f")), - (LINE, ("test_hotshot", f_lineno+1, "f")), - (LINE, ("test_hotshot", f_lineno+2, "f")), - (EXIT, ("test_hotshot", f_lineno, "f")), - (EXIT, ("test_hotshot", g_lineno, "g")), - ] - self.run_test(g, events, self.new_profiler(lineevents=1)) - - def test_start_stop(self): - # Make sure we don't return NULL in the start() and stop() - # methods when there isn't an error. Bug in 2.2 noted by - # Anthony Baxter. - profiler = self.new_profiler() - profiler.start() - profiler.stop() - profiler.close() - os.unlink(self.logfn) - - def test_bad_sys_path(self): - import sys - import os - orig_path = sys.path - coverage = hotshot._hotshot.coverage - try: - # verify we require a list for sys.path - sys.path = 'abc' - self.assertRaises(RuntimeError, coverage, test_support.TESTFN) - # verify that we require sys.path exists - del sys.path - self.assertRaises(RuntimeError, coverage, test_support.TESTFN) - finally: - sys.path = orig_path - if os.path.exists(test_support.TESTFN): - os.remove(test_support.TESTFN) - -def test_main(): - test_support.run_unittest(HotShotTestCase) - - -if __name__ == "__main__": - test_main() Modified: python/branches/py3k/Makefile.pre.in ============================================================================== --- python/branches/py3k/Makefile.pre.in (original) +++ python/branches/py3k/Makefile.pre.in Thu Oct 11 20:01:43 2007 @@ -737,7 +737,7 @@ PLATMACPATH=:plat-mac:plat-mac/lib-scriptpackages LIBSUBDIRS= lib-tk site-packages test test/output test/data \ test/decimaltestdata \ - encodings hotshot \ + encodings \ email email/mime email/test email/test/data \ sqlite3 sqlite3/test \ logging bsddb bsddb/test csv wsgiref \ Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Thu Oct 11 20:01:43 2007 @@ -4,6 +4,16 @@ (editors: check NEWS.help for information about editing NEWS using ReST.) +What's New in Python 3.0a2? + +*Unreleased* + +Extension Modules +----------------- + +- The `hotshot` profiler has been removed; use `cProfile` instead. + + What's New in Python 3.0a1? ========================== Deleted: /python/branches/py3k/Modules/_hotshot.c ============================================================================== --- /python/branches/py3k/Modules/_hotshot.c Thu Oct 11 20:01:43 2007 +++ (empty file) @@ -1,1645 +0,0 @@ -/* - * This is the High Performance Python Profiler portion of HotShot. - */ - -#include "Python.h" -#include "code.h" -#include "eval.h" -#include "frameobject.h" -#include "structmember.h" - -/* - * Which timer to use should be made more configurable, but that should not - * be difficult. This will do for now. - */ -#ifdef MS_WINDOWS -#include - -#ifdef HAVE_DIRECT_H -#include /* for getcwd() */ -#endif - -typedef __int64 hs_time; -#define GETTIMEOFDAY(P_HS_TIME) \ - { LARGE_INTEGER _temp; \ - QueryPerformanceCounter(&_temp); \ - *(P_HS_TIME) = _temp.QuadPart; } - - -#else -#ifndef HAVE_GETTIMEOFDAY -#error "This module requires gettimeofday() on non-Windows platforms!" -#endif -#if (defined(PYOS_OS2) && defined(PYCC_GCC)) || defined(__QNX__) -#include -#else -#include -#include -#endif -typedef struct timeval hs_time; -#endif - -#if !defined(__cplusplus) && !defined(inline) -#ifdef __GNUC__ -#define inline __inline -#endif -#endif - -#ifndef inline -#define inline -#endif - -#define BUFFERSIZE 10240 - -#if defined(PYOS_OS2) && defined(PYCC_GCC) -#define PATH_MAX 260 -#endif - -#if defined(__sgi) && _COMPILER_VERSION>700 && !defined(PATH_MAX) -/* fix PATH_MAX not being defined with MIPSPro 7.x - if mode is ANSI C (default) */ -#define PATH_MAX 1024 -#endif - -#ifndef PATH_MAX -# ifdef MAX_PATH -# define PATH_MAX MAX_PATH -# elif defined (_POSIX_PATH_MAX) -# define PATH_MAX _POSIX_PATH_MAX -# else -# error "Need a defn. for PATH_MAX in _hotshot.c" -# endif -#endif - -typedef struct { - PyObject_HEAD - PyObject *filemap; - PyObject *logfilename; - Py_ssize_t index; - unsigned char buffer[BUFFERSIZE]; - FILE *logfp; - int lineevents; - int linetimings; - int frametimings; - /* size_t filled; */ - int active; - int next_fileno; - hs_time prev_timeofday; -} ProfilerObject; - -typedef struct { - PyObject_HEAD - PyObject *info; - FILE *logfp; - int linetimings; - int frametimings; -} LogReaderObject; - -static PyObject * ProfilerError = NULL; - - -#ifndef MS_WINDOWS -#ifdef GETTIMEOFDAY_NO_TZ -#define GETTIMEOFDAY(ptv) gettimeofday((ptv)) -#else -#define GETTIMEOFDAY(ptv) gettimeofday((ptv), (struct timezone *)NULL) -#endif -#endif - - -/* The log reader... */ - -PyDoc_STRVAR(logreader_close__doc__, -"close()\n" -"Close the log file, preventing additional records from being read."); - -static PyObject * -logreader_close(LogReaderObject *self, PyObject *args) -{ - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_INCREF(Py_None); - - return Py_None; -} - -PyDoc_STRVAR(logreader_fileno__doc__, -"fileno() -> file descriptor\n" -"Returns the file descriptor for the log file, if open.\n" -"Raises ValueError if the log file is closed."); - -static PyObject * -logreader_fileno(LogReaderObject *self) -{ - if (self->logfp == NULL) { - PyErr_SetString(PyExc_ValueError, - "logreader's file object already closed"); - return NULL; - } - return PyInt_FromLong(fileno(self->logfp)); -} - - -/* Log File Format - * --------------- - * - * The log file consists of a sequence of variable-length records. - * Each record is identified with a record type identifier in two - * bits of the first byte. The two bits are the "least significant" - * bits of the byte. - * - * Low bits: Opcode: Meaning: - * 0x00 ENTER enter a frame - * 0x01 EXIT exit a frame - * 0x02 LINENO execution moved onto a different line - * 0x03 OTHER more bits are needed to deecode - * - * If the type is OTHER, the record is not packed so tightly, and the - * remaining bits are used to disambiguate the record type. These - * records are not used as frequently so compaction is not an issue. - * Each of the first three record types has a highly tailored - * structure that allows it to be packed tightly. - * - * The OTHER records have the following identifiers: - * - * First byte: Opcode: Meaning: - * 0x13 ADD_INFO define a key/value pair - * 0x23 DEFINE_FILE define an int->filename mapping - * 0x33 LINE_TIMES indicates if LINENO events have tdeltas - * 0x43 DEFINE_FUNC define a (fileno,lineno)->funcname mapping - * 0x53 FRAME_TIMES indicates if ENTER/EXIT events have tdeltas - * - * Packed Integers - * - * "Packed integers" are non-negative integer values encoded as a - * sequence of bytes. Each byte is encoded such that the most - * significant bit is set if the next byte is also part of the - * integer. Each byte provides bits to the least-significant end of - * the result; the accumulated value must be shifted up to place the - * new bits into the result. - * - * "Modified packed integers" are packed integers where only a portion - * of the first byte is used. In the rest of the specification, these - * are referred to as "MPI(n,name)", where "n" is the number of bits - * discarded from the least-signicant positions of the byte, and - * "name" is a name being given to those "discarded" bits, since they - * are a field themselves. - * - * ENTER records: - * - * MPI(2,type) fileno -- type is 0x00 - * PI lineno - * PI tdelta -- iff frame times are enabled - * - * EXIT records - * - * MPI(2,type) tdelta -- type is 0x01; tdelta will be 0 - * if frame times are disabled - * - * LINENO records - * - * MPI(2,type) lineno -- type is 0x02 - * PI tdelta -- iff LINENO includes it - * - * ADD_INFO records - * - * BYTE type -- always 0x13 - * PI len1 -- length of first string - * BYTE string1[len1] -- len1 bytes of string data - * PI len2 -- length of second string - * BYTE string2[len2] -- len2 bytes of string data - * - * DEFINE_FILE records - * - * BYTE type -- always 0x23 - * PI fileno - * PI len -- length of filename - * BYTE filename[len] -- len bytes of string data - * - * DEFINE_FUNC records - * - * BYTE type -- always 0x43 - * PI fileno - * PI lineno - * PI len -- length of funcname - * BYTE funcname[len] -- len bytes of string data - * - * LINE_TIMES records - * - * This record can be used only before the start of ENTER/EXIT/LINENO - * records. If have_tdelta is true, LINENO records will include the - * tdelta field, otherwise it will be omitted. If this record is not - * given, LINENO records will not contain the tdelta field. - * - * BYTE type -- always 0x33 - * BYTE have_tdelta -- 0 if LINENO does *not* have - * timing information - * FRAME_TIMES records - * - * This record can be used only before the start of ENTER/EXIT/LINENO - * records. If have_tdelta is true, ENTER and EXIT records will - * include the tdelta field, otherwise it will be omitted. If this - * record is not given, ENTER and EXIT records will contain the tdelta - * field. - * - * BYTE type -- always 0x53 - * BYTE have_tdelta -- 0 if ENTER/EXIT do *not* have - * timing information - */ - -#define WHAT_ENTER 0x00 -#define WHAT_EXIT 0x01 -#define WHAT_LINENO 0x02 -#define WHAT_OTHER 0x03 /* only used in decoding */ -#define WHAT_ADD_INFO 0x13 -#define WHAT_DEFINE_FILE 0x23 -#define WHAT_LINE_TIMES 0x33 -#define WHAT_DEFINE_FUNC 0x43 -#define WHAT_FRAME_TIMES 0x53 - -#define ERR_NONE 0 -#define ERR_EOF -1 -#define ERR_EXCEPTION -2 -#define ERR_BAD_RECTYPE -3 - -#define PISIZE (sizeof(int) + 1) -#define MPISIZE (PISIZE + 1) - -/* Maximum size of "normal" events -- nothing that contains string data */ -#define MAXEVENTSIZE (MPISIZE + PISIZE*2) - - -/* Unpack a packed integer; if "discard" is non-zero, unpack a modified - * packed integer with "discard" discarded bits. - */ -static int -unpack_packed_int(LogReaderObject *self, int *pvalue, int discard) -{ - int c; - int accum = 0; - int bits = 0; - int cont; - - do { - /* read byte */ - if ((c = fgetc(self->logfp)) == EOF) - return ERR_EOF; - accum |= ((c & 0x7F) >> discard) << bits; - bits += (7 - discard); - cont = c & 0x80; - discard = 0; - } while (cont); - - *pvalue = accum; - - return 0; -} - -/* Unpack a string, which is encoded as a packed integer giving the - * length of the string, followed by the string data. - */ -static int -unpack_string(LogReaderObject *self, PyObject **pvalue) -{ - int i; - int len; - int err; - int ch; - char *buf; - - if ((err = unpack_packed_int(self, &len, 0))) - return err; - - buf = (char *)malloc(len); - if (!buf) { - PyErr_NoMemory(); - return ERR_EXCEPTION; - } - - for (i=0; i < len; i++) { - ch = fgetc(self->logfp); - buf[i] = ch; - if (ch == EOF) { - free(buf); - return ERR_EOF; - } - } - *pvalue = PyString_FromStringAndSize(buf, len); - free(buf); - if (*pvalue == NULL) { - return ERR_EXCEPTION; - } - return 0; -} - - -static int -unpack_add_info(LogReaderObject *self) -{ - PyObject *key; - PyObject *value = NULL; - int err; - - err = unpack_string(self, &key); - if (!err) { - err = unpack_string(self, &value); - if (err) - Py_DECREF(key); - else { - PyObject *list = PyDict_GetItem(self->info, key); - if (list == NULL) { - list = PyList_New(0); - if (list == NULL) { - err = ERR_EXCEPTION; - goto finally; - } - if (PyDict_SetItem(self->info, key, list)) { - Py_DECREF(list); - err = ERR_EXCEPTION; - goto finally; - } - Py_DECREF(list); - } - if (PyList_Append(list, value)) - err = ERR_EXCEPTION; - } - } - finally: - Py_XDECREF(key); - Py_XDECREF(value); - return err; -} - - -static void -eof_error(LogReaderObject *self) -{ - fclose(self->logfp); - self->logfp = NULL; - PyErr_SetString(PyExc_EOFError, - "end of file with incomplete profile record"); -} - -static PyObject * -logreader_tp_iternext(LogReaderObject *self) -{ - int c; - int what; - int err = ERR_NONE; - int lineno = -1; - int fileno = -1; - int tdelta = -1; - PyObject *s1 = NULL, *s2 = NULL; - PyObject *result = NULL; -#if 0 - unsigned char b0, b1; -#endif - - if (self->logfp == NULL) { - PyErr_SetString(ProfilerError, - "cannot iterate over closed LogReader object"); - return NULL; - } - -restart: - /* decode the record type */ - if ((c = fgetc(self->logfp)) == EOF) { - fclose(self->logfp); - self->logfp = NULL; - return NULL; - } - what = c & WHAT_OTHER; - if (what == WHAT_OTHER) - what = c; /* need all the bits for type */ - else - ungetc(c, self->logfp); /* type byte includes packed int */ - - switch (what) { - case WHAT_ENTER: - err = unpack_packed_int(self, &fileno, 2); - if (!err) { - err = unpack_packed_int(self, &lineno, 0); - if (self->frametimings && !err) - err = unpack_packed_int(self, &tdelta, 0); - } - break; - case WHAT_EXIT: - err = unpack_packed_int(self, &tdelta, 2); - break; - case WHAT_LINENO: - err = unpack_packed_int(self, &lineno, 2); - if (self->linetimings && !err) - err = unpack_packed_int(self, &tdelta, 0); - break; - case WHAT_ADD_INFO: - err = unpack_add_info(self); - break; - case WHAT_DEFINE_FILE: - err = unpack_packed_int(self, &fileno, 0); - if (!err) { - err = unpack_string(self, &s1); - if (!err) { - Py_INCREF(Py_None); - s2 = Py_None; - } - } - break; - case WHAT_DEFINE_FUNC: - err = unpack_packed_int(self, &fileno, 0); - if (!err) { - err = unpack_packed_int(self, &lineno, 0); - if (!err) - err = unpack_string(self, &s1); - } - break; - case WHAT_LINE_TIMES: - if ((c = fgetc(self->logfp)) == EOF) - err = ERR_EOF; - else { - self->linetimings = c ? 1 : 0; - goto restart; - } - break; - case WHAT_FRAME_TIMES: - if ((c = fgetc(self->logfp)) == EOF) - err = ERR_EOF; - else { - self->frametimings = c ? 1 : 0; - goto restart; - } - break; - default: - err = ERR_BAD_RECTYPE; - } - if (err == ERR_BAD_RECTYPE) { - PyErr_SetString(PyExc_ValueError, - "unknown record type in log file"); - } - else if (err == ERR_EOF) { - eof_error(self); - } - else if (!err) { - result = PyTuple_New(4); - if (result == NULL) - return NULL; - PyTuple_SET_ITEM(result, 0, PyInt_FromLong(what)); - PyTuple_SET_ITEM(result, 2, PyInt_FromLong(fileno)); - if (s1 == NULL) - PyTuple_SET_ITEM(result, 1, PyInt_FromLong(tdelta)); - else - PyTuple_SET_ITEM(result, 1, s1); - if (s2 == NULL) - PyTuple_SET_ITEM(result, 3, PyInt_FromLong(lineno)); - else - PyTuple_SET_ITEM(result, 3, s2); - } - /* The only other case is err == ERR_EXCEPTION, in which case the - * exception is already set. - */ -#if 0 - b0 = self->buffer[self->index]; - b1 = self->buffer[self->index + 1]; - if (b0 & 1) { - /* This is a line-number event. */ - what = PyTrace_LINE; - lineno = ((b0 & ~1) << 7) + b1; - self->index += 2; - } - else { - what = (b0 & 0x0E) >> 1; - tdelta = ((b0 & 0xF0) << 4) + b1; - if (what == PyTrace_CALL) { - /* we know there's a 2-byte file ID & 2-byte line number */ - fileno = ((self->buffer[self->index + 2] << 8) - + self->buffer[self->index + 3]); - lineno = ((self->buffer[self->index + 4] << 8) - + self->buffer[self->index + 5]); - self->index += 6; - } - else - self->index += 2; - } -#endif - return result; -} - -static void -logreader_dealloc(LogReaderObject *self) -{ - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_XDECREF(self->info); - PyObject_Del(self); -} - -static PyObject * -logreader_sq_item(LogReaderObject *self, Py_ssize_t index) -{ - PyObject *result = logreader_tp_iternext(self); - if (result == NULL && !PyErr_Occurred()) { - PyErr_SetString(PyExc_IndexError, "no more events in log"); - return NULL; - } - return result; -} - -static void -do_stop(ProfilerObject *self); - -static int -flush_data(ProfilerObject *self) -{ - /* Need to dump data to the log file... */ - size_t written = fwrite(self->buffer, 1, self->index, self->logfp); - if (written == (size_t)self->index) - self->index = 0; - else { - memmove(self->buffer, &self->buffer[written], - self->index - written); - self->index -= written; - if (written == 0) { - char *s = PyString_AsString(self->logfilename); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, s); - do_stop(self); - return -1; - } - } - if (written > 0) { - if (fflush(self->logfp)) { - char *s = PyString_AsString(self->logfilename); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, s); - do_stop(self); - return -1; - } - } - return 0; -} - -static inline int -pack_packed_int(ProfilerObject *self, int value) -{ - unsigned char partial; - - do { - partial = value & 0x7F; - value >>= 7; - if (value) - partial |= 0x80; - self->buffer[self->index] = partial; - self->index++; - } while (value); - return 0; -} - -/* Encode a modified packed integer, with a subfield of modsize bits - * containing the value "subfield". The value of subfield is not - * checked to ensure it actually fits in modsize bits. - */ -static inline int -pack_modified_packed_int(ProfilerObject *self, int value, - int modsize, int subfield) -{ - const int maxvalues[] = {-1, 1, 3, 7, 15, 31, 63, 127}; - - int bits = 7 - modsize; - int partial = value & maxvalues[bits]; - unsigned char b = subfield | (partial << modsize); - - if (partial != value) { - b |= 0x80; - self->buffer[self->index] = b; - self->index++; - return pack_packed_int(self, value >> bits); - } - self->buffer[self->index] = b; - self->index++; - return 0; -} - -static int -pack_string(ProfilerObject *self, const char *s, Py_ssize_t len) -{ - if (len + PISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - assert(len < INT_MAX); - if (pack_packed_int(self, (int)len) < 0) - return -1; - memcpy(self->buffer + self->index, s, len); - self->index += len; - return 0; -} - -static int -pack_add_info(ProfilerObject *self, const char *s1, const char *s2) -{ - Py_ssize_t len1 = strlen(s1); - Py_ssize_t len2 = strlen(s2); - - if (len1 + len2 + PISIZE*2 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_ADD_INFO; - self->index++; - if (pack_string(self, s1, len1) < 0) - return -1; - return pack_string(self, s2, len2); -} - -static int -pack_define_file(ProfilerObject *self, int fileno, const char *filename) -{ - Py_ssize_t len = strlen(filename); - - if (len + PISIZE*2 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_DEFINE_FILE; - self->index++; - if (pack_packed_int(self, fileno) < 0) - return -1; - return pack_string(self, filename, len); -} - -static int -pack_define_func(ProfilerObject *self, int fileno, int lineno, - const char *funcname) -{ - Py_ssize_t len = strlen(funcname); - - if (len + PISIZE*3 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_DEFINE_FUNC; - self->index++; - if (pack_packed_int(self, fileno) < 0) - return -1; - if (pack_packed_int(self, lineno) < 0) - return -1; - return pack_string(self, funcname, len); -} - -static int -pack_line_times(ProfilerObject *self) -{ - if (2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_LINE_TIMES; - self->buffer[self->index + 1] = self->linetimings ? 1 : 0; - self->index += 2; - return 0; -} - -static int -pack_frame_times(ProfilerObject *self) -{ - if (2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_FRAME_TIMES; - self->buffer[self->index + 1] = self->frametimings ? 1 : 0; - self->index += 2; - return 0; -} - -static inline int -pack_enter(ProfilerObject *self, int fileno, int tdelta, int lineno) -{ - if (MPISIZE + PISIZE*2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - pack_modified_packed_int(self, fileno, 2, WHAT_ENTER); - pack_packed_int(self, lineno); - if (self->frametimings) - return pack_packed_int(self, tdelta); - else - return 0; -} - -static inline int -pack_exit(ProfilerObject *self, int tdelta) -{ - if (MPISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - if (self->frametimings) - return pack_modified_packed_int(self, tdelta, 2, WHAT_EXIT); - self->buffer[self->index] = WHAT_EXIT; - self->index++; - return 0; -} - -static inline int -pack_lineno(ProfilerObject *self, int lineno) -{ - if (MPISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - return pack_modified_packed_int(self, lineno, 2, WHAT_LINENO); -} - -static inline int -pack_lineno_tdelta(ProfilerObject *self, int lineno, int tdelta) -{ - if (MPISIZE + PISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return 0; - } - if (pack_modified_packed_int(self, lineno, 2, WHAT_LINENO) < 0) - return -1; - return pack_packed_int(self, tdelta); -} - -static inline int -get_fileno(ProfilerObject *self, PyCodeObject *fcode) -{ - /* This is only used for ENTER events. */ - - PyObject *obj; - PyObject *dict; - int fileno; - - obj = PyDict_GetItem(self->filemap, fcode->co_filename); - if (obj == NULL) { - /* first sighting of this file */ - dict = PyDict_New(); - if (dict == NULL) { - return -1; - } - fileno = self->next_fileno; - obj = Py_BuildValue("iN", fileno, dict); - if (obj == NULL) { - return -1; - } - if (PyDict_SetItem(self->filemap, fcode->co_filename, obj)) { - Py_DECREF(obj); - return -1; - } - self->next_fileno++; - Py_DECREF(obj); - if (pack_define_file(self, fileno, - PyString_AS_STRING(fcode->co_filename)) < 0) - return -1; - } - else { - /* already know this ID */ - fileno = PyInt_AS_LONG(PyTuple_GET_ITEM(obj, 0)); - dict = PyTuple_GET_ITEM(obj, 1); - } - /* make sure we save a function name for this (fileno, lineno) */ - obj = PyInt_FromLong(fcode->co_firstlineno); - if (obj == NULL) { - /* We just won't have it saved; too bad. */ - PyErr_Clear(); - } - else { - PyObject *name = PyDict_GetItem(dict, obj); - if (name == NULL) { - if (pack_define_func(self, fileno, fcode->co_firstlineno, - PyUnicode_AsString(fcode->co_name)) < 0) { - Py_DECREF(obj); - return -1; - } - if (PyDict_SetItem(dict, obj, fcode->co_name)) { - Py_DECREF(obj); - return -1; - } - } - Py_DECREF(obj); - } - return fileno; -} - -static inline int -get_tdelta(ProfilerObject *self) -{ - int tdelta; -#ifdef MS_WINDOWS - hs_time tv; - hs_time diff; - - GETTIMEOFDAY(&tv); - diff = tv - self->prev_timeofday; - tdelta = (int)diff; -#else - struct timeval tv; - - GETTIMEOFDAY(&tv); - - tdelta = tv.tv_usec - self->prev_timeofday.tv_usec; - if (tv.tv_sec != self->prev_timeofday.tv_sec) - tdelta += (tv.tv_sec - self->prev_timeofday.tv_sec) * 1000000; -#endif - /* time can go backwards on some multiprocessor systems or by NTP */ - if (tdelta < 0) - return 0; - - self->prev_timeofday = tv; - return tdelta; -} - - -/* The workhorse: the profiler callback function. */ - -static int -tracer_callback(ProfilerObject *self, PyFrameObject *frame, int what, - PyObject *arg) -{ - int fileno; - - switch (what) { - case PyTrace_CALL: - fileno = get_fileno(self, frame->f_code); - if (fileno < 0) - return -1; - return pack_enter(self, fileno, - self->frametimings ? get_tdelta(self) : -1, - frame->f_code->co_firstlineno); - - case PyTrace_RETURN: - return pack_exit(self, get_tdelta(self)); - - case PyTrace_LINE: /* we only get these events if we asked for them */ - if (self->linetimings) - return pack_lineno_tdelta(self, frame->f_lineno, - get_tdelta(self)); - else - return pack_lineno(self, frame->f_lineno); - - default: - /* ignore PyTrace_EXCEPTION */ - break; - } - return 0; -} - - -/* A couple of useful helper functions. */ - -#ifdef MS_WINDOWS -static LARGE_INTEGER frequency = {0, 0}; -#endif - -static unsigned long timeofday_diff = 0; -static unsigned long rusage_diff = 0; - -static void -calibrate(void) -{ - hs_time tv1, tv2; - -#ifdef MS_WINDOWS - hs_time diff; - QueryPerformanceFrequency(&frequency); -#endif - - GETTIMEOFDAY(&tv1); - while (1) { - GETTIMEOFDAY(&tv2); -#ifdef MS_WINDOWS - diff = tv2 - tv1; - if (diff != 0) { - timeofday_diff = (unsigned long)diff; - break; - } -#else - if (tv1.tv_sec != tv2.tv_sec || tv1.tv_usec != tv2.tv_usec) { - if (tv1.tv_sec == tv2.tv_sec) - timeofday_diff = tv2.tv_usec - tv1.tv_usec; - else - timeofday_diff = (1000000 - tv1.tv_usec) + tv2.tv_usec; - break; - } -#endif - } -#if defined(MS_WINDOWS) || defined(PYOS_OS2) || \ - defined(__VMS) || defined (__QNX__) - rusage_diff = -1; -#else - { - struct rusage ru1, ru2; - - getrusage(RUSAGE_SELF, &ru1); - while (1) { - getrusage(RUSAGE_SELF, &ru2); - if (ru1.ru_utime.tv_sec != ru2.ru_utime.tv_sec) { - rusage_diff = ((1000000 - ru1.ru_utime.tv_usec) - + ru2.ru_utime.tv_usec); - break; - } - else if (ru1.ru_utime.tv_usec != ru2.ru_utime.tv_usec) { - rusage_diff = ru2.ru_utime.tv_usec - ru1.ru_utime.tv_usec; - break; - } - else if (ru1.ru_stime.tv_sec != ru2.ru_stime.tv_sec) { - rusage_diff = ((1000000 - ru1.ru_stime.tv_usec) - + ru2.ru_stime.tv_usec); - break; - } - else if (ru1.ru_stime.tv_usec != ru2.ru_stime.tv_usec) { - rusage_diff = ru2.ru_stime.tv_usec - ru1.ru_stime.tv_usec; - break; - } - } - } -#endif -} - -static void -do_start(ProfilerObject *self) -{ - self->active = 1; - GETTIMEOFDAY(&self->prev_timeofday); - if (self->lineevents) - PyEval_SetTrace((Py_tracefunc) tracer_callback, (PyObject *)self); - else - PyEval_SetProfile((Py_tracefunc) tracer_callback, (PyObject *)self); -} - -static void -do_stop(ProfilerObject *self) -{ - if (self->active) { - self->active = 0; - if (self->lineevents) - PyEval_SetTrace(NULL, NULL); - else - PyEval_SetProfile(NULL, NULL); - } - if (self->index > 0) { - /* Best effort to dump out any remaining data. */ - flush_data(self); - } -} - -static int -is_available(ProfilerObject *self) -{ - if (self->active) { - PyErr_SetString(ProfilerError, "profiler already active"); - return 0; - } - if (self->logfp == NULL) { - PyErr_SetString(ProfilerError, "profiler already closed"); - return 0; - } - return 1; -} - - -/* Profiler object interface methods. */ - -PyDoc_STRVAR(addinfo__doc__, -"addinfo(key, value)\n" -"Insert an ADD_INFO record into the log."); - -static PyObject * -profiler_addinfo(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - char *key, *value; - - if (PyArg_ParseTuple(args, "ss:addinfo", &key, &value)) { - if (self->logfp == NULL) - PyErr_SetString(ProfilerError, "profiler already closed"); - else { - if (pack_add_info(self, key, value) == 0) { - result = Py_None; - Py_INCREF(result); - } - } - } - return result; -} - -PyDoc_STRVAR(close__doc__, -"close()\n" -"Shut down this profiler and close the log files, even if its active."); - -static PyObject * -profiler_close(ProfilerObject *self) -{ - do_stop(self); - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_INCREF(Py_None); - return Py_None; -} - -#define fileno__doc__ logreader_fileno__doc__ - -static PyObject * -profiler_fileno(ProfilerObject *self) -{ - if (self->logfp == NULL) { - PyErr_SetString(PyExc_ValueError, - "profiler's file object already closed"); - return NULL; - } - return PyInt_FromLong(fileno(self->logfp)); -} - -PyDoc_STRVAR(runcall__doc__, -"runcall(callable[, args[, kw]]) -> callable()\n" -"Profile a specific function call, returning the result of that call."); - -static PyObject * -profiler_runcall(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - PyObject *callargs = NULL; - PyObject *callkw = NULL; - PyObject *callable; - - if (PyArg_UnpackTuple(args, "runcall", 1, 3, - &callable, &callargs, &callkw)) { - if (is_available(self)) { - do_start(self); - result = PyEval_CallObjectWithKeywords(callable, callargs, callkw); - do_stop(self); - } - } - return result; -} - -PyDoc_STRVAR(runcode__doc__, -"runcode(code, globals[, locals])\n" -"Execute a code object while collecting profile data. If locals is\n" -"omitted, globals is used for the locals as well."); - -static PyObject * -profiler_runcode(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - PyCodeObject *code; - PyObject *globals; - PyObject *locals = NULL; - - if (PyArg_ParseTuple(args, "O!O!|O:runcode", - &PyCode_Type, &code, - &PyDict_Type, &globals, - &locals)) { - if (is_available(self)) { - if (locals == NULL || locals == Py_None) - locals = globals; - else if (!PyDict_Check(locals)) { - PyErr_SetString(PyExc_TypeError, - "locals must be a dictionary or None"); - return NULL; - } - do_start(self); - result = PyEval_EvalCode(code, globals, locals); - do_stop(self); -#if 0 - if (!PyErr_Occurred()) { - result = Py_None; - Py_INCREF(result); - } -#endif - } - } - return result; -} - -PyDoc_STRVAR(start__doc__, -"start()\n" -"Install this profiler for the current thread."); - -static PyObject * -profiler_start(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - - if (is_available(self)) { - do_start(self); - result = Py_None; - Py_INCREF(result); - } - return result; -} - -PyDoc_STRVAR(stop__doc__, -"stop()\n" -"Remove this profiler from the current thread."); - -static PyObject * -profiler_stop(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - - if (!self->active) - PyErr_SetString(ProfilerError, "profiler not active"); - else { - do_stop(self); - result = Py_None; - Py_INCREF(result); - } - return result; -} - - -/* Python API support. */ - -static void -profiler_dealloc(ProfilerObject *self) -{ - do_stop(self); - if (self->logfp != NULL) - fclose(self->logfp); - Py_XDECREF(self->filemap); - Py_XDECREF(self->logfilename); - PyObject_Del((PyObject *)self); -} - -static PyMethodDef profiler_methods[] = { - {"addinfo", (PyCFunction)profiler_addinfo, METH_VARARGS, addinfo__doc__}, - {"close", (PyCFunction)profiler_close, METH_NOARGS, close__doc__}, - {"fileno", (PyCFunction)profiler_fileno, METH_NOARGS, fileno__doc__}, - {"runcall", (PyCFunction)profiler_runcall, METH_VARARGS, runcall__doc__}, - {"runcode", (PyCFunction)profiler_runcode, METH_VARARGS, runcode__doc__}, - {"start", (PyCFunction)profiler_start, METH_NOARGS, start__doc__}, - {"stop", (PyCFunction)profiler_stop, METH_NOARGS, stop__doc__}, - {NULL, NULL} -}; - -static PyMemberDef profiler_members[] = { - {"frametimings", T_LONG, offsetof(ProfilerObject, linetimings), READONLY}, - {"lineevents", T_LONG, offsetof(ProfilerObject, lineevents), READONLY}, - {"linetimings", T_LONG, offsetof(ProfilerObject, linetimings), READONLY}, - {NULL} -}; - -static PyObject * -profiler_get_closed(ProfilerObject *self, void *closure) -{ - PyObject *result = (self->logfp == NULL) ? Py_True : Py_False; - Py_INCREF(result); - return result; -} - -static PyGetSetDef profiler_getsets[] = { - {"closed", (getter)profiler_get_closed, NULL, - PyDoc_STR("True if the profiler's output file has already been closed.")}, - {NULL} -}; - - -PyDoc_STRVAR(profiler_object__doc__, -"High-performance profiler object.\n" -"\n" -"Methods:\n" -"\n" -"close(): Stop the profiler and close the log files.\n" -"fileno(): Returns the file descriptor of the log file.\n" -"runcall(): Run a single function call with profiling enabled.\n" -"runcode(): Execute a code object with profiling enabled.\n" -"start(): Install the profiler and return.\n" -"stop(): Remove the profiler.\n" -"\n" -"Attributes (read-only):\n" -"\n" -"closed: True if the profiler has already been closed.\n" -"frametimings: True if ENTER/EXIT events collect timing information.\n" -"lineevents: True if line events are reported to the profiler.\n" -"linetimings: True if line events collect timing information."); - -static PyTypeObject ProfilerType = { - PyVarObject_HEAD_INIT(NULL, 0) - "_hotshot.ProfilerType", /* tp_name */ - (int) sizeof(ProfilerObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - (destructor)profiler_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - profiler_object__doc__, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - profiler_methods, /* tp_methods */ - profiler_members, /* tp_members */ - profiler_getsets, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ -}; - - -static PyMethodDef logreader_methods[] = { - {"close", (PyCFunction)logreader_close, METH_NOARGS, - logreader_close__doc__}, - {"fileno", (PyCFunction)logreader_fileno, METH_NOARGS, - logreader_fileno__doc__}, - {NULL, NULL} -}; - -static PyMemberDef logreader_members[] = { - {"info", T_OBJECT, offsetof(LogReaderObject, info), READONLY, - PyDoc_STR("Dictionary mapping informational keys to lists of values.")}, - {NULL} -}; - - -PyDoc_STRVAR(logreader__doc__, -"logreader(filename) --> log-iterator\n\ -Create a log-reader for the timing information file."); - -static PySequenceMethods logreader_as_sequence = { - 0, /* sq_length */ - 0, /* sq_concat */ - 0, /* sq_repeat */ - (ssizeargfunc)logreader_sq_item, /* sq_item */ - 0, /* sq_slice */ - 0, /* sq_ass_item */ - 0, /* sq_ass_slice */ - 0, /* sq_contains */ - 0, /* sq_inplace_concat */ - 0, /* sq_inplace_repeat */ -}; - -static PyObject * -logreader_get_closed(LogReaderObject *self, void *closure) -{ - PyObject *result = (self->logfp == NULL) ? Py_True : Py_False; - Py_INCREF(result); - return result; -} - -static PyGetSetDef logreader_getsets[] = { - {"closed", (getter)logreader_get_closed, NULL, - PyDoc_STR("True if the logreader's input file has already been closed.")}, - {NULL} -}; - -static PyTypeObject LogReaderType = { - PyVarObject_HEAD_INIT(NULL, 0) - "_hotshot.LogReaderType", /* tp_name */ - (int) sizeof(LogReaderObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - (destructor)logreader_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - &logreader_as_sequence, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - logreader__doc__, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - PyObject_SelfIter, /* tp_iter */ - (iternextfunc)logreader_tp_iternext,/* tp_iternext */ - logreader_methods, /* tp_methods */ - logreader_members, /* tp_members */ - logreader_getsets, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ -}; - -static PyObject * -hotshot_logreader(PyObject *unused, PyObject *args) -{ - LogReaderObject *self = NULL; - char *filename; - int c; - int err = 0; - - if (PyArg_ParseTuple(args, "s:logreader", &filename)) { - self = PyObject_New(LogReaderObject, &LogReaderType); - if (self != NULL) { - self->frametimings = 1; - self->linetimings = 0; - self->info = NULL; - self->logfp = fopen(filename, "rb"); - if (self->logfp == NULL) { - PyErr_SetFromErrnoWithFilename(PyExc_IOError, filename); - Py_DECREF(self); - self = NULL; - goto finally; - } - self->info = PyDict_New(); - if (self->info == NULL) { - Py_DECREF(self); - goto finally; - } - /* read initial info */ - for (;;) { - if ((c = fgetc(self->logfp)) == EOF) { - eof_error(self); - break; - } - if (c != WHAT_ADD_INFO) { - ungetc(c, self->logfp); - break; - } - err = unpack_add_info(self); - if (err) { - if (err == ERR_EOF) - eof_error(self); - else - PyErr_SetString(PyExc_RuntimeError, - "unexpected error"); - break; - } - } - } - } - finally: - return (PyObject *) self; -} - - -/* Return a Python string that represents the version number without the - * extra cruft added by revision control, even if the right options were - * given to the "cvs export" command to make it not include the extra - * cruft. - */ -static char * -get_version_string(void) -{ - static char *rcsid = "$Revision$"; - char *rev = rcsid; - char *buffer; - int i = 0; - - while (*rev && !isdigit(Py_CHARMASK(*rev))) - ++rev; - while (rev[i] != ' ' && rev[i] != '\0') - ++i; - buffer = (char *)malloc(i + 1); - if (buffer != NULL) { - memmove(buffer, rev, i); - buffer[i] = '\0'; - } - return buffer; -} - -/* Write out a RFC 822-style header with various useful bits of - * information to make the output easier to manage. - */ -static int -write_header(ProfilerObject *self) -{ - char *buffer; - char cwdbuffer[PATH_MAX]; - PyObject *temp; - Py_ssize_t i, len; - - buffer = get_version_string(); - if (buffer == NULL) { - PyErr_NoMemory(); - return -1; - } - pack_add_info(self, "hotshot-version", buffer); - pack_add_info(self, "requested-frame-timings", - (self->frametimings ? "yes" : "no")); - pack_add_info(self, "requested-line-events", - (self->lineevents ? "yes" : "no")); - pack_add_info(self, "requested-line-timings", - (self->linetimings ? "yes" : "no")); - pack_add_info(self, "platform", Py_GetPlatform()); - pack_add_info(self, "executable", Py_GetProgramFullPath()); - free(buffer); - buffer = (char *) Py_GetVersion(); - if (buffer == NULL) - PyErr_Clear(); - else - pack_add_info(self, "executable-version", buffer); - -#ifdef MS_WINDOWS - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%I64d", frequency.QuadPart); - pack_add_info(self, "reported-performance-frequency", cwdbuffer); -#else - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%lu", rusage_diff); - pack_add_info(self, "observed-interval-getrusage", cwdbuffer); - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%lu", timeofday_diff); - pack_add_info(self, "observed-interval-gettimeofday", cwdbuffer); -#endif - - pack_add_info(self, "current-directory", - getcwd(cwdbuffer, sizeof cwdbuffer)); - - temp = PySys_GetObject("path"); - if (temp == NULL || !PyList_Check(temp)) { - PyErr_SetString(PyExc_RuntimeError, "sys.path must be a list"); - return -1; - } - len = PyList_GET_SIZE(temp); - for (i = 0; i < len; ++i) { - PyObject *item = PyList_GET_ITEM(temp, i); - buffer = PyString_AsString(item); - if (buffer == NULL) { - pack_add_info(self, "sys-path-entry", ""); - PyErr_Clear(); - } - else { - pack_add_info(self, "sys-path-entry", buffer); - } - } - pack_frame_times(self); - pack_line_times(self); - - return 0; -} - -PyDoc_STRVAR(profiler__doc__, -"profiler(logfilename[, lineevents[, linetimes]]) -> profiler\n\ -Create a new profiler object."); - -static PyObject * -hotshot_profiler(PyObject *unused, PyObject *args) -{ - char *logfilename; - ProfilerObject *self = NULL; - int lineevents = 0; - int linetimings = 1; - - if (PyArg_ParseTuple(args, "s|ii:profiler", &logfilename, - &lineevents, &linetimings)) { - self = PyObject_New(ProfilerObject, &ProfilerType); - if (self == NULL) - return NULL; - self->frametimings = 1; - self->lineevents = lineevents ? 1 : 0; - self->linetimings = (lineevents && linetimings) ? 1 : 0; - self->index = 0; - self->active = 0; - self->next_fileno = 0; - self->logfp = NULL; - self->logfilename = PyTuple_GET_ITEM(args, 0); - Py_INCREF(self->logfilename); - self->filemap = PyDict_New(); - if (self->filemap == NULL) { - Py_DECREF(self); - return NULL; - } - self->logfp = fopen(logfilename, "wb"); - if (self->logfp == NULL) { - Py_DECREF(self); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, logfilename); - return NULL; - } - if (timeofday_diff == 0) { - /* Run this several times since sometimes the first - * doesn't give the lowest values, and we're really trying - * to determine the lowest. - */ - calibrate(); - calibrate(); - calibrate(); - } - if (write_header(self)) { - /* some error occurred, exception has been set */ - Py_DECREF(self); - self = NULL; - } - } - return (PyObject *) self; -} - -PyDoc_STRVAR(coverage__doc__, -"coverage(logfilename) -> profiler\n\ -Returns a profiler that doesn't collect any timing information, which is\n\ -useful in building a coverage analysis tool."); - -static PyObject * -hotshot_coverage(PyObject *unused, PyObject *args) -{ - char *logfilename; - PyObject *result = NULL; - - if (PyArg_ParseTuple(args, "s:coverage", &logfilename)) { - result = hotshot_profiler(unused, args); - if (result != NULL) { - ProfilerObject *self = (ProfilerObject *) result; - self->frametimings = 0; - self->linetimings = 0; - self->lineevents = 1; - } - } - return result; -} - -PyDoc_VAR(resolution__doc__) = -#ifdef MS_WINDOWS -PyDoc_STR( -"resolution() -> (performance-counter-ticks, update-frequency)\n" -"Return the resolution of the timer provided by the QueryPerformanceCounter()\n" -"function. The first value is the smallest observed change, and the second\n" -"is the result of QueryPerformanceFrequency()." -) -#else -PyDoc_STR( -"resolution() -> (gettimeofday-usecs, getrusage-usecs)\n" -"Return the resolution of the timers provided by the gettimeofday() and\n" -"getrusage() system calls, or -1 if the call is not supported." -) -#endif -; - -static PyObject * -hotshot_resolution(PyObject *self, PyObject *unused) -{ - if (timeofday_diff == 0) { - calibrate(); - calibrate(); - calibrate(); - } -#ifdef MS_WINDOWS - return Py_BuildValue("ii", timeofday_diff, frequency.LowPart); -#else - return Py_BuildValue("ii", timeofday_diff, rusage_diff); -#endif -} - - -static PyMethodDef functions[] = { - {"coverage", hotshot_coverage, METH_VARARGS, coverage__doc__}, - {"profiler", hotshot_profiler, METH_VARARGS, profiler__doc__}, - {"logreader", hotshot_logreader, METH_VARARGS, logreader__doc__}, - {"resolution", hotshot_resolution, METH_NOARGS, resolution__doc__}, - {NULL, NULL} -}; - - -void -init_hotshot(void) -{ - PyObject *module; - - Py_Type(&LogReaderType) = &PyType_Type; - Py_Type(&ProfilerType) = &PyType_Type; - module = Py_InitModule("_hotshot", functions); - if (module != NULL) { - char *s = get_version_string(); - - PyModule_AddStringConstant(module, "__version__", s); - free(s); - Py_INCREF(&LogReaderType); - PyModule_AddObject(module, "LogReaderType", - (PyObject *)&LogReaderType); - Py_INCREF(&ProfilerType); - PyModule_AddObject(module, "ProfilerType", - (PyObject *)&ProfilerType); - - if (ProfilerError == NULL) - ProfilerError = PyErr_NewException("hotshot.ProfilerError", - NULL, NULL); - if (ProfilerError != NULL) { - Py_INCREF(ProfilerError); - PyModule_AddObject(module, "ProfilerError", ProfilerError); - } - PyModule_AddIntConstant(module, "WHAT_ENTER", WHAT_ENTER); - PyModule_AddIntConstant(module, "WHAT_EXIT", WHAT_EXIT); - PyModule_AddIntConstant(module, "WHAT_LINENO", WHAT_LINENO); - PyModule_AddIntConstant(module, "WHAT_OTHER", WHAT_OTHER); - PyModule_AddIntConstant(module, "WHAT_ADD_INFO", WHAT_ADD_INFO); - PyModule_AddIntConstant(module, "WHAT_DEFINE_FILE", WHAT_DEFINE_FILE); - PyModule_AddIntConstant(module, "WHAT_DEFINE_FUNC", WHAT_DEFINE_FUNC); - PyModule_AddIntConstant(module, "WHAT_LINE_TIMES", WHAT_LINE_TIMES); - } -} Deleted: /python/branches/py3k/Tools/scripts/hotshotmain.py ============================================================================== --- /python/branches/py3k/Tools/scripts/hotshotmain.py Thu Oct 11 20:01:43 2007 +++ (empty file) @@ -1,60 +0,0 @@ -#!/usr/bin/env python -# -*- coding: iso-8859-1 -*- - -""" -Run a Python script under hotshot's control. - -Adapted from a posting on python-dev by Walter D?rwald - -usage %prog [ %prog args ] filename [ filename args ] - -Any arguments after the filename are used as sys.argv for the filename. -""" - -import sys -import optparse -import os -import hotshot -import hotshot.stats - -PROFILE = "hotshot.prof" - -def run_hotshot(filename, profile, args): - prof = hotshot.Profile(profile) - sys.path.insert(0, os.path.dirname(filename)) - sys.argv = [filename] + args - fp = open(filename) - try: - script = fp.read() - finally: - fp.close() - prof.run("exec(%r)" % script) - prof.close() - stats = hotshot.stats.load(profile) - stats.sort_stats("time", "calls") - - # print_stats uses unadorned print statements, so the only way - # to force output to stderr is to reassign sys.stdout temporarily - save_stdout = sys.stdout - sys.stdout = sys.stderr - stats.print_stats() - sys.stdout = save_stdout - - return 0 - -def main(args): - parser = optparse.OptionParser(__doc__) - parser.disable_interspersed_args() - parser.add_option("-p", "--profile", action="store", default=PROFILE, - dest="profile", help='Specify profile file to use') - (options, args) = parser.parse_args(args) - - if len(args) == 0: - parser.print_help("missing script to execute") - return 1 - - filename = args[0] - return run_hotshot(filename, options.profile, args[1:]) - -if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) Modified: python/branches/py3k/setup.py ============================================================================== --- python/branches/py3k/setup.py (original) +++ python/branches/py3k/setup.py Thu Oct 11 20:01:43 2007 @@ -413,8 +413,7 @@ exts.append( Extension("atexit", ["atexitmodule.c"]) ) # Python C API test module exts.append( Extension('_testcapi', ['_testcapimodule.c']) ) - # profilers (_lsprof is for cProfile.py) - exts.append( Extension('_hotshot', ['_hotshot.c']) ) + # profiler (_lsprof is for cProfile.py) exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) ) # static Unicode character database exts.append( Extension('unicodedata', ['unicodedata.c']) ) From python-3000-checkins at python.org Thu Oct 11 20:05:39 2007 From: python-3000-checkins at python.org (fred.drake) Date: Thu, 11 Oct 2007 20:05:39 +0200 (CEST) Subject: [Python-3000-checkins] r58414 - in python/branches/py3k: PC/config.c PCbuild/python.iss PCbuild/python20.wse PCbuild/pythoncore.vcproj Message-ID: <20071011180539.573671E401E@bag.python.org> Author: fred.drake Date: Thu Oct 11 20:05:38 2007 New Revision: 58414 Modified: python/branches/py3k/PC/config.c python/branches/py3k/PCbuild/python.iss python/branches/py3k/PCbuild/python20.wse python/branches/py3k/PCbuild/pythoncore.vcproj Log: clean out some hotshot-related build turds Modified: python/branches/py3k/PC/config.c ============================================================================== --- python/branches/py3k/PC/config.c (original) +++ python/branches/py3k/PC/config.c Thu Oct 11 20:05:38 2007 @@ -28,7 +28,6 @@ #endif extern void init_codecs(void); extern void init_weakref(void); -extern void init_hotshot(void); extern void initxxsubtype(void); extern void initzipimport(void); extern void init_random(void); Modified: python/branches/py3k/PCbuild/python.iss ============================================================================== --- python/branches/py3k/PCbuild/python.iss (original) +++ python/branches/py3k/PCbuild/python.iss Thu Oct 11 20:05:38 2007 @@ -174,10 +174,8 @@ Source: Lib\distutils\*.*; DestDir: {app}\Lib\distutils; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\email\*.*; DestDir: {app}\Lib\email; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\encodings\*.*; DestDir: {app}\Lib\encodings; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs -Source: Lib\hotshot\*.*; DestDir: {app}\Lib\hotshot; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\lib-old\*.*; DestDir: {app}\Lib\lib-old; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\xml\*.*; DestDir: {app}\Lib\xml; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs -Source: Lib\hotshot\*.*; DestDir: {app}\Lib\hotshot; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\test\*.*; DestDir: {app}\Lib\test; CopyMode: alwaysoverwrite; Components: test; Flags: recursesubdirs Source: Lib\site-packages\README.txt; DestDir: {app}\Lib\site-packages; CopyMode: alwaysoverwrite; Components: main @@ -284,9 +282,6 @@ Name: {app}\Lib\encodings\*.pyc; Type: files Name: {app}\Lib\encodings\*.pyo; Type: files Name: {app}\Lib\encodings; Type: dirifempty -Name: {app}\Lib\hotshot\*.pyc; Type: files -Name: {app}\Lib\hotshot\*.pyo; Type: files -Name: {app}\Lib\hotshot; Type: dirifempty Name: {app}\Lib\lib-old\*.pyc; Type: files Name: {app}\Lib\lib-old\*.pyo; Type: files Name: {app}\Lib\lib-old; Type: dirifempty Modified: python/branches/py3k/PCbuild/python20.wse ============================================================================== --- python/branches/py3k/PCbuild/python20.wse (original) +++ python/branches/py3k/PCbuild/python20.wse Thu Oct 11 20:05:38 2007 @@ -2024,14 +2024,6 @@ item: Remark end item: Install File - Source=..\lib\hotshot\*.py - Destination=%MAINDIR%\Lib\hotshot - Description=Fast Python profiler - Flags=0000000000000010 -end -item: Remark -end -item: Install File Source=..\lib\lib-old\*.py Destination=%MAINDIR%\Lib\lib-old Description=Obsolete modules Modified: python/branches/py3k/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/py3k/PCbuild/pythoncore.vcproj (original) +++ python/branches/py3k/PCbuild/pythoncore.vcproj Thu Oct 11 20:05:38 2007 @@ -380,9 +380,6 @@ RelativePath="..\Modules\_heapqmodule.c"> - - Author: brett.cannon Date: Thu Oct 11 23:44:13 2007 New Revision: 58416 Added: python/branches/py3k-importlib/Tools/scripts/make_ctype.py - copied unchanged from r58414, python/branches/py3k/Tools/scripts/make_ctype.py Removed: python/branches/py3k-importlib/Doc/library/hotshot.rst python/branches/py3k-importlib/Include/bufferobject.h python/branches/py3k-importlib/Lib/hotshot/ python/branches/py3k-importlib/Lib/test/test_buffer.py python/branches/py3k-importlib/Lib/test/test_hotshot.py python/branches/py3k-importlib/Modules/_hotshot.c python/branches/py3k-importlib/Objects/bufferobject.c python/branches/py3k-importlib/Tools/scripts/hotshotmain.py Modified: python/branches/py3k-importlib/ (props changed) python/branches/py3k-importlib/Doc/library/debug.rst python/branches/py3k-importlib/Doc/library/functions.rst python/branches/py3k-importlib/Doc/library/profile.rst python/branches/py3k-importlib/Doc/tutorial/datastructures.rst python/branches/py3k-importlib/Include/Python.h python/branches/py3k-importlib/Lib/_abcoll.py python/branches/py3k-importlib/Lib/ctypes/test/test_array_in_pointer.py python/branches/py3k-importlib/Lib/ctypes/test/test_byteswap.py python/branches/py3k-importlib/Lib/ctypes/test/test_strings.py python/branches/py3k-importlib/Lib/ftplib.py python/branches/py3k-importlib/Lib/idlelib/IOBinding.py python/branches/py3k-importlib/Lib/idlelib/NEWS.txt python/branches/py3k-importlib/Lib/pickle.py python/branches/py3k-importlib/Lib/sqlite3/dbapi2.py python/branches/py3k-importlib/Lib/sqlite3/test/dbapi.py python/branches/py3k-importlib/Lib/sqlite3/test/types.py python/branches/py3k-importlib/Lib/sqlite3/test/userfunctions.py python/branches/py3k-importlib/Lib/subprocess.py python/branches/py3k-importlib/Lib/test/test_array.py python/branches/py3k-importlib/Lib/test/test_bytes.py python/branches/py3k-importlib/Lib/test/test_io.py python/branches/py3k-importlib/Lib/test/test_marshal.py python/branches/py3k-importlib/Lib/test/test_multibytecodec_support.py python/branches/py3k-importlib/Lib/test/test_repr.py python/branches/py3k-importlib/Lib/test/test_socket_ssl.py python/branches/py3k-importlib/Lib/test/test_struct.py python/branches/py3k-importlib/Lib/test/test_types.py python/branches/py3k-importlib/Lib/test/test_unicode.py python/branches/py3k-importlib/Lib/test/test_xml_etree.py python/branches/py3k-importlib/Lib/test/test_xml_etree_c.py python/branches/py3k-importlib/Lib/types.py python/branches/py3k-importlib/Makefile.pre.in python/branches/py3k-importlib/Misc/NEWS python/branches/py3k-importlib/Modules/_ctypes/_ctypes.c python/branches/py3k-importlib/Modules/_sqlite/connection.c python/branches/py3k-importlib/Modules/_sqlite/cursor.c python/branches/py3k-importlib/Modules/_sqlite/statement.c python/branches/py3k-importlib/Objects/bytesobject.c python/branches/py3k-importlib/Objects/fileobject.c python/branches/py3k-importlib/Objects/memoryobject.c python/branches/py3k-importlib/Objects/stringobject.c python/branches/py3k-importlib/Objects/unicodeobject.c python/branches/py3k-importlib/PC/VC6/pythoncore.dsp python/branches/py3k-importlib/PC/config.c python/branches/py3k-importlib/PC/os2emx/Makefile python/branches/py3k-importlib/PC/os2emx/python25.def python/branches/py3k-importlib/PCbuild/python.iss python/branches/py3k-importlib/PCbuild/python20.wse python/branches/py3k-importlib/PCbuild/pythoncore.vcproj python/branches/py3k-importlib/PCbuild8/pythoncore/pythoncore.vcproj python/branches/py3k-importlib/Parser/tokenizer.c python/branches/py3k-importlib/Python/errors.c python/branches/py3k-importlib/Python/pythonrun.c python/branches/py3k-importlib/setup.py Log: Merged revisions 58355-58415 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k Modified: python/branches/py3k-importlib/Doc/library/debug.rst ============================================================================== --- python/branches/py3k-importlib/Doc/library/debug.rst (original) +++ python/branches/py3k-importlib/Doc/library/debug.rst Thu Oct 11 23:44:13 2007 @@ -12,6 +12,5 @@ bdb.rst pdb.rst profile.rst - hotshot.rst timeit.rst trace.rst \ No newline at end of file Modified: python/branches/py3k-importlib/Doc/library/functions.rst ============================================================================== --- python/branches/py3k-importlib/Doc/library/functions.rst (original) +++ python/branches/py3k-importlib/Doc/library/functions.rst Thu Oct 11 23:44:13 2007 @@ -1079,34 +1079,6 @@ returns an empty iterator. -.. % --------------------------------------------------------------------------- - - -.. _non-essential-built-in-funcs: - -Non-essential Built-in Functions -================================ - -There are several built-in functions that are no longer essential to learn, know -or use in modern Python programming. They have been kept here to maintain -backwards compatibility with programs written for older versions of Python. - -Python programmers, trainers, students and bookwriters should feel free to -bypass these functions without concerns about missing something important. - - -.. XXX does this go away? -.. function:: buffer(object[, offset[, size]]) - - The *object* argument must be an object that supports the buffer call interface - (such as strings, arrays, and buffers). A new buffer object will be created - which references the *object* argument. The buffer object will be a slice from - the beginning of *object* (or from the specified *offset*). The slice will - extend to the end of *object* (or will have a length given by the *size* - argument). - - - .. rubric:: Footnotes .. [#] Specifying a buffer size currently has no effect on systems that don't have Deleted: /python/branches/py3k-importlib/Doc/library/hotshot.rst ============================================================================== --- /python/branches/py3k-importlib/Doc/library/hotshot.rst Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,144 +0,0 @@ - -:mod:`hotshot` --- High performance logging profiler -==================================================== - -.. module:: hotshot - :synopsis: High performance logging profiler, mostly written in C. -.. moduleauthor:: Fred L. Drake, Jr. -.. sectionauthor:: Anthony Baxter - - -This module provides a nicer interface to the :mod:`_hotshot` C module. Hotshot -is a replacement for the existing :mod:`profile` module. As it's written mostly -in C, it should result in a much smaller performance impact than the existing -:mod:`profile` module. - -.. note:: - - The :mod:`hotshot` module focuses on minimizing the overhead while profiling, at - the expense of long data post-processing times. For common usages it is - recommended to use :mod:`cProfile` instead. :mod:`hotshot` is not maintained and - might be removed from the standard library in the future. - -.. warning:: - - The :mod:`hotshot` profiler does not yet work well with threads. It is useful to - use an unthreaded script to run the profiler over the code you're interested in - measuring if at all possible. - - -.. class:: Profile(logfile[, lineevents[, linetimings]]) - - The profiler object. The argument *logfile* is the name of a log file to use for - logged profile data. The argument *lineevents* specifies whether to generate - events for every source line, or just on function call/return. It defaults to - ``0`` (only log function call/return). The argument *linetimings* specifies - whether to record timing information. It defaults to ``1`` (store timing - information). - - -.. _hotshot-objects: - -Profile Objects ---------------- - -Profile objects have the following methods: - - -.. method:: Profile.addinfo(key, value) - - Add an arbitrary labelled value to the profile output. - - -.. method:: Profile.close() - - Close the logfile and terminate the profiler. - - -.. method:: Profile.fileno() - - Return the file descriptor of the profiler's log file. - - -.. method:: Profile.run(cmd) - - Profile an :func:`exec`\ -compatible string in the script environment. The - globals from the :mod:`__main__` module are used as both the globals and locals - for the script. - - -.. method:: Profile.runcall(func, *args, **keywords) - - Profile a single call of a callable. Additional positional and keyword arguments - may be passed along; the result of the call is returned, and exceptions are - allowed to propagate cleanly, while ensuring that profiling is disabled on the - way out. - - -.. method:: Profile.runctx(cmd, globals, locals) - - Profile an :func:`exec`\ -compatible string in a specific environment. The - string is compiled before profiling begins. - - -.. method:: Profile.start() - - Start the profiler. - - -.. method:: Profile.stop() - - Stop the profiler. - - -Using hotshot data ------------------- - -.. module:: hotshot.stats - :synopsis: Statistical analysis for Hotshot - - -This module loads hotshot profiling data into the standard :mod:`pstats` Stats -objects. - - -.. function:: load(filename) - - Load hotshot data from *filename*. Returns an instance of the - :class:`pstats.Stats` class. - - -.. seealso:: - - Module :mod:`profile` - The :mod:`profile` module's :class:`Stats` class - - -.. _hotshot-example: - -Example Usage -------------- - -Note that this example runs the python "benchmark" pystones. It can take some -time to run, and will produce large output files. :: - - >>> import hotshot, hotshot.stats, test.pystone - >>> prof = hotshot.Profile("stones.prof") - >>> benchtime, stones = prof.runcall(test.pystone.pystones) - >>> prof.close() - >>> stats = hotshot.stats.load("stones.prof") - >>> stats.strip_dirs() - >>> stats.sort_stats('time', 'calls') - >>> stats.print_stats(20) - 850004 function calls in 10.090 CPU seconds - - Ordered by: internal time, call count - - ncalls tottime percall cumtime percall filename:lineno(function) - 1 3.295 3.295 10.090 10.090 pystone.py:79(Proc0) - 150000 1.315 0.000 1.315 0.000 pystone.py:203(Proc7) - 50000 1.313 0.000 1.463 0.000 pystone.py:229(Func2) - . - . - . - Modified: python/branches/py3k-importlib/Doc/library/profile.rst ============================================================================== --- python/branches/py3k-importlib/Doc/library/profile.rst (original) +++ python/branches/py3k-importlib/Doc/library/profile.rst Thu Oct 11 23:44:13 2007 @@ -57,7 +57,7 @@ It also provides a series of report generation tools to allow users to rapidly examine the results of a profile operation. -The Python standard library provides three different profilers: +The Python standard library provides two different profilers: #. :mod:`profile`, a pure Python module, described in the sequel. Copyright ?? 1994, by InfoSeek Corporation. @@ -66,15 +66,11 @@ it suitable for profiling long-running programs. Based on :mod:`lsprof`, contributed by Brett Rosen and Ted Czotter. -#. :mod:`hotshot`, a C module focusing on minimizing the overhead while - profiling, at the expense of long data post-processing times. - The :mod:`profile` and :mod:`cProfile` modules export the same interface, so they are mostly interchangeables; :mod:`cProfile` has a much lower overhead but is not so far as well-tested and might not be available on all systems. :mod:`cProfile` is really a compatibility layer on top of the internal -:mod:`_lsprof` module. The :mod:`hotshot` module is reserved to specialized -usages. +:mod:`_lsprof` module. .. % \section{How Is This Profiler Different From The Old Profiler?} .. % \nodename{Profiler Changes} Modified: python/branches/py3k-importlib/Doc/tutorial/datastructures.rst ============================================================================== --- python/branches/py3k-importlib/Doc/tutorial/datastructures.rst (original) +++ python/branches/py3k-importlib/Doc/tutorial/datastructures.rst Thu Oct 11 23:44:13 2007 @@ -560,15 +560,8 @@ (1, 2, 3) == (1.0, 2.0, 3.0) (1, 2, ('aa', 'ab')) < (1, 2, ('abc', 'a'), 4) -Note that comparing objects of different types is legal. The outcome is -deterministic but arbitrary: the types are ordered by their name. Thus, a list -is always smaller than a string, a string is always smaller than a tuple, etc. -[#]_ Mixed numeric types are compared according to their numeric value, so 0 -equals 0.0, etc. - - -.. rubric:: Footnotes - -.. [#] The rules for comparing objects of different types should not be relied upon; - they may change in a future version of the language. - +Note that comparing objects of different types with ``<`` or ``>`` is legal +provided that the objects have appropriate comparison methods. For example, +mixed numeric types are compared according to their numeric value, so 0 equals +0.0, etc. Otherwise, rather than providing an arbitrary ordering, the +interpreter will raise a :exc:`TypeError` exception. Modified: python/branches/py3k-importlib/Include/Python.h ============================================================================== --- python/branches/py3k-importlib/Include/Python.h (original) +++ python/branches/py3k-importlib/Include/Python.h Thu Oct 11 23:44:13 2007 @@ -76,7 +76,6 @@ #endif #include "rangeobject.h" #include "stringobject.h" -#include "bufferobject.h" #include "memoryobject.h" #include "tupleobject.h" #include "listobject.h" Deleted: /python/branches/py3k-importlib/Include/bufferobject.h ============================================================================== --- /python/branches/py3k-importlib/Include/bufferobject.h Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,33 +0,0 @@ - -/* Buffer object interface */ - -/* Note: the object's structure is private */ - -#ifndef Py_BUFFEROBJECT_H -#define Py_BUFFEROBJECT_H -#ifdef __cplusplus -extern "C" { -#endif - - -PyAPI_DATA(PyTypeObject) PyBuffer_Type; - -#define PyBuffer_Check(op) (Py_Type(op) == &PyBuffer_Type) - -#define Py_END_OF_BUFFER (-1) - -PyAPI_FUNC(PyObject *) PyBuffer_FromObject(PyObject *base, - Py_ssize_t offset, Py_ssize_t size); -PyAPI_FUNC(PyObject *) PyBuffer_FromReadWriteObject(PyObject *base, - Py_ssize_t offset, - Py_ssize_t size); - -PyAPI_FUNC(PyObject *) PyBuffer_FromMemory(void *ptr, Py_ssize_t size); -PyAPI_FUNC(PyObject *) PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size); - -PyAPI_FUNC(PyObject *) PyBuffer_New(Py_ssize_t size); - -#ifdef __cplusplus -} -#endif -#endif /* !Py_BUFFEROBJECT_H */ Modified: python/branches/py3k-importlib/Lib/_abcoll.py ============================================================================== --- python/branches/py3k-importlib/Lib/_abcoll.py (original) +++ python/branches/py3k-importlib/Lib/_abcoll.py Thu Oct 11 23:44:13 2007 @@ -491,7 +491,7 @@ Sequence.register(tuple) Sequence.register(basestring) -Sequence.register(buffer) +Sequence.register(memoryview) class MutableSequence(Sequence): Modified: python/branches/py3k-importlib/Lib/ctypes/test/test_array_in_pointer.py ============================================================================== --- python/branches/py3k-importlib/Lib/ctypes/test/test_array_in_pointer.py (original) +++ python/branches/py3k-importlib/Lib/ctypes/test/test_array_in_pointer.py Thu Oct 11 23:44:13 2007 @@ -6,7 +6,7 @@ def dump(obj): # helper function to dump memory contents in hex, with a hyphen # between the bytes. - h = str(hexlify(buffer(obj))) + h = str(hexlify(memoryview(obj))) return re.sub(r"(..)", r"\1-", h)[:-1] Modified: python/branches/py3k-importlib/Lib/ctypes/test/test_byteswap.py ============================================================================== --- python/branches/py3k-importlib/Lib/ctypes/test/test_byteswap.py (original) +++ python/branches/py3k-importlib/Lib/ctypes/test/test_byteswap.py Thu Oct 11 23:44:13 2007 @@ -4,7 +4,7 @@ from ctypes import * def bin(s): - return str(hexlify(buffer(s))).upper() + return str(hexlify(memoryview(s))).upper() # Each *simple* type that supports different byte orders has an # __ctype_be__ attribute that specifies the same type in BIG ENDIAN Modified: python/branches/py3k-importlib/Lib/ctypes/test/test_strings.py ============================================================================== --- python/branches/py3k-importlib/Lib/ctypes/test/test_strings.py (original) +++ python/branches/py3k-importlib/Lib/ctypes/test/test_strings.py Thu Oct 11 23:44:13 2007 @@ -30,17 +30,17 @@ buf.value = "Hello, World" self.failUnlessEqual(buf.value, "Hello, World") - self.failUnlessRaises(TypeError, setattr, buf, "value", buffer("Hello, World")) - self.assertRaises(TypeError, setattr, buf, "value", buffer("abc")) - self.assertRaises(ValueError, setattr, buf, "raw", buffer("x" * 100)) + self.failUnlessRaises(TypeError, setattr, buf, "value", memoryview(b"Hello, World")) + self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) + self.assertRaises(ValueError, setattr, buf, "raw", memoryview(b"x" * 100)) def test_c_buffer_raw(self): buf = c_buffer(32) - buf.raw = buffer(b"Hello, World") + buf.raw = memoryview(b"Hello, World") self.failUnlessEqual(buf.value, "Hello, World") - self.assertRaises(TypeError, setattr, buf, "value", buffer("abc")) - self.assertRaises(ValueError, setattr, buf, "raw", buffer("x" * 100)) + self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) + self.assertRaises(ValueError, setattr, buf, "raw", memoryview(b"x" * 100)) def test_param_1(self): BUF = c_char * 4 Modified: python/branches/py3k-importlib/Lib/ftplib.py ============================================================================== --- python/branches/py3k-importlib/Lib/ftplib.py (original) +++ python/branches/py3k-importlib/Lib/ftplib.py Thu Oct 11 23:44:13 2007 @@ -103,7 +103,7 @@ file = None welcome = None passiveserver = 1 - encoding = "ASCII" + encoding = "latin1" # Initialization method (called by class instantiation). # Initialize host to localhost, port to standard ftp port Modified: python/branches/py3k-importlib/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/py3k-importlib/Lib/idlelib/IOBinding.py (original) +++ python/branches/py3k-importlib/Lib/idlelib/IOBinding.py Thu Oct 11 23:44:13 2007 @@ -242,7 +242,7 @@ eol = r"(\r\n)|\n|\r" # \r\n (Windows), \n (UNIX), or \r (Mac) eol_re = re.compile(eol) - eol_convention = os.linesep # Default + eol_convention = os.linesep # default def loadfile(self, filename): try: @@ -389,9 +389,10 @@ def writefile(self, filename): self.fixlastline() - chars = self.encode(self.text.get("1.0", "end-1c")) + text = self.text.get("1.0", "end-1c") if self.eol_convention != "\n": - chars = chars.replace("\n", self.eol_convention) + text = text.replace("\n", self.eol_convention) + chars = self.encode(self.text.get("1.0", "end-1c")) try: f = open(filename, "wb") f.write(chars) Modified: python/branches/py3k-importlib/Lib/idlelib/NEWS.txt ============================================================================== --- python/branches/py3k-importlib/Lib/idlelib/NEWS.txt (original) +++ python/branches/py3k-importlib/Lib/idlelib/NEWS.txt Thu Oct 11 23:44:13 2007 @@ -1,3 +1,12 @@ +What's New in IDLE 3.0a2? +========================= + +*Release date: XX-XXX-2007* + +- Windows EOL sequence not converted correctly, encoding error. + Caused file save to fail. Bug 1130. + + What's New in IDLE 3.0a1? ========================= @@ -19,7 +28,7 @@ What's New in IDLE 2.6a1? ========================= -*Release date: XX-XXX-200X* UNRELEASED +*Release date: XX-XXX-200X* UNRELEASED, but merged into 3.0a1 - Corrected some bugs in AutoComplete. Also, Page Up/Down in ACW implemented; mouse and cursor selection in ACWindow implemented; double Tab inserts Modified: python/branches/py3k-importlib/Lib/pickle.py ============================================================================== --- python/branches/py3k-importlib/Lib/pickle.py (original) +++ python/branches/py3k-importlib/Lib/pickle.py Thu Oct 11 23:44:13 2007 @@ -778,31 +778,17 @@ The protocol version of the pickle is detected automatically, so no proto argument is needed. - The file-like object must have two methods, a read() method that - takes an integer argument, and a readline() method that requires no - arguments. Both methods should return a string. Thus file-like - object can be a file object opened for reading, a StringIO object, - or any other custom object that meets this interface. + The file-like object must have two methods, a read() method + that takes an integer argument, and a readline() method that + requires no arguments. Both methods should return bytes. + Thus file-like object can be a binary file object opened for + reading, a BytesIO object, or any other custom object that + meets this interface. """ - try: - self.readline = file.readline - except AttributeError: - self.file = file + self.readline = file.readline self.read = file.read self.memo = {} - def readline(self): - # XXX Slow but at least correct - b = bytes() - while True: - c = self.file.read(1) - if not c: - break - b += c - if c == b'\n': - break - return b - def load(self): """Read a pickled object representation from the open file. @@ -895,7 +881,8 @@ dispatch[BININT2[0]] = load_binint2 def load_long(self): - self.append(int(str(self.readline()[:-1]), 0)) + val = self.readline()[:-1].decode("ascii") + self.append(int(val, 0)) dispatch[LONG[0]] = load_long def load_long1(self): @@ -1076,8 +1063,10 @@ def find_class(self, module, name): # Subclasses may override this - module = str(module) - name = str(name) + if isinstance(module, bytes): + module = module.decode("utf-8") + if isinstance(name, bytes): + name = name.decode("utf-8") __import__(module) mod = sys.modules[module] klass = getattr(mod, name) @@ -1110,7 +1099,7 @@ dispatch[DUP[0]] = load_dup def load_get(self): - self.append(self.memo[str8(self.readline())[:-1]]) + self.append(self.memo[str(self.readline())[:-1]]) dispatch[GET[0]] = load_get def load_binget(self): Modified: python/branches/py3k-importlib/Lib/sqlite3/dbapi2.py ============================================================================== --- python/branches/py3k-importlib/Lib/sqlite3/dbapi2.py (original) +++ python/branches/py3k-importlib/Lib/sqlite3/dbapi2.py Thu Oct 11 23:44:13 2007 @@ -50,7 +50,7 @@ version_info = tuple([int(x) for x in version.split(".")]) sqlite_version_info = tuple([int(x) for x in sqlite_version.split(".")]) -Binary = buffer +Binary = memoryview def register_adapters_and_converters(): def adapt_date(val): Modified: python/branches/py3k-importlib/Lib/sqlite3/test/dbapi.py ============================================================================== --- python/branches/py3k-importlib/Lib/sqlite3/test/dbapi.py (original) +++ python/branches/py3k-importlib/Lib/sqlite3/test/dbapi.py Thu Oct 11 23:44:13 2007 @@ -593,7 +593,7 @@ ts = sqlite.TimestampFromTicks(42) def CheckBinary(self): - b = sqlite.Binary(chr(0) + "'") + b = sqlite.Binary(b"\0'") class ExtensionTests(unittest.TestCase): def CheckScriptStringSql(self): Modified: python/branches/py3k-importlib/Lib/sqlite3/test/types.py ============================================================================== --- python/branches/py3k-importlib/Lib/sqlite3/test/types.py (original) +++ python/branches/py3k-importlib/Lib/sqlite3/test/types.py Thu Oct 11 23:44:13 2007 @@ -62,7 +62,7 @@ self.failUnlessEqual(row[0], val) def CheckBlob(self): - val = buffer(b"Guglhupf") + val = memoryview(b"Guglhupf") self.cur.execute("insert into test(b) values (?)", (val,)) self.cur.execute("select b from test") row = self.cur.fetchone() @@ -203,7 +203,7 @@ def CheckBlob(self): # default - val = buffer(b"Guglhupf") + val = memoryview(b"Guglhupf") self.cur.execute("insert into test(bin) values (?)", (val,)) self.cur.execute("select bin from test") row = self.cur.fetchone() @@ -305,7 +305,7 @@ def CheckBinaryInputForConverter(self): testdata = b"abcdefg" * 10 - result = self.con.execute('select ? as "x [bin]"', (buffer(bz2.compress(testdata)),)).fetchone()[0] + result = self.con.execute('select ? as "x [bin]"', (memoryview(bz2.compress(testdata)),)).fetchone()[0] self.failUnlessEqual(testdata, result) class DateTimeTests(unittest.TestCase): Modified: python/branches/py3k-importlib/Lib/sqlite3/test/userfunctions.py ============================================================================== --- python/branches/py3k-importlib/Lib/sqlite3/test/userfunctions.py (original) +++ python/branches/py3k-importlib/Lib/sqlite3/test/userfunctions.py Thu Oct 11 23:44:13 2007 @@ -36,7 +36,7 @@ def func_returnnull(): return None def func_returnblob(): - return buffer(b"blob") + return b"blob" def func_raiseexception(): 5/0 @@ -49,7 +49,7 @@ def func_isnone(v): return type(v) is type(None) def func_isblob(v): - return type(v) is buffer + return isinstance(v, (bytes, memoryview)) class AggrNoStep: def __init__(self): @@ -100,7 +100,8 @@ self.val = None def step(self, whichType, val): - theType = {"str": str, "int": int, "float": float, "None": type(None), "blob": buffer} + theType = {"str": str, "int": int, "float": float, "None": type(None), + "blob": bytes} self.val = int(theType[whichType] is type(val)) def finalize(self): @@ -196,8 +197,8 @@ cur = self.con.cursor() cur.execute("select returnblob()") val = cur.fetchone()[0] - self.failUnlessEqual(type(val), buffer) - self.failUnlessEqual(val, buffer(b"blob")) + self.failUnlessEqual(type(val), bytes) + self.failUnlessEqual(val, memoryview(b"blob")) def CheckFuncException(self): cur = self.con.cursor() @@ -234,7 +235,7 @@ def CheckParamBlob(self): cur = self.con.cursor() - cur.execute("select isblob(?)", (buffer(b"blob"),)) + cur.execute("select isblob(?)", (memoryview(b"blob"),)) val = cur.fetchone()[0] self.failUnlessEqual(val, 1) @@ -252,7 +253,7 @@ ) """) cur.execute("insert into test(t, i, f, n, b) values (?, ?, ?, ?, ?)", - ("foo", 5, 3.14, None, buffer(b"blob"),)) + ("foo", 5, 3.14, None, memoryview(b"blob"),)) self.con.create_aggregate("nostep", 1, AggrNoStep) self.con.create_aggregate("nofinalize", 1, AggrNoFinalize) @@ -344,7 +345,7 @@ def CheckAggrCheckParamBlob(self): cur = self.con.cursor() - cur.execute("select checkType('blob', ?)", (buffer(b"blob"),)) + cur.execute("select checkType('blob', ?)", (memoryview(b"blob"),)) val = cur.fetchone()[0] self.failUnlessEqual(val, 1) Modified: python/branches/py3k-importlib/Lib/subprocess.py ============================================================================== --- python/branches/py3k-importlib/Lib/subprocess.py (original) +++ python/branches/py3k-importlib/Lib/subprocess.py Thu Oct 11 23:44:13 2007 @@ -1041,8 +1041,11 @@ def _communicate(self, input): - if isinstance(input, str): # Unicode - input = input.encode("utf-8") # XXX What else? + if self.stdin: + if isinstance(input, str): # Unicode + input = input.encode("utf-8") # XXX What else? + if not isinstance(input, (bytes, str8)): + input = bytes(input) read_set = [] write_set = [] stdout = None # Return @@ -1071,7 +1074,8 @@ # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), buffer(input, input_offset, 512)) + chunk = input[input_offset : input_offset + 512] + bytes_written = os.write(self.stdin.fileno(), chunk) input_offset += bytes_written if input_offset >= len(input): self.stdin.close() Modified: python/branches/py3k-importlib/Lib/test/test_array.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_array.py (original) +++ python/branches/py3k-importlib/Lib/test/test_array.py Thu Oct 11 23:44:13 2007 @@ -708,7 +708,7 @@ def test_buffer(self): a = array.array(self.typecode, self.example) - b = bytes(buffer(a)) + b = bytes(memoryview(a)) self.assertEqual(b[0], a.tostring()[0]) def test_weakref(self): Deleted: /python/branches/py3k-importlib/Lib/test/test_buffer.py ============================================================================== --- /python/branches/py3k-importlib/Lib/test/test_buffer.py Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,56 +0,0 @@ -"""Unit tests for buffer objects. - -For now, we just test (the brand new) rich comparison. - -""" - -import unittest -from test import test_support - -class BufferTests(unittest.TestCase): - - def test_comparison(self): - a = buffer("a.b.c") - b = buffer("a.b" + ".c") - self.assert_(a == b) - self.assert_(a <= b) - self.assert_(a >= b) - self.assert_(a == "a.b.c") - self.assert_(a <= "a.b.c") - self.assert_(a >= "a.b.c") - b = buffer("a.b.c.d") - self.assert_(a != b) - self.assert_(a <= b) - self.assert_(a < b) - self.assert_(a != "a.b.c.d") - self.assert_(a < "a.b.c.d") - self.assert_(a <= "a.b.c.d") - b = buffer("a.b") - self.assert_(a != b) - self.assert_(a >= b) - self.assert_(a > b) - self.assert_(a != "a.b") - self.assert_(a > "a.b") - self.assert_(a >= "a.b") - b = object() - self.assert_(a != b) - self.failIf(a == b) - self.assertRaises(TypeError, lambda: a < b) - - def test_extended_getslice(self): - # Test extended slicing by comparing with list slicing. - s = bytes(range(255, -1, -1)) - b = buffer(s) - indices = (0, None, 1, 3, 19, 300, -1, -2, -31, -300) - for start in indices: - for stop in indices: - # Skip step 0 (invalid) - for step in indices[1:]: - self.assertEqual(b[start:stop:step], - s[start:stop:step]) - -def test_main(): - test_support.run_unittest(BufferTests) - -if __name__ == "__main__": - test_main() Modified: python/branches/py3k-importlib/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_bytes.py (original) +++ python/branches/py3k-importlib/Lib/test/test_bytes.py Thu Oct 11 23:44:13 2007 @@ -130,12 +130,14 @@ self.assertEqual(str8("abc") < b"ab", False) self.assertEqual(str8("abc") <= b"ab", False) - # Bytes can't be compared to Unicode! + # Byte comparisons with unicode should always fail! # Test this for all expected byte orders and Unicode character sizes - self.assertRaises(TypeError, lambda: b"\0a\0b\0c" == "abc") - self.assertRaises(TypeError, lambda: b"\0\0\0a\0\0\0b\0\0\0c" == "abc") - self.assertRaises(TypeError, lambda: b"a\0b\0c\0" == "abc") - self.assertRaises(TypeError, lambda: b"a\0\0\0b\0\0\0c\0\0\0" == "abc") + self.assertEqual(b"\0a\0b\0c" == "abc", False) + self.assertEqual(b"\0\0\0a\0\0\0b\0\0\0c" == "abc", False) + self.assertEqual(b"a\0b\0c\0" == "abc", False) + self.assertEqual(b"a\0\0\0b\0\0\0c\0\0\0" == "abc", False) + self.assertEqual(bytes() == str(), False) + self.assertEqual(bytes() != str(), True) def test_nohash(self): self.assertRaises(TypeError, hash, bytes()) @@ -343,7 +345,7 @@ def test_from_buffer(self): sample = str8("Hello world\n\x80\x81\xfe\xff") - buf = buffer(sample) + buf = memoryview(sample) b = bytes(buf) self.assertEqual(b, bytes(map(ord, sample))) @@ -456,8 +458,8 @@ b = bytes([0x1a, 0x2b, 0x30]) self.assertEquals(bytes.fromhex('1a2B30'), b) self.assertEquals(bytes.fromhex(' 1A 2B 30 '), b) - self.assertEquals(bytes.fromhex(buffer(b'')), bytes()) - self.assertEquals(bytes.fromhex(buffer(b'0000')), bytes([0, 0])) + self.assertEquals(bytes.fromhex(memoryview(b'')), bytes()) + self.assertEquals(bytes.fromhex(memoryview(b'0000')), bytes([0, 0])) self.assertRaises(ValueError, bytes.fromhex, 'a') self.assertRaises(ValueError, bytes.fromhex, 'rt') self.assertRaises(ValueError, bytes.fromhex, '1a b cd') @@ -630,7 +632,7 @@ self.assertEqual(b' a bb c '.split(None, 3), [b'a', b'bb', b'c']) def test_split_buffer(self): - self.assertEqual(b'a b'.split(buffer(b' ')), [b'a', b'b']) + self.assertEqual(b'a b'.split(memoryview(b' ')), [b'a', b'b']) def test_split_string_error(self): self.assertRaises(TypeError, b'a b'.split, ' ') @@ -653,7 +655,7 @@ self.assertEqual(b' a bb c '.rsplit(None, 3), [b'a', b'bb', b'c']) def test_rplit_buffer(self): - self.assertEqual(b'a b'.rsplit(buffer(b' ')), [b'a', b'b']) + self.assertEqual(b'a b'.rsplit(memoryview(b' ')), [b'a', b'b']) def test_rplit_string_error(self): self.assertRaises(TypeError, b'a b'.rsplit, ' ') @@ -707,9 +709,9 @@ self.assertEqual(b.rstrip(), b' \t\n\r\f\vabc') def test_strip_buffer(self): - self.assertEqual(b'abc'.strip(buffer(b'ac')), b'b') - self.assertEqual(b'abc'.lstrip(buffer(b'ac')), b'bc') - self.assertEqual(b'abc'.rstrip(buffer(b'ac')), b'ab') + self.assertEqual(b'abc'.strip(memoryview(b'ac')), b'b') + self.assertEqual(b'abc'.lstrip(memoryview(b'ac')), b'bc') + self.assertEqual(b'abc'.rstrip(memoryview(b'ac')), b'ab') def test_strip_string_error(self): self.assertRaises(TypeError, b'abc'.strip, 'b') Deleted: /python/branches/py3k-importlib/Lib/test/test_hotshot.py ============================================================================== --- /python/branches/py3k-importlib/Lib/test/test_hotshot.py Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,132 +0,0 @@ -import hotshot -import hotshot.log -import os -import pprint -import unittest - -from test import test_support - -from hotshot.log import ENTER, EXIT, LINE - - -def shortfilename(fn): - # We use a really shortened filename since an exact match is made, - # and the source may be either a Python source file or a - # pre-compiled bytecode file. - if fn: - return os.path.splitext(os.path.basename(fn))[0] - else: - return fn - - -class UnlinkingLogReader(hotshot.log.LogReader): - """Extend the LogReader so the log file is unlinked when we're - done with it.""" - - def __init__(self, logfn): - self.__logfn = logfn - hotshot.log.LogReader.__init__(self, logfn) - - def next(self, index=None): - try: - return hotshot.log.LogReader.next(self) - except StopIteration: - self.close() - os.unlink(self.__logfn) - raise - - -class HotShotTestCase(unittest.TestCase): - def new_profiler(self, lineevents=0, linetimings=1): - self.logfn = test_support.TESTFN - return hotshot.Profile(self.logfn, lineevents, linetimings) - - def get_logreader(self): - return UnlinkingLogReader(self.logfn) - - def get_events_wotime(self): - L = [] - for event in self.get_logreader(): - what, (filename, lineno, funcname), tdelta = event - L.append((what, (shortfilename(filename), lineno, funcname))) - return L - - def check_events(self, expected): - events = self.get_events_wotime() - if events != expected: - self.fail( - "events did not match expectation; got:\n%s\nexpected:\n%s" - % (pprint.pformat(events), pprint.pformat(expected))) - - def run_test(self, callable, events, profiler=None): - if profiler is None: - profiler = self.new_profiler() - self.failUnless(not profiler._prof.closed) - profiler.runcall(callable) - self.failUnless(not profiler._prof.closed) - profiler.close() - self.failUnless(profiler._prof.closed) - self.check_events(events) - - def test_addinfo(self): - def f(p): - p.addinfo("test-key", "test-value") - profiler = self.new_profiler() - profiler.runcall(f, profiler) - profiler.close() - log = self.get_logreader() - info = log._info - list(log) - self.assertEqual(info["test-key"], ["test-value"]) - - def test_line_numbers(self): - def f(): - y = 2 - x = 1 - def g(): - f() - f_lineno = f.__code__.co_firstlineno - g_lineno = g.__code__.co_firstlineno - events = [(ENTER, ("test_hotshot", g_lineno, "g")), - (LINE, ("test_hotshot", g_lineno+1, "g")), - (ENTER, ("test_hotshot", f_lineno, "f")), - (LINE, ("test_hotshot", f_lineno+1, "f")), - (LINE, ("test_hotshot", f_lineno+2, "f")), - (EXIT, ("test_hotshot", f_lineno, "f")), - (EXIT, ("test_hotshot", g_lineno, "g")), - ] - self.run_test(g, events, self.new_profiler(lineevents=1)) - - def test_start_stop(self): - # Make sure we don't return NULL in the start() and stop() - # methods when there isn't an error. Bug in 2.2 noted by - # Anthony Baxter. - profiler = self.new_profiler() - profiler.start() - profiler.stop() - profiler.close() - os.unlink(self.logfn) - - def test_bad_sys_path(self): - import sys - import os - orig_path = sys.path - coverage = hotshot._hotshot.coverage - try: - # verify we require a list for sys.path - sys.path = 'abc' - self.assertRaises(RuntimeError, coverage, test_support.TESTFN) - # verify that we require sys.path exists - del sys.path - self.assertRaises(RuntimeError, coverage, test_support.TESTFN) - finally: - sys.path = orig_path - if os.path.exists(test_support.TESTFN): - os.remove(test_support.TESTFN) - -def test_main(): - test_support.run_unittest(HotShotTestCase) - - -if __name__ == "__main__": - test_main() Modified: python/branches/py3k-importlib/Lib/test/test_io.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_io.py (original) +++ python/branches/py3k-importlib/Lib/test/test_io.py Thu Oct 11 23:44:13 2007 @@ -251,7 +251,7 @@ def test_array_writes(self): a = array.array('i', range(10)) - n = len(buffer(a)) + n = len(memoryview(a)) f = io.open(test_support.TESTFN, "wb", 0) self.assertEqual(f.write(a), n) f.close() Modified: python/branches/py3k-importlib/Lib/test/test_marshal.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_marshal.py (original) +++ python/branches/py3k-importlib/Lib/test/test_marshal.py Thu Oct 11 23:44:13 2007 @@ -98,9 +98,9 @@ for s in ["", "Andr\xe8 Previn", "abc", " "*10000]: self.helper(s) - def test_buffer(self): + def test_bytes(self): for s in [b"", b"Andr\xe8 Previn", b"abc", b" "*10000]: - self.helper(buffer(s)) + self.helper(s) class ExceptionTestCase(unittest.TestCase): def test_exceptions(self): Modified: python/branches/py3k-importlib/Lib/test/test_multibytecodec_support.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_multibytecodec_support.py (original) +++ python/branches/py3k-importlib/Lib/test/test_multibytecodec_support.py Thu Oct 11 23:44:13 2007 @@ -31,8 +31,14 @@ self.incrementaldecoder = self.codec.incrementaldecoder def test_chunkcoding(self): - for native, utf8 in zip(*[map(bytes, str8(f).splitlines(1)) - for f in self.tstring]): + tstring_lines = [] + for b in self.tstring: + lines = b.split(b"\n") + last = lines.pop() + assert last == b"" + lines = [line + b"\n" for line in lines] + tstring_lines.append(lines) + for native, utf8 in zip(*tstring_lines): u = self.decode(native)[0] self.assertEqual(u, utf8.decode('utf-8')) if self.roundtriptest: Modified: python/branches/py3k-importlib/Lib/test/test_repr.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_repr.py (original) +++ python/branches/py3k-importlib/Lib/test/test_repr.py Thu Oct 11 23:44:13 2007 @@ -163,12 +163,6 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") - def test_buffer(self): - # XXX doesn't test buffers with no b_base or read-write buffers (see - # bufferobject.c). The test is fairly incomplete too. Sigh. - x = buffer('foo') - self.failUnless(repr(x).startswith('I', data) vereq(value, 0x12345678) -# Test methods to pack and unpack from buffers rather than strings. +# Test methods to pack and unpack from memoryviews rather than strings. test_unpack_from() test_pack_into() test_pack_into_fn() -test_unpack_with_buffer() +test_unpack_with_memoryview() def test_bool(): for prefix in tuple("<>!=")+('',): Modified: python/branches/py3k-importlib/Lib/test/test_types.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_types.py (original) +++ python/branches/py3k-importlib/Lib/test/test_types.py Thu Oct 11 23:44:13 2007 @@ -203,54 +203,6 @@ self.assertRaises(TypeError, type, 1, 2) self.assertRaises(TypeError, type, 1, 2, 3, 4) - def test_buffers(self): - self.assertRaises(ValueError, buffer, 'asdf', -1) - self.assertRaises(TypeError, buffer, None) - - a = buffer(b'asdf') - hash(a) - b = a * 5 - if a == b: - self.fail('buffers should not be equal') - if str(b) != ('asdf' * 5): - self.fail('repeated buffer has wrong content') - if str(a * 0) != '': - self.fail('repeated buffer zero times has wrong content') - if str(a + buffer(b'def')) != 'asdfdef': - self.fail('concatenation of buffers yields wrong content') - if str(buffer(a)) != 'asdf': - self.fail('composing buffers failed') - if str(buffer(a, 2)) != 'df': - self.fail('specifying buffer offset failed') - if str(buffer(a, 0, 2)) != 'as': - self.fail('specifying buffer size failed') - if str(buffer(a, 1, 2)) != 'sd': - self.fail('specifying buffer offset and size failed') - self.assertRaises(ValueError, buffer, buffer(b'asdf', 1), -1) - if str(buffer(buffer(b'asdf', 0, 2), 0)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 0, 5000)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 0, -1)) != 'as': - self.fail('composing length-specified buffer failed') - if str(buffer(buffer(b'asdf', 0, 2), 1, 2)) != 's': - self.fail('composing length-specified buffer failed') - - try: a[1] = 'g' - except TypeError: pass - else: self.fail("buffer assignment should raise TypeError") - - try: a[0:1] = 'g' - except TypeError: pass - else: self.fail("buffer slice assignment should raise TypeError") - - # array.array() returns an object that does not implement a char buffer, - # something which int() uses for conversion. - import array - try: int(buffer(array.array('b'))) - except TypeError: pass - else: self.fail("char buffer (at C level) not working") - def test_main(): run_unittest(TypesTests) Modified: python/branches/py3k-importlib/Lib/test/test_unicode.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_unicode.py (original) +++ python/branches/py3k-importlib/Lib/test/test_unicode.py Thu Oct 11 23:44:13 2007 @@ -713,7 +713,7 @@ if not sys.platform.startswith('java'): self.assertEqual( str( - buffer(b'character buffers are decoded to unicode'), + memoryview(b'character buffers are decoded to unicode'), 'utf-8', 'strict' ), Modified: python/branches/py3k-importlib/Lib/test/test_xml_etree.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_xml_etree.py (original) +++ python/branches/py3k-importlib/Lib/test/test_xml_etree.py Thu Oct 11 23:44:13 2007 @@ -183,9 +183,8 @@ text >>> print(ET.tostring(element)) text - >>> print(ET.tostring(element, "ascii")) - - text + >>> print(repr(ET.tostring(element, "ascii"))) + b'\ntext' >>> _, ids = ET.XMLID("text") >>> len(ids) 0 Modified: python/branches/py3k-importlib/Lib/test/test_xml_etree_c.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_xml_etree_c.py (original) +++ python/branches/py3k-importlib/Lib/test/test_xml_etree_c.py Thu Oct 11 23:44:13 2007 @@ -175,9 +175,8 @@ text >>> print(ET.tostring(element)) text - >>> print(ET.tostring(element, "ascii")) - - text + >>> print(repr(ET.tostring(element, "ascii"))) + b'\ntext' >>> _, ids = ET.XMLID("text") >>> len(ids) 0 Modified: python/branches/py3k-importlib/Lib/types.py ============================================================================== --- python/branches/py3k-importlib/Lib/types.py (original) +++ python/branches/py3k-importlib/Lib/types.py Thu Oct 11 23:44:13 2007 @@ -22,8 +22,6 @@ except NameError: pass -BufferType = buffer - TupleType = tuple ListType = list DictType = DictionaryType = dict Modified: python/branches/py3k-importlib/Makefile.pre.in ============================================================================== --- python/branches/py3k-importlib/Makefile.pre.in (original) +++ python/branches/py3k-importlib/Makefile.pre.in Thu Oct 11 23:44:13 2007 @@ -285,7 +285,6 @@ OBJECT_OBJS= \ Objects/abstract.o \ Objects/boolobject.o \ - Objects/bufferobject.o \ Objects/bytesobject.o \ Objects/cellobject.o \ Objects/classobject.o \ @@ -530,7 +529,6 @@ Include/asdl.h \ Include/abstract.h \ Include/boolobject.h \ - Include/bufferobject.h \ Include/bytesobject.h \ Include/ceval.h \ Include/classobject.h \ @@ -739,7 +737,7 @@ PLATMACPATH=:plat-mac:plat-mac/lib-scriptpackages LIBSUBDIRS= lib-tk site-packages test test/output test/data \ test/decimaltestdata \ - encodings hotshot \ + encodings \ email email/mime email/test email/test/data \ sqlite3 sqlite3/test \ logging bsddb bsddb/test csv wsgiref \ Modified: python/branches/py3k-importlib/Misc/NEWS ============================================================================== --- python/branches/py3k-importlib/Misc/NEWS (original) +++ python/branches/py3k-importlib/Misc/NEWS Thu Oct 11 23:44:13 2007 @@ -4,6 +4,16 @@ (editors: check NEWS.help for information about editing NEWS using ReST.) +What's New in Python 3.0a2? + +*Unreleased* + +Extension Modules +----------------- + +- The `hotshot` profiler has been removed; use `cProfile` instead. + + What's New in Python 3.0a1? ========================== Modified: python/branches/py3k-importlib/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/py3k-importlib/Modules/_ctypes/_ctypes.c (original) +++ python/branches/py3k-importlib/Modules/_ctypes/_ctypes.c Thu Oct 11 23:44:13 2007 @@ -739,18 +739,12 @@ { char *ptr; Py_ssize_t size; - int rel = 0; Py_buffer view; - if (PyBuffer_Check(value)) { - if (PyObject_GetBuffer(value, &view, PyBUF_SIMPLE) < 0) - return -1; - size = view.len; - ptr = view.buf; - rel = 1; - } else if (-1 == PyString_AsStringAndSize(value, &ptr, &size)) { + if (PyObject_GetBuffer(value, &view, PyBUF_SIMPLE) < 0) return -1; - } + size = view.len; + ptr = view.buf; if (size > self->b_size) { PyErr_SetString(PyExc_ValueError, "string too long"); @@ -759,12 +753,10 @@ memcpy(self->b_ptr, ptr, size); - if (rel) - PyObject_ReleaseBuffer(value, &view); + PyObject_ReleaseBuffer(value, &view); return 0; fail: - if (rel) - PyObject_ReleaseBuffer(value, &view); + PyObject_ReleaseBuffer(value, &view); return -1; } Deleted: /python/branches/py3k-importlib/Modules/_hotshot.c ============================================================================== --- /python/branches/py3k-importlib/Modules/_hotshot.c Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,1645 +0,0 @@ -/* - * This is the High Performance Python Profiler portion of HotShot. - */ - -#include "Python.h" -#include "code.h" -#include "eval.h" -#include "frameobject.h" -#include "structmember.h" - -/* - * Which timer to use should be made more configurable, but that should not - * be difficult. This will do for now. - */ -#ifdef MS_WINDOWS -#include - -#ifdef HAVE_DIRECT_H -#include /* for getcwd() */ -#endif - -typedef __int64 hs_time; -#define GETTIMEOFDAY(P_HS_TIME) \ - { LARGE_INTEGER _temp; \ - QueryPerformanceCounter(&_temp); \ - *(P_HS_TIME) = _temp.QuadPart; } - - -#else -#ifndef HAVE_GETTIMEOFDAY -#error "This module requires gettimeofday() on non-Windows platforms!" -#endif -#if (defined(PYOS_OS2) && defined(PYCC_GCC)) || defined(__QNX__) -#include -#else -#include -#include -#endif -typedef struct timeval hs_time; -#endif - -#if !defined(__cplusplus) && !defined(inline) -#ifdef __GNUC__ -#define inline __inline -#endif -#endif - -#ifndef inline -#define inline -#endif - -#define BUFFERSIZE 10240 - -#if defined(PYOS_OS2) && defined(PYCC_GCC) -#define PATH_MAX 260 -#endif - -#if defined(__sgi) && _COMPILER_VERSION>700 && !defined(PATH_MAX) -/* fix PATH_MAX not being defined with MIPSPro 7.x - if mode is ANSI C (default) */ -#define PATH_MAX 1024 -#endif - -#ifndef PATH_MAX -# ifdef MAX_PATH -# define PATH_MAX MAX_PATH -# elif defined (_POSIX_PATH_MAX) -# define PATH_MAX _POSIX_PATH_MAX -# else -# error "Need a defn. for PATH_MAX in _hotshot.c" -# endif -#endif - -typedef struct { - PyObject_HEAD - PyObject *filemap; - PyObject *logfilename; - Py_ssize_t index; - unsigned char buffer[BUFFERSIZE]; - FILE *logfp; - int lineevents; - int linetimings; - int frametimings; - /* size_t filled; */ - int active; - int next_fileno; - hs_time prev_timeofday; -} ProfilerObject; - -typedef struct { - PyObject_HEAD - PyObject *info; - FILE *logfp; - int linetimings; - int frametimings; -} LogReaderObject; - -static PyObject * ProfilerError = NULL; - - -#ifndef MS_WINDOWS -#ifdef GETTIMEOFDAY_NO_TZ -#define GETTIMEOFDAY(ptv) gettimeofday((ptv)) -#else -#define GETTIMEOFDAY(ptv) gettimeofday((ptv), (struct timezone *)NULL) -#endif -#endif - - -/* The log reader... */ - -PyDoc_STRVAR(logreader_close__doc__, -"close()\n" -"Close the log file, preventing additional records from being read."); - -static PyObject * -logreader_close(LogReaderObject *self, PyObject *args) -{ - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_INCREF(Py_None); - - return Py_None; -} - -PyDoc_STRVAR(logreader_fileno__doc__, -"fileno() -> file descriptor\n" -"Returns the file descriptor for the log file, if open.\n" -"Raises ValueError if the log file is closed."); - -static PyObject * -logreader_fileno(LogReaderObject *self) -{ - if (self->logfp == NULL) { - PyErr_SetString(PyExc_ValueError, - "logreader's file object already closed"); - return NULL; - } - return PyInt_FromLong(fileno(self->logfp)); -} - - -/* Log File Format - * --------------- - * - * The log file consists of a sequence of variable-length records. - * Each record is identified with a record type identifier in two - * bits of the first byte. The two bits are the "least significant" - * bits of the byte. - * - * Low bits: Opcode: Meaning: - * 0x00 ENTER enter a frame - * 0x01 EXIT exit a frame - * 0x02 LINENO execution moved onto a different line - * 0x03 OTHER more bits are needed to deecode - * - * If the type is OTHER, the record is not packed so tightly, and the - * remaining bits are used to disambiguate the record type. These - * records are not used as frequently so compaction is not an issue. - * Each of the first three record types has a highly tailored - * structure that allows it to be packed tightly. - * - * The OTHER records have the following identifiers: - * - * First byte: Opcode: Meaning: - * 0x13 ADD_INFO define a key/value pair - * 0x23 DEFINE_FILE define an int->filename mapping - * 0x33 LINE_TIMES indicates if LINENO events have tdeltas - * 0x43 DEFINE_FUNC define a (fileno,lineno)->funcname mapping - * 0x53 FRAME_TIMES indicates if ENTER/EXIT events have tdeltas - * - * Packed Integers - * - * "Packed integers" are non-negative integer values encoded as a - * sequence of bytes. Each byte is encoded such that the most - * significant bit is set if the next byte is also part of the - * integer. Each byte provides bits to the least-significant end of - * the result; the accumulated value must be shifted up to place the - * new bits into the result. - * - * "Modified packed integers" are packed integers where only a portion - * of the first byte is used. In the rest of the specification, these - * are referred to as "MPI(n,name)", where "n" is the number of bits - * discarded from the least-signicant positions of the byte, and - * "name" is a name being given to those "discarded" bits, since they - * are a field themselves. - * - * ENTER records: - * - * MPI(2,type) fileno -- type is 0x00 - * PI lineno - * PI tdelta -- iff frame times are enabled - * - * EXIT records - * - * MPI(2,type) tdelta -- type is 0x01; tdelta will be 0 - * if frame times are disabled - * - * LINENO records - * - * MPI(2,type) lineno -- type is 0x02 - * PI tdelta -- iff LINENO includes it - * - * ADD_INFO records - * - * BYTE type -- always 0x13 - * PI len1 -- length of first string - * BYTE string1[len1] -- len1 bytes of string data - * PI len2 -- length of second string - * BYTE string2[len2] -- len2 bytes of string data - * - * DEFINE_FILE records - * - * BYTE type -- always 0x23 - * PI fileno - * PI len -- length of filename - * BYTE filename[len] -- len bytes of string data - * - * DEFINE_FUNC records - * - * BYTE type -- always 0x43 - * PI fileno - * PI lineno - * PI len -- length of funcname - * BYTE funcname[len] -- len bytes of string data - * - * LINE_TIMES records - * - * This record can be used only before the start of ENTER/EXIT/LINENO - * records. If have_tdelta is true, LINENO records will include the - * tdelta field, otherwise it will be omitted. If this record is not - * given, LINENO records will not contain the tdelta field. - * - * BYTE type -- always 0x33 - * BYTE have_tdelta -- 0 if LINENO does *not* have - * timing information - * FRAME_TIMES records - * - * This record can be used only before the start of ENTER/EXIT/LINENO - * records. If have_tdelta is true, ENTER and EXIT records will - * include the tdelta field, otherwise it will be omitted. If this - * record is not given, ENTER and EXIT records will contain the tdelta - * field. - * - * BYTE type -- always 0x53 - * BYTE have_tdelta -- 0 if ENTER/EXIT do *not* have - * timing information - */ - -#define WHAT_ENTER 0x00 -#define WHAT_EXIT 0x01 -#define WHAT_LINENO 0x02 -#define WHAT_OTHER 0x03 /* only used in decoding */ -#define WHAT_ADD_INFO 0x13 -#define WHAT_DEFINE_FILE 0x23 -#define WHAT_LINE_TIMES 0x33 -#define WHAT_DEFINE_FUNC 0x43 -#define WHAT_FRAME_TIMES 0x53 - -#define ERR_NONE 0 -#define ERR_EOF -1 -#define ERR_EXCEPTION -2 -#define ERR_BAD_RECTYPE -3 - -#define PISIZE (sizeof(int) + 1) -#define MPISIZE (PISIZE + 1) - -/* Maximum size of "normal" events -- nothing that contains string data */ -#define MAXEVENTSIZE (MPISIZE + PISIZE*2) - - -/* Unpack a packed integer; if "discard" is non-zero, unpack a modified - * packed integer with "discard" discarded bits. - */ -static int -unpack_packed_int(LogReaderObject *self, int *pvalue, int discard) -{ - int c; - int accum = 0; - int bits = 0; - int cont; - - do { - /* read byte */ - if ((c = fgetc(self->logfp)) == EOF) - return ERR_EOF; - accum |= ((c & 0x7F) >> discard) << bits; - bits += (7 - discard); - cont = c & 0x80; - discard = 0; - } while (cont); - - *pvalue = accum; - - return 0; -} - -/* Unpack a string, which is encoded as a packed integer giving the - * length of the string, followed by the string data. - */ -static int -unpack_string(LogReaderObject *self, PyObject **pvalue) -{ - int i; - int len; - int err; - int ch; - char *buf; - - if ((err = unpack_packed_int(self, &len, 0))) - return err; - - buf = (char *)malloc(len); - if (!buf) { - PyErr_NoMemory(); - return ERR_EXCEPTION; - } - - for (i=0; i < len; i++) { - ch = fgetc(self->logfp); - buf[i] = ch; - if (ch == EOF) { - free(buf); - return ERR_EOF; - } - } - *pvalue = PyString_FromStringAndSize(buf, len); - free(buf); - if (*pvalue == NULL) { - return ERR_EXCEPTION; - } - return 0; -} - - -static int -unpack_add_info(LogReaderObject *self) -{ - PyObject *key; - PyObject *value = NULL; - int err; - - err = unpack_string(self, &key); - if (!err) { - err = unpack_string(self, &value); - if (err) - Py_DECREF(key); - else { - PyObject *list = PyDict_GetItem(self->info, key); - if (list == NULL) { - list = PyList_New(0); - if (list == NULL) { - err = ERR_EXCEPTION; - goto finally; - } - if (PyDict_SetItem(self->info, key, list)) { - Py_DECREF(list); - err = ERR_EXCEPTION; - goto finally; - } - Py_DECREF(list); - } - if (PyList_Append(list, value)) - err = ERR_EXCEPTION; - } - } - finally: - Py_XDECREF(key); - Py_XDECREF(value); - return err; -} - - -static void -eof_error(LogReaderObject *self) -{ - fclose(self->logfp); - self->logfp = NULL; - PyErr_SetString(PyExc_EOFError, - "end of file with incomplete profile record"); -} - -static PyObject * -logreader_tp_iternext(LogReaderObject *self) -{ - int c; - int what; - int err = ERR_NONE; - int lineno = -1; - int fileno = -1; - int tdelta = -1; - PyObject *s1 = NULL, *s2 = NULL; - PyObject *result = NULL; -#if 0 - unsigned char b0, b1; -#endif - - if (self->logfp == NULL) { - PyErr_SetString(ProfilerError, - "cannot iterate over closed LogReader object"); - return NULL; - } - -restart: - /* decode the record type */ - if ((c = fgetc(self->logfp)) == EOF) { - fclose(self->logfp); - self->logfp = NULL; - return NULL; - } - what = c & WHAT_OTHER; - if (what == WHAT_OTHER) - what = c; /* need all the bits for type */ - else - ungetc(c, self->logfp); /* type byte includes packed int */ - - switch (what) { - case WHAT_ENTER: - err = unpack_packed_int(self, &fileno, 2); - if (!err) { - err = unpack_packed_int(self, &lineno, 0); - if (self->frametimings && !err) - err = unpack_packed_int(self, &tdelta, 0); - } - break; - case WHAT_EXIT: - err = unpack_packed_int(self, &tdelta, 2); - break; - case WHAT_LINENO: - err = unpack_packed_int(self, &lineno, 2); - if (self->linetimings && !err) - err = unpack_packed_int(self, &tdelta, 0); - break; - case WHAT_ADD_INFO: - err = unpack_add_info(self); - break; - case WHAT_DEFINE_FILE: - err = unpack_packed_int(self, &fileno, 0); - if (!err) { - err = unpack_string(self, &s1); - if (!err) { - Py_INCREF(Py_None); - s2 = Py_None; - } - } - break; - case WHAT_DEFINE_FUNC: - err = unpack_packed_int(self, &fileno, 0); - if (!err) { - err = unpack_packed_int(self, &lineno, 0); - if (!err) - err = unpack_string(self, &s1); - } - break; - case WHAT_LINE_TIMES: - if ((c = fgetc(self->logfp)) == EOF) - err = ERR_EOF; - else { - self->linetimings = c ? 1 : 0; - goto restart; - } - break; - case WHAT_FRAME_TIMES: - if ((c = fgetc(self->logfp)) == EOF) - err = ERR_EOF; - else { - self->frametimings = c ? 1 : 0; - goto restart; - } - break; - default: - err = ERR_BAD_RECTYPE; - } - if (err == ERR_BAD_RECTYPE) { - PyErr_SetString(PyExc_ValueError, - "unknown record type in log file"); - } - else if (err == ERR_EOF) { - eof_error(self); - } - else if (!err) { - result = PyTuple_New(4); - if (result == NULL) - return NULL; - PyTuple_SET_ITEM(result, 0, PyInt_FromLong(what)); - PyTuple_SET_ITEM(result, 2, PyInt_FromLong(fileno)); - if (s1 == NULL) - PyTuple_SET_ITEM(result, 1, PyInt_FromLong(tdelta)); - else - PyTuple_SET_ITEM(result, 1, s1); - if (s2 == NULL) - PyTuple_SET_ITEM(result, 3, PyInt_FromLong(lineno)); - else - PyTuple_SET_ITEM(result, 3, s2); - } - /* The only other case is err == ERR_EXCEPTION, in which case the - * exception is already set. - */ -#if 0 - b0 = self->buffer[self->index]; - b1 = self->buffer[self->index + 1]; - if (b0 & 1) { - /* This is a line-number event. */ - what = PyTrace_LINE; - lineno = ((b0 & ~1) << 7) + b1; - self->index += 2; - } - else { - what = (b0 & 0x0E) >> 1; - tdelta = ((b0 & 0xF0) << 4) + b1; - if (what == PyTrace_CALL) { - /* we know there's a 2-byte file ID & 2-byte line number */ - fileno = ((self->buffer[self->index + 2] << 8) - + self->buffer[self->index + 3]); - lineno = ((self->buffer[self->index + 4] << 8) - + self->buffer[self->index + 5]); - self->index += 6; - } - else - self->index += 2; - } -#endif - return result; -} - -static void -logreader_dealloc(LogReaderObject *self) -{ - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_XDECREF(self->info); - PyObject_Del(self); -} - -static PyObject * -logreader_sq_item(LogReaderObject *self, Py_ssize_t index) -{ - PyObject *result = logreader_tp_iternext(self); - if (result == NULL && !PyErr_Occurred()) { - PyErr_SetString(PyExc_IndexError, "no more events in log"); - return NULL; - } - return result; -} - -static void -do_stop(ProfilerObject *self); - -static int -flush_data(ProfilerObject *self) -{ - /* Need to dump data to the log file... */ - size_t written = fwrite(self->buffer, 1, self->index, self->logfp); - if (written == (size_t)self->index) - self->index = 0; - else { - memmove(self->buffer, &self->buffer[written], - self->index - written); - self->index -= written; - if (written == 0) { - char *s = PyString_AsString(self->logfilename); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, s); - do_stop(self); - return -1; - } - } - if (written > 0) { - if (fflush(self->logfp)) { - char *s = PyString_AsString(self->logfilename); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, s); - do_stop(self); - return -1; - } - } - return 0; -} - -static inline int -pack_packed_int(ProfilerObject *self, int value) -{ - unsigned char partial; - - do { - partial = value & 0x7F; - value >>= 7; - if (value) - partial |= 0x80; - self->buffer[self->index] = partial; - self->index++; - } while (value); - return 0; -} - -/* Encode a modified packed integer, with a subfield of modsize bits - * containing the value "subfield". The value of subfield is not - * checked to ensure it actually fits in modsize bits. - */ -static inline int -pack_modified_packed_int(ProfilerObject *self, int value, - int modsize, int subfield) -{ - const int maxvalues[] = {-1, 1, 3, 7, 15, 31, 63, 127}; - - int bits = 7 - modsize; - int partial = value & maxvalues[bits]; - unsigned char b = subfield | (partial << modsize); - - if (partial != value) { - b |= 0x80; - self->buffer[self->index] = b; - self->index++; - return pack_packed_int(self, value >> bits); - } - self->buffer[self->index] = b; - self->index++; - return 0; -} - -static int -pack_string(ProfilerObject *self, const char *s, Py_ssize_t len) -{ - if (len + PISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - assert(len < INT_MAX); - if (pack_packed_int(self, (int)len) < 0) - return -1; - memcpy(self->buffer + self->index, s, len); - self->index += len; - return 0; -} - -static int -pack_add_info(ProfilerObject *self, const char *s1, const char *s2) -{ - Py_ssize_t len1 = strlen(s1); - Py_ssize_t len2 = strlen(s2); - - if (len1 + len2 + PISIZE*2 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_ADD_INFO; - self->index++; - if (pack_string(self, s1, len1) < 0) - return -1; - return pack_string(self, s2, len2); -} - -static int -pack_define_file(ProfilerObject *self, int fileno, const char *filename) -{ - Py_ssize_t len = strlen(filename); - - if (len + PISIZE*2 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_DEFINE_FILE; - self->index++; - if (pack_packed_int(self, fileno) < 0) - return -1; - return pack_string(self, filename, len); -} - -static int -pack_define_func(ProfilerObject *self, int fileno, int lineno, - const char *funcname) -{ - Py_ssize_t len = strlen(funcname); - - if (len + PISIZE*3 + 1 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_DEFINE_FUNC; - self->index++; - if (pack_packed_int(self, fileno) < 0) - return -1; - if (pack_packed_int(self, lineno) < 0) - return -1; - return pack_string(self, funcname, len); -} - -static int -pack_line_times(ProfilerObject *self) -{ - if (2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_LINE_TIMES; - self->buffer[self->index + 1] = self->linetimings ? 1 : 0; - self->index += 2; - return 0; -} - -static int -pack_frame_times(ProfilerObject *self) -{ - if (2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - self->buffer[self->index] = WHAT_FRAME_TIMES; - self->buffer[self->index + 1] = self->frametimings ? 1 : 0; - self->index += 2; - return 0; -} - -static inline int -pack_enter(ProfilerObject *self, int fileno, int tdelta, int lineno) -{ - if (MPISIZE + PISIZE*2 + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - pack_modified_packed_int(self, fileno, 2, WHAT_ENTER); - pack_packed_int(self, lineno); - if (self->frametimings) - return pack_packed_int(self, tdelta); - else - return 0; -} - -static inline int -pack_exit(ProfilerObject *self, int tdelta) -{ - if (MPISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - if (self->frametimings) - return pack_modified_packed_int(self, tdelta, 2, WHAT_EXIT); - self->buffer[self->index] = WHAT_EXIT; - self->index++; - return 0; -} - -static inline int -pack_lineno(ProfilerObject *self, int lineno) -{ - if (MPISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return -1; - } - return pack_modified_packed_int(self, lineno, 2, WHAT_LINENO); -} - -static inline int -pack_lineno_tdelta(ProfilerObject *self, int lineno, int tdelta) -{ - if (MPISIZE + PISIZE + self->index >= BUFFERSIZE) { - if (flush_data(self) < 0) - return 0; - } - if (pack_modified_packed_int(self, lineno, 2, WHAT_LINENO) < 0) - return -1; - return pack_packed_int(self, tdelta); -} - -static inline int -get_fileno(ProfilerObject *self, PyCodeObject *fcode) -{ - /* This is only used for ENTER events. */ - - PyObject *obj; - PyObject *dict; - int fileno; - - obj = PyDict_GetItem(self->filemap, fcode->co_filename); - if (obj == NULL) { - /* first sighting of this file */ - dict = PyDict_New(); - if (dict == NULL) { - return -1; - } - fileno = self->next_fileno; - obj = Py_BuildValue("iN", fileno, dict); - if (obj == NULL) { - return -1; - } - if (PyDict_SetItem(self->filemap, fcode->co_filename, obj)) { - Py_DECREF(obj); - return -1; - } - self->next_fileno++; - Py_DECREF(obj); - if (pack_define_file(self, fileno, - PyString_AS_STRING(fcode->co_filename)) < 0) - return -1; - } - else { - /* already know this ID */ - fileno = PyInt_AS_LONG(PyTuple_GET_ITEM(obj, 0)); - dict = PyTuple_GET_ITEM(obj, 1); - } - /* make sure we save a function name for this (fileno, lineno) */ - obj = PyInt_FromLong(fcode->co_firstlineno); - if (obj == NULL) { - /* We just won't have it saved; too bad. */ - PyErr_Clear(); - } - else { - PyObject *name = PyDict_GetItem(dict, obj); - if (name == NULL) { - if (pack_define_func(self, fileno, fcode->co_firstlineno, - PyUnicode_AsString(fcode->co_name)) < 0) { - Py_DECREF(obj); - return -1; - } - if (PyDict_SetItem(dict, obj, fcode->co_name)) { - Py_DECREF(obj); - return -1; - } - } - Py_DECREF(obj); - } - return fileno; -} - -static inline int -get_tdelta(ProfilerObject *self) -{ - int tdelta; -#ifdef MS_WINDOWS - hs_time tv; - hs_time diff; - - GETTIMEOFDAY(&tv); - diff = tv - self->prev_timeofday; - tdelta = (int)diff; -#else - struct timeval tv; - - GETTIMEOFDAY(&tv); - - tdelta = tv.tv_usec - self->prev_timeofday.tv_usec; - if (tv.tv_sec != self->prev_timeofday.tv_sec) - tdelta += (tv.tv_sec - self->prev_timeofday.tv_sec) * 1000000; -#endif - /* time can go backwards on some multiprocessor systems or by NTP */ - if (tdelta < 0) - return 0; - - self->prev_timeofday = tv; - return tdelta; -} - - -/* The workhorse: the profiler callback function. */ - -static int -tracer_callback(ProfilerObject *self, PyFrameObject *frame, int what, - PyObject *arg) -{ - int fileno; - - switch (what) { - case PyTrace_CALL: - fileno = get_fileno(self, frame->f_code); - if (fileno < 0) - return -1; - return pack_enter(self, fileno, - self->frametimings ? get_tdelta(self) : -1, - frame->f_code->co_firstlineno); - - case PyTrace_RETURN: - return pack_exit(self, get_tdelta(self)); - - case PyTrace_LINE: /* we only get these events if we asked for them */ - if (self->linetimings) - return pack_lineno_tdelta(self, frame->f_lineno, - get_tdelta(self)); - else - return pack_lineno(self, frame->f_lineno); - - default: - /* ignore PyTrace_EXCEPTION */ - break; - } - return 0; -} - - -/* A couple of useful helper functions. */ - -#ifdef MS_WINDOWS -static LARGE_INTEGER frequency = {0, 0}; -#endif - -static unsigned long timeofday_diff = 0; -static unsigned long rusage_diff = 0; - -static void -calibrate(void) -{ - hs_time tv1, tv2; - -#ifdef MS_WINDOWS - hs_time diff; - QueryPerformanceFrequency(&frequency); -#endif - - GETTIMEOFDAY(&tv1); - while (1) { - GETTIMEOFDAY(&tv2); -#ifdef MS_WINDOWS - diff = tv2 - tv1; - if (diff != 0) { - timeofday_diff = (unsigned long)diff; - break; - } -#else - if (tv1.tv_sec != tv2.tv_sec || tv1.tv_usec != tv2.tv_usec) { - if (tv1.tv_sec == tv2.tv_sec) - timeofday_diff = tv2.tv_usec - tv1.tv_usec; - else - timeofday_diff = (1000000 - tv1.tv_usec) + tv2.tv_usec; - break; - } -#endif - } -#if defined(MS_WINDOWS) || defined(PYOS_OS2) || \ - defined(__VMS) || defined (__QNX__) - rusage_diff = -1; -#else - { - struct rusage ru1, ru2; - - getrusage(RUSAGE_SELF, &ru1); - while (1) { - getrusage(RUSAGE_SELF, &ru2); - if (ru1.ru_utime.tv_sec != ru2.ru_utime.tv_sec) { - rusage_diff = ((1000000 - ru1.ru_utime.tv_usec) - + ru2.ru_utime.tv_usec); - break; - } - else if (ru1.ru_utime.tv_usec != ru2.ru_utime.tv_usec) { - rusage_diff = ru2.ru_utime.tv_usec - ru1.ru_utime.tv_usec; - break; - } - else if (ru1.ru_stime.tv_sec != ru2.ru_stime.tv_sec) { - rusage_diff = ((1000000 - ru1.ru_stime.tv_usec) - + ru2.ru_stime.tv_usec); - break; - } - else if (ru1.ru_stime.tv_usec != ru2.ru_stime.tv_usec) { - rusage_diff = ru2.ru_stime.tv_usec - ru1.ru_stime.tv_usec; - break; - } - } - } -#endif -} - -static void -do_start(ProfilerObject *self) -{ - self->active = 1; - GETTIMEOFDAY(&self->prev_timeofday); - if (self->lineevents) - PyEval_SetTrace((Py_tracefunc) tracer_callback, (PyObject *)self); - else - PyEval_SetProfile((Py_tracefunc) tracer_callback, (PyObject *)self); -} - -static void -do_stop(ProfilerObject *self) -{ - if (self->active) { - self->active = 0; - if (self->lineevents) - PyEval_SetTrace(NULL, NULL); - else - PyEval_SetProfile(NULL, NULL); - } - if (self->index > 0) { - /* Best effort to dump out any remaining data. */ - flush_data(self); - } -} - -static int -is_available(ProfilerObject *self) -{ - if (self->active) { - PyErr_SetString(ProfilerError, "profiler already active"); - return 0; - } - if (self->logfp == NULL) { - PyErr_SetString(ProfilerError, "profiler already closed"); - return 0; - } - return 1; -} - - -/* Profiler object interface methods. */ - -PyDoc_STRVAR(addinfo__doc__, -"addinfo(key, value)\n" -"Insert an ADD_INFO record into the log."); - -static PyObject * -profiler_addinfo(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - char *key, *value; - - if (PyArg_ParseTuple(args, "ss:addinfo", &key, &value)) { - if (self->logfp == NULL) - PyErr_SetString(ProfilerError, "profiler already closed"); - else { - if (pack_add_info(self, key, value) == 0) { - result = Py_None; - Py_INCREF(result); - } - } - } - return result; -} - -PyDoc_STRVAR(close__doc__, -"close()\n" -"Shut down this profiler and close the log files, even if its active."); - -static PyObject * -profiler_close(ProfilerObject *self) -{ - do_stop(self); - if (self->logfp != NULL) { - fclose(self->logfp); - self->logfp = NULL; - } - Py_INCREF(Py_None); - return Py_None; -} - -#define fileno__doc__ logreader_fileno__doc__ - -static PyObject * -profiler_fileno(ProfilerObject *self) -{ - if (self->logfp == NULL) { - PyErr_SetString(PyExc_ValueError, - "profiler's file object already closed"); - return NULL; - } - return PyInt_FromLong(fileno(self->logfp)); -} - -PyDoc_STRVAR(runcall__doc__, -"runcall(callable[, args[, kw]]) -> callable()\n" -"Profile a specific function call, returning the result of that call."); - -static PyObject * -profiler_runcall(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - PyObject *callargs = NULL; - PyObject *callkw = NULL; - PyObject *callable; - - if (PyArg_UnpackTuple(args, "runcall", 1, 3, - &callable, &callargs, &callkw)) { - if (is_available(self)) { - do_start(self); - result = PyEval_CallObjectWithKeywords(callable, callargs, callkw); - do_stop(self); - } - } - return result; -} - -PyDoc_STRVAR(runcode__doc__, -"runcode(code, globals[, locals])\n" -"Execute a code object while collecting profile data. If locals is\n" -"omitted, globals is used for the locals as well."); - -static PyObject * -profiler_runcode(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - PyCodeObject *code; - PyObject *globals; - PyObject *locals = NULL; - - if (PyArg_ParseTuple(args, "O!O!|O:runcode", - &PyCode_Type, &code, - &PyDict_Type, &globals, - &locals)) { - if (is_available(self)) { - if (locals == NULL || locals == Py_None) - locals = globals; - else if (!PyDict_Check(locals)) { - PyErr_SetString(PyExc_TypeError, - "locals must be a dictionary or None"); - return NULL; - } - do_start(self); - result = PyEval_EvalCode(code, globals, locals); - do_stop(self); -#if 0 - if (!PyErr_Occurred()) { - result = Py_None; - Py_INCREF(result); - } -#endif - } - } - return result; -} - -PyDoc_STRVAR(start__doc__, -"start()\n" -"Install this profiler for the current thread."); - -static PyObject * -profiler_start(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - - if (is_available(self)) { - do_start(self); - result = Py_None; - Py_INCREF(result); - } - return result; -} - -PyDoc_STRVAR(stop__doc__, -"stop()\n" -"Remove this profiler from the current thread."); - -static PyObject * -profiler_stop(ProfilerObject *self, PyObject *args) -{ - PyObject *result = NULL; - - if (!self->active) - PyErr_SetString(ProfilerError, "profiler not active"); - else { - do_stop(self); - result = Py_None; - Py_INCREF(result); - } - return result; -} - - -/* Python API support. */ - -static void -profiler_dealloc(ProfilerObject *self) -{ - do_stop(self); - if (self->logfp != NULL) - fclose(self->logfp); - Py_XDECREF(self->filemap); - Py_XDECREF(self->logfilename); - PyObject_Del((PyObject *)self); -} - -static PyMethodDef profiler_methods[] = { - {"addinfo", (PyCFunction)profiler_addinfo, METH_VARARGS, addinfo__doc__}, - {"close", (PyCFunction)profiler_close, METH_NOARGS, close__doc__}, - {"fileno", (PyCFunction)profiler_fileno, METH_NOARGS, fileno__doc__}, - {"runcall", (PyCFunction)profiler_runcall, METH_VARARGS, runcall__doc__}, - {"runcode", (PyCFunction)profiler_runcode, METH_VARARGS, runcode__doc__}, - {"start", (PyCFunction)profiler_start, METH_NOARGS, start__doc__}, - {"stop", (PyCFunction)profiler_stop, METH_NOARGS, stop__doc__}, - {NULL, NULL} -}; - -static PyMemberDef profiler_members[] = { - {"frametimings", T_LONG, offsetof(ProfilerObject, linetimings), READONLY}, - {"lineevents", T_LONG, offsetof(ProfilerObject, lineevents), READONLY}, - {"linetimings", T_LONG, offsetof(ProfilerObject, linetimings), READONLY}, - {NULL} -}; - -static PyObject * -profiler_get_closed(ProfilerObject *self, void *closure) -{ - PyObject *result = (self->logfp == NULL) ? Py_True : Py_False; - Py_INCREF(result); - return result; -} - -static PyGetSetDef profiler_getsets[] = { - {"closed", (getter)profiler_get_closed, NULL, - PyDoc_STR("True if the profiler's output file has already been closed.")}, - {NULL} -}; - - -PyDoc_STRVAR(profiler_object__doc__, -"High-performance profiler object.\n" -"\n" -"Methods:\n" -"\n" -"close(): Stop the profiler and close the log files.\n" -"fileno(): Returns the file descriptor of the log file.\n" -"runcall(): Run a single function call with profiling enabled.\n" -"runcode(): Execute a code object with profiling enabled.\n" -"start(): Install the profiler and return.\n" -"stop(): Remove the profiler.\n" -"\n" -"Attributes (read-only):\n" -"\n" -"closed: True if the profiler has already been closed.\n" -"frametimings: True if ENTER/EXIT events collect timing information.\n" -"lineevents: True if line events are reported to the profiler.\n" -"linetimings: True if line events collect timing information."); - -static PyTypeObject ProfilerType = { - PyVarObject_HEAD_INIT(NULL, 0) - "_hotshot.ProfilerType", /* tp_name */ - (int) sizeof(ProfilerObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - (destructor)profiler_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - profiler_object__doc__, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - profiler_methods, /* tp_methods */ - profiler_members, /* tp_members */ - profiler_getsets, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ -}; - - -static PyMethodDef logreader_methods[] = { - {"close", (PyCFunction)logreader_close, METH_NOARGS, - logreader_close__doc__}, - {"fileno", (PyCFunction)logreader_fileno, METH_NOARGS, - logreader_fileno__doc__}, - {NULL, NULL} -}; - -static PyMemberDef logreader_members[] = { - {"info", T_OBJECT, offsetof(LogReaderObject, info), READONLY, - PyDoc_STR("Dictionary mapping informational keys to lists of values.")}, - {NULL} -}; - - -PyDoc_STRVAR(logreader__doc__, -"logreader(filename) --> log-iterator\n\ -Create a log-reader for the timing information file."); - -static PySequenceMethods logreader_as_sequence = { - 0, /* sq_length */ - 0, /* sq_concat */ - 0, /* sq_repeat */ - (ssizeargfunc)logreader_sq_item, /* sq_item */ - 0, /* sq_slice */ - 0, /* sq_ass_item */ - 0, /* sq_ass_slice */ - 0, /* sq_contains */ - 0, /* sq_inplace_concat */ - 0, /* sq_inplace_repeat */ -}; - -static PyObject * -logreader_get_closed(LogReaderObject *self, void *closure) -{ - PyObject *result = (self->logfp == NULL) ? Py_True : Py_False; - Py_INCREF(result); - return result; -} - -static PyGetSetDef logreader_getsets[] = { - {"closed", (getter)logreader_get_closed, NULL, - PyDoc_STR("True if the logreader's input file has already been closed.")}, - {NULL} -}; - -static PyTypeObject LogReaderType = { - PyVarObject_HEAD_INIT(NULL, 0) - "_hotshot.LogReaderType", /* tp_name */ - (int) sizeof(LogReaderObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - (destructor)logreader_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - &logreader_as_sequence, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - logreader__doc__, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - PyObject_SelfIter, /* tp_iter */ - (iternextfunc)logreader_tp_iternext,/* tp_iternext */ - logreader_methods, /* tp_methods */ - logreader_members, /* tp_members */ - logreader_getsets, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ -}; - -static PyObject * -hotshot_logreader(PyObject *unused, PyObject *args) -{ - LogReaderObject *self = NULL; - char *filename; - int c; - int err = 0; - - if (PyArg_ParseTuple(args, "s:logreader", &filename)) { - self = PyObject_New(LogReaderObject, &LogReaderType); - if (self != NULL) { - self->frametimings = 1; - self->linetimings = 0; - self->info = NULL; - self->logfp = fopen(filename, "rb"); - if (self->logfp == NULL) { - PyErr_SetFromErrnoWithFilename(PyExc_IOError, filename); - Py_DECREF(self); - self = NULL; - goto finally; - } - self->info = PyDict_New(); - if (self->info == NULL) { - Py_DECREF(self); - goto finally; - } - /* read initial info */ - for (;;) { - if ((c = fgetc(self->logfp)) == EOF) { - eof_error(self); - break; - } - if (c != WHAT_ADD_INFO) { - ungetc(c, self->logfp); - break; - } - err = unpack_add_info(self); - if (err) { - if (err == ERR_EOF) - eof_error(self); - else - PyErr_SetString(PyExc_RuntimeError, - "unexpected error"); - break; - } - } - } - } - finally: - return (PyObject *) self; -} - - -/* Return a Python string that represents the version number without the - * extra cruft added by revision control, even if the right options were - * given to the "cvs export" command to make it not include the extra - * cruft. - */ -static char * -get_version_string(void) -{ - static char *rcsid = "$Revision$"; - char *rev = rcsid; - char *buffer; - int i = 0; - - while (*rev && !isdigit(Py_CHARMASK(*rev))) - ++rev; - while (rev[i] != ' ' && rev[i] != '\0') - ++i; - buffer = (char *)malloc(i + 1); - if (buffer != NULL) { - memmove(buffer, rev, i); - buffer[i] = '\0'; - } - return buffer; -} - -/* Write out a RFC 822-style header with various useful bits of - * information to make the output easier to manage. - */ -static int -write_header(ProfilerObject *self) -{ - char *buffer; - char cwdbuffer[PATH_MAX]; - PyObject *temp; - Py_ssize_t i, len; - - buffer = get_version_string(); - if (buffer == NULL) { - PyErr_NoMemory(); - return -1; - } - pack_add_info(self, "hotshot-version", buffer); - pack_add_info(self, "requested-frame-timings", - (self->frametimings ? "yes" : "no")); - pack_add_info(self, "requested-line-events", - (self->lineevents ? "yes" : "no")); - pack_add_info(self, "requested-line-timings", - (self->linetimings ? "yes" : "no")); - pack_add_info(self, "platform", Py_GetPlatform()); - pack_add_info(self, "executable", Py_GetProgramFullPath()); - free(buffer); - buffer = (char *) Py_GetVersion(); - if (buffer == NULL) - PyErr_Clear(); - else - pack_add_info(self, "executable-version", buffer); - -#ifdef MS_WINDOWS - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%I64d", frequency.QuadPart); - pack_add_info(self, "reported-performance-frequency", cwdbuffer); -#else - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%lu", rusage_diff); - pack_add_info(self, "observed-interval-getrusage", cwdbuffer); - PyOS_snprintf(cwdbuffer, sizeof(cwdbuffer), "%lu", timeofday_diff); - pack_add_info(self, "observed-interval-gettimeofday", cwdbuffer); -#endif - - pack_add_info(self, "current-directory", - getcwd(cwdbuffer, sizeof cwdbuffer)); - - temp = PySys_GetObject("path"); - if (temp == NULL || !PyList_Check(temp)) { - PyErr_SetString(PyExc_RuntimeError, "sys.path must be a list"); - return -1; - } - len = PyList_GET_SIZE(temp); - for (i = 0; i < len; ++i) { - PyObject *item = PyList_GET_ITEM(temp, i); - buffer = PyString_AsString(item); - if (buffer == NULL) { - pack_add_info(self, "sys-path-entry", ""); - PyErr_Clear(); - } - else { - pack_add_info(self, "sys-path-entry", buffer); - } - } - pack_frame_times(self); - pack_line_times(self); - - return 0; -} - -PyDoc_STRVAR(profiler__doc__, -"profiler(logfilename[, lineevents[, linetimes]]) -> profiler\n\ -Create a new profiler object."); - -static PyObject * -hotshot_profiler(PyObject *unused, PyObject *args) -{ - char *logfilename; - ProfilerObject *self = NULL; - int lineevents = 0; - int linetimings = 1; - - if (PyArg_ParseTuple(args, "s|ii:profiler", &logfilename, - &lineevents, &linetimings)) { - self = PyObject_New(ProfilerObject, &ProfilerType); - if (self == NULL) - return NULL; - self->frametimings = 1; - self->lineevents = lineevents ? 1 : 0; - self->linetimings = (lineevents && linetimings) ? 1 : 0; - self->index = 0; - self->active = 0; - self->next_fileno = 0; - self->logfp = NULL; - self->logfilename = PyTuple_GET_ITEM(args, 0); - Py_INCREF(self->logfilename); - self->filemap = PyDict_New(); - if (self->filemap == NULL) { - Py_DECREF(self); - return NULL; - } - self->logfp = fopen(logfilename, "wb"); - if (self->logfp == NULL) { - Py_DECREF(self); - PyErr_SetFromErrnoWithFilename(PyExc_IOError, logfilename); - return NULL; - } - if (timeofday_diff == 0) { - /* Run this several times since sometimes the first - * doesn't give the lowest values, and we're really trying - * to determine the lowest. - */ - calibrate(); - calibrate(); - calibrate(); - } - if (write_header(self)) { - /* some error occurred, exception has been set */ - Py_DECREF(self); - self = NULL; - } - } - return (PyObject *) self; -} - -PyDoc_STRVAR(coverage__doc__, -"coverage(logfilename) -> profiler\n\ -Returns a profiler that doesn't collect any timing information, which is\n\ -useful in building a coverage analysis tool."); - -static PyObject * -hotshot_coverage(PyObject *unused, PyObject *args) -{ - char *logfilename; - PyObject *result = NULL; - - if (PyArg_ParseTuple(args, "s:coverage", &logfilename)) { - result = hotshot_profiler(unused, args); - if (result != NULL) { - ProfilerObject *self = (ProfilerObject *) result; - self->frametimings = 0; - self->linetimings = 0; - self->lineevents = 1; - } - } - return result; -} - -PyDoc_VAR(resolution__doc__) = -#ifdef MS_WINDOWS -PyDoc_STR( -"resolution() -> (performance-counter-ticks, update-frequency)\n" -"Return the resolution of the timer provided by the QueryPerformanceCounter()\n" -"function. The first value is the smallest observed change, and the second\n" -"is the result of QueryPerformanceFrequency()." -) -#else -PyDoc_STR( -"resolution() -> (gettimeofday-usecs, getrusage-usecs)\n" -"Return the resolution of the timers provided by the gettimeofday() and\n" -"getrusage() system calls, or -1 if the call is not supported." -) -#endif -; - -static PyObject * -hotshot_resolution(PyObject *self, PyObject *unused) -{ - if (timeofday_diff == 0) { - calibrate(); - calibrate(); - calibrate(); - } -#ifdef MS_WINDOWS - return Py_BuildValue("ii", timeofday_diff, frequency.LowPart); -#else - return Py_BuildValue("ii", timeofday_diff, rusage_diff); -#endif -} - - -static PyMethodDef functions[] = { - {"coverage", hotshot_coverage, METH_VARARGS, coverage__doc__}, - {"profiler", hotshot_profiler, METH_VARARGS, profiler__doc__}, - {"logreader", hotshot_logreader, METH_VARARGS, logreader__doc__}, - {"resolution", hotshot_resolution, METH_NOARGS, resolution__doc__}, - {NULL, NULL} -}; - - -void -init_hotshot(void) -{ - PyObject *module; - - Py_Type(&LogReaderType) = &PyType_Type; - Py_Type(&ProfilerType) = &PyType_Type; - module = Py_InitModule("_hotshot", functions); - if (module != NULL) { - char *s = get_version_string(); - - PyModule_AddStringConstant(module, "__version__", s); - free(s); - Py_INCREF(&LogReaderType); - PyModule_AddObject(module, "LogReaderType", - (PyObject *)&LogReaderType); - Py_INCREF(&ProfilerType); - PyModule_AddObject(module, "ProfilerType", - (PyObject *)&ProfilerType); - - if (ProfilerError == NULL) - ProfilerError = PyErr_NewException("hotshot.ProfilerError", - NULL, NULL); - if (ProfilerError != NULL) { - Py_INCREF(ProfilerError); - PyModule_AddObject(module, "ProfilerError", ProfilerError); - } - PyModule_AddIntConstant(module, "WHAT_ENTER", WHAT_ENTER); - PyModule_AddIntConstant(module, "WHAT_EXIT", WHAT_EXIT); - PyModule_AddIntConstant(module, "WHAT_LINENO", WHAT_LINENO); - PyModule_AddIntConstant(module, "WHAT_OTHER", WHAT_OTHER); - PyModule_AddIntConstant(module, "WHAT_ADD_INFO", WHAT_ADD_INFO); - PyModule_AddIntConstant(module, "WHAT_DEFINE_FILE", WHAT_DEFINE_FILE); - PyModule_AddIntConstant(module, "WHAT_DEFINE_FUNC", WHAT_DEFINE_FUNC); - PyModule_AddIntConstant(module, "WHAT_LINE_TIMES", WHAT_LINE_TIMES); - } -} Modified: python/branches/py3k-importlib/Modules/_sqlite/connection.c ============================================================================== --- python/branches/py3k-importlib/Modules/_sqlite/connection.c (original) +++ python/branches/py3k-importlib/Modules/_sqlite/connection.c Thu Oct 11 23:44:13 2007 @@ -425,16 +425,16 @@ sqlite3_result_int64(context, (PY_LONG_LONG)longval); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); - } else if (PyBuffer_Check(py_val)) { + } else if (PyString_Check(py_val)) { + sqlite3_result_text(context, PyString_AsString(py_val), -1, SQLITE_TRANSIENT); + } else if (PyUnicode_Check(py_val)) { + sqlite3_result_text(context, PyUnicode_AsString(py_val), -1, SQLITE_TRANSIENT); + } else if (PyObject_CheckBuffer(py_val)) { if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); } else { sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } - } else if (PyString_Check(py_val)) { - sqlite3_result_text(context, PyString_AsString(py_val), -1, SQLITE_TRANSIENT); - } else if (PyUnicode_Check(py_val)) { - sqlite3_result_text(context, PyUnicode_AsString(py_val), -1, SQLITE_TRANSIENT); } else { /* TODO: raise error */ } @@ -449,7 +449,6 @@ const char* val_str; PY_LONG_LONG val_int; Py_ssize_t buflen; - void* raw_buffer; args = PyTuple_New(argc); if (!args) { @@ -478,16 +477,8 @@ break; case SQLITE_BLOB: buflen = sqlite3_value_bytes(cur_value); - cur_py_value = PyBuffer_New(buflen); - if (!cur_py_value) { - break; - } - if (PyObject_AsWriteBuffer(cur_py_value, &raw_buffer, &buflen)) { - Py_DECREF(cur_py_value); - cur_py_value = NULL; - break; - } - memcpy(raw_buffer, sqlite3_value_blob(cur_value), buflen); + cur_py_value = PyBytes_FromStringAndSize( + sqlite3_value_blob(cur_value), buflen); break; case SQLITE_NULL: default: Modified: python/branches/py3k-importlib/Modules/_sqlite/cursor.c ============================================================================== --- python/branches/py3k-importlib/Modules/_sqlite/cursor.c (original) +++ python/branches/py3k-importlib/Modules/_sqlite/cursor.c Thu Oct 11 23:44:13 2007 @@ -296,7 +296,6 @@ PyObject* converted; Py_ssize_t nbytes; PyObject* buffer; - void* raw_buffer; const char* val_str; char buf[200]; const char* colname; @@ -380,14 +379,11 @@ } else { /* coltype == SQLITE_BLOB */ nbytes = sqlite3_column_bytes(self->statement->st, i); - buffer = PyBuffer_New(nbytes); + buffer = PyBytes_FromStringAndSize( + sqlite3_column_blob(self->statement->st, i), nbytes); if (!buffer) { break; } - if (PyObject_AsWriteBuffer(buffer, &raw_buffer, &nbytes)) { - break; - } - memcpy(raw_buffer, sqlite3_column_blob(self->statement->st, i), nbytes); converted = buffer; } } Modified: python/branches/py3k-importlib/Modules/_sqlite/statement.c ============================================================================== --- python/branches/py3k-importlib/Modules/_sqlite/statement.c (original) +++ python/branches/py3k-importlib/Modules/_sqlite/statement.c Thu Oct 11 23:44:13 2007 @@ -102,13 +102,6 @@ #endif } else if (PyFloat_Check(parameter)) { rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); - } else if (PyBuffer_Check(parameter)) { - if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { - rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT); - } else { - PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - rc = -1; - } } else if PyString_Check(parameter) { string = PyString_AsString(parameter); rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); @@ -118,6 +111,13 @@ rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); Py_DECREF(stringval); + } else if (PyObject_CheckBuffer(parameter)) { + if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { + rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT); + } else { + PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); + rc = -1; + } } else { rc = -1; } Deleted: /python/branches/py3k-importlib/Objects/bufferobject.c ============================================================================== --- /python/branches/py3k-importlib/Objects/bufferobject.c Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,724 +0,0 @@ - -/* Buffer object implementation */ - -#include "Python.h" - - -typedef struct { - PyObject_HEAD - PyObject *b_base; - void *b_ptr; - Py_ssize_t b_size; - Py_ssize_t b_offset; - int b_readonly; - long b_hash; -} PyBufferObject; - - -static int -get_buf(PyBufferObject *self, Py_buffer *view, int flags) -{ - if (self->b_base == NULL) { - view->buf = self->b_ptr; - view->len = self->b_size; - view->readonly = 0; - } - else { - Py_ssize_t count, offset; - PyBufferProcs *bp = self->b_base->ob_type->tp_as_buffer; - if ((*bp->bf_getbuffer)(self->b_base, view, flags) < 0) - return 0; - count = view->len; - /* apply constraints to the start/end */ - if (self->b_offset > count) - offset = count; - else - offset = self->b_offset; - view->buf = (char*)view->buf + offset; - if (self->b_size == Py_END_OF_BUFFER) - view->len = count; - else - view->len = self->b_size; - if (offset + view->len > count) - view->len = count - offset; - } - return 1; -} - - -static int -buffer_getbuf(PyBufferObject *self, Py_buffer *view, int flags) -{ - if (view == NULL) return 0; - if (!get_buf(self, view, flags)) - return -1; - return PyBuffer_FillInfo(view, view->buf, view->len, self->b_readonly, - flags); -} - - -static void -buffer_releasebuf(PyBufferObject *self, Py_buffer *view) -{ - /* No-op if there is no self->b_base */ - if (self->b_base != NULL) { - PyBufferProcs *bp = self->b_base->ob_type->tp_as_buffer; - if (bp->bf_releasebuffer != NULL) { - (*bp->bf_releasebuffer)(self->b_base, view); - } - } -} - -static PyObject * -buffer_from_memory(PyObject *base, Py_ssize_t size, Py_ssize_t offset, - void *ptr, int readonly) -{ - PyBufferObject * b; - - if (size < 0 && size != Py_END_OF_BUFFER) { - PyErr_SetString(PyExc_ValueError, - "size must be zero or positive"); - return NULL; - } - if (offset < 0) { - PyErr_SetString(PyExc_ValueError, - "offset must be zero or positive"); - return NULL; - } - - b = PyObject_NEW(PyBufferObject, &PyBuffer_Type); - if (b == NULL) - return NULL; - - Py_XINCREF(base); - b->b_base = base; - b->b_ptr = ptr; - b->b_size = size; - b->b_offset = offset; - b->b_readonly = readonly; - b->b_hash = -1; - - return (PyObject *) b; -} - -static PyObject * -buffer_from_object(PyObject *base, Py_ssize_t size, Py_ssize_t offset, - int readonly) -{ - if (offset < 0) { - PyErr_SetString(PyExc_ValueError, - "offset must be zero or positive"); - return NULL; - } - if (PyBuffer_Check(base) && (((PyBufferObject *)base)->b_base)) { - /* another buffer, refer to the base object */ - PyBufferObject *b = (PyBufferObject *)base; - if (b->b_size != Py_END_OF_BUFFER) { - Py_ssize_t base_size = b->b_size - offset; - if (base_size < 0) - base_size = 0; - if (size == Py_END_OF_BUFFER || size > base_size) - size = base_size; - } - offset += b->b_offset; - base = b->b_base; - } - return buffer_from_memory(base, size, offset, NULL, readonly); -} - - -PyObject * -PyBuffer_FromObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size) -{ - PyBufferProcs *pb = base->ob_type->tp_as_buffer; - - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_SetString(PyExc_TypeError, "buffer object expected"); - return NULL; - } - - return buffer_from_object(base, size, offset, 1); -} - -PyObject * -PyBuffer_FromReadWriteObject(PyObject *base, Py_ssize_t offset, - Py_ssize_t size) -{ - PyBufferProcs *pb = base->ob_type->tp_as_buffer; - - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_SetString(PyExc_TypeError, "buffer object expected"); - return NULL; - } - - return buffer_from_object(base, size, offset, 0); -} - -PyObject * -PyBuffer_FromMemory(void *ptr, Py_ssize_t size) -{ - return buffer_from_memory(NULL, size, 0, ptr, 1); -} - -PyObject * -PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size) -{ - return buffer_from_memory(NULL, size, 0, ptr, 0); -} - -PyObject * -PyBuffer_New(Py_ssize_t size) -{ - PyObject *o; - PyBufferObject * b; - - if (size < 0) { - PyErr_SetString(PyExc_ValueError, - "size must be zero or positive"); - return NULL; - } - /* XXX: check for overflow in multiply */ - /* Inline PyObject_New */ - o = (PyObject *)PyObject_MALLOC(sizeof(*b) + size); - if (o == NULL) - return PyErr_NoMemory(); - b = (PyBufferObject *) PyObject_INIT(o, &PyBuffer_Type); - - b->b_base = NULL; - b->b_ptr = (void *)(b + 1); - b->b_size = size; - b->b_offset = 0; - b->b_readonly = 0; - b->b_hash = -1; - - return o; -} - -/* Methods */ - -static PyObject * -buffer_new(PyTypeObject *type, PyObject *args, PyObject *kw) -{ - PyObject *ob; - Py_ssize_t offset = 0; - Py_ssize_t size = Py_END_OF_BUFFER; - - if (!_PyArg_NoKeywords("buffer()", kw)) - return NULL; - - if (!PyArg_ParseTuple(args, "O|nn:buffer", &ob, &offset, &size)) - return NULL; - return PyBuffer_FromObject(ob, offset, size); -} - -PyDoc_STRVAR(buffer_doc, -"buffer(object [, offset[, size]])\n\ -\n\ -Create a new buffer object which references the given object.\n\ -The buffer will reference a slice of the target object from the\n\ -start of the object (or at the specified offset). The slice will\n\ -extend to the end of the target object (or with the specified size)."); - - -static void -buffer_dealloc(PyBufferObject *self) -{ - Py_XDECREF(self->b_base); - PyObject_DEL(self); -} - -static int -get_bufx(PyObject *obj, Py_buffer *view, int flags) -{ - PyBufferProcs *bp; - - if (PyBuffer_Check(obj)) { - if (!get_buf((PyBufferObject *)obj, view, flags)) { - PyErr_Clear(); - return 0; - } - else - return 1; - } - bp = obj->ob_type->tp_as_buffer; - if (bp == NULL || - bp->bf_getbuffer == NULL) - return 0; - if ((*bp->bf_getbuffer)(obj, view, PyBUF_SIMPLE) < 0) - return 0; - return 1; -} - -static PyObject * -buffer_richcompare(PyObject *self, PyObject *other, int op) -{ - void *p1, *p2; - Py_ssize_t len1, len2, min_len; - int cmp, ok; - Py_buffer v1, v2; - - ok = 1; - if (!get_bufx(self, &v1, PyBUF_SIMPLE)) - ok = 0; - if (!get_bufx(other, &v2, PyBUF_SIMPLE)) { - if (ok) PyObject_ReleaseBuffer((PyObject *)self, &v1); - ok = 0; - } - if (!ok) { - /* If we can't get the buffers, - == and != are still defined - (and the objects are unequal) */ - PyObject *result; - if (op == Py_EQ) - result = Py_False; - else if (op == Py_NE) - result = Py_True; - else - result = Py_NotImplemented; - Py_INCREF(result); - return result; - } - len1 = v1.len; - len2 = v2.len; - p1 = v1.buf; - p2 = v2.buf; - min_len = (len1 < len2) ? len1 : len2; - cmp = memcmp(p1, p2, min_len); - if (cmp == 0) - cmp = (len1 < len2) ? -1 : - (len1 > len2) ? 1 : 0; - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(other, &v2); - return Py_CmpToRich(op, cmp); -} - -static PyObject * -buffer_repr(PyBufferObject *self) -{ - const char *status = self->b_readonly ? "read-only" : "read-write"; - - if (self->b_base == NULL) - return PyUnicode_FromFormat( - "<%s buffer ptr %p, size %zd at %p>", - status, - self->b_ptr, - self->b_size, - self); - else - return PyUnicode_FromFormat( - "<%s buffer for %p, size %zd, offset %zd at %p>", - status, - self->b_base, - self->b_size, - self->b_offset, - self); -} - -static long -buffer_hash(PyBufferObject *self) -{ - Py_buffer view; - register Py_ssize_t len; - register unsigned char *p; - register long x; - - if (self->b_hash != -1) - return self->b_hash; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - if (!(self->b_readonly)) { - PyErr_SetString(PyExc_TypeError, - "writable buffers are not hashable"); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - p = (unsigned char *) view.buf; - len = view.len; - x = *p << 7; - while (--len >= 0) - x = (1000003*x) ^ *p++; - x ^= view.len; - if (x == -1) - x = -2; - self->b_hash = x; - PyObject_ReleaseBuffer((PyObject *)self, &view); - return x; -} - -static PyObject * -buffer_str(PyBufferObject *self) -{ - Py_buffer view; - PyObject *res; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - res = PyString_FromStringAndSize((const char *)view.buf, view.len); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return res; -} - -/* Sequence methods */ - -static Py_ssize_t -buffer_length(PyBufferObject *self) -{ - Py_buffer view; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - PyObject_ReleaseBuffer((PyObject *)self, &view); - return view.len; -} - -static PyObject * -buffer_concat(PyBufferObject *self, PyObject *other) -{ - PyBufferProcs *pb = other->ob_type->tp_as_buffer; - char *p; - PyObject *ob; - Py_buffer view, view2; - - if (pb == NULL || - pb->bf_getbuffer == NULL) - { - PyErr_BadArgument(); - return NULL; - } - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - - /* optimize special case */ - /* XXX bad idea type-wise */ - if (view.len == 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - Py_INCREF(other); - return other; - } - - if (PyObject_GetBuffer((PyObject *)other, &view2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return NULL; - } - - /* XXX(nnorwitz): need to check for overflow! */ - ob = PyBytes_FromStringAndSize(NULL, view.len+view2.len); - if (ob == NULL) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return NULL; - } - p = PyBytes_AS_STRING(ob); - memcpy(p, view.buf, view.len); - memcpy(p + view.len, view2.buf, view2.len); - - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return ob; -} - -static PyObject * -buffer_repeat(PyBufferObject *self, Py_ssize_t count) -{ - PyObject *ob; - register char *p; - Py_buffer view; - - if (count < 0) - count = 0; - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - /* XXX(nnorwitz): need to check for overflow! */ - ob = PyBytes_FromStringAndSize(NULL, view.len * count); - if (ob == NULL) - return NULL; - - p = PyBytes_AS_STRING(ob); - while (count--) { - memcpy(p, view.buf, view.len); - p += view.len; - } - - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; -} - -static PyObject * -buffer_item(PyBufferObject *self, Py_ssize_t idx) -{ - Py_buffer view; - PyObject *ob; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - if (idx < 0 || idx >= view.len) { - PyErr_SetString(PyExc_IndexError, "buffer index out of range"); - return NULL; - } - ob = PyBytes_FromStringAndSize((char *)view.buf + idx, 1); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; -} - -static PyObject * -buffer_subscript(PyBufferObject *self, PyObject *item) -{ - Py_buffer view; - PyObject *ob; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return NULL; - if (PyIndex_Check(item)) { - Py_ssize_t idx = PyNumber_AsSsize_t(item, PyExc_IndexError); - - if (idx == -1 && PyErr_Occurred()) - return NULL; - if (idx < 0) - idx += view.len; - if (idx < 0 || idx >= view.len) { - PyErr_SetString(PyExc_IndexError, - "buffer index out of range"); - return NULL; - } - ob = PyBytes_FromStringAndSize((char *)view.buf + idx, 1); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - else if (PySlice_Check(item)) { - Py_ssize_t start, stop, step, slicelength, cur, i; - - if (PySlice_GetIndicesEx((PySliceObject*)item, view.len, - &start, &stop, &step, &slicelength) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return NULL; - } - - if (slicelength <= 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return PyBytes_FromStringAndSize("", 0); - } - else if (step == 1) { - ob = PyBytes_FromStringAndSize((char *)view.buf + - start, stop - start); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - else { - char *source_buf = (char *)view.buf; - char *result_buf = (char *)PyMem_Malloc(slicelength); - - if (result_buf == NULL) - return PyErr_NoMemory(); - - for (cur = start, i = 0; i < slicelength; - cur += step, i++) { - result_buf[i] = source_buf[cur]; - } - - ob = PyBytes_FromStringAndSize(result_buf, - slicelength); - PyMem_Free(result_buf); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return ob; - } - } - else { - PyErr_SetString(PyExc_TypeError, - "sequence index must be integer"); - return NULL; - } -} - -static int -buffer_ass_item(PyBufferObject *self, Py_ssize_t idx, PyObject *other) -{ - PyBufferProcs *pb; - Py_buffer view, view2; - - if (!get_buf(self, &view, PyBUF_SIMPLE)) - return -1; - - if (self->b_readonly || view.readonly) { - PyErr_SetString(PyExc_TypeError, - "buffer is read-only"); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - if (idx < 0 || idx >= view.len) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyErr_SetString(PyExc_IndexError, - "buffer assignment index out of range"); - return -1; - } - - pb = other ? other->ob_type->tp_as_buffer : NULL; - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyErr_BadArgument(); - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - - if (PyObject_GetBuffer(other, &view2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - return -1; - } - if (view.len != 1) { - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - PyErr_SetString(PyExc_TypeError, - "right operand must be a single byte"); - return -1; - } - - ((char *)(view.buf))[idx] = *((char *)(view2.buf)); - PyObject_ReleaseBuffer((PyObject *)self, &view); - PyObject_ReleaseBuffer(other, &view2); - return 0; -} - -static int -buffer_ass_subscript(PyBufferObject *self, PyObject *item, PyObject *value) -{ - Py_buffer v1; - - if (!get_buf(self, &v1, PyBUF_SIMPLE)) - return -1; - if (self->b_readonly || v1.readonly) { - PyErr_SetString(PyExc_TypeError, - "buffer is read-only"); - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - if (PyIndex_Check(item)) { - Py_ssize_t idx = PyNumber_AsSsize_t(item, PyExc_IndexError); - if (idx == -1 && PyErr_Occurred()) - return -1; - if (idx < 0) - idx += v1.len; - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return buffer_ass_item(self, idx, value); - } - else if (PySlice_Check(item)) { - Py_ssize_t start, stop, step, slicelength; - Py_buffer v2; - PyBufferProcs *pb; - - if (PySlice_GetIndicesEx((PySliceObject *)item, v1.len, - &start, &stop, &step, &slicelength) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - - pb = value ? value->ob_type->tp_as_buffer : NULL; - if (pb == NULL || - pb->bf_getbuffer == NULL) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyErr_BadArgument(); - return -1; - } - if ((*pb->bf_getbuffer)(value, &v2, PyBUF_SIMPLE) < 0) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } - - if (v2.len != slicelength) { - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(value, &v2); - PyErr_SetString(PyExc_TypeError, "right operand" - " length must match slice length"); - return -1; - } - - if (slicelength == 0) - /* nothing to do */; - else if (step == 1) - memcpy((char *)v1.buf + start, v2.buf, slicelength); - else { - Py_ssize_t cur, i; - - for (cur = start, i = 0; i < slicelength; - cur += step, i++) { - ((char *)v1.buf)[cur] = ((char *)v2.buf)[i]; - } - } - PyObject_ReleaseBuffer((PyObject *)self, &v1); - PyObject_ReleaseBuffer(value, &v2); - return 0; - } else { - PyErr_SetString(PyExc_TypeError, - "buffer indices must be integers"); - PyObject_ReleaseBuffer((PyObject *)self, &v1); - return -1; - } -} - -/* Buffer methods */ - -static PySequenceMethods buffer_as_sequence = { - (lenfunc)buffer_length, /*sq_length*/ - (binaryfunc)buffer_concat, /*sq_concat*/ - (ssizeargfunc)buffer_repeat, /*sq_repeat*/ - (ssizeargfunc)buffer_item, /*sq_item*/ - 0, /*sq_slice*/ - (ssizeobjargproc)buffer_ass_item, /*sq_ass_item*/ - 0, /*sq_ass_slice*/ -}; - -static PyMappingMethods buffer_as_mapping = { - (lenfunc)buffer_length, - (binaryfunc)buffer_subscript, - (objobjargproc)buffer_ass_subscript, -}; - -static PyBufferProcs buffer_as_buffer = { - (getbufferproc)buffer_getbuf, - (releasebufferproc)buffer_releasebuf, -}; - -PyTypeObject PyBuffer_Type = { - PyVarObject_HEAD_INIT(&PyType_Type, 0) - "buffer", - sizeof(PyBufferObject), - 0, - (destructor)buffer_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - (reprfunc)buffer_repr, /* tp_repr */ - 0, /* tp_as_number */ - &buffer_as_sequence, /* tp_as_sequence */ - &buffer_as_mapping, /* tp_as_mapping */ - (hashfunc)buffer_hash, /* tp_hash */ - 0, /* tp_call */ - (reprfunc)buffer_str, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - &buffer_as_buffer, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - buffer_doc, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - buffer_richcompare, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - buffer_new, /* tp_new */ -}; Modified: python/branches/py3k-importlib/Objects/bytesobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/bytesobject.c (original) +++ python/branches/py3k-importlib/Objects/bytesobject.c Thu Oct 11 23:44:13 2007 @@ -964,8 +964,8 @@ error, even if the comparison is for equality. */ if (PyObject_IsInstance(self, (PyObject*)&PyUnicode_Type) || PyObject_IsInstance(other, (PyObject*)&PyUnicode_Type)) { - PyErr_SetString(PyExc_TypeError, "can't compare bytes and str"); - return NULL; + Py_INCREF(Py_NotImplemented); + return Py_NotImplemented; } self_size = _getbuffer(self, &self_bytes); Modified: python/branches/py3k-importlib/Objects/fileobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/fileobject.c (original) +++ python/branches/py3k-importlib/Objects/fileobject.c Thu Oct 11 23:44:13 2007 @@ -145,7 +145,7 @@ value = _PyObject_Str(v); } else - value = PyObject_ReprStr8(v); + value = PyObject_Repr(v); if (value == NULL) { Py_DECREF(writer); return -1; Modified: python/branches/py3k-importlib/Objects/memoryobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/memoryobject.c (original) +++ python/branches/py3k-importlib/Objects/memoryobject.c Thu Oct 11 23:44:13 2007 @@ -8,6 +8,8 @@ { if (view != NULL) *view = self->view; + if (self->base == NULL) + return 0; return self->base->ob_type->tp_as_buffer->bf_getbuffer(self->base, NULL, PyBUF_FULL); } @@ -15,7 +17,8 @@ static void memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) { - PyObject_ReleaseBuffer(self->base, NULL); + if (self->base != NULL) + PyObject_ReleaseBuffer(self->base, NULL); } PyDoc_STRVAR(memory_doc, Modified: python/branches/py3k-importlib/Objects/stringobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/stringobject.c (original) +++ python/branches/py3k-importlib/Objects/stringobject.c Thu Oct 11 23:44:13 2007 @@ -4,7 +4,256 @@ #include "Python.h" -#include +/* Our own locale-independent ctype.h-like macros */ +/* XXX Move into a header file? */ + +#define FLAG_LOWER 0x01 +#define FLAG_UPPER 0x02 +#define FLAG_ALPHA (FLAG_LOWER|FLAG_UPPER) +#define FLAG_DIGIT 0x04 +#define FLAG_ALNUM (FLAG_ALPHA|FLAG_DIGIT) +#define FLAG_SPACE 0x08 +#define FLAG_XDIGIT 0x10 + +static unsigned int ctype_table[256] = { + 0, /* 0x0 '\x00' */ + 0, /* 0x1 '\x01' */ + 0, /* 0x2 '\x02' */ + 0, /* 0x3 '\x03' */ + 0, /* 0x4 '\x04' */ + 0, /* 0x5 '\x05' */ + 0, /* 0x6 '\x06' */ + 0, /* 0x7 '\x07' */ + 0, /* 0x8 '\x08' */ + FLAG_SPACE, /* 0x9 '\t' */ + FLAG_SPACE, /* 0xa '\n' */ + FLAG_SPACE, /* 0xb '\v' */ + FLAG_SPACE, /* 0xc '\f' */ + FLAG_SPACE, /* 0xd '\r' */ + 0, /* 0xe '\x0e' */ + 0, /* 0xf '\x0f' */ + 0, /* 0x10 '\x10' */ + 0, /* 0x11 '\x11' */ + 0, /* 0x12 '\x12' */ + 0, /* 0x13 '\x13' */ + 0, /* 0x14 '\x14' */ + 0, /* 0x15 '\x15' */ + 0, /* 0x16 '\x16' */ + 0, /* 0x17 '\x17' */ + 0, /* 0x18 '\x18' */ + 0, /* 0x19 '\x19' */ + 0, /* 0x1a '\x1a' */ + 0, /* 0x1b '\x1b' */ + 0, /* 0x1c '\x1c' */ + 0, /* 0x1d '\x1d' */ + 0, /* 0x1e '\x1e' */ + 0, /* 0x1f '\x1f' */ + FLAG_SPACE, /* 0x20 ' ' */ + 0, /* 0x21 '!' */ + 0, /* 0x22 '"' */ + 0, /* 0x23 '#' */ + 0, /* 0x24 '$' */ + 0, /* 0x25 '%' */ + 0, /* 0x26 '&' */ + 0, /* 0x27 "'" */ + 0, /* 0x28 '(' */ + 0, /* 0x29 ')' */ + 0, /* 0x2a '*' */ + 0, /* 0x2b '+' */ + 0, /* 0x2c ',' */ + 0, /* 0x2d '-' */ + 0, /* 0x2e '.' */ + 0, /* 0x2f '/' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x30 '0' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x31 '1' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x32 '2' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x33 '3' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x34 '4' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x35 '5' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x36 '6' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x37 '7' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x38 '8' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x39 '9' */ + 0, /* 0x3a ':' */ + 0, /* 0x3b ';' */ + 0, /* 0x3c '<' */ + 0, /* 0x3d '=' */ + 0, /* 0x3e '>' */ + 0, /* 0x3f '?' */ + 0, /* 0x40 '@' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x41 'A' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x42 'B' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x43 'C' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x44 'D' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x45 'E' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x46 'F' */ + FLAG_UPPER, /* 0x47 'G' */ + FLAG_UPPER, /* 0x48 'H' */ + FLAG_UPPER, /* 0x49 'I' */ + FLAG_UPPER, /* 0x4a 'J' */ + FLAG_UPPER, /* 0x4b 'K' */ + FLAG_UPPER, /* 0x4c 'L' */ + FLAG_UPPER, /* 0x4d 'M' */ + FLAG_UPPER, /* 0x4e 'N' */ + FLAG_UPPER, /* 0x4f 'O' */ + FLAG_UPPER, /* 0x50 'P' */ + FLAG_UPPER, /* 0x51 'Q' */ + FLAG_UPPER, /* 0x52 'R' */ + FLAG_UPPER, /* 0x53 'S' */ + FLAG_UPPER, /* 0x54 'T' */ + FLAG_UPPER, /* 0x55 'U' */ + FLAG_UPPER, /* 0x56 'V' */ + FLAG_UPPER, /* 0x57 'W' */ + FLAG_UPPER, /* 0x58 'X' */ + FLAG_UPPER, /* 0x59 'Y' */ + FLAG_UPPER, /* 0x5a 'Z' */ + 0, /* 0x5b '[' */ + 0, /* 0x5c '\\' */ + 0, /* 0x5d ']' */ + 0, /* 0x5e '^' */ + 0, /* 0x5f '_' */ + 0, /* 0x60 '`' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x61 'a' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x62 'b' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x63 'c' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x64 'd' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x65 'e' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x66 'f' */ + FLAG_LOWER, /* 0x67 'g' */ + FLAG_LOWER, /* 0x68 'h' */ + FLAG_LOWER, /* 0x69 'i' */ + FLAG_LOWER, /* 0x6a 'j' */ + FLAG_LOWER, /* 0x6b 'k' */ + FLAG_LOWER, /* 0x6c 'l' */ + FLAG_LOWER, /* 0x6d 'm' */ + FLAG_LOWER, /* 0x6e 'n' */ + FLAG_LOWER, /* 0x6f 'o' */ + FLAG_LOWER, /* 0x70 'p' */ + FLAG_LOWER, /* 0x71 'q' */ + FLAG_LOWER, /* 0x72 'r' */ + FLAG_LOWER, /* 0x73 's' */ + FLAG_LOWER, /* 0x74 't' */ + FLAG_LOWER, /* 0x75 'u' */ + FLAG_LOWER, /* 0x76 'v' */ + FLAG_LOWER, /* 0x77 'w' */ + FLAG_LOWER, /* 0x78 'x' */ + FLAG_LOWER, /* 0x79 'y' */ + FLAG_LOWER, /* 0x7a 'z' */ + 0, /* 0x7b '{' */ + 0, /* 0x7c '|' */ + 0, /* 0x7d '}' */ + 0, /* 0x7e '~' */ + 0, /* 0x7f '\x7f' */ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; + +#define ISLOWER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_LOWER) +#define ISUPPER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_UPPER) +#define ISALPHA(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALPHA) +#define ISDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_DIGIT) +#define ISXDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_XDIGIT) +#define ISALNUM(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALNUM) +#define ISSPACE(c) (ctype_table[Py_CHARMASK(c)] & FLAG_SPACE) + +#undef islower +#define islower(c) undefined_islower(c) +#undef isupper +#define isupper(c) undefined_isupper(c) +#undef isalpha +#define isalpha(c) undefined_isalpha(c) +#undef isdigit +#define isdigit(c) undefined_isdigit(c) +#undef isxdigit +#define isxdigit(c) undefined_isxdigit(c) +#undef isalnum +#define isalnum(c) undefined_isalnum(c) +#undef isspace +#define isspace(c) undefined_isspace(c) + +static unsigned char ctype_tolower[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + +static unsigned char ctype_toupper[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + +#define TOLOWER(c) (ctype_tolower[Py_CHARMASK(c)]) +#define TOUPPER(c) (ctype_toupper[Py_CHARMASK(c)]) + +#undef tolower +#define tolower(c) undefined_tolower(c) +#undef toupper +#define toupper(c) undefined_toupper(c) #ifdef COUNT_ALLOCS int null_strings, one_strings; @@ -173,7 +422,7 @@ for (f = format; *f; f++) { if (*f == '%') { const char* p = f; - while (*++f && *f != '%' && !isalpha(Py_CHARMASK(*f))) + while (*++f && *f != '%' && !ISALPHA(*f)) ; /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since @@ -242,15 +491,15 @@ /* parse the width.precision part (we're only interested in the precision value, if any) */ n = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) n = (n*10) + *f++ - '0'; if (*f == '.') { f++; n = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) n = (n*10) + *f++ - '0'; } - while (*f && *f != '%' && !isalpha(Py_CHARMASK(*f))) + while (*f && *f != '%' && !ISALPHA(*f)) f++; /* handle the long flag, but only for %ld and %lu. others can be added when necessary. */ @@ -606,23 +855,22 @@ *p++ = c; break; case 'x': - if (isxdigit(Py_CHARMASK(s[0])) - && isxdigit(Py_CHARMASK(s[1]))) { + if (ISXDIGIT(s[0]) && ISXDIGIT(s[1])) { unsigned int x = 0; c = Py_CHARMASK(*s); s++; - if (isdigit(c)) + if (ISDIGIT(c)) x = c - '0'; - else if (islower(c)) + else if (ISLOWER(c)) x = 10 + c - 'a'; else x = 10 + c - 'A'; x = x << 4; c = Py_CHARMASK(*s); s++; - if (isdigit(c)) + if (ISDIGIT(c)) x += c - '0'; - else if (islower(c)) + else if (ISLOWER(c)) x += 10 + c - 'a'; else x += 10 + c - 'A'; @@ -1250,10 +1498,10 @@ /* Always force the list to the expected size. */ #define FIX_PREALLOC_SIZE(list) Py_Size(list) = count -#define SKIP_SPACE(s, i, len) { while (i=0 && isspace(Py_CHARMASK(s[i]))) i--; } -#define RSKIP_NONSPACE(s, i) { while (i>=0 && !isspace(Py_CHARMASK(s[i]))) i--; } +#define SKIP_SPACE(s, i, len) { while (i=0 && ISSPACE(s[i])) i--; } +#define RSKIP_NONSPACE(s, i) { while (i>=0 && !ISSPACE(s[i])) i--; } Py_LOCAL_INLINE(PyObject *) split_whitespace(const char *s, Py_ssize_t len, Py_ssize_t maxsplit) @@ -1869,7 +2117,7 @@ i = 0; if (striptype != RIGHTSTRIP) { - while (i < len && isspace(Py_CHARMASK(s[i]))) { + while (i < len && ISSPACE(s[i])) { i++; } } @@ -1878,7 +2126,7 @@ if (striptype != LEFTSTRIP) { do { j--; - } while (j >= i && isspace(Py_CHARMASK(s[j]))); + } while (j >= i && ISSPACE(s[j])); j++; } @@ -1979,11 +2227,6 @@ \n\ Return a copy of the string S converted to lowercase."); -/* _tolower and _toupper are defined by SUSv2, but they're not ISO C */ -#ifndef _tolower -#define _tolower tolower -#endif - static PyObject * string_lower(PyStringObject *self) { @@ -2001,8 +2244,8 @@ for (i = 0; i < n; i++) { int c = Py_CHARMASK(s[i]); - if (isupper(c)) - s[i] = _tolower(c); + if (ISUPPER(c)) + s[i] = TOLOWER(c); } return newobj; @@ -2013,10 +2256,6 @@ \n\ Return a copy of the string S converted to uppercase."); -#ifndef _toupper -#define _toupper toupper -#endif - static PyObject * string_upper(PyStringObject *self) { @@ -2034,8 +2273,8 @@ for (i = 0; i < n; i++) { int c = Py_CHARMASK(s[i]); - if (islower(c)) - s[i] = _toupper(c); + if (ISLOWER(c)) + s[i] = TOUPPER(c); } return newobj; @@ -2061,13 +2300,13 @@ s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); - if (islower(c)) { + if (ISLOWER(c)) { if (!previous_is_cased) - c = toupper(c); + c = TOUPPER(c); previous_is_cased = 1; - } else if (isupper(c)) { + } else if (ISUPPER(c)) { if (previous_is_cased) - c = tolower(c); + c = TOLOWER(c); previous_is_cased = 1; } else previous_is_cased = 0; @@ -2095,16 +2334,16 @@ s_new = PyString_AsString(newobj); if (0 < n) { int c = Py_CHARMASK(*s++); - if (islower(c)) - *s_new = toupper(c); + if (ISLOWER(c)) + *s_new = TOUPPER(c); else *s_new = c; s_new++; } for (i = 1; i < n; i++) { int c = Py_CHARMASK(*s++); - if (isupper(c)) - *s_new = tolower(c); + if (ISUPPER(c)) + *s_new = TOLOWER(c); else *s_new = c; s_new++; @@ -2173,11 +2412,11 @@ s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); - if (islower(c)) { - *s_new = toupper(c); + if (ISLOWER(c)) { + *s_new = TOUPPER(c); } - else if (isupper(c)) { - *s_new = tolower(c); + else if (ISUPPER(c)) { + *s_new = TOLOWER(c); } else *s_new = c; @@ -3386,7 +3625,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1 && - isspace(*p)) + ISSPACE(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3395,7 +3634,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isspace(*p)) + if (!ISSPACE(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3417,7 +3656,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1 && - isalpha(*p)) + ISALPHA(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3426,7 +3665,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isalpha(*p)) + if (!ISALPHA(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3447,8 +3686,7 @@ register const unsigned char *e; /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - isalnum(*p)) + if (PyString_GET_SIZE(self) == 1 && ISALNUM(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3457,7 +3695,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isalnum(*p)) + if (!ISALNUM(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3478,8 +3716,7 @@ register const unsigned char *e; /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - isdigit(*p)) + if (PyString_GET_SIZE(self) == 1 && ISDIGIT(*p)) return PyBool_FromLong(1); /* Special case for empty strings */ @@ -3488,7 +3725,7 @@ e = p + PyString_GET_SIZE(self); for (; p < e; p++) { - if (!isdigit(*p)) + if (!ISDIGIT(*p)) return PyBool_FromLong(0); } return PyBool_FromLong(1); @@ -3511,7 +3748,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(islower(*p) != 0); + return PyBool_FromLong(ISLOWER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3520,9 +3757,9 @@ e = p + PyString_GET_SIZE(self); cased = 0; for (; p < e; p++) { - if (isupper(*p)) + if (ISUPPER(*p)) return PyBool_FromLong(0); - else if (!cased && islower(*p)) + else if (!cased && ISLOWER(*p)) cased = 1; } return PyBool_FromLong(cased); @@ -3545,7 +3782,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(isupper(*p) != 0); + return PyBool_FromLong(ISUPPER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3554,9 +3791,9 @@ e = p + PyString_GET_SIZE(self); cased = 0; for (; p < e; p++) { - if (islower(*p)) + if (ISLOWER(*p)) return PyBool_FromLong(0); - else if (!cased && isupper(*p)) + else if (!cased && ISUPPER(*p)) cased = 1; } return PyBool_FromLong(cased); @@ -3581,7 +3818,7 @@ /* Shortcut for single character strings */ if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(isupper(*p) != 0); + return PyBool_FromLong(ISUPPER(*p)); /* Special case for empty strings */ if (PyString_GET_SIZE(self) == 0) @@ -3593,13 +3830,13 @@ for (; p < e; p++) { register const unsigned char ch = *p; - if (isupper(ch)) { + if (ISUPPER(ch)) { if (previous_is_cased) return PyBool_FromLong(0); previous_is_cased = 1; cased = 1; } - else if (islower(ch)) { + else if (ISLOWER(ch)) { if (!previous_is_cased) return PyBool_FromLong(0); previous_is_cased = 1; @@ -4434,11 +4671,11 @@ if (--fmtcnt >= 0) c = *fmt++; } - else if (c >= 0 && isdigit(c)) { + else if (c >= 0 && ISDIGIT(c)) { width = c - '0'; while (--fmtcnt >= 0) { c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) + if (!ISDIGIT(c)) break; if ((width*10) / 10 != width) { PyErr_SetString( @@ -4471,11 +4708,11 @@ if (--fmtcnt >= 0) c = *fmt++; } - else if (c >= 0 && isdigit(c)) { + else if (c >= 0 && ISDIGIT(c)) { prec = c - '0'; while (--fmtcnt >= 0) { c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) + if (!ISDIGIT(c)) break; if ((prec*10) / 10 != prec) { PyErr_SetString( Modified: python/branches/py3k-importlib/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/unicodeobject.c (original) +++ python/branches/py3k-importlib/Objects/unicodeobject.c Thu Oct 11 23:44:13 2007 @@ -1051,6 +1051,7 @@ const char *errors) { PyObject *buffer = NULL, *unicode; + Py_buffer info; if (encoding == NULL) encoding = PyUnicode_GetDefaultEncoding(); @@ -1068,7 +1069,10 @@ return PyUnicode_DecodeASCII(s, size, errors); /* Decode via the codec registry */ - buffer = PyBuffer_FromMemory((void *)s, size); + buffer = NULL; + if (PyBuffer_FillInfo(&info, (void *)s, size, 1, PyBUF_SIMPLE) < 0) + goto onError; + buffer = PyMemoryView_FromMemory(&info); if (buffer == NULL) goto onError; unicode = PyCodec_Decode(buffer, encoding, errors); Modified: python/branches/py3k-importlib/PC/VC6/pythoncore.dsp ============================================================================== --- python/branches/py3k-importlib/PC/VC6/pythoncore.dsp (original) +++ python/branches/py3k-importlib/PC/VC6/pythoncore.dsp Thu Oct 11 23:44:13 2007 @@ -221,10 +221,6 @@ # End Source File # Begin Source File -SOURCE=..\..\Objects\bufferobject.c -# End Source File -# Begin Source File - SOURCE=..\..\Objects\cellobject.c # End Source File # Begin Source File Modified: python/branches/py3k-importlib/PC/config.c ============================================================================== --- python/branches/py3k-importlib/PC/config.c (original) +++ python/branches/py3k-importlib/PC/config.c Thu Oct 11 23:44:13 2007 @@ -28,7 +28,6 @@ #endif extern void init_codecs(void); extern void init_weakref(void); -extern void init_hotshot(void); extern void initxxsubtype(void); extern void initzipimport(void); extern void init_random(void); Modified: python/branches/py3k-importlib/PC/os2emx/Makefile ============================================================================== --- python/branches/py3k-importlib/PC/os2emx/Makefile (original) +++ python/branches/py3k-importlib/PC/os2emx/Makefile Thu Oct 11 23:44:13 2007 @@ -366,7 +366,6 @@ SRC.OBJECT= $(addprefix $(TOP), \ Objects/abstract.c \ Objects/boolobject.c \ - Objects/bufferobject.c \ Objects/cellobject.c \ Objects/classobject.c \ Objects/cobject.c \ Modified: python/branches/py3k-importlib/PC/os2emx/python25.def ============================================================================== --- python/branches/py3k-importlib/PC/os2emx/python25.def (original) +++ python/branches/py3k-importlib/PC/os2emx/python25.def Thu Oct 11 23:44:13 2007 @@ -181,14 +181,6 @@ "_Py_ZeroStruct" "_Py_TrueStruct" -; From python25_s.lib(bufferobject) - "PyBuffer_FromObject" - "PyBuffer_FromReadWriteObject" - "PyBuffer_FromMemory" - "PyBuffer_FromReadWriteMemory" - "PyBuffer_New" - "PyBuffer_Type" - ; From python25_s.lib(cellobject) "PyCell_New" "PyCell_Get" Modified: python/branches/py3k-importlib/PCbuild/python.iss ============================================================================== --- python/branches/py3k-importlib/PCbuild/python.iss (original) +++ python/branches/py3k-importlib/PCbuild/python.iss Thu Oct 11 23:44:13 2007 @@ -174,10 +174,8 @@ Source: Lib\distutils\*.*; DestDir: {app}\Lib\distutils; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\email\*.*; DestDir: {app}\Lib\email; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\encodings\*.*; DestDir: {app}\Lib\encodings; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs -Source: Lib\hotshot\*.*; DestDir: {app}\Lib\hotshot; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\lib-old\*.*; DestDir: {app}\Lib\lib-old; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\xml\*.*; DestDir: {app}\Lib\xml; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs -Source: Lib\hotshot\*.*; DestDir: {app}\Lib\hotshot; CopyMode: alwaysoverwrite; Components: main; Flags: recursesubdirs Source: Lib\test\*.*; DestDir: {app}\Lib\test; CopyMode: alwaysoverwrite; Components: test; Flags: recursesubdirs Source: Lib\site-packages\README.txt; DestDir: {app}\Lib\site-packages; CopyMode: alwaysoverwrite; Components: main @@ -284,9 +282,6 @@ Name: {app}\Lib\encodings\*.pyc; Type: files Name: {app}\Lib\encodings\*.pyo; Type: files Name: {app}\Lib\encodings; Type: dirifempty -Name: {app}\Lib\hotshot\*.pyc; Type: files -Name: {app}\Lib\hotshot\*.pyo; Type: files -Name: {app}\Lib\hotshot; Type: dirifempty Name: {app}\Lib\lib-old\*.pyc; Type: files Name: {app}\Lib\lib-old\*.pyo; Type: files Name: {app}\Lib\lib-old; Type: dirifempty Modified: python/branches/py3k-importlib/PCbuild/python20.wse ============================================================================== --- python/branches/py3k-importlib/PCbuild/python20.wse (original) +++ python/branches/py3k-importlib/PCbuild/python20.wse Thu Oct 11 23:44:13 2007 @@ -2024,14 +2024,6 @@ item: Remark end item: Install File - Source=..\lib\hotshot\*.py - Destination=%MAINDIR%\Lib\hotshot - Description=Fast Python profiler - Flags=0000000000000010 -end -item: Remark -end -item: Install File Source=..\lib\lib-old\*.py Destination=%MAINDIR%\Lib\lib-old Description=Obsolete modules Modified: python/branches/py3k-importlib/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/py3k-importlib/PCbuild/pythoncore.vcproj (original) +++ python/branches/py3k-importlib/PCbuild/pythoncore.vcproj Thu Oct 11 23:44:13 2007 @@ -380,9 +380,6 @@ RelativePath="..\Modules\_heapqmodule.c"> - - - - - - @@ -978,10 +974,6 @@ > - - Modified: python/branches/py3k-importlib/Parser/tokenizer.c ============================================================================== --- python/branches/py3k-importlib/Parser/tokenizer.c (original) +++ python/branches/py3k-importlib/Parser/tokenizer.c Thu Oct 11 23:44:13 2007 @@ -357,7 +357,7 @@ 1) NULL: need to call tok->decoding_readline to get a new line 2) PyUnicodeObject *: decoding_feof has called tok->decoding_readline and stored the result in tok->decoding_buffer - 3) PyStringObject *: previous call to fp_readl did not have enough room + 3) PyBytesObject *: previous call to fp_readl did not have enough room (in the s buffer) to copy entire contents of the line read by tok->decoding_readline. tok->decoding_buffer has the overflow. In this case, fp_readl is called in a loop (with an expanded buffer) @@ -625,7 +625,7 @@ utf8 = translate_into_utf8(str, tok->enc); if (utf8 == NULL) return error_ret(tok); - str = PyString_AsString(utf8); + str = PyBytes_AsString(utf8); } for (s = str;; s++) { if (*s == '\0') break; @@ -1556,7 +1556,10 @@ Py_DECREF(unicode_text); } if (!ret) { - PyErr_Print(); + PyErr_Clear(); + } + else { + assert(PyBytes_Check(ret)); } return ret; } @@ -1569,8 +1572,8 @@ /* convert source to original encondig */ PyObject *lineobj = dec_utf8(tok->encoding, tok->buf, len); if (lineobj != NULL) { - int linelen = PyString_Size(lineobj); - const char *line = PyString_AsString(lineobj); + int linelen = PyBytes_GET_SIZE(lineobj); + const char *line = PyBytes_AS_STRING(lineobj); text = PyObject_MALLOC(linelen + 1); if (text != NULL && line != NULL) { if (linelen) @@ -1582,9 +1585,11 @@ /* adjust error offset */ if (*offset > 1) { PyObject *offsetobj = dec_utf8(tok->encoding, - tok->buf, *offset-1); + tok->buf, + *offset-1); if (offsetobj) { - *offset = PyString_Size(offsetobj) + 1; + *offset = 1 + + PyBytes_GET_SIZE(offsetobj); Py_DECREF(offsetobj); } } Modified: python/branches/py3k-importlib/Python/errors.c ============================================================================== --- python/branches/py3k-importlib/Python/errors.c (original) +++ python/branches/py3k-importlib/Python/errors.c Thu Oct 11 23:44:13 2007 @@ -800,13 +800,11 @@ PyErr_Restore(exc, v, tb); } -/* com_fetch_program_text will attempt to load the line of text that - the exception refers to. If it fails, it will return NULL but will - not set an exception. +/* Attempt to load the line of text that the exception refers to. If it + fails, it will return NULL but will not set an exception. XXX The functionality of this function is quite similar to the - functionality in tb_displayline() in traceback.c. -*/ + functionality in tb_displayline() in traceback.c. */ PyObject * PyErr_ProgramText(const char *filename, int lineno) @@ -824,7 +822,8 @@ char *pLastChar = &linebuf[sizeof(linebuf) - 2]; do { *pLastChar = '\0'; - if (Py_UniversalNewlineFgets(linebuf, sizeof linebuf, fp, NULL) == NULL) + if (Py_UniversalNewlineFgets(linebuf, sizeof linebuf, + fp, NULL) == NULL) break; /* fgets read *something*; if it didn't get as far as pLastChar, it must have found a newline @@ -836,9 +835,13 @@ fclose(fp); if (i == lineno) { char *p = linebuf; + PyObject *res; while (*p == ' ' || *p == '\t' || *p == '\014') p++; - return PyUnicode_FromString(p); + res = PyUnicode_FromString(p); + if (res == NULL) + PyErr_Clear(); + return res; } return NULL; } Modified: python/branches/py3k-importlib/Python/pythonrun.c ============================================================================== --- python/branches/py3k-importlib/Python/pythonrun.c (original) +++ python/branches/py3k-importlib/Python/pythonrun.c Thu Oct 11 23:44:13 2007 @@ -764,19 +764,19 @@ } v = PySys_GetObject("ps1"); if (v != NULL) { - v = PyObject_Str(v); + v = PyObject_Unicode(v); if (v == NULL) PyErr_Clear(); - else if (PyString_Check(v)) - ps1 = PyString_AsString(v); + else if (PyUnicode_Check(v)) + ps1 = PyUnicode_AsString(v); } w = PySys_GetObject("ps2"); if (w != NULL) { - w = PyObject_Str(w); + w = PyObject_Unicode(w); if (w == NULL) PyErr_Clear(); - else if (PyString_Check(w)) - ps2 = PyString_AsString(w); + else if (PyUnicode_Check(w)) + ps2 = PyUnicode_AsString(w); } arena = PyArena_New(); if (arena == NULL) { @@ -981,7 +981,8 @@ goto finally; if (v == Py_None) *text = NULL; - else if (! (*text = PyString_AsString(v))) + else if (!PyUnicode_Check(v) || + !(*text = PyUnicode_AsString(v))) goto finally; Py_DECREF(v); return 1; @@ -1095,7 +1096,7 @@ if (set_sys_last_vars) { PySys_SetObject("last_type", exception); PySys_SetObject("last_value", v); - PySys_SetObject("last_traceback", tb); + PySys_SetObject("last_traceback", tb ? tb : Py_None); } hook = PySys_GetObject("excepthook"); if (hook) { @@ -1197,10 +1198,13 @@ } moduleName = PyObject_GetAttrString(exception, "__module__"); - if (moduleName == NULL) + if (moduleName == NULL || !PyUnicode_Check(moduleName)) + { + Py_DECREF(moduleName); err = PyFile_WriteString("", f); + } else { - char* modstr = PyString_AsString(moduleName); + char* modstr = PyUnicode_AsString(moduleName); if (modstr && strcmp(modstr, "__builtin__")) { err = PyFile_WriteString(modstr, f); @@ -1218,14 +1222,14 @@ else err = PyFile_WriteObject(exception, f, Py_PRINT_RAW); if (err == 0 && (value != Py_None)) { - PyObject *s = PyObject_Str(value); + PyObject *s = PyObject_Unicode(value); /* only print colon if the str() of the object is not the empty string */ if (s == NULL) err = -1; - else if (!PyString_Check(s) || - PyString_GET_SIZE(s) != 0) + else if (!PyUnicode_Check(s) || + PyUnicode_GetSize(s) != 0) err = PyFile_WriteString(": ", f); if (err == 0) err = PyFile_WriteObject(s, f, Py_PRINT_RAW); @@ -1532,9 +1536,9 @@ PyObject *type, *value, *tb; PyErr_Fetch(&type, &value, &tb); if (value != NULL) { - u = PyObject_Str(value); + u = PyObject_Unicode(value); if (u != NULL) { - msg = PyString_AsString(u); + msg = PyUnicode_AsString(u); } } if (msg == NULL) Deleted: /python/branches/py3k-importlib/Tools/scripts/hotshotmain.py ============================================================================== --- /python/branches/py3k-importlib/Tools/scripts/hotshotmain.py Thu Oct 11 23:44:13 2007 +++ (empty file) @@ -1,60 +0,0 @@ -#!/usr/bin/env python -# -*- coding: iso-8859-1 -*- - -""" -Run a Python script under hotshot's control. - -Adapted from a posting on python-dev by Walter D?rwald - -usage %prog [ %prog args ] filename [ filename args ] - -Any arguments after the filename are used as sys.argv for the filename. -""" - -import sys -import optparse -import os -import hotshot -import hotshot.stats - -PROFILE = "hotshot.prof" - -def run_hotshot(filename, profile, args): - prof = hotshot.Profile(profile) - sys.path.insert(0, os.path.dirname(filename)) - sys.argv = [filename] + args - fp = open(filename) - try: - script = fp.read() - finally: - fp.close() - prof.run("exec(%r)" % script) - prof.close() - stats = hotshot.stats.load(profile) - stats.sort_stats("time", "calls") - - # print_stats uses unadorned print statements, so the only way - # to force output to stderr is to reassign sys.stdout temporarily - save_stdout = sys.stdout - sys.stdout = sys.stderr - stats.print_stats() - sys.stdout = save_stdout - - return 0 - -def main(args): - parser = optparse.OptionParser(__doc__) - parser.disable_interspersed_args() - parser.add_option("-p", "--profile", action="store", default=PROFILE, - dest="profile", help='Specify profile file to use') - (options, args) = parser.parse_args(args) - - if len(args) == 0: - parser.print_help("missing script to execute") - return 1 - - filename = args[0] - return run_hotshot(filename, options.profile, args[1:]) - -if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) Modified: python/branches/py3k-importlib/setup.py ============================================================================== --- python/branches/py3k-importlib/setup.py (original) +++ python/branches/py3k-importlib/setup.py Thu Oct 11 23:44:13 2007 @@ -413,8 +413,7 @@ exts.append( Extension("atexit", ["atexitmodule.c"]) ) # Python C API test module exts.append( Extension('_testcapi', ['_testcapimodule.c']) ) - # profilers (_lsprof is for cProfile.py) - exts.append( Extension('_hotshot', ['_hotshot.c']) ) + # profiler (_lsprof is for cProfile.py) exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) ) # static Unicode character database exts.append( Extension('unicodedata', ['unicodedata.c']) ) From python-3000-checkins at python.org Thu Oct 11 23:50:17 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Thu, 11 Oct 2007 23:50:17 +0200 (CEST) Subject: [Python-3000-checkins] r58417 - python/branches/py3k-importlib/Python/bltinmodule.c Message-ID: <20071011215017.5896C1E4005@bag.python.org> Author: brett.cannon Date: Thu Oct 11 23:50:17 2007 New Revision: 58417 Modified: python/branches/py3k-importlib/Python/bltinmodule.c Log: Fix a boo-boo from the last merge where I accidentally undid the merged change. Modified: python/branches/py3k-importlib/Python/bltinmodule.c ============================================================================== --- python/branches/py3k-importlib/Python/bltinmodule.c (original) +++ python/branches/py3k-importlib/Python/bltinmodule.c Thu Oct 11 23:50:17 2007 @@ -1787,7 +1787,6 @@ SETBUILTIN("True", Py_True); SETBUILTIN("basestring", &PyBaseString_Type); SETBUILTIN("bool", &PyBool_Type); - SETBUILTIN("buffer", &PyBuffer_Type); SETBUILTIN("memoryview", &PyMemoryView_Type); SETBUILTIN("bytes", &PyBytes_Type); SETBUILTIN("classmethod", &PyClassMethod_Type); From python-3000-checkins at python.org Fri Oct 12 01:08:53 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Fri, 12 Oct 2007 01:08:53 +0200 (CEST) Subject: [Python-3000-checkins] r58418 - python/branches/py3k/Lib/io.py Message-ID: <20071011230853.A94CD1E4018@bag.python.org> Author: brett.cannon Date: Fri Oct 12 01:08:53 2007 New Revision: 58418 Modified: python/branches/py3k/Lib/io.py Log: Remove self-referential import. Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Fri Oct 12 01:08:53 2007 @@ -34,7 +34,6 @@ import sys import codecs import _fileio -import io import warnings # open() uses st_blksize whenever we can @@ -1031,7 +1030,7 @@ if encoding is None: try: encoding = os.device_encoding(buffer.fileno()) - except (AttributeError, io.UnsupportedOperation): + except (AttributeError, UnsupportedOperation): pass if encoding is None: try: From python-3000-checkins at python.org Fri Oct 12 09:20:04 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Fri, 12 Oct 2007 09:20:04 +0200 (CEST) Subject: [Python-3000-checkins] r58429 - in python/branches/py3k-importlib: Include/pythonrun.h NEWS Python/import.c Python/pythonrun.c Message-ID: <20071012072004.948371E401C@bag.python.org> Author: brett.cannon Date: Fri Oct 12 09:20:04 2007 New Revision: 58429 Modified: python/branches/py3k-importlib/Include/pythonrun.h python/branches/py3k-importlib/NEWS python/branches/py3k-importlib/Python/import.c python/branches/py3k-importlib/Python/pythonrun.c Log: Move over to using importlib for __import__. Squirrel away original __import__ in __old_import__. Had to create a dinky file class to handle the case where site.py has not been imported yet and thus has not set __builtins__.open(). Debugging is hard since sys.stderr is not set if site.py cannot be successfully imported, so one must be careful when changing the importlib code. This is not thoroughly tested! But test_import works sans the test requiring warnings; that won't pass until warnings is a built-in module. Modified: python/branches/py3k-importlib/Include/pythonrun.h ============================================================================== --- python/branches/py3k-importlib/Include/pythonrun.h (original) +++ python/branches/py3k-importlib/Include/pythonrun.h Fri Oct 12 09:20:04 2007 @@ -123,7 +123,7 @@ PyAPI_FUNC(PyObject *) _PyBuiltin_Init(void); PyAPI_FUNC(PyObject *) _PySys_Init(void); PyAPI_FUNC(void) _PyImport_Init(void); -PyAPI_FUNC(void) _PyImport_Importlib(void); +PyAPI_FUNC(void) _PyImport_Importlib(PyObject *, PyObject *); PyAPI_FUNC(void) _PyExc_Init(void); PyAPI_FUNC(void) _PyImportHooks_Init(void); PyAPI_FUNC(int) _PyFrame_Init(void); Modified: python/branches/py3k-importlib/NEWS ============================================================================== --- python/branches/py3k-importlib/NEWS (original) +++ python/branches/py3k-importlib/NEWS Fri Oct 12 09:20:04 2007 @@ -1,3 +1,5 @@ +* Store old __import__ into __old_import__ and use importlib. + * Add _r_long() and _w_long() to marshal. * Add _case_ok() to imp. Modified: python/branches/py3k-importlib/Python/import.c ============================================================================== --- python/branches/py3k-importlib/Python/import.c (original) +++ python/branches/py3k-importlib/Python/import.c Fri Oct 12 09:20:04 2007 @@ -525,7 +525,7 @@ void -_PyImport_Importlib(void) +_PyImport_Importlib(PyObject *builtins, PyObject *modules) { const char *importlib_path = Py_GetImportlibPath(); FILE *fp = NULL; @@ -536,6 +536,7 @@ unsigned int x; PyObject *modules_dict = NULL; PyObject *attr = NULL; + PyObject *obj = NULL; /* _importlib must have been found. */ @@ -556,22 +557,26 @@ Py_FatalError("could not initialize _importlib"); /* Get sys.modules so as to extract needed built-in modules. */ - modules_dict = PyImport_GetModuleDict(); for (x = 0; builtin_modules[x] != NULL; x += 1) { PyObject *module; const char *name = builtin_modules[x]; - /* Initialize the built-in module. */ - if (!init_builtin((char *)name)) - Py_FatalError("initialization of a built-in module failed"); - /* Get the module from sys.modules. */ - module = PyDict_GetItemString(modules_dict, name); - if (!module) + /* See if sys.modules already has the module to avoid a re-init. */ + module = PyDict_GetItemString(modules, name); + if (module == NULL) { + /* Initialize the built-in module. */ + if (!init_builtin((char *)name)) + Py_FatalError("initialization of a built-in " + "module failed"); + /* Get the module from sys.modules. */ + module = PyDict_GetItemString(modules, name); + } + if (module == NULL) Py_FatalError("built-in module lost"); /* Add the module to _importlib's globals. */ Py_INCREF(module); if (PyModule_AddObject(importlib, - (strcmp(name, PyOS_MODNAME) ? name : "_os"), + (strcmp(name, PyOS_MODNAME) != 0 ? name : "_os"), module)) Py_FatalError("could not add built-in module to _importlib"); } @@ -583,7 +588,25 @@ if (PyModule_AddObject(importlib, "path_sep", attr) < 0) Py_FatalError("could not add path_sep to _importlib"); + /* Store away old __import__. */ + obj = PyDict_GetItemString(builtins, "__import__"); + if (obj == NULL || PyErr_Occurred()) + Py_FatalError("error getting old __import__"); + if (PyDict_SetItemString(builtins, "__old_import__", obj)) + Py_FatalError("unable to store away old __import__"); + + /* Create an instance of Import and set __import__ to it. */ + modules_dict = PyModule_GetDict(importlib); + attr = PyDict_GetItemString(modules_dict, "Import"); + if (attr == NULL || PyErr_Occurred()) + Py_FatalError("couldn't get _importlib.Import"); + obj = PyObject_CallObject(attr, NULL); + if (obj == NULL) + Py_FatalError("unable to instantiate _importlib.Import"); + if (PyDict_SetItemString(builtins, "__import__", obj)) + Py_FatalError("could not set new __import__"); + Py_DECREF(obj); Py_DECREF(importlib); } Modified: python/branches/py3k-importlib/Python/pythonrun.c ============================================================================== --- python/branches/py3k-importlib/Python/pythonrun.c (original) +++ python/branches/py3k-importlib/Python/pythonrun.c Fri Oct 12 09:20:04 2007 @@ -237,12 +237,12 @@ _PyImportHooks_Init(); - _PyImport_Importlib(); - if (install_sigs) initsigs(); /* Signal handling stuff, including initintr() */ initmain(); /* Module __main__ */ + + _PyImport_Importlib(interp->builtins, interp->modules); if (!Py_NoSiteFlag) initsite(); /* Module site */ From python-3000-checkins at python.org Fri Oct 12 21:13:20 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Fri, 12 Oct 2007 21:13:20 +0200 (CEST) Subject: [Python-3000-checkins] r58438 - in python/branches/py3k/Lib/bsddb: dbshelve.py test/test_dbshelve.py Message-ID: <20071012191320.054571E4022@bag.python.org> Author: gregory.p.smith Date: Fri Oct 12 21:13:19 2007 New Revision: 58438 Modified: python/branches/py3k/Lib/bsddb/dbshelve.py python/branches/py3k/Lib/bsddb/test/test_dbshelve.py Log: Merge r58434: Fixes http://bugs.python.org/issue1233 - bsddb.dbshelve.DBShelf.append was useless due to inverted logic. Also adds a test case for RECNO dbs to test_dbshelve. Modified: python/branches/py3k/Lib/bsddb/dbshelve.py ============================================================================== --- python/branches/py3k/Lib/bsddb/dbshelve.py (original) +++ python/branches/py3k/Lib/bsddb/dbshelve.py Fri Oct 12 21:13:19 2007 @@ -77,6 +77,9 @@ #--------------------------------------------------------------------------- +class DBShelveError(db.DBError): pass + + class DBShelf(DictMixin): """A shelf to hold pickled objects, built upon a bsddb DB object. It automatically pickles/unpickles data objects going to/from the DB. @@ -152,10 +155,10 @@ return self.db.append(data, txn) def append(self, value, txn=None): - if self.get_type() != db.DB_RECNO: + if self.get_type() == db.DB_RECNO: self.append = self.__append return self.append(value, txn=txn) - raise db.DBError("append() only supported when dbshelve opened with filetype=dbshelve.db.DB_RECNO") + raise DBShelveError("append() only supported when dbshelve opened with filetype=dbshelve.db.DB_RECNO") def associate(self, secondaryDB, callback, flags=0): Modified: python/branches/py3k/Lib/bsddb/test/test_dbshelve.py ============================================================================== --- python/branches/py3k/Lib/bsddb/test/test_dbshelve.py (original) +++ python/branches/py3k/Lib/bsddb/test/test_dbshelve.py Fri Oct 12 21:13:19 2007 @@ -50,17 +50,22 @@ except os.error: pass + def mk(self, key): + """Turn key into an appropriate key type for this db""" + # override in child class for RECNO + return key.encode("ascii") + def populateDB(self, d): for x in letters: - d[('S' + x).encode("ascii")] = 10 * x # add a string - d[('I' + x).encode("ascii")] = ord(x) # add an integer - d[('L' + x).encode("ascii")] = [x] * 10 # add a list + d[self.mk('S' + x)] = 10 * x # add a string + d[self.mk('I' + x)] = ord(x) # add an integer + d[self.mk('L' + x)] = [x] * 10 # add a list inst = DataClass() # add an instance inst.S = 10 * x inst.I = ord(x) inst.L = [x] * 10 - d[('O' + x).encode("ascii")] = inst + d[self.mk('O' + x)] = inst # overridable in derived classes to affect how the shelf is created/opened @@ -100,14 +105,14 @@ print("keys:", k) print("stats:", s) - self.assertFalse(d.has_key(b'bad key')) - self.assertTrue(d.has_key(b'IA'), d.keys()) - self.assertTrue(d.has_key(b'OA')) - - d.delete(b'IA') - del d[b'OA'] - self.assertFalse(d.has_key(b'IA')) - self.assertFalse(d.has_key(b'OA')) + self.assertFalse(d.has_key(self.mk('bad key'))) + self.assertTrue(d.has_key(self.mk('IA')), d.keys()) + self.assertTrue(d.has_key(self.mk('OA'))) + + d.delete(self.mk('IA')) + del d[self.mk('OA')] + self.assertFalse(d.has_key(self.mk('IA'))) + self.assertFalse(d.has_key(self.mk('OA'))) self.assertEqual(len(d), l-2) values = [] @@ -129,18 +134,18 @@ for key, value in items: self.checkrec(key, value) - self.assertEqual(d.get(b'bad key'), None) - self.assertEqual(d.get(b'bad key', None), None) - self.assertEqual(d.get(b'bad key', b'a string'), b'a string') - self.assertEqual(d.get(b'bad key', [1, 2, 3]), [1, 2, 3]) + self.assertEqual(d.get(self.mk('bad key')), None) + self.assertEqual(d.get(self.mk('bad key'), None), None) + self.assertEqual(d.get(self.mk('bad key'), b'a string'), b'a string') + self.assertEqual(d.get(self.mk('bad key'), [1, 2, 3]), [1, 2, 3]) d.set_get_returns_none(0) - self.assertRaises(db.DBNotFoundError, d.get, b'bad key') + self.assertRaises(db.DBNotFoundError, d.get, self.mk('bad key')) d.set_get_returns_none(1) - d.put(b'new key', b'new data') - self.assertEqual(d.get(b'new key'), b'new data') - self.assertEqual(d[b'new key'], b'new data') + d.put(self.mk('new key'), b'new data') + self.assertEqual(d.get(self.mk('new key')), b'new data') + self.assertEqual(d[self.mk('new key')], b'new data') @@ -179,12 +184,23 @@ self.assertEqual(count, len(d)) - c.set(b'SS') + c.set(self.mk('SS')) key, value = c.current() self.checkrec(key, value) del c + def test03_append(self): + # NOTE: this is overridden in RECNO subclass, don't change its name. + if verbose: + print('\n', '-=' * 30) + print("Running %s.test03_append..." % self.__class__.__name__) + + self.assertRaises(dbshelve.DBShelveError, + self.d.append, b'unit test was here') + + def checkrec(self, key, value): + # override this in a subclass if the key type is different x = key[1:] if key[0:1] == b'S': self.assertEquals(type(value), str) @@ -289,7 +305,43 @@ #---------------------------------------------------------------------- -# TODO: Add test cases for a DBShelf in a RECNO DB. +# test cases for a DBShelf in a RECNO DB. + +class RecNoShelveTestCase(BasicShelveTestCase): + dbtype = db.DB_RECNO + dbflags = db.DB_CREATE + + def setUp(self): + BasicShelveTestCase.setUp(self) + + # pool to assign integer key values out of + self.key_pool = list(range(1, 5000)) + self.key_map = {} # map string keys to the number we gave them + self.intkey_map = {} # reverse map of above + + def mk(self, key): + if key not in self.key_map: + self.key_map[key] = self.key_pool.pop(0) + self.intkey_map[self.key_map[key]] = key.encode('ascii') + return self.key_map[key] + + def checkrec(self, intkey, value): + key = self.intkey_map[intkey] + BasicShelveTestCase.checkrec(self, key, value) + + def test03_append(self): + if verbose: + print('\n', '-=' * 30) + print("Running %s.test03_append..." % self.__class__.__name__) + + self.d[1] = b'spam' + self.d[5] = b'eggs' + self.assertEqual(6, self.d.append(b'spam')) + self.assertEqual(7, self.d.append(b'baked beans')) + self.assertEqual(b'spam', self.d.get(6)) + self.assertEqual(b'spam', self.d.get(1)) + self.assertEqual(b'baked beans', self.d.get(7)) + self.assertEqual(b'eggs', self.d.get(5)) #---------------------------------------------------------------------- @@ -306,6 +358,7 @@ suite.addTest(unittest.makeSuite(EnvHashShelveTestCase)) suite.addTest(unittest.makeSuite(EnvThreadBTreeShelveTestCase)) suite.addTest(unittest.makeSuite(EnvThreadHashShelveTestCase)) + suite.addTest(unittest.makeSuite(RecNoShelveTestCase)) return suite From python-3000-checkins at python.org Fri Oct 12 21:18:20 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Fri, 12 Oct 2007 21:18:20 +0200 (CEST) Subject: [Python-3000-checkins] r58439 - in python/branches/py3k: Lib/doctest.py Misc/NEWS Message-ID: <20071012191820.51BAF1E4024@bag.python.org> Author: brett.cannon Date: Fri Oct 12 21:18:19 2007 New Revision: 58439 Modified: python/branches/py3k/Lib/doctest.py python/branches/py3k/Misc/NEWS Log: Make _load_testfile() use its encoding argument when __loader__ is defined. Modified: python/branches/py3k/Lib/doctest.py ============================================================================== --- python/branches/py3k/Lib/doctest.py (original) +++ python/branches/py3k/Lib/doctest.py Fri Oct 12 21:18:19 2007 @@ -209,7 +209,8 @@ filename = _module_relative_path(package, filename) if hasattr(package, '__loader__'): if hasattr(package.__loader__, 'get_data'): - return package.__loader__.get_data(filename).decode('utf-8'), filename + return (package.__loader__.get_data(filename).decode(encoding), + filename) return open(filename, encoding=encoding).read(), filename def _indent(s, indent=4): Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Fri Oct 12 21:18:19 2007 @@ -14,6 +14,14 @@ - The `hotshot` profiler has been removed; use `cProfile` instead. +Library +------- + +- When loading an external file using testfile(), the passed-in encoding + argument was being ignored if __loader__ is defined and forcing the source to + be UTF-8. + + What's New in Python 3.0a1? ========================== From python-3000-checkins at python.org Sat Oct 13 00:05:15 2007 From: python-3000-checkins at python.org (travis.oliphant) Date: Sat, 13 Oct 2007 00:05:15 +0200 (CEST) Subject: [Python-3000-checkins] r58441 - python/branches/py3k/Modules/arraymodule.c Message-ID: <20071012220515.96B161E4016@bag.python.org> Author: travis.oliphant Date: Sat Oct 13 00:05:15 2007 New Revision: 58441 Modified: python/branches/py3k/Modules/arraymodule.c Log: Fix Issue 1268 with the array module by backing-out the 'w' addition. All builds will continue to use 'u' for unicode. Modified: python/branches/py3k/Modules/arraymodule.c ============================================================================== --- python/branches/py3k/Modules/arraymodule.c (original) +++ python/branches/py3k/Modules/arraymodule.c Sat Oct 13 00:05:15 2007 @@ -40,14 +40,6 @@ static PyTypeObject Arraytype; -#ifdef Py_UNICODE_WIDE -#define PyArr_UNI 'w' -#define PyArr_UNISTR "w" -#else -#define PyArr_UNI 'u' -#define PyArr_UNISTR "u" -#endif - #define array_Check(op) PyObject_TypeCheck(op, &Arraytype) #define array_CheckExact(op) (Py_Type(op) == &Arraytype) @@ -193,12 +185,14 @@ return 0; } + static PyObject * h_getitem(arrayobject *ap, Py_ssize_t i) { return PyInt_FromLong((long) ((short *)ap->ob_item)[i]); } + static int h_setitem(arrayobject *ap, Py_ssize_t i, PyObject *v) { @@ -389,9 +383,9 @@ /* Description of types */ static struct arraydescr descriptors[] = { - {'b', sizeof(char), b_getitem, b_setitem, "b"}, - {'B', sizeof(char), BB_getitem, BB_setitem, "B"}, - {PyArr_UNI, sizeof(Py_UNICODE), u_getitem, u_setitem, PyArr_UNISTR}, + {'b', 1, b_getitem, b_setitem, "b"}, + {'B', 1, BB_getitem, BB_setitem, "B"}, + {'u', sizeof(Py_UNICODE), u_getitem, u_setitem, "U"}, {'h', sizeof(short), h_getitem, h_setitem, "h"}, {'H', sizeof(short), HH_getitem, HH_setitem, "H"}, {'i', sizeof(int), i_getitem, i_setitem, "i"}, @@ -1418,11 +1412,13 @@ { Py_UNICODE *ustr; Py_ssize_t n; + char typecode; if (!PyArg_ParseTuple(args, "u#:fromunicode", &ustr, &n)) return NULL; - if (self->ob_descr->typecode != PyArr_UNI) { - PyErr_SetString(PyExc_ValueError, + typecode = self->ob_descr->typecode; + if ((typecode != 'u')) { + PyErr_SetString(PyExc_ValueError, "fromunicode() may only be called on " "unicode type arrays"); return NULL; @@ -1457,9 +1453,11 @@ static PyObject * array_tounicode(arrayobject *self, PyObject *unused) { - if (self->ob_descr->typecode != PyArr_UNI) { + char typecode; + typecode = self->ob_descr->typecode; + if ((typecode != 'u')) { PyErr_SetString(PyExc_ValueError, - "tounicode() may only be called on unicode type arrays"); + "tounicode() may only be called on unicode type arrays"); return NULL; } return PyUnicode_FromUnicode((Py_UNICODE *) self->ob_item, Py_Size(self)); @@ -1560,7 +1558,7 @@ if (len == 0) { return PyUnicode_FromFormat("array('%c')", typecode); } - if (typecode == PyArr_UNI) + if ((typecode == 'u')) v = array_tounicode(a, NULL); else v = array_tolist(a, NULL); @@ -1864,7 +1862,7 @@ if (!(initial == NULL || PyList_Check(initial) || PyBytes_Check(initial) || PyTuple_Check(initial) - || (c == PyArr_UNI && PyUnicode_Check(initial)))) { + || ((c=='u') && PyUnicode_Check(initial)))) { it = PyObject_GetIter(initial); if (it == NULL) return NULL; @@ -1966,7 +1964,7 @@ Type code C Type Minimum size in bytes \n\ 'b' signed integer 1 \n\ 'B' unsigned integer 1 \n\ - 'u' Unicode character 2 \n\ + 'u' Unicode character 2 (see note) \n\ 'h' signed integer 2 \n\ 'H' unsigned integer 2 \n\ 'i' signed integer 2 \n\ @@ -1977,6 +1975,9 @@ 'f' floating point 4 \n\ 'd' floating point 8 \n\ \n\ +NOTE: The 'u' typecode corresponds to Python's unicode character. On \n\ +narrow builds this is 2-bytes on wide builds this is 4-bytes.\n\ +\n\ The constructor is:\n\ \n\ array(typecode [, initializer]) -- create a new array\n\ @@ -2168,6 +2169,10 @@ initarray(void) { PyObject *m; + PyObject *typecodes; + Py_ssize_t size = 0; + register Py_UNICODE *p; + struct arraydescr *descr; if (PyType_Ready(&Arraytype) < 0) return; @@ -2180,5 +2185,18 @@ PyModule_AddObject(m, "ArrayType", (PyObject *)&Arraytype); Py_INCREF((PyObject *)&Arraytype); PyModule_AddObject(m, "array", (PyObject *)&Arraytype); + + for (descr=descriptors; descr->typecode != '\0'; descr++) { + size++; + } + + typecodes = PyUnicode_FromStringAndSize(NULL, size); + p = PyUnicode_AS_UNICODE(typecodes); + for (descr = descriptors; descr->typecode != '\0'; descr++) { + *p++ = (char)descr->typecode; + } + + PyModule_AddObject(m, "typecodes", (PyObject *)typecodes); + /* No need to check the error here, the caller will do that */ } From python-3000-checkins at python.org Sat Oct 13 00:06:37 2007 From: python-3000-checkins at python.org (travis.oliphant) Date: Sat, 13 Oct 2007 00:06:37 +0200 (CEST) Subject: [Python-3000-checkins] r58442 - python/branches/py3k/configure python/branches/py3k/configure.in python/branches/py3k/pyconfig.h.in Message-ID: <20071012220637.C26901E4012@bag.python.org> Author: travis.oliphant Date: Sat Oct 13 00:06:37 2007 New Revision: 58442 Modified: python/branches/py3k/configure python/branches/py3k/configure.in python/branches/py3k/pyconfig.h.in Log: Add macros for checking for long double and defining the size of long double on the platform. Ctypes and Struct module will both need them. Modified: python/branches/py3k/configure ============================================================================== --- python/branches/py3k/configure (original) +++ python/branches/py3k/configure Sat Oct 13 00:06:37 2007 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 57931 . +# From configure.in Revision: 58054 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.61 for python 3.0. # @@ -10589,6 +10589,468 @@ fi +{ echo "$as_me:$LINENO: checking for long double support" >&5 +echo $ECHO_N "checking for long double support... $ECHO_C" >&6; } +have_long_double=no +cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ + +int +main () +{ +long double x; x = (long double)0; + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + + +cat >>confdefs.h <<\_ACEOF +#define HAVE_LONG_DOUBLE 1 +_ACEOF + + have_long_double=yes + +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +{ echo "$as_me:$LINENO: result: $have_long_double" >&5 +echo "${ECHO_T}$have_long_double" >&6; } +if test "$have_long_double" = yes ; then +{ echo "$as_me:$LINENO: checking for long double" >&5 +echo $ECHO_N "checking for long double... $ECHO_C" >&6; } +if test "${ac_cv_type_long_double+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default +typedef long double ac__type_new_; +int +main () +{ +if ((ac__type_new_ *) 0) + return 0; +if (sizeof (ac__type_new_)) + return 0; + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_cv_type_long_double=yes +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_cv_type_long_double=no +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi +{ echo "$as_me:$LINENO: result: $ac_cv_type_long_double" >&5 +echo "${ECHO_T}$ac_cv_type_long_double" >&6; } + +# The cast to long int works around a bug in the HP C Compiler +# version HP92453-01 B.11.11.23709.GP, which incorrectly rejects +# declarations like `int a3[[(sizeof (unsigned char)) >= 0]];'. +# This bug is HP SR number 8606223364. +{ echo "$as_me:$LINENO: checking size of long double" >&5 +echo $ECHO_N "checking size of long double... $ECHO_C" >&6; } +if test "${ac_cv_sizeof_long_double+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + if test "$cross_compiling" = yes; then + # Depending upon the size, compute the lo and hi bounds. +cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) >= 0)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_lo=0 ac_mid=0 + while :; do + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) <= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=$ac_mid; break +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo=`expr $ac_mid + 1` + if test $ac_lo -le $ac_mid; then + ac_lo= ac_hi= + break + fi + ac_mid=`expr 2 '*' $ac_mid + 1` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + done +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) < 0)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=-1 ac_mid=-1 + while :; do + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) >= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_lo=$ac_mid; break +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_hi=`expr '(' $ac_mid ')' - 1` + if test $ac_mid -le $ac_hi; then + ac_lo= ac_hi= + break + fi + ac_mid=`expr 2 '*' $ac_mid` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + done +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo= ac_hi= +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +# Binary search between lo and hi bounds. +while test "x$ac_lo" != "x$ac_hi"; do + ac_mid=`expr '(' $ac_hi - $ac_lo ')' / 2 + $ac_lo` + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) <= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=$ac_mid +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo=`expr '(' $ac_mid ')' + 1` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +done +case $ac_lo in +?*) ac_cv_sizeof_long_double=$ac_lo;; +'') if test "$ac_cv_type_long_double" = yes; then + { { echo "$as_me:$LINENO: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&5 +echo "$as_me: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&2;} + { (exit 77); exit 77; }; } + else + ac_cv_sizeof_long_double=0 + fi ;; +esac +else + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +static long int longval () { return (long int) (sizeof (ac__type_sizeof_)); } +static unsigned long int ulongval () { return (long int) (sizeof (ac__type_sizeof_)); } +#include +#include +int +main () +{ + + FILE *f = fopen ("conftest.val", "w"); + if (! f) + return 1; + if (((long int) (sizeof (ac__type_sizeof_))) < 0) + { + long int i = longval (); + if (i != ((long int) (sizeof (ac__type_sizeof_)))) + return 1; + fprintf (f, "%ld\n", i); + } + else + { + unsigned long int i = ulongval (); + if (i != ((long int) (sizeof (ac__type_sizeof_)))) + return 1; + fprintf (f, "%lu\n", i); + } + return ferror (f) || fclose (f) != 0; + + ; + return 0; +} +_ACEOF +rm -f conftest$ac_exeext +if { (ac_try="$ac_link" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_link") 2>&5 + ac_status=$? + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { ac_try='./conftest$ac_exeext' + { (case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_try") 2>&5 + ac_status=$? + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); }; }; then + ac_cv_sizeof_long_double=`cat conftest.val` +else + echo "$as_me: program exited with status $ac_status" >&5 +echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + +( exit $ac_status ) +if test "$ac_cv_type_long_double" = yes; then + { { echo "$as_me:$LINENO: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&5 +echo "$as_me: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&2;} + { (exit 77); exit 77; }; } + else + ac_cv_sizeof_long_double=0 + fi +fi +rm -f core *.core core.conftest.* gmon.out bb.out conftest$ac_exeext conftest.$ac_objext conftest.$ac_ext +fi +rm -f conftest.val +fi +{ echo "$as_me:$LINENO: result: $ac_cv_sizeof_long_double" >&5 +echo "${ECHO_T}$ac_cv_sizeof_long_double" >&6; } + + + +cat >>confdefs.h <<_ACEOF +#define SIZEOF_LONG_DOUBLE $ac_cv_sizeof_long_double +_ACEOF + + +fi + + { echo "$as_me:$LINENO: checking for _Bool support" >&5 echo $ECHO_N "checking for _Bool support... $ECHO_C" >&6; } have_c99_bool=no Modified: python/branches/py3k/configure.in ============================================================================== --- python/branches/py3k/configure.in (original) +++ python/branches/py3k/configure.in Sat Oct 13 00:06:37 2007 @@ -1197,6 +1197,18 @@ AC_CHECK_SIZEOF(long long, 8) fi +AC_MSG_CHECKING(for long double support) +have_long_double=no +AC_TRY_COMPILE([], [long double x; x = (long double)0;], [ + AC_DEFINE(HAVE_LONG_DOUBLE, 1, [Define this if you have the type long double.]) + have_long_double=yes +]) +AC_MSG_RESULT($have_long_double) +if test "$have_long_double" = yes ; then +AC_CHECK_SIZEOF(long double, 16) +fi + + AC_MSG_CHECKING(for _Bool support) have_c99_bool=no AC_TRY_COMPILE([], [_Bool x; x = (_Bool)0;], [ Modified: python/branches/py3k/pyconfig.h.in ============================================================================== --- python/branches/py3k/pyconfig.h.in (original) +++ python/branches/py3k/pyconfig.h.in Sat Oct 13 00:06:37 2007 @@ -330,6 +330,9 @@ /* Define to 1 if you have the header file. */ #undef HAVE_LINUX_NETLINK_H +/* Define this if you have the type long double. */ +#undef HAVE_LONG_DOUBLE + /* Define this if you have the type long long. */ #undef HAVE_LONG_LONG @@ -824,6 +827,9 @@ /* The size of `long', as computed by sizeof. */ #undef SIZEOF_LONG +/* The size of `long double', as computed by sizeof. */ +#undef SIZEOF_LONG_DOUBLE + /* The size of `long long', as computed by sizeof. */ #undef SIZEOF_LONG_LONG From python-3000-checkins at python.org Sat Oct 13 01:27:53 2007 From: python-3000-checkins at python.org (travis.oliphant) Date: Sat, 13 Oct 2007 01:27:53 +0200 (CEST) Subject: [Python-3000-checkins] r58443 - in python/branches/py3k: Include/memoryobject.h Include/object.h Objects/abstract.c Objects/memoryobject.c Message-ID: <20071012232753.AC6C41E4023@bag.python.org> Author: travis.oliphant Date: Sat Oct 13 01:27:53 2007 New Revision: 58443 Modified: python/branches/py3k/Include/memoryobject.h python/branches/py3k/Include/object.h python/branches/py3k/Objects/abstract.c python/branches/py3k/Objects/memoryobject.c Log: Fix problems with memoryview object. There is still more to do to finish PEP 3118. The memory-view object needs to be fleshed out and the struct module needs to be modified. Modified: python/branches/py3k/Include/memoryobject.h ============================================================================== --- python/branches/py3k/Include/memoryobject.h (original) +++ python/branches/py3k/Include/memoryobject.h Sat Oct 13 01:27:53 2007 @@ -8,9 +8,9 @@ #endif typedef struct { - PyObject_HEAD - PyObject *base; - Py_buffer view; + PyObject_HEAD + PyObject *base; + Py_buffer view; } PyMemoryViewObject; @@ -21,39 +21,40 @@ #define Py_END_OF_MEMORY (-1) -PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, int buffertype, - char fort); +PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, + int buffertype, + char fort); /* Return a contiguous chunk of memory representing the buffer from an object in a memory view object. If a copy is made then the - base object for the memory view will be a *new* bytes object. - - Otherwise, the base-object will be the object itself and no - data-copying will be done. - - The buffertype argument can be PyBUF_READ, PyBUF_WRITE, - PyBUF_UPDATEIFCOPY to determine whether the returned buffer - should be READONLY, WRITABLE, or set to update the - original buffer if a copy must be made. If buffertype is - PyBUF_WRITE and the buffer is not contiguous an error will - be raised. In this circumstance, the user can use - PyBUF_UPDATEIFCOPY to ensure that a a writable temporary - contiguous buffer is returned. The contents of this - contiguous buffer will be copied back into the original - object after the memoryview object is deleted as long as - the original object is writable and allows setting its - memory to "readonly". If this is not allowed by the - original object, then a BufferError is raised. + base object for the memory view will be a *new* bytes object. - If the object is multi-dimensional and if fortran is 'F', - the first dimension of the underlying array will vary the - fastest in the buffer. If fortran is 'C', then the last - dimension will vary the fastest (C-style contiguous). If - fortran is 'A', then it does not matter and you will get - whatever the object decides is more efficient. + Otherwise, the base-object will be the object itself and no + data-copying will be done. - A new reference is returned that must be DECREF'd when finished. - */ + The buffertype argument can be PyBUF_READ, PyBUF_WRITE, + PyBUF_SHADOW to determine whether the returned buffer + should be READONLY, WRITABLE, or set to update the + original buffer if a copy must be made. If buffertype is + PyBUF_WRITE and the buffer is not contiguous an error will + be raised. In this circumstance, the user can use + PyBUF_SHADOW to ensure that a a writable temporary + contiguous buffer is returned. The contents of this + contiguous buffer will be copied back into the original + object after the memoryview object is deleted as long as + the original object is writable and allows setting an + exclusive write lock. If this is not allowed by the + original object, then a BufferError is raised. + + If the object is multi-dimensional and if fortran is 'F', + the first dimension of the underlying array will vary the + fastest in the buffer. If fortran is 'C', then the last + dimension will vary the fastest (C-style contiguous). If + fortran is 'A', then it does not matter and you will get + whatever the object decides is more efficient. + + A new reference is returned that must be DECREF'd when finished. + */ PyAPI_FUNC(PyObject *) PyMemoryView_FromObject(PyObject *base); Modified: python/branches/py3k/Include/object.h ============================================================================== --- python/branches/py3k/Include/object.h (original) +++ python/branches/py3k/Include/object.h Sat Oct 13 01:27:53 2007 @@ -164,7 +164,7 @@ #define PyBUF_WRITABLE 0x0002 /* we used to include an E, backwards compatible alias */ #define PyBUF_WRITEABLE PyBUF_WRITABLE -#define PyBUF_LOCKDATA 0x0004 +#define PyBUF_LOCK 0x0004 #define PyBUF_FORMAT 0x0008 #define PyBUF_ND 0x0010 #define PyBUF_STRIDES (0x0020 | PyBUF_ND) @@ -175,19 +175,25 @@ #define PyBUF_CONTIG (PyBUF_ND | PyBUF_WRITABLE) #define PyBUF_CONTIG_RO (PyBUF_ND) -#define PyBUF_CONTIG_LCK (PyBUF_ND | PyBUF_LOCKDATA) +#define PyBUF_CONTIG_LCK (PyBUF_ND | PyBUF_LOCK) +#define PyBUF_CONTIG_XLCK (PyBUF_ND | PyBUF_LOCK | PyBUF_WRITABLE) #define PyBUF_STRIDED (PyBUF_STRIDES | PyBUF_WRITABLE) #define PyBUF_STRIDED_RO (PyBUF_STRIDES) -#define PyBUF_STRIDED_LCK (PyBUF_STRIDES | PyBUF_LOCKDATA) +#define PyBUF_STRIDED_LCK (PyBUF_STRIDES | PyBUF_LOCK) +#define PyBUF_STRIDED_XLCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_WRITABLE) #define PyBUF_RECORDS (PyBUF_STRIDES | PyBUF_WRITABLE | PyBUF_FORMAT) #define PyBUF_RECORDS_RO (PyBUF_STRIDES | PyBUF_FORMAT) -#define PyBUF_RECORDS_LCK (PyBUF_STRIDES | PyBUF_LOCKDATA | PyBUF_FORMAT) +#define PyBUF_RECORDS_LCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_FORMAT) +#define PyBUF_RECORDS_XLCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_WRITABLE \ + | PyBUF_FORMAT) #define PyBUF_FULL (PyBUF_INDIRECT | PyBUF_WRITABLE | PyBUF_FORMAT) #define PyBUF_FULL_RO (PyBUF_INDIRECT | PyBUF_FORMAT) -#define PyBUF_FULL_LCK (PyBUF_INDIRECT | PyBUF_LOCKDATA | PyBUF_FORMAT) +#define PyBUF_FULL_LCK (PyBUF_INDIRECT | PyBUF_LOCK | PyBUF_FORMAT) +#define PyBUF_FULL_XLCK (PyBUF_INDIRECT | PyBUF_LOCK | PyBUF_WRITABLE \ + | PyBUF_FORMAT) #define PyBUF_READ 0x100 Modified: python/branches/py3k/Objects/abstract.c ============================================================================== --- python/branches/py3k/Objects/abstract.c (original) +++ python/branches/py3k/Objects/abstract.c Sat Oct 13 01:27:53 2007 @@ -220,11 +220,11 @@ */ int PyObject_AsCharBuffer(PyObject *obj, - const char **buffer, - Py_ssize_t *buffer_len) + const char **buffer, + Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -235,30 +235,30 @@ PyErr_SetString(PyExc_TypeError, "expected an object with the buffer interface"); return -1; - } - if ((*pb->bf_getbuffer)(obj, &view, PyBUF_CHARACTER)) return -1; + } + if ((*pb->bf_getbuffer)(obj, &view, PyBUF_CHARACTER)) return -1; *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } int PyObject_CheckReadBuffer(PyObject *obj) { - PyBufferProcs *pb = obj->ob_type->tp_as_buffer; + PyBufferProcs *pb = obj->ob_type->tp_as_buffer; if (pb == NULL || pb->bf_getbuffer == NULL) - return 0; - if ((*pb->bf_getbuffer)(obj, NULL, PyBUF_SIMPLE) == -1) { - PyErr_Clear(); - return 0; - } - if (*pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, NULL); + return 0; + if ((*pb->bf_getbuffer)(obj, NULL, PyBUF_SIMPLE) == -1) { + PyErr_Clear(); + return 0; + } + if (*pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, NULL); return 1; } @@ -267,7 +267,7 @@ Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -275,18 +275,18 @@ } pb = obj->ob_type->tp_as_buffer; if (pb == NULL || - pb->bf_getbuffer == NULL) { + pb->bf_getbuffer == NULL) { PyErr_SetString(PyExc_TypeError, "expected an object with a buffer interface"); return -1; } - if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; + if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } @@ -295,7 +295,7 @@ Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -303,17 +303,17 @@ } pb = obj->ob_type->tp_as_buffer; if (pb == NULL || - pb->bf_getbuffer == NULL || - ((*pb->bf_getbuffer)(obj, &view, PyBUF_WRITABLE) != 0)) { + pb->bf_getbuffer == NULL || + ((*pb->bf_getbuffer)(obj, &view, PyBUF_WRITABLE) != 0)) { PyErr_SetString(PyExc_TypeError, - "expected an object with a writable buffer interface"); + "expected an object with a writable buffer interface"); return -1; } *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } @@ -322,128 +322,128 @@ int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) { - if (!PyObject_CheckBuffer(obj)) { - PyErr_SetString(PyExc_TypeError, - "object does not have the buffer interface"); - return -1; - } - return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags); + if (!PyObject_CheckBuffer(obj)) { + PyErr_SetString(PyExc_TypeError, + "object does not have the buffer interface"); + return -1; + } + return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags); } void PyObject_ReleaseBuffer(PyObject *obj, Py_buffer *view) { - if (obj->ob_type->tp_as_buffer != NULL && - obj->ob_type->tp_as_buffer->bf_releasebuffer != NULL) { - (*(obj->ob_type->tp_as_buffer->bf_releasebuffer))(obj, view); - } + if (obj->ob_type->tp_as_buffer != NULL && + obj->ob_type->tp_as_buffer->bf_releasebuffer != NULL) { + (*(obj->ob_type->tp_as_buffer->bf_releasebuffer))(obj, view); + } } static int _IsFortranContiguous(Py_buffer *view) { - Py_ssize_t sd, dim; - int i; - - if (view->ndim == 0) return 1; - if (view->strides == NULL) return (view->ndim == 1); - - sd = view->itemsize; - if (view->ndim == 1) return (view->shape[0] == 1 || - sd == view->strides[0]); - for (i=0; indim; i++) { - dim = view->shape[i]; - if (dim == 0) return 1; - if (view->strides[i] != sd) return 0; - sd *= dim; - } - return 1; + Py_ssize_t sd, dim; + int i; + + if (view->ndim == 0) return 1; + if (view->strides == NULL) return (view->ndim == 1); + + sd = view->itemsize; + if (view->ndim == 1) return (view->shape[0] == 1 || + sd == view->strides[0]); + for (i=0; indim; i++) { + dim = view->shape[i]; + if (dim == 0) return 1; + if (view->strides[i] != sd) return 0; + sd *= dim; + } + return 1; } static int _IsCContiguous(Py_buffer *view) { - Py_ssize_t sd, dim; - int i; - - if (view->ndim == 0) return 1; - if (view->strides == NULL) return 1; - - sd = view->itemsize; - if (view->ndim == 1) return (view->shape[0] == 1 || - sd == view->strides[0]); - for (i=view->ndim-1; i>=0; i--) { - dim = view->shape[i]; - if (dim == 0) return 1; - if (view->strides[i] != sd) return 0; - sd *= dim; - } - return 1; + Py_ssize_t sd, dim; + int i; + + if (view->ndim == 0) return 1; + if (view->strides == NULL) return 1; + + sd = view->itemsize; + if (view->ndim == 1) return (view->shape[0] == 1 || + sd == view->strides[0]); + for (i=view->ndim-1; i>=0; i--) { + dim = view->shape[i]; + if (dim == 0) return 1; + if (view->strides[i] != sd) return 0; + sd *= dim; + } + return 1; } int PyBuffer_IsContiguous(Py_buffer *view, char fort) { - if (view->suboffsets != NULL) return 0; + if (view->suboffsets != NULL) return 0; - if (fort == 'C') - return _IsCContiguous(view); - else if (fort == 'F') - return _IsFortranContiguous(view); - else if (fort == 'A') - return (_IsCContiguous(view) || _IsFortranContiguous(view)); - return 0; + if (fort == 'C') + return _IsCContiguous(view); + else if (fort == 'F') + return _IsFortranContiguous(view); + else if (fort == 'A') + return (_IsCContiguous(view) || _IsFortranContiguous(view)); + return 0; } void* PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) { - char* pointer; - int i; - pointer = (char *)view->buf; - for (i = 0; i < view->ndim; i++) { - pointer += view->strides[i]*indices[i]; - if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { - pointer = *((char**)pointer) + view->suboffsets[i]; - } - } - return (void*)pointer; + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; } void _add_one_to_index_F(int nd, Py_ssize_t *index, Py_ssize_t *shape) { - int k; - - for (k=0; k=0; k--) { - if (index[k] < shape[k]-1) { - index[k]++; - break; - } - else { - index[k] = 0; - } - } + for (k=nd-1; k>=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } } /* view is not checked for consistency in either of these. It is @@ -454,235 +454,235 @@ int PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) { - int k; - void (*addone)(int, Py_ssize_t *, Py_ssize_t *); - Py_ssize_t *indices, elements; - char *dest, *ptr; - - if (len > view->len) { - len = view->len; - } - - if (PyBuffer_IsContiguous(view, fort)) { - /* simplest copy is all that is needed */ - memcpy(buf, view->buf, len); - return 0; - } + int k; + void (*addone)(int, Py_ssize_t *, Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } - /* Otherwise a more elaborate scheme is needed */ - + /* Otherwise a more elaborate scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); - if (indices == NULL) { - PyErr_NoMemory(); - return -1; - } - for (k=0; kndim;k++) { - indices[k] = 0; - } - - if (fort == 'F') { - addone = _add_one_to_index_F; - } - else { - addone = _add_one_to_index_C; - } - dest = buf; - /* XXX : This is not going to be the fastest code in the world - several optimizations are possible. - */ - elements = len / view->itemsize; - while (elements--) { - addone(view->ndim, indices, view->shape); - ptr = PyBuffer_GetPointer(view, indices); - memcpy(dest, ptr, view->itemsize); - dest += view->itemsize; - } - PyMem_Free(indices); - return 0; + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _add_one_to_index_F; + } + else { + addone = _add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + addone(view->ndim, indices, view->shape); + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + } + PyMem_Free(indices); + return 0; } int PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) { - int k; - void (*addone)(int, Py_ssize_t *, Py_ssize_t *); - Py_ssize_t *indices, elements; - char *src, *ptr; - - if (len > view->len) { - len = view->len; - } - - if (PyBuffer_IsContiguous(view, fort)) { - /* simplest copy is all that is needed */ - memcpy(view->buf, buf, len); - return 0; - } + int k; + void (*addone)(int, Py_ssize_t *, Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } - /* Otherwise a more elaborate scheme is needed */ - + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); - if (indices == NULL) { - PyErr_NoMemory(); - return -1; - } - for (k=0; kndim;k++) { - indices[k] = 0; - } - - if (fort == 'F') { - addone = _add_one_to_index_F; - } - else { - addone = _add_one_to_index_C; - } - src = buf; - /* XXX : This is not going to be the fastest code in the world - several optimizations are possible. - */ - elements = len / view->itemsize; - while (elements--) { - addone(view->ndim, indices, view->shape); - ptr = PyBuffer_GetPointer(view, indices); - memcpy(ptr, src, view->itemsize); - src += view->itemsize; - } - - PyMem_Free(indices); - return 0; + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _add_one_to_index_F; + } + else { + addone = _add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + addone(view->ndim, indices, view->shape); + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + } + + PyMem_Free(indices); + return 0; } int PyObject_CopyData(PyObject *dest, PyObject *src) { - Py_buffer view_dest, view_src; - int k; - Py_ssize_t *indices, elements; - char *dptr, *sptr; - - if (!PyObject_CheckBuffer(dest) || - !PyObject_CheckBuffer(src)) { - PyErr_SetString(PyExc_TypeError, - "both destination and source must have the "\ - "buffer interface"); - return -1; - } - - if (PyObject_GetBuffer(dest, &view_dest, PyBUF_FULL) != 0) return -1; - if (PyObject_GetBuffer(src, &view_src, PyBUF_FULL_RO) != 0) { - PyObject_ReleaseBuffer(dest, &view_dest); - return -1; - } - - if (view_dest.len < view_src.len) { - PyErr_SetString(PyExc_BufferError, - "destination is too small to receive data from source"); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return -1; - } - - if ((PyBuffer_IsContiguous(&view_dest, 'C') && - PyBuffer_IsContiguous(&view_src, 'C')) || - (PyBuffer_IsContiguous(&view_dest, 'F') && - PyBuffer_IsContiguous(&view_src, 'F'))) { - /* simplest copy is all that is needed */ - memcpy(view_dest.buf, view_src.buf, view_src.len); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return 0; - } + Py_buffer view_dest, view_src; + int k; + Py_ssize_t *indices, elements; + char *dptr, *sptr; + + if (!PyObject_CheckBuffer(dest) || + !PyObject_CheckBuffer(src)) { + PyErr_SetString(PyExc_TypeError, + "both destination and source must have the "\ + "buffer interface"); + return -1; + } + + if (PyObject_GetBuffer(dest, &view_dest, PyBUF_FULL) != 0) return -1; + if (PyObject_GetBuffer(src, &view_src, PyBUF_FULL_RO) != 0) { + PyObject_ReleaseBuffer(dest, &view_dest); + return -1; + } + + if (view_dest.len < view_src.len) { + PyErr_SetString(PyExc_BufferError, + "destination is too small to receive data from source"); + PyObject_ReleaseBuffer(dest, &view_dest); + PyObject_ReleaseBuffer(src, &view_src); + return -1; + } + + if ((PyBuffer_IsContiguous(&view_dest, 'C') && + PyBuffer_IsContiguous(&view_src, 'C')) || + (PyBuffer_IsContiguous(&view_dest, 'F') && + PyBuffer_IsContiguous(&view_src, 'F'))) { + /* simplest copy is all that is needed */ + memcpy(view_dest.buf, view_src.buf, view_src.len); + PyObject_ReleaseBuffer(dest, &view_dest); + PyObject_ReleaseBuffer(src, &view_src); + return 0; + } - /* Otherwise a more elaborate copy scheme is needed */ - + /* Otherwise a more elaborate copy scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*view_src.ndim); - if (indices == NULL) { - PyErr_NoMemory(); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return -1; - } - for (k=0; k=0; k--) { - strides[k] = sd; - sd *= shape[k]; - } - } - return; + int k; + Py_ssize_t sd; + + sd = itemsize; + if (fort == 'F') { + for (k=0; k=0; k--) { + strides[k] = sd; + sd *= shape[k]; + } + } + return; } int PyBuffer_FillInfo(Py_buffer *view, void *buf, Py_ssize_t len, - int readonly, int flags) -{ - if (view == NULL) return 0; - if (((flags & PyBUF_LOCKDATA) == PyBUF_LOCKDATA) && - readonly != -1) { - PyErr_SetString(PyExc_BufferError, - "Cannot make this object read-only."); - return -1; - } - if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && - readonly == 1) { - PyErr_SetString(PyExc_BufferError, - "Object is not writable."); - return -1; - } - - view->buf = buf; - view->len = len; - view->readonly = readonly; - view->itemsize = 1; - view->format = NULL; - if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) - view->format = "B"; - view->ndim = 1; - view->shape = NULL; - if ((flags & PyBUF_ND) == PyBUF_ND) - view->shape = &(view->len); - view->strides = NULL; - if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) - view->strides = &(view->itemsize); - view->suboffsets = NULL; - view->internal = NULL; - return 0; + int readonly, int flags) +{ + if (view == NULL) return 0; + if (((flags & PyBUF_LOCK) == PyBUF_LOCK) && + readonly >= 0) { + PyErr_SetString(PyExc_BufferError, + "Cannot lock this object."); + return -1; + } + if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && + (readonly == 1)) { + PyErr_SetString(PyExc_BufferError, + "Object is not writable."); + return -1; + } + + view->buf = buf; + view->len = len; + view->readonly = readonly; + view->itemsize = 1; + view->format = NULL; + if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) + view->format = "B"; + view->ndim = 1; + view->shape = NULL; + if ((flags & PyBUF_ND) == PyBUF_ND) + view->shape = &(view->len); + view->strides = NULL; + if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) + view->strides = &(view->itemsize); + view->suboffsets = NULL; + view->internal = NULL; + return 0; } /* Operations on numbers */ @@ -1025,7 +1025,7 @@ Py_DECREF(result); if (m != NULL) { binaryfunc f = NULL; - f = m->sq_inplace_concat; + f = m->sq_inplace_concat; if (f == NULL) f = m->sq_concat; if (f != NULL) @@ -1304,8 +1304,8 @@ PyObject *res = m->nb_float(o); if (res && !PyFloat_Check(res)) { PyErr_Format(PyExc_TypeError, - "__float__ returned non-float (type %.200s)", - res->ob_type->tp_name); + "__float__ returned non-float (type %.200s)", + res->ob_type->tp_name); Py_DECREF(res); return NULL; } @@ -1384,7 +1384,7 @@ return m->sq_concat(s, o); /* Instances of user classes defining an __add__() method only - have an nb_add slot, not an sq_concat slot. So we fall back + have an nb_add slot, not an sq_concat slot. So we fall back to nb_add if both arguments appear to be sequences. */ if (PySequence_Check(s) && PySequence_Check(o)) { PyObject *result = binary_op1(s, o, NB_SLOT(nb_add)); @@ -1629,7 +1629,7 @@ PySequence_Tuple(PyObject *v) { PyObject *it; /* iter(v) */ - Py_ssize_t n; /* guess for result tuple size */ + Py_ssize_t n; /* guess for result tuple size */ PyObject *result; Py_ssize_t j; @@ -1662,7 +1662,7 @@ return NULL; } PyErr_Clear(); - n = 10; /* arbitrary */ + n = 10; /* arbitrary */ } result = PyTuple_New(n); if (result == NULL) @@ -1718,7 +1718,7 @@ PySequence_List(PyObject *v) { PyObject *result; /* result list */ - PyObject *rv; /* return value from PyList_Extend */ + PyObject *rv; /* return value from PyList_Extend */ if (v == NULL) return null_error(); @@ -1749,7 +1749,7 @@ return v; } - it = PyObject_GetIter(v); + it = PyObject_GetIter(v); if (it == NULL) { if (PyErr_ExceptionMatches(PyExc_TypeError)) PyErr_SetString(PyExc_TypeError, m); @@ -1763,9 +1763,9 @@ } /* Iterate over seq. Result depends on the operation: - PY_ITERSEARCH_COUNT: -1 if error, else # of times obj appears in seq. - PY_ITERSEARCH_INDEX: 0-based index of first occurence of obj in seq; - set ValueError and return -1 if none found; also return -1 on error. + PY_ITERSEARCH_COUNT: -1 if error, else # of times obj appears in seq. + PY_ITERSEARCH_INDEX: 0-based index of first occurence of obj in seq; + set ValueError and return -1 if none found; also return -1 on error. Py_ITERSEARCH_CONTAINS: return 1 if obj in seq, else 0; -1 on error. */ Py_ssize_t @@ -1839,7 +1839,7 @@ goto Done; PyErr_SetString(PyExc_ValueError, - "sequence.index(x): x not in sequence"); + "sequence.index(x): x not in sequence"); /* fall into failure code */ Fail: n = -1; @@ -1865,7 +1865,7 @@ { Py_ssize_t result; PySequenceMethods *sqm = seq->ob_type->tp_as_sequence; - if (sqm != NULL && sqm->sq_contains != NULL) + if (sqm != NULL && sqm->sq_contains != NULL) return (*sqm->sq_contains)(seq, ob); result = _PySequence_IterSearch(seq, ob, PY_ITERSEARCH_CONTAINS); return Py_SAFE_DOWNCAST(result, Py_ssize_t, int); @@ -1890,7 +1890,7 @@ int PyMapping_Check(PyObject *o) { - return o && o->ob_type->tp_as_mapping && + return o && o->ob_type->tp_as_mapping && o->ob_type->tp_as_mapping->mp_subscript; } @@ -2044,7 +2044,7 @@ PyObject * PyObject_Call(PyObject *func, PyObject *arg, PyObject *kw) { - ternaryfunc call; + ternaryfunc call; if ((call = func->ob_type->tp_call) != NULL) { PyObject *result; @@ -2308,7 +2308,7 @@ * produce exactly the same results: NULL is returned and no error is set. * * If some exception other than AttributeError is raised, then NULL is also - * returned, but the exception is not cleared. That's because we want the + * returned, but the exception is not cleared. That's because we want the * exception to be propagated along. * * Callers are expected to test for PyErr_Occurred() when the return value @@ -2336,7 +2336,7 @@ return NULL; } if (!PyTuple_Check(bases)) { - Py_DECREF(bases); + Py_DECREF(bases); return NULL; } return bases; @@ -2428,18 +2428,18 @@ else if (PyTuple_Check(cls)) { Py_ssize_t i, n; - if (!recursion_depth) { - PyErr_SetString(PyExc_RuntimeError, - "nest level of tuple too deep"); - return -1; - } + if (!recursion_depth) { + PyErr_SetString(PyExc_RuntimeError, + "nest level of tuple too deep"); + return -1; + } n = PyTuple_GET_SIZE(cls); for (i = 0; i < n; i++) { retval = recursive_isinstance( - inst, - PyTuple_GET_ITEM(cls, i), - recursion_depth-1); + inst, + PyTuple_GET_ITEM(cls, i), + recursion_depth-1); if (retval != 0) break; } @@ -2490,12 +2490,12 @@ return recursive_isinstance(inst, cls, Py_GetRecursionLimit()); } -static int +static int recursive_issubclass(PyObject *derived, PyObject *cls, int recursion_depth) { int retval; - { + { if (!check_class(derived, "issubclass() arg 1 must be a class")) return -1; @@ -2504,16 +2504,16 @@ Py_ssize_t i; Py_ssize_t n = PyTuple_GET_SIZE(cls); - if (!recursion_depth) { - PyErr_SetString(PyExc_RuntimeError, - "nest level of tuple too deep"); - return -1; - } + if (!recursion_depth) { + PyErr_SetString(PyExc_RuntimeError, + "nest level of tuple too deep"); + return -1; + } for (i = 0; i < n; ++i) { retval = recursive_issubclass( - derived, - PyTuple_GET_ITEM(cls, i), - recursion_depth-1); + derived, + PyTuple_GET_ITEM(cls, i), + recursion_depth-1); if (retval != 0) { /* either found it, or got an error */ return retval; @@ -2590,7 +2590,7 @@ * If the iteration terminates normally, return NULL and clear the * PyExc_StopIteration exception (if it was set). PyErr_Occurred() * will be false. - * Else return the next object. PyErr_Occurred() will be false. + * Else return the next object. PyErr_Occurred() will be false. */ PyObject * PyIter_Next(PyObject *iter) Modified: python/branches/py3k/Objects/memoryobject.c ============================================================================== --- python/branches/py3k/Objects/memoryobject.c (original) +++ python/branches/py3k/Objects/memoryobject.c Sat Oct 13 01:27:53 2007 @@ -196,7 +196,9 @@ a contiguous buffer if it is not. The view will point to the shadow buffer which can be written to and then will be copied back into the other buffer when the memory - view is de-allocated. + view is de-allocated. While the shadow buffer is + being used, it will have an exclusive write lock on + the original buffer. */ PyObject * @@ -224,7 +226,7 @@ flags = PyBUF_FULL; break; case PyBUF_SHADOW: - flags = PyBUF_FULL_LCK; + flags = PyBUF_FULL_XLCK; break; } @@ -431,11 +433,7 @@ static PyObject * memory_repr(PyMemoryViewObject *self) { - /* XXX(nnorwitz): the code should be different or remove condition. */ - if (self->base == NULL) - return PyUnicode_FromFormat("", self); - else - return PyUnicode_FromFormat("", self); + return PyUnicode_FromFormat("", self); } @@ -502,6 +500,14 @@ /* Return a bytes object */ char *ptr; ptr = (char *)view->buf; + if (result < 0) { + result += view->shape[0]; + } + if ((result < 0) || (result > view->shape[0])) { + PyErr_SetString(PyExc_IndexError, + "index out of bounds"); + return NULL; + } if (view->strides == NULL) ptr += view->itemsize * result; else @@ -517,14 +523,20 @@ /* Return a new memory-view object */ Py_buffer newview; memset(&newview, 0, sizeof(newview)); + /* XXX: This needs to be fixed so it + actually returns a sub-view + */ return PyMemoryView_FromMemory(&newview); } } + /* Need to support getting a sliced view */ Py_INCREF(Py_NotImplemented); return Py_NotImplemented; } + +/* Need to support assigning memory if we can */ static int memory_ass_sub(PyMemoryViewObject *self, PyObject *key, PyObject *value) { From python-3000-checkins at python.org Sat Oct 13 20:56:01 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 13 Oct 2007 20:56:01 +0200 (CEST) Subject: [Python-3000-checkins] r58446 - in python/branches/py3k: PC/VC6/pythoncore.dsp PC/config.c PC/os2emx/Makefile PC/os2emx/config.c PCbuild8/pythoncore/pythoncore.vcproj Tools/scripts/README Message-ID: <20071013185601.2B2ED1E4006@bag.python.org> Author: neal.norwitz Date: Sat Oct 13 20:56:00 2007 New Revision: 58446 Modified: python/branches/py3k/PC/VC6/pythoncore.dsp python/branches/py3k/PC/config.c python/branches/py3k/PC/os2emx/Makefile python/branches/py3k/PC/os2emx/config.c python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj python/branches/py3k/Tools/scripts/README Log: Remove more hotshot cruft. Modified: python/branches/py3k/PC/VC6/pythoncore.dsp ============================================================================== --- python/branches/py3k/PC/VC6/pythoncore.dsp (original) +++ python/branches/py3k/PC/VC6/pythoncore.dsp Sat Oct 13 20:56:00 2007 @@ -137,10 +137,6 @@ # End Source File # Begin Source File -SOURCE=..\..\Modules\_hotshot.c -# End Source File -# Begin Source File - SOURCE=..\..\Modules\_localemodule.c # End Source File # Begin Source File Modified: python/branches/py3k/PC/config.c ============================================================================== --- python/branches/py3k/PC/config.c (original) +++ python/branches/py3k/PC/config.c Sat Oct 13 20:56:00 2007 @@ -99,7 +99,6 @@ {"_codecs", init_codecs}, {"_weakref", init_weakref}, - {"_hotshot", init_hotshot}, {"_random", init_random}, {"_bisect", init_bisect}, {"_heapq", init_heapq}, Modified: python/branches/py3k/PC/os2emx/Makefile ============================================================================== --- python/branches/py3k/PC/os2emx/Makefile (original) +++ python/branches/py3k/PC/os2emx/Makefile Sat Oct 13 20:56:00 2007 @@ -437,8 +437,7 @@ # Python modules to be dynamically loaded that need explicit build rules # (either multiple source files and/or non-standard module naming) # (NOTE: use shortened names for modules affected by 8 char name limit) -HARDEXTMODULES= _hotshot \ - _socket \ +HARDEXTMODULES= _socket \ _testcap \ unicoded @@ -567,9 +566,6 @@ # awkward handling (due e.g. to non-std naming, or multiple source files) # - standard modules -_hotshot$(MODULE.EXT): $(OUT)_hotshot$O $(OUT)_hotshot_m.def $(PYTHON.IMPLIB) - $(LD) $(LDFLAGS.DLL) -o $@ $(^^) $(L^) $(LIBS) - _socket$(MODULE.EXT): $(OUT)socketmodule$O $(OUT)_socket_m.def $(PYTHON.IMPLIB) $(LD) $(LDFLAGS.DLL) -o $@ $(^^) $(L^) $(LIBS) Modified: python/branches/py3k/PC/os2emx/config.c ============================================================================== --- python/branches/py3k/PC/os2emx/config.c (original) +++ python/branches/py3k/PC/os2emx/config.c Sat Oct 13 20:56:00 2007 @@ -75,7 +75,6 @@ #if !HAVE_DYNAMIC_LOADING extern void init_curses(); extern void init_curses_panel(); -extern void init_hotshot(); extern void init_testcapi(); extern void initbz2(); extern void initfpectl(); @@ -134,7 +133,6 @@ #if !HAVE_DYNAMIC_LOADING {"_curses", init_curses}, {"_curses_panel", init_curses_panel}, - {"_hotshot", init_hotshot}, {"_testcapi", init_testcapi}, {"bz2", initbz2}, {"fpectl", initfpectl}, Modified: python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj ============================================================================== --- python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj (original) +++ python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj Sat Oct 13 20:56:00 2007 @@ -1414,10 +1414,6 @@ > - - Modified: python/branches/py3k/Tools/scripts/README ============================================================================== --- python/branches/py3k/Tools/scripts/README (original) +++ python/branches/py3k/Tools/scripts/README Sat Oct 13 20:56:00 2007 @@ -30,7 +30,6 @@ google.py Open a webbrowser with Google. gprof2html.py Transform gprof(1) output into useful HTML. h2py.py Translate #define's into Python assignments -hotshotmain.py Main program to run script under control of hotshot idle Main program to start IDLE ifdef.py Remove #if(n)def groups from C sources lfcr.py Change LF line endings to CRLF (Unix to Windows) From python-3000-checkins at python.org Sat Oct 13 23:01:08 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 13 Oct 2007 23:01:08 +0200 (CEST) Subject: [Python-3000-checkins] r58447 - python/branches/py3k/Modules/arraymodule.c Message-ID: <20071013210108.5B3FD1E4006@bag.python.org> Author: guido.van.rossum Date: Sat Oct 13 23:01:08 2007 New Revision: 58447 Modified: python/branches/py3k/Modules/arraymodule.c Log: Fix typo in arraymodule.c (?) -- it's PyBUF_LOCK, not PyBUF_LOCKDATA. Modified: python/branches/py3k/Modules/arraymodule.c ============================================================================== --- python/branches/py3k/Modules/arraymodule.c (original) +++ python/branches/py3k/Modules/arraymodule.c Sat Oct 13 23:01:08 2007 @@ -1789,7 +1789,7 @@ "Cannot be a character buffer"); return -1; } - if ((flags & PyBUF_LOCKDATA)) { + if ((flags & PyBUF_LOCK)) { PyErr_SetString(PyExc_BufferError, "Cannot lock data"); return -1; From python-3000-checkins at python.org Sat Oct 13 23:03:28 2007 From: python-3000-checkins at python.org (travis.oliphant) Date: Sat, 13 Oct 2007 23:03:28 +0200 (CEST) Subject: [Python-3000-checkins] r58448 - in python/branches/py3k: Include/object.h Modules/arraymodule.c Objects/abstract.c Objects/unicodeobject.c Python/getargs.c Message-ID: <20071013210328.44F481E4006@bag.python.org> Author: travis.oliphant Date: Sat Oct 13 23:03:27 2007 New Revision: 58448 Modified: python/branches/py3k/Include/object.h python/branches/py3k/Modules/arraymodule.c python/branches/py3k/Objects/abstract.c python/branches/py3k/Objects/unicodeobject.c python/branches/py3k/Python/getargs.c Log: Eliminate use of PyBUF_CHARACTER flag which is no longer part of the buffer interface. Fix up array module to export the correct format for wide-builds. Modified: python/branches/py3k/Include/object.h ============================================================================== --- python/branches/py3k/Include/object.h (original) +++ python/branches/py3k/Include/object.h Sat Oct 13 23:03:27 2007 @@ -160,18 +160,17 @@ /* Flags for getting buffers */ #define PyBUF_SIMPLE 0 -#define PyBUF_CHARACTER 1 -#define PyBUF_WRITABLE 0x0002 +#define PyBUF_WRITABLE 0x0001 /* we used to include an E, backwards compatible alias */ #define PyBUF_WRITEABLE PyBUF_WRITABLE -#define PyBUF_LOCK 0x0004 -#define PyBUF_FORMAT 0x0008 -#define PyBUF_ND 0x0010 -#define PyBUF_STRIDES (0x0020 | PyBUF_ND) -#define PyBUF_C_CONTIGUOUS (0x0040 | PyBUF_STRIDES) -#define PyBUF_F_CONTIGUOUS (0x0080 | PyBUF_STRIDES) -#define PyBUF_ANY_CONTIGUOUS (0x0100 | PyBUF_STRIDES) -#define PyBUF_INDIRECT (0x0200 | PyBUF_STRIDES) +#define PyBUF_LOCK 0x0002 +#define PyBUF_FORMAT 0x0004 +#define PyBUF_ND 0x0008 +#define PyBUF_STRIDES (0x0010 | PyBUF_ND) +#define PyBUF_C_CONTIGUOUS (0x0020 | PyBUF_STRIDES) +#define PyBUF_F_CONTIGUOUS (0x0040 | PyBUF_STRIDES) +#define PyBUF_ANY_CONTIGUOUS (0x0080 | PyBUF_STRIDES) +#define PyBUF_INDIRECT (0x0100 | PyBUF_STRIDES) #define PyBUF_CONTIG (PyBUF_ND | PyBUF_WRITABLE) #define PyBUF_CONTIG_RO (PyBUF_ND) Modified: python/branches/py3k/Modules/arraymodule.c ============================================================================== --- python/branches/py3k/Modules/arraymodule.c (original) +++ python/branches/py3k/Modules/arraymodule.c Sat Oct 13 23:03:27 2007 @@ -385,7 +385,7 @@ static struct arraydescr descriptors[] = { {'b', 1, b_getitem, b_setitem, "b"}, {'B', 1, BB_getitem, BB_setitem, "B"}, - {'u', sizeof(Py_UNICODE), u_getitem, u_setitem, "U"}, + {'u', sizeof(Py_UNICODE), u_getitem, u_setitem, "u"}, {'h', sizeof(short), h_getitem, h_setitem, "h"}, {'H', sizeof(short), HH_getitem, HH_setitem, "H"}, {'i', sizeof(int), i_getitem, i_setitem, "i"}, @@ -1784,11 +1784,6 @@ static int array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags) { - if ((flags & PyBUF_CHARACTER)) { - PyErr_SetString(PyExc_TypeError, - "Cannot be a character buffer"); - return -1; - } if ((flags & PyBUF_LOCK)) { PyErr_SetString(PyExc_BufferError, "Cannot lock data"); @@ -1815,6 +1810,11 @@ view->internal = NULL; if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) { view->format = self->ob_descr->formats; +#ifdef Py_UNICODE_WIDE + if (self->ob_descr->typecode == 'u') { + view->formats = "w"; + } +#endif } finish: Modified: python/branches/py3k/Objects/abstract.c ============================================================================== --- python/branches/py3k/Objects/abstract.c (original) +++ python/branches/py3k/Objects/abstract.c Sat Oct 13 23:03:27 2007 @@ -236,7 +236,7 @@ "expected an object with the buffer interface"); return -1; } - if ((*pb->bf_getbuffer)(obj, &view, PyBUF_CHARACTER)) return -1; + if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; *buffer = view.buf; *buffer_len = view.len; Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Sat Oct 13 23:03:27 2007 @@ -8117,10 +8117,6 @@ unicode_buffer_getbuffer(PyUnicodeObject *self, Py_buffer *view, int flags) { - if (flags & PyBUF_CHARACTER) { - PyErr_SetString(PyExc_SystemError, "can't use str as char buffer"); - return -1; - } return PyBuffer_FillInfo(view, (void *)self->str, PyUnicode_GET_DATA_SIZE(self), 1, flags); } Modified: python/branches/py3k/Python/getargs.c ============================================================================== --- python/branches/py3k/Python/getargs.c (original) +++ python/branches/py3k/Python/getargs.c Sat Oct 13 23:03:27 2007 @@ -1237,7 +1237,9 @@ (*pb->bf_releasebuffer)(arg, &view); break; } - + + /*TEO: This can be eliminated --- here only for backward + compatibility */ case 't': { /* 8-bit character buffer, read-only access */ char **p = va_arg(*p_va, char **); PyBufferProcs *pb = arg->ob_type->tp_as_buffer; @@ -1253,7 +1255,7 @@ "string or read-only character buffer", arg, msgbuf, bufsize); - if ((*pb->bf_getbuffer)(arg, &view, PyBUF_CHARACTER) != 0) + if ((*pb->bf_getbuffer)(arg, &view, PyBUF_SIMPLE) != 0) return converterr("string or single-segment read-only buffer", arg, msgbuf, bufsize); From python-3000-checkins at python.org Sun Oct 14 01:23:58 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Sun, 14 Oct 2007 01:23:58 +0200 (CEST) Subject: [Python-3000-checkins] r58452 - python/branches/py3k/Lib/bsddb/dbshelve.py Message-ID: <20071013232358.56F631E4006@bag.python.org> Author: gregory.p.smith Date: Sun Oct 14 01:23:58 2007 New Revision: 58452 Modified: python/branches/py3k/Lib/bsddb/dbshelve.py Log: Merge 58450: fix uncollectable reference caused by bsddb.db.DBShelf.append Adds a DBShelf __repr__ method to not raise an exception when the DB is closed. Modified: python/branches/py3k/Lib/bsddb/dbshelve.py ============================================================================== --- python/branches/py3k/Lib/bsddb/dbshelve.py (original) +++ python/branches/py3k/Lib/bsddb/dbshelve.py Sun Oct 14 01:23:58 2007 @@ -86,6 +86,7 @@ """ def __init__(self, dbenv=None): self.db = db.DB(dbenv) + self._closed = True self.binary = 1 @@ -128,6 +129,23 @@ return self.db.keys() + def open(self, *args, **kwargs): + self.db.open(*args, **kwargs) + self._closed = False + + + def close(self, *args, **kwargs): + self.db.close(*args, **kwargs) + self._closed = True + + + def __repr__(self): + if self._closed: + return '' % (id(self)) + else: + return repr(dict(self.iteritems())) + + def items(self, txn=None): if txn != None: items = self.db.items(txn) @@ -156,8 +174,7 @@ def append(self, value, txn=None): if self.get_type() == db.DB_RECNO: - self.append = self.__append - return self.append(value, txn=txn) + return self.__append(value, txn=txn) raise DBShelveError("append() only supported when dbshelve opened with filetype=dbshelve.db.DB_RECNO") From python-3000-checkins at python.org Sun Oct 14 04:05:52 2007 From: python-3000-checkins at python.org (alexandre.vassalotti) Date: Sun, 14 Oct 2007 04:05:52 +0200 (CEST) Subject: [Python-3000-checkins] r58455 - in python/branches/py3k: Modules/_sre.c Modules/posixmodule.c Objects/unicodeobject.c Python/getargs.c Message-ID: <20071014020552.55D691E4006@bag.python.org> Author: alexandre.vassalotti Date: Sun Oct 14 04:05:51 2007 New Revision: 58455 Modified: python/branches/py3k/Modules/_sre.c python/branches/py3k/Modules/posixmodule.c python/branches/py3k/Objects/unicodeobject.c python/branches/py3k/Python/getargs.c Log: Remove the buffer API from PyUnicode as specified by PEP 3137. Also, fix the error message of the 't' format unit, in getargs.c, so that it asks for bytes, instead of string. Modified: python/branches/py3k/Modules/_sre.c ============================================================================== --- python/branches/py3k/Modules/_sre.c (original) +++ python/branches/py3k/Modules/_sre.c Sun Oct 14 04:05:51 2007 @@ -1674,6 +1674,15 @@ void* ptr; Py_buffer view; + /* Unicode objects do not support the buffer API. So, get the data + directly instead. */ + if (PyUnicode_Check(string)) { + ptr = (void *)PyUnicode_AS_DATA(string); + *p_length = PyUnicode_GET_SIZE(string); + *p_charsize = sizeof(Py_UNICODE); + return ptr; + } + /* get pointer to string buffer */ view.len = -1; buffer = Py_Type(string)->tp_as_buffer; Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Sun Oct 14 04:05:51 2007 @@ -2135,7 +2135,8 @@ FILEFINDBUF3 ep; APIRET rc; - if (!PyArg_ParseTuple(args, "t#:listdir", &name, &len)) + if (!PyArg_ParseTuple(args, "et#:listdir", + Py_FileSystemDefaultEncoding, &name, &len)) return NULL; if (len >= MAX_PATH) { PyErr_SetString(PyExc_ValueError, "path too long"); Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Sun Oct 14 04:05:51 2007 @@ -8113,15 +8113,6 @@ }; -static int -unicode_buffer_getbuffer(PyUnicodeObject *self, Py_buffer *view, int flags) -{ - - return PyBuffer_FillInfo(view, (void *)self->str, - PyUnicode_GET_DATA_SIZE(self), 1, flags); -} - - /* Helpers for PyUnicode_Format() */ static PyObject * @@ -8815,11 +8806,6 @@ return NULL; } -static PyBufferProcs unicode_as_buffer = { - (getbufferproc) unicode_buffer_getbuffer, - NULL, -}; - static PyObject * unicode_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds); @@ -8903,7 +8889,7 @@ (reprfunc) unicode_str, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ - &unicode_as_buffer, /* tp_as_buffer */ + 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_UNICODE_SUBCLASS, /* tp_flags */ unicode_doc, /* tp_doc */ Modified: python/branches/py3k/Python/getargs.c ============================================================================== --- python/branches/py3k/Python/getargs.c (original) +++ python/branches/py3k/Python/getargs.c Sun Oct 14 04:05:51 2007 @@ -1252,7 +1252,7 @@ arg, msgbuf, bufsize); if (pb == NULL || pb->bf_getbuffer == NULL) return converterr( - "string or read-only character buffer", + "bytes or read-only character buffer", arg, msgbuf, bufsize); if ((*pb->bf_getbuffer)(arg, &view, PyBUF_SIMPLE) != 0) From nnorwitz at gmail.com Sun Oct 14 04:22:51 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 13 Oct 2007 19:22:51 -0700 Subject: [Python-3000-checkins] r58455 - in python/branches/py3k: Modules/_sre.c Modules/posixmodule.c Objects/unicodeobject.c Python/getargs.c In-Reply-To: <20071014020552.55D691E4006@bag.python.org> References: <20071014020552.55D691E4006@bag.python.org> Message-ID: On Oct 13, 2007 7:05 PM, alexandre.vassalotti wrote: > Author: alexandre.vassalotti > Date: Sun Oct 14 04:05:51 2007 > New Revision: 58455 > > > Modified: python/branches/py3k/Modules/posixmodule.c > ============================================================================== > --- python/branches/py3k/Modules/posixmodule.c (original) > +++ python/branches/py3k/Modules/posixmodule.c Sun Oct 14 04:05:51 2007 > @@ -2135,7 +2135,8 @@ > FILEFINDBUF3 ep; > APIRET rc; > > - if (!PyArg_ParseTuple(args, "t#:listdir", &name, &len)) > + if (!PyArg_ParseTuple(args, "et#:listdir", > + Py_FileSystemDefaultEncoding, &name, &len)) > return NULL; I think this creates a memory leak. name needs to be deallocated when using et. See the section below for an example. n From python-3000-checkins at python.org Sun Oct 14 04:54:41 2007 From: python-3000-checkins at python.org (alexandre.vassalotti) Date: Sun, 14 Oct 2007 04:54:41 +0200 (CEST) Subject: [Python-3000-checkins] r58456 - python/branches/py3k/Modules/posixmodule.c Message-ID: <20071014025441.7EC0D1E400F@bag.python.org> Author: alexandre.vassalotti Date: Sun Oct 14 04:54:41 2007 New Revision: 58456 Modified: python/branches/py3k/Modules/posixmodule.c Log: Fix the memory leak introduced in r58455. The buffer reference returned by 'et' need to be freed after use. Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Sun Oct 14 04:54:41 2007 @@ -2150,8 +2150,10 @@ namebuf[len++] = SEP; strcpy(namebuf + len, "*.*"); - if ((d = PyList_New(0)) == NULL) + if ((d = PyList_New(0)) == NULL) { + PyMem_Free(name); return NULL; + } rc = DosFindFirst(namebuf, /* Wildcard Pattern to Match */ &hdir, /* Handle to Use While Search Directory */ @@ -2192,6 +2194,7 @@ } while (DosFindNext(hdir, &ep, sizeof(ep), &srchcnt) == NO_ERROR && srchcnt > 0); } + PyMem_Free(name); return d; #else From alexandre at peadrop.com Sun Oct 14 04:55:40 2007 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Sat, 13 Oct 2007 22:55:40 -0400 Subject: [Python-3000-checkins] r58455 - in python/branches/py3k: Modules/_sre.c Modules/posixmodule.c Objects/unicodeobject.c Python/getargs.c In-Reply-To: References: <20071014020552.55D691E4006@bag.python.org> Message-ID: On 10/13/07, Neal Norwitz wrote: > I think this creates a memory leak. name needs to be deallocated when > using et. See the section below for an example. > Thanks Neal! Fixed in r58456. -- Alexandre From python-3000-checkins at python.org Sun Oct 14 05:24:43 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sun, 14 Oct 2007 05:24:43 +0200 (CEST) Subject: [Python-3000-checkins] r58457 - python/branches/py3k/Modules/posixmodule.c Message-ID: <20071014032443.1967F1E400F@bag.python.org> Author: neal.norwitz Date: Sun Oct 14 05:23:09 2007 New Revision: 58457 Modified: python/branches/py3k/Modules/posixmodule.c Log: Fix some more memory leaks (in error conditions) introduced in r58455. Also fix some indentation. Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Sun Oct 14 05:23:09 2007 @@ -2139,7 +2139,8 @@ Py_FileSystemDefaultEncoding, &name, &len)) return NULL; if (len >= MAX_PATH) { - PyErr_SetString(PyExc_ValueError, "path too long"); + PyMem_Free(name); + PyErr_SetString(PyExc_ValueError, "path too long"); return NULL; } strcpy(namebuf, name); @@ -2150,7 +2151,7 @@ namebuf[len++] = SEP; strcpy(namebuf + len, "*.*"); - if ((d = PyList_New(0)) == NULL) { + if ((d = PyList_New(0)) == NULL) { PyMem_Free(name); return NULL; } @@ -2164,7 +2165,7 @@ if (rc != NO_ERROR) { errno = ENOENT; - return posix_error_with_filename(name); + return posix_error_with_allocated_filename(name); } if (srchcnt > 0) { /* If Directory is NOT Totally Empty, */ From alexandre at peadrop.com Sun Oct 14 06:41:45 2007 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Sun, 14 Oct 2007 00:41:45 -0400 Subject: [Python-3000-checkins] r58457 - python/branches/py3k/Modules/posixmodule.c In-Reply-To: <20071014032443.1967F1E400F@bag.python.org> References: <20071014032443.1967F1E400F@bag.python.org> Message-ID: On 10/13/07, neal.norwitz wrote: > Author: neal.norwitz > Date: Sun Oct 14 05:23:09 2007 > New Revision: 58457 > > Modified: > python/branches/py3k/Modules/posixmodule.c > Log: > Fix some more memory leaks (in error conditions) introduced in r58455. > Also fix some indentation. > Thanks for fixing this. -- Alexandre From python-3000-checkins at python.org Sun Oct 14 22:46:16 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sun, 14 Oct 2007 22:46:16 +0200 (CEST) Subject: [Python-3000-checkins] r58461 - python/branches/py3k/Modules/arraymodule.c Message-ID: <20071014204616.44A101E4027@bag.python.org> Author: guido.van.rossum Date: Sun Oct 14 22:46:15 2007 New Revision: 58461 Modified: python/branches/py3k/Modules/arraymodule.c Log: Fix typo: formats -> format. (Only mattered in wide unicode builds.) Modified: python/branches/py3k/Modules/arraymodule.c ============================================================================== --- python/branches/py3k/Modules/arraymodule.c (original) +++ python/branches/py3k/Modules/arraymodule.c Sun Oct 14 22:46:15 2007 @@ -1812,7 +1812,7 @@ view->format = self->ob_descr->formats; #ifdef Py_UNICODE_WIDE if (self->ob_descr->typecode == 'u') { - view->formats = "w"; + view->format = "w"; } #endif } From python-3000-checkins at python.org Mon Oct 15 02:25:56 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 15 Oct 2007 02:25:56 +0200 (CEST) Subject: [Python-3000-checkins] r58463 - python/branches/py3k/Lib/test/test_frozen.py Message-ID: <20071015002556.E12901E400A@bag.python.org> Author: guido.van.rossum Date: Mon Oct 15 02:25:56 2007 New Revision: 58463 Modified: python/branches/py3k/Lib/test/test_frozen.py Log: Make it possible to run this test stand-alone. Modified: python/branches/py3k/Lib/test/test_frozen.py ============================================================================== --- python/branches/py3k/Lib/test/test_frozen.py (original) +++ python/branches/py3k/Lib/test/test_frozen.py Mon Oct 15 02:25:56 2007 @@ -38,3 +38,6 @@ def test_main(): run_unittest(FrozenTests) + +if __name__ == "__main__": + test_main() From python-3000-checkins at python.org Mon Oct 15 03:27:53 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 15 Oct 2007 03:27:53 +0200 (CEST) Subject: [Python-3000-checkins] r58464 - python/branches/py3k/Lib/distutils/dist.py Message-ID: <20071015012753.BDDFA1E400A@bag.python.org> Author: guido.van.rossum Date: Mon Oct 15 03:27:53 2007 New Revision: 58464 Modified: python/branches/py3k/Lib/distutils/dist.py Log: Fix yet another stray 2.x-ism (maybe merged?). Modified: python/branches/py3k/Lib/distutils/dist.py ============================================================================== --- python/branches/py3k/Lib/distutils/dist.py (original) +++ python/branches/py3k/Lib/distutils/dist.py Mon Oct 15 03:27:53 2007 @@ -280,8 +280,7 @@ from pprint import pformat if commands is None: # dump all command option dicts - commands = self.command_options.keys() - commands.sort() + commands = sorted(self.command_options.keys()) if header is not None: print(indent + header) From python-3000-checkins at python.org Mon Oct 15 04:40:08 2007 From: python-3000-checkins at python.org (kurt.kaiser) Date: Mon, 15 Oct 2007 04:40:08 +0200 (CEST) Subject: [Python-3000-checkins] r58465 - python/branches/py3k/Lib/idlelib/IOBinding.py Message-ID: <20071015024008.DE9421E400A@bag.python.org> Author: kurt.kaiser Date: Mon Oct 15 04:40:08 2007 New Revision: 58465 Modified: python/branches/py3k/Lib/idlelib/IOBinding.py Log: Finish bug fix applied at 58398. I missed a piece, Tal Einat found the error. Modified: python/branches/py3k/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/py3k/Lib/idlelib/IOBinding.py (original) +++ python/branches/py3k/Lib/idlelib/IOBinding.py Mon Oct 15 04:40:08 2007 @@ -392,7 +392,7 @@ text = self.text.get("1.0", "end-1c") if self.eol_convention != "\n": text = text.replace("\n", self.eol_convention) - chars = self.encode(self.text.get("1.0", "end-1c")) + chars = self.encode(text) try: f = open(filename, "wb") f.write(chars) From python-3000-checkins at python.org Mon Oct 15 04:52:59 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 15 Oct 2007 04:52:59 +0200 (CEST) Subject: [Python-3000-checkins] r58466 - in python/branches/py3k: Include/code.h Include/unicodeobject.h Misc/ACKS Modules/_ctypes/callbacks.c Modules/posixmodule.c Modules/pyexpat.c Objects/codeobject.c Objects/moduleobject.c Objects/unicodeobject.c Python/bltinmodule.c Python/ceval.c Python/compile.c Python/frozen.c Python/import.c Python/importdl.c Python/pythonrun.c Python/traceback.c Message-ID: <20071015025259.470781E402C@bag.python.org> Author: guido.van.rossum Date: Mon Oct 15 04:52:41 2007 New Revision: 58466 Modified: python/branches/py3k/Include/code.h python/branches/py3k/Include/unicodeobject.h python/branches/py3k/Misc/ACKS python/branches/py3k/Modules/_ctypes/callbacks.c python/branches/py3k/Modules/posixmodule.c python/branches/py3k/Modules/pyexpat.c python/branches/py3k/Objects/codeobject.c python/branches/py3k/Objects/moduleobject.c python/branches/py3k/Objects/unicodeobject.c python/branches/py3k/Python/bltinmodule.c python/branches/py3k/Python/ceval.c python/branches/py3k/Python/compile.c python/branches/py3k/Python/frozen.c python/branches/py3k/Python/import.c python/branches/py3k/Python/importdl.c python/branches/py3k/Python/pythonrun.c python/branches/py3k/Python/traceback.c Log: Patch #1272, by Christian Heimes and Alexandre Vassalotti. Changes to make __file__ a proper Unicode object, using the default filesystem encoding. This is a bit tricky because the default filesystem encoding isn't set by the time we import the first modules; at that point we fudge things a bit. This is okay since __file__ isn't really used much except for error reporting. Tested on OSX and Linux only so far. Modified: python/branches/py3k/Include/code.h ============================================================================== --- python/branches/py3k/Include/code.h (original) +++ python/branches/py3k/Include/code.h Mon Oct 15 04:52:41 2007 @@ -21,8 +21,8 @@ PyObject *co_freevars; /* tuple of strings (free variable names) */ PyObject *co_cellvars; /* tuple of strings (cell variable names) */ /* The rest doesn't count for hash/cmp */ - PyObject *co_filename; /* string (where it was loaded from) */ - PyObject *co_name; /* string (name, for reference) */ + PyObject *co_filename; /* unicode (where it was loaded from) */ + PyObject *co_name; /* unicode (name, for reference) */ int co_firstlineno; /* first source line number */ PyObject *co_lnotab; /* string (encoding addr<->lineno mapping) */ void *co_zombieframe; /* for optimization only (see frameobject.c) */ Modified: python/branches/py3k/Include/unicodeobject.h ============================================================================== --- python/branches/py3k/Include/unicodeobject.h (original) +++ python/branches/py3k/Include/unicodeobject.h Mon Oct 15 04:52:41 2007 @@ -154,6 +154,7 @@ # define PyUnicode_DecodeASCII PyUnicodeUCS2_DecodeASCII # define PyUnicode_DecodeCharmap PyUnicodeUCS2_DecodeCharmap # define PyUnicode_DecodeLatin1 PyUnicodeUCS2_DecodeLatin1 +# define PyUnicode_DecodeFSDefault PyUnicodeUCS2_DecodeFSDefault # define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS2_DecodeRawUnicodeEscape # define PyUnicode_DecodeUTF32 PyUnicodeUCS2_DecodeUTF32 # define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS2_DecodeUTF32Stateful @@ -245,6 +246,7 @@ # define PyUnicode_DecodeASCII PyUnicodeUCS4_DecodeASCII # define PyUnicode_DecodeCharmap PyUnicodeUCS4_DecodeCharmap # define PyUnicode_DecodeLatin1 PyUnicodeUCS4_DecodeLatin1 +# define PyUnicode_DecodeFSDefault PyUnicodeUCS4_DecodeFSDefault # define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS4_DecodeRawUnicodeEscape # define PyUnicode_DecodeUTF32 PyUnicodeUCS4_DecodeUTF32 # define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS4_DecodeUTF32Stateful @@ -641,6 +643,20 @@ PyAPI_FUNC(PyObject *) _PyUnicode_AsDefaultEncodedString( PyObject *, const char *); +/* Decode a null-terminated string using Py_FileSystemDefaultEncoding. + + If the encoding is supported by one of the built-in codecs (i.e., UTF-8, + UTF-16, UTF-32, Latin-1 or MBCS), otherwise fallback to UTF-8 and replace + invalid characters with '?'. + + The function is intended to be used for paths and file names only + during bootstrapping process where the codecs are not set up. +*/ + +PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefault( + const char *s /* encoded string */ + ); + /* Return a char* holding the UTF-8 encoded value of the Unicode object. Modified: python/branches/py3k/Misc/ACKS ============================================================================== --- python/branches/py3k/Misc/ACKS (original) +++ python/branches/py3k/Misc/ACKS Mon Oct 15 04:52:41 2007 @@ -273,6 +273,7 @@ Shane Hathaway Rycharde Hawkes Jochen Hayek +Christian Heimes Thomas Heller Lance Finn Helsten Jonathan Hendry @@ -667,6 +668,7 @@ Hector Urtubia Atul Varma Dmitry Vasiliev +Alexandre Vassalotti Frank Vercruesse Mike Verdone Jaap Vermeulen Modified: python/branches/py3k/Modules/_ctypes/callbacks.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/callbacks.c (original) +++ python/branches/py3k/Modules/_ctypes/callbacks.c Mon Oct 15 04:52:41 2007 @@ -34,9 +34,9 @@ PyCodeObject *py_code = 0; PyFrameObject *py_frame = 0; - py_srcfile = PyString_FromString(filename); + py_srcfile = PyUnicode_DecodeFSDefault(filename); if (!py_srcfile) goto bad; - py_funcname = PyString_FromString(funcname); + py_funcname = PyUnicode_FromString(funcname); if (!py_funcname) goto bad; py_globals = PyDict_New(); if (!py_globals) goto bad; Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Mon Oct 15 04:52:41 2007 @@ -5370,7 +5370,7 @@ #endif if (name == NULL) return PyErr_NoMemory(); - result = PyString_FromString(name); + result = PyUnicode_DecodeFSDefault(name); free(name); return result; } @@ -5428,7 +5428,7 @@ Py_XDECREF(err); return NULL; } - return PyString_FromString(buffer); + return PyUnicode_DecodeFSDefault(buffer); } #endif Modified: python/branches/py3k/Modules/pyexpat.c ============================================================================== --- python/branches/py3k/Modules/pyexpat.c (original) +++ python/branches/py3k/Modules/pyexpat.c Mon Oct 15 04:52:41 2007 @@ -232,13 +232,13 @@ code = PyString_FromString(""); if (code == NULL) goto failed; - name = PyString_FromString(func_name); + name = PyUnicode_FromString(func_name); if (name == NULL) goto failed; nulltuple = PyTuple_New(0); if (nulltuple == NULL) goto failed; - filename = PyString_FromString(__FILE__); + filename = PyUnicode_DecodeFSDefault(__FILE__); handler_info[slot].tb_code = PyCode_New(0, /* argcount */ 0, /* kwonlyargcount */ Modified: python/branches/py3k/Objects/codeobject.c ============================================================================== --- python/branches/py3k/Objects/codeobject.c (original) +++ python/branches/py3k/Objects/codeobject.c Mon Oct 15 04:52:41 2007 @@ -50,6 +50,7 @@ { PyCodeObject *co; Py_ssize_t i; + /* Check argument types */ if (argcount < 0 || nlocals < 0 || code == NULL || @@ -58,20 +59,16 @@ varnames == NULL || !PyTuple_Check(varnames) || freevars == NULL || !PyTuple_Check(freevars) || cellvars == NULL || !PyTuple_Check(cellvars) || - name == NULL || (!PyString_Check(name) && !PyUnicode_Check(name)) || - filename == NULL || !PyString_Check(filename) || + name == NULL || !PyUnicode_Check(name) || + filename == NULL || !PyUnicode_Check(filename) || lnotab == NULL || !PyString_Check(lnotab) || !PyObject_CheckReadBuffer(code)) { PyErr_BadInternalCall(); return NULL; } - if (PyString_Check(name)) { - name = PyUnicode_FromString(PyString_AS_STRING(name)); - if (name == NULL) - return NULL; - } else { - Py_INCREF(name); - } + Py_INCREF(name); + Py_INCREF(filename); + intern_strings(names); intern_strings(varnames); intern_strings(freevars); @@ -299,8 +296,8 @@ if (co->co_firstlineno != 0) lineno = co->co_firstlineno; - if (co->co_filename && PyString_Check(co->co_filename)) - filename = PyString_AS_STRING(co->co_filename); + if (co->co_filename && PyUnicode_Check(co->co_filename)) + filename = PyUnicode_AsString(co->co_filename); return PyUnicode_FromFormat( "", co->co_name, co, filename, lineno); Modified: python/branches/py3k/Objects/moduleobject.c ============================================================================== --- python/branches/py3k/Objects/moduleobject.c (original) +++ python/branches/py3k/Objects/moduleobject.c Mon Oct 15 04:52:41 2007 @@ -86,12 +86,12 @@ d = ((PyModuleObject *)m)->md_dict; if (d == NULL || (fileobj = PyDict_GetItemString(d, "__file__")) == NULL || - !PyString_Check(fileobj)) + !PyUnicode_Check(fileobj)) { PyErr_SetString(PyExc_SystemError, "module filename missing"); return NULL; } - return PyString_AsString(fileobj); + return PyUnicode_AsString(fileobj); } void Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Mon Oct 15 04:52:41 2007 @@ -117,7 +117,11 @@ /* Default encoding to use and assume when NULL is passed as encoding parameter; it is fixed to "utf-8". Always use the - PyUnicode_GetDefaultEncoding() API to access this global. */ + PyUnicode_GetDefaultEncoding() API to access this global. + + Don't forget to alter Py_FileSystemDefaultEncoding() if you change the + hard coded default! +*/ static const char unicode_default_encoding[] = "utf-8"; Py_UNICODE @@ -1231,6 +1235,35 @@ return v; } +PyObject* +PyUnicode_DecodeFSDefault(const char *s) +{ + Py_ssize_t size = (Py_ssize_t)strlen(s); + + /* During the early bootstrapping process, Py_FileSystemDefaultEncoding + can be undefined. If it is case, decode using UTF-8. The following assumes + that Py_FileSystemDefaultEncoding is set to a built-in encoding during the + bootstrapping process where the codecs aren't ready yet. + */ + if (Py_FileSystemDefaultEncoding) { +#if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T) + if (strcmp(Py_FileSystemDefaultEncoding, "mbcs")) { + return PyUnicode_DecodeMBCS(s, size, "replace"); + } +#elif defined(__APPLE__) + if (strcmp(Py_FileSystemDefaultEncoding, "utf-8")) { + return PyUnicode_DecodeUTF8(s, size, "replace"); + } +#endif + return PyUnicode_Decode(s, size, + Py_FileSystemDefaultEncoding, + "replace"); + } + else { + return PyUnicode_DecodeUTF8(s, size, "replace"); + } +} + char* PyUnicode_AsStringAndSize(PyObject *unicode, Py_ssize_t *psize) { Modified: python/branches/py3k/Python/bltinmodule.c ============================================================================== --- python/branches/py3k/Python/bltinmodule.c (original) +++ python/branches/py3k/Python/bltinmodule.c Mon Oct 15 04:52:41 2007 @@ -10,6 +10,9 @@ /* The default encoding used by the platform file system APIs Can remain NULL for all platforms that don't have such a concept + + Don't forget to modify PyUnicode_DecodeFSDefault() if you touch any of the + values for Py_FileSystemDefaultEncoding! */ #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T) const char *Py_FileSystemDefaultEncoding = "mbcs"; Modified: python/branches/py3k/Python/ceval.c ============================================================================== --- python/branches/py3k/Python/ceval.c (original) +++ python/branches/py3k/Python/ceval.c Mon Oct 15 04:52:41 2007 @@ -767,7 +767,7 @@ lltrace = PyDict_GetItemString(f->f_globals, "__lltrace__") != NULL; #endif #if defined(Py_DEBUG) || defined(LLTRACE) - filename = PyString_AsString(co->co_filename); + filename = PyUnicode_AsString(co->co_filename); #endif why = WHY_NOT; @@ -2565,7 +2565,7 @@ if (argcount > co->co_argcount) { if (!(co->co_flags & CO_VARARGS)) { PyErr_Format(PyExc_TypeError, - "%S() takes %s %d " + "%U() takes %s %d " "%spositional argument%s (%d given)", co->co_name, defcount ? "at most" : "exactly", @@ -2599,7 +2599,7 @@ int j; if (keyword == NULL || !PyUnicode_Check(keyword)) { PyErr_Format(PyExc_TypeError, - "%S() keywords must be strings", + "%U() keywords must be strings", co->co_name); goto fail; } @@ -2622,7 +2622,7 @@ if (j >= co->co_argcount + co->co_kwonlyargcount) { if (kwdict == NULL) { PyErr_Format(PyExc_TypeError, - "%S() got an unexpected " + "%U() got an unexpected " "keyword argument '%S'", co->co_name, keyword); @@ -2633,7 +2633,7 @@ else { if (GETLOCAL(j) != NULL) { PyErr_Format(PyExc_TypeError, - "%S() got multiple " + "%U() got multiple " "values for keyword " "argument '%S'", co->co_name, @@ -2661,7 +2661,7 @@ continue; } PyErr_Format(PyExc_TypeError, - "%S() needs keyword-only argument %S", + "%U() needs keyword-only argument %S", co->co_name, name); goto fail; } @@ -2671,7 +2671,7 @@ for (i = argcount; i < m; i++) { if (GETLOCAL(i) == NULL) { PyErr_Format(PyExc_TypeError, - "%S() takes %s %d " + "%U() takes %s %d " "%spositional argument%s " "(%d given)", co->co_name, @@ -2699,7 +2699,7 @@ else { if (argcount > 0 || kwcount > 0) { PyErr_Format(PyExc_TypeError, - "%S() takes no arguments (%d given)", + "%U() takes no arguments (%d given)", co->co_name, argcount + kwcount); goto fail; Modified: python/branches/py3k/Python/compile.c ============================================================================== --- python/branches/py3k/Python/compile.c (original) +++ python/branches/py3k/Python/compile.c Mon Oct 15 04:52:41 2007 @@ -1247,7 +1247,7 @@ PyObject_REPR(name), PyString_AS_STRING(c->u->u_name), reftype, arg, - PyString_AS_STRING(co->co_name), + PyUnicode_AsString(co->co_name), PyObject_REPR(co->co_freevars)); Py_FatalError("compiler_make_closure()"); } @@ -4001,7 +4001,7 @@ freevars = dict_keys_inorder(c->u->u_freevars, PyTuple_Size(cellvars)); if (!freevars) goto error; - filename = PyString_FromString(c->c_filename); + filename = PyUnicode_DecodeFSDefault(c->c_filename); if (!filename) goto error; Modified: python/branches/py3k/Python/frozen.c ============================================================================== --- python/branches/py3k/Python/frozen.c (original) +++ python/branches/py3k/Python/frozen.c Mon Oct 15 04:52:41 2007 @@ -17,7 +17,7 @@ 131,1,0,1,100,1,0,83,40,2,0,0,0,117,14,0, 0,0,72,101,108,108,111,32,119,111,114,108,100,46,46,46, 78,40,1,0,0,0,117,5,0,0,0,112,114,105,110,116, - 40,0,0,0,0,40,0,0,0,0,40,0,0,0,0,115, + 40,0,0,0,0,40,0,0,0,0,40,0,0,0,0,117, 8,0,0,0,104,101,108,108,111,46,112,121,117,8,0,0, 0,60,109,111,100,117,108,101,62,1,0,0,0,115,0,0, 0,0, Modified: python/branches/py3k/Python/import.c ============================================================================== --- python/branches/py3k/Python/import.c (original) +++ python/branches/py3k/Python/import.c Mon Oct 15 04:52:41 2007 @@ -74,10 +74,11 @@ 3040 (added signature annotations) 3050 (print becomes a function) 3060 (PEP 3115 metaclass syntax) - 3070 (PEP 3109 raise changes) + 3070 (PEP 3109 raise changes) + 3080 (PEP 3137 make __file__ and __name__ unicode) . */ -#define MAGIC (3070 | ((long)'\r'<<16) | ((long)'\n'<<24)) +#define MAGIC (3080 | ((long)'\r'<<16) | ((long)'\n'<<24)) /* Magic word as global; note that _PyImport_Init() can change the value of this global to accommodate for alterations of how the @@ -652,7 +653,7 @@ /* Remember the filename as the __file__ attribute */ v = NULL; if (pathname != NULL) { - v = PyString_FromString(pathname); + v = PyUnicode_DecodeFSDefault(pathname); if (v == NULL) PyErr_Clear(); } @@ -983,7 +984,7 @@ PySys_WriteStderr("import %s # directory %s\n", name, pathname); d = PyModule_GetDict(m); - file = PyString_FromString(pathname); + file = PyUnicode_DecodeFSDefault(pathname); if (file == NULL) goto error; path = Py_BuildValue("[O]", file); Modified: python/branches/py3k/Python/importdl.c ============================================================================== --- python/branches/py3k/Python/importdl.c (original) +++ python/branches/py3k/Python/importdl.c Mon Oct 15 04:52:41 2007 @@ -62,7 +62,9 @@ return NULL; } /* Remember the filename as the __file__ attribute */ - if (PyModule_AddStringConstant(m, "__file__", pathname) < 0) + PyObject *path; + path = PyUnicode_DecodeFSDefault(pathname); + if (PyModule_AddObject(m, "__file__", path) < 0) PyErr_Clear(); /* Not important enough to report */ if (_PyImport_FixupExtension(name, pathname) == NULL) Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Mon Oct 15 04:52:41 2007 @@ -867,7 +867,8 @@ return -1; d = PyModule_GetDict(m); if (PyDict_GetItemString(d, "__file__") == NULL) { - PyObject *f = PyString_FromString(filename); + PyObject *f; + f = PyUnicode_DecodeFSDefault(filename); if (f == NULL) return -1; if (PyDict_SetItemString(d, "__file__", f) < 0) { Modified: python/branches/py3k/Python/traceback.c ============================================================================== --- python/branches/py3k/Python/traceback.c (original) +++ python/branches/py3k/Python/traceback.c Mon Oct 15 04:52:41 2007 @@ -229,10 +229,10 @@ while (tb != NULL && err == 0) { if (depth <= limit) { err = tb_displayline(f, - PyString_AsString( + PyUnicode_AsString( tb->tb_frame->f_code->co_filename), tb->tb_lineno, - PyString_AsString(tb->tb_frame->f_code->co_name)); + PyUnicode_AsString(tb->tb_frame->f_code->co_name)); } depth--; tb = tb->tb_next; From python-3000-checkins at python.org Mon Oct 15 17:42:31 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 15 Oct 2007 17:42:31 +0200 (CEST) Subject: [Python-3000-checkins] r58470 - python/branches/py3k/Doc/library/locale.rst Message-ID: <20071015154231.AC3A61E402D@bag.python.org> Author: guido.van.rossum Date: Mon Oct 15 17:42:31 2007 New Revision: 58470 Modified: python/branches/py3k/Doc/library/locale.rst Log: Update what the locale module documents about string operations. Modified: python/branches/py3k/Doc/library/locale.rst ============================================================================== --- python/branches/py3k/Doc/library/locale.rst (original) +++ python/branches/py3k/Doc/library/locale.rst Mon Oct 15 17:42:31 2007 @@ -498,26 +498,23 @@ before the settings have been restored. If, when coding a module for general use, you need a locale independent version -of an operation that is affected by the locale (such as :func:`string.lower`, or +of an operation that is affected by the locale (such as certain formats used with :func:`time.strftime`), you will have to find a way to do it without using the standard library routine. Even better is convincing yourself that using locale settings is okay. Only as a last resort should you document that your module is not compatible with non-\ ``C`` locale settings. -.. index:: module: string - -The case conversion functions in the :mod:`string` module are affected by the -locale settings. When a call to the :func:`setlocale` function changes the -:const:`LC_CTYPE` settings, the variables ``string.lowercase``, -``string.uppercase`` and ``string.letters`` are recalculated. Note that code -that uses these variable through ':keyword:`from` ... :keyword:`import` ...', -e.g. ``from string import letters``, is not affected by subsequent -:func:`setlocale` calls. - The only way to perform numeric operations according to the locale is to use the special functions defined by this module: :func:`atof`, :func:`atoi`, :func:`format`, :func:`str`. +There is no way to perform case conversions and character classifications +according to the locale. For (Unicode) text strings these are done according +to the character value only, while for byte strings, the conversions and +classifications are done according to the ASCII value of the byte, and bytes +whose high bit is set (i.e., non-ASCII bytes) are never converted or considered +part of a character class such as letter or whitespace. + .. _embedding-locale: From python-3000-checkins at python.org Mon Oct 15 18:08:27 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Mon, 15 Oct 2007 18:08:27 +0200 (CEST) Subject: [Python-3000-checkins] r58472 - python/branches/py3k/Python/importdl.c Message-ID: <20071015160827.1BB3E1E4020@bag.python.org> Author: neal.norwitz Date: Mon Oct 15 18:08:26 2007 New Revision: 58472 Modified: python/branches/py3k/Python/importdl.c Log: Move decl so it compiles with C89. Modified: python/branches/py3k/Python/importdl.c ============================================================================== --- python/branches/py3k/Python/importdl.c (original) +++ python/branches/py3k/Python/importdl.c Mon Oct 15 18:08:26 2007 @@ -22,6 +22,7 @@ _PyImport_LoadDynamicModule(char *name, char *pathname, FILE *fp) { PyObject *m; + PyObject *path; char *lastdot, *shortname, *packagecontext, *oldcontext; dl_funcptr p; @@ -62,7 +63,6 @@ return NULL; } /* Remember the filename as the __file__ attribute */ - PyObject *path; path = PyUnicode_DecodeFSDefault(pathname); if (PyModule_AddObject(m, "__file__", path) < 0) PyErr_Clear(); /* Not important enough to report */ From python-3000-checkins at python.org Mon Oct 15 21:39:59 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 15 Oct 2007 21:39:59 +0200 (CEST) Subject: [Python-3000-checkins] r58481 - in python/branches/py3k-importlib: Doc/library/locale.rst Include/code.h Include/memoryobject.h Include/object.h Include/unicodeobject.h Lib/bsddb/dbshelve.py Lib/bsddb/test/test_dbshelve.py Lib/distutils/dist.py Lib/doctest.py Lib/idlelib/IOBinding.py Lib/io.py Lib/test/test_frozen.py Misc/ACKS Misc/NEWS Modules/_ctypes/callbacks.c Modules/_sre.c Modules/arraymodule.c Modules/posixmodule.c Modules/pyexpat.c Objects/abstract.c Objects/codeobject.c Objects/memoryobject.c Objects/moduleobject.c Objects/unicodeobject.c PC/VC6/pythoncore.dsp PC/config.c PC/os2emx/Makefile PC/os2emx/config.c PCbuild8/pythoncore/pythoncore.vcproj Python/bltinmodule.c Python/ceval.c Python/compile.c Python/frozen.c Python/getargs.c Python/import.c Python/importdl.c Python/pythonrun.c Python/traceback.c Tools/scripts/README configure configure.in pyconfig.h.in Message-ID: <20071015193959.9DA321E42D6@bag.python.org> Author: brett.cannon Date: Mon Oct 15 21:39:57 2007 New Revision: 58481 Modified: python/branches/py3k-importlib/ (props changed) python/branches/py3k-importlib/Doc/library/locale.rst python/branches/py3k-importlib/Include/code.h python/branches/py3k-importlib/Include/memoryobject.h python/branches/py3k-importlib/Include/object.h python/branches/py3k-importlib/Include/unicodeobject.h python/branches/py3k-importlib/Lib/bsddb/dbshelve.py python/branches/py3k-importlib/Lib/bsddb/test/test_dbshelve.py python/branches/py3k-importlib/Lib/distutils/dist.py python/branches/py3k-importlib/Lib/doctest.py python/branches/py3k-importlib/Lib/idlelib/IOBinding.py python/branches/py3k-importlib/Lib/io.py python/branches/py3k-importlib/Lib/test/test_frozen.py python/branches/py3k-importlib/Misc/ACKS python/branches/py3k-importlib/Misc/NEWS python/branches/py3k-importlib/Modules/_ctypes/callbacks.c python/branches/py3k-importlib/Modules/_sre.c python/branches/py3k-importlib/Modules/arraymodule.c python/branches/py3k-importlib/Modules/posixmodule.c python/branches/py3k-importlib/Modules/pyexpat.c python/branches/py3k-importlib/Objects/abstract.c python/branches/py3k-importlib/Objects/codeobject.c python/branches/py3k-importlib/Objects/memoryobject.c python/branches/py3k-importlib/Objects/moduleobject.c python/branches/py3k-importlib/Objects/unicodeobject.c python/branches/py3k-importlib/PC/VC6/pythoncore.dsp python/branches/py3k-importlib/PC/config.c python/branches/py3k-importlib/PC/os2emx/Makefile python/branches/py3k-importlib/PC/os2emx/config.c python/branches/py3k-importlib/PCbuild8/pythoncore/pythoncore.vcproj python/branches/py3k-importlib/Python/bltinmodule.c python/branches/py3k-importlib/Python/ceval.c python/branches/py3k-importlib/Python/compile.c python/branches/py3k-importlib/Python/frozen.c python/branches/py3k-importlib/Python/getargs.c python/branches/py3k-importlib/Python/import.c python/branches/py3k-importlib/Python/importdl.c python/branches/py3k-importlib/Python/pythonrun.c python/branches/py3k-importlib/Python/traceback.c python/branches/py3k-importlib/Tools/scripts/README python/branches/py3k-importlib/configure python/branches/py3k-importlib/configure.in python/branches/py3k-importlib/pyconfig.h.in Log: Merged revisions 58416-58479 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k Modified: python/branches/py3k-importlib/Doc/library/locale.rst ============================================================================== --- python/branches/py3k-importlib/Doc/library/locale.rst (original) +++ python/branches/py3k-importlib/Doc/library/locale.rst Mon Oct 15 21:39:57 2007 @@ -498,26 +498,23 @@ before the settings have been restored. If, when coding a module for general use, you need a locale independent version -of an operation that is affected by the locale (such as :func:`string.lower`, or +of an operation that is affected by the locale (such as certain formats used with :func:`time.strftime`), you will have to find a way to do it without using the standard library routine. Even better is convincing yourself that using locale settings is okay. Only as a last resort should you document that your module is not compatible with non-\ ``C`` locale settings. -.. index:: module: string - -The case conversion functions in the :mod:`string` module are affected by the -locale settings. When a call to the :func:`setlocale` function changes the -:const:`LC_CTYPE` settings, the variables ``string.lowercase``, -``string.uppercase`` and ``string.letters`` are recalculated. Note that code -that uses these variable through ':keyword:`from` ... :keyword:`import` ...', -e.g. ``from string import letters``, is not affected by subsequent -:func:`setlocale` calls. - The only way to perform numeric operations according to the locale is to use the special functions defined by this module: :func:`atof`, :func:`atoi`, :func:`format`, :func:`str`. +There is no way to perform case conversions and character classifications +according to the locale. For (Unicode) text strings these are done according +to the character value only, while for byte strings, the conversions and +classifications are done according to the ASCII value of the byte, and bytes +whose high bit is set (i.e., non-ASCII bytes) are never converted or considered +part of a character class such as letter or whitespace. + .. _embedding-locale: Modified: python/branches/py3k-importlib/Include/code.h ============================================================================== --- python/branches/py3k-importlib/Include/code.h (original) +++ python/branches/py3k-importlib/Include/code.h Mon Oct 15 21:39:57 2007 @@ -21,8 +21,8 @@ PyObject *co_freevars; /* tuple of strings (free variable names) */ PyObject *co_cellvars; /* tuple of strings (cell variable names) */ /* The rest doesn't count for hash/cmp */ - PyObject *co_filename; /* string (where it was loaded from) */ - PyObject *co_name; /* string (name, for reference) */ + PyObject *co_filename; /* unicode (where it was loaded from) */ + PyObject *co_name; /* unicode (name, for reference) */ int co_firstlineno; /* first source line number */ PyObject *co_lnotab; /* string (encoding addr<->lineno mapping) */ void *co_zombieframe; /* for optimization only (see frameobject.c) */ Modified: python/branches/py3k-importlib/Include/memoryobject.h ============================================================================== --- python/branches/py3k-importlib/Include/memoryobject.h (original) +++ python/branches/py3k-importlib/Include/memoryobject.h Mon Oct 15 21:39:57 2007 @@ -8,9 +8,9 @@ #endif typedef struct { - PyObject_HEAD - PyObject *base; - Py_buffer view; + PyObject_HEAD + PyObject *base; + Py_buffer view; } PyMemoryViewObject; @@ -21,39 +21,40 @@ #define Py_END_OF_MEMORY (-1) -PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, int buffertype, - char fort); +PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, + int buffertype, + char fort); /* Return a contiguous chunk of memory representing the buffer from an object in a memory view object. If a copy is made then the - base object for the memory view will be a *new* bytes object. - - Otherwise, the base-object will be the object itself and no - data-copying will be done. - - The buffertype argument can be PyBUF_READ, PyBUF_WRITE, - PyBUF_UPDATEIFCOPY to determine whether the returned buffer - should be READONLY, WRITABLE, or set to update the - original buffer if a copy must be made. If buffertype is - PyBUF_WRITE and the buffer is not contiguous an error will - be raised. In this circumstance, the user can use - PyBUF_UPDATEIFCOPY to ensure that a a writable temporary - contiguous buffer is returned. The contents of this - contiguous buffer will be copied back into the original - object after the memoryview object is deleted as long as - the original object is writable and allows setting its - memory to "readonly". If this is not allowed by the - original object, then a BufferError is raised. + base object for the memory view will be a *new* bytes object. - If the object is multi-dimensional and if fortran is 'F', - the first dimension of the underlying array will vary the - fastest in the buffer. If fortran is 'C', then the last - dimension will vary the fastest (C-style contiguous). If - fortran is 'A', then it does not matter and you will get - whatever the object decides is more efficient. + Otherwise, the base-object will be the object itself and no + data-copying will be done. - A new reference is returned that must be DECREF'd when finished. - */ + The buffertype argument can be PyBUF_READ, PyBUF_WRITE, + PyBUF_SHADOW to determine whether the returned buffer + should be READONLY, WRITABLE, or set to update the + original buffer if a copy must be made. If buffertype is + PyBUF_WRITE and the buffer is not contiguous an error will + be raised. In this circumstance, the user can use + PyBUF_SHADOW to ensure that a a writable temporary + contiguous buffer is returned. The contents of this + contiguous buffer will be copied back into the original + object after the memoryview object is deleted as long as + the original object is writable and allows setting an + exclusive write lock. If this is not allowed by the + original object, then a BufferError is raised. + + If the object is multi-dimensional and if fortran is 'F', + the first dimension of the underlying array will vary the + fastest in the buffer. If fortran is 'C', then the last + dimension will vary the fastest (C-style contiguous). If + fortran is 'A', then it does not matter and you will get + whatever the object decides is more efficient. + + A new reference is returned that must be DECREF'd when finished. + */ PyAPI_FUNC(PyObject *) PyMemoryView_FromObject(PyObject *base); Modified: python/branches/py3k-importlib/Include/object.h ============================================================================== --- python/branches/py3k-importlib/Include/object.h (original) +++ python/branches/py3k-importlib/Include/object.h Mon Oct 15 21:39:57 2007 @@ -160,34 +160,39 @@ /* Flags for getting buffers */ #define PyBUF_SIMPLE 0 -#define PyBUF_CHARACTER 1 -#define PyBUF_WRITABLE 0x0002 +#define PyBUF_WRITABLE 0x0001 /* we used to include an E, backwards compatible alias */ #define PyBUF_WRITEABLE PyBUF_WRITABLE -#define PyBUF_LOCKDATA 0x0004 -#define PyBUF_FORMAT 0x0008 -#define PyBUF_ND 0x0010 -#define PyBUF_STRIDES (0x0020 | PyBUF_ND) -#define PyBUF_C_CONTIGUOUS (0x0040 | PyBUF_STRIDES) -#define PyBUF_F_CONTIGUOUS (0x0080 | PyBUF_STRIDES) -#define PyBUF_ANY_CONTIGUOUS (0x0100 | PyBUF_STRIDES) -#define PyBUF_INDIRECT (0x0200 | PyBUF_STRIDES) +#define PyBUF_LOCK 0x0002 +#define PyBUF_FORMAT 0x0004 +#define PyBUF_ND 0x0008 +#define PyBUF_STRIDES (0x0010 | PyBUF_ND) +#define PyBUF_C_CONTIGUOUS (0x0020 | PyBUF_STRIDES) +#define PyBUF_F_CONTIGUOUS (0x0040 | PyBUF_STRIDES) +#define PyBUF_ANY_CONTIGUOUS (0x0080 | PyBUF_STRIDES) +#define PyBUF_INDIRECT (0x0100 | PyBUF_STRIDES) #define PyBUF_CONTIG (PyBUF_ND | PyBUF_WRITABLE) #define PyBUF_CONTIG_RO (PyBUF_ND) -#define PyBUF_CONTIG_LCK (PyBUF_ND | PyBUF_LOCKDATA) +#define PyBUF_CONTIG_LCK (PyBUF_ND | PyBUF_LOCK) +#define PyBUF_CONTIG_XLCK (PyBUF_ND | PyBUF_LOCK | PyBUF_WRITABLE) #define PyBUF_STRIDED (PyBUF_STRIDES | PyBUF_WRITABLE) #define PyBUF_STRIDED_RO (PyBUF_STRIDES) -#define PyBUF_STRIDED_LCK (PyBUF_STRIDES | PyBUF_LOCKDATA) +#define PyBUF_STRIDED_LCK (PyBUF_STRIDES | PyBUF_LOCK) +#define PyBUF_STRIDED_XLCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_WRITABLE) #define PyBUF_RECORDS (PyBUF_STRIDES | PyBUF_WRITABLE | PyBUF_FORMAT) #define PyBUF_RECORDS_RO (PyBUF_STRIDES | PyBUF_FORMAT) -#define PyBUF_RECORDS_LCK (PyBUF_STRIDES | PyBUF_LOCKDATA | PyBUF_FORMAT) +#define PyBUF_RECORDS_LCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_FORMAT) +#define PyBUF_RECORDS_XLCK (PyBUF_STRIDES | PyBUF_LOCK | PyBUF_WRITABLE \ + | PyBUF_FORMAT) #define PyBUF_FULL (PyBUF_INDIRECT | PyBUF_WRITABLE | PyBUF_FORMAT) #define PyBUF_FULL_RO (PyBUF_INDIRECT | PyBUF_FORMAT) -#define PyBUF_FULL_LCK (PyBUF_INDIRECT | PyBUF_LOCKDATA | PyBUF_FORMAT) +#define PyBUF_FULL_LCK (PyBUF_INDIRECT | PyBUF_LOCK | PyBUF_FORMAT) +#define PyBUF_FULL_XLCK (PyBUF_INDIRECT | PyBUF_LOCK | PyBUF_WRITABLE \ + | PyBUF_FORMAT) #define PyBUF_READ 0x100 Modified: python/branches/py3k-importlib/Include/unicodeobject.h ============================================================================== --- python/branches/py3k-importlib/Include/unicodeobject.h (original) +++ python/branches/py3k-importlib/Include/unicodeobject.h Mon Oct 15 21:39:57 2007 @@ -154,6 +154,7 @@ # define PyUnicode_DecodeASCII PyUnicodeUCS2_DecodeASCII # define PyUnicode_DecodeCharmap PyUnicodeUCS2_DecodeCharmap # define PyUnicode_DecodeLatin1 PyUnicodeUCS2_DecodeLatin1 +# define PyUnicode_DecodeFSDefault PyUnicodeUCS2_DecodeFSDefault # define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS2_DecodeRawUnicodeEscape # define PyUnicode_DecodeUTF32 PyUnicodeUCS2_DecodeUTF32 # define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS2_DecodeUTF32Stateful @@ -245,6 +246,7 @@ # define PyUnicode_DecodeASCII PyUnicodeUCS4_DecodeASCII # define PyUnicode_DecodeCharmap PyUnicodeUCS4_DecodeCharmap # define PyUnicode_DecodeLatin1 PyUnicodeUCS4_DecodeLatin1 +# define PyUnicode_DecodeFSDefault PyUnicodeUCS4_DecodeFSDefault # define PyUnicode_DecodeRawUnicodeEscape PyUnicodeUCS4_DecodeRawUnicodeEscape # define PyUnicode_DecodeUTF32 PyUnicodeUCS4_DecodeUTF32 # define PyUnicode_DecodeUTF32Stateful PyUnicodeUCS4_DecodeUTF32Stateful @@ -641,6 +643,20 @@ PyAPI_FUNC(PyObject *) _PyUnicode_AsDefaultEncodedString( PyObject *, const char *); +/* Decode a null-terminated string using Py_FileSystemDefaultEncoding. + + If the encoding is supported by one of the built-in codecs (i.e., UTF-8, + UTF-16, UTF-32, Latin-1 or MBCS), otherwise fallback to UTF-8 and replace + invalid characters with '?'. + + The function is intended to be used for paths and file names only + during bootstrapping process where the codecs are not set up. +*/ + +PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefault( + const char *s /* encoded string */ + ); + /* Return a char* holding the UTF-8 encoded value of the Unicode object. Modified: python/branches/py3k-importlib/Lib/bsddb/dbshelve.py ============================================================================== --- python/branches/py3k-importlib/Lib/bsddb/dbshelve.py (original) +++ python/branches/py3k-importlib/Lib/bsddb/dbshelve.py Mon Oct 15 21:39:57 2007 @@ -77,12 +77,16 @@ #--------------------------------------------------------------------------- +class DBShelveError(db.DBError): pass + + class DBShelf(DictMixin): """A shelf to hold pickled objects, built upon a bsddb DB object. It automatically pickles/unpickles data objects going to/from the DB. """ def __init__(self, dbenv=None): self.db = db.DB(dbenv) + self._closed = True self.binary = 1 @@ -125,6 +129,23 @@ return self.db.keys() + def open(self, *args, **kwargs): + self.db.open(*args, **kwargs) + self._closed = False + + + def close(self, *args, **kwargs): + self.db.close(*args, **kwargs) + self._closed = True + + + def __repr__(self): + if self._closed: + return '' % (id(self)) + else: + return repr(dict(self.iteritems())) + + def items(self, txn=None): if txn != None: items = self.db.items(txn) @@ -152,10 +173,9 @@ return self.db.append(data, txn) def append(self, value, txn=None): - if self.get_type() != db.DB_RECNO: - self.append = self.__append - return self.append(value, txn=txn) - raise db.DBError("append() only supported when dbshelve opened with filetype=dbshelve.db.DB_RECNO") + if self.get_type() == db.DB_RECNO: + return self.__append(value, txn=txn) + raise DBShelveError("append() only supported when dbshelve opened with filetype=dbshelve.db.DB_RECNO") def associate(self, secondaryDB, callback, flags=0): Modified: python/branches/py3k-importlib/Lib/bsddb/test/test_dbshelve.py ============================================================================== --- python/branches/py3k-importlib/Lib/bsddb/test/test_dbshelve.py (original) +++ python/branches/py3k-importlib/Lib/bsddb/test/test_dbshelve.py Mon Oct 15 21:39:57 2007 @@ -50,17 +50,22 @@ except os.error: pass + def mk(self, key): + """Turn key into an appropriate key type for this db""" + # override in child class for RECNO + return key.encode("ascii") + def populateDB(self, d): for x in letters: - d[('S' + x).encode("ascii")] = 10 * x # add a string - d[('I' + x).encode("ascii")] = ord(x) # add an integer - d[('L' + x).encode("ascii")] = [x] * 10 # add a list + d[self.mk('S' + x)] = 10 * x # add a string + d[self.mk('I' + x)] = ord(x) # add an integer + d[self.mk('L' + x)] = [x] * 10 # add a list inst = DataClass() # add an instance inst.S = 10 * x inst.I = ord(x) inst.L = [x] * 10 - d[('O' + x).encode("ascii")] = inst + d[self.mk('O' + x)] = inst # overridable in derived classes to affect how the shelf is created/opened @@ -100,14 +105,14 @@ print("keys:", k) print("stats:", s) - self.assertFalse(d.has_key(b'bad key')) - self.assertTrue(d.has_key(b'IA'), d.keys()) - self.assertTrue(d.has_key(b'OA')) - - d.delete(b'IA') - del d[b'OA'] - self.assertFalse(d.has_key(b'IA')) - self.assertFalse(d.has_key(b'OA')) + self.assertFalse(d.has_key(self.mk('bad key'))) + self.assertTrue(d.has_key(self.mk('IA')), d.keys()) + self.assertTrue(d.has_key(self.mk('OA'))) + + d.delete(self.mk('IA')) + del d[self.mk('OA')] + self.assertFalse(d.has_key(self.mk('IA'))) + self.assertFalse(d.has_key(self.mk('OA'))) self.assertEqual(len(d), l-2) values = [] @@ -129,18 +134,18 @@ for key, value in items: self.checkrec(key, value) - self.assertEqual(d.get(b'bad key'), None) - self.assertEqual(d.get(b'bad key', None), None) - self.assertEqual(d.get(b'bad key', b'a string'), b'a string') - self.assertEqual(d.get(b'bad key', [1, 2, 3]), [1, 2, 3]) + self.assertEqual(d.get(self.mk('bad key')), None) + self.assertEqual(d.get(self.mk('bad key'), None), None) + self.assertEqual(d.get(self.mk('bad key'), b'a string'), b'a string') + self.assertEqual(d.get(self.mk('bad key'), [1, 2, 3]), [1, 2, 3]) d.set_get_returns_none(0) - self.assertRaises(db.DBNotFoundError, d.get, b'bad key') + self.assertRaises(db.DBNotFoundError, d.get, self.mk('bad key')) d.set_get_returns_none(1) - d.put(b'new key', b'new data') - self.assertEqual(d.get(b'new key'), b'new data') - self.assertEqual(d[b'new key'], b'new data') + d.put(self.mk('new key'), b'new data') + self.assertEqual(d.get(self.mk('new key')), b'new data') + self.assertEqual(d[self.mk('new key')], b'new data') @@ -179,12 +184,23 @@ self.assertEqual(count, len(d)) - c.set(b'SS') + c.set(self.mk('SS')) key, value = c.current() self.checkrec(key, value) del c + def test03_append(self): + # NOTE: this is overridden in RECNO subclass, don't change its name. + if verbose: + print('\n', '-=' * 30) + print("Running %s.test03_append..." % self.__class__.__name__) + + self.assertRaises(dbshelve.DBShelveError, + self.d.append, b'unit test was here') + + def checkrec(self, key, value): + # override this in a subclass if the key type is different x = key[1:] if key[0:1] == b'S': self.assertEquals(type(value), str) @@ -289,7 +305,43 @@ #---------------------------------------------------------------------- -# TODO: Add test cases for a DBShelf in a RECNO DB. +# test cases for a DBShelf in a RECNO DB. + +class RecNoShelveTestCase(BasicShelveTestCase): + dbtype = db.DB_RECNO + dbflags = db.DB_CREATE + + def setUp(self): + BasicShelveTestCase.setUp(self) + + # pool to assign integer key values out of + self.key_pool = list(range(1, 5000)) + self.key_map = {} # map string keys to the number we gave them + self.intkey_map = {} # reverse map of above + + def mk(self, key): + if key not in self.key_map: + self.key_map[key] = self.key_pool.pop(0) + self.intkey_map[self.key_map[key]] = key.encode('ascii') + return self.key_map[key] + + def checkrec(self, intkey, value): + key = self.intkey_map[intkey] + BasicShelveTestCase.checkrec(self, key, value) + + def test03_append(self): + if verbose: + print('\n', '-=' * 30) + print("Running %s.test03_append..." % self.__class__.__name__) + + self.d[1] = b'spam' + self.d[5] = b'eggs' + self.assertEqual(6, self.d.append(b'spam')) + self.assertEqual(7, self.d.append(b'baked beans')) + self.assertEqual(b'spam', self.d.get(6)) + self.assertEqual(b'spam', self.d.get(1)) + self.assertEqual(b'baked beans', self.d.get(7)) + self.assertEqual(b'eggs', self.d.get(5)) #---------------------------------------------------------------------- @@ -306,6 +358,7 @@ suite.addTest(unittest.makeSuite(EnvHashShelveTestCase)) suite.addTest(unittest.makeSuite(EnvThreadBTreeShelveTestCase)) suite.addTest(unittest.makeSuite(EnvThreadHashShelveTestCase)) + suite.addTest(unittest.makeSuite(RecNoShelveTestCase)) return suite Modified: python/branches/py3k-importlib/Lib/distutils/dist.py ============================================================================== --- python/branches/py3k-importlib/Lib/distutils/dist.py (original) +++ python/branches/py3k-importlib/Lib/distutils/dist.py Mon Oct 15 21:39:57 2007 @@ -280,8 +280,7 @@ from pprint import pformat if commands is None: # dump all command option dicts - commands = self.command_options.keys() - commands.sort() + commands = sorted(self.command_options.keys()) if header is not None: print(indent + header) Modified: python/branches/py3k-importlib/Lib/doctest.py ============================================================================== --- python/branches/py3k-importlib/Lib/doctest.py (original) +++ python/branches/py3k-importlib/Lib/doctest.py Mon Oct 15 21:39:57 2007 @@ -209,7 +209,8 @@ filename = _module_relative_path(package, filename) if hasattr(package, '__loader__'): if hasattr(package.__loader__, 'get_data'): - return package.__loader__.get_data(filename).decode('utf-8'), filename + return (package.__loader__.get_data(filename).decode(encoding), + filename) return open(filename, encoding=encoding).read(), filename def _indent(s, indent=4): Modified: python/branches/py3k-importlib/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/py3k-importlib/Lib/idlelib/IOBinding.py (original) +++ python/branches/py3k-importlib/Lib/idlelib/IOBinding.py Mon Oct 15 21:39:57 2007 @@ -392,7 +392,7 @@ text = self.text.get("1.0", "end-1c") if self.eol_convention != "\n": text = text.replace("\n", self.eol_convention) - chars = self.encode(self.text.get("1.0", "end-1c")) + chars = self.encode(text) try: f = open(filename, "wb") f.write(chars) Modified: python/branches/py3k-importlib/Lib/io.py ============================================================================== --- python/branches/py3k-importlib/Lib/io.py (original) +++ python/branches/py3k-importlib/Lib/io.py Mon Oct 15 21:39:57 2007 @@ -34,7 +34,6 @@ import sys import codecs import _fileio -import io import warnings # open() uses st_blksize whenever we can @@ -1031,7 +1030,7 @@ if encoding is None: try: encoding = os.device_encoding(buffer.fileno()) - except (AttributeError, io.UnsupportedOperation): + except (AttributeError, UnsupportedOperation): pass if encoding is None: try: Modified: python/branches/py3k-importlib/Lib/test/test_frozen.py ============================================================================== --- python/branches/py3k-importlib/Lib/test/test_frozen.py (original) +++ python/branches/py3k-importlib/Lib/test/test_frozen.py Mon Oct 15 21:39:57 2007 @@ -38,3 +38,6 @@ def test_main(): run_unittest(FrozenTests) + +if __name__ == "__main__": + test_main() Modified: python/branches/py3k-importlib/Misc/ACKS ============================================================================== --- python/branches/py3k-importlib/Misc/ACKS (original) +++ python/branches/py3k-importlib/Misc/ACKS Mon Oct 15 21:39:57 2007 @@ -273,6 +273,7 @@ Shane Hathaway Rycharde Hawkes Jochen Hayek +Christian Heimes Thomas Heller Lance Finn Helsten Jonathan Hendry @@ -667,6 +668,7 @@ Hector Urtubia Atul Varma Dmitry Vasiliev +Alexandre Vassalotti Frank Vercruesse Mike Verdone Jaap Vermeulen Modified: python/branches/py3k-importlib/Misc/NEWS ============================================================================== --- python/branches/py3k-importlib/Misc/NEWS (original) +++ python/branches/py3k-importlib/Misc/NEWS Mon Oct 15 21:39:57 2007 @@ -14,6 +14,14 @@ - The `hotshot` profiler has been removed; use `cProfile` instead. +Library +------- + +- When loading an external file using testfile(), the passed-in encoding + argument was being ignored if __loader__ is defined and forcing the source to + be UTF-8. + + What's New in Python 3.0a1? ========================== Modified: python/branches/py3k-importlib/Modules/_ctypes/callbacks.c ============================================================================== --- python/branches/py3k-importlib/Modules/_ctypes/callbacks.c (original) +++ python/branches/py3k-importlib/Modules/_ctypes/callbacks.c Mon Oct 15 21:39:57 2007 @@ -34,9 +34,9 @@ PyCodeObject *py_code = 0; PyFrameObject *py_frame = 0; - py_srcfile = PyString_FromString(filename); + py_srcfile = PyUnicode_DecodeFSDefault(filename); if (!py_srcfile) goto bad; - py_funcname = PyString_FromString(funcname); + py_funcname = PyUnicode_FromString(funcname); if (!py_funcname) goto bad; py_globals = PyDict_New(); if (!py_globals) goto bad; Modified: python/branches/py3k-importlib/Modules/_sre.c ============================================================================== --- python/branches/py3k-importlib/Modules/_sre.c (original) +++ python/branches/py3k-importlib/Modules/_sre.c Mon Oct 15 21:39:57 2007 @@ -1674,6 +1674,15 @@ void* ptr; Py_buffer view; + /* Unicode objects do not support the buffer API. So, get the data + directly instead. */ + if (PyUnicode_Check(string)) { + ptr = (void *)PyUnicode_AS_DATA(string); + *p_length = PyUnicode_GET_SIZE(string); + *p_charsize = sizeof(Py_UNICODE); + return ptr; + } + /* get pointer to string buffer */ view.len = -1; buffer = Py_Type(string)->tp_as_buffer; Modified: python/branches/py3k-importlib/Modules/arraymodule.c ============================================================================== --- python/branches/py3k-importlib/Modules/arraymodule.c (original) +++ python/branches/py3k-importlib/Modules/arraymodule.c Mon Oct 15 21:39:57 2007 @@ -40,14 +40,6 @@ static PyTypeObject Arraytype; -#ifdef Py_UNICODE_WIDE -#define PyArr_UNI 'w' -#define PyArr_UNISTR "w" -#else -#define PyArr_UNI 'u' -#define PyArr_UNISTR "u" -#endif - #define array_Check(op) PyObject_TypeCheck(op, &Arraytype) #define array_CheckExact(op) (Py_Type(op) == &Arraytype) @@ -193,12 +185,14 @@ return 0; } + static PyObject * h_getitem(arrayobject *ap, Py_ssize_t i) { return PyInt_FromLong((long) ((short *)ap->ob_item)[i]); } + static int h_setitem(arrayobject *ap, Py_ssize_t i, PyObject *v) { @@ -389,9 +383,9 @@ /* Description of types */ static struct arraydescr descriptors[] = { - {'b', sizeof(char), b_getitem, b_setitem, "b"}, - {'B', sizeof(char), BB_getitem, BB_setitem, "B"}, - {PyArr_UNI, sizeof(Py_UNICODE), u_getitem, u_setitem, PyArr_UNISTR}, + {'b', 1, b_getitem, b_setitem, "b"}, + {'B', 1, BB_getitem, BB_setitem, "B"}, + {'u', sizeof(Py_UNICODE), u_getitem, u_setitem, "u"}, {'h', sizeof(short), h_getitem, h_setitem, "h"}, {'H', sizeof(short), HH_getitem, HH_setitem, "H"}, {'i', sizeof(int), i_getitem, i_setitem, "i"}, @@ -1418,11 +1412,13 @@ { Py_UNICODE *ustr; Py_ssize_t n; + char typecode; if (!PyArg_ParseTuple(args, "u#:fromunicode", &ustr, &n)) return NULL; - if (self->ob_descr->typecode != PyArr_UNI) { - PyErr_SetString(PyExc_ValueError, + typecode = self->ob_descr->typecode; + if ((typecode != 'u')) { + PyErr_SetString(PyExc_ValueError, "fromunicode() may only be called on " "unicode type arrays"); return NULL; @@ -1457,9 +1453,11 @@ static PyObject * array_tounicode(arrayobject *self, PyObject *unused) { - if (self->ob_descr->typecode != PyArr_UNI) { + char typecode; + typecode = self->ob_descr->typecode; + if ((typecode != 'u')) { PyErr_SetString(PyExc_ValueError, - "tounicode() may only be called on unicode type arrays"); + "tounicode() may only be called on unicode type arrays"); return NULL; } return PyUnicode_FromUnicode((Py_UNICODE *) self->ob_item, Py_Size(self)); @@ -1560,7 +1558,7 @@ if (len == 0) { return PyUnicode_FromFormat("array('%c')", typecode); } - if (typecode == PyArr_UNI) + if ((typecode == 'u')) v = array_tounicode(a, NULL); else v = array_tolist(a, NULL); @@ -1786,12 +1784,7 @@ static int array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags) { - if ((flags & PyBUF_CHARACTER)) { - PyErr_SetString(PyExc_TypeError, - "Cannot be a character buffer"); - return -1; - } - if ((flags & PyBUF_LOCKDATA)) { + if ((flags & PyBUF_LOCK)) { PyErr_SetString(PyExc_BufferError, "Cannot lock data"); return -1; @@ -1817,6 +1810,11 @@ view->internal = NULL; if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) { view->format = self->ob_descr->formats; +#ifdef Py_UNICODE_WIDE + if (self->ob_descr->typecode == 'u') { + view->format = "w"; + } +#endif } finish: @@ -1864,7 +1862,7 @@ if (!(initial == NULL || PyList_Check(initial) || PyBytes_Check(initial) || PyTuple_Check(initial) - || (c == PyArr_UNI && PyUnicode_Check(initial)))) { + || ((c=='u') && PyUnicode_Check(initial)))) { it = PyObject_GetIter(initial); if (it == NULL) return NULL; @@ -1966,7 +1964,7 @@ Type code C Type Minimum size in bytes \n\ 'b' signed integer 1 \n\ 'B' unsigned integer 1 \n\ - 'u' Unicode character 2 \n\ + 'u' Unicode character 2 (see note) \n\ 'h' signed integer 2 \n\ 'H' unsigned integer 2 \n\ 'i' signed integer 2 \n\ @@ -1977,6 +1975,9 @@ 'f' floating point 4 \n\ 'd' floating point 8 \n\ \n\ +NOTE: The 'u' typecode corresponds to Python's unicode character. On \n\ +narrow builds this is 2-bytes on wide builds this is 4-bytes.\n\ +\n\ The constructor is:\n\ \n\ array(typecode [, initializer]) -- create a new array\n\ @@ -2168,6 +2169,10 @@ initarray(void) { PyObject *m; + PyObject *typecodes; + Py_ssize_t size = 0; + register Py_UNICODE *p; + struct arraydescr *descr; if (PyType_Ready(&Arraytype) < 0) return; @@ -2180,5 +2185,18 @@ PyModule_AddObject(m, "ArrayType", (PyObject *)&Arraytype); Py_INCREF((PyObject *)&Arraytype); PyModule_AddObject(m, "array", (PyObject *)&Arraytype); + + for (descr=descriptors; descr->typecode != '\0'; descr++) { + size++; + } + + typecodes = PyUnicode_FromStringAndSize(NULL, size); + p = PyUnicode_AS_UNICODE(typecodes); + for (descr = descriptors; descr->typecode != '\0'; descr++) { + *p++ = (char)descr->typecode; + } + + PyModule_AddObject(m, "typecodes", (PyObject *)typecodes); + /* No need to check the error here, the caller will do that */ } Modified: python/branches/py3k-importlib/Modules/posixmodule.c ============================================================================== --- python/branches/py3k-importlib/Modules/posixmodule.c (original) +++ python/branches/py3k-importlib/Modules/posixmodule.c Mon Oct 15 21:39:57 2007 @@ -2135,10 +2135,12 @@ FILEFINDBUF3 ep; APIRET rc; - if (!PyArg_ParseTuple(args, "t#:listdir", &name, &len)) + if (!PyArg_ParseTuple(args, "et#:listdir", + Py_FileSystemDefaultEncoding, &name, &len)) return NULL; if (len >= MAX_PATH) { - PyErr_SetString(PyExc_ValueError, "path too long"); + PyMem_Free(name); + PyErr_SetString(PyExc_ValueError, "path too long"); return NULL; } strcpy(namebuf, name); @@ -2149,8 +2151,10 @@ namebuf[len++] = SEP; strcpy(namebuf + len, "*.*"); - if ((d = PyList_New(0)) == NULL) + if ((d = PyList_New(0)) == NULL) { + PyMem_Free(name); return NULL; + } rc = DosFindFirst(namebuf, /* Wildcard Pattern to Match */ &hdir, /* Handle to Use While Search Directory */ @@ -2161,7 +2165,7 @@ if (rc != NO_ERROR) { errno = ENOENT; - return posix_error_with_filename(name); + return posix_error_with_allocated_filename(name); } if (srchcnt > 0) { /* If Directory is NOT Totally Empty, */ @@ -2191,6 +2195,7 @@ } while (DosFindNext(hdir, &ep, sizeof(ep), &srchcnt) == NO_ERROR && srchcnt > 0); } + PyMem_Free(name); return d; #else @@ -5365,7 +5370,7 @@ #endif if (name == NULL) return PyErr_NoMemory(); - result = PyString_FromString(name); + result = PyUnicode_DecodeFSDefault(name); free(name); return result; } @@ -5423,7 +5428,7 @@ Py_XDECREF(err); return NULL; } - return PyString_FromString(buffer); + return PyUnicode_DecodeFSDefault(buffer); } #endif Modified: python/branches/py3k-importlib/Modules/pyexpat.c ============================================================================== --- python/branches/py3k-importlib/Modules/pyexpat.c (original) +++ python/branches/py3k-importlib/Modules/pyexpat.c Mon Oct 15 21:39:57 2007 @@ -232,13 +232,13 @@ code = PyString_FromString(""); if (code == NULL) goto failed; - name = PyString_FromString(func_name); + name = PyUnicode_FromString(func_name); if (name == NULL) goto failed; nulltuple = PyTuple_New(0); if (nulltuple == NULL) goto failed; - filename = PyString_FromString(__FILE__); + filename = PyUnicode_DecodeFSDefault(__FILE__); handler_info[slot].tb_code = PyCode_New(0, /* argcount */ 0, /* kwonlyargcount */ Modified: python/branches/py3k-importlib/Objects/abstract.c ============================================================================== --- python/branches/py3k-importlib/Objects/abstract.c (original) +++ python/branches/py3k-importlib/Objects/abstract.c Mon Oct 15 21:39:57 2007 @@ -220,11 +220,11 @@ */ int PyObject_AsCharBuffer(PyObject *obj, - const char **buffer, - Py_ssize_t *buffer_len) + const char **buffer, + Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -235,30 +235,30 @@ PyErr_SetString(PyExc_TypeError, "expected an object with the buffer interface"); return -1; - } - if ((*pb->bf_getbuffer)(obj, &view, PyBUF_CHARACTER)) return -1; + } + if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } int PyObject_CheckReadBuffer(PyObject *obj) { - PyBufferProcs *pb = obj->ob_type->tp_as_buffer; + PyBufferProcs *pb = obj->ob_type->tp_as_buffer; if (pb == NULL || pb->bf_getbuffer == NULL) - return 0; - if ((*pb->bf_getbuffer)(obj, NULL, PyBUF_SIMPLE) == -1) { - PyErr_Clear(); - return 0; - } - if (*pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, NULL); + return 0; + if ((*pb->bf_getbuffer)(obj, NULL, PyBUF_SIMPLE) == -1) { + PyErr_Clear(); + return 0; + } + if (*pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, NULL); return 1; } @@ -267,7 +267,7 @@ Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -275,18 +275,18 @@ } pb = obj->ob_type->tp_as_buffer; if (pb == NULL || - pb->bf_getbuffer == NULL) { + pb->bf_getbuffer == NULL) { PyErr_SetString(PyExc_TypeError, "expected an object with a buffer interface"); return -1; } - if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; + if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1; *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } @@ -295,7 +295,7 @@ Py_ssize_t *buffer_len) { PyBufferProcs *pb; - Py_buffer view; + Py_buffer view; if (obj == NULL || buffer == NULL || buffer_len == NULL) { null_error(); @@ -303,17 +303,17 @@ } pb = obj->ob_type->tp_as_buffer; if (pb == NULL || - pb->bf_getbuffer == NULL || - ((*pb->bf_getbuffer)(obj, &view, PyBUF_WRITABLE) != 0)) { + pb->bf_getbuffer == NULL || + ((*pb->bf_getbuffer)(obj, &view, PyBUF_WRITABLE) != 0)) { PyErr_SetString(PyExc_TypeError, - "expected an object with a writable buffer interface"); + "expected an object with a writable buffer interface"); return -1; } *buffer = view.buf; *buffer_len = view.len; - if (pb->bf_releasebuffer != NULL) - (*pb->bf_releasebuffer)(obj, &view); + if (pb->bf_releasebuffer != NULL) + (*pb->bf_releasebuffer)(obj, &view); return 0; } @@ -322,128 +322,128 @@ int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) { - if (!PyObject_CheckBuffer(obj)) { - PyErr_SetString(PyExc_TypeError, - "object does not have the buffer interface"); - return -1; - } - return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags); + if (!PyObject_CheckBuffer(obj)) { + PyErr_SetString(PyExc_TypeError, + "object does not have the buffer interface"); + return -1; + } + return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags); } void PyObject_ReleaseBuffer(PyObject *obj, Py_buffer *view) { - if (obj->ob_type->tp_as_buffer != NULL && - obj->ob_type->tp_as_buffer->bf_releasebuffer != NULL) { - (*(obj->ob_type->tp_as_buffer->bf_releasebuffer))(obj, view); - } + if (obj->ob_type->tp_as_buffer != NULL && + obj->ob_type->tp_as_buffer->bf_releasebuffer != NULL) { + (*(obj->ob_type->tp_as_buffer->bf_releasebuffer))(obj, view); + } } static int _IsFortranContiguous(Py_buffer *view) { - Py_ssize_t sd, dim; - int i; - - if (view->ndim == 0) return 1; - if (view->strides == NULL) return (view->ndim == 1); - - sd = view->itemsize; - if (view->ndim == 1) return (view->shape[0] == 1 || - sd == view->strides[0]); - for (i=0; indim; i++) { - dim = view->shape[i]; - if (dim == 0) return 1; - if (view->strides[i] != sd) return 0; - sd *= dim; - } - return 1; + Py_ssize_t sd, dim; + int i; + + if (view->ndim == 0) return 1; + if (view->strides == NULL) return (view->ndim == 1); + + sd = view->itemsize; + if (view->ndim == 1) return (view->shape[0] == 1 || + sd == view->strides[0]); + for (i=0; indim; i++) { + dim = view->shape[i]; + if (dim == 0) return 1; + if (view->strides[i] != sd) return 0; + sd *= dim; + } + return 1; } static int _IsCContiguous(Py_buffer *view) { - Py_ssize_t sd, dim; - int i; - - if (view->ndim == 0) return 1; - if (view->strides == NULL) return 1; - - sd = view->itemsize; - if (view->ndim == 1) return (view->shape[0] == 1 || - sd == view->strides[0]); - for (i=view->ndim-1; i>=0; i--) { - dim = view->shape[i]; - if (dim == 0) return 1; - if (view->strides[i] != sd) return 0; - sd *= dim; - } - return 1; + Py_ssize_t sd, dim; + int i; + + if (view->ndim == 0) return 1; + if (view->strides == NULL) return 1; + + sd = view->itemsize; + if (view->ndim == 1) return (view->shape[0] == 1 || + sd == view->strides[0]); + for (i=view->ndim-1; i>=0; i--) { + dim = view->shape[i]; + if (dim == 0) return 1; + if (view->strides[i] != sd) return 0; + sd *= dim; + } + return 1; } int PyBuffer_IsContiguous(Py_buffer *view, char fort) { - if (view->suboffsets != NULL) return 0; + if (view->suboffsets != NULL) return 0; - if (fort == 'C') - return _IsCContiguous(view); - else if (fort == 'F') - return _IsFortranContiguous(view); - else if (fort == 'A') - return (_IsCContiguous(view) || _IsFortranContiguous(view)); - return 0; + if (fort == 'C') + return _IsCContiguous(view); + else if (fort == 'F') + return _IsFortranContiguous(view); + else if (fort == 'A') + return (_IsCContiguous(view) || _IsFortranContiguous(view)); + return 0; } void* PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) { - char* pointer; - int i; - pointer = (char *)view->buf; - for (i = 0; i < view->ndim; i++) { - pointer += view->strides[i]*indices[i]; - if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { - pointer = *((char**)pointer) + view->suboffsets[i]; - } - } - return (void*)pointer; + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; } void _add_one_to_index_F(int nd, Py_ssize_t *index, Py_ssize_t *shape) { - int k; - - for (k=0; k=0; k--) { - if (index[k] < shape[k]-1) { - index[k]++; - break; - } - else { - index[k] = 0; - } - } + for (k=nd-1; k>=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } } /* view is not checked for consistency in either of these. It is @@ -454,235 +454,235 @@ int PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) { - int k; - void (*addone)(int, Py_ssize_t *, Py_ssize_t *); - Py_ssize_t *indices, elements; - char *dest, *ptr; - - if (len > view->len) { - len = view->len; - } - - if (PyBuffer_IsContiguous(view, fort)) { - /* simplest copy is all that is needed */ - memcpy(buf, view->buf, len); - return 0; - } + int k; + void (*addone)(int, Py_ssize_t *, Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } - /* Otherwise a more elaborate scheme is needed */ - + /* Otherwise a more elaborate scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); - if (indices == NULL) { - PyErr_NoMemory(); - return -1; - } - for (k=0; kndim;k++) { - indices[k] = 0; - } - - if (fort == 'F') { - addone = _add_one_to_index_F; - } - else { - addone = _add_one_to_index_C; - } - dest = buf; - /* XXX : This is not going to be the fastest code in the world - several optimizations are possible. - */ - elements = len / view->itemsize; - while (elements--) { - addone(view->ndim, indices, view->shape); - ptr = PyBuffer_GetPointer(view, indices); - memcpy(dest, ptr, view->itemsize); - dest += view->itemsize; - } - PyMem_Free(indices); - return 0; + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _add_one_to_index_F; + } + else { + addone = _add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + addone(view->ndim, indices, view->shape); + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + } + PyMem_Free(indices); + return 0; } int PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) { - int k; - void (*addone)(int, Py_ssize_t *, Py_ssize_t *); - Py_ssize_t *indices, elements; - char *src, *ptr; - - if (len > view->len) { - len = view->len; - } - - if (PyBuffer_IsContiguous(view, fort)) { - /* simplest copy is all that is needed */ - memcpy(view->buf, buf, len); - return 0; - } + int k; + void (*addone)(int, Py_ssize_t *, Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } - /* Otherwise a more elaborate scheme is needed */ - + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); - if (indices == NULL) { - PyErr_NoMemory(); - return -1; - } - for (k=0; kndim;k++) { - indices[k] = 0; - } - - if (fort == 'F') { - addone = _add_one_to_index_F; - } - else { - addone = _add_one_to_index_C; - } - src = buf; - /* XXX : This is not going to be the fastest code in the world - several optimizations are possible. - */ - elements = len / view->itemsize; - while (elements--) { - addone(view->ndim, indices, view->shape); - ptr = PyBuffer_GetPointer(view, indices); - memcpy(ptr, src, view->itemsize); - src += view->itemsize; - } - - PyMem_Free(indices); - return 0; + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _add_one_to_index_F; + } + else { + addone = _add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + addone(view->ndim, indices, view->shape); + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + } + + PyMem_Free(indices); + return 0; } int PyObject_CopyData(PyObject *dest, PyObject *src) { - Py_buffer view_dest, view_src; - int k; - Py_ssize_t *indices, elements; - char *dptr, *sptr; - - if (!PyObject_CheckBuffer(dest) || - !PyObject_CheckBuffer(src)) { - PyErr_SetString(PyExc_TypeError, - "both destination and source must have the "\ - "buffer interface"); - return -1; - } - - if (PyObject_GetBuffer(dest, &view_dest, PyBUF_FULL) != 0) return -1; - if (PyObject_GetBuffer(src, &view_src, PyBUF_FULL_RO) != 0) { - PyObject_ReleaseBuffer(dest, &view_dest); - return -1; - } - - if (view_dest.len < view_src.len) { - PyErr_SetString(PyExc_BufferError, - "destination is too small to receive data from source"); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return -1; - } - - if ((PyBuffer_IsContiguous(&view_dest, 'C') && - PyBuffer_IsContiguous(&view_src, 'C')) || - (PyBuffer_IsContiguous(&view_dest, 'F') && - PyBuffer_IsContiguous(&view_src, 'F'))) { - /* simplest copy is all that is needed */ - memcpy(view_dest.buf, view_src.buf, view_src.len); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return 0; - } + Py_buffer view_dest, view_src; + int k; + Py_ssize_t *indices, elements; + char *dptr, *sptr; + + if (!PyObject_CheckBuffer(dest) || + !PyObject_CheckBuffer(src)) { + PyErr_SetString(PyExc_TypeError, + "both destination and source must have the "\ + "buffer interface"); + return -1; + } + + if (PyObject_GetBuffer(dest, &view_dest, PyBUF_FULL) != 0) return -1; + if (PyObject_GetBuffer(src, &view_src, PyBUF_FULL_RO) != 0) { + PyObject_ReleaseBuffer(dest, &view_dest); + return -1; + } + + if (view_dest.len < view_src.len) { + PyErr_SetString(PyExc_BufferError, + "destination is too small to receive data from source"); + PyObject_ReleaseBuffer(dest, &view_dest); + PyObject_ReleaseBuffer(src, &view_src); + return -1; + } + + if ((PyBuffer_IsContiguous(&view_dest, 'C') && + PyBuffer_IsContiguous(&view_src, 'C')) || + (PyBuffer_IsContiguous(&view_dest, 'F') && + PyBuffer_IsContiguous(&view_src, 'F'))) { + /* simplest copy is all that is needed */ + memcpy(view_dest.buf, view_src.buf, view_src.len); + PyObject_ReleaseBuffer(dest, &view_dest); + PyObject_ReleaseBuffer(src, &view_src); + return 0; + } - /* Otherwise a more elaborate copy scheme is needed */ - + /* Otherwise a more elaborate copy scheme is needed */ + /* XXX(nnorwitz): need to check for overflow! */ - indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*view_src.ndim); - if (indices == NULL) { - PyErr_NoMemory(); - PyObject_ReleaseBuffer(dest, &view_dest); - PyObject_ReleaseBuffer(src, &view_src); - return -1; - } - for (k=0; k=0; k--) { - strides[k] = sd; - sd *= shape[k]; - } - } - return; + int k; + Py_ssize_t sd; + + sd = itemsize; + if (fort == 'F') { + for (k=0; k=0; k--) { + strides[k] = sd; + sd *= shape[k]; + } + } + return; } int PyBuffer_FillInfo(Py_buffer *view, void *buf, Py_ssize_t len, - int readonly, int flags) -{ - if (view == NULL) return 0; - if (((flags & PyBUF_LOCKDATA) == PyBUF_LOCKDATA) && - readonly != -1) { - PyErr_SetString(PyExc_BufferError, - "Cannot make this object read-only."); - return -1; - } - if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && - readonly == 1) { - PyErr_SetString(PyExc_BufferError, - "Object is not writable."); - return -1; - } - - view->buf = buf; - view->len = len; - view->readonly = readonly; - view->itemsize = 1; - view->format = NULL; - if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) - view->format = "B"; - view->ndim = 1; - view->shape = NULL; - if ((flags & PyBUF_ND) == PyBUF_ND) - view->shape = &(view->len); - view->strides = NULL; - if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) - view->strides = &(view->itemsize); - view->suboffsets = NULL; - view->internal = NULL; - return 0; + int readonly, int flags) +{ + if (view == NULL) return 0; + if (((flags & PyBUF_LOCK) == PyBUF_LOCK) && + readonly >= 0) { + PyErr_SetString(PyExc_BufferError, + "Cannot lock this object."); + return -1; + } + if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && + (readonly == 1)) { + PyErr_SetString(PyExc_BufferError, + "Object is not writable."); + return -1; + } + + view->buf = buf; + view->len = len; + view->readonly = readonly; + view->itemsize = 1; + view->format = NULL; + if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) + view->format = "B"; + view->ndim = 1; + view->shape = NULL; + if ((flags & PyBUF_ND) == PyBUF_ND) + view->shape = &(view->len); + view->strides = NULL; + if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) + view->strides = &(view->itemsize); + view->suboffsets = NULL; + view->internal = NULL; + return 0; } /* Operations on numbers */ @@ -1025,7 +1025,7 @@ Py_DECREF(result); if (m != NULL) { binaryfunc f = NULL; - f = m->sq_inplace_concat; + f = m->sq_inplace_concat; if (f == NULL) f = m->sq_concat; if (f != NULL) @@ -1304,8 +1304,8 @@ PyObject *res = m->nb_float(o); if (res && !PyFloat_Check(res)) { PyErr_Format(PyExc_TypeError, - "__float__ returned non-float (type %.200s)", - res->ob_type->tp_name); + "__float__ returned non-float (type %.200s)", + res->ob_type->tp_name); Py_DECREF(res); return NULL; } @@ -1384,7 +1384,7 @@ return m->sq_concat(s, o); /* Instances of user classes defining an __add__() method only - have an nb_add slot, not an sq_concat slot. So we fall back + have an nb_add slot, not an sq_concat slot. So we fall back to nb_add if both arguments appear to be sequences. */ if (PySequence_Check(s) && PySequence_Check(o)) { PyObject *result = binary_op1(s, o, NB_SLOT(nb_add)); @@ -1629,7 +1629,7 @@ PySequence_Tuple(PyObject *v) { PyObject *it; /* iter(v) */ - Py_ssize_t n; /* guess for result tuple size */ + Py_ssize_t n; /* guess for result tuple size */ PyObject *result; Py_ssize_t j; @@ -1662,7 +1662,7 @@ return NULL; } PyErr_Clear(); - n = 10; /* arbitrary */ + n = 10; /* arbitrary */ } result = PyTuple_New(n); if (result == NULL) @@ -1718,7 +1718,7 @@ PySequence_List(PyObject *v) { PyObject *result; /* result list */ - PyObject *rv; /* return value from PyList_Extend */ + PyObject *rv; /* return value from PyList_Extend */ if (v == NULL) return null_error(); @@ -1749,7 +1749,7 @@ return v; } - it = PyObject_GetIter(v); + it = PyObject_GetIter(v); if (it == NULL) { if (PyErr_ExceptionMatches(PyExc_TypeError)) PyErr_SetString(PyExc_TypeError, m); @@ -1763,9 +1763,9 @@ } /* Iterate over seq. Result depends on the operation: - PY_ITERSEARCH_COUNT: -1 if error, else # of times obj appears in seq. - PY_ITERSEARCH_INDEX: 0-based index of first occurence of obj in seq; - set ValueError and return -1 if none found; also return -1 on error. + PY_ITERSEARCH_COUNT: -1 if error, else # of times obj appears in seq. + PY_ITERSEARCH_INDEX: 0-based index of first occurence of obj in seq; + set ValueError and return -1 if none found; also return -1 on error. Py_ITERSEARCH_CONTAINS: return 1 if obj in seq, else 0; -1 on error. */ Py_ssize_t @@ -1839,7 +1839,7 @@ goto Done; PyErr_SetString(PyExc_ValueError, - "sequence.index(x): x not in sequence"); + "sequence.index(x): x not in sequence"); /* fall into failure code */ Fail: n = -1; @@ -1865,7 +1865,7 @@ { Py_ssize_t result; PySequenceMethods *sqm = seq->ob_type->tp_as_sequence; - if (sqm != NULL && sqm->sq_contains != NULL) + if (sqm != NULL && sqm->sq_contains != NULL) return (*sqm->sq_contains)(seq, ob); result = _PySequence_IterSearch(seq, ob, PY_ITERSEARCH_CONTAINS); return Py_SAFE_DOWNCAST(result, Py_ssize_t, int); @@ -1890,7 +1890,7 @@ int PyMapping_Check(PyObject *o) { - return o && o->ob_type->tp_as_mapping && + return o && o->ob_type->tp_as_mapping && o->ob_type->tp_as_mapping->mp_subscript; } @@ -2044,7 +2044,7 @@ PyObject * PyObject_Call(PyObject *func, PyObject *arg, PyObject *kw) { - ternaryfunc call; + ternaryfunc call; if ((call = func->ob_type->tp_call) != NULL) { PyObject *result; @@ -2308,7 +2308,7 @@ * produce exactly the same results: NULL is returned and no error is set. * * If some exception other than AttributeError is raised, then NULL is also - * returned, but the exception is not cleared. That's because we want the + * returned, but the exception is not cleared. That's because we want the * exception to be propagated along. * * Callers are expected to test for PyErr_Occurred() when the return value @@ -2336,7 +2336,7 @@ return NULL; } if (!PyTuple_Check(bases)) { - Py_DECREF(bases); + Py_DECREF(bases); return NULL; } return bases; @@ -2428,18 +2428,18 @@ else if (PyTuple_Check(cls)) { Py_ssize_t i, n; - if (!recursion_depth) { - PyErr_SetString(PyExc_RuntimeError, - "nest level of tuple too deep"); - return -1; - } + if (!recursion_depth) { + PyErr_SetString(PyExc_RuntimeError, + "nest level of tuple too deep"); + return -1; + } n = PyTuple_GET_SIZE(cls); for (i = 0; i < n; i++) { retval = recursive_isinstance( - inst, - PyTuple_GET_ITEM(cls, i), - recursion_depth-1); + inst, + PyTuple_GET_ITEM(cls, i), + recursion_depth-1); if (retval != 0) break; } @@ -2490,12 +2490,12 @@ return recursive_isinstance(inst, cls, Py_GetRecursionLimit()); } -static int +static int recursive_issubclass(PyObject *derived, PyObject *cls, int recursion_depth) { int retval; - { + { if (!check_class(derived, "issubclass() arg 1 must be a class")) return -1; @@ -2504,16 +2504,16 @@ Py_ssize_t i; Py_ssize_t n = PyTuple_GET_SIZE(cls); - if (!recursion_depth) { - PyErr_SetString(PyExc_RuntimeError, - "nest level of tuple too deep"); - return -1; - } + if (!recursion_depth) { + PyErr_SetString(PyExc_RuntimeError, + "nest level of tuple too deep"); + return -1; + } for (i = 0; i < n; ++i) { retval = recursive_issubclass( - derived, - PyTuple_GET_ITEM(cls, i), - recursion_depth-1); + derived, + PyTuple_GET_ITEM(cls, i), + recursion_depth-1); if (retval != 0) { /* either found it, or got an error */ return retval; @@ -2590,7 +2590,7 @@ * If the iteration terminates normally, return NULL and clear the * PyExc_StopIteration exception (if it was set). PyErr_Occurred() * will be false. - * Else return the next object. PyErr_Occurred() will be false. + * Else return the next object. PyErr_Occurred() will be false. */ PyObject * PyIter_Next(PyObject *iter) Modified: python/branches/py3k-importlib/Objects/codeobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/codeobject.c (original) +++ python/branches/py3k-importlib/Objects/codeobject.c Mon Oct 15 21:39:57 2007 @@ -50,6 +50,7 @@ { PyCodeObject *co; Py_ssize_t i; + /* Check argument types */ if (argcount < 0 || nlocals < 0 || code == NULL || @@ -58,20 +59,16 @@ varnames == NULL || !PyTuple_Check(varnames) || freevars == NULL || !PyTuple_Check(freevars) || cellvars == NULL || !PyTuple_Check(cellvars) || - name == NULL || (!PyString_Check(name) && !PyUnicode_Check(name)) || - filename == NULL || !PyString_Check(filename) || + name == NULL || !PyUnicode_Check(name) || + filename == NULL || !PyUnicode_Check(filename) || lnotab == NULL || !PyString_Check(lnotab) || !PyObject_CheckReadBuffer(code)) { PyErr_BadInternalCall(); return NULL; } - if (PyString_Check(name)) { - name = PyUnicode_FromString(PyString_AS_STRING(name)); - if (name == NULL) - return NULL; - } else { - Py_INCREF(name); - } + Py_INCREF(name); + Py_INCREF(filename); + intern_strings(names); intern_strings(varnames); intern_strings(freevars); @@ -299,8 +296,8 @@ if (co->co_firstlineno != 0) lineno = co->co_firstlineno; - if (co->co_filename && PyString_Check(co->co_filename)) - filename = PyString_AS_STRING(co->co_filename); + if (co->co_filename && PyUnicode_Check(co->co_filename)) + filename = PyUnicode_AsString(co->co_filename); return PyUnicode_FromFormat( "", co->co_name, co, filename, lineno); Modified: python/branches/py3k-importlib/Objects/memoryobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/memoryobject.c (original) +++ python/branches/py3k-importlib/Objects/memoryobject.c Mon Oct 15 21:39:57 2007 @@ -196,7 +196,9 @@ a contiguous buffer if it is not. The view will point to the shadow buffer which can be written to and then will be copied back into the other buffer when the memory - view is de-allocated. + view is de-allocated. While the shadow buffer is + being used, it will have an exclusive write lock on + the original buffer. */ PyObject * @@ -224,7 +226,7 @@ flags = PyBUF_FULL; break; case PyBUF_SHADOW: - flags = PyBUF_FULL_LCK; + flags = PyBUF_FULL_XLCK; break; } @@ -431,11 +433,7 @@ static PyObject * memory_repr(PyMemoryViewObject *self) { - /* XXX(nnorwitz): the code should be different or remove condition. */ - if (self->base == NULL) - return PyUnicode_FromFormat("", self); - else - return PyUnicode_FromFormat("", self); + return PyUnicode_FromFormat("", self); } @@ -502,6 +500,14 @@ /* Return a bytes object */ char *ptr; ptr = (char *)view->buf; + if (result < 0) { + result += view->shape[0]; + } + if ((result < 0) || (result > view->shape[0])) { + PyErr_SetString(PyExc_IndexError, + "index out of bounds"); + return NULL; + } if (view->strides == NULL) ptr += view->itemsize * result; else @@ -517,14 +523,20 @@ /* Return a new memory-view object */ Py_buffer newview; memset(&newview, 0, sizeof(newview)); + /* XXX: This needs to be fixed so it + actually returns a sub-view + */ return PyMemoryView_FromMemory(&newview); } } + /* Need to support getting a sliced view */ Py_INCREF(Py_NotImplemented); return Py_NotImplemented; } + +/* Need to support assigning memory if we can */ static int memory_ass_sub(PyMemoryViewObject *self, PyObject *key, PyObject *value) { Modified: python/branches/py3k-importlib/Objects/moduleobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/moduleobject.c (original) +++ python/branches/py3k-importlib/Objects/moduleobject.c Mon Oct 15 21:39:57 2007 @@ -86,12 +86,12 @@ d = ((PyModuleObject *)m)->md_dict; if (d == NULL || (fileobj = PyDict_GetItemString(d, "__file__")) == NULL || - !PyString_Check(fileobj)) + !PyUnicode_Check(fileobj)) { PyErr_SetString(PyExc_SystemError, "module filename missing"); return NULL; } - return PyString_AsString(fileobj); + return PyUnicode_AsString(fileobj); } void Modified: python/branches/py3k-importlib/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-importlib/Objects/unicodeobject.c (original) +++ python/branches/py3k-importlib/Objects/unicodeobject.c Mon Oct 15 21:39:57 2007 @@ -117,7 +117,11 @@ /* Default encoding to use and assume when NULL is passed as encoding parameter; it is fixed to "utf-8". Always use the - PyUnicode_GetDefaultEncoding() API to access this global. */ + PyUnicode_GetDefaultEncoding() API to access this global. + + Don't forget to alter Py_FileSystemDefaultEncoding() if you change the + hard coded default! +*/ static const char unicode_default_encoding[] = "utf-8"; Py_UNICODE @@ -1231,6 +1235,35 @@ return v; } +PyObject* +PyUnicode_DecodeFSDefault(const char *s) +{ + Py_ssize_t size = (Py_ssize_t)strlen(s); + + /* During the early bootstrapping process, Py_FileSystemDefaultEncoding + can be undefined. If it is case, decode using UTF-8. The following assumes + that Py_FileSystemDefaultEncoding is set to a built-in encoding during the + bootstrapping process where the codecs aren't ready yet. + */ + if (Py_FileSystemDefaultEncoding) { +#if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T) + if (strcmp(Py_FileSystemDefaultEncoding, "mbcs")) { + return PyUnicode_DecodeMBCS(s, size, "replace"); + } +#elif defined(__APPLE__) + if (strcmp(Py_FileSystemDefaultEncoding, "utf-8")) { + return PyUnicode_DecodeUTF8(s, size, "replace"); + } +#endif + return PyUnicode_Decode(s, size, + Py_FileSystemDefaultEncoding, + "replace"); + } + else { + return PyUnicode_DecodeUTF8(s, size, "replace"); + } +} + char* PyUnicode_AsStringAndSize(PyObject *unicode, Py_ssize_t *psize) { @@ -8113,19 +8146,6 @@ }; -static int -unicode_buffer_getbuffer(PyUnicodeObject *self, Py_buffer *view, int flags) -{ - - if (flags & PyBUF_CHARACTER) { - PyErr_SetString(PyExc_SystemError, "can't use str as char buffer"); - return -1; - } - return PyBuffer_FillInfo(view, (void *)self->str, - PyUnicode_GET_DATA_SIZE(self), 1, flags); -} - - /* Helpers for PyUnicode_Format() */ static PyObject * @@ -8819,11 +8839,6 @@ return NULL; } -static PyBufferProcs unicode_as_buffer = { - (getbufferproc) unicode_buffer_getbuffer, - NULL, -}; - static PyObject * unicode_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds); @@ -8907,7 +8922,7 @@ (reprfunc) unicode_str, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ - &unicode_as_buffer, /* tp_as_buffer */ + 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_UNICODE_SUBCLASS, /* tp_flags */ unicode_doc, /* tp_doc */ Modified: python/branches/py3k-importlib/PC/VC6/pythoncore.dsp ============================================================================== --- python/branches/py3k-importlib/PC/VC6/pythoncore.dsp (original) +++ python/branches/py3k-importlib/PC/VC6/pythoncore.dsp Mon Oct 15 21:39:57 2007 @@ -137,10 +137,6 @@ # End Source File # Begin Source File -SOURCE=..\..\Modules\_hotshot.c -# End Source File -# Begin Source File - SOURCE=..\..\Modules\_localemodule.c # End Source File # Begin Source File Modified: python/branches/py3k-importlib/PC/config.c ============================================================================== --- python/branches/py3k-importlib/PC/config.c (original) +++ python/branches/py3k-importlib/PC/config.c Mon Oct 15 21:39:57 2007 @@ -99,7 +99,6 @@ {"_codecs", init_codecs}, {"_weakref", init_weakref}, - {"_hotshot", init_hotshot}, {"_random", init_random}, {"_bisect", init_bisect}, {"_heapq", init_heapq}, Modified: python/branches/py3k-importlib/PC/os2emx/Makefile ============================================================================== --- python/branches/py3k-importlib/PC/os2emx/Makefile (original) +++ python/branches/py3k-importlib/PC/os2emx/Makefile Mon Oct 15 21:39:57 2007 @@ -437,8 +437,7 @@ # Python modules to be dynamically loaded that need explicit build rules # (either multiple source files and/or non-standard module naming) # (NOTE: use shortened names for modules affected by 8 char name limit) -HARDEXTMODULES= _hotshot \ - _socket \ +HARDEXTMODULES= _socket \ _testcap \ unicoded @@ -567,9 +566,6 @@ # awkward handling (due e.g. to non-std naming, or multiple source files) # - standard modules -_hotshot$(MODULE.EXT): $(OUT)_hotshot$O $(OUT)_hotshot_m.def $(PYTHON.IMPLIB) - $(LD) $(LDFLAGS.DLL) -o $@ $(^^) $(L^) $(LIBS) - _socket$(MODULE.EXT): $(OUT)socketmodule$O $(OUT)_socket_m.def $(PYTHON.IMPLIB) $(LD) $(LDFLAGS.DLL) -o $@ $(^^) $(L^) $(LIBS) Modified: python/branches/py3k-importlib/PC/os2emx/config.c ============================================================================== --- python/branches/py3k-importlib/PC/os2emx/config.c (original) +++ python/branches/py3k-importlib/PC/os2emx/config.c Mon Oct 15 21:39:57 2007 @@ -75,7 +75,6 @@ #if !HAVE_DYNAMIC_LOADING extern void init_curses(); extern void init_curses_panel(); -extern void init_hotshot(); extern void init_testcapi(); extern void initbz2(); extern void initfpectl(); @@ -134,7 +133,6 @@ #if !HAVE_DYNAMIC_LOADING {"_curses", init_curses}, {"_curses_panel", init_curses_panel}, - {"_hotshot", init_hotshot}, {"_testcapi", init_testcapi}, {"bz2", initbz2}, {"fpectl", initfpectl}, Modified: python/branches/py3k-importlib/PCbuild8/pythoncore/pythoncore.vcproj ============================================================================== --- python/branches/py3k-importlib/PCbuild8/pythoncore/pythoncore.vcproj (original) +++ python/branches/py3k-importlib/PCbuild8/pythoncore/pythoncore.vcproj Mon Oct 15 21:39:57 2007 @@ -1414,10 +1414,6 @@ > - - Modified: python/branches/py3k-importlib/Python/bltinmodule.c ============================================================================== --- python/branches/py3k-importlib/Python/bltinmodule.c (original) +++ python/branches/py3k-importlib/Python/bltinmodule.c Mon Oct 15 21:39:57 2007 @@ -10,6 +10,9 @@ /* The default encoding used by the platform file system APIs Can remain NULL for all platforms that don't have such a concept + + Don't forget to modify PyUnicode_DecodeFSDefault() if you touch any of the + values for Py_FileSystemDefaultEncoding! */ #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T) const char *Py_FileSystemDefaultEncoding = "mbcs"; Modified: python/branches/py3k-importlib/Python/ceval.c ============================================================================== --- python/branches/py3k-importlib/Python/ceval.c (original) +++ python/branches/py3k-importlib/Python/ceval.c Mon Oct 15 21:39:57 2007 @@ -767,7 +767,7 @@ lltrace = PyDict_GetItemString(f->f_globals, "__lltrace__") != NULL; #endif #if defined(Py_DEBUG) || defined(LLTRACE) - filename = PyString_AsString(co->co_filename); + filename = PyUnicode_AsString(co->co_filename); #endif why = WHY_NOT; @@ -2565,7 +2565,7 @@ if (argcount > co->co_argcount) { if (!(co->co_flags & CO_VARARGS)) { PyErr_Format(PyExc_TypeError, - "%S() takes %s %d " + "%U() takes %s %d " "%spositional argument%s (%d given)", co->co_name, defcount ? "at most" : "exactly", @@ -2599,7 +2599,7 @@ int j; if (keyword == NULL || !PyUnicode_Check(keyword)) { PyErr_Format(PyExc_TypeError, - "%S() keywords must be strings", + "%U() keywords must be strings", co->co_name); goto fail; } @@ -2622,7 +2622,7 @@ if (j >= co->co_argcount + co->co_kwonlyargcount) { if (kwdict == NULL) { PyErr_Format(PyExc_TypeError, - "%S() got an unexpected " + "%U() got an unexpected " "keyword argument '%S'", co->co_name, keyword); @@ -2633,7 +2633,7 @@ else { if (GETLOCAL(j) != NULL) { PyErr_Format(PyExc_TypeError, - "%S() got multiple " + "%U() got multiple " "values for keyword " "argument '%S'", co->co_name, @@ -2661,7 +2661,7 @@ continue; } PyErr_Format(PyExc_TypeError, - "%S() needs keyword-only argument %S", + "%U() needs keyword-only argument %S", co->co_name, name); goto fail; } @@ -2671,7 +2671,7 @@ for (i = argcount; i < m; i++) { if (GETLOCAL(i) == NULL) { PyErr_Format(PyExc_TypeError, - "%S() takes %s %d " + "%U() takes %s %d " "%spositional argument%s " "(%d given)", co->co_name, @@ -2699,7 +2699,7 @@ else { if (argcount > 0 || kwcount > 0) { PyErr_Format(PyExc_TypeError, - "%S() takes no arguments (%d given)", + "%U() takes no arguments (%d given)", co->co_name, argcount + kwcount); goto fail; Modified: python/branches/py3k-importlib/Python/compile.c ============================================================================== --- python/branches/py3k-importlib/Python/compile.c (original) +++ python/branches/py3k-importlib/Python/compile.c Mon Oct 15 21:39:57 2007 @@ -1247,7 +1247,7 @@ PyObject_REPR(name), PyString_AS_STRING(c->u->u_name), reftype, arg, - PyString_AS_STRING(co->co_name), + PyUnicode_AsString(co->co_name), PyObject_REPR(co->co_freevars)); Py_FatalError("compiler_make_closure()"); } @@ -4001,7 +4001,7 @@ freevars = dict_keys_inorder(c->u->u_freevars, PyTuple_Size(cellvars)); if (!freevars) goto error; - filename = PyString_FromString(c->c_filename); + filename = PyUnicode_DecodeFSDefault(c->c_filename); if (!filename) goto error; Modified: python/branches/py3k-importlib/Python/frozen.c ============================================================================== --- python/branches/py3k-importlib/Python/frozen.c (original) +++ python/branches/py3k-importlib/Python/frozen.c Mon Oct 15 21:39:57 2007 @@ -17,7 +17,7 @@ 131,1,0,1,100,1,0,83,40,2,0,0,0,117,14,0, 0,0,72,101,108,108,111,32,119,111,114,108,100,46,46,46, 78,40,1,0,0,0,117,5,0,0,0,112,114,105,110,116, - 40,0,0,0,0,40,0,0,0,0,40,0,0,0,0,115, + 40,0,0,0,0,40,0,0,0,0,40,0,0,0,0,117, 8,0,0,0,104,101,108,108,111,46,112,121,117,8,0,0, 0,60,109,111,100,117,108,101,62,1,0,0,0,115,0,0, 0,0, Modified: python/branches/py3k-importlib/Python/getargs.c ============================================================================== --- python/branches/py3k-importlib/Python/getargs.c (original) +++ python/branches/py3k-importlib/Python/getargs.c Mon Oct 15 21:39:57 2007 @@ -1237,7 +1237,9 @@ (*pb->bf_releasebuffer)(arg, &view); break; } - + + /*TEO: This can be eliminated --- here only for backward + compatibility */ case 't': { /* 8-bit character buffer, read-only access */ char **p = va_arg(*p_va, char **); PyBufferProcs *pb = arg->ob_type->tp_as_buffer; @@ -1250,10 +1252,10 @@ arg, msgbuf, bufsize); if (pb == NULL || pb->bf_getbuffer == NULL) return converterr( - "string or read-only character buffer", + "bytes or read-only character buffer", arg, msgbuf, bufsize); - if ((*pb->bf_getbuffer)(arg, &view, PyBUF_CHARACTER) != 0) + if ((*pb->bf_getbuffer)(arg, &view, PyBUF_SIMPLE) != 0) return converterr("string or single-segment read-only buffer", arg, msgbuf, bufsize); Modified: python/branches/py3k-importlib/Python/import.c ============================================================================== --- python/branches/py3k-importlib/Python/import.c (original) +++ python/branches/py3k-importlib/Python/import.c Mon Oct 15 21:39:57 2007 @@ -74,10 +74,11 @@ 3040 (added signature annotations) 3050 (print becomes a function) 3060 (PEP 3115 metaclass syntax) - 3070 (PEP 3109 raise changes) + 3070 (PEP 3109 raise changes) + 3080 (PEP 3137 make __file__ and __name__ unicode) . */ -#define MAGIC (3070 | ((long)'\r'<<16) | ((long)'\n'<<24)) +#define MAGIC (3080 | ((long)'\r'<<16) | ((long)'\n'<<24)) /* Magic word as global; note that _PyImport_Init() can change the value of this global to accommodate for alterations of how the @@ -744,7 +745,7 @@ /* Remember the filename as the __file__ attribute */ v = NULL; if (pathname != NULL) { - v = PyString_FromString(pathname); + v = PyUnicode_DecodeFSDefault(pathname); if (v == NULL) PyErr_Clear(); } @@ -1075,7 +1076,7 @@ PySys_WriteStderr("import %s # directory %s\n", name, pathname); d = PyModule_GetDict(m); - file = PyString_FromString(pathname); + file = PyUnicode_DecodeFSDefault(pathname); if (file == NULL) goto error; path = Py_BuildValue("[O]", file); Modified: python/branches/py3k-importlib/Python/importdl.c ============================================================================== --- python/branches/py3k-importlib/Python/importdl.c (original) +++ python/branches/py3k-importlib/Python/importdl.c Mon Oct 15 21:39:57 2007 @@ -22,6 +22,7 @@ _PyImport_LoadDynamicModule(char *name, char *pathname, FILE *fp) { PyObject *m; + PyObject *path; char *lastdot, *shortname, *packagecontext, *oldcontext; dl_funcptr p; @@ -62,7 +63,8 @@ return NULL; } /* Remember the filename as the __file__ attribute */ - if (PyModule_AddStringConstant(m, "__file__", pathname) < 0) + path = PyUnicode_DecodeFSDefault(pathname); + if (PyModule_AddObject(m, "__file__", path) < 0) PyErr_Clear(); /* Not important enough to report */ if (_PyImport_FixupExtension(name, pathname) == NULL) Modified: python/branches/py3k-importlib/Python/pythonrun.c ============================================================================== --- python/branches/py3k-importlib/Python/pythonrun.c (original) +++ python/branches/py3k-importlib/Python/pythonrun.c Mon Oct 15 21:39:57 2007 @@ -869,7 +869,8 @@ return -1; d = PyModule_GetDict(m); if (PyDict_GetItemString(d, "__file__") == NULL) { - PyObject *f = PyString_FromString(filename); + PyObject *f; + f = PyUnicode_DecodeFSDefault(filename); if (f == NULL) return -1; if (PyDict_SetItemString(d, "__file__", f) < 0) { Modified: python/branches/py3k-importlib/Python/traceback.c ============================================================================== --- python/branches/py3k-importlib/Python/traceback.c (original) +++ python/branches/py3k-importlib/Python/traceback.c Mon Oct 15 21:39:57 2007 @@ -229,10 +229,10 @@ while (tb != NULL && err == 0) { if (depth <= limit) { err = tb_displayline(f, - PyString_AsString( + PyUnicode_AsString( tb->tb_frame->f_code->co_filename), tb->tb_lineno, - PyString_AsString(tb->tb_frame->f_code->co_name)); + PyUnicode_AsString(tb->tb_frame->f_code->co_name)); } depth--; tb = tb->tb_next; Modified: python/branches/py3k-importlib/Tools/scripts/README ============================================================================== --- python/branches/py3k-importlib/Tools/scripts/README (original) +++ python/branches/py3k-importlib/Tools/scripts/README Mon Oct 15 21:39:57 2007 @@ -30,7 +30,6 @@ google.py Open a webbrowser with Google. gprof2html.py Transform gprof(1) output into useful HTML. h2py.py Translate #define's into Python assignments -hotshotmain.py Main program to run script under control of hotshot idle Main program to start IDLE ifdef.py Remove #if(n)def groups from C sources lfcr.py Change LF line endings to CRLF (Unix to Windows) Modified: python/branches/py3k-importlib/configure ============================================================================== --- python/branches/py3k-importlib/configure (original) +++ python/branches/py3k-importlib/configure Mon Oct 15 21:39:57 2007 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 57931 . +# From configure.in Revision: 58054 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.61 for python 3.0. # @@ -10589,6 +10589,468 @@ fi +{ echo "$as_me:$LINENO: checking for long double support" >&5 +echo $ECHO_N "checking for long double support... $ECHO_C" >&6; } +have_long_double=no +cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ + +int +main () +{ +long double x; x = (long double)0; + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + + +cat >>confdefs.h <<\_ACEOF +#define HAVE_LONG_DOUBLE 1 +_ACEOF + + have_long_double=yes + +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +{ echo "$as_me:$LINENO: result: $have_long_double" >&5 +echo "${ECHO_T}$have_long_double" >&6; } +if test "$have_long_double" = yes ; then +{ echo "$as_me:$LINENO: checking for long double" >&5 +echo $ECHO_N "checking for long double... $ECHO_C" >&6; } +if test "${ac_cv_type_long_double+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default +typedef long double ac__type_new_; +int +main () +{ +if ((ac__type_new_ *) 0) + return 0; +if (sizeof (ac__type_new_)) + return 0; + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_cv_type_long_double=yes +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_cv_type_long_double=no +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi +{ echo "$as_me:$LINENO: result: $ac_cv_type_long_double" >&5 +echo "${ECHO_T}$ac_cv_type_long_double" >&6; } + +# The cast to long int works around a bug in the HP C Compiler +# version HP92453-01 B.11.11.23709.GP, which incorrectly rejects +# declarations like `int a3[[(sizeof (unsigned char)) >= 0]];'. +# This bug is HP SR number 8606223364. +{ echo "$as_me:$LINENO: checking size of long double" >&5 +echo $ECHO_N "checking size of long double... $ECHO_C" >&6; } +if test "${ac_cv_sizeof_long_double+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + if test "$cross_compiling" = yes; then + # Depending upon the size, compute the lo and hi bounds. +cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) >= 0)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_lo=0 ac_mid=0 + while :; do + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) <= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=$ac_mid; break +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo=`expr $ac_mid + 1` + if test $ac_lo -le $ac_mid; then + ac_lo= ac_hi= + break + fi + ac_mid=`expr 2 '*' $ac_mid + 1` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + done +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) < 0)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=-1 ac_mid=-1 + while :; do + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) >= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_lo=$ac_mid; break +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_hi=`expr '(' $ac_mid ')' - 1` + if test $ac_mid -le $ac_hi; then + ac_lo= ac_hi= + break + fi + ac_mid=`expr 2 '*' $ac_mid` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + done +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo= ac_hi= +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +# Binary search between lo and hi bounds. +while test "x$ac_lo" != "x$ac_hi"; do + ac_mid=`expr '(' $ac_hi - $ac_lo ')' / 2 + $ac_lo` + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +int +main () +{ +static int test_array [1 - 2 * !(((long int) (sizeof (ac__type_sizeof_))) <= $ac_mid)]; +test_array [0] = 0 + + ; + return 0; +} +_ACEOF +rm -f conftest.$ac_objext +if { (ac_try="$ac_compile" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_compile") 2>conftest.er1 + ac_status=$? + grep -v '^ *+' conftest.er1 >conftest.err + rm -f conftest.er1 + cat conftest.err >&5 + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { + test -z "$ac_c_werror_flag" || + test ! -s conftest.err + } && test -s conftest.$ac_objext; then + ac_hi=$ac_mid +else + echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + + ac_lo=`expr '(' $ac_mid ')' + 1` +fi + +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +done +case $ac_lo in +?*) ac_cv_sizeof_long_double=$ac_lo;; +'') if test "$ac_cv_type_long_double" = yes; then + { { echo "$as_me:$LINENO: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&5 +echo "$as_me: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&2;} + { (exit 77); exit 77; }; } + else + ac_cv_sizeof_long_double=0 + fi ;; +esac +else + cat >conftest.$ac_ext <<_ACEOF +/* confdefs.h. */ +_ACEOF +cat confdefs.h >>conftest.$ac_ext +cat >>conftest.$ac_ext <<_ACEOF +/* end confdefs.h. */ +$ac_includes_default + typedef long double ac__type_sizeof_; +static long int longval () { return (long int) (sizeof (ac__type_sizeof_)); } +static unsigned long int ulongval () { return (long int) (sizeof (ac__type_sizeof_)); } +#include +#include +int +main () +{ + + FILE *f = fopen ("conftest.val", "w"); + if (! f) + return 1; + if (((long int) (sizeof (ac__type_sizeof_))) < 0) + { + long int i = longval (); + if (i != ((long int) (sizeof (ac__type_sizeof_)))) + return 1; + fprintf (f, "%ld\n", i); + } + else + { + unsigned long int i = ulongval (); + if (i != ((long int) (sizeof (ac__type_sizeof_)))) + return 1; + fprintf (f, "%lu\n", i); + } + return ferror (f) || fclose (f) != 0; + + ; + return 0; +} +_ACEOF +rm -f conftest$ac_exeext +if { (ac_try="$ac_link" +case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_link") 2>&5 + ac_status=$? + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); } && { ac_try='./conftest$ac_exeext' + { (case "(($ac_try" in + *\"* | *\`* | *\\*) ac_try_echo=\$ac_try;; + *) ac_try_echo=$ac_try;; +esac +eval "echo \"\$as_me:$LINENO: $ac_try_echo\"") >&5 + (eval "$ac_try") 2>&5 + ac_status=$? + echo "$as_me:$LINENO: \$? = $ac_status" >&5 + (exit $ac_status); }; }; then + ac_cv_sizeof_long_double=`cat conftest.val` +else + echo "$as_me: program exited with status $ac_status" >&5 +echo "$as_me: failed program was:" >&5 +sed 's/^/| /' conftest.$ac_ext >&5 + +( exit $ac_status ) +if test "$ac_cv_type_long_double" = yes; then + { { echo "$as_me:$LINENO: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&5 +echo "$as_me: error: cannot compute sizeof (long double) +See \`config.log' for more details." >&2;} + { (exit 77); exit 77; }; } + else + ac_cv_sizeof_long_double=0 + fi +fi +rm -f core *.core core.conftest.* gmon.out bb.out conftest$ac_exeext conftest.$ac_objext conftest.$ac_ext +fi +rm -f conftest.val +fi +{ echo "$as_me:$LINENO: result: $ac_cv_sizeof_long_double" >&5 +echo "${ECHO_T}$ac_cv_sizeof_long_double" >&6; } + + + +cat >>confdefs.h <<_ACEOF +#define SIZEOF_LONG_DOUBLE $ac_cv_sizeof_long_double +_ACEOF + + +fi + + { echo "$as_me:$LINENO: checking for _Bool support" >&5 echo $ECHO_N "checking for _Bool support... $ECHO_C" >&6; } have_c99_bool=no Modified: python/branches/py3k-importlib/configure.in ============================================================================== --- python/branches/py3k-importlib/configure.in (original) +++ python/branches/py3k-importlib/configure.in Mon Oct 15 21:39:57 2007 @@ -1197,6 +1197,18 @@ AC_CHECK_SIZEOF(long long, 8) fi +AC_MSG_CHECKING(for long double support) +have_long_double=no +AC_TRY_COMPILE([], [long double x; x = (long double)0;], [ + AC_DEFINE(HAVE_LONG_DOUBLE, 1, [Define this if you have the type long double.]) + have_long_double=yes +]) +AC_MSG_RESULT($have_long_double) +if test "$have_long_double" = yes ; then +AC_CHECK_SIZEOF(long double, 16) +fi + + AC_MSG_CHECKING(for _Bool support) have_c99_bool=no AC_TRY_COMPILE([], [_Bool x; x = (_Bool)0;], [ Modified: python/branches/py3k-importlib/pyconfig.h.in ============================================================================== --- python/branches/py3k-importlib/pyconfig.h.in (original) +++ python/branches/py3k-importlib/pyconfig.h.in Mon Oct 15 21:39:57 2007 @@ -330,6 +330,9 @@ /* Define to 1 if you have the header file. */ #undef HAVE_LINUX_NETLINK_H +/* Define this if you have the type long double. */ +#undef HAVE_LONG_DOUBLE + /* Define this if you have the type long long. */ #undef HAVE_LONG_LONG @@ -824,6 +827,9 @@ /* The size of `long', as computed by sizeof. */ #undef SIZEOF_LONG +/* The size of `long double', as computed by sizeof. */ +#undef SIZEOF_LONG_DOUBLE + /* The size of `long long', as computed by sizeof. */ #undef SIZEOF_LONG_LONG From python-3000-checkins at python.org Mon Oct 15 22:17:02 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 15 Oct 2007 22:17:02 +0200 (CEST) Subject: [Python-3000-checkins] r58485 - python/branches/py3k-importlib/NEWS Message-ID: <20071015201702.62DB21E402F@bag.python.org> Author: brett.cannon Date: Mon Oct 15 22:17:02 2007 New Revision: 58485 Modified: python/branches/py3k-importlib/NEWS Log: Flesh out last NEWS entry. Modified: python/branches/py3k-importlib/NEWS ============================================================================== --- python/branches/py3k-importlib/NEWS (original) +++ python/branches/py3k-importlib/NEWS Mon Oct 15 22:17:02 2007 @@ -1,4 +1,6 @@ -* Store old __import__ into __old_import__ and use importlib. +* Store old __import__ into __old_import__ and use importlib. Required + introducing a somewhat fake file object that uses posix and file descriptors + as open() is defined in site.py which obviously has not been imported yet. * Add _r_long() and _w_long() to marshal. From python-3000-checkins at python.org Mon Oct 15 22:52:42 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 15 Oct 2007 22:52:42 +0200 (CEST) Subject: [Python-3000-checkins] r58486 - python/branches/py3k/Lib/io.py Message-ID: <20071015205242.2A7E81E402D@bag.python.org> Author: brett.cannon Date: Mon Oct 15 22:52:41 2007 New Revision: 58486 Modified: python/branches/py3k/Lib/io.py Log: Make the docstring for io.open() a raw string so that the explanation for the 'newline' argument is not a jumbled mess of newlines. Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Mon Oct 15 22:52:41 2007 @@ -50,7 +50,7 @@ def open(file, mode="r", buffering=None, encoding=None, newline=None): - """Replacement for the built-in open function. + r"""Replacement for the built-in open function. Args: file: string giving the name of the file to be opened; From python-3000-checkins at python.org Mon Oct 15 23:01:24 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 15 Oct 2007 23:01:24 +0200 (CEST) Subject: [Python-3000-checkins] r58488 - python/branches/py3k-importlib/NEWS Message-ID: <20071015210124.33C721E402D@bag.python.org> Author: brett.cannon Date: Mon Oct 15 23:01:23 2007 New Revision: 58488 Modified: python/branches/py3k-importlib/NEWS Log: Add even more detail to the last NEWS entry. Modified: python/branches/py3k-importlib/NEWS ============================================================================== --- python/branches/py3k-importlib/NEWS (original) +++ python/branches/py3k-importlib/NEWS Mon Oct 15 23:01:23 2007 @@ -1,6 +1,8 @@ * Store old __import__ into __old_import__ and use importlib. Required introducing a somewhat fake file object that uses posix and file descriptors as open() is defined in site.py which obviously has not been imported yet. + The faked file objects read all text in as ASCII since any codecs usage + requires the codecs package to be imported. * Add _r_long() and _w_long() to marshal. From python-3000-checkins at python.org Tue Oct 16 02:26:34 2007 From: python-3000-checkins at python.org (alexandre.vassalotti) Date: Tue, 16 Oct 2007 02:26:34 +0200 (CEST) Subject: [Python-3000-checkins] r58491 - python/branches/py3k/Objects/unicodeobject.c Message-ID: <20071016002634.2C3011E402D@bag.python.org> Author: alexandre.vassalotti Date: Tue Oct 16 02:26:33 2007 New Revision: 58491 Modified: python/branches/py3k/Objects/unicodeobject.c Log: Fix a small typo in the comment of unicode_default_encoding[]. Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Tue Oct 16 02:26:33 2007 @@ -119,7 +119,7 @@ parameter; it is fixed to "utf-8". Always use the PyUnicode_GetDefaultEncoding() API to access this global. - Don't forget to alter Py_FileSystemDefaultEncoding() if you change the + Don't forget to alter Py_FileSystemDefaultEncoding if you change the hard coded default! */ static const char unicode_default_encoding[] = "utf-8"; From python-3000-checkins at python.org Tue Oct 16 08:31:31 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Tue, 16 Oct 2007 08:31:31 +0200 (CEST) Subject: [Python-3000-checkins] r58493 - in python/branches/py3k: Include/bytes_methods.h Lib/test/buffer_tests.py Lib/test/test_bytes.py Makefile.pre.in Objects/bytes_methods.c Objects/bytesobject.c Objects/stringlib/README.txt Objects/stringlib/ctype.h Objects/stringlib/find.h Objects/stringlib/transmogrify.h Objects/stringlib/unicodedefs.h Objects/stringobject.c Message-ID: <20071016063131.7F4C21E4046@bag.python.org> Author: gregory.p.smith Date: Tue Oct 16 08:31:30 2007 New Revision: 58493 Added: python/branches/py3k/Include/bytes_methods.h python/branches/py3k/Lib/test/buffer_tests.py python/branches/py3k/Objects/bytes_methods.c python/branches/py3k/Objects/stringlib/ctype.h python/branches/py3k/Objects/stringlib/transmogrify.h Modified: python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Makefile.pre.in python/branches/py3k/Objects/bytesobject.c python/branches/py3k/Objects/stringlib/README.txt python/branches/py3k/Objects/stringlib/find.h python/branches/py3k/Objects/stringlib/unicodedefs.h python/branches/py3k/Objects/stringobject.c Log: For PEP3137: Adds missing methods to the mutable PyBytes object (soon to be called a buffer). Shares code with stringobject when possible. Adds unit tests with common code that should be usable to test the PEPs mutable buffer() and immutable bytes() types. http://bugs.python.org/issue1261 Added: python/branches/py3k/Include/bytes_methods.h ============================================================================== --- (empty file) +++ python/branches/py3k/Include/bytes_methods.h Tue Oct 16 08:31:30 2007 @@ -0,0 +1,84 @@ +#ifndef Py_BYTES_CTYPE_H +#define Py_BYTES_CTYPE_H + +/* + * The internal implementation behind PyString (bytes) and PyBytes (buffer) + * methods of the given names, they operate on ASCII byte strings. + */ +extern PyObject* _Py_bytes_isspace(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_isalpha(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_isalnum(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_isdigit(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_islower(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_isupper(const char *cptr, Py_ssize_t len); +extern PyObject* _Py_bytes_istitle(const char *cptr, Py_ssize_t len); + +/* These store their len sized answer in the given preallocated *result arg. */ +extern void _Py_bytes_lower(char *result, const char *cptr, Py_ssize_t len); +extern void _Py_bytes_upper(char *result, const char *cptr, Py_ssize_t len); +extern void _Py_bytes_title(char *result, char *s, Py_ssize_t len); +extern void _Py_bytes_capitalize(char *result, char *s, Py_ssize_t len); +extern void _Py_bytes_swapcase(char *result, char *s, Py_ssize_t len); + +/* Shared __doc__ strings. */ +extern const char _Py_isspace__doc__[]; +extern const char _Py_isalpha__doc__[]; +extern const char _Py_isalnum__doc__[]; +extern const char _Py_isdigit__doc__[]; +extern const char _Py_islower__doc__[]; +extern const char _Py_isupper__doc__[]; +extern const char _Py_istitle__doc__[]; +extern const char _Py_lower__doc__[]; +extern const char _Py_upper__doc__[]; +extern const char _Py_title__doc__[]; +extern const char _Py_capitalize__doc__[]; +extern const char _Py_swapcase__doc__[]; + +#define FLAG_LOWER 0x01 +#define FLAG_UPPER 0x02 +#define FLAG_ALPHA (FLAG_LOWER|FLAG_UPPER) +#define FLAG_DIGIT 0x04 +#define FLAG_ALNUM (FLAG_ALPHA|FLAG_DIGIT) +#define FLAG_SPACE 0x08 +#define FLAG_XDIGIT 0x10 + +extern const unsigned int _Py_ctype_table[256]; + +#define ISLOWER(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_LOWER) +#define ISUPPER(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_UPPER) +#define ISALPHA(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_ALPHA) +#define ISDIGIT(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_DIGIT) +#define ISXDIGIT(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_XDIGIT) +#define ISALNUM(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_ALNUM) +#define ISSPACE(c) (_Py_ctype_table[Py_CHARMASK(c)] & FLAG_SPACE) + +#undef islower +#define islower(c) undefined_islower(c) +#undef isupper +#define isupper(c) undefined_isupper(c) +#undef isalpha +#define isalpha(c) undefined_isalpha(c) +#undef isdigit +#define isdigit(c) undefined_isdigit(c) +#undef isxdigit +#define isxdigit(c) undefined_isxdigit(c) +#undef isalnum +#define isalnum(c) undefined_isalnum(c) +#undef isspace +#define isspace(c) undefined_isspace(c) + +extern const unsigned char _Py_ctype_tolower[256]; +extern const unsigned char _Py_ctype_toupper[256]; + +#define TOLOWER(c) (_Py_ctype_tolower[Py_CHARMASK(c)]) +#define TOUPPER(c) (_Py_ctype_toupper[Py_CHARMASK(c)]) + +#undef tolower +#define tolower(c) undefined_tolower(c) +#undef toupper +#define toupper(c) undefined_toupper(c) + +/* this is needed because some docs are shared from the .o, not static */ +#define PyDoc_STRVAR_shared(name,str) const char name[] = PyDoc_STR(str) + +#endif /* !Py_BYTES_CTYPE_H */ Added: python/branches/py3k/Lib/test/buffer_tests.py ============================================================================== --- (empty file) +++ python/branches/py3k/Lib/test/buffer_tests.py Tue Oct 16 08:31:30 2007 @@ -0,0 +1,206 @@ +# Tests that work for both str8 (bytes) and bytes (buffer) objects. +# See PEP 3137. + +import struct +import sys + +class MixinBytesBufferCommonTests(object): + """Tests that work for both str8 (bytes) and bytes (buffer) objects. + See PEP 3137. + """ + + def marshal(self, x): + """Convert x into the appropriate type for these tests.""" + raise RuntimeError('test class must provide a marshal method') + + def test_islower(self): + self.assertFalse(self.marshal(b'').islower()) + self.assert_(self.marshal(b'a').islower()) + self.assertFalse(self.marshal(b'A').islower()) + self.assertFalse(self.marshal(b'\n').islower()) + self.assert_(self.marshal(b'abc').islower()) + self.assertFalse(self.marshal(b'aBc').islower()) + self.assert_(self.marshal(b'abc\n').islower()) + self.assertRaises(TypeError, self.marshal(b'abc').islower, 42) + + def test_isupper(self): + self.assertFalse(self.marshal(b'').isupper()) + self.assertFalse(self.marshal(b'a').isupper()) + self.assert_(self.marshal(b'A').isupper()) + self.assertFalse(self.marshal(b'\n').isupper()) + self.assert_(self.marshal(b'ABC').isupper()) + self.assertFalse(self.marshal(b'AbC').isupper()) + self.assert_(self.marshal(b'ABC\n').isupper()) + self.assertRaises(TypeError, self.marshal(b'abc').isupper, 42) + + def test_istitle(self): + self.assertFalse(self.marshal(b'').istitle()) + self.assertFalse(self.marshal(b'a').istitle()) + self.assert_(self.marshal(b'A').istitle()) + self.assertFalse(self.marshal(b'\n').istitle()) + self.assert_(self.marshal(b'A Titlecased Line').istitle()) + self.assert_(self.marshal(b'A\nTitlecased Line').istitle()) + self.assert_(self.marshal(b'A Titlecased, Line').istitle()) + self.assertFalse(self.marshal(b'Not a capitalized String').istitle()) + self.assertFalse(self.marshal(b'Not\ta Titlecase String').istitle()) + self.assertFalse(self.marshal(b'Not--a Titlecase String').istitle()) + self.assertFalse(self.marshal(b'NOT').istitle()) + self.assertRaises(TypeError, self.marshal(b'abc').istitle, 42) + + def test_isspace(self): + self.assertFalse(self.marshal(b'').isspace()) + self.assertFalse(self.marshal(b'a').isspace()) + self.assert_(self.marshal(b' ').isspace()) + self.assert_(self.marshal(b'\t').isspace()) + self.assert_(self.marshal(b'\r').isspace()) + self.assert_(self.marshal(b'\n').isspace()) + self.assert_(self.marshal(b' \t\r\n').isspace()) + self.assertFalse(self.marshal(b' \t\r\na').isspace()) + self.assertRaises(TypeError, self.marshal(b'abc').isspace, 42) + + def test_isalpha(self): + self.assertFalse(self.marshal(b'').isalpha()) + self.assert_(self.marshal(b'a').isalpha()) + self.assert_(self.marshal(b'A').isalpha()) + self.assertFalse(self.marshal(b'\n').isalpha()) + self.assert_(self.marshal(b'abc').isalpha()) + self.assertFalse(self.marshal(b'aBc123').isalpha()) + self.assertFalse(self.marshal(b'abc\n').isalpha()) + self.assertRaises(TypeError, self.marshal(b'abc').isalpha, 42) + + def test_isalnum(self): + self.assertFalse(self.marshal(b'').isalnum()) + self.assert_(self.marshal(b'a').isalnum()) + self.assert_(self.marshal(b'A').isalnum()) + self.assertFalse(self.marshal(b'\n').isalnum()) + self.assert_(self.marshal(b'123abc456').isalnum()) + self.assert_(self.marshal(b'a1b3c').isalnum()) + self.assertFalse(self.marshal(b'aBc000 ').isalnum()) + self.assertFalse(self.marshal(b'abc\n').isalnum()) + self.assertRaises(TypeError, self.marshal(b'abc').isalnum, 42) + + def test_isdigit(self): + self.assertFalse(self.marshal(b'').isdigit()) + self.assertFalse(self.marshal(b'a').isdigit()) + self.assert_(self.marshal(b'0').isdigit()) + self.assert_(self.marshal(b'0123456789').isdigit()) + self.assertFalse(self.marshal(b'0123456789a').isdigit()) + + self.assertRaises(TypeError, self.marshal(b'abc').isdigit, 42) + + def test_lower(self): + self.assertEqual(b'hello', self.marshal(b'HeLLo').lower()) + self.assertEqual(b'hello', self.marshal(b'hello').lower()) + self.assertRaises(TypeError, self.marshal(b'hello').lower, 42) + + def test_upper(self): + self.assertEqual(b'HELLO', self.marshal(b'HeLLo').upper()) + self.assertEqual(b'HELLO', self.marshal(b'HELLO').upper()) + self.assertRaises(TypeError, self.marshal(b'hello').upper, 42) + + def test_capitalize(self): + self.assertEqual(b' hello ', self.marshal(b' hello ').capitalize()) + self.assertEqual(b'Hello ', self.marshal(b'Hello ').capitalize()) + self.assertEqual(b'Hello ', self.marshal(b'hello ').capitalize()) + self.assertEqual(b'Aaaa', self.marshal(b'aaaa').capitalize()) + self.assertEqual(b'Aaaa', self.marshal(b'AaAa').capitalize()) + + self.assertRaises(TypeError, self.marshal(b'hello').capitalize, 42) + + def test_ljust(self): + self.assertEqual(b'abc ', self.marshal(b'abc').ljust(10)) + self.assertEqual(b'abc ', self.marshal(b'abc').ljust(6)) + self.assertEqual(b'abc', self.marshal(b'abc').ljust(3)) + self.assertEqual(b'abc', self.marshal(b'abc').ljust(2)) + self.assertEqual(b'abc*******', self.marshal(b'abc').ljust(10, '*')) + self.assertRaises(TypeError, self.marshal(b'abc').ljust) + + def test_rjust(self): + self.assertEqual(b' abc', self.marshal(b'abc').rjust(10)) + self.assertEqual(b' abc', self.marshal(b'abc').rjust(6)) + self.assertEqual(b'abc', self.marshal(b'abc').rjust(3)) + self.assertEqual(b'abc', self.marshal(b'abc').rjust(2)) + self.assertEqual(b'*******abc', self.marshal(b'abc').rjust(10, '*')) + self.assertRaises(TypeError, self.marshal(b'abc').rjust) + + def test_center(self): + self.assertEqual(b' abc ', self.marshal(b'abc').center(10)) + self.assertEqual(b' abc ', self.marshal(b'abc').center(6)) + self.assertEqual(b'abc', self.marshal(b'abc').center(3)) + self.assertEqual(b'abc', self.marshal(b'abc').center(2)) + self.assertEqual(b'***abc****', self.marshal(b'abc').center(10, '*')) + self.assertRaises(TypeError, self.marshal(b'abc').center) + + def test_swapcase(self): + self.assertEqual(b'hEllO CoMPuTErS', + self.marshal(b'HeLLo cOmpUteRs').swapcase()) + + self.assertRaises(TypeError, self.marshal(b'hello').swapcase, 42) + + def test_zfill(self): + self.assertEqual(b'123', self.marshal(b'123').zfill(2)) + self.assertEqual(b'123', self.marshal(b'123').zfill(3)) + self.assertEqual(b'0123', self.marshal(b'123').zfill(4)) + self.assertEqual(b'+123', self.marshal(b'+123').zfill(3)) + self.assertEqual(b'+123', self.marshal(b'+123').zfill(4)) + self.assertEqual(b'+0123', self.marshal(b'+123').zfill(5)) + self.assertEqual(b'-123', self.marshal(b'-123').zfill(3)) + self.assertEqual(b'-123', self.marshal(b'-123').zfill(4)) + self.assertEqual(b'-0123', self.marshal(b'-123').zfill(5)) + self.assertEqual(b'000', self.marshal(b'').zfill(3)) + self.assertEqual(b'34', self.marshal(b'34').zfill(1)) + self.assertEqual(b'0034', self.marshal(b'34').zfill(4)) + + self.assertRaises(TypeError, self.marshal(b'123').zfill) + + def test_expandtabs(self): + self.assertEqual(b'abc\rab def\ng hi', + self.marshal(b'abc\rab\tdef\ng\thi').expandtabs()) + self.assertEqual(b'abc\rab def\ng hi', + self.marshal(b'abc\rab\tdef\ng\thi').expandtabs(8)) + self.assertEqual(b'abc\rab def\ng hi', + self.marshal(b'abc\rab\tdef\ng\thi').expandtabs(4)) + self.assertEqual(b'abc\r\nab def\ng hi', + self.marshal(b'abc\r\nab\tdef\ng\thi').expandtabs(4)) + self.assertEqual(b'abc\rab def\ng hi', + self.marshal(b'abc\rab\tdef\ng\thi').expandtabs()) + self.assertEqual(b'abc\rab def\ng hi', + self.marshal(b'abc\rab\tdef\ng\thi').expandtabs(8)) + self.assertEqual(b'abc\r\nab\r\ndef\ng\r\nhi', + self.marshal(b'abc\r\nab\r\ndef\ng\r\nhi').expandtabs(4)) + self.assertEqual(b' a\n b', self.marshal(b' \ta\n\tb').expandtabs(1)) + + self.assertRaises(TypeError, self.marshal(b'hello').expandtabs, 42, 42) + # This test is only valid when sizeof(int) == sizeof(void*) == 4. + if sys.maxint < (1 << 32) and struct.calcsize('P') == 4: + self.assertRaises(OverflowError, + self.marshal(b'\ta\n\tb').expandtabs, sys.maxint) + + def test_title(self): + self.assertEqual(b' Hello ', self.marshal(b' hello ').title()) + self.assertEqual(b'Hello ', self.marshal(b'hello ').title()) + self.assertEqual(b'Hello ', self.marshal(b'Hello ').title()) + self.assertEqual(b'Format This As Title String', + self.marshal(b'fOrMaT thIs aS titLe String').title()) + self.assertEqual(b'Format,This-As*Title;String', + self.marshal(b'fOrMaT,thIs-aS*titLe;String').title()) + self.assertEqual(b'Getint', self.marshal(b'getInt').title()) + self.assertRaises(TypeError, self.marshal(b'hello').title, 42) + + def test_splitlines(self): + self.assertEqual([b'abc', b'def', b'', b'ghi'], + self.marshal(b'abc\ndef\n\rghi').splitlines()) + self.assertEqual([b'abc', b'def', b'', b'ghi'], + self.marshal(b'abc\ndef\n\r\nghi').splitlines()) + self.assertEqual([b'abc', b'def', b'ghi'], + self.marshal(b'abc\ndef\r\nghi').splitlines()) + self.assertEqual([b'abc', b'def', b'ghi'], + self.marshal(b'abc\ndef\r\nghi\n').splitlines()) + self.assertEqual([b'abc', b'def', b'ghi', b''], + self.marshal(b'abc\ndef\r\nghi\n\r').splitlines()) + self.assertEqual([b'', b'abc', b'def', b'ghi', b''], + self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines()) + self.assertEqual([b'\n', b'abc\n', b'def\r\n', b'ghi\n', b'\r'], + self.marshal(b'\nabc\ndef\r\nghi\n\r').splitlines(1)) + + self.assertRaises(TypeError, self.marshal(b'abc').splitlines, 42, 42) Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Tue Oct 16 08:31:30 2007 @@ -8,6 +8,7 @@ import unittest import test.test_support import test.string_tests +import test.buffer_tests class BytesTest(unittest.TestCase): @@ -454,17 +455,18 @@ def test_fromhex(self): self.assertRaises(TypeError, bytes.fromhex) self.assertRaises(TypeError, bytes.fromhex, 1) - self.assertEquals(bytes.fromhex(''), bytes()) + self.assertEquals(bytes.fromhex(b''), bytes()) b = bytes([0x1a, 0x2b, 0x30]) - self.assertEquals(bytes.fromhex('1a2B30'), b) - self.assertEquals(bytes.fromhex(' 1A 2B 30 '), b) + self.assertEquals(bytes.fromhex(b'1a2B30'), b) + self.assertEquals(bytes.fromhex(b' 1A 2B 30 '), b) self.assertEquals(bytes.fromhex(memoryview(b'')), bytes()) self.assertEquals(bytes.fromhex(memoryview(b'0000')), bytes([0, 0])) - self.assertRaises(ValueError, bytes.fromhex, 'a') - self.assertRaises(ValueError, bytes.fromhex, 'rt') - self.assertRaises(ValueError, bytes.fromhex, '1a b cd') - self.assertRaises(ValueError, bytes.fromhex, '\x00') - self.assertRaises(ValueError, bytes.fromhex, '12 \x00 34') + self.assertRaises(TypeError, bytes.fromhex, '1B') + self.assertRaises(ValueError, bytes.fromhex, b'a') + self.assertRaises(ValueError, bytes.fromhex, b'rt') + self.assertRaises(ValueError, bytes.fromhex, b'1a b cd') + self.assertRaises(ValueError, bytes.fromhex, b'\x00') + self.assertRaises(ValueError, bytes.fromhex, b'12 \x00 34') def test_join(self): self.assertEqual(b"".join([]), bytes()) @@ -504,11 +506,12 @@ self.assertEqual(b, b'heo') self.assertRaises(ValueError, lambda: b.remove(ord('l'))) self.assertRaises(ValueError, lambda: b.remove(400)) - self.assertRaises(ValueError, lambda: b.remove('e')) + self.assertRaises(TypeError, lambda: b.remove('e')) # remove first and last b.remove(ord('o')) b.remove(ord('h')) self.assertEqual(b, b'e') + self.assertRaises(TypeError, lambda: b.remove(b'e')) def test_pop(self): b = b'world' @@ -542,6 +545,7 @@ b = bytes() b.append(ord('A')) self.assertEqual(len(b), 1) + self.assertRaises(TypeError, lambda: b.append(b'o')) def test_insert(self): b = b'msssspp' @@ -550,6 +554,7 @@ b.insert(-2, ord('i')) b.insert(1000, ord('i')) self.assertEqual(b, b'mississippi') + self.assertRaises(TypeError, lambda: b.insert(0, b'1')) def test_startswith(self): b = b'hello' @@ -734,6 +739,29 @@ # Unfortunately they are all bundled with tests that # are not appropriate for bytes + # I've started porting some of those into buffer_tests.py, we should port + # the rest that make sense (the code can be cleaned up to use modern + # unittest methods at the same time). + +class BufferPEP3137Test(unittest.TestCase, + test.buffer_tests.MixinBytesBufferCommonTests): + def marshal(self, x): + return bytes(x) + # TODO this should become: + #return buffer(x) + # once the bytes -> buffer and str8 -> bytes rename happens + + def test_returns_new_copy(self): + val = self.marshal(b'1234') + # On immutable types these MAY return a reference to themselves + # but on mutable types like buffer they MUST return a new copy. + for methname in ('zfill', 'rjust', 'ljust', 'center'): + method = getattr(val, methname) + newval = method(3) + self.assertEqual(val, newval) + self.assertTrue(val is not newval, + methname+' returned self on a mutable object') + class BytesAsStringTest(test.string_tests.BaseTest): type2test = bytes @@ -759,7 +787,7 @@ def test_main(): test.test_support.run_unittest(BytesTest) test.test_support.run_unittest(BytesAsStringTest) - + test.test_support.run_unittest(BufferPEP3137Test) if __name__ == "__main__": ##test_main() Modified: python/branches/py3k/Makefile.pre.in ============================================================================== --- python/branches/py3k/Makefile.pre.in (original) +++ python/branches/py3k/Makefile.pre.in Tue Oct 16 08:31:30 2007 @@ -285,6 +285,7 @@ OBJECT_OBJS= \ Objects/abstract.o \ Objects/boolobject.o \ + Objects/bytes_methods.o \ Objects/bytesobject.o \ Objects/cellobject.o \ Objects/classobject.o \ @@ -507,6 +508,18 @@ Objects/unicodectype.o: $(srcdir)/Objects/unicodectype.c \ $(srcdir)/Objects/unicodetype_db.h +BYTESTR_DEPS = Include/bytes_methods.h \ + $(srcdir)/Objects/stringlib/fastsearch.h \ + $(srcdir)/Objects/stringlib/count.h \ + $(srcdir)/Objects/stringlib/find.h \ + $(srcdir)/Objects/stringlib/partition.h \ + $(srcdir)/Objects/stringlib/ctype.h \ + $(srcdir)/Objects/stringlib/transmogrify.h + +Objects/stringobject.o: $(srcdir)/Objects/stringobject.c $(BYTESTR_DEPS) + +Objects/bytesobject.o: $(srcdir)/Objects/bytesobject.c $(BYTESTR_DEPS) + Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c \ $(srcdir)/Objects/stringlib/string_format.h \ $(srcdir)/Objects/stringlib/unicodedefs.h \ Added: python/branches/py3k/Objects/bytes_methods.c ============================================================================== --- (empty file) +++ python/branches/py3k/Objects/bytes_methods.c Tue Oct 16 08:31:30 2007 @@ -0,0 +1,610 @@ +#include "Python.h" +#include "bytes_methods.h" + +/* Our own locale-independent ctype.h-like macros */ + +const unsigned int _Py_ctype_table[256] = { + 0, /* 0x0 '\x00' */ + 0, /* 0x1 '\x01' */ + 0, /* 0x2 '\x02' */ + 0, /* 0x3 '\x03' */ + 0, /* 0x4 '\x04' */ + 0, /* 0x5 '\x05' */ + 0, /* 0x6 '\x06' */ + 0, /* 0x7 '\x07' */ + 0, /* 0x8 '\x08' */ + FLAG_SPACE, /* 0x9 '\t' */ + FLAG_SPACE, /* 0xa '\n' */ + FLAG_SPACE, /* 0xb '\v' */ + FLAG_SPACE, /* 0xc '\f' */ + FLAG_SPACE, /* 0xd '\r' */ + 0, /* 0xe '\x0e' */ + 0, /* 0xf '\x0f' */ + 0, /* 0x10 '\x10' */ + 0, /* 0x11 '\x11' */ + 0, /* 0x12 '\x12' */ + 0, /* 0x13 '\x13' */ + 0, /* 0x14 '\x14' */ + 0, /* 0x15 '\x15' */ + 0, /* 0x16 '\x16' */ + 0, /* 0x17 '\x17' */ + 0, /* 0x18 '\x18' */ + 0, /* 0x19 '\x19' */ + 0, /* 0x1a '\x1a' */ + 0, /* 0x1b '\x1b' */ + 0, /* 0x1c '\x1c' */ + 0, /* 0x1d '\x1d' */ + 0, /* 0x1e '\x1e' */ + 0, /* 0x1f '\x1f' */ + FLAG_SPACE, /* 0x20 ' ' */ + 0, /* 0x21 '!' */ + 0, /* 0x22 '"' */ + 0, /* 0x23 '#' */ + 0, /* 0x24 '$' */ + 0, /* 0x25 '%' */ + 0, /* 0x26 '&' */ + 0, /* 0x27 "'" */ + 0, /* 0x28 '(' */ + 0, /* 0x29 ')' */ + 0, /* 0x2a '*' */ + 0, /* 0x2b '+' */ + 0, /* 0x2c ',' */ + 0, /* 0x2d '-' */ + 0, /* 0x2e '.' */ + 0, /* 0x2f '/' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x30 '0' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x31 '1' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x32 '2' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x33 '3' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x34 '4' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x35 '5' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x36 '6' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x37 '7' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x38 '8' */ + FLAG_DIGIT|FLAG_XDIGIT, /* 0x39 '9' */ + 0, /* 0x3a ':' */ + 0, /* 0x3b ';' */ + 0, /* 0x3c '<' */ + 0, /* 0x3d '=' */ + 0, /* 0x3e '>' */ + 0, /* 0x3f '?' */ + 0, /* 0x40 '@' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x41 'A' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x42 'B' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x43 'C' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x44 'D' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x45 'E' */ + FLAG_UPPER|FLAG_XDIGIT, /* 0x46 'F' */ + FLAG_UPPER, /* 0x47 'G' */ + FLAG_UPPER, /* 0x48 'H' */ + FLAG_UPPER, /* 0x49 'I' */ + FLAG_UPPER, /* 0x4a 'J' */ + FLAG_UPPER, /* 0x4b 'K' */ + FLAG_UPPER, /* 0x4c 'L' */ + FLAG_UPPER, /* 0x4d 'M' */ + FLAG_UPPER, /* 0x4e 'N' */ + FLAG_UPPER, /* 0x4f 'O' */ + FLAG_UPPER, /* 0x50 'P' */ + FLAG_UPPER, /* 0x51 'Q' */ + FLAG_UPPER, /* 0x52 'R' */ + FLAG_UPPER, /* 0x53 'S' */ + FLAG_UPPER, /* 0x54 'T' */ + FLAG_UPPER, /* 0x55 'U' */ + FLAG_UPPER, /* 0x56 'V' */ + FLAG_UPPER, /* 0x57 'W' */ + FLAG_UPPER, /* 0x58 'X' */ + FLAG_UPPER, /* 0x59 'Y' */ + FLAG_UPPER, /* 0x5a 'Z' */ + 0, /* 0x5b '[' */ + 0, /* 0x5c '\\' */ + 0, /* 0x5d ']' */ + 0, /* 0x5e '^' */ + 0, /* 0x5f '_' */ + 0, /* 0x60 '`' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x61 'a' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x62 'b' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x63 'c' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x64 'd' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x65 'e' */ + FLAG_LOWER|FLAG_XDIGIT, /* 0x66 'f' */ + FLAG_LOWER, /* 0x67 'g' */ + FLAG_LOWER, /* 0x68 'h' */ + FLAG_LOWER, /* 0x69 'i' */ + FLAG_LOWER, /* 0x6a 'j' */ + FLAG_LOWER, /* 0x6b 'k' */ + FLAG_LOWER, /* 0x6c 'l' */ + FLAG_LOWER, /* 0x6d 'm' */ + FLAG_LOWER, /* 0x6e 'n' */ + FLAG_LOWER, /* 0x6f 'o' */ + FLAG_LOWER, /* 0x70 'p' */ + FLAG_LOWER, /* 0x71 'q' */ + FLAG_LOWER, /* 0x72 'r' */ + FLAG_LOWER, /* 0x73 's' */ + FLAG_LOWER, /* 0x74 't' */ + FLAG_LOWER, /* 0x75 'u' */ + FLAG_LOWER, /* 0x76 'v' */ + FLAG_LOWER, /* 0x77 'w' */ + FLAG_LOWER, /* 0x78 'x' */ + FLAG_LOWER, /* 0x79 'y' */ + FLAG_LOWER, /* 0x7a 'z' */ + 0, /* 0x7b '{' */ + 0, /* 0x7c '|' */ + 0, /* 0x7d '}' */ + 0, /* 0x7e '~' */ + 0, /* 0x7f '\x7f' */ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; + + +const unsigned char _Py_ctype_tolower[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, + 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, + 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, + 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + +const unsigned char _Py_ctype_toupper[256] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, + 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, + 0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, + 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, + 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, + 0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, + 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, + 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, + 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, + 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, + 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, + 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, + 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, + 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, + 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, + 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, + 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, + 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, + 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, + 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, + 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, + 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, +}; + + +PyDoc_STRVAR_shared(_Py_isspace__doc__, +"B.isspace() -> bool\n\ +\n\ +Return True if all characters in B are whitespace\n\ +and there is at least one character in B, False otherwise."); + +PyObject* +_Py_bytes_isspace(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + + /* Shortcut for single character strings */ + if (len == 1 && ISSPACE(*p)) + Py_RETURN_TRUE; + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + for (; p < e; p++) { + if (!ISSPACE(*p)) + Py_RETURN_FALSE; + } + Py_RETURN_TRUE; +} + + +PyDoc_STRVAR_shared(_Py_isalpha__doc__, +"B.isalpha() -> bool\n\ +\n\ +Return True if all characters in B are alphabetic\n\ +and there is at least one character in B, False otherwise."); + +PyObject* +_Py_bytes_isalpha(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + + /* Shortcut for single character strings */ + if (len == 1 && ISALPHA(*p)) + Py_RETURN_TRUE; + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + for (; p < e; p++) { + if (!ISALPHA(*p)) + Py_RETURN_FALSE; + } + Py_RETURN_TRUE; +} + + +PyDoc_STRVAR_shared(_Py_isalnum__doc__, +"B.isalnum() -> bool\n\ +\n\ +Return True if all characters in B are alphanumeric\n\ +and there is at least one character in B, False otherwise."); + +PyObject* +_Py_bytes_isalnum(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + + /* Shortcut for single character strings */ + if (len == 1 && ISALNUM(*p)) + Py_RETURN_TRUE; + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + for (; p < e; p++) { + if (!ISALNUM(*p)) + Py_RETURN_FALSE; + } + Py_RETURN_TRUE; +} + + +PyDoc_STRVAR_shared(_Py_isdigit__doc__, +"B.isdigit() -> bool\n\ +\n\ +Return True if all characters in B are digits\n\ +and there is at least one character in B, False otherwise."); + +PyObject* +_Py_bytes_isdigit(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + + /* Shortcut for single character strings */ + if (len == 1 && ISDIGIT(*p)) + Py_RETURN_TRUE; + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + for (; p < e; p++) { + if (!ISDIGIT(*p)) + Py_RETURN_FALSE; + } + Py_RETURN_TRUE; +} + + +PyDoc_STRVAR_shared(_Py_islower__doc__, +"B.islower() -> bool\n\ +\n\ +Return True if all cased characters in B are lowercase and there is\n\ +at least one cased character in B, False otherwise."); + +PyObject* +_Py_bytes_islower(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + int cased; + + /* Shortcut for single character strings */ + if (len == 1) + return PyBool_FromLong(ISLOWER(*p)); + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + cased = 0; + for (; p < e; p++) { + if (ISUPPER(*p)) + Py_RETURN_FALSE; + else if (!cased && ISLOWER(*p)) + cased = 1; + } + return PyBool_FromLong(cased); +} + + +PyDoc_STRVAR_shared(_Py_isupper__doc__, +"B.isupper() -> bool\n\ +\n\ +Return True if all cased characters in B are uppercase and there is\n\ +at least one cased character in B, False otherwise."); + +PyObject* +_Py_bytes_isupper(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + int cased; + + /* Shortcut for single character strings */ + if (len == 1) + return PyBool_FromLong(ISUPPER(*p)); + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + cased = 0; + for (; p < e; p++) { + if (ISLOWER(*p)) + Py_RETURN_FALSE; + else if (!cased && ISUPPER(*p)) + cased = 1; + } + return PyBool_FromLong(cased); +} + + +PyDoc_STRVAR_shared(_Py_istitle__doc__, +"B.istitle() -> bool\n\ +\n\ +Return True if B is a titlecased string and there is at least one\n\ +character in B, i.e. uppercase characters may only follow uncased\n\ +characters and lowercase characters only cased ones. Return False\n\ +otherwise."); + +PyObject* +_Py_bytes_istitle(const char *cptr, Py_ssize_t len) +{ + register const unsigned char *p + = (unsigned char *) cptr; + register const unsigned char *e; + int cased, previous_is_cased; + + /* Shortcut for single character strings */ + if (len == 1) + return PyBool_FromLong(ISUPPER(*p)); + + /* Special case for empty strings */ + if (len == 0) + Py_RETURN_FALSE; + + e = p + len; + cased = 0; + previous_is_cased = 0; + for (; p < e; p++) { + register const unsigned char ch = *p; + + if (ISUPPER(ch)) { + if (previous_is_cased) + Py_RETURN_FALSE; + previous_is_cased = 1; + cased = 1; + } + else if (ISLOWER(ch)) { + if (!previous_is_cased) + Py_RETURN_FALSE; + previous_is_cased = 1; + cased = 1; + } + else + previous_is_cased = 0; + } + return PyBool_FromLong(cased); +} + + +PyDoc_STRVAR_shared(_Py_lower__doc__, +"B.lower() -> copy of B\n\ +\n\ +Return a copy of B with all ASCII characters converted to lowercase."); + +void +_Py_bytes_lower(char *result, const char *cptr, Py_ssize_t len) +{ + Py_ssize_t i; + + /* + newobj = PyString_FromStringAndSize(NULL, len); + if (!newobj) + return NULL; + + s = PyString_AS_STRING(newobj); + */ + + Py_MEMCPY(result, cptr, len); + + for (i = 0; i < len; i++) { + int c = Py_CHARMASK(result[i]); + if (ISUPPER(c)) + result[i] = TOLOWER(c); + } +} + + +PyDoc_STRVAR_shared(_Py_upper__doc__, +"B.upper() -> copy of B\n\ +\n\ +Return a copy of B with all ASCII characters converted to uppercase."); + +void +_Py_bytes_upper(char *result, const char *cptr, Py_ssize_t len) +{ + Py_ssize_t i; + + /* + newobj = PyString_FromStringAndSize(NULL, len); + if (!newobj) + return NULL; + + s = PyString_AS_STRING(newobj); + */ + + Py_MEMCPY(result, cptr, len); + + for (i = 0; i < len; i++) { + int c = Py_CHARMASK(result[i]); + if (ISLOWER(c)) + result[i] = TOUPPER(c); + } +} + + +PyDoc_STRVAR_shared(_Py_title__doc__, +"B.title() -> copy of B\n\ +\n\ +Return a titlecased version of B, i.e. ASCII words start with uppercase\n\ +characters, all remaining cased characters have lowercase."); + +void +_Py_bytes_title(char *result, char *s, Py_ssize_t len) +{ + Py_ssize_t i; + int previous_is_cased = 0; + + /* + newobj = PyString_FromStringAndSize(NULL, len); + if (newobj == NULL) + return NULL; + s_new = PyString_AsString(newobj); + */ + for (i = 0; i < len; i++) { + int c = Py_CHARMASK(*s++); + if (ISLOWER(c)) { + if (!previous_is_cased) + c = TOUPPER(c); + previous_is_cased = 1; + } else if (ISUPPER(c)) { + if (previous_is_cased) + c = TOLOWER(c); + previous_is_cased = 1; + } else + previous_is_cased = 0; + *result++ = c; + } +} + + +PyDoc_STRVAR_shared(_Py_capitalize__doc__, +"B.capitalize() -> copy of B\n\ +\n\ +Return a copy of B with only its first character capitalized (ASCII)."); + +void +_Py_bytes_capitalize(char *result, char *s, Py_ssize_t len) +{ + Py_ssize_t i; + + /* + newobj = PyString_FromStringAndSize(NULL, len); + if (newobj == NULL) + return NULL; + s_new = PyString_AsString(newobj); + */ + if (0 < len) { + int c = Py_CHARMASK(*s++); + if (ISLOWER(c)) + *result = TOUPPER(c); + else + *result = c; + result++; + } + for (i = 1; i < len; i++) { + int c = Py_CHARMASK(*s++); + if (ISUPPER(c)) + *result = TOLOWER(c); + else + *result = c; + result++; + } +} + + +PyDoc_STRVAR_shared(_Py_swapcase__doc__, +"B.swapcase() -> copy of B\n\ +\n\ +Return a copy of B with uppercase ASCII characters converted\n\ +to lowercase ASCII and vice versa."); + +void +_Py_bytes_swapcase(char *result, char *s, Py_ssize_t len) +{ + Py_ssize_t i; + + /* + newobj = PyString_FromStringAndSize(NULL, len); + if (newobj == NULL) + return NULL; + s_new = PyString_AsString(newobj); + */ + for (i = 0; i < len; i++) { + int c = Py_CHARMASK(*s++); + if (ISLOWER(c)) { + *result = TOUPPER(c); + } + else if (ISUPPER(c)) { + *result = TOLOWER(c); + } + else + *result = c; + result++; + } +} + Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Tue Oct 16 08:31:30 2007 @@ -5,11 +5,8 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" #include "structmember.h" +#include "bytes_methods.h" -/* The nullbytes are used by the stringlib during partition. - * If partition is removed from bytes, nullbytes and its helper - * Init/Fini should also be removed. - */ static PyBytesObject *nullbytes = NULL; void @@ -37,15 +34,20 @@ static int _getbytevalue(PyObject* arg, int *value) { - PyObject *intarg = PyNumber_Int(arg); - if (! intarg) - return 0; - *value = PyInt_AsLong(intarg); - Py_DECREF(intarg); - if (*value < 0 || *value >= 256) { - PyErr_SetString(PyExc_ValueError, "byte must be in range(0, 256)"); + long face_value; + + if (PyInt_Check(arg)) { + face_value = PyInt_AsLong(arg); + if (face_value < 0 || face_value >= 256) { + PyErr_SetString(PyExc_ValueError, "byte must be in range(0, 256)"); + return 0; + } + } else { + PyErr_Format(PyExc_TypeError, "an integer is required"); return 0; } + + *value = face_value; return 1; } @@ -80,9 +82,7 @@ { PyBufferProcs *buffer = Py_Type(obj)->tp_as_buffer; - if (buffer == NULL || - PyUnicode_Check(obj) || - buffer->bf_getbuffer == NULL) + if (buffer == NULL || buffer->bf_getbuffer == NULL) { PyErr_Format(PyExc_TypeError, "Type %.100s doesn't support the buffer API", @@ -1035,13 +1035,18 @@ #define STRINGLIB_CHAR char #define STRINGLIB_CMP memcmp #define STRINGLIB_LEN PyBytes_GET_SIZE +#define STRINGLIB_STR PyBytes_AS_STRING #define STRINGLIB_NEW PyBytes_FromStringAndSize #define STRINGLIB_EMPTY nullbytes +#define STRINGLIB_CHECK_EXACT PyBytes_CheckExact +#define STRINGLIB_MUTABLE 1 #include "stringlib/fastsearch.h" #include "stringlib/count.h" #include "stringlib/find.h" #include "stringlib/partition.h" +#include "stringlib/ctype.h" +#include "stringlib/transmogrify.h" /* The following Py_LOCAL_INLINE and Py_LOCAL functions @@ -1088,7 +1093,6 @@ return res; } - PyDoc_STRVAR(find__doc__, "B.find(sub [,start [,end]]) -> int\n\ \n\ @@ -1118,27 +1122,25 @@ bytes_count(PyBytesObject *self, PyObject *args) { PyObject *sub_obj; - const char *str = PyBytes_AS_STRING(self), *sub; - Py_ssize_t sub_len; + const char *str = PyBytes_AS_STRING(self); Py_ssize_t start = 0, end = PY_SSIZE_T_MAX; + Py_buffer vsub; + PyObject *count_obj; if (!PyArg_ParseTuple(args, "O|O&O&:count", &sub_obj, _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end)) return NULL; - if (PyBytes_Check(sub_obj)) { - sub = PyBytes_AS_STRING(sub_obj); - sub_len = PyBytes_GET_SIZE(sub_obj); - } - /* XXX --> use the modern buffer interface */ - else if (PyObject_AsCharBuffer(sub_obj, &sub, &sub_len)) + if (_getbuffer(sub_obj, &vsub) < 0) return NULL; _adjust_indices(&start, &end, PyBytes_GET_SIZE(self)); - return PyInt_FromSsize_t( - stringlib_count(str + start, end - start, sub, sub_len) + count_obj = PyInt_FromSsize_t( + stringlib_count(str + start, end - start, vsub.buf, vsub.len) ); + PyObject_ReleaseBuffer(sub_obj, &vsub); + return count_obj; } @@ -1210,36 +1212,39 @@ Py_ssize_t end, int direction) { Py_ssize_t len = PyBytes_GET_SIZE(self); - Py_ssize_t slen; - const char* sub; const char* str; + Py_buffer vsubstr; + int rv; - if (PyBytes_Check(substr)) { - sub = PyBytes_AS_STRING(substr); - slen = PyBytes_GET_SIZE(substr); - } - /* XXX --> Use the modern buffer interface */ - else if (PyObject_AsCharBuffer(substr, &sub, &slen)) - return -1; str = PyBytes_AS_STRING(self); + if (_getbuffer(substr, &vsubstr) < 0) + return -1; + _adjust_indices(&start, &end, len); if (direction < 0) { /* startswith */ - if (start+slen > len) - return 0; + if (start+vsubstr.len > len) { + rv = 0; + goto done; + } } else { /* endswith */ - if (end-start < slen || start > len) - return 0; + if (end-start < vsubstr.len || start > len) { + rv = 0; + goto done; + } - if (end-slen > start) - start = end - slen; + if (end-vsubstr.len > start) + start = end - vsubstr.len; } - if (end-start >= slen) - return ! memcmp(str+start, sub, slen); - return 0; + if (end-start >= vsubstr.len) + rv = ! memcmp(str+start, vsubstr.buf, vsubstr.len); + +done: + PyObject_ReleaseBuffer(substr, &vsubstr); + return rv; } @@ -1324,7 +1329,6 @@ } - PyDoc_STRVAR(translate__doc__, "B.translate(table [,deletechars]) -> bytes\n\ \n\ @@ -1340,53 +1344,47 @@ register const char *table; register Py_ssize_t i, c, changed = 0; PyObject *input_obj = (PyObject*)self; - const char *table1, *output_start, *del_table=NULL; - Py_ssize_t inlen, tablen, dellen = 0; + const char *output_start; + Py_ssize_t inlen; PyObject *result; int trans_table[256]; PyObject *tableobj, *delobj = NULL; + Py_buffer vtable, vdel; if (!PyArg_UnpackTuple(args, "translate", 1, 2, &tableobj, &delobj)) return NULL; - if (PyBytes_Check(tableobj)) { - table1 = PyBytes_AS_STRING(tableobj); - tablen = PyBytes_GET_SIZE(tableobj); - } - /* XXX -> Use the modern buffer interface */ - else if (PyObject_AsCharBuffer(tableobj, &table1, &tablen)) + if (_getbuffer(tableobj, &vtable) < 0) return NULL; - if (tablen != 256) { + if (vtable.len != 256) { PyErr_SetString(PyExc_ValueError, "translation table must be 256 characters long"); - return NULL; + result = NULL; + goto done; } if (delobj != NULL) { - if (PyBytes_Check(delobj)) { - del_table = PyBytes_AS_STRING(delobj); - dellen = PyBytes_GET_SIZE(delobj); + if (_getbuffer(delobj, &vdel) < 0) { + result = NULL; + goto done; } - /* XXX -> use the modern buffer interface */ - else if (PyObject_AsCharBuffer(delobj, &del_table, &dellen)) - return NULL; } else { - del_table = NULL; - dellen = 0; + vdel.buf = NULL; + vdel.len = 0; } - table = table1; + table = (const char *)vtable.buf; inlen = PyBytes_GET_SIZE(input_obj); result = PyBytes_FromStringAndSize((char *)NULL, inlen); if (result == NULL) - return NULL; + goto done; output_start = output = PyBytes_AsString(result); input = PyBytes_AS_STRING(input_obj); - if (dellen == 0) { + if (vdel.len == 0) { /* If no deletions are required, use faster code */ for (i = inlen; --i >= 0; ) { c = Py_CHARMASK(*input++); @@ -1394,17 +1392,18 @@ changed = 1; } if (changed || !PyBytes_CheckExact(input_obj)) - return result; + goto done; Py_DECREF(result); Py_INCREF(input_obj); - return input_obj; + result = input_obj; + goto done; } for (i = 0; i < 256; i++) trans_table[i] = Py_CHARMASK(table[i]); - for (i = 0; i < dellen; i++) - trans_table[(int) Py_CHARMASK(del_table[i])] = -1; + for (i = 0; i < vdel.len; i++) + trans_table[(int) Py_CHARMASK( ((unsigned char*)vdel.buf)[i] )] = -1; for (i = inlen; --i >= 0; ) { c = Py_CHARMASK(*input++); @@ -1416,11 +1415,17 @@ if (!changed && PyBytes_CheckExact(input_obj)) { Py_DECREF(result); Py_INCREF(input_obj); - return input_obj; + result = input_obj; + goto done; } /* Fix the size of the resulting string */ if (inlen > 0) PyBytes_Resize(result, output - output_start); + +done: + PyObject_ReleaseBuffer(tableobj, &vtable); + if (delobj != NULL) + PyObject_ReleaseBuffer(delobj, &vdel); return result; } @@ -2021,6 +2026,7 @@ } } + PyDoc_STRVAR(replace__doc__, "B.replace (old, new[, count]) -> bytes\n\ \n\ @@ -2133,7 +2139,6 @@ return NULL; } -#define ISSPACE(c) (isspace(Py_CHARMASK(c)) && ((c) & 0x80) == 0) Py_LOCAL_INLINE(PyObject *) split_whitespace(const char *s, Py_ssize_t len, Py_ssize_t maxcount) @@ -2459,6 +2464,10 @@ static PyObject * bytes_extend(PyBytesObject *self, PyObject *arg) { + /* XXX(gps): The docstring says any iterable int will do but the + * bytes_setslice code only accepts something supporting PEP 3118. + * A list or tuple of 0 <= int <= 255 is supposed to work. */ + /* bug being tracked on: http://bugs.python.org/issue1283 */ if (bytes_setslice(self, Py_Size(self), Py_Size(self), arg) == -1) return NULL; Py_RETURN_NONE; @@ -2852,11 +2861,11 @@ static int hex_digit_to_int(int c) { - if (isdigit(c)) + if (ISDIGIT(c)) return c - '0'; else { - if (isupper(c)) - c = tolower(c); + if (ISUPPER(c)) + c = TOLOWER(c); if (c >= 'a' && c <= 'f') return c - 'a' + 10; } @@ -2866,26 +2875,34 @@ static PyObject * bytes_fromhex(PyObject *cls, PyObject *args) { - PyObject *newbytes; - char *hex, *buf; - Py_ssize_t len, byteslen, i, j; + PyObject *newbytes, *hexobj; + char *buf; + unsigned char *hex; + Py_ssize_t byteslen, i, j; int top, bot; + Py_buffer vhex; + + if (!PyArg_ParseTuple(args, "O:fromhex", &hexobj)) + return NULL; - if (!PyArg_ParseTuple(args, "s#:fromhex", &hex, &len)) + if (_getbuffer(hexobj, &vhex) < 0) return NULL; - byteslen = len / 2; /* max length if there are no spaces */ + byteslen = vhex.len / 2; /* max length if there are no spaces */ + hex = vhex.buf; newbytes = PyBytes_FromStringAndSize(NULL, byteslen); - if (!newbytes) + if (!newbytes) { + PyObject_ReleaseBuffer(hexobj, &vhex); return NULL; + } buf = PyBytes_AS_STRING(newbytes); - for (i = j = 0; i < len; i += 2) { + for (i = j = 0; i < vhex.len; i += 2) { /* skip over spaces in the input */ while (Py_CHARMASK(hex[i]) == ' ') i++; - if (i >= len) + if (i >= vhex.len) break; top = hex_digit_to_int(Py_CHARMASK(hex[i])); bot = hex_digit_to_int(Py_CHARMASK(hex[i+1])); @@ -2900,10 +2917,12 @@ } if (PyBytes_Resize(newbytes, j) < 0) goto error; + PyObject_ReleaseBuffer(hexobj, &vhex); return newbytes; error: Py_DECREF(newbytes); + PyObject_ReleaseBuffer(hexobj, &vhex); return NULL; } @@ -2955,6 +2974,19 @@ {"endswith", (PyCFunction)bytes_endswith, METH_VARARGS, endswith__doc__}, {"startswith", (PyCFunction)bytes_startswith, METH_VARARGS, startswith__doc__}, + {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, + {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, + {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, + _Py_capitalize__doc__}, + {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, + _Py_swapcase__doc__}, + {"islower", (PyCFunction)stringlib_islower, METH_NOARGS,_Py_islower__doc__}, + {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS,_Py_isupper__doc__}, + {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS,_Py_isspace__doc__}, + {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS,_Py_isdigit__doc__}, + {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS,_Py_istitle__doc__}, + {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS,_Py_isalpha__doc__}, + {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS,_Py_isalnum__doc__}, {"replace", (PyCFunction)bytes_replace, METH_VARARGS, replace__doc__}, {"translate", (PyCFunction)bytes_translate, METH_VARARGS, translate__doc__}, {"partition", (PyCFunction)bytes_partition, METH_O, partition__doc__}, @@ -2975,6 +3007,15 @@ {"fromhex", (PyCFunction)bytes_fromhex, METH_VARARGS|METH_CLASS, fromhex_doc}, {"join", (PyCFunction)bytes_join, METH_O, join_doc}, + {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, + {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, ljust__doc__}, + {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, rjust__doc__}, + {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, + {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, zfill__doc__}, + {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, + expandtabs__doc__}, + {"splitlines", (PyCFunction)stringlib_splitlines, METH_VARARGS, + splitlines__doc__}, {"__reduce__", (PyCFunction)bytes_reduce, METH_NOARGS, reduce_doc}, {NULL} }; Modified: python/branches/py3k/Objects/stringlib/README.txt ============================================================================== --- python/branches/py3k/Objects/stringlib/README.txt (original) +++ python/branches/py3k/Objects/stringlib/README.txt Tue Oct 16 08:31:30 2007 @@ -32,3 +32,12 @@ returns the pointer to the character data for the given string object (which must be of the right type) + +int STRINGLIB_CHECK_EXACT(PyObject *) + + returns true if the object is an instance of our type, not a subclass. + +STRINGLIB_MUTABLE + + Must be 0 or 1 to tell the cpp macros in stringlib code if the object + being operated on is mutable or not. Added: python/branches/py3k/Objects/stringlib/ctype.h ============================================================================== --- (empty file) +++ python/branches/py3k/Objects/stringlib/ctype.h Tue Oct 16 08:31:30 2007 @@ -0,0 +1,110 @@ +/* NOTE: this API is -ONLY- for use with single byte character strings. */ +/* Do not use it with Unicode. */ + +#include "bytes_methods.h" + +static PyObject* +stringlib_isspace(PyObject *self) +{ + return _Py_bytes_isspace(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_isalpha(PyObject *self) +{ + return _Py_bytes_isalpha(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_isalnum(PyObject *self) +{ + return _Py_bytes_isalnum(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_isdigit(PyObject *self) +{ + return _Py_bytes_isdigit(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_islower(PyObject *self) +{ + return _Py_bytes_islower(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_isupper(PyObject *self) +{ + return _Py_bytes_isupper(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + +static PyObject* +stringlib_istitle(PyObject *self) +{ + return _Py_bytes_istitle(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +} + + +/* functions that return a new object partially translated by ctype funcs: */ + +static PyObject* +stringlib_lower(PyObject *self) +{ + PyObject* newobj; + newobj = STRINGLIB_NEW(NULL, STRINGLIB_LEN(self)); + if (!newobj) + return NULL; + _Py_bytes_lower(STRINGLIB_STR(newobj), STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + return newobj; +} + +static PyObject* +stringlib_upper(PyObject *self) +{ + PyObject* newobj; + newobj = STRINGLIB_NEW(NULL, STRINGLIB_LEN(self)); + if (!newobj) + return NULL; + _Py_bytes_upper(STRINGLIB_STR(newobj), STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + return newobj; +} + +static PyObject* +stringlib_title(PyObject *self) +{ + PyObject* newobj; + newobj = STRINGLIB_NEW(NULL, STRINGLIB_LEN(self)); + if (!newobj) + return NULL; + _Py_bytes_title(STRINGLIB_STR(newobj), STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + return newobj; +} + +static PyObject* +stringlib_capitalize(PyObject *self) +{ + PyObject* newobj; + newobj = STRINGLIB_NEW(NULL, STRINGLIB_LEN(self)); + if (!newobj) + return NULL; + _Py_bytes_capitalize(STRINGLIB_STR(newobj), STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + return newobj; +} + +static PyObject* +stringlib_swapcase(PyObject *self) +{ + PyObject* newobj; + newobj = STRINGLIB_NEW(NULL, STRINGLIB_LEN(self)); + if (!newobj) + return NULL; + _Py_bytes_swapcase(STRINGLIB_STR(newobj), STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + return newobj; +} + Modified: python/branches/py3k/Objects/stringlib/find.h ============================================================================== --- python/branches/py3k/Objects/stringlib/find.h (original) +++ python/branches/py3k/Objects/stringlib/find.h Tue Oct 16 08:31:30 2007 @@ -90,7 +90,7 @@ return stringlib_rfind(str + start, end - start, sub, sub_len, start); } -#ifdef STRINGLIB_STR +#ifdef STRINGLIB_WANT_CONTAINS_OBJ Py_LOCAL_INLINE(int) stringlib_contains_obj(PyObject* str, PyObject* sub) Added: python/branches/py3k/Objects/stringlib/transmogrify.h ============================================================================== --- (empty file) +++ python/branches/py3k/Objects/stringlib/transmogrify.h Tue Oct 16 08:31:30 2007 @@ -0,0 +1,362 @@ +/* NOTE: this API is -ONLY- for use with single byte character strings. */ +/* Do not use it with Unicode. */ + +#include "bytes_methods.h" + +#ifndef STRINGLIB_MUTABLE +#warning "STRINGLIB_MUTABLE not defined before #include, assuming 0" +#define STRINGLIB_MUTABLE 0 +#endif + +/* the more complicated methods. parts of these should be pulled out into the + shared code in bytes_methods.c to cut down on duplicate code bloat. */ + +PyDoc_STRVAR(expandtabs__doc__, +"B.expandtabs([tabsize]) -> modified copy of B\n\ +\n\ +Return a copy of B where all tab characters are expanded using spaces.\n\ +If tabsize is not given, a tab size of 8 characters is assumed."); + +static PyObject* +stringlib_expandtabs(PyObject *self, PyObject *args) +{ + const char *e, *p; + char *q; + Py_ssize_t i, j, old_j; + PyObject *u; + int tabsize = 8; + + if (!PyArg_ParseTuple(args, "|i:expandtabs", &tabsize)) + return NULL; + + /* First pass: determine size of output string */ + i = j = old_j = 0; + e = STRINGLIB_STR(self) + STRINGLIB_LEN(self); + for (p = STRINGLIB_STR(self); p < e; p++) + if (*p == '\t') { + if (tabsize > 0) { + j += tabsize - (j % tabsize); + /* XXX: this depends on a signed integer overflow to < 0 */ + /* C compilers, including gcc, do -NOT- guarantee this. */ + if (old_j > j) { + PyErr_SetString(PyExc_OverflowError, + "result is too long"); + return NULL; + } + old_j = j; + } + } + else { + j++; + if (*p == '\n' || *p == '\r') { + i += j; + old_j = j = 0; + /* XXX: this depends on a signed integer overflow to < 0 */ + /* C compilers, including gcc, do -NOT- guarantee this. */ + if (i < 0) { + PyErr_SetString(PyExc_OverflowError, + "result is too long"); + return NULL; + } + } + } + + if ((i + j) < 0) { + /* XXX: this depends on a signed integer overflow to < 0 */ + /* C compilers, including gcc, do -NOT- guarantee this. */ + PyErr_SetString(PyExc_OverflowError, "result is too long"); + return NULL; + } + + /* Second pass: create output string and fill it */ + u = STRINGLIB_NEW(NULL, i + j); + if (!u) + return NULL; + + j = 0; + q = STRINGLIB_STR(u); + + for (p = STRINGLIB_STR(self); p < e; p++) + if (*p == '\t') { + if (tabsize > 0) { + i = tabsize - (j % tabsize); + j += i; + while (i--) + *q++ = ' '; + } + } + else { + j++; + *q++ = *p; + if (*p == '\n' || *p == '\r') + j = 0; + } + + return u; +} + +Py_LOCAL_INLINE(PyObject *) +pad(PyObject *self, Py_ssize_t left, Py_ssize_t right, char fill) +{ + PyObject *u; + + if (left < 0) + left = 0; + if (right < 0) + right = 0; + + if (left == 0 && right == 0 && STRINGLIB_CHECK_EXACT(self)) { +#if STRINGLIB_MUTABLE + /* We're defined as returning a copy; If the object is mutable + * that means we must make an identical copy. */ + return STRINGLIB_NEW(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +#else + Py_INCREF(self); + return (PyObject *)self; +#endif /* STRINGLIB_MUTABLE */ + } + + u = STRINGLIB_NEW(NULL, + left + STRINGLIB_LEN(self) + right); + if (u) { + if (left) + memset(STRINGLIB_STR(u), fill, left); + Py_MEMCPY(STRINGLIB_STR(u) + left, + STRINGLIB_STR(self), + STRINGLIB_LEN(self)); + if (right) + memset(STRINGLIB_STR(u) + left + STRINGLIB_LEN(self), + fill, right); + } + + return u; +} + +PyDoc_STRVAR(ljust__doc__, +"B.ljust(width[, fillchar]) -> modified copy of B\n" +"\n" +"Return B left justified in a string of length width. Padding is\n" +"done using the specified fill character (default is a space)."); + +static PyObject * +stringlib_ljust(PyObject *self, PyObject *args) +{ + Py_ssize_t width; + char fillchar = ' '; + + if (!PyArg_ParseTuple(args, "n|c:ljust", &width, &fillchar)) + return NULL; + + if (STRINGLIB_LEN(self) >= width && STRINGLIB_CHECK_EXACT(self)) { +#if STRINGLIB_MUTABLE + /* We're defined as returning a copy; If the object is mutable + * that means we must make an identical copy. */ + return STRINGLIB_NEW(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +#else + Py_INCREF(self); + return (PyObject*) self; +#endif + } + + return pad(self, 0, width - STRINGLIB_LEN(self), fillchar); +} + + +PyDoc_STRVAR(rjust__doc__, +"B.rjust(width[, fillchar]) -> modified copy of B\n" +"\n" +"Return B right justified in a string of length width. Padding is\n" +"done using the specified fill character (default is a space)"); + +static PyObject * +stringlib_rjust(PyObject *self, PyObject *args) +{ + Py_ssize_t width; + char fillchar = ' '; + + if (!PyArg_ParseTuple(args, "n|c:rjust", &width, &fillchar)) + return NULL; + + if (STRINGLIB_LEN(self) >= width && STRINGLIB_CHECK_EXACT(self)) { +#if STRINGLIB_MUTABLE + /* We're defined as returning a copy; If the object is mutable + * that means we must make an identical copy. */ + return STRINGLIB_NEW(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +#else + Py_INCREF(self); + return (PyObject*) self; +#endif + } + + return pad(self, width - STRINGLIB_LEN(self), 0, fillchar); +} + + +PyDoc_STRVAR(center__doc__, +"B.center(width[, fillchar]) -> modified copy of B\n" +"\n" +"Return B centered in a string of length width. Padding is\n" +"done using the specified fill character (default is a space)"); + +static PyObject * +stringlib_center(PyObject *self, PyObject *args) +{ + Py_ssize_t marg, left; + Py_ssize_t width; + char fillchar = ' '; + + if (!PyArg_ParseTuple(args, "n|c:center", &width, &fillchar)) + return NULL; + + if (STRINGLIB_LEN(self) >= width && STRINGLIB_CHECK_EXACT(self)) { +#if STRINGLIB_MUTABLE + /* We're defined as returning a copy; If the object is mutable + * that means we must make an identical copy. */ + return STRINGLIB_NEW(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +#else + Py_INCREF(self); + return (PyObject*) self; +#endif + } + + marg = width - STRINGLIB_LEN(self); + left = marg / 2 + (marg & width & 1); + + return pad(self, left, marg - left, fillchar); +} + +PyDoc_STRVAR(zfill__doc__, +"B.zfill(width) -> modified copy of B\n" +"\n" +"Pad a numeric string B with zeros on the left, to fill a field\n" +"of the specified width. B is never truncated."); + +static PyObject * +stringlib_zfill(PyObject *self, PyObject *args) +{ + Py_ssize_t fill; + PyObject *s; + char *p; + Py_ssize_t width; + + if (!PyArg_ParseTuple(args, "n:zfill", &width)) + return NULL; + + if (STRINGLIB_LEN(self) >= width) { + if (STRINGLIB_CHECK_EXACT(self)) { +#if STRINGLIB_MUTABLE + /* We're defined as returning a copy; If the object is mutable + * that means we must make an identical copy. */ + return STRINGLIB_NEW(STRINGLIB_STR(self), STRINGLIB_LEN(self)); +#else + Py_INCREF(self); + return (PyObject*) self; +#endif + } + else + return STRINGLIB_NEW( + STRINGLIB_STR(self), + STRINGLIB_LEN(self) + ); + } + + fill = width - STRINGLIB_LEN(self); + + s = pad(self, fill, 0, '0'); + + if (s == NULL) + return NULL; + + p = STRINGLIB_STR(s); + if (p[fill] == '+' || p[fill] == '-') { + /* move sign to beginning of string */ + p[0] = p[fill]; + p[fill] = '0'; + } + + return (PyObject*) s; +} + + +#define _STRINGLIB_SPLIT_APPEND(data, left, right) \ + str = STRINGLIB_NEW((data) + (left), \ + (right) - (left)); \ + if (str == NULL) \ + goto onError; \ + if (PyList_Append(list, str)) { \ + Py_DECREF(str); \ + goto onError; \ + } \ + else \ + Py_DECREF(str); + +PyDoc_STRVAR(splitlines__doc__, +"B.splitlines([keepends]) -> list of lines\n\ +\n\ +Return a list of the lines in B, breaking at line boundaries.\n\ +Line breaks are not included in the resulting list unless keepends\n\ +is given and true."); + +static PyObject* +stringlib_splitlines(PyObject *self, PyObject *args) +{ + register Py_ssize_t i; + register Py_ssize_t j; + Py_ssize_t len; + int keepends = 0; + PyObject *list; + PyObject *str; + char *data; + + if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends)) + return NULL; + + data = STRINGLIB_STR(self); + len = STRINGLIB_LEN(self); + + /* This does not use the preallocated list because splitlines is + usually run with hundreds of newlines. The overhead of + switching between PyList_SET_ITEM and append causes about a + 2-3% slowdown for that common case. A smarter implementation + could move the if check out, so the SET_ITEMs are done first + and the appends only done when the prealloc buffer is full. + That's too much work for little gain.*/ + + list = PyList_New(0); + if (!list) + goto onError; + + for (i = j = 0; i < len; ) { + Py_ssize_t eol; + + /* Find a line and append it */ + while (i < len && data[i] != '\n' && data[i] != '\r') + i++; + + /* Skip the line break reading CRLF as one line break */ + eol = i; + if (i < len) { + if (data[i] == '\r' && i + 1 < len && + data[i+1] == '\n') + i += 2; + else + i++; + if (keepends) + eol = i; + } + _STRINGLIB_SPLIT_APPEND(data, j, eol); + j = i; + } + if (j < len) { + _STRINGLIB_SPLIT_APPEND(data, j, len); + } + + return list; + + onError: + Py_XDECREF(list); + return NULL; +} + +#undef _STRINGLIB_SPLIT_APPEND + Modified: python/branches/py3k/Objects/stringlib/unicodedefs.h ============================================================================== --- python/branches/py3k/Objects/stringlib/unicodedefs.h (original) +++ python/branches/py3k/Objects/stringlib/unicodedefs.h Tue Oct 16 08:31:30 2007 @@ -22,6 +22,8 @@ #define STRINGLIB_CHECK PyUnicode_Check #define STRINGLIB_TOSTR PyObject_Unicode +#define STRINGLIB_WANT_CONTAINS_OBJ 1 + /* STRINGLIB_CMP was defined as: Py_LOCAL_INLINE(int) Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Tue Oct 16 08:31:30 2007 @@ -4,256 +4,7 @@ #include "Python.h" -/* Our own locale-independent ctype.h-like macros */ -/* XXX Move into a header file? */ - -#define FLAG_LOWER 0x01 -#define FLAG_UPPER 0x02 -#define FLAG_ALPHA (FLAG_LOWER|FLAG_UPPER) -#define FLAG_DIGIT 0x04 -#define FLAG_ALNUM (FLAG_ALPHA|FLAG_DIGIT) -#define FLAG_SPACE 0x08 -#define FLAG_XDIGIT 0x10 - -static unsigned int ctype_table[256] = { - 0, /* 0x0 '\x00' */ - 0, /* 0x1 '\x01' */ - 0, /* 0x2 '\x02' */ - 0, /* 0x3 '\x03' */ - 0, /* 0x4 '\x04' */ - 0, /* 0x5 '\x05' */ - 0, /* 0x6 '\x06' */ - 0, /* 0x7 '\x07' */ - 0, /* 0x8 '\x08' */ - FLAG_SPACE, /* 0x9 '\t' */ - FLAG_SPACE, /* 0xa '\n' */ - FLAG_SPACE, /* 0xb '\v' */ - FLAG_SPACE, /* 0xc '\f' */ - FLAG_SPACE, /* 0xd '\r' */ - 0, /* 0xe '\x0e' */ - 0, /* 0xf '\x0f' */ - 0, /* 0x10 '\x10' */ - 0, /* 0x11 '\x11' */ - 0, /* 0x12 '\x12' */ - 0, /* 0x13 '\x13' */ - 0, /* 0x14 '\x14' */ - 0, /* 0x15 '\x15' */ - 0, /* 0x16 '\x16' */ - 0, /* 0x17 '\x17' */ - 0, /* 0x18 '\x18' */ - 0, /* 0x19 '\x19' */ - 0, /* 0x1a '\x1a' */ - 0, /* 0x1b '\x1b' */ - 0, /* 0x1c '\x1c' */ - 0, /* 0x1d '\x1d' */ - 0, /* 0x1e '\x1e' */ - 0, /* 0x1f '\x1f' */ - FLAG_SPACE, /* 0x20 ' ' */ - 0, /* 0x21 '!' */ - 0, /* 0x22 '"' */ - 0, /* 0x23 '#' */ - 0, /* 0x24 '$' */ - 0, /* 0x25 '%' */ - 0, /* 0x26 '&' */ - 0, /* 0x27 "'" */ - 0, /* 0x28 '(' */ - 0, /* 0x29 ')' */ - 0, /* 0x2a '*' */ - 0, /* 0x2b '+' */ - 0, /* 0x2c ',' */ - 0, /* 0x2d '-' */ - 0, /* 0x2e '.' */ - 0, /* 0x2f '/' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x30 '0' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x31 '1' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x32 '2' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x33 '3' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x34 '4' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x35 '5' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x36 '6' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x37 '7' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x38 '8' */ - FLAG_DIGIT|FLAG_XDIGIT, /* 0x39 '9' */ - 0, /* 0x3a ':' */ - 0, /* 0x3b ';' */ - 0, /* 0x3c '<' */ - 0, /* 0x3d '=' */ - 0, /* 0x3e '>' */ - 0, /* 0x3f '?' */ - 0, /* 0x40 '@' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x41 'A' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x42 'B' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x43 'C' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x44 'D' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x45 'E' */ - FLAG_UPPER|FLAG_XDIGIT, /* 0x46 'F' */ - FLAG_UPPER, /* 0x47 'G' */ - FLAG_UPPER, /* 0x48 'H' */ - FLAG_UPPER, /* 0x49 'I' */ - FLAG_UPPER, /* 0x4a 'J' */ - FLAG_UPPER, /* 0x4b 'K' */ - FLAG_UPPER, /* 0x4c 'L' */ - FLAG_UPPER, /* 0x4d 'M' */ - FLAG_UPPER, /* 0x4e 'N' */ - FLAG_UPPER, /* 0x4f 'O' */ - FLAG_UPPER, /* 0x50 'P' */ - FLAG_UPPER, /* 0x51 'Q' */ - FLAG_UPPER, /* 0x52 'R' */ - FLAG_UPPER, /* 0x53 'S' */ - FLAG_UPPER, /* 0x54 'T' */ - FLAG_UPPER, /* 0x55 'U' */ - FLAG_UPPER, /* 0x56 'V' */ - FLAG_UPPER, /* 0x57 'W' */ - FLAG_UPPER, /* 0x58 'X' */ - FLAG_UPPER, /* 0x59 'Y' */ - FLAG_UPPER, /* 0x5a 'Z' */ - 0, /* 0x5b '[' */ - 0, /* 0x5c '\\' */ - 0, /* 0x5d ']' */ - 0, /* 0x5e '^' */ - 0, /* 0x5f '_' */ - 0, /* 0x60 '`' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x61 'a' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x62 'b' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x63 'c' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x64 'd' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x65 'e' */ - FLAG_LOWER|FLAG_XDIGIT, /* 0x66 'f' */ - FLAG_LOWER, /* 0x67 'g' */ - FLAG_LOWER, /* 0x68 'h' */ - FLAG_LOWER, /* 0x69 'i' */ - FLAG_LOWER, /* 0x6a 'j' */ - FLAG_LOWER, /* 0x6b 'k' */ - FLAG_LOWER, /* 0x6c 'l' */ - FLAG_LOWER, /* 0x6d 'm' */ - FLAG_LOWER, /* 0x6e 'n' */ - FLAG_LOWER, /* 0x6f 'o' */ - FLAG_LOWER, /* 0x70 'p' */ - FLAG_LOWER, /* 0x71 'q' */ - FLAG_LOWER, /* 0x72 'r' */ - FLAG_LOWER, /* 0x73 's' */ - FLAG_LOWER, /* 0x74 't' */ - FLAG_LOWER, /* 0x75 'u' */ - FLAG_LOWER, /* 0x76 'v' */ - FLAG_LOWER, /* 0x77 'w' */ - FLAG_LOWER, /* 0x78 'x' */ - FLAG_LOWER, /* 0x79 'y' */ - FLAG_LOWER, /* 0x7a 'z' */ - 0, /* 0x7b '{' */ - 0, /* 0x7c '|' */ - 0, /* 0x7d '}' */ - 0, /* 0x7e '~' */ - 0, /* 0x7f '\x7f' */ - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -}; - -#define ISLOWER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_LOWER) -#define ISUPPER(c) (ctype_table[Py_CHARMASK(c)] & FLAG_UPPER) -#define ISALPHA(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALPHA) -#define ISDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_DIGIT) -#define ISXDIGIT(c) (ctype_table[Py_CHARMASK(c)] & FLAG_XDIGIT) -#define ISALNUM(c) (ctype_table[Py_CHARMASK(c)] & FLAG_ALNUM) -#define ISSPACE(c) (ctype_table[Py_CHARMASK(c)] & FLAG_SPACE) - -#undef islower -#define islower(c) undefined_islower(c) -#undef isupper -#define isupper(c) undefined_isupper(c) -#undef isalpha -#define isalpha(c) undefined_isalpha(c) -#undef isdigit -#define isdigit(c) undefined_isdigit(c) -#undef isxdigit -#define isxdigit(c) undefined_isxdigit(c) -#undef isalnum -#define isalnum(c) undefined_isalnum(c) -#undef isspace -#define isspace(c) undefined_isspace(c) - -static unsigned char ctype_tolower[256] = { - 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, - 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, - 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, - 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, - 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, - 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, - 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, - 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, - 0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, - 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, - 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, - 0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, - 0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, - 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, - 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, - 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, - 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, - 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, - 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, - 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, - 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, - 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, - 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, - 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, - 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, - 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, - 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, - 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, - 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, - 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, - 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, - 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, -}; - -static unsigned char ctype_toupper[256] = { - 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, - 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, - 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, - 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, - 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, - 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, - 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, - 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, - 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, - 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, - 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, - 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, - 0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, - 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, - 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, - 0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, - 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, - 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, - 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, - 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, - 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, - 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, - 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, - 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, - 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, - 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, - 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, - 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, - 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, - 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, - 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, - 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, -}; - -#define TOLOWER(c) (ctype_tolower[Py_CHARMASK(c)]) -#define TOUPPER(c) (ctype_toupper[Py_CHARMASK(c)]) - -#undef tolower -#define tolower(c) undefined_tolower(c) -#undef toupper -#define toupper(c) undefined_toupper(c) +#include "bytes_methods.h" #ifdef COUNT_ALLOCS int null_strings, one_strings; @@ -1002,14 +753,19 @@ #define STRINGLIB_LEN PyString_GET_SIZE #define STRINGLIB_NEW PyString_FromStringAndSize #define STRINGLIB_STR PyString_AS_STRING +#define STRINGLIB_WANT_CONTAINS_OBJ 1 #define STRINGLIB_EMPTY nullstring +#define STRINGLIB_CHECK_EXACT PyString_CheckExact +#define STRINGLIB_MUTABLE 0 #include "stringlib/fastsearch.h" #include "stringlib/count.h" #include "stringlib/find.h" #include "stringlib/partition.h" +#include "stringlib/ctype.h" +#include "stringlib/transmogrify.h" PyObject * @@ -1466,18 +1222,6 @@ #define PREALLOC_SIZE(maxsplit) \ (maxsplit >= MAX_PREALLOC ? MAX_PREALLOC : maxsplit+1) -#define SPLIT_APPEND(data, left, right) \ - str = PyString_FromStringAndSize((data) + (left), \ - (right) - (left)); \ - if (str == NULL) \ - goto onError; \ - if (PyList_Append(list, str)) { \ - Py_DECREF(str); \ - goto onError; \ - } \ - else \ - Py_DECREF(str); - #define SPLIT_ADD(data, left, right) { \ str = PyString_FromStringAndSize((data) + (left), \ (right) - (left)); \ @@ -1839,6 +1583,10 @@ return NULL; } +#undef SPLIT_ADD +#undef MAX_PREALLOC +#undef PREALLOC_SIZE + PyDoc_STRVAR(join__doc__, "S.join(sequence) -> string\n\ @@ -2222,136 +1970,6 @@ } -PyDoc_STRVAR(lower__doc__, -"S.lower() -> string\n\ -\n\ -Return a copy of the string S converted to lowercase."); - -static PyObject * -string_lower(PyStringObject *self) -{ - char *s; - Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *newobj; - - newobj = PyString_FromStringAndSize(NULL, n); - if (!newobj) - return NULL; - - s = PyString_AS_STRING(newobj); - - Py_MEMCPY(s, PyString_AS_STRING(self), n); - - for (i = 0; i < n; i++) { - int c = Py_CHARMASK(s[i]); - if (ISUPPER(c)) - s[i] = TOLOWER(c); - } - - return newobj; -} - -PyDoc_STRVAR(upper__doc__, -"S.upper() -> string\n\ -\n\ -Return a copy of the string S converted to uppercase."); - -static PyObject * -string_upper(PyStringObject *self) -{ - char *s; - Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *newobj; - - newobj = PyString_FromStringAndSize(NULL, n); - if (!newobj) - return NULL; - - s = PyString_AS_STRING(newobj); - - Py_MEMCPY(s, PyString_AS_STRING(self), n); - - for (i = 0; i < n; i++) { - int c = Py_CHARMASK(s[i]); - if (ISLOWER(c)) - s[i] = TOUPPER(c); - } - - return newobj; -} - -PyDoc_STRVAR(title__doc__, -"S.title() -> string\n\ -\n\ -Return a titlecased version of S, i.e. words start with uppercase\n\ -characters, all remaining cased characters have lowercase."); - -static PyObject* -string_title(PyStringObject *self) -{ - char *s = PyString_AS_STRING(self), *s_new; - Py_ssize_t i, n = PyString_GET_SIZE(self); - int previous_is_cased = 0; - PyObject *newobj; - - newobj = PyString_FromStringAndSize(NULL, n); - if (newobj == NULL) - return NULL; - s_new = PyString_AsString(newobj); - for (i = 0; i < n; i++) { - int c = Py_CHARMASK(*s++); - if (ISLOWER(c)) { - if (!previous_is_cased) - c = TOUPPER(c); - previous_is_cased = 1; - } else if (ISUPPER(c)) { - if (previous_is_cased) - c = TOLOWER(c); - previous_is_cased = 1; - } else - previous_is_cased = 0; - *s_new++ = c; - } - return newobj; -} - -PyDoc_STRVAR(capitalize__doc__, -"S.capitalize() -> string\n\ -\n\ -Return a copy of the string S with only its first character\n\ -capitalized."); - -static PyObject * -string_capitalize(PyStringObject *self) -{ - char *s = PyString_AS_STRING(self), *s_new; - Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *newobj; - - newobj = PyString_FromStringAndSize(NULL, n); - if (newobj == NULL) - return NULL; - s_new = PyString_AsString(newobj); - if (0 < n) { - int c = Py_CHARMASK(*s++); - if (ISLOWER(c)) - *s_new = TOUPPER(c); - else - *s_new = c; - s_new++; - } - for (i = 1; i < n; i++) { - int c = Py_CHARMASK(*s++); - if (ISUPPER(c)) - *s_new = TOLOWER(c); - else - *s_new = c; - s_new++; - } - return newobj; -} - - PyDoc_STRVAR(count__doc__, "S.count(sub[, start[, end]]) -> int\n\ \n\ @@ -2393,38 +2011,6 @@ ); } -PyDoc_STRVAR(swapcase__doc__, -"S.swapcase() -> string\n\ -\n\ -Return a copy of the string S with uppercase characters\n\ -converted to lowercase and vice versa."); - -static PyObject * -string_swapcase(PyStringObject *self) -{ - char *s = PyString_AS_STRING(self), *s_new; - Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *newobj; - - newobj = PyString_FromStringAndSize(NULL, n); - if (newobj == NULL) - return NULL; - s_new = PyString_AsString(newobj); - for (i = 0; i < n; i++) { - int c = Py_CHARMASK(*s++); - if (ISLOWER(c)) { - *s_new = TOUPPER(c); - } - else if (ISUPPER(c)) { - *s_new = TOLOWER(c); - } - else - *s_new = c; - s_new++; - } - return newobj; -} - PyDoc_STRVAR(translate__doc__, "S.translate(table [,deletechars]) -> string\n\ @@ -3380,548 +2966,6 @@ } -PyDoc_STRVAR(expandtabs__doc__, -"S.expandtabs([tabsize]) -> string\n\ -\n\ -Return a copy of S where all tab characters are expanded using spaces.\n\ -If tabsize is not given, a tab size of 8 characters is assumed."); - -static PyObject* -string_expandtabs(PyStringObject *self, PyObject *args) -{ - const char *e, *p; - char *q; - Py_ssize_t i, j, old_j; - PyObject *u; - int tabsize = 8; - - if (!PyArg_ParseTuple(args, "|i:expandtabs", &tabsize)) - return NULL; - - /* First pass: determine size of output string */ - i = j = old_j = 0; - e = PyString_AS_STRING(self) + PyString_GET_SIZE(self); - for (p = PyString_AS_STRING(self); p < e; p++) - if (*p == '\t') { - if (tabsize > 0) { - j += tabsize - (j % tabsize); - if (old_j > j) { - PyErr_SetString(PyExc_OverflowError, - "new string is too long"); - return NULL; - } - old_j = j; - } - } - else { - j++; - if (*p == '\n' || *p == '\r') { - i += j; - old_j = j = 0; - if (i < 0) { - PyErr_SetString(PyExc_OverflowError, - "new string is too long"); - return NULL; - } - } - } - - if ((i + j) < 0) { - PyErr_SetString(PyExc_OverflowError, "new string is too long"); - return NULL; - } - - /* Second pass: create output string and fill it */ - u = PyString_FromStringAndSize(NULL, i + j); - if (!u) - return NULL; - - j = 0; - q = PyString_AS_STRING(u); - - for (p = PyString_AS_STRING(self); p < e; p++) - if (*p == '\t') { - if (tabsize > 0) { - i = tabsize - (j % tabsize); - j += i; - while (i--) - *q++ = ' '; - } - } - else { - j++; - *q++ = *p; - if (*p == '\n' || *p == '\r') - j = 0; - } - - return u; -} - -Py_LOCAL_INLINE(PyObject *) -pad(PyStringObject *self, Py_ssize_t left, Py_ssize_t right, char fill) -{ - PyObject *u; - - if (left < 0) - left = 0; - if (right < 0) - right = 0; - - if (left == 0 && right == 0 && PyString_CheckExact(self)) { - Py_INCREF(self); - return (PyObject *)self; - } - - u = PyString_FromStringAndSize(NULL, - left + PyString_GET_SIZE(self) + right); - if (u) { - if (left) - memset(PyString_AS_STRING(u), fill, left); - Py_MEMCPY(PyString_AS_STRING(u) + left, - PyString_AS_STRING(self), - PyString_GET_SIZE(self)); - if (right) - memset(PyString_AS_STRING(u) + left + PyString_GET_SIZE(self), - fill, right); - } - - return u; -} - -PyDoc_STRVAR(ljust__doc__, -"S.ljust(width[, fillchar]) -> string\n" -"\n" -"Return S left justified in a string of length width. Padding is\n" -"done using the specified fill character (default is a space)."); - -static PyObject * -string_ljust(PyStringObject *self, PyObject *args) -{ - Py_ssize_t width; - char fillchar = ' '; - - if (!PyArg_ParseTuple(args, "n|c:ljust", &width, &fillchar)) - return NULL; - - if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*) self; - } - - return pad(self, 0, width - PyString_GET_SIZE(self), fillchar); -} - - -PyDoc_STRVAR(rjust__doc__, -"S.rjust(width[, fillchar]) -> string\n" -"\n" -"Return S right justified in a string of length width. Padding is\n" -"done using the specified fill character (default is a space)"); - -static PyObject * -string_rjust(PyStringObject *self, PyObject *args) -{ - Py_ssize_t width; - char fillchar = ' '; - - if (!PyArg_ParseTuple(args, "n|c:rjust", &width, &fillchar)) - return NULL; - - if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*) self; - } - - return pad(self, width - PyString_GET_SIZE(self), 0, fillchar); -} - - -PyDoc_STRVAR(center__doc__, -"S.center(width[, fillchar]) -> string\n" -"\n" -"Return S centered in a string of length width. Padding is\n" -"done using the specified fill character (default is a space)"); - -static PyObject * -string_center(PyStringObject *self, PyObject *args) -{ - Py_ssize_t marg, left; - Py_ssize_t width; - char fillchar = ' '; - - if (!PyArg_ParseTuple(args, "n|c:center", &width, &fillchar)) - return NULL; - - if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*) self; - } - - marg = width - PyString_GET_SIZE(self); - left = marg / 2 + (marg & width & 1); - - return pad(self, left, marg - left, fillchar); -} - -PyDoc_STRVAR(zfill__doc__, -"S.zfill(width) -> string\n" -"\n" -"Pad a numeric string S with zeros on the left, to fill a field\n" -"of the specified width. The string S is never truncated."); - -static PyObject * -string_zfill(PyStringObject *self, PyObject *args) -{ - Py_ssize_t fill; - PyObject *s; - char *p; - Py_ssize_t width; - - if (!PyArg_ParseTuple(args, "n:zfill", &width)) - return NULL; - - if (PyString_GET_SIZE(self) >= width) { - if (PyString_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*) self; - } - else - return PyString_FromStringAndSize( - PyString_AS_STRING(self), - PyString_GET_SIZE(self) - ); - } - - fill = width - PyString_GET_SIZE(self); - - s = pad(self, fill, 0, '0'); - - if (s == NULL) - return NULL; - - p = PyString_AS_STRING(s); - if (p[fill] == '+' || p[fill] == '-') { - /* move sign to beginning of string */ - p[0] = p[fill]; - p[fill] = '0'; - } - - return (PyObject*) s; -} - -PyDoc_STRVAR(isspace__doc__, -"S.isspace() -> bool\n\ -\n\ -Return True if all characters in S are whitespace\n\ -and there is at least one character in S, False otherwise."); - -static PyObject* -string_isspace(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - ISSPACE(*p)) - return PyBool_FromLong(1); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - for (; p < e; p++) { - if (!ISSPACE(*p)) - return PyBool_FromLong(0); - } - return PyBool_FromLong(1); -} - - -PyDoc_STRVAR(isalpha__doc__, -"S.isalpha() -> bool\n\ -\n\ -Return True if all characters in S are alphabetic\n\ -and there is at least one character in S, False otherwise."); - -static PyObject* -string_isalpha(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && - ISALPHA(*p)) - return PyBool_FromLong(1); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - for (; p < e; p++) { - if (!ISALPHA(*p)) - return PyBool_FromLong(0); - } - return PyBool_FromLong(1); -} - - -PyDoc_STRVAR(isalnum__doc__, -"S.isalnum() -> bool\n\ -\n\ -Return True if all characters in S are alphanumeric\n\ -and there is at least one character in S, False otherwise."); - -static PyObject* -string_isalnum(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && ISALNUM(*p)) - return PyBool_FromLong(1); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - for (; p < e; p++) { - if (!ISALNUM(*p)) - return PyBool_FromLong(0); - } - return PyBool_FromLong(1); -} - - -PyDoc_STRVAR(isdigit__doc__, -"S.isdigit() -> bool\n\ -\n\ -Return True if all characters in S are digits\n\ -and there is at least one character in S, False otherwise."); - -static PyObject* -string_isdigit(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1 && ISDIGIT(*p)) - return PyBool_FromLong(1); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - for (; p < e; p++) { - if (!ISDIGIT(*p)) - return PyBool_FromLong(0); - } - return PyBool_FromLong(1); -} - - -PyDoc_STRVAR(islower__doc__, -"S.islower() -> bool\n\ -\n\ -Return True if all cased characters in S are lowercase and there is\n\ -at least one cased character in S, False otherwise."); - -static PyObject* -string_islower(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - int cased; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(ISLOWER(*p)); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - cased = 0; - for (; p < e; p++) { - if (ISUPPER(*p)) - return PyBool_FromLong(0); - else if (!cased && ISLOWER(*p)) - cased = 1; - } - return PyBool_FromLong(cased); -} - - -PyDoc_STRVAR(isupper__doc__, -"S.isupper() -> bool\n\ -\n\ -Return True if all cased characters in S are uppercase and there is\n\ -at least one cased character in S, False otherwise."); - -static PyObject* -string_isupper(PyStringObject *self) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - int cased; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(ISUPPER(*p)); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - cased = 0; - for (; p < e; p++) { - if (ISLOWER(*p)) - return PyBool_FromLong(0); - else if (!cased && ISUPPER(*p)) - cased = 1; - } - return PyBool_FromLong(cased); -} - - -PyDoc_STRVAR(istitle__doc__, -"S.istitle() -> bool\n\ -\n\ -Return True if S is a titlecased string and there is at least one\n\ -character in S, i.e. uppercase characters may only follow uncased\n\ -characters and lowercase characters only cased ones. Return False\n\ -otherwise."); - -static PyObject* -string_istitle(PyStringObject *self, PyObject *uncased) -{ - register const unsigned char *p - = (unsigned char *) PyString_AS_STRING(self); - register const unsigned char *e; - int cased, previous_is_cased; - - /* Shortcut for single character strings */ - if (PyString_GET_SIZE(self) == 1) - return PyBool_FromLong(ISUPPER(*p)); - - /* Special case for empty strings */ - if (PyString_GET_SIZE(self) == 0) - return PyBool_FromLong(0); - - e = p + PyString_GET_SIZE(self); - cased = 0; - previous_is_cased = 0; - for (; p < e; p++) { - register const unsigned char ch = *p; - - if (ISUPPER(ch)) { - if (previous_is_cased) - return PyBool_FromLong(0); - previous_is_cased = 1; - cased = 1; - } - else if (ISLOWER(ch)) { - if (!previous_is_cased) - return PyBool_FromLong(0); - previous_is_cased = 1; - cased = 1; - } - else - previous_is_cased = 0; - } - return PyBool_FromLong(cased); -} - - -PyDoc_STRVAR(splitlines__doc__, -"S.splitlines([keepends]) -> list of strings\n\ -\n\ -Return a list of the lines in S, breaking at line boundaries.\n\ -Line breaks are not included in the resulting list unless keepends\n\ -is given and true."); - -static PyObject* -string_splitlines(PyStringObject *self, PyObject *args) -{ - register Py_ssize_t i; - register Py_ssize_t j; - Py_ssize_t len; - int keepends = 0; - PyObject *list; - PyObject *str; - char *data; - - if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends)) - return NULL; - - data = PyString_AS_STRING(self); - len = PyString_GET_SIZE(self); - - /* This does not use the preallocated list because splitlines is - usually run with hundreds of newlines. The overhead of - switching between PyList_SET_ITEM and append causes about a - 2-3% slowdown for that common case. A smarter implementation - could move the if check out, so the SET_ITEMs are done first - and the appends only done when the prealloc buffer is full. - That's too much work for little gain.*/ - - list = PyList_New(0); - if (!list) - goto onError; - - for (i = j = 0; i < len; ) { - Py_ssize_t eol; - - /* Find a line and append it */ - while (i < len && data[i] != '\n' && data[i] != '\r') - i++; - - /* Skip the line break reading CRLF as one line break */ - eol = i; - if (i < len) { - if (data[i] == '\r' && i + 1 < len && - data[i+1] == '\n') - i += 2; - else - i++; - if (keepends) - eol = i; - } - SPLIT_APPEND(data, j, eol); - j = i; - } - if (j < len) { - SPLIT_APPEND(data, j, len); - } - - return list; - - onError: - Py_XDECREF(list); - return NULL; -} - -#undef SPLIT_APPEND -#undef SPLIT_ADD -#undef MAX_PREALLOC -#undef PREALLOC_SIZE - static PyObject * string_getnewargs(PyStringObject *v) { @@ -3934,17 +2978,24 @@ {"join", (PyCFunction)string_join, METH_O, join__doc__}, {"split", (PyCFunction)string_split, METH_VARARGS, split__doc__}, {"rsplit", (PyCFunction)string_rsplit, METH_VARARGS, rsplit__doc__}, - {"lower", (PyCFunction)string_lower, METH_NOARGS, lower__doc__}, - {"upper", (PyCFunction)string_upper, METH_NOARGS, upper__doc__}, - {"islower", (PyCFunction)string_islower, METH_NOARGS, islower__doc__}, - {"isupper", (PyCFunction)string_isupper, METH_NOARGS, isupper__doc__}, - {"isspace", (PyCFunction)string_isspace, METH_NOARGS, isspace__doc__}, - {"isdigit", (PyCFunction)string_isdigit, METH_NOARGS, isdigit__doc__}, - {"istitle", (PyCFunction)string_istitle, METH_NOARGS, istitle__doc__}, - {"isalpha", (PyCFunction)string_isalpha, METH_NOARGS, isalpha__doc__}, - {"isalnum", (PyCFunction)string_isalnum, METH_NOARGS, isalnum__doc__}, - {"capitalize", (PyCFunction)string_capitalize, METH_NOARGS, - capitalize__doc__}, + {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, + {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, + {"islower", (PyCFunction)stringlib_islower, METH_NOARGS, + _Py_islower__doc__}, + {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS, + _Py_isupper__doc__}, + {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS, + _Py_isspace__doc__}, + {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS, + _Py_isdigit__doc__}, + {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS, + _Py_istitle__doc__}, + {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS, + _Py_isalpha__doc__}, + {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS, + _Py_isalnum__doc__}, + {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, + _Py_capitalize__doc__}, {"count", (PyCFunction)string_count, METH_VARARGS, count__doc__}, {"endswith", (PyCFunction)string_endswith, METH_VARARGS, endswith__doc__}, @@ -3961,20 +3012,20 @@ {"startswith", (PyCFunction)string_startswith, METH_VARARGS, startswith__doc__}, {"strip", (PyCFunction)string_strip, METH_VARARGS, strip__doc__}, - {"swapcase", (PyCFunction)string_swapcase, METH_NOARGS, - swapcase__doc__}, + {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, + _Py_swapcase__doc__}, {"translate", (PyCFunction)string_translate, METH_VARARGS, translate__doc__}, - {"title", (PyCFunction)string_title, METH_NOARGS, title__doc__}, - {"ljust", (PyCFunction)string_ljust, METH_VARARGS, ljust__doc__}, - {"rjust", (PyCFunction)string_rjust, METH_VARARGS, rjust__doc__}, - {"center", (PyCFunction)string_center, METH_VARARGS, center__doc__}, - {"zfill", (PyCFunction)string_zfill, METH_VARARGS, zfill__doc__}, + {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, + {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, ljust__doc__}, + {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, rjust__doc__}, + {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, + {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, zfill__doc__}, {"encode", (PyCFunction)string_encode, METH_VARARGS, encode__doc__}, {"decode", (PyCFunction)string_decode, METH_VARARGS, decode__doc__}, - {"expandtabs", (PyCFunction)string_expandtabs, METH_VARARGS, + {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, expandtabs__doc__}, - {"splitlines", (PyCFunction)string_splitlines, METH_VARARGS, + {"splitlines", (PyCFunction)stringlib_splitlines, METH_VARARGS, splitlines__doc__}, {"__getnewargs__", (PyCFunction)string_getnewargs, METH_NOARGS}, {NULL, NULL} /* sentinel */ From python-3000-checkins at python.org Tue Oct 16 20:13:23 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 16 Oct 2007 20:13:23 +0200 (CEST) Subject: [Python-3000-checkins] r58495 - in python/branches/py3k: Include/stringobject.h Lib/ConfigParser.py Lib/UserString.py Lib/_abcoll.py Lib/binhex.py Lib/cProfile.py Lib/cmd.py Lib/cookielib.py Lib/copy_reg.py Lib/decimal.py Lib/distutils/ccompiler.py Lib/distutils/cmd.py Lib/distutils/command/build_clib.py Lib/distutils/command/build_ext.py Lib/distutils/command/build_py.py Lib/distutils/command/config.py Lib/distutils/command/install.py Lib/distutils/command/install_data.py Lib/distutils/dir_util.py Lib/distutils/dist.py Lib/distutils/extension.py Lib/distutils/fancy_getopt.py Lib/distutils/filelist.py Lib/distutils/unixccompiler.py Lib/doctest.py Lib/email/feedparser.py Lib/email/generator.py Lib/email/iterators.py Lib/fileinput.py Lib/inspect.py Lib/io.py Lib/lib-tk/Tkinter.py Lib/locale.py Lib/logging/__init__.py Lib/msilib/__init__.py Lib/optparse.py Lib/os.py Lib/pickle.py Lib/pickletools.py Lib/pkgutil.py Lib/pstats.py Lib/shlex.py Lib/smtplib.py Lib/sre_compile.py Lib/subprocess.py Lib/tarfile.py Lib/test/test___future__.py Lib/test/test_abc.py Lib/test/test_ast.py Lib/test/test_codeccallbacks.py Lib/test/test_codecs.py Lib/test/test_collections.py Lib/test/test_grp.py Lib/test/test_isinstance.py Lib/test/test_posixpath.py Lib/test/test_pwd.py Lib/test/test_re.py Lib/test/test_support.py Lib/test/test_sys.py Lib/test/test_tempfile.py Lib/test/test_textwrap.py Lib/test/test_unittest.py Lib/timeit.py Lib/unittest.py Lib/urllib2.py Lib/uu.py Lib/warnings.py Lib/wave.py Lib/webbrowser.py Lib/xml/dom/pulldom.py Lib/xml/sax/saxutils.py Lib/xmlrpclib.py Lib/zipfile.py Modules/_csv.c Objects/stringobject.c Objects/unicodeobject.c Python/bltinmodule.c Message-ID: <20071016181323.82AE21E4036@bag.python.org> Author: guido.van.rossum Date: Tue Oct 16 20:12:55 2007 New Revision: 58495 Modified: python/branches/py3k/Include/stringobject.h python/branches/py3k/Lib/ConfigParser.py python/branches/py3k/Lib/UserString.py python/branches/py3k/Lib/_abcoll.py python/branches/py3k/Lib/binhex.py python/branches/py3k/Lib/cProfile.py python/branches/py3k/Lib/cmd.py python/branches/py3k/Lib/cookielib.py python/branches/py3k/Lib/copy_reg.py python/branches/py3k/Lib/decimal.py python/branches/py3k/Lib/distutils/ccompiler.py python/branches/py3k/Lib/distutils/cmd.py python/branches/py3k/Lib/distutils/command/build_clib.py python/branches/py3k/Lib/distutils/command/build_ext.py python/branches/py3k/Lib/distutils/command/build_py.py python/branches/py3k/Lib/distutils/command/config.py python/branches/py3k/Lib/distutils/command/install.py python/branches/py3k/Lib/distutils/command/install_data.py python/branches/py3k/Lib/distutils/dir_util.py python/branches/py3k/Lib/distutils/dist.py python/branches/py3k/Lib/distutils/extension.py python/branches/py3k/Lib/distutils/fancy_getopt.py python/branches/py3k/Lib/distutils/filelist.py python/branches/py3k/Lib/distutils/unixccompiler.py python/branches/py3k/Lib/doctest.py python/branches/py3k/Lib/email/feedparser.py python/branches/py3k/Lib/email/generator.py python/branches/py3k/Lib/email/iterators.py python/branches/py3k/Lib/fileinput.py python/branches/py3k/Lib/inspect.py python/branches/py3k/Lib/io.py python/branches/py3k/Lib/lib-tk/Tkinter.py python/branches/py3k/Lib/locale.py python/branches/py3k/Lib/logging/__init__.py python/branches/py3k/Lib/msilib/__init__.py python/branches/py3k/Lib/optparse.py python/branches/py3k/Lib/os.py python/branches/py3k/Lib/pickle.py python/branches/py3k/Lib/pickletools.py python/branches/py3k/Lib/pkgutil.py python/branches/py3k/Lib/pstats.py python/branches/py3k/Lib/shlex.py python/branches/py3k/Lib/smtplib.py python/branches/py3k/Lib/sre_compile.py python/branches/py3k/Lib/subprocess.py python/branches/py3k/Lib/tarfile.py python/branches/py3k/Lib/test/test___future__.py python/branches/py3k/Lib/test/test_abc.py python/branches/py3k/Lib/test/test_ast.py python/branches/py3k/Lib/test/test_codeccallbacks.py python/branches/py3k/Lib/test/test_codecs.py python/branches/py3k/Lib/test/test_collections.py python/branches/py3k/Lib/test/test_grp.py python/branches/py3k/Lib/test/test_isinstance.py python/branches/py3k/Lib/test/test_posixpath.py python/branches/py3k/Lib/test/test_pwd.py python/branches/py3k/Lib/test/test_re.py python/branches/py3k/Lib/test/test_support.py python/branches/py3k/Lib/test/test_sys.py python/branches/py3k/Lib/test/test_tempfile.py python/branches/py3k/Lib/test/test_textwrap.py python/branches/py3k/Lib/test/test_unittest.py python/branches/py3k/Lib/timeit.py python/branches/py3k/Lib/unittest.py python/branches/py3k/Lib/urllib2.py python/branches/py3k/Lib/uu.py python/branches/py3k/Lib/warnings.py python/branches/py3k/Lib/wave.py python/branches/py3k/Lib/webbrowser.py python/branches/py3k/Lib/xml/dom/pulldom.py python/branches/py3k/Lib/xml/sax/saxutils.py python/branches/py3k/Lib/xmlrpclib.py python/branches/py3k/Lib/zipfile.py python/branches/py3k/Modules/_csv.c python/branches/py3k/Objects/stringobject.c python/branches/py3k/Objects/unicodeobject.c python/branches/py3k/Python/bltinmodule.c Log: Patch# 1258 by Christian Heimes: kill basestring. I like this because it makes the code shorter! :-) Modified: python/branches/py3k/Include/stringobject.h ============================================================================== --- python/branches/py3k/Include/stringobject.h (original) +++ python/branches/py3k/Include/stringobject.h Tue Oct 16 20:12:55 2007 @@ -48,7 +48,6 @@ */ } PyStringObject; -PyAPI_DATA(PyTypeObject) PyBaseString_Type; PyAPI_DATA(PyTypeObject) PyString_Type; #define PyString_Check(op) \ Modified: python/branches/py3k/Lib/ConfigParser.py ============================================================================== --- python/branches/py3k/Lib/ConfigParser.py (original) +++ python/branches/py3k/Lib/ConfigParser.py Tue Oct 16 20:12:55 2007 @@ -271,7 +271,7 @@ Return list of successfully read files. """ - if isinstance(filenames, basestring): + if isinstance(filenames, str): filenames = [filenames] read_ok = [] for filename in filenames: @@ -652,7 +652,7 @@ def set(self, section, option, value): """Set an option. Extend ConfigParser.set: check for string values.""" - if not isinstance(value, basestring): + if not isinstance(value, str): raise TypeError("option values must be strings") # check for bad percent signs: # first, replace all "good" interpolations Modified: python/branches/py3k/Lib/UserString.py ============================================================================== --- python/branches/py3k/Lib/UserString.py (original) +++ python/branches/py3k/Lib/UserString.py Tue Oct 16 20:12:55 2007 @@ -11,7 +11,7 @@ class UserString: def __init__(self, seq): - if isinstance(seq, basestring): + if isinstance(seq, str): self.data = seq elif isinstance(seq, UserString): self.data = seq.data[:] @@ -66,12 +66,12 @@ def __add__(self, other): if isinstance(other, UserString): return self.__class__(self.data + other.data) - elif isinstance(other, basestring): + elif isinstance(other, str): return self.__class__(self.data + other) else: return self.__class__(self.data + str(other)) def __radd__(self, other): - if isinstance(other, basestring): + if isinstance(other, str): return self.__class__(other + self.data) else: return self.__class__(str(other) + self.data) @@ -184,7 +184,7 @@ if isinstance(index, slice): if isinstance(sub, UserString): sub = sub.data - elif not isinstance(sub, basestring): + elif not isinstance(sub, str): sub = str(sub) start, stop, step = index.indices(len(self.data)) if step == -1: @@ -221,7 +221,7 @@ def __iadd__(self, other): if isinstance(other, UserString): self.data += other.data - elif isinstance(other, basestring): + elif isinstance(other, str): self.data += other else: self.data += str(other) Modified: python/branches/py3k/Lib/_abcoll.py ============================================================================== --- python/branches/py3k/Lib/_abcoll.py (original) +++ python/branches/py3k/Lib/_abcoll.py Tue Oct 16 20:12:55 2007 @@ -490,7 +490,8 @@ return sum(1 for v in self if v == value) Sequence.register(tuple) -Sequence.register(basestring) +Sequence.register(str) +Sequence.register(str8) Sequence.register(memoryview) Modified: python/branches/py3k/Lib/binhex.py ============================================================================== --- python/branches/py3k/Lib/binhex.py (original) +++ python/branches/py3k/Lib/binhex.py Tue Oct 16 20:12:55 2007 @@ -169,7 +169,7 @@ class BinHex: def __init__(self, name_finfo_dlen_rlen, ofp): name, finfo, dlen, rlen = name_finfo_dlen_rlen - if isinstance(ofp, basestring): + if isinstance(ofp, str): ofname = ofp ofp = io.open(ofname, 'wb') if os.name == 'mac': @@ -371,7 +371,7 @@ class HexBin: def __init__(self, ifp): - if isinstance(ifp, basestring): + if isinstance(ifp, str): ifp = io.open(ifp, 'rb') # # Find initial colon. Modified: python/branches/py3k/Lib/cProfile.py ============================================================================== --- python/branches/py3k/Lib/cProfile.py (original) +++ python/branches/py3k/Lib/cProfile.py Tue Oct 16 20:12:55 2007 @@ -153,7 +153,7 @@ # ____________________________________________________________ def label(code): - if isinstance(code, basestring): + if isinstance(code, str): return ('~', 0, code) # built-in functions ('~' sorts at the end) else: return (code.co_filename, code.co_firstlineno, code.co_name) Modified: python/branches/py3k/Lib/cmd.py ============================================================================== --- python/branches/py3k/Lib/cmd.py (original) +++ python/branches/py3k/Lib/cmd.py Tue Oct 16 20:12:55 2007 @@ -356,7 +356,7 @@ return nonstrings = [i for i in range(len(list)) - if not isinstance(list[i], basestring)] + if not isinstance(list[i], str)] if nonstrings: raise TypeError("list[i] not a string for i in %s" % ", ".join(map(str, nonstrings))) Modified: python/branches/py3k/Lib/cookielib.py ============================================================================== --- python/branches/py3k/Lib/cookielib.py (original) +++ python/branches/py3k/Lib/cookielib.py Tue Oct 16 20:12:55 2007 @@ -367,7 +367,7 @@ [[('Basic', None), ('realm', '"foobar"')]] """ - assert not isinstance(header_values, basestring) + assert not isinstance(header_values, str) result = [] for text in header_values: orig_text = text Modified: python/branches/py3k/Lib/copy_reg.py ============================================================================== --- python/branches/py3k/Lib/copy_reg.py (original) +++ python/branches/py3k/Lib/copy_reg.py Tue Oct 16 20:12:55 2007 @@ -114,7 +114,7 @@ if "__slots__" in c.__dict__: slots = c.__dict__['__slots__'] # if class has a single slot, it can be given as a string - if isinstance(slots, basestring): + if isinstance(slots, str): slots = (slots,) for name in slots: # special descriptors Modified: python/branches/py3k/Lib/decimal.py ============================================================================== --- python/branches/py3k/Lib/decimal.py (original) +++ python/branches/py3k/Lib/decimal.py Tue Oct 16 20:12:55 2007 @@ -588,7 +588,7 @@ # From a string # REs insist on real strings, so we can too. - if isinstance(value, basestring): + if isinstance(value, str): if _isinfinity(value): self._exp = 'F' self._int = (0,) Modified: python/branches/py3k/Lib/distutils/ccompiler.py ============================================================================== --- python/branches/py3k/Lib/distutils/ccompiler.py (original) +++ python/branches/py3k/Lib/distutils/ccompiler.py Tue Oct 16 20:12:55 2007 @@ -154,7 +154,7 @@ self.set_executable(key, value) def set_executable(self, key, value): - if isinstance(value, basestring): + if isinstance(value, str): setattr(self, key, split_quoted(value)) else: setattr(self, key, value) @@ -175,8 +175,8 @@ for defn in definitions: if not (isinstance(defn, tuple) and (len(defn) in (1, 2) and - (isinstance (defn[1], basestring) or defn[1] is None)) and - isinstance (defn[0], basestring)): + (isinstance (defn[1], str) or defn[1] is None)) and + isinstance (defn[0], str)): raise TypeError(("invalid macro definition '%s': " % defn) + \ "must be tuple (string,), (string, string), or " + \ "(string, None)") @@ -318,7 +318,7 @@ """ if outdir is None: outdir = self.output_dir - elif not isinstance(outdir, basestring): + elif not isinstance(outdir, str): raise TypeError("'output_dir' must be a string or None") if macros is None: @@ -415,7 +415,7 @@ """ if output_dir is None: output_dir = self.output_dir - elif not isinstance(output_dir, basestring): + elif not isinstance(output_dir, str): raise TypeError("'output_dir' must be a string or None") if macros is None: @@ -494,7 +494,7 @@ if output_dir is None: output_dir = self.output_dir - elif not isinstance(output_dir, basestring): + elif not isinstance(output_dir, str): raise TypeError("'output_dir' must be a string or None") return (objects, output_dir) Modified: python/branches/py3k/Lib/distutils/cmd.py ============================================================================== --- python/branches/py3k/Lib/distutils/cmd.py (original) +++ python/branches/py3k/Lib/distutils/cmd.py Tue Oct 16 20:12:55 2007 @@ -213,7 +213,7 @@ if val is None: setattr(self, option, default) return default - elif not isinstance(val, basestring): + elif not isinstance(val, str): raise DistutilsOptionError("'%s' must be a %s (got `%s`)" % (option, what, val)) return val @@ -233,11 +233,11 @@ val = getattr(self, option) if val is None: return - elif isinstance(val, basestring): + elif isinstance(val, str): setattr(self, option, re.split(r',\s*|\s+', val)) else: if isinstance(val, list): - ok = all(isinstance(v, basestring) for v in val) + ok = all(isinstance(v, str) for v in val) else: ok = False if not ok: @@ -390,7 +390,7 @@ # Allow 'infiles' to be a single string - if isinstance(infiles, basestring): + if isinstance(infiles, str): infiles = (infiles,) elif not isinstance(infiles, (list, tuple)): raise TypeError( Modified: python/branches/py3k/Lib/distutils/command/build_clib.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/build_clib.py (original) +++ python/branches/py3k/Lib/distutils/command/build_clib.py Tue Oct 16 20:12:55 2007 @@ -86,7 +86,7 @@ if self.include_dirs is None: self.include_dirs = self.distribution.include_dirs or [] - if isinstance(self.include_dirs, basestring): + if isinstance(self.include_dirs, str): self.include_dirs = self.include_dirs.split(os.pathsep) # XXX same as for build_ext -- what about 'self.define' and @@ -134,7 +134,7 @@ raise DistutilsSetupError( "each element of 'libraries' must a 2-tuple") - if isinstance(lib[0], basestring): + if isinstance(lib[0], str): raise DistutilsSetupError( "first element of each tuple in 'libraries' " "must be a string (the library name)") Modified: python/branches/py3k/Lib/distutils/command/build_ext.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/build_ext.py (original) +++ python/branches/py3k/Lib/distutils/command/build_ext.py Tue Oct 16 20:12:55 2007 @@ -133,7 +133,7 @@ plat_py_include = sysconfig.get_python_inc(plat_specific=1) if self.include_dirs is None: self.include_dirs = self.distribution.include_dirs or [] - if isinstance(self.include_dirs, basestring): + if isinstance(self.include_dirs, str): self.include_dirs = self.include_dirs.split(os.pathsep) # Put the Python "system" include dir at the end, so that @@ -142,7 +142,7 @@ if plat_py_include != py_include: self.include_dirs.append(plat_py_include) - if isinstance(self.libraries, basestring): + if isinstance(self.libraries, str): self.libraries = [self.libraries] # Life is easier if we're not forever checking for None, so @@ -151,12 +151,12 @@ self.libraries = [] if self.library_dirs is None: self.library_dirs = [] - elif isinstance(self.library_dirs, basestring): + elif isinstance(self.library_dirs, str): self.library_dirs = self.library_dirs.split(os.pathsep) if self.rpath is None: self.rpath = [] - elif isinstance(self.rpath, basestring): + elif isinstance(self.rpath, str): self.rpath = self.rpath.split(os.pathsep) # for extensions under windows use different directories @@ -309,7 +309,7 @@ "each element of 'ext_modules' option must be an " "Extension instance or 2-tuple") - if not (isinstance(ext_name, basestring) and + if not (isinstance(ext_name, str) and extension_name_re.match(ext_name)): raise DistutilsSetupError( "first element of each tuple in 'ext_modules' " Modified: python/branches/py3k/Lib/distutils/command/build_py.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/build_py.py (original) +++ python/branches/py3k/Lib/distutils/command/build_py.py Tue Oct 16 20:12:55 2007 @@ -325,7 +325,7 @@ return outputs def build_module(self, module, module_file, package): - if isinstance(package, basestring): + if isinstance(package, str): package = package.split('.') elif not isinstance(package, (list, tuple)): raise TypeError( Modified: python/branches/py3k/Lib/distutils/command/config.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/config.py (original) +++ python/branches/py3k/Lib/distutils/command/config.py Tue Oct 16 20:12:55 2007 @@ -69,17 +69,17 @@ def finalize_options(self): if self.include_dirs is None: self.include_dirs = self.distribution.include_dirs or [] - elif isinstance(self.include_dirs, basestring): + elif isinstance(self.include_dirs, str): self.include_dirs = self.include_dirs.split(os.pathsep) if self.libraries is None: self.libraries = [] - elif isinstance(self.libraries, basestring): + elif isinstance(self.libraries, str): self.libraries = [self.libraries] if self.library_dirs is None: self.library_dirs = [] - elif isinstance(self.library_dirs, basestring): + elif isinstance(self.library_dirs, str): self.library_dirs = self.library_dirs.split(os.pathsep) def run(self): @@ -204,7 +204,7 @@ self._check_compiler() (src, out) = self._preprocess(body, headers, include_dirs, lang) - if isinstance(pattern, basestring): + if isinstance(pattern, str): pattern = re.compile(pattern) file = open(out) Modified: python/branches/py3k/Lib/distutils/command/install.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/install.py (original) +++ python/branches/py3k/Lib/distutils/command/install.py Tue Oct 16 20:12:55 2007 @@ -449,7 +449,7 @@ self.extra_path = self.distribution.extra_path if self.extra_path is not None: - if isinstance(self.extra_path, basestring): + if isinstance(self.extra_path, str): self.extra_path = self.extra_path.split(',') if len(self.extra_path) == 1: Modified: python/branches/py3k/Lib/distutils/command/install_data.py ============================================================================== --- python/branches/py3k/Lib/distutils/command/install_data.py (original) +++ python/branches/py3k/Lib/distutils/command/install_data.py Tue Oct 16 20:12:55 2007 @@ -45,7 +45,7 @@ def run(self): self.mkpath(self.install_dir) for f in self.data_files: - if isinstance(f, basestring): + if isinstance(f, str): # it's a simple file, so copy it f = convert_path(f) if self.warn_dir: Modified: python/branches/py3k/Lib/distutils/dir_util.py ============================================================================== --- python/branches/py3k/Lib/distutils/dir_util.py (original) +++ python/branches/py3k/Lib/distutils/dir_util.py Tue Oct 16 20:12:55 2007 @@ -28,7 +28,7 @@ global _path_created # Detect a common bug -- name is None - if not isinstance(name, basestring): + if not isinstance(name, str): raise DistutilsInternalError( "mkpath: 'name' must be a string (got %r)" % (name,)) Modified: python/branches/py3k/Lib/distutils/dist.py ============================================================================== --- python/branches/py3k/Lib/distutils/dist.py (original) +++ python/branches/py3k/Lib/distutils/dist.py Tue Oct 16 20:12:55 2007 @@ -580,13 +580,13 @@ keywords = self.metadata.keywords if keywords is not None: - if isinstance(keywords, basestring): + if isinstance(keywords, str): keywordlist = keywords.split(',') self.metadata.keywords = [x.strip() for x in keywordlist] platforms = self.metadata.platforms if platforms is not None: - if isinstance(platforms, basestring): + if isinstance(platforms, str): platformlist = platforms.split(',') self.metadata.platforms = [x.strip() for x in platformlist] @@ -874,7 +874,7 @@ neg_opt = {} try: - is_string = isinstance(value, basestring) + is_string = isinstance(value, str) if option in neg_opt and is_string: setattr(command_obj, neg_opt[option], not strtobool(value)) elif option in bool_opts and is_string: Modified: python/branches/py3k/Lib/distutils/extension.py ============================================================================== --- python/branches/py3k/Lib/distutils/extension.py (original) +++ python/branches/py3k/Lib/distutils/extension.py Tue Oct 16 20:12:55 2007 @@ -102,9 +102,9 @@ language=None, **kw # To catch unknown keywords ): - assert isinstance(name, basestring), "'name' must be a string" + assert isinstance(name, str), "'name' must be a string" assert (isinstance(sources, list) and - all(isinstance(v, basestring) for v in sources)), \ + all(isinstance(v, str) for v in sources)), \ "'sources' must be a list of strings" self.name = name Modified: python/branches/py3k/Lib/distutils/fancy_getopt.py ============================================================================== --- python/branches/py3k/Lib/distutils/fancy_getopt.py (original) +++ python/branches/py3k/Lib/distutils/fancy_getopt.py Tue Oct 16 20:12:55 2007 @@ -154,12 +154,12 @@ raise ValueError("invalid option tuple: %r" % (option,)) # Type- and value-check the option names - if not isinstance(long, basestring) or len(long) < 2: + if not isinstance(long, str) or len(long) < 2: raise DistutilsGetoptError(("invalid long option '%s': " "must be a string of length >= 2") % long) if (not ((short is None) or - (isinstance(short, basestring) and len(short) == 1))): + (isinstance(short, str) and len(short) == 1))): raise DistutilsGetoptError("invalid short option '%s': " "must a single character or None" % short) Modified: python/branches/py3k/Lib/distutils/filelist.py ============================================================================== --- python/branches/py3k/Lib/distutils/filelist.py (original) +++ python/branches/py3k/Lib/distutils/filelist.py Tue Oct 16 20:12:55 2007 @@ -301,7 +301,7 @@ or just returned as-is (assumes it's a regex object). """ if is_regex: - if isinstance(pattern, basestring): + if isinstance(pattern, str): return re.compile(pattern) else: return pattern Modified: python/branches/py3k/Lib/distutils/unixccompiler.py ============================================================================== --- python/branches/py3k/Lib/distutils/unixccompiler.py (original) +++ python/branches/py3k/Lib/distutils/unixccompiler.py Tue Oct 16 20:12:55 2007 @@ -211,7 +211,7 @@ lib_opts = gen_lib_options(self, library_dirs, runtime_library_dirs, libraries) - if not isinstance(output_dir, (basestring, type(None))): + if not isinstance(output_dir, (str, type(None))): raise TypeError("'output_dir' must be a string or None") if output_dir is not None: output_filename = os.path.join(output_dir, output_filename) Modified: python/branches/py3k/Lib/doctest.py ============================================================================== --- python/branches/py3k/Lib/doctest.py (original) +++ python/branches/py3k/Lib/doctest.py Tue Oct 16 20:12:55 2007 @@ -443,7 +443,7 @@ Create a new DocTest containing the given examples. The DocTest's globals are initialized with a copy of `globs`. """ - assert not isinstance(examples, basestring), \ + assert not isinstance(examples, str), \ "DocTest no longer accepts str; use DocTestParser instead" self.examples = examples self.docstring = docstring @@ -875,13 +875,13 @@ # Look for tests in a module's __test__ dictionary. if inspect.ismodule(obj) and self._recurse: for valname, val in getattr(obj, '__test__', {}).items(): - if not isinstance(valname, basestring): + if not isinstance(valname, str): raise ValueError("DocTestFinder.find: __test__ keys " "must be strings: %r" % (type(valname),)) if not (inspect.isfunction(val) or inspect.isclass(val) or inspect.ismethod(val) or inspect.ismodule(val) or - isinstance(val, basestring)): + isinstance(val, str)): raise ValueError("DocTestFinder.find: __test__ values " "must be strings, functions, methods, " "classes, or modules: %r" % @@ -914,7 +914,7 @@ """ # Extract the object's docstring. If it doesn't have one, # then return None (no test for this object). - if isinstance(obj, basestring): + if isinstance(obj, str): docstring = obj else: try: @@ -922,7 +922,7 @@ docstring = '' else: docstring = obj.__doc__ - if not isinstance(docstring, basestring): + if not isinstance(docstring, str): docstring = str(docstring) except (TypeError, AttributeError): docstring = '' Modified: python/branches/py3k/Lib/email/feedparser.py ============================================================================== --- python/branches/py3k/Lib/email/feedparser.py (original) +++ python/branches/py3k/Lib/email/feedparser.py Tue Oct 16 20:12:55 2007 @@ -365,7 +365,7 @@ self._last.epilogue = epilogue[:-end] else: payload = self._last.get_payload() - if isinstance(payload, basestring): + if isinstance(payload, str): mo = NLCRE_eol.search(payload) if mo: payload = payload[:-len(mo.group(0))] Modified: python/branches/py3k/Lib/email/generator.py ============================================================================== --- python/branches/py3k/Lib/email/generator.py (original) +++ python/branches/py3k/Lib/email/generator.py Tue Oct 16 20:12:55 2007 @@ -151,7 +151,7 @@ payload = msg.get_payload() if payload is None: return - if not isinstance(payload, basestring): + if not isinstance(payload, str): raise TypeError('string payload expected: %s' % type(payload)) if self._mangle_from_: payload = fcre.sub('>From ', payload) @@ -168,7 +168,7 @@ subparts = msg.get_payload() if subparts is None: subparts = [] - elif isinstance(subparts, basestring): + elif isinstance(subparts, str): # e.g. a non-strict parse of a message with no starting boundary. self._fp.write(subparts) return @@ -288,7 +288,7 @@ for part in msg.walk(): maintype = part.get_content_maintype() if maintype == 'text': - print(part.get_payload(decode=True), file=self) + print(part.get_payload(decode=False), file=self) elif maintype == 'multipart': # Just skip this pass Modified: python/branches/py3k/Lib/email/iterators.py ============================================================================== --- python/branches/py3k/Lib/email/iterators.py (original) +++ python/branches/py3k/Lib/email/iterators.py Tue Oct 16 20:12:55 2007 @@ -39,7 +39,7 @@ """ for subpart in msg.walk(): payload = subpart.get_payload(decode=decode) - if isinstance(payload, basestring): + if isinstance(payload, str): for line in StringIO(payload): yield line Modified: python/branches/py3k/Lib/fileinput.py ============================================================================== --- python/branches/py3k/Lib/fileinput.py (original) +++ python/branches/py3k/Lib/fileinput.py Tue Oct 16 20:12:55 2007 @@ -196,7 +196,7 @@ def __init__(self, files=None, inplace=0, backup="", bufsize=0, mode="r", openhook=None): - if isinstance(files, basestring): + if isinstance(files, str): files = (files,) else: if files is None: Modified: python/branches/py3k/Lib/inspect.py ============================================================================== --- python/branches/py3k/Lib/inspect.py (original) +++ python/branches/py3k/Lib/inspect.py Tue Oct 16 20:12:55 2007 @@ -313,7 +313,7 @@ doc = object.__doc__ except AttributeError: return None - if not isinstance(doc, basestring): + if not isinstance(doc, str): return None try: lines = doc.expandtabs().split('\n') Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Tue Oct 16 20:12:55 2007 @@ -103,13 +103,13 @@ binary stream, a buffered binary stream, or a buffered text stream, open for reading and/or writing. """ - if not isinstance(file, (basestring, int)): + if not isinstance(file, (str, int)): raise TypeError("invalid file: %r" % file) - if not isinstance(mode, basestring): + if not isinstance(mode, str): raise TypeError("invalid mode: %r" % mode) if buffering is not None and not isinstance(buffering, int): raise TypeError("invalid buffering: %r" % buffering) - if encoding is not None and not isinstance(encoding, basestring): + if encoding is not None and not isinstance(encoding, str): raise TypeError("invalid encoding: %r" % encoding) modes = set(mode) if modes - set("arwb+tU") or len(mode) > len(modes): @@ -1092,7 +1092,7 @@ def write(self, s: str): if self.closed: raise ValueError("write to closed file") - if not isinstance(s, basestring): + if not isinstance(s, str): raise TypeError("can't write %s to text stream" % s.__class__.__name__) haslf = "\n" in s @@ -1394,7 +1394,7 @@ encoding=encoding, newline=newline) if initial_value: - if not isinstance(initial_value, basestring): + if not isinstance(initial_value, str): initial_value = str(initial_value) self.write(initial_value) self.seek(0) Modified: python/branches/py3k/Lib/lib-tk/Tkinter.py ============================================================================== --- python/branches/py3k/Lib/lib-tk/Tkinter.py (original) +++ python/branches/py3k/Lib/lib-tk/Tkinter.py Tue Oct 16 20:12:55 2007 @@ -255,7 +255,7 @@ def get(self): """Return value of variable as string.""" value = self._tk.globalgetvar(self._name) - if isinstance(value, basestring): + if isinstance(value, str): return value return str(value) Modified: python/branches/py3k/Lib/locale.py ============================================================================== --- python/branches/py3k/Lib/locale.py (original) +++ python/branches/py3k/Lib/locale.py Tue Oct 16 20:12:55 2007 @@ -12,6 +12,7 @@ """ import sys, encodings, encodings.aliases +from __builtin__ import str as _builtin_str # Try importing the _locale module. # @@ -472,7 +473,7 @@ category may be given as one of the LC_* values. """ - if locale and not isinstance(locale, basestring): + if locale and not isinstance(locale, _builtin_str): # convert to string locale = normalize(_build_localename(locale)) return _setlocale(category, locale) Modified: python/branches/py3k/Lib/logging/__init__.py ============================================================================== --- python/branches/py3k/Lib/logging/__init__.py (original) +++ python/branches/py3k/Lib/logging/__init__.py Tue Oct 16 20:12:55 2007 @@ -281,7 +281,7 @@ msg = str(self.msg) else: msg = self.msg - if not isinstance(msg, basestring): + if not isinstance(msg, str): try: msg = str(self.msg) except UnicodeError: Modified: python/branches/py3k/Lib/msilib/__init__.py ============================================================================== --- python/branches/py3k/Lib/msilib/__init__.py (original) +++ python/branches/py3k/Lib/msilib/__init__.py Tue Oct 16 20:12:55 2007 @@ -101,7 +101,7 @@ field = value[i] if isinstance(field, int): r.SetInteger(i+1,field) - elif isinstance(field, basestring): + elif isinstance(field, str): r.SetString(i+1,field) elif field is None: pass Modified: python/branches/py3k/Lib/optparse.py ============================================================================== --- python/branches/py3k/Lib/optparse.py (original) +++ python/branches/py3k/Lib/optparse.py Tue Oct 16 20:12:55 2007 @@ -815,9 +815,6 @@ SUPPRESS_HELP = "SUPPRESS"+"HELP" SUPPRESS_USAGE = "SUPPRESS"+"USAGE" -def isbasestring(x): - return isinstance(x, basestring) - class Values: def __init__(self, defaults=None): @@ -994,7 +991,7 @@ """add_option(Option) add_option(opt_str, ..., kwarg=val, ...) """ - if isbasestring(args[0]): + if isinstance(args[0], str): option = self.option_class(*args, **kwargs) elif len(args) == 1 and not kwargs: option = args[0] @@ -1294,7 +1291,7 @@ defaults = self.defaults.copy() for option in self._get_all_options(): default = defaults.get(option.dest) - if isbasestring(default): + if isinstance(default, str): opt_str = option.get_opt_string() defaults[option.dest] = option.check_value(opt_str, default) @@ -1305,7 +1302,7 @@ def add_option_group(self, *args, **kwargs): # XXX lots of overlap with OptionContainer.add_option() - if isbasestring(args[0]): + if isinstance(args[0], str): group = OptionGroup(self, *args, **kwargs) elif len(args) == 1 and not kwargs: group = args[0] Modified: python/branches/py3k/Lib/os.py ============================================================================== --- python/branches/py3k/Lib/os.py (original) +++ python/branches/py3k/Lib/os.py Tue Oct 16 20:12:55 2007 @@ -636,7 +636,7 @@ # Supply os.popen() def popen(cmd, mode="r", buffering=None): - if not isinstance(cmd, basestring): + if not isinstance(cmd, str): raise TypeError("invalid cmd type (%s, expected string)" % type(cmd)) if mode not in ("r", "w"): raise ValueError("invalid mode %r" % mode) Modified: python/branches/py3k/Lib/pickle.py ============================================================================== --- python/branches/py3k/Lib/pickle.py (original) +++ python/branches/py3k/Lib/pickle.py Tue Oct 16 20:12:55 2007 @@ -307,7 +307,7 @@ (t.__name__, obj)) # Check for string returned by reduce(), meaning "save as global" - if isinstance(rv, basestring): + if isinstance(rv, str): self.save_global(obj, rv) return Modified: python/branches/py3k/Lib/pickletools.py ============================================================================== --- python/branches/py3k/Lib/pickletools.py (original) +++ python/branches/py3k/Lib/pickletools.py Tue Oct 16 20:12:55 2007 @@ -701,7 +701,7 @@ ) def __init__(self, name, obtype, doc): - assert isinstance(name, basestring) + assert isinstance(name, str) self.name = name assert isinstance(obtype, type) or isinstance(obtype, tuple) @@ -710,7 +710,7 @@ assert isinstance(contained, type) self.obtype = obtype - assert isinstance(doc, basestring) + assert isinstance(doc, str) self.doc = doc def __repr__(self): @@ -846,10 +846,10 @@ def __init__(self, name, code, arg, stack_before, stack_after, proto, doc): - assert isinstance(name, basestring) + assert isinstance(name, str) self.name = name - assert isinstance(code, basestring) + assert isinstance(code, str) assert len(code) == 1 self.code = code @@ -869,7 +869,7 @@ assert isinstance(proto, int) and 0 <= proto <= 2 self.proto = proto - assert isinstance(doc, basestring) + assert isinstance(doc, str) self.doc = doc I = OpcodeInfo Modified: python/branches/py3k/Lib/pkgutil.py ============================================================================== --- python/branches/py3k/Lib/pkgutil.py (original) +++ python/branches/py3k/Lib/pkgutil.py Tue Oct 16 20:12:55 2007 @@ -516,7 +516,7 @@ path = path[:] # Start with a copy of the existing path for dir in sys.path: - if not isinstance(dir, basestring) or not os.path.isdir(dir): + if not isinstance(dir, str) or not os.path.isdir(dir): continue subdir = os.path.join(dir, pname) # XXX This may still add duplicate entries to path on Modified: python/branches/py3k/Lib/pstats.py ============================================================================== --- python/branches/py3k/Lib/pstats.py (original) +++ python/branches/py3k/Lib/pstats.py Tue Oct 16 20:12:55 2007 @@ -116,7 +116,7 @@ def load_stats(self, arg): if not arg: self.stats = {} - elif isinstance(arg, basestring): + elif isinstance(arg, str): f = open(arg, 'rb') self.stats = marshal.load(f) f.close() Modified: python/branches/py3k/Lib/shlex.py ============================================================================== --- python/branches/py3k/Lib/shlex.py (original) +++ python/branches/py3k/Lib/shlex.py Tue Oct 16 20:12:55 2007 @@ -18,7 +18,7 @@ class shlex: "A lexical analyzer class for simple shell-like syntaxes." def __init__(self, instream=None, infile=None, posix=False): - if isinstance(instream, basestring): + if isinstance(instream, str): instream = StringIO(instream) if instream is not None: self.instream = instream @@ -61,7 +61,7 @@ def push_source(self, newstream, newfile=None): "Push an input source onto the lexer's input source stack." - if isinstance(newstream, basestring): + if isinstance(newstream, str): newstream = StringIO(newstream) self.filestack.appendleft((self.infile, self.instream, self.lineno)) self.infile = newfile @@ -247,7 +247,7 @@ if newfile[0] == '"': newfile = newfile[1:-1] # This implements cpp-like semantics for relative-path inclusion. - if isinstance(self.infile, basestring) and not os.path.isabs(newfile): + if isinstance(self.infile, str) and not os.path.isabs(newfile): newfile = os.path.join(os.path.dirname(self.infile), newfile) return (newfile, open(newfile, "r")) Modified: python/branches/py3k/Lib/smtplib.py ============================================================================== --- python/branches/py3k/Lib/smtplib.py (original) +++ python/branches/py3k/Lib/smtplib.py Tue Oct 16 20:12:55 2007 @@ -668,7 +668,7 @@ self.rset() raise SMTPSenderRefused(code, resp, from_addr) senderrs={} - if isinstance(to_addrs, basestring): + if isinstance(to_addrs, str): to_addrs = [to_addrs] for each in to_addrs: (code,resp)=self.rcpt(each, rcpt_options) Modified: python/branches/py3k/Lib/sre_compile.py ============================================================================== --- python/branches/py3k/Lib/sre_compile.py (original) +++ python/branches/py3k/Lib/sre_compile.py Tue Oct 16 20:12:55 2007 @@ -472,7 +472,7 @@ code[skip] = len(code) - skip def isstring(obj): - return isinstance(obj, basestring) + return isinstance(obj, str) def _code(p, flags): Modified: python/branches/py3k/Lib/subprocess.py ============================================================================== --- python/branches/py3k/Lib/subprocess.py (original) +++ python/branches/py3k/Lib/subprocess.py Tue Oct 16 20:12:55 2007 @@ -698,7 +698,7 @@ errread, errwrite): """Execute program (MS Windows version)""" - if not isinstance(args, basestring): + if not isinstance(args, str): args = list2cmdline(args) # Process startup details @@ -913,7 +913,7 @@ errread, errwrite): """Execute program (POSIX version)""" - if isinstance(args, basestring): + if isinstance(args, str): args = [args] else: args = list(args) Modified: python/branches/py3k/Lib/tarfile.py ============================================================================== --- python/branches/py3k/Lib/tarfile.py (original) +++ python/branches/py3k/Lib/tarfile.py Tue Oct 16 20:12:55 2007 @@ -2041,7 +2041,7 @@ """ self._check("r") - if isinstance(member, basestring): + if isinstance(member, str): tarinfo = self.getmember(member) else: tarinfo = member @@ -2077,7 +2077,7 @@ """ self._check("r") - if isinstance(member, basestring): + if isinstance(member, str): tarinfo = self.getmember(member) else: tarinfo = member Modified: python/branches/py3k/Lib/test/test___future__.py ============================================================================== --- python/branches/py3k/Lib/test/test___future__.py (original) +++ python/branches/py3k/Lib/test/test___future__.py Tue Oct 16 20:12:55 2007 @@ -39,7 +39,7 @@ a(isinstance(major, int), "%s major isn't int" % name) a(isinstance(minor, int), "%s minor isn't int" % name) a(isinstance(micro, int), "%s micro isn't int" % name) - a(isinstance(level, basestring), + a(isinstance(level, str), "%s level isn't string" % name) a(level in GOOD_SERIALS, "%s level string has unknown value" % name) Modified: python/branches/py3k/Lib/test/test_abc.py ============================================================================== --- python/branches/py3k/Lib/test/test_abc.py (original) +++ python/branches/py3k/Lib/test/test_abc.py Tue Oct 16 20:12:55 2007 @@ -81,9 +81,11 @@ self.assertEqual(issubclass(int, A), True) class B(A): pass - B.register(basestring) + B.register(str) + class C(str): pass self.assertEqual(isinstance("", A), True) self.assertEqual(issubclass(str, A), True) + self.assertEqual(issubclass(C, A), True) def test_registration_edge_cases(self): class A(metaclass=abc.ABCMeta): Modified: python/branches/py3k/Lib/test/test_ast.py ============================================================================== --- python/branches/py3k/Lib/test/test_ast.py (original) +++ python/branches/py3k/Lib/test/test_ast.py Tue Oct 16 20:12:55 2007 @@ -2,7 +2,7 @@ import _ast def to_tuple(t): - if t is None or isinstance(t, (basestring, int, int, complex)): + if t is None or isinstance(t, (str, int, complex)): return t elif isinstance(t, list): return [to_tuple(e) for e in t] Modified: python/branches/py3k/Lib/test/test_codeccallbacks.py ============================================================================== --- python/branches/py3k/Lib/test/test_codeccallbacks.py (original) +++ python/branches/py3k/Lib/test/test_codeccallbacks.py Tue Oct 16 20:12:55 2007 @@ -140,17 +140,17 @@ sin += chr(sys.maxunicode) sout = b"a\\xac\\u1234\\u20ac\\u8000" if sys.maxunicode > 0xffff: - sout += bytes("\\U%08x" % sys.maxunicode) + sout += bytes("\\U%08x" % sys.maxunicode, "ascii") self.assertEqual(sin.encode("ascii", "backslashreplace"), sout) sout = b"a\xac\\u1234\\u20ac\\u8000" if sys.maxunicode > 0xffff: - sout += bytes("\\U%08x" % sys.maxunicode) + sout += bytes("\\U%08x" % sys.maxunicode, "ascii") self.assertEqual(sin.encode("latin-1", "backslashreplace"), sout) sout = b"a\xac\\u1234\xa4\\u8000" if sys.maxunicode > 0xffff: - sout += bytes("\\U%08x" % sys.maxunicode) + sout += bytes("\\U%08x" % sys.maxunicode, "ascii") self.assertEqual(sin.encode("iso-8859-15", "backslashreplace"), sout) def test_decoderelaxedutf8(self): Modified: python/branches/py3k/Lib/test/test_codecs.py ============================================================================== --- python/branches/py3k/Lib/test/test_codecs.py (original) +++ python/branches/py3k/Lib/test/test_codecs.py Tue Oct 16 20:12:55 2007 @@ -803,8 +803,9 @@ codecs.register_error("UnicodeInternalTest", codecs.ignore_errors) decoder = codecs.getdecoder("unicode_internal") ab = "ab".encode("unicode_internal") - ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:])), - "UnicodeInternalTest") + ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]), + "ascii"), + "UnicodeInternalTest") self.assertEquals(("ab", 12), ignored) # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html Modified: python/branches/py3k/Lib/test/test_collections.py ============================================================================== --- python/branches/py3k/Lib/test/test_collections.py (original) +++ python/branches/py3k/Lib/test/test_collections.py Tue Oct 16 20:12:55 2007 @@ -235,7 +235,7 @@ for sample in [tuple, list, bytes, str]: self.failUnless(isinstance(sample(), Sequence)) self.failUnless(issubclass(sample, Sequence)) - self.failUnless(issubclass(basestring, Sequence)) + self.failUnless(issubclass(str, Sequence)) def test_MutableSequence(self): for sample in [tuple, str]: @@ -244,7 +244,7 @@ for sample in [list, bytes]: self.failUnless(isinstance(sample(), MutableSequence)) self.failUnless(issubclass(sample, MutableSequence)) - self.failIf(issubclass(basestring, MutableSequence)) + self.failIf(issubclass(str, MutableSequence)) def test_main(verbose=None): Modified: python/branches/py3k/Lib/test/test_grp.py ============================================================================== --- python/branches/py3k/Lib/test/test_grp.py (original) +++ python/branches/py3k/Lib/test/test_grp.py Tue Oct 16 20:12:55 2007 @@ -11,9 +11,9 @@ # attributes promised by the docs self.assertEqual(len(value), 4) self.assertEqual(value[0], value.gr_name) - self.assert_(isinstance(value.gr_name, basestring)) + self.assert_(isinstance(value.gr_name, str)) self.assertEqual(value[1], value.gr_passwd) - self.assert_(isinstance(value.gr_passwd, basestring)) + self.assert_(isinstance(value.gr_passwd, str)) self.assertEqual(value[2], value.gr_gid) self.assert_(isinstance(value.gr_gid, int)) self.assertEqual(value[3], value.gr_mem) Modified: python/branches/py3k/Lib/test/test_isinstance.py ============================================================================== --- python/branches/py3k/Lib/test/test_isinstance.py (original) +++ python/branches/py3k/Lib/test/test_isinstance.py Tue Oct 16 20:12:55 2007 @@ -242,7 +242,7 @@ self.assertEqual(True, issubclass(NewSuper, (NewChild, (NewSuper,)))) self.assertEqual(True, issubclass(int, (int, (float, int)))) - self.assertEqual(True, issubclass(str, (str, (Child, NewChild, basestring)))) + self.assertEqual(True, issubclass(str, (str, (Child, NewChild, str)))) def test_subclass_recursion_limit(self): # make sure that issubclass raises RuntimeError before the C stack is Modified: python/branches/py3k/Lib/test/test_posixpath.py ============================================================================== --- python/branches/py3k/Lib/test/test_posixpath.py (original) +++ python/branches/py3k/Lib/test/test_posixpath.py Tue Oct 16 20:12:55 2007 @@ -335,15 +335,15 @@ except ImportError: pass else: - self.assert_(isinstance(posixpath.expanduser("~/"), basestring)) + self.assert_(isinstance(posixpath.expanduser("~/"), str)) # if home directory == root directory, this test makes no sense if posixpath.expanduser("~") != '/': self.assertEqual( posixpath.expanduser("~") + "/", posixpath.expanduser("~/") ) - self.assert_(isinstance(posixpath.expanduser("~root/"), basestring)) - self.assert_(isinstance(posixpath.expanduser("~foo/"), basestring)) + self.assert_(isinstance(posixpath.expanduser("~root/"), str)) + self.assert_(isinstance(posixpath.expanduser("~foo/"), str)) self.assertRaises(TypeError, posixpath.expanduser) Modified: python/branches/py3k/Lib/test/test_pwd.py ============================================================================== --- python/branches/py3k/Lib/test/test_pwd.py (original) +++ python/branches/py3k/Lib/test/test_pwd.py Tue Oct 16 20:12:55 2007 @@ -13,19 +13,19 @@ for e in entries: self.assertEqual(len(e), 7) self.assertEqual(e[0], e.pw_name) - self.assert_(isinstance(e.pw_name, basestring)) + self.assert_(isinstance(e.pw_name, str)) self.assertEqual(e[1], e.pw_passwd) - self.assert_(isinstance(e.pw_passwd, basestring)) + self.assert_(isinstance(e.pw_passwd, str)) self.assertEqual(e[2], e.pw_uid) self.assert_(isinstance(e.pw_uid, int)) self.assertEqual(e[3], e.pw_gid) self.assert_(isinstance(e.pw_gid, int)) self.assertEqual(e[4], e.pw_gecos) - self.assert_(isinstance(e.pw_gecos, basestring)) + self.assert_(isinstance(e.pw_gecos, str)) self.assertEqual(e[5], e.pw_dir) - self.assert_(isinstance(e.pw_dir, basestring)) + self.assert_(isinstance(e.pw_dir, str)) self.assertEqual(e[6], e.pw_shell) - self.assert_(isinstance(e.pw_shell, basestring)) + self.assert_(isinstance(e.pw_shell, str)) # The following won't work, because of duplicate entries # for one uid Modified: python/branches/py3k/Lib/test/test_re.py ============================================================================== --- python/branches/py3k/Lib/test/test_re.py (original) +++ python/branches/py3k/Lib/test/test_re.py Tue Oct 16 20:12:55 2007 @@ -591,9 +591,10 @@ self.assertEqual([item.group(0) for item in iter], [":", "::", ":::"]) - def test_bug_926075(self): - self.assert_(re.compile('bug_926075') is not - re.compile(str8('bug_926075'))) + # XXX This needs to be restored for str vs. bytes. +## def test_bug_926075(self): +## self.assert_(re.compile('bug_926075') is not +## re.compile(str8('bug_926075'))) def test_bug_931848(self): pattern = eval('"[\u002E\u3002\uFF0E\uFF61]"') Modified: python/branches/py3k/Lib/test/test_support.py ============================================================================== --- python/branches/py3k/Lib/test/test_support.py (original) +++ python/branches/py3k/Lib/test/test_support.py Tue Oct 16 20:12:55 2007 @@ -529,7 +529,7 @@ valid_types = (unittest.TestSuite, unittest.TestCase) suite = unittest.TestSuite() for cls in classes: - if isinstance(cls, basestring): + if isinstance(cls, str): if cls in sys.modules: suite.addTest(unittest.findTestCases(sys.modules[cls])) else: Modified: python/branches/py3k/Lib/test/test_sys.py ============================================================================== --- python/branches/py3k/Lib/test/test_sys.py (original) +++ python/branches/py3k/Lib/test/test_sys.py Tue Oct 16 20:12:55 2007 @@ -126,7 +126,7 @@ def test_getdefaultencoding(self): self.assertRaises(TypeError, sys.getdefaultencoding, 42) # can't check more than the type, as the user might have changed it - self.assert_(isinstance(sys.getdefaultencoding(), basestring)) + self.assert_(isinstance(sys.getdefaultencoding(), str)) # testing sys.settrace() is done in test_trace.py # testing sys.setprofile() is done in test_profile.py @@ -275,15 +275,15 @@ self.assert_(isinstance(sys.argv, list)) self.assert_(sys.byteorder in ("little", "big")) self.assert_(isinstance(sys.builtin_module_names, tuple)) - self.assert_(isinstance(sys.copyright, basestring)) - self.assert_(isinstance(sys.exec_prefix, basestring)) - self.assert_(isinstance(sys.executable, basestring)) + self.assert_(isinstance(sys.copyright, str)) + self.assert_(isinstance(sys.exec_prefix, str)) + self.assert_(isinstance(sys.executable, str)) self.assert_(isinstance(sys.hexversion, int)) self.assert_(isinstance(sys.maxint, int)) self.assert_(isinstance(sys.maxunicode, int)) - self.assert_(isinstance(sys.platform, basestring)) - self.assert_(isinstance(sys.prefix, basestring)) - self.assert_(isinstance(sys.version, basestring)) + self.assert_(isinstance(sys.platform, str)) + self.assert_(isinstance(sys.prefix, str)) + self.assert_(isinstance(sys.version, str)) vi = sys.version_info self.assert_(isinstance(vi, tuple)) self.assertEqual(len(vi), 5) Modified: python/branches/py3k/Lib/test/test_tempfile.py ============================================================================== --- python/branches/py3k/Lib/test/test_tempfile.py (original) +++ python/branches/py3k/Lib/test/test_tempfile.py Tue Oct 16 20:12:55 2007 @@ -143,7 +143,7 @@ self.failIf(len(cand) == 0) for c in cand: - self.assert_(isinstance(c, basestring), + self.assert_(isinstance(c, str), "%s is not a string" % c) def test_wanted_dirs(self): @@ -328,7 +328,7 @@ # gettempprefix returns a nonempty prefix string p = tempfile.gettempprefix() - self.assert_(isinstance(p, basestring)) + self.assert_(isinstance(p, str)) self.assert_(len(p) > 0) def test_usable_template(self): @@ -463,7 +463,7 @@ extant[i] = self.do_create(pre="aa") finally: for i in extant: - if(isinstance(i, basestring)): + if(isinstance(i, str)): os.rmdir(i) def test_choose_directory(self): Modified: python/branches/py3k/Lib/test/test_textwrap.py ============================================================================== --- python/branches/py3k/Lib/test/test_textwrap.py (original) +++ python/branches/py3k/Lib/test/test_textwrap.py Tue Oct 16 20:12:55 2007 @@ -23,7 +23,7 @@ for i in range(len(textin)): result.append(" %d: %r" % (i, textin[i])) result = '\n'.join(result) - elif isinstance(textin, basestring): + elif isinstance(textin, str): result = " %s\n" % repr(textin) return result Modified: python/branches/py3k/Lib/test/test_unittest.py ============================================================================== --- python/branches/py3k/Lib/test/test_unittest.py (original) +++ python/branches/py3k/Lib/test/test_unittest.py Tue Oct 16 20:12:55 2007 @@ -1718,7 +1718,7 @@ def test_id(self): test = unittest.FunctionTestCase(lambda: None) - self.failUnless(isinstance(test.id(), basestring)) + self.failUnless(isinstance(test.id(), str)) # "Returns a one-line description of the test, or None if no description # has been provided. The default implementation of this method returns @@ -2239,7 +2239,7 @@ def runTest(self): pass - self.failUnless(isinstance(Foo().id(), basestring)) + self.failUnless(isinstance(Foo().id(), str)) # "Returns a one-line description of the test, or None if no description # has been provided. The default implementation of this method returns Modified: python/branches/py3k/Lib/timeit.py ============================================================================== --- python/branches/py3k/Lib/timeit.py (original) +++ python/branches/py3k/Lib/timeit.py Tue Oct 16 20:12:55 2007 @@ -121,9 +121,9 @@ """Constructor. See class doc string.""" self.timer = timer ns = {} - if isinstance(stmt, basestring): + if isinstance(stmt, str): stmt = reindent(stmt, 8) - if isinstance(setup, basestring): + if isinstance(setup, str): setup = reindent(setup, 4) src = template % {'stmt': stmt, 'setup': setup} elif hasattr(setup, '__call__'): @@ -137,7 +137,7 @@ self.inner = ns["inner"] elif hasattr(stmt, '__call__'): self.src = None - if isinstance(setup, basestring): + if isinstance(setup, str): _setup = setup def setup(): exec(_setup, globals(), ns) Modified: python/branches/py3k/Lib/unittest.py ============================================================================== --- python/branches/py3k/Lib/unittest.py (original) +++ python/branches/py3k/Lib/unittest.py Tue Oct 16 20:12:55 2007 @@ -418,7 +418,7 @@ self._tests.append(test) def addTests(self, tests): - if isinstance(tests, basestring): + if isinstance(tests, str): raise TypeError("tests must be an iterable of tests, not a string") for test in tests: self.addTest(test) Modified: python/branches/py3k/Lib/urllib2.py ============================================================================== --- python/branches/py3k/Lib/urllib2.py (original) +++ python/branches/py3k/Lib/urllib2.py Tue Oct 16 20:12:55 2007 @@ -359,7 +359,7 @@ def open(self, fullurl, data=None, timeout=None): # accept a URL or a Request object - if isinstance(fullurl, basestring): + if isinstance(fullurl, str): req = Request(fullurl, data) else: req = fullurl @@ -702,7 +702,7 @@ def add_password(self, realm, uri, user, passwd): # uri could be a single URI or a sequence - if isinstance(uri, basestring): + if isinstance(uri, str): uri = [uri] if not realm in self.passwd: self.passwd[realm] = {} Modified: python/branches/py3k/Lib/uu.py ============================================================================== --- python/branches/py3k/Lib/uu.py (original) +++ python/branches/py3k/Lib/uu.py Tue Oct 16 20:12:55 2007 @@ -46,7 +46,7 @@ # if in_file == '-': in_file = sys.stdin.buffer - elif isinstance(in_file, basestring): + elif isinstance(in_file, str): if name is None: name = os.path.basename(in_file) if mode is None: @@ -60,7 +60,7 @@ # if out_file == '-': out_file = sys.stdout.buffer - elif isinstance(out_file, basestring): + elif isinstance(out_file, str): out_file = open(out_file, 'wb') # # Set defaults for name and mode @@ -87,7 +87,7 @@ # if in_file == '-': in_file = sys.stdin.buffer - elif isinstance(in_file, basestring): + elif isinstance(in_file, str): in_file = open(in_file, 'rb') # # Read until a begin is encountered or we've exhausted the file @@ -118,7 +118,7 @@ opened = False if out_file == '-': out_file = sys.stdout.buffer - elif isinstance(out_file, basestring): + elif isinstance(out_file, str): fp = open(out_file, 'wb') try: os.path.chmod(out_file, mode) @@ -169,7 +169,7 @@ if options.decode: if options.text: - if isinstance(output, basestring): + if isinstance(output, str): output = open(output, 'wb') else: print(sys.argv[0], ': cannot do -t to stdout') @@ -177,7 +177,7 @@ decode(input, output) else: if options.text: - if isinstance(input, basestring): + if isinstance(input, str): input = open(input, 'rb') else: print(sys.argv[0], ': cannot do -t from stdin') Modified: python/branches/py3k/Lib/warnings.py ============================================================================== --- python/branches/py3k/Lib/warnings.py (original) +++ python/branches/py3k/Lib/warnings.py Tue Oct 16 20:12:55 2007 @@ -150,10 +150,10 @@ import re assert action in ("error", "ignore", "always", "default", "module", "once"), "invalid action: %r" % (action,) - assert isinstance(message, basestring), "message must be a string" + assert isinstance(message, str), "message must be a string" assert isinstance(category, type), "category must be a class" assert issubclass(category, Warning), "category must be a Warning subclass" - assert isinstance(module, basestring), "module must be a string" + assert isinstance(module, str), "module must be a string" assert isinstance(lineno, int) and lineno >= 0, \ "lineno must be an int >= 0" item = (action, re.compile(message, re.I), category, Modified: python/branches/py3k/Lib/wave.py ============================================================================== --- python/branches/py3k/Lib/wave.py (original) +++ python/branches/py3k/Lib/wave.py Tue Oct 16 20:12:55 2007 @@ -155,7 +155,7 @@ def __init__(self, f): self._i_opened_the_file = None - if isinstance(f, basestring): + if isinstance(f, str): f = __builtin__.open(f, 'rb') self._i_opened_the_file = f # else, assume it is an open file object already @@ -299,7 +299,7 @@ def __init__(self, f): self._i_opened_the_file = None - if isinstance(f, basestring): + if isinstance(f, str): f = __builtin__.open(f, 'wb') self._i_opened_the_file = f try: Modified: python/branches/py3k/Lib/webbrowser.py ============================================================================== --- python/branches/py3k/Lib/webbrowser.py (original) +++ python/branches/py3k/Lib/webbrowser.py Tue Oct 16 20:12:55 2007 @@ -159,7 +159,7 @@ and without remote functionality.""" def __init__(self, name): - if isinstance(name, basestring): + if isinstance(name, str): self.name = name self.args = ["%s"] else: Modified: python/branches/py3k/Lib/xml/dom/pulldom.py ============================================================================== --- python/branches/py3k/Lib/xml/dom/pulldom.py (original) +++ python/branches/py3k/Lib/xml/dom/pulldom.py Tue Oct 16 20:12:55 2007 @@ -325,7 +325,7 @@ def parse(stream_or_string, parser=None, bufsize=None): if bufsize is None: bufsize = default_bufsize - if isinstance(stream_or_string, basestring): + if isinstance(stream_or_string, str): stream = open(stream_or_string) else: stream = stream_or_string Modified: python/branches/py3k/Lib/xml/sax/saxutils.py ============================================================================== --- python/branches/py3k/Lib/xml/sax/saxutils.py (original) +++ python/branches/py3k/Lib/xml/sax/saxutils.py Tue Oct 16 20:12:55 2007 @@ -272,7 +272,7 @@ """This function takes an InputSource and an optional base URL and returns a fully resolved InputSource object ready for reading.""" - if isinstance(source, basestring): + if isinstance(source, str): source = xmlreader.InputSource(source) elif hasattr(source, "read"): f = source Modified: python/branches/py3k/Lib/xmlrpclib.py ============================================================================== --- python/branches/py3k/Lib/xmlrpclib.py (original) +++ python/branches/py3k/Lib/xmlrpclib.py Tue Oct 16 20:12:55 2007 @@ -294,7 +294,7 @@ """ def __init__(self, value=0): - if not isinstance(value, basestring): + if not isinstance(value, str): if datetime and isinstance(value, datetime.datetime): self.value = value.strftime("%Y%m%dT%H:%M:%S") return @@ -653,7 +653,7 @@ write("\n") for k, v in value.items(): write("\n") - if not isinstance(k, basestring): + if not isinstance(k, str): raise TypeError("dictionary key must be string") write("%s\n" % escape(k)) dump(v, write) @@ -1031,7 +1031,7 @@ # standard XML-RPC wrappings if methodname: # a method call - if not isinstance(methodname, basestring): + if not isinstance(methodname, str): methodname = methodname.encode(encoding) data = ( xmlheader, Modified: python/branches/py3k/Lib/zipfile.py ============================================================================== --- python/branches/py3k/Lib/zipfile.py (original) +++ python/branches/py3k/Lib/zipfile.py Tue Oct 16 20:12:55 2007 @@ -596,7 +596,7 @@ self.pwd = None # Check if we were passed a file-like object - if isinstance(file, basestring): + if isinstance(file, str): # No, it's a filename self._filePassed = 0 self.filename = file Modified: python/branches/py3k/Modules/_csv.c ============================================================================== --- python/branches/py3k/Modules/_csv.c (original) +++ python/branches/py3k/Modules/_csv.c Tue Oct 16 20:12:55 2007 @@ -62,7 +62,7 @@ /* end 2.2 compatibility macros */ #define IS_BASESTRING(o) \ - PyObject_TypeCheck(o, &PyBaseString_Type) + PyUnicode_Check(o) static PyObject *error_obj; /* CSV exception */ static PyObject *dialects; /* Dialect registry */ Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Tue Oct 16 20:12:55 2007 @@ -3073,14 +3073,6 @@ } static PyObject * -basestring_new(PyTypeObject *type, PyObject *args, PyObject *kwds) -{ - PyErr_SetString(PyExc_TypeError, - "The basestring type cannot be instantiated"); - return NULL; -} - -static PyObject * string_mod(PyObject *v, PyObject *w) { if (!PyString_Check(v)) { @@ -3090,9 +3082,6 @@ return PyString_Format(v, w); } -PyDoc_STRVAR(basestring_doc, -"Type basestring cannot be instantiated; it is the base for str8 and str."); - static PyNumberMethods string_as_number = { 0, /*nb_add*/ 0, /*nb_subtract*/ @@ -3100,49 +3089,6 @@ string_mod, /*nb_remainder*/ }; - -PyTypeObject PyBaseString_Type = { - PyVarObject_HEAD_INIT(&PyType_Type, 0) - "basestring", - 0, - 0, - 0, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - 0, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - basestring_doc, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - &PyBaseObject_Type, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - basestring_new, /* tp_new */ - 0, /* tp_free */ -}; - PyDoc_STRVAR(string_doc, "str(object) -> string\n\ \n\ @@ -3183,7 +3129,7 @@ string_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ - &PyBaseString_Type, /* tp_base */ + &PyBaseObject_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ @@ -3615,7 +3561,7 @@ argidx = -2; } if (Py_Type(args)->tp_as_mapping && !PyTuple_Check(args) && - !PyObject_TypeCheck(args, &PyBaseString_Type)) + !PyString_Check(args) && !PyUnicode_Check(args)) dict = args; while (--fmtcnt >= 0) { if (*fmt != '%') { Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Tue Oct 16 20:12:55 2007 @@ -8441,7 +8441,7 @@ argidx = -2; } if (Py_Type(args)->tp_as_mapping && !PyTuple_Check(args) && - !PyObject_TypeCheck(args, &PyBaseString_Type)) + !PyString_Check(args) && !PyUnicode_Check(args)) dict = args; while (--fmtcnt >= 0) { @@ -8935,7 +8935,7 @@ unicode_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ - &PyBaseString_Type, /* tp_base */ + &PyBaseObject_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ Modified: python/branches/py3k/Python/bltinmodule.c ============================================================================== --- python/branches/py3k/Python/bltinmodule.c (original) +++ python/branches/py3k/Python/bltinmodule.c Tue Oct 16 20:12:55 2007 @@ -1595,12 +1595,19 @@ } } else { /* reject string values for 'start' parameter */ - if (PyObject_TypeCheck(result, &PyBaseString_Type)) { + if (PyUnicode_Check(result)) { PyErr_SetString(PyExc_TypeError, "sum() can't sum strings [use ''.join(seq) instead]"); Py_DECREF(iter); return NULL; } + if (PyBytes_Check(result)) { + PyErr_SetString(PyExc_TypeError, + "sum() can't sum bytes [use b''.join(seq) instead]"); + Py_DECREF(iter); + return NULL; + } + Py_INCREF(result); } @@ -1788,7 +1795,6 @@ SETBUILTIN("NotImplemented", Py_NotImplemented); SETBUILTIN("False", Py_False); SETBUILTIN("True", Py_True); - SETBUILTIN("basestring", &PyBaseString_Type); SETBUILTIN("bool", &PyBool_Type); SETBUILTIN("memoryview", &PyMemoryView_Type); SETBUILTIN("bytes", &PyBytes_Type); From python-3000-checkins at python.org Thu Oct 18 18:55:12 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Thu, 18 Oct 2007 18:55:12 +0200 (CEST) Subject: [Python-3000-checkins] r58535 - python/branches/py3k/Lib/bsddb/dbtables.py Message-ID: <20071018165512.A26C41E402F@bag.python.org> Author: gregory.p.smith Date: Thu Oct 18 18:55:12 2007 New Revision: 58535 Modified: python/branches/py3k/Lib/bsddb/dbtables.py Log: Merge 58532, 58533, 58534: bsddb.dbtables bug fixes - don't allow null bytes in random rowid strings, pass txn using a keyword where possible. Modified: python/branches/py3k/Lib/bsddb/dbtables.py ============================================================================== --- python/branches/py3k/Lib/bsddb/dbtables.py (original) +++ python/branches/py3k/Lib/bsddb/dbtables.py Thu Oct 18 18:55:12 2007 @@ -20,7 +20,7 @@ import re import sys import copy -import xdrlib +import struct import random import pickle @@ -255,7 +255,7 @@ flags=DB_RMW)) tablelist.append(table) # delete 1st, in case we opened with DB_DUP - self.db.delete(_E(_table_names_key), txn) + self.db.delete(_E(_table_names_key), txn=txn) self.db.put(_E(_table_names_key), pickle.dumps(tablelist, 1), txn=txn) txn.commit() @@ -330,7 +330,7 @@ # store the table's new extended column list if newcolumnlist != oldcolumnlist : # delete the old one first since we opened with DB_DUP - self.db.delete(columnlist_key, txn) + self.db.delete(columnlist_key, txn=txn) self.db.put(columnlist_key, pickle.dumps(newcolumnlist, 1), txn=txn) @@ -364,10 +364,11 @@ # Generate a random 64-bit row ID string # (note: this code has <64 bits of randomness # but it's plenty for our database id needs!) - p = xdrlib.Packer() - p.pack_int(int(random.random()*2147483647)) - p.pack_int(int(random.random()*2147483647)) - newid = p.get_buffer() + # We must ensure that no null bytes are in the id value. + blist = [] + for x in range(_rowid_str_len): + blist.append(random.randint(1,255)) + newid = bytes(blist) # Guarantee uniqueness by adding this key to the database try: @@ -451,10 +452,10 @@ try: dataitem = self.db.get( _data_key(table, column, rowid), - txn) + txn=txn) self.db.delete( _data_key(table, column, rowid), - txn) + txn=txn) except DBNotFoundError: # XXXXXXX row key somehow didn't exist, assume no # error @@ -497,13 +498,13 @@ try: self.db.delete(_data_key(table, column, rowid.encode("latin-1")), - txn) + txn=txn) except DBNotFoundError: # XXXXXXX column may not exist, assume no error pass try: - self.db.delete(_rowid_key(table, rowid.encode("latin-1")), txn) + self.db.delete(_rowid_key(table, rowid.encode("latin-1")), txn=txn) except DBNotFoundError: # XXXXXXX row key somehow didn't exist, assume no error pass @@ -659,7 +660,7 @@ txn = self.env.txn_begin() # delete the column list - self.db.delete(_columns_key(table), txn) + self.db.delete(_columns_key(table), txn=txn) cur = self.db.cursor(txn) @@ -698,7 +699,7 @@ # hmm, it wasn't there, oh well, that's what we want. pass # delete 1st, incase we opened with DB_DUP - self.db.delete(_E(_table_names_key), txn) + self.db.delete(_E(_table_names_key), txn=txn) self.db.put(_E(_table_names_key), pickle.dumps(tablelist, 1), txn=txn) txn.commit() From python-3000-checkins at python.org Fri Oct 19 09:34:49 2007 From: python-3000-checkins at python.org (gregory.p.smith) Date: Fri, 19 Oct 2007 09:34:49 +0200 (CEST) Subject: [Python-3000-checkins] r58540 - python/branches/py3k/Lib/bsddb/test/test_1413192.py Message-ID: <20071019073449.380171E4021@bag.python.org> Author: gregory.p.smith Date: Fri Oct 19 09:34:48 2007 New Revision: 58540 Modified: python/branches/py3k/Lib/bsddb/test/test_1413192.py Log: Merge 58539: squelch the warning that this test is intended to raise. Modified: python/branches/py3k/Lib/bsddb/test/test_1413192.py ============================================================================== --- python/branches/py3k/Lib/bsddb/test/test_1413192.py (original) +++ python/branches/py3k/Lib/bsddb/test/test_1413192.py Fri Oct 19 09:34:48 2007 @@ -5,6 +5,7 @@ import shutil import tempfile +import warnings try: # For Pythons w/distutils and add-on pybsddb from bsddb3 import db @@ -32,8 +33,12 @@ del self.the_txn -context = Context() -del context +warnings.filterwarnings('ignore', 'DBTxn aborted in destructor') +try: + context = Context() + del context +finally: + warnings.resetwarnings() # try not to leave a turd try: From python-3000-checkins at python.org Fri Oct 19 23:48:41 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 19 Oct 2007 23:48:41 +0200 (CEST) Subject: [Python-3000-checkins] r58551 - python/branches/py3k/Python/codecs.c python/branches/py3k/Python/structmember.c Message-ID: <20071019214841.9CF031E4009@bag.python.org> Author: guido.van.rossum Date: Fri Oct 19 23:48:41 2007 New Revision: 58551 Modified: python/branches/py3k/Python/codecs.c python/branches/py3k/Python/structmember.c Log: This is the uncontroversial half of patch 1263 by Thomas Lee: changes to codecs.c and structmember.c to use PyUnicode instead of PyString. Modified: python/branches/py3k/Python/codecs.c ============================================================================== --- python/branches/py3k/Python/codecs.c (original) +++ python/branches/py3k/Python/codecs.c Fri Oct 19 23:48:41 2007 @@ -55,16 +55,15 @@ size_t len = strlen(string); char *p; PyObject *v; - + if (len > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "string is too large"); return NULL; } - - v = PyString_FromStringAndSize(NULL, len); - if (v == NULL) - return NULL; - p = PyString_AS_STRING(v); + + p = PyMem_Malloc(len + 1); + if (p == NULL) + return NULL; for (i = 0; i < len; i++) { register char ch = string[i]; if (ch == ' ') @@ -73,6 +72,11 @@ ch = tolower(Py_CHARMASK(ch)); p[i] = ch; } + p[i] = '\0'; + v = PyUnicode_FromString(p); + if (v == NULL) + return NULL; + PyMem_Free(p); return v; } @@ -112,7 +116,7 @@ v = normalizestring(encoding); if (v == NULL) goto onError; - PyString_InternInPlace(&v); + PyUnicode_InternInPlace(&v); /* First, try to lookup the name in the registry dictionary */ result = PyDict_GetItem(interp->codec_search_cache, v); @@ -193,7 +197,7 @@ if (errors) { PyObject *v; - v = PyString_FromString(errors); + v = PyUnicode_FromString(errors); if (v == NULL) { Py_DECREF(args); return NULL; Modified: python/branches/py3k/Python/structmember.c ============================================================================== --- python/branches/py3k/Python/structmember.c (original) +++ python/branches/py3k/Python/structmember.c Fri Oct 19 23:48:41 2007 @@ -51,13 +51,13 @@ v = Py_None; } else - v = PyString_FromString(*(char**)addr); + v = PyUnicode_FromString(*(char**)addr); break; case T_STRING_INPLACE: - v = PyString_FromString((char*)addr); + v = PyUnicode_FromString((char*)addr); break; case T_CHAR: - v = PyString_FromStringAndSize((char*)addr, 1); + v = PyUnicode_FromStringAndSize((char*)addr, 1); break; case T_OBJECT: v = *(PyObject **)addr; @@ -225,8 +225,8 @@ Py_XDECREF(oldv); break; case T_CHAR: - if (PyString_Check(v) && PyString_Size(v) == 1) { - *(char*)addr = PyString_AsString(v)[0]; + if (PyUnicode_Check(v) && PyUnicode_GetSize(v) == 1) { + *(char*)addr = PyUnicode_AsString(v)[0]; } else { PyErr_BadArgument(); From python-3000-checkins at python.org Sat Oct 20 00:06:24 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 20 Oct 2007 00:06:24 +0200 (CEST) Subject: [Python-3000-checkins] r58552 - in python/branches/py3k: Lib/dis.py Lib/encodings/__init__.py Lib/modulefinder.py Lib/sre_parse.py Lib/test/string_tests.py Lib/test/test_bytes.py Lib/test/test_set.py Lib/test/test_struct.py Objects/stringobject.c Message-ID: <20071019220624.BC41F1E4009@bag.python.org> Author: guido.van.rossum Date: Sat Oct 20 00:06:24 2007 New Revision: 58552 Modified: python/branches/py3k/Lib/dis.py python/branches/py3k/Lib/encodings/__init__.py python/branches/py3k/Lib/modulefinder.py python/branches/py3k/Lib/sre_parse.py python/branches/py3k/Lib/test/string_tests.py python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Lib/test/test_set.py python/branches/py3k/Lib/test/test_struct.py python/branches/py3k/Objects/stringobject.c Log: Patch 1280, by Alexandre Vassalotti. Make PyString's indexing and iteration return integers. (I changed a few of Alexandre's decisions -- GvR.) Modified: python/branches/py3k/Lib/dis.py ============================================================================== --- python/branches/py3k/Lib/dis.py (original) +++ python/branches/py3k/Lib/dis.py Sat Oct 20 00:06:24 2007 @@ -117,8 +117,7 @@ extended_arg = 0 free = None while i < n: - c = code[i] - op = ord(c) + op = code[i] if i in linestarts: if i > 0: print() @@ -134,7 +133,7 @@ print(opname[op].ljust(20), end=' ') i = i+1 if op >= HAVE_ARGUMENT: - oparg = ord(code[i]) + ord(code[i+1])*256 + extended_arg + oparg = code[i] + code[i+1]*256 + extended_arg extended_arg = 0 i = i+2 if op == EXTENDED_ARG: @@ -162,8 +161,7 @@ n = len(code) i = 0 while i < n: - c = code[i] - op = ord(c) + op = code[i] if i == lasti: print('-->', end=' ') else: print(' ', end=' ') if i in labels: print('>>', end=' ') @@ -172,7 +170,7 @@ print(opname[op].ljust(15), end=' ') i = i+1 if op >= HAVE_ARGUMENT: - oparg = ord(code[i]) + ord(code[i+1])*256 + oparg = code[i] + code[i+1]*256 i = i+2 print(repr(oparg).rjust(5), end=' ') if op in hasconst: @@ -208,11 +206,10 @@ n = len(code) i = 0 while i < n: - c = code[i] - op = ord(c) + op = code[i] i = i+1 if op >= HAVE_ARGUMENT: - oparg = ord(code[i]) + ord(code[i+1])*256 + oparg = code[i] + code[i+1]*256 i = i+2 label = -1 if op in hasjrel: @@ -230,8 +227,8 @@ Generate pairs (offset, lineno) as described in Python/compile.c. """ - byte_increments = [ord(c) for c in code.co_lnotab[0::2]] - line_increments = [ord(c) for c in code.co_lnotab[1::2]] + byte_increments = list(code.co_lnotab[0::2]) + line_increments = list(code.co_lnotab[1::2]) lastlineno = None lineno = code.co_firstlineno Modified: python/branches/py3k/Lib/encodings/__init__.py ============================================================================== --- python/branches/py3k/Lib/encodings/__init__.py (original) +++ python/branches/py3k/Lib/encodings/__init__.py Sat Oct 20 00:06:24 2007 @@ -52,6 +52,8 @@ non-ASCII characters, these must be Latin-1 compatible. """ + if isinstance(encoding, str8): + encoding = str(encoding, "ascii") chars = [] punct = False for c in encoding: Modified: python/branches/py3k/Lib/modulefinder.py ============================================================================== --- python/branches/py3k/Lib/modulefinder.py (original) +++ python/branches/py3k/Lib/modulefinder.py Sat Oct 20 00:06:24 2007 @@ -367,7 +367,7 @@ consts = co.co_consts LOAD_LOAD_AND_IMPORT = LOAD_CONST + LOAD_CONST + IMPORT_NAME while code: - c = code[0] + c = chr(code[0]) if c in STORE_OPS: oparg, = unpack('= len(self.string): self.next = None return - char = self.string[self.index] - if char[0] == "\\": + char = self.string[self.index:self.index+1] + # Special case for the str8, since indexing returns a integer + # XXX This is only needed for test_bug_926075 in test_re.py + if isinstance(self.string, str8): + char = chr(char) + if char == "\\": try: c = self.string[self.index + 1] except IndexError: raise error("bogus escape (end of line)") + if isinstance(self.string, str8): + char = chr(c) char = char + c self.index = self.index + len(char) self.next = char Modified: python/branches/py3k/Lib/test/string_tests.py ============================================================================== --- python/branches/py3k/Lib/test/string_tests.py (original) +++ python/branches/py3k/Lib/test/string_tests.py Sat Oct 20 00:06:24 2007 @@ -558,6 +558,10 @@ a = self.type2test('DNSSEC') b = self.type2test('') for c in a: + # Special case for the str8, since indexing returns a integer + # XXX Maybe it would be a good idea to seperate str8's tests... + if self.type2test == str8: + c = chr(c) b += c hash(b) self.assertEqual(hash(a), hash(b)) Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Sat Oct 20 00:06:24 2007 @@ -348,7 +348,7 @@ sample = str8("Hello world\n\x80\x81\xfe\xff") buf = memoryview(sample) b = bytes(buf) - self.assertEqual(b, bytes(map(ord, sample))) + self.assertEqual(b, bytes(sample)) def test_to_str(self): sample = "Hello world\n\x80\x81\xfe\xff" Modified: python/branches/py3k/Lib/test/test_set.py ============================================================================== --- python/branches/py3k/Lib/test/test_set.py (original) +++ python/branches/py3k/Lib/test/test_set.py Sat Oct 20 00:06:24 2007 @@ -72,7 +72,7 @@ self.assertEqual(type(u), self.thetype) self.assertRaises(PassThru, self.s.union, check_pass_thru()) self.assertRaises(TypeError, self.s.union, [[]]) - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: self.assertEqual(self.thetype('abcba').union(C('cdc')), set('abcd')) self.assertEqual(self.thetype('abcba').union(C('efgfe')), set('abcefg')) self.assertEqual(self.thetype('abcba').union(C('ccb')), set('abc')) @@ -96,7 +96,7 @@ self.assertEqual(self.s, self.thetype(self.word)) self.assertEqual(type(i), self.thetype) self.assertRaises(PassThru, self.s.intersection, check_pass_thru()) - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: self.assertEqual(self.thetype('abcba').intersection(C('cdc')), set('cc')) self.assertEqual(self.thetype('abcba').intersection(C('efgfe')), set('')) self.assertEqual(self.thetype('abcba').intersection(C('ccb')), set('bc')) @@ -121,7 +121,7 @@ self.assertEqual(type(i), self.thetype) self.assertRaises(PassThru, self.s.difference, check_pass_thru()) self.assertRaises(TypeError, self.s.difference, [[]]) - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: self.assertEqual(self.thetype('abcba').difference(C('cdc')), set('ab')) self.assertEqual(self.thetype('abcba').difference(C('efgfe')), set('abc')) self.assertEqual(self.thetype('abcba').difference(C('ccb')), set('a')) @@ -146,7 +146,7 @@ self.assertEqual(type(i), self.thetype) self.assertRaises(PassThru, self.s.symmetric_difference, check_pass_thru()) self.assertRaises(TypeError, self.s.symmetric_difference, [[]]) - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: self.assertEqual(self.thetype('abcba').symmetric_difference(C('cdc')), set('abd')) self.assertEqual(self.thetype('abcba').symmetric_difference(C('efgfe')), set('abcefg')) self.assertEqual(self.thetype('abcba').symmetric_difference(C('ccb')), set('a')) @@ -390,7 +390,7 @@ self.assertRaises(PassThru, self.s.update, check_pass_thru()) self.assertRaises(TypeError, self.s.update, [[]]) for p, q in (('cdc', 'abcd'), ('efgfe', 'abcefg'), ('ccb', 'abc'), ('ef', 'abcef')): - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: s = self.thetype('abcba') self.assertEqual(s.update(C(p)), None) self.assertEqual(s, set(q)) @@ -411,7 +411,7 @@ self.assertRaises(PassThru, self.s.intersection_update, check_pass_thru()) self.assertRaises(TypeError, self.s.intersection_update, [[]]) for p, q in (('cdc', 'c'), ('efgfe', ''), ('ccb', 'bc'), ('ef', '')): - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: s = self.thetype('abcba') self.assertEqual(s.intersection_update(C(p)), None) self.assertEqual(s, set(q)) @@ -436,7 +436,7 @@ self.assertRaises(TypeError, self.s.difference_update, [[]]) self.assertRaises(TypeError, self.s.symmetric_difference_update, [[]]) for p, q in (('cdc', 'ab'), ('efgfe', 'abc'), ('ccb', 'a'), ('ef', 'abc')): - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: s = self.thetype('abcba') self.assertEqual(s.difference_update(C(p)), None) self.assertEqual(s, set(q)) @@ -460,7 +460,7 @@ self.assertRaises(PassThru, self.s.symmetric_difference_update, check_pass_thru()) self.assertRaises(TypeError, self.s.symmetric_difference_update, [[]]) for p, q in (('cdc', 'abd'), ('efgfe', 'abcefg'), ('ccb', 'a'), ('ef', 'abcef')): - for C in set, frozenset, dict.fromkeys, str, str8, list, tuple: + for C in set, frozenset, dict.fromkeys, str, list, tuple: s = self.thetype('abcba') self.assertEqual(s.symmetric_difference_update(C(p)), None) self.assertEqual(s, set(q)) Modified: python/branches/py3k/Lib/test/test_struct.py ============================================================================== --- python/branches/py3k/Lib/test/test_struct.py (original) +++ python/branches/py3k/Lib/test/test_struct.py Sat Oct 20 00:06:24 2007 @@ -674,8 +674,8 @@ elif not prefix and verbose: print('size of bool in native format is %i' % (len(packed))) - for c in str8('\x01\x7f\xff\x0f\xf0'): - if struct.unpack('>t', c)[0] is not True: + for c in b'\x01\x7f\xff\x0f\xf0': + if struct.unpack('>t', bytes([c]))[0] is not True: raise TestFailed('%c did not unpack as True' % c) test_bool() Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Sat Oct 20 00:06:24 2007 @@ -986,28 +986,6 @@ return stringlib_contains_obj(str_obj, sub_obj); } -static PyObject * -string_item(PyStringObject *a, register Py_ssize_t i) -{ - char pchar; - PyObject *v; - if (i < 0 || i >= Py_Size(a)) { - PyErr_SetString(PyExc_IndexError, "string index out of range"); - return NULL; - } - pchar = a->ob_sval[i]; - v = (PyObject *)characters[pchar & UCHAR_MAX]; - if (v == NULL) - v = PyString_FromStringAndSize(&pchar, 1); - else { -#ifdef COUNT_ALLOCS - one_strings++; -#endif - Py_INCREF(v); - } - return v; -} - static PyObject* string_richcompare(PyStringObject *a, PyStringObject *b, int op) { @@ -1110,7 +1088,12 @@ return NULL; if (i < 0) i += PyString_GET_SIZE(self); - return string_item(self, i); + if (i < 0 || i >= PyString_GET_SIZE(self)) { + PyErr_SetString(PyExc_IndexError, + "string index out of range"); + return NULL; + } + return PyInt_FromLong((unsigned char)self->ob_sval[i]); } else if (PySlice_Check(item)) { Py_ssize_t start, stop, step, slicelength, cur, i; @@ -1173,7 +1156,7 @@ (lenfunc)string_length, /*sq_length*/ (binaryfunc)string_concat, /*sq_concat*/ (ssizeargfunc)string_repeat, /*sq_repeat*/ - (ssizeargfunc)string_item, /*sq_item*/ + 0, /*sq_item*/ 0, /*sq_slice*/ 0, /*sq_ass_item*/ 0, /*sq_ass_slice*/ @@ -4147,8 +4130,8 @@ assert(PyString_Check(seq)); if (it->it_index < PyString_GET_SIZE(seq)) { - item = PyString_FromStringAndSize( - PyString_AS_STRING(seq)+it->it_index, 1); + item = PyInt_FromLong( + (unsigned char)seq->ob_sval[it->it_index]); if (item != NULL) ++it->it_index; return item; From python-3000-checkins at python.org Sat Oct 20 01:16:50 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 20 Oct 2007 01:16:50 +0200 (CEST) Subject: [Python-3000-checkins] r58553 - in python/branches/py3k: Doc/c-api/concrete.rst Doc/data/refcounts.dat Include/fileobject.h Lib/io.py Lib/site.py Lib/test/test_imp.py Objects/fileobject.c Parser/tokenizer.c Parser/tokenizer.h Python/import.c Python/pythonrun.c Message-ID: <20071019231650.F3CEB1E4025@bag.python.org> Author: guido.van.rossum Date: Sat Oct 20 01:16:50 2007 New Revision: 58553 Modified: python/branches/py3k/Doc/c-api/concrete.rst python/branches/py3k/Doc/data/refcounts.dat python/branches/py3k/Include/fileobject.h python/branches/py3k/Lib/io.py python/branches/py3k/Lib/site.py python/branches/py3k/Lib/test/test_imp.py python/branches/py3k/Objects/fileobject.c python/branches/py3k/Parser/tokenizer.c python/branches/py3k/Parser/tokenizer.h python/branches/py3k/Python/import.c python/branches/py3k/Python/pythonrun.c Log: Patch 1267 by Christian Heimes. Move the initialization of sys.std{in,out,err} and __builtin__.open to C code. This solves the problem that "python -S" wouldn't work. Modified: python/branches/py3k/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k/Doc/c-api/concrete.rst (original) +++ python/branches/py3k/Doc/c-api/concrete.rst Sat Oct 20 01:16:50 2007 @@ -2425,6 +2425,12 @@ pointer, *fp*. The function *close* will be called when the file should be closed. Return *NULL* on failure. +.. cfunction:: PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE *), int buffering, char *encoding, char *newline) + + Create a new :ctype:`PyFileObject` from the already-open standard C file + pointer, *fp*. The functions works similar to *PyFile_FromFile* but takes + optional arguments for *buffering*, *encoding* and *newline*. Use -1 resp. + *NULL* for default values. .. cfunction:: FILE* PyFile_AsFile(PyObject *p) Modified: python/branches/py3k/Doc/data/refcounts.dat ============================================================================== --- python/branches/py3k/Doc/data/refcounts.dat (original) +++ python/branches/py3k/Doc/data/refcounts.dat Sat Oct 20 01:16:50 2007 @@ -338,6 +338,15 @@ PyFile_FromFile:char*:mode:: PyFile_FromFile:int(*:close):: +PyFile_FromFileEx:PyObject*::+1: +PyFile_FromFileEx:FILE*:fp:: +PyFile_FromFileEx:char*:name:: +PyFile_FromFileEx:char*:mode:: +PyFile_FromFileEx:int(*:close):: +PyFile_FromFileEx:int:buffering:: +PyFile_FromFileEx:char*:encoding:: +PyFile_FromFileEx:char*:newline:: + PyFile_FromString:PyObject*::+1: PyFile_FromString:char*:name:: PyFile_FromString:char*:mode:: Modified: python/branches/py3k/Include/fileobject.h ============================================================================== --- python/branches/py3k/Include/fileobject.h (original) +++ python/branches/py3k/Include/fileobject.h Sat Oct 20 01:16:50 2007 @@ -9,6 +9,9 @@ #define PY_STDIOTEXTMODE "b" PyAPI_FUNC(PyObject *) PyFile_FromFile(FILE *, char *, char *, int (*)(FILE*)); +PyAPI_FUNC(PyObject *) PyFile_FromFileEx(FILE *, char *, char *, + int (*)(FILE *), int, char *, + char *); PyAPI_FUNC(PyObject *) PyFile_GetLine(PyObject *, int); PyAPI_FUNC(int) PyFile_WriteObject(PyObject *, PyObject *, int); PyAPI_FUNC(int) PyFile_WriteString(const char *, PyObject *); Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Sat Oct 20 01:16:50 2007 @@ -178,6 +178,18 @@ return text +class OpenWrapper: + """Wrapper for __builtin__.open + + Trick so that open won't become a bound method when stored + as a class variable (as dumbdbm does). + + See initstdio() in Python/pythonrun.c. + """ + def __new__(cls, *args, **kwargs): + return open(*args, **kwargs) + + class UnsupportedOperation(ValueError, IOError): pass Modified: python/branches/py3k/Lib/site.py ============================================================================== --- python/branches/py3k/Lib/site.py (original) +++ python/branches/py3k/Lib/site.py Sat Oct 20 01:16:50 2007 @@ -402,23 +402,6 @@ (err.__class__.__name__, err)) -def installnewio(): - """Install new I/O library as default.""" - import io - # Hack to avoid a nasty recursion issue when Python is invoked - # in verbose mode: pre-import the Latin-1 and UTF-8 codecs - from encodings import latin_1, utf_8 - # Trick so that open won't become a bound method when stored - # as a class variable (as dumbdbm does) - class open: - def __new__(cls, *args, **kwds): - return io.open(*args, **kwds) - __builtin__.open = open - sys.__stdin__ = sys.stdin = io.open(0, "r", newline='\n') - sys.__stdout__ = sys.stdout = io.open(1, "w", newline='\n') - sys.__stderr__ = sys.stderr = io.open(2, "w", newline='\n') - - def main(): abs__file__() paths_in_sys = removeduppaths() @@ -433,7 +416,6 @@ sethelper() aliasmbcs() setencoding() - installnewio() execsitecustomize() # Remove sys.setdefaultencoding() so that users cannot change the # encoding after initialization. The test for presence is needed when Modified: python/branches/py3k/Lib/test/test_imp.py ============================================================================== --- python/branches/py3k/Lib/test/test_imp.py (original) +++ python/branches/py3k/Lib/test/test_imp.py Sat Oct 20 01:16:50 2007 @@ -38,9 +38,16 @@ self.fail("release_lock() without lock should raise " "RuntimeError") +class ImportTests(unittest.TestCase): + + def test_find_module_encoding(self): + fd = imp.find_module("heapq")[0] + self.assertEqual(fd.encoding, "iso-8859-1") + def test_main(): test_support.run_unittest( LockTests, + ImportTests, ) if __name__ == "__main__": Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Sat Oct 20 01:16:50 2007 @@ -28,22 +28,32 @@ PyObject * PyFile_FromFile(FILE *fp, char *name, char *mode, int (*close)(FILE *)) { - PyObject *io, *stream, *nameobj; + return PyFile_FromFileEx(fp, name, mode, close, -1, NULL, NULL); +} + +PyObject * +PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE *), + int buffering, char *encoding, char *newline) +{ + PyObject *io, *stream, *nameobj=NULL; io = PyImport_ImportModule("io"); if (io == NULL) return NULL; - stream = PyObject_CallMethod(io, "open", "is", fileno(fp), mode); - Py_DECREF(io); + stream = PyObject_CallMethod(io, "open", "isiss", fileno(fp), mode, + buffering, encoding, newline); + Py_DECREF(io); if (stream == NULL) return NULL; - nameobj = PyUnicode_FromString(name); - if (nameobj == NULL) - PyErr_Clear(); - else { - if (PyObject_SetAttrString(stream, "name", nameobj) < 0) + if (name != NULL) { + nameobj = PyUnicode_FromString(name); + if (nameobj == NULL) PyErr_Clear(); - Py_DECREF(nameobj); + else { + if (PyObject_SetAttrString(stream, "name", nameobj) < 0) + PyErr_Clear(); + Py_DECREF(nameobj); + } } return stream; } Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Sat Oct 20 01:16:50 2007 @@ -1601,7 +1601,28 @@ } #endif - +/* Get -*- encoding -*- from a Python file + + PyTokenizer_FindEncoding returns NULL when it can't find the encoding in + the first or second line of the file. In this case the encoding is + PyUnicode_GetDefaultEncoding(). +*/ +char * +PyTokenizer_FindEncoding(FILE *fp) { + struct tok_state *tok; + char *p_start=NULL, *p_end=NULL; + + if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) { + rewind(fp); + return NULL; + } + while(((tok->lineno <= 2) && (tok->done == E_OK))) { + PyTokenizer_Get(tok, &p_start, &p_end); + } + + rewind(fp); + return tok->encoding; +} #ifdef Py_DEBUG Modified: python/branches/py3k/Parser/tokenizer.h ============================================================================== --- python/branches/py3k/Parser/tokenizer.h (original) +++ python/branches/py3k/Parser/tokenizer.h Sat Oct 20 01:16:50 2007 @@ -67,6 +67,7 @@ extern int PyTokenizer_Get(struct tok_state *, char **, char **); extern char * PyTokenizer_RestoreEncoding(struct tok_state* tok, int len, int *offset); +extern char * PyTokenizer_FindEncoding(FILE *fp); #ifdef __cplusplus } Modified: python/branches/py3k/Python/import.c ============================================================================== --- python/branches/py3k/Python/import.c (original) +++ python/branches/py3k/Python/import.c Sat Oct 20 01:16:50 2007 @@ -91,6 +91,9 @@ /* This table is defined in config.c: */ extern struct _inittab _PyImport_Inittab[]; +/* Method from Parser/tokenizer.c */ +extern char * PyTokenizer_FindEncoding(FILE *fp); + struct _inittab *PyImport_Inittab = _PyImport_Inittab; /* these tables define the module suffixes that Python recognizes */ @@ -2558,6 +2561,7 @@ struct filedescr *fdp; char pathname[MAXPATHLEN+1]; FILE *fp = NULL; + char *encoding = NULL; pathname[0] = '\0'; if (path == Py_None) @@ -2566,7 +2570,14 @@ if (fdp == NULL) return NULL; if (fp != NULL) { - fob = PyFile_FromFile(fp, pathname, fdp->mode, fclose); + if (strchr(fdp->mode, 'b') == NULL) { + /* Python text file, get encoding from tokenizer */ + encoding = PyTokenizer_FindEncoding(fp); + encoding = (encoding != NULL) ? encoding : + (char*)PyUnicode_GetDefaultEncoding(); + } + fob = PyFile_FromFileEx(fp, pathname, fdp->mode, fclose, -1, + (char*)encoding, NULL); if (fob == NULL) { fclose(fp); return NULL; Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Sat Oct 20 01:16:50 2007 @@ -51,6 +51,7 @@ /* Forward */ static void initmain(void); static void initsite(void); +static int initstdio(void); static PyObject *run_mod(mod_ty, const char *, PyObject *, PyObject *, PyCompilerFlags *, PyArena *); static PyObject *run_pyc_file(FILE *, const char *, PyObject *, PyObject *, @@ -241,6 +242,9 @@ initsigs(); /* Signal handling stuff, including initintr() */ initmain(); /* Module __main__ */ + if (initstdio() < 0) + Py_FatalError( + "Py_Initialize: can't initialize sys standard streams"); if (!Py_NoSiteFlag) initsite(); /* Module site */ @@ -676,6 +680,81 @@ } } +/* Initialize sys.stdin, stdout, stderr and __builtin__.open */ +static int +initstdio(void) +{ + PyObject *iomod = NULL, *wrapper; + PyObject *bimod = NULL; + PyObject *m; + PyObject *std = NULL; + int status = 0; + + /* Hack to avoid a nasty recursion issue when Python is invoked + in verbose mode: pre-import the Latin-1 and UTF-8 codecs */ + if ((m = PyImport_ImportModule("encodings.utf_8")) == NULL) { + goto error; + } + Py_DECREF(m); + + if (!(m = PyImport_ImportModule("encodings.latin_1"))) { + goto error; + } + Py_DECREF(m); + + if (!(bimod = PyImport_ImportModule("__builtin__"))) { + goto error; + } + + if (!(iomod = PyImport_ImportModule("io"))) { + goto error; + } + if (!(wrapper = PyObject_GetAttrString(iomod, "OpenWrapper"))) { + goto error; + } + + /* Set __builtin__.open */ + if (PyObject_SetAttrString(bimod, "open", wrapper) == -1) { + goto error; + } + + /* Set sys.stdin */ + if (!(std = PyFile_FromFileEx(stdin, "", "r", fclose, -1, + NULL, "\n"))) { + goto error; + } + PySys_SetObject("__stdin__", std); + PySys_SetObject("stdin", std); + Py_DECREF(std); + + /* Set sys.stdout */ + if (!(std = PyFile_FromFileEx(stdout, "", "w", fclose, -1, + NULL, "\n"))) { + goto error; + } + PySys_SetObject("__stdout__", std); + PySys_SetObject("stdout", std); + Py_DECREF(std); + + /* Set sys.stderr */ + if (!(std = PyFile_FromFileEx(stderr, "", "w", fclose, -1, + NULL, "\n"))) { + goto error; + } + PySys_SetObject("__stderr__", std); + PySys_SetObject("stderr", std); + Py_DECREF(std); + + if (0) { + error: + status = -1; + } + + Py_XDECREF(bimod); + Py_XDECREF(iomod); + return status; +} + /* Parse input from a file and execute it */ int @@ -1146,10 +1225,10 @@ int err = 0; PyObject *f = PySys_GetObject("stderr"); Py_INCREF(value); - if (f == NULL) + if (f == NULL) { _PyObject_Dump(value); - if (f == NULL) fprintf(stderr, "lost sys.stderr\n"); + } else { fflush(stdout); if (tb && tb != Py_None) @@ -1589,6 +1668,9 @@ Py_FatalError(const char *msg) { fprintf(stderr, "Fatal Python error: %s\n", msg); + if (PyErr_Occurred()) { + PyErr_Print(); + } #ifdef MS_WINDOWS OutputDebugString("Fatal Python error: "); OutputDebugString(msg); From python-3000-checkins at python.org Sat Oct 20 02:05:20 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 20 Oct 2007 02:05:20 +0200 (CEST) Subject: [Python-3000-checkins] r58554 - python/branches/py3k/Include/patchlevel.h Message-ID: <20071020000520.039F91E4009@bag.python.org> Author: guido.van.rossum Date: Sat Oct 20 02:05:19 2007 New Revision: 58554 Modified: python/branches/py3k/Include/patchlevel.h Log: Tweak the version *string* to clarify that this isn't your father's 3.0a1. :-) Modified: python/branches/py3k/Include/patchlevel.h ============================================================================== --- python/branches/py3k/Include/patchlevel.h (original) +++ python/branches/py3k/Include/patchlevel.h Sat Oct 20 02:05:19 2007 @@ -22,7 +22,7 @@ #define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.0a1" +#define PY_VERSION "3.0a1+" /* Subversion Revision number of this file (not of the repository) */ #define PY_PATCHLEVEL_REVISION "$Revision$" From python-3000-checkins at python.org Sat Oct 20 04:54:14 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Sat, 20 Oct 2007 04:54:14 +0200 (CEST) Subject: [Python-3000-checkins] r58555 - python/branches/py3k/Parser/tokenizer.c Message-ID: <20071020025414.7F9D71E400B@bag.python.org> Author: brett.cannon Date: Sat Oct 20 04:54:14 2007 New Revision: 58555 Modified: python/branches/py3k/Parser/tokenizer.c Log: Plug a memory leak where a struct tok_state was not being freed. Also tweak a comparison that was going farther than needed. Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Sat Oct 20 04:54:14 2007 @@ -52,6 +52,7 @@ static int tok_nextc(struct tok_state *tok); static void tok_backup(struct tok_state *tok, int c); + /* Token names */ char *_PyParser_TokenNames[] = { @@ -1610,18 +1611,25 @@ char * PyTokenizer_FindEncoding(FILE *fp) { struct tok_state *tok; - char *p_start=NULL, *p_end=NULL; + char *p_start=NULL, *p_end=NULL, *encoding=NULL; if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) { rewind(fp); return NULL; } - while(((tok->lineno <= 2) && (tok->done == E_OK))) { + while(((tok->lineno < 2) && (tok->done == E_OK))) { PyTokenizer_Get(tok, &p_start, &p_end); } rewind(fp); - return tok->encoding; + + if (tok->encoding) { + encoding = (char *)PyMem_MALLOC(strlen(tok->encoding)); + strcpy(encoding, tok->encoding); + } + PyTokenizer_Free(tok); + + return encoding; } #ifdef Py_DEBUG From python-3000-checkins at python.org Sat Oct 20 05:43:16 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Sat, 20 Oct 2007 05:43:16 +0200 (CEST) Subject: [Python-3000-checkins] r58556 - python/branches/py3k/Python/import.c Message-ID: <20071020034316.0A9881E400A@bag.python.org> Author: brett.cannon Date: Sat Oct 20 05:43:15 2007 New Revision: 58556 Modified: python/branches/py3k/Python/import.c Log: Fix a memory leak caused by PyTokenizer_FindEncoding() returning a char * that was PyMem_MALLOC'ed. Modified: python/branches/py3k/Python/import.c ============================================================================== --- python/branches/py3k/Python/import.c (original) +++ python/branches/py3k/Python/import.c Sat Oct 20 05:43:15 2007 @@ -2561,6 +2561,7 @@ struct filedescr *fdp; char pathname[MAXPATHLEN+1]; FILE *fp = NULL; + char *found_encoding = NULL; char *encoding = NULL; pathname[0] = '\0'; @@ -2571,15 +2572,17 @@ return NULL; if (fp != NULL) { if (strchr(fdp->mode, 'b') == NULL) { - /* Python text file, get encoding from tokenizer */ - encoding = PyTokenizer_FindEncoding(fp); - encoding = (encoding != NULL) ? encoding : + /* PyTokenizer_FindEncoding() returns PyMem_MALLOC'ed + memory. */ + found_encoding = PyTokenizer_FindEncoding(fp); + encoding = (found_encoding != NULL) ? found_encoding : (char*)PyUnicode_GetDefaultEncoding(); } fob = PyFile_FromFileEx(fp, pathname, fdp->mode, fclose, -1, (char*)encoding, NULL); if (fob == NULL) { fclose(fp); + PyMem_FREE(found_encoding); return NULL; } } @@ -2590,6 +2593,8 @@ ret = Py_BuildValue("Os(ssi)", fob, pathname, fdp->suffix, fdp->mode, fdp->type); Py_DECREF(fob); + PyMem_FREE(found_encoding); + return ret; } From python-3000-checkins at python.org Sat Oct 20 05:46:49 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Sat, 20 Oct 2007 05:46:49 +0200 (CEST) Subject: [Python-3000-checkins] r58557 - python/branches/py3k/Parser/tokenizer.c Message-ID: <20071020034649.6F7B61E4025@bag.python.org> Author: brett.cannon Date: Sat Oct 20 05:46:49 2007 New Revision: 58557 Modified: python/branches/py3k/Parser/tokenizer.c Log: Fix PyTokenizer_FindEncoding() for OS X 10.4. Turns out that seeking to the beginning of a file through a file pointer is not reflected when reading from a file descriptor. Using both fflush() and fpurge() does not solve it. One must use lseek() directly on the file descriptor to get the desired effect. This might suggest that we standardize on either file pointers (FILE) or file descriptors (int) for all C code used. Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Sat Oct 20 05:46:49 2007 @@ -1605,8 +1605,11 @@ /* Get -*- encoding -*- from a Python file PyTokenizer_FindEncoding returns NULL when it can't find the encoding in - the first or second line of the file. In this case the encoding is - PyUnicode_GetDefaultEncoding(). + the first or second line of the file (in which case the encoding + should be assumed to be PyUnicode_GetDefaultEncoding()). + + The char * returned was malloc'ed from PyMem_MALLOC() and thus must be freed + when no longer needed. */ char * PyTokenizer_FindEncoding(FILE *fp) { @@ -1614,14 +1617,18 @@ char *p_start=NULL, *p_end=NULL, *encoding=NULL; if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) { - rewind(fp); + /* lseek() usage is on purpose; see note later in code. */ + lseek(fileno(fp), 0, 0); return NULL; } while(((tok->lineno < 2) && (tok->done == E_OK))) { PyTokenizer_Get(tok, &p_start, &p_end); } - rewind(fp); + /* lseek() must be used instead of fseek()/rewind() as those fail on + OS X 10.4 to properly seek back to the beginning when reading from + the file descriptor instead of the file pointer. */ + lseek(fileno(fp), 0, 0); if (tok->encoding) { encoding = (char *)PyMem_MALLOC(strlen(tok->encoding)); From python-3000-checkins at python.org Sun Oct 21 04:45:33 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Sun, 21 Oct 2007 04:45:33 +0200 (CEST) Subject: [Python-3000-checkins] r58575 - python/branches/py3k/Parser/tokenizer.c Message-ID: <20071021024533.563871E426D@bag.python.org> Author: brett.cannon Date: Sun Oct 21 04:45:33 2007 New Revision: 58575 Modified: python/branches/py3k/Parser/tokenizer.c Log: Make sure the malloc'ed string has space for the null byte. Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Sun Oct 21 04:45:33 2007 @@ -1631,7 +1631,7 @@ lseek(fileno(fp), 0, 0); if (tok->encoding) { - encoding = (char *)PyMem_MALLOC(strlen(tok->encoding)); + encoding = (char *)PyMem_MALLOC(strlen(tok->encoding) + 1); strcpy(encoding, tok->encoding); } PyTokenizer_Free(tok); From python-3000-checkins at python.org Mon Oct 22 02:09:51 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 22 Oct 2007 02:09:51 +0200 (CEST) Subject: [Python-3000-checkins] r58587 - in python/branches/py3k: Doc/c-api/concrete.rst Doc/reference/introduction.rst Include/fileobject.h Lib/test/test_imp.py Misc/NEWS Modules/posixmodule.c Objects/bytesobject.c Objects/fileobject.c Parser/tokenizer.c Parser/tokenizer.h Python/import.c Python/pythonrun.c Message-ID: <20071022000951.D3C451E400E@bag.python.org> Author: guido.van.rossum Date: Mon Oct 22 02:09:51 2007 New Revision: 58587 Modified: python/branches/py3k/Doc/c-api/concrete.rst python/branches/py3k/Doc/reference/introduction.rst python/branches/py3k/Include/fileobject.h python/branches/py3k/Lib/test/test_imp.py python/branches/py3k/Misc/NEWS python/branches/py3k/Modules/posixmodule.c python/branches/py3k/Objects/bytesobject.c python/branches/py3k/Objects/fileobject.c python/branches/py3k/Parser/tokenizer.c python/branches/py3k/Parser/tokenizer.h python/branches/py3k/Python/import.c python/branches/py3k/Python/pythonrun.c Log: Issue 1267, continued. Additional patch by Christian Heimes to deal more cleanly with the FILE* vs file-descriptor issues. I cleaned up his code a bit, and moved the lseek() call into import.c. Modified: python/branches/py3k/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k/Doc/c-api/concrete.rst (original) +++ python/branches/py3k/Doc/c-api/concrete.rst Mon Oct 22 02:09:51 2007 @@ -2410,31 +2410,23 @@ :ctype:`PyFileObject`. -.. cfunction:: PyObject* PyFile_FromString(char *filename, char *mode) +.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *newline) - .. index:: single: fopen() + Create a new :ctype:`PyFileObject` from the file descriptor of an already + opened file *fd*. The arguments *name*, *encoding* and *newline* can be + *NULL* as well as buffering can be *-1* to use the defaults. Return *NULL* on + failure. - On success, return a new file object that is opened on the file given by - *filename*, with a file mode given by *mode*, where *mode* has the same - semantics as the standard C routine :cfunc:`fopen`. On failure, return *NULL*. + .. warning:: + Take care when you are mixing streams and descriptors! For more + information, see `GNU C Library + `_. -.. cfunction:: PyObject* PyFile_FromFile(FILE *fp, char *name, char *mode, int (*close)(FILE*)) - Create a new :ctype:`PyFileObject` from the already-open standard C file - pointer, *fp*. The function *close* will be called when the file should be - closed. Return *NULL* on failure. +.. cfunction:: int PyObject_AsFileDescriptor(PyObject *p) -.. cfunction:: PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE *), int buffering, char *encoding, char *newline) - - Create a new :ctype:`PyFileObject` from the already-open standard C file - pointer, *fp*. The functions works similar to *PyFile_FromFile* but takes - optional arguments for *buffering*, *encoding* and *newline*. Use -1 resp. - *NULL* for default values. - -.. cfunction:: FILE* PyFile_AsFile(PyObject *p) - - Return the file object associated with *p* as a :ctype:`FILE\*`. + Return the file descriptor associated with *p* as an :ctype:`int`. .. cfunction:: PyObject* PyFile_GetLine(PyObject *p, int n) Modified: python/branches/py3k/Doc/reference/introduction.rst ============================================================================== --- python/branches/py3k/Doc/reference/introduction.rst (original) +++ python/branches/py3k/Doc/reference/introduction.rst Mon Oct 22 02:09:51 2007 @@ -60,7 +60,7 @@ This implementation actually uses the CPython implementation, but is a managed .NET application and makes .NET libraries available. This was created by Brian Lloyd. For more information, see the `Python for .NET home page - `_. + `_. IronPython An alternate Python for .NET. Unlike Python.NET, this is a complete Python Modified: python/branches/py3k/Include/fileobject.h ============================================================================== --- python/branches/py3k/Include/fileobject.h (original) +++ python/branches/py3k/Include/fileobject.h Mon Oct 22 02:09:51 2007 @@ -8,10 +8,7 @@ #define PY_STDIOTEXTMODE "b" -PyAPI_FUNC(PyObject *) PyFile_FromFile(FILE *, char *, char *, int (*)(FILE*)); -PyAPI_FUNC(PyObject *) PyFile_FromFileEx(FILE *, char *, char *, - int (*)(FILE *), int, char *, - char *); +PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *); PyAPI_FUNC(PyObject *) PyFile_GetLine(PyObject *, int); PyAPI_FUNC(int) PyFile_WriteObject(PyObject *, PyObject *, int); PyAPI_FUNC(int) PyFile_WriteString(const char *, PyObject *); Modified: python/branches/py3k/Lib/test/test_imp.py ============================================================================== --- python/branches/py3k/Lib/test/test_imp.py (original) +++ python/branches/py3k/Lib/test/test_imp.py Mon Oct 22 02:09:51 2007 @@ -44,6 +44,23 @@ fd = imp.find_module("heapq")[0] self.assertEqual(fd.encoding, "iso-8859-1") + def test_issue1267(self): + fp, filename, info = imp.find_module("pydoc") + self.assertNotEqual(fp, None) + self.assertEqual(fp.encoding, "iso-8859-1") + self.assertEqual(fp.tell(), 0) + self.assertEqual(fp.readline(), '#!/usr/bin/env python\n') + fp.close() + + fp, filename, info = imp.find_module("tokenize") + self.assertNotEqual(fp, None) + self.assertEqual(fp.encoding, "utf-8") + self.assertEqual(fp.tell(), 0) + self.assertEqual(fp.readline(), + '"""Tokenization help for Python programs.\n') + fp.close() + + def test_main(): test_support.run_unittest( LockTests, Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Mon Oct 22 02:09:51 2007 @@ -8,6 +8,18 @@ *Unreleased* +Core and Builtins +----------------- + +- Replaced `PyFile_FromFile()` with `PyFile_FromFd(fd, name. mode, buffer, + encoding, newline)` + +- Fixed `imp.find_module()` to obey the -*- coding: -*- header. + +- Changed `__file__` and `co_filename` to unicode. The path names are decoded + with `Py_FileSystemDefaultEncoding` and a new API method + `PyUnicode_DecodeFSDefault(char*)` was added. + Extension Modules ----------------- Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Mon Oct 22 02:09:51 2007 @@ -5386,11 +5386,18 @@ posix_tmpfile(PyObject *self, PyObject *noargs) { FILE *fp; + int fd; fp = tmpfile(); if (fp == NULL) return posix_error(); - return PyFile_FromFile(fp, "", "w+b", fclose); + fd = fileno(fp); + if (fd != -1) + fd = dup(fd); + fclose(fp); + if (fd == -1) + return posix_error(); + return PyFile_FromFd(fd, "", "w+b", -1, NULL, NULL); } #endif Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Mon Oct 22 02:09:51 2007 @@ -1214,7 +1214,7 @@ Py_ssize_t len = PyBytes_GET_SIZE(self); const char* str; Py_buffer vsubstr; - int rv; + int rv = 0; str = PyBytes_AS_STRING(self); @@ -1226,13 +1226,11 @@ if (direction < 0) { /* startswith */ if (start+vsubstr.len > len) { - rv = 0; goto done; } } else { /* endswith */ if (end-start < vsubstr.len || start > len) { - rv = 0; goto done; } Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Mon Oct 22 02:09:51 2007 @@ -26,22 +26,16 @@ /* External C interface */ PyObject * -PyFile_FromFile(FILE *fp, char *name, char *mode, int (*close)(FILE *)) +PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, + char *newline) { - return PyFile_FromFileEx(fp, name, mode, close, -1, NULL, NULL); -} - -PyObject * -PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE *), - int buffering, char *encoding, char *newline) -{ - PyObject *io, *stream, *nameobj=NULL; + PyObject *io, *stream, *nameobj = NULL; io = PyImport_ImportModule("io"); if (io == NULL) return NULL; - stream = PyObject_CallMethod(io, "open", "isiss", fileno(fp), mode, - buffering, encoding, newline); + stream = PyObject_CallMethod(io, "open", "isiss", fd, mode, + buffering, encoding, newline); Py_DECREF(io); if (stream == NULL) return NULL; Modified: python/branches/py3k/Parser/tokenizer.c ============================================================================== --- python/branches/py3k/Parser/tokenizer.c (original) +++ python/branches/py3k/Parser/tokenizer.c Mon Oct 22 02:09:51 2007 @@ -1602,40 +1602,44 @@ } #endif -/* Get -*- encoding -*- from a Python file +/* Get -*- encoding -*- from a Python file. PyTokenizer_FindEncoding returns NULL when it can't find the encoding in the first or second line of the file (in which case the encoding should be assumed to be PyUnicode_GetDefaultEncoding()). - The char * returned was malloc'ed from PyMem_MALLOC() and thus must be freed - when no longer needed. + The char * returned is malloc'ed via PyMem_MALLOC() and thus must be freed + by the caller. */ char * -PyTokenizer_FindEncoding(FILE *fp) { +PyTokenizer_FindEncoding(int fd) +{ struct tok_state *tok; - char *p_start=NULL, *p_end=NULL, *encoding=NULL; + FILE *fp; + char *p_start =NULL , *p_end =NULL , *encoding = NULL; - if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) { - /* lseek() usage is on purpose; see note later in code. */ - lseek(fileno(fp), 0, 0); + fd = dup(fd); + if (fd < 0) { + return NULL; + } + fp = fdopen(fd, "r"); + if (fp == NULL) { + return NULL; + } + tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL); + if (tok == NULL) { + fclose(fp); return NULL; } - while(((tok->lineno < 2) && (tok->done == E_OK))) { + while (tok->lineno < 2 && tok->done == E_OK) { PyTokenizer_Get(tok, &p_start, &p_end); } - - /* lseek() must be used instead of fseek()/rewind() as those fail on - OS X 10.4 to properly seek back to the beginning when reading from - the file descriptor instead of the file pointer. */ - lseek(fileno(fp), 0, 0); - + fclose(fp); if (tok->encoding) { encoding = (char *)PyMem_MALLOC(strlen(tok->encoding) + 1); strcpy(encoding, tok->encoding); } PyTokenizer_Free(tok); - return encoding; } Modified: python/branches/py3k/Parser/tokenizer.h ============================================================================== --- python/branches/py3k/Parser/tokenizer.h (original) +++ python/branches/py3k/Parser/tokenizer.h Mon Oct 22 02:09:51 2007 @@ -67,7 +67,7 @@ extern int PyTokenizer_Get(struct tok_state *, char **, char **); extern char * PyTokenizer_RestoreEncoding(struct tok_state* tok, int len, int *offset); -extern char * PyTokenizer_FindEncoding(FILE *fp); +extern char * PyTokenizer_FindEncoding(int); #ifdef __cplusplus } Modified: python/branches/py3k/Python/import.c ============================================================================== --- python/branches/py3k/Python/import.c (original) +++ python/branches/py3k/Python/import.c Mon Oct 22 02:09:51 2007 @@ -92,7 +92,7 @@ extern struct _inittab _PyImport_Inittab[]; /* Method from Parser/tokenizer.c */ -extern char * PyTokenizer_FindEncoding(FILE *fp); +extern char * PyTokenizer_FindEncoding(int); struct _inittab *PyImport_Inittab = _PyImport_Inittab; @@ -2561,6 +2561,7 @@ struct filedescr *fdp; char pathname[MAXPATHLEN+1]; FILE *fp = NULL; + int fd = -1; char *found_encoding = NULL; char *encoding = NULL; @@ -2571,17 +2572,25 @@ if (fdp == NULL) return NULL; if (fp != NULL) { + fd = fileno(fp); + if (fd != -1) + fd = dup(fd); + fclose(fp); + fp = NULL; + } + if (fd != -1) { if (strchr(fdp->mode, 'b') == NULL) { /* PyTokenizer_FindEncoding() returns PyMem_MALLOC'ed memory. */ - found_encoding = PyTokenizer_FindEncoding(fp); + found_encoding = PyTokenizer_FindEncoding(fd); + lseek(fd, 0, 0); /* Reset position */ encoding = (found_encoding != NULL) ? found_encoding : (char*)PyUnicode_GetDefaultEncoding(); } - fob = PyFile_FromFileEx(fp, pathname, fdp->mode, fclose, -1, + fob = PyFile_FromFd(fd, pathname, fdp->mode, -1, (char*)encoding, NULL); if (fob == NULL) { - fclose(fp); + close(fd); PyMem_FREE(found_encoding); return NULL; } Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Mon Oct 22 02:09:51 2007 @@ -719,7 +719,7 @@ } /* Set sys.stdin */ - if (!(std = PyFile_FromFileEx(stdin, "", "r", fclose, -1, + if (!(std = PyFile_FromFd(fileno(stdin), "", "r", -1, NULL, "\n"))) { goto error; } @@ -728,7 +728,7 @@ Py_DECREF(std); /* Set sys.stdout */ - if (!(std = PyFile_FromFileEx(stdout, "", "w", fclose, -1, + if (!(std = PyFile_FromFd(fileno(stdout), "", "w", -1, NULL, "\n"))) { goto error; } @@ -737,7 +737,7 @@ Py_DECREF(std); /* Set sys.stderr */ - if (!(std = PyFile_FromFileEx(stderr, "", "w", fclose, -1, + if (!(std = PyFile_FromFd(fileno(stderr), "", "w", -1, NULL, "\n"))) { goto error; } From python-3000-checkins at python.org Mon Oct 22 14:42:47 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Mon, 22 Oct 2007 14:42:47 +0200 (CEST) Subject: [Python-3000-checkins] r58590 - python/branches/py3k/Lib/tempfile.py Message-ID: <20071022124247.288F21E4013@bag.python.org> Author: georg.brandl Date: Mon Oct 22 14:42:46 2007 New Revision: 58590 Modified: python/branches/py3k/Lib/tempfile.py Log: Fix exception indexing. Modified: python/branches/py3k/Lib/tempfile.py ============================================================================== --- python/branches/py3k/Lib/tempfile.py (original) +++ python/branches/py3k/Lib/tempfile.py Mon Oct 22 14:42:46 2007 @@ -201,7 +201,7 @@ del fp, fd return dir except (OSError, IOError) as e: - if e[0] != _errno.EEXIST: + if e.args[0] != _errno.EEXIST: break # no point trying more names in this directory pass raise IOError(_errno.ENOENT, From python-3000-checkins at python.org Mon Oct 22 18:00:45 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 22 Oct 2007 18:00:45 +0200 (CEST) Subject: [Python-3000-checkins] r58592 - python/branches/py3k/PCbuild/pythoncore.vcproj Message-ID: <20071022160045.E640E1E4010@bag.python.org> Author: guido.van.rossum Date: Mon Oct 22 18:00:45 2007 New Revision: 58592 Modified: python/branches/py3k/PCbuild/pythoncore.vcproj Log: ssue 1309: fix by Christian Heimes to be able to build on Windows. Modified: python/branches/py3k/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/py3k/PCbuild/pythoncore.vcproj (original) +++ python/branches/py3k/PCbuild/pythoncore.vcproj Mon Oct 22 18:00:45 2007 @@ -440,8 +440,11 @@ RelativePath="..\Objects\boolobject.c"> + RelativePath="..\Objects\bytes_methods.c"> + + From python-3000-checkins at python.org Mon Oct 22 18:16:14 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Mon, 22 Oct 2007 18:16:14 +0200 (CEST) Subject: [Python-3000-checkins] r58593 - in python/branches/py3k: Lib/asyncore.py Lib/idlelib/run.py Tools/i18n/pygettext.py Message-ID: <20071022161614.2CABA1E4031@bag.python.org> Author: georg.brandl Date: Mon Oct 22 18:16:13 2007 New Revision: 58593 Modified: python/branches/py3k/Lib/asyncore.py python/branches/py3k/Lib/idlelib/run.py python/branches/py3k/Tools/i18n/pygettext.py Log: In followup to #1310: Remove more exception indexing. Modified: python/branches/py3k/Lib/asyncore.py ============================================================================== --- python/branches/py3k/Lib/asyncore.py (original) +++ python/branches/py3k/Lib/asyncore.py Mon Oct 22 18:16:13 2007 @@ -120,7 +120,7 @@ try: r, w, e = select.select(r, w, e, timeout) except select.error as err: - if err[0] != EINTR: + if err.args[0] != EINTR: raise else: return @@ -166,7 +166,7 @@ try: r = pollster.poll(timeout) except select.error as err: - if err[0] != EINTR: + if err.args[0] != EINTR: raise r = [] for fd, flags in r: Modified: python/branches/py3k/Lib/idlelib/run.py ============================================================================== --- python/branches/py3k/Lib/idlelib/run.py (original) +++ python/branches/py3k/Lib/idlelib/run.py Mon Oct 22 18:16:13 2007 @@ -115,8 +115,8 @@ server = MyRPCServer(address, MyHandler) break except socket.error as err: - print("IDLE Subprocess: socket error: "\ - + err[1] + ", retrying....", file=sys.__stderr__) + print("IDLE Subprocess: socket error: " + err.args[1] + + ", retrying....", file=sys.__stderr__) else: print("IDLE Subprocess: Connection to "\ "IDLE GUI failed, exiting.", file=sys.__stderr__) @@ -131,14 +131,15 @@ import tkMessageBox root = Tkinter.Tk() root.withdraw() - if err[0] == 61: # connection refused + if err.args[0] == 61: # connection refused msg = "IDLE's subprocess can't connect to %s:%d. This may be due "\ "to your personal firewall configuration. It is safe to "\ "allow this internal connection because no data is visible on "\ "external ports." % address tkMessageBox.showerror("IDLE Subprocess Error", msg, parent=root) else: - tkMessageBox.showerror("IDLE Subprocess Error", "Socket Error: %s" % err[1]) + tkMessageBox.showerror("IDLE Subprocess Error", + "Socket Error: %s" % err.args[1]) root.destroy() def print_exception(): Modified: python/branches/py3k/Tools/i18n/pygettext.py ============================================================================== --- python/branches/py3k/Tools/i18n/pygettext.py (original) +++ python/branches/py3k/Tools/i18n/pygettext.py Mon Oct 22 18:16:13 2007 @@ -637,7 +637,8 @@ tokenize.tokenize(fp.readline, eater) except tokenize.TokenError as e: print('%s: %s, line %d, column %d' % ( - e[0], filename, e[1][0], e[1][1]), file=sys.stderr) + e.args[0], filename, e.args[1][0], e.args[1][1]), + file=sys.stderr) finally: if closep: fp.close() From python-3000-checkins at python.org Mon Oct 22 22:24:52 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Mon, 22 Oct 2007 22:24:52 +0200 (CEST) Subject: [Python-3000-checkins] r58596 - in python/branches/py3k: Lib/modulefinder.py Lib/sqlite3/test/types.py Lib/test/test_compile.py Lib/test/test_format.py Lib/test/test_str.py Lib/test/test_struct.py Lib/test/test_subprocess.py Lib/test/test_unicode.py Misc/NEWS Modules/_sqlite/cursor.c Objects/unicodeobject.c Message-ID: <20071022202452.6954E1E40E0@bag.python.org> Author: brett.cannon Date: Mon Oct 22 22:24:51 2007 New Revision: 58596 Modified: python/branches/py3k/Lib/modulefinder.py python/branches/py3k/Lib/sqlite3/test/types.py python/branches/py3k/Lib/test/test_compile.py python/branches/py3k/Lib/test/test_format.py python/branches/py3k/Lib/test/test_str.py python/branches/py3k/Lib/test/test_struct.py python/branches/py3k/Lib/test/test_subprocess.py python/branches/py3k/Lib/test/test_unicode.py python/branches/py3k/Misc/NEWS python/branches/py3k/Modules/_sqlite/cursor.c python/branches/py3k/Objects/unicodeobject.c Log: Make str/str8 comparisons return True/False for !=/==. Code that has been returning str8 becomes much more apparent thanks to this (e.g., struct module returning str8 for all string-related formats or sqlite3 passing in str8 instances when converting objects that had a __conform__ method). One also has to watch out in C code when making a key from char * using PyString in the C code but a str instance in Python code as that will not longer compare equal. Once str8 gains a constructor like the current bytes type then test_modulefinder needs a cleanup as the fix is a little messy in that file. Thanks goes to Thomas Lee for writing the patch for the change giving an initial run-down of why most of the tests were failing. Modified: python/branches/py3k/Lib/modulefinder.py ============================================================================== --- python/branches/py3k/Lib/modulefinder.py (original) +++ python/branches/py3k/Lib/modulefinder.py Mon Oct 22 22:24:51 2007 @@ -16,12 +16,13 @@ # remain compatible with Python < 2.3 READ_MODE = "r" -LOAD_CONST = chr(dis.opname.index('LOAD_CONST')) -IMPORT_NAME = chr(dis.opname.index('IMPORT_NAME')) -STORE_NAME = chr(dis.opname.index('STORE_NAME')) -STORE_GLOBAL = chr(dis.opname.index('STORE_GLOBAL')) +# XXX Clean up once str8's cstor matches bytes. +LOAD_CONST = str8(chr(dis.opname.index('LOAD_CONST'))) +IMPORT_NAME = str8(chr(dis.opname.index('IMPORT_NAME'))) +STORE_NAME = str8(chr(dis.opname.index('STORE_NAME'))) +STORE_GLOBAL = str8(chr(dis.opname.index('STORE_GLOBAL'))) STORE_OPS = [STORE_NAME, STORE_GLOBAL] -HAVE_ARGUMENT = chr(dis.HAVE_ARGUMENT) +HAVE_ARGUMENT = str8(chr(dis.HAVE_ARGUMENT)) # Modulefinder does a good job at simulating Python's, but it can not # handle __path__ modifications packages make at runtime. Therefore there @@ -367,7 +368,7 @@ consts = co.co_consts LOAD_LOAD_AND_IMPORT = LOAD_CONST + LOAD_CONST + IMPORT_NAME while code: - c = chr(code[0]) + c = str8(chr(code[0])) if c in STORE_OPS: oparg, = unpack(' %d # expected %d" % ( xfmt, n, len(res))) rev = struct.unpack(xfmt, res)[0] + if isinstance(arg, str): + # Strings are returned as str8 since you can't know the encoding of + # the string when packed. + arg = str8(arg) if rev != arg and not asy: raise TestFailed("unpack(%r, %r) -> (%r,) # expected (%r,)" % ( fmt, res, rev, arg)) @@ -424,14 +428,14 @@ def test_p_code(): for code, input, expected, expectedback in [ - ('p','abc', '\x00', ''), - ('1p', 'abc', '\x00', ''), - ('2p', 'abc', '\x01a', 'a'), - ('3p', 'abc', '\x02ab', 'ab'), - ('4p', 'abc', '\x03abc', 'abc'), - ('5p', 'abc', '\x03abc\x00', 'abc'), - ('6p', 'abc', '\x03abc\x00\x00', 'abc'), - ('1000p', 'x'*1000, '\xff' + 'x'*999, 'x'*255)]: + ('p','abc', '\x00', str8('')), + ('1p', 'abc', '\x00', str8('')), + ('2p', 'abc', '\x01a', str8('a')), + ('3p', 'abc', '\x02ab', str8('ab')), + ('4p', 'abc', '\x03abc', str8('abc')), + ('5p', 'abc', '\x03abc\x00', str8('abc')), + ('6p', 'abc', '\x03abc\x00\x00', str8('abc')), + ('1000p', 'x'*1000, '\xff' + 'x'*999, str8('x'*255))]: expected = bytes(expected, "latin-1") got = struct.pack(code, input) if got != expected: @@ -560,20 +564,20 @@ if verbose: print("test_unpack_from using", cls.__name__) data = cls(test_string) - vereq(s.unpack_from(data), ('abcd',)) - vereq(s.unpack_from(data, 2), ('cd01',)) - vereq(s.unpack_from(data, 4), ('0123',)) + vereq(s.unpack_from(data), (str8('abcd'),)) + vereq(s.unpack_from(data, 2), (str8('cd01'),)) + vereq(s.unpack_from(data, 4), (str8('0123'),)) for i in range(6): - vereq(s.unpack_from(data, i), (data[i:i+4],)) + vereq(s.unpack_from(data, i), (str8(data[i:i+4]),)) for i in range(6, len(test_string) + 1): simple_err(s.unpack_from, data, i) for cls in (str, str8, bytes): # XXX + memoryview data = cls(test_string) - vereq(struct.unpack_from(fmt, data), ('abcd',)) - vereq(struct.unpack_from(fmt, data, 2), ('cd01',)) - vereq(struct.unpack_from(fmt, data, 4), ('0123',)) + vereq(struct.unpack_from(fmt, data), (str8('abcd'),)) + vereq(struct.unpack_from(fmt, data, 2), (str8('cd01'),)) + vereq(struct.unpack_from(fmt, data, 4), (str8('0123'),)) for i in range(6): - vereq(struct.unpack_from(fmt, data, i), (data[i:i+4],)) + vereq(struct.unpack_from(fmt, data, i), (str8(data[i:i+4]),)) for i in range(6, len(test_string) + 1): simple_err(struct.unpack_from, fmt, data, i) Modified: python/branches/py3k/Lib/test/test_subprocess.py ============================================================================== --- python/branches/py3k/Lib/test/test_subprocess.py (original) +++ python/branches/py3k/Lib/test/test_subprocess.py Mon Oct 22 22:24:51 2007 @@ -24,7 +24,7 @@ # shutdown time. That frustrates tests trying to check stderr produced # from a spawned Python process. def remove_stderr_debug_decorations(stderr): - return re.sub(r"\[\d+ refs\]\r?\n?$", "", str8(stderr)) + return re.sub(r"\[\d+ refs\]\r?\n?$", "", str(stderr)) class ProcessTestCase(unittest.TestCase): def setUp(self): Modified: python/branches/py3k/Lib/test/test_unicode.py ============================================================================== --- python/branches/py3k/Lib/test/test_unicode.py (original) +++ python/branches/py3k/Lib/test/test_unicode.py Mon Oct 22 22:24:51 2007 @@ -200,6 +200,10 @@ self.checkequalnofix('one at two!three!', 'one!two!three!', 'replace', '!', '@', 1) self.assertRaises(TypeError, 'replace'.replace, "r", 42) + def test_str8_comparison(self): + self.assertEqual('abc' == str8('abc'), False) + self.assertEqual('abc' != str8('abc'), True) + def test_comparison(self): # Comparisons: self.assertEqual('abc', 'abc') Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Mon Oct 22 22:24:51 2007 @@ -11,6 +11,12 @@ Core and Builtins ----------------- +- Comparisons between str and str8 now return False/True for ==/!=. sqlite3 + returns str8 when recreating on object from it's __conform__ value. The + struct module returns str8 for all string-related formats. This was true + before this change, but becomes more apparent thanks to string comparisons + always being False. + - Replaced `PyFile_FromFile()` with `PyFile_FromFd(fd, name. mode, buffer, encoding, newline)` Modified: python/branches/py3k/Modules/_sqlite/cursor.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/cursor.c (original) +++ python/branches/py3k/Modules/_sqlite/cursor.c Mon Oct 22 22:24:51 2007 @@ -182,7 +182,7 @@ if (*pos == '[') { type_start = pos + 1; } else if (*pos == ']' && type_start != (const char*)-1) { - key = PyString_FromStringAndSize(type_start, pos - type_start); + key = PyUnicode_FromStringAndSize(type_start, pos - type_start); if (!key) { /* creating a string failed, but it is too complicated * to propagate the error here, we just assume there is @@ -203,7 +203,7 @@ if (decltype) { for (pos = decltype;;pos++) { if (*pos == ' ' || *pos == 0) { - py_decltype = PyString_FromStringAndSize(decltype, pos - decltype); + py_decltype = PyUnicode_FromStringAndSize(decltype, pos - decltype); if (!py_decltype) { return -1; } Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Mon Oct 22 22:24:51 2007 @@ -6224,16 +6224,6 @@ if (PyUnicode_Check(left) && PyUnicode_Check(right)) return unicode_compare((PyUnicodeObject *)left, (PyUnicodeObject *)right); - if ((PyString_Check(left) && PyUnicode_Check(right)) || - (PyUnicode_Check(left) && PyString_Check(right))) { - if (PyUnicode_Check(left)) - left = _PyUnicode_AsDefaultEncodedString(left, NULL); - if (PyUnicode_Check(right)) - right = _PyUnicode_AsDefaultEncodedString(right, NULL); - assert(PyString_Check(left)); - assert(PyString_Check(right)); - return PyObject_Compare(left, right); - } PyErr_Format(PyExc_TypeError, "Can't compare %.100s and %.100s", left->ob_type->tp_name, From python-3000-checkins at python.org Tue Oct 23 00:18:52 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 23 Oct 2007 00:18:52 +0200 (CEST) Subject: [Python-3000-checkins] r58597 - in python/branches/py3k: PC/config.c PCbuild/pythoncore.vcproj PCbuild8/pythoncore/pythoncore.vcproj Message-ID: <20071022221852.33D3C1E402E@bag.python.org> Author: guido.van.rossum Date: Tue Oct 23 00:18:51 2007 New Revision: 58597 Modified: python/branches/py3k/PC/config.c python/branches/py3k/PCbuild/pythoncore.vcproj python/branches/py3k/PCbuild8/pythoncore/pythoncore.vcproj Log: Patch 1304, by Amaury Forgeot d'Arc. Add md5module.c and sha1module.c to the project files, and in some cases bytes_methods.c and related .h files. Modified: python/branches/py3k/PC/config.c ============================================================================== --- python/branches/py3k/PC/config.c (original) +++ python/branches/py3k/PC/config.c Tue Oct 23 00:18:51 2007 @@ -14,9 +14,11 @@ extern void initerrno(void); extern void initgc(void); extern void initmath(void); +extern void init_md5(void); extern void initnt(void); extern void initoperator(void); extern void initsignal(void); +extern void init_sha1(void); extern void init_sha256(void); extern void init_sha512(void); extern void inittime(void); @@ -83,6 +85,8 @@ {"nt", initnt}, /* Use the NT os functions, not posix */ {"operator", initoperator}, {"signal", initsignal}, + {"_md5", init_md5}, + {"_sha1", init_sha1}, {"_sha256", init_sha256}, {"_sha512", init_sha512}, {"time", inittime}, Modified: python/branches/py3k/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/py3k/PCbuild/pythoncore.vcproj (original) +++ python/branches/py3k/PCbuild/pythoncore.vcproj Tue Oct 23 00:18:51 2007 @@ -442,9 +442,9 @@ - - + + @@ -632,6 +632,9 @@ RelativePath="..\Objects\methodobject.c"> + + + + + + @@ -974,6 +978,14 @@ > + + + + @@ -1490,6 +1502,10 @@ > + + @@ -1514,6 +1530,10 @@ > + + From python-3000-checkins at python.org Tue Oct 23 08:26:46 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 08:26:46 +0200 (CEST) Subject: [Python-3000-checkins] r58602 - in python/branches/py3k: Doc/library/weakref.rst Lib/abc.py Lib/weakref.py Modules/Setup.dist Message-ID: <20071023062646.D42D31E4012@bag.python.org> Author: georg.brandl Date: Tue Oct 23 08:26:46 2007 New Revision: 58602 Modified: python/branches/py3k/Doc/library/weakref.rst python/branches/py3k/Lib/abc.py python/branches/py3k/Lib/weakref.py python/branches/py3k/Modules/Setup.dist Log: #1061 (mainly by Thomas Wouters): use weak sets for abc caches. Modified: python/branches/py3k/Doc/library/weakref.rst ============================================================================== --- python/branches/py3k/Doc/library/weakref.rst (original) +++ python/branches/py3k/Doc/library/weakref.rst Tue Oct 23 08:26:46 2007 @@ -28,23 +28,26 @@ binary image objects, you may wish to associate a name with each. If you used a Python dictionary to map names to images, or images to names, the image objects would remain alive just because they appeared as values or keys in the -dictionaries. The :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` -classes supplied by the :mod:`weakref` module are an alternative, using weak -references to construct mappings that don't keep objects alive solely because -they appear in the mapping objects. If, for example, an image object is a value -in a :class:`WeakValueDictionary`, then when the last remaining references to -that image object are the weak references held by weak mappings, garbage -collection can reclaim the object, and its corresponding entries in weak -mappings are simply deleted. +dictionaries. The :class:`WeakKeyDictionary`, :class:`WeakValueDictionary` +and :class:`WeakSet` classes supplied by the :mod:`weakref` module are an +alternative, using weak references to construct mappings that don't keep objects +alive solely because they appear in the container objects. +If, for example, an image object is a value in a :class:`WeakValueDictionary`, +then when the last remaining references to that image object are the weak +references held by weak mappings, garbage collection can reclaim the object, +and its corresponding entries in weak mappings are simply deleted. :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` use weak references in their implementation, setting up callback functions on the weak references that notify the weak dictionaries when a key or value has been reclaimed by -garbage collection. Most programs should find that using one of these weak -dictionary types is all they need -- it's not usually necessary to create your -own weak references directly. The low-level machinery used by the weak -dictionary implementations is exposed by the :mod:`weakref` module for the -benefit of advanced uses. +garbage collection. :class:`WeakSet` implements the :class:`set` interface, +but keeps weak references to its elements, just like a +:class:`WeakKeyDictionary` does. + +Most programs should find that using one of these weak container types is all +they need -- it's not usually necessary to create your own weak references +directly. The low-level machinery used by the weak dictionary implementations +is exposed by the :mod:`weakref` module for the benefit of advanced uses. Not all objects can be weakly referenced; those objects which can include class instances, functions written in Python (but not in C), methods (both bound and @@ -179,6 +182,12 @@ Return a list of weak references to the values. +.. class:: WeakSet([elements]) + + Set class that keeps weak references to its elements. An element will be + discarded when no strong reference to it exists any more. + + .. data:: ReferenceType The type object for weak references objects. Modified: python/branches/py3k/Lib/abc.py ============================================================================== --- python/branches/py3k/Lib/abc.py (original) +++ python/branches/py3k/Lib/abc.py Tue Oct 23 08:26:46 2007 @@ -3,6 +3,7 @@ """Abstract Base Classes (ABCs) according to PEP 3119.""" +from weakref import WeakSet def abstractmethod(funcobj): """A decorator indicating abstract methods. @@ -130,9 +131,9 @@ abstracts.add(name) cls.__abstractmethods__ = abstracts # Set up inheritance registry - cls._abc_registry = set() - cls._abc_cache = set() - cls._abc_negative_cache = set() + cls._abc_registry = WeakSet() + cls._abc_cache = WeakSet() + cls._abc_negative_cache = WeakSet() cls._abc_negative_cache_version = ABCMeta._abc_invalidation_counter return cls @@ -172,7 +173,7 @@ # Check negative cache; may have to invalidate if cls._abc_negative_cache_version < ABCMeta._abc_invalidation_counter: # Invalidate the negative cache - cls._abc_negative_cache = set() + cls._abc_negative_cache = WeakSet() cls._abc_negative_cache_version = ABCMeta._abc_invalidation_counter elif subclass in cls._abc_negative_cache: return False Modified: python/branches/py3k/Lib/weakref.py ============================================================================== --- python/branches/py3k/Lib/weakref.py (original) +++ python/branches/py3k/Lib/weakref.py Tue Oct 23 08:26:46 2007 @@ -337,3 +337,108 @@ d[ref(key, self._remove)] = value if len(kwargs): self.update(kwargs) + + +class WeakSet: + def __init__(self, data=None): + self.data = set() + def _remove(item, selfref=ref(self)): + self = selfref() + if self is not None: + self.data.discard(item) + self._remove = _remove + if data is not None: + self.update(data) + + def __iter__(self): + for itemref in self.data: + item = itemref() + if item is not None: + yield item + + def __contains__(self, item): + return ref(item) in self.data + + def __reduce__(self): + return (self.__class__, (list(self),), + getattr(self, '__dict__', None)) + + def add(self, item): + self.data.add(ref(item, self._remove)) + + def clear(self): + self.data.clear() + + def copy(self): + return self.__class__(self) + + def pop(self): + while True: + itemref = self.data.pop() + item = itemref() + if item is not None: + return item + + def remove(self, item): + self.data.remove(ref(item)) + + def discard(self, item): + self.data.discard(ref(item)) + + def update(self, other): + if isinstance(other, self.__class__): + self.data.update(other.data) + else: + for element in other: + self.add(element) + __ior__ = update + + # Helper functions for simple delegating methods. + def _apply(self, other, method): + if not isinstance(other, self.__class__): + other = self.__class__(other) + newdata = method(other.data) + newset = self.__class__() + newset.data = newdata + return newset + + def _apply_mutate(self, other, method): + if not isinstance(other, self.__class__): + other = self.__class__(other) + method(other) + + def difference(self, other): + return self._apply(other, self.data.difference) + __sub__ = difference + + def difference_update(self, other): + self._apply_mutate(self, self.data.difference_update) + __isub__ = difference_update + + def intersection(self, other): + return self._apply(other, self.data.intersection) + __and__ = intersection + + def intersection_update(self, other): + self._apply_mutate(self, self.data.intersection_update) + __iand__ = intersection_update + + def issubset(self, other): + return self.data.issubset(ref(item) for item in other) + __lt__ = issubset + + def issuperset(self, other): + return self.data.issuperset(ref(item) for item in other) + __gt__ = issuperset + + def symmetric_difference(self, other): + return self._apply(other, self.data.symmetric_difference) + __xor__ = symmetric_difference + + def symmetric_difference_update(self, other): + self._apply_mutate(other, self.data.symmetric_difference_update) + __ixor__ = symmetric_difference_update + + def union(self, other): + self._apply_mutate(other, self.data.union) + __or__ = union Modified: python/branches/py3k/Modules/Setup.dist ============================================================================== --- python/branches/py3k/Modules/Setup.dist (original) +++ python/branches/py3k/Modules/Setup.dist Tue Oct 23 08:26:46 2007 @@ -116,6 +116,7 @@ _sre _sre.c # Fredrik Lundh's new regular expressions _codecs _codecsmodule.c # access to the builtin codecs and codec registry _fileio _fileio.c # Standard I/O baseline +_weakref _weakref.c # weak references # The zipimport module is always imported at startup. Having it as a # builtin module avoids some bootstrapping problems and reduces overhead. From python-3000-checkins at python.org Tue Oct 23 08:52:59 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 08:52:59 +0200 (CEST) Subject: [Python-3000-checkins] r58603 - in python/branches/py3k: Doc/library/stdtypes.rst Lib/test/test_unicode.py Objects/unicodeobject.c Message-ID: <20071023065300.0543B1E4012@bag.python.org> Author: georg.brandl Date: Tue Oct 23 08:52:59 2007 New Revision: 58603 Modified: python/branches/py3k/Doc/library/stdtypes.rst python/branches/py3k/Lib/test/test_unicode.py python/branches/py3k/Objects/unicodeobject.c Log: Patch #1071: Improve unicode.translate() so that you can pass unicode characters as mapping keys and invalid mapping keys are recognized and raise an error. Modified: python/branches/py3k/Doc/library/stdtypes.rst ============================================================================== --- python/branches/py3k/Doc/library/stdtypes.rst (original) +++ python/branches/py3k/Doc/library/stdtypes.rst Tue Oct 23 08:52:59 2007 @@ -931,7 +931,7 @@ Return a copy of the *s* where all characters have been mapped through the *map* which must be a dictionary of characters (strings of length 1) or Unicode ordinals (integers) to Unicode ordinals, strings or ``None``. - Unmapped characters are left untouched. Characters mapped to ``None`` are + Unmapped characters are left untouched. Characters mapped to ``None`` are deleted. .. note:: Modified: python/branches/py3k/Lib/test/test_unicode.py ============================================================================== --- python/branches/py3k/Lib/test/test_unicode.py (original) +++ python/branches/py3k/Lib/test/test_unicode.py Tue Oct 23 08:52:59 2007 @@ -160,12 +160,14 @@ self.checkequalnofix('bbbc', 'abababc', 'translate', {ord('a'):None}) self.checkequalnofix('iiic', 'abababc', 'translate', {ord('a'):None, ord('b'):ord('i')}) self.checkequalnofix('iiix', 'abababc', 'translate', {ord('a'):None, ord('b'):ord('i'), ord('c'):'x'}) - self.checkequalnofix('c', 'abababc', 'translate', {ord('a'):None, ord('b'):''}) + self.checkequalnofix('c', 'abababc', 'translate', {'a':None, 'b':''}) self.checkequalnofix('c', 'abababc', 'translate', {ord('a'):None, ord('b'):''}) self.checkequalnofix('xyyx', 'xzx', 'translate', {ord('z'):'yy'}) self.assertRaises(TypeError, 'hello'.translate) self.assertRaises(TypeError, 'abababc'.translate, 'abc', 'xyz') + self.assertRaises(ValueError, 'abababc'.translate, {'xy':2}) + self.assertRaises(TypeError, 'abababc'.translate, {(1,):2}) def test_split(self): string_tests.CommonTest.test_split(self) Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Tue Oct 23 08:52:59 2007 @@ -7810,10 +7810,54 @@ static PyObject* unicode_translate(PyUnicodeObject *self, PyObject *table) { - return PyUnicode_TranslateCharmap(self->str, - self->length, - table, - "ignore"); + PyObject *newtable = NULL; + Py_ssize_t i = 0; + PyObject *key, *value, *result; + + if (!PyDict_Check(table)) { + PyErr_SetString(PyExc_TypeError, "translate argument must be a dict"); + return NULL; + } + /* fixup the table -- allow size-1 string keys instead of only int keys */ + newtable = PyDict_Copy(table); + if (!newtable) return NULL; + while (PyDict_Next(table, &i, &key, &value)) { + if (PyUnicode_Check(key)) { + /* convert string keys to integer keys */ + PyObject *newkey; + int res; + if (PyUnicode_GET_SIZE(key) != 1) { + PyErr_SetString(PyExc_ValueError, "string items in translate " + "table must be 1 element long"); + goto err; + } + newkey = PyInt_FromLong(PyUnicode_AS_UNICODE(key)[0]); + if (!newkey) + goto err; + res = PyDict_SetItem(newtable, newkey, value); + Py_DECREF(newkey); + if (res < 0) + goto err; + } else if (PyInt_Check(key)) { + /* just keep integer keys */ + if (PyDict_SetItem(newtable, key, value) < 0) + goto err; + } else { + PyErr_SetString(PyExc_TypeError, "items in translate table must be " + "strings or integers"); + goto err; + } + } + + result = PyUnicode_TranslateCharmap(self->str, + self->length, + newtable, + "ignore"); + Py_DECREF(newtable); + return result; + err: + Py_DECREF(newtable); + return NULL; } PyDoc_STRVAR(upper__doc__, From python-3000-checkins at python.org Tue Oct 23 20:17:00 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 20:17:00 +0200 (CEST) Subject: [Python-3000-checkins] r58607 - python/branches/py3k/Doc/library/stdtypes.rst Message-ID: <20071023181700.9276E1E4018@bag.python.org> Author: georg.brandl Date: Tue Oct 23 20:17:00 2007 New Revision: 58607 Modified: python/branches/py3k/Doc/library/stdtypes.rst Log: Update w.r.t. PEP 3137. Modified: python/branches/py3k/Doc/library/stdtypes.rst ============================================================================== --- python/branches/py3k/Doc/library/stdtypes.rst (original) +++ python/branches/py3k/Doc/library/stdtypes.rst Tue Oct 23 20:17:00 2007 @@ -457,11 +457,11 @@ .. _typesseq: -Sequence Types --- :class:`str`, :class:`bytes`, :class:`list`, :class:`tuple`, :class:`buffer`, :class:`range` +Sequence Types --- :class:`str`, :class:`bytes`, :class:`buffer`, :class:`list`, :class:`tuple`, :class:`range` =============================================================================================================== -There are five sequence types: strings, byte sequences, lists, tuples, buffers, -and range objects. (For other containers see the built in :class:`dict`, +There are five sequence types: strings, byte sequences, buffers, lists, tuples, +and range objects. (For other containers see the built-in :class:`dict`, :class:`list`, :class:`set`, and :class:`tuple` classes, and the :mod:`collections` module.) @@ -469,31 +469,34 @@ object: sequence object: string object: bytes + object: buffer object: tuple object: list - object: buffer object: range -String literals are written in single or double quotes: ``'xyzzy'``, -``"frobozz"``. See :ref:`strings` for more about string literals. In addition -to the functionality described here, there are also string-specific methods -described in the :ref:`string-methods` section. Bytes objects can be -constructed from literals too; use a ``b`` prefix with normal string syntax: -``b'xyzzy'``. +Strings contain Unicode characters. Their literals are written in single or +double quotes: ``'xyzzy'``, ``"frobozz"``. See :ref:`strings` for more about +string literals. In addition to the functionality described here, there are +also string-specific methods described in the :ref:`string-methods` section. + +Bytes and buffer objects contain single bytes -- the former is immutable while +the latter is a mutable sequence. Bytes objects can be constructed from +literals too; use a ``b`` prefix with normal string syntax: ``b'xyzzy'``. +To construct buffer objects, use the :func:`buffer` function. .. warning:: While string objects are sequences of characters (represented by strings of - length 1), bytes objects are sequences of *integers* (between 0 and 255), - representing the ASCII value of single bytes. That means that for a bytes - object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytes - object of length 1. + length 1), bytes and buffer objects are sequences of *integers* (between 0 + and 255), representing the ASCII value of single bytes. That means that for + a bytes or buffer object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` + will be a bytes or buffer object of length 1. Also, while in previous Python versions, byte strings and Unicode strings could be exchanged for each other rather freely (barring encoding issues), - strings and bytes are completely separate concepts. There's no implicit - en-/decoding if you pass and object of the wrong type or try to e.g. compare - a string with a bytes object. + strings and bytes are now completely separate concepts. There's no implicit + en-/decoding if you pass and object of the wrong type. A string always + compares unequal to a bytes or buffer object. Lists are constructed with square brackets, separating items with commas: ``[a, b, c]``. Tuples are constructed by the comma operator (not within square @@ -501,10 +504,6 @@ the enclosing parentheses, such as ``a, b, c`` or ``()``. A single item tuple must have a trailing comma, such as ``(d,)``. -Buffer objects are not directly supported by Python syntax, but can be created -by calling the builtin function :func:`buffer`. They don't support -concatenation or repetition. - Objects of type range are similar to buffers in that there is no specific syntax to create them, but they are created using the :func:`range` function. They don't support slicing, concatenation or repetition, and using ``in``, ``not @@ -548,10 +547,10 @@ | ``max(s)`` | largest item of *s* | | +------------------+--------------------------------+----------+ -Sequence types also support comparisons. In particular, tuples and lists are -compared lexicographically by comparing corresponding elements. This means that +Sequence types also support comparisons. In particular, tuples and lists are +compared lexicographically by comparing corresponding elements. This means that to compare equal, every element must compare equal and the two sequences must be -of the same type and have the same length. (For full details see +of the same type and have the same length. (For full details see :ref:`comparisons` in the language reference.) .. index:: @@ -586,9 +585,9 @@ [[3], [3], [3]] What has happened is that ``[[]]`` is a one-element list containing an empty - list, so all three elements of ``[[]] * 3`` are (pointers to) this single empty - list. Modifying any of the elements of ``lists`` modifies this single list. - You can create a list of different lists this way:: + list, so all three elements of ``[[]] * 3`` are (pointers to) this single + empty list. Modifying any of the elements of ``lists`` modifies this single + list. You can create a list of different lists this way:: >>> lists = [[] for i in range(3)] >>> lists[0].append(3) @@ -599,8 +598,8 @@ (3) If *i* or *j* is negative, the index is relative to the end of the string: - ``len(s) + i`` or ``len(s) + j`` is substituted. But note that ``-0`` is still - ``0``. + ``len(s) + i`` or ``len(s) + j`` is substituted. But note that ``-0`` is + still ``0``. (4) The slice of *s* from *i* to *j* is defined as the sequence of items with index @@ -769,8 +768,8 @@ Return a string which is the concatenation of the values in the sequence *seq*. Non-string values in *seq* will be converted to a string using their - respective ``str()`` value. If there are any :class:`bytes` objects in - *seq*, a :exc:`TypeError` will be raised. The separator between elements is + respective ``str()`` value. If there are any :class:`bytes` objects in + *seq*, a :exc:`TypeError` will be raised. The separator between elements is the string providing this method. @@ -1160,16 +1159,16 @@ .. index:: triple: mutable; sequence; types object: list - object: bytes + object: buffer -List and bytes objects support additional operations that allow in-place +List and buffer objects support additional operations that allow in-place modification of the object. Other mutable sequence types (when added to the language) should also support these operations. Strings and tuples are immutable sequence types: such objects cannot be modified once created. The following operations are defined on mutable sequence types (where *x* is an arbitrary object). -Note that while lists allow their items to be of any type, bytes object +Note that while lists allow their items to be of any type, buffer object "items" are all integers in the range 0 <= x < 256. +------------------------------+--------------------------------+---------------------+ @@ -1263,7 +1262,7 @@ sequence. (7) - :meth:`sort` is not supported by bytes objects. + :meth:`sort` is not supported by buffer objects. The :meth:`sort` method takes optional arguments for controlling the comparisons. @@ -1297,51 +1296,34 @@ .. _bytes-methods: -Bytes Methods -------------- +Bytes and Buffer Methods +------------------------ .. index:: pair: bytes; methods + pair: buffer; methods -In addition to the operations on mutable sequence types (see -:ref:`typesseq-mutable`), bytes objects, being "mutable ASCII strings" have -further useful methods also found on strings. - -.. XXX "count" is documented as a mutable sequence method differently above -.. XXX perhaps just split bytes and list methods - -.. method:: bytes.count(sub[, start[, end]]) - - In contrast to the standard sequence ``count`` method, this returns the - number of occurrences of substring (not item) *sub* in the slice - ``[start:end]``. Optional arguments *start* and *end* are interpreted as in - slice notation. +Bytes and buffer objects, being "strings of bytes", have all methods found on +strings, with the exception of :func:`encode`, :func:`format` and +:func:`isidentifier`, which do not make sense with these types. Wherever one of +these methods needs to interpret the bytes as characters (e.g. the :func:`is...` +methods), the ASCII character set is assumed. +.. note:: -.. method:: bytes.decode([encoding[, errors]]) - - Decode the bytes using the codec registered for *encoding*. *encoding* - defaults to the default string encoding. *errors* may be given to set a - different error handling scheme. The default is ``'strict'``, meaning that - encoding errors raise :exc:`UnicodeError`. Other possible values are - ``'ignore'``, ``'replace'`` and any other name registered via - :func:`codecs.register_error`, see section :ref:`codec-base-classes`. + The methods on bytes and buffer objects don't accept strings as their + arguments, just as the methods on strings don't accept bytes as their + arguments. For example, you have to write :: + a = "abc" + b = a.replace("a", "f") -.. method:: bytes.endswith(suffix[, start[, end]]) + and :: - Return ``True`` if the bytes object ends with the specified *suffix*, - otherwise return ``False``. *suffix* can also be a tuple of suffixes to look - for. With optional *start*, test beginning at that position. With optional - *end*, stop comparing at that position. + a = b"abc" + b = a.replace(b"a", b"f") -.. method:: bytes.find(sub[, start[, end]]) - - Return the lowest index in the string where substring *sub* is found, such that - *sub* is contained in the range [*start*, *end*]. Optional arguments *start* - and *end* are interpreted as in slice notation. Return ``-1`` if *sub* is not - found. - +The bytes and buffer types have an additional class method: .. method:: bytes.fromhex(string) @@ -1354,113 +1336,9 @@ >>> bytes.fromhex('f0 f1f2 ') b'\xf0\xf1\xf2' +.. XXX verify/document translate() semantics! -.. method:: bytes.index(sub[, start[, end]]) - - Like :meth:`find`, but raise :exc:`ValueError` when the substring is not found. - - -.. method:: bytes.join(seq) - - Return a bytes object which is the concatenation of the bytes objects in the - sequence *seq*. The separator between elements is the bytes object providing - this method. - - -.. method:: bytes.lstrip(which) - - Return a copy of the bytes object with leading bytes removed. The *which* - argument is a bytes object specifying the set of bytes to be removed. As - with :meth:`str.lstrip`, the *which* argument is not a prefix; rather, all - combinations of its values are stripped. - - -.. method:: bytes.partition(sep) - - Split the bytes object at the first occurrence of *sep*, and return a 3-tuple - containing the part before the separator, the separator itself, and the part - after the separator. If the separator is not found, return a 3-tuple - containing the bytes object itself, followed by two empty strings. - - -.. method:: bytes.replace(old, new[, count]) - - Return a copy of the bytes object with all occurrences of substring *old* - replaced by *new*. If the optional argument *count* is given, only the first - *count* occurrences are replaced. - - -.. method:: bytes.rfind(sub[, start[, end]]) - - Return the highest index in the string where substring *sub* is found, such - that *sub* is contained within the slice ``[start:end]``. Optional arguments - *start* and *end* are interpreted as in slice notation. Return ``-1`` on - failure. - - -.. method:: bytes.rindex(sub[, start[, end]]) - - Like :meth:`rfind` but raises :exc:`ValueError` when the substring *sub* is - not found. - - -.. method:: bytes.rpartition(sep) - - Split the bytes object at the last occurrence of *sep*, and return a 3-tuple - containing the part before the separator, the separator itself, and the part - after the separator. If the separator is not found, return a 3-tuple - containing two empty strings, followed by the string itself. - - -.. method:: bytes.rsplit(sep[, maxsplit]) - - Return a list of substrings, using *sep* as the delimiter. If *maxsplit* is - given, at most *maxsplit* splits are done, the *rightmost* ones. Except for - splitting from the right, :meth:`rsplit` behaves like :meth:`split` which is - described in detail below. - - -.. method:: bytes.rstrip(which) - - Return a copy of the bytes object with trailing bytes removed. The *which* - argument is a bytes object specifying the set of bytes to be removed. As - with :meth:`str.rstrip`, The *chars* argument is not a suffix; rather, all - combinations of its values are stripped. - - -.. method:: bytes.split(sep[, maxsplit]) - - Return a list of substrings, using *sep* as the delimiter. If *maxsplit* is - given, at most *maxsplit* splits are done (thus, the list will have at most - ``maxsplit+1`` elements). If *maxsplit* is not specified, then there is no - limit on the number of splits (all possible splits are made). Consecutive - delimiters are not grouped together and are deemed to delimit empty strings - (for example, ``b'1,,2'.split(b',')`` returns ``[b'1', b'', b'2']``). The - *sep* argument may consist of multiple bytes (for example, ``b'1, 2, - 3'.split(b', ')`` returns ``[b'1', b'2', b'3']``). Splitting an empty string - with a specified separator returns ``[b'']``. - - -.. method:: bytes.startswith(prefix[, start[, end]]) - - Return ``True`` if the bytes object starts with the *prefix*, otherwise - return ``False``. *prefix* can also be a tuple of prefixes to look for. - With optional *start*, test string beginning at that position. With optional - *end*, stop comparing string at that position. - - -.. method:: bytes.strip(which) - - Return a copy of the bytes object with leading and trailing bytes found in - *which* removed. The *which* argument is a bytes object specifying the set - of characters to be removed. The *which* argument is not a prefix or suffix; - rather, all combinations of its values are stripped:: - - >>> b'www.example.com'.strip(b'cmowz.') - b'example' - - -.. method:: bytes.translate(table[, delete]) + .. method:: bytes.translate(table[, delete]) Return a copy of the bytes object where all bytes occurring in the optional argument *delete* are removed, and the remaining bytes have been mapped From python-3000-checkins at python.org Tue Oct 23 20:21:42 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 20:21:42 +0200 (CEST) Subject: [Python-3000-checkins] r58610 - python/branches/py3k/Doc/Makefile Message-ID: <20071023182142.1917A1E401D@bag.python.org> Author: georg.brandl Date: Tue Oct 23 20:21:41 2007 New Revision: 58610 Modified: python/branches/py3k/Doc/Makefile Log: Update Pygments version from externals. Modified: python/branches/py3k/Doc/Makefile ============================================================================== --- python/branches/py3k/Doc/Makefile (original) +++ python/branches/py3k/Doc/Makefile Tue Oct 23 20:21:41 2007 @@ -29,7 +29,7 @@ fi @if [ ! -d tools/pygments ]; then \ echo "Checking out Pygments..."; \ - svn checkout $(SVNROOT)/external/Pygments-0.8.1/pygments tools/pygments; \ + svn checkout $(SVNROOT)/external/Pygments-0.9/pygments tools/pygments; \ fi update: checkout From python-3000-checkins at python.org Tue Oct 23 20:25:21 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 20:25:21 +0200 (CEST) Subject: [Python-3000-checkins] r58612 - python/branches/py3k/Objects/fileobject.c Message-ID: <20071023182521.467111E4153@bag.python.org> Author: georg.brandl Date: Tue Oct 23 20:25:20 2007 New Revision: 58612 Modified: python/branches/py3k/Objects/fileobject.c Log: Remove redundant PyInt/PyLong checks. Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Tue Oct 23 20:25:20 2007 @@ -206,10 +206,7 @@ int fd; PyObject *meth; - if (PyInt_Check(o)) { - fd = PyInt_AsLong(o); - } - else if (PyLong_Check(o)) { + if (PyLong_Check(o)) { fd = PyLong_AsLong(o); } else if ((meth = PyObject_GetAttrString(o, "fileno")) != NULL) @@ -219,11 +216,7 @@ if (fno == NULL) return -1; - if (PyInt_Check(fno)) { - fd = PyInt_AsLong(fno); - Py_DECREF(fno); - } - else if (PyLong_Check(fno)) { + if (PyLong_Check(fno)) { fd = PyLong_AsLong(fno); Py_DECREF(fno); } From python-3000-checkins at python.org Tue Oct 23 20:26:17 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 20:26:17 +0200 (CEST) Subject: [Python-3000-checkins] r58613 - python/branches/py3k/Doc/c-api/abstract.rst python/branches/py3k/Doc/c-api/concrete.rst Message-ID: <20071023182617.F21461E45C1@bag.python.org> Author: georg.brandl Date: Tue Oct 23 20:26:17 2007 New Revision: 58613 Modified: python/branches/py3k/Doc/c-api/abstract.rst python/branches/py3k/Doc/c-api/concrete.rst Log: Remove duplicate entry for PyObject_AsFileDescriptor. Modified: python/branches/py3k/Doc/c-api/abstract.rst ============================================================================== --- python/branches/py3k/Doc/c-api/abstract.rst (original) +++ python/branches/py3k/Doc/c-api/abstract.rst Tue Oct 23 20:26:17 2007 @@ -327,10 +327,7 @@ .. cfunction:: int PyObject_AsFileDescriptor(PyObject *o) - Derives a file-descriptor from a Python object. If the object is an integer or - long integer, its value is returned. If not, the object's :meth:`fileno` method - is called if it exists; the method must return an integer or long integer, which - is returned as the file descriptor value. Returns ``-1`` on failure. + Derives a file-descriptor from a Python object. .. cfunction:: PyObject* PyObject_Dir(PyObject *o) Modified: python/branches/py3k/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k/Doc/c-api/concrete.rst (original) +++ python/branches/py3k/Doc/c-api/concrete.rst Tue Oct 23 20:26:17 2007 @@ -2420,13 +2420,17 @@ .. warning:: Take care when you are mixing streams and descriptors! For more - information, see `GNU C Library + information, see `the GNU C Library docs `_. .. cfunction:: int PyObject_AsFileDescriptor(PyObject *p) - Return the file descriptor associated with *p* as an :ctype:`int`. + Return the file descriptor associated with *p* as an :ctype:`int`. If the + object is an integer or long integer, its value is returned. If not, the + object's :meth:`fileno` method is called if it exists; the method must return + an integer, which is returned as the file descriptor value. Sets an + exception and returns ``-1`` on failure. .. cfunction:: PyObject* PyFile_GetLine(PyObject *p, int n) From python-3000-checkins at python.org Tue Oct 23 21:24:23 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 23 Oct 2007 21:24:23 +0200 (CEST) Subject: [Python-3000-checkins] r58617 - in python/branches/py3k: Modules/_ctypes/_ctypes.c Modules/_ctypes/cfield.c Modules/_sqlite/row.c Modules/_sqlite/statement.c Modules/cjkcodecs/multibytecodec.c Objects/sliceobject.c Python/getargs.c Message-ID: <20071023192423.67A351E4018@bag.python.org> Author: georg.brandl Date: Tue Oct 23 21:24:22 2007 New Revision: 58617 Modified: python/branches/py3k/Modules/_ctypes/_ctypes.c python/branches/py3k/Modules/_ctypes/cfield.c python/branches/py3k/Modules/_sqlite/row.c python/branches/py3k/Modules/_sqlite/statement.c python/branches/py3k/Modules/cjkcodecs/multibytecodec.c python/branches/py3k/Objects/sliceobject.c python/branches/py3k/Python/getargs.c Log: #1316: remove redundant PyLong_Check calls when PyInt_Check was already called. Modified: python/branches/py3k/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/_ctypes.c (original) +++ python/branches/py3k/Modules/_ctypes/_ctypes.c Tue Oct 23 21:24:22 2007 @@ -236,7 +236,7 @@ CDataType_from_address(PyObject *type, PyObject *value) { void *buf; - if (!PyInt_Check(value) && !PyLong_Check(value)) { + if (!PyInt_Check(value)) { PyErr_SetString(PyExc_TypeError, "integer expected"); return NULL; @@ -265,7 +265,7 @@ obj = PyObject_GetAttrString(dll, "_handle"); if (!obj) return NULL; - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!PyInt_Check(obj)) { PyErr_SetString(PyExc_TypeError, "the _handle attribute of the second argument must be an integer"); Py_DECREF(obj); @@ -1233,7 +1233,7 @@ } /* Should probably allow buffer interface as well */ /* int, long */ - if (PyInt_Check(value) || PyLong_Check(value)) { + if (PyInt_Check(value)) { PyCArgObject *parg; struct fielddesc *fd = getentry("P"); @@ -2697,7 +2697,7 @@ _get_name(PyObject *obj, char **pname) { #ifdef MS_WIN32 - if (PyInt_Check(obj) || PyLong_Check(obj)) { + if (PyInt_Check(obj)) { /* We have to use MAKEINTRESOURCEA for Windows CE. Works on Windows as well, of course. */ @@ -2734,7 +2734,7 @@ obj = PyObject_GetAttrString(dll, "_handle"); if (!obj) return NULL; - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!PyInt_Check(obj)) { PyErr_SetString(PyExc_TypeError, "the _handle attribute of the second argument must be an integer"); Py_DECREF(obj); @@ -2859,8 +2859,7 @@ #endif if (1 == PyTuple_GET_SIZE(args) - && (PyInt_Check(PyTuple_GET_ITEM(args, 0)) - || PyLong_Check(PyTuple_GET_ITEM(args, 0)))) { + && (PyInt_Check(PyTuple_GET_ITEM(args, 0)))) { CDataObject *ob; void *ptr = PyLong_AsVoidPtr(PyTuple_GET_ITEM(args, 0)); if (ptr == NULL) Modified: python/branches/py3k/Modules/_ctypes/cfield.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/cfield.c (original) +++ python/branches/py3k/Modules/_ctypes/cfield.c Tue Oct 23 21:24:22 2007 @@ -329,7 +329,7 @@ get_long(PyObject *v, long *p) { long x; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!PyInt_Check(v)) { PyErr_Format(PyExc_TypeError, "int expected instead of %s instance", v->ob_type->tp_name); @@ -348,7 +348,7 @@ get_ulong(PyObject *v, unsigned long *p) { unsigned long x; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!PyInt_Check(v)) { PyErr_Format(PyExc_TypeError, "int expected instead of %s instance", v->ob_type->tp_name); @@ -369,7 +369,7 @@ get_longlong(PyObject *v, PY_LONG_LONG *p) { PY_LONG_LONG x; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!PyInt_Check(v)) { PyErr_Format(PyExc_TypeError, "int expected instead of %s instance", v->ob_type->tp_name); @@ -388,7 +388,7 @@ get_ulonglong(PyObject *v, unsigned PY_LONG_LONG *p) { unsigned PY_LONG_LONG x; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!PyInt_Check(v)) { PyErr_Format(PyExc_TypeError, "int expected instead of %s instance", v->ob_type->tp_name); @@ -1373,7 +1373,7 @@ assert(PyBytes_Check(str)); *(char **)ptr = PyBytes_AS_STRING(str); return str; - } else if (PyInt_Check(value) || PyLong_Check(value)) { + } else if (PyInt_Check(value)) { #if SIZEOF_VOID_P == SIZEOF_LONG_LONG *(char **)ptr = (char *)PyInt_AsUnsignedLongLongMask(value); #else Modified: python/branches/py3k/Modules/_sqlite/row.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/row.c (original) +++ python/branches/py3k/Modules/_sqlite/row.c Tue Oct 23 21:24:22 2007 @@ -76,12 +76,7 @@ PyObject* item; - if (PyInt_Check(idx)) { - _idx = PyInt_AsLong(idx); - item = PyTuple_GetItem(self->data, _idx); - Py_XINCREF(item); - return item; - } else if (PyLong_Check(idx)) { + if (PyLong_Check(idx)) { _idx = PyLong_AsLong(idx); item = PyTuple_GetItem(self->data, _idx); Py_XINCREF(item); Modified: python/branches/py3k/Modules/_sqlite/statement.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/statement.c (original) +++ python/branches/py3k/Modules/_sqlite/statement.c Tue Oct 23 21:24:22 2007 @@ -80,9 +80,10 @@ int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter) { int rc = SQLITE_OK; - long longval; #ifdef HAVE_LONG_LONG PY_LONG_LONG longlongval; +#else + long longval; #endif const char* buffer; char* string; @@ -91,14 +92,16 @@ if (parameter == Py_None) { rc = sqlite3_bind_null(self->st, pos); - } else if (PyInt_CheckExact(parameter)) { - longval = PyInt_AsLong(parameter); - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longval); #ifdef HAVE_LONG_LONG } else if (PyLong_Check(parameter)) { longlongval = PyLong_AsLongLong(parameter); /* in the overflow error case, longlongval is -1, and an exception is set */ rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); +#else + } else if (PyLong_Check(parameter)) { + longval = PyLong_AsLong(parameter); + /* in the overflow error case, longval is -1, and an exception is set */ + rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longval); #endif } else if (PyFloat_Check(parameter)) { rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); Modified: python/branches/py3k/Modules/cjkcodecs/multibytecodec.c ============================================================================== --- python/branches/py3k/Modules/cjkcodecs/multibytecodec.c (original) +++ python/branches/py3k/Modules/cjkcodecs/multibytecodec.c Tue Oct 23 21:24:22 2007 @@ -313,8 +313,7 @@ if (!PyTuple_Check(retobj) || PyTuple_GET_SIZE(retobj) != 2 || !PyUnicode_Check((tobj = PyTuple_GET_ITEM(retobj, 0))) || - !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) || - PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) { + !PyInt_Check(PyTuple_GET_ITEM(retobj, 1))) { PyErr_SetString(PyExc_TypeError, "encoding error handler must return " "(unicode, int) tuple"); @@ -433,8 +432,7 @@ if (!PyTuple_Check(retobj) || PyTuple_GET_SIZE(retobj) != 2 || !PyUnicode_Check((retuni = PyTuple_GET_ITEM(retobj, 0))) || - !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) || - PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) { + !PyInt_Check(PyTuple_GET_ITEM(retobj, 1))) { PyErr_SetString(PyExc_TypeError, "decoding error handler must return " "(unicode, int) tuple"); Modified: python/branches/py3k/Objects/sliceobject.c ============================================================================== --- python/branches/py3k/Objects/sliceobject.c (original) +++ python/branches/py3k/Objects/sliceobject.c Tue Oct 23 21:24:22 2007 @@ -112,14 +112,14 @@ if (r->start == Py_None) { *start = *step < 0 ? length-1 : 0; } else { - if (!PyInt_Check(r->start) && !PyLong_Check(r->step)) return -1; + if (!PyInt_Check(r->start)) return -1; *start = PyInt_AsSsize_t(r->start); if (*start < 0) *start += length; } if (r->stop == Py_None) { *stop = *step < 0 ? -1 : length; } else { - if (!PyInt_Check(r->stop) && !PyLong_Check(r->step)) return -1; + if (!PyInt_Check(r->stop)) return -1; *stop = PyInt_AsSsize_t(r->stop); if (*stop < 0) *stop += length; } Modified: python/branches/py3k/Python/getargs.c ============================================================================== --- python/branches/py3k/Python/getargs.c (original) +++ python/branches/py3k/Python/getargs.c Tue Oct 23 21:24:22 2007 @@ -696,9 +696,7 @@ case 'k': { /* long sized bitfield */ unsigned long *p = va_arg(*p_va, unsigned long *); unsigned long ival; - if (PyInt_Check(arg)) - ival = PyInt_AsUnsignedLongMask(arg); - else if (PyLong_Check(arg)) + if (PyLong_Check(arg)) ival = PyLong_AsUnsignedLongMask(arg); else return converterr("integer", arg, msgbuf, bufsize); From python-3000-checkins at python.org Tue Oct 23 21:43:28 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 23 Oct 2007 21:43:28 +0200 (CEST) Subject: [Python-3000-checkins] r58619 - python/branches/py3k/Modules/timemodule.c Message-ID: <20071023194328.B29951E4018@bag.python.org> Author: guido.van.rossum Date: Tue Oct 23 21:43:28 2007 New Revision: 58619 Modified: python/branches/py3k/Modules/timemodule.c Log: Patch #1314 by Christian Heimes. Fix failing unittests for time and strptime on German and probably other localized Windows installations. Modified: python/branches/py3k/Modules/timemodule.c ============================================================================== --- python/branches/py3k/Modules/timemodule.c (original) +++ python/branches/py3k/Modules/timemodule.c Tue Oct 23 21:43:28 2007 @@ -528,7 +528,8 @@ e.g. an empty format, or %Z when the timezone is unknown. */ PyObject *ret; - ret = PyUnicode_FromStringAndSize(outbuf, buflen); + ret = PyUnicode_Decode(outbuf, buflen, + TZNAME_ENCODING, NULL); PyMem_Free(outbuf); return ret; } @@ -1035,5 +1036,3 @@ return 0; } - - From python-3000-checkins at python.org Wed Oct 24 20:41:20 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 24 Oct 2007 20:41:20 +0200 (CEST) Subject: [Python-3000-checkins] r58639 - in python/branches/py3k: Lib/ctypes/__init__.py Lib/ctypes/test/test_arrays.py Lib/ctypes/test/test_buffers.py Lib/ctypes/test/test_libc.py Lib/ctypes/test/test_memfunctions.py Lib/ctypes/test/test_python_api.py Lib/ctypes/test/test_returnfuncptrs.py Lib/ctypes/test/test_slicing.py Lib/ctypes/test/test_strings.py Lib/ctypes/test/test_unicode.py Modules/_ctypes/_ctypes.c Modules/_ctypes/callbacks.c Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Message-ID: <20071024184120.3B7861E4008@bag.python.org> Author: guido.van.rossum Date: Wed Oct 24 20:41:19 2007 New Revision: 58639 Modified: python/branches/py3k/Lib/ctypes/__init__.py python/branches/py3k/Lib/ctypes/test/test_arrays.py python/branches/py3k/Lib/ctypes/test/test_buffers.py python/branches/py3k/Lib/ctypes/test/test_libc.py python/branches/py3k/Lib/ctypes/test/test_memfunctions.py python/branches/py3k/Lib/ctypes/test/test_python_api.py python/branches/py3k/Lib/ctypes/test/test_returnfuncptrs.py python/branches/py3k/Lib/ctypes/test/test_slicing.py python/branches/py3k/Lib/ctypes/test/test_strings.py python/branches/py3k/Lib/ctypes/test/test_unicode.py python/branches/py3k/Modules/_ctypes/_ctypes.c python/branches/py3k/Modules/_ctypes/callbacks.c python/branches/py3k/Modules/_ctypes/callproc.c python/branches/py3k/Modules/_ctypes/cfield.c Log: Patch #1318 by Amaury Forgeot d'Arc. Updates to ctypes for python 3.0 to make the tests pass. Notable changes are: - return bytes instead of str8 - integers in range(256) are accepted as "one char string": libc.strchr("abcdef", 98) is now valid. - directly use the wide-char version of the win32 function LoadLibrary. Modified: python/branches/py3k/Lib/ctypes/__init__.py ============================================================================== --- python/branches/py3k/Lib/ctypes/__init__.py (original) +++ python/branches/py3k/Lib/ctypes/__init__.py Wed Oct 24 20:41:19 2007 @@ -52,11 +52,11 @@ """ def create_string_buffer(init, size=None): - """create_string_buffer(aString) -> character array + """create_string_buffer(aBytes) -> character array create_string_buffer(anInteger) -> character array create_string_buffer(aString, anInteger) -> character array """ - if isinstance(init, str): + if isinstance(init, (str, bytes)): if size is None: size = len(init)+1 buftype = c_char * size Modified: python/branches/py3k/Lib/ctypes/test/test_arrays.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_arrays.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_arrays.py Wed Oct 24 20:41:19 2007 @@ -94,12 +94,12 @@ # Failed with 0.9.8, reported by JUrner p = create_string_buffer("foo") sz = (c_char * 3).from_address(addressof(p)) - self.failUnlessEqual(sz[:], "foo") - self.failUnlessEqual(sz[::], "foo") - self.failUnlessEqual(sz[::-1], "oof") - self.failUnlessEqual(sz[::3], "f") - self.failUnlessEqual(sz[1:4:2], "o") - self.failUnlessEqual(sz.value, "foo") + self.failUnlessEqual(sz[:], b"foo") + self.failUnlessEqual(sz[::], b"foo") + self.failUnlessEqual(sz[::-1], b"oof") + self.failUnlessEqual(sz[::3], b"f") + self.failUnlessEqual(sz[1:4:2], b"o") + self.failUnlessEqual(sz.value, b"foo") try: create_unicode_buffer Modified: python/branches/py3k/Lib/ctypes/test/test_buffers.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_buffers.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_buffers.py Wed Oct 24 20:41:19 2007 @@ -14,11 +14,11 @@ self.failUnlessEqual(sizeof(b), 4 * sizeof(c_char)) self.failUnless(type(b[0]) is bytes) self.failUnlessEqual(b[0], b"a") - self.failUnlessEqual(b[:], "abc\0") - self.failUnlessEqual(b[::], "abc\0") - self.failUnlessEqual(b[::-1], "\0cba") - self.failUnlessEqual(b[::2], "ac") - self.failUnlessEqual(b[::5], "a") + self.failUnlessEqual(b[:], b"abc\0") + self.failUnlessEqual(b[::], b"abc\0") + self.failUnlessEqual(b[::-1], b"\0cba") + self.failUnlessEqual(b[::2], b"ac") + self.failUnlessEqual(b[::5], b"a") def test_string_conversion(self): b = create_string_buffer("abc") @@ -26,11 +26,11 @@ self.failUnlessEqual(sizeof(b), 4 * sizeof(c_char)) self.failUnless(type(b[0]) is bytes) self.failUnlessEqual(b[0], b"a") - self.failUnlessEqual(b[:], "abc\0") - self.failUnlessEqual(b[::], "abc\0") - self.failUnlessEqual(b[::-1], "\0cba") - self.failUnlessEqual(b[::2], "ac") - self.failUnlessEqual(b[::5], "a") + self.failUnlessEqual(b[:], b"abc\0") + self.failUnlessEqual(b[::], b"abc\0") + self.failUnlessEqual(b[::-1], b"\0cba") + self.failUnlessEqual(b[::2], b"ac") + self.failUnlessEqual(b[::5], b"a") try: c_wchar Modified: python/branches/py3k/Lib/ctypes/test/test_libc.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_libc.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_libc.py Wed Oct 24 20:41:19 2007 @@ -24,7 +24,7 @@ chars = create_string_buffer("spam, spam, and spam") lib.my_qsort(chars, len(chars)-1, sizeof(c_char), comparefunc(sort)) - self.failUnlessEqual(chars.raw, " ,,aaaadmmmnpppsss\x00") + self.failUnlessEqual(chars.raw, b" ,,aaaadmmmnpppsss\x00") if __name__ == "__main__": unittest.main() Modified: python/branches/py3k/Lib/ctypes/test/test_memfunctions.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_memfunctions.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_memfunctions.py Wed Oct 24 20:41:19 2007 @@ -9,21 +9,21 @@ a = create_string_buffer(1000000) p = b"Hello, World" result = memmove(a, p, len(p)) - self.failUnlessEqual(a.value, "Hello, World") + self.failUnlessEqual(a.value, b"Hello, World") - self.failUnlessEqual(string_at(result), "Hello, World") - self.failUnlessEqual(string_at(result, 5), "Hello") - self.failUnlessEqual(string_at(result, 16), "Hello, World\0\0\0\0") - self.failUnlessEqual(string_at(result, 0), "") + self.failUnlessEqual(string_at(result), b"Hello, World") + self.failUnlessEqual(string_at(result, 5), b"Hello") + self.failUnlessEqual(string_at(result, 16), b"Hello, World\0\0\0\0") + self.failUnlessEqual(string_at(result, 0), b"") def test_memset(self): a = create_string_buffer(1000000) result = memset(a, ord('x'), 16) - self.failUnlessEqual(a.value, "xxxxxxxxxxxxxxxx") + self.failUnlessEqual(a.value, b"xxxxxxxxxxxxxxxx") - self.failUnlessEqual(string_at(result), "xxxxxxxxxxxxxxxx") - self.failUnlessEqual(string_at(a), "xxxxxxxxxxxxxxxx") - self.failUnlessEqual(string_at(a, 20), "xxxxxxxxxxxxxxxx\0\0\0\0") + self.failUnlessEqual(string_at(result), b"xxxxxxxxxxxxxxxx") + self.failUnlessEqual(string_at(a), b"xxxxxxxxxxxxxxxx") + self.failUnlessEqual(string_at(a, 20), b"xxxxxxxxxxxxxxxx\0\0\0\0") def test_cast(self): a = (c_ubyte * 32)(*map(ord, "abcdef")) @@ -46,8 +46,8 @@ self.failUnlessEqual(2, sys.getrefcount(s)) self.failUnless(s, "foo bar") - self.failUnlessEqual(string_at(b"foo bar", 7), "foo bar") - self.failUnlessEqual(string_at(b"foo bar", 3), "foo") + self.failUnlessEqual(string_at(b"foo bar", 7), b"foo bar") + self.failUnlessEqual(string_at(b"foo bar", 3), b"foo") try: create_unicode_buffer Modified: python/branches/py3k/Lib/ctypes/test/test_python_api.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_python_api.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_python_api.py Wed Oct 24 20:41:19 2007 @@ -23,13 +23,13 @@ PyString_FromStringAndSize.restype = py_object PyString_FromStringAndSize.argtypes = c_char_p, c_py_ssize_t - self.failUnlessEqual(PyString_FromStringAndSize("abcdefghi", 3), "abc") + self.failUnlessEqual(PyString_FromStringAndSize(b"abcdefghi", 3), b"abc") def test_PyString_FromString(self): pythonapi.PyString_FromString.restype = py_object pythonapi.PyString_FromString.argtypes = (c_char_p,) - s = "abc" + s = b"abc" refcnt = grc(s) pyob = pythonapi.PyString_FromString(s) self.failUnlessEqual(grc(s), refcnt) @@ -73,10 +73,10 @@ buf = c_buffer(256) PyOS_snprintf(buf, sizeof(buf), "Hello from %s", b"ctypes") - self.failUnlessEqual(buf.value, "Hello from ctypes") + self.failUnlessEqual(buf.value, b"Hello from ctypes") PyOS_snprintf(buf, sizeof(buf), "Hello from %s (%d, %d, %d)", b"ctypes", 1, 2, 3) - self.failUnlessEqual(buf.value, "Hello from ctypes (1, 2, 3)") + self.failUnlessEqual(buf.value, b"Hello from ctypes (1, 2, 3)") # not enough arguments self.failUnlessRaises(TypeError, PyOS_snprintf, buf) Modified: python/branches/py3k/Lib/ctypes/test/test_returnfuncptrs.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_returnfuncptrs.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_returnfuncptrs.py Wed Oct 24 20:41:19 2007 @@ -14,7 +14,9 @@ strchr = get_strchr() self.failUnlessEqual(strchr("abcdef", "b"), "bcdef") self.failUnlessEqual(strchr("abcdef", "x"), None) - self.assertRaises(ArgumentError, strchr, "abcdef", 3) + self.failUnlessEqual(strchr("abcdef", 98), "bcdef") + self.failUnlessEqual(strchr("abcdef", 107), None) + self.assertRaises(ArgumentError, strchr, "abcdef", 3.0) self.assertRaises(TypeError, strchr, "abcdef") def test_without_prototype(self): @@ -28,7 +30,7 @@ strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(addr) self.failUnless(strchr("abcdef", "b"), "bcdef") self.failUnlessEqual(strchr("abcdef", "x"), None) - self.assertRaises(ArgumentError, strchr, "abcdef", 3) + self.assertRaises(ArgumentError, strchr, "abcdef", 3.0) self.assertRaises(TypeError, strchr, "abcdef") if __name__ == "__main__": Modified: python/branches/py3k/Lib/ctypes/test/test_slicing.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_slicing.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_slicing.py Wed Oct 24 20:41:19 2007 @@ -121,7 +121,7 @@ def test_char_array(self): - s = "abcdefghijklmnopqrstuvwxyz\0" + s = b"abcdefghijklmnopqrstuvwxyz\0" p = (c_char * 27)(*s) self.failUnlessEqual(p[:], s) Modified: python/branches/py3k/Lib/ctypes/test/test_strings.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_strings.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_strings.py Wed Oct 24 20:41:19 2007 @@ -6,20 +6,20 @@ BUF = c_char * 4 buf = BUF("a", "b", "c") - self.failUnlessEqual(buf.value, "abc") - self.failUnlessEqual(buf.raw, "abc\000") + self.failUnlessEqual(buf.value, b"abc") + self.failUnlessEqual(buf.raw, b"abc\000") buf.value = "ABCD" - self.failUnlessEqual(buf.value, "ABCD") - self.failUnlessEqual(buf.raw, "ABCD") + self.failUnlessEqual(buf.value, b"ABCD") + self.failUnlessEqual(buf.raw, b"ABCD") buf.value = "x" - self.failUnlessEqual(buf.value, "x") - self.failUnlessEqual(buf.raw, "x\000CD") + self.failUnlessEqual(buf.value, b"x") + self.failUnlessEqual(buf.raw, b"x\000CD") buf[1] = "Z" - self.failUnlessEqual(buf.value, "xZCD") - self.failUnlessEqual(buf.raw, "xZCD") + self.failUnlessEqual(buf.value, b"xZCD") + self.failUnlessEqual(buf.raw, b"xZCD") self.assertRaises(ValueError, setattr, buf, "value", "aaaaaaaa") self.assertRaises(TypeError, setattr, buf, "value", 42) @@ -27,8 +27,8 @@ def test_c_buffer_value(self): buf = c_buffer(32) - buf.value = "Hello, World" - self.failUnlessEqual(buf.value, "Hello, World") + buf.value = b"Hello, World" + self.failUnlessEqual(buf.value, b"Hello, World") self.failUnlessRaises(TypeError, setattr, buf, "value", memoryview(b"Hello, World")) self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) @@ -38,7 +38,7 @@ buf = c_buffer(32) buf.raw = memoryview(b"Hello, World") - self.failUnlessEqual(buf.value, "Hello, World") + self.failUnlessEqual(buf.value, b"Hello, World") self.assertRaises(TypeError, setattr, buf, "value", memoryview(b"abc")) self.assertRaises(ValueError, setattr, buf, "raw", memoryview(b"x" * 100)) Modified: python/branches/py3k/Lib/ctypes/test/test_unicode.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_unicode.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_unicode.py Wed Oct 24 20:41:19 2007 @@ -111,18 +111,18 @@ ctypes.set_conversion_mode("ascii", "replace") buf = ctypes.create_string_buffer("ab\xe4\xf6\xfc") - self.failUnlessEqual(buf[:], "ab???\0") - self.failUnlessEqual(buf[::], "ab???\0") - self.failUnlessEqual(buf[::-1], "\0???ba") - self.failUnlessEqual(buf[::2], "a??") - self.failUnlessEqual(buf[6:5:-1], "") + self.failUnlessEqual(buf[:], b"ab???\0") + self.failUnlessEqual(buf[::], b"ab???\0") + self.failUnlessEqual(buf[::-1], b"\0???ba") + self.failUnlessEqual(buf[::2], b"a??") + self.failUnlessEqual(buf[6:5:-1], b"") ctypes.set_conversion_mode("ascii", "ignore") buf = ctypes.create_string_buffer("ab\xe4\xf6\xfc") # is that correct? not sure. But with 'ignore', you get what you pay for.. - self.failUnlessEqual(buf[:], "ab\0\0\0\0") - self.failUnlessEqual(buf[::], "ab\0\0\0\0") - self.failUnlessEqual(buf[::-1], "\0\0\0\0ba") + self.failUnlessEqual(buf[:], b"ab\0\0\0\0") + self.failUnlessEqual(buf[::], b"ab\0\0\0\0") + self.failUnlessEqual(buf[::-1], b"\0\0\0\0ba") if __name__ == '__main__': unittest.main() Modified: python/branches/py3k/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/_ctypes.c (original) +++ python/branches/py3k/Modules/_ctypes/_ctypes.c Wed Oct 24 20:41:19 2007 @@ -763,7 +763,7 @@ static PyObject * CharArray_get_raw(CDataObject *self) { - return PyString_FromStringAndSize(self->b_ptr, self->b_size); + return PyBytes_FromStringAndSize(self->b_ptr, self->b_size); } static PyObject * @@ -774,7 +774,7 @@ for (i = 0; i < self->b_size; ++i) if (*ptr++ == '\0') break; - return PyString_FromStringAndSize(self->b_ptr, i); + return PyBytes_FromStringAndSize(self->b_ptr, i); } static int @@ -1251,7 +1251,7 @@ } /* XXX struni: remove later */ /* string */ - if (PyString_Check(value)) { + if (PyBytes_Check(value)) { PyCArgObject *parg; struct fielddesc *fd = getentry("z"); @@ -1452,7 +1452,7 @@ dict = PyObject_stgdict((PyObject *)self); assert(dict); /* Cannot be NULL for CDataObject instances */ - fmt = PyString_AsString(dict->proto); + fmt = PyUnicode_AsString(dict->proto); assert(fmt); fd = getentry(fmt); @@ -1644,7 +1644,7 @@ assert(dict); /* I think we can rely on this being a one-character string */ - fmt = PyString_AsString(dict->proto); + fmt = PyUnicode_AsString(dict->proto); assert(fmt); fd = getentry(fmt); @@ -2667,7 +2667,7 @@ char *name; PyObject *defval; PyObject *typ; - if (!PyArg_ParseTuple(item, "i|zO", &flag, &name, &defval)) { + if (!PyArg_ParseTuple(item, "i|ZO", &flag, &name, &defval)) { PyErr_SetString(PyExc_TypeError, "paramflags must be a sequence of (int [,string [,value]]) tuples"); return 0; @@ -2705,8 +2705,12 @@ return 1; } #endif - if (PyString_Check(obj) || PyUnicode_Check(obj)) { - *pname = PyString_AsString(obj); + if (PyBytes_Check(obj)) { + *pname = PyBytes_AS_STRING(obj); + return *pname ? 1 : 0; + } + if (PyUnicode_Check(obj)) { + *pname = PyUnicode_AsString(obj); return *pname ? 1 : 0; } PyErr_SetString(PyExc_TypeError, @@ -2966,7 +2970,7 @@ } static PyObject * -_get_arg(int *pindex, char *name, PyObject *defval, PyObject *inargs, PyObject *kwds) +_get_arg(int *pindex, PyObject *name, PyObject *defval, PyObject *inargs, PyObject *kwds) { PyObject *v; @@ -2976,7 +2980,7 @@ Py_INCREF(v); return v; } - if (kwds && (v = PyDict_GetItemString(kwds, name))) { + if (kwds && (v = PyDict_GetItem(kwds, name))) { ++*pindex; Py_INCREF(v); return v; @@ -3057,15 +3061,15 @@ PyObject *item = PyTuple_GET_ITEM(paramflags, i); PyObject *ob; int flag; - char *name = NULL; + PyObject *name = NULL; PyObject *defval = NULL; /* This way seems to be ~2 us faster than the PyArg_ParseTuple calls below. */ - /* We HAVE already checked that the tuple can be parsed with "i|zO", so... */ + /* We HAVE already checked that the tuple can be parsed with "i|ZO", so... */ Py_ssize_t tsize = PyTuple_GET_SIZE(item); flag = PyInt_AS_LONG(PyTuple_GET_ITEM(item, 0)); - name = tsize > 1 ? PyString_AS_STRING(PyTuple_GET_ITEM(item, 1)) : NULL; + name = tsize > 1 ? PyTuple_GET_ITEM(item, 1) : NULL; defval = tsize > 2 ? PyTuple_GET_ITEM(item, 2) : NULL; switch (flag & (PARAMFLAG_FIN | PARAMFLAG_FOUT | PARAMFLAG_FLCID)) { @@ -3730,10 +3734,10 @@ char *dest; if (slicelen <= 0) - return PyString_FromString(""); + return PyBytes_FromStringAndSize("", 0); if (step == 1) { - return PyString_FromStringAndSize(ptr + start, - slicelen); + return PyBytes_FromStringAndSize(ptr + start, + slicelen); } dest = (char *)PyMem_Malloc(slicelen); @@ -3745,7 +3749,7 @@ dest[i] = ptr[cur]; } - np = PyString_FromStringAndSize(dest, slicelen); + np = PyBytes_FromStringAndSize(dest, slicelen); PyMem_Free(dest); return np; } @@ -4407,10 +4411,10 @@ char *dest; if (len <= 0) - return PyString_FromString(""); + return PyBytes_FromStringAndSize("", 0); if (step == 1) { - return PyString_FromStringAndSize(ptr + start, - len); + return PyBytes_FromStringAndSize(ptr + start, + len); } dest = (char *)PyMem_Malloc(len); if (dest == NULL) @@ -4418,7 +4422,7 @@ for (cur = start, i = 0; i < len; cur += step, i++) { dest[i] = ptr[cur]; } - np = PyString_FromStringAndSize(dest, len); + np = PyBytes_FromStringAndSize(dest, len); PyMem_Free(dest); return np; } @@ -4629,7 +4633,7 @@ ++methods; } - s = PyString_FromString(comerror_doc); + s = PyUnicode_FromString(comerror_doc); if (s == NULL) return -1; status = PyDict_SetItemString(dict, "__doc__", s); @@ -4654,8 +4658,8 @@ string_at(const char *ptr, int size) { if (size == -1) - return PyString_FromString(ptr); - return PyString_FromStringAndSize(ptr, size); + return PyBytes_FromStringAndSize(ptr, strlen(ptr)); + return PyBytes_FromStringAndSize(ptr, size); } static int Modified: python/branches/py3k/Modules/_ctypes/callbacks.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/callbacks.c (original) +++ python/branches/py3k/Modules/_ctypes/callbacks.c Wed Oct 24 20:41:19 2007 @@ -365,7 +365,7 @@ static PyObject *context; if (context == NULL) - context = PyString_FromString("_ctypes.DllGetClassObject"); + context = PyUnicode_FromString("_ctypes.DllGetClassObject"); mod = PyImport_ImportModule("ctypes"); if (!mod) { @@ -444,7 +444,7 @@ static PyObject *context; if (context == NULL) - context = PyString_FromString("_ctypes.DllCanUnloadNow"); + context = PyUnicode_FromString("_ctypes.DllCanUnloadNow"); mod = PyImport_ImportModule("ctypes"); if (!mod) { Modified: python/branches/py3k/Modules/_ctypes/callproc.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/callproc.c (original) +++ python/branches/py3k/Modules/_ctypes/callproc.c Wed Oct 24 20:41:19 2007 @@ -770,7 +770,7 @@ PyObject *tp, *v, *tb, *s, *cls_str, *msg_str; va_start(vargs, fmt); - s = PyString_FromFormatV(fmt, vargs); + s = PyUnicode_FromFormatV(fmt, vargs); va_end(vargs); if (!s) return; @@ -779,18 +779,18 @@ PyErr_NormalizeException(&tp, &v, &tb); cls_str = PyObject_Str(tp); if (cls_str) { - PyString_ConcatAndDel(&s, cls_str); - PyString_ConcatAndDel(&s, PyString_FromString(": ")); + PyUnicode_AppendAndDel(&s, cls_str); + PyUnicode_AppendAndDel(&s, PyUnicode_FromString(": ")); if (s == NULL) goto error; } else PyErr_Clear(); - msg_str = PyObject_Str(v); + msg_str = PyObject_Unicode(v); if (msg_str) - PyString_ConcatAndDel(&s, msg_str); + PyUnicode_AppendAndDel(&s, msg_str); else { PyErr_Clear(); - PyString_ConcatAndDel(&s, PyString_FromString("???")); + PyUnicode_AppendAndDel(&s, PyUnicode_FromString("???")); if (s == NULL) goto error; } @@ -1087,34 +1087,18 @@ module.\n"; static PyObject *load_library(PyObject *self, PyObject *args) { - TCHAR *name; + WCHAR *name; PyObject *nameobj; PyObject *ignored; HMODULE hMod; if (!PyArg_ParseTuple(args, "O|O:LoadLibrary", &nameobj, &ignored)) return NULL; -#ifdef _UNICODE - name = alloca((PyString_Size(nameobj) + 1) * sizeof(WCHAR)); - if (!name) { - PyErr_NoMemory(); - return NULL; - } - { - int r; - char *aname = PyString_AsString(nameobj); - if(!aname) - return NULL; - r = MultiByteToWideChar(CP_ACP, 0, aname, -1, name, PyString_Size(nameobj) + 1); - name[r] = 0; - } -#else - name = PyString_AsString(nameobj); - if(!name) + name = PyUnicode_AsUnicode(nameobj); + if (!name) return NULL; -#endif - hMod = LoadLibrary(name); + hMod = LoadLibraryW(name); if (!hMod) return PyErr_SetFromWindowsErr(GetLastError()); #ifdef _WIN64 Modified: python/branches/py3k/Modules/_ctypes/cfield.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/cfield.c (original) +++ python/branches/py3k/Modules/_ctypes/cfield.c Wed Oct 24 20:41:19 2007 @@ -1169,6 +1169,14 @@ *(char *)ptr = PyBytes_AsString(value)[0]; _RET(value); } + if (PyInt_Check(value)) + { + long longval = PyInt_AS_LONG(value); + if (longval < 0 || longval >= 256) + goto error; + *(char *)ptr = (char)longval; + _RET(value); + } error: PyErr_Format(PyExc_TypeError, "one character string expected"); From python-3000-checkins at python.org Wed Oct 24 20:55:38 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Wed, 24 Oct 2007 20:55:38 +0200 (CEST) Subject: [Python-3000-checkins] r58640 - in python/branches/py3k: Lib/modulefinder.py Lib/pickletools.py Lib/struct.py Lib/test/test_builtin.py Lib/test/test_bytes.py Lib/test/test_codeccallbacks.py Lib/test/test_compile.py Lib/test/test_io.py Lib/test/test_locale.py Lib/test/test_struct.py Lib/test/test_sys.py Lib/test/test_unicode.py Lib/test/test_unicodedata.py Lib/test/testcodec.py Objects/stringobject.c Message-ID: <20071024185538.51E651E400F@bag.python.org> Author: georg.brandl Date: Wed Oct 24 20:55:37 2007 New Revision: 58640 Modified: python/branches/py3k/Lib/modulefinder.py python/branches/py3k/Lib/pickletools.py python/branches/py3k/Lib/struct.py python/branches/py3k/Lib/test/test_builtin.py python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Lib/test/test_codeccallbacks.py python/branches/py3k/Lib/test/test_compile.py python/branches/py3k/Lib/test/test_io.py python/branches/py3k/Lib/test/test_locale.py python/branches/py3k/Lib/test/test_struct.py python/branches/py3k/Lib/test/test_sys.py python/branches/py3k/Lib/test/test_unicode.py python/branches/py3k/Lib/test/test_unicodedata.py python/branches/py3k/Lib/test/testcodec.py python/branches/py3k/Objects/stringobject.c Log: Patch #1303: Adapt str8 constructor to bytes (now buffer) one. Modified: python/branches/py3k/Lib/modulefinder.py ============================================================================== --- python/branches/py3k/Lib/modulefinder.py (original) +++ python/branches/py3k/Lib/modulefinder.py Wed Oct 24 20:55:37 2007 @@ -17,12 +17,12 @@ READ_MODE = "r" # XXX Clean up once str8's cstor matches bytes. -LOAD_CONST = str8(chr(dis.opname.index('LOAD_CONST'))) -IMPORT_NAME = str8(chr(dis.opname.index('IMPORT_NAME'))) -STORE_NAME = str8(chr(dis.opname.index('STORE_NAME'))) -STORE_GLOBAL = str8(chr(dis.opname.index('STORE_GLOBAL'))) +LOAD_CONST = str8([dis.opname.index('LOAD_CONST')]) +IMPORT_NAME = str8([dis.opname.index('IMPORT_NAME')]) +STORE_NAME = str8([dis.opname.index('STORE_NAME')]) +STORE_GLOBAL = str8([dis.opname.index('STORE_GLOBAL')]) STORE_OPS = [STORE_NAME, STORE_GLOBAL] -HAVE_ARGUMENT = str8(chr(dis.HAVE_ARGUMENT)) +HAVE_ARGUMENT = str8([dis.HAVE_ARGUMENT]) # Modulefinder does a good job at simulating Python's, but it can not # handle __path__ modifications packages make at runtime. Therefore there @@ -368,7 +368,7 @@ consts = co.co_consts LOAD_LOAD_AND_IMPORT = LOAD_CONST + LOAD_CONST + IMPORT_NAME while code: - c = str8(chr(code[0])) + c = str8([code[0]]) if c in STORE_OPS: oparg, = unpack('>> import pickle ->>> x = [1, 2, (3, 4), {str8('abc'): "def"}] +>>> x = [1, 2, (3, 4), {str8(b'abc'): "def"}] >>> pkl = pickle.dumps(x, 0) >>> dis(pkl) 0: ( MARK Modified: python/branches/py3k/Lib/struct.py ============================================================================== --- python/branches/py3k/Lib/struct.py (original) +++ python/branches/py3k/Lib/struct.py Wed Oct 24 20:55:37 2007 @@ -36,7 +36,7 @@ class Struct(_Struct): def __init__(self, fmt): if isinstance(fmt, str): - fmt = str8(fmt) + fmt = str8(fmt, 'latin1') _Struct.__init__(self, fmt) _MAXCACHE = 100 Modified: python/branches/py3k/Lib/test/test_builtin.py ============================================================================== --- python/branches/py3k/Lib/test/test_builtin.py (original) +++ python/branches/py3k/Lib/test/test_builtin.py Wed Oct 24 20:55:37 2007 @@ -580,7 +580,8 @@ self.assertEqual(hash(1), hash(1)) self.assertEqual(hash(1), hash(1.0)) hash('spam') - self.assertEqual(hash('spam'), hash(str8('spam'))) + self.assertEqual(hash('spam'), hash(str8(b'spam'))) # remove str8() + # when b"" is immutable hash((0,1,2,3)) def f(): pass self.assertRaises(TypeError, hash, []) Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Wed Oct 24 20:55:37 2007 @@ -103,33 +103,33 @@ self.failIf(b3 <= b2) def test_compare_to_str(self): - self.assertEqual(b"abc" == str8("abc"), True) - self.assertEqual(b"ab" != str8("abc"), True) - self.assertEqual(b"ab" <= str8("abc"), True) - self.assertEqual(b"ab" < str8("abc"), True) - self.assertEqual(b"abc" >= str8("ab"), True) - self.assertEqual(b"abc" > str8("ab"), True) - - self.assertEqual(b"abc" != str8("abc"), False) - self.assertEqual(b"ab" == str8("abc"), False) - self.assertEqual(b"ab" > str8("abc"), False) - self.assertEqual(b"ab" >= str8("abc"), False) - self.assertEqual(b"abc" < str8("ab"), False) - self.assertEqual(b"abc" <= str8("ab"), False) - - self.assertEqual(str8("abc") == b"abc", True) - self.assertEqual(str8("ab") != b"abc", True) - self.assertEqual(str8("ab") <= b"abc", True) - self.assertEqual(str8("ab") < b"abc", True) - self.assertEqual(str8("abc") >= b"ab", True) - self.assertEqual(str8("abc") > b"ab", True) - - self.assertEqual(str8("abc") != b"abc", False) - self.assertEqual(str8("ab") == b"abc", False) - self.assertEqual(str8("ab") > b"abc", False) - self.assertEqual(str8("ab") >= b"abc", False) - self.assertEqual(str8("abc") < b"ab", False) - self.assertEqual(str8("abc") <= b"ab", False) + self.assertEqual(b"abc" == str8(b"abc"), True) + self.assertEqual(b"ab" != str8(b"abc"), True) + self.assertEqual(b"ab" <= str8(b"abc"), True) + self.assertEqual(b"ab" < str8(b"abc"), True) + self.assertEqual(b"abc" >= str8(b"ab"), True) + self.assertEqual(b"abc" > str8(b"ab"), True) + + self.assertEqual(b"abc" != str8(b"abc"), False) + self.assertEqual(b"ab" == str8(b"abc"), False) + self.assertEqual(b"ab" > str8(b"abc"), False) + self.assertEqual(b"ab" >= str8(b"abc"), False) + self.assertEqual(b"abc" < str8(b"ab"), False) + self.assertEqual(b"abc" <= str8(b"ab"), False) + + self.assertEqual(str8(b"abc") == b"abc", True) + self.assertEqual(str8(b"ab") != b"abc", True) + self.assertEqual(str8(b"ab") <= b"abc", True) + self.assertEqual(str8(b"ab") < b"abc", True) + self.assertEqual(str8(b"abc") >= b"ab", True) + self.assertEqual(str8(b"abc") > b"ab", True) + + self.assertEqual(str8(b"abc") != b"abc", False) + self.assertEqual(str8(b"ab") == b"abc", False) + self.assertEqual(str8(b"ab") > b"abc", False) + self.assertEqual(str8(b"ab") >= b"abc", False) + self.assertEqual(str8(b"abc") < b"ab", False) + self.assertEqual(str8(b"abc") <= b"ab", False) # Byte comparisons with unicode should always fail! # Test this for all expected byte orders and Unicode character sizes @@ -345,7 +345,7 @@ self.assertEqual(b.decode("utf8", "ignore"), "Hello world\n") def test_from_buffer(self): - sample = str8("Hello world\n\x80\x81\xfe\xff") + sample = str8(b"Hello world\n\x80\x81\xfe\xff") buf = memoryview(sample) b = bytes(buf) self.assertEqual(b, bytes(sample)) @@ -367,8 +367,8 @@ b1 = b"abc" b2 = b"def" self.assertEqual(b1 + b2, b"abcdef") - self.assertEqual(b1 + str8("def"), b"abcdef") - self.assertEqual(str8("def") + b1, b"defabc") + self.assertEqual(b1 + str8(b"def"), b"abcdef") + self.assertEqual(str8(b"def") + b1, b"defabc") self.assertRaises(TypeError, lambda: b1 + "def") self.assertRaises(TypeError, lambda: "abc" + b2) @@ -391,7 +391,7 @@ self.assertEqual(b, b"abcdef") self.assertEqual(b, b1) self.failUnless(b is b1) - b += str8("xyz") + b += str8(b"xyz") self.assertEqual(b, b"abcdefxyz") try: b += "" Modified: python/branches/py3k/Lib/test/test_codeccallbacks.py ============================================================================== --- python/branches/py3k/Lib/test/test_codeccallbacks.py (original) +++ python/branches/py3k/Lib/test/test_codeccallbacks.py Wed Oct 24 20:55:37 2007 @@ -181,7 +181,7 @@ # mapped through the encoding again. This means, that # to be able to use e.g. the "replace" handler, the # charmap has to have a mapping for "?". - charmap = dict((ord(c), str8(2*c.upper())) for c in "abcdefgh") + charmap = dict((ord(c), str8(2*c.upper(), 'ascii')) for c in "abcdefgh") sin = "abc" sout = b"AABBCC" self.assertEquals(codecs.charmap_encode(sin, "strict", charmap)[0], sout) @@ -189,7 +189,7 @@ sin = "abcA" self.assertRaises(UnicodeError, codecs.charmap_encode, sin, "strict", charmap) - charmap[ord("?")] = str8("XYZ") + charmap[ord("?")] = str8(b"XYZ") sin = "abcDEF" sout = b"AABBCCXYZXYZXYZ" self.assertEquals(codecs.charmap_encode(sin, "replace", charmap)[0], sout) @@ -309,7 +309,7 @@ # check with one argument too much self.assertRaises(TypeError, exctype, *(args + ["too much"])) # check with one argument of the wrong type - wrongargs = [ "spam", str8("eggs"), b"spam", 42, 1.0, None ] + wrongargs = [ "spam", str8(b"eggs"), b"spam", 42, 1.0, None ] for i in range(len(args)): for wrongarg in wrongargs: if type(wrongarg) is type(args[i]): Modified: python/branches/py3k/Lib/test/test_compile.py ============================================================================== --- python/branches/py3k/Lib/test/test_compile.py (original) +++ python/branches/py3k/Lib/test/test_compile.py Wed Oct 24 20:55:37 2007 @@ -157,7 +157,7 @@ s256 = "".join(["\n"] * 256 + ["spam"]) co = compile(s256, 'fn', 'exec') self.assertEqual(co.co_firstlineno, 257) - self.assertEqual(co.co_lnotab, str8('')) + self.assertEqual(co.co_lnotab, str8()) def test_literals_with_leading_zeroes(self): for arg in ["077787", "0xj", "0x.", "0e", "090000000000000", Modified: python/branches/py3k/Lib/test/test_io.py ============================================================================== --- python/branches/py3k/Lib/test/test_io.py (original) +++ python/branches/py3k/Lib/test/test_io.py Wed Oct 24 20:55:37 2007 @@ -88,7 +88,7 @@ self.assertEqual(f.tell(), 6) self.assertEqual(f.seek(-1, 1), 5) self.assertEqual(f.tell(), 5) - self.assertEqual(f.write(str8(" world\n\n\n")), 9) + self.assertEqual(f.write(str8(b" world\n\n\n")), 9) self.assertEqual(f.seek(0), 0) self.assertEqual(f.write(b"h"), 1) self.assertEqual(f.seek(-1, 2), 13) Modified: python/branches/py3k/Lib/test/test_locale.py ============================================================================== --- python/branches/py3k/Lib/test/test_locale.py (original) +++ python/branches/py3k/Lib/test/test_locale.py Wed Oct 24 20:55:37 2007 @@ -82,7 +82,7 @@ # Test BSD Rune locale's bug for isctype functions. def teststrop(s, method, output): - s = str8(s) + s = str8(s, 'latin1') # XXX if verbose: print("%s.%s() =? %s ..." % (repr(s), method, repr(output)), end=' ') result = getattr(s, method)() Modified: python/branches/py3k/Lib/test/test_struct.py ============================================================================== --- python/branches/py3k/Lib/test/test_struct.py (original) +++ python/branches/py3k/Lib/test/test_struct.py Wed Oct 24 20:55:37 2007 @@ -101,7 +101,7 @@ simple_err(struct.unpack, 'iii', s) simple_err(struct.unpack, 'i', s) -c = str8('a') +c = str8(b'a') b = 1 h = 255 i = 65535 @@ -186,7 +186,7 @@ if isinstance(arg, str): # Strings are returned as str8 since you can't know the encoding of # the string when packed. - arg = str8(arg) + arg = str8(arg, 'latin1') if rev != arg and not asy: raise TestFailed("unpack(%r, %r) -> (%r,) # expected (%r,)" % ( fmt, res, rev, arg)) @@ -428,14 +428,14 @@ def test_p_code(): for code, input, expected, expectedback in [ - ('p','abc', '\x00', str8('')), - ('1p', 'abc', '\x00', str8('')), - ('2p', 'abc', '\x01a', str8('a')), - ('3p', 'abc', '\x02ab', str8('ab')), - ('4p', 'abc', '\x03abc', str8('abc')), - ('5p', 'abc', '\x03abc\x00', str8('abc')), - ('6p', 'abc', '\x03abc\x00\x00', str8('abc')), - ('1000p', 'x'*1000, '\xff' + 'x'*999, str8('x'*255))]: + ('p','abc', '\x00', str8()), + ('1p', 'abc', '\x00', str8()), + ('2p', 'abc', '\x01a', str8(b'a')), + ('3p', 'abc', '\x02ab', str8(b'ab')), + ('4p', 'abc', '\x03abc', str8(b'abc')), + ('5p', 'abc', '\x03abc\x00', str8(b'abc')), + ('6p', 'abc', '\x03abc\x00\x00', str8(b'abc')), + ('1000p', 'x'*1000, '\xff' + 'x'*999, str8(b'x'*255))]: expected = bytes(expected, "latin-1") got = struct.pack(code, input) if got != expected: @@ -564,20 +564,24 @@ if verbose: print("test_unpack_from using", cls.__name__) data = cls(test_string) - vereq(s.unpack_from(data), (str8('abcd'),)) - vereq(s.unpack_from(data, 2), (str8('cd01'),)) - vereq(s.unpack_from(data, 4), (str8('0123'),)) + if not isinstance(data, (str8, bytes)): + bytes_data = str8(data, 'latin1') + else: + bytes_data = data + vereq(s.unpack_from(data), (str8(b'abcd'),)) + vereq(s.unpack_from(data, 2), (str8(b'cd01'),)) + vereq(s.unpack_from(data, 4), (str8(b'0123'),)) for i in range(6): - vereq(s.unpack_from(data, i), (str8(data[i:i+4]),)) + vereq(s.unpack_from(data, i), (bytes_data[i:i+4],)) for i in range(6, len(test_string) + 1): simple_err(s.unpack_from, data, i) for cls in (str, str8, bytes): # XXX + memoryview data = cls(test_string) - vereq(struct.unpack_from(fmt, data), (str8('abcd'),)) - vereq(struct.unpack_from(fmt, data, 2), (str8('cd01'),)) - vereq(struct.unpack_from(fmt, data, 4), (str8('0123'),)) + vereq(struct.unpack_from(fmt, data), (str8(b'abcd'),)) + vereq(struct.unpack_from(fmt, data, 2), (str8(b'cd01'),)) + vereq(struct.unpack_from(fmt, data, 4), (str8(b'0123'),)) for i in range(6): - vereq(struct.unpack_from(fmt, data, i), (str8(data[i:i+4]),)) + vereq(struct.unpack_from(fmt, data, i), (bytes_data[i:i+4],)) for i in range(6, len(test_string) + 1): simple_err(struct.unpack_from, fmt, data, i) Modified: python/branches/py3k/Lib/test/test_sys.py ============================================================================== --- python/branches/py3k/Lib/test/test_sys.py (original) +++ python/branches/py3k/Lib/test/test_sys.py Wed Oct 24 20:55:37 2007 @@ -300,7 +300,7 @@ def test_intern(self): self.assertRaises(TypeError, sys.intern) - s = str8("never interned before") + s = str8(b"never interned before") self.assert_(sys.intern(s) is s) s2 = s.swapcase().swapcase() self.assert_(sys.intern(s2) is s) @@ -314,7 +314,7 @@ def __hash__(self): return 123 - self.assertRaises(TypeError, sys.intern, S("abc")) + self.assertRaises(TypeError, sys.intern, S(b"abc")) s = "never interned as unicode before" self.assert_(sys.intern(s) is s) Modified: python/branches/py3k/Lib/test/test_unicode.py ============================================================================== --- python/branches/py3k/Lib/test/test_unicode.py (original) +++ python/branches/py3k/Lib/test/test_unicode.py Wed Oct 24 20:55:37 2007 @@ -203,8 +203,8 @@ self.assertRaises(TypeError, 'replace'.replace, "r", 42) def test_str8_comparison(self): - self.assertEqual('abc' == str8('abc'), False) - self.assertEqual('abc' != str8('abc'), True) + self.assertEqual('abc' == str8(b'abc'), False) + self.assertEqual('abc' != str8(b'abc'), True) def test_comparison(self): # Comparisons: Modified: python/branches/py3k/Lib/test/test_unicodedata.py ============================================================================== --- python/branches/py3k/Lib/test/test_unicodedata.py (original) +++ python/branches/py3k/Lib/test/test_unicodedata.py Wed Oct 24 20:55:37 2007 @@ -176,7 +176,7 @@ def test_east_asian_width(self): eaw = self.db.east_asian_width - self.assertRaises(TypeError, eaw, str8('a')) + self.assertRaises(TypeError, eaw, str8(b'a')) self.assertRaises(TypeError, eaw, '') self.assertRaises(TypeError, eaw, 'ra') self.assertEqual(eaw('\x1e'), 'N') Modified: python/branches/py3k/Lib/test/testcodec.py ============================================================================== --- python/branches/py3k/Lib/test/testcodec.py (original) +++ python/branches/py3k/Lib/test/testcodec.py Wed Oct 24 20:55:37 2007 @@ -36,7 +36,7 @@ decoding_map = codecs.make_identity_dict(range(256)) decoding_map.update({ 0x78: "abc", # 1-n decoding mapping - str8("abc"): 0x0078,# 1-n encoding mapping + str8(b"abc"): 0x0078,# 1-n encoding mapping 0x01: None, # decoding mapping to 0x79: "", # decoding mapping to }) Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Wed Oct 24 20:55:37 2007 @@ -3020,16 +3020,151 @@ static PyObject * string_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *x = NULL; - static char *kwlist[] = {"object", 0}; + PyObject *x = NULL, *it; + PyObject *(*iternext)(PyObject *); + const char *encoding = NULL; + const char *errors = NULL; + PyObject *new = NULL; + Py_ssize_t i, size; + static char *kwlist[] = {"object", "encoding", "errors", 0}; if (type != &PyString_Type) return str_subtype_new(type, args, kwds); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O:str8", kwlist, &x)) + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|Oss:str8", kwlist, &x, + &encoding, &errors)) return NULL; - if (x == NULL) + if (x == NULL) { + if (encoding != NULL || errors != NULL) { + PyErr_SetString(PyExc_TypeError, + "encoding or errors without sequence " + "argument"); + return NULL; + } return PyString_FromString(""); - return PyObject_Str(x); + } + + if (PyUnicode_Check(x)) { + /* Encode via the codec registry */ + if (encoding == NULL) { + PyErr_SetString(PyExc_TypeError, + "string argument without an encoding"); + return NULL; + } + new = PyCodec_Encode(x, encoding, errors); + if (new == NULL) + return NULL; + /* XXX(gb): must accept bytes here since codecs output bytes + at the moment */ + if (PyBytes_Check(new)) { + PyObject *str; + str = PyString_FromString(PyBytes_AsString(new)); + Py_DECREF(new); + if (!str) + return NULL; + return str; + } + if (!PyString_Check(new)) { + PyErr_Format(PyExc_TypeError, + "encoder did not return a str8 " + "object (type=%.400s)", + Py_Type(new)->tp_name); + Py_DECREF(new); + return NULL; + } + return new; + } + + /* If it's not unicode, there can't be encoding or errors */ + if (encoding != NULL || errors != NULL) { + PyErr_SetString(PyExc_TypeError, + "encoding or errors without a string argument"); + return NULL; + } + + /* Use the modern buffer interface */ + if (PyObject_CheckBuffer(x)) { + Py_buffer view; + if (PyObject_GetBuffer(x, &view, PyBUF_FULL_RO) < 0) + return NULL; + new = PyString_FromStringAndSize(NULL, view.len); + if (!new) + goto fail; + // XXX(brett.cannon): Better way to get to internal buffer? + if (PyBuffer_ToContiguous(((PyStringObject *)new)->ob_sval, + &view, view.len, 'C') < 0) + goto fail; + PyObject_ReleaseBuffer(x, &view); + return new; + fail: + Py_XDECREF(new); + PyObject_ReleaseBuffer(x, &view); + return NULL; + } + + /* For the iterator version, create a string object and resize as needed. */ + /* XXX(gb): is 64 a good value? also, optimize this if length is known */ + size = 64; + new = PyString_FromStringAndSize(NULL, size); + if (new == NULL) + return NULL; + + /* XXX Optimize this if the arguments is a list, tuple */ + + /* Get the iterator */ + it = PyObject_GetIter(x); + if (it == NULL) + goto error; + // XXX(brett.cannon): No API for this? + iternext = *Py_Type(it)->tp_iternext; + + /* Run the iterator to exhaustion */ + for (i = 0; ; i++) { + PyObject *item; + Py_ssize_t value; + + /* Get the next item */ + item = iternext(it); + if (item == NULL) { + if (PyErr_Occurred()) { + if (!PyErr_ExceptionMatches(PyExc_StopIteration)) + goto error; + PyErr_Clear(); + } + break; + } + + /* Interpret it as an int (__index__) */ + value = PyNumber_AsSsize_t(item, PyExc_ValueError); + Py_DECREF(item); + if (value == -1 && PyErr_Occurred()) + goto error; + + /* Range check */ + if (value < 0 || value >= 256) { + PyErr_SetString(PyExc_ValueError, + "bytes must be in range(0, 256)"); + goto error; + } + + /* Append the byte */ + if (i >= size) { + size *= 2; + if (_PyString_Resize(&new, size) < 0) + goto error; + } + ((PyStringObject *)new)->ob_sval[i] = value; + } + _PyString_Resize(&new, i); + + /* Clean up and return success */ + Py_DECREF(it); + return new; + + error: + /* Error handling when it != NULL */ + Py_XDECREF(it); + Py_DECREF(new); + return NULL; } static PyObject * From python-3000-checkins at python.org Wed Oct 24 21:37:27 2007 From: python-3000-checkins at python.org (thomas.heller) Date: Wed, 24 Oct 2007 21:37:27 +0200 (CEST) Subject: [Python-3000-checkins] r58642 - in python/branches/py3k: Lib/ctypes/test/test_prototypes.py Modules/_ctypes/_ctypes.c Message-ID: <20071024193727.CF0A81E402C@bag.python.org> Author: thomas.heller Date: Wed Oct 24 21:37:27 2007 New Revision: 58642 Modified: python/branches/py3k/Lib/ctypes/test/test_prototypes.py python/branches/py3k/Modules/_ctypes/_ctypes.c Log: A 'PyObject *' parameter in PyErr_Format must use %S parameter, not %s. Added unittest for calling a function with paramflags. Modified: python/branches/py3k/Lib/ctypes/test/test_prototypes.py ============================================================================== --- python/branches/py3k/Lib/ctypes/test/test_prototypes.py (original) +++ python/branches/py3k/Lib/ctypes/test/test_prototypes.py Wed Oct 24 21:37:27 2007 @@ -48,6 +48,24 @@ func.restype = c_long func.argtypes = None + def test_paramflags(self): + # function returns c_void_p result, + # and has a required parameter named 'input' + prototype = CFUNCTYPE(c_void_p, c_void_p) + func = prototype(("_testfunc_p_p", testdll), + ((1, "input"),)) + + try: + func() + except TypeError as details: + self.failUnlessEqual(str(details), "required argument 'input' missing") + else: + self.fail("TypeError not raised") + + self.failUnlessEqual(func(None), None) + self.failUnlessEqual(func(input=None), None) + + def test_int_pointer_arg(self): func = testdll._testfunc_p_p func.restype = c_long Modified: python/branches/py3k/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/py3k/Modules/_ctypes/_ctypes.c (original) +++ python/branches/py3k/Modules/_ctypes/_ctypes.c Wed Oct 24 21:37:27 2007 @@ -2992,7 +2992,7 @@ /* we can't currently emit a better error message */ if (name) PyErr_Format(PyExc_TypeError, - "required argument '%s' missing", name); + "required argument '%S' missing", name); else PyErr_Format(PyExc_TypeError, "not enough arguments"); From python-3000-checkins at python.org Wed Oct 24 22:18:37 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 24 Oct 2007 22:18:37 +0200 (CEST) Subject: [Python-3000-checkins] r58646 - python/branches/py3k/Lib/test/test_locale.py Message-ID: <20071024201837.A1E281E4007@bag.python.org> Author: guido.van.rossum Date: Wed Oct 24 22:18:37 2007 New Revision: 58646 Modified: python/branches/py3k/Lib/test/test_locale.py Log: Make test_locale pass by removing tests that were designed to handle issues with on various platforms. We no longer use . Modified: python/branches/py3k/Lib/test/test_locale.py ============================================================================== --- python/branches/py3k/Lib/test/test_locale.py (original) +++ python/branches/py3k/Lib/test/test_locale.py Wed Oct 24 22:18:37 2007 @@ -78,42 +78,3 @@ finally: locale.setlocale(locale.LC_NUMERIC, oldlocale) - - -# Test BSD Rune locale's bug for isctype functions. -def teststrop(s, method, output): - s = str8(s, 'latin1') # XXX - if verbose: - print("%s.%s() =? %s ..." % (repr(s), method, repr(output)), end=' ') - result = getattr(s, method)() - if result != output: - if verbose: - print("no") - print("%s.%s() == %s != %s" % (repr(s), method, repr(result), - repr(output))) - elif verbose: - print("yes") - -try: - if sys.platform == 'sunos5': - # On Solaris, in en_US.UTF-8, \xa0 is a space - raise locale.Error - oldlocale = locale.setlocale(locale.LC_CTYPE) - locale.setlocale(locale.LC_CTYPE, 'en_US.UTF-8') -except locale.Error: - pass -else: - try: - teststrop('\x20', 'isspace', True) - teststrop('\xa0', 'isspace', False) - teststrop('\xa1', 'isspace', False) - teststrop('\xc0', 'isalpha', False) - teststrop('\xc0', 'isalnum', False) - teststrop('\xc0', 'isupper', False) - teststrop('\xc0', 'islower', False) - teststrop('\xec\xa0\xbc', 'split', ['\xec\xa0\xbc']) - teststrop('\xed\x95\xa0', 'strip', '\xed\x95\xa0') - teststrop('\xcc\x85', 'lower', '\xcc\x85') - teststrop('\xed\x95\xa0', 'upper', '\xed\x95\xa0') - finally: - locale.setlocale(locale.LC_CTYPE, oldlocale) From python-3000-checkins at python.org Wed Oct 24 23:13:10 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 24 Oct 2007 23:13:10 +0200 (CEST) Subject: [Python-3000-checkins] r58648 - python/branches/py3k/Objects/unicodeobject.c Message-ID: <20071024211310.025981E4F04@bag.python.org> Author: guido.van.rossum Date: Wed Oct 24 23:13:09 2007 New Revision: 58648 Modified: python/branches/py3k/Objects/unicodeobject.c Log: Fix a broken format in a PyErr_Format() call: %lx is not supported. (It's still technically broken since the va_args code assumes %x is an int while we're passing a long, but that's mostly theoretical, and it's done all over the place.) Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Wed Oct 24 23:13:09 2007 @@ -4703,7 +4703,7 @@ long max = PyUnicode_GetMax(); if (value < 0 || value > max) { PyErr_Format(PyExc_TypeError, - "character mapping must be in range(0x%lx)", max+1); + "character mapping must be in range(0x%x)", max+1); Py_DECREF(x); return -1; } From python-3000-checkins at python.org Wed Oct 24 23:14:08 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Wed, 24 Oct 2007 23:14:08 +0200 (CEST) Subject: [Python-3000-checkins] r58649 - python/branches/py3k/Objects/codeobject.c Message-ID: <20071024211408.153811E4F09@bag.python.org> Author: georg.brandl Date: Wed Oct 24 23:14:07 2007 New Revision: 58649 Modified: python/branches/py3k/Objects/codeobject.c Log: Fix a refleak for `filename', introduced in rev. 58466. Also remove an unnecessary incref/decref for `name'. Modified: python/branches/py3k/Objects/codeobject.c ============================================================================== --- python/branches/py3k/Objects/codeobject.c (original) +++ python/branches/py3k/Objects/codeobject.c Wed Oct 24 23:14:07 2007 @@ -66,9 +66,6 @@ PyErr_BadInternalCall(); return NULL; } - Py_INCREF(name); - Py_INCREF(filename); - intern_strings(names); intern_strings(varnames); intern_strings(freevars); @@ -110,7 +107,6 @@ co->co_lnotab = lnotab; co->co_zombieframe = NULL; } - Py_DECREF(name); return co; } From python-3000-checkins at python.org Wed Oct 24 23:25:35 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Wed, 24 Oct 2007 23:25:35 +0200 (CEST) Subject: [Python-3000-checkins] r58650 - python/branches/py3k/Lib/test/test_codeccallbacks.py Message-ID: <20071024212535.2EA4E1E4007@bag.python.org> Author: georg.brandl Date: Wed Oct 24 23:25:34 2007 New Revision: 58650 Modified: python/branches/py3k/Lib/test/test_codeccallbacks.py Log: Remove a test case which is no longer valid. Modified: python/branches/py3k/Lib/test/test_codeccallbacks.py ============================================================================== --- python/branches/py3k/Lib/test/test_codeccallbacks.py (original) +++ python/branches/py3k/Lib/test/test_codeccallbacks.py Wed Oct 24 23:25:34 2007 @@ -792,7 +792,7 @@ class D(dict): def __getitem__(self, key): raise ValueError - self.assertRaises(ValueError, "\xff".translate, D()) + #self.assertRaises(ValueError, "\xff".translate, D()) self.assertRaises(TypeError, "\xff".translate, {0xff: sys.maxunicode+1}) self.assertRaises(TypeError, "\xff".translate, {0xff: ()}) From guido at python.org Thu Oct 25 00:27:37 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Oct 2007 15:27:37 -0700 Subject: [Python-3000-checkins] r58650 - python/branches/py3k/Lib/test/test_codeccallbacks.py In-Reply-To: <20071024212535.2EA4E1E4007@bag.python.org> References: <20071024212535.2EA4E1E4007@bag.python.org> Message-ID: I was looking at this too. The root cause of the test failure is that unicode_translate() calls PyDict_Copy() on its argument dict, and then modifies the copy to add extra keys. This seems awfully expensive. Isn't there another way? (I'm imagining that often the dict is rather large and the string rather short...) --Guido 2007/10/24, georg.brandl : > Author: georg.brandl > Date: Wed Oct 24 23:25:34 2007 > New Revision: 58650 > > Modified: > python/branches/py3k/Lib/test/test_codeccallbacks.py > Log: > Remove a test case which is no longer valid. > > > Modified: python/branches/py3k/Lib/test/test_codeccallbacks.py > ============================================================================== > --- python/branches/py3k/Lib/test/test_codeccallbacks.py (original) > +++ python/branches/py3k/Lib/test/test_codeccallbacks.py Wed Oct 24 23:25:34 2007 > @@ -792,7 +792,7 @@ > class D(dict): > def __getitem__(self, key): > raise ValueError > - self.assertRaises(ValueError, "\xff".translate, D()) > + #self.assertRaises(ValueError, "\xff".translate, D()) > self.assertRaises(TypeError, "\xff".translate, {0xff: sys.maxunicode+1}) > self.assertRaises(TypeError, "\xff".translate, {0xff: ()}) > > _______________________________________________ > Python-3000-checkins mailing list > Python-3000-checkins at python.org > http://mail.python.org/mailman/listinfo/python-3000-checkins > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python-3000-checkins at python.org Fri Oct 26 01:18:52 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 01:18:52 +0200 (CEST) Subject: [Python-3000-checkins] r58657 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Lib/test/test_posix.py Misc/NEWS Modules/posixmodule.c Message-ID: <20071025231852.095D41E4017@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 01:18:51 2007 New Revision: 58657 Modified: python/branches/py3k/Doc/library/os.rst python/branches/py3k/Lib/test/test_os.py python/branches/py3k/Lib/test/test_posix.py python/branches/py3k/Misc/NEWS python/branches/py3k/Modules/posixmodule.c Log: Patch 1318 by Christian Heimes: remove os.tmpnam(), os.tempnam(), and os.tmpfile(). Modified: python/branches/py3k/Doc/library/os.rst ============================================================================== --- python/branches/py3k/Doc/library/os.rst (original) +++ python/branches/py3k/Doc/library/os.rst Fri Oct 26 01:18:51 2007 @@ -371,14 +371,6 @@ This function is obsolete. Use the :mod:`subprocess` module. -.. function:: tmpfile() - - Return a new file object opened in update mode (``w+b``). The file has no - directory entries associated with it and will be automatically deleted once - there are no file descriptors for the file. Availability: Macintosh, Unix, - Windows. - - .. _os-fd-ops: File Descriptor Operations @@ -1077,53 +1069,6 @@ Create a symbolic link pointing to *src* named *dst*. Availability: Unix. -.. function:: tempnam([dir[, prefix]]) - - Return a unique path name that is reasonable for creating a temporary file. - This will be an absolute path that names a potential directory entry in the - directory *dir* or a common location for temporary files if *dir* is omitted or - ``None``. If given and not ``None``, *prefix* is used to provide a short prefix - to the filename. Applications are responsible for properly creating and - managing files created using paths returned by :func:`tempnam`; no automatic - cleanup is provided. On Unix, the environment variable :envvar:`TMPDIR` - overrides *dir*, while on Windows the :envvar:`TMP` is used. The specific - behavior of this function depends on the C library implementation; some aspects - are underspecified in system documentation. - - .. warning:: - - Use of :func:`tempnam` is vulnerable to symlink attacks; consider using - :func:`tmpfile` (section :ref:`os-newstreams`) instead. - - Availability: Macintosh, Unix, Windows. - - -.. function:: tmpnam() - - Return a unique path name that is reasonable for creating a temporary file. - This will be an absolute path that names a potential directory entry in a common - location for temporary files. Applications are responsible for properly - creating and managing files created using paths returned by :func:`tmpnam`; no - automatic cleanup is provided. - - .. warning:: - - Use of :func:`tmpnam` is vulnerable to symlink attacks; consider using - :func:`tmpfile` (section :ref:`os-newstreams`) instead. - - Availability: Unix, Windows. This function probably shouldn't be used on - Windows, though: Microsoft's implementation of :func:`tmpnam` always creates a - name in the root directory of the current drive, and that's generally a poor - location for a temp file (depending on privileges, you may not even be able to - open a file using this name). - - -.. data:: TMP_MAX - - The maximum number of unique names that :func:`tmpnam` will generate before - reusing names. - - .. function:: unlink(path) Remove the file *path*. This is the same function as :func:`remove`; the Modified: python/branches/py3k/Lib/test/test_os.py ============================================================================== --- python/branches/py3k/Lib/test/test_os.py (original) +++ python/branches/py3k/Lib/test/test_os.py Fri Oct 26 01:18:51 2007 @@ -8,9 +8,6 @@ import sys from test import test_support -warnings.filterwarnings("ignore", "tempnam", RuntimeWarning, __name__) -warnings.filterwarnings("ignore", "tmpnam", RuntimeWarning, __name__) - # Tests creating TESTFN class FileTests(unittest.TestCase): def setUp(self): @@ -24,76 +21,6 @@ self.assert_(os.access(test_support.TESTFN, os.W_OK)) -class TemporaryFileTests(unittest.TestCase): - def setUp(self): - self.files = [] - os.mkdir(test_support.TESTFN) - - def tearDown(self): - for name in self.files: - os.unlink(name) - os.rmdir(test_support.TESTFN) - - def check_tempfile(self, name): - # make sure it doesn't already exist: - self.failIf(os.path.exists(name), - "file already exists for temporary file") - # make sure we can create the file - open(name, "w") - self.files.append(name) - - def test_tempnam(self): - if not hasattr(os, "tempnam"): - return - warnings.filterwarnings("ignore", "tempnam", RuntimeWarning, - r"test_os$") - self.check_tempfile(os.tempnam()) - - name = os.tempnam(test_support.TESTFN) - self.check_tempfile(name) - - name = os.tempnam(test_support.TESTFN, "pfx") - self.assertEqual(os.path.basename(name)[:3], "pfx") - self.check_tempfile(name) - - def test_tmpfile(self): - if not hasattr(os, "tmpfile"): - return - fp = os.tmpfile() - fp.write(b"foobar") - fp.seek(0) - s = fp.read() - fp.close() - self.assertEquals(s, b"foobar") - - def test_tmpnam(self): - import sys - if not hasattr(os, "tmpnam"): - return - warnings.filterwarnings("ignore", "tmpnam", RuntimeWarning, - r"test_os$") - name = os.tmpnam() - if sys.platform in ("win32",): - # The Windows tmpnam() seems useless. From the MS docs: - # - # The character string that tmpnam creates consists of - # the path prefix, defined by the entry P_tmpdir in the - # file STDIO.H, followed by a sequence consisting of the - # digit characters '0' through '9'; the numerical value - # of this string is in the range 1 - 65,535. Changing the - # definitions of L_tmpnam or P_tmpdir in STDIO.H does not - # change the operation of tmpnam. - # - # The really bizarre part is that, at least under MSVC6, - # P_tmpdir is "\\". That is, the path returned refers to - # the root of the current drive. That's a terrible place to - # put temp files, and, depending on privileges, the user - # may not even be able to open a file in the root directory. - self.failIf(os.path.exists(name), - "file already exists for temporary file") - else: - self.check_tempfile(name) - # Test attributes on return values from os.*stat* family. class StatAttributeTests(unittest.TestCase): def setUp(self): @@ -483,7 +410,6 @@ def test_main(): test_support.run_unittest( FileTests, - TemporaryFileTests, StatAttributeTests, EnvironTests, WalkTests, Modified: python/branches/py3k/Lib/test/test_posix.py ============================================================================== --- python/branches/py3k/Lib/test/test_posix.py (original) +++ python/branches/py3k/Lib/test/test_posix.py Fri Oct 26 01:18:51 2007 @@ -29,7 +29,7 @@ # test posix functions which take no arguments and have # no side-effects which we need to cleanup (e.g., fork, wait, abort) NO_ARG_FUNCTIONS = [ "ctermid", "getcwd", "getcwdu", "uname", - "times", "getloadavg", "tmpnam", + "times", "getloadavg", "getegid", "geteuid", "getgid", "getgroups", "getpid", "getpgrp", "getppid", "getuid", ] @@ -171,17 +171,6 @@ os.close(reader) os.close(writer) - def test_tempnam(self): - if hasattr(posix, 'tempnam'): - self.assert_(posix.tempnam()) - self.assert_(posix.tempnam(os.curdir)) - self.assert_(posix.tempnam(os.curdir, 'blah')) - - def test_tmpfile(self): - if hasattr(posix, 'tmpfile'): - fp = posix.tmpfile() - fp.close() - def test_utime(self): if hasattr(posix, 'utime'): now = time.time() Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Fri Oct 26 01:18:51 2007 @@ -39,6 +39,9 @@ argument was being ignored if __loader__ is defined and forcing the source to be UTF-8. +- The methods `os.tmpnam()`, `os.tempnam()` and `os.tmpfile()` have been + removed in favor of the tempfile module. + What's New in Python 3.0a1? ========================== Modified: python/branches/py3k/Modules/posixmodule.c ============================================================================== --- python/branches/py3k/Modules/posixmodule.c (original) +++ python/branches/py3k/Modules/posixmodule.c Fri Oct 26 01:18:51 2007 @@ -302,10 +302,6 @@ #define USE_CTERMID_R #endif -#if defined(HAVE_TMPNAM_R) && defined(WITH_THREAD) -#define USE_TMPNAM_R -#endif - /* choose the appropriate stat and fstat functions and return structs */ #undef STAT #if defined(MS_WIN64) || defined(MS_WINDOWS) @@ -5339,107 +5335,6 @@ } #endif /* HAVE_STATVFS */ - -#ifdef HAVE_TEMPNAM -PyDoc_STRVAR(posix_tempnam__doc__, -"tempnam([dir[, prefix]]) -> string\n\n\ -Return a unique name for a temporary file.\n\ -The directory and a prefix may be specified as strings; they may be omitted\n\ -or None if not needed."); - -static PyObject * -posix_tempnam(PyObject *self, PyObject *args) -{ - PyObject *result = NULL; - char *dir = NULL; - char *pfx = NULL; - char *name; - - if (!PyArg_ParseTuple(args, "|zz:tempnam", &dir, &pfx)) - return NULL; - - if (PyErr_WarnEx(PyExc_RuntimeWarning, - "tempnam is a potential security risk to your program", - 1) < 0) - return NULL; - -#ifdef MS_WINDOWS - name = _tempnam(dir, pfx); -#else - name = tempnam(dir, pfx); -#endif - if (name == NULL) - return PyErr_NoMemory(); - result = PyUnicode_DecodeFSDefault(name); - free(name); - return result; -} -#endif - - -#ifdef HAVE_TMPFILE -PyDoc_STRVAR(posix_tmpfile__doc__, -"tmpfile() -> file object\n\n\ -Create a temporary file with no directory entries."); - -static PyObject * -posix_tmpfile(PyObject *self, PyObject *noargs) -{ - FILE *fp; - int fd; - - fp = tmpfile(); - if (fp == NULL) - return posix_error(); - fd = fileno(fp); - if (fd != -1) - fd = dup(fd); - fclose(fp); - if (fd == -1) - return posix_error(); - return PyFile_FromFd(fd, "", "w+b", -1, NULL, NULL); -} -#endif - - -#ifdef HAVE_TMPNAM -PyDoc_STRVAR(posix_tmpnam__doc__, -"tmpnam() -> string\n\n\ -Return a unique name for a temporary file."); - -static PyObject * -posix_tmpnam(PyObject *self, PyObject *noargs) -{ - char buffer[L_tmpnam]; - char *name; - - if (PyErr_WarnEx(PyExc_RuntimeWarning, - "tmpnam is a potential security risk to your program", - 1) < 0) - return NULL; - -#ifdef USE_TMPNAM_R - name = tmpnam_r(buffer); -#else - name = tmpnam(buffer); -#endif - if (name == NULL) { - PyObject *err = Py_BuildValue("is", 0, -#ifdef USE_TMPNAM_R - "unexpected NULL from tmpnam_r" -#else - "unexpected NULL from tmpnam" -#endif - ); - PyErr_SetObject(PyExc_OSError, err); - Py_XDECREF(err); - return NULL; - } - return PyUnicode_DecodeFSDefault(buffer); -} -#endif - - /* This is used for fpathconf(), pathconf(), confstr() and sysconf(). * It maps strings representing configuration variable names to * integer values, allowing those functions to be called with the @@ -6941,15 +6836,6 @@ #if defined(HAVE_STATVFS) && defined(HAVE_SYS_STATVFS_H) {"statvfs", posix_statvfs, METH_VARARGS, posix_statvfs__doc__}, #endif -#ifdef HAVE_TMPFILE - {"tmpfile", posix_tmpfile, METH_NOARGS, posix_tmpfile__doc__}, -#endif -#ifdef HAVE_TEMPNAM - {"tempnam", posix_tempnam, METH_VARARGS, posix_tempnam__doc__}, -#endif -#ifdef HAVE_TMPNAM - {"tmpnam", posix_tmpnam, METH_NOARGS, posix_tmpnam__doc__}, -#endif #ifdef HAVE_CONFSTR {"confstr", posix_confstr, METH_VARARGS, posix_confstr__doc__}, #endif From python-3000-checkins at python.org Fri Oct 26 01:21:03 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 01:21:03 +0200 (CEST) Subject: [Python-3000-checkins] r58658 - in python/branches/py3k/Lib: io.py test/test_file.py Message-ID: <20071025232103.D24CC1E4017@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 01:21:03 2007 New Revision: 58658 Modified: python/branches/py3k/Lib/io.py python/branches/py3k/Lib/test/test_file.py Log: Patch # 1323 by Amaury Forgeot d'Arc. This patch corrects a problem in test_file.py on Windows: f.truncate() seeks to the truncation point, but does not empty the buffers. In the test, f.tell() returns -1... Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Fri Oct 26 01:21:03 2007 @@ -597,9 +597,24 @@ return self.raw.tell() def truncate(self, pos=None): + # On Windows, the truncate operation changes the current position + # to the end of the file, which may leave us with desynchronized + # buffers. + # Since we promise that truncate() won't change the current position, + # the easiest thing is to capture current pos now and seek back to + # it at the end. + + initialpos = self.tell() if pos is None: - pos = self.tell() - return self.raw.truncate(pos) + pos = initialpos + + # Flush the stream. We're mixing buffered I/O with lower-level I/O, + # and a flush may be necessary to synch both views of the current + # file state. + self.flush() + newpos = self.raw.truncate(pos) + self.seek(initialpos) + return newpos ### Flush and close ### Modified: python/branches/py3k/Lib/test/test_file.py ============================================================================== --- python/branches/py3k/Lib/test/test_file.py (original) +++ python/branches/py3k/Lib/test/test_file.py Fri Oct 26 01:21:03 2007 @@ -181,12 +181,13 @@ self.assertEquals(d, s) def testTruncateOnWindows(self): + # SF bug + # "file.truncate fault on windows" + os.unlink(TESTFN) + f = open(TESTFN, 'wb') - def bug801631(): - # SF bug - # "file.truncate fault on windows" - f = open(TESTFN, 'wb') + try: f.write(b'12345678901') # 11 bytes f.close() @@ -205,10 +206,8 @@ size = os.path.getsize(TESTFN) if size != 5: self.fail("File size after ftruncate wrong %d" % size) - - try: - bug801631() finally: + f.close() os.unlink(TESTFN) def testIteration(self): From python-3000-checkins at python.org Fri Oct 26 01:47:33 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 01:47:33 +0200 (CEST) Subject: [Python-3000-checkins] r58659 - in python/branches/py3k: Lib/test/regrtest.py Objects/unicodeobject.c Message-ID: <20071025234733.7FFAD1E4017@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 01:47:33 2007 New Revision: 58659 Modified: python/branches/py3k/Lib/test/regrtest.py python/branches/py3k/Objects/unicodeobject.c Log: Patch # 1302 by Christian Heimes (with some love from me :-). The patch fixes the output for profile and cProfile. Another patch from Alexandre and me added additional calls to the UTF-8 codec. Modified: python/branches/py3k/Lib/test/regrtest.py ============================================================================== --- python/branches/py3k/Lib/test/regrtest.py (original) +++ python/branches/py3k/Lib/test/regrtest.py Fri Oct 26 01:47:33 2007 @@ -1119,6 +1119,15 @@ if not os.path.supports_unicode_filenames: self.expected.add('test_pep277') + # doctest, profile and cProfile tests fail when the codec for the fs + # encoding isn't built in because PyUnicode_Decode() adds two calls + # into Python. + encs = ("utf-8", "latin-1", "ascii", "mbcs", "utf-16", "utf-32") + if sys.getfilesystemencoding().lower() not in encs: + self.expected.add('test_profile') + self.expected.add('test_cProfile') + self.expected.add('test_doctest') + try: from test import test_socket_ssl except ImportError: Modified: python/branches/py3k/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k/Objects/unicodeobject.c (original) +++ python/branches/py3k/Objects/unicodeobject.c Fri Oct 26 01:47:33 2007 @@ -41,6 +41,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "bytes_methods.h" #include "unicodeobject.h" #include "ucnhash.h" @@ -592,9 +593,9 @@ if (*f == '%') { const char* p = f; width = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) width = (width*10) + *f++ - '0'; - while (*++f && *f != '%' && !isalpha(Py_CHARMASK(*f))) + while (*++f && *f != '%' && !ISALPHA(*f)) ; /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since @@ -755,12 +756,12 @@ zeropad = (*f == '0'); /* parse the width.precision part */ width = 0; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) width = (width*10) + *f++ - '0'; precision = 0; if (*f == '.') { f++; - while (isdigit(Py_CHARMASK(*f))) + while (ISDIGIT(*f)) precision = (precision*10) + *f++ - '0'; } /* handle the long flag, but only for %ld and %lu. @@ -1056,21 +1057,47 @@ { PyObject *buffer = NULL, *unicode; Py_buffer info; + char lower[20]; /* Enough for any encoding name we recognize */ + char *l; + const char *e; if (encoding == NULL) - encoding = PyUnicode_GetDefaultEncoding(); + encoding = PyUnicode_GetDefaultEncoding(); + + /* Convert encoding to lower case and replace '_' with '-' in order to + catch e.g. UTF_8 */ + e = encoding; + l = lower; + while (*e && l < &lower[(sizeof lower) - 2]) { + if (ISUPPER(*e)) { + *l++ = TOLOWER(*e++); + } + else if (*e == '_') { + *l++ = '-'; + e++; + } + else { + *l++ = *e++; + } + } + *l = '\0'; /* Shortcuts for common default encodings */ - if (strcmp(encoding, "utf-8") == 0) + if (strcmp(lower, "utf-8") == 0) return PyUnicode_DecodeUTF8(s, size, errors); - else if (strcmp(encoding, "latin-1") == 0) + else if ((strcmp(lower, "latin-1") == 0) || + (strcmp(lower, "iso-8859-1") == 0)) return PyUnicode_DecodeLatin1(s, size, errors); #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T) - else if (strcmp(encoding, "mbcs") == 0) + else if (strcmp(lower, "mbcs") == 0) return PyUnicode_DecodeMBCS(s, size, errors); #endif - else if (strcmp(encoding, "ascii") == 0) + else if (strcmp(lower, "ascii") == 0) return PyUnicode_DecodeASCII(s, size, errors); + else if (strcmp(lower, "utf-16") == 0) + return PyUnicode_DecodeUTF16(s, size, errors, 0); + else if (strcmp(lower, "utf-32") == 0) + return PyUnicode_DecodeUTF32(s, size, errors, 0); /* Decode via the codec registry */ buffer = NULL; @@ -1470,7 +1497,7 @@ #define B64(n) \ ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"[(n) & 0x3f]) #define B64CHAR(c) \ - (isalnum(c) || (c) == '+' || (c) == '/') + (ISALNUM(c) || (c) == '+' || (c) == '/') #define UB64(c) \ ((c) == '+' ? 62 : (c) == '/' ? 63 : (c) >= 'a' ? \ (c) - 71 : (c) >= 'A' ? (c) - 65 : (c) + 4 ) @@ -2703,7 +2730,7 @@ } for (i = 0; i < digits; ++i) { c = (unsigned char) s[i]; - if (!isxdigit(c)) { + if (!ISXDIGIT(c)) { endinpos = (s+i+1)-starts; if (unicode_decode_call_errorhandler( errors, &errorHandler, @@ -3077,7 +3104,7 @@ outpos = p-PyUnicode_AS_UNICODE(v); for (x = 0, i = 0; i < count; ++i, ++s) { c = (unsigned char)*s; - if (!isxdigit(c)) { + if (!ISXDIGIT(c)) { endinpos = s-starts; if (unicode_decode_call_errorhandler( errors, &errorHandler, From python-3000-checkins at python.org Fri Oct 26 06:29:24 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 06:29:24 +0200 (CEST) Subject: [Python-3000-checkins] r58662 - in python/branches/py3k/Lib: netrc.py subprocess.py test/regrtest.py test/test_mailbox.py test/test_netrc.py test/test_pep277.py test/test_subprocess.py Message-ID: <20071026042924.87E161E4017@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 06:29:23 2007 New Revision: 58662 Modified: python/branches/py3k/Lib/netrc.py python/branches/py3k/Lib/subprocess.py python/branches/py3k/Lib/test/regrtest.py python/branches/py3k/Lib/test/test_mailbox.py python/branches/py3k/Lib/test/test_netrc.py python/branches/py3k/Lib/test/test_pep277.py python/branches/py3k/Lib/test/test_subprocess.py Log: Patch # 1331 by Christian Heimes. The patch fixes some of the problems on Windows. It doesn't introduce addition problems on Linux. Modified: python/branches/py3k/Lib/netrc.py ============================================================================== --- python/branches/py3k/Lib/netrc.py (original) +++ python/branches/py3k/Lib/netrc.py Fri Oct 26 06:29:23 2007 @@ -26,9 +26,12 @@ file = os.path.join(os.environ['HOME'], ".netrc") except KeyError: raise IOError("Could not find .netrc: $HOME is not set") - fp = open(file) self.hosts = {} self.macros = {} + with open(file) as fp: + self._parse(file, fp) + + def _parse(self, file, fp): lexer = shlex.shlex(fp) lexer.wordchars += r"""!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~""" while 1: Modified: python/branches/py3k/Lib/subprocess.py ============================================================================== --- python/branches/py3k/Lib/subprocess.py (original) +++ python/branches/py3k/Lib/subprocess.py Fri Oct 26 06:29:23 2007 @@ -809,6 +809,8 @@ if self.stdin: if input is not None: + if isinstance(input, str): + input = input.encode() self.stdin.write(input) self.stdin.close() Modified: python/branches/py3k/Lib/test/regrtest.py ============================================================================== --- python/branches/py3k/Lib/test/regrtest.py (original) +++ python/branches/py3k/Lib/test/regrtest.py Fri Oct 26 06:29:23 2007 @@ -885,6 +885,7 @@ test_pwd test_resource test_signal + test_syslog test_threadsignals test_wait3 test_wait4 Modified: python/branches/py3k/Lib/test/test_mailbox.py ============================================================================== --- python/branches/py3k/Lib/test/test_mailbox.py (original) +++ python/branches/py3k/Lib/test/test_mailbox.py Fri Oct 26 06:29:23 2007 @@ -58,6 +58,7 @@ self._box = self._factory(self._path) def tearDown(self): + self._box.close() self._delete_recursively(self._path) def test_add(self): @@ -390,12 +391,14 @@ self._box.add(contents[0]) self._box.add(contents[1]) self._box.add(contents[2]) + oldbox = self._box method() self._box = self._factory(self._path) keys = self._box.keys() self.assertEqual(len(keys), 3) for key in keys: self.assert_(self._box.get_string(key) in contents) + oldbox.close() def test_dump_message(self): # Write message representations to disk @@ -403,7 +406,7 @@ _sample_message, io.StringIO(_sample_message)): output = io.StringIO() self._box._dump_message(input, output) - self.assert_(output.getvalue() == + self.assertEqual(output.getvalue(), _sample_message.replace('\n', os.linesep)) output = io.StringIO() self.assertRaises(TypeError, @@ -694,6 +697,7 @@ class _TestMboxMMDF(TestMailbox): def tearDown(self): + self._box.close() self._delete_recursively(self._path) for lock_remnant in glob.glob(self._path + '.*'): test_support.unlink(lock_remnant) @@ -916,6 +920,7 @@ _factory = lambda self, path, factory=None: mailbox.Babyl(path, factory) def tearDown(self): + self._box.close() self._delete_recursively(self._path) for lock_remnant in glob.glob(self._path + '.*'): test_support.unlink(lock_remnant) Modified: python/branches/py3k/Lib/test/test_netrc.py ============================================================================== --- python/branches/py3k/Lib/test/test_netrc.py (original) +++ python/branches/py3k/Lib/test/test_netrc.py Fri Oct 26 06:29:23 2007 @@ -21,25 +21,24 @@ class NetrcTestCase(unittest.TestCase): - def setUp (self): + def setUp(self): mode = 'w' if sys.platform not in ['cygwin']: mode += 't' fp = open(temp_filename, mode) fp.write(TEST_NETRC) fp.close() - self.netrc = netrc.netrc(temp_filename) - def tearDown (self): - del self.netrc + def tearDown(self): os.unlink(temp_filename) def test_case_1(self): - self.assert_(self.netrc.macros == {'macro1':['line1\n', 'line2\n'], + nrc = netrc.netrc(temp_filename) + self.assert_(nrc.macros == {'macro1':['line1\n', 'line2\n'], 'macro2':['line3\n', 'line4\n']} ) - self.assert_(self.netrc.hosts['foo'] == ('log1', 'acct1', 'pass1')) - self.assert_(self.netrc.hosts['default'] == ('log2', None, 'pass2')) + self.assert_(nrc.hosts['foo'] == ('log1', 'acct1', 'pass1')) + self.assert_(nrc.hosts['default'] == ('log2', None, 'pass2')) def test_main(): test_support.run_unittest(NetrcTestCase) Modified: python/branches/py3k/Lib/test/test_pep277.py ============================================================================== --- python/branches/py3k/Lib/test/test_pep277.py (original) +++ python/branches/py3k/Lib/test/test_pep277.py Fri Oct 26 06:29:23 2007 @@ -36,7 +36,7 @@ except OSError: pass for name in self.files: - f = open(name, 'w') + f = open(name, 'wb') f.write((name+'\n').encode("utf-8")) f.close() os.stat(name) @@ -71,7 +71,7 @@ def test_open(self): for name in self.files: - f = open(name, 'w') + f = open(name, 'wb') f.write((name+'\n').encode("utf-8")) f.close() os.stat(name) @@ -80,7 +80,7 @@ f1 = os.listdir(test_support.TESTFN) # Printing f1 is not appropriate, as specific filenames # returned depend on the local encoding - f2 = os.listdir(str(test_support.TESTFN, + f2 = os.listdir(str(test_support.TESTFN.encode("utf-8"), sys.getfilesystemencoding())) f2.sort() print(f2) @@ -96,7 +96,7 @@ oldwd = os.getcwd() os.mkdir(dirname) os.chdir(dirname) - f = open(filename, 'w') + f = open(filename, 'wb') f.write((filename + '\n').encode("utf-8")) f.close() print(repr(filename)) Modified: python/branches/py3k/Lib/test/test_subprocess.py ============================================================================== --- python/branches/py3k/Lib/test/test_subprocess.py (original) +++ python/branches/py3k/Lib/test/test_subprocess.py Fri Oct 26 06:29:23 2007 @@ -630,7 +630,7 @@ p = subprocess.Popen(["set"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(p.stdout.read().find(b"physalis"), -1) def test_shell_string(self): # Run command through the shell (string) @@ -639,7 +639,7 @@ p = subprocess.Popen("set", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(p.stdout.read().find(b"physalis"), -1) def test_call_string(self): # call() function with string argument on Windows From python-3000-checkins at python.org Fri Oct 26 07:42:09 2007 From: python-3000-checkins at python.org (brett.cannon) Date: Fri, 26 Oct 2007 07:42:09 +0200 (CEST) Subject: [Python-3000-checkins] r58663 - python/branches/py3k/Misc/NEWS Message-ID: <20071026054209.AED191E401E@bag.python.org> Author: brett.cannon Date: Fri Oct 26 07:42:09 2007 New Revision: 58663 Modified: python/branches/py3k/Misc/NEWS Log: Add a missing news entry about how str8's constructor signature now matches the bytes type. Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Fri Oct 26 07:42:09 2007 @@ -11,6 +11,8 @@ Core and Builtins ----------------- +- str8 now has the same construction signature as bytes. + - Comparisons between str and str8 now return False/True for ==/!=. sqlite3 returns str8 when recreating on object from it's __conform__ value. The struct module returns str8 for all string-related formats. This was true From python-3000-checkins at python.org Fri Oct 26 08:05:28 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Fri, 26 Oct 2007 08:05:28 +0200 (CEST) Subject: [Python-3000-checkins] r58664 - python/branches/py3k/Python/sysmodule.c Message-ID: <20071026060528.9BA0D1E401E@bag.python.org> Author: neal.norwitz Date: Fri Oct 26 08:05:28 2007 New Revision: 58664 Modified: python/branches/py3k/Python/sysmodule.c Log: Try to fix some of the failing tests on Win64. PY_SSIZE_T_MAX (8 bytes) is larger than what can fit into a long (4 bytes). Hopefully this will fix more problems than it creates. There are many, many compile warnings on Win64. Each of these should be investigated to determine if they are real problems or not. Many of these presumably affect the trunk too. Modified: python/branches/py3k/Python/sysmodule.c ============================================================================== --- python/branches/py3k/Python/sysmodule.c (original) +++ python/branches/py3k/Python/sysmodule.c Fri Oct 26 08:05:28 2007 @@ -1089,7 +1089,7 @@ SET_SYS_FROM_STRING("maxint", PyInt_FromLong(PyInt_GetMax())); SET_SYS_FROM_STRING("maxsize", - PyInt_FromLong(PY_SSIZE_T_MAX)); + PyInt_FromSsize_t(PY_SSIZE_T_MAX)); SET_SYS_FROM_STRING("maxunicode", PyInt_FromLong(PyUnicode_GetMax())); SET_SYS_FROM_STRING("builtin_module_names", From python-3000-checkins at python.org Fri Oct 26 08:44:10 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Fri, 26 Oct 2007 08:44:10 +0200 (CEST) Subject: [Python-3000-checkins] r58665 - python/branches/py3k/Objects/bytesobject.c python/branches/py3k/Objects/memoryobject.c Message-ID: <20071026064410.503C91E401E@bag.python.org> Author: neal.norwitz Date: Fri Oct 26 08:44:10 2007 New Revision: 58665 Modified: python/branches/py3k/Objects/bytesobject.c python/branches/py3k/Objects/memoryobject.c Log: Fix some Py_ssize_t warnings on Win64 that were probably bugs Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Fri Oct 26 08:44:10 2007 @@ -108,7 +108,7 @@ PyBytes_FromStringAndSize(const char *bytes, Py_ssize_t size) { PyBytesObject *new; - int alloc; + Py_ssize_t alloc; assert(size >= 0); @@ -1458,7 +1458,7 @@ } Py_LOCAL_INLINE(Py_ssize_t) -countchar(const char *target, int target_len, char c, Py_ssize_t maxcount) +countchar(const char *target, Py_ssize_t target_len, char c, Py_ssize_t maxcount) { Py_ssize_t count=0; const char *start=target; Modified: python/branches/py3k/Objects/memoryobject.c ============================================================================== --- python/branches/py3k/Objects/memoryobject.c (original) +++ python/branches/py3k/Objects/memoryobject.c Fri Oct 26 08:44:10 2007 @@ -78,7 +78,7 @@ static void _strided_copy_nd(char *dest, char *src, int nd, Py_ssize_t *shape, - Py_ssize_t *strides, int itemsize, char fort) + Py_ssize_t *strides, Py_ssize_t itemsize, char fort) { int k; Py_ssize_t outstride; @@ -298,7 +298,7 @@ static PyObject * memory_itemsize_get(PyMemoryViewObject *self) { - return PyInt_FromLong(self->view.itemsize); + return PyInt_FromSsize_t(self->view.itemsize); } static PyObject * From python-3000-checkins at python.org Fri Oct 26 19:19:34 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 19:19:34 +0200 (CEST) Subject: [Python-3000-checkins] r58673 - in python/branches/py3k: Lib/io.py Modules/_fileio.c Message-ID: <20071026171934.2D0D91E4003@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 19:19:33 2007 New Revision: 58673 Modified: python/branches/py3k/Lib/io.py python/branches/py3k/Modules/_fileio.c Log: Patch 1330 by Christian Heimes (with some TLC applied by myself). Move most of the messiness with truncate() on Windows into _fileio.c. Still keep the flush() call in io.py though. Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Fri Oct 26 19:19:33 2007 @@ -597,24 +597,14 @@ return self.raw.tell() def truncate(self, pos=None): - # On Windows, the truncate operation changes the current position - # to the end of the file, which may leave us with desynchronized - # buffers. - # Since we promise that truncate() won't change the current position, - # the easiest thing is to capture current pos now and seek back to - # it at the end. - - initialpos = self.tell() - if pos is None: - pos = initialpos - # Flush the stream. We're mixing buffered I/O with lower-level I/O, # and a flush may be necessary to synch both views of the current # file state. self.flush() - newpos = self.raw.truncate(pos) - self.seek(initialpos) - return newpos + + if pos is None: + pos = self.tell() + return self.raw.truncate(pos) ### Flush and close ### Modified: python/branches/py3k/Modules/_fileio.c ============================================================================== --- python/branches/py3k/Modules/_fileio.c (original) +++ python/branches/py3k/Modules/_fileio.c Fri Oct 26 19:19:33 2007 @@ -628,14 +628,21 @@ so don't even try using it. */ { HANDLE hFile; - PyObject *pos2; + PyObject *pos2, *oldposobj; + + /* store the current position */ + oldposobj = portable_lseek(self->fd, NULL, 1); + if (oldposobj == NULL) { + Py_DECREF(posobj); + return NULL; + } /* Have to move current pos to desired endpoint on Windows. */ errno = 0; pos2 = portable_lseek(fd, posobj, SEEK_SET); - if (pos2 == NULL) - { + if (pos2 == NULL) { Py_DECREF(posobj); + Py_DECREF(oldposobj); return NULL; } Py_DECREF(pos2); @@ -651,6 +658,18 @@ errno = EACCES; } Py_END_ALLOW_THREADS + + if (ret == 0) { + /* Move to the previous position in the file */ + pos2 = portable_lseek(fd, oldposobj, SEEK_SET); + if (pos2 == NULL) { + Py_DECREF(posobj); + Py_DECREF(oldposobj); + return NULL; + } + } + Py_DECREF(pos2); + Py_DECREF(oldposobj); } #else Py_BEGIN_ALLOW_THREADS From python-3000-checkins at python.org Fri Oct 26 19:39:48 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 19:39:48 +0200 (CEST) Subject: [Python-3000-checkins] r58674 - in python/branches/py3k: Lib/_abcoll.py Objects/bytesobject.c Message-ID: <20071026173948.BD5061E402A@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 19:39:48 2007 New Revision: 58674 Modified: python/branches/py3k/Lib/_abcoll.py python/branches/py3k/Objects/bytesobject.c Log: Patch 1335 by Christian Heimes. Add a bytes iterator (copied from stringobject.c and reindented :-). I (Guido) added a small change to _abcoll.py to remove the registration of bytes as a virtual subtype of Iterator -- the presence of __iter__ will handle that now. Modified: python/branches/py3k/Lib/_abcoll.py ============================================================================== --- python/branches/py3k/Lib/_abcoll.py (original) +++ python/branches/py3k/Lib/_abcoll.py Fri Oct 26 19:39:48 2007 @@ -51,8 +51,6 @@ return True return NotImplemented -Iterable.register(bytes) - class Iterator(metaclass=ABCMeta): Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Fri Oct 26 19:39:48 2007 @@ -3023,6 +3023,8 @@ \n\ If an argument is given it must be an iterable yielding ints in range(256)."); +static PyObject *bytes_iter(PyObject *seq); + PyTypeObject PyBytes_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "bytes", @@ -3050,7 +3052,7 @@ 0, /* tp_clear */ (richcmpfunc)bytes_richcompare, /* tp_richcompare */ 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ + bytes_iter, /* tp_iter */ 0, /* tp_iternext */ bytes_methods, /* tp_methods */ 0, /* tp_members */ @@ -3065,3 +3067,121 @@ PyType_GenericNew, /* tp_new */ PyObject_Del, /* tp_free */ }; + +/*********************** Bytes Iterator ****************************/ + +typedef struct { + PyObject_HEAD + Py_ssize_t it_index; + PyBytesObject *it_seq; /* Set to NULL when iterator is exhausted */ +} bytesiterobject; + +static void +bytesiter_dealloc(bytesiterobject *it) +{ + _PyObject_GC_UNTRACK(it); + Py_XDECREF(it->it_seq); + PyObject_GC_Del(it); +} + +static int +bytesiter_traverse(bytesiterobject *it, visitproc visit, void *arg) +{ + Py_VISIT(it->it_seq); + return 0; +} + +static PyObject * +bytesiter_next(bytesiterobject *it) +{ + PyBytesObject *seq; + PyObject *item; + + assert(it != NULL); + seq = it->it_seq; + if (seq == NULL) + return NULL; + assert(PyBytes_Check(seq)); + + if (it->it_index < PyBytes_GET_SIZE(seq)) { + item = PyInt_FromLong( + (unsigned char)seq->ob_bytes[it->it_index]); + if (item != NULL) + ++it->it_index; + return item; + } + + Py_DECREF(seq); + it->it_seq = NULL; + return NULL; +} + +static PyObject * +bytesiter_length_hint(bytesiterobject *it) +{ + Py_ssize_t len = 0; + if (it->it_seq) + len = PyBytes_GET_SIZE(it->it_seq) - it->it_index; + return PyInt_FromSsize_t(len); +} + +PyDoc_STRVAR(length_hint_doc, + "Private method returning an estimate of len(list(it))."); + +static PyMethodDef bytesiter_methods[] = { + {"__length_hint__", (PyCFunction)bytesiter_length_hint, METH_NOARGS, + length_hint_doc}, + {NULL, NULL} /* sentinel */ +}; + +PyTypeObject PyBytesIter_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "bytesiterator", /* tp_name */ + sizeof(bytesiterobject), /* tp_basicsize */ + 0, /* tp_itemsize */ + /* methods */ + (destructor)bytesiter_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ + 0, /* tp_doc */ + (traverseproc)bytesiter_traverse, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + PyObject_SelfIter, /* tp_iter */ + (iternextfunc)bytesiter_next, /* tp_iternext */ + bytesiter_methods, /* tp_methods */ + 0, +}; + +static PyObject * +bytes_iter(PyObject *seq) +{ + bytesiterobject *it; + + if (!PyBytes_Check(seq)) { + PyErr_BadInternalCall(); + return NULL; + } + it = PyObject_GC_New(bytesiterobject, &PyBytesIter_Type); + if (it == NULL) + return NULL; + it->it_index = 0; + Py_INCREF(seq); + it->it_seq = (PyBytesObject *)seq; + _PyObject_GC_TRACK(it); + return (PyObject *)it; +} From python-3000-checkins at python.org Fri Oct 26 20:43:10 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 20:43:10 +0200 (CEST) Subject: [Python-3000-checkins] r58676 - python/branches/py3k/Lib/test/test_str.py Message-ID: <20071026184310.92DE61E4002@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 20:43:10 2007 New Revision: 58676 Removed: python/branches/py3k/Lib/test/test_str.py Log: Delete test_str.py. There's not much there I care about, and it is confused about whether it's testing str8 or str... The stuff that matters is already tested in test_unicode.py anyway. Deleted: /python/branches/py3k/Lib/test/test_str.py ============================================================================== --- /python/branches/py3k/Lib/test/test_str.py Fri Oct 26 20:43:10 2007 +++ (empty file) @@ -1,66 +0,0 @@ - -import unittest -import struct -import sys -from test import test_support, string_tests - - -class StrTest( - string_tests.CommonTest, - string_tests.MixinStrUnicodeUserStringTest, - string_tests.MixinStrUnicodeTest, - ): - - type2test = str8 - - # We don't need to propagate to str - def fixtype(self, obj): - return obj - - def test_formatting(self): - string_tests.MixinStrUnicodeUserStringTest.test_formatting(self) - self.assertRaises(OverflowError, '%c'.__mod__, 0x12341234) - - def test_iterators(self): - # Make sure str objects have an __iter__ method - it = "abc".__iter__() - self.assertEqual(next(it), "a") - self.assertEqual(next(it), "b") - self.assertEqual(next(it), "c") - self.assertRaises(StopIteration, next, it) - - def test_conversion(self): - # Make sure __str__() behaves properly - - class Foo1: - def __str__(self): - return "foo" - - class Foo7(str): - def __str__(self): - return "foos" - - class Foo8(str): - def __new__(cls, content=""): - return str.__new__(cls, 2*content) - def __str__(self): - return self - - self.assertEqual(str(Foo1()), "foo") - self.assertEqual(str(Foo7("bar")), "foos") - self.assertEqual(str(Foo8("foo")), "foofoo") - - def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxint > (1 << 32) or struct.calcsize('P') != 4: - return - self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxint) - - -def test_main(): - test_support.run_unittest(StrTest) - -if __name__ == "__main__": - test_main() From python-3000-checkins at python.org Fri Oct 26 21:34:40 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 21:34:40 +0200 (CEST) Subject: [Python-3000-checkins] r58677 - in python/branches/py3k: Doc/c-api/concrete.rst Include/stringobject.h Message-ID: <20071026193440.735091E4002@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 21:34:40 2007 New Revision: 58677 Modified: python/branches/py3k/Doc/c-api/concrete.rst python/branches/py3k/Include/stringobject.h Log: Kill PyString_Encode(), which isn't used anywhere. (We should also kill the other PyString_ encode/decode APIs, but they still have some users.) Modified: python/branches/py3k/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k/Doc/c-api/concrete.rst (original) +++ python/branches/py3k/Doc/c-api/concrete.rst Fri Oct 26 21:34:40 2007 @@ -874,15 +874,6 @@ if an exception was raised by the codec. -.. cfunction:: PyObject* PyString_Encode(const char *s, Py_ssize_t size, const char *encoding, const char *errors) - - Encode the :ctype:`char` buffer of the given size by passing it to the codec - registered for *encoding* and return a Python object. *encoding* and *errors* - have the same meaning as the parameters of the same name in the string - :meth:`encode` method. The codec to be used is looked up using the Python codec - registry. Return *NULL* if an exception was raised by the codec. - - .. cfunction:: PyObject* PyString_AsEncodedObject(PyObject *str, const char *encoding, const char *errors) Encode a string object using the codec registered for *encoding* and return the Modified: python/branches/py3k/Include/stringobject.h ============================================================================== --- python/branches/py3k/Include/stringobject.h (original) +++ python/branches/py3k/Include/stringobject.h Fri Oct 26 21:34:40 2007 @@ -70,7 +70,7 @@ PyAPI_FUNC(PyObject *) PyString_Format(PyObject *, PyObject *); PyAPI_FUNC(PyObject *) _PyString_FormatLong(PyObject*, int, int, int, char**, int*); -PyAPI_FUNC(PyObject *) PyString_DecodeEscape(const char *, Py_ssize_t, +PyAPI_FUNC(PyObject *) PyString_DecodeEscape(const char *, Py_ssize_t, const char *, Py_ssize_t, const char *); @@ -102,17 +102,7 @@ const char *errors /* error handling */ ); -/* Encodes a char buffer of the given size and returns a - Python object. */ - -PyAPI_FUNC(PyObject*) PyString_Encode( - const char *s, /* string char buffer */ - Py_ssize_t size, /* number of chars to encode */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - -/* Encodes a string object and returns the result as Python +/* Encodes a string object and returns the result as Python object. */ PyAPI_FUNC(PyObject*) PyString_AsEncodedObject( @@ -122,8 +112,8 @@ ); /* Encodes a string object and returns the result as Python string - object. - + object. + If the codec returns an Unicode object, the object is converted back to a string using the default encoding. @@ -135,7 +125,7 @@ const char *errors /* error handling */ ); -/* Decodes a string object and returns the result as Python +/* Decodes a string object and returns the result as Python object. */ PyAPI_FUNC(PyObject*) PyString_AsDecodedObject( @@ -145,8 +135,8 @@ ); /* Decodes a string object and returns the result as Python string - object. - + object. + If the codec returns an Unicode object, the object is converted back to a string using the default encoding. @@ -171,7 +161,7 @@ (only possible for 0-terminated strings) */ ); - + #ifdef __cplusplus } From python-3000-checkins at python.org Fri Oct 26 23:46:44 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 23:46:44 +0200 (CEST) Subject: [Python-3000-checkins] r58678 - in python/branches/py3k: Lib/test/test_bytes.py Objects/bytesobject.c Objects/stringobject.c Message-ID: <20071026214644.7AC401E4002@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 23:46:44 2007 New Revision: 58678 Modified: python/branches/py3k/Lib/test/test_bytes.py python/branches/py3k/Objects/bytesobject.c python/branches/py3k/Objects/stringobject.c Log: Sort the method lists for str8 and bytes so differences are more apparent. Changed bytes.fromhex to take a str argument, as the docstring promises. Added str8.fromhex (untested so far). Modified: python/branches/py3k/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k/Lib/test/test_bytes.py (original) +++ python/branches/py3k/Lib/test/test_bytes.py Fri Oct 26 23:46:44 2007 @@ -450,23 +450,21 @@ self.assert_(alloc >= len(b)) if alloc not in seq: seq.append(alloc) - #print seq def test_fromhex(self): self.assertRaises(TypeError, bytes.fromhex) self.assertRaises(TypeError, bytes.fromhex, 1) - self.assertEquals(bytes.fromhex(b''), bytes()) + self.assertEquals(bytes.fromhex(''), bytes()) b = bytes([0x1a, 0x2b, 0x30]) - self.assertEquals(bytes.fromhex(b'1a2B30'), b) - self.assertEquals(bytes.fromhex(b' 1A 2B 30 '), b) - self.assertEquals(bytes.fromhex(memoryview(b'')), bytes()) - self.assertEquals(bytes.fromhex(memoryview(b'0000')), bytes([0, 0])) - self.assertRaises(TypeError, bytes.fromhex, '1B') - self.assertRaises(ValueError, bytes.fromhex, b'a') - self.assertRaises(ValueError, bytes.fromhex, b'rt') - self.assertRaises(ValueError, bytes.fromhex, b'1a b cd') - self.assertRaises(ValueError, bytes.fromhex, b'\x00') - self.assertRaises(ValueError, bytes.fromhex, b'12 \x00 34') + self.assertEquals(bytes.fromhex('1a2B30'), b) + self.assertEquals(bytes.fromhex(' 1A 2B 30 '), b) + self.assertEquals(bytes.fromhex('0000'), b'\0\0') + self.assertRaises(TypeError, bytes.fromhex, b'1B') + self.assertRaises(ValueError, bytes.fromhex, 'a') + self.assertRaises(ValueError, bytes.fromhex, 'rt') + self.assertRaises(ValueError, bytes.fromhex, '1a b cd') + self.assertRaises(ValueError, bytes.fromhex, '\x00') + self.assertRaises(ValueError, bytes.fromhex, '12 \x00 34') def test_join(self): self.assertEqual(b"".join([]), bytes()) Modified: python/branches/py3k/Objects/bytesobject.c ============================================================================== --- python/branches/py3k/Objects/bytesobject.c (original) +++ python/branches/py3k/Objects/bytesobject.c Fri Oct 26 23:46:44 2007 @@ -2854,11 +2854,13 @@ \n\ Create a bytes object from a string of hexadecimal numbers.\n\ Spaces between two numbers are accepted. Example:\n\ -bytes.fromhex('10 2030') -> bytes([0x10, 0x20, 0x30])."); +bytes.fromhex('10 1112') -> b'\\x10\\x11\\x12'."); static int -hex_digit_to_int(int c) +hex_digit_to_int(Py_UNICODE c) { + if (c >= 128) + return -1; if (ISDIGIT(c)) return c - '0'; else { @@ -2875,52 +2877,42 @@ { PyObject *newbytes, *hexobj; char *buf; - unsigned char *hex; - Py_ssize_t byteslen, i, j; + Py_UNICODE *hex; + Py_ssize_t hexlen, byteslen, i, j; int top, bot; - Py_buffer vhex; - if (!PyArg_ParseTuple(args, "O:fromhex", &hexobj)) + if (!PyArg_ParseTuple(args, "U:fromhex", &hexobj)) return NULL; - - if (_getbuffer(hexobj, &vhex) < 0) - return NULL; - - byteslen = vhex.len / 2; /* max length if there are no spaces */ - hex = vhex.buf; - + assert(PyUnicode_Check(hexobj)); + hexlen = PyUnicode_GET_SIZE(hexobj); + hex = PyUnicode_AS_UNICODE(hexobj); + byteslen = hexlen/2; /* This overestimates if there are spaces */ newbytes = PyBytes_FromStringAndSize(NULL, byteslen); - if (!newbytes) { - PyObject_ReleaseBuffer(hexobj, &vhex); + if (!newbytes) return NULL; - } buf = PyBytes_AS_STRING(newbytes); - - for (i = j = 0; i < vhex.len; i += 2) { + for (i = j = 0; i < hexlen; i += 2) { /* skip over spaces in the input */ - while (Py_CHARMASK(hex[i]) == ' ') + while (hex[i] == ' ') i++; - if (i >= vhex.len) + if (i >= hexlen) break; - top = hex_digit_to_int(Py_CHARMASK(hex[i])); - bot = hex_digit_to_int(Py_CHARMASK(hex[i+1])); + top = hex_digit_to_int(hex[i]); + bot = hex_digit_to_int(hex[i+1]); if (top == -1 || bot == -1) { PyErr_Format(PyExc_ValueError, - "non-hexadecimal number string '%c%c' found in " - "fromhex() arg at position %zd", - hex[i], hex[i+1], i); + "non-hexadecimal number found in " + "fromhex() arg at position %zd", i); goto error; } buf[j++] = (top << 4) + bot; } if (PyBytes_Resize(newbytes, j) < 0) goto error; - PyObject_ReleaseBuffer(hexobj, &vhex); return newbytes; error: Py_DECREF(newbytes); - PyObject_ReleaseBuffer(hexobj, &vhex); return NULL; } @@ -2964,57 +2956,65 @@ static PyMethodDef bytes_methods[] = { - {"find", (PyCFunction)bytes_find, METH_VARARGS, find__doc__}, - {"count", (PyCFunction)bytes_count, METH_VARARGS, count__doc__}, - {"index", (PyCFunction)bytes_index, METH_VARARGS, index__doc__}, - {"rfind", (PyCFunction)bytes_rfind, METH_VARARGS, rfind__doc__}, - {"rindex", (PyCFunction)bytes_rindex, METH_VARARGS, rindex__doc__}, - {"endswith", (PyCFunction)bytes_endswith, METH_VARARGS, endswith__doc__}, - {"startswith", (PyCFunction)bytes_startswith, METH_VARARGS, - startswith__doc__}, - {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, - {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, + {"__alloc__", (PyCFunction)bytes_alloc, METH_NOARGS, alloc_doc}, + {"__reduce__", (PyCFunction)bytes_reduce, METH_NOARGS, reduce_doc}, + {"append", (PyCFunction)bytes_append, METH_O, append__doc__}, {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, _Py_capitalize__doc__}, - {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, - _Py_swapcase__doc__}, - {"islower", (PyCFunction)stringlib_islower, METH_NOARGS,_Py_islower__doc__}, - {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS,_Py_isupper__doc__}, - {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS,_Py_isspace__doc__}, - {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS,_Py_isdigit__doc__}, - {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS,_Py_istitle__doc__}, - {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS,_Py_isalpha__doc__}, - {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS,_Py_isalnum__doc__}, - {"replace", (PyCFunction)bytes_replace, METH_VARARGS, replace__doc__}, - {"translate", (PyCFunction)bytes_translate, METH_VARARGS, translate__doc__}, - {"partition", (PyCFunction)bytes_partition, METH_O, partition__doc__}, - {"rpartition", (PyCFunction)bytes_rpartition, METH_O, rpartition__doc__}, - {"split", (PyCFunction)bytes_split, METH_VARARGS, split__doc__}, - {"rsplit", (PyCFunction)bytes_rsplit, METH_VARARGS, rsplit__doc__}, - {"extend", (PyCFunction)bytes_extend, METH_O, extend__doc__}, - {"insert", (PyCFunction)bytes_insert, METH_VARARGS, insert__doc__}, - {"append", (PyCFunction)bytes_append, METH_O, append__doc__}, - {"reverse", (PyCFunction)bytes_reverse, METH_NOARGS, reverse__doc__}, - {"pop", (PyCFunction)bytes_pop, METH_VARARGS, pop__doc__}, - {"remove", (PyCFunction)bytes_remove, METH_O, remove__doc__}, - {"strip", (PyCFunction)bytes_strip, METH_VARARGS, strip__doc__}, - {"lstrip", (PyCFunction)bytes_lstrip, METH_VARARGS, lstrip__doc__}, - {"rstrip", (PyCFunction)bytes_rstrip, METH_VARARGS, rstrip__doc__}, + {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, + {"count", (PyCFunction)bytes_count, METH_VARARGS, count__doc__}, {"decode", (PyCFunction)bytes_decode, METH_VARARGS, decode_doc}, - {"__alloc__", (PyCFunction)bytes_alloc, METH_NOARGS, alloc_doc}, + {"endswith", (PyCFunction)bytes_endswith, METH_VARARGS, endswith__doc__}, + {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, + expandtabs__doc__}, + {"extend", (PyCFunction)bytes_extend, METH_O, extend__doc__}, + {"find", (PyCFunction)bytes_find, METH_VARARGS, find__doc__}, {"fromhex", (PyCFunction)bytes_fromhex, METH_VARARGS|METH_CLASS, fromhex_doc}, + {"index", (PyCFunction)bytes_index, METH_VARARGS, index__doc__}, + {"insert", (PyCFunction)bytes_insert, METH_VARARGS, insert__doc__}, + {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS, + _Py_isalnum__doc__}, + {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS, + _Py_isalpha__doc__}, + {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS, + _Py_isdigit__doc__}, + {"islower", (PyCFunction)stringlib_islower, METH_NOARGS, + _Py_islower__doc__}, + {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS, + _Py_isspace__doc__}, + {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS, + _Py_istitle__doc__}, + {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS, + _Py_isupper__doc__}, {"join", (PyCFunction)bytes_join, METH_O, join_doc}, - {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, ljust__doc__}, + {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, + {"lstrip", (PyCFunction)bytes_lstrip, METH_VARARGS, lstrip__doc__}, + {"partition", (PyCFunction)bytes_partition, METH_O, partition__doc__}, + {"pop", (PyCFunction)bytes_pop, METH_VARARGS, pop__doc__}, + {"remove", (PyCFunction)bytes_remove, METH_O, remove__doc__}, + {"replace", (PyCFunction)bytes_replace, METH_VARARGS, replace__doc__}, + {"reverse", (PyCFunction)bytes_reverse, METH_NOARGS, reverse__doc__}, + {"rfind", (PyCFunction)bytes_rfind, METH_VARARGS, rfind__doc__}, + {"rindex", (PyCFunction)bytes_rindex, METH_VARARGS, rindex__doc__}, {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, rjust__doc__}, - {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, - {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, zfill__doc__}, - {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, - expandtabs__doc__}, + {"rpartition", (PyCFunction)bytes_rpartition, METH_O, rpartition__doc__}, + {"rsplit", (PyCFunction)bytes_rsplit, METH_VARARGS, rsplit__doc__}, + {"rstrip", (PyCFunction)bytes_rstrip, METH_VARARGS, rstrip__doc__}, + {"split", (PyCFunction)bytes_split, METH_VARARGS, split__doc__}, {"splitlines", (PyCFunction)stringlib_splitlines, METH_VARARGS, splitlines__doc__}, - {"__reduce__", (PyCFunction)bytes_reduce, METH_NOARGS, reduce_doc}, + {"startswith", (PyCFunction)bytes_startswith, METH_VARARGS , + startswith__doc__}, + {"strip", (PyCFunction)bytes_strip, METH_VARARGS, strip__doc__}, + {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, + _Py_swapcase__doc__}, + {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, + {"translate", (PyCFunction)bytes_translate, METH_VARARGS, + translate__doc__}, + {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, + {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, zfill__doc__}, {NULL} }; Modified: python/branches/py3k/Objects/stringobject.c ============================================================================== --- python/branches/py3k/Objects/stringobject.c (original) +++ python/branches/py3k/Objects/stringobject.c Fri Oct 26 23:46:44 2007 @@ -900,7 +900,7 @@ "strings are too large to concat"); return NULL; } - + /* Inline PyObject_NewVar */ op = (PyStringObject *)PyObject_MALLOC(sizeof(PyStringObject) + size); if (op == NULL) @@ -2260,7 +2260,7 @@ /* 1 at the end plus 1 after every character */ count = self_len+1; - if (maxcount < count) + if (maxcount < count) count = maxcount; /* Check for overflow */ @@ -2277,7 +2277,7 @@ "replace string is too long"); return NULL; } - + if (! (result = (PyStringObject *) PyString_FromStringAndSize(NULL, result_len)) ) return NULL; @@ -2291,7 +2291,7 @@ Py_MEMCPY(result_s, to_s, to_len); result_s += to_len; count -= 1; - + for (i=0; i=0); @@ -2540,7 +2540,7 @@ end = self_s + self_len; while (count-- > 0) { next = findchar(start, end-start, from_c); - if (next == NULL) + if (next == NULL) break; if (next == start) { @@ -2875,43 +2875,6 @@ } -PyDoc_STRVAR(encode__doc__, -"S.encode([encoding[,errors]]) -> object\n\ -\n\ -Encodes S using the codec registered for encoding. encoding defaults\n\ -to the default encoding. errors may be given to set a different error\n\ -handling scheme. Default is 'strict' meaning that encoding errors raise\n\ -a UnicodeEncodeError. Other possible values are 'ignore', 'replace' and\n\ -'xmlcharrefreplace' as well as any other name registered with\n\ -codecs.register_error that is able to handle UnicodeEncodeErrors."); - -static PyObject * -string_encode(PyStringObject *self, PyObject *args) -{ - char *encoding = NULL; - char *errors = NULL; - PyObject *v; - - if (!PyArg_ParseTuple(args, "|ss:encode", &encoding, &errors)) - return NULL; - v = PyString_AsEncodedObject((PyObject *)self, encoding, errors); - if (v == NULL) - goto onError; - if (!PyBytes_Check(v)) { - PyErr_Format(PyExc_TypeError, - "[str8] encoder did not return a bytes object " - "(type=%.400s)", - Py_Type(v)->tp_name); - Py_DECREF(v); - return NULL; - } - return v; - - onError: - return NULL; -} - - PyDoc_STRVAR(decode__doc__, "S.decode([encoding[,errors]]) -> object\n\ \n\ @@ -2949,6 +2912,74 @@ } +PyDoc_STRVAR(fromhex_doc, +"str8.fromhex(string) -> str8\n\ +\n\ +Create a str8 object from a string of hexadecimal numbers.\n\ +Spaces between two numbers are accepted. Example:\n\ +str8.fromhex('10 1112') -> s'\\x10\\x11\\x12'."); + +static int +hex_digit_to_int(Py_UNICODE c) +{ + if (c >= 128) + return -1; + if (ISDIGIT(c)) + return c - '0'; + else { + if (ISUPPER(c)) + c = TOLOWER(c); + if (c >= 'a' && c <= 'f') + return c - 'a' + 10; + } + return -1; +} + +static PyObject * +string_fromhex(PyObject *cls, PyObject *args) +{ + PyObject *newstring, *hexobj; + char *buf; + Py_UNICODE *hex; + Py_ssize_t hexlen, byteslen, i, j; + int top, bot; + + if (!PyArg_ParseTuple(args, "U:fromhex", &hexobj)) + return NULL; + assert(PyUnicode_Check(hexobj)); + hexlen = PyUnicode_GET_SIZE(hexobj); + hex = PyUnicode_AS_UNICODE(hexobj); + byteslen = hexlen/2; /* This overestimates if there are spaces */ + newstring = PyString_FromStringAndSize(NULL, byteslen); + if (!newstring) + return NULL; + buf = PyString_AS_STRING(newstring); + for (i = j = 0; i < hexlen; i += 2) { + /* skip over spaces in the input */ + while (hex[i] == ' ') + i++; + if (i >= hexlen) + break; + top = hex_digit_to_int(hex[i]); + bot = hex_digit_to_int(hex[i+1]); + if (top == -1 || bot == -1) { + PyErr_Format(PyExc_ValueError, + "non-hexadecimal number found in " + "fromhex() arg at position %zd", i); + goto error; + } + buf[j++] = (top << 4) + bot; + } + if (_PyString_Resize(&newstring, j) < 0) + goto error; + return newstring; + + error: + Py_DECREF(newstring); + return NULL; +} + + static PyObject * string_getnewargs(PyStringObject *v) { @@ -2958,59 +2989,60 @@ static PyMethodDef string_methods[] = { - {"join", (PyCFunction)string_join, METH_O, join__doc__}, - {"split", (PyCFunction)string_split, METH_VARARGS, split__doc__}, - {"rsplit", (PyCFunction)string_rsplit, METH_VARARGS, rsplit__doc__}, - {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, - {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, - {"islower", (PyCFunction)stringlib_islower, METH_NOARGS, - _Py_islower__doc__}, - {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS, - _Py_isupper__doc__}, - {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS, - _Py_isspace__doc__}, - {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS, - _Py_isdigit__doc__}, - {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS, - _Py_istitle__doc__}, - {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS, - _Py_isalpha__doc__}, - {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS, - _Py_isalnum__doc__}, + {"__getnewargs__", (PyCFunction)string_getnewargs, METH_NOARGS}, {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, _Py_capitalize__doc__}, + {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, {"count", (PyCFunction)string_count, METH_VARARGS, count__doc__}, + {"decode", (PyCFunction)string_decode, METH_VARARGS, decode__doc__}, {"endswith", (PyCFunction)string_endswith, METH_VARARGS, - endswith__doc__}, - {"partition", (PyCFunction)string_partition, METH_O, partition__doc__}, + endswith__doc__}, + {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, + expandtabs__doc__}, {"find", (PyCFunction)string_find, METH_VARARGS, find__doc__}, + {"fromhex", (PyCFunction)string_fromhex, METH_VARARGS|METH_CLASS, + fromhex_doc}, {"index", (PyCFunction)string_index, METH_VARARGS, index__doc__}, + {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS, + _Py_isalnum__doc__}, + {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS, + _Py_isalpha__doc__}, + {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS, + _Py_isdigit__doc__}, + {"islower", (PyCFunction)stringlib_islower, METH_NOARGS, + _Py_islower__doc__}, + {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS, + _Py_isspace__doc__}, + {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS, + _Py_istitle__doc__}, + {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS, + _Py_isupper__doc__}, + {"join", (PyCFunction)string_join, METH_O, join__doc__}, + {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, ljust__doc__}, + {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, {"lstrip", (PyCFunction)string_lstrip, METH_VARARGS, lstrip__doc__}, + {"partition", (PyCFunction)string_partition, METH_O, partition__doc__}, {"replace", (PyCFunction)string_replace, METH_VARARGS, replace__doc__}, {"rfind", (PyCFunction)string_rfind, METH_VARARGS, rfind__doc__}, {"rindex", (PyCFunction)string_rindex, METH_VARARGS, rindex__doc__}, - {"rstrip", (PyCFunction)string_rstrip, METH_VARARGS, rstrip__doc__}, + {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, rjust__doc__}, {"rpartition", (PyCFunction)string_rpartition, METH_O, rpartition__doc__}, + {"rsplit", (PyCFunction)string_rsplit, METH_VARARGS, rsplit__doc__}, + {"rstrip", (PyCFunction)string_rstrip, METH_VARARGS, rstrip__doc__}, + {"split", (PyCFunction)string_split, METH_VARARGS, split__doc__}, + {"splitlines", (PyCFunction)stringlib_splitlines, METH_VARARGS, + splitlines__doc__}, {"startswith", (PyCFunction)string_startswith, METH_VARARGS, - startswith__doc__}, + startswith__doc__}, {"strip", (PyCFunction)string_strip, METH_VARARGS, strip__doc__}, {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, _Py_swapcase__doc__}, + {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, {"translate", (PyCFunction)string_translate, METH_VARARGS, translate__doc__}, - {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, - {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, ljust__doc__}, - {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, rjust__doc__}, - {"center", (PyCFunction)stringlib_center, METH_VARARGS, center__doc__}, + {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, zfill__doc__}, - {"encode", (PyCFunction)string_encode, METH_VARARGS, encode__doc__}, - {"decode", (PyCFunction)string_decode, METH_VARARGS, decode__doc__}, - {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS, - expandtabs__doc__}, - {"splitlines", (PyCFunction)stringlib_splitlines, METH_VARARGS, - splitlines__doc__}, - {"__getnewargs__", (PyCFunction)string_getnewargs, METH_NOARGS}, {NULL, NULL} /* sentinel */ }; From python-3000-checkins at python.org Fri Oct 26 23:50:10 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Fri, 26 Oct 2007 23:50:10 +0200 (CEST) Subject: [Python-3000-checkins] r58679 - python/branches/py3k-pep3137 Message-ID: <20071026215010.955601E4002@bag.python.org> Author: guido.van.rossum Date: Fri Oct 26 23:50:10 2007 New Revision: 58679 Added: python/branches/py3k-pep3137/ - copied from r58678, python/branches/py3k/ Log: Create temporary branch for working on PEP 3137 stuff. From python-3000-checkins at python.org Sat Oct 27 00:35:34 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 00:35:34 +0200 (CEST) Subject: [Python-3000-checkins] r58680 - in python/branches/py3k-pep3137: Include/stringobject.h Objects/stringobject.c Message-ID: <20071026223535.008351E4002@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 00:35:34 2007 New Revision: 58680 Modified: python/branches/py3k-pep3137/Include/stringobject.h python/branches/py3k-pep3137/Objects/stringobject.c Log: Killed the remaining PyString_ decode/encode methods. Cleaned up some indentation. Modified: python/branches/py3k-pep3137/Include/stringobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/stringobject.h (original) +++ python/branches/py3k-pep3137/Include/stringobject.h Sat Oct 27 00:35:34 2007 @@ -83,77 +83,19 @@ #define PyString_CHECK_INTERNED(op) (((PyStringObject *)(op))->ob_sstate) /* Macro, trading safety for speed */ -#define PyString_AS_STRING(op) (assert(PyString_Check(op)),(((PyStringObject *)(op))->ob_sval)) +#define PyString_AS_STRING(op) (assert(PyString_Check(op)), \ + (((PyStringObject *)(op))->ob_sval)) #define PyString_GET_SIZE(op) (assert(PyString_Check(op)),Py_Size(op)) /* _PyString_Join(sep, x) is like sep.join(x). sep must be PyStringObject*, x must be an iterable object. */ PyAPI_FUNC(PyObject *) _PyString_Join(PyObject *sep, PyObject *x); -/* --- Generic Codecs ----------------------------------------------------- */ - -/* Create an object by decoding the encoded string s of the - given size. */ - -PyAPI_FUNC(PyObject*) PyString_Decode( - const char *s, /* encoded string */ - Py_ssize_t size, /* size of buffer */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - -/* Encodes a string object and returns the result as Python - object. */ - -PyAPI_FUNC(PyObject*) PyString_AsEncodedObject( - PyObject *str, /* string object */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - -/* Encodes a string object and returns the result as Python string - object. - - If the codec returns an Unicode object, the object is converted - back to a string using the default encoding. - - DEPRECATED - use PyString_AsEncodedObject() instead. */ - -PyAPI_FUNC(PyObject*) PyString_AsEncodedString( - PyObject *str, /* string object */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - -/* Decodes a string object and returns the result as Python - object. */ - -PyAPI_FUNC(PyObject*) PyString_AsDecodedObject( - PyObject *str, /* string object */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - -/* Decodes a string object and returns the result as Python string - object. - - If the codec returns an Unicode object, the object is converted - back to a string using the default encoding. - - DEPRECATED - use PyString_AsDecodedObject() instead. */ - -PyAPI_FUNC(PyObject*) PyString_AsDecodedString( - PyObject *str, /* string object */ - const char *encoding, /* encoding */ - const char *errors /* error handling */ - ); - /* Provides access to the internal data buffer and size of a string object or the default encoded version of an Unicode object. Passing NULL as *len parameter will force the string buffer to be 0-terminated (passing a string with embedded NULL characters will cause an exception). */ - PyAPI_FUNC(int) PyString_AsStringAndSize( register PyObject *obj, /* string or Unicode object */ register char **s, /* pointer to buffer variable */ Modified: python/branches/py3k-pep3137/Objects/stringobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/stringobject.c (original) +++ python/branches/py3k-pep3137/Objects/stringobject.c Sat Oct 27 00:35:34 2007 @@ -351,153 +351,6 @@ return ret; } - -PyObject *PyString_Decode(const char *s, - Py_ssize_t size, - const char *encoding, - const char *errors) -{ - PyObject *v, *str; - - str = PyString_FromStringAndSize(s, size); - if (str == NULL) - return NULL; - v = PyString_AsDecodedString(str, encoding, errors); - Py_DECREF(str); - return v; -} - -PyObject *PyString_AsDecodedObject(PyObject *str, - const char *encoding, - const char *errors) -{ - PyObject *v; - - if (!PyString_Check(str)) { - PyErr_BadArgument(); - goto onError; - } - - if (encoding == NULL) { - encoding = PyUnicode_GetDefaultEncoding(); - } - - /* Decode via the codec registry */ - v = PyCodec_Decode(str, encoding, errors); - if (v == NULL) - goto onError; - - return v; - - onError: - return NULL; -} - -PyObject *PyString_AsDecodedString(PyObject *str, - const char *encoding, - const char *errors) -{ - PyObject *v; - - v = PyString_AsDecodedObject(str, encoding, errors); - if (v == NULL) - goto onError; - - /* Convert Unicode to a string using the default encoding */ - if (PyUnicode_Check(v)) { - PyObject *temp = v; - v = PyUnicode_AsEncodedString(v, NULL, NULL); - Py_DECREF(temp); - if (v == NULL) - goto onError; - } - if (!PyString_Check(v)) { - PyErr_Format(PyExc_TypeError, - "decoder did not return a string object (type=%.400s)", - Py_Type(v)->tp_name); - Py_DECREF(v); - goto onError; - } - - return v; - - onError: - return NULL; -} - -PyObject *PyString_Encode(const char *s, - Py_ssize_t size, - const char *encoding, - const char *errors) -{ - PyObject *v, *str; - - str = PyString_FromStringAndSize(s, size); - if (str == NULL) - return NULL; - v = PyString_AsEncodedString(str, encoding, errors); - Py_DECREF(str); - return v; -} - -PyObject *PyString_AsEncodedObject(PyObject *str, - const char *encoding, - const char *errors) -{ - PyObject *v; - - if (!PyString_Check(str)) { - PyErr_BadArgument(); - goto onError; - } - - if (encoding == NULL) { - encoding = PyUnicode_GetDefaultEncoding(); - } - - /* Encode via the codec registry */ - v = PyCodec_Encode(str, encoding, errors); - if (v == NULL) - goto onError; - - return v; - - onError: - return NULL; -} - -PyObject *PyString_AsEncodedString(PyObject *str, - const char *encoding, - const char *errors) -{ - PyObject *v; - - v = PyString_AsEncodedObject(str, encoding, errors); - if (v == NULL) - goto onError; - - /* Convert Unicode to a string using the default encoding */ - if (PyUnicode_Check(v)) { - PyObject *temp = v; - v = PyUnicode_AsEncodedString(v, NULL, NULL); - Py_DECREF(temp); - if (v == NULL) - goto onError; - } - if (!PyString_Check(v)) { - PyErr_Format(PyExc_TypeError, - "encoder did not return a string object (type=%.400s)", - Py_Type(v)->tp_name); - Py_DECREF(v); - goto onError; - } - - return v; - - onError: - return NULL; -} - static void string_dealloc(PyObject *op) { @@ -577,7 +430,7 @@ continue; } s++; - if (s==end) { + if (s==end) { PyErr_SetString(PyExc_ValueError, "Trailing \\ in string"); goto failed; @@ -665,8 +518,8 @@ static Py_ssize_t string_getsize(register PyObject *op) { - char *s; - Py_ssize_t len; + char *s; + Py_ssize_t len; if (PyString_AsStringAndSize(op, &s, &len)) return -1; return len; @@ -675,8 +528,8 @@ static /*const*/ char * string_getbuffer(register PyObject *op) { - char *s; - Py_ssize_t len; + char *s; + Py_ssize_t len; if (PyString_AsStringAndSize(op, &s, &len)) return NULL; return s; @@ -876,7 +729,7 @@ if (!PyString_Check(bb)) { if (PyUnicode_Check(bb)) return PyUnicode_Concat((PyObject *)a, bb); - if (PyBytes_Check(bb)) + if (PyBytes_Check(bb)) return PyBytes_Concat((PyObject *)a, bb); PyErr_Format(PyExc_TypeError, "cannot concatenate 'str8' and '%.200s' objects", @@ -1053,9 +906,9 @@ { PyStringObject *a = (PyStringObject*) o1; PyStringObject *b = (PyStringObject*) o2; - return Py_Size(a) == Py_Size(b) - && *a->ob_sval == *b->ob_sval - && memcmp(a->ob_sval, b->ob_sval, Py_Size(a)) == 0; + return Py_Size(a) == Py_Size(b) + && *a->ob_sval == *b->ob_sval + && memcmp(a->ob_sval, b->ob_sval, Py_Size(a)) == 0; } static long @@ -1088,12 +941,12 @@ return NULL; if (i < 0) i += PyString_GET_SIZE(self); - if (i < 0 || i >= PyString_GET_SIZE(self)) { + if (i < 0 || i >= PyString_GET_SIZE(self)) { PyErr_SetString(PyExc_IndexError, "string index out of range"); return NULL; - } - return PyInt_FromLong((unsigned char)self->ob_sval[i]); + } + return PyInt_FromLong((unsigned char)self->ob_sval[i]); } else if (PySlice_Check(item)) { Py_ssize_t start, stop, step, slicelength, cur, i; @@ -1149,7 +1002,7 @@ static int string_buffer_getbuffer(PyStringObject *self, Py_buffer *view, int flags) { - return PyBuffer_FillInfo(view, (void *)self->ob_sval, Py_Size(self), 0, flags); + return PyBuffer_FillInfo(view, (void *)self->ob_sval, Py_Size(self), 0, flags); } static PySequenceMethods string_as_sequence = { @@ -1171,7 +1024,7 @@ static PyBufferProcs string_as_buffer = { (getbufferproc)string_buffer_getbuffer, - NULL, + NULL, }; @@ -2888,27 +2741,14 @@ static PyObject * string_decode(PyStringObject *self, PyObject *args) { - char *encoding = NULL; - char *errors = NULL; - PyObject *v; - - if (!PyArg_ParseTuple(args, "|ss:decode", &encoding, &errors)) - return NULL; - v = PyString_AsDecodedObject((PyObject *)self, encoding, errors); - if (v == NULL) - goto onError; - if (!PyString_Check(v) && !PyUnicode_Check(v)) { - PyErr_Format(PyExc_TypeError, - "decoder did not return a string/unicode object " - "(type=%.400s)", - Py_Type(v)->tp_name); - Py_DECREF(v); - return NULL; - } - return v; + char *encoding = NULL; + char *errors = NULL; - onError: - return NULL; + if (!PyArg_ParseTuple(args, "|ss:decode", &encoding, &errors)) + return NULL; + if (encoding == NULL) + encoding = PyUnicode_GetDefaultEncoding(); + return PyCodec_Decode(self, encoding, errors); } @@ -2922,17 +2762,17 @@ static int hex_digit_to_int(Py_UNICODE c) { - if (c >= 128) - return -1; - if (ISDIGIT(c)) - return c - '0'; - else { - if (ISUPPER(c)) - c = TOLOWER(c); - if (c >= 'a' && c <= 'f') - return c - 'a' + 10; - } - return -1; + if (c >= 128) + return -1; + if (ISDIGIT(c)) + return c - '0'; + else { + if (ISUPPER(c)) + c = TOLOWER(c); + if (c >= 'a' && c <= 'f') + return c - 'a' + 10; + } + return -1; } static PyObject * @@ -3392,7 +3232,7 @@ Py_LOCAL_INLINE(int) formatfloat(char *buf, size_t buflen, int flags, - int prec, int type, PyObject *v) + int prec, int type, PyObject *v) { /* fmt = '%#.' + `prec` + `type` worst case length = 3 + 10 (len of INT_MAX) + 1 = 14 (use 20)*/ @@ -3425,7 +3265,7 @@ */ if (((type == 'g' || type == 'G') && - buflen <= (size_t)10 + (size_t)prec) || + buflen <= (size_t)10 + (size_t)prec) || (type == 'f' && buflen <= (size_t)53 + (size_t)prec)) { PyErr_SetString(PyExc_OverflowError, "formatted float is too long (precision too large?)"); @@ -3434,7 +3274,7 @@ PyOS_snprintf(fmt, sizeof(fmt), "%%%s.%d%c", (flags&F_ALT) ? "#" : "", prec, type); - PyOS_ascii_formatd(buf, buflen, fmt, x); + PyOS_ascii_formatd(buf, buflen, fmt, x); return (int)strlen(buf); } @@ -3534,7 +3374,7 @@ (type == 'o' || type == 'x' || type == 'X'))) { assert(buf[sign] == '0'); assert(buf[sign+1] == 'x' || buf[sign+1] == 'X' || - buf[sign+1] == 'o'); + buf[sign+1] == 'o'); numnondigits -= 2; buf += 2; len -= 2; @@ -3582,7 +3422,7 @@ Py_LOCAL_INLINE(int) formatint(char *buf, size_t buflen, int flags, - int prec, int type, PyObject *v) + int prec, int type, PyObject *v) { /* fmt = '%#.' + `prec` + 'l' + `type` worst case length = 3 + 19 (worst len of INT_MAX on 64-bit machine) @@ -4065,18 +3905,20 @@ --rescnt; *res++ = ' '; } - if (dict && (argidx < arglen) && c != '%') { - PyErr_SetString(PyExc_TypeError, - "not all arguments converted during string formatting"); - Py_XDECREF(temp); - goto error; - } + if (dict && (argidx < arglen) && c != '%') { + PyErr_SetString(PyExc_TypeError, + "not all arguments converted " + "during string formatting"); + Py_XDECREF(temp); + goto error; + } Py_XDECREF(temp); } /* '%' */ } /* until end */ if (argidx < arglen && !dict) { PyErr_SetString(PyExc_TypeError, - "not all arguments converted during string formatting"); + "not all arguments converted " + "during string formatting"); goto error; } if (args_owned) { From python-3000-checkins at python.org Sat Oct 27 01:10:45 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 01:10:45 +0200 (CEST) Subject: [Python-3000-checkins] r58681 - in python/branches/py3k-pep3137: Include/stringobject.h Modules/_sqlite/cache.c Objects/stringobject.c Objects/unicodeobject.c Message-ID: <20071026231045.0FB891E402A@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 01:10:44 2007 New Revision: 58681 Modified: python/branches/py3k-pep3137/Include/stringobject.h python/branches/py3k-pep3137/Modules/_sqlite/cache.c python/branches/py3k-pep3137/Objects/stringobject.c python/branches/py3k-pep3137/Objects/unicodeobject.c Log: Fix compiler warnings in str8.decode() that I didn't notice before. Get rid of str8() % ... and PyString_Format(). Modified: python/branches/py3k-pep3137/Include/stringobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/stringobject.h (original) +++ python/branches/py3k-pep3137/Include/stringobject.h Sat Oct 27 01:10:44 2007 @@ -104,6 +104,12 @@ strings) */ ); +/* Flags used by string formatting */ +#define F_LJUST (1<<0) +#define F_SIGN (1<<1) +#define F_BLANK (1<<2) +#define F_ALT (1<<3) +#define F_ZERO (1<<4) #ifdef __cplusplus } Modified: python/branches/py3k-pep3137/Modules/_sqlite/cache.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_sqlite/cache.c (original) +++ python/branches/py3k-pep3137/Modules/_sqlite/cache.c Sat Oct 27 01:10:44 2007 @@ -246,7 +246,7 @@ Py_DECREF(fmt_args); return NULL; } - display_str = PyString_Format(template, fmt_args); + display_str = PyUnicode_Format(template, fmt_args); Py_DECREF(template); Py_DECREF(fmt_args); if (!display_str) { Modified: python/branches/py3k-pep3137/Objects/stringobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/stringobject.c (original) +++ python/branches/py3k-pep3137/Objects/stringobject.c Sat Oct 27 01:10:44 2007 @@ -2739,10 +2739,10 @@ able to handle UnicodeDecodeErrors."); static PyObject * -string_decode(PyStringObject *self, PyObject *args) +string_decode(PyObject *self, PyObject *args) { - char *encoding = NULL; - char *errors = NULL; + const char *encoding = NULL; + const char *errors = NULL; if (!PyArg_ParseTuple(args, "|ss:decode", &encoding, &errors)) return NULL; @@ -3062,23 +3062,6 @@ return pnew; } -static PyObject * -string_mod(PyObject *v, PyObject *w) -{ - if (!PyString_Check(v)) { - Py_INCREF(Py_NotImplemented); - return Py_NotImplemented; - } - return PyString_Format(v, w); -} - -static PyNumberMethods string_as_number = { - 0, /*nb_add*/ - 0, /*nb_subtract*/ - 0, /*nb_multiply*/ - string_mod, /*nb_remainder*/ -}; - PyDoc_STRVAR(string_doc, "str(object) -> string\n\ \n\ @@ -3098,7 +3081,7 @@ 0, /* tp_setattr */ 0, /* tp_compare */ string_repr, /* tp_repr */ - &string_as_number, /* tp_as_number */ + 0, /* tp_as_number */ &string_as_sequence, /* tp_as_sequence */ &string_as_mapping, /* tp_as_mapping */ (hashfunc)string_hash, /* tp_hash */ @@ -3199,85 +3182,6 @@ return 0; } -/* Helpers for formatstring */ - -Py_LOCAL_INLINE(PyObject *) -getnextarg(PyObject *args, Py_ssize_t arglen, Py_ssize_t *p_argidx) -{ - Py_ssize_t argidx = *p_argidx; - if (argidx < arglen) { - (*p_argidx)++; - if (arglen < 0) - return args; - else - return PyTuple_GetItem(args, argidx); - } - PyErr_SetString(PyExc_TypeError, - "not enough arguments for format string"); - return NULL; -} - -/* Format codes - * F_LJUST '-' - * F_SIGN '+' - * F_BLANK ' ' - * F_ALT '#' - * F_ZERO '0' - */ -#define F_LJUST (1<<0) -#define F_SIGN (1<<1) -#define F_BLANK (1<<2) -#define F_ALT (1<<3) -#define F_ZERO (1<<4) - -Py_LOCAL_INLINE(int) -formatfloat(char *buf, size_t buflen, int flags, - int prec, int type, PyObject *v) -{ - /* fmt = '%#.' + `prec` + `type` - worst case length = 3 + 10 (len of INT_MAX) + 1 = 14 (use 20)*/ - char fmt[20]; - double x; - x = PyFloat_AsDouble(v); - if (x == -1.0 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, "float argument required, " - "not %.200s", Py_Type(v)->tp_name); - return -1; - } - if (prec < 0) - prec = 6; - if (type == 'f' && fabs(x)/1e25 >= 1e25) - type = 'g'; - /* Worst case length calc to ensure no buffer overrun: - - 'g' formats: - fmt = %#.g - buf = '-' + [0-9]*prec + '.' + 'e+' + (longest exp - for any double rep.) - len = 1 + prec + 1 + 2 + 5 = 9 + prec - - 'f' formats: - buf = '-' + [0-9]*x + '.' + [0-9]*prec (with x < 50) - len = 1 + 50 + 1 + prec = 52 + prec - - If prec=0 the effective precision is 1 (the leading digit is - always given), therefore increase the length by one. - - */ - if (((type == 'g' || type == 'G') && - buflen <= (size_t)10 + (size_t)prec) || - (type == 'f' && buflen <= (size_t)53 + (size_t)prec)) { - PyErr_SetString(PyExc_OverflowError, - "formatted float is too long (precision too large?)"); - return -1; - } - PyOS_snprintf(fmt, sizeof(fmt), "%%%s.%d%c", - (flags&F_ALT) ? "#" : "", - prec, type); - PyOS_ascii_formatd(buf, buflen, fmt, x); - return (int)strlen(buf); -} - /* _PyString_FormatLong emulates the format codes d, u, o, x and X, and * the F_ALT flag, for Python's long (unbounded) ints. It's not used for * Python's regular ints. @@ -3356,7 +3260,8 @@ } llen = PyString_Size(result); if (llen > INT_MAX) { - PyErr_SetString(PyExc_ValueError, "string too large in _PyString_FormatLong"); + PyErr_SetString(PyExc_ValueError, + "string too large in _PyString_FormatLong"); return NULL; } len = (int)llen; @@ -3420,566 +3325,6 @@ return result; } -Py_LOCAL_INLINE(int) -formatint(char *buf, size_t buflen, int flags, - int prec, int type, PyObject *v) -{ - /* fmt = '%#.' + `prec` + 'l' + `type` - worst case length = 3 + 19 (worst len of INT_MAX on 64-bit machine) - + 1 + 1 = 24 */ - char fmt[64]; /* plenty big enough! */ - char *sign; - long x; - - x = PyInt_AsLong(v); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, "int argument required, not %.200s", - Py_Type(v)->tp_name); - return -1; - } - if (x < 0 && type == 'u') { - type = 'd'; - } - if (x < 0 && (type == 'x' || type == 'X' || type == 'o')) - sign = "-"; - else - sign = ""; - if (prec < 0) - prec = 1; - - if ((flags & F_ALT) && - (type == 'x' || type == 'X' || type == 'o')) { - /* When converting under %#o, %#x or %#X, there are a number - * of issues that cause pain: - * - for %#o, we want a different base marker than C - * - when 0 is being converted, the C standard leaves off - * the '0x' or '0X', which is inconsistent with other - * %#x/%#X conversions and inconsistent with Python's - * hex() function - * - there are platforms that violate the standard and - * convert 0 with the '0x' or '0X' - * (Metrowerks, Compaq Tru64) - * - there are platforms that give '0x' when converting - * under %#X, but convert 0 in accordance with the - * standard (OS/2 EMX) - * - * We can achieve the desired consistency by inserting our - * own '0x' or '0X' prefix, and substituting %x/%X in place - * of %#x/%#X. - * - * Note that this is the same approach as used in - * formatint() in unicodeobject.c - */ - PyOS_snprintf(fmt, sizeof(fmt), "%s0%c%%.%dl%c", - sign, type, prec, type); - } - else { - PyOS_snprintf(fmt, sizeof(fmt), "%s%%%s.%dl%c", - sign, (flags&F_ALT) ? "#" : "", - prec, type); - } - - /* buf = '+'/'-'/'' + '0o'/'0x'/'' + '[0-9]'*max(prec, len(x in octal)) - * worst case buf = '-0x' + [0-9]*prec, where prec >= 11 - */ - if (buflen <= 14 || buflen <= (size_t)3 + (size_t)prec) { - PyErr_SetString(PyExc_OverflowError, - "formatted integer is too long (precision too large?)"); - return -1; - } - if (sign[0]) - PyOS_snprintf(buf, buflen, fmt, -x); - else - PyOS_snprintf(buf, buflen, fmt, x); - return (int)strlen(buf); -} - -Py_LOCAL_INLINE(int) -formatchar(char *buf, size_t buflen, PyObject *v) -{ - /* presume that the buffer is at least 2 characters long */ - if (PyString_Check(v)) { - if (!PyArg_Parse(v, "c;%c requires int or char", &buf[0])) - return -1; - } - else { - if (!PyArg_Parse(v, "b;%c requires int or char", &buf[0])) - return -1; - } - buf[1] = '\0'; - return 1; -} - -/* fmt%(v1,v2,...) is roughly equivalent to sprintf(fmt, v1, v2, ...) - - FORMATBUFLEN is the length of the buffer in which the floats, ints, & - chars are formatted. XXX This is a magic number. Each formatting - routine does bounds checking to ensure no overflow, but a better - solution may be to malloc a buffer of appropriate size for each - format. For now, the current solution is sufficient. -*/ -#define FORMATBUFLEN (size_t)120 - -PyObject * -PyString_Format(PyObject *format, PyObject *args) -{ - char *fmt, *res; - Py_ssize_t arglen, argidx; - Py_ssize_t reslen, rescnt, fmtcnt; - int args_owned = 0; - PyObject *result, *orig_args; - PyObject *v, *w; - PyObject *dict = NULL; - if (format == NULL || !PyString_Check(format) || args == NULL) { - PyErr_BadInternalCall(); - return NULL; - } - orig_args = args; - fmt = PyString_AS_STRING(format); - fmtcnt = PyString_GET_SIZE(format); - reslen = rescnt = fmtcnt + 100; - result = PyString_FromStringAndSize((char *)NULL, reslen); - if (result == NULL) - return NULL; - res = PyString_AsString(result); - if (PyTuple_Check(args)) { - arglen = PyTuple_GET_SIZE(args); - argidx = 0; - } - else { - arglen = -1; - argidx = -2; - } - if (Py_Type(args)->tp_as_mapping && !PyTuple_Check(args) && - !PyString_Check(args) && !PyUnicode_Check(args)) - dict = args; - while (--fmtcnt >= 0) { - if (*fmt != '%') { - if (--rescnt < 0) { - rescnt = fmtcnt + 100; - reslen += rescnt; - if (_PyString_Resize(&result, reslen) < 0) - return NULL; - res = PyString_AS_STRING(result) - + reslen - rescnt; - --rescnt; - } - *res++ = *fmt++; - } - else { - /* Got a format specifier */ - int flags = 0; - Py_ssize_t width = -1; - int prec = -1; - int c = '\0'; - int fill; - PyObject *v = NULL; - PyObject *temp = NULL; - char *pbuf; - int sign; - Py_ssize_t len; - char formatbuf[FORMATBUFLEN]; - /* For format{float,int,char}() */ - char *fmt_start = fmt; - Py_ssize_t argidx_start = argidx; - - fmt++; - if (*fmt == '(') { - char *keystart; - Py_ssize_t keylen; - PyObject *key; - int pcount = 1; - - if (dict == NULL) { - PyErr_SetString(PyExc_TypeError, - "format requires a mapping"); - goto error; - } - ++fmt; - --fmtcnt; - keystart = fmt; - /* Skip over balanced parentheses */ - while (pcount > 0 && --fmtcnt >= 0) { - if (*fmt == ')') - --pcount; - else if (*fmt == '(') - ++pcount; - fmt++; - } - keylen = fmt - keystart - 1; - if (fmtcnt < 0 || pcount > 0) { - PyErr_SetString(PyExc_ValueError, - "incomplete format key"); - goto error; - } - key = PyString_FromStringAndSize(keystart, - keylen); - if (key == NULL) - goto error; - if (args_owned) { - Py_DECREF(args); - args_owned = 0; - } - args = PyObject_GetItem(dict, key); - Py_DECREF(key); - if (args == NULL) { - goto error; - } - args_owned = 1; - arglen = -1; - argidx = -2; - } - while (--fmtcnt >= 0) { - switch (c = *fmt++) { - case '-': flags |= F_LJUST; continue; - case '+': flags |= F_SIGN; continue; - case ' ': flags |= F_BLANK; continue; - case '#': flags |= F_ALT; continue; - case '0': flags |= F_ZERO; continue; - } - break; - } - if (c == '*') { - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - if (!PyInt_Check(v)) { - PyErr_SetString(PyExc_TypeError, - "* wants int"); - goto error; - } - width = PyInt_AsLong(v); - if (width == -1 && PyErr_Occurred()) - goto error; - if (width < 0) { - flags |= F_LJUST; - width = -width; - } - if (--fmtcnt >= 0) - c = *fmt++; - } - else if (c >= 0 && ISDIGIT(c)) { - width = c - '0'; - while (--fmtcnt >= 0) { - c = Py_CHARMASK(*fmt++); - if (!ISDIGIT(c)) - break; - if ((width*10) / 10 != width) { - PyErr_SetString( - PyExc_ValueError, - "width too big"); - goto error; - } - width = width*10 + (c - '0'); - } - } - if (c == '.') { - prec = 0; - if (--fmtcnt >= 0) - c = *fmt++; - if (c == '*') { - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - if (!PyInt_Check(v)) { - PyErr_SetString( - PyExc_TypeError, - "* wants int"); - goto error; - } - prec = PyInt_AsLong(v); - if (prec == -1 && PyErr_Occurred()) - goto error; - if (prec < 0) - prec = 0; - if (--fmtcnt >= 0) - c = *fmt++; - } - else if (c >= 0 && ISDIGIT(c)) { - prec = c - '0'; - while (--fmtcnt >= 0) { - c = Py_CHARMASK(*fmt++); - if (!ISDIGIT(c)) - break; - if ((prec*10) / 10 != prec) { - PyErr_SetString( - PyExc_ValueError, - "prec too big"); - goto error; - } - prec = prec*10 + (c - '0'); - } - } - } /* prec */ - if (fmtcnt >= 0) { - if (c == 'h' || c == 'l' || c == 'L') { - if (--fmtcnt >= 0) - c = *fmt++; - } - } - if (fmtcnt < 0) { - PyErr_SetString(PyExc_ValueError, - "incomplete format"); - goto error; - } - if (c != '%') { - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - } - sign = 0; - fill = ' '; - switch (c) { - case '%': - pbuf = "%"; - len = 1; - break; - case 's': - if (PyUnicode_Check(v)) { - fmt = fmt_start; - argidx = argidx_start; - goto unicode; - } - temp = _PyObject_Str(v); - if (temp != NULL && PyUnicode_Check(temp)) { - Py_DECREF(temp); - fmt = fmt_start; - argidx = argidx_start; - goto unicode; - } - /* Fall through */ - case 'r': - if (c == 'r') - temp = PyObject_ReprStr8(v); - if (temp == NULL) - goto error; - if (!PyString_Check(temp)) { - PyErr_SetString(PyExc_TypeError, - "%s argument has non-string str()/repr()"); - Py_DECREF(temp); - goto error; - } - pbuf = PyString_AS_STRING(temp); - len = PyString_GET_SIZE(temp); - if (prec >= 0 && len > prec) - len = prec; - break; - case 'i': - case 'd': - case 'u': - case 'o': - case 'x': - case 'X': - if (c == 'i') - c = 'd'; - if (PyLong_Check(v)) { - int ilen; - temp = _PyString_FormatLong(v, flags, - prec, c, &pbuf, &ilen); - len = ilen; - if (!temp) - goto error; - sign = 1; - } - else { - pbuf = formatbuf; - len = formatint(pbuf, - sizeof(formatbuf), - flags, prec, c, v); - if (len < 0) - goto error; - sign = 1; - } - if (flags & F_ZERO) - fill = '0'; - break; - case 'e': - case 'E': - case 'f': - case 'F': - case 'g': - case 'G': - if (c == 'F') - c = 'f'; - pbuf = formatbuf; - len = formatfloat(pbuf, sizeof(formatbuf), - flags, prec, c, v); - if (len < 0) - goto error; - sign = 1; - if (flags & F_ZERO) - fill = '0'; - break; - case 'c': - if (PyUnicode_Check(v)) { - fmt = fmt_start; - argidx = argidx_start; - goto unicode; - } - pbuf = formatbuf; - len = formatchar(pbuf, sizeof(formatbuf), v); - if (len < 0) - goto error; - break; - default: - PyErr_Format(PyExc_ValueError, - "unsupported format character '%c' (0x%x) " - "at index %zd", - c, c, - (Py_ssize_t)(fmt - 1 - - PyString_AsString(format))); - goto error; - } - if (sign) { - if (*pbuf == '-' || *pbuf == '+') { - sign = *pbuf++; - len--; - } - else if (flags & F_SIGN) - sign = '+'; - else if (flags & F_BLANK) - sign = ' '; - else - sign = 0; - } - if (width < len) - width = len; - if (rescnt - (sign != 0) < width) { - reslen -= rescnt; - rescnt = width + fmtcnt + 100; - reslen += rescnt; - if (reslen < 0) { - Py_DECREF(result); - Py_XDECREF(temp); - return PyErr_NoMemory(); - } - if (_PyString_Resize(&result, reslen) < 0) { - Py_XDECREF(temp); - return NULL; - } - res = PyString_AS_STRING(result) - + reslen - rescnt; - } - if (sign) { - if (fill != ' ') - *res++ = sign; - rescnt--; - if (width > len) - width--; - } - if ((flags & F_ALT) && - (c == 'x' || c == 'X' || c == 'o')) { - assert(pbuf[0] == '0'); - assert(pbuf[1] == c); - if (fill != ' ') { - *res++ = *pbuf++; - *res++ = *pbuf++; - } - rescnt -= 2; - width -= 2; - if (width < 0) - width = 0; - len -= 2; - } - if (width > len && !(flags & F_LJUST)) { - do { - --rescnt; - *res++ = fill; - } while (--width > len); - } - if (fill == ' ') { - if (sign) - *res++ = sign; - if ((flags & F_ALT) && - (c == 'x' || c == 'X' || c == 'o')) { - assert(pbuf[0] == '0'); - assert(pbuf[1] == c); - *res++ = *pbuf++; - *res++ = *pbuf++; - } - } - Py_MEMCPY(res, pbuf, len); - res += len; - rescnt -= len; - while (--width >= len) { - --rescnt; - *res++ = ' '; - } - if (dict && (argidx < arglen) && c != '%') { - PyErr_SetString(PyExc_TypeError, - "not all arguments converted " - "during string formatting"); - Py_XDECREF(temp); - goto error; - } - Py_XDECREF(temp); - } /* '%' */ - } /* until end */ - if (argidx < arglen && !dict) { - PyErr_SetString(PyExc_TypeError, - "not all arguments converted " - "during string formatting"); - goto error; - } - if (args_owned) { - Py_DECREF(args); - } - _PyString_Resize(&result, reslen - rescnt); - return result; - - unicode: - if (args_owned) { - Py_DECREF(args); - args_owned = 0; - } - /* Fiddle args right (remove the first argidx arguments) */ - if (PyTuple_Check(orig_args) && argidx > 0) { - PyObject *v; - Py_ssize_t n = PyTuple_GET_SIZE(orig_args) - argidx; - v = PyTuple_New(n); - if (v == NULL) - goto error; - while (--n >= 0) { - PyObject *w = PyTuple_GET_ITEM(orig_args, n + argidx); - Py_INCREF(w); - PyTuple_SET_ITEM(v, n, w); - } - args = v; - } else { - Py_INCREF(orig_args); - args = orig_args; - } - args_owned = 1; - /* Take what we have of the result and let the Unicode formatting - function format the rest of the input. */ - rescnt = res - PyString_AS_STRING(result); - if (_PyString_Resize(&result, rescnt)) - goto error; - fmtcnt = PyString_GET_SIZE(format) - \ - (fmt - PyString_AS_STRING(format)); - format = PyUnicode_Decode(fmt, fmtcnt, NULL, NULL); - if (format == NULL) - goto error; - v = PyUnicode_Format(format, args); - Py_DECREF(format); - if (v == NULL) - goto error; - /* Paste what we have (result) to what the Unicode formatting - function returned (v) and return the result (or error) */ - w = PyUnicode_Concat(result, v); - Py_DECREF(result); - Py_DECREF(v); - Py_DECREF(args); - return w; - - error: - Py_DECREF(result); - if (args_owned) { - Py_DECREF(args); - } - return NULL; -} - void PyString_InternInPlace(PyObject **p) { Modified: python/branches/py3k-pep3137/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/unicodeobject.c (original) +++ python/branches/py3k-pep3137/Objects/unicodeobject.c Sat Oct 27 01:10:44 2007 @@ -8225,12 +8225,6 @@ return NULL; } -#define F_LJUST (1<<0) -#define F_SIGN (1<<1) -#define F_BLANK (1<<2) -#define F_ALT (1<<3) -#define F_ZERO (1<<4) - static Py_ssize_t strtounicode(Py_UNICODE *buffer, const char *charbuffer) { From python-3000-checkins at python.org Sat Oct 27 04:50:52 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 04:50:52 +0200 (CEST) Subject: [Python-3000-checkins] r58682 - python/branches/py3k/Modules/_sqlite/connection.c python/branches/py3k/Modules/_sqlite/module.c Message-ID: <20071027025052.B11001E4002@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 04:50:52 2007 New Revision: 58682 Modified: python/branches/py3k/Modules/_sqlite/connection.c python/branches/py3k/Modules/_sqlite/module.c Log: Use unicode in more places. Fixes a problem with str8 + str in test. Modified: python/branches/py3k/Modules/_sqlite/connection.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/connection.c (original) +++ python/branches/py3k/Modules/_sqlite/connection.c Sat Oct 27 04:50:52 2007 @@ -87,7 +87,7 @@ } if (!isolation_level) { - isolation_level = PyString_FromString(""); + isolation_level = PyUnicode_FromString(""); if (!isolation_level) { return -1; } @@ -828,24 +828,28 @@ self->inTransaction = 0; } else { + const char *statement; + Py_ssize_t size; + Py_INCREF(isolation_level); self->isolation_level = isolation_level; - begin_statement = PyString_FromString("BEGIN "); + begin_statement = PyUnicode_FromString("BEGIN "); if (!begin_statement) { return -1; } - PyString_Concat(&begin_statement, isolation_level); + PyUnicode_Concat(begin_statement, isolation_level); if (!begin_statement) { return -1; } - self->begin_statement = PyMem_Malloc(PyString_Size(begin_statement) + 2); + statement = PyUnicode_AsStringAndSize(begin_statement, &size); + self->begin_statement = PyMem_Malloc(size + 2); if (!self->begin_statement) { return -1; } - strcpy(self->begin_statement, PyString_AsString(begin_statement)); + strcpy(self->begin_statement, statement); Py_DECREF(begin_statement); } Modified: python/branches/py3k/Modules/_sqlite/module.c ============================================================================== --- python/branches/py3k/Modules/_sqlite/module.c (original) +++ python/branches/py3k/Modules/_sqlite/module.c Sat Oct 27 04:50:52 2007 @@ -359,13 +359,13 @@ Py_DECREF(tmp_obj); } - if (!(tmp_obj = PyString_FromString(PYSQLITE_VERSION))) { + if (!(tmp_obj = PyUnicode_FromString(PYSQLITE_VERSION))) { goto error; } PyDict_SetItemString(dict, "version", tmp_obj); Py_DECREF(tmp_obj); - if (!(tmp_obj = PyString_FromString(sqlite3_libversion()))) { + if (!(tmp_obj = PyUnicode_FromString(sqlite3_libversion()))) { goto error; } PyDict_SetItemString(dict, "sqlite_version", tmp_obj); From python-3000-checkins at python.org Sat Oct 27 04:55:08 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 04:55:08 +0200 (CEST) Subject: [Python-3000-checkins] r58683 - python/branches/py3k/Lib/email/test/data/msg_15.txt Message-ID: <20071027025508.6005D1E4002@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 04:55:08 2007 New Revision: 58683 Modified: python/branches/py3k/Lib/email/test/data/msg_15.txt Log: Get test_email to pass. There is a problem reading the data file and making it unicode with the default encoding. I'm not sure if this is a problem in and of itself. However, the test seems to be testing something different, so at least get that working. Need to revisit the unicode problem. Modified: python/branches/py3k/Lib/email/test/data/msg_15.txt ============================================================================== --- python/branches/py3k/Lib/email/test/data/msg_15.txt (original) +++ python/branches/py3k/Lib/email/test/data/msg_15.txt Sat Oct 27 04:55:08 2007 @@ -9,7 +9,7 @@ Content-type: multipart/mixed; boundary="MS_Mac_OE_3071477847_720252_MIME_Part" -> Denne meddelelse er i MIME-format. Da dit postl?sningsprogram ikke forst?r dette format, kan del af eller hele meddelelsen v?re ul?selig. +> Denne meddelelse er i MIME-format. Da dit postl --MS_Mac_OE_3071477847_720252_MIME_Part Content-type: multipart/alternative; From python-3000-checkins at python.org Sat Oct 27 06:00:46 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 06:00:46 +0200 (CEST) Subject: [Python-3000-checkins] r58684 - in python/branches/py3k: Doc/includes/noddy2.c Doc/includes/noddy3.c Doc/includes/noddy4.c Doc/includes/run-func.c Modules/_lsprof.c Objects/exceptions.c Objects/frameobject.c Message-ID: <20071027040046.7289D1E4004@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 06:00:45 2007 New Revision: 58684 Modified: python/branches/py3k/Doc/includes/noddy2.c python/branches/py3k/Doc/includes/noddy3.c python/branches/py3k/Doc/includes/noddy4.c python/branches/py3k/Doc/includes/run-func.c python/branches/py3k/Modules/_lsprof.c python/branches/py3k/Objects/exceptions.c python/branches/py3k/Objects/frameobject.c Log: Get rid of more uses of string and use unicode Modified: python/branches/py3k/Doc/includes/noddy2.c ============================================================================== --- python/branches/py3k/Doc/includes/noddy2.c (original) +++ python/branches/py3k/Doc/includes/noddy2.c Sat Oct 27 06:00:45 2007 @@ -23,14 +23,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -90,7 +90,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -109,7 +109,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k/Doc/includes/noddy3.c ============================================================================== --- python/branches/py3k/Doc/includes/noddy3.c (original) +++ python/branches/py3k/Doc/includes/noddy3.c Sat Oct 27 06:00:45 2007 @@ -23,14 +23,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -93,7 +93,7 @@ return -1; } - if (! PyString_Check(value)) { + if (! PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string"); return -1; @@ -121,7 +121,7 @@ return -1; } - if (! PyString_Check(value)) { + if (! PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string"); return -1; @@ -153,7 +153,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -162,7 +162,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k/Doc/includes/noddy4.c ============================================================================== --- python/branches/py3k/Doc/includes/noddy4.c (original) +++ python/branches/py3k/Doc/includes/noddy4.c Sat Oct 27 06:00:45 2007 @@ -57,14 +57,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -124,7 +124,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -143,7 +143,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k/Doc/includes/run-func.c ============================================================================== --- python/branches/py3k/Doc/includes/run-func.c (original) +++ python/branches/py3k/Doc/includes/run-func.c Sat Oct 27 06:00:45 2007 @@ -13,7 +13,7 @@ } Py_Initialize(); - pName = PyString_FromString(argv[1]); + pName = PyUnicode_FromString(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); Modified: python/branches/py3k/Modules/_lsprof.c ============================================================================== --- python/branches/py3k/Modules/_lsprof.c (original) +++ python/branches/py3k/Modules/_lsprof.c Sat Oct 27 06:00:45 2007 @@ -179,10 +179,7 @@ /* built-in function: look up the module name */ PyObject *mod = fn->m_module; const char *modname; - if (mod && PyString_Check(mod)) { - modname = PyString_AS_STRING(mod); - } - else if (mod && PyUnicode_Check(mod)) { + if (mod && PyUnicode_Check(mod)) { modname = PyUnicode_AsString(mod); } else if (mod && PyModule_Check(mod)) { Modified: python/branches/py3k/Objects/exceptions.c ============================================================================== --- python/branches/py3k/Objects/exceptions.c (original) +++ python/branches/py3k/Objects/exceptions.c Sat Oct 27 06:00:45 2007 @@ -1882,7 +1882,7 @@ (PyBaseExceptionObject *)PyExc_RecursionErrorInst; PyObject *args_tuple; PyObject *exc_message; - exc_message = PyString_FromString("maximum recursion depth exceeded"); + exc_message = PyUnicode_FromString("maximum recursion depth exceeded"); if (!exc_message) Py_FatalError("cannot allocate argument for RuntimeError " "pre-allocation"); Modified: python/branches/py3k/Objects/frameobject.c ============================================================================== --- python/branches/py3k/Objects/frameobject.c (original) +++ python/branches/py3k/Objects/frameobject.c Sat Oct 27 06:00:45 2007 @@ -721,7 +721,7 @@ for (j = nmap; --j >= 0; ) { PyObject *key = PyTuple_GET_ITEM(map, j); PyObject *value = values[j]; - assert(PyString_Check(key)/*XXX this should go*/ || PyUnicode_Check(key)); + assert(PyUnicode_Check(key)); if (deref) { assert(PyCell_Check(value)); value = PyCell_GET(value); From python-3000-checkins at python.org Sat Oct 27 06:01:17 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 06:01:17 +0200 (CEST) Subject: [Python-3000-checkins] r58685 - python/branches/py3k/Modules/_testcapimodule.c Message-ID: <20071027040117.E33A21E4023@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 06:01:17 2007 New Revision: 58685 Modified: python/branches/py3k/Modules/_testcapimodule.c Log: Try to get this test to pass on Win64 by making clean for ssize_t Modified: python/branches/py3k/Modules/_testcapimodule.c ============================================================================== --- python/branches/py3k/Modules/_testcapimodule.c (original) +++ python/branches/py3k/Modules/_testcapimodule.c Sat Oct 27 06:01:17 2007 @@ -5,6 +5,8 @@ * standard Python regression test, via Lib/test/test_capi.py. */ +#define PY_SSIZE_T_CLEAN + #include "Python.h" #include #include "structmember.h" @@ -377,8 +379,8 @@ { Py_ssize_t value; if (!PyArg_ParseTuple(args, "n", &value)) - return NULL; - return PyInt_FromSsize_t(value); + return NULL; + return PyLong_FromSsize_t(value); } #ifdef HAVE_LONG_LONG @@ -465,7 +467,7 @@ { PyObject *tuple, *obj; Py_UNICODE *value; - int len; + Py_ssize_t len; tuple = PyTuple_New(1); if (tuple == NULL) @@ -503,7 +505,7 @@ { PyObject *tuple, *obj; Py_UNICODE *value1, *value2; - int len1, len2; + Py_ssize_t len1, len2; tuple = PyTuple_New(2); if (tuple == NULL) From python-3000-checkins at python.org Sat Oct 27 07:40:06 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 07:40:06 +0200 (CEST) Subject: [Python-3000-checkins] r58686 - python/branches/py3k/Modules/_localemodule.c python/branches/py3k/Modules/pwdmodule.c Message-ID: <20071027054006.F0C331E4002@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 07:40:06 2007 New Revision: 58686 Modified: python/branches/py3k/Modules/_localemodule.c python/branches/py3k/Modules/pwdmodule.c Log: Get the locale and pwd tests working on the Solaris box where there are some unicode values used. I'm not sure if this is the correct on all operating systems, but this works on Linux w/o unicode. Modified: python/branches/py3k/Modules/_localemodule.c ============================================================================== --- python/branches/py3k/Modules/_localemodule.c (original) +++ python/branches/py3k/Modules/_localemodule.c Sat Oct 27 07:40:06 2007 @@ -143,7 +143,7 @@ involved herein */ #define RESULT_STRING(s)\ - x = PyUnicode_FromString(l->s);\ + x = PyUnicode_DecodeUnicodeEscape(l->s, strlen(l->s), "strict");\ if (!x) goto failed;\ PyDict_SetItemString(result, #s, x);\ Py_XDECREF(x) @@ -471,8 +471,10 @@ /* Check NULL as a workaround for GNU libc's returning NULL instead of an empty string for nl_langinfo(ERA). */ const char *result = nl_langinfo(item); + result = result != NULL ? result : ""; /* XXX may have to convert this to wcs first. */ - return PyUnicode_FromString(result != NULL ? result : ""); + return PyUnicode_DecodeUnicodeEscape(result, strlen(result), + "strict"); } PyErr_SetString(PyExc_ValueError, "unsupported langinfo constant"); return NULL; Modified: python/branches/py3k/Modules/pwdmodule.c ============================================================================== --- python/branches/py3k/Modules/pwdmodule.c (original) +++ python/branches/py3k/Modules/pwdmodule.c Sat Oct 27 07:40:06 2007 @@ -48,8 +48,11 @@ static void sets(PyObject *v, int i, const char* val) { - if (val) - PyStructSequence_SET_ITEM(v, i, PyUnicode_FromString(val)); + if (val) { + PyObject *o = + PyUnicode_DecodeUnicodeEscape(val, strlen(val), "strict"); + PyStructSequence_SET_ITEM(v, i, o); + } else { PyStructSequence_SET_ITEM(v, i, Py_None); Py_INCREF(Py_None); From python-3000-checkins at python.org Sat Oct 27 08:04:52 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 08:04:52 +0200 (CEST) Subject: [Python-3000-checkins] r58687 - in python/branches/py3k: Lib/plat-mac/ic.py Mac/Modules/file/_Filemodule.c Message-ID: <20071027060452.E88871E4027@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 08:04:52 2007 New Revision: 58687 Modified: python/branches/py3k/Lib/plat-mac/ic.py python/branches/py3k/Mac/Modules/file/_Filemodule.c Log: Get a bunch of tests working on Mac OS. I suspect that a bunch of the ord()s in Lib/plat-mac/ic.py need to be removed. Modified: python/branches/py3k/Lib/plat-mac/ic.py ============================================================================== --- python/branches/py3k/Lib/plat-mac/ic.py (original) +++ python/branches/py3k/Lib/plat-mac/ic.py Sat Oct 27 08:04:52 2007 @@ -65,7 +65,7 @@ return size, face, data[5:5+namelen] def _decode_boolean(data, key): - return ord(data[0]) + return data[0] def _decode_text(data, key): return data Modified: python/branches/py3k/Mac/Modules/file/_Filemodule.c ============================================================================== --- python/branches/py3k/Mac/Modules/file/_Filemodule.c (original) +++ python/branches/py3k/Mac/Modules/file/_Filemodule.c Sat Oct 27 08:04:52 2007 @@ -1312,7 +1312,7 @@ PyMac_Error(err); return NULL; } - _res = PyString_FromString(strbuf); + _res = PyUnicode_FromString(strbuf); return _res; } From python-3000-checkins at python.org Sat Oct 27 18:22:41 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:22:41 +0200 (CEST) Subject: [Python-3000-checkins] r58688 - python/branches/py3k-pep3137/Python/ceval.c Message-ID: <20071027162241.656B31E405D@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:22:40 2007 New Revision: 58688 Modified: python/branches/py3k-pep3137/Python/ceval.c Log: Move the 's += t' optimization to PyUnicode; for PyString we'll have buffer. Modified: python/branches/py3k-pep3137/Python/ceval.c ============================================================================== --- python/branches/py3k-pep3137/Python/ceval.c (original) +++ python/branches/py3k-pep3137/Python/ceval.c Sat Oct 27 18:22:40 2007 @@ -119,8 +119,8 @@ static void set_exc_info(PyThreadState *, PyObject *, PyObject *, PyObject *); static void reset_exc_info(PyThreadState *); static void format_exc_check_arg(PyObject *, const char *, PyObject *); -static PyObject * string_concatenate(PyObject *, PyObject *, - PyFrameObject *, unsigned char *); +static PyObject * unicode_concatenate(PyObject *, PyObject *, + PyFrameObject *, unsigned char *); #define NAME_ERROR_MSG \ "name '%.200s' is not defined" @@ -1127,10 +1127,10 @@ goto slow_add; x = PyInt_FromLong(i); } - else if (PyString_CheckExact(v) && - PyString_CheckExact(w)) { - x = string_concatenate(v, w, f, next_instr); - /* string_concatenate consumed the ref to v */ + else if (PyUnicode_CheckExact(v) && + PyUnicode_CheckExact(w)) { + x = unicode_concatenate(v, w, f, next_instr); + /* unicode_concatenate consumed the ref to v */ goto skip_decref_vx; } else { @@ -1328,10 +1328,10 @@ goto slow_iadd; x = PyInt_FromLong(i); } - else if (PyString_CheckExact(v) && - PyString_CheckExact(w)) { - x = string_concatenate(v, w, f, next_instr); - /* string_concatenate consumed the ref to v */ + else if (PyUnicode_CheckExact(v) && + PyUnicode_CheckExact(w)) { + x = unicode_concatenate(v, w, f, next_instr); + /* unicode_concatenate consumed the ref to v */ goto skip_decref_v; } else { @@ -3961,13 +3961,13 @@ } static PyObject * -string_concatenate(PyObject *v, PyObject *w, +unicode_concatenate(PyObject *v, PyObject *w, PyFrameObject *f, unsigned char *next_instr) { /* This function implements 'variable += expr' when both arguments - are strings. */ - Py_ssize_t v_len = PyString_GET_SIZE(v); - Py_ssize_t w_len = PyString_GET_SIZE(w); + are (Unicode) strings. */ + Py_ssize_t v_len = PyUnicode_GET_SIZE(v); + Py_ssize_t w_len = PyUnicode_GET_SIZE(w); Py_ssize_t new_len = v_len + w_len; if (new_len < 0) { PyErr_SetString(PyExc_OverflowError, @@ -4016,12 +4016,12 @@ } } - if (v->ob_refcnt == 1 && !PyString_CHECK_INTERNED(v)) { + if (v->ob_refcnt == 1 && !PyUnicode_CHECK_INTERNED(v)) { /* Now we own the last reference to 'v', so we can resize it * in-place. */ - if (_PyString_Resize(&v, new_len) != 0) { - /* XXX if _PyString_Resize() fails, 'v' has been + if (PyUnicode_Resize(&v, new_len) != 0) { + /* XXX if PyUnicode_Resize() fails, 'v' has been * deallocated so it cannot be put back into * 'variable'. The MemoryError is raised when there * is no value in 'variable', which might (very @@ -4030,14 +4030,15 @@ return NULL; } /* copy 'w' into the newly allocated area of 'v' */ - memcpy(PyString_AS_STRING(v) + v_len, - PyString_AS_STRING(w), w_len); + memcpy(PyUnicode_AS_UNICODE(v) + v_len, + PyUnicode_AS_UNICODE(w), w_len*sizeof(Py_UNICODE)); return v; } else { /* When in-place resizing is not an option. */ - PyString_Concat(&v, w); - return v; + w = PyUnicode_Concat(v, w); + Py_DECREF(v); + return w; } } From python-3000-checkins at python.org Sat Oct 27 18:23:24 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:23:24 +0200 (CEST) Subject: [Python-3000-checkins] r58689 - python/branches/py3k-pep3137/Python/ast.c Message-ID: <20071027162324.A9D431E4051@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:23:24 2007 New Revision: 58689 Modified: python/branches/py3k-pep3137/Python/ast.c Log: Tiny cleanup of comment and unnecessary check for '_' in parsestr(). Modified: python/branches/py3k-pep3137/Python/ast.c ============================================================================== --- python/branches/py3k-pep3137/Python/ast.c (original) +++ python/branches/py3k-pep3137/Python/ast.c Sat Oct 27 18:23:24 2007 @@ -3174,7 +3174,7 @@ } /* s is a Python string literal, including the bracketing quote characters, - * and r &/or u prefixes (if any), and embedded escape sequences (if any). + * and r &/or b prefixes (if any), and embedded escape sequences (if any). * parsestr parses it, and returns the decoded Python string object. */ static PyObject * @@ -3186,7 +3186,7 @@ int rawmode = 0; int need_encoding; - if (isalpha(quote) || quote == '_') { + if (isalpha(quote)) { if (quote == 'b' || quote == 'B') { quote = *++s; *bytesmode = 1; From python-3000-checkins at python.org Sat Oct 27 18:24:06 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:24:06 +0200 (CEST) Subject: [Python-3000-checkins] r58690 - python/branches/py3k-pep3137/Objects/codeobject.c Message-ID: <20071027162406.DC4821E4024@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:24:06 2007 New Revision: 58690 Modified: python/branches/py3k-pep3137/Objects/codeobject.c Log: Intern PyUnicode, not PyString when creating a code object. Modified: python/branches/py3k-pep3137/Objects/codeobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/codeobject.c (original) +++ python/branches/py3k-pep3137/Objects/codeobject.c Sat Oct 27 18:24:06 2007 @@ -8,7 +8,7 @@ /* all_name_chars(s): true iff all chars in s are valid NAME_CHARS */ static int -all_name_chars(unsigned char *s) +all_name_chars(Py_UNICODE *s) { static char ok_name_char[256]; static unsigned char *name_chars = (unsigned char *)NAME_CHARS; @@ -19,6 +19,8 @@ ok_name_char[*p] = 1; } while (*s) { + if (*s >= 128) + return 0; if (ok_name_char[*s++] == 0) return 0; } @@ -73,11 +75,11 @@ /* Intern selected string constants */ for (i = PyTuple_Size(consts); --i >= 0; ) { PyObject *v = PyTuple_GetItem(consts, i); - if (!PyString_Check(v)) + if (!PyUnicode_Check(v)) continue; - if (!all_name_chars((unsigned char *)PyString_AS_STRING(v))) + if (!all_name_chars(PyUnicode_AS_UNICODE(v))) continue; - PyString_InternInPlace(&PyTuple_GET_ITEM(consts, i)); + PyUnicode_InternInPlace(&PyTuple_GET_ITEM(consts, i)); } co = PyObject_NEW(PyCodeObject, &PyCode_Type); if (co != NULL) { From python-3000-checkins at python.org Sat Oct 27 18:25:05 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:25:05 +0200 (CEST) Subject: [Python-3000-checkins] r58691 - python/branches/py3k-pep3137/Objects/unicodeobject.c Message-ID: <20071027162505.1139C1E4030@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:25:04 2007 New Revision: 58691 Modified: python/branches/py3k-pep3137/Objects/unicodeobject.c Log: Fix typo in comment (ob_sstate -> state). Modified: python/branches/py3k-pep3137/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/unicodeobject.c (original) +++ python/branches/py3k-pep3137/Objects/unicodeobject.c Sat Oct 27 18:25:04 2007 @@ -101,7 +101,7 @@ function will delete the reference from this dictionary. Another way to look at this is that to say that the actual reference - count of a string is: s->ob_refcnt + (s->ob_sstate?2:0) + count of a string is: s->ob_refcnt + (s->state ? 2 : 0) */ static PyObject *interned; From python-3000-checkins at python.org Sat Oct 27 18:56:33 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:56:33 +0200 (CEST) Subject: [Python-3000-checkins] r58692 - in python/branches/py3k-pep3137: Include/stringobject.h Lib/test/test_sys.py Objects/stringobject.c Python/import.c Python/marshal.c Python/sysmodule.c Message-ID: <20071027165633.11F5C1E4019@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:56:32 2007 New Revision: 58692 Modified: python/branches/py3k-pep3137/Include/stringobject.h python/branches/py3k-pep3137/Lib/test/test_sys.py python/branches/py3k-pep3137/Objects/stringobject.c python/branches/py3k-pep3137/Python/import.c python/branches/py3k-pep3137/Python/marshal.c python/branches/py3k-pep3137/Python/sysmodule.c Log: Kill PyString interning. There's one mystery: if I remove ob_sstate from the PyStringObject struct, some (unicode) string literals are mutilated, e.g. ('\\1', '\1') prints ('\\1', '\t'). This must be an out of bounds write or something that I can't track down. (It doesn't help that it doesn't occur in debug mode. And no, make clean + recompilation doesn't help either.) So, in the mean time, I just keep the field, renamed to 'ob_placeholder'. Modified: python/branches/py3k-pep3137/Include/stringobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/stringobject.h (original) +++ python/branches/py3k-pep3137/Include/stringobject.h Sat Oct 27 18:56:32 2007 @@ -25,26 +25,18 @@ */ /* Caching the hash (ob_shash) saves recalculation of a string's hash value. - Interning strings (ob_sstate) tries to ensure that only one string - object with a given value exists, so equality tests can be one pointer - comparison. This is generally restricted to strings that "look like" - Python identifiers, although the sys.intern() function can be used to force - interning of any string. - Together, these sped the interpreter by up to 20%. */ + This significantly speeds up dict lookups. */ typedef struct { PyObject_VAR_HEAD long ob_shash; - int ob_sstate; + int ob_placeholder; /* XXX If I remove this things break?!?! */ char ob_sval[1]; /* Invariants: * ob_sval contains space for 'ob_size+1' elements. * ob_sval[ob_size] == 0. * ob_shash is the hash of the string or -1 if not computed yet. - * ob_sstate != 0 iff the string object is in stringobject.c's - * 'interned' dictionary; in this case the two references - * from 'interned' to this object are *not counted* in ob_refcnt. */ } PyStringObject; @@ -74,14 +66,6 @@ const char *, Py_ssize_t, const char *); -PyAPI_FUNC(void) PyString_InternInPlace(PyObject **); -PyAPI_FUNC(void) PyString_InternImmortal(PyObject **); -PyAPI_FUNC(PyObject *) PyString_InternFromString(const char *); -PyAPI_FUNC(void) _Py_ReleaseInternedStrings(void); - -/* Use only if you know it's a string */ -#define PyString_CHECK_INTERNED(op) (((PyStringObject *)(op))->ob_sstate) - /* Macro, trading safety for speed */ #define PyString_AS_STRING(op) (assert(PyString_Check(op)), \ (((PyStringObject *)(op))->ob_sval)) Modified: python/branches/py3k-pep3137/Lib/test/test_sys.py ============================================================================== --- python/branches/py3k-pep3137/Lib/test/test_sys.py (original) +++ python/branches/py3k-pep3137/Lib/test/test_sys.py Sat Oct 27 18:56:32 2007 @@ -300,7 +300,7 @@ def test_intern(self): self.assertRaises(TypeError, sys.intern) - s = str8(b"never interned before") + s = "never interned before" self.assert_(sys.intern(s) is s) s2 = s.swapcase().swapcase() self.assert_(sys.intern(s2) is s) @@ -310,28 +310,11 @@ # We don't want them in the interned dict and if they aren't # actually interned, we don't want to create the appearance # that they are by allowing intern() to succeeed. - class S(str8): + class S(str): def __hash__(self): return 123 - self.assertRaises(TypeError, sys.intern, S(b"abc")) - - s = "never interned as unicode before" - self.assert_(sys.intern(s) is s) - s2 = s.swapcase().swapcase() - self.assert_(sys.intern(s2) is s) - - class U(str): - def __hash__(self): - return 123 - - self.assertRaises(TypeError, sys.intern, U("abc")) - - # It's still safe to pass these strings to routines that - # call intern internally, e.g. PyObject_SetAttr(). - s = U("abc") - setattr(s, s, s) - self.assertEqual(getattr(s, s), s) + self.assertRaises(TypeError, sys.intern, S("abc")) def test_main(): Modified: python/branches/py3k-pep3137/Objects/stringobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/stringobject.c (original) +++ python/branches/py3k-pep3137/Objects/stringobject.c Sat Oct 27 18:56:32 2007 @@ -13,16 +13,6 @@ static PyStringObject *characters[UCHAR_MAX + 1]; static PyStringObject *nullstring; -/* This dictionary holds all interned strings. Note that references to - strings in this dictionary are *not* counted in the string's ob_refcnt. - When the interned string reaches a refcnt of 0 the string deallocation - function will delete the reference from this dictionary. - - Another way to look at this is that to say that the actual reference - count of a string is: s->ob_refcnt + (s->ob_sstate?2:0) -*/ -static PyObject *interned; - /* For both PyString_FromString() and PyString_FromStringAndSize(), the parameter `size' denotes number of characters to allocate, not counting any @@ -77,21 +67,14 @@ return PyErr_NoMemory(); PyObject_INIT_VAR(op, &PyString_Type, size); op->ob_shash = -1; - op->ob_sstate = SSTATE_NOT_INTERNED; if (str != NULL) Py_MEMCPY(op->ob_sval, str, size); op->ob_sval[size] = '\0'; /* share short strings */ if (size == 0) { - PyObject *t = (PyObject *)op; - PyString_InternInPlace(&t); - op = (PyStringObject *)t; nullstring = op; Py_INCREF(op); } else if (size == 1 && str != NULL) { - PyObject *t = (PyObject *)op; - PyString_InternInPlace(&t); - op = (PyStringObject *)t; characters[*str & UCHAR_MAX] = op; Py_INCREF(op); } @@ -132,19 +115,12 @@ return PyErr_NoMemory(); PyObject_INIT_VAR(op, &PyString_Type, size); op->ob_shash = -1; - op->ob_sstate = SSTATE_NOT_INTERNED; Py_MEMCPY(op->ob_sval, str, size+1); /* share short strings */ if (size == 0) { - PyObject *t = (PyObject *)op; - PyString_InternInPlace(&t); - op = (PyStringObject *)t; nullstring = op; Py_INCREF(op); } else if (size == 1) { - PyObject *t = (PyObject *)op; - PyString_InternInPlace(&t); - op = (PyStringObject *)t; characters[*str & UCHAR_MAX] = op; Py_INCREF(op); } @@ -354,24 +330,6 @@ static void string_dealloc(PyObject *op) { - switch (PyString_CHECK_INTERNED(op)) { - case SSTATE_NOT_INTERNED: - break; - - case SSTATE_INTERNED_MORTAL: - /* revive dead object temporarily for DelItem */ - Py_Refcnt(op) = 3; - if (PyDict_DelItem(interned, op) != 0) - Py_FatalError( - "deletion of interned string failed"); - break; - - case SSTATE_INTERNED_IMMORTAL: - Py_FatalError("Immortal interned string died."); - - default: - Py_FatalError("Inconsistent interned string state."); - } Py_Type(op)->tp_free(op); } @@ -760,7 +718,6 @@ return PyErr_NoMemory(); PyObject_INIT_VAR(op, &PyString_Type, size); op->ob_shash = -1; - op->ob_sstate = SSTATE_NOT_INTERNED; Py_MEMCPY(op->ob_sval, a->ob_sval, Py_Size(a)); Py_MEMCPY(op->ob_sval + Py_Size(a), b->ob_sval, Py_Size(b)); op->ob_sval[size] = '\0'; @@ -803,7 +760,6 @@ return PyErr_NoMemory(); PyObject_INIT_VAR(op, &PyString_Type, size); op->ob_shash = -1; - op->ob_sstate = SSTATE_NOT_INTERNED; op->ob_sval[size] = '\0'; if (Py_Size(a) == 1 && n > 0) { memset(op->ob_sval, a->ob_sval[0] , n); @@ -3053,10 +3009,10 @@ n = PyString_GET_SIZE(tmp); pnew = type->tp_alloc(type, n); if (pnew != NULL) { - Py_MEMCPY(PyString_AS_STRING(pnew), PyString_AS_STRING(tmp), n+1); + Py_MEMCPY(PyString_AS_STRING(pnew), + PyString_AS_STRING(tmp), n+1); ((PyStringObject *)pnew)->ob_shash = ((PyStringObject *)tmp)->ob_shash; - ((PyStringObject *)pnew)->ob_sstate = SSTATE_NOT_INTERNED; } Py_DECREF(tmp); return pnew; @@ -3157,8 +3113,7 @@ register PyObject *v; register PyStringObject *sv; v = *pv; - if (!PyString_Check(v) || Py_Refcnt(v) != 1 || newsize < 0 || - PyString_CHECK_INTERNED(v)) { + if (!PyString_Check(v) || Py_Refcnt(v) != 1 || newsize < 0) { *pv = 0; Py_DECREF(v); PyErr_BadInternalCall(); @@ -3326,65 +3281,6 @@ } void -PyString_InternInPlace(PyObject **p) -{ - register PyStringObject *s = (PyStringObject *)(*p); - PyObject *t; - if (s == NULL || !PyString_Check(s)) - Py_FatalError("PyString_InternInPlace: strings only please!"); - /* If it's a string subclass, we don't really know what putting - it in the interned dict might do. */ - if (!PyString_CheckExact(s)) - return; - if (PyString_CHECK_INTERNED(s)) - return; - if (interned == NULL) { - interned = PyDict_New(); - if (interned == NULL) { - PyErr_Clear(); /* Don't leave an exception */ - return; - } - } - t = PyDict_GetItem(interned, (PyObject *)s); - if (t) { - Py_INCREF(t); - Py_DECREF(*p); - *p = t; - return; - } - - if (PyDict_SetItem(interned, (PyObject *)s, (PyObject *)s) < 0) { - PyErr_Clear(); - return; - } - /* The two references in interned are not counted by refcnt. - The string deallocator will take care of this */ - Py_Refcnt(s) -= 2; - PyString_CHECK_INTERNED(s) = SSTATE_INTERNED_MORTAL; -} - -void -PyString_InternImmortal(PyObject **p) -{ - PyString_InternInPlace(p); - if (PyString_CHECK_INTERNED(*p) != SSTATE_INTERNED_IMMORTAL) { - PyString_CHECK_INTERNED(*p) = SSTATE_INTERNED_IMMORTAL; - Py_INCREF(*p); - } -} - - -PyObject * -PyString_InternFromString(const char *cp) -{ - PyObject *s = PyString_FromString(cp); - if (s == NULL) - return NULL; - PyString_InternInPlace(&s); - return s; -} - -void PyString_Fini(void) { int i; @@ -3396,58 +3292,6 @@ nullstring = NULL; } -void _Py_ReleaseInternedStrings(void) -{ - PyObject *keys; - PyStringObject *s; - Py_ssize_t i, n; - Py_ssize_t immortal_size = 0, mortal_size = 0; - - if (interned == NULL || !PyDict_Check(interned)) - return; - keys = PyDict_Keys(interned); - if (keys == NULL || !PyList_Check(keys)) { - PyErr_Clear(); - return; - } - - /* Since _Py_ReleaseInternedStrings() is intended to help a leak - detector, interned strings are not forcibly deallocated; rather, we - give them their stolen references back, and then clear and DECREF - the interned dict. */ - - n = PyList_GET_SIZE(keys); - fprintf(stderr, "releasing %" PY_FORMAT_SIZE_T "d interned strings\n", - n); - for (i = 0; i < n; i++) { - s = (PyStringObject *) PyList_GET_ITEM(keys, i); - switch (s->ob_sstate) { - case SSTATE_NOT_INTERNED: - /* XXX Shouldn't happen */ - break; - case SSTATE_INTERNED_IMMORTAL: - Py_Refcnt(s) += 1; - immortal_size += Py_Size(s); - break; - case SSTATE_INTERNED_MORTAL: - Py_Refcnt(s) += 2; - mortal_size += Py_Size(s); - break; - default: - Py_FatalError("Inconsistent interned string state."); - } - s->ob_sstate = SSTATE_NOT_INTERNED; - } - fprintf(stderr, "total size of all interned strings: " - "%" PY_FORMAT_SIZE_T "d/%" PY_FORMAT_SIZE_T "d " - "mortal/immortal\n", mortal_size, immortal_size); - Py_DECREF(keys); - PyDict_Clear(interned); - Py_DECREF(interned); - interned = NULL; -} - - /*********************** Str Iterator ****************************/ typedef struct { Modified: python/branches/py3k-pep3137/Python/import.c ============================================================================== --- python/branches/py3k-pep3137/Python/import.c (original) +++ python/branches/py3k-pep3137/Python/import.c Sat Oct 27 18:56:32 2007 @@ -76,9 +76,10 @@ 3060 (PEP 3115 metaclass syntax) 3070 (PEP 3109 raise changes) 3080 (PEP 3137 make __file__ and __name__ unicode) + 3090 (kill str8 interning) . */ -#define MAGIC (3080 | ((long)'\r'<<16) | ((long)'\n'<<24)) +#define MAGIC (3090 | ((long)'\r'<<16) | ((long)'\n'<<24)) /* Magic word as global; note that _PyImport_Init() can change the value of this global to accommodate for alterations of how the Modified: python/branches/py3k-pep3137/Python/marshal.c ============================================================================== --- python/branches/py3k-pep3137/Python/marshal.c (original) +++ python/branches/py3k-pep3137/Python/marshal.c Sat Oct 27 18:56:32 2007 @@ -36,8 +36,6 @@ #define TYPE_BINARY_COMPLEX 'y' #define TYPE_LONG 'l' #define TYPE_STRING 's' -#define TYPE_INTERNED 't' -#define TYPE_STRINGREF 'R' #define TYPE_TUPLE '(' #define TYPE_LIST '[' #define TYPE_DICT '{' @@ -231,31 +229,7 @@ } #endif else if (PyString_Check(v)) { - if (p->strings && PyString_CHECK_INTERNED(v)) { - PyObject *o = PyDict_GetItem(p->strings, v); - if (o) { - long w = PyInt_AsLong(o); - w_byte(TYPE_STRINGREF, p); - w_long(w, p); - goto exit; - } - else { - int ok; - o = PyInt_FromSsize_t(PyDict_Size(p->strings)); - ok = o && - PyDict_SetItem(p->strings, v, o) >= 0; - Py_XDECREF(o); - if (!ok) { - p->depth--; - p->error = 1; - return; - } - w_byte(TYPE_INTERNED, p); - } - } - else { - w_byte(TYPE_STRING, p); - } + w_byte(TYPE_STRING, p); n = PyString_GET_SIZE(v); if (n > INT_MAX) { /* huge strings are not supported */ @@ -389,7 +363,6 @@ w_byte(TYPE_UNKNOWN, p); p->error = 1; } - exit: p->depth--; } @@ -703,7 +676,6 @@ } #endif - case TYPE_INTERNED: case TYPE_STRING: n = r_long(p); if (n < 0 || n > INT_MAX) { @@ -723,25 +695,6 @@ retval = NULL; break; } - if (type == TYPE_INTERNED) { - PyString_InternInPlace(&v); - if (PyList_Append(p->strings, v) < 0) { - retval = NULL; - break; - } - } - retval = v; - break; - - case TYPE_STRINGREF: - n = r_long(p); - if (n < 0 || n >= PyList_GET_SIZE(p->strings)) { - PyErr_SetString(PyExc_ValueError, "bad marshal data"); - retval = NULL; - break; - } - v = PyList_GET_ITEM(p->strings, n); - Py_INCREF(v); retval = v; break; Modified: python/branches/py3k-pep3137/Python/sysmodule.c ============================================================================== --- python/branches/py3k-pep3137/Python/sysmodule.c (original) +++ python/branches/py3k-pep3137/Python/sysmodule.c Sat Oct 27 18:56:32 2007 @@ -225,14 +225,9 @@ sys_intern(PyObject *self, PyObject *args) { PyObject *s; - if (!PyArg_ParseTuple(args, "S:intern", &s)) + if (!PyArg_ParseTuple(args, "U:intern", &s)) return NULL; - if (PyString_CheckExact(s)) { - Py_INCREF(s); - PyString_InternInPlace(&s); - return s; - } - else if (PyUnicode_CheckExact(s)) { + if (PyUnicode_CheckExact(s)) { Py_INCREF(s); PyUnicode_InternInPlace(&s); return s; From python-3000-checkins at python.org Sat Oct 27 18:58:56 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 18:58:56 +0200 (CEST) Subject: [Python-3000-checkins] r58693 - python/branches/py3k-pep3137 Message-ID: <20071027165856.19D271E402D@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 18:58:55 2007 New Revision: 58693 Modified: python/branches/py3k-pep3137/ (props changed) Log: Initialized merge tracking via "svnmerge" with revisions "1-58678" from svn+ssh://pythondev at svn.python.org/python/branches/py3k From python-3000-checkins at python.org Sat Oct 27 19:25:12 2007 From: python-3000-checkins at python.org (neal.norwitz) Date: Sat, 27 Oct 2007 19:25:12 +0200 (CEST) Subject: [Python-3000-checkins] r58694 - python/branches/py3k/Lib/urllib2.py Message-ID: <20071027172512.816B41E4015@bag.python.org> Author: neal.norwitz Date: Sat Oct 27 19:25:12 2007 New Revision: 58694 Modified: python/branches/py3k/Lib/urllib2.py Log: URLError now takes only a single parameter. This was causing errors in test_urllib2net on the MIPS buildbot. Modified: python/branches/py3k/Lib/urllib2.py ============================================================================== --- python/branches/py3k/Lib/urllib2.py (original) +++ python/branches/py3k/Lib/urllib2.py Sat Oct 27 19:25:12 2007 @@ -1240,7 +1240,7 @@ import mimetypes host = req.get_host() if not host: - raise URLError('ftp error', 'no host given') + raise URLError('ftp error: no host given') host, port = splitport(host) if port is None: port = ftplib.FTP_PORT @@ -1286,7 +1286,7 @@ headers = mimetools.Message(sf) return addinfourl(fp, headers, req.get_full_url()) except ftplib.all_errors as msg: - raise URLError('ftp error', msg).with_traceback(sys.exc_info()[2]) + raise URLError('ftp error %s' % msg).with_traceback(sys.exc_info()[2]) def connect_ftp(self, user, passwd, host, port, dirs, timeout): fw = ftpwrapper(user, passwd, host, port, dirs, timeout) From python-3000-checkins at python.org Sat Oct 27 19:30:08 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sat, 27 Oct 2007 19:30:08 +0200 (CEST) Subject: [Python-3000-checkins] r58695 - in python/branches/py3k-pep3137: Doc/includes/noddy2.c Doc/includes/noddy3.c Doc/includes/noddy4.c Doc/includes/run-func.c Lib/email/test/data/msg_15.txt Lib/plat-mac/ic.py Mac/Modules/file/_Filemodule.c Modules/_localemodule.c Modules/_lsprof.c Modules/_sqlite/connection.c Modules/_sqlite/module.c Modules/_testcapimodule.c Modules/pwdmodule.c Objects/exceptions.c Objects/frameobject.c Message-ID: <20071027173008.21C041E4019@bag.python.org> Author: guido.van.rossum Date: Sat Oct 27 19:30:07 2007 New Revision: 58695 Modified: python/branches/py3k-pep3137/ (props changed) python/branches/py3k-pep3137/Doc/includes/noddy2.c python/branches/py3k-pep3137/Doc/includes/noddy3.c python/branches/py3k-pep3137/Doc/includes/noddy4.c python/branches/py3k-pep3137/Doc/includes/run-func.c python/branches/py3k-pep3137/Lib/email/test/data/msg_15.txt python/branches/py3k-pep3137/Lib/plat-mac/ic.py python/branches/py3k-pep3137/Mac/Modules/file/_Filemodule.c python/branches/py3k-pep3137/Modules/_localemodule.c python/branches/py3k-pep3137/Modules/_lsprof.c python/branches/py3k-pep3137/Modules/_sqlite/connection.c python/branches/py3k-pep3137/Modules/_sqlite/module.c python/branches/py3k-pep3137/Modules/_testcapimodule.c python/branches/py3k-pep3137/Modules/pwdmodule.c python/branches/py3k-pep3137/Objects/exceptions.c python/branches/py3k-pep3137/Objects/frameobject.c Log: Merged revisions 58679-58693 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k ........ r58682 | neal.norwitz | 2007-10-26 19:50:52 -0700 (Fri, 26 Oct 2007) | 1 line Use unicode in more places. Fixes a problem with str8 + str in test. ........ r58683 | neal.norwitz | 2007-10-26 19:55:08 -0700 (Fri, 26 Oct 2007) | 6 lines Get test_email to pass. There is a problem reading the data file and making it unicode with the default encoding. I'm not sure if this is a problem in and of itself. However, the test seems to be testing something different, so at least get that working. Need to revisit the unicode problem. ........ r58684 | neal.norwitz | 2007-10-26 21:00:45 -0700 (Fri, 26 Oct 2007) | 1 line Get rid of more uses of string and use unicode ........ r58685 | neal.norwitz | 2007-10-26 21:01:17 -0700 (Fri, 26 Oct 2007) | 1 line Try to get this test to pass on Win64 by making clean for ssize_t ........ r58686 | neal.norwitz | 2007-10-26 22:40:06 -0700 (Fri, 26 Oct 2007) | 4 lines Get the locale and pwd tests working on the Solaris box where there are some unicode values used. I'm not sure if this is the correct on all operating systems, but this works on Linux w/o unicode. ........ r58687 | neal.norwitz | 2007-10-26 23:04:52 -0700 (Fri, 26 Oct 2007) | 3 lines Get a bunch of tests working on Mac OS. I suspect that a bunch of the ord()s in Lib/plat-mac/ic.py need to be removed. ........ Modified: python/branches/py3k-pep3137/Doc/includes/noddy2.c ============================================================================== --- python/branches/py3k-pep3137/Doc/includes/noddy2.c (original) +++ python/branches/py3k-pep3137/Doc/includes/noddy2.c Sat Oct 27 19:30:07 2007 @@ -23,14 +23,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -90,7 +90,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -109,7 +109,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k-pep3137/Doc/includes/noddy3.c ============================================================================== --- python/branches/py3k-pep3137/Doc/includes/noddy3.c (original) +++ python/branches/py3k-pep3137/Doc/includes/noddy3.c Sat Oct 27 19:30:07 2007 @@ -23,14 +23,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -93,7 +93,7 @@ return -1; } - if (! PyString_Check(value)) { + if (! PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string"); return -1; @@ -121,7 +121,7 @@ return -1; } - if (! PyString_Check(value)) { + if (! PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string"); return -1; @@ -153,7 +153,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -162,7 +162,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k-pep3137/Doc/includes/noddy4.c ============================================================================== --- python/branches/py3k-pep3137/Doc/includes/noddy4.c (original) +++ python/branches/py3k-pep3137/Doc/includes/noddy4.c Sat Oct 27 19:30:07 2007 @@ -57,14 +57,14 @@ self = (Noddy *)type->tp_alloc(type, 0); if (self != NULL) { - self->first = PyString_FromString(""); + self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } - self->last = PyString_FromString(""); + self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); @@ -124,7 +124,7 @@ PyObject *args, *result; if (format == NULL) { - format = PyString_FromString("%s %s"); + format = PyUnicode_FromString("%s %s"); if (format == NULL) return NULL; } @@ -143,7 +143,7 @@ if (args == NULL) return NULL; - result = PyString_Format(format, args); + result = PyUnicode_Format(format, args); Py_DECREF(args); return result; Modified: python/branches/py3k-pep3137/Doc/includes/run-func.c ============================================================================== --- python/branches/py3k-pep3137/Doc/includes/run-func.c (original) +++ python/branches/py3k-pep3137/Doc/includes/run-func.c Sat Oct 27 19:30:07 2007 @@ -13,7 +13,7 @@ } Py_Initialize(); - pName = PyString_FromString(argv[1]); + pName = PyUnicode_FromString(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); Modified: python/branches/py3k-pep3137/Lib/email/test/data/msg_15.txt ============================================================================== --- python/branches/py3k-pep3137/Lib/email/test/data/msg_15.txt (original) +++ python/branches/py3k-pep3137/Lib/email/test/data/msg_15.txt Sat Oct 27 19:30:07 2007 @@ -9,7 +9,7 @@ Content-type: multipart/mixed; boundary="MS_Mac_OE_3071477847_720252_MIME_Part" -> Denne meddelelse er i MIME-format. Da dit postl?sningsprogram ikke forst?r dette format, kan del af eller hele meddelelsen v?re ul?selig. +> Denne meddelelse er i MIME-format. Da dit postl --MS_Mac_OE_3071477847_720252_MIME_Part Content-type: multipart/alternative; Modified: python/branches/py3k-pep3137/Lib/plat-mac/ic.py ============================================================================== --- python/branches/py3k-pep3137/Lib/plat-mac/ic.py (original) +++ python/branches/py3k-pep3137/Lib/plat-mac/ic.py Sat Oct 27 19:30:07 2007 @@ -65,7 +65,7 @@ return size, face, data[5:5+namelen] def _decode_boolean(data, key): - return ord(data[0]) + return data[0] def _decode_text(data, key): return data Modified: python/branches/py3k-pep3137/Mac/Modules/file/_Filemodule.c ============================================================================== --- python/branches/py3k-pep3137/Mac/Modules/file/_Filemodule.c (original) +++ python/branches/py3k-pep3137/Mac/Modules/file/_Filemodule.c Sat Oct 27 19:30:07 2007 @@ -1312,7 +1312,7 @@ PyMac_Error(err); return NULL; } - _res = PyString_FromString(strbuf); + _res = PyUnicode_FromString(strbuf); return _res; } Modified: python/branches/py3k-pep3137/Modules/_localemodule.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_localemodule.c (original) +++ python/branches/py3k-pep3137/Modules/_localemodule.c Sat Oct 27 19:30:07 2007 @@ -143,7 +143,7 @@ involved herein */ #define RESULT_STRING(s)\ - x = PyUnicode_FromString(l->s);\ + x = PyUnicode_DecodeUnicodeEscape(l->s, strlen(l->s), "strict");\ if (!x) goto failed;\ PyDict_SetItemString(result, #s, x);\ Py_XDECREF(x) @@ -471,8 +471,10 @@ /* Check NULL as a workaround for GNU libc's returning NULL instead of an empty string for nl_langinfo(ERA). */ const char *result = nl_langinfo(item); + result = result != NULL ? result : ""; /* XXX may have to convert this to wcs first. */ - return PyUnicode_FromString(result != NULL ? result : ""); + return PyUnicode_DecodeUnicodeEscape(result, strlen(result), + "strict"); } PyErr_SetString(PyExc_ValueError, "unsupported langinfo constant"); return NULL; Modified: python/branches/py3k-pep3137/Modules/_lsprof.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_lsprof.c (original) +++ python/branches/py3k-pep3137/Modules/_lsprof.c Sat Oct 27 19:30:07 2007 @@ -179,10 +179,7 @@ /* built-in function: look up the module name */ PyObject *mod = fn->m_module; const char *modname; - if (mod && PyString_Check(mod)) { - modname = PyString_AS_STRING(mod); - } - else if (mod && PyUnicode_Check(mod)) { + if (mod && PyUnicode_Check(mod)) { modname = PyUnicode_AsString(mod); } else if (mod && PyModule_Check(mod)) { Modified: python/branches/py3k-pep3137/Modules/_sqlite/connection.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_sqlite/connection.c (original) +++ python/branches/py3k-pep3137/Modules/_sqlite/connection.c Sat Oct 27 19:30:07 2007 @@ -87,7 +87,7 @@ } if (!isolation_level) { - isolation_level = PyString_FromString(""); + isolation_level = PyUnicode_FromString(""); if (!isolation_level) { return -1; } @@ -828,24 +828,28 @@ self->inTransaction = 0; } else { + const char *statement; + Py_ssize_t size; + Py_INCREF(isolation_level); self->isolation_level = isolation_level; - begin_statement = PyString_FromString("BEGIN "); + begin_statement = PyUnicode_FromString("BEGIN "); if (!begin_statement) { return -1; } - PyString_Concat(&begin_statement, isolation_level); + PyUnicode_Concat(begin_statement, isolation_level); if (!begin_statement) { return -1; } - self->begin_statement = PyMem_Malloc(PyString_Size(begin_statement) + 2); + statement = PyUnicode_AsStringAndSize(begin_statement, &size); + self->begin_statement = PyMem_Malloc(size + 2); if (!self->begin_statement) { return -1; } - strcpy(self->begin_statement, PyString_AsString(begin_statement)); + strcpy(self->begin_statement, statement); Py_DECREF(begin_statement); } Modified: python/branches/py3k-pep3137/Modules/_sqlite/module.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_sqlite/module.c (original) +++ python/branches/py3k-pep3137/Modules/_sqlite/module.c Sat Oct 27 19:30:07 2007 @@ -359,13 +359,13 @@ Py_DECREF(tmp_obj); } - if (!(tmp_obj = PyString_FromString(PYSQLITE_VERSION))) { + if (!(tmp_obj = PyUnicode_FromString(PYSQLITE_VERSION))) { goto error; } PyDict_SetItemString(dict, "version", tmp_obj); Py_DECREF(tmp_obj); - if (!(tmp_obj = PyString_FromString(sqlite3_libversion()))) { + if (!(tmp_obj = PyUnicode_FromString(sqlite3_libversion()))) { goto error; } PyDict_SetItemString(dict, "sqlite_version", tmp_obj); Modified: python/branches/py3k-pep3137/Modules/_testcapimodule.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_testcapimodule.c (original) +++ python/branches/py3k-pep3137/Modules/_testcapimodule.c Sat Oct 27 19:30:07 2007 @@ -5,6 +5,8 @@ * standard Python regression test, via Lib/test/test_capi.py. */ +#define PY_SSIZE_T_CLEAN + #include "Python.h" #include #include "structmember.h" @@ -377,8 +379,8 @@ { Py_ssize_t value; if (!PyArg_ParseTuple(args, "n", &value)) - return NULL; - return PyInt_FromSsize_t(value); + return NULL; + return PyLong_FromSsize_t(value); } #ifdef HAVE_LONG_LONG @@ -465,7 +467,7 @@ { PyObject *tuple, *obj; Py_UNICODE *value; - int len; + Py_ssize_t len; tuple = PyTuple_New(1); if (tuple == NULL) @@ -503,7 +505,7 @@ { PyObject *tuple, *obj; Py_UNICODE *value1, *value2; - int len1, len2; + Py_ssize_t len1, len2; tuple = PyTuple_New(2); if (tuple == NULL) Modified: python/branches/py3k-pep3137/Modules/pwdmodule.c ============================================================================== --- python/branches/py3k-pep3137/Modules/pwdmodule.c (original) +++ python/branches/py3k-pep3137/Modules/pwdmodule.c Sat Oct 27 19:30:07 2007 @@ -48,8 +48,11 @@ static void sets(PyObject *v, int i, const char* val) { - if (val) - PyStructSequence_SET_ITEM(v, i, PyUnicode_FromString(val)); + if (val) { + PyObject *o = + PyUnicode_DecodeUnicodeEscape(val, strlen(val), "strict"); + PyStructSequence_SET_ITEM(v, i, o); + } else { PyStructSequence_SET_ITEM(v, i, Py_None); Py_INCREF(Py_None); Modified: python/branches/py3k-pep3137/Objects/exceptions.c ============================================================================== --- python/branches/py3k-pep3137/Objects/exceptions.c (original) +++ python/branches/py3k-pep3137/Objects/exceptions.c Sat Oct 27 19:30:07 2007 @@ -1882,7 +1882,7 @@ (PyBaseExceptionObject *)PyExc_RecursionErrorInst; PyObject *args_tuple; PyObject *exc_message; - exc_message = PyString_FromString("maximum recursion depth exceeded"); + exc_message = PyUnicode_FromString("maximum recursion depth exceeded"); if (!exc_message) Py_FatalError("cannot allocate argument for RuntimeError " "pre-allocation"); Modified: python/branches/py3k-pep3137/Objects/frameobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/frameobject.c (original) +++ python/branches/py3k-pep3137/Objects/frameobject.c Sat Oct 27 19:30:07 2007 @@ -721,7 +721,7 @@ for (j = nmap; --j >= 0; ) { PyObject *key = PyTuple_GET_ITEM(map, j); PyObject *value = values[j]; - assert(PyString_Check(key)/*XXX this should go*/ || PyUnicode_Check(key)); + assert(PyUnicode_Check(key)); if (deref) { assert(PyCell_Check(value)); value = PyCell_GET(value); From python-3000-checkins at python.org Sun Oct 28 16:09:42 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Sun, 28 Oct 2007 16:09:42 +0100 (CET) Subject: [Python-3000-checkins] r58699 - python/branches/py3k-pep3137/Lib/tempfile.py Message-ID: <20071028150942.CC8941E4016@bag.python.org> Author: guido.van.rossum Date: Sun Oct 28 16:09:42 2007 New Revision: 58699 Modified: python/branches/py3k-pep3137/Lib/tempfile.py Log: Issue 1340 by Amaury Forgeot d'Arc (with help from Christian Heimes, and my own interpretation). Don't pass the newline= flag to StringIO in SpooledTemporaryFile. This avoids doubling newlines when the file is rolled over. Modified: python/branches/py3k-pep3137/Lib/tempfile.py ============================================================================== --- python/branches/py3k-pep3137/Lib/tempfile.py (original) +++ python/branches/py3k-pep3137/Lib/tempfile.py Sun Oct 28 16:09:42 2007 @@ -495,7 +495,10 @@ if 'b' in mode: self._file = _io.BytesIO() else: - self._file = _io.StringIO(encoding=encoding, newline=newline) + # Setting newline="\n" avoids newline translation; + # this is important because otherwise on Windows we'd + # hget double newline translation upon rollover(). + self._file = _io.StringIO(encoding=encoding, newline="\n") self._max_size = max_size self._rolled = False self._TemporaryFileArgs = {'mode': mode, 'buffering': buffering, From python-3000-checkins at python.org Mon Oct 29 17:42:52 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 29 Oct 2007 17:42:52 +0100 (CET) Subject: [Python-3000-checkins] r58701 - python/branches/py3k/Lib/tempfile.py Message-ID: <20071029164252.19FC11E401A@bag.python.org> Author: guido.van.rossum Date: Mon Oct 29 17:42:51 2007 New Revision: 58701 Modified: python/branches/py3k/Lib/tempfile.py Log: Issue 1340 by Amaury Forgeot d'Arc (with help from Christian Heimes, and my own interpretation). Don't pass the newline= flag to StringIO in SpooledTemporaryFile. This avoids doubling newlines when the file is rolled over. Modified: python/branches/py3k/Lib/tempfile.py ============================================================================== --- python/branches/py3k/Lib/tempfile.py (original) +++ python/branches/py3k/Lib/tempfile.py Mon Oct 29 17:42:51 2007 @@ -495,7 +495,10 @@ if 'b' in mode: self._file = _io.BytesIO() else: - self._file = _io.StringIO(encoding=encoding, newline=newline) + # Setting newline="\n" avoids newline translation; + # this is important because otherwise on Windows we'd + # hget double newline translation upon rollover(). + self._file = _io.StringIO(encoding=encoding, newline="\n") self._max_size = max_size self._rolled = False self._TemporaryFileArgs = {'mode': mode, 'buffering': buffering, From python-3000-checkins at python.org Mon Oct 29 17:56:50 2007 From: python-3000-checkins at python.org (fred.drake) Date: Mon, 29 Oct 2007 17:56:50 +0100 (CET) Subject: [Python-3000-checkins] r58702 - python/branches/py3k/Lib/tempfile.py Message-ID: <20071029165650.C65CB1E401A@bag.python.org> Author: fred.drake Date: Mon Oct 29 17:56:50 2007 New Revision: 58702 Modified: python/branches/py3k/Lib/tempfile.py Log: fix typo Modified: python/branches/py3k/Lib/tempfile.py ============================================================================== --- python/branches/py3k/Lib/tempfile.py (original) +++ python/branches/py3k/Lib/tempfile.py Mon Oct 29 17:56:50 2007 @@ -497,7 +497,7 @@ else: # Setting newline="\n" avoids newline translation; # this is important because otherwise on Windows we'd - # hget double newline translation upon rollover(). + # get double newline translation upon rollover(). self._file = _io.StringIO(encoding=encoding, newline="\n") self._max_size = max_size self._rolled = False From python-3000-checkins at python.org Mon Oct 29 18:07:19 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 29 Oct 2007 18:07:19 +0100 (CET) Subject: [Python-3000-checkins] r58703 - in python/branches/py3k-pep3137: Lib/urllib2.py Message-ID: <20071029170719.BD5171E4012@bag.python.org> Author: guido.van.rossum Date: Mon Oct 29 18:07:19 2007 New Revision: 58703 Modified: python/branches/py3k-pep3137/ (props changed) python/branches/py3k-pep3137/Lib/urllib2.py Log: Merged revisions 58694-58702 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k ........ r58694 | neal.norwitz | 2007-10-27 10:25:12 -0700 (Sat, 27 Oct 2007) | 3 lines URLError now takes only a single parameter. This was causing errors in test_urllib2net on the MIPS buildbot. ........ r58701 | guido.van.rossum | 2007-10-29 09:42:51 -0700 (Mon, 29 Oct 2007) | 5 lines Issue 1340 by Amaury Forgeot d'Arc (with help from Christian Heimes, and my own interpretation). Don't pass the newline= flag to StringIO in SpooledTemporaryFile. This avoids doubling newlines when the file is rolled over. ........ r58702 | fred.drake | 2007-10-29 09:56:50 -0700 (Mon, 29 Oct 2007) | 1 line fix typo ........ Modified: python/branches/py3k-pep3137/Lib/urllib2.py ============================================================================== --- python/branches/py3k-pep3137/Lib/urllib2.py (original) +++ python/branches/py3k-pep3137/Lib/urllib2.py Mon Oct 29 18:07:19 2007 @@ -1240,7 +1240,7 @@ import mimetypes host = req.get_host() if not host: - raise URLError('ftp error', 'no host given') + raise URLError('ftp error: no host given') host, port = splitport(host) if port is None: port = ftplib.FTP_PORT @@ -1286,7 +1286,7 @@ headers = mimetools.Message(sf) return addinfourl(fp, headers, req.get_full_url()) except ftplib.all_errors as msg: - raise URLError('ftp error', msg).with_traceback(sys.exc_info()[2]) + raise URLError('ftp error %s' % msg).with_traceback(sys.exc_info()[2]) def connect_ftp(self, user, passwd, host, port, dirs, timeout): fw = ftpwrapper(user, passwd, host, port, dirs, timeout) From python-3000-checkins at python.org Mon Oct 29 18:39:59 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 29 Oct 2007 18:39:59 +0100 (CET) Subject: [Python-3000-checkins] r58704 - python/branches/py3k/Lib/fileinput.py Message-ID: <20071029173959.F037A1E4016@bag.python.org> Author: guido.van.rossum Date: Mon Oct 29 18:39:59 2007 New Revision: 58704 Modified: python/branches/py3k/Lib/fileinput.py Log: Patch 1341 by Amaury Forgeot d'Arc. This patch corrects test_fileinput on Windows: when redirecting stdout, os.open should be given O_BINARY, because the file descriptor is then wrapped in a text-mode file; os.fdopen(fd, "w"). Modified: python/branches/py3k/Lib/fileinput.py ============================================================================== --- python/branches/py3k/Lib/fileinput.py (original) +++ python/branches/py3k/Lib/fileinput.py Mon Oct 29 18:39:59 2007 @@ -326,9 +326,11 @@ except OSError: self._output = open(self._filename, "w") else: - fd = os.open(self._filename, - os.O_CREAT | os.O_WRONLY | os.O_TRUNC, - perm) + mode = os.O_CREAT | os.O_WRONLY | os.O_TRUNC + if hasattr(os, 'O_BINARY'): + mode |= os.O_BINARY + + fd = os.open(self._filename, mode, perm) self._output = os.fdopen(fd, "w") try: if hasattr(os, 'chmod'): From python-3000-checkins at python.org Mon Oct 29 20:54:25 2007 From: python-3000-checkins at python.org (martin.v.loewis) Date: Mon, 29 Oct 2007 20:54:25 +0100 (CET) Subject: [Python-3000-checkins] r58705 - python/branches/py3k/Python/symtable.c Message-ID: <20071029195425.11A4C1E4014@bag.python.org> Author: martin.v.loewis Date: Mon Oct 29 20:54:24 2007 New Revision: 58705 Modified: python/branches/py3k/Python/symtable.c Log: Bug #1356: Avoid using the C99 keyword 'restrict'. Modified: python/branches/py3k/Python/symtable.c ============================================================================== --- python/branches/py3k/Python/symtable.c (original) +++ python/branches/py3k/Python/symtable.c Mon Oct 29 20:54:24 2007 @@ -465,12 +465,12 @@ Note that the current block's free variables are included in free. That's safe because no name can be free and local in the same scope. - The 'restrict' argument may be set to a string to restrict the analysis + The 'restricted' argument may be set to a string to restrict the analysis to the one variable whose name equals that string (e.g. "__class__"). */ static int -analyze_cells(PyObject *scopes, PyObject *free, const char *restrict) +analyze_cells(PyObject *scopes, PyObject *free, const char *restricted) { PyObject *name, *v, *v_cell; int success = 0; @@ -487,8 +487,8 @@ continue; if (!PySet_Contains(free, name)) continue; - if (restrict != NULL && - PyUnicode_CompareWithASCIIString(name, restrict)) + if (restricted != NULL && + PyUnicode_CompareWithASCIIString(name, restricted)) continue; /* Replace LOCAL with CELL for this name, and remove from free. It is safe to replace the value of name From python-3000-checkins at python.org Mon Oct 29 22:45:35 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 29 Oct 2007 22:45:35 +0100 (CET) Subject: [Python-3000-checkins] r58707 - in python/branches/py3k-pep3137: Include/stringobject.h Objects/unicodeobject.c Message-ID: <20071029214535.ACED31E4007@bag.python.org> Author: guido.van.rossum Date: Mon Oct 29 22:45:35 2007 New Revision: 58707 Modified: python/branches/py3k-pep3137/Include/stringobject.h python/branches/py3k-pep3137/Objects/unicodeobject.c Log: Get rid of the placeholder in PyStringObject. Amaury Foregeot d'Arc found the culprit: the code for decoding octal escapes in PyUnicode_DecodeUnicodeEscape was reading past the end of the input buffer, which wasn't null-terminated. This probably ought to be backported; it seems pure luck it doesn't trigger in 2.x. Modified: python/branches/py3k-pep3137/Include/stringobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/stringobject.h (original) +++ python/branches/py3k-pep3137/Include/stringobject.h Mon Oct 29 22:45:35 2007 @@ -30,7 +30,6 @@ typedef struct { PyObject_VAR_HEAD long ob_shash; - int ob_placeholder; /* XXX If I remove this things break?!?! */ char ob_sval[1]; /* Invariants: Modified: python/branches/py3k-pep3137/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/unicodeobject.c (original) +++ python/branches/py3k-pep3137/Objects/unicodeobject.c Mon Oct 29 22:45:35 2007 @@ -2671,6 +2671,7 @@ startinpos = s-starts; /* \ - Escapes */ s++; + assert (s < end); /* If this fails, the parser let through garbage */ switch (*s++) { /* \x escapes */ @@ -2690,9 +2691,9 @@ case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': x = s[-1] - '0'; - if ('0' <= *s && *s <= '7') { + if (s < end && '0' <= *s && *s <= '7') { x = (x<<3) + *s++ - '0'; - if ('0' <= *s && *s <= '7') + if (s < end && '0' <= *s && *s <= '7') x = (x<<3) + *s++ - '0'; } *p++ = x; From python-3000-checkins at python.org Mon Oct 29 23:05:21 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Mon, 29 Oct 2007 23:05:21 +0100 (CET) Subject: [Python-3000-checkins] r58708 - python/branches/py3k-pep3137/Objects/stringobject.c python/branches/py3k-pep3137/Objects/unicodeobject.c Message-ID: <20071029220521.F324E1E4012@bag.python.org> Author: guido.van.rossum Date: Mon Oct 29 23:05:21 2007 New Revision: 58708 Modified: python/branches/py3k-pep3137/Objects/stringobject.c python/branches/py3k-pep3137/Objects/unicodeobject.c Log: Supplemental fix for issue 1359. Also fixing it for PyString, which has the same issue (and also for \x!). Modified: python/branches/py3k-pep3137/Objects/stringobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/stringobject.c (original) +++ python/branches/py3k-pep3137/Objects/stringobject.c Mon Oct 29 23:05:21 2007 @@ -409,15 +409,15 @@ case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': c = s[-1] - '0'; - if ('0' <= *s && *s <= '7') { + if (s < end && '0' <= *s && *s <= '7') { c = (c<<3) + *s++ - '0'; - if ('0' <= *s && *s <= '7') + if (s < end && '0' <= *s && *s <= '7') c = (c<<3) + *s++ - '0'; } *p++ = c; break; case 'x': - if (ISXDIGIT(s[0]) && ISXDIGIT(s[1])) { + if (s+1 < end && ISXDIGIT(s[0]) && ISXDIGIT(s[1])) { unsigned int x = 0; c = Py_CHARMASK(*s); s++; Modified: python/branches/py3k-pep3137/Objects/unicodeobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/unicodeobject.c (original) +++ python/branches/py3k-pep3137/Objects/unicodeobject.c Mon Oct 29 23:05:21 2007 @@ -2671,8 +2671,10 @@ startinpos = s-starts; /* \ - Escapes */ s++; - assert (s < end); /* If this fails, the parser let through garbage */ - switch (*s++) { + c = *s++; + if (s > end) + c = '\0'; /* Invalid after \ */ + switch (c) { /* \x escapes */ case '\n': break; From python-3000-checkins at python.org Tue Oct 30 18:27:31 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 30 Oct 2007 18:27:31 +0100 (CET) Subject: [Python-3000-checkins] r58711 - in python/branches/py3k: Doc/c-api/concrete.rst Include/fileobject.h Lib/io.py Lib/quopri.py Lib/test/test_io.py Misc/NEWS Modules/_fileio.c Objects/fileobject.c Python/import.c Python/pythonrun.c Message-ID: <20071030172731.0FBD31E478C@bag.python.org> Author: guido.van.rossum Date: Tue Oct 30 18:27:30 2007 New Revision: 58711 Modified: python/branches/py3k/Doc/c-api/concrete.rst python/branches/py3k/Include/fileobject.h python/branches/py3k/Lib/io.py python/branches/py3k/Lib/quopri.py python/branches/py3k/Lib/test/test_io.py python/branches/py3k/Misc/NEWS python/branches/py3k/Modules/_fileio.c python/branches/py3k/Objects/fileobject.c python/branches/py3k/Python/import.c python/branches/py3k/Python/pythonrun.c Log: Patch 1329 (partial) by Christian Heimes. Add a closefd flag to open() which can be set to False to prevent closing the file descriptor when close() is called or when the object is destroyed. Useful to ensure that sys.std{in,out,err} keep their file descriptors open when Python is uninitialized. (This was always a feature in 2.x, it just wasn't implemented in 3.0 yet.) Modified: python/branches/py3k/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k/Doc/c-api/concrete.rst (original) +++ python/branches/py3k/Doc/c-api/concrete.rst Tue Oct 30 18:27:30 2007 @@ -2401,12 +2401,12 @@ :ctype:`PyFileObject`. -.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *newline) +.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *newline, int closefd) Create a new :ctype:`PyFileObject` from the file descriptor of an already opened file *fd*. The arguments *name*, *encoding* and *newline* can be - *NULL* as well as buffering can be *-1* to use the defaults. Return *NULL* on - failure. + *NULL* to use the defaults; *buffering* can be *-1* to use the default. + Return *NULL* on failure. .. warning:: Modified: python/branches/py3k/Include/fileobject.h ============================================================================== --- python/branches/py3k/Include/fileobject.h (original) +++ python/branches/py3k/Include/fileobject.h Tue Oct 30 18:27:30 2007 @@ -8,7 +8,8 @@ #define PY_STDIOTEXTMODE "b" -PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *); +PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *, + int); PyAPI_FUNC(PyObject *) PyFile_GetLine(PyObject *, int); PyAPI_FUNC(int) PyFile_WriteObject(PyObject *, PyObject *, int); PyAPI_FUNC(int) PyFile_WriteString(const char *, PyObject *); Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Tue Oct 30 18:27:30 2007 @@ -49,7 +49,8 @@ self.characters_written = characters_written -def open(file, mode="r", buffering=None, encoding=None, newline=None): +def open(file, mode="r", buffering=None, encoding=None, newline=None, + closefd=True): r"""Replacement for the built-in open function. Args: @@ -81,9 +82,12 @@ other legal values, any `'\n'` characters written are translated to the given string. + closefd: optional argument to keep the underlying file descriptor + open when the file is closed. It must not be false when + a filename is given. + (*) If a file descriptor is given, it is closed when the returned - I/O object is closed. If you don't want this to happen, use - os.dup() to create a duplicate file descriptor. + I/O object is closed, unless closefd=False is give. Mode strings characters: 'r': open for reading (default) @@ -138,7 +142,8 @@ (reading and "r" or "") + (writing and "w" or "") + (appending and "a" or "") + - (updating and "+" or "")) + (updating and "+" or ""), + closefd) if buffering is None: buffering = -1 if buffering < 0 and raw.isatty(): Modified: python/branches/py3k/Lib/quopri.py ============================================================================== --- python/branches/py3k/Lib/quopri.py (original) +++ python/branches/py3k/Lib/quopri.py Tue Oct 30 18:27:30 2007 @@ -227,12 +227,14 @@ sys.stderr.write("%s: can't open (%s)\n" % (file, msg)) sts = 1 continue - if deco: - decode(fp, sys.stdout.buffer) - else: - encode(fp, sys.stdout.buffer, tabs) - if fp is not sys.stdin: - fp.close() + try: + if deco: + decode(fp, sys.stdout.buffer) + else: + encode(fp, sys.stdout.buffer, tabs) + finally: + if file != '-': + fp.close() if sts: sys.exit(sts) Modified: python/branches/py3k/Lib/test/test_io.py ============================================================================== --- python/branches/py3k/Lib/test/test_io.py (original) +++ python/branches/py3k/Lib/test/test_io.py Tue Oct 30 18:27:30 2007 @@ -259,6 +259,9 @@ self.assertEqual(f.write(a), n) f.close() + def test_closefd(self): + self.assertRaises(ValueError, io.open, test_support.TESTFN, 'w', + closefd=False) class MemorySeekTestMixin: Modified: python/branches/py3k/Misc/NEWS ============================================================================== --- python/branches/py3k/Misc/NEWS (original) +++ python/branches/py3k/Misc/NEWS Tue Oct 30 18:27:30 2007 @@ -28,6 +28,9 @@ with `Py_FileSystemDefaultEncoding` and a new API method `PyUnicode_DecodeFSDefault(char*)` was added. +- io.open() and _fileio.FileIO have grown a new argument closefd. A false + value disables the closing of the file descriptor. + Extension Modules ----------------- Modified: python/branches/py3k/Modules/_fileio.c ============================================================================== --- python/branches/py3k/Modules/_fileio.c (original) +++ python/branches/py3k/Modules/_fileio.c Tue Oct 30 18:27:30 2007 @@ -33,6 +33,7 @@ unsigned readable : 1; unsigned writable : 1; int seekable : 2; /* -1 means unknown */ + int closefd : 1; PyObject *weakreflist; } PyFileIOObject; @@ -59,6 +60,13 @@ static PyObject * fileio_close(PyFileIOObject *self) { + if (!self->closefd) { + if (PyErr_WarnEx(PyExc_RuntimeWarning, + "Trying to close unclosable fd!", 3) < 0) { + return NULL; + } + Py_RETURN_NONE; + } errno = internal_close(self); if (errno < 0) { PyErr_SetFromErrno(PyExc_IOError); @@ -119,7 +127,7 @@ fileio_init(PyObject *oself, PyObject *args, PyObject *kwds) { PyFileIOObject *self = (PyFileIOObject *) oself; - static char *kwlist[] = {"file", "mode", NULL}; + static char *kwlist[] = {"file", "mode", "closefd", NULL}; char *name = NULL; char *mode = "r"; char *s; @@ -130,6 +138,7 @@ int rwa = 0, plus = 0, append = 0; int flags = 0; int fd = -1; + int closefd = 1; assert(PyFileIO_Check(oself)); if (self->fd >= 0) { @@ -138,8 +147,8 @@ return -1; } - if (PyArg_ParseTupleAndKeywords(args, kwds, "i|s:fileio", - kwlist, &fd, &mode)) { + if (PyArg_ParseTupleAndKeywords(args, kwds, "i|si:fileio", + kwlist, &fd, &mode, &closefd)) { if (fd < 0) { PyErr_SetString(PyExc_ValueError, "Negative filedescriptor"); @@ -153,8 +162,9 @@ if (GetVersion() < 0x80000000) { /* On NT, so wide API available */ PyObject *po; - if (PyArg_ParseTupleAndKeywords(args, kwds, "U|s:fileio", - kwlist, &po, &mode)) { + if (PyArg_ParseTupleAndKeywords(args, kwds, "U|si:fileio", + kwlist, &po, &mode, &closefd) + ) { widename = PyUnicode_AS_UNICODE(po); } else { /* Drop the argument parsing error as narrow @@ -162,13 +172,13 @@ PyErr_Clear(); } } - if (widename == NULL) + if (widename == NULL) #endif { - if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|s:fileio", + if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|si:fileio", kwlist, Py_FileSystemDefaultEncoding, - &name, &mode)) + &name, &mode, &closefd)) goto error; } } @@ -237,8 +247,16 @@ if (fd >= 0) { self->fd = fd; + self->closefd = closefd; } else { + self->closefd = 1; + if (!closefd) { + PyErr_SetString(PyExc_ValueError, + "Cannot use closefd=True with file name"); + goto error; + } + Py_BEGIN_ALLOW_THREADS errno = 0; #ifdef MS_WINDOWS @@ -270,7 +288,7 @@ if (self->weakreflist != NULL) PyObject_ClearWeakRefs((PyObject *) self); - if (self->fd >= 0) { + if (self->fd >= 0 && self->closefd) { errno = internal_close(self); if (errno < 0) { #ifdef HAVE_STRERROR Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Tue Oct 30 18:27:30 2007 @@ -27,15 +27,15 @@ PyObject * PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, - char *newline) + char *newline, int closefd) { PyObject *io, *stream, *nameobj = NULL; io = PyImport_ImportModule("io"); if (io == NULL) return NULL; - stream = PyObject_CallMethod(io, "open", "isiss", fd, mode, - buffering, encoding, newline); + stream = PyObject_CallMethod(io, "open", "isissi", fd, mode, + buffering, encoding, newline, closefd); Py_DECREF(io); if (stream == NULL) return NULL; Modified: python/branches/py3k/Python/import.c ============================================================================== --- python/branches/py3k/Python/import.c (original) +++ python/branches/py3k/Python/import.c Tue Oct 30 18:27:30 2007 @@ -2588,7 +2588,7 @@ (char*)PyUnicode_GetDefaultEncoding(); } fob = PyFile_FromFd(fd, pathname, fdp->mode, -1, - (char*)encoding, NULL); + (char*)encoding, NULL, 1); if (fob == NULL) { close(fd); PyMem_FREE(found_encoding); Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Tue Oct 30 18:27:30 2007 @@ -720,7 +720,7 @@ /* Set sys.stdin */ if (!(std = PyFile_FromFd(fileno(stdin), "", "r", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stdin__", std); @@ -729,16 +729,16 @@ /* Set sys.stdout */ if (!(std = PyFile_FromFd(fileno(stdout), "", "w", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stdout__", std); PySys_SetObject("stdout", std); Py_DECREF(std); - /* Set sys.stderr */ + /* Set sys.stderr, replaces the preliminary stderr */ if (!(std = PyFile_FromFd(fileno(stderr), "", "w", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stderr__", std); From python-3000-checkins at python.org Tue Oct 30 18:35:35 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 30 Oct 2007 18:35:35 +0100 (CET) Subject: [Python-3000-checkins] r58713 - in python/branches/py3k-pep3137: Doc/c-api/concrete.rst Include/fileobject.h Lib/fileinput.py Lib/io.py Lib/quopri.py Lib/test/test_io.py Misc/NEWS Modules/_fileio.c Objects/fileobject.c Python/import.c Python/pythonrun.c Python/symtable.c Message-ID: <20071030173535.2288D1E47D8@bag.python.org> Author: guido.van.rossum Date: Tue Oct 30 18:35:34 2007 New Revision: 58713 Modified: python/branches/py3k-pep3137/ (props changed) python/branches/py3k-pep3137/Doc/c-api/concrete.rst python/branches/py3k-pep3137/Include/fileobject.h python/branches/py3k-pep3137/Lib/fileinput.py python/branches/py3k-pep3137/Lib/io.py python/branches/py3k-pep3137/Lib/quopri.py python/branches/py3k-pep3137/Lib/test/test_io.py python/branches/py3k-pep3137/Misc/NEWS python/branches/py3k-pep3137/Modules/_fileio.c python/branches/py3k-pep3137/Objects/fileobject.c python/branches/py3k-pep3137/Python/import.c python/branches/py3k-pep3137/Python/pythonrun.c python/branches/py3k-pep3137/Python/symtable.c Log: Merged revisions 58703-58712 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k ........ r58704 | guido.van.rossum | 2007-10-29 10:39:59 -0700 (Mon, 29 Oct 2007) | 5 lines Patch 1341 by Amaury Forgeot d'Arc. This patch corrects test_fileinput on Windows: when redirecting stdout, os.open should be given O_BINARY, because the file descriptor is then wrapped in a text-mode file; os.fdopen(fd, "w"). ........ r58705 | martin.v.loewis | 2007-10-29 12:54:24 -0700 (Mon, 29 Oct 2007) | 2 lines Bug #1356: Avoid using the C99 keyword 'restrict'. ........ r58711 | guido.van.rossum | 2007-10-30 10:27:30 -0700 (Tue, 30 Oct 2007) | 7 lines Patch 1329 (partial) by Christian Heimes. Add a closefd flag to open() which can be set to False to prevent closing the file descriptor when close() is called or when the object is destroyed. Useful to ensure that sys.std{in,out,err} keep their file descriptors open when Python is uninitialized. (This was always a feature in 2.x, it just wasn't implemented in 3.0 yet.) ........ Modified: python/branches/py3k-pep3137/Doc/c-api/concrete.rst ============================================================================== --- python/branches/py3k-pep3137/Doc/c-api/concrete.rst (original) +++ python/branches/py3k-pep3137/Doc/c-api/concrete.rst Tue Oct 30 18:35:34 2007 @@ -2401,12 +2401,12 @@ :ctype:`PyFileObject`. -.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *newline) +.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *newline, int closefd) Create a new :ctype:`PyFileObject` from the file descriptor of an already opened file *fd*. The arguments *name*, *encoding* and *newline* can be - *NULL* as well as buffering can be *-1* to use the defaults. Return *NULL* on - failure. + *NULL* to use the defaults; *buffering* can be *-1* to use the default. + Return *NULL* on failure. .. warning:: Modified: python/branches/py3k-pep3137/Include/fileobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/fileobject.h (original) +++ python/branches/py3k-pep3137/Include/fileobject.h Tue Oct 30 18:35:34 2007 @@ -8,7 +8,8 @@ #define PY_STDIOTEXTMODE "b" -PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *); +PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *, + int); PyAPI_FUNC(PyObject *) PyFile_GetLine(PyObject *, int); PyAPI_FUNC(int) PyFile_WriteObject(PyObject *, PyObject *, int); PyAPI_FUNC(int) PyFile_WriteString(const char *, PyObject *); Modified: python/branches/py3k-pep3137/Lib/fileinput.py ============================================================================== --- python/branches/py3k-pep3137/Lib/fileinput.py (original) +++ python/branches/py3k-pep3137/Lib/fileinput.py Tue Oct 30 18:35:34 2007 @@ -326,9 +326,11 @@ except OSError: self._output = open(self._filename, "w") else: - fd = os.open(self._filename, - os.O_CREAT | os.O_WRONLY | os.O_TRUNC, - perm) + mode = os.O_CREAT | os.O_WRONLY | os.O_TRUNC + if hasattr(os, 'O_BINARY'): + mode |= os.O_BINARY + + fd = os.open(self._filename, mode, perm) self._output = os.fdopen(fd, "w") try: if hasattr(os, 'chmod'): Modified: python/branches/py3k-pep3137/Lib/io.py ============================================================================== --- python/branches/py3k-pep3137/Lib/io.py (original) +++ python/branches/py3k-pep3137/Lib/io.py Tue Oct 30 18:35:34 2007 @@ -49,7 +49,8 @@ self.characters_written = characters_written -def open(file, mode="r", buffering=None, encoding=None, newline=None): +def open(file, mode="r", buffering=None, encoding=None, newline=None, + closefd=True): r"""Replacement for the built-in open function. Args: @@ -81,9 +82,12 @@ other legal values, any `'\n'` characters written are translated to the given string. + closefd: optional argument to keep the underlying file descriptor + open when the file is closed. It must not be false when + a filename is given. + (*) If a file descriptor is given, it is closed when the returned - I/O object is closed. If you don't want this to happen, use - os.dup() to create a duplicate file descriptor. + I/O object is closed, unless closefd=False is give. Mode strings characters: 'r': open for reading (default) @@ -138,7 +142,8 @@ (reading and "r" or "") + (writing and "w" or "") + (appending and "a" or "") + - (updating and "+" or "")) + (updating and "+" or ""), + closefd) if buffering is None: buffering = -1 if buffering < 0 and raw.isatty(): Modified: python/branches/py3k-pep3137/Lib/quopri.py ============================================================================== --- python/branches/py3k-pep3137/Lib/quopri.py (original) +++ python/branches/py3k-pep3137/Lib/quopri.py Tue Oct 30 18:35:34 2007 @@ -227,12 +227,14 @@ sys.stderr.write("%s: can't open (%s)\n" % (file, msg)) sts = 1 continue - if deco: - decode(fp, sys.stdout.buffer) - else: - encode(fp, sys.stdout.buffer, tabs) - if fp is not sys.stdin: - fp.close() + try: + if deco: + decode(fp, sys.stdout.buffer) + else: + encode(fp, sys.stdout.buffer, tabs) + finally: + if file != '-': + fp.close() if sts: sys.exit(sts) Modified: python/branches/py3k-pep3137/Lib/test/test_io.py ============================================================================== --- python/branches/py3k-pep3137/Lib/test/test_io.py (original) +++ python/branches/py3k-pep3137/Lib/test/test_io.py Tue Oct 30 18:35:34 2007 @@ -259,6 +259,9 @@ self.assertEqual(f.write(a), n) f.close() + def test_closefd(self): + self.assertRaises(ValueError, io.open, test_support.TESTFN, 'w', + closefd=False) class MemorySeekTestMixin: Modified: python/branches/py3k-pep3137/Misc/NEWS ============================================================================== --- python/branches/py3k-pep3137/Misc/NEWS (original) +++ python/branches/py3k-pep3137/Misc/NEWS Tue Oct 30 18:35:34 2007 @@ -28,6 +28,9 @@ with `Py_FileSystemDefaultEncoding` and a new API method `PyUnicode_DecodeFSDefault(char*)` was added. +- io.open() and _fileio.FileIO have grown a new argument closefd. A false + value disables the closing of the file descriptor. + Extension Modules ----------------- Modified: python/branches/py3k-pep3137/Modules/_fileio.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_fileio.c (original) +++ python/branches/py3k-pep3137/Modules/_fileio.c Tue Oct 30 18:35:34 2007 @@ -33,6 +33,7 @@ unsigned readable : 1; unsigned writable : 1; int seekable : 2; /* -1 means unknown */ + int closefd : 1; PyObject *weakreflist; } PyFileIOObject; @@ -59,6 +60,13 @@ static PyObject * fileio_close(PyFileIOObject *self) { + if (!self->closefd) { + if (PyErr_WarnEx(PyExc_RuntimeWarning, + "Trying to close unclosable fd!", 3) < 0) { + return NULL; + } + Py_RETURN_NONE; + } errno = internal_close(self); if (errno < 0) { PyErr_SetFromErrno(PyExc_IOError); @@ -119,7 +127,7 @@ fileio_init(PyObject *oself, PyObject *args, PyObject *kwds) { PyFileIOObject *self = (PyFileIOObject *) oself; - static char *kwlist[] = {"file", "mode", NULL}; + static char *kwlist[] = {"file", "mode", "closefd", NULL}; char *name = NULL; char *mode = "r"; char *s; @@ -130,6 +138,7 @@ int rwa = 0, plus = 0, append = 0; int flags = 0; int fd = -1; + int closefd = 1; assert(PyFileIO_Check(oself)); if (self->fd >= 0) { @@ -138,8 +147,8 @@ return -1; } - if (PyArg_ParseTupleAndKeywords(args, kwds, "i|s:fileio", - kwlist, &fd, &mode)) { + if (PyArg_ParseTupleAndKeywords(args, kwds, "i|si:fileio", + kwlist, &fd, &mode, &closefd)) { if (fd < 0) { PyErr_SetString(PyExc_ValueError, "Negative filedescriptor"); @@ -153,8 +162,9 @@ if (GetVersion() < 0x80000000) { /* On NT, so wide API available */ PyObject *po; - if (PyArg_ParseTupleAndKeywords(args, kwds, "U|s:fileio", - kwlist, &po, &mode)) { + if (PyArg_ParseTupleAndKeywords(args, kwds, "U|si:fileio", + kwlist, &po, &mode, &closefd) + ) { widename = PyUnicode_AS_UNICODE(po); } else { /* Drop the argument parsing error as narrow @@ -162,13 +172,13 @@ PyErr_Clear(); } } - if (widename == NULL) + if (widename == NULL) #endif { - if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|s:fileio", + if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|si:fileio", kwlist, Py_FileSystemDefaultEncoding, - &name, &mode)) + &name, &mode, &closefd)) goto error; } } @@ -237,8 +247,16 @@ if (fd >= 0) { self->fd = fd; + self->closefd = closefd; } else { + self->closefd = 1; + if (!closefd) { + PyErr_SetString(PyExc_ValueError, + "Cannot use closefd=True with file name"); + goto error; + } + Py_BEGIN_ALLOW_THREADS errno = 0; #ifdef MS_WINDOWS @@ -270,7 +288,7 @@ if (self->weakreflist != NULL) PyObject_ClearWeakRefs((PyObject *) self); - if (self->fd >= 0) { + if (self->fd >= 0 && self->closefd) { errno = internal_close(self); if (errno < 0) { #ifdef HAVE_STRERROR Modified: python/branches/py3k-pep3137/Objects/fileobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/fileobject.c (original) +++ python/branches/py3k-pep3137/Objects/fileobject.c Tue Oct 30 18:35:34 2007 @@ -27,15 +27,15 @@ PyObject * PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, - char *newline) + char *newline, int closefd) { PyObject *io, *stream, *nameobj = NULL; io = PyImport_ImportModule("io"); if (io == NULL) return NULL; - stream = PyObject_CallMethod(io, "open", "isiss", fd, mode, - buffering, encoding, newline); + stream = PyObject_CallMethod(io, "open", "isissi", fd, mode, + buffering, encoding, newline, closefd); Py_DECREF(io); if (stream == NULL) return NULL; Modified: python/branches/py3k-pep3137/Python/import.c ============================================================================== --- python/branches/py3k-pep3137/Python/import.c (original) +++ python/branches/py3k-pep3137/Python/import.c Tue Oct 30 18:35:34 2007 @@ -2589,7 +2589,7 @@ (char*)PyUnicode_GetDefaultEncoding(); } fob = PyFile_FromFd(fd, pathname, fdp->mode, -1, - (char*)encoding, NULL); + (char*)encoding, NULL, 1); if (fob == NULL) { close(fd); PyMem_FREE(found_encoding); Modified: python/branches/py3k-pep3137/Python/pythonrun.c ============================================================================== --- python/branches/py3k-pep3137/Python/pythonrun.c (original) +++ python/branches/py3k-pep3137/Python/pythonrun.c Tue Oct 30 18:35:34 2007 @@ -720,7 +720,7 @@ /* Set sys.stdin */ if (!(std = PyFile_FromFd(fileno(stdin), "", "r", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stdin__", std); @@ -729,16 +729,16 @@ /* Set sys.stdout */ if (!(std = PyFile_FromFd(fileno(stdout), "", "w", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stdout__", std); PySys_SetObject("stdout", std); Py_DECREF(std); - /* Set sys.stderr */ + /* Set sys.stderr, replaces the preliminary stderr */ if (!(std = PyFile_FromFd(fileno(stderr), "", "w", -1, - NULL, "\n"))) { + NULL, "\n", 0))) { goto error; } PySys_SetObject("__stderr__", std); Modified: python/branches/py3k-pep3137/Python/symtable.c ============================================================================== --- python/branches/py3k-pep3137/Python/symtable.c (original) +++ python/branches/py3k-pep3137/Python/symtable.c Tue Oct 30 18:35:34 2007 @@ -465,12 +465,12 @@ Note that the current block's free variables are included in free. That's safe because no name can be free and local in the same scope. - The 'restrict' argument may be set to a string to restrict the analysis + The 'restricted' argument may be set to a string to restrict the analysis to the one variable whose name equals that string (e.g. "__class__"). */ static int -analyze_cells(PyObject *scopes, PyObject *free, const char *restrict) +analyze_cells(PyObject *scopes, PyObject *free, const char *restricted) { PyObject *name, *v, *v_cell; int success = 0; @@ -487,8 +487,8 @@ continue; if (!PySet_Contains(free, name)) continue; - if (restrict != NULL && - PyUnicode_CompareWithASCIIString(name, restrict)) + if (restricted != NULL && + PyUnicode_CompareWithASCIIString(name, restricted)) continue; /* Replace LOCAL with CELL for this name, and remove from free. It is safe to replace the value of name From python-3000-checkins at python.org Tue Oct 30 18:42:20 2007 From: python-3000-checkins at python.org (georg.brandl) Date: Tue, 30 Oct 2007 18:42:20 +0100 (CET) Subject: [Python-3000-checkins] r58714 - python/branches/py3k/Lib/io.py Message-ID: <20071030174220.C54011E401F@bag.python.org> Author: georg.brandl Date: Tue Oct 30 18:42:20 2007 New Revision: 58714 Modified: python/branches/py3k/Lib/io.py Log: Fix typo. Modified: python/branches/py3k/Lib/io.py ============================================================================== --- python/branches/py3k/Lib/io.py (original) +++ python/branches/py3k/Lib/io.py Tue Oct 30 18:42:20 2007 @@ -87,7 +87,7 @@ a filename is given. (*) If a file descriptor is given, it is closed when the returned - I/O object is closed, unless closefd=False is give. + I/O object is closed, unless closefd=False is given. Mode strings characters: 'r': open for reading (default) From python-3000-checkins at python.org Tue Oct 30 19:12:39 2007 From: python-3000-checkins at python.org (bill.janssen) Date: Tue, 30 Oct 2007 19:12:39 +0100 (CET) Subject: [Python-3000-checkins] r58717 - python/branches/py3k/Lib/plat-mac/ic.py Message-ID: <20071030181239.83E7B1E4007@bag.python.org> Author: bill.janssen Date: Tue Oct 30 19:12:39 2007 New Revision: 58717 Modified: python/branches/py3k/Lib/plat-mac/ic.py Log: remove ord() calls on what are now integers rather than characters Modified: python/branches/py3k/Lib/plat-mac/ic.py ============================================================================== --- python/branches/py3k/Lib/plat-mac/ic.py (original) +++ python/branches/py3k/Lib/plat-mac/ic.py Tue Oct 30 19:12:39 2007 @@ -41,27 +41,27 @@ def _decode_default(data, key): if len(data) == 0: return data - if ord(data[0]) == len(data)-1: + if data[0] == len(data)-1: # Assume Pstring return data[1:] return ICOpaqueData(data) def _decode_multistr(data, key): - numstr = ord(data[0]) << 8 | ord(data[1]) + numstr = data[0] << 8 | data[1] rv = [] ptr = 2 for i in range(numstr): - strlen = ord(data[ptr]) + strlen = data[ptr] str = data[ptr+1:ptr+strlen+1] rv.append(str) ptr = ptr + strlen + 1 return rv def _decode_fontrecord(data, key): - size = ord(data[0]) << 8 | ord(data[1]) - face = ord(data[2]) - namelen = ord(data[4]) + size = data[0] << 8 | data[1] + face = data[2] + namelen = data[4] return size, face, data[5:5+namelen] def _decode_boolean(data, key): @@ -74,7 +74,7 @@ return data[:256], data[256:] def _decode_appspec(data, key): - namelen = ord(data[4]) + namelen = data[4] return data[0:4], data[5:5+namelen] def _code_default(data, key): From python-3000-checkins at python.org Tue Oct 30 19:13:17 2007 From: python-3000-checkins at python.org (bill.janssen) Date: Tue, 30 Oct 2007 19:13:17 +0100 (CET) Subject: [Python-3000-checkins] r58718 - python/branches/py3k/Lib/BaseHTTPServer.py Message-ID: <20071030181317.550841E4007@bag.python.org> Author: bill.janssen Date: Tue Oct 30 19:13:17 2007 New Revision: 58718 Modified: python/branches/py3k/Lib/BaseHTTPServer.py Log: make sure to write bytes instead of strings to underlying socket channel Modified: python/branches/py3k/Lib/BaseHTTPServer.py ============================================================================== --- python/branches/py3k/Lib/BaseHTTPServer.py (original) +++ python/branches/py3k/Lib/BaseHTTPServer.py Tue Oct 30 19:13:17 2007 @@ -356,11 +356,11 @@ content = (self.error_message_format % {'code': code, 'message': _quote_html(message), 'explain': explain}) self.send_response(code, message) - self.send_header("Content-Type", "text/html") + self.send_header("Content-Type", "text/html;charset=utf-8") self.send_header('Connection', 'close') self.end_headers() if self.command != 'HEAD' and code >= 200 and code not in (204, 304): - self.wfile.write(content) + self.wfile.write(content.encode('UTF-8', 'replace')) error_message_format = DEFAULT_ERROR_MESSAGE @@ -378,8 +378,8 @@ else: message = '' if self.request_version != 'HTTP/0.9': - self.wfile.write("%s %d %s\r\n" % - (self.protocol_version, code, message)) + self.wfile.write(("%s %d %s\r\n" % + (self.protocol_version, code, message)).encode('ASCII', 'strict')) # print (self.protocol_version, code, message) self.send_header('Server', self.version_string()) self.send_header('Date', self.date_time_string()) @@ -387,7 +387,7 @@ def send_header(self, keyword, value): """Send a MIME header.""" if self.request_version != 'HTTP/0.9': - self.wfile.write("%s: %s\r\n" % (keyword, value)) + self.wfile.write(("%s: %s\r\n" % (keyword, value)).encode('ASCII', 'strict')) if keyword.lower() == 'connection': if value.lower() == 'close': @@ -398,7 +398,7 @@ def end_headers(self): """Send the blank line ending the MIME headers.""" if self.request_version != 'HTTP/0.9': - self.wfile.write("\r\n") + self.wfile.write(b"\r\n") def log_request(self, code='-', size='-'): """Log an accepted request. From python-3000-checkins at python.org Tue Oct 30 19:34:07 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 30 Oct 2007 19:34:07 +0100 (CET) Subject: [Python-3000-checkins] r58719 - in python/branches/py3k: Include/fileobject.h Objects/fileobject.c Objects/object.c Python/pythonrun.c Message-ID: <20071030183408.40D5A1E4008@bag.python.org> Author: guido.van.rossum Date: Tue Oct 30 19:34:07 2007 New Revision: 58719 Modified: python/branches/py3k/Include/fileobject.h python/branches/py3k/Objects/fileobject.c python/branches/py3k/Objects/object.c python/branches/py3k/Python/pythonrun.c Log: Patch 1352 (continued in issue 1329) by Christian Heimes. Before sys.stderr is set to the proper thing, set it to a really simple file-like object that can print tracebacks using direct file descriptor I/O. This is handy for debugging. Modified: python/branches/py3k/Include/fileobject.h ============================================================================== --- python/branches/py3k/Include/fileobject.h (original) +++ python/branches/py3k/Include/fileobject.h Tue Oct 30 19:34:07 2007 @@ -21,6 +21,13 @@ */ PyAPI_DATA(const char *) Py_FileSystemDefaultEncoding; +/* Internal API + + The std printer acts as a preliminary sys.stderr until the new io + infrastructure is in place. */ +PyAPI_FUNC(PyObject *) PyFile_NewStdPrinter(int); +PyAPI_DATA(PyTypeObject) PyStdPrinter_Type; + #ifdef __cplusplus } #endif Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Tue Oct 30 19:34:07 2007 @@ -325,6 +325,124 @@ return buf; } +/* **************************** std printer **************************** */ + +typedef struct { + PyObject_HEAD + int fd; +} PyStdPrinter_Object; + +static PyObject * +stdprinter_new(PyTypeObject *type, PyObject *args, PyObject *kews) +{ + PyStdPrinter_Object *self; + + assert(type != NULL && type->tp_alloc != NULL); + + self = (PyStdPrinter_Object *) type->tp_alloc(type, 0); + if (self != NULL) { + self->fd = -1; + } + + return (PyObject *) self; +} + +PyObject * +PyFile_NewStdPrinter(int fd) +{ + PyStdPrinter_Object *self; + + if (fd != 1 && fd != 2) { + PyErr_BadInternalCall(); + return NULL; + } + + self = PyObject_New(PyStdPrinter_Object, + &PyStdPrinter_Type); + self->fd = fd; + return (PyObject*)self; +} + +PyObject * +stdprinter_write(PyStdPrinter_Object *self, PyObject *args) +{ + char *c; + Py_ssize_t n; + + if (self->fd < 0) { + PyErr_SetString(PyExc_ValueError, + "I/O operation on closed file"); + return NULL; + } + + if (!PyArg_ParseTuple(args, "s#", &c, &n)) { + return NULL; + } + + Py_BEGIN_ALLOW_THREADS + errno = 0; + n = write(self->fd, c, n); + Py_END_ALLOW_THREADS + + if (n < 0) { + if (errno == EAGAIN) + Py_RETURN_NONE; + PyErr_SetFromErrno(PyExc_IOError); + return NULL; + } + + return PyInt_FromSsize_t(n); +} + +static PyMethodDef stdprinter_methods[] = { + {"write", (PyCFunction)stdprinter_write, METH_VARARGS, ""}, + {NULL, NULL} /* sentinel */ +}; + +PyTypeObject PyStdPrinter_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "stderrprinter", /* tp_name */ + sizeof(PyStdPrinter_Object), /* tp_basicsize */ + 0, /* tp_itemsize */ + /* methods */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + 0, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + stdprinter_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + 0, /* tp_init */ + PyType_GenericAlloc, /* tp_alloc */ + stdprinter_new, /* tp_new */ + PyObject_Del, /* tp_free */ +}; + + #ifdef __cplusplus } #endif Modified: python/branches/py3k/Objects/object.c ============================================================================== --- python/branches/py3k/Objects/object.c (original) +++ python/branches/py3k/Objects/object.c Tue Oct 30 19:34:07 2007 @@ -1595,6 +1595,9 @@ if (PyType_Ready(&PyCode_Type) < 0) Py_FatalError("Can't initialize 'code'"); + + if (PyType_Ready(&PyStdPrinter_Type) < 0) + Py_FatalError("Can't initialize StdPrinter"); } Modified: python/branches/py3k/Python/pythonrun.c ============================================================================== --- python/branches/py3k/Python/pythonrun.c (original) +++ python/branches/py3k/Python/pythonrun.c Tue Oct 30 19:34:07 2007 @@ -151,7 +151,7 @@ { PyInterpreterState *interp; PyThreadState *tstate; - PyObject *bimod, *sysmod; + PyObject *bimod, *sysmod, *pstderr; char *p; #if defined(HAVE_LANGINFO_H) && defined(CODESET) char *codeset; @@ -228,6 +228,13 @@ PyDict_SetItemString(interp->sysdict, "modules", interp->modules); + /* Set up a preliminary stderr printer until we have enough + infrastructure for the io module in place. */ + pstderr = PyFile_NewStdPrinter(fileno(stderr)); + if (pstderr == NULL) + Py_FatalError("Py_Initialize: can't set preliminary stderr"); + PySys_SetObject("stderr", pstderr); + _PyImport_Init(); /* initialize builtin exceptions */ From python-3000-checkins at python.org Tue Oct 30 19:36:44 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 30 Oct 2007 19:36:44 +0100 (CET) Subject: [Python-3000-checkins] r58720 - python/branches/py3k/Objects/fileobject.c Message-ID: <20071030183644.5ACB81E401D@bag.python.org> Author: guido.van.rossum Date: Tue Oct 30 19:36:44 2007 New Revision: 58720 Modified: python/branches/py3k/Objects/fileobject.c Log: Minor correction to the stdprinter object. Modified: python/branches/py3k/Objects/fileobject.c ============================================================================== --- python/branches/py3k/Objects/fileobject.c (original) +++ python/branches/py3k/Objects/fileobject.c Tue Oct 30 19:36:44 2007 @@ -352,14 +352,16 @@ { PyStdPrinter_Object *self; - if (fd != 1 && fd != 2) { + if (fd != fileno(stdout) && fd != fileno(stderr)) { PyErr_BadInternalCall(); return NULL; } self = PyObject_New(PyStdPrinter_Object, &PyStdPrinter_Type); - self->fd = fd; + if (self != NULL) { + self->fd = fd; + } return (PyObject*)self; } From python-3000-checkins at python.org Tue Oct 30 20:14:52 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Tue, 30 Oct 2007 20:14:52 +0100 (CET) Subject: [Python-3000-checkins] r58721 - in python/branches/py3k-pep3137: Include/fileobject.h Lib/BaseHTTPServer.py Lib/io.py Lib/plat-mac/ic.py Objects/fileobject.c Objects/object.c Python/pythonrun.c Message-ID: <20071030191452.877EE1E401B@bag.python.org> Author: guido.van.rossum Date: Tue Oct 30 20:14:52 2007 New Revision: 58721 Modified: python/branches/py3k-pep3137/ (props changed) python/branches/py3k-pep3137/Include/fileobject.h python/branches/py3k-pep3137/Lib/BaseHTTPServer.py python/branches/py3k-pep3137/Lib/io.py python/branches/py3k-pep3137/Lib/plat-mac/ic.py python/branches/py3k-pep3137/Objects/fileobject.c python/branches/py3k-pep3137/Objects/object.c python/branches/py3k-pep3137/Python/pythonrun.c Log: Merged revisions 58713-58720 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k ........ r58714 | georg.brandl | 2007-10-30 10:42:20 -0700 (Tue, 30 Oct 2007) | 2 lines Fix typo. ........ r58717 | bill.janssen | 2007-10-30 11:12:39 -0700 (Tue, 30 Oct 2007) | 1 line remove ord() calls on what are now integers rather than characters ........ r58718 | bill.janssen | 2007-10-30 11:13:17 -0700 (Tue, 30 Oct 2007) | 1 line make sure to write bytes instead of strings to underlying socket channel ........ r58719 | guido.van.rossum | 2007-10-30 11:34:07 -0700 (Tue, 30 Oct 2007) | 5 lines Patch 1352 (continued in issue 1329) by Christian Heimes. Before sys.stderr is set to the proper thing, set it to a really simple file-like object that can print tracebacks using direct file descriptor I/O. This is handy for debugging. ........ r58720 | guido.van.rossum | 2007-10-30 11:36:44 -0700 (Tue, 30 Oct 2007) | 2 lines Minor correction to the stdprinter object. ........ Modified: python/branches/py3k-pep3137/Include/fileobject.h ============================================================================== --- python/branches/py3k-pep3137/Include/fileobject.h (original) +++ python/branches/py3k-pep3137/Include/fileobject.h Tue Oct 30 20:14:52 2007 @@ -21,6 +21,13 @@ */ PyAPI_DATA(const char *) Py_FileSystemDefaultEncoding; +/* Internal API + + The std printer acts as a preliminary sys.stderr until the new io + infrastructure is in place. */ +PyAPI_FUNC(PyObject *) PyFile_NewStdPrinter(int); +PyAPI_DATA(PyTypeObject) PyStdPrinter_Type; + #ifdef __cplusplus } #endif Modified: python/branches/py3k-pep3137/Lib/BaseHTTPServer.py ============================================================================== --- python/branches/py3k-pep3137/Lib/BaseHTTPServer.py (original) +++ python/branches/py3k-pep3137/Lib/BaseHTTPServer.py Tue Oct 30 20:14:52 2007 @@ -356,11 +356,11 @@ content = (self.error_message_format % {'code': code, 'message': _quote_html(message), 'explain': explain}) self.send_response(code, message) - self.send_header("Content-Type", "text/html") + self.send_header("Content-Type", "text/html;charset=utf-8") self.send_header('Connection', 'close') self.end_headers() if self.command != 'HEAD' and code >= 200 and code not in (204, 304): - self.wfile.write(content) + self.wfile.write(content.encode('UTF-8', 'replace')) error_message_format = DEFAULT_ERROR_MESSAGE @@ -378,8 +378,8 @@ else: message = '' if self.request_version != 'HTTP/0.9': - self.wfile.write("%s %d %s\r\n" % - (self.protocol_version, code, message)) + self.wfile.write(("%s %d %s\r\n" % + (self.protocol_version, code, message)).encode('ASCII', 'strict')) # print (self.protocol_version, code, message) self.send_header('Server', self.version_string()) self.send_header('Date', self.date_time_string()) @@ -387,7 +387,7 @@ def send_header(self, keyword, value): """Send a MIME header.""" if self.request_version != 'HTTP/0.9': - self.wfile.write("%s: %s\r\n" % (keyword, value)) + self.wfile.write(("%s: %s\r\n" % (keyword, value)).encode('ASCII', 'strict')) if keyword.lower() == 'connection': if value.lower() == 'close': @@ -398,7 +398,7 @@ def end_headers(self): """Send the blank line ending the MIME headers.""" if self.request_version != 'HTTP/0.9': - self.wfile.write("\r\n") + self.wfile.write(b"\r\n") def log_request(self, code='-', size='-'): """Log an accepted request. Modified: python/branches/py3k-pep3137/Lib/io.py ============================================================================== --- python/branches/py3k-pep3137/Lib/io.py (original) +++ python/branches/py3k-pep3137/Lib/io.py Tue Oct 30 20:14:52 2007 @@ -87,7 +87,7 @@ a filename is given. (*) If a file descriptor is given, it is closed when the returned - I/O object is closed, unless closefd=False is give. + I/O object is closed, unless closefd=False is given. Mode strings characters: 'r': open for reading (default) Modified: python/branches/py3k-pep3137/Lib/plat-mac/ic.py ============================================================================== --- python/branches/py3k-pep3137/Lib/plat-mac/ic.py (original) +++ python/branches/py3k-pep3137/Lib/plat-mac/ic.py Tue Oct 30 20:14:52 2007 @@ -41,27 +41,27 @@ def _decode_default(data, key): if len(data) == 0: return data - if ord(data[0]) == len(data)-1: + if data[0] == len(data)-1: # Assume Pstring return data[1:] return ICOpaqueData(data) def _decode_multistr(data, key): - numstr = ord(data[0]) << 8 | ord(data[1]) + numstr = data[0] << 8 | data[1] rv = [] ptr = 2 for i in range(numstr): - strlen = ord(data[ptr]) + strlen = data[ptr] str = data[ptr+1:ptr+strlen+1] rv.append(str) ptr = ptr + strlen + 1 return rv def _decode_fontrecord(data, key): - size = ord(data[0]) << 8 | ord(data[1]) - face = ord(data[2]) - namelen = ord(data[4]) + size = data[0] << 8 | data[1] + face = data[2] + namelen = data[4] return size, face, data[5:5+namelen] def _decode_boolean(data, key): @@ -74,7 +74,7 @@ return data[:256], data[256:] def _decode_appspec(data, key): - namelen = ord(data[4]) + namelen = data[4] return data[0:4], data[5:5+namelen] def _code_default(data, key): Modified: python/branches/py3k-pep3137/Objects/fileobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/fileobject.c (original) +++ python/branches/py3k-pep3137/Objects/fileobject.c Tue Oct 30 20:14:52 2007 @@ -325,6 +325,126 @@ return buf; } +/* **************************** std printer **************************** */ + +typedef struct { + PyObject_HEAD + int fd; +} PyStdPrinter_Object; + +static PyObject * +stdprinter_new(PyTypeObject *type, PyObject *args, PyObject *kews) +{ + PyStdPrinter_Object *self; + + assert(type != NULL && type->tp_alloc != NULL); + + self = (PyStdPrinter_Object *) type->tp_alloc(type, 0); + if (self != NULL) { + self->fd = -1; + } + + return (PyObject *) self; +} + +PyObject * +PyFile_NewStdPrinter(int fd) +{ + PyStdPrinter_Object *self; + + if (fd != fileno(stdout) && fd != fileno(stderr)) { + PyErr_BadInternalCall(); + return NULL; + } + + self = PyObject_New(PyStdPrinter_Object, + &PyStdPrinter_Type); + if (self != NULL) { + self->fd = fd; + } + return (PyObject*)self; +} + +PyObject * +stdprinter_write(PyStdPrinter_Object *self, PyObject *args) +{ + char *c; + Py_ssize_t n; + + if (self->fd < 0) { + PyErr_SetString(PyExc_ValueError, + "I/O operation on closed file"); + return NULL; + } + + if (!PyArg_ParseTuple(args, "s#", &c, &n)) { + return NULL; + } + + Py_BEGIN_ALLOW_THREADS + errno = 0; + n = write(self->fd, c, n); + Py_END_ALLOW_THREADS + + if (n < 0) { + if (errno == EAGAIN) + Py_RETURN_NONE; + PyErr_SetFromErrno(PyExc_IOError); + return NULL; + } + + return PyInt_FromSsize_t(n); +} + +static PyMethodDef stdprinter_methods[] = { + {"write", (PyCFunction)stdprinter_write, METH_VARARGS, ""}, + {NULL, NULL} /* sentinel */ +}; + +PyTypeObject PyStdPrinter_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "stderrprinter", /* tp_name */ + sizeof(PyStdPrinter_Object), /* tp_basicsize */ + 0, /* tp_itemsize */ + /* methods */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + 0, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + stdprinter_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + 0, /* tp_init */ + PyType_GenericAlloc, /* tp_alloc */ + stdprinter_new, /* tp_new */ + PyObject_Del, /* tp_free */ +}; + + #ifdef __cplusplus } #endif Modified: python/branches/py3k-pep3137/Objects/object.c ============================================================================== --- python/branches/py3k-pep3137/Objects/object.c (original) +++ python/branches/py3k-pep3137/Objects/object.c Tue Oct 30 20:14:52 2007 @@ -1595,6 +1595,9 @@ if (PyType_Ready(&PyCode_Type) < 0) Py_FatalError("Can't initialize 'code'"); + + if (PyType_Ready(&PyStdPrinter_Type) < 0) + Py_FatalError("Can't initialize StdPrinter"); } Modified: python/branches/py3k-pep3137/Python/pythonrun.c ============================================================================== --- python/branches/py3k-pep3137/Python/pythonrun.c (original) +++ python/branches/py3k-pep3137/Python/pythonrun.c Tue Oct 30 20:14:52 2007 @@ -151,7 +151,7 @@ { PyInterpreterState *interp; PyThreadState *tstate; - PyObject *bimod, *sysmod; + PyObject *bimod, *sysmod, *pstderr; char *p; #if defined(HAVE_LANGINFO_H) && defined(CODESET) char *codeset; @@ -228,6 +228,13 @@ PyDict_SetItemString(interp->sysdict, "modules", interp->modules); + /* Set up a preliminary stderr printer until we have enough + infrastructure for the io module in place. */ + pstderr = PyFile_NewStdPrinter(fileno(stderr)); + if (pstderr == NULL) + Py_FatalError("Py_Initialize: can't set preliminary stderr"); + PySys_SetObject("stderr", pstderr); + _PyImport_Init(); /* initialize builtin exceptions */ From python-3000-checkins at python.org Wed Oct 31 17:41:08 2007 From: python-3000-checkins at python.org (guido.van.rossum) Date: Wed, 31 Oct 2007 17:41:08 +0100 (CET) Subject: [Python-3000-checkins] r58730 - in python/branches/py3k-pep3137: Lib/test/test_bytes.py Objects/stringobject.c Message-ID: <20071031164108.683A11E400A@bag.python.org> Author: guido.van.rossum Date: Wed Oct 31 17:41:08 2007 New Revision: 58730 Modified: python/branches/py3k-pep3137/Lib/test/test_bytes.py python/branches/py3k-pep3137/Objects/stringobject.c Log: Patch 1365 by Christian Heimes. Implement str8(N) -> s'\0'*N, and a unit test for bytes(N) behavior. Modified: python/branches/py3k-pep3137/Lib/test/test_bytes.py ============================================================================== --- python/branches/py3k-pep3137/Lib/test/test_bytes.py (original) +++ python/branches/py3k-pep3137/Lib/test/test_bytes.py Wed Oct 31 17:41:08 2007 @@ -50,6 +50,15 @@ self.assertRaises(ValueError, bytes, [C(-1)]) self.assertRaises(ValueError, bytes, [C(256)]) + def test_from_ssize(self): + self.assertEqual(bytes(0), b'') + self.assertEqual(bytes(1), b'\x00') + self.assertEqual(bytes(5), b'\x00\x00\x00\x00\x00') + self.assertRaises(ValueError, bytes, -1) + + self.assertEqual(bytes('0', 'ascii'), b'0') + self.assertEqual(bytes(b'0'), b'0') + def test_constructor_type_errors(self): self.assertRaises(TypeError, bytes, 0.0) class C: Modified: python/branches/py3k-pep3137/Objects/stringobject.c ============================================================================== --- python/branches/py3k-pep3137/Objects/stringobject.c (original) +++ python/branches/py3k-pep3137/Objects/stringobject.c Wed Oct 31 17:41:08 2007 @@ -2909,6 +2909,26 @@ return NULL; } + /* Is it an int? */ + size = PyNumber_AsSsize_t(x, PyExc_ValueError); + if (size == -1 && PyErr_Occurred()) { + PyErr_Clear(); + } + else { + if (size < 0) { + PyErr_SetString(PyExc_ValueError, "negative count"); + return NULL; + } + new = PyString_FromStringAndSize(NULL, size); + if (new == NULL) { + return NULL; + } + if (size > 0) { + memset(((PyStringObject*)new)->ob_sval, 0, size); + } + return new; + } + /* Use the modern buffer interface */ if (PyObject_CheckBuffer(x)) { Py_buffer view; From python-3000-checkins at python.org Wed Oct 31 19:53:45 2007 From: python-3000-checkins at python.org (christian.heimes) Date: Wed, 31 Oct 2007 19:53:45 +0100 (CET) Subject: [Python-3000-checkins] r58732 - in python/branches/py3k/Lib: gettext.py test/test_gettext.py Message-ID: <20071031185345.071B41E400A@bag.python.org> Author: christian.heimes Date: Wed Oct 31 19:53:44 2007 New Revision: 58732 Modified: python/branches/py3k/Lib/gettext.py python/branches/py3k/Lib/test/test_gettext.py Log: Fixed gettext module for Windows. The metadata lines always end in \n and not in os.linesep Modified: python/branches/py3k/Lib/gettext.py ============================================================================== --- python/branches/py3k/Lib/gettext.py (original) +++ python/branches/py3k/Lib/gettext.py Wed Oct 31 19:53:44 2007 @@ -291,7 +291,7 @@ if mlen == 0: # Catalog description lastk = k = None - for b_item in tmsg.split(os.linesep.encode("ascii")): + for b_item in tmsg.split('\n'.encode("ascii")): item = str(b_item).strip() if not item: continue Modified: python/branches/py3k/Lib/test/test_gettext.py ============================================================================== --- python/branches/py3k/Lib/test/test_gettext.py (original) +++ python/branches/py3k/Lib/test/test_gettext.py Wed Oct 31 19:53:44 2007 @@ -332,6 +332,7 @@ def test_weird_metadata(self): info = self.t.info() + self.assertEqual(len(info), 9) self.assertEqual(info['last-translator'], 'John Doe \nJane Foobar ') From python-3000-checkins at python.org Wed Oct 31 20:20:49 2007 From: python-3000-checkins at python.org (christian.heimes) Date: Wed, 31 Oct 2007 20:20:49 +0100 (CET) Subject: [Python-3000-checkins] r58733 - python/branches/py3k/Modules/_fileio.c Message-ID: <20071031192049.1839C1E400C@bag.python.org> Author: christian.heimes Date: Wed Oct 31 20:20:48 2007 New Revision: 58733 Modified: python/branches/py3k/Modules/_fileio.c Log: Fixed bug in _fileio.c and test_pep277. On Windows IOError.filename was not set because the name is stored in widename. Modified: python/branches/py3k/Modules/_fileio.c ============================================================================== --- python/branches/py3k/Modules/_fileio.c (original) +++ python/branches/py3k/Modules/_fileio.c Wed Oct 31 20:20:48 2007 @@ -267,7 +267,11 @@ self->fd = open(name, flags, 0666); Py_END_ALLOW_THREADS if (self->fd < 0 || dircheck(self) < 0) { +#ifdef MS_WINDOWS + PyErr_SetFromErrnoWithUnicodeFilename(PyExc_IOError, widename); +#else PyErr_SetFromErrnoWithFilename(PyExc_IOError, name); +#endif goto error; } } From python-3000-checkins at python.org Wed Oct 31 20:40:17 2007 From: python-3000-checkins at python.org (christian.heimes) Date: Wed, 31 Oct 2007 20:40:17 +0100 (CET) Subject: [Python-3000-checkins] r58734 - python/branches/py3k/Lib/test/test_dumbdbm.py Message-ID: <20071031194017.7C7B61E400A@bag.python.org> Author: christian.heimes Date: Wed Oct 31 20:40:17 2007 New Revision: 58734 Modified: python/branches/py3k/Lib/test/test_dumbdbm.py Log: Fixed test_dumbdbm The test failed on Windows. I hope the change follows the spirit of the test. On Unix it checks if dumbdbm can handle Windows line separators and on Windows it tests with Unix line separators. Modified: python/branches/py3k/Lib/test/test_dumbdbm.py ============================================================================== --- python/branches/py3k/Lib/test/test_dumbdbm.py (original) +++ python/branches/py3k/Lib/test/test_dumbdbm.py Wed Oct 31 20:40:17 2007 @@ -113,9 +113,12 @@ f[b'2'] = b'hello2' f.close() - # Mangle the file by adding \r before each newline + # Mangle the file by changing the line separator to Windows or Unix data = io.open(_fname + '.dir', 'rb').read() - data = data.replace(b'\n', b'\r\n') + if os.linesep == b'\n': + data = data.replace(b'\n', b'\r\n') + else: + data = data.replace(b'\r\n', b'\n') io.open(_fname + '.dir', 'wb').write(data) f = dumbdbm.open(_fname) From python-3000-checkins at python.org Wed Oct 31 20:43:23 2007 From: python-3000-checkins at python.org (christian.heimes) Date: Wed, 31 Oct 2007 20:43:23 +0100 (CET) Subject: [Python-3000-checkins] r58735 - in python/branches/py3k-pep3137: Lib/gettext.py Lib/test/test_dumbdbm.py Lib/test/test_gettext.py Modules/_fileio.c Message-ID: <20071031194323.3C3A21E4029@bag.python.org> Author: christian.heimes Date: Wed Oct 31 20:43:22 2007 New Revision: 58735 Modified: python/branches/py3k-pep3137/ (props changed) python/branches/py3k-pep3137/Lib/gettext.py python/branches/py3k-pep3137/Lib/test/test_dumbdbm.py python/branches/py3k-pep3137/Lib/test/test_gettext.py python/branches/py3k-pep3137/Modules/_fileio.c Log: Merged revisions 58721-58734 via svnmerge from svn+ssh://pythondev at svn.python.org/python/branches/py3k ........ r58732 | christian.heimes | 2007-10-31 19:53:44 +0100 (Wed, 31 Oct 2007) | 1 line Fixed gettext module for Windows. The metadata lines always end in \n and not in os.linesep ........ r58733 | christian.heimes | 2007-10-31 20:20:48 +0100 (Wed, 31 Oct 2007) | 1 line Fixed bug in _fileio.c and test_pep277. On Windows IOError.filename was not set because the name is stored in widename. ........ r58734 | christian.heimes | 2007-10-31 20:40:17 +0100 (Wed, 31 Oct 2007) | 2 lines Fixed test_dumbdbm The test failed on Windows. I hope the change follows the spirit of the test. On Unix it checks if dumbdbm can handle Windows line separators and on Windows it tests with Unix line separators. ........ Modified: python/branches/py3k-pep3137/Lib/gettext.py ============================================================================== --- python/branches/py3k-pep3137/Lib/gettext.py (original) +++ python/branches/py3k-pep3137/Lib/gettext.py Wed Oct 31 20:43:22 2007 @@ -291,7 +291,7 @@ if mlen == 0: # Catalog description lastk = k = None - for b_item in tmsg.split(os.linesep.encode("ascii")): + for b_item in tmsg.split('\n'.encode("ascii")): item = str(b_item).strip() if not item: continue Modified: python/branches/py3k-pep3137/Lib/test/test_dumbdbm.py ============================================================================== --- python/branches/py3k-pep3137/Lib/test/test_dumbdbm.py (original) +++ python/branches/py3k-pep3137/Lib/test/test_dumbdbm.py Wed Oct 31 20:43:22 2007 @@ -113,9 +113,12 @@ f[b'2'] = b'hello2' f.close() - # Mangle the file by adding \r before each newline + # Mangle the file by changing the line separator to Windows or Unix data = io.open(_fname + '.dir', 'rb').read() - data = data.replace(b'\n', b'\r\n') + if os.linesep == b'\n': + data = data.replace(b'\n', b'\r\n') + else: + data = data.replace(b'\r\n', b'\n') io.open(_fname + '.dir', 'wb').write(data) f = dumbdbm.open(_fname) Modified: python/branches/py3k-pep3137/Lib/test/test_gettext.py ============================================================================== --- python/branches/py3k-pep3137/Lib/test/test_gettext.py (original) +++ python/branches/py3k-pep3137/Lib/test/test_gettext.py Wed Oct 31 20:43:22 2007 @@ -332,6 +332,7 @@ def test_weird_metadata(self): info = self.t.info() + self.assertEqual(len(info), 9) self.assertEqual(info['last-translator'], 'John Doe \nJane Foobar ') Modified: python/branches/py3k-pep3137/Modules/_fileio.c ============================================================================== --- python/branches/py3k-pep3137/Modules/_fileio.c (original) +++ python/branches/py3k-pep3137/Modules/_fileio.c Wed Oct 31 20:43:22 2007 @@ -267,7 +267,11 @@ self->fd = open(name, flags, 0666); Py_END_ALLOW_THREADS if (self->fd < 0 || dircheck(self) < 0) { +#ifdef MS_WINDOWS + PyErr_SetFromErrnoWithUnicodeFilename(PyExc_IOError, widename); +#else PyErr_SetFromErrnoWithFilename(PyExc_IOError, name); +#endif goto error; } } From python-3000-checkins at python.org Wed Oct 31 20:51:01 2007 From: python-3000-checkins at python.org (christian.heimes) Date: Wed, 31 Oct 2007 20:51:01 +0100 (CET) Subject: [Python-3000-checkins] r58736 - python/branches/py3k/Lib/test/output/test_pep277 Message-ID: <20071031195101.CB9041E400A@bag.python.org> Author: christian.heimes Date: Wed Oct 31 20:51:01 2007 New Revision: 58736 Modified: python/branches/py3k/Lib/test/output/test_pep277 Log: Oh, I missed the output part of the test ... Modified: python/branches/py3k/Lib/test/output/test_pep277 ============================================================================== --- python/branches/py3k/Lib/test/output/test_pep277 (original) +++ python/branches/py3k/Lib/test/output/test_pep277 Wed Oct 31 20:51:01 2007 @@ -1,3 +1,3 @@ test_pep277 -u'\xdf-\u66e8\u66e9\u66eb' -[u'Gr\xfc\xdf-Gott', u'abc', u'ascii', u'\u0393\u03b5\u03b9\u03ac-\u03c3\u03b1\u03c2', u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439\u0442\u0435', u'\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1', u'\u306b\u307d\u3093', u'\u66e8\u05e9\u3093\u0434\u0393\xdf', u'\u66e8\u66e9\u66eb'] +'\xdf-\u66e8\u66e9\u66eb' +['Gr\xfc\xdf-Gott', 'abc', 'ascii', '\u0393\u03b5\u03b9\u03ac-\u03c3\u03b1\u03c2', '\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439\u0442\u0435', '\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1', '\u306b\u307d\u3093', '\u66e8\u05e9\u3093\u0434\u0393\xdf', '\u66e8\u66e9\u66eb']