[Python-checkins] r64074 - in python/branches/tlee-ast-optimize: Doc/ACKS.txt Doc/library/_ast.rst Doc/library/ast.rst Doc/library/asynchat.rst Doc/library/asyncore.rst Doc/library/language.rst Lib/ast.py Lib/asynchat.py Lib/asyncore.py Lib/test/test_ast.py Lib/test/test_asyncore.py Lib/test/test_sys.py Lib/test/test_xrange.py Misc/ACKS Misc/NEWS Misc/developers.txt Modules/_ctypes/callproc.c Modules/itertoolsmodule.c Objects/rangeobject.c Objects/unicodeobject.c
thomas.lee
python-checkins at python.org
Tue Jun 10 17:03:16 CEST 2008
Author: thomas.lee
Date: Tue Jun 10 17:03:15 2008
New Revision: 64074
Log:
Merged revisions 64057-64072 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64057 | alexandre.vassalotti | 2008-06-10 13:34:53 +1000 (Tue, 10 Jun 2008) | 2 lines
Issue 2582: Fix pickling of xrange objects.
........
r64058 | alexandre.vassalotti | 2008-06-10 14:01:23 +1000 (Tue, 10 Jun 2008) | 3 lines
Added better pickling support to xrange objects.
Cleaned up the unit test.
........
r64062 | josiah.carlson | 2008-06-10 15:00:08 +1000 (Tue, 10 Jun 2008) | 5 lines
Applying updated patch from Issue 1736190, which addresses partial
issues in: 909005 and 17361001, as well as completely as possible issues
539444, 760475, 777588, 889153, 953599, 1025525, 1063924, and 658749.
This patch also includes doc and test updates as necessary.
........
r64063 | martin.v.loewis | 2008-06-10 15:03:35 +1000 (Tue, 10 Jun 2008) | 2 lines
Add Gregor Lingl.
........
r64064 | georg.brandl | 2008-06-10 17:45:28 +1000 (Tue, 10 Jun 2008) | 2 lines
Add the "ast" module, containing helpers to ease use of the "_ast" classes.
........
r64065 | raymond.hettinger | 2008-06-10 17:57:15 +1000 (Tue, 10 Jun 2008) | 1 line
Add Arnaud for his efforts on multi-arg set operations.
........
r64066 | robert.schuppenies | 2008-06-10 20:10:31 +1000 (Tue, 10 Jun 2008) | 2 lines
Issue 3048: Fixed sys.getsizeof for unicode objects.
........
r64067 | georg.brandl | 2008-06-10 22:46:39 +1000 (Tue, 10 Jun 2008) | 2 lines
#2536: fix itertools.permutations and itertools.combinations docstrings.
........
r64068 | benjamin.peterson | 2008-06-10 23:37:13 +1000 (Tue, 10 Jun 2008) | 2 lines
fix markup
........
r64069 | georg.brandl | 2008-06-10 23:53:24 +1000 (Tue, 10 Jun 2008) | 2 lines
more markup fix.
........
r64070 | thomas.heller | 2008-06-11 00:02:46 +1000 (Wed, 11 Jun 2008) | 2 lines
Add an optional 'offset' parameter to byref, defaultingto zero.
........
r64071 | thomas.heller | 2008-06-11 00:07:12 +1000 (Wed, 11 Jun 2008) | 3 lines
NEWS entry for:
Add an optional 'offset' parameter to byref, defaulting to zero.
........
Added:
python/branches/tlee-ast-optimize/Doc/library/ast.rst
- copied unchanged from r64071, /python/trunk/Doc/library/ast.rst
python/branches/tlee-ast-optimize/Lib/ast.py
- copied unchanged from r64071, /python/trunk/Lib/ast.py
Removed:
python/branches/tlee-ast-optimize/Doc/library/_ast.rst
Modified:
python/branches/tlee-ast-optimize/ (props changed)
python/branches/tlee-ast-optimize/Doc/ACKS.txt
python/branches/tlee-ast-optimize/Doc/library/asynchat.rst
python/branches/tlee-ast-optimize/Doc/library/asyncore.rst
python/branches/tlee-ast-optimize/Doc/library/language.rst
python/branches/tlee-ast-optimize/Lib/asynchat.py
python/branches/tlee-ast-optimize/Lib/asyncore.py
python/branches/tlee-ast-optimize/Lib/test/test_ast.py
python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py
python/branches/tlee-ast-optimize/Lib/test/test_sys.py
python/branches/tlee-ast-optimize/Lib/test/test_xrange.py
python/branches/tlee-ast-optimize/Misc/ACKS
python/branches/tlee-ast-optimize/Misc/NEWS
python/branches/tlee-ast-optimize/Misc/developers.txt
python/branches/tlee-ast-optimize/Modules/_ctypes/callproc.c
python/branches/tlee-ast-optimize/Modules/itertoolsmodule.c
python/branches/tlee-ast-optimize/Objects/rangeobject.c
python/branches/tlee-ast-optimize/Objects/unicodeobject.c
Modified: python/branches/tlee-ast-optimize/Doc/ACKS.txt
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/ACKS.txt (original)
+++ python/branches/tlee-ast-optimize/Doc/ACKS.txt Tue Jun 10 17:03:15 2008
@@ -157,6 +157,7 @@
* Bernhard Reiter
* Armin Rigo
* Wes Rishel
+ * Armin Ronacher
* Jim Roskind
* Guido van Rossum
* Donald Wallace Rouse II
Deleted: python/branches/tlee-ast-optimize/Doc/library/_ast.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/_ast.rst Tue Jun 10 17:03:15 2008
+++ (empty file)
@@ -1,85 +0,0 @@
-.. _ast:
-
-Abstract Syntax Trees
-=====================
-
-.. module:: _ast
- :synopsis: Abstract Syntax Tree classes.
-
-.. sectionauthor:: Martin v. Löwis <martin at v.loewis.de>
-
-
-.. versionadded:: 2.5
-
-The ``_ast`` module helps Python applications to process trees of the Python
-abstract syntax grammar. The abstract syntax itself might change with each
-Python release; this module helps to find out programmatically what the current
-grammar looks like.
-
-An abstract syntax tree can be generated by passing :data:`_ast.PyCF_ONLY_AST`
-as a flag to the :func:`compile` builtin function. The result will be a tree of
-objects whose classes all inherit from :class:`_ast.AST`.
-
-A modified abstract syntax tree can be compiled into a Python code object using
-the built-in :func:`compile` function.
-
-The actual classes are derived from the ``Parser/Python.asdl`` file, which is
-reproduced below. There is one class defined for each left-hand side symbol in
-the abstract grammar (for example, ``_ast.stmt`` or ``_ast.expr``). In addition,
-there is one class defined for each constructor on the right-hand side; these
-classes inherit from the classes for the left-hand side trees. For example,
-``_ast.BinOp`` inherits from ``_ast.expr``. For production rules with
-alternatives (aka "sums"), the left-hand side class is abstract: only instances
-of specific constructor nodes are ever created.
-
-Each concrete class has an attribute ``_fields`` which gives the names of all
-child nodes.
-
-Each instance of a concrete class has one attribute for each child node, of the
-type as defined in the grammar. For example, ``_ast.BinOp`` instances have an
-attribute ``left`` of type ``_ast.expr``. Instances of ``_ast.expr`` and
-``_ast.stmt`` subclasses also have lineno and col_offset attributes. The lineno
-is the line number of source text (1 indexed so the first line is line 1) and
-the col_offset is the utf8 byte offset of the first token that generated the
-node. The utf8 offset is recorded because the parser uses utf8 internally.
-
-If these attributes are marked as optional in the grammar (using a question
-mark), the value might be ``None``. If the attributes can have zero-or-more
-values (marked with an asterisk), the values are represented as Python lists.
-All possible attributes must be present and have valid values when compiling an
-AST with :func:`compile`.
-
-The constructor of a class ``_ast.T`` parses their arguments as follows:
-
-* If there are positional arguments, there must be as many as there are items in
- ``T._fields``; they will be assigned as attributes of these names.
-* If there are keyword arguments, they will set the attributes of the same names
- to the given values.
-
-For example, to create and populate a ``UnaryOp`` node, you could use ::
-
- node = _ast.UnaryOp()
- node.op = _ast.USub()
- node.operand = _ast.Num()
- node.operand.n = 5
- node.operand.lineno = 0
- node.operand.col_offset = 0
- node.lineno = 0
- node.col_offset = 0
-
-or the more compact ::
-
- node = _ast.UnaryOp(_ast.USub(), _ast.Num(5, lineno=0, col_offset=0),
- lineno=0, col_offset=0)
-
-
-
-Abstract Grammar
-----------------
-
-The module defines a string constant ``__version__`` which is the decimal
-Subversion revision number of the file shown below.
-
-The abstract grammar is currently defined as follows:
-
-.. literalinclude:: ../../Parser/Python.asdl
Modified: python/branches/tlee-ast-optimize/Doc/library/asynchat.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/asynchat.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/asynchat.rst Tue Jun 10 17:03:15 2008
@@ -81,6 +81,12 @@
:exc:`NotImplementedError` exception.
+.. method:: async_chat._collect_incoming_data(data)
+
+ Sample implementation of a data collection rutine to be used in conjunction
+ with :meth:`_get_data` in a user-specified :meth:`found_terminator`.
+
+
.. method:: async_chat.discard_buffers()
In emergencies this method will discard any data held in the input and/or
@@ -95,6 +101,12 @@
should be available via an instance attribute.
+.. method:: async_chat._get_data()
+
+ Will return and clear the data received with the sample
+ :meth:`_collect_incoming_data` implementation.
+
+
.. method:: async_chat.get_terminator()
Returns the current terminator for the channel.
Modified: python/branches/tlee-ast-optimize/Doc/library/asyncore.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/asyncore.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/asyncore.rst Tue Jun 10 17:03:15 2008
@@ -222,6 +222,21 @@
flushed). Sockets are automatically closed when they are
garbage-collected.
+.. class:: file_dispatcher()
+
+ A file_dispatcher takes a file descriptor or file object along with an
+ optional map argument and wraps it for use with the :cfunc:`poll` or
+ :cfunc:`loop` functions. If provided a file object or anything with a
+ :cfunc:`fileno` method, that method will be called and passed to the
+ :class:`file_wrapper` constructor. Availability: UNIX.
+
+.. class:: file_wrapper()
+
+ A file_wrapper takes an integer file descriptor and calls :func:`os.dup` to
+ duplicate the handle so that the original handle may be closed independently
+ of the file_wrapper. This class implements sufficient methods to emulate a
+ socket for use by the :class:`file_dispatcher` class. Availability: UNIX.
+
.. _asyncore-example:
Modified: python/branches/tlee-ast-optimize/Doc/library/language.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/language.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/language.rst Tue Jun 10 17:03:15 2008
@@ -15,7 +15,7 @@
.. toctree::
parser.rst
- _ast.rst
+ ast.rst
symbol.rst
token.rst
keyword.rst
Modified: python/branches/tlee-ast-optimize/Lib/asynchat.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/asynchat.py (original)
+++ python/branches/tlee-ast-optimize/Lib/asynchat.py Tue Jun 10 17:03:15 2008
@@ -60,16 +60,35 @@
ac_out_buffer_size = 4096
def __init__ (self, conn=None):
+ # for string terminator matching
self.ac_in_buffer = ''
- self.ac_out_buffer = ''
- self.producer_fifo = fifo()
+
+ # we use a list here rather than cStringIO for a few reasons...
+ # del lst[:] is faster than sio.truncate(0)
+ # lst = [] is faster than sio.truncate(0)
+ # cStringIO will be gaining unicode support in py3k, which
+ # will negatively affect the performance of bytes compared to
+ # a ''.join() equivalent
+ self.incoming = []
+
+ # we toss the use of the "simple producer" and replace it with
+ # a pure deque, which the original fifo was a wrapping of
+ self.producer_fifo = deque()
asyncore.dispatcher.__init__ (self, conn)
def collect_incoming_data(self, data):
- raise NotImplementedError, "must be implemented in subclass"
+ raise NotImplementedError("must be implemented in subclass")
+
+ def _collect_incoming_data(self, data):
+ self.incoming.append(data)
+
+ def _get_data(self):
+ d = ''.join(self.incoming)
+ del self.incoming[:]
+ return d
def found_terminator(self):
- raise NotImplementedError, "must be implemented in subclass"
+ raise NotImplementedError("must be implemented in subclass")
def set_terminator (self, term):
"Set the input delimiter. Can be a fixed string of any length, an integer, or None"
@@ -96,7 +115,7 @@
# Continue to search for self.terminator in self.ac_in_buffer,
# while calling self.collect_incoming_data. The while loop
# is necessary because we might read several data+terminator
- # combos with a single recv(1024).
+ # combos with a single recv(4096).
while self.ac_in_buffer:
lb = len(self.ac_in_buffer)
@@ -150,87 +169,82 @@
self.ac_in_buffer = ''
def handle_write (self):
- self.initiate_send ()
+ self.initiate_send()
def handle_close (self):
self.close()
def push (self, data):
- self.producer_fifo.push (simple_producer (data))
+ sabs = self.ac_out_buffer_size
+ if len(data) > sabs:
+ for i in xrange(0, len(data), sabs):
+ self.producer_fifo.append(data[i:i+sabs])
+ else:
+ self.producer_fifo.append(data)
self.initiate_send()
def push_with_producer (self, producer):
- self.producer_fifo.push (producer)
+ self.producer_fifo.append(producer)
self.initiate_send()
def readable (self):
"predicate for inclusion in the readable for select()"
- return (len(self.ac_in_buffer) <= self.ac_in_buffer_size)
+ # cannot use the old predicate, it violates the claim of the
+ # set_terminator method.
+
+ # return (len(self.ac_in_buffer) <= self.ac_in_buffer_size)
+ return 1
def writable (self):
"predicate for inclusion in the writable for select()"
- # return len(self.ac_out_buffer) or len(self.producer_fifo) or (not self.connected)
- # this is about twice as fast, though not as clear.
- return not (
- (self.ac_out_buffer == '') and
- self.producer_fifo.is_empty() and
- self.connected
- )
+ return self.producer_fifo or (not self.connected)
def close_when_done (self):
"automatically close this channel once the outgoing queue is empty"
- self.producer_fifo.push (None)
+ self.producer_fifo.append(None)
- # refill the outgoing buffer by calling the more() method
- # of the first producer in the queue
- def refill_buffer (self):
- while 1:
- if len(self.producer_fifo):
- p = self.producer_fifo.first()
- # a 'None' in the producer fifo is a sentinel,
- # telling us to close the channel.
- if p is None:
- if not self.ac_out_buffer:
- self.producer_fifo.pop()
- self.close()
- return
- elif isinstance(p, str):
- self.producer_fifo.pop()
- self.ac_out_buffer = self.ac_out_buffer + p
+ def initiate_send(self):
+ while self.producer_fifo and self.connected:
+ first = self.producer_fifo[0]
+ # handle empty string/buffer or None entry
+ if not first:
+ del self.producer_fifo[0]
+ if first is None:
+ self.handle_close()
return
- data = p.more()
+
+ # handle classic producer behavior
+ obs = self.ac_out_buffer_size
+ try:
+ data = buffer(first, 0, obs)
+ except TypeError:
+ data = first.more()
if data:
- self.ac_out_buffer = self.ac_out_buffer + data
- return
+ self.producer_fifo.appendleft(data)
else:
- self.producer_fifo.pop()
- else:
- return
+ del self.producer_fifo[0]
+ continue
- def initiate_send (self):
- obs = self.ac_out_buffer_size
- # try to refill the buffer
- if (len (self.ac_out_buffer) < obs):
- self.refill_buffer()
-
- if self.ac_out_buffer and self.connected:
- # try to send the buffer
+ # send the data
try:
- num_sent = self.send (self.ac_out_buffer[:obs])
- if num_sent:
- self.ac_out_buffer = self.ac_out_buffer[num_sent:]
-
- except socket.error, why:
+ num_sent = self.send(data)
+ except socket.error:
self.handle_error()
return
+ if num_sent:
+ if num_sent < len(data) or obs < len(first):
+ self.producer_fifo[0] = first[num_sent:]
+ else:
+ del self.producer_fifo[0]
+ # we tried to send some actual data
+ return
+
def discard_buffers (self):
# Emergencies only!
self.ac_in_buffer = ''
- self.ac_out_buffer = ''
- while self.producer_fifo:
- self.producer_fifo.pop()
-
+ del self.incoming[:]
+ self.producer_fifo.clear()
class simple_producer:
Modified: python/branches/tlee-ast-optimize/Lib/asyncore.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/asyncore.py (original)
+++ python/branches/tlee-ast-optimize/Lib/asyncore.py Tue Jun 10 17:03:15 2008
@@ -53,20 +53,26 @@
import os
from errno import EALREADY, EINPROGRESS, EWOULDBLOCK, ECONNRESET, \
- ENOTCONN, ESHUTDOWN, EINTR, EISCONN, errorcode
+ ENOTCONN, ESHUTDOWN, EINTR, EISCONN, EBADF, ECONNABORTED, errorcode
try:
socket_map
except NameError:
socket_map = {}
+def _strerror(err):
+ res = os.strerror(err)
+ if res == 'Unknown error':
+ res = errorcode[err]
+ return res
+
class ExitNow(Exception):
pass
def read(obj):
try:
obj.handle_read_event()
- except ExitNow:
+ except (ExitNow, KeyboardInterrupt, SystemExit):
raise
except:
obj.handle_error()
@@ -74,15 +80,15 @@
def write(obj):
try:
obj.handle_write_event()
- except ExitNow:
+ except (ExitNow, KeyboardInterrupt, SystemExit):
raise
except:
obj.handle_error()
-def _exception (obj):
+def _exception(obj):
try:
obj.handle_expt_event()
- except ExitNow:
+ except (ExitNow, KeyboardInterrupt, SystemExit):
raise
except:
obj.handle_error()
@@ -95,7 +101,7 @@
obj.handle_write_event()
if flags & (select.POLLERR | select.POLLHUP | select.POLLNVAL):
obj.handle_expt_event()
- except ExitNow:
+ except (ExitNow, KeyboardInterrupt, SystemExit):
raise
except:
obj.handle_error()
@@ -116,14 +122,15 @@
e.append(fd)
if [] == r == w == e:
time.sleep(timeout)
- else:
- try:
- r, w, e = select.select(r, w, e, timeout)
- except select.error, err:
- if err[0] != EINTR:
- raise
- else:
- return
+ return
+
+ try:
+ r, w, e = select.select(r, w, e, timeout)
+ except select.error, err:
+ if err[0] != EINTR:
+ raise
+ else:
+ return
for fd in r:
obj = map.get(fd)
@@ -209,18 +216,29 @@
else:
self._map = map
+ self._fileno = None
+
if sock:
+ # Set to nonblocking just to make sure for cases where we
+ # get a socket from a blocking source.
+ sock.setblocking(0)
self.set_socket(sock, map)
- # I think it should inherit this anyway
- self.socket.setblocking(0)
self.connected = True
- # XXX Does the constructor require that the socket passed
- # be connected?
+ # The constructor no longer requires that the socket
+ # passed be connected.
try:
self.addr = sock.getpeername()
except socket.error:
- # The addr isn't crucial
- pass
+ if err[0] == ENOTCONN:
+ # To handle the case where we got an unconnected
+ # socket.
+ self.connected = False
+ else:
+ # The socket is broken in some unknown way, alert
+ # the user and remove it from the map (to prevent
+ # polling of broken sockets).
+ self.del_channel(map)
+ raise
else:
self.socket = None
@@ -254,10 +272,9 @@
def create_socket(self, family, type):
self.family_and_type = family, type
- self.socket = socket.socket(family, type)
- self.socket.setblocking(0)
- self._fileno = self.socket.fileno()
- self.add_channel()
+ sock = socket.socket(family, type)
+ sock.setblocking(0)
+ self.set_socket(sock)
def set_socket(self, sock, map=None):
self.socket = sock
@@ -295,7 +312,7 @@
def listen(self, num):
self.accepting = True
if os.name == 'nt' and num > 5:
- num = 1
+ num = 5
return self.socket.listen(num)
def bind(self, addr):
@@ -310,10 +327,9 @@
return
if err in (0, EISCONN):
self.addr = address
- self.connected = True
- self.handle_connect()
+ self.handle_connect_event()
else:
- raise socket.error, (err, errorcode[err])
+ raise socket.error(err, errorcode[err])
def accept(self):
# XXX can return either an address pair or None
@@ -333,9 +349,11 @@
except socket.error, why:
if why[0] == EWOULDBLOCK:
return 0
+ elif why[0] in (ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED):
+ self.handle_close()
+ return 0
else:
raise
- return 0
def recv(self, buffer_size):
try:
@@ -349,15 +367,21 @@
return data
except socket.error, why:
# winsock sometimes throws ENOTCONN
- if why[0] in [ECONNRESET, ENOTCONN, ESHUTDOWN]:
+ if why[0] in [ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED]:
self.handle_close()
return ''
else:
raise
def close(self):
+ self.connected = False
+ self.accepting = False
self.del_channel()
- self.socket.close()
+ try:
+ self.socket.close()
+ except socket.error, why:
+ if why[0] not in (ENOTCONN, EBADF):
+ raise
# cheap inheritance, used to pass all other attribute
# references to the underlying socket object.
@@ -377,27 +401,53 @@
def handle_read_event(self):
if self.accepting:
- # for an accepting socket, getting a read implies
- # that we are connected
- if not self.connected:
- self.connected = True
+ # accepting sockets are never connected, they "spawn" new
+ # sockets that are connected
self.handle_accept()
elif not self.connected:
- self.handle_connect()
- self.connected = True
+ self.handle_connect_event()
self.handle_read()
else:
self.handle_read()
+ def handle_connect_event(self):
+ self.connected = True
+ self.handle_connect()
+
def handle_write_event(self):
- # getting a write implies that we are connected
+ if self.accepting:
+ # Accepting sockets shouldn't get a write event.
+ # We will pretend it didn't happen.
+ return
+
if not self.connected:
- self.handle_connect()
- self.connected = True
+ #check for errors
+ err = self.socket.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
+ if err != 0:
+ raise socket.error(err, strerror(err))
+
+ self.handle_connect_event()
self.handle_write()
def handle_expt_event(self):
- self.handle_expt()
+ # if the handle_expt is the same default worthless method,
+ # we'll not even bother calling it, we'll instead generate
+ # a useful error
+ x = True
+ try:
+ y1 = self.__class__.handle_expt.im_func
+ y2 = dispatcher.handle_expt.im_func
+ x = y1 is y2
+ except AttributeError:
+ pass
+
+ if x:
+ err = self.socket.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
+ msg = _strerror(err)
+
+ raise socket.error(err, msg)
+ else:
+ self.handle_expt()
def handle_error(self):
nil, t, v, tbinfo = compact_traceback()
@@ -473,7 +523,8 @@
def compact_traceback():
t, v, tb = sys.exc_info()
tbinfo = []
- assert tb # Must have a traceback
+ if not tb: # Must have a traceback
+ raise AssertionError("traceback does not exist")
while tb:
tbinfo.append((
tb.tb_frame.f_code.co_filename,
@@ -489,11 +540,22 @@
info = ' '.join(['[%s|%s|%s]' % x for x in tbinfo])
return (file, function, line), t, v, info
-def close_all(map=None):
+def close_all(map=None, ignore_all=False):
if map is None:
map = socket_map
for x in map.values():
- x.socket.close()
+ try:
+ x.close()
+ except OSError, x:
+ if x[0] == EBADF:
+ pass
+ elif not ignore_all:
+ raise
+ except (ExitNow, KeyboardInterrupt, SystemExit):
+ raise
+ except:
+ if not ignore_all:
+ raise
map.clear()
# Asynchronous File I/O:
@@ -513,11 +575,12 @@
import fcntl
class file_wrapper:
- # here we override just enough to make a file
+ # Here we override just enough to make a file
# look like a socket for the purposes of asyncore.
+ # The passed fd is automatically os.dup()'d
def __init__(self, fd):
- self.fd = fd
+ self.fd = os.dup(fd)
def recv(self, *args):
return os.read(self.fd, *args)
@@ -539,6 +602,10 @@
def __init__(self, fd, map=None):
dispatcher.__init__(self, None, map)
self.connected = True
+ try:
+ fd = fd.fileno()
+ except AttributeError:
+ pass
self.set_file(fd)
# set it to non-blocking mode
flags = fcntl.fcntl(fd, fcntl.F_GETFL, 0)
Modified: python/branches/tlee-ast-optimize/Lib/test/test_ast.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_ast.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_ast.py Tue Jun 10 17:03:15 2008
@@ -1,6 +1,6 @@
import sys, itertools, unittest
from test import test_support
-import _ast
+import ast
def to_tuple(t):
if t is None or isinstance(t, (basestring, int, long, complex)):
@@ -123,9 +123,9 @@
class AST_Tests(unittest.TestCase):
def _assert_order(self, ast_node, parent_pos):
- if not isinstance(ast_node, _ast.AST) or ast_node._fields is None:
+ if not isinstance(ast_node, ast.AST) or ast_node._fields is None:
return
- if isinstance(ast_node, (_ast.expr, _ast.stmt, _ast.excepthandler)):
+ if isinstance(ast_node, (ast.expr, ast.stmt, ast.excepthandler)):
node_pos = (ast_node.lineno, ast_node.col_offset)
self.assert_(node_pos >= parent_pos)
parent_pos = (ast_node.lineno, ast_node.col_offset)
@@ -148,24 +148,24 @@
self._assert_order(ast_tree, (0, 0))
def test_nodeclasses(self):
- x = _ast.BinOp(1, 2, 3, lineno=0)
+ x = ast.BinOp(1, 2, 3, lineno=0)
self.assertEquals(x.left, 1)
self.assertEquals(x.op, 2)
self.assertEquals(x.right, 3)
self.assertEquals(x.lineno, 0)
# node raises exception when not given enough arguments
- self.assertRaises(TypeError, _ast.BinOp, 1, 2)
+ self.assertRaises(TypeError, ast.BinOp, 1, 2)
# can set attributes through kwargs too
- x = _ast.BinOp(left=1, op=2, right=3, lineno=0)
+ x = ast.BinOp(left=1, op=2, right=3, lineno=0)
self.assertEquals(x.left, 1)
self.assertEquals(x.op, 2)
self.assertEquals(x.right, 3)
self.assertEquals(x.lineno, 0)
# this used to fail because Sub._fields was None
- x = _ast.Sub()
+ x = ast.Sub()
def test_pickling(self):
import pickle
@@ -182,8 +182,99 @@
ast2 = mod.loads(mod.dumps(ast, protocol))
self.assertEquals(to_tuple(ast2), to_tuple(ast))
+
+class ASTHelpers_Test(unittest.TestCase):
+
+ def test_parse(self):
+ a = ast.parse('foo(1 + 1)')
+ b = compile('foo(1 + 1)', '<unknown>', 'exec', ast.PyCF_ONLY_AST)
+ self.assertEqual(ast.dump(a), ast.dump(b))
+
+ def test_dump(self):
+ node = ast.parse('spam(eggs, "and cheese")')
+ self.assertEqual(ast.dump(node),
+ "Module(body=[Expr(value=Call(func=Name(id='spam', ctx=Load()), "
+ "args=[Name(id='eggs', ctx=Load()), Str(s='and cheese')], "
+ "keywords=[], starargs=None, kwargs=None))])"
+ )
+ self.assertEqual(ast.dump(node, annotate_fields=False),
+ "Module([Expr(Call(Name('spam', Load()), [Name('eggs', Load()), "
+ "Str('and cheese')], [], None, None))])"
+ )
+ self.assertEqual(ast.dump(node, include_attributes=True),
+ "Module(body=[Expr(value=Call(func=Name(id='spam', ctx=Load(), "
+ "lineno=1, col_offset=0), args=[Name(id='eggs', ctx=Load(), "
+ "lineno=1, col_offset=5), Str(s='and cheese', lineno=1, "
+ "col_offset=11)], keywords=[], starargs=None, kwargs=None, "
+ "lineno=1, col_offset=0), lineno=1, col_offset=0)])"
+ )
+
+ def test_copy_location(self):
+ src = ast.parse('1 + 1', mode='eval')
+ src.body.right = ast.copy_location(ast.Num(2), src.body.right)
+ self.assertEqual(ast.dump(src, include_attributes=True),
+ 'Expression(body=BinOp(left=Num(n=1, lineno=1, col_offset=0), '
+ 'op=Add(), right=Num(n=2, lineno=1, col_offset=4), lineno=1, '
+ 'col_offset=0))'
+ )
+
+ def test_fix_missing_locations(self):
+ src = ast.parse('write("spam")')
+ src.body.append(ast.Expr(ast.Call(ast.Name('spam', ast.Load()),
+ [ast.Str('eggs')], [], None, None)))
+ self.assertEqual(src, ast.fix_missing_locations(src))
+ self.assertEqual(ast.dump(src, include_attributes=True),
+ "Module(body=[Expr(value=Call(func=Name(id='write', ctx=Load(), "
+ "lineno=1, col_offset=0), args=[Str(s='spam', lineno=1, "
+ "col_offset=6)], keywords=[], starargs=None, kwargs=None, "
+ "lineno=1, col_offset=0), lineno=1, col_offset=0), "
+ "Expr(value=Call(func=Name(id='spam', ctx=Load(), lineno=1, "
+ "col_offset=0), args=[Str(s='eggs', lineno=1, col_offset=0)], "
+ "keywords=[], starargs=None, kwargs=None, lineno=1, "
+ "col_offset=0), lineno=1, col_offset=0)])"
+ )
+
+ def test_increment_lineno(self):
+ src = ast.parse('1 + 1', mode='eval')
+ self.assertEqual(ast.increment_lineno(src, n=3), src)
+ self.assertEqual(ast.dump(src, include_attributes=True),
+ 'Expression(body=BinOp(left=Num(n=1, lineno=4, col_offset=0), '
+ 'op=Add(), right=Num(n=1, lineno=4, col_offset=4), lineno=4, '
+ 'col_offset=0))'
+ )
+
+ def test_iter_fields(self):
+ node = ast.parse('foo()', mode='eval')
+ d = dict(ast.iter_fields(node.body))
+ self.assertEqual(d.pop('func').id, 'foo')
+ self.assertEqual(d, {'keywords': [], 'kwargs': None,
+ 'args': [], 'starargs': None})
+
+ def test_iter_child_nodes(self):
+ node = ast.parse("spam(23, 42, eggs='leek')", mode='eval')
+ self.assertEqual(len(list(ast.iter_child_nodes(node.body))), 4)
+ iterator = ast.iter_child_nodes(node.body)
+ self.assertEqual(next(iterator).id, 'spam')
+ self.assertEqual(next(iterator).n, 23)
+ self.assertEqual(next(iterator).n, 42)
+ self.assertEqual(ast.dump(next(iterator)),
+ "keyword(arg='eggs', value=Str(s='leek'))"
+ )
+
+ def test_get_docstring(self):
+ node = ast.parse('def foo():\n """line one\n line two"""')
+ self.assertEqual(ast.get_docstring(node.body[0]),
+ 'line one\nline two')
+
+ def test_literal_eval(self):
+ self.assertEqual(ast.literal_eval('[1, 2, 3]'), [1, 2, 3])
+ self.assertEqual(ast.literal_eval('{"foo": 42}'), {"foo": 42})
+ self.assertEqual(ast.literal_eval('(True, False, None)'), (True, False, None))
+ self.assertRaises(ValueError, ast.literal_eval, 'foo()')
+
+
def test_main():
- test_support.run_unittest(AST_Tests)
+ test_support.run_unittest(AST_Tests, ASTHelpers_Test)
def main():
if __name__ != '__main__':
Modified: python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py Tue Jun 10 17:03:15 2008
@@ -27,6 +27,9 @@
def __init__(self):
self.socket = dummysocket()
+ def close(self):
+ self.socket.close()
+
class exitingdummy:
def __init__(self):
pass
Modified: python/branches/tlee-ast-optimize/Lib/test/test_sys.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_sys.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_sys.py Tue Jun 10 17:03:15 2008
@@ -421,11 +421,14 @@
self.file.close()
test.test_support.unlink(test.test_support.TESTFN)
- def check_sizeof(self, o, size):
+ def check_sizeof(self, o, size, size2=None):
+ """Check size of o. Possible are size and optionally size2)."""
result = sys.getsizeof(o)
- msg = 'wrong size for %s: got %d, expected %d' \
- % (type(o), result, size)
- self.assertEqual(result, size, msg)
+ msg = 'wrong size for %s: got %d, expected ' % (type(o), result)
+ if (size2 != None) and (result != size):
+ self.assertEqual(result, size2, msg + str(size2))
+ else:
+ self.assertEqual(result, size, msg + str(size))
def align(self, value):
mod = value % self.p
@@ -517,10 +520,10 @@
pass
# type (PyTypeObject + PyNumberMethods + PyMappingMethods +
# PySequenceMethods + PyBufferProcs)
- len_typeobject = p + 2*l + 15*p + l + 4*p + l + 9*p + l + 11*p
+ len_typeobject = p + 2*l + 15*p + l + 4*p + l + 9*p +\
+ l + 11*p + self.align(4)
self.check_sizeof(class_newstyle,
- h + len_typeobject + 42*p + 10*p + 3*p + 6*p)
-
+ h + len_typeobject + 41*p + 10*p + 3*p + 6*p)
def test_specialtypes(self):
i = self.i
@@ -534,6 +537,24 @@
# list
self.check_sizeof([], h + l + p + l)
self.check_sizeof([1, 2, 3], h + l + p + l + 3*l)
+ # unicode
+ import math
+ usize = math.log(sys.maxunicode + 1, 2) / 8
+ samples = [u'', u'1'*100]
+ # we need to test for both sizes, because we don't know if the string
+ # has been cached
+ for s in samples:
+ basicsize = h + l + p + l + p + usize * (len(s) + 1)
+ self.check_sizeof(s, basicsize,\
+ size2=basicsize + sys.getsizeof(str(s)))
+ # XXX trigger caching encoded version as Python string
+ s = samples[1]
+ try:
+ getattr(sys, s)
+ except AttributeError:
+ pass
+ finally:
+ self.check_sizeof(s, basicsize + sys.getsizeof(str(s)))
h += l
# long
Modified: python/branches/tlee-ast-optimize/Lib/test/test_xrange.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_xrange.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_xrange.py Tue Jun 10 17:03:15 2008
@@ -2,6 +2,7 @@
import test.test_support, unittest
import sys
+import pickle
import warnings
warnings.filterwarnings("ignore", "integer argument expected",
@@ -57,6 +58,16 @@
self.assertEqual(len(r), sys.maxint)
self.assertRaises(OverflowError, xrange, -sys.maxint-1, sys.maxint, 2)
+ def test_pickling(self):
+ testcases = [(13,), (0, 11), (-22, 10), (20, 3, -1),
+ (13, 21, 3), (-2, 2, 2)]
+ for proto in range(pickle.HIGHEST_PROTOCOL):
+ for t in testcases:
+ r = xrange(*t)
+ self.assertEquals(list(pickle.loads(pickle.dumps(r, proto))),
+ list(r))
+
+
def test_main():
test.test_support.run_unittest(XrangeTest)
Modified: python/branches/tlee-ast-optimize/Misc/ACKS
==============================================================================
--- python/branches/tlee-ast-optimize/Misc/ACKS (original)
+++ python/branches/tlee-ast-optimize/Misc/ACKS Tue Jun 10 17:03:15 2008
@@ -156,6 +156,7 @@
Jonathan Dasteel
John DeGood
Vincent Delft
+Arnaud Delobelle
Erik Demaine
Roger Dev
Raghuram Devarakonda
@@ -574,6 +575,7 @@
Kevin Rodgers
Giampaolo Rodola
Mike Romberg
+Armin Ronacher
Case Roole
Timothy Roscoe
Jim Roskind
Modified: python/branches/tlee-ast-optimize/Misc/NEWS
==============================================================================
--- python/branches/tlee-ast-optimize/Misc/NEWS (original)
+++ python/branches/tlee-ast-optimize/Misc/NEWS Tue Jun 10 17:03:15 2008
@@ -77,6 +77,12 @@
Library
-------
+- The ctypes.byref function now takes an optional second parameter
+ which allows to specify an offset in bytes for the constructed
+ pointer-like object.
+
+- Added the ast module.
+
- Factored out the indentation cleaning from inspect.getdoc() into
inspect.cleandoc() to ease standalone use.
Modified: python/branches/tlee-ast-optimize/Misc/developers.txt
==============================================================================
--- python/branches/tlee-ast-optimize/Misc/developers.txt (original)
+++ python/branches/tlee-ast-optimize/Misc/developers.txt Tue Jun 10 17:03:15 2008
@@ -17,6 +17,9 @@
Permissions History
-------------------
+- Gregor Lingl was given SVN access on 10 June 2008 by MvL,
+ for work on the turtle module.
+
- Robert Schuppenies was given SVN access on 21 May 2008 by MvL,
for GSoC contributions.
Modified: python/branches/tlee-ast-optimize/Modules/_ctypes/callproc.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/_ctypes/callproc.c (original)
+++ python/branches/tlee-ast-optimize/Modules/_ctypes/callproc.c Tue Jun 10 17:03:15 2008
@@ -1561,7 +1561,7 @@
}
static char byref_doc[] =
-"byref(C instance) -> byref-object\n"
+"byref(C instance[, offset=0]) -> byref-object\n"
"Return a pointer lookalike to a C instance, only usable\n"
"as function argument";
@@ -1570,9 +1570,21 @@
* but still has a reference to self.
*/
static PyObject *
-byref(PyObject *self, PyObject *obj)
+byref(PyObject *self, PyObject *args)
{
PyCArgObject *parg;
+ PyObject *obj;
+ PyObject *pyoffset = NULL;
+ Py_ssize_t offset = 0;
+
+ if (!PyArg_UnpackTuple(args, "byref", 1, 2,
+ &obj, &pyoffset))
+ return NULL;
+ if (pyoffset) {
+ offset = PyNumber_AsSsize_t(pyoffset, NULL);
+ if (offset == -1 && PyErr_Occurred())
+ return NULL;
+ }
if (!CDataObject_Check(obj)) {
PyErr_Format(PyExc_TypeError,
"byref() argument must be a ctypes instance, not '%s'",
@@ -1588,7 +1600,7 @@
parg->pffi_type = &ffi_type_pointer;
Py_INCREF(obj);
parg->obj = obj;
- parg->value.p = ((CDataObject *)obj)->b_ptr;
+ parg->value.p = (char *)((CDataObject *)obj)->b_ptr + offset;
return (PyObject *)parg;
}
@@ -1876,7 +1888,7 @@
#endif
{"alignment", align_func, METH_O, alignment_doc},
{"sizeof", sizeof_func, METH_O, sizeof_doc},
- {"byref", byref, METH_O, byref_doc},
+ {"byref", byref, METH_VARARGS, byref_doc},
{"addressof", addressof, METH_O, addressof_doc},
{"call_function", call_function, METH_VARARGS },
{"call_cdeclfunction", call_cdeclfunction, METH_VARARGS },
Modified: python/branches/tlee-ast-optimize/Modules/itertoolsmodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/itertoolsmodule.c (original)
+++ python/branches/tlee-ast-optimize/Modules/itertoolsmodule.c Tue Jun 10 17:03:15 2008
@@ -2198,7 +2198,7 @@
}
PyDoc_STRVAR(combinations_doc,
-"combinations(iterables) --> combinations object\n\
+"combinations(iterable[, r]) --> combinations object\n\
\n\
Return successive r-length combinations of elements in the iterable.\n\n\
combinations(range(4), 3) --> (0,1,2), (0,1,3), (0,2,3), (1,2,3)");
@@ -2469,10 +2469,10 @@
}
PyDoc_STRVAR(permutations_doc,
-"permutations(iterables[, r]) --> permutations object\n\
+"permutations(iterable[, r]) --> permutations object\n\
\n\
Return successive r-length permutations of elements in the iterable.\n\n\
-permutations(range(4), 3) --> (0,1,2), (0,1,3), (0,2,3), (1,2,3)");
+permutations(range(3), 2) --> (0,1), (0,2), (1,0), (1,2), (2,0), (2,1)");
static PyTypeObject permutations_type = {
PyVarObject_HEAD_INIT(NULL, 0)
Modified: python/branches/tlee-ast-optimize/Objects/rangeobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/rangeobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/rangeobject.c Tue Jun 10 17:03:15 2008
@@ -129,6 +129,16 @@
return rtn;
}
+/* Pickling support */
+static PyObject *
+range_reduce(rangeobject *r, PyObject *args)
+{
+ return Py_BuildValue("(O(iii))", Py_TYPE(r),
+ r->start,
+ r->start + r->len * r->step,
+ r->step);
+}
+
static PySequenceMethods range_as_sequence = {
(lenfunc)range_length, /* sq_length */
0, /* sq_concat */
@@ -145,6 +155,7 @@
static PyMethodDef range_methods[] = {
{"__reversed__", (PyCFunction)range_reverse, METH_NOARGS, reverse_doc},
+ {"__reduce__", (PyCFunction)range_reduce, METH_VARARGS},
{NULL, NULL} /* sentinel */
};
Modified: python/branches/tlee-ast-optimize/Objects/unicodeobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/unicodeobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/unicodeobject.c Tue Jun 10 17:03:15 2008
@@ -7895,6 +7895,29 @@
\n\
");
+static PyObject *
+unicode__sizeof__(PyUnicodeObject *v)
+{
+ PyObject *res = NULL, *defsize = NULL;
+
+ res = PyInt_FromSsize_t(sizeof(PyUnicodeObject) +
+ sizeof(Py_UNICODE) * (v->length + 1));
+ if (v->defenc) {
+ defsize = PyObject_CallMethod(v->defenc, "__sizeof__", NULL);
+ if (defsize == NULL) {
+ Py_DECREF(res);
+ return NULL;
+ }
+ res = PyNumber_Add(res, defsize);
+ Py_DECREF(defsize);
+ }
+ return res;
+}
+
+PyDoc_STRVAR(sizeof__doc__,
+"S.__sizeof__() -> size of S in memory, in bytes\n\
+\n\
+");
static PyObject *
unicode_getnewargs(PyUnicodeObject *v)
@@ -7952,6 +7975,7 @@
{"__format__", (PyCFunction) unicode__format__, METH_VARARGS, p_format__doc__},
{"_formatter_field_name_split", (PyCFunction) formatter_field_name_split, METH_NOARGS},
{"_formatter_parser", (PyCFunction) formatter_parser, METH_NOARGS},
+ {"__sizeof__", (PyCFunction) unicode__sizeof__, METH_NOARGS, sizeof__doc__},
#if 0
{"capwords", (PyCFunction) unicode_capwords, METH_NOARGS, capwords__doc__},
#endif
More information about the Python-checkins
mailing list