[pypy-commit] pypy default: merged with default branch
brunogola
noreply at buildbot.pypy.org
Fri Oct 28 09:34:17 CEST 2011
Author: Bruno Gola <brunogola at gmail.com>
Branch:
Changeset: r48550:caf9fa132b86
Date: 2011-10-27 21:12 -0200
http://bitbucket.org/pypy/pypy/changeset/caf9fa132b86/
Log: merged with default branch
diff too long, truncating to 10000 out of 16297 lines
diff --git a/lib-python/modified-2.7/heapq.py b/lib-python/modified-2.7/heapq.py
new file mode 100644
--- /dev/null
+++ b/lib-python/modified-2.7/heapq.py
@@ -0,0 +1,442 @@
+# -*- coding: latin-1 -*-
+
+"""Heap queue algorithm (a.k.a. priority queue).
+
+Heaps are arrays for which a[k] <= a[2*k+1] and a[k] <= a[2*k+2] for
+all k, counting elements from 0. For the sake of comparison,
+non-existing elements are considered to be infinite. The interesting
+property of a heap is that a[0] is always its smallest element.
+
+Usage:
+
+heap = [] # creates an empty heap
+heappush(heap, item) # pushes a new item on the heap
+item = heappop(heap) # pops the smallest item from the heap
+item = heap[0] # smallest item on the heap without popping it
+heapify(x) # transforms list into a heap, in-place, in linear time
+item = heapreplace(heap, item) # pops and returns smallest item, and adds
+ # new item; the heap size is unchanged
+
+Our API differs from textbook heap algorithms as follows:
+
+- We use 0-based indexing. This makes the relationship between the
+ index for a node and the indexes for its children slightly less
+ obvious, but is more suitable since Python uses 0-based indexing.
+
+- Our heappop() method returns the smallest item, not the largest.
+
+These two make it possible to view the heap as a regular Python list
+without surprises: heap[0] is the smallest item, and heap.sort()
+maintains the heap invariant!
+"""
+
+# Original code by Kevin O'Connor, augmented by Tim Peters and Raymond Hettinger
+
+__about__ = """Heap queues
+
+[explanation by Fran�ois Pinard]
+
+Heaps are arrays for which a[k] <= a[2*k+1] and a[k] <= a[2*k+2] for
+all k, counting elements from 0. For the sake of comparison,
+non-existing elements are considered to be infinite. The interesting
+property of a heap is that a[0] is always its smallest element.
+
+The strange invariant above is meant to be an efficient memory
+representation for a tournament. The numbers below are `k', not a[k]:
+
+ 0
+
+ 1 2
+
+ 3 4 5 6
+
+ 7 8 9 10 11 12 13 14
+
+ 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
+
+
+In the tree above, each cell `k' is topping `2*k+1' and `2*k+2'. In
+an usual binary tournament we see in sports, each cell is the winner
+over the two cells it tops, and we can trace the winner down the tree
+to see all opponents s/he had. However, in many computer applications
+of such tournaments, we do not need to trace the history of a winner.
+To be more memory efficient, when a winner is promoted, we try to
+replace it by something else at a lower level, and the rule becomes
+that a cell and the two cells it tops contain three different items,
+but the top cell "wins" over the two topped cells.
+
+If this heap invariant is protected at all time, index 0 is clearly
+the overall winner. The simplest algorithmic way to remove it and
+find the "next" winner is to move some loser (let's say cell 30 in the
+diagram above) into the 0 position, and then percolate this new 0 down
+the tree, exchanging values, until the invariant is re-established.
+This is clearly logarithmic on the total number of items in the tree.
+By iterating over all items, you get an O(n ln n) sort.
+
+A nice feature of this sort is that you can efficiently insert new
+items while the sort is going on, provided that the inserted items are
+not "better" than the last 0'th element you extracted. This is
+especially useful in simulation contexts, where the tree holds all
+incoming events, and the "win" condition means the smallest scheduled
+time. When an event schedule other events for execution, they are
+scheduled into the future, so they can easily go into the heap. So, a
+heap is a good structure for implementing schedulers (this is what I
+used for my MIDI sequencer :-).
+
+Various structures for implementing schedulers have been extensively
+studied, and heaps are good for this, as they are reasonably speedy,
+the speed is almost constant, and the worst case is not much different
+than the average case. However, there are other representations which
+are more efficient overall, yet the worst cases might be terrible.
+
+Heaps are also very useful in big disk sorts. You most probably all
+know that a big sort implies producing "runs" (which are pre-sorted
+sequences, which size is usually related to the amount of CPU memory),
+followed by a merging passes for these runs, which merging is often
+very cleverly organised[1]. It is very important that the initial
+sort produces the longest runs possible. Tournaments are a good way
+to that. If, using all the memory available to hold a tournament, you
+replace and percolate items that happen to fit the current run, you'll
+produce runs which are twice the size of the memory for random input,
+and much better for input fuzzily ordered.
+
+Moreover, if you output the 0'th item on disk and get an input which
+may not fit in the current tournament (because the value "wins" over
+the last output value), it cannot fit in the heap, so the size of the
+heap decreases. The freed memory could be cleverly reused immediately
+for progressively building a second heap, which grows at exactly the
+same rate the first heap is melting. When the first heap completely
+vanishes, you switch heaps and start a new run. Clever and quite
+effective!
+
+In a word, heaps are useful memory structures to know. I use them in
+a few applications, and I think it is good to keep a `heap' module
+around. :-)
+
+--------------------
+[1] The disk balancing algorithms which are current, nowadays, are
+more annoying than clever, and this is a consequence of the seeking
+capabilities of the disks. On devices which cannot seek, like big
+tape drives, the story was quite different, and one had to be very
+clever to ensure (far in advance) that each tape movement will be the
+most effective possible (that is, will best participate at
+"progressing" the merge). Some tapes were even able to read
+backwards, and this was also used to avoid the rewinding time.
+Believe me, real good tape sorts were quite spectacular to watch!
+From all times, sorting has always been a Great Art! :-)
+"""
+
+__all__ = ['heappush', 'heappop', 'heapify', 'heapreplace', 'merge',
+ 'nlargest', 'nsmallest', 'heappushpop']
+
+from itertools import islice, repeat, count, imap, izip, tee, chain
+from operator import itemgetter
+import bisect
+
+def heappush(heap, item):
+ """Push item onto heap, maintaining the heap invariant."""
+ heap.append(item)
+ _siftdown(heap, 0, len(heap)-1)
+
+def heappop(heap):
+ """Pop the smallest item off the heap, maintaining the heap invariant."""
+ lastelt = heap.pop() # raises appropriate IndexError if heap is empty
+ if heap:
+ returnitem = heap[0]
+ heap[0] = lastelt
+ _siftup(heap, 0)
+ else:
+ returnitem = lastelt
+ return returnitem
+
+def heapreplace(heap, item):
+ """Pop and return the current smallest value, and add the new item.
+
+ This is more efficient than heappop() followed by heappush(), and can be
+ more appropriate when using a fixed-size heap. Note that the value
+ returned may be larger than item! That constrains reasonable uses of
+ this routine unless written as part of a conditional replacement:
+
+ if item > heap[0]:
+ item = heapreplace(heap, item)
+ """
+ returnitem = heap[0] # raises appropriate IndexError if heap is empty
+ heap[0] = item
+ _siftup(heap, 0)
+ return returnitem
+
+def heappushpop(heap, item):
+ """Fast version of a heappush followed by a heappop."""
+ if heap and heap[0] < item:
+ item, heap[0] = heap[0], item
+ _siftup(heap, 0)
+ return item
+
+def heapify(x):
+ """Transform list into a heap, in-place, in O(len(heap)) time."""
+ n = len(x)
+ # Transform bottom-up. The largest index there's any point to looking at
+ # is the largest with a child index in-range, so must have 2*i + 1 < n,
+ # or i < (n-1)/2. If n is even = 2*j, this is (2*j-1)/2 = j-1/2 so
+ # j-1 is the largest, which is n//2 - 1. If n is odd = 2*j+1, this is
+ # (2*j+1-1)/2 = j so j-1 is the largest, and that's again n//2-1.
+ for i in reversed(xrange(n//2)):
+ _siftup(x, i)
+
+def nlargest(n, iterable):
+ """Find the n largest elements in a dataset.
+
+ Equivalent to: sorted(iterable, reverse=True)[:n]
+ """
+ if n < 0: # for consistency with the c impl
+ return []
+ it = iter(iterable)
+ result = list(islice(it, n))
+ if not result:
+ return result
+ heapify(result)
+ _heappushpop = heappushpop
+ for elem in it:
+ _heappushpop(result, elem)
+ result.sort(reverse=True)
+ return result
+
+def nsmallest(n, iterable):
+ """Find the n smallest elements in a dataset.
+
+ Equivalent to: sorted(iterable)[:n]
+ """
+ if n < 0: # for consistency with the c impl
+ return []
+ if hasattr(iterable, '__len__') and n * 10 <= len(iterable):
+ # For smaller values of n, the bisect method is faster than a minheap.
+ # It is also memory efficient, consuming only n elements of space.
+ it = iter(iterable)
+ result = sorted(islice(it, 0, n))
+ if not result:
+ return result
+ insort = bisect.insort
+ pop = result.pop
+ los = result[-1] # los --> Largest of the nsmallest
+ for elem in it:
+ if los <= elem:
+ continue
+ insort(result, elem)
+ pop()
+ los = result[-1]
+ return result
+ # An alternative approach manifests the whole iterable in memory but
+ # saves comparisons by heapifying all at once. Also, saves time
+ # over bisect.insort() which has O(n) data movement time for every
+ # insertion. Finding the n smallest of an m length iterable requires
+ # O(m) + O(n log m) comparisons.
+ h = list(iterable)
+ heapify(h)
+ return map(heappop, repeat(h, min(n, len(h))))
+
+# 'heap' is a heap at all indices >= startpos, except possibly for pos. pos
+# is the index of a leaf with a possibly out-of-order value. Restore the
+# heap invariant.
+def _siftdown(heap, startpos, pos):
+ newitem = heap[pos]
+ # Follow the path to the root, moving parents down until finding a place
+ # newitem fits.
+ while pos > startpos:
+ parentpos = (pos - 1) >> 1
+ parent = heap[parentpos]
+ if newitem < parent:
+ heap[pos] = parent
+ pos = parentpos
+ continue
+ break
+ heap[pos] = newitem
+
+# The child indices of heap index pos are already heaps, and we want to make
+# a heap at index pos too. We do this by bubbling the smaller child of
+# pos up (and so on with that child's children, etc) until hitting a leaf,
+# then using _siftdown to move the oddball originally at index pos into place.
+#
+# We *could* break out of the loop as soon as we find a pos where newitem <=
+# both its children, but turns out that's not a good idea, and despite that
+# many books write the algorithm that way. During a heap pop, the last array
+# element is sifted in, and that tends to be large, so that comparing it
+# against values starting from the root usually doesn't pay (= usually doesn't
+# get us out of the loop early). See Knuth, Volume 3, where this is
+# explained and quantified in an exercise.
+#
+# Cutting the # of comparisons is important, since these routines have no
+# way to extract "the priority" from an array element, so that intelligence
+# is likely to be hiding in custom __cmp__ methods, or in array elements
+# storing (priority, record) tuples. Comparisons are thus potentially
+# expensive.
+#
+# On random arrays of length 1000, making this change cut the number of
+# comparisons made by heapify() a little, and those made by exhaustive
+# heappop() a lot, in accord with theory. Here are typical results from 3
+# runs (3 just to demonstrate how small the variance is):
+#
+# Compares needed by heapify Compares needed by 1000 heappops
+# -------------------------- --------------------------------
+# 1837 cut to 1663 14996 cut to 8680
+# 1855 cut to 1659 14966 cut to 8678
+# 1847 cut to 1660 15024 cut to 8703
+#
+# Building the heap by using heappush() 1000 times instead required
+# 2198, 2148, and 2219 compares: heapify() is more efficient, when
+# you can use it.
+#
+# The total compares needed by list.sort() on the same lists were 8627,
+# 8627, and 8632 (this should be compared to the sum of heapify() and
+# heappop() compares): list.sort() is (unsurprisingly!) more efficient
+# for sorting.
+
+def _siftup(heap, pos):
+ endpos = len(heap)
+ startpos = pos
+ newitem = heap[pos]
+ # Bubble up the smaller child until hitting a leaf.
+ childpos = 2*pos + 1 # leftmost child position
+ while childpos < endpos:
+ # Set childpos to index of smaller child.
+ rightpos = childpos + 1
+ if rightpos < endpos and not heap[childpos] < heap[rightpos]:
+ childpos = rightpos
+ # Move the smaller child up.
+ heap[pos] = heap[childpos]
+ pos = childpos
+ childpos = 2*pos + 1
+ # The leaf at pos is empty now. Put newitem there, and bubble it up
+ # to its final resting place (by sifting its parents down).
+ heap[pos] = newitem
+ _siftdown(heap, startpos, pos)
+
+# If available, use C implementation
+try:
+ from _heapq import *
+except ImportError:
+ pass
+
+def merge(*iterables):
+ '''Merge multiple sorted inputs into a single sorted output.
+
+ Similar to sorted(itertools.chain(*iterables)) but returns a generator,
+ does not pull the data into memory all at once, and assumes that each of
+ the input streams is already sorted (smallest to largest).
+
+ >>> list(merge([1,3,5,7], [0,2,4,8], [5,10,15,20], [], [25]))
+ [0, 1, 2, 3, 4, 5, 5, 7, 8, 10, 15, 20, 25]
+
+ '''
+ _heappop, _heapreplace, _StopIteration = heappop, heapreplace, StopIteration
+
+ h = []
+ h_append = h.append
+ for itnum, it in enumerate(map(iter, iterables)):
+ try:
+ next = it.next
+ h_append([next(), itnum, next])
+ except _StopIteration:
+ pass
+ heapify(h)
+
+ while 1:
+ try:
+ while 1:
+ v, itnum, next = s = h[0] # raises IndexError when h is empty
+ yield v
+ s[0] = next() # raises StopIteration when exhausted
+ _heapreplace(h, s) # restore heap condition
+ except _StopIteration:
+ _heappop(h) # remove empty iterator
+ except IndexError:
+ return
+
+# Extend the implementations of nsmallest and nlargest to use a key= argument
+_nsmallest = nsmallest
+def nsmallest(n, iterable, key=None):
+ """Find the n smallest elements in a dataset.
+
+ Equivalent to: sorted(iterable, key=key)[:n]
+ """
+ # Short-cut for n==1 is to use min() when len(iterable)>0
+ if n == 1:
+ it = iter(iterable)
+ head = list(islice(it, 1))
+ if not head:
+ return []
+ if key is None:
+ return [min(chain(head, it))]
+ return [min(chain(head, it), key=key)]
+
+ # When n>=size, it's faster to use sort()
+ try:
+ size = len(iterable)
+ except (TypeError, AttributeError):
+ pass
+ else:
+ if n >= size:
+ return sorted(iterable, key=key)[:n]
+
+ # When key is none, use simpler decoration
+ if key is None:
+ it = izip(iterable, count()) # decorate
+ result = _nsmallest(n, it)
+ return map(itemgetter(0), result) # undecorate
+
+ # General case, slowest method
+ in1, in2 = tee(iterable)
+ it = izip(imap(key, in1), count(), in2) # decorate
+ result = _nsmallest(n, it)
+ return map(itemgetter(2), result) # undecorate
+
+_nlargest = nlargest
+def nlargest(n, iterable, key=None):
+ """Find the n largest elements in a dataset.
+
+ Equivalent to: sorted(iterable, key=key, reverse=True)[:n]
+ """
+
+ # Short-cut for n==1 is to use max() when len(iterable)>0
+ if n == 1:
+ it = iter(iterable)
+ head = list(islice(it, 1))
+ if not head:
+ return []
+ if key is None:
+ return [max(chain(head, it))]
+ return [max(chain(head, it), key=key)]
+
+ # When n>=size, it's faster to use sort()
+ try:
+ size = len(iterable)
+ except (TypeError, AttributeError):
+ pass
+ else:
+ if n >= size:
+ return sorted(iterable, key=key, reverse=True)[:n]
+
+ # When key is none, use simpler decoration
+ if key is None:
+ it = izip(iterable, count(0,-1)) # decorate
+ result = _nlargest(n, it)
+ return map(itemgetter(0), result) # undecorate
+
+ # General case, slowest method
+ in1, in2 = tee(iterable)
+ it = izip(imap(key, in1), count(0,-1), in2) # decorate
+ result = _nlargest(n, it)
+ return map(itemgetter(2), result) # undecorate
+
+if __name__ == "__main__":
+ # Simple sanity test
+ heap = []
+ data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
+ for item in data:
+ heappush(heap, item)
+ sort = []
+ while heap:
+ sort.append(heappop(heap))
+ print sort
+
+ import doctest
+ doctest.testmod()
diff --git a/lib-python/modified-2.7/httplib.py b/lib-python/modified-2.7/httplib.py
new file mode 100644
--- /dev/null
+++ b/lib-python/modified-2.7/httplib.py
@@ -0,0 +1,1377 @@
+"""HTTP/1.1 client library
+
+<intro stuff goes here>
+<other stuff, too>
+
+HTTPConnection goes through a number of "states", which define when a client
+may legally make another request or fetch the response for a particular
+request. This diagram details these state transitions:
+
+ (null)
+ |
+ | HTTPConnection()
+ v
+ Idle
+ |
+ | putrequest()
+ v
+ Request-started
+ |
+ | ( putheader() )* endheaders()
+ v
+ Request-sent
+ |
+ | response = getresponse()
+ v
+ Unread-response [Response-headers-read]
+ |\____________________
+ | |
+ | response.read() | putrequest()
+ v v
+ Idle Req-started-unread-response
+ ______/|
+ / |
+ response.read() | | ( putheader() )* endheaders()
+ v v
+ Request-started Req-sent-unread-response
+ |
+ | response.read()
+ v
+ Request-sent
+
+This diagram presents the following rules:
+ -- a second request may not be started until {response-headers-read}
+ -- a response [object] cannot be retrieved until {request-sent}
+ -- there is no differentiation between an unread response body and a
+ partially read response body
+
+Note: this enforcement is applied by the HTTPConnection class. The
+ HTTPResponse class does not enforce this state machine, which
+ implies sophisticated clients may accelerate the request/response
+ pipeline. Caution should be taken, though: accelerating the states
+ beyond the above pattern may imply knowledge of the server's
+ connection-close behavior for certain requests. For example, it
+ is impossible to tell whether the server will close the connection
+ UNTIL the response headers have been read; this means that further
+ requests cannot be placed into the pipeline until it is known that
+ the server will NOT be closing the connection.
+
+Logical State __state __response
+------------- ------- ----------
+Idle _CS_IDLE None
+Request-started _CS_REQ_STARTED None
+Request-sent _CS_REQ_SENT None
+Unread-response _CS_IDLE <response_class>
+Req-started-unread-response _CS_REQ_STARTED <response_class>
+Req-sent-unread-response _CS_REQ_SENT <response_class>
+"""
+
+from array import array
+import os
+import socket
+from sys import py3kwarning
+from urlparse import urlsplit
+import warnings
+with warnings.catch_warnings():
+ if py3kwarning:
+ warnings.filterwarnings("ignore", ".*mimetools has been removed",
+ DeprecationWarning)
+ import mimetools
+
+try:
+ from cStringIO import StringIO
+except ImportError:
+ from StringIO import StringIO
+
+__all__ = ["HTTP", "HTTPResponse", "HTTPConnection",
+ "HTTPException", "NotConnected", "UnknownProtocol",
+ "UnknownTransferEncoding", "UnimplementedFileMode",
+ "IncompleteRead", "InvalidURL", "ImproperConnectionState",
+ "CannotSendRequest", "CannotSendHeader", "ResponseNotReady",
+ "BadStatusLine", "error", "responses"]
+
+HTTP_PORT = 80
+HTTPS_PORT = 443
+
+_UNKNOWN = 'UNKNOWN'
+
+# connection states
+_CS_IDLE = 'Idle'
+_CS_REQ_STARTED = 'Request-started'
+_CS_REQ_SENT = 'Request-sent'
+
+# status codes
+# informational
+CONTINUE = 100
+SWITCHING_PROTOCOLS = 101
+PROCESSING = 102
+
+# successful
+OK = 200
+CREATED = 201
+ACCEPTED = 202
+NON_AUTHORITATIVE_INFORMATION = 203
+NO_CONTENT = 204
+RESET_CONTENT = 205
+PARTIAL_CONTENT = 206
+MULTI_STATUS = 207
+IM_USED = 226
+
+# redirection
+MULTIPLE_CHOICES = 300
+MOVED_PERMANENTLY = 301
+FOUND = 302
+SEE_OTHER = 303
+NOT_MODIFIED = 304
+USE_PROXY = 305
+TEMPORARY_REDIRECT = 307
+
+# client error
+BAD_REQUEST = 400
+UNAUTHORIZED = 401
+PAYMENT_REQUIRED = 402
+FORBIDDEN = 403
+NOT_FOUND = 404
+METHOD_NOT_ALLOWED = 405
+NOT_ACCEPTABLE = 406
+PROXY_AUTHENTICATION_REQUIRED = 407
+REQUEST_TIMEOUT = 408
+CONFLICT = 409
+GONE = 410
+LENGTH_REQUIRED = 411
+PRECONDITION_FAILED = 412
+REQUEST_ENTITY_TOO_LARGE = 413
+REQUEST_URI_TOO_LONG = 414
+UNSUPPORTED_MEDIA_TYPE = 415
+REQUESTED_RANGE_NOT_SATISFIABLE = 416
+EXPECTATION_FAILED = 417
+UNPROCESSABLE_ENTITY = 422
+LOCKED = 423
+FAILED_DEPENDENCY = 424
+UPGRADE_REQUIRED = 426
+
+# server error
+INTERNAL_SERVER_ERROR = 500
+NOT_IMPLEMENTED = 501
+BAD_GATEWAY = 502
+SERVICE_UNAVAILABLE = 503
+GATEWAY_TIMEOUT = 504
+HTTP_VERSION_NOT_SUPPORTED = 505
+INSUFFICIENT_STORAGE = 507
+NOT_EXTENDED = 510
+
+# Mapping status codes to official W3C names
+responses = {
+ 100: 'Continue',
+ 101: 'Switching Protocols',
+
+ 200: 'OK',
+ 201: 'Created',
+ 202: 'Accepted',
+ 203: 'Non-Authoritative Information',
+ 204: 'No Content',
+ 205: 'Reset Content',
+ 206: 'Partial Content',
+
+ 300: 'Multiple Choices',
+ 301: 'Moved Permanently',
+ 302: 'Found',
+ 303: 'See Other',
+ 304: 'Not Modified',
+ 305: 'Use Proxy',
+ 306: '(Unused)',
+ 307: 'Temporary Redirect',
+
+ 400: 'Bad Request',
+ 401: 'Unauthorized',
+ 402: 'Payment Required',
+ 403: 'Forbidden',
+ 404: 'Not Found',
+ 405: 'Method Not Allowed',
+ 406: 'Not Acceptable',
+ 407: 'Proxy Authentication Required',
+ 408: 'Request Timeout',
+ 409: 'Conflict',
+ 410: 'Gone',
+ 411: 'Length Required',
+ 412: 'Precondition Failed',
+ 413: 'Request Entity Too Large',
+ 414: 'Request-URI Too Long',
+ 415: 'Unsupported Media Type',
+ 416: 'Requested Range Not Satisfiable',
+ 417: 'Expectation Failed',
+
+ 500: 'Internal Server Error',
+ 501: 'Not Implemented',
+ 502: 'Bad Gateway',
+ 503: 'Service Unavailable',
+ 504: 'Gateway Timeout',
+ 505: 'HTTP Version Not Supported',
+}
+
+# maximal amount of data to read at one time in _safe_read
+MAXAMOUNT = 1048576
+
+class HTTPMessage(mimetools.Message):
+
+ def addheader(self, key, value):
+ """Add header for field key handling repeats."""
+ prev = self.dict.get(key)
+ if prev is None:
+ self.dict[key] = value
+ else:
+ combined = ", ".join((prev, value))
+ self.dict[key] = combined
+
+ def addcontinue(self, key, more):
+ """Add more field data from a continuation line."""
+ prev = self.dict[key]
+ self.dict[key] = prev + "\n " + more
+
+ def readheaders(self):
+ """Read header lines.
+
+ Read header lines up to the entirely blank line that terminates them.
+ The (normally blank) line that ends the headers is skipped, but not
+ included in the returned list. If a non-header line ends the headers,
+ (which is an error), an attempt is made to backspace over it; it is
+ never included in the returned list.
+
+ The variable self.status is set to the empty string if all went well,
+ otherwise it is an error message. The variable self.headers is a
+ completely uninterpreted list of lines contained in the header (so
+ printing them will reproduce the header exactly as it appears in the
+ file).
+
+ If multiple header fields with the same name occur, they are combined
+ according to the rules in RFC 2616 sec 4.2:
+
+ Appending each subsequent field-value to the first, each separated
+ by a comma. The order in which header fields with the same field-name
+ are received is significant to the interpretation of the combined
+ field value.
+ """
+ # XXX The implementation overrides the readheaders() method of
+ # rfc822.Message. The base class design isn't amenable to
+ # customized behavior here so the method here is a copy of the
+ # base class code with a few small changes.
+
+ self.dict = {}
+ self.unixfrom = ''
+ self.headers = hlist = []
+ self.status = ''
+ headerseen = ""
+ firstline = 1
+ startofline = unread = tell = None
+ if hasattr(self.fp, 'unread'):
+ unread = self.fp.unread
+ elif self.seekable:
+ tell = self.fp.tell
+ while True:
+ if tell:
+ try:
+ startofline = tell()
+ except IOError:
+ startofline = tell = None
+ self.seekable = 0
+ line = self.fp.readline()
+ if not line:
+ self.status = 'EOF in headers'
+ break
+ # Skip unix From name time lines
+ if firstline and line.startswith('From '):
+ self.unixfrom = self.unixfrom + line
+ continue
+ firstline = 0
+ if headerseen and line[0] in ' \t':
+ # XXX Not sure if continuation lines are handled properly
+ # for http and/or for repeating headers
+ # It's a continuation line.
+ hlist.append(line)
+ self.addcontinue(headerseen, line.strip())
+ continue
+ elif self.iscomment(line):
+ # It's a comment. Ignore it.
+ continue
+ elif self.islast(line):
+ # Note! No pushback here! The delimiter line gets eaten.
+ break
+ headerseen = self.isheader(line)
+ if headerseen:
+ # It's a legal header line, save it.
+ hlist.append(line)
+ self.addheader(headerseen, line[len(headerseen)+1:].strip())
+ continue
+ else:
+ # It's not a header line; throw it back and stop here.
+ if not self.dict:
+ self.status = 'No headers'
+ else:
+ self.status = 'Non-header line where header expected'
+ # Try to undo the read.
+ if unread:
+ unread(line)
+ elif tell:
+ self.fp.seek(startofline)
+ else:
+ self.status = self.status + '; bad seek'
+ break
+
+class HTTPResponse:
+
+ # strict: If true, raise BadStatusLine if the status line can't be
+ # parsed as a valid HTTP/1.0 or 1.1 status line. By default it is
+ # false because it prevents clients from talking to HTTP/0.9
+ # servers. Note that a response with a sufficiently corrupted
+ # status line will look like an HTTP/0.9 response.
+
+ # See RFC 2616 sec 19.6 and RFC 1945 sec 6 for details.
+
+ def __init__(self, sock, debuglevel=0, strict=0, method=None, buffering=False):
+ if buffering:
+ # The caller won't be using any sock.recv() calls, so buffering
+ # is fine and recommended for performance.
+ self.fp = sock.makefile('rb')
+ else:
+ # The buffer size is specified as zero, because the headers of
+ # the response are read with readline(). If the reads were
+ # buffered the readline() calls could consume some of the
+ # response, which make be read via a recv() on the underlying
+ # socket.
+ self.fp = sock.makefile('rb', 0)
+ self.debuglevel = debuglevel
+ self.strict = strict
+ self._method = method
+
+ self.msg = None
+
+ # from the Status-Line of the response
+ self.version = _UNKNOWN # HTTP-Version
+ self.status = _UNKNOWN # Status-Code
+ self.reason = _UNKNOWN # Reason-Phrase
+
+ self.chunked = _UNKNOWN # is "chunked" being used?
+ self.chunk_left = _UNKNOWN # bytes left to read in current chunk
+ self.length = _UNKNOWN # number of bytes left in response
+ self.will_close = _UNKNOWN # conn will close at end of response
+
+ def _read_status(self):
+ # Initialize with Simple-Response defaults
+ line = self.fp.readline()
+ if self.debuglevel > 0:
+ print "reply:", repr(line)
+ if not line:
+ # Presumably, the server closed the connection before
+ # sending a valid response.
+ raise BadStatusLine(line)
+ try:
+ [version, status, reason] = line.split(None, 2)
+ except ValueError:
+ try:
+ [version, status] = line.split(None, 1)
+ reason = ""
+ except ValueError:
+ # empty version will cause next test to fail and status
+ # will be treated as 0.9 response.
+ version = ""
+ if not version.startswith('HTTP/'):
+ if self.strict:
+ self.close()
+ raise BadStatusLine(line)
+ else:
+ # assume it's a Simple-Response from an 0.9 server
+ self.fp = LineAndFileWrapper(line, self.fp)
+ return "HTTP/0.9", 200, ""
+
+ # The status code is a three-digit number
+ try:
+ status = int(status)
+ if status < 100 or status > 999:
+ raise BadStatusLine(line)
+ except ValueError:
+ raise BadStatusLine(line)
+ return version, status, reason
+
+ def begin(self):
+ if self.msg is not None:
+ # we've already started reading the response
+ return
+
+ # read until we get a non-100 response
+ while True:
+ version, status, reason = self._read_status()
+ if status != CONTINUE:
+ break
+ # skip the header from the 100 response
+ while True:
+ skip = self.fp.readline().strip()
+ if not skip:
+ break
+ if self.debuglevel > 0:
+ print "header:", skip
+
+ self.status = status
+ self.reason = reason.strip()
+ if version == 'HTTP/1.0':
+ self.version = 10
+ elif version.startswith('HTTP/1.'):
+ self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1
+ elif version == 'HTTP/0.9':
+ self.version = 9
+ else:
+ raise UnknownProtocol(version)
+
+ if self.version == 9:
+ self.length = None
+ self.chunked = 0
+ self.will_close = 1
+ self.msg = HTTPMessage(StringIO())
+ return
+
+ self.msg = HTTPMessage(self.fp, 0)
+ if self.debuglevel > 0:
+ for hdr in self.msg.headers:
+ print "header:", hdr,
+
+ # don't let the msg keep an fp
+ self.msg.fp = None
+
+ # are we using the chunked-style of transfer encoding?
+ tr_enc = self.msg.getheader('transfer-encoding')
+ if tr_enc and tr_enc.lower() == "chunked":
+ self.chunked = 1
+ self.chunk_left = None
+ else:
+ self.chunked = 0
+
+ # will the connection close at the end of the response?
+ self.will_close = self._check_close()
+
+ # do we have a Content-Length?
+ # NOTE: RFC 2616, S4.4, #3 says we ignore this if tr_enc is "chunked"
+ length = self.msg.getheader('content-length')
+ if length and not self.chunked:
+ try:
+ self.length = int(length)
+ except ValueError:
+ self.length = None
+ else:
+ if self.length < 0: # ignore nonsensical negative lengths
+ self.length = None
+ else:
+ self.length = None
+
+ # does the body have a fixed length? (of zero)
+ if (status == NO_CONTENT or status == NOT_MODIFIED or
+ 100 <= status < 200 or # 1xx codes
+ self._method == 'HEAD'):
+ self.length = 0
+
+ # if the connection remains open, and we aren't using chunked, and
+ # a content-length was not provided, then assume that the connection
+ # WILL close.
+ if not self.will_close and \
+ not self.chunked and \
+ self.length is None:
+ self.will_close = 1
+
+ def _check_close(self):
+ conn = self.msg.getheader('connection')
+ if self.version == 11:
+ # An HTTP/1.1 proxy is assumed to stay open unless
+ # explicitly closed.
+ conn = self.msg.getheader('connection')
+ if conn and "close" in conn.lower():
+ return True
+ return False
+
+ # Some HTTP/1.0 implementations have support for persistent
+ # connections, using rules different than HTTP/1.1.
+
+ # For older HTTP, Keep-Alive indicates persistent connection.
+ if self.msg.getheader('keep-alive'):
+ return False
+
+ # At least Akamai returns a "Connection: Keep-Alive" header,
+ # which was supposed to be sent by the client.
+ if conn and "keep-alive" in conn.lower():
+ return False
+
+ # Proxy-Connection is a netscape hack.
+ pconn = self.msg.getheader('proxy-connection')
+ if pconn and "keep-alive" in pconn.lower():
+ return False
+
+ # otherwise, assume it will close
+ return True
+
+ def close(self):
+ if self.fp:
+ self.fp.close()
+ self.fp = None
+
+ def isclosed(self):
+ # NOTE: it is possible that we will not ever call self.close(). This
+ # case occurs when will_close is TRUE, length is None, and we
+ # read up to the last byte, but NOT past it.
+ #
+ # IMPLIES: if will_close is FALSE, then self.close() will ALWAYS be
+ # called, meaning self.isclosed() is meaningful.
+ return self.fp is None
+
+ # XXX It would be nice to have readline and __iter__ for this, too.
+
+ def read(self, amt=None):
+ if self.fp is None:
+ return ''
+
+ if self._method == 'HEAD':
+ self.close()
+ return ''
+
+ if self.chunked:
+ return self._read_chunked(amt)
+
+ if amt is None:
+ # unbounded read
+ if self.length is None:
+ s = self.fp.read()
+ else:
+ s = self._safe_read(self.length)
+ self.length = 0
+ self.close() # we read everything
+ return s
+
+ if self.length is not None:
+ if amt > self.length:
+ # clip the read to the "end of response"
+ amt = self.length
+
+ # we do not use _safe_read() here because this may be a .will_close
+ # connection, and the user is reading more bytes than will be provided
+ # (for example, reading in 1k chunks)
+ s = self.fp.read(amt)
+ if self.length is not None:
+ self.length -= len(s)
+ if not self.length:
+ self.close()
+ return s
+
+ def _read_chunked(self, amt):
+ assert self.chunked != _UNKNOWN
+ chunk_left = self.chunk_left
+ value = []
+ while True:
+ if chunk_left is None:
+ line = self.fp.readline()
+ i = line.find(';')
+ if i >= 0:
+ line = line[:i] # strip chunk-extensions
+ try:
+ chunk_left = int(line, 16)
+ except ValueError:
+ # close the connection as protocol synchronisation is
+ # probably lost
+ self.close()
+ raise IncompleteRead(''.join(value))
+ if chunk_left == 0:
+ break
+ if amt is None:
+ value.append(self._safe_read(chunk_left))
+ elif amt < chunk_left:
+ value.append(self._safe_read(amt))
+ self.chunk_left = chunk_left - amt
+ return ''.join(value)
+ elif amt == chunk_left:
+ value.append(self._safe_read(amt))
+ self._safe_read(2) # toss the CRLF at the end of the chunk
+ self.chunk_left = None
+ return ''.join(value)
+ else:
+ value.append(self._safe_read(chunk_left))
+ amt -= chunk_left
+
+ # we read the whole chunk, get another
+ self._safe_read(2) # toss the CRLF at the end of the chunk
+ chunk_left = None
+
+ # read and discard trailer up to the CRLF terminator
+ ### note: we shouldn't have any trailers!
+ while True:
+ line = self.fp.readline()
+ if not line:
+ # a vanishingly small number of sites EOF without
+ # sending the trailer
+ break
+ if line == '\r\n':
+ break
+
+ # we read everything; close the "file"
+ self.close()
+
+ return ''.join(value)
+
+ def _safe_read(self, amt):
+ """Read the number of bytes requested, compensating for partial reads.
+
+ Normally, we have a blocking socket, but a read() can be interrupted
+ by a signal (resulting in a partial read).
+
+ Note that we cannot distinguish between EOF and an interrupt when zero
+ bytes have been read. IncompleteRead() will be raised in this
+ situation.
+
+ This function should be used when <amt> bytes "should" be present for
+ reading. If the bytes are truly not available (due to EOF), then the
+ IncompleteRead exception can be used to detect the problem.
+ """
+ # NOTE(gps): As of svn r74426 socket._fileobject.read(x) will never
+ # return less than x bytes unless EOF is encountered. It now handles
+ # signal interruptions (socket.error EINTR) internally. This code
+ # never caught that exception anyways. It seems largely pointless.
+ # self.fp.read(amt) will work fine.
+ s = []
+ while amt > 0:
+ chunk = self.fp.read(min(amt, MAXAMOUNT))
+ if not chunk:
+ raise IncompleteRead(''.join(s), amt)
+ s.append(chunk)
+ amt -= len(chunk)
+ return ''.join(s)
+
+ def fileno(self):
+ return self.fp.fileno()
+
+ def getheader(self, name, default=None):
+ if self.msg is None:
+ raise ResponseNotReady()
+ return self.msg.getheader(name, default)
+
+ def getheaders(self):
+ """Return list of (header, value) tuples."""
+ if self.msg is None:
+ raise ResponseNotReady()
+ return self.msg.items()
+
+
+class HTTPConnection:
+
+ _http_vsn = 11
+ _http_vsn_str = 'HTTP/1.1'
+
+ response_class = HTTPResponse
+ default_port = HTTP_PORT
+ auto_open = 1
+ debuglevel = 0
+ strict = 0
+
+ def __init__(self, host, port=None, strict=None,
+ timeout=socket._GLOBAL_DEFAULT_TIMEOUT, source_address=None):
+ self.timeout = timeout
+ self.source_address = source_address
+ self.sock = None
+ self._buffer = []
+ self.__response = None
+ self.__state = _CS_IDLE
+ self._method = None
+ self._tunnel_host = None
+ self._tunnel_port = None
+ self._tunnel_headers = {}
+
+ self._set_hostport(host, port)
+ if strict is not None:
+ self.strict = strict
+
+ def set_tunnel(self, host, port=None, headers=None):
+ """ Sets up the host and the port for the HTTP CONNECT Tunnelling.
+
+ The headers argument should be a mapping of extra HTTP headers
+ to send with the CONNECT request.
+ """
+ self._tunnel_host = host
+ self._tunnel_port = port
+ if headers:
+ self._tunnel_headers = headers
+ else:
+ self._tunnel_headers.clear()
+
+ def _set_hostport(self, host, port):
+ if port is None:
+ i = host.rfind(':')
+ j = host.rfind(']') # ipv6 addresses have [...]
+ if i > j:
+ try:
+ port = int(host[i+1:])
+ except ValueError:
+ raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
+ host = host[:i]
+ else:
+ port = self.default_port
+ if host and host[0] == '[' and host[-1] == ']':
+ host = host[1:-1]
+ self.host = host
+ self.port = port
+
+ def set_debuglevel(self, level):
+ self.debuglevel = level
+
+ def _tunnel(self):
+ self._set_hostport(self._tunnel_host, self._tunnel_port)
+ self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self.host, self.port))
+ for header, value in self._tunnel_headers.iteritems():
+ self.send("%s: %s\r\n" % (header, value))
+ self.send("\r\n")
+ response = self.response_class(self.sock, strict = self.strict,
+ method = self._method)
+ (version, code, message) = response._read_status()
+
+ if code != 200:
+ self.close()
+ raise socket.error("Tunnel connection failed: %d %s" % (code,
+ message.strip()))
+ while True:
+ line = response.fp.readline()
+ if line == '\r\n': break
+
+
+ def connect(self):
+ """Connect to the host and port specified in __init__."""
+ self.sock = socket.create_connection((self.host,self.port),
+ self.timeout, self.source_address)
+
+ if self._tunnel_host:
+ self._tunnel()
+
+ def close(self):
+ """Close the connection to the HTTP server."""
+ if self.sock:
+ self.sock.close() # close it manually... there may be other refs
+ self.sock = None
+ if self.__response:
+ self.__response.close()
+ self.__response = None
+ self.__state = _CS_IDLE
+
+ def send(self, data):
+ """Send `data' to the server."""
+ if self.sock is None:
+ if self.auto_open:
+ self.connect()
+ else:
+ raise NotConnected()
+
+ if self.debuglevel > 0:
+ print "send:", repr(data)
+ blocksize = 8192
+ if hasattr(data,'read') and not isinstance(data, array):
+ if self.debuglevel > 0: print "sendIng a read()able"
+ datablock = data.read(blocksize)
+ while datablock:
+ self.sock.sendall(datablock)
+ datablock = data.read(blocksize)
+ else:
+ self.sock.sendall(data)
+
+ def _output(self, s):
+ """Add a line of output to the current request buffer.
+
+ Assumes that the line does *not* end with \\r\\n.
+ """
+ self._buffer.append(s)
+
+ def _send_output(self, message_body=None):
+ """Send the currently buffered request and clear the buffer.
+
+ Appends an extra \\r\\n to the buffer.
+ A message_body may be specified, to be appended to the request.
+ """
+ self._buffer.extend(("", ""))
+ msg = "\r\n".join(self._buffer)
+ del self._buffer[:]
+ # If msg and message_body are sent in a single send() call,
+ # it will avoid performance problems caused by the interaction
+ # between delayed ack and the Nagle algorithim.
+ if isinstance(message_body, str):
+ msg += message_body
+ message_body = None
+ self.send(msg)
+ if message_body is not None:
+ #message_body was not a string (i.e. it is a file) and
+ #we must run the risk of Nagle
+ self.send(message_body)
+
+ def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
+ """Send a request to the server.
+
+ `method' specifies an HTTP request method, e.g. 'GET'.
+ `url' specifies the object being requested, e.g. '/index.html'.
+ `skip_host' if True does not add automatically a 'Host:' header
+ `skip_accept_encoding' if True does not add automatically an
+ 'Accept-Encoding:' header
+ """
+
+ # if a prior response has been completed, then forget about it.
+ if self.__response and self.__response.isclosed():
+ self.__response = None
+
+
+ # in certain cases, we cannot issue another request on this connection.
+ # this occurs when:
+ # 1) we are in the process of sending a request. (_CS_REQ_STARTED)
+ # 2) a response to a previous request has signalled that it is going
+ # to close the connection upon completion.
+ # 3) the headers for the previous response have not been read, thus
+ # we cannot determine whether point (2) is true. (_CS_REQ_SENT)
+ #
+ # if there is no prior response, then we can request at will.
+ #
+ # if point (2) is true, then we will have passed the socket to the
+ # response (effectively meaning, "there is no prior response"), and
+ # will open a new one when a new request is made.
+ #
+ # Note: if a prior response exists, then we *can* start a new request.
+ # We are not allowed to begin fetching the response to this new
+ # request, however, until that prior response is complete.
+ #
+ if self.__state == _CS_IDLE:
+ self.__state = _CS_REQ_STARTED
+ else:
+ raise CannotSendRequest()
+
+ # Save the method we use, we need it later in the response phase
+ self._method = method
+ if not url:
+ url = '/'
+ hdr = '%s %s %s' % (method, url, self._http_vsn_str)
+
+ self._output(hdr)
+
+ if self._http_vsn == 11:
+ # Issue some standard headers for better HTTP/1.1 compliance
+
+ if not skip_host:
+ # this header is issued *only* for HTTP/1.1
+ # connections. more specifically, this means it is
+ # only issued when the client uses the new
+ # HTTPConnection() class. backwards-compat clients
+ # will be using HTTP/1.0 and those clients may be
+ # issuing this header themselves. we should NOT issue
+ # it twice; some web servers (such as Apache) barf
+ # when they see two Host: headers
+
+ # If we need a non-standard port,include it in the
+ # header. If the request is going through a proxy,
+ # but the host of the actual URL, not the host of the
+ # proxy.
+
+ netloc = ''
+ if url.startswith('http'):
+ nil, netloc, nil, nil, nil = urlsplit(url)
+
+ if netloc:
+ try:
+ netloc_enc = netloc.encode("ascii")
+ except UnicodeEncodeError:
+ netloc_enc = netloc.encode("idna")
+ self.putheader('Host', netloc_enc)
+ else:
+ try:
+ host_enc = self.host.encode("ascii")
+ except UnicodeEncodeError:
+ host_enc = self.host.encode("idna")
+ # Wrap the IPv6 Host Header with [] (RFC 2732)
+ if host_enc.find(':') >= 0:
+ host_enc = "[" + host_enc + "]"
+ if self.port == self.default_port:
+ self.putheader('Host', host_enc)
+ else:
+ self.putheader('Host', "%s:%s" % (host_enc, self.port))
+
+ # note: we are assuming that clients will not attempt to set these
+ # headers since *this* library must deal with the
+ # consequences. this also means that when the supporting
+ # libraries are updated to recognize other forms, then this
+ # code should be changed (removed or updated).
+
+ # we only want a Content-Encoding of "identity" since we don't
+ # support encodings such as x-gzip or x-deflate.
+ if not skip_accept_encoding:
+ self.putheader('Accept-Encoding', 'identity')
+
+ # we can accept "chunked" Transfer-Encodings, but no others
+ # NOTE: no TE header implies *only* "chunked"
+ #self.putheader('TE', 'chunked')
+
+ # if TE is supplied in the header, then it must appear in a
+ # Connection header.
+ #self.putheader('Connection', 'TE')
+
+ else:
+ # For HTTP/1.0, the server will assume "not chunked"
+ pass
+
+ def putheader(self, header, *values):
+ """Send a request header line to the server.
+
+ For example: h.putheader('Accept', 'text/html')
+ """
+ if self.__state != _CS_REQ_STARTED:
+ raise CannotSendHeader()
+
+ hdr = '%s: %s' % (header, '\r\n\t'.join([str(v) for v in values]))
+ self._output(hdr)
+
+ def endheaders(self, message_body=None):
+ """Indicate that the last header line has been sent to the server.
+
+ This method sends the request to the server. The optional
+ message_body argument can be used to pass message body
+ associated with the request. The message body will be sent in
+ the same packet as the message headers if possible. The
+ message_body should be a string.
+ """
+ if self.__state == _CS_REQ_STARTED:
+ self.__state = _CS_REQ_SENT
+ else:
+ raise CannotSendHeader()
+ self._send_output(message_body)
+
+ def request(self, method, url, body=None, headers={}):
+ """Send a complete request to the server."""
+ self._send_request(method, url, body, headers)
+
+ def _set_content_length(self, body):
+ # Set the content-length based on the body.
+ thelen = None
+ try:
+ thelen = str(len(body))
+ except TypeError, te:
+ # If this is a file-like object, try to
+ # fstat its file descriptor
+ try:
+ thelen = str(os.fstat(body.fileno()).st_size)
+ except (AttributeError, OSError):
+ # Don't send a length if this failed
+ if self.debuglevel > 0: print "Cannot stat!!"
+
+ if thelen is not None:
+ self.putheader('Content-Length', thelen)
+
+ def _send_request(self, method, url, body, headers):
+ # Honor explicitly requested Host: and Accept-Encoding: headers.
+ header_names = dict.fromkeys([k.lower() for k in headers])
+ skips = {}
+ if 'host' in header_names:
+ skips['skip_host'] = 1
+ if 'accept-encoding' in header_names:
+ skips['skip_accept_encoding'] = 1
+
+ self.putrequest(method, url, **skips)
+
+ if body and ('content-length' not in header_names):
+ self._set_content_length(body)
+ for hdr, value in headers.iteritems():
+ self.putheader(hdr, value)
+ self.endheaders(body)
+
+ def getresponse(self, buffering=False):
+ "Get the response from the server."
+
+ # if a prior response has been completed, then forget about it.
+ if self.__response and self.__response.isclosed():
+ self.__response = None
+
+ #
+ # if a prior response exists, then it must be completed (otherwise, we
+ # cannot read this response's header to determine the connection-close
+ # behavior)
+ #
+ # note: if a prior response existed, but was connection-close, then the
+ # socket and response were made independent of this HTTPConnection
+ # object since a new request requires that we open a whole new
+ # connection
+ #
+ # this means the prior response had one of two states:
+ # 1) will_close: this connection was reset and the prior socket and
+ # response operate independently
+ # 2) persistent: the response was retained and we await its
+ # isclosed() status to become true.
+ #
+ if self.__state != _CS_REQ_SENT or self.__response:
+ raise ResponseNotReady()
+
+ args = (self.sock,)
+ kwds = {"strict":self.strict, "method":self._method}
+ if self.debuglevel > 0:
+ args += (self.debuglevel,)
+ if buffering:
+ #only add this keyword if non-default, for compatibility with
+ #other response_classes.
+ kwds["buffering"] = True;
+ response = self.response_class(*args, **kwds)
+
+ try:
+ response.begin()
+ except:
+ response.close()
+ raise
+ assert response.will_close != _UNKNOWN
+ self.__state = _CS_IDLE
+
+ if response.will_close:
+ # this effectively passes the connection to the response
+ self.close()
+ else:
+ # remember this, so we can tell when it is complete
+ self.__response = response
+
+ return response
+
+
+class HTTP:
+ "Compatibility class with httplib.py from 1.5."
+
+ _http_vsn = 10
+ _http_vsn_str = 'HTTP/1.0'
+
+ debuglevel = 0
+
+ _connection_class = HTTPConnection
+
+ def __init__(self, host='', port=None, strict=None):
+ "Provide a default host, since the superclass requires one."
+
+ # some joker passed 0 explicitly, meaning default port
+ if port == 0:
+ port = None
+
+ # Note that we may pass an empty string as the host; this will throw
+ # an error when we attempt to connect. Presumably, the client code
+ # will call connect before then, with a proper host.
+ self._setup(self._connection_class(host, port, strict))
+
+ def _setup(self, conn):
+ self._conn = conn
+
+ # set up delegation to flesh out interface
+ self.send = conn.send
+ self.putrequest = conn.putrequest
+ self.putheader = conn.putheader
+ self.endheaders = conn.endheaders
+ self.set_debuglevel = conn.set_debuglevel
+
+ conn._http_vsn = self._http_vsn
+ conn._http_vsn_str = self._http_vsn_str
+
+ self.file = None
+
+ def connect(self, host=None, port=None):
+ "Accept arguments to set the host/port, since the superclass doesn't."
+
+ if host is not None:
+ self._conn._set_hostport(host, port)
+ self._conn.connect()
+
+ def getfile(self):
+ "Provide a getfile, since the superclass' does not use this concept."
+ return self.file
+
+ def getreply(self, buffering=False):
+ """Compat definition since superclass does not define it.
+
+ Returns a tuple consisting of:
+ - server status code (e.g. '200' if all goes well)
+ - server "reason" corresponding to status code
+ - any RFC822 headers in the response from the server
+ """
+ try:
+ if not buffering:
+ response = self._conn.getresponse()
+ else:
+ #only add this keyword if non-default for compatibility
+ #with other connection classes
+ response = self._conn.getresponse(buffering)
+ except BadStatusLine, e:
+ ### hmm. if getresponse() ever closes the socket on a bad request,
+ ### then we are going to have problems with self.sock
+
+ ### should we keep this behavior? do people use it?
+ # keep the socket open (as a file), and return it
+ self.file = self._conn.sock.makefile('rb', 0)
+
+ # close our socket -- we want to restart after any protocol error
+ self.close()
+
+ self.headers = None
+ return -1, e.line, None
+
+ self.headers = response.msg
+ self.file = response.fp
+ return response.status, response.reason, response.msg
+
+ def close(self):
+ self._conn.close()
+
+ # note that self.file == response.fp, which gets closed by the
+ # superclass. just clear the object ref here.
+ ### hmm. messy. if status==-1, then self.file is owned by us.
+ ### well... we aren't explicitly closing, but losing this ref will
+ ### do it
+ self.file = None
+
+try:
+ import ssl
+except ImportError:
+ pass
+else:
+ class HTTPSConnection(HTTPConnection):
+ "This class allows communication via SSL."
+
+ default_port = HTTPS_PORT
+
+ def __init__(self, host, port=None, key_file=None, cert_file=None,
+ strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
+ source_address=None):
+ HTTPConnection.__init__(self, host, port, strict, timeout,
+ source_address)
+ self.key_file = key_file
+ self.cert_file = cert_file
+
+ def connect(self):
+ "Connect to a host on a given (SSL) port."
+
+ sock = socket.create_connection((self.host, self.port),
+ self.timeout, self.source_address)
+ if self._tunnel_host:
+ self.sock = sock
+ self._tunnel()
+ self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
+
+ __all__.append("HTTPSConnection")
+
+ class HTTPS(HTTP):
+ """Compatibility with 1.5 httplib interface
+
+ Python 1.5.2 did not have an HTTPS class, but it defined an
+ interface for sending http requests that is also useful for
+ https.
+ """
+
+ _connection_class = HTTPSConnection
+
+ def __init__(self, host='', port=None, key_file=None, cert_file=None,
+ strict=None):
+ # provide a default host, pass the X509 cert info
+
+ # urf. compensate for bad input.
+ if port == 0:
+ port = None
+ self._setup(self._connection_class(host, port, key_file,
+ cert_file, strict))
+
+ # we never actually use these for anything, but we keep them
+ # here for compatibility with post-1.5.2 CVS.
+ self.key_file = key_file
+ self.cert_file = cert_file
+
+
+ def FakeSocket (sock, sslobj):
+ warnings.warn("FakeSocket is deprecated, and won't be in 3.x. " +
+ "Use the result of ssl.wrap_socket() directly instead.",
+ DeprecationWarning, stacklevel=2)
+ return sslobj
+
+
+class HTTPException(Exception):
+ # Subclasses that define an __init__ must call Exception.__init__
+ # or define self.args. Otherwise, str() will fail.
+ pass
+
+class NotConnected(HTTPException):
+ pass
+
+class InvalidURL(HTTPException):
+ pass
+
+class UnknownProtocol(HTTPException):
+ def __init__(self, version):
+ self.args = version,
+ self.version = version
+
+class UnknownTransferEncoding(HTTPException):
+ pass
+
+class UnimplementedFileMode(HTTPException):
+ pass
+
+class IncompleteRead(HTTPException):
+ def __init__(self, partial, expected=None):
+ self.args = partial,
+ self.partial = partial
+ self.expected = expected
+ def __repr__(self):
+ if self.expected is not None:
+ e = ', %i more expected' % self.expected
+ else:
+ e = ''
+ return 'IncompleteRead(%i bytes read%s)' % (len(self.partial), e)
+ def __str__(self):
+ return repr(self)
+
+class ImproperConnectionState(HTTPException):
+ pass
+
+class CannotSendRequest(ImproperConnectionState):
+ pass
+
+class CannotSendHeader(ImproperConnectionState):
+ pass
+
+class ResponseNotReady(ImproperConnectionState):
+ pass
+
+class BadStatusLine(HTTPException):
+ def __init__(self, line):
+ if not line:
+ line = repr(line)
+ self.args = line,
+ self.line = line
+
+# for backwards compatibility
+error = HTTPException
+
+class LineAndFileWrapper:
+ """A limited file-like object for HTTP/0.9 responses."""
+
+ # The status-line parsing code calls readline(), which normally
+ # get the HTTP status line. For a 0.9 response, however, this is
+ # actually the first line of the body! Clients need to get a
+ # readable file object that contains that line.
+
+ def __init__(self, line, file):
+ self._line = line
+ self._file = file
+ self._line_consumed = 0
+ self._line_offset = 0
+ self._line_left = len(line)
+
+ def __getattr__(self, attr):
+ return getattr(self._file, attr)
+
+ def _done(self):
+ # called when the last byte is read from the line. After the
+ # call, all read methods are delegated to the underlying file
+ # object.
+ self._line_consumed = 1
+ self.read = self._file.read
+ self.readline = self._file.readline
+ self.readlines = self._file.readlines
+
+ def read(self, amt=None):
+ if self._line_consumed:
+ return self._file.read(amt)
+ assert self._line_left
+ if amt is None or amt > self._line_left:
+ s = self._line[self._line_offset:]
+ self._done()
+ if amt is None:
+ return s + self._file.read()
+ else:
+ return s + self._file.read(amt - len(s))
+ else:
+ assert amt <= self._line_left
+ i = self._line_offset
+ j = i + amt
+ s = self._line[i:j]
+ self._line_offset = j
+ self._line_left -= amt
+ if self._line_left == 0:
+ self._done()
+ return s
+
+ def readline(self):
+ if self._line_consumed:
+ return self._file.readline()
+ assert self._line_left
+ s = self._line[self._line_offset:]
+ self._done()
+ return s
+
+ def readlines(self, size=None):
+ if self._line_consumed:
+ return self._file.readlines(size)
+ assert self._line_left
+ L = [self._line[self._line_offset:]]
+ self._done()
+ if size is None:
+ return L + self._file.readlines()
+ else:
+ return L + self._file.readlines(size)
+
+def test():
+ """Test this module.
+
+ A hodge podge of tests collected here, because they have too many
+ external dependencies for the regular test suite.
+ """
+
+ import sys
+ import getopt
+ opts, args = getopt.getopt(sys.argv[1:], 'd')
+ dl = 0
+ for o, a in opts:
+ if o == '-d': dl = dl + 1
+ host = 'www.python.org'
+ selector = '/'
+ if args[0:]: host = args[0]
+ if args[1:]: selector = args[1]
+ h = HTTP()
+ h.set_debuglevel(dl)
+ h.connect(host)
+ h.putrequest('GET', selector)
+ h.endheaders()
+ status, reason, headers = h.getreply()
+ print 'status =', status
+ print 'reason =', reason
+ print "read", len(h.getfile().read())
+ print
+ if headers:
+ for header in headers.headers: print header.strip()
+ print
+
+ # minimal test that code to extract host from url works
+ class HTTP11(HTTP):
+ _http_vsn = 11
+ _http_vsn_str = 'HTTP/1.1'
+
+ h = HTTP11('www.python.org')
+ h.putrequest('GET', 'http://www.python.org/~jeremy/')
+ h.endheaders()
+ h.getreply()
+ h.close()
+
+ try:
+ import ssl
+ except ImportError:
+ pass
+ else:
+
+ for host, selector in (('sourceforge.net', '/projects/python'),
+ ):
+ print "https://%s%s" % (host, selector)
+ hs = HTTPS()
+ hs.set_debuglevel(dl)
+ hs.connect(host)
+ hs.putrequest('GET', selector)
+ hs.endheaders()
+ status, reason, headers = hs.getreply()
+ print 'status =', status
+ print 'reason =', reason
+ print "read", len(hs.getfile().read())
+ print
+ if headers:
+ for header in headers.headers: print header.strip()
+ print
+
+if __name__ == '__main__':
+ test()
diff --git a/lib-python/modified-2.7/json/encoder.py b/lib-python/modified-2.7/json/encoder.py
--- a/lib-python/modified-2.7/json/encoder.py
+++ b/lib-python/modified-2.7/json/encoder.py
@@ -2,14 +2,7 @@
"""
import re
-try:
- from _json import encode_basestring_ascii as c_encode_basestring_ascii
-except ImportError:
- c_encode_basestring_ascii = None
-try:
- from _json import make_encoder as c_make_encoder
-except ImportError:
- c_make_encoder = None
+from __pypy__.builders import StringBuilder, UnicodeBuilder
ESCAPE = re.compile(r'[\x00-\x1f\\"\b\f\n\r\t]')
ESCAPE_ASCII = re.compile(r'([\\"]|[^\ -~])')
@@ -24,8 +17,7 @@
'\t': '\\t',
}
for i in range(0x20):
- ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i))
- #ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,))
+ ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,))
# Assume this produces an infinity on all machines (probably not guaranteed)
INFINITY = float('1e66666')
@@ -37,10 +29,9 @@
"""
def replace(match):
return ESCAPE_DCT[match.group(0)]
- return '"' + ESCAPE.sub(replace, s) + '"'
+ return ESCAPE.sub(replace, s)
-
-def py_encode_basestring_ascii(s):
+def encode_basestring_ascii(s):
"""Return an ASCII-only JSON representation of a Python string
"""
@@ -53,20 +44,18 @@
except KeyError:
n = ord(s)
if n < 0x10000:
- return '\\u{0:04x}'.format(n)
- #return '\\u%04x' % (n,)
+ return '\\u%04x' % (n,)
else:
# surrogate pair
n -= 0x10000
s1 = 0xd800 | ((n >> 10) & 0x3ff)
s2 = 0xdc00 | (n & 0x3ff)
- return '\\u{0:04x}\\u{1:04x}'.format(s1, s2)
- #return '\\u%04x\\u%04x' % (s1, s2)
- return '"' + str(ESCAPE_ASCII.sub(replace, s)) + '"'
-
-
-encode_basestring_ascii = (
- c_encode_basestring_ascii or py_encode_basestring_ascii)
+ return '\\u%04x\\u%04x' % (s1, s2)
+ if ESCAPE_ASCII.search(s):
+ return str(ESCAPE_ASCII.sub(replace, s))
+ return s
+py_encode_basestring_ascii = lambda s: '"' + encode_basestring_ascii(s) + '"'
+c_encode_basestring_ascii = None
class JSONEncoder(object):
"""Extensible JSON <http://json.org> encoder for Python data structures.
@@ -147,6 +136,17 @@
self.skipkeys = skipkeys
self.ensure_ascii = ensure_ascii
+ if ensure_ascii:
+ self.encoder = encode_basestring_ascii
+ else:
+ self.encoder = encode_basestring
+ if encoding != 'utf-8':
+ orig_encoder = self.encoder
+ def encoder(o):
+ if isinstance(o, str):
+ o = o.decode(encoding)
+ return orig_encoder(o)
+ self.encoder = encoder
self.check_circular = check_circular
self.allow_nan = allow_nan
self.sort_keys = sort_keys
@@ -184,24 +184,126 @@
'{"foo": ["bar", "baz"]}'
"""
- # This is for extremely simple cases and benchmarks.
+ if self.check_circular:
+ markers = {}
+ else:
+ markers = None
+ if self.ensure_ascii:
+ builder = StringBuilder()
+ else:
+ builder = UnicodeBuilder()
+ self._encode(o, markers, builder, 0)
+ return builder.build()
+
+ def _emit_indent(self, builder, _current_indent_level):
+ if self.indent is not None:
+ _current_indent_level += 1
+ newline_indent = '\n' + (' ' * (self.indent *
+ _current_indent_level))
+ separator = self.item_separator + newline_indent
+ builder.append(newline_indent)
+ else:
+ separator = self.item_separator
+ return separator, _current_indent_level
+
+ def _emit_unindent(self, builder, _current_indent_level):
+ if self.indent is not None:
+ builder.append('\n')
+ builder.append(' ' * (self.indent * (_current_indent_level - 1)))
+
+ def _encode(self, o, markers, builder, _current_indent_level):
if isinstance(o, basestring):
- if isinstance(o, str):
- _encoding = self.encoding
- if (_encoding is not None
- and not (_encoding == 'utf-8')):
- o = o.decode(_encoding)
- if self.ensure_ascii:
- return encode_basestring_ascii(o)
+ builder.append('"')
+ builder.append(self.encoder(o))
+ builder.append('"')
+ elif o is None:
+ builder.append('null')
+ elif o is True:
+ builder.append('true')
+ elif o is False:
+ builder.append('false')
+ elif isinstance(o, (int, long)):
+ builder.append(str(o))
+ elif isinstance(o, float):
+ builder.append(self._floatstr(o))
+ elif isinstance(o, (list, tuple)):
+ if not o:
+ builder.append('[]')
+ return
+ self._encode_list(o, markers, builder, _current_indent_level)
+ elif isinstance(o, dict):
+ if not o:
+ builder.append('{}')
+ return
+ self._encode_dict(o, markers, builder, _current_indent_level)
+ else:
+ self._mark_markers(markers, o)
+ res = self.default(o)
+ self._encode(res, markers, builder, _current_indent_level)
+ self._remove_markers(markers, o)
+ return res
+
+ def _encode_list(self, l, markers, builder, _current_indent_level):
+ self._mark_markers(markers, l)
+ builder.append('[')
+ first = True
+ separator, _current_indent_level = self._emit_indent(builder,
+ _current_indent_level)
+ for elem in l:
+ if first:
+ first = False
else:
- return encode_basestring(o)
- # This doesn't pass the iterator directly to ''.join() because the
- # exceptions aren't as detailed. The list call should be roughly
- # equivalent to the PySequence_Fast that ''.join() would do.
- chunks = self.iterencode(o, _one_shot=True)
- if not isinstance(chunks, (list, tuple)):
- chunks = list(chunks)
- return ''.join(chunks)
+ builder.append(separator)
+ self._encode(elem, markers, builder, _current_indent_level)
+ del elem # XXX grumble
+ self._emit_unindent(builder, _current_indent_level)
+ builder.append(']')
+ self._remove_markers(markers, l)
+
+ def _encode_dict(self, d, markers, builder, _current_indent_level):
+ self._mark_markers(markers, d)
+ first = True
+ builder.append('{')
+ separator, _current_indent_level = self._emit_indent(builder,
+ _current_indent_level)
+ if self.sort_keys:
+ items = sorted(d.items(), key=lambda kv: kv[0])
+ else:
+ items = d.iteritems()
+
+ for key, v in items:
+ if first:
+ first = False
+ else:
+ builder.append(separator)
+ if isinstance(key, basestring):
+ pass
+ # JavaScript is weakly typed for these, so it makes sense to
+ # also allow them. Many encoders seem to do something like this.
+ elif isinstance(key, float):
+ key = self._floatstr(key)
+ elif key is True:
+ key = 'true'
+ elif key is False:
+ key = 'false'
+ elif key is None:
+ key = 'null'
+ elif isinstance(key, (int, long)):
+ key = str(key)
+ elif self.skipkeys:
+ continue
+ else:
+ raise TypeError("key " + repr(key) + " is not a string")
+ builder.append('"')
+ builder.append(self.encoder(key))
+ builder.append('"')
+ builder.append(self.key_separator)
+ self._encode(v, markers, builder, _current_indent_level)
+ del key
+ del v # XXX grumble
+ self._emit_unindent(builder, _current_indent_level)
+ builder.append('}')
+ self._remove_markers(markers, d)
def iterencode(self, o, _one_shot=False):
"""Encode the given object and yield each string
@@ -217,86 +319,54 @@
markers = {}
else:
markers = None
- if self.ensure_ascii:
- _encoder = encode_basestring_ascii
+ return self._iterencode(o, markers, 0)
+
+ def _floatstr(self, o):
+ # Check for specials. Note that this type of test is processor
+ # and/or platform-specific, so do tests which don't depend on the
+ # internals.
+
+ if o != o:
+ text = 'NaN'
+ elif o == INFINITY:
+ text = 'Infinity'
+ elif o == -INFINITY:
+ text = '-Infinity'
else:
- _encoder = encode_basestring
- if self.encoding != 'utf-8':
- def _encoder(o, _orig_encoder=_encoder, _encoding=self.encoding):
- if isinstance(o, str):
- o = o.decode(_encoding)
- return _orig_encoder(o)
+ return FLOAT_REPR(o)
- def floatstr(o, allow_nan=self.allow_nan,
- _repr=FLOAT_REPR, _inf=INFINITY, _neginf=-INFINITY):
- # Check for specials. Note that this type of test is processor
- # and/or platform-specific, so do tests which don't depend on the
- # internals.
+ if not self.allow_nan:
+ raise ValueError(
+ "Out of range float values are not JSON compliant: " +
+ repr(o))
- if o != o:
- text = 'NaN'
- elif o == _inf:
- text = 'Infinity'
- elif o == _neginf:
- text = '-Infinity'
- else:
- return _repr(o)
+ return text
- if not allow_nan:
- raise ValueError(
- "Out of range float values are not JSON compliant: " +
- repr(o))
+ def _mark_markers(self, markers, o):
+ if markers is not None:
+ if id(o) in markers:
+ raise ValueError("Circular reference detected")
+ markers[id(o)] = None
- return text
+ def _remove_markers(self, markers, o):
+ if markers is not None:
+ del markers[id(o)]
-
- if (_one_shot and c_make_encoder is not None
- and not self.indent and not self.sort_keys):
- _iterencode = c_make_encoder(
- markers, self.default, _encoder, self.indent,
- self.key_separator, self.item_separator, self.sort_keys,
- self.skipkeys, self.allow_nan)
- else:
- _iterencode = _make_iterencode(
- markers, self.default, _encoder, self.indent, floatstr,
- self.key_separator, self.item_separator, self.sort_keys,
- self.skipkeys, _one_shot)
- return _iterencode(o, 0)
-
-def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,
- _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
- ## HACK: hand-optimized bytecode; turn globals into locals
- ValueError=ValueError,
- basestring=basestring,
- dict=dict,
- float=float,
- id=id,
- int=int,
- isinstance=isinstance,
- list=list,
- long=long,
- str=str,
- tuple=tuple,
- ):
-
- def _iterencode_list(lst, _current_indent_level):
+ def _iterencode_list(self, lst, markers, _current_indent_level):
if not lst:
yield '[]'
return
- if markers is not None:
- markerid = id(lst)
- if markerid in markers:
- raise ValueError("Circular reference detected")
- markers[markerid] = lst
+ self._mark_markers(markers, lst)
buf = '['
- if _indent is not None:
+ if self.indent is not None:
_current_indent_level += 1
- newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
- separator = _item_separator + newline_indent
+ newline_indent = '\n' + (' ' * (self.indent *
+ _current_indent_level))
+ separator = self.item_separator + newline_indent
buf += newline_indent
else:
newline_indent = None
- separator = _item_separator
+ separator = self.item_separator
first = True
for value in lst:
if first:
@@ -304,7 +374,7 @@
else:
buf = separator
if isinstance(value, basestring):
- yield buf + _encoder(value)
+ yield buf + '"' + self.encoder(value) + '"'
elif value is None:
yield buf + 'null'
elif value is True:
@@ -314,44 +384,43 @@
elif isinstance(value, (int, long)):
yield buf + str(value)
elif isinstance(value, float):
- yield buf + _floatstr(value)
+ yield buf + self._floatstr(value)
else:
yield buf
if isinstance(value, (list, tuple)):
- chunks = _iterencode_list(value, _current_indent_level)
+ chunks = self._iterencode_list(value, markers,
+ _current_indent_level)
elif isinstance(value, dict):
- chunks = _iterencode_dict(value, _current_indent_level)
+ chunks = self._iterencode_dict(value, markers,
+ _current_indent_level)
else:
- chunks = _iterencode(value, _current_indent_level)
+ chunks = self._iterencode(value, markers,
+ _current_indent_level)
for chunk in chunks:
yield chunk
if newline_indent is not None:
_current_indent_level -= 1
- yield '\n' + (' ' * (_indent * _current_indent_level))
+ yield '\n' + (' ' * (self.indent * _current_indent_level))
yield ']'
- if markers is not None:
- del markers[markerid]
+ self._remove_markers(markers, lst)
- def _iterencode_dict(dct, _current_indent_level):
+ def _iterencode_dict(self, dct, markers, _current_indent_level):
if not dct:
yield '{}'
return
- if markers is not None:
- markerid = id(dct)
- if markerid in markers:
- raise ValueError("Circular reference detected")
- markers[markerid] = dct
+ self._mark_markers(markers, dct)
yield '{'
- if _indent is not None:
+ if self.indent is not None:
_current_indent_level += 1
- newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
- item_separator = _item_separator + newline_indent
+ newline_indent = '\n' + (' ' * (self.indent *
+ _current_indent_level))
+ item_separator = self.item_separator + newline_indent
yield newline_indent
else:
newline_indent = None
- item_separator = _item_separator
+ item_separator = self.item_separator
first = True
- if _sort_keys:
+ if self.sort_keys:
items = sorted(dct.items(), key=lambda kv: kv[0])
else:
items = dct.iteritems()
@@ -361,7 +430,7 @@
# JavaScript is weakly typed for these, so it makes sense to
# also allow them. Many encoders seem to do something like this.
elif isinstance(key, float):
- key = _floatstr(key)
+ key = self._floatstr(key)
elif key is True:
key = 'true'
elif key is False:
@@ -370,7 +439,7 @@
key = 'null'
elif isinstance(key, (int, long)):
key = str(key)
- elif _skipkeys:
+ elif self.skipkeys:
continue
else:
raise TypeError("key " + repr(key) + " is not a string")
@@ -378,10 +447,10 @@
first = False
else:
yield item_separator
- yield _encoder(key)
- yield _key_separator
+ yield '"' + self.encoder(key) + '"'
+ yield self.key_separator
if isinstance(value, basestring):
- yield _encoder(value)
+ yield '"' + self.encoder(value) + '"'
elif value is None:
yield 'null'
elif value is True:
@@ -391,26 +460,28 @@
elif isinstance(value, (int, long)):
yield str(value)
elif isinstance(value, float):
- yield _floatstr(value)
+ yield self._floatstr(value)
else:
if isinstance(value, (list, tuple)):
- chunks = _iterencode_list(value, _current_indent_level)
+ chunks = self._iterencode_list(value, markers,
+ _current_indent_level)
elif isinstance(value, dict):
- chunks = _iterencode_dict(value, _current_indent_level)
+ chunks = self._iterencode_dict(value, markers,
+ _current_indent_level)
else:
- chunks = _iterencode(value, _current_indent_level)
+ chunks = self._iterencode(value, markers,
+ _current_indent_level)
for chunk in chunks:
yield chunk
if newline_indent is not None:
_current_indent_level -= 1
- yield '\n' + (' ' * (_indent * _current_indent_level))
+ yield '\n' + (' ' * (self.indent * _current_indent_level))
yield '}'
- if markers is not None:
- del markers[markerid]
+ self._remove_markers(markers, dct)
- def _iterencode(o, _current_indent_level):
+ def _iterencode(self, o, markers, _current_indent_level):
if isinstance(o, basestring):
- yield _encoder(o)
+ yield '"' + self.encoder(o) + '"'
elif o is None:
yield 'null'
elif o is True:
@@ -420,23 +491,19 @@
elif isinstance(o, (int, long)):
yield str(o)
elif isinstance(o, float):
- yield _floatstr(o)
+ yield self._floatstr(o)
elif isinstance(o, (list, tuple)):
- for chunk in _iterencode_list(o, _current_indent_level):
+ for chunk in self._iterencode_list(o, markers,
+ _current_indent_level):
yield chunk
elif isinstance(o, dict):
- for chunk in _iterencode_dict(o, _current_indent_level):
+ for chunk in self._iterencode_dict(o, markers,
+ _current_indent_level):
yield chunk
else:
- if markers is not None:
- markerid = id(o)
- if markerid in markers:
- raise ValueError("Circular reference detected")
- markers[markerid] = o
- o = _default(o)
- for chunk in _iterencode(o, _current_indent_level):
+ self._mark_markers(markers, o)
+ obj = self.default(o)
+ for chunk in self._iterencode(obj, markers,
+ _current_indent_level):
yield chunk
- if markers is not None:
- del markers[markerid]
-
- return _iterencode
+ self._remove_markers(markers, o)
diff --git a/lib-python/modified-2.7/json/tests/test_unicode.py b/lib-python/modified-2.7/json/tests/test_unicode.py
--- a/lib-python/modified-2.7/json/tests/test_unicode.py
+++ b/lib-python/modified-2.7/json/tests/test_unicode.py
@@ -80,3 +80,9 @@
self.assertEqual(type(json.loads(u'["a"]')[0]), unicode)
# Issue 10038.
self.assertEqual(type(json.loads('"foo"')), unicode)
+
+ def test_encode_not_utf_8(self):
+ self.assertEqual(json.dumps('\xb1\xe6', encoding='iso8859-2'),
+ '"\\u0105\\u0107"')
+ self.assertEqual(json.dumps(['\xb1\xe6'], encoding='iso8859-2'),
+ '["\\u0105\\u0107"]')
diff --git a/lib-python/modified-2.7/test/test_array.py b/lib-python/modified-2.7/test/test_array.py
--- a/lib-python/modified-2.7/test/test_array.py
+++ b/lib-python/modified-2.7/test/test_array.py
@@ -295,9 +295,10 @@
)
b = array.array(self.badtypecode())
- self.assertRaises(TypeError, "a + b")
-
- self.assertRaises(TypeError, "a + 'bad'")
+ with self.assertRaises(TypeError):
+ a + b
+ with self.assertRaises(TypeError):
+ a + 'bad'
def test_iadd(self):
a = array.array(self.typecode, self.example[::-1])
@@ -316,9 +317,10 @@
)
b = array.array(self.badtypecode())
- self.assertRaises(TypeError, "a += b")
-
- self.assertRaises(TypeError, "a += 'bad'")
+ with self.assertRaises(TypeError):
+ a += b
+ with self.assertRaises(TypeError):
+ a += 'bad'
def test_mul(self):
a = 5*array.array(self.typecode, self.example)
@@ -345,7 +347,8 @@
array.array(self.typecode)
)
- self.assertRaises(TypeError, "a * 'bad'")
+ with self.assertRaises(TypeError):
+ a * 'bad'
def test_imul(self):
a = array.array(self.typecode, self.example)
@@ -374,7 +377,8 @@
a *= -1
self.assertEqual(a, array.array(self.typecode))
- self.assertRaises(TypeError, "a *= 'bad'")
+ with self.assertRaises(TypeError):
+ a *= 'bad'
def test_getitem(self):
a = array.array(self.typecode, self.example)
diff --git a/lib-python/modified-2.7/test/test_heapq.py b/lib-python/modified-2.7/test/test_heapq.py
--- a/lib-python/modified-2.7/test/test_heapq.py
+++ b/lib-python/modified-2.7/test/test_heapq.py
@@ -186,6 +186,11 @@
self.assertFalse(sys.modules['heapq'] is self.module)
self.assertTrue(hasattr(self.module.heapify, 'func_code'))
+ def test_islice_protection(self):
+ m = self.module
+ self.assertFalse(m.nsmallest(-1, [1]))
+ self.assertFalse(m.nlargest(-1, [1]))
+
class TestHeapC(TestHeap):
module = c_heapq
diff --git a/lib-python/modified-2.7/test/test_urllib2.py b/lib-python/modified-2.7/test/test_urllib2.py
--- a/lib-python/modified-2.7/test/test_urllib2.py
+++ b/lib-python/modified-2.7/test/test_urllib2.py
@@ -307,6 +307,9 @@
def getresponse(self):
return MockHTTPResponse(MockFile(), {}, 200, "OK")
+ def close(self):
+ pass
+
class MockHandler:
# useful for testing handler machinery
# see add_ordered_mock_handlers() docstring
diff --git a/lib-python/modified-2.7/urllib2.py b/lib-python/modified-2.7/urllib2.py
new file mode 100644
--- /dev/null
+++ b/lib-python/modified-2.7/urllib2.py
@@ -0,0 +1,1436 @@
+"""An extensible library for opening URLs using a variety of protocols
+
+The simplest way to use this module is to call the urlopen function,
+which accepts a string containing a URL or a Request object (described
+below). It opens the URL and returns the results as file-like
+object; the returned object has some extra methods described below.
+
+The OpenerDirector manages a collection of Handler objects that do
+all the actual work. Each Handler implements a particular protocol or
+option. The OpenerDirector is a composite object that invokes the
+Handlers needed to open the requested URL. For example, the
+HTTPHandler performs HTTP GET and POST requests and deals with
+non-error returns. The HTTPRedirectHandler automatically deals with
+HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler
+deals with digest authentication.
+
+urlopen(url, data=None) -- Basic usage is the same as original
+urllib. pass the url and optionally data to post to an HTTP URL, and
+get a file-like object back. One difference is that you can also pass
+a Request instance instead of URL. Raises a URLError (subclass of
+IOError); for HTTP errors, raises an HTTPError, which can also be
+treated as a valid response.
+
+build_opener -- Function that creates a new OpenerDirector instance.
+Will install the default handlers. Accepts one or more Handlers as
+arguments, either instances or Handler classes that it will
+instantiate. If one of the argument is a subclass of the default
+handler, the argument will be installed instead of the default.
+
+install_opener -- Installs a new opener as the default opener.
+
+objects of interest:
+
+OpenerDirector -- Sets up the User Agent as the Python-urllib client and manages
+the Handler classes, while dealing with requests and responses.
+
+Request -- An object that encapsulates the state of a request. The
+state can be as simple as the URL. It can also include extra HTTP
+headers, e.g. a User-Agent.
+
+BaseHandler --
+
+exceptions:
+URLError -- A subclass of IOError, individual protocols have their own
+specific subclass.
+
+HTTPError -- Also a valid HTTP response, so you can treat an HTTP error
+as an exceptional event or valid response.
+
+internals:
+BaseHandler and parent
+_call_chain conventions
+
+Example usage:
+
+import urllib2
+
+# set up authentication info
+authinfo = urllib2.HTTPBasicAuthHandler()
+authinfo.add_password(realm='PDQ Application',
+ uri='https://mahler:8092/site-updates.py',
+ user='klem',
+ passwd='geheim$parole')
+
+proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})
+
+# build a new opener that adds authentication and caching FTP handlers
+opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)
+
+# install it
+urllib2.install_opener(opener)
+
+f = urllib2.urlopen('http://www.python.org/')
+
+
+"""
+
+# XXX issues:
+# If an authentication error handler that tries to perform
+# authentication for some reason but fails, how should the error be
+# signalled? The client needs to know the HTTP error code. But if
+# the handler knows that the problem was, e.g., that it didn't know
+# that hash algo that requested in the challenge, it would be good to
+# pass that information along to the client, too.
+# ftp errors aren't handled cleanly
+# check digest against correct (i.e. non-apache) implementation
+
+# Possible extensions:
+# complex proxies XXX not sure what exactly was meant by this
+# abstract factory for opener
+
+import base64
+import hashlib
+import httplib
+import mimetools
+import os
+import posixpath
+import random
+import re
+import socket
+import sys
+import time
+import urlparse
+import bisect
+
+try:
+ from cStringIO import StringIO
+except ImportError:
+ from StringIO import StringIO
+
+from urllib import (unwrap, unquote, splittype, splithost, quote,
+ addinfourl, splitport, splittag,
+ splitattr, ftpwrapper, splituser, splitpasswd, splitvalue)
+
+# support for FileHandler, proxies via environment variables
+from urllib import localhost, url2pathname, getproxies, proxy_bypass
+
+# used in User-Agent header sent
+__version__ = sys.version[:3]
+
+_opener = None
+def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
+ global _opener
+ if _opener is None:
+ _opener = build_opener()
+ return _opener.open(url, data, timeout)
+
+def install_opener(opener):
+ global _opener
+ _opener = opener
+
+# do these error classes make sense?
+# make sure all of the IOError stuff is overridden. we just want to be
+# subtypes.
+
+class URLError(IOError):
+ # URLError is a sub-type of IOError, but it doesn't share any of
+ # the implementation. need to override __init__ and __str__.
+ # It sets self.args for compatibility with other EnvironmentError
+ # subclasses, but args doesn't have the typical format with errno in
+ # slot 0 and strerror in slot 1. This may be better than nothing.
+ def __init__(self, reason):
+ self.args = reason,
+ self.reason = reason
+
+ def __str__(self):
+ return '<urlopen error %s>' % self.reason
+
+class HTTPError(URLError, addinfourl):
+ """Raised when HTTP error occurs, but also acts like non-error return"""
+ __super_init = addinfourl.__init__
+
+ def __init__(self, url, code, msg, hdrs, fp):
+ self.code = code
+ self.msg = msg
+ self.hdrs = hdrs
+ self.fp = fp
+ self.filename = url
+ # The addinfourl classes depend on fp being a valid file
+ # object. In some cases, the HTTPError may not have a valid
+ # file object. If this happens, the simplest workaround is to
+ # not initialize the base classes.
+ if fp is not None:
+ self.__super_init(fp, hdrs, url, code)
+
+ def __str__(self):
+ return 'HTTP Error %s: %s' % (self.code, self.msg)
+
+# copied from cookielib.py
+_cut_port_re = re.compile(r":\d+$")
+def request_host(request):
+ """Return request-host, as defined by RFC 2965.
+
+ Variation from RFC: returned value is lowercased, for convenient
+ comparison.
+
+ """
+ url = request.get_full_url()
+ host = urlparse.urlparse(url)[1]
+ if host == "":
+ host = request.get_header("Host", "")
+
+ # remove port, if present
+ host = _cut_port_re.sub("", host, 1)
+ return host.lower()
+
+class Request:
+
+ def __init__(self, url, data=None, headers={},
+ origin_req_host=None, unverifiable=False):
+ # unwrap('<URL:type://host/path>') --> 'type://host/path'
+ self.__original = unwrap(url)
+ self.__original, fragment = splittag(self.__original)
+ self.type = None
+ # self.__r_type is what's left after doing the splittype
+ self.host = None
+ self.port = None
+ self._tunnel_host = None
+ self.data = data
+ self.headers = {}
+ for key, value in headers.items():
+ self.add_header(key, value)
+ self.unredirected_hdrs = {}
+ if origin_req_host is None:
+ origin_req_host = request_host(self)
+ self.origin_req_host = origin_req_host
+ self.unverifiable = unverifiable
+
+ def __getattr__(self, attr):
+ # XXX this is a fallback mechanism to guard against these
+ # methods getting called in a non-standard order. this may be
+ # too complicated and/or unnecessary.
+ # XXX should the __r_XXX attributes be public?
+ if attr[:12] == '_Request__r_':
+ name = attr[12:]
+ if hasattr(Request, 'get_' + name):
+ getattr(self, 'get_' + name)()
+ return getattr(self, attr)
+ raise AttributeError, attr
+
+ def get_method(self):
+ if self.has_data():
+ return "POST"
+ else:
+ return "GET"
+
+ # XXX these helper methods are lame
+
+ def add_data(self, data):
+ self.data = data
+
+ def has_data(self):
+ return self.data is not None
+
+ def get_data(self):
+ return self.data
+
+ def get_full_url(self):
+ return self.__original
+
+ def get_type(self):
+ if self.type is None:
+ self.type, self.__r_type = splittype(self.__original)
+ if self.type is None:
+ raise ValueError, "unknown url type: %s" % self.__original
+ return self.type
+
+ def get_host(self):
+ if self.host is None:
+ self.host, self.__r_host = splithost(self.__r_type)
+ if self.host:
+ self.host = unquote(self.host)
+ return self.host
+
+ def get_selector(self):
+ return self.__r_host
+
+ def set_proxy(self, host, type):
+ if self.type == 'https' and not self._tunnel_host:
+ self._tunnel_host = self.host
+ else:
+ self.type = type
+ self.__r_host = self.__original
+
+ self.host = host
+
+ def has_proxy(self):
+ return self.__r_host == self.__original
+
+ def get_origin_req_host(self):
+ return self.origin_req_host
+
+ def is_unverifiable(self):
+ return self.unverifiable
+
+ def add_header(self, key, val):
+ # useful for something like authentication
+ self.headers[key.capitalize()] = val
+
+ def add_unredirected_header(self, key, val):
+ # will not be added to a redirected request
+ self.unredirected_hdrs[key.capitalize()] = val
+
+ def has_header(self, header_name):
+ return (header_name in self.headers or
+ header_name in self.unredirected_hdrs)
+
+ def get_header(self, header_name, default=None):
+ return self.headers.get(
+ header_name,
+ self.unredirected_hdrs.get(header_name, default))
+
+ def header_items(self):
+ hdrs = self.unredirected_hdrs.copy()
+ hdrs.update(self.headers)
+ return hdrs.items()
+
+class OpenerDirector:
+ def __init__(self):
+ client_version = "Python-urllib/%s" % __version__
+ self.addheaders = [('User-agent', client_version)]
+ # manage the individual handlers
+ self.handlers = []
+ self.handle_open = {}
+ self.handle_error = {}
+ self.process_response = {}
+ self.process_request = {}
+
+ def add_handler(self, handler):
+ if not hasattr(handler, "add_parent"):
+ raise TypeError("expected BaseHandler instance, got %r" %
+ type(handler))
+
+ added = False
+ for meth in dir(handler):
+ if meth in ["redirect_request", "do_open", "proxy_open"]:
+ # oops, coincidental match
+ continue
+
+ i = meth.find("_")
+ protocol = meth[:i]
+ condition = meth[i+1:]
+
+ if condition.startswith("error"):
+ j = condition.find("_") + i + 1
+ kind = meth[j+1:]
+ try:
+ kind = int(kind)
+ except ValueError:
+ pass
+ lookup = self.handle_error.get(protocol, {})
+ self.handle_error[protocol] = lookup
+ elif condition == "open":
+ kind = protocol
+ lookup = self.handle_open
+ elif condition == "response":
+ kind = protocol
+ lookup = self.process_response
+ elif condition == "request":
+ kind = protocol
+ lookup = self.process_request
+ else:
+ continue
+
+ handlers = lookup.setdefault(kind, [])
+ if handlers:
+ bisect.insort(handlers, handler)
+ else:
+ handlers.append(handler)
+ added = True
+
+ if added:
+ # the handlers must work in an specific order, the order
+ # is specified in a Handler attribute
+ bisect.insort(self.handlers, handler)
+ handler.add_parent(self)
+
+ def close(self):
+ # Only exists for backwards compatibility.
+ pass
+
+ def _call_chain(self, chain, kind, meth_name, *args):
+ # Handlers raise an exception if no one else should try to handle
+ # the request, or return None if they can't but another handler
+ # could. Otherwise, they return the response.
+ handlers = chain.get(kind, ())
+ for handler in handlers:
+ func = getattr(handler, meth_name)
+
+ result = func(*args)
+ if result is not None:
+ return result
+
+ def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
+ # accept a URL or a Request object
+ if isinstance(fullurl, basestring):
+ req = Request(fullurl, data)
+ else:
+ req = fullurl
+ if data is not None:
+ req.add_data(data)
+
+ req.timeout = timeout
+ protocol = req.get_type()
+
+ # pre-process request
+ meth_name = protocol+"_request"
+ for processor in self.process_request.get(protocol, []):
+ meth = getattr(processor, meth_name)
+ req = meth(req)
+
+ response = self._open(req, data)
+
+ # post-process response
+ meth_name = protocol+"_response"
+ for processor in self.process_response.get(protocol, []):
+ meth = getattr(processor, meth_name)
+ response = meth(req, response)
+
+ return response
+
+ def _open(self, req, data=None):
+ result = self._call_chain(self.handle_open, 'default',
+ 'default_open', req)
+ if result:
+ return result
+
+ protocol = req.get_type()
+ result = self._call_chain(self.handle_open, protocol, protocol +
+ '_open', req)
+ if result:
+ return result
+
+ return self._call_chain(self.handle_open, 'unknown',
+ 'unknown_open', req)
+
+ def error(self, proto, *args):
+ if proto in ('http', 'https'):
+ # XXX http[s] protocols are special-cased
+ dict = self.handle_error['http'] # https is not different than http
+ proto = args[2] # YUCK!
+ meth_name = 'http_error_%s' % proto
+ http_err = 1
+ orig_args = args
+ else:
+ dict = self.handle_error
+ meth_name = proto + '_error'
+ http_err = 0
+ args = (dict, proto, meth_name) + args
+ result = self._call_chain(*args)
+ if result:
+ return result
+
+ if http_err:
+ args = (dict, 'default', 'http_error_default') + orig_args
+ return self._call_chain(*args)
+
+# XXX probably also want an abstract factory that knows when it makes
+# sense to skip a superclass in favor of a subclass and when it might
+# make sense to include both
+
+def build_opener(*handlers):
+ """Create an opener object from a list of handlers.
+
+ The opener will use several default handlers, including support
+ for HTTP, FTP and when applicable, HTTPS.
+
+ If any of the handlers passed as arguments are subclasses of the
+ default handlers, the default handlers will not be used.
+ """
+ import types
+ def isclass(obj):
+ return isinstance(obj, (types.ClassType, type))
+
+ opener = OpenerDirector()
+ default_classes = [ProxyHandler, UnknownHandler, HTTPHandler,
+ HTTPDefaultErrorHandler, HTTPRedirectHandler,
+ FTPHandler, FileHandler, HTTPErrorProcessor]
+ if hasattr(httplib, 'HTTPS'):
+ default_classes.append(HTTPSHandler)
+ skip = set()
+ for klass in default_classes:
+ for check in handlers:
+ if isclass(check):
+ if issubclass(check, klass):
+ skip.add(klass)
+ elif isinstance(check, klass):
+ skip.add(klass)
+ for klass in skip:
+ default_classes.remove(klass)
+
+ for klass in default_classes:
+ opener.add_handler(klass())
+
+ for h in handlers:
+ if isclass(h):
+ h = h()
+ opener.add_handler(h)
+ return opener
+
+class BaseHandler:
+ handler_order = 500
+
+ def add_parent(self, parent):
+ self.parent = parent
+
+ def close(self):
+ # Only exists for backwards compatibility
+ pass
+
+ def __lt__(self, other):
+ if not hasattr(other, "handler_order"):
+ # Try to preserve the old behavior of having custom classes
+ # inserted after default ones (works only for custom user
+ # classes which are not aware of handler_order).
+ return True
+ return self.handler_order < other.handler_order
+
+
+class HTTPErrorProcessor(BaseHandler):
+ """Process HTTP error responses."""
+ handler_order = 1000 # after all other processing
+
+ def http_response(self, request, response):
+ code, msg, hdrs = response.code, response.msg, response.info()
+
+ # According to RFC 2616, "2xx" code indicates that the client's
+ # request was successfully received, understood, and accepted.
+ if not (200 <= code < 300):
+ response = self.parent.error(
+ 'http', request, response, code, msg, hdrs)
+
+ return response
+
+ https_response = http_response
+
+class HTTPDefaultErrorHandler(BaseHandler):
+ def http_error_default(self, req, fp, code, msg, hdrs):
+ raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
+
+class HTTPRedirectHandler(BaseHandler):
+ # maximum number of redirections to any single URL
+ # this is needed because of the state that cookies introduce
+ max_repeats = 4
+ # maximum total number of redirections (regardless of URL) before
+ # assuming we're in a loop
+ max_redirections = 10
+
+ def redirect_request(self, req, fp, code, msg, headers, newurl):
+ """Return a Request or None in response to a redirect.
+
+ This is called by the http_error_30x methods when a
+ redirection response is received. If a redirection should
+ take place, return a new Request to allow http_error_30x to
+ perform the redirect. Otherwise, raise HTTPError if no-one
+ else should try to handle this url. Return None if you can't
+ but another Handler might.
+ """
+ m = req.get_method()
+ if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
+ or code in (301, 302, 303) and m == "POST"):
+ # Strictly (according to RFC 2616), 301 or 302 in response
+ # to a POST MUST NOT cause a redirection without confirmation
+ # from the user (of urllib2, in this case). In practice,
+ # essentially all clients do redirect in this case, so we
+ # do the same.
+ # be conciliant with URIs containing a space
+ newurl = newurl.replace(' ', '%20')
+ newheaders = dict((k,v) for k,v in req.headers.items()
+ if k.lower() not in ("content-length", "content-type")
+ )
+ return Request(newurl,
+ headers=newheaders,
+ origin_req_host=req.get_origin_req_host(),
+ unverifiable=True)
+ else:
+ raise HTTPError(req.get_full_url(), code, msg, headers, fp)
+
+ # Implementation note: To avoid the server sending us into an
+ # infinite loop, the request object needs to track what URLs we
+ # have already seen. Do this by adding a handler-specific
+ # attribute to the Request object.
+ def http_error_302(self, req, fp, code, msg, headers):
+ # Some servers (incorrectly) return multiple Location headers
+ # (so probably same goes for URI). Use first header.
+ if 'location' in headers:
+ newurl = headers.getheaders('location')[0]
+ elif 'uri' in headers:
+ newurl = headers.getheaders('uri')[0]
+ else:
+ return
+
+ # fix a possible malformed URL
+ urlparts = urlparse.urlparse(newurl)
+ if not urlparts.path:
+ urlparts = list(urlparts)
+ urlparts[2] = "/"
+ newurl = urlparse.urlunparse(urlparts)
+
+ newurl = urlparse.urljoin(req.get_full_url(), newurl)
+
+ # XXX Probably want to forget about the state of the current
+ # request, although that might interact poorly with other
+ # handlers that also use handler-specific request attributes
+ new = self.redirect_request(req, fp, code, msg, headers, newurl)
+ if new is None:
+ return
+
+ # loop detection
+ # .redirect_dict has a key url if url was previously visited.
+ if hasattr(req, 'redirect_dict'):
+ visited = new.redirect_dict = req.redirect_dict
+ if (visited.get(newurl, 0) >= self.max_repeats or
+ len(visited) >= self.max_redirections):
+ raise HTTPError(req.get_full_url(), code,
+ self.inf_msg + msg, headers, fp)
+ else:
+ visited = new.redirect_dict = req.redirect_dict = {}
+ visited[newurl] = visited.get(newurl, 0) + 1
+
+ # Don't close the fp until we are sure that we won't use it
+ # with HTTPError.
+ fp.read()
+ fp.close()
+
+ return self.parent.open(new, timeout=req.timeout)
+
+ http_error_301 = http_error_303 = http_error_307 = http_error_302
+
+ inf_msg = "The HTTP server returned a redirect error that would " \
+ "lead to an infinite loop.\n" \
+ "The last 30x error message was:\n"
+
+
+def _parse_proxy(proxy):
+ """Return (scheme, user, password, host/port) given a URL or an authority.
+
+ If a URL is supplied, it must have an authority (host:port) component.
+ According to RFC 3986, having an authority component means the URL must
+ have two slashes after the scheme:
+
+ >>> _parse_proxy('file:/ftp.example.com/')
+ Traceback (most recent call last):
+ ValueError: proxy URL with no authority: 'file:/ftp.example.com/'
+
+ The first three items of the returned tuple may be None.
+
+ Examples of authority parsing:
+
+ >>> _parse_proxy('proxy.example.com')
+ (None, None, None, 'proxy.example.com')
+ >>> _parse_proxy('proxy.example.com:3128')
+ (None, None, None, 'proxy.example.com:3128')
+
+ The authority component may optionally include userinfo (assumed to be
+ username:password):
+
+ >>> _parse_proxy('joe:password at proxy.example.com')
+ (None, 'joe', 'password', 'proxy.example.com')
+ >>> _parse_proxy('joe:password at proxy.example.com:3128')
+ (None, 'joe', 'password', 'proxy.example.com:3128')
+
+ Same examples, but with URLs instead:
+
+ >>> _parse_proxy('http://proxy.example.com/')
+ ('http', None, None, 'proxy.example.com')
+ >>> _parse_proxy('http://proxy.example.com:3128/')
+ ('http', None, None, 'proxy.example.com:3128')
+ >>> _parse_proxy('http://joe:password@proxy.example.com/')
+ ('http', 'joe', 'password', 'proxy.example.com')
+ >>> _parse_proxy('http://joe:password@proxy.example.com:3128')
+ ('http', 'joe', 'password', 'proxy.example.com:3128')
+
+ Everything after the authority is ignored:
+
+ >>> _parse_proxy('ftp://joe:password@proxy.example.com/rubbish:3128')
+ ('ftp', 'joe', 'password', 'proxy.example.com')
+
+ Test for no trailing '/' case:
+
+ >>> _parse_proxy('http://joe:password@proxy.example.com')
+ ('http', 'joe', 'password', 'proxy.example.com')
+
+ """
+ scheme, r_scheme = splittype(proxy)
+ if not r_scheme.startswith("/"):
+ # authority
+ scheme = None
+ authority = proxy
+ else:
+ # URL
+ if not r_scheme.startswith("//"):
+ raise ValueError("proxy URL with no authority: %r" % proxy)
+ # We have an authority, so for RFC 3986-compliant URLs (by ss 3.
+ # and 3.3.), path is empty or starts with '/'
+ end = r_scheme.find("/", 2)
+ if end == -1:
+ end = None
+ authority = r_scheme[2:end]
+ userinfo, hostport = splituser(authority)
+ if userinfo is not None:
+ user, password = splitpasswd(userinfo)
+ else:
+ user = password = None
+ return scheme, user, password, hostport
+
+class ProxyHandler(BaseHandler):
+ # Proxies must be in front
+ handler_order = 100
+
+ def __init__(self, proxies=None):
+ if proxies is None:
+ proxies = getproxies()
+ assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
+ self.proxies = proxies
+ for type, url in proxies.items():
+ setattr(self, '%s_open' % type,
+ lambda r, proxy=url, type=type, meth=self.proxy_open: \
+ meth(r, proxy, type))
+
+ def proxy_open(self, req, proxy, type):
+ orig_type = req.get_type()
+ proxy_type, user, password, hostport = _parse_proxy(proxy)
+
+ if proxy_type is None:
+ proxy_type = orig_type
+
+ if req.host and proxy_bypass(req.host):
+ return None
+
+ if user and password:
+ user_pass = '%s:%s' % (unquote(user), unquote(password))
+ creds = base64.b64encode(user_pass).strip()
+ req.add_header('Proxy-authorization', 'Basic ' + creds)
+ hostport = unquote(hostport)
+ req.set_proxy(hostport, proxy_type)
+
+ if orig_type == proxy_type or orig_type == 'https':
+ # let other handlers take care of it
+ return None
+ else:
+ # need to start over, because the other handlers don't
+ # grok the proxy's URL type
+ # e.g. if we have a constructor arg proxies like so:
+ # {'http': 'ftp://proxy.example.com'}, we may end up turning
+ # a request for http://acme.example.com/a into one for
+ # ftp://proxy.example.com/a
+ return self.parent.open(req, timeout=req.timeout)
+
+class HTTPPasswordMgr:
+
+ def __init__(self):
+ self.passwd = {}
+
+ def add_password(self, realm, uri, user, passwd):
+ # uri could be a single URI or a sequence
+ if isinstance(uri, basestring):
+ uri = [uri]
+ if not realm in self.passwd:
+ self.passwd[realm] = {}
+ for default_port in True, False:
+ reduced_uri = tuple(
+ [self.reduce_uri(u, default_port) for u in uri])
+ self.passwd[realm][reduced_uri] = (user, passwd)
+
+ def find_user_password(self, realm, authuri):
+ domains = self.passwd.get(realm, {})
+ for default_port in True, False:
+ reduced_authuri = self.reduce_uri(authuri, default_port)
+ for uris, authinfo in domains.iteritems():
+ for uri in uris:
+ if self.is_suburi(uri, reduced_authuri):
+ return authinfo
+ return None, None
+
+ def reduce_uri(self, uri, default_port=True):
+ """Accept authority or URI and extract only the authority and path."""
+ # note HTTP URLs do not have a userinfo component
+ parts = urlparse.urlsplit(uri)
+ if parts[1]:
+ # URI
+ scheme = parts[0]
+ authority = parts[1]
+ path = parts[2] or '/'
+ else:
+ # host or host:port
+ scheme = None
+ authority = uri
+ path = '/'
+ host, port = splitport(authority)
+ if default_port and port is None and scheme is not None:
+ dport = {"http": 80,
+ "https": 443,
+ }.get(scheme)
+ if dport is not None:
+ authority = "%s:%d" % (host, dport)
+ return authority, path
+
+ def is_suburi(self, base, test):
+ """Check if test is below base in a URI tree
+
+ Both args must be URIs in reduced form.
+ """
+ if base == test:
+ return True
+ if base[0] != test[0]:
+ return False
+ common = posixpath.commonprefix((base[1], test[1]))
+ if len(common) == len(base[1]):
+ return True
+ return False
+
+
+class HTTPPasswordMgrWithDefaultRealm(HTTPPasswordMgr):
+
+ def find_user_password(self, realm, authuri):
+ user, password = HTTPPasswordMgr.find_user_password(self, realm,
+ authuri)
+ if user is not None:
+ return user, password
+ return HTTPPasswordMgr.find_user_password(self, None, authuri)
+
+
+class AbstractBasicAuthHandler:
+
+ # XXX this allows for multiple auth-schemes, but will stupidly pick
+ # the last one with a realm specified.
+
+ # allow for double- and single-quoted realm values
+ # (single quotes are a violation of the RFC, but appear in the wild)
+ rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+'
+ 'realm=(["\'])(.*?)\\2', re.I)
+
+ # XXX could pre-emptively send auth info already accepted (RFC 2617,
+ # end of section 2, and section 1.2 immediately after "credentials"
+ # production).
+
+ def __init__(self, password_mgr=None):
+ if password_mgr is None:
+ password_mgr = HTTPPasswordMgr()
+ self.passwd = password_mgr
+ self.add_password = self.passwd.add_password
+ self.retried = 0
+
+ def reset_retry_count(self):
+ self.retried = 0
+
+ def http_error_auth_reqed(self, authreq, host, req, headers):
+ # host may be an authority (without userinfo) or a URL with an
+ # authority
+ # XXX could be multiple headers
+ authreq = headers.get(authreq, None)
+
+ if self.retried > 5:
+ # retry sending the username:password 5 times before failing.
+ raise HTTPError(req.get_full_url(), 401, "basic auth failed",
+ headers, None)
+ else:
+ self.retried += 1
+
+ if authreq:
+ mo = AbstractBasicAuthHandler.rx.search(authreq)
+ if mo:
+ scheme, quote, realm = mo.groups()
+ if scheme.lower() == 'basic':
+ response = self.retry_http_basic_auth(host, req, realm)
+ if response and response.code != 401:
+ self.retried = 0
+ return response
+
+ def retry_http_basic_auth(self, host, req, realm):
+ user, pw = self.passwd.find_user_password(realm, host)
+ if pw is not None:
+ raw = "%s:%s" % (user, pw)
+ auth = 'Basic %s' % base64.b64encode(raw).strip()
+ if req.headers.get(self.auth_header, None) == auth:
+ return None
+ req.add_unredirected_header(self.auth_header, auth)
+ return self.parent.open(req, timeout=req.timeout)
+ else:
+ return None
+
+
+class HTTPBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler):
+
+ auth_header = 'Authorization'
+
+ def http_error_401(self, req, fp, code, msg, headers):
+ url = req.get_full_url()
+ response = self.http_error_auth_reqed('www-authenticate',
+ url, req, headers)
+ self.reset_retry_count()
+ return response
+
+
+class ProxyBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler):
+
+ auth_header = 'Proxy-authorization'
+
+ def http_error_407(self, req, fp, code, msg, headers):
+ # http_error_auth_reqed requires that there is no userinfo component in
+ # authority. Assume there isn't one, since urllib2 does not (and
+ # should not, RFC 3986 s. 3.2.1) support requests for URLs containing
+ # userinfo.
+ authority = req.get_host()
+ response = self.http_error_auth_reqed('proxy-authenticate',
+ authority, req, headers)
+ self.reset_retry_count()
+ return response
+
+
+def randombytes(n):
+ """Return n random bytes."""
+ # Use /dev/urandom if it is available. Fall back to random module
+ # if not. It might be worthwhile to extend this function to use
+ # other platform-specific mechanisms for getting random bytes.
+ if os.path.exists("/dev/urandom"):
+ f = open("/dev/urandom")
+ s = f.read(n)
+ f.close()
+ return s
+ else:
+ L = [chr(random.randrange(0, 256)) for i in range(n)]
+ return "".join(L)
+
+class AbstractDigestAuthHandler:
+ # Digest authentication is specified in RFC 2617.
+
+ # XXX The client does not inspect the Authentication-Info header
+ # in a successful response.
+
+ # XXX It should be possible to test this implementation against
+ # a mock server that just generates a static set of challenges.
+
+ # XXX qop="auth-int" supports is shaky
+
+ def __init__(self, passwd=None):
+ if passwd is None:
+ passwd = HTTPPasswordMgr()
+ self.passwd = passwd
+ self.add_password = self.passwd.add_password
+ self.retried = 0
+ self.nonce_count = 0
+ self.last_nonce = None
+
+ def reset_retry_count(self):
+ self.retried = 0
+
+ def http_error_auth_reqed(self, auth_header, host, req, headers):
+ authreq = headers.get(auth_header, None)
+ if self.retried > 5:
+ # Don't fail endlessly - if we failed once, we'll probably
+ # fail a second time. Hm. Unless the Password Manager is
+ # prompting for the information. Crap. This isn't great
+ # but it's better than the current 'repeat until recursion
+ # depth exceeded' approach <wink>
+ raise HTTPError(req.get_full_url(), 401, "digest auth failed",
+ headers, None)
+ else:
+ self.retried += 1
+ if authreq:
+ scheme = authreq.split()[0]
+ if scheme.lower() == 'digest':
+ return self.retry_http_digest_auth(req, authreq)
+
+ def retry_http_digest_auth(self, req, auth):
+ token, challenge = auth.split(' ', 1)
+ chal = parse_keqv_list(parse_http_list(challenge))
+ auth = self.get_authorization(req, chal)
+ if auth:
+ auth_val = 'Digest %s' % auth
+ if req.headers.get(self.auth_header, None) == auth_val:
+ return None
+ req.add_unredirected_header(self.auth_header, auth_val)
+ resp = self.parent.open(req, timeout=req.timeout)
+ return resp
+
+ def get_cnonce(self, nonce):
+ # The cnonce-value is an opaque
+ # quoted string value provided by the client and used by both client
+ # and server to avoid chosen plaintext attacks, to provide mutual
+ # authentication, and to provide some message integrity protection.
+ # This isn't a fabulous effort, but it's probably Good Enough.
+ dig = hashlib.sha1("%s:%s:%s:%s" % (self.nonce_count, nonce, time.ctime(),
+ randombytes(8))).hexdigest()
+ return dig[:16]
+
+ def get_authorization(self, req, chal):
+ try:
+ realm = chal['realm']
+ nonce = chal['nonce']
+ qop = chal.get('qop')
+ algorithm = chal.get('algorithm', 'MD5')
+ # mod_digest doesn't send an opaque, even though it isn't
+ # supposed to be optional
+ opaque = chal.get('opaque', None)
+ except KeyError:
+ return None
+
+ H, KD = self.get_algorithm_impls(algorithm)
+ if H is None:
+ return None
+
+ user, pw = self.passwd.find_user_password(realm, req.get_full_url())
+ if user is None:
+ return None
+
+ # XXX not implemented yet
+ if req.has_data():
+ entdig = self.get_entity_digest(req.get_data(), chal)
+ else:
+ entdig = None
+
+ A1 = "%s:%s:%s" % (user, realm, pw)
+ A2 = "%s:%s" % (req.get_method(),
+ # XXX selector: what about proxies and full urls
+ req.get_selector())
+ if qop == 'auth':
+ if nonce == self.last_nonce:
+ self.nonce_count += 1
+ else:
+ self.nonce_count = 1
+ self.last_nonce = nonce
+
+ ncvalue = '%08x' % self.nonce_count
+ cnonce = self.get_cnonce(nonce)
+ noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, H(A2))
+ respdig = KD(H(A1), noncebit)
+ elif qop is None:
+ respdig = KD(H(A1), "%s:%s" % (nonce, H(A2)))
+ else:
+ # XXX handle auth-int.
+ raise URLError("qop '%s' is not supported." % qop)
+
+ # XXX should the partial digests be encoded too?
+
+ base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \
+ 'response="%s"' % (user, realm, nonce, req.get_selector(),
+ respdig)
+ if opaque:
+ base += ', opaque="%s"' % opaque
+ if entdig:
+ base += ', digest="%s"' % entdig
+ base += ', algorithm="%s"' % algorithm
+ if qop:
+ base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)
+ return base
+
+ def get_algorithm_impls(self, algorithm):
+ # algorithm should be case-insensitive according to RFC2617
+ algorithm = algorithm.upper()
+ # lambdas assume digest modules are imported at the top level
+ if algorithm == 'MD5':
+ H = lambda x: hashlib.md5(x).hexdigest()
+ elif algorithm == 'SHA':
+ H = lambda x: hashlib.sha1(x).hexdigest()
+ # XXX MD5-sess
+ KD = lambda s, d: H("%s:%s" % (s, d))
+ return H, KD
+
+ def get_entity_digest(self, data, chal):
+ # XXX not implemented yet
+ return None
+
+
+class HTTPDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler):
+ """An authentication protocol defined by RFC 2069
+
+ Digest authentication improves on basic authentication because it
+ does not transmit passwords in the clear.
+ """
+
+ auth_header = 'Authorization'
+ handler_order = 490 # before Basic auth
+
+ def http_error_401(self, req, fp, code, msg, headers):
+ host = urlparse.urlparse(req.get_full_url())[1]
+ retry = self.http_error_auth_reqed('www-authenticate',
+ host, req, headers)
+ self.reset_retry_count()
+ return retry
+
+
+class ProxyDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler):
+
+ auth_header = 'Proxy-Authorization'
+ handler_order = 490 # before Basic auth
+
+ def http_error_407(self, req, fp, code, msg, headers):
+ host = req.get_host()
+ retry = self.http_error_auth_reqed('proxy-authenticate',
+ host, req, headers)
+ self.reset_retry_count()
+ return retry
+
+class AbstractHTTPHandler(BaseHandler):
+
+ def __init__(self, debuglevel=0):
+ self._debuglevel = debuglevel
+
+ def set_http_debuglevel(self, level):
+ self._debuglevel = level
+
+ def do_request_(self, request):
+ host = request.get_host()
+ if not host:
+ raise URLError('no host given')
+
+ if request.has_data(): # POST
+ data = request.get_data()
+ if not request.has_header('Content-type'):
+ request.add_unredirected_header(
+ 'Content-type',
+ 'application/x-www-form-urlencoded')
+ if not request.has_header('Content-length'):
+ request.add_unredirected_header(
+ 'Content-length', '%d' % len(data))
+
+ sel_host = host
+ if request.has_proxy():
+ scheme, sel = splittype(request.get_selector())
+ sel_host, sel_path = splithost(sel)
+
+ if not request.has_header('Host'):
+ request.add_unredirected_header('Host', sel_host)
+ for name, value in self.parent.addheaders:
+ name = name.capitalize()
+ if not request.has_header(name):
+ request.add_unredirected_header(name, value)
+
+ return request
+
+ def do_open(self, http_class, req):
+ """Return an addinfourl object for the request, using http_class.
+
+ http_class must implement the HTTPConnection API from httplib.
+ The addinfourl return value is a file-like object. It also
+ has methods and attributes including:
+ - info(): return a mimetools.Message object for the headers
+ - geturl(): return the original request URL
+ - code: HTTP status code
+ """
+ host = req.get_host()
+ if not host:
+ raise URLError('no host given')
+
+ h = http_class(host, timeout=req.timeout) # will parse host:port
+ h.set_debuglevel(self._debuglevel)
+
+ headers = dict(req.unredirected_hdrs)
+ headers.update(dict((k, v) for k, v in req.headers.items()
+ if k not in headers))
+
+ # We want to make an HTTP/1.1 request, but the addinfourl
+ # class isn't prepared to deal with a persistent connection.
+ # It will try to read all remaining data from the socket,
+ # which will block while the server waits for the next request.
+ # So make sure the connection gets closed after the (only)
+ # request.
+ headers["Connection"] = "close"
+ headers = dict(
+ (name.title(), val) for name, val in headers.items())
+
+ if req._tunnel_host:
+ tunnel_headers = {}
+ proxy_auth_hdr = "Proxy-Authorization"
+ if proxy_auth_hdr in headers:
+ tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
+ # Proxy-Authorization should not be sent to origin
+ # server.
+ del headers[proxy_auth_hdr]
+ h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
+
+ try:
+ h.request(req.get_method(), req.get_selector(), req.data, headers)
+ try:
+ r = h.getresponse(buffering=True)
+ except TypeError: #buffering kw not supported
+ r = h.getresponse()
+ except socket.error, err: # XXX what error?
+ h.close()
+ raise URLError(err)
+
+ # Pick apart the HTTPResponse object to get the addinfourl
+ # object initialized properly.
+
+ # Wrap the HTTPResponse object in socket's file object adapter
+ # for Windows. That adapter calls recv(), so delegate recv()
+ # to read(). This weird wrapping allows the returned object to
+ # have readline() and readlines() methods.
+
+ # XXX It might be better to extract the read buffering code
+ # out of socket._fileobject() and into a base class.
+
+ r.recv = r.read
+ fp = socket._fileobject(r, close=True)
+
+ resp = addinfourl(fp, r.msg, req.get_full_url())
+ resp.code = r.status
+ resp.msg = r.reason
+ return resp
+
+
+class HTTPHandler(AbstractHTTPHandler):
+
+ def http_open(self, req):
+ return self.do_open(httplib.HTTPConnection, req)
+
+ http_request = AbstractHTTPHandler.do_request_
+
+if hasattr(httplib, 'HTTPS'):
+ class HTTPSHandler(AbstractHTTPHandler):
+
+ def https_open(self, req):
+ return self.do_open(httplib.HTTPSConnection, req)
+
+ https_request = AbstractHTTPHandler.do_request_
+
+class HTTPCookieProcessor(BaseHandler):
+ def __init__(self, cookiejar=None):
+ import cookielib
+ if cookiejar is None:
+ cookiejar = cookielib.CookieJar()
+ self.cookiejar = cookiejar
+
+ def http_request(self, request):
+ self.cookiejar.add_cookie_header(request)
+ return request
+
+ def http_response(self, request, response):
+ self.cookiejar.extract_cookies(response, request)
+ return response
+
+ https_request = http_request
+ https_response = http_response
+
+class UnknownHandler(BaseHandler):
+ def unknown_open(self, req):
+ type = req.get_type()
+ raise URLError('unknown url type: %s' % type)
+
+def parse_keqv_list(l):
+ """Parse list of key=value strings where keys are not duplicated."""
+ parsed = {}
+ for elt in l:
+ k, v = elt.split('=', 1)
+ if v[0] == '"' and v[-1] == '"':
+ v = v[1:-1]
+ parsed[k] = v
+ return parsed
+
+def parse_http_list(s):
+ """Parse lists as described by RFC 2068 Section 2.
+
+ In particular, parse comma-separated lists where the elements of
+ the list may include quoted-strings. A quoted-string could
+ contain a comma. A non-quoted string could have quotes in the
+ middle. Neither commas nor quotes count if they are escaped.
+ Only double-quotes count, not single-quotes.
+ """
+ res = []
+ part = ''
+
+ escape = quote = False
+ for cur in s:
+ if escape:
+ part += cur
+ escape = False
+ continue
+ if quote:
+ if cur == '\\':
+ escape = True
+ continue
+ elif cur == '"':
+ quote = False
+ part += cur
+ continue
+
+ if cur == ',':
+ res.append(part)
+ part = ''
+ continue
+
+ if cur == '"':
+ quote = True
+
+ part += cur
+
+ # append last part
+ if part:
+ res.append(part)
+
+ return [part.strip() for part in res]
+
+def _safe_gethostbyname(host):
+ try:
+ return socket.gethostbyname(host)
+ except socket.gaierror:
+ return None
+
+class FileHandler(BaseHandler):
+ # Use local file or FTP depending on form of URL
+ def file_open(self, req):
+ url = req.get_selector()
+ if url[:2] == '//' and url[2:3] != '/' and (req.host and
+ req.host != 'localhost'):
+ req.type = 'ftp'
+ return self.parent.open(req)
+ else:
+ return self.open_local_file(req)
+
+ # names for the localhost
+ names = None
+ def get_names(self):
+ if FileHandler.names is None:
+ try:
+ FileHandler.names = tuple(
+ socket.gethostbyname_ex('localhost')[2] +
+ socket.gethostbyname_ex(socket.gethostname())[2])
+ except socket.gaierror:
+ FileHandler.names = (socket.gethostbyname('localhost'),)
+ return FileHandler.names
+
+ # not entirely sure what the rules are here
+ def open_local_file(self, req):
+ import email.utils
+ import mimetypes
+ host = req.get_host()
+ filename = req.get_selector()
+ localfile = url2pathname(filename)
+ try:
+ stats = os.stat(localfile)
+ size = stats.st_size
+ modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
+ mtype = mimetypes.guess_type(filename)[0]
+ headers = mimetools.Message(StringIO(
+ 'Content-type: %s\nContent-length: %d\nLast-modified: %s\n' %
+ (mtype or 'text/plain', size, modified)))
+ if host:
+ host, port = splitport(host)
+ if not host or \
+ (not port and _safe_gethostbyname(host) in self.get_names()):
+ if host:
+ origurl = 'file://' + host + filename
+ else:
+ origurl = 'file://' + filename
+ return addinfourl(open(localfile, 'rb'), headers, origurl)
+ except OSError, msg:
+ # urllib2 users shouldn't expect OSErrors coming from urlopen()
+ raise URLError(msg)
+ raise URLError('file not on local host')
+
+class FTPHandler(BaseHandler):
+ def ftp_open(self, req):
+ import ftplib
+ import mimetypes
+ host = req.get_host()
+ if not host:
+ raise URLError('ftp error: no host given')
+ host, port = splitport(host)
+ if port is None:
+ port = ftplib.FTP_PORT
+ else:
+ port = int(port)
+
+ # username/password handling
+ user, host = splituser(host)
+ if user:
+ user, passwd = splitpasswd(user)
+ else:
+ passwd = None
+ host = unquote(host)
+ user = user or ''
+ passwd = passwd or ''
+
+ try:
+ host = socket.gethostbyname(host)
+ except socket.error, msg:
+ raise URLError(msg)
+ path, attrs = splitattr(req.get_selector())
+ dirs = path.split('/')
+ dirs = map(unquote, dirs)
+ dirs, file = dirs[:-1], dirs[-1]
+ if dirs and not dirs[0]:
+ dirs = dirs[1:]
+ try:
+ fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
+ type = file and 'I' or 'D'
+ for attr in attrs:
+ attr, value = splitvalue(attr)
+ if attr.lower() == 'type' and \
+ value in ('a', 'A', 'i', 'I', 'd', 'D'):
+ type = value.upper()
+ fp, retrlen = fw.retrfile(file, type)
+ headers = ""
+ mtype = mimetypes.guess_type(req.get_full_url())[0]
+ if mtype:
+ headers += "Content-type: %s\n" % mtype
+ if retrlen is not None and retrlen >= 0:
+ headers += "Content-length: %d\n" % retrlen
+ sf = StringIO(headers)
+ headers = mimetools.Message(sf)
+ return addinfourl(fp, headers, req.get_full_url())
+ except ftplib.all_errors, msg:
+ raise URLError, ('ftp error: %s' % msg), sys.exc_info()[2]
+
+ def connect_ftp(self, user, passwd, host, port, dirs, timeout):
+ fw = ftpwrapper(user, passwd, host, port, dirs, timeout)
+## fw.ftp.set_debuglevel(1)
+ return fw
+
+class CacheFTPHandler(FTPHandler):
+ # XXX would be nice to have pluggable cache strategies
+ # XXX this stuff is definitely not thread safe
+ def __init__(self):
+ self.cache = {}
+ self.timeout = {}
+ self.soonest = 0
+ self.delay = 60
+ self.max_conns = 16
+
+ def setTimeout(self, t):
+ self.delay = t
+
+ def setMaxConns(self, m):
+ self.max_conns = m
+
+ def connect_ftp(self, user, passwd, host, port, dirs, timeout):
+ key = user, host, port, '/'.join(dirs), timeout
+ if key in self.cache:
+ self.timeout[key] = time.time() + self.delay
+ else:
+ self.cache[key] = ftpwrapper(user, passwd, host, port, dirs, timeout)
+ self.timeout[key] = time.time() + self.delay
+ self.check_cache()
+ return self.cache[key]
+
+ def check_cache(self):
+ # first check for old ones
+ t = time.time()
+ if self.soonest <= t:
+ for k, v in self.timeout.items():
+ if v < t:
+ self.cache[k].close()
+ del self.cache[k]
+ del self.timeout[k]
+ self.soonest = min(self.timeout.values())
+
+ # then check the size
+ if len(self.cache) == self.max_conns:
+ for k, v in self.timeout.items():
+ if v == self.soonest:
+ del self.cache[k]
+ del self.timeout[k]
+ break
+ self.soonest = min(self.timeout.values())
diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -395,9 +395,21 @@
_wrapper.f_in = f_in
_wrapper.f_out = f_out
- if hasattr(sys, '__raw_input__'): # PyPy
- _old_raw_input = sys.__raw_input__
+ if '__pypy__' in sys.builtin_module_names: # PyPy
+
+ def _old_raw_input(prompt=''):
+ # sys.__raw_input__() is only called when stdin and stdout are
+ # as expected and are ttys. If it is the case, then get_reader()
+ # should not really fail in _wrapper.raw_input(). If it still
+ # does, then we will just cancel the redirection and call again
+ # the built-in raw_input().
+ try:
+ del sys.__raw_input__
+ except AttributeError:
+ pass
+ return raw_input(prompt)
sys.__raw_input__ = _wrapper.raw_input
+
else:
# this is not really what readline.c does. Better than nothing I guess
import __builtin__
diff --git a/lib_pypy/resource.py b/lib_pypy/resource.py
--- a/lib_pypy/resource.py
+++ b/lib_pypy/resource.py
@@ -7,7 +7,7 @@
from ctypes_support import standard_c_lib as libc
from ctypes_support import get_errno
-from ctypes import Structure, c_int, c_long, byref, sizeof, POINTER
+from ctypes import Structure, c_int, c_long, byref, POINTER
from errno import EINVAL, EPERM
import _structseq
@@ -165,7 +165,6 @@
@builtinify
def getpagesize():
- pagesize = 0
if _getpagesize:
return _getpagesize()
else:
diff --git a/pypy/annotation/classdef.py b/pypy/annotation/classdef.py
--- a/pypy/annotation/classdef.py
+++ b/pypy/annotation/classdef.py
@@ -276,8 +276,8 @@
# create the Attribute and do the generalization asked for
newattr = Attribute(attr, self.bookkeeper)
if s_value:
- if newattr.name == 'intval' and getattr(s_value, 'unsigned', False):
- import pdb; pdb.set_trace()
+ #if newattr.name == 'intval' and getattr(s_value, 'unsigned', False):
+ # import pdb; pdb.set_trace()
newattr.s_value = s_value
# keep all subattributes' values
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -72,6 +72,7 @@
del working_modules['fcntl'] # LOCK_NB not defined
del working_modules["_minimal_curses"]
del working_modules["termios"]
+ del working_modules["_multiprocessing"] # depends on rctime
@@ -127,7 +128,7 @@
pypy_optiondescription = OptionDescription("objspace", "Object Space Options", [
ChoiceOption("name", "Object Space name",
- ["std", "flow", "thunk", "dump", "taint"],
+ ["std", "flow", "thunk", "dump"],
"std",
cmdline='--objspace -o'),
diff --git a/pypy/doc/__pypy__-module.rst b/pypy/doc/__pypy__-module.rst
--- a/pypy/doc/__pypy__-module.rst
+++ b/pypy/doc/__pypy__-module.rst
@@ -37,29 +37,6 @@
.. _`thunk object space docs`: objspace-proxies.html#thunk
.. _`interface section of the thunk object space docs`: objspace-proxies.html#thunk-interface
-.. broken:
-
- Taint Object Space Functionality
- ================================
-
- When the taint object space is used (choose with :config:`objspace.name`),
- the following names are put into ``__pypy__``:
-
- - ``taint``
- - ``is_tainted``
- - ``untaint``
- - ``taint_atomic``
- - ``_taint_debug``
- - ``_taint_look``
- - ``TaintError``
-
- Those are all described in the `interface section of the taint object space
- docs`_.
-
- For more detailed explanations and examples see the `taint object space docs`_.
-
- .. _`taint object space docs`: objspace-proxies.html#taint
- .. _`interface section of the taint object space docs`: objspace-proxies.html#taint-interface
Transparent Proxy Functionality
===============================
diff --git a/pypy/doc/config/objspace.name.txt b/pypy/doc/config/objspace.name.txt
--- a/pypy/doc/config/objspace.name.txt
+++ b/pypy/doc/config/objspace.name.txt
@@ -4,7 +4,6 @@
for normal usage):
* thunk_: The thunk object space adds lazy evaluation to PyPy.
- * taint_: The taint object space adds soft security features.
* dump_: Using this object spaces results in the dumpimp of all operations
to a log.
@@ -12,5 +11,4 @@
.. _`Object Space Proxies`: ../objspace-proxies.html
.. _`Standard Object Space`: ../objspace.html#standard-object-space
.. _thunk: ../objspace-proxies.html#thunk
-.. _taint: ../objspace-proxies.html#taint
.. _dump: ../objspace-proxies.html#dump
diff --git a/pypy/doc/index.rst b/pypy/doc/index.rst
--- a/pypy/doc/index.rst
+++ b/pypy/doc/index.rst
@@ -309,7 +309,6 @@
.. _`object space`: objspace.html
.. _FlowObjSpace: objspace.html#the-flow-object-space
.. _`trace object space`: objspace.html#the-trace-object-space
-.. _`taint object space`: objspace-proxies.html#taint
.. _`thunk object space`: objspace-proxies.html#thunk
.. _`transparent proxies`: objspace-proxies.html#tproxy
.. _`Differences between PyPy and CPython`: cpython_differences.html
diff --git a/pypy/doc/objspace-proxies.rst b/pypy/doc/objspace-proxies.rst
--- a/pypy/doc/objspace-proxies.rst
+++ b/pypy/doc/objspace-proxies.rst
@@ -129,297 +129,6 @@
function behaves lazily: all calls to it return a thunk object.
-.. broken right now:
-
- .. _taint:
-
- The Taint Object Space
- ======================
-
- Motivation
- ----------
-
- The Taint Object Space provides a form of security: "tainted objects",
- inspired by various sources, see [D12.1]_ for a more detailed discussion.
-
- The basic idea of this kind of security is not to protect against
- malicious code but to help with handling and boxing sensitive data.
- It covers two kinds of sensitive data: secret data which should not leak,
- and untrusted data coming from an external source and that must be
- validated before it is used.
-
- The idea is that, considering a large application that handles these
- kinds of sensitive data, there are typically only a small number of
- places that need to explicitly manipulate that sensitive data; all the
- other places merely pass it around, or do entirely unrelated things.
-
- Nevertheless, if a large application needs to be reviewed for security,
- it must be entirely carefully checked, because it is possible that a
- bug at some apparently unrelated place could lead to a leak of sensitive
- information in a way that an external attacker could exploit. For
- example, if any part of the application provides web services, an
- attacker might be able to issue unexpected requests with a regular web
- browser and deduce secret information from the details of the answers he
- gets. Another example is the common CGI attack where an attacker sends
- malformed inputs and causes the CGI script to do unintended things.
-
- An approach like that of the Taint Object Space allows the small parts
- of the program that manipulate sensitive data to be explicitly marked.
- The effect of this is that although these small parts still need a
- careful security review, the rest of the application no longer does,
- because even a bug would be unable to leak the information.
-
- We have implemented a simple two-level model: objects are either
- regular (untainted), or sensitive (tainted). Objects are marked as
- sensitive if they are secret or untrusted, and only declassified at
- carefully-checked positions (e.g. where the secret data is needed, or
- after the untrusted data has been fully validated).
-
- It would be simple to extend the code for more fine-grained scales of
- secrecy. For example it is typical in the literature to consider
- user-specified lattices of secrecy levels, corresponding to multiple
- "owners" that cannot access data belonging to another "owner" unless
- explicitly authorized to do so.
-
- Tainting and untainting
- -----------------------
-
- Start a py.py with the Taint Object Space and try the following example::
-
- $ py.py -o taint
- >>>> from __pypy__ import taint
- >>>> x = taint(6)
-
- # x is hidden from now on. We can pass it around and
- # even operate on it, but not inspect it. Taintness
- # is propagated to operation results.
-
- >>>> x
- TaintError
-
- >>>> if x > 5: y = 2 # see below
- TaintError
-
- >>>> y = x + 5 # ok
- >>>> lst = [x, y]
- >>>> z = lst.pop()
- >>>> t = type(z) # type() works too, tainted answer
- >>>> t
- TaintError
- >>>> u = t is int # even 'is' works
- >>>> u
- TaintError
-
- Notice that using a tainted boolean like ``x > 5`` in an ``if``
- statement is forbidden. This is because knowing which path is followed
- would give away a hint about ``x``; in the example above, if the
- statement ``if x > 5: y = 2`` was allowed to run, we would know
- something about the value of ``x`` by looking at the (untainted) value
- in the variable ``y``.
-
- Of course, there is a way to inspect tainted objects. The basic way is
- to explicitly "declassify" it with the ``untaint()`` function. In an
- application, the places that use ``untaint()`` are the places that need
- careful security review. To avoid unexpected objects showing up, the
- ``untaint()`` function must be called with the exact type of the object
- to declassify. It will raise ``TaintError`` if the type doesn't match::
-
- >>>> from __pypy__ import taint
- >>>> untaint(int, x)
- 6
- >>>> untaint(int, z)
- 11
- >>>> untaint(bool, x > 5)
- True
- >>>> untaint(int, x > 5)
- TaintError
-
-
- Taint Bombs
- -----------
-
- In this area, a common problem is what to do about failing operations.
- If an operation raises an exception when manipulating a tainted object,
- then the very presence of the exception can leak information about the
- tainted object itself. Consider::
-
- >>>> 5 / (x-6)
-
- By checking if this raises ``ZeroDivisionError`` or not, we would know
- if ``x`` was equal to 6 or not. The solution to this problem in the
- Taint Object Space is to introduce *Taint Bombs*. They are a kind of
- tainted object that doesn't contain a real object, but a pending
- exception. Taint Bombs are indistinguishable from normal tainted
- objects to unprivileged code. See::
-
- >>>> x = taint(6)
- >>>> i = 5 / (x-6) # no exception here
- >>>> j = i + 1 # nor here
- >>>> k = j + 5 # nor here
- >>>> untaint(int, k)
- TaintError
-
- In the above example, all of ``i``, ``j`` and ``k`` contain a Taint
- Bomb. Trying to untaint it raises an exception - a generic
- ``TaintError``. What we win is that the exception gives little away,
- and most importantly it occurs at the point where ``untaint()`` is
- called, not where the operation failed. This means that all calls to
- ``untaint()`` - but not the rest of the code - must be carefully
- reviewed for what occurs if they receive a Taint Bomb; they might catch
- the ``TaintError`` and give the user a generic message that something
- went wrong, if we are reasonably careful that the message or even its
- presence doesn't give information away. This might be a
- problem by itself, but there is no satisfying general solution here:
- it must be considered on a case-by-case basis. Again, what the
- Taint Object Space approach achieves is not solving these problems, but
- localizing them to well-defined small parts of the application - namely,
- around calls to ``untaint()``.
-
- The ``TaintError`` exception deliberately does not include any
- useful error messages, because they might give information away.
- Of course, this makes debugging quite a bit harder; a difficult
- problem to solve properly. So far we have implemented a way to peek in a Taint
- Box or Bomb, ``__pypy__._taint_look(x)``, and a "debug mode" that
- prints the exception as soon as a Bomb is created - both write
- information to the low-level stderr of the application, where we hope
- that it is unlikely to be seen by anyone but the application
- developer.
-
-
- Taint Atomic functions
- ----------------------
-
- Occasionally, a more complicated computation must be performed on a
- tainted object. This requires first untainting the object, performing the
- computations, and then carefully tainting the result again (including
- hiding all exceptions into Bombs).
-
- There is a built-in decorator that does this for you::
-
- >>>> @__pypy__.taint_atomic
- >>>> def myop(x, y):
- .... while x > 0:
- .... x -= y
- .... return x
- ....
- >>>> myop(42, 10)
- -8
- >>>> z = myop(taint(42), 10)
- >>>> z
- TaintError
- >>>> untaint(int, z)
- -8
-
- The decorator makes a whole function behave like a built-in operation.
- If no tainted argument is passed in, the function behaves normally. But
- if any of the arguments is tainted, it is automatically untainted - so
- the function body always sees untainted arguments - and the eventual
- result is tainted again (possibly in a Taint Bomb).
-
- It is important for the function marked as ``taint_atomic`` to have no
- visible side effects, as these could cause information leakage.
- This is currently not enforced, which means that all ``taint_atomic``
- functions have to be carefully reviewed for security (but not the
- callers of ``taint_atomic`` functions).
-
- A possible future extension would be to forbid side-effects on
- non-tainted objects from all ``taint_atomic`` functions.
-
- An example of usage: given a tainted object ``passwords_db`` that
- references a database of passwords, we can write a function
- that checks if a password is valid as follows::
-
- @taint_atomic
- def validate(passwords_db, username, password):
- assert type(passwords_db) is PasswordDatabase
- assert type(username) is str
- assert type(password) is str
- ...load username entry from passwords_db...
- return expected_password == password
-
- It returns a tainted boolean answer, or a Taint Bomb if something
- went wrong. A caller can do::
-
- ok = validate(passwords_db, 'john', '1234')
- ok = untaint(bool, ok)
-
- This can give three outcomes: ``True``, ``False``, or a ``TaintError``
- exception (with no information on it) if anything went wrong. If even
- this is considered giving too much information away, the ``False`` case
- can be made indistinguishable from the ``TaintError`` case (simply by
- raising an exception in ``validate()`` if the password is wrong).
-
- In the above example, the security results achieved are the following:
- as long as ``validate()`` does not leak information, no other part of
- the code can obtain more information about a passwords database than a
- Yes/No answer to a precise query.
-
- A possible extension of the ``taint_atomic`` decorator would be to check
- the argument types, as ``untaint()`` does, for the same reason: to
- prevent bugs where a function like ``validate()`` above is accidentally
- called with the wrong kind of tainted object, which would make it
- misbehave. For now, all ``taint_atomic`` functions should be
- conservative and carefully check all assumptions on their input
- arguments.
-
-
- .. _`taint-interface`:
-
- Interface
- ---------
-
- .. _`like a built-in operation`:
-
- The basic rule of the Tainted Object Space is that it introduces two new
- kinds of objects, Tainted Boxes and Tainted Bombs (which are not types
- in the Python sense). Each box internally contains a regular object;
- each bomb internally contains an exception object. An operation
- involving Tainted Boxes is performed on the objects contained in the
- boxes, and gives a Tainted Box or a Tainted Bomb as a result (such an
- operation does not let an exception be raised). An operation called
- with a Tainted Bomb argument immediately returns the same Tainted Bomb.
-
- In a PyPy running with (or translated with) the Taint Object Space,
- the ``__pypy__`` module exposes the following interface:
-
- * ``taint(obj)``
-
- Return a new Tainted Box wrapping ``obj``. Return ``obj`` itself
- if it is already tainted (a Box or a Bomb).
-
- * ``is_tainted(obj)``
-
- Check if ``obj`` is tainted (a Box or a Bomb).
-
- * ``untaint(type, obj)``
-
- Untaints ``obj`` if it is tainted. Raise ``TaintError`` if the type
- of the untainted object is not exactly ``type``, or if ``obj`` is a
- Bomb.
-
- * ``taint_atomic(func)``
-
- Return a wrapper function around the callable ``func``. The wrapper
- behaves `like a built-in operation`_ with respect to untainting the
- arguments, tainting the result, and returning a Bomb.
-
- * ``TaintError``
-
- Exception. On purpose, it provides no attribute or error message.
-
- * ``_taint_debug(level)``
-
- Set the debugging level to ``level`` (0=off). At level 1 or above,
- all Taint Bombs print a diagnostic message to stderr when they are
- created.
-
- * ``_taint_look(obj)``
-
- For debugging purposes: prints (to stderr) the type and address of
- the object in a Tainted Box, or prints the exception if ``obj`` is
- a Taint Bomb.
-
-
.. _dump:
The Dump Object Space
diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst
--- a/pypy/doc/project-ideas.rst
+++ b/pypy/doc/project-ideas.rst
@@ -17,6 +17,12 @@
projects, or anything else in PyPy, pop up on IRC or write to us on the
`mailing list`_.
+Make big integers faster
+-------------------------
+
+PyPy's implementation of the Python ``long`` type is slower than CPython's.
+Find out why and optimize them.
+
Numpy improvements
------------------
diff --git a/pypy/interpreter/astcompiler/ast.py b/pypy/interpreter/astcompiler/ast.py
--- a/pypy/interpreter/astcompiler/ast.py
+++ b/pypy/interpreter/astcompiler/ast.py
@@ -2,7 +2,7 @@
from pypy.interpreter.baseobjspace import Wrappable
from pypy.interpreter import typedef
from pypy.interpreter.gateway import interp2app
-from pypy.interpreter.error import OperationError
+from pypy.interpreter.error import OperationError, operationerrfmt
from pypy.rlib.unroll import unrolling_iterable
from pypy.tool.pairtype import extendabletype
from pypy.tool.sourcetools import func_with_new_name
@@ -2925,14 +2925,13 @@
def Module_get_body(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -2968,14 +2967,13 @@
def Interactive_get_body(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3015,8 +3013,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
return space.wrap(w_self.body)
def Expression_set_body(space, w_self, w_new_value):
@@ -3057,14 +3054,13 @@
def Suite_get_body(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3104,8 +3100,7 @@
return w_obj
if not w_self.initialization_state & w_self._lineno_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'lineno'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'lineno')
return space.wrap(w_self.lineno)
def stmt_set_lineno(space, w_self, w_new_value):
@@ -3126,8 +3121,7 @@
return w_obj
if not w_self.initialization_state & w_self._col_offset_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'col_offset'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'col_offset')
return space.wrap(w_self.col_offset)
def stmt_set_col_offset(space, w_self, w_new_value):
@@ -3157,8 +3151,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'name'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'name')
return space.wrap(w_self.name)
def FunctionDef_set_name(space, w_self, w_new_value):
@@ -3179,8 +3172,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'args'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'args')
return space.wrap(w_self.args)
def FunctionDef_set_args(space, w_self, w_new_value):
@@ -3197,14 +3189,13 @@
def FunctionDef_get_body(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3215,14 +3206,13 @@
def FunctionDef_get_decorator_list(space, w_self):
if not w_self.initialization_state & 8:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'decorator_list'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'decorator_list')
if w_self.w_decorator_list is None:
if w_self.decorator_list is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.decorator_list]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_decorator_list = w_list
return w_self.w_decorator_list
@@ -3266,8 +3256,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'name'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'name')
return space.wrap(w_self.name)
def ClassDef_set_name(space, w_self, w_new_value):
@@ -3284,14 +3273,13 @@
def ClassDef_get_bases(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'bases'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'bases')
if w_self.w_bases is None:
if w_self.bases is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.bases]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_bases = w_list
return w_self.w_bases
@@ -3302,14 +3290,13 @@
def ClassDef_get_body(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3320,14 +3307,13 @@
def ClassDef_get_decorator_list(space, w_self):
if not w_self.initialization_state & 8:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'decorator_list'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'decorator_list')
if w_self.w_decorator_list is None:
if w_self.decorator_list is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.decorator_list]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_decorator_list = w_list
return w_self.w_decorator_list
@@ -3372,8 +3358,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Return_set_value(space, w_self, w_new_value):
@@ -3414,14 +3399,13 @@
def Delete_get_targets(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'targets'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'targets')
if w_self.w_targets is None:
if w_self.targets is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.targets]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_targets = w_list
return w_self.w_targets
@@ -3457,14 +3441,13 @@
def Assign_get_targets(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'targets'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'targets')
if w_self.w_targets is None:
if w_self.targets is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.targets]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_targets = w_list
return w_self.w_targets
@@ -3479,8 +3462,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Assign_set_value(space, w_self, w_new_value):
@@ -3527,8 +3509,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'target'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'target')
return space.wrap(w_self.target)
def AugAssign_set_target(space, w_self, w_new_value):
@@ -3549,8 +3530,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'op'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'op')
return operator_to_class[w_self.op - 1]()
def AugAssign_set_op(space, w_self, w_new_value):
@@ -3573,8 +3553,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def AugAssign_set_value(space, w_self, w_new_value):
@@ -3621,8 +3600,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'dest'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'dest')
return space.wrap(w_self.dest)
def Print_set_dest(space, w_self, w_new_value):
@@ -3639,14 +3617,13 @@
def Print_get_values(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'values'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'values')
if w_self.w_values is None:
if w_self.values is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.values]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_values = w_list
return w_self.w_values
@@ -3661,8 +3638,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'nl'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'nl')
return space.wrap(w_self.nl)
def Print_set_nl(space, w_self, w_new_value):
@@ -3710,8 +3686,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'target'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'target')
return space.wrap(w_self.target)
def For_set_target(space, w_self, w_new_value):
@@ -3732,8 +3707,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'iter'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'iter')
return space.wrap(w_self.iter)
def For_set_iter(space, w_self, w_new_value):
@@ -3750,14 +3724,13 @@
def For_get_body(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3768,14 +3741,13 @@
def For_get_orelse(space, w_self):
if not w_self.initialization_state & 8:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'orelse'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'orelse')
if w_self.w_orelse is None:
if w_self.orelse is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.orelse]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_orelse = w_list
return w_self.w_orelse
@@ -3819,8 +3791,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'test'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'test')
return space.wrap(w_self.test)
def While_set_test(space, w_self, w_new_value):
@@ -3837,14 +3808,13 @@
def While_get_body(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3855,14 +3825,13 @@
def While_get_orelse(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'orelse'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'orelse')
if w_self.w_orelse is None:
if w_self.orelse is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.orelse]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_orelse = w_list
return w_self.w_orelse
@@ -3905,8 +3874,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'test'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'test')
return space.wrap(w_self.test)
def If_set_test(space, w_self, w_new_value):
@@ -3923,14 +3891,13 @@
def If_get_body(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -3941,14 +3908,13 @@
def If_get_orelse(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'orelse'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'orelse')
if w_self.w_orelse is None:
if w_self.orelse is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.orelse]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_orelse = w_list
return w_self.w_orelse
@@ -3991,8 +3957,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'context_expr'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'context_expr')
return space.wrap(w_self.context_expr)
def With_set_context_expr(space, w_self, w_new_value):
@@ -4013,8 +3978,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'optional_vars'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'optional_vars')
return space.wrap(w_self.optional_vars)
def With_set_optional_vars(space, w_self, w_new_value):
@@ -4031,14 +3995,13 @@
def With_get_body(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -4080,8 +4043,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'type'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'type')
return space.wrap(w_self.type)
def Raise_set_type(space, w_self, w_new_value):
@@ -4102,8 +4064,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'inst'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'inst')
return space.wrap(w_self.inst)
def Raise_set_inst(space, w_self, w_new_value):
@@ -4124,8 +4085,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'tback'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'tback')
return space.wrap(w_self.tback)
def Raise_set_tback(space, w_self, w_new_value):
@@ -4168,14 +4128,13 @@
def TryExcept_get_body(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -4186,14 +4145,13 @@
def TryExcept_get_handlers(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'handlers'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'handlers')
if w_self.w_handlers is None:
if w_self.handlers is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.handlers]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_handlers = w_list
return w_self.w_handlers
@@ -4204,14 +4162,13 @@
def TryExcept_get_orelse(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'orelse'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'orelse')
if w_self.w_orelse is None:
if w_self.orelse is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.orelse]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_orelse = w_list
return w_self.w_orelse
@@ -4251,14 +4208,13 @@
def TryFinally_get_body(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -4269,14 +4225,13 @@
def TryFinally_get_finalbody(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'finalbody'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'finalbody')
if w_self.w_finalbody is None:
if w_self.finalbody is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.finalbody]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_finalbody = w_list
return w_self.w_finalbody
@@ -4318,8 +4273,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'test'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'test')
return space.wrap(w_self.test)
def Assert_set_test(space, w_self, w_new_value):
@@ -4340,8 +4294,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'msg'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'msg')
return space.wrap(w_self.msg)
def Assert_set_msg(space, w_self, w_new_value):
@@ -4383,14 +4336,13 @@
def Import_get_names(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'names'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'names')
if w_self.w_names is None:
if w_self.names is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.names]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_names = w_list
return w_self.w_names
@@ -4430,8 +4382,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'module'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'module')
return space.wrap(w_self.module)
def ImportFrom_set_module(space, w_self, w_new_value):
@@ -4451,14 +4402,13 @@
def ImportFrom_get_names(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'names'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'names')
if w_self.w_names is None:
if w_self.names is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.names]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_names = w_list
return w_self.w_names
@@ -4473,8 +4423,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'level'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'level')
return space.wrap(w_self.level)
def ImportFrom_set_level(space, w_self, w_new_value):
@@ -4522,8 +4471,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
return space.wrap(w_self.body)
def Exec_set_body(space, w_self, w_new_value):
@@ -4544,8 +4492,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'globals'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'globals')
return space.wrap(w_self.globals)
def Exec_set_globals(space, w_self, w_new_value):
@@ -4566,8 +4513,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'locals'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'locals')
return space.wrap(w_self.locals)
def Exec_set_locals(space, w_self, w_new_value):
@@ -4610,14 +4556,13 @@
def Global_get_names(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'names'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'names')
if w_self.w_names is None:
if w_self.names is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.names]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_names = w_list
return w_self.w_names
@@ -4657,8 +4602,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Expr_set_value(space, w_self, w_new_value):
@@ -4754,8 +4698,7 @@
return w_obj
if not w_self.initialization_state & w_self._lineno_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'lineno'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'lineno')
return space.wrap(w_self.lineno)
def expr_set_lineno(space, w_self, w_new_value):
@@ -4776,8 +4719,7 @@
return w_obj
if not w_self.initialization_state & w_self._col_offset_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'col_offset'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'col_offset')
return space.wrap(w_self.col_offset)
def expr_set_col_offset(space, w_self, w_new_value):
@@ -4807,8 +4749,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'op'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'op')
return boolop_to_class[w_self.op - 1]()
def BoolOp_set_op(space, w_self, w_new_value):
@@ -4827,14 +4768,13 @@
def BoolOp_get_values(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'values'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'values')
if w_self.w_values is None:
if w_self.values is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.values]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_values = w_list
return w_self.w_values
@@ -4875,8 +4815,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'left'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'left')
return space.wrap(w_self.left)
def BinOp_set_left(space, w_self, w_new_value):
@@ -4897,8 +4836,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'op'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'op')
return operator_to_class[w_self.op - 1]()
def BinOp_set_op(space, w_self, w_new_value):
@@ -4921,8 +4859,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'right'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'right')
return space.wrap(w_self.right)
def BinOp_set_right(space, w_self, w_new_value):
@@ -4969,8 +4906,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'op'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'op')
return unaryop_to_class[w_self.op - 1]()
def UnaryOp_set_op(space, w_self, w_new_value):
@@ -4993,8 +4929,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'operand'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'operand')
return space.wrap(w_self.operand)
def UnaryOp_set_operand(space, w_self, w_new_value):
@@ -5040,8 +4975,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'args'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'args')
return space.wrap(w_self.args)
def Lambda_set_args(space, w_self, w_new_value):
@@ -5062,8 +4996,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
return space.wrap(w_self.body)
def Lambda_set_body(space, w_self, w_new_value):
@@ -5109,8 +5042,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'test'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'test')
return space.wrap(w_self.test)
def IfExp_set_test(space, w_self, w_new_value):
@@ -5131,8 +5063,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
return space.wrap(w_self.body)
def IfExp_set_body(space, w_self, w_new_value):
@@ -5153,8 +5084,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'orelse'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'orelse')
return space.wrap(w_self.orelse)
def IfExp_set_orelse(space, w_self, w_new_value):
@@ -5197,14 +5127,13 @@
def Dict_get_keys(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'keys'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'keys')
if w_self.w_keys is None:
if w_self.keys is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.keys]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_keys = w_list
return w_self.w_keys
@@ -5215,14 +5144,13 @@
def Dict_get_values(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'values'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'values')
if w_self.w_values is None:
if w_self.values is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.values]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_values = w_list
return w_self.w_values
@@ -5260,14 +5188,13 @@
def Set_get_elts(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elts'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elts')
if w_self.w_elts is None:
if w_self.elts is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.elts]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_elts = w_list
return w_self.w_elts
@@ -5307,8 +5234,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elt'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elt')
return space.wrap(w_self.elt)
def ListComp_set_elt(space, w_self, w_new_value):
@@ -5325,14 +5251,13 @@
def ListComp_get_generators(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'generators'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'generators')
if w_self.w_generators is None:
if w_self.generators is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.generators]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_generators = w_list
return w_self.w_generators
@@ -5373,8 +5298,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elt'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elt')
return space.wrap(w_self.elt)
def SetComp_set_elt(space, w_self, w_new_value):
@@ -5391,14 +5315,13 @@
def SetComp_get_generators(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'generators'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'generators')
if w_self.w_generators is None:
if w_self.generators is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.generators]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_generators = w_list
return w_self.w_generators
@@ -5439,8 +5362,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'key'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'key')
return space.wrap(w_self.key)
def DictComp_set_key(space, w_self, w_new_value):
@@ -5461,8 +5383,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def DictComp_set_value(space, w_self, w_new_value):
@@ -5479,14 +5400,13 @@
def DictComp_get_generators(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'generators'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'generators')
if w_self.w_generators is None:
if w_self.generators is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.generators]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_generators = w_list
return w_self.w_generators
@@ -5528,8 +5448,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elt'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elt')
return space.wrap(w_self.elt)
def GeneratorExp_set_elt(space, w_self, w_new_value):
@@ -5546,14 +5465,13 @@
def GeneratorExp_get_generators(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'generators'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'generators')
if w_self.w_generators is None:
if w_self.generators is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.generators]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_generators = w_list
return w_self.w_generators
@@ -5594,8 +5512,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Yield_set_value(space, w_self, w_new_value):
@@ -5640,8 +5557,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'left'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'left')
return space.wrap(w_self.left)
def Compare_set_left(space, w_self, w_new_value):
@@ -5658,14 +5574,13 @@
def Compare_get_ops(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ops'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ops')
if w_self.w_ops is None:
if w_self.ops is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [cmpop_to_class[node - 1]() for node in w_self.ops]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_ops = w_list
return w_self.w_ops
@@ -5676,14 +5591,13 @@
def Compare_get_comparators(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'comparators'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'comparators')
if w_self.w_comparators is None:
if w_self.comparators is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.comparators]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_comparators = w_list
return w_self.w_comparators
@@ -5726,8 +5640,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'func'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'func')
return space.wrap(w_self.func)
def Call_set_func(space, w_self, w_new_value):
@@ -5744,14 +5657,13 @@
def Call_get_args(space, w_self):
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'args'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'args')
if w_self.w_args is None:
if w_self.args is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.args]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_args = w_list
return w_self.w_args
@@ -5762,14 +5674,13 @@
def Call_get_keywords(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'keywords'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'keywords')
if w_self.w_keywords is None:
if w_self.keywords is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.keywords]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_keywords = w_list
return w_self.w_keywords
@@ -5784,8 +5695,7 @@
return w_obj
if not w_self.initialization_state & 8:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'starargs'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'starargs')
return space.wrap(w_self.starargs)
def Call_set_starargs(space, w_self, w_new_value):
@@ -5806,8 +5716,7 @@
return w_obj
if not w_self.initialization_state & 16:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'kwargs'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'kwargs')
return space.wrap(w_self.kwargs)
def Call_set_kwargs(space, w_self, w_new_value):
@@ -5858,8 +5767,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Repr_set_value(space, w_self, w_new_value):
@@ -5904,8 +5812,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'n'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'n')
return w_self.n
def Num_set_n(space, w_self, w_new_value):
@@ -5950,8 +5857,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 's'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 's')
return w_self.s
def Str_set_s(space, w_self, w_new_value):
@@ -5996,8 +5902,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Attribute_set_value(space, w_self, w_new_value):
@@ -6018,8 +5923,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'attr'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'attr')
return space.wrap(w_self.attr)
def Attribute_set_attr(space, w_self, w_new_value):
@@ -6040,8 +5944,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ctx'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ctx')
return expr_context_to_class[w_self.ctx - 1]()
def Attribute_set_ctx(space, w_self, w_new_value):
@@ -6090,8 +5993,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Subscript_set_value(space, w_self, w_new_value):
@@ -6112,8 +6014,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'slice'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'slice')
return space.wrap(w_self.slice)
def Subscript_set_slice(space, w_self, w_new_value):
@@ -6134,8 +6035,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ctx'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ctx')
return expr_context_to_class[w_self.ctx - 1]()
def Subscript_set_ctx(space, w_self, w_new_value):
@@ -6184,8 +6084,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'id'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'id')
return space.wrap(w_self.id)
def Name_set_id(space, w_self, w_new_value):
@@ -6206,8 +6105,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ctx'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ctx')
return expr_context_to_class[w_self.ctx - 1]()
def Name_set_ctx(space, w_self, w_new_value):
@@ -6251,14 +6149,13 @@
def List_get_elts(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elts'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elts')
if w_self.w_elts is None:
if w_self.elts is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.elts]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_elts = w_list
return w_self.w_elts
@@ -6273,8 +6170,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ctx'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ctx')
return expr_context_to_class[w_self.ctx - 1]()
def List_set_ctx(space, w_self, w_new_value):
@@ -6319,14 +6215,13 @@
def Tuple_get_elts(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'elts'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'elts')
if w_self.w_elts is None:
if w_self.elts is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.elts]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_elts = w_list
return w_self.w_elts
@@ -6341,8 +6236,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ctx'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ctx')
return expr_context_to_class[w_self.ctx - 1]()
def Tuple_set_ctx(space, w_self, w_new_value):
@@ -6391,8 +6285,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return w_self.value
def Const_set_value(space, w_self, w_new_value):
@@ -6510,8 +6403,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'lower'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'lower')
return space.wrap(w_self.lower)
def Slice_set_lower(space, w_self, w_new_value):
@@ -6532,8 +6424,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'upper'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'upper')
return space.wrap(w_self.upper)
def Slice_set_upper(space, w_self, w_new_value):
@@ -6554,8 +6445,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'step'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'step')
return space.wrap(w_self.step)
def Slice_set_step(space, w_self, w_new_value):
@@ -6598,14 +6488,13 @@
def ExtSlice_get_dims(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'dims'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'dims')
if w_self.w_dims is None:
if w_self.dims is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.dims]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_dims = w_list
return w_self.w_dims
@@ -6645,8 +6534,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def Index_set_value(space, w_self, w_new_value):
@@ -6915,8 +6803,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'target'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'target')
return space.wrap(w_self.target)
def comprehension_set_target(space, w_self, w_new_value):
@@ -6937,8 +6824,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'iter'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'iter')
return space.wrap(w_self.iter)
def comprehension_set_iter(space, w_self, w_new_value):
@@ -6955,14 +6841,13 @@
def comprehension_get_ifs(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'ifs'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'ifs')
if w_self.w_ifs is None:
if w_self.ifs is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.ifs]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_ifs = w_list
return w_self.w_ifs
@@ -7004,8 +6889,7 @@
return w_obj
if not w_self.initialization_state & w_self._lineno_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'lineno'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'lineno')
return space.wrap(w_self.lineno)
def excepthandler_set_lineno(space, w_self, w_new_value):
@@ -7026,8 +6910,7 @@
return w_obj
if not w_self.initialization_state & w_self._col_offset_mask:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'col_offset'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'col_offset')
return space.wrap(w_self.col_offset)
def excepthandler_set_col_offset(space, w_self, w_new_value):
@@ -7057,8 +6940,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'type'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'type')
return space.wrap(w_self.type)
def ExceptHandler_set_type(space, w_self, w_new_value):
@@ -7079,8 +6961,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'name'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'name')
return space.wrap(w_self.name)
def ExceptHandler_set_name(space, w_self, w_new_value):
@@ -7097,14 +6978,13 @@
def ExceptHandler_get_body(space, w_self):
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'body'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'body')
if w_self.w_body is None:
if w_self.body is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.body]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_body = w_list
return w_self.w_body
@@ -7142,14 +7022,13 @@
def arguments_get_args(space, w_self):
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'args'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'args')
if w_self.w_args is None:
if w_self.args is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.args]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_args = w_list
return w_self.w_args
@@ -7164,8 +7043,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'vararg'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'vararg')
return space.wrap(w_self.vararg)
def arguments_set_vararg(space, w_self, w_new_value):
@@ -7189,8 +7067,7 @@
return w_obj
if not w_self.initialization_state & 4:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'kwarg'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'kwarg')
return space.wrap(w_self.kwarg)
def arguments_set_kwarg(space, w_self, w_new_value):
@@ -7210,14 +7087,13 @@
def arguments_get_defaults(space, w_self):
if not w_self.initialization_state & 8:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'defaults'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'defaults')
if w_self.w_defaults is None:
if w_self.defaults is None:
- w_list = space.newlist([])
+ list_w = []
else:
list_w = [space.wrap(node) for node in w_self.defaults]
- w_list = space.newlist(list_w)
+ w_list = space.newlist(list_w)
w_self.w_defaults = w_list
return w_self.w_defaults
@@ -7261,8 +7137,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'arg'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'arg')
return space.wrap(w_self.arg)
def keyword_set_arg(space, w_self, w_new_value):
@@ -7283,8 +7158,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'value'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'value')
return space.wrap(w_self.value)
def keyword_set_value(space, w_self, w_new_value):
@@ -7330,8 +7204,7 @@
return w_obj
if not w_self.initialization_state & 1:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'name'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'name')
return space.wrap(w_self.name)
def alias_set_name(space, w_self, w_new_value):
@@ -7352,8 +7225,7 @@
return w_obj
if not w_self.initialization_state & 2:
typename = space.type(w_self).getname(space)
- w_err = space.wrap("'%s' object has no attribute 'asname'" % typename)
- raise OperationError(space.w_AttributeError, w_err)
+ raise operationerrfmt(space.w_AttributeError, "'%s' object has no attribute '%s'", typename, 'asname')
return space.wrap(w_self.asname)
def alias_set_asname(space, w_self, w_new_value):
diff --git a/pypy/interpreter/astcompiler/tools/asdl_py.py b/pypy/interpreter/astcompiler/tools/asdl_py.py
--- a/pypy/interpreter/astcompiler/tools/asdl_py.py
+++ b/pypy/interpreter/astcompiler/tools/asdl_py.py
@@ -414,13 +414,12 @@
self.emit(" return w_obj", 1)
self.emit("if not w_self.initialization_state & %s:" % (flag,), 1)
self.emit("typename = space.type(w_self).getname(space)", 2)
- self.emit("w_err = space.wrap(\"'%%s' object has no attribute '%s'\" %% typename)" %
+ self.emit("raise operationerrfmt(space.w_AttributeError, \"'%%s' object has no attribute '%%s'\", typename, '%s')" %
(field.name,), 2)
- self.emit("raise OperationError(space.w_AttributeError, w_err)", 2)
if field.seq:
self.emit("if w_self.w_%s is None:" % (field.name,), 1)
self.emit("if w_self.%s is None:" % (field.name,), 2)
- self.emit("w_list = space.newlist([])", 3)
+ self.emit("list_w = []", 3)
self.emit("else:", 2)
if field.type.value in self.data.simple_types:
wrapper = "%s_to_class[node - 1]()" % (field.type,)
@@ -428,7 +427,7 @@
wrapper = "space.wrap(node)"
self.emit("list_w = [%s for node in w_self.%s]" %
(wrapper, field.name), 3)
- self.emit("w_list = space.newlist(list_w)", 3)
+ self.emit("w_list = space.newlist(list_w)", 2)
self.emit("w_self.w_%s = w_list" % (field.name,), 2)
self.emit("return w_self.w_%s" % (field.name,), 1)
elif field.type.value in self.data.simple_types:
@@ -540,7 +539,7 @@
from pypy.interpreter.baseobjspace import Wrappable
from pypy.interpreter import typedef
from pypy.interpreter.gateway import interp2app
-from pypy.interpreter.error import OperationError
+from pypy.interpreter.error import OperationError, operationerrfmt
from pypy.rlib.unroll import unrolling_iterable
from pypy.tool.pairtype import extendabletype
from pypy.tool.sourcetools import func_with_new_name
@@ -639,9 +638,7 @@
missing = required[i]
if missing is not None:
err = "required field \\"%s\\" missing from %s"
- err = err % (missing, host)
- w_err = space.wrap(err)
- raise OperationError(space.w_TypeError, w_err)
+ raise operationerrfmt(space.w_TypeError, err, missing, host)
raise AssertionError("should not reach here")
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -391,8 +391,11 @@
def decrement_ticker(self, by):
value = self._ticker
if self.has_bytecode_counter: # this 'if' is constant-folded
- value -= by
- self._ticker = value
+ if jit.isconstant(by) and by == 0:
+ pass # normally constant-folded too
+ else:
+ value -= by
+ self._ticker = value
return value
diff --git a/pypy/interpreter/pyparser/pytokenizer.py b/pypy/interpreter/pyparser/pytokenizer.py
--- a/pypy/interpreter/pyparser/pytokenizer.py
+++ b/pypy/interpreter/pyparser/pytokenizer.py
@@ -226,7 +226,7 @@
parenlev = parenlev - 1
if parenlev < 0:
raise TokenError("unmatched '%s'" % initial, line,
- lnum-1, 0, token_list)
+ lnum, start + 1, token_list)
if token in python_opmap:
punct = python_opmap[token]
else:
diff --git a/pypy/interpreter/pyparser/test/test_pyparse.py b/pypy/interpreter/pyparser/test/test_pyparse.py
--- a/pypy/interpreter/pyparser/test/test_pyparse.py
+++ b/pypy/interpreter/pyparser/test/test_pyparse.py
@@ -87,6 +87,10 @@
assert exc.lineno == 1
assert exc.offset == 5
assert exc.lastlineno == 5
+ exc = py.test.raises(SyntaxError, parse, "abc)").value
+ assert exc.msg == "unmatched ')'"
+ assert exc.lineno == 1
+ assert exc.offset == 4
def test_is(self):
self.parse("x is y")
diff --git a/pypy/jit/backend/llgraph/llimpl.py b/pypy/jit/backend/llgraph/llimpl.py
--- a/pypy/jit/backend/llgraph/llimpl.py
+++ b/pypy/jit/backend/llgraph/llimpl.py
@@ -7,9 +7,7 @@
import weakref
from pypy.objspace.flow.model import Variable, Constant
from pypy.annotation import model as annmodel
-from pypy.jit.metainterp.history import (ConstInt, ConstPtr,
- BoxInt, BoxPtr, BoxObj, BoxFloat,
- REF, INT, FLOAT)
+from pypy.jit.metainterp.history import REF, INT, FLOAT
from pypy.jit.codewriter import heaptracker
from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rffi
from pypy.rpython.ootypesystem import ootype
@@ -17,7 +15,7 @@
from pypy.rpython.llinterp import LLException
from pypy.rpython.extregistry import ExtRegistryEntry
-from pypy.jit.metainterp import resoperation, executor
+from pypy.jit.metainterp import resoperation
from pypy.jit.metainterp.resoperation import rop
from pypy.jit.backend.llgraph import symbolic
from pypy.jit.codewriter import longlong
@@ -165,6 +163,7 @@
'unicodegetitem' : (('ref', 'int'), 'int'),
'unicodesetitem' : (('ref', 'int', 'int'), 'int'),
'cast_ptr_to_int' : (('ref',), 'int'),
+ 'cast_int_to_ptr' : (('int',), 'ref'),
'debug_merge_point': (('ref', 'int'), None),
'force_token' : ((), 'int'),
'call_may_force' : (('int', 'varargs'), 'intorptr'),
@@ -333,6 +332,13 @@
assert isinstance(type, str) and len(type) == 1
op.descr = Descr(ofs, type, arg_types=arg_types)
+def compile_add_descr_arg(loop, ofs, type, arg_types):
+ from pypy.jit.backend.llgraph.runner import Descr
+ loop = _from_opaque(loop)
+ op = loop.operations[-1]
+ assert isinstance(type, str) and len(type) == 1
+ op.args.append(Descr(ofs, type, arg_types=arg_types))
+
def compile_add_loop_token(loop, descr):
if we_are_translated():
raise ValueError("CALL_ASSEMBLER not supported")
@@ -437,8 +443,11 @@
self._may_force = -1
def getenv(self, v):
+ from pypy.jit.backend.llgraph.runner import Descr
if isinstance(v, Constant):
return v.value
+ elif isinstance(v, Descr):
+ return v
else:
return self.env[v]
@@ -806,6 +815,29 @@
else:
raise NotImplementedError
+ def op_getinteriorfield_gc(self, descr, array, index):
+ if descr.typeinfo == REF:
+ return do_getinteriorfield_gc_ptr(array, index, descr.ofs)
+ elif descr.typeinfo == INT:
+ return do_getinteriorfield_gc_int(array, index, descr.ofs)
+ elif descr.typeinfo == FLOAT:
+ return do_getinteriorfield_gc_float(array, index, descr.ofs)
+ else:
+ raise NotImplementedError
+
+ def op_setinteriorfield_gc(self, descr, array, index, newvalue):
+ if descr.typeinfo == REF:
+ return do_setinteriorfield_gc_ptr(array, index, descr.ofs,
+ newvalue)
+ elif descr.typeinfo == INT:
+ return do_setinteriorfield_gc_int(array, index, descr.ofs,
+ newvalue)
+ elif descr.typeinfo == FLOAT:
+ return do_setinteriorfield_gc_float(array, index, descr.ofs,
+ newvalue)
+ else:
+ raise NotImplementedError
+
def op_setfield_gc(self, fielddescr, struct, newvalue):
if fielddescr.typeinfo == REF:
do_setfield_gc_ptr(struct, fielddescr.ofs, newvalue)
@@ -875,9 +907,6 @@
def op_new_array(self, arraydescr, count):
return do_new_array(arraydescr.ofs, count)
- def op_cast_ptr_to_int(self, descr, ptr):
- return cast_to_int(ptr)
-
def op_force_token(self, descr):
opaque_frame = _to_opaque(self)
return llmemory.cast_ptr_to_adr(opaque_frame)
@@ -1356,6 +1385,22 @@
def do_getfield_gc_ptr(struct, fieldnum):
return cast_to_ptr(_getfield_gc(struct, fieldnum))
+def _getinteriorfield_gc(struct, fieldnum):
+ STRUCT, fieldname = symbolic.TokenToField[fieldnum]
+ return getattr(struct, fieldname)
+
+def do_getinteriorfield_gc_int(array, index, fieldnum):
+ struct = array._obj.container.getitem(index)
+ return cast_to_int(_getinteriorfield_gc(struct, fieldnum))
+
+def do_getinteriorfield_gc_float(array, index, fieldnum):
+ struct = array._obj.container.getitem(index)
+ return cast_to_floatstorage(_getinteriorfield_gc(struct, fieldnum))
+
+def do_getinteriorfield_gc_ptr(array, index, fieldnum):
+ struct = array._obj.container.getitem(index)
+ return cast_to_ptr(_getinteriorfield_gc(struct, fieldnum))
+
def _getfield_raw(struct, fieldnum):
STRUCT, fieldname = symbolic.TokenToField[fieldnum]
ptr = cast_from_int(lltype.Ptr(STRUCT), struct)
@@ -1411,26 +1456,28 @@
newvalue = cast_from_ptr(ITEMTYPE, newvalue)
array.setitem(index, newvalue)
-def do_setfield_gc_int(struct, fieldnum, newvalue):
- STRUCT, fieldname = symbolic.TokenToField[fieldnum]
- ptr = lltype.cast_opaque_ptr(lltype.Ptr(STRUCT), struct)
- FIELDTYPE = getattr(STRUCT, fieldname)
- newvalue = cast_from_int(FIELDTYPE, newvalue)
- setattr(ptr, fieldname, newvalue)
+def new_setfield_gc(cast_func):
+ def do_setfield_gc(struct, fieldnum, newvalue):
+ STRUCT, fieldname = symbolic.TokenToField[fieldnum]
+ ptr = lltype.cast_opaque_ptr(lltype.Ptr(STRUCT), struct)
+ FIELDTYPE = getattr(STRUCT, fieldname)
+ newvalue = cast_func(FIELDTYPE, newvalue)
+ setattr(ptr, fieldname, newvalue)
+ return do_setfield_gc
+do_setfield_gc_int = new_setfield_gc(cast_from_int)
+do_setfield_gc_float = new_setfield_gc(cast_from_floatstorage)
+do_setfield_gc_ptr = new_setfield_gc(cast_from_ptr)
-def do_setfield_gc_float(struct, fieldnum, newvalue):
- STRUCT, fieldname = symbolic.TokenToField[fieldnum]
- ptr = lltype.cast_opaque_ptr(lltype.Ptr(STRUCT), struct)
- FIELDTYPE = getattr(STRUCT, fieldname)
- newvalue = cast_from_floatstorage(FIELDTYPE, newvalue)
- setattr(ptr, fieldname, newvalue)
-
-def do_setfield_gc_ptr(struct, fieldnum, newvalue):
- STRUCT, fieldname = symbolic.TokenToField[fieldnum]
- ptr = lltype.cast_opaque_ptr(lltype.Ptr(STRUCT), struct)
- FIELDTYPE = getattr(STRUCT, fieldname)
- newvalue = cast_from_ptr(FIELDTYPE, newvalue)
- setattr(ptr, fieldname, newvalue)
+def new_setinteriorfield_gc(cast_func):
+ def do_setinteriorfield_gc(array, index, fieldnum, newvalue):
+ STRUCT, fieldname = symbolic.TokenToField[fieldnum]
+ struct = array._obj.container.getitem(index)
+ FIELDTYPE = getattr(STRUCT, fieldname)
+ setattr(struct, fieldname, cast_func(FIELDTYPE, newvalue))
+ return do_setinteriorfield_gc
+do_setinteriorfield_gc_int = new_setinteriorfield_gc(cast_from_int)
+do_setinteriorfield_gc_float = new_setinteriorfield_gc(cast_from_floatstorage)
+do_setinteriorfield_gc_ptr = new_setinteriorfield_gc(cast_from_ptr)
def do_setfield_raw_int(struct, fieldnum, newvalue):
STRUCT, fieldname = symbolic.TokenToField[fieldnum]
@@ -1696,6 +1743,7 @@
setannotation(compile_start_float_var, annmodel.SomeInteger())
setannotation(compile_add, annmodel.s_None)
setannotation(compile_add_descr, annmodel.s_None)
+setannotation(compile_add_descr_arg, annmodel.s_None)
setannotation(compile_add_var, annmodel.s_None)
setannotation(compile_add_int_const, annmodel.s_None)
setannotation(compile_add_ref_const, annmodel.s_None)
@@ -1743,6 +1791,9 @@
setannotation(do_getfield_raw_int, annmodel.SomeInteger())
setannotation(do_getfield_raw_ptr, annmodel.SomePtr(llmemory.GCREF))
setannotation(do_getfield_raw_float, s_FloatStorage)
+setannotation(do_getinteriorfield_gc_int, annmodel.SomeInteger())
+setannotation(do_getinteriorfield_gc_ptr, annmodel.SomePtr(llmemory.GCREF))
+setannotation(do_getinteriorfield_gc_float, s_FloatStorage)
setannotation(do_new, annmodel.SomePtr(llmemory.GCREF))
setannotation(do_new_array, annmodel.SomePtr(llmemory.GCREF))
setannotation(do_setarrayitem_gc_int, annmodel.s_None)
@@ -1756,6 +1807,9 @@
setannotation(do_setfield_raw_int, annmodel.s_None)
setannotation(do_setfield_raw_ptr, annmodel.s_None)
setannotation(do_setfield_raw_float, annmodel.s_None)
+setannotation(do_setinteriorfield_gc_int, annmodel.s_None)
+setannotation(do_setinteriorfield_gc_ptr, annmodel.s_None)
+setannotation(do_setinteriorfield_gc_float, annmodel.s_None)
setannotation(do_newstr, annmodel.SomePtr(llmemory.GCREF))
setannotation(do_strsetitem, annmodel.s_None)
setannotation(do_newunicode, annmodel.SomePtr(llmemory.GCREF))
diff --git a/pypy/jit/backend/llgraph/runner.py b/pypy/jit/backend/llgraph/runner.py
--- a/pypy/jit/backend/llgraph/runner.py
+++ b/pypy/jit/backend/llgraph/runner.py
@@ -2,21 +2,19 @@
Minimal-API wrapper around the llinterpreter to run operations.
"""
-import sys
from pypy.rlib.unroll import unrolling_iterable
from pypy.rlib.objectmodel import we_are_translated
from pypy.rpython.lltypesystem import lltype, llmemory, rclass
from pypy.rpython.ootypesystem import ootype
from pypy.rpython.llinterp import LLInterpreter
from pypy.jit.metainterp import history
-from pypy.jit.metainterp.history import REF, INT, FLOAT
+from pypy.jit.metainterp.history import REF, INT, FLOAT, STRUCT
from pypy.jit.metainterp.warmstate import unwrap
-from pypy.jit.metainterp.resoperation import ResOperation, rop
+from pypy.jit.metainterp.resoperation import rop
from pypy.jit.backend import model
from pypy.jit.backend.llgraph import llimpl, symbolic
from pypy.jit.metainterp.typesystem import llhelper, oohelper
from pypy.jit.codewriter import heaptracker, longlong
-from pypy.rlib import rgc
class MiniStats:
pass
@@ -62,6 +60,9 @@
def is_array_of_floats(self):
return self.typeinfo == FLOAT
+ def is_array_of_structs(self):
+ return self.typeinfo == STRUCT
+
def as_vtable_size_descr(self):
return self
@@ -177,8 +178,10 @@
llimpl.compile_add(c, op.getopnum())
descr = op.getdescr()
if isinstance(descr, Descr):
- llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo, descr.arg_types)
- if isinstance(descr, history.LoopToken) and op.getopnum() != rop.JUMP:
+ llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo,
+ descr.arg_types)
+ if (isinstance(descr, history.LoopToken) and
+ op.getopnum() != rop.JUMP):
llimpl.compile_add_loop_token(c, descr)
if self.is_oo and isinstance(descr, (OODescr, MethDescr)):
# hack hack, not rpython
@@ -193,6 +196,9 @@
llimpl.compile_add_ref_const(c, x.value, self.ts.BASETYPE)
elif isinstance(x, history.ConstFloat):
llimpl.compile_add_float_const(c, x.value)
+ elif isinstance(x, Descr):
+ llimpl.compile_add_descr_arg(c, x.ofs, x.typeinfo,
+ x.arg_types)
else:
raise Exception("'%s' args contain: %r" % (op.getopname(),
x))
@@ -316,6 +322,13 @@
token = history.getkind(getattr(S, fieldname))
return self.getdescr(ofs, token[0], name=fieldname)
+ def interiorfielddescrof(self, A, fieldname):
+ S = A.OF
+ ofs2 = symbolic.get_size(A)
+ ofs, size = symbolic.get_field_token(S, fieldname)
+ token = history.getkind(getattr(S, fieldname))
+ return self.getdescr(ofs, token[0], name=fieldname, extrainfo=ofs2)
+
def calldescrof(self, FUNC, ARGS, RESULT, extrainfo):
arg_types = []
for ARG in ARGS:
@@ -353,8 +366,13 @@
def arraydescrof(self, A):
assert A.OF != lltype.Void
size = symbolic.get_size(A)
- token = history.getkind(A.OF)
- return self.getdescr(size, token[0])
+ if isinstance(A.OF, lltype.Ptr) or isinstance(A.OF, lltype.Primitive):
+ token = history.getkind(A.OF)[0]
+ elif isinstance(A.OF, lltype.Struct):
+ token = 's'
+ else:
+ token = '?'
+ return self.getdescr(size, token)
# ---------- the backend-dependent operations ----------
@@ -406,6 +424,29 @@
assert isinstance(fielddescr, Descr)
return llimpl.do_getfield_raw_float(struct, fielddescr.ofs)
+ def bh_getinteriorfield_gc_i(self, array, index, descr):
+ assert isinstance(descr, Descr)
+ return llimpl.do_getinteriorfield_gc_int(array, index, descr.ofs)
+ def bh_getinteriorfield_gc_r(self, array, index, descr):
+ assert isinstance(descr, Descr)
+ return llimpl.do_getinteriorfield_gc_ptr(array, index, descr.ofs)
+ def bh_getinteriorfield_gc_f(self, array, index, descr):
+ assert isinstance(descr, Descr)
+ return llimpl.do_getinteriorfield_gc_float(array, index, descr.ofs)
+
+ def bh_setinteriorfield_gc_i(self, array, index, descr, value):
+ assert isinstance(descr, Descr)
+ return llimpl.do_setinteriorfield_gc_int(array, index, descr.ofs,
+ value)
+ def bh_setinteriorfield_gc_r(self, array, index, descr, value):
+ assert isinstance(descr, Descr)
+ return llimpl.do_setinteriorfield_gc_ptr(array, index, descr.ofs,
+ value)
+ def bh_setinteriorfield_gc_f(self, array, index, descr, value):
+ assert isinstance(descr, Descr)
+ return llimpl.do_setinteriorfield_gc_float(array, index, descr.ofs,
+ value)
+
def bh_new(self, sizedescr):
assert isinstance(sizedescr, Descr)
return llimpl.do_new(sizedescr.ofs)
@@ -418,7 +459,6 @@
def bh_classof(self, struct):
struct = lltype.cast_opaque_ptr(rclass.OBJECTPTR, struct)
- result = struct.typeptr
result_adr = llmemory.cast_ptr_to_adr(struct.typeptr)
return heaptracker.adr2int(result_adr)
diff --git a/pypy/jit/backend/llsupport/descr.py b/pypy/jit/backend/llsupport/descr.py
--- a/pypy/jit/backend/llsupport/descr.py
+++ b/pypy/jit/backend/llsupport/descr.py
@@ -1,13 +1,10 @@
import py
-from pypy.rpython.lltypesystem import lltype, rffi, llmemory, rclass
+from pypy.rpython.lltypesystem import lltype, rffi, llmemory
from pypy.rpython.lltypesystem.lloperation import llop
from pypy.jit.backend.llsupport import symbolic, support
-from pypy.jit.metainterp.history import AbstractDescr, getkind, BoxInt, BoxPtr
-from pypy.jit.metainterp.history import BasicFailDescr, LoopToken, BoxFloat
+from pypy.jit.metainterp.history import AbstractDescr, getkind
from pypy.jit.metainterp import history
-from pypy.jit.metainterp.resoperation import ResOperation, rop
from pypy.jit.codewriter import heaptracker, longlong
-from pypy.rlib.rarithmetic import r_longlong, r_ulonglong
# The point of the class organization in this file is to make instances
# as compact as possible. This is done by not storing the field size or
@@ -23,6 +20,7 @@
self._cache_field = {}
self._cache_array = {}
self._cache_call = {}
+ self._cache_interiorfield = {}
def init_size_descr(self, STRUCT, sizedescr):
assert isinstance(STRUCT, lltype.GcStruct)
@@ -142,7 +140,6 @@
cachedict[fieldname] = fielddescr
return fielddescr
-
# ____________________________________________________________
# ArrayDescrs
@@ -167,6 +164,7 @@
_is_array_of_pointers = False # unless overridden by GcPtrArrayDescr
_is_array_of_floats = False # unless overridden by FloatArrayDescr
+ _is_array_of_structs = False # unless overridden by StructArrayDescr
_is_item_signed = False # unless overridden by XxxArrayDescr
def is_array_of_pointers(self):
@@ -175,6 +173,9 @@
def is_array_of_floats(self):
return self._is_array_of_floats
+ def is_array_of_structs(self):
+ return self._is_array_of_structs
+
def is_item_signed(self):
return self._is_item_signed
@@ -199,6 +200,10 @@
def get_item_size(self, translate_support_code):
return symbolic.get_size(lltype.Float, translate_support_code)
+class StructArrayDescr(BaseArrayDescr):
+ _clsname = 'StructArrayDescr'
+ _is_array_of_structs = True
+
class BaseArrayNoLengthDescr(BaseArrayDescr):
def get_base_size(self, translate_support_code):
return 0
@@ -218,6 +223,13 @@
def getArrayDescrClass(ARRAY):
if ARRAY.OF is lltype.Float:
return FloatArrayDescr
+ elif isinstance(ARRAY.OF, lltype.Struct):
+ class Descr(StructArrayDescr):
+ _clsname = '%sArrayDescr' % ARRAY.OF._name
+ def get_item_size(self, translate_support_code):
+ return symbolic.get_size(ARRAY.OF, translate_support_code)
+ Descr.__name__ = Descr._clsname
+ return Descr
return getDescrClass(ARRAY.OF, BaseArrayDescr, GcPtrArrayDescr,
NonGcPtrArrayDescr, 'Array', 'get_item_size',
'_is_array_of_floats', '_is_item_signed')
@@ -252,6 +264,39 @@
cache[ARRAY] = arraydescr
return arraydescr
+# ____________________________________________________________
+# InteriorFieldDescr
+
+class InteriorFieldDescr(AbstractDescr):
+ arraydescr = BaseArrayDescr() # workaround for the annotator
+ fielddescr = BaseFieldDescr('', 0)
+
+ def __init__(self, arraydescr, fielddescr):
+ self.arraydescr = arraydescr
+ self.fielddescr = fielddescr
+
+ def is_pointer_field(self):
+ return self.fielddescr.is_pointer_field()
+
+ def is_float_field(self):
+ return self.fielddescr.is_float_field()
+
+ def sort_key(self):
+ return self.fielddescr.sort_key()
+
+ def repr_of_descr(self):
+ return '<InteriorFieldDescr %s>' % self.fielddescr.repr_of_descr()
+
+def get_interiorfield_descr(gc_ll_descr, ARRAY, FIELDTP, name):
+ cache = gc_ll_descr._cache_interiorfield
+ try:
+ return cache[(ARRAY, FIELDTP, name)]
+ except KeyError:
+ arraydescr = get_array_descr(gc_ll_descr, ARRAY)
+ fielddescr = get_field_descr(gc_ll_descr, FIELDTP, name)
+ descr = InteriorFieldDescr(arraydescr, fielddescr)
+ cache[(ARRAY, FIELDTP, name)] = descr
+ return descr
# ____________________________________________________________
# CallDescrs
@@ -525,7 +570,8 @@
#
if TYPE is lltype.Float or is_longlong(TYPE):
setattr(Descr, floatattrname, True)
- elif TYPE is not lltype.Bool and rffi.cast(TYPE, -1) == -1:
+ elif (TYPE is not lltype.Bool and isinstance(TYPE, lltype.Number) and
+ rffi.cast(TYPE, -1) == -1):
setattr(Descr, signedattrname, True)
#
_cache[nameprefix, TYPE] = Descr
diff --git a/pypy/jit/backend/llsupport/gc.py b/pypy/jit/backend/llsupport/gc.py
--- a/pypy/jit/backend/llsupport/gc.py
+++ b/pypy/jit/backend/llsupport/gc.py
@@ -45,6 +45,22 @@
def freeing_block(self, start, stop):
pass
+ def get_funcptr_for_newarray(self):
+ return llhelper(self.GC_MALLOC_ARRAY, self.malloc_array)
+ def get_funcptr_for_newstr(self):
+ return llhelper(self.GC_MALLOC_STR_UNICODE, self.malloc_str)
+ def get_funcptr_for_newunicode(self):
+ return llhelper(self.GC_MALLOC_STR_UNICODE, self.malloc_unicode)
+
+
+ def record_constptrs(self, op, gcrefs_output_list):
+ for i in range(op.numargs()):
+ v = op.getarg(i)
+ if isinstance(v, ConstPtr) and bool(v.value):
+ p = v.value
+ rgc._make_sure_does_not_move(p)
+ gcrefs_output_list.append(p)
+
# ____________________________________________________________
class GcLLDescr_boehm(GcLLDescription):
@@ -88,6 +104,39 @@
malloc_fn_ptr = self.configure_boehm_once()
self.funcptr_for_new = malloc_fn_ptr
+ def malloc_array(basesize, itemsize, ofs_length, num_elem):
+ try:
+ size = ovfcheck(basesize + ovfcheck(itemsize * num_elem))
+ except OverflowError:
+ return lltype.nullptr(llmemory.GCREF.TO)
+ res = self.funcptr_for_new(size)
+ if not res:
+ return res
+ rffi.cast(rffi.CArrayPtr(lltype.Signed), res)[ofs_length/WORD] = num_elem
+ return res
+ self.malloc_array = malloc_array
+ self.GC_MALLOC_ARRAY = lltype.Ptr(lltype.FuncType(
+ [lltype.Signed] * 4, llmemory.GCREF))
+
+
+ (str_basesize, str_itemsize, str_ofs_length
+ ) = symbolic.get_array_token(rstr.STR, self.translate_support_code)
+ (unicode_basesize, unicode_itemsize, unicode_ofs_length
+ ) = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code)
+ def malloc_str(length):
+ return self.malloc_array(
+ str_basesize, str_itemsize, str_ofs_length, length
+ )
+ def malloc_unicode(length):
+ return self.malloc_array(
+ unicode_basesize, unicode_itemsize, unicode_ofs_length, length
+ )
+ self.malloc_str = malloc_str
+ self.malloc_unicode = malloc_unicode
+ self.GC_MALLOC_STR_UNICODE = lltype.Ptr(lltype.FuncType(
+ [lltype.Signed], llmemory.GCREF))
+
+
# on some platform GC_init is required before any other
# GC_* functions, call it here for the benefit of tests
# XXX move this to tests
@@ -108,38 +157,34 @@
ofs_length = arraydescr.get_ofs_length(self.translate_support_code)
basesize = arraydescr.get_base_size(self.translate_support_code)
itemsize = arraydescr.get_item_size(self.translate_support_code)
- size = basesize + itemsize * num_elem
- res = self.funcptr_for_new(size)
- rffi.cast(rffi.CArrayPtr(lltype.Signed), res)[ofs_length/WORD] = num_elem
- return res
+ return self.malloc_array(basesize, itemsize, ofs_length, num_elem)
def gc_malloc_str(self, num_elem):
- basesize, itemsize, ofs_length = symbolic.get_array_token(rstr.STR,
- self.translate_support_code)
- assert itemsize == 1
- size = basesize + num_elem
- res = self.funcptr_for_new(size)
- rffi.cast(rffi.CArrayPtr(lltype.Signed), res)[ofs_length/WORD] = num_elem
- return res
+ return self.malloc_str(num_elem)
def gc_malloc_unicode(self, num_elem):
- basesize, itemsize, ofs_length = symbolic.get_array_token(rstr.UNICODE,
- self.translate_support_code)
- size = basesize + num_elem * itemsize
- res = self.funcptr_for_new(size)
- rffi.cast(rffi.CArrayPtr(lltype.Signed), res)[ofs_length/WORD] = num_elem
- return res
+ return self.malloc_unicode(num_elem)
def args_for_new(self, sizedescr):
assert isinstance(sizedescr, BaseSizeDescr)
return [sizedescr.size]
+ def args_for_new_array(self, arraydescr):
+ ofs_length = arraydescr.get_ofs_length(self.translate_support_code)
+ basesize = arraydescr.get_base_size(self.translate_support_code)
+ itemsize = arraydescr.get_item_size(self.translate_support_code)
+ return [basesize, itemsize, ofs_length]
+
def get_funcptr_for_new(self):
return self.funcptr_for_new
- get_funcptr_for_newarray = None
- get_funcptr_for_newstr = None
- get_funcptr_for_newunicode = None
+ def rewrite_assembler(self, cpu, operations, gcrefs_output_list):
+ # record all GCREFs too, because Boehm cannot see them and keep them
+ # alive if they end up as constants in the assembler
+ for op in operations:
+ self.record_constptrs(op, gcrefs_output_list)
+ return GcLLDescription.rewrite_assembler(self, cpu, operations,
+ gcrefs_output_list)
# ____________________________________________________________
@@ -604,10 +649,13 @@
def malloc_basic(size, tid):
type_id = llop.extract_ushort(llgroup.HALFWORD, tid)
has_finalizer = bool(tid & (1<<llgroup.HALFSHIFT))
+ has_light_finalizer = bool(tid & (1<<(llgroup.HALFSHIFT + 1)))
check_typeid(type_id)
res = llop1.do_malloc_fixedsize_clear(llmemory.GCREF,
type_id, size,
- has_finalizer, False)
+ has_finalizer,
+ has_light_finalizer,
+ False)
# In case the operation above failed, we are returning NULL
# from this function to assembler. There is also an RPython
# exception set, typically MemoryError; but it's easier and
@@ -678,7 +726,7 @@
# also use it to allocate varsized objects. The tid
# and possibly the length are both set afterward.
gcref = llop1.do_malloc_fixedsize_clear(llmemory.GCREF,
- 0, size, False, False)
+ 0, size, False, False, False)
return rffi.cast(lltype.Signed, gcref)
self.malloc_slowpath = malloc_slowpath
self.MALLOC_SLOWPATH = lltype.FuncType([lltype.Signed], lltype.Signed)
@@ -702,7 +750,9 @@
type_id = self.layoutbuilder.get_type_id(S)
assert not self.layoutbuilder.is_weakref_type(S)
has_finalizer = bool(self.layoutbuilder.has_finalizer(S))
- flags = int(has_finalizer) << llgroup.HALFSHIFT
+ has_light_finalizer = bool(self.layoutbuilder.has_light_finalizer(S))
+ flags = (int(has_finalizer) << llgroup.HALFSHIFT |
+ int(has_light_finalizer) << (llgroup.HALFSHIFT + 1))
descr.tid = llop.combine_ushort(lltype.Signed, type_id, flags)
def init_array_descr(self, A, descr):
@@ -736,15 +786,6 @@
def get_funcptr_for_new(self):
return llhelper(self.GC_MALLOC_BASIC, self.malloc_basic)
- def get_funcptr_for_newarray(self):
- return llhelper(self.GC_MALLOC_ARRAY, self.malloc_array)
-
- def get_funcptr_for_newstr(self):
- return llhelper(self.GC_MALLOC_STR_UNICODE, self.malloc_str)
-
- def get_funcptr_for_newunicode(self):
- return llhelper(self.GC_MALLOC_STR_UNICODE, self.malloc_unicode)
-
def do_write_barrier(self, gcref_struct, gcref_newptr):
hdr_addr = llmemory.cast_ptr_to_adr(gcref_struct)
hdr_addr -= self.gcheaderbuilder.size_gc_header
@@ -757,14 +798,6 @@
funcptr(llmemory.cast_ptr_to_adr(gcref_struct),
llmemory.cast_ptr_to_adr(gcref_newptr))
- def record_constptrs(self, op, gcrefs_output_list):
- for i in range(op.numargs()):
- v = op.getarg(i)
- if isinstance(v, ConstPtr) and bool(v.value):
- p = v.value
- rgc._make_sure_does_not_move(p)
- gcrefs_output_list.append(p)
-
def rewrite_assembler(self, cpu, operations, gcrefs_output_list):
# Perform two kinds of rewrites in parallel:
#
diff --git a/pypy/jit/backend/llsupport/llmodel.py b/pypy/jit/backend/llsupport/llmodel.py
--- a/pypy/jit/backend/llsupport/llmodel.py
+++ b/pypy/jit/backend/llsupport/llmodel.py
@@ -1,24 +1,18 @@
-import sys
from pypy.rpython.lltypesystem import lltype, llmemory, rffi, rclass, rstr
from pypy.rpython.lltypesystem.lloperation import llop
-from pypy.rpython.llinterp import LLInterpreter, LLException
+from pypy.rpython.llinterp import LLInterpreter
from pypy.rpython.annlowlevel import llhelper
from pypy.rlib.objectmodel import we_are_translated, specialize
-from pypy.jit.metainterp.history import BoxInt, BoxPtr, set_future_values,\
- BoxFloat
from pypy.jit.metainterp import history
from pypy.jit.codewriter import heaptracker, longlong
from pypy.jit.backend.model import AbstractCPU
from pypy.jit.backend.llsupport import symbolic
from pypy.jit.backend.llsupport.symbolic import WORD, unroll_basic_sizes
-from pypy.jit.backend.llsupport.descr import get_size_descr, BaseSizeDescr
-from pypy.jit.backend.llsupport.descr import get_field_descr, BaseFieldDescr
-from pypy.jit.backend.llsupport.descr import get_array_descr, BaseArrayDescr
-from pypy.jit.backend.llsupport.descr import get_call_descr
-from pypy.jit.backend.llsupport.descr import BaseIntCallDescr, GcPtrCallDescr
-from pypy.jit.backend.llsupport.descr import FloatCallDescr, VoidCallDescr
+from pypy.jit.backend.llsupport.descr import (get_size_descr,
+ get_field_descr, BaseFieldDescr, get_array_descr, BaseArrayDescr,
+ get_call_descr, BaseIntCallDescr, GcPtrCallDescr, FloatCallDescr,
+ VoidCallDescr, InteriorFieldDescr, get_interiorfield_descr)
from pypy.jit.backend.llsupport.asmmemmgr import AsmMemoryManager
-from pypy.rpython.annlowlevel import cast_instance_to_base_ptr
class AbstractLLCPU(AbstractCPU):
@@ -241,6 +235,9 @@
def arraydescrof(self, A):
return get_array_descr(self.gc_ll_descr, A)
+ def interiorfielddescrof(self, A, fieldname):
+ return get_interiorfield_descr(self.gc_ll_descr, A, A.OF, fieldname)
+
def unpack_arraydescr(self, arraydescr):
assert isinstance(arraydescr, BaseArrayDescr)
return arraydescr.get_base_size(self.translate_support_code)
@@ -358,6 +355,100 @@
bh_getarrayitem_raw_i = bh_getarrayitem_gc_i
bh_getarrayitem_raw_f = bh_getarrayitem_gc_f
+ def bh_getinteriorfield_gc_i(self, gcref, itemindex, descr):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ fieldsize = descr.fielddescr.get_field_size(self.translate_support_code)
+ sign = descr.fielddescr.is_field_signed()
+ fullofs = itemindex * size + ofs
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), fullofs)
+ for STYPE, UTYPE, itemsize in unroll_basic_sizes:
+ if fieldsize == itemsize:
+ if sign:
+ item = rffi.cast(rffi.CArrayPtr(STYPE), items)
+ val = item[0]
+ val = rffi.cast(lltype.Signed, val)
+ else:
+ item = rffi.cast(rffi.CArrayPtr(UTYPE), items)
+ val = item[0]
+ val = rffi.cast(lltype.Signed, val)
+ # --- end of GC unsafe code ---
+ return val
+ else:
+ raise NotImplementedError("size = %d" % fieldsize)
+
+ def bh_getinteriorfield_gc_r(self, gcref, itemindex, descr):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs +
+ size * itemindex)
+ items = rffi.cast(rffi.CArrayPtr(lltype.Signed), items)
+ pval = self._cast_int_to_gcref(items[0])
+ # --- end of GC unsafe code ---
+ return pval
+
+ def bh_getinteriorfield_gc_f(self, gcref, itemindex, descr):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs +
+ size * itemindex)
+ items = rffi.cast(rffi.CArrayPtr(longlong.FLOATSTORAGE), items)
+ fval = items[0]
+ # --- end of GC unsafe code ---
+ return fval
+
+ def bh_setinteriorfield_gc_i(self, gcref, itemindex, descr, value):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ fieldsize = descr.fielddescr.get_field_size(self.translate_support_code)
+ ofs = itemindex * size + ofs
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
+ for TYPE, _, itemsize in unroll_basic_sizes:
+ if fieldsize == itemsize:
+ items = rffi.cast(rffi.CArrayPtr(TYPE), items)
+ items[0] = rffi.cast(TYPE, value)
+ # --- end of GC unsafe code ---
+ return
+ else:
+ raise NotImplementedError("size = %d" % fieldsize)
+
+ def bh_setinteriorfield_gc_r(self, gcref, itemindex, descr, newvalue):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ self.gc_ll_descr.do_write_barrier(gcref, newvalue)
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref),
+ ofs + size * itemindex)
+ items = rffi.cast(rffi.CArrayPtr(lltype.Signed), items)
+ items[0] = self.cast_gcref_to_int(newvalue)
+ # --- end of GC unsafe code ---
+
+ def bh_setinteriorfield_gc_f(self, gcref, itemindex, descr, newvalue):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs, size, _ = self.unpack_arraydescr_size(arraydescr)
+ ofs += descr.fielddescr.offset
+ # --- start of GC unsafe code (no GC operation!) ---
+ items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref),
+ ofs + size * itemindex)
+ items = rffi.cast(rffi.CArrayPtr(longlong.FLOATSTORAGE), items)
+ items[0] = newvalue
+ # --- end of GC unsafe code ---
+
def bh_strlen(self, string):
s = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), string)
return len(s.chars)
@@ -475,7 +566,6 @@
def bh_classof(self, struct):
struct = lltype.cast_opaque_ptr(rclass.OBJECTPTR, struct)
- result = struct.typeptr
result_adr = llmemory.cast_ptr_to_adr(struct.typeptr)
return heaptracker.adr2int(result_adr)
diff --git a/pypy/jit/backend/llsupport/regalloc.py b/pypy/jit/backend/llsupport/regalloc.py
--- a/pypy/jit/backend/llsupport/regalloc.py
+++ b/pypy/jit/backend/llsupport/regalloc.py
@@ -287,7 +287,6 @@
self.reg_bindings[to_v] = reg
def _move_variable_away(self, v, prev_loc):
- reg = None
if self.free_regs:
loc = self.free_regs.pop()
self.reg_bindings[v] = loc
diff --git a/pypy/jit/backend/llsupport/test/test_asmmemmgr.py b/pypy/jit/backend/llsupport/test/test_asmmemmgr.py
--- a/pypy/jit/backend/llsupport/test/test_asmmemmgr.py
+++ b/pypy/jit/backend/llsupport/test/test_asmmemmgr.py
@@ -211,14 +211,14 @@
debug._log = debug.DebugLog()
try:
mc._dump(addr, 'test-logname-section')
- log = list(debug._log)
+ log = list(debug._log)
finally:
debug._log = None
encoded = ''.join(writtencode).encode('hex').upper()
ataddr = '@%x' % addr
assert log == [('test-logname-section',
[('debug_print', 'CODE_DUMP', ataddr, '+0 ', encoded)])]
- #
+
lltype.free(p, flavor='raw')
def test_blockbuildermixin2():
diff --git a/pypy/jit/backend/llsupport/test/test_descr.py b/pypy/jit/backend/llsupport/test/test_descr.py
--- a/pypy/jit/backend/llsupport/test/test_descr.py
+++ b/pypy/jit/backend/llsupport/test/test_descr.py
@@ -3,7 +3,6 @@
from pypy.jit.backend.llsupport import symbolic
from pypy.rlib.objectmodel import Symbolic
from pypy.rpython.annlowlevel import llhelper
-from pypy.jit.metainterp.history import BoxInt, BoxFloat, BoxPtr
from pypy.jit.metainterp import history
from pypy.jit.codewriter import longlong
import sys, struct, py
@@ -149,7 +148,9 @@
A2 = lltype.GcArray(lltype.Ptr(T))
A3 = lltype.GcArray(lltype.Ptr(U))
A4 = lltype.GcArray(lltype.Float)
- A5 = lltype.GcArray(lltype.SingleFloat)
+ A5 = lltype.GcArray(lltype.Struct('x', ('v', lltype.Signed),
+ ('k', lltype.Signed)))
+ A6 = lltype.GcArray(lltype.SingleFloat)
assert getArrayDescrClass(A2) is GcPtrArrayDescr
assert getArrayDescrClass(A3) is NonGcPtrArrayDescr
cls = getArrayDescrClass(A1)
@@ -158,7 +159,7 @@
clsf = getArrayDescrClass(A4)
assert clsf != cls
assert clsf == getArrayDescrClass(lltype.GcArray(lltype.Float))
- clss = getArrayDescrClass(A5)
+ clss = getArrayDescrClass(A6)
assert clss not in (clsf, cls)
assert clss == getArrayDescrClass(lltype.GcArray(rffi.UINT))
#
@@ -168,11 +169,12 @@
descr3 = get_array_descr(c0, A3)
descr4 = get_array_descr(c0, A4)
descr5 = get_array_descr(c0, A5)
+ descr6 = get_array_descr(c0, A6)
assert descr1.__class__ is cls
assert descr2.__class__ is GcPtrArrayDescr
assert descr3.__class__ is NonGcPtrArrayDescr
assert descr4.__class__ is clsf
- assert descr5.__class__ is clss
+ assert descr6.__class__ is clss
assert descr1 == get_array_descr(c0, lltype.GcArray(lltype.Char))
assert not descr1.is_array_of_pointers()
assert descr2.is_array_of_pointers()
@@ -202,7 +204,8 @@
assert descr2.get_item_size(False) == rffi.sizeof(lltype.Ptr(T))
assert descr3.get_item_size(False) == rffi.sizeof(lltype.Ptr(U))
assert descr4.get_item_size(False) == rffi.sizeof(lltype.Float)
- assert descr5.get_item_size(False) == rffi.sizeof(lltype.SingleFloat)
+ assert descr5.get_item_size(False) == rffi.sizeof(lltype.Signed) * 2
+ assert descr6.get_item_size(False) == rffi.sizeof(lltype.SingleFloat)
#
assert isinstance(descr1.get_base_size(True), Symbolic)
assert isinstance(descr2.get_base_size(True), Symbolic)
@@ -348,7 +351,6 @@
(rffi.SHORT, True), (rffi.USHORT, False),
(rffi.INT, True), (rffi.UINT, False),
(rffi.LONG, True), (rffi.ULONG, False)]:
- A = lltype.GcArray(RESTYPE)
for tsc in [False, True]:
c2 = GcCache(tsc)
descr1 = get_call_descr(c2, [], RESTYPE)
@@ -379,7 +381,6 @@
descr3i = get_array_descr(c0, lltype.GcArray(lltype.Char))
assert descr3i.repr_of_descr() == '<CharArrayDescr>'
#
- cache = {}
descr4 = get_call_descr(c0, [lltype.Char, lltype.Ptr(S)], lltype.Ptr(S))
assert 'GcPtrCallDescr' in descr4.repr_of_descr()
#
@@ -412,10 +413,10 @@
ARGS = [lltype.Float, lltype.Ptr(ARRAY)]
RES = lltype.Float
- def f(a, b):
+ def f2(a, b):
return float(b[0]) + a
- fnptr = llhelper(lltype.Ptr(lltype.FuncType(ARGS, RES)), f)
+ fnptr = llhelper(lltype.Ptr(lltype.FuncType(ARGS, RES)), f2)
descr2 = get_call_descr(c0, ARGS, RES)
a = lltype.malloc(ARRAY, 3)
opaquea = lltype.cast_opaque_ptr(llmemory.GCREF, a)
diff --git a/pypy/jit/backend/llsupport/test/test_gc.py b/pypy/jit/backend/llsupport/test/test_gc.py
--- a/pypy/jit/backend/llsupport/test/test_gc.py
+++ b/pypy/jit/backend/llsupport/test/test_gc.py
@@ -247,12 +247,14 @@
self.record = []
def do_malloc_fixedsize_clear(self, RESTYPE, type_id, size,
- has_finalizer, contains_weakptr):
+ has_finalizer, has_light_finalizer,
+ contains_weakptr):
assert not contains_weakptr
+ assert not has_finalizer # in these tests
+ assert not has_light_finalizer # in these tests
p = llmemory.raw_malloc(size)
p = llmemory.cast_adr_to_ptr(p, RESTYPE)
- flags = int(has_finalizer) << 16
- tid = llop.combine_ushort(lltype.Signed, type_id, flags)
+ tid = llop.combine_ushort(lltype.Signed, type_id, 0)
self.record.append(("fixedsize", repr(size), tid, p))
return p
diff --git a/pypy/jit/backend/model.py b/pypy/jit/backend/model.py
--- a/pypy/jit/backend/model.py
+++ b/pypy/jit/backend/model.py
@@ -1,5 +1,5 @@
from pypy.rlib.debug import debug_start, debug_print, debug_stop
-from pypy.jit.metainterp import history, compile
+from pypy.jit.metainterp import history
class AbstractCPU(object):
@@ -213,6 +213,10 @@
def typedescrof(TYPE):
raise NotImplementedError
+ @staticmethod
+ def interiorfielddescrof(A, fieldname):
+ raise NotImplementedError
+
# ---------- the backend-dependent operations ----------
# lltype specific operations
diff --git a/pypy/jit/backend/test/runner_test.py b/pypy/jit/backend/test/runner_test.py
--- a/pypy/jit/backend/test/runner_test.py
+++ b/pypy/jit/backend/test/runner_test.py
@@ -5,7 +5,7 @@
BoxInt, Box, BoxPtr,
LoopToken,
ConstInt, ConstPtr,
- BoxObj, Const,
+ BoxObj,
ConstObj, BoxFloat, ConstFloat)
from pypy.jit.metainterp.resoperation import ResOperation, rop
from pypy.jit.metainterp.typesystem import deref
@@ -111,7 +111,7 @@
self.cpu.set_future_value_int(0, 2)
fail = self.cpu.execute_token(looptoken)
res = self.cpu.get_latest_value_int(0)
- assert res == 3
+ assert res == 3
assert fail.identifier == 1
def test_compile_loop(self):
@@ -127,7 +127,7 @@
]
inputargs = [i0]
operations[2].setfailargs([i1])
-
+
self.cpu.compile_loop(inputargs, operations, looptoken)
self.cpu.set_future_value_int(0, 2)
fail = self.cpu.execute_token(looptoken)
@@ -148,7 +148,7 @@
]
inputargs = [i0]
operations[2].setfailargs([None, None, i1, None])
-
+
self.cpu.compile_loop(inputargs, operations, looptoken)
self.cpu.set_future_value_int(0, 2)
fail = self.cpu.execute_token(looptoken)
@@ -372,7 +372,7 @@
for opnum, boxargs, retvalue in get_int_tests():
res = self.execute_operation(opnum, boxargs, 'int')
assert res.value == retvalue
-
+
def test_float_operations(self):
from pypy.jit.metainterp.test.test_executor import get_float_tests
for opnum, boxargs, rettype, retvalue in get_float_tests(self.cpu):
@@ -438,7 +438,7 @@
def test_ovf_operations_reversed(self):
self.test_ovf_operations(reversed=True)
-
+
def test_bh_call(self):
cpu = self.cpu
#
@@ -503,7 +503,7 @@
[funcbox, BoxInt(num), BoxInt(num)],
'int', descr=dyn_calldescr)
assert res.value == 2 * num
-
+
if cpu.supports_floats:
def func(f0, f1, f2, f3, f4, f5, f6, i0, i1, f7, f8, f9):
@@ -543,7 +543,7 @@
funcbox = self.get_funcbox(self.cpu, func_ptr)
res = self.execute_operation(rop.CALL, [funcbox] + map(BoxInt, args), 'int', descr=calldescr)
assert res.value == func(*args)
-
+
def test_call_stack_alignment(self):
# test stack alignment issues, notably for Mac OS/X.
# also test the ordering of the arguments.
@@ -615,7 +615,7 @@
res = self.execute_operation(rop.GETFIELD_GC, [t_box],
'int', descr=shortdescr)
assert res.value == 1331
-
+
#
u_box, U_box = self.alloc_instance(self.U)
fielddescr2 = self.cpu.fielddescrof(self.S, 'next')
@@ -695,7 +695,7 @@
def test_failing_guard_class(self):
t_box, T_box = self.alloc_instance(self.T)
- u_box, U_box = self.alloc_instance(self.U)
+ u_box, U_box = self.alloc_instance(self.U)
null_box = self.null_instance()
for opname, args in [(rop.GUARD_CLASS, [t_box, U_box]),
(rop.GUARD_CLASS, [u_box, T_box]),
@@ -787,7 +787,7 @@
r = self.execute_operation(rop.GETARRAYITEM_GC, [a_box, BoxInt(3)],
'int', descr=arraydescr)
assert r.value == 160
-
+
#
if isinstance(A, lltype.GcArray):
A = lltype.Ptr(A)
@@ -880,6 +880,73 @@
'int', descr=arraydescr)
assert r.value == 7441
+ def test_array_of_structs(self):
+ TP = lltype.GcStruct('x')
+ ITEM = lltype.Struct('x',
+ ('vs', lltype.Signed),
+ ('vu', lltype.Unsigned),
+ ('vsc', rffi.SIGNEDCHAR),
+ ('vuc', rffi.UCHAR),
+ ('vss', rffi.SHORT),
+ ('vus', rffi.USHORT),
+ ('vsi', rffi.INT),
+ ('vui', rffi.UINT),
+ ('k', lltype.Float),
+ ('p', lltype.Ptr(TP)))
+ a_box, A = self.alloc_array_of(ITEM, 15)
+ s_box, S = self.alloc_instance(TP)
+ kdescr = self.cpu.interiorfielddescrof(A, 'k')
+ pdescr = self.cpu.interiorfielddescrof(A, 'p')
+ self.execute_operation(rop.SETINTERIORFIELD_GC, [a_box, BoxInt(3),
+ boxfloat(1.5)],
+ 'void', descr=kdescr)
+ f = self.cpu.bh_getinteriorfield_gc_f(a_box.getref_base(), 3, kdescr)
+ assert longlong.getrealfloat(f) == 1.5
+ self.cpu.bh_setinteriorfield_gc_f(a_box.getref_base(), 3, kdescr, longlong.getfloatstorage(2.5))
+ r = self.execute_operation(rop.GETINTERIORFIELD_GC, [a_box, BoxInt(3)],
+ 'float', descr=kdescr)
+ assert r.getfloat() == 2.5
+ #
+ NUMBER_FIELDS = [('vs', lltype.Signed),
+ ('vu', lltype.Unsigned),
+ ('vsc', rffi.SIGNEDCHAR),
+ ('vuc', rffi.UCHAR),
+ ('vss', rffi.SHORT),
+ ('vus', rffi.USHORT),
+ ('vsi', rffi.INT),
+ ('vui', rffi.UINT)]
+ for name, TYPE in NUMBER_FIELDS[::-1]:
+ vdescr = self.cpu.interiorfielddescrof(A, name)
+ self.execute_operation(rop.SETINTERIORFIELD_GC, [a_box, BoxInt(3),
+ BoxInt(-15)],
+ 'void', descr=vdescr)
+ for name, TYPE in NUMBER_FIELDS:
+ vdescr = self.cpu.interiorfielddescrof(A, name)
+ i = self.cpu.bh_getinteriorfield_gc_i(a_box.getref_base(), 3,
+ vdescr)
+ assert i == rffi.cast(lltype.Signed, rffi.cast(TYPE, -15))
+ for name, TYPE in NUMBER_FIELDS[::-1]:
+ vdescr = self.cpu.interiorfielddescrof(A, name)
+ self.cpu.bh_setinteriorfield_gc_i(a_box.getref_base(), 3,
+ vdescr, -25)
+ for name, TYPE in NUMBER_FIELDS:
+ vdescr = self.cpu.interiorfielddescrof(A, name)
+ r = self.execute_operation(rop.GETINTERIORFIELD_GC,
+ [a_box, BoxInt(3)],
+ 'int', descr=vdescr)
+ assert r.getint() == rffi.cast(lltype.Signed, rffi.cast(TYPE, -25))
+ #
+ self.execute_operation(rop.SETINTERIORFIELD_GC, [a_box, BoxInt(4),
+ s_box],
+ 'void', descr=pdescr)
+ r = self.cpu.bh_getinteriorfield_gc_r(a_box.getref_base(), 4, pdescr)
+ assert r == s_box.getref_base()
+ self.cpu.bh_setinteriorfield_gc_r(a_box.getref_base(), 3, pdescr,
+ s_box.getref_base())
+ r = self.execute_operation(rop.GETINTERIORFIELD_GC, [a_box, BoxInt(3)],
+ 'ref', descr=pdescr)
+ assert r.getref_base() == s_box.getref_base()
+
def test_string_basic(self):
s_box = self.alloc_string("hello\xfe")
r = self.execute_operation(rop.STRLEN, [s_box], 'int')
@@ -1402,7 +1469,7 @@
addr = llmemory.cast_ptr_to_adr(func_ptr)
return ConstInt(heaptracker.adr2int(addr))
-
+
MY_VTABLE = rclass.OBJECT_VTABLE # for tests only
S = lltype.GcForwardReference()
@@ -1439,7 +1506,6 @@
return BoxPtr(lltype.nullptr(llmemory.GCREF.TO))
def alloc_array_of(self, ITEM, length):
- cpu = self.cpu
A = lltype.GcArray(ITEM)
a = lltype.malloc(A, length)
a_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, a))
@@ -1468,20 +1534,16 @@
return u''.join(u.chars)
- def test_casts(self):
- py.test.skip("xxx fix or kill")
- from pypy.rpython.lltypesystem import lltype, llmemory
- TP = lltype.GcStruct('x')
- x = lltype.malloc(TP)
- x = lltype.cast_opaque_ptr(llmemory.GCREF, x)
+ def test_cast_int_to_ptr(self):
+ res = self.execute_operation(rop.CAST_INT_TO_PTR,
+ [BoxInt(-17)], 'ref').value
+ assert lltype.cast_ptr_to_int(res) == -17
+
+ def test_cast_ptr_to_int(self):
+ x = lltype.cast_int_to_ptr(llmemory.GCREF, -19)
res = self.execute_operation(rop.CAST_PTR_TO_INT,
- [BoxPtr(x)], 'int').value
- expected = self.cpu.cast_adr_to_int(llmemory.cast_ptr_to_adr(x))
- assert rffi.get_real_int(res) == rffi.get_real_int(expected)
- res = self.execute_operation(rop.CAST_PTR_TO_INT,
- [ConstPtr(x)], 'int').value
- expected = self.cpu.cast_adr_to_int(llmemory.cast_ptr_to_adr(x))
- assert rffi.get_real_int(res) == rffi.get_real_int(expected)
+ [BoxPtr(x)], 'int').value
+ assert res == -19
def test_ooops_non_gc(self):
x = lltype.malloc(lltype.Struct('x'), flavor='raw')
@@ -2299,13 +2361,6 @@
#
cpu.bh_strsetitem(x, 4, ord('/'))
assert str.chars[4] == '/'
- #
-## x = cpu.bh_newstr(5)
-## y = cpu.bh_cast_ptr_to_int(x)
-## z = cpu.bh_cast_ptr_to_int(x)
-## y = rffi.get_real_int(y)
-## z = rffi.get_real_int(z)
-## assert type(y) == type(z) == int and y == z
def test_sorting_of_fields(self):
S = self.S
@@ -2329,7 +2384,7 @@
for opname, arg, res in ops:
self.execute_operation(opname, [arg], 'void')
assert self.guard_failed == res
-
+
lltype.free(x, flavor='raw')
def test_assembler_call(self):
@@ -2409,7 +2464,7 @@
FakeJitDriverSD.portal_calldescr = self.cpu.calldescrof(
lltype.Ptr(lltype.FuncType(ARGS, RES)), ARGS, RES,
EffectInfo.MOST_GENERAL)
-
+
ops = '''
[f0, f1]
f2 = float_add(f0, f1)
@@ -2500,7 +2555,7 @@
FakeJitDriverSD.portal_calldescr = self.cpu.calldescrof(
lltype.Ptr(lltype.FuncType(ARGS, RES)), ARGS, RES,
EffectInfo.MOST_GENERAL)
-
+
ops = '''
[f0, f1]
f2 = float_add(f0, f1)
@@ -2951,4 +3006,4 @@
def alloc_unicode(self, unicode):
py.test.skip("implement me")
-
+
diff --git a/pypy/jit/backend/test/test_ll_random.py b/pypy/jit/backend/test/test_ll_random.py
--- a/pypy/jit/backend/test/test_ll_random.py
+++ b/pypy/jit/backend/test/test_ll_random.py
@@ -28,16 +28,27 @@
fork.structure_types_and_vtables = self.structure_types_and_vtables
return fork
- def get_structptr_var(self, r, must_have_vtable=False, type=lltype.Struct):
+ def _choose_ptr_vars(self, from_, type, array_of_structs):
+ ptrvars = []
+ for i in range(len(from_)):
+ v, S = from_[i][:2]
+ if not isinstance(S, type):
+ continue
+ if ((isinstance(S, lltype.Array) and
+ isinstance(S.OF, lltype.Struct)) == array_of_structs):
+ ptrvars.append((v, S))
+ return ptrvars
+
+ def get_structptr_var(self, r, must_have_vtable=False, type=lltype.Struct,
+ array_of_structs=False):
while True:
- ptrvars = [(v, S) for (v, S) in self.ptrvars
- if isinstance(S, type)]
+ ptrvars = self._choose_ptr_vars(self.ptrvars, type,
+ array_of_structs)
if ptrvars and r.random() < 0.8:
v, S = r.choice(ptrvars)
else:
- prebuilt_ptr_consts = [(v, S)
- for (v, S, _) in self.prebuilt_ptr_consts
- if isinstance(S, type)]
+ prebuilt_ptr_consts = self._choose_ptr_vars(
+ self.prebuilt_ptr_consts, type, array_of_structs)
if prebuilt_ptr_consts and r.random() < 0.7:
v, S = r.choice(prebuilt_ptr_consts)
else:
@@ -48,7 +59,8 @@
has_vtable=must_have_vtable)
else:
# create a new constant array
- p = self.get_random_array(r)
+ p = self.get_random_array(r,
+ must_be_array_of_structs=array_of_structs)
S = lltype.typeOf(p).TO
v = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, p))
self.prebuilt_ptr_consts.append((v, S,
@@ -74,7 +86,8 @@
TYPE = lltype.Signed
return TYPE
- def get_random_structure_type(self, r, with_vtable=None, cache=True):
+ def get_random_structure_type(self, r, with_vtable=None, cache=True,
+ type=lltype.GcStruct):
if cache and self.structure_types and r.random() < 0.5:
return r.choice(self.structure_types)
fields = []
@@ -85,7 +98,7 @@
for i in range(r.randrange(1, 5)):
TYPE = self.get_random_primitive_type(r)
fields.append(('f%d' % i, TYPE))
- S = lltype.GcStruct('S%d' % self.counter, *fields, **kwds)
+ S = type('S%d' % self.counter, *fields, **kwds)
self.counter += 1
if cache:
self.structure_types.append(S)
@@ -125,17 +138,29 @@
setattr(p, fieldname, rffi.cast(TYPE, r.random_integer()))
return p
- def get_random_array_type(self, r):
- TYPE = self.get_random_primitive_type(r)
+ def get_random_array_type(self, r, can_be_array_of_struct=False,
+ must_be_array_of_structs=False):
+ if ((can_be_array_of_struct and r.random() < 0.1) or
+ must_be_array_of_structs):
+ TYPE = self.get_random_structure_type(r, cache=False,
+ type=lltype.Struct)
+ else:
+ TYPE = self.get_random_primitive_type(r)
return lltype.GcArray(TYPE)
- def get_random_array(self, r):
- A = self.get_random_array_type(r)
+ def get_random_array(self, r, must_be_array_of_structs=False):
+ A = self.get_random_array_type(r,
+ must_be_array_of_structs=must_be_array_of_structs)
length = (r.random_integer() // 15) % 300 # length: between 0 and 299
# likely to be small
p = lltype.malloc(A, length)
- for i in range(length):
- p[i] = rffi.cast(A.OF, r.random_integer())
+ if isinstance(A.OF, lltype.Primitive):
+ for i in range(length):
+ p[i] = rffi.cast(A.OF, r.random_integer())
+ else:
+ for i in range(length):
+ for fname, TP in A.OF._flds.iteritems():
+ setattr(p[i], fname, rffi.cast(TP, r.random_integer()))
return p
def get_index(self, length, r):
@@ -155,8 +180,16 @@
dic[fieldname] = getattr(p, fieldname)
else:
assert isinstance(S, lltype.Array)
- for i in range(len(p)):
- dic[i] = p[i]
+ if isinstance(S.OF, lltype.Struct):
+ for i in range(len(p)):
+ item = p[i]
+ s1 = {}
+ for fieldname in S.OF._names:
+ s1[fieldname] = getattr(item, fieldname)
+ dic[i] = s1
+ else:
+ for i in range(len(p)):
+ dic[i] = p[i]
return dic
def print_loop_prebuilt(self, names, writevar, s):
@@ -220,7 +253,7 @@
class GetFieldOperation(test_random.AbstractOperation):
def field_descr(self, builder, r):
- v, S = builder.get_structptr_var(r)
+ v, S = builder.get_structptr_var(r, )
names = S._names
if names[0] == 'parent':
names = names[1:]
@@ -239,6 +272,28 @@
continue
break
+class GetInteriorFieldOperation(test_random.AbstractOperation):
+ def field_descr(self, builder, r):
+ v, A = builder.get_structptr_var(r, type=lltype.Array,
+ array_of_structs=True)
+ array = v.getref(lltype.Ptr(A))
+ v_index = builder.get_index(len(array), r)
+ name = r.choice(A.OF._names)
+ descr = builder.cpu.interiorfielddescrof(A, name)
+ descr._random_info = 'cpu.interiorfielddescrof(%s, %r)' % (A.OF._name,
+ name)
+ TYPE = getattr(A.OF, name)
+ return v, v_index, descr, TYPE
+
+ def produce_into(self, builder, r):
+ while True:
+ try:
+ v, v_index, descr, _ = self.field_descr(builder, r)
+ self.put(builder, [v, v_index], descr)
+ except lltype.UninitializedMemoryAccess:
+ continue
+ break
+
class SetFieldOperation(GetFieldOperation):
def produce_into(self, builder, r):
v, descr, TYPE = self.field_descr(builder, r)
@@ -251,6 +306,18 @@
break
builder.do(self.opnum, [v, w], descr)
+class SetInteriorFieldOperation(GetInteriorFieldOperation):
+ def produce_into(self, builder, r):
+ v, v_index, descr, TYPE = self.field_descr(builder, r)
+ while True:
+ if r.random() < 0.3:
+ w = ConstInt(r.random_integer())
+ else:
+ w = r.choice(builder.intvars)
+ if rffi.cast(lltype.Signed, rffi.cast(TYPE, w.value)) == w.value:
+ break
+ builder.do(self.opnum, [v, v_index, w], descr)
+
class NewOperation(test_random.AbstractOperation):
def size_descr(self, builder, S):
descr = builder.cpu.sizeof(S)
@@ -306,7 +373,7 @@
class NewArrayOperation(ArrayOperation):
def produce_into(self, builder, r):
- A = builder.get_random_array_type(r)
+ A = builder.get_random_array_type(r, can_be_array_of_struct=True)
v_size = builder.get_index(300, r)
v_ptr = builder.do(self.opnum, [v_size], self.array_descr(builder, A))
builder.ptrvars.append((v_ptr, A))
@@ -586,7 +653,9 @@
for i in range(4): # make more common
OPERATIONS.append(GetFieldOperation(rop.GETFIELD_GC))
OPERATIONS.append(GetFieldOperation(rop.GETFIELD_GC))
+ OPERATIONS.append(GetInteriorFieldOperation(rop.GETINTERIORFIELD_GC))
OPERATIONS.append(SetFieldOperation(rop.SETFIELD_GC))
+ OPERATIONS.append(SetInteriorFieldOperation(rop.SETINTERIORFIELD_GC))
OPERATIONS.append(NewOperation(rop.NEW))
OPERATIONS.append(NewOperation(rop.NEW_WITH_VTABLE))
diff --git a/pypy/jit/backend/test/test_random.py b/pypy/jit/backend/test/test_random.py
--- a/pypy/jit/backend/test/test_random.py
+++ b/pypy/jit/backend/test/test_random.py
@@ -595,6 +595,10 @@
for name, value in fields.items():
if isinstance(name, str):
setattr(container, name, value)
+ elif isinstance(value, dict):
+ item = container.getitem(name)
+ for key1, value1 in value.items():
+ setattr(item, key1, value1)
else:
container.setitem(name, value)
diff --git a/pypy/jit/backend/x86/assembler.py b/pypy/jit/backend/x86/assembler.py
--- a/pypy/jit/backend/x86/assembler.py
+++ b/pypy/jit/backend/x86/assembler.py
@@ -1,7 +1,7 @@
import sys, os
from pypy.jit.backend.llsupport import symbolic
from pypy.jit.backend.llsupport.asmmemmgr import MachineDataBlockWrapper
-from pypy.jit.metainterp.history import Const, Box, BoxInt, BoxPtr, BoxFloat
+from pypy.jit.metainterp.history import Const, Box, BoxInt, ConstInt
from pypy.jit.metainterp.history import (AbstractFailDescr, INT, REF, FLOAT,
LoopToken)
from pypy.rpython.lltypesystem import lltype, rffi, rstr, llmemory
@@ -36,7 +36,6 @@
from pypy.rlib import rgc
from pypy.rlib.clibffi import FFI_DEFAULT_ABI
from pypy.jit.backend.x86.jump import remap_frame_layout
-from pypy.jit.metainterp.history import ConstInt, BoxInt
from pypy.jit.codewriter.effectinfo import EffectInfo
from pypy.jit.codewriter import longlong
@@ -729,8 +728,8 @@
# Also, make sure this is consistent with FRAME_FIXED_SIZE.
self.mc.PUSH_r(ebp.value)
self.mc.MOV_rr(ebp.value, esp.value)
- for regloc in self.cpu.CALLEE_SAVE_REGISTERS:
- self.mc.PUSH_r(regloc.value)
+ for loc in self.cpu.CALLEE_SAVE_REGISTERS:
+ self.mc.PUSH_r(loc.value)
gcrootmap = self.cpu.gc_ll_descr.gcrootmap
if gcrootmap and gcrootmap.is_shadow_stack:
@@ -994,7 +993,7 @@
effectinfo = op.getdescr().get_extra_info()
oopspecindex = effectinfo.oopspecindex
genop_llong_list[oopspecindex](self, op, arglocs, resloc)
-
+
def regalloc_perform_math(self, op, arglocs, resloc):
effectinfo = op.getdescr().get_extra_info()
oopspecindex = effectinfo.oopspecindex
@@ -1277,8 +1276,8 @@
genop_int_ne = _cmpop("NE", "NE")
genop_int_gt = _cmpop("G", "L")
genop_int_ge = _cmpop("GE", "LE")
- genop_ptr_eq = genop_int_eq
- genop_ptr_ne = genop_int_ne
+ genop_ptr_eq = genop_instance_ptr_eq = genop_int_eq
+ genop_ptr_ne = genop_instance_ptr_ne = genop_int_ne
genop_float_lt = _cmpop_float('B', 'A')
genop_float_le = _cmpop_float('BE', 'AE')
@@ -1298,8 +1297,8 @@
genop_guard_int_ne = _cmpop_guard("NE", "NE", "E", "E")
genop_guard_int_gt = _cmpop_guard("G", "L", "LE", "GE")
genop_guard_int_ge = _cmpop_guard("GE", "LE", "L", "G")
- genop_guard_ptr_eq = genop_guard_int_eq
- genop_guard_ptr_ne = genop_guard_int_ne
+ genop_guard_ptr_eq = genop_guard_instance_ptr_eq = genop_guard_int_eq
+ genop_guard_ptr_ne = genop_guard_instance_ptr_ne = genop_guard_int_ne
genop_guard_uint_gt = _cmpop_guard("A", "B", "BE", "AE")
genop_guard_uint_lt = _cmpop_guard("B", "A", "AE", "BE")
@@ -1311,7 +1310,7 @@
genop_guard_float_eq = _cmpop_guard_float("E", "E", "NE","NE")
genop_guard_float_gt = _cmpop_guard_float("A", "B", "BE","AE")
genop_guard_float_ge = _cmpop_guard_float("AE","BE", "B", "A")
-
+
def genop_math_sqrt(self, op, arglocs, resloc):
self.mc.SQRTSD(arglocs[0], resloc)
@@ -1387,7 +1386,8 @@
def genop_same_as(self, op, arglocs, resloc):
self.mov(arglocs[0], resloc)
- #genop_cast_ptr_to_int = genop_same_as
+ genop_cast_ptr_to_int = genop_same_as
+ genop_cast_int_to_ptr = genop_same_as
def genop_int_mod(self, op, arglocs, resloc):
if IS_X86_32:
@@ -1596,12 +1596,43 @@
genop_getarrayitem_gc_pure = genop_getarrayitem_gc
genop_getarrayitem_raw = genop_getarrayitem_gc
+ def _get_interiorfield_addr(self, temp_loc, index_loc, itemsize_loc,
+ base_loc, ofs_loc):
+ assert isinstance(itemsize_loc, ImmedLoc)
+ if isinstance(index_loc, ImmedLoc):
+ temp_loc = imm(index_loc.value * itemsize_loc.value)
+ else:
+ # XXX should not use IMUL in most cases
+ assert isinstance(temp_loc, RegLoc)
+ assert isinstance(index_loc, RegLoc)
+ self.mc.IMUL_rri(temp_loc.value, index_loc.value,
+ itemsize_loc.value)
+ assert isinstance(ofs_loc, ImmedLoc)
+ return AddressLoc(base_loc, temp_loc, 0, ofs_loc.value)
+
+ def genop_getinteriorfield_gc(self, op, arglocs, resloc):
+ (base_loc, ofs_loc, itemsize_loc, fieldsize_loc,
+ index_loc, sign_loc) = arglocs
+ src_addr = self._get_interiorfield_addr(resloc, index_loc,
+ itemsize_loc, base_loc,
+ ofs_loc)
+ self.load_from_mem(resloc, src_addr, fieldsize_loc, sign_loc)
+
+
def genop_discard_setfield_gc(self, op, arglocs):
base_loc, ofs_loc, size_loc, value_loc = arglocs
assert isinstance(size_loc, ImmedLoc)
dest_addr = AddressLoc(base_loc, ofs_loc)
self.save_into_mem(dest_addr, value_loc, size_loc)
+ def genop_discard_setinteriorfield_gc(self, op, arglocs):
+ (base_loc, ofs_loc, itemsize_loc, fieldsize_loc,
+ index_loc, temp_loc, value_loc) = arglocs
+ dest_addr = self._get_interiorfield_addr(temp_loc, index_loc,
+ itemsize_loc, base_loc,
+ ofs_loc)
+ self.save_into_mem(dest_addr, value_loc, fieldsize_loc)
+
def genop_discard_setarrayitem_gc(self, op, arglocs):
base_loc, ofs_loc, value_loc, size_loc, baseofs = arglocs
assert isinstance(baseofs, ImmedLoc)
diff --git a/pypy/jit/backend/x86/regalloc.py b/pypy/jit/backend/x86/regalloc.py
--- a/pypy/jit/backend/x86/regalloc.py
+++ b/pypy/jit/backend/x86/regalloc.py
@@ -7,7 +7,7 @@
ResOperation, BoxPtr, ConstFloat,
BoxFloat, LoopToken, INT, REF, FLOAT)
from pypy.jit.backend.x86.regloc import *
-from pypy.rpython.lltypesystem import lltype, ll2ctypes, rffi, rstr
+from pypy.rpython.lltypesystem import lltype, rffi, rstr
from pypy.rlib.objectmodel import we_are_translated
from pypy.rlib import rgc
from pypy.jit.backend.llsupport import symbolic
@@ -17,11 +17,12 @@
from pypy.jit.metainterp.resoperation import rop
from pypy.jit.backend.llsupport.descr import BaseFieldDescr, BaseArrayDescr
from pypy.jit.backend.llsupport.descr import BaseCallDescr, BaseSizeDescr
+from pypy.jit.backend.llsupport.descr import InteriorFieldDescr
from pypy.jit.backend.llsupport.regalloc import FrameManager, RegisterManager,\
TempBox
from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE
from pypy.jit.backend.x86.arch import IS_X86_32, IS_X86_64, MY_COPY_OF_REGS
-from pypy.rlib.rarithmetic import r_longlong, r_uint
+from pypy.rlib.rarithmetic import r_longlong
class X86RegisterManager(RegisterManager):
@@ -433,7 +434,7 @@
if self.can_merge_with_next_guard(op, i, operations):
oplist_with_guard[op.getopnum()](self, op, operations[i + 1])
i += 1
- elif not we_are_translated() and op.getopnum() == -124:
+ elif not we_are_translated() and op.getopnum() == -124:
self._consider_force_spill(op)
else:
oplist[op.getopnum()](self, op)
@@ -650,8 +651,8 @@
consider_uint_lt = _consider_compop
consider_uint_le = _consider_compop
consider_uint_ge = _consider_compop
- consider_ptr_eq = _consider_compop
- consider_ptr_ne = _consider_compop
+ consider_ptr_eq = consider_instance_ptr_eq = _consider_compop
+ consider_ptr_ne = consider_instance_ptr_ne = _consider_compop
def _consider_float_op(self, op):
loc1 = self.xrm.loc(op.getarg(1))
@@ -815,7 +816,7 @@
save_all_regs = guard_not_forced_op is not None
self.xrm.before_call(force_store, save_all_regs=save_all_regs)
if not save_all_regs:
- gcrootmap = gc_ll_descr = self.assembler.cpu.gc_ll_descr.gcrootmap
+ gcrootmap = self.assembler.cpu.gc_ll_descr.gcrootmap
if gcrootmap and gcrootmap.is_shadow_stack:
save_all_regs = 2
self.rm.before_call(force_store, save_all_regs=save_all_regs)
@@ -972,74 +973,27 @@
return self._call(op, arglocs)
def consider_newstr(self, op):
- gc_ll_descr = self.assembler.cpu.gc_ll_descr
- if gc_ll_descr.get_funcptr_for_newstr is not None:
- # framework GC
- loc = self.loc(op.getarg(0))
- return self._call(op, [loc])
- # boehm GC (XXX kill the following code at some point)
- ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.STR, self.translate_support_code)
- assert itemsize == 1
- return self._malloc_varsize(ofs_items, ofs, 0, op.getarg(0),
- op.result)
+ loc = self.loc(op.getarg(0))
+ return self._call(op, [loc])
def consider_newunicode(self, op):
- gc_ll_descr = self.assembler.cpu.gc_ll_descr
- if gc_ll_descr.get_funcptr_for_newunicode is not None:
- # framework GC
- loc = self.loc(op.getarg(0))
- return self._call(op, [loc])
- # boehm GC (XXX kill the following code at some point)
- ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE,
- self.translate_support_code)
- scale = self._get_unicode_item_scale()
- return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0),
- op.result)
-
- def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v):
- # XXX kill this function at some point
- if isinstance(v, Box):
- loc = self.rm.make_sure_var_in_reg(v, [v])
- tempbox = TempBox()
- other_loc = self.rm.force_allocate_reg(tempbox, [v])
- self.assembler.load_effective_addr(loc, ofs_items,scale, other_loc)
- else:
- tempbox = None
- other_loc = imm(ofs_items + (v.getint() << scale))
- self._call(ResOperation(rop.NEW, [], res_v),
- [other_loc], [v])
- loc = self.rm.make_sure_var_in_reg(v, [res_v])
- assert self.loc(res_v) == eax
- # now we have to reload length to some reasonable place
- self.rm.possibly_free_var(v)
- if tempbox is not None:
- self.rm.possibly_free_var(tempbox)
- self.PerformDiscard(ResOperation(rop.SETFIELD_GC, [None, None], None),
- [eax, imm(ofs_length), imm(WORD), loc])
+ loc = self.loc(op.getarg(0))
+ return self._call(op, [loc])
def consider_new_array(self, op):
gc_ll_descr = self.assembler.cpu.gc_ll_descr
- if gc_ll_descr.get_funcptr_for_newarray is not None:
- # framework GC
- box_num_elem = op.getarg(0)
- if isinstance(box_num_elem, ConstInt):
- num_elem = box_num_elem.value
- if gc_ll_descr.can_inline_malloc_varsize(op.getdescr(),
- num_elem):
- self.fastpath_malloc_varsize(op, op.getdescr(), num_elem)
- return
- args = self.assembler.cpu.gc_ll_descr.args_for_new_array(
- op.getdescr())
- arglocs = [imm(x) for x in args]
- arglocs.append(self.loc(box_num_elem))
- self._call(op, arglocs)
- return
- # boehm GC (XXX kill the following code at some point)
- itemsize, basesize, ofs_length, _, _ = (
- self._unpack_arraydescr(op.getdescr()))
- scale_of_field = _get_scale(itemsize)
- self._malloc_varsize(basesize, ofs_length, scale_of_field,
- op.getarg(0), op.result)
+ box_num_elem = op.getarg(0)
+ if isinstance(box_num_elem, ConstInt):
+ num_elem = box_num_elem.value
+ if gc_ll_descr.can_inline_malloc_varsize(op.getdescr(),
+ num_elem):
+ self.fastpath_malloc_varsize(op, op.getdescr(), num_elem)
+ return
+ args = self.assembler.cpu.gc_ll_descr.args_for_new_array(
+ op.getdescr())
+ arglocs = [imm(x) for x in args]
+ arglocs.append(self.loc(box_num_elem))
+ self._call(op, arglocs)
def _unpack_arraydescr(self, arraydescr):
assert isinstance(arraydescr, BaseArrayDescr)
@@ -1058,6 +1012,16 @@
sign = fielddescr.is_field_signed()
return imm(ofs), imm(size), ptr, sign
+ def _unpack_interiorfielddescr(self, descr):
+ assert isinstance(descr, InteriorFieldDescr)
+ arraydescr = descr.arraydescr
+ ofs = arraydescr.get_base_size(self.translate_support_code)
+ itemsize = arraydescr.get_item_size(self.translate_support_code)
+ fieldsize = descr.fielddescr.get_field_size(self.translate_support_code)
+ sign = descr.fielddescr.is_field_signed()
+ ofs += descr.fielddescr.offset
+ return imm(ofs), imm(itemsize), imm(fieldsize), sign
+
def consider_setfield_gc(self, op):
ofs_loc, size_loc, _, _ = self._unpack_fielddescr(op.getdescr())
assert isinstance(size_loc, ImmedLoc)
@@ -1074,6 +1038,35 @@
consider_setfield_raw = consider_setfield_gc
+ def consider_setinteriorfield_gc(self, op):
+ t = self._unpack_interiorfielddescr(op.getdescr())
+ ofs, itemsize, fieldsize, _ = t
+ args = op.getarglist()
+ if fieldsize.value == 1:
+ need_lower_byte = True
+ else:
+ need_lower_byte = False
+ box_base, box_index, box_value = args
+ base_loc = self.rm.make_sure_var_in_reg(box_base, args)
+ index_loc = self.rm.make_sure_var_in_reg(box_index, args)
+ value_loc = self.make_sure_var_in_reg(box_value, args,
+ need_lower_byte=need_lower_byte)
+ # If 'index_loc' is not an immediate, then we need a 'temp_loc' that
+ # is a register whose value will be destroyed. It's fine to destroy
+ # the same register as 'index_loc', but not the other ones.
+ self.rm.possibly_free_var(box_index)
+ if not isinstance(index_loc, ImmedLoc):
+ tempvar = TempBox()
+ temp_loc = self.rm.force_allocate_reg(tempvar, [box_base,
+ box_value])
+ self.rm.possibly_free_var(tempvar)
+ else:
+ temp_loc = None
+ self.rm.possibly_free_var(box_base)
+ self.possibly_free_var(box_value)
+ self.PerformDiscard(op, [base_loc, ofs, itemsize, fieldsize,
+ index_loc, temp_loc, value_loc])
+
def consider_strsetitem(self, op):
args = op.getarglist()
base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
@@ -1135,6 +1128,25 @@
consider_getarrayitem_raw = consider_getarrayitem_gc
consider_getarrayitem_gc_pure = consider_getarrayitem_gc
+ def consider_getinteriorfield_gc(self, op):
+ t = self._unpack_interiorfielddescr(op.getdescr())
+ ofs, itemsize, fieldsize, sign = t
+ if sign:
+ sign_loc = imm1
+ else:
+ sign_loc = imm0
+ args = op.getarglist()
+ base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
+ index_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
+ # 'base' and 'index' are put in two registers (or one if 'index'
+ # is an immediate). 'result' can be in the same register as
+ # 'index' but must be in a different register than 'base'.
+ self.rm.possibly_free_var(op.getarg(1))
+ result_loc = self.force_allocate_reg(op.result, [op.getarg(0)])
+ self.rm.possibly_free_var(op.getarg(0))
+ self.Perform(op, [base_loc, ofs, itemsize, fieldsize,
+ index_loc, sign_loc], result_loc)
+
def consider_int_is_true(self, op, guard_op):
# doesn't need arg to be in a register
argloc = self.loc(op.getarg(0))
@@ -1152,7 +1164,8 @@
self.possibly_free_var(op.getarg(0))
resloc = self.force_allocate_reg(op.result)
self.Perform(op, [argloc], resloc)
- #consider_cast_ptr_to_int = consider_same_as
+ consider_cast_ptr_to_int = consider_same_as
+ consider_cast_int_to_ptr = consider_same_as
def consider_strlen(self, op):
args = op.getarglist()
@@ -1240,7 +1253,6 @@
self.rm.possibly_free_var(srcaddr_box)
def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode):
- cpu = self.assembler.cpu
if is_unicode:
ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE,
self.translate_support_code)
@@ -1299,7 +1311,7 @@
tmpreg = X86RegisterManager.all_regs[0]
tmploc = self.rm.force_allocate_reg(box, selected_reg=tmpreg)
xmmtmp = X86XMMRegisterManager.all_regs[0]
- xmmtmploc = self.xrm.force_allocate_reg(box1, selected_reg=xmmtmp)
+ self.xrm.force_allocate_reg(box1, selected_reg=xmmtmp)
# Part about non-floats
# XXX we don't need a copy, we only just the original list
src_locations1 = [self.loc(op.getarg(i)) for i in range(op.numargs())
@@ -1379,7 +1391,7 @@
return lambda self, op: fn(self, op, None)
def is_comparison_or_ovf_op(opnum):
- from pypy.jit.metainterp.resoperation import opclasses, AbstractResOp
+ from pypy.jit.metainterp.resoperation import opclasses
cls = opclasses[opnum]
# hack hack: in theory they are instance method, but they don't use
# any instance field, we can use a fake object
diff --git a/pypy/jit/backend/x86/test/test_del.py b/pypy/jit/backend/x86/test/test_del.py
--- a/pypy/jit/backend/x86/test/test_del.py
+++ b/pypy/jit/backend/x86/test/test_del.py
@@ -1,5 +1,4 @@
-import py
from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
from pypy.jit.metainterp.test.test_del import DelTests
diff --git a/pypy/jit/backend/x86/test/test_dict.py b/pypy/jit/backend/x86/test/test_dict.py
new file mode 100644
--- /dev/null
+++ b/pypy/jit/backend/x86/test/test_dict.py
@@ -0,0 +1,9 @@
+
+from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
+from pypy.jit.metainterp.test.test_dict import DictTests
+
+
+class TestDict(Jit386Mixin, DictTests):
+ # for the individual tests see
+ # ====> ../../../metainterp/test/test_dict.py
+ pass
diff --git a/pypy/jit/backend/x86/test/test_runner.py b/pypy/jit/backend/x86/test/test_runner.py
--- a/pypy/jit/backend/x86/test/test_runner.py
+++ b/pypy/jit/backend/x86/test/test_runner.py
@@ -31,7 +31,7 @@
# for the individual tests see
# ====> ../../test/runner_test.py
-
+
def setup_method(self, meth):
self.cpu = CPU(rtyper=None, stats=FakeStats())
self.cpu.setup_once()
@@ -69,22 +69,16 @@
def test_allocations(self):
from pypy.rpython.lltypesystem import rstr
-
+
allocs = [None]
all = []
+ orig_new = self.cpu.gc_ll_descr.funcptr_for_new
def f(size):
allocs.insert(0, size)
- buf = ctypes.create_string_buffer(size)
- all.append(buf)
- return ctypes.cast(buf, ctypes.c_void_p).value
- func = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)(f)
- addr = ctypes.cast(func, ctypes.c_void_p).value
- # ctypes produces an unsigned value. We need it to be signed for, eg,
- # relative addressing to work properly.
- addr = rffi.cast(lltype.Signed, addr)
-
+ return orig_new(size)
+
self.cpu.assembler.setup_once()
- self.cpu.assembler.malloc_func_addr = addr
+ self.cpu.gc_ll_descr.funcptr_for_new = f
ofs = symbolic.get_field_token(rstr.STR, 'chars', False)[0]
res = self.execute_operation(rop.NEWSTR, [ConstInt(7)], 'ref')
@@ -108,7 +102,7 @@
res = self.execute_operation(rop.NEW_ARRAY, [ConstInt(10)],
'ref', descr)
assert allocs[0] == 10*WORD + ofs + WORD
- resbuf = self._resbuf(res)
+ resbuf = self._resbuf(res)
assert resbuf[ofs/WORD] == 10
# ------------------------------------------------------------
@@ -116,7 +110,7 @@
res = self.execute_operation(rop.NEW_ARRAY, [BoxInt(10)],
'ref', descr)
assert allocs[0] == 10*WORD + ofs + WORD
- resbuf = self._resbuf(res)
+ resbuf = self._resbuf(res)
assert resbuf[ofs/WORD] == 10
def test_stringitems(self):
@@ -146,7 +140,7 @@
ConstInt(2), BoxInt(38)],
'void', descr)
assert resbuf[itemsofs/WORD + 2] == 38
-
+
self.execute_operation(rop.SETARRAYITEM_GC, [res,
BoxInt(3), BoxInt(42)],
'void', descr)
@@ -167,7 +161,7 @@
BoxInt(2)],
'int', descr)
assert r.value == 38
-
+
r = self.execute_operation(rop.GETARRAYITEM_GC, [res, BoxInt(3)],
'int', descr)
assert r.value == 42
@@ -226,7 +220,7 @@
self.execute_operation(rop.SETFIELD_GC, [res, BoxInt(1234)], 'void', ofs_i)
i = self.execute_operation(rop.GETFIELD_GC, [res], 'int', ofs_i)
assert i.value == 1234
-
+
#u = self.execute_operation(rop.GETFIELD_GC, [res, ofs_u], 'int')
#assert u.value == 5
self.execute_operation(rop.SETFIELD_GC, [res, ConstInt(1)], 'void',
@@ -299,7 +293,7 @@
else:
assert result != execute(self.cpu, None,
op, None, b).value
-
+
def test_stuff_followed_by_guard(self):
boxes = [(BoxInt(1), BoxInt(0)),
@@ -523,7 +517,7 @@
def test_debugger_on(self):
from pypy.tool.logparser import parse_log_file, extract_category
from pypy.rlib import debug
-
+
loop = """
[i0]
debug_merge_point('xyz', 0)
diff --git a/pypy/jit/backend/x86/tool/viewcode.py b/pypy/jit/backend/x86/tool/viewcode.py
--- a/pypy/jit/backend/x86/tool/viewcode.py
+++ b/pypy/jit/backend/x86/tool/viewcode.py
@@ -9,7 +9,12 @@
"""
import autopath
-import operator, sys, os, re, py, new
+import new
+import operator
+import py
+import re
+import sys
+import subprocess
from bisect import bisect_left
# don't use pypy.tool.udir here to avoid removing old usessions which
@@ -44,14 +49,16 @@
f = open(tmpfile, 'wb')
f.write(data)
f.close()
- g = os.popen(objdump % {
+ p = subprocess.Popen(objdump % {
'file': tmpfile,
'origin': originaddr,
'backend': objdump_backend_option[backend_name],
- }, 'r')
- result = g.readlines()
- g.close()
- lines = result[6:] # drop some objdump cruft
+ }, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = p.communicate()
+ assert not p.returncode, ('Encountered an error running objdump: %s' %
+ stderr)
+ # drop some objdump cruft
+ lines = stdout.splitlines()[6:]
return format_code_dump_with_labels(originaddr, lines, label_list)
def format_code_dump_with_labels(originaddr, lines, label_list):
@@ -85,8 +92,12 @@
#
print 'loading symbols from %s...' % (filename,)
symbols = {}
- g = os.popen(symbollister % filename, "r")
- for line in g:
+ p = subprocess.Popen(symbollister % filename, shell=True,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = p.communicate()
+ assert not p.returncode, ('Encountered an error running nm: %s' %
+ stderr)
+ for line in stdout.splitlines():
match = re_symbolentry.match(line)
if match:
addr = long(match.group(1), 16)
@@ -94,7 +105,6 @@
if name.startswith('pypy_g_'):
name = '\xb7' + name[7:]
symbols[addr] = name
- g.close()
print '%d symbols found' % (len(symbols),)
return symbols
diff --git a/pypy/jit/codewriter/jtransform.py b/pypy/jit/codewriter/jtransform.py
--- a/pypy/jit/codewriter/jtransform.py
+++ b/pypy/jit/codewriter/jtransform.py
@@ -52,9 +52,11 @@
newoperations = []
#
def do_rename(var, var_or_const):
+ if var.concretetype is lltype.Void:
+ renamings[var] = Constant(None, lltype.Void)
+ return
renamings[var] = var_or_const
- if (isinstance(var_or_const, Constant)
- and var.concretetype != lltype.Void):
+ if isinstance(var_or_const, Constant):
value = var_or_const.value
value = lltype._cast_whatever(var.concretetype, value)
renamings_constants[var] = Constant(value, var.concretetype)
@@ -441,6 +443,8 @@
rewrite_op_gc_identityhash = _do_builtin_call
rewrite_op_gc_id = _do_builtin_call
rewrite_op_uint_mod = _do_builtin_call
+ rewrite_op_cast_float_to_uint = _do_builtin_call
+ rewrite_op_cast_uint_to_float = _do_builtin_call
# ----------
# getfield/setfield/mallocs etc.
@@ -455,6 +459,23 @@
# the special return value None forces op.result to be considered
# equal to op.args[0]
return [op0, op1, None]
+ if (hints.get('promote_string') and
+ op.args[0].concretetype is not lltype.Void):
+ S = lltype.Ptr(rstr.STR)
+ assert op.args[0].concretetype == S
+ self._register_extra_helper(EffectInfo.OS_STREQ_NONNULL,
+ "str.eq_nonnull",
+ [S, S],
+ lltype.Signed,
+ EffectInfo.EF_ELIDABLE_CANNOT_RAISE)
+ descr, p = self.callcontrol.callinfocollection.callinfo_for_oopspec(
+ EffectInfo.OS_STREQ_NONNULL)
+ # XXX this is fairly ugly way of creating a constant,
+ # however, callinfocollection has no better interface
+ c = Constant(p.adr.ptr, lltype.typeOf(p.adr.ptr))
+ op1 = SpaceOperation('str_guard_value', [op.args[0], c, descr],
+ op.result)
+ return [SpaceOperation('-live-', [], None), op1, None]
else:
log.WARNING('ignoring hint %r at %r' % (hints, self.graph))
@@ -718,29 +739,54 @@
return SpaceOperation(opname, [op.args[0]], op.result)
def rewrite_op_getinteriorfield(self, op):
- # only supports strings and unicodes
assert len(op.args) == 3
- assert op.args[1].value == 'chars'
optype = op.args[0].concretetype
if optype == lltype.Ptr(rstr.STR):
opname = "strgetitem"
+ return SpaceOperation(opname, [op.args[0], op.args[2]], op.result)
+ elif optype == lltype.Ptr(rstr.UNICODE):
+ opname = "unicodegetitem"
+ return SpaceOperation(opname, [op.args[0], op.args[2]], op.result)
else:
- assert optype == lltype.Ptr(rstr.UNICODE)
- opname = "unicodegetitem"
- return SpaceOperation(opname, [op.args[0], op.args[2]], op.result)
+ v_inst, v_index, c_field = op.args
+ if op.result.concretetype is lltype.Void:
+ return
+ # only GcArray of Struct supported
+ assert isinstance(v_inst.concretetype.TO, lltype.GcArray)
+ STRUCT = v_inst.concretetype.TO.OF
+ assert isinstance(STRUCT, lltype.Struct)
+ descr = self.cpu.interiorfielddescrof(v_inst.concretetype.TO,
+ c_field.value)
+ args = [v_inst, v_index, descr]
+ kind = getkind(op.result.concretetype)[0]
+ return SpaceOperation('getinteriorfield_gc_%s' % kind, args,
+ op.result)
def rewrite_op_setinteriorfield(self, op):
- # only supports strings and unicodes
assert len(op.args) == 4
- assert op.args[1].value == 'chars'
optype = op.args[0].concretetype
if optype == lltype.Ptr(rstr.STR):
opname = "strsetitem"
+ return SpaceOperation(opname, [op.args[0], op.args[2], op.args[3]],
+ op.result)
+ elif optype == lltype.Ptr(rstr.UNICODE):
+ opname = "unicodesetitem"
+ return SpaceOperation(opname, [op.args[0], op.args[2], op.args[3]],
+ op.result)
else:
- assert optype == lltype.Ptr(rstr.UNICODE)
- opname = "unicodesetitem"
- return SpaceOperation(opname, [op.args[0], op.args[2], op.args[3]],
- op.result)
+ v_inst, v_index, c_field, v_value = op.args
+ if v_value.concretetype is lltype.Void:
+ return
+ # only GcArray of Struct supported
+ assert isinstance(v_inst.concretetype.TO, lltype.GcArray)
+ STRUCT = v_inst.concretetype.TO.OF
+ assert isinstance(STRUCT, lltype.Struct)
+ descr = self.cpu.interiorfielddescrof(v_inst.concretetype.TO,
+ c_field.value)
+ kind = getkind(v_value.concretetype)[0]
+ args = [v_inst, v_index, v_value, descr]
+ return SpaceOperation('setinteriorfield_gc_%s' % kind, args,
+ op.result)
def _rewrite_equality(self, op, opname):
arg0, arg1 = op.args
@@ -754,6 +800,9 @@
def _is_gc(self, v):
return getattr(getattr(v.concretetype, "TO", None), "_gckind", "?") == 'gc'
+ def _is_rclass_instance(self, v):
+ return lltype._castdepth(v.concretetype.TO, rclass.OBJECT) >= 0
+
def _rewrite_cmp_ptrs(self, op):
if self._is_gc(op.args[0]):
return op
@@ -771,11 +820,21 @@
return self._rewrite_equality(op, 'int_is_true')
def rewrite_op_ptr_eq(self, op):
- op1 = self._rewrite_equality(op, 'ptr_iszero')
+ prefix = ''
+ if self._is_rclass_instance(op.args[0]):
+ assert self._is_rclass_instance(op.args[1])
+ op = SpaceOperation('instance_ptr_eq', op.args, op.result)
+ prefix = 'instance_'
+ op1 = self._rewrite_equality(op, prefix + 'ptr_iszero')
return self._rewrite_cmp_ptrs(op1)
def rewrite_op_ptr_ne(self, op):
- op1 = self._rewrite_equality(op, 'ptr_nonzero')
+ prefix = ''
+ if self._is_rclass_instance(op.args[0]):
+ assert self._is_rclass_instance(op.args[1])
+ op = SpaceOperation('instance_ptr_ne', op.args, op.result)
+ prefix = 'instance_'
+ op1 = self._rewrite_equality(op, prefix + 'ptr_nonzero')
return self._rewrite_cmp_ptrs(op1)
rewrite_op_ptr_iszero = _rewrite_cmp_ptrs
@@ -783,8 +842,11 @@
def rewrite_op_cast_ptr_to_int(self, op):
if self._is_gc(op.args[0]):
- #return op
- raise NotImplementedError("cast_ptr_to_int")
+ return op
+
+ def rewrite_op_cast_opaque_ptr(self, op):
+ # None causes the result of this op to get aliased to op.args[0]
+ return [SpaceOperation('mark_opaque_ptr', op.args, None), None]
def rewrite_op_force_cast(self, op):
v_arg = op.args[0]
@@ -805,26 +867,44 @@
elif not float_arg and float_res:
# some int -> some float
ops = []
- v1 = varoftype(lltype.Signed)
- oplist = self.rewrite_operation(
- SpaceOperation('force_cast', [v_arg], v1)
- )
- if oplist:
- ops.extend(oplist)
+ v2 = varoftype(lltype.Float)
+ sizesign = rffi.size_and_sign(v_arg.concretetype)
+ if sizesign <= rffi.size_and_sign(lltype.Signed):
+ # cast from a type that fits in an int: either the size is
+ # smaller, or it is equal and it is not unsigned
+ v1 = varoftype(lltype.Signed)
+ oplist = self.rewrite_operation(
+ SpaceOperation('force_cast', [v_arg], v1)
+ )
+ if oplist:
+ ops.extend(oplist)
+ else:
+ v1 = v_arg
+ op = self.rewrite_operation(
+ SpaceOperation('cast_int_to_float', [v1], v2)
+ )
+ ops.append(op)
else:
- v1 = v_arg
- v2 = varoftype(lltype.Float)
- op = self.rewrite_operation(
- SpaceOperation('cast_int_to_float', [v1], v2)
- )
- ops.append(op)
+ if sizesign == rffi.size_and_sign(lltype.Unsigned):
+ opname = 'cast_uint_to_float'
+ elif sizesign == rffi.size_and_sign(lltype.SignedLongLong):
+ opname = 'cast_longlong_to_float'
+ elif sizesign == rffi.size_and_sign(lltype.UnsignedLongLong):
+ opname = 'cast_ulonglong_to_float'
+ else:
+ raise AssertionError('cast_x_to_float: %r' % (sizesign,))
+ ops1 = self.rewrite_operation(
+ SpaceOperation(opname, [v_arg], v2)
+ )
+ if not isinstance(ops1, list): ops1 = [ops1]
+ ops.extend(ops1)
op2 = self.rewrite_operation(
SpaceOperation('force_cast', [v2], v_result)
)
if op2:
ops.append(op2)
else:
- op.result = v_result
+ ops[-1].result = v_result
return ops
elif float_arg and not float_res:
# some float -> some int
@@ -837,18 +917,36 @@
ops.append(op1)
else:
v1 = v_arg
- v2 = varoftype(lltype.Signed)
- op = self.rewrite_operation(
- SpaceOperation('cast_float_to_int', [v1], v2)
- )
- ops.append(op)
- oplist = self.rewrite_operation(
- SpaceOperation('force_cast', [v2], v_result)
- )
- if oplist:
- ops.extend(oplist)
+ sizesign = rffi.size_and_sign(v_result.concretetype)
+ if sizesign <= rffi.size_and_sign(lltype.Signed):
+ # cast to a type that fits in an int: either the size is
+ # smaller, or it is equal and it is not unsigned
+ v2 = varoftype(lltype.Signed)
+ op = self.rewrite_operation(
+ SpaceOperation('cast_float_to_int', [v1], v2)
+ )
+ ops.append(op)
+ oplist = self.rewrite_operation(
+ SpaceOperation('force_cast', [v2], v_result)
+ )
+ if oplist:
+ ops.extend(oplist)
+ else:
+ op.result = v_result
else:
- op.result = v_result
+ if sizesign == rffi.size_and_sign(lltype.Unsigned):
+ opname = 'cast_float_to_uint'
+ elif sizesign == rffi.size_and_sign(lltype.SignedLongLong):
+ opname = 'cast_float_to_longlong'
+ elif sizesign == rffi.size_and_sign(lltype.UnsignedLongLong):
+ opname = 'cast_float_to_ulonglong'
+ else:
+ raise AssertionError('cast_float_to_x: %r' % (sizesign,))
+ ops1 = self.rewrite_operation(
+ SpaceOperation(opname, [v1], v_result)
+ )
+ if not isinstance(ops1, list): ops1 = [ops1]
+ ops.extend(ops1)
return ops
else:
assert False
@@ -1054,8 +1152,6 @@
# The new operation is optionally further processed by rewrite_operation().
for _old, _new in [('bool_not', 'int_is_zero'),
('cast_bool_to_float', 'cast_int_to_float'),
- ('cast_uint_to_float', 'cast_int_to_float'),
- ('cast_float_to_uint', 'cast_float_to_int'),
('int_add_nonneg_ovf', 'int_add_ovf'),
('keepalive', '-live-'),
@@ -1526,6 +1622,10 @@
def rewrite_op_jit_force_virtual(self, op):
return self._do_builtin_call(op)
+ def rewrite_op_jit_is_virtual(self, op):
+ raise Exception, (
+ "'vref.virtual' should not be used from jit-visible code")
+
def rewrite_op_jit_force_virtualizable(self, op):
# this one is for virtualizables
vinfo = self.get_vinfo(op.args[0])
diff --git a/pypy/jit/codewriter/support.py b/pypy/jit/codewriter/support.py
--- a/pypy/jit/codewriter/support.py
+++ b/pypy/jit/codewriter/support.py
@@ -13,7 +13,6 @@
from pypy.translator.simplify import get_funcobj
from pypy.translator.unsimplify import split_block
from pypy.objspace.flow.model import Constant
-from pypy import conftest
from pypy.translator.translator import TranslationContext
from pypy.annotation.policy import AnnotatorPolicy
from pypy.annotation import model as annmodel
@@ -48,15 +47,13 @@
a.build_types(func, argtypes, main_entry_point=True)
rtyper = t.buildrtyper(type_system = type_system)
rtyper.specialize()
- if inline:
- auto_inlining(t, threshold=inline)
+ #if inline:
+ # auto_inlining(t, threshold=inline)
if backendoptimize:
from pypy.translator.backendopt.all import backend_optimizations
backend_optimizations(t, inline_threshold=inline or 0,
remove_asserts=True, really_remove_asserts=True)
- #if conftest.option.view:
- # t.view()
return rtyper
def getgraph(func, values):
@@ -232,6 +229,17 @@
else:
return x
+def _ll_1_cast_uint_to_float(x):
+ # XXX on 32-bit platforms, this should be done using cast_longlong_to_float
+ # (which is a residual call right now in the x86 backend)
+ return llop.cast_uint_to_float(lltype.Float, x)
+
+def _ll_1_cast_float_to_uint(x):
+ # XXX on 32-bit platforms, this should be done using cast_float_to_longlong
+ # (which is a residual call right now in the x86 backend)
+ return llop.cast_float_to_uint(lltype.Unsigned, x)
+
+
# math support
# ------------
@@ -456,6 +464,8 @@
return LLtypeHelpers._dictnext_items(lltype.Ptr(RES), iter)
_ll_1_dictiter_nextitems.need_result_type = True
+ _ll_1_dict_resize = ll_rdict.ll_dict_resize
+
# ---------- strings and unicode ----------
_ll_1_str_str2unicode = ll_rstr.LLHelpers.ll_str2unicode
diff --git a/pypy/jit/codewriter/test/test_flatten.py b/pypy/jit/codewriter/test/test_flatten.py
--- a/pypy/jit/codewriter/test/test_flatten.py
+++ b/pypy/jit/codewriter/test/test_flatten.py
@@ -8,7 +8,7 @@
from pypy.rpython.lltypesystem import lltype, rclass, rstr
from pypy.objspace.flow.model import SpaceOperation, Variable, Constant
from pypy.translator.unsimplify import varoftype
-from pypy.rlib.rarithmetic import ovfcheck, r_uint
+from pypy.rlib.rarithmetic import ovfcheck, r_uint, r_longlong, r_ulonglong
from pypy.rlib.jit import dont_look_inside, _we_are_jitted, JitDriver
from pypy.rlib.objectmodel import keepalive_until_here
from pypy.rlib import jit
@@ -70,7 +70,8 @@
return 'residual'
def getcalldescr(self, op, oopspecindex=None, extraeffect=None):
try:
- if 'cannot_raise' in op.args[0].value._obj.graph.name:
+ name = op.args[0].value._obj._name
+ if 'cannot_raise' in name or name.startswith('cast_'):
return self._descr_cannot_raise
except AttributeError:
pass
@@ -900,6 +901,67 @@
int_return %i4
""", transform=True)
+ def f(dbl):
+ return rffi.cast(rffi.UCHAR, dbl)
+ self.encoding_test(f, [12.456], """
+ cast_float_to_int %f0 -> %i0
+ int_and %i0, $255 -> %i1
+ int_return %i1
+ """, transform=True)
+
+ def f(dbl):
+ return rffi.cast(lltype.Unsigned, dbl)
+ self.encoding_test(f, [12.456], """
+ residual_call_irf_i $<* fn cast_float_to_uint>, <Descr>, I[], R[], F[%f0] -> %i0
+ int_return %i0
+ """, transform=True)
+
+ def f(i):
+ return rffi.cast(lltype.Float, chr(i)) # "char -> float"
+ self.encoding_test(f, [12], """
+ cast_int_to_float %i0 -> %f0
+ float_return %f0
+ """, transform=True)
+
+ def f(i):
+ return rffi.cast(lltype.Float, r_uint(i)) # "uint -> float"
+ self.encoding_test(f, [12], """
+ residual_call_irf_f $<* fn cast_uint_to_float>, <Descr>, I[%i0], R[], F[] -> %f0
+ float_return %f0
+ """, transform=True)
+
+ if not longlong.is_64_bit:
+ def f(dbl):
+ return rffi.cast(lltype.SignedLongLong, dbl)
+ self.encoding_test(f, [12.3], """
+ residual_call_irf_f $<* fn llong_from_float>, <Descr>, I[], R[], F[%f0] -> %f1
+ float_return %f1
+ """, transform=True)
+
+ def f(dbl):
+ return rffi.cast(lltype.UnsignedLongLong, dbl)
+ self.encoding_test(f, [12.3], """
+ residual_call_irf_f $<* fn ullong_from_float>, <Descr>, I[], R[], F[%f0] -> %f1
+ float_return %f1
+ """, transform=True)
+
+ def f(x):
+ ll = r_longlong(x)
+ return rffi.cast(lltype.Float, ll)
+ self.encoding_test(f, [12], """
+ residual_call_irf_f $<* fn llong_from_int>, <Descr>, I[%i0], R[], F[] -> %f0
+ residual_call_irf_f $<* fn llong_to_float>, <Descr>, I[], R[], F[%f0] -> %f1
+ float_return %f1
+ """, transform=True)
+
+ def f(x):
+ ll = r_ulonglong(x)
+ return rffi.cast(lltype.Float, ll)
+ self.encoding_test(f, [12], """
+ residual_call_irf_f $<* fn ullong_from_int>, <Descr>, I[%i0], R[], F[] -> %f0
+ residual_call_irf_f $<* fn ullong_u_to_float>, <Descr>, I[], R[], F[%f0] -> %f1
+ float_return %f1
+ """, transform=True)
def test_direct_ptradd(self):
from pypy.rpython.lltypesystem import rffi
diff --git a/pypy/jit/codewriter/test/test_jtransform.py b/pypy/jit/codewriter/test/test_jtransform.py
--- a/pypy/jit/codewriter/test/test_jtransform.py
+++ b/pypy/jit/codewriter/test/test_jtransform.py
@@ -1,14 +1,27 @@
-import py
import random
+try:
+ from itertools import product
+except ImportError:
+ # Python 2.5, this is taken from the CPython docs, but simplified.
+ def product(*args):
+ # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
+ # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
+ pools = map(tuple, args)
+ result = [[]]
+ for pool in pools:
+ result = [x+[y] for x in result for y in pool]
+ for prod in result:
+ yield tuple(prod)
+
from pypy.objspace.flow.model import FunctionGraph, Block, Link
from pypy.objspace.flow.model import SpaceOperation, Variable, Constant
-from pypy.jit.codewriter.jtransform import Transformer
-from pypy.jit.metainterp.history import getkind
-from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rlist
+from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr
from pypy.rpython.lltypesystem.module import ll_math
from pypy.translator.unsimplify import varoftype
from pypy.jit.codewriter import heaptracker, effectinfo
from pypy.jit.codewriter.flatten import ListOfKind
+from pypy.jit.codewriter.jtransform import Transformer
+from pypy.jit.metainterp.history import getkind
def const(x):
return Constant(x, lltype.typeOf(x))
@@ -23,6 +36,8 @@
return ('calldescr', FUNC, ARGS, RESULT)
def fielddescrof(self, STRUCT, name):
return ('fielddescr', STRUCT, name)
+ def interiorfielddescrof(self, ARRAY, name):
+ return ('interiorfielddescr', ARRAY, name)
def arraydescrof(self, ARRAY):
return FakeDescr(('arraydescr', ARRAY))
def sizeof(self, STRUCT):
@@ -85,6 +100,12 @@
if i == oopspecindex:
return True
return False
+ def callinfo_for_oopspec(self, oopspecindex):
+ assert oopspecindex == effectinfo.EffectInfo.OS_STREQ_NONNULL
+ class c:
+ class adr:
+ ptr = 1
+ return ('calldescr', c)
class FakeBuiltinCallControl:
def __init__(self):
@@ -105,6 +126,7 @@
EI.OS_STR2UNICODE:([PSTR], PUNICODE),
EI.OS_STR_CONCAT: ([PSTR, PSTR], PSTR),
EI.OS_STR_SLICE: ([PSTR, INT, INT], PSTR),
+ EI.OS_STREQ_NONNULL: ([PSTR, PSTR], INT),
EI.OS_UNI_CONCAT: ([PUNICODE, PUNICODE], PUNICODE),
EI.OS_UNI_SLICE: ([PUNICODE, INT, INT], PUNICODE),
EI.OS_UNI_EQUAL: ([PUNICODE, PUNICODE], lltype.Bool),
@@ -254,26 +276,35 @@
assert op1.result is None
def test_calls():
- for RESTYPE in [lltype.Signed, rclass.OBJECTPTR,
- lltype.Float, lltype.Void]:
- for with_void in [False, True]:
- for with_i in [False, True]:
- for with_r in [False, True]:
- for with_f in [False, True]:
- ARGS = []
- if with_void: ARGS += [lltype.Void, lltype.Void]
- if with_i: ARGS += [lltype.Signed, lltype.Char]
- if with_r: ARGS += [rclass.OBJECTPTR, lltype.Ptr(rstr.STR)]
- if with_f: ARGS += [lltype.Float, lltype.Float]
- random.shuffle(ARGS)
- if RESTYPE == lltype.Float: with_f = True
- if with_f: expectedkind = 'irf' # all kinds
- elif with_i: expectedkind = 'ir' # integers and references
- else: expectedkind = 'r' # only references
- yield residual_call_test, ARGS, RESTYPE, expectedkind
- yield direct_call_test, ARGS, RESTYPE, expectedkind
- yield indirect_residual_call_test, ARGS, RESTYPE, expectedkind
- yield indirect_regular_call_test, ARGS, RESTYPE, expectedkind
+ for RESTYPE, with_void, with_i, with_r, with_f in product(
+ [lltype.Signed, rclass.OBJECTPTR, lltype.Float, lltype.Void],
+ [False, True],
+ [False, True],
+ [False, True],
+ [False, True],
+ ):
+ ARGS = []
+ if with_void:
+ ARGS += [lltype.Void, lltype.Void]
+ if with_i:
+ ARGS += [lltype.Signed, lltype.Char]
+ if with_r:
+ ARGS += [rclass.OBJECTPTR, lltype.Ptr(rstr.STR)]
+ if with_f:
+ ARGS += [lltype.Float, lltype.Float]
+ random.shuffle(ARGS)
+ if RESTYPE == lltype.Float:
+ with_f = True
+ if with_f:
+ expectedkind = 'irf' # all kinds
+ elif with_i:
+ expectedkind = 'ir' # integers and references
+ else:
+ expectedkind = 'r' # only references
+ yield residual_call_test, ARGS, RESTYPE, expectedkind
+ yield direct_call_test, ARGS, RESTYPE, expectedkind
+ yield indirect_residual_call_test, ARGS, RESTYPE, expectedkind
+ yield indirect_regular_call_test, ARGS, RESTYPE, expectedkind
def get_direct_call_op(argtypes, restype):
FUNC = lltype.FuncType(argtypes, restype)
@@ -509,7 +540,7 @@
def test_rename_on_links():
v1 = Variable()
- v2 = Variable()
+ v2 = Variable(); v2.concretetype = llmemory.Address
v3 = Variable()
block = Block([v1])
block.operations = [SpaceOperation('cast_pointer', [v1], v2)]
@@ -545,10 +576,10 @@
assert op1.args == [v2]
def test_ptr_eq():
- v1 = varoftype(rclass.OBJECTPTR)
- v2 = varoftype(rclass.OBJECTPTR)
+ v1 = varoftype(lltype.Ptr(rstr.STR))
+ v2 = varoftype(lltype.Ptr(rstr.STR))
v3 = varoftype(lltype.Bool)
- c0 = const(lltype.nullptr(rclass.OBJECT))
+ c0 = const(lltype.nullptr(rstr.STR))
#
for opname, reducedname in [('ptr_eq', 'ptr_iszero'),
('ptr_ne', 'ptr_nonzero')]:
@@ -567,6 +598,31 @@
assert op1.opname == reducedname
assert op1.args == [v2]
+def test_instance_ptr_eq():
+ v1 = varoftype(rclass.OBJECTPTR)
+ v2 = varoftype(rclass.OBJECTPTR)
+ v3 = varoftype(lltype.Bool)
+ c0 = const(lltype.nullptr(rclass.OBJECT))
+
+ for opname, newopname, reducedname in [
+ ('ptr_eq', 'instance_ptr_eq', 'instance_ptr_iszero'),
+ ('ptr_ne', 'instance_ptr_ne', 'instance_ptr_nonzero')
+ ]:
+ op = SpaceOperation(opname, [v1, v2], v3)
+ op1 = Transformer().rewrite_operation(op)
+ assert op1.opname == newopname
+ assert op1.args == [v1, v2]
+
+ op = SpaceOperation(opname, [v1, c0], v3)
+ op1 = Transformer().rewrite_operation(op)
+ assert op1.opname == reducedname
+ assert op1.args == [v1]
+
+ op = SpaceOperation(opname, [c0, v1], v3)
+ op1 = Transformer().rewrite_operation(op)
+ assert op1.opname == reducedname
+ assert op1.args == [v1]
+
def test_nongc_ptr_eq():
v1 = varoftype(rclass.NONGCOBJECTPTR)
v2 = varoftype(rclass.NONGCOBJECTPTR)
@@ -646,6 +702,22 @@
assert op1.args == [v, v_index]
assert op1.result == v_result
+def test_dict_getinteriorfield():
+ DICT = lltype.GcArray(lltype.Struct('ENTRY', ('v', lltype.Signed),
+ ('k', lltype.Signed)))
+ v = varoftype(lltype.Ptr(DICT))
+ i = varoftype(lltype.Signed)
+ v_result = varoftype(lltype.Signed)
+ op = SpaceOperation('getinteriorfield', [v, i, Constant('v', lltype.Void)],
+ v_result)
+ op1 = Transformer(FakeCPU()).rewrite_operation(op)
+ assert op1.opname == 'getinteriorfield_gc_i'
+ assert op1.args == [v, i, ('interiorfielddescr', DICT, 'v')]
+ op = SpaceOperation('getinteriorfield', [v, i, Constant('v', lltype.Void)],
+ Constant(None, lltype.Void))
+ op1 = Transformer(FakeCPU()).rewrite_operation(op)
+ assert op1 is None
+
def test_str_setinteriorfield():
v = varoftype(lltype.Ptr(rstr.STR))
v_index = varoftype(lltype.Signed)
@@ -672,6 +744,23 @@
assert op1.args == [v, v_index, v_newchr]
assert op1.result == v_void
+def test_dict_setinteriorfield():
+ DICT = lltype.GcArray(lltype.Struct('ENTRY', ('v', lltype.Signed),
+ ('k', lltype.Signed)))
+ v = varoftype(lltype.Ptr(DICT))
+ i = varoftype(lltype.Signed)
+ v_void = varoftype(lltype.Void)
+ op = SpaceOperation('setinteriorfield', [v, i, Constant('v', lltype.Void),
+ i],
+ v_void)
+ op1 = Transformer(FakeCPU()).rewrite_operation(op)
+ assert op1.opname == 'setinteriorfield_gc_i'
+ assert op1.args == [v, i, i, ('interiorfielddescr', DICT, 'v')]
+ op = SpaceOperation('setinteriorfield', [v, i, Constant('v', lltype.Void),
+ v_void], v_void)
+ op1 = Transformer(FakeCPU()).rewrite_operation(op)
+ assert not op1
+
def test_promote_1():
v1 = varoftype(lltype.Signed)
v2 = varoftype(lltype.Signed)
@@ -821,6 +910,21 @@
assert op1.args[2] == ListOfKind('ref', [v1, v2])
assert op1.result == v3
+def test_str_promote():
+ PSTR = lltype.Ptr(rstr.STR)
+ v1 = varoftype(PSTR)
+ v2 = varoftype(PSTR)
+ op = SpaceOperation('hint',
+ [v1, Constant({'promote_string': True}, lltype.Void)],
+ v2)
+ tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+ op0, op1, _ = tr.rewrite_operation(op)
+ assert op1.opname == 'str_guard_value'
+ assert op1.args[0] == v1
+ assert op1.args[2] == 'calldescr'
+ assert op1.result == v2
+ assert op0.opname == '-live-'
+
def test_unicode_concat():
# test that the oopspec is present and correctly transformed
PSTR = lltype.Ptr(rstr.UNICODE)
@@ -1024,3 +1128,16 @@
varoftype(lltype.Signed))
tr = Transformer(None, None)
raises(NotImplementedError, tr.rewrite_operation, op)
+
+def test_cast_opaque_ptr():
+ S = lltype.GcStruct("S", ("x", lltype.Signed))
+ v1 = varoftype(lltype.Ptr(S))
+ v2 = varoftype(lltype.Ptr(rclass.OBJECT))
+
+ op = SpaceOperation('cast_opaque_ptr', [v1], v2)
+ tr = Transformer()
+ [op1, op2] = tr.rewrite_operation(op)
+ assert op1.opname == 'mark_opaque_ptr'
+ assert op1.args == [v1]
+ assert op1.result is None
+ assert op2 is None
\ No newline at end of file
diff --git a/pypy/jit/metainterp/blackhole.py b/pypy/jit/metainterp/blackhole.py
--- a/pypy/jit/metainterp/blackhole.py
+++ b/pypy/jit/metainterp/blackhole.py
@@ -2,11 +2,10 @@
from pypy.rlib.rtimer import read_timestamp
from pypy.rlib.rarithmetic import intmask, LONG_BIT, r_uint, ovfcheck
from pypy.rlib.objectmodel import we_are_translated
-from pypy.rlib.debug import debug_start, debug_stop
+from pypy.rlib.debug import debug_start, debug_stop, ll_assert
from pypy.rlib.debug import make_sure_not_resized
from pypy.rpython.lltypesystem import lltype, llmemory, rclass
from pypy.rpython.lltypesystem.lloperation import llop
-from pypy.rpython.llinterp import LLException
from pypy.jit.codewriter.jitcode import JitCode, SwitchDictDescr
from pypy.jit.codewriter import heaptracker, longlong
from pypy.jit.metainterp.jitexc import JitException, get_llexception, reraise
@@ -500,9 +499,25 @@
@arguments("r", returns="i")
def bhimpl_ptr_nonzero(a):
return bool(a)
- @arguments("r", returns="r")
- def bhimpl_cast_opaque_ptr(a):
- return a
+ @arguments("r", "r", returns="i")
+ def bhimpl_instance_ptr_eq(a, b):
+ return a == b
+ @arguments("r", "r", returns="i")
+ def bhimpl_instance_ptr_ne(a, b):
+ return a != b
+ @arguments("r", returns="i")
+ def bhimpl_cast_ptr_to_int(a):
+ i = lltype.cast_ptr_to_int(a)
+ ll_assert((i & 1) == 1, "bhimpl_cast_ptr_to_int: not an odd int")
+ return i
+ @arguments("i", returns="r")
+ def bhimpl_cast_int_to_ptr(i):
+ ll_assert((i & 1) == 1, "bhimpl_cast_int_to_ptr: not an odd int")
+ return lltype.cast_int_to_ptr(llmemory.GCREF, i)
+
+ @arguments("r")
+ def bhimpl_mark_opaque_ptr(a):
+ pass
@arguments("i", returns="i")
def bhimpl_int_copy(a):
@@ -523,6 +538,9 @@
@arguments("f")
def bhimpl_float_guard_value(a):
pass
+ @arguments("r", "i", "d", returns="r")
+ def bhimpl_str_guard_value(a, i, d):
+ return a
@arguments("self", "i")
def bhimpl_int_push(self, a):
@@ -619,6 +637,9 @@
a = longlong.getrealfloat(a)
# note: we need to call int() twice to care for the fact that
# int(-2147483648.0) returns a long :-(
+ # we could also call intmask() instead of the outermost int(), but
+ # it's probably better to explicitly crash (by getting a long) if a
+ # non-translated version tries to cast a too large float to an int.
return int(int(a))
@arguments("i", returns="f")
@@ -1142,6 +1163,26 @@
array = cpu.bh_getfield_gc_r(vable, fdescr)
return cpu.bh_arraylen_gc(adescr, array)
+ @arguments("cpu", "r", "i", "d", returns="i")
+ def bhimpl_getinteriorfield_gc_i(cpu, array, index, descr):
+ return cpu.bh_getinteriorfield_gc_i(array, index, descr)
+ @arguments("cpu", "r", "i", "d", returns="r")
+ def bhimpl_getinteriorfield_gc_r(cpu, array, index, descr):
+ return cpu.bh_getinteriorfield_gc_r(array, index, descr)
+ @arguments("cpu", "r", "i", "d", returns="f")
+ def bhimpl_getinteriorfield_gc_f(cpu, array, index, descr):
+ return cpu.bh_getinteriorfield_gc_f(array, index, descr)
+
+ @arguments("cpu", "r", "i", "d", "i")
+ def bhimpl_setinteriorfield_gc_i(cpu, array, index, descr, value):
+ cpu.bh_setinteriorfield_gc_i(array, index, descr, value)
+ @arguments("cpu", "r", "i", "d", "r")
+ def bhimpl_setinteriorfield_gc_r(cpu, array, index, descr, value):
+ cpu.bh_setinteriorfield_gc_r(array, index, descr, value)
+ @arguments("cpu", "r", "i", "d", "f")
+ def bhimpl_setinteriorfield_gc_f(cpu, array, index, descr, value):
+ cpu.bh_setinteriorfield_gc_f(array, index, descr, value)
+
@arguments("cpu", "r", "d", returns="i")
def bhimpl_getfield_gc_i(cpu, struct, fielddescr):
return cpu.bh_getfield_gc_i(struct, fielddescr)
diff --git a/pypy/jit/metainterp/executor.py b/pypy/jit/metainterp/executor.py
--- a/pypy/jit/metainterp/executor.py
+++ b/pypy/jit/metainterp/executor.py
@@ -1,11 +1,8 @@
"""This implements pyjitpl's execution of operations.
"""
-import py
-from pypy.rpython.lltypesystem import lltype, llmemory, rstr
-from pypy.rpython.ootypesystem import ootype
-from pypy.rpython.lltypesystem.lloperation import llop
-from pypy.rlib.rarithmetic import ovfcheck, r_uint, intmask, r_longlong
+from pypy.rpython.lltypesystem import lltype, rstr
+from pypy.rlib.rarithmetic import ovfcheck, r_longlong
from pypy.rlib.rtimer import read_timestamp
from pypy.rlib.unroll import unrolling_iterable
from pypy.jit.metainterp.history import BoxInt, BoxPtr, BoxFloat, check_descr
@@ -123,6 +120,29 @@
else:
cpu.bh_setarrayitem_raw_i(arraydescr, array, index, itembox.getint())
+def do_getinteriorfield_gc(cpu, _, arraybox, indexbox, descr):
+ array = arraybox.getref_base()
+ index = indexbox.getint()
+ if descr.is_pointer_field():
+ return BoxPtr(cpu.bh_getinteriorfield_gc_r(array, index, descr))
+ elif descr.is_float_field():
+ return BoxFloat(cpu.bh_getinteriorfield_gc_f(array, index, descr))
+ else:
+ return BoxInt(cpu.bh_getinteriorfield_gc_i(array, index, descr))
+
+def do_setinteriorfield_gc(cpu, _, arraybox, indexbox, valuebox, descr):
+ array = arraybox.getref_base()
+ index = indexbox.getint()
+ if descr.is_pointer_field():
+ cpu.bh_setinteriorfield_gc_r(array, index, descr,
+ valuebox.getref_base())
+ elif descr.is_float_field():
+ cpu.bh_setinteriorfield_gc_f(array, index, descr,
+ valuebox.getfloatstorage())
+ else:
+ cpu.bh_setinteriorfield_gc_i(array, index, descr,
+ valuebox.getint())
+
def do_getfield_gc(cpu, _, structbox, fielddescr):
struct = structbox.getref_base()
if fielddescr.is_pointer_field():
diff --git a/pypy/jit/metainterp/graphpage.py b/pypy/jit/metainterp/graphpage.py
--- a/pypy/jit/metainterp/graphpage.py
+++ b/pypy/jit/metainterp/graphpage.py
@@ -12,8 +12,8 @@
def get_display_text(self):
return None
-def display_loops(loops, errmsg=None, highlight_loops=()):
- graphs = [(loop, loop in highlight_loops) for loop in loops]
+def display_loops(loops, errmsg=None, highlight_loops={}):
+ graphs = [(loop, highlight_loops.get(loop, 0)) for loop in loops]
for graph, highlight in graphs:
for op in graph.get_operations():
if is_interesting_guard(op):
@@ -65,8 +65,7 @@
def add_graph(self, graph, highlight=False):
graphindex = len(self.graphs)
self.graphs.append(graph)
- if highlight:
- self.highlight_graphs[graph] = True
+ self.highlight_graphs[graph] = highlight
for i, op in enumerate(graph.get_operations()):
self.all_operations[op] = graphindex, i
@@ -126,10 +125,13 @@
self.dotgen.emit('subgraph cluster%d {' % graphindex)
label = graph.get_display_text()
if label is not None:
- if self.highlight_graphs.get(graph):
- fillcolor = '#f084c2'
+ colorindex = self.highlight_graphs.get(graph, 0)
+ if colorindex == 1:
+ fillcolor = '#f084c2' # highlighted graph
+ elif colorindex == 2:
+ fillcolor = '#808080' # invalidated graph
else:
- fillcolor = '#84f0c2'
+ fillcolor = '#84f0c2' # normal color
self.dotgen.emit_node(graphname, shape="octagon",
label=label, fillcolor=fillcolor)
self.pendingedges.append((graphname,
diff --git a/pypy/jit/metainterp/heapcache.py b/pypy/jit/metainterp/heapcache.py
--- a/pypy/jit/metainterp/heapcache.py
+++ b/pypy/jit/metainterp/heapcache.py
@@ -34,7 +34,6 @@
self.clear_caches(opnum, descr, argboxes)
def mark_escaped(self, opnum, argboxes):
- idx = 0
if opnum == rop.SETFIELD_GC:
assert len(argboxes) == 2
box, valuebox = argboxes
@@ -42,8 +41,20 @@
self.dependencies.setdefault(box, []).append(valuebox)
else:
self._escape(valuebox)
- # GETFIELD_GC doesn't escape it's argument
- elif opnum != rop.GETFIELD_GC:
+ elif opnum == rop.SETARRAYITEM_GC:
+ assert len(argboxes) == 3
+ box, indexbox, valuebox = argboxes
+ if self.is_unescaped(box) and self.is_unescaped(valuebox):
+ self.dependencies.setdefault(box, []).append(valuebox)
+ else:
+ self._escape(valuebox)
+ # GETFIELD_GC, MARK_OPAQUE_PTR, PTR_EQ, and PTR_NE don't escape their
+ # arguments
+ elif (opnum != rop.GETFIELD_GC and
+ opnum != rop.MARK_OPAQUE_PTR and
+ opnum != rop.PTR_EQ and
+ opnum != rop.PTR_NE):
+ idx = 0
for box in argboxes:
# setarrayitem_gc don't escape its first argument
if not (idx == 0 and opnum in [rop.SETARRAYITEM_GC]):
@@ -54,18 +65,19 @@
if box in self.new_boxes:
self.new_boxes[box] = False
if box in self.dependencies:
- for dep in self.dependencies[box]:
+ deps = self.dependencies[box]
+ del self.dependencies[box]
+ for dep in deps:
self._escape(dep)
- del self.dependencies[box]
def clear_caches(self, opnum, descr, argboxes):
- if opnum == rop.SETFIELD_GC:
- return
- if opnum == rop.SETARRAYITEM_GC:
- return
- if opnum == rop.SETFIELD_RAW:
- return
- if opnum == rop.SETARRAYITEM_RAW:
+ if (opnum == rop.SETFIELD_GC or
+ opnum == rop.SETARRAYITEM_GC or
+ opnum == rop.SETFIELD_RAW or
+ opnum == rop.SETARRAYITEM_RAW or
+ opnum == rop.SETINTERIORFIELD_GC or
+ opnum == rop.COPYSTRCONTENT or
+ opnum == rop.COPYUNICODECONTENT):
return
if rop._OVF_FIRST <= opnum <= rop._OVF_LAST:
return
@@ -74,9 +86,9 @@
if opnum == rop.CALL or opnum == rop.CALL_LOOPINVARIANT:
effectinfo = descr.get_extra_info()
ef = effectinfo.extraeffect
- if ef == effectinfo.EF_LOOPINVARIANT or \
- ef == effectinfo.EF_ELIDABLE_CANNOT_RAISE or \
- ef == effectinfo.EF_ELIDABLE_CAN_RAISE:
+ if (ef == effectinfo.EF_LOOPINVARIANT or
+ ef == effectinfo.EF_ELIDABLE_CANNOT_RAISE or
+ ef == effectinfo.EF_ELIDABLE_CAN_RAISE):
return
# A special case for ll_arraycopy, because it is so common, and its
# effects are so well defined.
diff --git a/pypy/jit/metainterp/history.py b/pypy/jit/metainterp/history.py
--- a/pypy/jit/metainterp/history.py
+++ b/pypy/jit/metainterp/history.py
@@ -9,12 +9,14 @@
from pypy.jit.metainterp.resoperation import ResOperation, rop
from pypy.jit.codewriter import heaptracker, longlong
+from pypy.rlib.objectmodel import compute_identity_hash
# ____________________________________________________________
INT = 'i'
REF = 'r'
FLOAT = 'f'
+STRUCT = 's'
HOLE = '_'
VOID = 'v'
@@ -104,7 +106,7 @@
getref._annspecialcase_ = 'specialize:arg(1)'
def _get_hash_(self):
- raise NotImplementedError
+ return compute_identity_hash(self)
def clonebox(self):
raise NotImplementedError
@@ -133,6 +135,9 @@
def _get_str(self):
raise NotImplementedError
+ def same_box(self, other):
+ return self is other
+
class AbstractDescr(AbstractValue):
__slots__ = ()
@@ -168,6 +173,11 @@
"""
raise NotImplementedError
+ def is_array_of_structs(self):
+ """ Implement for array descr
+ """
+ raise NotImplementedError
+
def is_pointer_field(self):
""" Implement for field descr
"""
@@ -241,32 +251,15 @@
def constbox(self):
return self
+ def same_box(self, other):
+ return self.same_constant(other)
+
def same_constant(self, other):
raise NotImplementedError
def __repr__(self):
return 'Const(%s)' % self._getrepr_()
- def __eq__(self, other):
- "NOT_RPYTHON"
- # Remember that you should not compare Consts with '==' in RPython.
- # Consts have no special __hash__, in order to force different Consts
- # from being considered as different keys when stored in dicts
- # (as they always are after translation). Use a dict_equal_consts()
- # to get the other behavior (i.e. using this __eq__).
- if self.__class__ is not other.__class__:
- return False
- try:
- return self.value == other.value
- except TypeError:
- if (isinstance(self.value, Symbolic) and
- isinstance(other.value, Symbolic)):
- return self.value is other.value
- raise
-
- def __ne__(self, other):
- return not (self == other)
-
class ConstInt(Const):
type = INT
@@ -688,33 +681,6 @@
# ____________________________________________________________
-def dict_equal_consts():
- "NOT_RPYTHON"
- # Returns a dict in which Consts that compare as equal
- # are identified when used as keys.
- return r_dict(dc_eq, dc_hash)
-
-def dc_eq(c1, c2):
- return c1 == c2
-
-def dc_hash(c):
- "NOT_RPYTHON"
- # This is called during translation only. Avoid using identityhash(),
- # to avoid forcing a hash, at least on lltype objects.
- if not isinstance(c, Const):
- return hash(c)
- if isinstance(c.value, Symbolic):
- return id(c.value)
- try:
- if isinstance(c, ConstPtr):
- p = lltype.normalizeptr(c.value)
- if p is not None:
- return hash(p._obj)
- else:
- return 0
- return c._get_hash_()
- except lltype.DelayedPointer:
- return -2 # xxx risk of changing hash...
def make_hashable_int(i):
from pypy.rpython.lltypesystem.ll2ctypes import NotCtypesAllocatedStructure
@@ -772,6 +738,7 @@
failed_states = None
retraced_count = 0
terminating = False # see TerminatingLoopToken in compile.py
+ invalidated = False
outermost_jitdriver_sd = None
# and more data specified by the backend when the loop is compiled
number = -1
@@ -974,6 +941,7 @@
self.loops = []
self.locations = []
self.aborted_keys = []
+ self.invalidated_token_numbers = set()
def set_history(self, history):
self.operations = history.operations
@@ -1052,7 +1020,12 @@
if loop in loops:
loops.remove(loop)
loops.append(loop)
- display_loops(loops, errmsg, extraloops)
+ highlight_loops = dict.fromkeys(extraloops, 1)
+ for loop in loops:
+ if hasattr(loop, '_looptoken_number') and (
+ loop._looptoken_number in self.invalidated_token_numbers):
+ highlight_loops.setdefault(loop, 2)
+ display_loops(loops, errmsg, highlight_loops)
# ----------------------------------------------------------------
diff --git a/pypy/jit/metainterp/memmgr.py b/pypy/jit/metainterp/memmgr.py
--- a/pypy/jit/metainterp/memmgr.py
+++ b/pypy/jit/metainterp/memmgr.py
@@ -68,7 +68,8 @@
debug_print("Loop tokens before:", oldtotal)
max_generation = self.current_generation - (self.max_age-1)
for looptoken in self.alive_loops.keys():
- if 0 <= looptoken.generation < max_generation:
+ if (0 <= looptoken.generation < max_generation or
+ looptoken.invalidated):
del self.alive_loops[looptoken]
newtotal = len(self.alive_loops)
debug_print("Loop tokens freed: ", oldtotal - newtotal)
diff --git a/pypy/jit/metainterp/optimizeopt/optimizer.py b/pypy/jit/metainterp/optimizeopt/optimizer.py
--- a/pypy/jit/metainterp/optimizeopt/optimizer.py
+++ b/pypy/jit/metainterp/optimizeopt/optimizer.py
@@ -209,13 +209,19 @@
def setfield(self, ofs, value):
raise NotImplementedError
+ def getlength(self):
+ raise NotImplementedError
+
def getitem(self, index):
raise NotImplementedError
- def getlength(self):
+ def setitem(self, index, value):
raise NotImplementedError
- def setitem(self, index, value):
+ def getinteriorfield(self, index, ofs, default):
+ raise NotImplementedError
+
+ def setinteriorfield(self, index, ofs, value):
raise NotImplementedError
@@ -283,11 +289,11 @@
return self.optimizer.optpure.has_pure_result(opnum, args, descr)
return False
- def get_pure_result(self, key):
+ def get_pure_result(self, key):
if self.optimizer.optpure:
return self.optimizer.optpure.get_pure_result(key)
return None
-
+
def setup(self):
pass
@@ -524,7 +530,7 @@
def replace_op(self, old_op, new_op):
# XXX: Do we want to cache indexes to prevent search?
- i = len(self._newoperations)
+ i = len(self._newoperations)
while i > 0:
i -= 1
if self._newoperations[i] is old_op:
diff --git a/pypy/jit/metainterp/optimizeopt/rewrite.py b/pypy/jit/metainterp/optimizeopt/rewrite.py
--- a/pypy/jit/metainterp/optimizeopt/rewrite.py
+++ b/pypy/jit/metainterp/optimizeopt/rewrite.py
@@ -106,10 +106,9 @@
self.make_equal_to(op.result, v1)
else:
self.emit_operation(op)
-
- # Synthesize the reverse ops for optimize_default to reuse
- self.pure(rop.INT_ADD, [op.result, op.getarg(1)], op.getarg(0))
- self.pure(rop.INT_SUB, [op.getarg(0), op.result], op.getarg(1))
+ # Synthesize the reverse ops for optimize_default to reuse
+ self.pure(rop.INT_ADD, [op.result, op.getarg(1)], op.getarg(0))
+ self.pure(rop.INT_SUB, [op.getarg(0), op.result], op.getarg(1))
def optimize_INT_ADD(self, op):
v1 = self.getvalue(op.getarg(0))
@@ -122,10 +121,9 @@
self.make_equal_to(op.result, v1)
else:
self.emit_operation(op)
-
- # Synthesize the reverse op for optimize_default to reuse
- self.pure(rop.INT_SUB, [op.result, op.getarg(1)], op.getarg(0))
- self.pure(rop.INT_SUB, [op.result, op.getarg(0)], op.getarg(1))
+ # Synthesize the reverse op for optimize_default to reuse
+ self.pure(rop.INT_SUB, [op.result, op.getarg(1)], op.getarg(0))
+ self.pure(rop.INT_SUB, [op.result, op.getarg(0)], op.getarg(1))
def optimize_INT_MUL(self, op):
v1 = self.getvalue(op.getarg(0))
@@ -141,13 +139,13 @@
self.make_constant_int(op.result, 0)
else:
for lhs, rhs in [(v1, v2), (v2, v1)]:
- # x & (x -1) == 0 is a quick test for power of 2
- if (lhs.is_constant() and
- (lhs.box.getint() & (lhs.box.getint() - 1)) == 0):
- new_rhs = ConstInt(highest_bit(lhs.box.getint()))
- op = op.copy_and_change(rop.INT_LSHIFT, args=[rhs.box, new_rhs])
- break
-
+ if lhs.is_constant():
+ x = lhs.box.getint()
+ # x & (x - 1) == 0 is a quick test for power of 2
+ if x & (x - 1) == 0:
+ new_rhs = ConstInt(highest_bit(lhs.box.getint()))
+ op = op.copy_and_change(rop.INT_LSHIFT, args=[rhs.box, new_rhs])
+ break
self.emit_operation(op)
def optimize_UINT_FLOORDIV(self, op):
@@ -339,7 +337,7 @@
def optimize_INT_IS_ZERO(self, op):
self._optimize_nullness(op, op.getarg(0), False)
- def _optimize_oois_ooisnot(self, op, expect_isnot):
+ def _optimize_oois_ooisnot(self, op, expect_isnot, instance):
value0 = self.getvalue(op.getarg(0))
value1 = self.getvalue(op.getarg(1))
if value0.is_virtual():
@@ -357,21 +355,28 @@
elif value0 is value1:
self.make_constant_int(op.result, not expect_isnot)
else:
- cls0 = value0.get_constant_class(self.optimizer.cpu)
- if cls0 is not None:
- cls1 = value1.get_constant_class(self.optimizer.cpu)
- if cls1 is not None and not cls0.same_constant(cls1):
- # cannot be the same object, as we know that their
- # class is different
- self.make_constant_int(op.result, expect_isnot)
- return
+ if instance:
+ cls0 = value0.get_constant_class(self.optimizer.cpu)
+ if cls0 is not None:
+ cls1 = value1.get_constant_class(self.optimizer.cpu)
+ if cls1 is not None and not cls0.same_constant(cls1):
+ # cannot be the same object, as we know that their
+ # class is different
+ self.make_constant_int(op.result, expect_isnot)
+ return
self.emit_operation(op)
+ def optimize_PTR_EQ(self, op):
+ self._optimize_oois_ooisnot(op, False, False)
+
def optimize_PTR_NE(self, op):
- self._optimize_oois_ooisnot(op, True)
+ self._optimize_oois_ooisnot(op, True, False)
- def optimize_PTR_EQ(self, op):
- self._optimize_oois_ooisnot(op, False)
+ def optimize_INSTANCE_PTR_EQ(self, op):
+ self._optimize_oois_ooisnot(op, False, True)
+
+ def optimize_INSTANCE_PTR_NE(self, op):
+ self._optimize_oois_ooisnot(op, True, True)
## def optimize_INSTANCEOF(self, op):
## value = self.getvalue(op.args[0])
@@ -450,6 +455,9 @@
if v2.is_constant() and v2.box.getint() == 1:
self.make_equal_to(op.result, v1)
return
+ elif v1.is_constant() and v1.box.getint() == 0:
+ self.make_constant_int(op.result, 0)
+ return
if v1.intbound.known_ge(IntBound(0, 0)) and v2.is_constant():
val = v2.box.getint()
if val & (val - 1) == 0 and val > 0: # val == 2**shift
@@ -457,10 +465,17 @@
args = [op.getarg(0), ConstInt(highest_bit(val))])
self.emit_operation(op)
- def optimize_CAST_OPAQUE_PTR(self, op):
+ def optimize_MARK_OPAQUE_PTR(self, op):
value = self.getvalue(op.getarg(0))
self.optimizer.opaque_pointers[value] = True
- self.make_equal_to(op.result, value)
+
+ def optimize_CAST_PTR_TO_INT(self, op):
+ self.pure(rop.CAST_INT_TO_PTR, [op.result], op.getarg(0))
+ self.emit_operation(op)
+
+ def optimize_CAST_INT_TO_PTR(self, op):
+ self.pure(rop.CAST_PTR_TO_INT, [op.result], op.getarg(0))
+ self.emit_operation(op)
dispatch_opt = make_dispatcher_method(OptRewrite, 'optimize_',
default=OptRewrite.emit_operation)
diff --git a/pypy/jit/metainterp/optimizeopt/simplify.py b/pypy/jit/metainterp/optimizeopt/simplify.py
--- a/pypy/jit/metainterp/optimizeopt/simplify.py
+++ b/pypy/jit/metainterp/optimizeopt/simplify.py
@@ -25,7 +25,8 @@
# but it's a bit hard to implement robustly if heap.py is also run
pass
- optimize_CAST_OPAQUE_PTR = optimize_VIRTUAL_REF
+ def optimize_MARK_OPAQUE_PTR(self, op):
+ pass
dispatch_opt = make_dispatcher_method(OptSimplify, 'optimize_',
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -508,13 +508,13 @@
ops = """
[p0]
guard_class(p0, ConstClass(node_vtable)) []
- i0 = ptr_ne(p0, NULL)
+ i0 = instance_ptr_ne(p0, NULL)
guard_true(i0) []
- i1 = ptr_eq(p0, NULL)
+ i1 = instance_ptr_eq(p0, NULL)
guard_false(i1) []
- i2 = ptr_ne(NULL, p0)
+ i2 = instance_ptr_ne(NULL, p0)
guard_true(i0) []
- i3 = ptr_eq(NULL, p0)
+ i3 = instance_ptr_eq(NULL, p0)
guard_false(i1) []
jump(p0)
"""
@@ -935,7 +935,6 @@
"""
self.optimize_loop(ops, expected)
-
def test_virtual_constant_isnonnull(self):
ops = """
[i0]
@@ -951,6 +950,32 @@
"""
self.optimize_loop(ops, expected)
+ def test_virtual_array_of_struct(self):
+ ops = """
+ [f0, f1, f2, f3]
+ p0 = new_array(2, descr=complexarraydescr)
+ setinteriorfield_gc(p0, 0, f0, descr=complexrealdescr)
+ setinteriorfield_gc(p0, 0, f1, descr=compleximagdescr)
+ setinteriorfield_gc(p0, 1, f2, descr=complexrealdescr)
+ setinteriorfield_gc(p0, 1, f3, descr=compleximagdescr)
+ f4 = getinteriorfield_gc(p0, 0, descr=complexrealdescr)
+ f5 = getinteriorfield_gc(p0, 1, descr=complexrealdescr)
+ f6 = float_mul(f4, f5)
+ f7 = getinteriorfield_gc(p0, 0, descr=compleximagdescr)
+ f8 = getinteriorfield_gc(p0, 1, descr=compleximagdescr)
+ f9 = float_mul(f7, f8)
+ f10 = float_add(f6, f9)
+ finish(f10)
+ """
+ expected = """
+ [f0, f1, f2, f3]
+ f4 = float_mul(f0, f2)
+ f5 = float_mul(f1, f3)
+ f6 = float_add(f4, f5)
+ finish(f6)
+ """
+ self.optimize_loop(ops, expected)
+
def test_nonvirtual_1(self):
ops = """
[i]
@@ -2026,7 +2051,7 @@
ops = """
[p1]
guard_class(p1, ConstClass(node_vtable2)) []
- i = ptr_ne(ConstPtr(myptr), p1)
+ i = instance_ptr_ne(ConstPtr(myptr), p1)
guard_true(i) []
jump(p1)
"""
@@ -2181,6 +2206,17 @@
"""
self.optimize_loop(ops, expected)
+ ops = """
+ [i0]
+ i1 = int_floordiv(0, i0)
+ jump(i1)
+ """
+ expected = """
+ [i0]
+ jump(0)
+ """
+ self.optimize_loop(ops, expected)
+
def test_fold_partially_constant_ops_ovf(self):
ops = """
[i0]
@@ -2329,7 +2365,7 @@
def _variables_equal(box, varname, strict):
if varname not in virtuals:
if strict:
- assert box == oparse.getvar(varname)
+ assert box.same_box(oparse.getvar(varname))
else:
assert box.value == oparse.getvar(varname).value
else:
@@ -4170,10 +4206,12 @@
class FakeCallInfoCollection:
def callinfo_for_oopspec(self, oopspecindex):
calldescrtype = type(LLtypeMixin.strequaldescr)
+ effectinfotype = type(LLtypeMixin.strequaldescr.get_extra_info())
for value in LLtypeMixin.__dict__.values():
if isinstance(value, calldescrtype):
extra = value.get_extra_info()
- if extra and extra.oopspecindex == oopspecindex:
+ if (extra and isinstance(extra, effectinfotype) and
+ extra.oopspecindex == oopspecindex):
# returns 0 for 'func' in this test
return value, 0
raise AssertionError("not found: oopspecindex=%d" %
@@ -4789,6 +4827,18 @@
"""
self.optimize_strunicode_loop(ops, expected)
+ def test_ptr_eq_str_constant(self):
+ ops = """
+ []
+ i0 = ptr_eq(s"abc", s"\x00")
+ finish(i0)
+ """
+ expected = """
+ []
+ finish(0)
+ """
+ self.optimize_loop(ops, expected)
+
class TestLLtype(BaseTestOptimizeBasic, LLtypeMixin):
pass
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py b/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
@@ -234,6 +234,30 @@
""" % expected_value
self.optimize_loop(ops, expected)
+ def test_reverse_of_cast(self):
+ ops = """
+ [i0]
+ p0 = cast_int_to_ptr(i0)
+ i1 = cast_ptr_to_int(p0)
+ jump(i1)
+ """
+ expected = """
+ [i0]
+ jump(i0)
+ """
+ self.optimize_loop(ops, expected)
+ ops = """
+ [p0]
+ i1 = cast_ptr_to_int(p0)
+ p1 = cast_int_to_ptr(i1)
+ jump(p1)
+ """
+ expected = """
+ [p0]
+ jump(p0)
+ """
+ self.optimize_loop(ops, expected)
+
# ----------
def test_remove_guard_class_1(self):
@@ -2659,7 +2683,7 @@
ops = """
[p1]
guard_class(p1, ConstClass(node_vtable2)) []
- i = ptr_ne(ConstPtr(myptr), p1)
+ i = instance_ptr_ne(ConstPtr(myptr), p1)
guard_true(i) []
jump(p1)
"""
@@ -3307,7 +3331,7 @@
jump(p1, i1, i2, i6)
'''
self.optimize_loop(ops, expected, preamble)
-
+
# ----------
@@ -5776,10 +5800,12 @@
class FakeCallInfoCollection:
def callinfo_for_oopspec(self, oopspecindex):
calldescrtype = type(LLtypeMixin.strequaldescr)
+ effectinfotype = type(LLtypeMixin.strequaldescr.get_extra_info())
for value in LLtypeMixin.__dict__.values():
if isinstance(value, calldescrtype):
extra = value.get_extra_info()
- if extra and extra.oopspecindex == oopspecindex:
+ if (extra and isinstance(extra, effectinfotype) and
+ extra.oopspecindex == oopspecindex):
# returns 0 for 'func' in this test
return value, 0
raise AssertionError("not found: oopspecindex=%d" %
@@ -7256,7 +7282,7 @@
ops = """
[p1, p2]
setarrayitem_gc(p1, 2, 10, descr=arraydescr)
- setarrayitem_gc(p2, 3, 13, descr=arraydescr)
+ setarrayitem_gc(p2, 3, 13, descr=arraydescr)
call(0, p1, p2, 0, 0, 10, descr=arraycopydescr)
jump(p1, p2)
"""
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_util.py b/pypy/jit/metainterp/optimizeopt/test/test_util.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_util.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_util.py
@@ -185,6 +185,18 @@
EffectInfo([], [arraydescr], [], [arraydescr],
oopspecindex=EffectInfo.OS_ARRAYCOPY))
+
+ # array of structs (complex data)
+ complexarray = lltype.GcArray(
+ lltype.Struct("complex",
+ ("real", lltype.Float),
+ ("imag", lltype.Float),
+ )
+ )
+ complexarraydescr = cpu.arraydescrof(complexarray)
+ complexrealdescr = cpu.interiorfielddescrof(complexarray, "real")
+ compleximagdescr = cpu.interiorfielddescrof(complexarray, "imag")
+
for _name, _os in [
('strconcatdescr', 'OS_STR_CONCAT'),
('strslicedescr', 'OS_STR_SLICE'),
@@ -240,7 +252,7 @@
## def get_class_of_box(self, box):
## root = box.getref(ootype.ROOT)
## return ootype.classof(root)
-
+
## cpu = runner.OOtypeCPU(None)
## NODE = ootype.Instance('NODE', ootype.ROOT, {})
## NODE._add_fields({'value': ootype.Signed,
diff --git a/pypy/jit/metainterp/optimizeopt/util.py b/pypy/jit/metainterp/optimizeopt/util.py
--- a/pypy/jit/metainterp/optimizeopt/util.py
+++ b/pypy/jit/metainterp/optimizeopt/util.py
@@ -90,14 +90,11 @@
for i in range(len(args1)):
arg1 = args1[i]
arg2 = args2[i]
- if isinstance(arg1, history.Const):
- if arg1.__class__ is not arg2.__class__:
+ if arg1 is None:
+ if arg2 is not None:
return False
- if not arg1.same_constant(arg2):
- return False
- else:
- if not arg1 is arg2:
- return False
+ elif not arg1.same_box(arg2):
+ return False
return True
def args_hash(args):
@@ -106,10 +103,8 @@
for arg in args:
if arg is None:
y = 17
- elif isinstance(arg, history.Const):
+ else:
y = arg._get_hash_()
- else:
- y = compute_identity_hash(arg)
res = intmask((1000003 * res) ^ y)
return res
@@ -145,9 +140,12 @@
for i in range(op1.numargs()):
x = op1.getarg(i)
y = op2.getarg(i)
- assert x == remap.get(y, y)
+ assert x.same_box(remap.get(y, y))
if op2.result in remap:
- assert op1.result == remap[op2.result]
+ if op2.result is None:
+ assert op1.result == remap[op2.result]
+ else:
+ assert op1.result.same_box(remap[op2.result])
else:
remap[op2.result] = op1.result
if op1.getopnum() != rop.JUMP: # xxx obscure
@@ -156,11 +154,20 @@
assert len(op1.getfailargs()) == len(op2.getfailargs())
if strict_fail_args:
for x, y in zip(op1.getfailargs(), op2.getfailargs()):
- assert x == remap.get(y, y)
+ if x is None:
+ assert remap.get(y, y) is None
+ else:
+ assert x.same_box(remap.get(y, y))
else:
fail_args1 = set(op1.getfailargs())
fail_args2 = set([remap.get(y, y) for y in op2.getfailargs()])
- assert fail_args1 == fail_args2
+ for x in fail_args1:
+ for y in fail_args2:
+ if x.same_box(y):
+ fail_args2.remove(y)
+ break
+ else:
+ assert False
assert len(oplist1) == len(oplist2)
print '-'*totwidth
return True
diff --git a/pypy/jit/metainterp/optimizeopt/virtualize.py b/pypy/jit/metainterp/optimizeopt/virtualize.py
--- a/pypy/jit/metainterp/optimizeopt/virtualize.py
+++ b/pypy/jit/metainterp/optimizeopt/virtualize.py
@@ -59,7 +59,7 @@
def import_from(self, other, optimizer):
raise NotImplementedError("should not be called at this level")
-
+
def get_fielddescrlist_cache(cpu):
if not hasattr(cpu, '_optimizeopt_fielddescrlist_cache'):
result = descrlist_dict()
@@ -113,7 +113,7 @@
#
if not we_are_translated():
op.name = 'FORCE ' + self.source_op.name
-
+
if self._is_immutable_and_filled_with_constants(optforce):
box = optforce.optimizer.constant_fold(op)
self.make_constant(box)
@@ -239,12 +239,12 @@
for index in range(len(self._items)):
self._items[index] = self._items[index].force_at_end_of_preamble(already_forced, optforce)
return self
-
+
def _really_force(self, optforce):
assert self.source_op is not None
if not we_are_translated():
self.source_op.name = 'FORCE ' + self.source_op.name
- optforce.emit_operation(self.source_op)
+ optforce.emit_operation(self.source_op)
self.box = box = self.source_op.result
for index in range(len(self._items)):
subvalue = self._items[index]
@@ -271,20 +271,86 @@
def _make_virtual(self, modifier):
return modifier.make_varray(self.arraydescr)
+class VArrayStructValue(AbstractVirtualValue):
+ def __init__(self, arraydescr, size, keybox, source_op=None):
+ AbstractVirtualValue.__init__(self, keybox, source_op)
+ self.arraydescr = arraydescr
+ self._items = [{} for _ in xrange(size)]
+
+ def getlength(self):
+ return len(self._items)
+
+ def getinteriorfield(self, index, ofs, default):
+ return self._items[index].get(ofs, default)
+
+ def setinteriorfield(self, index, ofs, itemvalue):
+ assert isinstance(itemvalue, optimizer.OptValue)
+ self._items[index][ofs] = itemvalue
+
+ def _really_force(self, optforce):
+ assert self.source_op is not None
+ if not we_are_translated():
+ self.source_op.name = 'FORCE ' + self.source_op.name
+ optforce.emit_operation(self.source_op)
+ self.box = box = self.source_op.result
+ for index in range(len(self._items)):
+ for descr, value in self._items[index].iteritems():
+ subbox = value.force_box(optforce)
+ op = ResOperation(rop.SETINTERIORFIELD_GC,
+ [box, ConstInt(index), subbox], None, descr=descr
+ )
+ optforce.emit_operation(op)
+
+ def _get_list_of_descrs(self):
+ descrs = []
+ for item in self._items:
+ item_descrs = item.keys()
+ sort_descrs(item_descrs)
+ descrs.append(item_descrs)
+ return descrs
+
+ def get_args_for_fail(self, modifier):
+ if self.box is None and not modifier.already_seen_virtual(self.keybox):
+ itemdescrs = self._get_list_of_descrs()
+ itemboxes = []
+ for i in range(len(self._items)):
+ for descr in itemdescrs[i]:
+ itemboxes.append(self._items[i][descr].get_key_box())
+ modifier.register_virtual_fields(self.keybox, itemboxes)
+ for i in range(len(self._items)):
+ for descr in itemdescrs[i]:
+ self._items[i][descr].get_args_for_fail(modifier)
+
+ def force_at_end_of_preamble(self, already_forced, optforce):
+ if self in already_forced:
+ return self
+ already_forced[self] = self
+ for index in range(len(self._items)):
+ for descr in self._items[index].keys():
+ self._items[index][descr] = self._items[index][descr].force_at_end_of_preamble(already_forced, optforce)
+ return self
+
+ def _make_virtual(self, modifier):
+ return modifier.make_varraystruct(self.arraydescr, self._get_list_of_descrs())
+
+
class OptVirtualize(optimizer.Optimization):
"Virtualize objects until they escape."
def new(self):
return OptVirtualize()
-
+
def make_virtual(self, known_class, box, source_op=None):
vvalue = VirtualValue(self.optimizer.cpu, known_class, box, source_op)
self.make_equal_to(box, vvalue)
return vvalue
def make_varray(self, arraydescr, size, box, source_op=None):
- constvalue = self.new_const_item(arraydescr)
- vvalue = VArrayValue(arraydescr, constvalue, size, box, source_op)
+ if arraydescr.is_array_of_structs():
+ vvalue = VArrayStructValue(arraydescr, size, box, source_op)
+ else:
+ constvalue = self.new_const_item(arraydescr)
+ vvalue = VArrayValue(arraydescr, constvalue, size, box, source_op)
self.make_equal_to(box, vvalue)
return vvalue
@@ -431,6 +497,34 @@
value.ensure_nonnull()
self.emit_operation(op)
+ def optimize_GETINTERIORFIELD_GC(self, op):
+ value = self.getvalue(op.getarg(0))
+ if value.is_virtual():
+ indexbox = self.get_constant_box(op.getarg(1))
+ if indexbox is not None:
+ descr = op.getdescr()
+ fieldvalue = value.getinteriorfield(
+ indexbox.getint(), descr, None
+ )
+ if fieldvalue is None:
+ fieldvalue = self.new_const(descr)
+ self.make_equal_to(op.result, fieldvalue)
+ return
+ value.ensure_nonnull()
+ self.emit_operation(op)
+
+ def optimize_SETINTERIORFIELD_GC(self, op):
+ value = self.getvalue(op.getarg(0))
+ if value.is_virtual():
+ indexbox = self.get_constant_box(op.getarg(1))
+ if indexbox is not None:
+ value.setinteriorfield(
+ indexbox.getint(), op.getdescr(), self.getvalue(op.getarg(2))
+ )
+ return
+ value.ensure_nonnull()
+ self.emit_operation(op)
+
dispatch_opt = make_dispatcher_method(OptVirtualize, 'optimize_',
default=OptVirtualize.emit_operation)
diff --git a/pypy/jit/metainterp/optimizeopt/virtualstate.py b/pypy/jit/metainterp/optimizeopt/virtualstate.py
--- a/pypy/jit/metainterp/optimizeopt/virtualstate.py
+++ b/pypy/jit/metainterp/optimizeopt/virtualstate.py
@@ -16,7 +16,7 @@
class AbstractVirtualStateInfo(resume.AbstractVirtualInfo):
position = -1
-
+
def generalization_of(self, other, renum, bad):
raise NotImplementedError
@@ -54,7 +54,7 @@
s.debug_print(indent + " ", seen, bad)
else:
debug_print(indent + " ...")
-
+
def debug_header(self, indent):
raise NotImplementedError
@@ -77,13 +77,15 @@
bad[self] = True
bad[other] = True
return False
+
+ assert isinstance(other, AbstractVirtualStructStateInfo)
assert len(self.fielddescrs) == len(self.fieldstate)
assert len(other.fielddescrs) == len(other.fieldstate)
if len(self.fielddescrs) != len(other.fielddescrs):
bad[self] = True
bad[other] = True
return False
-
+
for i in range(len(self.fielddescrs)):
if other.fielddescrs[i] is not self.fielddescrs[i]:
bad[self] = True
@@ -112,8 +114,8 @@
def _enum(self, virtual_state):
for s in self.fieldstate:
s.enum(virtual_state)
-
-
+
+
class VirtualStateInfo(AbstractVirtualStructStateInfo):
def __init__(self, known_class, fielddescrs):
AbstractVirtualStructStateInfo.__init__(self, fielddescrs)
@@ -128,13 +130,13 @@
def debug_header(self, indent):
debug_print(indent + 'VirtualStateInfo(%d):' % self.position)
-
+
class VStructStateInfo(AbstractVirtualStructStateInfo):
def __init__(self, typedescr, fielddescrs):
AbstractVirtualStructStateInfo.__init__(self, fielddescrs)
self.typedescr = typedescr
- def _generalization_of(self, other):
+ def _generalization_of(self, other):
if not isinstance(other, VStructStateInfo):
return False
if self.typedescr is not other.typedescr:
@@ -143,7 +145,7 @@
def debug_header(self, indent):
debug_print(indent + 'VStructStateInfo(%d):' % self.position)
-
+
class VArrayStateInfo(AbstractVirtualStateInfo):
def __init__(self, arraydescr):
self.arraydescr = arraydescr
@@ -157,11 +159,7 @@
bad[other] = True
return False
renum[self.position] = other.position
- if not isinstance(other, VArrayStateInfo):
- bad[self] = True
- bad[other] = True
- return False
- if self.arraydescr is not other.arraydescr:
+ if not self._generalization_of(other):
bad[self] = True
bad[other] = True
return False
@@ -177,6 +175,10 @@
return False
return True
+ def _generalization_of(self, other):
+ return (isinstance(other, VArrayStateInfo) and
+ self.arraydescr is other.arraydescr)
+
def enum_forced_boxes(self, boxes, value, optimizer):
assert isinstance(value, virtualize.VArrayValue)
assert value.is_virtual()
@@ -192,8 +194,75 @@
def debug_header(self, indent):
debug_print(indent + 'VArrayStateInfo(%d):' % self.position)
-
-
+
+class VArrayStructStateInfo(AbstractVirtualStateInfo):
+ def __init__(self, arraydescr, fielddescrs):
+ self.arraydescr = arraydescr
+ self.fielddescrs = fielddescrs
+
+ def generalization_of(self, other, renum, bad):
+ assert self.position != -1
+ if self.position in renum:
+ if renum[self.position] == other.position:
+ return True
+ bad[self] = True
+ bad[other] = True
+ return False
+ renum[self.position] = other.position
+ if not self._generalization_of(other):
+ bad[self] = True
+ bad[other] = True
+ return False
+
+ assert isinstance(other, VArrayStructStateInfo)
+ if len(self.fielddescrs) != len(other.fielddescrs):
+ bad[self] = True
More information about the pypy-commit
mailing list