+print fruit.data_text +fruit.set_attr('data-text', 'clementine') +print fruit.attr('data-text') # alternate method for accessing attributes +``` + +### Escaping + +Pyxl automatically escapes all data and attribute values, therefore all your markup is XSS safe by default. One can explicitly avoid escaping by wrapping data in a call to `rawhtml`, but that only applies to data inside a tag. Everything in attribute values is always escaped. Note that static text inside tags (i.e. anything not inside {}'s) is considered regular HTML and is not escaped. + +```py +safe_value = "Puppies!" +unsafe_value = "" +unsafe_attr = '">' +print (

+ {unsafe_value} + {rawhtml(safe_value)} +

) +``` + +The above script will print out: + +```html +

+ <script>bad();</script> + Puppies! +

+``` + +### UI Modules + +UI Modules are especially useful for creating re-usable building blocks in your application, making it quicker to implement new features, and keeping the UI consistent. Pyxl thinks of UI modules as user defined HTML tags, and so they are used just like you would use a `

` or any other tag. + +Creating UI modules in Pyxl simply means creating a class that inherits from [`x_element`](https://github.com/dropboxe/pyxl/blob/master/pyxl/pyxl/element.py) and implements the `render()` method. Modules must be prefixed with `x_`. This is an arbitrary requirement, but is useful in separating out pyxl modules from other things. + +Arguments to a UI module are passed as attributes to the UI module tag. Attribute values for these tags need not evaluate to samething that can be cast to unicode, ONLY if the attribute value is a single python expression i.e. the only thing inside the quotes is a {} wrapped python expression. This allows one to pass in any type to a UI module. To demonstrate, a useful UI module is a user badge, which displays a user profile picture with the user's name and some arbitrary content to the right of it: + +```py +# coding: pyxl +from pyxl.element import x_element + +class x_user_badge(x_element): + __attrs__ = { + 'user': object, + } + def render(self): + return ( +

{self.user.name}

+ {self.children()} +

) +``` + +This makes the tag `` available to us which accepts `user` as an attribute which is an object that contains the user's name and profile picture. Here is an example of this new UI module being used. + +```py +# coding: pyxl +from some_module import x_user_badge + +user = User.get(some_user_id) +content =

Any arbitrary content...

+print {content} +``` + +Some things to note about UI modules. + +* Modules names must begin with `x_` and be an instance of `x_element` +* Modules must specify the attributes they accept via the `__attrs__` class variable. This is a dictionary where the key is the attribute name, and the value is the attribute type. Passing an attribute that is not listed in `__attrs__` will result in an error. The only exceptions are attributes accepted by all pyxl elements i.e. id, class, style, onclick, title and anything prefixed with "data-" or "aria-" +* Providing a `class` attribute for a UI module element will automatically append the class string to the underlying HTML element the UI module renders. This is useful when you want to style UI modules differently based on where it is being rendered. + +### Fragments + +The [`pyxl.html`](https://github.com/dropbox/pyxl/blob/master/pyxl/pyxl/html.py) module provides the `` tag, which allows one to group a set of HTML tags without a parent. Rendering the `` tag simply renders all the children, and doesn't add to the markup. + +### Conditional HTML + +Pyxl avoids support for logic within the HTML flow, except for one case where we found it especially useful: conditionally rendering HTML. That is why Pyxl provides the `` tag, which takes an attr called `cond`. Children of an `` are only rendered if `cond` evaluates to True. + +## Implementation Details + +### Parsing + +Pyxl uses support for specifying source code encodings as described in [PEP 263](http://www.python.org/dev/peps/pep-0263/) to do what it does. The functionality was originally provided so that python developers could write code in non-ascii languages (eg. chinese variable names). Pyxl creates a custom encoding called pyxl which allows it to convert XML into regular python before the file is compiled. Once the pyxl codec is registered, any file starting with `# coding: pyxl` is run through the pyxl parser before compilation. + +To register the pyxl codec, one must import the [`pyxl.codec.register`](https://github.com/dropbox/pyxl/blob/master/pyxl/pyxl/codec/register.py) module. The **Installation Process** makes it so that this always happens at python startup via the final `sudo python finish_install.py` step. What this step is doing is adding a file called `pyxl.pth` in your python site-packages directory, which imports the `pyxl.codec.register` module. Anything with a `.pth` extension in the site-packages directory is run automatically at python startup. Read more about that [here](http://docs.python.org/library/site.html). + +Some people may prefer avoiding adding pyxl.pth to their site-packages directory, in which case they should skip the final step of the installation process and explicitly import `pyxl.codec.register` in the entry point of their application. + +The pyxl encoding is a wrapper around utf-8, but every time it encounters a blob of HTML in the file, it runs it through python's [`HTMLParser`](http://docs.python.org/library/htmlparser.html) and replaces the HTML with python objects. As explained above, opening tags are converted into object instantiations for the respective tag, nested tags are passed in as arguments to the `append_children` method, and closing tags close the bracket to the `append_children` call. The code for these conversions can be seen [here](https://github.com/dropbox/pyxl/blob/master/pyxl/pyxl/codec/parser.py). + +### HTML Objects + +Though the syntactic sugar of being able to write HTML in python is pyxl's biggest usefulness, pyxl does also provide a basic framework for dealing with HTML tags as objects. This is not a full DOM implementation, but provides most of the necessary functionality. All the basic HTML tags are represented by objects defined in the [`pyxl.html`](https://github.com/dropbox/pyxl/blob/master/pyxl/pyxl/html.py) module, all of which inherit from the [`x_base`](https://github.com/dropbox/pyxl/blob/master/pyxl/pyxl/base.py) class. + +An HTML tag is rendered by calling the `to_string()` method (called automatically when tags are cast to strings), which recursively calls `to_string()` on all its children. Therefore, it should be noted that almost all the work happens only once `to_string()` is called. It is also at this stage where attribute values and data is escaped. Most of the work consists of string concatenations, and performance based on applications we've written is equivalent to templating engines like Cheetah. Note that there is probably some low hanging fruit in performance improvements that we haven't looked in to (mostly because it hasn't been a problem). + +## Editor Support + +### Emacs + +Grab pyxl-mode.el from the downloaded package under `pyxl/emacs/pyxl-mode.el` or copy it from [here](https://github.com/dropbox/pyxl/blob/master/emacs/pyxl-mode.el). To install, drop the file anywhere on your load path, and add the following to your ~/.emacs file (GNU Emacs) or ~/.xemacs/init.el file (XEmacs): + +```py +(autoload 'pyxl-mode "pyxl-mode" "Major mode for editing pyxl" t) +(setq auto-mode-alist + (cons '("\\.py\\'" . pyxl-mode) auto-mode-alist)) +``` + +### Vim + +Pyxl detection, syntax, and indent files are in the `vim` directory. The easiest way to install the vim support is via [pathogen](https://github.com/tpope/vim-pathogen); with pathogen, you can simply link or copy the directory into your bundle directory. Without pathogen, place the various files in the corresponding subdirectories of your .vim directory. + +### Pycharm + +See [pycharm-pyxl](https://github.com/christoffer/pycharm-pyxl). + +### Sublime Text + +See [sublime-pyxl](https://github.com/yyjhao/sublime-pyxl). diff --git a/lib/pyxl/emacs/pyxl-mode.el b/lib/pyxl/emacs/pyxl-mode.el new file mode 100644 --- /dev/null +++ b/lib/pyxl/emacs/pyxl-mode.el @@ -0,0 +1,132 @@ +;;; pyxl-mode.el --- major mode for editing pyxl enabled Python +;;; +;;; @author Akhil Wable +;;; +;;; To install, drop this anywhere on your load path, and add the following to +;;; your ~/.emacs file (GNU Emacs) or ~/.xemacs/init.el file (XEmacs): +;;; +;;; (autoload 'pyxl-mode "pyxl-mode" "Major mode for editing pyxl" t) +;;; (setq auto-mode-alist +;;; (cons '("\\.py\\'" . pyxl-mode) auto-mode-alist)) +;;; + +(require 'cl) +(require 'python) + +(defcustom pyxl-mode-hook nil + "list of functions to be executed on entry to pyxl-mode." + :type 'hook + :group 'python) + +(defun pyxl-context-p () + "Does the range include some HTML?" + (let ((start-rexp "([ \n\t]*<") + (end-rexp ">[ \n\t]*)")) + (let ((backward-start (save-excursion (re-search-backward start-rexp nil t))) + (backward-end (save-excursion (re-search-backward end-rexp nil t)))) + (if (and backward-start + (or (not backward-end) (< backward-end backward-start))) + backward-start + nil)))) + +(defun pyxl-back-to-indentation () + (let ((first-non-indent + (save-excursion + (back-to-indentation) + (point)))) + (if (< (point) first-non-indent) + (back-to-indentation)))) + +(defun pyx-indent-line-helper () + "Indent a line containing html." + ;; nesting regex matches either an opening tag OR a closing tag + (let ((nesting-regex "\$<[:a-zA-Z][:a-zA-Z0-9_]*\$\\|\$\$") + (indent-from (line-beginning-position)) + (depth 1)) + (save-excursion + (re-search-backward "([ \n\t]*<" nil t) + (let ((starting-indent (current-indentation))) + (while (and (< (point) indent-from) + (re-search-forward nesting-regex indent-from t)) + (if (match-string 1) (incf depth)) + (if (match-string 2) (decf depth))) + (goto-char indent-from) + (indent-line-to + (+ starting-indent + (* 4 depth) + (if (looking-at "[ \t]*\$?:\$") -4 0))))) + (pyxl-back-to-indentation))) + +(defun pyxl-indent-line () + "Modify indent for a line of html." + (interactive) + (save-excursion + (if (pyxl-context-p) + ;; If a line is inside html, use the custom indent function + (pyx-indent-line-helper) + ;; Fall back to regular python indentation for no html + (python-indent-line))) + + (pyxl-back-to-indentation)) + +(defun pyxl-indent-region (start end) + (save-excursion + (goto-char end) + (setq end (point-marker)) + (goto-char start) + (or (bolp) (forward-line 1)) + (while (< (point) end) + (or (and (bolp) (eolp)) + (pyxl-indent-line)) + (forward-line 1)) + (move-marker end nil))) + +(defcustom pyxl-default-face 'default + "Default face in pyxl-mode buffers." + :type 'face + :group 'pyxl-mode) + +(defconst pyxl-font-lock-keywords + (append + (list + ;; tags + '("\\(]*-->" (0 font-lock-comment-face)) + + ;; XML entities + '("&\\w+;" . font-lock-constant-face) + ) + python-font-lock-keywords) + "Font Lock for pyxl mode.") + +;;;###autoload +(define-derived-mode pyxl-mode python-mode "pyxl" + "Major mode for editing Python code with pyxl." + + ;; Adapted from python-mode.el + (set (make-local-variable 'font-lock-defaults) + '(pyxl-font-lock-keywords + nil + nil + nil + nil + '(font-lock-syntactic-keywords . python-font-lock-syntactic-keywords) + ;; This probably isn't worth it. + ;; (font-lock-syntactic-face-function + ;; . python-font-lock-syntactic-face-function) + )) + + (setq indent-line-function 'pyxl-indent-line) + (setq indent-region-function 'pyxl-indent-region) + (run-hooks 'pyxl-mode-hook)) + +(provide 'pyxl-mode) + +;; In python-mode.el RET is bound to newline-and-indent, which indents the next line if necessary. +;; In python.el which we're extending, this is bound to C-j instead. +;; This binds RET to newline-and-indent +(add-hook + 'python-mode-hook + '(lambda () (define-key python-mode-map "\C-m" 'newline-and-indent))) diff --git a/lib/pyxl/finish_install.py b/lib/pyxl/finish_install.py new file mode 100644 --- /dev/null +++ b/lib/pyxl/finish_install.py @@ -0,0 +1,5 @@ +import shutil +from distutils.sysconfig import get_python_lib + +python_lib = get_python_lib() +shutil.copy('pyxl.pth', python_lib) diff --git a/lib/pyxl/pyxl.pth b/lib/pyxl/pyxl.pth new file mode 100644 --- /dev/null +++ b/lib/pyxl/pyxl.pth @@ -0,0 +1,1 @@ +import pyxl.codec.register diff --git a/lib/pyxl/pyxl/__init__.py b/lib/pyxl/pyxl/__init__.py new file mode 100755 --- /dev/null +++ b/lib/pyxl/pyxl/__init__.py @@ -0,0 +1,1 @@ +#!/usr/bin/env python diff --git a/lib/pyxl/pyxl/base.py b/lib/pyxl/pyxl/base.py new file mode 100755 --- /dev/null +++ b/lib/pyxl/pyxl/base.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python + +# We want a way to generate non-colliding 'pyxl' ids for elements, so we're +# using a non-cryptographically secure random number generator. We want it to be +# insecure because these aren't being used for anything cryptographic and it's +# much faster (2x). We're also not using NumPy (which is even faster) because +# it's a difficult dependency to fulfill purely to generate random numbers. +import random +import sys + +from pyxl.utils import escape + +class PyxlException(Exception): + pass + +class x_base_metaclass(type): + def __init__(self, name, parents, attrs): + super(x_base_metaclass, self).__init__(name, parents, attrs) + x_base_parents = [parent for parent in parents if hasattr(parent, '__attrs__')] + parent_attrs = x_base_parents[0].__attrs__ if len(x_base_parents) else {} + self_attrs = self.__dict__.get('__attrs__', {}) + + # Dont allow '_' in attr names + for attr_name in self_attrs: + assert '_' not in attr_name, ( + "%s: '_' not allowed in attr names, use '-' instead" % attr_name) + + combined_attrs = dict(parent_attrs) + combined_attrs.update(self_attrs) + setattr(self, '__attrs__', combined_attrs) + setattr(self, '__tag__', name[2:]) + +class x_base(object): + + __metaclass__ = x_base_metaclass + __attrs__ = { + # HTML attributes + 'accesskey': unicode, + 'class': unicode, + 'dir': unicode, + 'id': unicode, + 'lang': unicode, + 'maxlength': unicode, + 'role': unicode, + 'style': unicode, + 'tabindex': int, + 'title': unicode, + 'xml:lang': unicode, + + # Microdata HTML attributes + 'itemtype': unicode, + 'itemscope': unicode, + 'itemprop': unicode, + 'itemid': unicode, + 'itemref': unicode, + + # JS attributes + 'onabort': unicode, + 'onblur': unicode, + 'onchange': unicode, + 'onclick': unicode, + 'ondblclick': unicode, + 'onerror': unicode, + 'onfocus': unicode, + 'onkeydown': unicode, + 'onkeypress': unicode, + 'onkeyup': unicode, + 'onload': unicode, + 'onmousedown': unicode, + 'onmouseenter': unicode, + 'onmouseleave': unicode, + 'onmousemove': unicode, + 'onmouseout': unicode, + 'onmouseover': unicode, + 'onmouseup': unicode, + 'onreset': unicode, + 'onresize': unicode, + 'onselect': unicode, + 'onsubmit': unicode, + 'onunload': unicode, + } + + def __init__(self, **kwargs): + self.__attributes__ = {} + self.__children__ = [] + + for name, value in kwargs.iteritems(): + self.set_attr(x_base._fix_attribute_name(name), value) + + def __call__(self, *children): + self.append_children(children) + return self + + def get_id(self): + eid = self.attr('id') + if not eid: + eid = 'pyxl%d' % random.randint(0, sys.maxint) + self.set_attr('id', eid) + return eid + + def children(self, selector=None, exclude=False): + if not selector: + return self.__children__ + + # filter by class + if selector[0] == '.': + select = lambda x: selector[1:] in x.get_class() + + # filter by id + elif selector[0] == '#': + select = lambda x: selector[1:] == x.get_id() + + # filter by tag name + else: + select = lambda x: x.__class__.__name__ == ('x_%s' % selector) + + if exclude: + func = lambda x: not select(x) + else: + func = select + + return filter(func, self.__children__) + + def append(self, child): + if type(child) in (list, tuple) or hasattr(child, '__iter__'): + self.__children__.extend(c for c in child if c is not None and c is not False) + elif child is not None and child is not False: + self.__children__.append(child) + + def prepend(self, child): + if child is not None and child is not False: + self.__children__.insert(0, child) + + def __getattr__(self, name): + return self.attr(name.replace('_', '-')) + + def attr(self, name, default=None): + # this check is fairly expensive (~8% of cost) + if not self.allows_attribute(name): + raise PyxlException('<%s> has no attr named "%s"' % (self.__tag__, name)) + + value = self.__attributes__.get(name) + + if value is not None: + return value + + attr_type = self.__attrs__.get(name, unicode) + if type(attr_type) == list: + if not attr_type: + raise PyxlException('Invalid attribute definition') + + if None in attr_type[1:]: + raise PyxlException('None must be the first, default value') + + return attr_type[0] + + return default + + def transfer_attributes(self, element): + for name, value in self.__attributes__.iteritems(): + if element.allows_attribute(name) and element.attr(name) is None: + element.set_attr(name, value) + + def set_attr(self, name, value): + # this check is fairly expensive (~8% of cost) + if not self.allows_attribute(name): + raise PyxlException('<%s> has no attr named "%s"' % (self.__tag__, name)) + + if value is not None: + attr_type = self.__attrs__.get(name, unicode) + + if type(attr_type) == list: + # support for enum values in pyxl attributes + values_enum = attr_type + assert values_enum, 'Invalid attribute definition' + + if value not in values_enum: + msg = '%s: %s: incorrect value "%s" for "%s". Expecting enum value %s' % ( + self.__tag__, self.__class__.__name__, value, name, values_enum) + raise PyxlException(msg) + + else: + try: + # Validate type of attr and cast to correct type if possible + value = value if isinstance(value, attr_type) else attr_type(value) + except Exception: + exc_type, exc_obj, exc_tb = sys.exc_info() + msg = '%s: %s: incorrect type for "%s". expected %s, got %s' % ( + self.__tag__, self.__class__.__name__, name, attr_type, type(value)) + exception = PyxlException(msg) + raise exception, None, exc_tb + + self.__attributes__[name] = value + + elif name in self.__attributes__: + del self.__attributes__[name] + + def get_class(self): + return self.attr('class', '') + + def add_class(self, xclass): + if not xclass: return + current_class = self.attr('class') + if current_class: current_class += ' ' + xclass + else: current_class = xclass + self.set_attr('class', current_class) + + def append_children(self, children): + for child in children: + self.append(child) + + def attributes(self): + return self.__attributes__ + + def set_attributes(self, attrs_dict): + for name, value in attrs_dict.iteritems(): + self.set_attr(name, value) + + def allows_attribute(self, name): + return (name in self.__attrs__ or name.startswith('data-') or name.startswith('aria-')) + + def to_string(self): + l = [] + self._to_list(l) + return u''.join(l) + + def _to_list(self, l): + raise NotImplementedError() + + def __str__(self): + return self.to_string() + + def __unicode__(self): + return self.to_string() + + @staticmethod + def _render_child_to_list(child, l): + if isinstance(child, x_base): child._to_list(l) + elif child is not None: l.append(escape(child)) + + @staticmethod + def _fix_attribute_name(name): + if name == 'xclass': return 'class' + if name == 'xfor': return 'for' + return name.replace('_', '-').replace('COLON', ':') diff --git a/lib/pyxl/pyxl/browser_hacks.py b/lib/pyxl/pyxl/browser_hacks.py new file mode 100644 --- /dev/null +++ b/lib/pyxl/pyxl/browser_hacks.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from pyxl.base import x_base +from pyxl.utils import escape + +class x_cond_comment(x_base): + __attrs__ = { + 'cond': unicode, + } + + def _to_list(self, l): + # allow '&', escape everything else from cond + cond = self.__attributes__.get('cond', '') + cond = '&'.join(map(escape, cond.split('&'))) + + l.extend((u'') + +class x_cond_noncomment(x_base): + ''' This is a conditional comment where browsers which don't support conditional comments + will parse the children by default. ''' + __attrs__ = { + 'cond': unicode, + } + + def _to_list(self, l): + # allow '&', escape everything else from cond + cond = self.__attributes__.get('cond', '') + cond = '&'.join(map(escape, cond.split('&'))) + + l.extend((u'')) + + for child in self.__children__: + x_base._render_child_to_list(child, l) + + l.append(u'') + diff --git a/lib/pyxl/pyxl/codec/__init__.py b/lib/pyxl/pyxl/codec/__init__.py new file mode 100755 --- /dev/null +++ b/lib/pyxl/pyxl/codec/__init__.py @@ -0,0 +1,1 @@ +#!/usr/bin/env python diff --git a/lib/pyxl/pyxl/codec/html_tokenizer.py b/lib/pyxl/pyxl/codec/html_tokenizer.py new file mode 100644 --- /dev/null +++ b/lib/pyxl/pyxl/codec/html_tokenizer.py @@ -0,0 +1,416 @@ +""" +A naive but strict HTML tokenizer. Based directly on +http://www.w3.org/TR/2011/WD-html5-20110525/tokenization.html + +In the ATTRIBUTE_VALUE and BEFORE_ATTRIBUTE_VALUE states, python tokens are accepted. +""" + +import sys +from collections import OrderedDict + +class State(object): + DATA = 1 + # unused states: charrefs, RCDATA, script, RAWTEXT, PLAINTEXT + TAG_OPEN = 7 + END_TAG_OPEN = 8 + TAG_NAME = 9 + # unused states: RCDATA, RAWTEXT, script + BEFORE_ATTRIBUTE_NAME = 34 + ATTRIBUTE_NAME = 35 + AFTER_ATTRIBUTE_NAME = 36 + BEFORE_ATTRIBUTE_VALUE = 37 + ATTRIBUTE_VALUE_DOUBLE_QUOTED = 38 + ATTRIBUTE_VALUE_SINGLE_QUOTED = 39 + ATTRIBUTE_VALUE_UNQUOTED = 40 + # unused state: CHARREF_IN_ATTRIBUTE_VALUE = 41 + AFTER_ATTRIBUTE_VALUE = 42 + SELF_CLOSING_START_TAG = 43 + # unused state: BOGUS_COMMENT_STATE = 44 + MARKUP_DECLARATION_OPEN = 45 + COMMENT_START = 46 + COMMENT_START_DASH = 47 + COMMENT = 48 + COMMENT_END_DASH = 49 + COMMENT_END = 50 + # unused state: COMMENT_END_BANG = 51 + DOCTYPE = 52 + DOCTYPE_CONTENTS = 53 # Gross oversimplification. Not to spec. + # unused states: doctypes + CDATA_SECTION = 68 + + @classmethod + def state_name(cls, state_val): + for k, v in cls.__dict__.iteritems(): + if v == state_val: + return k + assert False, "impossible state value %r!" % state_val + +class Tag(object): + def __init__(self): + self.tag_name = None + self.attrs = OrderedDict() + self.endtag = False + self.startendtag = False + +class ParseError(Exception): + pass + +class BadCharError(Exception): + def __init__(self, state, char): + super(BadCharError, self).__init__("unexpected character %r in state %r" % + (char, State.state_name(state))) + +class Unimplemented(Exception): + pass + +class HTMLTokenizer(object): + + def __init__(self): + self.state = State.DATA + + # attribute_value is a list, where each element is either a string or a list of python + # tokens. + + self.data = "" + self.tag = None + self.tag_name = None + self.attribute_name = None + self.attribute_value = None + self.markup_declaration_buffer = None + + def handle_data(self, data): + assert False, "subclass should override" + + def handle_starttag(self, tag_name, attrs): + assert False, "subclass should override" + + def handle_startendtag(self, tag_name, attrs): + assert False, "subclass should override" + + def handle_endtag(self, tag_name): + assert False, "subclass should override" + + def handle_comment(self, tag_name): + assert False, "subclass should override" + + def handle_doctype(self, data): + assert False, "subclass should override" + + def handle_cdata(self, tag_name): + assert False, "subclass should override" + + def emit_data(self): + self.handle_data(self.data) + self.data = "" + + def emit_tag(self): + if self.tag.startendtag and self.tag.endtag: + raise ParseError("both startendtag and endtag!?") + if self.tag.startendtag: + self.handle_startendtag(self.tag.tag_name, self.tag.attrs) + elif self.tag.endtag: + self.handle_endtag(self.tag.tag_name) + else: + self.handle_starttag(self.tag.tag_name, self.tag.attrs) + + def emit_comment(self): + self.handle_comment(self.data) + self.data = "" + + def emit_doctype(self): + self.handle_doctype(self.data) + self.data = "" + + def emit_cdata(self): + self.handle_cdata(self.data) + self.data = "" + + def got_attribute(self): + if self.attribute_name in self.tag.attrs: + raise ParseError("repeat attribute name %r" % self.attribute_name) + self.tag.attrs[self.attribute_name] = self.attribute_value + self.attribute_name = None + self.attribute_value = None + + def add_data_char(self, build, c): + """ For adding a new character to e.g. an attribute value """ + if len(build) and type(build[-1]) == str: + build[-1] += c + else: + build.append(c) + + def feed(self, c): + if self.state == State.DATA: + if c == '<': + self.emit_data() + self.state = State.TAG_OPEN + # Pass through; it's the browser's problem to understand these. + #elif c == '&': + # raise Unimplemented + else: + self.data += c + + elif self.state == State.TAG_OPEN: + self.tag = Tag() + if c == '!': + self.markup_declaration_buffer = "" + self.state = State.MARKUP_DECLARATION_OPEN + elif c == '/': + self.state = State.END_TAG_OPEN + elif c.isalpha(): + self.tag.tag_name = c + self.state = State.TAG_NAME + else: + raise BadCharError(self.state, c) + + elif self.state == State.END_TAG_OPEN: + self.tag.endtag = True + if c.isalpha(): + self.tag.tag_name = c + self.state = State.TAG_NAME + else: + raise BadCharError(self.state, c) + + elif self.state == State.TAG_NAME: + if c in '\t\n\f ': + self.state = State.BEFORE_ATTRIBUTE_NAME + elif c == '/': + self.state = State.SELF_CLOSING_START_TAG + elif c == '>': + self.emit_tag() + self.state = State.DATA + else: + self.tag.tag_name += c + + elif self.state == State.BEFORE_ATTRIBUTE_NAME: + if c in '\t\n\f ': + pass + elif c == '/': + self.state = State.SELF_CLOSING_START_TAG + elif c == '>': + self.emit_tag() + self.state = State.DATA + elif c in "\"'<=": + raise BadCharError(self.state, c) + else: + self.attribute_name = c.lower() + self.state = State.ATTRIBUTE_NAME + + elif self.state == State.ATTRIBUTE_NAME: + if c in '\t\n\f ': + self.state = State.AFTER_ATTRIBUTE_NAME + elif c == '/': + self.got_attribute() + self.state = State.SELF_CLOSING_START_TAG + elif c == '=': + self.state = State.BEFORE_ATTRIBUTE_VALUE + elif c == '>': + self.emit_tag() + self.state = State.DATA + elif c in "\"'<": + raise BadCharError(self.state, c) + else: + self.attribute_name += c.lower() + + elif self.state == State.AFTER_ATTRIBUTE_NAME: + if c in '\t\n\f ': + pass + elif c == '/': + self.got_attribute() + self.state = State.SELF_CLOSING_START_TAG + elif c == '=': + self.state = State.BEFORE_ATTRIBUTE_VALUE + elif c == '>': + self.got_attribute() + self.emit_tag() + self.state = State.DATA + elif c in "\"'<": + raise BadCharError(self.state, c) + + elif self.state == State.BEFORE_ATTRIBUTE_VALUE: + if c in '\t\n\f ': + pass + elif c == '"': + self.attribute_value = [] + self.state = State.ATTRIBUTE_VALUE_DOUBLE_QUOTED + elif c == '&': + self.attribute_value = [] + self.state = State.ATTRIBUTE_VALUE_UNQUOTED + self.feed(c) # rehandle c + elif c == "'": + self.attribute_value = [] + self.state = State.ATTRIBUTE_VALUE_SINGLE_QUOTED + elif c in '><=`': + raise BadCharError(self.state, c) + else: + self.attribute_value = [c] + self.state = State.ATTRIBUTE_VALUE_UNQUOTED + + elif self.state == State.ATTRIBUTE_VALUE_DOUBLE_QUOTED: + if c == '"': + self.state = State.AFTER_ATTRIBUTE_VALUE + # Pass through; it's the browser's problem to understand these. + #elif c == '&': + # raise Unimplemented + else: + self.add_data_char(self.attribute_value, c) + + elif self.state == State.ATTRIBUTE_VALUE_SINGLE_QUOTED: + if c == "'": + self.state = State.AFTER_ATTRIBUTE_VALUE + # Pass through; it's the browser's problem to understand these. + #elif c == '&': + # raise Unimplemented + else: + self.add_data_char(self.attribute_value, c) + + elif self.state == State.ATTRIBUTE_VALUE_UNQUOTED: + if c in '\t\n\f ': + self.got_attribute() + self.state = State.BEFORE_ATTRIBUTE_NAME + elif c == '>': + self.got_attribute() + self.emit_tag() + self.state = State.DATA + elif c in "\"'<=`": + raise BadCharError(self.state, c) + # Pass through; it's the browser's problem to understand these. + #elif c == '&': + # raise Unimplemented + else: + self.add_data_char(self.attribute_value, c) + + elif self.state == State.AFTER_ATTRIBUTE_VALUE: + self.got_attribute() + if c in '\t\n\f ': + self.state = State.BEFORE_ATTRIBUTE_NAME + elif c == '/': + self.state = State.SELF_CLOSING_START_TAG + elif c == '>': + self.emit_tag() + self.state = State.DATA + else: + raise BadCharError(self.state, c) + + elif self.state == State.SELF_CLOSING_START_TAG: + self.tag.startendtag = True + if c == '>': + self.emit_tag() + self.state = State.DATA + else: + raise BadCharError(self.state, c) + + elif self.state == State.MARKUP_DECLARATION_OPEN: + self.markup_declaration_buffer += c + if self.markup_declaration_buffer == "--": + self.data = "" + self.state = State.COMMENT_START + elif self.markup_declaration_buffer.lower() == "DOCTYPE".lower(): + self.state = State.DOCTYPE + elif self.markup_declaration_buffer == "[CDATA[": + self.data = "" + self.cdata_buffer = "" + self.state = State.CDATA_SECTION + elif not ("--".startswith(self.markup_declaration_buffer) or + "DOCTYPE".lower().startswith(self.markup_declaration_buffer.lower()) or + "[CDATA[".startswith(self.markup_declaration_buffer)): + raise BadCharError(self.state, c) + + elif self.state == State.COMMENT_START: + if c == "-": + self.state = State.COMMENT_START_DASH + elif c == ">": + raise BadCharError(self.state, c) + else: + self.data += c + self.state = State.COMMENT + + elif self.state == State.COMMENT_START_DASH: + if c == "-": + self.state = State.COMMENT_END + elif c == ">": + raise BadCharError(self.state, c) + else: + self.data += "-" + c + self.state = State.COMMENT + + elif self.state == State.COMMENT: + if c == "-": + self.state = State.COMMENT_END_DASH + else: + self.data += c + + elif self.state == State.COMMENT_END_DASH: + if c == "-": + self.state = State.COMMENT_END + else: + self.data += "-" + c + self.state = State.COMMENT + + elif self.state == State.COMMENT_END: + if c == ">": + self.emit_comment() + self.state = State.DATA + else: + raise BadCharError(self.state, c) + + elif self.state == State.DOCTYPE: + if c in "\t\n\f ": + self.data = "" + self.state = State.DOCTYPE_CONTENTS + else: + raise BadCharError(self.state, c) + + elif self.state == State.DOCTYPE_CONTENTS: + if c == ">": + self.emit_doctype() + self.state = State.DATA + else: + self.data += c + + elif self.state == State.CDATA_SECTION: + self.cdata_buffer += c + if self.cdata_buffer == "]]>": + self.emit_cdata() + self.state = State.DATA + else: + while self.cdata_buffer and not "]]>".startswith(self.cdata_buffer): + self.data += self.cdata_buffer[0] + self.cdata_buffer = self.cdata_buffer[1:] + + else: + assert False, "bad state! %r" % self.state + + def feed_python(self, tokens): + if self.state == State.BEFORE_ATTRIBUTE_VALUE: + self.attribute_value = [tokens] + self.state = State.ATTRIBUTE_VALUE_UNQUOTED + elif self.state in [State.ATTRIBUTE_VALUE_DOUBLE_QUOTED, + State.ATTRIBUTE_VALUE_SINGLE_QUOTED, + State.ATTRIBUTE_VALUE_UNQUOTED]: + self.attribute_value.append(tokens) + else: + raise ParseError("python not allow in state %r" % State.state_name(self.state)) + +class HTMLTokenDumper(HTMLTokenizer): + def handle_data(self, data): + print "DATA %r" % data + + def handle_starttag(self, tag_name, attrs): + print "STARTTAG %r %r" % (tag_name, attrs) + + def handle_startendtag(self, tag_name, attrs): + print "STARTENDTAG %r %r" % (tag_name, attrs) + + def handle_endtag(self, tag_name): + print "ENDTAG %r" % tag_name + +def main(filename): + dumper = HTMLTokenDumper() + with open(filename) as f: + for line in f: + for c in line: + dumper.feed(c) + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/lib/pyxl/pyxl/codec/parser.py b/lib/pyxl/pyxl/codec/parser.py new file mode 100755 --- /dev/null +++ b/lib/pyxl/pyxl/codec/parser.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python + +import tokenize +from pyxl import html +from html_tokenizer import ( + HTMLTokenizer, + ParseError as TokenizerParseError, + State, +) +from pytokenize import Untokenizer + +class ParseError(Exception): + def __init__(self, message, pos=None): + if pos is not None: + super(ParseError, self).__init__("%s at line %d char %d" % ((message,) + pos)) + else: + super(ParseError, self).__init__(message) + +class PyxlParser(HTMLTokenizer): + def __init__(self, row, col): + super(PyxlParser, self).__init__() + self.start = self.end = (row, col) + self.output = [] + self.open_tags = [] + self.remainder = None + self.next_thing_is_python = False + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + + def feed(self, token): + ttype, tvalue, tstart, tend, tline = token + + assert tstart[0] >= self.end[0], "row went backwards" + if tstart[0] > self.end[0]: + self.output.append("\n" * (tstart[0] - self.end[0])) + + # interpret jumps on the same line as a single space + elif tstart[1] > self.end[1]: + super(PyxlParser, self).feed(" ") + + self.end = tstart + + if ttype != tokenize.INDENT: + while tvalue and not self.done(): + c, tvalue = tvalue[0], tvalue[1:] + if c == "\n": + self.end = (self.end[0]+1, 0) + else: + self.end = (self.end[0], self.end[1]+1) + try: + super(PyxlParser, self).feed(c) + except TokenizerParseError: + raise ParseError("HTML Parsing error", self.end) + if self.done(): + self.remainder = (ttype, tvalue, self.end, tend, tline) + else: + self.end = tend + + def feed_python(self, tokens): + ttype, tvalue, tstart, tend, tline = tokens[0] + assert tstart[0] >= self.end[0], "row went backwards" + if tstart[0] > self.end[0]: + self.output.append("\n" * (tstart[0] - self.end[0])) + ttype, tvalue, tstart, tend, tline = tokens[-1] + self.end = tend + + if self.state in [State.DATA, State.CDATA_SECTION]: + self.next_thing_is_python = True + self.emit_data() + self.output.append("%s, " % Untokenizer().untokenize(tokens)) + self.next_thing_is_python = False + self.last_thing_was_python = True + elif self.state in [State.BEFORE_ATTRIBUTE_VALUE, + State.ATTRIBUTE_VALUE_DOUBLE_QUOTED, + State.ATTRIBUTE_VALUE_SINGLE_QUOTED, + State.ATTRIBUTE_VALUE_UNQUOTED]: + super(PyxlParser, self).feed_python(tokens) + + def feed_position_only(self, token): + """update with any whitespace we might have missed, and advance position to after the + token""" + ttype, tvalue, tstart, tend, tline = token + self.feed((ttype, '', tstart, tstart, tline)) + self.end = tend + + def python_comment_allowed(self): + """Returns true if we're in a state where a # starts a comment. + + + # comment in data + Link text + + """ + return self.state in (State.DATA, State.TAG_NAME, + State.BEFORE_ATTRIBUTE_NAME, State.AFTER_ATTRIBUTE_NAME, + State.BEFORE_ATTRIBUTE_VALUE, State.AFTER_ATTRIBUTE_VALUE, + State.COMMENT, State.DOCTYPE_CONTENTS, State.CDATA_SECTION) + + def python_mode_allowed(self): + """Returns true if we're in a state where a { starts python mode. + + + """ + return self.state not in (State.COMMENT,) + + def feed_comment(self, token): + ttype, tvalue, tstart, tend, tline = token + self.feed((ttype, '', tstart, tstart, tline)) + self.output.append(tvalue) + self.end = tend + + def get_remainder(self): + return self.remainder + + def done(self): + return len(self.open_tags) == 0 and self.state == State.DATA and self.output + + def get_token(self): + return (tokenize.STRING, ''.join(self.output), self.start, self.end, '') + + @staticmethod + def safe_attr_name(name): + if name == "class": + return "xclass" + if name == "for": + return "xfor" + return name.replace('-', '_').replace(':', 'COLON') + + def _handle_attr_value(self, attr_value): + def format_parts(): + prev_was_python = False + for i, part in enumerate(attr_value): + if type(part) == list: + yield part + prev_was_python = True + else: + next_is_python = bool(i+1 < len(attr_value) and type(attr_value[i+1]) == list) + part = self._normalize_data_whitespace(part, prev_was_python, next_is_python) + if part: + yield part + prev_was_python = False + + attr_value = list(format_parts()) + if len(attr_value) == 1: + part = attr_value[0] + if type(part) == list: + self.output.append(Untokenizer().untokenize(part)) + else: + self.output.append(repr(part)) + else: + self.output.append('u"".join((') + for part in attr_value: + if type(part) == list: + self.output.append('unicode(') + self.output.append(Untokenizer().untokenize(part)) + self.output.append(')') + else: + self.output.append(repr(part)) + self.output.append(', ') + self.output.append('))') + + @staticmethod + def _normalize_data_whitespace(data, prev_was_py, next_is_py): + if not data: + return '' + if '\n' in data and not data.strip(): + if prev_was_py and next_is_py: + return ' ' + else: + return '' + if prev_was_py and data.startswith('\n'): + data = " " + data.lstrip('\n') + if next_is_py and data.endswith('\n'): + data = data.rstrip('\n') + " " + data = data.strip('\n') + data = data.replace('\r', ' ') + data = data.replace('\n', ' ') + return data + + def handle_starttag(self, tag, attrs, call=True): + self.open_tags.append({'tag':tag, 'row': self.end[0]}) + if tag == 'if': + if len(attrs) != 1: + raise ParseError("if tag only takes one attr called 'cond'", self.end) + if 'cond' not in attrs: + raise ParseError("if tag must contain the 'cond' attr", self.end) + + self.output.append('html._push_condition(bool(') + self._handle_attr_value(attrs['cond']) + self.output.append(')) and html.x_frag()(') + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + return + elif tag == 'else': + if len(attrs) != 0: + raise ParseError("else tag takes no attrs", self.end) + if not self.last_thing_was_close_if_tag: + raise ParseError(" tag must come right after ", self.end) + + self.output.append('(not html._last_if_condition) and html.x_frag()(') + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + return + + module, dot, identifier = tag.rpartition('.') + identifier = 'x_%s' % identifier + x_tag = module + dot + identifier + + if hasattr(html, x_tag): + self.output.append('html.') + self.output.append('%s(' % x_tag) + + first_attr = True + for attr_name, attr_value in attrs.iteritems(): + if first_attr: first_attr = False + else: self.output.append(', ') + + self.output.append(self.safe_attr_name(attr_name)) + self.output.append('=') + self._handle_attr_value(attr_value) + + self.output.append(')') + if call: + # start call to __call__ + self.output.append('(') + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + + def handle_endtag(self, tag_name, call=True): + if call: + # finish call to __call__ + self.output.append(")") + + assert self.open_tags, "got but tag stack empty; parsing should be over!" % tag_name + + open_tag = self.open_tags.pop() + if open_tag['tag'] != tag_name: + raise ParseError("<%s> on line %d closed by on line %d" % + (open_tag['tag'], open_tag['row'], tag_name, self.end[0])) + + if open_tag['tag'] == 'if': + self.output.append(',html._leave_if()') + self.last_thing_was_close_if_tag = True + else: + self.last_thing_was_close_if_tag = False + + if len(self.open_tags): + self.output.append(",") + self.last_thing_was_python = False + + def handle_startendtag(self, tag_name, attrs): + self.handle_starttag(tag_name, attrs, call=False) + self.handle_endtag(tag_name, call=False) + + def handle_data(self, data): + data = self._normalize_data_whitespace( + data, self.last_thing_was_python, self.next_thing_is_python) + if not data: + return + + # XXX XXX mimics old pyxl, but this is gross and likely wrong. I'm pretty sure we actually + # want %r instead of this crazy quote substitution and u"%s". + data = data.replace('"', '\\"') + self.output.append('html.rawhtml(u"%s"), ' % data) + + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + + def handle_comment(self, data): + self.handle_startendtag("html_comment", {"comment": [data.strip()]}) + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + + def handle_doctype(self, data): + self.handle_startendtag("html_decl", {"decl": ['DOCTYPE ' + data]}) + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False + + def handle_cdata(self, data): + self.handle_startendtag("html_marked_decl", {"decl": ['CDATA[' + data]}) + self.last_thing_was_python = False + self.last_thing_was_close_if_tag = False diff --git a/lib/pyxl/pyxl/codec/pytokenize.py b/lib/pyxl/pyxl/codec/pytokenize.py new file mode 100644 --- /dev/null +++ b/lib/pyxl/pyxl/codec/pytokenize.py @@ -0,0 +1,468 @@ +"""Tokenization help for Python programs. + +generate_tokens(readline) is a generator that breaks a stream of +text into Python tokens. It accepts a readline-like method which is called +repeatedly to get the next line of input (or "" for EOF). It generates +5-tuples with these members: + + the token type (see token.py) + the token (a string) + the starting (row, column) indices of the token (a 2-tuple of ints) + the ending (row, column) indices of the token (a 2-tuple of ints) + the original line (string) + +It is designed to match the working of the Python tokenizer exactly, except +that it produces COMMENT tokens for comments and gives type OP for all +operators + +Older entry points + tokenize_loop(readline, tokeneater) + tokenize(readline, tokeneater=printtoken) +are the same, except instead of generating tokens, tokeneater is a callback +function to which the 5 fields described above are passed as 5 arguments, +each time a new token is found. + + +This file was taken from the python 2.7.4 library and modified for use by +the Pyxl decoder. Changes made: + - When it encounters an unexpected EOF, the tokenizer does not raise an + exception, and instead yields an errortoken if appropriate. + - When it encounters an unexpected dedent, the tokenizer does not + raise an exception. + - The Untokenizer class was heavily modified. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013 Python Software Foundation; All Rights Reserved" are retained +in Python alone or in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. +""" + +__author__ = 'Ka-Ping Yee ' +__credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, ' + 'Skip Montanaro, Raymond Hettinger') + +import string, re +from token import * + +import token +__all__ = [x for x in dir(token) if not x.startswith("_")] +__all__ += ["COMMENT", "tokenize", "generate_tokens", "NL", "untokenize"] +del x +del token + +COMMENT = N_TOKENS +tok_name[COMMENT] = 'COMMENT' +NL = N_TOKENS + 1 +tok_name[NL] = 'NL' +N_TOKENS += 2 + +def group(*choices): return '(' + '|'.join(choices) + ')' +def any(*choices): return group(*choices) + '*' +def maybe(*choices): return group(*choices) + '?' + +Whitespace = r'[ \f\t]*' +Comment = r'#[^\r\n]*' +Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment) +Name = r'[a-zA-Z_]\w*' + +Hexnumber = r'0[xX][\da-fA-F]+[lL]?' +Octnumber = r'(0[oO][0-7]+)|(0[0-7]*)[lL]?' +Binnumber = r'0[bB][01]+[lL]?' +Decnumber = r'[1-9]\d*[lL]?' +Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber) +Exponent = r'[eE][-+]?\d+' +Pointfloat = group(r'\d+\.\d*', r'\.\d+') + maybe(Exponent) +Expfloat = r'\d+' + Exponent +Floatnumber = group(Pointfloat, Expfloat) +Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]') +Number = group(Imagnumber, Floatnumber, Intnumber) + +# Tail end of ' string. +Single = r"[^'\\]*(?:\\.[^'\\]*)*'" +# Tail end of " string. +Double = r'[^"\\]*(?:\\.[^"\\]*)*"' +# Tail end of ''' string. +Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''" +# Tail end of """ string. +Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""' +Triple = group("[uUbB]?[rR]?'''", '[uUbB]?[rR]?"""') +# Single-line ' or " string. +String = group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'", + r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"') + +# Because of leftmost-then-longest match semantics, be sure to put the +# longest operators first (e.g., if = came before ==, == would get +# recognized as two instances of =). +Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=", + r"//=?", + r"[+\-*/%&|^=<>]=?", + r"~") + +Bracket = '[][(){}]' +Special = group(r'\r?\n', r'[:;.,`@]') +Funny = group(Operator, Bracket, Special) + +PlainToken = group(Number, Funny, String, Name) +Token = Ignore + PlainToken + +# First (or only) line of ' or " string. +ContStr = group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" + + group("'", r'\\\r?\n'), + r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' + + group('"', r'\\\r?\n')) +PseudoExtras = group(r'\\\r?\n|\Z', Comment, Triple) +PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) + +tokenprog, pseudoprog, single3prog, double3prog = map( + re.compile, (Token, PseudoToken, Single3, Double3)) +endprogs = {"'": re.compile(Single), '"': re.compile(Double), + "'''": single3prog, '"""': double3prog, + "r'''": single3prog, 'r"""': double3prog, + "u'''": single3prog, 'u"""': double3prog, + "ur'''": single3prog, 'ur"""': double3prog, + "R'''": single3prog, 'R"""': double3prog, + "U'''": single3prog, 'U"""': double3prog, + "uR'''": single3prog, 'uR"""': double3prog, + "Ur'''": single3prog, 'Ur"""': double3prog, + "UR'''": single3prog, 'UR"""': double3prog, + "b'''": single3prog, 'b"""': double3prog, + "br'''": single3prog, 'br"""': double3prog, + "B'''": single3prog, 'B"""': double3prog, + "bR'''": single3prog, 'bR"""': double3prog, + "Br'''": single3prog, 'Br"""': double3prog, + "BR'''": single3prog, 'BR"""': double3prog, + 'r': None, 'R': None, 'u': None, 'U': None, + 'b': None, 'B': None} + +triple_quoted = {} +for t in ("'''", '"""', + "r'''", 'r"""', "R'''", 'R"""', + "u'''", 'u"""', "U'''", 'U"""', + "ur'''", 'ur"""', "Ur'''", 'Ur"""', + "uR'''", 'uR"""', "UR'''", 'UR"""', + "b'''", 'b"""', "B'''", 'B"""', + "br'''", 'br"""', "Br'''", 'Br"""', + "bR'''", 'bR"""', "BR'''", 'BR"""'): + triple_quoted[t] = t +single_quoted = {} +for t in ("'", '"', + "r'", 'r"', "R'", 'R"', + "u'", 'u"', "U'", 'U"', + "ur'", 'ur"', "Ur'", 'Ur"', + "uR'", 'uR"', "UR'", 'UR"', + "b'", 'b"', "B'", 'B"', + "br'", 'br"', "Br'", 'Br"', + "bR'", 'bR"', "BR'", 'BR"' ): + single_quoted[t] = t + +tabsize = 8 + +class TokenError(Exception): pass + +class StopTokenizing(Exception): pass + +def printtoken(type, token, srow_scol, erow_ecol, line): # for testing + srow, scol = srow_scol + erow, ecol = erow_ecol + print "%d,%d-%d,%d:\t%s\t%s" % \ + (srow, scol, erow, ecol, tok_name[type], repr(token)) + +def tokenize(readline, tokeneater=printtoken): + """ + The tokenize() function accepts two parameters: one representing the + input stream, and one providing an output mechanism for tokenize(). + + The first parameter, readline, must be a callable object which provides + the same interface as the readline() method of built-in file objects. + Each call to the function should return one line of input as a string. + + The second parameter, tokeneater, must also be a callable object. It is + called once for each token, with five arguments, corresponding to the + tuples generated by generate_tokens(). + """ + try: + tokenize_loop(readline, tokeneater) + except StopTokenizing: + pass + +# backwards compatible interface +def tokenize_loop(readline, tokeneater): + for token_info in generate_tokens(readline): + tokeneater(*token_info) + +class Untokenizer: + + # PYXL MODIFICATION: This entire class. + + def __init__(self, row=None, col=None): + self.tokens = [] + self.prev_row = row + self.prev_col = col + + def add_whitespace(self, start): + row, col = start + assert row >= self.prev_row, "row (%r) should be >= prev_row (%r)" % (row, self.prev_row) + row_offset = row - self.prev_row + if row_offset: + self.tokens.append("\n" * row_offset) + col_offset = col - self.prev_col + if col_offset: + self.tokens.append(" " * col_offset) + + def feed(self, t): + assert len(t) == 5 + tok_type, token, start, end, line = t + if (self.prev_row is None): + self.prev_row, self.prev_col = start + self.add_whitespace(start) + self.tokens.append(token) + self.prev_row, self.prev_col = end + if tok_type in (NEWLINE, NL): + self.prev_row += 1 + self.prev_col = 0 + + def finish(self): + return "".join(self.tokens) + + def untokenize(self, iterable): + for t in iterable: + self.feed(t) + return self.finish() + +def untokenize(iterable): + """Transform tokens back into Python source code. + + Each element returned by the iterable must be a token sequence + with at least two elements, a token number and token value. If + only two tokens are passed, the resulting output is poor. + + Round-trip invariant for full input: + Untokenized source will match input source exactly + + Round-trip invariant for limited intput: + # Output text will tokenize the back to the input + t1 = [tok[:2] for tok in generate_tokens(f.readline)] + newcode = untokenize(t1) + readline = iter(newcode.splitlines(1)).next + t2 = [tok[:2] for tok in generate_tokens(readline)] + assert t1 == t2 + """ + ut = Untokenizer() + return ut.untokenize(iterable) + +def generate_tokens(readline): + """ + The generate_tokens() generator requires one argment, readline, which + must be a callable object which provides the same interface as the + readline() method of built-in file objects. Each call to the function + should return one line of input as a string. Alternately, readline + can be a callable function terminating with StopIteration: + readline = open(myfile).next # Example of alternate readline + + The generator produces 5-tuples with these members: the token type; the + token string; a 2-tuple (srow, scol) of ints specifying the row and From pypy.commits at gmail.com Wed Jan 20 16:35:30 2016 From: pypy.commits at gmail.com (fijal) Date: Wed, 20 Jan 2016 13:35:30 -0800 (PST) Subject: [pypy-commit] benchmarks default: add some other pyston benchmarks Message-ID: <569ffda2.520e1c0a.1e25f.56dd@mx.google.com> Author: fijal Branch: Changeset: r346:947a8b49e3cd Date: 2016-01-20 22:31 +0100 http://bitbucket.org/pypy/benchmarks/changeset/947a8b49e3cd/ Log: add some other pyston benchmarks diff too long, truncating to 2000 out of 513606 lines diff --git a/benchmarks.py b/benchmarks.py --- a/benchmarks.py +++ b/benchmarks.py @@ -66,6 +66,11 @@ 'bm_dulwich_log': {'bm_env': {'PYTHONPATH': relative('lib/dulwich-0.9.1')}}, 'bm_chameleon': {'bm_env': {'PYTHONPATH': relative('lib/chameleon/src')}, 'iteration_scaling': 3}, + 'nqueens': {'iteration_scaling': .1}, + 'sqlalchemy_declarative': {'bm_env': {'PYTHONPATH': relative('lib/sqlalchemy/lib')}, + 'iteration_scaling': 3}, + 'sqlalchemy_imperative': {'bm_env': {'PYTHONPATH': relative('lib/sqlalchemy/lib')}, + 'iteration_scaling': 10}, } for name in ['expand', 'integrate', 'sum', 'str']: @@ -84,7 +89,8 @@ 'raytrace-simple', 'crypto_pyaes', 'bm_mako', 'bm_chameleon', 'json_bench', 'pidigits', 'hexiom2', 'eparse', 'deltablue', 'bm_dulwich_log', 'bm_krakatau', 'bm_mdp', 'pypy_interp', - 'sqlitesynth', 'pyxl_bench']: + 'sqlitesynth', 'pyxl_bench', 'nqueens', 'sqlalchemy_declarative', + 'sqlalchemy_imperative']: _register_new_bm(name, name, globals(), **opts.get(name, {})) for name in ['names', 'iteration', 'tcp', 'pb', ]:#'web']:#, 'accepts']: diff --git a/lib/sqlalchemy/AUTHORS b/lib/sqlalchemy/AUTHORS new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/AUTHORS @@ -0,0 +1,18 @@ +SQLAlchemy was created by Michael Bayer. + +Major contributing authors include: + +- Michael Bayer +- Jason Kirtland +- Gaetan de Menten +- Diana Clarke +- Michael Trier +- Philip Jenvey +- Ants Aasma +- Paul Johnston +- Jonathan Ellis + +For a larger list of SQLAlchemy contributors over time, see: + +http://www.sqlalchemy.org/trac/wiki/Contributors + diff --git a/lib/sqlalchemy/CHANGES b/lib/sqlalchemy/CHANGES new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/CHANGES @@ -0,0 +1,16 @@ +===== +MOVED +===== + +Please see: + + /doc/changelog/index.html + +or + + http://www.sqlalchemy.org/docs/latest/changelog/ + +for an index of all changelogs. + + + diff --git a/lib/sqlalchemy/LICENSE b/lib/sqlalchemy/LICENSE new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/LICENSE @@ -0,0 +1,20 @@ +This is the MIT license: http://www.opensource.org/licenses/mit-license.php + +Copyright (c) 2005-2015 the SQLAlchemy authors and contributors . +SQLAlchemy is a trademark of Michael Bayer. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons +to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE +FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/lib/sqlalchemy/MANIFEST.in b/lib/sqlalchemy/MANIFEST.in new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/MANIFEST.in @@ -0,0 +1,13 @@ +# any kind of "*" pulls in __init__.pyc files, +# so all extensions are explicit. + +recursive-include doc *.html *.css *.txt *.js *.jpg *.png *.py Makefile *.rst *.mako *.sty +recursive-include examples *.py *.xml +recursive-include test *.py *.dat + +# include the c extensions, which otherwise +# don't come in if --with-cextensions isn't specified. +recursive-include lib *.c *.txt + +include README* AUTHORS LICENSE distribute_setup.py sa2to3.py ez_setup.py sqla_nose.py CHANGES* tox.ini +prune doc/build/output diff --git a/lib/sqlalchemy/PKG-INFO b/lib/sqlalchemy/PKG-INFO new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/PKG-INFO @@ -0,0 +1,155 @@ +Metadata-Version: 1.1 +Name: SQLAlchemy +Version: 1.0.11 +Summary: Database Abstraction Library +Home-page: http://www.sqlalchemy.org +Author: Mike Bayer +Author-email: mike_mp at zzzcomputing.com +License: MIT License +Description: SQLAlchemy + ========== + + The Python SQL Toolkit and Object Relational Mapper + + Introduction + ------------- + + SQLAlchemy is the Python SQL toolkit and Object Relational Mapper + that gives application developers the full power and + flexibility of SQL. SQLAlchemy provides a full suite + of well known enterprise-level persistence patterns, + designed for efficient and high-performing database + access, adapted into a simple and Pythonic domain + language. + + Major SQLAlchemy features include: + + * An industrial strength ORM, built + from the core on the identity map, unit of work, + and data mapper patterns. These patterns + allow transparent persistence of objects + using a declarative configuration system. + Domain models + can be constructed and manipulated naturally, + and changes are synchronized with the + current transaction automatically. + * A relationally-oriented query system, exposing + the full range of SQL's capabilities + explicitly, including joins, subqueries, + correlation, and most everything else, + in terms of the object model. + Writing queries with the ORM uses the same + techniques of relational composition you use + when writing SQL. While you can drop into + literal SQL at any time, it's virtually never + needed. + * A comprehensive and flexible system + of eager loading for related collections and objects. + Collections are cached within a session, + and can be loaded on individual access, all + at once using joins, or by query per collection + across the full result set. + * A Core SQL construction system and DBAPI + interaction layer. The SQLAlchemy Core is + separate from the ORM and is a full database + abstraction layer in its own right, and includes + an extensible Python-based SQL expression + language, schema metadata, connection pooling, + type coercion, and custom types. + * All primary and foreign key constraints are + assumed to be composite and natural. Surrogate + integer primary keys are of course still the + norm, but SQLAlchemy never assumes or hardcodes + to this model. + * Database introspection and generation. Database + schemas can be "reflected" in one step into + Python structures representing database metadata; + those same structures can then generate + CREATE statements right back out - all within + the Core, independent of the ORM. + + SQLAlchemy's philosophy: + + * SQL databases behave less and less like object + collections the more size and performance start to + matter; object collections behave less and less like + tables and rows the more abstraction starts to matter. + SQLAlchemy aims to accommodate both of these + principles. + * An ORM doesn't need to hide the "R". A relational + database provides rich, set-based functionality + that should be fully exposed. SQLAlchemy's + ORM provides an open-ended set of patterns + that allow a developer to construct a custom + mediation layer between a domain model and + a relational schema, turning the so-called + "object relational impedance" issue into + a distant memory. + * The developer, in all cases, makes all decisions + regarding the design, structure, and naming conventions + of both the object model as well as the relational + schema. SQLAlchemy only provides the means + to automate the execution of these decisions. + * With SQLAlchemy, there's no such thing as + "the ORM generated a bad query" - you + retain full control over the structure of + queries, including how joins are organized, + how subqueries and correlation is used, what + columns are requested. Everything SQLAlchemy + does is ultimately the result of a developer- + initiated decision. + * Don't use an ORM if the problem doesn't need one. + SQLAlchemy consists of a Core and separate ORM + component. The Core offers a full SQL expression + language that allows Pythonic construction + of SQL constructs that render directly to SQL + strings for a target database, returning + result sets that are essentially enhanced DBAPI + cursors. + * Transactions should be the norm. With SQLAlchemy's + ORM, nothing goes to permanent storage until + commit() is called. SQLAlchemy encourages applications + to create a consistent means of delineating + the start and end of a series of operations. + * Never render a literal value in a SQL statement. + Bound parameters are used to the greatest degree + possible, allowing query optimizers to cache + query plans effectively and making SQL injection + attacks a non-issue. + + Documentation + ------------- + + Latest documentation is at: + + http://www.sqlalchemy.org/docs/ + + Installation / Requirements + --------------------------- + + Full documentation for installation is at + `Installation `_. + + Getting Help / Development / Bug reporting + ------------------------------------------ + + Please refer to the `SQLAlchemy Community Guide `_. + + License + ------- + + SQLAlchemy is distributed under the `MIT license + `_. + + +Platform: UNKNOWN +Classifier: Development Status :: 5 - Production/Stable +Classifier: Intended Audience :: Developers +Classifier: License :: OSI Approved :: MIT License +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: Implementation :: CPython +Classifier: Programming Language :: Python :: Implementation :: Jython +Classifier: Programming Language :: Python :: Implementation :: PyPy +Classifier: Topic :: Database :: Front-Ends +Classifier: Operating System :: OS Independent diff --git a/lib/sqlalchemy/README.dialects.rst b/lib/sqlalchemy/README.dialects.rst new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/README.dialects.rst @@ -0,0 +1,262 @@ +======================== +Developing new Dialects +======================== + +.. note:: + + When studying this file, it's probably a good idea to also + familiarize with the README.unittests.rst file, which discusses + SQLAlchemy's usage and extension of the Nose test runner. + +While SQLAlchemy includes many dialects within the core distribution, the +trend for new dialects should be that they are published as external +projects. SQLAlchemy has since version 0.5 featured a "plugin" system +which allows external dialects to be integrated into SQLAlchemy using +standard setuptools entry points. As of version 0.8, this system has +been enhanced, so that a dialect can also be "plugged in" at runtime. + +On the testing side, SQLAlchemy as of 0.8 also includes a "dialect +compliance suite" that is usable by third party libraries. There is no +longer a strong need for a new dialect to run through SQLAlchemy's full +testing suite, as a large portion of these tests do not have +dialect-sensitive functionality. The "dialect compliance suite" should +be viewed as the primary target for new dialects, and as it continues +to grow and mature it should become a more thorough and efficient system +of testing new dialects. + +As of SQLAlchemy 0.9.4, both nose and pytest are supported for running tests, +and pytest is now preferred. + +Dialect Layout +=============== + +The file structure of a dialect is typically similar to the following:: + + sqlalchemy-/ + setup.py + setup.cfg + run_tests.py + sqlalchemy_/ + __init__.py + base.py + .py + requirements.py + test/ + conftest.py + __init__.py + test_suite.py + test_.py + ... + +An example of this structure can be seen in the Access dialect at +https://bitbucket.org/zzzeek/sqlalchemy-access/. + +Key aspects of this file layout include: + +* setup.py - should specify setuptools entrypoints, allowing the + dialect to be usable from create_engine(), e.g.:: + + entry_points={ + 'sqlalchemy.dialects': [ + 'access = sqlalchemy_access.pyodbc:AccessDialect_pyodbc', + 'access.pyodbc = sqlalchemy_access.pyodbc:AccessDialect_pyodbc', + ] + } + + Above, the two entrypoints ``access`` and ``access.pyodbc`` allow URLs to be + used such as:: + + create_engine("access://user:pw at dsn") + + create_engine("access+pyodbc://user:pw at dsn") + +* setup.cfg - this file contains the traditional contents such as [egg_info], + [pytest] and [nosetests] directives, but also contains new directives that are used + by SQLAlchemy's testing framework. E.g. for Access:: + + [egg_info] + tag_build = dev + + [pytest] + addopts= --tb native -v -r fxX + python_files=test/*test_*.py + + [nosetests] + with-sqla_testing = true + where = test + cover-package = sqlalchemy_access + with-coverage = 1 + cover-erase = 1 + + [sqla_testing] + requirement_cls=sqlalchemy_access.requirements:Requirements + profile_file=.profiles.txt + + [db] + default=access+pyodbc://admin at access_test + sqlite=sqlite:///:memory: + + Above, the ``[sqla_testing]`` section contains configuration used by + SQLAlchemy's test plugin. The ``[pytest]`` and ``[nosetests]`` sections + include directives to help with these runners; in the case of + Nose, the directive ``with-sql_testing = true``, which indicates to Nose that + the SQLAlchemy nose plugin should be used. In the case of pytest, the + test/conftest.py file will bootstrap SQLAlchemy's plugin. + +* test/conftest.py - This script bootstraps SQLAlchemy's pytest plugin + into the pytest runner. This + script can also be used to install your third party dialect into + SQLAlchemy without using the setuptools entrypoint system; this allows + your dialect to be present without any explicit setup.py step needed. + The other portion invokes SQLAlchemy's pytest plugin:: + + from sqlalchemy.dialects import registry + + registry.register("access", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc") + registry.register("access.pyodbc", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc") + + from sqlalchemy.testing.plugin.pytestplugin import * + + Where above, the ``registry`` module, introduced in SQLAlchemy 0.8, provides + an in-Python means of installing the dialect entrypoints without the use + of setuptools, using the ``registry.register()`` function in a way that + is similar to the ``entry_points`` directive we placed in our ``setup.py``. + +* run_tests.py - This script is used when running the tests via Nose. + The purpose of the script is to plug in SQLAlchemy's nose plugin into + the Nose environment before the tests run. + + The format of this file is similar to that of conftest.py; first, + the optional but helpful step of registering your third party plugin, + then the other is to import SQLAlchemy's nose runner and invoke it:: + + from sqlalchemy.dialects import registry + + registry.register("access", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc") + registry.register("access.pyodbc", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc") + + from sqlalchemy.testing import runner + + # use this in setup.py 'test_suite': + # test_suite="run_tests.setup_py_test" + def setup_py_test(): + runner.setup_py_test() + + if __name__ == '__main__': + runner.main() + + The call to ``runner.main()`` then runs the Nose front end, which installs + SQLAlchemy's testing plugins. Invoking our custom runner looks like the + following:: + + $ python run_tests.py -v + +* requirements.py - The ``requirements.py`` file is where directives + regarding database and dialect capabilities are set up. + SQLAlchemy's tests are often annotated with decorators that mark + tests as "skip" or "fail" for particular backends. Over time, this + system has been refined such that specific database and DBAPI names + are mentioned less and less, in favor of @requires directives which + state a particular capability. The requirement directive is linked + to target dialects using a ``Requirements`` subclass. The custom + ``Requirements`` subclass is specified in the ``requirements.py`` file + and is made available to SQLAlchemy's test runner using the + ``requirement_cls`` directive inside the ``[sqla_testing]`` section. + + For a third-party dialect, the custom ``Requirements`` class can + usually specify a simple yes/no answer for a particular system. For + example, a requirements file that specifies a database that supports + the RETURNING construct but does not support reflection of tables + might look like this:: + + # sqlalchemy_access/requirements.py + + from sqlalchemy.testing.requirements import SuiteRequirements + + from sqlalchemy.testing import exclusions + + class Requirements(SuiteRequirements): + @property + def table_reflection(self): + return exclusions.closed() + + @property + def returning(self): + return exclusions.open() + + The ``SuiteRequirements`` class in + ``sqlalchemy.testing.requirements`` contains a large number of + requirements rules, which attempt to have reasonable defaults. The + tests will report on those requirements found as they are run. + + The requirements system can also be used when running SQLAlchemy's + primary test suite against the external dialect. In this use case, + a ``--dburi`` as well as a ``--requirements`` flag are passed to SQLAlchemy's + main test runner ``./sqla_nose.py`` so that exclusions specific to the + dialect take place:: + + cd /path/to/sqlalchemy + py.test -v \ + --requirements sqlalchemy_access.requirements:Requirements \ + --dburi access+pyodbc://admin at access_test + +* test_suite.py - Finally, the ``test_suite.py`` module represents a + stub test suite, which pulls in the actual SQLAlchemy test suite. + To pull in the suite as a whole, it can be imported in one step:: + + # test/test_suite.py + + from sqlalchemy.testing.suite import * + + That's all that's needed - the ``sqlalchemy.testing.suite`` package + contains an ever expanding series of tests, most of which should be + annotated with specific requirement decorators so that they can be + fully controlled. To specifically modify some of the tests, they can + be imported by name and subclassed:: + + from sqlalchemy.testing.suite import * + + from sqlalchemy.testing.suite import ComponentReflectionTest as _ComponentReflectionTest + + class ComponentReflectionTest(_ComponentReflectionTest): + @classmethod + def define_views(cls, metadata, schema): + # bypass the "define_views" section of the + # fixture + return + +Going Forward +============== + +The third-party dialect can be distributed like any other Python +module on Pypi. Links to prominent dialects can be featured within +SQLAlchemy's own documentation; contact the developers (see AUTHORS) +for help with this. + +While SQLAlchemy includes many dialects built in, it remains to be +seen if the project as a whole might move towards "plugin" model for +all dialects, including all those currently built in. Now that +SQLAlchemy's dialect API is mature and the test suite is not far +behind, it may be that a better maintenance experience can be +delivered by having all dialects separately maintained and released. + +As new versions of SQLAlchemy are released, the test suite and +requirements file will receive new tests and changes. The dialect +maintainer would normally keep track of these changes and make +adjustments as needed. + +Continuous Integration +====================== + +The most ideal scenario for ongoing dialect testing is continuous +integration, that is, an automated test runner that runs in response +to changes not just in the dialect itself but to new pushes to +SQLAlchemy as well. + +The SQLAlchemy project features a Jenkins installation that runs tests +on Amazon EC2 instances. It is possible for third-party dialect +developers to provide the SQLAlchemy project either with AMIs or EC2 +instance keys which feature test environments appropriate to the +dialect - SQLAlchemy's own Jenkins suite can invoke tests on these +environments. Contact the developers for further info. + diff --git a/lib/sqlalchemy/README.rst b/lib/sqlalchemy/README.rst new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/README.rst @@ -0,0 +1,135 @@ +SQLAlchemy +========== + +The Python SQL Toolkit and Object Relational Mapper + +Introduction +------------- + +SQLAlchemy is the Python SQL toolkit and Object Relational Mapper +that gives application developers the full power and +flexibility of SQL. SQLAlchemy provides a full suite +of well known enterprise-level persistence patterns, +designed for efficient and high-performing database +access, adapted into a simple and Pythonic domain +language. + +Major SQLAlchemy features include: + +* An industrial strength ORM, built + from the core on the identity map, unit of work, + and data mapper patterns. These patterns + allow transparent persistence of objects + using a declarative configuration system. + Domain models + can be constructed and manipulated naturally, + and changes are synchronized with the + current transaction automatically. +* A relationally-oriented query system, exposing + the full range of SQL's capabilities + explicitly, including joins, subqueries, + correlation, and most everything else, + in terms of the object model. + Writing queries with the ORM uses the same + techniques of relational composition you use + when writing SQL. While you can drop into + literal SQL at any time, it's virtually never + needed. +* A comprehensive and flexible system + of eager loading for related collections and objects. + Collections are cached within a session, + and can be loaded on individual access, all + at once using joins, or by query per collection + across the full result set. +* A Core SQL construction system and DBAPI + interaction layer. The SQLAlchemy Core is + separate from the ORM and is a full database + abstraction layer in its own right, and includes + an extensible Python-based SQL expression + language, schema metadata, connection pooling, + type coercion, and custom types. +* All primary and foreign key constraints are + assumed to be composite and natural. Surrogate + integer primary keys are of course still the + norm, but SQLAlchemy never assumes or hardcodes + to this model. +* Database introspection and generation. Database + schemas can be "reflected" in one step into + Python structures representing database metadata; + those same structures can then generate + CREATE statements right back out - all within + the Core, independent of the ORM. + +SQLAlchemy's philosophy: + +* SQL databases behave less and less like object + collections the more size and performance start to + matter; object collections behave less and less like + tables and rows the more abstraction starts to matter. + SQLAlchemy aims to accommodate both of these + principles. +* An ORM doesn't need to hide the "R". A relational + database provides rich, set-based functionality + that should be fully exposed. SQLAlchemy's + ORM provides an open-ended set of patterns + that allow a developer to construct a custom + mediation layer between a domain model and + a relational schema, turning the so-called + "object relational impedance" issue into + a distant memory. +* The developer, in all cases, makes all decisions + regarding the design, structure, and naming conventions + of both the object model as well as the relational + schema. SQLAlchemy only provides the means + to automate the execution of these decisions. +* With SQLAlchemy, there's no such thing as + "the ORM generated a bad query" - you + retain full control over the structure of + queries, including how joins are organized, + how subqueries and correlation is used, what + columns are requested. Everything SQLAlchemy + does is ultimately the result of a developer- + initiated decision. +* Don't use an ORM if the problem doesn't need one. + SQLAlchemy consists of a Core and separate ORM + component. The Core offers a full SQL expression + language that allows Pythonic construction + of SQL constructs that render directly to SQL + strings for a target database, returning + result sets that are essentially enhanced DBAPI + cursors. +* Transactions should be the norm. With SQLAlchemy's + ORM, nothing goes to permanent storage until + commit() is called. SQLAlchemy encourages applications + to create a consistent means of delineating + the start and end of a series of operations. +* Never render a literal value in a SQL statement. + Bound parameters are used to the greatest degree + possible, allowing query optimizers to cache + query plans effectively and making SQL injection + attacks a non-issue. + +Documentation +------------- + +Latest documentation is at: + +http://www.sqlalchemy.org/docs/ + +Installation / Requirements +--------------------------- + +Full documentation for installation is at +`Installation `_. + +Getting Help / Development / Bug reporting +------------------------------------------ + +Please refer to the `SQLAlchemy Community Guide `_. + +License +------- + +SQLAlchemy is distributed under the `MIT license +`_. + diff --git a/lib/sqlalchemy/README.unittests.rst b/lib/sqlalchemy/README.unittests.rst new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/README.unittests.rst @@ -0,0 +1,340 @@ +===================== +SQLALCHEMY UNIT TESTS +===================== + +**NOTE:** SQLAlchemy as of 0.9.4 now standardizes on `pytest `_ +for test running! However, the existing support for Nose **still remains**! +That is, you can now run the tests via pytest or nose. We hope to keep the +suite nose-compatible indefinitely however this might change at some point. + +SQLAlchemy unit tests by default run using Python's built-in sqlite3 +module. If running on a Python installation that doesn't include this +module, then pysqlite or compatible must be installed. + +Unit tests can be run with pytest or nose: + + py.test: http://pytest.org/ + + nose: https://pypi.python.org/pypi/nose/ + +The suite includes enhanced support when running with pytest. + +SQLAlchemy implements plugins for both pytest and nose that must be +present when tests are run. In the case of pytest, this plugin is automatically +used when pytest is run against the SQLAlchemy source tree. However, +for Nose support, a special test runner script must be used. + + +The test suite as also requires the mock library. While +mock is part of the Python standard library as of 3.3, previous versions +will need to have it installed, and is available at:: + + https://pypi.python.org/pypi/mock + +RUNNING TESTS VIA SETUP.PY +-------------------------- +A plain vanilla run of all tests using sqlite can be run via setup.py, and +requires that pytest is installed:: + + $ python setup.py test + + +RUNNING ALL TESTS - PYTEST +-------------------------- +To run all tests:: + + $ py.test + +The pytest configuration in setup.cfg will point the runner at the +test/ directory, where it consumes a conftest.py file that gets everything +else up and running. + + +RUNNING ALL TESTS - NOSE +-------------------------- + +When using Nose, a bootstrap script is provided which sets up sys.path +as well as installs the nose plugin:: + + $ ./sqla_nose.py + +Assuming all tests pass, this is a very unexciting output. To make it more +interesting:: + + $ ./sqla_nose.py -v + +RUNNING INDIVIDUAL TESTS +--------------------------------- + +Any directory of test modules can be run at once by specifying the directory +path, and a specific file can be specified as well:: + + $ py.test test/dialect + + $ py.test test/orm/test_mapper.py + +When using nose, the setup.cfg currently sets "where" to "test/", so the +"test/" prefix is omitted:: + + $ ./sqla_nose.py dialect/ + + $ ./sqla_nose.py orm/test_mapper.py + +With Nose, it is often more intuitive to specify tests as module paths:: + + $ ./sqla_nose.py test.orm.test_mapper + +Nose can also specify a test class and optional method using this syntax:: + + $ ./sqla_nose.py test.orm.test_mapper:MapperTest.test_utils + +With pytest, the -k flag is used to limit tests:: + + $ py.test test/orm/test_mapper.py -k "MapperTest and test_utils" + + +COMMAND LINE OPTIONS +-------------------- + +SQLAlchemy-specific options are added to both runners, which are viewable +within the help screen. With pytest, these options are easier to locate +as they are underneath the "sqlalchemy" grouping:: + + $ py.test --help + + $ ./sqla_nose.py --help + +The --help screen is a combination of common nose options and options which +the SQLAlchemy nose plugin adds. The most commonly SQLAlchemy-specific +options used are '--db' and '--dburi'. + +Both pytest and nose support the same set of SQLAlchemy options, though +pytest features a bit more capability with them. + + +DATABASE TARGETS +---------------- + +Tests will target an in-memory SQLite database by default. To test against +another database, use the --dburi option with any standard SQLAlchemy URL:: + + --dburi=postgresql://user:password at localhost/test + +If you'll be running the tests frequently, database aliases can save a lot of +typing. The --dbs option lists the built-in aliases and their matching URLs:: + + $ py.test --dbs + Available --db options (use --dburi to override) + mysql mysql://scott:tiger at 127.0.0.1:3306/test + oracle oracle://scott:tiger at 127.0.0.1:1521 + postgresql postgresql://scott:tiger at 127.0.0.1:5432/test + [...] + +To run tests against an aliased database:: + + $ py.test --db postgresql + +This list of database urls is present in the setup.cfg file. The list +can be modified/extended by adding a file ``test.cfg`` at the +top level of the SQLAlchemy source distribution which includes +additional entries:: + + [db] + postgresql=postgresql://myuser:mypass at localhost/mydb + +Your custom entries will override the defaults and you'll see them reflected +in the output of --dbs. + +MULTIPLE DATABASE TARGETS +------------------------- + +As of SQLAlchemy 0.9.4, the test runner supports **multiple databases at once**. +This doesn't mean that the entire test suite runs for each database, but +instead specific test suites may do so, while other tests may choose to +run on a specific target out of those available. For example, if the tests underneath +test/dialect/ are run, the majority of these tests are either specific to +a particular backend, or are marked as "multiple", meaning they will run repeatedly +for each database in use. If one runs the test suite as follows:: + + $ py.test test/dialect --db sqlite --db postgresql --db mysql + +The tests underneath test/dialect/test_suite.py will be tripled up, running +as appropriate for each target database, whereas dialect-specific tests +within test/dialect/mysql, test/dialect/postgresql/ test/dialect/test_sqlite.py +should run fully with no skips, as each suite has its target database available. + +The multiple targets feature is available both under pytest and nose, +however when running nose, the "multiple runner" feature won't be available; +instead, the first database target will be used. + +When running with multiple targets, tests that don't prefer a specific target +will be run against the first target specified. Putting sqlite first in +the list will lead to a much faster suite as the in-memory database is +extremely fast for setting up and tearing down tables. + + + +DATABASE CONFIGURATION +---------------------- + +Use an empty database and a database user with general DBA privileges. +The test suite will be creating and dropping many tables and other DDL, and +preexisting tables will interfere with the tests. + +Several tests require alternate usernames or schemas to be present, which +are used to test dotted-name access scenarios. On some databases such +as Oracle or Sybase, these are usernames, and others such as Postgresql +and MySQL they are schemas. The requirement applies to all backends +except SQLite and Firebird. The names are:: + + test_schema + test_schema_2 (only used on Postgresql) + +Please refer to your vendor documentation for the proper syntax to create +these namespaces - the database user must have permission to create and drop +tables within these schemas. Its perfectly fine to run the test suite +without these namespaces present, it only means that a handful of tests which +expect them to be present will fail. + +Additional steps specific to individual databases are as follows:: + + POSTGRESQL: To enable unicode testing with JSONB, create the + database with UTF8 encoding:: + + postgres=# create database test with owner=scott encoding='utf8' template=template0; + + To include tests for HSTORE, create the HSTORE type engine:: + + postgres=# \c test; + You are now connected to database "test" as user "postgresql". + test=# create extension hstore; + CREATE EXTENSION + + MYSQL: Default storage engine should be "MyISAM". Tests that require + "InnoDB" as the engine will specify this explicitly. + + ORACLE: a user named "test_schema" is created. + + The primary database user needs to be able to create and drop tables, + synonyms, and constraints within the "test_schema" user. For this + to work fully, including that the user has the "REFERENCES" role + in a remote schema for tables not yet defined (REFERENCES is per-table), + it is required that the test the user be present in the "DBA" role: + + grant dba to scott; + + SYBASE: Similar to Oracle, "test_schema" is created as a user, and the + primary test user needs to have the "sa_role". + + It's also recommended to turn on "trunc log on chkpt" and to use a + separate transaction log device - Sybase basically seizes up when + the transaction log is full otherwise. + + A full series of setup assuming sa/master: + + disk init name="translog", physname="/opt/sybase/data/translog.dat", size="10M" + create database sqlalchemy on default log on translog="10M" + sp_dboption sqlalchemy, "trunc log on chkpt", true + sp_addlogin scott, "tiger7" + sp_addlogin test_schema, "tiger7" + use sqlalchemy + sp_adduser scott + sp_adduser test_schema + grant all to scott + sp_role "grant", sa_role, scott + + Sybase will still freeze for up to a minute when the log becomes + full. To manually dump the log:: + + dump tran sqlalchemy with truncate_only + + MSSQL: Tests that involve multiple connections require Snapshot Isolation + ability implemented on the test database in order to prevent deadlocks that + will occur with record locking isolation. This feature is only available + with MSSQL 2005 and greater. You must enable snapshot isolation at the + database level and set the default cursor isolation with two SQL commands: + + ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON + + ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON + + MSSQL+zxJDBC: Trying to run the unit tests on Windows against SQL Server + requires using a test.cfg configuration file as the cmd.exe shell won't + properly pass the URL arguments into the nose test runner. + + POSTGRESQL: Full-text search configuration should be set to English, else + several tests of ``.match()`` will fail. This can be set (if it isn't so + already) with: + + ALTER DATABASE test SET default_text_search_config = 'pg_catalog.english' + + +CONFIGURING LOGGING +------------------- +SQLAlchemy logs its activity and debugging through Python's logging package. +Any log target can be directed to the console with command line options, such +as:: + + $ ./sqla_nose.py test.orm.unitofwork --log-info=sqlalchemy.orm.mapper \ + --log-debug=sqlalchemy.pool --log-info=sqlalchemy.engine + +This would log mapper configuration, connection pool checkouts, and SQL +statement execution. + + +BUILT-IN COVERAGE REPORTING +------------------------------ +Coverage is tracked using the coverage plugins built for pytest or nose:: + + $ py.test test/sql/test_query --cov=sqlalchemy + + $ ./sqla_nose.py test.sql.test_query --with-coverage + +BIG COVERAGE TIP !!! There is an issue where existing .pyc files may +store the incorrect filepaths, which will break the coverage system. If +coverage numbers are coming out as low/zero, try deleting all .pyc files. + +DEVELOPING AND TESTING NEW DIALECTS +----------------------------------- + +See the file README.dialects.rst for detail on dialects. + + +TESTING WITH MULTIPLE PYTHON VERSIONS USING TOX +----------------------------------------------- + +If you want to test across multiple versions of Python, you may find `tox +`_ useful. SQLAlchemy includes a tox.ini file:: + + tox -e full + +SQLAlchemy uses tox mostly for pre-fab testing configurations, to simplify +configuration of Jenkins jobs, and *not* for testing different Python +interpreters simultaneously. You can of course create whatever alternate +tox.ini file you want. + +Environments include:: + + "full" - runs a full py.test + + "coverage" - runs a py.test plus coverage, skipping memory/timing + intensive tests + + "pep8" - runs flake8 against the codebase (useful with --diff to check + against a patch) + + +PARALLEL TESTING +---------------- + +Parallel testing is supported using the Pytest xdist plugin. Supported +databases currently include sqlite, postgresql, and mysql. The username +for the database should have CREATE DATABASE and DROP DATABASE privileges. +After installing pytest-xdist, testing is run adding the -n option. +For example, to run against sqlite, mysql, postgresql with four processes:: + + tox -e -- -n 4 --db sqlite --db postgresql --db mysql + +Each backend has a different scheme for setting up the database. Postgresql +still needs the "test_schema" and "test_schema_2" schemas present, as the +parallel databases are created using the base database as a "template". diff --git a/lib/sqlalchemy/doc/_images/sqla_arch_small.png b/lib/sqlalchemy/doc/_images/sqla_arch_small.png new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..a1c09585ec8d45ee60ad356b315d5a4089421068 GIT binary patch [cut] diff --git a/lib/sqlalchemy/doc/_images/sqla_engine_arch.png b/lib/sqlalchemy/doc/_images/sqla_engine_arch.png new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..f040a2cf317c0118bc9d5b12c458224dea4eb690 GIT binary patch [cut] diff --git a/lib/sqlalchemy/doc/_modules/examples/adjacency_list/adjacency_list.html b/lib/sqlalchemy/doc/_modules/examples/adjacency_list/adjacency_list.html new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/doc/_modules/examples/adjacency_list/adjacency_list.html @@ -0,0 +1,253 @@ + + + + + + + + + + examples.adjacency_list.adjacency_list + — + SQLAlchemy 1.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ + + + + + + +

+ + + + + +

+ +

Source code for examples.adjacency_list.adjacency_list

+from sqlalchemy import Column, ForeignKey, Integer, String, create_engine
+from sqlalchemy.orm import Session, relationship, backref,\
+                                joinedload_all
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.collections import attribute_mapped_collection
+
+
+Base = declarative_base()
+
+class TreeNode(Base):
+    __tablename__ = 'tree'
+    id = Column(Integer, primary_key=True)
+    parent_id = Column(Integer, ForeignKey(id))
+    name = Column(String(50), nullable=False)
+
+    children = relationship("TreeNode",
+
+                        # cascade deletions
+                        cascade="all, delete-orphan",
+
+                        # many to one + adjacency list - remote_side
+                        # is required to reference the 'remote'
+                        # column in the join condition.
+                        backref=backref("parent", remote_side=id),
+
+                        # children will be represented as a dictionary
+                        # on the "name" attribute.
+                        collection_class=attribute_mapped_collection('name'),
+                    )
+
+    def __init__(self, name, parent=None):
+        self.name = name
+        self.parent = parent
+
+    def __repr__(self):
+        return "TreeNode(name=%r, id=%r, parent_id=%r)" % (
+                    self.name,
+                    self.id,
+                    self.parent_id
+                )
+
+    def dump(self, _indent=0):
+        return "   " * _indent + repr(self) + \
+                    "\n" + \
+                    "".join([
+                        c.dump(_indent + 1)
+                        for c in self.children.values()]
+                    )
+
+if __name__ == '__main__':
+    engine = create_engine('sqlite://', echo=True)
+
+    def msg(msg, *args):
+        msg = msg % args
+        print("\n\n\n" + "-" * len(msg.split("\n")[0]))
+        print(msg)
+        print("-" * len(msg.split("\n")[0]))
+
+    msg("Creating Tree Table:")
+
+    Base.metadata.create_all(engine)
+
+    session = Session(engine)
+
+    node = TreeNode('rootnode')
+    TreeNode('node1', parent=node)
+    TreeNode('node3', parent=node)
+
+    node2 = TreeNode('node2')
+    TreeNode('subnode1', parent=node2)
+    node.children['node2'] = node2
+    TreeNode('subnode2', parent=node.children['node2'])
+
+    msg("Created new tree structure:\n%s", node.dump())
+
+    msg("flush + commit:")
+
+    session.add(node)
+    session.commit()
+
+    msg("Tree After Save:\n %s", node.dump())
+
+    TreeNode('node4', parent=node)
+    TreeNode('subnode3', parent=node.children['node4'])
+    TreeNode('subnode4', parent=node.children['node4'])
+    TreeNode('subsubnode1', parent=node.children['node4'].children['subnode3'])
+
+    # remove node1 from the parent, which will trigger a delete
+    # via the delete-orphan cascade.
+    del node.children['node1']
+
+    msg("Removed node1.  flush + commit:")
+    session.commit()
+
+    msg("Tree after save:\n %s", node.dump())
+
+    msg("Emptying out the session entirely, "
+        "selecting tree on root, using eager loading to join four levels deep.")
+    session.expunge_all()
+    node = session.query(TreeNode).\
+                        options(joinedload_all("children", "children",
+                                                "children", "children")).\
+                        filter(TreeNode.name == "rootnode").\
+                        first()
+
+    msg("Full Tree:\n%s", node.dump())
+
+    msg("Marking root node as deleted, flush + commit:")
+
+    session.delete(node)
+    session.commit()

+ +

+ + + +

+ + + + + + + + + + + + + + + + + + + + + diff --git a/lib/sqlalchemy/doc/_modules/examples/association/basic_association.html b/lib/sqlalchemy/doc/_modules/examples/association/basic_association.html new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/doc/_modules/examples/association/basic_association.html @@ -0,0 +1,240 @@ + + + + + + + + + + examples.association.basic_association + — + SQLAlchemy 1.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ + + + + + + +

+ + + + + +

+ +

Source code for examples.association.basic_association

+"""basic_association.py
+
+illustrate a many-to-many relationship between an
+"Order" and a collection of "Item" objects, associating a purchase price
+with each via an association object called "OrderItem"
+
+The association object pattern is a form of many-to-many which
+associates additional data with each association between parent/child.
+
+The example illustrates an "order", referencing a collection
+of "items", with a particular price paid associated with each "item".
+
+"""
+
+from datetime import datetime
+
+from sqlalchemy import (create_engine, MetaData, Table, Column, Integer,
+    String, DateTime, Float, ForeignKey, and_)
+from sqlalchemy.orm import mapper, relationship, Session
+from sqlalchemy.ext.declarative import declarative_base
+
+Base = declarative_base()
+
+class Order(Base):
+    __tablename__ = 'order'
+
+    order_id = Column(Integer, primary_key=True)
+    customer_name = Column(String(30), nullable=False)
+    order_date = Column(DateTime, nullable=False, default=datetime.now())
+    order_items = relationship("OrderItem", cascade="all, delete-orphan",
+                            backref='order')
+
+    def __init__(self, customer_name):
+        self.customer_name = customer_name
+
+class Item(Base):
+    __tablename__ = 'item'
+    item_id = Column(Integer, primary_key=True)
+    description = Column(String(30), nullable=False)
+    price = Column(Float, nullable=False)
+
+    def __init__(self, description, price):
+        self.description = description
+        self.price = price
+
+    def __repr__(self):
+        return 'Item(%r, %r)' % (
+                    self.description, self.price
+                )
+
+class OrderItem(Base):
+    __tablename__ = 'orderitem'
+    order_id = Column(Integer, ForeignKey('order.order_id'), primary_key=True)
+    item_id = Column(Integer, ForeignKey('item.item_id'), primary_key=True)
+    price = Column(Float, nullable=False)
+
+    def __init__(self, item, price=None):
+        self.item = item
+        self.price = price or item.price
+    item = relationship(Item, lazy='joined')
+
+if __name__ == '__main__':
+    engine = create_engine('sqlite://')
+    Base.metadata.create_all(engine)
+
+    session = Session(engine)
+
+    # create catalog
+    tshirt, mug, hat, crowbar = (
+        Item('SA T-Shirt', 10.99),
+        Item('SA Mug', 6.50),
+        Item('SA Hat', 8.99),
+        Item('MySQL Crowbar', 16.99)
+    )
+    session.add_all([tshirt, mug, hat, crowbar])
+    session.commit()
+
+    # create an order
+    order = Order('john smith')
+
+    # add three OrderItem associations to the Order and save
+    order.order_items.append(OrderItem(mug))
+    order.order_items.append(OrderItem(crowbar, 10.99))
+    order.order_items.append(OrderItem(hat))
+    session.add(order)
+    session.commit()
+
+    # query the order, print items
+    order = session.query(Order).filter_by(customer_name='john smith').one()
+    print([(order_item.item.description, order_item.price)
+           for order_item in order.order_items])
+
+    # print customers who bought 'MySQL Crowbar' on sale
+    q = session.query(Order).join('order_items', 'item')
+    q = q.filter(and_(Item.description == 'MySQL Crowbar',
+                      Item.price > OrderItem.price))
+
+    print([order.customer_name for order in q])

+ +

+ + + +

+ + + + + + + + + + + + + + + + + + + + + diff --git a/lib/sqlalchemy/doc/_modules/examples/association/dict_of_sets_with_default.html b/lib/sqlalchemy/doc/_modules/examples/association/dict_of_sets_with_default.html new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/doc/_modules/examples/association/dict_of_sets_with_default.html @@ -0,0 +1,232 @@ + + + + + + + + + + examples.association.dict_of_sets_with_default + — + SQLAlchemy 1.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ + + + + + + +

+ + + + + +

+ +

Source code for examples.association.dict_of_sets_with_default

+"""dict_of_sets_with_default.py
+
+an advanced association proxy example which
+illustrates nesting of association proxies to produce multi-level Python
+collections, in this case a dictionary with string keys and sets of integers
+as values, which conceal the underlying mapped classes.
+
+This is a three table model which represents a parent table referencing a
+dictionary of string keys and sets as values, where each set stores a
+collection of integers. The association proxy extension is used to hide the
+details of this persistence. The dictionary also generates new collections
+upon access of a non-existent key, in the same manner as Python's
+"collections.defaultdict" object.
+
+"""
+
+from sqlalchemy import String, Integer, Column, create_engine, ForeignKey
+from sqlalchemy.orm import relationship, Session
+from sqlalchemy.orm.collections import MappedCollection
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.ext.associationproxy import association_proxy
+import operator
+
+class Base(object):
+    id = Column(Integer, primary_key=True)
+
+Base = declarative_base(cls=Base)
+
+class GenDefaultCollection(MappedCollection):
+    def __missing__(self, key):
+        self[key] = b = B(key)
+        return b
+
+class A(Base):
+    __tablename__ = "a"
+    associations = relationship("B",
+        collection_class=lambda: GenDefaultCollection(operator.attrgetter("key"))
+    )
+
+    collections = association_proxy("associations", "values")
+    """Bridge the association from 'associations' over to the 'values'
+    association proxy of B.
+    """
+
+class B(Base):
+    __tablename__ = "b"
+    a_id = Column(Integer, ForeignKey("a.id"), nullable=False)
+    elements = relationship("C", collection_class=set)
+    key = Column(String)
+
+    values = association_proxy("elements", "value")
+    """Bridge the association from 'elements' over to the
+    'value' element of C."""
+
+    def __init__(self, key, values=None):
+        self.key = key
+        if values:
+            self.values = values
+
+class C(Base):
+    __tablename__ = "c"
+    b_id = Column(Integer, ForeignKey("b.id"), nullable=False)
+    value = Column(Integer)
+    def __init__(self, value):
+        self.value = value
+
+if __name__ == '__main__':
+    engine = create_engine('sqlite://', echo=True)
+    Base.metadata.create_all(engine)
+    session = Session(engine)
+
+    # only "A" is referenced explicitly.  Using "collections",
+    # we deal with a dict of key/sets of integers directly.
+
+    session.add_all([
+        A(collections={
+            "1": set([1, 2, 3]),
+        })
+    ])
+    session.commit()
+
+    a1 = session.query(A).first()
+    print(a1.collections["1"])
+    a1.collections["1"].add(4)
+    session.commit()
+
+    a1.collections["2"].update([7, 8, 9])
+    session.commit()
+
+    print(a1.collections["2"])

+ +

+ + + +

+ + + + + + + + + + + + + + + + + + + + + diff --git a/lib/sqlalchemy/doc/_modules/examples/association/proxied_association.html b/lib/sqlalchemy/doc/_modules/examples/association/proxied_association.html new file mode 100644 --- /dev/null +++ b/lib/sqlalchemy/doc/_modules/examples/association/proxied_association.html @@ -0,0 +1,247 @@ + + + + + + + + + + examples.association.proxied_association + — + SQLAlchemy 1.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ + + + + + + +

+ + + + + +

+ +

Source code for examples.association.proxied_association

+"""proxied_association.py
+
+same example as basic_association, adding in
+usage of :mod:`sqlalchemy.ext.associationproxy` to make explicit references
+to ``OrderItem`` optional.
+
+
+"""
+
+from datetime import datetime
+
+from sqlalchemy import (create_engine, MetaData, Table, Column, Integer,
+    String, DateTime, Float, ForeignKey, and_)
+from sqlalchemy.orm import mapper, relationship, Session
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.ext.associationproxy import association_proxy
+
+Base = declarative_base()
+
+class Order(Base):
+    __tablename__ = 'order'
+
+    order_id = Column(Integer, primary_key=True)
+    customer_name = Column(String(30), nullable=False)
+    order_date = Column(DateTime, nullable=False, default=datetime.now())
+    order_items = relationship("OrderItem", cascade="all, delete-orphan",
+                            backref='order')
+    items = association_proxy("order_items", "item")
+
+    def __init__(self, customer_name):
+        self.customer_name = customer_name
+
+class Item(Base):
+    __tablename__ = 'item'
+    item_id = Column(Integer, primary_key=True)
+    description = Column(String(30), nullable=False)
+    price = Column(Float, nullable=False)
+
+    def __init__(self, description, price):
+        self.description = description
+        self.price = price
+
+    def __repr__(self):
+        return 'Item(%r, %r)' % (
+                    self.description, self.price
+                )
+
+class OrderItem(Base):
+    __tablename__ = 'orderitem'
+    order_id = Column(Integer, ForeignKey('order.order_id'), primary_key=True)
+    item_id = Column(Integer, ForeignKey('item.item_id'), primary_key=True)
+    price = Column(Float, nullable=False)
+
+    def __init__(self, item, price=None):
+        self.item = item
+        self.price = price or item.price
+    item = relationship(Item, lazy='joined')
+
+if __name__ == '__main__':
+    engine = create_engine('sqlite://')
+    Base.metadata.create_all(engine)
+
+    session = Session(engine)
+
+    # create catalog
+    tshirt, mug, hat, crowbar = (
+        Item('SA T-Shirt', 10.99),
+        Item('SA Mug', 6.50),
+        Item('SA Hat', 8.99),
+        Item('MySQL Crowbar', 16.99)
+    )
+    session.add_all([tshirt, mug, hat, crowbar])
+    session.commit()
+
+    # create an order
+    order = Order('john smith')
+
+    # add items via the association proxy.
+    # the OrderItem is created automatically.
+    order.items.append(mug)
+    order.items.append(hat)
+
+    # add an OrderItem explicitly.
+    order.order_items.append(OrderItem(crowbar, 10.99))
+
+    session.add(order)
+    session.commit()
+
+    # query the order, print items
+    order = session.query(Order).filter_by(customer_name='john smith').one()
+
+    # print items based on the OrderItem collection directly
+    print([(assoc.item.description, assoc.price, assoc.item.price)
+           for assoc in order.order_items])
+
+    # print items based on the "proxied" items collection
+    print([(item.description, item.price)
+           for item in order.items])
+
+    # print customers who bought 'MySQL Crowbar' on sale
+    orders = session.query(Order).\
+                    join('order_items', 'item').\
+                    filter(Item.description == 'MySQL Crowbar').\
+                    filter(Item.price > OrderItem.price)
+    print([o.customer_name for o in orders])

+ +

+ + + +

From pypy.commits at gmail.com Thu Jan 21 02:24:00 2016 From: pypy.commits at gmail.com (cfbolz) Date: Wed, 20 Jan 2016 23:24:00 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: fix the interaction between record_exact_class and the heapcache Message-ID: <56a08790.8205c20a.bf1e4.0054@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81881:7a18372094c5 Date: 2016-01-21 08:23 +0100 http://bitbucket.org/pypy/pypy/changeset/7a18372094c5/ Log: fix the interaction between record_exact_class and the heapcache diff --git a/rpython/jit/metainterp/heapcache.py b/rpython/jit/metainterp/heapcache.py --- a/rpython/jit/metainterp/heapcache.py +++ b/rpython/jit/metainterp/heapcache.py @@ -206,6 +206,7 @@ opnum == rop.SETFIELD_RAW or opnum == rop.SETARRAYITEM_RAW or opnum == rop.SETINTERIORFIELD_RAW or + opnum == rop.RECORD_EXACT_CLASS or opnum == rop.RAW_STORE): return if (rop._OVF_FIRST <= opnum <= rop._OVF_LAST or diff --git a/rpython/jit/metainterp/test/test_tracingopts.py b/rpython/jit/metainterp/test/test_tracingopts.py --- a/rpython/jit/metainterp/test/test_tracingopts.py +++ b/rpython/jit/metainterp/test/test_tracingopts.py @@ -707,3 +707,29 @@ res = self.interp_operations(fn, [0]) assert res == 0 self.check_operations_history(setfield_gc=0) + + def test_record_known_class_does_not_invalidate(self): + class A: + pass + class B(A): + pass + class C(object): + _immutable_fields_ = ['x?'] + c = C() + c.x = 5 + c.b = A() + c.b.x = 14 + def fn(n): + if n == 99: + c.x = 12 + c.b = B() + c.b.x = 12 + return 15 + b = c.b + x = b.x + jit.record_exact_class(c.b, A) + y = b.x + return x + y + res = self.interp_operations(fn, [1]) + assert res == 2 * 14 + self.check_operations_history(getfield_gc_i=1) From pypy.commits at gmail.com Thu Jan 21 03:04:24 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 00:04:24 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: fix test_pypy_c tests Message-ID: <56a09108.6adec20a.b3bf0.1040@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81882:7dfb24f5e8c5 Date: 2016-01-21 09:03 +0100 http://bitbucket.org/pypy/pypy/changeset/7dfb24f5e8c5/ Log: fix test_pypy_c tests diff --git a/pypy/module/pypyjit/test_pypy_c/test_00_model.py b/pypy/module/pypyjit/test_pypy_c/test_00_model.py --- a/pypy/module/pypyjit/test_pypy_c/test_00_model.py +++ b/pypy/module/pypyjit/test_pypy_c/test_00_model.py @@ -526,7 +526,7 @@ log = self.run(f) loop, = log.loops_by_filename(self.filepath) call_ops = log.opnames(loop.ops_by_id('call')) - assert call_ops == ['guard_not_invalidated', 'force_token'] # it does not follow inlining + assert call_ops == ['force_token'] # it does not follow inlining # add_ops = log.opnames(loop.ops_by_id('add')) assert add_ops == ['int_add'] @@ -535,7 +535,6 @@ assert ops == [ # this is the actual loop 'guard_not_invalidated', 'int_lt', 'guard_true', 'force_token', 'int_add', - 'force_token', 'int_add', # this is the signal checking stuff 'getfield_raw_i', 'int_lt', 'guard_false', 'jump' diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -71,7 +71,7 @@ entry_bridge, = log.loops_by_id('call', is_entry_bridge=True) # LOAD_GLOBAL of OFFSET ops = entry_bridge.ops_by_id('cond', opcode='LOAD_GLOBAL') - assert log.opnames(ops) == ["guard_value"] + assert log.opnames(ops) == [] ops = entry_bridge.ops_by_id('add', opcode='LOAD_GLOBAL') assert log.opnames(ops) == [] # @@ -218,7 +218,7 @@ loop, = log.loops_by_id('call') ops = log.opnames(loop.ops_by_id('call')) guards = [ops for ops in ops if ops.startswith('guard')] - assert guards == ["guard_not_invalidated", "guard_no_overflow"] + assert guards == ["guard_no_overflow"] def test_kwargs(self): # this is not a very precise test, could be improved diff --git a/pypy/module/pypyjit/test_pypy_c/test_ffi.py b/pypy/module/pypyjit/test_pypy_c/test_ffi.py --- a/pypy/module/pypyjit/test_pypy_c/test_ffi.py +++ b/pypy/module/pypyjit/test_pypy_c/test_ffi.py @@ -422,7 +422,6 @@ i114 = int_ne(i160, i112) guard_false(i114, descr=...) --TICK-- - i123 = arraylen_gc(p67, descr=) i119 = call_i(ConstClass(_ll_1_raw_malloc_varsize__Signed), 6, descr=) raw_store(i119, 0, i160, descr=) raw_store(i119, 2, i160, descr=) diff --git a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py --- a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py +++ b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py @@ -299,10 +299,12 @@ i129 = int_add(i55, i128) f149 = raw_load_f(i100, i129, descr=) i151 = int_add(i117, 1) + i92 = getfield_raw_i(54402752, descr=) setfield_gc(p156, i55, descr=) setarrayitem_gc(p150, 1, 0, descr=) setarrayitem_gc(p150, 0, 0, descr=) - --TICK-- + i95 = int_lt(i92, 0) + guard_false(i95, descr=...) jump(..., descr=...) """) @@ -354,12 +356,13 @@ guard_false(i92, descr=...) i93 = int_add(i91, 1) setfield_gc(p23, i93, descr=) + guard_not_invalidated? i94 = int_ge(i91, i56) guard_false(i94, descr=...) i96 = int_mul(i91, i58) i97 = int_add(i51, i96) f98 = raw_load_f(i63, i97, descr=) - guard_not_invalidated(descr=...) + guard_not_invalidated? f100 = float_mul(f98, 0.500000) i101 = int_add(i79, 1) i102 = arraylen_gc(p85, descr=) diff --git a/pypy/module/pypyjit/test_pypy_c/test_misc.py b/pypy/module/pypyjit/test_pypy_c/test_misc.py --- a/pypy/module/pypyjit/test_pypy_c/test_misc.py +++ b/pypy/module/pypyjit/test_pypy_c/test_misc.py @@ -114,16 +114,14 @@ assert log.result == main(1000) loop, = log.loops_by_filename(self.filepath) assert loop.match(""" + guard_not_invalidated? i12 = int_is_true(i4) guard_true(i12, descr=...) - guard_not_invalidated(descr=...) - i13 = int_add_ovf(i8, i9) + guard_not_invalidated? + i13 = int_add_ovf(i8, 2) guard_no_overflow(descr=...) - i10 = int_mul_ovf(2, i61) + i14 = int_add_ovf(i13, 2) guard_no_overflow(descr=...) - i14 = int_add_ovf(i13, i10) - guard_no_overflow(descr=...) - setfield_gc(p7, i11, descr=...) i17 = int_sub_ovf(i4, 1) guard_no_overflow(descr=...) --TICK-- @@ -151,9 +149,10 @@ setfield_gc(p9, i17, descr=<.* .*W_XRangeIterator.inst_current .*>) guard_not_invalidated(descr=...) i18 = force_token() + i75 = int_sub(i71, 1) i21 = int_lt(i10, 0) guard_false(i21, descr=...) - i22 = int_lt(i10, i14) + i22 = int_lt(i10, _) guard_true(i22, descr=...) i23 = int_add_ovf(i6, i10) guard_no_overflow(descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_thread.py b/pypy/module/pypyjit/test_pypy_c/test_thread.py --- a/pypy/module/pypyjit/test_pypy_c/test_thread.py +++ b/pypy/module/pypyjit/test_pypy_c/test_thread.py @@ -33,6 +33,7 @@ import thread local = thread._local() local.x = 1 + local.x = 2 # make it not constant i = 0 while i < n: i += local.x @@ -41,9 +42,10 @@ assert round(log.result, 6) == round(main(500), 6) loop, = log.loops_by_filename(self.filepath) assert loop.match(""" + guard_not_invalidated? i53 = int_lt(i48, i27) guard_true(i53, descr=...) - guard_not_invalidated(descr=...) + guard_not_invalidated? i54 = int_add_ovf(i48, i47) guard_no_overflow(descr=...) --TICK-- diff --git a/pypy/module/pypyjit/test_pypy_c/test_weakref.py b/pypy/module/pypyjit/test_pypy_c/test_weakref.py --- a/pypy/module/pypyjit/test_pypy_c/test_weakref.py +++ b/pypy/module/pypyjit/test_pypy_c/test_weakref.py @@ -25,8 +25,6 @@ i61 = int_add(i58, 1) setfield_gc(p18, i61, descr=) guard_not_invalidated(descr=...) - p62 = getfield_gc_r(ConstPtr(ptr37), descr=) - guard_value(p62, ConstPtr(ptr39), descr=...) p65 = getfield_gc_r(p14, descr=) guard_value(p65, ConstPtr(ptr45), descr=...) p67 = force_token() From pypy.commits at gmail.com Thu Jan 21 03:51:44 2016 From: pypy.commits at gmail.com (Vincent Legoll) Date: Thu, 21 Jan 2016 00:51:44 -0800 (PST) Subject: [pypy-commit] pypy default: Fix test_rabspath_absolute_nt Message-ID: <56a09c20.85e41c0a.93cdb.ffffe7eb@mx.google.com> Author: Vincent Legoll Branch: Changeset: r81883:49cd97514141 Date: 2016-01-21 08:52 +0100 http://bitbucket.org/pypy/pypy/changeset/49cd97514141/ Log: Fix test_rabspath_absolute_nt rpath._nt_rsplitdrive() returns a tuple, use the right part of it diff --git a/rpython/rlib/test/test_rpath.py b/rpython/rlib/test/test_rpath.py --- a/rpython/rlib/test/test_rpath.py +++ b/rpython/rlib/test/test_rpath.py @@ -68,8 +68,8 @@ assert rpath._nt_rabspath('d:\\foo\\bar\\..') == 'd:\\foo' assert rpath._nt_rabspath('d:\\foo\\bar\\..\\x') == 'd:\\foo\\x' curdrive = _ = rpath._nt_rsplitdrive(os.getcwd()) - assert len(curdrive) == 2 and curdrive[1] == ':' - assert rpath.rabspath('\\foo') == '%s\\foo' % curdrive + assert len(curdrive) == 2 and curdrive[0][1] == ':' + assert rpath.rabspath('\\foo') == '%s\\foo' % curdrive[0] def test_risabs_posix(): assert rpath._posix_risabs('/foo/bar') From pypy.commits at gmail.com Thu Jan 21 04:10:00 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 01:10:00 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: made register r13 dirty before flushing the non volatile registers to the stack, Message-ID: <56a0a068.2457c20a.a069d.2ac2@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81884:f5ab3026a4db Date: 2016-01-21 10:08 +0100 http://bitbucket.org/pypy/pypy/changeset/f5ab3026a4db/ Log: made register r13 dirty before flushing the non volatile registers to the stack, it seems that this cannot happen in the test suite because it is either ffi/ctypes that restores a constant r13 after the call and the translated tests might not need r13 after finishing the jit code diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -633,7 +633,6 @@ looptoken, clt.allgcrefs) self.pool.pre_assemble(self, operations) entrypos = self.mc.get_relative_pos() - self.mc.LARL(r.POOL, l.halfword(self.pool.pool_start - entrypos)) self._call_header_with_stack_check() looppos = self.mc.get_relative_pos() frame_depth_no_fixed_size = self._assemble(regalloc, inputargs, @@ -1000,6 +999,7 @@ def _call_header(self): # Build a new stackframe of size STD_FRAME_SIZE_IN_BYTES self.mc.STMG(r.r6, r.r15, l.addr(6*WORD, r.SP)) + self.mc.LARL(r.POOL, l.halfword(self.pool.pool_start - self.mc.get_relative_pos())) # save the back chain self.mc.STG(r.SP, l.addr(0, r.SP)) From pypy.commits at gmail.com Thu Jan 21 05:25:26 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 02:25:26 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: it can happen that longevity does not contain an entry for an operation (e.g. int_mul_ovf and result is not used), then when trying to spill a variable op can be in reg_bindings, but is not in longevity -> KeyError, Message-ID: <56a0b216.022f1c0a.31567.ffffb0f2@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81885:7ac200cdeecd Date: 2016-01-21 11:24 +0100 http://bitbucket.org/pypy/pypy/changeset/7ac200cdeecd/ Log: it can happen that longevity does not contain an entry for an operation (e.g. int_mul_ovf and result is not used), then when trying to spill a variable op can be in reg_bindings, but is not in longevity -> KeyError, fixed this by ensuring that the pair allocation happens at the latest point in the regalloc step diff --git a/rpython/jit/backend/zarch/helper/regalloc.py b/rpython/jit/backend/zarch/helper/regalloc.py --- a/rpython/jit/backend/zarch/helper/regalloc.py +++ b/rpython/jit/backend/zarch/helper/regalloc.py @@ -51,11 +51,11 @@ a1 = op.getarg(1) if a0.is_constant(): a0, a1 = a1, a0 - lr,lq = self.rm.ensure_even_odd_pair(a0, op, bind_first=False) if check_imm32(a1): l1 = imm(a1.getint()) else: l1 = self.ensure_reg(a1) + lr,lq = self.rm.ensure_even_odd_pair(a0, op, bind_first=False) self.free_op_vars() return [lr, lq, l1] @@ -63,15 +63,14 @@ def f(self, op): a0 = op.getarg(0) a1 = op.getarg(1) + l1 = self.ensure_reg(a1) if isinstance(a0, Const): poolloc = self.ensure_reg(a0) lr,lq = self.rm.ensure_even_odd_pair(a0, op, bind_first=modulus, must_exist=False) self.assembler.mc.LG(lq, poolloc) else: lr,lq = self.rm.ensure_even_odd_pair(a0, op, bind_first=modulus) - l1 = self.ensure_reg(a1) self.free_op_vars() - self.rm._check_invariants() return [lr, lq, l1] return f diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py --- a/rpython/jit/backend/zarch/regalloc.py +++ b/rpython/jit/backend/zarch/regalloc.py @@ -952,19 +952,19 @@ def prepare_zero_array(self, op): itemsize, ofs, _ = unpack_arraydescr(op.getdescr()) + startindex_loc = self.ensure_reg_or_16bit_imm(op.getarg(1)) + tempvar = TempInt() + self.rm.temp_boxes.append(tempvar) + ofs_loc = self.ensure_reg_or_16bit_imm(ConstInt(ofs)) + pad_byte, _ = self.rm.ensure_even_odd_pair(tempvar, tempvar, + bind_first=True, must_exist=False, move_regs=False) base_loc, length_loc = self.rm.ensure_even_odd_pair(op.getarg(0), op, bind_first=True, must_exist=False, load_loc_odd=False) - tempvar = TempInt() - self.rm.temp_boxes.append(tempvar) - pad_byte, _ = self.rm.ensure_even_odd_pair(tempvar, tempvar, - bind_first=True, must_exist=False, move_regs=False) - startindex_loc = self.ensure_reg_or_16bit_imm(op.getarg(1)) length_box = op.getarg(2) ll = self.rm.loc(length_box) if length_loc is not ll: self.assembler.regalloc_mov(ll, length_loc) - ofs_loc = self.ensure_reg_or_16bit_imm(ConstInt(ofs)) return [base_loc, startindex_loc, length_loc, ofs_loc, imm(itemsize), pad_byte] def prepare_cond_call(self, op): From pypy.commits at gmail.com Thu Jan 21 14:26:17 2016 From: pypy.commits at gmail.com (sbauman) Date: Thu, 21 Jan 2016 11:26:17 -0800 (PST) Subject: [pypy-commit] pypy remove-getfield-pure: Cleanup based on suggestions from fijal Message-ID: <56a130d9.c6e01c0a.6e68b.6111@mx.google.com> Author: Spenser Bauman Branch: remove-getfield-pure Changeset: r81886:3a92e4541f68 Date: 2016-01-21 14:13 -0500 http://bitbucket.org/pypy/pypy/changeset/3a92e4541f68/ Log: Cleanup based on suggestions from fijal diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py b/rpython/jit/metainterp/optimizeopt/optimizer.py --- a/rpython/jit/metainterp/optimizeopt/optimizer.py +++ b/rpython/jit/metainterp/optimizeopt/optimizer.py @@ -5,7 +5,7 @@ ConstIntBound, MININT, MAXINT, IntUnbounded from rpython.jit.metainterp.optimizeopt.util import make_dispatcher_method from rpython.jit.metainterp.resoperation import rop, AbstractResOp, GuardResOp,\ - OpHelpers, ResOperation, is_pure_getfield + OpHelpers, ResOperation from rpython.jit.metainterp.optimizeopt import info from rpython.jit.metainterp.typesystem import llhelper from rpython.rlib.objectmodel import specialize, we_are_translated @@ -756,7 +756,7 @@ opnum = op.getopnum() cpu = self.cpu - if is_pure_getfield(opnum, op.getdescr()): + if OpHelpers.is_pure_getfield(opnum, op.getdescr()): fielddescr = op.getdescr() ref = self.get_constant_box(op.getarg(0)).getref_base() cpu.protect_speculative_field(ref, fielddescr) diff --git a/rpython/jit/metainterp/pyjitpl.py b/rpython/jit/metainterp/pyjitpl.py --- a/rpython/jit/metainterp/pyjitpl.py +++ b/rpython/jit/metainterp/pyjitpl.py @@ -2095,21 +2095,7 @@ profiler = self.staticdata.profiler profiler.count_ops(opnum) resvalue = executor.execute(self.cpu, self, opnum, descr, *argboxes) - # - is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST - if not is_pure: - # TODO Don't base purity of an operation solely on opnum - if (opnum == rop.GETFIELD_RAW_I or - opnum == rop.GETFIELD_RAW_R or - opnum == rop.GETFIELD_RAW_F or - opnum == rop.GETFIELD_GC_I or - opnum == rop.GETFIELD_GC_R or - opnum == rop.GETFIELD_GC_F or - opnum == rop.GETARRAYITEM_RAW_I or - opnum == rop.GETARRAYITEM_RAW_F): - is_pure = descr.is_always_pure() - # - if is_pure: + if OpHelpers.is_pure_with_descr(opnum, descr): return self._record_helper_pure(opnum, resvalue, descr, *argboxes) if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: return self._record_helper_ovf(opnum, resvalue, descr, *argboxes) diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py --- a/rpython/jit/metainterp/resoperation.py +++ b/rpython/jit/metainterp/resoperation.py @@ -246,11 +246,6 @@ def forget_value(self): pass -def is_pure_getfield(opnum, descr): - if opnum not in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R): - return False - return descr is not None and descr.is_always_pure() - class AbstractResOp(AbstractResOpOrInputArg): """The central ResOperation class, representing one operation.""" @@ -1757,4 +1752,26 @@ opnum = rop.VEC_UNPACK_F return VecOperationNew(opnum, args, datatype, bytesize, signed, count) + @staticmethod + def is_pure_getfield(opnum, descr): + if (opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETFIELD_GC_R): + return descr is not None and descr.is_always_pure() + return False + @staticmethod + def is_pure_with_descr(opnum, descr): + is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST + if not is_pure: + if (opnum == rop.GETFIELD_RAW_I or + opnum == rop.GETFIELD_RAW_R or + opnum == rop.GETFIELD_RAW_F or + opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_R or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETARRAYITEM_RAW_I or + opnum == rop.GETARRAYITEM_RAW_F): + is_pure = descr.is_always_pure() + return is_pure + From pypy.commits at gmail.com Thu Jan 21 14:38:38 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 11:38:38 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: advancing the pointer to correctly return the value written from libffi, the reason why this does not happen on e.g. ppc bigendian 64 bit is: libffi ppc casts the return value to the requested type (s390x does not and we have discussed this enough already) Message-ID: <56a133be.c615c20a.64185.0c66@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81887:08606f22af4a Date: 2016-01-21 20:37 +0100 http://bitbucket.org/pypy/pypy/changeset/08606f22af4a/ Log: advancing the pointer to correctly return the value written from libffi, the reason why this does not happen on e.g. ppc bigendian 64 bit is: libffi ppc casts the return value to the requested type (s390x does not and we have discussed this enough already) diff --git a/pypy/module/_rawffi/interp_rawffi.py b/pypy/module/_rawffi/interp_rawffi.py --- a/pypy/module/_rawffi/interp_rawffi.py +++ b/pypy/module/_rawffi/interp_rawffi.py @@ -1,8 +1,10 @@ +import sys from pypy.interpreter.baseobjspace import W_Root from pypy.interpreter.error import OperationError, oefmt, wrap_oserror from pypy.interpreter.gateway import interp2app, unwrap_spec from pypy.interpreter.typedef import TypeDef, GetSetProperty +from rpython.jit.backend.llsupport.symbolic import WORD from rpython.rlib.clibffi import * from rpython.rtyper.lltypesystem import lltype, rffi from rpython.rtyper.tool import rffi_platform @@ -19,6 +21,8 @@ from pypy.module._rawffi.buffer import RawFFIBuffer from pypy.module._rawffi.tracker import tracker +BIGENDIAN = sys.byteorder == 'big' + TYPEMAP = { # XXX A mess with unsigned/signed/normal chars :-/ 'c' : ffi_type_uchar, @@ -331,10 +335,14 @@ if tracker.DO_TRACING: ll_buf = rffi.cast(lltype.Signed, self.ll_buffer) tracker.trace_allocation(ll_buf, self) + self._ll_buffer = self.ll_buffer def getbuffer(self, space): return space.wrap(rffi.cast(lltype.Unsigned, self.ll_buffer)) + def buffer_advance(self, n): + self.ll_buffer = rffi.ptradd(self.ll_buffer, n) + def byptr(self, space): from pypy.module._rawffi.array import ARRAY_OF_PTRS array = ARRAY_OF_PTRS.allocate(space, 1) @@ -342,16 +350,17 @@ return space.wrap(array) def free(self, space): - if not self.ll_buffer: + if not self._ll_buffer: raise segfault_exception(space, "freeing NULL pointer") self._free() def _free(self): if tracker.DO_TRACING: - ll_buf = rffi.cast(lltype.Signed, self.ll_buffer) + ll_buf = rffi.cast(lltype.Signed, self._ll_buffer) tracker.trace_free(ll_buf) - lltype.free(self.ll_buffer, flavor='raw') + lltype.free(self._ll_buffer, flavor='raw') self.ll_buffer = lltype.nullptr(rffi.VOIDP.TO) + self._ll_buffer = self.ll_buffer def buffer_w(self, space, flags): return RawFFIBuffer(self) @@ -497,6 +506,11 @@ result = self.resshape.allocate(space, 1, autofree=True) # adjust_return_size() was used here on result.ll_buffer self.ptr.call(args_ll, result.ll_buffer) + if BIGENDIAN and result.shape.size < WORD: + # we get a 8 byte value in big endian + n = WORD - result.shape.size + result.buffer_advance(n) + return space.wrap(result) else: self.ptr.call(args_ll, lltype.nullptr(rffi.VOIDP.TO)) From pypy.commits at gmail.com Thu Jan 21 14:54:06 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:06 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: another one, for loop constants Message-ID: <56a1375e.2851c20a.c3793.0fe8@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81889:28a4d2d1f711 Date: 2016-01-21 13:51 +0100 http://bitbucket.org/pypy/pypy/changeset/28a4d2d1f711/ Log: another one, for loop constants diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -625,3 +625,19 @@ i += 1 return 13 """, [1000]) + + def test_no_type_check_for_loopconst(self): + log = self.run(""" + + def main(stop): + a = [1, 2] # always a W_ListObject + i = 0 + while i < stop: + i += isinstance(a, list) # ID: isinstance + return 13 + """, [1000]) + entry_bridge, = log.loops_by_id('isinstance', is_entry_bridge=True) + entry_bridge.match_by_id("isinstance", """ + i1 = int_add(i0, 1) + --TICK-- + """) From pypy.commits at gmail.com Thu Jan 21 14:54:05 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:05 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: add some tests about the new instance features Message-ID: <56a1375d.6adec20a.b3bf0.1330@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81888:d9d8458cbfb3 Date: 2016-01-21 09:23 +0100 http://bitbucket.org/pypy/pypy/changeset/d9d8458cbfb3/ Log: add some tests about the new instance features diff --git a/pypy/module/pypyjit/test_pypy_c/test_instance.py b/pypy/module/pypyjit/test_pypy_c/test_instance.py --- a/pypy/module/pypyjit/test_pypy_c/test_instance.py +++ b/pypy/module/pypyjit/test_pypy_c/test_instance.py @@ -280,3 +280,68 @@ --TICK-- jump(..., descr=...) """) + + def test_mutate_instance_int(self): + def main(): + class A(object): + pass + a = A() + + a.x = 0 + a.upper = 1000 + while a.x < a.upper: + a.x += 1 + return a.x + log = self.run(main, []) + loop, = log.loops_by_filename(self.filepath) + assert loop.match(""" + guard_not_invalidated? + i33 = int_lt(i28, 1000) + guard_true(i33, descr=...) + i34 = int_add(i28, 1) + i35 = getfield_raw_i(..., descr=...) + setfield_gc(p23, i34, descr=...) + i36 = int_lt(i35, 0) + guard_false(i36, descr=...) + jump(..., descr=...) + """) + + def test_instances_class_is_known(self): + def main(): + class A(object): + pass + class B(object): + pass + a1 = A() + a2 = A() + + a1.a = B() + a2.a = B() + i = 0 + while i < 1000: + a1.a.x = i + 0.2 + a1, a2 = a2, a1 + i += 1 + return a1.a.x + a2.a.x + log = self.run(main, []) + loop, = log.loops_by_filename(self.filepath) + assert loop.match(""" + guard_not_invalidated? + i42 = int_lt(i37, 1000) + guard_true(i42, descr=...) + f50 = cast_int_to_float(i37) + f51 = float_add(f50, 0.200000) + p43 = getfield_gc_r(p16, descr=) + guard_value(p43, ConstPtr(ptr31), descr=...) + p44 = getfield_gc_r(p16, descr=) + p45 = getfield_gc_r(p44, descr=) + guard_value(p45, ConstPtr(ptr34), descr=...) + p46 = getfield_gc_r(p44, descr=) + i47 = int_add(i37, 1) + i48 = getfield_raw_i(54402752, descr=...) + setfield_gc(p46, f51, descr=) + i49 = int_lt(i48, 0) + guard_false(i49, descr=...) + jump(..., descr=...) + """) + From pypy.commits at gmail.com Thu Jan 21 14:54:12 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:12 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: document the profiler Message-ID: <56a13764.53ad1c0a.96b67.ffffe528@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81892:6fb5c569a60c Date: 2016-01-21 18:34 +0100 http://bitbucket.org/pypy/pypy/changeset/6fb5c569a60c/ Log: document the profiler diff --git a/pypy/interpreter/valueprof.py b/pypy/interpreter/valueprof.py --- a/pypy/interpreter/valueprof.py +++ b/pypy/interpreter/valueprof.py @@ -8,13 +8,39 @@ SEEN_TOO_MUCH = '?' class ValueProf(object): + """ Some reusal heap profiling infrastructure. Can be used either as a base + class, or as a mixin. + + The idea of this class is to have one ValueProf instance for many heap + storage cells that are likely to store the same content. An example is + having a ValueProf per field of a specific class. """ + _immutable_fields_ = ['_vprof_status?'] def __init__(self, msg=''): # only if you subclass normally self.init_valueprof(msg) + + # ________________________________________________________________________ + # abstract methods that need to be overridden: + + def is_int(self, w_obj): + """ returns whether the argument is a boxed integer. """ + raise NotImplementedError("abstract base") + + def get_int_val(self, w_obj): + """ w_obj must be a boxed integer. returns the unboxed value of that + integer. """ + raise NotImplementedError("abstract base") + + + # ________________________________________________________________________ + # public interface + def init_valueprof(self, msg=''): + """ initialize the profiler. must be called if ValueProf is used as a + mixin upon construction. """ self._vprof_status = SEEN_NOTHING self._vprof_value_int = 0 self._vprof_value_wref = dead_ref @@ -22,28 +48,6 @@ self._vprof_counter = 0 self._vprof_msg = msg - def is_int(self, w_obj): - raise NotImplementedError("abstract base") - - def get_int_val(self, w_obj): - raise NotImplementedError("abstract base") - - def write_necessary(self, w_value): - status = self._vprof_status - if status == SEEN_TOO_MUCH: - return True - # we must have seen something already, because it only makes sense to - # call write_necessary if there is already a value there - assert not status == SEEN_NOTHING - if status == SEEN_CONSTANT_INT: - return (self.is_int(w_value) and - self.read_constant_int() != self.get_int_val(w_value)) - elif status == SEEN_CONSTANT_OBJ: - prev_obj = self.try_read_constant_obj() - return prev_obj is not w_value - return True - - def see_write(self, w_value): """ inform the value profiler of a write.""" status = self._vprof_status @@ -93,26 +97,61 @@ self._vprof_status = SEEN_TOO_MUCH return + def write_necessary(self, w_value): + """ for an already initialized object check whether writing w_value + into the object is necessary. it is unnecessary if the profiler knows + the value is a constant and that constant is equal to w_value. """ + status = self._vprof_status + if status == SEEN_TOO_MUCH: + return True + # we must have seen something already, because it only makes sense to + # call write_necessary if there is already a value there + assert not status == SEEN_NOTHING + if status == SEEN_CONSTANT_INT: + return (self.is_int(w_value) and + self.read_constant_int() != self.get_int_val(w_value)) + elif status == SEEN_CONSTANT_OBJ: + prev_obj = self.try_read_constant_obj() + return prev_obj is not w_value + return True + def can_fold_read_int(self): + """ returns True if the heap profiler knows that the object stores a + constant integer. """ return self._vprof_status == SEEN_CONSTANT_INT def can_fold_read_obj(self): + """ returns True if the heap profiler knows that the object stores a + constant non-integer object. """ return self._vprof_status == SEEN_CONSTANT_OBJ def class_is_known(self): + """ returns True if the heap profiler knows the class of the stored + object. """ return self._vprof_status == SEEN_CONSTANT_CLASS @jit.elidable def read_constant_int(self): + """ returns the stored constant integer value in unboxed form. this + must only be called directly after having called + self.can_fold_read_int() and that returned True. """ assert self.can_fold_read_int() return self._vprof_value_int @jit.elidable def try_read_constant_obj(self): + """ tries to return the stored constant object. this must only be + called directly after having called self.can_fold_read_obj() and that + returned True. The method may still return False, if the constant + object was garbage collected in the meantime.""" assert self.can_fold_read_obj() return self._vprof_value_wref() @jit.elidable def read_constant_cls(self): + """ returns the class of the stored object. this must only be called + directly after having called self.class_is_known() and that returned + True. The returned class is typically used with + jit.record_exact_class(..., class)""" return self._vprof_const_cls From pypy.commits at gmail.com Thu Jan 21 14:54:14 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:14 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: move the valueprof to rlib Message-ID: <56a13766.552f1c0a.1091.ffffe302@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81893:cfa89ffc2f8a Date: 2016-01-21 18:37 +0100 http://bitbucket.org/pypy/pypy/changeset/cfa89ffc2f8a/ Log: move the valueprof to rlib diff --git a/pypy/objspace/std/mapdict.py b/pypy/objspace/std/mapdict.py --- a/pypy/objspace/std/mapdict.py +++ b/pypy/objspace/std/mapdict.py @@ -4,7 +4,7 @@ from rpython.rlib.rarithmetic import intmask, r_uint from pypy.interpreter.baseobjspace import W_Root -from pypy.interpreter import valueprof +from rpython.rlib import valueprof from pypy.objspace.std.dictmultiobject import ( W_DictMultiObject, DictStrategy, ObjectDictStrategy, BaseKeyIterator, BaseValueIterator, BaseItemIterator, _never_equal_to_string, diff --git a/pypy/interpreter/test/test_valueprof.py b/rpython/rlib/test/test_valueprof.py rename from pypy/interpreter/test/test_valueprof.py rename to rpython/rlib/test/test_valueprof.py --- a/pypy/interpreter/test/test_valueprof.py +++ b/rpython/rlib/test/test_valueprof.py @@ -1,4 +1,4 @@ -from pypy.interpreter.valueprof import * +from rpython.rlib.valueprof import * class Value(object): pass diff --git a/pypy/interpreter/valueprof.py b/rpython/rlib/valueprof.py rename from pypy/interpreter/valueprof.py rename to rpython/rlib/valueprof.py From pypy.commits at gmail.com Thu Jan 21 14:54:08 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:08 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: a test for lists of known types Message-ID: <56a13760.c6e01c0a.6e68b.6989@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81890:09164c7f58aa Date: 2016-01-21 13:58 +0100 http://bitbucket.org/pypy/pypy/changeset/09164c7f58aa/ Log: a test for lists of known types diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py --- a/pypy/module/pypyjit/test_pypy_c/test_containers.py +++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py @@ -269,3 +269,22 @@ loop, = log.loops_by_filename(self.filepath) opnames = log.opnames(loop.allops()) assert opnames.count('new_with_vtable') == 0 + + def test_list_of_known_types(self): + def main(n): + l = [[]] * 1000 + i = 0 + while i < 1000: + # l[i] is not None is always True, because l[i] has known type + # W_ListObject + i += l[i] is not None # ID: typecheck + + log = self.run(main, [1000]) + loop, = log.loops_by_filename(self.filepath) + assert loop.match_by_id("typecheck", """ + i40 = uint_ge(i34, i29) + guard_false(i40, descr=...) + p41 = getarrayitem_gc_r(p31, i34, descr=) + i42 = int_add(i34, 1) + --TICK-- + """) From pypy.commits at gmail.com Thu Jan 21 14:54:15 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:15 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: rename valueprof to heapprof Message-ID: <56a13767.ccaa1c0a.cdb63.ffffedb5@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81894:b6d1c55ce142 Date: 2016-01-21 18:39 +0100 http://bitbucket.org/pypy/pypy/changeset/b6d1c55ce142/ Log: rename valueprof to heapprof diff --git a/pypy/objspace/std/mapdict.py b/pypy/objspace/std/mapdict.py --- a/pypy/objspace/std/mapdict.py +++ b/pypy/objspace/std/mapdict.py @@ -4,7 +4,7 @@ from rpython.rlib.rarithmetic import intmask, r_uint from pypy.interpreter.baseobjspace import W_Root -from rpython.rlib import valueprof +from rpython.rlib import heapprof from pypy.objspace.std.dictmultiobject import ( W_DictMultiObject, DictStrategy, ObjectDictStrategy, BaseKeyIterator, BaseValueIterator, BaseItemIterator, _never_equal_to_string, @@ -335,7 +335,7 @@ class PlainAttribute(AbstractAttribute): _immutable_fields_ = ['name', 'index', 'storageindex', 'back', 'ever_mutated?', 'can_contain_mutable_cell?'] - objectmodel.import_from_mixin(valueprof.ValueProf) + objectmodel.import_from_mixin(heapprof.HeapProf) def __init__(self, name, index, back): AbstractAttribute.__init__(self, back.space, back.terminator) @@ -345,7 +345,7 @@ self.back = back self._size_estimate = self.length() * NUM_DIGITS_POW2 self.ever_mutated = False - self.init_valueprof('%s.%s' % (back.terminator.w_cls.name if back.terminator.w_cls else '???', name)) + self.init_heapprof('%s.%s' % (back.terminator.w_cls.name if back.terminator.w_cls else '???', name)) # this flag means: at some point there was an instance that used a # derivative of this map that had a MutableCell stored into the # corresponding field. @@ -360,7 +360,7 @@ return self._read_cell(result) # ____________________________________________________________ - # methods for ValueProf mixin + # methods for HeapProf mixin def is_int(self, w_obj): from pypy.objspace.std.intobject import W_IntObject return type(w_obj) is W_IntObject diff --git a/rpython/rlib/valueprof.py b/rpython/rlib/heapprof.py rename from rpython/rlib/valueprof.py rename to rpython/rlib/heapprof.py --- a/rpython/rlib/valueprof.py +++ b/rpython/rlib/heapprof.py @@ -7,19 +7,19 @@ SEEN_CONSTANT_CLASS = 'c' SEEN_TOO_MUCH = '?' -class ValueProf(object): +class HeapProf(object): """ Some reusal heap profiling infrastructure. Can be used either as a base class, or as a mixin. - The idea of this class is to have one ValueProf instance for many heap + The idea of this class is to have one HeapProf instance for many heap storage cells that are likely to store the same content. An example is - having a ValueProf per field of a specific class. """ + having a HeapProf per field of a specific class. """ - _immutable_fields_ = ['_vprof_status?'] + _immutable_fields_ = ['_hprof_status?'] def __init__(self, msg=''): # only if you subclass normally - self.init_valueprof(msg) + self.init_heapprof(msg) # ________________________________________________________________________ @@ -38,70 +38,70 @@ # ________________________________________________________________________ # public interface - def init_valueprof(self, msg=''): - """ initialize the profiler. must be called if ValueProf is used as a + def init_heapprof(self, msg=''): + """ initialize the profiler. must be called if HeapProf is used as a mixin upon construction. """ - self._vprof_status = SEEN_NOTHING - self._vprof_value_int = 0 - self._vprof_value_wref = dead_ref - self._vprof_const_cls = None - self._vprof_counter = 0 - self._vprof_msg = msg + self._hprof_status = SEEN_NOTHING + self._hprof_value_int = 0 + self._hprof_value_wref = dead_ref + self._hprof_const_cls = None + self._hprof_counter = 0 + self._hprof_msg = msg def see_write(self, w_value): """ inform the value profiler of a write.""" - status = self._vprof_status + status = self._hprof_status if status == SEEN_TOO_MUCH: return if w_value is None: - self._vprof_status = SEEN_TOO_MUCH + self._hprof_status = SEEN_TOO_MUCH return if status == SEEN_NOTHING: if self.is_int(w_value): - self._vprof_value_int = self.get_int_val(w_value) - self._vprof_status = SEEN_CONSTANT_INT + self._hprof_value_int = self.get_int_val(w_value) + self._hprof_status = SEEN_CONSTANT_INT else: try: - self._vprof_value_wref = ref(w_value) + self._hprof_value_wref = ref(w_value) except TypeError: # for tests, which really use unwrapped ints in a few places - self._vprof_status = SEEN_TOO_MUCH + self._hprof_status = SEEN_TOO_MUCH else: - self._vprof_const_cls = w_value.__class__ - self._vprof_status = SEEN_CONSTANT_OBJ + self._hprof_const_cls = w_value.__class__ + self._hprof_status = SEEN_CONSTANT_OBJ elif status == SEEN_CONSTANT_INT: if self.is_int(w_value): if self.read_constant_int() != self.get_int_val(w_value): - self._vprof_status = SEEN_CONSTANT_CLASS - self._vprof_const_cls = w_value.__class__ + self._hprof_status = SEEN_CONSTANT_CLASS + self._hprof_const_cls = w_value.__class__ else: return else: - self._vprof_status = SEEN_TOO_MUCH + self._hprof_status = SEEN_TOO_MUCH elif status == SEEN_CONSTANT_OBJ: prev_obj = self.try_read_constant_obj() if prev_obj is not w_value: prev_cls = self.read_constant_cls() if prev_cls is w_value.__class__: - self._vprof_const_cls = prev_cls - self._vprof_status = SEEN_CONSTANT_CLASS + self._hprof_const_cls = prev_cls + self._hprof_status = SEEN_CONSTANT_CLASS else: - self._vprof_status = SEEN_TOO_MUCH + self._hprof_status = SEEN_TOO_MUCH else: return elif status == SEEN_CONSTANT_CLASS: cls = self.read_constant_cls() if cls is not w_value.__class__: - self._vprof_status = SEEN_TOO_MUCH + self._hprof_status = SEEN_TOO_MUCH return def write_necessary(self, w_value): """ for an already initialized object check whether writing w_value into the object is necessary. it is unnecessary if the profiler knows the value is a constant and that constant is equal to w_value. """ - status = self._vprof_status + status = self._hprof_status if status == SEEN_TOO_MUCH: return True # we must have seen something already, because it only makes sense to @@ -118,17 +118,17 @@ def can_fold_read_int(self): """ returns True if the heap profiler knows that the object stores a constant integer. """ - return self._vprof_status == SEEN_CONSTANT_INT + return self._hprof_status == SEEN_CONSTANT_INT def can_fold_read_obj(self): """ returns True if the heap profiler knows that the object stores a constant non-integer object. """ - return self._vprof_status == SEEN_CONSTANT_OBJ + return self._hprof_status == SEEN_CONSTANT_OBJ def class_is_known(self): """ returns True if the heap profiler knows the class of the stored object. """ - return self._vprof_status == SEEN_CONSTANT_CLASS + return self._hprof_status == SEEN_CONSTANT_CLASS @jit.elidable def read_constant_int(self): @@ -136,7 +136,7 @@ must only be called directly after having called self.can_fold_read_int() and that returned True. """ assert self.can_fold_read_int() - return self._vprof_value_int + return self._hprof_value_int @jit.elidable def try_read_constant_obj(self): @@ -145,7 +145,7 @@ returned True. The method may still return False, if the constant object was garbage collected in the meantime.""" assert self.can_fold_read_obj() - return self._vprof_value_wref() + return self._hprof_value_wref() @jit.elidable def read_constant_cls(self): @@ -153,5 +153,5 @@ directly after having called self.class_is_known() and that returned True. The returned class is typically used with jit.record_exact_class(..., class)""" - return self._vprof_const_cls + return self._hprof_const_cls diff --git a/rpython/rlib/test/test_valueprof.py b/rpython/rlib/test/test_heapprof.py rename from rpython/rlib/test/test_valueprof.py rename to rpython/rlib/test/test_heapprof.py --- a/rpython/rlib/test/test_valueprof.py +++ b/rpython/rlib/test/test_heapprof.py @@ -1,4 +1,4 @@ -from rpython.rlib.valueprof import * +from rpython.rlib.heapprof import * class Value(object): pass @@ -11,7 +11,7 @@ self.intval = val -class ValueProf(ValueProf): +class HeapProf(HeapProf): def is_int(self, val): return isinstance(val, ValueInt) @@ -20,113 +20,113 @@ def test_int(): - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(ValueInt(1)) v.see_write(ValueInt(1)) v.see_write(ValueInt(1)) v.see_write(ValueInt(1)) assert v.read_constant_int() == 1 - assert v._vprof_status == SEEN_CONSTANT_INT + assert v._hprof_status == SEEN_CONSTANT_INT v.see_write(ValueInt(2)) - assert v._vprof_status == SEEN_CONSTANT_CLASS - assert v._vprof_const_cls is ValueInt + assert v._hprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_const_cls is ValueInt v.see_write(ValueInt(1)) - assert v._vprof_status == SEEN_CONSTANT_CLASS - assert v._vprof_const_cls is ValueInt + assert v._hprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_const_cls is ValueInt v.see_write(ValueInt(2)) - assert v._vprof_status == SEEN_CONSTANT_CLASS - assert v._vprof_const_cls is ValueInt + assert v._hprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_const_cls is ValueInt v.see_write(ValueInt(3)) - assert v._vprof_status == SEEN_CONSTANT_CLASS - assert v._vprof_const_cls is ValueInt + assert v._hprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_const_cls is ValueInt - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(ValueInt(1)) v.see_write(Value()) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH v.see_write(Value()) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH def test_obj(): - v = ValueProf() + v = HeapProf() value = Value() - assert v._vprof_status == SEEN_NOTHING + assert v._hprof_status == SEEN_NOTHING v.see_write(value) v.see_write(value) v.see_write(value) v.see_write(value) assert v.try_read_constant_obj() is value - assert v._vprof_status == SEEN_CONSTANT_OBJ + assert v._hprof_status == SEEN_CONSTANT_OBJ v.see_write(ValueInt(2)) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(Value()) v.see_write(OtherValue()) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH def test_none(): - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(None) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH v.see_write(None) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH - v = ValueProf() + v = HeapProf() v.see_write(ValueInt(1)) - assert v._vprof_status == SEEN_CONSTANT_INT + assert v._hprof_status == SEEN_CONSTANT_INT v.see_write(None) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH - v = ValueProf() + v = HeapProf() v.see_write(Value()) - assert v._vprof_status == SEEN_CONSTANT_OBJ + assert v._hprof_status == SEEN_CONSTANT_OBJ v.see_write(None) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH def test_known_class(): import gc - v = ValueProf() + v = HeapProf() value = Value() - assert v._vprof_status == SEEN_NOTHING + assert v._hprof_status == SEEN_NOTHING v.see_write(value) - assert v._vprof_status == SEEN_CONSTANT_OBJ + assert v._hprof_status == SEEN_CONSTANT_OBJ v.see_write(Value()) - assert v._vprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_status == SEEN_CONSTANT_CLASS v.see_write(OtherValue()) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(value) - assert v._vprof_status == SEEN_CONSTANT_OBJ + assert v._hprof_status == SEEN_CONSTANT_OBJ v.see_write(Value()) - assert v._vprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_status == SEEN_CONSTANT_CLASS v.see_write(ValueInt(5)) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(Value()) - assert v._vprof_status == SEEN_CONSTANT_OBJ + assert v._hprof_status == SEEN_CONSTANT_OBJ gc.collect() gc.collect() gc.collect() v.see_write(Value()) - assert v._vprof_status == SEEN_CONSTANT_CLASS + assert v._hprof_status == SEEN_CONSTANT_CLASS v.see_write(OtherValue()) - assert v._vprof_status == SEEN_TOO_MUCH + assert v._hprof_status == SEEN_TOO_MUCH def test_write_necessary_int(): - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING v.see_write(ValueInt(1)) res = v.write_necessary(ValueInt(1)) assert not res @@ -144,8 +144,8 @@ assert res def test_write_not_necessary_obj(): - v = ValueProf() - assert v._vprof_status == SEEN_NOTHING + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING val = Value() v.see_write(val) res = v.write_necessary(val) From pypy.commits at gmail.com Thu Jan 21 14:54:10 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 11:54:10 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: test write_necessary and make see_write return nothing Message-ID: <56a13762.cb571c0a.a54d.ffffe8a0@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81891:5812c89de335 Date: 2016-01-21 18:21 +0100 http://bitbucket.org/pypy/pypy/changeset/5812c89de335/ Log: test write_necessary and make see_write return nothing diff --git a/pypy/interpreter/test/test_valueprof.py b/pypy/interpreter/test/test_valueprof.py --- a/pypy/interpreter/test/test_valueprof.py +++ b/pypy/interpreter/test/test_valueprof.py @@ -124,37 +124,38 @@ v.see_write(OtherValue()) assert v._vprof_status == SEEN_TOO_MUCH -def test_write_not_necessary_int(): +def test_write_necessary_int(): v = ValueProf() assert v._vprof_status == SEEN_NOTHING - res = v.see_write(ValueInt(1)) + v.see_write(ValueInt(1)) + res = v.write_necessary(ValueInt(1)) assert not res - res = v.see_write(ValueInt(1)) - assert res - res = v.see_write(ValueInt(1)) + v.see_write(ValueInt(1)) + res = v.write_necessary(ValueInt(1)) + assert not res + res = v.see_write(ValueInt(2)) + res = v.write_necessary(ValueInt(1)) assert res res = v.see_write(ValueInt(2)) - assert not res - res = v.see_write(ValueInt(2)) - assert not res + res = v.write_necessary(ValueInt(1)) + assert res res = v.see_write(Value()) - assert not res + res = v.write_necessary(ValueInt(1)) + assert res def test_write_not_necessary_obj(): v = ValueProf() assert v._vprof_status == SEEN_NOTHING val = Value() - res = v.see_write(val) + v.see_write(val) + res = v.write_necessary(val) assert not res - res = v.see_write(val) + v.see_write(val) + res = v.write_necessary(val) + assert not res + v.see_write(ValueInt(1)) + res = v.write_necessary(ValueInt(1)) assert res - res = v.see_write(val) + v.see_write(Value()) + res = v.write_necessary(Value()) assert res - res = v.see_write(ValueInt(1)) - assert not res - res = v.see_write(ValueInt(2)) - assert not res - res = v.see_write(ValueInt(2)) - assert not res - res = v.see_write(Value()) - assert not res diff --git a/pypy/interpreter/valueprof.py b/pypy/interpreter/valueprof.py --- a/pypy/interpreter/valueprof.py +++ b/pypy/interpreter/valueprof.py @@ -45,17 +45,14 @@ def see_write(self, w_value): - """ inform the value profiler of a write. returns False, unless the - value is known to be a constant, and w_value that constant (in that - case the caller can elide the write to the actual object, if that - object already stores a value). """ + """ inform the value profiler of a write.""" status = self._vprof_status if status == SEEN_TOO_MUCH: - return False + return if w_value is None: self._vprof_status = SEEN_TOO_MUCH - return False + return if status == SEEN_NOTHING: if self.is_int(w_value): @@ -76,7 +73,7 @@ self._vprof_status = SEEN_CONSTANT_CLASS self._vprof_const_cls = w_value.__class__ else: - return True + return else: self._vprof_status = SEEN_TOO_MUCH elif status == SEEN_CONSTANT_OBJ: @@ -89,12 +86,12 @@ else: self._vprof_status = SEEN_TOO_MUCH else: - return True + return elif status == SEEN_CONSTANT_CLASS: cls = self.read_constant_cls() if cls is not w_value.__class__: self._vprof_status = SEEN_TOO_MUCH - return False + return def can_fold_read_int(self): return self._vprof_status == SEEN_CONSTANT_INT From pypy.commits at gmail.com Thu Jan 21 15:05:02 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 12:05:02 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: excluding structs from buffer advance (which would be wrong) Message-ID: <56a139ee.05bd1c0a.49f0d.ffff863d@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81895:60aa103a575b Date: 2016-01-21 21:04 +0100 http://bitbucket.org/pypy/pypy/changeset/60aa103a575b/ Log: excluding structs from buffer advance (which would be wrong) fixed wide_char test (tested little endian) diff --git a/pypy/module/_rawffi/interp_rawffi.py b/pypy/module/_rawffi/interp_rawffi.py --- a/pypy/module/_rawffi/interp_rawffi.py +++ b/pypy/module/_rawffi/interp_rawffi.py @@ -506,7 +506,7 @@ result = self.resshape.allocate(space, 1, autofree=True) # adjust_return_size() was used here on result.ll_buffer self.ptr.call(args_ll, result.ll_buffer) - if BIGENDIAN and result.shape.size < WORD: + if BIGENDIAN and result.shape.itemcode in ('c','h','i','C','H','I'): # we get a 8 byte value in big endian n = WORD - result.shape.size result.buffer_advance(n) diff --git a/pypy/module/_rawffi/test/test__rawffi.py b/pypy/module/_rawffi/test/test__rawffi.py --- a/pypy/module/_rawffi/test/test__rawffi.py +++ b/pypy/module/_rawffi/test/test__rawffi.py @@ -896,11 +896,21 @@ b = _rawffi.Array('c').fromaddress(a.buffer, 38) if sys.maxunicode > 65535: # UCS4 build - assert b[0] == 'x' - assert b[1] == '\x00' - assert b[2] == '\x00' - assert b[3] == '\x00' - assert b[4] == 'y' + if sys.byteorder == 'big': + assert b[0] == '\x00' + assert b[1] == '\x00' + assert b[2] == '\x00' + assert b[3] == 'x' + assert b[4] == '\x00' + assert b[5] == '\x00' + assert b[6] == '\x00' + assert b[7] == 'y' + else: + assert b[0] == 'x' + assert b[1] == '\x00' + assert b[2] == '\x00' + assert b[3] == '\x00' + assert b[4] == 'y' else: # UCS2 build assert b[0] == 'x' From pypy.commits at gmail.com Thu Jan 21 15:50:12 2016 From: pypy.commits at gmail.com (cfbolz) Date: Thu, 21 Jan 2016 12:50:12 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: fix translation by killing dead import Message-ID: <56a14484.6918c20a.261a0.2089@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81896:cf041c3153dd Date: 2016-01-21 21:49 +0100 http://bitbucket.org/pypy/pypy/changeset/cf041c3153dd/ Log: fix translation by killing dead import diff --git a/pypy/interpreter/pycode.py b/pypy/interpreter/pycode.py --- a/pypy/interpreter/pycode.py +++ b/pypy/interpreter/pycode.py @@ -6,7 +6,7 @@ import dis, imp, struct, types, new, sys, os -from pypy.interpreter import eval, valueprof +from pypy.interpreter import eval from pypy.interpreter.signature import Signature from pypy.interpreter.error import OperationError from pypy.interpreter.gateway import unwrap_spec From pypy.commits at gmail.com Thu Jan 21 16:05:56 2016 From: pypy.commits at gmail.com (sbauman) Date: Thu, 21 Jan 2016 13:05:56 -0800 (PST) Subject: [pypy-commit] pypy default: Merge remove-getfield-pure Message-ID: <56a14834.4f911c0a.60406.0539@mx.google.com> Author: Spenser Bauman Branch: Changeset: r81897:381c403794db Date: 2016-01-21 14:45 -0500 http://bitbucket.org/pypy/pypy/changeset/381c403794db/ Log: Merge remove-getfield-pure diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -83,9 +83,9 @@ p38 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p39 = getfield_gc_r(p38, descr=) i40 = force_token() - p41 = getfield_gc_pure_r(p38, descr=) + p41 = getfield_gc_r(p38, descr=) guard_value(p41, ConstPtr(ptr42), descr=...) - i42 = getfield_gc_pure_i(p38, descr=) + i42 = getfield_gc_i(p38, descr=) i43 = int_is_zero(i42) guard_true(i43, descr=...) i50 = force_token() @@ -435,21 +435,21 @@ guard_isnull(p5, descr=...) guard_nonnull_class(p12, ConstClass(W_IntObject), descr=...) guard_value(p2, ConstPtr(ptr21), descr=...) - i22 = getfield_gc_pure_i(p12, descr=) + i22 = getfield_gc_i(p12, descr=) i24 = int_lt(i22, 5000) guard_true(i24, descr=...) guard_not_invalidated(descr=...) p29 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p30 = getfield_gc_r(p29, descr=) p31 = force_token() - p32 = getfield_gc_pure_r(p29, descr=) + p32 = getfield_gc_r(p29, descr=) guard_value(p32, ConstPtr(ptr33), descr=...) - i34 = getfield_gc_pure_i(p29, descr=) + i34 = getfield_gc_i(p29, descr=) i35 = int_is_zero(i34) guard_true(i35, descr=...) p37 = getfield_gc_r(ConstPtr(ptr36), descr=) guard_nonnull_class(p37, ConstClass(W_IntObject), descr=...) - i39 = getfield_gc_pure_i(p37, descr=) + i39 = getfield_gc_i(p37, descr=) i40 = int_add_ovf(i22, i39) guard_no_overflow(descr=...) --TICK-- @@ -466,7 +466,7 @@ """, []) loop, = log.loops_by_id('call') assert loop.match(""" - i8 = getfield_gc_pure_i(p6, descr=) + i8 = getfield_gc_i(p6, descr=) i10 = int_lt(i8, 5000) guard_true(i10, descr=...) guard_not_invalidated? diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py --- a/pypy/module/pypyjit/test_pypy_c/test_containers.py +++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py @@ -84,7 +84,7 @@ guard_no_exception(descr=...) p20 = new_with_vtable(descr=...) call_n(ConstClass(_ll_dict_setitem_lookup_done_trampoline), p13, p10, p20, i12, i17, descr=) - setfield_gc(p20, i5, descr=) + setfield_gc(p20, i5, descr=) guard_no_exception(descr=...) i23 = call_i(ConstClass(ll_call_lookup_function), p13, p10, i12, 0, descr=) guard_no_exception(descr=...) @@ -93,7 +93,7 @@ p28 = getfield_gc_r(p13, descr=) p29 = getinteriorfield_gc_r(p28, i23, descr=>) guard_nonnull_class(p29, ConstClass(W_IntObject), descr=...) - i31 = getfield_gc_pure_i(p29, descr=) + i31 = getfield_gc_i(p29, descr=) i32 = int_sub_ovf(i31, i5) guard_no_overflow(descr=...) i34 = int_add_ovf(i32, 1) diff --git a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py --- a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py +++ b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py @@ -101,13 +101,13 @@ loop = log._filter(log.loops[0]) assert loop.match(""" guard_class(p1, #, descr=...) - p4 = getfield_gc_pure_r(p1, descr=) + p4 = getfield_gc_r(p1, descr=) i5 = getfield_gc_i(p0, descr=) - p6 = getfield_gc_pure_r(p4, descr=) - p7 = getfield_gc_pure_r(p6, descr=) + p6 = getfield_gc_r(p4, descr=) + p7 = getfield_gc_r(p6, descr=) guard_class(p7, ConstClass(Float64), descr=...) - i9 = getfield_gc_pure_i(p4, descr=) - i10 = getfield_gc_pure_i(p6, descr=) + i9 = getfield_gc_i(p4, descr=) + i10 = getfield_gc_i(p6, descr=) i12 = int_eq(i10, 61) i14 = int_eq(i10, 60) i15 = int_or(i12, i14) @@ -117,28 +117,28 @@ i18 = float_ne(f16, 0.000000) guard_true(i18, descr=...) guard_nonnull_class(p2, ConstClass(W_BoolBox), descr=...) - i20 = getfield_gc_pure_i(p2, descr=) + i20 = getfield_gc_i(p2, descr=) i21 = int_is_true(i20) guard_false(i21, descr=...) i22 = getfield_gc_i(p0, descr=) - i23 = getfield_gc_pure_i(p1, descr=) + i23 = getfield_gc_i(p1, descr=) guard_true(i23, descr=...) i25 = int_add(i22, 1) - p26 = getfield_gc_pure_r(p0, descr=) - i27 = getfield_gc_pure_i(p1, descr=) + p26 = getfield_gc_r(p0, descr=) + i27 = getfield_gc_i(p1, descr=) i28 = int_is_true(i27) guard_true(i28, descr=...) - i29 = getfield_gc_pure_i(p6, descr=) + i29 = getfield_gc_i(p6, descr=) guard_value(i29, 8, descr=...) i30 = int_add(i5, 8) - i31 = getfield_gc_pure_i(p1, descr=) + i31 = getfield_gc_i(p1, descr=) i32 = int_ge(i25, i31) guard_false(i32, descr=...) p34 = new_with_vtable(descr=...) {{{ - setfield_gc(p34, p1, descr=) + setfield_gc(p34, p1, descr=) setfield_gc(p34, i25, descr=) - setfield_gc(p34, p26, descr=) + setfield_gc(p34, p26, descr=) setfield_gc(p34, i30, descr=) }}} jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_min_max.py b/pypy/module/pypyjit/test_pypy_c/test_min_max.py --- a/pypy/module/pypyjit/test_pypy_c/test_min_max.py +++ b/pypy/module/pypyjit/test_pypy_c/test_min_max.py @@ -54,7 +54,7 @@ i19 = int_add(i11, 1) setfield_gc(p2, i19, descr=...) guard_nonnull_class(p18, ConstClass(W_IntObject), descr=...) - i20 = getfield_gc_pure_i(p18, descr=...) + i20 = getfield_gc_i(p18, descr=...) i21 = int_gt(i20, i14) guard_true(i21, descr=...) jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_misc.py b/pypy/module/pypyjit/test_pypy_c/test_misc.py --- a/pypy/module/pypyjit/test_pypy_c/test_misc.py +++ b/pypy/module/pypyjit/test_pypy_c/test_misc.py @@ -113,7 +113,7 @@ i12 = int_is_true(i4) guard_true(i12, descr=...) guard_not_invalidated(descr=...) - i10p = getfield_gc_pure_i(p10, descr=...) + i10p = getfield_gc_i(p10, descr=...) i10 = int_mul_ovf(2, i10p) guard_no_overflow(descr=...) i14 = int_add_ovf(i13, i10) diff --git a/pypy/module/pypyjit/test_pypy_c/test_string.py b/pypy/module/pypyjit/test_pypy_c/test_string.py --- a/pypy/module/pypyjit/test_pypy_c/test_string.py +++ b/pypy/module/pypyjit/test_pypy_c/test_string.py @@ -82,7 +82,7 @@ strsetitem(p25, 0, i23) p93 = call_r(ConstClass(fromstr), p25, 16, descr=) guard_no_exception(descr=...) - i95 = getfield_gc_pure_i(p93, descr=) + i95 = getfield_gc_i(p93, descr=) i96 = int_gt(i95, #) guard_false(i96, descr=...) i94 = call_i(ConstClass(rbigint._toint_helper), p93, descr=) diff --git a/rpython/jit/backend/llgraph/runner.py b/rpython/jit/backend/llgraph/runner.py --- a/rpython/jit/backend/llgraph/runner.py +++ b/rpython/jit/backend/llgraph/runner.py @@ -152,7 +152,7 @@ self.fieldname = fieldname self.FIELD = getattr(S, fieldname) self.index = heaptracker.get_fielddescr_index_in(S, fieldname) - self._is_pure = S._immutable_field(fieldname) + self._is_pure = S._immutable_field(fieldname) != False def is_always_pure(self): return self._is_pure @@ -608,9 +608,6 @@ p = support.cast_arg(lltype.Ptr(descr.S), p) return support.cast_result(descr.FIELD, getattr(p, descr.fieldname)) - bh_getfield_gc_pure_i = bh_getfield_gc - bh_getfield_gc_pure_r = bh_getfield_gc - bh_getfield_gc_pure_f = bh_getfield_gc bh_getfield_gc_i = bh_getfield_gc bh_getfield_gc_r = bh_getfield_gc bh_getfield_gc_f = bh_getfield_gc diff --git a/rpython/jit/backend/llsupport/descr.py b/rpython/jit/backend/llsupport/descr.py --- a/rpython/jit/backend/llsupport/descr.py +++ b/rpython/jit/backend/llsupport/descr.py @@ -180,7 +180,8 @@ return self.offset def repr_of_descr(self): - return '' % (self.flag, self.name, self.offset) + ispure = " pure" if self._is_pure else "" + return '' % (self.flag, self.name, self.offset, ispure) def get_parent_descr(self): return self.parent_descr @@ -200,7 +201,7 @@ flag = get_type_flag(FIELDTYPE) name = '%s.%s' % (STRUCT._name, fieldname) index_in_parent = heaptracker.get_fielddescr_index_in(STRUCT, fieldname) - is_pure = bool(STRUCT._immutable_field(fieldname)) + is_pure = STRUCT._immutable_field(fieldname) != False fielddescr = FieldDescr(name, offset, size, flag, index_in_parent, is_pure) cachedict = cache.setdefault(STRUCT, {}) diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py --- a/rpython/jit/backend/x86/assembler.py +++ b/rpython/jit/backend/x86/assembler.py @@ -1477,9 +1477,6 @@ genop_getfield_gc_f = _genop_getfield genop_getfield_raw_i = _genop_getfield genop_getfield_raw_f = _genop_getfield - genop_getfield_gc_pure_i = _genop_getfield - genop_getfield_gc_pure_r = _genop_getfield - genop_getfield_gc_pure_f = _genop_getfield def _genop_gc_load(self, op, arglocs, resloc): base_loc, ofs_loc, size_loc, sign_loc = arglocs diff --git a/rpython/jit/metainterp/heapcache.py b/rpython/jit/metainterp/heapcache.py --- a/rpython/jit/metainterp/heapcache.py +++ b/rpython/jit/metainterp/heapcache.py @@ -168,9 +168,6 @@ elif (opnum != rop.GETFIELD_GC_R and opnum != rop.GETFIELD_GC_I and opnum != rop.GETFIELD_GC_F and - opnum != rop.GETFIELD_GC_PURE_R and - opnum != rop.GETFIELD_GC_PURE_I and - opnum != rop.GETFIELD_GC_PURE_F and opnum != rop.PTR_EQ and opnum != rop.PTR_NE and opnum != rop.INSTANCE_PTR_EQ and diff --git a/rpython/jit/metainterp/history.py b/rpython/jit/metainterp/history.py --- a/rpython/jit/metainterp/history.py +++ b/rpython/jit/metainterp/history.py @@ -816,9 +816,6 @@ if 'getfield_gc' in check: assert check.pop('getfield_gc') == 0 check['getfield_gc_i'] = check['getfield_gc_r'] = check['getfield_gc_f'] = 0 - if 'getfield_gc_pure' in check: - assert check.pop('getfield_gc_pure') == 0 - check['getfield_gc_pure_i'] = check['getfield_gc_pure_r'] = check['getfield_gc_pure_f'] = 0 if 'getarrayitem_gc_pure' in check: assert check.pop('getarrayitem_gc_pure') == 0 check['getarrayitem_gc_pure_i'] = check['getarrayitem_gc_pure_r'] = check['getarrayitem_gc_pure_f'] = 0 diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py --- a/rpython/jit/metainterp/optimizeopt/heap.py +++ b/rpython/jit/metainterp/optimizeopt/heap.py @@ -183,6 +183,8 @@ return res def invalidate(self, descr): + if descr.is_always_pure(): + return for opinfo in self.cached_infos: assert isinstance(opinfo, info.AbstractStructPtrInfo) opinfo._fields[descr.get_index()] = None @@ -515,9 +517,14 @@ return pendingfields def optimize_GETFIELD_GC_I(self, op): + descr = op.getdescr() + if descr.is_always_pure() and self.get_constant_box(op.getarg(0)) is not None: + resbox = self.optimizer.constant_fold(op) + self.optimizer.make_constant(op, resbox) + return structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) + cf = self.field_cache(descr) + field = cf.getfield_from_cache(self, structinfo, descr) if field is not None: self.make_equal_to(op, field) return @@ -525,23 +532,10 @@ self.make_nonnull(op.getarg(0)) self.emit_operation(op) # then remember the result of reading the field - structinfo.setfield(op.getdescr(), op.getarg(0), op, optheap=self, cf=cf) + structinfo.setfield(descr, op.getarg(0), op, optheap=self, cf=cf) optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - def optimize_GETFIELD_GC_PURE_I(self, op): - structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) - if field is not None: - self.make_equal_to(op, field) - return - # default case: produce the operation - self.make_nonnull(op.getarg(0)) - self.emit_operation(op) - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_PURE_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_PURE_I - def optimize_SETFIELD_GC(self, op): self.setfield(op) #opnum = OpHelpers.getfield_pure_for_descr(op.getdescr()) @@ -631,12 +625,12 @@ def optimize_QUASIIMMUT_FIELD(self, op): # Pattern: QUASIIMMUT_FIELD(s, descr=QuasiImmutDescr) - # x = GETFIELD_GC_PURE(s, descr='inst_x') + # x = GETFIELD_GC(s, descr='inst_x') # pure # If 's' is a constant (after optimizations) we rely on the rest of the - # optimizations to constant-fold the following getfield_gc_pure. + # optimizations to constant-fold the following pure getfield_gc. # in addition, we record the dependency here to make invalidation work # correctly. - # NB: emitting the GETFIELD_GC_PURE is only safe because the + # NB: emitting the pure GETFIELD_GC is only safe because the # QUASIIMMUT_FIELD is also emitted to make sure the dependency is # registered. structvalue = self.ensure_ptr_info_arg0(op) diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py b/rpython/jit/metainterp/optimizeopt/optimizer.py --- a/rpython/jit/metainterp/optimizeopt/optimizer.py +++ b/rpython/jit/metainterp/optimizeopt/optimizer.py @@ -10,6 +10,7 @@ from rpython.jit.metainterp.typesystem import llhelper from rpython.rlib.objectmodel import specialize, we_are_translated from rpython.rlib.debug import debug_print +from rpython.jit.metainterp.optimize import SpeculativeError @@ -374,6 +375,7 @@ if (box.type == 'i' and box.get_forwarded() and box.get_forwarded().is_constant()): return ConstInt(box.get_forwarded().getint()) + return None #self.ensure_imported(value) def get_newoperations(self): @@ -736,12 +738,64 @@ self.emit_operation(op) def constant_fold(self, op): + self.protect_speculative_operation(op) argboxes = [self.get_constant_box(op.getarg(i)) for i in range(op.numargs())] return execute_nonspec_const(self.cpu, None, op.getopnum(), argboxes, op.getdescr(), op.type) + def protect_speculative_operation(self, op): + """When constant-folding a pure operation that reads memory from + a gcref, make sure that the gcref is non-null and of a valid type. + Otherwise, raise SpeculativeError. This should only occur when + unrolling and optimizing the unrolled loop. Note that if + cpu.supports_guard_gc_type is false, we can't really do this + check at all, but then we don't unroll in that case. + """ + opnum = op.getopnum() + cpu = self.cpu + + if OpHelpers.is_pure_getfield(opnum, op.getdescr()): + fielddescr = op.getdescr() + ref = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_field(ref, fielddescr) + return + + elif (opnum == rop.GETARRAYITEM_GC_PURE_I or + opnum == rop.GETARRAYITEM_GC_PURE_R or + opnum == rop.GETARRAYITEM_GC_PURE_F or + opnum == rop.ARRAYLEN_GC): + arraydescr = op.getdescr() + array = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_array(array, arraydescr) + if opnum == rop.ARRAYLEN_GC: + return + arraylength = cpu.bh_arraylen_gc(array, arraydescr) + + elif (opnum == rop.STRGETITEM or + opnum == rop.STRLEN): + string = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_string(string) + if opnum == rop.STRLEN: + return + arraylength = cpu.bh_strlen(string) + + elif (opnum == rop.UNICODEGETITEM or + opnum == rop.UNICODELEN): + unicode = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_unicode(unicode) + if opnum == rop.UNICODELEN: + return + arraylength = cpu.bh_unicodelen(unicode) + + else: + return + + index = self.get_constant_box(op.getarg(1)).getint() + if not (0 <= index < arraylength): + raise SpeculativeError + def is_virtual(self, op): if op.type == 'r': opinfo = self.getptrinfo(op) diff --git a/rpython/jit/metainterp/optimizeopt/pure.py b/rpython/jit/metainterp/optimizeopt/pure.py --- a/rpython/jit/metainterp/optimizeopt/pure.py +++ b/rpython/jit/metainterp/optimizeopt/pure.py @@ -94,7 +94,6 @@ break else: # all constant arguments: constant-fold away - self.protect_speculative_operation(op) resbox = self.optimizer.constant_fold(op) # note that INT_xxx_OVF is not done from here, and the # overflows in the INT_xxx operations are ignored @@ -119,59 +118,6 @@ if nextop: self.emit_operation(nextop) - def protect_speculative_operation(self, op): - """When constant-folding a pure operation that reads memory from - a gcref, make sure that the gcref is non-null and of a valid type. - Otherwise, raise SpeculativeError. This should only occur when - unrolling and optimizing the unrolled loop. Note that if - cpu.supports_guard_gc_type is false, we can't really do this - check at all, but then we don't unroll in that case. - """ - opnum = op.getopnum() - cpu = self.optimizer.cpu - - if (opnum == rop.GETFIELD_GC_PURE_I or - opnum == rop.GETFIELD_GC_PURE_R or - opnum == rop.GETFIELD_GC_PURE_F): - fielddescr = op.getdescr() - ref = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_field(ref, fielddescr) - return - - elif (opnum == rop.GETARRAYITEM_GC_PURE_I or - opnum == rop.GETARRAYITEM_GC_PURE_R or - opnum == rop.GETARRAYITEM_GC_PURE_F or - opnum == rop.ARRAYLEN_GC): - arraydescr = op.getdescr() - array = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_array(array, arraydescr) - if opnum == rop.ARRAYLEN_GC: - return - arraylength = cpu.bh_arraylen_gc(array, arraydescr) - - elif (opnum == rop.STRGETITEM or - opnum == rop.STRLEN): - string = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_string(string) - if opnum == rop.STRLEN: - return - arraylength = cpu.bh_strlen(string) - - elif (opnum == rop.UNICODEGETITEM or - opnum == rop.UNICODELEN): - unicode = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_unicode(unicode) - if opnum == rop.UNICODELEN: - return - arraylength = cpu.bh_unicodelen(unicode) - - else: - return - - index = self.get_constant_box(op.getarg(1)).getint() - if not (0 <= index < arraylength): - raise SpeculativeError - def getrecentops(self, opnum): if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: opnum = opnum - rop._OVF_FIRST diff --git a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py --- a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py @@ -521,8 +521,8 @@ def test_getfield(self): graph = self.build_dependency(""" [p0, p1] # 0: 1,2,5 - p2 = getfield_gc_r(p0) # 1: 3,5 - p3 = getfield_gc_r(p0) # 2: 4 + p2 = getfield_gc_r(p0, descr=valuedescr) # 1: 3,5 + p3 = getfield_gc_r(p0, descr=valuedescr) # 2: 4 guard_nonnull(p2) [p2] # 3: 4,5 guard_nonnull(p3) [p3] # 4: 5 jump(p0,p2) # 5: @@ -532,10 +532,10 @@ def test_cyclic(self): graph = self.build_dependency(""" [p0, p1, p5, p6, p7, p9, p11, p12] # 0: 1,6 - p13 = getfield_gc_r(p9) # 1: 2,5 + p13 = getfield_gc_r(p9, descr=valuedescr) # 1: 2,5 guard_nonnull(p13) [] # 2: 4,5 - i14 = getfield_gc_i(p9) # 3: 5 - p15 = getfield_gc_r(p13) # 4: 5 + i14 = getfield_gc_i(p9, descr=valuedescr) # 3: 5 + p15 = getfield_gc_r(p13, descr=valuedescr) # 4: 5 guard_class(p15, 14073732) [p1, p0, p9, i14, p15, p13, p5, p6, p7] # 5: 6 jump(p0,p1,p5,p6,p7,p9,p11,p12) # 6: """) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -955,12 +955,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_getfield_gc_1(self): ops = """ [i] - p1 = new_with_vtable(descr=nodesize) - setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + p1 = new_with_vtable(descr=nodesize3) + setfield_gc(p1, i, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) jump(i1) """ expected = """ @@ -969,17 +969,16 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ [i] - jump(5) - """ - self.node.value = 5 + jump(7) + """ self.optimize_loop(ops, expected) def test_getfield_gc_nonpure_2(self): @@ -1343,7 +1342,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -1355,7 +1354,7 @@ expected = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -1597,7 +1596,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -1608,7 +1607,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -5076,7 +5075,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i0 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i0 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i1 = call_pure_i(123, i0, descr=nonwritedescr) finish(i1) """ @@ -5462,15 +5461,15 @@ def test_getarrayitem_gc_pure_not_invalidated(self): ops = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) - i2 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i2 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(i2) jump(p0) """ expected = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) escape_n(i1) jump(p0) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py @@ -1409,12 +1409,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_pure_getfield_gc_1(self): ops = """ [i] p1 = new_with_vtable(descr=nodesize) setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + i1 = getfield_gc_i(p1, descr=valuedescr) jump(i1) """ expected = """ @@ -1423,10 +1423,10 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_pure_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ @@ -1436,20 +1436,20 @@ self.node.value = 5 self.optimize_loop(ops, expected) - def test_getfield_gc_pure_3(self): + def test_pure_getfield_gc_3(self): ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) - p3 = getfield_gc_pure_r(p1, descr=nextdescr) + p3 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p3) jump() """ expected = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) escape_n(p2) jump() @@ -2319,7 +2319,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -2332,7 +2332,7 @@ preamble = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2340,11 +2340,12 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - jump(p1, i1, i2, p3, i3) - """ - expected = """ - [p1, i1, i2, p3, i3] + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] # + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2352,8 +2353,7 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - ifoo = arraylen_gc(p3, descr=arraydescr) # killed by the backend - jump(p1, i1, i2, p3, i3) + jump(p1, i1, i2, p3) """ self.optimize_loop(ops, expected, preamble) @@ -2669,7 +2669,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -2680,7 +2680,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -3302,8 +3302,8 @@ [p8, p11, i24] p26 = new(descr=ssize) setfield_gc(p26, i24, descr=adescr) - i34 = getfield_gc_pure_i(p11, descr=abisdescr) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=abisdescr) + i35 = getfield_gc_i(p26, descr=adescr) i36 = int_add_ovf(i34, i35) guard_no_overflow() [] jump(p8, p11, i35) @@ -3330,8 +3330,8 @@ setfield_gc(p26, i24, descr=adescr) i28 = int_add(i17, 1) setfield_gc(p8, i28, descr=valuedescr) - i34 = getfield_gc_pure_i(p11, descr=valuedescr3) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=valuedescr3) + i35 = getfield_gc_i(p26, descr=adescr) guard_nonnull(p12) [] i36 = int_add_ovf(i34, i35) guard_no_overflow() [] @@ -3522,14 +3522,14 @@ def test_residual_call_does_not_invalidate_immutable_caches(self): ops = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) - i3 = getfield_gc_pure_i(p1, descr=valuedescr3) + i3 = getfield_gc_i(p1, descr=valuedescr3) jump(p1) """ expected_preamble = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) jump(p1, i1) """ @@ -4878,11 +4878,11 @@ def test_add_sub_ovf_virtual_unroll(self): ops = """ [p15] - i886 = getfield_gc_pure_i(p15, descr=valuedescr) + i886 = getfield_gc_i(p15, descr=valuedescr) i888 = int_sub_ovf(i886, 1) guard_no_overflow() [] escape_n(i888) - i4360 = getfield_gc_pure_i(p15, descr=valuedescr) + i4360 = getfield_gc_i(p15, descr=valuedescr) i4362 = int_add_ovf(i4360, 1) guard_no_overflow() [] i4360p = int_sub_ovf(i4362, 1) @@ -4972,18 +4972,16 @@ def test_pure(self): ops = """ [p42] - p53 = getfield_gc_r(ConstPtr(myptr), descr=nextdescr) - p59 = getfield_gc_pure_r(p53, descr=valuedescr) + p53 = getfield_gc_r(ConstPtr(myptr3), descr=nextdescr3) + p59 = getfield_gc_r(p53, descr=valuedescr3) i61 = call_i(1, p59, descr=nonwritedescr) jump(p42) """ expected = """ - [p42, p59] - i61 = call_i(1, p59, descr=nonwritedescr) - jump(p42, p59) - - """ - self.node.value = 5 + [p42] + i61 = call_i(1, 7, descr=nonwritedescr) + jump(p42) + """ self.optimize_loop(ops, expected) def test_complains_getfieldpure_setfield(self): @@ -4992,7 +4990,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) setfield_gc(p1, p3, descr=nextdescr) jump(p3) """ @@ -5002,7 +5000,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5010,7 +5008,7 @@ expected = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5021,7 +5019,7 @@ ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) p3 = escape_r() setfield_gc(p3, p1, descr=nextdescr) jump() @@ -6167,14 +6165,14 @@ def test_bug_unroll_with_immutables(self): ops = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) p1 = new_with_vtable(descr=immut_descr) setfield_gc(p1, 1242, descr=immut_intval) jump(p1) """ preamble = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) jump() """ expected = """ @@ -7229,13 +7227,13 @@ [p0, p1, i0] quasiimmut_field(p0, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ expected = """ [p0, p1, i0] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ @@ -7246,7 +7244,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) escape_n(i1) jump() """ @@ -7298,11 +7296,11 @@ [i0a, i0b] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i2 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7325,11 +7323,11 @@ setfield_gc(p, 421, descr=quasifielddescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p, descr=quasifielddescr) + i1 = getfield_gc_i(p, descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(p, descr=quasifielddescr) + i2 = getfield_gc_i(p, descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7568,7 +7566,7 @@ def test_forced_virtual_pure_getfield(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=valuedescr) + p1 = getfield_gc_r(p0, descr=valuedescr3) jump(p1) """ self.optimize_loop(ops, ops) @@ -7578,7 +7576,7 @@ p1 = new_with_vtable(descr=nodesize3) setfield_gc(p1, p0, descr=valuedescr3) escape_n(p1) - p2 = getfield_gc_pure_r(p1, descr=valuedescr3) + p2 = getfield_gc_r(p1, descr=valuedescr3) escape_n(p2) jump(p0) """ @@ -7852,14 +7850,14 @@ def test_loopinvariant_getarrayitem_gc_pure(self): ops = """ [p9, i1] - i843 = getarrayitem_gc_pure_i(p9, i1, descr=arraydescr) + i843 = getarrayitem_gc_pure_i(p9, i1, descr=arrayimmutdescr) call_n(i843, descr=nonwritedescr) jump(p9, i1) """ expected = """ [p9, i1, i843] call_n(i843, descr=nonwritedescr) - ifoo = arraylen_gc(p9, descr=arraydescr) + ifoo = arraylen_gc(p9, descr=arrayimmutdescr) jump(p9, i1, i843) """ self.optimize_loop(ops, expected) @@ -7868,7 +7866,7 @@ ops = """ [p0] p1 = getfield_gc_r(p0, descr=nextdescr) - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) call_n(p2, descr=nonwritedescr) jump(p0) """ @@ -7883,14 +7881,14 @@ i1 = arraylen_gc(p1, descr=gcarraydescr) i2 = int_ge(i1, 8) guard_true(i2) [] - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) - jump(p2, p1) - """ - expected = """ - [p0, p2, p1] + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) + jump(p1, p2) + """ + expected = """ + [p0, p1, p2] call_n(p2, descr=nonwritedescr) i3 = arraylen_gc(p1, descr=gcarraydescr) # Should be killed by backend - jump(p0, p2, p1) + jump(p0, p1, p2) """ self.optimize_loop(ops, expected, expected_short=short) @@ -8065,7 +8063,7 @@ def test_dont_mixup_equal_boxes(self): ops = """ [p8] - i9 = getfield_gc_pure_i(p8, descr=valuedescr) + i9 = getfield_gc_i(p8, descr=valuedescr3) i10 = int_gt(i9, 0) guard_true(i10) [] i29 = int_lshift(i9, 1) @@ -8160,9 +8158,9 @@ py.test.skip("would be fixed by make heap optimizer aware of virtual setfields") ops = """ [p5, p8] - i9 = getfield_gc_pure_i(p5, descr=valuedescr) + i9 = getfield_gc_i(p5, descr=valuedescr) call_n(i9, descr=nonwritedescr) - i11 = getfield_gc_pure_i(p8, descr=valuedescr) + i11 = getfield_gc_i(p8, descr=valuedescr) i13 = int_add_ovf(i11, 1) guard_no_overflow() [] p22 = new_with_vtable(descr=nodesize) @@ -8201,14 +8199,14 @@ ops = """ [p0] p10 = getfield_gc_r(ConstPtr(myptr), descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(ConstPtr(myptr), ConstPtr(myptr2), descr=otherdescr) + setfield_gc(ConstPtr(myptr), ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8232,14 +8230,14 @@ ops = """ [p0] p10 = getfield_gc_r(p0, descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(p0, ConstPtr(myptr2), descr=otherdescr) + setfield_gc(p0, ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8624,17 +8622,17 @@ [p10] p52 = getfield_gc_r(p10, descr=nextdescr) # inst_storage p54 = getarrayitem_gc_r(p52, 0, descr=arraydescr) - p69 = getfield_gc_pure_r(p54, descr=otherdescr) # inst_w_function + p69 = getfield_gc_r(p54, descr=otherdescr) # inst_w_function quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] p106 = new_with_vtable(descr=nodesize) p108 = new_array(3, descr=arraydescr) p110 = new_with_vtable(descr=nodesize) - setfield_gc(p110, ConstPtr(myptr2), descr=otherdescr) # inst_w_function + setfield_gc(p110, ConstPtr(myptrb), descr=otherdescr) # inst_w_function setarrayitem_gc(p108, 0, p110, descr=arraydescr) setfield_gc(p106, p108, descr=nextdescr) # inst_storage jump(p106) @@ -8650,7 +8648,7 @@ [p69] quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] jump(ConstPtr(myptr)) """ @@ -8852,13 +8850,13 @@ def test_virtual_back_and_forth(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=bdescr) + p1 = getfield_gc_r(p0, descr=nextdescr3) ptemp = new_with_vtable(descr=nodesize) setfield_gc(ptemp, p1, descr=nextdescr) p2 = getfield_gc_r(ptemp, descr=nextdescr) - ix = getarrayitem_gc_pure_i(p2, 0, descr=arraydescr) + ix = getarrayitem_gc_pure_i(p2, 0, descr=arrayimmutdescr) pfoo = getfield_gc_r(ptemp, descr=nextdescr) - guard_value(pfoo, ConstPtr(myarray)) [] + guard_value(pfoo, ConstPtr(immutarray)) [] ifoo = int_add(ix, 13) escape_n(ix) jump(p0) @@ -8888,13 +8886,13 @@ def test_constant_float_pure(self): ops = """ [p0] - f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarraydescr) + f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarrayimmutdescr) guard_value(f0, 1.03) [] jump(p0) """ expected = """ [p0] - ifoo = arraylen_gc(p0, descr=floatarraydescr) + ifoo = arraylen_gc(p0, descr=floatarrayimmutdescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -9102,7 +9100,7 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) jump(NULL, i3) """ @@ -9113,9 +9111,9 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) - jump(ConstPtr(myptr4), i3) + jump(ConstPtr(myptr2), i3) """ py.test.raises(InvalidLoop, self.optimize_loop, ops, ops) @@ -9265,9 +9263,126 @@ guard_value(i1, 5) [] jump() """ - a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE)), 5, zero=True) + a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE3)), 5, zero=True) self.optimize_loop(ops, expected, jump_values=[a]) + def test_large_number_of_immutable_references(self): + ops = """ + [p0] + i0 = getfield_gc_i(p0, descr=bigadescr) + i1 = getfield_gc_i(p0, descr=bigbdescr) + i2 = getfield_gc_i(p0, descr=bigcdescr) + i3 = getfield_gc_i(p0, descr=bigddescr) + i4 = getfield_gc_i(p0, descr=bigedescr) + i5 = getfield_gc_i(p0, descr=bigfdescr) + i6 = getfield_gc_i(p0, descr=biggdescr) + i7 = getfield_gc_i(p0, descr=bighdescr) + i8 = getfield_gc_i(p0, descr=bigidescr) + i9 = getfield_gc_i(p0, descr=bigjdescr) + i10 = getfield_gc_i(p0, descr=bigkdescr) + i11 = getfield_gc_i(p0, descr=bigldescr) + i12 = getfield_gc_i(p0, descr=bigmdescr) + i13 = getfield_gc_i(p0, descr=bigndescr) + i14 = getfield_gc_i(p0, descr=bigodescr) + i15 = getfield_gc_i(p0, descr=bigpdescr) + i16 = getfield_gc_i(p0, descr=bigqdescr) + i17 = getfield_gc_i(p0, descr=bigrdescr) + i18 = getfield_gc_i(p0, descr=bigsdescr) + i19 = getfield_gc_i(p0, descr=bigtdescr) + i20 = getfield_gc_i(p0, descr=bigudescr) + i21 = getfield_gc_i(p0, descr=bigvdescr) + i22 = getfield_gc_i(p0, descr=bigwdescr) + i23 = getfield_gc_i(p0, descr=bigxdescr) + i24 = getfield_gc_i(p0, descr=bigydescr) + i25 = getfield_gc_i(p0, descr=bigzdescr) + i27 = getfield_gc_i(p0, descr=bigbdescr) + i28 = getfield_gc_i(p0, descr=bigcdescr) + i29 = getfield_gc_i(p0, descr=bigddescr) + i30 = getfield_gc_i(p0, descr=bigedescr) + i31 = getfield_gc_i(p0, descr=bigfdescr) + i32 = getfield_gc_i(p0, descr=biggdescr) + i33 = getfield_gc_i(p0, descr=bighdescr) + i34 = getfield_gc_i(p0, descr=bigidescr) + i35 = getfield_gc_i(p0, descr=bigjdescr) + i36 = getfield_gc_i(p0, descr=bigkdescr) + i37 = getfield_gc_i(p0, descr=bigldescr) + i38 = getfield_gc_i(p0, descr=bigmdescr) + i39 = getfield_gc_i(p0, descr=bigndescr) + i40 = getfield_gc_i(p0, descr=bigodescr) + i41 = getfield_gc_i(p0, descr=bigpdescr) + i42 = getfield_gc_i(p0, descr=bigqdescr) + i43 = getfield_gc_i(p0, descr=bigrdescr) + i44 = getfield_gc_i(p0, descr=bigsdescr) + i45 = getfield_gc_i(p0, descr=bigtdescr) + i46 = getfield_gc_i(p0, descr=bigudescr) + i47 = getfield_gc_i(p0, descr=bigvdescr) + i48 = getfield_gc_i(p0, descr=bigwdescr) + i49 = getfield_gc_i(p0, descr=bigxdescr) + i50 = getfield_gc_i(p0, descr=bigydescr) + i51 = getfield_gc_i(p0, descr=bigzdescr) + i26 = getfield_gc_i(p0, descr=bigadescr) + i99 = int_add(i26, i51) + escape_i(i27) + escape_i(i28) + escape_i(i29) + escape_i(i30) + escape_i(i31) + escape_i(i32) + escape_i(i33) + escape_i(i34) + escape_i(i35) + escape_i(i36) + escape_i(i37) + escape_i(i38) + escape_i(i39) + escape_i(i40) + escape_i(i41) + escape_i(i42) + escape_i(i43) + escape_i(i44) + escape_i(i45) + escape_i(i46) + escape_i(i47) + escape_i(i48) + escape_i(i49) + escape_i(i50) + escape_i(i51) + escape_i(i26) + escape_i(i99) + jump(p0) + """ + expected = """ + [p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99] + escape_i(i1) + escape_i(i2) + escape_i(i3) + escape_i(i4) + escape_i(i5) + escape_i(i6) + escape_i(i7) + escape_i(i8) + escape_i(i9) + escape_i(i10) + escape_i(i11) + escape_i(i12) + escape_i(i13) + escape_i(i14) + escape_i(i15) + escape_i(i16) + escape_i(i17) + escape_i(i18) + escape_i(i19) + escape_i(i20) + escape_i(i21) + escape_i(i22) + escape_i(i23) + escape_i(i24) + escape_i(i25) + escape_i(i0) + escape_i(i99) + jump(p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99) + """ + self.optimize_loop(ops, expected) class TestLLtype(OptimizeOptTest, LLtypeMixin): pass diff --git a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py --- a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py @@ -220,16 +220,16 @@ def test_double_getfield_plus_pure(self): loop = """ [p0] - pc = getfield_gc_pure_r(p0, descr=nextdescr) + pc = getfield_gc_r(p0, descr=nextdescr3) escape_n(p0) # that should flush the caches - p1 = getfield_gc_r(pc, descr=nextdescr) - i0 = getfield_gc_i(p1, descr=valuedescr) + p1 = getfield_gc_r(pc, descr=nextdescr3) + i0 = getfield_gc_i(p1, descr=valuedescr3) jump(p0) """ es, loop, preamble = self.optimize(loop) assert len(es.short_boxes) == 4 # both getfields are available as - # well as getfield_gc_pure + # well as getfield_gc def test_p123_anti_nested(self): loop = """ diff --git a/rpython/jit/metainterp/optimizeopt/test/test_util.py b/rpython/jit/metainterp/optimizeopt/test/test_util.py --- a/rpython/jit/metainterp/optimizeopt/test/test_util.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_util.py @@ -1,4 +1,4 @@ -import py, random +import py, random, string from rpython.rlib.debug import debug_print from rpython.rtyper.lltypesystem import lltype, llmemory, rffi @@ -122,7 +122,14 @@ ('value', lltype.Signed), ('next', lltype.Ptr(NODE3)), hints={'immutable': True})) - + + big_fields = [('big' + i, lltype.Signed) for i in string.ascii_lowercase] + BIG = lltype.GcForwardReference() + BIG.become(lltype.GcStruct('BIG', *big_fields, hints={'immutable': True})) + + for field, _ in big_fields: + locals()[field + 'descr'] = cpu.fielddescrof(BIG, field) + node = lltype.malloc(NODE) node.value = 5 node.next = node @@ -133,16 +140,25 @@ node2.parent.parent.typeptr = node_vtable2 node2addr = lltype.cast_opaque_ptr(llmemory.GCREF, node2) myptr = lltype.cast_opaque_ptr(llmemory.GCREF, node) - mynode2 = lltype.malloc(NODE) + mynodeb = lltype.malloc(NODE) myarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(lltype.GcArray(lltype.Signed), 13, zero=True)) - mynode2.parent.typeptr = node_vtable - myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode2) - mynode3 = lltype.malloc(NODE2) - mynode3.parent.parent.typeptr = node_vtable2 + mynodeb.parent.typeptr = node_vtable + myptrb = lltype.cast_opaque_ptr(llmemory.GCREF, mynodeb) + myptr2 = lltype.malloc(NODE2) + myptr2.parent.parent.typeptr = node_vtable2 + myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, myptr2) + nullptr = lltype.nullptr(llmemory.GCREF.TO) + + mynode3 = lltype.malloc(NODE3) + mynode3.parent.typeptr = node_vtable3 + mynode3.value = 7 + mynode3.next = mynode3 myptr3 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode3) # a NODE2 mynode4 = lltype.malloc(NODE3) mynode4.parent.typeptr = node_vtable3 myptr4 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode4) # a NODE3 + + nullptr = lltype.nullptr(llmemory.GCREF.TO) #nodebox2 = InputArgRef(lltype.cast_opaque_ptr(llmemory.GCREF, node2)) nodesize = cpu.sizeof(NODE, node_vtable) @@ -203,7 +219,6 @@ arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed)) int32arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.INT)) int16arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.SHORT)) - floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) float32arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.SingleFloat)) arraydescr_tid = arraydescr.get_type_id() array = lltype.malloc(lltype.GcArray(lltype.Signed), 15, zero=True) @@ -212,6 +227,12 @@ array2ref = lltype.cast_opaque_ptr(llmemory.GCREF, array2) gcarraydescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF)) gcarraydescr_tid = gcarraydescr.get_type_id() + floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) + + arrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed, hints={"immutable": True})) + immutarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(arrayimmutdescr.A, 13, zero=True)) + gcarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF, hints={"immutable": True})) + floatarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Float, hints={"immutable": True})) # a GcStruct not inheriting from OBJECT tpl = lltype.malloc(S, zero=True) @@ -244,7 +265,7 @@ tsize = cpu.sizeof(T, None) cdescr = cpu.fielddescrof(T, 'c') ddescr = cpu.fielddescrof(T, 'd') - arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE))) + arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE3))) U = lltype.GcStruct('U', ('parent', OBJECT), diff --git a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py --- a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py @@ -1103,8 +1103,8 @@ jump(p0) """ self.optimize_bridge(loops, bridge, loops[0], 'Loop0', [self.myptr]) - self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr3]) - self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr3]) + self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr2]) + self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr2]) self.optimize_bridge(loops, loops[0], loops[0], 'Loop0', [self.nullptr]) self.optimize_bridge(loops, loops[1], loops[1], 'Loop1', [self.nullptr]) diff --git a/rpython/jit/metainterp/optimizeopt/virtualize.py b/rpython/jit/metainterp/optimizeopt/virtualize.py --- a/rpython/jit/metainterp/optimizeopt/virtualize.py +++ b/rpython/jit/metainterp/optimizeopt/virtualize.py @@ -188,12 +188,6 @@ optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - # note: the following line does not mean that the two operations are - # completely equivalent, because GETFIELD_GC_PURE is_always_pure(). - optimize_GETFIELD_GC_PURE_I = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_I - def optimize_SETFIELD_GC(self, op): struct = op.getarg(0) opinfo = self.getptrinfo(struct) diff --git a/rpython/jit/metainterp/pyjitpl.py b/rpython/jit/metainterp/pyjitpl.py --- a/rpython/jit/metainterp/pyjitpl.py +++ b/rpython/jit/metainterp/pyjitpl.py @@ -653,46 +653,37 @@ @arguments("box", "descr") def opimpl_getfield_gc_i(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resbox = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_I, fielddescr, box) + return ConstInt(resbox) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_I, box, fielddescr, 'i') + + @arguments("box", "descr") + def opimpl_getfield_gc_f(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resvalue = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_F, fielddescr, box) + return ConstFloat(resvalue) + return self._opimpl_getfield_gc_any_pureornot( + rop.GETFIELD_GC_F, box, fielddescr, 'f') + @arguments("box", "descr") def opimpl_getfield_gc_r(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + val = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_R, fielddescr, box) + return ConstPtr(val) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_R, box, fielddescr, 'r') - @arguments("box", "descr") - def opimpl_getfield_gc_f(self, box, fielddescr): - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_i_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resbox = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_I, fielddescr, box) - return ConstInt(resbox) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_I, box, fielddescr, 'i') - - @arguments("box", "descr") - def opimpl_getfield_gc_f_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resvalue = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_F, fielddescr, box) - return ConstFloat(resvalue) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_r_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - val = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_R, fielddescr, box) - return ConstPtr(val) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_R, box, fielddescr, 'r') + + opimpl_getfield_gc_i_pure = opimpl_getfield_gc_i + opimpl_getfield_gc_r_pure = opimpl_getfield_gc_r + opimpl_getfield_gc_f_pure = opimpl_getfield_gc_f @arguments("box", "box", "descr") def opimpl_getinteriorfield_gc_i(self, array, index, descr): @@ -733,7 +724,7 @@ @arguments("box", "descr", "orgpc") def _opimpl_getfield_gc_greenfield_any(self, box, fielddescr, pc): ginfo = self.metainterp.jitdriver_sd.greenfield_info - opnum = OpHelpers.getfield_pure_for_descr(fielddescr) + opnum = OpHelpers.getfield_for_descr(fielddescr) if (ginfo is not None and fielddescr in ginfo.green_field_descrs and not self._nonstandard_virtualizable(pc, box, fielddescr)): # fetch the result, but consider it as a Const box and don't @@ -2104,17 +2095,7 @@ profiler = self.staticdata.profiler profiler.count_ops(opnum) resvalue = executor.execute(self.cpu, self, opnum, descr, *argboxes) - # - is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST - if not is_pure: - if (opnum == rop.GETFIELD_RAW_I or - opnum == rop.GETFIELD_RAW_R or - opnum == rop.GETFIELD_RAW_F or - opnum == rop.GETARRAYITEM_RAW_I or - opnum == rop.GETARRAYITEM_RAW_F): - is_pure = descr.is_always_pure() - # - if is_pure: + if OpHelpers.is_pure_with_descr(opnum, descr): return self._record_helper_pure(opnum, resvalue, descr, *argboxes) if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: return self._record_helper_ovf(opnum, resvalue, descr, *argboxes) diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py --- a/rpython/jit/metainterp/resoperation.py +++ b/rpython/jit/metainterp/resoperation.py @@ -231,7 +231,7 @@ class AbstractResOpOrInputArg(AbstractValue): _attrs_ = ('_forwarded',) - _forwarded = None # either another resop or OptInfo + _forwarded = None # either another resop or OptInfo def get_forwarded(self): return self._forwarded @@ -412,6 +412,8 @@ return rop._JIT_DEBUG_FIRST <= self.getopnum() <= rop._JIT_DEBUG_LAST def is_always_pure(self): + # Tells whether an operation is pure based solely on the opcode. + # Other operations (e.g. getfield ops) may be pure in some cases are well. return rop._ALWAYS_PURE_FIRST <= self.getopnum() <= rop._ALWAYS_PURE_LAST def has_no_side_effect(self): @@ -434,9 +436,7 @@ return self.opnum in (rop.SAME_AS_I, rop.SAME_AS_F, rop.SAME_AS_R) def is_getfield(self): - return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, - rop.GETFIELD_GC_R, rop.GETFIELD_GC_PURE_I, - rop.GETFIELD_GC_PURE_R, rop.GETFIELD_GC_PURE_F) + return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R) def is_getarrayitem(self): return self.opnum in (rop.GETARRAYITEM_GC_I, rop.GETARRAYITEM_GC_F, @@ -1602,14 +1602,6 @@ return rop.CALL_LOOPINVARIANT_N @staticmethod - def getfield_pure_for_descr(descr): - if descr.is_pointer_field(): - return rop.GETFIELD_GC_PURE_R - elif descr.is_float_field(): - return rop.GETFIELD_GC_PURE_F - return rop.GETFIELD_GC_PURE_I - - @staticmethod def getfield_for_descr(descr): if descr.is_pointer_field(): return rop.GETFIELD_GC_R @@ -1760,4 +1752,26 @@ opnum = rop.VEC_UNPACK_F return VecOperationNew(opnum, args, datatype, bytesize, signed, count) + @staticmethod + def is_pure_getfield(opnum, descr): + if (opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETFIELD_GC_R): + return descr is not None and descr.is_always_pure() + return False + @staticmethod + def is_pure_with_descr(opnum, descr): + is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST + if not is_pure: + if (opnum == rop.GETFIELD_RAW_I or + opnum == rop.GETFIELD_RAW_R or + opnum == rop.GETFIELD_RAW_F or + opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_R or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETARRAYITEM_RAW_I or + opnum == rop.GETARRAYITEM_RAW_F): + is_pure = descr.is_always_pure() + return is_pure + diff --git a/rpython/jit/metainterp/test/test_ajit.py b/rpython/jit/metainterp/test/test_ajit.py --- a/rpython/jit/metainterp/test/test_ajit.py +++ b/rpython/jit/metainterp/test/test_ajit.py @@ -320,7 +320,7 @@ assert res == 252 self.check_trace_count(1) self.check_resops({'jump': 1, 'int_gt': 2, 'int_add': 2, - 'getfield_gc_pure_i': 1, 'int_mul': 1, + 'getfield_gc_i': 1, 'int_mul': 1, 'guard_true': 2, 'int_sub': 2}) def test_loops_are_transient(self): @@ -1405,7 +1405,7 @@ return tup[1] res = self.interp_operations(f, [3, 5]) assert res == 5 - self.check_operations_history(setfield_gc=2, getfield_gc_pure_i=0) + self.check_operations_history(setfield_gc=2, getfield_gc_i=0) def test_oosend_look_inside_only_one(self): class A: @@ -2522,7 +2522,7 @@ if counter > 10: return 7 assert self.meta_interp(build, []) == 7 - self.check_resops(getfield_gc_pure_r=2) + self.check_resops(getfield_gc_r=2) def test_args_becomming_equal(self): myjitdriver = JitDriver(greens = [], reds = ['n', 'i', 'sa', 'a', 'b']) diff --git a/rpython/jit/metainterp/test/test_immutable.py b/rpython/jit/metainterp/test/test_immutable.py --- a/rpython/jit/metainterp/test/test_immutable.py +++ b/rpython/jit/metainterp/test/test_immutable.py @@ -19,7 +19,7 @@ return y.x + 5 res = self.interp_operations(f, [23]) assert res == 28 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=1, int_add=1) + self.check_operations_history(getfield_gc_i=1, int_add=1) def test_fields_subclass(self): class X(object): @@ -41,8 +41,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def f(x, y): # this time, the field 'x' only shows up on subclass 'Y' @@ -50,8 +49,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def test_array(self): class X(object): @@ -66,8 +64,7 @@ return a.y[index] res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_index_error(self): class X(object): @@ -89,8 +86,7 @@ return a.get(index) res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_in_immutable(self): class X(object): @@ -106,8 +102,7 @@ return y.lst[index] + y.y + 5 res = self.interp_operations(f, [23, 0], listops=True) assert res == 23 + 24 + 5 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getfield_gc_pure_i=1, + self.check_operations_history(getfield_gc_r=1, getfield_gc_i=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1, int_add=3) diff --git a/rpython/jit/metainterp/test/test_quasiimmut.py b/rpython/jit/metainterp/test/test_quasiimmut.py --- a/rpython/jit/metainterp/test/test_quasiimmut.py +++ b/rpython/jit/metainterp/test/test_quasiimmut.py @@ -74,7 +74,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -101,7 +101,7 @@ res = self.meta_interp(f, [100, 7], enable_opts="") assert res == 700 # there should be no getfields, even though optimizations are turned off - self.check_resops(guard_not_invalidated=1, getfield_gc=0) + self.check_resops(guard_not_invalidated=1) def test_nonopt_1(self): myjitdriver = JitDriver(greens=[], reds=['x', 'total', 'lst']) @@ -124,8 +124,7 @@ assert f(100, 7) == 721 res = self.meta_interp(f, [100, 7]) assert res == 721 - self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, - getfield_gc_pure_i=2) + self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, getfield_gc_i=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -156,7 +155,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) def test_change_during_tracing_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -208,7 +207,7 @@ assert f(100, 7) == 700 res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=0, getfield_gc=0) + self.check_resops(guard_not_invalidated=0) def test_change_invalidate_reentering(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -234,7 +233,7 @@ assert g(100, 7) == 700707 res = self.meta_interp(g, [100, 7]) assert res == 700707 - self.check_resops(guard_not_invalidated=4, getfield_gc=0) + self.check_resops(guard_not_invalidated=4) def test_invalidate_while_running(self): jitdriver = JitDriver(greens=['foo'], reds=['i', 'total']) @@ -348,7 +347,7 @@ res = self.meta_interp(f, [100, 30]) assert res == 6019 self.check_resops(guard_not_invalidated=8, guard_not_forced=0, - call_may_force=0, getfield_gc=0) + call_may_force=0) def test_list_simple_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -374,8 +373,7 @@ getarrayitem_gc_pure_r=0, getarrayitem_gc_i=0, getarrayitem_gc_r=0, - getfield_gc_i=0, getfield_gc_pure_i=0, - getfield_gc_r=0, getfield_gC_pure_r=0) + getfield_gc_i=0, getfield_gc_r=0) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -405,9 +403,7 @@ assert res == 700 # operations must have been removed by the frontend self.check_resops(getarrayitem_gc_pure_i=0, guard_not_invalidated=1, - getarrayitem_gc_i=0, - getfield_gc=0, getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + getarrayitem_gc_i=0, getfield_gc_i=0, getfield_gc_r=0) def test_list_length_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) diff --git a/rpython/jit/metainterp/test/test_tracingopts.py b/rpython/jit/metainterp/test/test_tracingopts.py --- a/rpython/jit/metainterp/test/test_tracingopts.py +++ b/rpython/jit/metainterp/test/test_tracingopts.py @@ -436,10 +436,10 @@ return p.x[0] + p.x[1] res = self.interp_operations(fn, [7]) assert res == 7 + 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == -7 - 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) def test_heap_caching_and_elidable_function(self): class A: @@ -517,12 +517,12 @@ return a1[0] + a2[0] + gn(a1, a2) res = self.interp_operations(fn, [7]) assert res == 2 * 7 + 2 * 6 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == 2 * -7 + 2 * -8 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) def test_heap_caching_multiple_arrays(self): class Gbl(object): diff --git a/rpython/jit/metainterp/test/test_virtual.py b/rpython/jit/metainterp/test/test_virtual.py --- a/rpython/jit/metainterp/test/test_virtual.py +++ b/rpython/jit/metainterp/test/test_virtual.py @@ -1077,7 +1077,7 @@ res = self.meta_interp(f, [], repeat=7) assert res == f() - def test_getfield_gc_pure_nobug(self): + def test_pure_getfield_gc_nobug(self): mydriver = JitDriver(reds=['i', 's', 'a'], greens=[]) class A(object): From pypy.commits at gmail.com Thu Jan 21 16:05:58 2016 From: pypy.commits at gmail.com (sbauman) Date: Thu, 21 Jan 2016 13:05:58 -0800 (PST) Subject: [pypy-commit] pypy default: Merge latest changes Message-ID: <56a14836.17941c0a.2b56c.063a@mx.google.com> Author: Spenser Bauman Branch: Changeset: r81898:ee7c705360c7 Date: 2016-01-21 16:05 -0500 http://bitbucket.org/pypy/pypy/changeset/ee7c705360c7/ Log: Merge latest changes diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -83,9 +83,9 @@ p38 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p39 = getfield_gc_r(p38, descr=) i40 = force_token() - p41 = getfield_gc_pure_r(p38, descr=) + p41 = getfield_gc_r(p38, descr=) guard_value(p41, ConstPtr(ptr42), descr=...) - i42 = getfield_gc_pure_i(p38, descr=) + i42 = getfield_gc_i(p38, descr=) i43 = int_is_zero(i42) guard_true(i43, descr=...) i50 = force_token() @@ -435,21 +435,21 @@ guard_isnull(p5, descr=...) guard_nonnull_class(p12, ConstClass(W_IntObject), descr=...) guard_value(p2, ConstPtr(ptr21), descr=...) - i22 = getfield_gc_pure_i(p12, descr=) + i22 = getfield_gc_i(p12, descr=) i24 = int_lt(i22, 5000) guard_true(i24, descr=...) guard_not_invalidated(descr=...) p29 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p30 = getfield_gc_r(p29, descr=) p31 = force_token() - p32 = getfield_gc_pure_r(p29, descr=) + p32 = getfield_gc_r(p29, descr=) guard_value(p32, ConstPtr(ptr33), descr=...) - i34 = getfield_gc_pure_i(p29, descr=) + i34 = getfield_gc_i(p29, descr=) i35 = int_is_zero(i34) guard_true(i35, descr=...) p37 = getfield_gc_r(ConstPtr(ptr36), descr=) guard_nonnull_class(p37, ConstClass(W_IntObject), descr=...) - i39 = getfield_gc_pure_i(p37, descr=) + i39 = getfield_gc_i(p37, descr=) i40 = int_add_ovf(i22, i39) guard_no_overflow(descr=...) --TICK-- @@ -466,7 +466,7 @@ """, []) loop, = log.loops_by_id('call') assert loop.match(""" - i8 = getfield_gc_pure_i(p6, descr=) + i8 = getfield_gc_i(p6, descr=) i10 = int_lt(i8, 5000) guard_true(i10, descr=...) guard_not_invalidated? diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py --- a/pypy/module/pypyjit/test_pypy_c/test_containers.py +++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py @@ -84,7 +84,7 @@ guard_no_exception(descr=...) p20 = new_with_vtable(descr=...) call_n(ConstClass(_ll_dict_setitem_lookup_done_trampoline), p13, p10, p20, i12, i17, descr=) - setfield_gc(p20, i5, descr=) + setfield_gc(p20, i5, descr=) guard_no_exception(descr=...) i23 = call_i(ConstClass(ll_call_lookup_function), p13, p10, i12, 0, descr=) guard_no_exception(descr=...) @@ -93,7 +93,7 @@ p28 = getfield_gc_r(p13, descr=) p29 = getinteriorfield_gc_r(p28, i23, descr=>) guard_nonnull_class(p29, ConstClass(W_IntObject), descr=...) - i31 = getfield_gc_pure_i(p29, descr=) + i31 = getfield_gc_i(p29, descr=) i32 = int_sub_ovf(i31, i5) guard_no_overflow(descr=...) i34 = int_add_ovf(i32, 1) diff --git a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py --- a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py +++ b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py @@ -101,13 +101,13 @@ loop = log._filter(log.loops[0]) assert loop.match(""" guard_class(p1, #, descr=...) - p4 = getfield_gc_pure_r(p1, descr=) + p4 = getfield_gc_r(p1, descr=) i5 = getfield_gc_i(p0, descr=) - p6 = getfield_gc_pure_r(p4, descr=) - p7 = getfield_gc_pure_r(p6, descr=) + p6 = getfield_gc_r(p4, descr=) + p7 = getfield_gc_r(p6, descr=) guard_class(p7, ConstClass(Float64), descr=...) - i9 = getfield_gc_pure_i(p4, descr=) - i10 = getfield_gc_pure_i(p6, descr=) + i9 = getfield_gc_i(p4, descr=) + i10 = getfield_gc_i(p6, descr=) i12 = int_eq(i10, 61) i14 = int_eq(i10, 60) i15 = int_or(i12, i14) @@ -117,28 +117,28 @@ i18 = float_ne(f16, 0.000000) guard_true(i18, descr=...) guard_nonnull_class(p2, ConstClass(W_BoolBox), descr=...) - i20 = getfield_gc_pure_i(p2, descr=) + i20 = getfield_gc_i(p2, descr=) i21 = int_is_true(i20) guard_false(i21, descr=...) i22 = getfield_gc_i(p0, descr=) - i23 = getfield_gc_pure_i(p1, descr=) + i23 = getfield_gc_i(p1, descr=) guard_true(i23, descr=...) i25 = int_add(i22, 1) - p26 = getfield_gc_pure_r(p0, descr=) - i27 = getfield_gc_pure_i(p1, descr=) + p26 = getfield_gc_r(p0, descr=) + i27 = getfield_gc_i(p1, descr=) i28 = int_is_true(i27) guard_true(i28, descr=...) - i29 = getfield_gc_pure_i(p6, descr=) + i29 = getfield_gc_i(p6, descr=) guard_value(i29, 8, descr=...) i30 = int_add(i5, 8) - i31 = getfield_gc_pure_i(p1, descr=) + i31 = getfield_gc_i(p1, descr=) i32 = int_ge(i25, i31) guard_false(i32, descr=...) p34 = new_with_vtable(descr=...) {{{ - setfield_gc(p34, p1, descr=) + setfield_gc(p34, p1, descr=) setfield_gc(p34, i25, descr=) - setfield_gc(p34, p26, descr=) + setfield_gc(p34, p26, descr=) setfield_gc(p34, i30, descr=) }}} jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_min_max.py b/pypy/module/pypyjit/test_pypy_c/test_min_max.py --- a/pypy/module/pypyjit/test_pypy_c/test_min_max.py +++ b/pypy/module/pypyjit/test_pypy_c/test_min_max.py @@ -54,7 +54,7 @@ i19 = int_add(i11, 1) setfield_gc(p2, i19, descr=...) guard_nonnull_class(p18, ConstClass(W_IntObject), descr=...) - i20 = getfield_gc_pure_i(p18, descr=...) + i20 = getfield_gc_i(p18, descr=...) i21 = int_gt(i20, i14) guard_true(i21, descr=...) jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_misc.py b/pypy/module/pypyjit/test_pypy_c/test_misc.py --- a/pypy/module/pypyjit/test_pypy_c/test_misc.py +++ b/pypy/module/pypyjit/test_pypy_c/test_misc.py @@ -113,7 +113,7 @@ i12 = int_is_true(i4) guard_true(i12, descr=...) guard_not_invalidated(descr=...) - i10p = getfield_gc_pure_i(p10, descr=...) + i10p = getfield_gc_i(p10, descr=...) i10 = int_mul_ovf(2, i10p) guard_no_overflow(descr=...) i14 = int_add_ovf(i13, i10) diff --git a/pypy/module/pypyjit/test_pypy_c/test_string.py b/pypy/module/pypyjit/test_pypy_c/test_string.py --- a/pypy/module/pypyjit/test_pypy_c/test_string.py +++ b/pypy/module/pypyjit/test_pypy_c/test_string.py @@ -82,7 +82,7 @@ strsetitem(p25, 0, i23) p93 = call_r(ConstClass(fromstr), p25, 16, descr=) guard_no_exception(descr=...) - i95 = getfield_gc_pure_i(p93, descr=) + i95 = getfield_gc_i(p93, descr=) i96 = int_gt(i95, #) guard_false(i96, descr=...) i94 = call_i(ConstClass(rbigint._toint_helper), p93, descr=) diff --git a/rpython/jit/backend/llgraph/runner.py b/rpython/jit/backend/llgraph/runner.py --- a/rpython/jit/backend/llgraph/runner.py +++ b/rpython/jit/backend/llgraph/runner.py @@ -152,7 +152,7 @@ self.fieldname = fieldname self.FIELD = getattr(S, fieldname) self.index = heaptracker.get_fielddescr_index_in(S, fieldname) - self._is_pure = S._immutable_field(fieldname) + self._is_pure = S._immutable_field(fieldname) != False def is_always_pure(self): return self._is_pure @@ -608,9 +608,6 @@ p = support.cast_arg(lltype.Ptr(descr.S), p) return support.cast_result(descr.FIELD, getattr(p, descr.fieldname)) - bh_getfield_gc_pure_i = bh_getfield_gc - bh_getfield_gc_pure_r = bh_getfield_gc - bh_getfield_gc_pure_f = bh_getfield_gc bh_getfield_gc_i = bh_getfield_gc bh_getfield_gc_r = bh_getfield_gc bh_getfield_gc_f = bh_getfield_gc diff --git a/rpython/jit/backend/llsupport/descr.py b/rpython/jit/backend/llsupport/descr.py --- a/rpython/jit/backend/llsupport/descr.py +++ b/rpython/jit/backend/llsupport/descr.py @@ -180,7 +180,8 @@ return self.offset def repr_of_descr(self): - return '' % (self.flag, self.name, self.offset) + ispure = " pure" if self._is_pure else "" + return '' % (self.flag, self.name, self.offset, ispure) def get_parent_descr(self): return self.parent_descr @@ -200,7 +201,7 @@ flag = get_type_flag(FIELDTYPE) name = '%s.%s' % (STRUCT._name, fieldname) index_in_parent = heaptracker.get_fielddescr_index_in(STRUCT, fieldname) - is_pure = bool(STRUCT._immutable_field(fieldname)) + is_pure = STRUCT._immutable_field(fieldname) != False fielddescr = FieldDescr(name, offset, size, flag, index_in_parent, is_pure) cachedict = cache.setdefault(STRUCT, {}) diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py --- a/rpython/jit/backend/x86/assembler.py +++ b/rpython/jit/backend/x86/assembler.py @@ -1477,9 +1477,6 @@ genop_getfield_gc_f = _genop_getfield genop_getfield_raw_i = _genop_getfield genop_getfield_raw_f = _genop_getfield - genop_getfield_gc_pure_i = _genop_getfield - genop_getfield_gc_pure_r = _genop_getfield - genop_getfield_gc_pure_f = _genop_getfield def _genop_gc_load(self, op, arglocs, resloc): base_loc, ofs_loc, size_loc, sign_loc = arglocs diff --git a/rpython/jit/metainterp/heapcache.py b/rpython/jit/metainterp/heapcache.py --- a/rpython/jit/metainterp/heapcache.py +++ b/rpython/jit/metainterp/heapcache.py @@ -168,9 +168,6 @@ elif (opnum != rop.GETFIELD_GC_R and opnum != rop.GETFIELD_GC_I and opnum != rop.GETFIELD_GC_F and - opnum != rop.GETFIELD_GC_PURE_R and - opnum != rop.GETFIELD_GC_PURE_I and - opnum != rop.GETFIELD_GC_PURE_F and opnum != rop.PTR_EQ and opnum != rop.PTR_NE and opnum != rop.INSTANCE_PTR_EQ and diff --git a/rpython/jit/metainterp/history.py b/rpython/jit/metainterp/history.py --- a/rpython/jit/metainterp/history.py +++ b/rpython/jit/metainterp/history.py @@ -816,9 +816,6 @@ if 'getfield_gc' in check: assert check.pop('getfield_gc') == 0 check['getfield_gc_i'] = check['getfield_gc_r'] = check['getfield_gc_f'] = 0 - if 'getfield_gc_pure' in check: - assert check.pop('getfield_gc_pure') == 0 - check['getfield_gc_pure_i'] = check['getfield_gc_pure_r'] = check['getfield_gc_pure_f'] = 0 if 'getarrayitem_gc_pure' in check: assert check.pop('getarrayitem_gc_pure') == 0 check['getarrayitem_gc_pure_i'] = check['getarrayitem_gc_pure_r'] = check['getarrayitem_gc_pure_f'] = 0 diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py --- a/rpython/jit/metainterp/optimizeopt/heap.py +++ b/rpython/jit/metainterp/optimizeopt/heap.py @@ -183,6 +183,8 @@ return res def invalidate(self, descr): + if descr.is_always_pure(): + return for opinfo in self.cached_infos: assert isinstance(opinfo, info.AbstractStructPtrInfo) opinfo._fields[descr.get_index()] = None @@ -515,9 +517,14 @@ return pendingfields def optimize_GETFIELD_GC_I(self, op): + descr = op.getdescr() + if descr.is_always_pure() and self.get_constant_box(op.getarg(0)) is not None: + resbox = self.optimizer.constant_fold(op) + self.optimizer.make_constant(op, resbox) + return structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) + cf = self.field_cache(descr) + field = cf.getfield_from_cache(self, structinfo, descr) if field is not None: self.make_equal_to(op, field) return @@ -525,23 +532,10 @@ self.make_nonnull(op.getarg(0)) self.emit_operation(op) # then remember the result of reading the field - structinfo.setfield(op.getdescr(), op.getarg(0), op, optheap=self, cf=cf) + structinfo.setfield(descr, op.getarg(0), op, optheap=self, cf=cf) optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - def optimize_GETFIELD_GC_PURE_I(self, op): - structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) - if field is not None: - self.make_equal_to(op, field) - return - # default case: produce the operation - self.make_nonnull(op.getarg(0)) - self.emit_operation(op) - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_PURE_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_PURE_I - def optimize_SETFIELD_GC(self, op): self.setfield(op) #opnum = OpHelpers.getfield_pure_for_descr(op.getdescr()) @@ -631,12 +625,12 @@ def optimize_QUASIIMMUT_FIELD(self, op): # Pattern: QUASIIMMUT_FIELD(s, descr=QuasiImmutDescr) - # x = GETFIELD_GC_PURE(s, descr='inst_x') + # x = GETFIELD_GC(s, descr='inst_x') # pure # If 's' is a constant (after optimizations) we rely on the rest of the - # optimizations to constant-fold the following getfield_gc_pure. + # optimizations to constant-fold the following pure getfield_gc. # in addition, we record the dependency here to make invalidation work # correctly. - # NB: emitting the GETFIELD_GC_PURE is only safe because the + # NB: emitting the pure GETFIELD_GC is only safe because the # QUASIIMMUT_FIELD is also emitted to make sure the dependency is # registered. structvalue = self.ensure_ptr_info_arg0(op) diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py b/rpython/jit/metainterp/optimizeopt/optimizer.py --- a/rpython/jit/metainterp/optimizeopt/optimizer.py +++ b/rpython/jit/metainterp/optimizeopt/optimizer.py @@ -10,6 +10,7 @@ from rpython.jit.metainterp.typesystem import llhelper from rpython.rlib.objectmodel import specialize, we_are_translated from rpython.rlib.debug import debug_print +from rpython.jit.metainterp.optimize import SpeculativeError @@ -374,6 +375,7 @@ if (box.type == 'i' and box.get_forwarded() and box.get_forwarded().is_constant()): return ConstInt(box.get_forwarded().getint()) + return None #self.ensure_imported(value) def get_newoperations(self): @@ -736,12 +738,64 @@ self.emit_operation(op) def constant_fold(self, op): + self.protect_speculative_operation(op) argboxes = [self.get_constant_box(op.getarg(i)) for i in range(op.numargs())] return execute_nonspec_const(self.cpu, None, op.getopnum(), argboxes, op.getdescr(), op.type) + def protect_speculative_operation(self, op): + """When constant-folding a pure operation that reads memory from + a gcref, make sure that the gcref is non-null and of a valid type. + Otherwise, raise SpeculativeError. This should only occur when + unrolling and optimizing the unrolled loop. Note that if + cpu.supports_guard_gc_type is false, we can't really do this + check at all, but then we don't unroll in that case. + """ + opnum = op.getopnum() + cpu = self.cpu + + if OpHelpers.is_pure_getfield(opnum, op.getdescr()): + fielddescr = op.getdescr() + ref = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_field(ref, fielddescr) + return + + elif (opnum == rop.GETARRAYITEM_GC_PURE_I or + opnum == rop.GETARRAYITEM_GC_PURE_R or + opnum == rop.GETARRAYITEM_GC_PURE_F or + opnum == rop.ARRAYLEN_GC): + arraydescr = op.getdescr() + array = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_array(array, arraydescr) + if opnum == rop.ARRAYLEN_GC: + return + arraylength = cpu.bh_arraylen_gc(array, arraydescr) + + elif (opnum == rop.STRGETITEM or + opnum == rop.STRLEN): + string = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_string(string) + if opnum == rop.STRLEN: + return + arraylength = cpu.bh_strlen(string) + + elif (opnum == rop.UNICODEGETITEM or + opnum == rop.UNICODELEN): + unicode = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_unicode(unicode) + if opnum == rop.UNICODELEN: + return + arraylength = cpu.bh_unicodelen(unicode) + + else: + return + + index = self.get_constant_box(op.getarg(1)).getint() + if not (0 <= index < arraylength): + raise SpeculativeError + def is_virtual(self, op): if op.type == 'r': opinfo = self.getptrinfo(op) diff --git a/rpython/jit/metainterp/optimizeopt/pure.py b/rpython/jit/metainterp/optimizeopt/pure.py --- a/rpython/jit/metainterp/optimizeopt/pure.py +++ b/rpython/jit/metainterp/optimizeopt/pure.py @@ -94,7 +94,6 @@ break else: # all constant arguments: constant-fold away - self.protect_speculative_operation(op) resbox = self.optimizer.constant_fold(op) # note that INT_xxx_OVF is not done from here, and the # overflows in the INT_xxx operations are ignored @@ -119,59 +118,6 @@ if nextop: self.emit_operation(nextop) - def protect_speculative_operation(self, op): - """When constant-folding a pure operation that reads memory from - a gcref, make sure that the gcref is non-null and of a valid type. - Otherwise, raise SpeculativeError. This should only occur when - unrolling and optimizing the unrolled loop. Note that if - cpu.supports_guard_gc_type is false, we can't really do this - check at all, but then we don't unroll in that case. - """ - opnum = op.getopnum() - cpu = self.optimizer.cpu - - if (opnum == rop.GETFIELD_GC_PURE_I or - opnum == rop.GETFIELD_GC_PURE_R or - opnum == rop.GETFIELD_GC_PURE_F): - fielddescr = op.getdescr() - ref = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_field(ref, fielddescr) - return - - elif (opnum == rop.GETARRAYITEM_GC_PURE_I or - opnum == rop.GETARRAYITEM_GC_PURE_R or - opnum == rop.GETARRAYITEM_GC_PURE_F or - opnum == rop.ARRAYLEN_GC): - arraydescr = op.getdescr() - array = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_array(array, arraydescr) - if opnum == rop.ARRAYLEN_GC: - return - arraylength = cpu.bh_arraylen_gc(array, arraydescr) - - elif (opnum == rop.STRGETITEM or - opnum == rop.STRLEN): - string = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_string(string) - if opnum == rop.STRLEN: - return - arraylength = cpu.bh_strlen(string) - - elif (opnum == rop.UNICODEGETITEM or - opnum == rop.UNICODELEN): - unicode = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_unicode(unicode) - if opnum == rop.UNICODELEN: - return - arraylength = cpu.bh_unicodelen(unicode) - - else: - return - - index = self.get_constant_box(op.getarg(1)).getint() - if not (0 <= index < arraylength): - raise SpeculativeError - def getrecentops(self, opnum): if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: opnum = opnum - rop._OVF_FIRST diff --git a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py --- a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py @@ -521,8 +521,8 @@ def test_getfield(self): graph = self.build_dependency(""" [p0, p1] # 0: 1,2,5 - p2 = getfield_gc_r(p0) # 1: 3,5 - p3 = getfield_gc_r(p0) # 2: 4 + p2 = getfield_gc_r(p0, descr=valuedescr) # 1: 3,5 + p3 = getfield_gc_r(p0, descr=valuedescr) # 2: 4 guard_nonnull(p2) [p2] # 3: 4,5 guard_nonnull(p3) [p3] # 4: 5 jump(p0,p2) # 5: @@ -532,10 +532,10 @@ def test_cyclic(self): graph = self.build_dependency(""" [p0, p1, p5, p6, p7, p9, p11, p12] # 0: 1,6 - p13 = getfield_gc_r(p9) # 1: 2,5 + p13 = getfield_gc_r(p9, descr=valuedescr) # 1: 2,5 guard_nonnull(p13) [] # 2: 4,5 - i14 = getfield_gc_i(p9) # 3: 5 - p15 = getfield_gc_r(p13) # 4: 5 + i14 = getfield_gc_i(p9, descr=valuedescr) # 3: 5 + p15 = getfield_gc_r(p13, descr=valuedescr) # 4: 5 guard_class(p15, 14073732) [p1, p0, p9, i14, p15, p13, p5, p6, p7] # 5: 6 jump(p0,p1,p5,p6,p7,p9,p11,p12) # 6: """) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -955,12 +955,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_getfield_gc_1(self): ops = """ [i] - p1 = new_with_vtable(descr=nodesize) - setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + p1 = new_with_vtable(descr=nodesize3) + setfield_gc(p1, i, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) jump(i1) """ expected = """ @@ -969,17 +969,16 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ [i] - jump(5) - """ - self.node.value = 5 + jump(7) + """ self.optimize_loop(ops, expected) def test_getfield_gc_nonpure_2(self): @@ -1343,7 +1342,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -1355,7 +1354,7 @@ expected = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -1597,7 +1596,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -1608,7 +1607,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -5076,7 +5075,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i0 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i0 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i1 = call_pure_i(123, i0, descr=nonwritedescr) finish(i1) """ @@ -5462,15 +5461,15 @@ def test_getarrayitem_gc_pure_not_invalidated(self): ops = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) - i2 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i2 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(i2) jump(p0) """ expected = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) escape_n(i1) jump(p0) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py @@ -1409,12 +1409,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_pure_getfield_gc_1(self): ops = """ [i] p1 = new_with_vtable(descr=nodesize) setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + i1 = getfield_gc_i(p1, descr=valuedescr) jump(i1) """ expected = """ @@ -1423,10 +1423,10 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_pure_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ @@ -1436,20 +1436,20 @@ self.node.value = 5 self.optimize_loop(ops, expected) - def test_getfield_gc_pure_3(self): + def test_pure_getfield_gc_3(self): ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) - p3 = getfield_gc_pure_r(p1, descr=nextdescr) + p3 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p3) jump() """ expected = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) escape_n(p2) jump() @@ -2319,7 +2319,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -2332,7 +2332,7 @@ preamble = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2340,11 +2340,12 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - jump(p1, i1, i2, p3, i3) - """ - expected = """ - [p1, i1, i2, p3, i3] + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] # + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2352,8 +2353,7 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - ifoo = arraylen_gc(p3, descr=arraydescr) # killed by the backend - jump(p1, i1, i2, p3, i3) + jump(p1, i1, i2, p3) """ self.optimize_loop(ops, expected, preamble) @@ -2669,7 +2669,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -2680,7 +2680,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -3302,8 +3302,8 @@ [p8, p11, i24] p26 = new(descr=ssize) setfield_gc(p26, i24, descr=adescr) - i34 = getfield_gc_pure_i(p11, descr=abisdescr) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=abisdescr) + i35 = getfield_gc_i(p26, descr=adescr) i36 = int_add_ovf(i34, i35) guard_no_overflow() [] jump(p8, p11, i35) @@ -3330,8 +3330,8 @@ setfield_gc(p26, i24, descr=adescr) i28 = int_add(i17, 1) setfield_gc(p8, i28, descr=valuedescr) - i34 = getfield_gc_pure_i(p11, descr=valuedescr3) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=valuedescr3) + i35 = getfield_gc_i(p26, descr=adescr) guard_nonnull(p12) [] i36 = int_add_ovf(i34, i35) guard_no_overflow() [] @@ -3522,14 +3522,14 @@ def test_residual_call_does_not_invalidate_immutable_caches(self): ops = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) - i3 = getfield_gc_pure_i(p1, descr=valuedescr3) + i3 = getfield_gc_i(p1, descr=valuedescr3) jump(p1) """ expected_preamble = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) jump(p1, i1) """ @@ -4878,11 +4878,11 @@ def test_add_sub_ovf_virtual_unroll(self): ops = """ [p15] - i886 = getfield_gc_pure_i(p15, descr=valuedescr) + i886 = getfield_gc_i(p15, descr=valuedescr) i888 = int_sub_ovf(i886, 1) guard_no_overflow() [] escape_n(i888) - i4360 = getfield_gc_pure_i(p15, descr=valuedescr) + i4360 = getfield_gc_i(p15, descr=valuedescr) i4362 = int_add_ovf(i4360, 1) guard_no_overflow() [] i4360p = int_sub_ovf(i4362, 1) @@ -4972,18 +4972,16 @@ def test_pure(self): ops = """ [p42] - p53 = getfield_gc_r(ConstPtr(myptr), descr=nextdescr) - p59 = getfield_gc_pure_r(p53, descr=valuedescr) + p53 = getfield_gc_r(ConstPtr(myptr3), descr=nextdescr3) + p59 = getfield_gc_r(p53, descr=valuedescr3) i61 = call_i(1, p59, descr=nonwritedescr) jump(p42) """ expected = """ - [p42, p59] - i61 = call_i(1, p59, descr=nonwritedescr) - jump(p42, p59) - - """ - self.node.value = 5 + [p42] + i61 = call_i(1, 7, descr=nonwritedescr) + jump(p42) + """ self.optimize_loop(ops, expected) def test_complains_getfieldpure_setfield(self): @@ -4992,7 +4990,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) setfield_gc(p1, p3, descr=nextdescr) jump(p3) """ @@ -5002,7 +5000,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5010,7 +5008,7 @@ expected = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5021,7 +5019,7 @@ ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) p3 = escape_r() setfield_gc(p3, p1, descr=nextdescr) jump() @@ -6167,14 +6165,14 @@ def test_bug_unroll_with_immutables(self): ops = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) p1 = new_with_vtable(descr=immut_descr) setfield_gc(p1, 1242, descr=immut_intval) jump(p1) """ preamble = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) jump() """ expected = """ @@ -7229,13 +7227,13 @@ [p0, p1, i0] quasiimmut_field(p0, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ expected = """ [p0, p1, i0] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ @@ -7246,7 +7244,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) escape_n(i1) jump() """ @@ -7298,11 +7296,11 @@ [i0a, i0b] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i2 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7325,11 +7323,11 @@ setfield_gc(p, 421, descr=quasifielddescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p, descr=quasifielddescr) + i1 = getfield_gc_i(p, descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(p, descr=quasifielddescr) + i2 = getfield_gc_i(p, descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7568,7 +7566,7 @@ def test_forced_virtual_pure_getfield(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=valuedescr) + p1 = getfield_gc_r(p0, descr=valuedescr3) jump(p1) """ self.optimize_loop(ops, ops) @@ -7578,7 +7576,7 @@ p1 = new_with_vtable(descr=nodesize3) setfield_gc(p1, p0, descr=valuedescr3) escape_n(p1) - p2 = getfield_gc_pure_r(p1, descr=valuedescr3) + p2 = getfield_gc_r(p1, descr=valuedescr3) escape_n(p2) jump(p0) """ @@ -7852,14 +7850,14 @@ def test_loopinvariant_getarrayitem_gc_pure(self): ops = """ [p9, i1] - i843 = getarrayitem_gc_pure_i(p9, i1, descr=arraydescr) + i843 = getarrayitem_gc_pure_i(p9, i1, descr=arrayimmutdescr) call_n(i843, descr=nonwritedescr) jump(p9, i1) """ expected = """ [p9, i1, i843] call_n(i843, descr=nonwritedescr) - ifoo = arraylen_gc(p9, descr=arraydescr) + ifoo = arraylen_gc(p9, descr=arrayimmutdescr) jump(p9, i1, i843) """ self.optimize_loop(ops, expected) @@ -7868,7 +7866,7 @@ ops = """ [p0] p1 = getfield_gc_r(p0, descr=nextdescr) - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) call_n(p2, descr=nonwritedescr) jump(p0) """ @@ -7883,14 +7881,14 @@ i1 = arraylen_gc(p1, descr=gcarraydescr) i2 = int_ge(i1, 8) guard_true(i2) [] - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) - jump(p2, p1) - """ - expected = """ - [p0, p2, p1] + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) + jump(p1, p2) + """ + expected = """ + [p0, p1, p2] call_n(p2, descr=nonwritedescr) i3 = arraylen_gc(p1, descr=gcarraydescr) # Should be killed by backend - jump(p0, p2, p1) + jump(p0, p1, p2) """ self.optimize_loop(ops, expected, expected_short=short) @@ -8065,7 +8063,7 @@ def test_dont_mixup_equal_boxes(self): ops = """ [p8] - i9 = getfield_gc_pure_i(p8, descr=valuedescr) + i9 = getfield_gc_i(p8, descr=valuedescr3) i10 = int_gt(i9, 0) guard_true(i10) [] i29 = int_lshift(i9, 1) @@ -8160,9 +8158,9 @@ py.test.skip("would be fixed by make heap optimizer aware of virtual setfields") ops = """ [p5, p8] - i9 = getfield_gc_pure_i(p5, descr=valuedescr) + i9 = getfield_gc_i(p5, descr=valuedescr) call_n(i9, descr=nonwritedescr) - i11 = getfield_gc_pure_i(p8, descr=valuedescr) + i11 = getfield_gc_i(p8, descr=valuedescr) i13 = int_add_ovf(i11, 1) guard_no_overflow() [] p22 = new_with_vtable(descr=nodesize) @@ -8201,14 +8199,14 @@ ops = """ [p0] p10 = getfield_gc_r(ConstPtr(myptr), descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(ConstPtr(myptr), ConstPtr(myptr2), descr=otherdescr) + setfield_gc(ConstPtr(myptr), ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8232,14 +8230,14 @@ ops = """ [p0] p10 = getfield_gc_r(p0, descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(p0, ConstPtr(myptr2), descr=otherdescr) + setfield_gc(p0, ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8624,17 +8622,17 @@ [p10] p52 = getfield_gc_r(p10, descr=nextdescr) # inst_storage p54 = getarrayitem_gc_r(p52, 0, descr=arraydescr) - p69 = getfield_gc_pure_r(p54, descr=otherdescr) # inst_w_function + p69 = getfield_gc_r(p54, descr=otherdescr) # inst_w_function quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] p106 = new_with_vtable(descr=nodesize) p108 = new_array(3, descr=arraydescr) p110 = new_with_vtable(descr=nodesize) - setfield_gc(p110, ConstPtr(myptr2), descr=otherdescr) # inst_w_function + setfield_gc(p110, ConstPtr(myptrb), descr=otherdescr) # inst_w_function setarrayitem_gc(p108, 0, p110, descr=arraydescr) setfield_gc(p106, p108, descr=nextdescr) # inst_storage jump(p106) @@ -8650,7 +8648,7 @@ [p69] quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] jump(ConstPtr(myptr)) """ @@ -8852,13 +8850,13 @@ def test_virtual_back_and_forth(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=bdescr) + p1 = getfield_gc_r(p0, descr=nextdescr3) ptemp = new_with_vtable(descr=nodesize) setfield_gc(ptemp, p1, descr=nextdescr) p2 = getfield_gc_r(ptemp, descr=nextdescr) - ix = getarrayitem_gc_pure_i(p2, 0, descr=arraydescr) + ix = getarrayitem_gc_pure_i(p2, 0, descr=arrayimmutdescr) pfoo = getfield_gc_r(ptemp, descr=nextdescr) - guard_value(pfoo, ConstPtr(myarray)) [] + guard_value(pfoo, ConstPtr(immutarray)) [] ifoo = int_add(ix, 13) escape_n(ix) jump(p0) @@ -8888,13 +8886,13 @@ def test_constant_float_pure(self): ops = """ [p0] - f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarraydescr) + f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarrayimmutdescr) guard_value(f0, 1.03) [] jump(p0) """ expected = """ [p0] - ifoo = arraylen_gc(p0, descr=floatarraydescr) + ifoo = arraylen_gc(p0, descr=floatarrayimmutdescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -9102,7 +9100,7 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) jump(NULL, i3) """ @@ -9113,9 +9111,9 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) - jump(ConstPtr(myptr4), i3) + jump(ConstPtr(myptr2), i3) """ py.test.raises(InvalidLoop, self.optimize_loop, ops, ops) @@ -9265,9 +9263,126 @@ guard_value(i1, 5) [] jump() """ - a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE)), 5, zero=True) + a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE3)), 5, zero=True) self.optimize_loop(ops, expected, jump_values=[a]) + def test_large_number_of_immutable_references(self): + ops = """ + [p0] + i0 = getfield_gc_i(p0, descr=bigadescr) + i1 = getfield_gc_i(p0, descr=bigbdescr) + i2 = getfield_gc_i(p0, descr=bigcdescr) + i3 = getfield_gc_i(p0, descr=bigddescr) + i4 = getfield_gc_i(p0, descr=bigedescr) + i5 = getfield_gc_i(p0, descr=bigfdescr) + i6 = getfield_gc_i(p0, descr=biggdescr) + i7 = getfield_gc_i(p0, descr=bighdescr) + i8 = getfield_gc_i(p0, descr=bigidescr) + i9 = getfield_gc_i(p0, descr=bigjdescr) + i10 = getfield_gc_i(p0, descr=bigkdescr) + i11 = getfield_gc_i(p0, descr=bigldescr) + i12 = getfield_gc_i(p0, descr=bigmdescr) + i13 = getfield_gc_i(p0, descr=bigndescr) + i14 = getfield_gc_i(p0, descr=bigodescr) + i15 = getfield_gc_i(p0, descr=bigpdescr) + i16 = getfield_gc_i(p0, descr=bigqdescr) + i17 = getfield_gc_i(p0, descr=bigrdescr) + i18 = getfield_gc_i(p0, descr=bigsdescr) + i19 = getfield_gc_i(p0, descr=bigtdescr) + i20 = getfield_gc_i(p0, descr=bigudescr) + i21 = getfield_gc_i(p0, descr=bigvdescr) + i22 = getfield_gc_i(p0, descr=bigwdescr) + i23 = getfield_gc_i(p0, descr=bigxdescr) + i24 = getfield_gc_i(p0, descr=bigydescr) + i25 = getfield_gc_i(p0, descr=bigzdescr) + i27 = getfield_gc_i(p0, descr=bigbdescr) + i28 = getfield_gc_i(p0, descr=bigcdescr) + i29 = getfield_gc_i(p0, descr=bigddescr) + i30 = getfield_gc_i(p0, descr=bigedescr) + i31 = getfield_gc_i(p0, descr=bigfdescr) + i32 = getfield_gc_i(p0, descr=biggdescr) + i33 = getfield_gc_i(p0, descr=bighdescr) + i34 = getfield_gc_i(p0, descr=bigidescr) + i35 = getfield_gc_i(p0, descr=bigjdescr) + i36 = getfield_gc_i(p0, descr=bigkdescr) + i37 = getfield_gc_i(p0, descr=bigldescr) + i38 = getfield_gc_i(p0, descr=bigmdescr) + i39 = getfield_gc_i(p0, descr=bigndescr) + i40 = getfield_gc_i(p0, descr=bigodescr) + i41 = getfield_gc_i(p0, descr=bigpdescr) + i42 = getfield_gc_i(p0, descr=bigqdescr) + i43 = getfield_gc_i(p0, descr=bigrdescr) + i44 = getfield_gc_i(p0, descr=bigsdescr) + i45 = getfield_gc_i(p0, descr=bigtdescr) + i46 = getfield_gc_i(p0, descr=bigudescr) + i47 = getfield_gc_i(p0, descr=bigvdescr) + i48 = getfield_gc_i(p0, descr=bigwdescr) + i49 = getfield_gc_i(p0, descr=bigxdescr) + i50 = getfield_gc_i(p0, descr=bigydescr) + i51 = getfield_gc_i(p0, descr=bigzdescr) + i26 = getfield_gc_i(p0, descr=bigadescr) + i99 = int_add(i26, i51) + escape_i(i27) + escape_i(i28) + escape_i(i29) + escape_i(i30) + escape_i(i31) + escape_i(i32) + escape_i(i33) + escape_i(i34) + escape_i(i35) + escape_i(i36) + escape_i(i37) + escape_i(i38) + escape_i(i39) + escape_i(i40) + escape_i(i41) + escape_i(i42) + escape_i(i43) + escape_i(i44) + escape_i(i45) + escape_i(i46) + escape_i(i47) + escape_i(i48) + escape_i(i49) + escape_i(i50) + escape_i(i51) + escape_i(i26) + escape_i(i99) + jump(p0) + """ + expected = """ + [p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99] + escape_i(i1) + escape_i(i2) + escape_i(i3) + escape_i(i4) + escape_i(i5) + escape_i(i6) + escape_i(i7) + escape_i(i8) + escape_i(i9) + escape_i(i10) + escape_i(i11) + escape_i(i12) + escape_i(i13) + escape_i(i14) + escape_i(i15) + escape_i(i16) + escape_i(i17) + escape_i(i18) + escape_i(i19) + escape_i(i20) + escape_i(i21) + escape_i(i22) + escape_i(i23) + escape_i(i24) + escape_i(i25) + escape_i(i0) + escape_i(i99) + jump(p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99) + """ + self.optimize_loop(ops, expected) class TestLLtype(OptimizeOptTest, LLtypeMixin): pass diff --git a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py --- a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py @@ -220,16 +220,16 @@ def test_double_getfield_plus_pure(self): loop = """ [p0] - pc = getfield_gc_pure_r(p0, descr=nextdescr) + pc = getfield_gc_r(p0, descr=nextdescr3) escape_n(p0) # that should flush the caches - p1 = getfield_gc_r(pc, descr=nextdescr) - i0 = getfield_gc_i(p1, descr=valuedescr) + p1 = getfield_gc_r(pc, descr=nextdescr3) + i0 = getfield_gc_i(p1, descr=valuedescr3) jump(p0) """ es, loop, preamble = self.optimize(loop) assert len(es.short_boxes) == 4 # both getfields are available as - # well as getfield_gc_pure + # well as getfield_gc def test_p123_anti_nested(self): loop = """ diff --git a/rpython/jit/metainterp/optimizeopt/test/test_util.py b/rpython/jit/metainterp/optimizeopt/test/test_util.py --- a/rpython/jit/metainterp/optimizeopt/test/test_util.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_util.py @@ -1,4 +1,4 @@ -import py, random +import py, random, string from rpython.rlib.debug import debug_print from rpython.rtyper.lltypesystem import lltype, llmemory, rffi @@ -122,7 +122,14 @@ ('value', lltype.Signed), ('next', lltype.Ptr(NODE3)), hints={'immutable': True})) - + + big_fields = [('big' + i, lltype.Signed) for i in string.ascii_lowercase] + BIG = lltype.GcForwardReference() + BIG.become(lltype.GcStruct('BIG', *big_fields, hints={'immutable': True})) + + for field, _ in big_fields: + locals()[field + 'descr'] = cpu.fielddescrof(BIG, field) + node = lltype.malloc(NODE) node.value = 5 node.next = node @@ -133,16 +140,25 @@ node2.parent.parent.typeptr = node_vtable2 node2addr = lltype.cast_opaque_ptr(llmemory.GCREF, node2) myptr = lltype.cast_opaque_ptr(llmemory.GCREF, node) - mynode2 = lltype.malloc(NODE) + mynodeb = lltype.malloc(NODE) myarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(lltype.GcArray(lltype.Signed), 13, zero=True)) - mynode2.parent.typeptr = node_vtable - myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode2) - mynode3 = lltype.malloc(NODE2) - mynode3.parent.parent.typeptr = node_vtable2 + mynodeb.parent.typeptr = node_vtable + myptrb = lltype.cast_opaque_ptr(llmemory.GCREF, mynodeb) + myptr2 = lltype.malloc(NODE2) + myptr2.parent.parent.typeptr = node_vtable2 + myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, myptr2) + nullptr = lltype.nullptr(llmemory.GCREF.TO) + + mynode3 = lltype.malloc(NODE3) + mynode3.parent.typeptr = node_vtable3 + mynode3.value = 7 + mynode3.next = mynode3 myptr3 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode3) # a NODE2 mynode4 = lltype.malloc(NODE3) mynode4.parent.typeptr = node_vtable3 myptr4 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode4) # a NODE3 + + nullptr = lltype.nullptr(llmemory.GCREF.TO) #nodebox2 = InputArgRef(lltype.cast_opaque_ptr(llmemory.GCREF, node2)) nodesize = cpu.sizeof(NODE, node_vtable) @@ -203,7 +219,6 @@ arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed)) int32arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.INT)) int16arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.SHORT)) - floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) float32arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.SingleFloat)) arraydescr_tid = arraydescr.get_type_id() array = lltype.malloc(lltype.GcArray(lltype.Signed), 15, zero=True) @@ -212,6 +227,12 @@ array2ref = lltype.cast_opaque_ptr(llmemory.GCREF, array2) gcarraydescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF)) gcarraydescr_tid = gcarraydescr.get_type_id() + floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) + + arrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed, hints={"immutable": True})) + immutarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(arrayimmutdescr.A, 13, zero=True)) + gcarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF, hints={"immutable": True})) + floatarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Float, hints={"immutable": True})) # a GcStruct not inheriting from OBJECT tpl = lltype.malloc(S, zero=True) @@ -244,7 +265,7 @@ tsize = cpu.sizeof(T, None) cdescr = cpu.fielddescrof(T, 'c') ddescr = cpu.fielddescrof(T, 'd') - arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE))) + arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE3))) U = lltype.GcStruct('U', ('parent', OBJECT), diff --git a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py --- a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py @@ -1103,8 +1103,8 @@ jump(p0) """ self.optimize_bridge(loops, bridge, loops[0], 'Loop0', [self.myptr]) - self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr3]) - self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr3]) + self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr2]) + self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr2]) self.optimize_bridge(loops, loops[0], loops[0], 'Loop0', [self.nullptr]) self.optimize_bridge(loops, loops[1], loops[1], 'Loop1', [self.nullptr]) diff --git a/rpython/jit/metainterp/optimizeopt/virtualize.py b/rpython/jit/metainterp/optimizeopt/virtualize.py --- a/rpython/jit/metainterp/optimizeopt/virtualize.py +++ b/rpython/jit/metainterp/optimizeopt/virtualize.py @@ -188,12 +188,6 @@ optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - # note: the following line does not mean that the two operations are - # completely equivalent, because GETFIELD_GC_PURE is_always_pure(). - optimize_GETFIELD_GC_PURE_I = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_I - def optimize_SETFIELD_GC(self, op): struct = op.getarg(0) opinfo = self.getptrinfo(struct) diff --git a/rpython/jit/metainterp/pyjitpl.py b/rpython/jit/metainterp/pyjitpl.py --- a/rpython/jit/metainterp/pyjitpl.py +++ b/rpython/jit/metainterp/pyjitpl.py @@ -653,46 +653,37 @@ @arguments("box", "descr") def opimpl_getfield_gc_i(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resbox = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_I, fielddescr, box) + return ConstInt(resbox) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_I, box, fielddescr, 'i') + + @arguments("box", "descr") + def opimpl_getfield_gc_f(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resvalue = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_F, fielddescr, box) + return ConstFloat(resvalue) + return self._opimpl_getfield_gc_any_pureornot( + rop.GETFIELD_GC_F, box, fielddescr, 'f') + @arguments("box", "descr") def opimpl_getfield_gc_r(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + val = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_R, fielddescr, box) + return ConstPtr(val) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_R, box, fielddescr, 'r') - @arguments("box", "descr") - def opimpl_getfield_gc_f(self, box, fielddescr): - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_i_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resbox = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_I, fielddescr, box) - return ConstInt(resbox) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_I, box, fielddescr, 'i') - - @arguments("box", "descr") - def opimpl_getfield_gc_f_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resvalue = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_F, fielddescr, box) - return ConstFloat(resvalue) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_r_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - val = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_R, fielddescr, box) - return ConstPtr(val) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_R, box, fielddescr, 'r') + + opimpl_getfield_gc_i_pure = opimpl_getfield_gc_i + opimpl_getfield_gc_r_pure = opimpl_getfield_gc_r + opimpl_getfield_gc_f_pure = opimpl_getfield_gc_f @arguments("box", "box", "descr") def opimpl_getinteriorfield_gc_i(self, array, index, descr): @@ -733,7 +724,7 @@ @arguments("box", "descr", "orgpc") def _opimpl_getfield_gc_greenfield_any(self, box, fielddescr, pc): ginfo = self.metainterp.jitdriver_sd.greenfield_info - opnum = OpHelpers.getfield_pure_for_descr(fielddescr) + opnum = OpHelpers.getfield_for_descr(fielddescr) if (ginfo is not None and fielddescr in ginfo.green_field_descrs and not self._nonstandard_virtualizable(pc, box, fielddescr)): # fetch the result, but consider it as a Const box and don't @@ -2104,17 +2095,7 @@ profiler = self.staticdata.profiler profiler.count_ops(opnum) resvalue = executor.execute(self.cpu, self, opnum, descr, *argboxes) - # - is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST - if not is_pure: - if (opnum == rop.GETFIELD_RAW_I or - opnum == rop.GETFIELD_RAW_R or - opnum == rop.GETFIELD_RAW_F or - opnum == rop.GETARRAYITEM_RAW_I or - opnum == rop.GETARRAYITEM_RAW_F): - is_pure = descr.is_always_pure() - # - if is_pure: + if OpHelpers.is_pure_with_descr(opnum, descr): return self._record_helper_pure(opnum, resvalue, descr, *argboxes) if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: return self._record_helper_ovf(opnum, resvalue, descr, *argboxes) diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py --- a/rpython/jit/metainterp/resoperation.py +++ b/rpython/jit/metainterp/resoperation.py @@ -231,7 +231,7 @@ class AbstractResOpOrInputArg(AbstractValue): _attrs_ = ('_forwarded',) - _forwarded = None # either another resop or OptInfo + _forwarded = None # either another resop or OptInfo def get_forwarded(self): return self._forwarded @@ -412,6 +412,8 @@ return rop._JIT_DEBUG_FIRST <= self.getopnum() <= rop._JIT_DEBUG_LAST def is_always_pure(self): + # Tells whether an operation is pure based solely on the opcode. + # Other operations (e.g. getfield ops) may be pure in some cases are well. return rop._ALWAYS_PURE_FIRST <= self.getopnum() <= rop._ALWAYS_PURE_LAST def has_no_side_effect(self): @@ -434,9 +436,7 @@ return self.opnum in (rop.SAME_AS_I, rop.SAME_AS_F, rop.SAME_AS_R) def is_getfield(self): - return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, - rop.GETFIELD_GC_R, rop.GETFIELD_GC_PURE_I, - rop.GETFIELD_GC_PURE_R, rop.GETFIELD_GC_PURE_F) + return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R) def is_getarrayitem(self): return self.opnum in (rop.GETARRAYITEM_GC_I, rop.GETARRAYITEM_GC_F, @@ -1602,14 +1602,6 @@ return rop.CALL_LOOPINVARIANT_N @staticmethod - def getfield_pure_for_descr(descr): - if descr.is_pointer_field(): - return rop.GETFIELD_GC_PURE_R - elif descr.is_float_field(): - return rop.GETFIELD_GC_PURE_F - return rop.GETFIELD_GC_PURE_I - - @staticmethod def getfield_for_descr(descr): if descr.is_pointer_field(): return rop.GETFIELD_GC_R @@ -1760,4 +1752,26 @@ opnum = rop.VEC_UNPACK_F return VecOperationNew(opnum, args, datatype, bytesize, signed, count) + @staticmethod + def is_pure_getfield(opnum, descr): + if (opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETFIELD_GC_R): + return descr is not None and descr.is_always_pure() + return False + @staticmethod + def is_pure_with_descr(opnum, descr): + is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST + if not is_pure: + if (opnum == rop.GETFIELD_RAW_I or + opnum == rop.GETFIELD_RAW_R or + opnum == rop.GETFIELD_RAW_F or + opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_R or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETARRAYITEM_RAW_I or + opnum == rop.GETARRAYITEM_RAW_F): + is_pure = descr.is_always_pure() + return is_pure + diff --git a/rpython/jit/metainterp/test/test_ajit.py b/rpython/jit/metainterp/test/test_ajit.py --- a/rpython/jit/metainterp/test/test_ajit.py +++ b/rpython/jit/metainterp/test/test_ajit.py @@ -320,7 +320,7 @@ assert res == 252 self.check_trace_count(1) self.check_resops({'jump': 1, 'int_gt': 2, 'int_add': 2, - 'getfield_gc_pure_i': 1, 'int_mul': 1, + 'getfield_gc_i': 1, 'int_mul': 1, 'guard_true': 2, 'int_sub': 2}) def test_loops_are_transient(self): @@ -1405,7 +1405,7 @@ return tup[1] res = self.interp_operations(f, [3, 5]) assert res == 5 - self.check_operations_history(setfield_gc=2, getfield_gc_pure_i=0) + self.check_operations_history(setfield_gc=2, getfield_gc_i=0) def test_oosend_look_inside_only_one(self): class A: @@ -2522,7 +2522,7 @@ if counter > 10: return 7 assert self.meta_interp(build, []) == 7 - self.check_resops(getfield_gc_pure_r=2) + self.check_resops(getfield_gc_r=2) def test_args_becomming_equal(self): myjitdriver = JitDriver(greens = [], reds = ['n', 'i', 'sa', 'a', 'b']) diff --git a/rpython/jit/metainterp/test/test_immutable.py b/rpython/jit/metainterp/test/test_immutable.py --- a/rpython/jit/metainterp/test/test_immutable.py +++ b/rpython/jit/metainterp/test/test_immutable.py @@ -19,7 +19,7 @@ return y.x + 5 res = self.interp_operations(f, [23]) assert res == 28 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=1, int_add=1) + self.check_operations_history(getfield_gc_i=1, int_add=1) def test_fields_subclass(self): class X(object): @@ -41,8 +41,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def f(x, y): # this time, the field 'x' only shows up on subclass 'Y' @@ -50,8 +49,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def test_array(self): class X(object): @@ -66,8 +64,7 @@ return a.y[index] res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_index_error(self): class X(object): @@ -89,8 +86,7 @@ return a.get(index) res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_in_immutable(self): class X(object): @@ -106,8 +102,7 @@ return y.lst[index] + y.y + 5 res = self.interp_operations(f, [23, 0], listops=True) assert res == 23 + 24 + 5 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getfield_gc_pure_i=1, + self.check_operations_history(getfield_gc_r=1, getfield_gc_i=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1, int_add=3) diff --git a/rpython/jit/metainterp/test/test_quasiimmut.py b/rpython/jit/metainterp/test/test_quasiimmut.py --- a/rpython/jit/metainterp/test/test_quasiimmut.py +++ b/rpython/jit/metainterp/test/test_quasiimmut.py @@ -74,7 +74,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -101,7 +101,7 @@ res = self.meta_interp(f, [100, 7], enable_opts="") assert res == 700 # there should be no getfields, even though optimizations are turned off - self.check_resops(guard_not_invalidated=1, getfield_gc=0) + self.check_resops(guard_not_invalidated=1) def test_nonopt_1(self): myjitdriver = JitDriver(greens=[], reds=['x', 'total', 'lst']) @@ -124,8 +124,7 @@ assert f(100, 7) == 721 res = self.meta_interp(f, [100, 7]) assert res == 721 - self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, - getfield_gc_pure_i=2) + self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, getfield_gc_i=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -156,7 +155,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) def test_change_during_tracing_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -208,7 +207,7 @@ assert f(100, 7) == 700 res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=0, getfield_gc=0) + self.check_resops(guard_not_invalidated=0) def test_change_invalidate_reentering(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -234,7 +233,7 @@ assert g(100, 7) == 700707 res = self.meta_interp(g, [100, 7]) assert res == 700707 - self.check_resops(guard_not_invalidated=4, getfield_gc=0) + self.check_resops(guard_not_invalidated=4) def test_invalidate_while_running(self): jitdriver = JitDriver(greens=['foo'], reds=['i', 'total']) @@ -348,7 +347,7 @@ res = self.meta_interp(f, [100, 30]) assert res == 6019 self.check_resops(guard_not_invalidated=8, guard_not_forced=0, - call_may_force=0, getfield_gc=0) + call_may_force=0) def test_list_simple_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -374,8 +373,7 @@ getarrayitem_gc_pure_r=0, getarrayitem_gc_i=0, getarrayitem_gc_r=0, - getfield_gc_i=0, getfield_gc_pure_i=0, - getfield_gc_r=0, getfield_gC_pure_r=0) + getfield_gc_i=0, getfield_gc_r=0) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -405,9 +403,7 @@ assert res == 700 # operations must have been removed by the frontend self.check_resops(getarrayitem_gc_pure_i=0, guard_not_invalidated=1, - getarrayitem_gc_i=0, - getfield_gc=0, getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + getarrayitem_gc_i=0, getfield_gc_i=0, getfield_gc_r=0) def test_list_length_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) diff --git a/rpython/jit/metainterp/test/test_tracingopts.py b/rpython/jit/metainterp/test/test_tracingopts.py --- a/rpython/jit/metainterp/test/test_tracingopts.py +++ b/rpython/jit/metainterp/test/test_tracingopts.py @@ -436,10 +436,10 @@ return p.x[0] + p.x[1] res = self.interp_operations(fn, [7]) assert res == 7 + 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == -7 - 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) def test_heap_caching_and_elidable_function(self): class A: @@ -517,12 +517,12 @@ return a1[0] + a2[0] + gn(a1, a2) res = self.interp_operations(fn, [7]) assert res == 2 * 7 + 2 * 6 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == 2 * -7 + 2 * -8 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) def test_heap_caching_multiple_arrays(self): class Gbl(object): diff --git a/rpython/jit/metainterp/test/test_virtual.py b/rpython/jit/metainterp/test/test_virtual.py --- a/rpython/jit/metainterp/test/test_virtual.py +++ b/rpython/jit/metainterp/test/test_virtual.py @@ -1077,7 +1077,7 @@ res = self.meta_interp(f, [], repeat=7) assert res == f() - def test_getfield_gc_pure_nobug(self): + def test_pure_getfield_gc_nobug(self): mydriver = JitDriver(reds=['i', 's', 'a'], greens=[]) class A(object): From pypy.commits at gmail.com Thu Jan 21 16:22:48 2016 From: pypy.commits at gmail.com (sbauman) Date: Thu, 21 Jan 2016 13:22:48 -0800 (PST) Subject: [pypy-commit] pypy default: Mention remove-getfield-pure branch in whatsnew-head Message-ID: <56a14c28.05bd1c0a.82596.0b09@mx.google.com> Author: Spenser Bauman Branch: Changeset: r81899:35c27789e353 Date: 2016-01-21 16:22 -0500 http://bitbucket.org/pypy/pypy/changeset/35c27789e353/ Log: Mention remove-getfield-pure branch in whatsnew-head diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -123,3 +123,13 @@ .. branch: fix-cpython-ssl-tests-2.7 Fix SSL tests by importing cpython's patch + +.. branch: remove-getfield-pure + +Remove pure variants of ``getfield_gc_*`` operations from the JIT. Relevant +optimizations instead consult the field descriptor to determine the purity of +the operation. Additionally, pure ``getfield`` operations are now handled +entirely by `rpython/jit/metainterp/optimizeopt/heap.py` rather than +`rpython/jit/metainterp/optimizeopt/pure.py`, which can result in better codegen +for traces containing a large number of pure getfield operations. + From pypy.commits at gmail.com Fri Jan 22 02:37:00 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 23:37:00 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: unicode endian test issue in pypy module Message-ID: <56a1dc1c.c5321c0a.8b828.ffff82c0@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81900:d2af83605cf2 Date: 2016-01-22 08:36 +0100 http://bitbucket.org/pypy/pypy/changeset/d2af83605cf2/ Log: unicode endian test issue in pypy module diff --git a/pypy/module/_file/test/test_file_extra.py b/pypy/module/_file/test/test_file_extra.py --- a/pypy/module/_file/test/test_file_extra.py +++ b/pypy/module/_file/test/test_file_extra.py @@ -389,6 +389,7 @@ def test_writelines(self): import array + import sys fn = self.temptestfile with file(fn, 'w') as f: f.writelines(['abc']) @@ -406,7 +407,10 @@ exc = raises(TypeError, f.writelines, [memoryview('jkl')]) assert str(exc.value) == "writelines() argument must be a sequence of strings" out = open(fn, 'rb').readlines()[0] - assert out[0:5] == 'abcd\x00' + if sys.byteorder == 'big': + assert out[0:7] == 'abc\x00\x00\x00d' + else: + assert out[0:5] == 'abcd\x00' assert out[-3:] == 'ghi' with file(fn, 'wb') as f: From pypy.commits at gmail.com Fri Jan 22 02:44:31 2016 From: pypy.commits at gmail.com (plan_rich) Date: Thu, 21 Jan 2016 23:44:31 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: removed debug print statement, return value of closure changed to long. a closure return value must fill a full ffi_arg Message-ID: <56a1dddf.2851c20a.c3793.ffffa4e3@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81901:4cafa238dca9 Date: 2016-01-22 08:43 +0100 http://bitbucket.org/pypy/pypy/changeset/4cafa238dca9/ Log: removed debug print statement, return value of closure changed to long. a closure return value must fill a full ffi_arg diff --git a/pypy/module/_rawffi/test/test__rawffi.py b/pypy/module/_rawffi/test/test__rawffi.py --- a/pypy/module/_rawffi/test/test__rawffi.py +++ b/pypy/module/_rawffi/test/test__rawffi.py @@ -704,7 +704,6 @@ def compare(a, b): a1 = _rawffi.Array('i').fromaddress(_rawffi.Array('P').fromaddress(a, 1)[0], 1) a2 = _rawffi.Array('i').fromaddress(_rawffi.Array('P').fromaddress(b, 1)[0], 1) - print "comparing", a1[0], "with", a2[0] if a1[0] not in [1,2,3,4] or a2[0] not in [1,2,3,4]: bogus_args.append((a1[0], a2[0])) if a1[0] > a2[0]: @@ -715,7 +714,7 @@ a2[0] = len(ll_to_sort) a3 = _rawffi.Array('l')(1) a3[0] = struct.calcsize('i') - cb = _rawffi.CallbackPtr(compare, ['P', 'P'], 'i') + cb = _rawffi.CallbackPtr(compare, ['P', 'P'], 'l') a4 = cb.byptr() qsort(a1, a2, a3, a4) res = [ll_to_sort[i] for i in range(len(ll_to_sort))] From pypy.commits at gmail.com Fri Jan 22 03:18:22 2016 From: pypy.commits at gmail.com (cfbolz) Date: Fri, 22 Jan 2016 00:18:22 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: test and fix for write_necessary logic when mixing ints and objects Message-ID: <56a1e5ce.11301c0a.52de.ffff913c@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81902:c1866643eecd Date: 2016-01-22 09:13 +0100 http://bitbucket.org/pypy/pypy/changeset/c1866643eecd/ Log: test and fix for write_necessary logic when mixing ints and objects diff --git a/rpython/rlib/heapprof.py b/rpython/rlib/heapprof.py --- a/rpython/rlib/heapprof.py +++ b/rpython/rlib/heapprof.py @@ -108,8 +108,9 @@ # call write_necessary if there is already a value there assert not status == SEEN_NOTHING if status == SEEN_CONSTANT_INT: - return (self.is_int(w_value) and - self.read_constant_int() != self.get_int_val(w_value)) + if not self.is_int(w_value): + return True + return self.read_constant_int() != self.get_int_val(w_value) elif status == SEEN_CONSTANT_OBJ: prev_obj = self.try_read_constant_obj() return prev_obj is not w_value diff --git a/rpython/rlib/test/test_heapprof.py b/rpython/rlib/test/test_heapprof.py --- a/rpython/rlib/test/test_heapprof.py +++ b/rpython/rlib/test/test_heapprof.py @@ -143,6 +143,13 @@ res = v.write_necessary(ValueInt(1)) assert res + v = HeapProf() + assert v._hprof_status == SEEN_NOTHING + v.see_write(ValueInt(1)) + res = v.write_necessary(Value()) + assert res + + def test_write_not_necessary_obj(): v = HeapProf() assert v._hprof_status == SEEN_NOTHING @@ -159,3 +166,4 @@ v.see_write(Value()) res = v.write_necessary(Value()) assert res + From pypy.commits at gmail.com Fri Jan 22 03:18:24 2016 From: pypy.commits at gmail.com (cfbolz) Date: Fri, 22 Jan 2016 00:18:24 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: test that found the problem of the previous commit Message-ID: <56a1e5d0.a5c9c20a.809ac.ffffb4ca@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81903:7e6dd66f2318 Date: 2016-01-22 09:17 +0100 http://bitbucket.org/pypy/pypy/changeset/7e6dd66f2318/ Log: test that found the problem of the previous commit diff --git a/pypy/objspace/std/test/test_mapdict.py b/pypy/objspace/std/test/test_mapdict.py --- a/pypy/objspace/std/test/test_mapdict.py +++ b/pypy/objspace/std/test/test_mapdict.py @@ -931,6 +931,21 @@ d = x.__dict__ assert list(__pypy__.reversed_dict(d)) == d.keys()[::-1] + def test_bug_two_attributes(self): + class A(object): + def __setitem__(self, key, value): + self.setkey = key + self.setvalue = value + a1 = A() + a2 = A() + a1[a2] = 42 + assert a1.setkey is a2 + assert a1.setvalue == 42 + # + a1[42] = a2 + assert a1.setkey == 42 + assert a1.setvalue is a2 + class AppTestWithMapDictAndCounters(object): spaceconfig = {"objspace.std.withmapdict": True, From pypy.commits at gmail.com Fri Jan 22 03:21:24 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 00:21:24 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: big endian test issue cpyext Message-ID: <56a1e684.6217c20a.dcde6.ffffb5ef@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81904:43c90f2f520e Date: 2016-01-22 09:20 +0100 http://bitbucket.org/pypy/pypy/changeset/43c90f2f520e/ Log: big endian test issue cpyext diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py --- a/pypy/module/cpyext/test/test_arraymodule.py +++ b/pypy/module/cpyext/test/test_arraymodule.py @@ -51,13 +51,20 @@ assert arr.tolist() == [1, 23, 4] def test_buffer(self): + import sys module = self.import_module(name='array') arr = module.array('i', [1,2,3,4]) buf = buffer(arr) exc = raises(TypeError, "buf[1] = '1'") assert str(exc.value) == "buffer is read-only" # XXX big-endian - assert str(buf) == ('\x01\0\0\0' - '\x02\0\0\0' - '\x03\0\0\0' - '\x04\0\0\0') + if sys.byteorder == 'big': + assert str(buf) == ('\0\0\0\x01' + '\0\0\0\x02' + '\0\0\0\x03' + '\0\0\0\x04') + else: + assert str(buf) == ('\x01\0\0\0' + '\x02\0\0\0' + '\x03\0\0\0' + '\x04\0\0\0') From pypy.commits at gmail.com Fri Jan 22 03:35:05 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 00:35:05 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: big endian issue while testing cpyext (PyUnicode_DecodeUTF16) Message-ID: <56a1e9b9.034cc20a.9c04f.ffffb849@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81905:b1b31c094879 Date: 2016-01-22 09:34 +0100 http://bitbucket.org/pypy/pypy/changeset/b1b31c094879/ Log: big endian issue while testing cpyext (PyUnicode_DecodeUTF16) diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py --- a/pypy/module/cpyext/test/test_arraymodule.py +++ b/pypy/module/cpyext/test/test_arraymodule.py @@ -57,7 +57,6 @@ buf = buffer(arr) exc = raises(TypeError, "buf[1] = '1'") assert str(exc.value) == "buffer is read-only" - # XXX big-endian if sys.byteorder == 'big': assert str(buf) == ('\0\0\0\x01' '\0\0\0\x02' diff --git a/pypy/module/cpyext/test/test_unicodeobject.py b/pypy/module/cpyext/test/test_unicodeobject.py --- a/pypy/module/cpyext/test/test_unicodeobject.py +++ b/pypy/module/cpyext/test/test_unicodeobject.py @@ -386,11 +386,11 @@ lltype.free(pendian, flavor='raw') test("\x61\x00\x62\x00\x63\x00\x64\x00", -1) - - test("\x61\x00\x62\x00\x63\x00\x64\x00", None) - + if sys.byteorder == 'big': + test("\x00\x61\x00\x62\x00\x63\x00\x64", None) + else: + test("\x61\x00\x62\x00\x63\x00\x64\x00", None) test("\x00\x61\x00\x62\x00\x63\x00\x64", 1) - test("\xFE\xFF\x00\x61\x00\x62\x00\x63\x00\x64", 0, 1) test("\xFF\xFE\x61\x00\x62\x00\x63\x00\x64\x00", 0, -1) From pypy.commits at gmail.com Fri Jan 22 03:36:53 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 00:36:53 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: the same for the UTF32 test Message-ID: <56a1ea25.284cc20a.b44fb.ffffb77a@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81906:9df502324ca5 Date: 2016-01-22 09:36 +0100 http://bitbucket.org/pypy/pypy/changeset/9df502324ca5/ Log: the same for the UTF32 test diff --git a/pypy/module/cpyext/test/test_unicodeobject.py b/pypy/module/cpyext/test/test_unicodeobject.py --- a/pypy/module/cpyext/test/test_unicodeobject.py +++ b/pypy/module/cpyext/test/test_unicodeobject.py @@ -423,7 +423,10 @@ test("\x61\x00\x00\x00\x62\x00\x00\x00", -1) - test("\x61\x00\x00\x00\x62\x00\x00\x00", None) + if sys.byteorder == 'big': + test("\x00\x00\x00\x61\x00\x00\x00\x62", None) + else: + test("\x61\x00\x00\x00\x62\x00\x00\x00", None) test("\x00\x00\x00\x61\x00\x00\x00\x62", 1) From pypy.commits at gmail.com Fri Jan 22 04:00:18 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 01:00:18 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: endian test issue marshal, test decoded value using platform endianess (not desired for marshall module) Message-ID: <56a1efa2.e935c20a.8d22c.ffffc1ca@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81907:359e329036d5 Date: 2016-01-22 09:59 +0100 http://bitbucket.org/pypy/pypy/changeset/359e329036d5/ Log: endian test issue marshal, test decoded value using platform endianess (not desired for marshall module) diff --git a/pypy/module/marshal/test/test_marshalimpl.py b/pypy/module/marshal/test/test_marshalimpl.py --- a/pypy/module/marshal/test/test_marshalimpl.py +++ b/pypy/module/marshal/test/test_marshalimpl.py @@ -64,14 +64,17 @@ import marshal, struct class FakeM: + # NOTE: marshal is platform independent, running this test must assume + # that seen gets values from the endianess of the marshal module. + # (which is little endian!) def __init__(self): self.seen = [] def start(self, code): self.seen.append(code) def put_int(self, value): - self.seen.append(struct.pack("i", value)) + self.seen.append(struct.pack(" Author: Richard Plangger Branch: s390x-backend Changeset: r81908:6b28800745be Date: 2016-01-22 10:21 +0100 http://bitbucket.org/pypy/pypy/changeset/6b28800745be/ Log: numpy dtype fixes in the test suite diff --git a/pypy/module/marshal/test/test_marshalimpl.py b/pypy/module/marshal/test/test_marshalimpl.py --- a/pypy/module/marshal/test/test_marshalimpl.py +++ b/pypy/module/marshal/test/test_marshalimpl.py @@ -65,7 +65,7 @@ class FakeM: # NOTE: marshal is platform independent, running this test must assume - # that seen gets values from the endianess of the marshal module. + # that self.seen gets values from the endianess of the marshal module. # (which is little endian!) def __init__(self): self.seen = [] diff --git a/pypy/module/micronumpy/test/test_dtypes.py b/pypy/module/micronumpy/test/test_dtypes.py --- a/pypy/module/micronumpy/test/test_dtypes.py +++ b/pypy/module/micronumpy/test/test_dtypes.py @@ -351,7 +351,10 @@ assert np.dtype(xyz).name == 'xyz' # another obscure API, used in numpy record.py a = np.dtype((xyz, [('x', 'int32'), ('y', 'float32')])) - assert "[('x', 'i4'), ('y', '>f4')]" in repr(a) + else: + assert "[('x', 'i4" + E = '<' if sys.byteorder == 'little' else '>' + b = np.dtype((xyz, [("col1", E+"i4"), ("col2", E+"i4"), ("col3", E+"i4")])) data = [(1, 2,3), (4, 5, 6)] a = np.array(data, dtype=b) x = pickle.loads(pickle.dumps(a)) @@ -423,18 +429,20 @@ assert hash(t5) != hash(t6) def test_pickle(self): + import sys import numpy as np from numpy import array, dtype from cPickle import loads, dumps a = array([1,2,3]) + E = '<' if sys.byteorder == 'little' else '>' if self.ptr_size == 8: - assert a.dtype.__reduce__() == (dtype, ('i8', 0, 1), (3, '<', None, None, None, -1, -1, 0)) + assert a.dtype.__reduce__() == (dtype, ('i8', 0, 1), (3, E, None, None, None, -1, -1, 0)) else: - assert a.dtype.__reduce__() == (dtype, ('i4', 0, 1), (3, '<', None, None, None, -1, -1, 0)) + assert a.dtype.__reduce__() == (dtype, ('i4', 0, 1), (3, E, None, None, None, -1, -1, 0)) assert loads(dumps(a.dtype)) == a.dtype assert np.dtype('bool').__reduce__() == (dtype, ('b1', 0, 1), (3, '|', None, None, None, -1, -1, 0)) assert np.dtype('|V16').__reduce__() == (dtype, ('V16', 0, 1), (3, '|', None, None, None, 16, 1, 0)) - assert np.dtype(('' d = np.dtype('f8') d.__setstate__((3, '|', (np.dtype('float64'), (2,)), None, None, 20, 1, 0)) assert d.str == ('<' if sys.byteorder == 'little' else '>') + 'f8' @@ -1201,7 +1210,7 @@ assert d.shape == (2,) assert d.itemsize == 8 assert d.subdtype is not None - assert repr(d) == "dtype(('' + assert str(dt) == "{'names':['f0','f1'], 'formats':['%si4','u1'], 'offsets':[0,4], 'itemsize':8, 'aligned':True}" % E dt = np.dtype([('f1', 'u1'), ('f0', 'i4')], align=True) - assert str(dt) == "{'names':['f1','f0'], 'formats':['u1',' Author: Richard Plangger Branch: s390x-backend Changeset: r81909:272b467ba7dd Date: 2016-01-22 10:24 +0100 http://bitbucket.org/pypy/pypy/changeset/272b467ba7dd/ Log: two more dtype tests fixed (endian issues) diff --git a/pypy/module/micronumpy/test/test_dtypes.py b/pypy/module/micronumpy/test/test_dtypes.py --- a/pypy/module/micronumpy/test/test_dtypes.py +++ b/pypy/module/micronumpy/test/test_dtypes.py @@ -924,6 +924,7 @@ def test_dtype_str(self): from numpy import dtype + import sys byteorder = self.native_prefix assert dtype('i8').str == byteorder + 'i8' assert dtype('')+'U7' assert dtype([('', 'f8')]).str == "|V8" assert dtype(('f8', 2)).str == "|V16" @@ -976,8 +978,12 @@ def test_isnative(self): from numpy import dtype + import sys assert dtype('i4').isnative == True - assert dtype('>i8').isnative == False + if sys.byteorder == 'big': + assert dtype('i8').isnative == False def test_any_all_nonzero(self): import numpy From pypy.commits at gmail.com Fri Jan 22 04:56:59 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 01:56:59 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: half way through the ndarray tests (endian issues) Message-ID: <56a1fceb.2aacc20a.4b24d.ffffd6c6@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81910:d1a60e575946 Date: 2016-01-22 10:56 +0100 http://bitbucket.org/pypy/pypy/changeset/d1a60e575946/ Log: half way through the ndarray tests (endian issues) diff --git a/pypy/module/micronumpy/test/test_ndarray.py b/pypy/module/micronumpy/test/test_ndarray.py --- a/pypy/module/micronumpy/test/test_ndarray.py +++ b/pypy/module/micronumpy/test/test_ndarray.py @@ -1791,6 +1791,7 @@ def test_scalar_view(self): from numpy import array + import sys a = array(3, dtype='int32') b = a.view(dtype='float32') assert b.shape == () @@ -1799,17 +1800,27 @@ assert exc.value[0] == "new type not compatible with array." exc = raises(TypeError, a.view, 'string') assert exc.value[0] == "data-type must not be 0-sized" - assert a.view('S4') == '\x03' + if sys.byteorder == 'big': + assert a.view('S4') == '\x00\x00\x00\x03' + else: + assert a.view('S4') == '\x03' a = array('abc1', dtype='c') assert (a == ['a', 'b', 'c', '1']).all() assert a.view('S4') == 'abc1' b = a.view([('a', 'i2'), ('b', 'i2')]) assert b.shape == (1,) - assert b[0][0] == 25185 - assert b[0][1] == 12643 + if sys.byteorder == 'big': + assert b[0][0] == 0x6162 + assert b[0][1] == 0x6331 + else: + assert b[0][0] == 25185 + assert b[0][1] == 12643 a = array([(1, 2)], dtype=[('a', 'int64'), ('b', 'int64')])[0] assert a.shape == () - assert a.view('S16') == '\x01' + '\x00' * 7 + '\x02' + if sys.byteorder == 'big': + assert a.view('S16') == '\x00' * 7 + '\x01' + '\x00' * 7 + '\x02' + else: + assert a.view('S16') == '\x01' + '\x00' * 7 + '\x02' a = array(2, dtype=' Author: Richard Plangger Branch: s390x-backend Changeset: r81911:47a85e21bb1b Date: 2016-01-22 12:00 +0100 http://bitbucket.org/pypy/pypy/changeset/47a85e21bb1b/ Log: and the other part of the bigendian issues (micronumpy tests) diff --git a/pypy/module/micronumpy/test/test_ndarray.py b/pypy/module/micronumpy/test/test_ndarray.py --- a/pypy/module/micronumpy/test/test_ndarray.py +++ b/pypy/module/micronumpy/test/test_ndarray.py @@ -3500,7 +3500,11 @@ BaseNumpyAppTest.setup_class.im_func(cls) cls.w_data = cls.space.wrap(struct.pack('dddd', 1, 2, 3, 4)) cls.w_fdata = cls.space.wrap(struct.pack('f', 2.3)) - cls.w_float16val = cls.space.wrap('\x00E') # 5.0 in float16 + import sys + if sys.byteorder == 'big': + cls.w_float16val = cls.space.wrap('E\x00') # 5.0 in float16 + else: + cls.w_float16val = cls.space.wrap('\x00E') # 5.0 in float16 cls.w_float32val = cls.space.wrap(struct.pack('f', 5.2)) cls.w_float64val = cls.space.wrap(struct.pack('d', 300.4)) cls.w_ulongval = cls.space.wrap(struct.pack('L', 12)) @@ -3608,9 +3612,15 @@ assert (t == []).all() u = fromstring("\x01\x00\x00\x00\x00\x00\x00\x00", dtype=int) if sys.maxint > 2 ** 31 - 1: - assert (u == [1]).all() + if sys.byteorder == 'big': + assert (u == [0x0100000000000000]).all() + else: + assert (u == [1]).all() else: - assert (u == [1, 0]).all() + if sys.byteorder == 'big': + assert (u == [0x01000000, 0]).all() + else: + assert (u == [1, 0]).all() v = fromstring("abcd", dtype="|S2") assert v[0] == "ab" assert v[1] == "cd" @@ -3667,9 +3677,15 @@ k = fromstring(self.float16val, dtype='float16') assert k[0] == dtype('float16').type(5.) dt = array([5], dtype='longfloat').dtype + print(dt.itemsize) if dt.itemsize == 8: - m = fromstring('\x00\x00\x00\x00\x00\x00\x14@', - dtype='float64') + import sys + if sys.byteorder == 'big': + m = fromstring('@\x14\x00\x00\x00\x00\x00\x00', + dtype='float64') + else: + m = fromstring('\x00\x00\x00\x00\x00\x00\x14@', + dtype='float64') elif dt.itemsize == 12: m = fromstring('\x00\x00\x00\x00\x00\x00\x00\xa0\x01@\x00\x00', dtype='float96') @@ -3691,8 +3707,13 @@ def test_tostring(self): from numpy import array - assert array([1, 2, 3], 'i2').tostring() == '\x01\x00\x02\x00\x03\x00' - assert array([1, 2, 3], 'i2')[::2].tostring() == '\x01\x00\x03\x00' + import sys + if sys.byteorder == 'big': + assert array([1, 2, 3], 'i2').tostring() == '\x00\x01\x00\x02\x00\x03' + assert array([1, 2, 3], 'i2')[::2].tostring() == '\x00\x01\x00\x03' + else: + assert array([1, 2, 3], 'i2').tostring() == '\x01\x00\x02\x00\x03\x00' + assert array([1, 2, 3], 'i2')[::2].tostring() == '\x01\x00\x03\x00' assert array([1, 2, 3], 'i2')[::2].tostring() == '\x00\x01\x00\x03' assert array(0, dtype='i2').tostring() == '\x00\x00' @@ -4188,7 +4209,10 @@ v = a.view(('float32', 4)) assert v.dtype == np.dtype('float32') assert v.shape == (10, 4) - assert v[0][-1] == 2.53125 + if sys.byteorder == 'big': + assert v[0][-2] == 2.53125 + else: + assert v[0][-1] == 2.53125 exc = raises(ValueError, "a.view(('float32', 2))") assert exc.value[0] == 'new type not compatible with array.' diff --git a/pypy/module/micronumpy/test/test_scalar.py b/pypy/module/micronumpy/test/test_scalar.py --- a/pypy/module/micronumpy/test/test_scalar.py +++ b/pypy/module/micronumpy/test/test_scalar.py @@ -109,6 +109,7 @@ def test_pickle(self): from numpy import dtype, zeros + import sys try: from numpy.core.multiarray import scalar except ImportError: @@ -119,9 +120,11 @@ f = dtype('float64').type(13.37) c = dtype('complex128').type(13 + 37.j) - assert i.__reduce__() == (scalar, (dtype('int32'), '9\x05\x00\x00')) - assert f.__reduce__() == (scalar, (dtype('float64'), '=\n\xd7\xa3p\xbd*@')) - assert c.__reduce__() == (scalar, (dtype('complex128'), '\x00\x00\x00\x00\x00\x00*@\x00\x00\x00\x00\x00\x80B@')) + swap = lambda s: (''.join(reversed(s))) if sys.byteorder == 'big' else s + assert i.__reduce__() == (scalar, (dtype('int32'), swap('9\x05\x00\x00'))) + assert f.__reduce__() == (scalar, (dtype('float64'), swap('=\n\xd7\xa3p\xbd*@'))) + assert c.__reduce__() == (scalar, (dtype('complex128'), swap('\x00\x00\x00\x00\x00\x00*@') + \ + swap('\x00\x00\x00\x00\x00\x80B@'))) assert loads(dumps(i)) == i assert loads(dumps(f)) == f @@ -256,13 +259,20 @@ assert t < 7e-323 t = s.view('complex64') assert type(t) is np.complex64 - assert 0 < t.real < 1 - assert t.imag == 0 + if sys.byteorder == 'big': + assert 0 < t.imag < 1 + assert t.real == 0 + else: + assert 0 < t.real < 1 + assert t.imag == 0 exc = raises(TypeError, s.view, 'string') assert exc.value[0] == "data-type must not be 0-sized" t = s.view('S8') assert type(t) is np.string_ - assert t == '\x0c' + if sys.byteorder == 'big': + assert t == '\x00' * 7 + '\x0c' + else: + assert t == '\x0c' s = np.dtype('string').type('abc1') assert s.view('S4') == 'abc1' if '__pypy__' in sys.builtin_module_names: diff --git a/pypy/module/micronumpy/test/test_selection.py b/pypy/module/micronumpy/test/test_selection.py --- a/pypy/module/micronumpy/test/test_selection.py +++ b/pypy/module/micronumpy/test/test_selection.py @@ -327,10 +327,15 @@ # tests from numpy/core/tests/test_regression.py def test_sort_bigendian(self): from numpy import array, dtype - a = array(range(11), dtype='float64') - c = a.astype(dtype('' D.__module__ = 'mod' mod = new.module('mod') mod.D = D @@ -510,7 +511,7 @@ tp9 Rp10 (I3 - S'<' + S'{E}' p11 NNNI-1 I-1 @@ -520,7 +521,7 @@ S'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00@' p13 tp14 - b.'''.replace(' ','') + b.'''.replace(' ','').format(E=E) for ss,sn in zip(s.split('\n')[1:],s_from_numpy.split('\n')[1:]): if len(ss)>10: # ignore binary data, it will be checked later From pypy.commits at gmail.com Fri Jan 22 06:47:32 2016 From: pypy.commits at gmail.com (arigo) Date: Fri, 22 Jan 2016 03:47:32 -0800 (PST) Subject: [pypy-commit] pypy.org extradoc: update the values Message-ID: <56a216d4.cf0b1c0a.fc7b4.ffffdb3b@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r694:50a0e13fff46 Date: 2016-01-22 12:47 +0100 http://bitbucket.org/pypy/pypy.org/changeset/50a0e13fff46/ Log: update the values diff --git a/don1.html b/don1.html --- a/don1.html +++ b/don1.html @@ -9,13 +9,13 @@ - $62736 of $105000 (59.7%) + $62755 of $105000 (59.8%)

@@ -23,7 +23,7 @@

This donation goes towards supporting Python 3 in PyPy. Current status: - we have $7914 left + we have $7931 left in the account.
Read proposal

diff --git a/don4.html b/don4.html --- a/don4.html +++ b/don4.html @@ -17,7 +17,7 @@ 2nd call: - $30344 of $80000 (37.9%) + $30354 of $80000 (37.9%)

@@ -25,7 +25,7 @@

This donation goes towards supporting the Transactional Memory in PyPy. Current status: - we have $23076 left + we have $23084 left in the account.
Read proposal (2nd call)

From pypy.commits at gmail.com Fri Jan 22 06:52:44 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 03:52:44 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: fixed callsite of clibffi with the same big endian issues as found yesterday evening Message-ID: <56a2180c.6918c20a.261a0.fffffecf@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81912:6840459f9b22 Date: 2016-01-22 12:51 +0100 http://bitbucket.org/pypy/pypy/changeset/6840459f9b22/ Log: fixed callsite of clibffi with the same big endian issues as found yesterday evening diff --git a/pypy/module/micronumpy/test/test_ndarray.py b/pypy/module/micronumpy/test/test_ndarray.py --- a/pypy/module/micronumpy/test/test_ndarray.py +++ b/pypy/module/micronumpy/test/test_ndarray.py @@ -4209,6 +4209,7 @@ v = a.view(('float32', 4)) assert v.dtype == np.dtype('float32') assert v.shape == (10, 4) + import sys if sys.byteorder == 'big': assert v[0][-2] == 2.53125 else: diff --git a/rpython/rlib/clibffi.py b/rpython/rlib/clibffi.py --- a/rpython/rlib/clibffi.py +++ b/rpython/rlib/clibffi.py @@ -597,6 +597,9 @@ size = adjust_return_size(intmask(restype.c_size)) self.ll_result = lltype.malloc(rffi.VOIDP.TO, size, flavor='raw') + self.restype_size = intmask(restype.c_size) + else: + self.restype_size = -1 def push_arg(self, value): #if self.pushed_args == self.argnum: @@ -633,7 +636,12 @@ rffi.cast(VOIDPP, self.ll_args)) if RES_TP is not lltype.Void: TP = lltype.Ptr(rffi.CArray(RES_TP)) - res = rffi.cast(TP, self.ll_result)[0] + ptr = self.ll_result + if _BIG_ENDIAN and self.restype_size != -1: + # we get a 8 byte value in big endian + n = rffi.sizeof(lltype.Signed) - self.restype_size + ptr = rffi.ptradd(ptr, n) + res = rffi.cast(TP, ptr)[0] else: res = None self._clean_args() diff --git a/rpython/rlib/rstruct/test/test_runpack.py b/rpython/rlib/rstruct/test/test_runpack.py --- a/rpython/rlib/rstruct/test/test_runpack.py +++ b/rpython/rlib/rstruct/test/test_runpack.py @@ -6,11 +6,13 @@ class TestRStruct(BaseRtypingTest): def test_unpack(self): + import sys pad = '\x00' * (LONG_BIT//8-1) # 3 or 7 null bytes def fn(): return runpack('sll', 'a'+pad+'\x03'+pad+'\x04'+pad)[1] - assert fn() == 3 - assert self.interpret(fn, []) == 3 + result = 3 if sys.byteorder == 'little' else 3 << (LONG_BIT-8) + assert fn() == result + assert self.interpret(fn, []) == result def test_unpack_2(self): data = struct.pack('iiii', 0, 1, 2, 4) diff --git a/rpython/rlib/test/test_clibffi.py b/rpython/rlib/test/test_clibffi.py --- a/rpython/rlib/test/test_clibffi.py +++ b/rpython/rlib/test/test_clibffi.py @@ -181,11 +181,12 @@ p_a2 = rffi.cast(rffi.VOIDPP, ll_args[1])[0] a1 = rffi.cast(rffi.INTP, p_a1)[0] a2 = rffi.cast(rffi.INTP, p_a2)[0] - res = rffi.cast(rffi.INTP, ll_res) + res = rffi.cast(rffi.SIGNEDP, ll_res) + # must store a full ffi arg! if a1 > a2: - res[0] = rffi.cast(rffi.INT, 1) + res[0] = 1 else: - res[0] = rffi.cast(rffi.INT, -1) + res[0] = -1 ptr = CallbackFuncPtr([ffi_type_pointer, ffi_type_pointer], ffi_type_sint, callback) From pypy.commits at gmail.com Fri Jan 22 08:29:04 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 05:29:04 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: we can have is_recursive but no get_unique_id Message-ID: <56a22ea0.a3f6c20a.6894d.1d0c@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81913:9fcd011d9607 Date: 2016-01-22 14:28 +0100 http://bitbucket.org/pypy/pypy/changeset/9fcd011d9607/ Log: we can have is_recursive but no get_unique_id diff --git a/rpython/rlib/jit.py b/rpython/rlib/jit.py --- a/rpython/rlib/jit.py +++ b/rpython/rlib/jit.py @@ -623,8 +623,8 @@ raise AttributeError("no 'greens' or 'reds' supplied") if virtualizables is not None: self.virtualizables = virtualizables - if get_unique_id is not None or is_recursive: - assert get_unique_id is not None and is_recursive, "get_unique_id and is_recursive must be specified at the same time" + if get_unique_id is not None: + assert is_recursive, "get_unique_id and is_recursive must be specified at the same time" for v in self.virtualizables: assert v in self.reds # if reds are automatic, they won't be passed to jit_merge_point, so From pypy.commits at gmail.com Fri Jan 22 08:57:11 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 05:57:11 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: macros (e.g. WCOREDUMP) got parameter type Signed, on little endian this does not make a difference, but it does on big endian. changed to rffi.INT Message-ID: <56a23537.11181c0a.689f.06de@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81914:43db866bfbb3 Date: 2016-01-22 14:56 +0100 http://bitbucket.org/pypy/pypy/changeset/43db866bfbb3/ Log: macros (e.g. WCOREDUMP) got parameter type Signed, on little endian this does not make a difference, but it does on big endian. changed to rffi.INT diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -871,7 +871,7 @@ lltype.free(status_p, flavor='raw') def _make_waitmacro(name): - c_func = external(name, [lltype.Signed], lltype.Signed, + c_func = external(name, [rffi.INT], lltype.Signed, macro=_MACRO_ON_POSIX) returning_int = name in ('WEXITSTATUS', 'WSTOPSIG', 'WTERMSIG') From pypy.commits at gmail.com Fri Jan 22 09:13:09 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 06:13:09 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: fix that test Message-ID: <56a238f5.e935c20a.8d22c.334b@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81915:900de81cdc5e Date: 2016-01-22 15:11 +0100 http://bitbucket.org/pypy/pypy/changeset/900de81cdc5e/ Log: fix that test diff --git a/rpython/jit/metainterp/test/test_recursive.py b/rpython/jit/metainterp/test/test_recursive.py --- a/rpython/jit/metainterp/test/test_recursive.py +++ b/rpython/jit/metainterp/test/test_recursive.py @@ -1312,7 +1312,7 @@ return (code + 1) * 2 driver = JitDriver(greens=["pc", "code"], reds='auto', - get_unique_id=get_unique_id) + get_unique_id=get_unique_id, is_recursive=True) def f(pc, code): i = 0 From pypy.commits at gmail.com Fri Jan 22 09:13:10 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 06:13:10 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: import os x related hacks for threads & vmprof Message-ID: <56a238f6.44e21c0a.7de24.10a7@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81916:670f46cb2434 Date: 2016-01-22 15:12 +0100 http://bitbucket.org/pypy/pypy/changeset/670f46cb2434/ Log: import os x related hacks for threads & vmprof diff --git a/rpython/rlib/rvmprof/src/vmprof_common.h b/rpython/rlib/rvmprof/src/vmprof_common.h new file mode 100644 --- /dev/null +++ b/rpython/rlib/rvmprof/src/vmprof_common.h @@ -0,0 +1,71 @@ +#include + +#define MAX_FUNC_NAME 1024 + +static int profile_file = -1; +static long prepare_interval_usec = 0; +static long profile_interval_usec = 0; +static int opened_profile(char *interp_name); + +#define MAX_STACK_DEPTH \ + ((SINGLE_BUF_SIZE - sizeof(struct prof_stacktrace_s)) / sizeof(void *)) + +#define MARKER_STACKTRACE '\x01' +#define MARKER_VIRTUAL_IP '\x02' +#define MARKER_TRAILER '\x03' +#define MARKER_INTERP_NAME '\x04' /* deprecated */ +#define MARKER_HEADER '\x05' + +#define VERSION_BASE '\x00' +#define VERSION_THREAD_ID '\x01' + +typedef struct prof_stacktrace_s { + char padding[sizeof(long) - 1]; + char marker; + long count, depth; + void *stack[]; +} prof_stacktrace_s; + + +RPY_EXTERN +char *vmprof_init(int fd, double interval, char *interp_name) +{ + if (interval < 1e-6 || interval >= 1.0) + return "bad value for 'interval'"; + prepare_interval_usec = (int)(interval * 1000000.0); + + if (prepare_concurrent_bufs() < 0) + return "out of memory"; + + assert(fd >= 0); + profile_file = fd; + if (opened_profile(interp_name) < 0) { + profile_file = -1; + return strerror(errno); + } + return NULL; +} + +static int _write_all(const char *buf, size_t bufsize); + +static int opened_profile(char *interp_name) +{ + struct { + long hdr[5]; + char interp_name[259]; + } header; + + size_t namelen = strnlen(interp_name, 255); + + header.hdr[0] = 0; + header.hdr[1] = 3; + header.hdr[2] = 0; + header.hdr[3] = prepare_interval_usec; + header.hdr[4] = 0; + header.interp_name[0] = MARKER_HEADER; + header.interp_name[1] = '\x00'; + header.interp_name[2] = VERSION_THREAD_ID; + header.interp_name[3] = namelen; + memcpy(&header.interp_name[4], interp_name, namelen); + return _write_all((char*)&header, 5 * sizeof(long) + 4 + namelen); +} diff --git a/rpython/rlib/rvmprof/src/vmprof_main.h b/rpython/rlib/rvmprof/src/vmprof_main.h --- a/rpython/rlib/rvmprof/src/vmprof_main.h +++ b/rpython/rlib/rvmprof/src/vmprof_main.h @@ -35,11 +35,12 @@ #include "vmprof_getpc.h" #include "vmprof_mt.h" #include "vmprof_stack.h" +#include "vmprof_common.h" /************************************************************/ -static int profile_file = -1; static long prepare_interval_usec; +static long saved_profile_file; static struct profbuf_s *volatile current_codes; static void *(*mainloop_get_virtual_ip)(char *) = 0; @@ -47,26 +48,6 @@ static void flush_codes(void); - -RPY_EXTERN -char *vmprof_init(int fd, double interval, char *interp_name) -{ - if (interval < 1e-6 || interval >= 1.0) - return "bad value for 'interval'"; - prepare_interval_usec = (int)(interval * 1000000.0); - - if (prepare_concurrent_bufs() < 0) - return "out of memory"; - - assert(fd >= 0); - profile_file = fd; - if (opened_profile(interp_name) < 0) { - profile_file = -1; - return strerror(errno); - } - return NULL; -} - /************************************************************/ /* value: last bit is 1 if signals must be ignored; all other bits @@ -94,28 +75,6 @@ * ************************************************************* */ -#define MAX_FUNC_NAME 128 -#define MAX_STACK_DEPTH \ - ((SINGLE_BUF_SIZE - sizeof(struct prof_stacktrace_s)) / sizeof(void *)) - -#define MARKER_STACKTRACE '\x01' -#define MARKER_VIRTUAL_IP '\x02' -#define MARKER_TRAILER '\x03' -#define MARKER_INTERP_NAME '\x04' /* deprecated */ -#define MARKER_HEADER '\x05' - -#define VERSION_BASE '\x00' -#define VERSION_THREAD_ID '\x01' -#define VERSION_TAG '\x02' - -struct prof_stacktrace_s { - char padding[sizeof(long) - 1]; - char marker; - long count, depth; - intptr_t stack[]; -}; - -static long profile_interval_usec = 0; static char atfork_hook_installed = 0; @@ -194,8 +153,43 @@ * ************************************************************* */ +#include + +volatile int spinlock; +jmp_buf restore_point; + +static void segfault_handler(int arg) +{ + longjmp(restore_point, SIGSEGV); +} + static void sigprof_handler(int sig_nr, siginfo_t* info, void *ucontext) { +#ifdef __APPLE__ + // TERRIBLE HACK AHEAD + // on OS X, the thread local storage is sometimes uninitialized + // when the signal handler runs - it means it's impossible to read errno + // or call any syscall or read PyThread_Current or pthread_self. Additionally, + // it seems impossible to read the register gs. + // here we register segfault handler (all guarded by a spinlock) and call + // longjmp in case segfault happens while reading a thread local + while (__sync_lock_test_and_set(&spinlock, 1)) { + } + signal(SIGSEGV, &segfault_handler); + int fault_code = setjmp(restore_point); + if (fault_code == 0) { + pthread_self(); + get_current_thread_id(); + } else { + signal(SIGSEGV, SIG_DFL); + __sync_synchronize(); + spinlock = 0; + return; + } + signal(SIGSEGV, SIG_DFL); + __sync_synchronize(); + spinlock = 0; +#endif long val = __sync_fetch_and_add(&signal_handler_value, 2L); if ((val & 1) == 0) { @@ -212,10 +206,8 @@ struct prof_stacktrace_s *st = (struct prof_stacktrace_s *)p->data; st->marker = MARKER_STACKTRACE; st->count = 1; - //st->stack[0] = GetPC((ucontext_t*)ucontext); depth = get_stack_trace(st->stack, MAX_STACK_DEPTH-2, GetPC((ucontext_t*)ucontext), ucontext); - //depth++; // To account for pc value in stack[0]; st->depth = depth; st->stack[depth++] = get_current_thread_id(); p->data_offset = offsetof(struct prof_stacktrace_s, marker); @@ -280,12 +272,15 @@ static void atfork_disable_timer(void) { if (profile_interval_usec > 0) { + saved_profile_file = profile_file; + profile_file = -1; remove_sigprof_timer(); } } static void atfork_enable_timer(void) { if (profile_interval_usec > 0) { + profile_file = saved_profile_file; install_sigprof_timer(); } } @@ -332,7 +327,7 @@ return -1; } -static int _write_all(const void *buf, size_t bufsize) +static int _write_all(const char *buf, size_t bufsize) { while (bufsize > 0) { ssize_t count = write(profile_file, buf, bufsize); @@ -344,29 +339,6 @@ return 0; } -static int opened_profile(char *interp_name) -{ - struct { - long hdr[5]; - char interp_name[259]; - } header; - - size_t namelen = strnlen(interp_name, 255); - current_codes = NULL; - - header.hdr[0] = 0; - header.hdr[1] = 3; - header.hdr[2] = 0; - header.hdr[3] = prepare_interval_usec; - header.hdr[4] = 0; - header.interp_name[0] = MARKER_HEADER; - header.interp_name[1] = '\x00'; - header.interp_name[2] = VERSION_TAG; - header.interp_name[3] = namelen; - memcpy(&header.interp_name[4], interp_name, namelen); - return _write_all(&header, 5 * sizeof(long) + 4 + namelen); -} - static int close_profile(void) { unsigned char marker = MARKER_TRAILER; @@ -404,6 +376,9 @@ struct profbuf_s *p; char *t; + if (profile_file == -1) + return 0; // silently don't write it + retry: p = current_codes; if (p != NULL) { @@ -411,7 +386,7 @@ /* grabbed 'current_codes': we will append the current block to it if it contains enough room */ size_t freesize = SINGLE_BUF_SIZE - p->data_size; - if (freesize < blocklen) { + if (freesize < (size_t)blocklen) { /* full: flush it */ commit_buffer(profile_file, p); p = NULL; From pypy.commits at gmail.com Fri Jan 22 09:19:34 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 06:19:34 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: change pdb to some asserts Message-ID: <56a23a76.8673c20a.7e860.3255@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81917:ffec42c76f43 Date: 2016-01-22 15:18 +0100 http://bitbucket.org/pypy/pypy/changeset/ffec42c76f43/ Log: change pdb to some asserts diff --git a/rpython/jit/backend/llsupport/test/zrpy_vmprof_test.py b/rpython/jit/backend/llsupport/test/zrpy_vmprof_test.py --- a/rpython/jit/backend/llsupport/test/zrpy_vmprof_test.py +++ b/rpython/jit/backend/llsupport/test/zrpy_vmprof_test.py @@ -74,8 +74,8 @@ tmpfile = str(udir.join('test_rvmprof')) stats = read_profile(tmpfile) t = stats.get_tree() - import pdb - pdb.set_trace() + assert t.name == 'py:x:foo:3' + assert len(t.children) == 1 # jit self.meta_interp(f, [1000000], inline=True) try: From pypy.commits at gmail.com Fri Jan 22 10:46:17 2016 From: pypy.commits at gmail.com (arigo) Date: Fri, 22 Jan 2016 07:46:17 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: Fix for some tests: don't use .offset directly, call .getoffset() Message-ID: <56a24ec9.cdb81c0a.8c988.2a4c@mx.google.com> Author: Armin Rigo Branch: vmprof-newstack Changeset: r81918:fba43bc13dda Date: 2016-01-22 16:41 +0100 http://bitbucket.org/pypy/pypy/changeset/fba43bc13dda/ Log: Fix for some tests: don't use .offset directly, call .getoffset() diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py --- a/rpython/jit/backend/x86/assembler.py +++ b/rpython/jit/backend/x86/assembler.py @@ -857,7 +857,8 @@ # eax = address in the stack of a 3-words struct vmprof_stack_s self.mc.LEA_rs(eax.value, (FRAME_FIXED_SIZE - 4) * WORD) # old = current value of vmprof_tl_stack - self.mc.MOV_rm(old.value, (tloc.value, cintf.vmprof_tl_stack.offset)) + offset = cintf.vmprof_tl_stack.getoffset() + self.mc.MOV_rm(old.value, (tloc.value, offset)) # eax->next = old self.mc.MOV_mr((eax.value, 0), old.value) # eax->value = my esp @@ -865,7 +866,7 @@ # eax->kind = VMPROF_JITTED_TAG self.mc.MOV_mi((eax.value, WORD * 2), VMPROF_JITTED_TAG) # save in vmprof_tl_stack the new eax - self.mc.MOV_mr((tloc.value, cintf.vmprof_tl_stack.offset), eax.value) + self.mc.MOV_mr((tloc.value, offset), eax.value) def _call_footer_vmprof(self): from rpython.rlib.rvmprof.rvmprof import cintf @@ -874,7 +875,8 @@ # eax = (our local vmprof_tl_stack).next self.mc.MOV_rs(eax.value, (FRAME_FIXED_SIZE - 4 + 0) * WORD) # save in vmprof_tl_stack the value eax - self.mc.MOV_mr((edx.value, cintf.vmprof_tl_stack.offset), eax.value) + offset = cintf.vmprof_tl_stack.getoffset() + self.mc.MOV_mr((edx.value, offset), eax.value) def _call_header(self): self.mc.SUB_ri(esp.value, FRAME_FIXED_SIZE * WORD) diff --git a/rpython/rlib/rthread.py b/rpython/rlib/rthread.py --- a/rpython/rlib/rthread.py +++ b/rpython/rlib/rthread.py @@ -308,7 +308,7 @@ offset = CDefinedIntSymbolic('RPY_TLOFS_%s' % self.fieldname, default='?') offset.loop_invariant = loop_invariant - self.offset = offset + self._offset = offset def getraw(): if we_are_translated(): @@ -364,7 +364,7 @@ ThreadLocalField.__init__(self, lltype.Signed, 'tlref%d' % unique_id, loop_invariant=loop_invariant) setraw = self.setraw - offset = self.offset + offset = self._offset def get(): if we_are_translated(): From pypy.commits at gmail.com Fri Jan 22 11:32:27 2016 From: pypy.commits at gmail.com (arigo) Date: Fri, 22 Jan 2016 08:32:27 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: fix Message-ID: <56a2599b.e906c20a.3af87.5cf5@mx.google.com> Author: Armin Rigo Branch: vmprof-newstack Changeset: r81919:e45af82e0252 Date: 2016-01-22 17:31 +0100 http://bitbucket.org/pypy/pypy/changeset/e45af82e0252/ Log: fix diff --git a/rpython/jit/codewriter/test/test_jtransform.py b/rpython/jit/codewriter/test/test_jtransform.py --- a/rpython/jit/codewriter/test/test_jtransform.py +++ b/rpython/jit/codewriter/test/test_jtransform.py @@ -1332,7 +1332,7 @@ tlfield = ThreadLocalField(lltype.Signed, 'foobar_test_', loop_invariant=loop_inv) OS_THREADLOCALREF_GET = effectinfo.EffectInfo.OS_THREADLOCALREF_GET - c = const(tlfield.offset) + c = const(tlfield.getoffset()) v = varoftype(lltype.Signed) op = SpaceOperation('threadlocalref_get', [c], v) cc = FakeBuiltinCallControl() From pypy.commits at gmail.com Fri Jan 22 11:46:06 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 08:46:06 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: fix the test Message-ID: <56a25cce.a867c20a.41779.660b@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81920:b5543f370125 Date: 2016-01-22 17:33 +0100 http://bitbucket.org/pypy/pypy/changeset/b5543f370125/ Log: fix the test diff --git a/rpython/jit/metainterp/test/test_jitdriver.py b/rpython/jit/metainterp/test/test_jitdriver.py --- a/rpython/jit/metainterp/test/test_jitdriver.py +++ b/rpython/jit/metainterp/test/test_jitdriver.py @@ -193,7 +193,7 @@ return pc + 1 driver = JitDriver(greens=["pc"], reds='auto', - get_unique_id=get_unique_id) + get_unique_id=get_unique_id, is_recursive=True) def f(arg): i = 0 From pypy.commits at gmail.com Fri Jan 22 11:46:08 2016 From: pypy.commits at gmail.com (fijal) Date: Fri, 22 Jan 2016 08:46:08 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: merge Message-ID: <56a25cd0.e906c20a.3af87.61f5@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81921:4b79f234a111 Date: 2016-01-22 17:45 +0100 http://bitbucket.org/pypy/pypy/changeset/4b79f234a111/ Log: merge diff --git a/rpython/jit/codewriter/test/test_jtransform.py b/rpython/jit/codewriter/test/test_jtransform.py --- a/rpython/jit/codewriter/test/test_jtransform.py +++ b/rpython/jit/codewriter/test/test_jtransform.py @@ -1332,7 +1332,7 @@ tlfield = ThreadLocalField(lltype.Signed, 'foobar_test_', loop_invariant=loop_inv) OS_THREADLOCALREF_GET = effectinfo.EffectInfo.OS_THREADLOCALREF_GET - c = const(tlfield.offset) + c = const(tlfield.getoffset()) v = varoftype(lltype.Signed) op = SpaceOperation('threadlocalref_get', [c], v) cc = FakeBuiltinCallControl() From pypy.commits at gmail.com Fri Jan 22 12:04:55 2016 From: pypy.commits at gmail.com (plan_rich) Date: Fri, 22 Jan 2016 09:04:55 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: added comment to my last commit Message-ID: <56a26137.a151c20a.34a75.733a@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81922:dcf9b353608e Date: 2016-01-22 18:03 +0100 http://bitbucket.org/pypy/pypy/changeset/dcf9b353608e/ Log: added comment to my last commit diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -871,6 +871,9 @@ lltype.free(status_p, flavor='raw') def _make_waitmacro(name): + # note that rffi.INT as first parameter type is intentional. + # on s390x providing a lltype.Signed as param type, the + # macro wrapper function will always return 0 c_func = external(name, [rffi.INT], lltype.Signed, macro=_MACRO_ON_POSIX) returning_int = name in ('WEXITSTATUS', 'WSTOPSIG', 'WTERMSIG') From pypy.commits at gmail.com Sat Jan 23 01:51:47 2016 From: pypy.commits at gmail.com (cfbolz) Date: Fri, 22 Jan 2016 22:51:47 -0800 (PST) Subject: [pypy-commit] pypy value-profiling: fix interaction of known class lists and cpyext Message-ID: <56a32303.2467c20a.95f27.30c0@mx.google.com> Author: Carl Friedrich Bolz Branch: value-profiling Changeset: r81923:f66ef9ad7656 Date: 2016-01-22 17:47 +0100 http://bitbucket.org/pypy/pypy/changeset/f66ef9ad7656/ Log: fix interaction of known class lists and cpyext diff --git a/pypy/objspace/std/listobject.py b/pypy/objspace/std/listobject.py --- a/pypy/objspace/std/listobject.py +++ b/pypy/objspace/std/listobject.py @@ -88,6 +88,10 @@ return space.fromcache(EmptyListStrategy) w_firstobj = list_w[0] + if w_firstobj is None: + # this is done by cpyext: + return space.fromcache(ObjectListStrategy) + check_int_or_float = False if type(w_firstobj) is W_IntObject: diff --git a/pypy/objspace/std/test/test_listobject.py b/pypy/objspace/std/test/test_listobject.py --- a/pypy/objspace/std/test/test_listobject.py +++ b/pypy/objspace/std/test/test_listobject.py @@ -417,6 +417,13 @@ assert isinstance(w_lst.strategy, SizeListStrategy) assert w_lst.strategy.sizehint == 13 + def test_newlist_with_interplevel_None(self): + # needed for cpyext + space = self.space + w_lst = space.newlist([None] * 10) + assert w_lst.strategy._known_cls is None + + def test_find_fast_on_intlist(self, monkeypatch): monkeypatch.setattr(self.space, "eq_w", None) w = self.space.wrap From pypy.commits at gmail.com Sat Jan 23 08:22:11 2016 From: pypy.commits at gmail.com (fijal) Date: Sat, 23 Jan 2016 05:22:11 -0800 (PST) Subject: [pypy-commit] pypy vmprof-newstack: cargo-culting - maybe we should load esi first Message-ID: <56a37e83.c8b3c20a.8831c.ffff9cc7@mx.google.com> Author: fijal Branch: vmprof-newstack Changeset: r81924:9da0fc568a35 Date: 2016-01-23 14:21 +0100 http://bitbucket.org/pypy/pypy/changeset/9da0fc568a35/ Log: cargo-culting - maybe we should load esi first diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py --- a/rpython/jit/backend/x86/assembler.py +++ b/rpython/jit/backend/x86/assembler.py @@ -881,10 +881,11 @@ def _call_header(self): self.mc.SUB_ri(esp.value, FRAME_FIXED_SIZE * WORD) self.mc.MOV_sr(PASS_ON_MY_FRAME * WORD, ebp.value) + if IS_X86_64: + self.mc.MOV_sr(THREADLOCAL_OFS, esi.value) if self.cpu.translate_support_code: self._call_header_vmprof() # on X86_64, this uses esi if IS_X86_64: - self.mc.MOV_sr(THREADLOCAL_OFS, esi.value) self.mc.MOV_rr(ebp.value, edi.value) else: self.mc.MOV_rs(ebp.value, (FRAME_FIXED_SIZE + 1) * WORD) From pypy.commits at gmail.com Sat Jan 23 17:35:21 2016 From: pypy.commits at gmail.com (sbauman) Date: Sat, 23 Jan 2016 14:35:21 -0800 (PST) Subject: [pypy-commit] pypy default: Actually remove GETFIELD_GC_PURE_* from the opcode space Message-ID: <56a40029.17941c0a.2b56c.247a@mx.google.com> Author: Spenser Andrew Bauman Branch: Changeset: r81925:ceefeb71a4eb Date: 2016-01-23 16:01 -0500 http://bitbucket.org/pypy/pypy/changeset/ceefeb71a4eb/ Log: Actually remove GETFIELD_GC_PURE_* from the opcode space diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py --- a/rpython/jit/metainterp/resoperation.py +++ b/rpython/jit/metainterp/resoperation.py @@ -1154,7 +1154,6 @@ 'ARRAYLEN_GC/1d/i', 'STRLEN/1/i', 'STRGETITEM/2/i', - 'GETFIELD_GC_PURE/1d/rfi', 'GETARRAYITEM_GC_PURE/2d/rfi', #'GETFIELD_RAW_PURE/1d/rfi', these two operations not useful and #'GETARRAYITEM_RAW_PURE/2d/fi', dangerous when unrolling speculatively From pypy.commits at gmail.com Sat Jan 23 17:35:23 2016 From: pypy.commits at gmail.com (sbauman) Date: Sat, 23 Jan 2016 14:35:23 -0800 (PST) Subject: [pypy-commit] pypy default: Actually remove all references to GETFIELD_GC_PURE_* opcode Message-ID: <56a4002b.c9ebc20a.f7f2d.4068@mx.google.com> Author: Spenser Andrew Bauman Branch: Changeset: r81926:013f2f131193 Date: 2016-01-23 17:33 -0500 http://bitbucket.org/pypy/pypy/changeset/013f2f131193/ Log: Actually remove all references to GETFIELD_GC_PURE_* opcode diff --git a/rpython/jit/backend/llsupport/rewrite.py b/rpython/jit/backend/llsupport/rewrite.py --- a/rpython/jit/backend/llsupport/rewrite.py +++ b/rpython/jit/backend/llsupport/rewrite.py @@ -234,7 +234,6 @@ self.emit_gc_store_or_indexed(op, ptr_box, index_box, value_box, fieldsize, itemsize, ofs) elif op.getopnum() in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R, - rop.GETFIELD_GC_PURE_I, rop.GETFIELD_GC_PURE_F, rop.GETFIELD_GC_PURE_R, rop.GETFIELD_RAW_I, rop.GETFIELD_RAW_F, rop.GETFIELD_RAW_R): ofs, itemsize, sign = unpack_fielddescr(op.getdescr()) ptr_box = op.getarg(0) From pypy.commits at gmail.com Sun Jan 24 03:08:03 2016 From: pypy.commits at gmail.com (arigo) Date: Sun, 24 Jan 2016 00:08:03 -0800 (PST) Subject: [pypy-commit] pypy.org extradoc: update the values Message-ID: <56a48663.46bb1c0a.7b018.ffff89bb@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r695:13468fc54cb3 Date: 2016-01-24 09:07 +0100 http://bitbucket.org/pypy/pypy.org/changeset/13468fc54cb3/ Log: update the values diff --git a/don1.html b/don1.html --- a/don1.html +++ b/don1.html @@ -15,7 +15,7 @@ - $62755 of $105000 (59.8%) + $62765 of $105000 (59.8%)

@@ -23,7 +23,7 @@

This donation goes towards supporting Python 3 in PyPy. Current status: - we have $7931 left + we have $7939 left in the account.
Read proposal

diff --git a/don4.html b/don4.html --- a/don4.html +++ b/don4.html @@ -9,7 +9,7 @@ @@ -17,7 +17,7 @@ 2nd call: - $30354 of $80000 (37.9%) + $30379 of $80000 (38.0%)

@@ -25,7 +25,7 @@

This donation goes towards supporting the Transactional Memory in PyPy. Current status: - we have $23084 left + we have $23106 left in the account.
Read proposal (2nd call)

From pypy.commits at gmail.com Sun Jan 24 15:16:30 2016 From: pypy.commits at gmail.com (arigo) Date: Sun, 24 Jan 2016 12:16:30 -0800 (PST) Subject: [pypy-commit] pypy.org extradoc: update the values Message-ID: <56a5311e.29cec20a.57b07.ffffcba4@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r696:43d71eb8c5e7 Date: 2016-01-24 21:16 +0100 http://bitbucket.org/pypy/pypy.org/changeset/43d71eb8c5e7/ Log: update the values diff --git a/don1.html b/don1.html --- a/don1.html +++ b/don1.html @@ -15,7 +15,7 @@ - $62765 of $105000 (59.8%) + $62793 of $105000 (59.8%)

@@ -23,7 +23,7 @@

This donation goes towards supporting Python 3 in PyPy. Current status: - we have $7939 left + we have $7965 left in the account.
Read proposal

From pypy.commits at gmail.com Mon Jan 25 05:03:00 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 02:03:00 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: removed lldebug properties from test translation (left them for debugging), fixed a bug when Const arg is provided to force_allocate_reg, it could have consumed on of the forbidden_vars Message-ID: <56a5f2d4.0f811c0a.3314d.1f80@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81927:3430d1f200fa Date: 2016-01-25 11:02 +0100 http://bitbucket.org/pypy/pypy/changeset/3430d1f200fa/ Log: removed lldebug properties from test translation (left them for debugging), fixed a bug when Const arg is provided to force_allocate_reg, it could have consumed on of the forbidden_vars diff --git a/rpython/jit/backend/llsupport/test/test_zrpy_gc_direct.py b/rpython/jit/backend/llsupport/test/test_zrpy_gc_direct.py --- a/rpython/jit/backend/llsupport/test/test_zrpy_gc_direct.py +++ b/rpython/jit/backend/llsupport/test/test_zrpy_gc_direct.py @@ -26,7 +26,6 @@ t = TranslationContext() t.config.translation.gc = "minimark" t.config.translation.gcremovetypeptr = gcremovetypeptr - t.config.translation.lldebug = True ann = t.buildannotator() ann.build_types(main, [s_list_of_strings], main_entry_point=True) rtyper = t.buildrtyper() diff --git a/rpython/jit/backend/llsupport/test/zrpy_gc_test.py b/rpython/jit/backend/llsupport/test/zrpy_gc_test.py --- a/rpython/jit/backend/llsupport/test/zrpy_gc_test.py +++ b/rpython/jit/backend/llsupport/test/zrpy_gc_test.py @@ -84,7 +84,6 @@ # t = TranslationContext() t.config.translation.gc = gc - t.config.translation.lldebug = True # pretty useful when debugging assembly if gc != 'boehm': t.config.translation.gcremovetypeptr = True for name, value in kwds.items(): diff --git a/rpython/jit/backend/zarch/arch.py b/rpython/jit/backend/zarch/arch.py --- a/rpython/jit/backend/zarch/arch.py +++ b/rpython/jit/backend/zarch/arch.py @@ -29,12 +29,12 @@ # # -THREADLOCAL_BYTES = 8 +# THREADLOCAL_BYTES = 8 # in reverse order to SP STD_FRAME_SIZE_IN_BYTES = 160 -THREADLOCAL_ADDR_OFFSET = 8 +THREADLOCAL_ADDR_OFFSET = 16 # at position of r2, but r2 is never saved!! assert STD_FRAME_SIZE_IN_BYTES % 2 == 0 diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -199,7 +199,7 @@ # mc.STMG(r.r10, r.r12, l.addr(10*WORD, r.SP)) mc.STG(r.r2, l.addr(2*WORD, r.SP)) - mc.STD(r.f0, l.addr(3*WORD, r.SP)) # slot of r3 is not used here + mc.STD(r.f0, l.addr(16*WORD, r.SP)) saved_regs = None saved_fp_regs = None else: @@ -231,11 +231,11 @@ # Note: if not 'for_frame', argument_loc is r0, which must carefully # not be overwritten above mc.STG(r.SP, l.addr(0, r.SP)) # store the backchain - mc.AGHI(r.SP, l.imm(-STD_FRAME_SIZE_IN_BYTES)) + mc.push_std_frame() mc.load_imm(mc.RAW_CALL_REG, func) mc.LGR(r.r2, argument_loc) mc.raw_call() - mc.AGHI(r.SP, l.imm(STD_FRAME_SIZE_IN_BYTES)) + mc.pop_std_frame() if for_frame: self._restore_exception(mc, RCS2, RCS3) @@ -251,7 +251,7 @@ if for_frame: mc.LMG(r.r10, r.r12, l.addr(10*WORD, r.SP)) mc.LG(r.r2, l.addr(2*WORD, r.SP)) - mc.LD(r.f0, l.addr(3*WORD, r.SP)) # slot of r3 is not used here + mc.LD(r.f0, l.addr(16*WORD, r.SP)) else: self._pop_core_regs_from_jitframe(mc, saved_regs) self._pop_fp_regs_from_jitframe(mc, saved_fp_regs) @@ -516,13 +516,13 @@ # registers). mc = InstrBuilder() # - self._push_core_regs_to_jitframe(mc, [r.r14]) # store the link on the jit frame - # Do the call + # store the link backwards + self.mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) mc.push_std_frame() + mc.LGR(r.r2, r.SP) mc.load_imm(mc.RAW_CALL_REG, slowpathaddr) mc.raw_call() - mc.pop_std_frame() # # Check if it raised StackOverflow mc.load_imm(r.SCRATCH, self.cpu.pos_exception()) @@ -531,9 +531,11 @@ # else we have an exception mc.cmp_op(r.SCRATCH, l.imm(0), imm=True) # - self._pop_core_regs_from_jitframe(mc, [r.r14]) # restore the link on the jit frame + size = STD_FRAME_SIZE_IN_BYTES + self.mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) # restore the link # So we return to our caller, conditionally if "EQ" mc.BCR(c.EQ, r.r14) + mc.trap() # debug if this is EVER executed! # # Else, jump to propagate_exception_path assert self.propagate_exception_path @@ -565,6 +567,7 @@ jmp_pos = self.mc.currpos() self.mc.reserve_cond_jump() + mc.push_std_frame() mc.load_imm(r.r14, self.stack_check_slowpath) mc.BASR(r.r14, r.r14) @@ -1006,6 +1009,9 @@ # save r3, the second argument, to THREADLOCAL_ADDR_OFFSET self.mc.STG(r.r3, l.addr(THREADLOCAL_ADDR_OFFSET, r.SP)) + # push a standard frame for any call + self.mc.push_std_frame() + # move the first argument to SPP: the jitframe object self.mc.LGR(r.SPP, r.r2) @@ -1049,30 +1055,10 @@ self._call_footer_shadowstack(gcrootmap) # restore registers r6-r15 - self.mc.LMG(r.r6, r.r15, l.addr(6*WORD, r.SP)) + size = STD_FRAME_SIZE_IN_BYTES + self.mc.LMG(r.r6, r.r15, l.addr(size+6*WORD, r.SP)) self.jmpto(r.r14) - def _push_all_regs_to_frame(self, mc, ignored_regs, withfloats, callee_only=False): - # Push all general purpose registers - base_ofs = self.cpu.get_baseofs_of_frame_field() - if callee_only: - regs = gpr_reg_mgr_cls.save_around_call_regs - else: - regs = gpr_reg_mgr_cls.all_regs - for gpr in regs: - if gpr not in ignored_regs: - v = gpr_reg_mgr_cls.all_reg_indexes[gpr.value] - mc.MOV_br(v * WORD + base_ofs, gpr.value) - if withfloats: - if IS_X86_64: - coeff = 1 - else: - coeff = 2 - # Push all XMM regs - ofs = len(gpr_reg_mgr_cls.all_regs) - for i in range(len(xmm_reg_mgr_cls.all_regs)): - mc.MOVSD_bx((ofs + i * coeff) * WORD + base_ofs, i) - def _push_core_regs_to_jitframe(self, mc, includes=r.registers): self._multiple_to_or_from_jitframe(mc, includes, store=True) diff --git a/rpython/jit/backend/zarch/callbuilder.py b/rpython/jit/backend/zarch/callbuilder.py --- a/rpython/jit/backend/zarch/callbuilder.py +++ b/rpython/jit/backend/zarch/callbuilder.py @@ -62,7 +62,6 @@ # called function will in turn call further functions (which must be passed the # address of the new frame). This stack grows downwards from high addresses # """ - self.subtracted_to_sp = STD_FRAME_SIZE_IN_BYTES gpr_regs = 0 fpr_regs = 0 @@ -151,7 +150,8 @@ # save the SP back chain self.mc.STG(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) # move the frame pointer - self.mc.LAY(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) + if self.subtracted_to_sp != 0: + self.mc.LAY(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) self.mc.raw_call() # self.ensure_correct_signzero_extension() @@ -180,7 +180,8 @@ def restore_stack_pointer(self): # it must at LEAST be 160 bytes - self.mc.LAY(r.SP, l.addr(self.subtracted_to_sp, r.SP)) + if self.subtracted_to_sp != 0: + self.mc.LAY(r.SP, l.addr(self.subtracted_to_sp, r.SP)) def load_result(self): assert (self.resloc is None or diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py --- a/rpython/jit/backend/zarch/regalloc.py +++ b/rpython/jit/backend/zarch/regalloc.py @@ -138,9 +138,9 @@ poolloc = l.pool(offset) if force_in_reg: if selected_reg is None: - tmp = TempVar() + tmp = TempInt() + selected_reg = self.force_allocate_reg(tmp, forbidden_vars=self.temp_boxes) self.temp_boxes.append(tmp) - selected_reg = self.force_allocate_reg(tmp) self.assembler.mc.LG(selected_reg, poolloc) return selected_reg return poolloc @@ -152,7 +152,7 @@ return loc def get_scratch_reg(self): - box = TempVar() + box = TempInt() reg = self.force_allocate_reg(box, forbidden_vars=self.temp_boxes) self.temp_boxes.append(box) return reg @@ -465,7 +465,8 @@ # else, return a regular register (not SPP). if self.rm.reg_bindings.get(var, None) is not None: return self.rm.loc(var, must_exist=True) - return self.rm.force_allocate_reg(var) + forbidden_vars = self.rm.temp_boxes + return self.rm.force_allocate_reg(var, forbidden_vars) def walk_operations(self, inputargs, operations): from rpython.jit.backend.zarch.assembler import ( From pypy.commits at gmail.com Mon Jan 25 07:33:03 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 04:33:03 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: rewritten many calls to use one stack frame less Message-ID: <56a615ff.4c301c0a.1c7bf.5836@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81928:6a1b2984c003 Date: 2016-01-25 13:31 +0100 http://bitbucket.org/pypy/pypy/changeset/6a1b2984c003/ Log: rewritten many calls to use one stack frame less diff --git a/rpython/jit/backend/zarch/arch.py b/rpython/jit/backend/zarch/arch.py --- a/rpython/jit/backend/zarch/arch.py +++ b/rpython/jit/backend/zarch/arch.py @@ -34,7 +34,8 @@ # in reverse order to SP STD_FRAME_SIZE_IN_BYTES = 160 -THREADLOCAL_ADDR_OFFSET = 16 # at position of r2, but r2 is never saved!! +THREADLOCAL_ON_ENTER_JIT = 8 +THREADLOCAL_ADDR_OFFSET = STD_FRAME_SIZE_IN_BYTES + THREADLOCAL_ON_ENTER_JIT assert STD_FRAME_SIZE_IN_BYTES % 2 == 0 diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -16,7 +16,8 @@ from rpython.jit.backend.zarch.arch import (WORD, STD_FRAME_SIZE_IN_BYTES, THREADLOCAL_ADDR_OFFSET, RECOVERY_GCMAP_POOL_OFFSET, RECOVERY_TARGET_POOL_OFFSET, - JUMPABS_TARGET_ADDR__POOL_OFFSET, JUMPABS_POOL_ADDR_POOL_OFFSET) + JUMPABS_TARGET_ADDR__POOL_OFFSET, JUMPABS_POOL_ADDR_POOL_OFFSET, + THREADLOCAL_ON_ENTER_JIT) from rpython.jit.backend.zarch.opassembler import OpAssembler from rpython.jit.backend.zarch.regalloc import Regalloc from rpython.jit.codewriter.effectinfo import EffectInfo @@ -382,7 +383,7 @@ """ # signature of these cond_call_slowpath functions: # * on entry, r12 contains the function to call - # * r3, r4, r5, r6 contain arguments for the call + # * r2, r3, r4, r5 contain arguments for the call # * r0 is the gcmap # * the old value of these regs must already be stored in the jitframe # * on exit, all registers are restored from the jitframe @@ -391,6 +392,8 @@ self.mc = mc ofs2 = self.cpu.get_ofs_of_frame_field('jf_gcmap') mc.STG(r.SCRATCH2, l.addr(ofs2,r.SPP)) + mc.STMG(r.r14,r.r15,l.addr(14*WORD, r.SP)) + mc.push_std_frame() # copy registers to the frame, with the exception of r3 to r6 and r12, # because these have already been saved by the caller. Note that @@ -406,21 +409,21 @@ reg is not r.r4 and reg is not r.r5 and reg is not r.r12] - self._push_core_regs_to_jitframe(mc, regs + [r.r14]) + self._push_core_regs_to_jitframe(mc, regs) if supports_floats: self._push_fp_regs_to_jitframe(mc) # allocate a stack frame! - mc.push_std_frame() mc.raw_call(r.r12) - mc.pop_std_frame() # Finish self._reload_frame_if_necessary(mc) - self._pop_core_regs_from_jitframe(mc, saved_regs + [r.r14]) + self._pop_core_regs_from_jitframe(mc, saved_regs) if supports_floats: self._pop_fp_regs_from_jitframe(mc) + size = STD_FRAME_SIZE_IN_BYTES + mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) mc.BCR(c.ANY, r.RETURN) self.mc = None return mc.materialize(self.cpu, []) @@ -446,8 +449,11 @@ mc.STG(r.SCRATCH, l.addr(ofs2, r.SPP)) saved_regs = [reg for reg in r.MANAGED_REGS if reg is not r.RES and reg is not r.RSZ] - self._push_core_regs_to_jitframe(mc, saved_regs + [r.r14]) + self._push_core_regs_to_jitframe(mc, saved_regs) self._push_fp_regs_to_jitframe(mc) + # alloc a frame for the callee + mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) + mc.push_std_frame() # if kind == 'fixed': addr = self.cpu.gc_ll_descr.get_malloc_slowpath_addr() @@ -478,10 +484,8 @@ # Do the call addr = rffi.cast(lltype.Signed, addr) - mc.push_std_frame() mc.load_imm(mc.RAW_CALL_REG, addr) mc.raw_call() - mc.pop_std_frame() self._reload_frame_if_necessary(mc) @@ -490,7 +494,7 @@ # emit_call_malloc_gc()). self.propagate_memoryerror_if_r2_is_null() - self._pop_core_regs_from_jitframe(mc, saved_regs + [r.r14]) + self._pop_core_regs_from_jitframe(mc, saved_regs) self._pop_fp_regs_from_jitframe(mc) nursery_free_adr = self.cpu.gc_ll_descr.get_nursery_free_addr() @@ -501,6 +505,8 @@ # r.RSZ is loaded from [SCRATCH], to make the caller's store a no-op here mc.load(r.RSZ, r.r1, 0) # + size = STD_FRAME_SIZE_IN_BYTES + mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) mc.BCR(c.ANY, r.r14) self.mc = None return mc.materialize(self.cpu, []) @@ -517,7 +523,7 @@ mc = InstrBuilder() # # store the link backwards - self.mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) + mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) mc.push_std_frame() mc.LGR(r.r2, r.SP) @@ -532,7 +538,7 @@ mc.cmp_op(r.SCRATCH, l.imm(0), imm=True) # size = STD_FRAME_SIZE_IN_BYTES - self.mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) # restore the link + mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) # restore the link # So we return to our caller, conditionally if "EQ" mc.BCR(c.EQ, r.r14) mc.trap() # debug if this is EVER executed! @@ -590,11 +596,11 @@ # LGHI r0, ... (4 bytes) # sum -> (14 bytes) mc.write('\x00'*14) - self.mc.push_std_frame() + mc.push_std_frame() mc.load_imm(r.RETURN, self._frame_realloc_slowpath) self.load_gcmap(mc, r.r1, gcmap) mc.raw_call() - self.mc.pop_std_frame() + mc.pop_std_frame() self.frame_depth_to_patch.append((patch_pos, mc.currpos())) @@ -1006,8 +1012,8 @@ # save the back chain self.mc.STG(r.SP, l.addr(0, r.SP)) - # save r3, the second argument, to THREADLOCAL_ADDR_OFFSET - self.mc.STG(r.r3, l.addr(THREADLOCAL_ADDR_OFFSET, r.SP)) + # save r3, the second argument, to the thread local position + self.mc.STG(r.r3, l.addr(THREADLOCAL_ON_ENTER_JIT, r.SP)) # push a standard frame for any call self.mc.push_std_frame() @@ -1418,9 +1424,7 @@ raise AssertionError(kind) # # call! - mc.push_std_frame() mc.branch_absolute(addr) - mc.pop_std_frame() jmp_location = mc.currpos() mc.reserve_cond_jump(short=True) # jump forward, patched later diff --git a/rpython/jit/backend/zarch/callbuilder.py b/rpython/jit/backend/zarch/callbuilder.py --- a/rpython/jit/backend/zarch/callbuilder.py +++ b/rpython/jit/backend/zarch/callbuilder.py @@ -62,6 +62,7 @@ # called function will in turn call further functions (which must be passed the # address of the new frame). This stack grows downwards from high addresses # """ + self.subtracted_to_sp = 0 gpr_regs = 0 fpr_regs = 0 @@ -83,18 +84,18 @@ stack_params.append(i) self.subtracted_to_sp += len(stack_params) * WORD - base = -len(stack_params) * WORD + base = len(stack_params) * WORD if self.is_call_release_gil: self.subtracted_to_sp += 8*WORD - base -= 8*WORD - # one additional owrd for remap frame layout + base += 8*WORD + # one additional word for remap frame layout # regalloc_push will overwrite -8(r.SP) and destroy # a parameter if we would not reserve that space - base -= WORD - self.subtracted_to_sp += WORD + # base += WORD + # TODO self.subtracted_to_sp += WORD for idx,i in enumerate(stack_params): loc = arglocs[i] - offset = base + 8 * idx + offset = STD_FRAME_SIZE_IN_BYTES - base + 8 * idx if loc.type == FLOAT: if loc.is_fp_reg(): src = loc @@ -148,15 +149,23 @@ def emit_raw_call(self): # always allocate a stack frame for the new function # save the SP back chain - self.mc.STG(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) + #self.mc.STG(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) # move the frame pointer if self.subtracted_to_sp != 0: self.mc.LAY(r.SP, l.addr(-self.subtracted_to_sp, r.SP)) self.mc.raw_call() + + + def restore_stack_pointer(self): + # it must at LEAST be 160 bytes + if self.subtracted_to_sp != 0: + self.mc.LAY(r.SP, l.addr(self.subtracted_to_sp, r.SP)) + + def load_result(self): + assert (self.resloc is None or + self.resloc is r.GPR_RETURN or + self.resloc is r.FPR_RETURN) # - self.ensure_correct_signzero_extension() - - def ensure_correct_signzero_extension(self): if self.restype == 'i' and self.ressize != WORD: # we must be sure! libffi (s390x impl) will not return # a sane 64 bit zero/sign extended value. fix for this @@ -177,25 +186,14 @@ else: assert 0, "cannot zero extend size %d" % self.ressize - - def restore_stack_pointer(self): - # it must at LEAST be 160 bytes - if self.subtracted_to_sp != 0: - self.mc.LAY(r.SP, l.addr(self.subtracted_to_sp, r.SP)) - - def load_result(self): - assert (self.resloc is None or - self.resloc is r.GPR_RETURN or - self.resloc is r.FPR_RETURN) - - def call_releasegil_addr_and_move_real_arguments(self, fastgil): assert self.is_call_release_gil RSHADOWOLD = self.RSHADOWOLD RSHADOWPTR = self.RSHADOWPTR RFASTGILPTR = self.RFASTGILPTR # - self.mc.STMG(r.r8, r.r13, l.addr(-7*WORD, r.SP)) + pos = STD_FRAME_SIZE_IN_BYTES - 7*WORD + self.mc.STMG(r.r8, r.r13, l.addr(pos, r.SP)) # 6 registers, 1 for a floating point return value! # registered by prepare_arguments! # @@ -268,26 +266,27 @@ PARAM_SAVE_AREA_OFFSET = 0 if reg is not None: # save 1 word below the stack pointer + pos = STD_FRAME_SIZE_IN_BYTES if reg.is_core_reg(): - self.mc.STG(reg, l.addr(-1*WORD, r.SP)) + self.mc.STG(reg, l.addr(pos-1*WORD, r.SP)) elif reg.is_fp_reg(): - self.mc.STD(reg, l.addr(-1*WORD, r.SP)) - self.mc.push_std_frame(8*WORD) + self.mc.STD(reg, l.addr(pos-1*WORD, r.SP)) self.mc.load_imm(self.mc.RAW_CALL_REG, self.asm.reacqgil_addr) self.mc.raw_call() - self.mc.pop_std_frame(8*WORD) if reg is not None: + pos = STD_FRAME_SIZE_IN_BYTES if reg.is_core_reg(): - self.mc.LG(reg, l.addr(-1*WORD, r.SP)) + self.mc.LG(reg, l.addr(pos-1*WORD, r.SP)) elif reg.is_fp_reg(): - self.mc.LD(reg, l.addr(-1*WORD, r.SP)) + self.mc.LD(reg, l.addr(pos-1*WORD, r.SP)) # replace b1_location with BEQ(here) pmc = OverwritingBuilder(self.mc, b1_location, 1) pmc.BRCL(c.EQ, l.imm(self.mc.currpos() - b1_location)) pmc.overwrite() - self.mc.LMG(r.r8, r.r13, l.addr(-7*WORD, r.SP)) + pos = STD_FRAME_SIZE_IN_BYTES - 7*WORD + self.mc.LMG(r.r8, r.r13, l.addr(pos, r.SP)) def write_real_errno(self, save_err): if save_err & rffi.RFFI_READSAVED_ERRNO: diff --git a/rpython/jit/backend/zarch/codebuilder.py b/rpython/jit/backend/zarch/codebuilder.py --- a/rpython/jit/backend/zarch/codebuilder.py +++ b/rpython/jit/backend/zarch/codebuilder.py @@ -198,7 +198,7 @@ function pointer, which means on big-endian that it is actually the address of a three-words descriptor. """ - self.BASR(r.RETURN, call_reg) + self.BASR(r.r14, call_reg) def reserve_cond_jump(self, short=False): self.trap() # conditional jump, patched later diff --git a/rpython/jit/backend/zarch/opassembler.py b/rpython/jit/backend/zarch/opassembler.py --- a/rpython/jit/backend/zarch/opassembler.py +++ b/rpython/jit/backend/zarch/opassembler.py @@ -530,11 +530,7 @@ mc.LGR(r.r0, loc_base) # unusual argument location mc.load_imm(r.r14, self.wb_slowpath[helper_num]) - # alloc a stack frame - mc.push_std_frame() mc.BASR(r.r14, r.r14) - # destory the frame - mc.pop_std_frame() if card_marking_mask: # The helper ends again with a check of the flag in the object. From pypy.commits at gmail.com Mon Jan 25 07:55:08 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 04:55:08 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: fixed problem in build_propagate_exception_path, value was loaded into wrong register (that was overwritten after that immediatley) Message-ID: <56a61b2c.cf821c0a.e0c6.6ff7@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81929:2c1aa1edc33a Date: 2016-01-25 13:54 +0100 http://bitbucket.org/pypy/pypy/changeset/2c1aa1edc33a/ Log: fixed problem in build_propagate_exception_path, value was loaded into wrong register (that was overwritten after that immediatley) refactored some calls to use store_link/restore_link instead of manually specifing each STMG/LMG diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -178,6 +178,7 @@ # save the information mc.store_link() + mc.push_std_frame() RCS2 = r.r10 RCS3 = r.r12 @@ -231,12 +232,9 @@ func = rffi.cast(lltype.Signed, func) # Note: if not 'for_frame', argument_loc is r0, which must carefully # not be overwritten above - mc.STG(r.SP, l.addr(0, r.SP)) # store the backchain - mc.push_std_frame() mc.load_imm(mc.RAW_CALL_REG, func) mc.LGR(r.r2, argument_loc) mc.raw_call() - mc.pop_std_frame() if for_frame: self._restore_exception(mc, RCS2, RCS3) @@ -303,6 +301,9 @@ mc = InstrBuilder() self.mc = mc + mc.store_link() + mc.push_std_frame() + # signature of this _frame_realloc_slowpath function: # * on entry, r0 is the new size # * on entry, r1 is the gcmap @@ -314,7 +315,6 @@ self._push_core_regs_to_jitframe(mc, r.MANAGED_REGS) self._push_fp_regs_to_jitframe(mc) - self.mc.store_link() # First argument is SPP (= r31), which is the jitframe mc.LGR(r.r2, r.SPP) @@ -346,9 +346,10 @@ mc.load(r.r5, r.r5, diff) mc.store(r.r2, r.r5, -WORD) - mc.restore_link() self._pop_core_regs_from_jitframe(mc) self._pop_fp_regs_from_jitframe(mc) + + mc.restore_link() mc.BCR(c.ANY, r.RETURN) self._frame_realloc_slowpath = mc.materialize(self.cpu, []) @@ -367,7 +368,7 @@ ofs3 = self.cpu.get_ofs_of_frame_field('jf_guard_exc') ofs4 = self.cpu.get_ofs_of_frame_field('jf_descr') - self._store_and_reset_exception(self.mc, r.r3) + self._store_and_reset_exception(self.mc, r.r2) self.mc.load_imm(r.r3, propagate_exception_descr) self.mc.STG(r.r2, l.addr(ofs3, r.SPP)) self.mc.STG(r.r3, l.addr(ofs4, r.SPP)) @@ -392,7 +393,7 @@ self.mc = mc ofs2 = self.cpu.get_ofs_of_frame_field('jf_gcmap') mc.STG(r.SCRATCH2, l.addr(ofs2,r.SPP)) - mc.STMG(r.r14,r.r15,l.addr(14*WORD, r.SP)) + mc.store_link() mc.push_std_frame() # copy registers to the frame, with the exception of r3 to r6 and r12, @@ -422,8 +423,7 @@ self._pop_core_regs_from_jitframe(mc, saved_regs) if supports_floats: self._pop_fp_regs_from_jitframe(mc) - size = STD_FRAME_SIZE_IN_BYTES - mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) + mc.restore_link() mc.BCR(c.ANY, r.RETURN) self.mc = None return mc.materialize(self.cpu, []) @@ -445,15 +445,16 @@ assert kind in ['fixed', 'str', 'unicode', 'var'] mc = InstrBuilder() self.mc = mc + # alloc a frame for the callee + mc.store_link() + mc.push_std_frame() + # ofs2 = self.cpu.get_ofs_of_frame_field('jf_gcmap') mc.STG(r.SCRATCH, l.addr(ofs2, r.SPP)) saved_regs = [reg for reg in r.MANAGED_REGS if reg is not r.RES and reg is not r.RSZ] self._push_core_regs_to_jitframe(mc, saved_regs) self._push_fp_regs_to_jitframe(mc) - # alloc a frame for the callee - mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) - mc.push_std_frame() # if kind == 'fixed': addr = self.cpu.gc_ll_descr.get_malloc_slowpath_addr() @@ -505,8 +506,7 @@ # r.RSZ is loaded from [SCRATCH], to make the caller's store a no-op here mc.load(r.RSZ, r.r1, 0) # - size = STD_FRAME_SIZE_IN_BYTES - mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) + mc.restore_link() mc.BCR(c.ANY, r.r14) self.mc = None return mc.materialize(self.cpu, []) @@ -523,7 +523,7 @@ mc = InstrBuilder() # # store the link backwards - mc.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) + mc.store_link() mc.push_std_frame() mc.LGR(r.r2, r.SP) @@ -537,8 +537,7 @@ # else we have an exception mc.cmp_op(r.SCRATCH, l.imm(0), imm=True) # - size = STD_FRAME_SIZE_IN_BYTES - mc.LMG(r.r14, r.r15, l.addr(size+14*WORD, r.SP)) # restore the link + mc.restore_link() # So we return to our caller, conditionally if "EQ" mc.BCR(c.EQ, r.r14) mc.trap() # debug if this is EVER executed! diff --git a/rpython/jit/backend/zarch/codebuilder.py b/rpython/jit/backend/zarch/codebuilder.py --- a/rpython/jit/backend/zarch/codebuilder.py +++ b/rpython/jit/backend/zarch/codebuilder.py @@ -212,10 +212,11 @@ self.BASR(r.r14, r.r14) def store_link(self): - self.STG(r.RETURN, l.addr(14*WORD, r.SP)) + self.STMG(r.r14, r.r15, l.addr(14*WORD, r.SP)) def restore_link(self): - self.LG(r.RETURN, l.addr(14*WORD, r.SP)) + off = STD_FRAME_SIZE_IN_BYTES + self.LMG(r.r14, l.addr(off+14*WORD, r.SP)) def push_std_frame(self, additional_bytes=0): self.STG(r.SP, l.addr(-(STD_FRAME_SIZE_IN_BYTES + additional_bytes), r.SP)) From pypy.commits at gmail.com Mon Jan 25 08:03:26 2016 From: pypy.commits at gmail.com (Raemi) Date: Mon, 25 Jan 2016 05:03:26 -0800 (PST) Subject: [pypy-commit] pypy stmgc-c8: remove dummy allocation Message-ID: <56a61d1e.c177c20a.b064e.3f15@mx.google.com> Author: Remi Meier Branch: stmgc-c8 Changeset: r81930:89984a5ade1e Date: 2016-01-25 10:51 +0100 http://bitbucket.org/pypy/pypy/changeset/89984a5ade1e/ Log: remove dummy allocation AFAICS, we introduced this in C7 to make stop-the-world commits as fast as possible. It is only needed for major GCs in C8, so let's not pay the overhead. diff --git a/rpython/jit/backend/llsupport/stmrewrite.py b/rpython/jit/backend/llsupport/stmrewrite.py --- a/rpython/jit/backend/llsupport/stmrewrite.py +++ b/rpython/jit/backend/llsupport/stmrewrite.py @@ -137,16 +137,20 @@ self.handle_getfields(op) def possibly_add_dummy_allocation(self): - if not self.does_any_allocation: - # do a fake allocation since this is needed to check - # for requested safe-points: - self.does_any_allocation = True + # was necessary in C7 for others to commit, but in C8 it is only + # necessary for requesting major GCs. I think we better avoid this + # overhead for tight loops and wait a bit longer in that case. + pass + # if not self.does_any_allocation: + # # do a fake allocation since this is needed to check + # # for requested safe-points: + # self.does_any_allocation = True - # minimum size for the slowpath of MALLOC_NURSERY: - size = self.gc_ll_descr.minimal_size_in_nursery - op = ResOperation(rop.LABEL, []) # temp, will be replaced by gen_malloc_nursery - assert self._op_malloc_nursery is None # no ongoing allocation - self.gen_malloc_nursery(size, op) + # # minimum size for the slowpath of MALLOC_NURSERY: + # size = self.gc_ll_descr.minimal_size_in_nursery + # op = ResOperation(rop.LABEL, []) # temp, will be replaced by gen_malloc_nursery + # assert self._op_malloc_nursery is None # no ongoing allocation + # self.gen_malloc_nursery(size, op) def must_apply_write_barrier(self, val, v): # also apply for non-ref values From pypy.commits at gmail.com Mon Jan 25 08:03:28 2016 From: pypy.commits at gmail.com (Raemi) Date: Mon, 25 Jan 2016 05:03:28 -0800 (PST) Subject: [pypy-commit] pypy stmgc-c8: possible improvement Message-ID: <56a61d20.c8921c0a.886c4.67ad@mx.google.com> Author: Remi Meier Branch: stmgc-c8 Changeset: r81931:e9d25e9cc525 Date: 2016-01-25 11:56 +0100 http://bitbucket.org/pypy/pypy/changeset/e9d25e9cc525/ Log: possible improvement AFAICS at the level where the readbarrier transform gets executed, the operations listed in MALLOCS are already resolved. However, I failed to test this so far. diff --git a/rpython/translator/stm/readbarrier.py b/rpython/translator/stm/readbarrier.py --- a/rpython/translator/stm/readbarrier.py +++ b/rpython/translator/stm/readbarrier.py @@ -8,6 +8,7 @@ 'malloc', 'malloc_varsize', 'malloc_nonmovable', 'malloc_nonmovable_varsize', 'malloc_noconflict', 'malloc_noconflict_varsize', + 'stm_allocate_tid', ]) READ_OPS = set(['getfield', 'getarrayitem', 'getinteriorfield', 'raw_load']) diff --git a/rpython/translator/stm/test/test_readbarrier.py b/rpython/translator/stm/test/test_readbarrier.py --- a/rpython/translator/stm/test/test_readbarrier.py +++ b/rpython/translator/stm/test/test_readbarrier.py @@ -246,6 +246,37 @@ assert len(self.read_barriers) == 5 + + +# class TestAfterGCTransform(BaseTestTransform): +# do_read_barrier = True +# do_gc_transform = True + +# def test_malloc_result_readable(self): +# from rpython.flowspace.model import summary +# X = lltype.GcStruct('X', ('foo', lltype.Signed)) +# # +# def nobreak_escape(x): +# x.foo = 7 +# return x +# nobreak_escape._dont_inline_ = True +# # +# def f1(n): +# x = lltype.malloc(X) +# t = x.foo +# nobreak_escape(x) +# return t + +# self.interpret(f1, [4], run=False) +# g = self.graph +# from rpython.translator.translator import graphof +# #ff = graphof(g, f1) +# #ff.show() +# assert summary(g)['stm_read'] == 0 + +# assert self.read_barriers == [x1] + + external_release_gil = rffi.llexternal('external_release_gil', [], lltype.Void, _callable=lambda: None, random_effects_on_gcobjs=True, diff --git a/rpython/translator/stm/test/transform_support.py b/rpython/translator/stm/test/transform_support.py --- a/rpython/translator/stm/test/transform_support.py +++ b/rpython/translator/stm/test/transform_support.py @@ -23,6 +23,7 @@ do_read_barrier = False do_turn_inevitable = False do_jit_driver = False + do_gc_transform = False def build_state(self): self.read_barriers = [] @@ -36,7 +37,7 @@ return 'I' # allocated with immortal=True raise AssertionError("unknown category on %r" % (p,)) - def interpret(self, fn, args, gcremovetypeptr=False, run=True): + def interpret(self, fn, args, run=True): self.build_state() clear_tcache() interp, self.graph = get_interpreter(fn, args, view=False, @@ -45,14 +46,34 @@ interp.frame_class = LLSTMFrame # self.translator = interp.typer.annotator.translator - self.translator.config.translation.gcremovetypeptr = gcremovetypeptr + self.translator.config.translation.gc = "stmgc" + self.translator.config.translation.stm = True self.stmtransformer = STMTransformer(self.translator) if self.do_jit_driver: self.stmtransformer.transform_jit_driver() + if self.do_turn_inevitable: + self.stmtransformer.transform_turn_inevitable() + if self.do_gc_transform: + pass + # from rpython.translator.c.gc import StmFrameworkGcPolicy + # from rpython.translator.c.database import LowLevelDatabase + # from rpython.translator.backendopt.all import backend_optimizations + # from rpython.rtyper.lltypesystem.lltype import getfunctionptr + # self.translator.config.translation.backendopt.inline=True + # self.translator.config.translation.backendopt.inline_threshold=10000 + # self.translator.config.translation.backendopt.mallocs=True + # # backend_optimizations(self.translator, + # # inline_graph_from_anywhere=True, + # # secondary=True, inline=True, inline_threshold=0, + # # mallocs=True, print_statistics=True, + # # clever_malloc_removal=True) + # db = LowLevelDatabase(self.translator, gcpolicyclass=StmFrameworkGcPolicy) + # self.stmtransformer.transform_after_gc() + # list(db.gcpolicy.gc_startup_code()) + # db.get(getfunctionptr(self.graph)) + # db.complete() if self.do_read_barrier: self.stmtransformer.transform_read_barrier() - if self.do_turn_inevitable: - self.stmtransformer.transform_turn_inevitable() if option.view: self.translator.view() # From pypy.commits at gmail.com Mon Jan 25 08:05:32 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 05:05:32 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: added missing parameter to LMG call (refactoring issues :) Message-ID: <56a61d9c.8205c20a.bf1e4.7fef@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81932:0e52bea859f2 Date: 2016-01-25 14:04 +0100 http://bitbucket.org/pypy/pypy/changeset/0e52bea859f2/ Log: added missing parameter to LMG call (refactoring issues :) diff --git a/rpython/jit/backend/zarch/codebuilder.py b/rpython/jit/backend/zarch/codebuilder.py --- a/rpython/jit/backend/zarch/codebuilder.py +++ b/rpython/jit/backend/zarch/codebuilder.py @@ -216,7 +216,7 @@ def restore_link(self): off = STD_FRAME_SIZE_IN_BYTES - self.LMG(r.r14, l.addr(off+14*WORD, r.SP)) + self.LMG(r.r14, r.r15, l.addr(off+14*WORD, r.SP)) def push_std_frame(self, additional_bytes=0): self.STG(r.SP, l.addr(-(STD_FRAME_SIZE_IN_BYTES + additional_bytes), r.SP)) From pypy.commits at gmail.com Mon Jan 25 09:55:33 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 06:55:33 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: due to the refacotring the variable saving went to the wrong stack frame in wb_slowpath (now takes the right one) Message-ID: <56a63765.890bc30a.5c6f2.ffffb313@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81933:adce895c17e8 Date: 2016-01-25 15:54 +0100 http://bitbucket.org/pypy/pypy/changeset/adce895c17e8/ Log: due to the refacotring the variable saving went to the wrong stack frame in wb_slowpath (now takes the right one) diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -199,9 +199,10 @@ # and two more non-volatile registers (used to store # the RPython exception that occurred in the CALL, if any). # - mc.STMG(r.r10, r.r12, l.addr(10*WORD, r.SP)) - mc.STG(r.r2, l.addr(2*WORD, r.SP)) - mc.STD(r.f0, l.addr(16*WORD, r.SP)) + off = STD_FRAME_SIZE_IN_BYTES + mc.STMG(r.r10, r.r12, l.addr(off+10*WORD, r.SP)) + mc.STG(r.r2, l.addr(off+2*WORD, r.SP)) + mc.STD(r.f0, l.addr(off+16*WORD, r.SP)) saved_regs = None saved_fp_regs = None else: @@ -248,9 +249,10 @@ mc.NILL(RCS2, l.imm(card_marking_mask & 0xFF)) if for_frame: - mc.LMG(r.r10, r.r12, l.addr(10*WORD, r.SP)) - mc.LG(r.r2, l.addr(2*WORD, r.SP)) - mc.LD(r.f0, l.addr(16*WORD, r.SP)) + off = STD_FRAME_SIZE_IN_BYTES + mc.LMG(r.r10, r.r12, l.addr(off+10*WORD, r.SP)) + mc.LG(r.r2, l.addr(off+2*WORD, r.SP)) + mc.LD(r.f0, l.addr(off+16*WORD, r.SP)) else: self._pop_core_regs_from_jitframe(mc, saved_regs) self._pop_fp_regs_from_jitframe(mc, saved_fp_regs) @@ -396,7 +398,7 @@ mc.store_link() mc.push_std_frame() - # copy registers to the frame, with the exception of r3 to r6 and r12, + # copy registers to the frame, with the exception of r2 to r5 and r12, # because these have already been saved by the caller. Note that # this is not symmetrical: these 5 registers are saved by the caller # but restored here at the end of this function. @@ -450,7 +452,7 @@ mc.push_std_frame() # ofs2 = self.cpu.get_ofs_of_frame_field('jf_gcmap') - mc.STG(r.SCRATCH, l.addr(ofs2, r.SPP)) + mc.STG(r.r1, l.addr(ofs2, r.SPP)) saved_regs = [reg for reg in r.MANAGED_REGS if reg is not r.RES and reg is not r.RSZ] self._push_core_regs_to_jitframe(mc, saved_regs) @@ -572,7 +574,6 @@ jmp_pos = self.mc.currpos() self.mc.reserve_cond_jump() - mc.push_std_frame() mc.load_imm(r.r14, self.stack_check_slowpath) mc.BASR(r.r14, r.r14) @@ -1008,8 +1009,6 @@ # Build a new stackframe of size STD_FRAME_SIZE_IN_BYTES self.mc.STMG(r.r6, r.r15, l.addr(6*WORD, r.SP)) self.mc.LARL(r.POOL, l.halfword(self.pool.pool_start - self.mc.get_relative_pos())) - # save the back chain - self.mc.STG(r.SP, l.addr(0, r.SP)) # save r3, the second argument, to the thread local position self.mc.STG(r.r3, l.addr(THREADLOCAL_ON_ENTER_JIT, r.SP)) diff --git a/rpython/jit/backend/zarch/codebuilder.py b/rpython/jit/backend/zarch/codebuilder.py --- a/rpython/jit/backend/zarch/codebuilder.py +++ b/rpython/jit/backend/zarch/codebuilder.py @@ -219,7 +219,7 @@ self.LMG(r.r14, r.r15, l.addr(off+14*WORD, r.SP)) def push_std_frame(self, additional_bytes=0): - self.STG(r.SP, l.addr(-(STD_FRAME_SIZE_IN_BYTES + additional_bytes), r.SP)) + self.STG(r.SP, l.addr(0, r.SP)) self.LAY(r.SP, l.addr(-(STD_FRAME_SIZE_IN_BYTES + additional_bytes), r.SP)) def pop_std_frame(self, additional_bytes=0): From pypy.commits at gmail.com Mon Jan 25 12:34:30 2016 From: pypy.commits at gmail.com (devin.jeanpierre) Date: Mon, 25 Jan 2016 09:34:30 -0800 (PST) Subject: [pypy-commit] pypy default: Make C-API char* parameters const-correct with respect to CPython. Message-ID: <56a65ca6.6608c20a.ee2ce.fffff519@mx.google.com> Author: Devin Jeanpierre Branch: Changeset: r81934:2b601dbb5d24 Date: 2016-01-25 09:19 -0800 http://bitbucket.org/pypy/pypy/changeset/2b601dbb5d24/ Log: Make C-API char* parameters const-correct with respect to CPython. diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -59,7 +59,7 @@ return None return borrow_from(w_dict, w_res) - at cpython_api([PyObject, rffi.CCHARP], rffi.INT_real, error=-1) + at cpython_api([PyObject, CONST_STRING], rffi.INT_real, error=-1) def PyDict_DelItemString(space, w_dict, key_ptr): """Remove the entry in dictionary p which has a key specified by the string key. Return 0 on success or -1 on failure.""" diff --git a/pypy/module/cpyext/eval.py b/pypy/module/cpyext/eval.py --- a/pypy/module/cpyext/eval.py +++ b/pypy/module/cpyext/eval.py @@ -128,7 +128,7 @@ filename = "" return run_string(space, source, filename, start, w_globals, w_locals) - at cpython_api([rffi.CCHARP, rffi.INT_real, PyObject, PyObject, + at cpython_api([CONST_STRING, rffi.INT_real, PyObject, PyObject, PyCompilerFlagsPtr], PyObject) def PyRun_StringFlags(space, source, start, w_globals, w_locals, flagsptr): """Execute Python source code from str in the context specified by the @@ -189,7 +189,7 @@ pi[0] = space.getindex_w(w_obj, None) return 1 - at cpython_api([rffi.CCHARP, rffi.CCHARP, rffi.INT_real, PyCompilerFlagsPtr], + at cpython_api([CONST_STRING, CONST_STRING, rffi.INT_real, PyCompilerFlagsPtr], PyObject) def Py_CompileStringFlags(space, source, filename, start, flagsptr): """Parse and compile the Python source code in str, returning the diff --git a/pypy/module/cpyext/pystrtod.py b/pypy/module/cpyext/pystrtod.py --- a/pypy/module/cpyext/pystrtod.py +++ b/pypy/module/cpyext/pystrtod.py @@ -1,6 +1,6 @@ import errno from pypy.interpreter.error import OperationError -from pypy.module.cpyext.api import cpython_api +from pypy.module.cpyext.api import cpython_api, CONST_STRING from pypy.module.cpyext.pyobject import PyObject from rpython.rlib import rdtoa from rpython.rlib import rfloat @@ -22,7 +22,7 @@ rfloat.DIST_NAN: Py_DTST_NAN } - at cpython_api([rffi.CCHARP, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) + at cpython_api([CONST_STRING, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) @jit.dont_look_inside # direct use of _get_errno() def PyOS_string_to_double(space, s, endptr, w_overflow_exception): """Convert a string s to a double, raising a Python From pypy.commits at gmail.com Mon Jan 25 15:15:03 2016 From: pypy.commits at gmail.com (mattip) Date: Mon, 25 Jan 2016 12:15:03 -0800 (PST) Subject: [pypy-commit] pypy cpyext-ext: add passing test for multiple inheritance Message-ID: <56a68247.4e8e1c0a.e32dd.2412@mx.google.com> Author: mattip Branch: cpyext-ext Changeset: r81935:df9304236c27 Date: 2016-01-25 21:58 +0200 http://bitbucket.org/pypy/pypy/changeset/df9304236c27/ Log: add passing test for multiple inheritance diff --git a/pypy/module/cpyext/test/foo.c b/pypy/module/cpyext/test/foo.c --- a/pypy/module/cpyext/test/foo.c +++ b/pypy/module/cpyext/test/foo.c @@ -371,6 +371,12 @@ 0 /*tp_weaklist*/ }; +PyTypeObject UnicodeSubtype3 = { + PyObject_HEAD_INIT(NULL) + 0, + "foo.fuu3", + sizeof(UnicodeSubclassObject) +}; /* A Metatype */ @@ -777,6 +783,14 @@ CustomType.ob_type = &MetaType; if (PyType_Ready(&CustomType) < 0) return; + + UnicodeSubtype3.tp_flags = Py_TPFLAGS_DEFAULT; + UnicodeSubtype3.tp_base = &UnicodeSubtype; + UnicodeSubtype3.tp_bases = Py_BuildValue("(OO)", &UnicodeSubtype, + &CustomType); + if (PyType_Ready(&UnicodeSubtype3) < 0) + return; + m = Py_InitModule("foo", foo_functions); if (m == NULL) return; @@ -789,6 +803,8 @@ return; if (PyDict_SetItemString(d, "UnicodeSubtype2", (PyObject *) &UnicodeSubtype2) < 0) return; + if (PyDict_SetItemString(d, "UnicodeSubtype3", (PyObject *) &UnicodeSubtype3) < 0) + return; if (PyDict_SetItemString(d, "MetaType", (PyObject *) &MetaType) < 0) return; if (PyDict_SetItemString(d, "InitErrType", (PyObject *) &InitErrType) < 0) diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -150,6 +150,14 @@ assert fuu2(u"abc").baz().escape() raises(TypeError, module.fooType.object_member.__get__, 1) + def test_multiple_inheritance(self): + module = self.import_module(name='foo') + obj = module.UnicodeSubtype(u'xyz') + obj2 = module.UnicodeSubtype2() + obj3 = module.UnicodeSubtype3() + assert obj3.get_val() == 42 + assert len(type(obj3).mro()) == 6 + def test_init(self): module = self.import_module(name="foo") newobj = module.UnicodeSubtype() @@ -416,15 +424,15 @@ module = self.import_extension('foo', [ ("test_tp_getattro", "METH_VARARGS", ''' - PyObject *obj = PyTuple_GET_ITEM(args, 0); - PyIntObject *value = PyTuple_GET_ITEM(args, 1); + PyObject *name, *obj = PyTuple_GET_ITEM(args, 0); + PyIntObject *attr, *value = PyTuple_GET_ITEM(args, 1); if (!obj->ob_type->tp_getattro) { PyErr_SetString(PyExc_ValueError, "missing tp_getattro"); return NULL; } - PyObject *name = PyString_FromString("attr1"); - PyIntObject *attr = obj->ob_type->tp_getattro(obj, name); + name = PyString_FromString("attr1"); + attr = obj->ob_type->tp_getattro(obj, name); if (attr->ob_ival != value->ob_ival) { PyErr_SetString(PyExc_ValueError, @@ -705,13 +713,13 @@ static PyObject * intlike_nb_add(PyObject *self, PyObject *other) { - long val1 = ((IntLikeObject *)(self))->ival; + long val2, val1 = ((IntLikeObject *)(self))->ival; if (PyInt_Check(other)) { long val2 = PyInt_AsLong(other); return PyInt_FromLong(val1+val2); } - long val2 = ((IntLikeObject *)(other))->ival; + val2 = ((IntLikeObject *)(other))->ival; return PyInt_FromLong(val1+val2); } From pypy.commits at gmail.com Mon Jan 25 15:15:05 2016 From: pypy.commits at gmail.com (mattip) Date: Mon, 25 Jan 2016 12:15:05 -0800 (PST) Subject: [pypy-commit] pypy default: fix - link_flags is a tuple not a list Message-ID: <56a68249.878e1c0a.43a8f.21fa@mx.google.com> Author: mattip Branch: Changeset: r81936:e8e0169d3a13 Date: 2016-01-25 22:00 +0200 http://bitbucket.org/pypy/pypy/changeset/e8e0169d3a13/ Log: fix - link_flags is a tuple not a list diff --git a/rpython/translator/platform/windows.py b/rpython/translator/platform/windows.py --- a/rpython/translator/platform/windows.py +++ b/rpython/translator/platform/windows.py @@ -151,7 +151,7 @@ # Increase stack size, for the linker and the stack check code. stack_size = 8 << 20 # 8 Mb - self.link_flags.append('/STACK:%d' % stack_size) + self.link_flags = self.link_flags + ('/STACK:%d' % stack_size,) # The following symbol is used in c/src/stack.h self.cflags.append('/DMAX_STACK_SIZE=%d' % (stack_size - 1024)) From pypy.commits at gmail.com Mon Jan 25 15:15:08 2016 From: pypy.commits at gmail.com (mattip) Date: Mon, 25 Jan 2016 12:15:08 -0800 (PST) Subject: [pypy-commit] pypy cpyext-ext: remove debug cruft Message-ID: <56a6824c.01cdc20a.2b98b.4484@mx.google.com> Author: mattip Branch: cpyext-ext Changeset: r81938:d4fc8a14bff6 Date: 2016-01-18 20:46 +0200 http://bitbucket.org/pypy/pypy/changeset/d4fc8a14bff6/ Log: remove debug cruft diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -66,7 +66,6 @@ py_unicode = rffi.cast(PyUnicodeObject, py_obj) py_unicode.c_length = len(space.unicode_w(w_obj)) py_unicode.c_str = lltype.nullptr(rffi.CWCHARP.TO) - print w_obj py_unicode.c_hash = space.hash_w(w_obj) py_unicode.c_defenc = lltype.nullptr(PyObject.TO) From pypy.commits at gmail.com Mon Jan 25 15:15:07 2016 From: pypy.commits at gmail.com (mattip) Date: Mon, 25 Jan 2016 12:15:07 -0800 (PST) Subject: [pypy-commit] pypy cpyext-ext: test, fix - raise cpython compatible exception Message-ID: <56a6824b.624fc20a.f6238.2dbb@mx.google.com> Author: mattip Branch: cpyext-ext Changeset: r81937:26b7a0e96dc1 Date: 2016-01-17 08:17 +0200 http://bitbucket.org/pypy/pypy/changeset/26b7a0e96dc1/ Log: test, fix - raise cpython compatible exception diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -276,7 +276,12 @@ @cpython_api([PyObject], PyCFunction_typedef) def PyCFunction_GetFunction(space, w_obj): - cfunction = space.interp_w(W_PyCFunctionObject, w_obj) + try: + cfunction = space.interp_w(W_PyCFunctionObject, w_obj) + except OperationError, e: + if e.match(space, space.w_TypeError): + raise oefmt(space.w_SystemError, "bad argument to internal function") + raise return cfunction.ml.c_ml_meth @cpython_api([PyObject], PyObject) diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -90,7 +90,7 @@ assert mod.isCFunction(mod.getarg_O) == "getarg_O" assert mod.getModule(mod.getarg_O) == 'MyModule' assert mod.isSameFunction(mod.getarg_O) - raises(TypeError, mod.isSameFunction, 1) + raises(SystemError, mod.isSameFunction, 1) class TestPyCMethodObject(BaseApiTest): def test_repr(self, space, api): From pypy.commits at gmail.com Mon Jan 25 15:15:11 2016 From: pypy.commits at gmail.com (mattip) Date: Mon, 25 Jan 2016 12:15:11 -0800 (PST) Subject: [pypy-commit] pypy cpyext-ext: merge default into branch Message-ID: <56a6824f.41dfc20a.99dea.2ff8@mx.google.com> Author: mattip Branch: cpyext-ext Changeset: r81939:98e1616b19f0 Date: 2016-01-25 22:07 +0200 http://bitbucket.org/pypy/pypy/changeset/98e1616b19f0/ Log: merge default into branch diff too long, truncating to 2000 out of 9862 lines diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -28,7 +28,7 @@ DEALINGS IN THE SOFTWARE. -PyPy Copyright holders 2003-2015 +PyPy Copyright holders 2003-2016 ----------------------------------- Except when otherwise stated (look for LICENSE files or information at diff --git a/Makefile b/Makefile --- a/Makefile +++ b/Makefile @@ -39,5 +39,5 @@ # runs. We cannot get their original value either: # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html -cffi_imports: +cffi_imports: pypy-c PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py diff --git a/lib-python/2.7/distutils/command/build_ext.py b/lib-python/2.7/distutils/command/build_ext.py --- a/lib-python/2.7/distutils/command/build_ext.py +++ b/lib-python/2.7/distutils/command/build_ext.py @@ -685,13 +685,17 @@ # the previous version of this code did. This should work for # CPython too. The point is that on PyPy with cpyext, the # config var 'SO' is just ".so" but we want to return - # ".pypy-VERSION.so" instead. - so_ext = _get_c_extension_suffix() + # ".pypy-VERSION.so" instead. Note a further tweak for cffi's + # embedding mode: if EXT_SUFFIX is also defined, use that + # directly. + so_ext = get_config_var('EXT_SUFFIX') if so_ext is None: - so_ext = get_config_var('SO') # fall-back - # extensions in debug_mode are named 'module_d.pyd' under windows - if os.name == 'nt' and self.debug: - so_ext = '_d.pyd' + so_ext = _get_c_extension_suffix() + if so_ext is None: + so_ext = get_config_var('SO') # fall-back + # extensions in debug_mode are named 'module_d.pyd' under windows + if os.name == 'nt' and self.debug: + so_ext = '_d.pyd' return os.path.join(*ext_path) + so_ext def get_export_symbols (self, ext): diff --git a/lib-python/2.7/pickle.py b/lib-python/2.7/pickle.py --- a/lib-python/2.7/pickle.py +++ b/lib-python/2.7/pickle.py @@ -1376,6 +1376,7 @@ def decode_long(data): r"""Decode a long from a two's complement little-endian binary string. + This is overriden on PyPy by a RPython version that has linear complexity. >>> decode_long('') 0L @@ -1402,6 +1403,11 @@ n -= 1L << (nbytes * 8) return n +try: + from __pypy__ import decode_long +except ImportError: + pass + # Shorthands try: diff --git a/lib-python/2.7/sysconfig.py b/lib-python/2.7/sysconfig.py --- a/lib-python/2.7/sysconfig.py +++ b/lib-python/2.7/sysconfig.py @@ -524,6 +524,13 @@ import _osx_support _osx_support.customize_config_vars(_CONFIG_VARS) + # PyPy: + import imp + for suffix, mode, type_ in imp.get_suffixes(): + if type_ == imp.C_EXTENSION: + _CONFIG_VARS['SOABI'] = suffix.split('.')[1] + break + if args: vals = [] for name in args: diff --git a/lib-python/2.7/test/capath/0e4015b9.0 b/lib-python/2.7/test/capath/0e4015b9.0 new file mode 100644 --- /dev/null +++ b/lib-python/2.7/test/capath/0e4015b9.0 @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIIClTCCAf6gAwIBAgIJAKGU95wKR8pTMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNV +BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u +IFNvZnR3YXJlIEZvdW5kYXRpb24xIzAhBgNVBAMMGnNlbGYtc2lnbmVkLnB5dGhv +bnRlc3QubmV0MB4XDTE0MTEwMjE4MDkyOVoXDTI0MTAzMDE4MDkyOVowcDELMAkG +A1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMwIQYDVQQKDBpQeXRo +b24gU29mdHdhcmUgRm91bmRhdGlvbjEjMCEGA1UEAwwac2VsZi1zaWduZWQucHl0 +aG9udGVzdC5uZXQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANDXQXW9tjyZ +Xt0Iv2tLL1+jinr4wGg36ioLDLFkMf+2Y1GL0v0BnKYG4N1OKlAU15LXGeGer8vm +Sv/yIvmdrELvhAbbo3w4a9TMYQA4XkIVLdvu3mvNOAet+8PMJxn26dbDhG809ALv +EHY57lQsBS3G59RZyBPVqAqmImWNJnVzAgMBAAGjNzA1MCUGA1UdEQQeMByCGnNl +bGYtc2lnbmVkLnB5dGhvbnRlc3QubmV0MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcN +AQEFBQADgYEAIuzAhgMouJpNdf3URCHIineyoSt6WK/9+eyUcjlKOrDoXNZaD72h +TXMeKYoWvJyVcSLKL8ckPtDobgP2OTt0UkyAaj0n+ZHaqq1lH2yVfGUA1ILJv515 +C8BqbvVZuqm3i7ygmw3bqE/lYMgOrYtXXnqOrz6nvsE6Yc9V9rFflOM= +-----END CERTIFICATE----- diff --git a/lib-python/2.7/test/capath/ce7b8643.0 b/lib-python/2.7/test/capath/ce7b8643.0 new file mode 100644 --- /dev/null +++ b/lib-python/2.7/test/capath/ce7b8643.0 @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIIClTCCAf6gAwIBAgIJAKGU95wKR8pTMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNV +BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u +IFNvZnR3YXJlIEZvdW5kYXRpb24xIzAhBgNVBAMMGnNlbGYtc2lnbmVkLnB5dGhv +bnRlc3QubmV0MB4XDTE0MTEwMjE4MDkyOVoXDTI0MTAzMDE4MDkyOVowcDELMAkG +A1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMwIQYDVQQKDBpQeXRo +b24gU29mdHdhcmUgRm91bmRhdGlvbjEjMCEGA1UEAwwac2VsZi1zaWduZWQucHl0 +aG9udGVzdC5uZXQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANDXQXW9tjyZ +Xt0Iv2tLL1+jinr4wGg36ioLDLFkMf+2Y1GL0v0BnKYG4N1OKlAU15LXGeGer8vm +Sv/yIvmdrELvhAbbo3w4a9TMYQA4XkIVLdvu3mvNOAet+8PMJxn26dbDhG809ALv +EHY57lQsBS3G59RZyBPVqAqmImWNJnVzAgMBAAGjNzA1MCUGA1UdEQQeMByCGnNl +bGYtc2lnbmVkLnB5dGhvbnRlc3QubmV0MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcN +AQEFBQADgYEAIuzAhgMouJpNdf3URCHIineyoSt6WK/9+eyUcjlKOrDoXNZaD72h +TXMeKYoWvJyVcSLKL8ckPtDobgP2OTt0UkyAaj0n+ZHaqq1lH2yVfGUA1ILJv515 +C8BqbvVZuqm3i7ygmw3bqE/lYMgOrYtXXnqOrz6nvsE6Yc9V9rFflOM= +-----END CERTIFICATE----- diff --git a/lib-python/2.7/test/https_svn_python_org_root.pem b/lib-python/2.7/test/https_svn_python_org_root.pem deleted file mode 100644 --- a/lib-python/2.7/test/https_svn_python_org_root.pem +++ /dev/null @@ -1,41 +0,0 @@ ------BEGIN CERTIFICATE----- -MIIHPTCCBSWgAwIBAgIBADANBgkqhkiG9w0BAQQFADB5MRAwDgYDVQQKEwdSb290 -IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNB -IENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRA -Y2FjZXJ0Lm9yZzAeFw0wMzAzMzAxMjI5NDlaFw0zMzAzMjkxMjI5NDlaMHkxEDAO -BgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEi -MCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJ -ARYSc3VwcG9ydEBjYWNlcnQub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC -CgKCAgEAziLA4kZ97DYoB1CW8qAzQIxL8TtmPzHlawI229Z89vGIj053NgVBlfkJ -8BLPRoZzYLdufujAWGSuzbCtRRcMY/pnCujW0r8+55jE8Ez64AO7NV1sId6eINm6 -zWYyN3L69wj1x81YyY7nDl7qPv4coRQKFWyGhFtkZip6qUtTefWIonvuLwphK42y -fk1WpRPs6tqSnqxEQR5YYGUFZvjARL3LlPdCfgv3ZWiYUQXw8wWRBB0bF4LsyFe7 -w2t6iPGwcswlWyCR7BYCEo8y6RcYSNDHBS4CMEK4JZwFaz+qOqfrU0j36NK2B5jc -G8Y0f3/JHIJ6BVgrCFvzOKKrF11myZjXnhCLotLddJr3cQxyYN/Nb5gznZY0dj4k -epKwDpUeb+agRThHqtdB7Uq3EvbXG4OKDy7YCbZZ16oE/9KTfWgu3YtLq1i6L43q -laegw1SJpfvbi1EinbLDvhG+LJGGi5Z4rSDTii8aP8bQUWWHIbEZAWV/RRyH9XzQ -QUxPKZgh/TMfdQwEUfoZd9vUFBzugcMd9Zi3aQaRIt0AUMyBMawSB3s42mhb5ivU -fslfrejrckzzAeVLIL+aplfKkQABi6F1ITe1Yw1nPkZPcCBnzsXWWdsC4PDSy826 -YreQQejdIOQpvGQpQsgi3Hia/0PsmBsJUUtaWsJx8cTLc6nloQsCAwEAAaOCAc4w -ggHKMB0GA1UdDgQWBBQWtTIb1Mfz4OaO873SsDrusjkY0TCBowYDVR0jBIGbMIGY -gBQWtTIb1Mfz4OaO873SsDrusjkY0aF9pHsweTEQMA4GA1UEChMHUm9vdCBDQTEe -MBwGA1UECxMVaHR0cDovL3d3dy5jYWNlcnQub3JnMSIwIAYDVQQDExlDQSBDZXJ0 -IFNpZ25pbmcgQXV0aG9yaXR5MSEwHwYJKoZIhvcNAQkBFhJzdXBwb3J0QGNhY2Vy -dC5vcmeCAQAwDwYDVR0TAQH/BAUwAwEB/zAyBgNVHR8EKzApMCegJaAjhiFodHRw -czovL3d3dy5jYWNlcnQub3JnL3Jldm9rZS5jcmwwMAYJYIZIAYb4QgEEBCMWIWh0 -dHBzOi8vd3d3LmNhY2VydC5vcmcvcmV2b2tlLmNybDA0BglghkgBhvhCAQgEJxYl -aHR0cDovL3d3dy5jYWNlcnQub3JnL2luZGV4LnBocD9pZD0xMDBWBglghkgBhvhC -AQ0ESRZHVG8gZ2V0IHlvdXIgb3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQg -b3ZlciB0byBodHRwOi8vd3d3LmNhY2VydC5vcmcwDQYJKoZIhvcNAQEEBQADggIB -ACjH7pyCArpcgBLKNQodgW+JapnM8mgPf6fhjViVPr3yBsOQWqy1YPaZQwGjiHCc -nWKdpIevZ1gNMDY75q1I08t0AoZxPuIrA2jxNGJARjtT6ij0rPtmlVOKTV39O9lg -18p5aTuxZZKmxoGCXJzN600BiqXfEVWqFcofN8CCmHBh22p8lqOOLlQ+TyGpkO/c -gr/c6EWtTZBzCDyUZbAEmXZ/4rzCahWqlwQ3JNgelE5tDlG+1sSPypZt90Pf6DBl -Jzt7u0NDY8RD97LsaMzhGY4i+5jhe1o+ATc7iwiwovOVThrLm82asduycPAtStvY -sONvRUgzEv/+PDIqVPfE94rwiCPCR/5kenHA0R6mY7AHfqQv0wGP3J8rtsYIqQ+T -SCX8Ev2fQtzzxD72V7DX3WnRBnc0CkvSyqD/HMaMyRa+xMwyN2hzXwj7UfdJUzYF -CpUCTPJ5GhD22Dp1nPMd8aINcGeGG7MW9S/lpOt5hvk9C8JzC6WZrG/8Z7jlLwum -GCSNe9FINSkYQKyTYOGWhlC0elnYjyELn8+CkcY7v2vcB5G5l1YjqrZslMZIBjzk -zk6q5PYvCdxTby78dOs6Y5nCpqyJvKeyRKANihDjbPIky/qbn3BHLt4Ui9SyIAmW -omTxJBzcoTWcFbLUvFUufQb1nA5V9FrWk9p2rSVzTMVD ------END CERTIFICATE----- diff --git a/lib-python/2.7/test/selfsigned_pythontestdotnet.pem b/lib-python/2.7/test/selfsigned_pythontestdotnet.pem --- a/lib-python/2.7/test/selfsigned_pythontestdotnet.pem +++ b/lib-python/2.7/test/selfsigned_pythontestdotnet.pem @@ -1,5 +1,5 @@ -----BEGIN CERTIFICATE----- -MIIChzCCAfCgAwIBAgIJAKGU95wKR8pSMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNV +MIIClTCCAf6gAwIBAgIJAKGU95wKR8pTMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u IFNvZnR3YXJlIEZvdW5kYXRpb24xIzAhBgNVBAMMGnNlbGYtc2lnbmVkLnB5dGhv bnRlc3QubmV0MB4XDTE0MTEwMjE4MDkyOVoXDTI0MTAzMDE4MDkyOVowcDELMAkG @@ -8,9 +8,9 @@ aG9udGVzdC5uZXQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANDXQXW9tjyZ Xt0Iv2tLL1+jinr4wGg36ioLDLFkMf+2Y1GL0v0BnKYG4N1OKlAU15LXGeGer8vm Sv/yIvmdrELvhAbbo3w4a9TMYQA4XkIVLdvu3mvNOAet+8PMJxn26dbDhG809ALv -EHY57lQsBS3G59RZyBPVqAqmImWNJnVzAgMBAAGjKTAnMCUGA1UdEQQeMByCGnNl -bGYtc2lnbmVkLnB5dGhvbnRlc3QubmV0MA0GCSqGSIb3DQEBBQUAA4GBAIOXmdtM -eG9qzP9TiXW/Gc/zI4cBfdCpC+Y4gOfC9bQUC7hefix4iO3+iZjgy3X/FaRxUUoV -HKiXcXIaWqTSUWp45cSh0MbwZXudp6JIAptzdAhvvCrPKeC9i9GvxsPD4LtDAL97 -vSaxQBezA7hdxZd90/EeyMgVZgAnTCnvAWX9 +EHY57lQsBS3G59RZyBPVqAqmImWNJnVzAgMBAAGjNzA1MCUGA1UdEQQeMByCGnNl +bGYtc2lnbmVkLnB5dGhvbnRlc3QubmV0MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcN +AQEFBQADgYEAIuzAhgMouJpNdf3URCHIineyoSt6WK/9+eyUcjlKOrDoXNZaD72h +TXMeKYoWvJyVcSLKL8ckPtDobgP2OTt0UkyAaj0n+ZHaqq1lH2yVfGUA1ILJv515 +C8BqbvVZuqm3i7ygmw3bqE/lYMgOrYtXXnqOrz6nvsE6Yc9V9rFflOM= -----END CERTIFICATE----- diff --git a/lib-python/2.7/test/test_ssl.py b/lib-python/2.7/test/test_ssl.py --- a/lib-python/2.7/test/test_ssl.py +++ b/lib-python/2.7/test/test_ssl.py @@ -57,7 +57,8 @@ SIGNED_CERTFILE2 = data_file("keycert4.pem") SIGNING_CA = data_file("pycacert.pem") -SVN_PYTHON_ORG_ROOT_CERT = data_file("https_svn_python_org_root.pem") +REMOTE_HOST = "self-signed.pythontest.net" +REMOTE_ROOT_CERT = data_file("selfsigned_pythontestdotnet.pem") EMPTYCERT = data_file("nullcert.pem") BADCERT = data_file("badcert.pem") @@ -244,7 +245,7 @@ self.assertEqual(p['subjectAltName'], san) def test_DER_to_PEM(self): - with open(SVN_PYTHON_ORG_ROOT_CERT, 'r') as f: + with open(CAFILE_CACERT, 'r') as f: pem = f.read() d1 = ssl.PEM_cert_to_DER_cert(pem) p2 = ssl.DER_cert_to_PEM_cert(d1) @@ -792,7 +793,7 @@ # Mismatching key and cert ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) with self.assertRaisesRegexp(ssl.SSLError, "key values mismatch"): - ctx.load_cert_chain(SVN_PYTHON_ORG_ROOT_CERT, ONLYKEY) + ctx.load_cert_chain(CAFILE_CACERT, ONLYKEY) # Password protected key and cert ctx.load_cert_chain(CERTFILE_PROTECTED, password=KEY_PASSWORD) ctx.load_cert_chain(CERTFILE_PROTECTED, password=KEY_PASSWORD.encode()) @@ -1013,7 +1014,7 @@ ctx.load_verify_locations(CERTFILE) self.assertEqual(ctx.cert_store_stats(), {'x509_ca': 0, 'crl': 0, 'x509': 1}) - ctx.load_verify_locations(SVN_PYTHON_ORG_ROOT_CERT) + ctx.load_verify_locations(CAFILE_CACERT) self.assertEqual(ctx.cert_store_stats(), {'x509_ca': 1, 'crl': 0, 'x509': 2}) @@ -1023,8 +1024,8 @@ # CERTFILE is not flagged as X509v3 Basic Constraints: CA:TRUE ctx.load_verify_locations(CERTFILE) self.assertEqual(ctx.get_ca_certs(), []) - # but SVN_PYTHON_ORG_ROOT_CERT is a CA cert - ctx.load_verify_locations(SVN_PYTHON_ORG_ROOT_CERT) + # but CAFILE_CACERT is a CA cert + ctx.load_verify_locations(CAFILE_CACERT) self.assertEqual(ctx.get_ca_certs(), [{'issuer': ((('organizationName', 'Root CA'),), (('organizationalUnitName', 'http://www.cacert.org'),), @@ -1040,7 +1041,7 @@ (('emailAddress', 'support at cacert.org'),)), 'version': 3}]) - with open(SVN_PYTHON_ORG_ROOT_CERT) as f: + with open(CAFILE_CACERT) as f: pem = f.read() der = ssl.PEM_cert_to_DER_cert(pem) self.assertEqual(ctx.get_ca_certs(True), [der]) @@ -1215,11 +1216,11 @@ class NetworkedTests(unittest.TestCase): def test_connect(self): - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_NONE) try: - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) self.assertEqual({}, s.getpeercert()) finally: s.close() @@ -1228,27 +1229,27 @@ s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED) self.assertRaisesRegexp(ssl.SSLError, "certificate verify failed", - s.connect, ("svn.python.org", 443)) + s.connect, (REMOTE_HOST, 443)) s.close() # this should succeed because we specify the root cert s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED, - ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + ca_certs=REMOTE_ROOT_CERT) try: - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) self.assertTrue(s.getpeercert()) finally: s.close() def test_connect_ex(self): # Issue #11326: check connect_ex() implementation - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED, - ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + ca_certs=REMOTE_ROOT_CERT) try: - self.assertEqual(0, s.connect_ex(("svn.python.org", 443))) + self.assertEqual(0, s.connect_ex((REMOTE_HOST, 443))) self.assertTrue(s.getpeercert()) finally: s.close() @@ -1256,14 +1257,14 @@ def test_non_blocking_connect_ex(self): # Issue #11326: non-blocking connect_ex() should allow handshake # to proceed after the socket gets ready. - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED, - ca_certs=SVN_PYTHON_ORG_ROOT_CERT, + ca_certs=REMOTE_ROOT_CERT, do_handshake_on_connect=False) try: s.setblocking(False) - rc = s.connect_ex(('svn.python.org', 443)) + rc = s.connect_ex((REMOTE_HOST, 443)) # EWOULDBLOCK under Windows, EINPROGRESS elsewhere self.assertIn(rc, (0, errno.EINPROGRESS, errno.EWOULDBLOCK)) # Wait for connect to finish @@ -1285,58 +1286,62 @@ def test_timeout_connect_ex(self): # Issue #12065: on a timeout, connect_ex() should return the original # errno (mimicking the behaviour of non-SSL sockets). - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED, - ca_certs=SVN_PYTHON_ORG_ROOT_CERT, + ca_certs=REMOTE_ROOT_CERT, do_handshake_on_connect=False) try: s.settimeout(0.0000001) - rc = s.connect_ex(('svn.python.org', 443)) + rc = s.connect_ex((REMOTE_HOST, 443)) if rc == 0: - self.skipTest("svn.python.org responded too quickly") + self.skipTest("REMOTE_HOST responded too quickly") self.assertIn(rc, (errno.EAGAIN, errno.EWOULDBLOCK)) finally: s.close() def test_connect_ex_error(self): - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_REQUIRED, - ca_certs=SVN_PYTHON_ORG_ROOT_CERT) + ca_certs=REMOTE_ROOT_CERT) try: - rc = s.connect_ex(("svn.python.org", 444)) + rc = s.connect_ex((REMOTE_HOST, 444)) # Issue #19919: Windows machines or VMs hosted on Windows # machines sometimes return EWOULDBLOCK. - self.assertIn(rc, (errno.ECONNREFUSED, errno.EWOULDBLOCK)) + errors = ( + errno.ECONNREFUSED, errno.EHOSTUNREACH, errno.ETIMEDOUT, + errno.EWOULDBLOCK, + ) + self.assertIn(rc, errors) finally: s.close() def test_connect_with_context(self): - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): # Same as test_connect, but with a separately created context ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) s = ctx.wrap_socket(socket.socket(socket.AF_INET)) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) try: self.assertEqual({}, s.getpeercert()) finally: s.close() # Same with a server hostname s = ctx.wrap_socket(socket.socket(socket.AF_INET), - server_hostname="svn.python.org") - s.connect(("svn.python.org", 443)) + server_hostname=REMOTE_HOST) + s.connect((REMOTE_HOST, 443)) s.close() # This should fail because we have no verification certs ctx.verify_mode = ssl.CERT_REQUIRED s = ctx.wrap_socket(socket.socket(socket.AF_INET)) self.assertRaisesRegexp(ssl.SSLError, "certificate verify failed", - s.connect, ("svn.python.org", 443)) + s.connect, (REMOTE_HOST, 443)) s.close() # This should succeed because we specify the root cert - ctx.load_verify_locations(SVN_PYTHON_ORG_ROOT_CERT) + ctx.load_verify_locations(REMOTE_ROOT_CERT) s = ctx.wrap_socket(socket.socket(socket.AF_INET)) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) try: cert = s.getpeercert() self.assertTrue(cert) @@ -1349,12 +1354,12 @@ # OpenSSL 0.9.8n and 1.0.0, as a result the capath directory must # contain both versions of each certificate (same content, different # filename) for this test to be portable across OpenSSL releases. - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) ctx.verify_mode = ssl.CERT_REQUIRED ctx.load_verify_locations(capath=CAPATH) s = ctx.wrap_socket(socket.socket(socket.AF_INET)) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) try: cert = s.getpeercert() self.assertTrue(cert) @@ -1365,7 +1370,7 @@ ctx.verify_mode = ssl.CERT_REQUIRED ctx.load_verify_locations(capath=BYTES_CAPATH) s = ctx.wrap_socket(socket.socket(socket.AF_INET)) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) try: cert = s.getpeercert() self.assertTrue(cert) @@ -1373,15 +1378,15 @@ s.close() def test_connect_cadata(self): - with open(CAFILE_CACERT) as f: + with open(REMOTE_ROOT_CERT) as f: pem = f.read().decode('ascii') der = ssl.PEM_cert_to_DER_cert(pem) - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) ctx.verify_mode = ssl.CERT_REQUIRED ctx.load_verify_locations(cadata=pem) with closing(ctx.wrap_socket(socket.socket(socket.AF_INET))) as s: - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) cert = s.getpeercert() self.assertTrue(cert) @@ -1390,7 +1395,7 @@ ctx.verify_mode = ssl.CERT_REQUIRED ctx.load_verify_locations(cadata=der) with closing(ctx.wrap_socket(socket.socket(socket.AF_INET))) as s: - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) cert = s.getpeercert() self.assertTrue(cert) @@ -1399,9 +1404,9 @@ # Issue #5238: creating a file-like object with makefile() shouldn't # delay closing the underlying "real socket" (here tested with its # file descriptor, hence skipping the test under Windows). - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): ss = ssl.wrap_socket(socket.socket(socket.AF_INET)) - ss.connect(("svn.python.org", 443)) + ss.connect((REMOTE_HOST, 443)) fd = ss.fileno() f = ss.makefile() f.close() @@ -1415,9 +1420,9 @@ self.assertEqual(e.exception.errno, errno.EBADF) def test_non_blocking_handshake(self): - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): s = socket.socket(socket.AF_INET) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) s.setblocking(False) s = ssl.wrap_socket(s, cert_reqs=ssl.CERT_NONE, @@ -1460,12 +1465,12 @@ if support.verbose: sys.stdout.write("\nVerified certificate for %s:%s is\n%s\n" % (host, port ,pem)) - _test_get_server_certificate('svn.python.org', 443, SVN_PYTHON_ORG_ROOT_CERT) + _test_get_server_certificate(REMOTE_HOST, 443, REMOTE_ROOT_CERT) if support.IPV6_ENABLED: _test_get_server_certificate('ipv6.google.com', 443) def test_ciphers(self): - remote = ("svn.python.org", 443) + remote = (REMOTE_HOST, 443) with support.transient_internet(remote[0]): with closing(ssl.wrap_socket(socket.socket(socket.AF_INET), cert_reqs=ssl.CERT_NONE, ciphers="ALL")) as s: @@ -1510,13 +1515,13 @@ def test_get_ca_certs_capath(self): # capath certs are loaded on request - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) ctx.verify_mode = ssl.CERT_REQUIRED ctx.load_verify_locations(capath=CAPATH) self.assertEqual(ctx.get_ca_certs(), []) s = ctx.wrap_socket(socket.socket(socket.AF_INET)) - s.connect(("svn.python.org", 443)) + s.connect((REMOTE_HOST, 443)) try: cert = s.getpeercert() self.assertTrue(cert) @@ -1527,12 +1532,12 @@ @needs_sni def test_context_setget(self): # Check that the context of a connected socket can be replaced. - with support.transient_internet("svn.python.org"): + with support.transient_internet(REMOTE_HOST): ctx1 = ssl.SSLContext(ssl.PROTOCOL_TLSv1) ctx2 = ssl.SSLContext(ssl.PROTOCOL_SSLv23) s = socket.socket(socket.AF_INET) with closing(ctx1.wrap_socket(s)) as ss: - ss.connect(("svn.python.org", 443)) + ss.connect((REMOTE_HOST, 443)) self.assertIs(ss.context, ctx1) self.assertIs(ss._sslobj.context, ctx1) ss.context = ctx2 @@ -3026,7 +3031,7 @@ pass for filename in [ - CERTFILE, SVN_PYTHON_ORG_ROOT_CERT, BYTES_CERTFILE, + CERTFILE, REMOTE_ROOT_CERT, BYTES_CERTFILE, ONLYCERT, ONLYKEY, BYTES_ONLYCERT, BYTES_ONLYKEY, SIGNED_CERTFILE, SIGNED_CERTFILE2, SIGNING_CA, BADCERT, BADKEY, EMPTYCERT]: diff --git a/lib_pypy/cPickle.py b/lib_pypy/cPickle.py --- a/lib_pypy/cPickle.py +++ b/lib_pypy/cPickle.py @@ -167,7 +167,11 @@ try: key = ord(self.read(1)) while key != STOP: - self.dispatch[key](self) + try: + meth = self.dispatch[key] + except KeyError: + raise UnpicklingError("invalid load key, %r." % chr(key)) + meth(self) key = ord(self.read(1)) except TypeError: if self.read(1) == '': @@ -559,6 +563,7 @@ def decode_long(data): r"""Decode a long from a two's complement little-endian binary string. + This is overriden on PyPy by a RPython version that has linear complexity. >>> decode_long('') 0L @@ -592,6 +597,11 @@ n -= 1L << (nbytes << 3) return n +try: + from __pypy__ import decode_long +except ImportError: + pass + def load(f): return Unpickler(f).load() diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.4.2 +Version: 1.5.0 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI, CDefError, FFIError from .ffiplatform import VerificationError, VerificationMissing -__version__ = "1.4.2" -__version_info__ = (1, 4, 2) +__version__ = "1.5.0" +__version_info__ = (1, 5, 0) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -146,8 +146,9 @@ ((Py_ssize_t(*)(CTypeDescrObject *, PyObject *, char **))_cffi_exports[23]) #define _cffi_convert_array_from_object \ ((int(*)(char *, CTypeDescrObject *, PyObject *))_cffi_exports[24]) +#define _CFFI_CPIDX 25 #define _cffi_call_python \ - ((void(*)(struct _cffi_externpy_s *, char *))_cffi_exports[25]) + ((void(*)(struct _cffi_externpy_s *, char *))_cffi_exports[_CFFI_CPIDX]) #define _CFFI_NUM_EXPORTS 26 typedef struct _ctypedescr CTypeDescrObject; @@ -206,7 +207,8 @@ /********** end CPython-specific section **********/ #else _CFFI_UNUSED_FN -static void (*_cffi_call_python)(struct _cffi_externpy_s *, char *); +static void (*_cffi_call_python_org)(struct _cffi_externpy_s *, char *); +# define _cffi_call_python _cffi_call_python_org #endif diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py --- a/lib_pypy/cffi/api.py +++ b/lib_pypy/cffi/api.py @@ -74,6 +74,7 @@ self._windows_unicode = None self._init_once_cache = {} self._cdef_version = None + self._embedding = None if hasattr(backend, 'set_ffi'): backend.set_ffi(self) for name in backend.__dict__: @@ -101,13 +102,21 @@ If 'packed' is specified as True, all structs declared inside this cdef are packed, i.e. laid out without any field alignment at all. """ + self._cdef(csource, override=override, packed=packed) + + def embedding_api(self, csource, packed=False): + self._cdef(csource, packed=packed, dllexport=True) + if self._embedding is None: + self._embedding = '' + + def _cdef(self, csource, override=False, **options): if not isinstance(csource, str): # unicode, on Python 2 if not isinstance(csource, basestring): raise TypeError("cdef() argument must be a string") csource = csource.encode('ascii') with self._lock: self._cdef_version = object() - self._parser.parse(csource, override=override, packed=packed) + self._parser.parse(csource, override=override, **options) self._cdefsources.append(csource) if override: for cache in self._function_caches: @@ -533,6 +542,31 @@ ('_UNICODE', '1')] kwds['define_macros'] = defmacros + def _apply_embedding_fix(self, kwds): + # must include an argument like "-lpython2.7" for the compiler + if '__pypy__' in sys.builtin_module_names: + if hasattr(sys, 'prefix'): + import os + libdir = os.path.join(sys.prefix, 'bin') + dirs = kwds.setdefault('library_dirs', []) + if libdir not in dirs: + dirs.append(libdir) + pythonlib = "pypy-c" + else: + if sys.platform == "win32": + template = "python%d%d" + if sys.flags.debug: + template = template + '_d' + else: + template = "python%d.%d" + pythonlib = (template % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + if hasattr(sys, 'abiflags'): + pythonlib += sys.abiflags + libraries = kwds.setdefault('libraries', []) + if pythonlib not in libraries: + libraries.append(pythonlib) + def set_source(self, module_name, source, source_extension='.c', **kwds): if hasattr(self, '_assigned_source'): raise ValueError("set_source() cannot be called several times " @@ -592,14 +626,23 @@ recompile(self, module_name, source, c_file=filename, call_c_compiler=False, **kwds) - def compile(self, tmpdir='.', verbose=0): + def compile(self, tmpdir='.', verbose=0, target=None): + """The 'target' argument gives the final file name of the + compiled DLL. Use '*' to force distutils' choice, suitable for + regular CPython C API modules. Use a file name ending in '.*' + to ask for the system's default extension for dynamic libraries + (.so/.dll). + + The default is '*' when building a non-embedded C API extension, + and (module_name + '.*') when building an embedded library. + """ from .recompiler import recompile # if not hasattr(self, '_assigned_source'): raise ValueError("set_source() must be called before compile()") module_name, source, source_extension, kwds = self._assigned_source return recompile(self, module_name, source, tmpdir=tmpdir, - source_extension=source_extension, + target=target, source_extension=source_extension, compiler_verbose=verbose, **kwds) def init_once(self, func, tag): @@ -626,6 +669,32 @@ self._init_once_cache[tag] = (True, result) return result + def embedding_init_code(self, pysource): + if self._embedding: + raise ValueError("embedding_init_code() can only be called once") + # fix 'pysource' before it gets dumped into the C file: + # - remove empty lines at the beginning, so it starts at "line 1" + # - dedent, if all non-empty lines are indented + # - check for SyntaxErrors + import re + match = re.match(r'\s*\n', pysource) + if match: + pysource = pysource[match.end():] + lines = pysource.splitlines() or [''] + prefix = re.match(r'\s*', lines[0]).group() + for i in range(1, len(lines)): + line = lines[i] + if line.rstrip(): + while not line.startswith(prefix): + prefix = prefix[:-1] + i = len(prefix) + lines = [line[i:]+'\n' for line in lines] + pysource = ''.join(lines) + # + compile(pysource, "cffi_init", "exec") + # + self._embedding = pysource + def _load_backend_lib(backend, name, flags): if name is None: diff --git a/lib_pypy/cffi/cparser.py b/lib_pypy/cffi/cparser.py --- a/lib_pypy/cffi/cparser.py +++ b/lib_pypy/cffi/cparser.py @@ -220,8 +220,7 @@ self._included_declarations = set() self._anonymous_counter = 0 self._structnode2type = weakref.WeakKeyDictionary() - self._override = False - self._packed = False + self._options = None self._int_constants = {} self._recomplete = [] self._uses_new_feature = None @@ -281,16 +280,15 @@ msg = 'parse error\n%s' % (msg,) raise api.CDefError(msg) - def parse(self, csource, override=False, packed=False): - prev_override = self._override - prev_packed = self._packed + def parse(self, csource, override=False, packed=False, dllexport=False): + prev_options = self._options try: - self._override = override - self._packed = packed + self._options = {'override': override, + 'packed': packed, + 'dllexport': dllexport} self._internal_parse(csource) finally: - self._override = prev_override - self._packed = prev_packed + self._options = prev_options def _internal_parse(self, csource): ast, macros, csource = self._parse(csource) @@ -376,10 +374,13 @@ def _declare_function(self, tp, quals, decl): tp = self._get_type_pointer(tp, quals) - if self._inside_extern_python: - self._declare('extern_python ' + decl.name, tp) + if self._options['dllexport']: + tag = 'dllexport_python ' + elif self._inside_extern_python: + tag = 'extern_python ' else: - self._declare('function ' + decl.name, tp) + tag = 'function ' + self._declare(tag + decl.name, tp) def _parse_decl(self, decl): node = decl.type @@ -449,7 +450,7 @@ prevobj, prevquals = self._declarations[name] if prevobj is obj and prevquals == quals: return - if not self._override: + if not self._options['override']: raise api.FFIError( "multiple declarations of %s (for interactive usage, " "try cdef(xx, override=True))" % (name,)) @@ -728,7 +729,7 @@ if isinstance(tp, model.StructType) and tp.partial: raise NotImplementedError("%s: using both bitfields and '...;'" % (tp,)) - tp.packed = self._packed + tp.packed = self._options['packed'] if tp.completed: # must be re-completed: it is not opaque any more tp.completed = 0 self._recomplete.append(tp) diff --git a/lib_pypy/cffi/ffiplatform.py b/lib_pypy/cffi/ffiplatform.py --- a/lib_pypy/cffi/ffiplatform.py +++ b/lib_pypy/cffi/ffiplatform.py @@ -21,12 +21,14 @@ allsources.append(os.path.normpath(src)) return Extension(name=modname, sources=allsources, **kwds) -def compile(tmpdir, ext, compiler_verbose=0): +def compile(tmpdir, ext, compiler_verbose=0, target_extension=None, + embedding=False): """Compile a C extension module using distutils.""" saved_environ = os.environ.copy() try: - outputfilename = _build(tmpdir, ext, compiler_verbose) + outputfilename = _build(tmpdir, ext, compiler_verbose, + target_extension, embedding) outputfilename = os.path.abspath(outputfilename) finally: # workaround for a distutils bugs where some env vars can @@ -36,7 +38,32 @@ os.environ[key] = value return outputfilename -def _build(tmpdir, ext, compiler_verbose=0): +def _save_val(name): + import distutils.sysconfig + config_vars = distutils.sysconfig.get_config_vars() + return config_vars.get(name, Ellipsis) + +def _restore_val(name, value): + import distutils.sysconfig + config_vars = distutils.sysconfig.get_config_vars() + config_vars[name] = value + if value is Ellipsis: + del config_vars[name] + +def _win32_hack_for_embedding(): + from distutils.msvc9compiler import MSVCCompiler + if not hasattr(MSVCCompiler, '_remove_visual_c_ref_CFFI_BAK'): + MSVCCompiler._remove_visual_c_ref_CFFI_BAK = \ + MSVCCompiler._remove_visual_c_ref + MSVCCompiler._remove_visual_c_ref = lambda self,manifest_file: manifest_file + +def _win32_unhack_for_embedding(): + from distutils.msvc9compiler import MSVCCompiler + MSVCCompiler._remove_visual_c_ref = \ + MSVCCompiler._remove_visual_c_ref_CFFI_BAK + +def _build(tmpdir, ext, compiler_verbose=0, target_extension=None, + embedding=False): # XXX compact but horrible :-( from distutils.core import Distribution import distutils.errors, distutils.log @@ -49,18 +76,29 @@ options['build_temp'] = ('ffiplatform', tmpdir) # try: + if sys.platform == 'win32' and embedding: + _win32_hack_for_embedding() old_level = distutils.log.set_threshold(0) or 0 + old_SO = _save_val('SO') + old_EXT_SUFFIX = _save_val('EXT_SUFFIX') try: + if target_extension is not None: + _restore_val('SO', target_extension) + _restore_val('EXT_SUFFIX', target_extension) distutils.log.set_verbosity(compiler_verbose) dist.run_command('build_ext') + cmd_obj = dist.get_command_obj('build_ext') + [soname] = cmd_obj.get_outputs() finally: distutils.log.set_threshold(old_level) + _restore_val('SO', old_SO) + _restore_val('EXT_SUFFIX', old_EXT_SUFFIX) + if sys.platform == 'win32' and embedding: + _win32_unhack_for_embedding() except (distutils.errors.CompileError, distutils.errors.LinkError) as e: raise VerificationError('%s: %s' % (e.__class__.__name__, e)) # - cmd_obj = dist.get_command_obj('build_ext') - [soname] = cmd_obj.get_outputs() return soname try: diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py --- a/lib_pypy/cffi/recompiler.py +++ b/lib_pypy/cffi/recompiler.py @@ -3,6 +3,7 @@ from .cffi_opcode import * VERSION = "0x2601" +VERSION_EMBEDDED = "0x2701" class GlobalExpr: @@ -281,6 +282,29 @@ lines[i:i+1] = self._rel_readlines('parse_c_type.h') prnt(''.join(lines)) # + # if we have ffi._embedding != None, we give it here as a macro + # and include an extra file + base_module_name = self.module_name.split('.')[-1] + if self.ffi._embedding is not None: + prnt('#define _CFFI_MODULE_NAME "%s"' % (self.module_name,)) + prnt('#define _CFFI_PYTHON_STARTUP_CODE %s' % + (self._string_literal(self.ffi._embedding),)) + prnt('#ifdef PYPY_VERSION') + prnt('# define _CFFI_PYTHON_STARTUP_FUNC _cffi_pypyinit_%s' % ( + base_module_name,)) + prnt('#elif PY_MAJOR_VERSION >= 3') + prnt('# define _CFFI_PYTHON_STARTUP_FUNC PyInit_%s' % ( + base_module_name,)) + prnt('#else') + prnt('# define _CFFI_PYTHON_STARTUP_FUNC init%s' % ( + base_module_name,)) + prnt('#endif') + lines = self._rel_readlines('_embedding.h') + prnt(''.join(lines)) + version = VERSION_EMBEDDED + else: + version = VERSION + # # then paste the C source given by the user, verbatim. prnt('/************************************************************/') prnt() @@ -365,17 +389,16 @@ prnt() # # the init function - base_module_name = self.module_name.split('.')[-1] prnt('#ifdef PYPY_VERSION') prnt('PyMODINIT_FUNC') prnt('_cffi_pypyinit_%s(const void *p[])' % (base_module_name,)) prnt('{') if self._num_externpy: prnt(' if (((intptr_t)p[0]) >= 0x0A03) {') - prnt(' _cffi_call_python = ' + prnt(' _cffi_call_python_org = ' '(void(*)(struct _cffi_externpy_s *, char *))p[1];') prnt(' }') - prnt(' p[0] = (const void *)%s;' % VERSION) + prnt(' p[0] = (const void *)%s;' % version) prnt(' p[1] = &_cffi_type_context;') prnt('}') # on Windows, distutils insists on putting init_cffi_xyz in @@ -394,14 +417,14 @@ prnt('PyInit_%s(void)' % (base_module_name,)) prnt('{') prnt(' return _cffi_init("%s", %s, &_cffi_type_context);' % ( - self.module_name, VERSION)) + self.module_name, version)) prnt('}') prnt('#else') prnt('PyMODINIT_FUNC') prnt('init%s(void)' % (base_module_name,)) prnt('{') prnt(' _cffi_init("%s", %s, &_cffi_type_context);' % ( - self.module_name, VERSION)) + self.module_name, version)) prnt('}') prnt('#endif') @@ -1123,7 +1146,10 @@ assert isinstance(tp, model.FunctionPtrType) self._do_collect_type(tp) - def _generate_cpy_extern_python_decl(self, tp, name): + def _generate_cpy_dllexport_python_collecttype(self, tp, name): + self._generate_cpy_extern_python_collecttype(tp, name) + + def _generate_cpy_extern_python_decl(self, tp, name, dllexport=False): prnt = self._prnt if isinstance(tp.result, model.VoidType): size_of_result = '0' @@ -1156,7 +1182,11 @@ size_of_a = 'sizeof(%s) > %d ? sizeof(%s) : %d' % ( tp.result.get_c_name(''), size_of_a, tp.result.get_c_name(''), size_of_a) - prnt('static %s' % tp.result.get_c_name(name_and_arguments)) + if dllexport: + tag = 'CFFI_DLLEXPORT' + else: + tag = 'static' + prnt('%s %s' % (tag, tp.result.get_c_name(name_and_arguments))) prnt('{') prnt(' char a[%s];' % size_of_a) prnt(' char *p = a;') @@ -1174,6 +1204,9 @@ prnt() self._num_externpy += 1 + def _generate_cpy_dllexport_python_decl(self, tp, name): + self._generate_cpy_extern_python_decl(tp, name, dllexport=True) + def _generate_cpy_extern_python_ctx(self, tp, name): if self.target_is_python: raise ffiplatform.VerificationError( @@ -1185,6 +1218,21 @@ self._lsts["global"].append( GlobalExpr(name, '&_cffi_externpy__%s' % name, type_op, name)) + def _generate_cpy_dllexport_python_ctx(self, tp, name): + self._generate_cpy_extern_python_ctx(tp, name) + + def _string_literal(self, s): + def _char_repr(c): + # escape with a '\' the characters '\', '"' or (for trigraphs) '?' + if c in '\\"?': return '\\' + c + if ' ' <= c < '\x7F': return c + if c == '\n': return '\\n' + return '\\%03o' % ord(c) + lines = [] + for line in s.splitlines(True): + lines.append('"%s"' % ''.join([_char_repr(c) for c in line])) + return ' \\\n'.join(lines) + # ---------- # emitting the opcodes for individual types @@ -1311,12 +1359,15 @@ def recompile(ffi, module_name, preamble, tmpdir='.', call_c_compiler=True, c_file=None, source_extension='.c', extradir=None, - compiler_verbose=1, **kwds): + compiler_verbose=1, target=None, **kwds): if not isinstance(module_name, str): module_name = module_name.encode('ascii') if ffi._windows_unicode: ffi._apply_windows_unicode(kwds) if preamble is not None: + embedding = (ffi._embedding is not None) + if embedding: + ffi._apply_embedding_fix(kwds) if c_file is None: c_file, parts = _modname_to_file(tmpdir, module_name, source_extension) @@ -1325,13 +1376,40 @@ ext_c_file = os.path.join(*parts) else: ext_c_file = c_file - ext = ffiplatform.get_extension(ext_c_file, module_name, **kwds) + # + if target is None: + if embedding: + target = '%s.*' % module_name + else: + target = '*' + if target == '*': + target_module_name = module_name + target_extension = None # use default + else: + if target.endswith('.*'): + target = target[:-2] + if sys.platform == 'win32': + target += '.dll' + else: + target += '.so' + # split along the first '.' (not the last one, otherwise the + # preceeding dots are interpreted as splitting package names) + index = target.find('.') + if index < 0: + raise ValueError("target argument %r should be a file name " + "containing a '.'" % (target,)) + target_module_name = target[:index] + target_extension = target[index:] + # + ext = ffiplatform.get_extension(ext_c_file, target_module_name, **kwds) updated = make_c_source(ffi, module_name, preamble, c_file) if call_c_compiler: cwd = os.getcwd() try: os.chdir(tmpdir) - outputfilename = ffiplatform.compile('.', ext, compiler_verbose) + outputfilename = ffiplatform.compile('.', ext, compiler_verbose, + target_extension, + embedding=embedding) finally: os.chdir(cwd) return outputfilename diff --git a/pypy/doc/embedding.rst b/pypy/doc/embedding.rst --- a/pypy/doc/embedding.rst +++ b/pypy/doc/embedding.rst @@ -10,6 +10,15 @@ with a ``libpypy-c.so`` or ``pypy-c.dll`` file. This is the default in recent versions of PyPy. +.. note:: + + The interface described in this page is kept for backward compatibility. + From PyPy 4.1, it is recommended to use instead CFFI's `native embedding + support,`__ which gives a simpler approach that works on CPython as well + as PyPy. + +.. __: http://cffi.readthedocs.org/en/latest/embedding.html + The resulting shared library exports very few functions, however they are enough to accomplish everything you need, provided you follow a few principles. The API is: diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst --- a/pypy/doc/getting-started-dev.rst +++ b/pypy/doc/getting-started-dev.rst @@ -19,7 +19,9 @@ * Clone this new repo (i.e. the fork) to your local machine with the command ``hg clone ssh://hg at bitbucket.org/yourname/pypy``. It is a very slow - operation but only ever needs to be done once. If you already cloned + operation but only ever needs to be done once. See also + http://pypy.org/download.html#building-from-source . + If you already cloned ``https://bitbucket.org/pypy/pypy`` before, even if some time ago, then you can reuse the same clone by editing the file ``.hg/hgrc`` in your clone to contain the line ``default = diff --git a/pypy/doc/how-to-contribute.rst b/pypy/doc/how-to-contribute.rst --- a/pypy/doc/how-to-contribute.rst +++ b/pypy/doc/how-to-contribute.rst @@ -67,8 +67,8 @@ **module** directory contains extension modules written in RPython * **rpython compiler** that resides in ``rpython/annotator`` and - ``rpython/rtyper`` directories. Consult :doc:`introduction to RPython ` for - further reading + ``rpython/rtyper`` directories. Consult `Getting Started with RPython`_ + for further reading * **JIT generator** lives in ``rpython/jit`` directory. optimizations live in ``rpython/jit/metainterp/optimizeopt``, the main JIT in @@ -80,3 +80,14 @@ The rest of directories serve specific niche goal and are unlikely a good entry point. + + +More documentation +------------------ + +* `Getting Started Developing With PyPy`_ + +* `Getting Started with RPython`_ + +.. _`Getting Started Developing With PyPy`: getting-started-dev.html +.. _`Getting started with RPython`: http://rpython.readthedocs.org/en/latest/getting-started.html diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -5,6 +5,8 @@ .. this is a revision shortly after release-4.0.1 .. startrev: 4b5c840d0da2 +Fixed ``_PyLong_FromByteArray()``, which was buggy. + .. branch: numpy-1.10 Fix tests to run cleanly with -A and start to fix micronumpy for upstream numpy @@ -101,3 +103,33 @@ Fix the cryptic exception message when attempting to use extended slicing in rpython. Was issue #2211. + +.. branch: ec-keepalive + +Optimize the case where, in a new C-created thread, we keep invoking +short-running Python callbacks. (CFFI on CPython has a hack to achieve +the same result.) This can also be seen as a bug fix: previously, +thread-local objects would be reset between two such calls. + +.. branch: globals-quasiimmut + +Optimize global lookups. + +.. branch: cffi-static-callback-embedding + +Updated to CFFI 1.5, which supports a new way to do embedding. +Deprecates http://pypy.readthedocs.org/en/latest/embedding.html. + +.. branch: fix-cpython-ssl-tests-2.7 + +Fix SSL tests by importing cpython's patch + +.. branch: remove-getfield-pure + +Remove pure variants of ``getfield_gc_*`` operations from the JIT. Relevant +optimizations instead consult the field descriptor to determine the purity of +the operation. Additionally, pure ``getfield`` operations are now handled +entirely by `rpython/jit/metainterp/optimizeopt/heap.py` rather than +`rpython/jit/metainterp/optimizeopt/pure.py`, which can result in better codegen +for traces containing a large number of pure getfield operations. + diff --git a/pypy/goal/targetpypystandalone.py b/pypy/goal/targetpypystandalone.py --- a/pypy/goal/targetpypystandalone.py +++ b/pypy/goal/targetpypystandalone.py @@ -84,13 +84,6 @@ from rpython.rlib.entrypoint import entrypoint_highlevel from rpython.rtyper.lltypesystem import rffi, lltype - w_pathsetter = space.appexec([], """(): - def f(path): - import sys - sys.path[:] = path - return f - """) - @entrypoint_highlevel('main', [rffi.CCHARP, rffi.INT], c_name='pypy_setup_home') def pypy_setup_home(ll_home, verbose): @@ -109,7 +102,10 @@ " not found in '%s' or in any parent directory" % home1) return rffi.cast(rffi.INT, 1) space.startup() - space.call_function(w_pathsetter, w_path) + space.appexec([w_path], """(path): + import sys + sys.path[:] = path + """) # import site try: space.setattr(space.getbuiltinmodule('sys'), @@ -149,6 +145,9 @@ return os_thread.setup_threads(space) os_thread.bootstrapper.acquire(space, None, None) + # XXX this doesn't really work. Don't use os.fork(), and + # if your embedder program uses fork(), don't use any PyPy + # code in the fork rthread.gc_thread_start() os_thread.bootstrapper.nbthreads += 1 os_thread.bootstrapper.release() diff --git a/pypy/interpreter/eval.py b/pypy/interpreter/eval.py --- a/pypy/interpreter/eval.py +++ b/pypy/interpreter/eval.py @@ -9,8 +9,8 @@ class Code(W_Root): """A code is a compiled version of some source code. Abstract base class.""" - _immutable_ = True hidden_applevel = False + _immutable_fields_ = ['co_name', 'fast_natural_arity', 'hidden_applevel'] # n >= 0 : arity # FLATPYCALL = 0x100 diff --git a/pypy/interpreter/pycode.py b/pypy/interpreter/pycode.py --- a/pypy/interpreter/pycode.py +++ b/pypy/interpreter/pycode.py @@ -56,11 +56,13 @@ class PyCode(eval.Code): "CPython-style code objects." - _immutable_ = True - _immutable_fields_ = ["co_consts_w[*]", "co_names_w[*]", "co_varnames[*]", - "co_freevars[*]", "co_cellvars[*]", - "_args_as_cellvars[*]"] - + _immutable_fields_ = ["_signature", "co_argcount", "co_cellvars[*]", + "co_code", "co_consts_w[*]", "co_filename", + "co_firstlineno", "co_flags", "co_freevars[*]", + "co_lnotab", "co_names_w[*]", "co_nlocals", + "co_stacksize", "co_varnames[*]", + "_args_as_cellvars[*]", "w_globals?"] + def __init__(self, space, argcount, nlocals, stacksize, flags, code, consts, names, varnames, filename, name, firstlineno, lnotab, freevars, cellvars, @@ -84,6 +86,10 @@ self.co_name = name self.co_firstlineno = firstlineno self.co_lnotab = lnotab + # store the first globals object that the code object is run in in + # here. if a frame is run in that globals object, it does not need to + # store it at all + self.w_globals = None self.hidden_applevel = hidden_applevel self.magic = magic self._signature = cpython_code_signature(self) @@ -91,6 +97,14 @@ self._init_ready() self.new_code_hook() + def frame_stores_global(self, w_globals): + if self.w_globals is None: + self.w_globals = w_globals + return False + if self.w_globals is w_globals: + return False + return True + def new_code_hook(self): code_hook = self.space.fromcache(CodeHookCache)._code_hook if code_hook is not None: diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py --- a/pypy/interpreter/pyframe.py +++ b/pypy/interpreter/pyframe.py @@ -36,6 +36,7 @@ def __init__(self, pycode): self.f_lineno = pycode.co_firstlineno + self.w_globals = pycode.w_globals class PyFrame(W_Root): """Represents a frame for a regular Python function @@ -67,7 +68,6 @@ escaped = False # see mark_as_escaped() debugdata = None - w_globals = None pycode = None # code object executed by that frame locals_cells_stack_w = None # the list of all locals, cells and the valuestack valuestackdepth = 0 # number of items on valuestack @@ -90,8 +90,9 @@ self = hint(self, access_directly=True, fresh_virtualizable=True) assert isinstance(code, pycode.PyCode) self.space = space - self.w_globals = w_globals self.pycode = code + if code.frame_stores_global(w_globals): + self.getorcreatedebug().w_globals = w_globals ncellvars = len(code.co_cellvars) nfreevars = len(code.co_freevars) size = code.co_nlocals + ncellvars + nfreevars + code.co_stacksize @@ -116,6 +117,12 @@ self.debugdata = FrameDebugData(self.pycode) return self.debugdata + def get_w_globals(self): + debugdata = self.getdebug() + if debugdata is not None: + return debugdata.w_globals + return jit.promote(self.pycode).w_globals + def get_w_f_trace(self): d = self.getdebug() if d is None: @@ -201,8 +208,9 @@ if flags & pycode.CO_NEWLOCALS: self.getorcreatedebug().w_locals = self.space.newdict(module=True) else: - assert self.w_globals is not None - self.getorcreatedebug().w_locals = self.w_globals + w_globals = self.get_w_globals() + assert w_globals is not None + self.getorcreatedebug().w_locals = w_globals ncellvars = len(code.co_cellvars) nfreevars = len(code.co_freevars) @@ -449,7 +457,7 @@ w_blockstack, w_exc_value, # last_exception w_tb, # - self.w_globals, + self.get_w_globals(), w(self.last_instr), w(self.frame_finished_execution), w(f_lineno), @@ -658,6 +666,11 @@ def fget_getdictscope(self, space): return self.getdictscope() + def fget_w_globals(self, space): + # bit silly, but GetSetProperty passes a space + return self.get_w_globals() + + ### line numbers ### def fget_f_lineno(self, space): diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py --- a/pypy/interpreter/pyopcode.py +++ b/pypy/interpreter/pyopcode.py @@ -837,7 +837,7 @@ w_bases = self.popvalue() w_name = self.popvalue() w_metaclass = find_metaclass(self.space, w_bases, - w_methodsdict, self.w_globals, + w_methodsdict, self.get_w_globals(), self.space.wrap(self.get_builtin())) w_newclass = self.space.call_function(w_metaclass, w_name, w_bases, w_methodsdict) @@ -881,14 +881,14 @@ def STORE_GLOBAL(self, nameindex, next_instr): varname = self.getname_u(nameindex) w_newvalue = self.popvalue() - self.space.setitem_str(self.w_globals, varname, w_newvalue) + self.space.setitem_str(self.get_w_globals(), varname, w_newvalue) def DELETE_GLOBAL(self, nameindex, next_instr): w_varname = self.getname_w(nameindex) - self.space.delitem(self.w_globals, w_varname) + self.space.delitem(self.get_w_globals(), w_varname) def LOAD_NAME(self, nameindex, next_instr): - if self.getorcreatedebug().w_locals is not self.w_globals: + if self.getorcreatedebug().w_locals is not self.get_w_globals(): varname = self.getname_u(nameindex) w_value = self.space.finditem_str(self.getorcreatedebug().w_locals, varname) @@ -898,7 +898,7 @@ self.LOAD_GLOBAL(nameindex, next_instr) # fall-back def _load_global(self, varname): - w_value = self.space.finditem_str(self.w_globals, varname) + w_value = self.space.finditem_str(self.get_w_globals(), varname) if w_value is None: # not in the globals, now look in the built-ins w_value = self.get_builtin().getdictvalue(self.space, varname) @@ -1029,7 +1029,7 @@ if w_locals is None: # CPython does this w_locals = space.w_None w_modulename = space.wrap(modulename) - w_globals = self.w_globals + w_globals = self.get_w_globals() if w_flag is None: w_obj = space.call_function(w_import, w_modulename, w_globals, w_locals, w_fromlist) @@ -1237,7 +1237,7 @@ w_codeobj = self.popvalue() codeobj = self.space.interp_w(PyCode, w_codeobj) defaultarguments = self.popvalues(numdefaults) - fn = function.Function(self.space, codeobj, self.w_globals, + fn = function.Function(self.space, codeobj, self.get_w_globals(), defaultarguments) self.pushvalue(self.space.wrap(fn)) @@ -1249,7 +1249,7 @@ freevars = [self.space.interp_w(Cell, cell) for cell in self.space.fixedview(w_freevarstuple)] defaultarguments = self.popvalues(numdefaults) - fn = function.Function(self.space, codeobj, self.w_globals, + fn = function.Function(self.space, codeobj, self.get_w_globals(), defaultarguments, freevars) self.pushvalue(self.space.wrap(fn)) diff --git a/pypy/interpreter/test/test_pyframe.py b/pypy/interpreter/test/test_pyframe.py --- a/pypy/interpreter/test/test_pyframe.py +++ b/pypy/interpreter/test/test_pyframe.py @@ -34,6 +34,7 @@ import sys f = sys._getframe() assert f.f_globals is globals() + raises(TypeError, "f.f_globals = globals()") def test_f_builtins(self): import sys, __builtin__ diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py --- a/pypy/interpreter/typedef.py +++ b/pypy/interpreter/typedef.py @@ -772,7 +772,7 @@ f_restricted = GetSetProperty(PyFrame.fget_f_restricted), f_code = GetSetProperty(PyFrame.fget_code), f_locals = GetSetProperty(PyFrame.fget_getdictscope), - f_globals = interp_attrproperty_w('w_globals', cls=PyFrame), + f_globals = GetSetProperty(PyFrame.fget_w_globals), ) assert not PyFrame.typedef.acceptable_as_base_class # no __new__ diff --git a/pypy/module/__builtin__/compiling.py b/pypy/module/__builtin__/compiling.py --- a/pypy/module/__builtin__/compiling.py +++ b/pypy/module/__builtin__/compiling.py @@ -93,7 +93,7 @@ if space.is_none(w_locals): w_locals = w_globals else: - w_globals = caller.w_globals + w_globals = caller.get_w_globals() if space.is_none(w_locals): w_locals = caller.getdictscope() elif space.is_none(w_locals): diff --git a/pypy/module/__builtin__/interp_inspect.py b/pypy/module/__builtin__/interp_inspect.py --- a/pypy/module/__builtin__/interp_inspect.py +++ b/pypy/module/__builtin__/interp_inspect.py @@ -2,7 +2,7 @@ def globals(space): "Return the dictionary containing the current scope's global variables." ec = space.getexecutioncontext() - return ec.gettopframe_nohidden().w_globals + return ec.gettopframe_nohidden().get_w_globals() def locals(space): """Return a dictionary containing the current scope's local variables. diff --git a/pypy/module/__builtin__/test/test_classobj.py b/pypy/module/__builtin__/test/test_classobj.py --- a/pypy/module/__builtin__/test/test_classobj.py +++ b/pypy/module/__builtin__/test/test_classobj.py @@ -1084,7 +1084,7 @@ def is_strdict(space, w_class): from pypy.objspace.std.dictmultiobject import BytesDictStrategy w_d = w_class.getdict(space) - return space.wrap(isinstance(w_d.strategy, BytesDictStrategy)) + return space.wrap(isinstance(w_d.get_strategy(), BytesDictStrategy)) cls.w_is_strdict = cls.space.wrap(gateway.interp2app(is_strdict)) diff --git a/pypy/module/__pypy__/__init__.py b/pypy/module/__pypy__/__init__.py --- a/pypy/module/__pypy__/__init__.py +++ b/pypy/module/__pypy__/__init__.py @@ -89,6 +89,7 @@ 'set_code_callback' : 'interp_magic.set_code_callback', 'save_module_content_for_future_reload': 'interp_magic.save_module_content_for_future_reload', + 'decode_long' : 'interp_magic.decode_long', } if sys.platform == 'win32': interpleveldefs['get_console_cp'] = 'interp_magic.get_console_cp' diff --git a/pypy/module/__pypy__/interp_magic.py b/pypy/module/__pypy__/interp_magic.py --- a/pypy/module/__pypy__/interp_magic.py +++ b/pypy/module/__pypy__/interp_magic.py @@ -1,4 +1,4 @@ -from pypy.interpreter.error import OperationError, wrap_oserror +from pypy.interpreter.error import OperationError, oefmt, wrap_oserror from pypy.interpreter.gateway import unwrap_spec from pypy.interpreter.pycode import CodeHookCache from pypy.interpreter.pyframe import PyFrame @@ -93,7 +93,7 @@ Return the underlying strategy currently used by a dict, list or set object """ if isinstance(w_obj, W_DictMultiObject): - name = w_obj.strategy.__class__.__name__ + name = w_obj.get_strategy().__class__.__name__ elif isinstance(w_obj, W_ListObject): name = w_obj.strategy.__class__.__name__ elif isinstance(w_obj, W_BaseSetObject): @@ -158,4 +158,13 @@ if space.is_none(w_callable): cache._code_hook = None else: - cache._code_hook = w_callable \ No newline at end of file + cache._code_hook = w_callable + + at unwrap_spec(string=str, byteorder=str, signed=int) +def decode_long(space, string, byteorder='little', signed=1): + from rpython.rlib.rbigint import rbigint, InvalidEndiannessError + try: + result = rbigint.frombytes(string, byteorder, bool(signed)) + except InvalidEndiannessError: + raise oefmt(space.w_ValueError, "invalid byteorder argument") + return space.newlong_from_rbigint(result) diff --git a/pypy/module/__pypy__/test/test_magic.py b/pypy/module/__pypy__/test/test_magic.py --- a/pypy/module/__pypy__/test/test_magic.py +++ b/pypy/module/__pypy__/test/test_magic.py @@ -30,4 +30,20 @@ """ in d finally: __pypy__.set_code_callback(None) - assert d['f'].__code__ in l \ No newline at end of file + assert d['f'].__code__ in l + + def test_decode_long(self): + from __pypy__ import decode_long + assert decode_long('') == 0 + assert decode_long('\xff\x00') == 255 + assert decode_long('\xff\x7f') == 32767 + assert decode_long('\x00\xff') == -256 + assert decode_long('\x00\x80') == -32768 + assert decode_long('\x80') == -128 + assert decode_long('\x7f') == 127 + assert decode_long('\x55' * 97) == (1 << (97 * 8)) // 3 + assert decode_long('\x00\x80', 'big') == 128 + assert decode_long('\xff\x7f', 'little', False) == 32767 + assert decode_long('\x00\x80', 'little', False) == 32768 + assert decode_long('\x00\x80', 'little', True) == -32768 + raises(ValueError, decode_long, '', 'foo') diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -1,8 +1,9 @@ import sys from pypy.interpreter.mixedmodule import MixedModule -from rpython.rlib import rdynload, clibffi +from rpython.rlib import rdynload, clibffi, entrypoint +from rpython.rtyper.lltypesystem import rffi -VERSION = "1.4.2" +VERSION = "1.5.0" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: @@ -65,6 +66,10 @@ if has_stdcall: interpleveldefs['FFI_STDCALL'] = 'space.wrap(%d)' % FFI_STDCALL + def startup(self, space): + from pypy.module._cffi_backend import embedding + embedding.glob.space = space + def get_dict_rtld_constants(): found = {} @@ -78,3 +83,11 @@ for _name, _value in get_dict_rtld_constants().items(): Module.interpleveldefs[_name] = 'space.wrap(%d)' % _value + + +# write this entrypoint() here, to make sure it is registered early enough + at entrypoint.entrypoint_highlevel('main', [rffi.INT, rffi.VOIDP], + c_name='pypy_init_embedded_cffi_module') +def pypy_init_embedded_cffi_module(version, init_struct): + from pypy.module._cffi_backend import embedding + return embedding.pypy_init_embedded_cffi_module(version, init_struct) diff --git a/pypy/module/_cffi_backend/cffi1_module.py b/pypy/module/_cffi_backend/cffi1_module.py --- a/pypy/module/_cffi_backend/cffi1_module.py +++ b/pypy/module/_cffi_backend/cffi1_module.py @@ -2,24 +2,25 @@ from pypy.interpreter.error import oefmt from pypy.interpreter.module import Module +from pypy.module import _cffi_backend from pypy.module._cffi_backend import parse_c_type from pypy.module._cffi_backend.ffi_obj import W_FFIObject from pypy.module._cffi_backend.lib_obj import W_LibObject VERSION_MIN = 0x2601 -VERSION_MAX = 0x26FF +VERSION_MAX = 0x27FF VERSION_EXPORT = 0x0A03 -initfunctype = lltype.Ptr(lltype.FuncType([rffi.VOIDPP], lltype.Void)) +INITFUNCPTR = lltype.Ptr(lltype.FuncType([rffi.VOIDPP], lltype.Void)) def load_cffi1_module(space, name, path, initptr): # This is called from pypy.module.cpyext.api.load_extension_module() from pypy.module._cffi_backend.call_python import get_ll_cffi_call_python - initfunc = rffi.cast(initfunctype, initptr) + initfunc = rffi.cast(INITFUNCPTR, initptr) with lltype.scoped_alloc(rffi.VOIDPP.TO, 16, zero=True) as p: p[0] = rffi.cast(rffi.VOIDP, VERSION_EXPORT) p[1] = rffi.cast(rffi.VOIDP, get_ll_cffi_call_python()) @@ -27,8 +28,10 @@ version = rffi.cast(lltype.Signed, p[0]) if not (VERSION_MIN <= version <= VERSION_MAX): raise oefmt(space.w_ImportError, - "cffi extension module '%s' has unknown version %s", - name, hex(version)) + "cffi extension module '%s' uses an unknown version tag %s. " + "This module might need a more recent version of PyPy. " + "The current PyPy provides CFFI %s.", + name, hex(version), _cffi_backend.VERSION) src_ctx = rffi.cast(parse_c_type.PCTX, p[1]) ffi = W_FFIObject(space, src_ctx) @@ -38,7 +41,8 @@ w_name = space.wrap(name) module = Module(space, w_name) - module.setdictvalue(space, '__file__', space.wrap(path)) + if path is not None: + module.setdictvalue(space, '__file__', space.wrap(path)) module.setdictvalue(space, 'ffi', space.wrap(ffi)) module.setdictvalue(space, 'lib', space.wrap(lib)) w_modules_dict = space.sys.get('modules') diff --git a/pypy/module/_cffi_backend/embedding.py b/pypy/module/_cffi_backend/embedding.py new file mode 100644 --- /dev/null +++ b/pypy/module/_cffi_backend/embedding.py @@ -0,0 +1,146 @@ +import os +from rpython.rtyper.lltypesystem import lltype, rffi +from rpython.translator.tool.cbuild import ExternalCompilationInfo + +from pypy.interpreter.error import OperationError, oefmt + +# ____________________________________________________________ + + +EMBED_VERSION_MIN = 0xB011 +EMBED_VERSION_MAX = 0xB0FF + +STDERR = 2 +INITSTRUCTPTR = lltype.Ptr(lltype.Struct('CFFI_INIT', + ('name', rffi.CCHARP), + ('func', rffi.VOIDP), + ('code', rffi.CCHARP))) + +def load_embedded_cffi_module(space, version, init_struct): + from pypy.module._cffi_backend.cffi1_module import load_cffi1_module + declare_c_function() # translation-time hint only: + # declare _cffi_carefully_make_gil() + # + version = rffi.cast(lltype.Signed, version) + if not (EMBED_VERSION_MIN <= version <= EMBED_VERSION_MAX): + raise oefmt(space.w_ImportError, + "cffi embedded module has got unknown version tag %s", + hex(version)) + # + if space.config.objspace.usemodules.thread: + from pypy.module.thread import os_thread + os_thread.setup_threads(space) + # + name = rffi.charp2str(init_struct.name) + load_cffi1_module(space, name, None, init_struct.func) + code = rffi.charp2str(init_struct.code) + compiler = space.createcompiler() + pycode = compiler.compile(code, "" % name, 'exec', 0) + w_globals = space.newdict(module=True) + space.setitem_str(w_globals, "__builtins__", space.wrap(space.builtin)) + pycode.exec_code(space, w_globals, w_globals) + + +class Global: + pass +glob = Global() + +def pypy_init_embedded_cffi_module(version, init_struct): + # called from __init__.py + name = "?" + try: + init_struct = rffi.cast(INITSTRUCTPTR, init_struct) + name = rffi.charp2str(init_struct.name) + # + space = glob.space + must_leave = False + try: + must_leave = space.threadlocals.try_enter_thread(space) + load_embedded_cffi_module(space, version, init_struct) + res = 0 + except OperationError, operr: + operr.write_unraisable(space, "initialization of '%s'" % name, + with_traceback=True) + space.appexec([], r"""(): + import sys + sys.stderr.write('pypy version: %s.%s.%s\n' % + sys.pypy_version_info[:3]) + sys.stderr.write('sys.path: %r\n' % (sys.path,)) + """) + res = -1 + if must_leave: + space.threadlocals.leave_thread(space) + except Exception, e: + # oups! last-level attempt to recover. + try: + os.write(STDERR, "From initialization of '") + os.write(STDERR, name) + os.write(STDERR, "':\n") + os.write(STDERR, str(e)) + os.write(STDERR, "\n") + except: + pass + res = -1 + return rffi.cast(rffi.INT, res) + +# ____________________________________________________________ + + +eci = ExternalCompilationInfo(separate_module_sources=[ +r""" +/* XXX Windows missing */ +#include +#include +#include + +RPY_EXPORTED void rpython_startup_code(void); +RPY_EXPORTED int pypy_setup_home(char *, int); + +static unsigned char _cffi_ready = 0; +static const char *volatile _cffi_module_name; + +static void _cffi_init_error(const char *msg, const char *extra) +{ + fprintf(stderr, + "\nPyPy initialization failure when loading module '%s':\n%s%s\n", + _cffi_module_name, msg, extra); +} + +static void _cffi_init(void) +{ + Dl_info info; + char *home; + + rpython_startup_code(); + RPyGilAllocate(); + + if (dladdr(&_cffi_init, &info) == 0) { + _cffi_init_error("dladdr() failed: ", dlerror()); + return; + } + home = realpath(info.dli_fname, NULL); + if (pypy_setup_home(home, 1) != 0) { + _cffi_init_error("pypy_setup_home() failed", ""); + return; + } + _cffi_ready = 1; +} + +RPY_EXPORTED +int pypy_carefully_make_gil(const char *name) +{ + /* For CFFI: this initializes the GIL and loads the home path. + It can be called completely concurrently from unrelated threads. + It assumes that we don't hold the GIL before (if it exists), and we + don't hold it afterwards. + */ + static pthread_once_t once_control = PTHREAD_ONCE_INIT; + + _cffi_module_name = name; /* not really thread-safe, but better than + nothing */ + pthread_once(&once_control, _cffi_init); + return (int)_cffi_ready - 1; +} +"""]) + +declare_c_function = rffi.llexternal_use_eci(eci) diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.4.2", ("This test_c.py file is for testing a version" +assert __version__ == "1.5.0", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/pypy/module/_continuation/interp_continuation.py b/pypy/module/_continuation/interp_continuation.py --- a/pypy/module/_continuation/interp_continuation.py +++ b/pypy/module/_continuation/interp_continuation.py @@ -195,7 +195,7 @@ class SThread(StackletThread): def __init__(self, space, ec): - StackletThread.__init__(self, space.config) + StackletThread.__init__(self) self.space = space self.ec = ec # for unpickling diff --git a/pypy/module/_warnings/interp_warnings.py b/pypy/module/_warnings/interp_warnings.py --- a/pypy/module/_warnings/interp_warnings.py +++ b/pypy/module/_warnings/interp_warnings.py @@ -75,7 +75,7 @@ frame = ec.getnextframe_nohidden(frame) stacklevel -= 1 if frame: - w_globals = frame.w_globals + w_globals = frame.get_w_globals() lineno = frame.get_last_lineno() else: w_globals = space.sys.w_dict diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -59,7 +59,7 @@ return None return borrow_from(w_dict, w_res) - at cpython_api([PyObject, rffi.CCHARP], rffi.INT_real, error=-1) + at cpython_api([PyObject, CONST_STRING], rffi.INT_real, error=-1) def PyDict_DelItemString(space, w_dict, key_ptr): """Remove the entry in dictionary p which has a key specified by the string key. Return 0 on success or -1 on failure.""" diff --git a/pypy/module/cpyext/eval.py b/pypy/module/cpyext/eval.py --- a/pypy/module/cpyext/eval.py +++ b/pypy/module/cpyext/eval.py @@ -30,7 +30,7 @@ currently executing.""" caller = space.getexecutioncontext().gettopframe_nohidden() if caller is not None: - w_globals = caller.w_globals + w_globals = caller.get_w_globals() w_builtins = space.getitem(w_globals, space.wrap('__builtins__')) if not space.isinstance_w(w_builtins, space.w_dict): w_builtins = w_builtins.getdict(space) @@ -54,7 +54,7 @@ caller = space.getexecutioncontext().gettopframe_nohidden() if caller is None: return None - return borrow_from(None, caller.w_globals) + return borrow_from(None, caller.get_w_globals()) @cpython_api([PyCodeObject, PyObject, PyObject], PyObject) def PyEval_EvalCode(space, w_code, w_globals, w_locals): @@ -128,7 +128,7 @@ filename = "" return run_string(space, source, filename, start, w_globals, w_locals) - at cpython_api([rffi.CCHARP, rffi.INT_real, PyObject, PyObject, + at cpython_api([CONST_STRING, rffi.INT_real, PyObject, PyObject, PyCompilerFlagsPtr], PyObject) def PyRun_StringFlags(space, source, start, w_globals, w_locals, flagsptr): """Execute Python source code from str in the context specified by the @@ -189,7 +189,7 @@ pi[0] = space.getindex_w(w_obj, None) return 1 - at cpython_api([rffi.CCHARP, rffi.CCHARP, rffi.INT_real, PyCompilerFlagsPtr], + at cpython_api([CONST_STRING, CONST_STRING, rffi.INT_real, PyCompilerFlagsPtr], PyObject) def Py_CompileStringFlags(space, source, filename, start, flagsptr): """Parse and compile the Python source code in str, returning the diff --git a/pypy/module/cpyext/frameobject.py b/pypy/module/cpyext/frameobject.py --- a/pypy/module/cpyext/frameobject.py +++ b/pypy/module/cpyext/frameobject.py @@ -34,7 +34,7 @@ frame = space.interp_w(PyFrame, w_obj) py_frame = rffi.cast(PyFrameObject, py_obj) py_frame.c_f_code = rffi.cast(PyCodeObject, make_ref(space, frame.pycode)) - py_frame.c_f_globals = make_ref(space, frame.w_globals) + py_frame.c_f_globals = make_ref(space, frame.get_w_globals()) rffi.setintfield(py_frame, 'c_f_lineno', frame.getorcreatedebug().f_lineno) @cpython_api([PyObject], lltype.Void, external=False) diff --git a/pypy/module/cpyext/import_.py b/pypy/module/cpyext/import_.py --- a/pypy/module/cpyext/import_.py +++ b/pypy/module/cpyext/import_.py @@ -20,7 +20,7 @@ caller = space.getexecutioncontext().gettopframe_nohidden() # Get the builtins from current globals if caller is not None: - w_globals = caller.w_globals + w_globals = caller.get_w_globals() w_builtin = space.getitem(w_globals, space.wrap('__builtins__')) else: # No globals -- use standard builtins, and fake globals diff --git a/pypy/module/cpyext/longobject.py b/pypy/module/cpyext/longobject.py --- a/pypy/module/cpyext/longobject.py +++ b/pypy/module/cpyext/longobject.py @@ -228,26 +228,11 @@ def _PyLong_FromByteArray(space, bytes, n, little_endian, signed): little_endian = rffi.cast(lltype.Signed, little_endian) signed = rffi.cast(lltype.Signed, signed) - - result = rbigint() - negative = False - - for i in range(0, n): - if little_endian: - c = intmask(bytes[i]) - else: - c = intmask(bytes[n - i - 1]) - if i == 0 and signed and c & 0x80: - negative = True - if negative: - c = c ^ 0xFF - digit = rbigint.fromint(c) - - result = result.lshift(8) - result = result.add(digit) - - if negative: - result = result.neg() - + s = rffi.charpsize2str(rffi.cast(rffi.CCHARP, bytes), + rffi.cast(lltype.Signed, n)) + if little_endian: + byteorder = 'little' + else: + byteorder = 'big' + result = rbigint.frombytes(s, byteorder, signed != 0) return space.newlong_from_rbigint(result) - diff --git a/pypy/module/cpyext/Doc_stubgen_enable.patch b/pypy/module/cpyext/patches/Doc_stubgen_enable.patch rename from pypy/module/cpyext/Doc_stubgen_enable.patch rename to pypy/module/cpyext/patches/Doc_stubgen_enable.patch diff --git a/pypy/module/cpyext/pystrtod.py b/pypy/module/cpyext/pystrtod.py --- a/pypy/module/cpyext/pystrtod.py +++ b/pypy/module/cpyext/pystrtod.py @@ -1,6 +1,7 @@ import errno from pypy.interpreter.error import OperationError -from pypy.module.cpyext.api import cpython_api, PyObject, CONST_STRING +from pypy.module.cpyext.api import cpython_api, CONST_STRING +from pypy.module.cpyext.pyobject import PyObject from rpython.rlib import rdtoa from rpython.rlib import rfloat from rpython.rlib import rposix, jit diff --git a/pypy/module/cpyext/test/test_longobject.py b/pypy/module/cpyext/test/test_longobject.py --- a/pypy/module/cpyext/test/test_longobject.py +++ b/pypy/module/cpyext/test/test_longobject.py @@ -175,10 +175,26 @@ little_endian, is_signed); """), ]) - assert module.from_bytearray(True, False) == 0x9ABC - assert module.from_bytearray(True, True) == -0x6543 - assert module.from_bytearray(False, False) == 0xBC9A - assert module.from_bytearray(False, True) == -0x4365 + assert module.from_bytearray(True, False) == 0xBC9A + assert module.from_bytearray(True, True) == -0x4366 + assert module.from_bytearray(False, False) == 0x9ABC From pypy.commits at gmail.com Mon Jan 25 17:12:58 2016 From: pypy.commits at gmail.com (Pierre-Yves David) Date: Mon, 25 Jan 2016 14:12:58 -0800 (PST) Subject: [pypy-commit] extradoc extradoc: Registering for the Winter Sprint Message-ID: <56a69dea.53ad1c0a.ff3b1.3e9f@mx.google.com> Author: Pierre-Yves David Branch: extradoc Changeset: r5596:4f73dcd80597 Date: 2016-01-22 09:47 -0800 http://bitbucket.org/pypy/extradoc/changeset/4f73dcd80597/ Log: Registering for the Winter Sprint I'm a Mercurial developer, interested in making progress on having pypy + Mercurial moving toward production readyness and discussing your usage of Mercurial as a project. diff --git a/sprintinfo/leysin-winter-2016/announcement.txt b/sprintinfo/leysin-winter-2016/announcement.txt --- a/sprintinfo/leysin-winter-2016/announcement.txt +++ b/sprintinfo/leysin-winter-2016/announcement.txt @@ -30,6 +30,10 @@ the conflict reporting tools, and more generally to figure out how practical it is in large projects to avoid conflicts +* Running Mercurial on Pypy + +* Workflow of the Pypy Project with Mercurial + * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. diff --git a/sprintinfo/leysin-winter-2016/people.txt b/sprintinfo/leysin-winter-2016/people.txt --- a/sprintinfo/leysin-winter-2016/people.txt +++ b/sprintinfo/leysin-winter-2016/people.txt @@ -18,6 +18,7 @@ Richard Plangger 20-28 Ermina Maciej Fijalkowski 20-? Ermina (big room preferred) Ronan Lamy 20-27 Ermina? +Pierre-Yves David 20-27 Ermina ==================== ============== ======================= **NOTE:** we might have only a single double-bed room and a big room From pypy.commits at gmail.com Mon Jan 25 21:00:31 2016 From: pypy.commits at gmail.com (mjacob) Date: Mon, 25 Jan 2016 18:00:31 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Close this head. Message-ID: <56a6d33f.d4811c0a.40e2e.2c58@mx.google.com> Author: Manuel Jacob Branch: py3.3 Changeset: r81941:8abc47287812 Date: 2016-01-26 02:51 +0100 http://bitbucket.org/pypy/pypy/changeset/8abc47287812/ Log: Close this head. Something went wrong with the last merge from py3k. I'm trying to fix this by redoing the merge and all subsequent commits. From pypy.commits at gmail.com Mon Jan 25 21:00:29 2016 From: pypy.commits at gmail.com (mjacob) Date: Mon, 25 Jan 2016 18:00:29 -0800 (PST) Subject: [pypy-commit] pypy llvm-translation-backend: Add a hack to make functions linked in from C code inlinable. Message-ID: <56a6d33d.82e11c0a.12767.6c5a@mx.google.com> Author: Manuel Jacob Branch: llvm-translation-backend Changeset: r81940:697b7fe15e58 Date: 2016-01-16 01:34 +0100 http://bitbucket.org/pypy/pypy/changeset/697b7fe15e58/ Log: Add a hack to make functions linked in from C code inlinable. To make inlining these functions possible, their callers must (simplified) have the same target specified. This commits adds a hack to get the necessary attributes from clang. diff --git a/rpython/translator/llvm/genllvm.py b/rpython/translator/llvm/genllvm.py --- a/rpython/translator/llvm/genllvm.py +++ b/rpython/translator/llvm/genllvm.py @@ -942,13 +942,14 @@ def write_graph(self, ptr_type, name, graph, export): prevent_inline, llvmgcroot = self.prepare_graph(ptr_type, name, graph) - self.w('define {linkage}{retvar.T} {name}({a}){add}{gc} {{'.format( + self.w('define {linkage}{retvar.T} {name}({a}) {attrs}{gc} {{'.format( linkage='' if export else 'internal ', retvar=get_repr(graph.getreturnvar()), name=name, a=', '.join(get_repr(arg).TV for arg in graph.getargs() if arg.concretetype is not lltype.Void), - add=' noinline' if prevent_inline else '', + attrs=database.target_attributes + + (' noinline' if prevent_inline else ''), gc=' gc "pypy"' if llvmgcroot else ''), '') @@ -1842,6 +1843,32 @@ self.gcpolicy.finish() + def _get_target_information(self): + """ + Get datalayout, triple, and "target-cpu" and "target-features" + attributes for a generic version of the CPU we are compiling on. + + This passes a small C code snippet to clang and parses the output in a + way that could easily break with any future LLVM version. + """ + output = cmdexec( + 'echo "void test() {{}}" | clang -x c - -emit-llvm -S -o -' + .format(devnull)) + for line in output.splitlines(True): + if line.startswith('target '): + yield line + if line.startswith('attributes'): + assert line.startswith('attributes #0 = { ') + assert line.endswith(' }\n') + attributes_str = line[len('attributes #0 = { '):-len(' }\n')] + for attribute in attributes_str.split(): + if attribute.startswith('"target-cpu"='): + target_cpu_attribute = attribute + if attribute.startswith('"target-features"='): + target_features_attribute = attribute + database.target_attributes = '{} {}'.format( + target_cpu_attribute, target_features_attribute) + def _write_special_declarations(self, f): compiler_info_str = "LLVM " + cmdexec('llvm-config --version').strip() cstr_type = '[{} x i8]'.format(len(compiler_info_str) + 1) @@ -1877,13 +1904,10 @@ self.work_dir.mkdir() self.main_ll_file = self.work_dir.join('main.ll') with self.main_ll_file.open('w') as f: - output = cmdexec('clang -emit-llvm -S -x c {} -o -' - .format(devnull)) - for line in output.splitlines(True): - if line.startswith('target '): - f.write(line) + database = Database(self, f) - database = Database(self, f) + for line in self._get_target_information(): + f.write(line) from rpython.translator.c.database import LowLevelDatabase from rpython.translator.c.genc import gen_threadlocal_structdef From pypy.commits at gmail.com Mon Jan 25 21:00:33 2016 From: pypy.commits at gmail.com (mjacob) Date: Mon, 25 Jan 2016 18:00:33 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Remove trailing whitespace. Message-ID: <56a6d341.d7bc1c0a.c68bd.2a91@mx.google.com> Author: Manuel Jacob Branch: py3.3 Changeset: r81942:9b1940890dc0 Date: 2016-01-26 02:59 +0100 http://bitbucket.org/pypy/pypy/changeset/9b1940890dc0/ Log: Remove trailing whitespace. diff --git a/pypy/module/zlib/test/test_zlib.py b/pypy/module/zlib/test/test_zlib.py --- a/pypy/module/zlib/test/test_zlib.py +++ b/pypy/module/zlib/test/test_zlib.py @@ -183,7 +183,7 @@ assert dco.eof == True dco.flush() assert dco.eof == True - + def test_decompress_eof_incomplete_stream(self): import zlib x = b'x\x9cK\xcb\xcf\x07\x00\x02\x82\x01E' # 'foo' From pypy.commits at gmail.com Tue Jan 26 02:42:58 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 23:42:58 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: removed longlong from s390x from cpu Message-ID: <56a72382.2968c20a.c229f.08b3@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81943:47d8fe0b9435 Date: 2016-01-26 08:42 +0100 http://bitbucket.org/pypy/pypy/changeset/47d8fe0b9435/ Log: removed longlong from s390x from cpu diff --git a/rpython/jit/backend/detect_cpu.py b/rpython/jit/backend/detect_cpu.py --- a/rpython/jit/backend/detect_cpu.py +++ b/rpython/jit/backend/detect_cpu.py @@ -146,7 +146,7 @@ MODEL_X86_64_SSE4: ['floats', 'singlefloats'], MODEL_ARM: ['floats', 'singlefloats', 'longlong'], MODEL_PPC_64: [], # we don't even have PPC directory, so no - MODEL_S390_64: ['floats', 'longlong'], + MODEL_S390_64: ['floats'], }[backend_name] if __name__ == '__main__': From pypy.commits at gmail.com Tue Jan 26 02:47:54 2016 From: pypy.commits at gmail.com (plan_rich) Date: Mon, 25 Jan 2016 23:47:54 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: merged default Message-ID: <56a724aa.c6e01c0a.df7b1.ffffd0b9@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81944:c6029ee6abcc Date: 2016-01-26 08:47 +0100 http://bitbucket.org/pypy/pypy/changeset/c6029ee6abcc/ Log: merged default diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -124,6 +124,15 @@ Fix SSL tests by importing cpython's patch +.. branch: remove-getfield-pure + +Remove pure variants of ``getfield_gc_*`` operations from the JIT. Relevant +optimizations instead consult the field descriptor to determine the purity of +the operation. Additionally, pure ``getfield`` operations are now handled +entirely by `rpython/jit/metainterp/optimizeopt/heap.py` rather than +`rpython/jit/metainterp/optimizeopt/pure.py`, which can result in better codegen +for traces containing a large number of pure getfield operations. + .. branch: memop-simplify3 Further simplifying the backend operations malloc_cond_varsize and zero_array. diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -59,7 +59,7 @@ return None return borrow_from(w_dict, w_res) - at cpython_api([PyObject, rffi.CCHARP], rffi.INT_real, error=-1) + at cpython_api([PyObject, CONST_STRING], rffi.INT_real, error=-1) def PyDict_DelItemString(space, w_dict, key_ptr): """Remove the entry in dictionary p which has a key specified by the string key. Return 0 on success or -1 on failure.""" diff --git a/pypy/module/cpyext/eval.py b/pypy/module/cpyext/eval.py --- a/pypy/module/cpyext/eval.py +++ b/pypy/module/cpyext/eval.py @@ -128,7 +128,7 @@ filename = "" return run_string(space, source, filename, start, w_globals, w_locals) - at cpython_api([rffi.CCHARP, rffi.INT_real, PyObject, PyObject, + at cpython_api([CONST_STRING, rffi.INT_real, PyObject, PyObject, PyCompilerFlagsPtr], PyObject) def PyRun_StringFlags(space, source, start, w_globals, w_locals, flagsptr): """Execute Python source code from str in the context specified by the @@ -189,7 +189,7 @@ pi[0] = space.getindex_w(w_obj, None) return 1 - at cpython_api([rffi.CCHARP, rffi.CCHARP, rffi.INT_real, PyCompilerFlagsPtr], + at cpython_api([CONST_STRING, CONST_STRING, rffi.INT_real, PyCompilerFlagsPtr], PyObject) def Py_CompileStringFlags(space, source, filename, start, flagsptr): """Parse and compile the Python source code in str, returning the diff --git a/pypy/module/cpyext/pystrtod.py b/pypy/module/cpyext/pystrtod.py --- a/pypy/module/cpyext/pystrtod.py +++ b/pypy/module/cpyext/pystrtod.py @@ -1,6 +1,6 @@ import errno from pypy.interpreter.error import OperationError -from pypy.module.cpyext.api import cpython_api +from pypy.module.cpyext.api import cpython_api, CONST_STRING from pypy.module.cpyext.pyobject import PyObject from rpython.rlib import rdtoa from rpython.rlib import rfloat @@ -22,7 +22,7 @@ rfloat.DIST_NAN: Py_DTST_NAN } - at cpython_api([rffi.CCHARP, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) + at cpython_api([CONST_STRING, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) @jit.dont_look_inside # direct use of _get_errno() def PyOS_string_to_double(space, s, endptr, w_overflow_exception): """Convert a string s to a double, raising a Python diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -83,9 +83,9 @@ p38 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p39 = getfield_gc_r(p38, descr=) i40 = force_token() - p41 = getfield_gc_pure_r(p38, descr=) + p41 = getfield_gc_r(p38, descr=) guard_value(p41, ConstPtr(ptr42), descr=...) - i42 = getfield_gc_pure_i(p38, descr=) + i42 = getfield_gc_i(p38, descr=) i43 = int_is_zero(i42) guard_true(i43, descr=...) i50 = force_token() @@ -435,21 +435,21 @@ guard_isnull(p5, descr=...) guard_nonnull_class(p12, ConstClass(W_IntObject), descr=...) guard_value(p2, ConstPtr(ptr21), descr=...) - i22 = getfield_gc_pure_i(p12, descr=) + i22 = getfield_gc_i(p12, descr=) i24 = int_lt(i22, 5000) guard_true(i24, descr=...) guard_not_invalidated(descr=...) p29 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p30 = getfield_gc_r(p29, descr=) p31 = force_token() - p32 = getfield_gc_pure_r(p29, descr=) + p32 = getfield_gc_r(p29, descr=) guard_value(p32, ConstPtr(ptr33), descr=...) - i34 = getfield_gc_pure_i(p29, descr=) + i34 = getfield_gc_i(p29, descr=) i35 = int_is_zero(i34) guard_true(i35, descr=...) p37 = getfield_gc_r(ConstPtr(ptr36), descr=) guard_nonnull_class(p37, ConstClass(W_IntObject), descr=...) - i39 = getfield_gc_pure_i(p37, descr=) + i39 = getfield_gc_i(p37, descr=) i40 = int_add_ovf(i22, i39) guard_no_overflow(descr=...) --TICK-- @@ -466,7 +466,7 @@ """, []) loop, = log.loops_by_id('call') assert loop.match(""" - i8 = getfield_gc_pure_i(p6, descr=) + i8 = getfield_gc_i(p6, descr=) i10 = int_lt(i8, 5000) guard_true(i10, descr=...) guard_not_invalidated? diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py --- a/pypy/module/pypyjit/test_pypy_c/test_containers.py +++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py @@ -84,7 +84,7 @@ guard_no_exception(descr=...) p20 = new_with_vtable(descr=...) call_n(ConstClass(_ll_dict_setitem_lookup_done_trampoline), p13, p10, p20, i12, i17, descr=) - setfield_gc(p20, i5, descr=) + setfield_gc(p20, i5, descr=) guard_no_exception(descr=...) i23 = call_i(ConstClass(ll_call_lookup_function), p13, p10, i12, 0, descr=) guard_no_exception(descr=...) @@ -93,7 +93,7 @@ p28 = getfield_gc_r(p13, descr=) p29 = getinteriorfield_gc_r(p28, i23, descr=>) guard_nonnull_class(p29, ConstClass(W_IntObject), descr=...) - i31 = getfield_gc_pure_i(p29, descr=) + i31 = getfield_gc_i(p29, descr=) i32 = int_sub_ovf(i31, i5) guard_no_overflow(descr=...) i34 = int_add_ovf(i32, 1) diff --git a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py --- a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py +++ b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py @@ -101,13 +101,13 @@ loop = log._filter(log.loops[0]) assert loop.match(""" guard_class(p1, #, descr=...) - p4 = getfield_gc_pure_r(p1, descr=) + p4 = getfield_gc_r(p1, descr=) i5 = getfield_gc_i(p0, descr=) - p6 = getfield_gc_pure_r(p4, descr=) - p7 = getfield_gc_pure_r(p6, descr=) + p6 = getfield_gc_r(p4, descr=) + p7 = getfield_gc_r(p6, descr=) guard_class(p7, ConstClass(Float64), descr=...) - i9 = getfield_gc_pure_i(p4, descr=) - i10 = getfield_gc_pure_i(p6, descr=) + i9 = getfield_gc_i(p4, descr=) + i10 = getfield_gc_i(p6, descr=) i12 = int_eq(i10, 61) i14 = int_eq(i10, 60) i15 = int_or(i12, i14) @@ -117,28 +117,28 @@ i18 = float_ne(f16, 0.000000) guard_true(i18, descr=...) guard_nonnull_class(p2, ConstClass(W_BoolBox), descr=...) - i20 = getfield_gc_pure_i(p2, descr=) + i20 = getfield_gc_i(p2, descr=) i21 = int_is_true(i20) guard_false(i21, descr=...) i22 = getfield_gc_i(p0, descr=) - i23 = getfield_gc_pure_i(p1, descr=) + i23 = getfield_gc_i(p1, descr=) guard_true(i23, descr=...) i25 = int_add(i22, 1) - p26 = getfield_gc_pure_r(p0, descr=) - i27 = getfield_gc_pure_i(p1, descr=) + p26 = getfield_gc_r(p0, descr=) + i27 = getfield_gc_i(p1, descr=) i28 = int_is_true(i27) guard_true(i28, descr=...) - i29 = getfield_gc_pure_i(p6, descr=) + i29 = getfield_gc_i(p6, descr=) guard_value(i29, 8, descr=...) i30 = int_add(i5, 8) - i31 = getfield_gc_pure_i(p1, descr=) + i31 = getfield_gc_i(p1, descr=) i32 = int_ge(i25, i31) guard_false(i32, descr=...) p34 = new_with_vtable(descr=...) {{{ - setfield_gc(p34, p1, descr=) + setfield_gc(p34, p1, descr=) setfield_gc(p34, i25, descr=) - setfield_gc(p34, p26, descr=) + setfield_gc(p34, p26, descr=) setfield_gc(p34, i30, descr=) }}} jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_min_max.py b/pypy/module/pypyjit/test_pypy_c/test_min_max.py --- a/pypy/module/pypyjit/test_pypy_c/test_min_max.py +++ b/pypy/module/pypyjit/test_pypy_c/test_min_max.py @@ -54,7 +54,7 @@ i19 = int_add(i11, 1) setfield_gc(p2, i19, descr=...) guard_nonnull_class(p18, ConstClass(W_IntObject), descr=...) - i20 = getfield_gc_pure_i(p18, descr=...) + i20 = getfield_gc_i(p18, descr=...) i21 = int_gt(i20, i14) guard_true(i21, descr=...) jump(..., descr=...) diff --git a/pypy/module/pypyjit/test_pypy_c/test_misc.py b/pypy/module/pypyjit/test_pypy_c/test_misc.py --- a/pypy/module/pypyjit/test_pypy_c/test_misc.py +++ b/pypy/module/pypyjit/test_pypy_c/test_misc.py @@ -113,7 +113,7 @@ i12 = int_is_true(i4) guard_true(i12, descr=...) guard_not_invalidated(descr=...) - i10p = getfield_gc_pure_i(p10, descr=...) + i10p = getfield_gc_i(p10, descr=...) i10 = int_mul_ovf(2, i10p) guard_no_overflow(descr=...) i14 = int_add_ovf(i13, i10) diff --git a/pypy/module/pypyjit/test_pypy_c/test_string.py b/pypy/module/pypyjit/test_pypy_c/test_string.py --- a/pypy/module/pypyjit/test_pypy_c/test_string.py +++ b/pypy/module/pypyjit/test_pypy_c/test_string.py @@ -82,7 +82,7 @@ strsetitem(p25, 0, i23) p93 = call_r(ConstClass(fromstr), p25, 16, descr=) guard_no_exception(descr=...) - i95 = getfield_gc_pure_i(p93, descr=) + i95 = getfield_gc_i(p93, descr=) i96 = int_gt(i95, #) guard_false(i96, descr=...) i94 = call_i(ConstClass(rbigint._toint_helper), p93, descr=) diff --git a/rpython/jit/backend/llgraph/runner.py b/rpython/jit/backend/llgraph/runner.py --- a/rpython/jit/backend/llgraph/runner.py +++ b/rpython/jit/backend/llgraph/runner.py @@ -152,7 +152,7 @@ self.fieldname = fieldname self.FIELD = getattr(S, fieldname) self.index = heaptracker.get_fielddescr_index_in(S, fieldname) - self._is_pure = S._immutable_field(fieldname) + self._is_pure = S._immutable_field(fieldname) != False def is_always_pure(self): return self._is_pure @@ -608,9 +608,6 @@ p = support.cast_arg(lltype.Ptr(descr.S), p) return support.cast_result(descr.FIELD, getattr(p, descr.fieldname)) - bh_getfield_gc_pure_i = bh_getfield_gc - bh_getfield_gc_pure_r = bh_getfield_gc - bh_getfield_gc_pure_f = bh_getfield_gc bh_getfield_gc_i = bh_getfield_gc bh_getfield_gc_r = bh_getfield_gc bh_getfield_gc_f = bh_getfield_gc diff --git a/rpython/jit/backend/llsupport/descr.py b/rpython/jit/backend/llsupport/descr.py --- a/rpython/jit/backend/llsupport/descr.py +++ b/rpython/jit/backend/llsupport/descr.py @@ -180,7 +180,8 @@ return self.offset def repr_of_descr(self): - return '' % (self.flag, self.name, self.offset) + ispure = " pure" if self._is_pure else "" + return '' % (self.flag, self.name, self.offset, ispure) def get_parent_descr(self): return self.parent_descr @@ -200,7 +201,7 @@ flag = get_type_flag(FIELDTYPE) name = '%s.%s' % (STRUCT._name, fieldname) index_in_parent = heaptracker.get_fielddescr_index_in(STRUCT, fieldname) - is_pure = bool(STRUCT._immutable_field(fieldname)) + is_pure = STRUCT._immutable_field(fieldname) != False fielddescr = FieldDescr(name, offset, size, flag, index_in_parent, is_pure) cachedict = cache.setdefault(STRUCT, {}) diff --git a/rpython/jit/backend/llsupport/rewrite.py b/rpython/jit/backend/llsupport/rewrite.py --- a/rpython/jit/backend/llsupport/rewrite.py +++ b/rpython/jit/backend/llsupport/rewrite.py @@ -243,7 +243,6 @@ self.emit_gc_store_or_indexed(op, ptr_box, index_box, value_box, fieldsize, itemsize, ofs) elif opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R, - rop.GETFIELD_GC_PURE_I, rop.GETFIELD_GC_PURE_F, rop.GETFIELD_GC_PURE_R, rop.GETFIELD_RAW_I, rop.GETFIELD_RAW_F, rop.GETFIELD_RAW_R): ofs, itemsize, sign = unpack_fielddescr(op.getdescr()) ptr_box = op.getarg(0) @@ -497,8 +496,8 @@ elif arraydescr.itemsize == 0: total_size = arraydescr.basesize elif (self.gc_ll_descr.can_use_nursery_malloc(1) and - self.gen_malloc_nursery_varsize(arraydescr.itemsize, v_length, - op, arraydescr, kind=kind)): + self.gen_malloc_nursery_varsize(arraydescr.itemsize, + v_length, op, arraydescr, kind=kind)): # note that we cannot initialize tid here, because the array # might end up being allocated by malloc_external or some # stuff that initializes GC header fields differently @@ -534,8 +533,6 @@ # See emit_pending_zeros(). (This optimization is done by # hacking the object 'o' in-place: e.g., o.getarg(1) may be # replaced with another constant greater than 0.) - #o = ResOperation(rop.ZERO_ARRAY, [v_arr, self.c_zero, v_length], - # descr=arraydescr) assert isinstance(arraydescr, ArrayDescr) scale = arraydescr.itemsize v_length_scaled = v_length diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py --- a/rpython/jit/backend/x86/assembler.py +++ b/rpython/jit/backend/x86/assembler.py @@ -1477,9 +1477,6 @@ genop_getfield_gc_f = _genop_getfield genop_getfield_raw_i = _genop_getfield genop_getfield_raw_f = _genop_getfield - genop_getfield_gc_pure_i = _genop_getfield - genop_getfield_gc_pure_r = _genop_getfield - genop_getfield_gc_pure_f = _genop_getfield def _genop_gc_load(self, op, arglocs, resloc): base_loc, ofs_loc, size_loc, sign_loc = arglocs diff --git a/rpython/jit/metainterp/heapcache.py b/rpython/jit/metainterp/heapcache.py --- a/rpython/jit/metainterp/heapcache.py +++ b/rpython/jit/metainterp/heapcache.py @@ -168,9 +168,6 @@ elif (opnum != rop.GETFIELD_GC_R and opnum != rop.GETFIELD_GC_I and opnum != rop.GETFIELD_GC_F and - opnum != rop.GETFIELD_GC_PURE_R and - opnum != rop.GETFIELD_GC_PURE_I and - opnum != rop.GETFIELD_GC_PURE_F and opnum != rop.PTR_EQ and opnum != rop.PTR_NE and opnum != rop.INSTANCE_PTR_EQ and diff --git a/rpython/jit/metainterp/history.py b/rpython/jit/metainterp/history.py --- a/rpython/jit/metainterp/history.py +++ b/rpython/jit/metainterp/history.py @@ -816,9 +816,6 @@ if 'getfield_gc' in check: assert check.pop('getfield_gc') == 0 check['getfield_gc_i'] = check['getfield_gc_r'] = check['getfield_gc_f'] = 0 - if 'getfield_gc_pure' in check: - assert check.pop('getfield_gc_pure') == 0 - check['getfield_gc_pure_i'] = check['getfield_gc_pure_r'] = check['getfield_gc_pure_f'] = 0 if 'getarrayitem_gc_pure' in check: assert check.pop('getarrayitem_gc_pure') == 0 check['getarrayitem_gc_pure_i'] = check['getarrayitem_gc_pure_r'] = check['getarrayitem_gc_pure_f'] = 0 diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py --- a/rpython/jit/metainterp/optimizeopt/heap.py +++ b/rpython/jit/metainterp/optimizeopt/heap.py @@ -183,6 +183,8 @@ return res def invalidate(self, descr): + if descr.is_always_pure(): + return for opinfo in self.cached_infos: assert isinstance(opinfo, info.AbstractStructPtrInfo) opinfo._fields[descr.get_index()] = None @@ -515,9 +517,14 @@ return pendingfields def optimize_GETFIELD_GC_I(self, op): + descr = op.getdescr() + if descr.is_always_pure() and self.get_constant_box(op.getarg(0)) is not None: + resbox = self.optimizer.constant_fold(op) + self.optimizer.make_constant(op, resbox) + return structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) + cf = self.field_cache(descr) + field = cf.getfield_from_cache(self, structinfo, descr) if field is not None: self.make_equal_to(op, field) return @@ -525,23 +532,10 @@ self.make_nonnull(op.getarg(0)) self.emit_operation(op) # then remember the result of reading the field - structinfo.setfield(op.getdescr(), op.getarg(0), op, optheap=self, cf=cf) + structinfo.setfield(descr, op.getarg(0), op, optheap=self, cf=cf) optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - def optimize_GETFIELD_GC_PURE_I(self, op): - structinfo = self.ensure_ptr_info_arg0(op) - cf = self.field_cache(op.getdescr()) - field = cf.getfield_from_cache(self, structinfo, op.getdescr()) - if field is not None: - self.make_equal_to(op, field) - return - # default case: produce the operation - self.make_nonnull(op.getarg(0)) - self.emit_operation(op) - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_PURE_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_PURE_I - def optimize_SETFIELD_GC(self, op): self.setfield(op) #opnum = OpHelpers.getfield_pure_for_descr(op.getdescr()) @@ -631,12 +625,12 @@ def optimize_QUASIIMMUT_FIELD(self, op): # Pattern: QUASIIMMUT_FIELD(s, descr=QuasiImmutDescr) - # x = GETFIELD_GC_PURE(s, descr='inst_x') + # x = GETFIELD_GC(s, descr='inst_x') # pure # If 's' is a constant (after optimizations) we rely on the rest of the - # optimizations to constant-fold the following getfield_gc_pure. + # optimizations to constant-fold the following pure getfield_gc. # in addition, we record the dependency here to make invalidation work # correctly. - # NB: emitting the GETFIELD_GC_PURE is only safe because the + # NB: emitting the pure GETFIELD_GC is only safe because the # QUASIIMMUT_FIELD is also emitted to make sure the dependency is # registered. structvalue = self.ensure_ptr_info_arg0(op) diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py b/rpython/jit/metainterp/optimizeopt/optimizer.py --- a/rpython/jit/metainterp/optimizeopt/optimizer.py +++ b/rpython/jit/metainterp/optimizeopt/optimizer.py @@ -10,6 +10,7 @@ from rpython.jit.metainterp.typesystem import llhelper from rpython.rlib.objectmodel import specialize, we_are_translated from rpython.rlib.debug import debug_print +from rpython.jit.metainterp.optimize import SpeculativeError @@ -374,6 +375,7 @@ if (box.type == 'i' and box.get_forwarded() and box.get_forwarded().is_constant()): return ConstInt(box.get_forwarded().getint()) + return None #self.ensure_imported(value) def get_newoperations(self): @@ -736,12 +738,64 @@ self.emit_operation(op) def constant_fold(self, op): + self.protect_speculative_operation(op) argboxes = [self.get_constant_box(op.getarg(i)) for i in range(op.numargs())] return execute_nonspec_const(self.cpu, None, op.getopnum(), argboxes, op.getdescr(), op.type) + def protect_speculative_operation(self, op): + """When constant-folding a pure operation that reads memory from + a gcref, make sure that the gcref is non-null and of a valid type. + Otherwise, raise SpeculativeError. This should only occur when + unrolling and optimizing the unrolled loop. Note that if + cpu.supports_guard_gc_type is false, we can't really do this + check at all, but then we don't unroll in that case. + """ + opnum = op.getopnum() + cpu = self.cpu + + if OpHelpers.is_pure_getfield(opnum, op.getdescr()): + fielddescr = op.getdescr() + ref = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_field(ref, fielddescr) + return + + elif (opnum == rop.GETARRAYITEM_GC_PURE_I or + opnum == rop.GETARRAYITEM_GC_PURE_R or + opnum == rop.GETARRAYITEM_GC_PURE_F or + opnum == rop.ARRAYLEN_GC): + arraydescr = op.getdescr() + array = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_array(array, arraydescr) + if opnum == rop.ARRAYLEN_GC: + return + arraylength = cpu.bh_arraylen_gc(array, arraydescr) + + elif (opnum == rop.STRGETITEM or + opnum == rop.STRLEN): + string = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_string(string) + if opnum == rop.STRLEN: + return + arraylength = cpu.bh_strlen(string) + + elif (opnum == rop.UNICODEGETITEM or + opnum == rop.UNICODELEN): + unicode = self.get_constant_box(op.getarg(0)).getref_base() + cpu.protect_speculative_unicode(unicode) + if opnum == rop.UNICODELEN: + return + arraylength = cpu.bh_unicodelen(unicode) + + else: + return + + index = self.get_constant_box(op.getarg(1)).getint() + if not (0 <= index < arraylength): + raise SpeculativeError + def is_virtual(self, op): if op.type == 'r': opinfo = self.getptrinfo(op) diff --git a/rpython/jit/metainterp/optimizeopt/pure.py b/rpython/jit/metainterp/optimizeopt/pure.py --- a/rpython/jit/metainterp/optimizeopt/pure.py +++ b/rpython/jit/metainterp/optimizeopt/pure.py @@ -94,7 +94,6 @@ break else: # all constant arguments: constant-fold away - self.protect_speculative_operation(op) resbox = self.optimizer.constant_fold(op) # note that INT_xxx_OVF is not done from here, and the # overflows in the INT_xxx operations are ignored @@ -119,59 +118,6 @@ if nextop: self.emit_operation(nextop) - def protect_speculative_operation(self, op): - """When constant-folding a pure operation that reads memory from - a gcref, make sure that the gcref is non-null and of a valid type. - Otherwise, raise SpeculativeError. This should only occur when - unrolling and optimizing the unrolled loop. Note that if - cpu.supports_guard_gc_type is false, we can't really do this - check at all, but then we don't unroll in that case. - """ - opnum = op.getopnum() - cpu = self.optimizer.cpu - - if (opnum == rop.GETFIELD_GC_PURE_I or - opnum == rop.GETFIELD_GC_PURE_R or - opnum == rop.GETFIELD_GC_PURE_F): - fielddescr = op.getdescr() - ref = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_field(ref, fielddescr) - return - - elif (opnum == rop.GETARRAYITEM_GC_PURE_I or - opnum == rop.GETARRAYITEM_GC_PURE_R or - opnum == rop.GETARRAYITEM_GC_PURE_F or - opnum == rop.ARRAYLEN_GC): - arraydescr = op.getdescr() - array = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_array(array, arraydescr) - if opnum == rop.ARRAYLEN_GC: - return - arraylength = cpu.bh_arraylen_gc(array, arraydescr) - - elif (opnum == rop.STRGETITEM or - opnum == rop.STRLEN): - string = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_string(string) - if opnum == rop.STRLEN: - return - arraylength = cpu.bh_strlen(string) - - elif (opnum == rop.UNICODEGETITEM or - opnum == rop.UNICODELEN): - unicode = self.get_constant_box(op.getarg(0)).getref_base() - cpu.protect_speculative_unicode(unicode) - if opnum == rop.UNICODELEN: - return - arraylength = cpu.bh_unicodelen(unicode) - - else: - return - - index = self.get_constant_box(op.getarg(1)).getint() - if not (0 <= index < arraylength): - raise SpeculativeError - def getrecentops(self, opnum): if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: opnum = opnum - rop._OVF_FIRST diff --git a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py --- a/rpython/jit/metainterp/optimizeopt/test/test_dependency.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_dependency.py @@ -521,8 +521,8 @@ def test_getfield(self): graph = self.build_dependency(""" [p0, p1] # 0: 1,2,5 - p2 = getfield_gc_r(p0) # 1: 3,5 - p3 = getfield_gc_r(p0) # 2: 4 + p2 = getfield_gc_r(p0, descr=valuedescr) # 1: 3,5 + p3 = getfield_gc_r(p0, descr=valuedescr) # 2: 4 guard_nonnull(p2) [p2] # 3: 4,5 guard_nonnull(p3) [p3] # 4: 5 jump(p0,p2) # 5: @@ -532,10 +532,10 @@ def test_cyclic(self): graph = self.build_dependency(""" [p0, p1, p5, p6, p7, p9, p11, p12] # 0: 1,6 - p13 = getfield_gc_r(p9) # 1: 2,5 + p13 = getfield_gc_r(p9, descr=valuedescr) # 1: 2,5 guard_nonnull(p13) [] # 2: 4,5 - i14 = getfield_gc_i(p9) # 3: 5 - p15 = getfield_gc_r(p13) # 4: 5 + i14 = getfield_gc_i(p9, descr=valuedescr) # 3: 5 + p15 = getfield_gc_r(p13, descr=valuedescr) # 4: 5 guard_class(p15, 14073732) [p1, p0, p9, i14, p15, p13, p5, p6, p7] # 5: 6 jump(p0,p1,p5,p6,p7,p9,p11,p12) # 6: """) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -955,12 +955,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_getfield_gc_1(self): ops = """ [i] - p1 = new_with_vtable(descr=nodesize) - setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + p1 = new_with_vtable(descr=nodesize3) + setfield_gc(p1, i, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) jump(i1) """ expected = """ @@ -969,17 +969,16 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ [i] - jump(5) - """ - self.node.value = 5 + jump(7) + """ self.optimize_loop(ops, expected) def test_getfield_gc_nonpure_2(self): @@ -1343,7 +1342,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -1355,7 +1354,7 @@ expected = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -1597,7 +1596,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -1608,7 +1607,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -5076,7 +5075,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i0 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i0 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i1 = call_pure_i(123, i0, descr=nonwritedescr) finish(i1) """ @@ -5462,15 +5461,15 @@ def test_getarrayitem_gc_pure_not_invalidated(self): ops = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) - i2 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i2 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(i2) jump(p0) """ expected = """ [p0] - i1 = getarrayitem_gc_pure_i(p0, 1, descr=arraydescr) + i1 = getarrayitem_gc_pure_i(p0, 1, descr=arrayimmutdescr) escape_n(p0) escape_n(i1) jump(p0) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py @@ -1409,12 +1409,12 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_1(self): + def test_pure_getfield_gc_1(self): ops = """ [i] p1 = new_with_vtable(descr=nodesize) setfield_gc(p1, i, descr=valuedescr) - i1 = getfield_gc_pure_i(p1, descr=valuedescr) + i1 = getfield_gc_i(p1, descr=valuedescr) jump(i1) """ expected = """ @@ -1423,10 +1423,10 @@ """ self.optimize_loop(ops, expected) - def test_getfield_gc_pure_2(self): + def test_pure_getfield_gc_2(self): ops = """ [i] - i1 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i1 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) jump(i1) """ expected = """ @@ -1436,20 +1436,20 @@ self.node.value = 5 self.optimize_loop(ops, expected) - def test_getfield_gc_pure_3(self): + def test_pure_getfield_gc_3(self): ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) - p3 = getfield_gc_pure_r(p1, descr=nextdescr) + p3 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p3) jump() """ expected = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) escape_n(p2) escape_n(p2) jump() @@ -2319,7 +2319,7 @@ setfield_gc(p1, i1, descr=valuedescr) # # some operations on which the above setfield_gc cannot have effect - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) setarrayitem_gc(p3, 0, i5, descr=arraydescr) @@ -2332,7 +2332,7 @@ preamble = """ [p1, i1, i2, p3] # - i3 = getarrayitem_gc_pure_i(p3, 1, descr=arraydescr) + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2340,11 +2340,12 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - jump(p1, i1, i2, p3, i3) - """ - expected = """ - [p1, i1, i2, p3, i3] + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] # + i3 = getarrayitem_gc_i(p3, 1, descr=arraydescr) i4 = getarrayitem_gc_i(p3, i3, descr=arraydescr) i5 = int_add(i3, i4) # @@ -2352,8 +2353,7 @@ setfield_gc(p1, i4, descr=nextdescr) setarrayitem_gc(p3, 0, i5, descr=arraydescr) escape_n() - ifoo = arraylen_gc(p3, descr=arraydescr) # killed by the backend - jump(p1, i1, i2, p3, i3) + jump(p1, i1, i2, p3) """ self.optimize_loop(ops, expected, preamble) @@ -2669,7 +2669,7 @@ ops = """ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) - i4 = getfield_gc_pure_i(ConstPtr(myptr), descr=valuedescr) + i4 = getfield_gc_i(ConstPtr(myptr3), descr=valuedescr3) p5 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) escape_n(i4) @@ -2680,7 +2680,7 @@ [p1, p2] p3 = getarrayitem_gc_r(p1, 0, descr=arraydescr2) escape_n(p3) - escape_n(5) + escape_n(7) escape_n(p3) jump(p1, p2) """ @@ -3302,8 +3302,8 @@ [p8, p11, i24] p26 = new(descr=ssize) setfield_gc(p26, i24, descr=adescr) - i34 = getfield_gc_pure_i(p11, descr=abisdescr) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=abisdescr) + i35 = getfield_gc_i(p26, descr=adescr) i36 = int_add_ovf(i34, i35) guard_no_overflow() [] jump(p8, p11, i35) @@ -3330,8 +3330,8 @@ setfield_gc(p26, i24, descr=adescr) i28 = int_add(i17, 1) setfield_gc(p8, i28, descr=valuedescr) - i34 = getfield_gc_pure_i(p11, descr=valuedescr3) - i35 = getfield_gc_pure_i(p26, descr=adescr) + i34 = getfield_gc_i(p11, descr=valuedescr3) + i35 = getfield_gc_i(p26, descr=adescr) guard_nonnull(p12) [] i36 = int_add_ovf(i34, i35) guard_no_overflow() [] @@ -3522,14 +3522,14 @@ def test_residual_call_does_not_invalidate_immutable_caches(self): ops = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) - i3 = getfield_gc_pure_i(p1, descr=valuedescr3) + i3 = getfield_gc_i(p1, descr=valuedescr3) jump(p1) """ expected_preamble = """ [p1] - i1 = getfield_gc_pure_i(p1, descr=valuedescr3) + i1 = getfield_gc_i(p1, descr=valuedescr3) i2 = call_i(i1, descr=writevalue3descr) jump(p1, i1) """ @@ -4878,11 +4878,11 @@ def test_add_sub_ovf_virtual_unroll(self): ops = """ [p15] - i886 = getfield_gc_pure_i(p15, descr=valuedescr) + i886 = getfield_gc_i(p15, descr=valuedescr) i888 = int_sub_ovf(i886, 1) guard_no_overflow() [] escape_n(i888) - i4360 = getfield_gc_pure_i(p15, descr=valuedescr) + i4360 = getfield_gc_i(p15, descr=valuedescr) i4362 = int_add_ovf(i4360, 1) guard_no_overflow() [] i4360p = int_sub_ovf(i4362, 1) @@ -4972,18 +4972,16 @@ def test_pure(self): ops = """ [p42] - p53 = getfield_gc_r(ConstPtr(myptr), descr=nextdescr) - p59 = getfield_gc_pure_r(p53, descr=valuedescr) + p53 = getfield_gc_r(ConstPtr(myptr3), descr=nextdescr3) + p59 = getfield_gc_r(p53, descr=valuedescr3) i61 = call_i(1, p59, descr=nonwritedescr) jump(p42) """ expected = """ - [p42, p59] - i61 = call_i(1, p59, descr=nonwritedescr) - jump(p42, p59) - - """ - self.node.value = 5 + [p42] + i61 = call_i(1, 7, descr=nonwritedescr) + jump(p42) + """ self.optimize_loop(ops, expected) def test_complains_getfieldpure_setfield(self): @@ -4992,7 +4990,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) setfield_gc(p1, p3, descr=nextdescr) jump(p3) """ @@ -5002,7 +5000,7 @@ ops = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5010,7 +5008,7 @@ expected = """ [p3] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr3) setfield_gc(p1, p3, descr=otherdescr) escape_n(p2) jump(p3) @@ -5021,7 +5019,7 @@ ops = """ [] p1 = escape_r() - p2 = getfield_gc_pure_r(p1, descr=nextdescr) + p2 = getfield_gc_r(p1, descr=nextdescr) p3 = escape_r() setfield_gc(p3, p1, descr=nextdescr) jump() @@ -6167,14 +6165,14 @@ def test_bug_unroll_with_immutables(self): ops = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) p1 = new_with_vtable(descr=immut_descr) setfield_gc(p1, 1242, descr=immut_intval) jump(p1) """ preamble = """ [p0] - i2 = getfield_gc_pure_i(p0, descr=immut_intval) + i2 = getfield_gc_i(p0, descr=immut_intval) jump() """ expected = """ @@ -7229,13 +7227,13 @@ [p0, p1, i0] quasiimmut_field(p0, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ expected = """ [p0, p1, i0] - i1 = getfield_gc_pure_i(p0, descr=quasifielddescr) + i1 = getfield_gc_i(p0, descr=quasifielddescr) escape_n(i1) jump(p1, p0, i1) """ @@ -7246,7 +7244,7 @@ [] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) escape_n(i1) jump() """ @@ -7298,11 +7296,11 @@ [i0a, i0b] quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i1 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(ConstPtr(quasiptr), descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(ConstPtr(quasiptr), descr=quasifielddescr) + i2 = getfield_gc_i(ConstPtr(quasiptr), descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7325,11 +7323,11 @@ setfield_gc(p, 421, descr=quasifielddescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i1 = getfield_gc_pure_i(p, descr=quasifielddescr) + i1 = getfield_gc_i(p, descr=quasifielddescr) call_may_force_n(i0b, descr=mayforcevirtdescr) quasiimmut_field(p, descr=quasiimmutdescr) guard_not_invalidated() [] - i2 = getfield_gc_pure_i(p, descr=quasifielddescr) + i2 = getfield_gc_i(p, descr=quasifielddescr) i3 = escape_i(i1) i4 = escape_i(i2) jump(i3, i4) @@ -7568,7 +7566,7 @@ def test_forced_virtual_pure_getfield(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=valuedescr) + p1 = getfield_gc_r(p0, descr=valuedescr3) jump(p1) """ self.optimize_loop(ops, ops) @@ -7578,7 +7576,7 @@ p1 = new_with_vtable(descr=nodesize3) setfield_gc(p1, p0, descr=valuedescr3) escape_n(p1) - p2 = getfield_gc_pure_r(p1, descr=valuedescr3) + p2 = getfield_gc_r(p1, descr=valuedescr3) escape_n(p2) jump(p0) """ @@ -7852,14 +7850,14 @@ def test_loopinvariant_getarrayitem_gc_pure(self): ops = """ [p9, i1] - i843 = getarrayitem_gc_pure_i(p9, i1, descr=arraydescr) + i843 = getarrayitem_gc_pure_i(p9, i1, descr=arrayimmutdescr) call_n(i843, descr=nonwritedescr) jump(p9, i1) """ expected = """ [p9, i1, i843] call_n(i843, descr=nonwritedescr) - ifoo = arraylen_gc(p9, descr=arraydescr) + ifoo = arraylen_gc(p9, descr=arrayimmutdescr) jump(p9, i1, i843) """ self.optimize_loop(ops, expected) @@ -7868,7 +7866,7 @@ ops = """ [p0] p1 = getfield_gc_r(p0, descr=nextdescr) - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) call_n(p2, descr=nonwritedescr) jump(p0) """ @@ -7883,14 +7881,14 @@ i1 = arraylen_gc(p1, descr=gcarraydescr) i2 = int_ge(i1, 8) guard_true(i2) [] - p2 = getarrayitem_gc_pure_r(p1, 7, descr=gcarraydescr) - jump(p2, p1) - """ - expected = """ - [p0, p2, p1] + p2 = getarrayitem_gc_r(p1, 7, descr=gcarraydescr) + jump(p1, p2) + """ + expected = """ + [p0, p1, p2] call_n(p2, descr=nonwritedescr) i3 = arraylen_gc(p1, descr=gcarraydescr) # Should be killed by backend - jump(p0, p2, p1) + jump(p0, p1, p2) """ self.optimize_loop(ops, expected, expected_short=short) @@ -8065,7 +8063,7 @@ def test_dont_mixup_equal_boxes(self): ops = """ [p8] - i9 = getfield_gc_pure_i(p8, descr=valuedescr) + i9 = getfield_gc_i(p8, descr=valuedescr3) i10 = int_gt(i9, 0) guard_true(i10) [] i29 = int_lshift(i9, 1) @@ -8160,9 +8158,9 @@ py.test.skip("would be fixed by make heap optimizer aware of virtual setfields") ops = """ [p5, p8] - i9 = getfield_gc_pure_i(p5, descr=valuedescr) + i9 = getfield_gc_i(p5, descr=valuedescr) call_n(i9, descr=nonwritedescr) - i11 = getfield_gc_pure_i(p8, descr=valuedescr) + i11 = getfield_gc_i(p8, descr=valuedescr) i13 = int_add_ovf(i11, 1) guard_no_overflow() [] p22 = new_with_vtable(descr=nodesize) @@ -8201,14 +8199,14 @@ ops = """ [p0] p10 = getfield_gc_r(ConstPtr(myptr), descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(ConstPtr(myptr), ConstPtr(myptr2), descr=otherdescr) + setfield_gc(ConstPtr(myptr), ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8232,14 +8230,14 @@ ops = """ [p0] p10 = getfield_gc_r(p0, descr=otherdescr) - guard_value(p10, ConstPtr(myptr2)) [] + guard_value(p10, ConstPtr(myptrb)) [] call_n(p10, descr=nonwritedescr) - setfield_gc(p0, ConstPtr(myptr2), descr=otherdescr) + setfield_gc(p0, ConstPtr(myptrb), descr=otherdescr) jump(p0) """ expected = """ [p0] - call_n(ConstPtr(myptr2), descr=nonwritedescr) + call_n(ConstPtr(myptrb), descr=nonwritedescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -8624,17 +8622,17 @@ [p10] p52 = getfield_gc_r(p10, descr=nextdescr) # inst_storage p54 = getarrayitem_gc_r(p52, 0, descr=arraydescr) - p69 = getfield_gc_pure_r(p54, descr=otherdescr) # inst_w_function + p69 = getfield_gc_r(p54, descr=otherdescr) # inst_w_function quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] p106 = new_with_vtable(descr=nodesize) p108 = new_array(3, descr=arraydescr) p110 = new_with_vtable(descr=nodesize) - setfield_gc(p110, ConstPtr(myptr2), descr=otherdescr) # inst_w_function + setfield_gc(p110, ConstPtr(myptrb), descr=otherdescr) # inst_w_function setarrayitem_gc(p108, 0, p110, descr=arraydescr) setfield_gc(p106, p108, descr=nextdescr) # inst_storage jump(p106) @@ -8650,7 +8648,7 @@ [p69] quasiimmut_field(p69, descr=quasiimmutdescr) guard_not_invalidated() [] - p71 = getfield_gc_pure_r(p69, descr=quasifielddescr) # inst_code + p71 = getfield_gc_r(p69, descr=quasifielddescr) # inst_code guard_value(p71, -4247) [] jump(ConstPtr(myptr)) """ @@ -8852,13 +8850,13 @@ def test_virtual_back_and_forth(self): ops = """ [p0] - p1 = getfield_gc_pure_r(p0, descr=bdescr) + p1 = getfield_gc_r(p0, descr=nextdescr3) ptemp = new_with_vtable(descr=nodesize) setfield_gc(ptemp, p1, descr=nextdescr) p2 = getfield_gc_r(ptemp, descr=nextdescr) - ix = getarrayitem_gc_pure_i(p2, 0, descr=arraydescr) + ix = getarrayitem_gc_pure_i(p2, 0, descr=arrayimmutdescr) pfoo = getfield_gc_r(ptemp, descr=nextdescr) - guard_value(pfoo, ConstPtr(myarray)) [] + guard_value(pfoo, ConstPtr(immutarray)) [] ifoo = int_add(ix, 13) escape_n(ix) jump(p0) @@ -8888,13 +8886,13 @@ def test_constant_float_pure(self): ops = """ [p0] - f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarraydescr) + f0 = getarrayitem_gc_pure_f(p0, 3, descr=floatarrayimmutdescr) guard_value(f0, 1.03) [] jump(p0) """ expected = """ [p0] - ifoo = arraylen_gc(p0, descr=floatarraydescr) + ifoo = arraylen_gc(p0, descr=floatarrayimmutdescr) jump(p0) """ self.optimize_loop(ops, expected) @@ -9102,7 +9100,7 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) jump(NULL, i3) """ @@ -9113,9 +9111,9 @@ [p0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] - getfield_gc_pure_i(p0, descr=valuedescr) + getfield_gc_i(p0, descr=valuedescr3) i3 = int_sub(i1, 1) - jump(ConstPtr(myptr4), i3) + jump(ConstPtr(myptr2), i3) """ py.test.raises(InvalidLoop, self.optimize_loop, ops, ops) @@ -9265,9 +9263,126 @@ guard_value(i1, 5) [] jump() """ - a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE)), 5, zero=True) + a = lltype.malloc(lltype.GcArray(lltype.Ptr(self.NODE3)), 5, zero=True) self.optimize_loop(ops, expected, jump_values=[a]) + def test_large_number_of_immutable_references(self): + ops = """ + [p0] + i0 = getfield_gc_i(p0, descr=bigadescr) + i1 = getfield_gc_i(p0, descr=bigbdescr) + i2 = getfield_gc_i(p0, descr=bigcdescr) + i3 = getfield_gc_i(p0, descr=bigddescr) + i4 = getfield_gc_i(p0, descr=bigedescr) + i5 = getfield_gc_i(p0, descr=bigfdescr) + i6 = getfield_gc_i(p0, descr=biggdescr) + i7 = getfield_gc_i(p0, descr=bighdescr) + i8 = getfield_gc_i(p0, descr=bigidescr) + i9 = getfield_gc_i(p0, descr=bigjdescr) + i10 = getfield_gc_i(p0, descr=bigkdescr) + i11 = getfield_gc_i(p0, descr=bigldescr) + i12 = getfield_gc_i(p0, descr=bigmdescr) + i13 = getfield_gc_i(p0, descr=bigndescr) + i14 = getfield_gc_i(p0, descr=bigodescr) + i15 = getfield_gc_i(p0, descr=bigpdescr) + i16 = getfield_gc_i(p0, descr=bigqdescr) + i17 = getfield_gc_i(p0, descr=bigrdescr) + i18 = getfield_gc_i(p0, descr=bigsdescr) + i19 = getfield_gc_i(p0, descr=bigtdescr) + i20 = getfield_gc_i(p0, descr=bigudescr) + i21 = getfield_gc_i(p0, descr=bigvdescr) + i22 = getfield_gc_i(p0, descr=bigwdescr) + i23 = getfield_gc_i(p0, descr=bigxdescr) + i24 = getfield_gc_i(p0, descr=bigydescr) + i25 = getfield_gc_i(p0, descr=bigzdescr) + i27 = getfield_gc_i(p0, descr=bigbdescr) + i28 = getfield_gc_i(p0, descr=bigcdescr) + i29 = getfield_gc_i(p0, descr=bigddescr) + i30 = getfield_gc_i(p0, descr=bigedescr) + i31 = getfield_gc_i(p0, descr=bigfdescr) + i32 = getfield_gc_i(p0, descr=biggdescr) + i33 = getfield_gc_i(p0, descr=bighdescr) + i34 = getfield_gc_i(p0, descr=bigidescr) + i35 = getfield_gc_i(p0, descr=bigjdescr) + i36 = getfield_gc_i(p0, descr=bigkdescr) + i37 = getfield_gc_i(p0, descr=bigldescr) + i38 = getfield_gc_i(p0, descr=bigmdescr) + i39 = getfield_gc_i(p0, descr=bigndescr) + i40 = getfield_gc_i(p0, descr=bigodescr) + i41 = getfield_gc_i(p0, descr=bigpdescr) + i42 = getfield_gc_i(p0, descr=bigqdescr) + i43 = getfield_gc_i(p0, descr=bigrdescr) + i44 = getfield_gc_i(p0, descr=bigsdescr) + i45 = getfield_gc_i(p0, descr=bigtdescr) + i46 = getfield_gc_i(p0, descr=bigudescr) + i47 = getfield_gc_i(p0, descr=bigvdescr) + i48 = getfield_gc_i(p0, descr=bigwdescr) + i49 = getfield_gc_i(p0, descr=bigxdescr) + i50 = getfield_gc_i(p0, descr=bigydescr) + i51 = getfield_gc_i(p0, descr=bigzdescr) + i26 = getfield_gc_i(p0, descr=bigadescr) + i99 = int_add(i26, i51) + escape_i(i27) + escape_i(i28) + escape_i(i29) + escape_i(i30) + escape_i(i31) + escape_i(i32) + escape_i(i33) + escape_i(i34) + escape_i(i35) + escape_i(i36) + escape_i(i37) + escape_i(i38) + escape_i(i39) + escape_i(i40) + escape_i(i41) + escape_i(i42) + escape_i(i43) + escape_i(i44) + escape_i(i45) + escape_i(i46) + escape_i(i47) + escape_i(i48) + escape_i(i49) + escape_i(i50) + escape_i(i51) + escape_i(i26) + escape_i(i99) + jump(p0) + """ + expected = """ + [p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99] + escape_i(i1) + escape_i(i2) + escape_i(i3) + escape_i(i4) + escape_i(i5) + escape_i(i6) + escape_i(i7) + escape_i(i8) + escape_i(i9) + escape_i(i10) + escape_i(i11) + escape_i(i12) + escape_i(i13) + escape_i(i14) + escape_i(i15) + escape_i(i16) + escape_i(i17) + escape_i(i18) + escape_i(i19) + escape_i(i20) + escape_i(i21) + escape_i(i22) + escape_i(i23) + escape_i(i24) + escape_i(i25) + escape_i(i0) + escape_i(i99) + jump(p0,i1,i2,i3,i4,i5,i6,i7,i8,i9,i10,i11,i12,i13,i14,i15,i16,i17,i18,i19,i20,i21,i22,i23,i24,i25,i0,i99) + """ + self.optimize_loop(ops, expected) class TestLLtype(OptimizeOptTest, LLtypeMixin): pass diff --git a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py --- a/rpython/jit/metainterp/optimizeopt/test/test_unroll.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_unroll.py @@ -220,16 +220,16 @@ def test_double_getfield_plus_pure(self): loop = """ [p0] - pc = getfield_gc_pure_r(p0, descr=nextdescr) + pc = getfield_gc_r(p0, descr=nextdescr3) escape_n(p0) # that should flush the caches - p1 = getfield_gc_r(pc, descr=nextdescr) - i0 = getfield_gc_i(p1, descr=valuedescr) + p1 = getfield_gc_r(pc, descr=nextdescr3) + i0 = getfield_gc_i(p1, descr=valuedescr3) jump(p0) """ es, loop, preamble = self.optimize(loop) assert len(es.short_boxes) == 4 # both getfields are available as - # well as getfield_gc_pure + # well as getfield_gc def test_p123_anti_nested(self): loop = """ diff --git a/rpython/jit/metainterp/optimizeopt/test/test_util.py b/rpython/jit/metainterp/optimizeopt/test/test_util.py --- a/rpython/jit/metainterp/optimizeopt/test/test_util.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_util.py @@ -1,4 +1,4 @@ -import py, random +import py, random, string from rpython.rlib.debug import debug_print from rpython.rtyper.lltypesystem import lltype, llmemory, rffi @@ -122,7 +122,14 @@ ('value', lltype.Signed), ('next', lltype.Ptr(NODE3)), hints={'immutable': True})) - + + big_fields = [('big' + i, lltype.Signed) for i in string.ascii_lowercase] + BIG = lltype.GcForwardReference() + BIG.become(lltype.GcStruct('BIG', *big_fields, hints={'immutable': True})) + + for field, _ in big_fields: + locals()[field + 'descr'] = cpu.fielddescrof(BIG, field) + node = lltype.malloc(NODE) node.value = 5 node.next = node @@ -133,16 +140,25 @@ node2.parent.parent.typeptr = node_vtable2 node2addr = lltype.cast_opaque_ptr(llmemory.GCREF, node2) myptr = lltype.cast_opaque_ptr(llmemory.GCREF, node) - mynode2 = lltype.malloc(NODE) + mynodeb = lltype.malloc(NODE) myarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(lltype.GcArray(lltype.Signed), 13, zero=True)) - mynode2.parent.typeptr = node_vtable - myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode2) - mynode3 = lltype.malloc(NODE2) - mynode3.parent.parent.typeptr = node_vtable2 + mynodeb.parent.typeptr = node_vtable + myptrb = lltype.cast_opaque_ptr(llmemory.GCREF, mynodeb) + myptr2 = lltype.malloc(NODE2) + myptr2.parent.parent.typeptr = node_vtable2 + myptr2 = lltype.cast_opaque_ptr(llmemory.GCREF, myptr2) + nullptr = lltype.nullptr(llmemory.GCREF.TO) + + mynode3 = lltype.malloc(NODE3) + mynode3.parent.typeptr = node_vtable3 + mynode3.value = 7 + mynode3.next = mynode3 myptr3 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode3) # a NODE2 mynode4 = lltype.malloc(NODE3) mynode4.parent.typeptr = node_vtable3 myptr4 = lltype.cast_opaque_ptr(llmemory.GCREF, mynode4) # a NODE3 + + nullptr = lltype.nullptr(llmemory.GCREF.TO) #nodebox2 = InputArgRef(lltype.cast_opaque_ptr(llmemory.GCREF, node2)) nodesize = cpu.sizeof(NODE, node_vtable) @@ -203,7 +219,6 @@ arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed)) int32arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.INT)) int16arraydescr = cpu.arraydescrof(lltype.GcArray(rffi.SHORT)) - floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) float32arraydescr = cpu.arraydescrof(lltype.GcArray(lltype.SingleFloat)) arraydescr_tid = arraydescr.get_type_id() array = lltype.malloc(lltype.GcArray(lltype.Signed), 15, zero=True) @@ -212,6 +227,12 @@ array2ref = lltype.cast_opaque_ptr(llmemory.GCREF, array2) gcarraydescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF)) gcarraydescr_tid = gcarraydescr.get_type_id() + floatarraydescr = cpu.arraydescrof(lltype.GcArray(lltype.Float)) + + arrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Signed, hints={"immutable": True})) + immutarray = lltype.cast_opaque_ptr(llmemory.GCREF, lltype.malloc(arrayimmutdescr.A, 13, zero=True)) + gcarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(llmemory.GCREF, hints={"immutable": True})) + floatarrayimmutdescr = cpu.arraydescrof(lltype.GcArray(lltype.Float, hints={"immutable": True})) # a GcStruct not inheriting from OBJECT tpl = lltype.malloc(S, zero=True) @@ -244,7 +265,7 @@ tsize = cpu.sizeof(T, None) cdescr = cpu.fielddescrof(T, 'c') ddescr = cpu.fielddescrof(T, 'd') - arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE))) + arraydescr3 = cpu.arraydescrof(lltype.GcArray(lltype.Ptr(NODE3))) U = lltype.GcStruct('U', ('parent', OBJECT), diff --git a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py --- a/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_virtualstate.py @@ -1103,8 +1103,8 @@ jump(p0) """ self.optimize_bridge(loops, bridge, loops[0], 'Loop0', [self.myptr]) - self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr3]) - self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr3]) + self.optimize_bridge(loops, bridge, loops[1], 'Loop1', [self.myptr2]) + self.optimize_bridge(loops[0], bridge, 'RETRACE', [self.myptr2]) self.optimize_bridge(loops, loops[0], loops[0], 'Loop0', [self.nullptr]) self.optimize_bridge(loops, loops[1], loops[1], 'Loop1', [self.nullptr]) diff --git a/rpython/jit/metainterp/optimizeopt/virtualize.py b/rpython/jit/metainterp/optimizeopt/virtualize.py --- a/rpython/jit/metainterp/optimizeopt/virtualize.py +++ b/rpython/jit/metainterp/optimizeopt/virtualize.py @@ -188,12 +188,6 @@ optimize_GETFIELD_GC_R = optimize_GETFIELD_GC_I optimize_GETFIELD_GC_F = optimize_GETFIELD_GC_I - # note: the following line does not mean that the two operations are - # completely equivalent, because GETFIELD_GC_PURE is_always_pure(). - optimize_GETFIELD_GC_PURE_I = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_R = optimize_GETFIELD_GC_I - optimize_GETFIELD_GC_PURE_F = optimize_GETFIELD_GC_I - def optimize_SETFIELD_GC(self, op): struct = op.getarg(0) opinfo = self.getptrinfo(struct) diff --git a/rpython/jit/metainterp/pyjitpl.py b/rpython/jit/metainterp/pyjitpl.py --- a/rpython/jit/metainterp/pyjitpl.py +++ b/rpython/jit/metainterp/pyjitpl.py @@ -653,46 +653,37 @@ @arguments("box", "descr") def opimpl_getfield_gc_i(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resbox = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_I, fielddescr, box) + return ConstInt(resbox) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_I, box, fielddescr, 'i') + + @arguments("box", "descr") + def opimpl_getfield_gc_f(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + resvalue = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_F, fielddescr, box) + return ConstFloat(resvalue) + return self._opimpl_getfield_gc_any_pureornot( + rop.GETFIELD_GC_F, box, fielddescr, 'f') + @arguments("box", "descr") def opimpl_getfield_gc_r(self, box, fielddescr): + if fielddescr.is_always_pure() and isinstance(box, ConstPtr): + # if 'box' is directly a ConstPtr, bypass the heapcache completely + val = executor.execute(self.metainterp.cpu, self.metainterp, + rop.GETFIELD_GC_R, fielddescr, box) + return ConstPtr(val) return self._opimpl_getfield_gc_any_pureornot( rop.GETFIELD_GC_R, box, fielddescr, 'r') - @arguments("box", "descr") - def opimpl_getfield_gc_f(self, box, fielddescr): - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_i_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resbox = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_I, fielddescr, box) - return ConstInt(resbox) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_I, box, fielddescr, 'i') - - @arguments("box", "descr") - def opimpl_getfield_gc_f_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - resvalue = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_F, fielddescr, box) - return ConstFloat(resvalue) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_F, box, fielddescr, 'f') - - @arguments("box", "descr") - def opimpl_getfield_gc_r_pure(self, box, fielddescr): - if isinstance(box, ConstPtr): - # if 'box' is directly a ConstPtr, bypass the heapcache completely - val = executor.execute(self.metainterp.cpu, self.metainterp, - rop.GETFIELD_GC_PURE_R, fielddescr, box) - return ConstPtr(val) - return self._opimpl_getfield_gc_any_pureornot( - rop.GETFIELD_GC_PURE_R, box, fielddescr, 'r') + + opimpl_getfield_gc_i_pure = opimpl_getfield_gc_i + opimpl_getfield_gc_r_pure = opimpl_getfield_gc_r + opimpl_getfield_gc_f_pure = opimpl_getfield_gc_f @arguments("box", "box", "descr") def opimpl_getinteriorfield_gc_i(self, array, index, descr): @@ -733,7 +724,7 @@ @arguments("box", "descr", "orgpc") def _opimpl_getfield_gc_greenfield_any(self, box, fielddescr, pc): ginfo = self.metainterp.jitdriver_sd.greenfield_info - opnum = OpHelpers.getfield_pure_for_descr(fielddescr) + opnum = OpHelpers.getfield_for_descr(fielddescr) if (ginfo is not None and fielddescr in ginfo.green_field_descrs and not self._nonstandard_virtualizable(pc, box, fielddescr)): # fetch the result, but consider it as a Const box and don't @@ -2104,17 +2095,7 @@ profiler = self.staticdata.profiler profiler.count_ops(opnum) resvalue = executor.execute(self.cpu, self, opnum, descr, *argboxes) - # - is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST - if not is_pure: - if (opnum == rop.GETFIELD_RAW_I or - opnum == rop.GETFIELD_RAW_R or - opnum == rop.GETFIELD_RAW_F or - opnum == rop.GETARRAYITEM_RAW_I or - opnum == rop.GETARRAYITEM_RAW_F): - is_pure = descr.is_always_pure() - # - if is_pure: + if OpHelpers.is_pure_with_descr(opnum, descr): return self._record_helper_pure(opnum, resvalue, descr, *argboxes) if rop._OVF_FIRST <= opnum <= rop._OVF_LAST: return self._record_helper_ovf(opnum, resvalue, descr, *argboxes) diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py --- a/rpython/jit/metainterp/resoperation.py +++ b/rpython/jit/metainterp/resoperation.py @@ -231,7 +231,7 @@ class AbstractResOpOrInputArg(AbstractValue): _attrs_ = ('_forwarded',) - _forwarded = None # either another resop or OptInfo + _forwarded = None # either another resop or OptInfo def get_forwarded(self): return self._forwarded @@ -412,6 +412,8 @@ return rop._JIT_DEBUG_FIRST <= self.getopnum() <= rop._JIT_DEBUG_LAST def is_always_pure(self): + # Tells whether an operation is pure based solely on the opcode. + # Other operations (e.g. getfield ops) may be pure in some cases are well. return rop._ALWAYS_PURE_FIRST <= self.getopnum() <= rop._ALWAYS_PURE_LAST def has_no_side_effect(self): @@ -434,9 +436,7 @@ return self.opnum in (rop.SAME_AS_I, rop.SAME_AS_F, rop.SAME_AS_R) def is_getfield(self): - return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, - rop.GETFIELD_GC_R, rop.GETFIELD_GC_PURE_I, - rop.GETFIELD_GC_PURE_R, rop.GETFIELD_GC_PURE_F) + return self.opnum in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R) def is_getarrayitem(self): return self.opnum in (rop.GETARRAYITEM_GC_I, rop.GETARRAYITEM_GC_F, @@ -1154,7 +1154,6 @@ 'ARRAYLEN_GC/1d/i', 'STRLEN/1/i', 'STRGETITEM/2/i', - 'GETFIELD_GC_PURE/1d/rfi', 'GETARRAYITEM_GC_PURE/2d/rfi', #'GETFIELD_RAW_PURE/1d/rfi', these two operations not useful and #'GETARRAYITEM_RAW_PURE/2d/fi', dangerous when unrolling speculatively @@ -1602,14 +1601,6 @@ return rop.CALL_LOOPINVARIANT_N @staticmethod - def getfield_pure_for_descr(descr): - if descr.is_pointer_field(): - return rop.GETFIELD_GC_PURE_R - elif descr.is_float_field(): - return rop.GETFIELD_GC_PURE_F - return rop.GETFIELD_GC_PURE_I - - @staticmethod def getfield_for_descr(descr): if descr.is_pointer_field(): return rop.GETFIELD_GC_R @@ -1760,4 +1751,26 @@ opnum = rop.VEC_UNPACK_F return VecOperationNew(opnum, args, datatype, bytesize, signed, count) + @staticmethod + def is_pure_getfield(opnum, descr): + if (opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETFIELD_GC_R): + return descr is not None and descr.is_always_pure() + return False + @staticmethod + def is_pure_with_descr(opnum, descr): + is_pure = rop._ALWAYS_PURE_FIRST <= opnum <= rop._ALWAYS_PURE_LAST + if not is_pure: + if (opnum == rop.GETFIELD_RAW_I or + opnum == rop.GETFIELD_RAW_R or + opnum == rop.GETFIELD_RAW_F or + opnum == rop.GETFIELD_GC_I or + opnum == rop.GETFIELD_GC_R or + opnum == rop.GETFIELD_GC_F or + opnum == rop.GETARRAYITEM_RAW_I or + opnum == rop.GETARRAYITEM_RAW_F): + is_pure = descr.is_always_pure() + return is_pure + diff --git a/rpython/jit/metainterp/test/test_ajit.py b/rpython/jit/metainterp/test/test_ajit.py --- a/rpython/jit/metainterp/test/test_ajit.py +++ b/rpython/jit/metainterp/test/test_ajit.py @@ -320,7 +320,7 @@ assert res == 252 self.check_trace_count(1) self.check_resops({'jump': 1, 'int_gt': 2, 'int_add': 2, - 'getfield_gc_pure_i': 1, 'int_mul': 1, + 'getfield_gc_i': 1, 'int_mul': 1, 'guard_true': 2, 'int_sub': 2}) def test_loops_are_transient(self): @@ -1405,7 +1405,7 @@ return tup[1] res = self.interp_operations(f, [3, 5]) assert res == 5 - self.check_operations_history(setfield_gc=2, getfield_gc_pure_i=0) + self.check_operations_history(setfield_gc=2, getfield_gc_i=0) def test_oosend_look_inside_only_one(self): class A: @@ -2522,7 +2522,7 @@ if counter > 10: return 7 assert self.meta_interp(build, []) == 7 - self.check_resops(getfield_gc_pure_r=2) + self.check_resops(getfield_gc_r=2) def test_args_becomming_equal(self): myjitdriver = JitDriver(greens = [], reds = ['n', 'i', 'sa', 'a', 'b']) diff --git a/rpython/jit/metainterp/test/test_immutable.py b/rpython/jit/metainterp/test/test_immutable.py --- a/rpython/jit/metainterp/test/test_immutable.py +++ b/rpython/jit/metainterp/test/test_immutable.py @@ -19,7 +19,7 @@ return y.x + 5 res = self.interp_operations(f, [23]) assert res == 28 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=1, int_add=1) + self.check_operations_history(getfield_gc_i=1, int_add=1) def test_fields_subclass(self): class X(object): @@ -41,8 +41,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def f(x, y): # this time, the field 'x' only shows up on subclass 'Y' @@ -50,8 +49,7 @@ return z.x + z.y + 5 res = self.interp_operations(f, [23, 11]) assert res == 39 - self.check_operations_history(getfield_gc_i=0, getfield_gc_pure_i=2, - int_add=2) + self.check_operations_history(getfield_gc_i=2, int_add=2) def test_array(self): class X(object): @@ -66,8 +64,7 @@ return a.y[index] res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_index_error(self): class X(object): @@ -89,8 +86,7 @@ return a.get(index) res = self.interp_operations(f, [2], listops=True) assert res == 30 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) + self.check_operations_history(getfield_gc_r=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1) def test_array_in_immutable(self): class X(object): @@ -106,8 +102,7 @@ return y.lst[index] + y.y + 5 res = self.interp_operations(f, [23, 0], listops=True) assert res == 23 + 24 + 5 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=1, - getfield_gc_pure_i=1, + self.check_operations_history(getfield_gc_r=1, getfield_gc_i=1, getarrayitem_gc_i=0, getarrayitem_gc_pure_i=1, int_add=3) diff --git a/rpython/jit/metainterp/test/test_quasiimmut.py b/rpython/jit/metainterp/test/test_quasiimmut.py --- a/rpython/jit/metainterp/test/test_quasiimmut.py +++ b/rpython/jit/metainterp/test/test_quasiimmut.py @@ -74,7 +74,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -101,7 +101,7 @@ res = self.meta_interp(f, [100, 7], enable_opts="") assert res == 700 # there should be no getfields, even though optimizations are turned off - self.check_resops(guard_not_invalidated=1, getfield_gc=0) + self.check_resops(guard_not_invalidated=1) def test_nonopt_1(self): myjitdriver = JitDriver(greens=[], reds=['x', 'total', 'lst']) @@ -124,8 +124,7 @@ assert f(100, 7) == 721 res = self.meta_interp(f, [100, 7]) assert res == 721 - self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, - getfield_gc_pure_i=2) + self.check_resops(guard_not_invalidated=0, getfield_gc_r=1, getfield_gc_i=2) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -156,7 +155,7 @@ # res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=2, getfield_gc=0) + self.check_resops(guard_not_invalidated=2) def test_change_during_tracing_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -208,7 +207,7 @@ assert f(100, 7) == 700 res = self.meta_interp(f, [100, 7]) assert res == 700 - self.check_resops(guard_not_invalidated=0, getfield_gc=0) + self.check_resops(guard_not_invalidated=0) def test_change_invalidate_reentering(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -234,7 +233,7 @@ assert g(100, 7) == 700707 res = self.meta_interp(g, [100, 7]) assert res == 700707 - self.check_resops(guard_not_invalidated=4, getfield_gc=0) + self.check_resops(guard_not_invalidated=4) def test_invalidate_while_running(self): jitdriver = JitDriver(greens=['foo'], reds=['i', 'total']) @@ -348,7 +347,7 @@ res = self.meta_interp(f, [100, 30]) assert res == 6019 self.check_resops(guard_not_invalidated=8, guard_not_forced=0, - call_may_force=0, getfield_gc=0) + call_may_force=0) def test_list_simple_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) @@ -374,8 +373,7 @@ getarrayitem_gc_pure_r=0, getarrayitem_gc_i=0, getarrayitem_gc_r=0, - getfield_gc_i=0, getfield_gc_pure_i=0, - getfield_gc_r=0, getfield_gC_pure_r=0) + getfield_gc_i=0, getfield_gc_r=0) # from rpython.jit.metainterp.warmspot import get_stats loops = get_stats().loops @@ -405,9 +403,7 @@ assert res == 700 # operations must have been removed by the frontend self.check_resops(getarrayitem_gc_pure_i=0, guard_not_invalidated=1, - getarrayitem_gc_i=0, - getfield_gc=0, getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + getarrayitem_gc_i=0, getfield_gc_i=0, getfield_gc_r=0) def test_list_length_1(self): myjitdriver = JitDriver(greens=['foo'], reds=['x', 'total']) diff --git a/rpython/jit/metainterp/test/test_tracingopts.py b/rpython/jit/metainterp/test/test_tracingopts.py --- a/rpython/jit/metainterp/test/test_tracingopts.py +++ b/rpython/jit/metainterp/test/test_tracingopts.py @@ -436,10 +436,10 @@ return p.x[0] + p.x[1] res = self.interp_operations(fn, [7]) assert res == 7 + 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == -7 - 7 + 1 - self.check_operations_history(getfield_gc_r=0, getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_r=0) def test_heap_caching_and_elidable_function(self): class A: @@ -517,12 +517,12 @@ return a1[0] + a2[0] + gn(a1, a2) res = self.interp_operations(fn, [7]) assert res == 2 * 7 + 2 * 6 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) res = self.interp_operations(fn, [-7]) assert res == 2 * -7 + 2 * -8 - self.check_operations_history(getfield_gc_pure_i=0, - getfield_gc_pure_r=0) + self.check_operations_history(getfield_gc_i=0, + getfield_gc_r=0) def test_heap_caching_multiple_arrays(self): class Gbl(object): diff --git a/rpython/jit/metainterp/test/test_virtual.py b/rpython/jit/metainterp/test/test_virtual.py --- a/rpython/jit/metainterp/test/test_virtual.py +++ b/rpython/jit/metainterp/test/test_virtual.py @@ -1077,7 +1077,7 @@ res = self.meta_interp(f, [], repeat=7) assert res == f() - def test_getfield_gc_pure_nobug(self): + def test_pure_getfield_gc_nobug(self): mydriver = JitDriver(reds=['i', 's', 'a'], greens=[]) class A(object): diff --git a/rpython/rlib/test/test_rpath.py b/rpython/rlib/test/test_rpath.py --- a/rpython/rlib/test/test_rpath.py +++ b/rpython/rlib/test/test_rpath.py @@ -68,8 +68,8 @@ assert rpath._nt_rabspath('d:\\foo\\bar\\..') == 'd:\\foo' assert rpath._nt_rabspath('d:\\foo\\bar\\..\\x') == 'd:\\foo\\x' curdrive = _ = rpath._nt_rsplitdrive(os.getcwd()) - assert len(curdrive) == 2 and curdrive[1] == ':' - assert rpath.rabspath('\\foo') == '%s\\foo' % curdrive + assert len(curdrive) == 2 and curdrive[0][1] == ':' + assert rpath.rabspath('\\foo') == '%s\\foo' % curdrive[0] def test_risabs_posix(): assert rpath._posix_risabs('/foo/bar') diff --git a/rpython/translator/platform/windows.py b/rpython/translator/platform/windows.py --- a/rpython/translator/platform/windows.py +++ b/rpython/translator/platform/windows.py @@ -151,7 +151,7 @@ # Increase stack size, for the linker and the stack check code. stack_size = 8 << 20 # 8 Mb - self.link_flags.append('/STACK:%d' % stack_size) + self.link_flags = self.link_flags + ('/STACK:%d' % stack_size,) # The following symbol is used in c/src/stack.h self.cflags.append('/DMAX_STACK_SIZE=%d' % (stack_size - 1024)) From pypy.commits at gmail.com Tue Jan 26 03:07:50 2016 From: pypy.commits at gmail.com (Raemi) Date: Tue, 26 Jan 2016 00:07:50 -0800 (PST) Subject: [pypy-commit] pypy stmgc-c8: clarification Message-ID: <56a72956.85e41c0a.4104c.ffffc1d8@mx.google.com> Author: Remi Meier Branch: stmgc-c8 Changeset: r81945:23f42ac860b5 Date: 2016-01-26 09:08 +0100 http://bitbucket.org/pypy/pypy/changeset/23f42ac860b5/ Log: clarification diff --git a/rpython/jit/backend/llsupport/stmrewrite.py b/rpython/jit/backend/llsupport/stmrewrite.py --- a/rpython/jit/backend/llsupport/stmrewrite.py +++ b/rpython/jit/backend/llsupport/stmrewrite.py @@ -140,6 +140,12 @@ # was necessary in C7 for others to commit, but in C8 it is only # necessary for requesting major GCs. I think we better avoid this # overhead for tight loops and wait a bit longer in that case. + # ^^^ is not the entire truth: we currently measure the amount of work + # done in a transaction by number of bytes allocated. It means that + # now, tight loops not doing any allocation are not accounted for. + # However, given that not doing these allocations improves + # lee_router_tm.py by a factor of 2.5x, we better deal with it in + # another way. pass # if not self.does_any_allocation: # # do a fake allocation since this is needed to check From pypy.commits at gmail.com Tue Jan 26 03:08:16 2016 From: pypy.commits at gmail.com (plan_rich) Date: Tue, 26 Jan 2016 00:08:16 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: removed print statements, typo, removed push floats from stacklet because there is no need to save them Message-ID: <56a72970.d4811c0a.40e2e.ffff818d@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81946:62aeb645010b Date: 2016-01-26 09:07 +0100 http://bitbucket.org/pypy/pypy/changeset/62aeb645010b/ Log: removed print statements, typo, removed push floats from stacklet because there is no need to save them diff --git a/rpython/jit/backend/zarch/instructions.py b/rpython/jit/backend/zarch/instructions.py --- a/rpython/jit/backend/zarch/instructions.py +++ b/rpython/jit/backend/zarch/instructions.py @@ -287,9 +287,3 @@ all_mnemonic_codes.update(memory_mnemonic_codes) all_mnemonic_codes.update(floatingpoint_mnemonic_codes) all_mnemonic_codes.update(branch_mnemonic_codes) - - -if __name__ == "__main__": - print("%d instructions:" % len(all_mnemonic_codes)) - for name, (typeinstr, _) in all_mnemonic_codes.items(): - print(" %s\t(type: %s)" % (name, typeinstr)) diff --git a/rpython/jit/backend/zarch/pool.py b/rpython/jit/backend/zarch/pool.py --- a/rpython/jit/backend/zarch/pool.py +++ b/rpython/jit/backend/zarch/pool.py @@ -131,7 +131,7 @@ if size == -1: size = self.size if size >= 2**19: - msg = '[S390X/literalpool] size exceeded %d >= %d\n' % (size, 2**19-8) + msg = '[S390X/literalpool] size exceeded %d >= %d\n' % (size, 2**19) if we_are_translated(): llop.debug_print(lltype.Void, msg) raise PoolOverflow(msg) @@ -180,13 +180,8 @@ asm.mc.write('\x00' * self.size) wrote = 0 for val, offset in self.offset_map.items(): - if not we_are_translated(): - print('pool: %s at offset: %d' % (val, offset)) self.overwrite_64(asm.mc, offset, val) wrote += 8 - # for the descriptors - if not we_are_translated(): - print "pool with %d quad words" % (self.size // 8) def overwrite_64(self, mc, index, value): index += self.pool_start diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py --- a/rpython/jit/backend/zarch/regalloc.py +++ b/rpython/jit/backend/zarch/regalloc.py @@ -1215,40 +1215,3 @@ raise NotImplementedError(msg) prepare_oplist = [notimplemented] * (rop._LAST + 1) - -if not we_are_translated(): - implemented_count = 0 - total_count = 0 - missing = [] - for key, value in rop.__dict__.items(): - key = key.lower() - if key.startswith('_'): - continue - total_count += 1 - methname = 'prepare_%s' % key - if hasattr(Regalloc, methname): - func = getattr(Regalloc, methname).im_func - prepare_oplist[value] = func - implemented_count += 1 - else: - if not methname.startswith('prepare_vec') and \ - not methname.startswith('prepare_get') and \ - not methname.startswith('prepare_raw') and \ - not methname.startswith('prepare_unicodesetitem') and \ - not methname.startswith('prepare_unicodegetitem') and \ - not methname.startswith('prepare_strgetitem') and \ - not methname.startswith('prepare_strsetitem') and \ - not methname.startswith('prepare_call_loopinvariant') and \ - not methname.startswith('prepare_call_pure') and \ - not methname.startswith('prepare_new') and \ - not methname.startswith('prepare_set'): - missing.append(methname) - else: - implemented_count += 1 - - if __name__ == '__main__': - for m in missing: - print(" " * 4 + m) - print - print("regalloc implements %d of %d = %.2f%% of all resops" % \ - (implemented_count, total_count, (100.0 * implemented_count / total_count))) diff --git a/rpython/translator/c/src/stacklet/switch_s390x_gcc.h b/rpython/translator/c/src/stacklet/switch_s390x_gcc.h --- a/rpython/translator/c/src/stacklet/switch_s390x_gcc.h +++ b/rpython/translator/c/src/stacklet/switch_s390x_gcc.h @@ -12,10 +12,7 @@ "stmg 6,15,48(15)\n" - "std 0,128(15)\n" - "std 2,136(15)\n" - "std 4,144(15)\n" - "std 6,152(15)\n" + "lay 15,-160(15)\n" /* create stack frame */ "lgr 10, %[restore_state]\n" /* save 'restore_state' for later */ "lgr 11, %[extra]\n" /* save 'extra' for later */ @@ -23,9 +20,7 @@ "lgr 2, 15\n" /* arg 1: current (old) stack pointer */ "lgr 3, 11\n" /* arg 2: extra */ - "lay 15,-160(15)\n" /* create stack frame */ "basr 14, 14\n" /* call save_state() */ - "lay 15, 160(15)\n" /* destroy stack frame */ "cgij 2, 0, 8, zero\n" /* skip the rest if the return value is null */ @@ -38,10 +33,8 @@ "lgr 3, 11\n" /* arg 2: extra */ - "lay 15, -160(15)\n" /* create temp stack space for callee to use */ "lgr 14, 10\n" /* load restore_state */ "basr 14, 14\n" /* call restore_state() */ - "lay 15, 160(15)\n" /* destroy temp stack space */ /* The stack's content is now restored. */ @@ -49,13 +42,7 @@ /* Epilogue */ /* no need */ /* restore stack pointer */ - - "ld 0,128(15)\n" - "ld 2,136(15)\n" - "ld 4,144(15)\n" - "ld 6,152(15)\n" - - "lmg 6,15,48(15)\n" + "lmg 6,15,208(15)\n" : "=r"(result) /* output variable: expected to be r2 */ : [restore_state]"r"(restore_state), /* input variables */ diff --git a/rpython/translator/c/test/test_newgc.py b/rpython/translator/c/test/test_newgc.py --- a/rpython/translator/c/test/test_newgc.py +++ b/rpython/translator/c/test/test_newgc.py @@ -698,7 +698,7 @@ # related to libffi issue on s390x, we MUST # overwrite the full ffi result which is 64 bit # if not, this leaves garbage in the return value - # and qsort does not sorrt correctly + # and qsort does not sort correctly res = rffi.cast(rffi.SIGNEDP, ll_res) if a1 > a2: res[0] = 1 From pypy.commits at gmail.com Tue Jan 26 03:36:48 2016 From: pypy.commits at gmail.com (plan_rich) Date: Tue, 26 Jan 2016 00:36:48 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: simplified guard_class Message-ID: <56a73020.0c2e1c0a.18f57.ffffc9e4@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81947:1589c0b36069 Date: 2016-01-26 09:36 +0100 http://bitbucket.org/pypy/pypy/changeset/1589c0b36069/ Log: simplified guard_class diff --git a/rpython/jit/backend/zarch/opassembler.py b/rpython/jit/backend/zarch/opassembler.py --- a/rpython/jit/backend/zarch/opassembler.py +++ b/rpython/jit/backend/zarch/opassembler.py @@ -716,9 +716,6 @@ def _cmp_guard_gc_type(self, loc_ptr, expected_typeid): self._read_typeid(r.SCRATCH2, loc_ptr) assert 0 <= expected_typeid <= 0x7fffffff # 4 bytes are always enough - if expected_typeid > 0xffff: # if 2 bytes are not enough - self.mc.AGHI(r.SCRATCH2, l.imm(-(expected_typeid >> 16))) - expected_typeid = expected_typeid & 0xffff self.mc.cmp_op(r.SCRATCH2, l.imm(expected_typeid), imm=True, signed=False) From pypy.commits at gmail.com Tue Jan 26 08:02:10 2016 From: pypy.commits at gmail.com (plan_rich) Date: Tue, 26 Jan 2016 05:02:10 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: gotcha, guard nonnull class was implemented incorrectly (substracted value that should have been shifted 16 bits to the left) Message-ID: <56a76e52.88c8c20a.365f1.ffff83b3@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81948:acf6ac9d0d3d Date: 2016-01-26 14:01 +0100 http://bitbucket.org/pypy/pypy/changeset/acf6ac9d0d3d/ Log: gotcha, guard nonnull class was implemented incorrectly (substracted value that should have been shifted 16 bits to the left) diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -542,7 +542,6 @@ mc.restore_link() # So we return to our caller, conditionally if "EQ" mc.BCR(c.EQ, r.r14) - mc.trap() # debug if this is EVER executed! # # Else, jump to propagate_exception_path assert self.propagate_exception_path diff --git a/rpython/jit/backend/zarch/opassembler.py b/rpython/jit/backend/zarch/opassembler.py --- a/rpython/jit/backend/zarch/opassembler.py +++ b/rpython/jit/backend/zarch/opassembler.py @@ -682,11 +682,10 @@ def emit_guard_nonnull_class(self, op, arglocs, regalloc): self.mc.cmp_op(arglocs[0], l.imm(1), imm=True, signed=False) patch_pos = self.mc.currpos() - self.mc.trap() - self.mc.write('\x00' * 4) + self.mc.reserve_cond_jump(short=True) self._cmp_guard_class(op, arglocs, regalloc) pmc = OverwritingBuilder(self.mc, patch_pos, 1) - pmc.BRCL(c.LT, l.imm(self.mc.currpos() - patch_pos)) + pmc.BRC(c.LT, l.imm(self.mc.currpos() - patch_pos)) pmc.overwrite() self.guard_success_cc = c.EQ self._emit_guard(op, arglocs[2:]) @@ -716,6 +715,7 @@ def _cmp_guard_gc_type(self, loc_ptr, expected_typeid): self._read_typeid(r.SCRATCH2, loc_ptr) assert 0 <= expected_typeid <= 0x7fffffff # 4 bytes are always enough + # we can handle 4 byte compare immediate self.mc.cmp_op(r.SCRATCH2, l.imm(expected_typeid), imm=True, signed=False) From pypy.commits at gmail.com Tue Jan 26 10:46:06 2016 From: pypy.commits at gmail.com (plan_rich) Date: Tue, 26 Jan 2016 07:46:06 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: fixed translation issue. reverted the stacklet changes (did not pass test) Message-ID: <56a794be.2851c20a.bd9fa.ffffc408@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81949:fb7bbc354de2 Date: 2016-01-26 16:40 +0100 http://bitbucket.org/pypy/pypy/changeset/fb7bbc354de2/ Log: fixed translation issue. reverted the stacklet changes (did not pass test) diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py --- a/rpython/jit/backend/zarch/regalloc.py +++ b/rpython/jit/backend/zarch/regalloc.py @@ -1215,3 +1215,41 @@ raise NotImplementedError(msg) prepare_oplist = [notimplemented] * (rop._LAST + 1) + +if not we_are_translated(): + implemented_count = 0 + total_count = 0 + missing = [] + for key, value in rop.__dict__.items(): + key = key.lower() + if key.startswith('_'): + continue + total_count += 1 + methname = 'prepare_%s' % key + if hasattr(Regalloc, methname): + func = getattr(Regalloc, methname).im_func + prepare_oplist[value] = func + implemented_count += 1 + else: + if not methname.startswith('prepare_vec') and \ + not methname.startswith('prepare_get') and \ + not methname.startswith('prepare_raw') and \ + not methname.startswith('prepare_unicodesetitem') and \ + not methname.startswith('prepare_unicodegetitem') and \ + not methname.startswith('prepare_strgetitem') and \ + not methname.startswith('prepare_strsetitem') and \ + not methname.startswith('prepare_call_loopinvariant') and \ + not methname.startswith('prepare_call_pure') and \ + not methname.startswith('prepare_new') and \ + not methname.startswith('prepare_set'): + missing.append(methname) + else: + implemented_count += 1 + + if __name__ == '__main__': + for m in missing: + print(" " * 4 + m) + print + print("regalloc implements %d of %d = %.2f%% of all resops" % \ + (implemented_count, total_count, (100.0 * implemented_count / total_count))) + diff --git a/rpython/translator/c/src/stacklet/switch_s390x_gcc.h b/rpython/translator/c/src/stacklet/switch_s390x_gcc.h --- a/rpython/translator/c/src/stacklet/switch_s390x_gcc.h +++ b/rpython/translator/c/src/stacklet/switch_s390x_gcc.h @@ -12,7 +12,10 @@ "stmg 6,15,48(15)\n" - "lay 15,-160(15)\n" /* create stack frame */ + "std 0,128(15)\n" + "std 2,136(15)\n" + "std 4,144(15)\n" + "std 6,152(15)\n" "lgr 10, %[restore_state]\n" /* save 'restore_state' for later */ "lgr 11, %[extra]\n" /* save 'extra' for later */ @@ -20,7 +23,9 @@ "lgr 2, 15\n" /* arg 1: current (old) stack pointer */ "lgr 3, 11\n" /* arg 2: extra */ + "lay 15,-160(15)\n" /* create stack frame */ "basr 14, 14\n" /* call save_state() */ + "lay 15, 160(15)\n" /* destroy stack frame */ "cgij 2, 0, 8, zero\n" /* skip the rest if the return value is null */ @@ -33,8 +38,10 @@ "lgr 3, 11\n" /* arg 2: extra */ + "lay 15, -160(15)\n" /* create temp stack space for callee to use */ "lgr 14, 10\n" /* load restore_state */ "basr 14, 14\n" /* call restore_state() */ + "lay 15, 160(15)\n" /* destroy temp stack space */ /* The stack's content is now restored. */ @@ -42,7 +49,13 @@ /* Epilogue */ /* no need */ /* restore stack pointer */ - "lmg 6,15,208(15)\n" + + "ld 0,128(15)\n" + "ld 2,136(15)\n" + "ld 4,144(15)\n" + "ld 6,152(15)\n" + + "lmg 6,15,48(15)\n" : "=r"(result) /* output variable: expected to be r2 */ : [restore_state]"r"(restore_state), /* input variables */ From pypy.commits at gmail.com Tue Jan 26 10:46:08 2016 From: pypy.commits at gmail.com (plan_rich) Date: Tue, 26 Jan 2016 07:46:08 -0800 (PST) Subject: [pypy-commit] pypy s390x-backend: merged head Message-ID: <56a794c0.8205c20a.b6a17.ffffc2da@mx.google.com> Author: Richard Plangger Branch: s390x-backend Changeset: r81950:3d78eb62b0e5 Date: 2016-01-26 16:44 +0100 http://bitbucket.org/pypy/pypy/changeset/3d78eb62b0e5/ Log: merged head diff --git a/rpython/jit/backend/zarch/assembler.py b/rpython/jit/backend/zarch/assembler.py --- a/rpython/jit/backend/zarch/assembler.py +++ b/rpython/jit/backend/zarch/assembler.py @@ -542,7 +542,6 @@ mc.restore_link() # So we return to our caller, conditionally if "EQ" mc.BCR(c.EQ, r.r14) - mc.trap() # debug if this is EVER executed! # # Else, jump to propagate_exception_path assert self.propagate_exception_path diff --git a/rpython/jit/backend/zarch/opassembler.py b/rpython/jit/backend/zarch/opassembler.py --- a/rpython/jit/backend/zarch/opassembler.py +++ b/rpython/jit/backend/zarch/opassembler.py @@ -682,11 +682,10 @@ def emit_guard_nonnull_class(self, op, arglocs, regalloc): self.mc.cmp_op(arglocs[0], l.imm(1), imm=True, signed=False) patch_pos = self.mc.currpos() - self.mc.trap() - self.mc.write('\x00' * 4) + self.mc.reserve_cond_jump(short=True) self._cmp_guard_class(op, arglocs, regalloc) pmc = OverwritingBuilder(self.mc, patch_pos, 1) - pmc.BRCL(c.LT, l.imm(self.mc.currpos() - patch_pos)) + pmc.BRC(c.LT, l.imm(self.mc.currpos() - patch_pos)) pmc.overwrite() self.guard_success_cc = c.EQ self._emit_guard(op, arglocs[2:]) @@ -716,9 +715,7 @@ def _cmp_guard_gc_type(self, loc_ptr, expected_typeid): self._read_typeid(r.SCRATCH2, loc_ptr) assert 0 <= expected_typeid <= 0x7fffffff # 4 bytes are always enough - if expected_typeid > 0xffff: # if 2 bytes are not enough - self.mc.AGHI(r.SCRATCH2, l.imm(-(expected_typeid >> 16))) - expected_typeid = expected_typeid & 0xffff + # we can handle 4 byte compare immediate self.mc.cmp_op(r.SCRATCH2, l.imm(expected_typeid), imm=True, signed=False) From pypy.commits at gmail.com Tue Jan 26 13:11:17 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 10:11:17 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: Copy more code from cpyext-gc-support Message-ID: <56a7b6c5.e218c20a.e673a.03b5@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81951:23ac4994b18d Date: 2016-01-26 17:44 +0100 http://bitbucket.org/pypy/pypy/changeset/23ac4994b18d/ Log: Copy more code from cpyext-gc-support diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -192,7 +192,7 @@ class ApiFunction: def __init__(self, argtypes, restype, callable, error=_NOT_SPECIFIED, - c_name=None, gil=None): + c_name=None, gil=None, result_borrowed=False): self.argtypes = argtypes self.restype = restype self.functype = lltype.Ptr(lltype.FuncType(argtypes, restype)) @@ -209,17 +209,15 @@ self.argnames = argnames[1:] assert len(self.argnames) == len(self.argtypes) self.gil = gil + self.result_borrowed = result_borrowed + # + def get_llhelper(space): + return llhelper(self.functype, self.get_wrapper(space)) + self.get_llhelper = get_llhelper def _freeze_(self): return True - def get_llhelper(self, space): - llh = getattr(self, '_llhelper', None) - if llh is None: - llh = llhelper(self.functype, self.get_wrapper(space)) - self._llhelper = llh - return llh - @specialize.memo() def get_wrapper(self, space): wrapper = getattr(self, '_wrapper', None) @@ -232,7 +230,7 @@ return wrapper def cpython_api(argtypes, restype, error=_NOT_SPECIFIED, external=True, - gil=None): + gil=None, result_borrowed=False): """ Declares a function to be exported. - `argtypes`, `restype` are lltypes and describe the function signature. @@ -261,13 +259,15 @@ rffi.cast(restype, 0) == 0) def decorate(func): + func._always_inline_ = 'try' func_name = func.func_name if external: c_name = None else: c_name = func_name api_function = ApiFunction(argtypes, restype, func, error, - c_name=c_name, gil=gil) + c_name=c_name, gil=gil, + result_borrowed=result_borrowed) func.api_func = api_function if external: @@ -278,6 +278,10 @@ raise ValueError("function %s has no return value for exceptions" % func) def make_unwrapper(catch_exception): + # ZZZ is this whole logic really needed??? It seems to be only + # for RPython code calling PyXxx() functions directly. I would + # think that usually directly calling the function is clean + # enough now names = api_function.argnames types_names_enum_ui = unrolling_iterable(enumerate( zip(api_function.argtypes, @@ -289,52 +293,55 @@ from pypy.module.cpyext.pyobject import make_ref, from_ref from pypy.module.cpyext.pyobject import Reference newargs = () - to_decref = [] + keepalives = () assert len(args) == len(api_function.argtypes) for i, (ARG, is_wrapped) in types_names_enum_ui: input_arg = args[i] if is_PyObject(ARG) and not is_wrapped: - # build a reference - if input_arg is None: - arg = lltype.nullptr(PyObject.TO) - elif isinstance(input_arg, W_Root): - ref = make_ref(space, input_arg) - to_decref.append(ref) - arg = rffi.cast(ARG, ref) + # build a 'PyObject *' (not holding a reference) + if not is_pyobj(input_arg): + keepalives += (input_arg,) + arg = rffi.cast(ARG, as_xpyobj(space, input_arg)) + else: + arg = rffi.cast(ARG, input_arg) + elif is_PyObject(ARG) and is_wrapped: + # build a W_Root, possibly from a 'PyObject *' + if is_pyobj(input_arg): + arg = from_pyobj(space, input_arg) else: arg = input_arg - elif is_PyObject(ARG) and is_wrapped: - # convert to a wrapped object - if input_arg is None: - arg = input_arg - elif isinstance(input_arg, W_Root): - arg = input_arg - else: - try: - arg = from_ref(space, - rffi.cast(PyObject, input_arg)) - except TypeError, e: - err = OperationError(space.w_TypeError, - space.wrap( - "could not cast arg to PyObject")) - if not catch_exception: - raise err - state = space.fromcache(State) - state.set_exception(err) - if is_PyObject(restype): - return None - else: - return api_function.error_value + + ## ZZZ: for is_pyobj: + ## try: + ## arg = from_ref(space, + ## rffi.cast(PyObject, input_arg)) + ## except TypeError, e: + ## err = OperationError(space.w_TypeError, + ## space.wrap( + ## "could not cast arg to PyObject")) + ## if not catch_exception: + ## raise err + ## state = space.fromcache(State) + ## state.set_exception(err) + ## if is_PyObject(restype): + ## return None + ## else: + ## return api_function.error_value else: - # convert to a wrapped object + # arg is not declared as PyObject, no magic arg = input_arg newargs += (arg, ) - try: + if not catch_exception: + try: + res = func(space, *newargs) + finally: + keepalive_until_here(*keepalives) + else: + # non-rpython variant + assert not we_are_translated() try: res = func(space, *newargs) except OperationError, e: - if not catch_exception: - raise if not hasattr(api_function, "error_value"): raise state = space.fromcache(State) @@ -343,21 +350,13 @@ return None else: return api_function.error_value - if not we_are_translated(): - got_integer = isinstance(res, (int, long, float)) - assert got_integer == expect_integer,'got %r not integer' % res - if res is None: - return None - elif isinstance(res, Reference): - return res.get_wrapped(space) - else: - return res - finally: - for arg in to_decref: - Py_DecRef(space, arg) + # 'keepalives' is alive here (it's not rpython) + got_integer = isinstance(res, (int, long, float)) + assert got_integer == expect_integer, ( + 'got %r not integer' % (res,)) + return res unwrapper.func = func unwrapper.api_func = api_function - unwrapper._always_inline_ = 'try' return unwrapper unwrapper_catch = make_unwrapper(True) @@ -625,6 +624,7 @@ llop.gc_stack_bottom(lltype.Void) # marker for trackgcroot.py retval = fatal_value boxed_args = () + tb = None try: if not we_are_translated() and DEBUG_WRAPPER: print >>sys.stderr, callable, @@ -632,10 +632,8 @@ for i, (typ, is_wrapped) in argtypes_enum_ui: arg = args[i] if is_PyObject(typ) and is_wrapped: - if arg: - arg_conv = from_ref(space, rffi.cast(PyObject, arg)) - else: - arg_conv = None + assert is_pyobj(arg) + arg_conv = from_pyobj(space, arg) else: arg_conv = arg boxed_args += (arg_conv, ) @@ -650,6 +648,7 @@ except BaseException, e: failed = True if not we_are_translated(): + tb = sys.exc_info()[2] message = repr(e) import traceback traceback.print_exc() @@ -668,29 +667,34 @@ retval = error_value elif is_PyObject(callable.api_func.restype): - if result is None: - retval = rffi.cast(callable.api_func.restype, - make_ref(space, None)) - elif isinstance(result, Reference): - retval = result.get_ref(space) - elif not rffi._isllptr(result): - retval = rffi.cast(callable.api_func.restype, - make_ref(space, result)) + if is_pyobj(result): + retval = result else: - retval = result + if result is None: + if callable.api_func.result_borrowed: + retval = as_pyobj(space, result) + else: + retval = get_pyobj_and_incref(space, result) + retval = rffi.cast(callable.api_func.restype, retval) + else: + retval = lltype.nullptr(PyObject.TO) elif callable.api_func.restype is not lltype.Void: retval = rffi.cast(callable.api_func.restype, result) except Exception, e: print 'Fatal error in cpyext, CPython compatibility layer, calling', callable.__name__ print 'Either report a bug or consider not using this particular extension' if not we_are_translated(): + if tb is None: + tb = sys.exc_info()[2] import traceback traceback.print_exc() - print str(e) + if sys.stdout == sys.__stdout__: + import pdb; pdb.post_mortem(tb) # we can't do much here, since we're in ctypes, swallow else: print str(e) pypy_debug_catch_fatal_exception() + assert False rffi.stackcounter.stacks_counter -= 1 if gil_release: rgil.release() @@ -843,7 +847,7 @@ # populate static data for name, (typ, expr) in GLOBALS.iteritems(): - from pypy.module import cpyext + from pypy.module import cpyext # for the eval() below w_obj = eval(expr) if name.endswith('#'): name = name[:-1] @@ -1204,20 +1208,15 @@ @specialize.ll() def generic_cpy_call(space, func, *args): FT = lltype.typeOf(func).TO - return make_generic_cpy_call(FT, True, False)(space, func, *args) - - at specialize.ll() -def generic_cpy_call_dont_decref(space, func, *args): - FT = lltype.typeOf(func).TO - return make_generic_cpy_call(FT, False, False)(space, func, *args) + return make_generic_cpy_call(FT, False)(space, func, *args) @specialize.ll() def generic_cpy_call_expect_null(space, func, *args): FT = lltype.typeOf(func).TO - return make_generic_cpy_call(FT, True, True)(space, func, *args) + return make_generic_cpy_call(FT, True)(space, func, *args) @specialize.memo() -def make_generic_cpy_call(FT, decref_args, expect_null): +def make_generic_cpy_call(FT, expect_null): from pypy.module.cpyext.pyobject import make_ref, from_ref, Py_DecRef from pypy.module.cpyext.pyobject import RefcountState from pypy.module.cpyext.pyerrors import PyErr_Occurred @@ -1247,65 +1246,49 @@ @specialize.ll() def generic_cpy_call(space, func, *args): boxed_args = () - to_decref = [] + keepalives = () assert len(args) == len(FT.ARGS) for i, ARG in unrolling_arg_types: arg = args[i] if is_PyObject(ARG): - if arg is None: - boxed_args += (lltype.nullptr(PyObject.TO),) - elif isinstance(arg, W_Root): - ref = make_ref(space, arg) - boxed_args += (ref,) - if decref_args: - to_decref.append(ref) - else: - boxed_args += (arg,) - else: - boxed_args += (arg,) + if not is_pyobj(arg): + keepalives += (arg,) + arg = as_pyobj(space, arg) + boxed_args += (arg,) try: - # create a new container for borrowed references - state = space.fromcache(RefcountState) - old_container = state.swap_borrow_container(None) - try: - # Call the function - result = call_external_function(func, *boxed_args) - finally: - state.swap_borrow_container(old_container) + # Call the function + result = call_external_function(func, *boxed_args) + finally: + keepalive_until_here(*keepalives) - if is_PyObject(RESULT_TYPE): - if result is None: - ret = result - elif isinstance(result, W_Root): - ret = result + if is_PyObject(RESULT_TYPE): + if not is_pyobj(result): + ret = result + else: + # The object reference returned from a C function + # that is called from Python must be an owned reference + # - ownership is transferred from the function to its caller. + if result: + ret = get_w_obj_and_decref(space, result) else: - ret = from_ref(space, result) - # The object reference returned from a C function - # that is called from Python must be an owned reference - # - ownership is transferred from the function to its caller. - if result: - Py_DecRef(space, result) + ret = None - # Check for exception consistency - has_error = PyErr_Occurred(space) is not None - has_result = ret is not None - if has_error and has_result: - raise OperationError(space.w_SystemError, space.wrap( - "An exception was set, but function returned a value")) - elif not expect_null and not has_error and not has_result: - raise OperationError(space.w_SystemError, space.wrap( - "Function returned a NULL result without setting an exception")) + # Check for exception consistency + has_error = PyErr_Occurred(space) is not None + has_result = ret is not None + if has_error and has_result: + raise OperationError(space.w_SystemError, space.wrap( + "An exception was set, but function returned a value")) + elif not expect_null and not has_error and not has_result: + raise OperationError(space.w_SystemError, space.wrap( + "Function returned a NULL result without setting an exception")) - if has_error: - state = space.fromcache(State) - state.check_and_raise_exception() + if has_error: + state = space.fromcache(State) + state.check_and_raise_exception() - return ret - return result - finally: - if decref_args: - for ref in to_decref: - Py_DecRef(space, ref) + return ret + return result + return generic_cpy_call - diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -150,6 +150,7 @@ Allocates a PyObject, and fills its fields with info from the given intepreter object. """ + GOES_AWAY state = space.fromcache(RefcountState) w_type = space.type(w_obj) if w_type.is_cpytype(): @@ -169,6 +170,7 @@ """ Ties together a PyObject and an interpreter object. """ + GOES_AWAY # XXX looks like a PyObject_GC_TRACK ptr = rffi.cast(ADDR, py_obj) state = space.fromcache(RefcountState) @@ -185,6 +187,7 @@ """ Returns a new reference to an intepreter object. """ + GOES_AWAY if w_obj is None: return lltype.nullptr(PyObject.TO) assert isinstance(w_obj, W_Root) @@ -204,6 +207,7 @@ Finds the interpreter object corresponding to the given reference. If the object is not yet realized (see stringobject.py), creates it. """ + GOES_AWAY assert lltype.typeOf(ref) == PyObject if not ref: return None @@ -228,6 +232,7 @@ # XXX Optimize these functions and put them into macro definitions @cpython_api([PyObject], lltype.Void) def Py_DecRef(space, obj): + ZZZ if not obj: return assert lltype.typeOf(obj) == PyObject @@ -284,3 +289,11 @@ @cpython_api([rffi.VOIDP], lltype.Signed, error=CANNOT_FAIL) def _Py_HashPointer(space, ptr): return rffi.cast(lltype.Signed, ptr) + + +class RefcountState: + def __init__(self, *args): + GOES_AWAY + +def borrow_from(container, borrowed): + GOES_AWAY From pypy.commits at gmail.com Tue Jan 26 13:11:19 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 10:11:19 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: more copying Message-ID: <56a7b6c7.cf0b1c0a.4d6ce.719b@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81952:a6a694f2a019 Date: 2016-01-26 17:51 +0100 http://bitbucket.org/pypy/pypy/changeset/a6a694f2a019/ Log: more copying diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -17,7 +17,8 @@ #define staticforward static #define PyObject_HEAD \ - long ob_refcnt; \ + Py_ssize_t ob_refcnt; \ + Py_ssize_t ob_pypy_link; \ struct _typeobject *ob_type; #define PyObject_VAR_HEAD \ @@ -25,7 +26,7 @@ Py_ssize_t ob_size; /* Number of items in variable part */ #define PyObject_HEAD_INIT(type) \ - 1, type, + 1, 0, type, #define PyVarObject_HEAD_INIT(type, size) \ PyObject_HEAD_INIT(type) size, @@ -40,19 +41,19 @@ #ifdef PYPY_DEBUG_REFCOUNT /* Slow version, but useful for debugging */ -#define Py_INCREF(ob) (Py_IncRef((PyObject *)ob)) -#define Py_DECREF(ob) (Py_DecRef((PyObject *)ob)) -#define Py_XINCREF(ob) (Py_IncRef((PyObject *)ob)) -#define Py_XDECREF(ob) (Py_DecRef((PyObject *)ob)) +#define Py_INCREF(ob) (Py_IncRef((PyObject *)(ob))) +#define Py_DECREF(ob) (Py_DecRef((PyObject *)(ob))) +#define Py_XINCREF(ob) (Py_IncRef((PyObject *)(ob))) +#define Py_XDECREF(ob) (Py_DecRef((PyObject *)(ob))) #else /* Fast version */ -#define Py_INCREF(ob) (((PyObject *)ob)->ob_refcnt++) -#define Py_DECREF(ob) \ +#define Py_INCREF(ob) (((PyObject *)(ob))->ob_refcnt++) +#define Py_DECREF(op) \ do { \ - if (((PyObject *)ob)->ob_refcnt > 1) \ - ((PyObject *)ob)->ob_refcnt--; \ + if (--((PyObject *)(op))->ob_refcnt != 0) \ + ; \ else \ - Py_DecRef((PyObject *)ob); \ + _Py_Dealloc((PyObject *)(op)); \ } while (0) #define Py_XINCREF(op) do { if ((op) == NULL) ; else Py_INCREF(op); } while (0) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -134,6 +134,9 @@ #________________________________________________________ # refcounted object support +class InvalidPointerException(Exception): + pass + DEBUG_REFCOUNT = False def debug_refcount(*args, **kwargs): @@ -229,48 +232,13 @@ return get_typedescr(w_type.instancetypedef).realize(space, ref) -# XXX Optimize these functions and put them into macro definitions + at cpython_api([PyObject], lltype.Void) +def Py_IncRef(space, obj): + incref(obj) + @cpython_api([PyObject], lltype.Void) def Py_DecRef(space, obj): - ZZZ - if not obj: - return - assert lltype.typeOf(obj) == PyObject - - obj.c_ob_refcnt -= 1 - if DEBUG_REFCOUNT: - debug_refcount("DECREF", obj, obj.c_ob_refcnt, frame_stackdepth=3) - if obj.c_ob_refcnt == 0: - state = space.fromcache(RefcountState) - ptr = rffi.cast(ADDR, obj) - if ptr not in state.py_objects_r2w: - # this is a half-allocated object, lets call the deallocator - # without modifying the r2w/w2r dicts - _Py_Dealloc(space, obj) - else: - w_obj = state.py_objects_r2w[ptr] - del state.py_objects_r2w[ptr] - w_type = space.type(w_obj) - if not w_type.is_cpytype(): - _Py_Dealloc(space, obj) - del state.py_objects_w2r[w_obj] - # if the object was a container for borrowed references - state.delete_borrower(w_obj) - else: - if not we_are_translated() and obj.c_ob_refcnt < 0: - message = "Negative refcount for obj %s with type %s" % ( - obj, rffi.charp2str(obj.c_ob_type.c_tp_name)) - print >>sys.stderr, message - assert False, message - - at cpython_api([PyObject], lltype.Void) -def Py_IncRef(space, obj): - if not obj: - return - obj.c_ob_refcnt += 1 - assert obj.c_ob_refcnt > 0 - if DEBUG_REFCOUNT: - debug_refcount("INCREF", obj, obj.c_ob_refcnt, frame_stackdepth=3) + decref(space, obj) @cpython_api([PyObject], lltype.Void) def _Py_NewReference(space, obj): @@ -279,6 +247,7 @@ assert isinstance(w_type, W_TypeObject) get_typedescr(w_type.instancetypedef).realize(space, obj) + at cpython_api([PyObject], lltype.Void) def _Py_Dealloc(space, obj): from pypy.module.cpyext.api import generic_cpy_call_dont_decref pto = obj.c_ob_type From pypy.commits at gmail.com Tue Jan 26 13:11:21 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 10:11:21 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: in-progress Message-ID: <56a7b6c9.4c301c0a.ac10c.6936@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81953:3d1f9e7d01e1 Date: 2016-01-26 19:10 +0100 http://bitbucket.org/pypy/pypy/changeset/3d1f9e7d01e1/ Log: in-progress diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -2,9 +2,11 @@ from pypy.interpreter.baseobjspace import W_Root, SpaceCache from rpython.rtyper.lltypesystem import rffi, lltype +from rpython.rtyper.extregistry import ExtRegistryEntry from pypy.module.cpyext.api import ( cpython_api, bootstrap_function, PyObject, PyObjectP, ADDR, - CANNOT_FAIL, Py_TPFLAGS_HEAPTYPE, PyTypeObjectPtr) + CANNOT_FAIL, Py_TPFLAGS_HEAPTYPE, PyTypeObjectPtr, + INTERPLEVEL_API) from pypy.module.cpyext.state import State from pypy.objspace.std.typeobject import W_TypeObject from pypy.objspace.std.objectobject import W_ObjectObject @@ -151,10 +153,9 @@ def create_ref(space, w_obj, itemcount=0): """ Allocates a PyObject, and fills its fields with info from the given - intepreter object. + interpreter object. """ - GOES_AWAY - state = space.fromcache(RefcountState) + #state = space.fromcache(RefcountState) w_type = space.type(w_obj) if w_type.is_cpytype(): py_obj = state.get_from_lifeline(w_obj) @@ -173,18 +174,16 @@ """ Ties together a PyObject and an interpreter object. """ - GOES_AWAY # XXX looks like a PyObject_GC_TRACK - ptr = rffi.cast(ADDR, py_obj) - state = space.fromcache(RefcountState) + assert py_obj.c_ob_refcnt < rawrefcount.REFCNT_FROM_PYPY + py_obj.c_ob_refcnt += rawrefcount.REFCNT_FROM_PYPY if DEBUG_REFCOUNT: debug_refcount("MAKREF", py_obj, w_obj) + assert w_obj + assert py_obj if not replace: assert w_obj not in state.py_objects_w2r - assert ptr not in state.py_objects_r2w - state.py_objects_w2r[w_obj] = py_obj - if ptr: # init_typeobject() bootstraps with NULL references - state.py_objects_r2w[ptr] = w_obj + rawrefcount.create_link_pypy(py_obj, w_obj) def make_ref(space, w_obj): """ @@ -232,6 +231,124 @@ return get_typedescr(w_type.instancetypedef).realize(space, ref) +def debug_collect(): + rawrefcount._collect(track_allocation=False) + + +def as_pyobj(space, w_obj): + """ + Returns a 'PyObject *' representing the given intepreter object. + This doesn't give a new reference, but the returned 'PyObject *' + is valid at least as long as 'w_obj' is. To be safe, you should + use keepalive_until_here(w_obj) some time later. + + NOTE: get_pyobj_and_incref() is safer. + """ + if w_obj is not None: + assert not is_pyobj(w_obj) + return XXXXXXXXXXX + else: + return lltype.nullptr(PyObject.TO) +as_pyobj._always_inline_ = 'try' +INTERPLEVEL_API['as_pyobj'] = as_pyobj + +def pyobj_has_w_obj(pyobj): + return rawrefcount.to_obj(W_Root, pyobj) is not None +INTERPLEVEL_API['pyobj_has_w_obj'] = staticmethod(pyobj_has_w_obj) + + at specialize.ll() +def from_pyobj(space, pyobj): + assert is_pyobj(pyobj) + if pyobj: + pyobj = rffi.cast(PyObject, pyobj) + w_obj = rawrefcount.to_obj(W_Root, pyobj) + if w_obj is None: + XXXXXXXXXXX + return w_obj + else: + return None +from_pyobj._always_inline_ = 'try' +INTERPLEVEL_API['from_pyobj'] = from_pyobj + + +def is_pyobj(x): + if x is None or isinstance(x, W_Root): + return False + elif is_PyObject(lltype.typeOf(x)): + return True + else: + raise TypeError(repr(type(x))) +INTERPLEVEL_API['is_pyobj'] = staticmethod(is_pyobj) + +class Entry(ExtRegistryEntry): + _about_ = is_pyobj + def compute_result_annotation(self, s_x): + from rpython.rtyper.llannotation import SomePtr + return self.bookkeeper.immutablevalue(isinstance(s_x, SomePtr)) + def specialize_call(self, hop): + hop.exception_cannot_occur() + return hop.inputconst(lltype.Bool, hop.s_result.const) + + at specialize.ll() +def get_pyobj_and_incref(space, obj): + """Increment the reference counter of the PyObject and return it. + Can be called with either a PyObject or a W_Root. + """ + if obj: + if is_pyobj(obj): + pyobj = rffi.cast(PyObject, obj) + else: + pyobj = as_pyobj(space, obj) + assert pyobj.c_ob_refcnt > 0 + pyobj.c_ob_refcnt += 1 + if not is_pyobj(obj): + keepalive_until_here(obj) + return pyobj + else: + return lltype.nullptr(PyObject.TO) +INTERPLEVEL_API['get_pyobj_and_incref'] = get_pyobj_and_incref + + + at specialize.ll() +def get_w_obj_and_decref(space, obj): + """Decrement the reference counter of the PyObject and return the + corresponding W_Root object (so the reference count is at least + REFCNT_FROM_PYPY and cannot be zero). Can be called with either + a PyObject or a W_Root. + """ + if is_pyobj(obj): + pyobj = rffi.cast(PyObject, obj) + w_obj = from_pyobj(space, pyobj) + else: + w_obj = obj + pyobj = as_pyobj(space, w_obj) + if pyobj: + pyobj.c_ob_refcnt -= 1 + assert pyobj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY + keepalive_until_here(w_obj) + return w_obj +INTERPLEVEL_API['get_w_obj_and_decref'] = get_w_obj_and_decref + + + at specialize.ll() +def incref(space, obj): + get_pyobj_and_incref(space, obj) +INTERPLEVEL_API['incref'] = incref + + at specialize.ll() +def decref(space, obj): + if is_pyobj(obj): + obj = rffi.cast(PyObject, obj) + if obj: + assert obj.c_ob_refcnt > 0 + obj.c_ob_refcnt -= 1 + if obj.c_ob_refcnt == 0: + _Py_Dealloc(space, obj) + else: + get_w_obj_and_decref(space, obj) +INTERPLEVEL_API['decref'] = decref + + @cpython_api([PyObject], lltype.Void) def Py_IncRef(space, obj): incref(obj) diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -21,7 +21,8 @@ from pypy.module.cpyext.modsupport import convert_method_defs from pypy.module.cpyext.pyobject import ( PyObject, make_ref, create_ref, from_ref, get_typedescr, make_typedescr, - track_reference, RefcountState, borrow_from, Py_DecRef) + track_reference, RefcountState, borrow_from, Py_DecRef, + get_pyobj_and_incref) from pypy.module.cpyext.slotdefs import ( slotdefs_for_tp_slots, slotdefs_for_wrappers, get_slot_tp_function) from pypy.module.cpyext.state import State @@ -320,13 +321,13 @@ # - object.tp_bases is a tuple # - tuple.tp_bases is a tuple - # insert null placeholders to please create_ref() - track_reference(space, lltype.nullptr(PyObject.TO), space.w_type) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_object) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_tuple) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_str) - - # create the objects + # we create the types manually here + py_type = _type_alloc(space, lltype.nullptr(PyTypeObject)) + py_object = _type_alloc(space, lltype.nullptr(PyTypeObject)) + py_tuple = _type_alloc(space, lltype.nullptr(PyTypeObject)) + py_str = _type_alloc(space, lltype.nullptr(PyTypeObject)) + ... + py_type = create_ref(space, space.w_type) py_object = create_ref(space, space.w_object) py_tuple = create_ref(space, space.w_tuple) @@ -459,13 +460,16 @@ def type_alloc(space, w_metatype): - metatype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_metatype)) + metatype = get_pyobj_and_incref(space, w_metatype) + metatype = rffi.cast(PyTypeObjectPtr, metatype) + assert metatype # Don't increase refcount for non-heaptypes - if metatype: - flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) - if not flags & Py_TPFLAGS_HEAPTYPE: - Py_DecRef(space, w_metatype) + flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) + if not flags & Py_TPFLAGS_HEAPTYPE: + Py_DecRef(space, w_metatype) + return _type_alloc(space, metatype) +def _type_alloc(space, metatype): heaptype = lltype.malloc(PyHeapTypeObject.TO, flavor='raw', zero=True) pto = heaptype.c_ht_type From pypy.commits at gmail.com Tue Jan 26 15:39:29 2016 From: pypy.commits at gmail.com (rlamy) Date: Tue, 26 Jan 2016 12:39:29 -0800 (PST) Subject: [pypy-commit] pypy exctrans: Close branch exctrans Message-ID: <56a7d981.d69c1c0a.90d26.ffffa6c3@mx.google.com> Author: Ronan Lamy Branch: exctrans Changeset: r81954:e953e5f78446 Date: 2016-01-26 20:39 +0000 http://bitbucket.org/pypy/pypy/changeset/e953e5f78446/ Log: Close branch exctrans From pypy.commits at gmail.com Tue Jan 26 15:39:44 2016 From: pypy.commits at gmail.com (rlamy) Date: Tue, 26 Jan 2016 12:39:44 -0800 (PST) Subject: [pypy-commit] pypy default: Merged in exctrans (pull request #390) Message-ID: <56a7d990.c177c20a.46fc9.3c56@mx.google.com> Author: Ronan Lamy Branch: Changeset: r81955:467f4a616ad1 Date: 2016-01-26 20:39 +0000 http://bitbucket.org/pypy/pypy/changeset/467f4a616ad1/ Log: Merged in exctrans (pull request #390) Some refactoring of databasing diff --git a/rpython/memory/gctransform/boehm.py b/rpython/memory/gctransform/boehm.py --- a/rpython/memory/gctransform/boehm.py +++ b/rpython/memory/gctransform/boehm.py @@ -74,7 +74,7 @@ def gct_fv_gc_malloc_varsize(self, hop, flags, TYPE, v_length, c_const_size, c_item_size, c_offset_to_length): - # XXX same behavior for zero=True: in theory that's wrong + # XXX same behavior for zero=True: in theory that's wrong if c_offset_to_length is None: v_raw = hop.genop("direct_call", [self.malloc_varsize_no_length_ptr, v_length, @@ -156,6 +156,11 @@ resulttype = lltype.Signed) hop.genop('int_invert', [v_int], resultvar=hop.spaceop.result) + def gcheader_initdata(self, defnode): + hdr = lltype.malloc(self.HDR, immortal=True) + hdr.hash = lltype.identityhash_nocache(defnode.obj._as_ptr()) + return hdr._obj + ########## weakrefs ########## # Boehm: weakref objects are small structures containing only a Boehm diff --git a/rpython/memory/gctransform/framework.py b/rpython/memory/gctransform/framework.py --- a/rpython/memory/gctransform/framework.py +++ b/rpython/memory/gctransform/framework.py @@ -288,7 +288,6 @@ s_gcref = SomePtr(llmemory.GCREF) gcdata = self.gcdata - translator = self.translator #use the GC flag to find which malloc method to use #malloc_zero_filled == Ture -> malloc_fixedsize/varsize_clear #malloc_zero_filled == Flase -> malloc_fixedsize/varsize @@ -322,7 +321,7 @@ GCClass.malloc_varsize.im_func, [s_gc, s_typeid16] + [annmodel.SomeInteger(nonneg=True) for i in range(4)], s_gcref) - + self.collect_ptr = getfn(GCClass.collect.im_func, [s_gc, annmodel.SomeInteger()], annmodel.s_None) self.can_move_ptr = getfn(GCClass.can_move.im_func, @@ -1385,7 +1384,7 @@ [v] + previous_steps + [c_name, c_null]) else: llops.genop('bare_setfield', [v, c_name, c_null]) - + return elif isinstance(TYPE, lltype.Array): ITEM = TYPE.OF @@ -1412,6 +1411,25 @@ resulttype=llmemory.Address) llops.genop('raw_memclear', [v_adr, v_totalsize]) + def gcheader_initdata(self, defnode): + o = lltype.top_container(defnode.obj) + needs_hash = self.get_prebuilt_hash(o) is not None + hdr = self.gc_header_for(o, needs_hash) + return hdr._obj + + def get_prebuilt_hash(self, obj): + # for prebuilt objects that need to have their hash stored and + # restored. Note that only structures that are StructNodes all + # the way have their hash stored (and not e.g. structs with var- + # sized arrays at the end). 'obj' must be the top_container. + TYPE = lltype.typeOf(obj) + if not isinstance(TYPE, lltype.GcStruct): + return None + if TYPE._is_varsize(): + return None + return getattr(obj, '_hash_cache_', None) + + class TransformerLayoutBuilder(gctypelayout.TypeLayoutBuilder): diff --git a/rpython/memory/gctransform/refcounting.py b/rpython/memory/gctransform/refcounting.py --- a/rpython/memory/gctransform/refcounting.py +++ b/rpython/memory/gctransform/refcounting.py @@ -285,3 +285,7 @@ resulttype=llmemory.Address) hop.genop("direct_call", [self.identityhash_ptr, v_adr], resultvar=hop.spaceop.result) + + def gcheader_initdata(self, defnode): + top = lltype.top_container(defnode.obj) + return self.gcheaderbuilder.header_of_object(top)._obj diff --git a/rpython/memory/gctransform/test/test_framework.py b/rpython/memory/gctransform/test/test_framework.py --- a/rpython/memory/gctransform/test/test_framework.py +++ b/rpython/memory/gctransform/test/test_framework.py @@ -40,7 +40,7 @@ t.config.translation.gc = "minimark" cbuild = CStandaloneBuilder(t, entrypoint, t.config, gcpolicy=FrameworkGcPolicy2) - db = cbuild.generate_graphs_for_llinterp() + db = cbuild.build_database() entrypointptr = cbuild.getentrypointptr() entrygraph = entrypointptr._obj.graph @@ -69,7 +69,7 @@ return -x t = rtype(g, [int]) gg = graphof(t, g) - assert not CollectAnalyzer(t).analyze_direct_call(gg) + assert not CollectAnalyzer(t).analyze_direct_call(gg) def test_cancollect_external(): fext1 = rffi.llexternal('fext1', [], lltype.Void, releasegil=False) @@ -110,12 +110,12 @@ def entrypoint(argv): return g() + 2 - + t = rtype(entrypoint, [s_list_of_strings]) t.config.translation.gc = "minimark" cbuild = CStandaloneBuilder(t, entrypoint, t.config, gcpolicy=FrameworkGcPolicy2) - db = cbuild.generate_graphs_for_llinterp() + db = cbuild.build_database() def test_no_collect_detection(): from rpython.rlib import rgc @@ -134,12 +134,13 @@ def entrypoint(argv): return g() + 2 - + t = rtype(entrypoint, [s_list_of_strings]) t.config.translation.gc = "minimark" cbuild = CStandaloneBuilder(t, entrypoint, t.config, gcpolicy=FrameworkGcPolicy2) - f = py.test.raises(Exception, cbuild.generate_graphs_for_llinterp) + with py.test.raises(Exception) as f: + cbuild.build_database() expected = "'no_collect' function can trigger collection: 0: size = rffi.cast(rffi.SIZE_T, length) @@ -85,15 +85,24 @@ return loader def reraise_error(error, loader): - if error == 1: raise OSError(load_int(loader), "external error") - elif error == 2: raise IOError - elif error == 3: raise OverflowError - elif error == 4: raise ValueError - elif error == 5: raise ZeroDivisionError - elif error == 6: raise MemoryError - elif error == 7: raise KeyError - elif error == 8: raise IndexError - else: raise RuntimeError + if error == 1: + raise OSError(load_int(loader), "external error") + elif error == 2: + raise IOError + elif error == 3: + raise OverflowError + elif error == 4: + raise ValueError + elif error == 5: + raise ZeroDivisionError + elif error == 6: + raise MemoryError + elif error == 7: + raise KeyError + elif error == 8: + raise IndexError + else: + raise RuntimeError @signature(types.str(), returns=types.impossible()) @@ -101,51 +110,46 @@ STDERR = 2 with rffi.scoped_str2charp(msg + '\n') as buf: writeall_not_sandboxed(STDERR, buf, len(msg) + 1) - raise RuntimeError(msg) # XXX in RPython, the msg is ignored at the moment + raise RuntimeError(msg) # XXX in RPython, the msg is ignored + +def make_stub(fnname, msg): + """Build always-raising stub function to replace unsupported external.""" + log.WARNING(msg) + + def execute(*args): + not_implemented_stub(msg) + execute.__name__ = 'sandboxed_%s' % (fnname,) + return execute + +def sig_ll(fnobj): + FUNCTYPE = lltype.typeOf(fnobj) + args_s = [lltype_to_annotation(ARG) for ARG in FUNCTYPE.ARGS] + s_result = lltype_to_annotation(FUNCTYPE.RESULT) + return args_s, s_result dump_string = rmarshal.get_marshaller(str) -load_int = rmarshal.get_loader(int) +load_int = rmarshal.get_loader(int) -def get_external_function_sandbox_graph(fnobj, db, force_stub=False): - """Build the graph of a helper trampoline function to be used - in place of real calls to the external function 'fnobj'. The - trampoline marshals its input arguments, dumps them to STDOUT, - and waits for an answer on STDIN. +def get_sandbox_stub(fnobj, rtyper): + fnname = fnobj._name + args_s, s_result = sig_ll(fnobj) + msg = "Not implemented: sandboxing for external function '%s'" % (fnname,) + execute = make_stub(fnname, msg) + return _annotate(rtyper, execute, args_s, s_result) + +def make_sandbox_trampoline(fnname, args_s, s_result): + """Create a trampoline function with the specified signature. + + The trampoline is meant to be used in place of real calls to the external + function named 'fnname'. It marshals its input arguments, dumps them to + STDOUT, and waits for an answer on STDIN. """ - if getattr(getattr(fnobj, '_callable', None), - '_sandbox_external_name', None): - fnname = fnobj._callable._sandbox_external_name - else: - fnname = fnobj._name - if hasattr(fnobj, 'graph'): - # get the annotation of the input arguments and the result - graph = fnobj.graph - annotator = db.translator.annotator - args_s = [annotator.binding(v) for v in graph.getargs()] - s_result = annotator.binding(graph.getreturnvar()) - else: - # pure external function - fall back to the annotations - # corresponding to the ll types - FUNCTYPE = lltype.typeOf(fnobj) - args_s = [lltype_to_annotation(ARG) for ARG in FUNCTYPE.ARGS] - s_result = lltype_to_annotation(FUNCTYPE.RESULT) - try: - if force_stub: # old case - don't try to support suggested_primitive - raise NotImplementedError("sandboxing for external function '%s'" - % (fnname,)) - dump_arguments = rmarshal.get_marshaller(tuple(args_s)) load_result = rmarshal.get_loader(s_result) - - except (NotImplementedError, - rmarshal.CannotMarshal, - rmarshal.CannotUnmarshall), e: - msg = 'Not Implemented: %s' % (e,) - log.WARNING(msg) - def execute(*args): - not_implemented_stub(msg) - + except (rmarshal.CannotMarshal, rmarshal.CannotUnmarshall) as e: + msg = "Cannot sandbox function '%s': %s" % (fnname, e) + execute = make_stub(fnname, msg) else: def execute(*args): # marshal the function name and input arguments @@ -158,9 +162,12 @@ result = load_result(loader) loader.check_finished() return result - execute = func_with_new_name(execute, 'sandboxed_' + fnname) + execute.__name__ = 'sandboxed_%s' % (fnname,) + return execute - ann = MixLevelHelperAnnotator(db.translator.rtyper) - graph = ann.getgraph(execute, args_s, s_result) + +def _annotate(rtyper, f, args_s, s_result): + ann = MixLevelHelperAnnotator(rtyper) + graph = ann.getgraph(f, args_s, s_result) ann.finish() return graph From pypy.commits at gmail.com Tue Jan 26 17:36:36 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 14:36:36 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: in-progress Message-ID: <56a7f4f4.c177c20a.46fc9.5df3@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81956:20a0b177ddae Date: 2016-01-26 23:10 +0100 http://bitbucket.org/pypy/pypy/changeset/20a0b177ddae/ Log: in-progress diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -37,6 +37,7 @@ from py.builtin import BaseException from rpython.tool.sourcetools import func_with_new_name from rpython.rtyper.lltypesystem.lloperation import llop +from rpython.rlib import rawrefcount DEBUG_WRAPPER = True @@ -825,14 +826,6 @@ outputfilename=str(udir / "module_cache" / "pypyapi")) modulename = py.path.local(eci.libraries[-1]) - run_bootstrap_functions(space) - - # load the bridge, and init structure - import ctypes - bridge = ctypes.CDLL(str(modulename), mode=ctypes.RTLD_GLOBAL) - - space.fromcache(State).install_dll(eci) - def dealloc_trigger(): print 'dealloc_trigger...' while True: @@ -845,6 +838,14 @@ return "RETRY" rawrefcount.init(dealloc_trigger) + run_bootstrap_functions(space) + + # load the bridge, and init structure + import ctypes + bridge = ctypes.CDLL(str(modulename), mode=ctypes.RTLD_GLOBAL) + + space.fromcache(State).install_dll(eci) + # populate static data for name, (typ, expr) in GLOBALS.iteritems(): from pypy.module import cpyext # for the eval() below diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -13,6 +13,8 @@ from rpython.rlib.objectmodel import specialize, we_are_translated from rpython.rlib.rweakref import RWeakKeyDictionary from rpython.rtyper.annlowlevel import llhelper +from rpython.rlib import rawrefcount + #________________________________________________________ # type description @@ -31,12 +33,13 @@ # similar to PyType_GenericAlloc? # except that it's not related to any pypy object. - pytype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_type)) + pytype = get_pyobj_and_incref(space, w_type) + pytype = rffi.cast(PyTypeObjectPtr, pytype) + assert pytype # Don't increase refcount for non-heaptypes - if pytype: - flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) - if not flags & Py_TPFLAGS_HEAPTYPE: - Py_DecRef(space, w_type) + flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) + if not flags & Py_TPFLAGS_HEAPTYPE: + Py_DecRef(space, w_type) if pytype: size = pytype.c_tp_basicsize @@ -158,19 +161,18 @@ #state = space.fromcache(RefcountState) w_type = space.type(w_obj) if w_type.is_cpytype(): - py_obj = state.get_from_lifeline(w_obj) + ZZZ # py_obj = state.get_from_lifeline(w_obj) if py_obj: Py_IncRef(space, py_obj) return py_obj typedescr = get_typedescr(w_obj.typedef) py_obj = typedescr.allocate(space, w_type, itemcount=itemcount) - if w_type.is_cpytype(): - state.set_lifeline(w_obj, py_obj) + track_reference(space, py_obj, w_obj) typedescr.attach(space, py_obj, w_obj) return py_obj -def track_reference(space, py_obj, w_obj, replace=False): +def track_reference(space, py_obj, w_obj): """ Ties together a PyObject and an interpreter object. """ @@ -181,8 +183,6 @@ debug_refcount("MAKREF", py_obj, w_obj) assert w_obj assert py_obj - if not replace: - assert w_obj not in state.py_objects_w2r rawrefcount.create_link_pypy(py_obj, w_obj) def make_ref(space, w_obj): @@ -239,14 +239,17 @@ """ Returns a 'PyObject *' representing the given intepreter object. This doesn't give a new reference, but the returned 'PyObject *' - is valid at least as long as 'w_obj' is. To be safe, you should - use keepalive_until_here(w_obj) some time later. - - NOTE: get_pyobj_and_incref() is safer. + is valid at least as long as 'w_obj' is. **To be safe, you should + use keepalive_until_here(w_obj) some time later.** In case of + doubt, use the safer get_pyobj_and_incref(). """ if w_obj is not None: assert not is_pyobj(w_obj) - return XXXXXXXXXXX + py_obj = rawrefcount.from_obj(PyObject, w_obj) + if not py_obj: + py_obj = create_ref(space, w_obj) + #track_reference(space, py_obj, w_obj) -- included with create_ref() + return py_obj else: return lltype.nullptr(PyObject.TO) as_pyobj._always_inline_ = 'try' diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -311,54 +311,12 @@ realize=type_realize, dealloc=type_dealloc) - # some types are difficult to create because of cycles. - # - object.ob_type = type - # - type.ob_type = type - # - tuple.ob_type = type - # - type.tp_base = object - # - tuple.tp_base = object - # - type.tp_bases is a tuple - # - object.tp_bases is a tuple - # - tuple.tp_bases is a tuple - - # we create the types manually here + # we create the type "type" manually here, because of the cycle + # through its 'c_ob_type' field py_type = _type_alloc(space, lltype.nullptr(PyTypeObject)) - py_object = _type_alloc(space, lltype.nullptr(PyTypeObject)) - py_tuple = _type_alloc(space, lltype.nullptr(PyTypeObject)) - py_str = _type_alloc(space, lltype.nullptr(PyTypeObject)) - ... - - py_type = create_ref(space, space.w_type) - py_object = create_ref(space, space.w_object) - py_tuple = create_ref(space, space.w_tuple) - py_str = create_ref(space, space.w_str) - # XXX py_str is not initialized here correctly, because we are - # not tracking it, it gets an empty c_ob_type from py_basestring - - # form cycles - pto_type = rffi.cast(PyTypeObjectPtr, py_type) - py_type.c_ob_type = pto_type - py_object.c_ob_type = pto_type - py_tuple.c_ob_type = pto_type - - pto_object = rffi.cast(PyTypeObjectPtr, py_object) - pto_type.c_tp_base = pto_object - pto_tuple = rffi.cast(PyTypeObjectPtr, py_tuple) - pto_tuple.c_tp_base = pto_object - - pto_type.c_tp_bases.c_ob_type = pto_tuple - pto_object.c_tp_bases.c_ob_type = pto_tuple - pto_tuple.c_tp_bases.c_ob_type = pto_tuple - - for typ in (py_type, py_object, py_tuple, py_str): - heaptype = rffi.cast(PyHeapTypeObject, typ) - heaptype.c_ht_name.c_ob_type = pto_type - - # Restore the mapping - track_reference(space, py_type, space.w_type, replace=True) - track_reference(space, py_object, space.w_object, replace=True) - track_reference(space, py_tuple, space.w_tuple, replace=True) - track_reference(space, py_str, space.w_str, replace=True) + py_type.c_ob_type = rffi.cast(PyTypeObjectPtr, py_type) + track_reference(space, py_type, space.w_type) + type_attach(space, py_type, space.w_type) @cpython_api([PyObject], lltype.Void, external=False) @@ -510,7 +468,7 @@ if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: w_typename = space.getattr(w_type, space.wrap('__name__')) heaptype = rffi.cast(PyHeapTypeObject, pto) - heaptype.c_ht_name = make_ref(space, w_typename) + heaptype.c_ht_name = get_pyobj_and_incref(space, w_typename) from pypy.module.cpyext.stringobject import PyString_AsString pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) else: From pypy.commits at gmail.com Tue Jan 26 17:36:38 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 14:36:38 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: fixes, now getting a segfault Message-ID: <56a7f4f6.c96cc20a.32d30.5847@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81957:0da7861a5b1a Date: 2016-01-26 23:35 +0100 http://bitbucket.org/pypy/pypy/changeset/0da7861a5b1a/ Log: fixes, now getting a segfault diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -9,7 +9,7 @@ from rpython.rtyper.tool import rffi_platform from rpython.rtyper.lltypesystem import ll2ctypes from rpython.rtyper.annlowlevel import llhelper -from rpython.rlib.objectmodel import we_are_translated +from rpython.rlib.objectmodel import we_are_translated, keepalive_until_here from rpython.translator import cdir from rpython.translator.tool.cbuild import ExternalCompilationInfo from rpython.translator.gensupp import NameManager diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -11,7 +11,7 @@ from pypy.objspace.std.typeobject import W_TypeObject from pypy.objspace.std.objectobject import W_ObjectObject from rpython.rlib.objectmodel import specialize, we_are_translated -from rpython.rlib.rweakref import RWeakKeyDictionary +from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount @@ -183,7 +183,7 @@ debug_refcount("MAKREF", py_obj, w_obj) assert w_obj assert py_obj - rawrefcount.create_link_pypy(py_obj, w_obj) + rawrefcount.create_link_pypy(w_obj, py_obj) def make_ref(space, w_obj): """ diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py --- a/pypy/module/cpyext/test/test_api.py +++ b/pypy/module/cpyext/test/test_api.py @@ -6,6 +6,7 @@ from pypy.module.cpyext.test.test_cpyext import freeze_refcnts, LeakCheckingTest PyObject = api.PyObject from pypy.interpreter.error import OperationError +from rpython.rlib import rawrefcount import os @api.cpython_api([PyObject], lltype.Void) @@ -36,6 +37,9 @@ cls.api = CAPI() CAPI.__dict__.update(api.INTERPLEVEL_API) + print 'DONT_FREE_ANY_MORE' + rawrefcount._dont_free_any_more() + def raises(self, space, api, expected_exc, f, *args): if not callable(f): raise Exception("%s is not callable" % (f,)) @@ -49,6 +53,7 @@ return state.clear_exception() def setup_method(self, func): + #return # ZZZ freeze_refcnts(self) def teardown_method(self, func): From pypy.commits at gmail.com Tue Jan 26 18:08:49 2016 From: pypy.commits at gmail.com (amauryfa) Date: Tue, 26 Jan 2016 15:08:49 -0800 (PST) Subject: [pypy-commit] pypy py3.3: io.BufferedWriter: completely remove max_buffer_size argument. Message-ID: <56a7fc81.4f911c0a.b4082.ffffd153@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r81958:18cc712aac9b Date: 2016-01-26 22:52 +0100 http://bitbucket.org/pypy/pypy/changeset/18cc712aac9b/ Log: io.BufferedWriter: completely remove max_buffer_size argument. diff --git a/pypy/module/_io/interp_bufferedio.py b/pypy/module/_io/interp_bufferedio.py --- a/pypy/module/_io/interp_bufferedio.py +++ b/pypy/module/_io/interp_bufferedio.py @@ -91,10 +91,6 @@ rwbuffer.setslice(0, data) return space.wrap(len(data)) - def _complain_about_max_buffer_size(self, space): - space.warn(space.wrap("max_buffer_size is deprecated"), - space.w_DeprecationWarning) - W_BufferedIOBase.typedef = TypeDef( '_io._BufferedIOBase', W_IOBase.typedef, __new__ = generic_new_descr(W_BufferedIOBase), @@ -888,11 +884,8 @@ ) class W_BufferedWriter(BufferedMixin, W_BufferedIOBase): - @unwrap_spec(buffer_size=int, max_buffer_size=int) - def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE, - max_buffer_size=-234): - if max_buffer_size != -234: - self._complain_about_max_buffer_size(space) + @unwrap_spec(buffer_size=int) + def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE): self.state = STATE_ZERO check_writable_w(space, w_raw) @@ -954,12 +947,10 @@ w_reader = None w_writer = None - @unwrap_spec(buffer_size=int, max_buffer_size=int) + @unwrap_spec(buffer_size=int) def descr_init(self, space, w_reader, w_writer, - buffer_size=DEFAULT_BUFFER_SIZE, max_buffer_size=-234): + buffer_size=DEFAULT_BUFFER_SIZE): try: - if max_buffer_size != -234: - self._complain_about_max_buffer_size(space) self.w_reader = W_BufferedReader(space) self.w_reader.descr_init(space, w_reader, buffer_size) self.w_writer = W_BufferedWriter(space) @@ -1029,12 +1020,8 @@ ) class W_BufferedRandom(BufferedMixin, W_BufferedIOBase): - @unwrap_spec(buffer_size=int, max_buffer_size=int) - def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE, - max_buffer_size=-234): - if max_buffer_size != -234: - self._complain_about_max_buffer_size(space) - + @unwrap_spec(buffer_size=int) + def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE): self.state = STATE_ZERO check_readable_w(space, w_raw) check_writable_w(space, w_raw) diff --git a/pypy/module/_io/test/test_bufferedio.py b/pypy/module/_io/test/test_bufferedio.py --- a/pypy/module/_io/test/test_bufferedio.py +++ b/pypy/module/_io/test/test_bufferedio.py @@ -749,28 +749,17 @@ raises(TypeError, f.readline, 5.3) -class AppTestDeprecation: +class AppTestMaxBuffer: - def w_check_max_buffer_size_deprecation(self, test): + def w_check_max_buffer_size_removal(self, test): import _io - import _warnings - def simplefilter(action, category): - _warnings.filters.insert(0, (action, None, category, None, 0)) - simplefilter('error', DeprecationWarning) - try: - test(_io.BytesIO(), 8, 12) - except DeprecationWarning as e: - assert 'max_buffer_size is deprecated' in str(e) - else: - assert False, 'Expected DeprecationWarning' - finally: - simplefilter('default', DeprecationWarning) + raises(TypeError, test, _io.BytesIO(), 8, 12) - def test_max_buffer_size_deprecation(self): + def test_max_buffer_size_removal(self): import _io - self.check_max_buffer_size_deprecation(_io.BufferedWriter) - self.check_max_buffer_size_deprecation(_io.BufferedRandom) - self.check_max_buffer_size_deprecation( + self.check_max_buffer_size_removal(_io.BufferedWriter) + self.check_max_buffer_size_removal(_io.BufferedRandom) + self.check_max_buffer_size_removal ( lambda raw, *args: _io.BufferedRWPair(raw, raw, *args)) From pypy.commits at gmail.com Tue Jan 26 18:10:07 2016 From: pypy.commits at gmail.com (amauryfa) Date: Tue, 26 Jan 2016 15:10:07 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Relax a test: with pypy we cannot choose the name of the function when argument parsing fails. Message-ID: <56a7fccf.8f7e1c0a.e19ba.ffffd2a7@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r81959:43daeebf0ab1 Date: 2016-01-26 22:44 +0100 http://bitbucket.org/pypy/pypy/changeset/43daeebf0ab1/ Log: Relax a test: with pypy we cannot choose the name of the function when argument parsing fails. diff --git a/lib-python/3/test/test_io.py b/lib-python/3/test/test_io.py --- a/lib-python/3/test/test_io.py +++ b/lib-python/3/test/test_io.py @@ -1079,7 +1079,7 @@ def test_args_error(self): # Issue #17275 - with self.assertRaisesRegex(TypeError, "BufferedReader"): + with self.assertRaisesRegex(TypeError, "BufferedReader|__init__"): self.tp(io.BytesIO(), 1024, 1024, 1024) @@ -1386,7 +1386,7 @@ def test_args_error(self): # Issue #17275 - with self.assertRaisesRegex(TypeError, "BufferedWriter"): + with self.assertRaisesRegex(TypeError, "BufferedWriter|__init__"): self.tp(io.BytesIO(), 1024, 1024, 1024) @@ -1779,7 +1779,7 @@ def test_args_error(self): # Issue #17275 - with self.assertRaisesRegex(TypeError, "BufferedRandom"): + with self.assertRaisesRegex(TypeError, "BufferedRandom|__init__"): self.tp(io.BytesIO(), 1024, 1024, 1024) From pypy.commits at gmail.com Tue Jan 26 18:10:09 2016 From: pypy.commits at gmail.com (amauryfa) Date: Tue, 26 Jan 2016 15:10:09 -0800 (PST) Subject: [pypy-commit] pypy py3.3: TextIOWrapper: Ensure the constructor complains if passed a codec that isn't Message-ID: <56a7fcd1.a3f6c20a.2ed55.5f8e@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r81960:5ddd2d347061 Date: 2016-01-27 00:03 +0100 http://bitbucket.org/pypy/pypy/changeset/5ddd2d347061/ Log: TextIOWrapper: Ensure the constructor complains if passed a codec that isn't marked as a text encoding. diff --git a/pypy/module/_io/interp_textio.py b/pypy/module/_io/interp_textio.py --- a/pypy/module/_io/interp_textio.py +++ b/pypy/module/_io/interp_textio.py @@ -413,10 +413,16 @@ else: self.writenl = None + w_codec = interp_codecs.lookup_codec(space, + space.str_w(self.w_encoding)) + if not space.is_true(space.getattr(w_codec, + space.wrap('_is_text_encoding'))): + msg = ("%R is not a text encoding; " + "use codecs.open() to handle arbitrary codecs") + raise oefmt(space.w_LookupError, msg, self.w_encoding) + # build the decoder object if space.is_true(space.call_method(w_buffer, "readable")): - w_codec = interp_codecs.lookup_codec(space, - space.str_w(self.w_encoding)) self.w_decoder = space.call_method(w_codec, "incrementaldecoder", w_errors) if self.readuniversal: @@ -426,8 +432,6 @@ # build the encoder object if space.is_true(space.call_method(w_buffer, "writable")): - w_codec = interp_codecs.lookup_codec(space, - space.str_w(self.w_encoding)) self.w_encoder = space.call_method(w_codec, "incrementalencoder", w_errors) diff --git a/pypy/module/_io/test/test_textio.py b/pypy/module/_io/test/test_textio.py --- a/pypy/module/_io/test/test_textio.py +++ b/pypy/module/_io/test/test_textio.py @@ -367,15 +367,8 @@ def test_illegal_decoder(self): import _io - t = _io.TextIOWrapper(_io.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') - raises(TypeError, t.read, 1) - t = _io.TextIOWrapper(_io.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') - raises(TypeError, t.readline) - t = _io.TextIOWrapper(_io.BytesIO(b'aaaaaa'), newline='\n', - encoding='quopri_codec') - raises(TypeError, t.read) + raises(LookupError, _io.TextIOWrapper, _io.BytesIO(), + encoding='quopri_codec') def test_read_nonbytes(self): import _io From pypy.commits at gmail.com Tue Jan 26 18:19:58 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 15:19:58 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: Found the segfault's origin, made more explicit that way (not fixed yet) Message-ID: <56a7ff1e.a5c9c20a.42161.653d@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81961:d9a31cd29535 Date: 2016-01-27 00:19 +0100 http://bitbucket.org/pypy/pypy/changeset/d9a31cd29535/ Log: Found the segfault's origin, made more explicit that way (not fixed yet) diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -439,6 +439,9 @@ pto.c_tp_as_mapping = heaptype.c_as_mapping pto.c_tp_as_buffer = heaptype.c_as_buffer + pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out + pto.c_tp_itemsize = 0 + return rffi.cast(PyObject, heaptype) def type_attach(space, py_obj, w_type): @@ -473,8 +476,6 @@ pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) else: pto.c_tp_name = rffi.str2charp(w_type.name) - pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out - pto.c_tp_itemsize = 0 # uninitialized fields: # c_tp_print, c_tp_getattr, c_tp_setattr # XXX implement From pypy.commits at gmail.com Tue Jan 26 18:27:41 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 15:27:41 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: probably fixed by moving this part of the code later (this is part of an Message-ID: <56a800ed.88c8c20a.365f1.6515@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81962:834fd655953d Date: 2016-01-27 00:26 +0100 http://bitbucket.org/pypy/pypy/changeset/834fd655953d/ Log: probably fixed by moving this part of the code later (this is part of an attempt at simplifying the bootstrapping logic in init_typeobject, will see how far this theory goes or if I need to really reproduce it again) diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -468,20 +468,13 @@ PyObject_Del.api_func.get_wrapper(space)) pto.c_tp_alloc = llhelper(PyType_GenericAlloc.api_func.functype, PyType_GenericAlloc.api_func.get_wrapper(space)) - if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: - w_typename = space.getattr(w_type, space.wrap('__name__')) - heaptype = rffi.cast(PyHeapTypeObject, pto) - heaptype.c_ht_name = get_pyobj_and_incref(space, w_typename) - from pypy.module.cpyext.stringobject import PyString_AsString - pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) - else: - pto.c_tp_name = rffi.str2charp(w_type.name) # uninitialized fields: # c_tp_print, c_tp_getattr, c_tp_setattr # XXX implement # c_tp_compare and the following fields (see http://docs.python.org/c-api/typeobj.html ) w_base = best_base(space, w_type.bases_w) - pto.c_tp_base = rffi.cast(PyTypeObjectPtr, make_ref(space, w_base)) + py_base = get_pyobj_and_incref(space, w_base) + pto.c_tp_base = rffi.cast(PyTypeObjectPtr, py_base) finish_type_1(space, pto) finish_type_2(space, pto, w_type) @@ -496,6 +489,16 @@ if space.is_w(w_type, space.w_object): pto.c_tp_new = rffi.cast(newfunc, 1) update_all_slots(space, w_type, pto) + + if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: + w_typename = space.getattr(w_type, space.wrap('__name__')) + heaptype = rffi.cast(PyHeapTypeObject, pto) + heaptype.c_ht_name = get_pyobj_and_incref(space, w_typename) + from pypy.module.cpyext.stringobject import PyString_AsString + pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) + else: + pto.c_tp_name = rffi.str2charp(w_type.name) + pto.c_tp_flags |= Py_TPFLAGS_READY return pto @@ -562,8 +565,7 @@ if not py_type.c_tp_base: # borrowed reference, but w_object is unlikely to disappear - base = make_ref(space, space.w_object) - Py_DecRef(space, base) + base = as_pyobj(space, space.w_object) py_type.c_tp_base = rffi.cast(PyTypeObjectPtr, base) finish_type_1(space, py_type) @@ -597,7 +599,7 @@ bases = space.newtuple([]) else: bases = space.newtuple([from_ref(space, base_pyo)]) - pto.c_tp_bases = make_ref(space, bases) + pto.c_tp_bases = get_pyobj_and_incref(space, bases) def finish_type_2(space, pto, w_obj): """ From pypy.commits at gmail.com Tue Jan 26 18:36:51 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 15:36:51 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: Rename: this new function should really be a superset of make_ref() Message-ID: <56a80313.41dfc20a.e49d.6721@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81963:af2938e59cd3 Date: 2016-01-27 00:29 +0100 http://bitbucket.org/pypy/pypy/changeset/af2938e59cd3/ Log: Rename: this new function should really be a superset of make_ref() diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -675,7 +675,7 @@ if callable.api_func.result_borrowed: retval = as_pyobj(space, result) else: - retval = get_pyobj_and_incref(space, result) + retval = make_ref(space, result) retval = rffi.cast(callable.api_func.restype, retval) else: retval = lltype.nullptr(PyObject.TO) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -33,7 +33,7 @@ # similar to PyType_GenericAlloc? # except that it's not related to any pypy object. - pytype = get_pyobj_and_incref(space, w_type) + pytype = make_ref(space, w_type) pytype = rffi.cast(PyTypeObjectPtr, pytype) assert pytype # Don't increase refcount for non-heaptypes @@ -241,7 +241,7 @@ This doesn't give a new reference, but the returned 'PyObject *' is valid at least as long as 'w_obj' is. **To be safe, you should use keepalive_until_here(w_obj) some time later.** In case of - doubt, use the safer get_pyobj_and_incref(). + doubt, use the safer make_ref(). """ if w_obj is not None: assert not is_pyobj(w_obj) @@ -293,7 +293,7 @@ return hop.inputconst(lltype.Bool, hop.s_result.const) @specialize.ll() -def get_pyobj_and_incref(space, obj): +def make_ref(space, obj): """Increment the reference counter of the PyObject and return it. Can be called with either a PyObject or a W_Root. """ @@ -309,7 +309,7 @@ return pyobj else: return lltype.nullptr(PyObject.TO) -INTERPLEVEL_API['get_pyobj_and_incref'] = get_pyobj_and_incref +INTERPLEVEL_API['make_ref'] = make_ref @specialize.ll() @@ -335,7 +335,7 @@ @specialize.ll() def incref(space, obj): - get_pyobj_and_incref(space, obj) + make_ref(space, obj) INTERPLEVEL_API['incref'] = incref @specialize.ll() diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -21,8 +21,7 @@ from pypy.module.cpyext.modsupport import convert_method_defs from pypy.module.cpyext.pyobject import ( PyObject, make_ref, create_ref, from_ref, get_typedescr, make_typedescr, - track_reference, RefcountState, borrow_from, Py_DecRef, - get_pyobj_and_incref) + track_reference, RefcountState, borrow_from, Py_DecRef) from pypy.module.cpyext.slotdefs import ( slotdefs_for_tp_slots, slotdefs_for_wrappers, get_slot_tp_function) from pypy.module.cpyext.state import State @@ -418,7 +417,7 @@ def type_alloc(space, w_metatype): - metatype = get_pyobj_and_incref(space, w_metatype) + metatype = make_ref(space, w_metatype) metatype = rffi.cast(PyTypeObjectPtr, metatype) assert metatype # Don't increase refcount for non-heaptypes @@ -473,7 +472,7 @@ # XXX implement # c_tp_compare and the following fields (see http://docs.python.org/c-api/typeobj.html ) w_base = best_base(space, w_type.bases_w) - py_base = get_pyobj_and_incref(space, w_base) + py_base = make_ref(space, w_base) pto.c_tp_base = rffi.cast(PyTypeObjectPtr, py_base) finish_type_1(space, pto) @@ -493,7 +492,7 @@ if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: w_typename = space.getattr(w_type, space.wrap('__name__')) heaptype = rffi.cast(PyHeapTypeObject, pto) - heaptype.c_ht_name = get_pyobj_and_incref(space, w_typename) + heaptype.c_ht_name = make_ref(space, w_typename) from pypy.module.cpyext.stringobject import PyString_AsString pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) else: @@ -599,7 +598,7 @@ bases = space.newtuple([]) else: bases = space.newtuple([from_ref(space, base_pyo)]) - pto.c_tp_bases = get_pyobj_and_incref(space, bases) + pto.c_tp_bases = make_ref(space, bases) def finish_type_2(space, pto, w_obj): """ From pypy.commits at gmail.com Tue Jan 26 18:36:53 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 15:36:53 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: rename from_pyobj() back to from_ref() too Message-ID: <56a80315.520e1c0a.545fd.ffffd6f5@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81964:769d8cdc1853 Date: 2016-01-27 00:34 +0100 http://bitbucket.org/pypy/pypy/changeset/769d8cdc1853/ Log: rename from_pyobj() back to from_ref() too diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -308,7 +308,7 @@ elif is_PyObject(ARG) and is_wrapped: # build a W_Root, possibly from a 'PyObject *' if is_pyobj(input_arg): - arg = from_pyobj(space, input_arg) + arg = from_ref(space, input_arg) else: arg = input_arg @@ -634,7 +634,7 @@ arg = args[i] if is_PyObject(typ) and is_wrapped: assert is_pyobj(arg) - arg_conv = from_pyobj(space, arg) + arg_conv = from_ref(space, arg) else: arg_conv = arg boxed_args += (arg_conv, ) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -209,17 +209,12 @@ Finds the interpreter object corresponding to the given reference. If the object is not yet realized (see stringobject.py), creates it. """ - GOES_AWAY - assert lltype.typeOf(ref) == PyObject + assert is_pyobj(ref) if not ref: return None - state = space.fromcache(RefcountState) - ptr = rffi.cast(ADDR, ref) - - try: - return state.py_objects_r2w[ptr] - except KeyError: - pass + w_obj = rawrefcount.to_obj(W_Root, pyobj) + if w_obj is not None: + return w_obj # This reference is not yet a real interpreter object. # Realize it. @@ -259,20 +254,6 @@ return rawrefcount.to_obj(W_Root, pyobj) is not None INTERPLEVEL_API['pyobj_has_w_obj'] = staticmethod(pyobj_has_w_obj) - at specialize.ll() -def from_pyobj(space, pyobj): - assert is_pyobj(pyobj) - if pyobj: - pyobj = rffi.cast(PyObject, pyobj) - w_obj = rawrefcount.to_obj(W_Root, pyobj) - if w_obj is None: - XXXXXXXXXXX - return w_obj - else: - return None -from_pyobj._always_inline_ = 'try' -INTERPLEVEL_API['from_pyobj'] = from_pyobj - def is_pyobj(x): if x is None or isinstance(x, W_Root): @@ -321,7 +302,7 @@ """ if is_pyobj(obj): pyobj = rffi.cast(PyObject, obj) - w_obj = from_pyobj(space, pyobj) + w_obj = from_ref(space, pyobj) else: w_obj = obj pyobj = as_pyobj(space, w_obj) diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -21,7 +21,7 @@ from pypy.module.cpyext.modsupport import convert_method_defs from pypy.module.cpyext.pyobject import ( PyObject, make_ref, create_ref, from_ref, get_typedescr, make_typedescr, - track_reference, RefcountState, borrow_from, Py_DecRef) + track_reference, RefcountState, borrow_from, Py_DecRef, as_pyobj) from pypy.module.cpyext.slotdefs import ( slotdefs_for_tp_slots, slotdefs_for_wrappers, get_slot_tp_function) from pypy.module.cpyext.state import State From pypy.commits at gmail.com Tue Jan 26 19:19:56 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 16:19:56 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: Hopefully fix the bootstrap cycles in this way Message-ID: <56a80d2c.520e1c0a.545fd.ffffe019@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81965:ae316294940c Date: 2016-01-27 01:04 +0100 http://bitbucket.org/pypy/pypy/changeset/ae316294940c/ Log: Hopefully fix the bootstrap cycles in this way diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -310,13 +310,24 @@ realize=type_realize, dealloc=type_dealloc) - # we create the type "type" manually here, because of the cycle - # through its 'c_ob_type' field + # There is the obvious cycle of 'type(type) == type', but there are + # also several other ones, like 'tuple.tp_bases' being itself a + # tuple instance. We solve the first one by creating the type + # "type" manually here. For the other cycles, we fix them by delaying + # creation of the types here, and hoping nothing breaks by seeing + # uninitialized-yet types (only for a few basic types like 'type', + # 'tuple', 'object', 'str'). + space._cpyext_delay_type_creation = [] + py_type = _type_alloc(space, lltype.nullptr(PyTypeObject)) py_type.c_ob_type = rffi.cast(PyTypeObjectPtr, py_type) track_reference(space, py_type, space.w_type) type_attach(space, py_type, space.w_type) + while space._cpyext_delay_type_creation: + _type_really_attach(space, *space._cpyext_delay_type_creation.pop()) + del space._cpyext_delay_type_creation + @cpython_api([PyObject], lltype.Void, external=False) def subtype_dealloc(space, obj): @@ -447,6 +458,13 @@ """ Fills a newly allocated PyTypeObject from an existing type. """ + if hasattr(space, '_cpyext_delay_type_creation'): + space._cpyext_delay_type_creation.append((py_obj, w_type)) + else: + _type_really_attach(space, py_obj, w_type) + return rffi.cast(PyTypeObjectPtr, py_obj) + +def _type_really_attach(space, py_obj, w_type): from pypy.module.cpyext.object import PyObject_Del assert isinstance(w_type, W_TypeObject) @@ -499,7 +517,6 @@ pto.c_tp_name = rffi.str2charp(w_type.name) pto.c_tp_flags |= Py_TPFLAGS_READY - return pto def py_type_ready(space, pto): if pto.c_tp_flags & Py_TPFLAGS_READY: @@ -513,6 +530,7 @@ def type_realize(space, py_obj): pto = rffi.cast(PyTypeObjectPtr, py_obj) + assert pto.c_tp_flags & Py_TPFLAGS_READY == 0 assert pto.c_tp_flags & Py_TPFLAGS_READYING == 0 pto.c_tp_flags |= Py_TPFLAGS_READYING try: @@ -590,7 +608,8 @@ base = pto.c_tp_base base_pyo = rffi.cast(PyObject, pto.c_tp_base) if base and not base.c_tp_flags & Py_TPFLAGS_READY: - type_realize(space, rffi.cast(PyObject, base_pyo)) + if not hasattr(space, '_cpyext_delay_type_creation'): + type_realize(space, rffi.cast(PyObject, base_pyo)) if base and not pto.c_ob_type: # will be filled later pto.c_ob_type = base.c_ob_type if not pto.c_tp_bases: diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py --- a/pypy/objspace/std/typeobject.py +++ b/pypy/objspace/std/typeobject.py @@ -164,6 +164,10 @@ else: w_self.terminator = NoDictTerminator(space, w_self) + def __repr__(self): + "NOT_RPYTHON" + return '' % (self.name, id(self)) + def mutated(w_self, key): """ The type is being mutated. key is either the string containing the From pypy.commits at gmail.com Tue Jan 26 19:19:58 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 16:19:58 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: More tweaks for bootstrapping Message-ID: <56a80d2e.0f811c0a.1bc63.1ade@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81966:dfc37d8a80f8 Date: 2016-01-27 01:17 +0100 http://bitbucket.org/pypy/pypy/changeset/dfc37d8a80f8/ Log: More tweaks for bootstrapping diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -5,7 +5,7 @@ from rpython.rtyper.extregistry import ExtRegistryEntry from pypy.module.cpyext.api import ( cpython_api, bootstrap_function, PyObject, PyObjectP, ADDR, - CANNOT_FAIL, Py_TPFLAGS_HEAPTYPE, PyTypeObjectPtr, + CANNOT_FAIL, Py_TPFLAGS_HEAPTYPE, PyTypeObjectPtr, is_PyObject, INTERPLEVEL_API) from pypy.module.cpyext.state import State from pypy.objspace.std.typeobject import W_TypeObject @@ -212,7 +212,7 @@ assert is_pyobj(ref) if not ref: return None - w_obj = rawrefcount.to_obj(W_Root, pyobj) + w_obj = rawrefcount.to_obj(W_Root, ref) if w_obj is not None: return w_obj diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -324,9 +324,21 @@ track_reference(space, py_type, space.w_type) type_attach(space, py_type, space.w_type) + as_pyobj(space, space.w_str) + as_pyobj(space, space.w_tuple) + as_pyobj(space, space.w_object) + + delayed_types = [] while space._cpyext_delay_type_creation: - _type_really_attach(space, *space._cpyext_delay_type_creation.pop()) + (py_obj, w_type) = space._cpyext_delay_type_creation.pop() + _type_really_attach(space, py_obj, w_type) + delayed_types.append((py_obj, w_type)) del space._cpyext_delay_type_creation + for py_obj, w_type in delayed_types: + pto = rffi.cast(PyTypeObjectPtr, py_obj) + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) + finish_type_3(space, pto, w_type) @cpython_api([PyObject], lltype.Void, external=False) @@ -493,8 +505,9 @@ py_base = make_ref(space, w_base) pto.c_tp_base = rffi.cast(PyTypeObjectPtr, py_base) - finish_type_1(space, pto) - finish_type_2(space, pto, w_type) + if not hasattr(space, '_cpyext_delay_type_creation'): + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) pto.c_tp_basicsize = rffi.sizeof(typedescr.basestruct) if pto.c_tp_base: @@ -507,6 +520,12 @@ pto.c_tp_new = rffi.cast(newfunc, 1) update_all_slots(space, w_type, pto) + if not hasattr(space, '_cpyext_delay_type_creation'): + finish_type_3(space, pto, w_type) + + pto.c_tp_flags |= Py_TPFLAGS_READY + +def finish_type_3(space, pto, w_type): if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: w_typename = space.getattr(w_type, space.wrap('__name__')) heaptype = rffi.cast(PyHeapTypeObject, pto) @@ -516,8 +535,6 @@ else: pto.c_tp_name = rffi.str2charp(w_type.name) - pto.c_tp_flags |= Py_TPFLAGS_READY - def py_type_ready(space, pto): if pto.c_tp_flags & Py_TPFLAGS_READY: return @@ -608,8 +625,7 @@ base = pto.c_tp_base base_pyo = rffi.cast(PyObject, pto.c_tp_base) if base and not base.c_tp_flags & Py_TPFLAGS_READY: - if not hasattr(space, '_cpyext_delay_type_creation'): - type_realize(space, rffi.cast(PyObject, base_pyo)) + type_realize(space, rffi.cast(PyObject, base_pyo)) if base and not pto.c_ob_type: # will be filled later pto.c_ob_type = base.c_ob_type if not pto.c_tp_bases: From pypy.commits at gmail.com Tue Jan 26 19:20:00 2016 From: pypy.commits at gmail.com (arigo) Date: Tue, 26 Jan 2016 16:20:00 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: fix imports Message-ID: <56a80d30.a453c20a.d2467.6ef7@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81967:fcc70c53434c Date: 2016-01-27 01:19 +0100 http://bitbucket.org/pypy/pypy/changeset/fcc70c53434c/ Log: fix imports diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -290,9 +290,8 @@ @specialize.ll() def unwrapper(space, *args): - from pypy.module.cpyext.pyobject import Py_DecRef + from pypy.module.cpyext.pyobject import Py_DecRef, is_pyobj from pypy.module.cpyext.pyobject import make_ref, from_ref - from pypy.module.cpyext.pyobject import Reference newargs = () keepalives = () assert len(args) == len(api_function.argtypes) From pypy.commits at gmail.com Wed Jan 27 07:45:29 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 04:45:29 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: A branch to simplify the bootstrapping, to reduce the amount of changes Message-ID: <56a8bbe9.46bb1c0a.8ae80.ffffed8d@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81968:30bbc81f0b06 Date: 2016-01-27 13:42 +0100 http://bitbucket.org/pypy/pypy/changeset/30bbc81f0b06/ Log: A branch to simplify the bootstrapping, to reduce the amount of changes in cpyext-gc-support-2 From pypy.commits at gmail.com Wed Jan 27 07:45:31 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 04:45:31 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: Kill kill kill the logic in init_typeobject(). It was anyway very Message-ID: <56a8bbeb.cf821c0a.32477.ffffb9f5@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81969:3cd5ace97b49 Date: 2016-01-27 13:43 +0100 http://bitbucket.org/pypy/pypy/changeset/3cd5ace97b49/ Log: Kill kill kill the logic in init_typeobject(). It was anyway very strange, because it would create "heap" type objects, initialize them carefully, and then forget them happily by calling track_reference() again (which would crash if DEBUG_REFCOUNT=True) diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -828,6 +828,9 @@ space.fromcache(State).install_dll(eci) # populate static data + from pypy.module.cpyext.pyobject import track_reference, get_typedescr + from pypy.module.cpyext.typeobject import finish_type_1, finish_type_2 + to_attach = [] for name, (typ, expr) in GLOBALS.iteritems(): from pypy.module import cpyext w_obj = eval(expr) @@ -861,18 +864,25 @@ # we have a structure, get its address in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, name) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) - from pypy.module.cpyext.pyobject import ( - track_reference, get_typedescr) - w_type = space.type(w_obj) - typedescr = get_typedescr(w_type.instancetypedef) py_obj.c_ob_refcnt = 1 - py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, - make_ref(space, w_type)) - typedescr.attach(space, py_obj, w_obj) track_reference(space, py_obj, w_obj) + to_attach.append((py_obj, w_obj)) else: assert False, "Unknown static object: %s %s" % (typ, name) + space._cpyext_type_init = [] + for py_obj, w_obj in to_attach: + w_type = space.type(w_obj) + typedescr = get_typedescr(w_type.instancetypedef) + py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, + make_ref(space, w_type)) + typedescr.attach(space, py_obj, w_obj) + cpyext_type_init = space._cpyext_type_init + del space._cpyext_type_init + for pto, w_type in cpyext_type_init: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) + pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') # implement structure initialization code diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -310,55 +310,6 @@ realize=type_realize, dealloc=type_dealloc) - # some types are difficult to create because of cycles. - # - object.ob_type = type - # - type.ob_type = type - # - tuple.ob_type = type - # - type.tp_base = object - # - tuple.tp_base = object - # - type.tp_bases is a tuple - # - object.tp_bases is a tuple - # - tuple.tp_bases is a tuple - - # insert null placeholders to please create_ref() - track_reference(space, lltype.nullptr(PyObject.TO), space.w_type) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_object) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_tuple) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_str) - - # create the objects - py_type = create_ref(space, space.w_type) - py_object = create_ref(space, space.w_object) - py_tuple = create_ref(space, space.w_tuple) - py_str = create_ref(space, space.w_str) - # XXX py_str is not initialized here correctly, because we are - # not tracking it, it gets an empty c_ob_type from py_basestring - - # form cycles - pto_type = rffi.cast(PyTypeObjectPtr, py_type) - py_type.c_ob_type = pto_type - py_object.c_ob_type = pto_type - py_tuple.c_ob_type = pto_type - - pto_object = rffi.cast(PyTypeObjectPtr, py_object) - pto_type.c_tp_base = pto_object - pto_tuple = rffi.cast(PyTypeObjectPtr, py_tuple) - pto_tuple.c_tp_base = pto_object - - pto_type.c_tp_bases.c_ob_type = pto_tuple - pto_object.c_tp_bases.c_ob_type = pto_tuple - pto_tuple.c_tp_bases.c_ob_type = pto_tuple - - for typ in (py_type, py_object, py_tuple, py_str): - heaptype = rffi.cast(PyHeapTypeObject, typ) - heaptype.c_ht_name.c_ob_type = pto_type - - # Restore the mapping - track_reference(space, py_type, space.w_type, replace=True) - track_reference(space, py_object, space.w_object, replace=True) - track_reference(space, py_tuple, space.w_tuple, replace=True) - track_reference(space, py_str, space.w_str, replace=True) - @cpython_api([PyObject], lltype.Void, external=False) def subtype_dealloc(space, obj): @@ -520,8 +471,11 @@ w_base = best_base(space, w_type.bases_w) pto.c_tp_base = rffi.cast(PyTypeObjectPtr, make_ref(space, w_base)) - finish_type_1(space, pto) - finish_type_2(space, pto, w_type) + if hasattr(space, '_cpyext_type_init'): + space._cpyext_type_init.append((pto, w_type)) + else: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) pto.c_tp_basicsize = rffi.sizeof(typedescr.basestruct) if pto.c_tp_base: From pypy.commits at gmail.com Wed Jan 27 08:27:45 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 05:27:45 -0800 (PST) Subject: [pypy-commit] pypy default: Add "-m ensurepip" Message-ID: <56a8c5d1.c615c20a.6aa2b.5547@mx.google.com> Author: Armin Rigo Branch: Changeset: r81970:4974d1173ac2 Date: 2016-01-27 14:26 +0100 http://bitbucket.org/pypy/pypy/changeset/4974d1173ac2/ Log: Add "-m ensurepip" diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -54,7 +54,8 @@ It is quite common nowadays that xyz is available on PyPI_ and installable with ``pip install xyz``. The simplest solution is to `use virtualenv (as documented here)`_. Then enter (activate) the virtualenv -and type: ``pip install xyz``. +and type: ``pip install xyz``. If you don't know or don't want virtualenv, +you can also install ``pip`` globally by saying ``pypy -m ensurepip``. If you get errors from the C compiler, the module is a CPython C Extension module using unsupported features. `See below.`_ From pypy.commits at gmail.com Wed Jan 27 08:51:04 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 05:51:04 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: pfff, took me a while to figure out that memory corruption was going on here: Message-ID: <56a8cb48.4f911c0a.b4082.ffffcfbb@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81971:ed11e491b6c2 Date: 2016-01-27 14:50 +0100 http://bitbucket.org/pypy/pypy/changeset/ed11e491b6c2/ Log: pfff, took me a while to figure out that memory corruption was going on here: the globals False/True would be initialized as longs by writing 0/1 in the third word of memory, but they were only 2-words PyObjects... diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -442,8 +442,8 @@ TYPES = {} GLOBALS = { # this needs to include all prebuilt pto, otherwise segfaults occur '_Py_NoneStruct#': ('PyObject*', 'space.w_None'), - '_Py_TrueStruct#': ('PyObject*', 'space.w_True'), - '_Py_ZeroStruct#': ('PyObject*', 'space.w_False'), + '_Py_TrueStruct#': ('PyIntObject*', 'space.w_True'), + '_Py_ZeroStruct#': ('PyIntObject*', 'space.w_False'), '_Py_NotImplementedStruct#': ('PyObject*', 'space.w_NotImplemented'), '_Py_EllipsisObject#': ('PyObject*', 'space.w_Ellipsis'), 'PyDateTimeAPI': ('PyDateTime_CAPI*', 'None'), @@ -855,7 +855,7 @@ assert False, "Unknown static pointer: %s %s" % (typ, name) ptr.value = ctypes.cast(ll2ctypes.lltype2ctypes(value), ctypes.c_void_p).value - elif typ in ('PyObject*', 'PyTypeObject*'): + elif typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): if name.startswith('PyPyExc_') or name.startswith('cpyexttestExc_'): # we already have the pointer in_dll = ll2ctypes.get_ctypes_type(PyObject).in_dll(bridge, name) From pypy.commits at gmail.com Wed Jan 27 10:37:04 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 07:37:04 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: This test checks a condition which is actually not true, but which Message-ID: <56a8e420.d02d1c0a.b9ed.fffffdef@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81972:dd3a1abe4374 Date: 2016-01-27 16:35 +0100 http://bitbucket.org/pypy/pypy/changeset/dd3a1abe4374/ Log: This test checks a condition which is actually not true, but which happened to be true due to the two versions of some PyTypeObjects which this branch removes. The condition is false e.g. after translation anyway. diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -374,6 +374,11 @@ module = self.import_extension('foo', [ ("test_type", "METH_O", ''' + /* "args->ob_type" is a strange way to get at 'type', + which should have a different tp_getattro/tp_setattro + than its tp_base, which is 'object'. + */ + if (!args->ob_type->tp_setattro) { PyErr_SetString(PyExc_ValueError, "missing tp_setattro"); @@ -382,8 +387,12 @@ if (args->ob_type->tp_setattro == args->ob_type->tp_base->tp_setattro) { - PyErr_SetString(PyExc_ValueError, "recursive tp_setattro"); - return NULL; + /* Note that unlike CPython, in PyPy 'type.tp_setattro' + is the same function as 'object.tp_setattro'. This + test used to check that it was not, but that was an + artifact of the bootstrap logic only---in the final + C sources I checked and they are indeed the same. + So we ignore this problem here. */ } if (!args->ob_type->tp_getattro) { From pypy.commits at gmail.com Wed Jan 27 11:40:37 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 08:40:37 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: Attempt to fix translation, getting segfault while translating now Message-ID: <56a8f305.863f1c0a.33db4.4b60@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81973:036fa7ebc766 Date: 2016-01-27 17:03 +0100 http://bitbucket.org/pypy/pypy/changeset/036fa7ebc766/ Log: Attempt to fix translation, getting segfault while translating now diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -828,9 +828,7 @@ space.fromcache(State).install_dll(eci) # populate static data - from pypy.module.cpyext.pyobject import track_reference, get_typedescr - from pypy.module.cpyext.typeobject import finish_type_1, finish_type_2 - to_attach = [] + builder = StaticObjectBuilder(space) for name, (typ, expr) in GLOBALS.iteritems(): from pypy.module import cpyext w_obj = eval(expr) @@ -864,24 +862,10 @@ # we have a structure, get its address in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, name) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) - py_obj.c_ob_refcnt = 1 - track_reference(space, py_obj, w_obj) - to_attach.append((py_obj, w_obj)) + builder.prepare(py_obj, w_obj) else: assert False, "Unknown static object: %s %s" % (typ, name) - - space._cpyext_type_init = [] - for py_obj, w_obj in to_attach: - w_type = space.type(w_obj) - typedescr = get_typedescr(w_type.instancetypedef) - py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, - make_ref(space, w_type)) - typedescr.attach(space, py_obj, w_obj) - cpyext_type_init = space._cpyext_type_init - del space._cpyext_type_init - for pto, w_type in cpyext_type_init: - finish_type_1(space, pto) - finish_type_2(space, pto, w_type) + builder.attach_all() pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') @@ -898,6 +882,36 @@ setup_init_functions(eci, translating=False) return modulename.new(ext='') + +class StaticObjectBuilder: + def __init__(self, space): + self.space = space + self.to_attach = [] + + def prepare(self, py_obj, w_obj): + from pypy.module.cpyext.pyobject import track_reference + py_obj.c_ob_refcnt = 1 + track_reference(self.space, py_obj, w_obj) + self.to_attach.append((py_obj, w_obj)) + + def attach_all(self): + from pypy.module.cpyext.pyobject import get_typedescr, make_ref + from pypy.module.cpyext.typeobject import finish_type_1, finish_type_2 + space = self.space + space._cpyext_type_init = [] + for py_obj, w_obj in self.to_attach: + w_type = space.type(w_obj) + typedescr = get_typedescr(w_type.instancetypedef) + py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, + make_ref(space, w_type)) + typedescr.attach(space, py_obj, w_obj) + cpyext_type_init = space._cpyext_type_init + del space._cpyext_type_init + for pto, w_type in cpyext_type_init: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) + + def mangle_name(prefix, name): if name.startswith('Py'): return prefix + name[2:] @@ -1074,7 +1088,7 @@ def setup_library(space): "NOT_RPYTHON" - from pypy.module.cpyext.pyobject import make_ref + from pypy.module.cpyext.pyobject import get_typedescr, make_ref export_symbols = list(FUNCTIONS) + SYMBOLS_C + list(GLOBALS) from rpython.translator.c.database import LowLevelDatabase @@ -1092,14 +1106,36 @@ run_bootstrap_functions(space) setup_va_functions(eci) + from pypy.module import cpyext # for eval() below + + # Set up the types. This version of the code really allocates + # them: this is different from build_bridge(), where they are set + # up at the static address from the bridge library. This needs + # special logic to solve the cycles issue; otherwise, we could + # simply leave everything to make_ref() in the "populate static + # data" loop below. + builder = StaticObjectBuilder(space) + for name, (typ, expr) in GLOBALS.iteritems(): + if typ == 'PyTypeObject*': + w_type = eval(expr) + w_typetype = space.type(w_type) + if not space.is_w(w_typetype, space.w_type): + continue # skip types with a custom metaclass + typedescr = get_typedescr(w_typetype.instancetypedef) + py_obj = typedescr.allocate(space, None) + builder.prepare(py_obj, w_type) + py_typetype = rffi.cast(PyTypeObjectPtr, make_ref(space, space.w_type)) + for py_obj, w_type in builder.to_attach: + py_obj.c_ob_type = py_typetype + builder.attach_all() + # populate static data for name, (typ, expr) in GLOBALS.iteritems(): name = name.replace("#", "") if name.startswith('PyExc_'): name = '_' + name - from pypy.module import cpyext w_obj = eval(expr) - if typ in ('PyObject*', 'PyTypeObject*'): + if typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): struct_ptr = make_ref(space, w_obj) elif typ == 'PyDateTime_CAPI*': continue From pypy.commits at gmail.com Wed Jan 27 11:40:39 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 08:40:39 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: Some translation fixes Message-ID: <56a8f307.e906c20a.89ca3.ffffa289@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81974:a11965258086 Date: 2016-01-27 17:39 +0100 http://bitbucket.org/pypy/pypy/changeset/a11965258086/ Log: Some translation fixes diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -506,7 +506,9 @@ def get_structtype_for_ctype(ctype): from pypy.module.cpyext.typeobjectdefs import PyTypeObjectPtr from pypy.module.cpyext.cdatetime import PyDateTime_CAPI + from pypy.module.cpyext.intobject import PyIntObject return {"PyObject*": PyObject, "PyTypeObject*": PyTypeObjectPtr, + "PyIntObject*": PyIntObject, "PyDateTime_CAPI*": lltype.Ptr(PyDateTime_CAPI)}[ctype] PyTypeObject = lltype.ForwardReference() @@ -1088,7 +1090,7 @@ def setup_library(space): "NOT_RPYTHON" - from pypy.module.cpyext.pyobject import get_typedescr, make_ref + from pypy.module.cpyext.pyobject import make_ref export_symbols = list(FUNCTIONS) + SYMBOLS_C + list(GLOBALS) from rpython.translator.c.database import LowLevelDatabase @@ -1108,25 +1110,22 @@ from pypy.module import cpyext # for eval() below - # Set up the types. This version of the code really allocates - # them: this is different from build_bridge(), where they are set - # up at the static address from the bridge library. This needs - # special logic to solve the cycles issue; otherwise, we could - # simply leave everything to make_ref() in the "populate static - # data" loop below. + # Set up the types. Needs a special case, because of the + # immediate cycle involving 'c_ob_type', and because we don't + # want these types to be Py_TPFLAGS_HEAPTYPE. + static_types = {} + for name, (typ, expr) in GLOBALS.items(): + if typ == 'PyTypeObject*': + pto = lltype.malloc(PyTypeObject, immortal=True, + zero=True, flavor='raw') + pto.c_ob_refcnt = 1 + pto.c_tp_basicsize = -1 + static_types[name] = pto builder = StaticObjectBuilder(space) - for name, (typ, expr) in GLOBALS.iteritems(): - if typ == 'PyTypeObject*': - w_type = eval(expr) - w_typetype = space.type(w_type) - if not space.is_w(w_typetype, space.w_type): - continue # skip types with a custom metaclass - typedescr = get_typedescr(w_typetype.instancetypedef) - py_obj = typedescr.allocate(space, None) - builder.prepare(py_obj, w_type) - py_typetype = rffi.cast(PyTypeObjectPtr, make_ref(space, space.w_type)) - for py_obj, w_type in builder.to_attach: - py_obj.c_ob_type = py_typetype + for name, pto in static_types.items(): + pto.c_ob_type = static_types['PyType_Type#'] + w_type = eval(GLOBALS[name][1]) + builder.prepare(rffi.cast(PyObject, pto), w_type) builder.attach_all() # populate static data diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -146,7 +146,7 @@ assert len(slot_names) == 2 struct = getattr(pto, slot_names[0]) if not struct: - assert not space.config.translating + #assert not space.config.translating assert not pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE if slot_names[0] == 'c_tp_as_number': STRUCT_TYPE = PyNumberMethods @@ -427,6 +427,8 @@ pto.c_tp_as_sequence = heaptype.c_as_sequence pto.c_tp_as_mapping = heaptype.c_as_mapping pto.c_tp_as_buffer = heaptype.c_as_buffer + pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out + pto.c_tp_itemsize = 0 return rffi.cast(PyObject, heaptype) @@ -462,8 +464,6 @@ pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) else: pto.c_tp_name = rffi.str2charp(w_type.name) - pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out - pto.c_tp_itemsize = 0 # uninitialized fields: # c_tp_print, c_tp_getattr, c_tp_setattr # XXX implement From pypy.commits at gmail.com Wed Jan 27 12:06:24 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 09:06:24 -0800 (PST) Subject: [pypy-commit] pypy default: cpyext implies micronumpy, otherwise we get never-initialized ndarray subclasses Message-ID: <56a8f910.e218c20a.e673a.ffffb0d4@mx.google.com> Author: Armin Rigo Branch: Changeset: r81975:ec6457a2d845 Date: 2016-01-27 18:05 +0100 http://bitbucket.org/pypy/pypy/changeset/ec6457a2d845/ Log: cpyext implies micronumpy, otherwise we get never-initialized ndarray subclasses diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py --- a/pypy/config/pypyoption.py +++ b/pypy/config/pypyoption.py @@ -85,7 +85,8 @@ module_dependencies = { '_multiprocessing': [('objspace.usemodules.time', True), ('objspace.usemodules.thread', True)], - 'cpyext': [('objspace.usemodules.array', True)], + 'cpyext': [('objspace.usemodules.array', True), + ('objspace.usemodules.micronumpy', True)], 'cppyy': [('objspace.usemodules.cpyext', True)], } module_suggests = { From pypy.commits at gmail.com Wed Jan 27 13:03:03 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 10:03:03 -0800 (PST) Subject: [pypy-commit] pypy default: hg merge cpyext-bootstrap Message-ID: <56a90657.82e11c0a.12767.7089@mx.google.com> Author: Armin Rigo Branch: Changeset: r81977:7d1a37bf041c Date: 2016-01-27 19:02 +0100 http://bitbucket.org/pypy/pypy/changeset/7d1a37bf041c/ Log: hg merge cpyext-bootstrap simplify the bootstrap procedure, which was a bit manual and slightly buggy (e.g. untranslated, it made two versions of the same PyTypeObjects; and translated, the type objects ends up with Py_TPFLAGS_HEAPTYPE, even though they should not be). also fixes a potential memory corruption for initializing boolean objects. diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -442,8 +442,8 @@ TYPES = {} GLOBALS = { # this needs to include all prebuilt pto, otherwise segfaults occur '_Py_NoneStruct#': ('PyObject*', 'space.w_None'), - '_Py_TrueStruct#': ('PyObject*', 'space.w_True'), - '_Py_ZeroStruct#': ('PyObject*', 'space.w_False'), + '_Py_TrueStruct#': ('PyIntObject*', 'space.w_True'), + '_Py_ZeroStruct#': ('PyIntObject*', 'space.w_False'), '_Py_NotImplementedStruct#': ('PyObject*', 'space.w_NotImplemented'), '_Py_EllipsisObject#': ('PyObject*', 'space.w_Ellipsis'), 'PyDateTimeAPI': ('PyDateTime_CAPI*', 'None'), @@ -506,7 +506,9 @@ def get_structtype_for_ctype(ctype): from pypy.module.cpyext.typeobjectdefs import PyTypeObjectPtr from pypy.module.cpyext.cdatetime import PyDateTime_CAPI + from pypy.module.cpyext.intobject import PyIntObject return {"PyObject*": PyObject, "PyTypeObject*": PyTypeObjectPtr, + "PyIntObject*": PyIntObject, "PyDateTime_CAPI*": lltype.Ptr(PyDateTime_CAPI)}[ctype] PyTypeObject = lltype.ForwardReference() @@ -828,6 +830,7 @@ space.fromcache(State).install_dll(eci) # populate static data + builder = StaticObjectBuilder(space) for name, (typ, expr) in GLOBALS.iteritems(): from pypy.module import cpyext w_obj = eval(expr) @@ -852,7 +855,7 @@ assert False, "Unknown static pointer: %s %s" % (typ, name) ptr.value = ctypes.cast(ll2ctypes.lltype2ctypes(value), ctypes.c_void_p).value - elif typ in ('PyObject*', 'PyTypeObject*'): + elif typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): if name.startswith('PyPyExc_') or name.startswith('cpyexttestExc_'): # we already have the pointer in_dll = ll2ctypes.get_ctypes_type(PyObject).in_dll(bridge, name) @@ -861,17 +864,10 @@ # we have a structure, get its address in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, name) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) - from pypy.module.cpyext.pyobject import ( - track_reference, get_typedescr) - w_type = space.type(w_obj) - typedescr = get_typedescr(w_type.instancetypedef) - py_obj.c_ob_refcnt = 1 - py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, - make_ref(space, w_type)) - typedescr.attach(space, py_obj, w_obj) - track_reference(space, py_obj, w_obj) + builder.prepare(py_obj, w_obj) else: assert False, "Unknown static object: %s %s" % (typ, name) + builder.attach_all() pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') @@ -888,6 +884,36 @@ setup_init_functions(eci, translating=False) return modulename.new(ext='') + +class StaticObjectBuilder: + def __init__(self, space): + self.space = space + self.to_attach = [] + + def prepare(self, py_obj, w_obj): + from pypy.module.cpyext.pyobject import track_reference + py_obj.c_ob_refcnt = 1 + track_reference(self.space, py_obj, w_obj) + self.to_attach.append((py_obj, w_obj)) + + def attach_all(self): + from pypy.module.cpyext.pyobject import get_typedescr, make_ref + from pypy.module.cpyext.typeobject import finish_type_1, finish_type_2 + space = self.space + space._cpyext_type_init = [] + for py_obj, w_obj in self.to_attach: + w_type = space.type(w_obj) + typedescr = get_typedescr(w_type.instancetypedef) + py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, + make_ref(space, w_type)) + typedescr.attach(space, py_obj, w_obj) + cpyext_type_init = space._cpyext_type_init + del space._cpyext_type_init + for pto, w_type in cpyext_type_init: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) + + def mangle_name(prefix, name): if name.startswith('Py'): return prefix + name[2:] @@ -1082,14 +1108,33 @@ run_bootstrap_functions(space) setup_va_functions(eci) + from pypy.module import cpyext # for eval() below + + # Set up the types. Needs a special case, because of the + # immediate cycle involving 'c_ob_type', and because we don't + # want these types to be Py_TPFLAGS_HEAPTYPE. + static_types = {} + for name, (typ, expr) in GLOBALS.items(): + if typ == 'PyTypeObject*': + pto = lltype.malloc(PyTypeObject, immortal=True, + zero=True, flavor='raw') + pto.c_ob_refcnt = 1 + pto.c_tp_basicsize = -1 + static_types[name] = pto + builder = StaticObjectBuilder(space) + for name, pto in static_types.items(): + pto.c_ob_type = static_types['PyType_Type#'] + w_type = eval(GLOBALS[name][1]) + builder.prepare(rffi.cast(PyObject, pto), w_type) + builder.attach_all() + # populate static data for name, (typ, expr) in GLOBALS.iteritems(): name = name.replace("#", "") if name.startswith('PyExc_'): name = '_' + name - from pypy.module import cpyext w_obj = eval(expr) - if typ in ('PyObject*', 'PyTypeObject*'): + if typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): struct_ptr = make_ref(space, w_obj) elif typ == 'PyDateTime_CAPI*': continue diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -374,6 +374,11 @@ module = self.import_extension('foo', [ ("test_type", "METH_O", ''' + /* "args->ob_type" is a strange way to get at 'type', + which should have a different tp_getattro/tp_setattro + than its tp_base, which is 'object'. + */ + if (!args->ob_type->tp_setattro) { PyErr_SetString(PyExc_ValueError, "missing tp_setattro"); @@ -382,8 +387,12 @@ if (args->ob_type->tp_setattro == args->ob_type->tp_base->tp_setattro) { - PyErr_SetString(PyExc_ValueError, "recursive tp_setattro"); - return NULL; + /* Note that unlike CPython, in PyPy 'type.tp_setattro' + is the same function as 'object.tp_setattro'. This + test used to check that it was not, but that was an + artifact of the bootstrap logic only---in the final + C sources I checked and they are indeed the same. + So we ignore this problem here. */ } if (!args->ob_type->tp_getattro) { diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -146,7 +146,7 @@ assert len(slot_names) == 2 struct = getattr(pto, slot_names[0]) if not struct: - assert not space.config.translating + #assert not space.config.translating assert not pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE if slot_names[0] == 'c_tp_as_number': STRUCT_TYPE = PyNumberMethods @@ -310,55 +310,6 @@ realize=type_realize, dealloc=type_dealloc) - # some types are difficult to create because of cycles. - # - object.ob_type = type - # - type.ob_type = type - # - tuple.ob_type = type - # - type.tp_base = object - # - tuple.tp_base = object - # - type.tp_bases is a tuple - # - object.tp_bases is a tuple - # - tuple.tp_bases is a tuple - - # insert null placeholders to please create_ref() - track_reference(space, lltype.nullptr(PyObject.TO), space.w_type) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_object) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_tuple) - track_reference(space, lltype.nullptr(PyObject.TO), space.w_str) - - # create the objects - py_type = create_ref(space, space.w_type) - py_object = create_ref(space, space.w_object) - py_tuple = create_ref(space, space.w_tuple) - py_str = create_ref(space, space.w_str) - # XXX py_str is not initialized here correctly, because we are - # not tracking it, it gets an empty c_ob_type from py_basestring - - # form cycles - pto_type = rffi.cast(PyTypeObjectPtr, py_type) - py_type.c_ob_type = pto_type - py_object.c_ob_type = pto_type - py_tuple.c_ob_type = pto_type - - pto_object = rffi.cast(PyTypeObjectPtr, py_object) - pto_type.c_tp_base = pto_object - pto_tuple = rffi.cast(PyTypeObjectPtr, py_tuple) - pto_tuple.c_tp_base = pto_object - - pto_type.c_tp_bases.c_ob_type = pto_tuple - pto_object.c_tp_bases.c_ob_type = pto_tuple - pto_tuple.c_tp_bases.c_ob_type = pto_tuple - - for typ in (py_type, py_object, py_tuple, py_str): - heaptype = rffi.cast(PyHeapTypeObject, typ) - heaptype.c_ht_name.c_ob_type = pto_type - - # Restore the mapping - track_reference(space, py_type, space.w_type, replace=True) - track_reference(space, py_object, space.w_object, replace=True) - track_reference(space, py_tuple, space.w_tuple, replace=True) - track_reference(space, py_str, space.w_str, replace=True) - @cpython_api([PyObject], lltype.Void, external=False) def subtype_dealloc(space, obj): @@ -476,6 +427,8 @@ pto.c_tp_as_sequence = heaptype.c_as_sequence pto.c_tp_as_mapping = heaptype.c_as_mapping pto.c_tp_as_buffer = heaptype.c_as_buffer + pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out + pto.c_tp_itemsize = 0 return rffi.cast(PyObject, heaptype) @@ -511,8 +464,6 @@ pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) else: pto.c_tp_name = rffi.str2charp(w_type.name) - pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out - pto.c_tp_itemsize = 0 # uninitialized fields: # c_tp_print, c_tp_getattr, c_tp_setattr # XXX implement @@ -520,8 +471,11 @@ w_base = best_base(space, w_type.bases_w) pto.c_tp_base = rffi.cast(PyTypeObjectPtr, make_ref(space, w_base)) - finish_type_1(space, pto) - finish_type_2(space, pto, w_type) + if hasattr(space, '_cpyext_type_init'): + space._cpyext_type_init.append((pto, w_type)) + else: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) pto.c_tp_basicsize = rffi.sizeof(typedescr.basestruct) if pto.c_tp_base: From pypy.commits at gmail.com Wed Jan 27 13:03:01 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 10:03:01 -0800 (PST) Subject: [pypy-commit] pypy cpyext-bootstrap: Seems to work, merging Message-ID: <56a90655.6918c20a.2bb00.ffffc6f5@mx.google.com> Author: Armin Rigo Branch: cpyext-bootstrap Changeset: r81976:0fac859d23a0 Date: 2016-01-27 18:58 +0100 http://bitbucket.org/pypy/pypy/changeset/0fac859d23a0/ Log: Seems to work, merging From pypy.commits at gmail.com Wed Jan 27 13:12:05 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 10:12:05 -0800 (PST) Subject: [pypy-commit] pypy default: ignore branch Message-ID: <56a90875.01941c0a.adaea.3b7a@mx.google.com> Author: Armin Rigo Branch: Changeset: r81978:5edae6a9f5e3 Date: 2016-01-27 19:10 +0100 http://bitbucket.org/pypy/pypy/changeset/5edae6a9f5e3/ Log: ignore branch diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -133,3 +133,4 @@ `rpython/jit/metainterp/optimizeopt/pure.py`, which can result in better codegen for traces containing a large number of pure getfield operations. +.. branch: cpyext-bootstrap From pypy.commits at gmail.com Wed Jan 27 13:13:09 2016 From: pypy.commits at gmail.com (arigo) Date: Wed, 27 Jan 2016 10:13:09 -0800 (PST) Subject: [pypy-commit] pypy cpyext-gc-support-2: hg merge default Message-ID: <56a908b5.a151c20a.c3de6.ffffcd35@mx.google.com> Author: Armin Rigo Branch: cpyext-gc-support-2 Changeset: r81979:1fc97a564c99 Date: 2016-01-27 19:12 +0100 http://bitbucket.org/pypy/pypy/changeset/1fc97a564c99/ Log: hg merge default diff too long, truncating to 2000 out of 4461 lines diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py --- a/pypy/config/pypyoption.py +++ b/pypy/config/pypyoption.py @@ -85,7 +85,8 @@ module_dependencies = { '_multiprocessing': [('objspace.usemodules.time', True), ('objspace.usemodules.thread', True)], - 'cpyext': [('objspace.usemodules.array', True)], + 'cpyext': [('objspace.usemodules.array', True), + ('objspace.usemodules.micronumpy', True)], 'cppyy': [('objspace.usemodules.cpyext', True)], } module_suggests = { diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -54,7 +54,8 @@ It is quite common nowadays that xyz is available on PyPI_ and installable with ``pip install xyz``. The simplest solution is to `use virtualenv (as documented here)`_. Then enter (activate) the virtualenv -and type: ``pip install xyz``. +and type: ``pip install xyz``. If you don't know or don't want virtualenv, +you can also install ``pip`` globally by saying ``pypy -m ensurepip``. If you get errors from the C compiler, the module is a CPython C Extension module using unsupported features. `See below.`_ diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -123,3 +123,13 @@ .. branch: fix-cpython-ssl-tests-2.7 Fix SSL tests by importing cpython's patch + +.. branch: remove-getfield-pure + +Remove pure variants of ``getfield_gc_*`` operations from the JIT. Relevant +optimizations instead consult the field descriptor to determine the purity of +the operation. Additionally, pure ``getfield`` operations are now handled +entirely by `rpython/jit/metainterp/optimizeopt/heap.py` rather than +`rpython/jit/metainterp/optimizeopt/pure.py`, which can result in better codegen +for traces containing a large number of pure getfield operations. + diff --git a/pypy/module/cpyext/Doc_stubgen_enable.patch b/pypy/module/cpyext/Doc_stubgen_enable.patch deleted file mode 100644 --- a/pypy/module/cpyext/Doc_stubgen_enable.patch +++ /dev/null @@ -1,27 +0,0 @@ -Index: Doc/tools/sphinx/ext/refcounting.py -=================================================================== ---- Doc/tools/sphinx/ext/refcounting.py (Revision 79453) -+++ Doc/tools/sphinx/ext/refcounting.py (Arbeitskopie) -@@ -91,6 +91,7 @@ - if app.config.refcount_file: - refcounts = Refcounts.fromfile( - path.join(app.srcdir, app.config.refcount_file)) -+ app._refcounts = refcounts - app.connect('doctree-read', refcounts.add_refcount_annotations) - - -Index: Doc/conf.py -=================================================================== ---- Doc/conf.py (Revision 79421) -+++ Doc/conf.py (Arbeitskopie) -@@ -13,8 +13,8 @@ - # General configuration - # --------------------- - --extensions = ['sphinx.ext.refcounting', 'sphinx.ext.coverage', -- 'sphinx.ext.doctest', 'pyspecific'] -+extensions = ['pypy.module.cpyext.stubgen', 'sphinx.ext.refcounting', 'sphinx.ext.coverage', -+ 'sphinx.ext.doctest', 'pyspecific', ] - templates_path = ['tools/sphinxext'] - - # General substitutions. diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -441,8 +441,8 @@ TYPES = {} GLOBALS = { # this needs to include all prebuilt pto, otherwise segfaults occur '_Py_NoneStruct#': ('PyObject*', 'space.w_None'), - '_Py_TrueStruct#': ('PyObject*', 'space.w_True'), - '_Py_ZeroStruct#': ('PyObject*', 'space.w_False'), + '_Py_TrueStruct#': ('PyIntObject*', 'space.w_True'), + '_Py_ZeroStruct#': ('PyIntObject*', 'space.w_False'), '_Py_NotImplementedStruct#': ('PyObject*', 'space.w_NotImplemented'), '_Py_EllipsisObject#': ('PyObject*', 'space.w_Ellipsis'), 'PyDateTimeAPI': ('PyDateTime_CAPI*', 'None'), @@ -505,7 +505,9 @@ def get_structtype_for_ctype(ctype): from pypy.module.cpyext.typeobjectdefs import PyTypeObjectPtr from pypy.module.cpyext.cdatetime import PyDateTime_CAPI + from pypy.module.cpyext.intobject import PyIntObject return {"PyObject*": PyObject, "PyTypeObject*": PyTypeObjectPtr, + "PyIntObject*": PyIntObject, "PyDateTime_CAPI*": lltype.Ptr(PyDateTime_CAPI)}[ctype] # Note: as a special case, "PyObject" is the pointer type in RPython, @@ -846,6 +848,7 @@ space.fromcache(State).install_dll(eci) # populate static data + builder = StaticObjectBuilder(space) for name, (typ, expr) in GLOBALS.iteritems(): from pypy.module import cpyext # for the eval() below w_obj = eval(expr) @@ -870,7 +873,7 @@ assert False, "Unknown static pointer: %s %s" % (typ, name) ptr.value = ctypes.cast(ll2ctypes.lltype2ctypes(value), ctypes.c_void_p).value - elif typ in ('PyObject*', 'PyTypeObject*'): + elif typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): if name.startswith('PyPyExc_') or name.startswith('cpyexttestExc_'): # we already have the pointer in_dll = ll2ctypes.get_ctypes_type(PyObject).in_dll(bridge, name) @@ -879,17 +882,10 @@ # we have a structure, get its address in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, name) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) - from pypy.module.cpyext.pyobject import ( - track_reference, get_typedescr) - w_type = space.type(w_obj) - typedescr = get_typedescr(w_type.instancetypedef) - py_obj.c_ob_refcnt = 1 - py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, - make_ref(space, w_type)) - typedescr.attach(space, py_obj, w_obj) - track_reference(space, py_obj, w_obj) + builder.prepare(py_obj, w_obj) else: assert False, "Unknown static object: %s %s" % (typ, name) + builder.attach_all() pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') @@ -906,6 +902,36 @@ setup_init_functions(eci, translating=False) return modulename.new(ext='') + +class StaticObjectBuilder: + def __init__(self, space): + self.space = space + self.to_attach = [] + + def prepare(self, py_obj, w_obj): + from pypy.module.cpyext.pyobject import track_reference + py_obj.c_ob_refcnt = 1 + track_reference(self.space, py_obj, w_obj) + self.to_attach.append((py_obj, w_obj)) + + def attach_all(self): + from pypy.module.cpyext.pyobject import get_typedescr, make_ref + from pypy.module.cpyext.typeobject import finish_type_1, finish_type_2 + space = self.space + space._cpyext_type_init = [] + for py_obj, w_obj in self.to_attach: + w_type = space.type(w_obj) + typedescr = get_typedescr(w_type.instancetypedef) + py_obj.c_ob_type = rffi.cast(PyTypeObjectPtr, + make_ref(space, w_type)) + typedescr.attach(space, py_obj, w_obj) + cpyext_type_init = space._cpyext_type_init + del space._cpyext_type_init + for pto, w_type in cpyext_type_init: + finish_type_1(space, pto) + finish_type_2(space, pto, w_type) + + def mangle_name(prefix, name): if name.startswith('Py'): return prefix + name[2:] @@ -1100,14 +1126,33 @@ run_bootstrap_functions(space) setup_va_functions(eci) + from pypy.module import cpyext # for eval() below + + # Set up the types. Needs a special case, because of the + # immediate cycle involving 'c_ob_type', and because we don't + # want these types to be Py_TPFLAGS_HEAPTYPE. + static_types = {} + for name, (typ, expr) in GLOBALS.items(): + if typ == 'PyTypeObject*': + pto = lltype.malloc(PyTypeObject, immortal=True, + zero=True, flavor='raw') + pto.c_ob_refcnt = 1 + pto.c_tp_basicsize = -1 + static_types[name] = pto + builder = StaticObjectBuilder(space) + for name, pto in static_types.items(): + pto.c_ob_type = static_types['PyType_Type#'] + w_type = eval(GLOBALS[name][1]) + builder.prepare(rffi.cast(PyObject, pto), w_type) + builder.attach_all() + # populate static data for name, (typ, expr) in GLOBALS.iteritems(): name = name.replace("#", "") if name.startswith('PyExc_'): name = '_' + name - from pypy.module import cpyext w_obj = eval(expr) - if typ in ('PyObject*', 'PyTypeObject*'): + if typ in ('PyObject*', 'PyTypeObject*', 'PyIntObject*'): struct_ptr = make_ref(space, w_obj) elif typ == 'PyDateTime_CAPI*': continue diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -59,7 +59,7 @@ return None return borrow_from(w_dict, w_res) - at cpython_api([PyObject, rffi.CCHARP], rffi.INT_real, error=-1) + at cpython_api([PyObject, CONST_STRING], rffi.INT_real, error=-1) def PyDict_DelItemString(space, w_dict, key_ptr): """Remove the entry in dictionary p which has a key specified by the string key. Return 0 on success or -1 on failure.""" diff --git a/pypy/module/cpyext/eval.py b/pypy/module/cpyext/eval.py --- a/pypy/module/cpyext/eval.py +++ b/pypy/module/cpyext/eval.py @@ -128,7 +128,7 @@ filename = "" return run_string(space, source, filename, start, w_globals, w_locals) - at cpython_api([rffi.CCHARP, rffi.INT_real, PyObject, PyObject, + at cpython_api([CONST_STRING, rffi.INT_real, PyObject, PyObject, PyCompilerFlagsPtr], PyObject) def PyRun_StringFlags(space, source, start, w_globals, w_locals, flagsptr): """Execute Python source code from str in the context specified by the @@ -189,7 +189,7 @@ pi[0] = space.getindex_w(w_obj, None) return 1 - at cpython_api([rffi.CCHARP, rffi.CCHARP, rffi.INT_real, PyCompilerFlagsPtr], + at cpython_api([CONST_STRING, CONST_STRING, rffi.INT_real, PyCompilerFlagsPtr], PyObject) def Py_CompileStringFlags(space, source, filename, start, flagsptr): """Parse and compile the Python source code in str, returning the diff --git a/pypy/module/cpyext/patches/Doc_stubgen_enable.patch b/pypy/module/cpyext/patches/Doc_stubgen_enable.patch new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/patches/Doc_stubgen_enable.patch @@ -0,0 +1,27 @@ +Index: Doc/tools/sphinx/ext/refcounting.py +=================================================================== +--- Doc/tools/sphinx/ext/refcounting.py (Revision 79453) ++++ Doc/tools/sphinx/ext/refcounting.py (Arbeitskopie) +@@ -91,6 +91,7 @@ + if app.config.refcount_file: + refcounts = Refcounts.fromfile( + path.join(app.srcdir, app.config.refcount_file)) ++ app._refcounts = refcounts + app.connect('doctree-read', refcounts.add_refcount_annotations) + + +Index: Doc/conf.py +=================================================================== +--- Doc/conf.py (Revision 79421) ++++ Doc/conf.py (Arbeitskopie) +@@ -13,8 +13,8 @@ + # General configuration + # --------------------- + +-extensions = ['sphinx.ext.refcounting', 'sphinx.ext.coverage', +- 'sphinx.ext.doctest', 'pyspecific'] ++extensions = ['pypy.module.cpyext.stubgen', 'sphinx.ext.refcounting', 'sphinx.ext.coverage', ++ 'sphinx.ext.doctest', 'pyspecific', ] + templates_path = ['tools/sphinxext'] + + # General substitutions. diff --git a/pypy/module/cpyext/pystrtod.py b/pypy/module/cpyext/pystrtod.py --- a/pypy/module/cpyext/pystrtod.py +++ b/pypy/module/cpyext/pystrtod.py @@ -1,6 +1,6 @@ import errno from pypy.interpreter.error import OperationError -from pypy.module.cpyext.api import cpython_api +from pypy.module.cpyext.api import cpython_api, CONST_STRING from pypy.module.cpyext.pyobject import PyObject from rpython.rlib import rdtoa from rpython.rlib import rfloat @@ -22,7 +22,7 @@ rfloat.DIST_NAN: Py_DTST_NAN } - at cpython_api([rffi.CCHARP, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) + at cpython_api([CONST_STRING, rffi.CCHARPP, PyObject], rffi.DOUBLE, error=-1.0) @jit.dont_look_inside # direct use of _get_errno() def PyOS_string_to_double(space, s, endptr, w_overflow_exception): """Convert a string s to a double, raising a Python diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -374,6 +374,11 @@ module = self.import_extension('foo', [ ("test_type", "METH_O", ''' + /* "args->ob_type" is a strange way to get at 'type', + which should have a different tp_getattro/tp_setattro + than its tp_base, which is 'object'. + */ + if (!args->ob_type->tp_setattro) { PyErr_SetString(PyExc_ValueError, "missing tp_setattro"); @@ -382,8 +387,12 @@ if (args->ob_type->tp_setattro == args->ob_type->tp_base->tp_setattro) { - PyErr_SetString(PyExc_ValueError, "recursive tp_setattro"); - return NULL; + /* Note that unlike CPython, in PyPy 'type.tp_setattro' + is the same function as 'object.tp_setattro'. This + test used to check that it was not, but that was an + artifact of the bootstrap logic only---in the final + C sources I checked and they are indeed the same. + So we ignore this problem here. */ } if (!args->ob_type->tp_getattro) { diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -146,7 +146,7 @@ assert len(slot_names) == 2 struct = getattr(pto, slot_names[0]) if not struct: - assert not space.config.translating + #assert not space.config.translating assert not pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE if slot_names[0] == 'c_tp_as_number': STRUCT_TYPE = PyNumberMethods @@ -310,36 +310,6 @@ realize=type_realize, dealloc=type_dealloc) - # There is the obvious cycle of 'type(type) == type', but there are - # also several other ones, like 'tuple.tp_bases' being itself a - # tuple instance. We solve the first one by creating the type - # "type" manually here. For the other cycles, we fix them by delaying - # creation of the types here, and hoping nothing breaks by seeing - # uninitialized-yet types (only for a few basic types like 'type', - # 'tuple', 'object', 'str'). - space._cpyext_delay_type_creation = [] - - py_type = _type_alloc(space, lltype.nullptr(PyTypeObject)) - py_type.c_ob_type = rffi.cast(PyTypeObjectPtr, py_type) - track_reference(space, py_type, space.w_type) - type_attach(space, py_type, space.w_type) - - as_pyobj(space, space.w_str) - as_pyobj(space, space.w_tuple) - as_pyobj(space, space.w_object) - - delayed_types = [] - while space._cpyext_delay_type_creation: - (py_obj, w_type) = space._cpyext_delay_type_creation.pop() - _type_really_attach(space, py_obj, w_type) - delayed_types.append((py_obj, w_type)) - del space._cpyext_delay_type_creation - for py_obj, w_type in delayed_types: - pto = rffi.cast(PyTypeObjectPtr, py_obj) - finish_type_1(space, pto) - finish_type_2(space, pto, w_type) - finish_type_3(space, pto, w_type) - @cpython_api([PyObject], lltype.Void, external=False) def subtype_dealloc(space, obj): @@ -440,16 +410,13 @@ def type_alloc(space, w_metatype): - metatype = make_ref(space, w_metatype) - metatype = rffi.cast(PyTypeObjectPtr, metatype) - assert metatype + metatype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_metatype)) # Don't increase refcount for non-heaptypes - flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) - if not flags & Py_TPFLAGS_HEAPTYPE: - Py_DecRef(space, w_metatype) - return _type_alloc(space, metatype) + if metatype: + flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) + if not flags & Py_TPFLAGS_HEAPTYPE: + Py_DecRef(space, w_metatype) -def _type_alloc(space, metatype): heaptype = lltype.malloc(PyHeapTypeObject.TO, flavor='raw', zero=True) pto = heaptype.c_ht_type @@ -460,7 +427,6 @@ pto.c_tp_as_sequence = heaptype.c_as_sequence pto.c_tp_as_mapping = heaptype.c_as_mapping pto.c_tp_as_buffer = heaptype.c_as_buffer - pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out pto.c_tp_itemsize = 0 @@ -470,13 +436,6 @@ """ Fills a newly allocated PyTypeObject from an existing type. """ - if hasattr(space, '_cpyext_delay_type_creation'): - space._cpyext_delay_type_creation.append((py_obj, w_type)) - else: - _type_really_attach(space, py_obj, w_type) - return rffi.cast(PyTypeObjectPtr, py_obj) - -def _type_really_attach(space, py_obj, w_type): from pypy.module.cpyext.object import PyObject_Del assert isinstance(w_type, W_TypeObject) @@ -497,15 +456,24 @@ PyObject_Del.api_func.get_wrapper(space)) pto.c_tp_alloc = llhelper(PyType_GenericAlloc.api_func.functype, PyType_GenericAlloc.api_func.get_wrapper(space)) + if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: + w_typename = space.getattr(w_type, space.wrap('__name__')) + heaptype = rffi.cast(PyHeapTypeObject, pto) + heaptype.c_ht_name = make_ref(space, w_typename) + from pypy.module.cpyext.stringobject import PyString_AsString + pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) + else: + pto.c_tp_name = rffi.str2charp(w_type.name) # uninitialized fields: # c_tp_print, c_tp_getattr, c_tp_setattr # XXX implement # c_tp_compare and the following fields (see http://docs.python.org/c-api/typeobj.html ) w_base = best_base(space, w_type.bases_w) - py_base = make_ref(space, w_base) - pto.c_tp_base = rffi.cast(PyTypeObjectPtr, py_base) + pto.c_tp_base = rffi.cast(PyTypeObjectPtr, make_ref(space, w_base)) - if not hasattr(space, '_cpyext_delay_type_creation'): + if hasattr(space, '_cpyext_type_init'): + space._cpyext_type_init.append((pto, w_type)) + else: finish_type_1(space, pto) finish_type_2(space, pto, w_type) @@ -519,21 +487,8 @@ if space.is_w(w_type, space.w_object): pto.c_tp_new = rffi.cast(newfunc, 1) update_all_slots(space, w_type, pto) - - if not hasattr(space, '_cpyext_delay_type_creation'): - finish_type_3(space, pto, w_type) - pto.c_tp_flags |= Py_TPFLAGS_READY - -def finish_type_3(space, pto, w_type): - if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: - w_typename = space.getattr(w_type, space.wrap('__name__')) - heaptype = rffi.cast(PyHeapTypeObject, pto) - heaptype.c_ht_name = make_ref(space, w_typename) - from pypy.module.cpyext.stringobject import PyString_AsString - pto.c_tp_name = PyString_AsString(space, heaptype.c_ht_name) - else: - pto.c_tp_name = rffi.str2charp(w_type.name) + return pto def py_type_ready(space, pto): if pto.c_tp_flags & Py_TPFLAGS_READY: diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py --- a/pypy/module/pypyjit/test_pypy_c/test_call.py +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py @@ -83,9 +83,9 @@ p38 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p39 = getfield_gc_r(p38, descr=) i40 = force_token() - p41 = getfield_gc_pure_r(p38, descr=) + p41 = getfield_gc_r(p38, descr=) guard_value(p41, ConstPtr(ptr42), descr=...) - i42 = getfield_gc_pure_i(p38, descr=) + i42 = getfield_gc_i(p38, descr=) i43 = int_is_zero(i42) guard_true(i43, descr=...) i50 = force_token() @@ -435,21 +435,21 @@ guard_isnull(p5, descr=...) guard_nonnull_class(p12, ConstClass(W_IntObject), descr=...) guard_value(p2, ConstPtr(ptr21), descr=...) - i22 = getfield_gc_pure_i(p12, descr=) + i22 = getfield_gc_i(p12, descr=) i24 = int_lt(i22, 5000) guard_true(i24, descr=...) guard_not_invalidated(descr=...) p29 = call_r(ConstClass(_ll_1_threadlocalref_get__Ptr_GcStruct_objectLlT_Signed), #, descr=) p30 = getfield_gc_r(p29, descr=) p31 = force_token() - p32 = getfield_gc_pure_r(p29, descr=) + p32 = getfield_gc_r(p29, descr=) guard_value(p32, ConstPtr(ptr33), descr=...) - i34 = getfield_gc_pure_i(p29, descr=) + i34 = getfield_gc_i(p29, descr=) i35 = int_is_zero(i34) guard_true(i35, descr=...) p37 = getfield_gc_r(ConstPtr(ptr36), descr=) guard_nonnull_class(p37, ConstClass(W_IntObject), descr=...) - i39 = getfield_gc_pure_i(p37, descr=) + i39 = getfield_gc_i(p37, descr=) i40 = int_add_ovf(i22, i39) guard_no_overflow(descr=...) --TICK-- @@ -466,7 +466,7 @@ """, []) loop, = log.loops_by_id('call') assert loop.match(""" - i8 = getfield_gc_pure_i(p6, descr=) + i8 = getfield_gc_i(p6, descr=) i10 = int_lt(i8, 5000) guard_true(i10, descr=...) guard_not_invalidated? diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py --- a/pypy/module/pypyjit/test_pypy_c/test_containers.py +++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py @@ -84,7 +84,7 @@ guard_no_exception(descr=...) p20 = new_with_vtable(descr=...) call_n(ConstClass(_ll_dict_setitem_lookup_done_trampoline), p13, p10, p20, i12, i17, descr=) - setfield_gc(p20, i5, descr=) + setfield_gc(p20, i5, descr=) guard_no_exception(descr=...) i23 = call_i(ConstClass(ll_call_lookup_function), p13, p10, i12, 0, descr=) guard_no_exception(descr=...) @@ -93,7 +93,7 @@ p28 = getfield_gc_r(p13, descr=) p29 = getinteriorfield_gc_r(p28, i23, descr=>) guard_nonnull_class(p29, ConstClass(W_IntObject), descr=...) - i31 = getfield_gc_pure_i(p29, descr=) + i31 = getfield_gc_i(p29, descr=) i32 = int_sub_ovf(i31, i5) guard_no_overflow(descr=...) i34 = int_add_ovf(i32, 1) diff --git a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py --- a/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py +++ b/pypy/module/pypyjit/test_pypy_c/test_micronumpy.py @@ -101,13 +101,13 @@ loop = log._filter(log.loops[0]) assert loop.match(""" guard_class(p1, #, descr=...) - p4 = getfield_gc_pure_r(p1, descr=) + p4 = getfield_gc_r(p1, descr=) i5 = getfield_gc_i(p0, descr=) - p6 = getfield_gc_pure_r(p4, descr=) - p7 = getfield_gc_pure_r(p6, descr=) + p6 = getfield_gc_r(p4, descr=) + p7 = getfield_gc_r(p6, descr=) guard_class(p7, ConstClass(Float64), descr=...) - i9 = getfield_gc_pure_i(p4, descr=) - i10 = getfield_gc_pure_i(p6, descr=) + i9 = getfield_gc_i(p4, descr=) + i10 = getfield_gc_i(p6, descr=) i12 = int_eq(i10, 61) i14 = int_eq(i10, 60) i15 = int_or(i12, i14) @@ -117,28 +117,28 @@ i18 = float_ne(f16, 0.000000) guard_true(i18, descr=...) guard_nonnull_class(p2, ConstClass(W_BoolBox), descr=...) - i20 = getfield_gc_pure_i(p2, descr=) + i20 = getfield_gc_i(p2, descr=) i21 = int_is_true(i20) guard_false(i21, descr=...) i22 = getfield_gc_i(p0, descr=) - i23 = getfield_gc_pure_i(p1, descr=) + i23 = getfield_gc_i(p1, descr=) guard_true(i23, descr=...) i25 = int_add(i22, 1) - p26 = getfield_gc_pure_r(p0, descr=) - i27 = getfield_gc_pure_i(p1, descr=) + p26 = getfield_gc_r(p0, descr=) + i27 = getfield_gc_i(p1, descr=) i28 = int_is_true(i27) guard_true(i28, descr=...) - i29 = getfield_gc_pure_i(p6, descr=) + i29 = getfield_gc_i(p6, descr=) guard_value(i29, 8, descr=...) i30 = int_add(i5, 8) - i31 = getfield_gc_pure_i(p1, descr=) + i31 = getfield_gc_i(p1, descr=) i32 = int_ge(i25, i31) guard_false(i32, descr=...) p34 = new_with_vtable(descr=...) {{{ - setfield_gc(p34, p1, descr=) + setfield_gc(p34, p1, descr=

Python 3.2.5 compatible PyPy3 2.4.0

Linux x86 binary (32bit, tar.bz2 built on Ubuntu 10.04.4 LTS) (see [1] below)
Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 14.04) (see [1] below)

Branch: SomeRange Changeset: r82006:02f319beadff Date: 2016-01-30 04:47 +0000 http://bitbucket.org/pypy/pypy/changeset/02f319beadff/ Log: Separate bk.newrange() from bk.newlist() diff --git a/rpython/annotator/bookkeeper.py b/rpython/annotator/bookkeeper.py --- a/rpython/annotator/bookkeeper.py +++ b/rpython/annotator/bookkeeper.py @@ -180,15 +180,18 @@ listdef.listitem.__dict__.update(flags_if_new) return listdef - def newlist(self, *s_values, **flags): + def newlist(self, *s_values): """Make a SomeList associated with the current position, general enough to contain the s_values as items.""" - listdef = self.getlistdef(**flags) + listdef = self.getlistdef() for s_value in s_values: listdef.generalize(s_value) - if flags: - assert flags.keys() == ['range_step'] - listdef.generalize_range_step(flags['range_step']) + return SomeList(listdef) + + def newrange(self, s_item, step): + listdef = self.getlistdef(range_step=step) + listdef.generalize(s_item) + listdef.generalize_range_step(step) return SomeList(listdef) def getdictdef(self, is_r_dict=False, force_non_null=False): diff --git a/rpython/annotator/builtin.py b/rpython/annotator/builtin.py --- a/rpython/annotator/builtin.py +++ b/rpython/annotator/builtin.py @@ -80,7 +80,7 @@ nonneg = s_stop.nonneg or (s_stop.is_constant() and s_stop.const >= -1) s_item = SomeInteger(nonneg=nonneg) - return getbookkeeper().newlist(s_item, range_step=step) + return getbookkeeper().newrange(s_item, step) builtin_xrange = builtin_range # xxx for now allow it From pypy.commits at gmail.com Sat Jan 30 13:42:39 2016 From: pypy.commits at gmail.com (rlamy) Date: Sat, 30 Jan 2016 10:42:39 -0800 (PST) Subject: [pypy-commit] pypy SomeRange: Create SomeRange class Message-ID: <56ad041f.11301c0a.4a215.6b16@mx.google.com> Author: Ronan Lamy Branch: SomeRange Changeset: r82008:273c3ace813a Date: 2016-01-30 17:40 +0000 http://bitbucket.org/pypy/pypy/changeset/273c3ace813a/ Log: Create SomeRange class diff --git a/rpython/annotator/bookkeeper.py b/rpython/annotator/bookkeeper.py --- a/rpython/annotator/bookkeeper.py +++ b/rpython/annotator/bookkeeper.py @@ -13,7 +13,8 @@ SomeOrderedDict, SomeString, SomeChar, SomeFloat, unionof, SomeInstance, SomeDict, SomeBuiltin, SomePBC, SomeInteger, TLS, SomeUnicodeCodePoint, s_None, s_ImpossibleValue, SomeBool, SomeTuple, SomeException, - SomeImpossibleValue, SomeUnicodeString, SomeList, HarmlesslyBlocked, + SomeImpossibleValue, SomeUnicodeString, SomeList, SomeRange, + HarmlesslyBlocked, SomeWeakRef, SomeByteArray, SomeConstantType, SomeProperty) from rpython.annotator.classdesc import ClassDef, ClassDesc from rpython.annotator.listdef import ListDef, ListItem @@ -190,7 +191,7 @@ listdef.listitem.range_step = step listdef.generalize(s_item) listdef.generalize_range_step(step) - return SomeList(listdef) + return SomeRange(listdef) def getdictdef(self, is_r_dict=False, force_non_null=False): """Get the DictDef associated with the current position.""" diff --git a/rpython/annotator/model.py b/rpython/annotator/model.py --- a/rpython/annotator/model.py +++ b/rpython/annotator/model.py @@ -359,6 +359,9 @@ def noneify(self): return SomeList(self.listdef) +class SomeRange(SomeList): + pass + class SomeTuple(SomeObject): "Stands for a tuple of known length." diff --git a/rpython/rtyper/rrange.py b/rpython/rtyper/rrange.py --- a/rpython/rtyper/rrange.py +++ b/rpython/rtyper/rrange.py @@ -89,6 +89,9 @@ v_step = self._getstep(v_rng, hop) return hop.gendirectcall(ll_rangelen, v_rng, v_step) + def __eq__(self, other): + return other.__class__ is self.__class__ and other.__dict__ == self.__dict__ + class __extend__(pairtype(RangeRepr, IntegerRepr)): From pypy.commits at gmail.com Sat Jan 30 13:42:40 2016 From: pypy.commits at gmail.com (rlamy) Date: Sat, 30 Jan 2016 10:42:40 -0800 (PST) Subject: [pypy-commit] pypy SomeRange: Make range() lists immutable Message-ID: <56ad0420.cb371c0a.d64e2.36dd@mx.google.com> Author: Ronan Lamy Branch: SomeRange Changeset: r82009:3b3f8cea104f Date: 2016-01-30 18:16 +0000 http://bitbucket.org/pypy/pypy/changeset/3b3f8cea104f/ Log: Make range() lists immutable diff --git a/rpython/annotator/binaryop.py b/rpython/annotator/binaryop.py --- a/rpython/annotator/binaryop.py +++ b/rpython/annotator/binaryop.py @@ -9,7 +9,7 @@ SomeDict, SomeUnicodeCodePoint, SomeUnicodeString, SomeException, SomeTuple, SomeImpossibleValue, s_ImpossibleValue, SomeInstance, SomeBuiltinMethod, SomeIterator, SomePBC, SomeNone, SomeFloat, s_None, - SomeByteArray, SomeWeakRef, SomeSingleFloat, + SomeByteArray, SomeWeakRef, SomeSingleFloat, SomeRange, SomeLongFloat, SomeType, SomeTypeOf, SomeConstantType, unionof, UnionError, read_can_only_throw, add_knowntypedata, merge_knowntypedata,) @@ -486,6 +486,12 @@ return lst1 inplace_mul.can_only_throw = [] +class __extend__(pairtype(SomeRange, SomeObject)): + + def inplace_mul((lst1, obj2)): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + class __extend__(pairtype(SomeTuple, SomeTuple)): def union((tup1, tup2)): diff --git a/rpython/annotator/unaryop.py b/rpython/annotator/unaryop.py --- a/rpython/annotator/unaryop.py +++ b/rpython/annotator/unaryop.py @@ -14,7 +14,7 @@ SomeUnicodeCodePoint, SomeInstance, SomeBuiltin, SomeBuiltinMethod, SomeFloat, SomeIterator, SomePBC, SomeNone, SomeTypeOf, s_ImpossibleValue, s_Bool, s_None, s_Int, unionof, add_knowntypedata, - SomeWeakRef, SomeUnicodeString, SomeByteArray) + SomeWeakRef, SomeUnicodeString, SomeByteArray, SomeRange) from rpython.annotator.bookkeeper import getbookkeeper, immutablevalue from rpython.annotator.binaryop import _clone ## XXX where to put this? from rpython.annotator.binaryop import _dict_can_only_throw_keyerror @@ -429,6 +429,32 @@ check_negative_slice(s_start, s_stop) self.listdef.resize() +class __extend__(SomeRange): + + def method_append(self, s_value): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + + def method_extend(self, s_iterable): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + + def method_reverse(self): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + + def method_insert(self, s_index, s_value): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + + def method_remove(self, s_value): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + + def method_pop(self, s_index=None): + raise AnnotatorError( + "In RPython, lists returned by range() are immutable") + def check_negative_slice(s_start, s_stop, error="slicing"): if isinstance(s_start, SomeInteger) and not s_start.nonneg: raise AnnotatorError("%s: not proven to have non-negative start" % From pypy.commits at gmail.com Sat Jan 30 13:42:37 2016 From: pypy.commits at gmail.com (rlamy) Date: Sat, 30 Jan 2016 10:42:37 -0800 (PST) Subject: [pypy-commit] pypy SomeRange: Inline bk.getlistdef() Message-ID: <56ad041d.2457c20a.d484b.ffffca67@mx.google.com> Author: Ronan Lamy Branch: SomeRange Changeset: r82007:b1921df13e50 Date: 2016-01-30 06:03 +0000 http://bitbucket.org/pypy/pypy/changeset/b1921df13e50/ Log: Inline bk.getlistdef() diff --git a/rpython/annotator/bookkeeper.py b/rpython/annotator/bookkeeper.py --- a/rpython/annotator/bookkeeper.py +++ b/rpython/annotator/bookkeeper.py @@ -171,25 +171,23 @@ clsdefs = {self.getuniqueclassdef(cls) for cls in exc_classes} return SomeException(clsdefs) - def getlistdef(self, **flags_if_new): - """Get the ListDef associated with the current position.""" + def newlist(self, *s_values): + """Make a SomeList associated with the current position, general + enough to contain the s_values as items.""" try: listdef = self.listdefs[self.position_key] except KeyError: listdef = self.listdefs[self.position_key] = ListDef(self) - listdef.listitem.__dict__.update(flags_if_new) - return listdef - - def newlist(self, *s_values): - """Make a SomeList associated with the current position, general - enough to contain the s_values as items.""" - listdef = self.getlistdef() for s_value in s_values: listdef.generalize(s_value) return SomeList(listdef) def newrange(self, s_item, step): - listdef = self.getlistdef(range_step=step) + try: + listdef = self.listdefs[self.position_key] + except KeyError: + listdef = self.listdefs[self.position_key] = ListDef(self) + listdef.listitem.range_step = step listdef.generalize(s_item) listdef.generalize_range_step(step) return SomeList(listdef) From pypy.commits at gmail.com Sat Jan 30 13:42:33 2016 From: pypy.commits at gmail.com (rlamy) Date: Sat, 30 Jan 2016 10:42:33 -0800 (PST) Subject: [pypy-commit] pypy SomeRange: Merge rpython.rtyper.lltypesystem.rrange into rpython.rtyper.rrange Message-ID: <56ad0419.8378c20a.4e5cb.ffffc96f@mx.google.com> Author: Ronan Lamy Branch: SomeRange Changeset: r82005:d458d86b6ce2 Date: 2016-01-29 19:08 +0000 http://bitbucket.org/pypy/pypy/changeset/d458d86b6ce2/ Log: Merge rpython.rtyper.lltypesystem.rrange into rpython.rtyper.rrange diff --git a/rpython/rtyper/lltypesystem/rrange.py b/rpython/rtyper/lltypesystem/rrange.py deleted file mode 100644 --- a/rpython/rtyper/lltypesystem/rrange.py +++ /dev/null @@ -1,98 +0,0 @@ -from rpython.rtyper.lltypesystem.lltype import Ptr, GcStruct, Signed, malloc, Void -from rpython.rtyper.rrange import AbstractRangeRepr, AbstractRangeIteratorRepr -from rpython.rtyper.error import TyperError - -# ____________________________________________________________ -# -# Concrete implementation of RPython lists that are returned by range() -# and never mutated afterwards: -# -# struct range { -# Signed start, stop; // step is always constant -# } -# -# struct rangest { -# Signed start, stop, step; // rare case, for completeness -# } - -def ll_length(l): - if l.step > 0: - lo = l.start - hi = l.stop - step = l.step - else: - lo = l.stop - hi = l.start - step = -l.step - if hi <= lo: - return 0 - n = (hi - lo - 1) // step + 1 - return n - -def ll_getitem_fast(l, index): - return l.start + index * l.step - -RANGEST = GcStruct("range", ("start", Signed), ("stop", Signed), ("step", Signed), - adtmeths = { - "ll_length":ll_length, - "ll_getitem_fast":ll_getitem_fast, - }, - hints = {'immutable': True}) -RANGESTITER = GcStruct("range", ("next", Signed), ("stop", Signed), ("step", Signed)) - -class RangeRepr(AbstractRangeRepr): - - RANGEST = Ptr(RANGEST) - RANGESTITER = Ptr(RANGESTITER) - - getfield_opname = "getfield" - - def __init__(self, step, *args): - self.RANGE = Ptr(GcStruct("range", ("start", Signed), ("stop", Signed), - adtmeths = { - "ll_length":ll_length, - "ll_getitem_fast":ll_getitem_fast, - "step":step, - }, - hints = {'immutable': True})) - self.RANGEITER = Ptr(GcStruct("range", ("next", Signed), ("stop", Signed))) - AbstractRangeRepr.__init__(self, step, *args) - self.ll_newrange = ll_newrange - self.ll_newrangest = ll_newrangest - - def make_iterator_repr(self, variant=None): - if variant is not None: - raise TyperError("unsupported %r iterator over a range list" % - (variant,)) - return RangeIteratorRepr(self) - - -def ll_newrange(RANGE, start, stop): - l = malloc(RANGE.TO) - l.start = start - l.stop = stop - return l - -def ll_newrangest(start, stop, step): - if step == 0: - raise ValueError - l = malloc(RANGEST) - l.start = start - l.stop = stop - l.step = step - return l - -class RangeIteratorRepr(AbstractRangeIteratorRepr): - - def __init__(self, *args): - AbstractRangeIteratorRepr.__init__(self, *args) - self.ll_rangeiter = ll_rangeiter - -def ll_rangeiter(ITERPTR, rng): - iter = malloc(ITERPTR.TO) - iter.next = rng.start - iter.stop = rng.stop - if ITERPTR.TO is RANGESTITER: - iter.step = rng.step - return iter - diff --git a/rpython/rtyper/rlist.py b/rpython/rtyper/rlist.py --- a/rpython/rtyper/rlist.py +++ b/rpython/rtyper/rlist.py @@ -44,7 +44,7 @@ s_value = listitem.s_value if (listitem.range_step is not None and not listitem.mutated and not isinstance(s_value, annmodel.SomeImpossibleValue)): - from rpython.rtyper.lltypesystem.rrange import RangeRepr + from rpython.rtyper.rrange import RangeRepr return RangeRepr(listitem.range_step) else: # cannot do the rtyper.getrepr() call immediately, for the case diff --git a/rpython/rtyper/rrange.py b/rpython/rtyper/rrange.py --- a/rpython/rtyper/rrange.py +++ b/rpython/rtyper/rrange.py @@ -1,19 +1,79 @@ from rpython.flowspace.model import Constant from rpython.rtyper.error import TyperError -from rpython.rtyper.lltypesystem.lltype import Signed, Void, Ptr +from rpython.rtyper.lltypesystem.lltype import ( + Ptr, GcStruct, Signed, malloc, Void) from rpython.rtyper.rlist import dum_nocheck, dum_checkidx from rpython.rtyper.rmodel import Repr, IteratorRepr from rpython.rtyper.rint import IntegerRepr from rpython.tool.pairtype import pairtype +# ____________________________________________________________ +# +# Concrete implementation of RPython lists that are returned by range() +# and never mutated afterwards: +# +# struct range { +# Signed start, stop; // step is always constant +# } +# +# struct rangest { +# Signed start, stop, step; // rare case, for completeness +# } -class AbstractRangeRepr(Repr): +def ll_length(l): + if l.step > 0: + lo = l.start + hi = l.stop + step = l.step + else: + lo = l.stop + hi = l.start + step = -l.step + if hi <= lo: + return 0 + n = (hi - lo - 1) // step + 1 + return n + +def ll_getitem_fast(l, index): + return l.start + index * l.step + +RANGEST = GcStruct("range", ("start", Signed), ("stop", Signed), ("step", Signed), + adtmeths = { + "ll_length":ll_length, + "ll_getitem_fast":ll_getitem_fast, + }, + hints = {'immutable': True}) +RANGESTITER = GcStruct("range", ("next", Signed), ("stop", Signed), ("step", Signed)) + +class RangeRepr(Repr): + + RANGEST = Ptr(RANGEST) + RANGESTITER = Ptr(RANGESTITER) + + getfield_opname = "getfield" + def __init__(self, step): + self.RANGE = Ptr(GcStruct("range", ("start", Signed), ("stop", Signed), + adtmeths = { + "ll_length":ll_length, + "ll_getitem_fast":ll_getitem_fast, + "step":step, + }, + hints = {'immutable': True})) + self.RANGEITER = Ptr(GcStruct("range", ("next", Signed), ("stop", Signed))) self.step = step if step != 0: self.lowleveltype = self.RANGE else: self.lowleveltype = self.RANGEST + self.ll_newrange = ll_newrange + self.ll_newrangest = ll_newrangest + + def make_iterator_repr(self, variant=None): + if variant is not None: + raise TyperError("unsupported %r iterator over a range list" % + (variant,)) + return RangeIteratorRepr(self) def _getstep(self, v_rng, hop): return hop.genop(self.getfield_opname, @@ -29,7 +89,9 @@ v_step = self._getstep(v_rng, hop) return hop.gendirectcall(ll_rangelen, v_rng, v_step) -class __extend__(pairtype(AbstractRangeRepr, IntegerRepr)): + + +class __extend__(pairtype(RangeRepr, IntegerRepr)): def rtype_getitem((r_rng, r_int), hop): if hop.has_implicit_exception(IndexError): @@ -89,6 +151,21 @@ index += length return l.start + index * step +def ll_newrange(RANGE, start, stop): + l = malloc(RANGE.TO) + l.start = start + l.stop = stop + return l + +def ll_newrangest(start, stop, step): + if step == 0: + raise ValueError + l = malloc(RANGEST) + l.start = start + l.stop = stop + l.step = step + return l + # ____________________________________________________________ # # Irregular operations. @@ -105,7 +182,7 @@ if isinstance(vstep, Constant) and vstep.value == 0: # not really needed, annotator catches it. Just in case... raise TyperError("range cannot have a const step of zero") - if isinstance(hop.r_result, AbstractRangeRepr): + if isinstance(hop.r_result, RangeRepr): if hop.r_result.step != 0: c_rng = hop.inputconst(Void, hop.r_result.RANGE) hop.exception_is_here() @@ -142,13 +219,14 @@ # # Iteration. -class AbstractRangeIteratorRepr(IteratorRepr): +class RangeIteratorRepr(IteratorRepr): def __init__(self, r_rng): self.r_rng = r_rng if r_rng.step != 0: self.lowleveltype = r_rng.RANGEITER else: self.lowleveltype = r_rng.RANGESTITER + self.ll_rangeiter = ll_rangeiter def newiter(self, hop): v_rng, = hop.inputargs(self.r_rng) @@ -169,6 +247,14 @@ hop.exception_is_here() return hop.gendirectcall(llfn, v_iter, *args) +def ll_rangeiter(ITERPTR, rng): + iter = malloc(ITERPTR.TO) + iter.next = rng.start + iter.stop = rng.stop + if ITERPTR.TO is RANGESTITER: + iter.step = rng.step + return iter + def ll_rangenext_up(iter, step): next = iter.next if next >= iter.stop: diff --git a/rpython/rtyper/test/test_rrange.py b/rpython/rtyper/test/test_rrange.py --- a/rpython/rtyper/test/test_rrange.py +++ b/rpython/rtyper/test/test_rrange.py @@ -1,6 +1,7 @@ from rpython.rlib.rarithmetic import intmask -from rpython.rtyper.rrange import ll_rangelen, ll_rangeitem, ll_rangeitem_nonneg, dum_nocheck -from rpython.rtyper.lltypesystem import rrange +from rpython.rtyper.rrange import ( + ll_rangelen, ll_rangeitem, ll_rangeitem_nonneg, dum_nocheck, ll_newrangest, + RangeRepr, ll_newrange) from rpython.rtyper.test.tool import BaseRtypingTest @@ -11,11 +12,11 @@ expected = range(start, stop, step) length = len(expected) if varstep: - l = rrange.ll_newrangest(start, stop, step) + l = ll_newrangest(start, stop, step) step = l.step else: - RANGE = rrange.RangeRepr(step).RANGE - l = rrange.ll_newrange(RANGE, start, stop) + RANGE = RangeRepr(step).RANGE + l = ll_newrange(RANGE, start, stop) assert ll_rangelen(l, step) == length lst = [ll_rangeitem(dum_nocheck, l, i, step) for i in range(length)] assert lst == expected From pypy.commits at gmail.com Sat Jan 30 16:46:26 2016 From: pypy.commits at gmail.com (arigo) Date: Sat, 30 Jan 2016 13:46:26 -0800 (PST) Subject: [pypy-commit] extradoc extradoc: updated slides, check in the html version Message-ID: <56ad2f32.ccaa1c0a.adf90.6aa1@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r5598:8b46adc6da63 Date: 2016-01-30 22:46 +0100 http://bitbucket.org/pypy/extradoc/changeset/8b46adc6da63/ Log: updated slides, check in the html version diff --git a/talk/fosdem2016/slides b/talk/fosdem2016/slides --- a/talk/fosdem2016/slides +++ b/talk/fosdem2016/slides @@ -324,7 +324,7 @@ - it does not expose any part of the CPython C API - - everything is done with a different, minimal API on the ``ffi`` object + - everything is done with a minimal API on the ``ffi`` object which is closer to C - ``ffi.cast()``, ``ffi.new()``, etc. diff --git a/talk/fosdem2016/slides.html b/talk/fosdem2016/slides.html new file mode 100644 --- /dev/null +++ b/talk/fosdem2016/slides.html @@ -0,0 +1,728 @@ + + + + + + + +CFFI and PyPy + + + + + + + + + + + + + + +

+ + +

CFFI and PyPy

+ + +

CFFI

successful project according to PyPI
3.4 million downloads last month
total 19.2 millions, 27th place on pypi-ranking.info
- Django is 28th
+
some high-visibility projects have switched to it (Cryptography)

PyPy

harder to say, but probably not so successful
more later

CFFI

call C from Python
CFFI = C Foreign Function Interface
shares ideas from Cython, ctypes, and LuaJIT's FFI

CFFI demo

+$ man getpwuid
+
+SYNOPSIS
+       #include <sys/types.h>
+       #include <pwd.h>
+
+       struct passwd *getpwnam(const char *name);
+

CFFI demo

+.
+.
+.
+The passwd structure is defined in <pwd.h> as follows:
+
+    struct passwd {
+        char   *pw_name;       /* username */
+        char   *pw_passwd;     /* user password */
+        uid_t   pw_uid;        /* user ID */
+.
+.
+.
+

CFFI demo

+from cffi import FFI
+ffi = cffi.FFI()
+
+ffi.cdef("""
+    typedef int... uid_t;
+    struct passwd {
+        uid_t pw_uid;
+        ...;
+    };
+    struct passwd *getpwnam(const char *name);
+""")
+

CFFI demo

+ffi.set_source("_pwuid_cffi", """
+    #include <sys/types.h>
+    #include <pwd.h>
+""")
+
+ffi.compile()
+

------- ^^ put that in pwuid_build.py

CFFI demo

+python pwuid_build.py
+

creates _pwuid_cffi.so

CFFI demo

+from _pwuid_cffi import lib
+
+print lib.getpwnam("arigo").pw_uid
+

CFFI demo

+from _pwuid_cffi import ffi, lib
+

lib gives access to all functions from the cdef
+
ffi gives access to a few general helpers, e.g.
+
+
- ffi.cast("float", 42)
- p = ffi.new("struct passwd *")
- p = ffi.new("char[10]"); p[0] = 'X'; s = lib.getpwnam(p)
- p = ffi.new_handle(random_obj); ...; random_obj = ffi.from_handle(p)
+
+

CFFI

supports more or less the whole C
there is more than my short explanation suggests
read the docs: http://cffi.readthedocs.org/

PyPy

a Python interpreter
different from the standard, which is CPython
main goal of PyPy: speed

PyPy

+$ pypy
+
+Python 2.7.10 (5f8302b8bf9f, Nov 18 2015, 10:46:46)
+[PyPy 4.0.1 with GCC 4.8.4] on linux2
+Type "help", "copyright", "credits" or "license" for more information.
+>>>> 2+3
+5
+>>>>
+

PyPy

run pypy my_program.py
starts working like an interpreter
then a Just-in-Time Compiler kicks in
generate and execute machine code from the Python program
good or great speed-ups for the majority of long-running code

PyPy

different techniques than CPython also for "garbage collection"
works very well (arguably better than CPython's reference counting)

PyPy: Garbage Collection

"moving, generational, incremental GC"
objects don't have reference counters
allocated in a "nursery"
when nursery full, find surviving nursery objects and move them out
usually work on nursery objects only (fast), but rarely also perform +a full GC

PyPy: C extensions

PyPy works great for running Python
less great when there are CPython C extension modules involved

PyPy: C extensions

not directly possible: we have moving, non-reference-counted objects, +and the C code expects non-moving, reference-counted objects

PyPy: C extensions

PyPy has still some support for them, called its cpyext module
similar to IronPython's Ironclad
emulate all objects for C extensions with a shadow, non-movable, +reference-counted object

PyPy: C extensions

cpyext is slow
cpyext is actually really, really slow
- but we're working on making it only slow
+

PyPy: C extensions

cpyext will "often" work, but there are a some high-profile C +extension modules that are not supported so far
notably, numpy
(it is future work)

PyPy: ad

but, hey, if you need performance out of Python and don't rely +critically on C extension modules, then give PyPy a try
- typical area where it works well: web services
+

CPython C API: the problem

CPython comes with a C API
very large number of functions
assumes objects don't move
assumes a "reference counting" model

CPython C API

actually, the API is some large subset of the functions inside +CPython itself

CPython C API

easy to use from C
historically, part of the success of Python

CPython C API

further successful tools build on top of that API:
- SWIG
- Cython
- and other binding generators
- now CFFI
+

CFFI

but CFFI is a bit different
- it does not expose any part of the CPython C API
- everything is done with a minimal API on the ffi object +which is closer to C
  - ffi.cast(), ffi.new(), etc.
  +
- that means it can be directly ported
+

CFFI and PyPy

we have a PyPy version of CFFI
the demos I have given above work equally well on CPython or on PyPy
(supporting PyPy was part of the core motivation behind CFFI)

CFFI: performance

in PyPy, JIT compiler speeds up calls, so it's very fast
in CPython, it doesn't occur, but it is still reasonable when +compared with alternatives
main issue is that we write more code in Python with CFFI, +which makes it slower on CPython---but not really on PyPy

CFFI: summary

call C from Python
works natively on CPython and on PyPy
- and easy to port to other Python implementations
+
supports CPython 2.6, 2.7, 3.2 to 3.5, and +is integrated with PyPy

CFFI

independent on the particular details of the Python implementation
- using CFFI, you call C functions and manipulate C-pointer-like +objects directly from Python
- you do in Python all logic involving Python objects
- there are no (official) ways around this API to call the CPython C +API, and none are needed
+

CFFI

two reasons to switch to it :-)
- easy and cool
- better supported on non-CPython implementations
+

CFFI: latest news

support for "embedding" Python inside some other non-Python program
- now you really never need the CPython C API any more
+

CFFI

http://cffi.readthedocs.org/

+ + From pypy.commits at gmail.com Sun Jan 31 10:21:05 2016 From: pypy.commits at gmail.com (devin.jeanpierre) Date: Sun, 31 Jan 2016 07:21:05 -0800 (PST) Subject: [pypy-commit] pypy default: Cast to PyObject* in PyString_GET_SIZE/AS_STRING. Message-ID: <56ae2661.01cdc20a.16151.ffffe72a@mx.google.com> Author: Devin Jeanpierre Branch: Changeset: r82010:f1c5a5f9e0b8 Date: 2016-01-31 02:58 -0800 http://bitbucket.org/pypy/pypy/changeset/f1c5a5f9e0b8/ Log: Cast to PyObject* in PyString_GET_SIZE/AS_STRING. This way we accept PyStringObject*, much like CPython casts to PyStringObject* in order to accept PyObject*. diff --git a/pypy/module/cpyext/include/stringobject.h b/pypy/module/cpyext/include/stringobject.h --- a/pypy/module/cpyext/include/stringobject.h +++ b/pypy/module/cpyext/include/stringobject.h @@ -7,8 +7,8 @@ extern "C" { #endif -#define PyString_GET_SIZE(op) PyString_Size(op) -#define PyString_AS_STRING(op) PyString_AsString(op) +#define PyString_GET_SIZE(op) PyString_Size((PyObject*)(op)) +#define PyString_AS_STRING(op) PyString_AsString((PyObject*)(op)) typedef struct { PyObject_HEAD From pypy.commits at gmail.com Sun Jan 31 10:30:23 2016 From: pypy.commits at gmail.com (arigo) Date: Sun, 31 Jan 2016 07:30:23 -0800 (PST) Subject: [pypy-commit] cffi default: Issue #244: parse_type() calls the Parser logic but self._options used Message-ID: <56ae288f.a453c20a.d2467.00b7@mx.google.com> Author: Armin Rigo Branch: Changeset: r2615:6b198bbbad9f Date: 2016-01-31 16:29 +0100 http://bitbucket.org/cffi/cffi/changeset/6b198bbbad9f/ Log: Issue #244: parse_type() calls the Parser logic but self._options used to be None, crashing in corner cases diff --git a/cffi/cparser.py b/cffi/cparser.py --- a/cffi/cparser.py +++ b/cffi/cparser.py @@ -220,7 +220,7 @@ self._included_declarations = set() self._anonymous_counter = 0 self._structnode2type = weakref.WeakKeyDictionary() - self._options = None + self._options = {} self._int_constants = {} self._recomplete = [] self._uses_new_feature = None @@ -374,7 +374,7 @@ def _declare_function(self, tp, quals, decl): tp = self._get_type_pointer(tp, quals) - if self._options['dllexport']: + if self._options.get('dllexport'): tag = 'dllexport_python ' elif self._inside_extern_python: tag = 'extern_python ' @@ -450,7 +450,7 @@ prevobj, prevquals = self._declarations[name] if prevobj is obj and prevquals == quals: return - if not self._options['override']: + if not self._options.get('override'): raise api.FFIError( "multiple declarations of %s (for interactive usage, " "try cdef(xx, override=True))" % (name,)) @@ -729,7 +729,7 @@ if isinstance(tp, model.StructType) and tp.partial: raise NotImplementedError("%s: using both bitfields and '...;'" % (tp,)) - tp.packed = self._options['packed'] + tp.packed = self._options.get('packed') if tp.completed: # must be re-completed: it is not opaque any more tp.completed = 0 self._recomplete.append(tp) diff --git a/testing/cffi0/backend_tests.py b/testing/cffi0/backend_tests.py --- a/testing/cffi0/backend_tests.py +++ b/testing/cffi0/backend_tests.py @@ -1846,3 +1846,8 @@ thread.start_new_thread(f, ()) time.sleep(1.5) assert seen == ['init!', 'init done'] + 6 * [7] + + def test_sizeof_struct_directly(self): + # only works with the Python FFI instances + ffi = FFI(backend=self.Backend()) + assert ffi.sizeof("struct{int a;}") == ffi.sizeof("int") From pypy.commits at gmail.com Sun Jan 31 11:12:19 2016 From: pypy.commits at gmail.com (mattip) Date: Sun, 31 Jan 2016 08:12:19 -0800 (PST) Subject: [pypy-commit] cffi embedding-pypy-win32: allow embedding tests to run on pypy win32 Message-ID: <56ae3263.a453c20a.d2467.0d29@mx.google.com> Author: mattip Branch: embedding-pypy-win32 Changeset: r2616:6b5048b944cf Date: 2016-01-30 20:50 +0200 http://bitbucket.org/cffi/cffi/changeset/6b5048b944cf/ Log: allow embedding tests to run on pypy win32 diff --git a/cffi/api.py b/cffi/api.py --- a/cffi/api.py +++ b/cffi/api.py @@ -549,24 +549,25 @@ if value not in lst: lst.append(value) # - if '__pypy__' in sys.builtin_module_names: - if hasattr(sys, 'prefix'): - import os - ensure('library_dirs', os.path.join(sys.prefix, 'bin')) - pythonlib = "pypy-c" + if sys.platform == "win32": + # XXX pypy should not reuse the same import library name + template = "python%d%d" + if hasattr(sys, 'gettotalrefcount'): + template += '_d' else: - if sys.platform == "win32": - template = "python%d%d" - if hasattr(sys, 'gettotalrefcount'): - template += '_d' + if '__pypy__' in sys.builtin_module_names: + if hasattr(sys, 'prefix'): + import os + ensure('library_dirs', os.path.join(sys.prefix, 'bin')) + pythonlib = "pypy-c" else: template = "python%d.%d" if sysconfig.get_config_var('DEBUG_EXT'): template += sysconfig.get_config_var('DEBUG_EXT') - pythonlib = (template % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - if hasattr(sys, 'abiflags'): - pythonlib += sys.abiflags + pythonlib = (template % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + if hasattr(sys, 'abiflags'): + pythonlib += sys.abiflags ensure('libraries', pythonlib) if sys.platform == "win32": ensure('extra_link_args', '/MANIFEST') diff --git a/testing/embedding/test_basic.py b/testing/embedding/test_basic.py --- a/testing/embedding/test_basic.py +++ b/testing/embedding/test_basic.py @@ -118,12 +118,18 @@ def execute(self, name): path = self.get_path() env_extra = {'PYTHONPATH': prefix_pythonpath()} - libpath = os.environ.get('LD_LIBRARY_PATH') - if libpath: - libpath = path + ':' + libpath + if sys.platform == 'win32': + _path = os.environ.get('PATH') + # for libpypy-c.dll or Python27.dll + _path += ';' + os.path.split(sys.executable)[0] + env_extra['PATH'] = _path else: - libpath = path - env_extra['LD_LIBRARY_PATH'] = libpath + libpath = os.environ.get('LD_LIBRARY_PATH') + if libpath: + libpath = path + ':' + libpath + else: + libpath = path + env_extra['LD_LIBRARY_PATH'] = libpath print('running %r in %r' % (name, path)) executable_name = name if sys.platform == 'win32': From pypy.commits at gmail.com Sun Jan 31 15:12:50 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 12:12:50 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Fix class name. Message-ID: <56ae6ac2.8e301c0a.511ad.ffffbc1b@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82012:250569a438a0 Date: 2016-01-28 21:27 +0100 http://bitbucket.org/pypy/pypy/changeset/250569a438a0/ Log: Fix class name. diff --git a/pypy/module/itertools/interp_itertools.py b/pypy/module/itertools/interp_itertools.py --- a/pypy/module/itertools/interp_itertools.py +++ b/pypy/module/itertools/interp_itertools.py @@ -275,7 +275,7 @@ return space.wrap(r) W_FilterFalse.typedef = TypeDef( - 'itertools.ifilterfalse', + 'itertools.filterfalse', __new__ = interp2app(W_FilterFalse___new__), __iter__ = interp2app(W_FilterFalse.iter_w), __next__ = interp2app(W_FilterFalse.next_w), From pypy.commits at gmail.com Sun Jan 31 15:12:48 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 12:12:48 -0800 (PST) Subject: [pypy-commit] pypy py3.3: pickle support for itertools.takewhile and dropwhile. Message-ID: <56ae6ac0.42cbc20a.1c2fa.443b@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82011:6e3ddc38f7c0 Date: 2016-01-28 21:21 +0100 http://bitbucket.org/pypy/pypy/changeset/6e3ddc38f7c0/ Log: pickle support for itertools.takewhile and dropwhile. diff --git a/pypy/module/itertools/interp_itertools.py b/pypy/module/itertools/interp_itertools.py --- a/pypy/module/itertools/interp_itertools.py +++ b/pypy/module/itertools/interp_itertools.py @@ -148,7 +148,7 @@ def __init__(self, space, w_predicate, w_iterable): self.space = space self.w_predicate = w_predicate - self.iterable = space.iter(w_iterable) + self.w_iterable = space.iter(w_iterable) self.stopped = False def iter_w(self): @@ -158,7 +158,7 @@ if self.stopped: raise OperationError(self.space.w_StopIteration, self.space.w_None) - w_obj = self.space.next(self.iterable) # may raise a w_StopIteration + w_obj = self.space.next(self.w_iterable) # may raise a w_StopIteration w_bool = self.space.call_function(self.w_predicate, w_obj) if not self.space.is_true(w_bool): self.stopped = True @@ -166,6 +166,16 @@ return w_obj + def descr_reduce(self, space): + return space.newtuple([ + space.type(self), + space.newtuple([self.w_predicate, self.w_iterable]), + space.wrap(self.stopped) + ]) + + def descr_setstate(self, space, w_state): + self.stopped = space.bool_w(w_state) + def W_TakeWhile___new__(space, w_subtype, w_predicate, w_iterable): r = space.allocate_instance(W_TakeWhile, w_subtype) r.__init__(space, w_predicate, w_iterable) @@ -177,6 +187,8 @@ __new__ = interp2app(W_TakeWhile___new__), __iter__ = interp2app(W_TakeWhile.iter_w), __next__ = interp2app(W_TakeWhile.next_w), + __reduce__ = interp2app(W_TakeWhile.descr_reduce), + __setstate__ = interp2app(W_TakeWhile.descr_setstate), __doc__ = """Make an iterator that returns elements from the iterable as long as the predicate is true. @@ -195,7 +207,7 @@ def __init__(self, space, w_predicate, w_iterable): self.space = space self.w_predicate = w_predicate - self.iterable = space.iter(w_iterable) + self.w_iterable = space.iter(w_iterable) self.started = False def iter_w(self): @@ -203,10 +215,10 @@ def next_w(self): if self.started: - w_obj = self.space.next(self.iterable) # may raise w_StopIteration + w_obj = self.space.next(self.w_iterable) # may raise w_StopIter else: while True: - w_obj = self.space.next(self.iterable) # may raise w_StopIter + w_obj = self.space.next(self.w_iterable) # may raise w_StopIter w_bool = self.space.call_function(self.w_predicate, w_obj) if not self.space.is_true(w_bool): self.started = True @@ -214,6 +226,16 @@ return w_obj + def descr_reduce(self, space): + return space.newtuple([ + space.type(self), + space.newtuple([self.w_predicate, self.w_iterable]), + space.wrap(self.started) + ]) + + def descr_setstate(self, space, w_state): + self.started = space.bool_w(w_state) + def W_DropWhile___new__(space, w_subtype, w_predicate, w_iterable): r = space.allocate_instance(W_DropWhile, w_subtype) r.__init__(space, w_predicate, w_iterable) @@ -225,6 +247,8 @@ __new__ = interp2app(W_DropWhile___new__), __iter__ = interp2app(W_DropWhile.iter_w), __next__ = interp2app(W_DropWhile.next_w), + __reduce__ = interp2app(W_DropWhile.descr_reduce), + __setstate__ = interp2app(W_DropWhile.descr_setstate), __doc__ = """Make an iterator that drops elements from the iterable as long as the predicate is true; afterwards, returns every element. Note, the iterator does not produce any output until the diff --git a/pypy/module/itertools/test/test_itertools.py b/pypy/module/itertools/test/test_itertools.py --- a/pypy/module/itertools/test/test_itertools.py +++ b/pypy/module/itertools/test/test_itertools.py @@ -836,12 +836,6 @@ "usemodules": ['itertools', 'struct', 'binascii'], } - def setup_class(cls): - if cls.space.is_true(cls.space.appexec([], """(): - import sys; return sys.version_info < (2, 7) - """)): - py.test.skip("Requires Python 2.7") - def test_compress(self): import itertools it = itertools.compress(['a', 'b', 'c'], [0, 1, 0]) @@ -1001,6 +995,16 @@ assert list(itertools.islice( pickle.loads(pickle.dumps(c)), 10)) == list('bcabcabcab') + def test_takewhile_pickle(self): + data = [1, 2, 3, 0, 4, 5, 6] + import itertools, pickle + t = itertools.takewhile(bool, data) + next(t) + assert list(pickle.loads(pickle.dumps(t))) == [2, 3] + t = itertools.dropwhile(bool, data) + next(t) + assert list(pickle.loads(pickle.dumps(t))) == [4, 5, 6] + class AppTestItertools32: spaceconfig = dict(usemodules=['itertools']) From pypy.commits at gmail.com Sun Jan 31 15:12:52 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 12:12:52 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Pickle support for itertools.islice Message-ID: <56ae6ac4.624fc20a.d8ba2.4e9e@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82013:a5416d54ea4e Date: 2016-01-31 21:11 +0100 http://bitbucket.org/pypy/pypy/changeset/a5416d54ea4e/ Log: Pickle support for itertools.islice diff --git a/pypy/module/itertools/interp_itertools.py b/pypy/module/itertools/interp_itertools.py --- a/pypy/module/itertools/interp_itertools.py +++ b/pypy/module/itertools/interp_itertools.py @@ -404,6 +404,15 @@ if num <= 0: break + def descr_reduce(self, space): + return space.newtuple([ + space.type(self), + space.newtuple([self.iterable, + space.wrap(self.start), + space.wrap(self.stop), + space.wrap(self.ignore + 1)]), + ]) + def W_ISlice___new__(space, w_subtype, w_iterable, w_startstop, args_w): r = space.allocate_instance(W_ISlice, w_subtype) r.__init__(space, w_iterable, w_startstop, args_w) @@ -414,6 +423,7 @@ __new__ = interp2app(W_ISlice___new__), __iter__ = interp2app(W_ISlice.iter_w), __next__ = interp2app(W_ISlice.next_w), + __reduce__ = interp2app(W_ISlice.descr_reduce), __doc__ = """Make an iterator that returns selected elements from the iterable. If start is non-zero, then elements from the iterable are skipped until start is reached. Afterward, elements are diff --git a/pypy/module/itertools/test/test_itertools.py b/pypy/module/itertools/test/test_itertools.py --- a/pypy/module/itertools/test/test_itertools.py +++ b/pypy/module/itertools/test/test_itertools.py @@ -988,6 +988,11 @@ assert list(op(testIntermediate)) == [ (0,1,3), (0,2,3), (1,2,3)] + def test_islice_pickle(self): + import itertools, pickle + it = itertools.islice(range(100), 10, 20, 3) + assert list(pickle.loads(pickle.dumps(it))) == list(range(100)[10:20:3]) + def test_cycle_pickle(self): import itertools, pickle c = itertools.cycle('abc') From pypy.commits at gmail.com Sun Jan 31 16:19:47 2016 From: pypy.commits at gmail.com (mattip) Date: Sun, 31 Jan 2016 13:19:47 -0800 (PST) Subject: [pypy-commit] pypy default: skip for msvc Message-ID: <56ae7a73.01adc20a.c27d9.5a99@mx.google.com> Author: mattip Branch: Changeset: r82014:aa367ba93c8a Date: 2016-01-31 23:03 +0200 http://bitbucket.org/pypy/pypy/changeset/aa367ba93c8a/ Log: skip for msvc diff --git a/rpython/rtyper/tool/test/test_rffi_platform.py b/rpython/rtyper/tool/test/test_rffi_platform.py --- a/rpython/rtyper/tool/test/test_rffi_platform.py +++ b/rpython/rtyper/tool/test/test_rffi_platform.py @@ -277,10 +277,14 @@ assert not rffi_platform.has("x", "#include ") def test_has_0002(): + if platform.name == 'msvc': + py.test.skip('no m.lib in msvc') assert rffi_platform.has("pow", "#include ", libraries=["m"]) def test_has_0003(): """multiple libraries""" + if platform.name == 'msvc': + py.test.skip('no m.lib in msvc') assert rffi_platform.has("pow", "#include ", libraries=["m", "c"]) def test_has_0004(): From pypy.commits at gmail.com Sun Jan 31 16:37:30 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 13:37:30 -0800 (PST) Subject: [pypy-commit] pypy py3.3: For some reason this chunk was not copied from CPython 3.3 Message-ID: <56ae7e9a.6217c20a.5f363.70a6@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82015:b28c0f7ae8f3 Date: 2016-01-31 21:27 +0100 http://bitbucket.org/pypy/pypy/changeset/b28c0f7ae8f3/ Log: For some reason this chunk was not copied from CPython 3.3 diff --git a/pypy/module/_multibytecodec/src/cjkcodecs/_codecs_tw.c b/pypy/module/_multibytecodec/src/cjkcodecs/_codecs_tw.c --- a/pypy/module/_multibytecodec/src/cjkcodecs/_codecs_tw.c +++ b/pypy/module/_multibytecodec/src/cjkcodecs/_codecs_tw.c @@ -55,7 +55,7 @@ TRYMAP_DEC(big5, **outbuf, c, IN2) { NEXT(2, 1) } - else return 2; + else return 1; } return 0; @@ -109,7 +109,7 @@ TRYMAP_DEC(cp950ext, **outbuf, c, IN2); else TRYMAP_DEC(big5, **outbuf, c, IN2); - else return 2; + else return 1; NEXT(2, 1) } From pypy.commits at gmail.com Sun Jan 31 16:37:32 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 13:37:32 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Add _ssl.RAND_bytes Message-ID: <56ae7e9c.c9ebc20a.e381f.638f@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82016:9a47d88b9973 Date: 2016-01-31 22:29 +0100 http://bitbucket.org/pypy/pypy/changeset/9a47d88b9973/ Log: Add _ssl.RAND_bytes diff --git a/pypy/module/_ssl/__init__.py b/pypy/module/_ssl/__init__.py --- a/pypy/module/_ssl/__init__.py +++ b/pypy/module/_ssl/__init__.py @@ -43,8 +43,8 @@ if HAVE_OPENSSL_RAND: Module.interpleveldefs['RAND_add'] = "interp_ssl.RAND_add" - Module.interpleveldefs['RAND_bytes'] = "space.w_None" # so far - Module.interpleveldefs['RAND_pseudo_bytes'] = "space.w_None" # so far + Module.interpleveldefs['RAND_bytes'] = "interp_ssl.RAND_bytes" + Module.interpleveldefs['RAND_pseudo_bytes'] = "interp_ssl.RAND_pseudo_bytes" Module.interpleveldefs['RAND_status'] = "interp_ssl.RAND_status" Module.interpleveldefs['RAND_egd'] = "interp_ssl.RAND_egd" diff --git a/pypy/module/_ssl/interp_ssl.py b/pypy/module/_ssl/interp_ssl.py --- a/pypy/module/_ssl/interp_ssl.py +++ b/pypy/module/_ssl/interp_ssl.py @@ -237,6 +237,43 @@ with rffi.scoped_str2charp(string) as buf: libssl_RAND_add(buf, len(string), entropy) + def _RAND_bytes(space, n, pseudo): + if n < 0: + raise OperationError(space.w_ValueError, space.wrap( + "num must be positive")) + + with rffi.scoped_alloc_buffer(n) as buf: + if pseudo: + ok = libssl_RAND_pseudo_bytes( + rffi.cast(rffi.UCHARP, buf.raw), n) + if ok == 0 or ok == 1: + return space.newtuple([ + space.wrapbytes(buf.str(n)), + space.wrap(ok == 1), + ]) + else: + ok = libssl_RAND_bytes( + rffi.cast(rffi.UCHARP, buf.raw), n) + if ok == 1: + return space.wrapbytes(buf.str(n)) + + raise ssl_error(space, "", errcode=libssl_ERR_get_error()) + + @unwrap_spec(n=int) + def RAND_bytes(space, n): + """RAND_bytes(n) -> bytes + + Generate n cryptographically strong pseudo-random bytes.""" + return _RAND_bytes(space, n, pseudo=False) + + @unwrap_spec(n=int) + def RAND_pseudo_bytes(space, n): + """RAND_pseudo_bytes(n) -> (bytes, is_cryptographic) + + Generate n pseudo-random bytes. is_cryptographic is True if the bytes + generated are cryptographically strong.""" + return _RAND_bytes(space, n, pseudo=True) + def RAND_status(space): """RAND_status() -> 0 or 1 diff --git a/pypy/module/_ssl/test/test_ssl.py b/pypy/module/_ssl/test/test_ssl.py --- a/pypy/module/_ssl/test/test_ssl.py +++ b/pypy/module/_ssl/test/test_ssl.py @@ -48,6 +48,16 @@ raises(TypeError, _ssl.RAND_add, "xyz", "zyx") _ssl.RAND_add("xyz", 1.2345) + def test_RAND_bytes(self): + import _ssl + b = _ssl.RAND_bytes(3) + assert type(b) is bytes + assert len(b) == 3 + b, ok = _ssl.RAND_pseudo_bytes(3) + assert type(b) is bytes + assert len(b) == 3 + assert ok is True or ok is False + def test_RAND_status(self): import _ssl if not hasattr(_ssl, "RAND_status"): diff --git a/rpython/rlib/ropenssl.py b/rpython/rlib/ropenssl.py --- a/rpython/rlib/ropenssl.py +++ b/rpython/rlib/ropenssl.py @@ -311,6 +311,8 @@ if HAVE_OPENSSL_RAND: ssl_external('RAND_add', [rffi.CCHARP, rffi.INT, rffi.DOUBLE], lltype.Void) + ssl_external('RAND_bytes', [rffi.UCHARP, rffi.INT], rffi.INT) + ssl_external('RAND_pseudo_bytes', [rffi.UCHARP, rffi.INT], rffi.INT) ssl_external('RAND_status', [], rffi.INT) if HAVE_OPENSSL_RAND_EGD: ssl_external('RAND_egd', [rffi.CCHARP], rffi.INT) From pypy.commits at gmail.com Sun Jan 31 17:47:19 2016 From: pypy.commits at gmail.com (cfbolz) Date: Sun, 31 Jan 2016 14:47:19 -0800 (PST) Subject: [pypy-commit] pypy default: duh, check the index first(!) Message-ID: <56ae8ef7.a151c20a.c3de6.ffff8690@mx.google.com> Author: Carl Friedrich Bolz Branch: Changeset: r82018:3fe07a7b1c13 Date: 2016-01-31 19:10 +0100 http://bitbucket.org/pypy/pypy/changeset/3fe07a7b1c13/ Log: duh, check the index first(!) diff --git a/pypy/objspace/std/mapdict.py b/pypy/objspace/std/mapdict.py --- a/pypy/objspace/std/mapdict.py +++ b/pypy/objspace/std/mapdict.py @@ -116,7 +116,7 @@ def _find_map_attr(self, name, index): while isinstance(self, PlainAttribute): - if name == self.name and index == self.index: + if index == self.index and name == self.name: return self self = self.back return None @@ -296,7 +296,7 @@ new_obj._get_mapdict_map().add_attr(new_obj, self.name, self.index, w_value) def delete(self, obj, name, index): - if name == self.name and index == self.index: + if index == self.index and name == self.name: # ok, attribute is deleted if not self.ever_mutated: self.ever_mutated = True From pypy.commits at gmail.com Sun Jan 31 17:47:21 2016 From: pypy.commits at gmail.com (cfbolz) Date: Sun, 31 Jan 2016 14:47:21 -0800 (PST) Subject: [pypy-commit] pypy default: this comment is outdated now Message-ID: <56ae8ef9.2968c20a.c229f.ffff81fe@mx.google.com> Author: Carl Friedrich Bolz Branch: Changeset: r82019:a71c1f3776a9 Date: 2016-01-31 19:11 +0100 http://bitbucket.org/pypy/pypy/changeset/a71c1f3776a9/ Log: this comment is outdated now diff --git a/pypy/objspace/std/mapdict.py b/pypy/objspace/std/mapdict.py --- a/pypy/objspace/std/mapdict.py +++ b/pypy/objspace/std/mapdict.py @@ -156,7 +156,6 @@ jit.isconstant(name) and jit.isconstant(index)) def add_attr(self, obj, name, index, w_value): - # grumble, jit needs this attr = self._get_new_attr(name, index) oldattr = obj._get_mapdict_map() if not jit.we_are_jitted(): From pypy.commits at gmail.com Sun Jan 31 17:47:17 2016 From: pypy.commits at gmail.com (cfbolz) Date: Sun, 31 Jan 2016 14:47:17 -0800 (PST) Subject: [pypy-commit] pypy default: fix typo Message-ID: <56ae8ef5.0357c20a.58781.7513@mx.google.com> Author: Carl Friedrich Bolz Branch: Changeset: r82017:8a7217a547e2 Date: 2016-01-31 18:37 +0100 http://bitbucket.org/pypy/pypy/changeset/8a7217a547e2/ Log: fix typo diff --git a/pypy/objspace/std/setobject.py b/pypy/objspace/std/setobject.py --- a/pypy/objspace/std/setobject.py +++ b/pypy/objspace/std/setobject.py @@ -942,7 +942,7 @@ return False if w_set.length() == 0: return True - # it's possible to have 0-lenght strategy that's not empty + # it's possible to have 0-length strategy that's not empty if w_set.strategy is w_other.strategy: return self._issubset_unwrapped(w_set, w_other) if not self.may_contain_equal_elements(w_other.strategy): From pypy.commits at gmail.com Sun Jan 31 18:42:20 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 15:42:20 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Fix unicode.capitalize() test to pass with CPython3.3, Message-ID: <56ae9bdc.a3abc20a.ddb4.ffff9400@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82021:44aa48e4d16a Date: 2016-02-01 00:31 +0100 http://bitbucket.org/pypy/pypy/changeset/44aa48e4d16a/ Log: Fix unicode.capitalize() test to pass with CPython3.3, and implement it for PyPy. Probably not the fastest implementation... diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -217,7 +217,7 @@ # check that titlecased chars are lowered correctly # \u1ffc is the titlecased char assert ('\u1ff3\u1ff3\u1ffc\u1ffc'.capitalize() == - '\u1ffc\u1ff3\u1ff3\u1ff3') + '\u03a9\u0399\u1ff3\u1ff3\u1ff3') # check with cased non-letter chars assert ('\u24c5\u24ce\u24c9\u24bd\u24c4\u24c3'.capitalize() == '\u24c5\u24e8\u24e3\u24d7\u24de\u24dd') diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -155,13 +155,16 @@ return unicodedb.islinebreak(ord(ch)) def _upper(self, ch): - return unichr(unicodedb.toupper(ord(ch))) + return u''.join([unichr(x) for x in + unicodedb.toupper_full(ord(ch))]) def _lower(self, ch): - return unichr(unicodedb.tolower(ord(ch))) + return u''.join([unichr(x) for x in + unicodedb.tolower_full(ord(ch))]) def _title(self, ch): - return unichr(unicodedb.totitle(ord(ch))) + return u''.join([unichr(x) for x in + unicodedb.totitle_full(ord(ch))]) def _newlist_unwrapped(self, space, lst): return space.newlist_unicode(lst) From pypy.commits at gmail.com Sun Jan 31 18:42:22 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 15:42:22 -0800 (PST) Subject: [pypy-commit] pypy py3.3: Another test that needed a change for CPython3.3 Message-ID: <56ae9bde.4c5c1c0a.24191.fffffab2@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82022:f41f8a7c1fb8 Date: 2016-02-01 00:40 +0100 http://bitbucket.org/pypy/pypy/changeset/f41f8a7c1fb8/ Log: Another test that needed a change for CPython3.3 diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -729,7 +729,7 @@ assert 'ababa'.count('aba') == 1 def test_swapcase(self): - assert '\xe4\xc4\xdf'.swapcase() == '\xc4\xe4\xdf' + assert '\xe4\xc4\xdf'.swapcase() == '\xc4\xe4SS' def test_call_special_methods(self): # xxx not completely clear if these are implementation details or not From pypy.commits at gmail.com Sun Jan 31 18:42:18 2016 From: pypy.commits at gmail.com (amauryfa) Date: Sun, 31 Jan 2016 15:42:18 -0800 (PST) Subject: [pypy-commit] pypy py3.3: unicodedata: add tables for special_casing. Message-ID: <56ae9bda.8e301c0a.511ad.fffff8b0@mx.google.com> Author: Amaury Forgeot d'Arc Branch: py3.3 Changeset: r82020:c9f92e754efd Date: 2016-02-01 00:26 +0100 http://bitbucket.org/pypy/pypy/changeset/c9f92e754efd/ Log: unicodedata: add tables for special_casing. diff --git a/rpython/rlib/unicodedata/SpecialCasing-5.2.0.txt b/rpython/rlib/unicodedata/SpecialCasing-5.2.0.txt new file mode 100644 --- /dev/null +++ b/rpython/rlib/unicodedata/SpecialCasing-5.2.0.txt @@ -0,0 +1,273 @@ +# SpecialCasing-5.2.0.txt +# Date: 2009-09-22, 23:25:59 GMT [MD] +# +# Unicode Character Database +# Copyright (c) 1991-2009 Unicode, Inc. +# For terms of use, see http://www.unicode.org/terms_of_use.html +# For documentation, see http://www.unicode.org/reports/tr44/ +# +# Special Casing Properties +# +# This file is a supplement to the UnicodeData file. +# It contains additional information about the casing of Unicode characters. +# (For compatibility, the UnicodeData.txt file only contains case mappings for +# characters where they are 1-1, and independent of context and language. +# For more information, see the discussion of Case Mappings in the Unicode Standard. +# +# All code points not listed in this file that do not have a simple case mappings +# in UnicodeData.txt map to themselves. +# ================================================================================ +# Format +# ================================================================================ +# The entries in this file are in the following machine-readable format: +# +#

;  ;  ; <upper> ; (<condition_list> ;)? # <comment>
+#
+# <code>, <lower>, <title>, and <upper> provide character values in hex. If there is more
+# than one character, they are separated by spaces. Other than as used to separate 
+# elements, spaces are to be ignored.
+#
+# The <condition_list> is optional. Where present, it consists of one or more language IDs
+# or contexts, separated by spaces. In these conditions:
+# - A condition list overrides the normal behavior if all of the listed conditions are true.
+# - The context is always the context of the characters in the original string,
+#   NOT in the resulting string.
+# - Case distinctions in the condition list are not significant.
+# - Conditions preceded by "Not_" represent the negation of the condition.
+# The condition list is not represented in the UCD as a formal property.
+#
+# A language ID is defined by BCP 47, with '-' and '_' treated equivalently.
+#
+# A context for a character C is defined by Section 3.13 Default Case 
+# Operations, of The Unicode Standard, Version 5.0.
+# (This is identical to the context defined by Unicode 4.1.0,
+#  as specified in http://www.unicode.org/versions/Unicode4.1.0/)
+#
+# Parsers of this file must be prepared to deal with future additions to this format:
+#  * Additional contexts
+#  * Additional fields
+# ================================================================================
+# @missing 0000..10FFFF; <slc>; <stc>; <suc>
+# ================================================================================
+# Unconditional mappings
+# ================================================================================
+
+# The German es-zed is special--the normal mapping is to SS.
+# Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase(<es-zed>))
+
+00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S
+
+# Preserve canonical equivalence for I with dot. Turkic is handled below.
+
+0130; 0069 0307; 0130; 0130; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# Ligatures
+
+FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF
+FB01; FB01; 0046 0069; 0046 0049; # LATIN SMALL LIGATURE FI
+FB02; FB02; 0046 006C; 0046 004C; # LATIN SMALL LIGATURE FL
+FB03; FB03; 0046 0066 0069; 0046 0046 0049; # LATIN SMALL LIGATURE FFI
+FB04; FB04; 0046 0066 006C; 0046 0046 004C; # LATIN SMALL LIGATURE FFL
+FB05; FB05; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE LONG S T
+FB06; FB06; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE ST
+
+0587; 0587; 0535 0582; 0535 0552; # ARMENIAN SMALL LIGATURE ECH YIWN
+FB13; FB13; 0544 0576; 0544 0546; # ARMENIAN SMALL LIGATURE MEN NOW
+FB14; FB14; 0544 0565; 0544 0535; # ARMENIAN SMALL LIGATURE MEN ECH
+FB15; FB15; 0544 056B; 0544 053B; # ARMENIAN SMALL LIGATURE MEN INI
+FB16; FB16; 054E 0576; 054E 0546; # ARMENIAN SMALL LIGATURE VEW NOW
+FB17; FB17; 0544 056D; 0544 053D; # ARMENIAN SMALL LIGATURE MEN XEH
+
+# No corresponding uppercase precomposed character
+
+0149; 0149; 02BC 004E; 02BC 004E; # LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
+0390; 0390; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
+03B0; 03B0; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
+01F0; 01F0; 004A 030C; 004A 030C; # LATIN SMALL LETTER J WITH CARON
+1E96; 1E96; 0048 0331; 0048 0331; # LATIN SMALL LETTER H WITH LINE BELOW
+1E97; 1E97; 0054 0308; 0054 0308; # LATIN SMALL LETTER T WITH DIAERESIS
+1E98; 1E98; 0057 030A; 0057 030A; # LATIN SMALL LETTER W WITH RING ABOVE
+1E99; 1E99; 0059 030A; 0059 030A; # LATIN SMALL LETTER Y WITH RING ABOVE
+1E9A; 1E9A; 0041 02BE; 0041 02BE; # LATIN SMALL LETTER A WITH RIGHT HALF RING
+1F50; 1F50; 03A5 0313; 03A5 0313; # GREEK SMALL LETTER UPSILON WITH PSILI
+1F52; 1F52; 03A5 0313 0300; 03A5 0313 0300; # GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
+1F54; 1F54; 03A5 0313 0301; 03A5 0313 0301; # GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
+1F56; 1F56; 03A5 0313 0342; 03A5 0313 0342; # GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
+1FB6; 1FB6; 0391 0342; 0391 0342; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI
+1FC6; 1FC6; 0397 0342; 0397 0342; # GREEK SMALL LETTER ETA WITH PERISPOMENI
+1FD2; 1FD2; 0399 0308 0300; 0399 0308 0300; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
+1FD3; 1FD3; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
+1FD6; 1FD6; 0399 0342; 0399 0342; # GREEK SMALL LETTER IOTA WITH PERISPOMENI
+1FD7; 1FD7; 0399 0308 0342; 0399 0308 0342; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
+1FE2; 1FE2; 03A5 0308 0300; 03A5 0308 0300; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
+1FE3; 1FE3; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
+1FE4; 1FE4; 03A1 0313; 03A1 0313; # GREEK SMALL LETTER RHO WITH PSILI
+1FE6; 1FE6; 03A5 0342; 03A5 0342; # GREEK SMALL LETTER UPSILON WITH PERISPOMENI
+1FE7; 1FE7; 03A5 0308 0342; 03A5 0308 0342; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
+1FF6; 1FF6; 03A9 0342; 03A9 0342; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI
+
+# IMPORTANT-when iota-subscript (0345) is uppercased or titlecased,
+#  the result will be incorrect unless the iota-subscript is moved to the end
+#  of any sequence of combining marks. Otherwise, the accents will go on the capital iota.
+#  This process can be achieved by first transforming the text to NFC before casing.
+#  E.g. <alpha><iota_subscript><acute> is uppercased to <ALPHA><acute><IOTA>
+
+# The following cases are already in the UnicodeData file, so are only commented here.
+
+# 0345; 0345; 0345; 0399; # COMBINING GREEK YPOGEGRAMMENI
+
+# All letters with YPOGEGRAMMENI (iota-subscript) or PROSGEGRAMMENI (iota adscript)
+# have special uppercases.
+# Note: characters with PROSGEGRAMMENI are actually titlecase, not uppercase!
+
+1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI
+1F81; 1F81; 1F89; 1F09 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND YPOGEGRAMMENI
+1F82; 1F82; 1F8A; 1F0A 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F83; 1F83; 1F8B; 1F0B 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F84; 1F84; 1F8C; 1F0C 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F85; 1F85; 1F8D; 1F0D 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F86; 1F86; 1F8E; 1F0E 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F87; 1F87; 1F8F; 1F0F 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F88; 1F80; 1F88; 1F08 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PROSGEGRAMMENI
+1F89; 1F81; 1F89; 1F09 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PROSGEGRAMMENI
+1F8A; 1F82; 1F8A; 1F0A 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F8B; 1F83; 1F8B; 1F0B 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F8C; 1F84; 1F8C; 1F0C 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F8D; 1F85; 1F8D; 1F0D 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F8E; 1F86; 1F8E; 1F0E 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F8F; 1F87; 1F8F; 1F0F 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1F90; 1F90; 1F98; 1F28 0399; # GREEK SMALL LETTER ETA WITH PSILI AND YPOGEGRAMMENI
+1F91; 1F91; 1F99; 1F29 0399; # GREEK SMALL LETTER ETA WITH DASIA AND YPOGEGRAMMENI
+1F92; 1F92; 1F9A; 1F2A 0399; # GREEK SMALL LETTER ETA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F93; 1F93; 1F9B; 1F2B 0399; # GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F94; 1F94; 1F9C; 1F2C 0399; # GREEK SMALL LETTER ETA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F95; 1F95; 1F9D; 1F2D 0399; # GREEK SMALL LETTER ETA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F96; 1F96; 1F9E; 1F2E 0399; # GREEK SMALL LETTER ETA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F97; 1F97; 1F9F; 1F2F 0399; # GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F98; 1F90; 1F98; 1F28 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PROSGEGRAMMENI
+1F99; 1F91; 1F99; 1F29 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PROSGEGRAMMENI
+1F9A; 1F92; 1F9A; 1F2A 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F9B; 1F93; 1F9B; 1F2B 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F9C; 1F94; 1F9C; 1F2C 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F9D; 1F95; 1F9D; 1F2D 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F9E; 1F96; 1F9E; 1F2E 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F9F; 1F97; 1F9F; 1F2F 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FA0; 1FA0; 1FA8; 1F68 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND YPOGEGRAMMENI
+1FA1; 1FA1; 1FA9; 1F69 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND YPOGEGRAMMENI
+1FA2; 1FA2; 1FAA; 1F6A 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1FA3; 1FA3; 1FAB; 1F6B 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1FA4; 1FA4; 1FAC; 1F6C 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1FA5; 1FA5; 1FAD; 1F6D 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1FA6; 1FA6; 1FAE; 1F6E 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1FA7; 1FA7; 1FAF; 1F6F 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1FA8; 1FA0; 1FA8; 1F68 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PROSGEGRAMMENI
+1FA9; 1FA1; 1FA9; 1F69 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PROSGEGRAMMENI
+1FAA; 1FA2; 1FAA; 1F6A 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1FAB; 1FA3; 1FAB; 1F6B 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1FAC; 1FA4; 1FAC; 1F6C 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1FAD; 1FA5; 1FAD; 1F6D 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1FAE; 1FA6; 1FAE; 1F6E 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1FAF; 1FA7; 1FAF; 1F6F 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FB3; 1FB3; 1FBC; 0391 0399; # GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI
+1FBC; 1FB3; 1FBC; 0391 0399; # GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI
+1FC3; 1FC3; 1FCC; 0397 0399; # GREEK SMALL LETTER ETA WITH YPOGEGRAMMENI
+1FCC; 1FC3; 1FCC; 0397 0399; # GREEK CAPITAL LETTER ETA WITH PROSGEGRAMMENI
+1FF3; 1FF3; 1FFC; 03A9 0399; # GREEK SMALL LETTER OMEGA WITH YPOGEGRAMMENI
+1FFC; 1FF3; 1FFC; 03A9 0399; # GREEK CAPITAL LETTER OMEGA WITH PROSGEGRAMMENI
+
+# Some characters with YPOGEGRAMMENI also have no corresponding titlecases
+
+1FB2; 1FB2; 1FBA 0345; 1FBA 0399; # GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI
+1FB4; 1FB4; 0386 0345; 0386 0399; # GREEK SMALL LETTER ALPHA WITH OXIA AND YPOGEGRAMMENI
+1FC2; 1FC2; 1FCA 0345; 1FCA 0399; # GREEK SMALL LETTER ETA WITH VARIA AND YPOGEGRAMMENI
+1FC4; 1FC4; 0389 0345; 0389 0399; # GREEK SMALL LETTER ETA WITH OXIA AND YPOGEGRAMMENI
+1FF2; 1FF2; 1FFA 0345; 1FFA 0399; # GREEK SMALL LETTER OMEGA WITH VARIA AND YPOGEGRAMMENI
+1FF4; 1FF4; 038F 0345; 038F 0399; # GREEK SMALL LETTER OMEGA WITH OXIA AND YPOGEGRAMMENI
+
+1FB7; 1FB7; 0391 0342 0345; 0391 0342 0399; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FC7; 1FC7; 0397 0342 0345; 0397 0342 0399; # GREEK SMALL LETTER ETA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FF7; 1FF7; 03A9 0342 0345; 03A9 0342 0399; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI AND YPOGEGRAMMENI
+
+# ================================================================================
+# Conditional Mappings
+# The remainder of this file provides conditional casing data used to produce 
+# full case mappings.
+# ================================================================================
+# Language-Insensitive Mappings
+# These are characters whose full case mappings do not depend on language, but do
+# depend on context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Special case for final form of sigma
+
+03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
+
+# Note: the following cases for non-final are already in the UnicodeData file.
+
+# 03A3; 03C3; 03A3; 03A3; # GREEK CAPITAL LETTER SIGMA
+# 03C3; 03C3; 03A3; 03A3; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C2; 03A3; 03A3; # GREEK SMALL LETTER FINAL SIGMA
+
+# Note: the following cases are not included, since they would case-fold in lowercasing
+
+# 03C3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C3; 03A3; 03A3; Not_Final_Sigma; # GREEK SMALL LETTER FINAL SIGMA
+
+# ================================================================================
+# Language-Sensitive Mappings
+# These are characters whose full case mappings depend on language and perhaps also
+# context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Lithuanian
+
+# Lithuanian retains the dot in a lowercase i when followed by accents.
+
+# Remove DOT ABOVE after "i" with upper or titlecase
+
+0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
+
+# Introduce an explicit dot above when lowercasing capital I's and J's
+# whenever there are more accents above.
+# (of the accents used in Lithuanian: grave, acute, tilde above, and ogonek)
+
+0049; 0069 0307; 0049; 0049; lt More_Above; # LATIN CAPITAL LETTER I
+004A; 006A 0307; 004A; 004A; lt More_Above; # LATIN CAPITAL LETTER J
+012E; 012F 0307; 012E; 012E; lt More_Above; # LATIN CAPITAL LETTER I WITH OGONEK
+00CC; 0069 0307 0300; 00CC; 00CC; lt; # LATIN CAPITAL LETTER I WITH GRAVE
+00CD; 0069 0307 0301; 00CD; 00CD; lt; # LATIN CAPITAL LETTER I WITH ACUTE
+0128; 0069 0307 0303; 0128; 0128; lt; # LATIN CAPITAL LETTER I WITH TILDE
+
+# ================================================================================
+
+# Turkish and Azeri
+
+# I and i-dotless; I-dot and i are case pairs in Turkish and Azeri
+# The following rules handle those cases.
+
+0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# When lowercasing, remove dot_above in the sequence I + dot_above, which will turn into i.
+# This matches the behavior of the canonically equivalent I-dot_above
+
+0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE
+0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE
+
+# When lowercasing, unless an I is before a dot_above, it turns into a dotless i.
+
+0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I
+0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I
+
+# When uppercasing, i turns into a dotted capital I
+
+0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I
+0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I
+
+# Note: the following case is already in the UnicodeData file.
+
+# 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I
+
+# EOF
diff --git a/rpython/rlib/unicodedata/SpecialCasing-6.0.0.txt b/rpython/rlib/unicodedata/SpecialCasing-6.0.0.txt
new file mode 100644
--- /dev/null
+++ b/rpython/rlib/unicodedata/SpecialCasing-6.0.0.txt
@@ -0,0 +1,273 @@
+# SpecialCasing-6.0.0.txt
+# Date: 2010-05-18, 00:49:39 GMT [MD]
+#
+# Unicode Character Database
+# Copyright (c) 1991-2010 Unicode, Inc.
+# For terms of use, see http://www.unicode.org/terms_of_use.html
+# For documentation, see http://www.unicode.org/reports/tr44/
+#
+# Special Casing Properties
+#
+# This file is a supplement to the UnicodeData file.
+# It contains additional information about the casing of Unicode characters.
+# (For compatibility, the UnicodeData.txt file only contains case mappings for
+# characters where they are 1-1, and independent of context and language.
+# For more information, see the discussion of Case Mappings in the Unicode Standard.
+#
+# All code points not listed in this file that do not have a simple case mappings
+# in UnicodeData.txt map to themselves.
+# ================================================================================
+# Format
+# ================================================================================
+# The entries in this file are in the following machine-readable format:
+#
+# <code>; <lower> ; <title> ; <upper> ; (<condition_list> ;)? # <comment>
+#
+# <code>, <lower>, <title>, and <upper> provide character values in hex. If there is more
+# than one character, they are separated by spaces. Other than as used to separate 
+# elements, spaces are to be ignored.
+#
+# The <condition_list> is optional. Where present, it consists of one or more language IDs
+# or contexts, separated by spaces. In these conditions:
+# - A condition list overrides the normal behavior if all of the listed conditions are true.
+# - The context is always the context of the characters in the original string,
+#   NOT in the resulting string.
+# - Case distinctions in the condition list are not significant.
+# - Conditions preceded by "Not_" represent the negation of the condition.
+# The condition list is not represented in the UCD as a formal property.
+#
+# A language ID is defined by BCP 47, with '-' and '_' treated equivalently.
+#
+# A context for a character C is defined by Section 3.13 Default Case 
+# Operations, of The Unicode Standard, Version 5.0.
+# (This is identical to the context defined by Unicode 4.1.0,
+#  as specified in http://www.unicode.org/versions/Unicode4.1.0/)
+#
+# Parsers of this file must be prepared to deal with future additions to this format:
+#  * Additional contexts
+#  * Additional fields
+# ================================================================================
+# @missing 0000..10FFFF; <slc>; <stc>; <suc>
+# ================================================================================
+# Unconditional mappings
+# ================================================================================
+
+# The German es-zed is special--the normal mapping is to SS.
+# Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase(<es-zed>))
+
+00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S
+
+# Preserve canonical equivalence for I with dot. Turkic is handled below.
+
+0130; 0069 0307; 0130; 0130; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# Ligatures
+
+FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF
+FB01; FB01; 0046 0069; 0046 0049; # LATIN SMALL LIGATURE FI
+FB02; FB02; 0046 006C; 0046 004C; # LATIN SMALL LIGATURE FL
+FB03; FB03; 0046 0066 0069; 0046 0046 0049; # LATIN SMALL LIGATURE FFI
+FB04; FB04; 0046 0066 006C; 0046 0046 004C; # LATIN SMALL LIGATURE FFL
+FB05; FB05; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE LONG S T
+FB06; FB06; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE ST
+
+0587; 0587; 0535 0582; 0535 0552; # ARMENIAN SMALL LIGATURE ECH YIWN
+FB13; FB13; 0544 0576; 0544 0546; # ARMENIAN SMALL LIGATURE MEN NOW
+FB14; FB14; 0544 0565; 0544 0535; # ARMENIAN SMALL LIGATURE MEN ECH
+FB15; FB15; 0544 056B; 0544 053B; # ARMENIAN SMALL LIGATURE MEN INI
+FB16; FB16; 054E 0576; 054E 0546; # ARMENIAN SMALL LIGATURE VEW NOW
+FB17; FB17; 0544 056D; 0544 053D; # ARMENIAN SMALL LIGATURE MEN XEH
+
+# No corresponding uppercase precomposed character
+
+0149; 0149; 02BC 004E; 02BC 004E; # LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
+0390; 0390; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
+03B0; 03B0; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
+01F0; 01F0; 004A 030C; 004A 030C; # LATIN SMALL LETTER J WITH CARON
+1E96; 1E96; 0048 0331; 0048 0331; # LATIN SMALL LETTER H WITH LINE BELOW
+1E97; 1E97; 0054 0308; 0054 0308; # LATIN SMALL LETTER T WITH DIAERESIS
+1E98; 1E98; 0057 030A; 0057 030A; # LATIN SMALL LETTER W WITH RING ABOVE
+1E99; 1E99; 0059 030A; 0059 030A; # LATIN SMALL LETTER Y WITH RING ABOVE
+1E9A; 1E9A; 0041 02BE; 0041 02BE; # LATIN SMALL LETTER A WITH RIGHT HALF RING
+1F50; 1F50; 03A5 0313; 03A5 0313; # GREEK SMALL LETTER UPSILON WITH PSILI
+1F52; 1F52; 03A5 0313 0300; 03A5 0313 0300; # GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
+1F54; 1F54; 03A5 0313 0301; 03A5 0313 0301; # GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
+1F56; 1F56; 03A5 0313 0342; 03A5 0313 0342; # GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
+1FB6; 1FB6; 0391 0342; 0391 0342; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI
+1FC6; 1FC6; 0397 0342; 0397 0342; # GREEK SMALL LETTER ETA WITH PERISPOMENI
+1FD2; 1FD2; 0399 0308 0300; 0399 0308 0300; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
+1FD3; 1FD3; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
+1FD6; 1FD6; 0399 0342; 0399 0342; # GREEK SMALL LETTER IOTA WITH PERISPOMENI
+1FD7; 1FD7; 0399 0308 0342; 0399 0308 0342; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
+1FE2; 1FE2; 03A5 0308 0300; 03A5 0308 0300; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
+1FE3; 1FE3; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
+1FE4; 1FE4; 03A1 0313; 03A1 0313; # GREEK SMALL LETTER RHO WITH PSILI
+1FE6; 1FE6; 03A5 0342; 03A5 0342; # GREEK SMALL LETTER UPSILON WITH PERISPOMENI
+1FE7; 1FE7; 03A5 0308 0342; 03A5 0308 0342; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
+1FF6; 1FF6; 03A9 0342; 03A9 0342; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI
+
+# IMPORTANT-when iota-subscript (0345) is uppercased or titlecased,
+#  the result will be incorrect unless the iota-subscript is moved to the end
+#  of any sequence of combining marks. Otherwise, the accents will go on the capital iota.
+#  This process can be achieved by first transforming the text to NFC before casing.
+#  E.g. <alpha><iota_subscript><acute> is uppercased to <ALPHA><acute><IOTA>
+
+# The following cases are already in the UnicodeData file, so are only commented here.
+
+# 0345; 0345; 0345; 0399; # COMBINING GREEK YPOGEGRAMMENI
+
+# All letters with YPOGEGRAMMENI (iota-subscript) or PROSGEGRAMMENI (iota adscript)
+# have special uppercases.
+# Note: characters with PROSGEGRAMMENI are actually titlecase, not uppercase!
+
+1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI
+1F81; 1F81; 1F89; 1F09 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND YPOGEGRAMMENI
+1F82; 1F82; 1F8A; 1F0A 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F83; 1F83; 1F8B; 1F0B 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F84; 1F84; 1F8C; 1F0C 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F85; 1F85; 1F8D; 1F0D 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F86; 1F86; 1F8E; 1F0E 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F87; 1F87; 1F8F; 1F0F 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F88; 1F80; 1F88; 1F08 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PROSGEGRAMMENI
+1F89; 1F81; 1F89; 1F09 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PROSGEGRAMMENI
+1F8A; 1F82; 1F8A; 1F0A 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F8B; 1F83; 1F8B; 1F0B 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F8C; 1F84; 1F8C; 1F0C 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F8D; 1F85; 1F8D; 1F0D 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F8E; 1F86; 1F8E; 1F0E 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F8F; 1F87; 1F8F; 1F0F 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1F90; 1F90; 1F98; 1F28 0399; # GREEK SMALL LETTER ETA WITH PSILI AND YPOGEGRAMMENI
+1F91; 1F91; 1F99; 1F29 0399; # GREEK SMALL LETTER ETA WITH DASIA AND YPOGEGRAMMENI
+1F92; 1F92; 1F9A; 1F2A 0399; # GREEK SMALL LETTER ETA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F93; 1F93; 1F9B; 1F2B 0399; # GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F94; 1F94; 1F9C; 1F2C 0399; # GREEK SMALL LETTER ETA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F95; 1F95; 1F9D; 1F2D 0399; # GREEK SMALL LETTER ETA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F96; 1F96; 1F9E; 1F2E 0399; # GREEK SMALL LETTER ETA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F97; 1F97; 1F9F; 1F2F 0399; # GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F98; 1F90; 1F98; 1F28 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PROSGEGRAMMENI
+1F99; 1F91; 1F99; 1F29 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PROSGEGRAMMENI
+1F9A; 1F92; 1F9A; 1F2A 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F9B; 1F93; 1F9B; 1F2B 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F9C; 1F94; 1F9C; 1F2C 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F9D; 1F95; 1F9D; 1F2D 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F9E; 1F96; 1F9E; 1F2E 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F9F; 1F97; 1F9F; 1F2F 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FA0; 1FA0; 1FA8; 1F68 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND YPOGEGRAMMENI
+1FA1; 1FA1; 1FA9; 1F69 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND YPOGEGRAMMENI
+1FA2; 1FA2; 1FAA; 1F6A 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1FA3; 1FA3; 1FAB; 1F6B 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1FA4; 1FA4; 1FAC; 1F6C 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1FA5; 1FA5; 1FAD; 1F6D 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1FA6; 1FA6; 1FAE; 1F6E 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1FA7; 1FA7; 1FAF; 1F6F 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1FA8; 1FA0; 1FA8; 1F68 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PROSGEGRAMMENI
+1FA9; 1FA1; 1FA9; 1F69 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PROSGEGRAMMENI
+1FAA; 1FA2; 1FAA; 1F6A 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1FAB; 1FA3; 1FAB; 1F6B 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1FAC; 1FA4; 1FAC; 1F6C 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1FAD; 1FA5; 1FAD; 1F6D 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1FAE; 1FA6; 1FAE; 1F6E 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1FAF; 1FA7; 1FAF; 1F6F 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FB3; 1FB3; 1FBC; 0391 0399; # GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI
+1FBC; 1FB3; 1FBC; 0391 0399; # GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI
+1FC3; 1FC3; 1FCC; 0397 0399; # GREEK SMALL LETTER ETA WITH YPOGEGRAMMENI
+1FCC; 1FC3; 1FCC; 0397 0399; # GREEK CAPITAL LETTER ETA WITH PROSGEGRAMMENI
+1FF3; 1FF3; 1FFC; 03A9 0399; # GREEK SMALL LETTER OMEGA WITH YPOGEGRAMMENI
+1FFC; 1FF3; 1FFC; 03A9 0399; # GREEK CAPITAL LETTER OMEGA WITH PROSGEGRAMMENI
+
+# Some characters with YPOGEGRAMMENI also have no corresponding titlecases
+
+1FB2; 1FB2; 1FBA 0345; 1FBA 0399; # GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI
+1FB4; 1FB4; 0386 0345; 0386 0399; # GREEK SMALL LETTER ALPHA WITH OXIA AND YPOGEGRAMMENI
+1FC2; 1FC2; 1FCA 0345; 1FCA 0399; # GREEK SMALL LETTER ETA WITH VARIA AND YPOGEGRAMMENI
+1FC4; 1FC4; 0389 0345; 0389 0399; # GREEK SMALL LETTER ETA WITH OXIA AND YPOGEGRAMMENI
+1FF2; 1FF2; 1FFA 0345; 1FFA 0399; # GREEK SMALL LETTER OMEGA WITH VARIA AND YPOGEGRAMMENI
+1FF4; 1FF4; 038F 0345; 038F 0399; # GREEK SMALL LETTER OMEGA WITH OXIA AND YPOGEGRAMMENI
+
+1FB7; 1FB7; 0391 0342 0345; 0391 0342 0399; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FC7; 1FC7; 0397 0342 0345; 0397 0342 0399; # GREEK SMALL LETTER ETA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FF7; 1FF7; 03A9 0342 0345; 03A9 0342 0399; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI AND YPOGEGRAMMENI
+
+# ================================================================================
+# Conditional Mappings
+# The remainder of this file provides conditional casing data used to produce 
+# full case mappings.
+# ================================================================================
+# Language-Insensitive Mappings
+# These are characters whose full case mappings do not depend on language, but do
+# depend on context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Special case for final form of sigma
+
+03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
+
+# Note: the following cases for non-final are already in the UnicodeData file.
+
+# 03A3; 03C3; 03A3; 03A3; # GREEK CAPITAL LETTER SIGMA
+# 03C3; 03C3; 03A3; 03A3; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C2; 03A3; 03A3; # GREEK SMALL LETTER FINAL SIGMA
+
+# Note: the following cases are not included, since they would case-fold in lowercasing
+
+# 03C3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C3; 03A3; 03A3; Not_Final_Sigma; # GREEK SMALL LETTER FINAL SIGMA
+
+# ================================================================================
+# Language-Sensitive Mappings
+# These are characters whose full case mappings depend on language and perhaps also
+# context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Lithuanian
+
+# Lithuanian retains the dot in a lowercase i when followed by accents.
+
+# Remove DOT ABOVE after "i" with upper or titlecase
+
+0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
+
+# Introduce an explicit dot above when lowercasing capital I's and J's
+# whenever there are more accents above.
+# (of the accents used in Lithuanian: grave, acute, tilde above, and ogonek)
+
+0049; 0069 0307; 0049; 0049; lt More_Above; # LATIN CAPITAL LETTER I
+004A; 006A 0307; 004A; 004A; lt More_Above; # LATIN CAPITAL LETTER J
+012E; 012F 0307; 012E; 012E; lt More_Above; # LATIN CAPITAL LETTER I WITH OGONEK
+00CC; 0069 0307 0300; 00CC; 00CC; lt; # LATIN CAPITAL LETTER I WITH GRAVE
+00CD; 0069 0307 0301; 00CD; 00CD; lt; # LATIN CAPITAL LETTER I WITH ACUTE
+0128; 0069 0307 0303; 0128; 0128; lt; # LATIN CAPITAL LETTER I WITH TILDE
+
+# ================================================================================
+
+# Turkish and Azeri
+
+# I and i-dotless; I-dot and i are case pairs in Turkish and Azeri
+# The following rules handle those cases.
+
+0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# When lowercasing, remove dot_above in the sequence I + dot_above, which will turn into i.
+# This matches the behavior of the canonically equivalent I-dot_above
+
+0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE
+0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE
+
+# When lowercasing, unless an I is before a dot_above, it turns into a dotless i.
+
+0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I
+0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I
+
+# When uppercasing, i turns into a dotted capital I
+
+0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I
+0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I
+
+# Note: the following case is already in the UnicodeData file.
+
+# 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I
+
+# EOF
diff --git a/rpython/rlib/unicodedata/SpecialCasing-6.2.0.txt b/rpython/rlib/unicodedata/SpecialCasing-6.2.0.txt
new file mode 100644
--- /dev/null
+++ b/rpython/rlib/unicodedata/SpecialCasing-6.2.0.txt
@@ -0,0 +1,275 @@
+# SpecialCasing-6.2.0.txt
+# Date: 2012-05-23, 20:35:15 GMT [MD]
+#
+# Unicode Character Database
+# Copyright (c) 1991-2012 Unicode, Inc.
+# For terms of use, see http://www.unicode.org/terms_of_use.html
+# For documentation, see http://www.unicode.org/reports/tr44/
+#
+# Special Casing Properties
+#
+# This file is a supplement to the UnicodeData file.
+# It contains additional information about the casing of Unicode characters.
+# (For compatibility, the UnicodeData.txt file only contains case mappings for
+# characters where they are 1-1, and independent of context and language.
+# For more information, see the discussion of Case Mappings in the Unicode Standard.
+#
+# All code points not listed in this file that do not have a simple case mappings
+# in UnicodeData.txt map to themselves.
+# ================================================================================
+# Format
+# ================================================================================
+# The entries in this file are in the following machine-readable format:
+#
+# <code>; <lower> ; <title> ; <upper> ; (<condition_list> ;)? # <comment>
+#
+# <code>, <lower>, <title>, and <upper> provide character values in hex. If there is more
+# than one character, they are separated by spaces. Other than as used to separate 
+# elements, spaces are to be ignored.
+#
+# The <condition_list> is optional. Where present, it consists of one or more language IDs
+# or contexts, separated by spaces. In these conditions:
+# - A condition list overrides the normal behavior if all of the listed conditions are true.
+# - The context is always the context of the characters in the original string,
+#   NOT in the resulting string.
+# - Case distinctions in the condition list are not significant.
+# - Conditions preceded by "Not_" represent the negation of the condition.
+# The condition list is not represented in the UCD as a formal property.
+#
+# A language ID is defined by BCP 47, with '-' and '_' treated equivalently.
+#
+# A context for a character C is defined by Section 3.13 Default Case 
+# Operations, of The Unicode Standard, Version 5.0.
+# (This is identical to the context defined by Unicode 4.1.0,
+#  as specified in http://www.unicode.org/versions/Unicode4.1.0/)
+#
+# Parsers of this file must be prepared to deal with future additions to this format:
+#  * Additional contexts
+#  * Additional fields
+# ================================================================================
+
+# @missing: 0000..10FFFF; <slc>; <stc>; <suc>;
+
+# ================================================================================
+# Unconditional mappings
+# ================================================================================
+
+# The German es-zed is special--the normal mapping is to SS.
+# Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase(<es-zed>))
+
+00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S
+
+# Preserve canonical equivalence for I with dot. Turkic is handled below.
+
+0130; 0069 0307; 0130; 0130; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# Ligatures
+
+FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF
+FB01; FB01; 0046 0069; 0046 0049; # LATIN SMALL LIGATURE FI
+FB02; FB02; 0046 006C; 0046 004C; # LATIN SMALL LIGATURE FL
+FB03; FB03; 0046 0066 0069; 0046 0046 0049; # LATIN SMALL LIGATURE FFI
+FB04; FB04; 0046 0066 006C; 0046 0046 004C; # LATIN SMALL LIGATURE FFL
+FB05; FB05; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE LONG S T
+FB06; FB06; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE ST
+
+0587; 0587; 0535 0582; 0535 0552; # ARMENIAN SMALL LIGATURE ECH YIWN
+FB13; FB13; 0544 0576; 0544 0546; # ARMENIAN SMALL LIGATURE MEN NOW
+FB14; FB14; 0544 0565; 0544 0535; # ARMENIAN SMALL LIGATURE MEN ECH
+FB15; FB15; 0544 056B; 0544 053B; # ARMENIAN SMALL LIGATURE MEN INI
+FB16; FB16; 054E 0576; 054E 0546; # ARMENIAN SMALL LIGATURE VEW NOW
+FB17; FB17; 0544 056D; 0544 053D; # ARMENIAN SMALL LIGATURE MEN XEH
+
+# No corresponding uppercase precomposed character
+
+0149; 0149; 02BC 004E; 02BC 004E; # LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
+0390; 0390; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
+03B0; 03B0; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
+01F0; 01F0; 004A 030C; 004A 030C; # LATIN SMALL LETTER J WITH CARON
+1E96; 1E96; 0048 0331; 0048 0331; # LATIN SMALL LETTER H WITH LINE BELOW
+1E97; 1E97; 0054 0308; 0054 0308; # LATIN SMALL LETTER T WITH DIAERESIS
+1E98; 1E98; 0057 030A; 0057 030A; # LATIN SMALL LETTER W WITH RING ABOVE
+1E99; 1E99; 0059 030A; 0059 030A; # LATIN SMALL LETTER Y WITH RING ABOVE
+1E9A; 1E9A; 0041 02BE; 0041 02BE; # LATIN SMALL LETTER A WITH RIGHT HALF RING
+1F50; 1F50; 03A5 0313; 03A5 0313; # GREEK SMALL LETTER UPSILON WITH PSILI
+1F52; 1F52; 03A5 0313 0300; 03A5 0313 0300; # GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
+1F54; 1F54; 03A5 0313 0301; 03A5 0313 0301; # GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
+1F56; 1F56; 03A5 0313 0342; 03A5 0313 0342; # GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
+1FB6; 1FB6; 0391 0342; 0391 0342; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI
+1FC6; 1FC6; 0397 0342; 0397 0342; # GREEK SMALL LETTER ETA WITH PERISPOMENI
+1FD2; 1FD2; 0399 0308 0300; 0399 0308 0300; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
+1FD3; 1FD3; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
+1FD6; 1FD6; 0399 0342; 0399 0342; # GREEK SMALL LETTER IOTA WITH PERISPOMENI
+1FD7; 1FD7; 0399 0308 0342; 0399 0308 0342; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
+1FE2; 1FE2; 03A5 0308 0300; 03A5 0308 0300; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
+1FE3; 1FE3; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
+1FE4; 1FE4; 03A1 0313; 03A1 0313; # GREEK SMALL LETTER RHO WITH PSILI
+1FE6; 1FE6; 03A5 0342; 03A5 0342; # GREEK SMALL LETTER UPSILON WITH PERISPOMENI
+1FE7; 1FE7; 03A5 0308 0342; 03A5 0308 0342; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
+1FF6; 1FF6; 03A9 0342; 03A9 0342; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI
+
+# IMPORTANT-when iota-subscript (0345) is uppercased or titlecased,
+#  the result will be incorrect unless the iota-subscript is moved to the end
+#  of any sequence of combining marks. Otherwise, the accents will go on the capital iota.
+#  This process can be achieved by first transforming the text to NFC before casing.
+#  E.g. <alpha><iota_subscript><acute> is uppercased to <ALPHA><acute><IOTA>
+
+# The following cases are already in the UnicodeData file, so are only commented here.
+
+# 0345; 0345; 0345; 0399; # COMBINING GREEK YPOGEGRAMMENI
+
+# All letters with YPOGEGRAMMENI (iota-subscript) or PROSGEGRAMMENI (iota adscript)
+# have special uppercases.
+# Note: characters with PROSGEGRAMMENI are actually titlecase, not uppercase!
+
+1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI
+1F81; 1F81; 1F89; 1F09 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND YPOGEGRAMMENI
+1F82; 1F82; 1F8A; 1F0A 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F83; 1F83; 1F8B; 1F0B 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F84; 1F84; 1F8C; 1F0C 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F85; 1F85; 1F8D; 1F0D 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F86; 1F86; 1F8E; 1F0E 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F87; 1F87; 1F8F; 1F0F 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F88; 1F80; 1F88; 1F08 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PROSGEGRAMMENI
+1F89; 1F81; 1F89; 1F09 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PROSGEGRAMMENI
+1F8A; 1F82; 1F8A; 1F0A 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F8B; 1F83; 1F8B; 1F0B 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F8C; 1F84; 1F8C; 1F0C 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F8D; 1F85; 1F8D; 1F0D 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F8E; 1F86; 1F8E; 1F0E 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F8F; 1F87; 1F8F; 1F0F 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1F90; 1F90; 1F98; 1F28 0399; # GREEK SMALL LETTER ETA WITH PSILI AND YPOGEGRAMMENI
+1F91; 1F91; 1F99; 1F29 0399; # GREEK SMALL LETTER ETA WITH DASIA AND YPOGEGRAMMENI
+1F92; 1F92; 1F9A; 1F2A 0399; # GREEK SMALL LETTER ETA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1F93; 1F93; 1F9B; 1F2B 0399; # GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1F94; 1F94; 1F9C; 1F2C 0399; # GREEK SMALL LETTER ETA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1F95; 1F95; 1F9D; 1F2D 0399; # GREEK SMALL LETTER ETA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1F96; 1F96; 1F9E; 1F2E 0399; # GREEK SMALL LETTER ETA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1F97; 1F97; 1F9F; 1F2F 0399; # GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1F98; 1F90; 1F98; 1F28 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PROSGEGRAMMENI
+1F99; 1F91; 1F99; 1F29 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PROSGEGRAMMENI
+1F9A; 1F92; 1F9A; 1F2A 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1F9B; 1F93; 1F9B; 1F2B 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1F9C; 1F94; 1F9C; 1F2C 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1F9D; 1F95; 1F9D; 1F2D 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1F9E; 1F96; 1F9E; 1F2E 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1F9F; 1F97; 1F9F; 1F2F 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FA0; 1FA0; 1FA8; 1F68 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND YPOGEGRAMMENI
+1FA1; 1FA1; 1FA9; 1F69 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND YPOGEGRAMMENI
+1FA2; 1FA2; 1FAA; 1F6A 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND VARIA AND YPOGEGRAMMENI
+1FA3; 1FA3; 1FAB; 1F6B 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND VARIA AND YPOGEGRAMMENI
+1FA4; 1FA4; 1FAC; 1F6C 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND OXIA AND YPOGEGRAMMENI
+1FA5; 1FA5; 1FAD; 1F6D 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND OXIA AND YPOGEGRAMMENI
+1FA6; 1FA6; 1FAE; 1F6E 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
+1FA7; 1FA7; 1FAF; 1F6F 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
+1FA8; 1FA0; 1FA8; 1F68 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PROSGEGRAMMENI
+1FA9; 1FA1; 1FA9; 1F69 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PROSGEGRAMMENI
+1FAA; 1FA2; 1FAA; 1F6A 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND VARIA AND PROSGEGRAMMENI
+1FAB; 1FA3; 1FAB; 1F6B 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND VARIA AND PROSGEGRAMMENI
+1FAC; 1FA4; 1FAC; 1F6C 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND OXIA AND PROSGEGRAMMENI
+1FAD; 1FA5; 1FAD; 1F6D 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND OXIA AND PROSGEGRAMMENI
+1FAE; 1FA6; 1FAE; 1F6E 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
+1FAF; 1FA7; 1FAF; 1F6F 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
+1FB3; 1FB3; 1FBC; 0391 0399; # GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI
+1FBC; 1FB3; 1FBC; 0391 0399; # GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI
+1FC3; 1FC3; 1FCC; 0397 0399; # GREEK SMALL LETTER ETA WITH YPOGEGRAMMENI
+1FCC; 1FC3; 1FCC; 0397 0399; # GREEK CAPITAL LETTER ETA WITH PROSGEGRAMMENI
+1FF3; 1FF3; 1FFC; 03A9 0399; # GREEK SMALL LETTER OMEGA WITH YPOGEGRAMMENI
+1FFC; 1FF3; 1FFC; 03A9 0399; # GREEK CAPITAL LETTER OMEGA WITH PROSGEGRAMMENI
+
+# Some characters with YPOGEGRAMMENI also have no corresponding titlecases
+
+1FB2; 1FB2; 1FBA 0345; 1FBA 0399; # GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI
+1FB4; 1FB4; 0386 0345; 0386 0399; # GREEK SMALL LETTER ALPHA WITH OXIA AND YPOGEGRAMMENI
+1FC2; 1FC2; 1FCA 0345; 1FCA 0399; # GREEK SMALL LETTER ETA WITH VARIA AND YPOGEGRAMMENI
+1FC4; 1FC4; 0389 0345; 0389 0399; # GREEK SMALL LETTER ETA WITH OXIA AND YPOGEGRAMMENI
+1FF2; 1FF2; 1FFA 0345; 1FFA 0399; # GREEK SMALL LETTER OMEGA WITH VARIA AND YPOGEGRAMMENI
+1FF4; 1FF4; 038F 0345; 038F 0399; # GREEK SMALL LETTER OMEGA WITH OXIA AND YPOGEGRAMMENI
+
+1FB7; 1FB7; 0391 0342 0345; 0391 0342 0399; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FC7; 1FC7; 0397 0342 0345; 0397 0342 0399; # GREEK SMALL LETTER ETA WITH PERISPOMENI AND YPOGEGRAMMENI
+1FF7; 1FF7; 03A9 0342 0345; 03A9 0342 0399; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI AND YPOGEGRAMMENI
+
+# ================================================================================
+# Conditional Mappings
+# The remainder of this file provides conditional casing data used to produce 
+# full case mappings.
+# ================================================================================
+# Language-Insensitive Mappings
+# These are characters whose full case mappings do not depend on language, but do
+# depend on context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Special case for final form of sigma
+
+03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
+
+# Note: the following cases for non-final are already in the UnicodeData file.
+
+# 03A3; 03C3; 03A3; 03A3; # GREEK CAPITAL LETTER SIGMA
+# 03C3; 03C3; 03A3; 03A3; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C2; 03A3; 03A3; # GREEK SMALL LETTER FINAL SIGMA
+
+# Note: the following cases are not included, since they would case-fold in lowercasing
+
+# 03C3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK SMALL LETTER SIGMA
+# 03C2; 03C3; 03A3; 03A3; Not_Final_Sigma; # GREEK SMALL LETTER FINAL SIGMA
+
+# ================================================================================
+# Language-Sensitive Mappings
+# These are characters whose full case mappings depend on language and perhaps also
+# context (which characters come before or after). For more information
+# see the header of this file and the Unicode Standard.
+# ================================================================================
+
+# Lithuanian
+
+# Lithuanian retains the dot in a lowercase i when followed by accents.
+
+# Remove DOT ABOVE after "i" with upper or titlecase
+
+0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
+
+# Introduce an explicit dot above when lowercasing capital I's and J's
+# whenever there are more accents above.
+# (of the accents used in Lithuanian: grave, acute, tilde above, and ogonek)
+
+0049; 0069 0307; 0049; 0049; lt More_Above; # LATIN CAPITAL LETTER I
+004A; 006A 0307; 004A; 004A; lt More_Above; # LATIN CAPITAL LETTER J
+012E; 012F 0307; 012E; 012E; lt More_Above; # LATIN CAPITAL LETTER I WITH OGONEK
+00CC; 0069 0307 0300; 00CC; 00CC; lt; # LATIN CAPITAL LETTER I WITH GRAVE
+00CD; 0069 0307 0301; 00CD; 00CD; lt; # LATIN CAPITAL LETTER I WITH ACUTE
+0128; 0069 0307 0303; 0128; 0128; lt; # LATIN CAPITAL LETTER I WITH TILDE
+
+# ================================================================================
+
+# Turkish and Azeri
+
+# I and i-dotless; I-dot and i are case pairs in Turkish and Azeri
+# The following rules handle those cases.
+
+0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE
+
+# When lowercasing, remove dot_above in the sequence I + dot_above, which will turn into i.
+# This matches the behavior of the canonically equivalent I-dot_above
+
+0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE
+0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE
+
+# When lowercasing, unless an I is before a dot_above, it turns into a dotless i.
+
+0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I
+0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I
+
+# When uppercasing, i turns into a dotted capital I
+
+0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I
+0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I
+
+# Note: the following case is already in the UnicodeData file.
+
+# 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I
+
+# EOF
diff --git a/rpython/rlib/unicodedata/generate_unicodedb.py b/rpython/rlib/unicodedata/generate_unicodedb.py
--- a/rpython/rlib/unicodedata/generate_unicodedb.py
+++ b/rpython/rlib/unicodedata/generate_unicodedb.py
@@ -264,6 +264,24 @@
     for code, value in extra_numeric.iteritems():
         table.clone_char(code).numeric = value
 
+    table.special_casing = {}
+    if 'special_casing' in files:
+        for line in files['special_casing']:
+            line = line[:-1].split('#', 1)[0]
+            if not line:
+                continue
+            data = line.split("; ")
+            if data[4]:
+                # We ignore all conditionals (since they depend on
+                # languages) except for one, which is hardcoded. See
+                # handle_capital_sigma in unicodeobject.py.
+                continue
+            c = int(data[0], 16)
+            lower = [int(char, 16) for char in data[1].split()]
+            title = [int(char, 16) for char in data[2].split()]
+            upper = [int(char, 16) for char in data[3].split()]
+            table.special_casing[c] = (lower, title, upper)
+
     # Compute full decompositions.
     for code, char in table.enum_chars():
         table.get_canonical_decomposition(code)
@@ -703,6 +721,7 @@
     writeDict(outfile, '_toupper', toupper, base_mod)
     writeDict(outfile, '_tolower', tolower, base_mod)
     writeDict(outfile, '_totitle', totitle, base_mod)
+    writeDict(outfile, '_special_casing', table.special_casing, base_mod)
     print >> outfile, '''
 def toupper(code):
     try:
@@ -730,6 +749,39 @@
             return base_mod._totitle.get(code, code)
         else:
             return code
+
+def toupper_full(code):
+    try:
+        return _special_casing[code][2]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][2]
+            except KeyError:
+                pass
+    return [toupper(code)]
+
+def tolower_full(code):
+    try:
+        return _special_casing[code][0]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][0]
+            except KeyError:
+                pass
+    return [tolower(code)]
+
+def totitle_full(code):
+    try:
+        return _special_casing[code][1]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][1]
+            except KeyError:
+                pass
+    return [totitle(code)]
 '''
     # Decomposition
     decomposition = {}
@@ -854,6 +906,8 @@
         name_aliases='NameAliases-%(version)s.txt',
         named_sequences = 'NamedSequences-%(version)s.txt',
     )
+    if options.unidata_version > '5':
+        filenames['special_casing'] = 'SpecialCasing-%(version)s.txt'
     filenames = dict((name, filename % dict(version=options.unidata_version))
                      for (name, filename) in filenames.items())
     files = dict((name, open(filename))
diff --git a/rpython/rlib/unicodedata/test/test_unicodedata.py b/rpython/rlib/unicodedata/test/test_unicodedata.py
--- a/rpython/rlib/unicodedata/test/test_unicodedata.py
+++ b/rpython/rlib/unicodedata/test/test_unicodedata.py
@@ -118,3 +118,10 @@
         for un, name in additions.iteritems():
             assert unicodedb_6_0_0.name(un) == name
             assert unicodedb_6_0_0.isprintable(un)
+
+    def test_special_casing(self):
+        assert unicodedb_6_0_0.tolower_full(ord('A')) == [ord('a')]
+        # The German es-zed is special--the normal mapping is to SS.
+        assert unicodedb_6_0_0.tolower_full(ord(u'\xdf')) == [0xdf]
+        assert unicodedb_6_0_0.toupper_full(ord(u'\xdf')) == map(ord, 'SS')
+        assert unicodedb_6_0_0.totitle_full(ord(u'\xdf')) == map(ord, 'Ss')
diff --git a/rpython/rlib/unicodedata/unicodedb_3_2_0.py b/rpython/rlib/unicodedata/unicodedb_3_2_0.py
--- a/rpython/rlib/unicodedata/unicodedb_3_2_0.py
+++ b/rpython/rlib/unicodedata/unicodedb_3_2_0.py
@@ -19403,6 +19403,114 @@
 66638: None,
 66639: None,
 }
+_special_casing = {
+}
+
+_special_casing_corrected = {
+223: None,
+304: None,
+329: None,
+496: None,
+912: None,
+944: None,
+1415: None,
+7830: None,
+7831: None,
+7832: None,
+7833: None,
+7834: None,
+8016: None,
+8018: None,
+8020: None,
+8022: None,
+8064: None,
+8065: None,
+8066: None,
+8067: None,
+8068: None,
+8069: None,
+8070: None,
+8071: None,
+8072: None,
+8073: None,
+8074: None,
+8075: None,
+8076: None,
+8077: None,
+8078: None,
+8079: None,
+8080: None,
+8081: None,
+8082: None,
+8083: None,
+8084: None,
+8085: None,
+8086: None,
+8087: None,
+8088: None,
+8089: None,
+8090: None,
+8091: None,
+8092: None,
+8093: None,
+8094: None,
+8095: None,
+8096: None,
+8097: None,
+8098: None,
+8099: None,
+8100: None,
+8101: None,
+8102: None,
+8103: None,
+8104: None,
+8105: None,
+8106: None,
+8107: None,
+8108: None,
+8109: None,
+8110: None,
+8111: None,
+8114: None,
+8115: None,
+8116: None,
+8118: None,
+8119: None,
+8124: None,
+8130: None,
+8131: None,
+8132: None,
+8134: None,
+8135: None,
+8140: None,
+8146: None,
+8147: None,
+8150: None,
+8151: None,
+8162: None,
+8163: None,
+8164: None,
+8166: None,
+8167: None,
+8178: None,
+8179: None,
+8180: None,
+8182: None,
+8183: None,
+8188: None,
+64256: None,
+64257: None,
+64258: None,
+64259: None,
+64260: None,
+64261: None,
+64262: None,
+64275: None,
+64276: None,
+64277: None,
+64278: None,
+64279: None,
+}
 
 def toupper(code):
     try:
@@ -19431,6 +19539,39 @@
         else:
             return code
 
+def toupper_full(code):
+    try:
+        return _special_casing[code][2]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][2]
+            except KeyError:
+                pass
+    return [toupper(code)]
+
+def tolower_full(code):
+    try:
+        return _special_casing[code][0]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][0]
+            except KeyError:
+                pass
+    return [tolower(code)]
+
+def totitle_full(code):
+    try:
+        return _special_casing[code][1]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][1]
+            except KeyError:
+                pass
+    return [totitle(code)]
+
 _raw_decomposition = {
 194664: '2136A',
 194676: '5F33',
diff --git a/rpython/rlib/unicodedata/unicodedb_5_2_0.py b/rpython/rlib/unicodedata/unicodedb_5_2_0.py
--- a/rpython/rlib/unicodedata/unicodedb_5_2_0.py
+++ b/rpython/rlib/unicodedata/unicodedb_5_2_0.py
@@ -142996,6 +142996,114 @@
 
 _totitle_corrected = {
 }
+_special_casing = {
+223: ([223], [83, 115], [83, 83]),
+304: ([105, 775], [304], [304]),
+329: ([329], [700, 78], [700, 78]),
+496: ([496], [74, 780], [74, 780]),
+912: ([912], [921, 776, 769], [921, 776, 769]),
+944: ([944], [933, 776, 769], [933, 776, 769]),
+1415: ([1415], [1333, 1410], [1333, 1362]),
+7830: ([7830], [72, 817], [72, 817]),
+7831: ([7831], [84, 776], [84, 776]),
+7832: ([7832], [87, 778], [87, 778]),
+7833: ([7833], [89, 778], [89, 778]),
+7834: ([7834], [65, 702], [65, 702]),
+8016: ([8016], [933, 787], [933, 787]),
+8018: ([8018], [933, 787, 768], [933, 787, 768]),
+8020: ([8020], [933, 787, 769], [933, 787, 769]),
+8022: ([8022], [933, 787, 834], [933, 787, 834]),
+8064: ([8064], [8072], [7944, 921]),
+8065: ([8065], [8073], [7945, 921]),
+8066: ([8066], [8074], [7946, 921]),
+8067: ([8067], [8075], [7947, 921]),
+8068: ([8068], [8076], [7948, 921]),
+8069: ([8069], [8077], [7949, 921]),
+8070: ([8070], [8078], [7950, 921]),
+8071: ([8071], [8079], [7951, 921]),
+8072: ([8064], [8072], [7944, 921]),
+8073: ([8065], [8073], [7945, 921]),
+8074: ([8066], [8074], [7946, 921]),
+8075: ([8067], [8075], [7947, 921]),
+8076: ([8068], [8076], [7948, 921]),
+8077: ([8069], [8077], [7949, 921]),
+8078: ([8070], [8078], [7950, 921]),
+8079: ([8071], [8079], [7951, 921]),
+8080: ([8080], [8088], [7976, 921]),
+8081: ([8081], [8089], [7977, 921]),
+8082: ([8082], [8090], [7978, 921]),
+8083: ([8083], [8091], [7979, 921]),
+8084: ([8084], [8092], [7980, 921]),
+8085: ([8085], [8093], [7981, 921]),
+8086: ([8086], [8094], [7982, 921]),
+8087: ([8087], [8095], [7983, 921]),
+8088: ([8080], [8088], [7976, 921]),
+8089: ([8081], [8089], [7977, 921]),
+8090: ([8082], [8090], [7978, 921]),
+8091: ([8083], [8091], [7979, 921]),
+8092: ([8084], [8092], [7980, 921]),
+8093: ([8085], [8093], [7981, 921]),
+8094: ([8086], [8094], [7982, 921]),
+8095: ([8087], [8095], [7983, 921]),
+8096: ([8096], [8104], [8040, 921]),
+8097: ([8097], [8105], [8041, 921]),
+8098: ([8098], [8106], [8042, 921]),
+8099: ([8099], [8107], [8043, 921]),
+8100: ([8100], [8108], [8044, 921]),
+8101: ([8101], [8109], [8045, 921]),
+8102: ([8102], [8110], [8046, 921]),
+8103: ([8103], [8111], [8047, 921]),
+8104: ([8096], [8104], [8040, 921]),
+8105: ([8097], [8105], [8041, 921]),
+8106: ([8098], [8106], [8042, 921]),
+8107: ([8099], [8107], [8043, 921]),
+8108: ([8100], [8108], [8044, 921]),
+8109: ([8101], [8109], [8045, 921]),
+8110: ([8102], [8110], [8046, 921]),
+8111: ([8103], [8111], [8047, 921]),
+8114: ([8114], [8122, 837], [8122, 921]),
+8115: ([8115], [8124], [913, 921]),
+8116: ([8116], [902, 837], [902, 921]),
+8118: ([8118], [913, 834], [913, 834]),
+8119: ([8119], [913, 834, 837], [913, 834, 921]),
+8124: ([8115], [8124], [913, 921]),
+8130: ([8130], [8138, 837], [8138, 921]),
+8131: ([8131], [8140], [919, 921]),
+8132: ([8132], [905, 837], [905, 921]),
+8134: ([8134], [919, 834], [919, 834]),
+8135: ([8135], [919, 834, 837], [919, 834, 921]),
+8140: ([8131], [8140], [919, 921]),
+8146: ([8146], [921, 776, 768], [921, 776, 768]),
+8147: ([8147], [921, 776, 769], [921, 776, 769]),
+8150: ([8150], [921, 834], [921, 834]),
+8151: ([8151], [921, 776, 834], [921, 776, 834]),
+8162: ([8162], [933, 776, 768], [933, 776, 768]),
+8163: ([8163], [933, 776, 769], [933, 776, 769]),
+8164: ([8164], [929, 787], [929, 787]),
+8166: ([8166], [933, 834], [933, 834]),
+8167: ([8167], [933, 776, 834], [933, 776, 834]),
+8178: ([8178], [8186, 837], [8186, 921]),
+8179: ([8179], [8188], [937, 921]),
+8180: ([8180], [911, 837], [911, 921]),
+8182: ([8182], [937, 834], [937, 834]),
+8183: ([8183], [937, 834, 837], [937, 834, 921]),
+8188: ([8179], [8188], [937, 921]),
+64256: ([64256], [70, 102], [70, 70]),
+64257: ([64257], [70, 105], [70, 73]),
+64258: ([64258], [70, 108], [70, 76]),
+64259: ([64259], [70, 102, 105], [70, 70, 73]),
+64260: ([64260], [70, 102, 108], [70, 70, 76]),
+64261: ([64261], [83, 116], [83, 84]),
+64262: ([64262], [83, 116], [83, 84]),
+64275: ([64275], [1348, 1398], [1348, 1350]),
+64276: ([64276], [1348, 1381], [1348, 1333]),
+64277: ([64277], [1348, 1387], [1348, 1339]),
+64278: ([64278], [1358, 1398], [1358, 1350]),
+64279: ([64279], [1348, 1389], [1348, 1341]),
+}
+
+_special_casing_corrected = {
+}
 
 def toupper(code):
     try:
@@ -143024,6 +143132,39 @@
         else:
             return code
 
+def toupper_full(code):
+    try:
+        return _special_casing[code][2]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][2]
+            except KeyError:
+                pass
+    return [toupper(code)]
+
+def tolower_full(code):
+    try:
+        return _special_casing[code][0]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][0]
+            except KeyError:
+                pass
+    return [tolower(code)]
+
+def totitle_full(code):
+    try:
+        return _special_casing[code][1]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][1]
+            except KeyError:
+                pass
+    return [totitle(code)]
+
 _raw_decomposition = {
 160: '<noBreak> 0020',
 168: '<compat> 0020 0308',
diff --git a/rpython/rlib/unicodedata/unicodedb_6_0_0.py b/rpython/rlib/unicodedata/unicodedb_6_0_0.py
--- a/rpython/rlib/unicodedata/unicodedb_6_0_0.py
+++ b/rpython/rlib/unicodedata/unicodedb_6_0_0.py
@@ -5710,6 +5710,11 @@
 
 _totitle_corrected = {
 }
+_special_casing = {
+}
+
+_special_casing_corrected = {
+}
 
 def toupper(code):
     try:
@@ -5738,6 +5743,39 @@
         else:
             return code
 
+def toupper_full(code):
+    try:
+        return _special_casing[code][2]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][2]
+            except KeyError:
+                pass
+    return [toupper(code)]
+
+def tolower_full(code):
+    try:
+        return _special_casing[code][0]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][0]
+            except KeyError:
+                pass
+    return [tolower(code)]
+
+def totitle_full(code):
+    try:
+        return _special_casing[code][1]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][1]
+            except KeyError:
+                pass
+    return [totitle(code)]
+
 _raw_decomposition = {
 8341: '<sub> 0068',
 8342: '<sub> 006B',
diff --git a/rpython/rlib/unicodedata/unicodedb_6_2_0.py b/rpython/rlib/unicodedata/unicodedb_6_2_0.py
--- a/rpython/rlib/unicodedata/unicodedb_6_2_0.py
+++ b/rpython/rlib/unicodedata/unicodedb_6_2_0.py
@@ -8245,6 +8245,11 @@
 
 _totitle_corrected = {
 }
+_special_casing = {
+}
+
+_special_casing_corrected = {
+}
 
 def toupper(code):
     try:
@@ -8273,6 +8278,39 @@
         else:
             return code
 
+def toupper_full(code):
+    try:
+        return _special_casing[code][2]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][2]
+            except KeyError:
+                pass
+    return [toupper(code)]
+
+def tolower_full(code):
+    try:
+        return _special_casing[code][0]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][0]
+            except KeyError:
+                pass
+    return [tolower(code)]
+
+def totitle_full(code):
+    try:
+        return _special_casing[code][1]
+    except KeyError:
+        if base_mod is not None and code not in _special_casing_corrected:
+            try:
+                return base_mod._special_casing[code][1]
+            except KeyError:
+                pass
+    return [totitle(code)]
+
 _raw_decomposition = {
 8341: '<sub> 0068',
 8342: '<sub> 006B',