From pypy.commits at gmail.com Sat Oct 7 09:10:28 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 07 Oct 2017 06:10:28 -0700 (PDT) Subject: [pypy-commit] pypy.org extradoc: tweaks Message-ID: <59d8d244.0ac0df0a.7dbd3.930c@mx.google.com> Author: Matti Picus Branch: extradoc Changeset: r901:275468ee16dc Date: 2017-10-06 00:14 +1100 http://bitbucket.org/pypy/pypy.org/changeset/275468ee16dc/ Log: tweaks diff --git a/download.html b/download.html --- a/download.html +++ b/download.html @@ -407,7 +407,7 @@ 2e464bcbc8216e55bb2433ace712130244fd1f3fa78de0c0c98745fd8ff12b03 pypy2-v5.8.0-ppc64.tar.bz2 5746823904df74423376e0326046e1171df9693a6d4c95e8ce14ca83534bae72 pypy2-v5.8.0-ppc64le.tar.bz2 -

pypy2.7-5.9.0 sha256

pypy2.7-5.9.0 sha256:

a2431a9e4ef879da1a2b56b111013b4a6efb87d4173a37bf650de47834ac5fe4 pypy2-v5.9.0-linux32.tar.bz2 790febd4f09e22d6e2f81154efc7dc4b2feec72712aaf4f82aa91b550abb4b48 pypy2-v5.9.0-linux64.tar.bz2 diff --git a/features.html b/features.html --- a/features.html +++ b/features.html @@ -74,7 +74,7 @@
PyPy is a replacement for CPython. It is built using the RPython language that was co-developed with it. The main reason to use it instead of CPython is speed: it runs generally faster (see next section).
-
PyPy implements Python 2.7.13. +
PyPy implements Python 2.7.13 and 3.5.3. It supports all of the core language, passing the Python test suite (with minor modifications that were already accepted in the main python in newer versions). It supports most of the commonly used Python diff --git a/source/download.txt b/source/download.txt --- a/source/download.txt +++ b/source/download.txt @@ -450,7 +450,7 @@ 2e464bcbc8216e55bb2433ace712130244fd1f3fa78de0c0c98745fd8ff12b03 pypy2-v5.8.0-ppc64.tar.bz2 5746823904df74423376e0326046e1171df9693a6d4c95e8ce14ca83534bae72 pypy2-v5.8.0-ppc64le.tar.bz2 -pypy2.7-5.9.0 sha256 +pypy2.7-5.9.0 sha256: a2431a9e4ef879da1a2b56b111013b4a6efb87d4173a37bf650de47834ac5fe4 pypy2-v5.9.0-linux32.tar.bz2 790febd4f09e22d6e2f81154efc7dc4b2feec72712aaf4f82aa91b550abb4b48 pypy2-v5.9.0-linux64.tar.bz2 diff --git a/source/features.txt b/source/features.txt --- a/source/features.txt +++ b/source/features.txt @@ -10,7 +10,7 @@ language that was co-developed with it. The main reason to use it instead of CPython is speed: it runs generally faster (see next section). -**PyPy** implements **Python 2.7.13**. +**PyPy** implements **Python 2.7.13 and 3.5.3**. It supports all of the core language, passing the Python test suite (with minor modifications that were already accepted in the main python in newer versions). It supports most of the commonly used Python From pypy.commits at gmail.com Sat Oct 7 09:10:32 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 06:10:32 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: Remove obsolete description of the api object Message-ID: <59d8d248.c88b1c0a.4662b.2a93@mx.google.com> Author: Ronan Lamy Branch: cpyext-nowrapper Changeset: r92611:433e01a0e7fc Date: 2017-10-05 16:23 +0200 http://bitbucket.org/pypy/pypy/changeset/433e01a0e7fc/ Log: Remove obsolete description of the api object diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -181,21 +181,20 @@ # The same function can be called in three different contexts: # (1) from C code -# (2) in the test suite, though the "api" object -# (3) from RPython code, for example in the implementation of another function. +# (2) from RPython code, for example in the implementation of another function +# or in tests. # -# In contexts (2) and (3), a function declaring a PyObject argument type will +# In context (2), a function declaring a PyObject argument type will # receive a wrapped pypy object if the parameter name starts with 'w_', a -# reference (= rffi pointer) otherwise; conversion is automatic. Context (2) -# only allows calls with a wrapped object. +# reference (= rffi pointer) otherwise; conversion is automatic. # # Functions with a PyObject return type should return a wrapped object. # -# Functions may raise exceptions. In context (3), the exception flows normally -# through the calling function. In context (1) and (2), the exception is +# Functions may raise exceptions. In context (2), the exception flows normally +# through the calling function. In context (1), the exception is # caught; if it is an OperationError, it is stored in the thread state; other -# exceptions generate a OperationError(w_SystemError); and the funtion returns -# the error value specifed in the API. +# exceptions generate a OperationError(w_SystemError); and the function returns +# the error value specified in the API. # # Handling of the GIL # ------------------- From pypy.commits at gmail.com Sat Oct 7 09:10:35 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 06:10:35 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: oops Message-ID: <59d8d24b.472e1c0a.59b2b.9503@mx.google.com> Author: Ronan Lamy Branch: cpyext-nowrapper Changeset: r92612:67d393c4cc8c Date: 2017-10-05 16:26 +0200 http://bitbucket.org/pypy/pypy/changeset/67d393c4cc8c/ Log: oops diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -179,7 +179,7 @@ pass CANNOT_FAIL = CannotFail() -# The same function can be called in three different contexts: +# The same function can be called in two different contexts: # (1) from C code # (2) from RPython code, for example in the implementation of another function # or in tests. From pypy.commits at gmail.com Sat Oct 7 09:10:38 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 06:10:38 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: whack whack whack; Message-ID: <59d8d24e.4fabdf0a.5fbf4.d83a@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92613:15eb01ac7f57 Date: 2017-10-05 17:14 +0200 http://bitbucket.org/pypy/pypy/changeset/15eb01ac7f57/ Log: whack whack whack; diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -126,7 +126,7 @@ self.orig = handler def handle(self, errors, encoding, msg, s, pos, endpos): - s, p, lgt = self.orig(errors, encoding, msg, s, pos, endpos) + s, p = self.orig(errors, encoding, msg, s, pos, endpos) return s.decode("utf8"), p class EncodeWrapper(object): @@ -134,8 +134,7 @@ self.orig = handler def handle(self, errors, encoding, msg, s, pos, endpos): - s, rs, p, lgt = self.orig(errors, encoding, msg, s.encode("utf8"), pos, endpos) - return s, rs, p + return self.orig(errors, encoding, msg, s.encode("utf8"), pos, endpos) # some irregular interfaces def str_decode_utf8(s, slen, errors, final, errorhandler): diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,7 +66,7 @@ "position %d from error handler out of bounds", newpos) w_replace = space.convert_to_w_unicode(w_replace) - return w_replace._utf8, newpos, w_replace._length + return w_replace._utf8, newpos return call_errorhandler def make_decode_errorhandler(self, space): @@ -443,8 +443,7 @@ # "allow_surrogates=True" @unwrap_spec(utf8='utf8', errors='text_or_none') def utf_8_encode(space, utf8, errors="strict"): - raise Exception('foo') - return space.newtuple([space.newbytes(utf8), space.newint(utf8len)]) + return space.newtuple([space.newbytes(utf8), space.newint(rutf8.check_utf8(utf8))]) #@unwrap_spec(uni=unicode, errors='text_or_none') #def utf_8_encode(space, uni, errors="strict"): # if errors is None: diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -776,31 +776,28 @@ def encode_object(space, w_object, encoding, errors): + w_encoder = None if encoding is None: # Get the encoder functions as a wrapped object. # This lookup is cached. w_encoder = space.sys.get_w_default_encoder() - else: - if errors is None or errors == 'strict': - if encoding == 'ascii': - s = space.utf8_w(w_object) - try: - rutf8.check_ascii(s) - except rutf8.CheckError as a: - eh = unicodehelper.encode_error_handler(space) - u_len = w_object._len() - eh(None, "ascii", "ordinal not in range(128)", s, u_len, - a.pos, a.pos + 1) - assert False, "always raises" - return space.newbytes(s) - if encoding == 'utf-8': - u = space.utf8_w(w_object) - return space.newbytes(u) - # XXX is this enough? - #eh = unicodehelper.raise_unicode_exception_encode - #return space.newbytes(unicode_encode_utf_8( - # u, len(u), None, errorhandler=eh, - # allow_surrogates=True)) + if errors is None or errors == 'strict': + if ((encoding is None and space.sys.defaultencoding == 'ascii') or + encoding == 'ascii'): + s = space.utf8_w(w_object) + try: + rutf8.check_ascii(s) + except rutf8.CheckError as a: + eh = unicodehelper.encode_error_handler(space) + u_len = w_object._len() + eh(None, "ascii", "ordinal not in range(128)", s, u_len, + a.pos, a.pos + 1) + assert False, "always raises" + return space.newbytes(s) + if ((encoding is None and space.sys.defaultencoding == 'utf8') or + encoding == 'utf-8'): + return space.newbytes(space.utf8_w(w_object)) + if w_encoder is None: from pypy.module._codecs.interp_codecs import lookup_codec w_encoder = space.getitem(lookup_codec(space, encoding), space.newint(0)) if errors is None: @@ -821,7 +818,6 @@ encoding = getdefaultencoding(space) if errors is None or errors == 'strict': if encoding == 'ascii': - # XXX error handling s = space.charbuf_w(w_obj) try: rutf8.check_ascii(s) diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -1280,8 +1280,9 @@ collend = pos+1 while collend < len(p) and ord(p[collend]) >= limit: collend += 1 - ru, rs, pos = errorhandler(errors, encoding, reason, p, + ru, pos = errorhandler(errors, encoding, reason, p, collstart, collend) + rs = None if rs is not None: # py3k only result.append(rs) From pypy.commits at gmail.com Sat Oct 7 09:10:41 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 06:10:41 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: whack Message-ID: <59d8d251.1986df0a.ede72.a80d@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92614:5344b04bd9c1 Date: 2017-10-05 17:14 +0200 http://bitbucket.org/pypy/pypy/changeset/5344b04bd9c1/ Log: whack diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -113,7 +113,7 @@ def str_decode_ascii(s, slen, errors, final, errorhandler): try: - rutf8.check_ascii(s, slen) + rutf8.check_ascii(s) return s, slen, len(s) except rutf8.CheckError: raise Exception("foo") From pypy.commits at gmail.com Sat Oct 7 09:10:45 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 06:10:45 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: whack whack whack whack. I hate RPython Message-ID: <59d8d255.9b88df0a.6e93f.2cdb@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92615:0379d71a32bf Date: 2017-10-05 18:40 +0200 http://bitbucket.org/pypy/pypy/changeset/0379d71a32bf/ Log: whack whack whack whack. I hate RPython diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -1,1 +1,2 @@ * unskip tests in test_unicodeobject.py +* rutf8.prev_codepoint_pos should use r_uint \ No newline at end of file diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -70,7 +70,7 @@ try: length = rutf8.check_utf8(string, allow_surrogates=True) except rutf8.CheckError as e: - XXX + raise Exception("foo") decode_error_handler(space)('strict', 'utf8', e.msg, string, e.startpos, e.endpos) raise False, "unreachable" diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -37,7 +37,7 @@ assert length >= 0 self._utf8 = utf8str self._length = length - self._index_storage = None + self._index_storage = rutf8.null_storage() if not we_are_translated(): assert rutf8.check_utf8(utf8str, allow_surrogates=True) == length @@ -521,6 +521,8 @@ if keepends: eol = pos lgt += line_end_chars + assert eol >= 0 + assert sol >= 0 strs_w.append(W_UnicodeObject(value[sol:eol], lgt)) return space.newlist(strs_w) @@ -636,7 +638,7 @@ def _getitem_result(self, space, index): if index >= self._length: raise oefmt(space.w_IndexError, "string index out of range") - if self._index_storage is None: + if self._index_storage == rutf8.null_storage(): self._index_storage = rutf8.create_utf8_index_storage(self._utf8, self._length) start = rutf8.codepoint_position_at_index(self._utf8, diff --git a/rpython/rlib/rstring.py b/rpython/rlib/rstring.py --- a/rpython/rlib/rstring.py +++ b/rpython/rlib/rstring.py @@ -7,7 +7,7 @@ from rpython.rtyper.llannotation import SomePtr from rpython.rlib import jit from rpython.rlib.objectmodel import newlist_hint, resizelist_hint, specialize, not_rpython -from rpython.rlib.rarithmetic import ovfcheck, LONG_BIT as BLOOM_WIDTH +from rpython.rlib.rarithmetic import ovfcheck, LONG_BIT as BLOOM_WIDTH, intmask from rpython.rlib.unicodedata import unicodedb_5_2_0 as unicodedb from rpython.rtyper.extregistry import ExtRegistryEntry from rpython.tool.pairtype import pairtype @@ -32,7 +32,9 @@ if isutf8: from rpython.rlib.rutf8 import next_codepoint_pos assert pos >= 0 - return next_codepoint_pos(s, pos) + r = next_codepoint_pos(s, pos) + assert r >= 0 + return r else: return pos + 1 @@ -42,7 +44,7 @@ from rpython.rlib.rutf8 import prev_codepoint_pos if pos <= 0: return -1 - return prev_codepoint_pos(s, pos) + return intmask(prev_codepoint_pos(s, pos)) else: return pos - 1 diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -18,7 +18,7 @@ from rpython.rlib.objectmodel import enforceargs from rpython.rlib.rstring import StringBuilder from rpython.rlib import jit -from rpython.rlib.rarithmetic import r_uint +from rpython.rlib.rarithmetic import r_uint, intmask from rpython.rtyper.lltypesystem import lltype @@ -81,6 +81,7 @@ Assumes valid utf8. 'pos' must be before the end of the string. """ chr1 = ord(code[pos]) + assert pos >= 0 if chr1 <= 0x7F: return pos + 1 if chr1 <= 0xDF: @@ -93,20 +94,24 @@ """Gives the position of the previous codepoint. 'pos' must not be zero. """ - pos = r_uint(pos) - pos -= 1 + pos -= 1 # ruint if pos >= len(code): # for the case where pos - 1 == len(code): + assert pos >= 0 return pos # assume there is an extra '\x00' character chr1 = ord(code[pos]) if chr1 <= 0x7F: + assert pos >= 0 return pos pos -= 1 if ord(code[pos]) >= 0xC0: + assert pos >= 0 return pos pos -= 1 if ord(code[pos]) >= 0xC0: + assert pos >= 0 return pos pos -= 1 + assert pos >= 0 return pos def compute_length_utf8(s): @@ -375,6 +380,9 @@ for _j in range(16): ASCII_INDEX_STORAGE[_i].ofs[_j] = chr(_j * 4 + 1) +def null_storage(): + return lltype.nullptr(UTF8_INDEX_STORAGE) + def create_utf8_index_storage(utf8, utf8len): """ Create an index storage which stores index of each 4th character in utf8 encoded unicode string. @@ -421,6 +429,7 @@ if index == 0: return prev_codepoint_pos(utf8, bytepos) elif index == 1: + assert bytepos >= 0 return bytepos elif index == 2: return next_codepoint_pos(utf8, bytepos) From pypy.commits at gmail.com Sat Oct 7 09:10:48 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 06:10:48 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: change the signature of never called functions Message-ID: <59d8d258.c7341c0a.2c97e.886c@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92616:5118754f8566 Date: 2017-10-05 18:51 +0200 http://bitbucket.org/pypy/pypy/changeset/5118754f8566/ Log: change the signature of never called functions diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -95,17 +95,17 @@ def default_unicode_error_decode(errors, encoding, msg, s, startingpos, endingpos): if errors == 'replace': - return u'\ufffd', endingpos + return u'\ufffd'.encode('utf8'), endingpos if errors == 'ignore': - return u'', endingpos + return '', endingpos raise UnicodeDecodeError(encoding, s, startingpos, endingpos, msg) def default_unicode_error_encode(errors, encoding, msg, u, startingpos, endingpos): if errors == 'replace': - return u'?', None, endingpos + return '?', None, endingpos if errors == 'ignore': - return u'', None, endingpos + return '', None, endingpos raise UnicodeEncodeError(encoding, u, startingpos, endingpos, msg) # ____________________________________________________________ From pypy.commits at gmail.com Sat Oct 7 09:10:51 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:10:51 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: Small tweaks Message-ID: <59d8d25b.c7341c0a.2c97e.8874@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r5834:5de46ea94c8c Date: 2017-10-05 22:28 +0200 http://bitbucket.org/pypy/extradoc/changeset/5de46ea94c8c/ Log: Small tweaks diff --git a/talk/pyconza2017/pypy-whats-new/slides.pdf b/talk/pyconza2017/pypy-whats-new/slides.pdf index 7fd5fdeae3fe34927641ade3c151867aa43c75f7..ed2a435d5ce44d13c41f9f642829c2b8ff7fc3d3 GIT binary patch [cut] diff --git a/talk/pyconza2017/pypy-whats-new/slides.rst b/talk/pyconza2017/pypy-whats-new/slides.rst --- a/talk/pyconza2017/pypy-whats-new/slides.rst +++ b/talk/pyconza2017/pypy-whats-new/slides.rst @@ -180,9 +180,9 @@ * Another possibility: a GIL-free but non-STM PyPy -* Large amount of work, not looking forward to do it +* Large amount of work, large maintenance burden -* Would require serious funding first +* Lots of hype, but we would need serious commercial funding and backing From pypy.commits at gmail.com Sat Oct 7 09:10:54 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:10:54 -0700 (PDT) Subject: [pypy-commit] pypy default: Add direct tests for latin1 siphash computation Message-ID: <59d8d25e.c5061c0a.a63cd.a4b0@mx.google.com> Author: Armin Rigo Branch: Changeset: r92617:4a184c225a22 Date: 2017-10-06 02:43 +0200 http://bitbucket.org/pypy/pypy/changeset/4a184c225a22/ Log: Add direct tests for latin1 siphash computation diff --git a/rpython/rlib/test/test_rsiphash.py b/rpython/rlib/test/test_rsiphash.py --- a/rpython/rlib/test/test_rsiphash.py +++ b/rpython/rlib/test/test_rsiphash.py @@ -1,8 +1,10 @@ import os from rpython.rlib.rsiphash import siphash24, _siphash24, choosen_seed from rpython.rlib.rsiphash import initialize_from_env, enable_siphash24 +from rpython.rlib.rsiphash import ll_hash_string_siphash24 from rpython.rlib.objectmodel import compute_hash from rpython.rlib.rarithmetic import intmask +from rpython.rtyper.annlowlevel import llstr, llunicode from rpython.rtyper.lltypesystem import llmemory, rffi from rpython.translator.c.test.test_genc import compile @@ -38,14 +40,26 @@ test_misaligned_path=True): x = siphash24(s) y = _siphash24(llmemory.cast_ptr_to_adr(rffi.ptradd(q, 1)), len(s)) + z = ll_hash_string_siphash24(llstr(s)) rffi.free_charp(q) assert x == y + assert z == intmask(x) return x def test_siphash24(): for expected, string in CASES: assert check(string) == expected +def check_latin1(s, expected): + with choosen_seed(0x8a9f065a358479f4, 0x11cb1e9ee7f40e1f, + test_misaligned_path=True): + z = ll_hash_string_siphash24(llunicode(s)) + assert z == intmask(expected) + +def test_siphash24_latin1_unicode(): + for expected, string in CASES: + check_latin1(string.decode('latin1'), expected) + def test_fix_seed(): old_val = os.environ.get('PYTHONHASHSEED', None) try: From pypy.commits at gmail.com Sat Oct 7 09:10:56 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:10:56 -0700 (PDT) Subject: [pypy-commit] pypy default: Optimize the case of siphash computing the hash of a latin1 unicode string Message-ID: <59d8d260.c5a4df0a.6609c.27c2@mx.google.com> Author: Armin Rigo Branch: Changeset: r92618:120728f3d439 Date: 2017-10-06 02:58 +0200 http://bitbucket.org/pypy/pypy/changeset/120728f3d439/ Log: Optimize the case of siphash computing the hash of a latin1 unicode string diff --git a/rpython/rlib/rsiphash.py b/rpython/rlib/rsiphash.py --- a/rpython/rlib/rsiphash.py +++ b/rpython/rlib/rsiphash.py @@ -11,6 +11,7 @@ from rpython.rlib.objectmodel import not_rpython, always_inline from rpython.rlib.objectmodel import we_are_translated, dont_inline from rpython.rlib.objectmodel import keepalive_until_here +from rpython.rlib.objectmodel import specialize from rpython.rlib import rgc, jit, rposix from rpython.rlib.rarithmetic import r_uint64, r_uint32, r_uint from rpython.rlib.rawstorage import misaligned_is_fine @@ -26,9 +27,11 @@ return x def _le32toh(x): return x + BIG_ENDIAN = False else: _le64toh = rarithmetic.byteswap _le32toh = rarithmetic.byteswap + BIG_ENDIAN = True def _decode64(s): return (r_uint64(ord(s[0])) | @@ -151,23 +154,20 @@ else: # NOTE: a latin-1 unicode string must have the same hash as the # corresponding byte string. If the unicode is all within - # 0-255, then we need to allocate a byte buffer and copy the - # latin-1 encoding in it manually. Note also that we give a + # 0-255, then we call _siphash24() with a special argument that + # will make it load only one byte for every unicode char. + # Note also that we give a # different hash result than CPython on ucs4 platforms, for # unicode strings where CPython uses 2 bytes per character. + addr = rstr._get_raw_buf_unicode(rstr.UNICODE, ll_s, 0) + SZ = rffi.sizeof(rstr.UNICODE.chars.OF) for i in range(length): if ord(ll_s.chars[i]) > 0xFF: - addr = rstr._get_raw_buf_unicode(rstr.UNICODE, ll_s, 0) - length *= rffi.sizeof(rstr.UNICODE.chars.OF) + length *= SZ break else: - p = lltype.malloc(rffi.CCHARP.TO, length, flavor='raw') - i = 0 - while i < length: - p[i] = chr(ord(ll_s.chars[i])) - i += 1 - x = _siphash24(llmemory.cast_ptr_to_adr(p), length) - lltype.free(p, flavor='raw') + x = _siphash24(addr, length, SZ) + keepalive_until_here(ll_s) return intmask(x) x = _siphash24(addr, length) keepalive_until_here(ll_s) @@ -215,7 +215,8 @@ @rgc.no_collect -def _siphash24(addr_in, size): + at specialize.arg(2) +def _siphash24(addr_in, size, SZ=1): """Takes an address pointer and a size. Returns the hash as a r_uint64, which can then be casted to the expected type.""" @@ -227,10 +228,14 @@ v2 = k0 ^ magic2 v3 = k1 ^ magic3 - direct = (misaligned_is_fine or + direct = (SZ == 1) and (misaligned_is_fine or (rffi.cast(lltype.Signed, addr_in) & 7) == 0) - index = 0 + if BIG_ENDIAN: + index = SZ - 1 + else: + index = 0 if direct: + assert SZ == 1 while size >= 8: mi = llop.raw_load(rffi.ULONGLONG, addr_in, index) mi = _le64toh(mi) @@ -242,30 +247,30 @@ else: while size >= 8: mi = ( - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1)) << 8 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2)) << 16 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3)) << 24 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4)) << 32 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5)) << 40 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6)) << 48 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 7)) << 56 + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1*SZ)) << 8 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2*SZ)) << 16 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3*SZ)) << 24 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4*SZ)) << 32 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5*SZ)) << 40 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6*SZ)) << 48 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 7*SZ)) << 56 ) size -= 8 - index += 8 + index += 8*SZ v3 ^= mi v0, v1, v2, v3 = _double_round(v0, v1, v2, v3) v0 ^= mi t = r_uint64(0) if size == 7: - t = r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6)) << 48 + t = r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6*SZ)) << 48 size = 6 if size == 6: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5)) << 40 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5*SZ)) << 40 size = 5 if size == 5: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4)) << 32 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4*SZ)) << 32 size = 4 if size == 4: if direct: @@ -273,13 +278,13 @@ t |= r_uint64(v) size = 0 else: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3)) << 24 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3*SZ))<< 24 size = 3 if size == 3: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2)) << 16 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2*SZ)) << 16 size = 2 if size == 2: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1)) << 8 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1*SZ)) << 8 size = 1 if size == 1: t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) From pypy.commits at gmail.com Sat Oct 7 09:10:59 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:10:59 -0700 (PDT) Subject: [pypy-commit] pypy default: Cache the siphash24 of 0 and 1 bytes strings Message-ID: <59d8d263.908a1c0a.e4026.6b2d@mx.google.com> Author: Armin Rigo Branch: Changeset: r92619:c10873611793 Date: 2017-10-06 04:21 +0200 http://bitbucket.org/pypy/pypy/changeset/c10873611793/ Log: Cache the siphash24 of 0 and 1 bytes strings diff --git a/rpython/rlib/rsiphash.py b/rpython/rlib/rsiphash.py --- a/rpython/rlib/rsiphash.py +++ b/rpython/rlib/rsiphash.py @@ -52,6 +52,7 @@ """'s' is a string of length 16""" seed.k0l = _decode64(s) seed.k1l = _decode64(s[8:16]) + _update_prebuilt_hashes() random_ctx = rurandom.init_urandom() @@ -175,16 +176,22 @@ @contextmanager -def choosen_seed(new_k0, new_k1, test_misaligned_path=False): +def choosen_seed(new_k0, new_k1, test_misaligned_path=False, + test_prebuilt=False): """For tests.""" - global misaligned_is_fine - old = seed.k0l, seed.k1l, misaligned_is_fine + global misaligned_is_fine, seed + old = seed, misaligned_is_fine + seed = Seed() seed.k0l = r_uint64(new_k0) seed.k1l = r_uint64(new_k1) + if test_prebuilt: + _update_prebuilt_hashes() + else: + seed.bound_prebuilt_size = 0 if test_misaligned_path: misaligned_is_fine = False yield - seed.k0l, seed.k1l, misaligned_is_fine = old + seed, misaligned_is_fine = old magic0 = r_uint64(0x736f6d6570736575) magic1 = r_uint64(0x646f72616e646f6d) @@ -220,6 +227,17 @@ """Takes an address pointer and a size. Returns the hash as a r_uint64, which can then be casted to the expected type.""" + if BIG_ENDIAN: + index = SZ - 1 + else: + index = 0 + if size < seed.bound_prebuilt_size: + if size <= 0: + return seed.hash_empty + else: + t = rarithmetic.intmask(llop.raw_load(rffi.UCHAR, addr_in, index)) + return seed.hash_single[t] + k0 = seed.k0l k1 = seed.k1l b = r_uint64(size) << 56 @@ -230,10 +248,6 @@ direct = (SZ == 1) and (misaligned_is_fine or (rffi.cast(lltype.Signed, addr_in) & 7) == 0) - if BIG_ENDIAN: - index = SZ - 1 - else: - index = 0 if direct: assert SZ == 1 while size >= 8: @@ -311,3 +325,17 @@ """ with rffi.scoped_nonmovingbuffer(s) as p: return _siphash24(llmemory.cast_ptr_to_adr(p), len(s)) + + +# Prebuilt hashes are precomputed here +def _update_prebuilt_hashes(): + seed.bound_prebuilt_size = 0 + with lltype.scoped_alloc(rffi.CCHARP.TO, 1) as p: + addr = llmemory.cast_ptr_to_adr(p) + seed.hash_single = [r_uint64(0)] * 256 + for i in range(256): + p[0] = chr(i) + seed.hash_single[i] = _siphash24(addr, 1) + seed.hash_empty = _siphash24(addr, 0) + seed.bound_prebuilt_size = 2 +_update_prebuilt_hashes() diff --git a/rpython/rlib/test/test_rsiphash.py b/rpython/rlib/test/test_rsiphash.py --- a/rpython/rlib/test/test_rsiphash.py +++ b/rpython/rlib/test/test_rsiphash.py @@ -50,9 +50,9 @@ for expected, string in CASES: assert check(string) == expected -def check_latin1(s, expected): +def check_latin1(s, expected, test_prebuilt=False): with choosen_seed(0x8a9f065a358479f4, 0x11cb1e9ee7f40e1f, - test_misaligned_path=True): + test_misaligned_path=True, test_prebuilt=test_prebuilt): z = ll_hash_string_siphash24(llunicode(s)) assert z == intmask(expected) @@ -60,6 +60,10 @@ for expected, string in CASES: check_latin1(string.decode('latin1'), expected) +def test_siphash24_latin1_unicode_prebuilt(): + for expected, string in CASES: + check_latin1(string.decode('latin1'), expected, test_prebuilt=True) + def test_fix_seed(): old_val = os.environ.get('PYTHONHASHSEED', None) try: From pypy.commits at gmail.com Sat Oct 7 09:11:03 2017 From: pypy.commits at gmail.com (wlav) Date: Sat, 07 Oct 2017 06:11:03 -0700 (PDT) Subject: [pypy-commit] pypy default: fix library name lookup for presence test Message-ID: <59d8d267.84881c0a.fb681.c4d1@mx.google.com> Author: Wim Lavrijsen Branch: Changeset: r92620:692fe5cc3bb7 Date: 2017-10-05 20:30 -0700 http://bitbucket.org/pypy/pypy/changeset/692fe5cc3bb7/ Log: fix library name lookup for presence test diff --git a/pypy/module/_cppyy/test/conftest.py b/pypy/module/_cppyy/test/conftest.py --- a/pypy/module/_cppyy/test/conftest.py +++ b/pypy/module/_cppyy/test/conftest.py @@ -4,7 +4,7 @@ def pytest_runtest_setup(item): if py.path.local.sysfind('genreflex') is None: import pypy.module._cppyy.capi.loadable_capi as lcapi - if 'dummy' in lcapi.reflection_library: + if 'dummy' in lcapi.backend_library: # run only tests that are covered by the dummy backend and tests # that do not rely on reflex import os @@ -33,7 +33,7 @@ import pypy.module._cppyy.capi.loadable_capi as lcapi try: import ctypes - ctypes.CDLL(lcapi.reflection_library) + ctypes.CDLL(lcapi.backend_library) except Exception as e: if config.option.runappdirect: return # "can't run dummy tests in -A" @@ -71,4 +71,4 @@ return raise - lcapi.reflection_library = str(soname) + lcapi.backend_library = str(soname) From pypy.commits at gmail.com Sat Oct 7 09:11:07 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:11:07 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: hg merge default Message-ID: <59d8d26b.1986df0a.ede72.a83f@mx.google.com> Author: Armin Rigo Branch: py3.5 Changeset: r92621:e1232edfd557 Date: 2017-10-06 04:26 +0200 http://bitbucket.org/pypy/pypy/changeset/e1232edfd557/ Log: hg merge default diff too long, truncating to 2000 out of 8213 lines diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -3,13 +3,6 @@ =========================== .. this is a revision shortly after release-pypy2.7-v5.9.0 -.. startrev:899e5245de1e +.. startrev:d56dadcef996 -.. branch: cpyext-jit -Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: -this allows to write specialized code which is much faster than previous -completely generic version. Moreover, let the JIT to look inside the cpyext -module: the net result is that cpyext calls are up to 7x faster. However, this -is true only for very simple situations: in all real life code, we are still -much slower than CPython (more optimizations to come) diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst --- a/pypy/doc/whatsnew-pypy2-5.9.0.rst +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -85,3 +85,12 @@ .. branch: py_ssize_t Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'math/mathcore/src/Fitter.cxx' -if os.path.exists(inp): - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp): - if '#include "TF1.h"' in line: - continue - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - -# done -os.chdir(os.path.pardir) - -# debugging: run a test build -if DEBUG_TESTBUILD: - print('running a debug test build') - tb = "test_builddir" - if os.path.exists(tb): - shutil.rmtree(tb) - os.mkdir(tb) - os.chdir(tb) - os.system('cmake ../%s -DCMAKE_INSTALL_PREFIX=../install -Dminimal=ON -Dasimage=OFF' % pkgdir) - os.system('make -j 32') - - -# -## package creation -# -countdown = 0 -pidir = 'Package-'+args.release -print('creating package', pidir) -if not os.path.exists(pidir): - os.mkdir(pidir) -os.chdir(pidir); countdown += 1 - -print('creating LICENSE.txt') -with open('LICENSE.txt', 'w') as outp: - outp.write("""There are three main parts: - - LLVM: distributed under University of Illinois/NCSA Open Source License - https://opensource.org/licenses/UoI-NCSA.php - ROOT: distributed under LGPL 2.1 - https://root.cern.ch/license - Cppyy: distributed under LBNL BSD - https://fedoraproject.org/wiki/Licensing/LBNLBSD -""") - -print('creating MANIFEST.in') -with open('MANIFEST.in', 'w') as outp: - outp.write("""# Include the license file -include LICENSE.txt - -# Include the data files -recursive-include src * -""") - -print('creating README.rst') -with open('README.rst', 'w') as outp: - outp.write("""PyPy cling-support -================== - ----- - -Find the documentation here: - http://doc.pypy.org/en/latest/cppyy.html -""") - -print('creating setup.cfg') -with open('setup.cfg', 'w') as outp: - outp.write("""[bdist_wheel] -universal=0 -""") - -print('creating setup.py') -with open('setup.py', 'w') as outp: - outp.write("""import os, sys, subprocess -from setuptools import setup, find_packages -from distutils import log -from distutils.command.build import build as _build -from setuptools.command.install import install as _install -from distutils.sysconfig import get_python_lib -from distutils.errors import DistutilsSetupError -from codecs import open - -here = os.path.abspath(os.path.dirname(__file__)) -with open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: - long_description = f.read() - -builddir = None -def get_builddir(): - global builddir - if builddir is None: - topdir = os.getcwd() - builddir = os.path.join(topdir, 'builddir') - return builddir - -srcdir = None -def get_srcdir(): - global srcdir - if srcdir is None: - topdir = os.getcwd() - srcdir = os.path.join(topdir, 'src', 'backend') - return srcdir - -class my_cmake_build(_build): - def __init__(self, dist, *args, **kwargs): - _build.__init__(self, dist, *args, **kwargs) - # TODO: can't seem to find a better way of getting hold of - # the install_lib parameter during the build phase ... - prefix = '' - try: - prefix = dist.get_command_obj('install').install_lib - except AttributeError: - pass - if not prefix: - prefix = get_python_lib(1, 0) - self.prefix = os.path.join(prefix, 'cppyy_backend') - - def run(self): - # base run - _build.run(self) - - # custom run - log.info('Now building libcppyy_backend.so and dependencies') - builddir = get_builddir() - srcdir = get_srcdir() - if not os.path.exists(builddir): - log.info('Creating build directory %s ...' % builddir) - os.makedirs(builddir) - - os.chdir(builddir) - log.info('Running cmake for cppyy_backend') - if subprocess.call([ - 'cmake', srcdir, '-Dminimal=ON -Dasimage=OFF', - '-DCMAKE_INSTALL_PREFIX='+self.prefix]) != 0: - raise DistutilsSetupError('Failed to configure cppyy_backend') - - nprocs = os.getenv("MAKE_NPROCS") - if nprocs: - try: - ival = int(nprocs) - nprocs = '-j'+nprocs - except ValueError: - log.warn("Integer expected for MAKE_NPROCS, but got %s (ignored)", nprocs) - nprocs = '-j1' - else: - nprocs = '-j1' - log.info('Now building cppyy_backend and dependencies ...') - if subprocess.call(['make', nprocs]) != 0: - raise DistutilsSetupError('Failed to build cppyy_backend') - - log.info('build finished') - -class my_libs_install(_install): - def run(self): - # base install - _install.run(self) - - # custom install - log.info('Now installing libcppyy_backend.so and dependencies') - builddir = get_builddir() - if not os.path.exists(builddir): - raise DistutilsSetupError('Failed to find build dir!') - os.chdir(builddir) - - prefix = self.install_lib - log.info('Now installing in %s ...', prefix) - if subprocess.call(['make', 'install']) != 0: - raise DistutilsSetupError('Failed to install cppyy_backend') - - log.info('install finished') - - def get_outputs(self): - outputs = _install.get_outputs(self) - outputs.append(os.path.join(self.install_lib, 'cppyy_backend')) - return outputs - -setup( - name='PyPy-cppyy-backend', -""") - outp.write(" version='%s', # corresponds to ROOT %s, extra number is for packager\n"\ - % (args.version, args.release)) - outp.write(""" description='Cling support for PyPy', - long_description=long_description, - - url='http://pypy.org', - - # Author details - author='PyPy Developers', - author_email='pypy-dev at python.org', - - license='LLVM: UoI-NCSA; ROOT: LGPL 2.1; Cppyy: LBNL BSD', - - classifiers=[ - 'Development Status :: 4 - Beta', - - 'Intended Audience :: Developers', - - 'Topic :: Software Development', - 'Topic :: Software Development :: Interpreters', - - #'License :: OSI Approved :: MIT License', - - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: Implementation :: PyPy', - 'Programming Language :: C', - 'Programming Language :: C++', - - 'Natural Language :: English' - ], - - keywords='interpreter development', - - packages=find_packages('src', ['backend']), - include_package_data=True, - - extras_require={ - }, - - cmdclass = { - 'build': my_cmake_build, - 'install': my_libs_install, - }, -) -""") - - -print('creating src ... ROOT part') -if not os.path.exists('src'): - os.mkdir('src') -os.chdir('src'); countdown += 1 -if not os.path.exists('backend'): - src = os.path.join(os.path.pardir, os.path.pardir, pkgdir) - print('now copying', src) - shutil.copytree(src, 'backend') - -print('creating src ... cppyy part') -os.chdir('backend'); countdown += 1 -if not os.path.exists('cppyy'): - os.mkdir('cppyy') - os.chdir('cppyy'); countdown += 1 - - with open('CMakeLists.txt', 'w') as outp: - outp.write("""############################################################################ -# CMakeLists.txt file for building cppyy package -############################################################################ - -ROOT_GLOB_SOURCES(sources ${CMAKE_CURRENT_SOURCE_DIR}/src/*.cxx) -set_source_files_properties(${sources} COMPILE_FLAGS "-fomit-frame-pointer -fvisibility=hidden -DRPY_EXTERN=RPY_EXPORTED -DRPYTHON_LL2CTYPES") - -add_definitions(${CLING_CXXFLAGS}) - -ROOT_LINKER_LIBRARY(cppyy_backend ${sources} - LIBRARIES ${CMAKE_DL_LIBS} - DEPENDENCIES Core Cling RIO Thread) - -add_dependencies(cppyy_backend CLING) -""") - - os.mkdir('src') - os.chdir('src'); countdown += 1 - print('pulling cppyy/clingcwrapper.cxx from pypy') - base = 'https://bitbucket.org/pypy/pypy/raw/default/pypy/module/cppyy/' - for cppyy_file in ['src/callcontext.h', 'include/capi.h', 'src/clingcwrapper.cxx', - 'include/clingcwrapper.h', 'include/cpp_cppyy.h', 'include/cppyy.h']: - resp = urllib2.urlopen(base+cppyy_file) - with open(os.path.basename(cppyy_file), 'w') as outp: - outp.write(resp.read()) - - # fix include - inp = 'capi.h' - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if 'src/precommondefs.h' in line: - line = '#include "precommondefs.h"\n' - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - - with open('precommondefs.h', 'w') as outp: - outp.write("""/***** Start of precommondefs.h *****/ - -/* This is extracted from pyconfig.h from CPython. It sets the macros - that affect the features we get from system include files. - It must not #include anything. */ - -#ifndef __PYPY_PRECOMMONDEFS_H -#define __PYPY_PRECOMMONDEFS_H - - -/* Define on Darwin to activate all library features */ -#define _DARWIN_C_SOURCE 1 -/* This must be set to 64 on some systems to enable large file support. */ -#define _FILE_OFFSET_BITS 64 -/* Define on Linux to activate all library features */ -#define _GNU_SOURCE 1 -/* This must be defined on some systems to enable large file support. */ -#define _LARGEFILE_SOURCE 1 -/* Define on NetBSD to activate all library features */ -#define _NETBSD_SOURCE 1 -/* Define to activate features from IEEE Stds 1003.1-2001 */ -#ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L -#endif -/* Define on FreeBSD to activate all library features */ -#define __BSD_VISIBLE 1 -#define __XSI_VISIBLE 700 -/* Windows: winsock/winsock2 mess */ -#define WIN32_LEAN_AND_MEAN -#ifdef _WIN64 - typedef __int64 Signed; - typedef unsigned __int64 Unsigned; -# define SIGNED_MIN LLONG_MIN -#else - typedef long Signed; - typedef unsigned long Unsigned; -# define SIGNED_MIN LONG_MIN -#endif - -#if !defined(RPY_ASSERT) && !defined(RPY_LL_ASSERT) && !defined(NDEBUG) -# define NDEBUG -#endif - - -/* All functions and global variables declared anywhere should use - one of the following attributes: - - RPY_EXPORTED: the symbol is exported out of libpypy-c.so. - - RPY_EXTERN: the symbol is not exported out of libpypy-c.so, but - otherwise works like 'extern' by being available to - other C sources. - - static: as usual, this means the symbol is local to this C file. - - Don't use _RPY_HIDDEN directly. For tests involving building a custom - .so, translator/tool/cbuild.py overrides RPY_EXTERN so that it becomes - equal to RPY_EXPORTED. - - Any function or global variable declared with no attribute at all is - a bug; please report or fix it. -*/ -#ifdef __GNUC__ -# define RPY_EXPORTED extern __attribute__((visibility("default"))) -# define _RPY_HIDDEN __attribute__((visibility("hidden"))) -#else -# define RPY_EXPORTED extern __declspec(dllexport) -# define _RPY_HIDDEN /* nothing */ -#endif -#ifndef RPY_EXTERN -# define RPY_EXTERN extern _RPY_HIDDEN -#endif - - -#endif /* __PYPY_PRECOMMONDEFS_H */ - -/***** End of precommondefs.h *****/ -""") - -# back up to pip package top -for i in range(countdown-1): - os.chdir(os.path.pardir) - -# add cppyy module to cmake -os.chdir('src/backend') -inp = 'CMakeLists.txt' -print('adding cppyy to cmake') -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'add_subdirectory' in line and 'net' in line: - line += 'add_subdirectory (cppyy)\n' - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# done! diff --git a/pypy/module/_cppyy/bench/Makefile b/pypy/module/_cppyy/bench/Makefile deleted file mode 100644 --- a/pypy/module/_cppyy/bench/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -all: bench02Dict_reflex.so - -ROOTSYS := ${ROOTSYS} - -ifeq ($(ROOTSYS),) - genreflex=genreflex - cppflags= -else - genreflex=$(ROOTSYS)/bin/genreflex - cppflags=-I$(ROOTSYS)/include -L$(ROOTSYS)/lib -endif - -PLATFORM := $(shell uname -s) -ifeq ($(PLATFORM),Darwin) - cppflags+=-dynamiclib -single_module -arch x86_64 -endif - -ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),) - genreflexflags= - cppflags2=-O3 -fPIC -else - genreflexflags=--with-methptrgetter - cppflags2=-Wno-pmf-conversions -O3 -fPIC -endif - - -bench02Dict_reflex.so: bench02.h bench02.cxx bench02.xml - $(genreflex) bench02.h $(genreflexflags) --selection=bench02.xml -I$(ROOTSYS)/include - g++ -o $@ bench02.cxx bench02_rflx.cpp -I$(ROOTSYS)/include -shared -std=c++11 -lHistPainter `root-config --libs` $(cppflags) $(cppflags2) diff --git a/pypy/module/_cppyy/bench/bench02.cxx b/pypy/module/_cppyy/bench/bench02.cxx deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.cxx +++ /dev/null @@ -1,79 +0,0 @@ -#include "bench02.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TDirectory.h" -#include "TInterpreter.h" -#include "TSystem.h" -#include "TBenchmark.h" -#include "TStyle.h" -#include "TError.h" -#include "Getline.h" -#include "TVirtualX.h" - -#include "Api.h" - -#include - -TClass *TClass::GetClass(const char*, Bool_t, Bool_t) { - static TClass* dummy = new TClass("__dummy__", kTRUE); - return dummy; // is deleted by gROOT at shutdown -} - -class TTestApplication : public TApplication { -public: - TTestApplication( - const char* acn, Int_t* argc, char** argv, Bool_t bLoadLibs = kTRUE); - virtual ~TTestApplication(); -}; - -TTestApplication::TTestApplication( - const char* acn, int* argc, char** argv, bool do_load) : TApplication(acn, argc, argv) { - if (do_load) { - // follow TRint to minimize differences with CINT - ProcessLine("#include ", kTRUE); - ProcessLine("#include <_string>", kTRUE); // for std::string iostream. - ProcessLine("#include ", kTRUE); // needed because they're used within the - ProcessLine("#include ", kTRUE); // core ROOT dicts and CINT won't be able - // to properly unload these files - } - - // save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // prevent crashes on accessing history - Gl_histinit((char*)"-"); - - // prevent ROOT from exiting python - SetReturnFromRun(kTRUE); -} - -TTestApplication::~TTestApplication() {} - -static const char* appname = "pypy-cppyy"; - -Bench02RootApp::Bench02RootApp() { - gROOT->SetBatch(kTRUE); - if (!gApplication) { - int argc = 1; - char* argv[1]; argv[0] = (char*)appname; - gApplication = new TTestApplication(appname, &argc, argv, kFALSE); - } -} - -Bench02RootApp::~Bench02RootApp() { - // TODO: ROOT globals cleanup ... (?) -} - -void Bench02RootApp::report() { - std::cout << "gROOT is: " << gROOT << std::endl; - std::cout << "gApplication is: " << gApplication << std::endl; -} - -void Bench02RootApp::close_file(TFile* f) { - std::cout << "closing file " << f->GetName() << " ... " << std::endl; - f->Write(); - f->Close(); - std::cout << "... file closed" << std::endl; -} diff --git a/pypy/module/_cppyy/bench/bench02.h b/pypy/module/_cppyy/bench/bench02.h deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.h +++ /dev/null @@ -1,72 +0,0 @@ -#include "TString.h" - -#include "TCanvas.h" -#include "TFile.h" -#include "TProfile.h" -#include "TNtuple.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TRandom.h" -#include "TRandom3.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TSystem.h" - -#include "TArchiveFile.h" -#include "TBasket.h" -#include "TBenchmark.h" -#include "TBox.h" -#include "TBranchRef.h" -#include "TBrowser.h" -#include "TClassGenerator.h" -#include "TClassRef.h" -#include "TClassStreamer.h" -#include "TContextMenu.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TF1.h" -#include "TFileCacheRead.h" -#include "TFileCacheWrite.h" -#include "TFileMergeInfo.h" -#include "TFitResult.h" -#include "TFolder.h" -//#include "TFormulaPrimitive.h" -#include "TFunction.h" -#include "TFrame.h" -#include "TGlobal.h" -#include "THashList.h" -#include "TInetAddress.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TLegend.h" -#include "TMethodCall.h" -#include "TPluginManager.h" -#include "TProcessUUID.h" -#include "TSchemaRuleSet.h" -#include "TStyle.h" -#include "TSysEvtHandler.h" -#include "TTimer.h" -#include "TView.h" -//#include "TVirtualCollectionProxy.h" -#include "TVirtualFFT.h" -#include "TVirtualHistPainter.h" -#include "TVirtualIndex.h" -#include "TVirtualIsAProxy.h" -#include "TVirtualPadPainter.h" -#include "TVirtualRefProxy.h" -#include "TVirtualStreamerInfo.h" -#include "TVirtualViewer3D.h" - -#include -#include - - -class Bench02RootApp { -public: - Bench02RootApp(); - ~Bench02RootApp(); - - void report(); - void close_file(TFile* f); -}; diff --git a/pypy/module/_cppyy/bench/bench02.xml b/pypy/module/_cppyy/bench/bench02.xml deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pypy/module/_cppyy/bench/hsimple.C b/pypy/module/_cppyy/bench/hsimple.C deleted file mode 100644 --- a/pypy/module/_cppyy/bench/hsimple.C +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -TFile *hsimple(Int_t get=0) -{ -// This program creates : -// - a one dimensional histogram -// - a two dimensional histogram -// - a profile histogram -// - a memory-resident ntuple -// -// These objects are filled with some random numbers and saved on a file. -// If get=1 the macro returns a pointer to the TFile of "hsimple.root" -// if this file exists, otherwise it is created. -// The file "hsimple.root" is created in $ROOTSYS/tutorials if the caller has -// write access to this directory, otherwise the file is created in $PWD - - TString filename = "hsimple.root"; - TString dir = gSystem->UnixPathName(gInterpreter->GetCurrentMacroName()); - dir.ReplaceAll("hsimple.C",""); - dir.ReplaceAll("/./","/"); - TFile *hfile = 0; - if (get) { - // if the argument get =1 return the file "hsimple.root" - // if the file does not exist, it is created - TString fullPath = dir+"hsimple.root"; - if (!gSystem->AccessPathName(fullPath,kFileExists)) { - hfile = TFile::Open(fullPath); //in $ROOTSYS/tutorials - if (hfile) return hfile; - } - //otherwise try $PWD/hsimple.root - if (!gSystem->AccessPathName("hsimple.root",kFileExists)) { - hfile = TFile::Open("hsimple.root"); //in current dir - if (hfile) return hfile; - } - } - //no hsimple.root file found. Must generate it ! - //generate hsimple.root in $ROOTSYS/tutorials if we have write access - if (!gSystem->AccessPathName(dir,kWritePermission)) { - filename = dir+"hsimple.root"; - } else if (!gSystem->AccessPathName(".",kWritePermission)) { - //otherwise generate hsimple.root in the current directory - } else { - printf("you must run the script in a directory with write access\n"); - return 0; - } - hfile = (TFile*)gROOT->FindObject(filename); if (hfile) hfile->Close(); - hfile = new TFile(filename,"RECREATE","Demo ROOT file with histograms"); - - // Create some histograms, a profile histogram and an ntuple - TH1F *hpx = new TH1F("hpx","This is the px distribution",100,-4,4); - hpx->SetFillColor(48); - TH2F *hpxpy = new TH2F("hpxpy","py vs px",40,-4,4,40,-4,4); - TProfile *hprof = new TProfile("hprof","Profile of pz versus px",100,-4,4,0,20); - TNtuple *ntuple = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - - gBenchmark->Start("hsimple"); - - // Create a new canvas. - TCanvas *c1 = new TCanvas("c1","Dynamic Filling Example",200,10,700,500); - c1->SetFillColor(42); - c1->GetFrame()->SetFillColor(21); - c1->GetFrame()->SetBorderSize(6); - c1->GetFrame()->SetBorderMode(-1); - - - // Fill histograms randomly - TRandom3 random; - Float_t px, py, pz; - const Int_t kUPDATE = 1000; - for (Int_t i = 0; i < 50000; i++) { - // random.Rannor(px,py); - px = random.Gaus(0, 1); - py = random.Gaus(0, 1); - pz = px*px + py*py; - Float_t rnd = random.Rndm(1); - hpx->Fill(px); - hpxpy->Fill(px,py); - hprof->Fill(px,pz); - ntuple->Fill(px,py,pz,rnd,i); - if (i && (i%kUPDATE) == 0) { - if (i == kUPDATE) hpx->Draw(); - c1->Modified(); - c1->Update(); - if (gSystem->ProcessEvents()) - break; - } - } - gBenchmark->Show("hsimple"); - - // Save all objects in this file - hpx->SetFillColor(0); - hfile->Write(); - hpx->SetFillColor(48); - c1->Modified(); - return hfile; - -// Note that the file is automatically close when application terminates -// or when the file destructor is called. -} diff --git a/pypy/module/_cppyy/bench/hsimple.py b/pypy/module/_cppyy/bench/hsimple.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple.py +++ /dev/null @@ -1,110 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -_reflex = True # to keep things equal, set to False for full macro - -try: - import cppyy, random - - if not hasattr(cppyy.gbl, 'gROOT'): - cppyy.load_reflection_info('bench02Dict_reflex.so') - _reflex = True - - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom3 = cppyy.gbl.TRandom3 - - gROOT = cppyy.gbl.gROOT - gBenchmark = cppyy.gbl.TBenchmark() - gSystem = cppyy.gbl.gSystem - -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom3 - from ROOT import gROOT, gBenchmark, gSystem - import random - -if _reflex: - gROOT.SetBatch(True) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -if not _reflex: - hfile = gROOT.FindObject('hsimple.root') - if hfile: - hfile.Close() - hfile = TFile('hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.SetFillColor(48) -hpxpy = TH2F('hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4) -hprof = TProfile('hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20) -if not _reflex: - ntuple = TNtuple('ntuple', 'Demo ntuple', 'px:py:pz:random:i') - -gBenchmark.Start('hsimple') - -# Create a new canvas, and customize it. -c1 = TCanvas('c1', 'Dynamic Filling Example', 200, 10, 700, 500) -c1.SetFillColor(42) -c1.GetFrame().SetFillColor(21) -c1.GetFrame().SetBorderSize(6) -c1.GetFrame().SetBorderMode(-1) - -# Fill histograms randomly. -random = TRandom3() -kUPDATE = 1000 -for i in xrange(50000): - # Generate random numbers -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) - pz = px*px + py*py -# rnd = random.random() - rnd = random.Rndm(1) - - # Fill histograms - hpx.Fill(px) - hpxpy.Fill(px, py) - hprof.Fill(px, pz) - if not _reflex: - ntuple.Fill(px, py, pz, rnd, i) - - # Update display every kUPDATE events - if i and i%kUPDATE == 0: - if i == kUPDATE: - hpx.Draw() - - c1.Modified(True) - c1.Update() - - if gSystem.ProcessEvents(): # allow user interrupt - break - -gBenchmark.Show( 'hsimple' ) - -# Save all objects in this file -hpx.SetFillColor(0) -if not _reflex: - hfile.Write() -hpx.SetFillColor(48) -c1.Modified(True) -c1.Update() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/bench/hsimple_rflx.py b/pypy/module/_cppyy/bench/hsimple_rflx.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple_rflx.py +++ /dev/null @@ -1,120 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -try: - import warnings - warnings.simplefilter("ignore") - - import cppyy, random - cppyy.load_reflection_info('bench02Dict_reflex.so') - - app = cppyy.gbl.Bench02RootApp() - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom = cppyy.gbl.TRandom -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom - import random - -import math - -#gROOT = cppyy.gbl.gROOT -#gBenchmark = cppyy.gbl.gBenchmark -#gRandom = cppyy.gbl.gRandom -#gSystem = cppyy.gbl.gSystem - -#gROOT.Reset() - -# Create a new canvas, and customize it. -#c1 = TCanvas( 'c1', 'Dynamic Filling Example', 200, 10, 700, 500 ) -#c1.SetFillColor( 42 ) -#c1.GetFrame().SetFillColor( 21 ) -#c1.GetFrame().SetBorderSize( 6 ) -#c1.GetFrame().SetBorderMode( -1 ) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -#hfile = gROOT.FindObject( 'hsimple.root' ) -#if hfile: -# hfile.Close() -#hfile = TFile( 'hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.Print() -#hpxpy = TH2F( 'hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4 ) -#hprof = TProfile( 'hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20 ) -#ntuple = TNtuple( 'ntuple', 'Demo ntuple', 'px:py:pz:random:i' ) - -# Set canvas/frame attributes. -#hpx.SetFillColor( 48 ) - -#gBenchmark.Start( 'hsimple' ) - -# Initialize random number generator. -#gRandom.SetSeed() -#rannor, rndm = gRandom.Rannor, gRandom.Rndm - -random = TRandom() -random.SetSeed(0) - -# Fill histograms randomly. -#px, py = Double(), Double() -kUPDATE = 1000 -for i in xrange(2500000): - # Generate random values. -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) -# pt = (px*px + py*py)**0.5 - pt = math.sqrt(px*px + py*py) -# pt = (px*px + py*py) -# random = rndm(1) - - # Fill histograms. - hpx.Fill(pt) -# hpxpyFill( px, py ) -# hprofFill( px, pz ) -# ntupleFill( px, py, pz, random, i ) - - # Update display every kUPDATE events. -# if i and i%kUPDATE == 0: -# if i == kUPDATE: -# hpx.Draw() - -# c1.Modified() -# c1.Update() - -# if gSystem.ProcessEvents(): # allow user interrupt -# break - -#gBenchmark.Show( 'hsimple' ) - -hpx.Print() - -# Save all objects in this file. -#hpx.SetFillColor( 0 ) -#hfile.Write() -#hfile.Close() -#hpx.SetFillColor( 48 ) -#c1.Modified() -#c1.Update() -#c1.Draw() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/capi/__init__.py b/pypy/module/_cppyy/capi/__init__.py --- a/pypy/module/_cppyy/capi/__init__.py +++ b/pypy/module/_cppyy/capi/__init__.py @@ -1,16 +1,6 @@ from rpython.rtyper.lltypesystem import rffi, lltype -# There are two possible ways of accessing the backend through the reflection -# C-API: built it into pypy-c, or load it dynamically. The latter is preferred -# (and is the default) for use with Reflex. B/c of some builtin pythonizations, -# the former is recommended (for now) with CINT. - -# Note: if builtin_capi is chosen, then inside builtin_capi.py, there is still -# the selection of the desired backend (default is Reflex). - -# choose C-API access method: from pypy.module._cppyy.capi.loadable_capi import * -#from pypy.module._cppyy.capi.builtin_capi import * from pypy.module._cppyy.capi.capi_types import C_OBJECT,\ C_NULL_TYPE, C_NULL_OBJECT diff --git a/pypy/module/_cppyy/capi/builtin_capi.py b/pypy/module/_cppyy/capi/builtin_capi.py deleted file mode 100644 --- a/pypy/module/_cppyy/capi/builtin_capi.py +++ /dev/null @@ -1,590 +0,0 @@ -from rpython.rtyper.lltypesystem import rffi, lltype -from rpython.rlib.rarithmetic import intmask -from rpython.rlib import jit - -import cling_capi as backend - -from pypy.module._cppyy.capi.capi_types import C_SCOPE, C_TYPE, C_OBJECT,\ - C_METHOD, C_INDEX, C_INDEX_ARRAY, WLAVC_INDEX, C_FUNC_PTR - -identify = backend.identify -pythonize = backend.pythonize -register_pythonizations = backend.register_pythonizations -std_string_name = backend.std_string_name - -ts_reflect = backend.ts_reflect -ts_call = backend.ts_call -ts_memory = backend.ts_memory -ts_helper = backend.ts_helper - -def verify_backend(space): - return True # by definition - -c_load_dictionary = backend.c_load_dictionary - -# name to opaque C++ scope representation ------------------------------------ -_c_num_scopes = rffi.llexternal( - "cppyy_num_scopes", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_scopes(space, cppscope): - return _c_num_scopes(cppscope.handle) -_c_scope_name = rffi.llexternal( - "cppyy_scope_name", - [C_SCOPE, rffi.INT], rffi.CCHARP, - compilation_info = backend.eci) -def c_scope_name(space, cppscope, iscope): - return charp2str_free(space, _c_scope_name(cppscope.handle, iscope)) - -_c_resolve_name = rffi.llexternal( - "cppyy_resolve_name", - [rffi.CCHARP], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_resolve_name(space, name): - return charp2str_free(space, _c_resolve_name(name)) -_c_get_scope_opaque = rffi.llexternal( - "cppyy_get_scope", - [rffi.CCHARP], C_SCOPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_scope_opaque(space, name): - return _c_get_scope_opaque(name) -_c_actual_class = rffi.llexternal( - "cppyy_actual_class", - [C_TYPE, C_OBJECT], C_TYPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_actual_class(space, cppclass, cppobj): - return _c_actual_class(cppclass.handle, cppobj) - -# memory management ---------------------------------------------------------- -_c_allocate = rffi.llexternal( - "cppyy_allocate", - [C_TYPE], C_OBJECT, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate(space, cppclass): - return _c_allocate(cppclass.handle) -_c_deallocate = rffi.llexternal( - "cppyy_deallocate", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate(space, cppclass, cppobject): - _c_deallocate(cppclass.handle, cppobject) -_c_destruct = rffi.llexternal( - "cppyy_destruct", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_destruct(space, cppclass, cppobject): - _c_destruct(cppclass.handle, cppobject) - -# method/function dispatching ------------------------------------------------ -_c_call_v = rffi.llexternal( - "cppyy_call_v", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_v(space, cppmethod, cppobject, nargs, args): - _c_call_v(cppmethod, cppobject, nargs, args) -_c_call_b = rffi.llexternal( - "cppyy_call_b", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.UCHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_b(space, cppmethod, cppobject, nargs, args): - return _c_call_b(cppmethod, cppobject, nargs, args) -_c_call_c = rffi.llexternal( - "cppyy_call_c", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.CHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_c(space, cppmethod, cppobject, nargs, args): - return _c_call_c(cppmethod, cppobject, nargs, args) -_c_call_h = rffi.llexternal( - "cppyy_call_h", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.SHORT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_h(space, cppmethod, cppobject, nargs, args): - return _c_call_h(cppmethod, cppobject, nargs, args) -_c_call_i = rffi.llexternal( - "cppyy_call_i", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.INT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_i(space, cppmethod, cppobject, nargs, args): - return _c_call_i(cppmethod, cppobject, nargs, args) -_c_call_l = rffi.llexternal( - "cppyy_call_l", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_l(space, cppmethod, cppobject, nargs, args): - return _c_call_l(cppmethod, cppobject, nargs, args) -_c_call_ll = rffi.llexternal( - "cppyy_call_ll", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGLONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ll(space, cppmethod, cppobject, nargs, args): - return _c_call_ll(cppmethod, cppobject, nargs, args) -_c_call_f = rffi.llexternal( - "cppyy_call_f", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.FLOAT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_f(space, cppmethod, cppobject, nargs, args): - return _c_call_f(cppmethod, cppobject, nargs, args) -_c_call_d = rffi.llexternal( - "cppyy_call_d", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.DOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_d(space, cppmethod, cppobject, nargs, args): - return _c_call_d(cppmethod, cppobject, nargs, args) -_c_call_ld = rffi.llexternal( - "cppyy_call_ld", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGDOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ld(space, cppmethod, cppobject, nargs, args): - return _c_call_ld(cppmethod, cppobject, nargs, args) - -_c_call_r = rffi.llexternal( - "cppyy_call_r", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.VOIDP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_r(space, cppmethod, cppobject, nargs, args): - return _c_call_r(cppmethod, cppobject, nargs, args) -_c_call_s = rffi.llexternal( - "cppyy_call_s", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, rffi.SIZE_TP], rffi.CCHARP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_s(space, cppmethod, cppobject, nargs, args): - length = lltype.malloc(rffi.SIZE_TP.TO, 1, flavor='raw') - try: - cstr = _c_call_s(cppmethod, cppobject, nargs, args, length) - cstr_len = intmask(length[0]) - finally: - lltype.free(length, flavor='raw') - return cstr, cstr_len - -_c_constructor = rffi.llexternal( - "cppyy_constructor", - [C_METHOD, C_TYPE, rffi.INT, rffi.VOIDP], C_OBJECT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_constructor(space, cppmethod, cppobject, nargs, args): - return _c_constructor(cppmethod, cppobject, nargs, args) -_c_call_o = rffi.llexternal( - "cppyy_call_o", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, C_TYPE], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_o(space, method, cppobj, nargs, args, cppclass): - return _c_call_o(method, cppobj, nargs, args, cppclass.handle) - -_c_get_function_address = rffi.llexternal( - "cppyy_get_function_address", - [C_SCOPE, C_INDEX], C_FUNC_PTR, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) -def c_get_function_address(space, cppscope, index): - return _c_get_function_address(cppscope.handle, index) - -# handling of function argument buffer --------------------------------------- -_c_allocate_function_args = rffi.llexternal( - "cppyy_allocate_function_args", - [rffi.SIZE_T], rffi.VOIDP, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate_function_args(space, size): - return _c_allocate_function_args(size) -_c_deallocate_function_args = rffi.llexternal( - "cppyy_deallocate_function_args", - [rffi.VOIDP], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate_function_args(space, args): - _c_deallocate_function_args(args) -_c_function_arg_sizeof = rffi.llexternal( - "cppyy_function_arg_sizeof", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_sizeof(space): - return _c_function_arg_sizeof() -_c_function_arg_typeoffset = rffi.llexternal( - "cppyy_function_arg_typeoffset", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_typeoffset(space): - return _c_function_arg_typeoffset() - -# scope reflection information ----------------------------------------------- -_c_is_namespace = rffi.llexternal( - "cppyy_is_namespace", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_namespace(space, scope): - return _c_is_namespace(scope) -_c_is_template = rffi.llexternal( - "cppyy_is_template", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_template(space, name): - return _c_is_template(name) -_c_is_abstract = rffi.llexternal( - "cppyy_is_abstract", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_abstract(space, cpptype): - return _c_is_abstract(cpptype) -_c_is_enum = rffi.llexternal( - "cppyy_is_enum", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_enum(space, name): - return _c_is_enum(name) - -# type/class reflection information ------------------------------------------ -_c_final_name = rffi.llexternal( - "cppyy_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_final_name(space, cpptype): - return charp2str_free(space, _c_final_name(cpptype)) -_c_scoped_final_name = rffi.llexternal( - "cppyy_scoped_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_scoped_final_name(space, cpptype): - return charp2str_free(space, _c_scoped_final_name(cpptype)) -_c_has_complex_hierarchy = rffi.llexternal( - "cppyy_has_complex_hierarchy", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_has_complex_hierarchy(space, cpptype): - return _c_has_complex_hierarchy(cpptype) -_c_num_bases = rffi.llexternal( - "cppyy_num_bases", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_bases(space, cppclass): - return _c_num_bases(cppclass.handle) -_c_base_name = rffi.llexternal( - "cppyy_base_name", - [C_TYPE, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_base_name(space, cppclass, base_index): - return charp2str_free(space, _c_base_name(cppclass.handle, base_index)) -_c_is_subtype = rffi.llexternal( - "cppyy_is_subtype", - [C_TYPE, C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_is_subtype(space, derived, base): - if derived == base: - return 1 - return _c_is_subtype(derived.handle, base.handle) - -_c_base_offset = rffi.llexternal( - "cppyy_base_offset", - [C_TYPE, C_TYPE, C_OBJECT, rffi.INT], rffi.LONG, # actually ptrdiff_t - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_base_offset(space, derived, base, address, direction): - if derived == base: - return 0 - return _c_base_offset(derived.handle, base.handle, address, direction) -def c_base_offset1(space, derived_h, base, address, direction): - return _c_base_offset(derived_h, base.handle, address, direction) - -# method/function reflection information ------------------------------------- -_c_num_methods = rffi.llexternal( - "cppyy_num_methods", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_methods(space, cppscope): - return _c_num_methods(cppscope.handle) -_c_method_index_at = rffi.llexternal( - "cppyy_method_index_at", - [C_SCOPE, rffi.INT], C_INDEX, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_index_at(space, cppscope, imethod): - return _c_method_index_at(cppscope.handle, imethod) -_c_method_indices_from_name = rffi.llexternal( - "cppyy_method_indices_from_name", - [C_SCOPE, rffi.CCHARP], C_INDEX_ARRAY, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_indices_from_name(space, cppscope, name): - indices = _c_method_indices_from_name(cppscope.handle, name) - if not indices: - return [] - py_indices = [] - i = 0 - index = indices[i] - while index != -1: - i += 1 - py_indices.append(index) - index = indices[i] - c_free(space, rffi.cast(rffi.VOIDP, indices)) # c_free defined below - return py_indices - -_c_method_name = rffi.llexternal( - "cppyy_method_name", - [C_SCOPE, C_INDEX], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_name(space, cppscope, index): - return charp2str_free(space, _c_method_name(cppscope.handle, index)) -_c_method_result_type = rffi.llexternal( - "cppyy_method_result_type", - [C_SCOPE, C_INDEX], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_result_type(space, cppscope, index): - return charp2str_free(space, _c_method_result_type(cppscope.handle, index)) -_c_method_num_args = rffi.llexternal( - "cppyy_method_num_args", - [C_SCOPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_num_args(space, cppscope, index): - return _c_method_num_args(cppscope.handle, index) -_c_method_req_args = rffi.llexternal( - "cppyy_method_req_args", - [C_SCOPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_req_args(space, cppscope, index): - return _c_method_req_args(cppscope.handle, index) -_c_method_arg_type = rffi.llexternal( - "cppyy_method_arg_type", - [C_SCOPE, C_INDEX, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_arg_type(space, cppscope, index, arg_index): - return charp2str_free(space, _c_method_arg_type(cppscope.handle, index, arg_index)) -_c_method_arg_default = rffi.llexternal( - "cppyy_method_arg_default", - [C_SCOPE, C_INDEX, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_arg_default(space, cppscope, index, arg_index): - return charp2str_free(space, _c_method_arg_default(cppscope.handle, index, arg_index)) -_c_method_signature = rffi.llexternal( - "cppyy_method_signature", - [C_SCOPE, C_INDEX], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_signature(space, cppscope, index): - return charp2str_free(space, _c_method_signature(cppscope.handle, index)) - -_c_method_is_template = rffi.llexternal( - "cppyy_method_is_template", - [C_SCOPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_is_template(space, cppscope, index): - return _c_method_is_template(cppscope.handle, index) -_c_method_num_template_args = rffi.llexternal( - "cppyy_method_num_template_args", - [C_SCOPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -_c_method_template_arg_name = rffi.llexternal( - "cppyy_method_template_arg_name", - [C_SCOPE, C_INDEX, C_INDEX], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_template_args(space, cppscope, index): - nargs = _c_method_num_template_args(cppscope.handle, index) - args = [c_resolve_name(space, - charp2str_free(space, _c_method_template_arg_name(cppscope.handle, index, iarg))) - for iarg in range(nargs)] - return args - -_c_get_method = rffi.llexternal( - "cppyy_get_method", - [C_SCOPE, C_INDEX], C_METHOD, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_method(space, cppscope, index): - return _c_get_method(cppscope.handle, index) -_c_get_global_operator = rffi.llexternal( - "cppyy_get_global_operator", - [C_SCOPE, C_SCOPE, C_SCOPE, rffi.CCHARP], WLAVC_INDEX, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_global_operator(space, nss, lc, rc, op): - if nss is not None: - return _c_get_global_operator(nss.handle, lc.handle, rc.handle, op) - return rffi.cast(WLAVC_INDEX, -1) - -# method properties ---------------------------------------------------------- -_c_is_constructor = rffi.llexternal( - "cppyy_is_constructor", - [C_TYPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_constructor(space, cppclass, index): - return _c_is_constructor(cppclass.handle, index) -_c_is_staticmethod = rffi.llexternal( - "cppyy_is_staticmethod", - [C_TYPE, C_INDEX], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_staticmethod(space, cppclass, index): - return _c_is_staticmethod(cppclass.handle, index) - -# data member reflection information ----------------------------------------- -_c_num_datamembers = rffi.llexternal( - "cppyy_num_datamembers", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_datamembers(space, cppscope): - return _c_num_datamembers(cppscope.handle) -_c_datamember_name = rffi.llexternal( - "cppyy_datamember_name", - [C_SCOPE, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_datamember_name(space, cppscope, datamember_index): - return charp2str_free(space, _c_datamember_name(cppscope.handle, datamember_index)) -_c_datamember_type = rffi.llexternal( - "cppyy_datamember_type", - [C_SCOPE, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_datamember_type(space, cppscope, datamember_index): - return charp2str_free(space, _c_datamember_type(cppscope.handle, datamember_index)) -_c_datamember_offset = rffi.llexternal( - "cppyy_datamember_offset", - [C_SCOPE, rffi.INT], rffi.LONG, # actually ptrdiff_t - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_datamember_offset(space, cppscope, datamember_index): - return _c_datamember_offset(cppscope.handle, datamember_index) - -_c_datamember_index = rffi.llexternal( - "cppyy_datamember_index", - [C_SCOPE, rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_datamember_index(space, cppscope, name): - return _c_datamember_index(cppscope.handle, name) - -# data member properties ----------------------------------------------------- -_c_is_publicdata = rffi.llexternal( - "cppyy_is_publicdata", - [C_SCOPE, rffi.INT], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_publicdata(space, cppscope, datamember_index): - return _c_is_publicdata(cppscope.handle, datamember_index) -_c_is_staticdata = rffi.llexternal( - "cppyy_is_staticdata", - [C_SCOPE, rffi.INT], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_staticdata(space, cppscope, datamember_index): - return _c_is_staticdata(cppscope.handle, datamember_index) - -# misc helpers --------------------------------------------------------------- -_c_strtoll = rffi.llexternal( - "cppyy_strtoll", - [rffi.CCHARP], rffi.LONGLONG, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_strtoll(space, svalue): - return _c_strtoll(svalue) -_c_strtoull = rffi.llexternal( - "cppyy_strtoull", - [rffi.CCHARP], rffi.ULONGLONG, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_strtoull(space, svalue): - return _c_strtoull(svalue) -_c_free = rffi.llexternal( - "cppyy_free", - [rffi.VOIDP], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_free(space, voidp): - return _c_free(voidp) - -def charp2str_free(space, charp): - string = rffi.charp2str(charp) - voidp = rffi.cast(rffi.VOIDP, charp) - _c_free(voidp) - return string - -_c_charp2stdstring = rffi.llexternal( - "cppyy_charp2stdstring", - [rffi.CCHARP, rffi.SIZE_T], C_OBJECT, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_charp2stdstring(space, pystr, sz): - with rffi.scoped_view_charp(pystr) as cstr: - cppstr = _c_charp2stdstring(cstr, sz) - return cppstr -_c_stdstring2stdstring = rffi.llexternal( - "cppyy_stdstring2stdstring", - [C_OBJECT], C_OBJECT, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_stdstring2stdstring(space, cppobject): - return _c_stdstring2stdstring(cppobject) - -_c_stdvector_valuetype = rffi.llexternal( - "cppyy_stdvector_valuetype", - [rffi.CCHARP], rffi.CCHARP, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_stdvector_valuetype(space, pystr): - cstr = rffi.str2charp(pystr) - result = _c_stdvector_valuetype(cstr) - rffi.free_charp(cstr) - if result: - return charp2str_free(space, result) - return "" -_c_stdvector_valuesize = rffi.llexternal( - "cppyy_stdvector_valuesize", - [rffi.CCHARP], rffi.SIZE_T, - releasegil=ts_helper, - compilation_info=backend.eci) -def c_stdvector_valuesize(space, pystr): - cstr = rffi.str2charp(pystr) - result = _c_stdvector_valuesize(cstr) - rffi.free_charp(cstr) - return result diff --git a/pypy/module/_cppyy/capi/cling_capi.py b/pypy/module/_cppyy/capi/cling_capi.py deleted file mode 100644 --- a/pypy/module/_cppyy/capi/cling_capi.py +++ /dev/null @@ -1,196 +0,0 @@ -import py, os - -from pypy.objspace.std.iterobject import W_AbstractSeqIterObject - -from pypy.interpreter.error import OperationError -from pypy.interpreter.gateway import interp2app - -from rpython.translator.tool.cbuild import ExternalCompilationInfo -from rpython.rtyper.lltypesystem import rffi, lltype -from rpython.rlib.rarithmetic import intmask -from rpython.rlib import jit, libffi, rdynload - -from pypy.module._rawffi.array import W_ArrayInstance -from pypy.module._cppyy.capi.capi_types import C_OBJECT - -__all__ = ['identify', 'std_string_name', 'eci', 'c_load_dictionary'] - -pkgpath = py.path.local(__file__).dirpath().join(os.pardir) -srcpath = pkgpath.join("src") -incpath = pkgpath.join("include") - -import commands -(config_stat, incdir) = commands.getstatusoutput("root-config --incdir") - -if os.environ.get("ROOTSYS"): - if config_stat != 0: # presumably Reflex-only - rootincpath = [os.path.join(os.environ["ROOTSYS"], "interpreter/cling/include"), - os.path.join(os.environ["ROOTSYS"], "interpreter/llvm/inst/include"), - os.path.join(os.environ["ROOTSYS"], "include"),] - rootlibpath = [os.path.join(os.environ["ROOTSYS"], "lib64"), os.path.join(os.environ["ROOTSYS"], "lib")] - else: - rootincpath = [incdir] - rootlibpath = commands.getoutput("root-config --libdir").split() -else: - if config_stat == 0: - rootincpath = [incdir] - rootlibpath = commands.getoutput("root-config --libdir").split() - else: - rootincpath = [] - rootlibpath = [] - -def identify(): - return 'Cling' - -ts_reflect = False -ts_call = 'auto' -ts_memory = 'auto' -ts_helper = 'auto' - -std_string_name = 'std::basic_string' - -# force loading (and exposure) of libCore symbols From pypy.commits at gmail.com Sat Oct 7 09:11:10 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 06:11:10 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix str.encode() when passing errors but no encoding Message-ID: <59d8d26e.53571c0a.af4aa.a75f@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92622:31a461517920 Date: 2017-10-06 10:13 +0200 http://bitbucket.org/pypy/pypy/changeset/31a461517920/ Log: Fix str.encode() when passing errors but no encoding diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1010,6 +1010,9 @@ raises(TypeError, "str(b'', encoding=None)") raises(TypeError, 'u"".encode("utf-8", None)') + def test_encode_wrong_errors(self): + assert ''.encode(errors='some_wrong_name') == b'' + def test_casefold(self): assert u'hello'.casefold() == u'hello' assert u'hELlo'.casefold() == u'hello' diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -573,6 +573,8 @@ u, len(u), errors, errorhandler=eh)) from pypy.module._codecs.interp_codecs import encode_text + if encoding is None: + encoding = space.sys.defaultencoding w_retval = encode_text(space, w_object, encoding, errors) if not space.isinstance_w(w_retval, space.w_bytes): raise oefmt(space.w_TypeError, From pypy.commits at gmail.com Sat Oct 7 09:11:14 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 06:11:14 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: whack whack whack until we get to the point of getitem working Message-ID: <59d8d272.52c6df0a.821dd.5cca@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92623:ecf3b7cd79eb Date: 2017-10-06 12:14 +0200 http://bitbucket.org/pypy/pypy/changeset/ecf3b7cd79eb/ Log: whack whack whack until we get to the point of getitem working diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -1,2 +1,3 @@ * unskip tests in test_unicodeobject.py -* rutf8.prev_codepoint_pos should use r_uint \ No newline at end of file +* rutf8.prev_codepoint_pos should use r_uint +* elidable in rutf8.check_utf8, WTF is wrong with that \ No newline at end of file diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -126,8 +126,7 @@ self.orig = handler def handle(self, errors, encoding, msg, s, pos, endpos): - s, p = self.orig(errors, encoding, msg, s, pos, endpos) - return s.decode("utf8"), p + return self.orig(errors, encoding, msg, s, pos, endpos) class EncodeWrapper(object): def __init__(self, handler): @@ -145,7 +144,8 @@ def str_decode_unicode_escape(s, slen, errors, final, errorhandler, ud_handler): w = DecodeWrapper(errorhandler) - u, pos = runicode.str_decode_unicode_escape(s, slen, errors, final, w.handle, + u, pos = runicode.str_decode_unicode_escape(s, slen, errors, final, + w.handle, ud_handler) return u.encode('utf8'), pos, len(u) @@ -159,7 +159,7 @@ return getattr(runicode, encoder_call_name)(u, len(u), errors, w.handle) def decoder(s, slen, errors, final, errorhandler): - w = DecodeWrapper(errorhandler) + w = DecodeWrapper((errorhandler)) u, pos = getattr(runicode, decoder_name)(s, slen, errors, final, w.handle) return u.encode('utf8'), pos, len(u) encoder.__name__ = encoder_name diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,7 +66,7 @@ "position %d from error handler out of bounds", newpos) w_replace = space.convert_to_w_unicode(w_replace) - return w_replace._utf8, newpos + return w_replace._utf8.decode('utf8'), newpos return call_errorhandler def make_decode_errorhandler(self, space): diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -38,8 +38,8 @@ self._utf8 = utf8str self._length = length self._index_storage = rutf8.null_storage() - if not we_are_translated(): - assert rutf8.check_utf8(utf8str, allow_surrogates=True) == length + #if not we_are_translated(): + # assert rutf8.check_utf8(utf8str, allow_surrogates=True) == length def __repr__(self): """representation for debugging purposes""" diff --git a/rpython/jit/codewriter/effectinfo.py b/rpython/jit/codewriter/effectinfo.py --- a/rpython/jit/codewriter/effectinfo.py +++ b/rpython/jit/codewriter/effectinfo.py @@ -6,6 +6,10 @@ from rpython.tool.algo import bitstring +class UnsupportedFieldExc(Exception): + pass + + class EffectInfo(object): _cache = {} @@ -313,7 +317,10 @@ return if getattr(T.OF, fieldname) is lltype.Void: return - descr = cpu.interiorfielddescrof(T, fieldname) + try: + descr = cpu.interiorfielddescrof(T, fieldname) + except UnsupportedFieldExc: + return descrs_interiorfields.append(descr) # a read or a write to an interiorfield, inside an array of diff --git a/rpython/jit/codewriter/heaptracker.py b/rpython/jit/codewriter/heaptracker.py --- a/rpython/jit/codewriter/heaptracker.py +++ b/rpython/jit/codewriter/heaptracker.py @@ -94,6 +94,7 @@ def all_interiorfielddescrs(gccache, ARRAY, get_field_descr=None): from rpython.jit.backend.llsupport import descr + from rpython.jit.codewriter.effectinfo import UnsupportedFieldExc if get_field_descr is None: get_field_descr = descr.get_field_descr @@ -107,7 +108,7 @@ if name == 'typeptr': continue # dealt otherwise elif isinstance(FIELD, lltype.Struct): - raise Exception("unexpected array(struct(struct))") + raise UnsupportedFieldExc("unexpected array(struct(struct))") res.append(get_field_descr(gccache, ARRAY, name)) return res diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -94,18 +94,20 @@ def default_unicode_error_decode(errors, encoding, msg, s, startingpos, endingpos): + assert endingpos >= 0 if errors == 'replace': - return u'\ufffd'.encode('utf8'), endingpos + return u'\ufffd', endingpos if errors == 'ignore': - return '', endingpos + return u'', endingpos raise UnicodeDecodeError(encoding, s, startingpos, endingpos, msg) def default_unicode_error_encode(errors, encoding, msg, u, startingpos, endingpos): + assert endingpos >= 0 if errors == 'replace': - return '?', None, endingpos + return u'?', None, endingpos if errors == 'ignore': - return '', None, endingpos + return u'', None, endingpos raise UnicodeEncodeError(encoding, u, startingpos, endingpos, msg) # ____________________________________________________________ diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -191,7 +191,7 @@ def __init__(self, pos): self.pos = pos - at jit.elidable +#@jit.elidable def check_ascii(s): for i in range(len(s)): if ord(s[i]) > 0x7F: @@ -289,12 +289,14 @@ (ordch1 == 0xf4 and ordch2 > 0x8f)) - at jit.elidable +#@jit.elidable def check_utf8(s, allow_surrogates=False): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. Note that surrogates are not handled specially here. """ + import pdb + pdb.set_trace() pos = 0 continuation_bytes = 0 while pos < len(s): @@ -416,6 +418,7 @@ break return storage + at jit.dont_look_inside def codepoint_position_at_index(utf8, storage, index): """ Return byte index of a character inside utf8 encoded string, given storage of type UTF8_INDEX_STORAGE. The index must be smaller than @@ -436,6 +439,7 @@ else: return next_codepoint_pos(utf8, next_codepoint_pos(utf8, bytepos)) + at jit.dont_look_inside def codepoint_at_index(utf8, storage, index): """ Return codepoint of a character inside utf8 encoded string, given storage of type UTF8_INDEX_STORAGE diff --git a/rpython/rtyper/rstr.py b/rpython/rtyper/rstr.py --- a/rpython/rtyper/rstr.py +++ b/rpython/rtyper/rstr.py @@ -8,7 +8,88 @@ from rpython.rtyper.rint import IntegerRepr from rpython.rtyper.rfloat import FloatRepr from rpython.tool.pairtype import pairtype, pair -from rpython.tool.sourcetools import func_with_new_name + +def str_decode_utf8(s): + from rpython.rlib.rstring import UnicodeBuilder + from rpython.rlib import runicode + + size = len(s) + if size == 0: + return u'' + + result = UnicodeBuilder(size) + pos = 0 + while pos < size: + ordch1 = ord(s[pos]) + # fast path for ASCII + # XXX maybe use a while loop here + if ordch1 < 0x80: + result.append(unichr(ordch1)) + pos += 1 + continue + + n = ord(runicode._utf8_code_length[ordch1 - 0x80]) + if pos + n > size: + raise UnicodeDecodeError('utf8', s, pos, pos + 1, + 'whatever') + if n == 0: + raise UnicodeDecodeError('utf8', s, pos, pos + 1, + 'whatever') + elif n == 1: + assert 0, "ascii should have gone through the fast path" + + elif n == 2: + ordch2 = ord(s[pos+1]) + if runicode._invalid_byte_2_of_2(ordch2): + + raise UnicodeDecodeError('utf8', s, pos, pos + 1, + 'whatever') + # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz + result.append(unichr(((ordch1 & 0x1F) << 6) + # 0b00011111 + (ordch2 & 0x3F))) # 0b00111111 + pos += 2 + + elif n == 3: + ordch2 = ord(s[pos+1]) + ordch3 = ord(s[pos+2]) + if (runicode._invalid_byte_2_of_3(ordch1, ordch2, True) or + runicode._invalid_byte_3_of_3(ordch3)): + raise UnicodeDecodeError('utf8', s, pos, pos + 1, + 'whatever') + # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz + result.append(unichr(((ordch1 & 0x0F) << 12) + # 0b00001111 + ((ordch2 & 0x3F) << 6) + # 0b00111111 + (ordch3 & 0x3F))) # 0b00111111 + pos += 3 + + elif n == 4: + ordch2 = ord(s[pos+1]) + ordch3 = ord(s[pos+2]) + ordch4 = ord(s[pos+3]) + if (runicode._invalid_byte_2_of_4(ordch1, ordch2) or + runicode._invalid_byte_3_of_4(ordch3) or + runicode._invalid_byte_4_of_4(ordch4)): + + raise UnicodeDecodeError('utf8', s, pos, pos + 1, + 'whatever') + # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz + c = (((ordch1 & 0x07) << 18) + # 0b00000111 + ((ordch2 & 0x3F) << 12) + # 0b00111111 + ((ordch3 & 0x3F) << 6) + # 0b00111111 + (ordch4 & 0x3F)) # 0b00111111 + if c <= runicode.MAXUNICODE: + result.append(runicode.UNICHR(c)) + else: + # compute and append the two surrogates: + # translate from 10000..10FFFF to 0..FFFF + c -= 0x10000 + # high surrogate = top 10 bits added to D800 + result.append(unichr(0xD800 + (c >> 10))) + # low surrogate = bottom 10 bits added to DC00 + result.append(unichr(0xDC00 + (c & 0x03FF))) + pos += 4 + + return result.build() class AbstractStringRepr(Repr): @@ -16,13 +97,10 @@ @jit.elidable def ll_decode_utf8(self, llvalue): from rpython.rtyper.annlowlevel import hlstr - from rpython.rlib import runicode value = hlstr(llvalue) assert value is not None - errorhandler = runicode.default_unicode_error_decode # NB. keep the arguments in sync with annotator/unaryop.py - u, pos = runicode.str_decode_utf_8_elidable( - value, len(value), 'strict', True, errorhandler, True) + u = str_decode_utf8(value) # XXX maybe the whole ''.decode('utf-8') should be not RPython. return self.ll.llunicode(u) From pypy.commits at gmail.com Sat Oct 7 09:11:17 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 06:11:17 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix for get_wch() on platforms where wint_t != int Message-ID: <59d8d275.14a1df0a.1061f.299a@mx.google.com> Author: Armin Rigo Branch: py3.5 Changeset: r92624:8103736dfa9f Date: 2017-10-06 14:02 +0200 http://bitbucket.org/pypy/pypy/changeset/8103736dfa9f/ Log: Fix for get_wch() on platforms where wint_t != int diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py --- a/lib_pypy/_curses.py +++ b/lib_pypy/_curses.py @@ -404,7 +404,7 @@ return val def get_wch(self, *args): - wch = ffi.new("int[1]") + wch = ffi.new("wint_t[1]") if len(args) == 0: val = lib.wget_wch(self._win, wch) elif len(args) == 2: From pypy.commits at gmail.com Sat Oct 7 09:11:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:21 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: use a better quality picture Message-ID: <59d8d279.53571c0a.af4aa.a76e@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5835:6cd4c639387a Date: 2017-10-05 23:22 +0200 http://bitbucket.org/pypy/extradoc/changeset/6cd4c639387a/ Log: use a better quality picture diff --git a/sprintinfo/cape-town-2017/2017-10-04-155524.jpg b/sprintinfo/cape-town-2017/2017-10-04-155524.jpg index 4bbb25cda6773e5d1b87986645f036127a38bce6..60691b685f56f59dd084e060cedb31fa96ebb798 GIT binary patch [cut] From pypy.commits at gmail.com Sat Oct 7 09:11:24 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:24 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: WIP: manually revert part of the previous commit and go back to the old implementation. The plan is to do changes one by one in order to spot when/where we break things Message-ID: <59d8d27c.90921c0a.3f8a8.368a@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92625:2d5f1e54e903 Date: 2017-10-07 01:00 +0200 http://bitbucket.org/pypy/pypy/changeset/2d5f1e54e903/ Log: WIP: manually revert part of the previous commit and go back to the old implementation. The plan is to do changes one by one in order to spot when/where we break things diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -49,23 +49,35 @@ @cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def _PyObject_NewVar(space, tp, nitems): - size = _VAR_SIZE(tp, nitems) - assert size >= rffi.sizeof(PyObject.TO) - buf = lltype.malloc(rffi.VOIDP.TO, size, - flavor='raw', zero=True, - add_memory_pressure=True) # XXX add_memory_pressure? - pyobj = rffi.cast(PyObject, buf) - # XXX should we do this? - ## if (pyobj == NULL) - ## return PyErr_NoMemory() - # - pyobj.c_ob_refcnt = 1 # why do we need this? - # - # XXX: incref tp? In the previous version, this was done by - # typedescr.alllocate(); however, CPython doesn't. - #incref(space, tp) - PyObject_InitVar(space, pyobj, tp, nitems) - return pyobj + w_type = from_ref(space, rffi.cast(PyObject, tp)) + assert isinstance(w_type, W_TypeObject) + typedescr = get_typedescr(w_type.layout.typedef) + py_obj = typedescr.allocate(space, w_type, itemcount=nitems) + #py_obj.c_ob_refcnt = 0 --- will be set to 1 again by PyObject_Init{Var} + if tp.c_tp_itemsize == 0: + w_obj = PyObject_Init(space, py_obj, tp) + else: + py_objvar = rffi.cast(PyVarObject, py_obj) + w_obj = PyObject_InitVar(space, py_objvar, tp, nitems) + return py_obj + + ## size = _VAR_SIZE(tp, nitems) + ## assert size >= rffi.sizeof(PyObject.TO) + ## buf = lltype.malloc(rffi.VOIDP.TO, size, + ## flavor='raw', zero=True, + ## add_memory_pressure=True) # XXX add_memory_pressure? + ## pyobj = rffi.cast(PyObject, buf) + ## # XXX should we do this? + ## ## if (pyobj == NULL) + ## ## return PyErr_NoMemory() + ## # + ## pyobj.c_ob_refcnt = 1 # why do we need this? + ## # + ## # XXX: incref tp? In the previous version, this was done by + ## # typedescr.alllocate(); however, CPython doesn't. + ## #incref(space, tp) + ## PyObject_InitVar(space, pyobj, tp, nitems) + ## return pyobj @slot_function([PyObject], lltype.Void, no_gc=True) def PyObject_dealloc(obj): From pypy.commits at gmail.com Sat Oct 7 09:11:26 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:26 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: WIP: in the quest to remove references to space in _PyObject_NewVar, manually inline the content of BaseCpyTypedescr.allocate; for the w_type case, we still go through typedescr.allocate(). Let's see if tests pass Message-ID: <59d8d27e.1986df0a.ede72.a863@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92626:15a548a8aebf Date: 2017-10-07 01:23 +0200 http://bitbucket.org/pypy/pypy/changeset/15a548a8aebf/ Log: WIP: in the quest to remove references to space in _PyObject_NewVar, manually inline the content of BaseCpyTypedescr.allocate; for the w_type case, we still go through typedescr.allocate(). Let's see if tests pass diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -51,33 +51,41 @@ def _PyObject_NewVar(space, tp, nitems): w_type = from_ref(space, rffi.cast(PyObject, tp)) assert isinstance(w_type, W_TypeObject) - typedescr = get_typedescr(w_type.layout.typedef) - py_obj = typedescr.allocate(space, w_type, itemcount=nitems) - #py_obj.c_ob_refcnt = 0 --- will be set to 1 again by PyObject_Init{Var} + if w_type is space.w_type: + # XXX: integrate this logic with the one below + typedescr = get_typedescr(w_type.layout.typedef) + pyobj = typedescr.allocate(space, w_type, itemcount=nitems) + else: + # std case: do the same logic as BaseCpyTypedescr.allocate + # CPython doesn't do this: investigate whether we can remove + flags = rffi.cast(lltype.Signed, tp.c_tp_flags) + if flags & Py_TPFLAGS_HEAPTYPE: + Py_IncRef(space, w_type) + # + size = _VAR_SIZE(tp, nitems) + assert size >= rffi.sizeof(PyObject.TO) + # XXX: zero=True? add_memory_pressure? + buf = lltype.malloc(rffi.VOIDP.TO, size, + flavor='raw', zero=True, + add_memory_pressure=True) + pyobj = rffi.cast(PyObject, buf) + # XXX CPython does this + ## if (pyobj == NULL) + ## return PyErr_NoMemory() + pyobj.c_ob_refcnt = 1 + if tp.c_tp_itemsize: + pyvarobj = rffi.cast(PyVarObject, pyobj) + pyvarobj.c_ob_size = nitems + pyobj.c_ob_refcnt = 1 + #pyobj.c_ob_pypy_link should get assigned very quickly + pyobj.c_ob_type = tp + # if tp.c_tp_itemsize == 0: - w_obj = PyObject_Init(space, py_obj, tp) + w_obj = PyObject_Init(space, pyobj, tp) else: - py_objvar = rffi.cast(PyVarObject, py_obj) + py_objvar = rffi.cast(PyVarObject, pyobj) w_obj = PyObject_InitVar(space, py_objvar, tp, nitems) - return py_obj - - ## size = _VAR_SIZE(tp, nitems) - ## assert size >= rffi.sizeof(PyObject.TO) - ## buf = lltype.malloc(rffi.VOIDP.TO, size, - ## flavor='raw', zero=True, - ## add_memory_pressure=True) # XXX add_memory_pressure? - ## pyobj = rffi.cast(PyObject, buf) - ## # XXX should we do this? - ## ## if (pyobj == NULL) - ## ## return PyErr_NoMemory() - ## # - ## pyobj.c_ob_refcnt = 1 # why do we need this? - ## # - ## # XXX: incref tp? In the previous version, this was done by - ## # typedescr.alllocate(); however, CPython doesn't. - ## #incref(space, tp) - ## PyObject_InitVar(space, pyobj, tp, nitems) - ## return pyobj + return pyobj @slot_function([PyObject], lltype.Void, no_gc=True) def PyObject_dealloc(obj): From pypy.commits at gmail.com Sat Oct 7 09:11:28 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:28 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: merge heads Message-ID: <59d8d280.dc4a1c0a.4a601.6069@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92627:512a517b4e70 Date: 2017-10-07 01:34 +0200 http://bitbucket.org/pypy/pypy/changeset/512a517b4e70/ Log: merge heads diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -179,23 +179,22 @@ pass CANNOT_FAIL = CannotFail() -# The same function can be called in three different contexts: +# The same function can be called in two different contexts: # (1) from C code -# (2) in the test suite, though the "api" object -# (3) from RPython code, for example in the implementation of another function. +# (2) from RPython code, for example in the implementation of another function +# or in tests. # -# In contexts (2) and (3), a function declaring a PyObject argument type will +# In context (2), a function declaring a PyObject argument type will # receive a wrapped pypy object if the parameter name starts with 'w_', a -# reference (= rffi pointer) otherwise; conversion is automatic. Context (2) -# only allows calls with a wrapped object. +# reference (= rffi pointer) otherwise; conversion is automatic. # # Functions with a PyObject return type should return a wrapped object. # -# Functions may raise exceptions. In context (3), the exception flows normally -# through the calling function. In context (1) and (2), the exception is +# Functions may raise exceptions. In context (2), the exception flows normally +# through the calling function. In context (1), the exception is # caught; if it is an OperationError, it is stored in the thread state; other -# exceptions generate a OperationError(w_SystemError); and the funtion returns -# the error value specifed in the API. +# exceptions generate a OperationError(w_SystemError); and the function returns +# the error value specified in the API. # # Handling of the GIL # ------------------- From pypy.commits at gmail.com Sat Oct 7 09:11:31 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:31 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: change strategy: instead of trying to mimic what CPython does, for now just try to refactor things in a way whih will allow us to remove all references to the space. We will need to refactor Py_IncRef first Message-ID: <59d8d283.68addf0a.a3ea5.ce74@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92628:d9e6994b8085 Date: 2017-10-07 10:39 +0100 http://bitbucket.org/pypy/pypy/changeset/d9e6994b8085/ Log: change strategy: instead of trying to mimic what CPython does, for now just try to refactor things in a way whih will allow us to remove all references to the space. We will need to refactor Py_IncRef first diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -49,6 +49,8 @@ @cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def _PyObject_NewVar(space, tp, nitems): + from pypy.module.cpyext.pyobject import _allocate_generic_object + w_type = from_ref(space, rffi.cast(PyObject, tp)) assert isinstance(w_type, W_TypeObject) if w_type is space.w_type: @@ -56,29 +58,7 @@ typedescr = get_typedescr(w_type.layout.typedef) pyobj = typedescr.allocate(space, w_type, itemcount=nitems) else: - # std case: do the same logic as BaseCpyTypedescr.allocate - # CPython doesn't do this: investigate whether we can remove - flags = rffi.cast(lltype.Signed, tp.c_tp_flags) - if flags & Py_TPFLAGS_HEAPTYPE: - Py_IncRef(space, w_type) - # - size = _VAR_SIZE(tp, nitems) - assert size >= rffi.sizeof(PyObject.TO) - # XXX: zero=True? add_memory_pressure? - buf = lltype.malloc(rffi.VOIDP.TO, size, - flavor='raw', zero=True, - add_memory_pressure=True) - pyobj = rffi.cast(PyObject, buf) - # XXX CPython does this - ## if (pyobj == NULL) - ## return PyErr_NoMemory() - pyobj.c_ob_refcnt = 1 - if tp.c_tp_itemsize: - pyvarobj = rffi.cast(PyVarObject, pyobj) - pyvarobj.c_ob_size = nitems - pyobj.c_ob_refcnt = 1 - #pyobj.c_ob_pypy_link should get assigned very quickly - pyobj.c_ob_type = tp + return _allocate_generic_object(space, tp, nitems) # if tp.c_tp_itemsize == 0: w_obj = PyObject_Init(space, pyobj, tp) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -21,6 +21,29 @@ #________________________________________________________ # type description +def _allocate_generic_object(space, pytype, itemcount, immortal=False): + # Don't increase refcount for non-heaptypes + flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) + if flags & Py_TPFLAGS_HEAPTYPE: + Py_IncRef(space, pytype) + + size = pytype.c_tp_basicsize + if pytype.c_tp_itemsize: + size += itemcount * pytype.c_tp_itemsize + assert size >= rffi.sizeof(PyObject.TO) + buf = lltype.malloc(rffi.VOIDP.TO, size, + flavor='raw', zero=True, + add_memory_pressure=True, immortal=immortal) + pyobj = rffi.cast(PyObject, buf) + if pytype.c_tp_itemsize: + pyvarobj = rffi.cast(PyVarObject, pyobj) + pyvarobj.c_ob_size = itemcount + pyobj.c_ob_refcnt = 1 + #pyobj.c_ob_pypy_link should get assigned very quickly + pyobj.c_ob_type = pytype + return pyobj + + class BaseCpyTypedescr(object): basestruct = PyObject.TO W_BaseObject = W_ObjectObject @@ -32,33 +55,10 @@ def allocate(self, space, w_type, itemcount=0, immortal=False): # typically called from PyType_GenericAlloc via typedescr.allocate # this returns a PyObject with ob_refcnt == 1. - pytype = as_pyobj(space, w_type) pytype = rffi.cast(PyTypeObjectPtr, pytype) assert pytype - # Don't increase refcount for non-heaptypes - flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) - if flags & Py_TPFLAGS_HEAPTYPE: - Py_IncRef(space, w_type) - - if pytype: - size = pytype.c_tp_basicsize - else: - size = rffi.sizeof(self.basestruct) - if pytype.c_tp_itemsize: - size += itemcount * pytype.c_tp_itemsize - assert size >= rffi.sizeof(PyObject.TO) - buf = lltype.malloc(rffi.VOIDP.TO, size, - flavor='raw', zero=True, - add_memory_pressure=True, immortal=immortal) - pyobj = rffi.cast(PyObject, buf) - if pytype.c_tp_itemsize: - pyvarobj = rffi.cast(PyVarObject, pyobj) - pyvarobj.c_ob_size = itemcount - pyobj.c_ob_refcnt = 1 - #pyobj.c_ob_pypy_link should get assigned very quickly - pyobj.c_ob_type = pytype - return pyobj + return _allocate_generic_object(space, pytype, itemcount, immortal) def attach(self, space, pyobj, w_obj, w_userdata=None): pass From pypy.commits at gmail.com Sat Oct 7 09:11:33 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:33 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: copy&adapt state.C from the branch cpyext-avoid-roundtrip, to be able to directly call functions defined in C; improve _PyObject_NewVar to compare tp directly with PyType_Type, without passing through the space Message-ID: <59d8d285.040c1c0a.75a0a.de76@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92629:9c8e90ccce77 Date: 2017-10-07 12:33 +0200 http://bitbucket.org/pypy/pypy/changeset/9c8e90ccce77/ Log: copy&adapt state.C from the branch cpyext-avoid-roundtrip, to be able to directly call functions defined in C; improve _PyObject_NewVar to compare tp directly with PyType_Type, without passing through the space diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -1108,6 +1108,14 @@ _reinit_tls() add_fork_hook('child', reinit_tls) + +def attach_c_functions(space, eci, prefix): + state = space.fromcache(State) + state.C._PyPy_get_PyType_Type = rffi.llexternal( + '_PyPy_get_PyType_Type', [], PyTypeObjectPtr, + compilation_info=eci, _nowrapper=True) + + def init_function(func): INIT_FUNCTIONS.append(func) return func @@ -1176,6 +1184,7 @@ space.fromcache(State).install_dll(eci) modulename = py.path.local(eci.libraries[-1]) + attach_c_functions(space, eci, prefix) run_bootstrap_functions(space) # load the bridge, and init structure @@ -1419,6 +1428,7 @@ source_dir / "pythread.c", source_dir / "missing.c", source_dir / "pymem.c", + source_dir / "typeobject.c", ] def build_eci(code, use_micronumpy=False, translating=False): @@ -1513,6 +1523,7 @@ eci = build_eci(code, use_micronumpy, translating=True) space.fromcache(State).install_dll(eci) + attach_c_functions(space, eci, prefix) run_bootstrap_functions(space) # emit uninitialized static data diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -330,7 +330,7 @@ PyAPI_FUNC(int) PyPyType_Register(PyTypeObject *); #define PyObject_Length PyObject_Size #define _PyObject_GC_Del PyObject_GC_Del - +PyAPI_FUNC(PyTypeObject*) _PyPy_get_PyType_Type(void); #ifdef __cplusplus } diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -9,6 +9,7 @@ get_typedescr) from pypy.module.cpyext.typeobject import PyTypeObjectPtr from pypy.module.cpyext.pyerrors import PyErr_NoMemory, PyErr_BadInternalCall +from pypy.module.cpyext.state import State from pypy.objspace.std.typeobject import W_TypeObject from pypy.interpreter.error import OperationError, oefmt import pypy.module.__builtin__.operation as operation @@ -50,11 +51,10 @@ @cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def _PyObject_NewVar(space, tp, nitems): from pypy.module.cpyext.pyobject import _allocate_generic_object - - w_type = from_ref(space, rffi.cast(PyObject, tp)) - assert isinstance(w_type, W_TypeObject) - if w_type is space.w_type: - # XXX: integrate this logic with the one below + state = space.fromcache(State) + if tp is state.C._PyPy_get_PyType_Type(): + w_type = from_ref(space, rffi.cast(PyObject, tp)) + assert isinstance(w_type, W_TypeObject) typedescr = get_typedescr(w_type.layout.typedef) pyobj = typedescr.allocate(space, w_type, itemcount=nitems) else: diff --git a/pypy/module/cpyext/src/typeobject.c b/pypy/module/cpyext/src/typeobject.c new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/src/typeobject.c @@ -0,0 +1,7 @@ +#include "Python.h" + +PyTypeObject* +_PyPy_get_PyType_Type(void) +{ + return &PyType_Type; +} diff --git a/pypy/module/cpyext/state.py b/pypy/module/cpyext/state.py --- a/pypy/module/cpyext/state.py +++ b/pypy/module/cpyext/state.py @@ -14,6 +14,7 @@ self.programname = lltype.nullptr(rffi.CCHARP.TO) self.version = lltype.nullptr(rffi.CCHARP.TO) self.builder = None + self.C = CNamespace() def reset(self): from pypy.module.cpyext.modsupport import PyMethodDef @@ -168,6 +169,11 @@ return w_mod +class CNamespace: + def _freeze_(self): + return True + + def _rawrefcount_perform(space): from pypy.module.cpyext.pyobject import PyObject, decref while True: From pypy.commits at gmail.com Sat Oct 7 09:11:36 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 07 Oct 2017 06:11:36 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: bah Message-ID: <59d8d288.30b8df0a.bb5f7.0e0d@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92630:259118ef27be Date: 2017-10-07 12:45 +0200 http://bitbucket.org/pypy/pypy/changeset/259118ef27be/ Log: bah diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -52,7 +52,7 @@ def _PyObject_NewVar(space, tp, nitems): from pypy.module.cpyext.pyobject import _allocate_generic_object state = space.fromcache(State) - if tp is state.C._PyPy_get_PyType_Type(): + if tp == state.C._PyPy_get_PyType_Type(): w_type = from_ref(space, rffi.cast(PyObject, tp)) assert isinstance(w_type, W_TypeObject) typedescr = get_typedescr(w_type.layout.typedef) From pypy.commits at gmail.com Sat Oct 7 09:11:39 2017 From: pypy.commits at gmail.com (jerith) Date: Sat, 07 Oct 2017 06:11:39 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: expandtabs and [is]title. Message-ID: <59d8d28b.55ad1c0a.a83e5.894b@mx.google.com> Author: Jeremy Thurgood Branch: unicode-utf8 Changeset: r92631:842f2cbd6d78 Date: 2017-10-07 14:54 +0200 http://bitbucket.org/pypy/pypy/changeset/842f2cbd6d78/ Log: expandtabs and [is]title. diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -230,6 +230,7 @@ assert u"bROWN fOX".title() == u"Brown Fox" assert u"Brown Fox".title() == u"Brown Fox" assert u"bro!wn fox".title() == u"Bro!Wn Fox" + assert u"brow\u4321n fox".title() == u"Brow\u4321N Fox" def test_istitle(self): assert u"".istitle() == False diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -5,6 +5,7 @@ enforceargs, newlist_hint, specialize, we_are_translated) from rpython.rlib.buffer import StringBuffer from rpython.rlib.mutbuffer import MutableStringBuffer +from rpython.rlib.rarithmetic import ovfcheck from rpython.rlib.rstring import StringBuilder, split, rsplit, UnicodeBuilder,\ replace_count from rpython.rlib.runicode import make_unicode_escape_function @@ -349,6 +350,28 @@ def descr_rmod(self, space, w_values): return mod_format(space, w_values, self, do_unicode=True) + def descr_title(self, space): + if len(self._utf8) == 0: + return self + return W_UnicodeObject(self.title(self._utf8), self._len()) + + @jit.elidable + def title(self, value): + input = self._utf8 + builder = StringBuilder(len(input)) + i = 0 + previous_is_cased = False + while i < len(input): + ch = rutf8.codepoint_at_pos(input, i) + i = rutf8.next_codepoint_pos(input, i) + if not previous_is_cased: + ch = unicodedb.totitle(ch) + else: + ch = unicodedb.tolower(ch) + rutf8.unichr_as_utf8_append(builder, ch) + previous_is_cased = unicodedb.iscased(ch) + return builder.build() + def descr_translate(self, space, w_table): input = self._utf8 result = StringBuilder(len(input)) @@ -389,6 +412,30 @@ w_errors) return encode_object(space, self, encoding, errors) + @unwrap_spec(tabsize=int) + def descr_expandtabs(self, space, tabsize=8): + value = self._utf8 + if not value: + return self._empty() + + splitted = value.split('\t') + + try: + if tabsize > 0: + ovfcheck(len(splitted) * tabsize) + except OverflowError: + raise oefmt(space.w_OverflowError, "new string is too long") + expanded = oldtoken = splitted.pop(0) + newlen = self._len() - len(splitted) + + for token in splitted: + dist = self._tabindent(oldtoken, tabsize) + expanded += ' ' * dist + token + newlen += dist + oldtoken = token + + return W_UnicodeObject(expanded, newlen) + _StringMethods_descr_join = descr_join def descr_join(self, space, w_list): l = space.listview_unicode(w_list) @@ -438,6 +485,27 @@ i = rutf8.next_codepoint_pos(val, i) return space.newbool(cased) + def descr_istitle(self, space): + cased = False + previous_is_cased = False + val = self._utf8 + i = 0 + while i < len(val): + uchar = rutf8.codepoint_at_pos(val, i) + i = rutf8.next_codepoint_pos(val, i) + if unicodedb.isupper(uchar) or unicodedb.istitle(uchar): + if previous_is_cased: + return space.w_False + previous_is_cased = True + cased = True + elif unicodedb.islower(uchar): + if not previous_is_cased: + return space.w_False + cased = True + else: + previous_is_cased = False + return space.newbool(cased) + def descr_isupper(self, space): cased = False i = 0 From pypy.commits at gmail.com Sat Oct 7 09:18:55 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 07 Oct 2017 06:18:55 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: fix for win32 posix => os Message-ID: <59d8d43f.4f861c0a.d22d9.3152@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92632:724034639995 Date: 2017-10-08 00:17 +1100 http://bitbucket.org/pypy/pypy/changeset/724034639995/ Log: fix for win32 posix => os diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -560,14 +560,14 @@ s.ioctl(_socket.SIO_KEEPALIVE_VALS, (1, 100, 100)) def test_dup(self): - import _socket as socket, posix + import _socket as socket, os s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('localhost', 0)) fd = socket.dup(s.fileno()) assert s.fileno() != fd - assert posix.get_inheritable(s.fileno()) is False - assert posix.get_inheritable(fd) is False - posix.close(fd) + assert os.get_inheritable(s.fileno()) is False + assert os.get_inheritable(fd) is False + os.close(fd) s.close() def test_dup_error(self): @@ -682,18 +682,18 @@ raises(ValueError, _socket.socket, fileno=-1) def test_socket_non_inheritable(self): - import _socket, posix + import _socket, os s1 = _socket.socket() - assert posix.get_inheritable(s1.fileno()) is False + assert os.get_inheritable(s1.fileno()) is False s1.close() def test_socketpair_non_inheritable(self): - import _socket, posix + import _socket, os if not hasattr(_socket, 'socketpair'): skip("no socketpair") s1, s2 = _socket.socketpair() - assert posix.get_inheritable(s1.fileno()) is False - assert posix.get_inheritable(s2.fileno()) is False + assert os.get_inheritable(s1.fileno()) is False + assert os.get_inheritable(s2.fileno()) is False s1.close() s2.close() @@ -887,12 +887,12 @@ def test_accept_non_inheritable(self): - import _socket, posix + import _socket, os cli = _socket.socket() cli.connect(self.serv.getsockname()) fileno, addr = self.serv._accept() - assert posix.get_inheritable(fileno) is False - posix.close(fileno) + assert os.get_inheritable(fileno) is False + os.close(fileno) cli.close() def test_recv_into_params(self): From pypy.commits at gmail.com Sat Oct 7 09:23:00 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 07 Oct 2017 06:23:00 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge default into py3.5 Message-ID: <59d8d534.478f1c0a.a6499.9f5d@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92633:e3a0b4e339d0 Date: 2017-10-08 00:21 +1100 http://bitbucket.org/pypy/pypy/changeset/e3a0b4e339d0/ Log: merge default into py3.5 diff --git a/pypy/module/_cppyy/test/conftest.py b/pypy/module/_cppyy/test/conftest.py --- a/pypy/module/_cppyy/test/conftest.py +++ b/pypy/module/_cppyy/test/conftest.py @@ -4,7 +4,7 @@ def pytest_runtest_setup(item): if py.path.local.sysfind('genreflex') is None: import pypy.module._cppyy.capi.loadable_capi as lcapi - if 'dummy' in lcapi.reflection_library: + if 'dummy' in lcapi.backend_library: # run only tests that are covered by the dummy backend and tests # that do not rely on reflex import os @@ -33,7 +33,7 @@ import pypy.module._cppyy.capi.loadable_capi as lcapi try: import ctypes - ctypes.CDLL(lcapi.reflection_library) + ctypes.CDLL(lcapi.backend_library) except Exception as e: if config.option.runappdirect: return # "can't run dummy tests in -A" @@ -71,4 +71,4 @@ return raise - lcapi.reflection_library = str(soname) + lcapi.backend_library = str(soname) From pypy.commits at gmail.com Sat Oct 7 10:29:24 2017 From: pypy.commits at gmail.com (jerith) Date: Sat, 07 Oct 2017 07:29:24 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: capitalize and {starts, ends}with. Message-ID: <59d8e4c4.2587df0a.e5230.9be7@mx.google.com> Author: Jeremy Thurgood Branch: unicode-utf8 Changeset: r92634:2d6fe4fc14a3 Date: 2017-10-07 16:28 +0200 http://bitbucket.org/pypy/pypy/changeset/2d6fe4fc14a3/ Log: capitalize and {starts,ends}with. diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py --- a/pypy/objspace/std/stringmethods.py +++ b/pypy/objspace/std/stringmethods.py @@ -632,16 +632,13 @@ return space.w_True return space.w_False + # This is overridden in unicodeobject, but the two above are not. def _startswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) if start > len(value): - return self._starts_ends_overflow(prefix) + return False return startswith(value, prefix, start, end) - def _starts_ends_overflow(self, prefix): - return False # bug-to-bug compat: this is for strings and - # bytearrays, but overridden for unicodes - def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): value, start, end, _ = self._convert_idx_params(space, w_start, w_end) if space.isinstance_w(w_suffix, space.w_tuple): @@ -655,10 +652,11 @@ return space.w_True return space.w_False + # This is overridden in unicodeobject, but the two above are not. def _endswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) if start > len(value): - return self._starts_ends_overflow(prefix) + return False return endswith(value, prefix, start, end) def _strip(self, space, w_chars, left, right, name='strip'): diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -6,8 +6,9 @@ from rpython.rlib.buffer import StringBuffer from rpython.rlib.mutbuffer import MutableStringBuffer from rpython.rlib.rarithmetic import ovfcheck -from rpython.rlib.rstring import StringBuilder, split, rsplit, UnicodeBuilder,\ - replace_count +from rpython.rlib.rstring import ( + StringBuilder, split, rsplit, UnicodeBuilder, replace_count, startswith, + endswith) from rpython.rlib.runicode import make_unicode_escape_function from rpython.rlib import rutf8, jit @@ -139,6 +140,10 @@ return True @staticmethod + def _op_utf8(space, w_other, strict=None): + return W_UnicodeObject.convert_arg_to_w_unicode(space, w_other, strict)._utf8 + + @staticmethod def _op_val(space, w_other, strict=None): return W_UnicodeObject.convert_arg_to_w_unicode(space, w_other, strict)._utf8.decode('utf8') @@ -520,8 +525,17 @@ i = rutf8.next_codepoint_pos(val, i) return space.newbool(cased) - def _starts_ends_overflow(self, prefix): - return len(prefix) == 0 + def _startswith(self, space, value, w_prefix, start, end): + prefix = self._op_utf8(space, w_prefix) + if start > len(value): + return len(prefix) == 0 + return startswith(value, prefix, start, end) + + def _endswith(self, space, value, w_prefix, start, end): + prefix = self._op_utf8(space, w_prefix) + if start > len(value): + return len(prefix) == 0 + return endswith(value, prefix, start, end) def descr_add(self, space, w_other): try: @@ -644,6 +658,21 @@ return space.newlist_utf8(res) + def descr_capitalize(self, space): + value = self._utf8 + if len(value) == 0: + return self._empty() + + builder = StringBuilder(len(value)) + uchar = rutf8.codepoint_at_pos(value, 0) + i = rutf8.next_codepoint_pos(value, 0) + rutf8.unichr_as_utf8_append(builder, unicodedb.toupper(uchar)) + while i < len(value): + uchar = rutf8.codepoint_at_pos(value, i) + i = rutf8.next_codepoint_pos(value, i) + rutf8.unichr_as_utf8_append(builder, unicodedb.tolower(uchar)) + return W_UnicodeObject(builder.build(), self._len()) + @unwrap_spec(width=int, w_fillchar=WrappedDefault(' ')) def descr_center(self, space, width, w_fillchar): value = self._utf8 From pypy.commits at gmail.com Sat Oct 7 11:47:30 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 08:47:30 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Create ready pyobjects when calling create_ref() on str; fixes #2673 Message-ID: <59d8f712.8a821c0a.f5f88.9549@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92635:0037bb35a103 Date: 2017-10-07 17:45 +0200 http://bitbucket.org/pypy/pypy/changeset/0037bb35a103/ Log: Create ready pyobjects when calling create_ref() on str; fixes #2673 diff --git a/pypy/module/cpyext/test/test_unicodeobject.py b/pypy/module/cpyext/test/test_unicodeobject.py --- a/pypy/module/cpyext/test/test_unicodeobject.py +++ b/pypy/module/cpyext/test/test_unicodeobject.py @@ -307,6 +307,18 @@ assert module.utf8('xyz') == b'xyz' assert module.utf8('café') == 'café'.encode('utf-8') + def test_ready(self): + module = self.import_extension('foo', [ + ("unsafe_len", "METH_O", + """ + Py_ssize_t size = PyUnicode_GET_LENGTH(args); + return PyLong_FromSsize_t(size); + """)]) + assert module.unsafe_len(u"abc") == 3 + assert module.unsafe_len(u"café") == 4 + assert module.unsafe_len(u'aАbБcСdД') == 8 + assert module.unsafe_len(u"café\U0001F4A9") == 5 + class TestUnicode(BaseApiTest): def test_unicodeobject(self, space): @@ -863,30 +875,31 @@ assert space.unwrap(PyUnicode_Substring(space, w_str, 4, 3)) == u"" def test_Ready(self, space): - w_str = space.wrap(u'abc') # ASCII - py_str = as_pyobj(space, w_str) + def as_py_uni(val): + py_obj = new_empty_unicode(space, len(val)) + set_wbuffer(py_obj, rffi.unicode2wcharp(val)) + return py_obj + + py_str = as_py_uni(u'abc') # ASCII assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 1 assert get_ascii(py_str) == 1 - w_str = space.wrap(u'café') # latin1 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'café') # latin1 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 1 assert get_ascii(py_str) == 0 - w_str = space.wrap(u'Росси́я') # UCS2 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'Росси́я') # UCS2 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 2 assert get_ascii(py_str) == 0 - w_str = space.wrap(u'***\U0001f4a9***') # UCS4 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'***\U0001f4a9***') # UCS4 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 4 assert get_ascii(py_str) == 0 diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -70,6 +70,7 @@ "Fills a newly allocated PyUnicodeObject with a unicode string" set_wsize(py_obj, len(space.unicode_w(w_obj))) set_wbuffer(py_obj, lltype.nullptr(rffi.CWCHARP.TO)) + _readify(space, py_obj, w_obj._value) def unicode_realize(space, py_obj): """ @@ -265,8 +266,11 @@ assert isinstance(w_obj, unicodeobject.W_UnicodeObject) py_obj = as_pyobj(space, w_obj) assert get_kind(py_obj) == WCHAR_KIND + return _readify(space, py_obj, w_obj._value) + +def _readify(space, py_obj, value): maxchar = 0 - for c in w_obj._value: + for c in value: if ord(c) > maxchar: maxchar = ord(c) if maxchar > MAX_UNICODE: @@ -275,7 +279,7 @@ maxchar) if maxchar < 256: ucs1_data = rffi.str2charp(unicode_encode_latin_1( - w_obj._value, len(w_obj._value), errors='strict')) + value, len(value), errors='strict')) set_data(py_obj, cts.cast('void*', ucs1_data)) set_kind(py_obj, _1BYTE_KIND) set_len(py_obj, get_wsize(py_obj)) @@ -290,7 +294,7 @@ elif maxchar < 65536: # XXX: assumes that sizeof(wchar_t) == 4 ucs2_str = unicode_encode_utf_16_helper( - w_obj._value, len(w_obj._value), errors='strict', + value, len(value), errors='strict', byteorder=runicode.BYTEORDER) ucs2_data = cts.cast('Py_UCS2 *', rffi.str2charp(ucs2_str)) set_data(py_obj, cts.cast('void*', ucs2_data)) @@ -302,9 +306,8 @@ # XXX: assumes that sizeof(wchar_t) == 4 if not get_wbuffer(py_obj): # Copy unicode buffer - u = w_obj._value - set_wbuffer(py_obj, rffi.unicode2wcharp(u)) - set_wsize(py_obj, len(u)) + set_wbuffer(py_obj, rffi.unicode2wcharp(value)) + set_wsize(py_obj, len(value)) ucs4_data = get_wbuffer(py_obj) set_data(py_obj, cts.cast('void*', ucs4_data)) set_len(py_obj, get_wsize(py_obj)) @@ -332,9 +335,7 @@ w_res = utf_32_decode(space, value, w_final=space.w_False) else: raise oefmt(space.w_SystemError, "invalid kind") - w_ret = space.unpackiterable(w_res)[0] - _PyUnicode_Ready(space, w_ret) - return w_ret + return space.unpackiterable(w_res)[0] @cts.decl("Py_UNICODE * PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)") def PyUnicode_AsUnicodeAndSize(space, ref, psize): From pypy.commits at gmail.com Sat Oct 7 11:58:32 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 08:58:32 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: fix translation Message-ID: <59d8f9a8.138a1c0a.7e4b1.0378@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92636:3f5373832c7a Date: 2017-10-07 17:57 +0200 http://bitbucket.org/pypy/pypy/changeset/3f5373832c7a/ Log: fix translation diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -68,9 +68,10 @@ def unicode_attach(space, py_obj, w_obj, w_userdata=None): "Fills a newly allocated PyUnicodeObject with a unicode string" - set_wsize(py_obj, len(space.unicode_w(w_obj))) + value = space.unicode_w(w_obj) + set_wsize(py_obj, len(value) set_wbuffer(py_obj, lltype.nullptr(rffi.CWCHARP.TO)) - _readify(space, py_obj, w_obj._value) + _readify(space, py_obj, value) def unicode_realize(space, py_obj): """ From pypy.commits at gmail.com Sat Oct 7 12:10:13 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 09:10:13 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: oops Message-ID: <59d8fc65.90d71c0a.45829.0534@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92637:093af3bb83a7 Date: 2017-10-07 18:09 +0200 http://bitbucket.org/pypy/pypy/changeset/093af3bb83a7/ Log: oops diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -69,7 +69,7 @@ def unicode_attach(space, py_obj, w_obj, w_userdata=None): "Fills a newly allocated PyUnicodeObject with a unicode string" value = space.unicode_w(w_obj) - set_wsize(py_obj, len(value) + set_wsize(py_obj, len(value)) set_wbuffer(py_obj, lltype.nullptr(rffi.CWCHARP.TO)) _readify(space, py_obj, value) From pypy.commits at gmail.com Sat Oct 7 12:26:55 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 09:26:55 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: first attempt at find Message-ID: <59d9004f.298fdf0a.664c4.c88c@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92638:3ff5e711f1a0 Date: 2017-10-07 18:26 +0200 http://bitbucket.org/pypy/pypy/changeset/3ff5e711f1a0/ Log: first attempt at find diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -117,6 +117,7 @@ def _new_from_list(self, value): u = u''.join(value) return W_UnicodeObject(u.encode('utf8'), len(u)) + def _empty(self): return W_UnicodeObject.EMPTY @@ -412,6 +413,23 @@ "character mapping must be in range(0x110000)") return W_UnicodeObject(result.build(), result_length) + def descr_find(self, space, w_sub, w_start=None, w_end=None): + value, start, end, ofs = self._convert_idx_params(space, w_start, w_end) + + w_sub = self.convert_arg_to_w_unicode(space, w_sub) + # XXX for now just create index + storage = self._get_index_storage() + start_index = rutf8.codepoint_position_at_index(self._utf8, storage, + start) + end_index = rutf8.codepoint_position_at_index(self._utf8, storage, end) + + res_index = value.find(w_sub._utf8, start_index, end_index) + if res_index == -1: + return space.newint(-1) + + res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + return space.newint(res) + def descr_encode(self, space, w_encoding=None, w_errors=None): encoding, errors = _get_encoding_and_errors(space, w_encoding, w_errors) @@ -732,14 +750,17 @@ descr_rmul = descr_mul + def _get_index_storage(self): + if self._index_storage == rutf8.null_storage(): + self._index_storage = rutf8.create_utf8_index_storage(self._utf8, + self._length) + return self._index_storage + def _getitem_result(self, space, index): if index >= self._length: raise oefmt(space.w_IndexError, "string index out of range") - if self._index_storage == rutf8.null_storage(): - self._index_storage = rutf8.create_utf8_index_storage(self._utf8, - self._length) - start = rutf8.codepoint_position_at_index(self._utf8, - self._index_storage, index) + storage = self._get_index_storage() + start = rutf8.codepoint_position_at_index(self._utf8, storage, index) end = rutf8.next_codepoint_pos(self._utf8, start) return W_UnicodeObject(self._utf8[start:end], 1) diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -290,16 +290,18 @@ #@jit.elidable -def check_utf8(s, allow_surrogates=False): +def check_utf8(s, allow_surrogates=False, force_len=-1): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. Note that surrogates are not handled specially here. """ - import pdb - pdb.set_trace() pos = 0 continuation_bytes = 0 - while pos < len(s): + if force_len == -1: + end = len(s) + else: + end = force_len + while pos < end: ordch1 = ord(s[pos]) pos += 1 # fast path for ASCII @@ -310,7 +312,7 @@ raise CheckError(pos - 1) if ordch1 <= 0xDF: - if pos >= len(s): + if pos >= end: raise CheckError(pos - 1) ordch2 = ord(s[pos]) pos += 1 @@ -322,7 +324,7 @@ continue if ordch1 <= 0xEF: - if (pos + 2) > len(s): + if (pos + 2) > end: raise CheckError(pos - 1) ordch2 = ord(s[pos]) ordch3 = ord(s[pos + 1]) @@ -336,7 +338,7 @@ continue if ordch1 <= 0xF4: - if (pos + 3) > len(s): + if (pos + 3) > end: raise CheckError(pos - 1) ordch2 = ord(s[pos]) ordch3 = ord(s[pos + 1]) @@ -353,7 +355,7 @@ raise CheckError(pos - 1) - assert pos == len(s) + assert pos == end return pos - continuation_bytes @jit.elidable From pypy.commits at gmail.com Sat Oct 7 12:38:35 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 09:38:35 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fastpath for find() without any args, also mention in TODO Message-ID: <59d9030b.551d1c0a.10035.7dcc@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92639:fcee89117042 Date: 2017-10-07 18:33 +0200 http://bitbucket.org/pypy/pypy/changeset/fcee89117042/ Log: fastpath for find() without any args, also mention in TODO diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -1,3 +1,5 @@ * unskip tests in test_unicodeobject.py * rutf8.prev_codepoint_pos should use r_uint -* elidable in rutf8.check_utf8, WTF is wrong with that \ No newline at end of file +* elidable in rutf8.check_utf8, WTF is wrong with that +* find a better way to run "find" without creating the index storage, + if one is not already readily available \ No newline at end of file diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -418,10 +418,16 @@ w_sub = self.convert_arg_to_w_unicode(space, w_sub) # XXX for now just create index - storage = self._get_index_storage() - start_index = rutf8.codepoint_position_at_index(self._utf8, storage, - start) - end_index = rutf8.codepoint_position_at_index(self._utf8, storage, end) + start_index = 0 + end_index = len(self._utf8) + if start > 0 or end != self._length: + storage = self._get_index_storage() + if start > 0: + start_index = rutf8.codepoint_position_at_index(self._utf8, + storage, start) + if end != self.length: + end_index = rutf8.codepoint_position_at_index(self._utf8, + storage, end) res_index = value.find(w_sub._utf8, start_index, end_index) if res_index == -1: From pypy.commits at gmail.com Sat Oct 7 12:38:37 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 09:38:37 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fix Message-ID: <59d9030d.f586df0a.c427c.f851@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92640:a90ccea247f0 Date: 2017-10-07 18:37 +0200 http://bitbucket.org/pypy/pypy/changeset/a90ccea247f0/ Log: fix diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -23,6 +23,7 @@ from pypy.objspace.std.formatting import mod_format from pypy.objspace.std.stringmethods import StringMethods from pypy.objspace.std.util import IDTAG_SPECIAL, IDTAG_SHIFT +from pypy.objspace.std.sliceobject import unwrap_start_stop __all__ = ['W_UnicodeObject', 'wrapunicode', 'plain_str2unicode', 'encode_object', 'decode_object', 'unicode_from_object', @@ -414,7 +415,7 @@ return W_UnicodeObject(result.build(), result_length) def descr_find(self, space, w_sub, w_start=None, w_end=None): - value, start, end, ofs = self._convert_idx_params(space, w_start, w_end) + start, end = unwrap_start_stop(space, self._length, w_start, w_end) w_sub = self.convert_arg_to_w_unicode(space, w_sub) # XXX for now just create index @@ -429,7 +430,7 @@ end_index = rutf8.codepoint_position_at_index(self._utf8, storage, end) - res_index = value.find(w_sub._utf8, start_index, end_index) + res_index = self._utf8.find(w_sub._utf8, start_index, end_index) if res_index == -1: return space.newint(-1) From pypy.commits at gmail.com Sat Oct 7 12:43:05 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 09:43:05 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: is this code never executed in tests? Message-ID: <59d90419.c7c4df0a.668f3.875f@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92641:0013993b27a1 Date: 2017-10-07 18:42 +0200 http://bitbucket.org/pypy/pypy/changeset/0013993b27a1/ Log: is this code never executed in tests? diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -426,7 +426,7 @@ if start > 0: start_index = rutf8.codepoint_position_at_index(self._utf8, storage, start) - if end != self.length: + if end != self._length: end_index = rutf8.codepoint_position_at_index(self._utf8, storage, end) From pypy.commits at gmail.com Sat Oct 7 12:53:40 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 07 Oct 2017 09:53:40 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: actually test stuff Message-ID: <59d90694.298fdf0a.664c4.d66e@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92642:dbc8e7ddf1ed Date: 2017-10-07 18:52 +0200 http://bitbucket.org/pypy/pypy/changeset/dbc8e7ddf1ed/ Log: actually test stuff diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1,4 +1,5 @@ +# -*- encoding: utf-8 -*- import py import sys @@ -725,7 +726,9 @@ assert u'abc'.rfind('', 4) == -1 def test_find_index_str_unicode(self): - assert 'abcdefghiabc'.find(u'bc') == 1 + assert u'abcdefghiabc'.find(u'bc') == 1 + assert u'ab\u0105b\u0107'.find('b', 2) == 3 + assert u'ab\u0105b\u0107'.find('b', 0, 1) == -1 assert 'abcdefghiabc'.rfind(u'abc') == 9 raises(UnicodeDecodeError, '\x80'.find, u'') raises(UnicodeDecodeError, '\x80'.rfind, u'') From pypy.commits at gmail.com Sat Oct 7 16:28:29 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 13:28:29 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Merged in Dodan/pypy_sendmsg_patch/py3.5 (pull request #569) Message-ID: <59d938ed.c5a4df0a.6609c.dbc9@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92645:ee5f3be5fea2 Date: 2017-10-07 20:27 +0000 http://bitbucket.org/pypy/pypy/changeset/ee5f3be5fea2/ Log: Merged in Dodan/pypy_sendmsg_patch/py3.5 (pull request #569) Fix issues #2648 and #2649 diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -574,6 +574,22 @@ import _socket raises(_socket.error, _socket.dup, 123456) + def test_recvmsg_issue2649(self): + import _socket as socket + listener = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + listener.bind(('::1', 1234)) + + s = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + IPV6_RECVERR = 25 + s.setsockopt(socket.IPPROTO_IPV6, IPV6_RECVERR, 1) + + s.sendto(b'x', ('::1', 1234)) + try: + queue = s.recvmsg(1024, 1024, socket.MSG_ERRQUEUE) + except BlockingIOError as e: + assert True + def test_buffer(self): # Test that send/sendall/sendto accept a buffer as arg import _socket, os diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py --- a/rpython/rlib/_rsocket_rffi.py +++ b/rpython/rlib/_rsocket_rffi.py @@ -162,7 +162,7 @@ IP_RECVRETOPTS IP_RETOPTS IP_TOS IP_TTL MSG_BTAG MSG_ETAG MSG_CTRUNC MSG_DONTROUTE MSG_DONTWAIT MSG_EOR MSG_OOB -MSG_PEEK MSG_TRUNC MSG_WAITALL +MSG_PEEK MSG_TRUNC MSG_WAITALL MSG_ERRQUEUE NI_DGRAM NI_MAXHOST NI_MAXSERV NI_NAMEREQD NI_NOFQDN NI_NUMERICHOST NI_NUMERICSERV @@ -535,7 +535,7 @@ int cmsg_status; struct iovec iov; struct recvmsg_info* retinfo; - int error_flag; // variable to be set in case of special errors. + int error_flag = 0; // variable to be set in case of special errors. int cmsgdatalen = 0; // variables that are set to 1, if the message charp has been allocated @@ -709,6 +709,7 @@ free(retinfo); } } + if (error_flag==0) error_flag = -1; return error_flag; err_closefds: diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py --- a/rpython/rlib/rsocket.py +++ b/rpython/rlib/rsocket.py @@ -1074,7 +1074,7 @@ if address is not None: address.unlock() - if _c.geterrno() == _c.EINTR: + if (_c.geterrno() == _c.EINTR) or (_c.geterrno() == 11): raise last_error() if (reply == -10000): raise RSocketError("Invalid message size") From pypy.commits at gmail.com Sat Oct 7 16:28:38 2017 From: pypy.commits at gmail.com (miha...@gmail.com) Date: Sat, 07 Oct 2017 13:28:38 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Recverr hotfix Message-ID: <59d938f6.5282df0a.fb159.3de2@mx.google.com> Author: mihai.dodan at gmail.com Branch: py3.5 Changeset: r92643:66e8282940aa Date: 2017-09-25 18:18 +0300 http://bitbucket.org/pypy/pypy/changeset/66e8282940aa/ Log: Recverr hotfix diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py --- a/rpython/rlib/_rsocket_rffi.py +++ b/rpython/rlib/_rsocket_rffi.py @@ -534,7 +534,7 @@ int cmsg_status; struct iovec iov; struct recvmsg_info* retinfo; - int error_flag; // variable to be set in case of special errors. + int error_flag = 0; // variable to be set in case of special errors. int cmsgdatalen = 0; // variables that are set to 1, if the message charp has been allocated @@ -708,6 +708,7 @@ free(retinfo); } } + if (error_flag==0) error_flag = -1; return error_flag; err_closefds: diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py --- a/rpython/rlib/rsocket.py +++ b/rpython/rlib/rsocket.py @@ -1074,7 +1074,7 @@ if address is not None: address.unlock() - if _c.geterrno() == _c.EINTR: + if (_c.geterrno() == _c.EINTR) or (_c.geterrno() == 11): raise last_error() if (reply == -10000): raise RSocketError("Invalid message size") From pypy.commits at gmail.com Sat Oct 7 16:28:40 2017 From: pypy.commits at gmail.com (miha...@gmail.com) Date: Sat, 07 Oct 2017 13:28:40 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix for issue 2648 and 2649 Message-ID: <59d938f8.b796df0a.d9abe.6504@mx.google.com> Author: mihai.dodan at gmail.com Branch: py3.5 Changeset: r92644:0a45774261bb Date: 2017-09-26 16:44 +0300 http://bitbucket.org/pypy/pypy/changeset/0a45774261bb/ Log: Fix for issue 2648 and 2649 diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -574,6 +574,22 @@ import _socket raises(_socket.error, _socket.dup, 123456) + def test_recvmsg_issue2649(self): + import _socket as socket + listener = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + listener.bind(('::1', 1234)) + + s = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + IPV6_RECVERR = 25 + s.setsockopt(socket.IPPROTO_IPV6, IPV6_RECVERR, 1) + + s.sendto(b'x', ('::1', 1234)) + try: + queue = s.recvmsg(1024, 1024, socket.MSG_ERRQUEUE) + except BlockingIOError as e: + assert True + def test_buffer(self): # Test that send/sendall/sendto accept a buffer as arg import _socket, os diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py --- a/rpython/rlib/_rsocket_rffi.py +++ b/rpython/rlib/_rsocket_rffi.py @@ -162,7 +162,7 @@ IP_RECVRETOPTS IP_RETOPTS IP_TOS IP_TTL MSG_BTAG MSG_ETAG MSG_CTRUNC MSG_DONTROUTE MSG_DONTWAIT MSG_EOR MSG_OOB -MSG_PEEK MSG_TRUNC MSG_WAITALL +MSG_PEEK MSG_TRUNC MSG_WAITALL MSG_ERRQUEUE NI_DGRAM NI_MAXHOST NI_MAXSERV NI_NAMEREQD NI_NOFQDN NI_NUMERICHOST NI_NUMERICSERV From pypy.commits at gmail.com Sat Oct 7 16:56:53 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 07 Oct 2017 13:56:53 -0700 (PDT) Subject: [pypy-commit] pypy default: Merged in loganchien/pypy (pull request #570) Message-ID: <59d93f95.43b9df0a.1e711.245a@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92647:f3b1730b5499 Date: 2017-10-07 20:56 +0000 http://bitbucket.org/pypy/pypy/changeset/f3b1730b5499/ Log: Merged in loganchien/pypy (pull request #570) fix the PYPYLOG link in rpython doc diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: From pypy.commits at gmail.com Sat Oct 7 16:57:02 2017 From: pypy.commits at gmail.com (loganchien) Date: Sat, 07 Oct 2017 13:57:02 -0700 (PDT) Subject: [pypy-commit] pypy default: fix the PYPYLOG link in rpython doc Message-ID: <59d93f9e.d28bdf0a.4de21.b7e4@mx.google.com> Author: Logan Chien Branch: Changeset: r92646:ecd865bf542a Date: 2017-10-04 22:05 +0800 http://bitbucket.org/pypy/pypy/changeset/ecd865bf542a/ Log: fix the PYPYLOG link in rpython doc diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: From pypy.commits at gmail.com Sun Oct 8 02:22:37 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 23:22:37 -0700 (PDT) Subject: [pypy-commit] pypy default: Issue #2676 Message-ID: <59d9c42d.2587df0a.e5230.d164@mx.google.com> Author: Armin Rigo Branch: Changeset: r92648:08b4e5ae6bfa Date: 2017-10-08 08:21 +0200 http://bitbucket.org/pypy/pypy/changeset/08b4e5ae6bfa/ Log: Issue #2676 Seems that Ubuntu 16.04 should have the same "new" package list. diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ From pypy.commits at gmail.com Sun Oct 8 02:22:40 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 23:22:40 -0700 (PDT) Subject: [pypy-commit] pypy default: merge heads Message-ID: <59d9c430.4f861c0a.d22d9.d2bb@mx.google.com> Author: Armin Rigo Branch: Changeset: r92649:aedd9ba91c12 Date: 2017-10-08 08:22 +0200 http://bitbucket.org/pypy/pypy/changeset/aedd9ba91c12/ Log: merge heads diff --git a/pypy/module/_cppyy/test/conftest.py b/pypy/module/_cppyy/test/conftest.py --- a/pypy/module/_cppyy/test/conftest.py +++ b/pypy/module/_cppyy/test/conftest.py @@ -4,7 +4,7 @@ def pytest_runtest_setup(item): if py.path.local.sysfind('genreflex') is None: import pypy.module._cppyy.capi.loadable_capi as lcapi - if 'dummy' in lcapi.reflection_library: + if 'dummy' in lcapi.backend_library: # run only tests that are covered by the dummy backend and tests # that do not rely on reflex import os @@ -33,7 +33,7 @@ import pypy.module._cppyy.capi.loadable_capi as lcapi try: import ctypes - ctypes.CDLL(lcapi.reflection_library) + ctypes.CDLL(lcapi.backend_library) except Exception as e: if config.option.runappdirect: return # "can't run dummy tests in -A" @@ -71,4 +71,4 @@ return raise - lcapi.reflection_library = str(soname) + lcapi.backend_library = str(soname) diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: From pypy.commits at gmail.com Sun Oct 8 02:50:17 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 23:50:17 -0700 (PDT) Subject: [pypy-commit] cffi default: Bump version number to 1.11.2 Message-ID: <59d9caa9.2587df0a.e5230.d8b1@mx.google.com> Author: Armin Rigo Branch: Changeset: r3045:d0a2021a388d Date: 2017-10-08 08:50 +0200 http://bitbucket.org/cffi/cffi/changeset/d0a2021a388d/ Log: Bump version number to 1.11.2 diff --git a/c/_cffi_backend.c b/c/_cffi_backend.c --- a/c/_cffi_backend.c +++ b/c/_cffi_backend.c @@ -2,7 +2,7 @@ #include #include "structmember.h" -#define CFFI_VERSION "1.11.1" +#define CFFI_VERSION "1.11.2" #ifdef MS_WIN32 #include diff --git a/c/test_c.py b/c/test_c.py --- a/c/test_c.py +++ b/c/test_c.py @@ -12,7 +12,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/cffi/__init__.py b/cffi/__init__.py --- a/cffi/__init__.py +++ b/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/cffi/_embedding.h b/cffi/_embedding.h --- a/cffi/_embedding.h +++ b/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/doc/source/conf.py b/doc/source/conf.py --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -47,7 +47,7 @@ # The short X.Y version. version = '1.11' # The full version, including alpha/beta/rc tags. -release = '1.11.1' +release = '1.11.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/doc/source/installation.rst b/doc/source/installation.rst --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -53,13 +53,13 @@ * https://pypi.python.org/pypi/cffi -* Checksums of the "source" package version 1.11.1: +* Checksums of the "source" package version 1.11.2: - - MD5: 89dfa1bb3b3e28eafb5da3e345a19f56 + - MD5: ... - - SHA: f3a61a3b53b482a7d05704344e0f055d4672fa01 + - SHA: ... - - SHA256: 4c40817cc0f71b5351eb0bdd0b585db4a285c2bcc03fbcb961b79bb8086b7576 + - SHA256: ... * Or grab the most current version from the `Bitbucket page`_: ``hg clone https://bitbucket.org/cffi/cffi`` diff --git a/doc/source/whatsnew.rst b/doc/source/whatsnew.rst --- a/doc/source/whatsnew.rst +++ b/doc/source/whatsnew.rst @@ -3,6 +3,12 @@ ====================== +v1.11.2 +======= + +* Fix Windows issue with managing the thread-state on CPython 3.0 to 3.5 + + v1.11.1 ======= diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -186,7 +186,7 @@ `Mailing list `_ """, - version='1.11.1', + version='1.11.2', packages=['cffi'] if cpython else [], package_data={'cffi': ['_cffi_include.h', 'parse_c_type.h', '_embedding.h', '_cffi_errors.h']} From pypy.commits at gmail.com Sun Oct 8 02:50:20 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 23:50:20 -0700 (PDT) Subject: [pypy-commit] cffi release-1.11: hg merge default Message-ID: <59d9caac.51421c0a.a6174.3859@mx.google.com> Author: Armin Rigo Branch: release-1.11 Changeset: r3046:59ead014cdee Date: 2017-10-08 08:50 +0200 http://bitbucket.org/cffi/cffi/changeset/59ead014cdee/ Log: hg merge default diff --git a/c/_cffi_backend.c b/c/_cffi_backend.c --- a/c/_cffi_backend.c +++ b/c/_cffi_backend.c @@ -2,7 +2,7 @@ #include #include "structmember.h" -#define CFFI_VERSION "1.11.1" +#define CFFI_VERSION "1.11.2" #ifdef MS_WIN32 #include diff --git a/c/misc_thread_common.h b/c/misc_thread_common.h --- a/c/misc_thread_common.h +++ b/c/misc_thread_common.h @@ -62,33 +62,35 @@ #endif -/* Seems that CPython 3.5.1 made our job harder. Did not find out how - to do that without these hacks. We can't use PyThreadState_GET(), - because that calls PyThreadState_Get() which fails an assert if the - result is NULL. We can use _PyThreadState_UncheckedGet() from 3.6, - though. It was added in 3.5.2 but should never be used in 3.5.x - because it is not available in 3.5.0 or 3.5.1. */ -#if PY_VERSION_HEX >= 0x03060000 +/* MESS. We can't use PyThreadState_GET(), because that calls + PyThreadState_Get() which fails an assert if the result is NULL. + + * in Python 2.7 and <= 3.4, the variable _PyThreadState_Current + is directly available, so use that. + + * in Python 3.5, the variable is available too, but it might be + the case that the headers don't define it (this changed in 3.5.1). + In case we're compiling with 3.5.x with x >= 1, we need to + manually define this variable. + + * in Python >= 3.6 there is _PyThreadState_UncheckedGet(). + It was added in 3.5.2 but should never be used in 3.5.x + because it is not available in 3.5.0 or 3.5.1. +*/ +#if PY_VERSION_HEX >= 0x03050100 && PY_VERSION_HEX < 0x03060000 +PyAPI_DATA(void *volatile) _PyThreadState_Current; +#endif + static PyThreadState *get_current_ts(void) { +#if PY_VERSION_HEX >= 0x03060000 return _PyThreadState_UncheckedGet(); +#elif defined(_Py_atomic_load_relaxed) + return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current); +#else + return (PyThreadState*)_PyThreadState_Current; /* assume atomic read */ +#endif } -#else -# if PY_MAJOR_VERSION >= 3 && !defined(_Py_atomic_load_relaxed) - /* this was abruptly un-defined in 3.5.1 */ -extern void *volatile _PyThreadState_Current; - /* XXX simple volatile access is assumed atomic */ -# define _Py_atomic_load_relaxed(pp) (*(pp)) -# endif -static PyThreadState *get_current_ts(void) -{ -# if defined(_Py_atomic_load_relaxed) - return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current); -# else - return _PyThreadState_Current; -# endif -} -#endif static PyGILState_STATE gil_ensure(void) { diff --git a/c/test_c.py b/c/test_c.py --- a/c/test_c.py +++ b/c/test_c.py @@ -12,7 +12,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/cffi/__init__.py b/cffi/__init__.py --- a/cffi/__init__.py +++ b/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/cffi/_embedding.h b/cffi/_embedding.h --- a/cffi/_embedding.h +++ b/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/doc/source/conf.py b/doc/source/conf.py --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -47,7 +47,7 @@ # The short X.Y version. version = '1.11' # The full version, including alpha/beta/rc tags. -release = '1.11.1' +release = '1.11.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/doc/source/installation.rst b/doc/source/installation.rst --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -53,13 +53,13 @@ * https://pypi.python.org/pypi/cffi -* Checksums of the "source" package version 1.11.1: +* Checksums of the "source" package version 1.11.2: - - MD5: 89dfa1bb3b3e28eafb5da3e345a19f56 + - MD5: ... - - SHA: f3a61a3b53b482a7d05704344e0f055d4672fa01 + - SHA: ... - - SHA256: 4c40817cc0f71b5351eb0bdd0b585db4a285c2bcc03fbcb961b79bb8086b7576 + - SHA256: ... * Or grab the most current version from the `Bitbucket page`_: ``hg clone https://bitbucket.org/cffi/cffi`` diff --git a/doc/source/whatsnew.rst b/doc/source/whatsnew.rst --- a/doc/source/whatsnew.rst +++ b/doc/source/whatsnew.rst @@ -3,6 +3,12 @@ ====================== +v1.11.2 +======= + +* Fix Windows issue with managing the thread-state on CPython 3.0 to 3.5 + + v1.11.1 ======= diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -186,7 +186,7 @@ `Mailing list `_ """, - version='1.11.1', + version='1.11.2', packages=['cffi'] if cpython else [], package_data={'cffi': ['_cffi_include.h', 'parse_c_type.h', '_embedding.h', '_cffi_errors.h']} From pypy.commits at gmail.com Sun Oct 8 02:54:19 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 07 Oct 2017 23:54:19 -0700 (PDT) Subject: [pypy-commit] pypy default: Bump cffi version to 1.11.2 Message-ID: <59d9cb9b.c23a1c0a.a1b02.516b@mx.google.com> Author: Armin Rigo Branch: Changeset: r92650:16f298b6276f Date: 2017-10-08 08:50 +0200 http://bitbucket.org/pypy/pypy/changeset/16f298b6276f/ Log: Bump cffi version to 1.11.2 diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py --- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py +++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py @@ -37,7 +37,7 @@ v = cffi.__version__.replace('+', '') p = os.path.join(parent, 'doc', 'source', 'installation.rst') content = open(p).read() - assert ("/cffi-%s.tar.gz" % v) in content + assert (" package version %s:" % v) in content def test_setup_version(): parent = os.path.dirname(os.path.dirname(cffi.__file__)) From pypy.commits at gmail.com Sun Oct 8 03:40:41 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 08 Oct 2017 00:40:41 -0700 (PDT) Subject: [pypy-commit] pypy default: Merged in cjrh/pypy/docs-osx-brew-openssl (pull request #572) Message-ID: <59d9d679.ccabdf0a.c440.336e@mx.google.com> Author: Armin Rigo Branch: Changeset: r92652:38a0a4871739 Date: 2017-10-08 07:40 +0000 http://bitbucket.org/pypy/pypy/changeset/38a0a4871739/ Log: Merged in cjrh/pypy/docs-osx-brew-openssl (pull request #572) DOCS: For translation on MacOS, get OpenSSL from Homebrew and set env var. diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. From pypy.commits at gmail.com Sun Oct 8 03:40:49 2017 From: pypy.commits at gmail.com (cjrh) Date: Sun, 08 Oct 2017 00:40:49 -0700 (PDT) Subject: [pypy-commit] pypy docs-osx-brew-openssl: DOC: For translation on MacOS, get OpenSSL from Homebrew and set env var. Message-ID: <59d9d681.8e861c0a.873f.cf0b@mx.google.com> Author: Caleb Hattingh Branch: docs-osx-brew-openssl Changeset: r92651:9eb4dfce15c7 Date: 2017-10-08 14:57 +1000 http://bitbucket.org/pypy/pypy/changeset/9eb4dfce15c7/ Log: DOC: For translation on MacOS, get OpenSSL from Homebrew and set env var. diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. From pypy.commits at gmail.com Sun Oct 8 06:01:49 2017 From: pypy.commits at gmail.com (jerith) Date: Sun, 08 Oct 2017 03:01:49 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: getslice and fix {starts, ends}with. Message-ID: <59d9f78d.90921c0a.3f8a8.2976@mx.google.com> Author: Jeremy Thurgood Branch: unicode-utf8 Changeset: r92653:c46107c45a6e Date: 2017-10-08 12:01 +0200 http://bitbucket.org/pypy/pypy/changeset/c46107c45a6e/ Log: getslice and fix {starts,ends}with. diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py --- a/pypy/objspace/std/stringmethods.py +++ b/pypy/objspace/std/stringmethods.py @@ -619,6 +619,7 @@ strs.append(value[pos:length]) return self._newlist_unwrapped(space, strs) + # This is overridden in unicodeobject, _startswith_tuple is not. def descr_startswith(self, space, w_prefix, w_start=None, w_end=None): value, start, end, _ = self._convert_idx_params(space, w_start, w_end) if space.isinstance_w(w_prefix, space.w_tuple): @@ -632,13 +633,14 @@ return space.w_True return space.w_False - # This is overridden in unicodeobject, but the two above are not. + # This is overridden in unicodeobject, _startswith_tuple is not. def _startswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) if start > len(value): return False return startswith(value, prefix, start, end) + # This is overridden in unicodeobject, _endswith_tuple is not. def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): value, start, end, _ = self._convert_idx_params(space, w_start, w_end) if space.isinstance_w(w_suffix, space.w_tuple): @@ -652,7 +654,7 @@ return space.w_True return space.w_False - # This is overridden in unicodeobject, but the two above are not. + # This is overridden in unicodeobject, but _endswith_tuple is not. def _endswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) if start > len(value): diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -21,9 +21,10 @@ from pypy.objspace.std import newformat from pypy.objspace.std.basestringtype import basestring_typedef from pypy.objspace.std.formatting import mod_format +from pypy.objspace.std.sliceobject import ( + unwrap_start_stop, normalize_simple_slice) from pypy.objspace.std.stringmethods import StringMethods from pypy.objspace.std.util import IDTAG_SPECIAL, IDTAG_SHIFT -from pypy.objspace.std.sliceobject import unwrap_start_stop __all__ = ['W_UnicodeObject', 'wrapunicode', 'plain_str2unicode', 'encode_object', 'decode_object', 'unicode_from_object', @@ -142,10 +143,6 @@ return True @staticmethod - def _op_utf8(space, w_other, strict=None): - return W_UnicodeObject.convert_arg_to_w_unicode(space, w_other, strict)._utf8 - - @staticmethod def _op_val(space, w_other, strict=None): return W_UnicodeObject.convert_arg_to_w_unicode(space, w_other, strict)._utf8.decode('utf8') @@ -550,14 +547,30 @@ i = rutf8.next_codepoint_pos(val, i) return space.newbool(cased) + def descr_startswith(self, space, w_prefix, w_start=None, w_end=None): + start, end = unwrap_start_stop(space, self._length, w_start, w_end) + value = self._utf8 + if space.isinstance_w(w_prefix, space.w_tuple): + return self._startswith_tuple(space, value, w_prefix, start, end) + return space.newbool(self._startswith(space, value, w_prefix, start, + end)) + def _startswith(self, space, value, w_prefix, start, end): - prefix = self._op_utf8(space, w_prefix) + prefix = self.convert_arg_to_w_unicode(space, w_prefix)._utf8 if start > len(value): return len(prefix) == 0 return startswith(value, prefix, start, end) + def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): + start, end = unwrap_start_stop(space, self._length, w_start, w_end) + value = self._utf8 + if space.isinstance_w(w_suffix, space.w_tuple): + return self._endswith_tuple(space, value, w_suffix, start, end) + return space.newbool(self._endswith(space, value, w_suffix, start, + end)) + def _endswith(self, space, value, w_prefix, start, end): - prefix = self._op_utf8(space, w_prefix) + prefix = self.convert_arg_to_w_unicode(space, w_prefix)._utf8 if start > len(value): return len(prefix) == 0 return endswith(value, prefix, start, end) @@ -683,6 +696,15 @@ return space.newlist_utf8(res) + def descr_getslice(self, space, w_start, w_stop): + selfvalue = self._utf8 + start, stop = normalize_simple_slice( + space, self._len(), w_start, w_stop) + if start == stop: + return self._empty() + else: + return self._sliced(space, selfvalue, start, stop, self) + def descr_capitalize(self, space): value = self._utf8 if len(value) == 0: From pypy.commits at gmail.com Sun Oct 8 11:10:24 2017 From: pypy.commits at gmail.com (jerith) Date: Sun, 08 Oct 2017 08:10:24 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Some more unicode methods. Message-ID: <59da3fe0.97addf0a.92518.4d80@mx.google.com> Author: Jeremy Thurgood Branch: unicode-utf8 Changeset: r92654:3e28fa9641ac Date: 2017-10-08 17:09 +0200 http://bitbucket.org/pypy/pypy/changeset/3e28fa9641ac/ Log: Some more unicode methods. diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py --- a/pypy/objspace/std/stringmethods.py +++ b/pypy/objspace/std/stringmethods.py @@ -489,60 +489,44 @@ builder.append(self._lower(value[i])) return self._new(builder.build()) + # This is not used for W_UnicodeObject. def descr_partition(self, space, w_sub): from pypy.objspace.std.bytearrayobject import W_BytearrayObject value = self._val(space) - if self._use_rstr_ops(space, w_sub): - sub = self._op_val(space, w_sub) - sublen = len(sub) - if sublen == 0: - raise oefmt(space.w_ValueError, "empty separator") + sub = _get_buffer(space, w_sub) + sublen = sub.getlength() + if sublen == 0: + raise oefmt(space.w_ValueError, "empty separator") - pos = value.find(sub) - else: - sub = _get_buffer(space, w_sub) - sublen = sub.getlength() - if sublen == 0: - raise oefmt(space.w_ValueError, "empty separator") - - pos = find(value, sub, 0, len(value)) - if pos != -1 and isinstance(self, W_BytearrayObject): - w_sub = self._new_from_buffer(sub) + pos = find(value, sub, 0, len(value)) + if pos != -1 and isinstance(self, W_BytearrayObject): + w_sub = self._new_from_buffer(sub) if pos == -1: - if isinstance(self, W_BytearrayObject): - self = self._new(value) + self = self._new(value) return space.newtuple([self, self._empty(), self._empty()]) else: return space.newtuple( [self._sliced(space, value, 0, pos, self), w_sub, self._sliced(space, value, pos + sublen, len(value), self)]) + # This is not used for W_UnicodeObject. def descr_rpartition(self, space, w_sub): from pypy.objspace.std.bytearrayobject import W_BytearrayObject value = self._val(space) - if self._use_rstr_ops(space, w_sub): - sub = self._op_val(space, w_sub) - sublen = len(sub) - if sublen == 0: - raise oefmt(space.w_ValueError, "empty separator") + sub = _get_buffer(space, w_sub) + sublen = sub.getlength() + if sublen == 0: + raise oefmt(space.w_ValueError, "empty separator") - pos = value.rfind(sub) - else: - sub = _get_buffer(space, w_sub) - sublen = sub.getlength() - if sublen == 0: - raise oefmt(space.w_ValueError, "empty separator") - - pos = rfind(value, sub, 0, len(value)) - if pos != -1 and isinstance(self, W_BytearrayObject): - w_sub = self._new_from_buffer(sub) + pos = rfind(value, sub, 0, len(value)) + if pos != -1 and isinstance(self, W_BytearrayObject): + w_sub = self._new_from_buffer(sub) if pos == -1: - if isinstance(self, W_BytearrayObject): - self = self._new(value) + self = self._new(value) return space.newtuple([self._empty(), self._empty(), self]) else: return space.newtuple( diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -412,20 +412,9 @@ return W_UnicodeObject(result.build(), result_length) def descr_find(self, space, w_sub, w_start=None, w_end=None): - start, end = unwrap_start_stop(space, self._length, w_start, w_end) - w_sub = self.convert_arg_to_w_unicode(space, w_sub) - # XXX for now just create index - start_index = 0 - end_index = len(self._utf8) - if start > 0 or end != self._length: - storage = self._get_index_storage() - if start > 0: - start_index = rutf8.codepoint_position_at_index(self._utf8, - storage, start) - if end != self._length: - end_index = rutf8.codepoint_position_at_index(self._utf8, - storage, end) + start_index, end_index = self._unwrap_and_compute_idx_params( + space, w_start, w_end) res_index = self._utf8.find(w_sub._utf8, start_index, end_index) if res_index == -1: @@ -434,6 +423,44 @@ res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise return space.newint(res) + def descr_rfind(self, space, w_sub, w_start=None, w_end=None): + w_sub = self.convert_arg_to_w_unicode(space, w_sub) + start_index, end_index = self._unwrap_and_compute_idx_params( + space, w_start, w_end) + + res_index = self._utf8.rfind(w_sub._utf8, start_index, end_index) + if res_index == -1: + return space.newint(-1) + + res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + return space.newint(res) + + def descr_index(self, space, w_sub, w_start=None, w_end=None): + w_sub = self.convert_arg_to_w_unicode(space, w_sub) + start_index, end_index = self._unwrap_and_compute_idx_params( + space, w_start, w_end) + + res_index = self._utf8.find(w_sub._utf8, start_index, end_index) + if res_index == -1: + raise oefmt(space.w_ValueError, + "substring not found in string.index") + + res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + return space.newint(res) + + def descr_rindex(self, space, w_sub, w_start=None, w_end=None): + w_sub = self.convert_arg_to_w_unicode(space, w_sub) + start_index, end_index = self._unwrap_and_compute_idx_params( + space, w_start, w_end) + + res_index = self._utf8.rfind(w_sub._utf8, start_index, end_index) + if res_index == -1: + raise oefmt(space.w_ValueError, + "substring not found in string.rindex") + + res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + return space.newint(res) + def descr_encode(self, space, w_encoding=None, w_errors=None): encoding, errors = _get_encoding_and_errors(space, w_encoding, w_errors) @@ -548,7 +575,7 @@ return space.newbool(cased) def descr_startswith(self, space, w_prefix, w_start=None, w_end=None): - start, end = unwrap_start_stop(space, self._length, w_start, w_end) + start, end = self._unwrap_and_compute_idx_params(space, w_start, w_end) value = self._utf8 if space.isinstance_w(w_prefix, space.w_tuple): return self._startswith_tuple(space, value, w_prefix, start, end) @@ -562,7 +589,7 @@ return startswith(value, prefix, start, end) def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): - start, end = unwrap_start_stop(space, self._length, w_start, w_end) + start, end = self._unwrap_and_compute_idx_params(space, w_start, w_end) value = self._utf8 if space.isinstance_w(w_suffix, space.w_tuple): return self._endswith_tuple(space, value, w_suffix, start, end) @@ -739,11 +766,50 @@ return W_UnicodeObject(centered, self._len() + d) + def descr_count(self, space, w_sub, w_start=None, w_end=None): + value = self._utf8 + start_index, end_index = self._unwrap_and_compute_idx_params( + space, w_start, w_end) + sub = self.convert_arg_to_w_unicode(space, w_sub)._utf8 + return space.newint(value.count(sub, start_index, end_index)) + def descr_contains(self, space, w_sub): value = self._utf8 w_other = self.convert_arg_to_w_unicode(space, w_sub) return space.newbool(value.find(w_other._utf8) >= 0) + def descr_partition(self, space, w_sub): + value = self._utf8 + sub = self.convert_arg_to_w_unicode(space, w_sub) + sublen = sub._len() + if sublen == 0: + raise oefmt(space.w_ValueError, "empty separator") + + pos = value.find(sub._utf8) + + if pos == -1: + return space.newtuple([self, self._empty(), self._empty()]) + else: + return space.newtuple( + [self._sliced(space, value, 0, pos, self), w_sub, + self._sliced(space, value, pos + sublen, len(value), self)]) + + def descr_rpartition(self, space, w_sub): + value = self._utf8 + sub = self.convert_arg_to_w_unicode(space, w_sub) + sublen = sub._len() + if sublen == 0: + raise oefmt(space.w_ValueError, "empty separator") + + pos = value.rfind(sub._utf8) + + if pos == -1: + return space.newtuple([self._empty(), self._empty(), self]) + else: + return space.newtuple( + [self._sliced(space, value, 0, pos, self), w_sub, + self._sliced(space, value, pos + sublen, len(value), self)]) + @unwrap_spec(count=int) def descr_replace(self, space, w_old, w_new, count=-1): @@ -786,13 +852,38 @@ return self._index_storage def _getitem_result(self, space, index): - if index >= self._length: + if index < 0: + index += self._length + if index < 0 or index >= self._length: raise oefmt(space.w_IndexError, "string index out of range") storage = self._get_index_storage() start = rutf8.codepoint_position_at_index(self._utf8, storage, index) end = rutf8.next_codepoint_pos(self._utf8, start) return W_UnicodeObject(self._utf8[start:end], 1) + def _unwrap_and_compute_idx_params(self, space, w_start, w_end): + start, end = unwrap_start_stop(space, self._length, w_start, w_end) + # XXX for now just create index + start_index = 0 + end_index = len(self._utf8) + if start > 0 or end != self._length: + storage = self._get_index_storage() + if start > 0: + # :-( + if start > self._length: + start_index = start + else: + start_index = rutf8.codepoint_position_at_index( + self._utf8, storage, start) + if end != self._length: + # :-( + if end > self._length: + end_index = end + else: + end_index = rutf8.codepoint_position_at_index( + self._utf8, storage, end) + return (start_index, end_index) + @unwrap_spec(width=int, w_fillchar=WrappedDefault(' ')) def descr_rjust(self, space, width, w_fillchar): value = self._utf8 From pypy.commits at gmail.com Sun Oct 8 12:53:13 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 08 Oct 2017 09:53:13 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59da57f9.5ce81c0a.ce8dd.c313@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92655:6d880cc2b4d6 Date: 2017-10-08 18:52 +0200 http://bitbucket.org/pypy/pypy/changeset/6d880cc2b4d6/ Log: (antocuni, arigo) Hack around: add the @c_only decorator that makes it possible to write a function in RPython that is callable (only) from C, with no magic. Use it for free(): we want it to occur in RPython because of lltype.malloc(), and the tests are not happy because ll2ctypes never sees the free. diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -255,14 +255,69 @@ cpyext_namespace = NameManager('cpyext_') -class ApiFunction(object): - def __init__(self, argtypes, restype, callable, error=CANNOT_FAIL, - c_name=None, cdecl=None, gil=None, - result_borrowed=False, result_is_ll=False): +class BaseApiFunction(object): + def __init__(self, argtypes, restype, callable): self.argtypes = argtypes self.restype = restype self.functype = lltype.Ptr(lltype.FuncType(argtypes, restype)) self.callable = callable + self.cdecl = None # default + + def get_api_decl(self, name, c_writer): + restype = self.get_c_restype(c_writer) + args = self.get_c_args(c_writer) + res = self.API_VISIBILITY % (restype,) + return "{res} {name}({args});".format(**locals()) + + def get_c_restype(self, c_writer): + if self.cdecl: + return self.cdecl.tp.result.get_c_name() + return c_writer.gettype(self.restype).replace('@', '').strip() + + def get_c_args(self, c_writer): + if self.cdecl: + args = [tp.get_c_name('arg%d' % i) for i, tp in + enumerate(self.cdecl.tp.args)] + return ', '.join(args) or "void" + args = [] + for i, argtype in enumerate(self.argtypes): + if argtype is CONST_STRING: + arg = 'const char *@' + elif argtype is CONST_STRINGP: + arg = 'const char **@' + elif argtype is CONST_WSTRING: + arg = 'const wchar_t *@' + else: + arg = c_writer.gettype(argtype) + arg = arg.replace('@', 'arg%d' % (i,)).strip() + args.append(arg) + args = ', '.join(args) or "void" + return args + + def get_ptr_decl(self, name, c_writer): + restype = self.get_c_restype(c_writer) + args = self.get_c_args(c_writer) + return "{restype} (*{name})({args});".format(**locals()) + + def get_ctypes_impl(self, name, c_writer): + restype = self.get_c_restype(c_writer) + args = self.get_c_args(c_writer) + callargs = ', '.join('arg%d' % (i,) + for i in range(len(self.argtypes))) + if self.restype is lltype.Void: + body = "{ _pypyAPI.%s(%s); }" % (name, callargs) + else: + body = "{ return _pypyAPI.%s(%s); }" % (name, callargs) + return '%s %s(%s)\n%s' % (restype, name, args, body) + + +class ApiFunction(BaseApiFunction): + API_VISIBILITY = "PyAPI_FUNC(%s)" + + def __init__(self, argtypes, restype, callable, error=CANNOT_FAIL, + c_name=None, cdecl=None, gil=None, + result_borrowed=False, result_is_ll=False): + BaseApiFunction.__init__(self, argtypes, restype, callable) self.error_value = error self.c_name = c_name self.cdecl = cdecl @@ -383,52 +438,6 @@ keepalive_until_here(*keepalives) return unwrapper - def get_c_restype(self, c_writer): - if self.cdecl: - return self.cdecl.tp.result.get_c_name() - return c_writer.gettype(self.restype).replace('@', '').strip() - - def get_c_args(self, c_writer): - if self.cdecl: - args = [tp.get_c_name('arg%d' % i) for i, tp in - enumerate(self.cdecl.tp.args)] - return ', '.join(args) or "void" - args = [] - for i, argtype in enumerate(self.argtypes): - if argtype is CONST_STRING: - arg = 'const char *@' - elif argtype is CONST_STRINGP: - arg = 'const char **@' - elif argtype is CONST_WSTRING: - arg = 'const wchar_t *@' - else: - arg = c_writer.gettype(argtype) - arg = arg.replace('@', 'arg%d' % (i,)).strip() - args.append(arg) - args = ', '.join(args) or "void" - return args - - def get_api_decl(self, name, c_writer): - restype = self.get_c_restype(c_writer) - args = self.get_c_args(c_writer) - return "PyAPI_FUNC({restype}) {name}({args});".format(**locals()) - - def get_ptr_decl(self, name, c_writer): - restype = self.get_c_restype(c_writer) - args = self.get_c_args(c_writer) - return "{restype} (*{name})({args});".format(**locals()) - - def get_ctypes_impl(self, name, c_writer): - restype = self.get_c_restype(c_writer) - args = self.get_c_args(c_writer) - callargs = ', '.join('arg%d' % (i,) - for i in range(len(self.argtypes))) - if self.restype is lltype.Void: - body = "{ _pypyAPI.%s(%s); }" % (name, callargs) - else: - body = "{ return _pypyAPI.%s(%s); }" % (name, callargs) - return '%s %s(%s)\n%s' % (restype, name, args, body) - DEFAULT_HEADER = 'pypy_decl.h' def cpython_api(argtypes, restype, error=_NOT_SPECIFIED, header=DEFAULT_HEADER, @@ -466,6 +475,30 @@ return unwrapper return decorate +class COnlyApiFunction(BaseApiFunction): + API_VISIBILITY = "extern %s" + + def __init__(self, *args, **kwds): + BaseApiFunction.__init__(self, *args, **kwds) + # + def get_llhelper(space): + return llhelper(self.functype, self.callable) + self.get_llhelper = get_llhelper + + def __call__(self, *args): + raise TypeError("the function %s should not be directly " + "called from RPython, but only from C" % (self.func,)) + +def c_only(argtypes, restype): + def decorate(func): + header = DEFAULT_HEADER + if func.__name__ in FUNCTIONS_BY_HEADER[header]: + raise ValueError("%s already registered" % func.__name__) + api_function = COnlyApiFunction(argtypes, restype, func) + FUNCTIONS_BY_HEADER[header][func.__name__] = api_function + return api_function + return decorate + def api_func_from_cdef(func, cdef, cts, error=_NOT_SPECIFIED, header=DEFAULT_HEADER, result_is_ll=False): diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -3,7 +3,7 @@ cpython_api, generic_cpy_call, CANNOT_FAIL, Py_ssize_t, Py_ssize_tP, PyVarObject, size_t, slot_function, Py_TPFLAGS_HEAPTYPE, Py_LT, Py_LE, Py_EQ, Py_NE, Py_GT, - Py_GE, CONST_STRING, FILEP, fwrite) + Py_GE, CONST_STRING, FILEP, fwrite, c_only) from pypy.module.cpyext.pyobject import ( PyObject, PyObjectP, from_ref, Py_IncRef, Py_DecRef, get_typedescr) @@ -32,6 +32,10 @@ # XXX FIXME return realloc(ptr, size) + at c_only([rffi.VOIDP], lltype.Void) +def _PyPy_Free(ptr): + lltype.free(ptr, flavor='raw') + @cpython_api([PyTypeObjectPtr], PyObject, result_is_ll=True) def _PyObject_New(space, type): return _PyObject_NewVar(space, type, 0) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -36,16 +36,18 @@ Py_DECREF(pto); } +extern void _PyPy_Free(void *ptr); + void PyObject_Free(void *obj) { - free(obj); + _PyPy_Free(obj); } void PyObject_GC_Del(void *obj) { - free(obj); + _PyPy_Free(obj); } PyObject * From pypy.commits at gmail.com Sun Oct 8 12:58:08 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 08 Oct 2017 09:58:08 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59da5920.2587df0a.e5230.5cda@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92656:48682b264cfe Date: 2017-10-08 18:57 +0200 http://bitbucket.org/pypy/pypy/changeset/48682b264cfe/ Log: (antocuni, arigo) fix translation (1st step?) diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -262,6 +262,10 @@ self.functype = lltype.Ptr(lltype.FuncType(argtypes, restype)) self.callable = callable self.cdecl = None # default + # + def get_llhelper(space): + return llhelper(self.functype, self.get_wrapper(space)) + self.get_llhelper = get_llhelper def get_api_decl(self, name, c_writer): restype = self.get_c_restype(c_writer) @@ -335,10 +339,6 @@ self.gil = gil self.result_borrowed = result_borrowed self.result_is_ll = result_is_ll - # - def get_llhelper(space): - return llhelper(self.functype, self.get_wrapper(space)) - self.get_llhelper = get_llhelper def _freeze_(self): return True @@ -478,12 +478,8 @@ class COnlyApiFunction(BaseApiFunction): API_VISIBILITY = "extern %s" - def __init__(self, *args, **kwds): - BaseApiFunction.__init__(self, *args, **kwds) - # - def get_llhelper(space): - return llhelper(self.functype, self.callable) - self.get_llhelper = get_llhelper + def get_wrapper(self, space): + return self.callable def __call__(self, *args): raise TypeError("the function %s should not be directly " From pypy.commits at gmail.com Sun Oct 8 13:13:39 2017 From: pypy.commits at gmail.com (jerith) Date: Sun, 08 Oct 2017 10:13:39 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Almost all objspace/std tests pass now. Message-ID: <59da5cc3.d2b0df0a.de073.bc1b@mx.google.com> Author: Jeremy Thurgood Branch: unicode-utf8 Changeset: r92657:76983639ace6 Date: 2017-10-08 19:10 +0200 http://bitbucket.org/pypy/pypy/changeset/76983639ace6/ Log: Almost all objspace/std tests pass now. diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -171,34 +171,34 @@ _builder = UnicodeBuilder def _isupper(self, ch): - return unicodedb.isupper(ord(ch)) + return unicodedb.isupper(ch) def _islower(self, ch): - return unicodedb.islower(ord(ch)) + return unicodedb.islower(ch) def _isnumeric(self, ch): - return unicodedb.isnumeric(ord(ch)) + return unicodedb.isnumeric(ch) def _istitle(self, ch): - return unicodedb.isupper(ord(ch)) or unicodedb.istitle(ord(ch)) + return unicodedb.isupper(ch) or unicodedb.istitle(ch) def _isspace(self, ch): - return unicodedb.isspace(ord(ch)) + return unicodedb.isspace(ch) def _isalpha(self, ch): - return unicodedb.isalpha(ord(ch)) + return unicodedb.isalpha(ch) def _isalnum(self, ch): - return unicodedb.isalnum(ord(ch)) + return unicodedb.isalnum(ch) def _isdigit(self, ch): - return unicodedb.isdigit(ord(ch)) + return unicodedb.isdigit(ch) def _isdecimal(self, ch): - return unicodedb.isdecimal(ord(ch)) + return unicodedb.isdecimal(ch) def _iscased(self, ch): - return unicodedb.iscased(ord(ch)) + return unicodedb.iscased(ch) def _islinebreak(self, s, pos): return rutf8.islinebreak(s, pos) @@ -354,6 +354,21 @@ def descr_rmod(self, space, w_values): return mod_format(space, w_values, self, do_unicode=True) + def descr_swapcase(self, space): + selfvalue = self._utf8 + builder = StringBuilder(len(selfvalue)) + i = 0 + while i < len(selfvalue): + ch = rutf8.codepoint_at_pos(selfvalue, i) + i = rutf8.next_codepoint_pos(selfvalue, i) + if unicodedb.isupper(ch): + rutf8.unichr_as_utf8_append(builder, unicodedb.tolower(ch)) + elif unicodedb.islower(ch): + rutf8.unichr_as_utf8_append(builder, unicodedb.toupper(ch)) + else: + rutf8.unichr_as_utf8_append(builder, ch) + return W_UnicodeObject(builder.build(), self._length) + def descr_title(self, space): if len(self._utf8) == 0: return self @@ -461,6 +476,28 @@ res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise return space.newint(res) + @specialize.arg(2) + def _is_generic(self, space, func_name): + func = getattr(self, func_name) + if self._length == 0: + return space.w_False + if self._length == 1: + return space.newbool(func(rutf8.codepoint_at_pos(self._utf8, 0))) + else: + return self._is_generic_loop(space, self._utf8, func_name) + + @specialize.arg(3) + def _is_generic_loop(self, space, v, func_name): + func = getattr(self, func_name) + val = self._utf8 + i = 0 + while i < len(val): + uchar = rutf8.codepoint_at_pos(val, i) + i = rutf8.next_codepoint_pos(val, i) + if not func(uchar): + return space.w_False + return space.w_True + def descr_encode(self, space, w_encoding=None, w_errors=None): encoding, errors = _get_encoding_and_errors(space, w_encoding, w_errors) @@ -673,6 +710,16 @@ strs_w.append(W_UnicodeObject(value[sol:eol], lgt)) return space.newlist(strs_w) + def descr_upper(self, space): + value = self._utf8 + builder = StringBuilder(len(value)) + i = 0 + while i < len(value): + uchar = rutf8.codepoint_at_pos(value, i) + i = rutf8.next_codepoint_pos(value, i) + rutf8.unichr_as_utf8_append(builder, unicodedb.toupper(uchar)) + return W_UnicodeObject(builder.build(), self._length) + @unwrap_spec(width=int) def descr_zfill(self, space, width): selfval = self._utf8 From pypy.commits at gmail.com Sun Oct 8 16:55:16 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 08 Oct 2017 13:55:16 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: to-do for cpyext Message-ID: <59da90b4.6896df0a.720ec.7e0b@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r5836:5d964bc7add5 Date: 2017-10-08 22:55 +0200 http://bitbucket.org/pypy/extradoc/changeset/5d964bc7add5/ Log: to-do for cpyext diff --git a/planning/cpyext.txt b/planning/cpyext.txt new file mode 100644 --- /dev/null +++ b/planning/cpyext.txt @@ -0,0 +1,10 @@ + + +- BaseCpyTypedescr.allocate() calls + lltype.malloc(add_memory_pressure=True). We should reduce the memory + pressure in case a DECREF() frees the object before it becomes + attached to a W_Root object. + + +- make_typedescr() for the type object (typeobject.py) should pass + basestruct=PyHeapTypeObject?? From pypy.commits at gmail.com Sun Oct 8 18:12:55 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:12:55 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: (antocuni, ronan): make Py_IncRef no_gc=True, rewrite incref not to take the space Message-ID: <59daa2e7.42da1c0a.536f5.fc48@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92658:aae2bc78be74 Date: 2017-10-07 18:35 +0200 http://bitbucket.org/pypy/pypy/changeset/aae2bc78be74/ Log: (antocuni, ronan): make Py_IncRef no_gc=True, rewrite incref not to take the space diff --git a/pypy/module/cpyext/buffer.py b/pypy/module/cpyext/buffer.py --- a/pypy/module/cpyext/buffer.py +++ b/pypy/module/cpyext/buffer.py @@ -188,7 +188,7 @@ view.c_len = length view.c_obj = obj if obj: - incref(space, obj) + incref(obj) view.c_itemsize = 1 rffi.setintfield(view, 'c_readonly', readonly) rffi.setintfield(view, 'c_ndim', 1) diff --git a/pypy/module/cpyext/bytesobject.py b/pypy/module/cpyext/bytesobject.py --- a/pypy/module/cpyext/bytesobject.py +++ b/pypy/module/cpyext/bytesobject.py @@ -6,7 +6,7 @@ from pypy.module.cpyext.pyerrors import PyErr_BadArgument from pypy.module.cpyext.pyobject import ( PyObject, PyObjectP, decref, make_ref, from_ref, track_reference, - make_typedescr, get_typedescr, as_pyobj, Py_IncRef, get_w_obj_and_decref, + make_typedescr, get_typedescr, as_pyobj, get_w_obj_and_decref, pyobj_has_w_obj) from pypy.objspace.std.bytesobject import W_BytesObject diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -9,7 +9,7 @@ bootstrap_function, slot_function) from pypy.module.cpyext.pyobject import (PyObject, PyObjectP, as_pyobj, make_typedescr, track_reference, create_ref, from_ref, decref, - Py_IncRef) + incref) from pypy.module.cpyext.object import _dealloc from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -324,7 +324,7 @@ @cpython_api([PyObject], PyObject, result_is_ll=True) def PyObject_SelfIter(space, ref): """Undocumented function, this is what CPython does.""" - Py_IncRef(space, ref) + incref(ref) return ref @cpython_api([PyObject, PyObject], PyObject) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -25,7 +25,7 @@ # Don't increase refcount for non-heaptypes flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) if flags & Py_TPFLAGS_HEAPTYPE: - Py_IncRef(space, pytype) + incref(pytype) size = pytype.c_tp_basicsize if pytype.c_tp_itemsize: @@ -309,9 +309,12 @@ keepalive_until_here(w_obj) return w_obj +def incref(obj): + assert is_pyobj(obj) + pyobj = rffi.cast(PyObject, obj) + pyobj.c_ob_refcnt += 1 - at specialize.ll() -def incref(space, obj): +def incref_w_obj(space, obj): make_ref(space, obj) def decref(obj): @@ -331,12 +334,15 @@ def decref_w_obj(space, obj): get_w_obj_and_decref(space, obj) - at cpython_api([PyObject], lltype.Void) -def Py_IncRef(space, obj): - incref(space, obj) -# you should not call this function from RPython directly; call decref(), it's -# slightly faster because it doesn't go through the unwrapper +# you should not call Py_IncRef and Py_DecRef from RPython directly; call +# incref() and decref: they are slightly faster because they don't go through +# the unwrapper + + at cpython_api([PyObject], lltype.Void, no_gc=True) +def Py_IncRef(obj): + incref(obj) + @cpython_api([PyObject], lltype.Void, no_gc=True) def Py_DecRef(obj): decref(obj) diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -197,7 +197,7 @@ to_cp = newsize for i in range(to_cp): ob = oldref.c_ob_item[i] - incref(space, ob) + incref(ob) newref.c_ob_item[i] = ob except: decref(p_ref[0]) From pypy.commits at gmail.com Sun Oct 8 18:12:59 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:12:59 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: forgot to save and commit this change Message-ID: <59daa2eb.196b1c0a.40752.79ec@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92660:32f97b462003 Date: 2017-10-07 18:47 +0200 http://bitbucket.org/pypy/pypy/changeset/32f97b462003/ Log: forgot to save and commit this change diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -267,7 +267,7 @@ decref(py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) py_dict.c__tmpkeys = create_ref(space, w_keys) - Py_IncRef(space, py_dict.c__tmpkeys) + incref(py_dict.c__tmpkeys) else: if not py_dict.c__tmpkeys: # pos should have been 0, cannot fail so return 0 From pypy.commits at gmail.com Sun Oct 8 18:13:02 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:02 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: call directly ll_type_allocate from here, bypassing the space Message-ID: <59daa2ee.cdb51c0a.dcfce.1c55@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92661:194a4564f528 Date: 2017-10-07 19:23 +0200 http://bitbucket.org/pypy/pypy/changeset/194a4564f528/ Log: call directly ll_type_allocate from here, bypassing the space diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -51,12 +51,11 @@ @cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def _PyObject_NewVar(space, tp, nitems): from pypy.module.cpyext.pyobject import _allocate_generic_object + from pypy.module.cpyext.typeobject import ll_type_alloc state = space.fromcache(State) if tp == state.C._PyPy_get_PyType_Type(): - w_type = from_ref(space, rffi.cast(PyObject, tp)) - assert isinstance(w_type, W_TypeObject) - typedescr = get_typedescr(w_type.layout.typedef) - pyobj = typedescr.allocate(space, w_type, itemcount=nitems) + metatype = rffi.cast(PyTypeObjectPtr, tp) + pyobj = ll_type_alloc(metatype, nitems) else: return _allocate_generic_object(tp, nitems) # diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -702,7 +702,9 @@ flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) if not flags & Py_TPFLAGS_HEAPTYPE: decref_w_obj(space, w_metatype) + return ll_type_alloc(metatype, itemsize) +def ll_type_alloc(metatype, itemsize=0): heaptype = lltype.malloc(PyHeapTypeObject.TO, flavor='raw', zero=True, add_memory_pressure=True) @@ -717,7 +719,6 @@ pto.c_tp_as_buffer = heaptype.c_as_buffer pto.c_tp_basicsize = -1 # hopefully this makes malloc bail out pto.c_tp_itemsize = 0 - return rffi.cast(PyObject, heaptype) def type_attach(space, py_obj, w_type, w_userdata=None): From pypy.commits at gmail.com Sun Oct 8 18:13:04 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:04 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: this was supposed to continue, not to return; it's 'funny' that tests pass anyway :-/ Message-ID: <59daa2f0.0ab81c0a.2d23f.e225@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92662:aa3ffaa3a670 Date: 2017-10-07 19:24 +0200 http://bitbucket.org/pypy/pypy/changeset/aa3ffaa3a670/ Log: this was supposed to continue, not to return; it's 'funny' that tests pass anyway :-/ diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -57,7 +57,7 @@ metatype = rffi.cast(PyTypeObjectPtr, tp) pyobj = ll_type_alloc(metatype, nitems) else: - return _allocate_generic_object(tp, nitems) + pyobj = _allocate_generic_object(tp, nitems) # if tp.c_tp_itemsize == 0: w_obj = PyObject_Init(space, pyobj, tp) From pypy.commits at gmail.com Sun Oct 8 18:12:57 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:12:57 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: finally, this no longer needs the space Message-ID: <59daa2e9.0ec61c0a.6f3a9.267e@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92659:0fd504cab8d9 Date: 2017-10-07 18:38 +0200 http://bitbucket.org/pypy/pypy/changeset/0fd504cab8d9/ Log: finally, this no longer needs the space diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -58,7 +58,7 @@ typedescr = get_typedescr(w_type.layout.typedef) pyobj = typedescr.allocate(space, w_type, itemcount=nitems) else: - return _allocate_generic_object(space, tp, nitems) + return _allocate_generic_object(tp, nitems) # if tp.c_tp_itemsize == 0: w_obj = PyObject_Init(space, pyobj, tp) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -21,7 +21,7 @@ #________________________________________________________ # type description -def _allocate_generic_object(space, pytype, itemcount, immortal=False): +def _allocate_generic_object(pytype, itemcount, immortal=False): # Don't increase refcount for non-heaptypes flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) if flags & Py_TPFLAGS_HEAPTYPE: @@ -58,7 +58,7 @@ pytype = as_pyobj(space, w_type) pytype = rffi.cast(PyTypeObjectPtr, pytype) assert pytype - return _allocate_generic_object(space, pytype, itemcount, immortal) + return _allocate_generic_object(pytype, itemcount, immortal) def attach(self, space, pyobj, w_obj, w_userdata=None): pass From pypy.commits at gmail.com Sun Oct 8 18:13:07 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:07 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: this is no longer needed Message-ID: <59daa2f3.90b8df0a.264cf.1aff@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92663:2754d331493b Date: 2017-10-07 19:36 +0200 http://bitbucket.org/pypy/pypy/changeset/2754d331493b/ Log: this is no longer needed diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -41,12 +41,6 @@ def _PyObject_New(space, type): return _PyObject_NewVar(space, type, 0) -def _VAR_SIZE(typeobj, nitems): - # equivalent to _PyObject_VAR_SIZE - SIZEOF_VOID_P = rffi.sizeof(rffi.VOIDP) - return (typeobj.c_tp_basicsize + - nitems * typeobj.c_tp_itemsize + - (SIZEOF_VOID_P - 1)) & ~(SIZEOF_VOID_P - 1) @cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def _PyObject_NewVar(space, tp, nitems): From pypy.commits at gmail.com Sun Oct 8 18:13:12 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:12 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: (antocuni, ronan): try to fix translation Message-ID: <59daa2f8.9c471c0a.f631e.b9a4@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92665:e088311fc427 Date: 2017-10-08 00:50 +0200 http://bitbucket.org/pypy/pypy/changeset/e088311fc427/ Log: (antocuni, ronan): try to fix translation diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -25,7 +25,7 @@ # Don't increase refcount for non-heaptypes flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) if flags & Py_TPFLAGS_HEAPTYPE: - incref(pytype) + incref(rffi.cast(PyObject, pytype)) size = pytype.c_tp_basicsize if pytype.c_tp_itemsize: From pypy.commits at gmail.com Sun Oct 8 18:13:15 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:15 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: implement _PyObject_New, _PyObject_NewVar and _PyObject_GC_New in C Message-ID: <59daa2fb.9c471c0a.f631e.b9ae@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92666:e5413301904e Date: 2017-10-09 00:12 +0200 http://bitbucket.org/pypy/pypy/changeset/e5413301904e/ Log: implement _PyObject_New, _PyObject_NewVar and _PyObject_GC_New in C diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -627,6 +627,7 @@ '_Py_QnewFlag', 'Py_Py3kWarningFlag', 'Py_HashRandomizationFlag', '_Py_PackageContext', '_PyTraceMalloc_Track', '_PyTraceMalloc_Untrack', 'PyMem_Malloc', 'Py_IncRef', 'Py_DecRef', 'PyObject_Free', 'PyObject_GC_Del', 'PyType_GenericAlloc', + '_PyObject_New', '_PyObject_NewVar', '_PyObject_GC_New', ] TYPES = {} FORWARD_DECLS = [] diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -342,6 +342,10 @@ #define PyObject_Del PyObject_Free #define PyObject_DEL PyObject_Free +PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *); +PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t); +PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *); + /* PyPy internal ----------------------------------- */ PyAPI_FUNC(int) PyPyType_Register(PyTypeObject *); diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -36,24 +36,16 @@ def _PyPy_Free(ptr): lltype.free(ptr, flavor='raw') - at cpython_api([PyTypeObjectPtr], PyObject, result_is_ll=True) -def _PyObject_New(space, type): - return _PyObject_NewVar(space, type, 0) + at c_only([Py_ssize_t], rffi.VOIDP) +def _PyPy_Malloc(size): + # XXX: the malloc inside BaseCpyTypedescr.allocate and + # typeobject.type_alloc specify zero=True, so this is why we use it also + # here. However, CPython does a simple non-initialized malloc, so we + # should investigate whether we can remove zero=True as well + return lltype.malloc(rffi.VOIDP.TO, size, + flavor='raw', zero=True, + add_memory_pressure=True) -# CCC port to C - at cpython_api([PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) -def _PyObject_NewVar(space, type, itemcount): - w_type = from_ref(space, rffi.cast(PyObject, type)) - assert isinstance(w_type, W_TypeObject) - typedescr = get_typedescr(w_type.layout.typedef) - py_obj = typedescr.allocate(space, w_type, itemcount=itemcount) - #py_obj.c_ob_refcnt = 0 --- will be set to 1 again by PyObject_Init{Var} - if type.c_tp_itemsize == 0: - w_obj = PyObject_Init(space, py_obj, type) - else: - py_objvar = rffi.cast(PyVarObject, py_obj) - w_obj = PyObject_InitVar(space, py_objvar, type, itemcount) - return py_obj def _dealloc(space, obj): # This frees an object after its refcount dropped to zero, so we @@ -65,10 +57,6 @@ if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: Py_DecRef(space, rffi.cast(PyObject, pto)) - at cpython_api([PyTypeObjectPtr], PyObject, result_is_ll=True) -def _PyObject_GC_New(space, type): - return _PyObject_New(space, type) - @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL) def _PyObject_GetDictPtr(space, op): return lltype.nullptr(PyObjectP.TO) @@ -173,6 +161,7 @@ space.delitem(w_obj, w_key) return 0 +# CCC port to C @cpython_api([PyObject, PyTypeObjectPtr], PyObject, result_is_ll=True) def PyObject_Init(space, obj, type): """Initialize a newly-allocated object op with its type and initial @@ -187,6 +176,7 @@ obj.c_ob_refcnt = 1 return obj +# CCC port to C @cpython_api([PyVarObject, PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) def PyObject_InitVar(space, py_obj, type, size): """This does everything PyObject_Init() does, and also initializes the diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -2,6 +2,9 @@ #include "Python.h" +extern void _PyPy_Free(void *ptr); +extern void *_PyPy_Malloc(Py_ssize_t size); + void Py_IncRef(PyObject *o) { @@ -36,8 +39,6 @@ Py_DECREF(pto); } -extern void _PyPy_Free(void *ptr); - void PyObject_Free(void *obj) { @@ -53,5 +54,77 @@ PyObject * PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems) { - return _PyObject_NewVar(type, nitems); + return (PyObject*)_PyObject_NewVar(type, nitems); } + +PyObject * +_PyObject_New(PyTypeObject *type) +{ + return (PyObject*)_PyObject_NewVar(type, 0); +} + +PyObject * _PyObject_GC_New(PyTypeObject *type) +{ + return _PyObject_New(type); +} + + +static PyObject * +_type_alloc(PyTypeObject *metatype) +{ + PyHeapTypeObject *heaptype = (PyHeapTypeObject*)_PyPy_Malloc(sizeof(PyTypeObject)); + PyTypeObject *pto = &heaptype->ht_type; + + pto->ob_refcnt = 1; + pto->ob_pypy_link = 0; + pto->ob_type = metatype; + pto->tp_flags |= Py_TPFLAGS_HEAPTYPE; + pto->tp_as_number = &heaptype->as_number; + pto->tp_as_sequence = &heaptype->as_sequence; + pto->tp_as_mapping = &heaptype->as_mapping; + pto->tp_as_buffer = &heaptype->as_buffer; + pto->tp_basicsize = -1; /* hopefully this makes malloc bail out */ + pto->tp_itemsize = 0; + return (PyObject*)heaptype; +} + +static PyObject * +_generic_alloc(PyTypeObject *type, Py_ssize_t nitems) +{ + if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) + Py_INCREF(type); + + Py_ssize_t size = type->tp_basicsize; + if (type->tp_itemsize) + size += nitems * type->tp_itemsize; + + PyObject *pyobj = (PyObject*)_PyPy_Malloc(size); + + if (type->tp_itemsize) + ((PyVarObject*)pyobj)->ob_size = nitems; + + pyobj->ob_refcnt = 1; + /* pyobj->ob_pypy_link should get assigned very quickly */ + pyobj->ob_type = type; + return pyobj; +} + +PyVarObject * +_PyObject_NewVar(PyTypeObject *type, Py_ssize_t nitems) +{ + /* Note that this logic is slightly different than the one used by + CPython. The plan is to try to follow as closely as possible the + current cpyext logic here, and fix it when the migration to C is + completed + */ + PyObject *py_obj; + if (type == &PyType_Type) + py_obj = _type_alloc(type); + else + py_obj = _generic_alloc(type, nitems); + + if (type->tp_itemsize == 0) + return PyObject_Init(py_obj, type); + else + return PyObject_InitVar((PyVarObject*)py_obj, type, nitems); +} From pypy.commits at gmail.com Sun Oct 8 18:13:09 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:13:09 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-nowrapper: (antocuni, ronan): try to avoid using the space inside PyErr_NoMemory, so that we can later declare it as no_gc=True Message-ID: <59daa2f5.196b1c0a.40752.7a0c@mx.google.com> Author: Antonio Cuni Branch: cpyext-nowrapper Changeset: r92664:ab603c05de2d Date: 2017-10-08 00:41 +0200 http://bitbucket.org/pypy/pypy/changeset/ab603c05de2d/ Log: (antocuni, ronan): try to avoid using the space inside PyErr_NoMemory, so that we can later declare it as no_gc=True diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -1114,6 +1114,9 @@ state.C._PyPy_get_PyType_Type = rffi.llexternal( '_PyPy_get_PyType_Type', [], PyTypeObjectPtr, compilation_info=eci, _nowrapper=True) + state.C._PyPy_get_PyExc_MemoryError = rffi.llexternal( + '_PyPy_get_PyExc_MemoryError', [], PyObject, + compilation_info=eci, _nowrapper=True) def init_function(func): @@ -1217,11 +1220,12 @@ ctypes.c_void_p).value else: if name.startswith('PyExc_'): - # we already have the pointer + # PyExc_* are C variables of type PyObject* in_dll = ll2ctypes.get_ctypes_type(PyObject).in_dll(bridge, mname) py_obj = ll2ctypes.ctypes2lltype(PyObject, in_dll) else: - # we have a structure, get its address + # the other global names actual C structures (NOT pointers to + # C structs, as PyExc_*) in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, mname) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) builder.prepare(py_obj, w_obj) @@ -1428,7 +1432,7 @@ source_dir / "pythread.c", source_dir / "missing.c", source_dir / "pymem.c", - source_dir / "typeobject.c", + source_dir / "_pypy_internal.c", ] def build_eci(code, use_micronumpy=False, translating=False): diff --git a/pypy/module/cpyext/include/Python.h b/pypy/module/cpyext/include/Python.h --- a/pypy/module/cpyext/include/Python.h +++ b/pypy/module/cpyext/include/Python.h @@ -130,6 +130,8 @@ /* Missing definitions */ #include "missing.h" +#include "_pypy_internal.h" + /* The declarations of most API functions are generated in a separate file */ /* Don't include them while building PyPy, RPython also generated signatures diff --git a/pypy/module/cpyext/include/_pypy_internal.h b/pypy/module/cpyext/include/_pypy_internal.h new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/include/_pypy_internal.h @@ -0,0 +1,17 @@ +#ifndef PyPy_INTERNAL_H +#define PyPy_INTERNAL_H + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +PyAPI_FUNC(PyTypeObject*) _PyPy_get_PyType_Type(void); +PyAPI_FUNC(PyObject*) _PyPy_get_PyExc_MemoryError(void); + + +#ifdef __cplusplus +} +#endif +#endif /* !PyPy_INTERNAL_H */ diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -330,7 +330,6 @@ PyAPI_FUNC(int) PyPyType_Register(PyTypeObject *); #define PyObject_Length PyObject_Size #define _PyObject_GC_Del PyObject_GC_Del -PyAPI_FUNC(PyTypeObject*) _PyPy_get_PyType_Type(void); #ifdef __cplusplus } diff --git a/pypy/module/cpyext/pyerrors.py b/pypy/module/cpyext/pyerrors.py --- a/pypy/module/cpyext/pyerrors.py +++ b/pypy/module/cpyext/pyerrors.py @@ -132,7 +132,8 @@ so an object allocation function can write return PyErr_NoMemory(); when it runs out of memory. Return value: always NULL.""" - PyErr_SetNone(space, space.w_MemoryError) + state = space.fromcache(State) + PyErr_SetNone(space, state.C._PyPy_get_PyExc_MemoryError()) @cpython_api([PyObject], PyObject) def PyErr_SetFromErrno(space, w_type): diff --git a/pypy/module/cpyext/src/typeobject.c b/pypy/module/cpyext/src/_pypy_internal.c rename from pypy/module/cpyext/src/typeobject.c rename to pypy/module/cpyext/src/_pypy_internal.c --- a/pypy/module/cpyext/src/typeobject.c +++ b/pypy/module/cpyext/src/_pypy_internal.c @@ -5,3 +5,9 @@ { return &PyType_Type; } + +PyObject* +_PyPy_get_PyExc_MemoryError(void) +{ + return PyExc_MemoryError; +} From pypy.commits at gmail.com Sun Oct 8 18:42:04 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 08 Oct 2017 15:42:04 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: now the signature of _PyObject_NewVar returns PyVarObject*, as CPython; previously, it returned PyObject*; fix this test by inserting the proper cast Message-ID: <59daa9bc.93d31c0a.264af.dd0c@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92667:a7a6b42a7803 Date: 2017-10-09 00:41 +0200 http://bitbucket.org/pypy/pypy/changeset/a7a6b42a7803/ Log: now the signature of _PyObject_NewVar returns PyVarObject*, as CPython; previously, it returned PyObject*; fix this test by inserting the proper cast diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py --- a/pypy/module/cpyext/test/test_bytesobject.py +++ b/pypy/module/cpyext/test/test_bytesobject.py @@ -104,7 +104,7 @@ """), ('alloc_rw', "METH_NOARGS", ''' - PyObject *obj = _PyObject_NewVar(&PyBytes_Type, 10); + PyObject *obj = (PyObject*)_PyObject_NewVar(&PyBytes_Type, 10); memcpy(PyBytes_AS_STRING(obj), "works", 6); return (PyObject*)obj; '''), From pypy.commits at gmail.com Mon Oct 9 05:21:17 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 02:21:17 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: port PyObject_Init{, Var} to C Message-ID: <59db3f8d.2587df0a.e5230.7456@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92668:ed10b210f7b9 Date: 2017-10-09 08:56 +0200 http://bitbucket.org/pypy/pypy/changeset/ed10b210f7b9/ Log: port PyObject_Init{,Var} to C diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -628,6 +628,7 @@ '_PyTraceMalloc_Track', '_PyTraceMalloc_Untrack', 'PyMem_Malloc', 'Py_IncRef', 'Py_DecRef', 'PyObject_Free', 'PyObject_GC_Del', 'PyType_GenericAlloc', '_PyObject_New', '_PyObject_NewVar', '_PyObject_GC_New', + 'PyObject_Init', 'PyObject_InitVar', ] TYPES = {} FORWARD_DECLS = [] diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -346,6 +346,10 @@ PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t); PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *); +PyAPI_FUNC(PyObject *) PyObject_Init(PyObject *, PyTypeObject *); +PyAPI_FUNC(PyVarObject *) PyObject_InitVar(PyVarObject *, + PyTypeObject *, Py_ssize_t); + /* PyPy internal ----------------------------------- */ PyAPI_FUNC(int) PyPyType_Register(PyTypeObject *); diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -161,31 +161,6 @@ space.delitem(w_obj, w_key) return 0 -# CCC port to C - at cpython_api([PyObject, PyTypeObjectPtr], PyObject, result_is_ll=True) -def PyObject_Init(space, obj, type): - """Initialize a newly-allocated object op with its type and initial - reference. Returns the initialized object. If type indicates that the - object participates in the cyclic garbage detector, it is added to the - detector's set of observed objects. Other fields of the object are not - affected.""" - if not obj: - PyErr_NoMemory(space) - obj.c_ob_type = type - obj.c_ob_pypy_link = 0 - obj.c_ob_refcnt = 1 - return obj - -# CCC port to C - at cpython_api([PyVarObject, PyTypeObjectPtr, Py_ssize_t], PyObject, result_is_ll=True) -def PyObject_InitVar(space, py_obj, type, size): - """This does everything PyObject_Init() does, and also initializes the - length information for a variable-size object.""" - if not py_obj: - PyErr_NoMemory(space) - py_obj.c_ob_size = size - return PyObject_Init(space, rffi.cast(PyObject, py_obj), type) - @cpython_api([PyObject], PyObject) def PyObject_Type(space, w_obj): """When o is non-NULL, returns a type object corresponding to the object type diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -128,3 +128,23 @@ else return PyObject_InitVar((PyVarObject*)py_obj, type, nitems); } + +PyObject * +PyObject_Init(PyObject *obj, PyTypeObject *type) +{ + if (!obj) + PyErr_NoMemory(); + obj->ob_type = type; + obj->ob_pypy_link = 0; + obj->ob_refcnt = 1; + return obj; +} + +PyVarObject * +PyObject_InitVar(PyVarObject *obj, PyTypeObject *type, Py_ssize_t size) +{ + if (!obj) + PyErr_NoMemory(); + obj->ob_size = size; + return PyObject_Init((PyObject*)obj, type); +} From pypy.commits at gmail.com Mon Oct 9 05:21:19 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 02:21:19 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: try to be a bit more similar to CPython and raise PyExc_NoMem() inside _PyObject_NewVar() instead of PyObject_Init*() Message-ID: <59db3f8f.8e841c0a.df599.fd2c@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92669:f91e599505d5 Date: 2017-10-09 09:00 +0200 http://bitbucket.org/pypy/pypy/changeset/f91e599505d5/ Log: try to be a bit more similar to CPython and raise PyExc_NoMem() inside _PyObject_NewVar() instead of PyObject_Init*() diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -123,6 +123,9 @@ else py_obj = _generic_alloc(type, nitems); + if (!py_obj) + return (PyVarObject*)PyErr_NoMemory(); + if (type->tp_itemsize == 0) return PyObject_Init(py_obj, type); else @@ -132,8 +135,6 @@ PyObject * PyObject_Init(PyObject *obj, PyTypeObject *type) { - if (!obj) - PyErr_NoMemory(); obj->ob_type = type; obj->ob_pypy_link = 0; obj->ob_refcnt = 1; @@ -143,8 +144,6 @@ PyVarObject * PyObject_InitVar(PyVarObject *obj, PyTypeObject *type, Py_ssize_t size) { - if (!obj) - PyErr_NoMemory(); obj->ob_size = size; return PyObject_Init((PyObject*)obj, type); } From pypy.commits at gmail.com Mon Oct 9 05:21:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 02:21:21 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: we can use the macros now Message-ID: <59db3f91.d1d81c0a.c1ee0.36db@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92670:c0c845f0324a Date: 2017-10-09 09:04 +0200 http://bitbucket.org/pypy/pypy/changeset/c0c845f0324a/ Log: we can use the macros now diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -127,9 +127,9 @@ return (PyVarObject*)PyErr_NoMemory(); if (type->tp_itemsize == 0) - return PyObject_Init(py_obj, type); + return PyObject_INIT(py_obj, type); else - return PyObject_InitVar((PyVarObject*)py_obj, type, nitems); + return PyObject_INIT_VAR((PyVarObject*)py_obj, type, nitems); } PyObject * From pypy.commits at gmail.com Mon Oct 9 05:21:23 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 02:21:23 -0700 (PDT) Subject: [pypy-commit] pypy default: (antocuni, arigo) Message-ID: <59db3f93.8e841c0a.df599.fd34@mx.google.com> Author: Armin Rigo Branch: Changeset: r92671:00a9596553c6 Date: 2017-10-09 11:20 +0200 http://bitbucket.org/pypy/pypy/changeset/00a9596553c6/ Log: (antocuni, arigo) This test with -A fails on CPython. Fix it, guessing that this is the non-checked-in intermediate state of what ended up as commit 98f8c7e. diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -353,12 +353,8 @@ PyObject* name = PyString_FromString("mymodule"); PyObject *obj = PyType_Type.tp_alloc(&PyType_Type, 0); PyHeapTypeObject *type = (PyHeapTypeObject*)obj; - if ((type->ht_type.tp_flags & Py_TPFLAGS_HEAPTYPE) == 0) - { - PyErr_SetString(PyExc_ValueError, - "Py_TPFLAGS_HEAPTYPE not set"); - return NULL; - } + /* this is issue #2434: logic from pybind11 */ + type->ht_type.tp_flags |= Py_TPFLAGS_HEAPTYPE; type->ht_type.tp_name = ((PyTypeObject*)args)->tp_name; PyType_Ready(&type->ht_type); ret = PyObject_SetAttrString((PyObject*)&type->ht_type, From pypy.commits at gmail.com Mon Oct 9 05:24:23 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 02:24:23 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59db4047.3499df0a.9e01c.8336@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92672:508aeed657b4 Date: 2017-10-09 11:20 +0200 http://bitbucket.org/pypy/pypy/changeset/508aeed657b4/ Log: (antocuni, arigo) This test with -A fails on CPython. Fix it, guessing that this is the non-checked-in intermediate state of what ended up as commit 98f8c7e. (grafted from 00a9596553c660c47da9223757358d02a1299545) diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -353,12 +353,8 @@ PyObject* name = PyString_FromString("mymodule"); PyObject *obj = PyType_Type.tp_alloc(&PyType_Type, 0); PyHeapTypeObject *type = (PyHeapTypeObject*)obj; - if ((type->ht_type.tp_flags & Py_TPFLAGS_HEAPTYPE) == 0) - { - PyErr_SetString(PyExc_ValueError, - "Py_TPFLAGS_HEAPTYPE not set"); - return NULL; - } + /* this is issue #2434: logic from pybind11 */ + type->ht_type.tp_flags |= Py_TPFLAGS_HEAPTYPE; type->ht_type.tp_name = ((PyTypeObject*)args)->tp_name; PyType_Ready(&type->ht_type); ret = PyObject_SetAttrString((PyObject*)&type->ht_type, From pypy.commits at gmail.com Mon Oct 9 06:16:14 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 03:16:14 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59db4c6e.dc4a1c0a.4a601.b7d7@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92673:f74697a4cbd1 Date: 2017-10-09 11:31 +0200 http://bitbucket.org/pypy/pypy/changeset/f74697a4cbd1/ Log: (antocuni, arigo) Fix diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -72,7 +72,7 @@ static PyObject * _type_alloc(PyTypeObject *metatype) { - PyHeapTypeObject *heaptype = (PyHeapTypeObject*)_PyPy_Malloc(sizeof(PyTypeObject)); + PyHeapTypeObject *heaptype = (PyHeapTypeObject*)_PyPy_Malloc(sizeof(PyHeapTypeObject)); PyTypeObject *pto = &heaptype->ht_type; pto->ob_refcnt = 1; From pypy.commits at gmail.com Mon Oct 9 06:16:16 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 03:16:16 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59db4c70.d185df0a.9fd68.9cf9@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92674:22870e1d5f60 Date: 2017-10-09 12:12 +0200 http://bitbucket.org/pypy/pypy/changeset/22870e1d5f60/ Log: (antocuni, arigo) Yay. We are seemingly managing to remove _type_alloc() now. diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -99,6 +99,7 @@ assert not kw, "Extra arguments to make_typedescr" null_dealloc = lltype.nullptr(lltype.FuncType([PyObject], lltype.Void)) + assert not isinstance(tp_basestruct, lltype.Ptr), "should pass .TO" class CpyTypedescr(BaseCpyTypedescr): basestruct = tp_basestruct diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -70,25 +70,6 @@ static PyObject * -_type_alloc(PyTypeObject *metatype) -{ - PyHeapTypeObject *heaptype = (PyHeapTypeObject*)_PyPy_Malloc(sizeof(PyHeapTypeObject)); - PyTypeObject *pto = &heaptype->ht_type; - - pto->ob_refcnt = 1; - pto->ob_pypy_link = 0; - pto->ob_type = metatype; - pto->tp_flags |= Py_TPFLAGS_HEAPTYPE; - pto->tp_as_number = &heaptype->as_number; - pto->tp_as_sequence = &heaptype->as_sequence; - pto->tp_as_mapping = &heaptype->as_mapping; - pto->tp_as_buffer = &heaptype->as_buffer; - pto->tp_basicsize = -1; /* hopefully this makes malloc bail out */ - pto->tp_itemsize = 0; - return (PyObject*)heaptype; -} - -static PyObject * _generic_alloc(PyTypeObject *type, Py_ssize_t nitems) { if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) @@ -99,7 +80,9 @@ size += nitems * type->tp_itemsize; PyObject *pyobj = (PyObject*)_PyPy_Malloc(size); - + if (pyobj == NULL) + return NULL; + if (type->tp_itemsize) ((PyVarObject*)pyobj)->ob_size = nitems; @@ -117,12 +100,7 @@ current cpyext logic here, and fix it when the migration to C is completed */ - PyObject *py_obj; - if (type == &PyType_Type) - py_obj = _type_alloc(type); - else - py_obj = _generic_alloc(type, nitems); - + PyObject *py_obj = _generic_alloc(type, nitems); if (!py_obj) return (PyVarObject*)PyErr_NoMemory(); diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -523,7 +523,7 @@ @bootstrap_function def init_typeobject(space): make_typedescr(space.w_type.layout.typedef, - basestruct=PyTypeObject, + basestruct=PyHeapTypeObject.TO, alloc=type_alloc, attach=type_attach, realize=type_realize, From pypy.commits at gmail.com Mon Oct 9 08:35:54 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 05:35:54 -0700 (PDT) Subject: [pypy-commit] cffi release-1.11: released cffi 1.11.2 Message-ID: <59db6d2a.c7341c0a.2e787.ad66@mx.google.com> Author: Armin Rigo Branch: release-1.11 Changeset: r3047:5f9690f5832b Date: 2017-10-09 14:35 +0200 http://bitbucket.org/cffi/cffi/changeset/5f9690f5832b/ Log: released cffi 1.11.2 diff --git a/doc/source/installation.rst b/doc/source/installation.rst --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -55,11 +55,11 @@ * Checksums of the "source" package version 1.11.2: - - MD5: ... + - MD5: a731487324b501c8295221b629d3f5f3 - - SHA: ... + - SHA: 04d2df85eb1921630b4f9206886737eb37200c19 - - SHA256: ... + - SHA256: ab87dd91c0c4073758d07334c1e5f712ce8fe48f007b86f8238773963ee700a6 * Or grab the most current version from the `Bitbucket page`_: ``hg clone https://bitbucket.org/cffi/cffi`` From pypy.commits at gmail.com Mon Oct 9 08:35:56 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 05:35:56 -0700 (PDT) Subject: [pypy-commit] cffi release-1.11: Added tag v1.11.2 for changeset 5f9690f5832b Message-ID: <59db6d2c.472e1c0a.59b2b.4040@mx.google.com> Author: Armin Rigo Branch: release-1.11 Changeset: r3048:f06cbc514392 Date: 2017-10-09 14:35 +0200 http://bitbucket.org/cffi/cffi/changeset/f06cbc514392/ Log: Added tag v1.11.2 for changeset 5f9690f5832b diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -13,3 +13,4 @@ ca6e81df7f1ea58d891129ad016a8888c08f238b release-0.1 0000000000000000000000000000000000000000 release-0.1 ada126bd7d1e96cc76303c1fca64a556912549d8 v1.11.1 +5f9690f5832b0292056df45f72314d69c191f75a v1.11.2 From pypy.commits at gmail.com Mon Oct 9 08:35:58 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 05:35:58 -0700 (PDT) Subject: [pypy-commit] cffi default: hg merge release-1.11 Message-ID: <59db6d2e.84ad1c0a.f88b8.3eba@mx.google.com> Author: Armin Rigo Branch: Changeset: r3049:19b4f570695d Date: 2017-10-09 14:35 +0200 http://bitbucket.org/cffi/cffi/changeset/19b4f570695d/ Log: hg merge release-1.11 diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -13,3 +13,4 @@ ca6e81df7f1ea58d891129ad016a8888c08f238b release-0.1 0000000000000000000000000000000000000000 release-0.1 ada126bd7d1e96cc76303c1fca64a556912549d8 v1.11.1 +5f9690f5832b0292056df45f72314d69c191f75a v1.11.2 diff --git a/doc/source/installation.rst b/doc/source/installation.rst --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -55,11 +55,11 @@ * Checksums of the "source" package version 1.11.2: - - MD5: ... + - MD5: a731487324b501c8295221b629d3f5f3 - - SHA: ... + - SHA: 04d2df85eb1921630b4f9206886737eb37200c19 - - SHA256: ... + - SHA256: ab87dd91c0c4073758d07334c1e5f712ce8fe48f007b86f8238773963ee700a6 * Or grab the most current version from the `Bitbucket page`_: ``hg clone https://bitbucket.org/cffi/cffi`` From pypy.commits at gmail.com Mon Oct 9 08:56:00 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 05:56:00 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: (antocuni, arigo): another idea Message-ID: <59db71e0.0a98df0a.f2484.8247@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5837:5a8c64c3a71f Date: 2017-10-09 14:55 +0200 http://bitbucket.org/pypy/extradoc/changeset/5a8c64c3a71f/ Log: (antocuni, arigo): another idea diff --git a/planning/cpyext.txt b/planning/cpyext.txt --- a/planning/cpyext.txt +++ b/planning/cpyext.txt @@ -8,3 +8,12 @@ - make_typedescr() for the type object (typeobject.py) should pass basestruct=PyHeapTypeObject?? + +- right now the conversion from space.w_None to Py_None always does a dict + lookup (inside as_pyobj); investigate whether it is worth to add a special + case for it. A possible more general solution is to add a special field to + some selected W_* classes (for example W_TypeObject, W_NoneObject and an + hypotetical W_CPyObject) which contains a PyObject* field, so that doing the + w_* -> py_* lookup is just a getfield (there is alreadt support for this in + rawrefcount.create_link_pyobj) + From pypy.commits at gmail.com Mon Oct 9 10:05:23 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 07:05:23 -0700 (PDT) Subject: [pypy-commit] pypy default: (arigo, ronan) Make sure that the keys returned from PyDict_Next are kept alive. Message-ID: <59db8223.0a181c0a.d08d8.ef03@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92675:1b7397a88e79 Date: 2017-10-09 16:04 +0200 http://bitbucket.org/pypy/pypy/changeset/1b7397a88e79/ Log: (arigo, ronan) Make sure that the keys returned from PyDict_Next are kept alive. And try to make the general problem more easily debuggable in as_pyobj(). diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -258,7 +258,7 @@ if w_dict is None: return 0 if not space.isinstance_w(w_dict, space.w_dict): - return 0 + return 0 pos = ppos[0] py_obj = as_pyobj(space, w_dict) py_dict = rffi.cast(PyDictObject, py_obj) @@ -266,6 +266,8 @@ # Store the current keys in the PyDictObject. decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) + # w_keys must use the object strategy in order to keep the keys alive + w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) else: @@ -278,10 +280,10 @@ decref(space, py_dict.c__tmpkeys) py_dict.c__tmpkeys = lltype.nullptr(PyObject.TO) return 0 - w_key = space.listview(w_keys)[pos] + w_key = space.listview(w_keys)[pos] # fast iff w_keys uses object strat w_value = space.getitem(w_dict, w_key) if pkey: - pkey[0] = as_pyobj(space, w_key) + pkey[0] = as_pyobj(space, w_key) if pvalue: pvalue[0] = as_pyobj(space, w_value) return 1 diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -15,7 +15,7 @@ from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount, jit -from rpython.rlib.debug import fatalerror +from rpython.rlib.debug import ll_assert, fatalerror #________________________________________________________ @@ -243,6 +243,11 @@ py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) + # + # Try to crash here, instead of randomly, if we don't keep w_obj alive + ll_assert(py_obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY, + "Bug in cpyext: The W_Root object was garbage-collected " + "while being converted to PyObject.") return py_obj else: return lltype.nullptr(PyObject.TO) From pypy.commits at gmail.com Mon Oct 9 10:09:04 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 07:09:04 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): use make_ref+decref instead of as_pyobj to convert arguments inside generic_cpy_call: the plan is to add a fast-path to make_ref to allocate int objects more quickly, without passing through the whole .allocate() dance Message-ID: <59db8300.cdb51c0a.dcfce.51ad@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92676:c84be963f330 Date: 2017-10-09 16:08 +0200 http://bitbucket.org/pypy/pypy/changeset/c84be963f330/ Log: (antocuni, arigo): use make_ref+decref instead of as_pyobj to convert arguments inside generic_cpy_call: the plan is to add a fast-path to make_ref to allocate int objects more quickly, without passing through the whole .allocate() dance diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -1677,7 +1677,7 @@ @specialize.memo() def make_generic_cpy_call(FT, expect_null): - from pypy.module.cpyext.pyobject import is_pyobj, as_pyobj + from pypy.module.cpyext.pyobject import is_pyobj, make_ref, decref from pypy.module.cpyext.pyobject import get_w_obj_and_decref from pypy.module.cpyext.pyerrors import PyErr_Occurred unrolling_arg_types = unrolling_iterable(enumerate(FT.ARGS)) @@ -1705,15 +1705,17 @@ @specialize.ll() def generic_cpy_call(space, func, *args): boxed_args = () - keepalives = () + to_decref = () assert len(args) == len(FT.ARGS) for i, ARG in unrolling_arg_types: arg = args[i] + _pyobj = None if is_PyObject(ARG): if not is_pyobj(arg): - keepalives += (arg,) - arg = as_pyobj(space, arg) + arg = make_ref(space, arg) + _pyobj = arg boxed_args += (arg,) + to_decref += (_pyobj,) # see "Handling of the GIL" above tid = rthread.get_ident() @@ -1727,7 +1729,11 @@ finally: assert cpyext_glob_tid_ptr[0] == tid cpyext_glob_tid_ptr[0] = 0 - keepalive_until_here(*keepalives) + for i, ARG in unrolling_arg_types: + # note that this loop is nicely unrolled statically by RPython + _pyobj = to_decref[i] + if _pyobj is not None: + decref(space, _pyobj) if is_PyObject(RESULT_TYPE): if not is_pyobj(result): From pypy.commits at gmail.com Mon Oct 9 10:29:56 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:29:56 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59db87e4.72a6df0a.cb095.c19d@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92677:8bbae2a41f6b Date: 2017-10-09 16:26 +0200 http://bitbucket.org/pypy/pypy/changeset/8bbae2a41f6b/ Log: (antocuni, arigo) Small refactoring to make it possible to return something different from make_ref() diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -279,10 +279,7 @@ return hop.inputconst(lltype.Bool, hop.s_result.const) @specialize.ll() -def make_ref(space, obj, w_userdata=None, immortal=False): - """Increment the reference counter of the PyObject and return it. - Can be called with either a PyObject or a W_Root. - """ +def get_pyobj_and_incref(space, obj, w_userdata=None, immortal=False): if is_pyobj(obj): pyobj = rffi.cast(PyObject, obj) at_least = 1 @@ -295,6 +292,12 @@ keepalive_until_here(obj) return pyobj + at specialize.ll() +def make_ref(space, obj, w_userdata=None, immortal=False): + """Increment the reference counter of the PyObject and return it. + Can be called with either a PyObject or a W_Root. + """ + return get_pyobj_and_incref(space, obj, w_userdata, immortal=False) @specialize.ll() def get_w_obj_and_decref(space, obj): @@ -318,7 +321,7 @@ @specialize.ll() def incref(space, obj): - make_ref(space, obj) + get_pyobj_and_incref(space, obj) @specialize.ll() def decref(space, obj): From pypy.commits at gmail.com Mon Oct 9 10:29:58 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:29:58 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: Fix for leaks in case of NotImplemented return values Message-ID: <59db87e6.f0b8df0a.fe466.5e80@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92678:55577c485f1b Date: 2017-10-09 16:27 +0200 http://bitbucket.org/pypy/pypy/changeset/55577c485f1b/ Log: Fix for leaks in case of NotImplemented return values diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, readbufferproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,26 +90,29 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -127,11 +130,10 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -323,12 +325,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -341,9 +341,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -359,12 +357,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: From pypy.commits at gmail.com Mon Oct 9 10:30:00 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:30:00 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads Message-ID: <59db87e8.8a821c0a.f5f88.3fbe@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92679:2290915cd14c Date: 2017-10-09 16:29 +0200 http://bitbucket.org/pypy/pypy/changeset/2290915cd14c/ Log: merge heads diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -1677,7 +1677,7 @@ @specialize.memo() def make_generic_cpy_call(FT, expect_null): - from pypy.module.cpyext.pyobject import is_pyobj, as_pyobj + from pypy.module.cpyext.pyobject import is_pyobj, make_ref, decref from pypy.module.cpyext.pyobject import get_w_obj_and_decref from pypy.module.cpyext.pyerrors import PyErr_Occurred unrolling_arg_types = unrolling_iterable(enumerate(FT.ARGS)) @@ -1705,15 +1705,17 @@ @specialize.ll() def generic_cpy_call(space, func, *args): boxed_args = () - keepalives = () + to_decref = () assert len(args) == len(FT.ARGS) for i, ARG in unrolling_arg_types: arg = args[i] + _pyobj = None if is_PyObject(ARG): if not is_pyobj(arg): - keepalives += (arg,) - arg = as_pyobj(space, arg) + arg = make_ref(space, arg) + _pyobj = arg boxed_args += (arg,) + to_decref += (_pyobj,) # see "Handling of the GIL" above tid = rthread.get_ident() @@ -1727,7 +1729,11 @@ finally: assert cpyext_glob_tid_ptr[0] == tid cpyext_glob_tid_ptr[0] = 0 - keepalive_until_here(*keepalives) + for i, ARG in unrolling_arg_types: + # note that this loop is nicely unrolled statically by RPython + _pyobj = to_decref[i] + if _pyobj is not None: + decref(space, _pyobj) if is_PyObject(RESULT_TYPE): if not is_pyobj(result): From pypy.commits at gmail.com Mon Oct 9 10:41:19 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 07:41:19 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix performance and correctness of PyDict_Next by storing a list of keys instead of a view Message-ID: <59db8a8f.472e1c0a.59b2b.5e15@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92681:61c11feab4e6 Date: 2017-10-09 16:40 +0200 http://bitbucket.org/pypy/pypy/changeset/61c11feab4e6/ Log: Fix performance and correctness of PyDict_Next by storing a list of keys instead of a view diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -274,8 +274,9 @@ if pos == 0: # Store the current keys in the PyDictObject. decref(space, py_dict.c__tmpkeys) - w_keys = space.call_method(space.w_dict, "keys", w_dict) + w_keyview = space.call_method(space.w_dict, "keys", w_dict) # w_keys must use the object strategy in order to keep the keys alive + w_keys = space.newlist(space.listview(w_keyview)) w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) From pypy.commits at gmail.com Mon Oct 9 10:41:17 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 07:41:17 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: hg merge default Message-ID: <59db8a8d.4680df0a.ae9c7.7d64@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92680:ebfa699998d4 Date: 2017-10-09 16:11 +0200 http://bitbucket.org/pypy/pypy/changeset/ebfa699998d4/ Log: hg merge default diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -275,6 +275,8 @@ # Store the current keys in the PyDictObject. decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) + # w_keys must use the object strategy in order to keep the keys alive + w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) else: @@ -287,10 +289,10 @@ decref(space, py_dict.c__tmpkeys) py_dict.c__tmpkeys = lltype.nullptr(PyObject.TO) return 0 - w_key = space.listview(w_keys)[pos] + w_key = space.listview(w_keys)[pos] # fast iff w_keys uses object strat w_value = space.getitem(w_dict, w_key) if pkey: - pkey[0] = as_pyobj(space, w_key) + pkey[0] = as_pyobj(space, w_key) if pvalue: pvalue[0] = as_pyobj(space, w_value) return 1 diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -15,7 +15,7 @@ from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount, jit -from rpython.rlib.debug import fatalerror +from rpython.rlib.debug import ll_assert, fatalerror #________________________________________________________ @@ -243,6 +243,11 @@ py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) + # + # Try to crash here, instead of randomly, if we don't keep w_obj alive + ll_assert(py_obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY, + "Bug in cpyext: The W_Root object was garbage-collected " + "while being converted to PyObject.") return py_obj else: return lltype.nullptr(PyObject.TO) diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -337,12 +337,8 @@ PyObject* name = PyBytes_FromString("mymodule"); PyObject *obj = PyType_Type.tp_alloc(&PyType_Type, 0); PyHeapTypeObject *type = (PyHeapTypeObject*)obj; - if ((type->ht_type.tp_flags & Py_TPFLAGS_HEAPTYPE) == 0) - { - PyErr_SetString(PyExc_ValueError, - "Py_TPFLAGS_HEAPTYPE not set"); - return NULL; - } + /* this is issue #2434: logic from pybind11 */ + type->ht_type.tp_flags |= Py_TPFLAGS_HEAPTYPE; type->ht_type.tp_name = ((PyTypeObject*)args)->tp_name; PyType_Ready(&type->ht_type); ret = PyObject_SetAttrString((PyObject*)&type->ht_type, diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py --- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py +++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py @@ -37,7 +37,7 @@ v = cffi.__version__.replace('+', '') p = os.path.join(parent, 'doc', 'source', 'installation.rst') content = open(p).read() - assert ("/cffi-%s.tar.gz" % v) in content + assert (" package version %s:" % v) in content def test_setup_version(): parent = os.path.dirname(os.path.dirname(cffi.__file__)) diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: From pypy.commits at gmail.com Mon Oct 9 10:55:16 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:55:16 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: Fix for leaks in case of NotImplemented return values Message-ID: <59db8dd4.14421c0a.303f6.128a@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92682:9bfec319cb44 Date: 2017-10-09 16:42 +0200 http://bitbucket.org/pypy/pypy/changeset/9bfec319cb44/ Log: Fix for leaks in case of NotImplemented return values diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, readbufferproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref, as_pyobj from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,26 +90,29 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -127,11 +130,10 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -323,12 +325,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -341,9 +341,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -359,12 +357,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: From pypy.commits at gmail.com Mon Oct 9 10:55:18 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:55:18 -0700 (PDT) Subject: [pypy-commit] pypy default: Fix for leaks in case of NotImplemented return values Message-ID: <59db8dd6.196b1c0a.40752.63c7@mx.google.com> Author: Armin Rigo Branch: Changeset: r92683:4e7960df8e0c Date: 2017-10-09 16:42 +0200 http://bitbucket.org/pypy/pypy/changeset/4e7960df8e0c/ Log: Fix for leaks in case of NotImplemented return values diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, readbufferproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref, as_pyobj from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,26 +90,29 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -127,11 +130,10 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -323,12 +325,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -341,9 +341,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -359,12 +357,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: From pypy.commits at gmail.com Mon Oct 9 10:55:20 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:55:20 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads Message-ID: <59db8dd8.68a0df0a.a3eab.c988@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92684:77686cdbb6fe Date: 2017-10-09 16:54 +0200 http://bitbucket.org/pypy/pypy/changeset/77686cdbb6fe/ Log: merge heads diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -1677,7 +1677,7 @@ @specialize.memo() def make_generic_cpy_call(FT, expect_null): - from pypy.module.cpyext.pyobject import is_pyobj, as_pyobj + from pypy.module.cpyext.pyobject import is_pyobj, make_ref, decref from pypy.module.cpyext.pyobject import get_w_obj_and_decref from pypy.module.cpyext.pyerrors import PyErr_Occurred unrolling_arg_types = unrolling_iterable(enumerate(FT.ARGS)) @@ -1705,15 +1705,17 @@ @specialize.ll() def generic_cpy_call(space, func, *args): boxed_args = () - keepalives = () + to_decref = () assert len(args) == len(FT.ARGS) for i, ARG in unrolling_arg_types: arg = args[i] + _pyobj = None if is_PyObject(ARG): if not is_pyobj(arg): - keepalives += (arg,) - arg = as_pyobj(space, arg) + arg = make_ref(space, arg) + _pyobj = arg boxed_args += (arg,) + to_decref += (_pyobj,) # see "Handling of the GIL" above tid = rthread.get_ident() @@ -1727,7 +1729,11 @@ finally: assert cpyext_glob_tid_ptr[0] == tid cpyext_glob_tid_ptr[0] = 0 - keepalive_until_here(*keepalives) + for i, ARG in unrolling_arg_types: + # note that this loop is nicely unrolled statically by RPython + _pyobj = to_decref[i] + if _pyobj is not None: + decref(space, _pyobj) if is_PyObject(RESULT_TYPE): if not is_pyobj(result): From pypy.commits at gmail.com Mon Oct 9 10:55:22 2017 From: pypy.commits at gmail.com (arigo) Date: Mon, 09 Oct 2017 07:55:22 -0700 (PDT) Subject: [pypy-commit] pypy default: merge heads Message-ID: <59db8dda.94e51c0a.f6367.9e84@mx.google.com> Author: Armin Rigo Branch: Changeset: r92685:386fcca6c58f Date: 2017-10-09 16:54 +0200 http://bitbucket.org/pypy/pypy/changeset/386fcca6c58f/ Log: merge heads diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -258,7 +258,7 @@ if w_dict is None: return 0 if not space.isinstance_w(w_dict, space.w_dict): - return 0 + return 0 pos = ppos[0] py_obj = as_pyobj(space, w_dict) py_dict = rffi.cast(PyDictObject, py_obj) @@ -266,6 +266,8 @@ # Store the current keys in the PyDictObject. decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) + # w_keys must use the object strategy in order to keep the keys alive + w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) else: @@ -278,10 +280,10 @@ decref(space, py_dict.c__tmpkeys) py_dict.c__tmpkeys = lltype.nullptr(PyObject.TO) return 0 - w_key = space.listview(w_keys)[pos] + w_key = space.listview(w_keys)[pos] # fast iff w_keys uses object strat w_value = space.getitem(w_dict, w_key) if pkey: - pkey[0] = as_pyobj(space, w_key) + pkey[0] = as_pyobj(space, w_key) if pvalue: pvalue[0] = as_pyobj(space, w_value) return 1 diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -15,7 +15,7 @@ from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount, jit -from rpython.rlib.debug import fatalerror +from rpython.rlib.debug import ll_assert, fatalerror #________________________________________________________ @@ -243,6 +243,11 @@ py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) + # + # Try to crash here, instead of randomly, if we don't keep w_obj alive + ll_assert(py_obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY, + "Bug in cpyext: The W_Root object was garbage-collected " + "while being converted to PyObject.") return py_obj else: return lltype.nullptr(PyObject.TO) From pypy.commits at gmail.com Mon Oct 9 12:42:41 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 09:42:41 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): implement PyInt_FromLong in C and use freelists for allocation, copying the code from CPython; plus, some smaller changes here and there which are needed to implement this. Message-ID: <59dba701.f586df0a.c427c.6553@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92686:e5c7b7f85187 Date: 2017-10-09 18:26 +0200 http://bitbucket.org/pypy/pypy/changeset/e5c7b7f85187/ Log: (antocuni, arigo): implement PyInt_FromLong in C and use freelists for allocation, copying the code from CPython; plus, some smaller changes here and there which are needed to implement this. diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -628,7 +628,7 @@ '_PyTraceMalloc_Track', '_PyTraceMalloc_Untrack', 'PyMem_Malloc', 'Py_IncRef', 'Py_DecRef', 'PyObject_Free', 'PyObject_GC_Del', 'PyType_GenericAlloc', '_PyObject_New', '_PyObject_NewVar', '_PyObject_GC_New', - 'PyObject_Init', 'PyObject_InitVar', + 'PyObject_Init', 'PyObject_InitVar', 'PyInt_FromLong', ] TYPES = {} FORWARD_DECLS = [] @@ -1120,9 +1120,17 @@ [PyTypeObjectPtr, Py_ssize_t], PyObject, compilation_info=eci, _nowrapper=True) + state.C.PyInt_FromLong = rffi.llexternal( + mangle_name(prefix, 'PyInt_FromLong'), + [rffi.LONG], PyObject, + compilation_info=eci, + _nowrapper=True) _, state.C.set_marker = rffi.CExternVariable( Py_ssize_t, '_pypy_rawrefcount_w_marker_deallocating', eci, _nowrapper=True, c_type='Py_ssize_t') + state.C._PyPy_int_dealloc = rffi.llexternal( + '_PyPy_int_dealloc', [PyObject], lltype.Void, + compilation_info=eci, _nowrapper=True) state.C._PyPy_subtype_dealloc = rffi.llexternal( '_PyPy_subtype_dealloc', [PyObject], lltype.Void, compilation_info=eci, _nowrapper=True) @@ -1445,6 +1453,7 @@ source_dir / "pymem.c", source_dir / "object.c", source_dir / "typeobject.c", + source_dir / "intobject.c", ] def build_eci(code, use_micronumpy=False, translating=False): diff --git a/pypy/module/cpyext/include/intobject.h b/pypy/module/cpyext/include/intobject.h --- a/pypy/module/cpyext/include/intobject.h +++ b/pypy/module/cpyext/include/intobject.h @@ -16,6 +16,9 @@ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_INT_SUBCLASS) #define PyInt_CheckExact(op) ((op)->ob_type == &PyInt_Type) +PyAPI_FUNC(PyObject *) PyInt_FromLong(long); +PyAPI_FUNC(void) _PyPy_int_dealloc(PyObject *); + #ifdef __cplusplus } #endif diff --git a/pypy/module/cpyext/intobject.py b/pypy/module/cpyext/intobject.py --- a/pypy/module/cpyext/intobject.py +++ b/pypy/module/cpyext/intobject.py @@ -5,7 +5,8 @@ cpython_api, cpython_struct, build_type_checkers_flags, bootstrap_function, PyObject, PyObjectFields, CONST_STRING, CANNOT_FAIL, Py_ssize_t) from pypy.module.cpyext.pyobject import ( - make_typedescr, track_reference, from_ref) + make_typedescr, track_reference, from_ref, BaseCpyTypedescr) +from pypy.module.cpyext.state import State from rpython.rlib.rarithmetic import r_uint, intmask, LONG_TEST, r_ulonglong from pypy.objspace.std.intobject import W_IntObject import sys @@ -19,11 +20,26 @@ @bootstrap_function def init_intobject(space): "Type description of PyIntObject" + state = space.fromcache(State) make_typedescr(space.w_int.layout.typedef, basestruct=PyIntObject.TO, attach=int_attach, + alloc=int_alloc, + dealloc=state.C._PyPy_int_dealloc, realize=int_realize) +def int_alloc(typedescr, space, w_type, itemcount): + state = space.fromcache(State) + if w_type is space.w_int: + # in theory here we just want to allocate, without initializing the + # value. However, it's just easier to call PyInt_FromLong with a dummy + # value; make sure it's big enough to avoid the smallint optimization + # (if it will ever be enabled) + return state.C.PyInt_FromLong(0xDEADBEEF) + else: + return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) + +# CCC kill this? def int_attach(space, py_obj, w_obj, w_userdata=None): """ Fills a newly allocated PyIntObject with the given int object. The @@ -32,6 +48,7 @@ py_int = rffi.cast(PyIntObject, py_obj) py_int.c_ob_ival = space.int_w(w_obj) +# CCC kill this? def int_realize(space, obj): intval = rffi.cast(lltype.Signed, rffi.cast(PyIntObject, obj).c_ob_ival) w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type)) @@ -48,12 +65,6 @@ as defined in the system header files).""" return sys.maxint - at cpython_api([lltype.Signed], PyObject) -def PyInt_FromLong(space, ival): - """Create a new integer object with a value of ival. - - """ - return space.newint(ival) @cpython_api([PyObject], lltype.Signed, error=-1) def PyInt_AsLong(space, w_obj): diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -106,7 +106,7 @@ if tp_alloc: def allocate(self, space, w_type, itemcount=0, immortal=False): - return tp_alloc(space, w_type, itemcount) + return tp_alloc(self, space, w_type, itemcount) if hasattr(tp_dealloc, 'api_func'): def get_dealloc(self, space): diff --git a/pypy/module/cpyext/src/intobject.c b/pypy/module/cpyext/src/intobject.c new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/src/intobject.c @@ -0,0 +1,108 @@ + +/* Integer object implementation -- copied&adapted from CPython */ + +#include "Python.h" + +/* Integers are quite normal objects, to make object handling uniform. + (Using odd pointers to represent integers would save much space + but require extra checks for this special case throughout the code.) + Since a typical Python program spends much of its time allocating + and deallocating integers, these operations should be very fast. + Therefore we use a dedicated allocation scheme with a much lower + overhead (in space and time) than straight malloc(): a simple + dedicated free list, filled when necessary with memory from malloc(). + + block_list is a singly-linked list of all PyIntBlocks ever allocated, + linked via their next members. PyIntBlocks are never returned to the + system before shutdown (PyInt_Fini). + + free_list is a singly-linked list of available PyIntObjects, linked + via abuse of their ob_type members. +*/ + +#define BLOCK_SIZE 1000 /* 1K less typical malloc overhead */ +#define BHEAD_SIZE 8 /* Enough for a 64-bit pointer */ +#define N_INTOBJECTS ((BLOCK_SIZE - BHEAD_SIZE) / sizeof(PyIntObject)) + +struct _intblock { + struct _intblock *next; + PyIntObject objects[N_INTOBJECTS]; +}; + +typedef struct _intblock PyIntBlock; + +static PyIntBlock *block_list = NULL; +static PyIntObject *free_list = NULL; + +static PyIntObject * +fill_free_list(void) +{ + PyIntObject *p, *q; + /* Python's object allocator isn't appropriate for large blocks. */ + p = (PyIntObject *) PyMem_MALLOC(sizeof(PyIntBlock)); + if (p == NULL) + return (PyIntObject *) PyErr_NoMemory(); + ((PyIntBlock *)p)->next = block_list; + block_list = (PyIntBlock *)p; + /* Link the int objects together, from rear to front, then return + the address of the last int object in the block. */ + p = &((PyIntBlock *)p)->objects[0]; + q = p + N_INTOBJECTS; + while (--q > p) + Py_TYPE(q) = (struct _typeobject *)(q-1); + Py_TYPE(q) = NULL; + return p + N_INTOBJECTS - 1; +} + +#ifndef NSMALLPOSINTS +#define NSMALLPOSINTS 257 +#endif +#ifndef NSMALLNEGINTS +#define NSMALLNEGINTS 5 +#endif +#if NSMALLNEGINTS + NSMALLPOSINTS > 0 +/* References to small integers are saved in this array so that they + can be shared. + The integers that are saved are those in the range + -NSMALLNEGINTS (inclusive) to NSMALLPOSINTS (not inclusive). +*/ +static PyIntObject *small_ints[NSMALLNEGINTS + NSMALLPOSINTS]; +#endif + +PyObject * +PyInt_FromLong(long ival) +{ + register PyIntObject *v; + /* +#if NSMALLNEGINTS + NSMALLPOSINTS > 0 + if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) { + v = small_ints[ival + NSMALLNEGINTS]; + Py_INCREF(v); + return (PyObject *) v; + } +#endif + */ + if (free_list == NULL) { + if ((free_list = fill_free_list()) == NULL) + return NULL; + } + /* Inline PyObject_New */ + v = free_list; + free_list = (PyIntObject *)Py_TYPE(v); + (void)PyObject_INIT(v, &PyInt_Type); + v->ob_ival = ival; + return (PyObject *) v; +} + +/* this is CPython's int_dealloc */ +void +_PyPy_int_dealloc(PyObject *obj) +{ + PyIntObject *v = (PyIntObject *)obj; + if (PyInt_CheckExact(v)) { + Py_TYPE(v) = (struct _typeobject *)free_list; + free_list = v; + } + else + Py_TYPE(v)->tp_free((PyObject *)v); +} diff --git a/pypy/module/cpyext/test/test_intobject.py b/pypy/module/cpyext/test/test_intobject.py --- a/pypy/module/cpyext/test/test_intobject.py +++ b/pypy/module/cpyext/test/test_intobject.py @@ -1,13 +1,17 @@ from pypy.module.cpyext.test.test_api import BaseApiTest, raises_w from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from pypy.module.cpyext.intobject import ( - PyInt_Check, PyInt_AsLong, PyInt_AS_LONG, PyInt_FromLong, + PyInt_Check, PyInt_AsLong, PyInt_AS_LONG, PyInt_AsUnsignedLong, PyInt_AsUnsignedLongMask, PyInt_AsUnsignedLongLongMask) +from pypy.module.cpyext.pyobject import (decref, make_ref, + get_w_obj_and_decref) +from pypy.module.cpyext.state import State import sys class TestIntObject(BaseApiTest): def test_intobject(self, space): + state = space.fromcache(State) assert PyInt_Check(space, space.wrap(3)) assert PyInt_Check(space, space.w_True) assert not PyInt_Check(space, space.wrap((1, 2, 3))) @@ -16,7 +20,8 @@ y = PyInt_AS_LONG(space, space.wrap(i)) assert x == i assert y == i - w_x = PyInt_FromLong(space, x + 1) + py_x = state.C.PyInt_FromLong(x + 1) + w_x = get_w_obj_and_decref(space, py_x) assert space.type(w_x) is space.w_int assert space.eq_w(w_x, space.wrap(i + 1)) @@ -40,6 +45,14 @@ assert (PyInt_AsUnsignedLongLongMask(space, space.wrap(10 ** 30)) == 10 ** 30 % (2 ** 64)) + def test_freelist_direct(self, space): + state = space.fromcache(State) + p_x = state.C.PyInt_FromLong(12345678) + decref(space, p_x) + p_y = state.C.PyInt_FromLong(87654321) + assert p_x == p_y + decref(space, p_y) + def test_coerce(self, space): w_obj = space.appexec([], """(): class Coerce(object): diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -658,7 +658,7 @@ # CCC port it to C -def type_alloc(space, w_metatype, itemsize=0): +def type_alloc(typedescr, space, w_metatype, itemsize=0): metatype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_metatype)) # Don't increase refcount for non-heaptypes if metatype: From pypy.commits at gmail.com Mon Oct 9 12:42:43 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 09:42:43 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): hoorray: finally add an optimized special case to make_ref for integers, so that they are allocated from the free list Message-ID: <59dba703.76b0df0a.1e0f8.3229@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92687:79a097e55085 Date: 2017-10-09 18:35 +0200 http://bitbucket.org/pypy/pypy/changeset/79a097e55085/ Log: (antocuni, arigo): hoorray: finally add an optimized special case to make_ref for integers, so that they are allocated from the free list diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -297,6 +297,12 @@ """Increment the reference counter of the PyObject and return it. Can be called with either a PyObject or a W_Root. """ + if not is_pyobj(obj): + w_obj = obj + if w_obj is not None and space.type(w_obj) is space.w_int: + state = space.fromcache(State) + intval = space.int_w(w_obj) + return state.C.PyInt_FromLong(intval) return get_pyobj_and_incref(space, obj, w_userdata, immortal=False) @specialize.ll() diff --git a/pypy/module/cpyext/test/test_intobject.py b/pypy/module/cpyext/test/test_intobject.py --- a/pypy/module/cpyext/test/test_intobject.py +++ b/pypy/module/cpyext/test/test_intobject.py @@ -50,6 +50,19 @@ p_x = state.C.PyInt_FromLong(12345678) decref(space, p_x) p_y = state.C.PyInt_FromLong(87654321) + # check that the address is the same, i.e. that the freelist did its + # job + assert p_x == p_y + decref(space, p_y) + + def test_freelist_make_ref(self, space): + w_x = space.newint(12345678) + w_y = space.newint(87654321) + p_x = make_ref(space, w_x) + decref(space, p_x) + p_y = make_ref(space, w_y) + # check that the address is the same: note that w_x does NOT keep p_x + # alive, because in make_ref we have a special case for ints assert p_x == p_y decref(space, p_y) From pypy.commits at gmail.com Mon Oct 9 12:42:47 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 09:42:47 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): we concluded that we cannot kill these: they are needed e.g. by as_pyobj and int subclasses Message-ID: <59dba707.53b4df0a.8611e.6868@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92689:3f54b35142d4 Date: 2017-10-09 18:42 +0200 http://bitbucket.org/pypy/pypy/changeset/3f54b35142d4/ Log: (antocuni, arigo): we concluded that we cannot kill these: they are needed e.g. by as_pyobj and int subclasses diff --git a/pypy/module/cpyext/intobject.py b/pypy/module/cpyext/intobject.py --- a/pypy/module/cpyext/intobject.py +++ b/pypy/module/cpyext/intobject.py @@ -39,7 +39,6 @@ else: return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) -# CCC kill this? def int_attach(space, py_obj, w_obj, w_userdata=None): """ Fills a newly allocated PyIntObject with the given int object. The @@ -48,7 +47,6 @@ py_int = rffi.cast(PyIntObject, py_obj) py_int.c_ob_ival = space.int_w(w_obj) -# CCC kill this? def int_realize(space, obj): intval = rffi.cast(lltype.Signed, rffi.cast(PyIntObject, obj).c_ob_ival) w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type)) From pypy.commits at gmail.com Mon Oct 9 12:42:45 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 09 Oct 2017 09:42:45 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): write a passing test Message-ID: <59dba705.02c51c0a.abb3.2402@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92688:abffbc3948ea Date: 2017-10-09 18:40 +0200 http://bitbucket.org/pypy/pypy/changeset/abffbc3948ea/ Log: (antocuni, arigo): write a passing test diff --git a/pypy/module/cpyext/test/test_intobject.py b/pypy/module/cpyext/test/test_intobject.py --- a/pypy/module/cpyext/test/test_intobject.py +++ b/pypy/module/cpyext/test/test_intobject.py @@ -66,6 +66,21 @@ assert p_x == p_y decref(space, p_y) + def test_freelist_int_subclass(self, space): + w_MyInt = space.appexec([], """(): + class MyInt(int): + pass + return MyInt""") + w_x = space.call_function(w_MyInt, space.newint(12345678)) + w_y = space.call_function(w_MyInt, space.newint(87654321)) + p_x = make_ref(space, w_x) + decref(space, p_x) + p_y = make_ref(space, w_y) + # now the address is different because the freelist does not work for + # int subclasses + assert p_x != p_y + decref(space, p_y) + def test_coerce(self, space): w_obj = space.appexec([], """(): class Coerce(object): From pypy.commits at gmail.com Mon Oct 9 19:19:12 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 16:19:12 -0700 (PDT) Subject: [pypy-commit] pypy default: Fix handling of high surrogates in unicode_encode_utf_32 Message-ID: <59dc03f0.aea6df0a.6fe4e.87b3@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92690:0aec7a27d5ba Date: 2017-10-10 01:18 +0200 http://bitbucket.org/pypy/pypy/changeset/0aec7a27d5ba/ Log: Fix handling of high surrogates in unicode_encode_utf_32 diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -877,32 +877,31 @@ ch = ord(s[pos]) pos += 1 ch2 = 0 - if 0xD800 <= ch < 0xDC00: - if not allow_surrogates: - ru, rs, pos = errorhandler(errors, public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - if rs is not None: - # py3k only - if len(rs) % 4 != 0: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - result.append(rs) - continue - for ch in ru: - if ord(ch) < 0xD800: - _STORECHAR32(result, ord(ch), byteorder) - else: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) + if not allow_surrogates and 0xD800 <= ch < 0xE000: + ru, rs, pos = errorhandler(errors, public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + if rs is not None: + # py3k only + if len(rs) % 4 != 0: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + result.append(rs) continue - elif MAXUNICODE < 65536 and pos < size: - ch2 = ord(s[pos]) - if 0xDC00 <= ch2 < 0xE000: - ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; - pos += 1 + for ch in ru: + if ord(ch) < 0xD800: + _STORECHAR32(result, ord(ch), byteorder) + else: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + continue + if 0xD800 <= ch < 0xDC00 and MAXUNICODE < 65536 and pos < size: + ch2 = ord(s[pos]) + if 0xDC00 <= ch2 < 0xE000: + ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; + pos += 1 _STORECHAR32(result, ch, byteorder) return result.build() diff --git a/rpython/rlib/test/test_runicode.py b/rpython/rlib/test/test_runicode.py --- a/rpython/rlib/test/test_runicode.py +++ b/rpython/rlib/test/test_runicode.py @@ -2,6 +2,7 @@ import py import sys, random +import struct from rpython.rlib import runicode from hypothesis import given, settings, strategies @@ -266,11 +267,12 @@ assert replace_with(u'rep', None) == '\x00<\x00r\x00e\x00p\x00>' assert replace_with(None, '\xca\xfe') == '\x00<\xca\xfe\x00>' - def test_utf32_surrogates(self): + @py.test.mark.parametrize('unich',[u"\ud800", u"\udc80"]) + def test_utf32_surrogates(self, unich): assert runicode.unicode_encode_utf_32_be( - u"\ud800", 1, None) == '\x00\x00\xd8\x00' + unich, 1, None) == struct.pack('>i', ord(unich)) py.test.raises(UnicodeEncodeError, runicode.unicode_encode_utf_32_be, - u"\ud800", 1, None, allow_surrogates=False) + unich, 1, None, allow_surrogates=False) def replace_with(ru, rs): def errorhandler(errors, enc, msg, u, startingpos, endingpos): if errors == 'strict': @@ -278,7 +280,7 @@ endingpos, msg) return ru, rs, endingpos return runicode.unicode_encode_utf_32_be( - u"<\ud800>", 3, None, + u"<%s>" % unich, 3, None, errorhandler, allow_surrogates=False) assert replace_with(u'rep', None) == u''.encode('utf-32-be') assert replace_with(None, '\xca\xfe\xca\xfe') == '\x00\x00\x00<\xca\xfe\xca\xfe\x00\x00\x00>' @@ -432,7 +434,7 @@ assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'ignore', final=True) == (u'aaaabbbb', len(seq) + 8)) assert (self.decoder(seq, len(seq), 'custom', final=True, - errorhandler=self.custom_replace) == + errorhandler=self.custom_replace) == (FOO * len(seq), len(seq))) assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'custom', final=True, errorhandler=self.custom_replace) == @@ -628,7 +630,7 @@ msg='invalid continuation byte') assert self.decoder(seq, len(seq), 'replace', final=True ) == (res, len(seq)) - assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, + assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'replace', final=True) == (u'aaaa' + res + u'bbbb', len(seq) + 8)) res = res.replace(FFFD, u'') From pypy.commits at gmail.com Mon Oct 9 19:54:25 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 16:54:25 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: hg merge default Message-ID: <59dc0c31.03a7df0a.bbb3.9c98@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92691:be539692837f Date: 2017-10-10 01:30 +0200 http://bitbucket.org/pypy/pypy/changeset/be539692837f/ Log: hg merge default diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref, as_pyobj from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,20 +90,21 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -121,8 +122,6 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -314,12 +313,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -332,9 +329,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -350,12 +345,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -877,32 +877,31 @@ ch = ord(s[pos]) pos += 1 ch2 = 0 - if 0xD800 <= ch < 0xDC00: - if not allow_surrogates: - ru, rs, pos = errorhandler(errors, public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - if rs is not None: - # py3k only - if len(rs) % 4 != 0: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - result.append(rs) - continue - for ch in ru: - if ord(ch) < 0xD800: - _STORECHAR32(result, ord(ch), byteorder) - else: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) + if not allow_surrogates and 0xD800 <= ch < 0xE000: + ru, rs, pos = errorhandler(errors, public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + if rs is not None: + # py3k only + if len(rs) % 4 != 0: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + result.append(rs) continue - elif MAXUNICODE < 65536 and pos < size: - ch2 = ord(s[pos]) - if 0xDC00 <= ch2 < 0xE000: - ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; - pos += 1 + for ch in ru: + if ord(ch) < 0xD800: + _STORECHAR32(result, ord(ch), byteorder) + else: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + continue + if 0xD800 <= ch < 0xDC00 and MAXUNICODE < 65536 and pos < size: + ch2 = ord(s[pos]) + if 0xDC00 <= ch2 < 0xE000: + ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; + pos += 1 _STORECHAR32(result, ch, byteorder) return result.build() diff --git a/rpython/rlib/test/test_runicode.py b/rpython/rlib/test/test_runicode.py --- a/rpython/rlib/test/test_runicode.py +++ b/rpython/rlib/test/test_runicode.py @@ -2,6 +2,7 @@ import py import sys, random +import struct from rpython.rlib import runicode from hypothesis import given, settings, strategies @@ -266,11 +267,12 @@ assert replace_with(u'rep', None) == '\x00<\x00r\x00e\x00p\x00>' assert replace_with(None, '\xca\xfe') == '\x00<\xca\xfe\x00>' - def test_utf32_surrogates(self): + @py.test.mark.parametrize('unich',[u"\ud800", u"\udc80"]) + def test_utf32_surrogates(self, unich): assert runicode.unicode_encode_utf_32_be( - u"\ud800", 1, None) == '\x00\x00\xd8\x00' + unich, 1, None) == struct.pack('>i', ord(unich)) py.test.raises(UnicodeEncodeError, runicode.unicode_encode_utf_32_be, - u"\ud800", 1, None, allow_surrogates=False) + unich, 1, None, allow_surrogates=False) def replace_with(ru, rs): def errorhandler(errors, enc, msg, u, startingpos, endingpos): if errors == 'strict': @@ -278,7 +280,7 @@ endingpos, msg) return ru, rs, endingpos return runicode.unicode_encode_utf_32_be( - u"<\ud800>", 3, None, + u"<%s>" % unich, 3, None, errorhandler, allow_surrogates=False) assert replace_with(u'rep', None) == u''.encode('utf-32-be') assert replace_with(None, '\xca\xfe\xca\xfe') == '\x00\x00\x00<\xca\xfe\xca\xfe\x00\x00\x00>' @@ -432,7 +434,7 @@ assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'ignore', final=True) == (u'aaaabbbb', len(seq) + 8)) assert (self.decoder(seq, len(seq), 'custom', final=True, - errorhandler=self.custom_replace) == + errorhandler=self.custom_replace) == (FOO * len(seq), len(seq))) assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'custom', final=True, errorhandler=self.custom_replace) == @@ -628,7 +630,7 @@ msg='invalid continuation byte') assert self.decoder(seq, len(seq), 'replace', final=True ) == (res, len(seq)) - assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, + assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'replace', final=True) == (u'aaaa' + res + u'bbbb', len(seq) + 8)) res = res.replace(FFFD, u'') From pypy.commits at gmail.com Mon Oct 9 19:54:27 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 09 Oct 2017 16:54:27 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Test behaviour of lone surrogates with non-default error handlers Message-ID: <59dc0c33.ddafdf0a.26f91.b55a@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92692:86bb8174a0e9 Date: 2017-10-10 01:53 +0200 http://bitbucket.org/pypy/pypy/changeset/86bb8174a0e9/ Log: Test behaviour of lone surrogates with non-default error handlers diff --git a/pypy/module/_codecs/test/test_codecs.py b/pypy/module/_codecs/test/test_codecs.py --- a/pypy/module/_codecs/test/test_codecs.py +++ b/pypy/module/_codecs/test/test_codecs.py @@ -762,9 +762,16 @@ assert s == b'\xe9' def test_lone_surrogates(self): - for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', - 'utf-32', 'utf-32-le', 'utf-32-be'): + encodings = ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', + 'utf-32', 'utf-32-le', 'utf-32-be') + for encoding in encodings: raises(UnicodeEncodeError, u'\ud800'.encode, encoding) + assert (u'[\udc80]'.encode(encoding, "backslashreplace") == + '[\\udc80]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "ignore") == + '[]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "replace") == + '[?]'.encode(encoding)) def test_charmap_encode(self): assert 'xxx'.encode('charmap') == b'xxx' From pypy.commits at gmail.com Tue Oct 10 03:35:21 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 00:35:21 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: add support for hashing the fetched archive and verifying it, and patching sources Message-ID: <59dc7839.8d9ddf0a.c2504.f6ac@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92693:397dbd08c702 Date: 2017-10-07 15:05 +0200 http://bitbucket.org/pypy/pypy/changeset/397dbd08c702/ Log: add support for hashing the fetched archive and verifying it, and patching sources diff --git a/pypy/tool/build_cffi_imports.py b/pypy/tool/build_cffi_imports.py --- a/pypy/tool/build_cffi_imports.py +++ b/pypy/tool/build_cffi_imports.py @@ -1,5 +1,5 @@ from __future__ import print_function -import sys, shutil, os +import sys, shutil, os, tempfile, hashlib from os.path import join class MissingDependenciesError(Exception): @@ -25,10 +25,14 @@ # for distribution, we may want to fetch dependencies not provided by # the OS, such as a recent openssl/libressl or liblzma/xz. cffi_dependencies = { - 'lzma': ('https://tukaani.org/xz/xz-5.2.3.tar.gz', []), - 'ssl': ('http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.6.0.tar.gz', + 'lzma': ('https://tukaani.org/xz/xz-5.2.3.tar.gz', + '71928b357d0a09a12a4b4c5fafca8c31c19b0e7d3b8ebb19622e96f26dbf28cb', + []), + 'ssl': ('http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.6.2.tar.gz', + 'b029d2492b72a9ba5b5fcd9f3d602c9fd0baa087912f2aaecc28f52f567ec478', ['--without-openssldir']), - '_gdbm': ('ftp://ftp.gnu.org/gnu/gdbm/gdbm-1.13.tar.gz', + '_gdbm': ('http://ftp.gnu.org/gnu/gdbm/gdbm-1.13.tar.gz', + '9d252cbd7d793f7b12bcceaddda98d257c14f4d1890d851c386c37207000a253', ['--without-readline']), } @@ -47,39 +51,66 @@ finally: tarobj.close() +def _sha256(filename): + dgst = hashlib.sha256() -def _build_dependency(name, destdir): + with open(filename, 'rb') as fp: + dgst.update(fp.read()) + return dgst.hexdigest() + + +def _build_dependency(name, destdir, patches=[]): import multiprocessing import shutil import subprocess + from rpython.tool.runsubprocess import run_subprocess + try: from urllib.request import urlretrieve except ImportError: from urllib import urlretrieve try: - url, args = cffi_dependencies[name] + url, dgst, args = cffi_dependencies[name] except KeyError: return 0, None, None - archive = os.path.join(destdir, url.rsplit('/', 1)[-1]) + archive_dir = os.path.join(tempfile.gettempdir(), 'pypy-archives') + + if not os.path.isdir(archive_dir): + os.makedirs(archive_dir) + + archive = os.path.join(archive_dir, url.rsplit('/', 1)[-1]) # next, fetch the archive to disk, if needed - if not os.path.exists(archive): + if not os.path.exists(archive) or _sha256(archive) != dgst: + print('fetching archive', url, file=sys.stderr) urlretrieve(url, archive) # extract the archive into our destination directory + print('unpacking archive', archive, file=sys.stderr) _unpack_tarfile(archive, destdir) - # configure & build it sources = os.path.join( destdir, os.path.basename(archive)[:-7], ) - from rpython.tool.runsubprocess import run_subprocess + # apply any patches + if patches: + for patch in patches: + print('applying patch', patch, file=sys.stderr) + status, stdout, stderr = run_subprocess( + '/usr/bin/patch', ['-p1', '-i', patch], cwd=sources, + ) + if status != 0: + return status, stdout, stderr + + print('configuring', sources, file=sys.stderr) + + # configure & build it status, stdout, stderr = run_subprocess( './configure', [ @@ -94,6 +125,8 @@ if status != 0: return status, stdout, stderr + print('building', sources, file=sys.stderr) + status, stdout, stderr = run_subprocess( 'make', [ @@ -130,8 +163,9 @@ print('*', ' '.join(args), file=sys.stderr) if embed_dependencies: - destdir = os.path.join(os.path.abspath(os.path.dirname(__file__)), - 'dest') + curdir = os.path.abspath(os.path.dirname(__file__)) + destdir = os.path.join(curdir, 'dest') + shutil.rmtree(destdir, ignore_errors=True) os.makedirs(destdir) From pypy.commits at gmail.com Tue Oct 10 03:35:23 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 00:35:23 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: ssl: add custom libressl patch that makes it fall back to the OS X roots Message-ID: <59dc783b.d2851c0a.e384d.dda6@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92694:edfc27431cae Date: 2017-10-07 15:04 +0200 http://bitbucket.org/pypy/pypy/changeset/edfc27431cae/ Log: ssl: add custom libressl patch that makes it fall back to the OS X roots diff --git a/lib_pypy/_cffi_ssl/osx-roots.diff b/lib_pypy/_cffi_ssl/osx-roots.diff new file mode 100644 --- /dev/null +++ b/lib_pypy/_cffi_ssl/osx-roots.diff @@ -0,0 +1,479 @@ +diff -Naur libressl-2.6.2.orig/crypto/Makefile.am libressl-2.6.2/crypto/Makefile.am +--- libressl-2.6.2.orig/crypto/Makefile.am 2017-09-02 01:49:55.000000000 +0200 ++++ libressl-2.6.2/crypto/Makefile.am 2017-10-07 14:05:16.000000000 +0200 +@@ -92,7 +92,7 @@ + -mv crypto_portable.sym.tmp crypto_portable.sym + endif + +-libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym ++libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym -framework Security -framework CoreFoundation + libcrypto_la_LIBADD = libcompat.la + if !HAVE_EXPLICIT_BZERO + libcrypto_la_LIBADD += libcompatnoopt.la +@@ -863,6 +863,7 @@ + libcrypto_la_SOURCES += x509/x509_txt.c + libcrypto_la_SOURCES += x509/x509_v3.c + libcrypto_la_SOURCES += x509/x509_vfy.c ++libcrypto_la_SOURCES += x509/x509_vfy_apple.c + libcrypto_la_SOURCES += x509/x509_vpm.c + libcrypto_la_SOURCES += x509/x509cset.c + libcrypto_la_SOURCES += x509/x509name.c +diff -Naur libressl-2.6.2.orig/crypto/Makefile.in libressl-2.6.2/crypto/Makefile.in +--- libressl-2.6.2.orig/crypto/Makefile.in 2017-09-26 06:07:03.000000000 +0200 ++++ libressl-2.6.2/crypto/Makefile.in 2017-10-07 14:05:24.000000000 +0200 +@@ -426,20 +426,20 @@ + x509/x509_err.c x509/x509_ext.c x509/x509_lu.c x509/x509_obj.c \ + x509/x509_r2x.c x509/x509_req.c x509/x509_set.c \ + x509/x509_trs.c x509/x509_txt.c x509/x509_v3.c x509/x509_vfy.c \ +- x509/x509_vpm.c x509/x509cset.c x509/x509name.c \ +- x509/x509rset.c x509/x509spki.c x509/x509type.c x509/x_all.c \ +- x509v3/pcy_cache.c x509v3/pcy_data.c x509v3/pcy_lib.c \ +- x509v3/pcy_map.c x509v3/pcy_node.c x509v3/pcy_tree.c \ +- x509v3/v3_akey.c x509v3/v3_akeya.c x509v3/v3_alt.c \ +- x509v3/v3_bcons.c x509v3/v3_bitst.c x509v3/v3_conf.c \ +- x509v3/v3_cpols.c x509v3/v3_crld.c x509v3/v3_enum.c \ +- x509v3/v3_extku.c x509v3/v3_genn.c x509v3/v3_ia5.c \ +- x509v3/v3_info.c x509v3/v3_int.c x509v3/v3_lib.c \ +- x509v3/v3_ncons.c x509v3/v3_ocsp.c x509v3/v3_pci.c \ +- x509v3/v3_pcia.c x509v3/v3_pcons.c x509v3/v3_pku.c \ +- x509v3/v3_pmaps.c x509v3/v3_prn.c x509v3/v3_purp.c \ +- x509v3/v3_skey.c x509v3/v3_sxnet.c x509v3/v3_utl.c \ +- x509v3/v3err.c ++ x509/x509_vfy_apple.c x509/x509_vpm.c x509/x509cset.c \ ++ x509/x509name.c x509/x509rset.c x509/x509spki.c \ ++ x509/x509type.c x509/x_all.c x509v3/pcy_cache.c \ ++ x509v3/pcy_data.c x509v3/pcy_lib.c x509v3/pcy_map.c \ ++ x509v3/pcy_node.c x509v3/pcy_tree.c x509v3/v3_akey.c \ ++ x509v3/v3_akeya.c x509v3/v3_alt.c x509v3/v3_bcons.c \ ++ x509v3/v3_bitst.c x509v3/v3_conf.c x509v3/v3_cpols.c \ ++ x509v3/v3_crld.c x509v3/v3_enum.c x509v3/v3_extku.c \ ++ x509v3/v3_genn.c x509v3/v3_ia5.c x509v3/v3_info.c \ ++ x509v3/v3_int.c x509v3/v3_lib.c x509v3/v3_ncons.c \ ++ x509v3/v3_ocsp.c x509v3/v3_pci.c x509v3/v3_pcia.c \ ++ x509v3/v3_pcons.c x509v3/v3_pku.c x509v3/v3_pmaps.c \ ++ x509v3/v3_prn.c x509v3/v3_purp.c x509v3/v3_skey.c \ ++ x509v3/v3_sxnet.c x509v3/v3_utl.c x509v3/v3err.c + am__objects_27 = aes/libcrypto_la-aes-elf-x86_64.lo \ + aes/libcrypto_la-bsaes-elf-x86_64.lo \ + aes/libcrypto_la-vpaes-elf-x86_64.lo \ +@@ -759,11 +759,12 @@ + x509/libcrypto_la-x509_r2x.lo x509/libcrypto_la-x509_req.lo \ + x509/libcrypto_la-x509_set.lo x509/libcrypto_la-x509_trs.lo \ + x509/libcrypto_la-x509_txt.lo x509/libcrypto_la-x509_v3.lo \ +- x509/libcrypto_la-x509_vfy.lo x509/libcrypto_la-x509_vpm.lo \ +- x509/libcrypto_la-x509cset.lo x509/libcrypto_la-x509name.lo \ +- x509/libcrypto_la-x509rset.lo x509/libcrypto_la-x509spki.lo \ +- x509/libcrypto_la-x509type.lo x509/libcrypto_la-x_all.lo \ +- x509v3/libcrypto_la-pcy_cache.lo \ ++ x509/libcrypto_la-x509_vfy.lo \ ++ x509/libcrypto_la-x509_vfy_apple.lo \ ++ x509/libcrypto_la-x509_vpm.lo x509/libcrypto_la-x509cset.lo \ ++ x509/libcrypto_la-x509name.lo x509/libcrypto_la-x509rset.lo \ ++ x509/libcrypto_la-x509spki.lo x509/libcrypto_la-x509type.lo \ ++ x509/libcrypto_la-x_all.lo x509v3/libcrypto_la-pcy_cache.lo \ + x509v3/libcrypto_la-pcy_data.lo x509v3/libcrypto_la-pcy_lib.lo \ + x509v3/libcrypto_la-pcy_map.lo x509v3/libcrypto_la-pcy_node.lo \ + x509v3/libcrypto_la-pcy_tree.lo x509v3/libcrypto_la-v3_akey.lo \ +@@ -1000,7 +1001,7 @@ + $(ASM_X86_64_ELF) $(ASM_X86_64_MACOSX) + BUILT_SOURCES = crypto_portable.sym + CLEANFILES = crypto_portable.sym +-libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym ++libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym -framework Security -framework CoreFoundation + libcrypto_la_LIBADD = libcompat.la $(am__append_1) + libcrypto_la_CPPFLAGS = $(AM_CPPFLAGS) -DLIBRESSL_INTERNAL \ + -DOPENSSL_NO_HW_PADLOCK $(am__append_2) $(am__append_3) \ +@@ -1272,20 +1273,20 @@ + x509/x509_err.c x509/x509_ext.c x509/x509_lu.c x509/x509_obj.c \ + x509/x509_r2x.c x509/x509_req.c x509/x509_set.c \ + x509/x509_trs.c x509/x509_txt.c x509/x509_v3.c x509/x509_vfy.c \ +- x509/x509_vpm.c x509/x509cset.c x509/x509name.c \ +- x509/x509rset.c x509/x509spki.c x509/x509type.c x509/x_all.c \ +- x509v3/pcy_cache.c x509v3/pcy_data.c x509v3/pcy_lib.c \ +- x509v3/pcy_map.c x509v3/pcy_node.c x509v3/pcy_tree.c \ +- x509v3/v3_akey.c x509v3/v3_akeya.c x509v3/v3_alt.c \ +- x509v3/v3_bcons.c x509v3/v3_bitst.c x509v3/v3_conf.c \ +- x509v3/v3_cpols.c x509v3/v3_crld.c x509v3/v3_enum.c \ +- x509v3/v3_extku.c x509v3/v3_genn.c x509v3/v3_ia5.c \ +- x509v3/v3_info.c x509v3/v3_int.c x509v3/v3_lib.c \ +- x509v3/v3_ncons.c x509v3/v3_ocsp.c x509v3/v3_pci.c \ +- x509v3/v3_pcia.c x509v3/v3_pcons.c x509v3/v3_pku.c \ +- x509v3/v3_pmaps.c x509v3/v3_prn.c x509v3/v3_purp.c \ +- x509v3/v3_skey.c x509v3/v3_sxnet.c x509v3/v3_utl.c \ +- x509v3/v3err.c ++ x509/x509_vfy_apple.c x509/x509_vpm.c x509/x509cset.c \ ++ x509/x509name.c x509/x509rset.c x509/x509spki.c \ ++ x509/x509type.c x509/x_all.c x509v3/pcy_cache.c \ ++ x509v3/pcy_data.c x509v3/pcy_lib.c x509v3/pcy_map.c \ ++ x509v3/pcy_node.c x509v3/pcy_tree.c x509v3/v3_akey.c \ ++ x509v3/v3_akeya.c x509v3/v3_alt.c x509v3/v3_bcons.c \ ++ x509v3/v3_bitst.c x509v3/v3_conf.c x509v3/v3_cpols.c \ ++ x509v3/v3_crld.c x509v3/v3_enum.c x509v3/v3_extku.c \ ++ x509v3/v3_genn.c x509v3/v3_ia5.c x509v3/v3_info.c \ ++ x509v3/v3_int.c x509v3/v3_lib.c x509v3/v3_ncons.c \ ++ x509v3/v3_ocsp.c x509v3/v3_pci.c x509v3/v3_pcia.c \ ++ x509v3/v3_pcons.c x509v3/v3_pku.c x509v3/v3_pmaps.c \ ++ x509v3/v3_prn.c x509v3/v3_purp.c x509v3/v3_skey.c \ ++ x509v3/v3_sxnet.c x509v3/v3_utl.c x509v3/v3err.c + + # chacha + +@@ -2808,6 +2809,8 @@ + x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509_vfy.lo: x509/$(am__dirstamp) \ + x509/$(DEPDIR)/$(am__dirstamp) ++x509/libcrypto_la-x509_vfy_apple.lo: x509/$(am__dirstamp) \ ++ x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509_vpm.lo: x509/$(am__dirstamp) \ + x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509cset.lo: x509/$(am__dirstamp) \ +@@ -3583,6 +3586,7 @@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_txt.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_v3.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vfy.Plo at am__quote@ ++ at AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vpm.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509cset.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509name.Plo at am__quote@ +@@ -7460,6 +7464,13 @@ + @AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ + @am__fastdepCC_FALSE@ $(AM_V_CC at am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o x509/libcrypto_la-x509_vfy.lo `test -f 'x509/x509_vfy.c' || echo '$(srcdir)/'`x509/x509_vfy.c + ++x509/libcrypto_la-x509_vfy_apple.lo: x509/x509_vfy_apple.c ++ at am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT x509/libcrypto_la-x509_vfy_apple.lo -MD -MP -MF x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Tpo -c -o x509/libcrypto_la-x509_vfy_apple.lo `test -f 'x509/x509_vfy_apple.c' || echo '$(srcdir)/'`x509/x509_vfy_apple.c ++ at am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Tpo x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Plo ++ at AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='x509/x509_vfy_apple.c' object='x509/libcrypto_la-x509_vfy_apple.lo' libtool=yes @AMDEPBACKSLASH@ ++ at AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ ++ at am__fastdepCC_FALSE@ $(AM_V_CC at am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o x509/libcrypto_la-x509_vfy_apple.lo `test -f 'x509/x509_vfy_apple.c' || echo '$(srcdir)/'`x509/x509_vfy_apple.c ++ + x509/libcrypto_la-x509_vpm.lo: x509/x509_vpm.c + @am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT x509/libcrypto_la-x509_vpm.lo -MD -MP -MF x509/$(DEPDIR)/libcrypto_la-x509_vpm.Tpo -c -o x509/libcrypto_la-x509_vpm.lo `test -f 'x509/x509_vpm.c' || echo '$(srcdir)/'`x509/x509_vpm.c + @am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) x509/$(DEPDIR)/libcrypto_la-x509_vpm.Tpo x509/$(DEPDIR)/libcrypto_la-x509_vpm.Plo +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy.c libressl-2.6.2/crypto/x509/x509_vfy.c +--- libressl-2.6.2.orig/crypto/x509/x509_vfy.c 2017-09-02 14:01:08.000000000 +0200 ++++ libressl-2.6.2/crypto/x509/x509_vfy.c 2017-10-07 14:05:16.000000000 +0200 +@@ -115,6 +115,13 @@ + + #define CRL_SCORE_TIME_DELTA 0x002 + ++/* ++ * If we are using Trust Evaluation Agent, rename the original function ++ */ ++#ifdef __APPLE__ ++#define X509_verify_cert X509_verify_cert_orig ++#endif ++ + static int null_callback(int ok, X509_STORE_CTX *e); + static int check_issued(X509_STORE_CTX *ctx, X509 *x, X509 *issuer); + static X509 *find_issuer(X509_STORE_CTX *ctx, STACK_OF(X509) *sk, X509 *x); +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c libressl-2.6.2/crypto/x509/x509_vfy_apple.c +--- libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c 1970-01-01 01:00:00.000000000 +0100 ++++ libressl-2.6.2/crypto/x509/x509_vfy_apple.c 2017-10-07 14:05:16.000000000 +0200 +@@ -0,0 +1,229 @@ ++/* ++ * Copyright (c) 2009 Apple Inc. All Rights Reserved. ++ * ++ * @APPLE_LICENSE_HEADER_START@ ++ * ++ * This file contains Original Code and/or Modifications of Original Code ++ * as defined in and that are subject to the Apple Public Source License ++ * Version 2.0 (the 'License'). You may not use this file except in ++ * compliance with the License. Please obtain a copy of the License at ++ * http://www.opensource.apple.com/apsl/ and read it before using this ++ * file. ++ * ++ * The Original Code and all software distributed under the License are ++ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER ++ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, ++ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, ++ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. ++ * Please see the License for the specific language governing rights and ++ * limitations under the License. ++ * ++ * @APPLE_LICENSE_HEADER_END@ ++ * ++ */ ++ ++#include ++#include ++#include ++ ++#include ++ ++#include ++ ++#include ++#include ++#include ++ ++#include "cryptlib.h" ++#include "vpm_int.h" ++#include "x509_vfy_apple.h" ++ ++#define TEA_might_correct_error(err) (err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY || err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT || err == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN) ++ ++ ++static bool add_cert_to_array(CFMutableArrayRef array, X509 *x509) ++{ ++ unsigned char *asn1_cert_data = NULL; ++ int asn1_cert_len = i2d_X509(x509, &asn1_cert_data); ++ ++ CFDataRef data = CFDataCreate(kCFAllocatorDefault, asn1_cert_data, asn1_cert_len); ++ ++ if (data == NULL) { ++ return false; ++ } ++ ++ SecCertificateRef cert = SecCertificateCreateWithData(NULL, data); ++ ++ free(asn1_cert_data); ++ ++ if (cert == NULL) { ++ CFRelease(data); ++ return false; ++ } ++ ++ CFArrayAppendValue(array, cert); ++ CFRelease(data); ++ ++ return true; ++} ++ ++static CFStringRef to_string(const char *s) { ++ if (s == NULL) ++ return NULL; ++ return CFStringCreateWithCString(kCFAllocatorDefault, s, ++ kCFStringEncodingASCII); ++} ++ ++static SecPolicyRef get_policy(X509_VERIFY_PARAM *param) { ++ switch (param->purpose) { ++ case X509_PURPOSE_SSL_CLIENT: ++ case X509_PURPOSE_SSL_SERVER: { ++ ++ if (!param->id) { ++ fprintf(stderr, "got no ID!\n"); ++ return NULL; ++ } ++ ++ CFStringRef hostname; ++ int nhosts = sk_OPENSSL_STRING_num(param->id->hosts); ++ ++ if (nhosts != 1) { ++ fprintf(stderr, "got %d hosts?\n", nhosts); ++ hostname = NULL; ++ ++ } else { ++ fprintf(stderr, "got a host!\n"); ++ hostname = to_string(sk_OPENSSL_STRING_value(param->id->hosts, 0)); ++ CFShow(hostname); ++ } ++ ++ return SecPolicyCreateSSL(param->purpose == X509_PURPOSE_SSL_SERVER, ++ hostname); ++ } ++ ++ case X509_PURPOSE_NS_SSL_SERVER: ++ case X509_PURPOSE_SMIME_SIGN: ++ case X509_PURPOSE_SMIME_ENCRYPT: ++ case X509_PURPOSE_CRL_SIGN: ++ case X509_PURPOSE_ANY: ++ case X509_PURPOSE_OCSP_HELPER: ++ case X509_PURPOSE_TIMESTAMP_SIGN: ++ default: ++ fprintf(stderr, "unsupported purpose %d", param->purpose); ++ return NULL; ++ } ++} ++ ++/* ++ * Please see comment in x509_vfy_apple.h ++ */ ++int ++X509_verify_cert(X509_STORE_CTX *ctx) ++{ ++ uint64_t certLastIndex = 0; ++ uint64_t i = 0; ++ ++ /* Try OpenSSL, if we get a local certificate issue verify against trusted roots */ ++ int ret = X509_verify_cert_orig(ctx); ++ ++ /* Verify TEA is enabled and should be used. */ ++ if (0 == X509_TEA_is_enabled() || ++ ret == 1 || !TEA_might_correct_error(ctx->error)) { ++ return ret; ++ } ++ ++ /* Verify that the certificate chain exists, otherwise make it. */ ++ if (ctx->chain == NULL && (ctx->chain = sk_X509_new_null()) == NULL) { ++ fprintf(stderr, "Could not create the certificate chain"); ++ ctx->error = X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY; ++ return -1; ++ } ++ ++ /* Verify chain depth */ ++ certLastIndex = sk_X509_num(ctx->untrusted); ++ if (certLastIndex > ctx->param->depth) { ++ fprintf(stderr, "Pruning certificate chain to %" PRIu64, certLastIndex); ++ certLastIndex = ctx->param->depth; ++ } ++ ++ CFMutableArrayRef certArray = CFArrayCreateMutable(NULL, certLastIndex + 1, NULL); ++ CFRetain(certArray); ++ ++ if (!add_cert_to_array(certArray, ctx->cert)) { ++ fprintf(stderr, "Failed to add certificate to array"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return -1; ++ } ++ ++ for (i = 0; i < certLastIndex; ++i) { ++ X509 *t = sk_X509_value(ctx->untrusted, i); ++ if (!add_cert_to_array(certArray, t)) { ++ fprintf(stderr, "Failed to add chain certificate %lld to array", i); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return 0; ++ } ++ } ++ ++ // We put ASN.1 encoded X509 on the CertificateChain, so we don't call TEACertificateChainSetEncodingHandler ++ SecPolicyRef policy = get_policy(ctx->param); ++ ++ if (policy == NULL) { ++ fprintf(stderr, "Failed to create policy!\n"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return -1; ++ } ++ ++ SecTrustRef trust = NULL; ++ ++ if (SecTrustCreateWithCertificates(certArray, policy, &trust) != errSecSuccess) { ++ fprintf(stderr, "Failed to create trust!\n"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_CERT_UNTRUSTED; ++ return -1; ++ } ++ ++ if (ctx->param->flags & X509_V_FLAG_USE_CHECK_TIME) { ++ fprintf(stderr, "Setting time not supported yet?\n"); ++ SecTrustSetVerifyDate(trust, CFDateCreate(NULL, ctx->param->check_time)); ++ } ++ ++ SecTrustResultType result = 0; ++ ++ if (SecTrustEvaluate(trust, &result) != errSecSuccess || result != kSecTrustResultUnspecified) { ++ fprintf(stderr, "Not trusted! %d\n", result); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_CERT_UNTRUSTED; ++ return 0; ++ } ++ ++ fprintf(stderr, "OK!\n"); ++ CFRelease(certArray); ++ ctx->error = 0; ++ return 1; ++} ++ ++#pragma mark Trust Evaluation Agent ++ ++/* -1: not set ++ * 0: set to false ++ * 1: set to true ++ */ ++static int tea_enabled = -1; ++ ++void ++X509_TEA_set_state(int change) ++{ ++ tea_enabled = (change) ? 1 : 0; ++} ++ ++int ++X509_TEA_is_enabled() ++{ ++ if (tea_enabled < 0) ++ tea_enabled = (NULL == getenv(X509_TEA_ENV_DISABLE)); ++ ++ return tea_enabled != 0; ++} +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.h libressl-2.6.2/crypto/x509/x509_vfy_apple.h +--- libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.h 1970-01-01 01:00:00.000000000 +0100 ++++ libressl-2.6.2/crypto/x509/x509_vfy_apple.h 2017-10-07 14:05:16.000000000 +0200 +@@ -0,0 +1,74 @@ ++/* ++ * Copyright (c) 2009 Apple Inc. All Rights Reserved. ++ * ++ * @APPLE_LICENSE_HEADER_START@ ++ * ++ * This file contains Original Code and/or Modifications of Original Code ++ * as defined in and that are subject to the Apple Public Source License ++ * Version 2.0 (the 'License'). You may not use this file except in ++ * compliance with the License. Please obtain a copy of the License at ++ * http://www.opensource.apple.com/apsl/ and read it before using this ++ * file. ++ * ++ * The Original Code and all software distributed under the License are ++ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER ++ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, ++ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, ++ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. ++ * Please see the License for the specific language governing rights and ++ * limitations under the License. ++ * ++ * @APPLE_LICENSE_HEADER_END@ ++ * ++ */ ++ ++#ifndef HEADER_X509_H ++#include ++#endif ++ ++#ifndef HEADER_X509_VFY_APPLE_H ++#define HEADER_X509_VFY_APPLE_H ++ ++/* Environment variable name to disable TEA. */ ++#define X509_TEA_ENV_DISABLE "OPENSSL_X509_TEA_DISABLE" ++ ++/* ++ * X509_verify_cert ++ * ++ * Originally located in x509_vfy.c. ++ * ++ * Verify certificate with OpenSSL created X509_verify_cert. If and only if ++ * OpenSSL cannot get certificate issuer locally then OS X security API will ++ * verify the certificate, using Trust Evaluation Agent. ++ * ++ * Return values: ++ * -------------- ++ * -1: Null was passed for either ctx or ctx->cert. ++ * 0: Certificate is trusted. ++ * 1: Certificate is not trusted. ++ */ ++int X509_verify_cert(X509_STORE_CTX *ctx); ++ ++/* ++ * X509_TEA_is_enabled ++ * ++ * Is the Trust Evaluation Agent (TEA) used for certificate verification when ++ * the issuer cannot be verified. ++ * ++ * Returns 0 if TEA is disabled and 1 if TEA is enabled. ++ */ ++int X509_TEA_is_enabled(); ++ ++/* ++ * X509_TEA_set_state ++ * ++ * Enables/disables certificate verification with Trust Evaluation Agent (TEA) ++ * when the issuer cannot be verified. ++ * ++ * Pass 0 to disable TEA and non-zero to enable TEA. ++ */ ++void X509_TEA_set_state(int change); ++ ++int X509_verify_cert_orig(X509_STORE_CTX *ctx); ++ ++#endif /* HEADER_X509_VFY_APPLE_H */ diff --git a/pypy/tool/build_cffi_imports.py b/pypy/tool/build_cffi_imports.py --- a/pypy/tool/build_cffi_imports.py +++ b/pypy/tool/build_cffi_imports.py @@ -169,7 +169,18 @@ shutil.rmtree(destdir, ignore_errors=True) os.makedirs(destdir) - status, stdout, stderr = _build_dependency(key, destdir) + if key == 'ssl' and sys.platform == 'darwin': + # this patch is loosely inspired by an Apple and adds + # a fallback to the OS X roots when none are available + patches = [ + os.path.join(curdir, + '../../lib_pypy/_cffi_ssl/osx-roots.diff'), + ] + else: + patches = [] + + status, stdout, stderr = _build_dependency(key, destdir, + patches=patches) if status != 0: failures.append((key, module)) @@ -184,6 +195,10 @@ env['LDFLAGS'] = \ '-L{}/usr/lib {}'.format(destdir, env.get('LDFLAGS', '')) + if key == 'ssl' and sys.platform == 'darwin': + # needed for our roots patch + env['LDFLAGS'] += ' -framework CoreFoundation -framework Security' + try: status, stdout, stderr = run_subprocess(str(pypy_c), args, cwd=cwd, env=env) From pypy.commits at gmail.com Tue Oct 10 03:35:32 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 00:35:32 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: merge py3.5 into py3.5-mac-embedding Message-ID: <59dc7844.c23a1c0a.a1b02.8a90@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92695:1dae3639ba33 Date: 2017-10-07 15:32 +0200 http://bitbucket.org/pypy/pypy/changeset/1dae3639ba33/ Log: merge py3.5 into py3.5-mac-embedding diff too long, truncating to 2000 out of 50513 lines diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -27,16 +27,17 @@ ^pypy/module/cpyext/test/.+\.manifest$ ^pypy/module/test_lib_pypy/ctypes_tests/.+\.o$ ^pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test\.o$ -^pypy/module/cppyy/src/.+\.o$ -^pypy/module/cppyy/bench/.+\.so$ -^pypy/module/cppyy/bench/.+\.root$ -^pypy/module/cppyy/bench/.+\.d$ -^pypy/module/cppyy/src/.+\.errors$ -^pypy/module/cppyy/test/.+_rflx\.cpp$ -^pypy/module/cppyy/test/.+\.so$ -^pypy/module/cppyy/test/.+\.rootmap$ -^pypy/module/cppyy/test/.+\.exe$ -^pypy/module/cppyy/test/.+_cint.h$ +^pypy/module/_cppyy/src/.+\.o$ +^pypy/module/_cppyy/bench/.+\.so$ +^pypy/module/_cppyy/bench/.+\.root$ +^pypy/module/_cppyy/bench/.+\.d$ +^pypy/module/_cppyy/src/.+\.errors$ +^pypy/module/_cppyy/test/.+_rflx\.cpp$ +^pypy/module/_cppyy/test/.+\.so$ +^pypy/module/_cppyy/test/.+\.rootmap$ +^pypy/module/_cppyy/test/.+\.exe$ +^pypy/module/_cppyy/test/.+_cint.h$ +^pypy/module/_cppyy/.+/*\.pcm$ ^pypy/module/test_lib_pypy/cffi_tests/__pycache__.+$ ^pypy/doc/.+\.html$ ^pypy/doc/config/.+\.rst$ @@ -94,6 +95,3 @@ ^release/ ^rpython/_cache$ -pypy/module/cppyy/.+/*\.pcm - - diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -60,8 +60,8 @@ Wim Lavrijsen Eric van Riet Paap Richard Emslie + Remi Meier Alexander Schremmer - Remi Meier Dan Villiom Podlaski Christiansen Lukas Diekmann Sven Hager @@ -102,6 +102,7 @@ Michael Foord Stephan Diehl Stefano Rivera + Jean-Paul Calderone Stefan Schwarzer Tomek Meka Valentino Volonghi @@ -110,14 +111,13 @@ Bob Ippolito Bruno Gola David Malcolm - Jean-Paul Calderone Squeaky Edd Barrett Timo Paulssen Marius Gedminas + Nicolas Truessel Alexandre Fayolle Simon Burton - Nicolas Truessel Martin Matusiak Laurence Tratt Wenzhu Man @@ -156,6 +156,7 @@ Stefan H. Muller Tim Felgentreff Eugene Oden + Dodan Mihai Jeff Terrace Henry Mason Vasily Kuznetsov @@ -182,11 +183,13 @@ Rocco Moretti Gintautas Miliauskas Lucian Branescu Mihaila + Mariano Anaya anatoly techtonik - Dodan Mihai Karl Bartel + Stefan Beyer Gabriel Lavoie Jared Grubb + Alecsandru Patrascu Olivier Dormond Wouter van Heyst Sebastian Pawluś @@ -194,6 +197,7 @@ Victor Stinner Andrews Medina Aaron Iles + p_zieschang at yahoo.de Toby Watson Daniel Patrick Stuart Williams @@ -204,6 +208,7 @@ Michael Cheng Mikael Schönenberg Stanislaw Halik + Mihnea Saracin Berkin Ilbeyi Gasper Zejn Faye Zhao @@ -214,14 +219,12 @@ Jonathan David Riehl Beatrice During Alex Perry - p_zieschang at yahoo.de Robert Zaremba Alan McIntyre Alexander Sedov Vaibhav Sood Reuben Cummings Attila Gobi - Alecsandru Patrascu Christopher Pope Tristan Arthur Christian Tismer @@ -243,7 +246,6 @@ Jacek Generowicz Sylvain Thenault Jakub Stasiak - Stefan Beyer Andrew Dalke Alejandro J. Cura Vladimir Kryachko @@ -275,6 +277,7 @@ Christoph Gerum Miguel de Val Borro Artur Lisiecki + afteryu Toni Mattis Laurens Van Houtven Bobby Impollonia @@ -305,6 +308,7 @@ Anna Katrina Dominguez Kim Jin Su Amber Brown + Anthony Sottile Nate Bragg Ben Darnell Juan Francisco Cantero Hurtado @@ -325,12 +329,14 @@ Mike Bayer Rodrigo Araújo Daniil Yarancev + Min RK OlivierBlanvillain Jonas Pfannschmidt Zearin Andrey Churin Dan Crosta reubano at gmail.com + Stanisław Halik Julien Phalip Roman Podoliaka Eli Stevens diff --git a/Makefile b/Makefile --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ RUNINTERP = $(PYPY_EXECUTABLE) endif -.PHONY: cffi_imports +.PHONY: pypy-c cffi_imports pypy-c: @echo @@ -32,7 +32,7 @@ @echo "====================================================================" @echo @sleep 5 - $(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py + cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py # Note: the -jN option, or MAKEFLAGS=-jN, are not usable. They are # replaced with an opaque --jobserver option by the time this Makefile @@ -40,4 +40,4 @@ # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html cffi_imports: pypy-c - PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true + PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py --- a/lib-python/2.7/ctypes/__init__.py +++ b/lib-python/2.7/ctypes/__init__.py @@ -361,17 +361,20 @@ if handle is None: if flags & _FUNCFLAG_CDECL: - self._handle = _ffi.CDLL(name, mode) + pypy_dll = _ffi.CDLL(name, mode) else: - self._handle = _ffi.WinDLL(name, mode) - else: - self._handle = handle + pypy_dll = _ffi.WinDLL(name, mode) + self.__pypy_dll__ = pypy_dll + handle = int(pypy_dll) + if _sys.maxint > 2 ** 32: + handle = int(handle) # long -> int + self._handle = handle def __repr__(self): - return "<%s '%s', handle %r at 0x%x>" % ( - self.__class__.__name__, self._name, self._handle, - id(self) & (_sys.maxint * 2 + 1)) - + return "<%s '%s', handle %x at %x>" % \ + (self.__class__.__name__, self._name, + (self._handle & (_sys.maxint*2 + 1)), + id(self) & (_sys.maxint*2 + 1)) def __getattr__(self, name): if name.startswith('__') and name.endswith('__'): diff --git a/lib-python/2.7/ctypes/test/test_byteswap.py b/lib-python/2.7/ctypes/test/test_byteswap.py --- a/lib-python/2.7/ctypes/test/test_byteswap.py +++ b/lib-python/2.7/ctypes/test/test_byteswap.py @@ -23,7 +23,6 @@ setattr(bits, "i%s" % i, 1) dump(bits) - @xfail def test_endian_short(self): if sys.byteorder == "little": self.assertIs(c_short.__ctype_le__, c_short) @@ -51,7 +50,6 @@ self.assertEqual(bin(s), "3412") self.assertEqual(s.value, 0x1234) - @xfail def test_endian_int(self): if sys.byteorder == "little": self.assertIs(c_int.__ctype_le__, c_int) @@ -80,7 +78,6 @@ self.assertEqual(bin(s), "78563412") self.assertEqual(s.value, 0x12345678) - @xfail def test_endian_longlong(self): if sys.byteorder == "little": self.assertIs(c_longlong.__ctype_le__, c_longlong) @@ -109,7 +106,6 @@ self.assertEqual(bin(s), "EFCDAB9078563412") self.assertEqual(s.value, 0x1234567890ABCDEF) - @xfail def test_endian_float(self): if sys.byteorder == "little": self.assertIs(c_float.__ctype_le__, c_float) @@ -128,7 +124,6 @@ self.assertAlmostEqual(s.value, math.pi, 6) self.assertEqual(bin(struct.pack(">f", math.pi)), bin(s)) - @xfail def test_endian_double(self): if sys.byteorder == "little": self.assertIs(c_double.__ctype_le__, c_double) @@ -156,7 +151,6 @@ self.assertIs(c_char.__ctype_le__, c_char) self.assertIs(c_char.__ctype_be__, c_char) - @xfail def test_struct_fields_1(self): if sys.byteorder == "little": base = BigEndianStructure @@ -192,7 +186,6 @@ pass self.assertRaises(TypeError, setattr, T, "_fields_", [("x", typ)]) - @xfail def test_struct_struct(self): # nested structures with different byteorders @@ -221,7 +214,6 @@ self.assertEqual(s.point.x, 1) self.assertEqual(s.point.y, 2) - @xfail def test_struct_fields_2(self): # standard packing in struct uses no alignment. # So, we have to align using pad bytes. @@ -245,7 +237,6 @@ s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14) self.assertEqual(bin(s1), bin(s2)) - @xfail def test_unaligned_nonnative_struct_fields(self): if sys.byteorder == "little": base = BigEndianStructure diff --git a/lib-python/2.7/ctypes/test/test_unaligned_structures.py b/lib-python/2.7/ctypes/test/test_unaligned_structures.py --- a/lib-python/2.7/ctypes/test/test_unaligned_structures.py +++ b/lib-python/2.7/ctypes/test/test_unaligned_structures.py @@ -37,10 +37,7 @@ for typ in byteswapped_structures: ## print >> sys.stderr, typ.value self.assertEqual(typ.value.offset, 1) - try: - o = typ() - except NotImplementedError as e: - self.skipTest(str(e)) # for PyPy + o = typ() o.value = 4 self.assertEqual(o.value, 4) diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py --- a/lib-python/2.7/distutils/sysconfig_pypy.py +++ b/lib-python/2.7/distutils/sysconfig_pypy.py @@ -218,6 +218,10 @@ compiler.shared_lib_extension = so_ext +def get_config_h_filename(): + """Returns the path of pyconfig.h.""" + inc_dir = get_python_inc(plat_specific=1) + return os.path.join(inc_dir, 'pyconfig.h') from sysconfig_cpython import ( parse_makefile, _variable_rx, expand_makefile_vars) diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py --- a/lib-python/2.7/distutils/unixccompiler.py +++ b/lib-python/2.7/distutils/unixccompiler.py @@ -226,7 +226,19 @@ return "-L" + dir def _is_gcc(self, compiler_name): - return "gcc" in compiler_name or "g++" in compiler_name + # XXX PyPy workaround, look at the big comment below for more + # context. On CPython, the hack below works fine because + # `compiler_name` contains the name of the actual compiler which was + # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine). + # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end + # result is that we pass the wrong option to the compiler. + # + # The workaround is to *always* pretend to be GCC if we are on Linux: + # this should cover the vast majority of real systems, including the + # ones which use clang (which understands the '-Wl,-rpath' syntax as + # well) + return (sys.platform == "linux2" or + "gcc" in compiler_name or "g++" in compiler_name) def runtime_library_dir_option(self, dir): # XXX Hackish, at the very least. See Python bug #445902: diff --git a/lib-python/2.7/inspect.py b/lib-python/2.7/inspect.py --- a/lib-python/2.7/inspect.py +++ b/lib-python/2.7/inspect.py @@ -203,7 +203,7 @@ f_locals local namespace seen by this frame f_restricted 0 or 1 if frame is in restricted execution mode f_trace tracing function for this frame, or None""" - return isinstance(object, types.FrameType) + return isinstance(object, (types.FrameType, types.FakeFrameType)) def iscode(object): """Return true if the object is a code object. diff --git a/lib-python/2.7/multiprocessing/heap.py b/lib-python/2.7/multiprocessing/heap.py --- a/lib-python/2.7/multiprocessing/heap.py +++ b/lib-python/2.7/multiprocessing/heap.py @@ -62,7 +62,7 @@ self.size = size self.name = 'pym-%d-%d' % (os.getpid(), Arena._counter.next()) self.buffer = mmap.mmap(-1, self.size, tagname=self.name) - assert win32.GetLastError() == 0, 'tagname already in use' + #assert win32.GetLastError() == 0, 'tagname already in use' self._state = (self.size, self.name) def __getstate__(self): @@ -72,7 +72,7 @@ def __setstate__(self, state): self.size, self.name = self._state = state self.buffer = mmap.mmap(-1, self.size, tagname=self.name) - assert win32.GetLastError() == win32.ERROR_ALREADY_EXISTS + #assert win32.GetLastError() == win32.ERROR_ALREADY_EXISTS else: diff --git a/lib-python/2.7/string.py b/lib-python/2.7/string.py --- a/lib-python/2.7/string.py +++ b/lib-python/2.7/string.py @@ -75,7 +75,7 @@ for i in range(256): buf[i] = i for i in range(n): - buf[ord(fromstr[i])] = tostr[i] + buf[ord(fromstr[i])] = ord(tostr[i]) return str(buf) diff --git a/lib-python/2.7/test/test_os.py b/lib-python/2.7/test/test_os.py --- a/lib-python/2.7/test/test_os.py +++ b/lib-python/2.7/test/test_os.py @@ -580,6 +580,7 @@ "getentropy() does not use a file descriptor") class URandomFDTests(unittest.TestCase): @unittest.skipUnless(resource, "test requires the resource module") + @test_support.impl_detail(pypy=False) # on Linux, may use getrandom() def test_urandom_failure(self): # Check urandom() failing when it is not able to open /dev/random. # We spawn a new process to make the test more robust (if getrlimit() diff --git a/lib-python/2.7/types.py b/lib-python/2.7/types.py --- a/lib-python/2.7/types.py +++ b/lib-python/2.7/types.py @@ -71,6 +71,12 @@ FrameType = type(tb.tb_frame) del tb +# PyPy extension +try: + FakeFrameType = type(next(sys._current_frames().itervalues())) +except (AttributeError, StopIteration): + FakeFrameType = FrameType + SliceType = slice EllipsisType = type(Ellipsis) diff --git a/lib-python/3/ctypes/__init__.py b/lib-python/3/ctypes/__init__.py --- a/lib-python/3/ctypes/__init__.py +++ b/lib-python/3/ctypes/__init__.py @@ -346,16 +346,18 @@ if handle is None: if flags & _FUNCFLAG_CDECL: - self._handle = _ffi.CDLL(name, mode) + pypy_dll = _ffi.CDLL(name, mode) else: - self._handle = _ffi.WinDLL(name, mode) - else: - self._handle = handle + pypy_dll = _ffi.WinDLL(name, mode) + self.__pypy_dll__ = pypy_dll + handle = int(pypy_dll) + self._handle = handle def __repr__(self): - return "<%s '%s', handle %r at 0x%x>" % ( - self.__class__.__name__, self._name, self._handle, - id(self) & (_sys.maxsize * 2 + 1)) + return "<%s '%s', handle %x at 0x%x>" % \ + (self.__class__.__name__, self._name, + (self._handle & (_sys.maxsize*2 + 1)), + id(self) & (_sys.maxsize*2 + 1)) def __getattr__(self, name): if name.startswith('__') and name.endswith('__'): diff --git a/lib-python/3/ctypes/test/test_byteswap.py b/lib-python/3/ctypes/test/test_byteswap.py --- a/lib-python/3/ctypes/test/test_byteswap.py +++ b/lib-python/3/ctypes/test/test_byteswap.py @@ -2,7 +2,6 @@ from binascii import hexlify from ctypes import * -from ctypes.test import xfail def bin(s): return hexlify(memoryview(s)).decode().upper() @@ -43,7 +42,6 @@ with self.assertRaises(AttributeError): little.z = 24 - @xfail def test_endian_short(self): if sys.byteorder == "little": self.assertIs(c_short.__ctype_le__, c_short) @@ -71,7 +69,6 @@ self.assertEqual(bin(s), "3412") self.assertEqual(s.value, 0x1234) - @xfail def test_endian_int(self): if sys.byteorder == "little": self.assertIs(c_int.__ctype_le__, c_int) @@ -100,7 +97,6 @@ self.assertEqual(bin(s), "78563412") self.assertEqual(s.value, 0x12345678) - @xfail def test_endian_longlong(self): if sys.byteorder == "little": self.assertIs(c_longlong.__ctype_le__, c_longlong) @@ -129,7 +125,6 @@ self.assertEqual(bin(s), "EFCDAB9078563412") self.assertEqual(s.value, 0x1234567890ABCDEF) - @xfail def test_endian_float(self): if sys.byteorder == "little": self.assertIs(c_float.__ctype_le__, c_float) @@ -148,7 +143,6 @@ self.assertAlmostEqual(s.value, math.pi, places=6) self.assertEqual(bin(struct.pack(">f", math.pi)), bin(s)) - @xfail def test_endian_double(self): if sys.byteorder == "little": self.assertIs(c_double.__ctype_le__, c_double) @@ -176,7 +170,6 @@ self.assertIs(c_char.__ctype_le__, c_char) self.assertIs(c_char.__ctype_be__, c_char) - @xfail def test_struct_fields_1(self): if sys.byteorder == "little": base = BigEndianStructure @@ -212,7 +205,6 @@ pass self.assertRaises(TypeError, setattr, T, "_fields_", [("x", typ)]) - @xfail def test_struct_struct(self): # nested structures with different byteorders @@ -241,7 +233,6 @@ self.assertEqual(s.point.x, 1) self.assertEqual(s.point.y, 2) - @xfail def test_struct_fields_2(self): # standard packing in struct uses no alignment. # So, we have to align using pad bytes. @@ -265,7 +256,6 @@ s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14) self.assertEqual(bin(s1), bin(s2)) - @xfail def test_unaligned_nonnative_struct_fields(self): if sys.byteorder == "little": base = BigEndianStructure diff --git a/lib-python/3/ctypes/test/test_values.py b/lib-python/3/ctypes/test/test_values.py --- a/lib-python/3/ctypes/test/test_values.py +++ b/lib-python/3/ctypes/test/test_values.py @@ -4,6 +4,7 @@ import unittest import sys +from test.support import cpython_only from ctypes import * import _ctypes_test @@ -28,6 +29,7 @@ ctdll = CDLL(_ctypes_test.__file__) self.assertRaises(ValueError, c_int.in_dll, ctdll, "Undefined_Symbol") + at cpython_only class PythonValuesTestCase(unittest.TestCase): """This test only works when python itself is a dll/shared library""" diff --git a/lib-python/3/datetime.py b/lib-python/3/datetime.py --- a/lib-python/3/datetime.py +++ b/lib-python/3/datetime.py @@ -810,7 +810,8 @@ month = self._month if day is None: day = self._day - return date(year, month, day) + # PyPy fix: returns type(self)() instead of date() + return type(self)(year, month, day) # Comparisons of date objects with other. @@ -1285,7 +1286,8 @@ microsecond = self.microsecond if tzinfo is True: tzinfo = self.tzinfo - return time(hour, minute, second, microsecond, tzinfo) + # PyPy fix: returns type(self)() instead of time() + return type(self)(hour, minute, second, microsecond, tzinfo) # Pickle support. @@ -1497,7 +1499,8 @@ microsecond = self.microsecond if tzinfo is True: tzinfo = self.tzinfo - return datetime(year, month, day, hour, minute, second, microsecond, + # PyPy fix: returns type(self)() instead of datetime() + return type(self)(year, month, day, hour, minute, second, microsecond, tzinfo) def astimezone(self, tz=None): diff --git a/lib-python/3/distutils/sysconfig_pypy.py b/lib-python/3/distutils/sysconfig_pypy.py --- a/lib-python/3/distutils/sysconfig_pypy.py +++ b/lib-python/3/distutils/sysconfig_pypy.py @@ -73,7 +73,7 @@ g['CCSHARED'] = "-fPIC" g['LDSHARED'] = "cc -pthread -shared" g['EXT_SUFFIX'] = so_ext - g['SHLIB_SUFFIX'] = so_ext + g['SHLIB_SUFFIX'] = ".so" g['SO'] = so_ext # deprecated in Python 3, for backward compatibility g['AR'] = "ar" g['ARFLAGS'] = "rc" @@ -81,6 +81,19 @@ g['LIBDIR'] = os.path.join(sys.prefix, 'lib') g['VERSION'] = get_python_version() + if sys.platform[:6] == "darwin": + import platform + if platform.machine() == 'i386': + if platform.architecture()[0] == '32bit': + arch = 'i386' + else: + arch = 'x86_64' + else: + # just a guess + arch = platform.machine() + g['LDSHARED'] += ' -undefined dynamic_lookup' + g['CC'] += ' -arch %s' % (arch,) + global _config_vars _config_vars = g diff --git a/lib-python/3/stat.py b/lib-python/3/stat.py --- a/lib-python/3/stat.py +++ b/lib-python/3/stat.py @@ -139,13 +139,21 @@ def filemode(mode): """Convert a file's mode to a string of the form '-rwxrwxrwx'.""" perm = [] + + # The first group gets a question mark if none of the bits match the mode. + empty = "?" + for table in _filemode_table: for bit, char in table: if mode & bit == bit: perm.append(char) break else: - perm.append("-") + perm.append(empty) + + # All the rest of the positions get a - if the bits don't match. + empty = "-" + return "".join(perm) diff --git a/lib-python/3/test/test_asyncio/test_base_events.py b/lib-python/3/test/test_asyncio/test_base_events.py --- a/lib-python/3/test/test_asyncio/test_base_events.py +++ b/lib-python/3/test/test_asyncio/test_base_events.py @@ -1588,9 +1588,15 @@ sock.getsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR)) if reuseport_supported: - self.assertFalse( - sock.getsockopt( - socket.SOL_SOCKET, socket.SO_REUSEPORT)) + try: + self.assertFalse( + sock.getsockopt( + socket.SOL_SOCKET, socket.SO_REUSEPORT)) + except OSError: + # Python's socket module was compiled using modern headers + # thus defining SO_REUSEPORT but this process is running + # under an older kernel that does not support SO_REUSEPORT. + reuseport_supported = False self.assertFalse( sock.getsockopt( socket.SOL_SOCKET, socket.SO_BROADCAST)) diff --git a/lib-python/3/test/test_importlib/builtin/test_loader.py b/lib-python/3/test/test_importlib/builtin/test_loader.py --- a/lib-python/3/test/test_importlib/builtin/test_loader.py +++ b/lib-python/3/test/test_importlib/builtin/test_loader.py @@ -1,6 +1,8 @@ from .. import abc from .. import util +from importlib.machinery import BuiltinImporter + machinery = util.import_importlib('importlib.machinery') import sys @@ -14,7 +16,7 @@ def setUp(self): self.verification = {'__name__': 'errno', '__package__': '', - '__loader__': self.machinery.BuiltinImporter} + '__loader__': BuiltinImporter} # PyPy change def verify(self, module): """Verify that the module matches against what it should have.""" diff --git a/lib-python/3/test/test_importlib/extension/test_loader.py b/lib-python/3/test/test_importlib/extension/test_loader.py --- a/lib-python/3/test/test_importlib/extension/test_loader.py +++ b/lib-python/3/test/test_importlib/extension/test_loader.py @@ -88,6 +88,7 @@ def setUp(self): self.name = '_testmultiphase' + __import__(self.name) # PyPy hack finder = self.machinery.FileFinder(None) self.spec = importlib.util.find_spec(self.name) assert self.spec @@ -145,7 +146,8 @@ importlib.reload(module) self.assertIs(ex_class, module.Example) - def test_try_registration(self): + # XXX: PyPy doesn't support the PyState_* functions yet + def XXXtest_try_registration(self): '''Assert that the PyState_{Find,Add,Remove}Module C API doesn't work''' module = self.load_module() with self.subTest('PyState_FindModule'): diff --git a/lib-python/3/test/test_marshal.py b/lib-python/3/test/test_marshal.py --- a/lib-python/3/test/test_marshal.py +++ b/lib-python/3/test/test_marshal.py @@ -271,6 +271,11 @@ if n is not None and n > 4: n += 10**6 return n + def read(self, n): # PyPy calls read(), not readinto() + result = super().read(n) + if len(result) > 4: + result += b'\x00' * (10**6) + return result for value in (1.0, 1j, b'0123456789', '0123456789'): self.assertRaises(ValueError, marshal.load, BadReader(marshal.dumps(value))) @@ -348,7 +353,8 @@ strobj = "abcde"*3 dictobj = {"hello":floatobj, "goodbye":floatobj, floatobj:"hello"} - def helper3(self, rsample, recursive=False, simple=False): + def helper3(self, rsample, recursive=False, simple=False, + check_sharing=True, check_non_sharing=True): #we have two instances sample = (rsample, rsample) @@ -358,28 +364,35 @@ n3 = CollectObjectIDs(set(), marshal.loads(s3)) #same number of instances generated - self.assertEqual(n3, n0) + # except in one corner case on top of pypy, for code objects + if check_sharing: + self.assertEqual(n3, n0) if not recursive: #can compare with version 2 s2 = marshal.dumps(sample, 2) n2 = CollectObjectIDs(set(), marshal.loads(s2)) #old format generated more instances - self.assertGreater(n2, n0) + # except on pypy where equal ints or floats always have + # the same id anyway + if check_non_sharing: + self.assertGreater(n2, n0) #if complex objects are in there, old format is larger - if not simple: + if check_non_sharing and not simple: self.assertGreater(len(s2), len(s3)) else: self.assertGreaterEqual(len(s2), len(s3)) def testInt(self): self.helper(self.intobj) - self.helper3(self.intobj, simple=True) + self.helper3(self.intobj, simple=True, + check_non_sharing=support.check_impl_detail()) def testFloat(self): self.helper(self.floatobj) - self.helper3(self.floatobj) + self.helper3(self.floatobj, + check_non_sharing=support.check_impl_detail()) def testStr(self): self.helper(self.strobj) @@ -395,7 +408,7 @@ if __file__.endswith(".py"): code = compile(code, __file__, "exec") self.helper(code) - self.helper3(code) + self.helper3(code, check_sharing=support.check_impl_detail()) def testRecursion(self): d = dict(self.dictobj) diff --git a/lib-python/3/test/test_pyexpat.py b/lib-python/3/test/test_pyexpat.py --- a/lib-python/3/test/test_pyexpat.py +++ b/lib-python/3/test/test_pyexpat.py @@ -11,7 +11,7 @@ from xml.parsers import expat from xml.parsers.expat import errors -from test.support import sortdict +from test.support import sortdict, impl_detail class SetAttributeTest(unittest.TestCase): @@ -446,6 +446,7 @@ self.assertEqual(os.path.basename(entry[0]), filename) self.assertEqual(entry[2], funcname) + @impl_detail("PyPy does not have pyexpat.c", pypy=False) def test_exception(self): parser = expat.ParserCreate() parser.StartElementHandler = self.StartElementHandler diff --git a/lib-python/3/test/test_stat.py b/lib-python/3/test/test_stat.py --- a/lib-python/3/test/test_stat.py +++ b/lib-python/3/test/test_stat.py @@ -138,6 +138,10 @@ self.assertS_IS("REG", st_mode) self.assertEqual(modestr, '-r--r--r--') self.assertEqual(self.statmod.S_IMODE(st_mode), 0o444) + + # If there are only permission bits, no type bytes, a question + # mark is rendered in the type field. + self.assertEqual(self.statmod.filemode(0o420), '?r---w----') else: os.chmod(TESTFN, 0o700) st_mode, modestr = self.get_mode() diff --git a/lib-python/3/test/test_sysconfig.py b/lib-python/3/test/test_sysconfig.py --- a/lib-python/3/test/test_sysconfig.py +++ b/lib-python/3/test/test_sysconfig.py @@ -397,9 +397,16 @@ self.assertTrue('linux' in suffix, suffix) if re.match('(i[3-6]86|x86_64)$', machine): if ctypes.sizeof(ctypes.c_char_p()) == 4: - self.assertTrue(suffix.endswith('i386-linux-gnu.so') \ - or suffix.endswith('x86_64-linux-gnux32.so'), - suffix) + self.assertTrue( + suffix.endswith(( + 'i386-linux-gnu.so', + 'i486-linux-gnu.so', + 'i586-linux-gnu.so', + 'i686-linux-gnu.so', + 'x86_64-linux-gnux32.so', + )), + suffix, + ) else: # 8 byte pointer size self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix) diff --git a/lib_pypy/_cffi_ssl/README.md b/lib_pypy/_cffi_ssl/README.md --- a/lib_pypy/_cffi_ssl/README.md +++ b/lib_pypy/_cffi_ssl/README.md @@ -5,9 +5,15 @@ it renames the compiled shared object to _pypy_openssl.so (which means that cryptography can ship their own cffi backend) -NOTE: currently, we have changed ``_cffi_src/openssl/callbacks.py`` to -not rely on the CPython C API, and ``_cffi_src/utils.py`` for issue #2575 -(29c9a89359e4). (The first change is now backported.) +NOTE: currently, we have the following changes: + +* ``_cffi_src/openssl/callbacks.py`` to not rely on the CPython C API + (this change is now backported) + +* ``_cffi_src/utils.py`` for issue #2575 (29c9a89359e4) + +* ``_cffi_src/openssl/x509_vfy.py`` for issue #2605 (ca4d0c90f5a1) + # Tests? diff --git a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py --- a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py +++ b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py @@ -221,9 +221,13 @@ static const long X509_V_ERR_SUITE_B_INVALID_SIGNATURE_ALGORITHM = 0; static const long X509_V_ERR_SUITE_B_LOS_NOT_ALLOWED = 0; static const long X509_V_ERR_SUITE_B_CANNOT_SIGN_P_384_WITH_P_256 = 0; -#if !CRYPTOGRAPHY_LIBRESSL_251_OR_GREATER +#ifndef X509_V_ERR_HOSTNAME_MISMATCH static const long X509_V_ERR_HOSTNAME_MISMATCH = 0; +#endif +#ifndef X509_V_ERR_EMAIL_MISMATCH static const long X509_V_ERR_EMAIL_MISMATCH = 0; +#endif +#ifndef X509_V_ERR_IP_ADDRESS_MISMATCH static const long X509_V_ERR_IP_ADDRESS_MISMATCH = 0; #endif #endif diff --git a/lib_pypy/_cffi_ssl/_stdssl/certificate.py b/lib_pypy/_cffi_ssl/_stdssl/certificate.py --- a/lib_pypy/_cffi_ssl/_stdssl/certificate.py +++ b/lib_pypy/_cffi_ssl/_stdssl/certificate.py @@ -173,14 +173,13 @@ return tuple(dn) -STATIC_BIO_BUF = ffi.new("char[]", 2048) - def _bio_get_str(biobuf): - length = lib.BIO_gets(biobuf, STATIC_BIO_BUF, len(STATIC_BIO_BUF)-1) + bio_buf = ffi.new("char[]", 2048) + length = lib.BIO_gets(biobuf, bio_buf, len(bio_buf)-1) if length < 0: if biobuf: lib.BIO_free(biobuf) raise ssl_error(None) - return _str_with_len(STATIC_BIO_BUF, length) + return _str_with_len(bio_buf, length) def _decode_certificate(certificate): retval = {} diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -82,7 +82,7 @@ return False def in_dll(self, dll, name): - return self.from_address(dll._handle.getaddressindll(name)) + return self.from_address(dll.__pypy_dll__.getaddressindll(name)) def from_buffer(self, obj, offset=0): size = self._sizeofinstances() diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py --- a/lib_pypy/_ctypes/function.py +++ b/lib_pypy/_ctypes/function.py @@ -430,7 +430,7 @@ ffires = restype.get_ffi_argtype() return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_) - cdll = self.dll._handle + cdll = self.dll.__pypy_dll__ try: ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes] ffi_restype = restype.get_ffi_argtype() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -141,6 +141,10 @@ ptr._buffer = tp._ffiarray(1, autofree=True) ptr._buffer[0] = obj._buffer result = ptr + elif isinstance(obj, bytes): + result = tp() + result._buffer[0] = memoryview(obj)._pypy_raw_address() + return result elif not (isinstance(obj, _CData) and type(obj)._is_pointer_like()): raise TypeError("cast() argument 1 must be a pointer, not %s" % (type(obj),)) diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py --- a/lib_pypy/_ctypes/primitive.py +++ b/lib_pypy/_ctypes/primitive.py @@ -61,6 +61,54 @@ pyobj_container = GlobalPyobjContainer() +def swap_bytes(value, sizeof, typeof, get_or_set): + def swap_2(): + return ((value >> 8) & 0x00FF) | ((value << 8) & 0xFF00) + + def swap_4(): + return ((value & 0x000000FF) << 24) | \ + ((value & 0x0000FF00) << 8) | \ + ((value & 0x00FF0000) >> 8) | \ + ((value >> 24) & 0xFF) + + def swap_8(): + return ((value & 0x00000000000000FF) << 56) | \ + ((value & 0x000000000000FF00) << 40) | \ + ((value & 0x0000000000FF0000) << 24) | \ + ((value & 0x00000000FF000000) << 8) | \ + ((value & 0x000000FF00000000) >> 8) | \ + ((value & 0x0000FF0000000000) >> 24) | \ + ((value & 0x00FF000000000000) >> 40) | \ + ((value >> 56) & 0xFF) + + def swap_double_float(typ): + from struct import pack, unpack + if get_or_set == 'set': + if sys.byteorder == 'little': + st = pack(''.join(['>', typ]), value) + else: + st = pack(''.join(['<', typ]), value) + return unpack(typ, st)[0] + else: + packed = pack(typ, value) + if sys.byteorder == 'little': + st = unpack(''.join(['>', typ]), packed) + else: + st = unpack(''.join(['<', typ]), packed) + return st[0] + + if typeof in ('c_float', 'c_float_le', 'c_float_be'): + return swap_double_float('f') + elif typeof in ('c_double', 'c_double_le', 'c_double_be'): + return swap_double_float('d') + else: + if sizeof == 2: + return swap_2() + elif sizeof == 4: + return swap_4() + elif sizeof == 8: + return swap_8() + def generic_xxx_p_from_param(cls, value): if value is None: return cls(None) @@ -265,6 +313,31 @@ def _as_ffi_pointer_(self, ffitype): return as_ffi_pointer(self, ffitype) result._as_ffi_pointer_ = _as_ffi_pointer_ + if name[-2:] != '_p' and name[-3:] not in ('_le', '_be') \ + and name not in ('c_wchar', '_SimpleCData', 'c_longdouble', 'c_bool', 'py_object'): + from sys import byteorder + if byteorder == 'big': + name += '_le' + swapped = self.__new__(self, name, bases, dct) + result.__ctype_le__ = swapped + result.__ctype_be__ = result + swapped.__ctype_be__ = result + swapped.__ctype_le__ = swapped + else: + name += '_be' + swapped = self.__new__(self, name, bases, dct) + result.__ctype_be__ = swapped + result.__ctype_le__ = result + swapped.__ctype_le__ = result + swapped.__ctype_be__ = swapped + from _ctypes import sizeof + def _getval(self): + return swap_bytes(self._buffer[0], sizeof(self), name, 'get') + def _setval(self, value): + d = result() + d.value = value + self._buffer[0] = swap_bytes(d.value, sizeof(self), name, 'set') + swapped.value = property(_getval, _setval) return result diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -40,6 +40,22 @@ else: rawfields.append((f[0], f[1]._ffishape_)) + # hack for duplicate field names + already_seen = set() + names1 = names + names = [] + for f in names1: + if f not in already_seen: + names.append(f) + already_seen.add(f) + already_seen = set() + for i in reversed(range(len(rawfields))): + if rawfields[i][0] in already_seen: + rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),) + + rawfields[i][1:]) + already_seen.add(rawfields[i][0]) + # /hack + _set_shape(self, rawfields, self._is_union) fields = {} @@ -130,6 +146,7 @@ obj._buffer.__setattr__(self.name, arg) + def _set_shape(tp, rawfields, is_union=False): tp._ffistruct_ = _rawffi.Structure(rawfields, is_union, getattr(tp, '_pack_', 0)) @@ -224,18 +241,26 @@ res.__dict__['_index'] = -1 return res - class StructOrUnion(_CData, metaclass=StructOrUnionMeta): def __new__(cls, *args, **kwds): from _ctypes import union - self = super(_CData, cls).__new__(cls) - if ('_abstract_' in cls.__dict__ or cls is Structure + if ('_abstract_' in cls.__dict__ or cls is Structure or cls is union.Union): raise TypeError("abstract class") if hasattr(cls, '_swappedbytes_'): - raise NotImplementedError("missing in PyPy: structure/union with " - "swapped (non-native) byte ordering") + fields = [None] * len(cls._fields_) + for i in range(len(cls._fields_)): + if cls._fields_[i][1] == cls._fields_[i][1].__dict__.get('__ctype_be__', None): + swapped = cls._fields_[i][1].__dict__.get('__ctype_le__', cls._fields_[i][1]) + else: + swapped = cls._fields_[i][1].__dict__.get('__ctype_be__', cls._fields_[i][1]) + if len(cls._fields_[i]) < 3: + fields[i] = (cls._fields_[i][0], swapped) + else: + fields[i] = (cls._fields_[i][0], swapped, cls._fields_[i][2]) + names_and_fields(cls, fields, _CData, cls.__dict__.get('_anonymous_', None)) + self = super(_CData, cls).__new__(cls) if hasattr(cls, '_ffistruct_'): self.__dict__['_buffer'] = self._ffistruct_(autofree=True) return self diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py --- a/lib_pypy/_curses.py +++ b/lib_pypy/_curses.py @@ -404,14 +404,14 @@ return val def get_wch(self, *args): - wch = ffi.new("int[1]") + wch = ffi.new("wint_t[1]") if len(args) == 0: val = lib.wget_wch(self._win, wch) elif len(args) == 2: val = lib.mvwget_wch(self._win, *args, wch) else: raise error("get_wch requires 0 or 2 arguments") - _check_ERR(val, "get_wch"): + _check_ERR(val, "get_wch") return wch[0] def getkey(self, *args): diff --git a/lib_pypy/_testmultiphase.c b/lib_pypy/_testmultiphase.c new file mode 100644 --- /dev/null +++ b/lib_pypy/_testmultiphase.c @@ -0,0 +1,627 @@ +/* Copied from CPython's Modules/_testmultiphase.c */ +/***************************************************/ + +/* Testing module for multi-phase initialization of extension modules (PEP 489) + */ + +#include "Python.h" + +/* Example objects */ +typedef struct { + PyObject_HEAD + PyObject *x_attr; /* Attributes dictionary */ +} ExampleObject; + +/* Example methods */ + +static int +Example_traverse(ExampleObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->x_attr); + return 0; +} + +static int +Example_finalize(ExampleObject *self) +{ + Py_CLEAR(self->x_attr); + return 0; +} + +static PyObject * +Example_demo(ExampleObject *self, PyObject *args) +{ + PyObject *o = NULL; + if (!PyArg_ParseTuple(args, "|O:demo", &o)) + return NULL; + if (o != NULL && PyUnicode_Check(o)) { + Py_INCREF(o); + return o; + } + Py_INCREF(Py_None); + return Py_None; +} + + +static PyMethodDef Example_methods[] = { + {"demo", (PyCFunction)Example_demo, METH_VARARGS, + PyDoc_STR("demo() -> None")}, + {NULL, NULL} /* sentinel */ +}; + +static PyObject * +Example_getattro(ExampleObject *self, PyObject *name) +{ + if (self->x_attr != NULL) { + PyObject *v = PyDict_GetItem(self->x_attr, name); + if (v != NULL) { + Py_INCREF(v); + return v; + } + } + return PyObject_GenericGetAttr((PyObject *)self, name); +} + +static int +Example_setattr(ExampleObject *self, char *name, PyObject *v) +{ + if (self->x_attr == NULL) { + self->x_attr = PyDict_New(); + if (self->x_attr == NULL) + return -1; + } + if (v == NULL) { + int rv = PyDict_DelItemString(self->x_attr, name); + if (rv < 0) + PyErr_SetString(PyExc_AttributeError, + "delete non-existing Example attribute"); + return rv; + } + else + return PyDict_SetItemString(self->x_attr, name, v); +} + +static PyType_Slot Example_Type_slots[] = { + {Py_tp_doc, "The Example type"}, +// {Py_tp_finalize, Example_finalize}, + {Py_tp_traverse, Example_traverse}, + {Py_tp_getattro, Example_getattro}, + {Py_tp_setattr, Example_setattr}, + {Py_tp_methods, Example_methods}, + {0, 0}, +}; + +static PyType_Spec Example_Type_spec = { + "_testimportexec.Example", + sizeof(ExampleObject), + 0, + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, + Example_Type_slots +}; + +/* Function of two integers returning integer */ + +PyDoc_STRVAR(testexport_foo_doc, +"foo(i,j)\n\ +\n\ +Return the sum of i and j."); + +static PyObject * +testexport_foo(PyObject *self, PyObject *args) +{ + long i, j; + long res; + if (!PyArg_ParseTuple(args, "ll:foo", &i, &j)) + return NULL; + res = i + j; + return PyLong_FromLong(res); +} + +/* Test that PyState registration fails */ + +//PyDoc_STRVAR(call_state_registration_func_doc, +//"register_state(0): call PyState_FindModule()\n\ +//register_state(1): call PyState_AddModule()\n\ +//register_state(2): call PyState_RemoveModule()"); +// +//static PyObject * +//call_state_registration_func(PyObject *mod, PyObject *args) +//{ +// int i, ret; +// PyModuleDef *def = PyModule_GetDef(mod); +// if (def == NULL) { +// return NULL; +// } +// if (!PyArg_ParseTuple(args, "i:call_state_registration_func", &i)) +// return NULL; +// switch (i) { +// case 0: +// mod = PyState_FindModule(def); +// if (mod == NULL) { +// Py_RETURN_NONE; +// } +// return mod; +// case 1: +// ret = PyState_AddModule(mod, def); +// if (ret != 0) { +// return NULL; +// } +// break; +// case 2: +// ret = PyState_RemoveModule(def); +// if (ret != 0) { +// return NULL; +// } +// break; +// } +// Py_RETURN_NONE; +//} + + +static PyType_Slot Str_Type_slots[] = { + {Py_tp_base, NULL}, /* filled out in module exec function */ + {0, 0}, +}; + +static PyType_Spec Str_Type_spec = { + "_testimportexec.Str", + 0, + 0, + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, + Str_Type_slots +}; + +static PyMethodDef testexport_methods[] = { + {"foo", testexport_foo, METH_VARARGS, + testexport_foo_doc}, +// {"call_state_registration_func", call_state_registration_func, +// METH_VARARGS, call_state_registration_func_doc}, + {NULL, NULL} /* sentinel */ +}; + +static int execfunc(PyObject *m) +{ + PyObject *temp = NULL; + + /* Due to cross platform compiler issues the slots must be filled + * here. It's required for portability to Windows without requiring + * C++. */ + Str_Type_slots[0].pfunc = &PyUnicode_Type; + + /* Add a custom type */ + temp = PyType_FromSpec(&Example_Type_spec); + if (temp == NULL) + goto fail; + if (PyModule_AddObject(m, "Example", temp) != 0) + goto fail; + + /* Add an exception type */ + temp = PyErr_NewException("_testimportexec.error", NULL, NULL); + if (temp == NULL) + goto fail; + if (PyModule_AddObject(m, "error", temp) != 0) + goto fail; + + /* Add Str */ + temp = PyType_FromSpec(&Str_Type_spec); + if (temp == NULL) + goto fail; + if (PyModule_AddObject(m, "Str", temp) != 0) + goto fail; + + if (PyModule_AddIntConstant(m, "int_const", 1969) != 0) + goto fail; + + if (PyModule_AddStringConstant(m, "str_const", "something different") != 0) + goto fail; + + return 0; + fail: + return -1; +} + +/* Helper for module definitions; there'll be a lot of them */ +#define TEST_MODULE_DEF(name, slots, methods) { \ + PyModuleDef_HEAD_INIT, /* m_base */ \ + name, /* m_name */ \ + PyDoc_STR("Test module " name), /* m_doc */ \ + 0, /* m_size */ \ + methods, /* m_methods */ \ + slots, /* m_slots */ \ + NULL, /* m_traverse */ \ + NULL, /* m_clear */ \ + NULL, /* m_free */ \ +} + +PyModuleDef_Slot main_slots[] = { + {Py_mod_exec, execfunc}, + {0, NULL}, +}; + +static PyModuleDef main_def = TEST_MODULE_DEF("main", main_slots, testexport_methods); + +PyMODINIT_FUNC +PyInit__testmultiphase(PyObject *spec) +{ + return PyModuleDef_Init(&main_def); +} + + +/**** Importing a non-module object ****/ + +static PyModuleDef def_nonmodule; +static PyModuleDef def_nonmodule_with_methods; + +/* Create a SimpleNamespace(three=3) */ +static PyObject* +createfunc_nonmodule(PyObject *spec, PyModuleDef *def) +{ + PyObject *dct, *ns, *three; + + if (def != &def_nonmodule && def != &def_nonmodule_with_methods) { + PyErr_SetString(PyExc_SystemError, "def does not match"); + return NULL; + } + + dct = PyDict_New(); + if (dct == NULL) + return NULL; + + three = PyLong_FromLong(3); + if (three == NULL) { + Py_DECREF(dct); + return NULL; + } + PyDict_SetItemString(dct, "three", three); + Py_DECREF(three); + + ns = _PyNamespace_New(dct); + Py_DECREF(dct); + return ns; +} + +static PyModuleDef_Slot slots_create_nonmodule[] = { + {Py_mod_create, createfunc_nonmodule}, + {0, NULL}, +}; + +static PyModuleDef def_nonmodule = TEST_MODULE_DEF( + "_testmultiphase_nonmodule", slots_create_nonmodule, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_nonmodule(PyObject *spec) +{ + return PyModuleDef_Init(&def_nonmodule); +} + +PyDoc_STRVAR(nonmodule_bar_doc, +"bar(i,j)\n\ +\n\ +Return the difference of i - j."); + +static PyObject * +nonmodule_bar(PyObject *self, PyObject *args) +{ + long i, j; + long res; + if (!PyArg_ParseTuple(args, "ll:bar", &i, &j)) + return NULL; + res = i - j; + return PyLong_FromLong(res); +} + +static PyMethodDef nonmodule_methods[] = { + {"bar", nonmodule_bar, METH_VARARGS, nonmodule_bar_doc}, + {NULL, NULL} /* sentinel */ +}; + +static PyModuleDef def_nonmodule_with_methods = TEST_MODULE_DEF( + "_testmultiphase_nonmodule_with_methods", slots_create_nonmodule, nonmodule_methods); + +PyMODINIT_FUNC +PyInit__testmultiphase_nonmodule_with_methods(PyObject *spec) +{ + return PyModuleDef_Init(&def_nonmodule_with_methods); +} + +/**** Non-ASCII-named modules ****/ + +static PyModuleDef def_nonascii_latin = { \ + PyModuleDef_HEAD_INIT, /* m_base */ + "_testmultiphase_nonascii_latin", /* m_name */ + PyDoc_STR("Module named in Czech"), /* m_doc */ + 0, /* m_size */ + NULL, /* m_methods */ + NULL, /* m_slots */ + NULL, /* m_traverse */ + NULL, /* m_clear */ + NULL, /* m_free */ +}; + +PyMODINIT_FUNC +PyInitU__testmultiphase_zkouka_naten_evc07gi8e(PyObject *spec) +{ + return PyModuleDef_Init(&def_nonascii_latin); +} + +static PyModuleDef def_nonascii_kana = { \ + PyModuleDef_HEAD_INIT, /* m_base */ + "_testmultiphase_nonascii_kana", /* m_name */ + PyDoc_STR("Module named in Japanese"), /* m_doc */ + 0, /* m_size */ + NULL, /* m_methods */ + NULL, /* m_slots */ + NULL, /* m_traverse */ + NULL, /* m_clear */ + NULL, /* m_free */ +}; + +PyMODINIT_FUNC +PyInitU_eckzbwbhc6jpgzcx415x(PyObject *spec) +{ + return PyModuleDef_Init(&def_nonascii_kana); +} + +/*** Module with a single-character name ***/ + +PyMODINIT_FUNC +PyInit_x(PyObject *spec) +{ + return PyModuleDef_Init(&main_def); +} + +/**** Testing NULL slots ****/ + +static PyModuleDef null_slots_def = TEST_MODULE_DEF( + "_testmultiphase_null_slots", NULL, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_null_slots(PyObject *spec) +{ + return PyModuleDef_Init(&null_slots_def); +} + +/**** Problematic modules ****/ + +static PyModuleDef_Slot slots_bad_large[] = { + {_Py_mod_LAST_SLOT + 1, NULL}, + {0, NULL}, +}; + +static PyModuleDef def_bad_large = TEST_MODULE_DEF( + "_testmultiphase_bad_slot_large", slots_bad_large, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_bad_slot_large(PyObject *spec) +{ + return PyModuleDef_Init(&def_bad_large); +} + +static PyModuleDef_Slot slots_bad_negative[] = { + {-1, NULL}, + {0, NULL}, +}; + +static PyModuleDef def_bad_negative = TEST_MODULE_DEF( + "_testmultiphase_bad_slot_negative", slots_bad_negative, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_bad_slot_negative(PyObject *spec) +{ + return PyModuleDef_Init(&def_bad_negative); +} + +static PyModuleDef def_create_int_with_state = { \ + PyModuleDef_HEAD_INIT, /* m_base */ + "create_with_state", /* m_name */ + PyDoc_STR("Not a PyModuleObject object, but requests per-module state"), + 10, /* m_size */ + NULL, /* m_methods */ + slots_create_nonmodule, /* m_slots */ + NULL, /* m_traverse */ + NULL, /* m_clear */ + NULL, /* m_free */ +}; + +PyMODINIT_FUNC +PyInit__testmultiphase_create_int_with_state(PyObject *spec) +{ + return PyModuleDef_Init(&def_create_int_with_state); +} + + +static PyModuleDef def_negative_size = { \ + PyModuleDef_HEAD_INIT, /* m_base */ + "negative_size", /* m_name */ + PyDoc_STR("PyModuleDef with negative m_size"), + -1, /* m_size */ + NULL, /* m_methods */ + slots_create_nonmodule, /* m_slots */ + NULL, /* m_traverse */ + NULL, /* m_clear */ + NULL, /* m_free */ +}; + +PyMODINIT_FUNC +PyInit__testmultiphase_negative_size(PyObject *spec) +{ + return PyModuleDef_Init(&def_negative_size); +} + + +static PyModuleDef uninitialized_def = TEST_MODULE_DEF("main", main_slots, testexport_methods); + +PyMODINIT_FUNC +PyInit__testmultiphase_export_uninitialized(PyObject *spec) +{ + return (PyObject*) &uninitialized_def; +} + +PyMODINIT_FUNC +PyInit__testmultiphase_export_null(PyObject *spec) +{ + return NULL; +} + +PyMODINIT_FUNC +PyInit__testmultiphase_export_raise(PyObject *spec) +{ + PyErr_SetString(PyExc_SystemError, "bad export function"); + return NULL; +} + +PyMODINIT_FUNC +PyInit__testmultiphase_export_unreported_exception(PyObject *spec) +{ + PyErr_SetString(PyExc_SystemError, "bad export function"); + return PyModuleDef_Init(&main_def); +} + +static PyObject* +createfunc_null(PyObject *spec, PyModuleDef *def) +{ + return NULL; +} + +PyModuleDef_Slot slots_create_null[] = { + {Py_mod_create, createfunc_null}, + {0, NULL}, +}; + +static PyModuleDef def_create_null = TEST_MODULE_DEF( + "_testmultiphase_create_null", slots_create_null, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_create_null(PyObject *spec) +{ + return PyModuleDef_Init(&def_create_null); +} + +static PyObject* +createfunc_raise(PyObject *spec, PyModuleDef *def) +{ + PyErr_SetString(PyExc_SystemError, "bad create function"); + return NULL; +} + +static PyModuleDef_Slot slots_create_raise[] = { + {Py_mod_create, createfunc_raise}, + {0, NULL}, +}; + +static PyModuleDef def_create_raise = TEST_MODULE_DEF( + "_testmultiphase_create_null", slots_create_raise, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_create_raise(PyObject *spec) +{ + return PyModuleDef_Init(&def_create_raise); +} + +static PyObject* +createfunc_unreported_exception(PyObject *spec, PyModuleDef *def) +{ + PyErr_SetString(PyExc_SystemError, "bad create function"); + return PyModule_New("foo"); +} + +static PyModuleDef_Slot slots_create_unreported_exception[] = { + {Py_mod_create, createfunc_unreported_exception}, + {0, NULL}, +}; + +static PyModuleDef def_create_unreported_exception = TEST_MODULE_DEF( + "_testmultiphase_create_unreported_exception", slots_create_unreported_exception, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_create_unreported_exception(PyObject *spec) +{ + return PyModuleDef_Init(&def_create_unreported_exception); +} + +static PyModuleDef_Slot slots_nonmodule_with_exec_slots[] = { + {Py_mod_create, createfunc_nonmodule}, + {Py_mod_exec, execfunc}, + {0, NULL}, +}; + +static PyModuleDef def_nonmodule_with_exec_slots = TEST_MODULE_DEF( + "_testmultiphase_nonmodule_with_exec_slots", slots_nonmodule_with_exec_slots, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_nonmodule_with_exec_slots(PyObject *spec) +{ + return PyModuleDef_Init(&def_nonmodule_with_exec_slots); +} + +static int +execfunc_err(PyObject *mod) +{ + return -1; +} + +static PyModuleDef_Slot slots_exec_err[] = { + {Py_mod_exec, execfunc_err}, + {0, NULL}, +}; + +static PyModuleDef def_exec_err = TEST_MODULE_DEF( + "_testmultiphase_exec_err", slots_exec_err, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_exec_err(PyObject *spec) +{ + return PyModuleDef_Init(&def_exec_err); +} + +static int +execfunc_raise(PyObject *spec) +{ + PyErr_SetString(PyExc_SystemError, "bad exec function"); + return -1; +} + +static PyModuleDef_Slot slots_exec_raise[] = { + {Py_mod_exec, execfunc_raise}, + {0, NULL}, +}; + +static PyModuleDef def_exec_raise = TEST_MODULE_DEF( + "_testmultiphase_exec_raise", slots_exec_raise, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_exec_raise(PyObject *mod) +{ + return PyModuleDef_Init(&def_exec_raise); +} + +static int +execfunc_unreported_exception(PyObject *mod) +{ + PyErr_SetString(PyExc_SystemError, "bad exec function"); + return 0; +} + +static PyModuleDef_Slot slots_exec_unreported_exception[] = { + {Py_mod_exec, execfunc_unreported_exception}, + {0, NULL}, +}; + +static PyModuleDef def_exec_unreported_exception = TEST_MODULE_DEF( + "_testmultiphase_exec_unreported_exception", slots_exec_unreported_exception, NULL); + +PyMODINIT_FUNC +PyInit__testmultiphase_exec_unreported_exception(PyObject *spec) +{ + return PyModuleDef_Init(&def_exec_unreported_exception); +} + +/*** Helper for imp test ***/ + +static PyModuleDef imp_dummy_def = TEST_MODULE_DEF("imp_dummy", main_slots, testexport_methods); + +PyMODINIT_FUNC +PyInit_imp_dummy(PyObject *spec) +{ + return PyModuleDef_Init(&imp_dummy_def); +} diff --git a/lib_pypy/_testmultiphase.py b/lib_pypy/_testmultiphase.py new file mode 100644 --- /dev/null +++ b/lib_pypy/_testmultiphase.py @@ -0,0 +1,18 @@ +import imp +import os + +try: + import cpyext +except ImportError: + raise ImportError("No module named '_testmultiphase'") +import _pypy_testcapi +cfile = '_testmultiphase.c' +thisdir = os.path.dirname(__file__) +output_dir = _pypy_testcapi.get_hashed_dir(os.path.join(thisdir, cfile)) +try: + fp, filename, description = imp.find_module('_test_multiphase', path=[output_dir]) + with fp: + imp.load_module('_testmultiphase', fp, filename, description) +except ImportError: + print('could not find _testmultiphase in %s' % output_dir) + _pypy_testcapi.compile_shared('_testmultiphase.c', '_testmultiphase', output_dir) diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py --- a/lib_pypy/_tkinter/tklib_build.py +++ b/lib_pypy/_tkinter/tklib_build.py @@ -22,12 +22,27 @@ linklibs = ['tcl', 'tk'] libdirs = [] else: - for _ver in ['', '8.6', '8.5', '']: + # On some Linux distributions, the tcl and tk libraries are + # stored in /usr/include, so we must check this case also + libdirs = [] + found = False + for _ver in ['', '8.6', '8.5']: incdirs = ['/usr/include/tcl' + _ver] linklibs = ['tcl' + _ver, 'tk' + _ver] - libdirs = [] if os.path.isdir(incdirs[0]): + found = True break + if not found: + for _ver in ['8.6', '8.5', '']: + incdirs = [] + linklibs = ['tcl' + _ver, 'tk' + _ver] + if os.path.isfile(''.join(['/usr/lib/lib', linklibs[1], '.so'])): + found = True + break + if not found: + sys.stderr.write("*** TCL libraries not found! Falling back...\n") + incdirs = [] + linklibs = ['tcl', 'tk'] config_ffi = FFI() config_ffi.cdef(""" diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.0 +Version: 1.11.1 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.0" -__version_info__ = (1, 11, 0) +__version__ = "1.11.1" +__version_info__ = (1, 11, 1) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -95,6 +95,7 @@ #define _cffi_from_c_ulong PyLong_FromUnsignedLong #define _cffi_from_c_longlong PyLong_FromLongLong #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong +#define _cffi_from_c__Bool PyBool_FromLong #define _cffi_to_c_double PyFloat_AsDouble #define _cffi_to_c_float PyFloat_AsDouble diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -1,7 +1,12 @@ /***** Support code for embedding *****/ -#if defined(_MSC_VER) +#ifdef __cplusplus +extern "C" { +#endif + + +#if defined(_WIN32) # define CFFI_DLLEXPORT __declspec(dllexport) #elif defined(__GNUC__) # define CFFI_DLLEXPORT __attribute__((visibility("default"))) @@ -242,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.0" + "\ncompiled with cffi version: 1.11.1" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); @@ -525,3 +530,7 @@ #undef cffi_compare_and_swap #undef cffi_write_barrier #undef cffi_read_barrier + +#ifdef __cplusplus +} +#endif diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py --- a/lib_pypy/cffi/api.py +++ b/lib_pypy/cffi/api.py @@ -394,12 +394,17 @@ replace_with = ' ' + replace_with return self._backend.getcname(cdecl, replace_with) - def gc(self, cdata, destructor): + def gc(self, cdata, destructor, size=0): """Return a new cdata object that points to the same data. Later, when this new cdata object is garbage-collected, 'destructor(old_cdata_object)' will be called. + + The optional 'size' gives an estimate of the size, used to + trigger the garbage collection more eagerly. So far only used + on PyPy. It tells the GC that the returned object keeps alive + roughly 'size' bytes of external memory. """ - return self._backend.gcp(cdata, destructor) + return self._backend.gcp(cdata, destructor, size) def _get_cached_btype(self, type): assert self._lock.acquire(False) is False diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py --- a/lib_pypy/cffi/backend_ctypes.py +++ b/lib_pypy/cffi/backend_ctypes.py @@ -1002,7 +1002,7 @@ _weakref_cache_ref = None - def gcp(self, cdata, destructor): + def gcp(self, cdata, destructor, size=0): if self._weakref_cache_ref is None: import weakref class MyRef(weakref.ref): diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py --- a/lib_pypy/cffi/recompiler.py +++ b/lib_pypy/cffi/recompiler.py @@ -412,6 +412,9 @@ prnt(' }') prnt(' p[0] = (const void *)0x%x;' % self._version) prnt(' p[1] = &_cffi_type_context;') + prnt('#if PY_MAJOR_VERSION >= 3') + prnt(' return NULL;') + prnt('#endif') prnt('}') # on Windows, distutils insists on putting init_cffi_xyz in # 'export_symbols', so instead of fighting it, just give up and @@ -578,7 +581,7 @@ def _convert_expr_from_c(self, tp, var, context): if isinstance(tp, model.BasePrimitiveType): - if tp.is_integer_type(): + if tp.is_integer_type() and tp.name != '_Bool': return '_cffi_from_c_int(%s, %s)' % (var, tp.name) elif isinstance(tp, model.UnknownFloatType): return '_cffi_from_c_double(%s)' % (var,) diff --git a/lib_pypy/cffi/vengine_cpy.py b/lib_pypy/cffi/vengine_cpy.py --- a/lib_pypy/cffi/vengine_cpy.py +++ b/lib_pypy/cffi/vengine_cpy.py @@ -296,7 +296,7 @@ def _convert_expr_from_c(self, tp, var, context): if isinstance(tp, model.PrimitiveType): - if tp.is_integer_type(): + if tp.is_integer_type() and tp.name != '_Bool': return '_cffi_from_c_int(%s, %s)' % (var, tp.name) elif tp.name != 'long double': return '_cffi_from_c_%s(%s)' % (tp.name.replace(' ', '_'), var) @@ -872,6 +872,7 @@ #define _cffi_from_c_ulong PyLong_FromUnsignedLong #define _cffi_from_c_longlong PyLong_FromLongLong #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong +#define _cffi_from_c__Bool PyBool_FromLong #define _cffi_to_c_double PyFloat_AsDouble #define _cffi_to_c_float PyFloat_AsDouble diff --git a/lib_pypy/pyrepl/historical_reader.py b/lib_pypy/pyrepl/historical_reader.py --- a/lib_pypy/pyrepl/historical_reader.py +++ b/lib_pypy/pyrepl/historical_reader.py @@ -17,7 +17,7 @@ # CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN # CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. -from pyrepl import reader, commands +from pyrepl import reader, commands, input from pyrepl.reader import Reader as R isearch_keymap = tuple( @@ -215,7 +215,6 @@ isearch_forwards, isearch_backwards, operate_and_get_next]: self.commands[c.__name__] = c self.commands[c.__name__.replace('_', '-')] = c - from pyrepl import input self.isearch_trans = input.KeymapTranslator( isearch_keymap, invalid_cls=isearch_end, character_cls=isearch_add_character) diff --git a/lib_pypy/pyrepl/reader.py b/lib_pypy/pyrepl/reader.py --- a/lib_pypy/pyrepl/reader.py +++ b/lib_pypy/pyrepl/reader.py @@ -239,6 +239,10 @@ def __init__(self, console): self.buffer = [] + # Enable the use of `insert` without a `prepare` call - necessary to + # facilitate the tab completion hack implemented for + # . + self.pos = 0 self.ps1 = "->> " self.ps2 = "/>> " self.ps3 = "|.. " From pypy.commits at gmail.com Tue Oct 10 03:35:34 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 00:35:34 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: default to embedding dependencies on OS X Message-ID: <59dc7846.532f1c0a.49fef.a899@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92696:c26613e63483 Date: 2017-10-07 15:03 +0200 http://bitbucket.org/pypy/pypy/changeset/c26613e63483/ Log: default to embedding dependencies on OS X diff --git a/pypy/tool/release/package.py b/pypy/tool/release/package.py --- a/pypy/tool/release/package.py +++ b/pypy/tool/release/package.py @@ -270,6 +270,15 @@ def package(*args, **kwds): import argparse + + class NegateAction(argparse.Action): + def __init__(self, option_strings, dest, nargs=0, **kwargs): + super(NegateAction, self).__init__(option_strings, dest, nargs, + **kwargs) + + def __call__(self, parser, ns, values, option): + setattr(ns, self.dest, option[2:4] != 'no') + if sys.platform == 'win32': pypy_exe = 'pypy3.exe' else: @@ -300,9 +309,12 @@ help='destination dir for archive') parser.add_argument('--override_pypy_c', type=str, default='', help='use as pypy3 exe instead of pypy/goal/pypy3-c') - parser.add_argument('--embedded-dependencies', dest='embed_dependencies', - action='store_true', - help='embed dependencies for distribution') + parser.add_argument('--embedded-dependencies', '--no-embedded-dependencies', + dest='embed_dependencies', + action=NegateAction, + default=sys.platform == 'darwin', + help='whether to embed dependencies for distribution ' + '(default on OS X)') options = parser.parse_args(args) if os.environ.has_key("PYPY_PACKAGE_NOSTRIP"): From pypy.commits at gmail.com Tue Oct 10 04:33:37 2017 From: pypy.commits at gmail.com (antocuni) Date: Tue, 10 Oct 2017 01:33:37 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): we can no longer call decref on w_obj now, because make_ref returns an unliked PyObject for integers. Fix the test Message-ID: <59dc85e1.09a0df0a.ea0b7.2f66@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92697:a6058690cb67 Date: 2017-10-10 10:32 +0200 http://bitbucket.org/pypy/pypy/changeset/a6058690cb67/ Log: (antocuni, arigo): we can no longer call decref on w_obj now, because make_ref returns an unliked PyObject for integers. Fix the test diff --git a/pypy/module/cpyext/test/test_number.py b/pypy/module/cpyext/test/test_number.py --- a/pypy/module/cpyext/test/test_number.py +++ b/pypy/module/cpyext/test/test_number.py @@ -54,19 +54,23 @@ def test_coerce(self, space): w_obj1 = space.wrap(123) w_obj2 = space.wrap(456.789) + p1 = make_ref(space, w_obj1) + p2 = make_ref(space, w_obj2) pp1 = lltype.malloc(PyObjectP.TO, 1, flavor='raw') - pp1[0] = make_ref(space, w_obj1) + pp1[0] = p1 pp2 = lltype.malloc(PyObjectP.TO, 1, flavor='raw') - pp2[0] = make_ref(space, w_obj2) + pp2[0] = p2 assert PyNumber_Coerce(space, pp1, pp2) == 0 assert space.str_w(space.repr(from_ref(space, pp1[0]))) == '123.0' assert space.str_w(space.repr(from_ref(space, pp2[0]))) == '456.789' + # + # We need to decref twice because PyNumber_Coerce does an incref and + # possibly changes the content of pp1 and pp2 + Py_DecRef(space, p1) Py_DecRef(space, pp1[0]) + Py_DecRef(space, p2) Py_DecRef(space, pp2[0]) lltype.free(pp1, flavor='raw') - # Yes, decrement twice since we decoupled between w_obj* and pp*[0]. - Py_DecRef(space, w_obj1) - Py_DecRef(space, w_obj2) lltype.free(pp2, flavor='raw') def test_number_coerce_ex(self, space): From pypy.commits at gmail.com Tue Oct 10 05:14:14 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 02:14:14 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59dc8f66.08691c0a.f9da8.b1b9@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92698:51b3443b1148 Date: 2017-10-10 11:11 +0200 http://bitbucket.org/pypy/pypy/changeset/51b3443b1148/ Log: (antocuni, arigo) Try to be less magic and avoid the basic functions like decref() accepting both PyObjects and W_Roots. Rename Py_DecRef() to decref(). diff --git a/pypy/module/cpyext/bufferobject.py b/pypy/module/cpyext/bufferobject.py --- a/pypy/module/cpyext/bufferobject.py +++ b/pypy/module/cpyext/bufferobject.py @@ -4,7 +4,7 @@ from pypy.module.cpyext.api import ( cpython_api, Py_ssize_t, cpython_struct, bootstrap_function, slot_function, PyObjectFields, PyObject) -from pypy.module.cpyext.pyobject import make_typedescr, Py_DecRef, make_ref +from pypy.module.cpyext.pyobject import make_typedescr, decref, make_ref from pypy.module.array.interp_array import ArrayBuffer from pypy.objspace.std.bufferobject import W_Buffer @@ -76,7 +76,7 @@ def buffer_dealloc(space, py_obj): py_buf = rffi.cast(PyBufferObject, py_obj) if py_buf.c_b_base: - Py_DecRef(space, py_buf.c_b_base) + decref(space, py_buf.c_b_base) else: rffi.free_charp(rffi.cast(rffi.CCHARP, py_buf.c_b_ptr)) from pypy.module.cpyext.object import _dealloc diff --git a/pypy/module/cpyext/bytearrayobject.py b/pypy/module/cpyext/bytearrayobject.py --- a/pypy/module/cpyext/bytearrayobject.py +++ b/pypy/module/cpyext/bytearrayobject.py @@ -7,8 +7,8 @@ PyVarObjectFields, Py_ssize_t, CONST_STRING, CANNOT_FAIL) from pypy.module.cpyext.pyerrors import PyErr_BadArgument from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, Py_DecRef, make_ref, from_ref, - make_typedescr, get_typedescr, Py_IncRef) + PyObject, PyObjectP, make_ref, from_ref, + make_typedescr, get_typedescr) # Type PyByteArrayObject represents a mutable array of bytes. # The Python API is that of a sequence; # the bytes are mapped to ints in [0, 256). diff --git a/pypy/module/cpyext/bytesobject.py b/pypy/module/cpyext/bytesobject.py --- a/pypy/module/cpyext/bytesobject.py +++ b/pypy/module/cpyext/bytesobject.py @@ -5,8 +5,8 @@ PyVarObjectFields, Py_ssize_t, CONST_STRING, CANNOT_FAIL, slot_function) from pypy.module.cpyext.pyerrors import PyErr_BadArgument from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, Py_DecRef, make_ref, from_ref, track_reference, - make_typedescr, get_typedescr, as_pyobj, Py_IncRef, get_w_obj_and_decref, + PyObject, PyObjectP, decref, make_ref, from_ref, track_reference, + make_typedescr, get_typedescr, as_pyobj, get_w_obj_and_decref, pyobj_has_w_obj) from pypy.objspace.std.bytesobject import W_BytesObject @@ -215,7 +215,7 @@ try: py_newstr = new_empty_str(space, newsize) except MemoryError: - Py_DecRef(space, ref[0]) + decref(space, ref[0]) ref[0] = lltype.nullptr(PyObject.TO) raise to_cp = newsize @@ -224,7 +224,7 @@ to_cp = oldsize for i in range(to_cp): py_newstr.c_ob_sval[i] = py_str.c_ob_sval[i] - Py_DecRef(space, ref[0]) + decref(space, ref[0]) ref[0] = rffi.cast(PyObject, py_newstr) return 0 @@ -260,7 +260,7 @@ try: PyString_Concat(space, ref, newpart) finally: - Py_DecRef(space, newpart) + decref(space, newpart) @cpython_api([PyObject, PyObject], PyObject) def PyString_Format(space, w_format, w_args): @@ -294,7 +294,7 @@ alias.""" w_str = from_ref(space, string[0]) w_str = space.new_interned_w_str(w_str) - Py_DecRef(space, string[0]) + decref(space, string[0]) string[0] = make_ref(space, w_str) @cpython_api([PyObject, CONST_STRING, CONST_STRING], PyObject) diff --git a/pypy/module/cpyext/classobject.py b/pypy/module/cpyext/classobject.py --- a/pypy/module/cpyext/classobject.py +++ b/pypy/module/cpyext/classobject.py @@ -2,7 +2,8 @@ from pypy.module.cpyext.api import ( PyObjectFields, CANNOT_FAIL, cpython_api, bootstrap_function, build_type_checkers) -from pypy.module.cpyext.pyobject import PyObject, make_ref, from_ref, Py_DecRef, make_typedescr +from pypy.module.cpyext.pyobject import ( + PyObject, make_ref, from_ref, make_typedescr) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.module.__builtin__.interp_classobj import W_ClassObject, W_InstanceObject diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -9,7 +9,7 @@ bootstrap_function, slot_function) from pypy.module.cpyext.pyobject import (PyObject, PyObjectP, as_pyobj, make_typedescr, track_reference, create_ref, from_ref, decref, - Py_IncRef) + incref) from pypy.module.cpyext.object import _dealloc from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall @@ -267,7 +267,7 @@ decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) py_dict.c__tmpkeys = create_ref(space, w_keys) - Py_IncRef(space, py_dict.c__tmpkeys) + incref(space, py_dict.c__tmpkeys) else: if not py_dict.c__tmpkeys: # pos should have been 0, cannot fail so return 0 diff --git a/pypy/module/cpyext/frameobject.py b/pypy/module/cpyext/frameobject.py --- a/pypy/module/cpyext/frameobject.py +++ b/pypy/module/cpyext/frameobject.py @@ -3,7 +3,7 @@ cpython_api, bootstrap_function, PyObjectFields, cpython_struct, CANNOT_FAIL, slot_function) from pypy.module.cpyext.pyobject import ( - PyObject, Py_DecRef, make_ref, from_ref, track_reference, + PyObject, decref, make_ref, from_ref, track_reference, make_typedescr, get_typedescr) from pypy.module.cpyext.state import State from pypy.module.cpyext.pystate import PyThreadState @@ -43,9 +43,9 @@ def frame_dealloc(space, py_obj): py_frame = rffi.cast(PyFrameObject, py_obj) py_code = rffi.cast(PyObject, py_frame.c_f_code) - Py_DecRef(space, py_code) - Py_DecRef(space, py_frame.c_f_globals) - Py_DecRef(space, py_frame.c_f_locals) + decref(space, py_code) + decref(space, py_frame.c_f_globals) + decref(space, py_frame.c_f_locals) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) diff --git a/pypy/module/cpyext/funcobject.py b/pypy/module/cpyext/funcobject.py --- a/pypy/module/cpyext/funcobject.py +++ b/pypy/module/cpyext/funcobject.py @@ -4,7 +4,7 @@ cpython_api, bootstrap_function, cpython_struct, build_type_checkers, slot_function) from pypy.module.cpyext.pyobject import ( - PyObject, make_ref, from_ref, Py_DecRef, make_typedescr) + PyObject, make_ref, from_ref, decref, make_typedescr) from rpython.rlib.unroll import unrolling_iterable from pypy.interpreter.error import OperationError from pypy.interpreter.function import Function, Method @@ -60,7 +60,7 @@ @slot_function([PyObject], lltype.Void) def function_dealloc(space, py_obj): py_func = rffi.cast(PyFunctionObject, py_obj) - Py_DecRef(space, py_func.c_func_name) + decref(space, py_func.c_func_name) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) @@ -79,8 +79,8 @@ @slot_function([PyObject], lltype.Void) def code_dealloc(space, py_obj): py_code = rffi.cast(PyCodeObject, py_obj) - Py_DecRef(space, py_code.c_co_name) - Py_DecRef(space, py_code.c_co_filename) + decref(space, py_code.c_co_name) + decref(space, py_code.c_co_filename) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -14,7 +14,7 @@ cpython_api, generic_cpy_call, CANNOT_FAIL, slot_function, cts, build_type_checkers) from pypy.module.cpyext.pyobject import ( - Py_DecRef, from_ref, make_ref, as_pyobj, make_typedescr) + decref, from_ref, make_ref, as_pyobj, make_typedescr) PyMethodDef = cts.gettype('PyMethodDef') PyCFunction = cts.gettype('PyCFunction') @@ -38,8 +38,8 @@ @slot_function([PyObject], lltype.Void) def cfunction_dealloc(space, py_obj): py_func = rffi.cast(PyCFunctionObject, py_obj) - Py_DecRef(space, py_func.c_m_self) - Py_DecRef(space, py_func.c_m_module) + decref(space, py_func.c_m_self) + decref(space, py_func.c_m_module) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) diff --git a/pypy/module/cpyext/number.py b/pypy/module/cpyext/number.py --- a/pypy/module/cpyext/number.py +++ b/pypy/module/cpyext/number.py @@ -1,6 +1,6 @@ from pypy.interpreter.error import OperationError, oefmt from pypy.module.cpyext.api import cpython_api, CANNOT_FAIL, Py_ssize_t -from pypy.module.cpyext.pyobject import PyObject, PyObjectP, from_ref, make_ref, Py_DecRef +from pypy.module.cpyext.pyobject import PyObject, PyObjectP, from_ref, make_ref from rpython.rtyper.lltypesystem import rffi, lltype from rpython.tool.sourcetools import func_with_new_name from pypy.module.cpyext.state import State diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -5,7 +5,7 @@ Py_TPFLAGS_HEAPTYPE, Py_LT, Py_LE, Py_EQ, Py_NE, Py_GT, Py_GE, CONST_STRING, FILEP, fwrite, c_only) from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, from_ref, Py_IncRef, Py_DecRef, + PyObject, PyObjectP, from_ref, incref, decref, get_typedescr) from pypy.module.cpyext.typeobject import PyTypeObjectPtr from pypy.module.cpyext.pyerrors import PyErr_NoMemory, PyErr_BadInternalCall @@ -55,7 +55,7 @@ obj_voidp = rffi.cast(rffi.VOIDP, obj) generic_cpy_call(space, pto.c_tp_free, obj_voidp) if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: - Py_DecRef(space, rffi.cast(PyObject, pto)) + decref(space, rffi.cast(PyObject, pto)) @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL) def _PyObject_GetDictPtr(space, op): @@ -267,7 +267,7 @@ @cpython_api([PyObject], PyObject, result_is_ll=True) def PyObject_SelfIter(space, ref): """Undocumented function, this is what CPython does.""" - Py_IncRef(space, ref) + incref(space, ref) return ref @cpython_api([PyObject, PyObject], PyObject) diff --git a/pypy/module/cpyext/pyerrors.py b/pypy/module/cpyext/pyerrors.py --- a/pypy/module/cpyext/pyerrors.py +++ b/pypy/module/cpyext/pyerrors.py @@ -6,7 +6,7 @@ from pypy.module.cpyext.api import cpython_api, CANNOT_FAIL, CONST_STRING from pypy.module.exceptions.interp_exceptions import W_RuntimeWarning from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, make_ref, from_ref, Py_DecRef) + PyObject, PyObjectP, make_ref, from_ref, decref) from pypy.module.cpyext.state import State from pypy.module.cpyext.import_ import PyImport_Import from rpython.rlib import rposix, jit @@ -65,7 +65,7 @@ ptraceback[0] = lltype.nullptr(PyObject.TO) @cpython_api([PyObject, PyObject, PyObject], lltype.Void) -def PyErr_Restore(space, w_type, w_value, w_traceback): +def PyErr_Restore(space, py_type, py_value, py_traceback): """Set the error indicator from the three objects. If the error indicator is already set, it is cleared first. If the objects are NULL, the error indicator is cleared. Do not pass a NULL type and non-NULL value or @@ -80,13 +80,13 @@ error indicator temporarily; use PyErr_Fetch() to save the current exception state.""" state = space.fromcache(State) + w_type = get_w_obj_and_decref(py_type) + w_value = get_w_obj_and_decref(py_value) + w_traceback = get_w_obj_and_decref(py_traceback) #XXX do something with that if w_type is None: state.clear_exception() return state.set_exception(OperationError(w_type, w_value)) - Py_DecRef(space, w_type) - Py_DecRef(space, w_value) - Py_DecRef(space, w_traceback) @cpython_api([PyObjectP, PyObjectP, PyObjectP], lltype.Void) def PyErr_NormalizeException(space, exc_p, val_p, tb_p): @@ -108,8 +108,8 @@ w_evalue = space.w_None operr = OperationError(w_etype, w_evalue) operr.normalize_exception(space) - Py_DecRef(space, exc_p[0]) - Py_DecRef(space, val_p[0]) + decref(space, exc_p[0]) + decref(space, val_p[0]) exc_p[0] = make_ref(space, operr.w_type) val_p[0] = make_ref(space, operr.get_w_value(space)) @@ -390,7 +390,7 @@ ptraceback[0] = lltype.nullptr(PyObject.TO) @cpython_api([PyObject, PyObject, PyObject], lltype.Void) -def PyErr_SetExcInfo(space, w_type, w_value, w_traceback): +def PyErr_SetExcInfo(space, py_type, py_value, py_traceback): """---Cython extension--- Set the exception info, as known from ``sys.exc_info()``. This refers @@ -406,6 +406,9 @@ restore the exception state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception state. """ + w_type = get_w_obj_and_decref(py_type) + w_value = get_w_obj_and_decref(py_value) + w_traceback = get_w_obj_and_decref(py_traceback) if w_value is None or space.is_w(w_value, space.w_None): operror = None else: @@ -419,9 +422,6 @@ # ec = space.getexecutioncontext() ec.set_sys_exc_info(operror) - Py_DecRef(space, w_type) - Py_DecRef(space, w_value) - Py_DecRef(space, w_traceback) @cpython_api([], rffi.INT_real, error=CANNOT_FAIL) def PyOS_InterruptOccurred(space): diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -41,7 +41,7 @@ # Don't increase refcount for non-heaptypes flags = rffi.cast(lltype.Signed, pytype.c_tp_flags) if flags & Py_TPFLAGS_HEAPTYPE: - Py_IncRef(space, w_type) + incref(space, pytype) if pytype: size = pytype.c_tp_basicsize @@ -186,7 +186,7 @@ """ Ties together a PyObject and an interpreter object. The PyObject's refcnt is increased by REFCNT_FROM_PYPY. - The reference in 'py_obj' is not stolen! Remember to Py_DecRef() + The reference in 'py_obj' is not stolen! Remember to decref() it is you need to. """ # XXX looks like a PyObject_GC_TRACK @@ -244,8 +244,8 @@ use keepalive_until_here(w_obj) some time later.** In case of doubt, use the safer make_ref(). """ + assert not is_pyobj(w_obj) if w_obj is not None: - assert not is_pyobj(w_obj) py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) @@ -278,46 +278,36 @@ hop.exception_cannot_occur() return hop.inputconst(lltype.Bool, hop.s_result.const) - at specialize.ll() -def get_pyobj_and_incref(space, obj, w_userdata=None, immortal=False): - if is_pyobj(obj): - pyobj = rffi.cast(PyObject, obj) - at_least = 1 - else: - pyobj = as_pyobj(space, obj, w_userdata, immortal=immortal) - at_least = rawrefcount.REFCNT_FROM_PYPY - if pyobj: - assert pyobj.c_ob_refcnt >= at_least +def get_pyobj_and_incref(space, w_obj, w_userdata=None, immortal=False): + pyobj = as_pyobj(space, w_obj, w_userdata, immortal=immortal) + if pyobj: # != NULL + assert pyobj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY pyobj.c_ob_refcnt += 1 - keepalive_until_here(obj) + keepalive_until_here(w_obj) return pyobj - at specialize.ll() -def make_ref(space, obj, w_userdata=None, immortal=False): - """Increment the reference counter of the PyObject and return it. - Can be called with either a PyObject or a W_Root. +def make_ref(space, w_obj, w_userdata=None, immortal=False): + """Turn the W_Root into a corresponding PyObject. You should + decref the returned PyObject later. Note that it is often the + case, but not guaranteed, that make_ref() returns always the + same PyObject for the same W_Root; for example, integers. """ - if not is_pyobj(obj): - w_obj = obj - if w_obj is not None and space.type(w_obj) is space.w_int: - state = space.fromcache(State) - intval = space.int_w(w_obj) - return state.C.PyInt_FromLong(intval) - return get_pyobj_and_incref(space, obj, w_userdata, immortal=False) + assert not is_pyobj(w_obj) + if w_obj is not None and space.type(w_obj) is space.w_int: + state = space.fromcache(State) + intval = space.int_w(w_obj) + return state.C.PyInt_FromLong(intval) + return get_pyobj_and_incref(space, w_obj, w_userdata, immortal=False) @specialize.ll() -def get_w_obj_and_decref(space, obj): +def get_w_obj_and_decref(space, pyobj): """Decrement the reference counter of the PyObject and return the - corresponding W_Root object (so the reference count is at least - REFCNT_FROM_PYPY and cannot be zero). Can be called with either - a PyObject or a W_Root. + corresponding W_Root object (so the reference count after the decref + is at least REFCNT_FROM_PYPY and cannot be zero). """ - if is_pyobj(obj): - pyobj = rffi.cast(PyObject, obj) - w_obj = from_ref(space, pyobj) - else: - w_obj = obj - pyobj = as_pyobj(space, w_obj) + assert is_pyobj(pyobj) + pyobj = rffi.cast(PyObject, pyobj) + w_obj = from_ref(space, pyobj) if pyobj: pyobj.c_ob_refcnt -= 1 assert pyobj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY @@ -326,30 +316,30 @@ @specialize.ll() -def incref(space, obj): - get_pyobj_and_incref(space, obj) +def incref(space, pyobj): + assert is_pyobj(pyobj) + pyobj = rffi.cast(PyObject, pyobj) + assert pyobj.c_ob_refcnt >= 1 + pyobj.c_ob_refcnt += 1 @specialize.ll() -def decref(space, obj): +def decref(space, pyobj): from pypy.module.cpyext.api import generic_cpy_call - if is_pyobj(obj): - obj = rffi.cast(PyObject, obj) - if obj: - assert obj.c_ob_refcnt > 0 - assert obj.c_ob_pypy_link == 0 or obj.c_ob_refcnt > rawrefcount.REFCNT_FROM_PYPY - obj.c_ob_refcnt -= 1 - if obj.c_ob_refcnt == 0: - state = space.fromcache(State) - generic_cpy_call(space, state.C._Py_Dealloc, obj) - #else: - # w_obj = rawrefcount.to_obj(W_Root, ref) - # if w_obj is not None: - # assert obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY - else: - get_w_obj_and_decref(space, obj) + assert is_pyobj(pyobj) + pyobj = rffi.cast(PyObject, pyobj) + if pyobj: + assert pyobj.c_ob_refcnt > 0 + assert (pyobj.c_ob_pypy_link == 0 or + pyobj.c_ob_refcnt > rawrefcount.REFCNT_FROM_PYPY) + pyobj.c_ob_refcnt -= 1 + if pyobj.c_ob_refcnt == 0: + state = space.fromcache(State) + generic_cpy_call(space, state.C._Py_Dealloc, pyobj) + #else: + # w_obj = rawrefcount.to_obj(W_Root, ref) + # if w_obj is not None: + # assert pyobj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY -Py_IncRef = incref # XXX remove me and kill all the Py_IncRef usages from RPython -Py_DecRef = decref # XXX remove me and kill all the Py_DecRef usages from RPython @cpython_api([PyObject], lltype.Void) def _Py_NewReference(space, obj): diff --git a/pypy/module/cpyext/pystate.py b/pypy/module/cpyext/pystate.py --- a/pypy/module/cpyext/pystate.py +++ b/pypy/module/cpyext/pystate.py @@ -1,6 +1,6 @@ from pypy.module.cpyext.api import ( cpython_api, CANNOT_FAIL, cpython_struct) -from pypy.module.cpyext.pyobject import PyObject, Py_DecRef, make_ref +from pypy.module.cpyext.pyobject import PyObject, decref, make_ref from pypy.interpreter.error import OperationError from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib import rthread @@ -75,7 +75,7 @@ def ThreadState_dealloc(ts, space): assert space is not None - Py_DecRef(space, ts.c_dict) + decref(space, ts.c_dict) ThreadStateCapsule = encapsulator(PyThreadState.TO, dealloc=ThreadState_dealloc) @@ -319,7 +319,7 @@ interpreter lock must be held.""" if not space.config.translation.thread: raise NoThreads - Py_DecRef(space, tstate.c_dict) + decref(space, tstate.c_dict) tstate.c_dict = lltype.nullptr(PyObject.TO) space.threadlocals.leave_thread(space) space.getexecutioncontext().cleanup_cpyext_state() diff --git a/pypy/module/cpyext/pytraceback.py b/pypy/module/cpyext/pytraceback.py --- a/pypy/module/cpyext/pytraceback.py +++ b/pypy/module/cpyext/pytraceback.py @@ -4,7 +4,7 @@ cpython_api, bootstrap_function, cpython_struct, slot_function) from pypy.module.cpyext.pyobject import ( - PyObject, make_ref, from_ref, Py_DecRef, make_typedescr) + PyObject, make_ref, from_ref, decref, make_typedescr) from pypy.module.cpyext.frameobject import PyFrameObject from pypy.interpreter.error import OperationError from pypy.interpreter.pytraceback import PyTraceback @@ -44,7 +44,7 @@ @slot_function([PyObject], lltype.Void) def traceback_dealloc(space, py_obj): py_traceback = rffi.cast(PyTracebackObject, py_obj) - Py_DecRef(space, rffi.cast(PyObject, py_traceback.c_tb_next)) - Py_DecRef(space, rffi.cast(PyObject, py_traceback.c_tb_frame)) + decref(space, rffi.cast(PyObject, py_traceback.c_tb_next)) + decref(space, rffi.cast(PyObject, py_traceback.c_tb_frame)) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) diff --git a/pypy/module/cpyext/setobject.py b/pypy/module/cpyext/setobject.py --- a/pypy/module/cpyext/setobject.py +++ b/pypy/module/cpyext/setobject.py @@ -2,7 +2,7 @@ from rpython.rtyper.lltypesystem import rffi, lltype from pypy.module.cpyext.api import (cpython_api, Py_ssize_t, CANNOT_FAIL, build_type_checkers) -from pypy.module.cpyext.pyobject import (PyObject, PyObjectP, Py_DecRef, +from pypy.module.cpyext.pyobject import (PyObject, PyObjectP, make_ref, from_ref) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject, newset diff --git a/pypy/module/cpyext/sliceobject.py b/pypy/module/cpyext/sliceobject.py --- a/pypy/module/cpyext/sliceobject.py +++ b/pypy/module/cpyext/sliceobject.py @@ -3,7 +3,7 @@ cpython_api, cpython_struct, bootstrap_function, build_type_checkers, CANNOT_FAIL, Py_ssize_t, Py_ssize_tP, PyObjectFields, slot_function) from pypy.module.cpyext.pyobject import ( - Py_DecRef, PyObject, make_ref, make_typedescr) + decref, PyObject, make_ref, make_typedescr) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.interpreter.error import OperationError from pypy.objspace.std.sliceobject import W_SliceObject @@ -41,9 +41,9 @@ """Frees allocated PyBytesObject resources. """ py_slice = rffi.cast(PySliceObject, py_obj) - Py_DecRef(space, py_slice.c_start) - Py_DecRef(space, py_slice.c_stop) - Py_DecRef(space, py_slice.c_step) + decref(space, py_slice.c_start) + decref(space, py_slice.c_stop) + decref(space, py_slice.c_step) from pypy.module.cpyext.object import _dealloc _dealloc(space, py_obj) diff --git a/pypy/module/cpyext/structmember.py b/pypy/module/cpyext/structmember.py --- a/pypy/module/cpyext/structmember.py +++ b/pypy/module/cpyext/structmember.py @@ -5,7 +5,7 @@ from pypy.module.cpyext.api import ADDR, PyObjectP, cpython_api, CONST_STRING from pypy.module.cpyext.intobject import PyInt_AsLong, PyInt_AsUnsignedLong from pypy.module.cpyext.pyerrors import PyErr_Occurred -from pypy.module.cpyext.pyobject import PyObject, Py_DecRef, from_ref, make_ref +from pypy.module.cpyext.pyobject import PyObject, decref, from_ref, make_ref from pypy.module.cpyext.bytesobject import ( PyString_FromString, PyString_FromStringAndSize) from pypy.module.cpyext.floatobject import PyFloat_AsDouble @@ -123,7 +123,7 @@ elif member_type in [T_OBJECT, T_OBJECT_EX]: array = rffi.cast(PyObjectP, addr) if array[0]: - Py_DecRef(space, array[0]) + decref(space, array[0]) array[0] = make_ref(space, w_value) else: raise oefmt(space.w_SystemError, "bad memberdescr type") diff --git a/pypy/module/cpyext/test/test_bufferobject.py b/pypy/module/cpyext/test/test_bufferobject.py --- a/pypy/module/cpyext/test/test_bufferobject.py +++ b/pypy/module/cpyext/test/test_bufferobject.py @@ -2,7 +2,6 @@ from pypy.module.cpyext.test.test_api import BaseApiTest from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from pypy.module.cpyext.api import PyObject -from pypy.module.cpyext.pyobject import Py_DecRef class AppTestBufferObject(AppTestCpythonExtensionBase): def test_FromMemory(self): diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py --- a/pypy/module/cpyext/test/test_bytesobject.py +++ b/pypy/module/cpyext/test/test_bytesobject.py @@ -10,7 +10,7 @@ PyString_AsEncodedObject, PyString_AsDecodedObject, _PyString_Eq, _PyString_Join) from pypy.module.cpyext.api import PyObjectP, PyObject, Py_ssize_tP, generic_cpy_call -from pypy.module.cpyext.pyobject import Py_DecRef, from_ref, make_ref +from pypy.module.cpyext.pyobject import decref, from_ref, make_ref from pypy.module.cpyext.buffer import PyObject_AsCharBuffer from pypy.module.cpyext.api import PyTypeObjectPtr @@ -485,7 +485,7 @@ assert py_str.c_ob_size == 10 assert py_str.c_ob_sval[1] == 'b' assert py_str.c_ob_sval[10] == '\x00' - Py_DecRef(space, ar[0]) + decref(space, ar[0]) lltype.free(ar, flavor='raw') def test_string_buffer(self, space): @@ -504,7 +504,7 @@ assert generic_cpy_call(space, c_buf.c_bf_getreadbuffer, py_obj, 0, ref) == 10 lltype.free(ref, flavor='raw') - Py_DecRef(space, py_obj) + decref(space, py_obj) def test_Concat(self, space): ref = make_ref(space, space.wrap('abc')) @@ -530,7 +530,7 @@ PyString_ConcatAndDel(space, ptr, ref2) assert space.str_w(from_ref(space, ptr[0])) == 'abcdef' assert ref2.c_ob_refcnt == prev_refcnf - 1 - Py_DecRef(space, ptr[0]) + decref(space, ptr[0]) ptr[0] = lltype.nullptr(PyObject.TO) ref2 = make_ref(space, space.wrap('foo')) prev_refcnf = ref2.c_ob_refcnt @@ -555,7 +555,7 @@ assert rffi.charp2str(bufp[0]) == 'text' lltype.free(bufp, flavor='raw') lltype.free(lenp, flavor='raw') - Py_DecRef(space, ref) + decref(space, ref) def test_intern(self, space): buf = rffi.str2charp("test") diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -2,7 +2,7 @@ from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from pypy.module.cpyext.methodobject import PyMethodDef from pypy.module.cpyext.api import ApiFunction -from pypy.module.cpyext.pyobject import PyObject, make_ref, Py_DecRef +from pypy.module.cpyext.pyobject import PyObject, make_ref from pypy.module.cpyext.methodobject import ( PyDescr_NewMethod, PyCFunction) from rpython.rtyper.lltypesystem import rffi, lltype diff --git a/pypy/module/cpyext/test/test_number.py b/pypy/module/cpyext/test/test_number.py --- a/pypy/module/cpyext/test/test_number.py +++ b/pypy/module/cpyext/test/test_number.py @@ -4,7 +4,7 @@ from pypy.interpreter.error import OperationError from pypy.module.cpyext.test.test_api import BaseApiTest from pypy.module.cpyext.pyobject import ( - PyObjectP, from_ref, make_ref, Py_DecRef) + PyObjectP, from_ref, make_ref, decref) from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from pypy.module.cpyext.number import ( PyIndex_Check, PyNumber_Check, PyNumber_Long, PyNumber_Int, @@ -66,10 +66,10 @@ # # We need to decref twice because PyNumber_Coerce does an incref and # possibly changes the content of pp1 and pp2 - Py_DecRef(space, p1) - Py_DecRef(space, pp1[0]) - Py_DecRef(space, p2) - Py_DecRef(space, pp2[0]) + decref(space, p1) + decref(space, pp1[0]) + decref(space, p2) + decref(space, pp2[0]) lltype.free(pp1, flavor='raw') lltype.free(pp2, flavor='raw') @@ -88,10 +88,10 @@ assert PyFloat_Check(space, w_res) assert space.unwrap(w_res) == 123. - Py_DecRef(space, pl) - Py_DecRef(space, pf) - Py_DecRef(space, ppl[0]) - Py_DecRef(space, ppf[0]) + decref(space, pl) + decref(space, pf) + decref(space, ppl[0]) + decref(space, ppf[0]) lltype.free(ppl, flavor='raw') lltype.free(ppf, flavor='raw') diff --git a/pypy/module/cpyext/test/test_structseq.py b/pypy/module/cpyext/test/test_structseq.py --- a/pypy/module/cpyext/test/test_structseq.py +++ b/pypy/module/cpyext/test/test_structseq.py @@ -2,7 +2,6 @@ from pypy.module.cpyext.test.test_api import BaseApiTest from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from pypy.module.cpyext.api import PyObject -from pypy.module.cpyext.pyobject import Py_DecRef class AppTestStructSeq(AppTestCpythonExtensionBase): def test_StructSeq(self): diff --git a/pypy/module/cpyext/test/test_unicodeobject.py b/pypy/module/cpyext/test/test_unicodeobject.py --- a/pypy/module/cpyext/test/test_unicodeobject.py +++ b/pypy/module/cpyext/test/test_unicodeobject.py @@ -5,7 +5,7 @@ from pypy.module.cpyext.unicodeobject import ( Py_UNICODE, PyUnicodeObject, new_empty_unicode) from pypy.module.cpyext.api import PyObjectP, PyObject -from pypy.module.cpyext.pyobject import Py_DecRef, from_ref +from pypy.module.cpyext.pyobject import decref, from_ref from rpython.rtyper.lltypesystem import rffi, lltype import sys, py from pypy.module.cpyext.unicodeobject import * @@ -220,7 +220,7 @@ res = PyUnicode_FromStringAndSize(space, s, 4) w_res = from_ref(space, res) - Py_DecRef(space, res) + decref(space, res) assert space.unwrap(w_res) == u'sp\x09m' rffi.free_charp(s) @@ -243,7 +243,7 @@ assert py_uni.c_length == 10 assert py_uni.c_str[1] == 'b' assert py_uni.c_str[10] == '\x00' - Py_DecRef(space, ar[0]) + decref(space, ar[0]) lltype.free(ar, flavor='raw') def test_AsUTF8String(self, space): diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -29,7 +29,7 @@ from pypy.module.cpyext.modsupport import convert_method_defs from pypy.module.cpyext.pyobject import ( PyObject, make_ref, from_ref, get_typedescr, make_typedescr, - track_reference, Py_DecRef, as_pyobj) + track_reference, decref, as_pyobj) from pypy.module.cpyext.slotdefs import ( slotdefs_for_tp_slots, slotdefs_for_wrappers, get_slot_tp_function, llslot) @@ -367,11 +367,11 @@ if not space.is_true(w_kwds): w_kwds = None + subtype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_subtype)) try: - subtype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_subtype)) w_obj = generic_cpy_call(space, tp_new, subtype, w_args, w_kwds) finally: - Py_DecRef(space, w_subtype) + decref(space, subtype) return w_obj @specialize.memo() @@ -465,7 +465,7 @@ return PyMember_GetOne( space, rffi.cast(rffi.CCHARP, pyref), self.member) finally: - Py_DecRef(space, pyref) + decref(space, pyref) def member_delete(self, space, w_self): assert isinstance(self, W_MemberDescr) @@ -475,7 +475,7 @@ PyMember_SetOne( space, rffi.cast(rffi.CCHARP, pyref), self.member, None) finally: - Py_DecRef(space, pyref) + decref(space, pyref) def member_setter(self, space, w_self, w_value): assert isinstance(self, W_MemberDescr) @@ -485,7 +485,7 @@ PyMember_SetOne( space, rffi.cast(rffi.CCHARP, pyref), self.member, w_value) finally: - Py_DecRef(space, pyref) + decref(space, pyref) class W_PyCTypeObject(W_TypeObject): @jit.dont_look_inside @@ -571,7 +571,7 @@ pyref = make_ref(space, w_str) ref[0] = PyString_AsString(space, pyref) # Stolen reference: the object has better exist somewhere else - Py_DecRef(space, pyref) + decref(space, pyref) return space.len_w(w_str) @slot_function([PyObject, Py_ssize_t, rffi.VOIDPP], lltype.Signed, error=-1) @@ -584,7 +584,7 @@ pyref = make_ref(space, w_str) ref[0] = PyUnicode_AS_UNICODE(space, pyref) # Stolen reference: the object has better exist somewhere else - Py_DecRef(space, pyref) + decref(space, pyref) return PyUnicode_GET_DATA_SIZE(space, w_str) @slot_function([PyObject, Py_ssize_t, rffi.CCHARPP], lltype.Signed, error=-1) @@ -646,14 +646,14 @@ from pypy.module.cpyext.object import _dealloc obj_pto = rffi.cast(PyTypeObjectPtr, obj) base_pyo = rffi.cast(PyObject, obj_pto.c_tp_base) - Py_DecRef(space, obj_pto.c_tp_bases) - Py_DecRef(space, obj_pto.c_tp_mro) - Py_DecRef(space, obj_pto.c_tp_cache) # let's do it like cpython - Py_DecRef(space, obj_pto.c_tp_dict) + decref(space, obj_pto.c_tp_bases) + decref(space, obj_pto.c_tp_mro) + decref(space, obj_pto.c_tp_cache) # let's do it like cpython + decref(space, obj_pto.c_tp_dict) if obj_pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE: heaptype = rffi.cast(PyHeapTypeObject, obj) - Py_DecRef(space, heaptype.c_ht_name) - Py_DecRef(space, base_pyo) + decref(space, heaptype.c_ht_name) + decref(space, base_pyo) _dealloc(space, obj) @@ -664,7 +664,7 @@ if metatype: flags = rffi.cast(lltype.Signed, metatype.c_tp_flags) if not flags & Py_TPFLAGS_HEAPTYPE: - Py_DecRef(space, w_metatype) + decref(space, metatype) heaptype = lltype.malloc(PyHeapTypeObject.TO, flavor='raw', zero=True, @@ -756,7 +756,7 @@ flags = rffi.cast(lltype.Signed, pto.c_tp_flags) if pto.c_tp_base != base_object_pto or flags & Py_TPFLAGS_HEAPTYPE: pto.c_tp_new = pto.c_tp_base.c_tp_new - Py_DecRef(space, base_object_pyo) + decref(space, base_object_pyo) pto.c_tp_flags |= Py_TPFLAGS_READY return pto @@ -829,7 +829,7 @@ if not pto_as.c_bf_releasebuffer: pto_as.c_bf_releasebuffer = base_as.c_bf_releasebuffer finally: - Py_DecRef(space, base_pyo) + decref(space, base_pyo) def _type_realize(space, py_obj): """ @@ -922,7 +922,7 @@ pto.c_tp_getattro = llslot(space, PyObject_GenericGetAttr) if w_obj.is_cpytype(): - Py_DecRef(space, pto.c_tp_dict) + decref(space, pto.c_tp_dict) w_dict = w_obj.getdict(space) # pass in the w_obj to convert any values that are # unbound GetSetProperty into bound PyGetSetDescrObject diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -7,7 +7,7 @@ CONST_WSTRING, slot_function, cts, parse_dir) from pypy.module.cpyext.pyerrors import PyErr_BadArgument from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, Py_DecRef, make_ref, from_ref, track_reference, + PyObject, PyObjectP, decref, make_ref, from_ref, track_reference, make_typedescr, get_typedescr) from pypy.module.cpyext.bytesobject import PyString_Check from pypy.module.sys.interp_encoding import setdefaultencoding @@ -84,7 +84,7 @@ @slot_function([PyObject], lltype.Void) def unicode_dealloc(space, py_obj): py_unicode = rffi.cast(PyUnicodeObject, py_obj) - Py_DecRef(space, py_unicode.c_defenc) + decref(space, py_unicode.c_defenc) if py_unicode.c_str: lltype.free(py_unicode.c_str, flavor="raw") @@ -460,7 +460,7 @@ try: py_newuni = new_empty_unicode(space, newsize) except MemoryError: - Py_DecRef(space, ref[0]) + decref(space, ref[0]) ref[0] = lltype.nullptr(PyObject.TO) raise to_cp = newsize @@ -469,7 +469,7 @@ to_cp = oldsize for i in range(to_cp): py_newuni.c_str[i] = py_uni.c_str[i] - Py_DecRef(space, ref[0]) + decref(space, ref[0]) ref[0] = rffi.cast(PyObject, py_newuni) return 0 From pypy.commits at gmail.com Tue Oct 10 05:30:36 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 02:30:36 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: Move this numpypy test to test_ndarrayobject. It fails if it is placed Message-ID: <59dc933c.53b4df0a.8611e.b960@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92699:630e99e82bff Date: 2017-10-10 11:29 +0200 http://bitbucket.org/pypy/pypy/changeset/630e99e82bff/ Log: Move this numpypy test to test_ndarrayobject. It fails if it is placed elsewhere and we run only that test file. diff --git a/pypy/module/cpyext/test/test_ndarrayobject.py b/pypy/module/cpyext/test/test_ndarrayobject.py --- a/pypy/module/cpyext/test/test_ndarrayobject.py +++ b/pypy/module/cpyext/test/test_ndarrayobject.py @@ -1,6 +1,7 @@ import pytest import os from pypy.interpreter.error import OperationError +from pypy.module.cpyext.pyobject import make_ref, decref from pypy.module.cpyext.test.test_api import BaseApiTest from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from rpython.rtyper.lltypesystem import rffi, lltype @@ -231,6 +232,13 @@ signature) ''' + def test_ndarray_ref(self, space, api): + w_obj = space.appexec([], """(): + import _numpypy + return _numpypy.multiarray.dtype('int64').type(2)""") + ref = make_ref(space, w_obj) + decref(space, ref) + class AppTestNDArray(AppTestCpythonExtensionBase): def setup_class(cls): diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -522,12 +522,6 @@ assert w_obj is None assert api.PyErr_Occurred() is None - def test_ndarray_ref(self, space, api): - w_obj = space.appexec([], """(): - import _numpypy - return _numpypy.multiarray.dtype('int64').type(2)""") - ref = make_ref(space, w_obj) - decref(space, ref) class AppTestSlots(AppTestCpythonExtensionBase): def setup_class(cls): From pypy.commits at gmail.com Tue Oct 10 05:30:38 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 02:30:38 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: oops, fixes Message-ID: <59dc933e.72a6df0a.cb095.a8cc@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92700:3c9c5746d625 Date: 2017-10-10 11:29 +0200 http://bitbucket.org/pypy/pypy/changeset/3c9c5746d625/ Log: oops, fixes diff --git a/pypy/module/cpyext/pyerrors.py b/pypy/module/cpyext/pyerrors.py --- a/pypy/module/cpyext/pyerrors.py +++ b/pypy/module/cpyext/pyerrors.py @@ -6,7 +6,7 @@ from pypy.module.cpyext.api import cpython_api, CANNOT_FAIL, CONST_STRING from pypy.module.exceptions.interp_exceptions import W_RuntimeWarning from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, make_ref, from_ref, decref) + PyObject, PyObjectP, make_ref, from_ref, decref, get_w_obj_and_decref) from pypy.module.cpyext.state import State from pypy.module.cpyext.import_ import PyImport_Import from rpython.rlib import rposix, jit @@ -80,9 +80,10 @@ error indicator temporarily; use PyErr_Fetch() to save the current exception state.""" state = space.fromcache(State) - w_type = get_w_obj_and_decref(py_type) - w_value = get_w_obj_and_decref(py_value) - w_traceback = get_w_obj_and_decref(py_traceback) #XXX do something with that + w_type = get_w_obj_and_decref(space, py_type) + w_value = get_w_obj_and_decref(space, py_value) + w_traceback = get_w_obj_and_decref(space, py_traceback) + # XXX do something with w_traceback if w_type is None: state.clear_exception() return @@ -406,9 +407,9 @@ restore the exception state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception state. """ - w_type = get_w_obj_and_decref(py_type) - w_value = get_w_obj_and_decref(py_value) - w_traceback = get_w_obj_and_decref(py_traceback) + w_type = get_w_obj_and_decref(space, py_type) + w_value = get_w_obj_and_decref(space, py_value) + w_traceback = get_w_obj_and_decref(space, py_traceback) if w_value is None or space.is_w(w_value, space.w_None): operror = None else: diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py --- a/pypy/module/cpyext/test/test_memoryobject.py +++ b/pypy/module/cpyext/test/test_memoryobject.py @@ -32,7 +32,7 @@ assert space.eq_w(space.getattr(w_mv, w_f), space.getattr(w_memoryview, w_f)) decref(space, ref) - decref(space, w_memoryview) + decref(space, c_memoryview) class AppTestPyBuffer_FillInfo(AppTestCpythonExtensionBase): def test_fillWithObject(self): From pypy.commits at gmail.com Tue Oct 10 08:22:07 2017 From: pypy.commits at gmail.com (mattip) Date: Tue, 10 Oct 2017 05:22:07 -0700 (PDT) Subject: [pypy-commit] pypy matplotlib: add Tk structs needed for matplotlib Message-ID: <59dcbb6f.138a1c0a.7e4b1.09c6@mx.google.com> Author: Matti Picus Branch: matplotlib Changeset: r92701:e1572b9ceb2a Date: 2017-10-10 12:34 +1100 http://bitbucket.org/pypy/pypy/changeset/e1572b9ceb2a/ Log: add Tk structs needed for matplotlib diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py --- a/lib_pypy/_tkinter/tklib_build.py +++ b/lib_pypy/_tkinter/tklib_build.py @@ -163,6 +163,7 @@ int Tcl_ListObjLength(Tcl_Interp* interp, Tcl_Obj* listPtr, int* intPtr); int Tcl_ListObjIndex(Tcl_Interp* interp, Tcl_Obj* listPtr, int index, Tcl_Obj** objPtrPtr); int Tcl_SplitList(Tcl_Interp* interp, char* list, int* argcPtr, const char*** argvPtr); +char* Tcl_Merge(int argc, const char * const* argv); int Tcl_Eval(Tcl_Interp* interp, const char* script); int Tcl_EvalFile(Tcl_Interp* interp, const char* filename); @@ -187,6 +188,38 @@ int Tk_GetNumMainWindows(); void Tcl_FindExecutable(char *argv0); + +/* for matplotlib, from tkDecls.h, tk.h */ +typedef struct Tk_Window_ *Tk_Window; + +/* + * The following structure describes a block of pixels in memory: + */ + +typedef struct Tk_PhotoImageBlock { + unsigned char *pixelPtr; /* Pointer to the first pixel. */ + int width; /* Width of block, in pixels. */ + int height; /* Height of block, in pixels. */ + int pitch; /* Address difference between corresponding + * pixels in successive lines. */ + int pixelSize; /* Address difference between successive + * pixels in the same line. */ + int offset[4]; /* Address differences between the red, green, + * blue and alpha components of the pixel and + * the pixel as a whole. */ +} Tk_PhotoImageBlock; + + +typedef void *Tk_PhotoHandle; +Tk_Window Tk_MainWindow (Tcl_Interp * interp); +Tk_PhotoHandle Tk_FindPhoto (Tcl_Interp * interp, const char * imageName); +void Tk_PhotoBlank(Tk_PhotoHandle); +void Tk_PhotoPutBlock_NoComposite(Tk_PhotoHandle handle, + Tk_PhotoImageBlock *blockPtr, int x, int y, + int width, int height); +void Tcl_AppendResult (Tcl_Interp * interp, ...); + + """) if HAVE_WIDE_INT_TYPE: From pypy.commits at gmail.com Tue Oct 10 08:22:09 2017 From: pypy.commits at gmail.com (mattip) Date: Tue, 10 Oct 2017 05:22:09 -0700 (PDT) Subject: [pypy-commit] pypy matplotlib: together with branch of matlab. tkinter plots begin to work Message-ID: <59dcbb71.43aadf0a.13937.6b52@mx.google.com> Author: Matti Picus Branch: matplotlib Changeset: r92702:296db0043bc2 Date: 2017-10-10 14:47 +1100 http://bitbucket.org/pypy/pypy/changeset/296db0043bc2/ Log: together with branch of matlab. tkinter plots begin to work This converts the ndarray data to a string, the non-CFFI version passes a pointer to the data diff --git a/lib_pypy/_tkinter/app.py b/lib_pypy/_tkinter/app.py --- a/lib_pypy/_tkinter/app.py +++ b/lib_pypy/_tkinter/app.py @@ -119,7 +119,7 @@ tklib.TCL_GLOBAL_ONLY) # This is used to get the application class for Tk 4.1 and up - argv0 = className.lower() + argv0 = className.lower().encode('ascii') tklib.Tcl_SetVar(self.interp, "argv0", argv0, tklib.TCL_GLOBAL_ONLY) diff --git a/lib_pypy/_tkinter/tclobj.py b/lib_pypy/_tkinter/tclobj.py --- a/lib_pypy/_tkinter/tclobj.py +++ b/lib_pypy/_tkinter/tclobj.py @@ -2,6 +2,11 @@ from .tklib_cffi import ffi as tkffi, lib as tklib import binascii +try: + import numpy as np + hasNumpy = True +except ImportError: + hasNumpy = False class TypeCache(object): def __init__(self): @@ -86,6 +91,28 @@ finally: tklib.mp_clear(bigValue) +def AsObjNDArray(value): + # XXX there must be a better way + argv = tkffi.new("Tcl_Obj*[]", 3) + argv[0] = AsObj(' '.join([str(x) for x in value.shape])) + argv[1] = AsObj(value.dtype.str) + asstr = value.tostring() + argv[2] = AsObj(binascii.b2a_hex(asstr)) + return tklib.Tcl_NewListObj(3, argv) + +def FromTclStringNDArray(data): + # unconvert data, assuming it is stringified from AsObjNDArray + indx1 = data.find(b'}') + shape = map(int, data[1:indx1].split()) + size = np.prod(shape) + indx2 = data.find(b' ', indx1 + 2) + dtype = np.dtype(data[indx1 + 2:indx2]) + start = indx2+1 + stop = start + size * dtype.itemsize * 2 + if stop > len(data): + raise ValueError('data too short') + vals = binascii.a2b_hex(data[start:stop]) + return np.fromstring(vals, dtype=dtype).reshape(shape) def FromObj(app, value): """Convert a TclObj pointer into a Python object.""" @@ -175,6 +202,8 @@ if isinstance(value, TclObject): tklib.Tcl_IncrRefCount(value._value) return value._value + if hasNumpy and isinstance(value, np.ndarray): + return AsObjNDArray(value) return AsObj(str(value)) From pypy.commits at gmail.com Tue Oct 10 09:53:52 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 06:53:52 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: osx root patch: remove printout from common code paths Message-ID: <59dcd0f0.43581c0a.12a55.9b9a@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92703:e3423154d0f1 Date: 2017-10-10 15:38 +0200 http://bitbucket.org/pypy/pypy/changeset/e3423154d0f1/ Log: osx root patch: remove printout from common code paths We shouldn't print out anything if the return code is sufficient for debugging. diff --git a/lib_pypy/_cffi_ssl/osx-roots.diff b/lib_pypy/_cffi_ssl/osx-roots.diff --- a/lib_pypy/_cffi_ssl/osx-roots.diff +++ b/lib_pypy/_cffi_ssl/osx-roots.diff @@ -169,7 +169,7 @@ diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c libressl-2.6.2/crypto/x509/x509_vfy_apple.c --- libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c 1970-01-01 01:00:00.000000000 +0100 +++ libressl-2.6.2/crypto/x509/x509_vfy_apple.c 2017-10-07 14:05:16.000000000 +0200 -@@ -0,0 +1,229 @@ +@@ -0,0 +1,225 @@ +/* + * Copyright (c) 2009 Apple Inc. All Rights Reserved. + * @@ -260,11 +260,9 @@ + int nhosts = sk_OPENSSL_STRING_num(param->id->hosts); + + if (nhosts != 1) { -+ fprintf(stderr, "got %d hosts?\n", nhosts); + hostname = NULL; + + } else { -+ fprintf(stderr, "got a host!\n"); + hostname = to_string(sk_OPENSSL_STRING_value(param->id->hosts, 0)); + CFShow(hostname); + } @@ -365,13 +363,11 @@ + SecTrustResultType result = 0; + + if (SecTrustEvaluate(trust, &result) != errSecSuccess || result != kSecTrustResultUnspecified) { -+ fprintf(stderr, "Not trusted! %d\n", result); + CFRelease(certArray); + ctx->error = X509_V_ERR_CERT_UNTRUSTED; + return 0; + } + -+ fprintf(stderr, "OK!\n"); + CFRelease(certArray); + ctx->error = 0; + return 1; From pypy.commits at gmail.com Tue Oct 10 09:53:55 2017 From: pypy.commits at gmail.com (danchr) Date: Tue, 10 Oct 2017 06:53:55 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: build_cffi_imports: use cpu count rather than cpu count + 1 Message-ID: <59dcd0f3.43581c0a.12a55.9ba0@mx.google.com> Author: Dan Villiom Podlaski Christiansen Branch: py3.5-mac-embedding Changeset: r92704:05b4d8cf38cc Date: 2017-10-10 15:51 +0200 http://bitbucket.org/pypy/pypy/changeset/05b4d8cf38cc/ Log: build_cffi_imports: use cpu count rather than cpu count + 1 For consistency with the regular translation process, we should only use the CPU count when building dependencies. diff --git a/pypy/tool/build_cffi_imports.py b/pypy/tool/build_cffi_imports.py --- a/pypy/tool/build_cffi_imports.py +++ b/pypy/tool/build_cffi_imports.py @@ -130,7 +130,7 @@ status, stdout, stderr = run_subprocess( 'make', [ - '-s', '-j' + str(multiprocessing.cpu_count() + 1), + '-s', '-j' + str(multiprocessing.cpu_count()), 'install', 'DESTDIR={}/'.format(destdir), ], cwd=sources, From pypy.commits at gmail.com Tue Oct 10 10:07:40 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 07:07:40 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59dcd42c.05d21c0a.9658.1f15@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92706:b2d92fe45d11 Date: 2017-10-10 16:06 +0200 http://bitbucket.org/pypy/pypy/changeset/b2d92fe45d11/ Log: (antocuni, arigo) Test and fix diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py --- a/pypy/module/cpyext/sequence.py +++ b/pypy/module/cpyext/sequence.py @@ -49,9 +49,8 @@ converted to a sequence, and raises a TypeError, raise a new TypeError with m as the message text. If the conversion otherwise, fails, reraise the original exception""" - # CCC test and enable these two lines: - #if isinstance(w_obj, tupleobject.W_TupleObject): - # return w_obj + if isinstance(w_obj, tupleobject.W_TupleObject): + return w_obj # CCC avoid the double conversion that occurs here if isinstance(w_obj, W_ListObject): # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM w_obj.convert_to_cpy_strategy(space) @@ -63,6 +62,7 @@ raise OperationError(space.w_TypeError, space.newtext(rffi.charp2str(m))) raise e +# CCC this should be written as a C macro @cpython_api([rffi.VOIDP, Py_ssize_t], PyObject, result_borrowed=True) def PySequence_Fast_GET_ITEM(space, w_obj, index): """Return the ith element of o, assuming that o was returned by diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -14,6 +14,9 @@ assert not api.PySequence_Check(space.newdict()) def test_sequence_api(self, space, api): + w_tup = space.wrap((1, 2, 3, 4)) + assert api.PySequence_Fast(w_tup, "message") is w_tup + w_l = space.wrap([1, 2, 3, 4]) assert api.PySequence_Fast(w_l, "message") is w_l From pypy.commits at gmail.com Tue Oct 10 10:07:38 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 07:07:38 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59dcd42a.c88b1c0a.4662b.3743@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92705:86616775c934 Date: 2017-10-10 16:02 +0200 http://bitbucket.org/pypy/pypy/changeset/86616775c934/ Log: (antocuni, arigo) Test and "fix": PySequence_GetItem() should return a new reference to an *existing* object that is kept alive by the original tuple or list too. PyArg_ParseTuple() and probably 3rd-party code depends on that. diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py --- a/pypy/module/cpyext/sequence.py +++ b/pypy/module/cpyext/sequence.py @@ -1,11 +1,13 @@ from rpython.rlib import rerased, jit +from rpython.rlib.objectmodel import keepalive_until_here from pypy.interpreter.error import OperationError, oefmt from pypy.objspace.std.listobject import ( ListStrategy, UNROLL_CUTOFF, W_ListObject, ObjectListStrategy) from pypy.module.cpyext.api import ( cpython_api, CANNOT_FAIL, CONST_STRING, Py_ssize_t, PyObject, PyObjectP) from pypy.module.cpyext.pyobject import PyObject, make_ref, from_ref +from pypy.module.cpyext.pyobject import as_pyobj, incref from rpython.rtyper.lltypesystem import rffi, lltype from pypy.objspace.std import tupleobject @@ -47,6 +49,9 @@ converted to a sequence, and raises a TypeError, raise a new TypeError with m as the message text. If the conversion otherwise, fails, reraise the original exception""" + # CCC test and enable these two lines: + #if isinstance(w_obj, tupleobject.W_TupleObject): + # return w_obj if isinstance(w_obj, W_ListObject): # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM w_obj.convert_to_cpy_strategy(space) @@ -132,7 +137,21 @@ This function used an int type for i. This might require changes in your code for properly supporting 64-bit systems.""" - return space.getitem(w_obj, space.newint(i)) + if isinstance(w_obj, tupleobject.W_TupleObject): + from pypy.module.cpyext.tupleobject import PyTuple_GetItem + py_obj = as_pyobj(space, w_obj) + py_res = PyTuple_GetItem(space, py_obj, i) + incref(space, py_res) + keepalive_until_here(w_obj) + return py_res + if isinstance(w_obj, W_ListObject): + from pypy.module.cpyext.listobject import PyList_GET_ITEM + py_obj = as_pyobj(space, w_obj) + py_res = PyList_GET_ITEM(space, py_obj, i) + incref(space, py_res) + keepalive_until_here(w_obj) + return py_res + return make_ref(space, space.getitem(w_obj, space.newint(i))) @cpython_api([PyObject, Py_ssize_t], PyObject) def PySequence_GetItem(space, w_obj, i): diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -156,6 +156,29 @@ result = api.PySequence_Index(w_gen, w_tofind) assert result == 4 + def test_sequence_getitem(self, space, api): + # PySequence_GetItem() is defined to return a new reference. + # When it happens to be called on a list or tuple, it returns + # a new reference that is also kept alive by the fact that it + # lives in the list/tuple. Some code like PyArg_ParseTuple() + # relies on this fact: it decrefs the result of + # PySequence_GetItem() but then expects it to stay alive. Meh. + # Here, we check that we try hard not to break this kind of + # code: if written naively, it could return a fresh PyIntObject, + # for example. + w1 = space.wrap((41, 42, 43)) + p1 = api.PySequence_GetItem(w1, 1) + p2 = api.PySequence_GetItem(w1, 1) + assert p1 == p2 + assert p1.c_ob_refcnt > 1 + # + w1 = space.wrap([41, 42, 43]) + p1 = api.PySequence_GetItem(w1, 1) + p2 = api.PySequence_GetItem(w1, 1) + assert p1 == p2 + assert p1.c_ob_refcnt > 1 + + class AppTestSetObject(AppTestCpythonExtensionBase): def test_sequence_macro_cast(self): module = self.import_extension('foo', [ From pypy.commits at gmail.com Tue Oct 10 11:52:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Tue, 10 Oct 2017 08:52:21 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): hopefully fix translation, and add a sanity check to catch these errors earlier in the tests Message-ID: <59dcecb5.d2b0df0a.de073.5429@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92707:648fa5150fa6 Date: 2017-10-10 17:51 +0200 http://bitbucket.org/pypy/pypy/changeset/648fa5150fa6/ Log: (antocuni, arigo): hopefully fix translation, and add a sanity check to catch these errors earlier in the tests diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -433,9 +433,15 @@ arg = input_arg newargs += (arg, ) try: - return self.callable(space, *newargs) + result = self.callable(space, *newargs) finally: keepalive_until_here(*keepalives) + # + # this is just a sanity check to ensure that we don't forget to + # specify result_is_ll=True + if self.restype == PyObject: + assert self.result_is_ll == is_pyobj(result) + return result return unwrapper diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py --- a/pypy/module/cpyext/sequence.py +++ b/pypy/module/cpyext/sequence.py @@ -128,7 +128,7 @@ space.delslice(w_obj, space.newint(start), space.newint(end)) return 0 - at cpython_api([rffi.VOIDP, Py_ssize_t], PyObject) + at cpython_api([rffi.VOIDP, Py_ssize_t], PyObject, result_is_ll=True) def PySequence_ITEM(space, w_obj, i): """Return the ith element of o or NULL on failure. Macro form of PySequence_GetItem() but without checking that @@ -153,7 +153,7 @@ return py_res return make_ref(space, space.getitem(w_obj, space.newint(i))) - at cpython_api([PyObject, Py_ssize_t], PyObject) + at cpython_api([PyObject, Py_ssize_t], PyObject, result_is_ll=True) def PySequence_GetItem(space, w_obj, i): """Return the ith element of o, or NULL on failure. This is the equivalent of the Python expression o[i].""" From pypy.commits at gmail.com Tue Oct 10 11:54:55 2017 From: pypy.commits at gmail.com (antocuni) Date: Tue, 10 Oct 2017 08:54:55 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo): blindly try to fix cppyy Message-ID: <59dced4f.c55c1c0a.f8353.862f@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92708:160e55b245b0 Date: 2017-10-10 17:54 +0200 http://bitbucket.org/pypy/pypy/changeset/160e55b245b0/ Log: (antocuni, arigo): blindly try to fix cppyy diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -658,8 +658,8 @@ if hasattr(space, "fake"): raise NotImplementedError space.getbuiltinmodule("cpyext") - from pypy.module.cpyext.pyobject import Py_DecRef, PyObject - Py_DecRef(space, rffi.cast(PyObject, rffi.cast(rffi.VOIDPP, arg)[0])) + from pypy.module.cpyext.pyobject import decref, PyObject + decref(space, rffi.cast(PyObject, rffi.cast(rffi.VOIDPP, arg)[0])) class MacroConverter(TypeConverter): diff --git a/pypy/module/_cppyy/executor.py b/pypy/module/_cppyy/executor.py --- a/pypy/module/_cppyy/executor.py +++ b/pypy/module/_cppyy/executor.py @@ -220,11 +220,11 @@ def wrap_result(self, space, lresult): space.getbuiltinmodule("cpyext") - from pypy.module.cpyext.pyobject import PyObject, from_ref, make_ref, Py_DecRef + from pypy.module.cpyext.pyobject import PyObject, from_ref, make_ref, decref result = rffi.cast(PyObject, lresult) w_obj = from_ref(space, result) if result: - Py_DecRef(space, result) + decref(space, result) return w_obj def execute(self, space, cppmethod, cppthis, num_args, args): From pypy.commits at gmail.com Tue Oct 10 12:25:50 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 09:25:50 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59dcf48e.85c1df0a.e65e5.b0ae@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92709:8d6f9c10803c Date: 2017-10-10 18:25 +0200 http://bitbucket.org/pypy/pypy/changeset/8d6f9c10803c/ Log: (antocuni, arigo) Fix two different tests diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py --- a/pypy/module/cpyext/sequence.py +++ b/pypy/module/cpyext/sequence.py @@ -137,6 +137,9 @@ This function used an int type for i. This might require changes in your code for properly supporting 64-bit systems.""" + # XXX we should call Py*_GET_ITEM() instead of Py*_GetItem() + # from here, but we cannot because we are also called from + # PySequence_GetItem() if isinstance(w_obj, tupleobject.W_TupleObject): from pypy.module.cpyext.tupleobject import PyTuple_GetItem py_obj = as_pyobj(space, w_obj) @@ -145,9 +148,9 @@ keepalive_until_here(w_obj) return py_res if isinstance(w_obj, W_ListObject): - from pypy.module.cpyext.listobject import PyList_GET_ITEM + from pypy.module.cpyext.listobject import PyList_GetItem py_obj = as_pyobj(space, w_obj) - py_res = PyList_GET_ITEM(space, py_obj, i) + py_res = PyList_GetItem(space, py_obj, i) incref(space, py_res) keepalive_until_here(w_obj) return py_res diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -5,6 +5,7 @@ from pypy.module.cpyext.sequence import ( PySequence_Fast, PySequence_Contains, PySequence_Index, PySequence_GetItem, PySequence_SetItem, PySequence_DelItem) +from pypy.module.cpyext.pyobject import get_w_obj_and_decref import pytest @@ -134,9 +135,11 @@ def test_getitem(self, space, api): thelist = [8, 7, 6, 5, 4, 3, 2, 1] w_l = space.wrap(thelist) - result = api.PySequence_GetItem(w_l, 4) + py_result = api.PySequence_GetItem(w_l, 4) + result = get_w_obj_and_decref(space, py_result) assert space.is_true(space.eq(result, space.wrap(4))) - result = api.PySequence_ITEM(w_l, 4) + py_result = api.PySequence_ITEM(w_l, 4) + result = get_w_obj_and_decref(space, py_result) assert space.is_true(space.eq(result, space.wrap(4))) with raises_w(space, IndexError): PySequence_GetItem(space, w_l, 9000) From pypy.commits at gmail.com Tue Oct 10 13:57:15 2017 From: pypy.commits at gmail.com (arigo) Date: Tue, 10 Oct 2017 10:57:15 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, ronan, arigo) Message-ID: <59dd09fb.d1141c0a.6c657.238d@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92710:fa425a186e38 Date: 2017-10-10 19:56 +0200 http://bitbucket.org/pypy/pypy/changeset/fa425a186e38/ Log: (antocuni, ronan, arigo) Fix and improve the test, and fix the code diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py --- a/pypy/module/cpyext/sequence.py +++ b/pypy/module/cpyext/sequence.py @@ -104,6 +104,11 @@ cpy_strategy = space.fromcache(CPyListStrategy) if w_obj.strategy is cpy_strategy: return w_obj.get_raw_items() # asserts it's a cpyext strategy + elif isinstance(w_obj, tupleobject.W_TupleObject): + from pypy.module.cpyext.tupleobject import PyTupleObject + py_obj = as_pyobj(space, w_obj) + py_tuple = rffi.cast(PyTupleObject, py_obj) + return rffi.cast(PyObjectP, py_tuple.c_ob_item) raise oefmt(space.w_TypeError, "PySequence_Fast_ITEMS called but object is not the result of " "PySequence_Fast") diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -274,12 +274,13 @@ PyTypeObject * common_type; PyObject *foo, **objects; PyObject * seq = PyTuple_GetItem(args, 0); - /* XXX assert it is a tuple */ if (seq == NULL) Py_RETURN_NONE; foo = PySequence_Fast(seq, "some string"); objects = PySequence_Fast_ITEMS(foo); - size = PySequence_Fast_GET_SIZE(seq); + if (objects == NULL) + return NULL; + size = PySequence_Fast_GET_SIZE(foo); common_type = size > 0 ? Py_TYPE(objects[0]) : NULL; for (i = 1; i < size; ++i) { if (Py_TYPE(objects[i]) != common_type) { @@ -288,12 +289,20 @@ } } Py_DECREF(foo); + if (common_type == NULL) + return PyBool_FromLong(0); Py_DECREF(common_type); return PyBool_FromLong(1); """)]) s = [1, 2, 3, 4] assert module.test_fast_sequence(s[0:-1]) assert module.test_fast_sequence(s[::-1]) + s = (1, 2, 3, 4) + assert module.test_fast_sequence(s[0:-1]) + assert module.test_fast_sequence(s[::-1]) + s = "1234" + assert module.test_fast_sequence(s[0:-1]) + assert module.test_fast_sequence(s[::-1]) def test_fast_keyerror(self): module = self.import_extension('foo', [ From pypy.commits at gmail.com Tue Oct 10 19:53:22 2017 From: pypy.commits at gmail.com (mattip) Date: Tue, 10 Oct 2017 16:53:22 -0700 (PDT) Subject: [pypy-commit] pypy default: fix annotation for 1b7397a88e79 - same signature for {set, list}.switch_to_object_strategy() Message-ID: <59dd5d72.c8c5df0a.c34a2.f8ed@mx.google.com> Author: Matti Picus Branch: Changeset: r92712:382f89d5d1c5 Date: 2017-10-11 10:52 +1100 http://bitbucket.org/pypy/pypy/changeset/382f89d5d1c5/ Log: fix annotation for 1b7397a88e79 - same signature for {set, list}.switch_to_object_strategy() diff --git a/pypy/objspace/std/setobject.py b/pypy/objspace/std/setobject.py --- a/pypy/objspace/std/setobject.py +++ b/pypy/objspace/std/setobject.py @@ -47,9 +47,9 @@ def delweakref(self): self._lifeline_ = None - def switch_to_object_strategy(self, space): + def switch_to_object_strategy(self): d = self.strategy.getdict_w(self) - self.strategy = strategy = space.fromcache(ObjectSetStrategy) + self.strategy = strategy = self.space.fromcache(ObjectSetStrategy) self.sstorage = strategy.erase(d) def switch_to_empty_strategy(self): @@ -912,14 +912,14 @@ d = self.unerase(w_set.sstorage) d[self.unwrap(w_key)] = None else: - w_set.switch_to_object_strategy(self.space) + w_set.switch_to_object_strategy() w_set.add(w_key) def remove(self, w_set, w_item): d = self.unerase(w_set.sstorage) if not self.is_correct_type(w_item): #XXX check type of w_item and immediately return False in some cases - w_set.switch_to_object_strategy(self.space) + w_set.switch_to_object_strategy() return w_set.remove(w_item) key = self.unwrap(w_item) @@ -949,7 +949,7 @@ def has_key(self, w_set, w_key): if not self.is_correct_type(w_key): #XXX check type of w_item and immediately return False in some cases - w_set.switch_to_object_strategy(self.space) + w_set.switch_to_object_strategy() return w_set.has_key(w_key) d = self.unerase(w_set.sstorage) return self.unwrap(w_key) in d @@ -1205,7 +1205,7 @@ return if w_other.length() == 0: return - w_set.switch_to_object_strategy(self.space) + w_set.switch_to_object_strategy() w_set.update(w_other) def popitem(self, w_set): From pypy.commits at gmail.com Tue Oct 10 19:53:20 2017 From: pypy.commits at gmail.com (mattip) Date: Tue, 10 Oct 2017 16:53:20 -0700 (PDT) Subject: [pypy-commit] pypy default: document branches, matplotlib progress Message-ID: <59dd5d70.1b88df0a.d65d7.67cd@mx.google.com> Author: Matti Picus Branch: Changeset: r92711:f5cc7168a27b Date: 2017-10-11 10:47 +1100 http://bitbucket.org/pypy/pypy/changeset/f5cc7168a27b/ Log: document branches, matplotlib progress diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -5,4 +5,8 @@ .. this is a revision shortly after release-pypy2.7-v5.9.0 .. startrev:d56dadcef996 +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging +.. branch: docs-osx-brew-openssl + From pypy.commits at gmail.com Wed Oct 11 04:41:44 2017 From: pypy.commits at gmail.com (cfbolz) Date: Wed, 11 Oct 2017 01:41:44 -0700 (PDT) Subject: [pypy-commit] pypy default: some passing cases, and comments how they work Message-ID: <59ddd948.512f1c0a.becb5.0d14@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: Changeset: r92713:47ef5366ae73 Date: 2017-10-11 10:41 +0200 http://bitbucket.org/pypy/pypy/changeset/47ef5366ae73/ Log: some passing cases, and comments how they work diff --git a/rpython/jit/metainterp/optimizeopt/intbounds.py b/rpython/jit/metainterp/optimizeopt/intbounds.py --- a/rpython/jit/metainterp/optimizeopt/intbounds.py +++ b/rpython/jit/metainterp/optimizeopt/intbounds.py @@ -305,6 +305,10 @@ # Transform into INT_ADD. The following guard will be killed # by optimize_GUARD_NO_OVERFLOW; if we see instead an # optimize_GUARD_OVERFLOW, then InvalidLoop. + + # NB: this case also takes care of int_add_ovf with 0 as on of the + # arguments: the result will be bounded, and then the optimization + # for int_add with 0 as argument will remove the op. op = self.replace_op_with(op, rop.INT_ADD) return self.emit(op) @@ -325,6 +329,7 @@ return None resbound = b0.sub_bound(b1) if resbound.bounded(): + # this case takes care of int_sub_ovf(x, 0) as well op = self.replace_op_with(op, rop.INT_SUB) return self.emit(op) @@ -342,6 +347,7 @@ b2 = self.getintbound(op.getarg(1)) resbound = b1.mul_bound(b2) if resbound.bounded(): + # this case also takes care of multiplication with 0 and 1 op = self.replace_op_with(op, rop.INT_MUL) return self.emit(op) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -1962,6 +1962,55 @@ """ self.optimize_loop(ops, expected) + ops = """ + [i0] + i1 = int_mul_ovf(0, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(1, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 1) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + def test_fold_constant_partial_ops_float(self): ops = """ [f0] From pypy.commits at gmail.com Wed Oct 11 05:11:44 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 02:11:44 -0700 (PDT) Subject: [pypy-commit] pypy default: Backed out changeset 382f89d5d1c5 Message-ID: <59dde050.099fdf0a.f8f3f.dba2@mx.google.com> Author: Armin Rigo Branch: Changeset: r92714:38ac9e4520a1 Date: 2017-10-11 11:11 +0200 http://bitbucket.org/pypy/pypy/changeset/38ac9e4520a1/ Log: Backed out changeset 382f89d5d1c5 Oops, our fault, but 1b7397a88e79 should be fixed in cpyext/ diff --git a/pypy/objspace/std/setobject.py b/pypy/objspace/std/setobject.py --- a/pypy/objspace/std/setobject.py +++ b/pypy/objspace/std/setobject.py @@ -47,9 +47,9 @@ def delweakref(self): self._lifeline_ = None - def switch_to_object_strategy(self): + def switch_to_object_strategy(self, space): d = self.strategy.getdict_w(self) - self.strategy = strategy = self.space.fromcache(ObjectSetStrategy) + self.strategy = strategy = space.fromcache(ObjectSetStrategy) self.sstorage = strategy.erase(d) def switch_to_empty_strategy(self): @@ -912,14 +912,14 @@ d = self.unerase(w_set.sstorage) d[self.unwrap(w_key)] = None else: - w_set.switch_to_object_strategy() + w_set.switch_to_object_strategy(self.space) w_set.add(w_key) def remove(self, w_set, w_item): d = self.unerase(w_set.sstorage) if not self.is_correct_type(w_item): #XXX check type of w_item and immediately return False in some cases - w_set.switch_to_object_strategy() + w_set.switch_to_object_strategy(self.space) return w_set.remove(w_item) key = self.unwrap(w_item) @@ -949,7 +949,7 @@ def has_key(self, w_set, w_key): if not self.is_correct_type(w_key): #XXX check type of w_item and immediately return False in some cases - w_set.switch_to_object_strategy() + w_set.switch_to_object_strategy(self.space) return w_set.has_key(w_key) d = self.unerase(w_set.sstorage) return self.unwrap(w_key) in d @@ -1205,7 +1205,7 @@ return if w_other.length() == 0: return - w_set.switch_to_object_strategy() + w_set.switch_to_object_strategy(self.space) w_set.update(w_other) def popitem(self, w_set): From pypy.commits at gmail.com Wed Oct 11 05:26:30 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 02:26:30 -0700 (PDT) Subject: [pypy-commit] pypy default: Translation fix Message-ID: <59dde3c6.14421c0a.303f6.3125@mx.google.com> Author: Armin Rigo Branch: Changeset: r92715:32e65dca6a08 Date: 2017-10-11 11:21 +0200 http://bitbucket.org/pypy/pypy/changeset/32e65dca6a08/ Log: Translation fix diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -264,9 +264,12 @@ py_dict = rffi.cast(PyDictObject, py_obj) if pos == 0: # Store the current keys in the PyDictObject. + from pypy.objspace.std.listobject import W_ListObject decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) # w_keys must use the object strategy in order to keep the keys alive + if not isinstance(w_keys, W_ListObject): + return 0 # XXX should not call keys() above w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) From pypy.commits at gmail.com Wed Oct 11 06:19:15 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 03:19:15 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, ronan, arigo) Message-ID: <59ddf023.8ba9df0a.92399.a8c7@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92716:0d92b4ee9763 Date: 2017-10-11 12:18 +0200 http://bitbucket.org/pypy/pypy/changeset/0d92b4ee9763/ Log: (antocuni, ronan, arigo) Add hack_for_result_often_existing_obj(), used in a few functions which officially return a new reference, but where the caller might be bogus code that expects the new reference to stay alive even after a Py_DecRef() (which is "often" the case in CPython) diff --git a/pypy/module/cpyext/mapping.py b/pypy/module/cpyext/mapping.py --- a/pypy/module/cpyext/mapping.py +++ b/pypy/module/cpyext/mapping.py @@ -1,7 +1,8 @@ from rpython.rtyper.lltypesystem import lltype, rffi from pypy.module.cpyext.api import ( cpython_api, CANNOT_FAIL, CONST_STRING, Py_ssize_t) -from pypy.module.cpyext.pyobject import PyObject +from pypy.module.cpyext.pyobject import ( + PyObject, hack_for_result_often_existing_obj) @cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) @@ -37,12 +38,13 @@ the Python expression o.items().""" return space.call_method(w_obj, "items") - at cpython_api([PyObject, CONST_STRING], PyObject) + at cpython_api([PyObject, CONST_STRING], PyObject, result_is_ll=True) def PyMapping_GetItemString(space, w_obj, key): """Return element of o corresponding to the object key or NULL on failure. This is the equivalent of the Python expression o[key].""" w_key = space.newtext(rffi.charp2str(key)) - return space.getitem(w_obj, w_key) + w_res = space.getitem(w_obj, w_key) + return hack_for_result_often_existing_obj(space, w_res) @cpython_api([PyObject, CONST_STRING, PyObject], rffi.INT_real, error=-1) def PyMapping_SetItemString(space, w_obj, key, w_value): diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py --- a/pypy/module/cpyext/object.py +++ b/pypy/module/cpyext/object.py @@ -6,7 +6,7 @@ Py_GE, CONST_STRING, FILEP, fwrite, c_only) from pypy.module.cpyext.pyobject import ( PyObject, PyObjectP, from_ref, incref, decref, - get_typedescr) + get_typedescr, hack_for_result_often_existing_obj) from pypy.module.cpyext.typeobject import PyTypeObjectPtr from pypy.module.cpyext.pyerrors import PyErr_NoMemory, PyErr_BadInternalCall from pypy.objspace.std.typeobject import W_TypeObject @@ -69,20 +69,22 @@ def PyObject_Not(space, w_obj): return not space.is_true(w_obj) - at cpython_api([PyObject, PyObject], PyObject) + at cpython_api([PyObject, PyObject], PyObject, result_is_ll=True) def PyObject_GetAttr(space, w_obj, w_name): """Retrieve an attribute named attr_name from object o. Returns the attribute value on success, or NULL on failure. This is the equivalent of the Python expression o.attr_name.""" - return space.getattr(w_obj, w_name) + w_res = space.getattr(w_obj, w_name) + return hack_for_result_often_existing_obj(space, w_res) - at cpython_api([PyObject, CONST_STRING], PyObject) + at cpython_api([PyObject, CONST_STRING], PyObject, result_is_ll=True) def PyObject_GetAttrString(space, w_obj, name_ptr): """Retrieve an attribute named attr_name from object o. Returns the attribute value on success, or NULL on failure. This is the equivalent of the Python expression o.attr_name.""" name = rffi.charp2str(name_ptr) - return space.getattr(w_obj, space.newtext(name)) + w_res = space.getattr(w_obj, space.newtext(name)) + return hack_for_result_often_existing_obj(space, w_res) @cpython_api([PyObject, PyObject], rffi.INT_real, error=CANNOT_FAIL) def PyObject_HasAttr(space, w_obj, w_name): @@ -141,11 +143,12 @@ and 0 otherwise. This function always succeeds.""" return int(space.is_true(space.callable(w_obj))) - at cpython_api([PyObject, PyObject], PyObject) + at cpython_api([PyObject, PyObject], PyObject, result_is_ll=True) def PyObject_GetItem(space, w_obj, w_key): """Return element of o corresponding to the object key or NULL on failure. This is the equivalent of the Python expression o[key].""" - return space.getitem(w_obj, w_key) + w_res = space.getitem(w_obj, w_key) + return hack_for_result_often_existing_obj(space, w_res) @cpython_api([PyObject, PyObject, PyObject], rffi.INT_real, error=-1) def PyObject_SetItem(space, w_obj, w_key, w_value): diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -286,6 +286,13 @@ keepalive_until_here(w_obj) return pyobj +def hack_for_result_often_existing_obj(space, w_obj): + # Equivalent to get_pyobj_and_incref() and not to make_ref(): + # it builds a PyObject from a W_Root, but ensures that the result + # gets attached to the original W_Root. This is needed to work around + # some obscure abuses: https://github.com/numpy/numpy/issues/9850 + return get_pyobj_and_incref(space, w_obj) + def make_ref(space, w_obj, w_userdata=None, immortal=False): """Turn the W_Root into a corresponding PyObject. You should decref the returned PyObject later. Note that it is often the diff --git a/pypy/module/cpyext/test/test_mapping.py b/pypy/module/cpyext/test/test_mapping.py --- a/pypy/module/cpyext/test/test_mapping.py +++ b/pypy/module/cpyext/test/test_mapping.py @@ -1,5 +1,8 @@ from pypy.module.cpyext.test.test_api import BaseApiTest +from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from rpython.rtyper.lltypesystem import lltype, rffi +from pypy.module.cpyext.pyobject import get_w_obj_and_decref + class TestMapping(BaseApiTest): def test_check(self, space, api): @@ -25,8 +28,8 @@ w_d = space.newdict() key = rffi.str2charp("key") api.PyMapping_SetItemString(w_d, key, space.wrap(42)) - assert 42 == space.unwrap( - api.PyMapping_GetItemString(w_d, key)) + assert 42 == space.unwrap(get_w_obj_and_decref(space, + api.PyMapping_GetItemString(w_d, key))) rffi.free_charp(key) def test_haskey(self, space, api): @@ -38,3 +41,23 @@ assert api.PyMapping_HasKey(w_d, w_d) == 0 # and no error is set + + +class AppTestMapping(AppTestCpythonExtensionBase): + + def test_getitemstring_returns_new_but_borrowed_ref(self): + module = self.import_extension('foo', [ + ("test_mapping", "METH_O", + ''' + PyObject *value = PyMapping_GetItemString(args, "a"); + /* officially, "value" can have a refcount equal to one, + but some code out there assumes that it has a refcnt + of at least two --- which is bogus --- because it + is generally kept alive by the container. */ + PyObject *refcnt = PyInt_FromLong(value->ob_refcnt); + Py_DECREF(value); + return refcnt; + '''),]) + d = {"a": 42} + res = module.test_mapping(d) + assert res > 1 diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py --- a/pypy/module/cpyext/test/test_object.py +++ b/pypy/module/cpyext/test/test_object.py @@ -3,6 +3,7 @@ from pypy.module.cpyext.test.test_api import BaseApiTest, raises_w from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from rpython.rtyper.lltypesystem import rffi, lltype +from pypy.module.cpyext.pyobject import get_w_obj_and_decref from pypy.module.cpyext.api import ( Py_LT, Py_LE, Py_NE, Py_EQ, Py_GE, Py_GT) from pypy.module.cpyext.object import ( @@ -70,7 +71,8 @@ def test_getattr(self, space): charp1 = rffi.str2charp("__len__") charp2 = rffi.str2charp("not_real") - assert PyObject_GetAttrString(space, space.wrap(""), charp1) + assert get_w_obj_and_decref(space, + PyObject_GetAttrString(space, space.wrap(""), charp1)) with raises_w(space, AttributeError): PyObject_GetAttrString(space, space.wrap(""), charp2) @@ -79,17 +81,20 @@ rffi.free_charp(charp1) rffi.free_charp(charp2) - assert PyObject_GetAttr(space, space.wrap(""), space.wrap("__len__")) + assert get_w_obj_and_decref(space, + PyObject_GetAttr(space, space.wrap(""), space.wrap("__len__"))) with raises_w(space, AttributeError): PyObject_DelAttr(space, space.wrap(""), space.wrap("__len__")) def test_getitem(self, space, api): w_t = space.wrap((1, 2, 3, 4, 5)) - assert space.unwrap(api.PyObject_GetItem(w_t, space.wrap(3))) == 4 + assert space.unwrap(get_w_obj_and_decref(space, + api.PyObject_GetItem(w_t, space.wrap(3)))) == 4 w_d = space.newdict() space.setitem(w_d, space.wrap("a key!"), space.wrap(72)) - assert space.unwrap(api.PyObject_GetItem(w_d, space.wrap("a key!"))) == 72 + assert space.unwrap(get_w_obj_and_decref(space, + api.PyObject_GetItem(w_d, space.wrap("a key!")))) == 72 assert api.PyObject_SetItem(w_d, space.wrap("key"), space.w_None) == 0 assert space.getitem(w_d, space.wrap("key")) is space.w_None From pypy.commits at gmail.com Wed Oct 11 08:39:49 2017 From: pypy.commits at gmail.com (antocuni) Date: Wed, 11 Oct 2017 05:39:49 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo, ronan): hg merge default Message-ID: <59de1115.4f091c0a.afc3a.38c7@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92717:70284cb08d7e Date: 2017-10-11 14:39 +0200 http://bitbucket.org/pypy/pypy/changeset/70284cb08d7e/ Log: (antocuni, arigo, ronan): hg merge default diff too long, truncating to 2000 out of 8986 lines diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/release-v5.9.0.rst b/pypy/doc/release-v5.9.0.rst --- a/pypy/doc/release-v5.9.0.rst +++ b/pypy/doc/release-v5.9.0.rst @@ -10,15 +10,18 @@ This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 includes the upstream stdlib version 3.5.3. -NumPy and Pandas now work on PyPy2.7. Issues that appeared as excessive memory +NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues +that appeared as excessive memory use were cleared up and other incompatibilities were resolved. The C-API compatibility layer does slow down code which crosses the python-c interface often, we have ideas on how it could be improved, and still recommend using pure python on PyPy or interfacing via CFFI_. Many other modules based on C-API exentions now work on PyPy as well. -Cython 0.27 (released last week) should support more projects with PyPy, both -on PyPy2.7 and PyPy3.5 beta. +Cython 0.27.1 (released very recently) supports more projects with PyPy, both +on PyPy2.7 and PyPy3.5 beta. Note version **0.27.1** is now the minimum +version that supports this version of PyPy, due to some interactions with +updated C-API interface code. We optimized the JSON parser for recurring string keys, which should decrease memory use to 50% and increase parsing speed by up to 15% for large JSON files @@ -148,8 +151,7 @@ * Issue 2590_: fix the bounds in the GC when allocating a lot of objects with finalizers * Replace magical NOT RPYTHON comment with a decorator * Implement ``socket.sendmsg()``/``.recvmsg()`` for py3.5 - * Reduce excessive ``memory_pressure`` for ``_SSLContext`` objects and add - ``memory_pressure`` for ``_SSLSocket`` objects + * Add ``memory_pressure`` for ``_SSLSocket`` objects * Degredations diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -3,4 +3,10 @@ =========================== .. this is a revision shortly after release-pypy2.7-v5.9.0 -.. startrev:899e5245de1e +.. startrev:d56dadcef996 + +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging + +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst --- a/pypy/doc/whatsnew-pypy2-5.9.0.rst +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -85,3 +85,12 @@ .. branch: py_ssize_t Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/ctypeptr.py b/pypy/module/_cffi_backend/ctypeptr.py --- a/pypy/module/_cffi_backend/ctypeptr.py +++ b/pypy/module/_cffi_backend/ctypeptr.py @@ -156,10 +156,11 @@ class W_CTypePtrBase(W_CTypePtrOrArray): # base class for both pointers and pointers-to-functions - _attrs_ = ['is_void_ptr', 'is_voidchar_ptr'] - _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr'] + _attrs_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] + _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] is_void_ptr = False is_voidchar_ptr = False + is_onebyte_ptr = False def convert_to_object(self, cdata): ptrdata = rffi.cast(rffi.CCHARPP, cdata)[0] @@ -179,12 +180,20 @@ if self.is_void_ptr or other.is_void_ptr: pass # cast from or to 'void *' elif self.is_voidchar_ptr or other.is_voidchar_ptr: - space = self.space - msg = ("implicit cast from '%s' to '%s' " - "will be forbidden in the future (check that the types " - "are as you expect; use an explicit ffi.cast() if they " - "are correct)" % (other.name, self.name)) - space.warn(space.newtext(msg), space.w_UserWarning) + # for backward compatibility, accept "char *" as either + # source of target. This is not what C does, though, + # so emit a warning that will eventually turn into an + # error. The warning is turned off if both types are + # pointers to single bytes. + if self.is_onebyte_ptr and other.is_onebyte_ptr: + pass # no warning + else: + space = self.space + msg = ("implicit cast from '%s' to '%s' " + "will be forbidden in the future (check that the types " + "are as you expect; use an explicit ffi.cast() if they " + "are correct)" % (other.name, self.name)) + space.warn(space.newtext(msg), space.w_UserWarning) else: raise self._convert_error("compatible pointer", w_ob) @@ -214,6 +223,7 @@ self.is_void_ptr = isinstance(ctitem, ctypevoid.W_CTypeVoid) self.is_voidchar_ptr = (self.is_void_ptr or isinstance(ctitem, ctypeprim.W_CTypePrimitiveChar)) + self.is_onebyte_ptr = (ctitem.size == 1) W_CTypePtrBase.__init__(self, space, size, extra, 2, ctitem) def newp(self, w_init, allocator): diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): @@ -2099,7 +2099,8 @@ if sys.platform.startswith("linux"): BWChar = new_primitive_type("wchar_t") assert sizeof(BWChar) == 4 - assert int(cast(BWChar, -1)) == -1 # signed, on linux + # wchar_t is often signed on Linux, but not always (e.g. on ARM) + assert int(cast(BWChar, -1)) in (-1, 4294967295) def test_char16(): BChar16 = new_primitive_type("char16_t") @@ -3903,9 +3904,11 @@ BCharP = new_pointer_type(new_primitive_type("char")) BIntP = new_pointer_type(new_primitive_type("int")) BVoidP = new_pointer_type(new_void_type()) + BUCharP = new_pointer_type(new_primitive_type("unsigned char")) z1 = cast(BCharP, 0) z2 = cast(BIntP, 0) z3 = cast(BVoidP, 0) + z4 = cast(BUCharP, 0) with warnings.catch_warnings(record=True) as w: newp(new_pointer_type(BIntP), z1) # warn assert len(w) == 1 @@ -3919,6 +3922,12 @@ assert len(w) == 2 newp(new_pointer_type(BIntP), z3) # fine assert len(w) == 2 + newp(new_pointer_type(BCharP), z4) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z1) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z3) # fine + assert len(w) == 2 # check that the warnings are associated with lines in this file assert w[1].lineno == w[0].lineno + 4 diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'math/mathcore/src/Fitter.cxx' -if os.path.exists(inp): - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp): - if '#include "TF1.h"' in line: - continue - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - -# done -os.chdir(os.path.pardir) - -# debugging: run a test build -if DEBUG_TESTBUILD: - print('running a debug test build') - tb = "test_builddir" - if os.path.exists(tb): - shutil.rmtree(tb) - os.mkdir(tb) - os.chdir(tb) - os.system('cmake ../%s -DCMAKE_INSTALL_PREFIX=../install -Dminimal=ON -Dasimage=OFF' % pkgdir) - os.system('make -j 32') - - -# -## package creation -# -countdown = 0 -pidir = 'Package-'+args.release -print('creating package', pidir) -if not os.path.exists(pidir): - os.mkdir(pidir) -os.chdir(pidir); countdown += 1 - -print('creating LICENSE.txt') -with open('LICENSE.txt', 'w') as outp: - outp.write("""There are three main parts: - - LLVM: distributed under University of Illinois/NCSA Open Source License - https://opensource.org/licenses/UoI-NCSA.php - ROOT: distributed under LGPL 2.1 - https://root.cern.ch/license - Cppyy: distributed under LBNL BSD - https://fedoraproject.org/wiki/Licensing/LBNLBSD -""") - -print('creating MANIFEST.in') -with open('MANIFEST.in', 'w') as outp: - outp.write("""# Include the license file -include LICENSE.txt - -# Include the data files -recursive-include src * -""") - -print('creating README.rst') -with open('README.rst', 'w') as outp: - outp.write("""PyPy cling-support -================== - ----- - -Find the documentation here: - http://doc.pypy.org/en/latest/cppyy.html -""") - -print('creating setup.cfg') -with open('setup.cfg', 'w') as outp: - outp.write("""[bdist_wheel] -universal=0 -""") - -print('creating setup.py') -with open('setup.py', 'w') as outp: - outp.write("""import os, sys, subprocess -from setuptools import setup, find_packages -from distutils import log -from distutils.command.build import build as _build -from setuptools.command.install import install as _install -from distutils.sysconfig import get_python_lib -from distutils.errors import DistutilsSetupError -from codecs import open - -here = os.path.abspath(os.path.dirname(__file__)) -with open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: - long_description = f.read() - -builddir = None -def get_builddir(): - global builddir - if builddir is None: - topdir = os.getcwd() - builddir = os.path.join(topdir, 'builddir') - return builddir - -srcdir = None -def get_srcdir(): - global srcdir - if srcdir is None: - topdir = os.getcwd() - srcdir = os.path.join(topdir, 'src', 'backend') - return srcdir - -class my_cmake_build(_build): - def __init__(self, dist, *args, **kwargs): - _build.__init__(self, dist, *args, **kwargs) - # TODO: can't seem to find a better way of getting hold of - # the install_lib parameter during the build phase ... - prefix = '' - try: - prefix = dist.get_command_obj('install').install_lib - except AttributeError: - pass - if not prefix: - prefix = get_python_lib(1, 0) - self.prefix = os.path.join(prefix, 'cppyy_backend') - - def run(self): - # base run - _build.run(self) - - # custom run - log.info('Now building libcppyy_backend.so and dependencies') - builddir = get_builddir() - srcdir = get_srcdir() - if not os.path.exists(builddir): - log.info('Creating build directory %s ...' % builddir) - os.makedirs(builddir) - - os.chdir(builddir) - log.info('Running cmake for cppyy_backend') - if subprocess.call([ - 'cmake', srcdir, '-Dminimal=ON -Dasimage=OFF', - '-DCMAKE_INSTALL_PREFIX='+self.prefix]) != 0: - raise DistutilsSetupError('Failed to configure cppyy_backend') - - nprocs = os.getenv("MAKE_NPROCS") - if nprocs: - try: - ival = int(nprocs) - nprocs = '-j'+nprocs - except ValueError: - log.warn("Integer expected for MAKE_NPROCS, but got %s (ignored)", nprocs) - nprocs = '-j1' - else: - nprocs = '-j1' - log.info('Now building cppyy_backend and dependencies ...') - if subprocess.call(['make', nprocs]) != 0: - raise DistutilsSetupError('Failed to build cppyy_backend') - - log.info('build finished') - -class my_libs_install(_install): - def run(self): - # base install - _install.run(self) - - # custom install - log.info('Now installing libcppyy_backend.so and dependencies') - builddir = get_builddir() - if not os.path.exists(builddir): - raise DistutilsSetupError('Failed to find build dir!') - os.chdir(builddir) - - prefix = self.install_lib - log.info('Now installing in %s ...', prefix) - if subprocess.call(['make', 'install']) != 0: - raise DistutilsSetupError('Failed to install cppyy_backend') - - log.info('install finished') - - def get_outputs(self): - outputs = _install.get_outputs(self) - outputs.append(os.path.join(self.install_lib, 'cppyy_backend')) - return outputs - -setup( - name='PyPy-cppyy-backend', -""") - outp.write(" version='%s', # corresponds to ROOT %s, extra number is for packager\n"\ - % (args.version, args.release)) - outp.write(""" description='Cling support for PyPy', - long_description=long_description, - - url='http://pypy.org', - - # Author details - author='PyPy Developers', - author_email='pypy-dev at python.org', - - license='LLVM: UoI-NCSA; ROOT: LGPL 2.1; Cppyy: LBNL BSD', - - classifiers=[ - 'Development Status :: 4 - Beta', - - 'Intended Audience :: Developers', - - 'Topic :: Software Development', - 'Topic :: Software Development :: Interpreters', - - #'License :: OSI Approved :: MIT License', - - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: Implementation :: PyPy', - 'Programming Language :: C', - 'Programming Language :: C++', - - 'Natural Language :: English' - ], - - keywords='interpreter development', - - packages=find_packages('src', ['backend']), - include_package_data=True, - - extras_require={ - }, - - cmdclass = { - 'build': my_cmake_build, - 'install': my_libs_install, - }, -) -""") - - -print('creating src ... ROOT part') -if not os.path.exists('src'): - os.mkdir('src') -os.chdir('src'); countdown += 1 -if not os.path.exists('backend'): - src = os.path.join(os.path.pardir, os.path.pardir, pkgdir) - print('now copying', src) - shutil.copytree(src, 'backend') - -print('creating src ... cppyy part') -os.chdir('backend'); countdown += 1 -if not os.path.exists('cppyy'): - os.mkdir('cppyy') - os.chdir('cppyy'); countdown += 1 - - with open('CMakeLists.txt', 'w') as outp: - outp.write("""############################################################################ -# CMakeLists.txt file for building cppyy package -############################################################################ - -ROOT_GLOB_SOURCES(sources ${CMAKE_CURRENT_SOURCE_DIR}/src/*.cxx) -set_source_files_properties(${sources} COMPILE_FLAGS "-fomit-frame-pointer -fvisibility=hidden -DRPY_EXTERN=RPY_EXPORTED -DRPYTHON_LL2CTYPES") - -add_definitions(${CLING_CXXFLAGS}) - -ROOT_LINKER_LIBRARY(cppyy_backend ${sources} - LIBRARIES ${CMAKE_DL_LIBS} - DEPENDENCIES Core Cling RIO Thread) - -add_dependencies(cppyy_backend CLING) -""") - - os.mkdir('src') - os.chdir('src'); countdown += 1 - print('pulling cppyy/clingcwrapper.cxx from pypy') - base = 'https://bitbucket.org/pypy/pypy/raw/default/pypy/module/cppyy/' - for cppyy_file in ['src/callcontext.h', 'include/capi.h', 'src/clingcwrapper.cxx', - 'include/clingcwrapper.h', 'include/cpp_cppyy.h', 'include/cppyy.h']: - resp = urllib2.urlopen(base+cppyy_file) - with open(os.path.basename(cppyy_file), 'w') as outp: - outp.write(resp.read()) - - # fix include - inp = 'capi.h' - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if 'src/precommondefs.h' in line: - line = '#include "precommondefs.h"\n' - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - - with open('precommondefs.h', 'w') as outp: - outp.write("""/***** Start of precommondefs.h *****/ - -/* This is extracted from pyconfig.h from CPython. It sets the macros - that affect the features we get from system include files. - It must not #include anything. */ - -#ifndef __PYPY_PRECOMMONDEFS_H -#define __PYPY_PRECOMMONDEFS_H - - -/* Define on Darwin to activate all library features */ -#define _DARWIN_C_SOURCE 1 -/* This must be set to 64 on some systems to enable large file support. */ -#define _FILE_OFFSET_BITS 64 -/* Define on Linux to activate all library features */ -#define _GNU_SOURCE 1 -/* This must be defined on some systems to enable large file support. */ -#define _LARGEFILE_SOURCE 1 -/* Define on NetBSD to activate all library features */ -#define _NETBSD_SOURCE 1 -/* Define to activate features from IEEE Stds 1003.1-2001 */ -#ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L -#endif -/* Define on FreeBSD to activate all library features */ -#define __BSD_VISIBLE 1 -#define __XSI_VISIBLE 700 -/* Windows: winsock/winsock2 mess */ -#define WIN32_LEAN_AND_MEAN -#ifdef _WIN64 - typedef __int64 Signed; - typedef unsigned __int64 Unsigned; -# define SIGNED_MIN LLONG_MIN -#else - typedef long Signed; - typedef unsigned long Unsigned; -# define SIGNED_MIN LONG_MIN -#endif - -#if !defined(RPY_ASSERT) && !defined(RPY_LL_ASSERT) && !defined(NDEBUG) -# define NDEBUG -#endif - - -/* All functions and global variables declared anywhere should use - one of the following attributes: - - RPY_EXPORTED: the symbol is exported out of libpypy-c.so. - - RPY_EXTERN: the symbol is not exported out of libpypy-c.so, but - otherwise works like 'extern' by being available to - other C sources. - - static: as usual, this means the symbol is local to this C file. - - Don't use _RPY_HIDDEN directly. For tests involving building a custom - .so, translator/tool/cbuild.py overrides RPY_EXTERN so that it becomes - equal to RPY_EXPORTED. - - Any function or global variable declared with no attribute at all is - a bug; please report or fix it. -*/ -#ifdef __GNUC__ -# define RPY_EXPORTED extern __attribute__((visibility("default"))) -# define _RPY_HIDDEN __attribute__((visibility("hidden"))) -#else -# define RPY_EXPORTED extern __declspec(dllexport) -# define _RPY_HIDDEN /* nothing */ -#endif -#ifndef RPY_EXTERN -# define RPY_EXTERN extern _RPY_HIDDEN -#endif - - -#endif /* __PYPY_PRECOMMONDEFS_H */ - -/***** End of precommondefs.h *****/ -""") - -# back up to pip package top -for i in range(countdown-1): - os.chdir(os.path.pardir) - -# add cppyy module to cmake -os.chdir('src/backend') -inp = 'CMakeLists.txt' -print('adding cppyy to cmake') -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'add_subdirectory' in line and 'net' in line: - line += 'add_subdirectory (cppyy)\n' - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# done! diff --git a/pypy/module/_cppyy/bench/Makefile b/pypy/module/_cppyy/bench/Makefile deleted file mode 100644 --- a/pypy/module/_cppyy/bench/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -all: bench02Dict_reflex.so - -ROOTSYS := ${ROOTSYS} - -ifeq ($(ROOTSYS),) - genreflex=genreflex - cppflags= -else - genreflex=$(ROOTSYS)/bin/genreflex - cppflags=-I$(ROOTSYS)/include -L$(ROOTSYS)/lib -endif - -PLATFORM := $(shell uname -s) -ifeq ($(PLATFORM),Darwin) - cppflags+=-dynamiclib -single_module -arch x86_64 -endif - -ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),) - genreflexflags= - cppflags2=-O3 -fPIC -else - genreflexflags=--with-methptrgetter - cppflags2=-Wno-pmf-conversions -O3 -fPIC -endif - - -bench02Dict_reflex.so: bench02.h bench02.cxx bench02.xml - $(genreflex) bench02.h $(genreflexflags) --selection=bench02.xml -I$(ROOTSYS)/include - g++ -o $@ bench02.cxx bench02_rflx.cpp -I$(ROOTSYS)/include -shared -std=c++11 -lHistPainter `root-config --libs` $(cppflags) $(cppflags2) diff --git a/pypy/module/_cppyy/bench/bench02.cxx b/pypy/module/_cppyy/bench/bench02.cxx deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.cxx +++ /dev/null @@ -1,79 +0,0 @@ -#include "bench02.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TDirectory.h" -#include "TInterpreter.h" -#include "TSystem.h" -#include "TBenchmark.h" -#include "TStyle.h" -#include "TError.h" -#include "Getline.h" -#include "TVirtualX.h" - -#include "Api.h" - -#include - -TClass *TClass::GetClass(const char*, Bool_t, Bool_t) { - static TClass* dummy = new TClass("__dummy__", kTRUE); - return dummy; // is deleted by gROOT at shutdown -} - -class TTestApplication : public TApplication { -public: - TTestApplication( - const char* acn, Int_t* argc, char** argv, Bool_t bLoadLibs = kTRUE); - virtual ~TTestApplication(); -}; - -TTestApplication::TTestApplication( - const char* acn, int* argc, char** argv, bool do_load) : TApplication(acn, argc, argv) { - if (do_load) { - // follow TRint to minimize differences with CINT - ProcessLine("#include ", kTRUE); - ProcessLine("#include <_string>", kTRUE); // for std::string iostream. - ProcessLine("#include ", kTRUE); // needed because they're used within the - ProcessLine("#include ", kTRUE); // core ROOT dicts and CINT won't be able - // to properly unload these files - } - - // save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // prevent crashes on accessing history - Gl_histinit((char*)"-"); - - // prevent ROOT from exiting python - SetReturnFromRun(kTRUE); -} - -TTestApplication::~TTestApplication() {} - -static const char* appname = "pypy-cppyy"; - -Bench02RootApp::Bench02RootApp() { - gROOT->SetBatch(kTRUE); - if (!gApplication) { - int argc = 1; - char* argv[1]; argv[0] = (char*)appname; - gApplication = new TTestApplication(appname, &argc, argv, kFALSE); - } -} - -Bench02RootApp::~Bench02RootApp() { - // TODO: ROOT globals cleanup ... (?) -} - -void Bench02RootApp::report() { - std::cout << "gROOT is: " << gROOT << std::endl; - std::cout << "gApplication is: " << gApplication << std::endl; -} - -void Bench02RootApp::close_file(TFile* f) { - std::cout << "closing file " << f->GetName() << " ... " << std::endl; - f->Write(); - f->Close(); - std::cout << "... file closed" << std::endl; -} diff --git a/pypy/module/_cppyy/bench/bench02.h b/pypy/module/_cppyy/bench/bench02.h deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.h +++ /dev/null @@ -1,72 +0,0 @@ -#include "TString.h" - -#include "TCanvas.h" -#include "TFile.h" -#include "TProfile.h" -#include "TNtuple.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TRandom.h" -#include "TRandom3.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TSystem.h" - -#include "TArchiveFile.h" -#include "TBasket.h" -#include "TBenchmark.h" -#include "TBox.h" -#include "TBranchRef.h" -#include "TBrowser.h" -#include "TClassGenerator.h" -#include "TClassRef.h" -#include "TClassStreamer.h" -#include "TContextMenu.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TF1.h" -#include "TFileCacheRead.h" -#include "TFileCacheWrite.h" -#include "TFileMergeInfo.h" -#include "TFitResult.h" -#include "TFolder.h" -//#include "TFormulaPrimitive.h" -#include "TFunction.h" -#include "TFrame.h" -#include "TGlobal.h" -#include "THashList.h" -#include "TInetAddress.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TLegend.h" -#include "TMethodCall.h" -#include "TPluginManager.h" -#include "TProcessUUID.h" -#include "TSchemaRuleSet.h" -#include "TStyle.h" -#include "TSysEvtHandler.h" -#include "TTimer.h" -#include "TView.h" -//#include "TVirtualCollectionProxy.h" -#include "TVirtualFFT.h" -#include "TVirtualHistPainter.h" -#include "TVirtualIndex.h" -#include "TVirtualIsAProxy.h" -#include "TVirtualPadPainter.h" -#include "TVirtualRefProxy.h" -#include "TVirtualStreamerInfo.h" -#include "TVirtualViewer3D.h" - -#include -#include - - -class Bench02RootApp { -public: - Bench02RootApp(); - ~Bench02RootApp(); - - void report(); - void close_file(TFile* f); -}; diff --git a/pypy/module/_cppyy/bench/bench02.xml b/pypy/module/_cppyy/bench/bench02.xml deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pypy/module/_cppyy/bench/hsimple.C b/pypy/module/_cppyy/bench/hsimple.C deleted file mode 100644 --- a/pypy/module/_cppyy/bench/hsimple.C +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -TFile *hsimple(Int_t get=0) -{ -// This program creates : -// - a one dimensional histogram -// - a two dimensional histogram -// - a profile histogram -// - a memory-resident ntuple -// -// These objects are filled with some random numbers and saved on a file. -// If get=1 the macro returns a pointer to the TFile of "hsimple.root" -// if this file exists, otherwise it is created. -// The file "hsimple.root" is created in $ROOTSYS/tutorials if the caller has -// write access to this directory, otherwise the file is created in $PWD - - TString filename = "hsimple.root"; - TString dir = gSystem->UnixPathName(gInterpreter->GetCurrentMacroName()); - dir.ReplaceAll("hsimple.C",""); - dir.ReplaceAll("/./","/"); - TFile *hfile = 0; - if (get) { - // if the argument get =1 return the file "hsimple.root" - // if the file does not exist, it is created - TString fullPath = dir+"hsimple.root"; - if (!gSystem->AccessPathName(fullPath,kFileExists)) { - hfile = TFile::Open(fullPath); //in $ROOTSYS/tutorials - if (hfile) return hfile; - } - //otherwise try $PWD/hsimple.root - if (!gSystem->AccessPathName("hsimple.root",kFileExists)) { - hfile = TFile::Open("hsimple.root"); //in current dir - if (hfile) return hfile; - } - } - //no hsimple.root file found. Must generate it ! - //generate hsimple.root in $ROOTSYS/tutorials if we have write access - if (!gSystem->AccessPathName(dir,kWritePermission)) { - filename = dir+"hsimple.root"; - } else if (!gSystem->AccessPathName(".",kWritePermission)) { - //otherwise generate hsimple.root in the current directory - } else { - printf("you must run the script in a directory with write access\n"); - return 0; - } - hfile = (TFile*)gROOT->FindObject(filename); if (hfile) hfile->Close(); - hfile = new TFile(filename,"RECREATE","Demo ROOT file with histograms"); - - // Create some histograms, a profile histogram and an ntuple - TH1F *hpx = new TH1F("hpx","This is the px distribution",100,-4,4); - hpx->SetFillColor(48); - TH2F *hpxpy = new TH2F("hpxpy","py vs px",40,-4,4,40,-4,4); - TProfile *hprof = new TProfile("hprof","Profile of pz versus px",100,-4,4,0,20); - TNtuple *ntuple = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - - gBenchmark->Start("hsimple"); - - // Create a new canvas. - TCanvas *c1 = new TCanvas("c1","Dynamic Filling Example",200,10,700,500); - c1->SetFillColor(42); - c1->GetFrame()->SetFillColor(21); - c1->GetFrame()->SetBorderSize(6); - c1->GetFrame()->SetBorderMode(-1); - - - // Fill histograms randomly - TRandom3 random; - Float_t px, py, pz; - const Int_t kUPDATE = 1000; - for (Int_t i = 0; i < 50000; i++) { - // random.Rannor(px,py); - px = random.Gaus(0, 1); - py = random.Gaus(0, 1); - pz = px*px + py*py; - Float_t rnd = random.Rndm(1); - hpx->Fill(px); - hpxpy->Fill(px,py); - hprof->Fill(px,pz); - ntuple->Fill(px,py,pz,rnd,i); - if (i && (i%kUPDATE) == 0) { - if (i == kUPDATE) hpx->Draw(); - c1->Modified(); - c1->Update(); - if (gSystem->ProcessEvents()) - break; - } - } - gBenchmark->Show("hsimple"); - - // Save all objects in this file - hpx->SetFillColor(0); - hfile->Write(); - hpx->SetFillColor(48); - c1->Modified(); - return hfile; - -// Note that the file is automatically close when application terminates -// or when the file destructor is called. -} diff --git a/pypy/module/_cppyy/bench/hsimple.py b/pypy/module/_cppyy/bench/hsimple.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple.py +++ /dev/null @@ -1,110 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -_reflex = True # to keep things equal, set to False for full macro - -try: - import cppyy, random - - if not hasattr(cppyy.gbl, 'gROOT'): - cppyy.load_reflection_info('bench02Dict_reflex.so') - _reflex = True - - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom3 = cppyy.gbl.TRandom3 - - gROOT = cppyy.gbl.gROOT - gBenchmark = cppyy.gbl.TBenchmark() - gSystem = cppyy.gbl.gSystem - -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom3 - from ROOT import gROOT, gBenchmark, gSystem - import random - -if _reflex: - gROOT.SetBatch(True) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -if not _reflex: - hfile = gROOT.FindObject('hsimple.root') - if hfile: - hfile.Close() - hfile = TFile('hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.SetFillColor(48) -hpxpy = TH2F('hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4) -hprof = TProfile('hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20) -if not _reflex: - ntuple = TNtuple('ntuple', 'Demo ntuple', 'px:py:pz:random:i') - -gBenchmark.Start('hsimple') - -# Create a new canvas, and customize it. -c1 = TCanvas('c1', 'Dynamic Filling Example', 200, 10, 700, 500) -c1.SetFillColor(42) -c1.GetFrame().SetFillColor(21) -c1.GetFrame().SetBorderSize(6) -c1.GetFrame().SetBorderMode(-1) - -# Fill histograms randomly. -random = TRandom3() -kUPDATE = 1000 -for i in xrange(50000): - # Generate random numbers -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) - pz = px*px + py*py -# rnd = random.random() - rnd = random.Rndm(1) - - # Fill histograms - hpx.Fill(px) - hpxpy.Fill(px, py) - hprof.Fill(px, pz) - if not _reflex: - ntuple.Fill(px, py, pz, rnd, i) - - # Update display every kUPDATE events - if i and i%kUPDATE == 0: - if i == kUPDATE: - hpx.Draw() - - c1.Modified(True) - c1.Update() - - if gSystem.ProcessEvents(): # allow user interrupt - break - -gBenchmark.Show( 'hsimple' ) - -# Save all objects in this file -hpx.SetFillColor(0) -if not _reflex: - hfile.Write() -hpx.SetFillColor(48) -c1.Modified(True) -c1.Update() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/bench/hsimple_rflx.py b/pypy/module/_cppyy/bench/hsimple_rflx.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple_rflx.py +++ /dev/null @@ -1,120 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -try: - import warnings - warnings.simplefilter("ignore") - - import cppyy, random - cppyy.load_reflection_info('bench02Dict_reflex.so') - - app = cppyy.gbl.Bench02RootApp() - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom = cppyy.gbl.TRandom -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom - import random - -import math - -#gROOT = cppyy.gbl.gROOT -#gBenchmark = cppyy.gbl.gBenchmark -#gRandom = cppyy.gbl.gRandom -#gSystem = cppyy.gbl.gSystem - -#gROOT.Reset() - -# Create a new canvas, and customize it. -#c1 = TCanvas( 'c1', 'Dynamic Filling Example', 200, 10, 700, 500 ) -#c1.SetFillColor( 42 ) -#c1.GetFrame().SetFillColor( 21 ) -#c1.GetFrame().SetBorderSize( 6 ) -#c1.GetFrame().SetBorderMode( -1 ) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -#hfile = gROOT.FindObject( 'hsimple.root' ) -#if hfile: -# hfile.Close() -#hfile = TFile( 'hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.Print() -#hpxpy = TH2F( 'hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4 ) -#hprof = TProfile( 'hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20 ) -#ntuple = TNtuple( 'ntuple', 'Demo ntuple', 'px:py:pz:random:i' ) - -# Set canvas/frame attributes. -#hpx.SetFillColor( 48 ) - -#gBenchmark.Start( 'hsimple' ) - -# Initialize random number generator. -#gRandom.SetSeed() -#rannor, rndm = gRandom.Rannor, gRandom.Rndm - -random = TRandom() -random.SetSeed(0) - -# Fill histograms randomly. -#px, py = Double(), Double() -kUPDATE = 1000 -for i in xrange(2500000): - # Generate random values. -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) -# pt = (px*px + py*py)**0.5 - pt = math.sqrt(px*px + py*py) -# pt = (px*px + py*py) -# random = rndm(1) - - # Fill histograms. - hpx.Fill(pt) -# hpxpyFill( px, py ) -# hprofFill( px, pz ) -# ntupleFill( px, py, pz, random, i ) - - # Update display every kUPDATE events. -# if i and i%kUPDATE == 0: -# if i == kUPDATE: -# hpx.Draw() - -# c1.Modified() -# c1.Update() - -# if gSystem.ProcessEvents(): # allow user interrupt -# break - -#gBenchmark.Show( 'hsimple' ) - -hpx.Print() - -# Save all objects in this file. -#hpx.SetFillColor( 0 ) -#hfile.Write() -#hfile.Close() -#hpx.SetFillColor( 48 ) -#c1.Modified() -#c1.Update() -#c1.Draw() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/capi/__init__.py b/pypy/module/_cppyy/capi/__init__.py --- a/pypy/module/_cppyy/capi/__init__.py +++ b/pypy/module/_cppyy/capi/__init__.py @@ -1,16 +1,6 @@ from rpython.rtyper.lltypesystem import rffi, lltype -# There are two possible ways of accessing the backend through the reflection -# C-API: built it into pypy-c, or load it dynamically. The latter is preferred -# (and is the default) for use with Reflex. B/c of some builtin pythonizations, -# the former is recommended (for now) with CINT. - -# Note: if builtin_capi is chosen, then inside builtin_capi.py, there is still -# the selection of the desired backend (default is Reflex). - -# choose C-API access method: from pypy.module._cppyy.capi.loadable_capi import * -#from pypy.module._cppyy.capi.builtin_capi import * from pypy.module._cppyy.capi.capi_types import C_OBJECT,\ C_NULL_TYPE, C_NULL_OBJECT diff --git a/pypy/module/_cppyy/capi/builtin_capi.py b/pypy/module/_cppyy/capi/builtin_capi.py deleted file mode 100644 --- a/pypy/module/_cppyy/capi/builtin_capi.py +++ /dev/null @@ -1,590 +0,0 @@ -from rpython.rtyper.lltypesystem import rffi, lltype -from rpython.rlib.rarithmetic import intmask -from rpython.rlib import jit - -import cling_capi as backend - -from pypy.module._cppyy.capi.capi_types import C_SCOPE, C_TYPE, C_OBJECT,\ - C_METHOD, C_INDEX, C_INDEX_ARRAY, WLAVC_INDEX, C_FUNC_PTR - -identify = backend.identify -pythonize = backend.pythonize -register_pythonizations = backend.register_pythonizations -std_string_name = backend.std_string_name - -ts_reflect = backend.ts_reflect -ts_call = backend.ts_call -ts_memory = backend.ts_memory -ts_helper = backend.ts_helper - -def verify_backend(space): - return True # by definition - -c_load_dictionary = backend.c_load_dictionary - -# name to opaque C++ scope representation ------------------------------------ -_c_num_scopes = rffi.llexternal( - "cppyy_num_scopes", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_scopes(space, cppscope): - return _c_num_scopes(cppscope.handle) -_c_scope_name = rffi.llexternal( - "cppyy_scope_name", - [C_SCOPE, rffi.INT], rffi.CCHARP, - compilation_info = backend.eci) -def c_scope_name(space, cppscope, iscope): - return charp2str_free(space, _c_scope_name(cppscope.handle, iscope)) - -_c_resolve_name = rffi.llexternal( - "cppyy_resolve_name", - [rffi.CCHARP], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_resolve_name(space, name): - return charp2str_free(space, _c_resolve_name(name)) -_c_get_scope_opaque = rffi.llexternal( - "cppyy_get_scope", - [rffi.CCHARP], C_SCOPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_scope_opaque(space, name): - return _c_get_scope_opaque(name) -_c_actual_class = rffi.llexternal( - "cppyy_actual_class", - [C_TYPE, C_OBJECT], C_TYPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_actual_class(space, cppclass, cppobj): - return _c_actual_class(cppclass.handle, cppobj) - -# memory management ---------------------------------------------------------- -_c_allocate = rffi.llexternal( - "cppyy_allocate", - [C_TYPE], C_OBJECT, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate(space, cppclass): - return _c_allocate(cppclass.handle) -_c_deallocate = rffi.llexternal( - "cppyy_deallocate", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate(space, cppclass, cppobject): - _c_deallocate(cppclass.handle, cppobject) -_c_destruct = rffi.llexternal( - "cppyy_destruct", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_destruct(space, cppclass, cppobject): - _c_destruct(cppclass.handle, cppobject) - -# method/function dispatching ------------------------------------------------ -_c_call_v = rffi.llexternal( - "cppyy_call_v", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_v(space, cppmethod, cppobject, nargs, args): - _c_call_v(cppmethod, cppobject, nargs, args) -_c_call_b = rffi.llexternal( - "cppyy_call_b", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.UCHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_b(space, cppmethod, cppobject, nargs, args): - return _c_call_b(cppmethod, cppobject, nargs, args) -_c_call_c = rffi.llexternal( - "cppyy_call_c", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.CHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_c(space, cppmethod, cppobject, nargs, args): - return _c_call_c(cppmethod, cppobject, nargs, args) -_c_call_h = rffi.llexternal( - "cppyy_call_h", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.SHORT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_h(space, cppmethod, cppobject, nargs, args): - return _c_call_h(cppmethod, cppobject, nargs, args) -_c_call_i = rffi.llexternal( - "cppyy_call_i", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.INT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_i(space, cppmethod, cppobject, nargs, args): - return _c_call_i(cppmethod, cppobject, nargs, args) -_c_call_l = rffi.llexternal( - "cppyy_call_l", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_l(space, cppmethod, cppobject, nargs, args): - return _c_call_l(cppmethod, cppobject, nargs, args) -_c_call_ll = rffi.llexternal( - "cppyy_call_ll", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGLONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ll(space, cppmethod, cppobject, nargs, args): - return _c_call_ll(cppmethod, cppobject, nargs, args) -_c_call_f = rffi.llexternal( - "cppyy_call_f", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.FLOAT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_f(space, cppmethod, cppobject, nargs, args): - return _c_call_f(cppmethod, cppobject, nargs, args) -_c_call_d = rffi.llexternal( - "cppyy_call_d", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.DOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_d(space, cppmethod, cppobject, nargs, args): - return _c_call_d(cppmethod, cppobject, nargs, args) -_c_call_ld = rffi.llexternal( - "cppyy_call_ld", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGDOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ld(space, cppmethod, cppobject, nargs, args): - return _c_call_ld(cppmethod, cppobject, nargs, args) - -_c_call_r = rffi.llexternal( - "cppyy_call_r", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.VOIDP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_r(space, cppmethod, cppobject, nargs, args): - return _c_call_r(cppmethod, cppobject, nargs, args) -_c_call_s = rffi.llexternal( - "cppyy_call_s", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, rffi.SIZE_TP], rffi.CCHARP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_s(space, cppmethod, cppobject, nargs, args): - length = lltype.malloc(rffi.SIZE_TP.TO, 1, flavor='raw') - try: - cstr = _c_call_s(cppmethod, cppobject, nargs, args, length) - cstr_len = intmask(length[0]) - finally: - lltype.free(length, flavor='raw') - return cstr, cstr_len - -_c_constructor = rffi.llexternal( - "cppyy_constructor", - [C_METHOD, C_TYPE, rffi.INT, rffi.VOIDP], C_OBJECT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_constructor(space, cppmethod, cppobject, nargs, args): - return _c_constructor(cppmethod, cppobject, nargs, args) -_c_call_o = rffi.llexternal( - "cppyy_call_o", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, C_TYPE], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_o(space, method, cppobj, nargs, args, cppclass): - return _c_call_o(method, cppobj, nargs, args, cppclass.handle) - -_c_get_function_address = rffi.llexternal( - "cppyy_get_function_address", - [C_SCOPE, C_INDEX], C_FUNC_PTR, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) -def c_get_function_address(space, cppscope, index): - return _c_get_function_address(cppscope.handle, index) - -# handling of function argument buffer --------------------------------------- -_c_allocate_function_args = rffi.llexternal( - "cppyy_allocate_function_args", - [rffi.SIZE_T], rffi.VOIDP, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate_function_args(space, size): - return _c_allocate_function_args(size) -_c_deallocate_function_args = rffi.llexternal( - "cppyy_deallocate_function_args", - [rffi.VOIDP], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate_function_args(space, args): - _c_deallocate_function_args(args) -_c_function_arg_sizeof = rffi.llexternal( - "cppyy_function_arg_sizeof", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_sizeof(space): - return _c_function_arg_sizeof() -_c_function_arg_typeoffset = rffi.llexternal( - "cppyy_function_arg_typeoffset", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_typeoffset(space): - return _c_function_arg_typeoffset() - -# scope reflection information ----------------------------------------------- -_c_is_namespace = rffi.llexternal( - "cppyy_is_namespace", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_namespace(space, scope): - return _c_is_namespace(scope) -_c_is_template = rffi.llexternal( - "cppyy_is_template", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_template(space, name): - return _c_is_template(name) -_c_is_abstract = rffi.llexternal( - "cppyy_is_abstract", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_abstract(space, cpptype): - return _c_is_abstract(cpptype) -_c_is_enum = rffi.llexternal( - "cppyy_is_enum", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_enum(space, name): - return _c_is_enum(name) - -# type/class reflection information ------------------------------------------ -_c_final_name = rffi.llexternal( - "cppyy_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_final_name(space, cpptype): - return charp2str_free(space, _c_final_name(cpptype)) -_c_scoped_final_name = rffi.llexternal( - "cppyy_scoped_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_scoped_final_name(space, cpptype): - return charp2str_free(space, _c_scoped_final_name(cpptype)) -_c_has_complex_hierarchy = rffi.llexternal( - "cppyy_has_complex_hierarchy", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_has_complex_hierarchy(space, cpptype): - return _c_has_complex_hierarchy(cpptype) -_c_num_bases = rffi.llexternal( - "cppyy_num_bases", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_bases(space, cppclass): - return _c_num_bases(cppclass.handle) -_c_base_name = rffi.llexternal( - "cppyy_base_name", - [C_TYPE, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_base_name(space, cppclass, base_index): - return charp2str_free(space, _c_base_name(cppclass.handle, base_index)) -_c_is_subtype = rffi.llexternal( - "cppyy_is_subtype", - [C_TYPE, C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_is_subtype(space, derived, base): - if derived == base: - return 1 - return _c_is_subtype(derived.handle, base.handle) - -_c_base_offset = rffi.llexternal( - "cppyy_base_offset", - [C_TYPE, C_TYPE, C_OBJECT, rffi.INT], rffi.LONG, # actually ptrdiff_t - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_base_offset(space, derived, base, address, direction): - if derived == base: - return 0 - return _c_base_offset(derived.handle, base.handle, address, direction) -def c_base_offset1(space, derived_h, base, address, direction): - return _c_base_offset(derived_h, base.handle, address, direction) - -# method/function reflection information ------------------------------------- -_c_num_methods = rffi.llexternal( - "cppyy_num_methods", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_methods(space, cppscope): - return _c_num_methods(cppscope.handle) -_c_method_index_at = rffi.llexternal( - "cppyy_method_index_at", - [C_SCOPE, rffi.INT], C_INDEX, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_index_at(space, cppscope, imethod): - return _c_method_index_at(cppscope.handle, imethod) -_c_method_indices_from_name = rffi.llexternal( - "cppyy_method_indices_from_name", - [C_SCOPE, rffi.CCHARP], C_INDEX_ARRAY, From pypy.commits at gmail.com Wed Oct 11 09:41:48 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 06:41:48 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: (antocuni, arigo) Message-ID: <59de1f9c.81c11c0a.e10c1.71d4@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92719:a2e018841ca5 Date: 2017-10-11 15:03 +0200 http://bitbucket.org/pypy/pypy/changeset/a2e018841ca5/ Log: (antocuni, arigo) Ha ha ha argh 32-bit translations diff --git a/pypy/module/cpyext/intobject.py b/pypy/module/cpyext/intobject.py --- a/pypy/module/cpyext/intobject.py +++ b/pypy/module/cpyext/intobject.py @@ -35,7 +35,7 @@ # value. However, it's just easier to call PyInt_FromLong with a dummy # value; make sure it's big enough to avoid the smallint optimization # (if it will ever be enabled) - return state.C.PyInt_FromLong(0xDEADBEEF) + return state.C.PyInt_FromLong(0x0DEADBEE) else: return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) From pypy.commits at gmail.com Wed Oct 11 09:41:46 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 06:41:46 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: fix tests Message-ID: <59de1f9a.56acdf0a.95e3.e9fa@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92718:67a15f1e9e7b Date: 2017-10-11 15:01 +0200 http://bitbucket.org/pypy/pypy/changeset/67a15f1e9e7b/ Log: fix tests diff --git a/pypy/module/cpyext/test/test_classobject.py b/pypy/module/cpyext/test/test_classobject.py --- a/pypy/module/cpyext/test/test_classobject.py +++ b/pypy/module/cpyext/test/test_classobject.py @@ -5,6 +5,7 @@ PyClass_Check, PyClass_New, PyInstance_Check, PyInstance_New, PyInstance_NewRaw, _PyInstance_Lookup) from pypy.module.cpyext.object import PyObject_GetAttr +from pypy.module.cpyext.pyobject import get_w_obj_and_decref class TestClassObject(BaseApiTest): def test_newinstance(self, space): @@ -44,13 +45,14 @@ """) assert PyInstance_Check(space, w_instance) - assert PyObject_GetAttr(space, w_instance, space.wrap('x')) is space.w_None + py_obj = PyObject_GetAttr(space, w_instance, space.wrap('x')) + assert get_w_obj_and_decref(space, py_obj) is space.w_None assert _PyInstance_Lookup(space, w_instance, space.wrap('x')) is space.w_None assert _PyInstance_Lookup(space, w_instance, space.wrap('y')) is None # getattr returns a bound method - assert not isinstance( - PyObject_GetAttr(space, w_instance, space.wrap('f')), Function) + py_obj = PyObject_GetAttr(space, w_instance, space.wrap('f')) + assert not isinstance(get_w_obj_and_decref(space, py_obj), Function) # _PyInstance_Lookup returns the raw descriptor assert isinstance( _PyInstance_Lookup(space, w_instance, space.wrap('f')), Function) diff --git a/pypy/module/cpyext/test/test_eval.py b/pypy/module/cpyext/test/test_eval.py --- a/pypy/module/cpyext/test/test_eval.py +++ b/pypy/module/cpyext/test/test_eval.py @@ -14,6 +14,7 @@ _PyEval_SliceIndex) from pypy.module.cpyext.api import ( c_fopen, c_fclose, c_fileno, Py_ssize_tP, is_valid_fd) +from pypy.module.cpyext.pyobject import get_w_obj_and_decref from pypy.interpreter.gateway import interp2app from pypy.interpreter.error import OperationError from pypy.interpreter.astcompiler import consts @@ -117,8 +118,8 @@ assert run("a = 42 * 43", Py_single_input, w_globals, w_globals) == space.w_None - assert 42 * 43 == space.unwrap( - PyObject_GetItem(space, w_globals, space.wrap("a"))) + py_obj = PyObject_GetItem(space, w_globals, space.wrap("a")) + assert 42 * 43 == space.unwrap(get_w_obj_and_decref(space, py_obj)) def test_run_string_flags(self, space): flags = lltype.malloc(PyCompilerFlags, flavor='raw') From pypy.commits at gmail.com Wed Oct 11 09:41:50 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 11 Oct 2017 06:41:50 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads Message-ID: <59de1f9e.84ad1c0a.f88b8.76cb@mx.google.com> Author: Armin Rigo Branch: cpyext-avoid-roundtrip Changeset: r92720:bdbc6d0e6a01 Date: 2017-10-11 15:41 +0200 http://bitbucket.org/pypy/pypy/changeset/bdbc6d0e6a01/ Log: merge heads diff too long, truncating to 2000 out of 8986 lines diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/release-v5.9.0.rst b/pypy/doc/release-v5.9.0.rst --- a/pypy/doc/release-v5.9.0.rst +++ b/pypy/doc/release-v5.9.0.rst @@ -10,15 +10,18 @@ This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 includes the upstream stdlib version 3.5.3. -NumPy and Pandas now work on PyPy2.7. Issues that appeared as excessive memory +NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues +that appeared as excessive memory use were cleared up and other incompatibilities were resolved. The C-API compatibility layer does slow down code which crosses the python-c interface often, we have ideas on how it could be improved, and still recommend using pure python on PyPy or interfacing via CFFI_. Many other modules based on C-API exentions now work on PyPy as well. -Cython 0.27 (released last week) should support more projects with PyPy, both -on PyPy2.7 and PyPy3.5 beta. +Cython 0.27.1 (released very recently) supports more projects with PyPy, both +on PyPy2.7 and PyPy3.5 beta. Note version **0.27.1** is now the minimum +version that supports this version of PyPy, due to some interactions with +updated C-API interface code. We optimized the JSON parser for recurring string keys, which should decrease memory use to 50% and increase parsing speed by up to 15% for large JSON files @@ -148,8 +151,7 @@ * Issue 2590_: fix the bounds in the GC when allocating a lot of objects with finalizers * Replace magical NOT RPYTHON comment with a decorator * Implement ``socket.sendmsg()``/``.recvmsg()`` for py3.5 - * Reduce excessive ``memory_pressure`` for ``_SSLContext`` objects and add - ``memory_pressure`` for ``_SSLSocket`` objects + * Add ``memory_pressure`` for ``_SSLSocket`` objects * Degredations diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -3,4 +3,10 @@ =========================== .. this is a revision shortly after release-pypy2.7-v5.9.0 -.. startrev:899e5245de1e +.. startrev:d56dadcef996 + +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging + +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst --- a/pypy/doc/whatsnew-pypy2-5.9.0.rst +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -85,3 +85,12 @@ .. branch: py_ssize_t Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/ctypeptr.py b/pypy/module/_cffi_backend/ctypeptr.py --- a/pypy/module/_cffi_backend/ctypeptr.py +++ b/pypy/module/_cffi_backend/ctypeptr.py @@ -156,10 +156,11 @@ class W_CTypePtrBase(W_CTypePtrOrArray): # base class for both pointers and pointers-to-functions - _attrs_ = ['is_void_ptr', 'is_voidchar_ptr'] - _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr'] + _attrs_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] + _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] is_void_ptr = False is_voidchar_ptr = False + is_onebyte_ptr = False def convert_to_object(self, cdata): ptrdata = rffi.cast(rffi.CCHARPP, cdata)[0] @@ -179,12 +180,20 @@ if self.is_void_ptr or other.is_void_ptr: pass # cast from or to 'void *' elif self.is_voidchar_ptr or other.is_voidchar_ptr: - space = self.space - msg = ("implicit cast from '%s' to '%s' " - "will be forbidden in the future (check that the types " - "are as you expect; use an explicit ffi.cast() if they " - "are correct)" % (other.name, self.name)) - space.warn(space.newtext(msg), space.w_UserWarning) + # for backward compatibility, accept "char *" as either + # source of target. This is not what C does, though, + # so emit a warning that will eventually turn into an + # error. The warning is turned off if both types are + # pointers to single bytes. + if self.is_onebyte_ptr and other.is_onebyte_ptr: + pass # no warning + else: + space = self.space + msg = ("implicit cast from '%s' to '%s' " + "will be forbidden in the future (check that the types " + "are as you expect; use an explicit ffi.cast() if they " + "are correct)" % (other.name, self.name)) + space.warn(space.newtext(msg), space.w_UserWarning) else: raise self._convert_error("compatible pointer", w_ob) @@ -214,6 +223,7 @@ self.is_void_ptr = isinstance(ctitem, ctypevoid.W_CTypeVoid) self.is_voidchar_ptr = (self.is_void_ptr or isinstance(ctitem, ctypeprim.W_CTypePrimitiveChar)) + self.is_onebyte_ptr = (ctitem.size == 1) W_CTypePtrBase.__init__(self, space, size, extra, 2, ctitem) def newp(self, w_init, allocator): diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): @@ -2099,7 +2099,8 @@ if sys.platform.startswith("linux"): BWChar = new_primitive_type("wchar_t") assert sizeof(BWChar) == 4 - assert int(cast(BWChar, -1)) == -1 # signed, on linux + # wchar_t is often signed on Linux, but not always (e.g. on ARM) + assert int(cast(BWChar, -1)) in (-1, 4294967295) def test_char16(): BChar16 = new_primitive_type("char16_t") @@ -3903,9 +3904,11 @@ BCharP = new_pointer_type(new_primitive_type("char")) BIntP = new_pointer_type(new_primitive_type("int")) BVoidP = new_pointer_type(new_void_type()) + BUCharP = new_pointer_type(new_primitive_type("unsigned char")) z1 = cast(BCharP, 0) z2 = cast(BIntP, 0) z3 = cast(BVoidP, 0) + z4 = cast(BUCharP, 0) with warnings.catch_warnings(record=True) as w: newp(new_pointer_type(BIntP), z1) # warn assert len(w) == 1 @@ -3919,6 +3922,12 @@ assert len(w) == 2 newp(new_pointer_type(BIntP), z3) # fine assert len(w) == 2 + newp(new_pointer_type(BCharP), z4) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z1) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z3) # fine + assert len(w) == 2 # check that the warnings are associated with lines in this file assert w[1].lineno == w[0].lineno + 4 diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'math/mathcore/src/Fitter.cxx' -if os.path.exists(inp): - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp): - if '#include "TF1.h"' in line: - continue - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - -# done -os.chdir(os.path.pardir) - -# debugging: run a test build -if DEBUG_TESTBUILD: - print('running a debug test build') - tb = "test_builddir" - if os.path.exists(tb): - shutil.rmtree(tb) - os.mkdir(tb) - os.chdir(tb) - os.system('cmake ../%s -DCMAKE_INSTALL_PREFIX=../install -Dminimal=ON -Dasimage=OFF' % pkgdir) - os.system('make -j 32') - - -# -## package creation -# -countdown = 0 -pidir = 'Package-'+args.release -print('creating package', pidir) -if not os.path.exists(pidir): - os.mkdir(pidir) -os.chdir(pidir); countdown += 1 - -print('creating LICENSE.txt') -with open('LICENSE.txt', 'w') as outp: - outp.write("""There are three main parts: - - LLVM: distributed under University of Illinois/NCSA Open Source License - https://opensource.org/licenses/UoI-NCSA.php - ROOT: distributed under LGPL 2.1 - https://root.cern.ch/license - Cppyy: distributed under LBNL BSD - https://fedoraproject.org/wiki/Licensing/LBNLBSD -""") - -print('creating MANIFEST.in') -with open('MANIFEST.in', 'w') as outp: - outp.write("""# Include the license file -include LICENSE.txt - -# Include the data files -recursive-include src * -""") - -print('creating README.rst') -with open('README.rst', 'w') as outp: - outp.write("""PyPy cling-support -================== - ----- - -Find the documentation here: - http://doc.pypy.org/en/latest/cppyy.html -""") - -print('creating setup.cfg') -with open('setup.cfg', 'w') as outp: - outp.write("""[bdist_wheel] -universal=0 -""") - -print('creating setup.py') -with open('setup.py', 'w') as outp: - outp.write("""import os, sys, subprocess -from setuptools import setup, find_packages -from distutils import log -from distutils.command.build import build as _build -from setuptools.command.install import install as _install -from distutils.sysconfig import get_python_lib -from distutils.errors import DistutilsSetupError -from codecs import open - -here = os.path.abspath(os.path.dirname(__file__)) -with open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: - long_description = f.read() - -builddir = None -def get_builddir(): - global builddir - if builddir is None: - topdir = os.getcwd() - builddir = os.path.join(topdir, 'builddir') - return builddir - -srcdir = None -def get_srcdir(): - global srcdir - if srcdir is None: - topdir = os.getcwd() - srcdir = os.path.join(topdir, 'src', 'backend') - return srcdir - -class my_cmake_build(_build): - def __init__(self, dist, *args, **kwargs): - _build.__init__(self, dist, *args, **kwargs) - # TODO: can't seem to find a better way of getting hold of - # the install_lib parameter during the build phase ... - prefix = '' - try: - prefix = dist.get_command_obj('install').install_lib - except AttributeError: - pass - if not prefix: - prefix = get_python_lib(1, 0) - self.prefix = os.path.join(prefix, 'cppyy_backend') - - def run(self): - # base run - _build.run(self) - - # custom run - log.info('Now building libcppyy_backend.so and dependencies') - builddir = get_builddir() - srcdir = get_srcdir() - if not os.path.exists(builddir): - log.info('Creating build directory %s ...' % builddir) - os.makedirs(builddir) - - os.chdir(builddir) - log.info('Running cmake for cppyy_backend') - if subprocess.call([ - 'cmake', srcdir, '-Dminimal=ON -Dasimage=OFF', - '-DCMAKE_INSTALL_PREFIX='+self.prefix]) != 0: - raise DistutilsSetupError('Failed to configure cppyy_backend') - - nprocs = os.getenv("MAKE_NPROCS") - if nprocs: - try: - ival = int(nprocs) - nprocs = '-j'+nprocs - except ValueError: - log.warn("Integer expected for MAKE_NPROCS, but got %s (ignored)", nprocs) - nprocs = '-j1' - else: - nprocs = '-j1' - log.info('Now building cppyy_backend and dependencies ...') - if subprocess.call(['make', nprocs]) != 0: - raise DistutilsSetupError('Failed to build cppyy_backend') - - log.info('build finished') - -class my_libs_install(_install): - def run(self): - # base install - _install.run(self) - - # custom install - log.info('Now installing libcppyy_backend.so and dependencies') - builddir = get_builddir() - if not os.path.exists(builddir): - raise DistutilsSetupError('Failed to find build dir!') - os.chdir(builddir) - - prefix = self.install_lib - log.info('Now installing in %s ...', prefix) - if subprocess.call(['make', 'install']) != 0: - raise DistutilsSetupError('Failed to install cppyy_backend') - - log.info('install finished') - - def get_outputs(self): - outputs = _install.get_outputs(self) - outputs.append(os.path.join(self.install_lib, 'cppyy_backend')) - return outputs - -setup( - name='PyPy-cppyy-backend', -""") - outp.write(" version='%s', # corresponds to ROOT %s, extra number is for packager\n"\ - % (args.version, args.release)) - outp.write(""" description='Cling support for PyPy', - long_description=long_description, - - url='http://pypy.org', - - # Author details - author='PyPy Developers', - author_email='pypy-dev at python.org', - - license='LLVM: UoI-NCSA; ROOT: LGPL 2.1; Cppyy: LBNL BSD', - - classifiers=[ - 'Development Status :: 4 - Beta', - - 'Intended Audience :: Developers', - - 'Topic :: Software Development', - 'Topic :: Software Development :: Interpreters', - - #'License :: OSI Approved :: MIT License', - - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: Implementation :: PyPy', - 'Programming Language :: C', - 'Programming Language :: C++', - - 'Natural Language :: English' - ], - - keywords='interpreter development', - - packages=find_packages('src', ['backend']), - include_package_data=True, - - extras_require={ - }, - - cmdclass = { - 'build': my_cmake_build, - 'install': my_libs_install, - }, -) -""") - - -print('creating src ... ROOT part') -if not os.path.exists('src'): - os.mkdir('src') -os.chdir('src'); countdown += 1 -if not os.path.exists('backend'): - src = os.path.join(os.path.pardir, os.path.pardir, pkgdir) - print('now copying', src) - shutil.copytree(src, 'backend') - -print('creating src ... cppyy part') -os.chdir('backend'); countdown += 1 -if not os.path.exists('cppyy'): - os.mkdir('cppyy') - os.chdir('cppyy'); countdown += 1 - - with open('CMakeLists.txt', 'w') as outp: - outp.write("""############################################################################ -# CMakeLists.txt file for building cppyy package -############################################################################ - -ROOT_GLOB_SOURCES(sources ${CMAKE_CURRENT_SOURCE_DIR}/src/*.cxx) -set_source_files_properties(${sources} COMPILE_FLAGS "-fomit-frame-pointer -fvisibility=hidden -DRPY_EXTERN=RPY_EXPORTED -DRPYTHON_LL2CTYPES") - -add_definitions(${CLING_CXXFLAGS}) - -ROOT_LINKER_LIBRARY(cppyy_backend ${sources} - LIBRARIES ${CMAKE_DL_LIBS} - DEPENDENCIES Core Cling RIO Thread) - -add_dependencies(cppyy_backend CLING) -""") - - os.mkdir('src') - os.chdir('src'); countdown += 1 - print('pulling cppyy/clingcwrapper.cxx from pypy') - base = 'https://bitbucket.org/pypy/pypy/raw/default/pypy/module/cppyy/' - for cppyy_file in ['src/callcontext.h', 'include/capi.h', 'src/clingcwrapper.cxx', - 'include/clingcwrapper.h', 'include/cpp_cppyy.h', 'include/cppyy.h']: - resp = urllib2.urlopen(base+cppyy_file) - with open(os.path.basename(cppyy_file), 'w') as outp: - outp.write(resp.read()) - - # fix include - inp = 'capi.h' - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if 'src/precommondefs.h' in line: - line = '#include "precommondefs.h"\n' - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - - with open('precommondefs.h', 'w') as outp: - outp.write("""/***** Start of precommondefs.h *****/ - -/* This is extracted from pyconfig.h from CPython. It sets the macros - that affect the features we get from system include files. - It must not #include anything. */ - -#ifndef __PYPY_PRECOMMONDEFS_H -#define __PYPY_PRECOMMONDEFS_H - - -/* Define on Darwin to activate all library features */ -#define _DARWIN_C_SOURCE 1 -/* This must be set to 64 on some systems to enable large file support. */ -#define _FILE_OFFSET_BITS 64 -/* Define on Linux to activate all library features */ -#define _GNU_SOURCE 1 -/* This must be defined on some systems to enable large file support. */ -#define _LARGEFILE_SOURCE 1 -/* Define on NetBSD to activate all library features */ -#define _NETBSD_SOURCE 1 -/* Define to activate features from IEEE Stds 1003.1-2001 */ -#ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L -#endif -/* Define on FreeBSD to activate all library features */ -#define __BSD_VISIBLE 1 -#define __XSI_VISIBLE 700 -/* Windows: winsock/winsock2 mess */ -#define WIN32_LEAN_AND_MEAN -#ifdef _WIN64 - typedef __int64 Signed; - typedef unsigned __int64 Unsigned; -# define SIGNED_MIN LLONG_MIN -#else - typedef long Signed; - typedef unsigned long Unsigned; -# define SIGNED_MIN LONG_MIN -#endif - -#if !defined(RPY_ASSERT) && !defined(RPY_LL_ASSERT) && !defined(NDEBUG) -# define NDEBUG -#endif - - -/* All functions and global variables declared anywhere should use - one of the following attributes: - - RPY_EXPORTED: the symbol is exported out of libpypy-c.so. - - RPY_EXTERN: the symbol is not exported out of libpypy-c.so, but - otherwise works like 'extern' by being available to - other C sources. - - static: as usual, this means the symbol is local to this C file. - - Don't use _RPY_HIDDEN directly. For tests involving building a custom - .so, translator/tool/cbuild.py overrides RPY_EXTERN so that it becomes - equal to RPY_EXPORTED. - - Any function or global variable declared with no attribute at all is - a bug; please report or fix it. -*/ -#ifdef __GNUC__ -# define RPY_EXPORTED extern __attribute__((visibility("default"))) -# define _RPY_HIDDEN __attribute__((visibility("hidden"))) -#else -# define RPY_EXPORTED extern __declspec(dllexport) -# define _RPY_HIDDEN /* nothing */ -#endif -#ifndef RPY_EXTERN -# define RPY_EXTERN extern _RPY_HIDDEN -#endif - - -#endif /* __PYPY_PRECOMMONDEFS_H */ - -/***** End of precommondefs.h *****/ -""") - -# back up to pip package top -for i in range(countdown-1): - os.chdir(os.path.pardir) - -# add cppyy module to cmake -os.chdir('src/backend') -inp = 'CMakeLists.txt' -print('adding cppyy to cmake') -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'add_subdirectory' in line and 'net' in line: - line += 'add_subdirectory (cppyy)\n' - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# done! diff --git a/pypy/module/_cppyy/bench/Makefile b/pypy/module/_cppyy/bench/Makefile deleted file mode 100644 --- a/pypy/module/_cppyy/bench/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -all: bench02Dict_reflex.so - -ROOTSYS := ${ROOTSYS} - -ifeq ($(ROOTSYS),) - genreflex=genreflex - cppflags= -else - genreflex=$(ROOTSYS)/bin/genreflex - cppflags=-I$(ROOTSYS)/include -L$(ROOTSYS)/lib -endif - -PLATFORM := $(shell uname -s) -ifeq ($(PLATFORM),Darwin) - cppflags+=-dynamiclib -single_module -arch x86_64 -endif - -ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),) - genreflexflags= - cppflags2=-O3 -fPIC -else - genreflexflags=--with-methptrgetter - cppflags2=-Wno-pmf-conversions -O3 -fPIC -endif - - -bench02Dict_reflex.so: bench02.h bench02.cxx bench02.xml - $(genreflex) bench02.h $(genreflexflags) --selection=bench02.xml -I$(ROOTSYS)/include - g++ -o $@ bench02.cxx bench02_rflx.cpp -I$(ROOTSYS)/include -shared -std=c++11 -lHistPainter `root-config --libs` $(cppflags) $(cppflags2) diff --git a/pypy/module/_cppyy/bench/bench02.cxx b/pypy/module/_cppyy/bench/bench02.cxx deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.cxx +++ /dev/null @@ -1,79 +0,0 @@ -#include "bench02.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TDirectory.h" -#include "TInterpreter.h" -#include "TSystem.h" -#include "TBenchmark.h" -#include "TStyle.h" -#include "TError.h" -#include "Getline.h" -#include "TVirtualX.h" - -#include "Api.h" - -#include - -TClass *TClass::GetClass(const char*, Bool_t, Bool_t) { - static TClass* dummy = new TClass("__dummy__", kTRUE); - return dummy; // is deleted by gROOT at shutdown -} - -class TTestApplication : public TApplication { -public: - TTestApplication( - const char* acn, Int_t* argc, char** argv, Bool_t bLoadLibs = kTRUE); - virtual ~TTestApplication(); -}; - -TTestApplication::TTestApplication( - const char* acn, int* argc, char** argv, bool do_load) : TApplication(acn, argc, argv) { - if (do_load) { - // follow TRint to minimize differences with CINT - ProcessLine("#include ", kTRUE); - ProcessLine("#include <_string>", kTRUE); // for std::string iostream. - ProcessLine("#include ", kTRUE); // needed because they're used within the - ProcessLine("#include ", kTRUE); // core ROOT dicts and CINT won't be able - // to properly unload these files - } - - // save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // prevent crashes on accessing history - Gl_histinit((char*)"-"); - - // prevent ROOT from exiting python - SetReturnFromRun(kTRUE); -} - -TTestApplication::~TTestApplication() {} - -static const char* appname = "pypy-cppyy"; - -Bench02RootApp::Bench02RootApp() { - gROOT->SetBatch(kTRUE); - if (!gApplication) { - int argc = 1; - char* argv[1]; argv[0] = (char*)appname; - gApplication = new TTestApplication(appname, &argc, argv, kFALSE); - } -} - -Bench02RootApp::~Bench02RootApp() { - // TODO: ROOT globals cleanup ... (?) -} - -void Bench02RootApp::report() { - std::cout << "gROOT is: " << gROOT << std::endl; - std::cout << "gApplication is: " << gApplication << std::endl; -} - -void Bench02RootApp::close_file(TFile* f) { - std::cout << "closing file " << f->GetName() << " ... " << std::endl; - f->Write(); - f->Close(); - std::cout << "... file closed" << std::endl; -} diff --git a/pypy/module/_cppyy/bench/bench02.h b/pypy/module/_cppyy/bench/bench02.h deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.h +++ /dev/null @@ -1,72 +0,0 @@ -#include "TString.h" - -#include "TCanvas.h" -#include "TFile.h" -#include "TProfile.h" -#include "TNtuple.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TRandom.h" -#include "TRandom3.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TSystem.h" - -#include "TArchiveFile.h" -#include "TBasket.h" -#include "TBenchmark.h" -#include "TBox.h" -#include "TBranchRef.h" -#include "TBrowser.h" -#include "TClassGenerator.h" -#include "TClassRef.h" -#include "TClassStreamer.h" -#include "TContextMenu.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TF1.h" -#include "TFileCacheRead.h" -#include "TFileCacheWrite.h" -#include "TFileMergeInfo.h" -#include "TFitResult.h" -#include "TFolder.h" -//#include "TFormulaPrimitive.h" -#include "TFunction.h" -#include "TFrame.h" -#include "TGlobal.h" -#include "THashList.h" -#include "TInetAddress.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TLegend.h" -#include "TMethodCall.h" -#include "TPluginManager.h" -#include "TProcessUUID.h" -#include "TSchemaRuleSet.h" -#include "TStyle.h" -#include "TSysEvtHandler.h" -#include "TTimer.h" -#include "TView.h" -//#include "TVirtualCollectionProxy.h" -#include "TVirtualFFT.h" -#include "TVirtualHistPainter.h" -#include "TVirtualIndex.h" -#include "TVirtualIsAProxy.h" -#include "TVirtualPadPainter.h" -#include "TVirtualRefProxy.h" -#include "TVirtualStreamerInfo.h" -#include "TVirtualViewer3D.h" - -#include -#include - - -class Bench02RootApp { -public: - Bench02RootApp(); - ~Bench02RootApp(); - - void report(); - void close_file(TFile* f); -}; diff --git a/pypy/module/_cppyy/bench/bench02.xml b/pypy/module/_cppyy/bench/bench02.xml deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pypy/module/_cppyy/bench/hsimple.C b/pypy/module/_cppyy/bench/hsimple.C deleted file mode 100644 --- a/pypy/module/_cppyy/bench/hsimple.C +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -TFile *hsimple(Int_t get=0) -{ -// This program creates : -// - a one dimensional histogram -// - a two dimensional histogram -// - a profile histogram -// - a memory-resident ntuple -// -// These objects are filled with some random numbers and saved on a file. -// If get=1 the macro returns a pointer to the TFile of "hsimple.root" -// if this file exists, otherwise it is created. -// The file "hsimple.root" is created in $ROOTSYS/tutorials if the caller has -// write access to this directory, otherwise the file is created in $PWD - - TString filename = "hsimple.root"; - TString dir = gSystem->UnixPathName(gInterpreter->GetCurrentMacroName()); - dir.ReplaceAll("hsimple.C",""); - dir.ReplaceAll("/./","/"); - TFile *hfile = 0; - if (get) { - // if the argument get =1 return the file "hsimple.root" - // if the file does not exist, it is created - TString fullPath = dir+"hsimple.root"; - if (!gSystem->AccessPathName(fullPath,kFileExists)) { - hfile = TFile::Open(fullPath); //in $ROOTSYS/tutorials - if (hfile) return hfile; - } - //otherwise try $PWD/hsimple.root - if (!gSystem->AccessPathName("hsimple.root",kFileExists)) { - hfile = TFile::Open("hsimple.root"); //in current dir - if (hfile) return hfile; - } - } - //no hsimple.root file found. Must generate it ! - //generate hsimple.root in $ROOTSYS/tutorials if we have write access - if (!gSystem->AccessPathName(dir,kWritePermission)) { - filename = dir+"hsimple.root"; - } else if (!gSystem->AccessPathName(".",kWritePermission)) { - //otherwise generate hsimple.root in the current directory - } else { - printf("you must run the script in a directory with write access\n"); - return 0; - } - hfile = (TFile*)gROOT->FindObject(filename); if (hfile) hfile->Close(); - hfile = new TFile(filename,"RECREATE","Demo ROOT file with histograms"); - - // Create some histograms, a profile histogram and an ntuple - TH1F *hpx = new TH1F("hpx","This is the px distribution",100,-4,4); - hpx->SetFillColor(48); - TH2F *hpxpy = new TH2F("hpxpy","py vs px",40,-4,4,40,-4,4); - TProfile *hprof = new TProfile("hprof","Profile of pz versus px",100,-4,4,0,20); - TNtuple *ntuple = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - - gBenchmark->Start("hsimple"); - - // Create a new canvas. - TCanvas *c1 = new TCanvas("c1","Dynamic Filling Example",200,10,700,500); - c1->SetFillColor(42); - c1->GetFrame()->SetFillColor(21); - c1->GetFrame()->SetBorderSize(6); - c1->GetFrame()->SetBorderMode(-1); - - - // Fill histograms randomly - TRandom3 random; - Float_t px, py, pz; - const Int_t kUPDATE = 1000; - for (Int_t i = 0; i < 50000; i++) { - // random.Rannor(px,py); - px = random.Gaus(0, 1); - py = random.Gaus(0, 1); - pz = px*px + py*py; - Float_t rnd = random.Rndm(1); - hpx->Fill(px); - hpxpy->Fill(px,py); - hprof->Fill(px,pz); - ntuple->Fill(px,py,pz,rnd,i); - if (i && (i%kUPDATE) == 0) { - if (i == kUPDATE) hpx->Draw(); - c1->Modified(); - c1->Update(); - if (gSystem->ProcessEvents()) - break; - } - } - gBenchmark->Show("hsimple"); - - // Save all objects in this file - hpx->SetFillColor(0); - hfile->Write(); - hpx->SetFillColor(48); - c1->Modified(); - return hfile; - -// Note that the file is automatically close when application terminates -// or when the file destructor is called. -} diff --git a/pypy/module/_cppyy/bench/hsimple.py b/pypy/module/_cppyy/bench/hsimple.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple.py +++ /dev/null @@ -1,110 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -_reflex = True # to keep things equal, set to False for full macro - -try: - import cppyy, random - - if not hasattr(cppyy.gbl, 'gROOT'): - cppyy.load_reflection_info('bench02Dict_reflex.so') - _reflex = True - - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom3 = cppyy.gbl.TRandom3 - - gROOT = cppyy.gbl.gROOT - gBenchmark = cppyy.gbl.TBenchmark() - gSystem = cppyy.gbl.gSystem - -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom3 - from ROOT import gROOT, gBenchmark, gSystem - import random - -if _reflex: - gROOT.SetBatch(True) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -if not _reflex: - hfile = gROOT.FindObject('hsimple.root') - if hfile: - hfile.Close() - hfile = TFile('hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.SetFillColor(48) -hpxpy = TH2F('hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4) -hprof = TProfile('hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20) -if not _reflex: - ntuple = TNtuple('ntuple', 'Demo ntuple', 'px:py:pz:random:i') - -gBenchmark.Start('hsimple') - -# Create a new canvas, and customize it. -c1 = TCanvas('c1', 'Dynamic Filling Example', 200, 10, 700, 500) -c1.SetFillColor(42) -c1.GetFrame().SetFillColor(21) -c1.GetFrame().SetBorderSize(6) -c1.GetFrame().SetBorderMode(-1) - -# Fill histograms randomly. -random = TRandom3() -kUPDATE = 1000 -for i in xrange(50000): - # Generate random numbers -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) - pz = px*px + py*py -# rnd = random.random() - rnd = random.Rndm(1) - - # Fill histograms - hpx.Fill(px) - hpxpy.Fill(px, py) - hprof.Fill(px, pz) - if not _reflex: - ntuple.Fill(px, py, pz, rnd, i) - - # Update display every kUPDATE events - if i and i%kUPDATE == 0: - if i == kUPDATE: - hpx.Draw() - - c1.Modified(True) - c1.Update() - - if gSystem.ProcessEvents(): # allow user interrupt - break - -gBenchmark.Show( 'hsimple' ) - -# Save all objects in this file -hpx.SetFillColor(0) -if not _reflex: - hfile.Write() -hpx.SetFillColor(48) -c1.Modified(True) -c1.Update() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/bench/hsimple_rflx.py b/pypy/module/_cppyy/bench/hsimple_rflx.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple_rflx.py +++ /dev/null @@ -1,120 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -try: - import warnings - warnings.simplefilter("ignore") - - import cppyy, random - cppyy.load_reflection_info('bench02Dict_reflex.so') - - app = cppyy.gbl.Bench02RootApp() - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom = cppyy.gbl.TRandom -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom - import random - -import math - -#gROOT = cppyy.gbl.gROOT -#gBenchmark = cppyy.gbl.gBenchmark -#gRandom = cppyy.gbl.gRandom -#gSystem = cppyy.gbl.gSystem - -#gROOT.Reset() - -# Create a new canvas, and customize it. -#c1 = TCanvas( 'c1', 'Dynamic Filling Example', 200, 10, 700, 500 ) -#c1.SetFillColor( 42 ) -#c1.GetFrame().SetFillColor( 21 ) -#c1.GetFrame().SetBorderSize( 6 ) -#c1.GetFrame().SetBorderMode( -1 ) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -#hfile = gROOT.FindObject( 'hsimple.root' ) -#if hfile: -# hfile.Close() -#hfile = TFile( 'hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.Print() -#hpxpy = TH2F( 'hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4 ) -#hprof = TProfile( 'hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20 ) -#ntuple = TNtuple( 'ntuple', 'Demo ntuple', 'px:py:pz:random:i' ) - -# Set canvas/frame attributes. -#hpx.SetFillColor( 48 ) - -#gBenchmark.Start( 'hsimple' ) - -# Initialize random number generator. -#gRandom.SetSeed() -#rannor, rndm = gRandom.Rannor, gRandom.Rndm - -random = TRandom() -random.SetSeed(0) - -# Fill histograms randomly. -#px, py = Double(), Double() -kUPDATE = 1000 -for i in xrange(2500000): - # Generate random values. -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) -# pt = (px*px + py*py)**0.5 - pt = math.sqrt(px*px + py*py) -# pt = (px*px + py*py) -# random = rndm(1) - - # Fill histograms. - hpx.Fill(pt) -# hpxpyFill( px, py ) -# hprofFill( px, pz ) -# ntupleFill( px, py, pz, random, i ) - - # Update display every kUPDATE events. -# if i and i%kUPDATE == 0: -# if i == kUPDATE: -# hpx.Draw() - -# c1.Modified() -# c1.Update() - -# if gSystem.ProcessEvents(): # allow user interrupt -# break - -#gBenchmark.Show( 'hsimple' ) - -hpx.Print() - -# Save all objects in this file. -#hpx.SetFillColor( 0 ) -#hfile.Write() -#hfile.Close() -#hpx.SetFillColor( 48 ) -#c1.Modified() -#c1.Update() -#c1.Draw() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/capi/__init__.py b/pypy/module/_cppyy/capi/__init__.py --- a/pypy/module/_cppyy/capi/__init__.py +++ b/pypy/module/_cppyy/capi/__init__.py @@ -1,16 +1,6 @@ from rpython.rtyper.lltypesystem import rffi, lltype -# There are two possible ways of accessing the backend through the reflection -# C-API: built it into pypy-c, or load it dynamically. The latter is preferred -# (and is the default) for use with Reflex. B/c of some builtin pythonizations, -# the former is recommended (for now) with CINT. - -# Note: if builtin_capi is chosen, then inside builtin_capi.py, there is still -# the selection of the desired backend (default is Reflex). - -# choose C-API access method: from pypy.module._cppyy.capi.loadable_capi import * -#from pypy.module._cppyy.capi.builtin_capi import * from pypy.module._cppyy.capi.capi_types import C_OBJECT,\ C_NULL_TYPE, C_NULL_OBJECT diff --git a/pypy/module/_cppyy/capi/builtin_capi.py b/pypy/module/_cppyy/capi/builtin_capi.py deleted file mode 100644 --- a/pypy/module/_cppyy/capi/builtin_capi.py +++ /dev/null @@ -1,590 +0,0 @@ -from rpython.rtyper.lltypesystem import rffi, lltype -from rpython.rlib.rarithmetic import intmask -from rpython.rlib import jit - -import cling_capi as backend - -from pypy.module._cppyy.capi.capi_types import C_SCOPE, C_TYPE, C_OBJECT,\ - C_METHOD, C_INDEX, C_INDEX_ARRAY, WLAVC_INDEX, C_FUNC_PTR - -identify = backend.identify -pythonize = backend.pythonize -register_pythonizations = backend.register_pythonizations -std_string_name = backend.std_string_name - -ts_reflect = backend.ts_reflect -ts_call = backend.ts_call -ts_memory = backend.ts_memory -ts_helper = backend.ts_helper - -def verify_backend(space): - return True # by definition - -c_load_dictionary = backend.c_load_dictionary - -# name to opaque C++ scope representation ------------------------------------ -_c_num_scopes = rffi.llexternal( - "cppyy_num_scopes", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_scopes(space, cppscope): - return _c_num_scopes(cppscope.handle) -_c_scope_name = rffi.llexternal( - "cppyy_scope_name", - [C_SCOPE, rffi.INT], rffi.CCHARP, - compilation_info = backend.eci) -def c_scope_name(space, cppscope, iscope): - return charp2str_free(space, _c_scope_name(cppscope.handle, iscope)) - -_c_resolve_name = rffi.llexternal( - "cppyy_resolve_name", - [rffi.CCHARP], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_resolve_name(space, name): - return charp2str_free(space, _c_resolve_name(name)) -_c_get_scope_opaque = rffi.llexternal( - "cppyy_get_scope", - [rffi.CCHARP], C_SCOPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_scope_opaque(space, name): - return _c_get_scope_opaque(name) -_c_actual_class = rffi.llexternal( - "cppyy_actual_class", - [C_TYPE, C_OBJECT], C_TYPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_actual_class(space, cppclass, cppobj): - return _c_actual_class(cppclass.handle, cppobj) - -# memory management ---------------------------------------------------------- -_c_allocate = rffi.llexternal( - "cppyy_allocate", - [C_TYPE], C_OBJECT, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate(space, cppclass): - return _c_allocate(cppclass.handle) -_c_deallocate = rffi.llexternal( - "cppyy_deallocate", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate(space, cppclass, cppobject): - _c_deallocate(cppclass.handle, cppobject) -_c_destruct = rffi.llexternal( - "cppyy_destruct", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_destruct(space, cppclass, cppobject): - _c_destruct(cppclass.handle, cppobject) - -# method/function dispatching ------------------------------------------------ -_c_call_v = rffi.llexternal( - "cppyy_call_v", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_v(space, cppmethod, cppobject, nargs, args): - _c_call_v(cppmethod, cppobject, nargs, args) -_c_call_b = rffi.llexternal( - "cppyy_call_b", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.UCHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_b(space, cppmethod, cppobject, nargs, args): - return _c_call_b(cppmethod, cppobject, nargs, args) -_c_call_c = rffi.llexternal( - "cppyy_call_c", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.CHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_c(space, cppmethod, cppobject, nargs, args): - return _c_call_c(cppmethod, cppobject, nargs, args) -_c_call_h = rffi.llexternal( - "cppyy_call_h", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.SHORT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_h(space, cppmethod, cppobject, nargs, args): - return _c_call_h(cppmethod, cppobject, nargs, args) -_c_call_i = rffi.llexternal( - "cppyy_call_i", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.INT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_i(space, cppmethod, cppobject, nargs, args): - return _c_call_i(cppmethod, cppobject, nargs, args) -_c_call_l = rffi.llexternal( - "cppyy_call_l", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_l(space, cppmethod, cppobject, nargs, args): - return _c_call_l(cppmethod, cppobject, nargs, args) -_c_call_ll = rffi.llexternal( - "cppyy_call_ll", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGLONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ll(space, cppmethod, cppobject, nargs, args): - return _c_call_ll(cppmethod, cppobject, nargs, args) -_c_call_f = rffi.llexternal( - "cppyy_call_f", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.FLOAT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_f(space, cppmethod, cppobject, nargs, args): - return _c_call_f(cppmethod, cppobject, nargs, args) -_c_call_d = rffi.llexternal( - "cppyy_call_d", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.DOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_d(space, cppmethod, cppobject, nargs, args): - return _c_call_d(cppmethod, cppobject, nargs, args) -_c_call_ld = rffi.llexternal( - "cppyy_call_ld", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGDOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ld(space, cppmethod, cppobject, nargs, args): - return _c_call_ld(cppmethod, cppobject, nargs, args) - -_c_call_r = rffi.llexternal( - "cppyy_call_r", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.VOIDP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_r(space, cppmethod, cppobject, nargs, args): - return _c_call_r(cppmethod, cppobject, nargs, args) -_c_call_s = rffi.llexternal( - "cppyy_call_s", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, rffi.SIZE_TP], rffi.CCHARP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_s(space, cppmethod, cppobject, nargs, args): - length = lltype.malloc(rffi.SIZE_TP.TO, 1, flavor='raw') - try: - cstr = _c_call_s(cppmethod, cppobject, nargs, args, length) - cstr_len = intmask(length[0]) - finally: - lltype.free(length, flavor='raw') - return cstr, cstr_len - -_c_constructor = rffi.llexternal( - "cppyy_constructor", - [C_METHOD, C_TYPE, rffi.INT, rffi.VOIDP], C_OBJECT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_constructor(space, cppmethod, cppobject, nargs, args): - return _c_constructor(cppmethod, cppobject, nargs, args) -_c_call_o = rffi.llexternal( - "cppyy_call_o", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, C_TYPE], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_o(space, method, cppobj, nargs, args, cppclass): - return _c_call_o(method, cppobj, nargs, args, cppclass.handle) - -_c_get_function_address = rffi.llexternal( - "cppyy_get_function_address", - [C_SCOPE, C_INDEX], C_FUNC_PTR, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) -def c_get_function_address(space, cppscope, index): - return _c_get_function_address(cppscope.handle, index) - -# handling of function argument buffer --------------------------------------- -_c_allocate_function_args = rffi.llexternal( - "cppyy_allocate_function_args", - [rffi.SIZE_T], rffi.VOIDP, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate_function_args(space, size): - return _c_allocate_function_args(size) -_c_deallocate_function_args = rffi.llexternal( - "cppyy_deallocate_function_args", - [rffi.VOIDP], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate_function_args(space, args): - _c_deallocate_function_args(args) -_c_function_arg_sizeof = rffi.llexternal( - "cppyy_function_arg_sizeof", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_sizeof(space): - return _c_function_arg_sizeof() -_c_function_arg_typeoffset = rffi.llexternal( - "cppyy_function_arg_typeoffset", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_typeoffset(space): - return _c_function_arg_typeoffset() - -# scope reflection information ----------------------------------------------- -_c_is_namespace = rffi.llexternal( - "cppyy_is_namespace", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_namespace(space, scope): - return _c_is_namespace(scope) -_c_is_template = rffi.llexternal( - "cppyy_is_template", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_template(space, name): - return _c_is_template(name) -_c_is_abstract = rffi.llexternal( - "cppyy_is_abstract", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_abstract(space, cpptype): - return _c_is_abstract(cpptype) -_c_is_enum = rffi.llexternal( - "cppyy_is_enum", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_enum(space, name): - return _c_is_enum(name) - -# type/class reflection information ------------------------------------------ -_c_final_name = rffi.llexternal( - "cppyy_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_final_name(space, cpptype): - return charp2str_free(space, _c_final_name(cpptype)) -_c_scoped_final_name = rffi.llexternal( - "cppyy_scoped_final_name", - [C_TYPE], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_scoped_final_name(space, cpptype): - return charp2str_free(space, _c_scoped_final_name(cpptype)) -_c_has_complex_hierarchy = rffi.llexternal( - "cppyy_has_complex_hierarchy", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_has_complex_hierarchy(space, cpptype): - return _c_has_complex_hierarchy(cpptype) -_c_num_bases = rffi.llexternal( - "cppyy_num_bases", - [C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_bases(space, cppclass): - return _c_num_bases(cppclass.handle) -_c_base_name = rffi.llexternal( - "cppyy_base_name", - [C_TYPE, rffi.INT], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_base_name(space, cppclass, base_index): - return charp2str_free(space, _c_base_name(cppclass.handle, base_index)) -_c_is_subtype = rffi.llexternal( - "cppyy_is_subtype", - [C_TYPE, C_TYPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_is_subtype(space, derived, base): - if derived == base: - return 1 - return _c_is_subtype(derived.handle, base.handle) - -_c_base_offset = rffi.llexternal( - "cppyy_base_offset", - [C_TYPE, C_TYPE, C_OBJECT, rffi.INT], rffi.LONG, # actually ptrdiff_t - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_base_offset(space, derived, base, address, direction): - if derived == base: - return 0 - return _c_base_offset(derived.handle, base.handle, address, direction) -def c_base_offset1(space, derived_h, base, address, direction): - return _c_base_offset(derived_h, base.handle, address, direction) - -# method/function reflection information ------------------------------------- -_c_num_methods = rffi.llexternal( - "cppyy_num_methods", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_methods(space, cppscope): - return _c_num_methods(cppscope.handle) -_c_method_index_at = rffi.llexternal( - "cppyy_method_index_at", - [C_SCOPE, rffi.INT], C_INDEX, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_method_index_at(space, cppscope, imethod): - return _c_method_index_at(cppscope.handle, imethod) -_c_method_indices_from_name = rffi.llexternal( - "cppyy_method_indices_from_name", - [C_SCOPE, rffi.CCHARP], C_INDEX_ARRAY, From pypy.commits at gmail.com Wed Oct 11 10:24:38 2017 From: pypy.commits at gmail.com (antocuni) Date: Wed, 11 Oct 2017 07:24:38 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: (antocuni, arigo): first draft for the sprint report Message-ID: <59de29a6.09a0df0a.ea0b7.050e@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5838:fada16ff748b Date: 2017-10-11 16:24 +0200 http://bitbucket.org/pypy/extradoc/changeset/fada16ff748b/ Log: (antocuni, arigo): first draft for the sprint report diff --git a/blog/draft/2017-10-cape-town-report.rst b/blog/draft/2017-10-cape-town-report.rst new file mode 100644 --- /dev/null +++ b/blog/draft/2017-10-cape-town-report.rst @@ -0,0 +1,56 @@ +(Cape of) Good Hope for PyPy +---------------------------- + +Hello from the other side of the world (for most of you)! + +With the excuse of coming to PyCon ZA[1], during the last two weeks Armin, +Ronan, Antonio and Maciek had a very nice and productive sprint in Cape Town, +as pictures show :). + +XXX insert pic + +We spent most of the time hacking at cpyext, our CPython C-API compatibility +layer: during the last years, the focus was to make it working and compatible +with CPython, in order to run existing libraries such as numpy and +pandas. However, we never paid too much attention to performance, so the net +result is that with the latest released version of PyPy, C extensions +generally work but their speed ranges from "slow" to "horribly slow". + +For example, these `very simple micro-benchmarks`_ measure the speed of +calling (empty) C functions, i.e. the time you spend to "cross the border" +between RPython and C. These are the results on CPython, on PyPy 5.9, and on +our newest in-progress version:: + + bla bla bla + + +.. origin git at github.com:antocuni/cpyext-benchmarks.git (fetch) + + +To reach this result, we did various improvements, such as: + + 1. teach the JIT how to look (a bit) inside the cpyext module + + 2. write specialized code for calling ``METH_NOARGS``, ``METH_O`` and + ``METH_VARARGS`` functions; previously, we always used a very general and + slow logic + + 3. the `cpyext-avoid-roundtrip`_ branch: crossing the RPython/C border is + slowish, but the real problem was (and still is for many cases) we often + cross it many times for no good reason. So, depending on the actual API + call, you might end up in the C land, which calls back into the RPython + land, which goes to C, etc. etc. (ad libitum). + +The branch tries to fix such nonsense: so far, we fixed only some cases, which +are enough to speed up the benchmarks shown above. But most importantly, we +now have a clear path to improve cpyext more and more. + +For example, the ``METH_VARARGS`` case is still a bit slowish because we need +to ``malloc()`` a new cpyext tuple for every call: the plan is to use the same +strategy as CPython, i.e. to use free lists to be able to quickly reuse the +memory as soon as the tuples die. + +bla bla bla + + + From pypy.commits at gmail.com Wed Oct 11 14:03:43 2017 From: pypy.commits at gmail.com (fijal) Date: Wed, 11 Oct 2017 11:03:43 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: essential Message-ID: <59de5cff.12a4df0a.67999.c392@mx.google.com> Author: fijal Branch: extradoc Changeset: r5839:622f1baa8223 Date: 2017-10-11 20:03 +0200 http://bitbucket.org/pypy/extradoc/changeset/622f1baa8223/ Log: essential diff --git a/blog/draft/2017-10-cape-town-report.rst b/blog/draft/2017-10-cape-town-report.rst --- a/blog/draft/2017-10-cape-town-report.rst +++ b/blog/draft/2017-10-cape-town-report.rst @@ -4,7 +4,7 @@ Hello from the other side of the world (for most of you)! With the excuse of coming to PyCon ZA[1], during the last two weeks Armin, -Ronan, Antonio and Maciek had a very nice and productive sprint in Cape Town, +Ronan, Antonio and sometimes Maciek had a very nice and productive sprint in Cape Town, as pictures show :). XXX insert pic diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst --- a/blog/draft/remove-gil.rst +++ b/blog/draft/remove-gil.rst @@ -15,6 +15,7 @@ counting. See the section below for further details. .. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html +.. _`gilectomy`: https://github.com/larryhastings/gilectomy As we announced at EuroPython, what we have so far is a GIL-less PyPy which can run **very simple** multi-threaded, nicely parallelized, programs. From pypy.commits at gmail.com Wed Oct 11 19:48:55 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:48:55 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: merge default into branch Message-ID: <59deade7.0ec61c0a.6f3a9.29b4@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92721:63eea9e3fded Date: 2017-10-06 14:24 -0700 http://bitbucket.org/pypy/pypy/changeset/63eea9e3fded/ Log: merge default into branch diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/pypy/module/_cppyy/test/conftest.py b/pypy/module/_cppyy/test/conftest.py --- a/pypy/module/_cppyy/test/conftest.py +++ b/pypy/module/_cppyy/test/conftest.py @@ -4,7 +4,7 @@ def pytest_runtest_setup(item): if py.path.local.sysfind('genreflex') is None: import pypy.module._cppyy.capi.loadable_capi as lcapi - if 'dummy' in lcapi.reflection_library: + if 'dummy' in lcapi.backend_library: # run only tests that are covered by the dummy backend and tests # that do not rely on reflex import os @@ -33,7 +33,7 @@ import pypy.module._cppyy.capi.loadable_capi as lcapi try: import ctypes - ctypes.CDLL(lcapi.reflection_library) + ctypes.CDLL(lcapi.backend_library) except Exception as e: if config.option.runappdirect: return # "can't run dummy tests in -A" @@ -71,4 +71,4 @@ return raise - lcapi.reflection_library = str(soname) + lcapi.backend_library = str(soname) diff --git a/rpython/rlib/rsiphash.py b/rpython/rlib/rsiphash.py --- a/rpython/rlib/rsiphash.py +++ b/rpython/rlib/rsiphash.py @@ -11,6 +11,7 @@ from rpython.rlib.objectmodel import not_rpython, always_inline from rpython.rlib.objectmodel import we_are_translated, dont_inline from rpython.rlib.objectmodel import keepalive_until_here +from rpython.rlib.objectmodel import specialize from rpython.rlib import rgc, jit, rposix from rpython.rlib.rarithmetic import r_uint64, r_uint32, r_uint from rpython.rlib.rawstorage import misaligned_is_fine @@ -26,9 +27,11 @@ return x def _le32toh(x): return x + BIG_ENDIAN = False else: _le64toh = rarithmetic.byteswap _le32toh = rarithmetic.byteswap + BIG_ENDIAN = True def _decode64(s): return (r_uint64(ord(s[0])) | @@ -49,6 +52,7 @@ """'s' is a string of length 16""" seed.k0l = _decode64(s) seed.k1l = _decode64(s[8:16]) + _update_prebuilt_hashes() random_ctx = rurandom.init_urandom() @@ -151,23 +155,20 @@ else: # NOTE: a latin-1 unicode string must have the same hash as the # corresponding byte string. If the unicode is all within - # 0-255, then we need to allocate a byte buffer and copy the - # latin-1 encoding in it manually. Note also that we give a + # 0-255, then we call _siphash24() with a special argument that + # will make it load only one byte for every unicode char. + # Note also that we give a # different hash result than CPython on ucs4 platforms, for # unicode strings where CPython uses 2 bytes per character. + addr = rstr._get_raw_buf_unicode(rstr.UNICODE, ll_s, 0) + SZ = rffi.sizeof(rstr.UNICODE.chars.OF) for i in range(length): if ord(ll_s.chars[i]) > 0xFF: - addr = rstr._get_raw_buf_unicode(rstr.UNICODE, ll_s, 0) - length *= rffi.sizeof(rstr.UNICODE.chars.OF) + length *= SZ break else: - p = lltype.malloc(rffi.CCHARP.TO, length, flavor='raw') - i = 0 - while i < length: - p[i] = chr(ord(ll_s.chars[i])) - i += 1 - x = _siphash24(llmemory.cast_ptr_to_adr(p), length) - lltype.free(p, flavor='raw') + x = _siphash24(addr, length, SZ) + keepalive_until_here(ll_s) return intmask(x) x = _siphash24(addr, length) keepalive_until_here(ll_s) @@ -175,16 +176,22 @@ @contextmanager -def choosen_seed(new_k0, new_k1, test_misaligned_path=False): +def choosen_seed(new_k0, new_k1, test_misaligned_path=False, + test_prebuilt=False): """For tests.""" - global misaligned_is_fine - old = seed.k0l, seed.k1l, misaligned_is_fine + global misaligned_is_fine, seed + old = seed, misaligned_is_fine + seed = Seed() seed.k0l = r_uint64(new_k0) seed.k1l = r_uint64(new_k1) + if test_prebuilt: + _update_prebuilt_hashes() + else: + seed.bound_prebuilt_size = 0 if test_misaligned_path: misaligned_is_fine = False yield - seed.k0l, seed.k1l, misaligned_is_fine = old + seed, misaligned_is_fine = old magic0 = r_uint64(0x736f6d6570736575) magic1 = r_uint64(0x646f72616e646f6d) @@ -215,10 +222,22 @@ @rgc.no_collect -def _siphash24(addr_in, size): + at specialize.arg(2) +def _siphash24(addr_in, size, SZ=1): """Takes an address pointer and a size. Returns the hash as a r_uint64, which can then be casted to the expected type.""" + if BIG_ENDIAN: + index = SZ - 1 + else: + index = 0 + if size < seed.bound_prebuilt_size: + if size <= 0: + return seed.hash_empty + else: + t = rarithmetic.intmask(llop.raw_load(rffi.UCHAR, addr_in, index)) + return seed.hash_single[t] + k0 = seed.k0l k1 = seed.k1l b = r_uint64(size) << 56 @@ -227,10 +246,10 @@ v2 = k0 ^ magic2 v3 = k1 ^ magic3 - direct = (misaligned_is_fine or + direct = (SZ == 1) and (misaligned_is_fine or (rffi.cast(lltype.Signed, addr_in) & 7) == 0) - index = 0 if direct: + assert SZ == 1 while size >= 8: mi = llop.raw_load(rffi.ULONGLONG, addr_in, index) mi = _le64toh(mi) @@ -242,30 +261,30 @@ else: while size >= 8: mi = ( - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1)) << 8 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2)) << 16 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3)) << 24 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4)) << 32 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5)) << 40 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6)) << 48 | - r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 7)) << 56 + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1*SZ)) << 8 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2*SZ)) << 16 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3*SZ)) << 24 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4*SZ)) << 32 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5*SZ)) << 40 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6*SZ)) << 48 | + r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 7*SZ)) << 56 ) size -= 8 - index += 8 + index += 8*SZ v3 ^= mi v0, v1, v2, v3 = _double_round(v0, v1, v2, v3) v0 ^= mi t = r_uint64(0) if size == 7: - t = r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6)) << 48 + t = r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 6*SZ)) << 48 size = 6 if size == 6: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5)) << 40 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 5*SZ)) << 40 size = 5 if size == 5: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4)) << 32 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 4*SZ)) << 32 size = 4 if size == 4: if direct: @@ -273,13 +292,13 @@ t |= r_uint64(v) size = 0 else: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3)) << 24 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 3*SZ))<< 24 size = 3 if size == 3: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2)) << 16 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 2*SZ)) << 16 size = 2 if size == 2: - t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1)) << 8 + t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index + 1*SZ)) << 8 size = 1 if size == 1: t |= r_uint64(llop.raw_load(rffi.UCHAR, addr_in, index)) @@ -306,3 +325,17 @@ """ with rffi.scoped_nonmovingbuffer(s) as p: return _siphash24(llmemory.cast_ptr_to_adr(p), len(s)) + + +# Prebuilt hashes are precomputed here +def _update_prebuilt_hashes(): + seed.bound_prebuilt_size = 0 + with lltype.scoped_alloc(rffi.CCHARP.TO, 1) as p: + addr = llmemory.cast_ptr_to_adr(p) + seed.hash_single = [r_uint64(0)] * 256 + for i in range(256): + p[0] = chr(i) + seed.hash_single[i] = _siphash24(addr, 1) + seed.hash_empty = _siphash24(addr, 0) + seed.bound_prebuilt_size = 2 +_update_prebuilt_hashes() diff --git a/rpython/rlib/test/test_rsiphash.py b/rpython/rlib/test/test_rsiphash.py --- a/rpython/rlib/test/test_rsiphash.py +++ b/rpython/rlib/test/test_rsiphash.py @@ -1,8 +1,10 @@ import os from rpython.rlib.rsiphash import siphash24, _siphash24, choosen_seed from rpython.rlib.rsiphash import initialize_from_env, enable_siphash24 +from rpython.rlib.rsiphash import ll_hash_string_siphash24 from rpython.rlib.objectmodel import compute_hash from rpython.rlib.rarithmetic import intmask +from rpython.rtyper.annlowlevel import llstr, llunicode from rpython.rtyper.lltypesystem import llmemory, rffi from rpython.translator.c.test.test_genc import compile @@ -38,14 +40,30 @@ test_misaligned_path=True): x = siphash24(s) y = _siphash24(llmemory.cast_ptr_to_adr(rffi.ptradd(q, 1)), len(s)) + z = ll_hash_string_siphash24(llstr(s)) rffi.free_charp(q) assert x == y + assert z == intmask(x) return x def test_siphash24(): for expected, string in CASES: assert check(string) == expected +def check_latin1(s, expected, test_prebuilt=False): + with choosen_seed(0x8a9f065a358479f4, 0x11cb1e9ee7f40e1f, + test_misaligned_path=True, test_prebuilt=test_prebuilt): + z = ll_hash_string_siphash24(llunicode(s)) + assert z == intmask(expected) + +def test_siphash24_latin1_unicode(): + for expected, string in CASES: + check_latin1(string.decode('latin1'), expected) + +def test_siphash24_latin1_unicode_prebuilt(): + for expected, string in CASES: + check_latin1(string.decode('latin1'), expected, test_prebuilt=True) + def test_fix_seed(): old_val = os.environ.get('PYTHONHASHSEED', None) try: From pypy.commits at gmail.com Wed Oct 11 19:48:57 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:48:57 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: naming consistency and equivalence with CPython Message-ID: <59deade9.c7c4df0a.668f3.f42b@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92722:b547e0ca6aba Date: 2017-10-11 11:34 -0700 http://bitbucket.org/pypy/pypy/changeset/b547e0ca6aba/ Log: naming consistency and equivalence with CPython diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -7,7 +7,7 @@ interpleveldefs = { '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', - '_template_byname' : 'interp_cppyy.template_byname', + '_is_template' : 'interp_cppyy.is_template', '_std_string_name' : 'interp_cppyy.std_string_name', '_set_class_generator' : 'interp_cppyy.set_class_generator', '_set_function_generator': 'interp_cppyy.set_function_generator', @@ -15,6 +15,7 @@ '_get_nullptr' : 'interp_cppyy.get_nullptr', 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', + '_bind_object' : 'interp_cppyy._bind_object', 'bind_object' : 'interp_cppyy.bind_object', } diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -495,9 +495,9 @@ def _unwrap_object(self, space, w_obj): from pypy.module._cppyy.interp_cppyy import W_CPPClass if isinstance(w_obj, W_CPPClass): - if capi.c_is_subtype(space, w_obj.cppclass, self.clsdecl): + if capi.c_is_subtype(space, w_obj.clsdecl, self.clsdecl): rawobject = w_obj.get_rawobject() - offset = capi.c_base_offset(space, w_obj.cppclass, self.clsdecl, rawobject, 1) + offset = capi.c_base_offset(space, w_obj.clsdecl, self.clsdecl, rawobject, 1) obj_address = capi.direct_ptradd(rawobject, offset) return rffi.cast(capi.C_OBJECT, obj_address) raise oefmt(space.w_TypeError, @@ -527,7 +527,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): self._is_abstract(space) @@ -548,7 +548,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): address = rffi.cast(rffi.VOIDPP, self._get_raw_address(space, w_obj, offset)) @@ -582,8 +582,8 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, - do_cast=False, is_ref=True) + return interp_cppyy.wrap_cppinstance( + space, address, self.clsdecl, do_cast=False, is_ref=True) class StdStringConverter(InstanceConverter): @@ -606,7 +606,7 @@ assign = self.clsdecl.get_overload("__assign__") from pypy.module._cppyy import interp_cppyy assign.call( - interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False), [w_value]) + interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False), [w_value]) except Exception: InstanceConverter.to_memory(self, space, w_obj, w_value, offset) diff --git a/pypy/module/_cppyy/executor.py b/pypy/module/_cppyy/executor.py --- a/pypy/module/_cppyy/executor.py +++ b/pypy/module/_cppyy/executor.py @@ -159,7 +159,7 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_l(space, cppmethod, cppthis, num_args, args) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - pyres = interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + pyres = interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) return pyres def execute_libffi(self, space, cif_descr, funcaddr, buffer): @@ -167,7 +167,7 @@ result = rffi.ptradd(buffer, cif_descr.exchange_result) from pypy.module._cppyy import interp_cppyy ptr_result = rffi.cast(capi.C_OBJECT, rffi.cast(rffi.VOIDPP, result)[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) class InstancePtrPtrExecutor(InstancePtrExecutor): @@ -176,7 +176,7 @@ voidp_result = capi.c_call_r(space, cppmethod, cppthis, num_args, args) ref_address = rffi.cast(rffi.VOIDPP, voidp_result) ptr_result = rffi.cast(capi.C_OBJECT, ref_address[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible @@ -188,8 +188,8 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_o(space, cppmethod, cppthis, num_args, args, self.cppclass) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass, - do_cast=False, python_owns=True, fresh=True) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass, + do_cast=False, python_owns=True, fresh=True) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -2,7 +2,7 @@ from pypy.interpreter.error import OperationError, oefmt from pypy.interpreter.gateway import interp2app, unwrap_spec -from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty, interp_attrproperty_w +from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty from pypy.interpreter.baseobjspace import W_Root from rpython.rtyper.lltypesystem import rffi, lltype, llmemory @@ -33,16 +33,21 @@ class State(object): def __init__(self, space): + # final scoped name -> opaque handle self.cppscope_cache = { - "void" : W_CPPClassDecl(space, "void", capi.C_NULL_TYPE) } + 'void' : W_CPPClassDecl(space, capi.C_NULL_TYPE, 'void') } + # opaque handle -> app-level python class + self.cppclass_registry = {} + # app-level class generator callback + self.w_clgen_callback = None + # app-level function generator callback (currently not used) + self.w_fngen_callback = None + # C++11's nullptr self.w_nullptr = None - self.cpptemplate_cache = {} - self.cppclass_registry = {} - self.w_clgen_callback = None - self.w_fngen_callback = None def get_nullptr(space): - if hasattr(space, "fake"): + # construct a unique address that compares to NULL, serves as nullptr + if hasattr(space, 'fake'): raise NotImplementedError state = space.fromcache(State) if state.w_nullptr is None: @@ -58,52 +63,48 @@ state.w_nullptr = nullarr return state.w_nullptr - at unwrap_spec(name='text') -def resolve_name(space, name): - return space.newtext(capi.c_resolve_name(space, name)) + at unwrap_spec(scoped_name='text') +def resolve_name(space, scoped_name): + return space.newtext(capi.c_resolve_name(space, scoped_name)) - at unwrap_spec(name='text') -def scope_byname(space, name): - true_name = capi.c_resolve_name(space, name) +# memoized lookup of handles by final, scoped, name of classes/namespaces + at unwrap_spec(final_scoped_name='text') +def scope_byname(space, final_scoped_name): state = space.fromcache(State) try: - return state.cppscope_cache[true_name] + return state.cppscope_cache[final_scoped_name] except KeyError: pass - opaque_handle = capi.c_get_scope_opaque(space, true_name) + opaque_handle = capi.c_get_scope_opaque(space, final_scoped_name) assert lltype.typeOf(opaque_handle) == capi.C_SCOPE if opaque_handle: - final_name = capi.c_final_name(space, opaque_handle) - if capi.c_is_namespace(space, opaque_handle): - cppscope = W_CPPNamespaceDecl(space, final_name, opaque_handle) - elif capi.c_has_complex_hierarchy(space, opaque_handle): - cppscope = W_CPPComplexClassDecl(space, final_name, opaque_handle) + isns = capi.c_is_namespace(space, opaque_handle) + if isns: + cppscope = W_CPPNamespaceDecl(space, opaque_handle, final_scoped_name) else: - cppscope = W_CPPClassDecl(space, final_name, opaque_handle) - state.cppscope_cache[name] = cppscope + if capi.c_has_complex_hierarchy(space, opaque_handle): + cppscope = W_CPPComplexClassDecl(space, opaque_handle, final_scoped_name) + else: + cppscope = W_CPPClassDecl(space, opaque_handle, final_scoped_name) - cppscope._build_methods() - cppscope._find_datamembers() + # store in the cache to prevent recursion + state.cppscope_cache[final_scoped_name] = cppscope + + if not isns: + # build methods/data; TODO: also defer this for classes (a functional __dir__ + # and instrospection for help() is enough and allows more lazy loading) + cppscope._build_methods() + cppscope._find_datamembers() + return cppscope return None - at unwrap_spec(name='text') -def template_byname(space, name): - state = space.fromcache(State) - try: - return state.cpptemplate_cache[name] - except KeyError: - pass - - if capi.c_is_template(space, name): - cpptemplate = W_CPPTemplateType(space, name) - state.cpptemplate_cache[name] = cpptemplate - return cpptemplate - - return None + at unwrap_spec(final_scoped_name='text') +def is_template(space, final_scoped_name): + return space.wrap(capi.c_is_template(space, final_scoped_name)) def std_string_name(space): return space.newtext(capi.std_string_name) @@ -591,15 +592,17 @@ @jit.unroll_safe @unwrap_spec(args_w='args_w') def call(self, w_cppinstance, args_w): + # TODO: factor out the following: + if capi.c_is_abstract(self.space, self.scope.handle): + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.scope.name) w_result = W_CPPOverload.call(self, w_cppinstance, args_w) newthis = rffi.cast(capi.C_OBJECT, self.space.uint_w(w_result)) cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if cppinstance is not None: cppinstance._rawobject = newthis memory_regulator.register(cppinstance) - return w_cppinstance - return wrap_cppobject(self.space, newthis, self.functions[0].scope, - do_cast=False, python_owns=True, fresh=True) def __repr__(self): return "W_CPPConstructorOverload(%s)" % [f.signature() for f in self.functions] @@ -643,8 +646,8 @@ def _get_offset(self, cppinstance): if cppinstance: - assert lltype.typeOf(cppinstance.cppclass.handle) == lltype.typeOf(self.scope.handle) - offset = self.offset + cppinstance.cppclass.get_base_offset(cppinstance, self.scope) + assert lltype.typeOf(cppinstance.clsdecl.handle) == lltype.typeOf(self.scope.handle) + offset = self.offset + cppinstance.clsdecl.get_base_offset(cppinstance, self.scope) else: offset = self.offset return offset @@ -705,12 +708,12 @@ return space.w_False class W_CPPScopeDecl(W_Root): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] _immutable_fields_ = ['handle', 'name'] - def __init__(self, space, name, opaque_handle): + def __init__(self, space, opaque_handle, final_scoped_name): self.space = space - self.name = name + self.name = final_scoped_name assert lltype.typeOf(opaque_handle) == capi.C_SCOPE self.handle = opaque_handle self.methods = {} @@ -769,6 +772,9 @@ # classes for inheritance. Both are python classes, though, and refactoring # may be in order at some point. class W_CPPNamespaceDecl(W_CPPScopeDecl): + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name'] + def _make_cppfunction(self, pyname, index): num_args = capi.c_method_num_args(self.space, self, index) args_required = capi.c_method_req_args(self.space, self, index) @@ -779,9 +785,6 @@ arg_defs.append((arg_type, arg_dflt)) return CPPFunction(self.space, self, index, arg_defs, args_required) - def _build_methods(self): - pass # force lazy lookups in namespaces - def _make_datamember(self, dm_name, dm_idx): type_name = capi.c_datamember_type(self.space, self, dm_idx) offset = capi.c_datamember_offset(self.space, self, dm_idx) @@ -791,9 +794,6 @@ self.datamembers[dm_name] = datamember return datamember - def _find_datamembers(self): - pass # force lazy lookups in namespaces - def find_overload(self, meth_name): indices = capi.c_method_indices_from_name(self.space, self, meth_name) if not indices: @@ -855,18 +855,21 @@ class W_CPPClassDecl(W_CPPScopeDecl): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] - _immutable_fields_ = ['handle', 'constructor', 'methods[*]', 'datamembers[*]'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name', 'methods[*]', 'datamembers[*]'] def _build_methods(self): assert len(self.methods) == 0 methods_temp = {} for i in range(capi.c_num_methods(self.space, self)): idx = capi.c_method_index_at(self.space, self, i) - pyname = helper.map_operator_name(self.space, - capi.c_method_name(self.space, self, idx), - capi.c_method_num_args(self.space, self, idx), - capi.c_method_result_type(self.space, self, idx)) + if capi.c_is_constructor(self.space, self, idx): + pyname = '__init__' + else: + pyname = helper.map_operator_name(self.space, + capi.c_method_name(self.space, self, idx), + capi.c_method_num_args(self.space, self, idx), + capi.c_method_result_type(self.space, self, idx)) cppmethod = self._make_cppfunction(pyname, idx) methods_temp.setdefault(pyname, []).append(cppmethod) # the following covers the case where the only kind of operator[](idx) @@ -883,7 +886,7 @@ # create the overload methods from the method sets for pyname, methods in methods_temp.iteritems(): CPPMethodSort(methods).sort() - if pyname == self.name: + if pyname == '__init__': overload = W_CPPConstructorOverload(self.space, self, methods[:]) else: overload = W_CPPOverload(self.space, self, methods[:]) @@ -934,11 +937,11 @@ raise self.missing_attribute_error(name) def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return 0 def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return cppinstance.get_rawobject() def is_namespace(self): @@ -973,13 +976,13 @@ class W_CPPComplexClassDecl(W_CPPClassDecl): def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = capi.c_base_offset(self.space, self, calling_scope, cppinstance.get_rawobject(), 1) return offset def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = self.get_base_offset(cppinstance, calling_scope) return capi.direct_ptradd(cppinstance.get_rawobject(), offset) @@ -997,37 +1000,16 @@ W_CPPComplexClassDecl.typedef.acceptable_as_base_class = False -class W_CPPTemplateType(W_Root): - _attrs_ = ['space', 'name'] - _immutable_fields = ['name'] - - def __init__(self, space, name): - self.space = space - self.name = name - - @unwrap_spec(args_w='args_w') - def __call__(self, args_w): - # TODO: this is broken but unused (see pythonify.py) - fullname = "".join([self.name, '<', self.space.text_w(args_w[0]), '>']) - return scope_byname(self.space, fullname) - -W_CPPTemplateType.typedef = TypeDef( - 'CPPTemplateType', - __call__ = interp2app(W_CPPTemplateType.__call__), -) -W_CPPTemplateType.typedef.acceptable_as_base_class = False - - class W_CPPClass(W_Root): - _attrs_ = ['space', 'cppclass', '_rawobject', 'isref', 'python_owns', + _attrs_ = ['space', 'clsdecl', '_rawobject', 'isref', 'python_owns', 'finalizer_registered'] - _immutable_fields_ = ["cppclass", "isref"] + _immutable_fields_ = ['clsdecl', 'isref'] finalizer_registered = False - def __init__(self, space, cppclass, rawobject, isref, python_owns): + def __init__(self, space, decl, rawobject, isref, python_owns): self.space = space - self.cppclass = cppclass + self.clsdecl = decl assert lltype.typeOf(rawobject) == capi.C_OBJECT assert not isref or rawobject self._rawobject = rawobject @@ -1057,7 +1039,7 @@ self._opt_register_finalizer() def get_cppthis(self, calling_scope): - return self.cppclass.get_cppthis(self, calling_scope) + return self.clsdecl.get_cppthis(self, calling_scope) def get_rawobject(self): if not self.isref: @@ -1078,12 +1060,9 @@ return None def instance__init__(self, args_w): - if capi.c_is_abstract(self.space, self.cppclass.handle): - raise oefmt(self.space.w_TypeError, - "cannot instantiate abstract class '%s'", - self.cppclass.name) - constructor_overload = self.cppclass.get_overload(self.cppclass.name) - constructor_overload.call(self, args_w) + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.clsdecl.name) def instance__eq__(self, w_other): # special case: if other is None, compare pointer-style @@ -1099,7 +1078,7 @@ for name in ["", "__gnu_cxx", "__1"]: nss = scope_byname(self.space, name) meth_idx = capi.c_get_global_operator( - self.space, nss, self.cppclass, other.cppclass, "operator==") + self.space, nss, self.clsdecl, other.clsdecl, "operator==") if meth_idx != -1: f = nss._make_cppfunction("operator==", meth_idx) ol = W_CPPOverload(self.space, nss, [f]) @@ -1118,7 +1097,7 @@ # fallback 2: direct pointer comparison (the class comparison is needed since # the first data member in a struct and the struct have the same address) other = self.space.interp_w(W_CPPClass, w_other, can_be_None=False) # TODO: factor out - iseq = (self._rawobject == other._rawobject) and (self.cppclass == other.cppclass) + iseq = (self._rawobject == other._rawobject) and (self.clsdecl == other.clsdecl) return self.space.newbool(iseq) def instance__ne__(self, w_other): @@ -1134,26 +1113,26 @@ if w_as_builtin is not None: return self.space.len(w_as_builtin) raise oefmt(self.space.w_TypeError, - "'%s' has no length", self.cppclass.name) + "'%s' has no length", self.clsdecl.name) def instance__cmp__(self, w_other): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.cmp(w_as_builtin, w_other) raise oefmt(self.space.w_AttributeError, - "'%s' has no attribute __cmp__", self.cppclass.name) + "'%s' has no attribute __cmp__", self.clsdecl.name) def instance__repr__(self): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.repr(w_as_builtin) return self.space.newtext("<%s object at 0x%x>" % - (self.cppclass.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) + (self.clsdecl.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) def destruct(self): if self._rawobject and not self.isref: memory_regulator.unregister(self) - capi.c_destruct(self.space, self.cppclass, self._rawobject) + capi.c_destruct(self.space, self.clsdecl, self._rawobject) self._rawobject = capi.C_NULL_OBJECT def _finalize_(self): @@ -1162,7 +1141,6 @@ W_CPPClass.typedef = TypeDef( 'CPPClass', - cppclass = interp_attrproperty_w('cppclass', cls=W_CPPClass), _python_owns = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), __init__ = interp2app(W_CPPClass.instance__init__), __eq__ = interp2app(W_CPPClass.instance__eq__), @@ -1220,21 +1198,21 @@ state = space.fromcache(State) return space.call_function(state.w_fngen_callback, w_callable, space.newint(npar)) -def wrap_cppobject(space, rawobject, cppclass, - do_cast=True, python_owns=False, is_ref=False, fresh=False): +def wrap_cppinstance(space, rawobject, clsdecl, + do_cast=True, python_owns=False, is_ref=False, fresh=False): rawobject = rffi.cast(capi.C_OBJECT, rawobject) # cast to actual if requested and possible w_pycppclass = None if do_cast and rawobject: - actual = capi.c_actual_class(space, cppclass, rawobject) - if actual != cppclass.handle: + actual = capi.c_actual_class(space, clsdecl, rawobject) + if actual != clsdecl.handle: try: w_pycppclass = get_pythonized_cppclass(space, actual) - offset = capi.c_base_offset1(space, actual, cppclass, rawobject, -1) + offset = capi.c_base_offset1(space, actual, clsdecl, rawobject, -1) rawobject = capi.direct_ptradd(rawobject, offset) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) + w_cppdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + clsdecl = space.interp_w(W_CPPClassDecl, w_cppdecl, can_be_None=False) except Exception: # failed to locate/build the derived class, so stick to the base (note # that only get_pythonized_cppclass is expected to raise, so none of @@ -1242,18 +1220,18 @@ pass if w_pycppclass is None: - w_pycppclass = get_pythonized_cppclass(space, cppclass.handle) + w_pycppclass = get_pythonized_cppclass(space, clsdecl.handle) # try to recycle existing object if this one is not newly created if not fresh and rawobject: obj = memory_regulator.retrieve(rawobject) - if obj is not None and obj.cppclass is cppclass: + if obj is not None and obj.clsdecl is clsdecl: return obj # fresh creation w_cppinstance = space.allocate_instance(W_CPPClass, w_pycppclass) cppinstance = space.interp_w(W_CPPClass, w_cppinstance, can_be_None=False) - cppinstance.__init__(space, cppclass, rawobject, is_ref, python_owns) + cppinstance.__init__(space, clsdecl, rawobject, is_ref, python_owns) memory_regulator.register(cppinstance) return w_cppinstance @@ -1273,19 +1251,24 @@ return space.newlong(address) @unwrap_spec(owns=bool, cast=bool) -def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): - """Takes an address and a bound C++ class proxy, returns a bound instance.""" +def _bind_object(space, w_obj, w_clsdecl, owns=False, cast=False): try: # attempt address from array or C++ instance rawobject = rffi.cast(capi.C_OBJECT, _addressof(space, w_obj)) except Exception: # accept integer value as address rawobject = rffi.cast(capi.C_OBJECT, space.uint_w(w_obj)) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - if not w_cppclass: - w_cppclass = scope_byname(space, space.text_w(w_pycppclass)) - if not w_cppclass: + decl = space.interp_w(W_CPPClassDecl, w_clsdecl, can_be_None=False) + return wrap_cppinstance(space, rawobject, decl, python_owns=owns, do_cast=cast) + + at unwrap_spec(owns=bool, cast=bool) +def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): + """Takes an address and a bound C++ class proxy, returns a bound instance.""" + w_clsdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + if not w_clsdecl: + w_clsdecl = scope_byname(space, space.text_w(w_pycppclass)) + if not w_clsdecl: raise oefmt(space.w_TypeError, "no such class: %s", space.text_w(w_pycppclass)) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) - return wrap_cppobject(space, rawobject, cppclass, do_cast=cast, python_owns=owns) + return _bind_object(space, w_obj, w_clsdecl, owns, cast) + diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -10,7 +10,7 @@ class CPPMetaScope(type): def __getattr__(self, name): try: - return get_pycppitem(self, name) # will cache on self + return get_scoped_pycppitem(self, name) # will cache on self except Exception as e: raise AttributeError("%s object has no attribute '%s' (details: %s)" % (self, name, str(e))) @@ -36,11 +36,14 @@ self._scope = scope def _arg_to_str(self, arg): - if arg == str: - import _cppyy - arg = _cppyy._std_string_name() - elif type(arg) != str: - arg = arg.__name__ + try: + arg = arg.__cppname__ + except AttributeError: + if arg == str: + import _cppyy + arg = _cppyy._std_string_name() + elif type(arg) != str: + arg = arg.__name__ return arg def __call__(self, *args): @@ -58,8 +61,36 @@ return self.__call__(*args) -def clgen_callback(name): - return get_pycppclass(name) +def scope_splitter(name): + is_open_template, scope = 0, "" + for c in name: + if c == ':' and not is_open_template: + if scope: + yield scope + scope = "" + continue + elif c == '<': + is_open_template += 1 + elif c == '>': + is_open_template -= 1 + scope += c + yield scope + +def get_pycppitem(final_scoped_name): + # walk scopes recursively down from global namespace ("::") to get the + # actual (i.e. not typedef'ed) class, triggering all necessary creation + scope = gbl + for name in scope_splitter(final_scoped_name): + scope = getattr(scope, name) + return scope +get_pycppclass = get_pycppitem # currently no distinction, but might + # in future for performance + + +# callbacks (originating from interp_cppyy.py) to allow interp-level to +# initiate creation of app-level classes and function +def clgen_callback(final_scoped_name): + return get_pycppclass(final_scoped_name) def fngen_callback(func, npar): # todo, some kind of arg transform spec if npar == 0: @@ -75,6 +106,12 @@ return wrapper +# construction of namespaces and classes, and their helpers +def make_module_name(scope): + if scope: + return scope.__module__ + '.' + scope.__name__ + return 'cppyy' + def make_static_function(func_name, cppol): def function(*args): return cppol.call(None, *args) @@ -82,13 +119,6 @@ function.__doc__ = cppol.signature() return staticmethod(function) -def make_method(meth_name, cppol): - def method(self, *args): - return cppol.call(self, *args) - method.__name__ = meth_name - method.__doc__ = cppol.signature() - return method - def make_cppnamespace(scope, name, decl): # build up a representation of a C++ namespace (namespaces are classes) @@ -98,20 +128,19 @@ ns_meta = type(name+'_meta', (CPPMetaNamespace,), {}) # create the python-side C++ namespace representation, cache in scope if given - d = {"__cppdecl__" : decl, "__cppname__" : decl.__cppname__ } + d = {"__cppdecl__" : decl, + "__module__" : make_module_name(scope), + "__cppname__" : decl.__cppname__ } pyns = ns_meta(name, (CPPNamespace,), d) if scope: setattr(scope, name, pyns) # install as modules to allow importing from (note naming: cppyy) - modname = 'cppyy.gbl' - if scope: - modname = 'cppyy.gbl.'+pyns.__cppname__.replace('::', '.') - sys.modules[modname] = pyns + sys.modules[make_module_name(pyns)] = pyns return pyns def _drop_cycles(bases): - # TODO: figure this out, as it seems to be a PyPy bug?! + # TODO: figure out why this is necessary? for b1 in bases: for b2 in bases: if not (b1 is b2) and issubclass(b2, b1): @@ -119,27 +148,37 @@ break return tuple(bases) -def make_new(class_name): + +def make_new(decl): def __new__(cls, *args): # create a place-holder only as there may be a derived class defined + # TODO: get rid of the import and add user-land bind_object that uses + # _bind_object (see interp_cppyy.py) import _cppyy - instance = _cppyy.bind_object(0, class_name, True) + instance = _cppyy._bind_object(0, decl, True) if not instance.__class__ is cls: instance.__class__ = cls # happens for derived class return instance return __new__ -def make_cppclass(scope, class_name, final_class_name, decl): +def make_method(meth_name, cppol): + def method(self, *args): + return cppol.call(self, *args) + method.__name__ = meth_name + method.__doc__ = cppol.signature() + return method + +def make_cppclass(scope, cl_name, decl): # get a list of base classes for class creation bases = [get_pycppclass(base) for base in decl.get_base_names()] if not bases: bases = [CPPClass,] else: - # it's technically possible that the required class now has been built - # if one of the base classes uses it in e.g. a function interface + # it's possible that the required class now has been built if one of + # the base classes uses it in e.g. a function interface try: - return scope.__dict__[final_class_name] + return scope.__dict__[cl_name] except KeyError: pass @@ -147,39 +186,41 @@ d_meta = {} # prepare dictionary for python-side C++ class representation - def dispatch(self, name, signature): - cppol = decl.dispatch(name, signature) - return types.MethodType(make_method(name, cppol), self, type(self)) + def dispatch(self, m_name, signature): + cppol = decl.__dispatch__(m_name, signature) + return types.MethodType(make_method(m_name, cppol), self, type(self)) d_class = {"__cppdecl__" : decl, + "__new__" : make_new(decl), + "__module__" : make_module_name(scope), "__cppname__" : decl.__cppname__, - "__new__" : make_new(class_name), + "__dispatch__" : dispatch, } # insert (static) methods into the class dictionary - for name in decl.get_method_names(): - cppol = decl.get_overload(name) + for m_name in decl.get_method_names(): + cppol = decl.get_overload(m_name) if cppol.is_static(): - d_class[name] = make_static_function(name, cppol) + d_class[m_name] = make_static_function(m_name, cppol) else: - d_class[name] = make_method(name, cppol) + d_class[m_name] = make_method(m_name, cppol) # add all data members to the dictionary of the class to be created, and # static ones also to the metaclass (needed for property setters) - for name in decl.get_datamember_names(): - cppdm = decl.get_datamember(name) - d_class[name] = cppdm + for d_name in decl.get_datamember_names(): + cppdm = decl.get_datamember(d_name) + d_class[d_name] = cppdm if cppdm.is_static(): - d_meta[name] = cppdm + d_meta[d_name] = cppdm # create a metaclass to allow properties (for static data write access) metabases = [type(base) for base in bases] - metacpp = type(CPPMetaScope)(class_name+'_meta', _drop_cycles(metabases), d_meta) + metacpp = type(CPPMetaScope)(cl_name+'_meta', _drop_cycles(metabases), d_meta) # create the python-side C++ class - pycls = metacpp(class_name, _drop_cycles(bases), d_class) + pycls = metacpp(cl_name, _drop_cycles(bases), d_class) # store the class on its outer scope - setattr(scope, final_class_name, pycls) + setattr(scope, cl_name, pycls) # the call to register will add back-end specific pythonizations and thus # needs to run first, so that the generic pythonizations can use them @@ -192,32 +233,32 @@ return CPPTemplate(template_name, scope) -def get_pycppitem(scope, name): +def get_scoped_pycppitem(scope, name): import _cppyy - # resolve typedefs/aliases - full_name = (scope == gbl) and name or (scope.__name__+'::'+name) - true_name = _cppyy._resolve_name(full_name) - if true_name != full_name: - return get_pycppclass(true_name) + # resolve typedefs/aliases: these may cross namespaces, in which case + # the lookup must trigger the creation of all necessary scopes + scoped_name = (scope == gbl) and name or (scope.__cppname__+'::'+name) + final_scoped_name = _cppyy._resolve_name(scoped_name) + if final_scoped_name != scoped_name: + pycppitem = get_pycppitem(final_scoped_name) + # also store on the requested scope (effectively a typedef or pointer copy) + setattr(scope, name, pycppitem) + return pycppitem pycppitem = None - # classes - cppitem = _cppyy._scope_byname(true_name) + # scopes (classes and namespaces) + cppitem = _cppyy._scope_byname(final_scoped_name) if cppitem: - name = true_name - if scope != gbl: - name = true_name[len(scope.__cppname__)+2:] if cppitem.is_namespace(): pycppitem = make_cppnamespace(scope, name, cppitem) - setattr(scope, name, pycppitem) else: - pycppitem = make_cppclass(scope, name, true_name, cppitem) + pycppitem = make_cppclass(scope, name, cppitem) # templates if not cppitem: - cppitem = _cppyy._template_byname(true_name) + cppitem = _cppyy._is_template(final_scoped_name) if cppitem: pycppitem = make_cpptemplatetype(scope, name) setattr(scope, name, pycppitem) @@ -249,29 +290,6 @@ raise AttributeError("'%s' has no attribute '%s'" % (str(scope), name)) -def scope_splitter(name): - is_open_template, scope = 0, "" - for c in name: - if c == ':' and not is_open_template: - if scope: - yield scope - scope = "" - continue - elif c == '<': - is_open_template += 1 - elif c == '>': - is_open_template -= 1 - scope += c - yield scope - -def get_pycppclass(name): - # break up the name, to walk the scopes and get the class recursively - scope = gbl - for part in scope_splitter(name): - scope = getattr(scope, part) - return scope - - # pythonization by decoration (move to their own file?) def python_style_getitem(self, idx): # python-style indexing: check for size and allow indexing from the back @@ -346,8 +364,8 @@ # also the fallback on the indexed __getitem__, but that is slower) if not 'vector' in pyclass.__name__[:11] and \ ('begin' in pyclass.__dict__ and 'end' in pyclass.__dict__): - if _cppyy._scope_byname(pyclass.__name__+'::iterator') or \ - _cppyy._scope_byname(pyclass.__name__+'::const_iterator'): + if _cppyy._scope_byname(pyclass.__cppname__+'::iterator') or \ + _cppyy._scope_byname(pyclass.__cppname__+'::const_iterator'): def __iter__(self): i = self.begin() while i != self.end(): diff --git a/pypy/module/_cppyy/test/test_cppyy.py b/pypy/module/_cppyy/test/test_cppyy.py --- a/pypy/module/_cppyy/test/test_cppyy.py +++ b/pypy/module/_cppyy/test/test_cppyy.py @@ -31,12 +31,16 @@ spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools']) def setup_class(cls): - cls.w_lib, cls.w_example01, cls.w_payload = \ + cls.w_lib, cls.w_instantiate, cls.w_example01, cls.w_payload = \ cls.space.unpackiterable(cls.space.appexec([], """(): import _cppyy, ctypes lib = ctypes.CDLL(%r, ctypes.RTLD_GLOBAL) - return lib, _cppyy._scope_byname('example01'), _cppyy._scope_byname('payload')"""\ - % (test_dct, ))) + def cpp_instantiate(tt, *args): + inst = _cppyy._bind_object(0, tt, True) + tt.get_overload("__init__").call(inst, *args) + return inst + return lib, cpp_instantiate, _cppyy._scope_byname('example01'),\ + _cppyy._scope_byname('payload')""" % (test_dct, ))) def test01_static_int(self): """Test passing of an int, returning of an int, and overloading on a @@ -95,7 +99,7 @@ assert t.get_overload("getCount").call(None) == 0 - e1 = t.get_overload(t.__cppname__).call(None, 7) + e1 = self.instantiate(t, 7) assert t.get_overload("getCount").call(None) == 1 res = t.get_overload("addDataToInt").call(e1, 4) assert res == 11 @@ -105,8 +109,8 @@ assert t.get_overload("getCount").call(None) == 0 raises(ReferenceError, 't.get_overload("addDataToInt").call(e1, 4)') - e1 = t.get_overload(t.__cppname__).call(None, 7) - e2 = t.get_overload(t.__cppname__).call(None, 8) + e1 = self.instantiate(t, 7) + e2 = self.instantiate(t, 8) assert t.get_overload("getCount").call(None) == 2 e1.__destruct__() assert t.get_overload("getCount").call(None) == 1 @@ -128,7 +132,7 @@ assert t.get_overload("getCount").call(None) == 0 - e1 = t.get_overload(t.__cppname__).call(None, 7) + e1 = self.instantiate(t, 7) assert t.get_overload("getCount").call(None) == 1 res = t.get_overload("addDataToInt").call(e1, 4) assert res == 11 @@ -138,8 +142,8 @@ gc.collect() assert t.get_overload("getCount").call(None) == 0 - e1 = t.get_overload(t.__cppname__).call(None, 7) - e2 = t.get_overload(t.__cppname__).call(None, 8) + e1 = self.instantiate(t, 7) + e2 = self.instantiate(t, 8) assert t.get_overload("getCount").call(None) == 2 e1 = None gc.collect() @@ -159,7 +163,7 @@ assert t.get_overload("getCount").call(None) == 0 - e1 = t.get_overload(t.__cppname__).call(None, 7) + e1 = self.instantiate(t, 7) assert t.get_overload("getCount").call(None) == 1 assert e1._python_owns == True e1._python_owns = False @@ -178,12 +182,12 @@ t = self.example01 - e = t.get_overload(t.__cppname__).call(None, 13) + e = self.instantiate(t, 13) res = t.get_overload("addDataToDouble").call(e, 16) assert round(res-29, 8) == 0. e.__destruct__() - e = t.get_overload(t.__cppname__).call(None, -13) + e = self.instantiate(t, -13) res = t.get_overload("addDataToDouble").call(e, 16) assert round(res-3, 8) == 0. e.__destruct__() @@ -196,7 +200,7 @@ t = self.example01 - e = t.get_overload(t.__cppname__).call(None, 42) + e = self.instantiate(t, 42) res = t.get_overload("addDataToAtoi").call(e, "13") assert res == 55 res = t.get_overload("addToStringValue").call(e, "12") # TODO: this leaks @@ -213,12 +217,12 @@ t1 = self.example01 t2 = self.payload - pl = t2.get_overload(t2.__cppname__).call(None, 3.14) + pl = self.instantiate(t2, 3.14) assert round(t2.get_overload("getData").call(pl)-3.14, 8) == 0 t1.get_overload("staticSetPayload").call(None, pl, 41.) assert t2.get_overload("getData").call(pl) == 41. - e = t1.get_overload(t1.__cppname__).call(None, 50) + e = self.instantiate(t1, 50) t1.get_overload("setPayload").call(e, pl); assert round(t2.get_overload("getData").call(pl)-50., 8) == 0 @@ -233,12 +237,12 @@ t1 = self.example01 t2 = self.payload - pl1 = t2.get_overload(t2.__cppname__).call(None, 3.14) + pl1 = self.instantiate(t2, 3.14) assert round(t2.get_overload("getData").call(pl1)-3.14, 8) == 0 pl2 = t1.get_overload("staticCyclePayload").call(None, pl1, 38.) assert t2.get_overload("getData").call(pl2) == 38. - e = t1.get_overload(t1.__cppname__).call(None, 50) + e = self.instantiate(t1, 50) pl2 = t1.get_overload("cyclePayload").call(e, pl1); assert round(t2.get_overload("getData").call(pl2)-50., 8) == 0 diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py --- a/pypy/module/_cppyy/test/test_fragile.py +++ b/pypy/module/_cppyy/test/test_fragile.py @@ -189,7 +189,7 @@ o = fragile.O() # raises TypeError assert 0 except TypeError as e: - assert "cannot instantiate abstract class 'O'" in str(e) + assert "cannot instantiate abstract class 'fragile::O'" in str(e) def test10_dir(self): """Test __dir__ method""" @@ -239,7 +239,7 @@ assert _cppyy.gbl.fragile.nested1 is nested1 assert nested1.__name__ == 'nested1' assert nested1.__module__ == 'cppyy.gbl.fragile' - assert nested1.__cppname__ == 'nested1' + assert nested1.__cppname__ == 'fragile::nested1' from cppyy.gbl.fragile.nested1 import A, nested2 assert _cppyy.gbl.fragile.nested1.A is A @@ -254,17 +254,17 @@ from cppyy.gbl.fragile.nested1.nested2 import A, nested3 assert _cppyy.gbl.fragile.nested1.nested2.A is A assert A.__name__ == 'A' - assert A.__module__ == 'cppyy.gbl.fragile.nested1' + assert A.__module__ == 'cppyy.gbl.fragile.nested1.nested2' assert A.__cppname__ == 'fragile::nested1::nested2::A' assert _cppyy.gbl.fragile.nested1.nested2.nested3 is nested3 assert A.__name__ == 'A' - assert A.__module__ == 'cppyy.gbl.fragile.nested1' + assert A.__module__ == 'cppyy.gbl.fragile.nested1.nested2' assert nested3.__cppname__ == 'fragile::nested1::nested2::nested3' from cppyy.gbl.fragile.nested1.nested2.nested3 import A assert _cppyy.gbl.fragile.nested1.nested2.nested3.A is nested3.A assert A.__name__ == 'A' - assert A.__module__ == 'cppyy.gbl.fragile.nested1' + assert A.__module__ == 'cppyy.gbl.fragile.nested1.nested2.nested3' assert A.__cppname__ == 'fragile::nested1::nested2::nested3::A' def test12_missing_casts(self): diff --git a/pypy/module/_cppyy/test/test_zjit.py b/pypy/module/_cppyy/test/test_zjit.py --- a/pypy/module/_cppyy/test/test_zjit.py +++ b/pypy/module/_cppyy/test/test_zjit.py @@ -279,7 +279,8 @@ drv = jit.JitDriver(greens=[], reds=["i", "inst", "cppmethod"]) def f(): cls = interp_cppyy.scope_byname(space, "example01") - inst = cls.get_overload("example01").call(None, [FakeInt(0)]) + inst = interp_cppyy._bind_object(space, FakeInt(0), cls, True) + cls.get_overload("__init__").call(inst, [FakeInt(0)]) cppmethod = cls.get_overload(method_name) assert isinstance(inst, interp_cppyy.W_CPPClass) i = 10 From pypy.commits at gmail.com Wed Oct 11 19:49:00 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:49:00 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: fix translator error Message-ID: <59deadec.ccabdf0a.c440.0a43@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92723:e98bb35e4396 Date: 2017-10-11 12:53 -0700 http://bitbucket.org/pypy/pypy/changeset/e98bb35e4396/ Log: fix translator error diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -104,7 +104,7 @@ @unwrap_spec(final_scoped_name='text') def is_template(space, final_scoped_name): - return space.wrap(capi.c_is_template(space, final_scoped_name)) + return space.newbool(capi.c_is_template(space, final_scoped_name)) def std_string_name(space): return space.newtext(capi.std_string_name) From pypy.commits at gmail.com Wed Oct 11 19:49:04 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:49:04 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: basic std::move implementation Message-ID: <59deadf0.978adf0a.6c412.0130@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92725:1ab7076e5b66 Date: 2017-10-11 15:39 -0700 http://bitbucket.org/pypy/pypy/changeset/1ab7076e5b66/ Log: basic std::move implementation diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -17,6 +17,7 @@ 'addressof' : 'interp_cppyy.addressof', '_bind_object' : 'interp_cppyy._bind_object', 'bind_object' : 'interp_cppyy.bind_object', + 'move' : 'interp_cppyy.move', } appleveldefs = { diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -4,7 +4,7 @@ from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib.rarithmetic import r_singlefloat, r_longfloat -from rpython.rlib import rfloat +from rpython.rlib import rfloat, rawrefcount from pypy.module._rawffi.interp_rawffi import letter2tp from pypy.module._rawffi.array import W_Array, W_ArrayInstance @@ -495,6 +495,10 @@ def _unwrap_object(self, space, w_obj): from pypy.module._cppyy.interp_cppyy import W_CPPClass if isinstance(w_obj, W_CPPClass): + from pypy.module._cppyy.interp_cppyy import INSTANCE_FLAGS_IS_R_VALUE + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + # reject moves as all are explicit + raise ValueError("lvalue expected") if capi.c_is_subtype(space, w_obj.clsdecl, self.clsdecl): rawobject = w_obj.get_rawobject() offset = capi.c_base_offset(space, w_obj.clsdecl, self.clsdecl, rawobject, 1) @@ -518,6 +522,17 @@ x = rffi.cast(rffi.VOIDPP, address) x[0] = rffi.cast(rffi.VOIDP, self._unwrap_object(space, w_obj)) +class InstanceMoveConverter(InstanceRefConverter): + def _unwrap_object(self, space, w_obj): + # moving is same as by-ref, but have to check that move is allowed + from pypy.module._cppyy.interp_cppyy import W_CPPClass, INSTANCE_FLAGS_IS_R_VALUE + if isinstance(w_obj, W_CPPClass): + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + w_obj.flags &= ~INSTANCE_FLAGS_IS_R_VALUE + return InstanceRefConverter._unwrap_object(self, space, w_obj) + raise oefmt(space.w_ValueError, "object is not an rvalue") + + class InstanceConverter(InstanceRefConverter): def convert_argument_libffi(self, space, w_obj, address, call_local): @@ -719,6 +734,8 @@ return InstancePtrConverter(space, clsdecl) elif compound == "&": return InstanceRefConverter(space, clsdecl) + elif compound == "&&": + return InstanceMoveConverter(space, clsdecl) elif compound == "**": return InstancePtrPtrConverter(space, clsdecl) elif compound == "": diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -15,6 +15,10 @@ from pypy.module._cppyy import converter, executor, ffitypes, helper +INSTANCE_FLAGS_PYTHON_OWNS = 0x0001 +INSTANCE_FLAGS_IS_REF = 0x0002 +INSTANCE_FLAGS_IS_R_VALUE = 0x0004 + class FastCallNotPossible(Exception): pass @@ -1001,9 +1005,9 @@ class W_CPPClass(W_Root): - _attrs_ = ['space', 'clsdecl', '_rawobject', 'isref', 'python_owns', + _attrs_ = ['space', 'clsdecl', '_rawobject', 'flags', 'finalizer_registered'] - _immutable_fields_ = ['clsdecl', 'isref'] + _immutable_fields_ = ['clsdecl'] finalizer_registered = False @@ -1014,35 +1018,42 @@ assert not isref or rawobject self._rawobject = rawobject assert not isref or not python_owns - self.isref = isref - self.python_owns = python_owns - self._opt_register_finalizer() + self.flags = 0 + if isref: + self.flags |= INSTANCE_FLAGS_IS_REF + if python_owns: + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() def _opt_register_finalizer(self): - if self.python_owns and not self.finalizer_registered \ - and not hasattr(self.space, "fake"): + if not self.finalizer_registered and not hasattr(self.space, "fake"): + assert self.flags & INSTANCE_FLAGS_PYTHON_OWNS self.register_finalizer(self.space) self.finalizer_registered = True def _nullcheck(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): raise oefmt(self.space.w_ReferenceError, "trying to access a NULL pointer") # allow user to determine ownership rules on a per object level def fget_python_owns(self, space): - return space.newbool(self.python_owns) + return space.newbool(self.flags & INSTANCE_FLAGS_PYTHON_OWNS) @unwrap_spec(value=bool) def fset_python_owns(self, space, value): - self.python_owns = space.is_true(value) - self._opt_register_finalizer() + if space.is_true(value): + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() + else: + self.flags &= ~INSTANCE_FLAGS_PYTHON_OWNS def get_cppthis(self, calling_scope): return self.clsdecl.get_cppthis(self, calling_scope) def get_rawobject(self): - if not self.isref: + if not (self.flags & INSTANCE_FLAGS_IS_REF): return self._rawobject else: ptrptr = rffi.cast(rffi.VOIDPP, self._rawobject) @@ -1104,7 +1115,8 @@ return self.space.not_(self.instance__eq__(w_other)) def instance__nonzero__(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): return self.space.w_False return self.space.w_True @@ -1130,13 +1142,13 @@ (self.clsdecl.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) def destruct(self): - if self._rawobject and not self.isref: + if self._rawobject and not (self.flags & INSTANCE_FLAGS_IS_REF): memory_regulator.unregister(self) capi.c_destruct(self.space, self.clsdecl, self._rawobject) self._rawobject = capi.C_NULL_OBJECT def _finalize_(self): - if self.python_owns: + if self.flags & INSTANCE_FLAGS_PYTHON_OWNS: self.destruct() W_CPPClass.typedef = TypeDef( @@ -1272,3 +1284,9 @@ "no such class: %s", space.text_w(w_pycppclass)) return _bind_object(space, w_obj, w_clsdecl, owns, cast) +def move(space, w_obj): + """Casts the given instance into an C++-style rvalue.""" + obj = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + if obj: + obj.flags |= INSTANCE_FLAGS_IS_R_VALUE + return w_obj diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -434,6 +434,9 @@ # pre-create std to allow direct importing gbl.std = make_cppnamespace(gbl, 'std', _cppyy._scope_byname('std')) + # add move cast + gbl.std.move = _cppyy.move + # install a type for enums to refer to # TODO: this is correct for C++98, not for C++11 and in general there will # be the same issue for all typedef'd builtin types diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile --- a/pypy/module/_cppyy/test/Makefile +++ b/pypy/module/_cppyy/test/Makefile @@ -1,5 +1,6 @@ dicts = advancedcppDict.so \ advancedcpp2Dict.so \ + cpp11featuresDict.so \ crossingDict.so \ datatypesDict.so \ example01Dict.so \ diff --git a/pypy/module/_cppyy/test/cpp11features.cxx b/pypy/module/_cppyy/test/cpp11features.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.cxx @@ -0,0 +1,18 @@ +#if __cplusplus >= 201103L + +#include "cpp11features.h" + + +// for std::shared_ptr<> testing +int TestSharedPtr::s_counter = 0; + +std::shared_ptr create_shared_ptr_instance() { + return std::shared_ptr(new TestSharedPtr); +} + + +// for move ctors etc. +int TestMoving1::s_move_counter = 0; +int TestMoving2::s_move_counter = 0; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.h b/pypy/module/_cppyy/test/cpp11features.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.h @@ -0,0 +1,45 @@ +#if __cplusplus >= 201103L + +#include + + +//=========================================================================== +class TestSharedPtr { // for std::shared_ptr<> testing +public: + static int s_counter; + +public: + TestSharedPtr() { ++s_counter; } + TestSharedPtr(const TestSharedPtr&) { ++s_counter; } + ~TestSharedPtr() { --s_counter; } +}; + +std::shared_ptr create_shared_ptr_instance(); + + +//=========================================================================== +class TestMoving1 { // for move ctors etc. +public: + static int s_move_counter; + +public: + TestMoving1() {} + TestMoving1(TestMoving1&&) { ++s_move_counter; } + TestMoving1(const TestMoving1&) {} + TestMoving1& operator=(TestMoving1&&) { ++s_move_counter; return *this; } + TestMoving1& operator=(TestMoving1&) { return *this; } +}; + +class TestMoving2 { // note opposite method order from TestMoving1 +public: + static int s_move_counter; + +public: + TestMoving2() {} + TestMoving2(const TestMoving2&) {} + TestMoving2(TestMoving2&& other) { ++s_move_counter; } + TestMoving2& operator=(TestMoving2&) { return *this; } + TestMoving2& operator=(TestMoving2&&) { ++s_move_counter; return *this; } +}; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.xml b/pypy/module/_cppyy/test/cpp11features.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/test_cpp11features.py b/pypy/module/_cppyy/test/test_cpp11features.py new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/test_cpp11features.py @@ -0,0 +1,94 @@ +import py, os, sys +from .support import setup_make + + +currpath = py.path.local(__file__).dirpath() +test_dct = str(currpath.join("cpp11featuresDict.so")) + +def setup_module(mod): + setup_make("cpp11featuresDict.so") + +class AppTestCPP11FEATURES: + spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools']) + + def setup_class(cls): + cls.w_test_dct = cls.space.newtext(test_dct) + cls.w_example01 = cls.space.appexec([], """(): + import ctypes + return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, )) + + def test01_shared_ptr(self): + """Usage and access of std::shared_ptr<>""" + + import _cppyy + TestSharedPtr = _cppyy.gbl.TestSharedPtr + create_shared_ptr_instance = _cppyy.gbl.create_shared_ptr_instance + + # proper memory accounting + assert TestSharedPtr.s_counter == 0 + + ptr1 = create_shared_ptr_instance() + assert ptr1 + assert not not ptr1 + assert TestSharedPtr.s_counter == 1 + + ptr2 = create_shared_ptr_instance() + assert ptr2 + assert not not ptr2 + assert TestSharedPtr.s_counter == 2 + + del ptr2 + import gc; gc.collect() + assert TestSharedPtr.s_counter == 1 + + del ptr1 + gc.collect() + assert TestSharedPtr.s_counter == 0 + + def test02_nullptr(self): + """Allow the programmer to pass NULL in certain cases""" + + import _cppyy + + # test existence + nullptr = _cppyy.nullptr + assert not hasattr(_cppyy.gbl, 'nullptr') + + # usage is tested in datatypes.py:test15_nullptr_passing + + def test03_move(self): + """Move construction, assignment, and methods""" + + import _cppyy + + def moveit(T): + std = _cppyy.gbl.std + + # move constructor + i1 = T() + assert T.s_move_counter == 0 + + i2 = T(i1) # cctor + assert T.s_move_counter == 0 + + i3 = T(std.move(T())) # Note: in CPython can check for + # ref-count == 1, so no move() needed + assert T.s_move_counter == 1 + + i4 = T(std.move(i1)) + assert T.s_move_counter == 2 + + # move assignment + i4.__assign__(i2) + assert T.s_move_counter == 2 + + i4.__assign__(std.move(T())) # same note as above move ctor + assert T.s_move_counter == 3 + + i4.__assign__(std.move(i2)) + assert T.s_move_counter == 4 + + # order of moving and normal functions are reversed in 1, 2, for + # overload resolution testing + moveit(_cppyy.gbl.TestMoving1) + moveit(_cppyy.gbl.TestMoving2) From pypy.commits at gmail.com Wed Oct 11 19:49:02 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:49:02 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: place C++11's nullptr in _cppyy rather then gbl Message-ID: <59deadee.56acdf0a.95e3.9373@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92724:2fe6d8f50de7 Date: 2017-10-11 13:06 -0700 http://bitbucket.org/pypy/pypy/changeset/2fe6d8f50de7/ Log: place C++11's nullptr in _cppyy rather then gbl diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -439,12 +439,13 @@ # be the same issue for all typedef'd builtin types setattr(gbl, 'internal_enum_type_t', int) - # install nullptr as a unique reference - setattr(gbl, 'nullptr', _cppyy._get_nullptr()) - # install for user access _cppyy.gbl = gbl + # install nullptr as a unique reference + _cppyy.nullptr = _cppyy._get_nullptr() + + # user-defined pythonizations interface _pythonizations = {} def add_pythonization(class_name, callback): diff --git a/pypy/module/_cppyy/test/test_datatypes.py b/pypy/module/_cppyy/test/test_datatypes.py --- a/pypy/module/_cppyy/test/test_datatypes.py +++ b/pypy/module/_cppyy/test/test_datatypes.py @@ -226,8 +226,8 @@ raises(Exception, c.pass_array(0).__getitem__, 0) # raises SegfaultException assert not c.pass_array(None) raises(Exception, c.pass_array(None).__getitem__, 0) # id. - assert not c.pass_array(_cppyy.gbl.nullptr) - raises(Exception, c.pass_array(_cppyy.gbl.nullptr).__getitem__, 0) # id. id. + assert not c.pass_array(_cppyy.nullptr) + raises(Exception, c.pass_array(_cppyy.nullptr).__getitem__, 0) # id. id. c.__destruct__() @@ -694,22 +694,22 @@ c = CppyyTestData() - assert not _cppyy.gbl.nullptr + assert not _cppyy.nullptr - assert c.s_voidp is _cppyy.gbl.nullptr - assert CppyyTestData.s_voidp is _cppyy.gbl.nullptr + assert c.s_voidp is _cppyy.nullptr + assert CppyyTestData.s_voidp is _cppyy.nullptr - assert c.m_voidp is _cppyy.gbl.nullptr - assert c.get_voidp() is _cppyy.gbl.nullptr + assert c.m_voidp is _cppyy.nullptr + assert c.get_voidp() is _cppyy.nullptr c2 = CppyyTestData() - assert c2.m_voidp is _cppyy.gbl.nullptr + assert c2.m_voidp is _cppyy.nullptr c.set_voidp(c2.m_voidp) - assert c.m_voidp is _cppyy.gbl.nullptr + assert c.m_voidp is _cppyy.nullptr c.set_voidp(c2.get_voidp()) - assert c.m_voidp is _cppyy.gbl.nullptr - c.set_voidp(_cppyy.gbl.nullptr) - assert c.m_voidp is _cppyy.gbl.nullptr + assert c.m_voidp is _cppyy.nullptr + c.set_voidp(_cppyy.nullptr) + assert c.m_voidp is _cppyy.nullptr c.set_voidp(c2) def address_equality_test(a, b): @@ -724,8 +724,8 @@ def null_test(null): c.m_voidp = null - assert c.m_voidp is _cppyy.gbl.nullptr - map(null_test, [0, None, _cppyy.gbl.nullptr]) + assert c.m_voidp is _cppyy.nullptr + map(null_test, [0, None, _cppyy.nullptr]) c.m_voidp = c2 address_equality_test(c.m_voidp, c2) diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py --- a/pypy/module/_cppyy/test/test_fragile.py +++ b/pypy/module/_cppyy/test/test_fragile.py @@ -93,7 +93,7 @@ # 0, None, and nullptr allowed assert _cppyy.addressof(0) == 0 assert _cppyy.addressof(None) == 0 - assert _cppyy.addressof(_cppyy.gbl.nullptr) == 0 + assert _cppyy.addressof(_cppyy.nullptr) == 0 def test05_wrong_this(self): """Test that using an incorrect self argument raises""" From pypy.commits at gmail.com Wed Oct 11 19:49:07 2017 From: pypy.commits at gmail.com (wlav) Date: Wed, 11 Oct 2017 16:49:07 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: translation fix Message-ID: <59deadf3.8cc11c0a.d2b9f.8a7d@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92726:bda46aec4528 Date: 2017-10-11 16:11 -0700 http://bitbucket.org/pypy/pypy/changeset/bda46aec4528/ Log: translation fix diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -1039,7 +1039,7 @@ # allow user to determine ownership rules on a per object level def fget_python_owns(self, space): - return space.newbool(self.flags & INSTANCE_FLAGS_PYTHON_OWNS) + return space.newbool(bool(self.flags & INSTANCE_FLAGS_PYTHON_OWNS)) @unwrap_spec(value=bool) def fset_python_owns(self, space, value): From pypy.commits at gmail.com Wed Oct 11 20:50:20 2017 From: pypy.commits at gmail.com (mjacob) Date: Wed, 11 Oct 2017 17:50:20 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Update test_whatsnew for renamed py3k -> py3.5. Message-ID: <59debc4c.ddafdf0a.26f91.1b22@mx.google.com> Author: Manuel Jacob Branch: py3.5 Changeset: r92727:8966e708ff7c Date: 2017-10-12 02:29 +0200 http://bitbucket.org/pypy/pypy/changeset/8966e708ff7c/ Log: Update test_whatsnew for renamed py3k -> py3.5. diff --git a/pypy/doc/test/test_whatsnew.py b/pypy/doc/test/test_whatsnew.py --- a/pypy/doc/test/test_whatsnew.py +++ b/pypy/doc/test/test_whatsnew.py @@ -89,7 +89,7 @@ startrev, documented = parse_doc(last_whatsnew) merged, branch = get_merged_branches(ROOT, startrev, '') merged.discard('default') - merged.discard('py3k') + merged.discard('py3.5') merged.discard('') not_documented = merged.difference(documented) not_merged = documented.difference(merged) @@ -100,7 +100,7 @@ print '\n'.join(not_merged) print assert not not_documented - if branch == 'py3k': + if branch == 'py3.5': assert not not_merged else: assert branch in documented, 'Please document this branch before merging: %s' % branch From pypy.commits at gmail.com Wed Oct 11 20:50:22 2017 From: pypy.commits at gmail.com (mjacob) Date: Wed, 11 Oct 2017 17:50:22 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Seems like I forgot to remove this when merging default. Message-ID: <59debc4e.d185df0a.9fd68.92d3@mx.google.com> Author: Manuel Jacob Branch: py3.5 Changeset: r92728:d21f1c2cb4f9 Date: 2017-10-12 02:49 +0200 http://bitbucket.org/pypy/pypy/changeset/d21f1c2cb4f9/ Log: Seems like I forgot to remove this when merging default. diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -4,8 +4,3 @@ .. this is the revision after release-pypy3.5-5.9 .. startrev: be41e3ac0a29 - -.. branch: multiphase - -Implement PyType_FromSpec (PEP 384) and fix issues with PEP 489 support. - From pypy.commits at gmail.com Thu Oct 12 06:26:12 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 12 Oct 2017 03:26:12 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: WIP: copy the tupleobject freelist logic from CPython, and whack around until we can use it from cpyext/tupleobject.py. The first test passes, other are broken Message-ID: <59df4344.0a181c0a.d08d8.cea0@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92729:ccf12107e805 Date: 2017-10-12 10:09 +0200 http://bitbucket.org/pypy/pypy/changeset/ccf12107e805/ Log: WIP: copy the tupleobject freelist logic from CPython, and whack around until we can use it from cpyext/tupleobject.py. The first test passes, other are broken diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -633,8 +633,10 @@ '_Py_QnewFlag', 'Py_Py3kWarningFlag', 'Py_HashRandomizationFlag', '_Py_PackageContext', '_PyTraceMalloc_Track', '_PyTraceMalloc_Untrack', 'PyMem_Malloc', 'Py_IncRef', 'Py_DecRef', 'PyObject_Free', 'PyObject_GC_Del', 'PyType_GenericAlloc', - '_PyObject_New', '_PyObject_NewVar', '_PyObject_GC_New', + '_PyObject_New', '_PyObject_NewVar', + '_PyObject_GC_New', '_PyObject_GC_NewVar', 'PyObject_Init', 'PyObject_InitVar', 'PyInt_FromLong', + 'PyTuple_New', ] TYPES = {} FORWARD_DECLS = [] @@ -1135,12 +1137,20 @@ [rffi.LONG], PyObject, compilation_info=eci, _nowrapper=True) + state.C._PyPy_int_dealloc = rffi.llexternal( + '_PyPy_int_dealloc', [PyObject], lltype.Void, + compilation_info=eci, _nowrapper=True) + state.C.PyTuple_New = rffi.llexternal( + mangle_name(prefix, 'PyTuple_New'), + [Py_ssize_t], PyObject, + compilation_info=eci, + _nowrapper=True) + state.C._PyPy_tuple_dealloc = rffi.llexternal( + '_PyPy_tuple_dealloc', [PyObject], lltype.Void, + compilation_info=eci, _nowrapper=True) _, state.C.set_marker = rffi.CExternVariable( Py_ssize_t, '_pypy_rawrefcount_w_marker_deallocating', eci, _nowrapper=True, c_type='Py_ssize_t') - state.C._PyPy_int_dealloc = rffi.llexternal( - '_PyPy_int_dealloc', [PyObject], lltype.Void, - compilation_info=eci, _nowrapper=True) state.C._PyPy_subtype_dealloc = rffi.llexternal( '_PyPy_subtype_dealloc', [PyObject], lltype.Void, compilation_info=eci, _nowrapper=True) @@ -1258,7 +1268,6 @@ in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, mname) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) builder.prepare(py_obj, w_obj) - builder.attach_all(space) pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') @@ -1269,6 +1278,12 @@ ll2ctypes.lltype2ctypes(func.get_llhelper(space)), ctypes.c_void_p) + # we need to call this *after* the init code above, because it might + # indirectly call some functions which are attached to pypyAPI (e.g., we + # if do tuple_attach of the prebuilt empty tuple, we need to call + # _PyPy_Malloc) + builder.attach_all(space) + setup_init_functions(eci, prefix) return modulename.new(ext='') @@ -1464,6 +1479,7 @@ source_dir / "object.c", source_dir / "typeobject.c", source_dir / "intobject.c", + source_dir / "tupleobject.c", ] def build_eci(code, use_micronumpy=False, translating=False): diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -290,6 +290,9 @@ #define PyObject_GC_New(type, typeobj) \ ( (type *) _PyObject_GC_New(typeobj) ) +#define PyObject_GC_NewVar(type, typeobj, n) \ + ( (type *) _PyObject_GC_NewVar((typeobj), (n)) ) + /* A dummy PyGC_Head, just to please some tests. Don't use it! */ typedef union _gc_head { char dummy; @@ -345,6 +348,7 @@ PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *); PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t); PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *); +PyAPI_FUNC(PyVarObject *) _PyObject_GC_NewVar(PyTypeObject *, Py_ssize_t); PyAPI_FUNC(PyObject *) PyObject_Init(PyObject *, PyTypeObject *); PyAPI_FUNC(PyVarObject *) PyObject_InitVar(PyVarObject *, diff --git a/pypy/module/cpyext/include/tupleobject.h b/pypy/module/cpyext/include/tupleobject.h --- a/pypy/module/cpyext/include/tupleobject.h +++ b/pypy/module/cpyext/include/tupleobject.h @@ -16,9 +16,13 @@ */ } PyTupleObject; +PyAPI_FUNC(PyObject *) PyTuple_New(Py_ssize_t size); +PyAPI_FUNC(void) _PyPy_tuple_dealloc(PyObject *); + /* defined in varargswrapper.c */ PyAPI_FUNC(PyObject *) PyTuple_Pack(Py_ssize_t, ...); + /* Macro, trading safety for speed */ #define PyTuple_GET_ITEM(op, i) (((PyTupleObject *)(op))->ob_item[i]) #define PyTuple_GET_SIZE(op) Py_SIZE(op) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -68,6 +68,10 @@ return _PyObject_New(type); } +PyVarObject * _PyObject_GC_NewVar(PyTypeObject *type, Py_ssize_t nitems) +{ + return _PyObject_NewVar(type, nitems); +} static PyObject * _generic_alloc(PyTypeObject *type, Py_ssize_t nitems) diff --git a/pypy/module/cpyext/src/tupleobject.c b/pypy/module/cpyext/src/tupleobject.c new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/src/tupleobject.c @@ -0,0 +1,98 @@ + +/* Tuple object implementation, stolen&adapted from CPython */ + +#include "Python.h" + +/* Speed optimization to avoid frequent malloc/free of small tuples */ +#ifndef PyTuple_MAXSAVESIZE +#define PyTuple_MAXSAVESIZE 20 /* Largest tuple to save on free list */ +#endif +#ifndef PyTuple_MAXFREELIST +#define PyTuple_MAXFREELIST 2000 /* Maximum number of tuples of each size to save */ +#endif + +#if PyTuple_MAXSAVESIZE > 0 +/* Entries 1 up to PyTuple_MAXSAVESIZE are free lists, entry 0 is the empty + tuple () of which at most one instance will be allocated. +*/ +static PyTupleObject *free_list[PyTuple_MAXSAVESIZE]; +static int numfree[PyTuple_MAXSAVESIZE]; +#endif + +PyObject * +PyTuple_New(register Py_ssize_t size) +{ + register PyTupleObject *op; + Py_ssize_t i; + if (size < 0) { + PyErr_BadInternalCall(); + return NULL; + } +#if PyTuple_MAXSAVESIZE > 0 + if (size == 0 && free_list[0]) { + op = free_list[0]; + Py_INCREF(op); + return (PyObject *) op; + } + if (size < PyTuple_MAXSAVESIZE && (op = free_list[size]) != NULL) { + free_list[size] = (PyTupleObject *) op->ob_item[0]; + numfree[size]--; + _Py_NewReference((PyObject *)op); + } + else +#endif + { + Py_ssize_t nbytes = size * sizeof(PyObject *); + /* Check for overflow */ + if (nbytes / sizeof(PyObject *) != (size_t)size || + (nbytes > PY_SSIZE_T_MAX - sizeof(PyTupleObject) - sizeof(PyObject *))) + { + return PyErr_NoMemory(); + } + + op = PyObject_GC_NewVar(PyTupleObject, &PyTuple_Type, size); + if (op == NULL) + return NULL; + } + for (i=0; i < size; i++) + op->ob_item[i] = NULL; +#if PyTuple_MAXSAVESIZE > 0 + if (size == 0) { + free_list[0] = op; + ++numfree[0]; + Py_INCREF(op); /* extra INCREF so that this is never freed */ + } +#endif + _PyObject_GC_TRACK(op); + return (PyObject *) op; +} + +/* this is CPython's tupledealloc */ +void +_PyPy_tuple_dealloc(register PyObject *_op) +{ + register PyTupleObject *op = (PyTupleObject *)_op; + register Py_ssize_t i; + register Py_ssize_t len = Py_SIZE(op); + PyObject_GC_UnTrack(op); + Py_TRASHCAN_SAFE_BEGIN(op) + if (len > 0) { + i = len; + while (--i >= 0) + Py_XDECREF(op->ob_item[i]); +#if PyTuple_MAXSAVESIZE > 0 + if (len < PyTuple_MAXSAVESIZE && + numfree[len] < PyTuple_MAXFREELIST && + Py_TYPE(op) == &PyTuple_Type) + { + op->ob_item[0] = (PyObject *) free_list[len]; + numfree[len]++; + free_list[len] = op; + goto done; /* return */ + } +#endif + } + Py_TYPE(op)->tp_free((PyObject *)op); +done: + Py_TRASHCAN_SAFE_END(op) +} diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -7,6 +7,7 @@ from pypy.module.cpyext.pyobject import ( PyObject, PyObjectP, make_ref, from_ref, decref, incref, track_reference, make_typedescr, get_typedescr, pyobj_has_w_obj) +from pypy.module.cpyext.state import State from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.objspace.std.tupleobject import W_TupleObject @@ -36,10 +37,12 @@ @bootstrap_function def init_tupleobject(space): "Type description of PyTupleObject" + state = space.fromcache(State) make_typedescr(space.w_tuple.layout.typedef, basestruct=PyTupleObject.TO, attach=tuple_attach, - dealloc=tuple_dealloc, + alloc=tuple_alloc, + dealloc=state.C._PyPy_tuple_dealloc, realize=tuple_realize) PyTuple_Check, PyTuple_CheckExact = build_type_checkers_flags("Tuple") @@ -49,12 +52,20 @@ return (w_type is space.w_tuple or space.issubtype_w(w_type, space.w_tuple)) +def tuple_alloc(typedescr, space, w_type, itemcount): + state = space.fromcache(State) + if w_type is space.w_tuple: + return state.C.PyTuple_New(itemcount) + else: + return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) + def new_empty_tuple(space, length): """ Allocate a PyTupleObject and its array of PyObject *, but without a corresponding interpreter object. The array may be mutated, until tuple_realize() is called. Refcount of the result is 1. """ + assert False, 'kill this?' typedescr = get_typedescr(space.w_tuple.layout.typedef) py_obj = typedescr.allocate(space, space.w_tuple, length) py_tup = rffi.cast(PyTupleObject, py_obj) @@ -113,22 +124,22 @@ track_reference(space, py_obj, w_obj) return w_obj - at slot_function([PyObject], lltype.Void) -def tuple_dealloc(space, py_obj): - """Frees allocated PyTupleObject resources. - """ - py_tup = rffi.cast(PyTupleObject, py_obj) - p = py_tup.c_ob_item - for i in range(py_tup.c_ob_size): - decref(space, p[i]) - from pypy.module.cpyext.object import _dealloc - _dealloc(space, py_obj) +## @slot_function([PyObject], lltype.Void) +## def tuple_dealloc(space, py_obj): +## """Frees allocated PyTupleObject resources. +## """ +## py_tup = rffi.cast(PyTupleObject, py_obj) +## p = py_tup.c_ob_item +## for i in range(py_tup.c_ob_size): +## decref(space, p[i]) +## from pypy.module.cpyext.object import _dealloc +## _dealloc(space, py_obj) #_______________________________________________________________________ - at cpython_api([Py_ssize_t], PyObject, result_is_ll=True) -def PyTuple_New(space, size): - return new_empty_tuple(space, size) +## @cpython_api([Py_ssize_t], PyObject, result_is_ll=True) +## def PyTuple_New(space, size): +## return new_empty_tuple(space, size) @cpython_api([PyObject, Py_ssize_t, PyObject], rffi.INT_real, error=-1) def PyTuple_SetItem(space, ref, index, py_obj): From pypy.commits at gmail.com Thu Oct 12 06:26:14 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 12 Oct 2017 03:26:14 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: add a proper repr for this class, else its repr() is StdObjSpace, which is veeeery confusing. Hoorray old style classes Message-ID: <59df4346.521a1c0a.801de.7578@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92730:a79aad3bfd78 Date: 2017-10-12 10:20 +0200 http://bitbucket.org/pypy/pypy/changeset/a79aad3bfd78/ Log: add a proper repr for this class, else its repr() is StdObjSpace, which is veeeery confusing. Hoorray old style classes diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py --- a/pypy/module/cpyext/test/test_api.py +++ b/pypy/module/cpyext/test/test_api.py @@ -25,6 +25,10 @@ cls.preload_builtins(space) class CAPI: + def __repr__(self): + return '<%s.%s instance>' % (self.__class__.__module__, + self.__class__.__name__) + def __getattr__(self, name): return getattr(cls.space, name) cls.api = CAPI() From pypy.commits at gmail.com Thu Oct 12 06:26:16 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 12 Oct 2017 03:26:16 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: call PyTuple_New from here Message-ID: <59df4348.c384df0a.2c9fd.28c2@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92731:aab2478d4f85 Date: 2017-10-12 10:29 +0200 http://bitbucket.org/pypy/pypy/changeset/aab2478d4f85/ Log: call PyTuple_New from here diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -7,7 +7,7 @@ from rpython.rlib.debug import FatalError from pypy.module.cpyext.tupleobject import ( PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size) - +from pypy.module.cpyext.state import State class TestTupleObject(BaseApiTest): @@ -22,17 +22,19 @@ PyTuple_Size(space, space.newlist([])) def test_tuple_realize_refuses_nulls(self, space, api): - py_tuple = api.PyTuple_New(1) + state = space.fromcache(State) + py_tuple = state.C.PyTuple_New(1) py.test.raises(FatalError, from_ref, space, py_tuple) decref(space, py_tuple) def test_tuple_resize(self, space, api): + state = space.fromcache(State) w_42 = space.wrap(42) w_43 = space.wrap(43) w_44 = space.wrap(44) ar = lltype.malloc(PyObjectP.TO, 1, flavor='raw') - py_tuple = api.PyTuple_New(3) + py_tuple = state.C.PyTuple_New(3) # inside py_tuple is an array of "PyObject *" items which each hold # a reference rffi.cast(PyTupleObject, py_tuple).c_ob_item[0] = make_ref(space, w_42) @@ -45,7 +47,7 @@ assert space.int_w(space.getitem(w_tuple, space.wrap(1))) == 43 decref(space, ar[0]) - py_tuple = api.PyTuple_New(3) + py_tuple = state.C.PyTuple_New(3) rffi.cast(PyTupleObject, py_tuple).c_ob_item[0] = make_ref(space, w_42) rffi.cast(PyTupleObject, py_tuple).c_ob_item[1] = make_ref(space, w_43) rffi.cast(PyTupleObject, py_tuple).c_ob_item[2] = make_ref(space, w_44) @@ -64,7 +66,8 @@ lltype.free(ar, flavor='raw') def test_setitem(self, space, api): - py_tuple = api.PyTuple_New(2) + state = space.fromcache(State) + py_tuple = state.C.PyTuple_New(2) api.PyTuple_SetItem(py_tuple, 0, make_ref(space, space.wrap(42))) api.PyTuple_SetItem(py_tuple, 1, make_ref(space, space.wrap(43))) From pypy.commits at gmail.com Thu Oct 12 06:26:18 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 12 Oct 2017 03:26:18 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads" Message-ID: <59df434a.84ad1c0a.f88b8.4720@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92732:226ccce93e52 Date: 2017-10-12 10:30 +0200 http://bitbucket.org/pypy/pypy/changeset/226ccce93e52/ Log: merge heads" diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py --- a/pypy/module/cpyext/test/test_api.py +++ b/pypy/module/cpyext/test/test_api.py @@ -25,6 +25,10 @@ cls.preload_builtins(space) class CAPI: + def __repr__(self): + return '<%s.%s instance>' % (self.__class__.__module__, + self.__class__.__name__) + def __getattr__(self, name): return getattr(cls.space, name) cls.api = CAPI() From pypy.commits at gmail.com Thu Oct 12 06:28:08 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 12 Oct 2017 03:28:08 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: don't forget this Message-ID: <59df43b8.e291df0a.67c4.b69a@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5840:af8c66a99b6f Date: 2017-10-12 12:27 +0200 http://bitbucket.org/pypy/extradoc/changeset/af8c66a99b6f/ Log: don't forget this diff --git a/blog/draft/2017-10-cape-town-report.rst b/blog/draft/2017-10-cape-town-report.rst --- a/blog/draft/2017-10-cape-town-report.rst +++ b/blog/draft/2017-10-cape-town-report.rst @@ -7,6 +7,8 @@ Ronan, Antonio and sometimes Maciek had a very nice and productive sprint in Cape Town, as pictures show :). +XXX thank kiwi https://www.kiwi.com/sourcelift/ + XXX insert pic We spent most of the time hacking at cpyext, our CPython C-API compatibility From pypy.commits at gmail.com Thu Oct 12 07:10:23 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 04:10:23 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: more to-do items Message-ID: <59df4d9f.41a1df0a.d8081.f9fc@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r5841:d93bb9e6a71f Date: 2017-10-12 13:10 +0200 http://bitbucket.org/pypy/extradoc/changeset/d93bb9e6a71f/ Log: more to-do items diff --git a/planning/cpyext.txt b/planning/cpyext.txt --- a/planning/cpyext.txt +++ b/planning/cpyext.txt @@ -16,4 +16,11 @@ hypotetical W_CPyObject) which contains a PyObject* field, so that doing the w_* -> py_* lookup is just a getfield (there is alreadt support for this in rawrefcount.create_link_pyobj) - + +- make sure that we run all tests with -A, too + +- make PyTypeObject be a pointer, like all other PyXxxObject + +- as_pyobj() should be a context manager which calls + keepalive_until_here() + From pypy.commits at gmail.com Thu Oct 12 07:26:58 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 04:26:58 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Re-enable this, seems to work fine Message-ID: <59df5182.298fdf0a.664c4.e44d@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92733:e3cdc0f10924 Date: 2017-10-12 13:26 +0200 http://bitbucket.org/pypy/pypy/changeset/e3cdc0f10924/ Log: Re-enable this, seems to work fine diff --git a/pypy/objspace/std/dictmultiobject.py b/pypy/objspace/std/dictmultiobject.py --- a/pypy/objspace/std/dictmultiobject.py +++ b/pypy/objspace/std/dictmultiobject.py @@ -640,9 +640,9 @@ if type(w_key) is self.space.StringObjectCls: self.switch_to_bytes_strategy(w_dict) return - #elif type(w_key) is self.space.UnicodeObjectCls: - # self.switch_to_unicode_strategy(w_dict) - # return + elif type(w_key) is self.space.UnicodeObjectCls: + self.switch_to_unicode_strategy(w_dict) + return w_type = self.space.type(w_key) if self.space.is_w(w_type, self.space.w_int): self.switch_to_int_strategy(w_dict) @@ -658,7 +658,6 @@ w_dict.dstorage = storage def switch_to_unicode_strategy(self, w_dict): - xxx strategy = self.space.fromcache(UnicodeDictStrategy) storage = strategy.get_empty_storage() w_dict.set_strategy(strategy) From pypy.commits at gmail.com Thu Oct 12 08:27:06 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 05:27:06 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Random in-progress clean-up Message-ID: <59df5f9a.512f1c0a.3fda.07eb@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92734:13a4b012e64e Date: 2017-10-12 14:26 +0200 http://bitbucket.org/pypy/pypy/changeset/13a4b012e64e/ Log: Random in-progress clean-up diff --git a/pypy/interpreter/test/test_unicodehelper.py b/pypy/interpreter/test/test_unicodehelper.py --- a/pypy/interpreter/test/test_unicodehelper.py +++ b/pypy/interpreter/test/test_unicodehelper.py @@ -1,4 +1,7 @@ +import py from pypy.interpreter.unicodehelper import encode_utf8, decode_utf8 +from pypy.interpreter.unicodehelper import utf8_encode_ascii, str_decode_ascii + class FakeSpace: pass @@ -16,11 +19,14 @@ def test_decode_utf8(): space = FakeSpace() - assert decode_utf8(space, "abc") == u"abc" - assert decode_utf8(space, "\xe1\x88\xb4") == u"\u1234" - assert decode_utf8(space, "\xed\xa0\x80") == u"\ud800" - assert decode_utf8(space, "\xed\xb0\x80") == u"\udc00" - got = decode_utf8(space, "\xed\xa0\x80\xed\xb0\x80") - assert map(ord, got) == [0xd800, 0xdc00] - got = decode_utf8(space, "\xf0\x90\x80\x80") - assert map(ord, got) == [0x10000] + assert decode_utf8(space, "abc") == ("abc", 3) + assert decode_utf8(space, "\xe1\x88\xb4") == ("\xe1\x88\xb4", 1) + assert decode_utf8(space, "\xed\xa0\x80") == ("\xed\xa0\x80", 1) + assert decode_utf8(space, "\xed\xb0\x80") == ("\xed\xb0\x80", 1) + assert decode_utf8(space, "\xed\xa0\x80\xed\xb0\x80") == ( + "\xed\xa0\x80\xed\xb0\x80", 2) + assert decode_utf8(space, "\xf0\x90\x80\x80") == ("\xf0\x90\x80\x80", 1) + +def test_utf8_encode_ascii(): + assert utf8_encode_ascii("abc", 3, "??", "??") == "abc" + py.test.skip("test me more...") diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -62,7 +62,16 @@ # XXX argh. we want each surrogate to be encoded separately return ''.join([u.encode('utf8') for u in result_u]), len(result_u) -def check_utf8(space, string): +def check_ascii_or_raise(space, string): + try: + rutf8.check_ascii(string) + except rutf8.CheckError as e: + decode_error_handler(space)('strict', 'ascii', + 'ordinal not in range(128)', string, + e.pos, e.pos + 1) + assert False, "unreachable" + +def check_utf8_or_raise(space, string): # Surrogates are accepted and not treated specially at all. # If there happen to be two 3-bytes encoding a pair of surrogates, # you still get two surrogate unicode characters in the result. @@ -70,13 +79,13 @@ try: length = rutf8.check_utf8(string, allow_surrogates=True) except rutf8.CheckError as e: - raise Exception("foo") - decode_error_handler(space)('strict', 'utf8', e.msg, string, e.startpos, - e.endpos) - raise False, "unreachable" + decode_error_handler(space)('strict', 'utf8', 'invalid utf-8', string, + e.pos, e.pos + 1) + assert False, "unreachable" return length def encode_utf8(space, uni): + # DEPRECATED # Note that this function never raises UnicodeEncodeError, # since surrogates are allowed, either paired or lone. # A paired surrogate is considered like the non-BMP character @@ -87,11 +96,8 @@ allow_surrogates=True) def decode_utf8(space, s): - u, _ = runicode.str_decode_utf_8(s, len(s), - "strict", final=True, - errorhandler=decode_error_handler(space), - allow_surrogates=True) - return u.encode('utf8'), len(u) + # DEPRECATED + return (s, check_utf8_or_raise(space, s)) def utf8_encode_ascii(utf8, utf8len, errors, errorhandler): if len(utf8) == utf8len: @@ -109,7 +115,7 @@ else: utf8_repl, newpos, length = errorhandler(errors, 'ascii', 'ordinal not in range (128)', utf8, lgt, lgt + 1) - return b.build(), lgt + return b.build() def str_decode_ascii(s, slen, errors, final, errorhandler): try: diff --git a/pypy/objspace/std/marshal_impl.py b/pypy/objspace/std/marshal_impl.py --- a/pypy/objspace/std/marshal_impl.py +++ b/pypy/objspace/std/marshal_impl.py @@ -403,7 +403,7 @@ @unmarshaller(TYPE_UNICODE) def unmarshal_unicode(space, u, tc): arg = u.get_str() - length = unicodehelper.check_utf8(space, arg) + length = unicodehelper.check_utf8_or_raise(space, arg) return space.newutf8(arg, length) @marshaller(W_SetObject) diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -1106,22 +1106,11 @@ if errors is None or errors == 'strict': if encoding == 'ascii': s = space.charbuf_w(w_obj) - try: - rutf8.check_ascii(s) - except rutf8.CheckError as e: - unicodehelper.decode_error_handler(space)(None, - 'ascii', "ordinal not in range(128)", s, e.pos, e.pos+1) - assert False + unicodehelper.check_ascii_or_raise(space, s) return space.newutf8(s, len(s)) if encoding == 'utf-8': s = space.charbuf_w(w_obj) - try: - lgt = rutf8.check_utf8(s, allow_surrogates=True) - except rutf8.CheckError: - assert False, "fix in the future" - eh = unicodehelper.decode_error_handler(space) - eh(None, 'utf8', e.msg, s, e.startpos, e.endpos) - assert False, "has to raise" + lgt = unicodehelper.check_utf8_or_raise(space, s) return space.newutf8(s, lgt) w_codecs = space.getbuiltinmodule("_codecs") w_decode = space.getattr(w_codecs, space.newtext("decode")) @@ -1176,11 +1165,7 @@ if encoding != 'ascii': return unicode_from_encoded_object(space, w_bytes, encoding, "strict") s = space.bytes_w(w_bytes) - try: - rutf8.check_ascii(s) - except rutf8.CheckError: - # raising UnicodeDecodeError is messy, "please crash for me" - return unicode_from_encoded_object(space, w_bytes, "ascii", "strict") + unicodehelper.check_ascii_or_raise(space, s) return W_UnicodeObject(s, len(s)) From pypy.commits at gmail.com Thu Oct 12 10:54:10 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 07:54:10 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Implement and test these Message-ID: <59df8212.09a0df0a.ea0b7.69d9@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92735:cb0586abb276 Date: 2017-10-12 16:53 +0200 http://bitbucket.org/pypy/pypy/changeset/cb0586abb276/ Log: Implement and test these diff --git a/pypy/interpreter/test/test_unicodehelper.py b/pypy/interpreter/test/test_unicodehelper.py --- a/pypy/interpreter/test/test_unicodehelper.py +++ b/pypy/interpreter/test/test_unicodehelper.py @@ -29,4 +29,34 @@ def test_utf8_encode_ascii(): assert utf8_encode_ascii("abc", 3, "??", "??") == "abc" - py.test.skip("test me more...") + def eh(errors, encoding, reason, p, start, end): + lst.append((errors, encoding, p, start, end)) + return "", end + lst = [] + input = u"\u1234".encode("utf8") + assert utf8_encode_ascii(input, 1, "??", eh) == "" + assert lst == [("??", "ascii", input, 0, 1)] + lst = [] + input = u"\u1234\u5678abc\u8765\u4321".encode("utf8") + assert utf8_encode_ascii(input, 7, "??", eh) == "abc" + assert lst == [("??", "ascii", input, 0, 2), + ("??", "ascii", input, 5, 7)] + +def test_str_decode_ascii(): + assert str_decode_ascii("abc", 3, "??", True, "??") == ("abc", 3, 3) + def eh(errors, encoding, reason, p, start, end): + lst.append((errors, encoding, p, start, end)) + return u"\u1234\u5678", end + lst = [] + input = "\xe8" + exp = u"\u1234\u5678".encode("utf8") + assert str_decode_ascii(input, 1, "??", True, eh) == (exp, 1, 2) + assert lst == [("??", "ascii", input, 0, 1)] + lst = [] + input = "\xe8\xe9abc\xea\xeb" + assert str_decode_ascii(input, 7, "??", True, eh) == ( + exp + exp + "abc" + exp + exp, 7, 11) + assert lst == [("??", "ascii", input, 0, 1), + ("??", "ascii", input, 1, 2), + ("??", "ascii", input, 5, 6), + ("??", "ascii", input, 6, 7)] diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -102,28 +102,20 @@ def utf8_encode_ascii(utf8, utf8len, errors, errorhandler): if len(utf8) == utf8len: return utf8 - assert False, "implement" - b = StringBuilder(utf8len) - i = 0 - lgt = 0 - while i < len(utf8): - c = ord(utf8[i]) - if c <= 0x7F: - b.append(chr(c)) - lgt += 1 - i += 1 - else: - utf8_repl, newpos, length = errorhandler(errors, 'ascii', - 'ordinal not in range (128)', utf8, lgt, lgt + 1) - return b.build() + # No Way At All to emulate the calls to the error handler in + # less than three pages, so better not. + u = utf8.decode("utf8") + w = EncodeWrapper(errorhandler) + return runicode.unicode_encode_ascii(u, len(u), errors, w.handle) def str_decode_ascii(s, slen, errors, final, errorhandler): try: rutf8.check_ascii(s) return s, slen, len(s) except rutf8.CheckError: - raise Exception("foo") - return rutf8.str_decode_ascii(s, slen, errors, errorhandler) + w = DecodeWrapper((errorhandler)) + u, pos = runicode.str_decode_ascii(s, slen, errors, final, w.handle) + return u.encode('utf8'), pos, len(u) # XXX wrappers, think about speed From pypy.commits at gmail.com Thu Oct 12 11:52:39 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 08:52:39 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Re-enable this Message-ID: <59df8fc7.c5061c0a.87d5d.39af@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92736:f7ce6bd07d6a Date: 2017-10-12 17:52 +0200 http://bitbucket.org/pypy/pypy/changeset/f7ce6bd07d6a/ Log: Re-enable this diff --git a/pypy/objspace/std/listobject.py b/pypy/objspace/std/listobject.py --- a/pypy/objspace/std/listobject.py +++ b/pypy/objspace/std/listobject.py @@ -1027,7 +1027,6 @@ unilist = space.listview_unicode(w_iterable) if unilist is not None: - assert False, "disabled" w_list.strategy = strategy = space.fromcache(UnicodeListStrategy) # need to copy because intlist can share with w_iterable w_list.lstorage = strategy.erase(unilist[:]) From pypy.commits at gmail.com Thu Oct 12 12:23:59 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 09:23:59 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Re-enable this Message-ID: <59df971f.040c1c0a.d8798.50f7@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92737:28266792f75a Date: 2017-10-12 18:15 +0200 http://bitbucket.org/pypy/pypy/changeset/28266792f75a/ Log: Re-enable this diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -625,7 +625,6 @@ @unwrap_spec(utf8='utf8', errors='text_or_none') def charmap_encode(space, utf8, errors="strict", w_mapping=None): from pypy.interpreter.unicodehelper import EncodeWrapper - raise Exception("foo") if errors is None: errors = 'strict' @@ -645,12 +644,10 @@ @unwrap_spec(chars='utf8') def charmap_build(space, chars): # XXX CPython sometimes uses a three-level trie - raise Exception("foo") - XXXXXX w_charmap = space.newdict() pos = 0 num = 0 - while num < charslen: + while pos < len(chars): w_char = space.newint(rutf8.codepoint_at_pos(chars, pos)) space.setitem(w_charmap, w_char, space.newint(num)) pos = rutf8.next_codepoint_pos(chars, pos) From pypy.commits at gmail.com Thu Oct 12 12:24:01 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 09:24:01 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Revert runicode to be exactly like in default Message-ID: <59df9721.56acdf0a.95e3.62bb@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92738:4ab3528a9b0c Date: 2017-10-12 18:15 +0200 http://bitbucket.org/pypy/pypy/changeset/4ab3528a9b0c/ Log: Revert runicode to be exactly like in default diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -43,7 +43,7 @@ if len(u) == 1: return ord(u[0]) raise TypeError - + if MAXUNICODE > sys.maxunicode: # A version of unichr which allows codes outside the BMP # even on narrow unicode builds. @@ -340,14 +340,13 @@ # # See also unicode_encode_utf8sp(). # + if errorhandler is None: + errorhandler = default_unicode_error_encode return unicode_encode_utf_8_elidable(s, size, errors, errorhandler, allow_surrogates=allow_surrogates) def unicode_encode_utf_8_impl(s, size, errors, errorhandler, allow_surrogates=False): - # XXX hack - if errorhandler is None: - errorhandler = default_unicode_error_encode assert(size >= 0) result = StringBuilder(size) pos = 0 @@ -1282,9 +1281,8 @@ collend = pos+1 while collend < len(p) and ord(p[collend]) >= limit: collend += 1 - ru, pos = errorhandler(errors, encoding, reason, p, + ru, rs, pos = errorhandler(errors, encoding, reason, p, collstart, collend) - rs = None if rs is not None: # py3k only result.append(rs) @@ -1333,8 +1331,8 @@ c = mapping.get(ch, ERROR_CHAR) if c == ERROR_CHAR: r, pos = errorhandler(errors, "charmap", - "character maps to ", - s, pos, pos + 1) + "character maps to ", + s, pos, pos + 1) result.append(r) continue result.append(c) From pypy.commits at gmail.com Thu Oct 12 12:24:03 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 09:24:03 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Another fix by reverting this piece of code to default Message-ID: <59df9723.d78edf0a.e71ba.bdc8@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92739:6c1d3144913e Date: 2017-10-12 18:23 +0200 http://bitbucket.org/pypy/pypy/changeset/6c1d3144913e/ Log: Another fix by reverting this piece of code to default diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -73,7 +73,13 @@ return self._make_errorhandler(space, True) def make_encode_errorhandler(self, space): - return self._make_errorhandler(space, False) + errorhandler = self._make_errorhandler(space, False) + def encode_call_errorhandler(errors, encoding, reason, input, startpos, + endpos): + replace, newpos = errorhandler(errors, encoding, reason, input, + startpos, endpos) + return replace, None, newpos + return encode_call_errorhandler def get_unicodedata_handler(self, space): if self.unicodedata_handler: From pypy.commits at gmail.com Thu Oct 12 12:31:43 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 09:31:43 -0700 (PDT) Subject: [pypy-commit] pypy default: Simplification (and speed-up) Message-ID: <59df98ef.e291df0a.67c4.0192@mx.google.com> Author: Armin Rigo Branch: Changeset: r92740:926092bd058d Date: 2017-10-12 18:31 +0200 http://bitbucket.org/pypy/pypy/changeset/926092bd058d/ Log: Simplification (and speed-up) diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,20 +66,17 @@ "position %d from error handler out of bounds", newpos) replace = space.unicode_w(w_replace) - return replace, newpos + if decode: + return replace, newpos + else: + return replace, None, newpos return call_errorhandler def make_decode_errorhandler(self, space): return self._make_errorhandler(space, True) def make_encode_errorhandler(self, space): - errorhandler = self._make_errorhandler(space, False) - def encode_call_errorhandler(errors, encoding, reason, input, startpos, - endpos): - replace, newpos = errorhandler(errors, encoding, reason, input, - startpos, endpos) - return replace, None, newpos - return encode_call_errorhandler + return self._make_errorhandler(space, False) def get_unicodedata_handler(self, space): if self.unicodedata_handler: From pypy.commits at gmail.com Thu Oct 12 12:40:31 2017 From: pypy.commits at gmail.com (arigo) Date: Thu, 12 Oct 2017 09:40:31 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: hg merge default Message-ID: <59df9aff.84b5df0a.2d9af.ddc1@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92741:194830b5fc5a Date: 2017-10-12 18:37 +0200 http://bitbucket.org/pypy/pypy/changeset/194830b5fc5a/ Log: hg merge default diff too long, truncating to 2000 out of 16196 lines diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -60,8 +60,8 @@ Wim Lavrijsen Eric van Riet Paap Richard Emslie + Remi Meier Alexander Schremmer - Remi Meier Dan Villiom Podlaski Christiansen Lukas Diekmann Sven Hager @@ -102,6 +102,7 @@ Michael Foord Stephan Diehl Stefano Rivera + Jean-Paul Calderone Stefan Schwarzer Tomek Meka Valentino Volonghi @@ -110,14 +111,13 @@ Bob Ippolito Bruno Gola David Malcolm - Jean-Paul Calderone Squeaky Edd Barrett Timo Paulssen Marius Gedminas + Nicolas Truessel Alexandre Fayolle Simon Burton - Nicolas Truessel Martin Matusiak Laurence Tratt Wenzhu Man @@ -156,6 +156,7 @@ Stefan H. Muller Tim Felgentreff Eugene Oden + Dodan Mihai Jeff Terrace Henry Mason Vasily Kuznetsov @@ -182,11 +183,13 @@ Rocco Moretti Gintautas Miliauskas Lucian Branescu Mihaila + Mariano Anaya anatoly techtonik - Dodan Mihai Karl Bartel + Stefan Beyer Gabriel Lavoie Jared Grubb + Alecsandru Patrascu Olivier Dormond Wouter van Heyst Sebastian Pawluś @@ -194,6 +197,7 @@ Victor Stinner Andrews Medina Aaron Iles + p_zieschang at yahoo.de Toby Watson Daniel Patrick Stuart Williams @@ -204,6 +208,7 @@ Michael Cheng Mikael Schönenberg Stanislaw Halik + Mihnea Saracin Berkin Ilbeyi Gasper Zejn Faye Zhao @@ -214,14 +219,12 @@ Jonathan David Riehl Beatrice During Alex Perry - p_zieschang at yahoo.de Robert Zaremba Alan McIntyre Alexander Sedov Vaibhav Sood Reuben Cummings Attila Gobi - Alecsandru Patrascu Christopher Pope Tristan Arthur Christian Tismer @@ -243,7 +246,6 @@ Jacek Generowicz Sylvain Thenault Jakub Stasiak - Stefan Beyer Andrew Dalke Alejandro J. Cura Vladimir Kryachko @@ -275,6 +277,7 @@ Christoph Gerum Miguel de Val Borro Artur Lisiecki + afteryu Toni Mattis Laurens Van Houtven Bobby Impollonia @@ -305,6 +308,7 @@ Anna Katrina Dominguez Kim Jin Su Amber Brown + Anthony Sottile Nate Bragg Ben Darnell Juan Francisco Cantero Hurtado @@ -325,12 +329,14 @@ Mike Bayer Rodrigo Araújo Daniil Yarancev + Min RK OlivierBlanvillain Jonas Pfannschmidt Zearin Andrey Churin Dan Crosta reubano at gmail.com + Stanisław Halik Julien Phalip Roman Podoliaka Eli Stevens diff --git a/lib-python/2.7/ctypes/test/test_byteswap.py b/lib-python/2.7/ctypes/test/test_byteswap.py --- a/lib-python/2.7/ctypes/test/test_byteswap.py +++ b/lib-python/2.7/ctypes/test/test_byteswap.py @@ -23,7 +23,6 @@ setattr(bits, "i%s" % i, 1) dump(bits) - @xfail def test_endian_short(self): if sys.byteorder == "little": self.assertIs(c_short.__ctype_le__, c_short) @@ -51,7 +50,6 @@ self.assertEqual(bin(s), "3412") self.assertEqual(s.value, 0x1234) - @xfail def test_endian_int(self): if sys.byteorder == "little": self.assertIs(c_int.__ctype_le__, c_int) @@ -80,7 +78,6 @@ self.assertEqual(bin(s), "78563412") self.assertEqual(s.value, 0x12345678) - @xfail def test_endian_longlong(self): if sys.byteorder == "little": self.assertIs(c_longlong.__ctype_le__, c_longlong) @@ -109,7 +106,6 @@ self.assertEqual(bin(s), "EFCDAB9078563412") self.assertEqual(s.value, 0x1234567890ABCDEF) - @xfail def test_endian_float(self): if sys.byteorder == "little": self.assertIs(c_float.__ctype_le__, c_float) @@ -128,7 +124,6 @@ self.assertAlmostEqual(s.value, math.pi, 6) self.assertEqual(bin(struct.pack(">f", math.pi)), bin(s)) - @xfail def test_endian_double(self): if sys.byteorder == "little": self.assertIs(c_double.__ctype_le__, c_double) @@ -156,7 +151,6 @@ self.assertIs(c_char.__ctype_le__, c_char) self.assertIs(c_char.__ctype_be__, c_char) - @xfail def test_struct_fields_1(self): if sys.byteorder == "little": base = BigEndianStructure @@ -192,7 +186,6 @@ pass self.assertRaises(TypeError, setattr, T, "_fields_", [("x", typ)]) - @xfail def test_struct_struct(self): # nested structures with different byteorders @@ -221,7 +214,6 @@ self.assertEqual(s.point.x, 1) self.assertEqual(s.point.y, 2) - @xfail def test_struct_fields_2(self): # standard packing in struct uses no alignment. # So, we have to align using pad bytes. @@ -245,7 +237,6 @@ s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14) self.assertEqual(bin(s1), bin(s2)) - @xfail def test_unaligned_nonnative_struct_fields(self): if sys.byteorder == "little": base = BigEndianStructure diff --git a/lib-python/2.7/ctypes/test/test_unaligned_structures.py b/lib-python/2.7/ctypes/test/test_unaligned_structures.py --- a/lib-python/2.7/ctypes/test/test_unaligned_structures.py +++ b/lib-python/2.7/ctypes/test/test_unaligned_structures.py @@ -37,10 +37,7 @@ for typ in byteswapped_structures: ## print >> sys.stderr, typ.value self.assertEqual(typ.value.offset, 1) - try: - o = typ() - except NotImplementedError as e: - self.skipTest(str(e)) # for PyPy + o = typ() o.value = 4 self.assertEqual(o.value, 4) diff --git a/lib-python/2.7/inspect.py b/lib-python/2.7/inspect.py --- a/lib-python/2.7/inspect.py +++ b/lib-python/2.7/inspect.py @@ -203,7 +203,7 @@ f_locals local namespace seen by this frame f_restricted 0 or 1 if frame is in restricted execution mode f_trace tracing function for this frame, or None""" - return isinstance(object, types.FrameType) + return isinstance(object, (types.FrameType, types.FakeFrameType)) def iscode(object): """Return true if the object is a code object. diff --git a/lib-python/2.7/multiprocessing/heap.py b/lib-python/2.7/multiprocessing/heap.py --- a/lib-python/2.7/multiprocessing/heap.py +++ b/lib-python/2.7/multiprocessing/heap.py @@ -62,7 +62,7 @@ self.size = size self.name = 'pym-%d-%d' % (os.getpid(), Arena._counter.next()) self.buffer = mmap.mmap(-1, self.size, tagname=self.name) - assert win32.GetLastError() == 0, 'tagname already in use' + #assert win32.GetLastError() == 0, 'tagname already in use' self._state = (self.size, self.name) def __getstate__(self): @@ -72,7 +72,7 @@ def __setstate__(self, state): self.size, self.name = self._state = state self.buffer = mmap.mmap(-1, self.size, tagname=self.name) - assert win32.GetLastError() == win32.ERROR_ALREADY_EXISTS + #assert win32.GetLastError() == win32.ERROR_ALREADY_EXISTS else: diff --git a/lib-python/2.7/string.py b/lib-python/2.7/string.py --- a/lib-python/2.7/string.py +++ b/lib-python/2.7/string.py @@ -75,7 +75,7 @@ for i in range(256): buf[i] = i for i in range(n): - buf[ord(fromstr[i])] = tostr[i] + buf[ord(fromstr[i])] = ord(tostr[i]) return str(buf) diff --git a/lib-python/2.7/types.py b/lib-python/2.7/types.py --- a/lib-python/2.7/types.py +++ b/lib-python/2.7/types.py @@ -71,6 +71,12 @@ FrameType = type(tb.tb_frame) del tb +# PyPy extension +try: + FakeFrameType = type(next(sys._current_frames().itervalues())) +except (AttributeError, StopIteration): + FakeFrameType = FrameType + SliceType = slice EllipsisType = type(Ellipsis) diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -142,6 +142,10 @@ ptr._buffer = tp._ffiarray(1, autofree=True) ptr._buffer[0] = obj._buffer result = ptr + elif isinstance(obj, bytes): + result = tp() + result._buffer[0] = buffer(obj)._pypy_raw_address() + return result elif not (isinstance(obj, _CData) and type(obj)._is_pointer_like()): raise TypeError("cast() argument 1 must be a pointer, not %s" % (type(obj),)) diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py --- a/lib_pypy/_ctypes/primitive.py +++ b/lib_pypy/_ctypes/primitive.py @@ -61,6 +61,54 @@ pyobj_container = GlobalPyobjContainer() +def swap_bytes(value, sizeof, typeof, get_or_set): + def swap_2(): + return ((value >> 8) & 0x00FF) | ((value << 8) & 0xFF00) + + def swap_4(): + return ((value & 0x000000FF) << 24) | \ + ((value & 0x0000FF00) << 8) | \ + ((value & 0x00FF0000) >> 8) | \ + ((value >> 24) & 0xFF) + + def swap_8(): + return ((value & 0x00000000000000FFL) << 56) | \ + ((value & 0x000000000000FF00L) << 40) | \ + ((value & 0x0000000000FF0000L) << 24) | \ + ((value & 0x00000000FF000000L) << 8) | \ + ((value & 0x000000FF00000000L) >> 8) | \ + ((value & 0x0000FF0000000000L) >> 24) | \ + ((value & 0x00FF000000000000L) >> 40) | \ + ((value >> 56) & 0xFF) + + def swap_double_float(typ): + from struct import pack, unpack + if get_or_set == 'set': + if sys.byteorder == 'little': + st = pack(''.join(['>', typ]), value) + else: + st = pack(''.join(['<', typ]), value) + return unpack(typ, st)[0] + else: + packed = pack(typ, value) + if sys.byteorder == 'little': + st = unpack(''.join(['>', typ]), packed) + else: + st = unpack(''.join(['<', typ]), packed) + return st[0] + + if typeof in ('c_float', 'c_float_le', 'c_float_be'): + return swap_double_float('f') + elif typeof in ('c_double', 'c_double_le', 'c_double_be'): + return swap_double_float('d') + else: + if sizeof == 2: + return swap_2() + elif sizeof == 4: + return swap_4() + elif sizeof == 8: + return swap_8() + def generic_xxx_p_from_param(cls, value): if value is None: return cls(None) @@ -271,6 +319,31 @@ def _as_ffi_pointer_(self, ffitype): return as_ffi_pointer(self, ffitype) result._as_ffi_pointer_ = _as_ffi_pointer_ + if name[-2:] != '_p' and name[-3:] not in ('_le', '_be') \ + and name not in ('c_wchar', '_SimpleCData', 'c_longdouble', 'c_bool', 'py_object'): + from sys import byteorder + if byteorder == 'big': + name += '_le' + swapped = self.__new__(self, name, bases, dct) + result.__ctype_le__ = swapped + result.__ctype_be__ = result + swapped.__ctype_be__ = result + swapped.__ctype_le__ = swapped + else: + name += '_be' + swapped = self.__new__(self, name, bases, dct) + result.__ctype_be__ = swapped + result.__ctype_le__ = result + swapped.__ctype_le__ = result + swapped.__ctype_be__ = swapped + from _ctypes import sizeof + def _getval(self): + return swap_bytes(self._buffer[0], sizeof(self), name, 'get') + def _setval(self, value): + d = result() + d.value = value + self._buffer[0] = swap_bytes(d.value, sizeof(self), name, 'set') + swapped.value = property(_getval, _setval) return result diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -146,6 +146,7 @@ obj._buffer.__setattr__(self.name, arg) + def _set_shape(tp, rawfields, is_union=False): tp._ffistruct_ = _rawffi.Structure(rawfields, is_union, getattr(tp, '_pack_', 0)) @@ -240,19 +241,27 @@ res.__dict__['_index'] = -1 return res - class StructOrUnion(_CData): __metaclass__ = StructOrUnionMeta def __new__(cls, *args, **kwds): from _ctypes import union - self = super(_CData, cls).__new__(cls) - if ('_abstract_' in cls.__dict__ or cls is Structure + if ('_abstract_' in cls.__dict__ or cls is Structure or cls is union.Union): raise TypeError("abstract class") if hasattr(cls, '_swappedbytes_'): - raise NotImplementedError("missing in PyPy: structure/union with " - "swapped (non-native) byte ordering") + fields = [None] * len(cls._fields_) + for i in range(len(cls._fields_)): + if cls._fields_[i][1] == cls._fields_[i][1].__dict__.get('__ctype_be__', None): + swapped = cls._fields_[i][1].__dict__.get('__ctype_le__', cls._fields_[i][1]) + else: + swapped = cls._fields_[i][1].__dict__.get('__ctype_be__', cls._fields_[i][1]) + if len(cls._fields_[i]) < 3: + fields[i] = (cls._fields_[i][0], swapped) + else: + fields[i] = (cls._fields_[i][0], swapped, cls._fields_[i][2]) + names_and_fields(cls, fields, _CData, cls.__dict__.get('_anonymous_', None)) + self = super(_CData, cls).__new__(cls) if hasattr(cls, '_ffistruct_'): self.__dict__['_buffer'] = self._ffistruct_(autofree=True) return self diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.0 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.0" -__version_info__ = (1, 11, 0) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.0" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/lib_pypy/pyrepl/historical_reader.py b/lib_pypy/pyrepl/historical_reader.py --- a/lib_pypy/pyrepl/historical_reader.py +++ b/lib_pypy/pyrepl/historical_reader.py @@ -17,7 +17,7 @@ # CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN # CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. -from pyrepl import reader, commands +from pyrepl import reader, commands, input from pyrepl.reader import Reader as R isearch_keymap = tuple( @@ -214,7 +214,6 @@ isearch_forwards, isearch_backwards, operate_and_get_next]: self.commands[c.__name__] = c self.commands[c.__name__.replace('_', '-')] = c - from pyrepl import input self.isearch_trans = input.KeymapTranslator( isearch_keymap, invalid_cls=isearch_end, character_cls=isearch_add_character) diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,8 +119,15 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On Debian and Ubuntu, this is the command to install all build-time -dependencies:: +On recent Debian and Ubuntu (16.04 onwards), this is the command to install +all build-time dependencies:: + + apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ + libsqlite3-dev libncurses5-dev libexpat1-dev libssl-dev libgdbm-dev \ + tk-dev libgc-dev python-cffi \ + liblzma-dev libncursesw5-dev # these two only needed on PyPy3 + +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -142,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -180,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -248,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/contributor.rst b/pypy/doc/contributor.rst --- a/pypy/doc/contributor.rst +++ b/pypy/doc/contributor.rst @@ -27,8 +27,8 @@ Wim Lavrijsen Eric van Riet Paap Richard Emslie + Remi Meier Alexander Schremmer - Remi Meier Dan Villiom Podlaski Christiansen Lukas Diekmann Sven Hager @@ -69,6 +69,7 @@ Michael Foord Stephan Diehl Stefano Rivera + Jean-Paul Calderone Stefan Schwarzer Tomek Meka Valentino Volonghi @@ -77,14 +78,13 @@ Bob Ippolito Bruno Gola David Malcolm - Jean-Paul Calderone Squeaky Edd Barrett Timo Paulssen Marius Gedminas + Nicolas Truessel Alexandre Fayolle Simon Burton - Nicolas Truessel Martin Matusiak Laurence Tratt Wenzhu Man @@ -123,6 +123,7 @@ Stefan H. Muller Tim Felgentreff Eugene Oden + Dodan Mihai Jeff Terrace Henry Mason Vasily Kuznetsov @@ -149,11 +150,13 @@ Rocco Moretti Gintautas Miliauskas Lucian Branescu Mihaila + Mariano Anaya anatoly techtonik - Dodan Mihai Karl Bartel + Stefan Beyer Gabriel Lavoie Jared Grubb + Alecsandru Patrascu Olivier Dormond Wouter van Heyst Sebastian Pawluś @@ -161,6 +164,7 @@ Victor Stinner Andrews Medina Aaron Iles + p_zieschang at yahoo.de Toby Watson Daniel Patrick Stuart Williams @@ -171,6 +175,7 @@ Michael Cheng Mikael Schönenberg Stanislaw Halik + Mihnea Saracin Berkin Ilbeyi Gasper Zejn Faye Zhao @@ -181,14 +186,12 @@ Jonathan David Riehl Beatrice During Alex Perry - p_zieschang at yahoo.de Robert Zaremba Alan McIntyre Alexander Sedov Vaibhav Sood Reuben Cummings Attila Gobi - Alecsandru Patrascu Christopher Pope Tristan Arthur Christian Tismer @@ -210,7 +213,6 @@ Jacek Generowicz Sylvain Thenault Jakub Stasiak - Stefan Beyer Andrew Dalke Alejandro J. Cura Vladimir Kryachko @@ -242,6 +244,7 @@ Christoph Gerum Miguel de Val Borro Artur Lisiecki + afteryu Toni Mattis Laurens Van Houtven Bobby Impollonia @@ -272,6 +275,7 @@ Anna Katrina Dominguez Kim Jin Su Amber Brown + Anthony Sottile Nate Bragg Ben Darnell Juan Francisco Cantero Hurtado @@ -292,12 +296,14 @@ Mike Bayer Rodrigo Araújo Daniil Yarancev + Min RK OlivierBlanvillain Jonas Pfannschmidt Zearin Andrey Churin Dan Crosta reubano at gmail.com + Stanisław Halik Julien Phalip Roman Podoliaka Eli Stevens diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst --- a/pypy/doc/cpython_differences.rst +++ b/pypy/doc/cpython_differences.rst @@ -429,7 +429,8 @@ * the ``__builtins__`` name is always referencing the ``__builtin__`` module, never a dictionary as it sometimes is in CPython. Assigning to - ``__builtins__`` has no effect. + ``__builtins__`` has no effect. (For usages of tools like + RestrictedPython, see `issue #2653`_.) * directly calling the internal magic methods of a few built-in types with invalid arguments may have a slightly different result. For @@ -535,7 +536,12 @@ or ``float`` subtypes. Currently PyPy does not support the ``__class__`` attribute assignment for any non heaptype subtype. +* In PyPy, module and class dictionaries are optimized under the assumption + that deleting attributes from them are rare. Because of this, e.g. + ``del foo.bar`` where ``foo`` is a module (or class) that contains the + function ``bar``, is significantly slower than CPython. + .. _`is ignored in PyPy`: http://bugs.python.org/issue14621 .. _`little point`: http://events.ccc.de/congress/2012/Fahrplan/events/5152.en.html .. _`#2072`: https://bitbucket.org/pypy/pypy/issue/2072/ - +.. _`issue #2653`: https://bitbucket.org/pypy/pypy/issues/2653/ diff --git a/pypy/doc/index-of-release-notes.rst b/pypy/doc/index-of-release-notes.rst --- a/pypy/doc/index-of-release-notes.rst +++ b/pypy/doc/index-of-release-notes.rst @@ -6,6 +6,7 @@ .. toctree:: + release-v5.9.0.rst release-v5.8.0.rst release-v5.7.1.rst release-v5.7.0.rst diff --git a/pypy/doc/index-of-whatsnew.rst b/pypy/doc/index-of-whatsnew.rst --- a/pypy/doc/index-of-whatsnew.rst +++ b/pypy/doc/index-of-whatsnew.rst @@ -7,6 +7,7 @@ .. toctree:: whatsnew-head.rst + whatsnew-pypy2-5.9.0.rst whatsnew-pypy2-5.8.0.rst whatsnew-pypy2-5.7.0.rst whatsnew-pypy2-5.6.0.rst @@ -36,6 +37,7 @@ .. toctree:: whatsnew-pypy3-head.rst + whatsnew-pypy3-5.9.0.rst whatsnew-pypy3-5.8.0.rst whatsnew-pypy3-5.7.0.rst diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/release-v5.9.0.rst b/pypy/doc/release-v5.9.0.rst new file mode 100644 --- /dev/null +++ b/pypy/doc/release-v5.9.0.rst @@ -0,0 +1,217 @@ +===================================== +PyPy2.7 and PyPy3.5 v5.9 dual release +===================================== + +The PyPy team is proud to release both PyPy2.7 v5.9 (an interpreter supporting +Python 2.7 syntax), and a beta-quality PyPy3.5 v5.9 (an interpreter for Python +3.5 syntax). The two releases are both based on much the same codebase, thus +the dual release. Note that PyPy3.5 supports Linux 64bit only for now. + +This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and +PyPy3.5 includes the upstream stdlib version 3.5.3. + +NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues +that appeared as excessive memory +use were cleared up and other incompatibilities were resolved. The C-API +compatibility layer does slow down code which crosses the python-c interface +often, we have ideas on how it could be improved, and still recommend +using pure python on PyPy or interfacing via CFFI_. Many other modules +based on C-API exentions now work on PyPy as well. + +Cython 0.27.1 (released very recently) supports more projects with PyPy, both +on PyPy2.7 and PyPy3.5 beta. Note version **0.27.1** is now the minimum +version that supports this version of PyPy, due to some interactions with +updated C-API interface code. + +We optimized the JSON parser for recurring string keys, which should decrease +memory use to 50% and increase parsing speed by up to 15% for large JSON files +with many repeating dictionary keys (which is quite common). + +CFFI_, which is part of the PyPy release, has been updated to 1.11.1, +improving an already great package for interfacing with C. CFFI now supports +complex arguments in API mode, as well as ``char16_t`` and ``char32_t`` and has +improved support for callbacks. + +Please let us know if your use case is slow, we have ideas how to make things +faster but need real-world examples (not micro-benchmarks) of problematic code. + +Work sponsored by a Mozilla grant_ continues on PyPy3.5; numerous fixes from +CPython were ported to PyPy. Of course the bug fixes and performance enhancements +mentioned above are part of both PyPy2.7 and PyPy3.5 beta. + +As always, this release fixed many other issues and bugs raised by the +growing community of PyPy users. We strongly recommend updating. + +You can download the v5.9 releases here: + + http://pypy.org/download.html + +We would like to thank our donors for the continued support of the PyPy +project. + +We would also like to thank our contributors and +encourage new people to join the project. PyPy has many +layers and we need help with all of them: `PyPy`_ and `RPython`_ documentation +improvements, tweaking popular `modules`_ to run on pypy, or general `help`_ +with making RPython's JIT even better. + +.. _vmprof: http://vmprof.readthedocs.io +.. _CFFI: https://cffi.readthedocs.io/en/latest/whatsnew.html +.. _grant: https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from-mozilla-for.html +.. _`PyPy`: index.html +.. _`RPython`: https://rpython.readthedocs.org +.. _`modules`: project-ideas.html#make-more-python-modules-pypy-friendly +.. _`help`: project-ideas.html + +What is PyPy? +============= + +PyPy is a very compliant Python interpreter, almost a drop-in replacement for +CPython 2.7 and CPython 3.5. It's fast (`PyPy and CPython 2.7.x`_ performance comparison) +due to its integrated tracing JIT compiler. + +We also welcome developers of other `dynamic languages`_ to see what RPython +can do for them. + +The PyPy 2.7 release supports: + + * **x86** machines on most common operating systems + (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD) + + * newer **ARM** hardware (ARMv6 or ARMv7, with VFPv3) running Linux, + + * big- and little-endian variants of **PPC64** running Linux, + + * **s390x** running Linux + +.. _`PyPy and CPython 2.7.x`: http://speed.pypy.org +.. _`dynamic languages`: http://rpython.readthedocs.io/en/latest/examples.html + +Highlights of the PyPy2.7, cpyext, and RPython changes (since 5.8 released June, 2017) +====================================================================================== + +See also issues that were resolved_ + +Note that these are also merged into PyPy 3.5 + +* New features and cleanups + + * Add support for ``PyFrozenSet_New``, ``PyObject_HashNotImplemented``, + ``PyObject_Print(NULL, ...)``, ``PyObject_RichCompareBool(a, a, ...)``, + ``PyType_IS_GC`` (does nothing), ``PyUnicode_FromFormat`` + * ctypes ``char_p`` and ``unichar_p`` indexing now CPython compatible + * ``gcdump`` now reports largest object + * More complete support in the ``_curses`` CFFI module + * Add cPickle.Unpickler.find_global (issue 1853_) + * Fix ``PyErr_Fetch`` + ``PyErr_NormalizeException`` with no exception set + * Simplify ``gc.get_referrers()`` to return the opposite of ``gc.get_referents()`` + * Update RevDB to version pypy2.7-v5.6.2 + * Previously, ``instance.method`` would return always the same bound method + object, when gotten from the same instance (as far as ``is`` and ``id()`` + can tell). CPython doesn't do that. Now PyPy, like CPython, returns a + different bound method object every time. For ``type.method``, PyPy2 still + returns always the same *unbound* method object; CPython does it for built-in + types but not for user-defined types + * Link to disable PaX protection for the JIT when needed + * Update build instructions and an rarely used Makefile + * Recreate support for using leakfinder in cpyext tests which had suffered + bit-rot, disable due to many false positives + * Add more functionality to ``sysconfig`` + * Added ``_swappedbytes_`` support for ``ctypes.Structure`` + * Better support the ``inspect`` module on ``frames`` + +* Bug Fixes + + * Fix issue 2592_ - cpyext ``PyListObject.pop``, ``pop_end`` must return a value + * Implement ``PyListOjbect.getstorage_copy`` + * Fix for ``reversed(dictproxy)`` issue 2601_ + * Fix for duplicate names in ctypes' ``_fields__``, issue 2621_ + * Update built-in ``pyexpat`` module on win32 to use UTF-8 version not UTF-16 + * ``gc.get_objects`` now handles objects with finalizers more consistently + * Fixed memory leak in ``SSLContext.getpeercert`` returning validated + certificates and ``SSLContext.get_ca_certs(binary_mode=True)`` + (_get_crl_dp) `CPython issue 29738`_ + +* Performance improvements: + + * Improve performance of ``bytearray.extend`` by rewriting portions in app-level + * Optimize list accesses with constant indexes better by retaining more + information about them + * Add a jit driver for ``array.count`` and ``array.index`` + * Improve information retained in a bridge wrt ``array`` + * Move some dummy CAPI functions and ``Py*_Check`` functions from RPython into + pure C macros + * In the fast ``zip(intlist1, intlist2)`` implementation, don't wrap and unwrap + all the ints + * Cache string keys that occur in JSON dicts, as they are likely to repeat + +* RPython improvements + + * Do not preallocate a RPython list if we only know an upper bound on its size + * Issue 2590_: fix the bounds in the GC when allocating a lot of objects with finalizers + * Replace magical NOT RPYTHON comment with a decorator + * Implement ``socket.sendmsg()``/``.recvmsg()`` for py3.5 + * Add ``memory_pressure`` for ``_SSLSocket`` objects + +* Degredations + + * Disable vmprof on win32, due to upstream changes that break the internal ``_vmprof`` module + +.. _here: cpython_differences.html +.. _1853: https://bitbucket.org/pypy/pypy/issues/1853 +.. _2592: https://bitbucket.org/pypy/pypy/issues/2592 +.. _2590: https://bitbucket.org/pypy/pypy/issues/2590 +.. _2621: https://bitbucket.org/pypy/pypy/issues/2621 + +Highlights of the PyPy3.5 release (since 5.8 beta released June 2017) +====================================================================== + +* New features + + * Add support for ``_PyNamespace_New``, ``PyMemoryView_FromMemory``, + ``Py_EnterRecursiveCall`` raising RecursionError, ``PyObject_LengthHint``, + ``PyUnicode_FromKindAndData``, ``PyDict_SetDefault``, ``PyGenObject``, + ``PyGenObject``, ``PyUnicode_Substring``, ``PyLong_FromUnicodeObject`` + * Implement ``PyType_FromSpec`` (PEP 384) and fix issues with PEP 489 support + * Support the new version of ``os.stat()`` on win32 + * Use ``stat3()`` on Posix + * Accept buffer objects as filenames, except for `oslistdir`` + * Make slices of array ``memoryview`` s usable as writable buffers if contiguous + * Better handling of ``'%s'`` formatting for byte strings which might be utf-8 encoded + * Update the macros ``Py_DECREF`` and similar to use the CPython 3.5 version + * Ensure that ``mappingproxy`` is recognised as a mapping, not a sequence + * Enable PGO for CLang + * Rework ``cppyy`` packaging and rename the backend to ``_cppyy`` + * Support for libressl 2.5.4 + * Mirror CPython ``classmethod __reduce__`` which fixes pickling test + * Use utf-8 for ``readline`` history file + * Allow assigning ``'__class__'`` between ``ModuleType`` and its subclasses + * Add async slot functions in cpyext + +* Bug Fixes + + * Try to make ``openssl`` CFFI bindings more general and future-proof + * Better support ``importlib`` by only listing built-in modules in ``sys.builtin`` + * Add ``memory_pressure`` to large CFFI allocations in ``_lzma``, issue 2579_ + * Fix for ``reversed(mapping object)`` issue 2601_ + * Fixing regression with non-started generator receiving non-``None``, should + always raise ``TypeError`` + * ``itertools.islice``: use same logic as CPython, fixes 2643_ + +* Performance improvements: + + * + +* The following features of Python 3.5 are not implemented yet in PyPy: + + * PEP 442: Safe object finalization + +.. _resolved: whatsnew-pypy2-5.9.0.html +.. _2579: https://bitbucket.org/pypy/pypy/issues/2579 +.. _2601: https://bitbucket.org/pypy/pypy/issues/2601 +.. _2643: https://bitbucket.org/pypy/pypy/issues/2643 +.. _CPython issue 29738: https://bugs.python.org/issue29738 + +Please update, and continue to help us make PyPy better. + +Cheers diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -1,75 +1,12 @@ -========================== -What's new in PyPy2.7 5.9+ -========================== - -.. this is a revision shortly after release-pypy2.7-v5.8.0 -.. startrev: 558bd00b3dd8 - -In previous versions of PyPy, ``instance.method`` would return always -the same bound method object, when gotten out of the same instance (as -far as ``is`` and ``id()`` can tell). CPython doesn't do that. Now -PyPy, like CPython, returns a different bound method object every time. -For ``type.method``, PyPy2 still returns always the same *unbound* -method object; CPython does it for built-in types but not for -user-defined types. - -.. branch: cffi-complex -.. branch: cffi-char16-char32 - -The two ``cffi-*`` branches are part of the upgrade to cffi 1.11. - -.. branch: ctypes_char_indexing - -Indexing into char* behaves differently than CPython - -.. branch: vmprof-0.4.8 - -Improve and fix issues with vmprof - -.. branch: issue-2592 - -CPyext PyListObject.pop must return the value - -.. branch: cpyext-hash_notimpl - -If ``tp_hash`` is ``PyObject_HashNotImplemented``, set ``obj.__dict__['__hash__']`` to None - -.. branch: cppyy-packaging - -Renaming of ``cppyy`` to ``_cppyy``. -The former is now an external package installable with ``pip install cppyy``. - -.. branch: Enable_PGO_for_clang - -.. branch: nopax - -At the end of translation, run ``attr -q -s pax.flags -V m`` on -PAX-enabled systems on the produced binary. This seems necessary -because PyPy uses a JIT. - -.. branch: pypy_bytearray - -Improve ``bytearray`` performance (backported from py3.5) - -.. branch: gc-del-limit-growth - -Fix the bounds in the GC when allocating a lot of objects with finalizers, -fixes issue #2590 - -.. branch: arrays-force-less - -Small improvement to optimize list accesses with constant indexes better by -throwing away information about them less eagerly. - - -.. branch: getarrayitem-into-bridges - -More information is retained into a bridge: knowledge about the content of -arrays (at fixed indices) is stored in guards (and thus available at the -beginning of bridges). Also, some better feeding of information about known -fields of constant objects into bridges. - -.. branch: cpyext-leakchecking - -Add support for leakfinder in cpyext tests (disabled for now, due to too many -failures). +=========================== +What's new in PyPy2.7 5.10+ +=========================== + +.. this is a revision shortly after release-pypy2.7-v5.9.0 +.. startrev:d56dadcef996 + +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging + +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst new file mode 100644 --- /dev/null +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -0,0 +1,96 @@ +========================= +What's new in PyPy2.7 5.9 +========================= + +.. this is a revision shortly after release-pypy2.7-v5.8.0 +.. startrev: 558bd00b3dd8 + +In previous versions of PyPy, ``instance.method`` would return always +the same bound method object, when gotten out of the same instance (as +far as ``is`` and ``id()`` can tell). CPython doesn't do that. Now +PyPy, like CPython, returns a different bound method object every time. +For ``type.method``, PyPy2 still returns always the same *unbound* +method object; CPython does it for built-in types but not for +user-defined types. + +.. branch: cffi-complex +.. branch: cffi-char16-char32 + +The two ``cffi-*`` branches are part of the upgrade to cffi 1.11. + +.. branch: ctypes_char_indexing + +Indexing into char* behaves differently than CPython + +.. branch: vmprof-0.4.8 + +Improve and fix issues with vmprof + +.. branch: issue-2592 + +CPyext PyListObject.pop must return the value + +.. branch: cpyext-hash_notimpl + +If ``tp_hash`` is ``PyObject_HashNotImplemented``, set ``obj.__dict__['__hash__']`` to None + +.. branch: cppyy-packaging + +Renaming of ``cppyy`` to ``_cppyy``. +The former is now an external package installable with ``pip install cppyy``. + +.. branch: Enable_PGO_for_clang + +.. branch: nopax + +At the end of translation, run ``attr -q -s pax.flags -V m`` on +PAX-enabled systems on the produced binary. This seems necessary +because PyPy uses a JIT. + +.. branch: pypy_bytearray + +Improve ``bytearray`` performance (backported from py3.5) + +.. branch: gc-del-limit-growth + +Fix the bounds in the GC when allocating a lot of objects with finalizers, +fixes issue #2590 + +.. branch: arrays-force-less + +Small improvement to optimize list accesses with constant indexes better by +throwing away information about them less eagerly. + + +.. branch: getarrayitem-into-bridges + +More information is retained into a bridge: knowledge about the content of +arrays (at fixed indices) is stored in guards (and thus available at the +beginning of bridges). Also, some better feeding of information about known +fields of constant objects into bridges. + +.. branch: cpyext-leakchecking + +Add support for leakfinder in cpyext tests (disabled for now, due to too many +failures). + +.. branch: pypy_swappedbytes + +Added ``_swappedbytes_`` support for ``ctypes.Structure`` + +.. branch: pycheck-macros + +Convert many Py*_Check cpyext functions into macros, like CPython. + +.. branch: py_ssize_t + +Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/doc/whatsnew-pypy3-5.9.0.rst b/pypy/doc/whatsnew-pypy3-5.9.0.rst new file mode 100644 --- /dev/null +++ b/pypy/doc/whatsnew-pypy3-5.9.0.rst @@ -0,0 +1,7 @@ +======================= +What's new in PyPy3 5.9 +======================= + +.. this is the revision after release-pypy3.5-5.8 +.. startrev: afbf09453369 + diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -1,7 +1,7 @@ ========================= -What's new in PyPy3 5.7+ +What's new in PyPy3 5.9+ ========================= -.. this is the revision after release-pypy3.3-5.7.x was branched -.. startrev: afbf09453369 +.. this is the revision after release-pypy3.5-5.9 +.. startrev: be41e3ac0a29 diff --git a/pypy/doc/windows.rst b/pypy/doc/windows.rst --- a/pypy/doc/windows.rst +++ b/pypy/doc/windows.rst @@ -114,12 +114,15 @@ INCLUDE, LIB and PATH (for DLLs) environment variables appropriately. -Abridged method (for -Ojit builds using Visual Studio 2008) ------------------------------------------------------------ +Abridged method (using Visual Studio 2008) +------------------------------------------ Download the versions of all the external packages from +https://bitbucket.org/pypy/pypy/downloads/local_59.zip +(for post-5.8 builds) with sha256 checksum +``6344230e90ab7a9cb84efbae1ba22051cdeeb40a31823e0808545b705aba8911`` https://bitbucket.org/pypy/pypy/downloads/local_5.8.zip -(for post-5.7.1 builds) with sha256 checksum +(to reproduce 5.8 builds) with sha256 checksum ``fbe769bf3a4ab6f5a8b0a05b61930fc7f37da2a9a85a8f609cf5a9bad06e2554`` or https://bitbucket.org/pypy/pypy/downloads/local_2.4.zip (for 2.4 release and later) or @@ -135,8 +138,8 @@ Now you should be good to go. If you choose this method, you do not need to download/build anything else. -Nonabrided method (building from scratch) ------------------------------------------ +Nonabridged method (building from scratch) +------------------------------------------ If you want to, you can rebuild everything from scratch by continuing. @@ -209,17 +212,85 @@ The expat XML parser ~~~~~~~~~~~~~~~~~~~~ -Download the source code of expat on sourceforge: -https://github.com/libexpat/libexpat/releases and extract it in the base directory. -Version 2.1.1 is known to pass tests. Then open the project file ``expat.dsw`` -with Visual Studio; follow the instruction for converting the project files, -switch to the "Release" configuration, use the ``expat_static`` project, -reconfigure the runtime for Multi-threaded DLL (/MD) and build. Do the same for -the ``expat`` project to build the ``expat.dll`` (for tests via ll2ctypes) +CPython compiles expat from source as part of the build. PyPy uses the same +code base, but expects to link to a static lib of expat. Here are instructions +to reproduce the static lib in version 2.2.4. -Then, copy the file ``win32\bin\release\libexpat.lib`` into -LIB, and both ``lib\expat.h`` and ``lib\expat_external.h`` in -INCLUDE, and ``win32\bin\release\libexpat.dll`` into PATH. +Download the source code of expat: https://github.com/libexpat/libexpat. +``git checkout`` the proper tag, in this case ``R_2_2_4``. Run +``vcvars.bat`` to set up the visual compiler tools, and CD into the source +directory. Create a file ``stdbool.h`` with the content + +.. code-block:: c + + #pragma once + + #define false 0 + #define true 1 + + #define bool int + +and put it in a place on the ``INCLUDE`` path, or create it in the local +directory and add ``.`` to the ``INCLUDE`` path:: + + SET INCLUDE=%INCLUDE%;. + +Then compile all the ``*.c`` file into ``*.obj``:: + + cl.exe /nologo /MD /O2 *c /c + rem for debug + cl.exe /nologo /MD /O0 /Ob0 /Zi *c /c + +You may need to move some variable declarations to the beginning of the +function, to be compliant with C89 standard. Here is the diff for version 2.2.4 + +.. code-block:: diff + + diff --git a/expat/lib/xmltok.c b/expat/lib/xmltok.c + index 007aed0..a2dcaad 100644 + --- a/expat/lib/xmltok.c + +++ b/expat/lib/xmltok.c + @@ -399,19 +399,21 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc), + /* Avoid copying partial characters (due to limited space). */ + const ptrdiff_t bytesAvailable = fromLim - *fromP; + const ptrdiff_t bytesStorable = toLim - *toP; + + const char * fromLimBefore; + + ptrdiff_t bytesToCopy; + if (bytesAvailable > bytesStorable) { + fromLim = *fromP + bytesStorable; + output_exhausted = true; + } + + /* Avoid copying partial characters (from incomplete input). */ + - const char * const fromLimBefore = fromLim; + + fromLimBefore = fromLim; + align_limit_to_full_utf8_characters(*fromP, &fromLim); + if (fromLim < fromLimBefore) { + input_incomplete = true; + } + + - const ptrdiff_t bytesToCopy = fromLim - *fromP; + + bytesToCopy = fromLim - *fromP; + memcpy((void *)*toP, (const void *)*fromP, (size_t)bytesToCopy); + *fromP += bytesToCopy; + *toP += bytesToCopy; + + +Create ``libexpat.lib`` (for translation) and ``libexpat.dll`` (for tests):: + + cl /LD *.obj libexpat.def /Felibexpat.dll + rem for debug + rem cl /LDd /Zi *.obj libexpat.def /Felibexpat.dll + + rem this will override the export library created in the step above + rem but tests do not need the export library, they load the dll dynamically + lib *.obj /out:libexpat.lib + +Then, copy + +- ``libexpat.lib`` into LIB +- both ``lib\expat.h`` and ``lib\expat_external.h`` in INCLUDE +- ``libexpat.dll`` into PATH The OpenSSL library @@ -363,7 +434,7 @@ It is probably not too much work if the goal is only to get a translated PyPy executable, and to run all tests before translation. But you need to start somewhere, and you should start with some tests in -rpython/translator/c/test/, like ``test_standalone.py`` and +``rpython/translator/c/test/``, like ``test_standalone.py`` and ``test_newgc.py``: try to have them pass on top of CPython64/64. Keep in mind that this runs small translations, and some details may go @@ -373,7 +444,7 @@ should be something like ``long long``. What is more generally needed is to review all the C files in -rpython/translator/c/src for the word ``long``, because this means a +``rpython/translator/c/src`` for the word ``long``, because this means a 32-bit integer even on Win64. Replace it with ``Signed`` most of the times. You can replace one with the other without breaking anything on any other platform, so feel free to. diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py --- a/pypy/interpreter/baseobjspace.py +++ b/pypy/interpreter/baseobjspace.py @@ -1445,7 +1445,7 @@ length = 1 return start, stop, step, length - def getindex_w(self, w_obj, w_exception, objdescr=None): + def getindex_w(self, w_obj, w_exception, objdescr=None, errmsg=None): """Return w_obj.__index__() as an RPython int. If w_exception is None, silently clamp in case of overflow; else raise w_exception. @@ -1455,8 +1455,10 @@ except OperationError as err: if objdescr is None or not err.match(self, self.w_TypeError): raise - raise oefmt(self.w_TypeError, "%s must be an integer, not %T", - objdescr, w_obj) + if errmsg is None: + errmsg = " must be an integer" + raise oefmt(self.w_TypeError, "%s%s, not %T", + objdescr, errmsg, w_obj) try: # allow_conversion=False it's not really necessary because the # return type of __index__ is already checked by space.index(), @@ -1632,12 +1634,15 @@ # return text_w(w_obj) or None return None if self.is_none(w_obj) else self.text_w(w_obj) + @specialize.argtype(1) def bytes_w(self, w_obj): """ Takes an application level :py:class:`bytes` (on PyPy2 this equals `str`) and returns a rpython byte string. """ + assert w_obj is not None return w_obj.str_w(self) + @specialize.argtype(1) def text_w(self, w_obj): """ PyPy2 takes either a :py:class:`str` and returns a rpython byte string, or it takes an :py:class:`unicode` @@ -1647,6 +1652,7 @@ On PyPy3 it takes a :py:class:`str` and it will return an utf-8 encoded rpython string. """ + assert w_obj is not None return w_obj.str_w(self) @not_rpython # tests only; should be replaced with bytes_w or text_w @@ -1689,12 +1695,14 @@ if len(string) != 1: raise oefmt(self.w_ValueError, "string must be of size 1") return string[0] - value = self.getindex_w(w_obj, None) + value = self.getindex_w(w_obj, None, "", + "an integer or string of size 1 is required") if not 0 <= value < 256: # this includes the OverflowError in case the long is too large raise oefmt(self.w_ValueError, "byte must be in range(0, 256)") return chr(value) + @specialize.argtype(1) def int_w(self, w_obj, allow_conversion=True): """ Unwrap an app-level int object into an interpret-level int. @@ -1707,26 +1715,35 @@ If allow_conversion=False, w_obj needs to be an app-level int or a subclass. """ + assert w_obj is not None return w_obj.int_w(self, allow_conversion) + @specialize.argtype(1) def int(self, w_obj): + assert w_obj is not None return w_obj.int(self) + @specialize.argtype(1) def uint_w(self, w_obj): + assert w_obj is not None return w_obj.uint_w(self) + @specialize.argtype(1) def bigint_w(self, w_obj, allow_conversion=True): """ Like int_w, but return a rlib.rbigint object and call __long__ if allow_conversion is True. """ + assert w_obj is not None return w_obj.bigint_w(self, allow_conversion) + @specialize.argtype(1) def float_w(self, w_obj, allow_conversion=True): """ Like int_w, but return an interp-level float and call __float__ if allow_conversion is True. """ + assert w_obj is not None return w_obj.float_w(self, allow_conversion) def realtext_w(self, w_obj): @@ -1739,6 +1756,7 @@ def utf8_w(self, w_obj): return w_obj.utf8_w(self) + @specialize.argtype(1) def unicode_w(self, w_obj): return self.utf8_w(w_obj).decode('utf8') @@ -1775,7 +1793,9 @@ # This is here mostly just for gateway.int_unwrapping_space_method(). return bool(self.int_w(w_obj)) + @specialize.argtype(1) def ord(self, w_obj): + assert w_obj is not None return w_obj.ord(self) # This is all interface for gateway.py. diff --git a/pypy/interpreter/test/test_typedef.py b/pypy/interpreter/test/test_typedef.py --- a/pypy/interpreter/test/test_typedef.py +++ b/pypy/interpreter/test/test_typedef.py @@ -419,3 +419,7 @@ def f(): return x assert f.__closure__[0].cell_contents is x + + def test_get_with_none_arg(self): + raises(TypeError, type.__dict__['__mro__'].__get__, None) + raises(TypeError, type.__dict__['__mro__'].__get__, None, None) diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py --- a/pypy/interpreter/typedef.py +++ b/pypy/interpreter/typedef.py @@ -297,6 +297,8 @@ if (space.is_w(w_obj, space.w_None) and not space.is_w(w_cls, space.type(space.w_None))): #print self, w_obj, w_cls + if space.is_w(w_cls, space.w_None): + raise oefmt(space.w_TypeError, "__get__(None, None) is invalid") return self else: try: diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.0" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/cffi1_module.py b/pypy/module/_cffi_backend/cffi1_module.py --- a/pypy/module/_cffi_backend/cffi1_module.py +++ b/pypy/module/_cffi_backend/cffi1_module.py @@ -1,4 +1,5 @@ from rpython.rtyper.lltypesystem import lltype, rffi +from rpython.rlib import jit from pypy.interpreter.error import oefmt from pypy.interpreter.module import Module @@ -15,7 +16,7 @@ INITFUNCPTR = lltype.Ptr(lltype.FuncType([rffi.VOIDPP], lltype.Void)) - + at jit.dont_look_inside def load_cffi1_module(space, name, path, initptr): # This is called from pypy.module.cpyext.api.load_extension_module() from pypy.module._cffi_backend.call_python import get_ll_cffi_call_python diff --git a/pypy/module/_cffi_backend/ctypeptr.py b/pypy/module/_cffi_backend/ctypeptr.py --- a/pypy/module/_cffi_backend/ctypeptr.py +++ b/pypy/module/_cffi_backend/ctypeptr.py @@ -156,10 +156,11 @@ class W_CTypePtrBase(W_CTypePtrOrArray): # base class for both pointers and pointers-to-functions - _attrs_ = ['is_void_ptr', 'is_voidchar_ptr'] - _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr'] + _attrs_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] + _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] is_void_ptr = False is_voidchar_ptr = False + is_onebyte_ptr = False def convert_to_object(self, cdata): ptrdata = rffi.cast(rffi.CCHARPP, cdata)[0] @@ -179,12 +180,20 @@ if self.is_void_ptr or other.is_void_ptr: pass # cast from or to 'void *' elif self.is_voidchar_ptr or other.is_voidchar_ptr: - space = self.space - msg = ("implicit cast from '%s' to '%s' " - "will be forbidden in the future (check that the types " - "are as you expect; use an explicit ffi.cast() if they " - "are correct)" % (other.name, self.name)) - space.warn(space.newtext(msg), space.w_UserWarning) + # for backward compatibility, accept "char *" as either + # source of target. This is not what C does, though, + # so emit a warning that will eventually turn into an + # error. The warning is turned off if both types are + # pointers to single bytes. + if self.is_onebyte_ptr and other.is_onebyte_ptr: + pass # no warning + else: + space = self.space + msg = ("implicit cast from '%s' to '%s' " + "will be forbidden in the future (check that the types " + "are as you expect; use an explicit ffi.cast() if they " + "are correct)" % (other.name, self.name)) + space.warn(space.newtext(msg), space.w_UserWarning) else: raise self._convert_error("compatible pointer", w_ob) @@ -214,6 +223,7 @@ self.is_void_ptr = isinstance(ctitem, ctypevoid.W_CTypeVoid) self.is_voidchar_ptr = (self.is_void_ptr or isinstance(ctitem, ctypeprim.W_CTypePrimitiveChar)) + self.is_onebyte_ptr = (ctitem.size == 1) W_CTypePtrBase.__init__(self, space, size, extra, 2, ctitem) def newp(self, w_init, allocator): diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.0", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): @@ -2099,7 +2099,8 @@ if sys.platform.startswith("linux"): BWChar = new_primitive_type("wchar_t") assert sizeof(BWChar) == 4 - assert int(cast(BWChar, -1)) == -1 # signed, on linux + # wchar_t is often signed on Linux, but not always (e.g. on ARM) + assert int(cast(BWChar, -1)) in (-1, 4294967295) def test_char16(): BChar16 = new_primitive_type("char16_t") @@ -3903,9 +3904,11 @@ BCharP = new_pointer_type(new_primitive_type("char")) BIntP = new_pointer_type(new_primitive_type("int")) BVoidP = new_pointer_type(new_void_type()) + BUCharP = new_pointer_type(new_primitive_type("unsigned char")) z1 = cast(BCharP, 0) z2 = cast(BIntP, 0) z3 = cast(BVoidP, 0) + z4 = cast(BUCharP, 0) with warnings.catch_warnings(record=True) as w: newp(new_pointer_type(BIntP), z1) # warn assert len(w) == 1 @@ -3919,6 +3922,12 @@ assert len(w) == 2 newp(new_pointer_type(BIntP), z3) # fine assert len(w) == 2 + newp(new_pointer_type(BCharP), z4) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z1) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z3) # fine + assert len(w) == 2 # check that the warnings are associated with lines in this file assert w[1].lineno == w[0].lineno + 4 diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,20 +66,18 @@ "position %d from error handler out of bounds", newpos) w_replace = space.convert_to_w_unicode(w_replace) - return w_replace._utf8.decode('utf8'), newpos + replace = w_replace._utf8.decode('utf8') + if decode: + return replace, newpos + else: + return replace, None, newpos return call_errorhandler def make_decode_errorhandler(self, space): return self._make_errorhandler(space, True) def make_encode_errorhandler(self, space): - errorhandler = self._make_errorhandler(space, False) - def encode_call_errorhandler(errors, encoding, reason, input, startpos, - endpos): - replace, newpos = errorhandler(errors, encoding, reason, input, - startpos, endpos) - return replace, None, newpos - return encode_call_errorhandler + return self._make_errorhandler(space, False) def get_unicodedata_handler(self, space): if self.unicodedata_handler: diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() From pypy.commits at gmail.com Thu Oct 12 16:04:48 2017 From: pypy.commits at gmail.com (wlav) Date: Thu, 12 Oct 2017 13:04:48 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: improve consistency PyPy <-> CPython Message-ID: <59dfcae0.c7341c0a.36d13.647f@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92742:eb3d59d5546b Date: 2017-10-12 12:54 -0700 http://bitbucket.org/pypy/pypy/changeset/eb3d59d5546b/ Log: improve consistency PyPy <-> CPython diff --git a/pypy/module/_cppyy/capi/loadable_capi.py b/pypy/module/_cppyy/capi/loadable_capi.py --- a/pypy/module/_cppyy/capi/loadable_capi.py +++ b/pypy/module/_cppyy/capi/loadable_capi.py @@ -217,7 +217,8 @@ 'method_req_args' : ([c_scope, c_index], c_int), 'method_arg_type' : ([c_scope, c_index, c_int], c_ccharp), 'method_arg_default' : ([c_scope, c_index, c_int], c_ccharp), - 'method_signature' : ([c_scope, c_index], c_ccharp), + 'method_signature' : ([c_scope, c_index, c_int], c_ccharp), + 'method_prototype' : ([c_scope, c_index, c_int], c_ccharp), 'method_is_template' : ([c_scope, c_index], c_int), 'method_num_template_args' : ([c_scope, c_index], c_int), @@ -498,9 +499,12 @@ def c_method_arg_default(space, cppscope, index, arg_index): args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(arg_index)] return charp2str_free(space, call_capi(space, 'method_arg_default', args)) -def c_method_signature(space, cppscope, index): - args = [_ArgH(cppscope.handle), _ArgL(index)] +def c_method_signature(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] return charp2str_free(space, call_capi(space, 'method_signature', args)) +def c_method_prototype(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] + return charp2str_free(space, call_capi(space, 'method_prototype', args)) def c_method_is_template(space, cppscope, index): args = [_ArgH(cppscope.handle), _ArgL(index)] diff --git a/pypy/module/_cppyy/include/capi.h b/pypy/module/_cppyy/include/capi.h --- a/pypy/module/_cppyy/include/capi.h +++ b/pypy/module/_cppyy/include/capi.h @@ -19,14 +19,15 @@ RPY_EXTERN int cppyy_num_scopes(cppyy_scope_t parent); RPY_EXTERN - char* cppyy_scope_name(cppyy_scope_t parent, int iscope); - + char* cppyy_scope_name(cppyy_scope_t parent, cppyy_index_t iscope); RPY_EXTERN char* cppyy_resolve_name(const char* cppitem_name); RPY_EXTERN cppyy_scope_t cppyy_get_scope(const char* scope_name); RPY_EXTERN cppyy_type_t cppyy_actual_class(cppyy_type_t klass, cppyy_object_t obj); + RPY_EXTERN + size_t cppyy_size_of(cppyy_type_t klass); /* memory management ------------------------------------------------------ */ RPY_EXTERN @@ -120,6 +121,8 @@ RPY_EXTERN char* cppyy_method_name(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN + char* cppyy_method_mangled_name(cppyy_scope_t scope, cppyy_index_t idx); + RPY_EXTERN char* cppyy_method_result_type(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN int cppyy_method_num_args(cppyy_scope_t scope, cppyy_index_t idx); @@ -130,7 +133,9 @@ RPY_EXTERN char* cppyy_method_arg_default(cppyy_scope_t scope, cppyy_index_t idx, int arg_index); RPY_EXTERN - char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx); + char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); + RPY_EXTERN + char* cppyy_method_prototype(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); RPY_EXTERN int cppyy_method_is_template(cppyy_scope_t scope, cppyy_index_t idx); @@ -147,8 +152,12 @@ /* method properties ------------------------------------------------------ */ RPY_EXTERN + int cppyy_is_publicmethod(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_constructor(cppyy_type_t type, cppyy_index_t idx); RPY_EXTERN + int cppyy_is_destructor(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_staticmethod(cppyy_type_t type, cppyy_index_t idx); /* data member reflection information ------------------------------------- */ diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -194,8 +194,13 @@ # check number of given arguments against required (== total - defaults) args_expected = len(self.arg_defs) args_given = len(args_w) - if args_expected < args_given or args_given < self.args_required: - raise oefmt(self.space.w_TypeError, "wrong number of arguments") + + if args_given < self.args_required: + raise oefmt(self.space.w_TypeError, + "takes at least %d arguments (%d given)", self.args_required, args_given) + elif args_expected < args_given: + raise oefmt(self.space.w_TypeError, + "takes at most %d arguments (%d given)", args_expected, args_given) # initial setup of converters, executors, and libffi (if available) if self.converters is None: @@ -381,8 +386,11 @@ conv.free_argument(self.space, rffi.cast(capi.C_OBJECT, arg_i), loc_i) capi.c_deallocate_function_args(self.space, args) - def signature(self): - return capi.c_method_signature(self.space, self.scope, self.index) + def signature(self, show_formalargs=True): + return capi.c_method_signature(self.space, self.scope, self.index, show_formalargs) + + def prototype(self, show_formalargs=True): + return capi.c_method_prototype(self.space, self.scope, self.index, show_formalargs) def priority(self): total_arg_priority = 0 @@ -396,7 +404,7 @@ lltype.free(self.cif_descr, flavor='raw') def __repr__(self): - return "CPPMethod: %s" % self.signature() + return "CPPMethod: %s" % self.prototype() def _freeze_(self): assert 0, "you should never have a pre-built instance of this!" @@ -412,7 +420,7 @@ return capi.C_NULL_OBJECT def __repr__(self): - return "CPPFunction: %s" % self.signature() + return "CPPFunction: %s" % self.prototype() class CPPTemplatedCall(CPPMethod): @@ -445,7 +453,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPTemplatedCall: %s" % self.signature() + return "CPPTemplatedCall: %s" % self.prototype() class CPPConstructor(CPPMethod): @@ -467,7 +475,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPConstructor: %s" % self.signature() + return "CPPConstructor: %s" % self.prototype() class CPPSetItem(CPPMethod): @@ -554,12 +562,12 @@ w_exc_type = e.w_type elif all_same_type and not e.match(self.space, w_exc_type): all_same_type = False - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' '+e.errorstr(self.space) except Exception as e: # can not special case this for non-overloaded functions as we anyway need an # OperationError error down from here - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' Exception: '+str(e) if all_same_type and w_exc_type is not None: @@ -567,20 +575,20 @@ else: raise OperationError(self.space.w_TypeError, self.space.newtext(errmsg)) - def signature(self): - sig = self.functions[0].signature() + def prototype(self): + sig = self.functions[0].prototype() for i in range(1, len(self.functions)): - sig += '\n'+self.functions[i].signature() + sig += '\n'+self.functions[i].prototype() return self.space.newtext(sig) def __repr__(self): - return "W_CPPOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPOverload(%s)" % [f.prototype() for f in self.functions] W_CPPOverload.typedef = TypeDef( 'CPPOverload', is_static = interp2app(W_CPPOverload.is_static), call = interp2app(W_CPPOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPOverload.prototype), ) @@ -609,13 +617,13 @@ memory_regulator.register(cppinstance) def __repr__(self): - return "W_CPPConstructorOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPConstructorOverload(%s)" % [f.prototype() for f in self.functions] W_CPPConstructorOverload.typedef = TypeDef( 'CPPConstructorOverload', is_static = interp2app(W_CPPConstructorOverload.is_static), call = interp2app(W_CPPConstructorOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPOverload.prototype), ) @@ -760,7 +768,7 @@ overload = self.get_overload(name) sig = '(%s)' % signature for f in overload.functions: - if 0 < f.signature().find(sig): + if f.signature(False) == sig: return W_CPPOverload(self.space, self, [f]) raise oefmt(self.space.w_LookupError, "no overload matches signature") diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -116,7 +116,7 @@ def function(*args): return cppol.call(None, *args) function.__name__ = func_name - function.__doc__ = cppol.signature() + function.__doc__ = cppol.prototype() return staticmethod(function) @@ -165,7 +165,7 @@ def method(self, *args): return cppol.call(self, *args) method.__name__ = meth_name - method.__doc__ = cppol.signature() + method.__doc__ = cppol.prototype() return method def make_cppclass(scope, cl_name, decl): diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py --- a/pypy/module/_cppyy/test/test_fragile.py +++ b/pypy/module/_cppyy/test/test_fragile.py @@ -163,20 +163,23 @@ assert 0 except TypeError as e: assert "fragile::D::check()" in str(e) - assert "TypeError: wrong number of arguments" in str(e) + assert "TypeError: takes at most 0 arguments (1 given)" in str(e) + assert "TypeError: takes at least 2 arguments (1 given)" in str(e) try: d.overload(None) # raises TypeError assert 0 except TypeError as e: + # TODO: pypy-c does not indicate which argument failed to convert, CPython does + # likewise there are still minor differences in descriptiveness of messages assert "fragile::D::overload()" in str(e) - assert "TypeError: wrong number of arguments" in str(e) + assert "TypeError: takes at most 0 arguments (1 given)" in str(e) assert "fragile::D::overload(fragile::no_such_class*)" in str(e) - assert "TypeError: no converter available for 'fragile::no_such_class*'" in str(e) - assert "fragile::D::overload(char, int)" in str(e) - assert "TypeError: expected string, got NoneType object" in str(e) - assert "fragile::D::overload(int, fragile::no_such_class*)" in str(e) - assert "TypeError: expected integer, got NoneType object" in str(e) + #assert "no converter available for 'fragile::no_such_class*'" in str(e) + assert "void fragile::D::overload(char, int i = 0)" in str(e) + #assert "char or small int type expected" in str(e) + assert "void fragile::D::overload(int, fragile::no_such_class* p = 0)" in str(e) + #assert "int/long conversion expects an integer object" in str(e) j = fragile.J() assert fragile.J.method1.__doc__ == j.method1.__doc__ From pypy.commits at gmail.com Fri Oct 13 08:45:03 2017 From: pypy.commits at gmail.com (arigo) Date: Fri, 13 Oct 2017 05:45:03 -0700 (PDT) Subject: [pypy-commit] cffi default: Document for issue 340 Message-ID: <59e0b54f.4f091c0a.84a92.883c@mx.google.com> Author: Armin Rigo Branch: Changeset: r3050:fa84d0ccb9f4 Date: 2017-10-13 14:44 +0200 http://bitbucket.org/cffi/cffi/changeset/fa84d0ccb9f4/ Log: Document for issue 340 diff --git a/doc/source/ref.rst b/doc/source/ref.rst --- a/doc/source/ref.rst +++ b/doc/source/ref.rst @@ -401,7 +401,12 @@ nothing gets out of sync if the size estimates do not match. It only makes the next GC start more or less early. +Note that if you have several ``ffi.gc()`` objects, the corresponding +destructors will be called in a random order. If you need a particular +order, see the discussion in `issue 340`__. + .. __: http://bugs.python.org/issue31105 +.. __: https://bitbucket.org/cffi/cffi/issues/340/resources-release-issues .. _ffi-new-handle: From pypy.commits at gmail.com Fri Oct 13 09:10:30 2017 From: pypy.commits at gmail.com (arigo) Date: Fri, 13 Oct 2017 06:10:30 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fix Message-ID: <59e0bb46.90bbdf0a.13098.8139@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92743:b9d04d54fecd Date: 2017-10-13 08:38 +0200 http://bitbucket.org/pypy/pypy/changeset/b9d04d54fecd/ Log: fix diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -487,7 +487,8 @@ @unwrap_spec(data='bufferstr', errors='text_or_none', byteorder=int, w_final=WrappedDefault(False)) def utf_16_ex_decode(space, data, errors='strict', byteorder=0, w_final=None): - assert False, "fix in the future" + from pypy.interpreter.unicodehelper import DecodeWrapper + if errors is None: errors = 'strict' final = space.is_true(w_final) @@ -502,14 +503,16 @@ if final: consumed = 0 res, consumed, byteorder = runicode.str_decode_utf_16_helper( - data, len(data), errors, final, state.decode_error_handler, byteorder) + data, len(data), errors, final, + DecodeWrapper(state.decode_error_handler).handle, byteorder) return space.newtuple([space.newunicode(res), space.newint(consumed), space.newint(byteorder)]) @unwrap_spec(data='bufferstr', errors='text_or_none', byteorder=int, w_final=WrappedDefault(False)) def utf_32_ex_decode(space, data, errors='strict', byteorder=0, w_final=None): - assert False, "fix in the future" + from pypy.interpreter.unicodehelper import DecodeWrapper + final = space.is_true(w_final) state = space.fromcache(CodecState) if byteorder == 0: @@ -522,7 +525,8 @@ if final: consumed = 0 res, consumed, byteorder = runicode.str_decode_utf_32_helper( - data, len(data), errors, final, state.decode_error_handler, byteorder) + data, len(data), errors, final, + DecodeWrapper(state.decode_error_handler).handle, byteorder) return space.newtuple([space.newunicode(res), space.newint(consumed), space.newint(byteorder)]) From pypy.commits at gmail.com Fri Oct 13 09:10:32 2017 From: pypy.commits at gmail.com (arigo) Date: Fri, 13 Oct 2017 06:10:32 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Fix for module/_codecs/test/test_ztranslation Message-ID: <59e0bb48.f586df0a.31717.6b2f@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92744:29bfc6b969b9 Date: 2017-10-13 15:09 +0200 http://bitbucket.org/pypy/pypy/changeset/29bfc6b969b9/ Log: Fix for module/_codecs/test/test_ztranslation diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py --- a/pypy/interpreter/unicodehelper.py +++ b/pypy/interpreter/unicodehelper.py @@ -32,6 +32,7 @@ def convert_arg_to_w_unicode(space, w_arg, strict=None): from pypy.objspace.std.unicodeobject import W_UnicodeObject + assert not hasattr(space, 'is_fake_objspace') return W_UnicodeObject.convert_arg_to_w_unicode(space, w_arg, strict) # ____________________________________________________________ From pypy.commits at gmail.com Fri Oct 13 13:03:11 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 13 Oct 2017 10:03:11 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: implement _Py_NewReference directly in C Message-ID: <59e0f1cf.55ad1c0a.cbd26.2682@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92745:888c6f93963d Date: 2017-10-13 18:02 +0100 http://bitbucket.org/pypy/pypy/changeset/888c6f93963d/ Log: implement _Py_NewReference directly in C diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -65,6 +65,7 @@ #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) #define Py_SIZE(ob) (((PyVarObject*)(ob))->ob_size) +#define _Py_NewReference(ob) (((PyObject *)(ob))->ob_refcnt = 1) #define _Py_ForgetReference(ob) /* nothing */ #define Py_None (&_Py_NoneStruct) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -352,14 +352,6 @@ # assert pyobj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY - at cpython_api([PyObject], lltype.Void) -def _Py_NewReference(space, obj): - obj.c_ob_refcnt = 1 - # XXX is it always useful to create the W_Root object here? - w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type)) - assert isinstance(w_type, W_TypeObject) - get_typedescr(w_type.layout.typedef).realize(space, obj) - @init_function def write_w_marker_deallocating(space): if we_are_translated(): From pypy.commits at gmail.com Fri Oct 13 13:03:13 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 13 Oct 2017 10:03:13 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads Message-ID: <59e0f1d1.d79d1c0a.7a137.466c@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92746:5b20c5296b9f Date: 2017-10-13 18:02 +0100 http://bitbucket.org/pypy/pypy/changeset/5b20c5296b9f/ Log: merge heads diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -633,8 +633,10 @@ '_Py_QnewFlag', 'Py_Py3kWarningFlag', 'Py_HashRandomizationFlag', '_Py_PackageContext', '_PyTraceMalloc_Track', '_PyTraceMalloc_Untrack', 'PyMem_Malloc', 'Py_IncRef', 'Py_DecRef', 'PyObject_Free', 'PyObject_GC_Del', 'PyType_GenericAlloc', - '_PyObject_New', '_PyObject_NewVar', '_PyObject_GC_New', + '_PyObject_New', '_PyObject_NewVar', + '_PyObject_GC_New', '_PyObject_GC_NewVar', 'PyObject_Init', 'PyObject_InitVar', 'PyInt_FromLong', + 'PyTuple_New', ] TYPES = {} FORWARD_DECLS = [] @@ -1135,12 +1137,20 @@ [rffi.LONG], PyObject, compilation_info=eci, _nowrapper=True) + state.C._PyPy_int_dealloc = rffi.llexternal( + '_PyPy_int_dealloc', [PyObject], lltype.Void, + compilation_info=eci, _nowrapper=True) + state.C.PyTuple_New = rffi.llexternal( + mangle_name(prefix, 'PyTuple_New'), + [Py_ssize_t], PyObject, + compilation_info=eci, + _nowrapper=True) + state.C._PyPy_tuple_dealloc = rffi.llexternal( + '_PyPy_tuple_dealloc', [PyObject], lltype.Void, + compilation_info=eci, _nowrapper=True) _, state.C.set_marker = rffi.CExternVariable( Py_ssize_t, '_pypy_rawrefcount_w_marker_deallocating', eci, _nowrapper=True, c_type='Py_ssize_t') - state.C._PyPy_int_dealloc = rffi.llexternal( - '_PyPy_int_dealloc', [PyObject], lltype.Void, - compilation_info=eci, _nowrapper=True) state.C._PyPy_subtype_dealloc = rffi.llexternal( '_PyPy_subtype_dealloc', [PyObject], lltype.Void, compilation_info=eci, _nowrapper=True) @@ -1258,7 +1268,6 @@ in_dll = ll2ctypes.get_ctypes_type(PyObject.TO).in_dll(bridge, mname) py_obj = ll2ctypes.ctypes2lltype(PyObject, ctypes.pointer(in_dll)) builder.prepare(py_obj, w_obj) - builder.attach_all(space) pypyAPI = ctypes.POINTER(ctypes.c_void_p).in_dll(bridge, 'pypyAPI') @@ -1269,6 +1278,12 @@ ll2ctypes.lltype2ctypes(func.get_llhelper(space)), ctypes.c_void_p) + # we need to call this *after* the init code above, because it might + # indirectly call some functions which are attached to pypyAPI (e.g., we + # if do tuple_attach of the prebuilt empty tuple, we need to call + # _PyPy_Malloc) + builder.attach_all(space) + setup_init_functions(eci, prefix) return modulename.new(ext='') @@ -1464,6 +1479,7 @@ source_dir / "object.c", source_dir / "typeobject.c", source_dir / "intobject.c", + source_dir / "tupleobject.c", ] def build_eci(code, use_micronumpy=False, translating=False): diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -291,6 +291,9 @@ #define PyObject_GC_New(type, typeobj) \ ( (type *) _PyObject_GC_New(typeobj) ) +#define PyObject_GC_NewVar(type, typeobj, n) \ + ( (type *) _PyObject_GC_NewVar((typeobj), (n)) ) + /* A dummy PyGC_Head, just to please some tests. Don't use it! */ typedef union _gc_head { char dummy; @@ -346,6 +349,7 @@ PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *); PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t); PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *); +PyAPI_FUNC(PyVarObject *) _PyObject_GC_NewVar(PyTypeObject *, Py_ssize_t); PyAPI_FUNC(PyObject *) PyObject_Init(PyObject *, PyTypeObject *); PyAPI_FUNC(PyVarObject *) PyObject_InitVar(PyVarObject *, diff --git a/pypy/module/cpyext/include/tupleobject.h b/pypy/module/cpyext/include/tupleobject.h --- a/pypy/module/cpyext/include/tupleobject.h +++ b/pypy/module/cpyext/include/tupleobject.h @@ -16,9 +16,13 @@ */ } PyTupleObject; +PyAPI_FUNC(PyObject *) PyTuple_New(Py_ssize_t size); +PyAPI_FUNC(void) _PyPy_tuple_dealloc(PyObject *); + /* defined in varargswrapper.c */ PyAPI_FUNC(PyObject *) PyTuple_Pack(Py_ssize_t, ...); + /* Macro, trading safety for speed */ #define PyTuple_GET_ITEM(op, i) (((PyTupleObject *)(op))->ob_item[i]) #define PyTuple_GET_SIZE(op) Py_SIZE(op) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -68,6 +68,10 @@ return _PyObject_New(type); } +PyVarObject * _PyObject_GC_NewVar(PyTypeObject *type, Py_ssize_t nitems) +{ + return _PyObject_NewVar(type, nitems); +} static PyObject * _generic_alloc(PyTypeObject *type, Py_ssize_t nitems) diff --git a/pypy/module/cpyext/src/tupleobject.c b/pypy/module/cpyext/src/tupleobject.c new file mode 100644 --- /dev/null +++ b/pypy/module/cpyext/src/tupleobject.c @@ -0,0 +1,98 @@ + +/* Tuple object implementation, stolen&adapted from CPython */ + +#include "Python.h" + +/* Speed optimization to avoid frequent malloc/free of small tuples */ +#ifndef PyTuple_MAXSAVESIZE +#define PyTuple_MAXSAVESIZE 20 /* Largest tuple to save on free list */ +#endif +#ifndef PyTuple_MAXFREELIST +#define PyTuple_MAXFREELIST 2000 /* Maximum number of tuples of each size to save */ +#endif + +#if PyTuple_MAXSAVESIZE > 0 +/* Entries 1 up to PyTuple_MAXSAVESIZE are free lists, entry 0 is the empty + tuple () of which at most one instance will be allocated. +*/ +static PyTupleObject *free_list[PyTuple_MAXSAVESIZE]; +static int numfree[PyTuple_MAXSAVESIZE]; +#endif + +PyObject * +PyTuple_New(register Py_ssize_t size) +{ + register PyTupleObject *op; + Py_ssize_t i; + if (size < 0) { + PyErr_BadInternalCall(); + return NULL; + } +#if PyTuple_MAXSAVESIZE > 0 + if (size == 0 && free_list[0]) { + op = free_list[0]; + Py_INCREF(op); + return (PyObject *) op; + } + if (size < PyTuple_MAXSAVESIZE && (op = free_list[size]) != NULL) { + free_list[size] = (PyTupleObject *) op->ob_item[0]; + numfree[size]--; + _Py_NewReference((PyObject *)op); + } + else +#endif + { + Py_ssize_t nbytes = size * sizeof(PyObject *); + /* Check for overflow */ + if (nbytes / sizeof(PyObject *) != (size_t)size || + (nbytes > PY_SSIZE_T_MAX - sizeof(PyTupleObject) - sizeof(PyObject *))) + { + return PyErr_NoMemory(); + } + + op = PyObject_GC_NewVar(PyTupleObject, &PyTuple_Type, size); + if (op == NULL) + return NULL; + } + for (i=0; i < size; i++) + op->ob_item[i] = NULL; +#if PyTuple_MAXSAVESIZE > 0 + if (size == 0) { + free_list[0] = op; + ++numfree[0]; + Py_INCREF(op); /* extra INCREF so that this is never freed */ + } +#endif + _PyObject_GC_TRACK(op); + return (PyObject *) op; +} + +/* this is CPython's tupledealloc */ +void +_PyPy_tuple_dealloc(register PyObject *_op) +{ + register PyTupleObject *op = (PyTupleObject *)_op; + register Py_ssize_t i; + register Py_ssize_t len = Py_SIZE(op); + PyObject_GC_UnTrack(op); + Py_TRASHCAN_SAFE_BEGIN(op) + if (len > 0) { + i = len; + while (--i >= 0) + Py_XDECREF(op->ob_item[i]); +#if PyTuple_MAXSAVESIZE > 0 + if (len < PyTuple_MAXSAVESIZE && + numfree[len] < PyTuple_MAXFREELIST && + Py_TYPE(op) == &PyTuple_Type) + { + op->ob_item[0] = (PyObject *) free_list[len]; + numfree[len]++; + free_list[len] = op; + goto done; /* return */ + } +#endif + } + Py_TYPE(op)->tp_free((PyObject *)op); +done: + Py_TRASHCAN_SAFE_END(op) +} diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py --- a/pypy/module/cpyext/test/test_api.py +++ b/pypy/module/cpyext/test/test_api.py @@ -25,6 +25,10 @@ cls.preload_builtins(space) class CAPI: + def __repr__(self): + return '<%s.%s instance>' % (self.__class__.__module__, + self.__class__.__name__) + def __getattr__(self, name): return getattr(cls.space, name) cls.api = CAPI() diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -7,7 +7,7 @@ from rpython.rlib.debug import FatalError from pypy.module.cpyext.tupleobject import ( PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size) - +from pypy.module.cpyext.state import State class TestTupleObject(BaseApiTest): @@ -22,17 +22,19 @@ PyTuple_Size(space, space.newlist([])) def test_tuple_realize_refuses_nulls(self, space, api): - py_tuple = api.PyTuple_New(1) + state = space.fromcache(State) + py_tuple = state.C.PyTuple_New(1) py.test.raises(FatalError, from_ref, space, py_tuple) decref(space, py_tuple) def test_tuple_resize(self, space, api): + state = space.fromcache(State) w_42 = space.wrap(42) w_43 = space.wrap(43) w_44 = space.wrap(44) ar = lltype.malloc(PyObjectP.TO, 1, flavor='raw') - py_tuple = api.PyTuple_New(3) + py_tuple = state.C.PyTuple_New(3) # inside py_tuple is an array of "PyObject *" items which each hold # a reference rffi.cast(PyTupleObject, py_tuple).c_ob_item[0] = make_ref(space, w_42) @@ -45,7 +47,7 @@ assert space.int_w(space.getitem(w_tuple, space.wrap(1))) == 43 decref(space, ar[0]) - py_tuple = api.PyTuple_New(3) + py_tuple = state.C.PyTuple_New(3) rffi.cast(PyTupleObject, py_tuple).c_ob_item[0] = make_ref(space, w_42) rffi.cast(PyTupleObject, py_tuple).c_ob_item[1] = make_ref(space, w_43) rffi.cast(PyTupleObject, py_tuple).c_ob_item[2] = make_ref(space, w_44) @@ -64,7 +66,8 @@ lltype.free(ar, flavor='raw') def test_setitem(self, space, api): - py_tuple = api.PyTuple_New(2) + state = space.fromcache(State) + py_tuple = state.C.PyTuple_New(2) api.PyTuple_SetItem(py_tuple, 0, make_ref(space, space.wrap(42))) api.PyTuple_SetItem(py_tuple, 1, make_ref(space, space.wrap(43))) diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -7,6 +7,7 @@ from pypy.module.cpyext.pyobject import ( PyObject, PyObjectP, make_ref, from_ref, decref, incref, track_reference, make_typedescr, get_typedescr, pyobj_has_w_obj) +from pypy.module.cpyext.state import State from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.objspace.std.tupleobject import W_TupleObject @@ -36,10 +37,12 @@ @bootstrap_function def init_tupleobject(space): "Type description of PyTupleObject" + state = space.fromcache(State) make_typedescr(space.w_tuple.layout.typedef, basestruct=PyTupleObject.TO, attach=tuple_attach, - dealloc=tuple_dealloc, + alloc=tuple_alloc, + dealloc=state.C._PyPy_tuple_dealloc, realize=tuple_realize) PyTuple_Check, PyTuple_CheckExact = build_type_checkers_flags("Tuple") @@ -49,12 +52,20 @@ return (w_type is space.w_tuple or space.issubtype_w(w_type, space.w_tuple)) +def tuple_alloc(typedescr, space, w_type, itemcount): + state = space.fromcache(State) + if w_type is space.w_tuple: + return state.C.PyTuple_New(itemcount) + else: + return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) + def new_empty_tuple(space, length): """ Allocate a PyTupleObject and its array of PyObject *, but without a corresponding interpreter object. The array may be mutated, until tuple_realize() is called. Refcount of the result is 1. """ + assert False, 'kill this?' typedescr = get_typedescr(space.w_tuple.layout.typedef) py_obj = typedescr.allocate(space, space.w_tuple, length) py_tup = rffi.cast(PyTupleObject, py_obj) @@ -113,22 +124,22 @@ track_reference(space, py_obj, w_obj) return w_obj - at slot_function([PyObject], lltype.Void) -def tuple_dealloc(space, py_obj): - """Frees allocated PyTupleObject resources. - """ - py_tup = rffi.cast(PyTupleObject, py_obj) - p = py_tup.c_ob_item - for i in range(py_tup.c_ob_size): - decref(space, p[i]) - from pypy.module.cpyext.object import _dealloc - _dealloc(space, py_obj) +## @slot_function([PyObject], lltype.Void) +## def tuple_dealloc(space, py_obj): +## """Frees allocated PyTupleObject resources. +## """ +## py_tup = rffi.cast(PyTupleObject, py_obj) +## p = py_tup.c_ob_item +## for i in range(py_tup.c_ob_size): +## decref(space, p[i]) +## from pypy.module.cpyext.object import _dealloc +## _dealloc(space, py_obj) #_______________________________________________________________________ - at cpython_api([Py_ssize_t], PyObject, result_is_ll=True) -def PyTuple_New(space, size): - return new_empty_tuple(space, size) +## @cpython_api([Py_ssize_t], PyObject, result_is_ll=True) +## def PyTuple_New(space, size): +## return new_empty_tuple(space, size) @cpython_api([PyObject, Py_ssize_t, PyObject], rffi.INT_real, error=-1) def PyTuple_SetItem(space, ref, index, py_obj): From pypy.commits at gmail.com Fri Oct 13 13:15:33 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 13 Oct 2017 10:15:33 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: kill new_empty_tuple and fix PyTuple_Resize Message-ID: <59e0f4b5.4e301c0a.d96f7.69b4@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92747:014910472387 Date: 2017-10-13 19:06 +0200 http://bitbucket.org/pypy/pypy/changeset/014910472387/ Log: kill new_empty_tuple and fix PyTuple_Resize diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -59,21 +59,6 @@ else: return BaseCpyTypedescr.allocate(typedescr, space, w_type, itemcount) -def new_empty_tuple(space, length): - """ - Allocate a PyTupleObject and its array of PyObject *, but without a - corresponding interpreter object. The array may be mutated, until - tuple_realize() is called. Refcount of the result is 1. - """ - assert False, 'kill this?' - typedescr = get_typedescr(space.w_tuple.layout.typedef) - py_obj = typedescr.allocate(space, space.w_tuple, length) - py_tup = rffi.cast(PyTupleObject, py_obj) - p = py_tup.c_ob_item - for i in range(py_tup.c_ob_size): - p[i] = lltype.nullptr(PyObject.TO) - return py_obj - def tuple_attach(space, py_obj, w_obj, w_userdata=None): """ Fills a newly allocated PyTupleObject with the given tuple object. The @@ -124,22 +109,6 @@ track_reference(space, py_obj, w_obj) return w_obj -## @slot_function([PyObject], lltype.Void) -## def tuple_dealloc(space, py_obj): -## """Frees allocated PyTupleObject resources. -## """ -## py_tup = rffi.cast(PyTupleObject, py_obj) -## p = py_tup.c_ob_item -## for i in range(py_tup.c_ob_size): -## decref(space, p[i]) -## from pypy.module.cpyext.object import _dealloc -## _dealloc(space, py_obj) - -#_______________________________________________________________________ - -## @cpython_api([Py_ssize_t], PyObject, result_is_ll=True) -## def PyTuple_New(space, size): -## return new_empty_tuple(space, size) @cpython_api([PyObject, Py_ssize_t, PyObject], rffi.INT_real, error=-1) def PyTuple_SetItem(space, ref, index, py_obj): @@ -194,12 +163,13 @@ this function. If the object referenced by *p is replaced, the original *p is destroyed. On failure, returns -1 and sets *p to NULL, and raises MemoryError or SystemError.""" + state = space.fromcache(State) ref = p_ref[0] if not tuple_check_ref(space, ref): PyErr_BadInternalCall(space) oldref = rffi.cast(PyTupleObject, ref) oldsize = oldref.c_ob_size - p_ref[0] = new_empty_tuple(space, newsize) + p_ref[0] = state.C.PyTuple_New(newsize) newref = rffi.cast(PyTupleObject, p_ref[0]) try: if oldsize < newsize: From pypy.commits at gmail.com Fri Oct 13 17:30:33 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 13 Oct 2017 14:30:33 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: fix this test, now that the empty tuple is cached also on pypy; fix also the type of initial_item_refcount, because int is not large enough to contain REFCNT_FROM_PYPY Message-ID: <59e13079.11b51c0a.3059d.e735@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92748:cfae668eb6e9 Date: 2017-10-13 23:25 +0200 http://bitbucket.org/pypy/pypy/changeset/cfae668eb6e9/ Log: fix this test, now that the empty tuple is cached also on pypy; fix also the type of initial_item_refcount, because int is not large enough to contain REFCNT_FROM_PYPY diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -90,13 +90,9 @@ """ PyObject *item = PyTuple_New(0); PyObject *t = PyTuple_New(1); -#ifdef PYPY_VERSION - // PyPy starts even empty tuples with a refcount of 1. - const int initial_item_refcount = 1; -#else - // CPython can cache (). - const int initial_item_refcount = item->ob_refcnt; -#endif // PYPY_VERSION + // the empty tuple is cached, so we can't assume its refcnt is 1 + const Py_ssize_t initial_item_refcount = item->ob_refcnt; + if (t->ob_refcnt != 1 || item->ob_refcnt != initial_item_refcount) { PyErr_SetString(PyExc_SystemError, "bad initial refcnt"); return NULL; From pypy.commits at gmail.com Sat Oct 14 05:43:23 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:23 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Revert some changes that are not needed any more here Message-ID: <59e1dc3b.2fb9df0a.8d596.fe35@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92749:e6e4622c8bc2 Date: 2017-10-14 07:04 +0200 http://bitbucket.org/pypy/pypy/changeset/e6e4622c8bc2/ Log: Revert some changes that are not needed any more here diff --git a/pypy/module/_multibytecodec/c_codecs.py b/pypy/module/_multibytecodec/c_codecs.py --- a/pypy/module/_multibytecodec/c_codecs.py +++ b/pypy/module/_multibytecodec/c_codecs.py @@ -3,7 +3,7 @@ from rpython.translator.tool.cbuild import ExternalCompilationInfo from rpython.translator import cdir -UNICODE_REPLACEMENT_CHARACTER = u'\uFFFD'.encode("utf8") +UNICODE_REPLACEMENT_CHARACTER = u'\uFFFD' class EncodeDecodeError(Exception): @@ -148,17 +148,16 @@ if errors == "strict": raise EncodeDecodeError(start, end, reason) elif errors == "ignore": - replace = "" - lgt = 0 + replace = u"" elif errors == "replace": replace = UNICODE_REPLACEMENT_CHARACTER - lgt = 1 else: assert errorcb - replace, end, lgt = errorcb(errors, namecb, reason, + replace, end = errorcb(errors, namecb, reason, stringdata, start, end) - with rffi.scoped_nonmoving_unicodebuffer(replace.decode("utf8")) as inbuf: - r = pypy_cjk_dec_replace_on_error(decodebuf, inbuf, lgt, end) + # 'replace' is RPython unicode here + with rffi.scoped_nonmoving_unicodebuffer(replace) as inbuf: + r = pypy_cjk_dec_replace_on_error(decodebuf, inbuf, len(replace), end) if r == MBERR_NOMEMORY: raise MemoryError @@ -257,7 +256,7 @@ replace = "?" else: assert errorcb - retu, rets, end, lgt = errorcb(errors, namecb, reason, + retu, rets, end = errorcb(errors, namecb, reason, unicodedata.encode("utf8"), start, end) if rets is not None: # py3k only diff --git a/pypy/module/_multibytecodec/interp_incremental.py b/pypy/module/_multibytecodec/interp_incremental.py --- a/pypy/module/_multibytecodec/interp_incremental.py +++ b/pypy/module/_multibytecodec/interp_incremental.py @@ -96,24 +96,25 @@ c_codecs.pypy_cjk_enc_free(self.encodebuf) self.encodebuf = lltype.nullptr(c_codecs.ENCODEBUF_P.TO) - @unwrap_spec(utf8object='utf8', final=bool) - def encode_w(self, utf8object, objlen, final=False): - object = utf8object.decode('utf8') + @unwrap_spec(object='utf8', final=bool) + def encode_w(self, object, final=False): + u_object = object.decode('utf8') space = self.space state = space.fromcache(CodecState) if len(self.pending) > 0: - object = self.pending + object + u_object = self.pending + u_object try: - output = c_codecs.encodeex(self.encodebuf, object, self.errors, + output = c_codecs.encodeex(self.encodebuf, u_object, self.errors, state.encode_error_handler, self.name, get_ignore_error(final)) except c_codecs.EncodeDecodeError as e: - raise wrap_unicodeencodeerror(space, e, utf8object, self.name) + raise wrap_unicodeencodeerror(space, e, object, len(u_object), + self.name) except RuntimeError: raise wrap_runtimeerror(space) pos = c_codecs.pypy_cjk_enc_inbuf_consumed(self.encodebuf) - assert 0 <= pos <= len(object) - self.pending = object[pos:] + assert 0 <= pos <= len(u_object) + self.pending = u_object[pos:] return space.newbytes(output) diff --git a/pypy/module/_multibytecodec/interp_multibytecodec.py b/pypy/module/_multibytecodec/interp_multibytecodec.py --- a/pypy/module/_multibytecodec/interp_multibytecodec.py +++ b/pypy/module/_multibytecodec/interp_multibytecodec.py @@ -28,20 +28,22 @@ space.newint(len(input))]) @unwrap_spec(input='utf8', errors="text_or_none") - def encode(self, space, input, inputlen, errors=None): + def encode(self, space, input, errors=None): if errors is None: errors = 'strict' state = space.fromcache(CodecState) # + u_input = input.decode('utf8') try: - output = c_codecs.encode(self.codec, input.decode('utf8'), errors, + output = c_codecs.encode(self.codec, u_input, errors, state.encode_error_handler, self.name) except c_codecs.EncodeDecodeError as e: - raise wrap_unicodeencodeerror(space, e, input, self.name) + raise wrap_unicodeencodeerror(space, e, input, len(u_input), + self.name) except RuntimeError: raise wrap_runtimeerror(space) return space.newtuple([space.newbytes(output), - space.newint(inputlen)]) + space.newint(len(u_input))]) MultibyteCodec.typedef = TypeDef( @@ -71,12 +73,12 @@ space.newint(e.end), space.newtext(e.reason)])) -def wrap_unicodeencodeerror(space, e, input, name): +def wrap_unicodeencodeerror(space, e, input, inputlen, name): raise OperationError( space.w_UnicodeEncodeError, space.newtuple([ space.newtext(name), - space.newutf8(input, -1), + space.newutf8(input, inputlen), space.newint(e.start), space.newint(e.end), space.newtext(e.reason)])) diff --git a/pypy/module/_multibytecodec/test/test_c_codecs.py b/pypy/module/_multibytecodec/test/test_c_codecs.py --- a/pypy/module/_multibytecodec/test/test_c_codecs.py +++ b/pypy/module/_multibytecodec/test/test_c_codecs.py @@ -126,6 +126,6 @@ def test_encode_custom_error_handler_bytes(): c = getcodec("hz") def errorhandler(errors, enc, msg, t, startingpos, endingpos): - return None, '\xc3', endingpos, -1 + return None, '\xc3', endingpos s = encode(c, u'abc\u1234def', 'foo', errorhandler) assert '\xc3' in s From pypy.commits at gmail.com Sat Oct 14 05:43:25 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:25 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Tweak for the JIT, similar to W_UnicodeObject in the py3.5 branch Message-ID: <59e1dc3d.93d31c0a.6fd06.7e97@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92750:40990369b37a Date: 2017-10-14 07:15 +0200 http://bitbucket.org/pypy/pypy/changeset/40990369b37a/ Log: Tweak for the JIT, similar to W_UnicodeObject in the py3.5 branch diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -893,10 +893,11 @@ descr_rmul = descr_mul def _get_index_storage(self): - if self._index_storage == rutf8.null_storage(): - self._index_storage = rutf8.create_utf8_index_storage(self._utf8, - self._length) - return self._index_storage + storage = jit.conditional_call_elidable(self._index_storage, + rutf8.create_utf8_index_storage, self._utf8, self._length) + if not jit.isconstant(self): + self._index_storage = storage + return storage def _getitem_result(self, space, index): if index < 0: diff --git a/rpython/rlib/jit.py b/rpython/rlib/jit.py --- a/rpython/rlib/jit.py +++ b/rpython/rlib/jit.py @@ -1206,7 +1206,7 @@ def conditional_call_elidable(value, function, *args): """Does the same as: - if value == <0 or None>: + if value == <0 or None or NULL>: value = function(*args) return value @@ -1236,7 +1236,7 @@ value = function(*args) assert isinstance(value, int) else: - if value is None: + if not value: value = function(*args) assert not isinstance(value, int) return value From pypy.commits at gmail.com Sat Oct 14 05:43:30 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:30 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Rewrite unicode.index/find to use rutf8.codepoints_in_utf8(). It should Message-ID: <59e1dc42.472e1c0a.532db.d33c@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92752:6bc169e4d079 Date: 2017-10-14 10:21 +0200 http://bitbucket.org/pypy/pypy/changeset/6bc169e4d079/ Log: Rewrite unicode.index/find to use rutf8.codepoints_in_utf8(). It should ensure that the complexity of these methods is always correct. Previously it was O(result) instead of O(chars-searched), which can get very bad if .find() is called with a non-zero starting position. diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -2,6 +2,8 @@ # -*- encoding: utf-8 -*- import py import sys +from hypothesis import given, strategies, settings, example +from pypy.interpreter.error import OperationError class TestUnicodeObject: @@ -35,6 +37,49 @@ space.w_unicode, "__new__", space.w_unicode, w_uni) assert w_new is w_uni + @given(strategies.text(), strategies.integers(min_value=0, max_value=10), + strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(self, u, start, len1): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + space = self.space + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + class AppTestUnicodeStringStdOnly: def test_compares(self): @@ -698,6 +743,7 @@ def test_index(self): assert u"rrarrrrrrrrra".index(u'a', 4, None) == 12 assert u"rrarrrrrrrrra".index(u'a', None, 6) == 2 + assert u"\u1234\u4321\u5678".index(u'\u5678', 1) == 2 def test_rindex(self): from sys import maxint @@ -707,6 +753,7 @@ assert u'abcdefghiabc'.rindex(u'abc', 0, -1) == 0 assert u'abcdefghiabc'.rindex(u'abc', -4*maxint, 4*maxint) == 9 assert u'rrarrrrrrrrra'.rindex(u'a', 4, None) == 12 + assert u"\u1234\u5678".rindex(u'\u5678') == 1 raises(ValueError, u'abcdefghiabc'.rindex, u'hib') raises(ValueError, u'defghiabc'.rindex, u'def', 1) @@ -721,6 +768,7 @@ assert u'abcdefghiabc'.rfind(u'') == 12 assert u'abcdefghiabc'.rfind(u'abcd') == 0 assert u'abcdefghiabc'.rfind(u'abcz') == -1 + assert u"\u1234\u5678".rfind(u'\u5678') == 1 def test_rfind_corner_case(self): assert u'abc'.rfind('', 4) == -1 @@ -736,6 +784,7 @@ assert 'abcdefghiabc'.rindex(u'abc') == 9 raises(UnicodeDecodeError, '\x80'.index, u'') raises(UnicodeDecodeError, '\x80'.rindex, u'') + assert u"\u1234\u5678".find(u'\u5678') == 1 def test_count(self): assert u"".count(u"x") ==0 diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -1,7 +1,7 @@ """The builtin unicode implementation""" from rpython.rlib.objectmodel import ( - compute_hash, compute_unique_id, import_from_mixin, + compute_hash, compute_unique_id, import_from_mixin, always_inline, enforceargs, newlist_hint, specialize, we_are_translated) from rpython.rlib.buffer import StringBuffer from rpython.rlib.mutbuffer import MutableStringBuffer @@ -427,58 +427,32 @@ return W_UnicodeObject(result.build(), result_length) def descr_find(self, space, w_sub, w_start=None, w_end=None): - w_sub = self.convert_arg_to_w_unicode(space, w_sub) - start_index, end_index = self._unwrap_and_compute_idx_params( - space, w_start, w_end) - - res_index = self._utf8.find(w_sub._utf8, start_index, end_index) - if res_index == -1: - return space.newint(-1) - - res = rutf8.check_utf8(self._utf8, allow_surrogates=True, - force_len=res_index) # can't raise - return space.newint(res) + w_result = self._unwrap_and_search(space, w_sub, w_start, w_end) + if w_result is None: + w_result = space.newint(-1) + return w_result def descr_rfind(self, space, w_sub, w_start=None, w_end=None): - w_sub = self.convert_arg_to_w_unicode(space, w_sub) - start_index, end_index = self._unwrap_and_compute_idx_params( - space, w_start, w_end) - - res_index = self._utf8.rfind(w_sub._utf8, start_index, end_index) - if res_index == -1: - return space.newint(-1) - - res = rutf8.check_utf8(self._utf8, allow_surrogates=True, - force_len=res_index) # can't raise - return space.newint(res) + w_result = self._unwrap_and_search(space, w_sub, w_start, w_end, + forward=False) + if w_result is None: + w_result = space.newint(-1) + return w_result def descr_index(self, space, w_sub, w_start=None, w_end=None): - w_sub = self.convert_arg_to_w_unicode(space, w_sub) - start_index, end_index = self._unwrap_and_compute_idx_params( - space, w_start, w_end) - - res_index = self._utf8.find(w_sub._utf8, start_index, end_index) - if res_index == -1: + w_result = self._unwrap_and_search(space, w_sub, w_start, w_end) + if w_result is None: raise oefmt(space.w_ValueError, "substring not found in string.index") - - res = rutf8.check_utf8(self._utf8, allow_surrogates=True, - force_len=res_index) # can't raise - return space.newint(res) + return w_result def descr_rindex(self, space, w_sub, w_start=None, w_end=None): - w_sub = self.convert_arg_to_w_unicode(space, w_sub) - start_index, end_index = self._unwrap_and_compute_idx_params( - space, w_start, w_end) - - res_index = self._utf8.rfind(w_sub._utf8, start_index, end_index) - if res_index == -1: + w_result = self._unwrap_and_search(space, w_sub, w_start, w_end, + forward=False) + if w_result is None: raise oefmt(space.w_ValueError, "substring not found in string.rindex") - - res = rutf8.check_utf8(self._utf8, allow_surrogates=True, - force_len=res_index) # can't raise - return space.newint(res) + return w_result @specialize.arg(2) def _is_generic(self, space, func_name): @@ -908,11 +882,48 @@ index += self._length if index < 0 or index >= self._length: raise oefmt(space.w_IndexError, "string index out of range") - storage = self._get_index_storage() - start = rutf8.codepoint_position_at_index(self._utf8, storage, index) + start = self._index_to_byte(index) end = rutf8.next_codepoint_pos(self._utf8, start) return W_UnicodeObject(self._utf8[start:end], 1) + def _index_to_byte(self, index): + return rutf8.codepoint_position_at_index( + self._utf8, self._get_index_storage(), index) + + @always_inline + def _unwrap_and_search(self, space, w_sub, w_start, w_end, forward=True): + w_sub = self.convert_arg_to_w_unicode(space, w_sub) + start, end = unwrap_start_stop(space, self._length, w_start, w_end) + if start == 0: + start_index = 0 + elif start > self._length: + return None + else: + start_index = self._index_to_byte(start) + + if end >= self._length: + end = self._length + end_index = len(self._utf8) + else: + end_index = self._index_to_byte(end) + + if forward: + res_index = self._utf8.find(w_sub._utf8, start_index, end_index) + if res_index < 0: + return None + skip = rutf8.codepoints_in_utf8(self._utf8, start_index, res_index) + res = start + skip + assert res >= 0 + return space.newint(res) + else: + res_index = self._utf8.rfind(w_sub._utf8, start_index, end_index) + if res_index < 0: + return None + skip = rutf8.codepoints_in_utf8(self._utf8, res_index, end_index) + res = end - skip + assert res >= 0 + return space.newint(res) + def _unwrap_and_compute_idx_params(self, space, w_start, w_end): start, end = unwrap_start_stop(space, self._length, w_start, w_end) # XXX for now just create index diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -15,11 +15,12 @@ extra code in the middle for error handlers and so on. """ +import sys from rpython.rlib.objectmodel import enforceargs from rpython.rlib.rstring import StringBuilder from rpython.rlib import jit from rpython.rlib.rarithmetic import r_uint, intmask -from rpython.rtyper.lltypesystem import lltype +from rpython.rtyper.lltypesystem import lltype, rffi def unichr_as_utf8(code, allow_surrogates=False): @@ -290,17 +291,16 @@ #@jit.elidable -def check_utf8(s, allow_surrogates, force_len=-1): +def check_utf8(s, allow_surrogates): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. - Note that surrogates are not handled specially here. + If allow_surrogates is False, then also raise if we see any. + Note also codepoints_in_utf8(), which also computes the length + faster by assuming that 's' is valid utf-8. """ pos = 0 continuation_bytes = 0 - if force_len == -1: - end = len(s) - else: - end = force_len + end = len(s) while pos < end: ordch1 = ord(s[pos]) pos += 1 @@ -359,6 +359,23 @@ return pos - continuation_bytes @jit.elidable +def codepoints_in_utf8(value, start=0, end=sys.maxint): + """Return the number of codepoints in the UTF-8 byte string + 'value[start:end]'. Assumes 0 <= start <= len(value) and start <= end. + """ + if end > len(value): + end = len(value) + assert 0 <= start <= end + length = 0 + for i in range(start, end): + # we want to count the number of chars not between 0x80 and 0xBF; + # we do that by casting the char to a signed integer + signedchar = rffi.cast(rffi.SIGNEDCHAR, ord(value[i])) + if rffi.cast(lltype.Signed, signedchar) >= -0x40: + length += 1 + return length + + at jit.elidable def surrogate_in_utf8(value): """Check if the UTF-8 byte string 'value' contains a surrogate. The 'value' argument must be otherwise correctly formed for UTF-8. diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py --- a/rpython/rlib/test/test_rutf8.py +++ b/rpython/rlib/test/test_rutf8.py @@ -85,6 +85,19 @@ r = (ch in txt) assert r == response + at given(strategies.text(), strategies.integers(min_value=0), + strategies.integers(min_value=0)) +def test_codepoints_in_utf8(u, start, len1): + end = start + len1 + if end > len(u): + extra = end - len(u) + else: + extra = 0 + count = rutf8.codepoints_in_utf8(u.encode('utf8'), + len(u[:start].encode('utf8')), + len(u[:end].encode('utf8')) + extra) + assert count == len(u[start:end]) + @given(strategies.text()) def test_utf8_index_storage(u): index = rutf8.create_utf8_index_storage(u.encode('utf8'), len(u)) From pypy.commits at gmail.com Sat Oct 14 05:43:32 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:32 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Add startswith/endswith in the hypothesis test. Fixes. Includes a fix Message-ID: <59e1dc44.cd5e1c0a.647ea.958e@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92753:f029e18069d3 Date: 2017-10-14 10:47 +0200 http://bitbucket.org/pypy/pypy/changeset/f029e18069d3/ Log: Add startswith/endswith in the hypothesis test. Fixes. Includes a fix that should also be in default... diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -80,6 +80,28 @@ space.newint(start + len1)) assert space.int_w(w_index) == rexpected + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + class AppTestUnicodeStringStdOnly: def test_compares(self): @@ -475,6 +497,9 @@ assert u''.startswith(u'a') is False assert u'x'.startswith(u'xx') is False assert u'y'.startswith(u'xx') is False + assert u'\u1234\u5678\u4321'.startswith(u'\u1234') is True + assert u'\u1234\u5678\u4321'.startswith(u'\u1234\u4321') is False + assert u'\u1234'.startswith(u'', 1, 0) is True def test_startswith_more(self): assert u'ab'.startswith(u'a', 0) is True diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -599,8 +599,8 @@ def _startswith(self, space, value, w_prefix, start, end): prefix = self.convert_arg_to_w_unicode(space, w_prefix)._utf8 - if start > len(value): - return len(prefix) == 0 + if len(prefix) == 0: + return True return startswith(value, prefix, start, end) def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): @@ -613,8 +613,8 @@ def _endswith(self, space, value, w_prefix, start, end): prefix = self.convert_arg_to_w_unicode(space, w_prefix)._utf8 - if start > len(value): - return len(prefix) == 0 + if len(prefix) == 0: + return True return endswith(value, prefix, start, end) def descr_add(self, space, w_other): @@ -925,26 +925,20 @@ return space.newint(res) def _unwrap_and_compute_idx_params(self, space, w_start, w_end): + # unwrap start and stop indices, optimized for the case where + # start == 0 and end == self._length. Note that 'start' and + # 'end' are measured in codepoints whereas 'start_index' and + # 'end_index' are measured in bytes. start, end = unwrap_start_stop(space, self._length, w_start, w_end) - # XXX for now just create index start_index = 0 end_index = len(self._utf8) - if start > 0 or end != self._length: - storage = self._get_index_storage() - if start > 0: - # :-( - if start > self._length: - start_index = start - else: - start_index = rutf8.codepoint_position_at_index( - self._utf8, storage, start) - if end != self._length: - # :-( - if end > self._length: - end_index = end - else: - end_index = rutf8.codepoint_position_at_index( - self._utf8, storage, end) + if start > 0: + if start > self._length: + start_index = end_index + else: + start_index = self._index_to_byte(start) + if end < self._length: + end_index = self._index_to_byte(end) return (start_index, end_index) @unwrap_spec(width=int, w_fillchar=WrappedDefault(' ')) From pypy.commits at gmail.com Sat Oct 14 05:43:28 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:28 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Force 'allow_surrogates' to be specified in check_utf8() Message-ID: <59e1dc40.cdb51c0a.19139.c621@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92751:d9fe594a639b Date: 2017-10-14 07:34 +0200 http://bitbucket.org/pypy/pypy/changeset/d9fe594a639b/ Log: Force 'allow_surrogates' to be specified in check_utf8() diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -39,7 +39,8 @@ w_input = space.newbytes(input) else: w_cls = space.w_UnicodeEncodeError - w_input = space.newutf8(input, rutf8.check_utf8(input)) + length = rutf8.check_utf8(input, allow_surrogates=True) + w_input = space.newutf8(input, length) w_exc = space.call_function( w_cls, space.newtext(encoding), @@ -447,7 +448,8 @@ # "allow_surrogates=True" @unwrap_spec(utf8='utf8', errors='text_or_none') def utf_8_encode(space, utf8, errors="strict"): - return space.newtuple([space.newbytes(utf8), space.newint(rutf8.check_utf8(utf8))]) + length = rutf8.check_utf8(utf8, allow_surrogates=True) + return space.newtuple([space.newbytes(utf8), space.newint(length)]) #@unwrap_spec(uni=unicode, errors='text_or_none') #def utf_8_encode(space, uni, errors="strict"): # if errors is None: @@ -472,7 +474,7 @@ state = space.fromcache(CodecState) # call the fast version for checking try: - lgt = rutf8.check_utf8(string) + lgt = rutf8.check_utf8(string, allow_surrogates=True) except rutf8.CheckError as e: # XXX do the way around runicode - we can optimize it later if we # decide we care about obscure cases diff --git a/pypy/objspace/std/objspace.py b/pypy/objspace/std/objspace.py --- a/pypy/objspace/std/objspace.py +++ b/pypy/objspace/std/objspace.py @@ -317,7 +317,8 @@ for utf in lst: assert utf is not None assert isinstance(utf, str) - res_w.append(self.newutf8(utf, rutf8.check_utf8(utf))) + length = rutf8.check_utf8(utf, allow_surrogates=True) + res_w.append(self.newutf8(utf, length)) return self.newlist(res_w) def newlist_int(self, list_i): diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -435,7 +435,8 @@ if res_index == -1: return space.newint(-1) - res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + res = rutf8.check_utf8(self._utf8, allow_surrogates=True, + force_len=res_index) # can't raise return space.newint(res) def descr_rfind(self, space, w_sub, w_start=None, w_end=None): @@ -447,7 +448,8 @@ if res_index == -1: return space.newint(-1) - res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + res = rutf8.check_utf8(self._utf8, allow_surrogates=True, + force_len=res_index) # can't raise return space.newint(res) def descr_index(self, space, w_sub, w_start=None, w_end=None): @@ -460,7 +462,8 @@ raise oefmt(space.w_ValueError, "substring not found in string.index") - res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + res = rutf8.check_utf8(self._utf8, allow_surrogates=True, + force_len=res_index) # can't raise return space.newint(res) def descr_rindex(self, space, w_sub, w_start=None, w_end=None): @@ -473,7 +476,8 @@ raise oefmt(space.w_ValueError, "substring not found in string.rindex") - res = rutf8.check_utf8(self._utf8, force_len=res_index) # can't raise + res = rutf8.check_utf8(self._utf8, allow_surrogates=True, + force_len=res_index) # can't raise return space.newint(res) @specialize.arg(2) diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -290,7 +290,7 @@ #@jit.elidable -def check_utf8(s, allow_surrogates=False, force_len=-1): +def check_utf8(s, allow_surrogates, force_len=-1): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. Note that surrogates are not handled specially here. @@ -424,8 +424,8 @@ def codepoint_position_at_index(utf8, storage, index): """ Return byte index of a character inside utf8 encoded string, given storage of type UTF8_INDEX_STORAGE. The index must be smaller than - the utf8 length: if needed, check explicitly before calling this - function. + or equal to the utf8 length: if needed, check explicitly before calling + this function. """ current = index >> 6 ofs = ord(storage[current].ofs[(index >> 2) & 0x0F]) From pypy.commits at gmail.com Sat Oct 14 05:43:34 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 02:43:34 -0700 (PDT) Subject: [pypy-commit] pypy default: Test with hypothesis, fix for a corner case of startswith() Message-ID: <59e1dc46.ccabdf0a.d46da.2dbe@mx.google.com> Author: Armin Rigo Branch: Changeset: r92754:88bed3bb8ad4 Date: 2017-10-14 11:16 +0200 http://bitbucket.org/pypy/pypy/changeset/88bed3bb8ad4/ Log: Test with hypothesis, fix for a corner case of startswith() diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py --- a/pypy/objspace/std/stringmethods.py +++ b/pypy/objspace/std/stringmethods.py @@ -633,13 +633,16 @@ def _startswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) - if start > len(value): - return self._starts_ends_overflow(prefix) + if self._starts_ends_unicode: # bug-to-bug compat + if len(prefix) == 0: + return True + else: + if start > len(value): + return False return startswith(value, prefix, start, end) - def _starts_ends_overflow(self, prefix): - return False # bug-to-bug compat: this is for strings and - # bytearrays, but overridden for unicodes + _starts_ends_unicode = False # bug-to-bug compat: this is for strings and + # bytearrays, but overridden for unicodes def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): value, start, end, _ = self._convert_idx_params(space, w_start, w_end) @@ -656,8 +659,12 @@ def _endswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) - if start > len(value): - return self._starts_ends_overflow(prefix) + if self._starts_ends_unicode: # bug-to-bug compat + if len(prefix) == 0: + return True + else: + if start > len(value): + return False return endswith(value, prefix, start, end) def _strip(self, space, w_chars, left, right, name='strip'): diff --git a/pypy/objspace/std/test/test_bytesobject.py b/pypy/objspace/std/test/test_bytesobject.py --- a/pypy/objspace/std/test/test_bytesobject.py +++ b/pypy/objspace/std/test/test_bytesobject.py @@ -1,3 +1,7 @@ +from hypothesis import given, strategies, settings, example +from pypy.interpreter.error import OperationError + + class TestW_BytesObject: def teardown_method(self, method): @@ -86,6 +90,72 @@ w_bytes = self.space.newbytes('abcd') assert self.space.listview_bytes(w_bytes) == list("abcd") + @given(strategies.binary(), strategies.integers(min_value=0, max_value=10), + strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(self, u, start, len1): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + space = self.space + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + class AppTestBytesObject: def test_format_wrongchar(self): diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1,5 +1,7 @@ import py import sys +from hypothesis import given, strategies, settings, example +from pypy.interpreter.error import OperationError class TestUnicodeObject: @@ -32,6 +34,71 @@ space.w_unicode, "__new__", space.w_unicode, w_uni) assert w_new is w_uni + @given(strategies.text(), strategies.integers(min_value=0, max_value=10), + strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(self, u, start, len1): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + space = self.space + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + class AppTestUnicodeStringStdOnly: def test_compares(self): diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -419,8 +419,7 @@ cased = True return space.newbool(cased) - def _starts_ends_overflow(self, prefix): - return len(prefix) == 0 + _starts_ends_unicode = True def wrapunicode(space, uni): From pypy.commits at gmail.com Sat Oct 14 09:37:36 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 06:37:36 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Re-disable, turns out to be manually disabled almost everywhere else anyway. Message-ID: <59e21320.11b51c0a.3059d.827b@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92755:0aa72483c0e8 Date: 2017-10-14 15:27 +0200 http://bitbucket.org/pypy/pypy/changeset/0aa72483c0e8/ Log: Re-disable, turns out to be manually disabled almost everywhere else anyway. diff --git a/pypy/objspace/std/listobject.py b/pypy/objspace/std/listobject.py --- a/pypy/objspace/std/listobject.py +++ b/pypy/objspace/std/listobject.py @@ -1025,8 +1025,9 @@ w_list.lstorage = strategy.erase(byteslist[:]) return - unilist = space.listview_unicode(w_iterable) - if unilist is not None: + if False: + unilist = space.listview_unicode(w_iterable) + if unilist is not None: w_list.strategy = strategy = space.fromcache(UnicodeListStrategy) # need to copy because intlist can share with w_iterable w_list.lstorage = strategy.erase(unilist[:]) From pypy.commits at gmail.com Sat Oct 14 09:37:38 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 06:37:38 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: Translation fixes. The 'fix' in unicodeobject merely shows the next problem Message-ID: <59e21322.d185df0a.75cbe.e6d2@mx.google.com> Author: Armin Rigo Branch: unicode-utf8 Changeset: r92756:a5262fe7cebf Date: 2017-10-14 15:37 +0200 http://bitbucket.org/pypy/pypy/changeset/a5262fe7cebf/ Log: Translation fixes. The 'fix' in unicodeobject merely shows the next problem diff --git a/pypy/module/_multibytecodec/c_codecs.py b/pypy/module/_multibytecodec/c_codecs.py --- a/pypy/module/_multibytecodec/c_codecs.py +++ b/pypy/module/_multibytecodec/c_codecs.py @@ -264,7 +264,7 @@ else: assert retu is not None codec = pypy_cjk_enc_getcodec(encodebuf) - replace = encode(codec, retu.decode("utf8"), "strict", errorcb, namecb) + replace = encode(codec, retu, "strict", errorcb, namecb) with rffi.scoped_nonmovingbuffer(replace) as inbuf: r = pypy_cjk_enc_replace_on_error(encodebuf, inbuf, len(replace), end) if r == MBERR_NOMEMORY: diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py --- a/pypy/objspace/std/bytesobject.py +++ b/pypy/objspace/std/bytesobject.py @@ -663,7 +663,7 @@ if space.isinstance_w(w_prefix, space.w_unicode): self_as_unicode = unicode_from_encoded_object(space, self, None, None) - return self_as_unicode._startswith(space, self_as_unicode._utf8.decode("utf8"), + return self_as_unicode._startswith(space, self_as_unicode._utf8, w_prefix, start, end) return self._StringMethods__startswith(space, value, w_prefix, start, end) @@ -673,7 +673,7 @@ if space.isinstance_w(w_suffix, space.w_unicode): self_as_unicode = unicode_from_encoded_object(space, self, None, None) - return self_as_unicode._endswith(space, self_as_unicode._utf8.decode("utf8"), + return self_as_unicode._endswith(space, self_as_unicode._utf8, w_suffix, start, end) return self._StringMethods__endswith(space, value, w_suffix, start, end) diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -114,6 +114,7 @@ return space.newint(rutf8.codepoint_at_pos(self._utf8, 0)) def _new(self, value): + assert isinstance(value, unicode) return W_UnicodeObject(value.encode('utf8'), len(value)) def _new_from_list(self, value): From pypy.commits at gmail.com Sat Oct 14 09:57:43 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:43 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: we cannot cache the empty tuple as CPython does, else we end up having the same py_obj for different w_objs Message-ID: <59e217d7.46161c0a.a9a4d.d148@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92757:7860795a7204 Date: 2017-10-14 00:06 +0200 http://bitbucket.org/pypy/pypy/changeset/7860795a7204/ Log: we cannot cache the empty tuple as CPython does, else we end up having the same py_obj for different w_objs diff --git a/pypy/module/cpyext/src/tupleobject.c b/pypy/module/cpyext/src/tupleobject.c --- a/pypy/module/cpyext/src/tupleobject.c +++ b/pypy/module/cpyext/src/tupleobject.c @@ -1,5 +1,13 @@ -/* Tuple object implementation, stolen&adapted from CPython */ +/* Tuple object implementation, stolen&adapted from CPython. + + One important difference is that CPython caches the empty tuple separately + and always return the very same object, while we always return a fresh one. + The reasons is that space.newtuple([]) always return different objects, and + we want to ensure that the following is always true: + + w_a != w_b ==> as_pyobj(w_b) != as_pyobj(w_b) + */ #include "Python.h" @@ -12,9 +20,6 @@ #endif #if PyTuple_MAXSAVESIZE > 0 -/* Entries 1 up to PyTuple_MAXSAVESIZE are free lists, entry 0 is the empty - tuple () of which at most one instance will be allocated. -*/ static PyTupleObject *free_list[PyTuple_MAXSAVESIZE]; static int numfree[PyTuple_MAXSAVESIZE]; #endif @@ -29,11 +34,6 @@ return NULL; } #if PyTuple_MAXSAVESIZE > 0 - if (size == 0 && free_list[0]) { - op = free_list[0]; - Py_INCREF(op); - return (PyObject *) op; - } if (size < PyTuple_MAXSAVESIZE && (op = free_list[size]) != NULL) { free_list[size] = (PyTupleObject *) op->ob_item[0]; numfree[size]--; @@ -56,13 +56,6 @@ } for (i=0; i < size; i++) op->ob_item[i] = NULL; -#if PyTuple_MAXSAVESIZE > 0 - if (size == 0) { - free_list[0] = op; - ++numfree[0]; - Py_INCREF(op); /* extra INCREF so that this is never freed */ - } -#endif _PyObject_GC_TRACK(op); return (PyObject *) op; } @@ -76,7 +69,7 @@ register Py_ssize_t len = Py_SIZE(op); PyObject_GC_UnTrack(op); Py_TRASHCAN_SAFE_BEGIN(op) - if (len > 0) { + if (len >= 0) { i = len; while (--i >= 0) Py_XDECREF(op->ob_item[i]); diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -1,6 +1,6 @@ import py -from pypy.module.cpyext.pyobject import PyObject, PyObjectP, make_ref, from_ref, decref +from pypy.module.cpyext.pyobject import PyObject, PyObjectP, make_ref, from_ref, decref, as_pyobj from pypy.module.cpyext.test.test_api import BaseApiTest, raises_w from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase from rpython.rtyper.lltypesystem import rffi, lltype @@ -27,6 +27,31 @@ py.test.raises(FatalError, from_ref, space, py_tuple) decref(space, py_tuple) + def test_freelist(self, space, api): + state = space.fromcache(State) + # check that we don't cache the empty tuple + py_a = state.C.PyTuple_New(0) + py_b = state.C.PyTuple_New(0) + assert py_a != py_b + assert py_a.c_ob_refcnt == 1 + assert py_b.c_ob_refcnt == 1 + decref(space, py_a) + decref(space, py_b) + # + # check that the freelist is working + py_c = state.C.PyTuple_New(0) + assert py_c == py_b + decref(space, py_c) + + def test_empty_tuple_as_pyobj(self, space, api): + state = space.fromcache(State) + w_a = space.newtuple([]) + w_b = space.newtuple([]) + assert w_a is not w_b + py_a = as_pyobj(space, w_a) + py_b = as_pyobj(space, w_b) + assert py_a != py_b + def test_tuple_resize(self, space, api): state = space.fromcache(State) w_42 = space.wrap(42) From pypy.commits at gmail.com Sat Oct 14 09:57:45 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:45 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: missing import Message-ID: <59e217d9.69a9df0a.ba575.e1b9@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92758:59c2a72f8c1f Date: 2017-10-14 00:24 +0200 http://bitbucket.org/pypy/pypy/changeset/59c2a72f8c1f/ Log: missing import diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -5,7 +5,7 @@ cpython_api, Py_ssize_t, build_type_checkers_flags, PyVarObjectFields, cpython_struct, bootstrap_function, slot_function) from pypy.module.cpyext.pyobject import ( - PyObject, PyObjectP, make_ref, from_ref, decref, incref, + PyObject, PyObjectP, make_ref, from_ref, decref, incref, BaseCpyTypedescr, track_reference, make_typedescr, get_typedescr, pyobj_has_w_obj) from pypy.module.cpyext.state import State from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall From pypy.commits at gmail.com Sat Oct 14 09:57:47 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:47 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: a (hopefully quick) 'sub-branch' of cpyext-avoid-roundtrip to refactor and cleanup methodobject.py Message-ID: <59e217db.508d1c0a.90ca5.87df@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92759:c549ac0e8be0 Date: 2017-10-14 11:08 +0200 http://bitbucket.org/pypy/pypy/changeset/c549ac0e8be0/ Log: a (hopefully quick) 'sub-branch' of cpyext-avoid-roundtrip to refactor and cleanup methodobject.py From pypy.commits at gmail.com Sat Oct 14 09:57:53 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:53 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: rename these classes Message-ID: <59e217e1.c23a1c0a.b322e.bae6@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92762:b05962717d68 Date: 2017-10-14 15:32 +0200 http://bitbucket.org/pypy/pypy/changeset/b05962717d68/ Log: rename these classes diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -97,7 +97,7 @@ else: return space.w_None -class W_PyCFunctionObjectNoArgs(W_PyCFunctionObject): +class W_PyCFunctionObject_NOARGS(W_PyCFunctionObject): def call(self, space, w_self, w_args, w_kw): # Call the C function if w_self is None: @@ -105,7 +105,7 @@ func = self.ml.c_ml_meth return generic_cpy_call(space, func, w_self, None) -class W_PyCFunctionObjectSingleObject(W_PyCFunctionObject): +class W_PyCFunctionObject_O(W_PyCFunctionObject): def call(self, space, w_self, w_o, w_kw): if w_self is None: w_self = self.w_self @@ -243,12 +243,12 @@ def cfunction_descr_call_noargs(space, w_self): # special case for calling with flags METH_NOARGS - self = space.interp_w(W_PyCFunctionObjectNoArgs, w_self) + self = space.interp_w(W_PyCFunctionObject_NOARGS, w_self) return self.call(space, None, None, None) def cfunction_descr_call_single_object(space, w_self, w_o): # special case for calling with flags METH_O - self = space.interp_w(W_PyCFunctionObjectSingleObject, w_self) + self = space.interp_w(W_PyCFunctionObject_O, w_self) return self.call(space, None, w_o, None) @jit.dont_look_inside @@ -305,25 +305,25 @@ ) W_PyCFunctionObject.typedef.acceptable_as_base_class = False -W_PyCFunctionObjectNoArgs.typedef = TypeDef( +W_PyCFunctionObject_NOARGS.typedef = TypeDef( 'builtin_function_or_method', W_PyCFunctionObject.typedef, __call__ = interp2app(cfunction_descr_call_noargs), - __doc__ = GetSetProperty(W_PyCFunctionObjectNoArgs.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObjectNoArgs), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObjectNoArgs, + __doc__ = GetSetProperty(W_PyCFunctionObject_NOARGS.get_doc), + __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_NOARGS), + __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_NOARGS, wrapfn="newtext_or_none"), ) -W_PyCFunctionObjectNoArgs.typedef.acceptable_as_base_class = False +W_PyCFunctionObject_NOARGS.typedef.acceptable_as_base_class = False -W_PyCFunctionObjectSingleObject.typedef = TypeDef( +W_PyCFunctionObject_O.typedef = TypeDef( 'builtin_function_or_method', W_PyCFunctionObject.typedef, __call__ = interp2app(cfunction_descr_call_single_object), - __doc__ = GetSetProperty(W_PyCFunctionObjectSingleObject.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObjectSingleObject), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObjectSingleObject, + __doc__ = GetSetProperty(W_PyCFunctionObject_O.get_doc), + __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_O), + __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_O, wrapfn="newtext_or_none"), ) -W_PyCFunctionObjectSingleObject.typedef.acceptable_as_base_class = False +W_PyCFunctionObject_O.typedef.acceptable_as_base_class = False W_PyCMethodObject.typedef = TypeDef( 'method_descriptor', diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py --- a/pypy/module/cpyext/modsupport.py +++ b/pypy/module/cpyext/modsupport.py @@ -7,7 +7,7 @@ from pypy.module.cpyext.methodobject import ( W_PyCFunctionObject, PyCFunction_NewEx, PyDescr_NewMethod, PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New, - W_PyCFunctionObjectNoArgs, W_PyCFunctionObjectSingleObject) + W_PyCFunctionObject_NOARGS, W_PyCFunctionObject_O) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.module.cpyext.state import State from pypy.interpreter.error import oefmt @@ -84,9 +84,9 @@ def _create_pyc_function_object(space, method, w_self, w_name, flags): flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) if flags == METH_NOARGS: - return W_PyCFunctionObjectNoArgs(space, method, w_self, w_name) + return W_PyCFunctionObject_NOARGS(space, method, w_self, w_name) if flags == METH_O: - return W_PyCFunctionObjectSingleObject(space, method, w_self, w_name) + return W_PyCFunctionObject_O(space, method, w_self, w_name) return W_PyCFunctionObject(space, method, w_self, w_name) def convert_method_defs(space, dict_w, methods, w_type, w_self=None, name=None): From pypy.commits at gmail.com Sat Oct 14 09:57:55 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:55 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: move the descr_call inside the classes, to reduce a bit the cluttering Message-ID: <59e217e3.50b9df0a.69dfc.0f55@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92763:0509cab76af7 Date: 2017-10-14 15:54 +0200 http://bitbucket.org/pypy/pypy/changeset/0509cab76af7/ Log: move the descr_call inside the classes, to reduce a bit the cluttering diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -98,6 +98,11 @@ return space.w_None class W_PyCFunctionObject_NOARGS(W_PyCFunctionObject): + # METH_NOARGS + + def descr_call(self, space): + return self.call(space, None, None, None) + def call(self, space, w_self, w_args, w_kw): # Call the C function if w_self is None: @@ -106,6 +111,11 @@ return generic_cpy_call(space, func, w_self, None) class W_PyCFunctionObject_O(W_PyCFunctionObject): + # METH_O + + def descr_call(self, space, w_o): + return self.call(space, None, w_o, None) + def call(self, space, w_self, w_o, w_kw): if w_self is None: w_self = self.w_self @@ -241,16 +251,6 @@ space.setitem(w_kw, space.newtext(key), w_obj) return self.call(space, w_self, w_args, w_kw) -def cfunction_descr_call_noargs(space, w_self): - # special case for calling with flags METH_NOARGS - self = space.interp_w(W_PyCFunctionObject_NOARGS, w_self) - return self.call(space, None, None, None) - -def cfunction_descr_call_single_object(space, w_self, w_o): - # special case for calling with flags METH_O - self = space.interp_w(W_PyCFunctionObject_O, w_self) - return self.call(space, None, w_o, None) - @jit.dont_look_inside def cfunction_descr_call(space, w_self, __args__): # specialize depending on the W_PyCFunctionObject @@ -307,7 +307,7 @@ W_PyCFunctionObject_NOARGS.typedef = TypeDef( 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(cfunction_descr_call_noargs), + __call__ = interp2app(W_PyCFunctionObject_NOARGS.descr_call), __doc__ = GetSetProperty(W_PyCFunctionObject_NOARGS.get_doc), __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_NOARGS), __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_NOARGS, @@ -317,7 +317,7 @@ W_PyCFunctionObject_O.typedef = TypeDef( 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(cfunction_descr_call_single_object), + __call__ = interp2app(W_PyCFunctionObject_O.descr_call), __doc__ = GetSetProperty(W_PyCFunctionObject_O.get_doc), __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_O), __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_O, From pypy.commits at gmail.com Sat Oct 14 09:57:51 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:51 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix Message-ID: <59e217df.2f97df0a.f843c.d647@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92761:5c27ff8fb77b Date: 2017-10-14 15:27 +0200 http://bitbucket.org/pypy/pypy/changeset/5c27ff8fb77b/ Log: fix diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -112,8 +112,8 @@ ''' ), ]) - assert mod.isCFunction(mod.getarg_O) == "getarg_O" - assert mod.getModule(mod.getarg_O) == 'MyModule' + assert mod.isCFunction(mod.getModule) == "getModule" + assert mod.getModule(mod.getModule) == 'MyModule' if self.runappdirect: # XXX: fails untranslated assert mod.isSameFunction(mod.getModule) raises(SystemError, mod.isSameFunction, 1) From pypy.commits at gmail.com Sat Oct 14 09:57:49 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 06:57:49 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: split this one gigantic test into smaller ones Message-ID: <59e217dd.8d811c0a.827c6.8549@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92760:1ca2a0f2262f Date: 2017-10-14 11:47 +0200 http://bitbucket.org/pypy/pypy/changeset/1ca2a0f2262f/ Log: split this one gigantic test into smaller ones diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -8,14 +8,9 @@ from rpython.rtyper.lltypesystem import rffi, lltype class AppTestMethodObject(AppTestCpythonExtensionBase): - def test_call_METH(self): + + def test_call_METH_NOARGS(self): mod = self.import_extension('MyModule', [ - ('getarg_O', 'METH_O', - ''' - Py_INCREF(args); - return args; - ''' - ), ('getarg_NO', 'METH_NOARGS', ''' if(args) { @@ -28,6 +23,27 @@ } ''' ), + ]) + assert mod.getarg_NO() is None + raises(TypeError, mod.getarg_NO, 1) + raises(TypeError, mod.getarg_NO, 1, 1) + + def test_call_METH_O(self): + mod = self.import_extension('MyModule', [ + ('getarg_O', 'METH_O', + ''' + Py_INCREF(args); + return args; + ''' + ), + ]) + assert mod.getarg_O(1) == 1 + assert mod.getarg_O.__name__ == "getarg_O" + raises(TypeError, mod.getarg_O) + raises(TypeError, mod.getarg_O, 1, 1) + + def test_call_METH_OLDARGS(self): + mod = self.import_extension('MyModule', [ ('getarg_OLD', 'METH_OLDARGS', ''' if(args) { @@ -40,6 +56,28 @@ } ''' ), + ]) + assert mod.getarg_OLD(1) == 1 + assert mod.getarg_OLD() is None + assert mod.getarg_OLD(1, 2) == (1, 2) + + def test_call_METH_VARARGS(self): + mod = self.import_extension('MyModule', [ + ('getarg_VARARGS', 'METH_VARARGS', + ''' + PyObject * i; + i = PyLong_FromLong((long)PyObject_Length(args)); + Py_INCREF(i); + return i; + ''' + ), + ]) + assert mod.getarg_VARARGS() == 0 + assert mod.getarg_VARARGS(1) == 1 + raises(TypeError, mod.getarg_VARARGS, k=1) + + def test_func_attributes(self): + mod = self.import_extension('MyModule', [ ('isCFunction', 'METH_O', ''' if(PyCFunction_Check(args)) { @@ -67,30 +105,17 @@ ''' PyCFunction ptr = PyCFunction_GetFunction(args); if (!ptr) return NULL; - if (ptr == (PyCFunction)MyModule_getarg_O) + if (ptr == (PyCFunction)MyModule_getModule) Py_RETURN_TRUE; else Py_RETURN_FALSE; ''' ), ]) - assert mod.getarg_O(1) == 1 - assert mod.getarg_O.__name__ == "getarg_O" - raises(TypeError, mod.getarg_O) - raises(TypeError, mod.getarg_O, 1, 1) - - assert mod.getarg_NO() is None - raises(TypeError, mod.getarg_NO, 1) - raises(TypeError, mod.getarg_NO, 1, 1) - - assert mod.getarg_OLD(1) == 1 - assert mod.getarg_OLD() is None - assert mod.getarg_OLD(1, 2) == (1, 2) - assert mod.isCFunction(mod.getarg_O) == "getarg_O" assert mod.getModule(mod.getarg_O) == 'MyModule' if self.runappdirect: # XXX: fails untranslated - assert mod.isSameFunction(mod.getarg_O) + assert mod.isSameFunction(mod.getModule) raises(SystemError, mod.isSameFunction, 1) def test_check(self): From pypy.commits at gmail.com Sat Oct 14 11:24:45 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 08:24:45 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: fix this test Message-ID: <59e22c3d.4c891c0a.62176.1a39@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92764:1fe00f592e00 Date: 2017-10-14 17:23 +0200 http://bitbucket.org/pypy/pypy/changeset/1fe00f592e00/ Log: fix this test diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -6,7 +6,7 @@ PySequence_Fast, PySequence_Contains, PySequence_Index, PySequence_GetItem, PySequence_SetItem, PySequence_DelItem) from pypy.module.cpyext.pyobject import get_w_obj_and_decref - +from pypy.module.cpyext.state import State import pytest class TestSequence(BaseApiTest): @@ -108,6 +108,7 @@ PySequence_Contains(space, space.w_None, space.wrap(2)) def test_setitem(self, space, api): + state = space.fromcache(State) w_value = space.wrap(42) l = api.PyList_New(1) @@ -117,7 +118,7 @@ with raises_w(space, IndexError): PySequence_SetItem(space, l, 3, w_value) - t = api.PyTuple_New(1) + t = state.C.PyTuple_New(1) api.PyTuple_SetItem(t, 0, l) with raises_w(space, TypeError): PySequence_SetItem(space, t, 0, w_value) From pypy.commits at gmail.com Sat Oct 14 11:24:47 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 08:24:47 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: merge the fix from cpyext-avoid-roundtrip Message-ID: <59e22c3f.099fdf0a.83eb6.24cb@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92765:e627947095d3 Date: 2017-10-14 17:24 +0200 http://bitbucket.org/pypy/pypy/changeset/e627947095d3/ Log: merge the fix from cpyext-avoid-roundtrip diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py --- a/pypy/module/cpyext/test/test_sequence.py +++ b/pypy/module/cpyext/test/test_sequence.py @@ -6,7 +6,7 @@ PySequence_Fast, PySequence_Contains, PySequence_Index, PySequence_GetItem, PySequence_SetItem, PySequence_DelItem) from pypy.module.cpyext.pyobject import get_w_obj_and_decref - +from pypy.module.cpyext.state import State import pytest class TestSequence(BaseApiTest): @@ -108,6 +108,7 @@ PySequence_Contains(space, space.w_None, space.wrap(2)) def test_setitem(self, space, api): + state = space.fromcache(State) w_value = space.wrap(42) l = api.PyList_New(1) @@ -117,7 +118,7 @@ with raises_w(space, IndexError): PySequence_SetItem(space, l, 3, w_value) - t = api.PyTuple_New(1) + t = state.C.PyTuple_New(1) api.PyTuple_SetItem(t, 0, l) with raises_w(space, TypeError): PySequence_SetItem(space, t, 0, w_value) From pypy.commits at gmail.com Sat Oct 14 12:33:56 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 09:33:56 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: introduce a specialized class for METH_VARARGS functions Message-ID: <59e23c74.99471c0a.a5304.5329@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92766:e06951cd3fe6 Date: 2017-10-14 18:11 +0200 http://bitbucket.org/pypy/pypy/changeset/e06951cd3fe6/ Log: introduce a specialized class for METH_VARARGS functions diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -100,6 +100,7 @@ class W_PyCFunctionObject_NOARGS(W_PyCFunctionObject): # METH_NOARGS + # CCC I think we can condense descr_call and call into one, can't we? def descr_call(self, space): return self.call(space, None, None, None) @@ -113,6 +114,7 @@ class W_PyCFunctionObject_O(W_PyCFunctionObject): # METH_O + # CCC I think we can condense descr_call and call into one, can't we? def descr_call(self, space, w_o): return self.call(space, None, w_o, None) @@ -122,6 +124,15 @@ func = self.ml.c_ml_meth return generic_cpy_call(space, func, w_self, w_o) +class W_PyCFunctionObject_VARARGS(W_PyCFunctionObject): + # METH_VARARGS + + def descr_call(self, space, args_w): + w_self = self.w_self + func = self.ml.c_ml_meth + w_args = space.newtuple(args_w) + return generic_cpy_call(space, func, w_self, w_args) + class W_PyCMethodObject(W_PyCFunctionObject): w_self = None def __init__(self, space, ml, w_type): @@ -325,6 +336,16 @@ ) W_PyCFunctionObject_O.typedef.acceptable_as_base_class = False +W_PyCFunctionObject_VARARGS.typedef = TypeDef( + 'builtin_function_or_method', W_PyCFunctionObject.typedef, + __call__ = interp2app(W_PyCFunctionObject_VARARGS.descr_call), + __doc__ = GetSetProperty(W_PyCFunctionObject_VARARGS.get_doc), + __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_VARARGS), + __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_VARARGS, + wrapfn="newtext_or_none"), + ) +W_PyCFunctionObject_VARARGS.typedef.acceptable_as_base_class = False + W_PyCMethodObject.typedef = TypeDef( 'method_descriptor', __get__ = interp2app(cmethod_descr_get), diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py --- a/pypy/module/cpyext/modsupport.py +++ b/pypy/module/cpyext/modsupport.py @@ -1,13 +1,14 @@ from rpython.rtyper.lltypesystem import rffi, lltype from pypy.module.cpyext.api import cpython_api, cpython_struct, \ METH_STATIC, METH_CLASS, METH_COEXIST, CANNOT_FAIL, CONST_STRING, \ - METH_NOARGS, METH_O + METH_NOARGS, METH_O, METH_VARARGS from pypy.module.cpyext.pyobject import PyObject, as_pyobj from pypy.interpreter.module import Module from pypy.module.cpyext.methodobject import ( W_PyCFunctionObject, PyCFunction_NewEx, PyDescr_NewMethod, PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New, - W_PyCFunctionObject_NOARGS, W_PyCFunctionObject_O) + W_PyCFunctionObject_NOARGS, W_PyCFunctionObject_O, + W_PyCFunctionObject_VARARGS) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.module.cpyext.state import State from pypy.interpreter.error import oefmt @@ -87,6 +88,8 @@ return W_PyCFunctionObject_NOARGS(space, method, w_self, w_name) if flags == METH_O: return W_PyCFunctionObject_O(space, method, w_self, w_name) + if flags == METH_VARARGS: + return W_PyCFunctionObject_VARARGS(space, method, w_self, w_name) return W_PyCFunctionObject(space, method, w_self, w_name) def convert_method_defs(space, dict_w, methods, w_type, w_self=None, name=None): diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -65,15 +65,19 @@ mod = self.import_extension('MyModule', [ ('getarg_VARARGS', 'METH_VARARGS', ''' - PyObject * i; - i = PyLong_FromLong((long)PyObject_Length(args)); - Py_INCREF(i); - return i; + return Py_BuildValue("Ol", args, args->ob_refcnt); ''' ), ]) - assert mod.getarg_VARARGS() == 0 - assert mod.getarg_VARARGS(1) == 1 + tup, _ = mod.getarg_VARARGS() + assert tup == () + # + tup, _ = mod.getarg_VARARGS(1) + assert tup == (1,) + # + tup, _ = mod.getarg_VARARGS(1, 2, 3) + assert tup == (1, 2, 3) + # raises(TypeError, mod.getarg_VARARGS, k=1) def test_func_attributes(self): From pypy.commits at gmail.com Sat Oct 14 12:33:58 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 14 Oct 2017 09:33:58 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: when calling a METH_VARARGS, create directly a PyTuple, instead of passing from a w_tuple; in theory, this should make the args tuple to die very quickly, and thus make a more effective usage of freelists Message-ID: <59e23c76.56acdf0a.89f47.d438@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92767:cefeba1a1de2 Date: 2017-10-14 18:30 +0200 http://bitbucket.org/pypy/pypy/changeset/cefeba1a1de2/ Log: when calling a METH_VARARGS, create directly a PyTuple, instead of passing from a w_tuple; in theory, this should make the args tuple to die very quickly, and thus make a more effective usage of freelists diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -15,6 +15,8 @@ build_type_checkers) from pypy.module.cpyext.pyobject import ( decref, from_ref, make_ref, as_pyobj, make_typedescr) +from pypy.module.cpyext.state import State +from pypy.module.cpyext.tupleobject import PyTuple_SetItem PyMethodDef = cts.gettype('PyMethodDef') PyCFunction = cts.gettype('PyCFunction') @@ -128,10 +130,18 @@ # METH_VARARGS def descr_call(self, space, args_w): + state = space.fromcache(State) w_self = self.w_self func = self.ml.c_ml_meth - w_args = space.newtuple(args_w) - return generic_cpy_call(space, func, w_self, w_args) + n = len(args_w) + py_args = state.C.PyTuple_New(n) + for i, w_item in enumerate(args_w): + py_item = make_ref(space, w_item) + PyTuple_SetItem(space, py_args, i, py_item) + try: + return generic_cpy_call(space, func, w_self, py_args) + finally: + decref(space, py_args) class W_PyCMethodObject(W_PyCFunctionObject): w_self = None diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -69,14 +69,26 @@ ''' ), ]) - tup, _ = mod.getarg_VARARGS() + # check that we pass the expected tuple of arguments AND that the + # recnt is 1. In particular, on PyPy refcnt==1 means that we created + # the PyObject tuple directly, without passing from a w_tuple; as + # such, the tuple will be immediately freed after the call, without + # having to wait until the GC runs. + # + tup, refcnt = mod.getarg_VARARGS() assert tup == () + # the empty tuple is shared on CPython, so the refcnt will be >1. On + # PyPy it is not shared, though. + if not self.runappdirect: + assert refcnt == 1 # - tup, _ = mod.getarg_VARARGS(1) + tup, refcnt = mod.getarg_VARARGS(1) assert tup == (1,) + assert refcnt == 1 # - tup, _ = mod.getarg_VARARGS(1, 2, 3) + tup, refcnt = mod.getarg_VARARGS(1, 2, 3) assert tup == (1, 2, 3) + assert refcnt == 1 # raises(TypeError, mod.getarg_VARARGS, k=1) From pypy.commits at gmail.com Sat Oct 14 13:55:35 2017 From: pypy.commits at gmail.com (arigo) Date: Sat, 14 Oct 2017 10:55:35 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: write down these to-do items Message-ID: <59e24f97.040c1c0a.a2c93.19e8@mx.google.com> Author: Armin Rigo Branch: extradoc Changeset: r5842:111393a517be Date: 2017-10-14 19:55 +0200 http://bitbucket.org/pypy/extradoc/changeset/111393a517be/ Log: write down these to-do items diff --git a/planning/cpyext.txt b/planning/cpyext.txt --- a/planning/cpyext.txt +++ b/planning/cpyext.txt @@ -24,3 +24,8 @@ - as_pyobj() should be a context manager which calls keepalive_until_here() +- tupleobject.c: PyErr_BadInternalCall() and PyErr_NoMemory() + cannot be called from PyTuple_New() right now, think again + +- methodobject.py: be ready for make_ref() to fail, and free the + partially-contructed tuple (same logic as tuple_attach) From pypy.commits at gmail.com Sun Oct 15 16:19:20 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 15 Oct 2017 13:19:20 -0700 (PDT) Subject: [pypy-commit] pypy default: Hypothesis found this failure on Windows Message-ID: <59e3c2c8.4a9fdf0a.63991.12da@mx.google.com> Author: Armin Rigo Branch: Changeset: r92768:a9f41bdd18f8 Date: 2017-10-15 22:17 +0200 http://bitbucket.org/pypy/pypy/changeset/a9f41bdd18f8/ Log: Hypothesis found this failure on Windows diff --git a/pypy/module/unicodedata/test/test_hyp.py b/pypy/module/unicodedata/test/test_hyp.py --- a/pypy/module/unicodedata/test/test_hyp.py +++ b/pypy/module/unicodedata/test/test_hyp.py @@ -40,6 +40,7 @@ @pytest.mark.parametrize('NF1, NF2, NF3', compositions) @example(s=u'---\uafb8\u11a7---') # issue 2289 + at example(s=u'\ufacf') @settings(max_examples=1000) @given(s=st.text()) def test_composition(s, space, NF1, NF2, NF3): From pypy.commits at gmail.com Sun Oct 15 16:25:30 2017 From: pypy.commits at gmail.com (arigo) Date: Sun, 15 Oct 2017 13:25:30 -0700 (PDT) Subject: [pypy-commit] pypy default: Make these tests import even if 'hypothesis' is not available Message-ID: <59e3c43a.95a2df0a.8db27.0af5@mx.google.com> Author: Armin Rigo Branch: Changeset: r92769:f29e5fba625b Date: 2017-10-15 22:24 +0200 http://bitbucket.org/pypy/pypy/changeset/f29e5fba625b/ Log: Make these tests import even if 'hypothesis' is not available diff --git a/pypy/objspace/std/test/test_bytesobject.py b/pypy/objspace/std/test/test_bytesobject.py --- a/pypy/objspace/std/test/test_bytesobject.py +++ b/pypy/objspace/std/test/test_bytesobject.py @@ -1,4 +1,3 @@ -from hypothesis import given, strategies, settings, example from pypy.interpreter.error import OperationError @@ -90,13 +89,19 @@ w_bytes = self.space.newbytes('abcd') assert self.space.listview_bytes(w_bytes) == list("abcd") - @given(strategies.binary(), strategies.integers(min_value=0, max_value=10), - strategies.integers(min_value=-1, max_value=10)) - def test_hypo_index_find(self, u, start, len1): + +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.binary(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): if start + len1 < 0: return # skip this case v = u[start : start + len1] - space = self.space w_u = space.wrap(u) w_v = space.wrap(v) expected = u.find(v, start, start + len1) diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1,6 +1,5 @@ import py import sys -from hypothesis import given, strategies, settings, example from pypy.interpreter.error import OperationError @@ -34,13 +33,19 @@ space.w_unicode, "__new__", space.w_unicode, w_uni) assert w_new is w_uni - @given(strategies.text(), strategies.integers(min_value=0, max_value=10), - strategies.integers(min_value=-1, max_value=10)) - def test_hypo_index_find(self, u, start, len1): + +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.text(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): if start + len1 < 0: return # skip this case v = u[start : start + len1] - space = self.space w_u = space.wrap(u) w_v = space.wrap(v) expected = u.find(v, start, start + len1) From pypy.commits at gmail.com Sun Oct 15 16:49:59 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 15 Oct 2017 13:49:59 -0700 (PDT) Subject: [pypy-commit] pypy default: make sure argv0 is bytes, not unicode (for matplotlib tkagg backend) Message-ID: <59e3c9f7.d598df0a.9f3db.1c57@mx.google.com> Author: Matti Picus Branch: Changeset: r92771:4da3d7649d2b Date: 2017-10-15 19:18 +0300 http://bitbucket.org/pypy/pypy/changeset/4da3d7649d2b/ Log: make sure argv0 is bytes, not unicode (for matplotlib tkagg backend) diff --git a/lib_pypy/_tkinter/app.py b/lib_pypy/_tkinter/app.py --- a/lib_pypy/_tkinter/app.py +++ b/lib_pypy/_tkinter/app.py @@ -119,7 +119,7 @@ tklib.TCL_GLOBAL_ONLY) # This is used to get the application class for Tk 4.1 and up - argv0 = className.lower() + argv0 = className.lower().encode('ascii') tklib.Tcl_SetVar(self.interp, "argv0", argv0, tklib.TCL_GLOBAL_ONLY) From pypy.commits at gmail.com Sun Oct 15 16:49:57 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 15 Oct 2017 13:49:57 -0700 (PDT) Subject: [pypy-commit] pypy default: ignore cffi build artifacts on windows Message-ID: <59e3c9f5.8b141c0a.79ae.cb04@mx.google.com> Author: Matti Picus Branch: Changeset: r92770:f995cc10eaaf Date: 2017-10-15 19:18 +0300 http://bitbucket.org/pypy/pypy/changeset/f995cc10eaaf/ Log: ignore cffi build artifacts on windows diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -71,6 +71,8 @@ ^lib_pypy/.+.c$ ^lib_pypy/.+.o$ ^lib_pypy/.+.so$ +^lib_pypy/.+.pyd$ +^lib_pypy/Release/ ^pypy/doc/discussion/.+\.html$ ^include/.+\.h$ ^include/.+\.inl$ From pypy.commits at gmail.com Sun Oct 15 19:10:20 2017 From: pypy.commits at gmail.com (mjacob) Date: Sun, 15 Oct 2017 16:10:20 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Port 773018d6ae59 to 3.5 standard library. Message-ID: <59e3eadc.978adf0a.ef0d8.c362@mx.google.com> Author: Manuel Jacob Branch: py3.5 Changeset: r92772:7126604d932e Date: 2017-10-16 01:09 +0200 http://bitbucket.org/pypy/pypy/changeset/7126604d932e/ Log: Port 773018d6ae59 to 3.5 standard library. diff --git a/lib-python/3/test/test_bytes.py b/lib-python/3/test/test_bytes.py --- a/lib-python/3/test/test_bytes.py +++ b/lib-python/3/test/test_bytes.py @@ -721,9 +721,12 @@ self.assertIs(type(BytesSubclass(A())), BytesSubclass) # Test PyBytes_FromFormat() - @test.support.impl_detail("don't test cpyext here") def test_from_format(self): test.support.import_module('ctypes') + try: + from ctypes import pythonapi + except ImportError: + self.skipTest( "no pythonapi in ctypes") from ctypes import pythonapi, py_object, c_int, c_char_p PyBytes_FromFormat = pythonapi.PyBytes_FromFormat PyBytes_FromFormat.restype = py_object diff --git a/lib-python/3/test/test_unicode.py b/lib-python/3/test/test_unicode.py --- a/lib-python/3/test/test_unicode.py +++ b/lib-python/3/test/test_unicode.py @@ -2396,6 +2396,10 @@ # Test PyUnicode_FromFormat() def test_from_format(self): support.import_module('ctypes') + try: + from ctypes import pythonapi + except ImportError: + self.skipTest( "no pythonapi in ctypes") from ctypes import ( pythonapi, py_object, sizeof, c_int, c_long, c_longlong, c_ssize_t, From pypy.commits at gmail.com Sun Oct 15 23:21:21 2017 From: pypy.commits at gmail.com (mjacob) Date: Sun, 15 Oct 2017 20:21:21 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Remove some entries from the TODO which I believe are solved or not relevant any more. Message-ID: <59e425b1.11b51c0a.3059d.17c6@mx.google.com> Author: Manuel Jacob Branch: py3.5 Changeset: r92773:c5011e45fadc Date: 2017-10-16 05:20 +0200 http://bitbucket.org/pypy/pypy/changeset/c5011e45fadc/ Log: Remove some entries from the TODO which I believe are solved or not relevant any more. diff --git a/pypy/TODO b/pypy/TODO --- a/pypy/TODO +++ b/pypy/TODO @@ -1,18 +1,4 @@ -TODO for the python3 test suite: - -* test_memoryview - Needs bytes/str changes. Probably easy. Work for this has begun on - py3k-memoryview (by mjacob) https://bugs.pypy.org/issue1542 - -own-tests: - -* module/test_lib_pypy - These crash the buildbots (via SyntaxErrors): others were really - made to run under Python 2.x and so simply fail - -* module.cpyext.test.test_structseq test_StructSeq - structseq now subclasses tuple on py3, which breaks how - BaseCpyTypeDescr.realize allocates it +... antocuni's older TODO: @@ -20,14 +6,6 @@ * run coverage against the parser/astbuilder/astcompiler: it's probably full of dead code because the grammar changed -* re-enable strategies https://bugs.pypy.org/issue1540 : - - re-enable IntDictStrategy - - re-enable StdObjSpace.listview_str - - re-enable the kwargs dict strategy in dictmultiobject.py - - re-enable view_as_kwargs - -* unskip numpypy tests in module/test_lib_pypy/numpypy/ - * optimize W_UnicodeObject, right now it stores both an unicode and an utf8 version of the same string From pypy.commits at gmail.com Mon Oct 16 06:23:53 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 03:23:53 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: add a comment about a potential issue Message-ID: <59e488b9.c82d1c0a.e55cf.2ff0@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92774:718a91452806 Date: 2017-10-16 11:42 +0200 http://bitbucket.org/pypy/pypy/changeset/718a91452806/ Log: add a comment about a potential issue diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -27,6 +27,10 @@ ## Then the macro PyTuple_GET_ITEM can be implemented like CPython. ## +# CCC: we have a problem in the branch: some of the functions defined in C, +# like PyTuple_New, might call PyErr_*, but since these are implemented in +# RPython, they are not GIL-safe. We need to think about it :( + PyTupleObjectStruct = lltype.ForwardReference() PyTupleObject = lltype.Ptr(PyTupleObjectStruct) ObjectItems = rffi.CArray(PyObject) From pypy.commits at gmail.com Mon Oct 16 06:23:55 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 03:23:55 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: _Py_NewReference did not initialize pypy_link: as such, reused tuples from the freelist could contain a wrong value. Test&fix Message-ID: <59e488bb.d598df0a.9f3db.a3a3@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92775:0cd1a1fbbad9 Date: 2017-10-16 12:22 +0200 http://bitbucket.org/pypy/pypy/changeset/0cd1a1fbbad9/ Log: _Py_NewReference did not initialize pypy_link: as such, reused tuples from the freelist could contain a wrong value. Test&fix diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -65,7 +65,10 @@ #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) #define Py_SIZE(ob) (((PyVarObject*)(ob))->ob_size) -#define _Py_NewReference(ob) (((PyObject *)(ob))->ob_refcnt = 1) +#define _Py_NewReference(op) \ + ( ((PyObject *)(op))->ob_refcnt = 1, \ + ((PyObject *)(op))->ob_pypy_link = 0 ) + #define _Py_ForgetReference(ob) /* nothing */ #define Py_None (&_Py_NoneStruct) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -17,7 +17,16 @@ Py_XDECREF(o); } -Py_ssize_t _pypy_rawrefcount_w_marker_deallocating; /* set from pyobject.py */ +/* + * The actual value of this variable will be the address of + * pyobject.w_marker_deallocating, and will be set by + * pyobject.write_w_marker_deallocating(). + * + * The value set here is used only as a marker by tests (because during the + * tests we cannot call set_marker(), so we need to set a special value + * directly here) + */ +Py_ssize_t _pypy_rawrefcount_w_marker_deallocating = 0xDEADFFF; void _Py_Dealloc(PyObject *obj) diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -52,6 +52,31 @@ py_b = as_pyobj(space, w_b) assert py_a != py_b + def test_PyTuple_New_initialize_pypy_link(self, space, api): + from rpython.rlib.rawrefcount import _collect + state = space.fromcache(State) + # see object.c:_pypy_rawrefcount_w_marker_deallocating + MARKER = 0xDEADFFF + # + # first: create a pytuple, attach a w_obj, decref the pytuple and let + # the GC to collect the w_obj: this way, c_ob_pypy_link is set to + # w_marker_deallocating + py_a = state.C.PyTuple_New(0) + assert py_a.c_ob_pypy_link == 0 + w_a = from_ref(space, py_a) + assert py_a.c_ob_pypy_link != 0 + decref(space, py_a) + w_a = None + _collect() + assert py_a.c_ob_pypy_link == MARKER + # + # second: create another tuple, which will reuse the same memory as + # before thanks to the freelist. Check that c_ob_pypy_link has been + # initialized to 0. + py_b = state.C.PyTuple_New(0) + assert py_b == py_a + assert py_b.c_ob_pypy_link == 0 + def test_tuple_resize(self, space, api): state = space.fromcache(State) w_42 = space.wrap(42) From pypy.commits at gmail.com Mon Oct 16 06:23:57 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 03:23:57 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: merge cpyext-avoid-roundtrip inside this Message-ID: <59e488bd.11b51c0a.3059d.a520@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92776:a5eb32fe2228 Date: 2017-10-16 12:23 +0200 http://bitbucket.org/pypy/pypy/changeset/a5eb32fe2228/ Log: merge cpyext-avoid-roundtrip inside this diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -65,7 +65,10 @@ #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) #define Py_SIZE(ob) (((PyVarObject*)(ob))->ob_size) -#define _Py_NewReference(ob) (((PyObject *)(ob))->ob_refcnt = 1) +#define _Py_NewReference(op) \ + ( ((PyObject *)(op))->ob_refcnt = 1, \ + ((PyObject *)(op))->ob_pypy_link = 0 ) + #define _Py_ForgetReference(ob) /* nothing */ #define Py_None (&_Py_NoneStruct) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -17,7 +17,16 @@ Py_XDECREF(o); } -Py_ssize_t _pypy_rawrefcount_w_marker_deallocating; /* set from pyobject.py */ +/* + * The actual value of this variable will be the address of + * pyobject.w_marker_deallocating, and will be set by + * pyobject.write_w_marker_deallocating(). + * + * The value set here is used only as a marker by tests (because during the + * tests we cannot call set_marker(), so we need to set a special value + * directly here) + */ +Py_ssize_t _pypy_rawrefcount_w_marker_deallocating = 0xDEADFFF; void _Py_Dealloc(PyObject *obj) diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -52,6 +52,31 @@ py_b = as_pyobj(space, w_b) assert py_a != py_b + def test_PyTuple_New_initialize_pypy_link(self, space, api): + from rpython.rlib.rawrefcount import _collect + state = space.fromcache(State) + # see object.c:_pypy_rawrefcount_w_marker_deallocating + MARKER = 0xDEADFFF + # + # first: create a pytuple, attach a w_obj, decref the pytuple and let + # the GC to collect the w_obj: this way, c_ob_pypy_link is set to + # w_marker_deallocating + py_a = state.C.PyTuple_New(0) + assert py_a.c_ob_pypy_link == 0 + w_a = from_ref(space, py_a) + assert py_a.c_ob_pypy_link != 0 + decref(space, py_a) + w_a = None + _collect() + assert py_a.c_ob_pypy_link == MARKER + # + # second: create another tuple, which will reuse the same memory as + # before thanks to the freelist. Check that c_ob_pypy_link has been + # initialized to 0. + py_b = state.C.PyTuple_New(0) + assert py_b == py_a + assert py_b.c_ob_pypy_link == 0 + def test_tuple_resize(self, space, api): state = space.fromcache(State) w_42 = space.wrap(42) diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -27,6 +27,10 @@ ## Then the macro PyTuple_GET_ITEM can be implemented like CPython. ## +# CCC: we have a problem in the branch: some of the functions defined in C, +# like PyTuple_New, might call PyErr_*, but since these are implemented in +# RPython, they are not GIL-safe. We need to think about it :( + PyTupleObjectStruct = lltype.ForwardReference() PyTupleObject = lltype.Ptr(PyTupleObjectStruct) ObjectItems = rffi.CArray(PyObject) From pypy.commits at gmail.com Mon Oct 16 11:53:24 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 16 Oct 2017 08:53:24 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Don't allow passing str to ctypes.c_char or bytes to ctypes.c_wchar Message-ID: <59e4d5f4.952d1c0a.a17d3.a42b@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92780:09772d33fc36 Date: 2017-10-16 17:50 +0200 http://bitbucket.org/pypy/pypy/changeset/09772d33fc36/ Log: Don't allow passing str to ctypes.c_char or bytes to ctypes.c_wchar diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py --- a/lib_pypy/_ctypes/primitive.py +++ b/lib_pypy/_ctypes/primitive.py @@ -232,9 +232,6 @@ elif tp == 'u': def _setvalue(self, val): - if isinstance(val, bytes): - val = val.decode(ConvMode.encoding, ConvMode.errors) - # possible if we use 'ignore' if val: self._buffer[0] = val def _getvalue(self): @@ -243,8 +240,6 @@ elif tp == 'c': def _setvalue(self, val): - if isinstance(val, str): - val = val.encode(ConvMode.encoding, ConvMode.errors) if val: self._buffer[0] = val def _getvalue(self): From pypy.commits at gmail.com Mon Oct 16 12:41:14 2017 From: pypy.commits at gmail.com (rlamy) Date: Mon, 16 Oct 2017 09:41:14 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Mark slots test as an implementation detail Message-ID: <59e4e12a.0291df0a.cc3bf.c52b@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92781:9f84023a5df7 Date: 2017-10-16 18:40 +0200 http://bitbucket.org/pypy/pypy/changeset/9f84023a5df7/ Log: Mark slots test as an implementation detail diff --git a/lib-python/3/ctypes/test/test_byteswap.py b/lib-python/3/ctypes/test/test_byteswap.py --- a/lib-python/3/ctypes/test/test_byteswap.py +++ b/lib-python/3/ctypes/test/test_byteswap.py @@ -2,6 +2,7 @@ from binascii import hexlify from ctypes import * +from test.support import impl_detail def bin(s): return hexlify(memoryview(s)).decode().upper() @@ -22,6 +23,7 @@ setattr(bits, "i%s" % i, 1) dump(bits) + @impl_detail("slots are irrelevant on PyPy", pypy=False) def test_slots(self): class BigPoint(BigEndianStructure): __slots__ = () From pypy.commits at gmail.com Mon Oct 16 18:54:35 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 15:54:35 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: write a specialized function to convert args_w into a py_tuple, which sets directly c_ob_item[*] instead of going through the slowish PyTuple_SetItem Message-ID: <59e538ab.9b88df0a.6da25.d753@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92782:17fd1d984f1d Date: 2017-10-17 00:47 +0200 http://bitbucket.org/pypy/pypy/changeset/17fd1d984f1d/ Log: write a specialized function to convert args_w into a py_tuple, which sets directly c_ob_item[*] instead of going through the slowish PyTuple_SetItem diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -16,7 +16,7 @@ from pypy.module.cpyext.pyobject import ( decref, from_ref, make_ref, as_pyobj, make_typedescr) from pypy.module.cpyext.state import State -from pypy.module.cpyext.tupleobject import PyTuple_SetItem +from pypy.module.cpyext.tupleobject import tuple_from_args_w PyMethodDef = cts.gettype('PyMethodDef') PyCFunction = cts.gettype('PyCFunction') @@ -133,11 +133,7 @@ state = space.fromcache(State) w_self = self.w_self func = self.ml.c_ml_meth - n = len(args_w) - py_args = state.C.PyTuple_New(n) - for i, w_item in enumerate(args_w): - py_item = make_ref(space, w_item) - PyTuple_SetItem(space, py_args, i, py_item) + py_args = tuple_from_args_w(space, args_w) try: return generic_cpy_call(space, func, w_self, py_args) finally: diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -6,7 +6,8 @@ from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib.debug import FatalError from pypy.module.cpyext.tupleobject import ( - PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size) + PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size, + tuple_from_args_w) from pypy.module.cpyext.state import State class TestTupleObject(BaseApiTest): @@ -77,6 +78,16 @@ assert py_b == py_a assert py_b.c_ob_pypy_link == 0 + def test_tuple_from_args_w(self, space, api): + args_w = [space.newint(i) for i in (40, 41, 42)] + py_tuple = tuple_from_args_w(space, args_w) + assert py_tuple.c_ob_refcnt == 1 + assert api.PyTuple_Size(py_tuple) == 3 + py_items = [api.PyTuple_GetItem(py_tuple, i) for i in range(3)] + items = [api.PyInt_AsLong(py_obj) for py_obj in py_items] + assert items == [40, 41, 42] + decref(space, py_tuple) + def test_tuple_resize(self, space, api): state = space.fromcache(State) w_42 = space.wrap(42) diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -113,6 +113,14 @@ track_reference(space, py_obj, w_obj) return w_obj +def tuple_from_args_w(space, args_w): + state = space.fromcache(State) + n = len(args_w) + py_tuple = state.C.PyTuple_New(n) # XXX: check for errors? + py_tuple = rffi.cast(PyTupleObject, py_tuple) + for i, w_obj in enumerate(args_w): + py_tuple.c_ob_item[i] = make_ref(space, w_obj) + return rffi.cast(PyObject, py_tuple) @cpython_api([PyObject, Py_ssize_t, PyObject], rffi.INT_real, error=-1) def PyTuple_SetItem(space, ref, index, py_obj): From pypy.commits at gmail.com Mon Oct 16 18:54:37 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 15:54:37 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix test_arraymodule:test_pickle by teaching pickle about all the W_PyCFunctionObject variants Message-ID: <59e538ad.2f97df0a.f843c.ba85@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92783:371b5a6f2316 Date: 2017-10-17 00:53 +0200 http://bitbucket.org/pypy/pypy/changeset/371b5a6f2316/ Log: fix test_arraymodule:test_pickle by teaching pickle about all the W_PyCFunctionObject variants diff --git a/pypy/module/cpyext/__init__.py b/pypy/module/cpyext/__init__.py --- a/pypy/module/cpyext/__init__.py +++ b/pypy/module/cpyext/__init__.py @@ -16,11 +16,17 @@ space.fromcache(State).startup(space) method = pypy.module.cpyext.typeobject.get_new_method_def(space) # the w_self argument here is a dummy, the only thing done with w_obj - # is call space.type on it - w_obj = pypy.module.cpyext.methodobject.W_PyCFunctionObject(space, method, space.w_None) - space.appexec([space.type(w_obj)], """(methodtype): + # is call type() on it + objs_w = [cls(space, method, space.w_None) for cls in ( + pypy.module.cpyext.methodobject.W_PyCFunctionObject, + pypy.module.cpyext.methodobject.W_PyCFunctionObject_NOARGS, + pypy.module.cpyext.methodobject.W_PyCFunctionObject_O, + pypy.module.cpyext.methodobject.W_PyCFunctionObject_VARARGS)] + w_objs = space.newtuple(objs_w) + space.appexec([w_objs], """(methods): from pickle import Pickler - Pickler.dispatch[methodtype] = Pickler.save_global + for meth in methods: + Pickler.dispatch[type(meth)] = Pickler.save_global """) def register_atexit(self, function): From pypy.commits at gmail.com Mon Oct 16 19:33:04 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 16 Oct 2017 16:33:04 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix translation Message-ID: <59e541b0.090b1c0a.de929.b0eb@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92784:84feef42c1ac Date: 2017-10-17 01:32 +0200 http://bitbucket.org/pypy/pypy/changeset/84feef42c1ac/ Log: fix translation diff --git a/pypy/module/cpyext/__init__.py b/pypy/module/cpyext/__init__.py --- a/pypy/module/cpyext/__init__.py +++ b/pypy/module/cpyext/__init__.py @@ -17,11 +17,11 @@ method = pypy.module.cpyext.typeobject.get_new_method_def(space) # the w_self argument here is a dummy, the only thing done with w_obj # is call type() on it - objs_w = [cls(space, method, space.w_None) for cls in ( + objs_w = [cls(space, method, space.w_None) for cls in [ pypy.module.cpyext.methodobject.W_PyCFunctionObject, pypy.module.cpyext.methodobject.W_PyCFunctionObject_NOARGS, pypy.module.cpyext.methodobject.W_PyCFunctionObject_O, - pypy.module.cpyext.methodobject.W_PyCFunctionObject_VARARGS)] + pypy.module.cpyext.methodobject.W_PyCFunctionObject_VARARGS]] w_objs = space.newtuple(objs_w) space.appexec([w_objs], """(methods): from pickle import Pickler From pypy.commits at gmail.com Tue Oct 17 15:14:44 2017 From: pypy.commits at gmail.com (rlamy) Date: Tue, 17 Oct 2017 12:14:44 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Remove @xfail from passing test Message-ID: <59e656a4.93d31c0a.6fd06.d92a@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92785:d4c89ad97c6d Date: 2017-10-17 20:14 +0100 http://bitbucket.org/pypy/pypy/changeset/d4c89ad97c6d/ Log: Remove @xfail from passing test diff --git a/lib-python/3/ctypes/test/test_frombuffer.py b/lib-python/3/ctypes/test/test_frombuffer.py --- a/lib-python/3/ctypes/test/test_frombuffer.py +++ b/lib-python/3/ctypes/test/test_frombuffer.py @@ -85,7 +85,6 @@ del a gc.collect() # Should not crash - @xfail def test_from_buffer_copy(self): a = array.array("i", range(16)) x = (c_int * 16).from_buffer_copy(a) From pypy.commits at gmail.com Wed Oct 18 06:46:38 2017 From: pypy.commits at gmail.com (cfbolz) Date: Wed, 18 Oct 2017 03:46:38 -0700 (PDT) Subject: [pypy-commit] pypy regalloc-playground: start preparing for 32bit support Message-ID: <59e7310e.06b6df0a.5c8b5.2021@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: regalloc-playground Changeset: r92786:a98e7c269025 Date: 2017-10-18 08:56 +0200 http://bitbucket.org/pypy/pypy/changeset/a98e7c269025/ Log: start preparing for 32bit support diff --git a/rpython/jit/backend/x86/reghint.py b/rpython/jit/backend/x86/reghint.py --- a/rpython/jit/backend/x86/reghint.py +++ b/rpython/jit/backend/x86/reghint.py @@ -14,8 +14,11 @@ SAVE_GCREF_REGS, SAVE_ALL_REGS) from rpython.jit.backend.x86.regalloc import ( + X86RegisterManager, X86XMMRegisterManager, X86_64_RegisterManager, X86_64_XMMRegisterManager, compute_gc_level - ) +) + +from rpython.jit.backend.x86.arch import IS_X86_32, IS_X86_64 # tell the register allocator hints about which variables should be placed in # what registers (those are just hints, the register allocator will try its @@ -153,34 +156,44 @@ num = getattr(rop, name.upper()) oplist[num] = value +class CallHints32(object): + RegisterManager = X86RegisterManager + XMMRegisterManager = X86XMMRegisterManager + def _block_non_caller_save(self, position, save_all_regs, hinted_gpr=None, hinted_xmm=None): + if hinted_gpr is None: + hinted_gpr = [] + if hinted_xmm is None: + hinted_xmm = [] + # block all remaining registers that are not caller save -class CallHints64(object): + # XXX the case save_all_regs == SAVE_GCREF_REGS + # (save callee-save regs + gc ptrs) is no expressible atm + if save_all_regs == SAVE_ALL_REGS: + regs = self.RegisterManager.all_regs + else: + regs = self.RegisterManager.save_around_call_regs + for reg in regs: + if reg not in hinted_gpr: + self.longevity.fixed_register(position, reg) + for reg in self.XMMRegisterManager.all_regs: + if reg not in hinted_xmm: + self.longevity.fixed_register(position, reg) + def hint(self, position, args, argtypes, save_all_regs): + self._block_non_caller_save(position, save_all_regs) + + +class CallHints64(CallHints32): ARGUMENTS_GPR = [edi, esi, edx, ecx, r8, r9] ARGUMENTS_XMM = [xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7] + RegisterManager = X86_64_RegisterManager + XMMRegisterManager = X86_64_XMMRegisterManager def __init__(self, longevity): self.longevity = longevity - def _unused_gpr(self): - i = self.next_arg_gpr - self.next_arg_gpr = i + 1 - try: - res = self.ARGUMENTS_GPR[i] - except IndexError: - return None - return res - - def _unused_xmm(self): - i = self.next_arg_xmm - self.next_arg_xmm = i + 1 - try: - return self.ARGUMENTS_XMM[i] - except IndexError: - return None - def hint(self, position, args, argtypes, save_all_regs): hinted_xmm = [] hinted_gpr = [] @@ -205,17 +218,10 @@ hinted_gpr.append(tgt) hinted_args.append(arg) next_arg_gpr += 1 - # block all remaining registers that are not caller save + self._block_non_caller_save(position, save_all_regs, hinted_gpr, hinted_xmm) - # XXX the case save_all_regs == SAVE_GCREF_REGS - # (save callee-save regs + gc ptrs) is no expressible atm - if save_all_regs == SAVE_ALL_REGS: - regs = X86_64_RegisterManager.all_regs - else: - regs = X86_64_RegisterManager.save_around_call_regs - for reg in regs: - if reg not in hinted_gpr: - self.longevity.fixed_register(position, reg) - for reg in X86_64_XMMRegisterManager.all_regs: - if reg not in hinted_xmm: - self.longevity.fixed_register(position, reg) + +if IS_X86_32: + CallHints = CallHints32 +if IS_X86_64: + CallHints = CallHints64 From pypy.commits at gmail.com Wed Oct 18 06:46:40 2017 From: pypy.commits at gmail.com (cfbolz) Date: Wed, 18 Oct 2017 03:46:40 -0700 (PDT) Subject: [pypy-commit] pypy regalloc-playground: try to fix 32bit Message-ID: <59e73110.84addf0a.be752.6fdb@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: regalloc-playground Changeset: r92787:e02007d88e4b Date: 2017-10-18 12:46 +0200 http://bitbucket.org/pypy/pypy/changeset/e02007d88e4b/ Log: try to fix 32bit diff --git a/rpython/jit/backend/x86/callbuilder.py b/rpython/jit/backend/x86/callbuilder.py --- a/rpython/jit/backend/x86/callbuilder.py +++ b/rpython/jit/backend/x86/callbuilder.py @@ -436,9 +436,11 @@ self.subtract_esp_aligned(stack_depth - self.stack_max) # p = 0 + num_moves = 0 for i in range(n): loc = arglocs[i] if isinstance(loc, RegLoc): + num_moves += 1 if loc.is_xmm: self.mc.MOVSD_sx(p, loc.value) else: @@ -449,11 +451,14 @@ loc = arglocs[i] if not isinstance(loc, RegLoc): if loc.get_width() == 8: + num_moves += 2 self.mc.MOVSD(xmm0, loc) self.mc.MOVSD_sx(p, xmm0.value) elif isinstance(loc, ImmedLoc): + num_moves += 1 self.mc.MOV_si(p, loc.value) else: + num_moves += 2 self.mc.MOV(eax, loc) self.mc.MOV_sr(p, eax.value) p += loc.get_width() @@ -461,7 +466,7 @@ # if not self.fnloc_is_immediate: # the last "argument" pushed above self.fnloc = RawEspLoc(p - WORD, INT) - + self.num_moves = num_moves def emit_raw_call(self): if stdcall_or_cdecl and self.is_call_release_gil: diff --git a/rpython/jit/backend/x86/test/test_regalloc.py b/rpython/jit/backend/x86/test/test_regalloc.py --- a/rpython/jit/backend/x86/test/test_regalloc.py +++ b/rpython/jit/backend/x86/test/test_regalloc.py @@ -4,6 +4,7 @@ from rpython.jit.backend.llsupport.test import test_regalloc_integration from rpython.jit.backend.x86.assembler import Assembler386 +from rpython.jit.backend.x86.arch import IS_X86_64 class LogEntry(object): def __init__(self, position, name, *args): @@ -59,7 +60,7 @@ self._log("jump", self._regalloc.final_jump_op.getdescr()._x86_arglocs) return Assembler386.closing_jump(self, jump_target_descr) -class TestCheckRegistersExplicitly(test_regalloc_integration.BaseTestRegalloc): +class BaseTestCheckRegistersExplicitly(test_regalloc_integration.BaseTestRegalloc): def setup_class(cls): cls.cpu.assembler = LoggingAssembler(cls.cpu, False) cls.cpu.assembler.setup_once() @@ -74,6 +75,7 @@ def filter_log_moves(self): return [entry for entry in self.log if entry.name == "mov"] +class TestCheckRegistersExplicitly(BaseTestCheckRegistersExplicitly): def test_unused(self): ops = ''' [i0, i1, i2, i3] @@ -121,22 +123,6 @@ # i0 and i1, one for the result to the stack assert len(self.filter_log_moves()) == 3 - def test_call_use_argument_twice(self): - ops = ''' - [i0, i1, i2, i3] - i7 = int_add(i0, i1) - i8 = int_add(i2, 13) - i9 = call_i(ConstClass(f2ptr), i7, i7, descr=f2_calldescr) - i10 = int_is_true(i9) - guard_true(i10) [i8] - finish(i9) - ''' - self.interpret(ops, [5, 6, 7, 8]) - # two moves are needed from the stack frame to registers for arguments - # i0 and i1, one for the result to the stack - # one for the copy to the other argument register - assert len(self.filter_log_moves()) == 4 - @pytest.mark.skip("later") def test_same_stack_entry_many_times(self): ops = ''' @@ -227,39 +213,6 @@ # that would break coalescing between i7 and i9) assert op.args[1][0] is add1.args[-1] - def test_coalescing_float(self): - ops = ''' - [f0, f1, f3] - f7 = float_add(f0, f1) - f8 = float_add(f7, f3) - f9 = call_f(ConstClass(ffptr), f8, 1.0, descr=ff_calldescr) - i10 = float_ne(f9, 0.0) - guard_true(i10) [] - finish(f9) - ''' - self.interpret(ops, [5.0, 6.0, 8.0]) - assert len(self.filter_log_moves()) == 3 - - def test_malloc(self, monkeypatch): - ops = ''' - [i0] - i1 = int_add(i0, 1) # this is using ecx or edx because it fits - i6 = int_add(i0, 6) # this is using ecx or edx because it fits - i2 = int_add(i6, i1) - p0 = call_malloc_nursery(16) - gc_store(p0, 0, 83944, 8) - gc_store(p0, 8, i2, 8) - i10 = int_is_true(i2) - guard_true(i10) [p0, i0] - finish(p0) - ''' - monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_top_addr", lambda: 61) - monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_free_addr", lambda: 68) - self.interpret(ops, [0], run=False) - # 2 moves, because the call_malloc_nursery hints prevent using ecx and - # edx for any of the integer results - assert len(self.filter_log_moves()) == 2 - def test_jump_hinting(self): self.targettoken._ll_loop_code = 0 ops = ''' @@ -366,6 +319,60 @@ # 4 moves, three for args, one for result assert len(self.filter_log_moves()) == 4 +class TestCheckRegistersExplicitly64(BaseTestCheckRegistersExplicitly): + def setup_class(self): + if not IS_X86_64: + pytest.skip("needs 64 bit") + + def test_call_use_argument_twice(self): + ops = ''' + [i0, i1, i2, i3] + i7 = int_add(i0, i1) + i8 = int_add(i2, 13) + i9 = call_i(ConstClass(f2ptr), i7, i7, descr=f2_calldescr) + i10 = int_is_true(i9) + guard_true(i10) [i8] + finish(i9) + ''' + self.interpret(ops, [5, 6, 7, 8]) + # two moves are needed from the stack frame to registers for arguments + # i0 and i1, one for the result to the stack + # one for the copy to the other argument register + assert len(self.filter_log_moves()) == 4 + + def test_coalescing_float(self): + ops = ''' + [f0, f1, f3] + f7 = float_add(f0, f1) + f8 = float_add(f7, f3) + f9 = call_f(ConstClass(ffptr), f8, 1.0, descr=ff_calldescr) + i10 = float_ne(f9, 0.0) + guard_true(i10) [] + finish(f9) + ''' + self.interpret(ops, [5.0, 6.0, 8.0]) + assert len(self.filter_log_moves()) == 3 + + def test_malloc(self, monkeypatch): + ops = ''' + [i0] + i1 = int_add(i0, 1) # this is using ecx or edx because it fits + i6 = int_add(i0, 6) # this is using ecx or edx because it fits + i2 = int_add(i6, i1) + p0 = call_malloc_nursery(16) + gc_store(p0, 0, 83944, 8) + gc_store(p0, 8, i2, 8) + i10 = int_is_true(i2) + guard_true(i10) [p0, i0] + finish(p0) + ''' + monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_top_addr", lambda: 61) + monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_free_addr", lambda: 68) + self.interpret(ops, [0], run=False) + # 2 moves, because the call_malloc_nursery hints prevent using ecx and + # edx for any of the integer results + assert len(self.filter_log_moves()) == 2 + def test_dict_lookup(self, monkeypatch): monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_top_addr", lambda: 61) monkeypatch.setattr(self.cpu.gc_ll_descr, "get_nursery_free_addr", lambda: 68) From pypy.commits at gmail.com Wed Oct 18 08:20:05 2017 From: pypy.commits at gmail.com (cfbolz) Date: Wed, 18 Oct 2017 05:20:05 -0700 (PDT) Subject: [pypy-commit] pypy regalloc-playground: merge default Message-ID: <59e746f5.cc981c0a.802bd.5bf7@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: regalloc-playground Changeset: r92788:c408f259eed1 Date: 2017-10-18 14:19 +0200 http://bitbucket.org/pypy/pypy/changeset/c408f259eed1/ Log: merge default diff too long, truncating to 2000 out of 10381 lines diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -71,6 +71,8 @@ ^lib_pypy/.+.c$ ^lib_pypy/.+.o$ ^lib_pypy/.+.so$ +^lib_pypy/.+.pyd$ +^lib_pypy/Release/ ^pypy/doc/discussion/.+\.html$ ^include/.+\.h$ ^include/.+\.inl$ diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/lib_pypy/_tkinter/app.py b/lib_pypy/_tkinter/app.py --- a/lib_pypy/_tkinter/app.py +++ b/lib_pypy/_tkinter/app.py @@ -119,7 +119,7 @@ tklib.TCL_GLOBAL_ONLY) # This is used to get the application class for Tk 4.1 and up - argv0 = className.lower() + argv0 = className.lower().encode('ascii') tklib.Tcl_SetVar(self.interp, "argv0", argv0, tklib.TCL_GLOBAL_ONLY) diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/release-v5.9.0.rst b/pypy/doc/release-v5.9.0.rst --- a/pypy/doc/release-v5.9.0.rst +++ b/pypy/doc/release-v5.9.0.rst @@ -10,18 +10,24 @@ This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 includes the upstream stdlib version 3.5.3. -Only a handful of failing tests remain in NumPy and Pandas on PyPy2.7, issues -that appeared as excessive memory use were cleared up and other incompatibilities -were resolved. +NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues +that appeared as excessive memory +use were cleared up and other incompatibilities were resolved. The C-API +compatibility layer does slow down code which crosses the python-c interface +often, we have ideas on how it could be improved, and still recommend +using pure python on PyPy or interfacing via CFFI_. Many other modules +based on C-API exentions now work on PyPy as well. -Cython 0.27 (released last week) should support more projects with PyPy, both -on PyPy2.7 and PyPy3.5 beta. +Cython 0.27.1 (released very recently) supports more projects with PyPy, both +on PyPy2.7 and PyPy3.5 beta. Note version **0.27.1** is now the minimum +version that supports this version of PyPy, due to some interactions with +updated C-API interface code. We optimized the JSON parser for recurring string keys, which should decrease memory use to 50% and increase parsing speed by up to 15% for large JSON files with many repeating dictionary keys (which is quite common). -CFFI_, which is part of the PyPy release, has been updated to 1.11, +CFFI_, which is part of the PyPy release, has been updated to 1.11.1, improving an already great package for interfacing with C. CFFI now supports complex arguments in API mode, as well as ``char16_t`` and ``char32_t`` and has improved support for callbacks. @@ -145,6 +151,7 @@ * Issue 2590_: fix the bounds in the GC when allocating a lot of objects with finalizers * Replace magical NOT RPYTHON comment with a decorator * Implement ``socket.sendmsg()``/``.recvmsg()`` for py3.5 + * Add ``memory_pressure`` for ``_SSLSocket`` objects * Degredations @@ -163,7 +170,8 @@ * Add support for ``_PyNamespace_New``, ``PyMemoryView_FromMemory``, ``Py_EnterRecursiveCall`` raising RecursionError, ``PyObject_LengthHint``, - ``PyUnicode_FromKindAndData``, ``PyDict_SetDefault``, ``PyGenObject`` + ``PyUnicode_FromKindAndData``, ``PyDict_SetDefault``, ``PyGenObject``, + ``PyGenObject``, ``PyUnicode_Substring``, ``PyLong_FromUnicodeObject`` * Implement ``PyType_FromSpec`` (PEP 384) and fix issues with PEP 489 support * Support the new version of ``os.stat()`` on win32 * Use ``stat3()`` on Posix diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -3,4 +3,10 @@ =========================== .. this is a revision shortly after release-pypy2.7-v5.9.0 -.. startrev:899e5245de1e +.. startrev:d56dadcef996 + +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging + +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst --- a/pypy/doc/whatsnew-pypy2-5.9.0.rst +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -85,3 +85,12 @@ .. branch: py_ssize_t Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/cffi1_module.py b/pypy/module/_cffi_backend/cffi1_module.py --- a/pypy/module/_cffi_backend/cffi1_module.py +++ b/pypy/module/_cffi_backend/cffi1_module.py @@ -1,4 +1,5 @@ from rpython.rtyper.lltypesystem import lltype, rffi +from rpython.rlib import jit from pypy.interpreter.error import oefmt from pypy.interpreter.module import Module @@ -15,7 +16,7 @@ INITFUNCPTR = lltype.Ptr(lltype.FuncType([rffi.VOIDPP], lltype.Void)) - + at jit.dont_look_inside def load_cffi1_module(space, name, path, initptr): # This is called from pypy.module.cpyext.api.load_extension_module() from pypy.module._cffi_backend.call_python import get_ll_cffi_call_python diff --git a/pypy/module/_cffi_backend/ctypeptr.py b/pypy/module/_cffi_backend/ctypeptr.py --- a/pypy/module/_cffi_backend/ctypeptr.py +++ b/pypy/module/_cffi_backend/ctypeptr.py @@ -156,10 +156,11 @@ class W_CTypePtrBase(W_CTypePtrOrArray): # base class for both pointers and pointers-to-functions - _attrs_ = ['is_void_ptr', 'is_voidchar_ptr'] - _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr'] + _attrs_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] + _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] is_void_ptr = False is_voidchar_ptr = False + is_onebyte_ptr = False def convert_to_object(self, cdata): ptrdata = rffi.cast(rffi.CCHARPP, cdata)[0] @@ -179,12 +180,20 @@ if self.is_void_ptr or other.is_void_ptr: pass # cast from or to 'void *' elif self.is_voidchar_ptr or other.is_voidchar_ptr: - space = self.space - msg = ("implicit cast from '%s' to '%s' " - "will be forbidden in the future (check that the types " - "are as you expect; use an explicit ffi.cast() if they " - "are correct)" % (other.name, self.name)) - space.warn(space.newtext(msg), space.w_UserWarning) + # for backward compatibility, accept "char *" as either + # source of target. This is not what C does, though, + # so emit a warning that will eventually turn into an + # error. The warning is turned off if both types are + # pointers to single bytes. + if self.is_onebyte_ptr and other.is_onebyte_ptr: + pass # no warning + else: + space = self.space + msg = ("implicit cast from '%s' to '%s' " + "will be forbidden in the future (check that the types " + "are as you expect; use an explicit ffi.cast() if they " + "are correct)" % (other.name, self.name)) + space.warn(space.newtext(msg), space.w_UserWarning) else: raise self._convert_error("compatible pointer", w_ob) @@ -214,6 +223,7 @@ self.is_void_ptr = isinstance(ctitem, ctypevoid.W_CTypeVoid) self.is_voidchar_ptr = (self.is_void_ptr or isinstance(ctitem, ctypeprim.W_CTypePrimitiveChar)) + self.is_onebyte_ptr = (ctitem.size == 1) W_CTypePtrBase.__init__(self, space, size, extra, 2, ctitem) def newp(self, w_init, allocator): diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): @@ -2099,7 +2099,8 @@ if sys.platform.startswith("linux"): BWChar = new_primitive_type("wchar_t") assert sizeof(BWChar) == 4 - assert int(cast(BWChar, -1)) == -1 # signed, on linux + # wchar_t is often signed on Linux, but not always (e.g. on ARM) + assert int(cast(BWChar, -1)) in (-1, 4294967295) def test_char16(): BChar16 = new_primitive_type("char16_t") @@ -3903,9 +3904,11 @@ BCharP = new_pointer_type(new_primitive_type("char")) BIntP = new_pointer_type(new_primitive_type("int")) BVoidP = new_pointer_type(new_void_type()) + BUCharP = new_pointer_type(new_primitive_type("unsigned char")) z1 = cast(BCharP, 0) z2 = cast(BIntP, 0) z3 = cast(BVoidP, 0) + z4 = cast(BUCharP, 0) with warnings.catch_warnings(record=True) as w: newp(new_pointer_type(BIntP), z1) # warn assert len(w) == 1 @@ -3919,6 +3922,12 @@ assert len(w) == 2 newp(new_pointer_type(BIntP), z3) # fine assert len(w) == 2 + newp(new_pointer_type(BCharP), z4) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z1) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z3) # fine + assert len(w) == 2 # check that the warnings are associated with lines in this file assert w[1].lineno == w[0].lineno + 4 diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,20 +66,17 @@ "position %d from error handler out of bounds", newpos) replace = space.unicode_w(w_replace) - return replace, newpos + if decode: + return replace, newpos + else: + return replace, None, newpos return call_errorhandler def make_decode_errorhandler(self, space): return self._make_errorhandler(space, True) def make_encode_errorhandler(self, space): - errorhandler = self._make_errorhandler(space, False) - def encode_call_errorhandler(errors, encoding, reason, input, startpos, - endpos): - replace, newpos = errorhandler(errors, encoding, reason, input, - startpos, endpos) - return replace, None, newpos - return encode_call_errorhandler + return self._make_errorhandler(space, False) def get_unicodedata_handler(self, space): if self.unicodedata_handler: diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'math/mathcore/src/Fitter.cxx' -if os.path.exists(inp): - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp): - if '#include "TF1.h"' in line: - continue - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - -# done -os.chdir(os.path.pardir) - -# debugging: run a test build -if DEBUG_TESTBUILD: - print('running a debug test build') - tb = "test_builddir" - if os.path.exists(tb): - shutil.rmtree(tb) - os.mkdir(tb) - os.chdir(tb) - os.system('cmake ../%s -DCMAKE_INSTALL_PREFIX=../install -Dminimal=ON -Dasimage=OFF' % pkgdir) - os.system('make -j 32') - - -# -## package creation -# -countdown = 0 -pidir = 'Package-'+args.release -print('creating package', pidir) -if not os.path.exists(pidir): - os.mkdir(pidir) -os.chdir(pidir); countdown += 1 - -print('creating LICENSE.txt') -with open('LICENSE.txt', 'w') as outp: - outp.write("""There are three main parts: - - LLVM: distributed under University of Illinois/NCSA Open Source License - https://opensource.org/licenses/UoI-NCSA.php - ROOT: distributed under LGPL 2.1 - https://root.cern.ch/license - Cppyy: distributed under LBNL BSD - https://fedoraproject.org/wiki/Licensing/LBNLBSD -""") - -print('creating MANIFEST.in') -with open('MANIFEST.in', 'w') as outp: - outp.write("""# Include the license file -include LICENSE.txt - -# Include the data files -recursive-include src * -""") - -print('creating README.rst') -with open('README.rst', 'w') as outp: - outp.write("""PyPy cling-support -================== - ----- - -Find the documentation here: - http://doc.pypy.org/en/latest/cppyy.html -""") - -print('creating setup.cfg') -with open('setup.cfg', 'w') as outp: - outp.write("""[bdist_wheel] -universal=0 -""") - -print('creating setup.py') -with open('setup.py', 'w') as outp: - outp.write("""import os, sys, subprocess -from setuptools import setup, find_packages -from distutils import log -from distutils.command.build import build as _build -from setuptools.command.install import install as _install -from distutils.sysconfig import get_python_lib -from distutils.errors import DistutilsSetupError -from codecs import open - -here = os.path.abspath(os.path.dirname(__file__)) -with open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: - long_description = f.read() - -builddir = None -def get_builddir(): - global builddir - if builddir is None: - topdir = os.getcwd() - builddir = os.path.join(topdir, 'builddir') - return builddir - -srcdir = None -def get_srcdir(): - global srcdir - if srcdir is None: - topdir = os.getcwd() - srcdir = os.path.join(topdir, 'src', 'backend') - return srcdir - -class my_cmake_build(_build): - def __init__(self, dist, *args, **kwargs): - _build.__init__(self, dist, *args, **kwargs) - # TODO: can't seem to find a better way of getting hold of - # the install_lib parameter during the build phase ... - prefix = '' - try: - prefix = dist.get_command_obj('install').install_lib - except AttributeError: - pass - if not prefix: - prefix = get_python_lib(1, 0) - self.prefix = os.path.join(prefix, 'cppyy_backend') - - def run(self): - # base run - _build.run(self) - - # custom run - log.info('Now building libcppyy_backend.so and dependencies') - builddir = get_builddir() - srcdir = get_srcdir() - if not os.path.exists(builddir): - log.info('Creating build directory %s ...' % builddir) - os.makedirs(builddir) - - os.chdir(builddir) - log.info('Running cmake for cppyy_backend') - if subprocess.call([ - 'cmake', srcdir, '-Dminimal=ON -Dasimage=OFF', - '-DCMAKE_INSTALL_PREFIX='+self.prefix]) != 0: - raise DistutilsSetupError('Failed to configure cppyy_backend') - - nprocs = os.getenv("MAKE_NPROCS") - if nprocs: - try: - ival = int(nprocs) - nprocs = '-j'+nprocs - except ValueError: - log.warn("Integer expected for MAKE_NPROCS, but got %s (ignored)", nprocs) - nprocs = '-j1' - else: - nprocs = '-j1' - log.info('Now building cppyy_backend and dependencies ...') - if subprocess.call(['make', nprocs]) != 0: - raise DistutilsSetupError('Failed to build cppyy_backend') - - log.info('build finished') - -class my_libs_install(_install): - def run(self): - # base install - _install.run(self) - - # custom install - log.info('Now installing libcppyy_backend.so and dependencies') - builddir = get_builddir() - if not os.path.exists(builddir): - raise DistutilsSetupError('Failed to find build dir!') - os.chdir(builddir) - - prefix = self.install_lib - log.info('Now installing in %s ...', prefix) - if subprocess.call(['make', 'install']) != 0: - raise DistutilsSetupError('Failed to install cppyy_backend') - - log.info('install finished') - - def get_outputs(self): - outputs = _install.get_outputs(self) - outputs.append(os.path.join(self.install_lib, 'cppyy_backend')) - return outputs - -setup( - name='PyPy-cppyy-backend', -""") - outp.write(" version='%s', # corresponds to ROOT %s, extra number is for packager\n"\ - % (args.version, args.release)) - outp.write(""" description='Cling support for PyPy', - long_description=long_description, - - url='http://pypy.org', - - # Author details - author='PyPy Developers', - author_email='pypy-dev at python.org', - - license='LLVM: UoI-NCSA; ROOT: LGPL 2.1; Cppyy: LBNL BSD', - - classifiers=[ - 'Development Status :: 4 - Beta', - - 'Intended Audience :: Developers', - - 'Topic :: Software Development', - 'Topic :: Software Development :: Interpreters', - - #'License :: OSI Approved :: MIT License', - - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: Implementation :: PyPy', - 'Programming Language :: C', - 'Programming Language :: C++', - - 'Natural Language :: English' - ], - - keywords='interpreter development', - - packages=find_packages('src', ['backend']), - include_package_data=True, - - extras_require={ - }, - - cmdclass = { - 'build': my_cmake_build, - 'install': my_libs_install, - }, -) -""") - - -print('creating src ... ROOT part') -if not os.path.exists('src'): - os.mkdir('src') -os.chdir('src'); countdown += 1 -if not os.path.exists('backend'): - src = os.path.join(os.path.pardir, os.path.pardir, pkgdir) - print('now copying', src) - shutil.copytree(src, 'backend') - -print('creating src ... cppyy part') -os.chdir('backend'); countdown += 1 -if not os.path.exists('cppyy'): - os.mkdir('cppyy') - os.chdir('cppyy'); countdown += 1 - - with open('CMakeLists.txt', 'w') as outp: - outp.write("""############################################################################ -# CMakeLists.txt file for building cppyy package -############################################################################ - -ROOT_GLOB_SOURCES(sources ${CMAKE_CURRENT_SOURCE_DIR}/src/*.cxx) -set_source_files_properties(${sources} COMPILE_FLAGS "-fomit-frame-pointer -fvisibility=hidden -DRPY_EXTERN=RPY_EXPORTED -DRPYTHON_LL2CTYPES") - -add_definitions(${CLING_CXXFLAGS}) - -ROOT_LINKER_LIBRARY(cppyy_backend ${sources} - LIBRARIES ${CMAKE_DL_LIBS} - DEPENDENCIES Core Cling RIO Thread) - -add_dependencies(cppyy_backend CLING) -""") - - os.mkdir('src') - os.chdir('src'); countdown += 1 - print('pulling cppyy/clingcwrapper.cxx from pypy') - base = 'https://bitbucket.org/pypy/pypy/raw/default/pypy/module/cppyy/' - for cppyy_file in ['src/callcontext.h', 'include/capi.h', 'src/clingcwrapper.cxx', - 'include/clingcwrapper.h', 'include/cpp_cppyy.h', 'include/cppyy.h']: - resp = urllib2.urlopen(base+cppyy_file) - with open(os.path.basename(cppyy_file), 'w') as outp: - outp.write(resp.read()) - - # fix include - inp = 'capi.h' - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if 'src/precommondefs.h' in line: - line = '#include "precommondefs.h"\n' - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - - with open('precommondefs.h', 'w') as outp: - outp.write("""/***** Start of precommondefs.h *****/ - -/* This is extracted from pyconfig.h from CPython. It sets the macros - that affect the features we get from system include files. - It must not #include anything. */ - -#ifndef __PYPY_PRECOMMONDEFS_H -#define __PYPY_PRECOMMONDEFS_H - - -/* Define on Darwin to activate all library features */ -#define _DARWIN_C_SOURCE 1 -/* This must be set to 64 on some systems to enable large file support. */ -#define _FILE_OFFSET_BITS 64 -/* Define on Linux to activate all library features */ -#define _GNU_SOURCE 1 -/* This must be defined on some systems to enable large file support. */ -#define _LARGEFILE_SOURCE 1 -/* Define on NetBSD to activate all library features */ -#define _NETBSD_SOURCE 1 -/* Define to activate features from IEEE Stds 1003.1-2001 */ -#ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L -#endif -/* Define on FreeBSD to activate all library features */ -#define __BSD_VISIBLE 1 -#define __XSI_VISIBLE 700 -/* Windows: winsock/winsock2 mess */ -#define WIN32_LEAN_AND_MEAN -#ifdef _WIN64 - typedef __int64 Signed; - typedef unsigned __int64 Unsigned; -# define SIGNED_MIN LLONG_MIN -#else - typedef long Signed; - typedef unsigned long Unsigned; -# define SIGNED_MIN LONG_MIN -#endif - -#if !defined(RPY_ASSERT) && !defined(RPY_LL_ASSERT) && !defined(NDEBUG) -# define NDEBUG -#endif - - -/* All functions and global variables declared anywhere should use - one of the following attributes: - - RPY_EXPORTED: the symbol is exported out of libpypy-c.so. - - RPY_EXTERN: the symbol is not exported out of libpypy-c.so, but - otherwise works like 'extern' by being available to - other C sources. - - static: as usual, this means the symbol is local to this C file. - - Don't use _RPY_HIDDEN directly. For tests involving building a custom - .so, translator/tool/cbuild.py overrides RPY_EXTERN so that it becomes - equal to RPY_EXPORTED. - - Any function or global variable declared with no attribute at all is - a bug; please report or fix it. -*/ -#ifdef __GNUC__ -# define RPY_EXPORTED extern __attribute__((visibility("default"))) -# define _RPY_HIDDEN __attribute__((visibility("hidden"))) -#else -# define RPY_EXPORTED extern __declspec(dllexport) -# define _RPY_HIDDEN /* nothing */ -#endif -#ifndef RPY_EXTERN -# define RPY_EXTERN extern _RPY_HIDDEN -#endif - - -#endif /* __PYPY_PRECOMMONDEFS_H */ - -/***** End of precommondefs.h *****/ -""") - -# back up to pip package top -for i in range(countdown-1): - os.chdir(os.path.pardir) - -# add cppyy module to cmake -os.chdir('src/backend') -inp = 'CMakeLists.txt' -print('adding cppyy to cmake') -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'add_subdirectory' in line and 'net' in line: - line += 'add_subdirectory (cppyy)\n' - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# done! diff --git a/pypy/module/_cppyy/bench/Makefile b/pypy/module/_cppyy/bench/Makefile deleted file mode 100644 --- a/pypy/module/_cppyy/bench/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -all: bench02Dict_reflex.so - -ROOTSYS := ${ROOTSYS} - -ifeq ($(ROOTSYS),) - genreflex=genreflex - cppflags= -else - genreflex=$(ROOTSYS)/bin/genreflex - cppflags=-I$(ROOTSYS)/include -L$(ROOTSYS)/lib -endif - -PLATFORM := $(shell uname -s) -ifeq ($(PLATFORM),Darwin) - cppflags+=-dynamiclib -single_module -arch x86_64 -endif - -ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),) - genreflexflags= - cppflags2=-O3 -fPIC -else - genreflexflags=--with-methptrgetter - cppflags2=-Wno-pmf-conversions -O3 -fPIC -endif - - -bench02Dict_reflex.so: bench02.h bench02.cxx bench02.xml - $(genreflex) bench02.h $(genreflexflags) --selection=bench02.xml -I$(ROOTSYS)/include - g++ -o $@ bench02.cxx bench02_rflx.cpp -I$(ROOTSYS)/include -shared -std=c++11 -lHistPainter `root-config --libs` $(cppflags) $(cppflags2) diff --git a/pypy/module/_cppyy/bench/bench02.cxx b/pypy/module/_cppyy/bench/bench02.cxx deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.cxx +++ /dev/null @@ -1,79 +0,0 @@ -#include "bench02.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TDirectory.h" -#include "TInterpreter.h" -#include "TSystem.h" -#include "TBenchmark.h" -#include "TStyle.h" -#include "TError.h" -#include "Getline.h" -#include "TVirtualX.h" - -#include "Api.h" - -#include - -TClass *TClass::GetClass(const char*, Bool_t, Bool_t) { - static TClass* dummy = new TClass("__dummy__", kTRUE); - return dummy; // is deleted by gROOT at shutdown -} - -class TTestApplication : public TApplication { -public: - TTestApplication( - const char* acn, Int_t* argc, char** argv, Bool_t bLoadLibs = kTRUE); - virtual ~TTestApplication(); -}; - -TTestApplication::TTestApplication( - const char* acn, int* argc, char** argv, bool do_load) : TApplication(acn, argc, argv) { - if (do_load) { - // follow TRint to minimize differences with CINT - ProcessLine("#include ", kTRUE); - ProcessLine("#include <_string>", kTRUE); // for std::string iostream. - ProcessLine("#include ", kTRUE); // needed because they're used within the - ProcessLine("#include ", kTRUE); // core ROOT dicts and CINT won't be able - // to properly unload these files - } - - // save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // prevent crashes on accessing history - Gl_histinit((char*)"-"); - - // prevent ROOT from exiting python - SetReturnFromRun(kTRUE); -} - -TTestApplication::~TTestApplication() {} - -static const char* appname = "pypy-cppyy"; - -Bench02RootApp::Bench02RootApp() { - gROOT->SetBatch(kTRUE); - if (!gApplication) { - int argc = 1; - char* argv[1]; argv[0] = (char*)appname; - gApplication = new TTestApplication(appname, &argc, argv, kFALSE); - } -} - -Bench02RootApp::~Bench02RootApp() { - // TODO: ROOT globals cleanup ... (?) -} - -void Bench02RootApp::report() { - std::cout << "gROOT is: " << gROOT << std::endl; - std::cout << "gApplication is: " << gApplication << std::endl; -} - -void Bench02RootApp::close_file(TFile* f) { - std::cout << "closing file " << f->GetName() << " ... " << std::endl; - f->Write(); - f->Close(); - std::cout << "... file closed" << std::endl; -} diff --git a/pypy/module/_cppyy/bench/bench02.h b/pypy/module/_cppyy/bench/bench02.h deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.h +++ /dev/null @@ -1,72 +0,0 @@ -#include "TString.h" - -#include "TCanvas.h" -#include "TFile.h" -#include "TProfile.h" -#include "TNtuple.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TRandom.h" -#include "TRandom3.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TSystem.h" - -#include "TArchiveFile.h" -#include "TBasket.h" -#include "TBenchmark.h" -#include "TBox.h" -#include "TBranchRef.h" -#include "TBrowser.h" -#include "TClassGenerator.h" -#include "TClassRef.h" -#include "TClassStreamer.h" -#include "TContextMenu.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TF1.h" -#include "TFileCacheRead.h" -#include "TFileCacheWrite.h" -#include "TFileMergeInfo.h" -#include "TFitResult.h" -#include "TFolder.h" -//#include "TFormulaPrimitive.h" -#include "TFunction.h" -#include "TFrame.h" -#include "TGlobal.h" -#include "THashList.h" -#include "TInetAddress.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TLegend.h" -#include "TMethodCall.h" -#include "TPluginManager.h" -#include "TProcessUUID.h" -#include "TSchemaRuleSet.h" -#include "TStyle.h" -#include "TSysEvtHandler.h" -#include "TTimer.h" -#include "TView.h" -//#include "TVirtualCollectionProxy.h" -#include "TVirtualFFT.h" -#include "TVirtualHistPainter.h" -#include "TVirtualIndex.h" -#include "TVirtualIsAProxy.h" -#include "TVirtualPadPainter.h" -#include "TVirtualRefProxy.h" -#include "TVirtualStreamerInfo.h" -#include "TVirtualViewer3D.h" - -#include -#include - - -class Bench02RootApp { -public: - Bench02RootApp(); - ~Bench02RootApp(); - - void report(); - void close_file(TFile* f); -}; diff --git a/pypy/module/_cppyy/bench/bench02.xml b/pypy/module/_cppyy/bench/bench02.xml deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pypy/module/_cppyy/bench/hsimple.C b/pypy/module/_cppyy/bench/hsimple.C deleted file mode 100644 --- a/pypy/module/_cppyy/bench/hsimple.C +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -TFile *hsimple(Int_t get=0) -{ -// This program creates : -// - a one dimensional histogram -// - a two dimensional histogram -// - a profile histogram -// - a memory-resident ntuple -// -// These objects are filled with some random numbers and saved on a file. -// If get=1 the macro returns a pointer to the TFile of "hsimple.root" -// if this file exists, otherwise it is created. -// The file "hsimple.root" is created in $ROOTSYS/tutorials if the caller has -// write access to this directory, otherwise the file is created in $PWD - - TString filename = "hsimple.root"; - TString dir = gSystem->UnixPathName(gInterpreter->GetCurrentMacroName()); - dir.ReplaceAll("hsimple.C",""); - dir.ReplaceAll("/./","/"); - TFile *hfile = 0; - if (get) { - // if the argument get =1 return the file "hsimple.root" - // if the file does not exist, it is created - TString fullPath = dir+"hsimple.root"; - if (!gSystem->AccessPathName(fullPath,kFileExists)) { - hfile = TFile::Open(fullPath); //in $ROOTSYS/tutorials - if (hfile) return hfile; - } - //otherwise try $PWD/hsimple.root - if (!gSystem->AccessPathName("hsimple.root",kFileExists)) { - hfile = TFile::Open("hsimple.root"); //in current dir - if (hfile) return hfile; - } - } - //no hsimple.root file found. Must generate it ! - //generate hsimple.root in $ROOTSYS/tutorials if we have write access - if (!gSystem->AccessPathName(dir,kWritePermission)) { - filename = dir+"hsimple.root"; - } else if (!gSystem->AccessPathName(".",kWritePermission)) { - //otherwise generate hsimple.root in the current directory - } else { - printf("you must run the script in a directory with write access\n"); - return 0; - } - hfile = (TFile*)gROOT->FindObject(filename); if (hfile) hfile->Close(); - hfile = new TFile(filename,"RECREATE","Demo ROOT file with histograms"); - - // Create some histograms, a profile histogram and an ntuple - TH1F *hpx = new TH1F("hpx","This is the px distribution",100,-4,4); - hpx->SetFillColor(48); - TH2F *hpxpy = new TH2F("hpxpy","py vs px",40,-4,4,40,-4,4); - TProfile *hprof = new TProfile("hprof","Profile of pz versus px",100,-4,4,0,20); - TNtuple *ntuple = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - - gBenchmark->Start("hsimple"); - - // Create a new canvas. - TCanvas *c1 = new TCanvas("c1","Dynamic Filling Example",200,10,700,500); - c1->SetFillColor(42); - c1->GetFrame()->SetFillColor(21); - c1->GetFrame()->SetBorderSize(6); - c1->GetFrame()->SetBorderMode(-1); - - - // Fill histograms randomly - TRandom3 random; - Float_t px, py, pz; - const Int_t kUPDATE = 1000; - for (Int_t i = 0; i < 50000; i++) { - // random.Rannor(px,py); - px = random.Gaus(0, 1); - py = random.Gaus(0, 1); - pz = px*px + py*py; - Float_t rnd = random.Rndm(1); - hpx->Fill(px); - hpxpy->Fill(px,py); - hprof->Fill(px,pz); - ntuple->Fill(px,py,pz,rnd,i); - if (i && (i%kUPDATE) == 0) { - if (i == kUPDATE) hpx->Draw(); - c1->Modified(); - c1->Update(); - if (gSystem->ProcessEvents()) - break; - } - } - gBenchmark->Show("hsimple"); - - // Save all objects in this file - hpx->SetFillColor(0); - hfile->Write(); - hpx->SetFillColor(48); - c1->Modified(); - return hfile; - -// Note that the file is automatically close when application terminates -// or when the file destructor is called. -} diff --git a/pypy/module/_cppyy/bench/hsimple.py b/pypy/module/_cppyy/bench/hsimple.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple.py +++ /dev/null @@ -1,110 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -_reflex = True # to keep things equal, set to False for full macro - -try: - import cppyy, random - - if not hasattr(cppyy.gbl, 'gROOT'): - cppyy.load_reflection_info('bench02Dict_reflex.so') - _reflex = True - - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom3 = cppyy.gbl.TRandom3 - - gROOT = cppyy.gbl.gROOT - gBenchmark = cppyy.gbl.TBenchmark() - gSystem = cppyy.gbl.gSystem - -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom3 - from ROOT import gROOT, gBenchmark, gSystem - import random - -if _reflex: - gROOT.SetBatch(True) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -if not _reflex: - hfile = gROOT.FindObject('hsimple.root') - if hfile: - hfile.Close() - hfile = TFile('hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.SetFillColor(48) -hpxpy = TH2F('hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4) -hprof = TProfile('hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20) -if not _reflex: - ntuple = TNtuple('ntuple', 'Demo ntuple', 'px:py:pz:random:i') - -gBenchmark.Start('hsimple') - -# Create a new canvas, and customize it. -c1 = TCanvas('c1', 'Dynamic Filling Example', 200, 10, 700, 500) -c1.SetFillColor(42) -c1.GetFrame().SetFillColor(21) -c1.GetFrame().SetBorderSize(6) -c1.GetFrame().SetBorderMode(-1) - -# Fill histograms randomly. -random = TRandom3() -kUPDATE = 1000 -for i in xrange(50000): - # Generate random numbers -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) - pz = px*px + py*py -# rnd = random.random() - rnd = random.Rndm(1) - - # Fill histograms - hpx.Fill(px) - hpxpy.Fill(px, py) - hprof.Fill(px, pz) - if not _reflex: - ntuple.Fill(px, py, pz, rnd, i) - - # Update display every kUPDATE events - if i and i%kUPDATE == 0: - if i == kUPDATE: - hpx.Draw() - - c1.Modified(True) - c1.Update() - - if gSystem.ProcessEvents(): # allow user interrupt - break - -gBenchmark.Show( 'hsimple' ) - -# Save all objects in this file -hpx.SetFillColor(0) -if not _reflex: - hfile.Write() -hpx.SetFillColor(48) -c1.Modified(True) -c1.Update() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/bench/hsimple_rflx.py b/pypy/module/_cppyy/bench/hsimple_rflx.py deleted file mode 100755 --- a/pypy/module/_cppyy/bench/hsimple_rflx.py +++ /dev/null @@ -1,120 +0,0 @@ -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -#*-* -#*-* This program creates : -#*-* - a one dimensional histogram -#*-* - a two dimensional histogram -#*-* - a profile histogram -#*-* - a memory-resident ntuple -#*-* -#*-* These objects are filled with some random numbers and saved on a file. -#*-* -#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* - -try: - import warnings - warnings.simplefilter("ignore") - - import cppyy, random - cppyy.load_reflection_info('bench02Dict_reflex.so') - - app = cppyy.gbl.Bench02RootApp() - TCanvas = cppyy.gbl.TCanvas - TFile = cppyy.gbl.TFile - TProfile = cppyy.gbl.TProfile - TNtuple = cppyy.gbl.TNtuple - TH1F = cppyy.gbl.TH1F - TH2F = cppyy.gbl.TH2F - TRandom = cppyy.gbl.TRandom -except ImportError: - from ROOT import TCanvas, TFile, TProfile, TNtuple, TH1F, TH2F, TRandom - import random - -import math - -#gROOT = cppyy.gbl.gROOT -#gBenchmark = cppyy.gbl.gBenchmark -#gRandom = cppyy.gbl.gRandom -#gSystem = cppyy.gbl.gSystem - -#gROOT.Reset() - -# Create a new canvas, and customize it. -#c1 = TCanvas( 'c1', 'Dynamic Filling Example', 200, 10, 700, 500 ) -#c1.SetFillColor( 42 ) -#c1.GetFrame().SetFillColor( 21 ) -#c1.GetFrame().SetBorderSize( 6 ) -#c1.GetFrame().SetBorderMode( -1 ) - -# Create a new ROOT binary machine independent file. -# Note that this file may contain any kind of ROOT objects, histograms, -# pictures, graphics objects, detector geometries, tracks, events, etc.. -# This file is now becoming the current directory. - -#hfile = gROOT.FindObject( 'hsimple.root' ) -#if hfile: -# hfile.Close() -#hfile = TFile( 'hsimple.root', 'RECREATE', 'Demo ROOT file with histograms' ) - -# Create some histograms, a profile histogram and an ntuple -hpx = TH1F('hpx', 'This is the px distribution', 100, -4, 4) -hpx.Print() -#hpxpy = TH2F( 'hpxpy', 'py vs px', 40, -4, 4, 40, -4, 4 ) -#hprof = TProfile( 'hprof', 'Profile of pz versus px', 100, -4, 4, 0, 20 ) -#ntuple = TNtuple( 'ntuple', 'Demo ntuple', 'px:py:pz:random:i' ) - -# Set canvas/frame attributes. -#hpx.SetFillColor( 48 ) - -#gBenchmark.Start( 'hsimple' ) - -# Initialize random number generator. -#gRandom.SetSeed() -#rannor, rndm = gRandom.Rannor, gRandom.Rndm - -random = TRandom() -random.SetSeed(0) - -# Fill histograms randomly. -#px, py = Double(), Double() -kUPDATE = 1000 -for i in xrange(2500000): - # Generate random values. -# px, py = random.gauss(0, 1), random.gauss(0, 1) - px, py = random.Gaus(0, 1), random.Gaus(0, 1) -# pt = (px*px + py*py)**0.5 - pt = math.sqrt(px*px + py*py) -# pt = (px*px + py*py) -# random = rndm(1) - - # Fill histograms. - hpx.Fill(pt) -# hpxpyFill( px, py ) -# hprofFill( px, pz ) -# ntupleFill( px, py, pz, random, i ) - - # Update display every kUPDATE events. -# if i and i%kUPDATE == 0: -# if i == kUPDATE: -# hpx.Draw() - -# c1.Modified() -# c1.Update() - -# if gSystem.ProcessEvents(): # allow user interrupt -# break - -#gBenchmark.Show( 'hsimple' ) - -hpx.Print() - -# Save all objects in this file. -#hpx.SetFillColor( 0 ) -#hfile.Write() -#hfile.Close() -#hpx.SetFillColor( 48 ) -#c1.Modified() -#c1.Update() -#c1.Draw() - -# Note that the file is automatically closed when application terminates -# or when the file destructor is called. diff --git a/pypy/module/_cppyy/capi/__init__.py b/pypy/module/_cppyy/capi/__init__.py --- a/pypy/module/_cppyy/capi/__init__.py +++ b/pypy/module/_cppyy/capi/__init__.py @@ -1,16 +1,6 @@ from rpython.rtyper.lltypesystem import rffi, lltype -# There are two possible ways of accessing the backend through the reflection -# C-API: built it into pypy-c, or load it dynamically. The latter is preferred -# (and is the default) for use with Reflex. B/c of some builtin pythonizations, -# the former is recommended (for now) with CINT. - -# Note: if builtin_capi is chosen, then inside builtin_capi.py, there is still -# the selection of the desired backend (default is Reflex). - -# choose C-API access method: from pypy.module._cppyy.capi.loadable_capi import * -#from pypy.module._cppyy.capi.builtin_capi import * from pypy.module._cppyy.capi.capi_types import C_OBJECT,\ C_NULL_TYPE, C_NULL_OBJECT diff --git a/pypy/module/_cppyy/capi/builtin_capi.py b/pypy/module/_cppyy/capi/builtin_capi.py deleted file mode 100644 --- a/pypy/module/_cppyy/capi/builtin_capi.py +++ /dev/null @@ -1,590 +0,0 @@ -from rpython.rtyper.lltypesystem import rffi, lltype -from rpython.rlib.rarithmetic import intmask -from rpython.rlib import jit - -import cling_capi as backend - -from pypy.module._cppyy.capi.capi_types import C_SCOPE, C_TYPE, C_OBJECT,\ - C_METHOD, C_INDEX, C_INDEX_ARRAY, WLAVC_INDEX, C_FUNC_PTR - -identify = backend.identify -pythonize = backend.pythonize -register_pythonizations = backend.register_pythonizations -std_string_name = backend.std_string_name - -ts_reflect = backend.ts_reflect -ts_call = backend.ts_call -ts_memory = backend.ts_memory -ts_helper = backend.ts_helper - -def verify_backend(space): - return True # by definition - -c_load_dictionary = backend.c_load_dictionary - -# name to opaque C++ scope representation ------------------------------------ -_c_num_scopes = rffi.llexternal( - "cppyy_num_scopes", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_num_scopes(space, cppscope): - return _c_num_scopes(cppscope.handle) -_c_scope_name = rffi.llexternal( - "cppyy_scope_name", - [C_SCOPE, rffi.INT], rffi.CCHARP, - compilation_info = backend.eci) -def c_scope_name(space, cppscope, iscope): - return charp2str_free(space, _c_scope_name(cppscope.handle, iscope)) - -_c_resolve_name = rffi.llexternal( - "cppyy_resolve_name", - [rffi.CCHARP], rffi.CCHARP, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_resolve_name(space, name): - return charp2str_free(space, _c_resolve_name(name)) -_c_get_scope_opaque = rffi.llexternal( - "cppyy_get_scope", - [rffi.CCHARP], C_SCOPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_get_scope_opaque(space, name): - return _c_get_scope_opaque(name) -_c_actual_class = rffi.llexternal( - "cppyy_actual_class", - [C_TYPE, C_OBJECT], C_TYPE, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_actual_class(space, cppclass, cppobj): - return _c_actual_class(cppclass.handle, cppobj) - -# memory management ---------------------------------------------------------- -_c_allocate = rffi.llexternal( - "cppyy_allocate", - [C_TYPE], C_OBJECT, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate(space, cppclass): - return _c_allocate(cppclass.handle) -_c_deallocate = rffi.llexternal( - "cppyy_deallocate", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate(space, cppclass, cppobject): - _c_deallocate(cppclass.handle, cppobject) -_c_destruct = rffi.llexternal( - "cppyy_destruct", - [C_TYPE, C_OBJECT], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_destruct(space, cppclass, cppobject): - _c_destruct(cppclass.handle, cppobject) - -# method/function dispatching ------------------------------------------------ -_c_call_v = rffi.llexternal( - "cppyy_call_v", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], lltype.Void, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_v(space, cppmethod, cppobject, nargs, args): - _c_call_v(cppmethod, cppobject, nargs, args) -_c_call_b = rffi.llexternal( - "cppyy_call_b", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.UCHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_b(space, cppmethod, cppobject, nargs, args): - return _c_call_b(cppmethod, cppobject, nargs, args) -_c_call_c = rffi.llexternal( - "cppyy_call_c", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.CHAR, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_c(space, cppmethod, cppobject, nargs, args): - return _c_call_c(cppmethod, cppobject, nargs, args) -_c_call_h = rffi.llexternal( - "cppyy_call_h", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.SHORT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_h(space, cppmethod, cppobject, nargs, args): - return _c_call_h(cppmethod, cppobject, nargs, args) -_c_call_i = rffi.llexternal( - "cppyy_call_i", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.INT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_i(space, cppmethod, cppobject, nargs, args): - return _c_call_i(cppmethod, cppobject, nargs, args) -_c_call_l = rffi.llexternal( - "cppyy_call_l", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_l(space, cppmethod, cppobject, nargs, args): - return _c_call_l(cppmethod, cppobject, nargs, args) -_c_call_ll = rffi.llexternal( - "cppyy_call_ll", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGLONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ll(space, cppmethod, cppobject, nargs, args): - return _c_call_ll(cppmethod, cppobject, nargs, args) -_c_call_f = rffi.llexternal( - "cppyy_call_f", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.FLOAT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_f(space, cppmethod, cppobject, nargs, args): - return _c_call_f(cppmethod, cppobject, nargs, args) -_c_call_d = rffi.llexternal( - "cppyy_call_d", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.DOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_d(space, cppmethod, cppobject, nargs, args): - return _c_call_d(cppmethod, cppobject, nargs, args) -_c_call_ld = rffi.llexternal( - "cppyy_call_ld", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.LONGDOUBLE, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_ld(space, cppmethod, cppobject, nargs, args): - return _c_call_ld(cppmethod, cppobject, nargs, args) - -_c_call_r = rffi.llexternal( - "cppyy_call_r", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP], rffi.VOIDP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_r(space, cppmethod, cppobject, nargs, args): - return _c_call_r(cppmethod, cppobject, nargs, args) -_c_call_s = rffi.llexternal( - "cppyy_call_s", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, rffi.SIZE_TP], rffi.CCHARP, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_s(space, cppmethod, cppobject, nargs, args): - length = lltype.malloc(rffi.SIZE_TP.TO, 1, flavor='raw') - try: - cstr = _c_call_s(cppmethod, cppobject, nargs, args, length) - cstr_len = intmask(length[0]) - finally: - lltype.free(length, flavor='raw') - return cstr, cstr_len - -_c_constructor = rffi.llexternal( - "cppyy_constructor", - [C_METHOD, C_TYPE, rffi.INT, rffi.VOIDP], C_OBJECT, - releasegil=ts_call, - compilation_info=backend.eci) -def c_constructor(space, cppmethod, cppobject, nargs, args): - return _c_constructor(cppmethod, cppobject, nargs, args) -_c_call_o = rffi.llexternal( - "cppyy_call_o", - [C_METHOD, C_OBJECT, rffi.INT, rffi.VOIDP, C_TYPE], rffi.LONG, - releasegil=ts_call, - compilation_info=backend.eci) -def c_call_o(space, method, cppobj, nargs, args, cppclass): - return _c_call_o(method, cppobj, nargs, args, cppclass.handle) - -_c_get_function_address = rffi.llexternal( - "cppyy_get_function_address", - [C_SCOPE, C_INDEX], C_FUNC_PTR, - releasegil=ts_reflect, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) -def c_get_function_address(space, cppscope, index): - return _c_get_function_address(cppscope.handle, index) - -# handling of function argument buffer --------------------------------------- -_c_allocate_function_args = rffi.llexternal( - "cppyy_allocate_function_args", - [rffi.SIZE_T], rffi.VOIDP, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_allocate_function_args(space, size): - return _c_allocate_function_args(size) -_c_deallocate_function_args = rffi.llexternal( - "cppyy_deallocate_function_args", - [rffi.VOIDP], lltype.Void, - releasegil=ts_memory, - compilation_info=backend.eci) -def c_deallocate_function_args(space, args): - _c_deallocate_function_args(args) -_c_function_arg_sizeof = rffi.llexternal( - "cppyy_function_arg_sizeof", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_sizeof(space): - return _c_function_arg_sizeof() -_c_function_arg_typeoffset = rffi.llexternal( - "cppyy_function_arg_typeoffset", - [], rffi.SIZE_T, - releasegil=ts_memory, - compilation_info=backend.eci, - random_effects_on_gcobjs=False) - at jit.elidable -def c_function_arg_typeoffset(space): - return _c_function_arg_typeoffset() - -# scope reflection information ----------------------------------------------- -_c_is_namespace = rffi.llexternal( - "cppyy_is_namespace", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_namespace(space, scope): - return _c_is_namespace(scope) -_c_is_template = rffi.llexternal( - "cppyy_is_template", - [rffi.CCHARP], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_template(space, name): - return _c_is_template(name) -_c_is_abstract = rffi.llexternal( - "cppyy_is_abstract", - [C_SCOPE], rffi.INT, - releasegil=ts_reflect, - compilation_info=backend.eci) -def c_is_abstract(space, cpptype): - return _c_is_abstract(cpptype) From pypy.commits at gmail.com Wed Oct 18 11:49:16 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 08:49:16 -0700 (PDT) Subject: [pypy-commit] pypy default: Add missing int_floordiv, int_divmod. Improves pidigits by about 5% Message-ID: <59e777fc.d08edf0a.9c3a.fc6a@mx.google.com> Author: stian Branch: Changeset: r92789:d420391a020a Date: 2017-10-18 17:48 +0200 http://bitbucket.org/pypy/pypy/changeset/d420391a020a/ Log: Add missing int_floordiv, int_divmod. Improves pidigits by about 5% diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -491,17 +491,18 @@ "long division or modulo by zero") return newlong(space, z) - def _floordiv(self, space, w_other): + def _int_floordiv(self, space, w_other): try: - z = self.num.floordiv(w_other.asbigint()) + z = self.num.int_floordiv(w_other) except ZeroDivisionError: raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return newlong(space, z) - descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv) + descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv, _int_floordiv) _div = func_with_new_name(_floordiv, '_div') - descr_div, descr_rdiv = _make_descr_binop(_div) + _int_div = func_with_new_name(_int_floordiv, '_int_div') + descr_div, descr_rdiv = _make_descr_binop(_div, _int_div) def _mod(self, space, w_other): try: @@ -527,7 +528,16 @@ raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return space.newtuple([newlong(space, div), newlong(space, mod)]) - descr_divmod, descr_rdivmod = _make_descr_binop(_divmod) + + def _int_divmod(self, space, w_other): + try: + div, mod = self.num.int_divmod(w_other) + except ZeroDivisionError: + raise oefmt(space.w_ZeroDivisionError, + "long division or modulo by zero") + return space.newtuple([newlong(space, div), newlong(space, mod)]) + + descr_divmod, descr_rdivmod = _make_descr_binop(_divmod, _int_divmod) def newlong(space, bigint): diff --git a/pypy/objspace/std/test/test_longobject.py b/pypy/objspace/std/test/test_longobject.py --- a/pypy/objspace/std/test/test_longobject.py +++ b/pypy/objspace/std/test/test_longobject.py @@ -70,6 +70,17 @@ a = x // 10000000L assert a == 3L + def test_int_floordiv(self): + x = 3000L + a = x // 1000 + assert a == 3L + + x = 3000L + a = x // -1000 + assert a == -3L + + + def test_numerator_denominator(self): assert (1L).numerator == 1L assert (1L).denominator == 1L @@ -208,6 +219,11 @@ check_division(x, y) raises(ZeroDivisionError, "x // 0L") + def test_int_divmod(self): + q, r = divmod(100L, 11) + assert q == 9L + assert r == 1L + def test_format(self): assert repr(12345678901234567890) == '12345678901234567890L' assert str(12345678901234567890) == '12345678901234567890' @@ -386,3 +402,4 @@ n = "a" * size expected = (2 << (size * 4)) // 3 assert long(n, 16) == expected + diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -781,7 +781,34 @@ def div(self, other): return self.floordiv(other) - + + @jit.elidable + def int_floordiv(self, b): + if not int_in_valid_range(b): + # Fallback to long. + return self.mul(rbigint.fromint(b)) + + digit = abs(b) + assert digit > 0 + + if self.sign == 1 and b > 0: + if digit == 1: + return self + """elif digit & (digit - 1) == 0: + return self.rshift(ptwotable[digit]) + """ + div, mod = _divrem1(self, digit) + + if mod != 0 and self.sign * (-1 if b < 0 else 1) == -1: + if div.sign == 0: + return ONENEGATIVERBIGINT + div = div.int_add(1) + div.sign = self.sign * (-1 if b < 0 else 1) + return div + + def int_div(self, other): + return self.int_floordiv(other) + @jit.elidable def mod(self, other): if self.sign == 0: @@ -888,6 +915,30 @@ return div, mod @jit.elidable + def int_divmod(v, w): + """ Divmod with int """ + + if w == 0: + raise ZeroDivisionError("long division or modulo by zero") + + wsign = (-1 if w < 0 else 1) + if not int_in_valid_range(w) or v.sign != wsign: + # Divrem1 doesn't deal with the sign difference. Instead of having yet another copy, + # Just fallback. + return v.divmod(rbigint.fromint(w)) + + digit = abs(w) + assert digit > 0 + + div, mod = _divrem1(v, digit) + mod = rbigint.fromint(mod) + + mod.sign = wsign + div.sign = v.sign * wsign + + return div, mod + + @jit.elidable def pow(a, b, c=None): negativeOutput = False # if x<0 return negative output From pypy.commits at gmail.com Wed Oct 18 15:05:24 2017 From: pypy.commits at gmail.com (arigo) Date: Wed, 18 Oct 2017 12:05:24 -0700 (PDT) Subject: [pypy-commit] pypy default: Backed out changeset d420391a020a Message-ID: <59e7a5f4.025d1c0a.32f41.d17e@mx.google.com> Author: Armin Rigo Branch: Changeset: r92790:bd1d07287bb4 Date: 2017-10-18 21:04 +0200 http://bitbucket.org/pypy/pypy/changeset/bd1d07287bb4/ Log: Backed out changeset d420391a020a A few problems: 'b // 0' fails an internal asserts now; and 'n / int(-n)' gives bogus results if n == sys.maxint+1. Moreover, it disables the replacement of 'long_num // 2' with a simple shift, which costs a lot of performance in some cases. diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -491,18 +491,17 @@ "long division or modulo by zero") return newlong(space, z) - def _int_floordiv(self, space, w_other): + def _floordiv(self, space, w_other): try: - z = self.num.int_floordiv(w_other) + z = self.num.floordiv(w_other.asbigint()) except ZeroDivisionError: raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return newlong(space, z) - descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv, _int_floordiv) + descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv) _div = func_with_new_name(_floordiv, '_div') - _int_div = func_with_new_name(_int_floordiv, '_int_div') - descr_div, descr_rdiv = _make_descr_binop(_div, _int_div) + descr_div, descr_rdiv = _make_descr_binop(_div) def _mod(self, space, w_other): try: @@ -528,16 +527,7 @@ raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return space.newtuple([newlong(space, div), newlong(space, mod)]) - - def _int_divmod(self, space, w_other): - try: - div, mod = self.num.int_divmod(w_other) - except ZeroDivisionError: - raise oefmt(space.w_ZeroDivisionError, - "long division or modulo by zero") - return space.newtuple([newlong(space, div), newlong(space, mod)]) - - descr_divmod, descr_rdivmod = _make_descr_binop(_divmod, _int_divmod) + descr_divmod, descr_rdivmod = _make_descr_binop(_divmod) def newlong(space, bigint): diff --git a/pypy/objspace/std/test/test_longobject.py b/pypy/objspace/std/test/test_longobject.py --- a/pypy/objspace/std/test/test_longobject.py +++ b/pypy/objspace/std/test/test_longobject.py @@ -70,17 +70,6 @@ a = x // 10000000L assert a == 3L - def test_int_floordiv(self): - x = 3000L - a = x // 1000 - assert a == 3L - - x = 3000L - a = x // -1000 - assert a == -3L - - - def test_numerator_denominator(self): assert (1L).numerator == 1L assert (1L).denominator == 1L @@ -219,11 +208,6 @@ check_division(x, y) raises(ZeroDivisionError, "x // 0L") - def test_int_divmod(self): - q, r = divmod(100L, 11) - assert q == 9L - assert r == 1L - def test_format(self): assert repr(12345678901234567890) == '12345678901234567890L' assert str(12345678901234567890) == '12345678901234567890' @@ -402,4 +386,3 @@ n = "a" * size expected = (2 << (size * 4)) // 3 assert long(n, 16) == expected - diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -781,34 +781,7 @@ def div(self, other): return self.floordiv(other) - - @jit.elidable - def int_floordiv(self, b): - if not int_in_valid_range(b): - # Fallback to long. - return self.mul(rbigint.fromint(b)) - - digit = abs(b) - assert digit > 0 - - if self.sign == 1 and b > 0: - if digit == 1: - return self - """elif digit & (digit - 1) == 0: - return self.rshift(ptwotable[digit]) - """ - div, mod = _divrem1(self, digit) - - if mod != 0 and self.sign * (-1 if b < 0 else 1) == -1: - if div.sign == 0: - return ONENEGATIVERBIGINT - div = div.int_add(1) - div.sign = self.sign * (-1 if b < 0 else 1) - return div - - def int_div(self, other): - return self.int_floordiv(other) - + @jit.elidable def mod(self, other): if self.sign == 0: @@ -915,30 +888,6 @@ return div, mod @jit.elidable - def int_divmod(v, w): - """ Divmod with int """ - - if w == 0: - raise ZeroDivisionError("long division or modulo by zero") - - wsign = (-1 if w < 0 else 1) - if not int_in_valid_range(w) or v.sign != wsign: - # Divrem1 doesn't deal with the sign difference. Instead of having yet another copy, - # Just fallback. - return v.divmod(rbigint.fromint(w)) - - digit = abs(w) - assert digit > 0 - - div, mod = _divrem1(v, digit) - mod = rbigint.fromint(mod) - - mod.sign = wsign - div.sign = v.sign * wsign - - return div, mod - - @jit.elidable def pow(a, b, c=None): negativeOutput = False # if x<0 return negative output From pypy.commits at gmail.com Wed Oct 18 16:58:17 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 13:58:17 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: branch with some rbigint/longobject/intobject improvements Message-ID: <59e7c069.13121c0a.54e3.5743@mx.google.com> Author: stian Branch: math-improvements Changeset: r92791:ec9abaebdc28 Date: 2017-10-18 22:10 +0200 http://bitbucket.org/pypy/pypy/changeset/ec9abaebdc28/ Log: branch with some rbigint/longobject/intobject improvements From pypy.commits at gmail.com Wed Oct 18 16:58:19 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 13:58:19 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Test + fix for bugs found by Armin Rigo Message-ID: <59e7c06b.92acdf0a.bdb1d.61a6@mx.google.com> Author: stian Branch: math-improvements Changeset: r92792:06129c0b5e0e Date: 2017-10-18 22:13 +0200 http://bitbucket.org/pypy/pypy/changeset/06129c0b5e0e/ Log: Test + fix for bugs found by Armin Rigo diff --git a/pypy/objspace/std/test/test_longobject.py b/pypy/objspace/std/test/test_longobject.py --- a/pypy/objspace/std/test/test_longobject.py +++ b/pypy/objspace/std/test/test_longobject.py @@ -2,7 +2,6 @@ from pypy.objspace.std import longobject as lobj from rpython.rlib.rbigint import rbigint - class TestW_LongObject: def test_bigint_w(self): space = self.space @@ -70,6 +69,23 @@ a = x // 10000000L assert a == 3L + def test_int_floordiv(self): + import sys + + x = 3000L + a = x // 1000 + assert a == 3L + + x = 3000L + a = x // -1000 + assert a == -3L + + x = 3000L + raises(ZeroDivisionError, "x // 0") + + n = sys.maxint+1 + assert n / int(-n) == -1L + def test_numerator_denominator(self): assert (1L).numerator == 1L assert (1L).denominator == 1L @@ -208,6 +224,11 @@ check_division(x, y) raises(ZeroDivisionError, "x // 0L") + def test_int_divmod(self): + q, r = divmod(100L, 11) + assert q == 9L + assert r == 1L + def test_format(self): assert repr(12345678901234567890) == '12345678901234567890L' assert str(12345678901234567890) == '12345678901234567890' @@ -386,3 +407,4 @@ n = "a" * size expected = (2 << (size * 4)) // 3 assert long(n, 16) == expected + diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -145,6 +145,7 @@ make_sure_not_resized(digits) self._digits = digits assert size >= 0 + self.size = size or len(digits) self.sign = sign @@ -183,7 +184,9 @@ setdigit._always_inline_ = True def numdigits(self): - return self.size + w = self.size + assert w > 0 + return w numdigits._always_inline_ = True @staticmethod @@ -510,7 +513,6 @@ @jit.elidable def int_eq(self, other): """ eq with int """ - if not int_in_valid_range(other): # Fallback to Long. return self.eq(rbigint.fromint(other)) @@ -657,7 +659,7 @@ if other.sign == 0: return self elif self.sign == 0: - return rbigint(other._digits[:other.size], -other.sign, other.size) + return rbigint(other._digits[:other.numdigits()], -other.sign, other.numdigits()) elif self.sign == other.sign: result = _x_sub(self, other) else: @@ -698,7 +700,7 @@ if a._digits[0] == NULLDIGIT: return NULLRBIGINT elif a._digits[0] == ONEDIGIT: - return rbigint(b._digits[:b.size], a.sign * b.sign, b.size) + return rbigint(b._digits[:b.numdigits()], a.sign * b.sign, b.numdigits()) elif bsize == 1: res = b.widedigit(0) * a.widedigit(0) carry = res >> SHIFT @@ -740,7 +742,7 @@ bsign = -1 if b < 0 else 1 if digit == 1: - return rbigint(self._digits[:self.size], self.sign * bsign, asize) + return rbigint(self._digits[:self.numdigits()], self.sign * bsign, asize) elif asize == 1: res = self.widedigit(0) * digit carry = res >> SHIFT @@ -767,7 +769,7 @@ if self.sign == 1 and other.numdigits() == 1 and other.sign == 1: digit = other.digit(0) if digit == 1: - return rbigint(self._digits[:self.size], 1, self.size) + return rbigint(self._digits[:self.numdigits()], 1, self.numdigits()) elif digit and digit & (digit - 1) == 0: return self.rshift(ptwotable[digit]) @@ -781,7 +783,37 @@ def div(self, other): return self.floordiv(other) - + + @jit.elidable + def int_floordiv(self, b): + if not int_in_valid_range(b): + # Fallback to long. + return self.floordiv(rbigint.fromint(b)) + + if b == 0: + raise ZeroDivisionError("long division by zero") + + digit = abs(b) + assert digit > 0 + + if self.sign == 1 and b > 0: + if digit == 1: + return self + elif digit & (digit - 1) == 0: + return self.rshift(ptwotable[digit]) + + div, mod = _divrem1(self, digit) + + if mod != 0 and self.sign * (-1 if b < 0 else 1) == -1: + if div.sign == 0: + return ONENEGATIVERBIGINT + div = div.int_add(1) + div.sign = self.sign * (-1 if b < 0 else 1) + return div + + def int_div(self, other): + return self.int_floordiv(other) + @jit.elidable def mod(self, other): if self.sign == 0: @@ -888,6 +920,30 @@ return div, mod @jit.elidable + def int_divmod(v, w): + """ Divmod with int """ + + if w == 0: + raise ZeroDivisionError("long division or modulo by zero") + + wsign = (-1 if w < 0 else 1) + if not int_in_valid_range(w) or v.sign != wsign: + # Divrem1 doesn't deal with the sign difference. Instead of having yet another copy, + # Just fallback. + return v.divmod(rbigint.fromint(w)) + + digit = abs(w) + assert digit > 0 + + div, mod = _divrem1(v, digit) + mod = rbigint.fromint(mod * wsign) + + #mod.sign = wsign + div.sign = v.sign * wsign + + return div, mod + + @jit.elidable def pow(a, b, c=None): negativeOutput = False # if x<0 return negative output @@ -1029,13 +1085,13 @@ @jit.elidable def neg(self): - return rbigint(self._digits, -self.sign, self.size) + return rbigint(self._digits, -self.sign, self.numdigits()) @jit.elidable def abs(self): if self.sign != -1: return self - return rbigint(self._digits, 1, self.size) + return rbigint(self._digits, 1, self.numdigits()) @jit.elidable def invert(self): #Implement ~x as -(x + 1) @@ -1061,7 +1117,7 @@ # So we can avoid problems with eq, AND avoid the need for normalize. if self.sign == 0: return self - return rbigint([NULLDIGIT] * wordshift + self._digits, self.sign, self.size + wordshift) + return rbigint([NULLDIGIT] * wordshift + self._digits, self.sign, self.numdigits() + wordshift) oldsize = self.numdigits() newsize = oldsize + wordshift + 1 @@ -1276,7 +1332,7 @@ def __repr__(self): return "" % (self._digits, - self.sign, self.size, len(self._digits), + self.sign, self.numdigits(), len(self._digits), self.str()) ONERBIGINT = rbigint([ONEDIGIT], 1, 1) From pypy.commits at gmail.com Wed Oct 18 16:58:21 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 13:58:21 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Missing from earlier Message-ID: <59e7c06d.6583df0a.72744.0cd3@mx.google.com> Author: stian Branch: math-improvements Changeset: r92793:b4e5817c17d8 Date: 2017-10-18 22:14 +0200 http://bitbucket.org/pypy/pypy/changeset/b4e5817c17d8/ Log: Missing from earlier diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -491,17 +491,18 @@ "long division or modulo by zero") return newlong(space, z) - def _floordiv(self, space, w_other): + def _int_floordiv(self, space, w_other): try: - z = self.num.floordiv(w_other.asbigint()) + z = self.num.int_floordiv(w_other) except ZeroDivisionError: raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return newlong(space, z) - descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv) + descr_floordiv, descr_rfloordiv = _make_descr_binop(_floordiv, _int_floordiv) _div = func_with_new_name(_floordiv, '_div') - descr_div, descr_rdiv = _make_descr_binop(_div) + _int_div = func_with_new_name(_int_floordiv, '_int_div') + descr_div, descr_rdiv = _make_descr_binop(_div, _int_div) def _mod(self, space, w_other): try: @@ -527,7 +528,16 @@ raise oefmt(space.w_ZeroDivisionError, "long division or modulo by zero") return space.newtuple([newlong(space, div), newlong(space, mod)]) - descr_divmod, descr_rdivmod = _make_descr_binop(_divmod) + + def _int_divmod(self, space, w_other): + try: + div, mod = self.num.int_divmod(w_other) + except ZeroDivisionError: + raise oefmt(space.w_ZeroDivisionError, + "long division or modulo by zero") + return space.newtuple([newlong(space, div), newlong(space, mod)]) + + descr_divmod, descr_rdivmod = _make_descr_binop(_divmod, _int_divmod) def newlong(space, bigint): From pypy.commits at gmail.com Wed Oct 18 16:58:23 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 13:58:23 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Use long with int when overflowing Message-ID: <59e7c06f.cc981c0a.802bd.ca8e@mx.google.com> Author: stian Branch: math-improvements Changeset: r92794:c1d74e03b736 Date: 2017-10-18 22:15 +0200 http://bitbucket.org/pypy/pypy/changeset/c1d74e03b736/ Log: Use long with int when overflowing diff --git a/pypy/objspace/std/intobject.py b/pypy/objspace/std/intobject.py --- a/pypy/objspace/std/intobject.py +++ b/pypy/objspace/std/intobject.py @@ -287,7 +287,7 @@ op = getattr(operator, opname, None) assert op or ovf2small - def ovf2long(space, x, y): + def ovf2long(space, x, y, y_obj): """Handle overflowing to smalllong or long""" if _recover_with_smalllong(space): if ovf2small: @@ -298,11 +298,12 @@ a = r_longlong(x) b = r_longlong(y) return W_SmallLongObject(op(a, b)) - + from pypy.objspace.std.longobject import W_LongObject w_x = W_LongObject.fromint(space, x) - w_y = W_LongObject.fromint(space, y) - return getattr(w_x, 'descr_' + opname)(space, w_y) + #w_y = W_LongObject.fromint(space, y) + + return getattr(w_x, 'descr_' + opname)(space, y_obj) return ovf2long @@ -521,7 +522,7 @@ try: z = ovfcheck(op(x, y)) except OverflowError: - return ovf2long(space, x, y) + return ovf2long(space, x, y, w_other) else: z = op(x, y) return wrapint(space, z) @@ -543,7 +544,7 @@ try: z = ovfcheck(op(y, x)) except OverflowError: - return ovf2long(space, y, x) + return ovf2long(space, y, x, w_other) else: z = op(y, x) return wrapint(space, z) @@ -574,7 +575,7 @@ try: return func(space, x, y) except OverflowError: - return ovf2long(space, x, y) + return ovf2long(space, x, y, w_other) else: return func(space, x, y) @@ -589,7 +590,7 @@ try: return func(space, y, x) except OverflowError: - return ovf2long(space, y, x) + return ovf2long(space, y, x, w_other) else: return func(space, y, x) From pypy.commits at gmail.com Wed Oct 18 16:58:25 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 13:58:25 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Add rlib test Message-ID: <59e7c071.55ad1c0a.cbd26.78f0@mx.google.com> Author: stian Branch: math-improvements Changeset: r92795:3a37e2483546 Date: 2017-10-18 22:57 +0200 http://bitbucket.org/pypy/pypy/changeset/3a37e2483546/ Log: Add rlib test diff --git a/rpython/rlib/test/test_rbigint.py b/rpython/rlib/test/test_rbigint.py --- a/rpython/rlib/test/test_rbigint.py +++ b/rpython/rlib/test/test_rbigint.py @@ -64,6 +64,29 @@ r2 = op1 // op2 assert r1.tolong() == r2 + def test_int_floordiv(self): + x = 1000L + r = rbigint.fromlong(x) + r2 = r.int_floordiv(10) + assert r2.tolong() == 100L + + assert py.test.raises(ZeroDivisionError, r.int_floordiv, 0) + + # Error pointed out by Armin Rigo + n = sys.maxint+1 + r = rbigint.fromlong(n) + assert r.int_floordiv(int(-n)).tolong() == -1L + + for x in (1, 1000, sys.maxint): + r = rbigint.fromlong(x) + rn = rbigint.fromlong(-x) + res = r.int_floordiv(x) + res2 = r.int_floordiv(-x) + res3 = rn.int_floordiv(x) + assert res.tolong() == 1L + assert res2.tolong() == -1L + assert res3.tolong() == -1L + def test_truediv(self): for op1 in gen_signs(long_vals_not_too_big): for op2 in gen_signs(long_vals): From pypy.commits at gmail.com Wed Oct 18 18:12:40 2017 From: pypy.commits at gmail.com (stian) Date: Wed, 18 Oct 2017 15:12:40 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Make use of int_pow Message-ID: <59e7d1d8.6896df0a.aaabd.37c5@mx.google.com> Author: stian Branch: math-improvements Changeset: r92796:ab5af3aa60e3 Date: 2017-10-18 23:16 +0200 http://bitbucket.org/pypy/pypy/changeset/ab5af3aa60e3/ Log: Make use of int_pow diff --git a/pypy/objspace/std/intobject.py b/pypy/objspace/std/intobject.py --- a/pypy/objspace/std/intobject.py +++ b/pypy/objspace/std/intobject.py @@ -268,7 +268,7 @@ return ix -def _pow_ovf2long(space, iv, iw, w_modulus): +def _pow_ovf2long(space, iv, iw, iw_obj, w_modulus): if space.is_none(w_modulus) and _recover_with_smalllong(space): from pypy.objspace.std.smalllongobject import _pow as _pow_small try: @@ -279,8 +279,8 @@ pass from pypy.objspace.std.longobject import W_LongObject w_iv = W_LongObject.fromint(space, iv) - w_iw = W_LongObject.fromint(space, iw) - return w_iv.descr_pow(space, w_iw, w_modulus) + + return w_iv.descr_pow(space, iw_obj, w_modulus) def _make_ovf2long(opname, ovf2small=None): @@ -292,17 +292,16 @@ if _recover_with_smalllong(space): if ovf2small: return ovf2small(space, x, y) - # Assume a generic operation without an explicit ovf2small + # Assume a generic operation without an explicit #iovf2small # handler from pypy.objspace.std.smalllongobject import W_SmallLongObject a = r_longlong(x) b = r_longlong(y) return W_SmallLongObject(op(a, b)) - + from pypy.objspace.std.longobject import W_LongObject w_x = W_LongObject.fromint(space, x) - #w_y = W_LongObject.fromint(space, y) - + return getattr(w_x, 'descr_' + opname)(space, y_obj) return ovf2long @@ -472,12 +471,12 @@ # can't return NotImplemented (space.pow doesn't do full # ternary, i.e. w_modulus.__zpow__(self, w_exponent)), so # handle it ourselves - return _pow_ovf2long(space, x, y, w_modulus) + return _pow_ovf2long(space, x, y, w_exponent, w_modulus) try: result = _pow(space, x, y, z) except (OverflowError, ValueError): - return _pow_ovf2long(space, x, y, w_modulus) + return _pow_ovf2long(space, x, y, w_exponent, w_modulus) return space.newint(result) @unwrap_spec(w_modulus=WrappedDefault(None)) From pypy.commits at gmail.com Thu Oct 19 11:21:55 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 08:21:55 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Add one more case to divmod, add tests to int_divmod, add int_pow tests. Message-ID: <59e8c313.090b1c0a.26551.8571@mx.google.com> Author: stian Branch: math-improvements Changeset: r92797:6eedf6a4b154 Date: 2017-10-19 12:04 +0200 http://bitbucket.org/pypy/pypy/changeset/6eedf6a4b154/ Log: Add one more case to divmod, add tests to int_divmod, add int_pow tests. diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -309,27 +309,45 @@ @unwrap_spec(w_modulus=WrappedDefault(None)) def descr_pow(self, space, w_exponent, w_modulus=None): if isinstance(w_exponent, W_AbstractIntObject): - w_exponent = w_exponent.descr_long(space) + exp_int = w_exponent.int_w(space) + sign = 0 + use_int = True + if exp_int > 0: + sign = 1 + elif exp_int < 0: + sign = -1 elif not isinstance(w_exponent, W_AbstractLongObject): return space.w_NotImplemented - + else: + exp_bigint = w_exponent.asbigint() + sign = exp_bigint.sign + use_int = False + if space.is_none(w_modulus): - if w_exponent.asbigint().sign < 0: + if sign < 0: self = self.descr_float(space) w_exponent = w_exponent.descr_float(space) return space.pow(self, w_exponent, space.w_None) - return W_LongObject(self.num.pow(w_exponent.asbigint())) + if use_int: + return W_LongObject(self.num.int_pow(exp_int)) + else: + return W_LongObject(self.num.pow(exp_bigint)) + elif isinstance(w_modulus, W_AbstractIntObject): w_modulus = w_modulus.descr_long(space) + elif not isinstance(w_modulus, W_AbstractLongObject): return space.w_NotImplemented - if w_exponent.asbigint().sign < 0: + if sign < 0: raise oefmt(space.w_TypeError, "pow() 2nd argument cannot be negative when 3rd " "argument specified") try: - result = self.num.pow(w_exponent.asbigint(), w_modulus.asbigint()) + if use_int: + result = self.num.int_pow(exp_int, w_modulus.asbigint()) + else: + result = self.num.pow(exp_bigint, w_modulus.asbigint()) except ValueError: raise oefmt(space.w_ValueError, "pow 3rd argument cannot be 0") return W_LongObject(result) diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -927,20 +927,29 @@ raise ZeroDivisionError("long division or modulo by zero") wsign = (-1 if w < 0 else 1) - if not int_in_valid_range(w) or v.sign != wsign: - # Divrem1 doesn't deal with the sign difference. Instead of having yet another copy, + if not int_in_valid_range(w) or (wsign == -1 and v.sign != wsign): # Just fallback. return v.divmod(rbigint.fromint(w)) - + + digit = abs(w) assert digit > 0 div, mod = _divrem1(v, digit) - mod = rbigint.fromint(mod * wsign) + + div.sign = v.sign * wsign + - #mod.sign = wsign - div.sign = v.sign * wsign - + + if v.sign != wsign: + if div.sign == 0: + div = NEGATIVERBIGINT + else: + div = div.int_sub(1) + mod = w - mod + + mod = rbigint.fromint(wsign * mod) + return div, mod @jit.elidable @@ -994,9 +1003,7 @@ elif a.sign == 0: return NULLRBIGINT elif size_b == 1: - if b._digits[0] == NULLDIGIT: - return ONERBIGINT if a.sign == 1 else ONENEGATIVERBIGINT - elif b._digits[0] == ONEDIGIT: + if b._digits[0] == ONEDIGIT: return a elif a.numdigits() == 1: adigit = a.digit(0) @@ -1084,6 +1091,89 @@ return z @jit.elidable + def int_pow(a, b, c=None): + negativeOutput = False # if x<0 return negative output + + # 5-ary values. If the exponent is large enough, table is + # precomputed so that table[i] == a**i % c for i in range(32). + # python translation: the table is computed when needed. + + if b < 0: # if exponent is negative + if c is not None: + raise TypeError( + "pow() 2nd argument " + "cannot be negative when 3rd argument specified") + # XXX failed to implement + raise ValueError("bigint pow() too negative") + + if c is not None: + if c.sign == 0: + raise ValueError("pow() 3rd argument cannot be 0") + + # if modulus < 0: + # negativeOutput = True + # modulus = -modulus + if c.sign < 0: + negativeOutput = True + c = c.neg() + + # if modulus == 1: + # return 0 + if c.numdigits() == 1 and c._digits[0] == ONEDIGIT: + return NULLRBIGINT + + # Reduce base by modulus in some cases: + # 1. If base < 0. Forcing the base non-neg makes things easier. + # 2. If base is obviously larger than the modulus. The "small + # exponent" case later can multiply directly by base repeatedly, + # while the "large exponent" case multiplies directly by base 31 + # times. It can be unboundedly faster to multiply by + # base % modulus instead. + # We could _always_ do this reduction, but mod() isn't cheap, + # so we only do it when it buys something. + if a.sign < 0 or a.numdigits() > c.numdigits(): + a = a.mod(c) + + elif b == 0: + return ONERBIGINT + elif a.sign == 0: + return NULLRBIGINT + + if b == 1: + return a + elif a.numdigits() == 1: + adigit = a.digit(0) + if adigit == 1: + if a.sign == -1 and b % 2: + return ONENEGATIVERBIGINT + return ONERBIGINT + elif adigit & (adigit - 1) == 0: + ret = a.lshift(((b-1)*(ptwotable[adigit]-1)) + b-1) + if a.sign == -1 and not b % 2: + ret.sign = 1 + return ret + + # At this point a, b, and c are guaranteed non-negative UNLESS + # c is NULL, in which case a may be negative. */ + + z = rbigint([ONEDIGIT], 1, 1) + + # python adaptation: moved macros REDUCE(X) and MULT(X, Y, result) + # into helper function result = _help_mult(x, y, c) + # Left-to-right binary exponentiation (HAC Algorithm 14.79) + # http://www.cacr.math.uwaterloo.ca/hac/about/chap14.pdf + j = 1 << (SHIFT-1) + while j != 0: + z = _help_mult(z, z, c) + if b & j: + z = _help_mult(z, a, c) + j >>= 1 + + if negativeOutput and z.sign != 0: + z = z.sub(c) + return z + + @jit.elidable def neg(self): return rbigint(self._digits, -self.sign, self.numdigits()) diff --git a/rpython/rlib/test/test_rbigint.py b/rpython/rlib/test/test_rbigint.py --- a/rpython/rlib/test/test_rbigint.py +++ b/rpython/rlib/test/test_rbigint.py @@ -164,6 +164,13 @@ r2 = op1 ** op2 assert r1.tolong() == r2 + def test_int_pow(self): + for op1 in gen_signs(long_vals_not_too_big[:-2]): + for op2 in [0, 1, 2, 8, 9, 10, 11]: + rl_op1 = rbigint.fromlong(op1) + r1 = rl_op1.int_pow(op2) + r2 = op1 ** op2 + assert r1.tolong() == r2 def test_touint(self): result = r_uint(sys.maxint + 42) rl = rbigint.fromint(sys.maxint).add(rbigint.fromint(42)) @@ -781,6 +788,20 @@ assert div.tolong() == _div assert rem.tolong() == _rem + def test_int_divmod(self): + x = 12345678901234567890L + for i in range(10): + y = randint(0, 1 << 12) + for sx, sy in (1, 1), (1, -1), (-1, -1), (-1, 1): + sx *= x + sy *= y + f1 = rbigint.fromlong(sx) + div, rem = f1.int_divmod(sy) + _div, _rem = divmod(sx, sy) + print sx, sy, " | ", div.tolong(), rem.tolong() + assert div.tolong() == _div + assert rem.tolong() == _rem + # testing Karatsuba stuff def test__v_iadd(self): f1 = bigint([lobj.MASK] * 10, 1) From pypy.commits at gmail.com Thu Oct 19 11:21:57 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 08:21:57 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Cleanup functio Message-ID: <59e8c315.03a7df0a.e264c.3f25@mx.google.com> Author: stian Branch: math-improvements Changeset: r92798:114b6da85abb Date: 2017-10-19 12:05 +0200 http://bitbucket.org/pypy/pypy/changeset/114b6da85abb/ Log: Cleanup functio diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -931,16 +931,12 @@ # Just fallback. return v.divmod(rbigint.fromint(w)) - digit = abs(w) assert digit > 0 div, mod = _divrem1(v, digit) - div.sign = v.sign * wsign - - if v.sign != wsign: if div.sign == 0: div = NEGATIVERBIGINT From pypy.commits at gmail.com Thu Oct 19 11:21:59 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 08:21:59 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Fix build issues Message-ID: <59e8c317.daa3df0a.9b720.6259@mx.google.com> Author: stian Branch: math-improvements Changeset: r92799:4e50c695d26e Date: 2017-10-19 12:27 +0200 http://bitbucket.org/pypy/pypy/changeset/4e50c695d26e/ Log: Fix build issues diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -308,10 +308,12 @@ @unwrap_spec(w_modulus=WrappedDefault(None)) def descr_pow(self, space, w_exponent, w_modulus=None): + exp_int = 0 + exp_bigint = None + sign = 0 + if isinstance(w_exponent, W_AbstractIntObject): exp_int = w_exponent.int_w(space) - sign = 0 - use_int = True if exp_int > 0: sign = 1 elif exp_int < 0: @@ -321,14 +323,13 @@ else: exp_bigint = w_exponent.asbigint() sign = exp_bigint.sign - use_int = False if space.is_none(w_modulus): if sign < 0: self = self.descr_float(space) w_exponent = w_exponent.descr_float(space) return space.pow(self, w_exponent, space.w_None) - if use_int: + if not exp_bigint: return W_LongObject(self.num.int_pow(exp_int)) else: return W_LongObject(self.num.pow(exp_bigint)) @@ -344,7 +345,7 @@ "pow() 2nd argument cannot be negative when 3rd " "argument specified") try: - if use_int: + if not exp_bigint: result = self.num.int_pow(exp_int, w_modulus.asbigint()) else: result = self.num.pow(exp_bigint, w_modulus.asbigint()) diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -939,7 +939,7 @@ if v.sign != wsign: if div.sign == 0: - div = NEGATIVERBIGINT + div = ONENEGATIVERBIGINT else: div = div.int_sub(1) mod = w - mod From pypy.commits at gmail.com Thu Oct 19 12:51:03 2017 From: pypy.commits at gmail.com (rlamy) Date: Thu, 19 Oct 2017 09:51:03 -0700 (PDT) Subject: [pypy-commit] pypy default: Code cleanup (to reduce diff with py3.5) Message-ID: <59e8d7f7.55ad1c0a.dc832.2b63@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92800:058674d3d73a Date: 2017-10-19 17:50 +0100 http://bitbucket.org/pypy/pypy/changeset/058674d3d73a/ Log: Code cleanup (to reduce diff with py3.5) diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,60 +8,60 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) - if '_type_' in typedict: - ffiarray = _rawffi.Array(typedict['_type_']._ffishape_) - res._ffiarray = ffiarray - subletter = getattr(typedict['_type_'], '_type_', None) - if subletter == 'c': - def getvalue(self): - return _rawffi.charp2string(self._buffer.buffer, + if '_type_' not in typedict: + res._ffiarray = None + return res + ffiarray = _rawffi.Array(typedict['_type_']._ffishape_) + res._ffiarray = ffiarray + subletter = getattr(typedict['_type_'], '_type_', None) + if subletter == 'c': + def getvalue(self): + return _rawffi.charp2string(self._buffer.buffer, + self._length_) + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, str): + _rawffi.rawstring2charp(self._buffer.buffer, val) + else: + for i in range(len(val)): + self[i] = val[i] + if len(val) < self._length_: + self._buffer[len(val)] = '\x00' + res.value = property(getvalue, setvalue) + + def getraw(self): + return _rawffi.charp2rawstring(self._buffer.buffer, self._length_) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, str): - _rawffi.rawstring2charp(self._buffer.buffer, val) - else: - for i in range(len(val)): - self[i] = val[i] - if len(val) < self._length_: - self._buffer[len(val)] = '\x00' - res.value = property(getvalue, setvalue) - def getraw(self): - return _rawffi.charp2rawstring(self._buffer.buffer, - self._length_) + def setraw(self, buffer): + if len(buffer) > self._length_: + raise ValueError("%r too long" % (buffer,)) + _rawffi.rawstring2charp(self._buffer.buffer, buffer) + res.raw = property(getraw, setraw) + elif subletter == 'u': + def getvalue(self): + return _rawffi.wcharp2unicode(self._buffer.buffer, + self._length_) - def setraw(self, buffer): - if len(buffer) > self._length_: - raise ValueError("%r too long" % (buffer,)) - _rawffi.rawstring2charp(self._buffer.buffer, buffer) - res.raw = property(getraw, setraw) - elif subletter == 'u': - def getvalue(self): - return _rawffi.wcharp2unicode(self._buffer.buffer, - self._length_) + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, unicode): + target = self._buffer + else: + target = self + for i in range(len(val)): + target[i] = val[i] + if len(val) < self._length_: + target[len(val)] = u'\x00' + res.value = property(getvalue, setvalue) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, unicode): - target = self._buffer - else: - target = self - for i in range(len(val)): - target[i] = val[i] - if len(val) < self._length_: - target[len(val)] = u'\x00' - res.value = property(getvalue, setvalue) - - if '_length_' in typedict: - res._ffishape_ = (ffiarray, typedict['_length_']) - res._fficompositesize_ = res._sizeofinstances() - else: - res._ffiarray = None + if '_length_' in typedict: + res._ffishape_ = (ffiarray, typedict['_length_']) + res._fficompositesize_ = res._sizeofinstances() return res from_address = cdata_from_address From pypy.commits at gmail.com Thu Oct 19 13:21:08 2017 From: pypy.commits at gmail.com (rlamy) Date: Thu, 19 Oct 2017 10:21:08 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix test_abstract in ctypes/test/test_frombuffer.py Message-ID: <59e8df04.87bbdf0a.d95c4.048b@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92801:05b154658972 Date: 2017-10-19 18:20 +0100 http://bitbucket.org/pypy/pypy/changeset/05b154658972/ Log: Fix test_abstract in ctypes/test/test_frombuffer.py diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,9 +8,14 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) + if cls == (_CData,): # this is the Array class defined below + res._ffiarray = None return res - + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) subletter = getattr(res._type_, '_type_', None) if subletter == 'c': @@ -55,7 +60,7 @@ for i in range(len(val)): target[i] = val[i] if len(val) < self._length_: - target[len(val)] = '\x00' + target[len(val)] = u'\x00' res.value = property(getvalue, setvalue) res._ffishape_ = (ffiarray, res._length_) @@ -164,7 +169,7 @@ if letter == 'c': return b"".join(l) if letter == 'u': - return "".join(l) + return u"".join(l) return l class Array(_CData, metaclass=ArrayMeta): From pypy.commits at gmail.com Thu Oct 19 13:39:48 2017 From: pypy.commits at gmail.com (rlamy) Date: Thu, 19 Oct 2017 10:39:48 -0700 (PDT) Subject: [pypy-commit] pypy default: backport changes from py3.5 Message-ID: <59e8e364.c5061c0a.d9b9.d4fa@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92802:e0fe31dff118 Date: 2017-10-19 18:38 +0100 http://bitbucket.org/pypy/pypy/changeset/e0fe31dff118/ Log: backport changes from py3.5 diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,12 +8,16 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) - if '_type_' not in typedict: + + if cls == (_CData,): # this is the Array class defined below res._ffiarray = None return res - ffiarray = _rawffi.Array(typedict['_type_']._ffishape_) - res._ffiarray = ffiarray - subletter = getattr(typedict['_type_'], '_type_', None) + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") + ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) + subletter = getattr(res._type_, '_type_', None) if subletter == 'c': def getvalue(self): return _rawffi.charp2string(self._buffer.buffer, @@ -28,12 +32,12 @@ for i in range(len(val)): self[i] = val[i] if len(val) < self._length_: - self._buffer[len(val)] = '\x00' + self._buffer[len(val)] = b'\x00' res.value = property(getvalue, setvalue) def getraw(self): return _rawffi.charp2rawstring(self._buffer.buffer, - self._length_) + self._length_) def setraw(self, buffer): if len(buffer) > self._length_: @@ -43,7 +47,7 @@ elif subletter == 'u': def getvalue(self): return _rawffi.wcharp2unicode(self._buffer.buffer, - self._length_) + self._length_) def setvalue(self, val): # we don't want to have buffers here @@ -59,9 +63,8 @@ target[len(val)] = u'\x00' res.value = property(getvalue, setvalue) - if '_length_' in typedict: - res._ffishape_ = (ffiarray, typedict['_length_']) - res._fficompositesize_ = res._sizeofinstances() + res._ffishape_ = (ffiarray, res._length_) + res._fficompositesize_ = res._sizeofinstances() return res from_address = cdata_from_address @@ -156,7 +159,7 @@ l = [self[i] for i in range(start, stop, step)] letter = getattr(self._type_, '_type_', None) if letter == 'c': - return "".join(l) + return b"".join(l) if letter == 'u': return u"".join(l) return l From pypy.commits at gmail.com Thu Oct 19 19:18:23 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 16:18:23 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Ups, not suppose to include this code Message-ID: <59e932bf.2281df0a.bc00.0de2@mx.google.com> Author: stian Branch: math-improvements Changeset: r92806:6e8fd621566a Date: 2017-10-20 01:17 +0200 http://bitbucket.org/pypy/pypy/changeset/6e8fd621566a/ Log: Ups, not suppose to include this code diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -748,8 +748,6 @@ return rbigint(self._digits[:asize], self.sign * bsign, asize) elif digit & (digit - 1) == 0: result = self.lqshift(ptwotable[digit]) - elif digit == 10: - result = self.lqshift(3).add(self.lqshift(1)) elif asize == 1: res = self.widedigit(0) * digit carry = res >> SHIFT From pypy.commits at gmail.com Thu Oct 19 19:18:17 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 16:18:17 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Make use of int_sub in longobjects Message-ID: <59e932b9.4cb91c0a.71688.323c@mx.google.com> Author: stian Branch: math-improvements Changeset: r92803:62ba73fa6cac Date: 2017-10-19 22:47 +0200 http://bitbucket.org/pypy/pypy/changeset/62ba73fa6cac/ Log: Make use of int_sub in longobjects diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -395,10 +395,14 @@ methname = opname + '_' if opname in ('and', 'or') else opname descr_rname = 'descr_r' + opname op = getattr(rbigint, methname) + intop = getattr(rbigint, 'int_' + methname) @func_renamer('descr_' + opname) - @delegate_other def descr_binop(self, space, w_other): + if isinstance(w_other, W_AbstractIntObject): + return W_LongObject(intop(self.num, w_other.int_w(space))) + elif not isinstance(w_other, W_AbstractLongObject): + return space.w_NotImplemented return W_LongObject(op(self.num, w_other.asbigint())) @func_renamer(descr_rname) From pypy.commits at gmail.com Thu Oct 19 19:18:19 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 16:18:19 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Avoid copy if the sign stay the same in int_mul Message-ID: <59e932bb.6896df0a.aaabd.13ca@mx.google.com> Author: stian Branch: math-improvements Changeset: r92804:0f49fdf5b940 Date: 2017-10-19 23:03 +0200 http://bitbucket.org/pypy/pypy/changeset/0f49fdf5b940/ Log: Avoid copy if the sign stay the same in int_mul diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -742,6 +742,8 @@ bsign = -1 if b < 0 else 1 if digit == 1: + if self.sign == bsign: + return self return rbigint(self._digits[:self.numdigits()], self.sign * bsign, asize) elif asize == 1: res = self.widedigit(0) * digit From pypy.commits at gmail.com Thu Oct 19 19:18:21 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 16:18:21 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Prebuild nulldigits list for use on null results where the list is not modified, reorder if operations in int_mul so power of two happens before single size check Message-ID: <59e932bd.26afdf0a.2a535.652e@mx.google.com> Author: stian Branch: math-improvements Changeset: r92805:587f1f780bd1 Date: 2017-10-19 23:43 +0200 http://bitbucket.org/pypy/pypy/changeset/587f1f780bd1/ Log: Prebuild nulldigits list for use on null results where the list is not modified, reorder if operations in int_mul so power of two happens before single size check diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -108,6 +108,7 @@ NULLDIGIT = _store_digit(0) ONEDIGIT = _store_digit(1) +NULLDIGITS = [NULLDIGIT] def _check_digits(l): for x in l: @@ -139,7 +140,7 @@ _immutable_ = True _immutable_fields_ = ["_digits"] - def __init__(self, digits=[NULLDIGIT], sign=0, size=0): + def __init__(self, digits=NULLDIGITS, sign=0, size=0): if not we_are_translated(): _check_digits(digits) make_sure_not_resized(digits) @@ -742,9 +743,13 @@ bsign = -1 if b < 0 else 1 if digit == 1: - if self.sign == bsign: + if bsign == 1: return self - return rbigint(self._digits[:self.numdigits()], self.sign * bsign, asize) + return rbigint(self._digits[:asize], self.sign * bsign, asize) + elif digit & (digit - 1) == 0: + result = self.lqshift(ptwotable[digit]) + elif digit == 10: + result = self.lqshift(3).add(self.lqshift(1)) elif asize == 1: res = self.widedigit(0) * digit carry = res >> SHIFT @@ -752,9 +757,6 @@ return rbigint([_store_digit(res & MASK), _store_digit(carry)], self.sign * bsign, 2) else: return rbigint([_store_digit(res & MASK)], self.sign * bsign, 1) - - elif digit & (digit - 1) == 0: - result = self.lqshift(ptwotable[digit]) else: result = _muladd1(self, digit) @@ -1395,7 +1397,7 @@ self.size = i if self.numdigits() == 1 and self._digits[0] == NULLDIGIT: self.sign = 0 - self._digits = [NULLDIGIT] + self._digits = NULLDIGITS _normalize._always_inline_ = True @@ -1486,7 +1488,7 @@ if x > 0: return digits_from_nonneg_long(x), 1 elif x == 0: - return [NULLDIGIT], 0 + return NULLDIGITS, 0 elif x != most_neg_value_of_same_type(x): # normal case return digits_from_nonneg_long(-x), -1 @@ -1504,7 +1506,7 @@ def args_from_long(x): if x >= 0: if x == 0: - return [NULLDIGIT], 0 + return NULLDIGITS, 0 else: return digits_from_nonneg_long(x), 1 else: @@ -1766,8 +1768,8 @@ size_lo = min(size_n, size) # We use "or" her to avoid having a check where list can be empty in _normalize. - lo = rbigint(n._digits[:size_lo] or [NULLDIGIT], 1) - hi = rbigint(n._digits[size_lo:n.size] or [NULLDIGIT], 1) + lo = rbigint(n._digits[:size_lo] or NULLDIGITS, 1) + hi = rbigint(n._digits[size_lo:n.size] or NULLDIGITS, 1) lo._normalize() hi._normalize() return hi, lo diff --git a/rpython/rlib/test/test_rbigint.py b/rpython/rlib/test/test_rbigint.py --- a/rpython/rlib/test/test_rbigint.py +++ b/rpython/rlib/test/test_rbigint.py @@ -28,8 +28,8 @@ class TestRLong(object): def test_simple(self): - for op1 in [-2, -1, 0, 1, 2, 50]: - for op2 in [-2, -1, 0, 1, 2, 50]: + for op1 in [-2, -1, 0, 1, 2, 10, 50]: + for op2 in [-2, -1, 0, 1, 2, 10, 50]: rl_op1 = rbigint.fromint(op1) rl_op2 = rbigint.fromint(op2) for op in "add sub mul".split(): From pypy.commits at gmail.com Thu Oct 19 20:07:35 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 19 Oct 2017 17:07:35 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Undo this move, it is actully 50% slower for power of two calculations on single digit longs Message-ID: <59e93e47.2f97df0a.f843c.37aa@mx.google.com> Author: stian Branch: math-improvements Changeset: r92807:bbeb35d80f5f Date: 2017-10-20 02:05 +0200 http://bitbucket.org/pypy/pypy/changeset/bbeb35d80f5f/ Log: Undo this move, it is actully 50% slower for power of two calculations on single digit longs diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -746,8 +746,6 @@ if bsign == 1: return self return rbigint(self._digits[:asize], self.sign * bsign, asize) - elif digit & (digit - 1) == 0: - result = self.lqshift(ptwotable[digit]) elif asize == 1: res = self.widedigit(0) * digit carry = res >> SHIFT @@ -755,6 +753,8 @@ return rbigint([_store_digit(res & MASK), _store_digit(carry)], self.sign * bsign, 2) else: return rbigint([_store_digit(res & MASK)], self.sign * bsign, 1) + elif digit & (digit - 1) == 0: + result = self.lqshift(ptwotable[digit]) else: result = _muladd1(self, digit) From pypy.commits at gmail.com Fri Oct 20 06:48:30 2017 From: pypy.commits at gmail.com (mattip) Date: Fri, 20 Oct 2017 03:48:30 -0700 (PDT) Subject: [pypy-commit] buildbot default: setup nightly scheduler for rpython builds, add change_source conditional filter Message-ID: <59e9d47e.c7c4df0a.3450d.6380@mx.google.com> Author: Matti Picus Branch: Changeset: r1031:f7ccb0e5313a Date: 2017-10-18 16:42 +0300 http://bitbucket.org/pypy/buildbot/changeset/f7ccb0e5313a/ Log: setup nightly scheduler for rpython builds, add change_source conditional filter diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -11,6 +11,8 @@ from pypybuildbot.pypylist import PyPyList, NumpyStatusList from pypybuildbot.ircbot import IRC # side effects from pypybuildbot.util import we_are_debugging +from buildbot.changes import filter +from buildbot.changes.hgpoller import HgPoller # Forbid "force build" with empty user name class CustomForceScheduler(ForceScheduler): @@ -270,7 +272,10 @@ BuildmasterConfig = { 'slavePortnum': slavePortnum, - 'change_source': [], + 'change_source': [ + HgPoller('https://bitbucket.org/pypy/pypy/', workdir='hgpoller-workdir', + branch='default', pollinterval=300), + ], 'schedulers': [ # the benchmarks run on tannit and (planned) speed-old.python.org. @@ -280,9 +285,6 @@ LINUX32OWN, # on tannit32, uses all cores LINUX64OWN, # on bencher4, uses all cores WIN32OWN, # on allegro_win32, SalsaSalsa - LINUX32RPYTHON, # on tannit32, uses all cores - LINUX64RPYTHON, # on bencher4, uses all cores - WIN32RPYTHON, # on allegro_win32, SalsaSalsa JITLINUX32, # on tannit32, uses 1 core JITLINUX64, # on bencher4, uses 1 core #APPLVLLINUX32, # on tannit32, uses 1 core @@ -298,6 +300,15 @@ #PYPYBUILDBOT # on cobra ], branch='default', hour=0, minute=0), + Nightly("nightly-0-01", [ + LINUX32RPYTHON, # on tannit32, uses all cores + LINUX64RPYTHON, # on bencher4, uses all cores + WIN32RPYTHON, # on allegro_win32, SalsaSalsa + ], branch='default', hour=0, minute=0, onlyIfChanged=True, + fileIsImportant=None, # set this to only rpython changes + change_filter=filter.ChangeFilter(branch='default'), + ), + Nightly("nightly-1-00", [ JITBENCH, # on tannit32, uses 1 core (in part exclusively) JITBENCH64, # on tannit64, uses 1 core (in part exclusively) From pypy.commits at gmail.com Fri Oct 20 06:48:32 2017 From: pypy.commits at gmail.com (mattip) Date: Fri, 20 Oct 2017 03:48:32 -0700 (PDT) Subject: [pypy-commit] buildbot default: add fileIsImportant filter for rpython, should it exclude doc subdirectory? Message-ID: <59e9d480.c23a1c0a.20b1b.a036@mx.google.com> Author: Matti Picus Branch: Changeset: r1032:3118f11e324e Date: 2017-10-20 08:51 +0300 http://bitbucket.org/pypy/buildbot/changeset/3118f11e324e/ Log: add fileIsImportant filter for rpython, should it exclude doc subdirectory? diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -4,6 +4,7 @@ from buildbot.schedulers.forcesched import ForceScheduler from buildbot.schedulers.forcesched import ValidationError from buildbot.buildslave import BuildSlave +from buildbot.buildslave.base import log from buildbot.status.html import WebStatus from buildbot.status.web import authz from buildbot.process.build import Build @@ -269,12 +270,19 @@ 'hhu-pypy-pi2': {'max_builds': 1}, } +def isRPython(change): + for fname in change.files: + if fname.startswith('rpython'): + log.msg('fileIsImportant filter isRPython got "%s"' % fname) + return True + return False + BuildmasterConfig = { 'slavePortnum': slavePortnum, 'change_source': [ HgPoller('https://bitbucket.org/pypy/pypy/', workdir='hgpoller-workdir', - branch='default', pollinterval=300), + branch='default', pollinterval=20*60), ], 'schedulers': [ @@ -305,7 +313,7 @@ LINUX64RPYTHON, # on bencher4, uses all cores WIN32RPYTHON, # on allegro_win32, SalsaSalsa ], branch='default', hour=0, minute=0, onlyIfChanged=True, - fileIsImportant=None, # set this to only rpython changes + fileIsImportant=isRPython, change_filter=filter.ChangeFilter(branch='default'), ), From pypy.commits at gmail.com Fri Oct 20 06:48:28 2017 From: pypy.commits at gmail.com (mattip) Date: Fri, 20 Oct 2017 03:48:28 -0700 (PDT) Subject: [pypy-commit] buildbot default: refactor {Own} into {Own, RPython} Untranslated factories, remove indiana bot Message-ID: <59e9d47c.908a1c0a.7e5c9.6904@mx.google.com> Author: Matti Picus Branch: Changeset: r1030:53cae65f1475 Date: 2017-10-17 20:44 +0300 http://bitbucket.org/pypy/buildbot/changeset/53cae65f1475/ Log: refactor {Own} into {Own, RPython} Untranslated factories, remove indiana bot diff --git a/bot2/pypybuildbot/arm_master.py b/bot2/pypybuildbot/arm_master.py --- a/bot2/pypybuildbot/arm_master.py +++ b/bot2/pypybuildbot/arm_master.py @@ -18,6 +18,7 @@ timeout=36000) pypyJitOnlyOwnTestFactoryARM = pypybuilds.Own(cherrypick="jit", timeout=2 * 3600) pypyOwnTestFactoryARM = pypybuilds.Own(timeout=2 * 3600) +pypyRPythonTestFactoryARM = pypybuilds.RPython(timeout=2 * 3600) pypyCrossTranslationFactoryARM = pypybuilds.NightlyBuild( translationArgs=crosstranslationargs + ['-O2'], @@ -100,7 +101,8 @@ platform='linux-armhf-raring', ) # -LINUXARMHF = "own-linux-armhf" +LINUXARMHFOWN = "own-linux-armhf" +LINUXARMHFRPYTHON = "rpython-linux-armhf" APPLVLLINUXARM = "pypy-c-app-level-linux-armel" APPLVLLINUXARMHF_v7 = "pypy-c-app-level-linux-armhf-v7" APPLVLLINUXARMHF_RASPBIAN = "pypy-c-app-level-linux-armhf-raspbian" diff --git a/bot2/pypybuildbot/builds.py b/bot2/pypybuildbot/builds.py --- a/bot2/pypybuildbot/builds.py +++ b/bot2/pypybuildbot/builds.py @@ -479,14 +479,14 @@ # ---- -class Own(factory.BuildFactory): +class Untranslated(factory.BuildFactory): def __init__(self, platform='linux', cherrypick='', extra_cfgs=[], **kwargs): factory.BuildFactory.__init__(self) setup_steps(platform, self) - timeout=kwargs.get('timeout', 4000) + self.timeout=kwargs.get('timeout', 4000) self.addStep(shell.SetPropertyFromCommand( command=['python', '-c', "import tempfile, os ;print" @@ -494,21 +494,21 @@ property="target_tmpdir")) # If target_tmpdir is empty, crash. tmp_or_crazy = '%(prop:target_tmpdir:-crazy/name/so/mkdir/fails/)s' - pytest = "pytest" + self.pytest = "pytest" self.addStep(ShellCmd( description="mkdir for tests", command=['python', '-c', Interpolate("import os; os.mkdir(r'" + \ - tmp_or_crazy + pytest + "') if not os.path.exists(r'" + \ - tmp_or_crazy + pytest + "') else True")], + tmp_or_crazy + self.pytest + "') if not os.path.exists(r'" + \ + tmp_or_crazy + self.pytest + "') else True")], haltOnFailure=True, )) nDays = '3' #str, not int if platform == 'win32': - command = ['FORFILES', '/P', Interpolate(tmp_or_crazy + pytest), + command = ['FORFILES', '/P', Interpolate(tmp_or_crazy + self.pytest), '/D', '-' + nDays, '/c', "cmd /c rmdir /q /s @path"] else: - command = ['find', Interpolate(tmp_or_crazy + pytest), '-mtime', + command = ['find', Interpolate(tmp_or_crazy + self.pytest), '-mtime', '+' + nDays, '-exec', 'rm', '-r', '{}', ';'] self.addStep(SuccessAlways( description="cleanout old test files", @@ -524,44 +524,55 @@ )) if platform == 'win32': - virt_python = r'virt_test\Scripts\python.exe' + self.virt_python = r'virt_test\Scripts\python.exe' else: - virt_python = 'virt_test/bin/python' + self.virt_python = 'virt_test/bin/python' self.addStep(ShellCmd( description="install requirments to virtual environment", - command=[virt_python, '-mpip', 'install', '-r', 'requirements.txt'], + command=[self.virt_python, '-mpip', 'install', '-r', + 'requirements.txt'], haltOnFailure=True, )) + + +class Own(Untranslated): + def __init__(self, platform='linux', cherrypick='', extra_cfgs=[], **kwargs): + Untranslated.__init__(self, platform=platform, cherrypick=cherrypick, + extra_cfgs=extra_cfgs, **kwargs) self.addStep(PytestCmd( description="pytest pypy", - command=[virt_python, "testrunner/runner.py", + command=[self.virt_python, "testrunner/runner.py", "--logfile=testrun.log", "--config=pypy/testrunner_cfg.py", "--config=~/machine_cfg.py", - "--root=pypy", "--timeout=%s" % (timeout,) + "--root=pypy", "--timeout=%s" % (self.timeout,) ] + ["--config=%s" % cfg for cfg in extra_cfgs], logfiles={'pytestLog': 'testrun.log'}, - timeout=timeout, + timeout=self.timeout, env={"PYTHONPATH": ['.'], "PYPYCHERRYPICK": cherrypick, - "TMPDIR": Interpolate('%(prop:target_tmpdir)s' + pytest), + "TMPDIR": Interpolate('%(prop:target_tmpdir)s' + self.pytest), })) +class RPython(Untranslated): + def __init__(self, platform='linux', cherrypick='', extra_cfgs=[], **kwargs): + Untranslated.__init__(self, platform=platform, cherrypick=cherrypick, + extra_cfgs=extra_cfgs, **kwargs) self.addStep(PytestCmd( description="pytest rpython", - command=[virt_python, "testrunner/runner.py", + command=[self.virt_python, "testrunner/runner.py", "--logfile=testrun.log", "--config=pypy/testrunner_cfg.py", "--config=~/machine_cfg.py", - "--root=rpython", "--timeout=%s" % (timeout,) + "--root=rpython", "--timeout=%s" % (self.timeout,) ] + ["--config=%s" % cfg for cfg in extra_cfgs], logfiles={'pytestLog': 'testrun.log'}, - timeout=timeout, + timeout=self.timeout, env={"PYTHONPATH": ['.'], "PYPYCHERRYPICK": cherrypick, - "TMPDIR": Interpolate('%(prop:target_tmpdir)s' + pytest), + "TMPDIR": Interpolate('%(prop:target_tmpdir)s' + self.pytest), })) diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -69,11 +69,17 @@ pypyOwnTestFactory = pypybuilds.Own() pypyOwnTestFactoryWin = pypybuilds.Own(platform="win32") -pypyOwnTestFactoryIndiana = pypybuilds.Own(platform="indiana32") pypyJitOnlyOwnTestFactory = pypybuilds.Own(cherrypick="jit") # OSX 32bit tests require a larger timeout to finish -pypyOwnTestFactoryOSX32 = pypybuilds.Own(timeout=3*3600) +pypyOwnTestFactoryOSX32 = pypybuilds.Own(timeout=3*3600) # XXX Own or RPython? + +pypyRPythonTestFactory = pypybuilds.RPython() +pypyRPythonTestFactoryWin = pypybuilds.RPython(platform="win32") +pypyJitOnlyRPythonTestFactory = pypybuilds.RPython(cherrypick="jit") + +# OSX 32bit tests require a larger timeout to finish +pypyRPythonTestFactoryOSX32 = pypybuilds.RPython(timeout=3*3600) # XXX Own or RPython? pypyTranslatedAppLevelTestFactory = pypybuilds.Translated(lib_python=True, app_tests=True) @@ -118,15 +124,6 @@ trigger='NUMPY64_scheduler', ) -pypyJITTranslatedTestFactoryIndiana = pypybuilds.Translated( - translationArgs=jit_translation_args, - targetArgs=[], - lib_python=True, - pypyjit=True, - app_tests=True, - platform='openindiana32', - ) - pypyJITTranslatedTestFactoryOSX = pypybuilds.Translated( platform='osx', translationArgs=jit_translation_args + ['--make-jobs=1'], @@ -187,12 +184,20 @@ # -LINUX32 = "own-linux-x86-32" -LINUX64 = "own-linux-x86-64" -LINUX_S390X = "own-linux-s390x" +LINUX32OWN = "own-linux-x86-32" +LINUX64OWN = "own-linux-x86-64" +LINUX_S390XOWN = "own-linux-s390x" +MACOSX32OWN = "own-macosx-x86-32" +WIN32OWN = "own-win-x86-32" +WIN64OWN = "own-win-x86-64" -MACOSX32 = "own-macosx-x86-32" -WIN32 = "own-win-x86-32" +LINUX32RPYTHON = "rpython-linux-x86-32" +LINUX64RPYTHON = "rpython-linux-x86-64" +LINUX_S390XRPYTHON = "rpython-linux-s390x" +MACOSX32RPYTHON = "rpython-macosx-x86-32" +WIN32RPYTHON = "rpython-win-x86-32" +WIN64RPYTHON = "rpython-win-x86-64" + APPLVLLINUX32 = "pypy-c-app-level-linux-x86-32" APPLVLLINUX64 = "pypy-c-app-level-linux-x86-64" APPLVLWIN32 = "pypy-c-app-level-win-x86-32" @@ -218,26 +223,23 @@ PYPYBUILDBOT = 'pypy-buildbot' JITFREEBSD964 = 'pypy-c-jit-freebsd-9-x86-64' -WIN64 = "own-win-x86-64" -INDIANA32 = "own-indiana-x86-32" JITWIN64 = "pypy-c-jit-win-x86-64" JITFREEBSD764 = 'pypy-c-jit-freebsd-7-x86-64' JITFREEBSD864 = 'pypy-c-jit-freebsd-8-x86-64' -JITINDIANA32 = "pypy-c-jit-indiana-x86-32" JITBENCH64_NEW = 'jit-benchmark-linux-x86-64-single-run' inactive_slaves = [ - {"name": WIN64, + {"name": WIN64OWN, "slavenames": [], - "builddir": WIN64, + "builddir": WIN64OWN, "factory": pypyOwnTestFactoryWin, "category": 'win32' }, - {'name': INDIANA32, - 'slavenames': [], - 'builddir': INDIANA32, - 'factory': pypyOwnTestFactoryIndiana, - 'category': 'openindiana32', - }, + {"name": WIN64RPYTHON, + "slavenames": [], + "builddir": WIN64RPYTHON, + "factory": pypyOwnTestFactoryWin, + "category": 'win32' + }, {"name" : JITWIN64, "slavenames": [], 'builddir' : JITWIN64, @@ -256,13 +258,6 @@ 'factory' : pypyJITTranslatedTestFactoryFreeBSD, "category": 'freebsd64' }, - # openindiana - {'name': JITINDIANA32, - 'slavenames': [], - 'builddir': JITINDIANA32, - 'factory': pypyJITTranslatedTestFactoryIndiana, - 'category': 'openindiana32', - }, ] extra_opts = {'xerxes': {'keepalive_interval': 15}, 'aurora': {'max_builds': 1}, @@ -282,8 +277,12 @@ # All the other linux tests run on bencher4.soft-dev.org. Nightly("nightly-0-00", [ # linux tests - LINUX32, # on tannit32, uses all cores - LINUX64, # on bencher4, uses all cores + LINUX32OWN, # on tannit32, uses all cores + LINUX64OWN, # on bencher4, uses all cores + WIN32OWN, # on allegro_win32, SalsaSalsa + LINUX32RPYTHON, # on tannit32, uses all cores + LINUX64RPYTHON, # on bencher4, uses all cores + WIN32RPYTHON, # on allegro_win32, SalsaSalsa JITLINUX32, # on tannit32, uses 1 core JITLINUX64, # on bencher4, uses 1 core #APPLVLLINUX32, # on tannit32, uses 1 core @@ -291,7 +290,6 @@ # other platforms #MACOSX32, # on minime JITWIN32, # on allegro_win32, SalsaSalsa - WIN32, # on allegro_win32, SalsaSalsa #JITFREEBSD764, # on headless #JITFREEBSD864, # on ananke JITFREEBSD964, # on tavendo @@ -319,16 +317,21 @@ # branch="py3.5", hour=3, minute=0), Nightly("nightly-3-00-py3.5", [ - LINUX32, # on tannit32, uses all cores + LINUX32OWN, # on tannit32, uses all cores + LINUX32RPYTHON, # on tannit32, uses all cores JITLINUX32, # on tannit32, uses 1 core - LINUX64, # on bencher4, uses all cores + LINUX64OWN, # on bencher4, uses all cores + LINUX64RPYTHON, # on bencher4, uses all cores JITLINUX64, # on bencher4, uses 1 core JITMACOSX64, # on xerxes JITWIN32, # on allegro_win32, SalsaSalsa ], branch="py3.5", hour=3, minute=0), # S390X vm (ibm-research) - Nightly("nightly-4-00", [LINUX_S390X], branch='default', hour=0, minute=0), + Nightly("nightly-4-00", [ + LINUX_S390XOWN, + LINUX_S390XRPYTHON, + ], branch='default', hour=0, minute=0), Nightly("nightly-4-01", [JITLINUX_S390X], branch='default', hour=2, minute=0), # this one has faithfully run every night even though the latest @@ -340,11 +343,15 @@ CustomForceScheduler('Force Scheduler', builderNames=[ PYPYBUILDBOT, - LINUX32, - LINUX64, + LINUX32OWN, + LINUX64OWN, + MACOSX32OWN, + WIN32OWN, + LINUX32RPYTHON, + LINUX64RPYTHON, + MACOSX32RPYTHON, + WIN32RPYTHON, - MACOSX32, - WIN32, APPLVLLINUX32, APPLVLLINUX64, APPLVLWIN32, @@ -364,15 +371,14 @@ JITBENCH64_NEW, NUMPY_64, NUMPY_WIN, - #INDIANA32, - #WIN64, + #WIN64OWN, #JITMACOSX64_2, #JITWIN64, #JITFREEBSD764, #JITFREEBSD864, - #JITINDIANA32, - LINUX_S390X, + LINUX_S390XOWN, + LINUX_S390XRPYTHON, JITLINUX_S390X, ] + ARM.builderNames, properties=[]), @@ -385,21 +391,36 @@ in passwords.iteritems()], 'builders': [ - {"name": LINUX32, + {"name": LINUX32OWN, "slavenames": ["tannit32"], - "builddir": LINUX32, + "builddir": LINUX32OWN, "factory": pypyOwnTestFactory, "category": 'linux32', "locks": [TannitCPU.access('counting')], }, - {"name": LINUX64, + {"name": LINUX32RPYTHON, + "slavenames": ["tannit32"], + "builddir": LINUX32RPYTHON, + "factory": pypyRPythonTestFactory, + "category": 'linux32', + "locks": [TannitCPU.access('counting')], + }, + {"name": LINUX64OWN, #"slavenames": ["bencher4", "speed-old"], "slavenames": ["bencher4"], - "builddir": LINUX64, + "builddir": LINUX64OWN, "factory": pypyOwnTestFactory, "category": 'linux64', "locks": [Bencher4Lock.access('counting')], }, + {"name": LINUX64RPYTHON, + #"slavenames": ["bencher4", "speed-old"], + "slavenames": ["bencher4"], + "builddir": LINUX64RPYTHON, + "factory": pypyRPythonTestFactory, + "category": 'linux64', + "locks": [Bencher4Lock.access('counting')], + }, {"name": APPLVLLINUX32, #"slavenames": ["allegro32"], "slavenames": ["tannit32"], @@ -469,12 +490,18 @@ "category": "benchmark-run", "locks": [Bencher4Lock.access('exclusive')], }, - {"name": MACOSX32, + {"name": MACOSX32OWN, "slavenames": ["minime"], - "builddir": MACOSX32, + "builddir": MACOSX32OWN, "factory": pypyOwnTestFactoryOSX32, "category": 'mac32' }, + {"name": MACOSX32RPYTHON, + "slavenames": ["minime"], + "builddir": MACOSX32RPYTHON, + "factory": pypyRPythonTestFactoryOSX32, + "category": 'mac32' + }, {"name" : JITMACOSX64, "slavenames": ["rebuy-de", "tosh", "osx-10.9-x64-dw"], # "xerxes" 'builddir' : JITMACOSX64, @@ -487,13 +514,20 @@ # 'factory' : pypyJITTranslatedTestFactoryOSX64, # 'category' : 'mac64', # }, - {"name": WIN32, + {"name": WIN32OWN, "slavenames": ["SalsaSalsa", "allegro_win32", "anubis64"], - "builddir": WIN32, + "builddir": WIN32OWN, "factory": pypyOwnTestFactoryWin, "locks": [WinSlaveLock.access('counting')], "category": 'win32', }, + {"name": WIN32RPYTHON, + "slavenames": ["SalsaSalsa", "allegro_win32", "anubis64"], + "builddir": WIN32RPYTHON, + "factory": pypyRPythonTestFactoryWin, + "locks": [WinSlaveLock.access('counting')], + "category": 'win32', + }, {"name": APPLVLWIN32, "slavenames": ["SalsaSalsa", "allegro_win32"], "builddir": APPLVLWIN32, @@ -545,12 +579,18 @@ "locks": [Bencher4Lock.access('counting')], }, # S390X - {"name": LINUX_S390X, + {"name": LINUX_S390XOWN, "slavenames": ["s390x-slave"], - "builddir": LINUX_S390X, + "builddir": LINUX_S390XOWN, "factory": pypyOwnTestFactory, "category": 'linux-s390x', }, + {"name": LINUX_S390XRPYTHON, + "slavenames": ["s390x-slave"], + "builddir": LINUX_S390XRPYTHON, + "factory": pypyRPythonTestFactory, + "category": 'linux-s390x', + }, {'name': JITLINUX_S390X, 'slavenames': ['s390x-slave'], 'builddir': JITLINUX_S390X, From pypy.commits at gmail.com Fri Oct 20 17:16:41 2017 From: pypy.commits at gmail.com (rlamy) Date: Fri, 20 Oct 2017 14:16:41 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: @xfail a test: supporting bitfields in other-endian Structures is too messy Message-ID: <59ea67b9.ddb1df0a.9ccf2.d1c3@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92808:680d99dade82 Date: 2017-10-20 22:16 +0100 http://bitbucket.org/pypy/pypy/changeset/680d99dade82/ Log: @xfail a test: supporting bitfields in other-endian Structures is too messy diff --git a/lib-python/3/ctypes/test/test_bitfields.py b/lib-python/3/ctypes/test/test_bitfields.py --- a/lib-python/3/ctypes/test/test_bitfields.py +++ b/lib-python/3/ctypes/test/test_bitfields.py @@ -1,5 +1,5 @@ from ctypes import * -from ctypes.test import need_symbol +from ctypes.test import need_symbol, xfail import unittest import os @@ -279,6 +279,7 @@ self.assertEqual(b, b'\xef\xcd\xab\x21') @need_symbol('c_uint32') + @xfail def test_uint32_swap_big_endian(self): # Issue #23319 class Big(BigEndianStructure): From pypy.commits at gmail.com Fri Oct 20 18:08:31 2017 From: pypy.commits at gmail.com (rlamy) Date: Fri, 20 Oct 2017 15:08:31 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Kill save_module_content_for_future_reload(): reloading a built-in module is a no-op now Message-ID: <59ea73df.8388500a.60fec.c042@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92809:f80cb4368b95 Date: 2017-10-20 23:07 +0100 http://bitbucket.org/pypy/pypy/changeset/f80cb4368b95/ Log: Kill save_module_content_for_future_reload(): reloading a built-in module is a no-op now diff --git a/pypy/interpreter/app_main.py b/pypy/interpreter/app_main.py --- a/pypy/interpreter/app_main.py +++ b/pypy/interpreter/app_main.py @@ -574,10 +574,6 @@ except ValueError: pass # ignore "2 is not a valid file descriptor" - if we_are_translated(): - import __pypy__ - __pypy__.save_module_content_for_future_reload(sys) - mainmodule = type(sys)('__main__') sys.modules['__main__'] = mainmodule diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py --- a/pypy/interpreter/baseobjspace.py +++ b/pypy/interpreter/baseobjspace.py @@ -515,9 +515,8 @@ # patcher relies on this behaviour. w_mod2 = Module(self, w_name) self.setitem(w_modules, w_name, w_mod2) - w_mod.getdict(self) # unlazy w_initialdict - self.call_method(w_mod2.getdict(self), 'update', - w_mod.w_initialdict) + w_dict = w_mod.getdict(self) # unlazy + self.call_method(w_mod2.getdict(self), 'update', w_dict) return w_mod2 self.setitem(w_modules, w_name, w_mod) w_mod.init(self) diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py --- a/pypy/interpreter/mixedmodule.py +++ b/pypy/interpreter/mixedmodule.py @@ -10,11 +10,7 @@ class MixedModule(Module): applevel_name = None - - # The following attribute is None as long as the module has not been - # imported yet, and when it has been, it is mod.__dict__.copy() just - # after startup(). - w_initialdict = None + initialized = False lazy = False submodule_name = None @@ -23,7 +19,6 @@ Module.__init__(self, space, w_name) init_extra_module_attrs(space, self) self.lazy = True - self.lazy_initial_values_w = {} self.__class__.buildloaders() self.loaders = self.loaders.copy() # copy from the class to the inst self.submodules_w = [] @@ -47,27 +42,14 @@ def init(self, space): """This is called each time the module is imported or reloaded """ - if self.w_initialdict is not None: - # the module was already imported. Refresh its content with - # the saved dict, as done with built-in and extension modules - # on CPython. - space.call_method(self.w_dict, 'update', self.w_initialdict) - for w_submodule in self.submodules_w: name = space.text0_w(w_submodule.w_name) space.setitem(self.w_dict, space.newtext(name.split(".")[-1]), w_submodule) space.getbuiltinmodule(name) - if self.w_initialdict is None: + if not self.initialized: Module.init(self, space) - if not self.lazy and self.w_initialdict is None: - self.save_module_content_for_future_reload() - - def save_module_content_for_future_reload(self): - # Save the current dictionary in w_initialdict, for future - # reloads. This forces the dictionary if needed. - w_dict = self.getdict(self.space) - self.w_initialdict = self.space.call_method(w_dict, 'copy') + self.initialized = True @classmethod @not_rpython @@ -96,13 +78,6 @@ return w_value def setdictvalue(self, space, attr, w_value): - if self.lazy and attr not in self.lazy_initial_values_w: - # in lazy mode, the first time an attribute changes, - # we save away the old (initial) value. This allows - # a future getdict() call to build the correct - # self.w_initialdict, containing the initial value. - w_initial_value = self._load_lazily(space, attr) - self.lazy_initial_values_w[attr] = w_initial_value space.setitem_str(self.w_dict, attr, w_value) return True @@ -142,23 +117,11 @@ def _force_lazy_dict_now(self): # Force the dictionary by calling all lazy loaders now. - # This also saves in self.w_initialdict a copy of all the - # initial values, including if they have already been - # modified by setdictvalue(). space = self.space for name in self.loaders: w_value = self.get(name) space.setitem(self.w_dict, space.new_interned_str(name), w_value) self.lazy = False - self.save_module_content_for_future_reload() - for key, w_initial_value in self.lazy_initial_values_w.items(): - w_key = space.new_interned_str(key) - if w_initial_value is not None: - space.setitem(self.w_initialdict, w_key, w_initial_value) - else: - if space.finditem(self.w_initialdict, w_key) is not None: - space.delitem(self.w_initialdict, w_key) - del self.lazy_initial_values_w def _cleanup_(self): self.getdict(self.space) diff --git a/pypy/module/__pypy__/__init__.py b/pypy/module/__pypy__/__init__.py --- a/pypy/module/__pypy__/__init__.py +++ b/pypy/module/__pypy__/__init__.py @@ -92,8 +92,6 @@ 'set_debug' : 'interp_magic.set_debug', 'locals_to_fast' : 'interp_magic.locals_to_fast', 'set_code_callback' : 'interp_magic.set_code_callback', - 'save_module_content_for_future_reload': - 'interp_magic.save_module_content_for_future_reload', 'decode_long' : 'interp_magic.decode_long', '_promote' : 'interp_magic._promote', 'normalize_exc' : 'interp_magic.normalize_exc', @@ -131,7 +129,7 @@ raise else: pass # ok fine to ignore in this case - + if self.space.config.translation.jit: features = detect_cpu.getcpufeatures(model) self.extra_interpdef('jit_backend_features', diff --git a/pypy/module/__pypy__/interp_magic.py b/pypy/module/__pypy__/interp_magic.py --- a/pypy/module/__pypy__/interp_magic.py +++ b/pypy/module/__pypy__/interp_magic.py @@ -145,10 +145,6 @@ assert isinstance(w_frame, PyFrame) w_frame.locals2fast() - at unwrap_spec(w_module=MixedModule) -def save_module_content_for_future_reload(space, w_module): - w_module.save_module_content_for_future_reload() - def set_code_callback(space, w_callable): cache = space.fromcache(CodeHookCache) if space.is_none(w_callable): diff --git a/pypy/module/__pypy__/test/test_magic.py b/pypy/module/__pypy__/test/test_magic.py --- a/pypy/module/__pypy__/test/test_magic.py +++ b/pypy/module/__pypy__/test/test_magic.py @@ -2,18 +2,6 @@ class AppTestMagic: spaceconfig = dict(usemodules=['__pypy__']) - def test_save_module_content_for_future_reload(self): - import sys, __pypy__, imp - d = sys.dont_write_bytecode - sys.dont_write_bytecode = "hello world" - __pypy__.save_module_content_for_future_reload(sys) - sys.dont_write_bytecode = d - imp.reload(sys) - assert sys.dont_write_bytecode == "hello world" - # - sys.dont_write_bytecode = d - __pypy__.save_module_content_for_future_reload(sys) - def test_new_code_hook(self): # workaround for running on top of old CPython 2.7 versions def exec_(code, d): From pypy.commits at gmail.com Fri Oct 20 23:56:49 2017 From: pypy.commits at gmail.com (stian) Date: Fri, 20 Oct 2017 20:56:49 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Add uint128_t, and use it to speed up mul and lshift slightly. TODO: Tests missing. Message-ID: <59eac581.d1b8df0a.b4e74.9932@mx.google.com> Author: stian Branch: math-improvements Changeset: r92810:e4fc38341ad9 Date: 2017-10-21 05:56 +0200 http://bitbucket.org/pypy/pypy/changeset/e4fc38341ad9/ Log: Add uint128_t, and use it to speed up mul and lshift slightly. TODO: Tests missing. diff --git a/pypy/objspace/std/longobject.py b/pypy/objspace/std/longobject.py --- a/pypy/objspace/std/longobject.py +++ b/pypy/objspace/std/longobject.py @@ -460,11 +460,9 @@ return space.w_NotImplemented return func(self, space, w_other) else: - @delegate_other @func_renamer('descr_' + opname) def descr_binop(self, space, w_other): return func(self, space, w_other) - @delegate_other @func_renamer('descr_r' + opname) def descr_rbinop(self, space, w_other): if not isinstance(w_other, W_LongObject): diff --git a/rpython/rlib/rarithmetic.py b/rpython/rlib/rarithmetic.py --- a/rpython/rlib/rarithmetic.py +++ b/rpython/rlib/rarithmetic.py @@ -614,6 +614,7 @@ r_ulonglong = build_int('r_ulonglong', False, 64) r_longlonglong = build_int('r_longlonglong', True, 128) +r_ulonglonglong = build_int('r_ulonglonglong', False, 128) longlongmax = r_longlong(LONGLONG_TEST - 1) if r_longlong is not r_int: diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -27,6 +27,7 @@ else: UDIGIT_MASK = longlongmask LONG_TYPE = rffi.__INT128_T + ULONG_TYPE = rffi.__UINT128_T if LONG_BIT > SHIFT: STORE_TYPE = lltype.Signed UNSIGNED_TYPE = lltype.Unsigned @@ -40,7 +41,8 @@ STORE_TYPE = lltype.Signed UNSIGNED_TYPE = lltype.Unsigned LONG_TYPE = rffi.LONGLONG - + ULONG_TYPE = rffi.ULONGLONG + MASK = int((1 << SHIFT) - 1) FLOAT_MULTIPLIER = float(1 << SHIFT) @@ -97,6 +99,9 @@ def _widen_digit(x): return rffi.cast(LONG_TYPE, x) +def _unsigned_widen_digit(x): + return rffi.cast(ULONG_TYPE, x) + @specialize.argtype(0) def _store_digit(x): return rffi.cast(STORE_TYPE, x) @@ -139,15 +144,16 @@ """This is a reimplementation of longs using a list of digits.""" _immutable_ = True _immutable_fields_ = ["_digits"] - + def __init__(self, digits=NULLDIGITS, sign=0, size=0): if not we_are_translated(): _check_digits(digits) make_sure_not_resized(digits) self._digits = digits + assert size >= 0 + self.size = size or len(digits) - self.size = size or len(digits) self.sign = sign # __eq__ and __ne__ method exist for testingl only, they are not RPython! @@ -172,6 +178,12 @@ return _widen_digit(self._digits[x]) widedigit._always_inline_ = True + def uwidedigit(self, x): + """Return the x'th digit, as a long long int if needed + to have enough room to contain two digits.""" + return _unsigned_widen_digit(self._digits[x]) + uwidedigit._always_inline_ = True + def udigit(self, x): """Return the x'th digit, as an unsigned int.""" return _load_unsigned_digit(self._digits[x]) @@ -701,9 +713,9 @@ if a._digits[0] == NULLDIGIT: return NULLRBIGINT elif a._digits[0] == ONEDIGIT: - return rbigint(b._digits[:b.numdigits()], a.sign * b.sign, b.numdigits()) + return rbigint(b._digits[:bsize], a.sign * b.sign, bsize) elif bsize == 1: - res = b.widedigit(0) * a.widedigit(0) + res = b.uwidedigit(0) * a.uwidedigit(0) carry = res >> SHIFT if carry: return rbigint([_store_digit(res & MASK), _store_digit(carry)], a.sign * b.sign, 2) @@ -740,6 +752,7 @@ asize = self.numdigits() digit = abs(b) + bsign = -1 if b < 0 else 1 if digit == 1: @@ -747,7 +760,8 @@ return self return rbigint(self._digits[:asize], self.sign * bsign, asize) elif asize == 1: - res = self.widedigit(0) * digit + udigit = r_uint(digit) + res = self.uwidedigit(0) * udigit carry = res >> SHIFT if carry: return rbigint([_store_digit(res & MASK), _store_digit(carry)], self.sign * bsign, 2) @@ -839,7 +853,7 @@ rem = self.widedigit(size) size -= 1 while size >= 0: - rem = ((rem << SHIFT) + self.widedigit(size)) % digit + rem = ((rem << SHIFT) | self.digit(size)) % digit size -= 1 else: rem = self.digit(0) % digit @@ -880,7 +894,7 @@ rem = self.widedigit(size) size -= 1 while size >= 0: - rem = ((rem << SHIFT) + self.widedigit(size)) % digit + rem = ((rem << SHIFT) | self.digit(size)) % digit size -= 1 else: rem = self.digit(0) % digit @@ -1210,10 +1224,10 @@ oldsize = self.numdigits() newsize = oldsize + wordshift + 1 z = rbigint([NULLDIGIT] * newsize, self.sign, newsize) - accum = _widen_digit(0) + accum = _unsigned_widen_digit(0) j = 0 while j < oldsize: - accum += self.widedigit(j) << remshift + accum += self.uwidedigit(j) << remshift z.setdigit(wordshift, accum) accum >>= SHIFT wordshift += 1 @@ -1235,10 +1249,10 @@ oldsize = self.numdigits() z = rbigint([NULLDIGIT] * (oldsize + 1), self.sign, (oldsize + 1)) - accum = _widen_digit(0) + accum = _unsigned_widen_digit(0) i = 0 while i < oldsize: - accum += self.widedigit(i) << int_other + accum += self.uwidedigit(i) << int_other z.setdigit(i, accum) accum >>= SHIFT i += 1 @@ -1267,9 +1281,9 @@ z = rbigint([NULLDIGIT] * newsize, self.sign, newsize) i = 0 while i < newsize: - newdigit = (self.digit(wordshift) >> loshift) + newdigit = (self.udigit(wordshift) >> loshift) if i+1 < newsize: - newdigit |= (self.digit(wordshift+1) << hishift) + newdigit |= (self.udigit(wordshift+1) << hishift) z.setdigit(i, newdigit) i += 1 wordshift += 1 @@ -1661,11 +1675,11 @@ z = rbigint([NULLDIGIT] * (size_a + size_b), 1) i = UDIGIT_TYPE(0) while i < size_a: - f = a.widedigit(i) + f = a.uwidedigit(i) pz = i << 1 pa = i + 1 - carry = z.widedigit(pz) + f * f + carry = z.uwidedigit(pz) + f * f z.setdigit(pz, carry) pz += 1 carry >>= SHIFT @@ -1675,18 +1689,18 @@ # pyramid it appears. Same as adding f<<1 once. f <<= 1 while pa < size_a: - carry += z.widedigit(pz) + a.widedigit(pa) * f + carry += z.uwidedigit(pz) + a.uwidedigit(pa) * f pa += 1 z.setdigit(pz, carry) pz += 1 carry >>= SHIFT if carry: - carry += z.widedigit(pz) + carry += z.uwidedigit(pz) z.setdigit(pz, carry) pz += 1 carry >>= SHIFT if carry: - z.setdigit(pz, z.widedigit(pz) + carry) + z.setdigit(pz, z.uwidedigit(pz) + carry) assert (carry >> SHIFT) == 0 i += 1 z._normalize() @@ -1707,10 +1721,10 @@ size_a1 = UDIGIT_TYPE(size_a - 1) size_b1 = UDIGIT_TYPE(size_b - 1) while i < size_a1: - f0 = a.widedigit(i) - f1 = a.widedigit(i + 1) + f0 = a.uwidedigit(i) + f1 = a.uwidedigit(i + 1) pz = i - carry = z.widedigit(pz) + b.widedigit(0) * f0 + carry = z.uwidedigit(pz) + b.uwidedigit(0) * f0 z.setdigit(pz, carry) pz += 1 carry >>= SHIFT @@ -1722,14 +1736,14 @@ # b.widedigit(j + 1) * f0 < (2**(B-1) - 1)**2; so # carry + z.widedigit(pz) + b.widedigit(j + 1) * f0 + # b.widedigit(j) * f1 < 2**(2*B - 1) - 2**B < 2**LONG)BIT - 1 - carry += z.widedigit(pz) + b.widedigit(j + 1) * f0 + \ - b.widedigit(j) * f1 + carry += z.uwidedigit(pz) + b.uwidedigit(j + 1) * f0 + \ + b.uwidedigit(j) * f1 z.setdigit(pz, carry) pz += 1 carry >>= SHIFT j += 1 # carry < 2**(B + 1) - 2 - carry += z.widedigit(pz) + b.widedigit(size_b1) * f1 + carry += z.uwidedigit(pz) + b.uwidedigit(size_b1) * f1 z.setdigit(pz, carry) pz += 1 carry >>= SHIFT @@ -1740,17 +1754,17 @@ i += 2 if size_a & 1: pz = size_a1 - f = a.widedigit(pz) + f = a.uwidedigit(pz) pb = 0 - carry = _widen_digit(0) + carry = _unsigned_widen_digit(0) while pb < size_b: - carry += z.widedigit(pz) + b.widedigit(pb) * f + carry += z.uwidedigit(pz) + b.uwidedigit(pb) * f pb += 1 z.setdigit(pz, carry) pz += 1 carry >>= SHIFT if carry: - z.setdigit(pz, z.widedigit(pz) + carry) + z.setdigit(pz, z.uwidedigit(pz) + carry) z._normalize() return z @@ -1767,7 +1781,7 @@ # We use "or" her to avoid having a check where list can be empty in _normalize. lo = rbigint(n._digits[:size_lo] or NULLDIGITS, 1) - hi = rbigint(n._digits[size_lo:n.size] or NULLDIGITS, 1) + hi = rbigint(n._digits[size_lo:size_n] or NULLDIGITS, 1) lo._normalize() hi._normalize() return hi, lo @@ -1980,7 +1994,7 @@ size = pin.numdigits() size -= 1 while size >= 0: - rem = (rem << SHIFT) | pin.widedigit(size) + rem = (rem << SHIFT) | pin.digit(size) hi = rem // n pout.setdigit(size, hi) rem -= hi * n @@ -2057,14 +2071,15 @@ def _muladd1(a, n, extra=0): """Multiply by a single digit and add a single digit, ignoring the sign. """ - + assert n > 0 + size_a = a.numdigits() z = rbigint([NULLDIGIT] * (size_a+1), 1) assert extra & MASK == extra - carry = _widen_digit(extra) + carry = _unsigned_widen_digit(extra) i = 0 while i < size_a: - carry += a.widedigit(i) * n + carry += a.uwidedigit(i) * n z.setdigit(i, carry) carry >>= SHIFT i += 1 @@ -2081,7 +2096,7 @@ assert 0 <= d and d < SHIFT i = 0 while i < m: - acc = a.widedigit(i) << d | carry + acc = a.uwidedigit(i) << d | carry z.setdigit(i, acc) carry = acc >> SHIFT i += 1 @@ -2093,14 +2108,14 @@ * result in z[0:m], and return the d bits shifted out of the bottom. """ - carry = _widen_digit(0) - acc = _widen_digit(0) + carry = _unsigned_widen_digit(0) + acc = _unsigned_widen_digit(0) mask = (1 << d) - 1 assert 0 <= d and d < SHIFT i = m-1 while i >= 0: - acc = (carry << SHIFT) | a.widedigit(i) + acc = (carry << SHIFT) | a.uwidedigit(i) carry = acc & mask z.setdigit(i, acc >> d) i -= 1 diff --git a/rpython/rtyper/lltypesystem/ll2ctypes.py b/rpython/rtyper/lltypesystem/ll2ctypes.py --- a/rpython/rtyper/lltypesystem/ll2ctypes.py +++ b/rpython/rtyper/lltypesystem/ll2ctypes.py @@ -175,7 +175,16 @@ if res >= (1 << 127): res -= 1 << 128 return res + class c_uint128(ctypes.Array): # based on 2 ulongs + _type_ = ctypes.c_uint64 + _length_ = 2 + @property + def value(self): + res = self[0] | (self[1] << 64) + return res + _ctypes_cache[rffi.__INT128_T] = c_int128 + _ctypes_cache[rffi.__UINT128_T] = c_uint128 # for unicode strings, do not use ctypes.c_wchar because ctypes # automatically converts arrays into unicode strings. diff --git a/rpython/rtyper/lltypesystem/lloperation.py b/rpython/rtyper/lltypesystem/lloperation.py --- a/rpython/rtyper/lltypesystem/lloperation.py +++ b/rpython/rtyper/lltypesystem/lloperation.py @@ -320,6 +320,28 @@ 'lllong_rshift': LLOp(canfold=True), # args (r_longlonglong, int) 'lllong_xor': LLOp(canfold=True), + 'ulllong_is_true': LLOp(canfold=True), + 'ulllong_neg': LLOp(canfold=True), + 'ulllong_abs': LLOp(canfold=True), + 'ulllong_invert': LLOp(canfold=True), + + 'ulllong_add': LLOp(canfold=True), + 'ulllong_sub': LLOp(canfold=True), + 'ulllong_mul': LLOp(canfold=True), + 'ulllong_floordiv': LLOp(canfold=True), + 'ulllong_mod': LLOp(canfold=True), + 'ulllong_lt': LLOp(canfold=True), + 'ulllong_le': LLOp(canfold=True), + 'ulllong_eq': LLOp(canfold=True), + 'ulllong_ne': LLOp(canfold=True), + 'ulllong_gt': LLOp(canfold=True), + 'ulllong_ge': LLOp(canfold=True), + 'ulllong_and': LLOp(canfold=True), + 'ulllong_or': LLOp(canfold=True), + 'ulllong_lshift': LLOp(canfold=True), # args (r_ulonglonglong, int) + 'ulllong_rshift': LLOp(canfold=True), # args (r_ulonglonglong, int) + 'ulllong_xor': LLOp(canfold=True), + 'cast_primitive': LLOp(canfold=True), 'cast_bool_to_int': LLOp(canfold=True), 'cast_bool_to_uint': LLOp(canfold=True), diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py --- a/rpython/rtyper/lltypesystem/lltype.py +++ b/rpython/rtyper/lltypesystem/lltype.py @@ -8,7 +8,7 @@ from rpython.rlib.rarithmetic import ( base_int, intmask, is_emulated_long, is_valid_int, longlonglongmask, longlongmask, maxint, normalizedinttype, r_int, r_longfloat, r_longlong, - r_longlonglong, r_singlefloat, r_uint, r_ulonglong) + r_longlonglong, r_singlefloat, r_uint, r_ulonglong, r_ulonglonglong) from rpython.rtyper.extregistry import ExtRegistryEntry from rpython.tool import leakfinder from rpython.tool.identity_dict import identity_dict @@ -676,6 +676,7 @@ _numbertypes[r_int] = _numbertypes[int] _numbertypes[r_longlonglong] = Number("SignedLongLongLong", r_longlonglong, longlonglongmask) + if r_longlong is not r_int: _numbertypes[r_longlong] = Number("SignedLongLong", r_longlong, longlongmask) @@ -700,6 +701,7 @@ SignedLongLong = build_number("SignedLongLong", r_longlong) SignedLongLongLong = build_number("SignedLongLongLong", r_longlonglong) UnsignedLongLong = build_number("UnsignedLongLong", r_ulonglong) +UnsignedLongLongLong = build_number("UnsignedLongLongLong", r_ulonglonglong) Float = Primitive("Float", 0.0) # C type 'double' SingleFloat = Primitive("SingleFloat", r_singlefloat(0.0)) # 'float' diff --git a/rpython/rtyper/lltypesystem/opimpl.py b/rpython/rtyper/lltypesystem/opimpl.py --- a/rpython/rtyper/lltypesystem/opimpl.py +++ b/rpython/rtyper/lltypesystem/opimpl.py @@ -17,7 +17,7 @@ # global synonyms for some types from rpython.rlib.rarithmetic import intmask -from rpython.rlib.rarithmetic import r_int, r_uint, r_longlong, r_ulonglong, r_longlonglong +from rpython.rlib.rarithmetic import r_int, r_uint, r_longlong, r_ulonglong, r_longlonglong, r_ulonglonglong from rpython.rtyper.lltypesystem.llmemory import AddressAsInt if r_longlong is r_int: @@ -38,6 +38,7 @@ 'llong': r_longlong_arg, 'ullong': r_ulonglong, 'lllong': r_longlonglong, + 'ulllong': r_ulonglonglong, } def no_op(x): @@ -366,6 +367,16 @@ assert is_valid_int(y) return r_ulonglong(x >> y) +def op_ulllong_lshift(x, y): + assert isinstance(x, r_ulonglonglong) + assert isinstance(y, int) + return r_ulonglonglong(x << y) + +def op_ulllong_rshift(x, y): + assert isinstance(x, r_ulonglonglong) + assert is_valid_int(y) + return r_ulonglonglong(x >> y) + def op_same_as(x): return x diff --git a/rpython/rtyper/lltypesystem/rffi.py b/rpython/rtyper/lltypesystem/rffi.py --- a/rpython/rtyper/lltypesystem/rffi.py +++ b/rpython/rtyper/lltypesystem/rffi.py @@ -506,7 +506,7 @@ # This is a bit of a hack since we can't use rffi_platform here. try: sizeof_c_type('__int128_t', ignore_errors=True) - TYPES += ['__int128_t'] + TYPES += ['__int128_t', '__uint128_t'] except CompilationError: pass @@ -538,6 +538,8 @@ if name.startswith('unsigned'): name = 'u' + name[9:] signed = False + elif name.startswith('__u'): + signed = False elif name == 'size_t' or name.startswith('uint'): signed = False else: diff --git a/rpython/rtyper/rint.py b/rpython/rtyper/rint.py --- a/rpython/rtyper/rint.py +++ b/rpython/rtyper/rint.py @@ -4,11 +4,11 @@ from rpython.flowspace.operation import op_appendices from rpython.rlib import objectmodel, jit from rpython.rlib.rarithmetic import intmask, longlongmask, r_int, r_longlong -from rpython.rlib.rarithmetic import r_uint, r_ulonglong, r_longlonglong +from rpython.rlib.rarithmetic import r_uint, r_ulonglong, r_longlonglong, r_ulonglonglong from rpython.rtyper.error import TyperError from rpython.rtyper.lltypesystem.lltype import (Signed, Unsigned, Bool, Float, Char, UniChar, UnsignedLongLong, SignedLongLong, build_number, Number, - cast_primitive, typeOf, SignedLongLongLong) + cast_primitive, typeOf, SignedLongLongLong, UnsignedLongLongLong) from rpython.rtyper.rfloat import FloatRepr from rpython.rtyper.rmodel import inputconst, log from rpython.tool.pairtype import pairtype @@ -188,6 +188,7 @@ signedlonglonglong_repr = getintegerrepr(SignedLongLongLong, 'lllong_') unsigned_repr = getintegerrepr(Unsigned, 'uint_') unsignedlonglong_repr = getintegerrepr(UnsignedLongLong, 'ullong_') +signedlonglonglong_repr = getintegerrepr(UnsignedLongLongLong, 'ulllong_') class __extend__(pairtype(IntegerRepr, IntegerRepr)): @@ -473,7 +474,15 @@ raise ZeroDivisionError("longlonglong division") return ll_lllong_py_div(x, y) + at jit.dont_look_inside +def ll_ulllong_py_div(x, y): + return llop.ullong_floordiv(UnsignedLongLongLong, x, y) +def ll_ulllong_py_div_zer(x, y): + if y == 0: + raise ZeroDivisionError("unsigned longlonglong division") + return ll_ulllong_py_div(x, y) + # ---------- mod ---------- @jit.oopspec("int.py_mod(x, y)") @@ -553,6 +562,15 @@ if y == 0: raise ZeroDivisionError return ll_lllong_py_mod(x, y) + + at jit.dont_look_inside +def ll_ulllong_py_mod(x, y): + return llop.ullong_mod(UnsignedLongLongLong, x, y) + +def ll_ulllong_py_mod_zer(x, y): + if y == 0: + raise ZeroDivisionError + return ll_ulllong_py_mod(x, y) # ---------- lshift, neg, abs ---------- diff --git a/rpython/translator/c/primitive.py b/rpython/translator/c/primitive.py --- a/rpython/translator/c/primitive.py +++ b/rpython/translator/c/primitive.py @@ -263,3 +263,4 @@ define_c_primitive(rffi.ULONGLONG, 'unsigned long long', 'ULL') if SUPPORT_INT128: define_c_primitive(rffi.__INT128_T, '__int128_t', 'LL') # Unless it's a 128bit platform, LL is the biggest + define_c_primitive(rffi.__UINT128_T, '__uint128_t', 'ULL') diff --git a/rpython/translator/c/src/int.h b/rpython/translator/c/src/int.h --- a/rpython/translator/c/src/int.h +++ b/rpython/translator/c/src/int.h @@ -113,6 +113,7 @@ #define OP_ULLONG_RSHIFT(x,y,r) CHECK_SHIFT_RANGE(y, PYPY_LONGLONG_BIT); \ r = (x) >> (y) #define OP_LLLONG_RSHIFT(x,y,r) CHECK_SHIFT_RANGE(y, 128); r = (x) >> (y) +#define OP_ULLLONG_RSHIFT(x,y,r) CHECK_SHIFT_RANGE(y, 128); r = (x) >> (y) /* left-shift of a signed value: C99 makes the result undefined if the value is negative. Force the left-shift to occur on unsigned instead. */ @@ -127,6 +128,8 @@ r = (x) << (y) #define OP_ULLONG_LSHIFT(x,y,r) CHECK_SHIFT_RANGE(y, PYPY_LONGLONG_BIT); \ r = (x) << (y) +#define OP_ULLLONG_LSHIFT(x,y,r) CHECK_SHIFT_RANGE(y, 128); \ + r = (x) << (y) /* floor division */ @@ -135,6 +138,7 @@ #define OP_LLONG_FLOORDIV(x,y,r) r = (x) / (y) #define OP_ULLONG_FLOORDIV(x,y,r) r = (x) / (y) #define OP_LLLONG_FLOORDIV(x,y,r) r = (x) / (y) +#define OP_ULLLONG_FLOORDIV(x,y,r) r = (x) / (y) /* modulus */ @@ -143,6 +147,7 @@ #define OP_LLONG_MOD(x,y,r) r = (x) % (y) #define OP_ULLONG_MOD(x,y,r) r = (x) % (y) #define OP_LLLONG_MOD(x,y,r) r = (x) % (y) +#define OP_ULLLONG_MOD(x,y,r) r = (x) % (y) /* bit operations */ @@ -253,3 +258,19 @@ #define OP_ULLONG_AND OP_LLONG_AND #define OP_ULLONG_OR OP_LLONG_OR #define OP_ULLONG_XOR OP_LLONG_XOR + +#define OP_ULLLONG_IS_TRUE OP_LLONG_IS_TRUE +#define OP_ULLLONG_INVERT OP_LLONG_INVERT +#define OP_ULLLONG_ADD OP_LLONG_ADD +#define OP_ULLLONG_SUB OP_LLONG_SUB +#define OP_ULLLONG_MUL OP_LLONG_MUL +#define OP_ULLLONG_LT OP_LLONG_LT +#define OP_ULLLONG_LE OP_LLONG_LE +#define OP_ULLLONG_EQ OP_LLONG_EQ +#define OP_ULLLONG_NE OP_LLONG_NE +#define OP_ULLLONG_GT OP_LLONG_GT +#define OP_ULLLONG_GE OP_LLONG_GE +#define OP_ULLLONG_AND OP_LLONG_AND +#define OP_ULLLONG_OR OP_LLONG_OR +#define OP_ULLLONG_XOR OP_LLONG_XOR + From pypy.commits at gmail.com Sat Oct 21 08:45:13 2017 From: pypy.commits at gmail.com (amauryfa) Date: Sat, 21 Oct 2017 05:45:13 -0700 (PDT) Subject: [pypy-commit] pypy default: Issue2684: pyexpat.CreateParser() did not handle correctly namespace_separator="" Message-ID: <59eb4159.87b41c0a.96dc6.6aa1@mx.google.com> Author: Amaury Forgeot d'Arc Branch: Changeset: r92811:df15dab6a182 Date: 2017-10-20 21:36 +0200 http://bitbucket.org/pypy/pypy/changeset/df15dab6a182/ Log: Issue2684: pyexpat.CreateParser() did not handle correctly namespace_separator="" diff --git a/pypy/module/pyexpat/interp_pyexpat.py b/pypy/module/pyexpat/interp_pyexpat.py --- a/pypy/module/pyexpat/interp_pyexpat.py +++ b/pypy/module/pyexpat/interp_pyexpat.py @@ -808,7 +808,7 @@ w_encoding) if space.is_none(w_namespace_separator): - namespace_separator = 0 + namespace_separator = -1 elif space.isinstance_w(w_namespace_separator, space.w_text): separator = space.text_w(w_namespace_separator) if len(separator) == 0: @@ -831,7 +831,7 @@ elif space.is_w(w_intern, space.w_None): w_intern = None - if namespace_separator: + if namespace_separator >= 0: xmlparser = XML_ParserCreateNS( encoding, rffi.cast(rffi.CHAR, namespace_separator)) diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py --- a/pypy/module/pyexpat/test/test_parser.py +++ b/pypy/module/pyexpat/test/test_parser.py @@ -59,7 +59,7 @@ p.CharacterDataHandler = lambda s: data.append(s) encoding = encoding_arg is None and 'utf-8' or encoding_arg - res = p.Parse(u"\u00f6".encode(encoding), isfinal=True) + res = p.Parse(u"\u00f6".encode(encoding), True) assert res == 1 assert data == [u"\u00f6"] @@ -188,6 +188,34 @@ p.ParseFile(fake_reader) assert fake_reader.read_count == 4 + def test_entities(self): + import pyexpat + parser = pyexpat.ParserCreate(None, "") + + def startElement(tag, attrs): + assert tag == 'http://www.w3.org/1999/02/22-rdf-syntax-ns#RDF' + assert attrs == { + 'http://www.w3.org/XML/1998/namespacebase': + 'http://www.semanticweb.org/jiba/ontologies/2017/0/test'} + parser.StartElementHandler = startElement + parser.Parse(""" + + + + + + ]> + + + + """, True) + class AppTestPyexpat2: spaceconfig = dict(usemodules=['pyexpat', 'itertools', '_socket', 'time', 'struct', 'binascii']) From pypy.commits at gmail.com Sat Oct 21 08:53:57 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 21 Oct 2017 05:53:57 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Backed out changeset f80cb4368b95 Message-ID: <59eb4365.eda0df0a.1b6b7.0b7a@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92812:142c785bb994 Date: 2017-10-21 13:53 +0100 http://bitbucket.org/pypy/pypy/changeset/142c785bb994/ Log: Backed out changeset f80cb4368b95 diff --git a/pypy/interpreter/app_main.py b/pypy/interpreter/app_main.py --- a/pypy/interpreter/app_main.py +++ b/pypy/interpreter/app_main.py @@ -574,6 +574,10 @@ except ValueError: pass # ignore "2 is not a valid file descriptor" + if we_are_translated(): + import __pypy__ + __pypy__.save_module_content_for_future_reload(sys) + mainmodule = type(sys)('__main__') sys.modules['__main__'] = mainmodule diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py --- a/pypy/interpreter/baseobjspace.py +++ b/pypy/interpreter/baseobjspace.py @@ -515,8 +515,9 @@ # patcher relies on this behaviour. w_mod2 = Module(self, w_name) self.setitem(w_modules, w_name, w_mod2) - w_dict = w_mod.getdict(self) # unlazy - self.call_method(w_mod2.getdict(self), 'update', w_dict) + w_mod.getdict(self) # unlazy w_initialdict + self.call_method(w_mod2.getdict(self), 'update', + w_mod.w_initialdict) return w_mod2 self.setitem(w_modules, w_name, w_mod) w_mod.init(self) diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py --- a/pypy/interpreter/mixedmodule.py +++ b/pypy/interpreter/mixedmodule.py @@ -10,7 +10,11 @@ class MixedModule(Module): applevel_name = None - initialized = False + + # The following attribute is None as long as the module has not been + # imported yet, and when it has been, it is mod.__dict__.copy() just + # after startup(). + w_initialdict = None lazy = False submodule_name = None @@ -19,6 +23,7 @@ Module.__init__(self, space, w_name) init_extra_module_attrs(space, self) self.lazy = True + self.lazy_initial_values_w = {} self.__class__.buildloaders() self.loaders = self.loaders.copy() # copy from the class to the inst self.submodules_w = [] @@ -42,14 +47,27 @@ def init(self, space): """This is called each time the module is imported or reloaded """ + if self.w_initialdict is not None: + # the module was already imported. Refresh its content with + # the saved dict, as done with built-in and extension modules + # on CPython. + space.call_method(self.w_dict, 'update', self.w_initialdict) + for w_submodule in self.submodules_w: name = space.text0_w(w_submodule.w_name) space.setitem(self.w_dict, space.newtext(name.split(".")[-1]), w_submodule) space.getbuiltinmodule(name) - if not self.initialized: + if self.w_initialdict is None: Module.init(self, space) - self.initialized = True + if not self.lazy and self.w_initialdict is None: + self.save_module_content_for_future_reload() + + def save_module_content_for_future_reload(self): + # Save the current dictionary in w_initialdict, for future + # reloads. This forces the dictionary if needed. + w_dict = self.getdict(self.space) + self.w_initialdict = self.space.call_method(w_dict, 'copy') @classmethod @not_rpython @@ -78,6 +96,13 @@ return w_value def setdictvalue(self, space, attr, w_value): + if self.lazy and attr not in self.lazy_initial_values_w: + # in lazy mode, the first time an attribute changes, + # we save away the old (initial) value. This allows + # a future getdict() call to build the correct + # self.w_initialdict, containing the initial value. + w_initial_value = self._load_lazily(space, attr) + self.lazy_initial_values_w[attr] = w_initial_value space.setitem_str(self.w_dict, attr, w_value) return True @@ -117,11 +142,23 @@ def _force_lazy_dict_now(self): # Force the dictionary by calling all lazy loaders now. + # This also saves in self.w_initialdict a copy of all the + # initial values, including if they have already been + # modified by setdictvalue(). space = self.space for name in self.loaders: w_value = self.get(name) space.setitem(self.w_dict, space.new_interned_str(name), w_value) self.lazy = False + self.save_module_content_for_future_reload() + for key, w_initial_value in self.lazy_initial_values_w.items(): + w_key = space.new_interned_str(key) + if w_initial_value is not None: + space.setitem(self.w_initialdict, w_key, w_initial_value) + else: + if space.finditem(self.w_initialdict, w_key) is not None: + space.delitem(self.w_initialdict, w_key) + del self.lazy_initial_values_w def _cleanup_(self): self.getdict(self.space) diff --git a/pypy/module/__pypy__/__init__.py b/pypy/module/__pypy__/__init__.py --- a/pypy/module/__pypy__/__init__.py +++ b/pypy/module/__pypy__/__init__.py @@ -92,6 +92,8 @@ 'set_debug' : 'interp_magic.set_debug', 'locals_to_fast' : 'interp_magic.locals_to_fast', 'set_code_callback' : 'interp_magic.set_code_callback', + 'save_module_content_for_future_reload': + 'interp_magic.save_module_content_for_future_reload', 'decode_long' : 'interp_magic.decode_long', '_promote' : 'interp_magic._promote', 'normalize_exc' : 'interp_magic.normalize_exc', @@ -129,7 +131,7 @@ raise else: pass # ok fine to ignore in this case - + if self.space.config.translation.jit: features = detect_cpu.getcpufeatures(model) self.extra_interpdef('jit_backend_features', diff --git a/pypy/module/__pypy__/interp_magic.py b/pypy/module/__pypy__/interp_magic.py --- a/pypy/module/__pypy__/interp_magic.py +++ b/pypy/module/__pypy__/interp_magic.py @@ -145,6 +145,10 @@ assert isinstance(w_frame, PyFrame) w_frame.locals2fast() + at unwrap_spec(w_module=MixedModule) +def save_module_content_for_future_reload(space, w_module): + w_module.save_module_content_for_future_reload() + def set_code_callback(space, w_callable): cache = space.fromcache(CodeHookCache) if space.is_none(w_callable): diff --git a/pypy/module/__pypy__/test/test_magic.py b/pypy/module/__pypy__/test/test_magic.py --- a/pypy/module/__pypy__/test/test_magic.py +++ b/pypy/module/__pypy__/test/test_magic.py @@ -2,6 +2,18 @@ class AppTestMagic: spaceconfig = dict(usemodules=['__pypy__']) + def test_save_module_content_for_future_reload(self): + import sys, __pypy__, imp + d = sys.dont_write_bytecode + sys.dont_write_bytecode = "hello world" + __pypy__.save_module_content_for_future_reload(sys) + sys.dont_write_bytecode = d + imp.reload(sys) + assert sys.dont_write_bytecode == "hello world" + # + sys.dont_write_bytecode = d + __pypy__.save_module_content_for_future_reload(sys) + def test_new_code_hook(self): # workaround for running on top of old CPython 2.7 versions def exec_(code, d): From pypy.commits at gmail.com Sat Oct 21 12:56:54 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 21 Oct 2017 09:56:54 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Fix untranslated test_reimport_builtin(): time.tzname is initialised in a non-standard way that seems to break in untranslated tests Message-ID: <59eb7c56.1cbf1c0a.d5b6b.0be7@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92813:21d9bc2c1120 Date: 2017-10-21 17:56 +0100 http://bitbucket.org/pypy/pypy/changeset/21d9bc2c1120/ Log: Fix untranslated test_reimport_builtin(): time.tzname is initialised in a non-standard way that seems to break in untranslated tests diff --git a/pypy/module/imp/test/test_import.py b/pypy/module/imp/test/test_import.py --- a/pypy/module/imp/test/test_import.py +++ b/pypy/module/imp/test/test_import.py @@ -637,18 +637,18 @@ def test_reimport_builtin(self): import imp, sys, time - oldpath = sys.path - time.tzname = "" + old_sleep = time.sleep + time.sleep = "" del sys.modules['time'] import time as time1 assert sys.modules['time'] is time1 - assert time.tzname == "" + assert time.sleep == "" - imp.reload(time1) # don't leave a broken time.tzname behind + imp.reload(time1) # don't leave a broken time.sleep behind import time - assert time.tzname != "" + assert time.sleep is old_sleep def test_reload_infinite(self): import infinite_reload From pypy.commits at gmail.com Sat Oct 21 13:12:48 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 21 Oct 2017 10:12:48 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Don't mess too much with the sys module Message-ID: <59eb8010.ced71c0a.12757.02a8@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92814:7746dde0b2d4 Date: 2017-10-21 18:12 +0100 http://bitbucket.org/pypy/pypy/changeset/7746dde0b2d4/ Log: Don't mess too much with the sys module diff --git a/pypy/module/imp/test/test_import.py b/pypy/module/imp/test/test_import.py --- a/pypy/module/imp/test/test_import.py +++ b/pypy/module/imp/test/test_import.py @@ -601,25 +601,15 @@ import pkg.a, imp imp.reload(pkg.a) - def test_reload_builtin(self): - import sys, imp - oldpath = sys.path - try: - del sys.settrace - except AttributeError: - pass - - imp.reload(sys) - - assert sys.path is oldpath - assert 'settrace' not in dir(sys) # at least on CPython 3.5.2 - def test_reload_builtin_doesnt_clear(self): import imp import sys sys.foobar = "baz" - imp.reload(sys) - assert sys.foobar == "baz" + try: + imp.reload(sys) + assert sys.foobar == "baz" + finally: + del sys.foobar def test_reimport_builtin_simple_case_1(self): import sys, time From pypy.commits at gmail.com Sat Oct 21 13:36:39 2017 From: pypy.commits at gmail.com (rlamy) Date: Sat, 21 Oct 2017 10:36:39 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Allow __pypy__ module apptests to run translated Message-ID: <59eb85a7.90b8df0a.550c.9f24@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92815:c41fe009ac33 Date: 2017-10-21 18:36 +0100 http://bitbucket.org/pypy/pypy/changeset/c41fe009ac33/ Log: Allow __pypy__ module apptests to run translated diff --git a/pypy/tool/pytest/objspace.py b/pypy/tool/pytest/objspace.py --- a/pypy/tool/pytest/objspace.py +++ b/pypy/tool/pytest/objspace.py @@ -57,9 +57,6 @@ if not ok: py.test.skip("cannot runappdirect test: " "module %r required" % (modname,)) - else: - if '__pypy__' in value: - py.test.skip("no module __pypy__ on top of CPython") continue if info is None: py.test.skip("cannot runappdirect this test on top of CPython") From pypy.commits at gmail.com Sat Oct 21 19:59:16 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:16 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: XXX: this is just a test to see how much performance we loose if we go through the slow __args__ dance instead of using the fastcalls Message-ID: <59ebdf54.521a1c0a.8aadd.b13e@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92816:8823f6116665 Date: 2017-10-21 02:28 +0200 http://bitbucket.org/pypy/pypy/changeset/8823f6116665/ Log: XXX: this is just a test to see how much performance we loose if we go through the slow __args__ dance instead of using the fastcalls diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -103,7 +103,10 @@ # METH_NOARGS # CCC I think we can condense descr_call and call into one, can't we? - def descr_call(self, space): + def descr_call(self, space, __args__): + if len(__args__.arguments_w) != 0: + raise oefmt(space.w_TypeError, + "%s() takes no arguments", self.name) return self.call(space, None, None, None) def call(self, space, w_self, w_args, w_kw): @@ -117,7 +120,11 @@ # METH_O # CCC I think we can condense descr_call and call into one, can't we? - def descr_call(self, space, w_o): + def descr_call(self, space, __args__): + if len(__args__.arguments_w) != 1: + raise oefmt(space.w_TypeError, + "%s() takes exactly one argument", self.name) + w_o = __args__.arguments_w[0] return self.call(space, None, w_o, None) def call(self, space, w_self, w_o, w_kw): From pypy.commits at gmail.com Sat Oct 21 19:59:18 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:18 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: WIP: try a different approach for descr_call: instead of creating a different Message-ID: <59ebdf56.61acdf0a.367b6.e8cc@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92817:26eed5075896 Date: 2017-10-21 15:22 +0200 http://bitbucket.org/pypy/pypy/changeset/26eed5075896/ Log: WIP: try a different approach for descr_call: instead of creating a different class/typedef for each calling convention, use a very generic descr_call and manipulate the __args__ manually. The test done by 8823f6116665 shows that the JIT generates the very same code. By doing this, we probably loose some performance without JIT, because we don't go through fastcall_{0,1} for the NOARGS and O cases. However, the advantage is that this approach should work out of the box also for W_PyCMethodObject. diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -46,31 +46,27 @@ _dealloc(space, py_obj) class W_PyCFunctionObject(W_Root): - # TODO create a slightly different class depending on the c_ml_flags + # CCC: make c_ml_flags an immutable_field, to constant-fold the checks + # inside descr_call? + def __init__(self, space, ml, w_self, w_module=None): self.ml = ml self.name = rffi.charp2str(rffi.cast(rffi.CCHARP,self.ml.c_ml_name)) self.w_self = w_self self.w_module = w_module - def call(self, space, w_self, w_args, w_kw): - # Call the C function - if w_self is None: - w_self = self.w_self + def descr_call(self, space, __args__): flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) - if not flags & METH_KEYWORDS and space.is_true(w_kw): + length = len(__args__.arguments_w) + if not flags & METH_KEYWORDS and __args__.keywords: raise oefmt(space.w_TypeError, "%s() takes no keyword arguments", self.name) - - func = self.ml.c_ml_meth - length = space.int_w(space.len(w_args)) if flags & METH_KEYWORDS: - func = rffi.cast(PyCFunctionKwArgs, self.ml.c_ml_meth) - return generic_cpy_call(space, func, w_self, w_args, w_kw) + return self.call_keywords(space, self.w_self, __args__) elif flags & METH_NOARGS: if length == 0: - return generic_cpy_call(space, func, w_self, None) + return self.call_noargs(space, self.w_self, __args__) raise oefmt(space.w_TypeError, "%s() takes no arguments", self.name) elif flags & METH_O: @@ -78,19 +74,46 @@ raise oefmt(space.w_TypeError, "%s() takes exactly one argument (%d given)", self.name, length) - w_arg = space.getitem(w_args, space.newint(0)) - return generic_cpy_call(space, func, w_self, w_arg) + return self.call_o(space, self.w_self, __args__) elif flags & METH_VARARGS: - return generic_cpy_call(space, func, w_self, w_args) - else: # METH_OLDARGS, the really old style - size = length - if size == 1: - w_arg = space.getitem(w_args, space.newint(0)) - elif size == 0: - w_arg = None - else: - w_arg = w_args - return generic_cpy_call(space, func, w_self, w_arg) + return self.call_varargs(space, self.w_self, __args__) + else: + return self.call_oldargs(space, self.w_self, __args__) + + def call_noargs(self, space, w_self, __args__): + func = self.ml.c_ml_meth + return generic_cpy_call(space, func, w_self, None) + + def call_o(self, space, w_self, __args__): + func = self.ml.c_ml_meth + w_o = __args__.arguments_w[0] + return generic_cpy_call(space, func, w_self, w_o) + + def call_varargs(self, space, w_self, __args__): + state = space.fromcache(State) + func = self.ml.c_ml_meth + py_args = tuple_from_args_w(space, __args__.arguments_w) + try: + return generic_cpy_call(space, func, w_self, py_args) + finally: + decref(space, py_args) + + def call_keywords(self, space, w_self, __args__): + import pdb;pdb.set_trace() + func = rffi.cast(PyCFunctionKwArgs, self.ml.c_ml_meth) + return generic_cpy_call(space, func, w_self, w_args, w_kw) + + def call_oldargs(self, space, w_self, __args__): + func = self.ml.c_ml_meth + length = len(__args__.arguments_w) + if length == 0: + w_args = None + elif length == 1: + w_args = __args__.arguments_w[0] + else: + # CCC: create a py_tuple + w_args = space.newtuple(__args__.arguments_w) + return generic_cpy_call(space, func, w_self, w_args) def get_doc(self, space): doc = self.ml.c_ml_doc @@ -276,19 +299,6 @@ return self.call(space, w_self, w_args, w_kw) @jit.dont_look_inside -def cfunction_descr_call(space, w_self, __args__): - # specialize depending on the W_PyCFunctionObject - self = space.interp_w(W_PyCFunctionObject, w_self) - args_w, kw_w = __args__.unpack() - # XXX __args__.unpack is slow - w_args = space.newtuple(args_w) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, None, w_args, w_kw) - return ret - - at jit.dont_look_inside def cclassmethod_descr_call(space, w_self, __args__): self = space.interp_w(W_PyCFunctionObject, w_self) args_w, kw_w = __args__.unpack() @@ -321,7 +331,7 @@ W_PyCFunctionObject.typedef = TypeDef( 'builtin_function_or_method', - __call__ = interp2app(cfunction_descr_call), + __call__ = interp2app(W_PyCFunctionObject.descr_call), __doc__ = GetSetProperty(W_PyCFunctionObject.get_doc), __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject), __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject, diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py --- a/pypy/module/cpyext/modsupport.py +++ b/pypy/module/cpyext/modsupport.py @@ -83,13 +83,13 @@ return w_mod # borrowed result kept alive in PyImport_AddModule() def _create_pyc_function_object(space, method, w_self, w_name, flags): - flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) - if flags == METH_NOARGS: - return W_PyCFunctionObject_NOARGS(space, method, w_self, w_name) - if flags == METH_O: - return W_PyCFunctionObject_O(space, method, w_self, w_name) - if flags == METH_VARARGS: - return W_PyCFunctionObject_VARARGS(space, method, w_self, w_name) + ## flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) + ## if flags == METH_NOARGS: + ## return W_PyCFunctionObject_NOARGS(space, method, w_self, w_name) + ## if flags == METH_O: + ## return W_PyCFunctionObject_O(space, method, w_self, w_name) + ## if flags == METH_VARARGS: + ## return W_PyCFunctionObject_VARARGS(space, method, w_self, w_name) return W_PyCFunctionObject(space, method, w_self, w_name) def convert_method_defs(space, dict_w, methods, w_type, w_self=None, name=None): From pypy.commits at gmail.com Sat Oct 21 19:59:22 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:22 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: implement call_KEYWORDS and fix the test Message-ID: <59ebdf5a.89ce1c0a.7cd31.8d76@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92819:1c00ece5b913 Date: 2017-10-21 15:55 +0200 http://bitbucket.org/pypy/pypy/changeset/1c00ece5b913/ Log: implement call_KEYWORDS and fix the test diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -99,9 +99,21 @@ decref(space, py_args) def call_keywords(self, space, w_self, __args__): - import pdb;pdb.set_trace() func = rffi.cast(PyCFunctionKwArgs, self.ml.c_ml_meth) - return generic_cpy_call(space, func, w_self, w_args, w_kw) + py_args = tuple_from_args_w(space, __args__.arguments_w) + w_kwargs = None + if __args__.keywords: + # CCC: we should probably have a @jit.look_inside_iff if the + # keyword count is constant, as we do in Arguments.unpack + w_kwargs = space.newdict() + for i in range(len(__args__.keywords)): + key = __args__.keywords[i] + w_obj = __args__.keywords_w[i] + space.setitem(w_kwargs, space.newtext(key), w_obj) + try: + return generic_cpy_call(space, func, w_self, py_args, w_kwargs) + finally: + decref(space, py_args) def call_oldargs(self, space, w_self, __args__): func = self.ml.c_ml_meth From pypy.commits at gmail.com Sat Oct 21 19:59:20 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:20 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: add a test for METH_KEYWORDS; the test passes with -A, but is currently broken in pypy Message-ID: <59ebdf58.8faedf0a.7e508.6bc3@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92818:094b65ff1d96 Date: 2017-10-21 15:46 +0200 http://bitbucket.org/pypy/pypy/changeset/094b65ff1d96/ Log: add a test for METH_KEYWORDS; the test passes with -A, but is currently broken in pypy diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -92,6 +92,21 @@ # raises(TypeError, mod.getarg_VARARGS, k=1) + def test_call_METH_KEYWORDS(self): + mod = self.import_extension('MyModule', [ + ('getarg_KW', 'METH_VARARGS | METH_KEYWORDS', + ''' + if (!kwargs) kwargs = Py_None; + return Py_BuildValue("OO", args, kwargs); + ''' + ), + ]) + assert mod.getarg_KW(1) == ((1,), None) + assert mod.getarg_KW(1, 2) == ((1, 2), None) + assert mod.getarg_KW(a=3, b=4) == ((), {'a': 3, 'b': 4}) + assert mod.getarg_KW(1, 2, a=3, b=4) == ((1, 2), {'a': 3, 'b': 4}) + assert mod.getarg_KW.__name__ == "getarg_KW" + def test_func_attributes(self): mod = self.import_extension('MyModule', [ ('isCFunction', 'METH_O', diff --git a/pypy/tool/cpyext/extbuild.py b/pypy/tool/cpyext/extbuild.py --- a/pypy/tool/cpyext/extbuild.py +++ b/pypy/tool/cpyext/extbuild.py @@ -114,14 +114,18 @@ codes = [] for funcname, flags, code in functions: cfuncname = "%s_%s" % (modname, funcname) + if 'METH_KEYWORDS' in flags: + signature = '(PyObject *self, PyObject *args, PyObject *kwargs)' + else: + signature = '(PyObject *self, PyObject *args)' methods_table.append( - "{\"%s\", %s, %s}," % (funcname, cfuncname, flags)) + "{\"%s\", (PyCFunction)%s, %s}," % (funcname, cfuncname, flags)) func_code = """ - static PyObject* %s(PyObject* self, PyObject* args) - { - %s - } - """ % (cfuncname, code) + static PyObject* {cfuncname}{signature} + {{ + {code} + }} + """.format(cfuncname=cfuncname, signature=signature, code=code) codes.append(func_code) body = "\n".join(codes) + """ From pypy.commits at gmail.com Sat Oct 21 19:59:26 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:26 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: kill the now-obsolete W_PyCFunctionObject_{NOARGS, O, VARARGS} Message-ID: <59ebdf5e.61acdf0a.367b6.e8d7@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92821:6a0f6b5d4571 Date: 2017-10-22 00:01 +0200 http://bitbucket.org/pypy/pypy/changeset/6a0f6b5d4571/ Log: kill the now-obsolete W_PyCFunctionObject_{NOARGS,O,VARARGS} diff --git a/pypy/module/cpyext/__init__.py b/pypy/module/cpyext/__init__.py --- a/pypy/module/cpyext/__init__.py +++ b/pypy/module/cpyext/__init__.py @@ -17,16 +17,11 @@ method = pypy.module.cpyext.typeobject.get_new_method_def(space) # the w_self argument here is a dummy, the only thing done with w_obj # is call type() on it - objs_w = [cls(space, method, space.w_None) for cls in [ - pypy.module.cpyext.methodobject.W_PyCFunctionObject, - pypy.module.cpyext.methodobject.W_PyCFunctionObject_NOARGS, - pypy.module.cpyext.methodobject.W_PyCFunctionObject_O, - pypy.module.cpyext.methodobject.W_PyCFunctionObject_VARARGS]] - w_objs = space.newtuple(objs_w) - space.appexec([w_objs], """(methods): + w_obj = pypy.module.cpyext.methodobject.W_PyCFunctionObject(space, + method, space.w_None) + space.appexec([w_obj], """(meth): from pickle import Pickler - for meth in methods: - Pickler.dispatch[type(meth)] = Pickler.save_global + Pickler.dispatch[type(meth)] = Pickler.save_global """) def register_atexit(self, function): diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -137,53 +137,6 @@ else: return space.w_None -class W_PyCFunctionObject_NOARGS(W_PyCFunctionObject): - # METH_NOARGS - - # CCC I think we can condense descr_call and call into one, can't we? - def descr_call(self, space, __args__): - if len(__args__.arguments_w) != 0: - raise oefmt(space.w_TypeError, - "%s() takes no arguments", self.name) - return self.call(space, None, None, None) - - def call(self, space, w_self, w_args, w_kw): - # Call the C function - if w_self is None: - w_self = self.w_self - func = self.ml.c_ml_meth - return generic_cpy_call(space, func, w_self, None) - -class W_PyCFunctionObject_O(W_PyCFunctionObject): - # METH_O - - # CCC I think we can condense descr_call and call into one, can't we? - def descr_call(self, space, __args__): - if len(__args__.arguments_w) != 1: - raise oefmt(space.w_TypeError, - "%s() takes exactly one argument", self.name) - w_o = __args__.arguments_w[0] - return self.call(space, None, w_o, None) - - def call(self, space, w_self, w_o, w_kw): - if w_self is None: - w_self = self.w_self - func = self.ml.c_ml_meth - return generic_cpy_call(space, func, w_self, w_o) - -class W_PyCFunctionObject_VARARGS(W_PyCFunctionObject): - # METH_VARARGS - - def descr_call(self, space, args_w): - state = space.fromcache(State) - w_self = self.w_self - func = self.ml.c_ml_meth - py_args = tuple_from_args_w(space, args_w) - try: - return generic_cpy_call(space, func, w_self, py_args) - finally: - decref(space, py_args) - class W_PyCMethodObject(W_PyCFunctionObject): w_self = None def __init__(self, space, ml, w_type): @@ -351,36 +304,6 @@ ) W_PyCFunctionObject.typedef.acceptable_as_base_class = False -W_PyCFunctionObject_NOARGS.typedef = TypeDef( - 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(W_PyCFunctionObject_NOARGS.descr_call), - __doc__ = GetSetProperty(W_PyCFunctionObject_NOARGS.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_NOARGS), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_NOARGS, - wrapfn="newtext_or_none"), - ) -W_PyCFunctionObject_NOARGS.typedef.acceptable_as_base_class = False - -W_PyCFunctionObject_O.typedef = TypeDef( - 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(W_PyCFunctionObject_O.descr_call), - __doc__ = GetSetProperty(W_PyCFunctionObject_O.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_O), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_O, - wrapfn="newtext_or_none"), - ) -W_PyCFunctionObject_O.typedef.acceptable_as_base_class = False - -W_PyCFunctionObject_VARARGS.typedef = TypeDef( - 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(W_PyCFunctionObject_VARARGS.descr_call), - __doc__ = GetSetProperty(W_PyCFunctionObject_VARARGS.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject_VARARGS), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject_VARARGS, - wrapfn="newtext_or_none"), - ) -W_PyCFunctionObject_VARARGS.typedef.acceptable_as_base_class = False - W_PyCMethodObject.typedef = TypeDef( 'method_descriptor', __get__ = interp2app(cmethod_descr_get), diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py --- a/pypy/module/cpyext/modsupport.py +++ b/pypy/module/cpyext/modsupport.py @@ -6,9 +6,7 @@ from pypy.interpreter.module import Module from pypy.module.cpyext.methodobject import ( W_PyCFunctionObject, PyCFunction_NewEx, PyDescr_NewMethod, - PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New, - W_PyCFunctionObject_NOARGS, W_PyCFunctionObject_O, - W_PyCFunctionObject_VARARGS) + PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.module.cpyext.state import State from pypy.interpreter.error import oefmt @@ -82,16 +80,6 @@ space.newtext(rffi.charp2str(doc))) return w_mod # borrowed result kept alive in PyImport_AddModule() -def _create_pyc_function_object(space, method, w_self, w_name, flags): - ## flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) - ## if flags == METH_NOARGS: - ## return W_PyCFunctionObject_NOARGS(space, method, w_self, w_name) - ## if flags == METH_O: - ## return W_PyCFunctionObject_O(space, method, w_self, w_name) - ## if flags == METH_VARARGS: - ## return W_PyCFunctionObject_VARARGS(space, method, w_self, w_name) - return W_PyCFunctionObject(space, method, w_self, w_name) - def convert_method_defs(space, dict_w, methods, w_type, w_self=None, name=None): w_name = space.newtext_or_none(name) methods = rffi.cast(rffi.CArrayPtr(PyMethodDef), methods) @@ -110,8 +98,7 @@ raise oefmt(space.w_ValueError, "module functions cannot set METH_CLASS or " "METH_STATIC") - w_obj = _create_pyc_function_object(space, method, w_self, - w_name, flags) + w_obj = W_PyCFunctionObject(space, method, w_self, w_name) else: if methodname in dict_w and not (flags & METH_COEXIST): continue From pypy.commits at gmail.com Sat Oct 21 19:59:28 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:28 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix W_PyCClassMethodObject.descr_call Message-ID: <59ebdf60.c23a1c0a.a290c.8a65@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92822:ae86725f484f Date: 2017-10-22 00:26 +0200 http://bitbucket.org/pypy/pypy/changeset/ae86725f484f/ Log: fix W_PyCClassMethodObject.descr_call diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -198,6 +198,16 @@ def __repr__(self): return self.space.unwrap(self.descr_method_repr()) + def descr_call(self, space, __args__): + if len(__args__.arguments_w) == 0: + raise oefmt(space.w_TypeError, + "descriptor '%s' of '%s' object needs an argument", + self.name, self.w_objclass.getname(space)) + w_instance = __args__.arguments_w[0] # XXX typecheck missing + # CCC: we can surely do better than this + __args__ = __args__.replace_arguments(__args__.arguments_w[1:]) + return self.call(space, w_instance, __args__) + def descr_method_repr(self): return self.getrepr(self.space, "built-in method '%s' of '%s' object" % @@ -263,22 +273,6 @@ space.setitem(w_kw, space.newtext(key), w_obj) return self.call(space, w_self, w_args, w_kw) - at jit.dont_look_inside -def cclassmethod_descr_call(space, w_self, __args__): - self = space.interp_w(W_PyCFunctionObject, w_self) - args_w, kw_w = __args__.unpack() - if len(args_w) < 1: - raise oefmt(space.w_TypeError, - "descriptor '%s' of '%s' object needs an argument", - self.name, self.w_objclass.getname(space)) - w_instance = args_w[0] # XXX typecheck missing - w_args = space.newtuple(args_w[1:]) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, w_instance, w_args, w_kw) - return ret - def cmethod_descr_get(space, w_function, w_obj, w_cls=None): asking_for_bound = (space.is_none(w_cls) or not space.is_w(w_obj, space.w_None) or @@ -318,7 +312,7 @@ W_PyCClassMethodObject.typedef = TypeDef( 'classmethod', __get__ = interp2app(cclassmethod_descr_get), - __call__ = interp2app(cclassmethod_descr_call), + __call__ = interp2app(W_PyCClassMethodObject.descr_call), __name__ = interp_attrproperty('name', cls=W_PyCClassMethodObject, wrapfn="newtext_or_none"), __objclass__ = interp_attrproperty_w('w_objclass', From pypy.commits at gmail.com Sat Oct 21 19:59:24 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:24 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix W_PyCMethodObject.descr_call, so that it uses the same logic as W_PyCFunctionObject Message-ID: <59ebdf5c.4f091c0a.1bb65.73b2@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92820:071be7b3d869 Date: 2017-10-21 23:54 +0200 http://bitbucket.org/pypy/pypy/changeset/071be7b3d869/ Log: fix W_PyCMethodObject.descr_call, so that it uses the same logic as W_PyCFunctionObject diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -56,6 +56,9 @@ self.w_module = w_module def descr_call(self, space, __args__): + return self.call(space, self.w_self, __args__) + + def call(self, space, w_self, __args__): flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) length = len(__args__.arguments_w) @@ -63,10 +66,10 @@ raise oefmt(space.w_TypeError, "%s() takes no keyword arguments", self.name) if flags & METH_KEYWORDS: - return self.call_keywords(space, self.w_self, __args__) + return self.call_keywords(space, w_self, __args__) elif flags & METH_NOARGS: if length == 0: - return self.call_noargs(space, self.w_self, __args__) + return self.call_noargs(space, w_self, __args__) raise oefmt(space.w_TypeError, "%s() takes no arguments", self.name) elif flags & METH_O: @@ -74,11 +77,11 @@ raise oefmt(space.w_TypeError, "%s() takes exactly one argument (%d given)", self.name, length) - return self.call_o(space, self.w_self, __args__) + return self.call_o(space, w_self, __args__) elif flags & METH_VARARGS: - return self.call_varargs(space, self.w_self, __args__) + return self.call_varargs(space, w_self, __args__) else: - return self.call_oldargs(space, self.w_self, __args__) + return self.call_oldargs(space, w_self, __args__) def call_noargs(self, space, w_self, __args__): func = self.ml.c_ml_meth @@ -199,14 +202,13 @@ self.name, w_objclass.name)) def descr_call(self, space, __args__): - args_w, kw_w = __args__.unpack() - if len(args_w) < 1: + if len(__args__.arguments_w) == 0: w_objclass = self.w_objclass assert isinstance(w_objclass, W_TypeObject) raise oefmt(space.w_TypeError, "descriptor '%s' of '%s' object needs an argument", self.name, w_objclass.name) - w_instance = args_w[0] + w_instance = __args__.arguments_w[0] # XXX: needs a stricter test if not space.isinstance_w(w_instance, self.w_objclass): w_objclass = self.w_objclass @@ -214,12 +216,10 @@ raise oefmt(space.w_TypeError, "descriptor '%s' requires a '%s' object but received a '%T'", self.name, w_objclass.name, w_instance) - w_args = space.newtuple(args_w[1:]) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, w_instance, w_args, w_kw) - return ret + # + # CCC: we can surely do better than this + __args__ = __args__.replace_arguments(__args__.arguments_w[1:]) + return self.call(space, w_instance, __args__) # PyPy addition, for Cython _, _ = build_type_checkers("MethodDescr", W_PyCMethodObject) From pypy.commits at gmail.com Sat Oct 21 19:59:31 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 21 Oct 2017 16:59:31 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: add this fix, because now pass w_kwds=None directly; we should investigate better whether the check is still needed, but first I want to test the heavy changes which I did in the previous checkins Message-ID: <59ebdf63.28361c0a.2fc0c.7e7d@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92823:7501b268c626 Date: 2017-10-22 00:43 +0200 http://bitbucket.org/pypy/pypy/changeset/7501b268c626/ Log: add this fix, because now pass w_kwds=None directly; we should investigate better whether the check is still needed, but first I want to test the heavy changes which I did in the previous checkins diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -378,7 +378,10 @@ args_w = space.fixedview(w_args) w_subtype = args_w[0] w_args = space.newtuple(args_w[1:]) - if not space.is_true(w_kwds): + # CCC: is this check still needed? I think we can safely remove it now + # that we manually handle __args__ inside W_PyCFunctionObject.descr_call, + # but we need to double check + if w_kwds and not space.is_true(w_kwds): w_kwds = None subtype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_subtype)) From pypy.commits at gmail.com Sun Oct 22 05:49:51 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 02:49:51 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: try to add flags as an immutable fields, to constant-fold away the checks inside call() Message-ID: <59ec69bf.26afdf0a.5224b.defd@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92824:a41f31f17cfe Date: 2017-10-22 11:48 +0200 http://bitbucket.org/pypy/pypy/changeset/a41f31f17cfe/ Log: try to add flags as an immutable fields, to constant-fold away the checks inside call() diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -46,12 +46,12 @@ _dealloc(space, py_obj) class W_PyCFunctionObject(W_Root): - # CCC: make c_ml_flags an immutable_field, to constant-fold the checks - # inside descr_call? - + _immutable_fields_ = ["flags"] + def __init__(self, space, ml, w_self, w_module=None): self.ml = ml self.name = rffi.charp2str(rffi.cast(rffi.CCHARP,self.ml.c_ml_name)) + self.flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) self.w_self = w_self self.w_module = w_module @@ -59,8 +59,7 @@ return self.call(space, self.w_self, __args__) def call(self, space, w_self, __args__): - flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) - flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) + flags = self.flags & ~(METH_CLASS | METH_STATIC | METH_COEXIST) length = len(__args__.arguments_w) if not flags & METH_KEYWORDS and __args__.keywords: raise oefmt(space.w_TypeError, From pypy.commits at gmail.com Sun Oct 22 11:39:52 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 08:39:52 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: fix this CCC and pass a py_tuple also to METH_OLDARGS calls Message-ID: <59ecbbc8.06b6df0a.6f437.ba5b@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92825:42834b4486db Date: 2017-10-22 12:05 +0200 http://bitbucket.org/pypy/pypy/changeset/42834b4486db/ Log: fix this CCC and pass a py_tuple also to METH_OLDARGS calls diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -121,13 +121,15 @@ func = self.ml.c_ml_meth length = len(__args__.arguments_w) if length == 0: - w_args = None + py_args = lltype.nullptr(PyObject.TO) elif length == 1: - w_args = __args__.arguments_w[0] + py_args = make_ref(space, __args__.arguments_w[0]) else: - # CCC: create a py_tuple - w_args = space.newtuple(__args__.arguments_w) - return generic_cpy_call(space, func, w_self, w_args) + py_args = tuple_from_args_w(space, __args__.arguments_w) + try: + return generic_cpy_call(space, func, w_self, py_args) + finally: + decref(space, py_args) def get_doc(self, space): doc = self.ml.c_ml_doc diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -42,25 +42,6 @@ raises(TypeError, mod.getarg_O) raises(TypeError, mod.getarg_O, 1, 1) - def test_call_METH_OLDARGS(self): - mod = self.import_extension('MyModule', [ - ('getarg_OLD', 'METH_OLDARGS', - ''' - if(args) { - Py_INCREF(args); - return args; - } - else { - Py_INCREF(Py_None); - return Py_None; - } - ''' - ), - ]) - assert mod.getarg_OLD(1) == 1 - assert mod.getarg_OLD() is None - assert mod.getarg_OLD(1, 2) == (1, 2) - def test_call_METH_VARARGS(self): mod = self.import_extension('MyModule', [ ('getarg_VARARGS', 'METH_VARARGS', @@ -92,6 +73,27 @@ # raises(TypeError, mod.getarg_VARARGS, k=1) + def test_call_METH_OLDARGS(self): + mod = self.import_extension('MyModule', [ + ('getarg_OLD', 'METH_OLDARGS', + ''' + if(args) { + return Py_BuildValue("Ol", args, args->ob_refcnt); + } + else { + Py_INCREF(Py_None); + return Py_None; + } + ''' + ), + ]) + assert mod.getarg_OLD() is None + val, refcnt = mod.getarg_OLD(1) + assert val == 1 + val, refcnt = mod.getarg_OLD(1, 2) + assert val == (1, 2) + assert refcnt == 1 # see the comments in the test above + def test_call_METH_KEYWORDS(self): mod = self.import_extension('MyModule', [ ('getarg_KW', 'METH_VARARGS | METH_KEYWORDS', From pypy.commits at gmail.com Sun Oct 22 11:39:56 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 08:39:56 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge the branch cpyext-refactor-methodobject: now W_PyCMethodObject and W_PyCFunctionObject use the same code to dispatch calls, with the effect that the latter is as fast as the former; moreover, METH_VARARGS uses a special, faster logic Message-ID: <59ecbbcc.46c3df0a.f9b72.8fc2@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92827:770b53602445 Date: 2017-10-22 17:39 +0200 http://bitbucket.org/pypy/pypy/changeset/770b53602445/ Log: merge the branch cpyext-refactor-methodobject: now W_PyCMethodObject and W_PyCFunctionObject use the same code to dispatch calls, with the effect that the latter is as fast as the former; moreover, METH_VARARGS uses a special, faster logic diff --git a/pypy/module/cpyext/__init__.py b/pypy/module/cpyext/__init__.py --- a/pypy/module/cpyext/__init__.py +++ b/pypy/module/cpyext/__init__.py @@ -16,11 +16,12 @@ space.fromcache(State).startup(space) method = pypy.module.cpyext.typeobject.get_new_method_def(space) # the w_self argument here is a dummy, the only thing done with w_obj - # is call space.type on it - w_obj = pypy.module.cpyext.methodobject.W_PyCFunctionObject(space, method, space.w_None) - space.appexec([space.type(w_obj)], """(methodtype): + # is call type() on it + w_obj = pypy.module.cpyext.methodobject.W_PyCFunctionObject(space, + method, space.w_None) + space.appexec([w_obj], """(meth): from pickle import Pickler - Pickler.dispatch[methodtype] = Pickler.save_global + Pickler.dispatch[type(meth)] = Pickler.save_global """) def register_atexit(self, function): diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -15,6 +15,8 @@ build_type_checkers) from pypy.module.cpyext.pyobject import ( decref, from_ref, make_ref, as_pyobj, make_typedescr) +from pypy.module.cpyext.state import State +from pypy.module.cpyext.tupleobject import tuple_from_args_w PyMethodDef = cts.gettype('PyMethodDef') PyCFunction = cts.gettype('PyCFunction') @@ -44,31 +46,29 @@ _dealloc(space, py_obj) class W_PyCFunctionObject(W_Root): - # TODO create a slightly different class depending on the c_ml_flags + _immutable_fields_ = ["flags"] + def __init__(self, space, ml, w_self, w_module=None): self.ml = ml self.name = rffi.charp2str(rffi.cast(rffi.CCHARP,self.ml.c_ml_name)) + self.flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) self.w_self = w_self self.w_module = w_module - def call(self, space, w_self, w_args, w_kw): - # Call the C function - if w_self is None: - w_self = self.w_self - flags = rffi.cast(lltype.Signed, self.ml.c_ml_flags) - flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) - if not flags & METH_KEYWORDS and space.is_true(w_kw): + def descr_call(self, space, __args__): + return self.call(space, self.w_self, __args__) + + def call(self, space, w_self, __args__): + flags = self.flags & ~(METH_CLASS | METH_STATIC | METH_COEXIST) + length = len(__args__.arguments_w) + if not flags & METH_KEYWORDS and __args__.keywords: raise oefmt(space.w_TypeError, "%s() takes no keyword arguments", self.name) - - func = self.ml.c_ml_meth - length = space.int_w(space.len(w_args)) if flags & METH_KEYWORDS: - func = rffi.cast(PyCFunctionKwArgs, self.ml.c_ml_meth) - return generic_cpy_call(space, func, w_self, w_args, w_kw) + return self.call_keywords(space, w_self, __args__) elif flags & METH_NOARGS: if length == 0: - return generic_cpy_call(space, func, w_self, None) + return self.call_noargs(space, w_self, __args__) raise oefmt(space.w_TypeError, "%s() takes no arguments", self.name) elif flags & METH_O: @@ -76,19 +76,60 @@ raise oefmt(space.w_TypeError, "%s() takes exactly one argument (%d given)", self.name, length) - w_arg = space.getitem(w_args, space.newint(0)) - return generic_cpy_call(space, func, w_self, w_arg) + return self.call_o(space, w_self, __args__) elif flags & METH_VARARGS: - return generic_cpy_call(space, func, w_self, w_args) - else: # METH_OLDARGS, the really old style - size = length - if size == 1: - w_arg = space.getitem(w_args, space.newint(0)) - elif size == 0: - w_arg = None - else: - w_arg = w_args - return generic_cpy_call(space, func, w_self, w_arg) + return self.call_varargs(space, w_self, __args__) + else: + return self.call_oldargs(space, w_self, __args__) + + def call_noargs(self, space, w_self, __args__): + func = self.ml.c_ml_meth + return generic_cpy_call(space, func, w_self, None) + + def call_o(self, space, w_self, __args__): + func = self.ml.c_ml_meth + w_o = __args__.arguments_w[0] + return generic_cpy_call(space, func, w_self, w_o) + + def call_varargs(self, space, w_self, __args__): + state = space.fromcache(State) + func = self.ml.c_ml_meth + py_args = tuple_from_args_w(space, __args__.arguments_w) + try: + return generic_cpy_call(space, func, w_self, py_args) + finally: + decref(space, py_args) + + def call_keywords(self, space, w_self, __args__): + func = rffi.cast(PyCFunctionKwArgs, self.ml.c_ml_meth) + py_args = tuple_from_args_w(space, __args__.arguments_w) + w_kwargs = None + if __args__.keywords: + # CCC: we should probably have a @jit.look_inside_iff if the + # keyword count is constant, as we do in Arguments.unpack + w_kwargs = space.newdict() + for i in range(len(__args__.keywords)): + key = __args__.keywords[i] + w_obj = __args__.keywords_w[i] + space.setitem(w_kwargs, space.newtext(key), w_obj) + try: + return generic_cpy_call(space, func, w_self, py_args, w_kwargs) + finally: + decref(space, py_args) + + def call_oldargs(self, space, w_self, __args__): + func = self.ml.c_ml_meth + length = len(__args__.arguments_w) + if length == 0: + py_args = lltype.nullptr(PyObject.TO) + elif length == 1: + py_args = make_ref(space, __args__.arguments_w[0]) + else: + py_args = tuple_from_args_w(space, __args__.arguments_w) + try: + return generic_cpy_call(space, func, w_self, py_args) + finally: + decref(space, py_args) def get_doc(self, space): doc = self.ml.c_ml_doc @@ -97,21 +138,6 @@ else: return space.w_None -class W_PyCFunctionObjectNoArgs(W_PyCFunctionObject): - def call(self, space, w_self, w_args, w_kw): - # Call the C function - if w_self is None: - w_self = self.w_self - func = self.ml.c_ml_meth - return generic_cpy_call(space, func, w_self, None) - -class W_PyCFunctionObjectSingleObject(W_PyCFunctionObject): - def call(self, space, w_self, w_o, w_kw): - if w_self is None: - w_self = self.w_self - func = self.ml.c_ml_meth - return generic_cpy_call(space, func, w_self, w_o) - class W_PyCMethodObject(W_PyCFunctionObject): w_self = None def __init__(self, space, ml, w_type): @@ -130,14 +156,13 @@ self.name, w_objclass.name)) def descr_call(self, space, __args__): - args_w, kw_w = __args__.unpack() - if len(args_w) < 1: + if len(__args__.arguments_w) == 0: w_objclass = self.w_objclass assert isinstance(w_objclass, W_TypeObject) raise oefmt(space.w_TypeError, "descriptor '%s' of '%s' object needs an argument", self.name, w_objclass.name) - w_instance = args_w[0] + w_instance = __args__.arguments_w[0] # XXX: needs a stricter test if not space.isinstance_w(w_instance, self.w_objclass): w_objclass = self.w_objclass @@ -145,12 +170,10 @@ raise oefmt(space.w_TypeError, "descriptor '%s' requires a '%s' object but received a '%T'", self.name, w_objclass.name, w_instance) - w_args = space.newtuple(args_w[1:]) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, w_instance, w_args, w_kw) - return ret + # + # CCC: we can surely do better than this + __args__ = __args__.replace_arguments(__args__.arguments_w[1:]) + return self.call(space, w_instance, __args__) # PyPy addition, for Cython _, _ = build_type_checkers("MethodDescr", W_PyCMethodObject) @@ -176,6 +199,16 @@ def __repr__(self): return self.space.unwrap(self.descr_method_repr()) + def descr_call(self, space, __args__): + if len(__args__.arguments_w) == 0: + raise oefmt(space.w_TypeError, + "descriptor '%s' of '%s' object needs an argument", + self.name, self.w_objclass.getname(space)) + w_instance = __args__.arguments_w[0] # XXX typecheck missing + # CCC: we can surely do better than this + __args__ = __args__.replace_arguments(__args__.arguments_w[1:]) + return self.call(space, w_instance, __args__) + def descr_method_repr(self): return self.getrepr(self.space, "built-in method '%s' of '%s' object" % @@ -241,45 +274,6 @@ space.setitem(w_kw, space.newtext(key), w_obj) return self.call(space, w_self, w_args, w_kw) -def cfunction_descr_call_noargs(space, w_self): - # special case for calling with flags METH_NOARGS - self = space.interp_w(W_PyCFunctionObjectNoArgs, w_self) - return self.call(space, None, None, None) - -def cfunction_descr_call_single_object(space, w_self, w_o): - # special case for calling with flags METH_O - self = space.interp_w(W_PyCFunctionObjectSingleObject, w_self) - return self.call(space, None, w_o, None) - - at jit.dont_look_inside -def cfunction_descr_call(space, w_self, __args__): - # specialize depending on the W_PyCFunctionObject - self = space.interp_w(W_PyCFunctionObject, w_self) - args_w, kw_w = __args__.unpack() - # XXX __args__.unpack is slow - w_args = space.newtuple(args_w) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, None, w_args, w_kw) - return ret - - at jit.dont_look_inside -def cclassmethod_descr_call(space, w_self, __args__): - self = space.interp_w(W_PyCFunctionObject, w_self) - args_w, kw_w = __args__.unpack() - if len(args_w) < 1: - raise oefmt(space.w_TypeError, - "descriptor '%s' of '%s' object needs an argument", - self.name, self.w_objclass.getname(space)) - w_instance = args_w[0] # XXX typecheck missing - w_args = space.newtuple(args_w[1:]) - w_kw = space.newdict() - for key, w_obj in kw_w.items(): - space.setitem(w_kw, space.newtext(key), w_obj) - ret = self.call(space, w_instance, w_args, w_kw) - return ret - def cmethod_descr_get(space, w_function, w_obj, w_cls=None): asking_for_bound = (space.is_none(w_cls) or not space.is_w(w_obj, space.w_None) or @@ -297,7 +291,7 @@ W_PyCFunctionObject.typedef = TypeDef( 'builtin_function_or_method', - __call__ = interp2app(cfunction_descr_call), + __call__ = interp2app(W_PyCFunctionObject.descr_call), __doc__ = GetSetProperty(W_PyCFunctionObject.get_doc), __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObject), __name__ = interp_attrproperty('name', cls=W_PyCFunctionObject, @@ -305,26 +299,6 @@ ) W_PyCFunctionObject.typedef.acceptable_as_base_class = False -W_PyCFunctionObjectNoArgs.typedef = TypeDef( - 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(cfunction_descr_call_noargs), - __doc__ = GetSetProperty(W_PyCFunctionObjectNoArgs.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObjectNoArgs), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObjectNoArgs, - wrapfn="newtext_or_none"), - ) -W_PyCFunctionObjectNoArgs.typedef.acceptable_as_base_class = False - -W_PyCFunctionObjectSingleObject.typedef = TypeDef( - 'builtin_function_or_method', W_PyCFunctionObject.typedef, - __call__ = interp2app(cfunction_descr_call_single_object), - __doc__ = GetSetProperty(W_PyCFunctionObjectSingleObject.get_doc), - __module__ = interp_attrproperty_w('w_module', cls=W_PyCFunctionObjectSingleObject), - __name__ = interp_attrproperty('name', cls=W_PyCFunctionObjectSingleObject, - wrapfn="newtext_or_none"), - ) -W_PyCFunctionObjectSingleObject.typedef.acceptable_as_base_class = False - W_PyCMethodObject.typedef = TypeDef( 'method_descriptor', __get__ = interp2app(cmethod_descr_get), @@ -339,7 +313,7 @@ W_PyCClassMethodObject.typedef = TypeDef( 'classmethod', __get__ = interp2app(cclassmethod_descr_get), - __call__ = interp2app(cclassmethod_descr_call), + __call__ = interp2app(W_PyCClassMethodObject.descr_call), __name__ = interp_attrproperty('name', cls=W_PyCClassMethodObject, wrapfn="newtext_or_none"), __objclass__ = interp_attrproperty_w('w_objclass', diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py --- a/pypy/module/cpyext/modsupport.py +++ b/pypy/module/cpyext/modsupport.py @@ -1,13 +1,12 @@ from rpython.rtyper.lltypesystem import rffi, lltype from pypy.module.cpyext.api import cpython_api, cpython_struct, \ METH_STATIC, METH_CLASS, METH_COEXIST, CANNOT_FAIL, CONST_STRING, \ - METH_NOARGS, METH_O + METH_NOARGS, METH_O, METH_VARARGS from pypy.module.cpyext.pyobject import PyObject, as_pyobj from pypy.interpreter.module import Module from pypy.module.cpyext.methodobject import ( W_PyCFunctionObject, PyCFunction_NewEx, PyDescr_NewMethod, - PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New, - W_PyCFunctionObjectNoArgs, W_PyCFunctionObjectSingleObject) + PyMethodDef, PyDescr_NewClassMethod, PyStaticMethod_New) from pypy.module.cpyext.pyerrors import PyErr_BadInternalCall from pypy.module.cpyext.state import State from pypy.interpreter.error import oefmt @@ -81,14 +80,6 @@ space.newtext(rffi.charp2str(doc))) return w_mod # borrowed result kept alive in PyImport_AddModule() -def _create_pyc_function_object(space, method, w_self, w_name, flags): - flags &= ~(METH_CLASS | METH_STATIC | METH_COEXIST) - if flags == METH_NOARGS: - return W_PyCFunctionObjectNoArgs(space, method, w_self, w_name) - if flags == METH_O: - return W_PyCFunctionObjectSingleObject(space, method, w_self, w_name) - return W_PyCFunctionObject(space, method, w_self, w_name) - def convert_method_defs(space, dict_w, methods, w_type, w_self=None, name=None): w_name = space.newtext_or_none(name) methods = rffi.cast(rffi.CArrayPtr(PyMethodDef), methods) @@ -107,8 +98,7 @@ raise oefmt(space.w_ValueError, "module functions cannot set METH_CLASS or " "METH_STATIC") - w_obj = _create_pyc_function_object(space, method, w_self, - w_name, flags) + w_obj = W_PyCFunctionObject(space, method, w_self, w_name) else: if methodname in dict_w and not (flags & METH_COEXIST): continue diff --git a/pypy/module/cpyext/test/test_methodobject.py b/pypy/module/cpyext/test/test_methodobject.py --- a/pypy/module/cpyext/test/test_methodobject.py +++ b/pypy/module/cpyext/test/test_methodobject.py @@ -8,14 +8,9 @@ from rpython.rtyper.lltypesystem import rffi, lltype class AppTestMethodObject(AppTestCpythonExtensionBase): - def test_call_METH(self): + + def test_call_METH_NOARGS(self): mod = self.import_extension('MyModule', [ - ('getarg_O', 'METH_O', - ''' - Py_INCREF(args); - return args; - ''' - ), ('getarg_NO', 'METH_NOARGS', ''' if(args) { @@ -28,11 +23,62 @@ } ''' ), + ]) + assert mod.getarg_NO() is None + raises(TypeError, mod.getarg_NO, 1) + raises(TypeError, mod.getarg_NO, 1, 1) + + def test_call_METH_O(self): + mod = self.import_extension('MyModule', [ + ('getarg_O', 'METH_O', + ''' + Py_INCREF(args); + return args; + ''' + ), + ]) + assert mod.getarg_O(1) == 1 + assert mod.getarg_O.__name__ == "getarg_O" + raises(TypeError, mod.getarg_O) + raises(TypeError, mod.getarg_O, 1, 1) + + def test_call_METH_VARARGS(self): + mod = self.import_extension('MyModule', [ + ('getarg_VARARGS', 'METH_VARARGS', + ''' + return Py_BuildValue("Ol", args, args->ob_refcnt); + ''' + ), + ]) + # check that we pass the expected tuple of arguments AND that the + # recnt is 1. In particular, on PyPy refcnt==1 means that we created + # the PyObject tuple directly, without passing from a w_tuple; as + # such, the tuple will be immediately freed after the call, without + # having to wait until the GC runs. + # + tup, refcnt = mod.getarg_VARARGS() + assert tup == () + # the empty tuple is shared on CPython, so the refcnt will be >1. On + # PyPy it is not shared, though. + if not self.runappdirect: + assert refcnt == 1 + # + tup, refcnt = mod.getarg_VARARGS(1) + assert tup == (1,) + assert refcnt == 1 + # + tup, refcnt = mod.getarg_VARARGS(1, 2, 3) + assert tup == (1, 2, 3) + assert refcnt == 1 + # + raises(TypeError, mod.getarg_VARARGS, k=1) + + def test_call_METH_OLDARGS(self): + mod = self.import_extension('MyModule', [ ('getarg_OLD', 'METH_OLDARGS', ''' if(args) { - Py_INCREF(args); - return args; + return Py_BuildValue("Ol", args, args->ob_refcnt); } else { Py_INCREF(Py_None); @@ -40,6 +86,31 @@ } ''' ), + ]) + assert mod.getarg_OLD() is None + val, refcnt = mod.getarg_OLD(1) + assert val == 1 + val, refcnt = mod.getarg_OLD(1, 2) + assert val == (1, 2) + assert refcnt == 1 # see the comments in the test above + + def test_call_METH_KEYWORDS(self): + mod = self.import_extension('MyModule', [ + ('getarg_KW', 'METH_VARARGS | METH_KEYWORDS', + ''' + if (!kwargs) kwargs = Py_None; + return Py_BuildValue("OO", args, kwargs); + ''' + ), + ]) + assert mod.getarg_KW(1) == ((1,), None) + assert mod.getarg_KW(1, 2) == ((1, 2), None) + assert mod.getarg_KW(a=3, b=4) == ((), {'a': 3, 'b': 4}) + assert mod.getarg_KW(1, 2, a=3, b=4) == ((1, 2), {'a': 3, 'b': 4}) + assert mod.getarg_KW.__name__ == "getarg_KW" + + def test_func_attributes(self): + mod = self.import_extension('MyModule', [ ('isCFunction', 'METH_O', ''' if(PyCFunction_Check(args)) { @@ -67,30 +138,17 @@ ''' PyCFunction ptr = PyCFunction_GetFunction(args); if (!ptr) return NULL; - if (ptr == (PyCFunction)MyModule_getarg_O) + if (ptr == (PyCFunction)MyModule_getModule) Py_RETURN_TRUE; else Py_RETURN_FALSE; ''' ), ]) - assert mod.getarg_O(1) == 1 - assert mod.getarg_O.__name__ == "getarg_O" - raises(TypeError, mod.getarg_O) - raises(TypeError, mod.getarg_O, 1, 1) - - assert mod.getarg_NO() is None - raises(TypeError, mod.getarg_NO, 1) - raises(TypeError, mod.getarg_NO, 1, 1) - - assert mod.getarg_OLD(1) == 1 - assert mod.getarg_OLD() is None - assert mod.getarg_OLD(1, 2) == (1, 2) - - assert mod.isCFunction(mod.getarg_O) == "getarg_O" - assert mod.getModule(mod.getarg_O) == 'MyModule' + assert mod.isCFunction(mod.getModule) == "getModule" + assert mod.getModule(mod.getModule) == 'MyModule' if self.runappdirect: # XXX: fails untranslated - assert mod.isSameFunction(mod.getarg_O) + assert mod.isSameFunction(mod.getModule) raises(SystemError, mod.isSameFunction, 1) def test_check(self): diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py --- a/pypy/module/cpyext/test/test_tupleobject.py +++ b/pypy/module/cpyext/test/test_tupleobject.py @@ -6,7 +6,8 @@ from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib.debug import FatalError from pypy.module.cpyext.tupleobject import ( - PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size) + PyTupleObject, PyTuple_Check, PyTuple_SetItem, PyTuple_Size, + tuple_from_args_w) from pypy.module.cpyext.state import State class TestTupleObject(BaseApiTest): @@ -77,6 +78,16 @@ assert py_b == py_a assert py_b.c_ob_pypy_link == 0 + def test_tuple_from_args_w(self, space, api): + args_w = [space.newint(i) for i in (40, 41, 42)] + py_tuple = tuple_from_args_w(space, args_w) + assert py_tuple.c_ob_refcnt == 1 + assert api.PyTuple_Size(py_tuple) == 3 + py_items = [api.PyTuple_GetItem(py_tuple, i) for i in range(3)] + items = [api.PyInt_AsLong(py_obj) for py_obj in py_items] + assert items == [40, 41, 42] + decref(space, py_tuple) + def test_tuple_resize(self, space, api): state = space.fromcache(State) w_42 = space.wrap(42) diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py --- a/pypy/module/cpyext/tupleobject.py +++ b/pypy/module/cpyext/tupleobject.py @@ -113,6 +113,14 @@ track_reference(space, py_obj, w_obj) return w_obj +def tuple_from_args_w(space, args_w): + state = space.fromcache(State) + n = len(args_w) + py_tuple = state.C.PyTuple_New(n) # XXX: check for errors? + py_tuple = rffi.cast(PyTupleObject, py_tuple) + for i, w_obj in enumerate(args_w): + py_tuple.c_ob_item[i] = make_ref(space, w_obj) + return rffi.cast(PyObject, py_tuple) @cpython_api([PyObject, Py_ssize_t, PyObject], rffi.INT_real, error=-1) def PyTuple_SetItem(space, ref, index, py_obj): diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -378,7 +378,10 @@ args_w = space.fixedview(w_args) w_subtype = args_w[0] w_args = space.newtuple(args_w[1:]) - if not space.is_true(w_kwds): + # CCC: is this check still needed? I think we can safely remove it now + # that we manually handle __args__ inside W_PyCFunctionObject.descr_call, + # but we need to double check + if w_kwds and not space.is_true(w_kwds): w_kwds = None subtype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_subtype)) diff --git a/pypy/tool/cpyext/extbuild.py b/pypy/tool/cpyext/extbuild.py --- a/pypy/tool/cpyext/extbuild.py +++ b/pypy/tool/cpyext/extbuild.py @@ -114,14 +114,18 @@ codes = [] for funcname, flags, code in functions: cfuncname = "%s_%s" % (modname, funcname) + if 'METH_KEYWORDS' in flags: + signature = '(PyObject *self, PyObject *args, PyObject *kwargs)' + else: + signature = '(PyObject *self, PyObject *args)' methods_table.append( - "{\"%s\", %s, %s}," % (funcname, cfuncname, flags)) + "{\"%s\", (PyCFunction)%s, %s}," % (funcname, cfuncname, flags)) func_code = """ - static PyObject* %s(PyObject* self, PyObject* args) - { - %s - } - """ % (cfuncname, code) + static PyObject* {cfuncname}{signature} + {{ + {code} + }} + """.format(cfuncname=cfuncname, signature=signature, code=code) codes.append(func_code) body = "\n".join(codes) + """ From pypy.commits at gmail.com Sun Oct 22 11:39:54 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 08:39:54 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-refactor-methodobject: close this branch, will be merged into cpyext-avoid-roundtrip Message-ID: <59ecbbca.cd5e1c0a.92025.bbf2@mx.google.com> Author: Antonio Cuni Branch: cpyext-refactor-methodobject Changeset: r92826:1bd92743748b Date: 2017-10-22 17:36 +0200 http://bitbucket.org/pypy/pypy/changeset/1bd92743748b/ Log: close this branch, will be merged into cpyext-avoid-roundtrip From pypy.commits at gmail.com Sun Oct 22 11:50:16 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 08:50:16 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: bah, the flags attribute was not set because this class didn't call the parent __init__; fix Message-ID: <59ecbe38.13121c0a.24c7e.cad3@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92828:85dbd5c26f1d Date: 2017-10-22 17:49 +0200 http://bitbucket.org/pypy/pypy/changeset/85dbd5c26f1d/ Log: bah, the flags attribute was not set because this class didn't call the parent __init__; fix diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -139,11 +139,10 @@ return space.w_None class W_PyCMethodObject(W_PyCFunctionObject): - w_self = None + def __init__(self, space, ml, w_type): + W_PyCFunctionObject.__init__(self, space, ml, w_self=None) self.space = space - self.ml = ml - self.name = rffi.charp2str(rffi.cast(rffi.CCHARP, ml.c_ml_name)) self.w_objclass = w_type def __repr__(self): From pypy.commits at gmail.com Sun Oct 22 12:00:07 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 22 Oct 2017 09:00:07 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: add embedding arg to build_cffi_imports during translation too Message-ID: <59ecc087.4fc7df0a.269ff.d242@mx.google.com> Author: Matti Picus Branch: py3.5-mac-embedding Changeset: r92829:825f3ec1239e Date: 2017-10-22 18:55 +0300 http://bitbucket.org/pypy/pypy/changeset/825f3ec1239e/ Log: add embedding arg to build_cffi_imports during translation too diff --git a/pypy/goal/targetpypystandalone.py b/pypy/goal/targetpypystandalone.py --- a/pypy/goal/targetpypystandalone.py +++ b/pypy/goal/targetpypystandalone.py @@ -352,8 +352,9 @@ def hack_for_cffi_modules(self, driver): # HACKHACKHACK - # ugly hack to modify target goal from compile_* to build_cffi_imports - # this should probably get cleaned up and merged with driver.create_exe + # ugly hack to modify target goal from compile_* to build_cffi_imports, + # as done in package.py + # this is needed by the benchmark buildbot run, maybe do it as a seperate step there? from rpython.tool.runsubprocess import run_subprocess from rpython.translator.driver import taskdef import types @@ -363,11 +364,14 @@ def task_build_cffi_imports(self): ''' Use cffi to compile cffi interfaces to modules''' filename = os.path.join(pypydir, 'tool', 'build_cffi_imports.py') + if system.platform == 'darwin': + argv = [filename, '--embed-dependencies'] + else: + argv = [filename,] status, out, err = run_subprocess(str(driver.compute_exe_name()), - [filename]) + argv) sys.stdout.write(out) sys.stderr.write(err) - # otherwise, ignore errors driver.task_build_cffi_imports = types.MethodType(task_build_cffi_imports, driver) driver.tasks['build_cffi_imports'] = driver.task_build_cffi_imports, [compile_goal] driver.default_goal = 'build_cffi_imports' diff --git a/pypy/tool/release/package.py b/pypy/tool/release/package.py --- a/pypy/tool/release/package.py +++ b/pypy/tool/release/package.py @@ -312,7 +312,7 @@ parser.add_argument('--embedded-dependencies', '--no-embedded-dependencies', dest='embed_dependencies', action=NegateAction, - default=sys.platform == 'darwin', + default=(sys.platform == 'darwin'), help='whether to embed dependencies for distribution ' '(default on OS X)') options = parser.parse_args(args) From pypy.commits at gmail.com Sun Oct 22 12:00:12 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 22 Oct 2017 09:00:12 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: merge py3.5 into branch Message-ID: <59ecc08c.57abdf0a.93258.978c@mx.google.com> Author: Matti Picus Branch: py3.5-mac-embedding Changeset: r92830:7693d7783f04 Date: 2017-10-22 18:56 +0300 http://bitbucket.org/pypy/pypy/changeset/7693d7783f04/ Log: merge py3.5 into branch diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -76,6 +76,8 @@ ^lib_pypy/.+.c$ ^lib_pypy/.+.o$ ^lib_pypy/.+.so$ +^lib_pypy/.+.pyd$ +^lib_pypy/Release/ ^pypy/doc/discussion/.+\.html$ ^include/.+\.h$ ^include/.+\.inl$ diff --git a/lib-python/3/ctypes/test/test_bitfields.py b/lib-python/3/ctypes/test/test_bitfields.py --- a/lib-python/3/ctypes/test/test_bitfields.py +++ b/lib-python/3/ctypes/test/test_bitfields.py @@ -1,5 +1,5 @@ from ctypes import * -from ctypes.test import need_symbol +from ctypes.test import need_symbol, xfail import unittest import os @@ -279,6 +279,7 @@ self.assertEqual(b, b'\xef\xcd\xab\x21') @need_symbol('c_uint32') + @xfail def test_uint32_swap_big_endian(self): # Issue #23319 class Big(BigEndianStructure): diff --git a/lib-python/3/ctypes/test/test_byteswap.py b/lib-python/3/ctypes/test/test_byteswap.py --- a/lib-python/3/ctypes/test/test_byteswap.py +++ b/lib-python/3/ctypes/test/test_byteswap.py @@ -2,6 +2,7 @@ from binascii import hexlify from ctypes import * +from test.support import impl_detail def bin(s): return hexlify(memoryview(s)).decode().upper() @@ -22,6 +23,7 @@ setattr(bits, "i%s" % i, 1) dump(bits) + @impl_detail("slots are irrelevant on PyPy", pypy=False) def test_slots(self): class BigPoint(BigEndianStructure): __slots__ = () diff --git a/lib-python/3/ctypes/test/test_frombuffer.py b/lib-python/3/ctypes/test/test_frombuffer.py --- a/lib-python/3/ctypes/test/test_frombuffer.py +++ b/lib-python/3/ctypes/test/test_frombuffer.py @@ -85,7 +85,6 @@ del a gc.collect() # Should not crash - @xfail def test_from_buffer_copy(self): a = array.array("i", range(16)) x = (c_int * 16).from_buffer_copy(a) diff --git a/lib-python/3/test/test_bytes.py b/lib-python/3/test/test_bytes.py --- a/lib-python/3/test/test_bytes.py +++ b/lib-python/3/test/test_bytes.py @@ -721,9 +721,12 @@ self.assertIs(type(BytesSubclass(A())), BytesSubclass) # Test PyBytes_FromFormat() - @test.support.impl_detail("don't test cpyext here") def test_from_format(self): test.support.import_module('ctypes') + try: + from ctypes import pythonapi + except ImportError: + self.skipTest( "no pythonapi in ctypes") from ctypes import pythonapi, py_object, c_int, c_char_p PyBytes_FromFormat = pythonapi.PyBytes_FromFormat PyBytes_FromFormat.restype = py_object diff --git a/lib-python/3/test/test_unicode.py b/lib-python/3/test/test_unicode.py --- a/lib-python/3/test/test_unicode.py +++ b/lib-python/3/test/test_unicode.py @@ -2396,6 +2396,10 @@ # Test PyUnicode_FromFormat() def test_from_format(self): support.import_module('ctypes') + try: + from ctypes import pythonapi + except ImportError: + self.skipTest( "no pythonapi in ctypes") from ctypes import ( pythonapi, py_object, sizeof, c_int, c_long, c_longlong, c_ssize_t, diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,9 +8,14 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) + if cls == (_CData,): # this is the Array class defined below + res._ffiarray = None return res - + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) subletter = getattr(res._type_, '_type_', None) if subletter == 'c': @@ -55,7 +60,7 @@ for i in range(len(val)): target[i] = val[i] if len(val) < self._length_: - target[len(val)] = '\x00' + target[len(val)] = u'\x00' res.value = property(getvalue, setvalue) res._ffishape_ = (ffiarray, res._length_) @@ -164,7 +169,7 @@ if letter == 'c': return b"".join(l) if letter == 'u': - return "".join(l) + return u"".join(l) return l class Array(_CData, metaclass=ArrayMeta): diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py --- a/lib_pypy/_ctypes/primitive.py +++ b/lib_pypy/_ctypes/primitive.py @@ -232,9 +232,6 @@ elif tp == 'u': def _setvalue(self, val): - if isinstance(val, bytes): - val = val.decode(ConvMode.encoding, ConvMode.errors) - # possible if we use 'ignore' if val: self._buffer[0] = val def _getvalue(self): @@ -243,8 +240,6 @@ elif tp == 'c': def _setvalue(self, val): - if isinstance(val, str): - val = val.encode(ConvMode.encoding, ConvMode.errors) if val: self._buffer[0] = val def _getvalue(self): diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/TODO b/pypy/TODO --- a/pypy/TODO +++ b/pypy/TODO @@ -1,18 +1,4 @@ -TODO for the python3 test suite: - -* test_memoryview - Needs bytes/str changes. Probably easy. Work for this has begun on - py3k-memoryview (by mjacob) https://bugs.pypy.org/issue1542 - -own-tests: - -* module/test_lib_pypy - These crash the buildbots (via SyntaxErrors): others were really - made to run under Python 2.x and so simply fail - -* module.cpyext.test.test_structseq test_StructSeq - structseq now subclasses tuple on py3, which breaks how - BaseCpyTypeDescr.realize allocates it +... antocuni's older TODO: @@ -20,14 +6,6 @@ * run coverage against the parser/astbuilder/astcompiler: it's probably full of dead code because the grammar changed -* re-enable strategies https://bugs.pypy.org/issue1540 : - - re-enable IntDictStrategy - - re-enable StdObjSpace.listview_str - - re-enable the kwargs dict strategy in dictmultiobject.py - - re-enable view_as_kwargs - -* unskip numpypy tests in module/test_lib_pypy/numpypy/ - * optimize W_UnicodeObject, right now it stores both an unicode and an utf8 version of the same string diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/test/test_whatsnew.py b/pypy/doc/test/test_whatsnew.py --- a/pypy/doc/test/test_whatsnew.py +++ b/pypy/doc/test/test_whatsnew.py @@ -89,7 +89,7 @@ startrev, documented = parse_doc(last_whatsnew) merged, branch = get_merged_branches(ROOT, startrev, '') merged.discard('default') - merged.discard('py3k') + merged.discard('py3.5') merged.discard('') not_documented = merged.difference(documented) not_merged = documented.difference(merged) @@ -100,7 +100,7 @@ print '\n'.join(not_merged) print assert not not_documented - if branch == 'py3k': + if branch == 'py3.5': assert not not_merged else: assert branch in documented, 'Please document this branch before merging: %s' % branch diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -5,4 +5,8 @@ .. this is a revision shortly after release-pypy2.7-v5.9.0 .. startrev:d56dadcef996 +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -4,8 +4,3 @@ .. this is the revision after release-pypy3.5-5.9 .. startrev: be41e3ac0a29 - -.. branch: multiphase - -Implement PyType_FromSpec (PEP 384) and fix issues with PEP 489 support. - diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/pypy/module/_codecs/test/test_codecs.py b/pypy/module/_codecs/test/test_codecs.py --- a/pypy/module/_codecs/test/test_codecs.py +++ b/pypy/module/_codecs/test/test_codecs.py @@ -762,9 +762,16 @@ assert s == b'\xe9' def test_lone_surrogates(self): - for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', - 'utf-32', 'utf-32-le', 'utf-32-be'): + encodings = ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', + 'utf-32', 'utf-32-le', 'utf-32-be') + for encoding in encodings: raises(UnicodeEncodeError, u'\ud800'.encode, encoding) + assert (u'[\udc80]'.encode(encoding, "backslashreplace") == + '[\\udc80]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "ignore") == + '[]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "replace") == + '[?]'.encode(encoding)) def test_charmap_encode(self): assert 'xxx'.encode('charmap') == b'xxx' diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -574,6 +574,22 @@ import _socket raises(_socket.error, _socket.dup, 123456) + def test_recvmsg_issue2649(self): + import _socket as socket + listener = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + listener.bind(('::1', 1234)) + + s = socket.socket(family=socket.AF_INET6, type=socket.SOCK_DGRAM) + IPV6_RECVERR = 25 + s.setsockopt(socket.IPPROTO_IPV6, IPV6_RECVERR, 1) + + s.sendto(b'x', ('::1', 1234)) + try: + queue = s.recvmsg(1024, 1024, socket.MSG_ERRQUEUE) + except BlockingIOError as e: + assert True + def test_buffer(self): # Test that send/sendall/sendto accept a buffer as arg import _socket, os diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -274,7 +274,10 @@ if pos == 0: # Store the current keys in the PyDictObject. decref(space, py_dict.c__tmpkeys) - w_keys = space.call_method(space.w_dict, "keys", w_dict) + w_keyview = space.call_method(space.w_dict, "keys", w_dict) + # w_keys must use the object strategy in order to keep the keys alive + w_keys = space.newlist(space.listview(w_keyview)) + w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) else: @@ -287,10 +290,10 @@ decref(space, py_dict.c__tmpkeys) py_dict.c__tmpkeys = lltype.nullptr(PyObject.TO) return 0 - w_key = space.listview(w_keys)[pos] + w_key = space.listview(w_keys)[pos] # fast iff w_keys uses object strat w_value = space.getitem(w_dict, w_key) if pkey: - pkey[0] = as_pyobj(space, w_key) + pkey[0] = as_pyobj(space, w_key) if pvalue: pvalue[0] = as_pyobj(space, w_value) return 1 diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -15,7 +15,7 @@ from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount, jit -from rpython.rlib.debug import fatalerror +from rpython.rlib.debug import ll_assert, fatalerror #________________________________________________________ @@ -243,6 +243,11 @@ py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) + # + # Try to crash here, instead of randomly, if we don't keep w_obj alive + ll_assert(py_obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY, + "Bug in cpyext: The W_Root object was garbage-collected " + "while being converted to PyObject.") return py_obj else: return lltype.nullptr(PyObject.TO) diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref, as_pyobj from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,20 +90,21 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -121,8 +122,6 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -314,12 +313,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -332,9 +329,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -350,12 +345,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -337,12 +337,8 @@ PyObject* name = PyBytes_FromString("mymodule"); PyObject *obj = PyType_Type.tp_alloc(&PyType_Type, 0); PyHeapTypeObject *type = (PyHeapTypeObject*)obj; - if ((type->ht_type.tp_flags & Py_TPFLAGS_HEAPTYPE) == 0) - { - PyErr_SetString(PyExc_ValueError, - "Py_TPFLAGS_HEAPTYPE not set"); - return NULL; - } + /* this is issue #2434: logic from pybind11 */ + type->ht_type.tp_flags |= Py_TPFLAGS_HEAPTYPE; type->ht_type.tp_name = ((PyTypeObject*)args)->tp_name; PyType_Ready(&type->ht_type); ret = PyObject_SetAttrString((PyObject*)&type->ht_type, diff --git a/pypy/module/cpyext/test/test_unicodeobject.py b/pypy/module/cpyext/test/test_unicodeobject.py --- a/pypy/module/cpyext/test/test_unicodeobject.py +++ b/pypy/module/cpyext/test/test_unicodeobject.py @@ -307,6 +307,18 @@ assert module.utf8('xyz') == b'xyz' assert module.utf8('café') == 'café'.encode('utf-8') + def test_ready(self): + module = self.import_extension('foo', [ + ("unsafe_len", "METH_O", + """ + Py_ssize_t size = PyUnicode_GET_LENGTH(args); + return PyLong_FromSsize_t(size); + """)]) + assert module.unsafe_len(u"abc") == 3 + assert module.unsafe_len(u"café") == 4 + assert module.unsafe_len(u'aАbБcСdД') == 8 + assert module.unsafe_len(u"café\U0001F4A9") == 5 + class TestUnicode(BaseApiTest): def test_unicodeobject(self, space): @@ -863,30 +875,31 @@ assert space.unwrap(PyUnicode_Substring(space, w_str, 4, 3)) == u"" def test_Ready(self, space): - w_str = space.wrap(u'abc') # ASCII - py_str = as_pyobj(space, w_str) + def as_py_uni(val): + py_obj = new_empty_unicode(space, len(val)) + set_wbuffer(py_obj, rffi.unicode2wcharp(val)) + return py_obj + + py_str = as_py_uni(u'abc') # ASCII assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 1 assert get_ascii(py_str) == 1 - w_str = space.wrap(u'café') # latin1 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'café') # latin1 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 1 assert get_ascii(py_str) == 0 - w_str = space.wrap(u'Росси́я') # UCS2 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'Росси́я') # UCS2 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 2 assert get_ascii(py_str) == 0 - w_str = space.wrap(u'***\U0001f4a9***') # UCS4 - py_str = as_pyobj(space, w_str) + py_str = as_py_uni(u'***\U0001f4a9***') # UCS4 assert get_kind(py_str) == 0 - _PyUnicode_Ready(space, w_str) + _PyUnicode_Ready(space, py_str) assert get_kind(py_str) == 4 assert get_ascii(py_str) == 0 diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py --- a/pypy/module/cpyext/unicodeobject.py +++ b/pypy/module/cpyext/unicodeobject.py @@ -68,8 +68,10 @@ def unicode_attach(space, py_obj, w_obj, w_userdata=None): "Fills a newly allocated PyUnicodeObject with a unicode string" - set_wsize(py_obj, len(space.unicode_w(w_obj))) + value = space.unicode_w(w_obj) + set_wsize(py_obj, len(value)) set_wbuffer(py_obj, lltype.nullptr(rffi.CWCHARP.TO)) + _readify(space, py_obj, value) def unicode_realize(space, py_obj): """ @@ -265,8 +267,11 @@ assert isinstance(w_obj, unicodeobject.W_UnicodeObject) py_obj = as_pyobj(space, w_obj) assert get_kind(py_obj) == WCHAR_KIND + return _readify(space, py_obj, w_obj._value) + +def _readify(space, py_obj, value): maxchar = 0 - for c in w_obj._value: + for c in value: if ord(c) > maxchar: maxchar = ord(c) if maxchar > MAX_UNICODE: @@ -275,7 +280,7 @@ maxchar) if maxchar < 256: ucs1_data = rffi.str2charp(unicode_encode_latin_1( - w_obj._value, len(w_obj._value), errors='strict')) + value, len(value), errors='strict')) set_data(py_obj, cts.cast('void*', ucs1_data)) set_kind(py_obj, _1BYTE_KIND) set_len(py_obj, get_wsize(py_obj)) @@ -290,7 +295,7 @@ elif maxchar < 65536: # XXX: assumes that sizeof(wchar_t) == 4 ucs2_str = unicode_encode_utf_16_helper( - w_obj._value, len(w_obj._value), errors='strict', + value, len(value), errors='strict', byteorder=runicode.BYTEORDER) ucs2_data = cts.cast('Py_UCS2 *', rffi.str2charp(ucs2_str)) set_data(py_obj, cts.cast('void*', ucs2_data)) @@ -302,9 +307,8 @@ # XXX: assumes that sizeof(wchar_t) == 4 if not get_wbuffer(py_obj): # Copy unicode buffer - u = w_obj._value - set_wbuffer(py_obj, rffi.unicode2wcharp(u)) - set_wsize(py_obj, len(u)) + set_wbuffer(py_obj, rffi.unicode2wcharp(value)) + set_wsize(py_obj, len(value)) ucs4_data = get_wbuffer(py_obj) set_data(py_obj, cts.cast('void*', ucs4_data)) set_len(py_obj, get_wsize(py_obj)) @@ -332,9 +336,7 @@ w_res = utf_32_decode(space, value, w_final=space.w_False) else: raise oefmt(space.w_SystemError, "invalid kind") - w_ret = space.unpackiterable(w_res)[0] - _PyUnicode_Ready(space, w_ret) - return w_ret + return space.unpackiterable(w_res)[0] @cts.decl("Py_UNICODE * PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)") def PyUnicode_AsUnicodeAndSize(space, ref, psize): diff --git a/pypy/module/imp/test/test_import.py b/pypy/module/imp/test/test_import.py --- a/pypy/module/imp/test/test_import.py +++ b/pypy/module/imp/test/test_import.py @@ -601,25 +601,15 @@ import pkg.a, imp imp.reload(pkg.a) - def test_reload_builtin(self): - import sys, imp - oldpath = sys.path - try: - del sys.settrace - except AttributeError: - pass - - imp.reload(sys) - - assert sys.path is oldpath - assert 'settrace' not in dir(sys) # at least on CPython 3.5.2 - def test_reload_builtin_doesnt_clear(self): import imp import sys sys.foobar = "baz" - imp.reload(sys) - assert sys.foobar == "baz" + try: + imp.reload(sys) + assert sys.foobar == "baz" + finally: + del sys.foobar def test_reimport_builtin_simple_case_1(self): import sys, time @@ -637,18 +627,18 @@ def test_reimport_builtin(self): import imp, sys, time - oldpath = sys.path - time.tzname = "" + old_sleep = time.sleep + time.sleep = "" del sys.modules['time'] import time as time1 assert sys.modules['time'] is time1 - assert time.tzname == "" + assert time.sleep == "" - imp.reload(time1) # don't leave a broken time.tzname behind + imp.reload(time1) # don't leave a broken time.sleep behind import time - assert time.tzname != "" + assert time.sleep is old_sleep def test_reload_infinite(self): import infinite_reload diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py --- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py +++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py @@ -37,7 +37,7 @@ v = cffi.__version__.replace('+', '') p = os.path.join(parent, 'doc', 'source', 'installation.rst') content = open(p).read() - assert ("/cffi-%s.tar.gz" % v) in content + assert (" package version %s:" % v) in content def test_setup_version(): parent = os.path.dirname(os.path.dirname(cffi.__file__)) diff --git a/pypy/module/unicodedata/test/test_hyp.py b/pypy/module/unicodedata/test/test_hyp.py --- a/pypy/module/unicodedata/test/test_hyp.py +++ b/pypy/module/unicodedata/test/test_hyp.py @@ -40,6 +40,7 @@ @pytest.mark.parametrize('NF1, NF2, NF3', compositions) @example(s=u'---\uafb8\u11a7---') # issue 2289 + at example(s=u'\ufacf') @settings(max_examples=1000) @given(s=st.text()) def test_composition(s, space, NF1, NF2, NF3): diff --git a/pypy/objspace/std/test/test_bytesobject.py b/pypy/objspace/std/test/test_bytesobject.py --- a/pypy/objspace/std/test/test_bytesobject.py +++ b/pypy/objspace/std/test/test_bytesobject.py @@ -1,4 +1,7 @@ # coding: utf-8 + +from pypy.interpreter.error import OperationError + class TestW_BytesObject: def teardown_method(self, method): @@ -96,6 +99,78 @@ w_res = space.call_function(space.w_bytes, space.wrap([42])) assert space.str_w(w_res) == '*' + +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.binary(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + class AppTestBytesObject: def setup_class(cls): diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1,6 +1,7 @@ # -*- encoding: utf-8 -*- import py import sys +from pypy.interpreter.error import OperationError class TestUnicodeObject: @@ -38,6 +39,55 @@ space.raises_w(space.w_UnicodeEncodeError, space.text_w, w_uni) +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.text(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + class AppTestUnicodeStringStdOnly: def test_compares(self): assert type('a') != type(b'a') diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -748,17 +748,6 @@ interpreted as in slice notation. """ - def decode(): - """S.decode(encoding=None, errors='strict') -> string or unicode - - Decode S using the codec registered for encoding. encoding defaults - to the default encoding. errors may be given to set a different error - handling scheme. Default is 'strict' meaning that encoding errors raise - a UnicodeDecodeError. Other possible values are 'ignore' and 'replace' - as well as any other name registered with codecs.register_error that is - able to handle UnicodeDecodeErrors. - """ - def encode(): """S.encode(encoding=None, errors='strict') -> string or unicode diff --git a/pypy/tool/pytest/objspace.py b/pypy/tool/pytest/objspace.py --- a/pypy/tool/pytest/objspace.py +++ b/pypy/tool/pytest/objspace.py @@ -57,9 +57,6 @@ if not ok: py.test.skip("cannot runappdirect test: " "module %r required" % (modname,)) - else: - if '__pypy__' in value: - py.test.skip("no module __pypy__ on top of CPython") continue if info is None: py.test.skip("cannot runappdirect this test on top of CPython") diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: diff --git a/rpython/jit/metainterp/optimizeopt/intbounds.py b/rpython/jit/metainterp/optimizeopt/intbounds.py --- a/rpython/jit/metainterp/optimizeopt/intbounds.py +++ b/rpython/jit/metainterp/optimizeopt/intbounds.py @@ -305,6 +305,10 @@ # Transform into INT_ADD. The following guard will be killed # by optimize_GUARD_NO_OVERFLOW; if we see instead an # optimize_GUARD_OVERFLOW, then InvalidLoop. + + # NB: this case also takes care of int_add_ovf with 0 as on of the + # arguments: the result will be bounded, and then the optimization + # for int_add with 0 as argument will remove the op. op = self.replace_op_with(op, rop.INT_ADD) return self.emit(op) @@ -325,6 +329,7 @@ return None resbound = b0.sub_bound(b1) if resbound.bounded(): + # this case takes care of int_sub_ovf(x, 0) as well op = self.replace_op_with(op, rop.INT_SUB) return self.emit(op) @@ -342,6 +347,7 @@ b2 = self.getintbound(op.getarg(1)) resbound = b1.mul_bound(b2) if resbound.bounded(): + # this case also takes care of multiplication with 0 and 1 op = self.replace_op_with(op, rop.INT_MUL) return self.emit(op) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -1962,6 +1962,55 @@ """ self.optimize_loop(ops, expected) + ops = """ + [i0] + i1 = int_mul_ovf(0, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(1, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 1) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + def test_fold_constant_partial_ops_float(self): ops = """ [f0] diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py --- a/rpython/rlib/_rsocket_rffi.py +++ b/rpython/rlib/_rsocket_rffi.py @@ -162,7 +162,7 @@ IP_RECVRETOPTS IP_RETOPTS IP_TOS IP_TTL MSG_BTAG MSG_ETAG MSG_CTRUNC MSG_DONTROUTE MSG_DONTWAIT MSG_EOR MSG_OOB -MSG_PEEK MSG_TRUNC MSG_WAITALL +MSG_PEEK MSG_TRUNC MSG_WAITALL MSG_ERRQUEUE NI_DGRAM NI_MAXHOST NI_MAXSERV NI_NAMEREQD NI_NOFQDN NI_NUMERICHOST NI_NUMERICSERV @@ -535,7 +535,7 @@ int cmsg_status; struct iovec iov; struct recvmsg_info* retinfo; - int error_flag; // variable to be set in case of special errors. + int error_flag = 0; // variable to be set in case of special errors. int cmsgdatalen = 0; // variables that are set to 1, if the message charp has been allocated @@ -709,6 +709,7 @@ free(retinfo); } } + if (error_flag==0) error_flag = -1; return error_flag; err_closefds: diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py --- a/rpython/rlib/rsocket.py +++ b/rpython/rlib/rsocket.py @@ -1074,7 +1074,7 @@ if address is not None: address.unlock() - if _c.geterrno() == _c.EINTR: + if (_c.geterrno() == _c.EINTR) or (_c.geterrno() == 11): raise last_error() if (reply == -10000): raise RSocketError("Invalid message size") diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -877,32 +877,31 @@ ch = ord(s[pos]) pos += 1 ch2 = 0 - if 0xD800 <= ch < 0xDC00: - if not allow_surrogates: - ru, rs, pos = errorhandler(errors, public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - if rs is not None: - # py3k only - if len(rs) % 4 != 0: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - result.append(rs) - continue - for ch in ru: - if ord(ch) < 0xD800: - _STORECHAR32(result, ord(ch), byteorder) - else: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) + if not allow_surrogates and 0xD800 <= ch < 0xE000: + ru, rs, pos = errorhandler(errors, public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + if rs is not None: + # py3k only + if len(rs) % 4 != 0: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + result.append(rs) continue - elif MAXUNICODE < 65536 and pos < size: - ch2 = ord(s[pos]) - if 0xDC00 <= ch2 < 0xE000: - ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; - pos += 1 + for ch in ru: + if ord(ch) < 0xD800: + _STORECHAR32(result, ord(ch), byteorder) + else: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + continue + if 0xD800 <= ch < 0xDC00 and MAXUNICODE < 65536 and pos < size: + ch2 = ord(s[pos]) + if 0xDC00 <= ch2 < 0xE000: + ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; + pos += 1 _STORECHAR32(result, ch, byteorder) return result.build() diff --git a/rpython/rlib/test/test_runicode.py b/rpython/rlib/test/test_runicode.py --- a/rpython/rlib/test/test_runicode.py +++ b/rpython/rlib/test/test_runicode.py @@ -2,6 +2,7 @@ import py import sys, random +import struct from rpython.rlib import runicode from hypothesis import given, settings, strategies @@ -266,11 +267,12 @@ assert replace_with(u'rep', None) == '\x00<\x00r\x00e\x00p\x00>' assert replace_with(None, '\xca\xfe') == '\x00<\xca\xfe\x00>' - def test_utf32_surrogates(self): + @py.test.mark.parametrize('unich',[u"\ud800", u"\udc80"]) + def test_utf32_surrogates(self, unich): assert runicode.unicode_encode_utf_32_be( - u"\ud800", 1, None) == '\x00\x00\xd8\x00' + unich, 1, None) == struct.pack('>i', ord(unich)) py.test.raises(UnicodeEncodeError, runicode.unicode_encode_utf_32_be, - u"\ud800", 1, None, allow_surrogates=False) + unich, 1, None, allow_surrogates=False) def replace_with(ru, rs): def errorhandler(errors, enc, msg, u, startingpos, endingpos): if errors == 'strict': @@ -278,7 +280,7 @@ endingpos, msg) return ru, rs, endingpos return runicode.unicode_encode_utf_32_be( - u"<\ud800>", 3, None, + u"<%s>" % unich, 3, None, errorhandler, allow_surrogates=False) assert replace_with(u'rep', None) == u''.encode('utf-32-be') assert replace_with(None, '\xca\xfe\xca\xfe') == '\x00\x00\x00<\xca\xfe\xca\xfe\x00\x00\x00>' @@ -432,7 +434,7 @@ assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'ignore', final=True) == (u'aaaabbbb', len(seq) + 8)) assert (self.decoder(seq, len(seq), 'custom', final=True, - errorhandler=self.custom_replace) == + errorhandler=self.custom_replace) == (FOO * len(seq), len(seq))) assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'custom', final=True, errorhandler=self.custom_replace) == @@ -628,7 +630,7 @@ msg='invalid continuation byte') assert self.decoder(seq, len(seq), 'replace', final=True ) == (res, len(seq)) - assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, + assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'replace', final=True) == (u'aaaa' + res + u'bbbb', len(seq) + 8)) res = res.replace(FFFD, u'') From pypy.commits at gmail.com Sun Oct 22 13:42:12 2017 From: pypy.commits at gmail.com (antocuni) Date: Sun, 22 Oct 2017 10:42:12 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: bah, fix this one as well Message-ID: <59ecd874.09a0df0a.bcf4c.d30e@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92831:79e9438da998 Date: 2017-10-22 19:41 +0200 http://bitbucket.org/pypy/pypy/changeset/79e9438da998/ Log: bah, fix this one as well diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -188,11 +188,10 @@ return isinstance(w_obj, BuiltinFunction) class W_PyCClassMethodObject(W_PyCFunctionObject): - w_self = None + def __init__(self, space, ml, w_type): + W_PyCFunctionObject.__init__(self, space, ml, w_self=None) self.space = space - self.ml = ml - self.name = rffi.charp2str(rffi.cast(rffi.CCHARP, ml.c_ml_name)) self.w_objclass = w_type def __repr__(self): From pypy.commits at gmail.com Mon Oct 23 11:56:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 23 Oct 2017 08:56:21 -0700 (PDT) Subject: [pypy-commit] pypy default: improve this test by using the same 'seen' pattern also used by other tests, so that it's easier to follow the control flow Message-ID: <59ee1125.8d9edf0a.33627.727a@mx.google.com> Author: Antonio Cuni Branch: Changeset: r92832:b015a2924278 Date: 2017-10-23 17:55 +0200 http://bitbucket.org/pypy/pypy/changeset/b015a2924278/ Log: improve this test by using the same 'seen' pattern also used by other tests, so that it's easier to follow the control flow diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -290,30 +290,51 @@ def test_random_switching(self): from _continuation import continulet # + seen = [] + def t1(c1): - return c1.switch() + seen.append(3) + res = c1.switch() + seen.append(6) + return res + def s1(c1, n): + seen.append(2) assert n == 123 c2 = t1(c1) - return c1.switch('a') + 1 + seen.append(7) + res = c1.switch('a') + 1 + seen.append(10) + return res # def s2(c2, c1): + seen.append(5) res = c1.switch(c2) + seen.append(8) assert res == 'a' - return c2.switch('b') + 2 + res = c2.switch('b') + 2 + seen.append(12) + return res # def f(): + seen.append(1) c1 = continulet(s1, 123) c2 = continulet(s2, c1) c1.switch() + seen.append(4) res = c2.switch() + seen.append(9) assert res == 'b' res = c1.switch(1000) + seen.append(11) assert res == 1001 - return c2.switch(2000) + res = c2.switch(2000) + seen.append(13) + return res # res = f() assert res == 2002 + assert seen == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] def test_f_back(self): import sys From pypy.commits at gmail.com Mon Oct 23 11:57:25 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 23 Oct 2017 08:57:25 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: kill this check; all the tests pass Message-ID: <59ee1165.87bbdf0a.6e63e.6f91@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92833:a1ed00dd3eba Date: 2017-10-22 20:08 +0200 http://bitbucket.org/pypy/pypy/changeset/a1ed00dd3eba/ Log: kill this check; all the tests pass diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py --- a/pypy/module/cpyext/typeobject.py +++ b/pypy/module/cpyext/typeobject.py @@ -378,12 +378,6 @@ args_w = space.fixedview(w_args) w_subtype = args_w[0] w_args = space.newtuple(args_w[1:]) - # CCC: is this check still needed? I think we can safely remove it now - # that we manually handle __args__ inside W_PyCFunctionObject.descr_call, - # but we need to double check - if w_kwds and not space.is_true(w_kwds): - w_kwds = None - subtype = rffi.cast(PyTypeObjectPtr, make_ref(space, w_subtype)) try: w_obj = generic_cpy_call(space, tp_new, subtype, w_args, w_kwds) From pypy.commits at gmail.com Mon Oct 23 11:57:28 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 23 Oct 2017 08:57:28 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: merge heads Message-ID: <59ee1168.119adf0a.d021f.100e@mx.google.com> Author: Antonio Cuni Branch: cpyext-avoid-roundtrip Changeset: r92834:7b4e95e7c3b7 Date: 2017-10-23 17:56 +0200 http://bitbucket.org/pypy/pypy/changeset/7b4e95e7c3b7/ Log: merge heads diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py --- a/pypy/module/cpyext/methodobject.py +++ b/pypy/module/cpyext/methodobject.py @@ -188,11 +188,10 @@ return isinstance(w_obj, BuiltinFunction) class W_PyCClassMethodObject(W_PyCFunctionObject): - w_self = None + def __init__(self, space, ml, w_type): + W_PyCFunctionObject.__init__(self, space, ml, w_self=None) self.space = space - self.ml = ml - self.name = rffi.charp2str(rffi.cast(rffi.CCHARP, ml.c_ml_name)) self.w_objclass = w_type def __repr__(self): From pypy.commits at gmail.com Mon Oct 23 11:59:18 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 23 Oct 2017 08:59:18 -0700 (PDT) Subject: [pypy-commit] pypy default: fix whitespace Message-ID: <59ee11d6.8d811c0a.c44bd.374b@mx.google.com> Author: Antonio Cuni Branch: Changeset: r92835:94ddc680fce1 Date: 2017-10-23 17:58 +0200 http://bitbucket.org/pypy/pypy/changeset/94ddc680fce1/ Log: fix whitespace diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -291,13 +291,13 @@ from _continuation import continulet # seen = [] - + # def t1(c1): seen.append(3) res = c1.switch() seen.append(6) return res - + # def s1(c1, n): seen.append(2) assert n == 123 From pypy.commits at gmail.com Mon Oct 23 12:22:18 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 23 Oct 2017 09:22:18 -0700 (PDT) Subject: [pypy-commit] pypy default: improve this test by giving more descriptive names to variables; in particular, f vs f* was a bit confusing, at least for me :) Message-ID: <59ee173a.0a181c0a.9703f.3c0d@mx.google.com> Author: Antonio Cuni Branch: Changeset: r92836:1c3af3c4a095 Date: 2017-10-23 18:21 +0200 http://bitbucket.org/pypy/pypy/changeset/1c3af3c4a095/ Log: improve this test by giving more descriptive names to variables; in particular, f vs f* was a bit confusing, at least for me :) diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -340,37 +340,37 @@ import sys from _continuation import continulet # - def g(c): + def bar(c): c.switch(sys._getframe(0)) c.switch(sys._getframe(0).f_back) c.switch(sys._getframe(1)) c.switch(sys._getframe(1).f_back) - assert sys._getframe(2) is f3.f_back + assert sys._getframe(2) is f3_foo.f_back c.switch(sys._getframe(2)) - def f(c): - g(c) + def foo(c): + bar(c) # - c = continulet(f) - f1 = c.switch() - assert f1.f_code.co_name == 'g' - f2 = c.switch() - assert f2.f_code.co_name == 'f' - f3 = c.switch() - assert f3 is f2 - assert f1.f_back is f3 + c = continulet(foo) + f1_bar = c.switch() + assert f1_bar.f_code.co_name == 'bar' + f2_foo = c.switch() + assert f2_foo.f_code.co_name == 'foo' + f3_foo = c.switch() + assert f3_foo is f2_foo + assert f1_bar.f_back is f3_foo def main(): - f4 = c.switch() - assert f4.f_code.co_name == 'main', repr(f4.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f4_main = c.switch() + assert f4_main.f_code.co_name == 'main' + assert f3_foo.f_back is f1_bar # not running, so a loop def main2(): - f5 = c.switch() - assert f5.f_code.co_name == 'main2', repr(f5.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f5_main2 = c.switch() + assert f5_main2.f_code.co_name == 'main2' + assert f3_foo.f_back is f1_bar # not running, so a loop main() main2() res = c.switch() assert res is None - assert f3.f_back is None + assert f3_foo.f_back is None def test_traceback_is_complete(self): import sys From pypy.commits at gmail.com Mon Oct 23 15:32:36 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 23 Oct 2017 12:32:36 -0700 (PDT) Subject: [pypy-commit] pypy cpyext-avoid-roundtrip: fix for C-89 (windows vc 9) Message-ID: <59ee43d4.69a8df0a.29688.85c1@mx.google.com> Author: Matti Picus Branch: cpyext-avoid-roundtrip Changeset: r92837:76c31eb1696c Date: 2017-10-23 22:31 +0300 http://bitbucket.org/pypy/pypy/changeset/76c31eb1696c/ Log: fix for C-89 (windows vc 9) diff --git a/pypy/module/cpyext/src/object.c b/pypy/module/cpyext/src/object.c --- a/pypy/module/cpyext/src/object.c +++ b/pypy/module/cpyext/src/object.c @@ -85,14 +85,16 @@ static PyObject * _generic_alloc(PyTypeObject *type, Py_ssize_t nitems) { + Py_ssize_t size; + PyObject *pyobj; if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_INCREF(type); - Py_ssize_t size = type->tp_basicsize; + size = type->tp_basicsize; if (type->tp_itemsize) size += nitems * type->tp_itemsize; - PyObject *pyobj = (PyObject*)_PyPy_Malloc(size); + pyobj = (PyObject*)_PyPy_Malloc(size); if (pyobj == NULL) return NULL; From pypy.commits at gmail.com Tue Oct 24 17:38:21 2017 From: pypy.commits at gmail.com (stian) Date: Tue, 24 Oct 2017 14:38:21 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Fix int_mod with mod argument Message-ID: <59efb2cd.450c1c0a.a7e8.95a8@mx.google.com> Author: stian Branch: math-improvements Changeset: r92840:e80b5cdab0a6 Date: 2017-10-24 21:38 +0200 http://bitbucket.org/pypy/pypy/changeset/e80b5cdab0a6/ Log: Fix int_mod with mod argument diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -982,7 +982,9 @@ size_b = b.numdigits() - if c is not None: + if b.sign == 0: + return ONERBIGINT + elif c is not None: if c.sign == 0: raise ValueError("pow() 3rd argument cannot be 0") @@ -1009,15 +1011,12 @@ # so we only do it when it buys something. if a.sign < 0 or a.numdigits() > c.numdigits(): a = a.mod(c) - - elif b.sign == 0: - return ONERBIGINT elif a.sign == 0: return NULLRBIGINT elif size_b == 1: if b._digits[0] == ONEDIGIT: return a - elif a.numdigits() == 1: + elif a.numdigits() == 1 and c is None: adigit = a.digit(0) digit = b.digit(0) if adigit == 1: @@ -1118,7 +1117,10 @@ # XXX failed to implement raise ValueError("bigint pow() too negative") - if c is not None: + assert b >= 0 + if b == 0: + return ONERBIGINT + elif c is not None: if c.sign == 0: raise ValueError("pow() 3rd argument cannot be 0") @@ -1145,13 +1147,9 @@ # so we only do it when it buys something. if a.sign < 0 or a.numdigits() > c.numdigits(): a = a.mod(c) - - elif b == 0: - return ONERBIGINT elif a.sign == 0: return NULLRBIGINT - - if b == 1: + elif b == 1: return a elif a.numdigits() == 1: adigit = a.digit(0) @@ -1175,6 +1173,7 @@ # Left-to-right binary exponentiation (HAC Algorithm 14.79) # http://www.cacr.math.uwaterloo.ca/hac/about/chap14.pdf j = 1 << (SHIFT-1) + while j != 0: z = _help_mult(z, z, c) if b & j: diff --git a/rpython/rlib/test/test_rbigint.py b/rpython/rlib/test/test_rbigint.py --- a/rpython/rlib/test/test_rbigint.py +++ b/rpython/rlib/test/test_rbigint.py @@ -164,6 +164,11 @@ r2 = op1 ** op2 assert r1.tolong() == r2 + r3 = rl_op1.int_pow(op2, rbigint.fromint(1000)) + r4 = pow(op1, op2, 1000) + print op1, op2 + assert r3.tolong() == r4 + def test_int_pow(self): for op1 in gen_signs(long_vals_not_too_big[:-2]): for op2 in [0, 1, 2, 8, 9, 10, 11]: @@ -171,6 +176,12 @@ r1 = rl_op1.int_pow(op2) r2 = op1 ** op2 assert r1.tolong() == r2 + + r3 = rl_op1.int_pow(op2, rbigint.fromint(1000)) + r4 = pow(op1, op2, 1000) + print op1, op2 + assert r3.tolong() == r4 + def test_touint(self): result = r_uint(sys.maxint + 42) rl = rbigint.fromint(sys.maxint).add(rbigint.fromint(42)) From pypy.commits at gmail.com Tue Oct 24 17:38:17 2017 From: pypy.commits at gmail.com (stian) Date: Tue, 24 Oct 2017 14:38:17 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: replace construction of a onedigit with premade, and mark sign and size as immutable (which is true post contruction) Message-ID: <59efb2c9.c7341c0a.57fd1.893c@mx.google.com> Author: stian Branch: math-improvements Changeset: r92838:2a90db93be84 Date: 2017-10-24 15:03 +0200 http://bitbucket.org/pypy/pypy/changeset/2a90db93be84/ Log: replace construction of a onedigit with premade, and mark sign and size as immutable (which is true post contruction) diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -143,7 +143,7 @@ class rbigint(object): """This is a reimplementation of longs using a list of digits.""" _immutable_ = True - _immutable_fields_ = ["_digits"] + _immutable_fields_ = ["_digits", "size", "sign"] def __init__(self, digits=NULLDIGITS, sign=0, size=0): if not we_are_translated(): @@ -1033,7 +1033,7 @@ # At this point a, b, and c are guaranteed non-negative UNLESS # c is NULL, in which case a may be negative. */ - z = rbigint([ONEDIGIT], 1, 1) + z = ONERBIGINT # python adaptation: moved macros REDUCE(X) and MULT(X, Y, result) # into helper function result = _help_mult(x, y, c) @@ -1168,7 +1168,7 @@ # At this point a, b, and c are guaranteed non-negative UNLESS # c is NULL, in which case a may be negative. */ - z = rbigint([ONEDIGIT], 1, 1) + z = ONERBIGINT # python adaptation: moved macros REDUCE(X) and MULT(X, Y, result) # into helper function result = _help_mult(x, y, c) From pypy.commits at gmail.com Tue Oct 24 17:38:19 2017 From: pypy.commits at gmail.com (stian) Date: Tue, 24 Oct 2017 14:38:19 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: Digits don't change post-construction Message-ID: <59efb2cb.88d31c0a.40030.9f0b@mx.google.com> Author: stian Branch: math-improvements Changeset: r92839:7427553eee6b Date: 2017-10-24 18:42 +0200 http://bitbucket.org/pypy/pypy/changeset/7427553eee6b/ Log: Digits don't change post-construction diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -143,7 +143,7 @@ class rbigint(object): """This is a reimplementation of longs using a list of digits.""" _immutable_ = True - _immutable_fields_ = ["_digits", "size", "sign"] + _immutable_fields_ = ["_digits[*]", "size", "sign"] def __init__(self, digits=NULLDIGITS, sign=0, size=0): if not we_are_translated(): From pypy.commits at gmail.com Tue Oct 24 22:54:45 2017 From: pypy.commits at gmail.com (rlamy) Date: Tue, 24 Oct 2017 19:54:45 -0700 (PDT) Subject: [pypy-commit] pypy default: ctypes: allow ptr[0] = foo when ptr is a pointer to struct Message-ID: <59effcf5.14a1df0a.569fe.9bfd@mx.google.com> Author: Ronan Lamy Branch: Changeset: r92841:c2cf8296095b Date: 2017-10-25 03:54 +0100 http://bitbucket.org/pypy/pypy/changeset/c2cf8296095b/ Log: ctypes: allow ptr[0] = foo when ptr is a pointer to struct diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -176,6 +176,10 @@ def _get_buffer_value(self): return self._buffer[0] + def _copy_to(self, addr): + target = type(self).from_address(addr)._buffer + target[0] = self._get_buffer_value() + def _to_ffi_param(self): if self.__class__._is_pointer_like(): return self._get_buffer_value() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -114,7 +114,9 @@ cobj = self._type_.from_param(value) if ensure_objects(cobj) is not None: store_reference(self, index, cobj._objects) - self._subarray(index)[0] = cobj._get_buffer_value() + address = self._buffer[0] + address += index * sizeof(self._type_) + cobj._copy_to(address) def __nonzero__(self): return self._buffer[0] != 0 diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -291,6 +291,11 @@ def _get_buffer_value(self): return self._buffer.buffer + def _copy_to(self, addr): + from ctypes import memmove + origin = self._get_buffer_value() + memmove(addr, origin, self._fficompositesize_) + def _to_ffi_param(self): return self._buffer From pypy.commits at gmail.com Tue Oct 24 23:06:02 2017 From: pypy.commits at gmail.com (rlamy) Date: Tue, 24 Oct 2017 20:06:02 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: hg merge default Message-ID: <59efff9a.21b9df0a.88c5.381e@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92842:1c9a8dfaca36 Date: 2017-10-25 04:05 +0100 http://bitbucket.org/pypy/pypy/changeset/1c9a8dfaca36/ Log: hg merge default diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -165,6 +165,10 @@ def _get_buffer_value(self): return self._buffer[0] + def _copy_to(self, addr): + target = type(self).from_address(addr)._buffer + target[0] = self._get_buffer_value() + def _to_ffi_param(self): if self.__class__._is_pointer_like(): return self._get_buffer_value() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -113,7 +113,9 @@ cobj = self._type_.from_param(value) if ensure_objects(cobj) is not None: store_reference(self, index, cobj._objects) - self._subarray(index)[0] = cobj._get_buffer_value() + address = self._buffer[0] + address += index * sizeof(self._type_) + cobj._copy_to(address) def __bool__(self): return self._buffer[0] != 0 diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -290,6 +290,11 @@ def _get_buffer_value(self): return self._buffer.buffer + def _copy_to(self, addr): + from ctypes import memmove + origin = self._get_buffer_value() + memmove(addr, origin, self._fficompositesize_) + def _to_ffi_param(self): return self._buffer diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -290,66 +290,87 @@ def test_random_switching(self): from _continuation import continulet # + seen = [] + # def t1(c1): - return c1.switch() + seen.append(3) + res = c1.switch() + seen.append(6) + return res + # def s1(c1, n): + seen.append(2) assert n == 123 c2 = t1(c1) - return c1.switch('a') + 1 + seen.append(7) + res = c1.switch('a') + 1 + seen.append(10) + return res # def s2(c2, c1): + seen.append(5) res = c1.switch(c2) + seen.append(8) assert res == 'a' - return c2.switch('b') + 2 + res = c2.switch('b') + 2 + seen.append(12) + return res # def f(): + seen.append(1) c1 = continulet(s1, 123) c2 = continulet(s2, c1) c1.switch() + seen.append(4) res = c2.switch() + seen.append(9) assert res == 'b' res = c1.switch(1000) + seen.append(11) assert res == 1001 - return c2.switch(2000) + res = c2.switch(2000) + seen.append(13) + return res # res = f() assert res == 2002 + assert seen == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] def test_f_back(self): import sys from _continuation import continulet # - def g(c): + def bar(c): c.switch(sys._getframe(0)) c.switch(sys._getframe(0).f_back) c.switch(sys._getframe(1)) c.switch(sys._getframe(1).f_back) - assert sys._getframe(2) is f3.f_back + assert sys._getframe(2) is f3_foo.f_back c.switch(sys._getframe(2)) - def f(c): - g(c) + def foo(c): + bar(c) # - c = continulet(f) - f1 = c.switch() - assert f1.f_code.co_name == 'g' - f2 = c.switch() - assert f2.f_code.co_name == 'f' - f3 = c.switch() - assert f3 is f2 - assert f1.f_back is f3 + c = continulet(foo) + f1_bar = c.switch() + assert f1_bar.f_code.co_name == 'bar' + f2_foo = c.switch() + assert f2_foo.f_code.co_name == 'foo' + f3_foo = c.switch() + assert f3_foo is f2_foo + assert f1_bar.f_back is f3_foo def main(): - f4 = c.switch() - assert f4.f_code.co_name == 'main', repr(f4.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f4_main = c.switch() + assert f4_main.f_code.co_name == 'main' + assert f3_foo.f_back is f1_bar # not running, so a loop def main2(): - f5 = c.switch() - assert f5.f_code.co_name == 'main2', repr(f5.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f5_main2 = c.switch() + assert f5_main2.f_code.co_name == 'main2' + assert f3_foo.f_back is f1_bar # not running, so a loop main() main2() res = c.switch() assert res is None - assert f3.f_back is None + assert f3_foo.f_back is None def test_traceback_is_complete(self): import sys diff --git a/pypy/module/pyexpat/interp_pyexpat.py b/pypy/module/pyexpat/interp_pyexpat.py --- a/pypy/module/pyexpat/interp_pyexpat.py +++ b/pypy/module/pyexpat/interp_pyexpat.py @@ -811,7 +811,7 @@ w_encoding) if space.is_none(w_namespace_separator): - namespace_separator = 0 + namespace_separator = -1 elif space.isinstance_w(w_namespace_separator, space.w_text): separator = space.text_w(w_namespace_separator) if len(separator) == 0: @@ -834,7 +834,7 @@ elif space.is_w(w_intern, space.w_None): w_intern = None - if namespace_separator: + if namespace_separator >= 0: xmlparser = XML_ParserCreateNS( encoding, rffi.cast(rffi.CHAR, namespace_separator)) diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py --- a/pypy/module/pyexpat/test/test_parser.py +++ b/pypy/module/pyexpat/test/test_parser.py @@ -58,9 +58,9 @@ p.CharacterDataHandler = lambda s: data.append(s) encoding = encoding_arg is None and 'utf-8' or encoding_arg - res = p.Parse("\u00f6".encode(encoding), True) + res = p.Parse(u"\u00f6".encode(encoding), True) assert res == 1 - assert data == ["\u00f6"] + assert data == [u"\u00f6"] def test_get_handler(self): import pyexpat @@ -210,6 +210,34 @@ p.ParseFile(fake_reader) assert fake_reader.read_count == 4 + def test_entities(self): + import pyexpat + parser = pyexpat.ParserCreate(None, "") + + def startElement(tag, attrs): + assert tag == 'http://www.w3.org/1999/02/22-rdf-syntax-ns#RDF' + assert attrs == { + 'http://www.w3.org/XML/1998/namespacebase': + 'http://www.semanticweb.org/jiba/ontologies/2017/0/test'} + parser.StartElementHandler = startElement + parser.Parse(""" + + + + + + ]> + + + + """, True) + def test_exception(self): """ From pypy.commits at gmail.com Wed Oct 25 03:25:18 2017 From: pypy.commits at gmail.com (fijal) Date: Wed, 25 Oct 2017 00:25:18 -0700 (PDT) Subject: [pypy-commit] pypy ssl-context-share: new branch to experiment with ssl context sharing Message-ID: <59f03c5e.508d1c0a.a54fd.e995@mx.google.com> Author: fijal Branch: ssl-context-share Changeset: r92843:0531fecb7205 Date: 2017-10-25 09:24 +0200 http://bitbucket.org/pypy/pypy/changeset/0531fecb7205/ Log: new branch to experiment with ssl context sharing From pypy.commits at gmail.com Wed Oct 25 04:00:43 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 01:00:43 -0700 (PDT) Subject: [pypy-commit] buildbot default: add new macos buildbot Message-ID: <59f044ab.4dbbdf0a.cb49a.b6ba@mx.google.com> Author: Matti Picus Branch: Changeset: r1033:d495e76c5761 Date: 2017-10-25 11:00 +0300 http://bitbucket.org/pypy/buildbot/changeset/d495e76c5761/ Log: add new macos buildbot diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -510,19 +510,19 @@ "locks": [Bencher4Lock.access('exclusive')], }, {"name": MACOSX32OWN, - "slavenames": ["minime"], + "slavenames": ["minime", "billenstein-sierra"], "builddir": MACOSX32OWN, "factory": pypyOwnTestFactoryOSX32, "category": 'mac32' }, {"name": MACOSX32RPYTHON, - "slavenames": ["minime"], + "slavenames": ["minime", "billenstein-sierra"], "builddir": MACOSX32RPYTHON, "factory": pypyRPythonTestFactoryOSX32, "category": 'mac32' }, {"name" : JITMACOSX64, - "slavenames": ["rebuy-de", "tosh", "osx-10.9-x64-dw"], # "xerxes" + "slavenames": ["rebuy-de", "tosh", "osx-10.9-x64-dw", "billenstein-sierra"], # "xerxes" 'builddir' : JITMACOSX64, 'factory' : pypyJITTranslatedTestFactoryOSX64, 'category' : 'mac64', From pypy.commits at gmail.com Wed Oct 25 05:38:12 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 02:38:12 -0700 (PDT) Subject: [pypy-commit] buildbot default: allow pause, tweak slave requirements Message-ID: <59f05b84.0bbadf0a.174bd.fb89@mx.google.com> Author: Matti Picus Branch: Changeset: r1034:dbccd8b5c54f Date: 2017-10-25 12:37 +0300 http://bitbucket.org/pypy/buildbot/changeset/dbccd8b5c54f/ Log: allow pause, tweak slave requirements diff --git a/README_BUILDSLAVE b/README_BUILDSLAVE --- a/README_BUILDSLAVE +++ b/README_BUILDSLAVE @@ -1,9 +1,17 @@ How to setup a buildslave for PyPy ================================== -The recommended setup is to have a "pypy" in your path that will translate and -a "python" (cpython) in your path that will run the test suites. You will also -need a "virtualenv" in your path that will create a cpython virtualenv. +The required setup is:: +- hg +- "python" should run a cpython 2.6-2.7 that will run the test suites. +- virtualenv that will use the "python" from above +- all the build dependencies list here + http://doc.pypy.org/en/latest/build.html?highlight=dependencies#install-build-time-dependencies + +If your slave will be translating, you will also need +- pypy in your path +- at least 4GB of RAM for 32 bit, 6GB for 64 bit translations + Then you will need to install the ``buildbot-slave`` package, which will install many other packages like twised, so you may prefer to run the diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -37,7 +37,8 @@ channel = '#pypy' authz_cfg = authz.Authz(pingBuilder=False, forceBuild=True, forceAllBuilds=True, - stopBuild=True, stopAllBuilds=True, cancelPendingBuild=True) + stopBuild=True, stopAllBuilds=True, + cancelPendingBuild=True, pauseSlave=True) # xxx could maybe also say 'default_action=True' instead of all the # '*=True' in the line above status = WebStatus(httpPortNumber, authz=authz_cfg) From pypy.commits at gmail.com Wed Oct 25 08:56:51 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 05:56:51 -0700 (PDT) Subject: [pypy-commit] pypy default: fix issue #2682, split firstword on any whitespace Message-ID: <59f08a13.8faedf0a.c6fed.c39a@mx.google.com> Author: Matti Picus Branch: Changeset: r92844:7dec6a241b75 Date: 2017-10-25 15:52 +0300 http://bitbucket.org/pypy/pypy/changeset/7dec6a241b75/ Log: fix issue #2682, split firstword on any whitespace diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1027,7 +1027,7 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split(" ")[0].upper() + first_word = sql.lstrip().split()[0].upper() if first_word == "": self._type = _STMT_TYPE_INVALID elif first_word == "SELECT": diff --git a/pypy/module/test_lib_pypy/test_sqlite3.py b/pypy/module/test_lib_pypy/test_sqlite3.py --- a/pypy/module/test_lib_pypy/test_sqlite3.py +++ b/pypy/module/test_lib_pypy/test_sqlite3.py @@ -228,6 +228,14 @@ cur.execute("create table test(a)") cur.executemany("insert into test values (?)", [[1], [2], [3]]) assert cur.lastrowid is None + # issue 2682 + cur.execute('''insert + into test + values (?) + ''', (1, )) + assert cur.lastrowid is not None + cur.execute('''insert\t into test values (?) ''', (1, )) + assert cur.lastrowid is not None def test_authorizer_bad_value(self, con): def authorizer_cb(action, arg1, arg2, dbname, source): From pypy.commits at gmail.com Wed Oct 25 08:56:53 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 05:56:53 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge default into py3.5 Message-ID: <59f08a15.4db3df0a.d93cb.e551@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92845:7d0732ed45b9 Date: 2017-10-25 15:53 +0300 http://bitbucket.org/pypy/pypy/changeset/7d0732ed45b9/ Log: merge default into py3.5 diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1080,7 +1080,7 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split(" ")[0].upper() + first_word = sql.lstrip().split()[0].upper() if first_word == "": self._type = _STMT_TYPE_INVALID elif first_word == "SELECT": diff --git a/pypy/module/test_lib_pypy/test_sqlite3.py b/pypy/module/test_lib_pypy/test_sqlite3.py --- a/pypy/module/test_lib_pypy/test_sqlite3.py +++ b/pypy/module/test_lib_pypy/test_sqlite3.py @@ -228,6 +228,14 @@ cur.execute("create table test(a)") cur.executemany("insert into test values (?)", [[1], [2], [3]]) assert cur.lastrowid is None + # issue 2682 + cur.execute('''insert + into test + values (?) + ''', (1, )) + assert cur.lastrowid is not None + cur.execute('''insert\t into test values (?) ''', (1, )) + assert cur.lastrowid is not None def test_authorizer_bad_value(self, con): def authorizer_cb(action, arg1, arg2, dbname, source): From pypy.commits at gmail.com Wed Oct 25 11:40:22 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 08:40:22 -0700 (PDT) Subject: [pypy-commit] pypy default: test, fix issue #2677 copy pure c PyBuffer_{From, To}Contiguous from cpython Message-ID: <59f0b066.5c8bdf0a.cee1d.f6a6@mx.google.com> Author: Matti Picus Branch: Changeset: r92846:94ab09b73962 Date: 2017-10-25 17:50 +0300 http://bitbucket.org/pypy/pypy/changeset/94ab09b73962/ Log: test, fix issue #2677 copy pure c PyBuffer_{From,To}Contiguous from cpython diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -577,6 +577,7 @@ 'PyComplex_AsCComplex', 'PyComplex_FromCComplex', 'PyObject_AsReadBuffer', 'PyObject_AsWriteBuffer', 'PyObject_CheckReadBuffer', + 'PyBuffer_GetPointer', 'PyBuffer_ToContiguous', 'PyBuffer_FromContiguous', 'PyOS_getsig', 'PyOS_setsig', 'PyThread_get_thread_ident', 'PyThread_allocate_lock', 'PyThread_free_lock', diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -317,6 +317,31 @@ PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *, const void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_AsWriteBuffer(PyObject *, void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_CheckReadBuffer(PyObject *); +PyAPI_FUNC(void *) PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices); +/* Get the memory area pointed to by the indices for the buffer given. + Note that view->ndim is the assumed size of indices +*/ + +PyAPI_FUNC(int) PyBuffer_ToContiguous(void *buf, Py_buffer *view, + Py_ssize_t len, char fort); +PyAPI_FUNC(int) PyBuffer_FromContiguous(Py_buffer *view, void *buf, + Py_ssize_t len, char fort); +/* Copy len bytes of data from the contiguous chunk of memory + pointed to by buf into the buffer exported by obj. Return + 0 on success and return -1 and raise a PyBuffer_Error on + error (i.e. the object does not have a buffer interface or + it is not working). + + If fort is 'F' and the object is multi-dimensional, + then the data will be copied into the array in + Fortran-style (first dimension varies the fastest). If + fort is 'C', then the data will be copied into the array + in C-style (last dimension varies the fastest). If fort + is 'A', then it does not matter and the copy will be made + in whatever way is more efficient. + +*/ + #define PyObject_MALLOC PyObject_Malloc #define PyObject_REALLOC PyObject_Realloc diff --git a/pypy/module/cpyext/src/abstract.c b/pypy/module/cpyext/src/abstract.c --- a/pypy/module/cpyext/src/abstract.c +++ b/pypy/module/cpyext/src/abstract.c @@ -101,6 +101,163 @@ return 0; } +void* +PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) +{ + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; +} + +void +_Py_add_one_to_index_F(int nd, Py_ssize_t *index, const Py_ssize_t *shape) +{ + int k; + + for (k=0; k=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } +} + + /* view is not checked for consistency in either of these. It is + assumed that the size of the buffer is view->len in + view->len / view->itemsize elements. + */ + +int +PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + addone(view->ndim, indices, view->shape); + } + PyMem_Free(indices); + return 0; +} + +int +PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + addone(view->ndim, indices, view->shape); + } + + PyMem_Free(indices); + return 0; +} + + + /* Buffer C-API for Python 3.0 */ int diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py --- a/pypy/module/cpyext/test/test_memoryobject.py +++ b/pypy/module/cpyext/test/test_memoryobject.py @@ -115,7 +115,36 @@ view = PyMemoryView_GET_BUFFER(memoryview); Py_DECREF(memoryview); return PyLong_FromLong(view->len / view->itemsize); - """)]) + """), + ("test_contiguous", "METH_O", + """ + Py_buffer* view; + PyObject * memoryview; + void * buf = NULL; + int ret; + Py_ssize_t len; + memoryview = PyMemoryView_FromObject(args); + if (memoryview == NULL) + return NULL; + view = PyMemoryView_GET_BUFFER(memoryview); + Py_DECREF(memoryview); + len = view->len; + if (len == 0) + return NULL; + buf = malloc(len); + ret = PyBuffer_ToContiguous(buf, view, view->len, 'A'); + if (ret != 0) + { + free(buf); + return NULL; + } + ret = PyBuffer_FromContiguous(view, buf, view->len, 'A'); + free(buf); + if (ret != 0) + return NULL; + Py_RETURN_NONE; + """), + ]) module = self.import_module(name='buffer_test') arr = module.PyMyArray(10) ten = foo.get_len(arr) @@ -124,6 +153,7 @@ assert ten == 10 ten = foo.test_buffer(arr) assert ten == 10 + foo.test_contiguous(arr) def test_releasebuffer(self): module = self.import_extension('foo', [ From pypy.commits at gmail.com Wed Oct 25 15:12:25 2017 From: pypy.commits at gmail.com (mattip) Date: Wed, 25 Oct 2017 12:12:25 -0700 (PDT) Subject: [pypy-commit] pypy default: whoops for empty sql Message-ID: <59f0e219.13121c0a.31c1.598d@mx.google.com> Author: Matti Picus Branch: Changeset: r92847:89d1f31fabc8 Date: 2017-10-25 22:11 +0300 http://bitbucket.org/pypy/pypy/changeset/89d1f31fabc8/ Log: whoops for empty sql diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1027,21 +1027,25 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split()[0].upper() - if first_word == "": + + if sql: + first_word = sql.lstrip().split()[0].upper() + if first_word == '': + self._type = _STMT_TYPE_INVALID + if first_word == "SELECT": + self._type = _STMT_TYPE_SELECT + elif first_word == "INSERT": + self._type = _STMT_TYPE_INSERT + elif first_word == "UPDATE": + self._type = _STMT_TYPE_UPDATE + elif first_word == "DELETE": + self._type = _STMT_TYPE_DELETE + elif first_word == "REPLACE": + self._type = _STMT_TYPE_REPLACE + else: + self._type = _STMT_TYPE_OTHER + else: self._type = _STMT_TYPE_INVALID - elif first_word == "SELECT": - self._type = _STMT_TYPE_SELECT - elif first_word == "INSERT": - self._type = _STMT_TYPE_INSERT - elif first_word == "UPDATE": - self._type = _STMT_TYPE_UPDATE - elif first_word == "DELETE": - self._type = _STMT_TYPE_DELETE - elif first_word == "REPLACE": - self._type = _STMT_TYPE_REPLACE - else: - self._type = _STMT_TYPE_OTHER if isinstance(sql, unicode): sql = sql.encode('utf-8') From pypy.commits at gmail.com Thu Oct 26 05:45:01 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 02:45:01 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: write down _sliced version for unicode Message-ID: <59f1ae9d.07c41c0a.332bc.3529@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92848:e1df137224a1 Date: 2017-10-26 11:44 +0200 http://bitbucket.org/pypy/pypy/changeset/e1df137224a1/ Log: write down _sliced version for unicode diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -756,7 +756,16 @@ if start == stop: return self._empty() else: - return self._sliced(space, selfvalue, start, stop, self) + return self._unicode_sliced(space, start, stop) + + def _unicode_sliced(self, space, start, stop): + # XXX maybe some heuristic, like first slice does not create + # full index, but second does? + assert start >= 0 + assert stop >= 0 + byte_start = self._index_to_byte(start) + byte_stop = self._index_to_byte(stop) + return W_UnicodeObject(self._utf8[byte_start:byte_stop], stop - start) def descr_capitalize(self, space): value = self._utf8 From pypy.commits at gmail.com Thu Oct 26 05:56:31 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 02:56:31 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: partition/rpartition Message-ID: <59f1b14f.89ce1c0a.1afaf.41e2@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92849:b80499557864 Date: 2017-10-26 11:55 +0200 http://bitbucket.org/pypy/pypy/changeset/b80499557864/ Log: partition/rpartition diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -825,9 +825,11 @@ if pos == -1: return space.newtuple([self, self._empty(), self._empty()]) else: + lgt = rutf8.check_utf8(value, True, stop=pos) return space.newtuple( - [self._sliced(space, value, 0, pos, self), w_sub, - self._sliced(space, value, pos + sublen, len(value), self)]) + [W_UnicodeObject(value[0:pos], lgt), w_sub, + W_UnicodeObject(value[pos + len(sub._utf8):len(value)], + self._len() - lgt - sublen)]) def descr_rpartition(self, space, w_sub): value = self._utf8 @@ -841,10 +843,11 @@ if pos == -1: return space.newtuple([self._empty(), self._empty(), self]) else: + lgt = rutf8.check_utf8(value, True, stop=pos) return space.newtuple( - [self._sliced(space, value, 0, pos, self), w_sub, - self._sliced(space, value, pos + sublen, len(value), self)]) - + [W_UnicodeObject(value[0:pos], lgt), w_sub, + W_UnicodeObject(value[pos + len(sub._utf8):len(value)], + self._len() - lgt - sublen)]) @unwrap_spec(count=int) def descr_replace(self, space, w_old, w_new, count=-1): diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -291,16 +291,19 @@ #@jit.elidable -def check_utf8(s, allow_surrogates): +def check_utf8(s, allow_surrogates, start=0, stop=-1): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. If allow_surrogates is False, then also raise if we see any. Note also codepoints_in_utf8(), which also computes the length faster by assuming that 's' is valid utf-8. """ - pos = 0 + pos = start continuation_bytes = 0 - end = len(s) + if stop < 0: + end = len(s) + else: + end = stop while pos < end: ordch1 = ord(s[pos]) pos += 1 From pypy.commits at gmail.com Thu Oct 26 09:47:50 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 06:47:50 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: kill a bunch of no longer needed integrations with stringmethods Message-ID: <59f1e786.532f1c0a.19345.bb09@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92850:2137079f21c5 Date: 2017-10-26 12:05 +0200 http://bitbucket.org/pypy/pypy/changeset/2137079f21c5/ Log: kill a bunch of no longer needed integrations with stringmethods diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -113,30 +113,12 @@ "found", self._len()) return space.newint(rutf8.codepoint_at_pos(self._utf8, 0)) - def _new(self, value): - assert isinstance(value, unicode) - return W_UnicodeObject(value.encode('utf8'), len(value)) - - def _new_from_list(self, value): - u = u''.join(value) - return W_UnicodeObject(u.encode('utf8'), len(u)) - def _empty(self): return W_UnicodeObject.EMPTY def _len(self): - #if self._length == -1: - # self._length = self._compute_length() return self._length - def _compute_length(self): - return rutf8.compute_length_utf8(self._utf8) - - def _val(self, space): - #import pdb - #pdb.set_trace() - return self._utf8.decode('utf8') - @staticmethod def _use_rstr_ops(space, w_other): # Always return true because we always need to copy the other From pypy.commits at gmail.com Thu Oct 26 09:47:54 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 06:47:54 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fix rpython Message-ID: <59f1e78a.8a821c0a.82d96.69d0@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92852:5ac470b3f3bb Date: 2017-10-26 15:47 +0200 http://bitbucket.org/pypy/pypy/changeset/5ac470b3f3bb/ Log: fix rpython diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -797,7 +797,7 @@ pos = value.find(sub._utf8) - if pos == -1: + if pos < 0: return space.newtuple([self, self._empty(), self._empty()]) else: lgt = rutf8.check_utf8(value, True, stop=pos) @@ -815,7 +815,7 @@ pos = value.rfind(sub._utf8) - if pos == -1: + if pos < 0: return space.newtuple([self._empty(), self._empty(), self]) else: lgt = rutf8.check_utf8(value, True, stop=pos) From pypy.commits at gmail.com Thu Oct 26 09:47:52 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 06:47:52 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: remove decode from repr function Message-ID: <59f1e788.17f71c0a.8866e.9735@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92851:196f5e9026d4 Date: 2017-10-26 15:45 +0200 http://bitbucket.org/pypy/pypy/changeset/196f5e9026d4/ Log: remove decode from repr function diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -126,10 +126,6 @@ return True @staticmethod - def _op_val(space, w_other, strict=None): - return W_UnicodeObject.convert_arg_to_w_unicode(space, w_other, strict)._utf8.decode('utf8') - - @staticmethod def convert_arg_to_w_unicode(space, w_other, strict=None): if isinstance(w_other, W_UnicodeObject): return w_other @@ -237,10 +233,7 @@ return w_newobj def descr_repr(self, space): - chars = self._utf8.decode('utf8') - size = len(chars) - s = _repr_function(chars, size, "strict") - return space.newtext(s) + return space.newtext(_repr_function(self._utf8)) def descr_str(self, space): return encode_object(space, self, None, None) @@ -1752,5 +1745,5 @@ return ''.join(result) -_repr_function, _ = make_unicode_escape_function( - pass_printable=False, unicode_output=False, quotes=True, prefix='u') +_repr_function = rutf8.make_utf8_escape_function( + pass_printable=False, quotes=True, prefix='u') diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -16,10 +16,11 @@ """ import sys -from rpython.rlib.objectmodel import enforceargs +from rpython.rlib.objectmodel import enforceargs, we_are_translated from rpython.rlib.rstring import StringBuilder from rpython.rlib import jit from rpython.rlib.rarithmetic import r_uint, intmask +from rpython.rlib.unicodedata import unicodedb from rpython.rtyper.lltypesystem import lltype, rffi @@ -478,3 +479,100 @@ if index == 2: bytepos = next_codepoint_pos(utf8, bytepos) return codepoint_at_pos(utf8, bytepos) + +def make_utf8_escape_function(pass_printable=False, quotes=False, prefix=None): + @jit.elidable + def unicode_escape(s): + size = len(s) + result = StringBuilder(size) + + if quotes: + if prefix: + result.append(prefix) + if s.find('\'') != -1 and s.find('\"') == -1: + quote = ord('\"') + result.append('"') + else: + quote = ord('\'') + result.append('\'') + else: + quote = 0 + + if size == 0: + return '' + + pos = 0 + while pos < size: + oc = codepoint_at_pos(s, pos) + ch = s[pos] + + # Escape quotes + if quotes and (oc == quote or ch == '\\'): + result.append('\\') + next_pos = next_codepoint_pos(s, pos) + result.append_slice(s, pos, next_pos) + pos = next_pos + continue + + # The following logic is enabled only if MAXUNICODE == 0xffff, or + # for testing on top of a host Python where sys.maxunicode == 0xffff + if (not we_are_translated() and sys.maxunicode == 0xFFFF and + 0xD800 <= oc < 0xDC00 and pos + 3 < size): + # Map UTF-16 surrogate pairs to Unicode \UXXXXXXXX escapes + pos += 3 + oc2 = codepoint_at_pos(s, pos) + + if 0xDC00 <= oc2 <= 0xDFFF: + ucs = (((oc & 0x03FF) << 10) | (oc2 & 0x03FF)) + 0x00010000 + char_escape_helper(result, ucs) + pos += 3 + continue + # Fall through: isolated surrogates are copied as-is + pos -= 3 + + # Map special whitespace to '\t', \n', '\r' + if ch == '\t': + result.append('\\t') + elif ch == '\n': + result.append('\\n') + elif ch == '\r': + result.append('\\r') + elif ch == '\\': + result.append('\\\\') + + # Map non-printable or non-ascii to '\xhh' or '\uhhhh' + elif pass_printable and not (oc <= 0x10ffff and unicodedb.isprintable(oc)): + char_escape_helper(result, oc) + elif not pass_printable and (oc < 32 or oc >= 0x7F): + char_escape_helper(result, oc) + + # Copy everything else as-is + else: + if oc < 128: + result.append(ch) + else: + next_pos = next_codepoint_pos(s, pos) + result.append_slice(s, pos, next_pos) + pos = next_codepoint_pos(s, pos) + + if quotes: + result.append(chr(quote)) + return result.build() + + TABLE = '0123456789abcdef' + + def char_escape_helper(result, char): + if char >= 0x10000 or char < 0: + result.append("\\U") + zeros = 8 + elif char >= 0x100: + result.append("\\u") + zeros = 4 + else: + result.append("\\x") + zeros = 2 + for i in range(zeros-1, -1, -1): + result.append(TABLE[(char >> (4 * i)) & 0x0f]) + + return unicode_escape #, char_escape_helper + diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py --- a/rpython/rlib/test/test_rutf8.py +++ b/rpython/rlib/test/test_rutf8.py @@ -114,6 +114,13 @@ assert (rutf8.codepoint_position_at_index(u.encode('utf8'), index, i) == len(u[:i].encode('utf8'))) +repr_func = rutf8.make_utf8_escape_function(prefix='u', pass_printable=False, + quotes=True) + + at given(strategies.text()) +def test_repr(u): + assert repr(u) == repr_func(u.encode('utf8')) + @given(strategies.lists(strategies.characters())) def test_surrogate_in_utf8(unichars): uni = u''.join(unichars).encode('utf-8') From pypy.commits at gmail.com Thu Oct 26 12:00:51 2017 From: pypy.commits at gmail.com (mattip) Date: Thu, 26 Oct 2017 09:00:51 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-mac-embedding: typo Message-ID: <59f206b3.57abdf0a.c021f.b838@mx.google.com> Author: Matti Picus Branch: py3.5-mac-embedding Changeset: r92853:8d9cebe3885b Date: 2017-10-26 18:31 +0300 http://bitbucket.org/pypy/pypy/changeset/8d9cebe3885b/ Log: typo diff --git a/pypy/goal/targetpypystandalone.py b/pypy/goal/targetpypystandalone.py --- a/pypy/goal/targetpypystandalone.py +++ b/pypy/goal/targetpypystandalone.py @@ -364,7 +364,7 @@ def task_build_cffi_imports(self): ''' Use cffi to compile cffi interfaces to modules''' filename = os.path.join(pypydir, 'tool', 'build_cffi_imports.py') - if system.platform == 'darwin': + if sys.platform == 'darwin': argv = [filename, '--embed-dependencies'] else: argv = [filename,] From pypy.commits at gmail.com Thu Oct 26 12:10:31 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 26 Oct 2017 09:10:31 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: finish the draft Message-ID: <59f208f7.13121c0a.457bd.9b2b@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5843:eaef98c9e980 Date: 2017-10-18 01:35 +0200 http://bitbucket.org/pypy/extradoc/changeset/eaef98c9e980/ Log: finish the draft diff --git a/blog/draft/2017-10-cape-town-report.rst b/blog/draft/2017-10-cape-town-report.rst --- a/blog/draft/2017-10-cape-town-report.rst +++ b/blog/draft/2017-10-cape-town-report.rst @@ -3,41 +3,63 @@ Hello from the other side of the world (for most of you)! -With the excuse of coming to PyCon ZA[1], during the last two weeks Armin, -Ronan, Antonio and sometimes Maciek had a very nice and productive sprint in Cape Town, -as pictures show :). +With the excuse of coming to `PyCon ZA`_ during the last two weeks Armin, +Ronan, Antonio and sometimes Maciek had a very nice and productive sprint in +Cape Town, as pictures show :). We would like to say a big thank you to +Kiwi.com, which sponsored part of the travel costs via its awesome Sourcelift_ +program to help Open Source projects. -XXX thank kiwi https://www.kiwi.com/sourcelift/ +.. _`PyCon ZA`: https://za.pycon.org/ +.. _Sourcelift: https://www.kiwi.com/sourcelift/ XXX insert pic -We spent most of the time hacking at cpyext, our CPython C-API compatibility -layer: during the last years, the focus was to make it working and compatible -with CPython, in order to run existing libraries such as numpy and -pandas. However, we never paid too much attention to performance, so the net -result is that with the latest released version of PyPy, C extensions -generally work but their speed ranges from "slow" to "horribly slow". +Armin, Ronan and Anto spent most of the time hacking at cpyext, our CPython +C-API compatibility layer: during the last years, the focus was to make it +working and compatible with CPython, in order to run existing libraries such +as numpy and pandas. However, we never paid too much attention to performance, +so the net result is that with the latest released version of PyPy, C +extensions generally work but their speed ranges from "slow" to "horribly +slow". -For example, these `very simple micro-benchmarks`_ measure the speed of +For example, these very simple microbenchmarks_ measure the speed of calling (empty) C functions, i.e. the time you spend to "cross the border" between RPython and C. These are the results on CPython, on PyPy 5.9, and on our newest in-progress version:: + + $ python bench.py # CPython + noargs : 0.51 secs + onearg : 0.45 secs + varargs: 0.60 secs + + $ pypy-5.8 bench.py # PyPy 5.8 + noargs : 1.11 secs + onearg : 1.83 secs + varargs: 3.50 secs + + $ pypy bench.py # cpyext-refactor-methodobject branch + noargs : 0.31 secs + onearg : 0.37 secs + varargs: 0.49 secs + + +.. _microbenchmarks: https://github.com/antocuni/cpyext-benchmarks - bla bla bla +So yes: before the sprint, we were ~2-6x slower than CPython. Now, we are +**faster** than it! - -.. origin git at github.com:antocuni/cpyext-benchmarks.git (fetch) - - To reach this result, we did various improvements, such as: 1. teach the JIT how to look (a bit) inside the cpyext module 2. write specialized code for calling ``METH_NOARGS``, ``METH_O`` and ``METH_VARARGS`` functions; previously, we always used a very general and - slow logic + slow logic; - 3. the `cpyext-avoid-roundtrip`_ branch: crossing the RPython/C border is + 3. we implemented freelists to allocate the cpyext versions of ``int`` and + ``tuple`` objects, as CPython does. + + 4. the `cpyext-avoid-roundtrip`_ branch: crossing the RPython/C border is slowish, but the real problem was (and still is for many cases) we often cross it many times for no good reason. So, depending on the actual API call, you might end up in the C land, which calls back into the RPython @@ -45,14 +67,31 @@ The branch tries to fix such nonsense: so far, we fixed only some cases, which are enough to speed up the benchmarks shown above. But most importantly, we -now have a clear path to improve cpyext more and more. +now have a clear path and an actual plan to improve cpyext more and +more. Ideally, we would like to reach a point in which cpyext-intensive +programs run at worst at the same speed of CPython. -For example, the ``METH_VARARGS`` case is still a bit slowish because we need -to ``malloc()`` a new cpyext tuple for every call: the plan is to use the same -strategy as CPython, i.e. to use free lists to be able to quickly reuse the -memory as soon as the tuples die. +Among the other things, Armin and Maciej did a lot of work on the +`unicode-utf8`_ branch: the goal of the branch is to always use UTF-8 as the +internal representation of unicode strings. The advantages are various: -bla bla bla + - decoding an UTF-8 stream is super fast, as you just need to check that the + stream is valid; + - encoding to UTF-8 is a no-op; + - in most cases, UTF-8 is a more compact representation than the currently + used UCS-4. +Before you ask: yes, this branch contains special logic to ensure that random +access of single unicode chars is still O(1), as it is on both CPython and the +current PyPy. + +In summary, this was a long and profitable sprint, in which we achieved lots +of interesting results. However, what we liked even more was the privilege of +doing commits_ from awesome places such as the top of Table Mountain: + +.. _commits: https://bitbucket.org/pypy/pypy/commits/a4307fb5912e + +XXX: embed this tweet +https://twitter.com/ronanlamy/status/915575026107240449 From pypy.commits at gmail.com Thu Oct 26 12:10:33 2017 From: pypy.commits at gmail.com (antocuni) Date: Thu, 26 Oct 2017 09:10:33 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: draft of a new blog post Message-ID: <59f208f9.06b6df0a.1db93.b809@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5844:fc3e9ca8c4df Date: 2017-10-26 18:10 +0200 http://bitbucket.org/pypy/extradoc/changeset/fc3e9ca8c4df/ Log: draft of a new blog post diff --git a/blog/draft/2017-10-how-to-make-50x-faster.rst b/blog/draft/2017-10-how-to-make-50x-faster.rst new file mode 100644 --- /dev/null +++ b/blog/draft/2017-10-how-to-make-50x-faster.rst @@ -0,0 +1,238 @@ +How to make your code 80 times faster +====================================== + +I often hear people who are happy because PyPy makes their code 2 times faster +or so. Here is a short personal story which shows that PyPy can go well beyond +that. + +**DISCLAIMER**: this is not a silver bullet or a general recipe: it worked in +this particular case, it might not work so well in other cases. But I think it +is still an interesting technique. Moreover, the various steps and +implementations are showed in the same order as I tried them during the +development, so it is a real-life example of how to proceed when optimizing +for PyPy. + +Some months ago I `played a bit`_ with evolutionary algorithms: the ambitious +plan was to automatically evolve a logic which could control a (simulated) +quadcopter, i.e. a `PID controller`_ (**spoiler**: it doesn't fly). + +.. _`played a bit`: https://github.com/antocuni/evolvingcopter +.. _`PID controller`: https://en.wikipedia.org/wiki/PID_controller + +The idea is to have an initial population of random creatures: at each +generation, the ones with the best fitness survive and reproduce with small, +random variations. + +However, for the scope of this post, the actual task at hand is not so +important, so let's jump straight to the code. To drive the quadcopter, a +``Creature`` has a ``run_step`` method which runs at each delta-t (`full +code`_):: + + class Creature(object): + INPUTS = 2 # z_setpoint, current z position + OUTPUTS = 1 # PWM for all 4 motors + STATE_VARS = 1 + ... + + def run_step(self, inputs): + # state: [state_vars ... inputs] + # out_values: [state_vars, ... outputs] + self.state[self.STATE_VARS:] = inputs + out_values = np.dot(self.matrix, self.state) + self.constant + self.state[:self.STATE_VARS] = out_values[:self.STATE_VARS] + outputs = out_values[self.STATE_VARS:] + return outputs + +- ``inputs`` is a numpy array containing the desired setpoint and the current + position on the Z axis; + +- ``outputs`` is a numpy array containing the thrust to give to the motors. To + start easy, all the 4 motors are constrained to have the same thrust, so + that the quadcopter only travels up and down the Z axis; + +- ``self.state`` contains an arbitrary amount of values which are passed from + one step to the next; + +- ``self.matrix`` and ``self.constant`` contains the actual logic: by putting + the "right" values there, in theory we could get a perfectly tuned PID + controller. These are randomly mutated between generations. + +.. _`full code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py + +``run_step`` is run at 100Hz (in the virtual time of the simulation). At each +generation, we test 500 creatures for a total of 12 virtual seconds each. So, +we have a total of 600,000 executions of ``run_step`` at each generation. + +At first, I simply tried to run this code on CPython; here is the result:: + + $ python -m ev.main + Generation 1: ... [population = 500] [12.06 secs] + Generation 2: ... [population = 500] [6.13 secs] + Generation 3: ... [population = 500] [6.11 secs] + Generation 4: ... [population = 500] [6.09 secs] + Generation 5: ... [population = 500] [6.18 secs] + Generation 6: ... [population = 500] [6.26 secs] + +Which means ~6.15 seconds/generation, excluding the first. + +Then I tried with PyPy 5.9:: + + $ pypy -m ev.main + Generation 1: ... [population = 500] [63.90 secs] + Generation 2: ... [population = 500] [33.92 secs] + Generation 3: ... [population = 500] [34.21 secs] + Generation 4: ... [population = 500] [33.75 secs] + +Ouch! We are ~5.5x slower than CPython. This was kind of expected: numpy is +based on cpyext, which is infamously slow (although `we are working on that`_). + +So, let's try to avoid cpyext. The first obvious step is to use numpypy +instead of numpy; for more info about numpypy, see the next section. In the +meantime, here is a link to the hack_ which enables for PyPy. Let's see if the +speed improves:: + + $ pypy -m ev.main # using numpypy + Generation 1: ... [population = 500] [5.60 secs] + Generation 2: ... [population = 500] [2.90 secs] + Generation 3: ... [population = 500] [2.78 secs] + Generation 4: ... [population = 500] [2.69 secs] + Generation 5: ... [population = 500] [2.72 secs] + Generation 6: ... [population = 500] [2.73 secs] + +So, ~2.7 seconds on average: this is 12x faster than PyPy+numpy, and more than +2x faster than the original CPython. At this point, most people would be happy +and go tweeting how good is PyPy. + +.. _`we are working on that`: https://morepypy.blogspot.it/2017/10/cape-of-good-hope-for-pypy-hello-from.html +.. _hack: https://github.com/antocuni/evolvingcopter/blob/master/ev/pypycompat.py + +In general, when talking of CPython vs PyPy, I am rarely satified of a 2x +speedup: I know that PyPy can do much better than this, especially if you +write code which is specifically optimized for the JIT. For a real-life +example, have a look at `capnpy benchmarks`_, in which the PyPy version is +~15x faster than the heavily optimized CPython+Cython version (both have been +written by me, and I tried hard to write the fastest code for both +implementations). + +.. _`capnpy benchmarks`: http://capnpy.readthedocs.io/en/latest/benchmarks.html + +So, let's try to do better. As usual, the first thing to do is to profile and +see where we spend most of the time. Here is the `vmprof profile`_. We spend a +lot of time inside the internals of numpypy, and to allocate tons of temporary +arrays to store the results of the various operations. + +Also, let's look at the `jit traces`_ and search for the function ``run``: +this is loop in which we spend most of the time, and it is composed by a total +of 1796 operations. The operations emitted for the line ``np.dot(...) + +self.constant`` are listed between lines 1217 and 1456; 239 low level +operations are a lot: if we look at them, we can see for example that there is +a call to the RPython function `descr_dot`_, at line 1232. But there are also +calls to ``raw_malloc``, at line 1295, which allocates the space to store the +result of ``... + self.constant``. + +.. _`vmprof profile`: http://vmprof.com/#/449ca8ee-3ab2-49d4-b6f0-9099987e9000 +.. _`jit traces`: http://vmprof.com/#/28fd6e8f-f103-4bf4-a76a-4b65dbd637f4/traces +.. _`descr_dot`: https://bitbucket.org/pypy/pypy/src/89d1f31fabc86778cfaa1034b1102887c063de66/pypy/module/micronumpy/ndarray.py?at=default&fileviewer=file-view-default#ndarray.py-1168 + +All of this is very suboptimal: in this particular case, we know that the +shape of ``self.matrix`` is always ``(3, 2)``: so, we are doing an incredible +amount of work and ``malloc()``ing a temporary array just to call an RPython +function which ultimately does a total of 6 multiplications and 8 additions. + +One possible solution to this nonsense is a well known compiler optimization: +loop unrolling. From the compiler point of view, unrolling the loop is always +risky because if the matrix is too big you might end up emitting a huge blob +of code, possibly uselss if the shape of the matrices change frequently: this +is the main reason why the PyPy JIT does not even try to do it in this case. + +However, we **know** that the matrix is small, and always of the same +shape. So, let's unroll the loop manually:: + + class SpecializedCreature(Creature): + + def __init__(self, *args, **kwargs): + Creature.__init__(self, *args, **kwargs) + # store the data in a plain Python list, which pypy is able to + # optimize as a float array + self.data = list(self.matrix.ravel()) + list(self.constant) + self.data_state = [0.0] + assert self.matrix.shape == (2, 3) + assert len(self.data) == 8 + + def run_step(self, inputs): + # state: [state_vars ... inputs] + # out_values: [state_vars, ... outputs] + k0, k1, k2, q0, q1, q2, c0, c1 = self.data + s0 = self.data_state[0] + z_sp, z = inputs + # + # compute the output + out0 = s0*k0 + z_sp*k1 + z*k2 + c0 + out1 = s0*q0 + z_sp*q1 + z*q2 + c1 + # + self.data_state[0] = out0 + outputs = [out1] + return outputs + +In the `actual code`_ there is also a sanity check which asserts that the +computed output is the very same as the one returned by ``Creature.run_step``. + +Note that is code is particularly PyPy-friendly, because ``self.data`` is a +simple list of floats: thanks to list strategies, it is internally represented +as a flat array of C doubles, i.e. very fast and compact. + +.. _`actual code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py#L100 +.. _`list strategies`: https://morepypy.blogspot.it/2011/10/more-compact-lists-with-list-strategies.html + +So, let's try to see how it performs. First, with CPython:: + + $ python -m ev.main + Generation 1: ... [population = 500] [7.61 secs] + Generation 2: ... [population = 500] [3.96 secs] + Generation 3: ... [population = 500] [3.79 secs] + Generation 4: ... [population = 500] [3.74 secs] + Generation 5: ... [population = 500] [3.84 secs] + Generation 6: ... [population = 500] [3.69 secs] + +This looks good: 60% faster than the original CPython+numpy +implementation. Let's try on PyPy:: + + Generation 1: ... [population = 500] [0.39 secs] + Generation 2: ... [population = 500] [0.10 secs] + Generation 3: ... [population = 500] [0.11 secs] + Generation 4: ... [population = 500] [0.09 secs] + Generation 5: ... [population = 500] [0.08 secs] + Generation 6: ... [population = 500] [0.12 secs] + Generation 7: ... [population = 500] [0.09 secs] + Generation 8: ... [population = 500] [0.08 secs] + Generation 9: ... [population = 500] [0.08 secs] + Generation 10: ... [population = 500] [0.08 secs] + Generation 11: ... [population = 500] [0.08 secs] + Generation 12: ... [population = 500] [0.07 secs] + Generation 13: ... [population = 500] [0.07 secs] + Generation 14: ... [population = 500] [0.08 secs] + Generation 15: ... [population = 500] [0.07 secs] + +Yes, it's not an error. After a couple of generations, it stabilizes at around +~0.07-0.08 seconds per generation. This is around **80 (eighty) times faster** +than the original CPython+numpy implementation, and around 35-40x faster than +the naive PyPy+numpypy one. + +Let's look at the trace_ again: it no longer contains expensive calls, and +certainly no more temporary ``malloc()``s: the core of the logic is between +lines XX-YY, where we can see that it does fast C-level multiplications and +additions. + +.. _trace: http://vmprof.com/#/402af746-2966-4403-a61d-93015abac033/traces + +As I said before, this is a very particular example, and the techniques +described here do not always apply: it is not realistic to expect an 80x +speedup, unfortunately. However, it clearly shows the potential of PyPy when +it comes to high-speed computing. And most importantly, it's not a toy +benchmark which was designed specifically to have good performance on PyPy: +it's a real world example, albeit small. + +Numpy vs numpypy +----------------- + +bla bla From pypy.commits at gmail.com Thu Oct 26 12:35:44 2017 From: pypy.commits at gmail.com (mattip) Date: Thu, 26 Oct 2017 09:35:44 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: edit a bit Message-ID: <59f20ee0.d1b8df0a.ddbfa.1626@mx.google.com> Author: Matti Picus Branch: extradoc Changeset: r5845:c149e5d02b88 Date: 2017-10-26 19:35 +0300 http://bitbucket.org/pypy/extradoc/changeset/c149e5d02b88/ Log: edit a bit diff --git a/blog/draft/2017-10-how-to-make-50x-faster.rst b/blog/draft/2017-10-how-to-make-50x-faster.rst --- a/blog/draft/2017-10-how-to-make-50x-faster.rst +++ b/blog/draft/2017-10-how-to-make-50x-faster.rst @@ -2,7 +2,7 @@ ====================================== I often hear people who are happy because PyPy makes their code 2 times faster -or so. Here is a short personal story which shows that PyPy can go well beyond +or so. Here is a short personal story which shows PyPy can go well beyond that. **DISCLAIMER**: this is not a silver bullet or a general recipe: it worked in @@ -25,7 +25,7 @@ However, for the scope of this post, the actual task at hand is not so important, so let's jump straight to the code. To drive the quadcopter, a -``Creature`` has a ``run_step`` method which runs at each delta-t (`full +``Creature`` has a ``run_step`` method which runs at each ``delta - t`` (`full code`_):: class Creature(object): @@ -50,16 +50,16 @@ start easy, all the 4 motors are constrained to have the same thrust, so that the quadcopter only travels up and down the Z axis; -- ``self.state`` contains an arbitrary amount of values which are passed from +- ``self.state`` contains arbitrary values of unknown size which are passed from one step to the next; -- ``self.matrix`` and ``self.constant`` contains the actual logic: by putting +- ``self.matrix`` and ``self.constant`` contains the actual logic. By putting the "right" values there, in theory we could get a perfectly tuned PID controller. These are randomly mutated between generations. .. _`full code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py -``run_step`` is run at 100Hz (in the virtual time of the simulation). At each +``run_step`` is run at 100Hz (in the virtual time frame of the simulation). At each generation, we test 500 creatures for a total of 12 virtual seconds each. So, we have a total of 600,000 executions of ``run_step`` at each generation. @@ -101,7 +101,7 @@ So, ~2.7 seconds on average: this is 12x faster than PyPy+numpy, and more than 2x faster than the original CPython. At this point, most people would be happy -and go tweeting how good is PyPy. +and go tweeting how PyPy is great. .. _`we are working on that`: https://morepypy.blogspot.it/2017/10/cape-of-good-hope-for-pypy-hello-from.html .. _hack: https://github.com/antocuni/evolvingcopter/blob/master/ev/pypycompat.py @@ -118,14 +118,14 @@ So, let's try to do better. As usual, the first thing to do is to profile and see where we spend most of the time. Here is the `vmprof profile`_. We spend a -lot of time inside the internals of numpypy, and to allocate tons of temporary +lot of time inside the internals of numpypy, and allocating tons of temporary arrays to store the results of the various operations. Also, let's look at the `jit traces`_ and search for the function ``run``: -this is loop in which we spend most of the time, and it is composed by a total +this is loop in which we spend most of the time, and it is composed of 1796 operations. The operations emitted for the line ``np.dot(...) + self.constant`` are listed between lines 1217 and 1456; 239 low level -operations are a lot: if we look at them, we can see for example that there is +operations are a lot. If we look at them, we can see for example that there is a call to the RPython function `descr_dot`_, at line 1232. But there are also calls to ``raw_malloc``, at line 1295, which allocates the space to store the result of ``... + self.constant``. @@ -136,7 +136,7 @@ All of this is very suboptimal: in this particular case, we know that the shape of ``self.matrix`` is always ``(3, 2)``: so, we are doing an incredible -amount of work and ``malloc()``ing a temporary array just to call an RPython +amount of work. We also use ``malloc()`` to create a temporary array just to call an RPython function which ultimately does a total of 6 multiplications and 8 additions. One possible solution to this nonsense is a well known compiler optimization: @@ -177,8 +177,8 @@ In the `actual code`_ there is also a sanity check which asserts that the computed output is the very same as the one returned by ``Creature.run_step``. -Note that is code is particularly PyPy-friendly, because ``self.data`` is a -simple list of floats: thanks to list strategies, it is internally represented +Note that is code is particularly PyPy-friendly. Thanks to PyPy's `list strategies`_ +optimizations, ``self.data`` as a simple list of floats is internally represented as a flat array of C doubles, i.e. very fast and compact. .. _`actual code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py#L100 @@ -219,7 +219,7 @@ the naive PyPy+numpypy one. Let's look at the trace_ again: it no longer contains expensive calls, and -certainly no more temporary ``malloc()``s: the core of the logic is between +certainly no more temporary ``malloc()`` s. The core of the logic is between lines XX-YY, where we can see that it does fast C-level multiplications and additions. @@ -235,4 +235,15 @@ Numpy vs numpypy ----------------- -bla bla +Way back in 2011, the PyPy team `started to reimplement`_ NumPy in PyPy. +It has two pieces: the micronumpy RPython module that roughly covers the +multiarray numpy module, and a fork of the python-code called numpypy. +Over the years the project slowly matured, eventually it was able to call +out to the LAPACK and BLAS libraries to speed matrix calculations just like +NumPy, and reached around an 80% parity with the upstream project. But 80% +is far from 100%, and once the cpyext layer of PyPy matured to the point it +could pass 99.9% of the NumPy test suite, we no longer recommend using numpypy. + +XXX more needed? + +.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html From pypy.commits at gmail.com Thu Oct 26 12:49:22 2017 From: pypy.commits at gmail.com (stian) Date: Thu, 26 Oct 2017 09:49:22 -0700 (PDT) Subject: [pypy-commit] pypy math-improvements: uint128_t test and a tiny optimalization Message-ID: <59f21212.85891c0a.91a5.9ea2@mx.google.com> Author: stian Branch: math-improvements Changeset: r92854:12d7e0578291 Date: 2017-10-26 18:48 +0200 http://bitbucket.org/pypy/pypy/changeset/12d7e0578291/ Log: uint128_t test and a tiny optimalization diff --git a/rpython/rlib/rbigint.py b/rpython/rlib/rbigint.py --- a/rpython/rlib/rbigint.py +++ b/rpython/rlib/rbigint.py @@ -849,6 +849,7 @@ else: # Perform size = self.numdigits() - 1 + if size > 0: rem = self.widedigit(size) size -= 1 @@ -856,7 +857,7 @@ rem = ((rem << SHIFT) | self.digit(size)) % digit size -= 1 else: - rem = self.digit(0) % digit + rem = self.widedigit(0) % digit if rem == 0: return NULLRBIGINT @@ -2091,7 +2092,7 @@ * result in z[0:m], and return the d bits shifted out of the top. """ - carry = 0 + carry = _unsigned_widen_digit(0) assert 0 <= d and d < SHIFT i = 0 while i < m: diff --git a/rpython/translator/c/test/test_typed.py b/rpython/translator/c/test/test_typed.py --- a/rpython/translator/c/test/test_typed.py +++ b/rpython/translator/c/test/test_typed.py @@ -955,6 +955,20 @@ res = f(217) assert res == 305123851 + def test_uint128(self): + if not hasattr(rffi, '__UINT128_T'): + py.test.skip("no '__uint128_t'") + def func(n): + x = rffi.cast(getattr(rffi, '__UINT128_T'), n) + x *= x + x *= x + x *= x + x *= x + return intmask(x >> 96) + f = self.getcompiled(func, [int]) + res = f(217) + assert res == 305123851 + def test_bool_2(self): def func(n): x = rffi.cast(lltype.Bool, n) From pypy.commits at gmail.com Thu Oct 26 12:57:18 2017 From: pypy.commits at gmail.com (mattip) Date: Thu, 26 Oct 2017 09:57:18 -0700 (PDT) Subject: [pypy-commit] buildbot default: move more rpython builds to filtered schedulers Message-ID: <59f213ee.131f1c0a.e5e42.7e55@mx.google.com> Author: Matti Picus Branch: Changeset: r1035:2336f92eb562 Date: 2017-10-26 19:56 +0300 http://bitbucket.org/pypy/buildbot/changeset/2336f92eb562/ Log: move more rpython builds to filtered schedulers diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -313,6 +313,7 @@ LINUX32RPYTHON, # on tannit32, uses all cores LINUX64RPYTHON, # on bencher4, uses all cores WIN32RPYTHON, # on allegro_win32, SalsaSalsa + LINUX_S390XRPYTHON, ], branch='default', hour=0, minute=0, onlyIfChanged=True, fileIsImportant=isRPython, change_filter=filter.ChangeFilter(branch='default'), @@ -338,19 +339,25 @@ Nightly("nightly-3-00-py3.5", [ LINUX32OWN, # on tannit32, uses all cores - LINUX32RPYTHON, # on tannit32, uses all cores JITLINUX32, # on tannit32, uses 1 core LINUX64OWN, # on bencher4, uses all cores - LINUX64RPYTHON, # on bencher4, uses all cores JITLINUX64, # on bencher4, uses 1 core JITMACOSX64, # on xerxes JITWIN32, # on allegro_win32, SalsaSalsa ], branch="py3.5", hour=3, minute=0), + # XXX disable this if you are SURE py3.5 and default rpython are identical + Nightly("nightly-3-01-py3.5", [ + LINUX32RPYTHON, # on tannit32, uses all cores + LINUX64RPYTHON, # on bencher4, uses all cores + ], branch="py3.5", hour=3, minute=0, onlyIfChanged=True, + fileIsImportant=isRPython, + change_filter=filter.ChangeFilter(branch='py3.5'), + ), + # S390X vm (ibm-research) Nightly("nightly-4-00", [ LINUX_S390XOWN, - LINUX_S390XRPYTHON, ], branch='default', hour=0, minute=0), Nightly("nightly-4-01", [JITLINUX_S390X], branch='default', hour=2, minute=0), @@ -511,13 +518,13 @@ "locks": [Bencher4Lock.access('exclusive')], }, {"name": MACOSX32OWN, - "slavenames": ["minime", "billenstein-sierra"], + "slavenames": ["minime"], "builddir": MACOSX32OWN, "factory": pypyOwnTestFactoryOSX32, "category": 'mac32' }, {"name": MACOSX32RPYTHON, - "slavenames": ["minime", "billenstein-sierra"], + "slavenames": ["minime"], "builddir": MACOSX32RPYTHON, "factory": pypyRPythonTestFactoryOSX32, "category": 'mac32' From pypy.commits at gmail.com Thu Oct 26 14:11:41 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 11:11:41 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: improve the slice tests and fix it Message-ID: <59f2255d.46811c0a.6e9b9.d6d2@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92855:84d1ebd9002d Date: 2017-10-26 20:11 +0200 http://bitbucket.org/pypy/pypy/changeset/84d1ebd9002d/ Log: improve the slice tests and fix it diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -907,16 +907,31 @@ def test_getslice(self): assert u'123456'.__getslice__(1, 5) == u'2345' - s = u"abc" - assert s[:] == "abc" - assert s[1:] == "bc" - assert s[:2] == "ab" - assert s[1:2] == "b" - assert s[-2:] == "bc" - assert s[:-1] == "ab" - assert s[-2:2] == "b" - assert s[1:-1] == "b" - assert s[-2:-1] == "b" + s = u"\u0105b\u0107" + assert s[:] == u"\u0105b\u0107" + assert s[1:] == u"b\u0107" + assert s[:2] == u"\u0105b" + assert s[1:2] == u"b" + assert s[-2:] == u"b\u0107" + assert s[:-1] == u"\u0105b" + assert s[-2:2] == u"b" + assert s[1:-1] == u"b" + assert s[-2:-1] == u"b" + + def test_getitem_slice(self): + assert u'123456'.__getitem__(slice(1, 5)) == u'2345' + s = u"\u0105b\u0107" + assert s[slice(3)] == u"\u0105b\u0107" + assert s[slice(1, 3)] == u"b\u0107" + assert s[slice(2)] == u"\u0105b" + assert s[slice(1,2)] == u"b" + assert s[slice(-2,3)] == u"b\u0107" + assert s[slice(-1)] == u"\u0105b" + assert s[slice(-2,2)] == u"b" + assert s[slice(1,-1)] == u"b" + assert s[slice(-2,-1)] == u"b" + assert u"abcde"[::2] == u"ace" + assert u"\u0105\u0106\u0107abcd"[::2] == u"\u0105\u0107bd" def test_no_len_on_str_iter(self): iterable = u"hello" diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -21,7 +21,7 @@ from pypy.objspace.std import newformat from pypy.objspace.std.basestringtype import basestring_typedef from pypy.objspace.std.formatting import mod_format -from pypy.objspace.std.sliceobject import ( +from pypy.objspace.std.sliceobject import (W_SliceObject, unwrap_start_stop, normalize_simple_slice) from pypy.objspace.std.stringmethods import StringMethods from pypy.objspace.std.util import IDTAG_SPECIAL, IDTAG_SHIFT @@ -724,8 +724,36 @@ return space.newlist_utf8(res) + def descr_getitem(self, space, w_index): + if isinstance(w_index, W_SliceObject): + length = self._len() + start, stop, step, sl = w_index.indices4(space, length) + if sl == 0: + return self._empty() + elif step == 1: + assert start >= 0 and stop >= 0 + return self._unicode_sliced(space, start, stop) + else: + return self._getitem_slice_slowpath(space, start, step, sl) + + index = space.getindex_w(w_index, space.w_IndexError, "string index") + return self._getitem_result(space, index) + + def _getitem_slice_slowpath(self, space, start, step, sl): + # XXX same comment as in _unicode_sliced + builder = StringBuilder(step * sl) + byte_pos = self._index_to_byte(start) + i = 0 + while True: + next_pos = rutf8.next_codepoint_pos(self._utf8, byte_pos) + builder.append(self._utf8[byte_pos:next_pos]) + if i == sl - 1: + break + i += 1 + byte_pos = self._index_to_byte(start + i * step) + return W_UnicodeObject(builder.build(), sl) + def descr_getslice(self, space, w_start, w_stop): - selfvalue = self._utf8 start, stop = normalize_simple_slice( space, self._len(), w_start, w_stop) if start == stop: From pypy.commits at gmail.com Thu Oct 26 14:21:59 2017 From: pypy.commits at gmail.com (fijal) Date: Thu, 26 Oct 2017 11:21:59 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: implement lower Message-ID: <59f227c7.89ce1c0a.16e7.cd6f@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92856:e5017df1fbdd Date: 2017-10-26 20:21 +0200 http://bitbucket.org/pypy/pypy/changeset/e5017df1fbdd/ Log: implement lower diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -323,6 +323,16 @@ assert not u'\u01c5abc'.islower() assert not u'\u01c5ABC'.isupper() + def test_lower_upper(self): + assert u'a'.lower() == u'a' + assert u'A'.lower() == u'a' + assert u'\u0105'.lower() == u'\u0105' + assert u'\u0104'.lower() == u'\u0105' + assert u'a'.upper() == u'A' + assert u'A'.upper() == u'A' + assert u'\u0105'.upper() == u'\u0104' + assert u'\u0104'.upper() == u'\u0104' + def test_capitalize(self): assert u"brown fox".capitalize() == u"Brown fox" assert u' hello '.capitalize() == u' hello ' diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -182,15 +182,6 @@ def _islinebreak(self, s, pos): return rutf8.islinebreak(s, pos) - def _upper(self, ch): - return unichr(unicodedb.toupper(ord(ch))) - - def _lower(self, ch): - return unichr(unicodedb.tolower(ord(ch))) - - def _title(self, ch): - return unichr(unicodedb.totitle(ord(ch))) - def _newlist_unwrapped(self, space, lst): assert False, "should not be called" return space.newlist_unicode(lst) @@ -510,6 +501,15 @@ tformat = unicode_template_formatter(space, space.unicode_w(self)) return tformat.formatter_field_name_split() + def descr_lower(self, space): + builder = StringBuilder(len(self._utf8)) + pos = 0 + while pos < len(self._utf8): + lower = unicodedb.tolower(rutf8.codepoint_at_pos(self._utf8, pos)) + rutf8.unichr_as_utf8_append(builder, lower) # XXX allow surrogates? + pos = rutf8.next_codepoint_pos(self._utf8, pos) + return W_UnicodeObject(builder.build(), self._len()) + def descr_isdecimal(self, space): return self._is_generic(space, '_isdecimal') From pypy.commits at gmail.com Fri Oct 27 08:29:49 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 05:29:49 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: rename... it's actually 80x faster, not just 50x :) Message-ID: <59f326bd.c78c1c0a.79475.18da@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5846:afd53b75cebe Date: 2017-10-27 12:00 +0200 http://bitbucket.org/pypy/extradoc/changeset/afd53b75cebe/ Log: rename... it's actually 80x faster, not just 50x :) diff --git a/blog/draft/2017-10-how-to-make-50x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst rename from blog/draft/2017-10-how-to-make-50x-faster.rst rename to blog/draft/2017-10-how-to-make-80x-faster.rst From pypy.commits at gmail.com Fri Oct 27 08:29:51 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 05:29:51 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: tweaks Message-ID: <59f326bf.6896df0a.9740d.27d9@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5847:ac20780052f6 Date: 2017-10-27 14:28 +0200 http://bitbucket.org/pypy/extradoc/changeset/ac20780052f6/ Log: tweaks diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -25,7 +25,7 @@ However, for the scope of this post, the actual task at hand is not so important, so let's jump straight to the code. To drive the quadcopter, a -``Creature`` has a ``run_step`` method which runs at each ``delta - t`` (`full +``Creature`` has a ``run_step`` method which runs at each ``delta_t`` (`full code`_):: class Creature(object): @@ -220,8 +220,9 @@ Let's look at the trace_ again: it no longer contains expensive calls, and certainly no more temporary ``malloc()`` s. The core of the logic is between -lines XX-YY, where we can see that it does fast C-level multiplications and -additions. +lines 386-416, where we can see that it does fast C-level multiplications and +additions: ``float_mul`` and ``float_add`` are translated straight into +``mulsd`` and ``addsd`` x86 instructions. .. _trace: http://vmprof.com/#/402af746-2966-4403-a61d-93015abac033/traces @@ -232,6 +233,15 @@ benchmark which was designed specifically to have good performance on PyPy: it's a real world example, albeit small. +You might be also interested in the talk I gave at last EuroPython, in which I +talk about a similar topic: "The Joy of PyPy JIT: abstractions for free" +(abstract_, slides_ and video_). + +.. _abstract: https://ep2017.europython.eu/conference/talks/the-joy-of-pypy-jit-abstractions-for-free +.. _slides: https://speakerdeck.com/antocuni/the-joy-of-pypy-jit-abstractions-for-free +.. _video: https://www.youtube.com/watch?v=NQfpHQII2cU + + Numpy vs numpypy ----------------- From pypy.commits at gmail.com Fri Oct 27 09:07:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 06:07:21 -0700 (PDT) Subject: [pypy-commit] pypy default: finally, add two FAQs about numpy and numpypy Message-ID: <59f32f89.579c1c0a.aad37.4134@mx.google.com> Author: Antonio Cuni Branch: Changeset: r92857:dfa3189e8e9f Date: 2017-10-27 15:06 +0200 http://bitbucket.org/pypy/pypy/changeset/dfa3189e8e9f/ Log: finally, add two FAQs about numpy and numpypy diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -182,6 +182,57 @@ technical difficulties. +What about numpy, numpypy, micronumpy? +-------------------------------------- + +Way back in 2011, the PyPy team `started to reimplement`_ numpy in PyPy. It +has two pieces: + + * the builtin module :source:`pypy/module/micronumpy`: this is written in + RPython and roughly covers the content of the ``numpy.core.multiarray`` + module. Confusingly enough, this is available in PyPy under the name + ``_numpypy``. It is included by default in all the official releases of + PyPy (but it might be dropped in the future). + + * a fork_ of the official numpy repository maintained by us and informally + called ``numpypy``: even more confusing, the name of the repo on bitbucket + is ``numpy``. The main difference with the upstream numpy, is that it is + based on the micronumpy module written in RPython, instead of of + ``numpy.core.multiarray`` which is written in C. + +Moreover, it is also possible to install the upstream version of ``numpy``: +its core is written in C and it runs on PyPy under the cpyext compatibility +layer. This is what you get if you do ``pypy -m pip install numpy``. + + +Should I install numpy or numpypy? +----------------------------------- + +TL;DR version: you should use numpy. You can install it by doing ``pypy -m pip +install numpy``. You might also be interested in using the experimental `PyPy +binary wheels`_ to save compilation time. + +The upstream ``numpy`` is written in C, and runs under the cpyext +compatibility layer. Nowadays, cpyext is mature enough that you can simply +use the upstream ``numpy``, since it passes 99.9% of the test suite. At the +moment of writing (October 2017) the main drawback of ``numpy`` is that cpyext +is infamously slow, and thus it has worse performance compared to +``numpypy``. However, we are actively working on improving it, as we expect to +reach the same speed, eventually. + +On the other hand, ``numpypy`` is more JIT-friendly and very fast to call, +since it is written in RPython: but it is a reimplementation, and it's hard to +be completely compatible: over the years the project slowly matured and +eventually it was able to call out to the LAPACK and BLAS libraries to speed +matrix calculations, and reached around an 80% parity with the upstream +numpy. However, 80% is far from 100%. Since cpyext/numpy compatibility is +progressing fast, we have discontinued support for ``numpypy``. + +.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html +.. _fork: https://bitbucket.org/pypy/numpy +.. _`PyPy binary wheels`: https://github.com/antocuni/pypy-wheels + + Is PyPy more clever than CPython about Tail Calls? -------------------------------------------------- From pypy.commits at gmail.com Fri Oct 27 09:10:35 2017 From: pypy.commits at gmail.com (cfbolz) Date: Fri, 27 Oct 2017 06:10:35 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: circumvent the problem of elidable with exceptions by using an elidable helper Message-ID: <59f3304b.3499df0a.52b7c.7d86@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: unicode-utf8 Changeset: r92858:877286df2904 Date: 2017-10-27 15:09 +0200 http://bitbucket.org/pypy/pypy/changeset/877286df2904/ Log: circumvent the problem of elidable with exceptions by using an elidable helper function that doesn't raise. the API stays the same. diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -1,5 +1,4 @@ * unskip tests in test_unicodeobject.py * rutf8.prev_codepoint_pos should use r_uint -* elidable in rutf8.check_utf8, WTF is wrong with that * find a better way to run "find" without creating the index storage, - if one is not already readily available \ No newline at end of file + if one is not already readily available diff --git a/rpython/jit/metainterp/test/test_string.py b/rpython/jit/metainterp/test/test_string.py --- a/rpython/jit/metainterp/test/test_string.py +++ b/rpython/jit/metainterp/test/test_string.py @@ -952,6 +952,23 @@ self.check_simple_loop({'guard_true': 1, 'int_add': 1, 'int_lt': 1, 'jump': 1}) + def test_check_ascii(self): + from rpython.rlib.rutf8 import check_ascii + jitdriver = JitDriver(greens=['x', 'y'], reds=['z']) + def f(x, y): + z = 0 + while z < 10: + jitdriver.jit_merge_point(x=x, y=y, z=z) + if x > 0: + s = "abc" + else: + s = "def" + check_ascii(s) + z += 1 + return 0 + self.meta_interp(f, [222, 3333]) + self.check_simple_loop(call_i=0) + def test_string_hashing(self): def f(i): s = str(i) diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py --- a/rpython/rlib/rutf8.py +++ b/rpython/rlib/rutf8.py @@ -193,11 +193,19 @@ def __init__(self, pos): self.pos = pos -#@jit.elidable def check_ascii(s): + res = _check_ascii(s) + if res < 0: + return + raise CheckError(res) + + + at jit.elidable +def _check_ascii(s): for i in range(len(s)): if ord(s[i]) > 0x7F: - raise CheckError(i) + return i + return -1 def islinebreak(s, pos): chr1 = ord(s[pos]) @@ -291,7 +299,6 @@ (ordch1 == 0xf4 and ordch2 > 0x8f)) -#@jit.elidable def check_utf8(s, allow_surrogates, start=0, stop=-1): """Check that 's' is a utf-8-encoded byte string. Returns the length (number of chars) or raise CheckError. @@ -299,6 +306,13 @@ Note also codepoints_in_utf8(), which also computes the length faster by assuming that 's' is valid utf-8. """ + res = _check_utf8(s, allow_surrogates, start, stop) + if res >= 0: + return res + raise CheckError(~res) + + at jit.elidable +def _check_utf8(s, allow_surrogates, start, stop): pos = start continuation_bytes = 0 if stop < 0: @@ -313,37 +327,37 @@ continue if ordch1 <= 0xC1: - raise CheckError(pos - 1) + return ~(pos - 1) if ordch1 <= 0xDF: if pos >= end: - raise CheckError(pos - 1) + return ~(pos - 1) ordch2 = ord(s[pos]) pos += 1 if _invalid_byte_2_of_2(ordch2): - raise CheckError(pos - 2) + return ~(pos - 2) # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz continuation_bytes += 1 continue if ordch1 <= 0xEF: if (pos + 2) > end: - raise CheckError(pos - 1) + return ~(pos - 1) ordch2 = ord(s[pos]) ordch3 = ord(s[pos + 1]) pos += 2 if (_invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates) or _invalid_byte_3_of_3(ordch3)): - raise CheckError(pos - 3) + return ~(pos - 3) # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz continuation_bytes += 2 continue if ordch1 <= 0xF4: if (pos + 3) > end: - raise CheckError(pos - 1) + return ~(pos - 1) ordch2 = ord(s[pos]) ordch3 = ord(s[pos + 1]) ordch4 = ord(s[pos + 2]) @@ -352,14 +366,15 @@ if (_invalid_byte_2_of_4(ordch1, ordch2) or _invalid_byte_3_of_4(ordch3) or _invalid_byte_4_of_4(ordch4)): - raise CheckError(pos - 4) + return ~(pos - 4) # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz continuation_bytes += 3 continue - raise CheckError(pos - 1) + return ~(pos - 1) assert pos == end + assert pos - continuation_bytes >= 0 return pos - continuation_bytes @jit.elidable From pypy.commits at gmail.com Fri Oct 27 09:22:19 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 06:22:19 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: insert a link to the numpypy FAQ Message-ID: <59f3330b.4db3df0a.d93cb.d8a7@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5848:3c4d6ae486f3 Date: 2017-10-27 15:19 +0200 http://bitbucket.org/pypy/extradoc/changeset/3c4d6ae486f3/ Log: insert a link to the numpypy FAQ diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -86,10 +86,9 @@ Ouch! We are ~5.5x slower than CPython. This was kind of expected: numpy is based on cpyext, which is infamously slow (although `we are working on that`_). -So, let's try to avoid cpyext. The first obvious step is to use numpypy -instead of numpy; for more info about numpypy, see the next section. In the -meantime, here is a link to the hack_ which enables for PyPy. Let's see if the -speed improves:: +So, let's try to avoid cpyext. The first obvious step is to use numpypy_ +instead of numpy (actually, there a hack_ to use just the micronumpy +part). Let's see if the speed improves:: $ pypy -m ev.main # using numpypy Generation 1: ... [population = 500] [5.60 secs] @@ -104,6 +103,7 @@ and go tweeting how PyPy is great. .. _`we are working on that`: https://morepypy.blogspot.it/2017/10/cape-of-good-hope-for-pypy-hello-from.html +.. _numpypy: http://doc.pypy.org/en/latest/faq.html#what-about-numpy-numpypy-micronumpy .. _hack: https://github.com/antocuni/evolvingcopter/blob/master/ev/pypycompat.py In general, when talking of CPython vs PyPy, I am rarely satified of a 2x @@ -240,20 +240,3 @@ .. _abstract: https://ep2017.europython.eu/conference/talks/the-joy-of-pypy-jit-abstractions-for-free .. _slides: https://speakerdeck.com/antocuni/the-joy-of-pypy-jit-abstractions-for-free .. _video: https://www.youtube.com/watch?v=NQfpHQII2cU - - -Numpy vs numpypy ------------------ - -Way back in 2011, the PyPy team `started to reimplement`_ NumPy in PyPy. -It has two pieces: the micronumpy RPython module that roughly covers the -multiarray numpy module, and a fork of the python-code called numpypy. -Over the years the project slowly matured, eventually it was able to call -out to the LAPACK and BLAS libraries to speed matrix calculations just like -NumPy, and reached around an 80% parity with the upstream project. But 80% -is far from 100%, and once the cpyext layer of PyPy matured to the point it -could pass 99.9% of the NumPy test suite, we no longer recommend using numpypy. - -XXX more needed? - -.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html From pypy.commits at gmail.com Fri Oct 27 09:25:20 2017 From: pypy.commits at gmail.com (cfbolz) Date: Fri, 27 Oct 2017 06:25:20 -0700 (PDT) Subject: [pypy-commit] pypy regalloc-playground: merge default Message-ID: <59f333c0.e887df0a.2dc18.b185@mx.google.com> Author: Carl Friedrich Bolz-Tereick Branch: regalloc-playground Changeset: r92859:4d898e5c1b6c Date: 2017-10-27 15:24 +0200 http://bitbucket.org/pypy/pypy/changeset/4d898e5c1b6c/ Log: merge default diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,60 +8,63 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) - if '_type_' in typedict: - ffiarray = _rawffi.Array(typedict['_type_']._ffishape_) - res._ffiarray = ffiarray - subletter = getattr(typedict['_type_'], '_type_', None) - if subletter == 'c': - def getvalue(self): - return _rawffi.charp2string(self._buffer.buffer, - self._length_) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, str): - _rawffi.rawstring2charp(self._buffer.buffer, val) - else: - for i in range(len(val)): - self[i] = val[i] - if len(val) < self._length_: - self._buffer[len(val)] = '\x00' - res.value = property(getvalue, setvalue) - def getraw(self): - return _rawffi.charp2rawstring(self._buffer.buffer, - self._length_) + if cls == (_CData,): # this is the Array class defined below + res._ffiarray = None + return res + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") + ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) + subletter = getattr(res._type_, '_type_', None) + if subletter == 'c': + def getvalue(self): + return _rawffi.charp2string(self._buffer.buffer, + self._length_) + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, str): + _rawffi.rawstring2charp(self._buffer.buffer, val) + else: + for i in range(len(val)): + self[i] = val[i] + if len(val) < self._length_: + self._buffer[len(val)] = b'\x00' + res.value = property(getvalue, setvalue) - def setraw(self, buffer): - if len(buffer) > self._length_: - raise ValueError("%r too long" % (buffer,)) - _rawffi.rawstring2charp(self._buffer.buffer, buffer) - res.raw = property(getraw, setraw) - elif subletter == 'u': - def getvalue(self): - return _rawffi.wcharp2unicode(self._buffer.buffer, - self._length_) + def getraw(self): + return _rawffi.charp2rawstring(self._buffer.buffer, + self._length_) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, unicode): - target = self._buffer - else: - target = self - for i in range(len(val)): - target[i] = val[i] - if len(val) < self._length_: - target[len(val)] = u'\x00' - res.value = property(getvalue, setvalue) - - if '_length_' in typedict: - res._ffishape_ = (ffiarray, typedict['_length_']) - res._fficompositesize_ = res._sizeofinstances() - else: - res._ffiarray = None + def setraw(self, buffer): + if len(buffer) > self._length_: + raise ValueError("%r too long" % (buffer,)) + _rawffi.rawstring2charp(self._buffer.buffer, buffer) + res.raw = property(getraw, setraw) + elif subletter == 'u': + def getvalue(self): + return _rawffi.wcharp2unicode(self._buffer.buffer, + self._length_) + + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, unicode): + target = self._buffer + else: + target = self + for i in range(len(val)): + target[i] = val[i] + if len(val) < self._length_: + target[len(val)] = u'\x00' + res.value = property(getvalue, setvalue) + + res._ffishape_ = (ffiarray, res._length_) + res._fficompositesize_ = res._sizeofinstances() return res from_address = cdata_from_address @@ -156,7 +159,7 @@ l = [self[i] for i in range(start, stop, step)] letter = getattr(self._type_, '_type_', None) if letter == 'c': - return "".join(l) + return b"".join(l) if letter == 'u': return u"".join(l) return l diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -176,6 +176,10 @@ def _get_buffer_value(self): return self._buffer[0] + def _copy_to(self, addr): + target = type(self).from_address(addr)._buffer + target[0] = self._get_buffer_value() + def _to_ffi_param(self): if self.__class__._is_pointer_like(): return self._get_buffer_value() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -114,7 +114,9 @@ cobj = self._type_.from_param(value) if ensure_objects(cobj) is not None: store_reference(self, index, cobj._objects) - self._subarray(index)[0] = cobj._get_buffer_value() + address = self._buffer[0] + address += index * sizeof(self._type_) + cobj._copy_to(address) def __nonzero__(self): return self._buffer[0] != 0 diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -291,6 +291,11 @@ def _get_buffer_value(self): return self._buffer.buffer + def _copy_to(self, addr): + from ctypes import memmove + origin = self._get_buffer_value() + memmove(addr, origin, self._fficompositesize_) + def _to_ffi_param(self): return self._buffer diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1027,21 +1027,25 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split(" ")[0].upper() - if first_word == "": + + if sql: + first_word = sql.lstrip().split()[0].upper() + if first_word == '': + self._type = _STMT_TYPE_INVALID + if first_word == "SELECT": + self._type = _STMT_TYPE_SELECT + elif first_word == "INSERT": + self._type = _STMT_TYPE_INSERT + elif first_word == "UPDATE": + self._type = _STMT_TYPE_UPDATE + elif first_word == "DELETE": + self._type = _STMT_TYPE_DELETE + elif first_word == "REPLACE": + self._type = _STMT_TYPE_REPLACE + else: + self._type = _STMT_TYPE_OTHER + else: self._type = _STMT_TYPE_INVALID - elif first_word == "SELECT": - self._type = _STMT_TYPE_SELECT - elif first_word == "INSERT": - self._type = _STMT_TYPE_INSERT - elif first_word == "UPDATE": - self._type = _STMT_TYPE_UPDATE - elif first_word == "DELETE": - self._type = _STMT_TYPE_DELETE - elif first_word == "REPLACE": - self._type = _STMT_TYPE_REPLACE - else: - self._type = _STMT_TYPE_OTHER if isinstance(sql, unicode): sql = sql.encode('utf-8') diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -182,6 +182,57 @@ technical difficulties. +What about numpy, numpypy, micronumpy? +-------------------------------------- + +Way back in 2011, the PyPy team `started to reimplement`_ numpy in PyPy. It +has two pieces: + + * the builtin module :source:`pypy/module/micronumpy`: this is written in + RPython and roughly covers the content of the ``numpy.core.multiarray`` + module. Confusingly enough, this is available in PyPy under the name + ``_numpypy``. It is included by default in all the official releases of + PyPy (but it might be dropped in the future). + + * a fork_ of the official numpy repository maintained by us and informally + called ``numpypy``: even more confusing, the name of the repo on bitbucket + is ``numpy``. The main difference with the upstream numpy, is that it is + based on the micronumpy module written in RPython, instead of of + ``numpy.core.multiarray`` which is written in C. + +Moreover, it is also possible to install the upstream version of ``numpy``: +its core is written in C and it runs on PyPy under the cpyext compatibility +layer. This is what you get if you do ``pypy -m pip install numpy``. + + +Should I install numpy or numpypy? +----------------------------------- + +TL;DR version: you should use numpy. You can install it by doing ``pypy -m pip +install numpy``. You might also be interested in using the experimental `PyPy +binary wheels`_ to save compilation time. + +The upstream ``numpy`` is written in C, and runs under the cpyext +compatibility layer. Nowadays, cpyext is mature enough that you can simply +use the upstream ``numpy``, since it passes 99.9% of the test suite. At the +moment of writing (October 2017) the main drawback of ``numpy`` is that cpyext +is infamously slow, and thus it has worse performance compared to +``numpypy``. However, we are actively working on improving it, as we expect to +reach the same speed, eventually. + +On the other hand, ``numpypy`` is more JIT-friendly and very fast to call, +since it is written in RPython: but it is a reimplementation, and it's hard to +be completely compatible: over the years the project slowly matured and +eventually it was able to call out to the LAPACK and BLAS libraries to speed +matrix calculations, and reached around an 80% parity with the upstream +numpy. However, 80% is far from 100%. Since cpyext/numpy compatibility is +progressing fast, we have discontinued support for ``numpypy``. + +.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html +.. _fork: https://bitbucket.org/pypy/numpy +.. _`PyPy binary wheels`: https://github.com/antocuni/pypy-wheels + + Is PyPy more clever than CPython about Tail Calls? -------------------------------------------------- diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -290,66 +290,87 @@ def test_random_switching(self): from _continuation import continulet # + seen = [] + # def t1(c1): - return c1.switch() + seen.append(3) + res = c1.switch() + seen.append(6) + return res + # def s1(c1, n): + seen.append(2) assert n == 123 c2 = t1(c1) - return c1.switch('a') + 1 + seen.append(7) + res = c1.switch('a') + 1 + seen.append(10) + return res # def s2(c2, c1): + seen.append(5) res = c1.switch(c2) + seen.append(8) assert res == 'a' - return c2.switch('b') + 2 + res = c2.switch('b') + 2 + seen.append(12) + return res # def f(): + seen.append(1) c1 = continulet(s1, 123) c2 = continulet(s2, c1) c1.switch() + seen.append(4) res = c2.switch() + seen.append(9) assert res == 'b' res = c1.switch(1000) + seen.append(11) assert res == 1001 - return c2.switch(2000) + res = c2.switch(2000) + seen.append(13) + return res # res = f() assert res == 2002 + assert seen == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] def test_f_back(self): import sys from _continuation import continulet # - def g(c): + def bar(c): c.switch(sys._getframe(0)) c.switch(sys._getframe(0).f_back) c.switch(sys._getframe(1)) c.switch(sys._getframe(1).f_back) - assert sys._getframe(2) is f3.f_back + assert sys._getframe(2) is f3_foo.f_back c.switch(sys._getframe(2)) - def f(c): - g(c) + def foo(c): + bar(c) # - c = continulet(f) - f1 = c.switch() - assert f1.f_code.co_name == 'g' - f2 = c.switch() - assert f2.f_code.co_name == 'f' - f3 = c.switch() - assert f3 is f2 - assert f1.f_back is f3 + c = continulet(foo) + f1_bar = c.switch() + assert f1_bar.f_code.co_name == 'bar' + f2_foo = c.switch() + assert f2_foo.f_code.co_name == 'foo' + f3_foo = c.switch() + assert f3_foo is f2_foo + assert f1_bar.f_back is f3_foo def main(): - f4 = c.switch() - assert f4.f_code.co_name == 'main', repr(f4.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f4_main = c.switch() + assert f4_main.f_code.co_name == 'main' + assert f3_foo.f_back is f1_bar # not running, so a loop def main2(): - f5 = c.switch() - assert f5.f_code.co_name == 'main2', repr(f5.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f5_main2 = c.switch() + assert f5_main2.f_code.co_name == 'main2' + assert f3_foo.f_back is f1_bar # not running, so a loop main() main2() res = c.switch() assert res is None - assert f3.f_back is None + assert f3_foo.f_back is None def test_traceback_is_complete(self): import sys diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -577,6 +577,7 @@ 'PyComplex_AsCComplex', 'PyComplex_FromCComplex', 'PyObject_AsReadBuffer', 'PyObject_AsWriteBuffer', 'PyObject_CheckReadBuffer', + 'PyBuffer_GetPointer', 'PyBuffer_ToContiguous', 'PyBuffer_FromContiguous', 'PyOS_getsig', 'PyOS_setsig', 'PyThread_get_thread_ident', 'PyThread_allocate_lock', 'PyThread_free_lock', diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -317,6 +317,31 @@ PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *, const void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_AsWriteBuffer(PyObject *, void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_CheckReadBuffer(PyObject *); +PyAPI_FUNC(void *) PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices); +/* Get the memory area pointed to by the indices for the buffer given. + Note that view->ndim is the assumed size of indices +*/ + +PyAPI_FUNC(int) PyBuffer_ToContiguous(void *buf, Py_buffer *view, + Py_ssize_t len, char fort); +PyAPI_FUNC(int) PyBuffer_FromContiguous(Py_buffer *view, void *buf, + Py_ssize_t len, char fort); +/* Copy len bytes of data from the contiguous chunk of memory + pointed to by buf into the buffer exported by obj. Return + 0 on success and return -1 and raise a PyBuffer_Error on + error (i.e. the object does not have a buffer interface or + it is not working). + + If fort is 'F' and the object is multi-dimensional, + then the data will be copied into the array in + Fortran-style (first dimension varies the fastest). If + fort is 'C', then the data will be copied into the array + in C-style (last dimension varies the fastest). If fort + is 'A', then it does not matter and the copy will be made + in whatever way is more efficient. + +*/ + #define PyObject_MALLOC PyObject_Malloc #define PyObject_REALLOC PyObject_Realloc diff --git a/pypy/module/cpyext/src/abstract.c b/pypy/module/cpyext/src/abstract.c --- a/pypy/module/cpyext/src/abstract.c +++ b/pypy/module/cpyext/src/abstract.c @@ -101,6 +101,163 @@ return 0; } +void* +PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) +{ + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; +} + +void +_Py_add_one_to_index_F(int nd, Py_ssize_t *index, const Py_ssize_t *shape) +{ + int k; + + for (k=0; k=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } +} + + /* view is not checked for consistency in either of these. It is + assumed that the size of the buffer is view->len in + view->len / view->itemsize elements. + */ + +int +PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + addone(view->ndim, indices, view->shape); + } + PyMem_Free(indices); + return 0; +} + +int +PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + addone(view->ndim, indices, view->shape); + } + + PyMem_Free(indices); + return 0; +} + + + /* Buffer C-API for Python 3.0 */ int diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py --- a/pypy/module/cpyext/test/test_memoryobject.py +++ b/pypy/module/cpyext/test/test_memoryobject.py @@ -115,7 +115,36 @@ view = PyMemoryView_GET_BUFFER(memoryview); Py_DECREF(memoryview); return PyLong_FromLong(view->len / view->itemsize); - """)]) + """), + ("test_contiguous", "METH_O", + """ + Py_buffer* view; + PyObject * memoryview; + void * buf = NULL; + int ret; + Py_ssize_t len; + memoryview = PyMemoryView_FromObject(args); + if (memoryview == NULL) + return NULL; + view = PyMemoryView_GET_BUFFER(memoryview); + Py_DECREF(memoryview); + len = view->len; + if (len == 0) + return NULL; + buf = malloc(len); + ret = PyBuffer_ToContiguous(buf, view, view->len, 'A'); + if (ret != 0) + { + free(buf); + return NULL; + } + ret = PyBuffer_FromContiguous(view, buf, view->len, 'A'); + free(buf); + if (ret != 0) + return NULL; + Py_RETURN_NONE; + """), + ]) module = self.import_module(name='buffer_test') arr = module.PyMyArray(10) ten = foo.get_len(arr) @@ -124,6 +153,7 @@ assert ten == 10 ten = foo.test_buffer(arr) assert ten == 10 + foo.test_contiguous(arr) def test_releasebuffer(self): module = self.import_extension('foo', [ diff --git a/pypy/module/pyexpat/interp_pyexpat.py b/pypy/module/pyexpat/interp_pyexpat.py --- a/pypy/module/pyexpat/interp_pyexpat.py +++ b/pypy/module/pyexpat/interp_pyexpat.py @@ -808,7 +808,7 @@ w_encoding) if space.is_none(w_namespace_separator): - namespace_separator = 0 + namespace_separator = -1 elif space.isinstance_w(w_namespace_separator, space.w_text): separator = space.text_w(w_namespace_separator) if len(separator) == 0: @@ -831,7 +831,7 @@ elif space.is_w(w_intern, space.w_None): w_intern = None - if namespace_separator: + if namespace_separator >= 0: xmlparser = XML_ParserCreateNS( encoding, rffi.cast(rffi.CHAR, namespace_separator)) diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py --- a/pypy/module/pyexpat/test/test_parser.py +++ b/pypy/module/pyexpat/test/test_parser.py @@ -59,7 +59,7 @@ p.CharacterDataHandler = lambda s: data.append(s) encoding = encoding_arg is None and 'utf-8' or encoding_arg - res = p.Parse(u"\u00f6".encode(encoding), isfinal=True) + res = p.Parse(u"\u00f6".encode(encoding), True) assert res == 1 assert data == [u"\u00f6"] @@ -188,6 +188,34 @@ p.ParseFile(fake_reader) assert fake_reader.read_count == 4 + def test_entities(self): + import pyexpat + parser = pyexpat.ParserCreate(None, "") + + def startElement(tag, attrs): + assert tag == 'http://www.w3.org/1999/02/22-rdf-syntax-ns#RDF' + assert attrs == { + 'http://www.w3.org/XML/1998/namespacebase': + 'http://www.semanticweb.org/jiba/ontologies/2017/0/test'} + parser.StartElementHandler = startElement + parser.Parse(""" + + + + + + ]> + + + + """, True) + class AppTestPyexpat2: spaceconfig = dict(usemodules=['pyexpat', 'itertools', '_socket', 'time', 'struct', 'binascii']) diff --git a/pypy/module/test_lib_pypy/test_sqlite3.py b/pypy/module/test_lib_pypy/test_sqlite3.py --- a/pypy/module/test_lib_pypy/test_sqlite3.py +++ b/pypy/module/test_lib_pypy/test_sqlite3.py @@ -228,6 +228,14 @@ cur.execute("create table test(a)") cur.executemany("insert into test values (?)", [[1], [2], [3]]) assert cur.lastrowid is None + # issue 2682 + cur.execute('''insert + into test + values (?) + ''', (1, )) + assert cur.lastrowid is not None + cur.execute('''insert\t into test values (?) ''', (1, )) + assert cur.lastrowid is not None def test_authorizer_bad_value(self, con): def authorizer_cb(action, arg1, arg2, dbname, source): From pypy.commits at gmail.com Fri Oct 27 09:22:21 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 06:22:21 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: explain how to reproduce the results Message-ID: <59f3330d.c7341c0a.1707b.b750@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5849:9e8e252f6afa Date: 2017-10-27 15:22 +0200 http://bitbucket.org/pypy/extradoc/changeset/9e8e252f6afa/ Log: explain how to reproduce the results diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -240,3 +240,15 @@ .. _abstract: https://ep2017.europython.eu/conference/talks/the-joy-of-pypy-jit-abstractions-for-free .. _slides: https://speakerdeck.com/antocuni/the-joy-of-pypy-jit-abstractions-for-free .. _video: https://www.youtube.com/watch?v=NQfpHQII2cU + + +How to reproduce the results +----------------------------- + +:: + + $ git clone https://github.com/antocuni/evolvingcopter + $ cd evolvingcopter + $ {python,pypy} -m ev.main --no-specialized --no-numpypy + $ {python,pypy} -m ev.main --no-specialized + $ {python,pypy} -m ev.main From pypy.commits at gmail.com Fri Oct 27 09:43:58 2017 From: pypy.commits at gmail.com (antocuni) Date: Fri, 27 Oct 2017 06:43:58 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: rename all the files to follow YYYY-MM-title, because this directory is a mess Message-ID: <59f3381e.ddb1df0a.e3857.2b60@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5850:b876e412d58c Date: 2017-10-27 15:43 +0200 http://bitbucket.org/pypy/extradoc/changeset/b876e412d58c/ Log: rename all the files to follow YYYY-MM-title, because this directory is a mess diff --git a/blog/draft/thank-you-psf.rst b/blog/draft/2011-03-thank-you-psf.rst rename from blog/draft/thank-you-psf.rst rename to blog/draft/2011-03-thank-you-psf.rst diff --git a/blog/draft/tkinter.rst b/blog/draft/2011-04-tkinter.rst rename from blog/draft/tkinter.rst rename to blog/draft/2011-04-tkinter.rst diff --git a/blog/draft/numpy_followup.rst b/blog/draft/2011-05-numpy_followup.rst rename from blog/draft/numpy_followup.rst rename to blog/draft/2011-05-numpy_followup.rst diff --git a/blog/draft/numpy_roadmap.rst b/blog/draft/2011-05-numpy_roadmap.rst rename from blog/draft/numpy_roadmap.rst rename to blog/draft/2011-05-numpy_roadmap.rst diff --git a/blog/draft/usage_survey.rst b/blog/draft/2011-05-usage_survey.rst rename from blog/draft/usage_survey.rst rename to blog/draft/2011-05-usage_survey.rst diff --git a/blog/draft/gil.rst b/blog/draft/2011-06-gil.rst rename from blog/draft/gil.rst rename to blog/draft/2011-06-gil.rst diff --git a/blog/draft/survey_results.rst b/blog/draft/2011-06-survey_results.rst rename from blog/draft/survey_results.rst rename to blog/draft/2011-06-survey_results.rst diff --git a/blog/draft/realtime_image_processing.rst b/blog/draft/2011-07-realtime_image_processing.rst rename from blog/draft/realtime_image_processing.rst rename to blog/draft/2011-07-realtime_image_processing.rst diff --git a/blog/draft/string-formatting.rst b/blog/draft/2011-07-string-formatting.rst rename from blog/draft/string-formatting.rst rename to blog/draft/2011-07-string-formatting.rst diff --git a/blog/draft/success_stories.rst b/blog/draft/2011-07-success_stories.rst rename from blog/draft/success_stories.rst rename to blog/draft/2011-07-success_stories.rst diff --git a/blog/draft/jitviewer.rst b/blog/draft/2011-08-jitviewer.rst rename from blog/draft/jitviewer.rst rename to blog/draft/2011-08-jitviewer.rst diff --git a/blog/draft/py3donate.rst b/blog/draft/2011-09-py3donate.rst rename from blog/draft/py3donate.rst rename to blog/draft/2011-09-py3donate.rst diff --git a/blog/draft/faster-json.rst b/blog/draft/2011-10-faster-json.rst rename from blog/draft/faster-json.rst rename to blog/draft/2011-10-faster-json.rst diff --git a/blog/draft/numpy-status-update.rst b/blog/draft/2011-10-numpy-status-update.rst rename from blog/draft/numpy-status-update.rst rename to blog/draft/2011-10-numpy-status-update.rst diff --git a/blog/draft/matplotlib.rst b/blog/draft/2011-12-matplotlib.rst rename from blog/draft/matplotlib.rst rename to blog/draft/2011-12-matplotlib.rst diff --git a/blog/draft/pycon-2012-teaser.rst b/blog/draft/2011-12-pycon-2012-teaser.rst rename from blog/draft/pycon-2012-teaser.rst rename to blog/draft/2011-12-pycon-2012-teaser.rst diff --git a/blog/draft/laplace.rst b/blog/draft/2012-01-laplace.rst rename from blog/draft/laplace.rst rename to blog/draft/2012-01-laplace.rst diff --git a/blog/draft/numpy-internship.rst b/blog/draft/2012-01-numpy-internship.rst rename from blog/draft/numpy-internship.rst rename to blog/draft/2012-01-numpy-internship.rst diff --git a/blog/draft/numpy-status-update-2.rst b/blog/draft/2012-01-numpy-status-update-2.rst rename from blog/draft/numpy-status-update-2.rst rename to blog/draft/2012-01-numpy-status-update-2.rst diff --git a/blog/draft/pypy-2011.rst b/blog/draft/2012-01-pypy-2011.rst rename from blog/draft/pypy-2011.rst rename to blog/draft/2012-01-pypy-2011.rst diff --git a/blog/draft/tm.rst b/blog/draft/2012-01-tm.rst rename from blog/draft/tm.rst rename to blog/draft/2012-01-tm.rst diff --git a/blog/draft/py3k-status-update-1.rst b/blog/draft/2012-02-py3k-status-update-1.rst rename from blog/draft/py3k-status-update-1.rst rename to blog/draft/2012-02-py3k-status-update-1.rst diff --git a/blog/draft/py3k-status-update-2.rst b/blog/draft/2012-03-py3k-status-update-2.rst rename from blog/draft/py3k-status-update-2.rst rename to blog/draft/2012-03-py3k-status-update-2.rst diff --git a/blog/draft/pycon-wrapup.rst b/blog/draft/2012-03-pycon-wrapup.rst rename from blog/draft/pycon-wrapup.rst rename to blog/draft/2012-03-pycon-wrapup.rst diff --git a/blog/draft/numpy-status-update-3.rst b/blog/draft/2012-04-numpy-status-update-3.rst rename from blog/draft/numpy-status-update-3.rst rename to blog/draft/2012-04-numpy-status-update-3.rst diff --git a/blog/draft/py3k-status-update-3.rst b/blog/draft/2012-04-py3k-status-update-3.rst rename from blog/draft/py3k-status-update-3.rst rename to blog/draft/2012-04-py3k-status-update-3.rst diff --git a/blog/draft/stm-apr2012.rst b/blog/draft/2012-04-stm-apr2012.rst rename from blog/draft/stm-apr2012.rst rename to blog/draft/2012-04-stm-apr2012.rst diff --git a/blog/draft/cffi-release-0.1.rst b/blog/draft/2012-06-cffi-release-0.1.rst rename from blog/draft/cffi-release-0.1.rst rename to blog/draft/2012-06-cffi-release-0.1.rst diff --git a/blog/draft/py3k-status-update-4.rst b/blog/draft/2012-06-py3k-status-update-4.rst rename from blog/draft/py3k-status-update-4.rst rename to blog/draft/2012-06-py3k-status-update-4.rst diff --git a/blog/draft/stm-jun2012.rst b/blog/draft/2012-06-stm-jun2012.rst rename from blog/draft/stm-jun2012.rst rename to blog/draft/2012-06-stm-jun2012.rst diff --git a/blog/draft/cffi-release-0.2.rst b/blog/draft/2012-07-cffi-release-0.2.rst rename from blog/draft/cffi-release-0.2.rst rename to blog/draft/2012-07-cffi-release-0.2.rst diff --git a/blog/draft/plans-for-2-years.rst b/blog/draft/2012-07-plans-for-2-years.rst rename from blog/draft/plans-for-2-years.rst rename to blog/draft/2012-07-plans-for-2-years.rst diff --git a/blog/draft/py3k-status-update-5.rst b/blog/draft/2012-07-py3k-status-update-5.rst rename from blog/draft/py3k-status-update-5.rst rename to blog/draft/2012-07-py3k-status-update-5.rst diff --git a/blog/draft/stm-jul2012.rst b/blog/draft/2012-07-stm-jul2012.rst rename from blog/draft/stm-jul2012.rst rename to blog/draft/2012-07-stm-jul2012.rst diff --git a/blog/draft/cffi-release-0.3.rst b/blog/draft/2012-08-cffi-release-0.3.rst rename from blog/draft/cffi-release-0.3.rst rename to blog/draft/2012-08-cffi-release-0.3.rst diff --git a/blog/draft/numpy-non-progress.rst b/blog/draft/2012-08-numpy-non-progress.rst rename from blog/draft/numpy-non-progress.rst rename to blog/draft/2012-08-numpy-non-progress.rst diff --git a/blog/draft/arm-status-update.rst b/blog/draft/2012-09-arm-status-update.rst rename from blog/draft/arm-status-update.rst rename to blog/draft/2012-09-arm-status-update.rst diff --git a/blog/draft/numpy-status-update-4.rst b/blog/draft/2012-09-numpy-status-update-4.rst rename from blog/draft/numpy-status-update-4.rst rename to blog/draft/2012-09-numpy-status-update-4.rst diff --git a/blog/draft/py3k-status-update-6.rst b/blog/draft/2012-09-py3k-status-update-6.rst rename from blog/draft/py3k-status-update-6.rst rename to blog/draft/2012-09-py3k-status-update-6.rst diff --git a/blog/draft/py3k-status-update-7.rst b/blog/draft/2012-10-py3k-status-update-7.rst rename from blog/draft/py3k-status-update-7.rst rename to blog/draft/2012-10-py3k-status-update-7.rst diff --git a/blog/draft/za-sprint-report.rst b/blog/draft/2012-10-za-sprint-report.rst rename from blog/draft/za-sprint-report.rst rename to blog/draft/2012-10-za-sprint-report.rst diff --git a/blog/draft/numpy-status-update-5.rst b/blog/draft/2012-11-numpy-status-update-5.rst rename from blog/draft/numpy-status-update-5.rst rename to blog/draft/2012-11-numpy-status-update-5.rst diff --git a/blog/draft/numpy-internship-2.rst b/blog/draft/2012-12-numpy-internship-2.rst rename from blog/draft/numpy-internship-2.rst rename to blog/draft/2012-12-numpy-internship-2.rst diff --git a/blog/draft/numpy-status-update-6.rst b/blog/draft/2012-12-numpy-status-update-6.rst rename from blog/draft/numpy-status-update-6.rst rename to blog/draft/2012-12-numpy-status-update-6.rst diff --git a/blog/draft/py3k-status-update-8.rst b/blog/draft/2012-12-py3k-status-update-8.rst rename from blog/draft/py3k-status-update-8.rst rename to blog/draft/2012-12-py3k-status-update-8.rst diff --git a/blog/draft/py3k-status-update-9.rst b/blog/draft/2013-01-py3k-status-update-9.rst rename from blog/draft/py3k-status-update-9.rst rename to blog/draft/2013-01-py3k-status-update-9.rst diff --git a/blog/draft/10years.rst b/blog/draft/2013-02-10years.rst rename from blog/draft/10years.rst rename to blog/draft/2013-02-10years.rst diff --git a/blog/draft/py3k-status-update-10.rst b/blog/draft/2013-03-py3k-status-update-10.rst rename from blog/draft/py3k-status-update-10.rst rename to blog/draft/2013-03-py3k-status-update-10.rst diff --git a/blog/draft/so_you_want.rst b/blog/draft/2013-03-so_you_want.rst rename from blog/draft/so_you_want.rst rename to blog/draft/2013-03-so_you_want.rst diff --git a/blog/draft/web-server-survey.rst b/blog/draft/2013-04-web-server-survey.rst rename from blog/draft/web-server-survey.rst rename to blog/draft/2013-04-web-server-survey.rst diff --git a/blog/draft/pypy-alpha-arm.rst b/blog/draft/2013-05-pypy-alpha-arm.rst rename from blog/draft/pypy-alpha-arm.rst rename to blog/draft/2013-05-pypy-alpha-arm.rst diff --git a/blog/draft/py3k-status-update-11.rst b/blog/draft/2013-06-py3k-status-update-11.rst rename from blog/draft/py3k-status-update-11.rst rename to blog/draft/2013-06-py3k-status-update-11.rst diff --git a/blog/draft/duhton.rst b/blog/draft/2013-07-duhton.rst rename from blog/draft/duhton.rst rename to blog/draft/2013-07-duhton.rst diff --git a/blog/draft/numpy-road-forward.rst b/blog/draft/2013-08-numpy-road-forward.rst rename from blog/draft/numpy-road-forward.rst rename to blog/draft/2013-08-numpy-road-forward.rst diff --git a/blog/draft/coverage.rst b/blog/draft/2013-10-coverage.rst rename from blog/draft/coverage.rst rename to blog/draft/2013-10-coverage.rst diff --git a/blog/draft/incremental-gc.rst b/blog/draft/2013-10-incremental-gc.rst rename from blog/draft/incremental-gc.rst rename to blog/draft/2013-10-incremental-gc.rst diff --git a/blog/draft/numpy_oct_2013.rst b/blog/draft/2013-10-numpy_oct_2013.rst rename from blog/draft/numpy_oct_2013.rst rename to blog/draft/2013-10-numpy_oct_2013.rst diff --git a/blog/draft/stm-oct2013.rst b/blog/draft/2013-10-stm-oct2013.rst rename from blog/draft/stm-oct2013.rst rename to blog/draft/2013-10-stm-oct2013.rst diff --git a/blog/draft/py3k-status-update-12.rst b/blog/draft/2013-11-py3k-status-update-12.rst rename from blog/draft/py3k-status-update-12.rst rename to blog/draft/2013-11-py3k-status-update-12.rst diff --git a/blog/draft/running-numpy.rst b/blog/draft/2014-02-running-numpy.rst rename from blog/draft/running-numpy.rst rename to blog/draft/2014-02-running-numpy.rst diff --git a/blog/draft/stm-feb2014.txt b/blog/draft/2014-02-stm.txt rename from blog/draft/stm-feb2014.txt rename to blog/draft/2014-02-stm.txt diff --git a/blog/draft/stm-mar2014.txt b/blog/draft/2014-03-stm.txt rename from blog/draft/stm-mar2014.txt rename to blog/draft/2014-03-stm.txt diff --git a/blog/draft/py3k-status-update-13.rst b/blog/draft/2014-07-py3k-status-update-13.rst rename from blog/draft/py3k-status-update-13.rst rename to blog/draft/2014-07-py3k-status-update-13.rst diff --git a/blog/draft/io-improvements.rst b/blog/draft/2014-10-io-improvements.rst rename from blog/draft/io-improvements.rst rename to blog/draft/2014-10-io-improvements.rst diff --git a/blog/draft/thank-you-psf-2.rst b/blog/draft/2014-11-thank-you-psf-2.rst rename from blog/draft/thank-you-psf-2.rst rename to blog/draft/2014-11-thank-you-psf-2.rst diff --git a/blog/draft/tornado-stm.rst b/blog/draft/2014-11-tornado-stm.rst rename from blog/draft/tornado-stm.rst rename to blog/draft/2014-11-tornado-stm.rst diff --git a/blog/draft/ordered-dicts.rst b/blog/draft/2015-01-ordered-dicts.rst rename from blog/draft/ordered-dicts.rst rename to blog/draft/2015-01-ordered-dicts.rst diff --git a/blog/draft/stm-mar2015.txt b/blog/draft/2015-03-stm.txt rename from blog/draft/stm-mar2015.txt rename to blog/draft/2015-03-stm.txt diff --git a/blog/draft/warmup-improvements.rst b/blog/draft/2015-09-warmup-improvements.rst rename from blog/draft/warmup-improvements.rst rename to blog/draft/2015-09-warmup-improvements.rst diff --git a/blog/draft/ppc-backend.rst b/blog/draft/2015-10-ppc-backend.rst rename from blog/draft/ppc-backend.rst rename to blog/draft/2015-10-ppc-backend.rst diff --git a/blog/draft/warmup-improvements-2.rst b/blog/draft/2015-10-warmup-improvements-2.rst rename from blog/draft/warmup-improvements-2.rst rename to blog/draft/2015-10-warmup-improvements-2.rst diff --git a/blog/draft/cffi-embedding.rst b/blog/draft/2016-01-cffi-embedding.rst rename from blog/draft/cffi-embedding.rst rename to blog/draft/2016-01-cffi-embedding.rst diff --git a/blog/draft/cpyext-gcsupport.rst b/blog/draft/2016-02-cpyext-gcsupport.rst rename from blog/draft/cpyext-gcsupport.rst rename to blog/draft/2016-02-cpyext-gcsupport.rst diff --git a/blog/draft/cpyext-ext.rst b/blog/draft/2016-04-cpyext-ext.rst rename from blog/draft/cpyext-ext.rst rename to blog/draft/2016-04-cpyext-ext.rst diff --git a/blog/draft/jit-leaner-frontend.rst b/blog/draft/2016-04-jit-leaner-frontend.rst rename from blog/draft/jit-leaner-frontend.rst rename to blog/draft/2016-04-jit-leaner-frontend.rst diff --git a/blog/draft/revdb.rst b/blog/draft/2016-07-revdb.rst rename from blog/draft/revdb.rst rename to blog/draft/2016-07-revdb.rst diff --git a/blog/draft/new-jit-log.rst b/blog/draft/2016-08-new-jit-log.rst rename from blog/draft/new-jit-log.rst rename to blog/draft/2016-08-new-jit-log.rst diff --git a/blog/draft/vectorization_extended.rst b/blog/draft/2016-10-vectorization_extended.rst rename from blog/draft/vectorization_extended.rst rename to blog/draft/2016-10-vectorization_extended.rst diff --git a/blog/draft/vmprof-native.rst b/blog/draft/2017-04-vmprof-native.rst rename from blog/draft/vmprof-native.rst rename to blog/draft/2017-04-vmprof-native.rst diff --git a/blog/draft/remove-gil.rst b/blog/draft/2017-08-remove-gil.rst rename from blog/draft/remove-gil.rst rename to blog/draft/2017-08-remove-gil.rst From pypy.commits at gmail.com Fri Oct 27 10:33:09 2017 From: pypy.commits at gmail.com (mattip) Date: Fri, 27 Oct 2017 07:33:09 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: when running tests on windows, the block may be formatted with unix linesep Message-ID: <59f343a5.f1ae500a.f6ce7.f93c@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92860:0b92ac004fe5 Date: 2017-10-27 17:30 +0300 http://bitbucket.org/pypy/pypy/changeset/0b92ac004fe5/ Log: when running tests on windows, the block may be formatted with unix linesep diff --git a/pypy/tool/pytest/apptest.py b/pypy/tool/pytest/apptest.py --- a/pypy/tool/pytest/apptest.py +++ b/pypy/tool/pytest/apptest.py @@ -90,7 +90,7 @@ import os try: if isinstance(func, str): - if func.startswith((' ', os.linesep)): + if func.startswith((' ', os.linesep, '\n')): # it's probably an indented block, so we prefix if True: # to avoid SyntaxError func = "if True:\n" + func From pypy.commits at gmail.com Fri Oct 27 10:33:15 2017 From: pypy.commits at gmail.com (mattip) Date: Fri, 27 Oct 2017 07:33:15 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge default into py3.5 Message-ID: <59f343ab.06ab500a.5850b.f7b1@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92861:b45ee2feb1b1 Date: 2017-10-27 17:32 +0300 http://bitbucket.org/pypy/pypy/changeset/b45ee2feb1b1/ Log: merge default into py3.5 diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1080,21 +1080,25 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split()[0].upper() - if first_word == "": + + if sql: + first_word = sql.lstrip().split()[0].upper() + if first_word == '': + self._type = _STMT_TYPE_INVALID + if first_word == "SELECT": + self._type = _STMT_TYPE_SELECT + elif first_word == "INSERT": + self._type = _STMT_TYPE_INSERT + elif first_word == "UPDATE": + self._type = _STMT_TYPE_UPDATE + elif first_word == "DELETE": + self._type = _STMT_TYPE_DELETE + elif first_word == "REPLACE": + self._type = _STMT_TYPE_REPLACE + else: + self._type = _STMT_TYPE_OTHER + else: self._type = _STMT_TYPE_INVALID - elif first_word == "SELECT": - self._type = _STMT_TYPE_SELECT - elif first_word == "INSERT": - self._type = _STMT_TYPE_INSERT - elif first_word == "UPDATE": - self._type = _STMT_TYPE_UPDATE - elif first_word == "DELETE": - self._type = _STMT_TYPE_DELETE - elif first_word == "REPLACE": - self._type = _STMT_TYPE_REPLACE - else: - self._type = _STMT_TYPE_OTHER if isinstance(sql, unicode): sql = sql.encode('utf-8') diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -582,6 +582,7 @@ 'PyComplex_AsCComplex', 'PyComplex_FromCComplex', 'PyObject_AsReadBuffer', 'PyObject_AsWriteBuffer', 'PyObject_CheckReadBuffer', + 'PyBuffer_GetPointer', 'PyBuffer_ToContiguous', 'PyBuffer_FromContiguous', 'PyImport_ImportModuleLevel', diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -308,6 +308,31 @@ PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *, const void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_AsWriteBuffer(PyObject *, void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_CheckReadBuffer(PyObject *); +PyAPI_FUNC(void *) PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices); +/* Get the memory area pointed to by the indices for the buffer given. + Note that view->ndim is the assumed size of indices +*/ + +PyAPI_FUNC(int) PyBuffer_ToContiguous(void *buf, Py_buffer *view, + Py_ssize_t len, char fort); +PyAPI_FUNC(int) PyBuffer_FromContiguous(Py_buffer *view, void *buf, + Py_ssize_t len, char fort); +/* Copy len bytes of data from the contiguous chunk of memory + pointed to by buf into the buffer exported by obj. Return + 0 on success and return -1 and raise a PyBuffer_Error on + error (i.e. the object does not have a buffer interface or + it is not working). + + If fort is 'F' and the object is multi-dimensional, + then the data will be copied into the array in + Fortran-style (first dimension varies the fastest). If + fort is 'C', then the data will be copied into the array + in C-style (last dimension varies the fastest). If fort + is 'A', then it does not matter and the copy will be made + in whatever way is more efficient. + +*/ + #define PyObject_MALLOC PyObject_Malloc #define PyObject_REALLOC PyObject_Realloc diff --git a/pypy/module/cpyext/src/abstract.c b/pypy/module/cpyext/src/abstract.c --- a/pypy/module/cpyext/src/abstract.c +++ b/pypy/module/cpyext/src/abstract.c @@ -96,6 +96,163 @@ return 0; } +void* +PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) +{ + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; +} + +void +_Py_add_one_to_index_F(int nd, Py_ssize_t *index, const Py_ssize_t *shape) +{ + int k; + + for (k=0; k=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } +} + + /* view is not checked for consistency in either of these. It is + assumed that the size of the buffer is view->len in + view->len / view->itemsize elements. + */ + +int +PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + addone(view->ndim, indices, view->shape); + } + PyMem_Free(indices); + return 0; +} + +int +PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + addone(view->ndim, indices, view->shape); + } + + PyMem_Free(indices); + return 0; +} + + + /* Buffer C-API for Python 3.0 */ int diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py --- a/pypy/module/cpyext/test/test_memoryobject.py +++ b/pypy/module/cpyext/test/test_memoryobject.py @@ -137,7 +137,36 @@ view = PyMemoryView_GET_BUFFER(memoryview); Py_DECREF(memoryview); return PyLong_FromLong(view->len / view->itemsize); - """)]) + """), + ("test_contiguous", "METH_O", + """ + Py_buffer* view; + PyObject * memoryview; + void * buf = NULL; + int ret; + Py_ssize_t len; + memoryview = PyMemoryView_FromObject(args); + if (memoryview == NULL) + return NULL; + view = PyMemoryView_GET_BUFFER(memoryview); + Py_DECREF(memoryview); + len = view->len; + if (len == 0) + return NULL; + buf = malloc(len); + ret = PyBuffer_ToContiguous(buf, view, view->len, 'A'); + if (ret != 0) + { + free(buf); + return NULL; + } + ret = PyBuffer_FromContiguous(view, buf, view->len, 'A'); + free(buf); + if (ret != 0) + return NULL; + Py_RETURN_NONE; + """), + ]) module = self.import_module(name='buffer_test') arr = module.PyMyArray(10) ten = foo.get_len(arr) @@ -146,6 +175,7 @@ assert ten == 10 ten = foo.test_buffer(arr) assert ten == 10 + foo.test_contiguous(arr) def test_releasebuffer(self): module = self.import_extension('foo', [ From pypy.commits at gmail.com Sat Oct 28 05:56:44 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 28 Oct 2017 02:56:44 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fix pyexpat Message-ID: <59f4545c.90921c0a.f1310.bd12@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92862:44cd30c7b69a Date: 2017-10-28 11:56 +0200 http://bitbucket.org/pypy/pypy/changeset/44cd30c7b69a/ Log: fix pyexpat diff --git a/pypy/module/pyexpat/interp_pyexpat.py b/pypy/module/pyexpat/interp_pyexpat.py --- a/pypy/module/pyexpat/interp_pyexpat.py +++ b/pypy/module/pyexpat/interp_pyexpat.py @@ -478,12 +478,14 @@ # I suppose this is a valid utf8, but there is noone to check # and noone to catch an error either try: - rutf8.str_check_utf8(s, len(s), final=True) - return space.newutf8(s, -1) - except rutf8.Utf8CheckError as e: + lgt = rutf8.check_utf8(s, True) + return space.newutf8(s, lgt) + except rutf8.CheckError: from pypy.interpreter import unicodehelper - unicodehelper.decode_error_handler(space)('strict', 'utf-8', - e.msg, s, e.startpos, e.endpos) + # get the correct error msg + unicodehelper.str_decode_utf8(s, len(s), 'string', True, + unicodehelper.decode_error_handler(space)) + assert False, "always raises" else: return space.newtext(s) From pypy.commits at gmail.com Sat Oct 28 06:04:23 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 28 Oct 2017 03:04:23 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: disable for now and document Message-ID: <59f45627.d08edf0a.73d35.512e@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92863:e653587f0ad5 Date: 2017-10-28 12:03 +0200 http://bitbucket.org/pypy/pypy/changeset/e653587f0ad5/ Log: disable for now and document diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -2,3 +2,4 @@ * rutf8.prev_codepoint_pos should use r_uint * find a better way to run "find" without creating the index storage, if one is not already readily available +* fix _pypyjson From pypy.commits at gmail.com Sat Oct 28 06:16:04 2017 From: pypy.commits at gmail.com (fijal) Date: Sat, 28 Oct 2017 03:16:04 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: remove that too Message-ID: <59f458e4.c7341c0a.13fb8.2d40@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92864:08329fddbd93 Date: 2017-10-28 12:15 +0200 http://bitbucket.org/pypy/pypy/changeset/08329fddbd93/ Log: remove that too diff --git a/TODO b/TODO --- a/TODO +++ b/TODO @@ -3,3 +3,4 @@ * find a better way to run "find" without creating the index storage, if one is not already readily available * fix _pypyjson +* fix cpyext From pypy.commits at gmail.com Sat Oct 28 06:17:18 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 28 Oct 2017 03:17:18 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: put screenshots of parts of the trace, as suggested by fijal Message-ID: <59f4592e.13121c0a.29d6b.0a1e@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5851:c08fe2c2dbef Date: 2017-10-28 12:10 +0200 http://bitbucket.org/pypy/extradoc/changeset/c08fe2c2dbef/ Log: put screenshots of parts of the trace, as suggested by fijal diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -122,13 +122,29 @@ arrays to store the results of the various operations. Also, let's look at the `jit traces`_ and search for the function ``run``: -this is loop in which we spend most of the time, and it is composed -of 1796 operations. The operations emitted for the line ``np.dot(...) + -self.constant`` are listed between lines 1217 and 1456; 239 low level -operations are a lot. If we look at them, we can see for example that there is -a call to the RPython function `descr_dot`_, at line 1232. But there are also -calls to ``raw_malloc``, at line 1295, which allocates the space to store the -result of ``... + self.constant``. +this is loop in which we spend most of the time, and it is composed of 1796 +operations. The operations emitted for the line ``np.dot(...) + +self.constant`` are listed between lines 1217 and 1456. Here is the excerpt +which calls ``np.dot(...)``; most of the ops are cheap, but at line 1232 we +see a call to the RPython function `descr_dot`_; by looking at the +implementation we see that it creates a new ``W_NDimArray`` to store the +result, which means it has to do a ``malloc()``: + +.. image:: 2017-10-trace1.png + +The implementation of the ``+ self.constant`` part is also interesting: +contrary the former, the call to ``W_NDimArray.descr_add`` has been inlined by +the JIT, so we have a better picture of what's happening; in particular, we +can see the call to ``__0_alloc_with_del____`` which allocates the +``W_NDimArray`` for the result, and the ``raw_malloc`` which allocates the +actual array. Then we have a long list of 149 simple operations which set the +fields of the resulting array, construct an iterator, and finally do a +``call_assembler``: this is the actual logic to do the addition, which was +JITtted indipendently; ``call_assembler`` is one of the operations to do +JIT-to-JIT calls: + +.. image:: 2017-10-trace1.png + .. _`vmprof profile`: http://vmprof.com/#/449ca8ee-3ab2-49d4-b6f0-9099987e9000 .. _`jit traces`: http://vmprof.com/#/28fd6e8f-f103-4bf4-a76a-4b65dbd637f4/traces @@ -136,8 +152,10 @@ All of this is very suboptimal: in this particular case, we know that the shape of ``self.matrix`` is always ``(3, 2)``: so, we are doing an incredible -amount of work. We also use ``malloc()`` to create a temporary array just to call an RPython -function which ultimately does a total of 6 multiplications and 8 additions. +amount of work. We also use ``malloc()`` to create a temporary array just to +call an RPython function which ultimately does a total of 6 multiplications +and 8 additions. Note also that this is not a fault of the JIT: CPython+numpy +has to do the same amount of work, just hidden inside C calls. One possible solution to this nonsense is a well known compiler optimization: loop unrolling. From the compiler point of view, unrolling the loop is always diff --git a/blog/draft/2017-10-trace1.png b/blog/draft/2017-10-trace1.png new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..491f07997fa4d66f2dc7d44fb1ed77a32981251f GIT binary patch [cut] diff --git a/blog/draft/2017-10-trace2.png b/blog/draft/2017-10-trace2.png new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ad0002021d8bc0f21b392c35dfbea900865d2d75 GIT binary patch [cut] From pypy.commits at gmail.com Sat Oct 28 06:17:19 2017 From: pypy.commits at gmail.com (antocuni) Date: Sat, 28 Oct 2017 03:17:19 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: another disclaimer Message-ID: <59f4592f.26afdf0a.91ae9.de90@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5852:fb3978e5d935 Date: 2017-10-28 12:11 +0200 http://bitbucket.org/pypy/extradoc/changeset/fb3978e5d935/ Log: another disclaimer diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -84,7 +84,9 @@ Generation 4: ... [population = 500] [33.75 secs] Ouch! We are ~5.5x slower than CPython. This was kind of expected: numpy is -based on cpyext, which is infamously slow (although `we are working on that`_). +based on cpyext, which is infamously slow. (Actually, `we are working on +that`_ and on the ``cpyext-avoid-roundtrip`` branch we are already faster than +CPython, but this will be the subject of another blog post.) So, let's try to avoid cpyext. The first obvious step is to use numpypy_ instead of numpy (actually, there a hack_ to use just the micronumpy From pypy.commits at gmail.com Sat Oct 28 12:25:38 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 28 Oct 2017 09:25:38 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Make tests import even if 'hypothesis' is not available Message-ID: <59f4af82.4e9d1c0a.69940.eda4@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92865:ff734c1f0ec6 Date: 2017-10-28 19:24 +0300 http://bitbucket.org/pypy/pypy/changeset/ff734c1f0ec6/ Log: Make tests import even if 'hypothesis' is not available diff --git a/pypy/interpreter/test/test_interpreter.py b/pypy/interpreter/test/test_interpreter.py --- a/pypy/interpreter/test/test_interpreter.py +++ b/pypy/interpreter/test/test_interpreter.py @@ -1,7 +1,6 @@ import py import sys from pypy.interpreter import gateway, module, error -from hypothesis import given, strategies class TestInterpreter: @@ -300,30 +299,6 @@ assert "TypeError:" in res assert "'tuple' object is not a mapping" in res - @given(strategies.lists(strategies.one_of(strategies.none(), - strategies.lists(strategies.none())))) - def test_build_map_order(self, shape): - value = [10] - def build_expr(shape): - if shape is None: - value[0] += 1 - return '0: %d' % value[0] - else: - return '**{%s}' % (', '.join( - [build_expr(shape1) for shape1 in shape]),) - - expr = build_expr(shape)[2:] - code = """ - def f(): - return %s - """ % (expr, ) - res = self.codetest(code, 'f', []) - if value[0] == 10: - expected = {} - else: - expected = {0: value[0]} - assert res == expected, "got %r for %r" % (res, expr) - def test_build_map_unpack_with_call(self): code = """ def f(a,b,c,d): @@ -348,6 +323,36 @@ assert "TypeError:" in resg4 assert "got multiple values for keyword argument 'a'" in resg4 +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + class TestHypothesisInterpreter(TestInterpreter): + @given(strategies.lists(strategies.one_of(strategies.none(), + strategies.lists(strategies.none())))) + def test_build_map_order(self, shape): + value = [10] + def build_expr(shape): + if shape is None: + value[0] += 1 + return '0: %d' % value[0] + else: + return '**{%s}' % (', '.join( + [build_expr(shape1) for shape1 in shape]),) + + expr = build_expr(shape)[2:] + code = """ + def f(): + return %s + """ % (expr, ) + res = self.codetest(code, 'f', []) + if value[0] == 10: + expected = {} + else: + expected = {0: value[0]} + assert res == expected, "got %r for %r" % (res, expr) + class AppTestInterpreter: def test_trivial(self): From pypy.commits at gmail.com Sat Oct 28 12:54:05 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 28 Oct 2017 09:54:05 -0700 (PDT) Subject: [pypy-commit] pypy.org extradoc: update numpy installation info, refering to the FAQ answer in docs.pypy.org Message-ID: <59f4b62d.83b91c0a.1c7df.3bf5@mx.google.com> Author: Matti Picus Branch: extradoc Changeset: r902:97b16db806af Date: 2017-10-28 19:53 +0300 http://bitbucket.org/pypy/pypy.org/changeset/97b16db806af/ Log: update numpy installation info, refering to the FAQ answer in docs.pypy.org diff --git a/download.html b/download.html --- a/download.html +++ b/download.html @@ -212,38 +212,30 @@

Installing NumPy

There are two different versions of NumPy for PyPy.

1. NumPy via cpyext

The generally recommended way is to install the original NumPy via the -CPython C API compatibility layer, cpyext. Modern versions of PyPy -support enough of the C API to make this a reasonable choice in many -cases. Performance-wise, the speed is mostly the same as CPython's -NumPy (it is the same code); the exception is that interactions between -the Python side and NumPy objects are mediated through the slower cpyext -layer (which hurts a few benchmarks that do a lot of element-by-element -array accesses, for example).

Installation works on any recent PyPy (the release above is fine, a recent -nightly will implement more of the new buffer protocol). -The currently released numpy 1.12 works except for nditers with the +

There are two different versions of NumPy for PyPy. For details see this +FAQ question.

1. Standard NumPy

Installation works on any recent PyPy (the release above is fine). +The currently released numpy 1.13 works except for nditers with the updateifcopy flag. For example, without using a virtualenv:

 $ ./pypy-xxx/bin/pypy -m ensurepip
-$ ./pypy-xxx/bin/pypy -mpip install -U wheel # to update wheel
 $ ./pypy-xxx/bin/pip install cython numpy

(See the general installation documentation for more.)

2. NumPyPy

The “numpy” module can be installed from our own repository rather -than from the official source. This version uses internally our -built-in _numpypy module. This module is slightly incomplete. -Also, its performance is hard to predict exactly. For regular NumPy +

The “numpy” module can also be installed from our own repository rather +than from the official source. This version uses our +built-in _numpypy multiarray replacement module, written in RPython. +This module is not complete, but if it works it should give correct answers. +Its performance is hard to predict exactly. For regular NumPy source code that handles large arrays, it is likely to be slower than -the native NumPy with cpyext. It is faster on the kind of code that -contains many Python loops doing things on an element-by-element basis.

+the standard NumPy. It is faster on pure python code that loop over ndarrays +doing things on an element-by-element basis.

Installation (see the installation documentation for installing pip):

 pypy -m pip install git+https://bitbucket.org/pypy/numpy.git
@@ -258,9 +250,10 @@
  sudo pypy -c 'import numpy'
 
-Note again that this version is still a work-in-progress: many things do
+
Note again that this version is incomplete: many things do
 not work and those that do may not be any faster than NumPy on CPython.
-For further instructions see the pypy/numpy repository.
+For further instructions see the pypy/numpy repository and the
+FAQ question about the difference between the two.

diff --git a/source/download.txt b/source/download.txt --- a/source/download.txt +++ b/source/download.txt @@ -223,29 +223,18 @@ Installing NumPy ------------------------------- -**There are two different versions of NumPy for PyPy.** +**There are two different versions of NumPy for PyPy.** For details see this +`FAQ question`_. +1. Standard NumPy ++++++++++++++++++ -1. NumPy via cpyext -+++++++++++++++++++ - -The generally recommended way is to install the original NumPy via the -CPython C API compatibility layer, cpyext. Modern versions of PyPy -support enough of the C API to make this a reasonable choice in many -cases. Performance-wise, the speed is mostly the same as CPython's -NumPy (it is the same code); the exception is that interactions between -the Python side and NumPy objects are mediated through the slower cpyext -layer (which hurts a few benchmarks that do a lot of element-by-element -array accesses, for example). - -Installation works on any recent PyPy (the release_ above is fine, a recent -nightly will implement more of the new buffer protocol). -The currently released numpy 1.12 works except for ``nditers`` with the +Installation works on any recent PyPy (the release_ above is fine). +The currently released numpy 1.13 works except for ``nditers`` with the ``updateifcopy`` flag. For example, without using a virtualenv:: $ ./pypy-xxx/bin/pypy -m ensurepip - $ ./pypy-xxx/bin/pypy -mpip install -U wheel # to update wheel $ ./pypy-xxx/bin/pip install cython numpy (See the general `installation documentation`_ for more.) @@ -254,13 +243,14 @@ 2. NumPyPy ++++++++++ -The "numpy" module can be installed from `our own repository`__ rather -than from the official source. This version uses internally our -built-in ``_numpypy`` module. This module is slightly incomplete. -Also, its performance is hard to predict exactly. For regular NumPy +The "numpy" module can also be installed from `our own repository`__ rather +than from the official source. This version uses our +built-in ``_numpypy`` multiarray replacement module, written in RPython. +This module is not complete, but if it works it should give correct answers. +Its performance is hard to predict exactly. For regular NumPy source code that handles large arrays, it is likely to be slower than -the native NumPy with cpyext. It is faster on the kind of code that -contains many Python loops doing things on an element-by-element basis. +the standard NumPy. It is faster on pure python code that loop over ndarrays +doing things on an element-by-element basis. .. __: https://bitbucket.org/pypy/numpy @@ -278,11 +268,13 @@ sudo pypy -c 'import numpy' -Note again that this version is still a work-in-progress: many things do +Note again that this version is incomplete: many things do not work and those that do may not be any faster than NumPy on CPython. -For further instructions see `the pypy/numpy repository`__. +For further instructions see `the pypy/numpy repository`__ and the +`FAQ question`_ about the difference between the two. .. __: https://bitbucket.org/pypy/numpy +.. _`FAQ question`: http://doc.pypy.org/en/latest/faq.html#should-i-install-numpy-or-numpypy .. _translate: From pypy.commits at gmail.com Sat Oct 28 15:19:24 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 28 Oct 2017 12:19:24 -0700 (PDT) Subject: [pypy-commit] pypy default: move test summary to end on terminal, after the multiple pages of failure info, typos Message-ID: <59f4d83c.09a0df0a.b232.9b2c@mx.google.com> Author: Matti Picus Branch: Changeset: r92866:0edecbafb119 Date: 2017-10-28 22:18 +0300 http://bitbucket.org/pypy/pypy/changeset/0edecbafb119/ Log: move test summary to end on terminal, after the multiple pages of failure info, typos diff --git a/_pytest/terminal.py b/_pytest/terminal.py --- a/_pytest/terminal.py +++ b/_pytest/terminal.py @@ -366,11 +366,11 @@ EXIT_OK, EXIT_TESTSFAILED, EXIT_INTERRUPTED, EXIT_USAGEERROR, EXIT_NOTESTSCOLLECTED) if exitstatus in summary_exit_codes: - self.config.hook.pytest_terminal_summary(terminalreporter=self) self.summary_errors() self.summary_failures() self.summary_warnings() self.summary_passes() + self.config.hook.pytest_terminal_summary(terminalreporter=self) if exitstatus == EXIT_INTERRUPTED: self._report_keyboardinterrupt() del self._keyboardinterrupt_memo diff --git a/pypy/test_all.py b/pypy/test_all.py --- a/pypy/test_all.py +++ b/pypy/test_all.py @@ -27,7 +27,7 @@ sys.path.insert(0,os.path.dirname(os.path.dirname(os.path.realpath(__file__)))) import pytest if sys.platform == 'win32': - #Try to avoid opeing a dialog box if one of the tests causes a system error + #Try to avoid opening a dialog box if one of the tests causes a system error # We do this in runner.py, but buildbots run twisted which ruins inheritance # in windows subprocesses. import ctypes diff --git a/testrunner/runner.py b/testrunner/runner.py --- a/testrunner/runner.py +++ b/testrunner/runner.py @@ -21,7 +21,7 @@ win32api.CloseHandle(proch) except pywintypes.error, e: pass - #Try to avoid opeing a dialog box if one of the tests causes a system error + #Try to avoid opening a dialog box if one of the tests causes a system error import ctypes winapi = ctypes.windll.kernel32 SetErrorMode = winapi.SetErrorMode From pypy.commits at gmail.com Sat Oct 28 16:42:24 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 28 Oct 2017 13:42:24 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-appexec: make appexec for applevel tests identical to default Message-ID: <59f4ebb0.83b91c0a.d0a5b.38a0@mx.google.com> Author: Matti Picus Branch: py3.5-appexec Changeset: r92868:ec989ce318a9 Date: 2017-10-28 23:40 +0300 http://bitbucket.org/pypy/pypy/changeset/ec989ce318a9/ Log: make appexec for applevel tests identical to default diff --git a/pypy/tool/pytest/objspace.py b/pypy/tool/pytest/objspace.py --- a/pypy/tool/pytest/objspace.py +++ b/pypy/tool/pytest/objspace.py @@ -79,7 +79,9 @@ body = body.lstrip() assert body.startswith('(') src = py.code.Source("def anonymous" + body) - return (src, args) + d = {} + exec src.compile() in d + return d['anonymous'](*args) def wrap(self, obj): if isinstance(obj, str): From pypy.commits at gmail.com Sat Oct 28 16:42:22 2017 From: pypy.commits at gmail.com (mattip) Date: Sat, 28 Oct 2017 13:42:22 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: tweak run_with_python() Message-ID: <59f4ebae.4f931c0a.1441.2f84@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92867:2e9a42fcdebf Date: 2017-10-28 23:23 +0300 http://bitbucket.org/pypy/pypy/changeset/2e9a42fcdebf/ Log: tweak run_with_python() diff --git a/pypy/tool/pytest/apptest.py b/pypy/tool/pytest/apptest.py --- a/pypy/tool/pytest/apptest.py +++ b/pypy/tool/pytest/apptest.py @@ -12,7 +12,7 @@ from pypy.interpreter.gateway import app2interp_temp from pypy.interpreter.error import OperationError from pypy.interpreter.function import Method -from rpython.tool import runsubprocess +from rpython.tool.runsubprocess import run_subprocess from pypy.tool.pytest import appsupport from pypy.tool.pytest.objspace import gettestobjspace from rpython.tool.udir import udir @@ -67,14 +67,10 @@ def _rename_module(name): return str(RENAMED_USEMODULES.get(name, name)) - -def run_with_python(python_, target_, usemodules, **definitions): - if python_ is None: - py.test.skip("Cannot find the default python3 interpreter to run with -A") - # we assume that the source of target_ is in utf-8. Unfortunately, we don't - # have any easy/standard way to determine from here the original encoding - # of the source file - helpers = r"""# -*- encoding: utf-8 -*- +# we assume that the source of target_ is in utf-8. Unfortunately, we don't +# have any easy/standard way to determine from here the original encoding +# of the source file +helpers = r"""# -*- encoding: utf-8 -*- if 1: import sys sys.path.append(%r) @@ -109,6 +105,10 @@ pass self = Test() """ + +def run_with_python(python_, target_, usemodules, **definitions): + if python_ is None: + py.test.skip("Cannot find the default python3 interpreter to run with -A") defs = [] for symbol, value in sorted(definitions.items()): if isinstance(value, tuple) and isinstance(value[0], py.code.Source): @@ -181,7 +181,7 @@ helper_dir = os.path.join(pypydir, 'tool', 'cpyext') env = os.environ.copy() env['PYTHONPATH'] = helper_dir - res, stdout, stderr = runsubprocess.run_subprocess( + res, stdout, stderr = run_subprocess( python_, [str(pyfile)], env=env) print pyfile.read() print >> sys.stdout, stdout From pypy.commits at gmail.com Sat Oct 28 18:58:11 2017 From: pypy.commits at gmail.com (wlav) Date: Sat, 28 Oct 2017 15:58:11 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: cleanup Message-ID: <59f50b83.46c3df0a.23392.5604@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92869:b9ac9a79e9e5 Date: 2017-10-12 13:34 -0700 http://bitbucket.org/pypy/pypy/changeset/b9ac9a79e9e5/ Log: cleanup diff --git a/pypy/module/_cppyy/test/test_datatypes.py b/pypy/module/_cppyy/test/test_datatypes.py --- a/pypy/module/_cppyy/test/test_datatypes.py +++ b/pypy/module/_cppyy/test/test_datatypes.py @@ -149,15 +149,15 @@ # integer types names = ['short', 'ushort', 'int', 'uint', 'long', 'ulong', 'llong', 'ullong'] for i in range(len(names)): - exec 'c.m_%s = %d' % (names[i],i) + setattr(c, 'm_'+names[i], i) assert eval('c.get_%s()' % names[i]) == i for i in range(len(names)): - exec 'c.set_%s(%d)' % (names[i],2*i) + getattr(c, 'set_'+names[i])(2*i) assert eval('c.m_%s' % names[i]) == 2*i for i in range(len(names)): - exec 'c.set_%s_cr(%d)' % (names[i],3*i) + getattr(c, 'set_'+names[i]+'_cr')(3*i) assert eval('c.m_%s' % names[i]) == 3*i # float types through functions @@ -182,11 +182,11 @@ atypes = ['h', 'H', 'i', 'I', 'l', 'L' ] for j in range(len(names)): b = array.array(atypes[j], a) - exec 'c.m_%s_array = b' % names[j] # buffer copies + setattr(c, 'm_'+names[j]+'_array', b) # buffer copies for i in range(self.N): assert eval('c.m_%s_array[i]' % names[j]) == b[i] - exec 'c.m_%s_array2 = b' % names[j] # pointer copies + setattr(c, 'm_'+names[j]+'_array2', b) # pointer copies b[i] = 28 for i in range(self.N): assert eval('c.m_%s_array2[i]' % names[j]) == b[i] @@ -241,10 +241,10 @@ assert isinstance(c, CppyyTestData) # char types - assert CppyyTestData.s_char == 'c' - assert c.s_char == 'c' - assert c.s_uchar == 'u' - assert CppyyTestData.s_uchar == 'u' + assert CppyyTestData.s_char == 'c' + assert c.s_char == 'c' + assert c.s_uchar == 'u' + assert CppyyTestData.s_uchar == 'u' # integer types assert CppyyTestData.s_short == -101 @@ -255,20 +255,20 @@ assert c.s_int == -202 assert c.s_uint == 202 assert CppyyTestData.s_uint == 202 - assert CppyyTestData.s_long == -303L - assert c.s_long == -303L - assert c.s_ulong == 303L - assert CppyyTestData.s_ulong == 303L - assert CppyyTestData.s_llong == -404L - assert c.s_llong == -404L - assert c.s_ullong == 404L - assert CppyyTestData.s_ullong == 404L + assert CppyyTestData.s_long == -303 + assert c.s_long == -303 + assert c.s_ulong == 303 + assert CppyyTestData.s_ulong == 303 + assert CppyyTestData.s_llong == -404 + assert c.s_llong == -404 + assert c.s_ullong == 404 + assert CppyyTestData.s_ullong == 404 # floating point types - assert round(CppyyTestData.s_float + 606., 5) == 0 - assert round(c.s_float + 606., 5) == 0 - assert round(CppyyTestData.s_double + 707., 8) == 0 - assert round(c.s_double + 707., 8) == 0 + assert round(CppyyTestData.s_float + 606., 5) == 0 + assert round(c.s_float + 606., 5) == 0 + assert round(CppyyTestData.s_double + 707., 8) == 0 + assert round(c.s_double + 707., 8) == 0 c.__destruct__() @@ -282,56 +282,56 @@ assert isinstance(c, CppyyTestData) # char types - CppyyTestData.s_char = 'a' - assert c.s_char == 'a' - c.s_char = 'b' - assert CppyyTestData.s_char == 'b' - CppyyTestData.s_uchar = 'c' - assert c.s_uchar == 'c' - c.s_uchar = 'd' - assert CppyyTestData.s_uchar == 'd' + CppyyTestData.s_char = 'a' + assert c.s_char == 'a' + c.s_char = 'b' + assert CppyyTestData.s_char == 'b' + CppyyTestData.s_uchar = 'c' + assert c.s_uchar == 'c' + c.s_uchar = 'd' + assert CppyyTestData.s_uchar == 'd' raises(ValueError, setattr, CppyyTestData, 's_uchar', -1) - raises(ValueError, setattr, c, 's_uchar', -1) + raises(ValueError, setattr, c, 's_uchar', -1) # integer types c.s_short = -102 - assert CppyyTestData.s_short == -102 - CppyyTestData.s_short = -203 + assert CppyyTestData.s_short == -102 + CppyyTestData.s_short = -203 assert c.s_short == -203 c.s_ushort = 127 - assert CppyyTestData.s_ushort == 127 - CppyyTestData.s_ushort = 227 + assert CppyyTestData.s_ushort == 127 + CppyyTestData.s_ushort = 227 assert c.s_ushort == 227 - CppyyTestData.s_int = -234 + CppyyTestData.s_int = -234 assert c.s_int == -234 c.s_int = -321 - assert CppyyTestData.s_int == -321 - CppyyTestData.s_uint = 1234 + assert CppyyTestData.s_int == -321 + CppyyTestData.s_uint = 1234 assert c.s_uint == 1234 c.s_uint = 4321 - assert CppyyTestData.s_uint == 4321 - raises(ValueError, setattr, c, 's_uint', -1) + assert CppyyTestData.s_uint == 4321 + raises(ValueError, setattr, c, 's_uint', -1) raises(ValueError, setattr, CppyyTestData, 's_uint', -1) - CppyyTestData.s_long = -87L - assert c.s_long == -87L - c.s_long = 876L - assert CppyyTestData.s_long == 876L - CppyyTestData.s_ulong = 876L - assert c.s_ulong == 876L - c.s_ulong = 678L - assert CppyyTestData.s_ulong == 678L + CppyyTestData.s_long = -87 + assert c.s_long == -87 + c.s_long = 876 + assert CppyyTestData.s_long == 876 + CppyyTestData.s_ulong = 876 + assert c.s_ulong == 876 + c.s_ulong = 678 + assert CppyyTestData.s_ulong == 678 raises(ValueError, setattr, CppyyTestData, 's_ulong', -1) - raises(ValueError, setattr, c, 's_ulong', -1) + raises(ValueError, setattr, c, 's_ulong', -1) # floating point types - CppyyTestData.s_float = -3.1415 - assert round(c.s_float, 5 ) == -3.1415 + CppyyTestData.s_float = -3.1415 + assert round(c.s_float, 5) == -3.1415 c.s_float = 3.1415 - assert round(CppyyTestData.s_float, 5 ) == 3.1415 + assert round(CppyyTestData.s_float, 5) == 3.1415 import math c.s_double = -math.pi - assert CppyyTestData.s_double == -math.pi - CppyyTestData.s_double = math.pi + assert CppyyTestData.s_double == -math.pi + CppyyTestData.s_double = math.pi assert c.s_double == math.pi c.__destruct__() From pypy.commits at gmail.com Sat Oct 28 18:58:13 2017 From: pypy.commits at gmail.com (wlav) Date: Sat, 28 Oct 2017 15:58:13 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: consistency with CPython/cppyy (incl. capi and test upgrades) Message-ID: <59f50b85.d0e61c0a.20503.2bfa@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92870:c10f636528b7 Date: 2017-10-28 10:14 -0700 http://bitbucket.org/pypy/pypy/changeset/c10f636528b7/ Log: consistency with CPython/cppyy (incl. capi and test upgrades) diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -21,9 +21,9 @@ # match for the qualified type. -def get_rawobject(space, w_obj): +def get_rawobject(space, w_obj, can_be_None=True): from pypy.module._cppyy.interp_cppyy import W_CPPClass - cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=can_be_None) if cppinstance: rawobject = cppinstance.get_rawobject() assert lltype.typeOf(rawobject) == capi.C_OBJECT @@ -48,17 +48,16 @@ return capi.C_NULL_OBJECT def is_nullpointer_specialcase(space, w_obj): - # 0, None, and nullptr may serve as "NULL", check for any of them + # 0 and nullptr may serve as "NULL" # integer 0 try: return space.int_w(w_obj) == 0 except Exception: pass - # None or nullptr + # C++-style nullptr from pypy.module._cppyy import interp_cppyy - return space.is_true(space.is_(w_obj, space.w_None)) or \ - space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) + return space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) def get_rawbuffer(space, w_obj): # raw buffer @@ -74,7 +73,7 @@ return rffi.cast(rffi.VOIDP, space.uint_w(arr.getbuffer(space))) except Exception: pass - # pre-defined NULL + # pre-defined nullptr if is_nullpointer_specialcase(space, w_obj): return rffi.cast(rffi.VOIDP, 0) raise TypeError("not an addressable buffer") @@ -392,6 +391,7 @@ _immutable_fields_ = ['typecode'] typecode = 'g' + class CStringConverter(TypeConverter): def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.LONGP, address) @@ -408,18 +408,27 @@ def free_argument(self, space, arg, call_local): lltype.free(rffi.cast(rffi.CCHARPP, arg)[0], flavor='raw') +class CStringConverterWithSize(CStringConverter): + _immutable_fields_ = ['size'] + + def __init__(self, space, extra): + self.size = extra + + def from_memory(self, space, w_obj, w_pycppclass, offset): + address = self._get_raw_address(space, w_obj, offset) + charpptr = rffi.cast(rffi.CCHARP, address) + strsize = self.size + if charpptr[self.size-1] == '\0': + strsize = self.size-1 # rffi will add \0 back + return space.newbytes(rffi.charpsize2str(charpptr, strsize)) + class VoidPtrConverter(TypeConverter): def _unwrap_object(self, space, w_obj): try: obj = get_rawbuffer(space, w_obj) except TypeError: - try: - # TODO: accept a 'capsule' rather than naked int - # (do accept int(0), though) - obj = rffi.cast(rffi.VOIDP, space.uint_w(w_obj)) - except Exception: - obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) + obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj, False)) return obj def cffi_type(self, space): @@ -463,12 +472,12 @@ def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.VOIDPP, address) ba = rffi.cast(rffi.CCHARP, address) - r = rffi.cast(rffi.VOIDPP, call_local) try: - r[0] = get_rawbuffer(space, w_obj) + x[0] = get_rawbuffer(space, w_obj) except TypeError: + r = rffi.cast(rffi.VOIDPP, call_local) r[0] = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) - x[0] = rffi.cast(rffi.VOIDP, call_local) + x[0] = rffi.cast(rffi.VOIDP, call_local) ba[capi.c_function_arg_typeoffset(space)] = self.typecode def finalize_call(self, space, w_obj, call_local): @@ -687,7 +696,7 @@ _converters = {} # builtin and custom types _a_converters = {} # array and ptr versions of above -def get_converter(space, name, default): +def get_converter(space, _name, default): # The matching of the name to a converter should follow: # 1) full, exact match # 1a) const-removed match @@ -695,9 +704,9 @@ # 3) accept ref as pointer (for the stubs, const& can be # by value, but that does not work for the ffi path) # 4) generalized cases (covers basically all user classes) - # 5) void converter, which fails on use + # 5) void* or void converter (which fails on use) - name = capi.c_resolve_name(space, name) + name = capi.c_resolve_name(space, _name) # 1) full, exact match try: @@ -716,7 +725,7 @@ clean_name = capi.c_resolve_name(space, helper.clean_type(name)) try: # array_index may be negative to indicate no size or no size found - array_size = helper.array_size(name) + array_size = helper.array_size(_name) # uses original arg return _a_converters[clean_name+compound](space, array_size) except KeyError: pass @@ -743,11 +752,13 @@ elif capi.c_is_enum(space, clean_name): return _converters['unsigned'](space, default) - # 5) void converter, which fails on use - # + # 5) void* or void converter (which fails on use) + if 0 <= compound.find('*'): + return VoidPtrConverter(space, default) # "user knows best" + # return a void converter here, so that the class can be build even - # when some types are unknown; this overload will simply fail on use - return VoidConverter(space, name) + # when some types are unknown + return VoidConverter(space, name) # fails on use _converters["bool"] = BoolConverter @@ -864,6 +875,10 @@ for name in names: _a_converters[name+'[]'] = ArrayConverter _a_converters[name+'*'] = PtrConverter + + # special case, const char* w/ size and w/o '\0' + _a_converters["const char[]"] = CStringConverterWithSize + _build_array_converters() # add another set of aliased names diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -623,7 +623,21 @@ 'CPPConstructorOverload', is_static = interp2app(W_CPPConstructorOverload.is_static), call = interp2app(W_CPPConstructorOverload.call), - prototype = interp2app(W_CPPOverload.prototype), + prototype = interp2app(W_CPPConstructorOverload.prototype), +) + + +class W_CPPTemplateOverload(W_CPPOverload): + @unwrap_spec(args_w='args_w') + def __getitem__(self, args_w): + pass + + def __repr__(self): + return "W_CPPTemplateOverload(%s)" % [f.prototype() for f in self.functions] + +W_CPPTemplateOverload.typedef = TypeDef( + 'CPPTemplateOverload', + __getitem__ = interp2app(W_CPPTemplateOverload.call), ) @@ -637,6 +651,9 @@ def __call__(self, args_w): return self.method.bound_call(self.cppthis, args_w) + def __repr__(self): + return "W_CPPBoundMethod(%s)" % [f.prototype() for f in self.functions] + W_CPPBoundMethod.typedef = TypeDef( 'CPPBoundMethod', __call__ = interp2app(W_CPPBoundMethod.__call__), @@ -667,7 +684,7 @@ def get(self, w_cppinstance, w_pycppclass): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) return self.converter.from_memory(self.space, w_cppinstance, w_pycppclass, offset) @@ -675,7 +692,7 @@ def set(self, w_cppinstance, w_value): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) self.converter.to_memory(self.space, w_cppinstance, w_value, offset) @@ -1161,7 +1178,7 @@ W_CPPClass.typedef = TypeDef( 'CPPClass', - _python_owns = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), + __python_owns__ = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), __init__ = interp2app(W_CPPClass.instance__init__), __eq__ = interp2app(W_CPPClass.instance__eq__), __ne__ = interp2app(W_CPPClass.instance__ne__), @@ -1262,7 +1279,7 @@ except TypeError: pass # attempt to get address of C++ instance - return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj)) + return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj, False)) @unwrap_spec(w_obj=W_Root) def addressof(space, w_obj): diff --git a/pypy/module/_cppyy/src/dummy_backend.cxx b/pypy/module/_cppyy/src/dummy_backend.cxx --- a/pypy/module/_cppyy/src/dummy_backend.cxx +++ b/pypy/module/_cppyy/src/dummy_backend.cxx @@ -955,7 +955,13 @@ return cppstring_to_cstring(""); } -char* cppyy_method_signature(cppyy_scope_t /* handle */, cppyy_index_t /* method_index */) { +char* cppyy_method_signature( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { + return cppstring_to_cstring(""); +} + +char* cppyy_method_prototype( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { return cppstring_to_cstring(""); } diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile --- a/pypy/module/_cppyy/test/Makefile +++ b/pypy/module/_cppyy/test/Makefile @@ -7,7 +7,8 @@ fragileDict.so \ operatorsDict.so \ overloadsDict.so \ - stltypesDict.so + stltypesDict.so \ + templatesDict.so all : $(dicts) diff --git a/pypy/module/_cppyy/test/advancedcpp.cxx b/pypy/module/_cppyy/test/advancedcpp.cxx --- a/pypy/module/_cppyy/test/advancedcpp.cxx +++ b/pypy/module/_cppyy/test/advancedcpp.cxx @@ -106,17 +106,6 @@ } -// more template testing -long my_templated_method_class::get_size() { return -1; } - -long my_templated_method_class::get_char_size() { return (long)sizeof(char); } -long my_templated_method_class::get_int_size() { return (long)sizeof(int); } -long my_templated_method_class::get_long_size() { return (long)sizeof(long); } -long my_templated_method_class::get_float_size() { return (long)sizeof(float); } -long my_templated_method_class::get_double_size() { return (long)sizeof(double); } -long my_templated_method_class::get_self_size() { return (long)sizeof(my_templated_method_class); } - - // overload order testing int overload_one_way::gime() const { return 1; } std::string overload_one_way::gime() { return "aap"; } diff --git a/pypy/module/_cppyy/test/advancedcpp.h b/pypy/module/_cppyy/test/advancedcpp.h --- a/pypy/module/_cppyy/test/advancedcpp.h +++ b/pypy/module/_cppyy/test/advancedcpp.h @@ -246,8 +246,6 @@ int m_i; }; -template class std::vector; - //=========================================================================== class some_convertible { // for math conversions testing @@ -275,6 +273,7 @@ extern double my_global_double; // a couple of globals for access testing extern double my_global_array[500]; extern double* my_global_ptr; +static const char my_global_string[] = "aap " " noot " " mies"; //=========================================================================== class some_class_with_data { // for life-line and identity testing @@ -387,37 +386,6 @@ template char my_templated_function(char); template double my_templated_function(double); -class my_templated_method_class { -public: - long get_size(); // to get around bug in genreflex - template long get_size(); - - long get_char_size(); - long get_int_size(); - long get_long_size(); - long get_float_size(); - long get_double_size(); - - long get_self_size(); - -private: - double m_data[3]; -}; - -template -inline long my_templated_method_class::get_size() { - return sizeof(B); -} - -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); - -typedef my_templated_method_class my_typedef_t; -template long my_templated_method_class::get_size(); - //=========================================================================== class overload_one_way { // overload order testing diff --git a/pypy/module/_cppyy/test/advancedcpp.xml b/pypy/module/_cppyy/test/advancedcpp.xml --- a/pypy/module/_cppyy/test/advancedcpp.xml +++ b/pypy/module/_cppyy/test/advancedcpp.xml @@ -53,8 +53,6 @@ - - diff --git a/pypy/module/_cppyy/test/fragile.h b/pypy/module/_cppyy/test/fragile.h --- a/pypy/module/_cppyy/test/fragile.h +++ b/pypy/module/_cppyy/test/fragile.h @@ -30,9 +30,11 @@ void overload(int, no_such_class* p = 0) {} }; +static const int dummy_location = 0xdead; + class E { public: - E() : m_pp_no_such(0), m_pp_a(0) {} + E() : m_pp_no_such((no_such_class**)&dummy_location), m_pp_a(0) {} virtual int check() { return (int)'E'; } void overload(no_such_class**) {} diff --git a/pypy/module/_cppyy/test/templates.cxx b/pypy/module/_cppyy/test/templates.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.cxx @@ -0,0 +1,12 @@ +#include "templates.h" + + +// template methods +long MyTemplatedMethodClass::get_size() { return -1; } + +long MyTemplatedMethodClass::get_char_size() { return (long)sizeof(char); } +long MyTemplatedMethodClass::get_int_size() { return (long)sizeof(int); } +long MyTemplatedMethodClass::get_long_size() { return (long)42; /* "lying" */ } +long MyTemplatedMethodClass::get_float_size() { return (long)sizeof(float); } +long MyTemplatedMethodClass::get_double_size() { return (long)sizeof(double); } +long MyTemplatedMethodClass::get_self_size() { return (long)sizeof(MyTemplatedMethodClass); } diff --git a/pypy/module/_cppyy/test/templates.h b/pypy/module/_cppyy/test/templates.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.h @@ -0,0 +1,35 @@ +//=========================================================================== +class MyTemplatedMethodClass { // template methods +public: + long get_size(); // to get around bug in genreflex + template long get_size(); + + long get_char_size(); + long get_int_size(); + long get_long_size(); + long get_float_size(); + long get_double_size(); + + long get_self_size(); + +private: + double m_data[3]; +}; + +template +inline long MyTemplatedMethodClass::get_size() { + return sizeof(B); +} + +// +typedef MyTemplatedMethodClass MyTMCTypedef_t; + +// explicit instantiation +template long MyTemplatedMethodClass::get_size(); +template long MyTemplatedMethodClass::get_size(); + +// "lying" specialization +template<> +inline long MyTemplatedMethodClass::get_size() { + return 42; +} diff --git a/pypy/module/_cppyy/test/templates.xml b/pypy/module/_cppyy/test/templates.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/test_advancedcpp.py b/pypy/module/_cppyy/test/test_advancedcpp.py --- a/pypy/module/_cppyy/test/test_advancedcpp.py +++ b/pypy/module/_cppyy/test/test_advancedcpp.py @@ -28,9 +28,9 @@ def test01_default_arguments(self): """Test usage of default arguments""" - import _cppyy + import _cppyy as cppyy def test_defaulter(n, t): - defaulter = getattr(_cppyy.gbl, '%s_defaulter' % n) + defaulter = getattr(cppyy.gbl, '%s_defaulter' % n) d = defaulter() assert d.m_a == t(11) @@ -55,23 +55,23 @@ assert d.m_b == t(4) assert d.m_c == t(5) d.__destruct__() - test_defaulter('short', int) + test_defaulter('short', int) test_defaulter('ushort', int) - test_defaulter('int', int) - test_defaulter('uint', int) - test_defaulter('long', long) - test_defaulter('ulong', long) - test_defaulter('llong', long) + test_defaulter('int', int) + test_defaulter('uint', int) + test_defaulter('long', long) + test_defaulter('ulong', long) + test_defaulter('llong', long) test_defaulter('ullong', long) - test_defaulter('float', float) + test_defaulter('float', float) test_defaulter('double', float) def test02_simple_inheritance(self): """Test binding of a basic inheritance structure""" - import _cppyy - base_class = _cppyy.gbl.base_class - derived_class = _cppyy.gbl.derived_class + import _cppyy as cppyy + base_class = cppyy.gbl.base_class + derived_class = cppyy.gbl.derived_class assert issubclass(derived_class, base_class) assert not issubclass(base_class, derived_class) @@ -123,8 +123,8 @@ def test03_namespaces(self): """Test access to namespaces and inner classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.a_ns is gbl.a_ns assert gbl.a_ns.d_ns is gbl.a_ns.d_ns @@ -150,8 +150,8 @@ def test03a_namespace_lookup_on_update(self): """Test whether namespaces can be shared across dictionaries.""" - import _cppyy, ctypes - gbl = _cppyy.gbl + import _cppyy as cppyy, ctypes + gbl = cppyy.gbl lib2 = ctypes.CDLL("./advancedcpp2Dict.so", ctypes.RTLD_GLOBAL) @@ -179,8 +179,8 @@ def test04_template_types(self): """Test bindings of templated types""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.T1 is gbl.T1 assert gbl.T2 is gbl.T2 @@ -245,8 +245,8 @@ def test05_abstract_classes(self): """Test non-instatiatability of abstract classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl raises(TypeError, gbl.a_class) raises(TypeError, gbl.some_abstract_class) @@ -260,12 +260,12 @@ def test06_datamembers(self): """Test data member access when using virtual inheritence""" - import _cppyy - a_class = _cppyy.gbl.a_class - b_class = _cppyy.gbl.b_class - c_class_1 = _cppyy.gbl.c_class_1 - c_class_2 = _cppyy.gbl.c_class_2 - d_class = _cppyy.gbl.d_class + import _cppyy as cppyy + a_class = cppyy.gbl.a_class + b_class = cppyy.gbl.b_class + c_class_1 = cppyy.gbl.c_class_1 + c_class_2 = cppyy.gbl.c_class_2 + d_class = cppyy.gbl.d_class assert issubclass(b_class, a_class) assert issubclass(c_class_1, a_class) @@ -354,8 +354,8 @@ def test07_pass_by_reference(self): """Test reference passing when using virtual inheritance""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl b_class = gbl.b_class c_class = gbl.c_class_2 d_class = gbl.d_class @@ -387,71 +387,75 @@ def test08_void_pointer_passing(self): """Test passing of variants of void pointer arguments""" - import _cppyy - pointer_pass = _cppyy.gbl.pointer_pass - some_concrete_class = _cppyy.gbl.some_concrete_class + import _cppyy as cppyy + pointer_pass = cppyy.gbl.pointer_pass + some_concrete_class = cppyy.gbl.some_concrete_class pp = pointer_pass() o = some_concrete_class() - assert _cppyy.addressof(o) == pp.gime_address_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ref(o) + assert cppyy.addressof(o) == pp.gime_address_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ref(o) import array - addressofo = array.array('l', [_cppyy.addressof(o)]) - assert addressofo.buffer_info()[0] == pp.gime_address_ptr_ptr(addressofo) + addressofo = array.array('l', [cppyy.addressof(o)]) + assert addressofo[0] == pp.gime_address_ptr_ptr(addressofo) assert 0 == pp.gime_address_ptr(0) - assert 0 == pp.gime_address_ptr(None) + raises(TypeError, pp.gime_address_ptr, None) - ptr = _cppyy.bind_object(0, some_concrete_class) - assert _cppyy.addressof(ptr) == 0 + ptr = cppyy.bind_object(0, some_concrete_class) + assert cppyy.addressof(ptr) == 0 pp.set_address_ptr_ref(ptr) - assert _cppyy.addressof(ptr) == 0x1234 + assert cppyy.addressof(ptr) == 0x1234 pp.set_address_ptr_ptr(ptr) - assert _cppyy.addressof(ptr) == 0x4321 + assert cppyy.addressof(ptr) == 0x4321 + + assert cppyy.addressof(cppyy.nullptr) == 0 + raises(TypeError, cppyy.addressof, None) + assert cppyy.addressof(0) == 0 def test09_opaque_pointer_passing(self): """Test passing around of opaque pointers""" - import _cppyy - some_concrete_class = _cppyy.gbl.some_concrete_class + import _cppyy as cppyy + some_concrete_class = cppyy.gbl.some_concrete_class o = some_concrete_class() # TODO: figure out the PyPy equivalent of CObject (may have to do this # through the C-API from C++) - #cobj = _cppyy.as_cobject(o) - addr = _cppyy.addressof(o) + #cobj = cppyy.as_cobject(o) + addr = cppyy.addressof(o) - #assert o == _cppyy.bind_object(cobj, some_concrete_class) - #assert o == _cppyy.bind_object(cobj, type(o)) - #assert o == _cppyy.bind_object(cobj, o.__class__) - #assert o == _cppyy.bind_object(cobj, "some_concrete_class") - assert _cppyy.addressof(o) == _cppyy.addressof(_cppyy.bind_object(addr, some_concrete_class)) - assert o == _cppyy.bind_object(addr, some_concrete_class) - assert o == _cppyy.bind_object(addr, type(o)) - assert o == _cppyy.bind_object(addr, o.__class__) - assert o == _cppyy.bind_object(addr, "some_concrete_class") - raises(TypeError, _cppyy.bind_object, addr, "does_not_exist") - raises(TypeError, _cppyy.bind_object, addr, 1) + #assert o == cppyy.bind_object(cobj, some_concrete_class) + #assert o == cppyy.bind_object(cobj, type(o)) + #assert o == cppyy.bind_object(cobj, o.__class__) + #assert o == cppyy.bind_object(cobj, "some_concrete_class") + assert cppyy.addressof(o) == cppyy.addressof(cppyy.bind_object(addr, some_concrete_class)) + assert o == cppyy.bind_object(addr, some_concrete_class) + assert o == cppyy.bind_object(addr, type(o)) + assert o == cppyy.bind_object(addr, o.__class__) + assert o == cppyy.bind_object(addr, "some_concrete_class") + raises(TypeError, cppyy.bind_object, addr, "does_not_exist") + raises(TypeError, cppyy.bind_object, addr, 1) def test10_object_identity(self): """Test object identity""" - import _cppyy - some_concrete_class = _cppyy.gbl.some_concrete_class - some_class_with_data = _cppyy.gbl.some_class_with_data + import _cppyy as cppyy + some_concrete_class = cppyy.gbl.some_concrete_class + some_class_with_data = cppyy.gbl.some_class_with_data o = some_concrete_class() - addr = _cppyy.addressof(o) + addr = cppyy.addressof(o) - o2 = _cppyy.bind_object(addr, some_concrete_class) + o2 = cppyy.bind_object(addr, some_concrete_class) assert o is o2 - o3 = _cppyy.bind_object(addr, some_class_with_data) + o3 = cppyy.bind_object(addr, some_class_with_data) assert not o is o3 d1 = some_class_with_data() @@ -472,13 +476,13 @@ def test11_multi_methods(self): """Test calling of methods from multiple inheritance""" - import _cppyy - multi = _cppyy.gbl.multi + import _cppyy as cppyy + multi = cppyy.gbl.multi - assert _cppyy.gbl.multi1 is multi.__bases__[0] - assert _cppyy.gbl.multi2 is multi.__bases__[1] + assert cppyy.gbl.multi1 is multi.__bases__[0] + assert cppyy.gbl.multi2 is multi.__bases__[1] - dict_keys = multi.__dict__.keys() + dict_keys = list(multi.__dict__.keys()) assert dict_keys.count('get_my_own_int') == 1 assert dict_keys.count('get_multi1_int') == 0 assert dict_keys.count('get_multi2_int') == 0 @@ -491,9 +495,9 @@ def test12_actual_type(self): """Test that a pointer to base return does an auto-downcast""" - import _cppyy - base_class = _cppyy.gbl.base_class - derived_class = _cppyy.gbl.derived_class + import _cppyy as cppyy + base_class = cppyy.gbl.base_class + derived_class = cppyy.gbl.derived_class b = base_class() d = derived_class() @@ -520,85 +524,70 @@ assert not isinstance(voidp, base_class) assert not isinstance(voidp, derived_class) - d1 = _cppyy.bind_object(voidp, base_class, cast=True) + d1 = cppyy.bind_object(voidp, base_class, cast=True) assert isinstance(d1, derived_class) assert d1 is d - b1 = _cppyy.bind_object(voidp, base_class) + b1 = cppyy.bind_object(voidp, base_class) assert isinstance(b1, base_class) - assert _cppyy.addressof(b1) == _cppyy.addressof(d) + assert cppyy.addressof(b1) == cppyy.addressof(d) assert not (b1 is d) def test13_actual_type_virtual_multi(self): """Test auto-downcast in adverse inheritance situation""" - import _cppyy + import _cppyy as cppyy - c1 = _cppyy.gbl.create_c1() - assert type(c1) == _cppyy.gbl.c_class_1 + c1 = cppyy.gbl.create_c1() + assert type(c1) == cppyy.gbl.c_class_1 assert c1.m_c == 3 c1.__destruct__() - c2 = _cppyy.gbl.create_c2() - assert type(c2) == _cppyy.gbl.c_class_2 + c2 = cppyy.gbl.create_c2() + assert type(c2) == cppyy.gbl.c_class_2 assert c2.m_c == 3 c2.__destruct__() def test14_new_overloader(self): """Verify that class-level overloaded new/delete are called""" - import _cppyy + import _cppyy as cppyy - assert _cppyy.gbl.new_overloader.s_instances == 0 - nl = _cppyy.gbl.new_overloader() - assert _cppyy.gbl.new_overloader.s_instances == 1 + assert cppyy.gbl.new_overloader.s_instances == 0 + nl = cppyy.gbl.new_overloader() + assert cppyy.gbl.new_overloader.s_instances == 1 nl.__destruct__() import gc gc.collect() - assert _cppyy.gbl.new_overloader.s_instances == 0 + assert cppyy.gbl.new_overloader.s_instances == 0 def test15_template_instantiation_with_vector_of_float(self): """Test template instantiation with a std::vector""" - import _cppyy + import _cppyy as cppyy # the following will simply fail if there is a naming problem (e.g. # std::, allocator, etc., etc.); note the parsing required ... - b = _cppyy.gbl.my_templated_class(_cppyy.gbl.std.vector(float))() + b = cppyy.gbl.my_templated_class(cppyy.gbl.std.vector(float))() for i in range(5): b.m_b.push_back(i) assert round(b.m_b[i], 5) == float(i) - def test16_template_member_functions(self): - """Test template member functions lookup and calls""" - - import _cppyy - - m = _cppyy.gbl.my_templated_method_class() - - assert m.get_size('char')() == m.get_char_size() - assert m.get_size(int)() == m.get_int_size() - assert m.get_size(long)() == m.get_long_size() - assert m.get_size(float)() == m.get_float_size() - assert m.get_size('double')() == m.get_double_size() - assert m.get_size('my_templated_method_class')() == m.get_self_size() - assert m.get_size('my_typedef_t')() == m.get_self_size() - - def test17_template_global_functions(self): + def test16_template_global_functions(self): """Test template global function lookup and calls""" - import _cppyy + import _cppyy as cppyy - f = _cppyy.gbl.my_templated_function + f = cppyy.gbl.my_templated_function assert f('c') == 'c' assert type(f('c')) == type('c') assert f(3.) == 3. assert type(f(4.)) == type(4.) - def test18_assign_to_return_byref( self ): + def test17_assign_to_return_byref( self ): """Test assignment to an instance returned by reference""" from _cppyy import gbl @@ -614,7 +603,7 @@ # assert len(a) == 1 # assert a[0].m_i == 33 - def test19_math_converters(self): + def test18_math_converters(self): """Test operator int/long/double incl. typedef""" from _cppyy import gbl @@ -623,14 +612,14 @@ a.m_i = 1234 a.m_d = 4321. - assert int(a) == 1234 - assert int(a) == a.m_i - assert long(a) == a.m_i + assert int(a) == 1234 + assert int(a) == a.m_i + assert long(a) == a.m_i - assert float(a) == 4321. - assert float(a) == a.m_d + assert float(a) == 4321. + assert float(a) == a.m_d - def test20_comparator(self): + def test19_comparator(self): """Check that the global operator!=/== is picked up""" from _cppyy import gbl @@ -648,21 +637,22 @@ assert a.__eq__(a) == False assert b.__eq__(b) == False - def test21_overload_order_with_proper_return(self): + def test20_overload_order_with_proper_return(self): """Test return type against proper overload w/ const and covariance""" - import _cppyy + import _cppyy as cppyy - assert _cppyy.gbl.overload_one_way().gime() == 1 - assert _cppyy.gbl.overload_the_other_way().gime() == "aap" + assert cppyy.gbl.overload_one_way().gime() == 1 + assert cppyy.gbl.overload_the_other_way().gime() == "aap" - def test22_access_to_global_variables(self): + def test21_access_to_global_variables(self): """Access global_variables_and_pointers""" - import _cppyy + import _cppyy as cppyy - assert _cppyy.gbl.my_global_double == 12. - assert len(_cppyy.gbl.my_global_array) == 500 + assert cppyy.gbl.my_global_double == 12. + assert len(cppyy.gbl.my_global_array) == 500 + assert cppyy.gbl.my_global_string == "aap noot mies" # TODO: currently fails b/c double** not understood as &double* - #assert _cppyy.gbl.my_global_ptr[0] == 1234. + #assert cppyy.gbl.my_global_ptr[0] == 1234. diff --git a/pypy/module/_cppyy/test/test_cppyy.py b/pypy/module/_cppyy/test/test_cppyy.py --- a/pypy/module/_cppyy/test/test_cppyy.py +++ b/pypy/module/_cppyy/test/test_cppyy.py @@ -1,5 +1,4 @@ import py, os, sys -import subprocess from pypy.module._cppyy import interp_cppyy, executor from .support import setup_make @@ -165,8 +164,8 @@ e1 = self.instantiate(t, 7) assert t.get_overload("getCount").call(None) == 1 - assert e1._python_owns == True - e1._python_owns = False + assert e1.__python_owns__ == True + e1.__python_owns__ = False e1 = None gc.collect() assert t.get_overload("getCount").call(None) == 1 diff --git a/pypy/module/_cppyy/test/test_datatypes.py b/pypy/module/_cppyy/test/test_datatypes.py --- a/pypy/module/_cppyy/test/test_datatypes.py +++ b/pypy/module/_cppyy/test/test_datatypes.py @@ -21,8 +21,8 @@ def test01_instance_data_read_access(self): """Read access to instance public data and verify values""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -98,8 +98,8 @@ raises(IndexError, c.m_double_array.__getitem__, self.N) # can not access an instance member on the class - raises(ReferenceError, getattr, CppyyTestData, 'm_bool') - raises(ReferenceError, getattr, CppyyTestData, 'm_int') + raises(AttributeError, getattr, CppyyTestData, 'm_bool') + raises(AttributeError, getattr, CppyyTestData, 'm_int') assert not hasattr(CppyyTestData, 'm_bool') assert not hasattr(CppyyTestData, 'm_int') @@ -109,8 +109,8 @@ def test02_instance_data_write_access(self): """Test write access to instance public data and verify values""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -196,8 +196,8 @@ def test03_array_passing(self): """Test passing of array arguments""" - import _cppyy, array, sys - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy, array, sys + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -221,21 +221,20 @@ for i in range(self.N): assert ca[i] == b[i] - # NULL/None/nullptr passing (will use short*) + # NULL/nullptr passing (will use short*) assert not c.pass_array(0) raises(Exception, c.pass_array(0).__getitem__, 0) # raises SegfaultException - assert not c.pass_array(None) - raises(Exception, c.pass_array(None).__getitem__, 0) # id. - assert not c.pass_array(_cppyy.nullptr) - raises(Exception, c.pass_array(_cppyy.nullptr).__getitem__, 0) # id. id. + assert raises(TypeError, c.pass_array, None) + assert not c.pass_array(cppyy.nullptr) + raises(Exception, c.pass_array(cppyy.nullptr).__getitem__, 0) # id. id. c.__destruct__() def test04_class_read_access(self): """Test read access to class public data and verify values""" - import _cppyy, sys - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy, sys + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -275,8 +274,8 @@ def test05_class_data_write_access(self): """Test write access to class public data and verify values""" - import _cppyy, sys - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy, sys + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -339,8 +338,8 @@ def test06_range_access(self): """Test the ranges of integer types""" - import _cppyy, sys - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy, sys + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -355,8 +354,8 @@ def test07_type_conversions(self): """Test conversions between builtin types""" - import _cppyy, sys - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy, sys + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -373,8 +372,8 @@ def test08_global_builtin_type(self): """Test access to a global builtin type""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.g_int == gbl.get_global_int() @@ -389,8 +388,8 @@ def test09_global_ptr(self): """Test access of global objects through a pointer""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl raises(ReferenceError, 'gbl.g_pod.m_int') @@ -420,10 +419,10 @@ def test10_enum(self): """Test access to enums""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl - CppyyTestData = _cppyy.gbl.CppyyTestData + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -468,8 +467,8 @@ def test11_string_passing(self): """Test passing/returning of a const char*""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert c.get_valid_string('aap') == 'aap' @@ -478,8 +477,8 @@ def test12_copy_contructor(self): """Test copy constructor""" - import _cppyy - FourVector = _cppyy.gbl.FourVector + import _cppyy as cppyy + FourVector = cppyy.gbl.FourVector t1 = FourVector(1., 2., 3., -4.) t2 = FourVector(0., 0., 0., 0.) @@ -494,9 +493,9 @@ def test13_object_returns(self): """Test access to and return of PODs""" - import _cppyy + import _cppyy as cppyy - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() assert c.m_pod.m_int == 888 assert c.m_pod.m_double == 3.14 @@ -521,13 +520,13 @@ def test14_object_arguments(self): """Test setting and returning of a POD through arguments""" - import _cppyy + import _cppyy as cppyy - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() assert c.m_pod.m_int == 888 assert c.m_pod.m_double == 3.14 - p = _cppyy.gbl.CppyyTestPod() + p = cppyy.gbl.CppyyTestPod() p.m_int = 123 assert p.m_int == 123 p.m_double = 321. @@ -537,12 +536,12 @@ assert c.m_pod.m_int == 123 assert c.m_pod.m_double == 321. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_ptr_in(p) assert c.m_pod.m_int == 123 assert c.m_pod.m_double == 321. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_ptr_out(p) assert p.m_int == 888 assert p.m_double == 3.14 @@ -550,26 +549,26 @@ p.m_int = 555 p.m_double = 666. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_ref(p) assert c.m_pod.m_int == 555 assert c.m_pod.m_double == 666. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_ptrptr_in(p) assert c.m_pod.m_int == 555 assert c.m_pod.m_double == 666. assert p.m_int == 555 assert p.m_double == 666. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_void_ptrptr_in(p) assert c.m_pod.m_int == 555 assert c.m_pod.m_double == 666. assert p.m_int == 555 assert p.m_double == 666. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_ptrptr_out(p) assert c.m_pod.m_int == 888 assert c.m_pod.m_double == 3.14 @@ -579,7 +578,7 @@ p.m_int = 777 p.m_double = 888. - c = _cppyy.gbl.CppyyTestData() + c = cppyy.gbl.CppyyTestData() c.set_pod_void_ptrptr_out(p) assert c.m_pod.m_int == 888 assert c.m_pod.m_double == 3.14 @@ -587,12 +586,12 @@ assert p.m_double == 3.14 def test15_nullptr_passing(self): - """Integer 0 ('NULL') and None allowed to pass through instance*""" + """Integer 0 ('NULL') and nullptr allowed to pass through instance*""" - import _cppyy + import _cppyy as cppyy - for o in (0, None): - c = _cppyy.gbl.CppyyTestData() + for o in (0, cppyy.nullptr): + c = cppyy.gbl.CppyyTestData() assert c.m_pod.m_int == 888 assert c.m_pod.m_double == 3.14 assert not not c.m_ppod @@ -604,8 +603,8 @@ def test16_respect_privacy(self): """Test that privacy settings are respected""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() assert isinstance(c, CppyyTestData) @@ -617,26 +616,26 @@ def test17_object_and_pointer_comparisons(self): """Verify object and pointer comparisons""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl - c1 = _cppyy.bind_object(0, gbl.CppyyTestData) + c1 = cppyy.bind_object(0, gbl.CppyyTestData) assert c1 == None assert None == c1 - c2 = _cppyy.bind_object(0, gbl.CppyyTestData) + c2 = cppyy.bind_object(0, gbl.CppyyTestData) assert c1 == c2 assert c2 == c1 # FourVector overrides operator== - l1 = _cppyy.bind_object(0, gbl.FourVector) + l1 = cppyy.bind_object(0, gbl.FourVector) assert l1 == None assert None == l1 assert c1 != l1 assert l1 != c1 - l2 = _cppyy.bind_object(0, gbl.FourVector) + l2 = cppyy.bind_object(0, gbl.FourVector) assert l1 == l2 assert l2 == l1 @@ -668,8 +667,8 @@ def test19_buffer_reshaping(self): """Test usage of buffer sizing""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() for func in ['get_bool_array', 'get_bool_array2', @@ -680,43 +679,52 @@ 'get_ulong_array', 'get_ulong_array2']: arr = getattr(c, func)() arr = arr.shape.fromaddress(arr.itemaddress(0), self.N) + + """TODO: interface change ... + arr.reshape((self.N,)) + assert len(arr) == self.N + + raises(TypeError, arr.reshape, (1, 2)) + assert len(arr) == self.N + + raises(TypeError, arr.reshape, 2*self.N) assert len(arr) == self.N l = list(arr) for i in range(self.N): - assert arr[i] == l[i] + assert arr[i] == l[i]""" def test20_voidp(self): """Test usage of void* data""" - import _cppyy - CppyyTestData = _cppyy.gbl.CppyyTestData + import _cppyy as cppyy + CppyyTestData = cppyy.gbl.CppyyTestData c = CppyyTestData() - assert not _cppyy.nullptr + assert not cppyy.nullptr - assert c.s_voidp is _cppyy.nullptr - assert CppyyTestData.s_voidp is _cppyy.nullptr + assert c.s_voidp is cppyy.nullptr + assert CppyyTestData.s_voidp is cppyy.nullptr - assert c.m_voidp is _cppyy.nullptr - assert c.get_voidp() is _cppyy.nullptr + assert c.m_voidp is cppyy.nullptr + assert c.get_voidp() is cppyy.nullptr c2 = CppyyTestData() - assert c2.m_voidp is _cppyy.nullptr + assert c2.m_voidp is cppyy.nullptr c.set_voidp(c2.m_voidp) - assert c.m_voidp is _cppyy.nullptr + assert c.m_voidp is cppyy.nullptr c.set_voidp(c2.get_voidp()) - assert c.m_voidp is _cppyy.nullptr - c.set_voidp(_cppyy.nullptr) - assert c.m_voidp is _cppyy.nullptr + assert c.m_voidp is cppyy.nullptr + c.set_voidp(cppyy.nullptr) + assert c.m_voidp is cppyy.nullptr c.set_voidp(c2) def address_equality_test(a, b): - assert _cppyy.addressof(a) == _cppyy.addressof(b) - b2 = _cppyy.bind_object(a, CppyyTestData) + assert cppyy.addressof(a) == cppyy.addressof(b) + b2 = cppyy.bind_object(a, CppyyTestData) assert b is b2 # memory regulator recycles - b3 = _cppyy.bind_object(_cppyy.addressof(a), CppyyTestData) + b3 = cppyy.bind_object(cppyy.addressof(a), CppyyTestData) assert b is b3 # likewise address_equality_test(c.m_voidp, c2) @@ -724,8 +732,8 @@ def null_test(null): c.m_voidp = null - assert c.m_voidp is _cppyy.nullptr - map(null_test, [0, None, _cppyy.nullptr]) + assert c.m_voidp is cppyy.nullptr + map(null_test, [0, cppyy.nullptr]) c.m_voidp = c2 address_equality_test(c.m_voidp, c2) diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py --- a/pypy/module/_cppyy/test/test_fragile.py +++ b/pypy/module/_cppyy/test/test_fragile.py @@ -71,7 +71,9 @@ e = fragile.E() raises(TypeError, e.overload, None) - raises(TypeError, getattr, e, 'm_pp_no_such') + # allowing access to e.m_pp_no_such is debatable, but it provides a raw pointer + # which may be useful ... + assert e.m_pp_no_such[0] == 0xdead def test04_wrong_arg_addressof(self): """Test addressof() error reporting""" @@ -90,10 +92,7 @@ _cppyy.addressof(f) raises(TypeError, _cppyy.addressof, o) raises(TypeError, _cppyy.addressof, 1) - # 0, None, and nullptr allowed - assert _cppyy.addressof(0) == 0 - assert _cppyy.addressof(None) == 0 - assert _cppyy.addressof(_cppyy.nullptr) == 0 + # see also test08_void_pointer_passing in test_advancedcpp.py def test05_wrong_this(self): """Test that using an incorrect self argument raises""" diff --git a/pypy/module/_cppyy/test/test_templates.py b/pypy/module/_cppyy/test/test_templates.py new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/test_templates.py @@ -0,0 +1,42 @@ +import py, os, sys +from .support import setup_make + + +currpath = py.path.local(__file__).dirpath() +test_dct = str(currpath.join("templatesDict.so")) + +def setup_module(mod): + setup_make("templatesDict.so") + +class AppTestTEMPLATES: + spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools']) + + def setup_class(cls): + cls.w_test_dct = cls.space.newtext(test_dct) + cls.w_datatypes = cls.space.appexec([], """(): + import ctypes + return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, )) + + def test01_template_member_functions(self): + """Template member functions lookup and calls""" + + import _cppyy + + m = _cppyy.gbl.MyTemplatedMethodClass() + + return + + # pre-instantiated + assert m.get_size['char']() == m.get_char_size() + assert m.get_size[int]() == m.get_int_size() + + # specialized + assert m.get_size[long]() == m.get_long_size() + + # auto-instantiation + assert m.get_size[float]() == m.get_float_size() + assert m.get_size['double']() == m.get_double_size() + assert m.get_size['MyTemplatedMethodClass']() == m.get_self_size() + + # auto through typedef + assert m.get_size['MyTMCTypedef_t']() == m.get_self_size() From pypy.commits at gmail.com Sat Oct 28 18:58:17 2017 From: pypy.commits at gmail.com (wlav) Date: Sat, 28 Oct 2017 15:58:17 -0700 (PDT) Subject: [pypy-commit] pypy cppyy-packaging: merge default into branch for testing Message-ID: <59f50b89.d1b8df0a.ddbfa.fa95@mx.google.com> Author: Wim Lavrijsen Branch: cppyy-packaging Changeset: r92871:766bf4f67f39 Date: 2017-10-28 15:47 -0700 http://bitbucket.org/pypy/pypy/changeset/766bf4f67f39/ Log: merge default into branch for testing diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -71,6 +71,8 @@ ^lib_pypy/.+.c$ ^lib_pypy/.+.o$ ^lib_pypy/.+.so$ +^lib_pypy/.+.pyd$ +^lib_pypy/Release/ ^pypy/doc/discussion/.+\.html$ ^include/.+\.h$ ^include/.+\.inl$ diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,60 +8,63 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) - if '_type_' in typedict: - ffiarray = _rawffi.Array(typedict['_type_']._ffishape_) - res._ffiarray = ffiarray - subletter = getattr(typedict['_type_'], '_type_', None) - if subletter == 'c': - def getvalue(self): - return _rawffi.charp2string(self._buffer.buffer, - self._length_) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, str): - _rawffi.rawstring2charp(self._buffer.buffer, val) - else: - for i in range(len(val)): - self[i] = val[i] - if len(val) < self._length_: - self._buffer[len(val)] = '\x00' - res.value = property(getvalue, setvalue) - def getraw(self): - return _rawffi.charp2rawstring(self._buffer.buffer, - self._length_) + if cls == (_CData,): # this is the Array class defined below + res._ffiarray = None + return res + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") + ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) + subletter = getattr(res._type_, '_type_', None) + if subletter == 'c': + def getvalue(self): + return _rawffi.charp2string(self._buffer.buffer, + self._length_) + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, str): + _rawffi.rawstring2charp(self._buffer.buffer, val) + else: + for i in range(len(val)): + self[i] = val[i] + if len(val) < self._length_: + self._buffer[len(val)] = b'\x00' + res.value = property(getvalue, setvalue) - def setraw(self, buffer): - if len(buffer) > self._length_: - raise ValueError("%r too long" % (buffer,)) - _rawffi.rawstring2charp(self._buffer.buffer, buffer) - res.raw = property(getraw, setraw) - elif subletter == 'u': - def getvalue(self): - return _rawffi.wcharp2unicode(self._buffer.buffer, - self._length_) + def getraw(self): + return _rawffi.charp2rawstring(self._buffer.buffer, + self._length_) - def setvalue(self, val): - # we don't want to have buffers here - if len(val) > self._length_: - raise ValueError("%r too long" % (val,)) - if isinstance(val, unicode): - target = self._buffer - else: - target = self - for i in range(len(val)): - target[i] = val[i] - if len(val) < self._length_: - target[len(val)] = u'\x00' - res.value = property(getvalue, setvalue) - - if '_length_' in typedict: - res._ffishape_ = (ffiarray, typedict['_length_']) - res._fficompositesize_ = res._sizeofinstances() - else: - res._ffiarray = None + def setraw(self, buffer): + if len(buffer) > self._length_: + raise ValueError("%r too long" % (buffer,)) + _rawffi.rawstring2charp(self._buffer.buffer, buffer) + res.raw = property(getraw, setraw) + elif subletter == 'u': + def getvalue(self): + return _rawffi.wcharp2unicode(self._buffer.buffer, + self._length_) + + def setvalue(self, val): + # we don't want to have buffers here + if len(val) > self._length_: + raise ValueError("%r too long" % (val,)) + if isinstance(val, unicode): + target = self._buffer + else: + target = self + for i in range(len(val)): + target[i] = val[i] + if len(val) < self._length_: + target[len(val)] = u'\x00' + res.value = property(getvalue, setvalue) + + res._ffishape_ = (ffiarray, res._length_) + res._fficompositesize_ = res._sizeofinstances() return res from_address = cdata_from_address @@ -156,7 +159,7 @@ l = [self[i] for i in range(start, stop, step)] letter = getattr(self._type_, '_type_', None) if letter == 'c': - return "".join(l) + return b"".join(l) if letter == 'u': return u"".join(l) return l diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -176,6 +176,10 @@ def _get_buffer_value(self): return self._buffer[0] + def _copy_to(self, addr): + target = type(self).from_address(addr)._buffer + target[0] = self._get_buffer_value() + def _to_ffi_param(self): if self.__class__._is_pointer_like(): return self._get_buffer_value() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -114,7 +114,9 @@ cobj = self._type_.from_param(value) if ensure_objects(cobj) is not None: store_reference(self, index, cobj._objects) - self._subarray(index)[0] = cobj._get_buffer_value() + address = self._buffer[0] + address += index * sizeof(self._type_) + cobj._copy_to(address) def __nonzero__(self): return self._buffer[0] != 0 diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -291,6 +291,11 @@ def _get_buffer_value(self): return self._buffer.buffer + def _copy_to(self, addr): + from ctypes import memmove + origin = self._get_buffer_value() + memmove(addr, origin, self._fficompositesize_) + def _to_ffi_param(self): return self._buffer diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1027,21 +1027,25 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split(" ")[0].upper() - if first_word == "": + + if sql: + first_word = sql.lstrip().split()[0].upper() + if first_word == '': + self._type = _STMT_TYPE_INVALID + if first_word == "SELECT": + self._type = _STMT_TYPE_SELECT + elif first_word == "INSERT": + self._type = _STMT_TYPE_INSERT + elif first_word == "UPDATE": + self._type = _STMT_TYPE_UPDATE + elif first_word == "DELETE": + self._type = _STMT_TYPE_DELETE + elif first_word == "REPLACE": + self._type = _STMT_TYPE_REPLACE + else: + self._type = _STMT_TYPE_OTHER + else: self._type = _STMT_TYPE_INVALID - elif first_word == "SELECT": - self._type = _STMT_TYPE_SELECT - elif first_word == "INSERT": - self._type = _STMT_TYPE_INSERT - elif first_word == "UPDATE": - self._type = _STMT_TYPE_UPDATE - elif first_word == "DELETE": - self._type = _STMT_TYPE_DELETE - elif first_word == "REPLACE": - self._type = _STMT_TYPE_REPLACE - else: - self._type = _STMT_TYPE_OTHER if isinstance(sql, unicode): sql = sql.encode('utf-8') diff --git a/lib_pypy/_tkinter/app.py b/lib_pypy/_tkinter/app.py --- a/lib_pypy/_tkinter/app.py +++ b/lib_pypy/_tkinter/app.py @@ -119,7 +119,7 @@ tklib.TCL_GLOBAL_ONLY) # This is used to get the application class for Tk 4.1 and up - argv0 = className.lower() + argv0 = className.lower().encode('ascii') tklib.Tcl_SetVar(self.interp, "argv0", argv0, tklib.TCL_GLOBAL_ONLY) diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -182,6 +182,57 @@ technical difficulties. +What about numpy, numpypy, micronumpy? +-------------------------------------- + +Way back in 2011, the PyPy team `started to reimplement`_ numpy in PyPy. It +has two pieces: + + * the builtin module :source:`pypy/module/micronumpy`: this is written in + RPython and roughly covers the content of the ``numpy.core.multiarray`` + module. Confusingly enough, this is available in PyPy under the name + ``_numpypy``. It is included by default in all the official releases of + PyPy (but it might be dropped in the future). + + * a fork_ of the official numpy repository maintained by us and informally + called ``numpypy``: even more confusing, the name of the repo on bitbucket + is ``numpy``. The main difference with the upstream numpy, is that it is + based on the micronumpy module written in RPython, instead of of + ``numpy.core.multiarray`` which is written in C. + +Moreover, it is also possible to install the upstream version of ``numpy``: +its core is written in C and it runs on PyPy under the cpyext compatibility +layer. This is what you get if you do ``pypy -m pip install numpy``. + + +Should I install numpy or numpypy? +----------------------------------- + +TL;DR version: you should use numpy. You can install it by doing ``pypy -m pip +install numpy``. You might also be interested in using the experimental `PyPy +binary wheels`_ to save compilation time. + +The upstream ``numpy`` is written in C, and runs under the cpyext +compatibility layer. Nowadays, cpyext is mature enough that you can simply +use the upstream ``numpy``, since it passes 99.9% of the test suite. At the +moment of writing (October 2017) the main drawback of ``numpy`` is that cpyext +is infamously slow, and thus it has worse performance compared to +``numpypy``. However, we are actively working on improving it, as we expect to +reach the same speed, eventually. + +On the other hand, ``numpypy`` is more JIT-friendly and very fast to call, +since it is written in RPython: but it is a reimplementation, and it's hard to +be completely compatible: over the years the project slowly matured and +eventually it was able to call out to the LAPACK and BLAS libraries to speed +matrix calculations, and reached around an 80% parity with the upstream +numpy. However, 80% is far from 100%. Since cpyext/numpy compatibility is +progressing fast, we have discontinued support for ``numpypy``. + +.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html +.. _fork: https://bitbucket.org/pypy/numpy +.. _`PyPy binary wheels`: https://github.com/antocuni/pypy-wheels + + Is PyPy more clever than CPython about Tail Calls? -------------------------------------------------- diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -5,4 +5,8 @@ .. this is a revision shortly after release-pypy2.7-v5.9.0 .. startrev:d56dadcef996 +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging +.. branch: docs-osx-brew-openssl + diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py --- a/pypy/module/_codecs/interp_codecs.py +++ b/pypy/module/_codecs/interp_codecs.py @@ -66,20 +66,17 @@ "position %d from error handler out of bounds", newpos) replace = space.unicode_w(w_replace) - return replace, newpos + if decode: + return replace, newpos + else: + return replace, None, newpos return call_errorhandler def make_decode_errorhandler(self, space): return self._make_errorhandler(space, True) def make_encode_errorhandler(self, space): - errorhandler = self._make_errorhandler(space, False) - def encode_call_errorhandler(errors, encoding, reason, input, startpos, - endpos): - replace, newpos = errorhandler(errors, encoding, reason, input, - startpos, endpos) - return replace, None, newpos - return encode_call_errorhandler + return self._make_errorhandler(space, False) def get_unicodedata_handler(self, space): if self.unicodedata_handler: diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -290,66 +290,87 @@ def test_random_switching(self): from _continuation import continulet # + seen = [] + # def t1(c1): - return c1.switch() + seen.append(3) + res = c1.switch() + seen.append(6) + return res + # def s1(c1, n): + seen.append(2) assert n == 123 c2 = t1(c1) - return c1.switch('a') + 1 + seen.append(7) + res = c1.switch('a') + 1 + seen.append(10) + return res # def s2(c2, c1): + seen.append(5) res = c1.switch(c2) + seen.append(8) assert res == 'a' - return c2.switch('b') + 2 + res = c2.switch('b') + 2 + seen.append(12) + return res # def f(): + seen.append(1) c1 = continulet(s1, 123) c2 = continulet(s2, c1) c1.switch() + seen.append(4) res = c2.switch() + seen.append(9) assert res == 'b' res = c1.switch(1000) + seen.append(11) assert res == 1001 - return c2.switch(2000) + res = c2.switch(2000) + seen.append(13) + return res # res = f() assert res == 2002 + assert seen == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] def test_f_back(self): import sys from _continuation import continulet # - def g(c): + def bar(c): c.switch(sys._getframe(0)) c.switch(sys._getframe(0).f_back) c.switch(sys._getframe(1)) c.switch(sys._getframe(1).f_back) - assert sys._getframe(2) is f3.f_back + assert sys._getframe(2) is f3_foo.f_back c.switch(sys._getframe(2)) - def f(c): - g(c) + def foo(c): + bar(c) # - c = continulet(f) - f1 = c.switch() - assert f1.f_code.co_name == 'g' - f2 = c.switch() - assert f2.f_code.co_name == 'f' - f3 = c.switch() - assert f3 is f2 - assert f1.f_back is f3 + c = continulet(foo) + f1_bar = c.switch() + assert f1_bar.f_code.co_name == 'bar' + f2_foo = c.switch() + assert f2_foo.f_code.co_name == 'foo' + f3_foo = c.switch() + assert f3_foo is f2_foo + assert f1_bar.f_back is f3_foo def main(): - f4 = c.switch() - assert f4.f_code.co_name == 'main', repr(f4.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f4_main = c.switch() + assert f4_main.f_code.co_name == 'main' + assert f3_foo.f_back is f1_bar # not running, so a loop def main2(): - f5 = c.switch() - assert f5.f_code.co_name == 'main2', repr(f5.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f5_main2 = c.switch() + assert f5_main2.f_code.co_name == 'main2' + assert f3_foo.f_back is f1_bar # not running, so a loop main() main2() res = c.switch() assert res is None - assert f3.f_back is None + assert f3_foo.f_back is None def test_traceback_is_complete(self): import sys diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py --- a/pypy/module/cpyext/api.py +++ b/pypy/module/cpyext/api.py @@ -577,6 +577,7 @@ 'PyComplex_AsCComplex', 'PyComplex_FromCComplex', 'PyObject_AsReadBuffer', 'PyObject_AsWriteBuffer', 'PyObject_CheckReadBuffer', + 'PyBuffer_GetPointer', 'PyBuffer_ToContiguous', 'PyBuffer_FromContiguous', 'PyOS_getsig', 'PyOS_setsig', 'PyThread_get_thread_ident', 'PyThread_allocate_lock', 'PyThread_free_lock', diff --git a/pypy/module/cpyext/dictobject.py b/pypy/module/cpyext/dictobject.py --- a/pypy/module/cpyext/dictobject.py +++ b/pypy/module/cpyext/dictobject.py @@ -258,14 +258,19 @@ if w_dict is None: return 0 if not space.isinstance_w(w_dict, space.w_dict): - return 0 + return 0 pos = ppos[0] py_obj = as_pyobj(space, w_dict) py_dict = rffi.cast(PyDictObject, py_obj) if pos == 0: # Store the current keys in the PyDictObject. + from pypy.objspace.std.listobject import W_ListObject decref(space, py_dict.c__tmpkeys) w_keys = space.call_method(space.w_dict, "keys", w_dict) + # w_keys must use the object strategy in order to keep the keys alive + if not isinstance(w_keys, W_ListObject): + return 0 # XXX should not call keys() above + w_keys.switch_to_object_strategy() py_dict.c__tmpkeys = create_ref(space, w_keys) Py_IncRef(space, py_dict.c__tmpkeys) else: @@ -278,10 +283,10 @@ decref(space, py_dict.c__tmpkeys) py_dict.c__tmpkeys = lltype.nullptr(PyObject.TO) return 0 - w_key = space.listview(w_keys)[pos] + w_key = space.listview(w_keys)[pos] # fast iff w_keys uses object strat w_value = space.getitem(w_dict, w_key) if pkey: - pkey[0] = as_pyobj(space, w_key) + pkey[0] = as_pyobj(space, w_key) if pvalue: pvalue[0] = as_pyobj(space, w_value) return 1 diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h --- a/pypy/module/cpyext/include/object.h +++ b/pypy/module/cpyext/include/object.h @@ -317,6 +317,31 @@ PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *, const void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_AsWriteBuffer(PyObject *, void **, Py_ssize_t *); PyAPI_FUNC(int) PyObject_CheckReadBuffer(PyObject *); +PyAPI_FUNC(void *) PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices); +/* Get the memory area pointed to by the indices for the buffer given. + Note that view->ndim is the assumed size of indices +*/ + +PyAPI_FUNC(int) PyBuffer_ToContiguous(void *buf, Py_buffer *view, + Py_ssize_t len, char fort); +PyAPI_FUNC(int) PyBuffer_FromContiguous(Py_buffer *view, void *buf, + Py_ssize_t len, char fort); +/* Copy len bytes of data from the contiguous chunk of memory + pointed to by buf into the buffer exported by obj. Return + 0 on success and return -1 and raise a PyBuffer_Error on + error (i.e. the object does not have a buffer interface or + it is not working). + + If fort is 'F' and the object is multi-dimensional, + then the data will be copied into the array in + Fortran-style (first dimension varies the fastest). If + fort is 'C', then the data will be copied into the array + in C-style (last dimension varies the fastest). If fort + is 'A', then it does not matter and the copy will be made + in whatever way is more efficient. + +*/ + #define PyObject_MALLOC PyObject_Malloc #define PyObject_REALLOC PyObject_Realloc diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -15,7 +15,7 @@ from rpython.rlib.objectmodel import keepalive_until_here from rpython.rtyper.annlowlevel import llhelper from rpython.rlib import rawrefcount, jit -from rpython.rlib.debug import fatalerror +from rpython.rlib.debug import ll_assert, fatalerror #________________________________________________________ @@ -243,6 +243,11 @@ py_obj = rawrefcount.from_obj(PyObject, w_obj) if not py_obj: py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal) + # + # Try to crash here, instead of randomly, if we don't keep w_obj alive + ll_assert(py_obj.c_ob_refcnt >= rawrefcount.REFCNT_FROM_PYPY, + "Bug in cpyext: The W_Root object was garbage-collected " + "while being converted to PyObject.") return py_obj else: return lltype.nullptr(PyObject.TO) diff --git a/pypy/module/cpyext/slotdefs.py b/pypy/module/cpyext/slotdefs.py --- a/pypy/module/cpyext/slotdefs.py +++ b/pypy/module/cpyext/slotdefs.py @@ -13,7 +13,7 @@ ssizessizeargfunc, ssizeobjargproc, iternextfunc, initproc, richcmpfunc, cmpfunc, hashfunc, descrgetfunc, descrsetfunc, objobjproc, objobjargproc, readbufferproc, getbufferproc, ssizessizeobjargproc) -from pypy.module.cpyext.pyobject import make_ref, decref, from_ref +from pypy.module.cpyext.pyobject import make_ref, from_ref, as_pyobj from pypy.module.cpyext.pyerrors import PyErr_Occurred from pypy.module.cpyext.memoryobject import fill_Py_buffer from pypy.module.cpyext.state import State @@ -90,26 +90,29 @@ args_w = space.fixedview(w_args) return generic_cpy_call(space, func_binary, w_self, args_w[0]) +def _get_ob_type(space, w_obj): + # please ensure that w_obj stays alive + ob_type = as_pyobj(space, space.type(w_obj)) + return rffi.cast(PyTypeObjectPtr, ob_type) + def wrap_binaryfunc_l(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, w_self, args_w[0]) def wrap_binaryfunc_r(space, w_self, w_args, func): func_binary = rffi.cast(binaryfunc, func) check_num_args(space, w_args, 1) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) return generic_cpy_call(space, func_binary, args_w[0], w_self) def wrap_ternaryfunc(space, w_self, w_args, func): @@ -127,11 +130,10 @@ func_ternary = rffi.cast(ternaryfunc, func) check_num_argsv(space, w_args, 1, 2) args_w = space.fixedview(w_args) - ref = make_ref(space, w_self) - if (not ref.c_ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and + ob_type = _get_ob_type(space, w_self) + if (not ob_type.c_tp_flags & Py_TPFLAGS_CHECKTYPES and not space.issubtype_w(space.type(args_w[0]), space.type(w_self))): return space.w_NotImplemented - decref(space, ref) arg3 = space.w_None if len(args_w) > 1: arg3 = args_w[1] @@ -323,12 +325,10 @@ def wrap_getreadbuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(rffi.VOIDPP.TO, 1) as ptr: index = rffi.cast(Py_ssize_t, 0) size = generic_cpy_call(space, func_target, w_self, index, ptr) @@ -341,9 +341,7 @@ def wrap_getwritebuffer(space, w_self, w_args, func): func_target = rffi.cast(readbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type - decref(space, py_obj) + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) @@ -359,12 +357,10 @@ def wrap_getbuffer(space, w_self, w_args, func): func_target = rffi.cast(getbufferproc, func) - py_obj = make_ref(space, w_self) - py_type = py_obj.c_ob_type + py_type = _get_ob_type(space, w_self) rbp = rffi.cast(rffi.VOIDP, 0) if py_type.c_tp_as_buffer: rbp = rffi.cast(rffi.VOIDP, py_type.c_tp_as_buffer.c_bf_releasebuffer) - decref(space, py_obj) with lltype.scoped_alloc(Py_buffer) as pybuf: _flags = 0 if space.len_w(w_args) > 0: diff --git a/pypy/module/cpyext/src/abstract.c b/pypy/module/cpyext/src/abstract.c --- a/pypy/module/cpyext/src/abstract.c +++ b/pypy/module/cpyext/src/abstract.c @@ -101,6 +101,163 @@ return 0; } +void* +PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) +{ + char* pointer; + int i; + pointer = (char *)view->buf; + for (i = 0; i < view->ndim; i++) { + pointer += view->strides[i]*indices[i]; + if ((view->suboffsets != NULL) && (view->suboffsets[i] >= 0)) { + pointer = *((char**)pointer) + view->suboffsets[i]; + } + } + return (void*)pointer; +} + +void +_Py_add_one_to_index_F(int nd, Py_ssize_t *index, const Py_ssize_t *shape) +{ + int k; + + for (k=0; k=0; k--) { + if (index[k] < shape[k]-1) { + index[k]++; + break; + } + else { + index[k] = 0; + } + } +} + + /* view is not checked for consistency in either of these. It is + assumed that the size of the buffer is view->len in + view->len / view->itemsize elements. + */ + +int +PyBuffer_ToContiguous(void *buf, Py_buffer *view, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *dest, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(buf, view->buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + dest = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(dest, ptr, view->itemsize); + dest += view->itemsize; + addone(view->ndim, indices, view->shape); + } + PyMem_Free(indices); + return 0; +} + +int +PyBuffer_FromContiguous(Py_buffer *view, void *buf, Py_ssize_t len, char fort) +{ + int k; + void (*addone)(int, Py_ssize_t *, const Py_ssize_t *); + Py_ssize_t *indices, elements; + char *src, *ptr; + + if (len > view->len) { + len = view->len; + } + + if (PyBuffer_IsContiguous(view, fort)) { + /* simplest copy is all that is needed */ + memcpy(view->buf, buf, len); + return 0; + } + + /* Otherwise a more elaborate scheme is needed */ + + /* view->ndim <= 64 */ + indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); + if (indices == NULL) { + PyErr_NoMemory(); + return -1; + } + for (k=0; kndim;k++) { + indices[k] = 0; + } + + if (fort == 'F') { + addone = _Py_add_one_to_index_F; + } + else { + addone = _Py_add_one_to_index_C; + } + src = buf; + /* XXX : This is not going to be the fastest code in the world + several optimizations are possible. + */ + elements = len / view->itemsize; + while (elements--) { + ptr = PyBuffer_GetPointer(view, indices); + memcpy(ptr, src, view->itemsize); + src += view->itemsize; + addone(view->ndim, indices, view->shape); + } + + PyMem_Free(indices); + return 0; +} + + + /* Buffer C-API for Python 3.0 */ int diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py --- a/pypy/module/cpyext/test/test_memoryobject.py +++ b/pypy/module/cpyext/test/test_memoryobject.py @@ -115,7 +115,36 @@ view = PyMemoryView_GET_BUFFER(memoryview); Py_DECREF(memoryview); return PyLong_FromLong(view->len / view->itemsize); - """)]) + """), + ("test_contiguous", "METH_O", + """ + Py_buffer* view; + PyObject * memoryview; + void * buf = NULL; + int ret; + Py_ssize_t len; + memoryview = PyMemoryView_FromObject(args); + if (memoryview == NULL) + return NULL; + view = PyMemoryView_GET_BUFFER(memoryview); + Py_DECREF(memoryview); + len = view->len; + if (len == 0) + return NULL; + buf = malloc(len); + ret = PyBuffer_ToContiguous(buf, view, view->len, 'A'); + if (ret != 0) + { + free(buf); + return NULL; + } + ret = PyBuffer_FromContiguous(view, buf, view->len, 'A'); + free(buf); + if (ret != 0) + return NULL; + Py_RETURN_NONE; + """), + ]) module = self.import_module(name='buffer_test') arr = module.PyMyArray(10) ten = foo.get_len(arr) @@ -124,6 +153,7 @@ assert ten == 10 ten = foo.test_buffer(arr) assert ten == 10 + foo.test_contiguous(arr) def test_releasebuffer(self): module = self.import_extension('foo', [ diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py --- a/pypy/module/cpyext/test/test_typeobject.py +++ b/pypy/module/cpyext/test/test_typeobject.py @@ -353,12 +353,8 @@ PyObject* name = PyString_FromString("mymodule"); PyObject *obj = PyType_Type.tp_alloc(&PyType_Type, 0); PyHeapTypeObject *type = (PyHeapTypeObject*)obj; - if ((type->ht_type.tp_flags & Py_TPFLAGS_HEAPTYPE) == 0) - { - PyErr_SetString(PyExc_ValueError, - "Py_TPFLAGS_HEAPTYPE not set"); - return NULL; - } + /* this is issue #2434: logic from pybind11 */ + type->ht_type.tp_flags |= Py_TPFLAGS_HEAPTYPE; type->ht_type.tp_name = ((PyTypeObject*)args)->tp_name; PyType_Ready(&type->ht_type); ret = PyObject_SetAttrString((PyObject*)&type->ht_type, diff --git a/pypy/module/pyexpat/interp_pyexpat.py b/pypy/module/pyexpat/interp_pyexpat.py --- a/pypy/module/pyexpat/interp_pyexpat.py +++ b/pypy/module/pyexpat/interp_pyexpat.py @@ -808,7 +808,7 @@ w_encoding) if space.is_none(w_namespace_separator): - namespace_separator = 0 + namespace_separator = -1 elif space.isinstance_w(w_namespace_separator, space.w_text): separator = space.text_w(w_namespace_separator) if len(separator) == 0: @@ -831,7 +831,7 @@ elif space.is_w(w_intern, space.w_None): w_intern = None - if namespace_separator: + if namespace_separator >= 0: xmlparser = XML_ParserCreateNS( encoding, rffi.cast(rffi.CHAR, namespace_separator)) diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py --- a/pypy/module/pyexpat/test/test_parser.py +++ b/pypy/module/pyexpat/test/test_parser.py @@ -59,7 +59,7 @@ p.CharacterDataHandler = lambda s: data.append(s) encoding = encoding_arg is None and 'utf-8' or encoding_arg - res = p.Parse(u"\u00f6".encode(encoding), isfinal=True) + res = p.Parse(u"\u00f6".encode(encoding), True) assert res == 1 assert data == [u"\u00f6"] @@ -188,6 +188,34 @@ p.ParseFile(fake_reader) assert fake_reader.read_count == 4 + def test_entities(self): + import pyexpat + parser = pyexpat.ParserCreate(None, "") + + def startElement(tag, attrs): + assert tag == 'http://www.w3.org/1999/02/22-rdf-syntax-ns#RDF' + assert attrs == { + 'http://www.w3.org/XML/1998/namespacebase': + 'http://www.semanticweb.org/jiba/ontologies/2017/0/test'} + parser.StartElementHandler = startElement + parser.Parse(""" + + + + + + ]> + + + + """, True) + class AppTestPyexpat2: spaceconfig = dict(usemodules=['pyexpat', 'itertools', '_socket', 'time', 'struct', 'binascii']) diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py --- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py +++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_version.py @@ -37,7 +37,7 @@ v = cffi.__version__.replace('+', '') p = os.path.join(parent, 'doc', 'source', 'installation.rst') content = open(p).read() - assert ("/cffi-%s.tar.gz" % v) in content + assert (" package version %s:" % v) in content def test_setup_version(): parent = os.path.dirname(os.path.dirname(cffi.__file__)) diff --git a/pypy/module/test_lib_pypy/test_sqlite3.py b/pypy/module/test_lib_pypy/test_sqlite3.py --- a/pypy/module/test_lib_pypy/test_sqlite3.py +++ b/pypy/module/test_lib_pypy/test_sqlite3.py @@ -228,6 +228,14 @@ cur.execute("create table test(a)") cur.executemany("insert into test values (?)", [[1], [2], [3]]) assert cur.lastrowid is None + # issue 2682 + cur.execute('''insert + into test + values (?) + ''', (1, )) + assert cur.lastrowid is not None + cur.execute('''insert\t into test values (?) ''', (1, )) + assert cur.lastrowid is not None def test_authorizer_bad_value(self, con): def authorizer_cb(action, arg1, arg2, dbname, source): diff --git a/pypy/module/unicodedata/test/test_hyp.py b/pypy/module/unicodedata/test/test_hyp.py --- a/pypy/module/unicodedata/test/test_hyp.py +++ b/pypy/module/unicodedata/test/test_hyp.py @@ -40,6 +40,7 @@ @pytest.mark.parametrize('NF1, NF2, NF3', compositions) @example(s=u'---\uafb8\u11a7---') # issue 2289 + at example(s=u'\ufacf') @settings(max_examples=1000) @given(s=st.text()) def test_composition(s, space, NF1, NF2, NF3): diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py --- a/pypy/objspace/std/stringmethods.py +++ b/pypy/objspace/std/stringmethods.py @@ -633,13 +633,16 @@ def _startswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) - if start > len(value): - return self._starts_ends_overflow(prefix) + if self._starts_ends_unicode: # bug-to-bug compat + if len(prefix) == 0: + return True + else: + if start > len(value): + return False return startswith(value, prefix, start, end) - def _starts_ends_overflow(self, prefix): - return False # bug-to-bug compat: this is for strings and - # bytearrays, but overridden for unicodes + _starts_ends_unicode = False # bug-to-bug compat: this is for strings and + # bytearrays, but overridden for unicodes def descr_endswith(self, space, w_suffix, w_start=None, w_end=None): value, start, end, _ = self._convert_idx_params(space, w_start, w_end) @@ -656,8 +659,12 @@ def _endswith(self, space, value, w_prefix, start, end): prefix = self._op_val(space, w_prefix) - if start > len(value): - return self._starts_ends_overflow(prefix) + if self._starts_ends_unicode: # bug-to-bug compat + if len(prefix) == 0: + return True + else: + if start > len(value): + return False return endswith(value, prefix, start, end) def _strip(self, space, w_chars, left, right, name='strip'): diff --git a/pypy/objspace/std/test/test_bytesobject.py b/pypy/objspace/std/test/test_bytesobject.py --- a/pypy/objspace/std/test/test_bytesobject.py +++ b/pypy/objspace/std/test/test_bytesobject.py @@ -1,3 +1,6 @@ +from pypy.interpreter.error import OperationError + + class TestW_BytesObject: def teardown_method(self, method): @@ -86,6 +89,78 @@ w_bytes = self.space.newbytes('abcd') assert self.space.listview_bytes(w_bytes) == list("abcd") + +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.binary(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + class AppTestBytesObject: def test_format_wrongchar(self): diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py --- a/pypy/objspace/std/test/test_unicodeobject.py +++ b/pypy/objspace/std/test/test_unicodeobject.py @@ -1,5 +1,6 @@ import py import sys +from pypy.interpreter.error import OperationError class TestUnicodeObject: @@ -33,6 +34,77 @@ assert w_new is w_uni +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + @given(u=strategies.text(), + start=strategies.integers(min_value=0, max_value=10), + len1=strategies.integers(min_value=-1, max_value=10)) + def test_hypo_index_find(u, start, len1, space): + if start + len1 < 0: + return # skip this case + v = u[start : start + len1] + w_u = space.wrap(u) + w_v = space.wrap(v) + expected = u.find(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'index', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert expected == -1 + else: + assert space.int_w(w_index) == expected >= 0 + + w_index = space.call_method(w_u, 'find', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == expected + + rexpected = u.rfind(v, start, start + len1) + try: + w_index = space.call_method(w_u, 'rindex', w_v, + space.newint(start), + space.newint(start + len1)) + except OperationError as e: + if not e.match(space, space.w_ValueError): + raise + assert rexpected == -1 + else: + assert space.int_w(w_index) == rexpected >= 0 + + w_index = space.call_method(w_u, 'rfind', w_v, + space.newint(start), + space.newint(start + len1)) + assert space.int_w(w_index) == rexpected + + expected = u.startswith(v, start) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.startswith(v, start, start + len1) + w_res = space.call_method(w_u, 'startswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start)) + assert w_res is space.newbool(expected) + + expected = u.endswith(v, start, start + len1) + w_res = space.call_method(w_u, 'endswith', w_v, + space.newint(start), + space.newint(start + len1)) + assert w_res is space.newbool(expected) + + class AppTestUnicodeStringStdOnly: def test_compares(self): assert u'a' == 'a' diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -419,8 +419,7 @@ cased = True return space.newbool(cased) - def _starts_ends_overflow(self, prefix): - return len(prefix) == 0 + _starts_ends_unicode = True def wrapunicode(space, uni): diff --git a/rpython/doc/jit/optimizer.rst b/rpython/doc/jit/optimizer.rst --- a/rpython/doc/jit/optimizer.rst +++ b/rpython/doc/jit/optimizer.rst @@ -42,10 +42,9 @@ There are better ways to compute the sum from ``[0..100]``, but it gives a better intuition on how traces are constructed than ``sum(range(101))``. Note that the trace syntax is the one used in the test suite. It is also very -similar to traces printed at runtime by PYPYLOG_. The first line gives the input variables, the -second line is a ``label`` operation, the last one is the backwards ``jump`` operation. - -.. _PYPYLOG: logging.html +similar to traces printed at runtime by :doc:`PYPYLOG <../logging>`. The first +line gives the input variables, the second line is a ``label`` operation, the +last one is the backwards ``jump`` operation. These instructions mentioned earlier are special: diff --git a/rpython/jit/metainterp/optimizeopt/intbounds.py b/rpython/jit/metainterp/optimizeopt/intbounds.py --- a/rpython/jit/metainterp/optimizeopt/intbounds.py +++ b/rpython/jit/metainterp/optimizeopt/intbounds.py @@ -305,6 +305,10 @@ # Transform into INT_ADD. The following guard will be killed # by optimize_GUARD_NO_OVERFLOW; if we see instead an # optimize_GUARD_OVERFLOW, then InvalidLoop. + + # NB: this case also takes care of int_add_ovf with 0 as on of the + # arguments: the result will be bounded, and then the optimization + # for int_add with 0 as argument will remove the op. op = self.replace_op_with(op, rop.INT_ADD) return self.emit(op) @@ -325,6 +329,7 @@ return None resbound = b0.sub_bound(b1) if resbound.bounded(): + # this case takes care of int_sub_ovf(x, 0) as well op = self.replace_op_with(op, rop.INT_SUB) return self.emit(op) @@ -342,6 +347,7 @@ b2 = self.getintbound(op.getarg(1)) resbound = b1.mul_bound(b2) if resbound.bounded(): + # this case also takes care of multiplication with 0 and 1 op = self.replace_op_with(op, rop.INT_MUL) return self.emit(op) diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py --- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py +++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py @@ -1962,6 +1962,55 @@ """ self.optimize_loop(ops, expected) + ops = """ + [i0] + i1 = int_mul_ovf(0, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(1, i0) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + ops = """ + [i0] + i1 = int_mul_ovf(i0, 1) + guard_no_overflow() [] + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, expected) + + def test_fold_constant_partial_ops_float(self): ops = """ [f0] diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py --- a/rpython/rlib/runicode.py +++ b/rpython/rlib/runicode.py @@ -877,32 +877,31 @@ ch = ord(s[pos]) pos += 1 ch2 = 0 - if 0xD800 <= ch < 0xDC00: - if not allow_surrogates: - ru, rs, pos = errorhandler(errors, public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - if rs is not None: - # py3k only - if len(rs) % 4 != 0: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) - result.append(rs) - continue - for ch in ru: - if ord(ch) < 0xD800: - _STORECHAR32(result, ord(ch), byteorder) - else: - errorhandler('strict', public_encoding_name, - 'surrogates not allowed', - s, pos-1, pos) + if not allow_surrogates and 0xD800 <= ch < 0xE000: + ru, rs, pos = errorhandler(errors, public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + if rs is not None: + # py3k only + if len(rs) % 4 != 0: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + result.append(rs) continue - elif MAXUNICODE < 65536 and pos < size: - ch2 = ord(s[pos]) - if 0xDC00 <= ch2 < 0xE000: - ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; - pos += 1 + for ch in ru: + if ord(ch) < 0xD800: + _STORECHAR32(result, ord(ch), byteorder) + else: + errorhandler('strict', public_encoding_name, + 'surrogates not allowed', + s, pos-1, pos) + continue + if 0xD800 <= ch < 0xDC00 and MAXUNICODE < 65536 and pos < size: + ch2 = ord(s[pos]) + if 0xDC00 <= ch2 < 0xE000: + ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000; + pos += 1 _STORECHAR32(result, ch, byteorder) return result.build() diff --git a/rpython/rlib/test/test_runicode.py b/rpython/rlib/test/test_runicode.py --- a/rpython/rlib/test/test_runicode.py +++ b/rpython/rlib/test/test_runicode.py @@ -2,6 +2,7 @@ import py import sys, random +import struct from rpython.rlib import runicode from hypothesis import given, settings, strategies @@ -266,11 +267,12 @@ assert replace_with(u'rep', None) == '\x00<\x00r\x00e\x00p\x00>' assert replace_with(None, '\xca\xfe') == '\x00<\xca\xfe\x00>' - def test_utf32_surrogates(self): + @py.test.mark.parametrize('unich',[u"\ud800", u"\udc80"]) + def test_utf32_surrogates(self, unich): assert runicode.unicode_encode_utf_32_be( - u"\ud800", 1, None) == '\x00\x00\xd8\x00' + unich, 1, None) == struct.pack('>i', ord(unich)) py.test.raises(UnicodeEncodeError, runicode.unicode_encode_utf_32_be, - u"\ud800", 1, None, allow_surrogates=False) + unich, 1, None, allow_surrogates=False) def replace_with(ru, rs): def errorhandler(errors, enc, msg, u, startingpos, endingpos): if errors == 'strict': @@ -278,7 +280,7 @@ endingpos, msg) return ru, rs, endingpos return runicode.unicode_encode_utf_32_be( - u"<\ud800>", 3, None, + u"<%s>" % unich, 3, None, errorhandler, allow_surrogates=False) assert replace_with(u'rep', None) == u''.encode('utf-32-be') assert replace_with(None, '\xca\xfe\xca\xfe') == '\x00\x00\x00<\xca\xfe\xca\xfe\x00\x00\x00>' @@ -432,7 +434,7 @@ assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'ignore', final=True) == (u'aaaabbbb', len(seq) + 8)) assert (self.decoder(seq, len(seq), 'custom', final=True, - errorhandler=self.custom_replace) == + errorhandler=self.custom_replace) == (FOO * len(seq), len(seq))) assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'custom', final=True, errorhandler=self.custom_replace) == @@ -628,7 +630,7 @@ msg='invalid continuation byte') assert self.decoder(seq, len(seq), 'replace', final=True ) == (res, len(seq)) - assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, + assert (self.decoder('aaaa' + seq + 'bbbb', len(seq) + 8, 'replace', final=True) == (u'aaaa' + res + u'bbbb', len(seq) + 8)) res = res.replace(FFFD, u'') From pypy.commits at gmail.com Sat Oct 28 19:05:28 2017 From: pypy.commits at gmail.com (wlav) Date: Sat, 28 Oct 2017 16:05:28 -0700 (PDT) Subject: [pypy-commit] pypy default: merge cppyy-packaging with improved consistency for cppyy CPython <-> PyPy Message-ID: <59f50d38.06b6df0a.1db93.99c6@mx.google.com> Author: Wim Lavrijsen Branch: Changeset: r92872:0b8528722439 Date: 2017-10-28 15:55 -0700 http://bitbucket.org/pypy/pypy/changeset/0b8528722439/ Log: merge cppyy-packaging with improved consistency for cppyy CPython <-> PyPy diff too long, truncating to 2000 out of 3147 lines diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -7,7 +7,7 @@ interpleveldefs = { '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', - '_template_byname' : 'interp_cppyy.template_byname', + '_is_template' : 'interp_cppyy.is_template', '_std_string_name' : 'interp_cppyy.std_string_name', '_set_class_generator' : 'interp_cppyy.set_class_generator', '_set_function_generator': 'interp_cppyy.set_function_generator', @@ -15,7 +15,9 @@ '_get_nullptr' : 'interp_cppyy.get_nullptr', 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', + '_bind_object' : 'interp_cppyy._bind_object', 'bind_object' : 'interp_cppyy.bind_object', + 'move' : 'interp_cppyy.move', } appleveldefs = { diff --git a/pypy/module/_cppyy/capi/loadable_capi.py b/pypy/module/_cppyy/capi/loadable_capi.py --- a/pypy/module/_cppyy/capi/loadable_capi.py +++ b/pypy/module/_cppyy/capi/loadable_capi.py @@ -217,7 +217,8 @@ 'method_req_args' : ([c_scope, c_index], c_int), 'method_arg_type' : ([c_scope, c_index, c_int], c_ccharp), 'method_arg_default' : ([c_scope, c_index, c_int], c_ccharp), - 'method_signature' : ([c_scope, c_index], c_ccharp), + 'method_signature' : ([c_scope, c_index, c_int], c_ccharp), + 'method_prototype' : ([c_scope, c_index, c_int], c_ccharp), 'method_is_template' : ([c_scope, c_index], c_int), 'method_num_template_args' : ([c_scope, c_index], c_int), @@ -498,9 +499,12 @@ def c_method_arg_default(space, cppscope, index, arg_index): args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(arg_index)] return charp2str_free(space, call_capi(space, 'method_arg_default', args)) -def c_method_signature(space, cppscope, index): - args = [_ArgH(cppscope.handle), _ArgL(index)] +def c_method_signature(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] return charp2str_free(space, call_capi(space, 'method_signature', args)) +def c_method_prototype(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] + return charp2str_free(space, call_capi(space, 'method_prototype', args)) def c_method_is_template(space, cppscope, index): args = [_ArgH(cppscope.handle), _ArgL(index)] diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -4,7 +4,7 @@ from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib.rarithmetic import r_singlefloat, r_longfloat -from rpython.rlib import rfloat +from rpython.rlib import rfloat, rawrefcount from pypy.module._rawffi.interp_rawffi import letter2tp from pypy.module._rawffi.array import W_Array, W_ArrayInstance @@ -21,9 +21,9 @@ # match for the qualified type. -def get_rawobject(space, w_obj): +def get_rawobject(space, w_obj, can_be_None=True): from pypy.module._cppyy.interp_cppyy import W_CPPClass - cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=can_be_None) if cppinstance: rawobject = cppinstance.get_rawobject() assert lltype.typeOf(rawobject) == capi.C_OBJECT @@ -48,17 +48,16 @@ return capi.C_NULL_OBJECT def is_nullpointer_specialcase(space, w_obj): - # 0, None, and nullptr may serve as "NULL", check for any of them + # 0 and nullptr may serve as "NULL" # integer 0 try: return space.int_w(w_obj) == 0 except Exception: pass - # None or nullptr + # C++-style nullptr from pypy.module._cppyy import interp_cppyy - return space.is_true(space.is_(w_obj, space.w_None)) or \ - space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) + return space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) def get_rawbuffer(space, w_obj): # raw buffer @@ -74,7 +73,7 @@ return rffi.cast(rffi.VOIDP, space.uint_w(arr.getbuffer(space))) except Exception: pass - # pre-defined NULL + # pre-defined nullptr if is_nullpointer_specialcase(space, w_obj): return rffi.cast(rffi.VOIDP, 0) raise TypeError("not an addressable buffer") @@ -392,6 +391,7 @@ _immutable_fields_ = ['typecode'] typecode = 'g' + class CStringConverter(TypeConverter): def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.LONGP, address) @@ -408,18 +408,27 @@ def free_argument(self, space, arg, call_local): lltype.free(rffi.cast(rffi.CCHARPP, arg)[0], flavor='raw') +class CStringConverterWithSize(CStringConverter): + _immutable_fields_ = ['size'] + + def __init__(self, space, extra): + self.size = extra + + def from_memory(self, space, w_obj, w_pycppclass, offset): + address = self._get_raw_address(space, w_obj, offset) + charpptr = rffi.cast(rffi.CCHARP, address) + strsize = self.size + if charpptr[self.size-1] == '\0': + strsize = self.size-1 # rffi will add \0 back + return space.newbytes(rffi.charpsize2str(charpptr, strsize)) + class VoidPtrConverter(TypeConverter): def _unwrap_object(self, space, w_obj): try: obj = get_rawbuffer(space, w_obj) except TypeError: - try: - # TODO: accept a 'capsule' rather than naked int - # (do accept int(0), though) - obj = rffi.cast(rffi.VOIDP, space.uint_w(w_obj)) - except Exception: - obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) + obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj, False)) return obj def cffi_type(self, space): @@ -463,12 +472,12 @@ def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.VOIDPP, address) ba = rffi.cast(rffi.CCHARP, address) - r = rffi.cast(rffi.VOIDPP, call_local) try: - r[0] = get_rawbuffer(space, w_obj) + x[0] = get_rawbuffer(space, w_obj) except TypeError: + r = rffi.cast(rffi.VOIDPP, call_local) r[0] = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) - x[0] = rffi.cast(rffi.VOIDP, call_local) + x[0] = rffi.cast(rffi.VOIDP, call_local) ba[capi.c_function_arg_typeoffset(space)] = self.typecode def finalize_call(self, space, w_obj, call_local): @@ -495,9 +504,13 @@ def _unwrap_object(self, space, w_obj): from pypy.module._cppyy.interp_cppyy import W_CPPClass if isinstance(w_obj, W_CPPClass): - if capi.c_is_subtype(space, w_obj.cppclass, self.clsdecl): + from pypy.module._cppyy.interp_cppyy import INSTANCE_FLAGS_IS_R_VALUE + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + # reject moves as all are explicit + raise ValueError("lvalue expected") + if capi.c_is_subtype(space, w_obj.clsdecl, self.clsdecl): rawobject = w_obj.get_rawobject() - offset = capi.c_base_offset(space, w_obj.cppclass, self.clsdecl, rawobject, 1) + offset = capi.c_base_offset(space, w_obj.clsdecl, self.clsdecl, rawobject, 1) obj_address = capi.direct_ptradd(rawobject, offset) return rffi.cast(capi.C_OBJECT, obj_address) raise oefmt(space.w_TypeError, @@ -518,6 +531,17 @@ x = rffi.cast(rffi.VOIDPP, address) x[0] = rffi.cast(rffi.VOIDP, self._unwrap_object(space, w_obj)) +class InstanceMoveConverter(InstanceRefConverter): + def _unwrap_object(self, space, w_obj): + # moving is same as by-ref, but have to check that move is allowed + from pypy.module._cppyy.interp_cppyy import W_CPPClass, INSTANCE_FLAGS_IS_R_VALUE + if isinstance(w_obj, W_CPPClass): + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + w_obj.flags &= ~INSTANCE_FLAGS_IS_R_VALUE + return InstanceRefConverter._unwrap_object(self, space, w_obj) + raise oefmt(space.w_ValueError, "object is not an rvalue") + + class InstanceConverter(InstanceRefConverter): def convert_argument_libffi(self, space, w_obj, address, call_local): @@ -527,7 +551,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): self._is_abstract(space) @@ -548,7 +572,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): address = rffi.cast(rffi.VOIDPP, self._get_raw_address(space, w_obj, offset)) @@ -582,8 +606,8 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, - do_cast=False, is_ref=True) + return interp_cppyy.wrap_cppinstance( + space, address, self.clsdecl, do_cast=False, is_ref=True) class StdStringConverter(InstanceConverter): @@ -606,7 +630,7 @@ assign = self.clsdecl.get_overload("__assign__") from pypy.module._cppyy import interp_cppyy assign.call( - interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False), [w_value]) + interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False), [w_value]) except Exception: InstanceConverter.to_memory(self, space, w_obj, w_value, offset) @@ -672,7 +696,7 @@ _converters = {} # builtin and custom types _a_converters = {} # array and ptr versions of above -def get_converter(space, name, default): +def get_converter(space, _name, default): # The matching of the name to a converter should follow: # 1) full, exact match # 1a) const-removed match @@ -680,9 +704,9 @@ # 3) accept ref as pointer (for the stubs, const& can be # by value, but that does not work for the ffi path) # 4) generalized cases (covers basically all user classes) - # 5) void converter, which fails on use + # 5) void* or void converter (which fails on use) - name = capi.c_resolve_name(space, name) + name = capi.c_resolve_name(space, _name) # 1) full, exact match try: @@ -701,7 +725,7 @@ clean_name = capi.c_resolve_name(space, helper.clean_type(name)) try: # array_index may be negative to indicate no size or no size found - array_size = helper.array_size(name) + array_size = helper.array_size(_name) # uses original arg return _a_converters[clean_name+compound](space, array_size) except KeyError: pass @@ -719,6 +743,8 @@ return InstancePtrConverter(space, clsdecl) elif compound == "&": return InstanceRefConverter(space, clsdecl) + elif compound == "&&": + return InstanceMoveConverter(space, clsdecl) elif compound == "**": return InstancePtrPtrConverter(space, clsdecl) elif compound == "": @@ -726,11 +752,13 @@ elif capi.c_is_enum(space, clean_name): return _converters['unsigned'](space, default) - # 5) void converter, which fails on use - # + # 5) void* or void converter (which fails on use) + if 0 <= compound.find('*'): + return VoidPtrConverter(space, default) # "user knows best" + # return a void converter here, so that the class can be build even - # when some types are unknown; this overload will simply fail on use - return VoidConverter(space, name) + # when some types are unknown + return VoidConverter(space, name) # fails on use _converters["bool"] = BoolConverter @@ -847,6 +875,10 @@ for name in names: _a_converters[name+'[]'] = ArrayConverter _a_converters[name+'*'] = PtrConverter + + # special case, const char* w/ size and w/o '\0' + _a_converters["const char[]"] = CStringConverterWithSize + _build_array_converters() # add another set of aliased names diff --git a/pypy/module/_cppyy/executor.py b/pypy/module/_cppyy/executor.py --- a/pypy/module/_cppyy/executor.py +++ b/pypy/module/_cppyy/executor.py @@ -159,7 +159,7 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_l(space, cppmethod, cppthis, num_args, args) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - pyres = interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + pyres = interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) return pyres def execute_libffi(self, space, cif_descr, funcaddr, buffer): @@ -167,7 +167,7 @@ result = rffi.ptradd(buffer, cif_descr.exchange_result) from pypy.module._cppyy import interp_cppyy ptr_result = rffi.cast(capi.C_OBJECT, rffi.cast(rffi.VOIDPP, result)[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) class InstancePtrPtrExecutor(InstancePtrExecutor): @@ -176,7 +176,7 @@ voidp_result = capi.c_call_r(space, cppmethod, cppthis, num_args, args) ref_address = rffi.cast(rffi.VOIDPP, voidp_result) ptr_result = rffi.cast(capi.C_OBJECT, ref_address[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible @@ -188,8 +188,8 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_o(space, cppmethod, cppthis, num_args, args, self.cppclass) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass, - do_cast=False, python_owns=True, fresh=True) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass, + do_cast=False, python_owns=True, fresh=True) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible diff --git a/pypy/module/_cppyy/include/capi.h b/pypy/module/_cppyy/include/capi.h --- a/pypy/module/_cppyy/include/capi.h +++ b/pypy/module/_cppyy/include/capi.h @@ -19,14 +19,15 @@ RPY_EXTERN int cppyy_num_scopes(cppyy_scope_t parent); RPY_EXTERN - char* cppyy_scope_name(cppyy_scope_t parent, int iscope); - + char* cppyy_scope_name(cppyy_scope_t parent, cppyy_index_t iscope); RPY_EXTERN char* cppyy_resolve_name(const char* cppitem_name); RPY_EXTERN cppyy_scope_t cppyy_get_scope(const char* scope_name); RPY_EXTERN cppyy_type_t cppyy_actual_class(cppyy_type_t klass, cppyy_object_t obj); + RPY_EXTERN + size_t cppyy_size_of(cppyy_type_t klass); /* memory management ------------------------------------------------------ */ RPY_EXTERN @@ -120,6 +121,8 @@ RPY_EXTERN char* cppyy_method_name(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN + char* cppyy_method_mangled_name(cppyy_scope_t scope, cppyy_index_t idx); + RPY_EXTERN char* cppyy_method_result_type(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN int cppyy_method_num_args(cppyy_scope_t scope, cppyy_index_t idx); @@ -130,7 +133,9 @@ RPY_EXTERN char* cppyy_method_arg_default(cppyy_scope_t scope, cppyy_index_t idx, int arg_index); RPY_EXTERN - char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx); + char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); + RPY_EXTERN + char* cppyy_method_prototype(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); RPY_EXTERN int cppyy_method_is_template(cppyy_scope_t scope, cppyy_index_t idx); @@ -147,8 +152,12 @@ /* method properties ------------------------------------------------------ */ RPY_EXTERN + int cppyy_is_publicmethod(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_constructor(cppyy_type_t type, cppyy_index_t idx); RPY_EXTERN + int cppyy_is_destructor(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_staticmethod(cppyy_type_t type, cppyy_index_t idx); /* data member reflection information ------------------------------------- */ diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -2,7 +2,7 @@ from pypy.interpreter.error import OperationError, oefmt from pypy.interpreter.gateway import interp2app, unwrap_spec -from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty, interp_attrproperty_w +from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty from pypy.interpreter.baseobjspace import W_Root from rpython.rtyper.lltypesystem import rffi, lltype, llmemory @@ -15,6 +15,10 @@ from pypy.module._cppyy import converter, executor, ffitypes, helper +INSTANCE_FLAGS_PYTHON_OWNS = 0x0001 +INSTANCE_FLAGS_IS_REF = 0x0002 +INSTANCE_FLAGS_IS_R_VALUE = 0x0004 + class FastCallNotPossible(Exception): pass @@ -33,16 +37,21 @@ class State(object): def __init__(self, space): + # final scoped name -> opaque handle self.cppscope_cache = { - "void" : W_CPPClassDecl(space, "void", capi.C_NULL_TYPE) } + 'void' : W_CPPClassDecl(space, capi.C_NULL_TYPE, 'void') } + # opaque handle -> app-level python class + self.cppclass_registry = {} + # app-level class generator callback + self.w_clgen_callback = None + # app-level function generator callback (currently not used) + self.w_fngen_callback = None + # C++11's nullptr self.w_nullptr = None - self.cpptemplate_cache = {} - self.cppclass_registry = {} - self.w_clgen_callback = None - self.w_fngen_callback = None def get_nullptr(space): - if hasattr(space, "fake"): + # construct a unique address that compares to NULL, serves as nullptr + if hasattr(space, 'fake'): raise NotImplementedError state = space.fromcache(State) if state.w_nullptr is None: @@ -58,52 +67,48 @@ state.w_nullptr = nullarr return state.w_nullptr - at unwrap_spec(name='text') -def resolve_name(space, name): - return space.newtext(capi.c_resolve_name(space, name)) + at unwrap_spec(scoped_name='text') +def resolve_name(space, scoped_name): + return space.newtext(capi.c_resolve_name(space, scoped_name)) - at unwrap_spec(name='text') -def scope_byname(space, name): - true_name = capi.c_resolve_name(space, name) +# memoized lookup of handles by final, scoped, name of classes/namespaces + at unwrap_spec(final_scoped_name='text') +def scope_byname(space, final_scoped_name): state = space.fromcache(State) try: - return state.cppscope_cache[true_name] + return state.cppscope_cache[final_scoped_name] except KeyError: pass - opaque_handle = capi.c_get_scope_opaque(space, true_name) + opaque_handle = capi.c_get_scope_opaque(space, final_scoped_name) assert lltype.typeOf(opaque_handle) == capi.C_SCOPE if opaque_handle: - final_name = capi.c_final_name(space, opaque_handle) - if capi.c_is_namespace(space, opaque_handle): - cppscope = W_CPPNamespaceDecl(space, final_name, opaque_handle) - elif capi.c_has_complex_hierarchy(space, opaque_handle): - cppscope = W_CPPComplexClassDecl(space, final_name, opaque_handle) + isns = capi.c_is_namespace(space, opaque_handle) + if isns: + cppscope = W_CPPNamespaceDecl(space, opaque_handle, final_scoped_name) else: - cppscope = W_CPPClassDecl(space, final_name, opaque_handle) - state.cppscope_cache[name] = cppscope + if capi.c_has_complex_hierarchy(space, opaque_handle): + cppscope = W_CPPComplexClassDecl(space, opaque_handle, final_scoped_name) + else: + cppscope = W_CPPClassDecl(space, opaque_handle, final_scoped_name) - cppscope._build_methods() - cppscope._find_datamembers() + # store in the cache to prevent recursion + state.cppscope_cache[final_scoped_name] = cppscope + + if not isns: + # build methods/data; TODO: also defer this for classes (a functional __dir__ + # and instrospection for help() is enough and allows more lazy loading) + cppscope._build_methods() + cppscope._find_datamembers() + return cppscope return None - at unwrap_spec(name='text') -def template_byname(space, name): - state = space.fromcache(State) - try: - return state.cpptemplate_cache[name] - except KeyError: - pass - - if capi.c_is_template(space, name): - cpptemplate = W_CPPTemplateType(space, name) - state.cpptemplate_cache[name] = cpptemplate - return cpptemplate - - return None + at unwrap_spec(final_scoped_name='text') +def is_template(space, final_scoped_name): + return space.newbool(capi.c_is_template(space, final_scoped_name)) def std_string_name(space): return space.newtext(capi.std_string_name) @@ -189,8 +194,13 @@ # check number of given arguments against required (== total - defaults) args_expected = len(self.arg_defs) args_given = len(args_w) - if args_expected < args_given or args_given < self.args_required: - raise oefmt(self.space.w_TypeError, "wrong number of arguments") + + if args_given < self.args_required: + raise oefmt(self.space.w_TypeError, + "takes at least %d arguments (%d given)", self.args_required, args_given) + elif args_expected < args_given: + raise oefmt(self.space.w_TypeError, + "takes at most %d arguments (%d given)", args_expected, args_given) # initial setup of converters, executors, and libffi (if available) if self.converters is None: @@ -376,8 +386,11 @@ conv.free_argument(self.space, rffi.cast(capi.C_OBJECT, arg_i), loc_i) capi.c_deallocate_function_args(self.space, args) - def signature(self): - return capi.c_method_signature(self.space, self.scope, self.index) + def signature(self, show_formalargs=True): + return capi.c_method_signature(self.space, self.scope, self.index, show_formalargs) + + def prototype(self, show_formalargs=True): + return capi.c_method_prototype(self.space, self.scope, self.index, show_formalargs) def priority(self): total_arg_priority = 0 @@ -391,7 +404,7 @@ lltype.free(self.cif_descr, flavor='raw') def __repr__(self): - return "CPPMethod: %s" % self.signature() + return "CPPMethod: %s" % self.prototype() def _freeze_(self): assert 0, "you should never have a pre-built instance of this!" @@ -407,7 +420,7 @@ return capi.C_NULL_OBJECT def __repr__(self): - return "CPPFunction: %s" % self.signature() + return "CPPFunction: %s" % self.prototype() class CPPTemplatedCall(CPPMethod): @@ -440,7 +453,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPTemplatedCall: %s" % self.signature() + return "CPPTemplatedCall: %s" % self.prototype() class CPPConstructor(CPPMethod): @@ -462,7 +475,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPConstructor: %s" % self.signature() + return "CPPConstructor: %s" % self.prototype() class CPPSetItem(CPPMethod): @@ -549,12 +562,12 @@ w_exc_type = e.w_type elif all_same_type and not e.match(self.space, w_exc_type): all_same_type = False - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' '+e.errorstr(self.space) except Exception as e: # can not special case this for non-overloaded functions as we anyway need an # OperationError error down from here - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' Exception: '+str(e) if all_same_type and w_exc_type is not None: @@ -562,20 +575,20 @@ else: raise OperationError(self.space.w_TypeError, self.space.newtext(errmsg)) - def signature(self): - sig = self.functions[0].signature() + def prototype(self): + sig = self.functions[0].prototype() for i in range(1, len(self.functions)): - sig += '\n'+self.functions[i].signature() + sig += '\n'+self.functions[i].prototype() return self.space.newtext(sig) def __repr__(self): - return "W_CPPOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPOverload(%s)" % [f.prototype() for f in self.functions] W_CPPOverload.typedef = TypeDef( 'CPPOverload', is_static = interp2app(W_CPPOverload.is_static), call = interp2app(W_CPPOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPOverload.prototype), ) @@ -591,24 +604,40 @@ @jit.unroll_safe @unwrap_spec(args_w='args_w') def call(self, w_cppinstance, args_w): + # TODO: factor out the following: + if capi.c_is_abstract(self.space, self.scope.handle): + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.scope.name) w_result = W_CPPOverload.call(self, w_cppinstance, args_w) newthis = rffi.cast(capi.C_OBJECT, self.space.uint_w(w_result)) cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if cppinstance is not None: cppinstance._rawobject = newthis memory_regulator.register(cppinstance) - return w_cppinstance - return wrap_cppobject(self.space, newthis, self.functions[0].scope, - do_cast=False, python_owns=True, fresh=True) def __repr__(self): - return "W_CPPConstructorOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPConstructorOverload(%s)" % [f.prototype() for f in self.functions] W_CPPConstructorOverload.typedef = TypeDef( 'CPPConstructorOverload', is_static = interp2app(W_CPPConstructorOverload.is_static), call = interp2app(W_CPPConstructorOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPConstructorOverload.prototype), +) + + +class W_CPPTemplateOverload(W_CPPOverload): + @unwrap_spec(args_w='args_w') + def __getitem__(self, args_w): + pass + + def __repr__(self): + return "W_CPPTemplateOverload(%s)" % [f.prototype() for f in self.functions] + +W_CPPTemplateOverload.typedef = TypeDef( + 'CPPTemplateOverload', + __getitem__ = interp2app(W_CPPTemplateOverload.call), ) @@ -622,6 +651,9 @@ def __call__(self, args_w): return self.method.bound_call(self.cppthis, args_w) + def __repr__(self): + return "W_CPPBoundMethod(%s)" % [f.prototype() for f in self.functions] + W_CPPBoundMethod.typedef = TypeDef( 'CPPBoundMethod', __call__ = interp2app(W_CPPBoundMethod.__call__), @@ -643,8 +675,8 @@ def _get_offset(self, cppinstance): if cppinstance: - assert lltype.typeOf(cppinstance.cppclass.handle) == lltype.typeOf(self.scope.handle) - offset = self.offset + cppinstance.cppclass.get_base_offset(cppinstance, self.scope) + assert lltype.typeOf(cppinstance.clsdecl.handle) == lltype.typeOf(self.scope.handle) + offset = self.offset + cppinstance.clsdecl.get_base_offset(cppinstance, self.scope) else: offset = self.offset return offset @@ -652,7 +684,7 @@ def get(self, w_cppinstance, w_pycppclass): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) return self.converter.from_memory(self.space, w_cppinstance, w_pycppclass, offset) @@ -660,7 +692,7 @@ def set(self, w_cppinstance, w_value): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) self.converter.to_memory(self.space, w_cppinstance, w_value, offset) @@ -705,12 +737,12 @@ return space.w_False class W_CPPScopeDecl(W_Root): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] _immutable_fields_ = ['handle', 'name'] - def __init__(self, space, name, opaque_handle): + def __init__(self, space, opaque_handle, final_scoped_name): self.space = space - self.name = name + self.name = final_scoped_name assert lltype.typeOf(opaque_handle) == capi.C_SCOPE self.handle = opaque_handle self.methods = {} @@ -753,7 +785,7 @@ overload = self.get_overload(name) sig = '(%s)' % signature for f in overload.functions: - if 0 < f.signature().find(sig): + if f.signature(False) == sig: return W_CPPOverload(self.space, self, [f]) raise oefmt(self.space.w_LookupError, "no overload matches signature") @@ -769,6 +801,9 @@ # classes for inheritance. Both are python classes, though, and refactoring # may be in order at some point. class W_CPPNamespaceDecl(W_CPPScopeDecl): + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name'] + def _make_cppfunction(self, pyname, index): num_args = capi.c_method_num_args(self.space, self, index) args_required = capi.c_method_req_args(self.space, self, index) @@ -779,9 +814,6 @@ arg_defs.append((arg_type, arg_dflt)) return CPPFunction(self.space, self, index, arg_defs, args_required) - def _build_methods(self): - pass # force lazy lookups in namespaces - def _make_datamember(self, dm_name, dm_idx): type_name = capi.c_datamember_type(self.space, self, dm_idx) offset = capi.c_datamember_offset(self.space, self, dm_idx) @@ -791,9 +823,6 @@ self.datamembers[dm_name] = datamember return datamember - def _find_datamembers(self): - pass # force lazy lookups in namespaces - def find_overload(self, meth_name): indices = capi.c_method_indices_from_name(self.space, self, meth_name) if not indices: @@ -855,18 +884,21 @@ class W_CPPClassDecl(W_CPPScopeDecl): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] - _immutable_fields_ = ['handle', 'constructor', 'methods[*]', 'datamembers[*]'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name', 'methods[*]', 'datamembers[*]'] def _build_methods(self): assert len(self.methods) == 0 methods_temp = {} for i in range(capi.c_num_methods(self.space, self)): idx = capi.c_method_index_at(self.space, self, i) - pyname = helper.map_operator_name(self.space, - capi.c_method_name(self.space, self, idx), - capi.c_method_num_args(self.space, self, idx), - capi.c_method_result_type(self.space, self, idx)) + if capi.c_is_constructor(self.space, self, idx): + pyname = '__init__' + else: + pyname = helper.map_operator_name(self.space, + capi.c_method_name(self.space, self, idx), + capi.c_method_num_args(self.space, self, idx), + capi.c_method_result_type(self.space, self, idx)) cppmethod = self._make_cppfunction(pyname, idx) methods_temp.setdefault(pyname, []).append(cppmethod) # the following covers the case where the only kind of operator[](idx) @@ -883,7 +915,7 @@ # create the overload methods from the method sets for pyname, methods in methods_temp.iteritems(): CPPMethodSort(methods).sort() - if pyname == self.name: + if pyname == '__init__': overload = W_CPPConstructorOverload(self.space, self, methods[:]) else: overload = W_CPPOverload(self.space, self, methods[:]) @@ -934,11 +966,11 @@ raise self.missing_attribute_error(name) def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return 0 def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return cppinstance.get_rawobject() def is_namespace(self): @@ -973,13 +1005,13 @@ class W_CPPComplexClassDecl(W_CPPClassDecl): def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = capi.c_base_offset(self.space, self, calling_scope, cppinstance.get_rawobject(), 1) return offset def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = self.get_base_offset(cppinstance, calling_scope) return capi.direct_ptradd(cppinstance.get_rawobject(), offset) @@ -997,70 +1029,56 @@ W_CPPComplexClassDecl.typedef.acceptable_as_base_class = False -class W_CPPTemplateType(W_Root): - _attrs_ = ['space', 'name'] - _immutable_fields = ['name'] - - def __init__(self, space, name): - self.space = space - self.name = name - - @unwrap_spec(args_w='args_w') - def __call__(self, args_w): - # TODO: this is broken but unused (see pythonify.py) - fullname = "".join([self.name, '<', self.space.text_w(args_w[0]), '>']) - return scope_byname(self.space, fullname) - -W_CPPTemplateType.typedef = TypeDef( - 'CPPTemplateType', - __call__ = interp2app(W_CPPTemplateType.__call__), -) -W_CPPTemplateType.typedef.acceptable_as_base_class = False - - class W_CPPClass(W_Root): - _attrs_ = ['space', 'cppclass', '_rawobject', 'isref', 'python_owns', + _attrs_ = ['space', 'clsdecl', '_rawobject', 'flags', 'finalizer_registered'] - _immutable_fields_ = ["cppclass", "isref"] + _immutable_fields_ = ['clsdecl'] finalizer_registered = False - def __init__(self, space, cppclass, rawobject, isref, python_owns): + def __init__(self, space, decl, rawobject, isref, python_owns): self.space = space - self.cppclass = cppclass + self.clsdecl = decl assert lltype.typeOf(rawobject) == capi.C_OBJECT assert not isref or rawobject self._rawobject = rawobject assert not isref or not python_owns - self.isref = isref - self.python_owns = python_owns - self._opt_register_finalizer() + self.flags = 0 + if isref: + self.flags |= INSTANCE_FLAGS_IS_REF + if python_owns: + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() def _opt_register_finalizer(self): - if self.python_owns and not self.finalizer_registered \ - and not hasattr(self.space, "fake"): + if not self.finalizer_registered and not hasattr(self.space, "fake"): + assert self.flags & INSTANCE_FLAGS_PYTHON_OWNS self.register_finalizer(self.space) self.finalizer_registered = True def _nullcheck(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): raise oefmt(self.space.w_ReferenceError, "trying to access a NULL pointer") # allow user to determine ownership rules on a per object level def fget_python_owns(self, space): - return space.newbool(self.python_owns) + return space.newbool(bool(self.flags & INSTANCE_FLAGS_PYTHON_OWNS)) @unwrap_spec(value=bool) def fset_python_owns(self, space, value): - self.python_owns = space.is_true(value) - self._opt_register_finalizer() + if space.is_true(value): + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() + else: + self.flags &= ~INSTANCE_FLAGS_PYTHON_OWNS def get_cppthis(self, calling_scope): - return self.cppclass.get_cppthis(self, calling_scope) + return self.clsdecl.get_cppthis(self, calling_scope) def get_rawobject(self): - if not self.isref: + if not (self.flags & INSTANCE_FLAGS_IS_REF): return self._rawobject else: ptrptr = rffi.cast(rffi.VOIDPP, self._rawobject) @@ -1078,12 +1096,9 @@ return None def instance__init__(self, args_w): - if capi.c_is_abstract(self.space, self.cppclass.handle): - raise oefmt(self.space.w_TypeError, - "cannot instantiate abstract class '%s'", - self.cppclass.name) - constructor_overload = self.cppclass.get_overload(self.cppclass.name) - constructor_overload.call(self, args_w) + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.clsdecl.name) def instance__eq__(self, w_other): # special case: if other is None, compare pointer-style @@ -1099,7 +1114,7 @@ for name in ["", "__gnu_cxx", "__1"]: nss = scope_byname(self.space, name) meth_idx = capi.c_get_global_operator( - self.space, nss, self.cppclass, other.cppclass, "operator==") + self.space, nss, self.clsdecl, other.clsdecl, "operator==") if meth_idx != -1: f = nss._make_cppfunction("operator==", meth_idx) ol = W_CPPOverload(self.space, nss, [f]) @@ -1118,14 +1133,15 @@ # fallback 2: direct pointer comparison (the class comparison is needed since # the first data member in a struct and the struct have the same address) other = self.space.interp_w(W_CPPClass, w_other, can_be_None=False) # TODO: factor out - iseq = (self._rawobject == other._rawobject) and (self.cppclass == other.cppclass) + iseq = (self._rawobject == other._rawobject) and (self.clsdecl == other.clsdecl) return self.space.newbool(iseq) def instance__ne__(self, w_other): return self.space.not_(self.instance__eq__(w_other)) def instance__nonzero__(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): return self.space.w_False return self.space.w_True @@ -1134,36 +1150,35 @@ if w_as_builtin is not None: return self.space.len(w_as_builtin) raise oefmt(self.space.w_TypeError, - "'%s' has no length", self.cppclass.name) + "'%s' has no length", self.clsdecl.name) def instance__cmp__(self, w_other): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.cmp(w_as_builtin, w_other) raise oefmt(self.space.w_AttributeError, - "'%s' has no attribute __cmp__", self.cppclass.name) + "'%s' has no attribute __cmp__", self.clsdecl.name) def instance__repr__(self): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.repr(w_as_builtin) return self.space.newtext("<%s object at 0x%x>" % - (self.cppclass.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) + (self.clsdecl.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) def destruct(self): - if self._rawobject and not self.isref: + if self._rawobject and not (self.flags & INSTANCE_FLAGS_IS_REF): memory_regulator.unregister(self) - capi.c_destruct(self.space, self.cppclass, self._rawobject) + capi.c_destruct(self.space, self.clsdecl, self._rawobject) self._rawobject = capi.C_NULL_OBJECT def _finalize_(self): - if self.python_owns: + if self.flags & INSTANCE_FLAGS_PYTHON_OWNS: self.destruct() W_CPPClass.typedef = TypeDef( 'CPPClass', - cppclass = interp_attrproperty_w('cppclass', cls=W_CPPClass), - _python_owns = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), + __python_owns__ = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), __init__ = interp2app(W_CPPClass.instance__init__), __eq__ = interp2app(W_CPPClass.instance__eq__), __ne__ = interp2app(W_CPPClass.instance__ne__), @@ -1220,21 +1235,21 @@ state = space.fromcache(State) return space.call_function(state.w_fngen_callback, w_callable, space.newint(npar)) -def wrap_cppobject(space, rawobject, cppclass, - do_cast=True, python_owns=False, is_ref=False, fresh=False): +def wrap_cppinstance(space, rawobject, clsdecl, + do_cast=True, python_owns=False, is_ref=False, fresh=False): rawobject = rffi.cast(capi.C_OBJECT, rawobject) # cast to actual if requested and possible w_pycppclass = None if do_cast and rawobject: - actual = capi.c_actual_class(space, cppclass, rawobject) - if actual != cppclass.handle: + actual = capi.c_actual_class(space, clsdecl, rawobject) + if actual != clsdecl.handle: try: w_pycppclass = get_pythonized_cppclass(space, actual) - offset = capi.c_base_offset1(space, actual, cppclass, rawobject, -1) + offset = capi.c_base_offset1(space, actual, clsdecl, rawobject, -1) rawobject = capi.direct_ptradd(rawobject, offset) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) + w_cppdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + clsdecl = space.interp_w(W_CPPClassDecl, w_cppdecl, can_be_None=False) except Exception: # failed to locate/build the derived class, so stick to the base (note # that only get_pythonized_cppclass is expected to raise, so none of @@ -1242,18 +1257,18 @@ pass if w_pycppclass is None: - w_pycppclass = get_pythonized_cppclass(space, cppclass.handle) + w_pycppclass = get_pythonized_cppclass(space, clsdecl.handle) # try to recycle existing object if this one is not newly created if not fresh and rawobject: obj = memory_regulator.retrieve(rawobject) - if obj is not None and obj.cppclass is cppclass: + if obj is not None and obj.clsdecl is clsdecl: return obj # fresh creation w_cppinstance = space.allocate_instance(W_CPPClass, w_pycppclass) cppinstance = space.interp_w(W_CPPClass, w_cppinstance, can_be_None=False) - cppinstance.__init__(space, cppclass, rawobject, is_ref, python_owns) + cppinstance.__init__(space, clsdecl, rawobject, is_ref, python_owns) memory_regulator.register(cppinstance) return w_cppinstance @@ -1264,7 +1279,7 @@ except TypeError: pass # attempt to get address of C++ instance - return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj)) + return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj, False)) @unwrap_spec(w_obj=W_Root) def addressof(space, w_obj): @@ -1273,19 +1288,30 @@ return space.newlong(address) @unwrap_spec(owns=bool, cast=bool) -def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): - """Takes an address and a bound C++ class proxy, returns a bound instance.""" +def _bind_object(space, w_obj, w_clsdecl, owns=False, cast=False): try: # attempt address from array or C++ instance rawobject = rffi.cast(capi.C_OBJECT, _addressof(space, w_obj)) except Exception: # accept integer value as address rawobject = rffi.cast(capi.C_OBJECT, space.uint_w(w_obj)) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - if not w_cppclass: - w_cppclass = scope_byname(space, space.text_w(w_pycppclass)) - if not w_cppclass: + decl = space.interp_w(W_CPPClassDecl, w_clsdecl, can_be_None=False) + return wrap_cppinstance(space, rawobject, decl, python_owns=owns, do_cast=cast) + + at unwrap_spec(owns=bool, cast=bool) +def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): + """Takes an address and a bound C++ class proxy, returns a bound instance.""" + w_clsdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + if not w_clsdecl: + w_clsdecl = scope_byname(space, space.text_w(w_pycppclass)) + if not w_clsdecl: raise oefmt(space.w_TypeError, "no such class: %s", space.text_w(w_pycppclass)) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) - return wrap_cppobject(space, rawobject, cppclass, do_cast=cast, python_owns=owns) + return _bind_object(space, w_obj, w_clsdecl, owns, cast) + +def move(space, w_obj): + """Casts the given instance into an C++-style rvalue.""" + obj = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + if obj: + obj.flags |= INSTANCE_FLAGS_IS_R_VALUE + return w_obj diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -10,7 +10,7 @@ class CPPMetaScope(type): def __getattr__(self, name): try: - return get_pycppitem(self, name) # will cache on self + return get_scoped_pycppitem(self, name) # will cache on self except Exception as e: raise AttributeError("%s object has no attribute '%s' (details: %s)" % (self, name, str(e))) @@ -36,11 +36,14 @@ self._scope = scope def _arg_to_str(self, arg): - if arg == str: - import _cppyy - arg = _cppyy._std_string_name() - elif type(arg) != str: - arg = arg.__name__ + try: + arg = arg.__cppname__ + except AttributeError: + if arg == str: + import _cppyy + arg = _cppyy._std_string_name() + elif type(arg) != str: + arg = arg.__name__ return arg def __call__(self, *args): @@ -58,8 +61,36 @@ return self.__call__(*args) -def clgen_callback(name): - return get_pycppclass(name) +def scope_splitter(name): + is_open_template, scope = 0, "" + for c in name: + if c == ':' and not is_open_template: + if scope: + yield scope + scope = "" + continue + elif c == '<': + is_open_template += 1 + elif c == '>': + is_open_template -= 1 + scope += c + yield scope + +def get_pycppitem(final_scoped_name): + # walk scopes recursively down from global namespace ("::") to get the + # actual (i.e. not typedef'ed) class, triggering all necessary creation + scope = gbl + for name in scope_splitter(final_scoped_name): + scope = getattr(scope, name) + return scope +get_pycppclass = get_pycppitem # currently no distinction, but might + # in future for performance + + +# callbacks (originating from interp_cppyy.py) to allow interp-level to +# initiate creation of app-level classes and function +def clgen_callback(final_scoped_name): + return get_pycppclass(final_scoped_name) def fngen_callback(func, npar): # todo, some kind of arg transform spec if npar == 0: @@ -75,20 +106,19 @@ return wrapper +# construction of namespaces and classes, and their helpers +def make_module_name(scope): + if scope: + return scope.__module__ + '.' + scope.__name__ + return 'cppyy' + def make_static_function(func_name, cppol): def function(*args): return cppol.call(None, *args) function.__name__ = func_name - function.__doc__ = cppol.signature() + function.__doc__ = cppol.prototype() return staticmethod(function) -def make_method(meth_name, cppol): - def method(self, *args): - return cppol.call(self, *args) - method.__name__ = meth_name - method.__doc__ = cppol.signature() - return method - def make_cppnamespace(scope, name, decl): # build up a representation of a C++ namespace (namespaces are classes) @@ -98,20 +128,19 @@ ns_meta = type(name+'_meta', (CPPMetaNamespace,), {}) # create the python-side C++ namespace representation, cache in scope if given - d = {"__cppdecl__" : decl, "__cppname__" : decl.__cppname__ } + d = {"__cppdecl__" : decl, + "__module__" : make_module_name(scope), + "__cppname__" : decl.__cppname__ } pyns = ns_meta(name, (CPPNamespace,), d) if scope: setattr(scope, name, pyns) # install as modules to allow importing from (note naming: cppyy) - modname = 'cppyy.gbl' - if scope: - modname = 'cppyy.gbl.'+pyns.__cppname__.replace('::', '.') - sys.modules[modname] = pyns + sys.modules[make_module_name(pyns)] = pyns return pyns def _drop_cycles(bases): - # TODO: figure this out, as it seems to be a PyPy bug?! + # TODO: figure out why this is necessary? for b1 in bases: for b2 in bases: if not (b1 is b2) and issubclass(b2, b1): @@ -119,27 +148,37 @@ break return tuple(bases) -def make_new(class_name): + +def make_new(decl): def __new__(cls, *args): # create a place-holder only as there may be a derived class defined + # TODO: get rid of the import and add user-land bind_object that uses + # _bind_object (see interp_cppyy.py) import _cppyy - instance = _cppyy.bind_object(0, class_name, True) + instance = _cppyy._bind_object(0, decl, True) if not instance.__class__ is cls: instance.__class__ = cls # happens for derived class return instance return __new__ -def make_cppclass(scope, class_name, final_class_name, decl): +def make_method(meth_name, cppol): + def method(self, *args): + return cppol.call(self, *args) + method.__name__ = meth_name + method.__doc__ = cppol.prototype() + return method + +def make_cppclass(scope, cl_name, decl): # get a list of base classes for class creation bases = [get_pycppclass(base) for base in decl.get_base_names()] if not bases: bases = [CPPClass,] else: - # it's technically possible that the required class now has been built - # if one of the base classes uses it in e.g. a function interface + # it's possible that the required class now has been built if one of + # the base classes uses it in e.g. a function interface try: - return scope.__dict__[final_class_name] + return scope.__dict__[cl_name] except KeyError: pass @@ -147,39 +186,41 @@ d_meta = {} # prepare dictionary for python-side C++ class representation - def dispatch(self, name, signature): - cppol = decl.dispatch(name, signature) - return types.MethodType(make_method(name, cppol), self, type(self)) + def dispatch(self, m_name, signature): + cppol = decl.__dispatch__(m_name, signature) + return types.MethodType(make_method(m_name, cppol), self, type(self)) d_class = {"__cppdecl__" : decl, + "__new__" : make_new(decl), + "__module__" : make_module_name(scope), "__cppname__" : decl.__cppname__, - "__new__" : make_new(class_name), + "__dispatch__" : dispatch, } # insert (static) methods into the class dictionary - for name in decl.get_method_names(): - cppol = decl.get_overload(name) + for m_name in decl.get_method_names(): + cppol = decl.get_overload(m_name) if cppol.is_static(): - d_class[name] = make_static_function(name, cppol) + d_class[m_name] = make_static_function(m_name, cppol) else: - d_class[name] = make_method(name, cppol) + d_class[m_name] = make_method(m_name, cppol) # add all data members to the dictionary of the class to be created, and # static ones also to the metaclass (needed for property setters) - for name in decl.get_datamember_names(): - cppdm = decl.get_datamember(name) - d_class[name] = cppdm + for d_name in decl.get_datamember_names(): + cppdm = decl.get_datamember(d_name) + d_class[d_name] = cppdm if cppdm.is_static(): - d_meta[name] = cppdm + d_meta[d_name] = cppdm # create a metaclass to allow properties (for static data write access) metabases = [type(base) for base in bases] - metacpp = type(CPPMetaScope)(class_name+'_meta', _drop_cycles(metabases), d_meta) + metacpp = type(CPPMetaScope)(cl_name+'_meta', _drop_cycles(metabases), d_meta) # create the python-side C++ class - pycls = metacpp(class_name, _drop_cycles(bases), d_class) + pycls = metacpp(cl_name, _drop_cycles(bases), d_class) # store the class on its outer scope - setattr(scope, final_class_name, pycls) + setattr(scope, cl_name, pycls) # the call to register will add back-end specific pythonizations and thus # needs to run first, so that the generic pythonizations can use them @@ -192,32 +233,32 @@ return CPPTemplate(template_name, scope) -def get_pycppitem(scope, name): +def get_scoped_pycppitem(scope, name): import _cppyy - # resolve typedefs/aliases - full_name = (scope == gbl) and name or (scope.__name__+'::'+name) - true_name = _cppyy._resolve_name(full_name) - if true_name != full_name: - return get_pycppclass(true_name) + # resolve typedefs/aliases: these may cross namespaces, in which case + # the lookup must trigger the creation of all necessary scopes + scoped_name = (scope == gbl) and name or (scope.__cppname__+'::'+name) + final_scoped_name = _cppyy._resolve_name(scoped_name) + if final_scoped_name != scoped_name: + pycppitem = get_pycppitem(final_scoped_name) + # also store on the requested scope (effectively a typedef or pointer copy) + setattr(scope, name, pycppitem) + return pycppitem pycppitem = None - # classes - cppitem = _cppyy._scope_byname(true_name) + # scopes (classes and namespaces) + cppitem = _cppyy._scope_byname(final_scoped_name) if cppitem: - name = true_name - if scope != gbl: - name = true_name[len(scope.__cppname__)+2:] if cppitem.is_namespace(): pycppitem = make_cppnamespace(scope, name, cppitem) - setattr(scope, name, pycppitem) else: - pycppitem = make_cppclass(scope, name, true_name, cppitem) + pycppitem = make_cppclass(scope, name, cppitem) # templates if not cppitem: - cppitem = _cppyy._template_byname(true_name) + cppitem = _cppyy._is_template(final_scoped_name) if cppitem: pycppitem = make_cpptemplatetype(scope, name) setattr(scope, name, pycppitem) @@ -249,29 +290,6 @@ raise AttributeError("'%s' has no attribute '%s'" % (str(scope), name)) -def scope_splitter(name): - is_open_template, scope = 0, "" - for c in name: - if c == ':' and not is_open_template: - if scope: - yield scope - scope = "" - continue - elif c == '<': - is_open_template += 1 - elif c == '>': - is_open_template -= 1 - scope += c - yield scope - -def get_pycppclass(name): - # break up the name, to walk the scopes and get the class recursively - scope = gbl - for part in scope_splitter(name): - scope = getattr(scope, part) - return scope - - # pythonization by decoration (move to their own file?) def python_style_getitem(self, idx): # python-style indexing: check for size and allow indexing from the back @@ -346,8 +364,8 @@ # also the fallback on the indexed __getitem__, but that is slower) if not 'vector' in pyclass.__name__[:11] and \ ('begin' in pyclass.__dict__ and 'end' in pyclass.__dict__): - if _cppyy._scope_byname(pyclass.__name__+'::iterator') or \ - _cppyy._scope_byname(pyclass.__name__+'::const_iterator'): + if _cppyy._scope_byname(pyclass.__cppname__+'::iterator') or \ + _cppyy._scope_byname(pyclass.__cppname__+'::const_iterator'): def __iter__(self): i = self.begin() while i != self.end(): @@ -416,17 +434,21 @@ # pre-create std to allow direct importing gbl.std = make_cppnamespace(gbl, 'std', _cppyy._scope_byname('std')) + # add move cast + gbl.std.move = _cppyy.move + # install a type for enums to refer to # TODO: this is correct for C++98, not for C++11 and in general there will # be the same issue for all typedef'd builtin types setattr(gbl, 'internal_enum_type_t', int) - # install nullptr as a unique reference - setattr(gbl, 'nullptr', _cppyy._get_nullptr()) - # install for user access _cppyy.gbl = gbl + # install nullptr as a unique reference + _cppyy.nullptr = _cppyy._get_nullptr() + + # user-defined pythonizations interface _pythonizations = {} def add_pythonization(class_name, callback): diff --git a/pypy/module/_cppyy/src/dummy_backend.cxx b/pypy/module/_cppyy/src/dummy_backend.cxx --- a/pypy/module/_cppyy/src/dummy_backend.cxx +++ b/pypy/module/_cppyy/src/dummy_backend.cxx @@ -955,7 +955,13 @@ return cppstring_to_cstring(""); } -char* cppyy_method_signature(cppyy_scope_t /* handle */, cppyy_index_t /* method_index */) { +char* cppyy_method_signature( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { + return cppstring_to_cstring(""); +} + +char* cppyy_method_prototype( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { return cppstring_to_cstring(""); } diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile --- a/pypy/module/_cppyy/test/Makefile +++ b/pypy/module/_cppyy/test/Makefile @@ -1,12 +1,14 @@ dicts = advancedcppDict.so \ advancedcpp2Dict.so \ + cpp11featuresDict.so \ crossingDict.so \ datatypesDict.so \ example01Dict.so \ fragileDict.so \ operatorsDict.so \ overloadsDict.so \ - stltypesDict.so + stltypesDict.so \ + templatesDict.so all : $(dicts) diff --git a/pypy/module/_cppyy/test/advancedcpp.cxx b/pypy/module/_cppyy/test/advancedcpp.cxx --- a/pypy/module/_cppyy/test/advancedcpp.cxx +++ b/pypy/module/_cppyy/test/advancedcpp.cxx @@ -106,17 +106,6 @@ } -// more template testing -long my_templated_method_class::get_size() { return -1; } - -long my_templated_method_class::get_char_size() { return (long)sizeof(char); } -long my_templated_method_class::get_int_size() { return (long)sizeof(int); } -long my_templated_method_class::get_long_size() { return (long)sizeof(long); } -long my_templated_method_class::get_float_size() { return (long)sizeof(float); } -long my_templated_method_class::get_double_size() { return (long)sizeof(double); } -long my_templated_method_class::get_self_size() { return (long)sizeof(my_templated_method_class); } - - // overload order testing int overload_one_way::gime() const { return 1; } std::string overload_one_way::gime() { return "aap"; } diff --git a/pypy/module/_cppyy/test/advancedcpp.h b/pypy/module/_cppyy/test/advancedcpp.h --- a/pypy/module/_cppyy/test/advancedcpp.h +++ b/pypy/module/_cppyy/test/advancedcpp.h @@ -246,8 +246,6 @@ int m_i; }; -template class std::vector; - //=========================================================================== class some_convertible { // for math conversions testing @@ -275,6 +273,7 @@ extern double my_global_double; // a couple of globals for access testing extern double my_global_array[500]; extern double* my_global_ptr; +static const char my_global_string[] = "aap " " noot " " mies"; //=========================================================================== class some_class_with_data { // for life-line and identity testing @@ -387,37 +386,6 @@ template char my_templated_function(char); template double my_templated_function(double); -class my_templated_method_class { -public: - long get_size(); // to get around bug in genreflex - template long get_size(); - - long get_char_size(); - long get_int_size(); - long get_long_size(); - long get_float_size(); - long get_double_size(); - - long get_self_size(); - -private: - double m_data[3]; -}; - -template -inline long my_templated_method_class::get_size() { - return sizeof(B); -} - -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); - -typedef my_templated_method_class my_typedef_t; -template long my_templated_method_class::get_size(); - //=========================================================================== class overload_one_way { // overload order testing diff --git a/pypy/module/_cppyy/test/advancedcpp.xml b/pypy/module/_cppyy/test/advancedcpp.xml --- a/pypy/module/_cppyy/test/advancedcpp.xml +++ b/pypy/module/_cppyy/test/advancedcpp.xml @@ -53,8 +53,6 @@ - - diff --git a/pypy/module/_cppyy/test/cpp11features.cxx b/pypy/module/_cppyy/test/cpp11features.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.cxx @@ -0,0 +1,18 @@ +#if __cplusplus >= 201103L + +#include "cpp11features.h" + + +// for std::shared_ptr<> testing +int TestSharedPtr::s_counter = 0; + +std::shared_ptr create_shared_ptr_instance() { + return std::shared_ptr(new TestSharedPtr); +} + + +// for move ctors etc. +int TestMoving1::s_move_counter = 0; +int TestMoving2::s_move_counter = 0; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.h b/pypy/module/_cppyy/test/cpp11features.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.h @@ -0,0 +1,45 @@ +#if __cplusplus >= 201103L + +#include + + +//=========================================================================== +class TestSharedPtr { // for std::shared_ptr<> testing +public: + static int s_counter; + +public: + TestSharedPtr() { ++s_counter; } + TestSharedPtr(const TestSharedPtr&) { ++s_counter; } + ~TestSharedPtr() { --s_counter; } +}; + +std::shared_ptr create_shared_ptr_instance(); + + +//=========================================================================== +class TestMoving1 { // for move ctors etc. +public: + static int s_move_counter; + +public: + TestMoving1() {} + TestMoving1(TestMoving1&&) { ++s_move_counter; } + TestMoving1(const TestMoving1&) {} + TestMoving1& operator=(TestMoving1&&) { ++s_move_counter; return *this; } + TestMoving1& operator=(TestMoving1&) { return *this; } +}; + +class TestMoving2 { // note opposite method order from TestMoving1 +public: + static int s_move_counter; + +public: + TestMoving2() {} + TestMoving2(const TestMoving2&) {} + TestMoving2(TestMoving2&& other) { ++s_move_counter; } + TestMoving2& operator=(TestMoving2&) { return *this; } + TestMoving2& operator=(TestMoving2&&) { ++s_move_counter; return *this; } +}; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.xml b/pypy/module/_cppyy/test/cpp11features.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/fragile.h b/pypy/module/_cppyy/test/fragile.h --- a/pypy/module/_cppyy/test/fragile.h +++ b/pypy/module/_cppyy/test/fragile.h @@ -30,9 +30,11 @@ void overload(int, no_such_class* p = 0) {} }; +static const int dummy_location = 0xdead; + class E { public: - E() : m_pp_no_such(0), m_pp_a(0) {} + E() : m_pp_no_such((no_such_class**)&dummy_location), m_pp_a(0) {} virtual int check() { return (int)'E'; } void overload(no_such_class**) {} diff --git a/pypy/module/_cppyy/test/templates.cxx b/pypy/module/_cppyy/test/templates.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.cxx @@ -0,0 +1,12 @@ +#include "templates.h" + + +// template methods +long MyTemplatedMethodClass::get_size() { return -1; } + +long MyTemplatedMethodClass::get_char_size() { return (long)sizeof(char); } +long MyTemplatedMethodClass::get_int_size() { return (long)sizeof(int); } +long MyTemplatedMethodClass::get_long_size() { return (long)42; /* "lying" */ } +long MyTemplatedMethodClass::get_float_size() { return (long)sizeof(float); } +long MyTemplatedMethodClass::get_double_size() { return (long)sizeof(double); } +long MyTemplatedMethodClass::get_self_size() { return (long)sizeof(MyTemplatedMethodClass); } diff --git a/pypy/module/_cppyy/test/templates.h b/pypy/module/_cppyy/test/templates.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.h @@ -0,0 +1,35 @@ +//=========================================================================== +class MyTemplatedMethodClass { // template methods +public: + long get_size(); // to get around bug in genreflex + template long get_size(); + + long get_char_size(); + long get_int_size(); + long get_long_size(); + long get_float_size(); + long get_double_size(); + + long get_self_size(); + +private: + double m_data[3]; +}; + +template +inline long MyTemplatedMethodClass::get_size() { + return sizeof(B); +} + +// +typedef MyTemplatedMethodClass MyTMCTypedef_t; + +// explicit instantiation +template long MyTemplatedMethodClass::get_size(); +template long MyTemplatedMethodClass::get_size(); + +// "lying" specialization +template<> +inline long MyTemplatedMethodClass::get_size() { + return 42; +} diff --git a/pypy/module/_cppyy/test/templates.xml b/pypy/module/_cppyy/test/templates.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/test_advancedcpp.py b/pypy/module/_cppyy/test/test_advancedcpp.py --- a/pypy/module/_cppyy/test/test_advancedcpp.py +++ b/pypy/module/_cppyy/test/test_advancedcpp.py @@ -28,9 +28,9 @@ def test01_default_arguments(self): """Test usage of default arguments""" - import _cppyy + import _cppyy as cppyy def test_defaulter(n, t): - defaulter = getattr(_cppyy.gbl, '%s_defaulter' % n) + defaulter = getattr(cppyy.gbl, '%s_defaulter' % n) d = defaulter() assert d.m_a == t(11) @@ -55,23 +55,23 @@ assert d.m_b == t(4) assert d.m_c == t(5) d.__destruct__() - test_defaulter('short', int) + test_defaulter('short', int) test_defaulter('ushort', int) - test_defaulter('int', int) - test_defaulter('uint', int) - test_defaulter('long', long) - test_defaulter('ulong', long) - test_defaulter('llong', long) + test_defaulter('int', int) + test_defaulter('uint', int) + test_defaulter('long', long) + test_defaulter('ulong', long) + test_defaulter('llong', long) test_defaulter('ullong', long) - test_defaulter('float', float) + test_defaulter('float', float) test_defaulter('double', float) def test02_simple_inheritance(self): """Test binding of a basic inheritance structure""" - import _cppyy - base_class = _cppyy.gbl.base_class - derived_class = _cppyy.gbl.derived_class + import _cppyy as cppyy + base_class = cppyy.gbl.base_class + derived_class = cppyy.gbl.derived_class assert issubclass(derived_class, base_class) assert not issubclass(base_class, derived_class) @@ -123,8 +123,8 @@ def test03_namespaces(self): """Test access to namespaces and inner classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.a_ns is gbl.a_ns assert gbl.a_ns.d_ns is gbl.a_ns.d_ns @@ -150,8 +150,8 @@ def test03a_namespace_lookup_on_update(self): """Test whether namespaces can be shared across dictionaries.""" - import _cppyy, ctypes - gbl = _cppyy.gbl + import _cppyy as cppyy, ctypes + gbl = cppyy.gbl lib2 = ctypes.CDLL("./advancedcpp2Dict.so", ctypes.RTLD_GLOBAL) @@ -179,8 +179,8 @@ def test04_template_types(self): """Test bindings of templated types""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.T1 is gbl.T1 assert gbl.T2 is gbl.T2 @@ -245,8 +245,8 @@ def test05_abstract_classes(self): """Test non-instatiatability of abstract classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl raises(TypeError, gbl.a_class) raises(TypeError, gbl.some_abstract_class) @@ -260,12 +260,12 @@ def test06_datamembers(self): """Test data member access when using virtual inheritence""" - import _cppyy - a_class = _cppyy.gbl.a_class - b_class = _cppyy.gbl.b_class - c_class_1 = _cppyy.gbl.c_class_1 - c_class_2 = _cppyy.gbl.c_class_2 - d_class = _cppyy.gbl.d_class + import _cppyy as cppyy + a_class = cppyy.gbl.a_class + b_class = cppyy.gbl.b_class + c_class_1 = cppyy.gbl.c_class_1 + c_class_2 = cppyy.gbl.c_class_2 + d_class = cppyy.gbl.d_class assert issubclass(b_class, a_class) assert issubclass(c_class_1, a_class) @@ -354,8 +354,8 @@ def test07_pass_by_reference(self): """Test reference passing when using virtual inheritance""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl b_class = gbl.b_class c_class = gbl.c_class_2 d_class = gbl.d_class @@ -387,71 +387,75 @@ def test08_void_pointer_passing(self): """Test passing of variants of void pointer arguments""" - import _cppyy - pointer_pass = _cppyy.gbl.pointer_pass - some_concrete_class = _cppyy.gbl.some_concrete_class + import _cppyy as cppyy + pointer_pass = cppyy.gbl.pointer_pass + some_concrete_class = cppyy.gbl.some_concrete_class pp = pointer_pass() o = some_concrete_class() - assert _cppyy.addressof(o) == pp.gime_address_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ref(o) + assert cppyy.addressof(o) == pp.gime_address_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ref(o) import array - addressofo = array.array('l', [_cppyy.addressof(o)]) - assert addressofo.buffer_info()[0] == pp.gime_address_ptr_ptr(addressofo) + addressofo = array.array('l', [cppyy.addressof(o)]) + assert addressofo[0] == pp.gime_address_ptr_ptr(addressofo) assert 0 == pp.gime_address_ptr(0) - assert 0 == pp.gime_address_ptr(None) + raises(TypeError, pp.gime_address_ptr, None) - ptr = _cppyy.bind_object(0, some_concrete_class) - assert _cppyy.addressof(ptr) == 0 + ptr = cppyy.bind_object(0, some_concrete_class) + assert cppyy.addressof(ptr) == 0 pp.set_address_ptr_ref(ptr) - assert _cppyy.addressof(ptr) == 0x1234 + assert cppyy.addressof(ptr) == 0x1234 pp.set_address_ptr_ptr(ptr) - assert _cppyy.addressof(ptr) == 0x4321 + assert cppyy.addressof(ptr) == 0x4321 + + assert cppyy.addressof(cppyy.nullptr) == 0 + raises(TypeError, cppyy.addressof, None) + assert cppyy.addressof(0) == 0 def test09_opaque_pointer_passing(self): """Test passing around of opaque pointers""" - import _cppyy - some_concrete_class = _cppyy.gbl.some_concrete_class + import _cppyy as cppyy + some_concrete_class = cppyy.gbl.some_concrete_class o = some_concrete_class() # TODO: figure out the PyPy equivalent of CObject (may have to do this # through the C-API from C++) - #cobj = _cppyy.as_cobject(o) - addr = _cppyy.addressof(o) + #cobj = cppyy.as_cobject(o) + addr = cppyy.addressof(o) - #assert o == _cppyy.bind_object(cobj, some_concrete_class) - #assert o == _cppyy.bind_object(cobj, type(o)) - #assert o == _cppyy.bind_object(cobj, o.__class__) - #assert o == _cppyy.bind_object(cobj, "some_concrete_class") - assert _cppyy.addressof(o) == _cppyy.addressof(_cppyy.bind_object(addr, some_concrete_class)) - assert o == _cppyy.bind_object(addr, some_concrete_class) - assert o == _cppyy.bind_object(addr, type(o)) - assert o == _cppyy.bind_object(addr, o.__class__) - assert o == _cppyy.bind_object(addr, "some_concrete_class") - raises(TypeError, _cppyy.bind_object, addr, "does_not_exist") - raises(TypeError, _cppyy.bind_object, addr, 1) + #assert o == cppyy.bind_object(cobj, some_concrete_class) + #assert o == cppyy.bind_object(cobj, type(o)) + #assert o == cppyy.bind_object(cobj, o.__class__) + #assert o == cppyy.bind_object(cobj, "some_concrete_class") + assert cppyy.addressof(o) == cppyy.addressof(cppyy.bind_object(addr, some_concrete_class)) + assert o == cppyy.bind_object(addr, some_concrete_class) + assert o == cppyy.bind_object(addr, type(o)) + assert o == cppyy.bind_object(addr, o.__class__) + assert o == cppyy.bind_object(addr, "some_concrete_class") + raises(TypeError, cppyy.bind_object, addr, "does_not_exist") + raises(TypeError, cppyy.bind_object, addr, 1) def test10_object_identity(self): """Test object identity""" - import _cppyy - some_concrete_class = _cppyy.gbl.some_concrete_class - some_class_with_data = _cppyy.gbl.some_class_with_data + import _cppyy as cppyy + some_concrete_class = cppyy.gbl.some_concrete_class + some_class_with_data = cppyy.gbl.some_class_with_data o = some_concrete_class() - addr = _cppyy.addressof(o) + addr = cppyy.addressof(o) - o2 = _cppyy.bind_object(addr, some_concrete_class) + o2 = cppyy.bind_object(addr, some_concrete_class) assert o is o2 - o3 = _cppyy.bind_object(addr, some_class_with_data) + o3 = cppyy.bind_object(addr, some_class_with_data) assert not o is o3 d1 = some_class_with_data() @@ -472,13 +476,13 @@ def test11_multi_methods(self): """Test calling of methods from multiple inheritance""" - import _cppyy - multi = _cppyy.gbl.multi + import _cppyy as cppyy + multi = cppyy.gbl.multi - assert _cppyy.gbl.multi1 is multi.__bases__[0] - assert _cppyy.gbl.multi2 is multi.__bases__[1] From pypy.commits at gmail.com Sun Oct 29 09:49:37 2017 From: pypy.commits at gmail.com (rlamy) Date: Sun, 29 Oct 2017 06:49:37 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: Merged in nanjekye/pypy/sched_yield (pull request #574) Message-ID: <59f5dc71.2587df0a.3a662.a1da@mx.google.com> Author: Ronan Lamy Branch: py3.5 Changeset: r92874:4edfa3e729be Date: 2017-10-29 13:49 +0000 http://bitbucket.org/pypy/pypy/changeset/4edfa3e729be/ Log: Merged in nanjekye/pypy/sched_yield (pull request #574) sched_yield posix attribute diff --git a/pypy/module/posix/__init__.py b/pypy/module/posix/__init__.py --- a/pypy/module/posix/__init__.py +++ b/pypy/module/posix/__init__.py @@ -237,6 +237,9 @@ if getattr(rposix, _name) is not None: interpleveldefs[_name] = 'space.wrap(%d)' % getattr(rposix, _name) + if hasattr(rposix, 'sched_yield'): + interpleveldefs['sched_yield'] = 'interp_posix.sched_yield' + for _name in ["O_CLOEXEC"]: if getattr(rposix, _name) is not None: interpleveldefs[_name] = 'space.wrap(%d)' % getattr(rposix, _name) diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py --- a/pypy/module/posix/interp_posix.py +++ b/pypy/module/posix/interp_posix.py @@ -2468,3 +2468,13 @@ wrap_oserror(space, e, eintr_retry=True) else: return space.newint(s) + +def sched_yield(space): + """ Voluntarily relinquish the CPU""" + while True: + try: + res = rposix.sched_yield() + except OSError as e: + wrap_oserror(space, e, eintr_retry=True) + else: + return space.newint(res) diff --git a/pypy/module/posix/test/test_posix2.py b/pypy/module/posix/test/test_posix2.py --- a/pypy/module/posix/test/test_posix2.py +++ b/pypy/module/posix/test/test_posix2.py @@ -975,6 +975,12 @@ assert isinstance(high, int) == True assert high > low + if hasattr(rposix, 'sched_yield'): + def test_sched_yield(self): + os = self.posix + #Always suceeds on Linux + os.sched_yield() + def test_write_buffer(self): os = self.posix fd = os.open(self.path2 + 'test_write_buffer', diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -1851,6 +1851,8 @@ rffi.INT, save_err=rffi.RFFI_FULL_ERRNO_ZERO) c_sched_get_priority_min = external('sched_get_priority_min', [rffi.INT], rffi.INT, save_err=rffi.RFFI_SAVE_ERRNO) + if not _WIN32: + c_sched_yield = external('sched_yield', [], rffi.INT) @enforceargs(int) def sched_get_priority_max(policy): @@ -1860,9 +1862,9 @@ def sched_get_priority_min(policy): return handle_posix_error('sched_get_priority_min', c_sched_get_priority_min(policy)) - - - + def sched_yield(): + return handle_posix_error('sched_yield', c_sched_yield()) + #___________________________________________________________________ c_chroot = external('chroot', [rffi.CCHARP], rffi.INT, diff --git a/rpython/rlib/test/test_rposix.py b/rpython/rlib/test/test_rposix.py --- a/rpython/rlib/test/test_rposix.py +++ b/rpython/rlib/test/test_rposix.py @@ -810,4 +810,9 @@ assert isinstance(low, int) == True assert isinstance(high, int) == True assert high > low + + at rposix_requires('sched_yield') +def test_sched_yield(): + if sys.platform != 'win32': + rposix.sched_yield() From pypy.commits at gmail.com Sun Oct 29 09:49:43 2017 From: pypy.commits at gmail.com (nanjekye) Date: Sun, 29 Oct 2017 06:49:43 -0700 (PDT) Subject: [pypy-commit] pypy sched_yield: sched_yield posix attribute Message-ID: <59f5dc77.8a821c0a.ad7aa.5d90@mx.google.com> Author: Joannah Nanjekye Branch: sched_yield Changeset: r92873:f06ed6b90651 Date: 2017-10-27 19:57 +0300 http://bitbucket.org/pypy/pypy/changeset/f06ed6b90651/ Log: sched_yield posix attribute diff --git a/pypy/module/posix/__init__.py b/pypy/module/posix/__init__.py --- a/pypy/module/posix/__init__.py +++ b/pypy/module/posix/__init__.py @@ -237,6 +237,9 @@ if getattr(rposix, _name) is not None: interpleveldefs[_name] = 'space.wrap(%d)' % getattr(rposix, _name) + if hasattr(rposix, 'sched_yield'): + interpleveldefs['sched_yield'] = 'interp_posix.sched_yield' + for _name in ["O_CLOEXEC"]: if getattr(rposix, _name) is not None: interpleveldefs[_name] = 'space.wrap(%d)' % getattr(rposix, _name) diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py --- a/pypy/module/posix/interp_posix.py +++ b/pypy/module/posix/interp_posix.py @@ -2468,3 +2468,13 @@ wrap_oserror(space, e, eintr_retry=True) else: return space.newint(s) + +def sched_yield(space): + """ Voluntarily relinquish the CPU""" + while True: + try: + res = rposix.sched_yield() + except OSError as e: + wrap_oserror(space, e, eintr_retry=True) + else: + return space.newint(res) diff --git a/pypy/module/posix/test/test_posix2.py b/pypy/module/posix/test/test_posix2.py --- a/pypy/module/posix/test/test_posix2.py +++ b/pypy/module/posix/test/test_posix2.py @@ -975,6 +975,12 @@ assert isinstance(high, int) == True assert high > low + if hasattr(rposix, 'sched_yield'): + def test_sched_yield(self): + os = self.posix + #Always suceeds on Linux + os.sched_yield() + def test_write_buffer(self): os = self.posix fd = os.open(self.path2 + 'test_write_buffer', diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -1851,6 +1851,8 @@ rffi.INT, save_err=rffi.RFFI_FULL_ERRNO_ZERO) c_sched_get_priority_min = external('sched_get_priority_min', [rffi.INT], rffi.INT, save_err=rffi.RFFI_SAVE_ERRNO) + if not _WIN32: + c_sched_yield = external('sched_yield', [], rffi.INT) @enforceargs(int) def sched_get_priority_max(policy): @@ -1860,9 +1862,9 @@ def sched_get_priority_min(policy): return handle_posix_error('sched_get_priority_min', c_sched_get_priority_min(policy)) - - - + def sched_yield(): + return handle_posix_error('sched_yield', c_sched_yield()) + #___________________________________________________________________ c_chroot = external('chroot', [rffi.CCHARP], rffi.INT, diff --git a/rpython/rlib/test/test_rposix.py b/rpython/rlib/test/test_rposix.py --- a/rpython/rlib/test/test_rposix.py +++ b/rpython/rlib/test/test_rposix.py @@ -810,4 +810,9 @@ assert isinstance(low, int) == True assert isinstance(high, int) == True assert high > low + + at rposix_requires('sched_yield') +def test_sched_yield(): + if sys.platform != 'win32': + rposix.sched_yield() From pypy.commits at gmail.com Sun Oct 29 13:41:43 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 29 Oct 2017 10:41:43 -0700 (PDT) Subject: [pypy-commit] pypy py3.5-appexec: revert ec989ce318a9 and raise if space.appexec passed to space.is_true on runappdirect Message-ID: <59f612d7.ddafdf0a.7e53f.5435@mx.google.com> Author: Matti Picus Branch: py3.5-appexec Changeset: r92875:69ba67aff206 Date: 2017-10-29 19:40 +0200 http://bitbucket.org/pypy/pypy/changeset/69ba67aff206/ Log: revert ec989ce318a9 and raise if space.appexec passed to space.is_true on runappdirect diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -729,7 +729,12 @@ def setup_class(cls): if not hasattr(os, 'getpid'): pytest.skip("AF_NETLINK needs os.getpid()") - w_ok = space.appexec([], "(): import _socket; " + + + if cls.runappdirect: + import _socket + w_ok = hasattr(_socket, 'AF_NETLINK') + else: + w_ok = space.appexec([], "(): import _socket; " + "return hasattr(_socket, 'AF_NETLINK')") if not space.is_true(w_ok): pytest.skip("no AF_NETLINK on this platform") @@ -737,7 +742,8 @@ def test_connect_to_kernel_netlink_routing_socket(self): import _socket, os - s = _socket.socket(_socket.AF_NETLINK, _socket.SOCK_DGRAM, _socket.NETLINK_ROUTE) + s = _socket.socket(_socket.AF_NETLINK, _socket.SOCK_DGRAM, + _socket.NETLINK_ROUTE) assert s.getsockname() == (0, 0) s.bind((0, 0)) a, b = s.getsockname() @@ -749,7 +755,11 @@ def setup_class(cls): if not hasattr(os, 'getuid') or os.getuid() != 0: pytest.skip("AF_PACKET needs to be root for testing") - w_ok = space.appexec([], "(): import _socket; " + + if cls.runappdirect: + import _socket + w_ok = hasattr(_socket, 'AF_PACKET') + else: + w_ok = space.appexec([], "(): import _socket; " + "return hasattr(_socket, 'AF_PACKET')") if not space.is_true(w_ok): pytest.skip("no AF_PACKET on this platform") diff --git a/pypy/tool/pytest/apptest.py b/pypy/tool/pytest/apptest.py --- a/pypy/tool/pytest/apptest.py +++ b/pypy/tool/pytest/apptest.py @@ -239,7 +239,7 @@ space = gettestobjspace() filename = self._getdynfilename(target) func = app2interp_temp(src, filename=filename) - print "executing", func + # print "executing", func self.execute_appex(space, func, space) def repr_failure(self, excinfo): diff --git a/pypy/tool/pytest/objspace.py b/pypy/tool/pytest/objspace.py --- a/pypy/tool/pytest/objspace.py +++ b/pypy/tool/pytest/objspace.py @@ -79,9 +79,7 @@ body = body.lstrip() assert body.startswith('(') src = py.code.Source("def anonymous" + body) - d = {} - exec src.compile() in d - return d['anonymous'](*args) + return (src, args) def wrap(self, obj): if isinstance(obj, str): @@ -102,6 +100,8 @@ return list(itr) def is_true(self, obj): + if isinstance(obj, tuple) and isinstance(obj[0], py.code.Source): + raise ValueError('bool(appexec object) unknown') return bool(obj) def is_none(self, obj): From pypy.commits at gmail.com Sun Oct 29 15:06:48 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 29 Oct 2017 12:06:48 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge py3.5-appexec which will now raise on improper use of appexec in app tests Message-ID: <59f626c8.c4321c0a.2b71.8146@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92876:955b8000658b Date: 2017-10-29 21:01 +0200 http://bitbucket.org/pypy/pypy/changeset/955b8000658b/ Log: merge py3.5-appexec which will now raise on improper use of appexec in app tests diff --git a/pypy/module/_socket/test/test_sock_app.py b/pypy/module/_socket/test/test_sock_app.py --- a/pypy/module/_socket/test/test_sock_app.py +++ b/pypy/module/_socket/test/test_sock_app.py @@ -729,7 +729,12 @@ def setup_class(cls): if not hasattr(os, 'getpid'): pytest.skip("AF_NETLINK needs os.getpid()") - w_ok = space.appexec([], "(): import _socket; " + + + if cls.runappdirect: + import _socket + w_ok = hasattr(_socket, 'AF_NETLINK') + else: + w_ok = space.appexec([], "(): import _socket; " + "return hasattr(_socket, 'AF_NETLINK')") if not space.is_true(w_ok): pytest.skip("no AF_NETLINK on this platform") @@ -737,7 +742,8 @@ def test_connect_to_kernel_netlink_routing_socket(self): import _socket, os - s = _socket.socket(_socket.AF_NETLINK, _socket.SOCK_DGRAM, _socket.NETLINK_ROUTE) + s = _socket.socket(_socket.AF_NETLINK, _socket.SOCK_DGRAM, + _socket.NETLINK_ROUTE) assert s.getsockname() == (0, 0) s.bind((0, 0)) a, b = s.getsockname() @@ -749,7 +755,11 @@ def setup_class(cls): if not hasattr(os, 'getuid') or os.getuid() != 0: pytest.skip("AF_PACKET needs to be root for testing") - w_ok = space.appexec([], "(): import _socket; " + + if cls.runappdirect: + import _socket + w_ok = hasattr(_socket, 'AF_PACKET') + else: + w_ok = space.appexec([], "(): import _socket; " + "return hasattr(_socket, 'AF_PACKET')") if not space.is_true(w_ok): pytest.skip("no AF_PACKET on this platform") diff --git a/pypy/tool/pytest/apptest.py b/pypy/tool/pytest/apptest.py --- a/pypy/tool/pytest/apptest.py +++ b/pypy/tool/pytest/apptest.py @@ -239,7 +239,7 @@ space = gettestobjspace() filename = self._getdynfilename(target) func = app2interp_temp(src, filename=filename) - print "executing", func + # print "executing", func self.execute_appex(space, func, space) def repr_failure(self, excinfo): diff --git a/pypy/tool/pytest/objspace.py b/pypy/tool/pytest/objspace.py --- a/pypy/tool/pytest/objspace.py +++ b/pypy/tool/pytest/objspace.py @@ -100,6 +100,8 @@ return list(itr) def is_true(self, obj): + if isinstance(obj, tuple) and isinstance(obj[0], py.code.Source): + raise ValueError('bool(appexec object) unknown') return bool(obj) def is_none(self, obj): From pypy.commits at gmail.com Sun Oct 29 15:06:50 2017 From: pypy.commits at gmail.com (mattip) Date: Sun, 29 Oct 2017 12:06:50 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: document merged branches Message-ID: <59f626ca.06b6df0a.1db93.f530@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92877:41578430cbc4 Date: 2017-10-29 21:04 +0200 http://bitbucket.org/pypy/pypy/changeset/41578430cbc4/ Log: document merged branches diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -4,3 +4,10 @@ .. this is the revision after release-pypy3.5-5.9 .. startrev: be41e3ac0a29 + +.. branch: sched_yield +Add sched_yield posix attribute + +.. branch: py3.5-appexec +Raise if space.is_true(space.appexec()) used in app level tests, fix tests +that did this From pypy.commits at gmail.com Sun Oct 29 21:19:49 2017 From: pypy.commits at gmail.com (rlamy) Date: Sun, 29 Oct 2017 18:19:49 -0700 (PDT) Subject: [pypy-commit] pypy py3.6: hg merge py3.5 Message-ID: <59f67e35.8fa3df0a.8d0a6.a50a@mx.google.com> Author: Ronan Lamy Branch: py3.6 Changeset: r92878:40e3c2a486e9 Date: 2017-10-30 01:18 +0000 http://bitbucket.org/pypy/pypy/changeset/40e3c2a486e9/ Log: hg merge py3.5 diff too long, truncating to 2000 out of 11351 lines diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -75,6 +75,8 @@ ^lib_pypy/.+.c$ ^lib_pypy/.+.o$ ^lib_pypy/.+.so$ +^lib_pypy/.+.pyd$ +^lib_pypy/Release/ ^pypy/doc/discussion/.+\.html$ ^include/.+\.h$ ^include/.+\.inl$ diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -40,3 +40,7 @@ 2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1 c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0 a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +d72f9800a42b46a8056951b1da2426d2c2d8d502 release-pypy3.5-v5.9.0 +03d614975835870da65ff0481e1edad68ebbcb8d release-pypy2.7-v5.9.0 +84a2f3e6a7f88f2fe698e473998755b3bd1a12e2 release-pypy2.7-v5.9.0 diff --git a/lib-python/3/ctypes/test/test_bitfields.py b/lib-python/3/ctypes/test/test_bitfields.py --- a/lib-python/3/ctypes/test/test_bitfields.py +++ b/lib-python/3/ctypes/test/test_bitfields.py @@ -1,5 +1,5 @@ from ctypes import * -from ctypes.test import need_symbol +from ctypes.test import need_symbol, xfail import unittest import os @@ -278,6 +278,7 @@ self.assertEqual(b, b'\xef\xcd\xab\x21') @need_symbol('c_uint32') + @xfail def test_uint32_swap_big_endian(self): # Issue #23319 class Big(BigEndianStructure): diff --git a/lib-python/3/ctypes/test/test_byteswap.py b/lib-python/3/ctypes/test/test_byteswap.py --- a/lib-python/3/ctypes/test/test_byteswap.py +++ b/lib-python/3/ctypes/test/test_byteswap.py @@ -2,6 +2,7 @@ from binascii import hexlify from ctypes import * +from test.support import impl_detail def bin(s): return hexlify(memoryview(s)).decode().upper() @@ -22,6 +23,7 @@ setattr(bits, "i%s" % i, 1) dump(bits) + @impl_detail("slots are irrelevant on PyPy", pypy=False) def test_slots(self): class BigPoint(BigEndianStructure): __slots__ = () diff --git a/lib-python/3/ctypes/test/test_frombuffer.py b/lib-python/3/ctypes/test/test_frombuffer.py --- a/lib-python/3/ctypes/test/test_frombuffer.py +++ b/lib-python/3/ctypes/test/test_frombuffer.py @@ -85,7 +85,6 @@ del a gc.collect() # Should not crash - @xfail def test_from_buffer_copy(self): a = array.array("i", range(16)) x = (c_int * 16).from_buffer_copy(a) diff --git a/lib-python/3/ctypes/test/test_values.py b/lib-python/3/ctypes/test/test_values.py --- a/lib-python/3/ctypes/test/test_values.py +++ b/lib-python/3/ctypes/test/test_values.py @@ -4,6 +4,7 @@ import unittest import sys +from test.support import cpython_only from ctypes import * import _ctypes_test @@ -28,6 +29,7 @@ ctdll = CDLL(_ctypes_test.__file__) self.assertRaises(ValueError, c_int.in_dll, ctdll, "Undefined_Symbol") + at cpython_only class PythonValuesTestCase(unittest.TestCase): """This test only works when python itself is a dll/shared library""" diff --git a/lib-python/3/test/test_bytes.py b/lib-python/3/test/test_bytes.py --- a/lib-python/3/test/test_bytes.py +++ b/lib-python/3/test/test_bytes.py @@ -780,7 +780,6 @@ self.assertIs(type(BytesSubclass(A())), BytesSubclass) # Test PyBytes_FromFormat() - @test.support.impl_detail("don't test cpyext here") def test_from_format(self): ctypes = test.support.import_module('ctypes') _testcapi = test.support.import_module('_testcapi') diff --git a/lib-python/3/test/test_unicode.py b/lib-python/3/test/test_unicode.py --- a/lib-python/3/test/test_unicode.py +++ b/lib-python/3/test/test_unicode.py @@ -2436,6 +2436,10 @@ # Test PyUnicode_FromFormat() def test_from_format(self): support.import_module('ctypes') + try: + from ctypes import pythonapi + except ImportError: + self.skipTest( "no pythonapi in ctypes") from ctypes import ( pythonapi, py_object, sizeof, c_int, c_long, c_longlong, c_ssize_t, diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py --- a/lib_pypy/_ctypes/array.py +++ b/lib_pypy/_ctypes/array.py @@ -8,9 +8,14 @@ class ArrayMeta(_CDataMeta): def __new__(self, name, cls, typedict): res = type.__new__(self, name, cls, typedict) + if cls == (_CData,): # this is the Array class defined below + res._ffiarray = None return res - + if not hasattr(res, '_length_') or not isinstance(res._length_, int): + raise AttributeError( + "class must define a '_length_' attribute, " + "which must be a positive integer") ffiarray = res._ffiarray = _rawffi.Array(res._type_._ffishape_) subletter = getattr(res._type_, '_type_', None) if subletter == 'c': @@ -55,7 +60,7 @@ for i in range(len(val)): target[i] = val[i] if len(val) < self._length_: - target[len(val)] = '\x00' + target[len(val)] = u'\x00' res.value = property(getvalue, setvalue) res._ffishape_ = (ffiarray, res._length_) @@ -164,7 +169,7 @@ if letter == 'c': return b"".join(l) if letter == 'u': - return "".join(l) + return u"".join(l) return l class Array(_CData, metaclass=ArrayMeta): diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -165,6 +165,10 @@ def _get_buffer_value(self): return self._buffer[0] + def _copy_to(self, addr): + target = type(self).from_address(addr)._buffer + target[0] = self._get_buffer_value() + def _to_ffi_param(self): if self.__class__._is_pointer_like(): return self._get_buffer_value() diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py --- a/lib_pypy/_ctypes/pointer.py +++ b/lib_pypy/_ctypes/pointer.py @@ -113,7 +113,9 @@ cobj = self._type_.from_param(value) if ensure_objects(cobj) is not None: store_reference(self, index, cobj._objects) - self._subarray(index)[0] = cobj._get_buffer_value() + address = self._buffer[0] + address += index * sizeof(self._type_) + cobj._copy_to(address) def __bool__(self): return self._buffer[0] != 0 diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py --- a/lib_pypy/_ctypes/primitive.py +++ b/lib_pypy/_ctypes/primitive.py @@ -232,9 +232,6 @@ elif tp == 'u': def _setvalue(self, val): - if isinstance(val, bytes): - val = val.decode(ConvMode.encoding, ConvMode.errors) - # possible if we use 'ignore' if val: self._buffer[0] = val def _getvalue(self): @@ -243,8 +240,6 @@ elif tp == 'c': def _setvalue(self, val): - if isinstance(val, str): - val = val.encode(ConvMode.encoding, ConvMode.errors) if val: self._buffer[0] = val def _getvalue(self): diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -290,6 +290,11 @@ def _get_buffer_value(self): return self._buffer.buffer + def _copy_to(self, addr): + from ctypes import memmove + origin = self._get_buffer_value() + memmove(addr, origin, self._fficompositesize_) + def _to_ffi_param(self): return self._buffer diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py --- a/lib_pypy/_curses.py +++ b/lib_pypy/_curses.py @@ -404,7 +404,7 @@ return val def get_wch(self, *args): - wch = ffi.new("int[1]") + wch = ffi.new("wint_t[1]") if len(args) == 0: val = lib.wget_wch(self._win, wch) elif len(args) == 2: diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py --- a/lib_pypy/_sqlite3.py +++ b/lib_pypy/_sqlite3.py @@ -1080,21 +1080,25 @@ if '\0' in sql: raise ValueError("the query contains a null character") - first_word = sql.lstrip().split(" ")[0].upper() - if first_word == "": + + if sql: + first_word = sql.lstrip().split()[0].upper() + if first_word == '': + self._type = _STMT_TYPE_INVALID + if first_word == "SELECT": + self._type = _STMT_TYPE_SELECT + elif first_word == "INSERT": + self._type = _STMT_TYPE_INSERT + elif first_word == "UPDATE": + self._type = _STMT_TYPE_UPDATE + elif first_word == "DELETE": + self._type = _STMT_TYPE_DELETE + elif first_word == "REPLACE": + self._type = _STMT_TYPE_REPLACE + else: + self._type = _STMT_TYPE_OTHER + else: self._type = _STMT_TYPE_INVALID - elif first_word == "SELECT": - self._type = _STMT_TYPE_SELECT - elif first_word == "INSERT": - self._type = _STMT_TYPE_INSERT - elif first_word == "UPDATE": - self._type = _STMT_TYPE_UPDATE - elif first_word == "DELETE": - self._type = _STMT_TYPE_DELETE - elif first_word == "REPLACE": - self._type = _STMT_TYPE_REPLACE - else: - self._type = _STMT_TYPE_OTHER if isinstance(sql, unicode): sql = sql.encode('utf-8') diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO --- a/lib_pypy/cffi.egg-info/PKG-INFO +++ b/lib_pypy/cffi.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cffi -Version: 1.11.1 +Version: 1.11.2 Summary: Foreign Function Interface for Python calling C code. Home-page: http://cffi.readthedocs.org Author: Armin Rigo, Maciej Fijalkowski diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -4,8 +4,8 @@ from .api import FFI from .error import CDefError, FFIError, VerificationError, VerificationMissing -__version__ = "1.11.1" -__version_info__ = (1, 11, 1) +__version__ = "1.11.2" +__version_info__ = (1, 11, 2) # The verifier module file names are based on the CRC32 of a string that # contains the following version number. It may be older than __version__ diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h --- a/lib_pypy/cffi/_cffi_include.h +++ b/lib_pypy/cffi/_cffi_include.h @@ -238,9 +238,9 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x) { if (sizeof(_cffi_wchar_t) == 2) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else - return _cffi_from_c_wchar3216_t(x); + return _cffi_from_c_wchar3216_t((int)x); } _CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o) @@ -254,7 +254,7 @@ _CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x) { if (sizeof(_cffi_wchar_t) == 4) - return _cffi_from_c_wchar_t(x); + return _cffi_from_c_wchar_t((_cffi_wchar_t)x); else return _cffi_from_c_wchar3216_t(x); } diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h --- a/lib_pypy/cffi/_embedding.h +++ b/lib_pypy/cffi/_embedding.h @@ -247,7 +247,7 @@ if (f != NULL && f != Py_None) { PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME - "\ncompiled with cffi version: 1.11.1" + "\ncompiled with cffi version: 1.11.2" "\n_cffi_backend module: ", f); modules = PyImport_GetModuleDict(); mod = PyDict_GetItemString(modules, "_cffi_backend"); diff --git a/pypy/TODO b/pypy/TODO --- a/pypy/TODO +++ b/pypy/TODO @@ -1,18 +1,4 @@ -TODO for the python3 test suite: - -* test_memoryview - Needs bytes/str changes. Probably easy. Work for this has begun on - py3k-memoryview (by mjacob) https://bugs.pypy.org/issue1542 - -own-tests: - -* module/test_lib_pypy - These crash the buildbots (via SyntaxErrors): others were really - made to run under Python 2.x and so simply fail - -* module.cpyext.test.test_structseq test_StructSeq - structseq now subclasses tuple on py3, which breaks how - BaseCpyTypeDescr.realize allocates it +... antocuni's older TODO: @@ -20,14 +6,6 @@ * run coverage against the parser/astbuilder/astcompiler: it's probably full of dead code because the grammar changed -* re-enable strategies https://bugs.pypy.org/issue1540 : - - re-enable IntDictStrategy - - re-enable StdObjSpace.listview_str - - re-enable the kwargs dict strategy in dictmultiobject.py - - re-enable view_as_kwargs - -* unskip numpypy tests in module/test_lib_pypy/numpypy/ - * optimize W_UnicodeObject, right now it stores both an unicode and an utf8 version of the same string diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -119,7 +119,7 @@ To run untranslated tests, you need the Boehm garbage collector libgc. -On recent Debian and Ubuntu (like 17.04), this is the command to install +On recent Debian and Ubuntu (16.04 onwards), this is the command to install all build-time dependencies:: apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \ @@ -127,7 +127,7 @@ tk-dev libgc-dev python-cffi \ liblzma-dev libncursesw5-dev # these two only needed on PyPy3 -On older Debian and Ubuntu (12.04 to 16.04):: +On older Debian and Ubuntu (12.04-14.04):: apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ @@ -149,12 +149,23 @@ xz-devel # For lzma on PyPy3. (XXX plus the SLES11 version of libgdbm-dev and tk-dev) -On Mac OS X, most of these build-time dependencies are installed alongside +On Mac OS X:: + +Most of these build-time dependencies are installed alongside the Developer Tools. However, note that in order for the installation to find them you may need to run:: xcode-select --install +An exception is OpenSSL, which is no longer provided with the operating +system. It can be obtained via Homebrew (with ``$ brew install openssl``), +but it will not be available on the system path by default. The easiest +way to enable it for building pypy is to set an environment variable:: + + export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig + +After setting this, translation (described next) will find the OpenSSL libs +as expected. Run the translation ------------------- @@ -187,18 +198,18 @@ entire pypy interpreter. This step is currently singe threaded, and RAM hungry. As part of this step, the chain creates a large number of C code files and a Makefile to compile them in a - directory controlled by the ``PYPY_USESSION_DIR`` environment variable. + directory controlled by the ``PYPY_USESSION_DIR`` environment variable. 2. Create an executable ``pypy-c`` by running the Makefile. This step can - utilize all possible cores on the machine. -3. Copy the needed binaries to the current directory. -4. Generate c-extension modules for any cffi-based stdlib modules. + utilize all possible cores on the machine. +3. Copy the needed binaries to the current directory. +4. Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Python interpreter (see :doc:`cpython_differences`), and is ready for testing, for use as a base interpreter for a new virtualenv, or for packaging into a binary suitable for installation on another machine running the same OS as the build -machine. +machine. Note that step 4 is merely done as a convenience, any of the steps may be rerun without rerunning the previous steps. @@ -255,7 +266,7 @@ * PyPy 2.5.1 or earlier: normal users would see permission errors. Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in + commands at install time; the exact list is in :source:`pypy/tool/release/package.py `. Users seeing a broken installation of PyPy can fix it after-the-fact if they have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst --- a/pypy/doc/project-ideas.rst +++ b/pypy/doc/project-ideas.rst @@ -240,9 +240,12 @@ **matplotlib** https://github.com/matplotlib/matplotlib - TODO: the tkagg backend does not work, which makes tests fail on downstream - projects like Pandas, SciPy. It uses id(obj) as a c-pointer to obj in - tkagg.py, which requires refactoring + Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of + matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the + tkagg backend can function. + + TODO: the matplotlib branch passes numpy arrays by value (copying all the + data), this proof-of-concept needs help to become completely compliant **wxPython** https://bitbucket.org/amauryfa/wxpython-cffi diff --git a/pypy/doc/release-v5.9.0.rst b/pypy/doc/release-v5.9.0.rst --- a/pypy/doc/release-v5.9.0.rst +++ b/pypy/doc/release-v5.9.0.rst @@ -10,15 +10,18 @@ This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 includes the upstream stdlib version 3.5.3. -NumPy and Pandas now work on PyPy2.7. Issues that appeared as excessive memory +NumPy and Pandas now work on PyPy2.7 (together with Cython 0.27.1). Issues +that appeared as excessive memory use were cleared up and other incompatibilities were resolved. The C-API compatibility layer does slow down code which crosses the python-c interface often, we have ideas on how it could be improved, and still recommend using pure python on PyPy or interfacing via CFFI_. Many other modules based on C-API exentions now work on PyPy as well. -Cython 0.27 (released last week) should support more projects with PyPy, both -on PyPy2.7 and PyPy3.5 beta. +Cython 0.27.1 (released very recently) supports more projects with PyPy, both +on PyPy2.7 and PyPy3.5 beta. Note version **0.27.1** is now the minimum +version that supports this version of PyPy, due to some interactions with +updated C-API interface code. We optimized the JSON parser for recurring string keys, which should decrease memory use to 50% and increase parsing speed by up to 15% for large JSON files @@ -148,8 +151,7 @@ * Issue 2590_: fix the bounds in the GC when allocating a lot of objects with finalizers * Replace magical NOT RPYTHON comment with a decorator * Implement ``socket.sendmsg()``/``.recvmsg()`` for py3.5 - * Reduce excessive ``memory_pressure`` for ``_SSLContext`` objects and add - ``memory_pressure`` for ``_SSLSocket`` objects + * Add ``memory_pressure`` for ``_SSLSocket`` objects * Degredations diff --git a/pypy/doc/test/test_whatsnew.py b/pypy/doc/test/test_whatsnew.py --- a/pypy/doc/test/test_whatsnew.py +++ b/pypy/doc/test/test_whatsnew.py @@ -89,7 +89,7 @@ startrev, documented = parse_doc(last_whatsnew) merged, branch = get_merged_branches(ROOT, startrev, '') merged.discard('default') - merged.discard('py3k') + merged.discard('py3.5') merged.discard('') not_documented = merged.difference(documented) not_merged = documented.difference(merged) @@ -100,7 +100,7 @@ print '\n'.join(not_merged) print assert not not_documented - if branch == 'py3k': + if branch == 'py3.5': assert not not_merged else: assert branch in documented, 'Please document this branch before merging: %s' % branch diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -3,4 +3,10 @@ =========================== .. this is a revision shortly after release-pypy2.7-v5.9.0 -.. startrev:899e5245de1e +.. startrev:d56dadcef996 + +.. branch: cppyy-packaging +Cleanup and improve cppyy packaging + +.. branch: docs-osx-brew-openssl + diff --git a/pypy/doc/whatsnew-pypy2-5.9.0.rst b/pypy/doc/whatsnew-pypy2-5.9.0.rst --- a/pypy/doc/whatsnew-pypy2-5.9.0.rst +++ b/pypy/doc/whatsnew-pypy2-5.9.0.rst @@ -85,3 +85,12 @@ .. branch: py_ssize_t Explicitly use Py_ssize_t as the Signed type in pypy c-api + +.. branch: cpyext-jit + +Differentiate the code to call METH_NOARGS, METH_O and METH_VARARGS in cpyext: +this allows to write specialized code which is much faster than previous +completely generic version. Moreover, let the JIT to look inside the cpyext +module: the net result is that cpyext calls are up to 7x faster. However, this +is true only for very simple situations: in all real life code, we are still +much slower than CPython (more optimizations to come) diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -5,7 +5,9 @@ .. this is the revision after release-pypy3.5-5.9 .. startrev: be41e3ac0a29 -.. branch: multiphase +.. branch: sched_yield +Add sched_yield posix attribute -Implement PyType_FromSpec (PEP 384) and fix issues with PEP 489 support. - +.. branch: py3.5-appexec +Raise if space.is_true(space.appexec()) used in app level tests, fix tests +that did this diff --git a/pypy/interpreter/test/test_app_main.py b/pypy/interpreter/test/test_app_main.py --- a/pypy/interpreter/test/test_app_main.py +++ b/pypy/interpreter/test/test_app_main.py @@ -1162,9 +1162,13 @@ import os old_sys_path = sys.path[:] sys.path.append(self.goal_dir) + if sys.platform == 'win32': + exename = 'pypy3-c.exe' + else: + exename = 'pypy3-c' try: import app_main - pypy_c = os.path.join(self.trunkdir, 'pypy', 'goal', 'pypy3-c') + pypy_c = os.path.join(self.trunkdir, 'pypy', 'goal', exename) app_main.setup_bootstrap_path(pypy_c) newpath = sys.path[:] # we get at least lib_pypy @@ -1180,9 +1184,13 @@ import os old_sys_path = sys.path[:] sys.path.append(self.goal_dir) + if sys.platform == 'win32': + exename = 'pypy3-c.exe' + else: + exename = 'pypy3-c' try: import app_main - pypy_c = os.path.join(self.trunkdir, 'pypy', 'goal', 'pypy3-c') + pypy_c = os.path.join(self.trunkdir, 'pypy', 'goal', exename) app_main.entry_point(pypy_c, [self.foo_py]) # assert it did not crash finally: diff --git a/pypy/interpreter/test/test_interpreter.py b/pypy/interpreter/test/test_interpreter.py --- a/pypy/interpreter/test/test_interpreter.py +++ b/pypy/interpreter/test/test_interpreter.py @@ -1,7 +1,6 @@ import py import sys from pypy.interpreter import gateway, module, error -from hypothesis import given, strategies class TestInterpreter: @@ -300,30 +299,6 @@ assert "TypeError:" in res assert "'tuple' object is not a mapping" in res - @given(strategies.lists(strategies.one_of(strategies.none(), - strategies.lists(strategies.none())))) - def test_build_map_order(self, shape): - value = [10] - def build_expr(shape): - if shape is None: - value[0] += 1 - return '0: %d' % value[0] - else: - return '**{%s}' % (', '.join( - [build_expr(shape1) for shape1 in shape]),) - - expr = build_expr(shape)[2:] - code = """ - def f(): - return %s - """ % (expr, ) - res = self.codetest(code, 'f', []) - if value[0] == 10: - expected = {} - else: - expected = {0: value[0]} - assert res == expected, "got %r for %r" % (res, expr) - def test_build_map_unpack_with_call(self): code = """ def f(a,b,c,d): @@ -348,6 +323,36 @@ assert "TypeError:" in resg4 assert "got multiple values for keyword argument 'a'" in resg4 +try: + from hypothesis import given, strategies +except ImportError: + pass +else: + class TestHypothesisInterpreter(TestInterpreter): + @given(strategies.lists(strategies.one_of(strategies.none(), + strategies.lists(strategies.none())))) + def test_build_map_order(self, shape): + value = [10] + def build_expr(shape): + if shape is None: + value[0] += 1 + return '0: %d' % value[0] + else: + return '**{%s}' % (', '.join( + [build_expr(shape1) for shape1 in shape]),) + + expr = build_expr(shape)[2:] + code = """ + def f(): + return %s + """ % (expr, ) + res = self.codetest(code, 'f', []) + if value[0] == 10: + expected = {} + else: + expected = {0: value[0]} + assert res == expected, "got %r for %r" % (res, expr) + class AppTestInterpreter: def test_trivial(self): diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py --- a/pypy/module/__builtin__/test/test_descriptor.py +++ b/pypy/module/__builtin__/test/test_descriptor.py @@ -258,6 +258,17 @@ raises(TypeError, "super(D).__get__(12)") raises(TypeError, "super(D).__get__(C())") + def test_super_incomplete(self): + """ + class M(type): + def mro(cls): + if cls.__mro__ is None: + raises(AttributeError, lambda: super(cls, cls).xxx) + return type.mro(cls) + class A(metaclass=M): + pass + """ + def test_classmethods_various(self): class C(object): def foo(*a): return a diff --git a/pypy/module/_cffi_backend/__init__.py b/pypy/module/_cffi_backend/__init__.py --- a/pypy/module/_cffi_backend/__init__.py +++ b/pypy/module/_cffi_backend/__init__.py @@ -3,7 +3,7 @@ from rpython.rlib import rdynload, clibffi from rpython.rtyper.lltypesystem import rffi -VERSION = "1.11.1" +VERSION = "1.11.2" FFI_DEFAULT_ABI = clibffi.FFI_DEFAULT_ABI try: diff --git a/pypy/module/_cffi_backend/cffi1_module.py b/pypy/module/_cffi_backend/cffi1_module.py --- a/pypy/module/_cffi_backend/cffi1_module.py +++ b/pypy/module/_cffi_backend/cffi1_module.py @@ -1,4 +1,5 @@ from rpython.rtyper.lltypesystem import lltype, rffi +from rpython.rlib import jit from pypy.interpreter.error import oefmt from pypy.interpreter.module import Module @@ -15,7 +16,7 @@ INITFUNCPTR = lltype.Ptr(lltype.FuncType([rffi.VOIDPP], lltype.Void)) - + at jit.dont_look_inside def load_cffi1_module(space, name, path, initptr): # This is called from pypy.module.cpyext.api.load_extension_module() from pypy.module._cffi_backend.call_python import get_ll_cffi_call_python diff --git a/pypy/module/_cffi_backend/ctypeptr.py b/pypy/module/_cffi_backend/ctypeptr.py --- a/pypy/module/_cffi_backend/ctypeptr.py +++ b/pypy/module/_cffi_backend/ctypeptr.py @@ -158,10 +158,11 @@ class W_CTypePtrBase(W_CTypePtrOrArray): # base class for both pointers and pointers-to-functions - _attrs_ = ['is_void_ptr', 'is_voidchar_ptr'] - _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr'] + _attrs_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] + _immutable_fields_ = ['is_void_ptr', 'is_voidchar_ptr', 'is_onebyte_ptr'] is_void_ptr = False is_voidchar_ptr = False + is_onebyte_ptr = False def convert_to_object(self, cdata): ptrdata = rffi.cast(rffi.CCHARPP, cdata)[0] @@ -181,12 +182,20 @@ if self.is_void_ptr or other.is_void_ptr: pass # cast from or to 'void *' elif self.is_voidchar_ptr or other.is_voidchar_ptr: - space = self.space - msg = ("implicit cast from '%s' to '%s' " - "will be forbidden in the future (check that the types " - "are as you expect; use an explicit ffi.cast() if they " - "are correct)" % (other.name, self.name)) - space.warn(space.newtext(msg), space.w_UserWarning) + # for backward compatibility, accept "char *" as either + # source of target. This is not what C does, though, + # so emit a warning that will eventually turn into an + # error. The warning is turned off if both types are + # pointers to single bytes. + if self.is_onebyte_ptr and other.is_onebyte_ptr: + pass # no warning + else: + space = self.space + msg = ("implicit cast from '%s' to '%s' " + "will be forbidden in the future (check that the types " + "are as you expect; use an explicit ffi.cast() if they " + "are correct)" % (other.name, self.name)) + space.warn(space.newtext(msg), space.w_UserWarning) else: raise self._convert_error("compatible pointer", w_ob) @@ -216,6 +225,7 @@ self.is_void_ptr = isinstance(ctitem, ctypevoid.W_CTypeVoid) self.is_voidchar_ptr = (self.is_void_ptr or isinstance(ctitem, ctypeprim.W_CTypePrimitiveChar)) + self.is_onebyte_ptr = (ctitem.size == 1) W_CTypePtrBase.__init__(self, space, size, extra, 2, ctitem) def newp(self, w_init, allocator): diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py --- a/pypy/module/_cffi_backend/test/_backend_test_c.py +++ b/pypy/module/_cffi_backend/test/_backend_test_c.py @@ -1,7 +1,7 @@ # ____________________________________________________________ import sys -assert __version__ == "1.11.1", ("This test_c.py file is for testing a version" +assert __version__ == "1.11.2", ("This test_c.py file is for testing a version" " of cffi that differs from the one that we" " get from 'import _cffi_backend'") if sys.version_info < (3,): @@ -2099,7 +2099,8 @@ if sys.platform.startswith("linux"): BWChar = new_primitive_type("wchar_t") assert sizeof(BWChar) == 4 - assert int(cast(BWChar, -1)) == -1 # signed, on linux + # wchar_t is often signed on Linux, but not always (e.g. on ARM) + assert int(cast(BWChar, -1)) in (-1, 4294967295) def test_char16(): BChar16 = new_primitive_type("char16_t") @@ -3903,9 +3904,11 @@ BCharP = new_pointer_type(new_primitive_type("char")) BIntP = new_pointer_type(new_primitive_type("int")) BVoidP = new_pointer_type(new_void_type()) + BUCharP = new_pointer_type(new_primitive_type("unsigned char")) z1 = cast(BCharP, 0) z2 = cast(BIntP, 0) z3 = cast(BVoidP, 0) + z4 = cast(BUCharP, 0) with warnings.catch_warnings(record=True) as w: newp(new_pointer_type(BIntP), z1) # warn assert len(w) == 1 @@ -3919,6 +3922,12 @@ assert len(w) == 2 newp(new_pointer_type(BIntP), z3) # fine assert len(w) == 2 + newp(new_pointer_type(BCharP), z4) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z1) # fine (ignore signedness here) + assert len(w) == 2 + newp(new_pointer_type(BUCharP), z3) # fine + assert len(w) == 2 # check that the warnings are associated with lines in this file assert w[1].lineno == w[0].lineno + 4 diff --git a/pypy/module/_codecs/test/test_codecs.py b/pypy/module/_codecs/test/test_codecs.py --- a/pypy/module/_codecs/test/test_codecs.py +++ b/pypy/module/_codecs/test/test_codecs.py @@ -762,9 +762,16 @@ assert s == b'\xe9' def test_lone_surrogates(self): - for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', - 'utf-32', 'utf-32-le', 'utf-32-be'): + encodings = ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be', + 'utf-32', 'utf-32-le', 'utf-32-be') + for encoding in encodings: raises(UnicodeEncodeError, u'\ud800'.encode, encoding) + assert (u'[\udc80]'.encode(encoding, "backslashreplace") == + '[\\udc80]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "ignore") == + '[]'.encode(encoding)) + assert (u'[\udc80]'.encode(encoding, "replace") == + '[?]'.encode(encoding)) def test_charmap_encode(self): assert 'xxx'.encode('charmap') == b'xxx' diff --git a/pypy/module/_continuation/test/test_stacklet.py b/pypy/module/_continuation/test/test_stacklet.py --- a/pypy/module/_continuation/test/test_stacklet.py +++ b/pypy/module/_continuation/test/test_stacklet.py @@ -290,66 +290,87 @@ def test_random_switching(self): from _continuation import continulet # + seen = [] + # def t1(c1): - return c1.switch() + seen.append(3) + res = c1.switch() + seen.append(6) + return res + # def s1(c1, n): + seen.append(2) assert n == 123 c2 = t1(c1) - return c1.switch('a') + 1 + seen.append(7) + res = c1.switch('a') + 1 + seen.append(10) + return res # def s2(c2, c1): + seen.append(5) res = c1.switch(c2) + seen.append(8) assert res == 'a' - return c2.switch('b') + 2 + res = c2.switch('b') + 2 + seen.append(12) + return res # def f(): + seen.append(1) c1 = continulet(s1, 123) c2 = continulet(s2, c1) c1.switch() + seen.append(4) res = c2.switch() + seen.append(9) assert res == 'b' res = c1.switch(1000) + seen.append(11) assert res == 1001 - return c2.switch(2000) + res = c2.switch(2000) + seen.append(13) + return res # res = f() assert res == 2002 + assert seen == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] def test_f_back(self): import sys from _continuation import continulet # - def g(c): + def bar(c): c.switch(sys._getframe(0)) c.switch(sys._getframe(0).f_back) c.switch(sys._getframe(1)) c.switch(sys._getframe(1).f_back) - assert sys._getframe(2) is f3.f_back + assert sys._getframe(2) is f3_foo.f_back c.switch(sys._getframe(2)) - def f(c): - g(c) + def foo(c): + bar(c) # - c = continulet(f) - f1 = c.switch() - assert f1.f_code.co_name == 'g' - f2 = c.switch() - assert f2.f_code.co_name == 'f' - f3 = c.switch() - assert f3 is f2 - assert f1.f_back is f3 + c = continulet(foo) + f1_bar = c.switch() + assert f1_bar.f_code.co_name == 'bar' + f2_foo = c.switch() + assert f2_foo.f_code.co_name == 'foo' + f3_foo = c.switch() + assert f3_foo is f2_foo + assert f1_bar.f_back is f3_foo def main(): - f4 = c.switch() - assert f4.f_code.co_name == 'main', repr(f4.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f4_main = c.switch() + assert f4_main.f_code.co_name == 'main' + assert f3_foo.f_back is f1_bar # not running, so a loop def main2(): - f5 = c.switch() - assert f5.f_code.co_name == 'main2', repr(f5.f_code.co_name) - assert f3.f_back is f1 # not running, so a loop + f5_main2 = c.switch() + assert f5_main2.f_code.co_name == 'main2' + assert f3_foo.f_back is f1_bar # not running, so a loop main() main2() res = c.switch() assert res is None - assert f3.f_back is None + assert f3_foo.f_back is None def test_traceback_is_complete(self): import sys diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -1,12 +1,10 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - "This module provides runtime bindings to C++ code for which reflection\n\ - info has been generated. Current supported back-ends are Reflex and CINT.\n\ - See http://doc.pypy.org/en/latest/cppyy.html for full details." + "This module brigdes the cppyy frontend with its backend, through PyPy.\n\ + See http://cppyy.readthedocs.io/en/latest for full details." interpleveldefs = { - '_load_dictionary' : 'interp_cppyy.load_dictionary', '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', '_template_byname' : 'interp_cppyy.template_byname', @@ -15,14 +13,13 @@ '_set_function_generator': 'interp_cppyy.set_function_generator', '_register_class' : 'interp_cppyy.register_class', '_get_nullptr' : 'interp_cppyy.get_nullptr', - 'CPPInstanceBase' : 'interp_cppyy.W_CPPInstance', + 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', 'bind_object' : 'interp_cppyy.bind_object', } appleveldefs = { '_init_pythonify' : 'pythonify._init_pythonify', - 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', 'Template' : 'pythonify.CPPTemplate', } diff --git a/pypy/module/_cppyy/backend/create_cppyy_package.py b/pypy/module/_cppyy/backend/create_cppyy_package.py deleted file mode 100755 --- a/pypy/module/_cppyy/backend/create_cppyy_package.py +++ /dev/null @@ -1,649 +0,0 @@ -#!/usr/bin/env python -from __future__ import print_function - -import os, sys -import argparse, re, shutil, tarfile, urllib2 - - -DEBUG_TESTBUILD = False - -TARBALL_CACHE_DIR = 'releases' - -ROOT_KEEP = ['build', 'cmake', 'config', 'core', 'etc', 'interpreter', - 'io', 'LICENSE', 'net', 'Makefile', 'CMakeLists.txt', 'math', - 'main'] # main only needed in more recent root b/c of rootcling -ROOT_CORE_KEEP = ['CMakeLists.txt', 'base', 'clib', 'clingutils', 'cont', - 'dictgen', 'foundation', 'lzma', 'macosx', 'meta', - 'metacling', 'metautils', 'rootcling_stage1', 'textinput', - 'thread', 'unix', 'utils', 'winnt', 'zip'] -ROOT_IO_KEEP = ['CMakeLists.txt', 'io', 'rootpcm'] -ROOT_NET_KEEP = ['CMakeLists.txt', 'net'] -ROOT_MATH_KEEP = ['CMakeLists.txt', 'mathcore'] -ROOT_ETC_KEEP = ['Makefile.arch', 'class.rules', 'cmake', 'dictpch', - 'gdb-backtrace.sh', 'gitinfo.txt', 'helgrind-root.supp', - 'hostcert.conf', 'system.plugins-ios', - 'valgrind-root-python.supp', 'valgrind-root.supp', 'vmc'] - -ROOT_EXPLICIT_REMOVE = ['core/base/v7', 'math/mathcore/v7', 'io/io/v7'] - - -ERR_RELEASE_NOT_FOUND = 2 - - -# -## CLI arguments -# -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - - -# -## ROOT source pull and cleansing -# -def clean_directory(directory, keeplist, trim_cmake=True): - removed_entries = [] - for entry in os.listdir(directory): - if entry[0] == '.' or entry in keeplist: - continue - removed_entries.append(entry) - entry = os.path.join(directory, entry) - print('now removing', entry) - if os.path.isdir(entry): - shutil.rmtree(entry) - else: - os.remove(entry) - - if not trim_cmake: - return - - # now take the removed entries out of the CMakeLists.txt - if removed_entries: - inp = os.path.join(directory, 'CMakeLists.txt') - print('trimming', inp) - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if ('add_subdirectory' in line) or\ - ('COMMAND' in line and 'copy' in line) or\ - ('ROOT_ADD_TEST_SUBDIRECTORY' in line) or\ - ('install(DIRECTORY' in line): - for sub in removed_entries: - if sub in line: - line = '#'+line - break - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - else: - print('reusing existing %s/CMakeLists.txt' % (directory,)) - - -class ReleaseValidation(argparse.Action): - def __call__(self, parser, namespace, value, option_string=None): - if not re.match(r'6\.\d\d\.\d\d', value): - raise argparse.ArgumentTypeError( - "release number should of the form '6.dd.dd'") - setattr(namespace, self.dest, value) - return value - -parser = argparse.ArgumentParser( - description='Build PyPi package for cppyy containing the minimum of ROOT') -parser.add_argument('-r', '--release', type=str, nargs='?', - action=ReleaseValidation, help='ROOT release to use') - -args = parser.parse_args() - -if not os.path.exists(TARBALL_CACHE_DIR): - os.mkdir(TARBALL_CACHE_DIR) - -if args.release: - # use provided release - fn = 'root_v%s.source.tar.gz' % args.release - addr = 'https://root.cern.ch/download/'+fn - if not os.path.exists(os.path.join(TARBALL_CACHE_DIR, fn)): - try: - print('retrieving', fn) - resp = urllib2.urlopen(addr, fn) - out = open(os.path.join(TARBALL_CACHE_DIR, fn), 'wb') - out.write(resp.read()) - out.close() - except urllib2.HTTPError: - print('release %s not found' % args.release) - sys.exit(ERR_RELEASE_NOT_FOUND) - else: - print('reusing', fn, 'from local directory') -else: - print('provide release ... getting latest release is not yet implemented ...') - sys.exit(1) - # get latest and set fn, args.release, etc. - -# construct version for package -args.version = '' -testnext = False -for c in args.release: - if testnext: - testnext = False - if c == '0': - continue - if c == '.': - testnext = True - args.version += c -args.version += '.0' - -fn = os.path.join(TARBALL_CACHE_DIR, fn) -pkgdir = os.path.join('root-'+args.release) -if not os.path.exists(pkgdir): - print('now extracting', args.release) - tf = tarfile.TarFile.gzopen(fn) - tf.extractall() - tf.close() -else: - print('reusing existing directory', pkgdir) - -# remove everything except for the listed set of libraries -os.chdir(pkgdir) - -clean_directory(os.path.curdir, ROOT_KEEP) -clean_directory('core', ROOT_CORE_KEEP) -clean_directory('etc', ROOT_ETC_KEEP, trim_cmake=False) -clean_directory('io', ROOT_IO_KEEP) -clean_directory('math', ROOT_MATH_KEEP) -clean_directory('net', ROOT_NET_KEEP) - - -# trim main (only need rootcling) -print('trimming main') -for entry in os.listdir('main/src'): - if entry != 'rootcling.cxx': - os.remove('main/src/'+entry) -inp = 'main/CMakeLists.txt' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if ('ROOT_EXECUTABLE' in line or\ - 'SET_TARGET_PROPERTIES' in line) and\ - not 'rootcling' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove afterimage and ftgl explicitly -print('trimming externals') -for cmf in ['AfterImage', 'FTGL']: - os.remove('cmake/modules/Find%s.cmake' % (cmf,)) -inp = 'cmake/modules/SearchInstalledSoftware.cmake' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Check for ftgl if needed' == line[0:28] or\ - '#---Check for AfterImage' == line[0:24]: - now_stripping = True - elif '#---Check' == line[0:9]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'cmake/modules/RootBuildOptions.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'ROOT_BUILD_OPTION(builtin_ftgl' in line or\ - 'ROOT_BUILD_OPTION(builtin_afterimage' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - - -# remove testing and examples -print('trimming testing') -inp = 'CMakeLists.txt' -outp = inp+'.new' -now_stripping = False -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if '#---Configure Testing using CTest' == line[0:33] or\ - '#---hsimple.root' == line[0:16]: - now_stripping = True - elif '#---Packaging' == line[0:13] or\ - '#---version' == line[0:11]: - now_stripping = False - if now_stripping: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -print('trimming RootCPack') -inp = 'cmake/modules/RootCPack.cmake' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if 'README.txt' in line: - line = '#'+line - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# some more explicit removes: -for dir_to_remove in ROOT_EXPLICIT_REMOVE: - try: - shutil.rmtree(dir_to_remove) - except OSError: - pass - -# special fixes -inp = 'core/base/src/TVirtualPad.cxx' -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp): - if '#include "X3DBuffer.h"' == line[0:22]: - line = """//#include "X3DBuffer.h" -typedef struct _x3d_sizeof_ { - int numPoints; - int numSegs; - int numPolys; -} Size3D; -""" - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -inp = 'math/mathcore/src/Fitter.cxx' -if os.path.exists(inp): - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp): - if '#include "TF1.h"' in line: - continue - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - -# done -os.chdir(os.path.pardir) - -# debugging: run a test build -if DEBUG_TESTBUILD: - print('running a debug test build') - tb = "test_builddir" - if os.path.exists(tb): - shutil.rmtree(tb) - os.mkdir(tb) - os.chdir(tb) - os.system('cmake ../%s -DCMAKE_INSTALL_PREFIX=../install -Dminimal=ON -Dasimage=OFF' % pkgdir) - os.system('make -j 32') - - -# -## package creation -# -countdown = 0 -pidir = 'Package-'+args.release -print('creating package', pidir) -if not os.path.exists(pidir): - os.mkdir(pidir) -os.chdir(pidir); countdown += 1 - -print('creating LICENSE.txt') -with open('LICENSE.txt', 'w') as outp: - outp.write("""There are three main parts: - - LLVM: distributed under University of Illinois/NCSA Open Source License - https://opensource.org/licenses/UoI-NCSA.php - ROOT: distributed under LGPL 2.1 - https://root.cern.ch/license - Cppyy: distributed under LBNL BSD - https://fedoraproject.org/wiki/Licensing/LBNLBSD -""") - -print('creating MANIFEST.in') -with open('MANIFEST.in', 'w') as outp: - outp.write("""# Include the license file -include LICENSE.txt - -# Include the data files -recursive-include src * -""") - -print('creating README.rst') -with open('README.rst', 'w') as outp: - outp.write("""PyPy cling-support -================== - ----- - -Find the documentation here: - http://doc.pypy.org/en/latest/cppyy.html -""") - -print('creating setup.cfg') -with open('setup.cfg', 'w') as outp: - outp.write("""[bdist_wheel] -universal=0 -""") - -print('creating setup.py') -with open('setup.py', 'w') as outp: - outp.write("""import os, sys, subprocess -from setuptools import setup, find_packages -from distutils import log -from distutils.command.build import build as _build -from setuptools.command.install import install as _install -from distutils.sysconfig import get_python_lib -from distutils.errors import DistutilsSetupError -from codecs import open - -here = os.path.abspath(os.path.dirname(__file__)) -with open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: - long_description = f.read() - -builddir = None -def get_builddir(): - global builddir - if builddir is None: - topdir = os.getcwd() - builddir = os.path.join(topdir, 'builddir') - return builddir - -srcdir = None -def get_srcdir(): - global srcdir - if srcdir is None: - topdir = os.getcwd() - srcdir = os.path.join(topdir, 'src', 'backend') - return srcdir - -class my_cmake_build(_build): - def __init__(self, dist, *args, **kwargs): - _build.__init__(self, dist, *args, **kwargs) - # TODO: can't seem to find a better way of getting hold of - # the install_lib parameter during the build phase ... - prefix = '' - try: - prefix = dist.get_command_obj('install').install_lib - except AttributeError: - pass - if not prefix: - prefix = get_python_lib(1, 0) - self.prefix = os.path.join(prefix, 'cppyy_backend') - - def run(self): - # base run - _build.run(self) - - # custom run - log.info('Now building libcppyy_backend.so and dependencies') - builddir = get_builddir() - srcdir = get_srcdir() - if not os.path.exists(builddir): - log.info('Creating build directory %s ...' % builddir) - os.makedirs(builddir) - - os.chdir(builddir) - log.info('Running cmake for cppyy_backend') - if subprocess.call([ - 'cmake', srcdir, '-Dminimal=ON -Dasimage=OFF', - '-DCMAKE_INSTALL_PREFIX='+self.prefix]) != 0: - raise DistutilsSetupError('Failed to configure cppyy_backend') - - nprocs = os.getenv("MAKE_NPROCS") - if nprocs: - try: - ival = int(nprocs) - nprocs = '-j'+nprocs - except ValueError: - log.warn("Integer expected for MAKE_NPROCS, but got %s (ignored)", nprocs) - nprocs = '-j1' - else: - nprocs = '-j1' - log.info('Now building cppyy_backend and dependencies ...') - if subprocess.call(['make', nprocs]) != 0: - raise DistutilsSetupError('Failed to build cppyy_backend') - - log.info('build finished') - -class my_libs_install(_install): - def run(self): - # base install - _install.run(self) - - # custom install - log.info('Now installing libcppyy_backend.so and dependencies') - builddir = get_builddir() - if not os.path.exists(builddir): - raise DistutilsSetupError('Failed to find build dir!') - os.chdir(builddir) - - prefix = self.install_lib - log.info('Now installing in %s ...', prefix) - if subprocess.call(['make', 'install']) != 0: - raise DistutilsSetupError('Failed to install cppyy_backend') - - log.info('install finished') - - def get_outputs(self): - outputs = _install.get_outputs(self) - outputs.append(os.path.join(self.install_lib, 'cppyy_backend')) - return outputs - -setup( - name='PyPy-cppyy-backend', -""") - outp.write(" version='%s', # corresponds to ROOT %s, extra number is for packager\n"\ - % (args.version, args.release)) - outp.write(""" description='Cling support for PyPy', - long_description=long_description, - - url='http://pypy.org', - - # Author details - author='PyPy Developers', - author_email='pypy-dev at python.org', - - license='LLVM: UoI-NCSA; ROOT: LGPL 2.1; Cppyy: LBNL BSD', - - classifiers=[ - 'Development Status :: 4 - Beta', - - 'Intended Audience :: Developers', - - 'Topic :: Software Development', - 'Topic :: Software Development :: Interpreters', - - #'License :: OSI Approved :: MIT License', - - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: Implementation :: PyPy', - 'Programming Language :: C', - 'Programming Language :: C++', - - 'Natural Language :: English' - ], - - keywords='interpreter development', - - packages=find_packages('src', ['backend']), - include_package_data=True, - - extras_require={ - }, - - cmdclass = { - 'build': my_cmake_build, - 'install': my_libs_install, - }, -) -""") - - -print('creating src ... ROOT part') -if not os.path.exists('src'): - os.mkdir('src') -os.chdir('src'); countdown += 1 -if not os.path.exists('backend'): - src = os.path.join(os.path.pardir, os.path.pardir, pkgdir) - print('now copying', src) - shutil.copytree(src, 'backend') - -print('creating src ... cppyy part') -os.chdir('backend'); countdown += 1 -if not os.path.exists('cppyy'): - os.mkdir('cppyy') - os.chdir('cppyy'); countdown += 1 - - with open('CMakeLists.txt', 'w') as outp: - outp.write("""############################################################################ -# CMakeLists.txt file for building cppyy package -############################################################################ - -ROOT_GLOB_SOURCES(sources ${CMAKE_CURRENT_SOURCE_DIR}/src/*.cxx) -set_source_files_properties(${sources} COMPILE_FLAGS "-fomit-frame-pointer -fvisibility=hidden -DRPY_EXTERN=RPY_EXPORTED -DRPYTHON_LL2CTYPES") - -add_definitions(${CLING_CXXFLAGS}) - -ROOT_LINKER_LIBRARY(cppyy_backend ${sources} - LIBRARIES ${CMAKE_DL_LIBS} - DEPENDENCIES Core Cling RIO Thread) - -add_dependencies(cppyy_backend CLING) -""") - - os.mkdir('src') - os.chdir('src'); countdown += 1 - print('pulling cppyy/clingcwrapper.cxx from pypy') - base = 'https://bitbucket.org/pypy/pypy/raw/default/pypy/module/cppyy/' - for cppyy_file in ['src/callcontext.h', 'include/capi.h', 'src/clingcwrapper.cxx', - 'include/clingcwrapper.h', 'include/cpp_cppyy.h', 'include/cppyy.h']: - resp = urllib2.urlopen(base+cppyy_file) - with open(os.path.basename(cppyy_file), 'w') as outp: - outp.write(resp.read()) - - # fix include - inp = 'capi.h' - outp = inp+'.new' - new_cml = open(outp, 'w') - for line in open(inp).readlines(): - if 'src/precommondefs.h' in line: - line = '#include "precommondefs.h"\n' - new_cml.write(line) - new_cml.close() - os.rename(outp, inp) - - with open('precommondefs.h', 'w') as outp: - outp.write("""/***** Start of precommondefs.h *****/ - -/* This is extracted from pyconfig.h from CPython. It sets the macros - that affect the features we get from system include files. - It must not #include anything. */ - -#ifndef __PYPY_PRECOMMONDEFS_H -#define __PYPY_PRECOMMONDEFS_H - - -/* Define on Darwin to activate all library features */ -#define _DARWIN_C_SOURCE 1 -/* This must be set to 64 on some systems to enable large file support. */ -#define _FILE_OFFSET_BITS 64 -/* Define on Linux to activate all library features */ -#define _GNU_SOURCE 1 -/* This must be defined on some systems to enable large file support. */ -#define _LARGEFILE_SOURCE 1 -/* Define on NetBSD to activate all library features */ -#define _NETBSD_SOURCE 1 -/* Define to activate features from IEEE Stds 1003.1-2001 */ -#ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L -#endif -/* Define on FreeBSD to activate all library features */ -#define __BSD_VISIBLE 1 -#define __XSI_VISIBLE 700 -/* Windows: winsock/winsock2 mess */ -#define WIN32_LEAN_AND_MEAN -#ifdef _WIN64 - typedef __int64 Signed; - typedef unsigned __int64 Unsigned; -# define SIGNED_MIN LLONG_MIN -#else - typedef long Signed; - typedef unsigned long Unsigned; -# define SIGNED_MIN LONG_MIN -#endif - -#if !defined(RPY_ASSERT) && !defined(RPY_LL_ASSERT) && !defined(NDEBUG) -# define NDEBUG -#endif - - -/* All functions and global variables declared anywhere should use - one of the following attributes: - - RPY_EXPORTED: the symbol is exported out of libpypy-c.so. - - RPY_EXTERN: the symbol is not exported out of libpypy-c.so, but - otherwise works like 'extern' by being available to - other C sources. - - static: as usual, this means the symbol is local to this C file. - - Don't use _RPY_HIDDEN directly. For tests involving building a custom - .so, translator/tool/cbuild.py overrides RPY_EXTERN so that it becomes - equal to RPY_EXPORTED. - - Any function or global variable declared with no attribute at all is - a bug; please report or fix it. -*/ -#ifdef __GNUC__ -# define RPY_EXPORTED extern __attribute__((visibility("default"))) -# define _RPY_HIDDEN __attribute__((visibility("hidden"))) -#else -# define RPY_EXPORTED extern __declspec(dllexport) -# define _RPY_HIDDEN /* nothing */ -#endif -#ifndef RPY_EXTERN -# define RPY_EXTERN extern _RPY_HIDDEN -#endif - - -#endif /* __PYPY_PRECOMMONDEFS_H */ - -/***** End of precommondefs.h *****/ -""") - -# back up to pip package top -for i in range(countdown-1): - os.chdir(os.path.pardir) - -# add cppyy module to cmake -os.chdir('src/backend') -inp = 'CMakeLists.txt' -print('adding cppyy to cmake') -outp = inp+'.new' -new_cml = open(outp, 'w') -for line in open(inp).readlines(): - if 'add_subdirectory' in line and 'net' in line: - line += 'add_subdirectory (cppyy)\n' - new_cml.write(line) -new_cml.close() -os.rename(outp, inp) - -# done! diff --git a/pypy/module/_cppyy/bench/Makefile b/pypy/module/_cppyy/bench/Makefile deleted file mode 100644 --- a/pypy/module/_cppyy/bench/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -all: bench02Dict_reflex.so - -ROOTSYS := ${ROOTSYS} - -ifeq ($(ROOTSYS),) - genreflex=genreflex - cppflags= -else - genreflex=$(ROOTSYS)/bin/genreflex - cppflags=-I$(ROOTSYS)/include -L$(ROOTSYS)/lib -endif - -PLATFORM := $(shell uname -s) -ifeq ($(PLATFORM),Darwin) - cppflags+=-dynamiclib -single_module -arch x86_64 -endif - -ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),) - genreflexflags= - cppflags2=-O3 -fPIC -else - genreflexflags=--with-methptrgetter - cppflags2=-Wno-pmf-conversions -O3 -fPIC -endif - - -bench02Dict_reflex.so: bench02.h bench02.cxx bench02.xml - $(genreflex) bench02.h $(genreflexflags) --selection=bench02.xml -I$(ROOTSYS)/include - g++ -o $@ bench02.cxx bench02_rflx.cpp -I$(ROOTSYS)/include -shared -std=c++11 -lHistPainter `root-config --libs` $(cppflags) $(cppflags2) diff --git a/pypy/module/_cppyy/bench/bench02.cxx b/pypy/module/_cppyy/bench/bench02.cxx deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.cxx +++ /dev/null @@ -1,79 +0,0 @@ -#include "bench02.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TDirectory.h" -#include "TInterpreter.h" -#include "TSystem.h" -#include "TBenchmark.h" -#include "TStyle.h" -#include "TError.h" -#include "Getline.h" -#include "TVirtualX.h" - -#include "Api.h" - -#include - -TClass *TClass::GetClass(const char*, Bool_t, Bool_t) { - static TClass* dummy = new TClass("__dummy__", kTRUE); - return dummy; // is deleted by gROOT at shutdown -} - -class TTestApplication : public TApplication { -public: - TTestApplication( - const char* acn, Int_t* argc, char** argv, Bool_t bLoadLibs = kTRUE); - virtual ~TTestApplication(); -}; - -TTestApplication::TTestApplication( - const char* acn, int* argc, char** argv, bool do_load) : TApplication(acn, argc, argv) { - if (do_load) { - // follow TRint to minimize differences with CINT - ProcessLine("#include ", kTRUE); - ProcessLine("#include <_string>", kTRUE); // for std::string iostream. - ProcessLine("#include ", kTRUE); // needed because they're used within the - ProcessLine("#include ", kTRUE); // core ROOT dicts and CINT won't be able - // to properly unload these files - } - - // save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // prevent crashes on accessing history - Gl_histinit((char*)"-"); - - // prevent ROOT from exiting python - SetReturnFromRun(kTRUE); -} - -TTestApplication::~TTestApplication() {} - -static const char* appname = "pypy-cppyy"; - -Bench02RootApp::Bench02RootApp() { - gROOT->SetBatch(kTRUE); - if (!gApplication) { - int argc = 1; - char* argv[1]; argv[0] = (char*)appname; - gApplication = new TTestApplication(appname, &argc, argv, kFALSE); - } -} - -Bench02RootApp::~Bench02RootApp() { - // TODO: ROOT globals cleanup ... (?) -} - -void Bench02RootApp::report() { - std::cout << "gROOT is: " << gROOT << std::endl; - std::cout << "gApplication is: " << gApplication << std::endl; -} - -void Bench02RootApp::close_file(TFile* f) { - std::cout << "closing file " << f->GetName() << " ... " << std::endl; - f->Write(); - f->Close(); - std::cout << "... file closed" << std::endl; -} diff --git a/pypy/module/_cppyy/bench/bench02.h b/pypy/module/_cppyy/bench/bench02.h deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.h +++ /dev/null @@ -1,72 +0,0 @@ -#include "TString.h" - -#include "TCanvas.h" -#include "TFile.h" -#include "TProfile.h" -#include "TNtuple.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TRandom.h" -#include "TRandom3.h" - -#include "TROOT.h" -#include "TApplication.h" -#include "TSystem.h" - -#include "TArchiveFile.h" -#include "TBasket.h" -#include "TBenchmark.h" -#include "TBox.h" -#include "TBranchRef.h" -#include "TBrowser.h" -#include "TClassGenerator.h" -#include "TClassRef.h" -#include "TClassStreamer.h" -#include "TContextMenu.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TF1.h" -#include "TFileCacheRead.h" -#include "TFileCacheWrite.h" -#include "TFileMergeInfo.h" -#include "TFitResult.h" -#include "TFolder.h" -//#include "TFormulaPrimitive.h" -#include "TFunction.h" -#include "TFrame.h" -#include "TGlobal.h" -#include "THashList.h" -#include "TInetAddress.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TLegend.h" -#include "TMethodCall.h" -#include "TPluginManager.h" -#include "TProcessUUID.h" -#include "TSchemaRuleSet.h" -#include "TStyle.h" -#include "TSysEvtHandler.h" -#include "TTimer.h" -#include "TView.h" -//#include "TVirtualCollectionProxy.h" -#include "TVirtualFFT.h" -#include "TVirtualHistPainter.h" -#include "TVirtualIndex.h" -#include "TVirtualIsAProxy.h" -#include "TVirtualPadPainter.h" -#include "TVirtualRefProxy.h" -#include "TVirtualStreamerInfo.h" -#include "TVirtualViewer3D.h" - -#include -#include - - -class Bench02RootApp { -public: - Bench02RootApp(); - ~Bench02RootApp(); - - void report(); - void close_file(TFile* f); -}; diff --git a/pypy/module/_cppyy/bench/bench02.xml b/pypy/module/_cppyy/bench/bench02.xml deleted file mode 100644 --- a/pypy/module/_cppyy/bench/bench02.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pypy/module/_cppyy/bench/hsimple.C b/pypy/module/_cppyy/bench/hsimple.C deleted file mode 100644 --- a/pypy/module/_cppyy/bench/hsimple.C +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -TFile *hsimple(Int_t get=0) -{ -// This program creates : -// - a one dimensional histogram -// - a two dimensional histogram -// - a profile histogram -// - a memory-resident ntuple -// -// These objects are filled with some random numbers and saved on a file. -// If get=1 the macro returns a pointer to the TFile of "hsimple.root" -// if this file exists, otherwise it is created. -// The file "hsimple.root" is created in $ROOTSYS/tutorials if the caller has -// write access to this directory, otherwise the file is created in $PWD - - TString filename = "hsimple.root"; - TString dir = gSystem->UnixPathName(gInterpreter->GetCurrentMacroName()); - dir.ReplaceAll("hsimple.C",""); - dir.ReplaceAll("/./","/"); - TFile *hfile = 0; - if (get) { - // if the argument get =1 return the file "hsimple.root" - // if the file does not exist, it is created - TString fullPath = dir+"hsimple.root"; - if (!gSystem->AccessPathName(fullPath,kFileExists)) { - hfile = TFile::Open(fullPath); //in $ROOTSYS/tutorials - if (hfile) return hfile; - } - //otherwise try $PWD/hsimple.root - if (!gSystem->AccessPathName("hsimple.root",kFileExists)) { - hfile = TFile::Open("hsimple.root"); //in current dir - if (hfile) return hfile; - } - } - //no hsimple.root file found. Must generate it ! - //generate hsimple.root in $ROOTSYS/tutorials if we have write access - if (!gSystem->AccessPathName(dir,kWritePermission)) { - filename = dir+"hsimple.root"; - } else if (!gSystem->AccessPathName(".",kWritePermission)) { - //otherwise generate hsimple.root in the current directory - } else { - printf("you must run the script in a directory with write access\n"); - return 0; - } - hfile = (TFile*)gROOT->FindObject(filename); if (hfile) hfile->Close(); - hfile = new TFile(filename,"RECREATE","Demo ROOT file with histograms"); - - // Create some histograms, a profile histogram and an ntuple - TH1F *hpx = new TH1F("hpx","This is the px distribution",100,-4,4); - hpx->SetFillColor(48); - TH2F *hpxpy = new TH2F("hpxpy","py vs px",40,-4,4,40,-4,4); - TProfile *hprof = new TProfile("hprof","Profile of pz versus px",100,-4,4,0,20); - TNtuple *ntuple = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - - gBenchmark->Start("hsimple"); - - // Create a new canvas. - TCanvas *c1 = new TCanvas("c1","Dynamic Filling Example",200,10,700,500); - c1->SetFillColor(42); - c1->GetFrame()->SetFillColor(21); - c1->GetFrame()->SetBorderSize(6); - c1->GetFrame()->SetBorderMode(-1); - - - // Fill histograms randomly - TRandom3 random; - Float_t px, py, pz; - const Int_t kUPDATE = 1000; - for (Int_t i = 0; i < 50000; i++) { - // random.Rannor(px,py); - px = random.Gaus(0, 1); - py = random.Gaus(0, 1); - pz = px*px + py*py; - Float_t rnd = random.Rndm(1); - hpx->Fill(px); - hpxpy->Fill(px,py); - hprof->Fill(px,pz); - ntuple->Fill(px,py,pz,rnd,i); - if (i && (i%kUPDATE) == 0) { - if (i == kUPDATE) hpx->Draw(); - c1->Modified(); - c1->Update(); - if (gSystem->ProcessEvents()) - break; - } From pypy.commits at gmail.com Mon Oct 30 06:19:07 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 30 Oct 2017 03:19:07 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: syntax highlighting, and a tweak Message-ID: <59f6fc9b.6896df0a.9740d.c45d@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5853:8eccb15f4dd9 Date: 2017-10-30 10:45 +0100 http://bitbucket.org/pypy/extradoc/changeset/8eccb15f4dd9/ Log: syntax highlighting, and a tweak diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -26,7 +26,9 @@ However, for the scope of this post, the actual task at hand is not so important, so let's jump straight to the code. To drive the quadcopter, a ``Creature`` has a ``run_step`` method which runs at each ``delta_t`` (`full -code`_):: +code`_): + +.. sourcecode:: python class Creature(object): INPUTS = 2 # z_setpoint, current z position @@ -59,11 +61,13 @@ .. _`full code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py -``run_step`` is run at 100Hz (in the virtual time frame of the simulation). At each +``run_step`` is called at 100Hz (in the virtual time frame of the simulation). At each generation, we test 500 creatures for a total of 12 virtual seconds each. So, we have a total of 600,000 executions of ``run_step`` at each generation. -At first, I simply tried to run this code on CPython; here is the result:: +At first, I simply tried to run this code on CPython; here is the result: + +.. sourcecode:: $ python -m ev.main Generation 1: ... [population = 500] [12.06 secs] @@ -75,7 +79,9 @@ Which means ~6.15 seconds/generation, excluding the first. -Then I tried with PyPy 5.9:: +Then I tried with PyPy 5.9: + +.. sourcecode:: $ pypy -m ev.main Generation 1: ... [population = 500] [63.90 secs] @@ -90,7 +96,9 @@ So, let's try to avoid cpyext. The first obvious step is to use numpypy_ instead of numpy (actually, there a hack_ to use just the micronumpy -part). Let's see if the speed improves:: +part). Let's see if the speed improves: + +.. sourcecode:: $ pypy -m ev.main # using numpypy Generation 1: ... [population = 500] [5.60 secs] @@ -154,8 +162,8 @@ All of this is very suboptimal: in this particular case, we know that the shape of ``self.matrix`` is always ``(3, 2)``: so, we are doing an incredible -amount of work. We also use ``malloc()`` to create a temporary array just to -call an RPython function which ultimately does a total of 6 multiplications +amount of work. We also ``malloc()`` two temporary arrays just to +call two functions which ultimately do a total of 6 multiplications and 8 additions. Note also that this is not a fault of the JIT: CPython+numpy has to do the same amount of work, just hidden inside C calls. @@ -166,7 +174,9 @@ is the main reason why the PyPy JIT does not even try to do it in this case. However, we **know** that the matrix is small, and always of the same -shape. So, let's unroll the loop manually:: +shape. So, let's unroll the loop manually: + +.. sourcecode:: python class SpecializedCreature(Creature): @@ -204,7 +214,9 @@ .. _`actual code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py#L100 .. _`list strategies`: https://morepypy.blogspot.it/2011/10/more-compact-lists-with-list-strategies.html -So, let's try to see how it performs. First, with CPython:: +So, let's try to see how it performs. First, with CPython: + +.. sourcecode:: $ python -m ev.main Generation 1: ... [population = 500] [7.61 secs] @@ -215,7 +227,9 @@ Generation 6: ... [population = 500] [3.69 secs] This looks good: 60% faster than the original CPython+numpy -implementation. Let's try on PyPy:: +implementation. Let's try on PyPy: + +.. sourcecode:: Generation 1: ... [population = 500] [0.39 secs] Generation 2: ... [population = 500] [0.10 secs] @@ -265,7 +279,7 @@ How to reproduce the results ----------------------------- -:: +.. sourcecode:: $ git clone https://github.com/antocuni/evolvingcopter $ cd evolvingcopter From pypy.commits at gmail.com Mon Oct 30 06:19:09 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 30 Oct 2017 03:19:09 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: remove the reference to list strategies: it's not true, since self.data items are of type numpy.float64. Bah. Message-ID: <59f6fc9d.e887df0a.2dc18.98e0@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5854:03cc5eb071c6 Date: 2017-10-30 10:47 +0100 http://bitbucket.org/pypy/extradoc/changeset/03cc5eb071c6/ Log: remove the reference to list strategies: it's not true, since self.data items are of type numpy.float64. Bah. diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -182,8 +182,7 @@ def __init__(self, *args, **kwargs): Creature.__init__(self, *args, **kwargs) - # store the data in a plain Python list, which pypy is able to - # optimize as a float array + # store the data in a plain Python list self.data = list(self.matrix.ravel()) + list(self.constant) self.data_state = [0.0] assert self.matrix.shape == (2, 3) @@ -207,12 +206,7 @@ In the `actual code`_ there is also a sanity check which asserts that the computed output is the very same as the one returned by ``Creature.run_step``. -Note that is code is particularly PyPy-friendly. Thanks to PyPy's `list strategies`_ -optimizations, ``self.data`` as a simple list of floats is internally represented -as a flat array of C doubles, i.e. very fast and compact. - .. _`actual code`: https://github.com/antocuni/evolvingcopter/blob/master/ev/creature.py#L100 -.. _`list strategies`: https://morepypy.blogspot.it/2011/10/more-compact-lists-with-list-strategies.html So, let's try to see how it performs. First, with CPython: From pypy.commits at gmail.com Mon Oct 30 06:19:12 2017 From: pypy.commits at gmail.com (antocuni) Date: Mon, 30 Oct 2017 03:19:12 -0700 (PDT) Subject: [pypy-commit] extradoc extradoc: another tweak Message-ID: <59f6fca0.e291df0a.1229.ed9c@mx.google.com> Author: Antonio Cuni Branch: extradoc Changeset: r5855:c2f10979b271 Date: 2017-10-30 10:49 +0100 http://bitbucket.org/pypy/extradoc/changeset/c2f10979b271/ Log: another tweak diff --git a/blog/draft/2017-10-how-to-make-80x-faster.rst b/blog/draft/2017-10-how-to-make-80x-faster.rst --- a/blog/draft/2017-10-how-to-make-80x-faster.rst +++ b/blog/draft/2017-10-how-to-make-80x-faster.rst @@ -256,7 +256,7 @@ As I said before, this is a very particular example, and the techniques described here do not always apply: it is not realistic to expect an 80x -speedup, unfortunately. However, it clearly shows the potential of PyPy when +speedup on arbitrary code, unfortunately. However, it clearly shows the potential of PyPy when it comes to high-speed computing. And most importantly, it's not a toy benchmark which was designed specifically to have good performance on PyPy: it's a real world example, albeit small. From pypy.commits at gmail.com Mon Oct 30 08:52:52 2017 From: pypy.commits at gmail.com (spapanik) Date: Mon, 30 Oct 2017 05:52:52 -0700 (PDT) Subject: [pypy-commit] pypy py3.6: Add inf and nan to cmath Message-ID: <59f720a4.178fdf0a.a7138.ac0b@mx.google.com> Author: Steve Papanik Branch: py3.6 Changeset: r92879:4a3e6bc30cc8 Date: 2017-10-30 12:46 +0000 http://bitbucket.org/pypy/pypy/changeset/4a3e6bc30cc8/ Log: Add inf and nan to cmath This also adds the corresponding complex constants. diff --git a/pypy/module/cmath/__init__.py b/pypy/module/cmath/__init__.py --- a/pypy/module/cmath/__init__.py +++ b/pypy/module/cmath/__init__.py @@ -40,6 +40,10 @@ interpleveldefs = { 'pi': 'space.newfloat(interp_cmath.pi)', 'e': 'space.newfloat(interp_cmath.e)', + 'inf': 'space.newfloat(interp_cmath.inf)', + 'nan': 'space.newfloat(interp_cmath.nan)', + 'infj': 'space.newcomplex(0.0, interp_cmath.inf)', + 'nanj': 'space.newcomplex(0.0, interp_cmath.nan)', 'isclose': 'interp_cmath.isclose', } interpleveldefs.update(dict([(name, 'interp_cmath.wrapped_' + name) diff --git a/pypy/module/cmath/interp_cmath.py b/pypy/module/cmath/interp_cmath.py --- a/pypy/module/cmath/interp_cmath.py +++ b/pypy/module/cmath/interp_cmath.py @@ -6,8 +6,13 @@ from pypy.module.cmath import names_and_docstrings from rpython.rlib import rcomplex, rfloat -pi = math.pi -e = math.e +pi = math.pi +e = math.e +inf = float('inf') +nan = float('nan') +infj = complex(0.0, inf) +nanj = complex(0.0, nan) + @specialize.arg(0) def call_c_func(c_func, space, x, y): diff --git a/pypy/module/cmath/test/test_cmath.py b/pypy/module/cmath/test/test_cmath.py --- a/pypy/module/cmath/test/test_cmath.py +++ b/pypy/module/cmath/test/test_cmath.py @@ -134,6 +134,24 @@ assert cmath.isclose(100000j, 100001j, abs_tol=1.5) assert not cmath.isclose(100000j, 100001j, abs_tol=0.5) + def test_infinity_and_nan_constants(self): + import cmath, math + assert cmath.inf.real == math.inf + assert cmath.inf.imag == 0.0 + assert cmath.infj.real == 0.0 + assert cmath.infj.imag == math.inf + + assert math.isnan(cmath.nan.real) + assert cmath.nan.imag == 0.0 + assert cmath.nanj.real == 0.0 + assert math.isnan(cmath.nanj.imag) + + # Check consistency with reprs. + assert repr(cmath.inf) == "inf" + assert repr(cmath.infj) == "infj" + assert repr(cmath.nan) == "nan" + assert repr(cmath.nanj) == "nanj" + def parse_testfile(fname): """Parse a file with test values @@ -211,7 +229,7 @@ def test_specific_values(): #if not float.__getformat__("double").startswith("IEEE"): # return - + import rpython # too fragile... fname = os.path.join(os.path.dirname(rpython.rlib.__file__), 'test', 'rcomplex_testcases.txt') From pypy.commits at gmail.com Mon Oct 30 10:10:49 2017 From: pypy.commits at gmail.com (thisch) Date: Mon, 30 Oct 2017 07:10:49 -0700 (PDT) Subject: [pypy-commit] pypy py3.6: Comment-out unsupported crypto algorithms Message-ID: <59f732e9.28361c0a.ea46.ae89@mx.google.com> Author: Thomas Hisch Branch: py3.6 Changeset: r92880:4b8536756542 Date: 2017-10-30 13:59 +0000 http://bitbucket.org/pypy/pypy/changeset/4b8536756542/ Log: Comment-out unsupported crypto algorithms diff --git a/lib-python/3/hashlib.py b/lib-python/3/hashlib.py --- a/lib-python/3/hashlib.py +++ b/lib-python/3/hashlib.py @@ -56,9 +56,12 @@ # This tuple and __get_builtin_constructor() must be modified if a new # always available algorithm is added. __always_supported = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', - 'blake2b', 'blake2s', - 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', - 'shake_128', 'shake_256') + # FIXME the following algorithms were added in + # cpython3.6 but are missing in pypy + # 'blake2b', 'blake2s', + # 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', # + # 'shake_128', 'shake_256' +) algorithms_guaranteed = set(__always_supported) @@ -78,7 +81,7 @@ try: if name in ('SHA1', 'sha1'): import _sha1 - cache['SHA1'] = cache['sha1'] = _sha1.sha1 + cache['SHA1'] = cache['sha1'] = _sha1py.sha1 elif name in ('MD5', 'md5'): import _md5 cache['MD5'] = cache['md5'] = _md5.md5 From pypy.commits at gmail.com Mon Oct 30 10:10:51 2017 From: pypy.commits at gmail.com (thisch) Date: Mon, 30 Oct 2017 07:10:51 -0700 (PDT) Subject: [pypy-commit] pypy py3.6: Fix typo introduced previous commit Message-ID: <59f732eb.87bbdf0a.c2a6c.5196@mx.google.com> Author: Thomas Hisch Branch: py3.6 Changeset: r92881:bcfbc11f8ebd Date: 2017-10-30 14:07 +0000 http://bitbucket.org/pypy/pypy/changeset/bcfbc11f8ebd/ Log: Fix typo introduced previous commit diff --git a/lib-python/3/hashlib.py b/lib-python/3/hashlib.py --- a/lib-python/3/hashlib.py +++ b/lib-python/3/hashlib.py @@ -59,7 +59,7 @@ # FIXME the following algorithms were added in # cpython3.6 but are missing in pypy # 'blake2b', 'blake2s', - # 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', # + # 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', # 'shake_128', 'shake_256' ) @@ -81,7 +81,7 @@ try: if name in ('SHA1', 'sha1'): import _sha1 - cache['SHA1'] = cache['sha1'] = _sha1py.sha1 + cache['SHA1'] = cache['sha1'] = _sha1.sha1 elif name in ('MD5', 'md5'): import _md5 cache['MD5'] = cache['md5'] = _md5.md5 From pypy.commits at gmail.com Mon Oct 30 11:29:58 2017 From: pypy.commits at gmail.com (...@funkyhat.org) Date: Mon, 30 Oct 2017 08:29:58 -0700 (PDT) Subject: [pypy-commit] pypy py3.6: Support b2a_base64 newline kwarg (new in 3.6) Message-ID: <59f74576.8dd71c0a.ee3de.dd99@mx.google.com> Author: m at funkyhat.org Branch: py3.6 Changeset: r92882:652e45c71368 Date: 2017-10-30 15:18 +0000 http://bitbucket.org/pypy/pypy/changeset/652e45c71368/ Log: Support b2a_base64 newline kwarg (new in 3.6) diff --git a/pypy/module/binascii/interp_base64.py b/pypy/module/binascii/interp_base64.py --- a/pypy/module/binascii/interp_base64.py +++ b/pypy/module/binascii/interp_base64.py @@ -78,8 +78,8 @@ table_b2a_base64 = ( "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/") - at unwrap_spec(bin='bufferstr') -def b2a_base64(space, bin): + at unwrap_spec(bin='bufferstr', newline=bool) +def b2a_base64(space, bin, __kwonly__, newline=True): "Base64-code line of data." newlength = (len(bin) + 2) // 3 @@ -109,5 +109,6 @@ elif leftbits == 4: res.append(table_b2a_base64[(leftchar & 0xf) << 2]) res.append(PAD) - res.append('\n') + if newline: + res.append('\n') return space.newbytes(res.build()) diff --git a/pypy/module/binascii/test/test_binascii.py b/pypy/module/binascii/test/test_binascii.py --- a/pypy/module/binascii/test/test_binascii.py +++ b/pypy/module/binascii/test/test_binascii.py @@ -121,16 +121,22 @@ raises(UnicodeEncodeError, self.binascii.a2b_base64, u"caf\xe9") def test_b2a_base64(self): - for input, expected in [ - (b"", b""), - (b"b", b"Yg=="), - (b"i\xb7\x1d", b"abcd"), - (b"i\xb7\x1dy", b"abcdeQ=="), - (b"i\xb7\x1dy\xf8", b"abcdefg="), - (b"i\xb7\x1dy\xf8!", b"abcdefgh"), - (b"i\xb7\x1d" * 345, b"abcd" * 345), - ]: - assert self.binascii.b2a_base64(input) == expected + b'\n' + for newline in (True, False, None): + for input, expected in [ + (b"", b""), + (b"b", b"Yg=="), + (b"i\xb7\x1d", b"abcd"), + (b"i\xb7\x1dy", b"abcdeQ=="), + (b"i\xb7\x1dy\xf8", b"abcdefg="), + (b"i\xb7\x1dy\xf8!", b"abcdefgh"), + (b"i\xb7\x1d" * 345, b"abcd" * 345), + ]: + kwargs = {} + if isinstance(newline, bool): + kwargs['newline'] = newline + if newline is not False: + expected += b'\n' + assert self.binascii.b2a_base64(input, **kwargs) == expected def test_a2b_qp(self): for input, expected in [ From pypy.commits at gmail.com Mon Oct 30 13:12:37 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 30 Oct 2017 10:12:37 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge default into branch Message-ID: <59f75d85.84addf0a.e128.1f9f@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92885:623c5cbb6132 Date: 2017-10-30 18:58 +0200 http://bitbucket.org/pypy/pypy/changeset/623c5cbb6132/ Log: merge default into branch diff too long, truncating to 2000 out of 3230 lines diff --git a/_pytest/terminal.py b/_pytest/terminal.py --- a/_pytest/terminal.py +++ b/_pytest/terminal.py @@ -366,11 +366,11 @@ EXIT_OK, EXIT_TESTSFAILED, EXIT_INTERRUPTED, EXIT_USAGEERROR, EXIT_NOTESTSCOLLECTED) if exitstatus in summary_exit_codes: - self.config.hook.pytest_terminal_summary(terminalreporter=self) self.summary_errors() self.summary_failures() self.summary_warnings() self.summary_passes() + self.config.hook.pytest_terminal_summary(terminalreporter=self) if exitstatus == EXIT_INTERRUPTED: self._report_keyboardinterrupt() del self._keyboardinterrupt_memo diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -182,6 +182,57 @@ technical difficulties. +What about numpy, numpypy, micronumpy? +-------------------------------------- + +Way back in 2011, the PyPy team `started to reimplement`_ numpy in PyPy. It +has two pieces: + + * the builtin module :source:`pypy/module/micronumpy`: this is written in + RPython and roughly covers the content of the ``numpy.core.multiarray`` + module. Confusingly enough, this is available in PyPy under the name + ``_numpypy``. It is included by default in all the official releases of + PyPy (but it might be dropped in the future). + + * a fork_ of the official numpy repository maintained by us and informally + called ``numpypy``: even more confusing, the name of the repo on bitbucket + is ``numpy``. The main difference with the upstream numpy, is that it is + based on the micronumpy module written in RPython, instead of of + ``numpy.core.multiarray`` which is written in C. + +Moreover, it is also possible to install the upstream version of ``numpy``: +its core is written in C and it runs on PyPy under the cpyext compatibility +layer. This is what you get if you do ``pypy -m pip install numpy``. + + +Should I install numpy or numpypy? +----------------------------------- + +TL;DR version: you should use numpy. You can install it by doing ``pypy -m pip +install numpy``. You might also be interested in using the experimental `PyPy +binary wheels`_ to save compilation time. + +The upstream ``numpy`` is written in C, and runs under the cpyext +compatibility layer. Nowadays, cpyext is mature enough that you can simply +use the upstream ``numpy``, since it passes 99.9% of the test suite. At the +moment of writing (October 2017) the main drawback of ``numpy`` is that cpyext +is infamously slow, and thus it has worse performance compared to +``numpypy``. However, we are actively working on improving it, as we expect to +reach the same speed, eventually. + +On the other hand, ``numpypy`` is more JIT-friendly and very fast to call, +since it is written in RPython: but it is a reimplementation, and it's hard to +be completely compatible: over the years the project slowly matured and +eventually it was able to call out to the LAPACK and BLAS libraries to speed +matrix calculations, and reached around an 80% parity with the upstream +numpy. However, 80% is far from 100%. Since cpyext/numpy compatibility is +progressing fast, we have discontinued support for ``numpypy``. + +.. _`started to reimplement`: https://morepypy.blogspot.co.il/2011/05/numpy-in-pypy-status-and-roadmap.html +.. _fork: https://bitbucket.org/pypy/numpy +.. _`PyPy binary wheels`: https://github.com/antocuni/pypy-wheels + + Is PyPy more clever than CPython about Tail Calls? -------------------------------------------------- diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py --- a/pypy/module/_cppyy/__init__.py +++ b/pypy/module/_cppyy/__init__.py @@ -7,7 +7,7 @@ interpleveldefs = { '_resolve_name' : 'interp_cppyy.resolve_name', '_scope_byname' : 'interp_cppyy.scope_byname', - '_template_byname' : 'interp_cppyy.template_byname', + '_is_template' : 'interp_cppyy.is_template', '_std_string_name' : 'interp_cppyy.std_string_name', '_set_class_generator' : 'interp_cppyy.set_class_generator', '_set_function_generator': 'interp_cppyy.set_function_generator', @@ -15,7 +15,9 @@ '_get_nullptr' : 'interp_cppyy.get_nullptr', 'CPPClassBase' : 'interp_cppyy.W_CPPClass', 'addressof' : 'interp_cppyy.addressof', + '_bind_object' : 'interp_cppyy._bind_object', 'bind_object' : 'interp_cppyy.bind_object', + 'move' : 'interp_cppyy.move', } appleveldefs = { diff --git a/pypy/module/_cppyy/capi/loadable_capi.py b/pypy/module/_cppyy/capi/loadable_capi.py --- a/pypy/module/_cppyy/capi/loadable_capi.py +++ b/pypy/module/_cppyy/capi/loadable_capi.py @@ -217,7 +217,8 @@ 'method_req_args' : ([c_scope, c_index], c_int), 'method_arg_type' : ([c_scope, c_index, c_int], c_ccharp), 'method_arg_default' : ([c_scope, c_index, c_int], c_ccharp), - 'method_signature' : ([c_scope, c_index], c_ccharp), + 'method_signature' : ([c_scope, c_index, c_int], c_ccharp), + 'method_prototype' : ([c_scope, c_index, c_int], c_ccharp), 'method_is_template' : ([c_scope, c_index], c_int), 'method_num_template_args' : ([c_scope, c_index], c_int), @@ -498,9 +499,12 @@ def c_method_arg_default(space, cppscope, index, arg_index): args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(arg_index)] return charp2str_free(space, call_capi(space, 'method_arg_default', args)) -def c_method_signature(space, cppscope, index): - args = [_ArgH(cppscope.handle), _ArgL(index)] +def c_method_signature(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] return charp2str_free(space, call_capi(space, 'method_signature', args)) +def c_method_prototype(space, cppscope, index, show_formalargs=True): + args = [_ArgH(cppscope.handle), _ArgL(index), _ArgL(show_formalargs)] + return charp2str_free(space, call_capi(space, 'method_prototype', args)) def c_method_is_template(space, cppscope, index): args = [_ArgH(cppscope.handle), _ArgL(index)] diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py --- a/pypy/module/_cppyy/converter.py +++ b/pypy/module/_cppyy/converter.py @@ -4,7 +4,7 @@ from rpython.rtyper.lltypesystem import rffi, lltype from rpython.rlib.rarithmetic import r_singlefloat, r_longfloat -from rpython.rlib import rfloat +from rpython.rlib import rfloat, rawrefcount from pypy.module._rawffi.interp_rawffi import letter2tp from pypy.module._rawffi.array import W_Array, W_ArrayInstance @@ -21,9 +21,9 @@ # match for the qualified type. -def get_rawobject(space, w_obj): +def get_rawobject(space, w_obj, can_be_None=True): from pypy.module._cppyy.interp_cppyy import W_CPPClass - cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + cppinstance = space.interp_w(W_CPPClass, w_obj, can_be_None=can_be_None) if cppinstance: rawobject = cppinstance.get_rawobject() assert lltype.typeOf(rawobject) == capi.C_OBJECT @@ -48,17 +48,16 @@ return capi.C_NULL_OBJECT def is_nullpointer_specialcase(space, w_obj): - # 0, None, and nullptr may serve as "NULL", check for any of them + # 0 and nullptr may serve as "NULL" # integer 0 try: return space.int_w(w_obj) == 0 except Exception: pass - # None or nullptr + # C++-style nullptr from pypy.module._cppyy import interp_cppyy - return space.is_true(space.is_(w_obj, space.w_None)) or \ - space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) + return space.is_true(space.is_(w_obj, interp_cppyy.get_nullptr(space))) def get_rawbuffer(space, w_obj): # raw buffer @@ -74,7 +73,7 @@ return rffi.cast(rffi.VOIDP, space.uint_w(arr.getbuffer(space))) except Exception: pass - # pre-defined NULL + # pre-defined nullptr if is_nullpointer_specialcase(space, w_obj): return rffi.cast(rffi.VOIDP, 0) raise TypeError("not an addressable buffer") @@ -392,6 +391,7 @@ _immutable_fields_ = ['typecode'] typecode = 'g' + class CStringConverter(TypeConverter): def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.LONGP, address) @@ -408,18 +408,27 @@ def free_argument(self, space, arg, call_local): lltype.free(rffi.cast(rffi.CCHARPP, arg)[0], flavor='raw') +class CStringConverterWithSize(CStringConverter): + _immutable_fields_ = ['size'] + + def __init__(self, space, extra): + self.size = extra + + def from_memory(self, space, w_obj, w_pycppclass, offset): + address = self._get_raw_address(space, w_obj, offset) + charpptr = rffi.cast(rffi.CCHARP, address) + strsize = self.size + if charpptr[self.size-1] == '\0': + strsize = self.size-1 # rffi will add \0 back + return space.newbytes(rffi.charpsize2str(charpptr, strsize)) + class VoidPtrConverter(TypeConverter): def _unwrap_object(self, space, w_obj): try: obj = get_rawbuffer(space, w_obj) except TypeError: - try: - # TODO: accept a 'capsule' rather than naked int - # (do accept int(0), though) - obj = rffi.cast(rffi.VOIDP, space.uint_w(w_obj)) - except Exception: - obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) + obj = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj, False)) return obj def cffi_type(self, space): @@ -463,12 +472,12 @@ def convert_argument(self, space, w_obj, address, call_local): x = rffi.cast(rffi.VOIDPP, address) ba = rffi.cast(rffi.CCHARP, address) - r = rffi.cast(rffi.VOIDPP, call_local) try: - r[0] = get_rawbuffer(space, w_obj) + x[0] = get_rawbuffer(space, w_obj) except TypeError: + r = rffi.cast(rffi.VOIDPP, call_local) r[0] = rffi.cast(rffi.VOIDP, get_rawobject(space, w_obj)) - x[0] = rffi.cast(rffi.VOIDP, call_local) + x[0] = rffi.cast(rffi.VOIDP, call_local) ba[capi.c_function_arg_typeoffset(space)] = self.typecode def finalize_call(self, space, w_obj, call_local): @@ -495,9 +504,13 @@ def _unwrap_object(self, space, w_obj): from pypy.module._cppyy.interp_cppyy import W_CPPClass if isinstance(w_obj, W_CPPClass): - if capi.c_is_subtype(space, w_obj.cppclass, self.clsdecl): + from pypy.module._cppyy.interp_cppyy import INSTANCE_FLAGS_IS_R_VALUE + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + # reject moves as all are explicit + raise ValueError("lvalue expected") + if capi.c_is_subtype(space, w_obj.clsdecl, self.clsdecl): rawobject = w_obj.get_rawobject() - offset = capi.c_base_offset(space, w_obj.cppclass, self.clsdecl, rawobject, 1) + offset = capi.c_base_offset(space, w_obj.clsdecl, self.clsdecl, rawobject, 1) obj_address = capi.direct_ptradd(rawobject, offset) return rffi.cast(capi.C_OBJECT, obj_address) raise oefmt(space.w_TypeError, @@ -518,6 +531,17 @@ x = rffi.cast(rffi.VOIDPP, address) x[0] = rffi.cast(rffi.VOIDP, self._unwrap_object(space, w_obj)) +class InstanceMoveConverter(InstanceRefConverter): + def _unwrap_object(self, space, w_obj): + # moving is same as by-ref, but have to check that move is allowed + from pypy.module._cppyy.interp_cppyy import W_CPPClass, INSTANCE_FLAGS_IS_R_VALUE + if isinstance(w_obj, W_CPPClass): + if w_obj.flags & INSTANCE_FLAGS_IS_R_VALUE: + w_obj.flags &= ~INSTANCE_FLAGS_IS_R_VALUE + return InstanceRefConverter._unwrap_object(self, space, w_obj) + raise oefmt(space.w_ValueError, "object is not an rvalue") + + class InstanceConverter(InstanceRefConverter): def convert_argument_libffi(self, space, w_obj, address, call_local): @@ -527,7 +551,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): self._is_abstract(space) @@ -548,7 +572,7 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False) + return interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False) def to_memory(self, space, w_obj, w_value, offset): address = rffi.cast(rffi.VOIDPP, self._get_raw_address(space, w_obj, offset)) @@ -582,8 +606,8 @@ def from_memory(self, space, w_obj, w_pycppclass, offset): address = rffi.cast(capi.C_OBJECT, self._get_raw_address(space, w_obj, offset)) from pypy.module._cppyy import interp_cppyy - return interp_cppyy.wrap_cppobject(space, address, self.clsdecl, - do_cast=False, is_ref=True) + return interp_cppyy.wrap_cppinstance( + space, address, self.clsdecl, do_cast=False, is_ref=True) class StdStringConverter(InstanceConverter): @@ -606,7 +630,7 @@ assign = self.clsdecl.get_overload("__assign__") from pypy.module._cppyy import interp_cppyy assign.call( - interp_cppyy.wrap_cppobject(space, address, self.clsdecl, do_cast=False), [w_value]) + interp_cppyy.wrap_cppinstance(space, address, self.clsdecl, do_cast=False), [w_value]) except Exception: InstanceConverter.to_memory(self, space, w_obj, w_value, offset) @@ -672,7 +696,7 @@ _converters = {} # builtin and custom types _a_converters = {} # array and ptr versions of above -def get_converter(space, name, default): +def get_converter(space, _name, default): # The matching of the name to a converter should follow: # 1) full, exact match # 1a) const-removed match @@ -680,9 +704,9 @@ # 3) accept ref as pointer (for the stubs, const& can be # by value, but that does not work for the ffi path) # 4) generalized cases (covers basically all user classes) - # 5) void converter, which fails on use + # 5) void* or void converter (which fails on use) - name = capi.c_resolve_name(space, name) + name = capi.c_resolve_name(space, _name) # 1) full, exact match try: @@ -701,7 +725,7 @@ clean_name = capi.c_resolve_name(space, helper.clean_type(name)) try: # array_index may be negative to indicate no size or no size found - array_size = helper.array_size(name) + array_size = helper.array_size(_name) # uses original arg return _a_converters[clean_name+compound](space, array_size) except KeyError: pass @@ -719,6 +743,8 @@ return InstancePtrConverter(space, clsdecl) elif compound == "&": return InstanceRefConverter(space, clsdecl) + elif compound == "&&": + return InstanceMoveConverter(space, clsdecl) elif compound == "**": return InstancePtrPtrConverter(space, clsdecl) elif compound == "": @@ -726,11 +752,13 @@ elif capi.c_is_enum(space, clean_name): return _converters['unsigned'](space, default) - # 5) void converter, which fails on use - # + # 5) void* or void converter (which fails on use) + if 0 <= compound.find('*'): + return VoidPtrConverter(space, default) # "user knows best" + # return a void converter here, so that the class can be build even - # when some types are unknown; this overload will simply fail on use - return VoidConverter(space, name) + # when some types are unknown + return VoidConverter(space, name) # fails on use _converters["bool"] = BoolConverter @@ -847,6 +875,10 @@ for name in names: _a_converters[name+'[]'] = ArrayConverter _a_converters[name+'*'] = PtrConverter + + # special case, const char* w/ size and w/o '\0' + _a_converters["const char[]"] = CStringConverterWithSize + _build_array_converters() # add another set of aliased names diff --git a/pypy/module/_cppyy/executor.py b/pypy/module/_cppyy/executor.py --- a/pypy/module/_cppyy/executor.py +++ b/pypy/module/_cppyy/executor.py @@ -163,7 +163,7 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_l(space, cppmethod, cppthis, num_args, args) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - pyres = interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + pyres = interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) return pyres def execute_libffi(self, space, cif_descr, funcaddr, buffer): @@ -171,7 +171,7 @@ result = rffi.ptradd(buffer, cif_descr.exchange_result) from pypy.module._cppyy import interp_cppyy ptr_result = rffi.cast(capi.C_OBJECT, rffi.cast(rffi.VOIDPP, result)[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) class InstancePtrPtrExecutor(InstancePtrExecutor): @@ -180,7 +180,7 @@ voidp_result = capi.c_call_r(space, cppmethod, cppthis, num_args, args) ref_address = rffi.cast(rffi.VOIDPP, voidp_result) ptr_result = rffi.cast(capi.C_OBJECT, ref_address[0]) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible @@ -192,8 +192,8 @@ from pypy.module._cppyy import interp_cppyy long_result = capi.c_call_o(space, cppmethod, cppthis, num_args, args, self.cppclass) ptr_result = rffi.cast(capi.C_OBJECT, long_result) - return interp_cppyy.wrap_cppobject(space, ptr_result, self.cppclass, - do_cast=False, python_owns=True, fresh=True) + return interp_cppyy.wrap_cppinstance(space, ptr_result, self.cppclass, + do_cast=False, python_owns=True, fresh=True) def execute_libffi(self, space, cif_descr, funcaddr, buffer): from pypy.module._cppyy.interp_cppyy import FastCallNotPossible diff --git a/pypy/module/_cppyy/include/capi.h b/pypy/module/_cppyy/include/capi.h --- a/pypy/module/_cppyy/include/capi.h +++ b/pypy/module/_cppyy/include/capi.h @@ -19,14 +19,15 @@ RPY_EXTERN int cppyy_num_scopes(cppyy_scope_t parent); RPY_EXTERN - char* cppyy_scope_name(cppyy_scope_t parent, int iscope); - + char* cppyy_scope_name(cppyy_scope_t parent, cppyy_index_t iscope); RPY_EXTERN char* cppyy_resolve_name(const char* cppitem_name); RPY_EXTERN cppyy_scope_t cppyy_get_scope(const char* scope_name); RPY_EXTERN cppyy_type_t cppyy_actual_class(cppyy_type_t klass, cppyy_object_t obj); + RPY_EXTERN + size_t cppyy_size_of(cppyy_type_t klass); /* memory management ------------------------------------------------------ */ RPY_EXTERN @@ -120,6 +121,8 @@ RPY_EXTERN char* cppyy_method_name(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN + char* cppyy_method_mangled_name(cppyy_scope_t scope, cppyy_index_t idx); + RPY_EXTERN char* cppyy_method_result_type(cppyy_scope_t scope, cppyy_index_t idx); RPY_EXTERN int cppyy_method_num_args(cppyy_scope_t scope, cppyy_index_t idx); @@ -130,7 +133,9 @@ RPY_EXTERN char* cppyy_method_arg_default(cppyy_scope_t scope, cppyy_index_t idx, int arg_index); RPY_EXTERN - char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx); + char* cppyy_method_signature(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); + RPY_EXTERN + char* cppyy_method_prototype(cppyy_scope_t scope, cppyy_index_t idx, int show_formalargs); RPY_EXTERN int cppyy_method_is_template(cppyy_scope_t scope, cppyy_index_t idx); @@ -147,8 +152,12 @@ /* method properties ------------------------------------------------------ */ RPY_EXTERN + int cppyy_is_publicmethod(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_constructor(cppyy_type_t type, cppyy_index_t idx); RPY_EXTERN + int cppyy_is_destructor(cppyy_type_t type, cppyy_index_t idx); + RPY_EXTERN int cppyy_is_staticmethod(cppyy_type_t type, cppyy_index_t idx); /* data member reflection information ------------------------------------- */ diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py --- a/pypy/module/_cppyy/interp_cppyy.py +++ b/pypy/module/_cppyy/interp_cppyy.py @@ -2,7 +2,7 @@ from pypy.interpreter.error import OperationError, oefmt from pypy.interpreter.gateway import interp2app, unwrap_spec -from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty, interp_attrproperty_w +from pypy.interpreter.typedef import TypeDef, GetSetProperty, interp_attrproperty from pypy.interpreter.baseobjspace import W_Root from rpython.rtyper.lltypesystem import rffi, lltype, llmemory @@ -15,6 +15,10 @@ from pypy.module._cppyy import converter, executor, ffitypes, helper +INSTANCE_FLAGS_PYTHON_OWNS = 0x0001 +INSTANCE_FLAGS_IS_REF = 0x0002 +INSTANCE_FLAGS_IS_R_VALUE = 0x0004 + class FastCallNotPossible(Exception): pass @@ -33,16 +37,21 @@ class State(object): def __init__(self, space): + # final scoped name -> opaque handle self.cppscope_cache = { - "void" : W_CPPClassDecl(space, "void", capi.C_NULL_TYPE) } + 'void' : W_CPPClassDecl(space, capi.C_NULL_TYPE, 'void') } + # opaque handle -> app-level python class + self.cppclass_registry = {} + # app-level class generator callback + self.w_clgen_callback = None + # app-level function generator callback (currently not used) + self.w_fngen_callback = None + # C++11's nullptr self.w_nullptr = None - self.cpptemplate_cache = {} - self.cppclass_registry = {} - self.w_clgen_callback = None - self.w_fngen_callback = None def get_nullptr(space): - if hasattr(space, "fake"): + # construct a unique address that compares to NULL, serves as nullptr + if hasattr(space, 'fake'): raise NotImplementedError state = space.fromcache(State) if state.w_nullptr is None: @@ -58,52 +67,48 @@ state.w_nullptr = nullarr return state.w_nullptr - at unwrap_spec(name='text') -def resolve_name(space, name): - return space.newtext(capi.c_resolve_name(space, name)) + at unwrap_spec(scoped_name='text') +def resolve_name(space, scoped_name): + return space.newtext(capi.c_resolve_name(space, scoped_name)) - at unwrap_spec(name='text') -def scope_byname(space, name): - true_name = capi.c_resolve_name(space, name) +# memoized lookup of handles by final, scoped, name of classes/namespaces + at unwrap_spec(final_scoped_name='text') +def scope_byname(space, final_scoped_name): state = space.fromcache(State) try: - return state.cppscope_cache[true_name] + return state.cppscope_cache[final_scoped_name] except KeyError: pass - opaque_handle = capi.c_get_scope_opaque(space, true_name) + opaque_handle = capi.c_get_scope_opaque(space, final_scoped_name) assert lltype.typeOf(opaque_handle) == capi.C_SCOPE if opaque_handle: - final_name = capi.c_final_name(space, opaque_handle) - if capi.c_is_namespace(space, opaque_handle): - cppscope = W_CPPNamespaceDecl(space, final_name, opaque_handle) - elif capi.c_has_complex_hierarchy(space, opaque_handle): - cppscope = W_CPPComplexClassDecl(space, final_name, opaque_handle) + isns = capi.c_is_namespace(space, opaque_handle) + if isns: + cppscope = W_CPPNamespaceDecl(space, opaque_handle, final_scoped_name) else: - cppscope = W_CPPClassDecl(space, final_name, opaque_handle) - state.cppscope_cache[name] = cppscope + if capi.c_has_complex_hierarchy(space, opaque_handle): + cppscope = W_CPPComplexClassDecl(space, opaque_handle, final_scoped_name) + else: + cppscope = W_CPPClassDecl(space, opaque_handle, final_scoped_name) - cppscope._build_methods() - cppscope._find_datamembers() + # store in the cache to prevent recursion + state.cppscope_cache[final_scoped_name] = cppscope + + if not isns: + # build methods/data; TODO: also defer this for classes (a functional __dir__ + # and instrospection for help() is enough and allows more lazy loading) + cppscope._build_methods() + cppscope._find_datamembers() + return cppscope return None - at unwrap_spec(name='text') -def template_byname(space, name): - state = space.fromcache(State) - try: - return state.cpptemplate_cache[name] - except KeyError: - pass - - if capi.c_is_template(space, name): - cpptemplate = W_CPPTemplateType(space, name) - state.cpptemplate_cache[name] = cpptemplate - return cpptemplate - - return None + at unwrap_spec(final_scoped_name='text') +def is_template(space, final_scoped_name): + return space.newbool(capi.c_is_template(space, final_scoped_name)) def std_string_name(space): return space.newtext(capi.std_string_name) @@ -189,8 +194,13 @@ # check number of given arguments against required (== total - defaults) args_expected = len(self.arg_defs) args_given = len(args_w) - if args_expected < args_given or args_given < self.args_required: - raise oefmt(self.space.w_TypeError, "wrong number of arguments") + + if args_given < self.args_required: + raise oefmt(self.space.w_TypeError, + "takes at least %d arguments (%d given)", self.args_required, args_given) + elif args_expected < args_given: + raise oefmt(self.space.w_TypeError, + "takes at most %d arguments (%d given)", args_expected, args_given) # initial setup of converters, executors, and libffi (if available) if self.converters is None: @@ -376,8 +386,11 @@ conv.free_argument(self.space, rffi.cast(capi.C_OBJECT, arg_i), loc_i) capi.c_deallocate_function_args(self.space, args) - def signature(self): - return capi.c_method_signature(self.space, self.scope, self.index) + def signature(self, show_formalargs=True): + return capi.c_method_signature(self.space, self.scope, self.index, show_formalargs) + + def prototype(self, show_formalargs=True): + return capi.c_method_prototype(self.space, self.scope, self.index, show_formalargs) def priority(self): total_arg_priority = 0 @@ -391,7 +404,7 @@ lltype.free(self.cif_descr, flavor='raw') def __repr__(self): - return "CPPMethod: %s" % self.signature() + return "CPPMethod: %s" % self.prototype() def _freeze_(self): assert 0, "you should never have a pre-built instance of this!" @@ -407,7 +420,7 @@ return capi.C_NULL_OBJECT def __repr__(self): - return "CPPFunction: %s" % self.signature() + return "CPPFunction: %s" % self.prototype() class CPPTemplatedCall(CPPMethod): @@ -440,7 +453,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPTemplatedCall: %s" % self.signature() + return "CPPTemplatedCall: %s" % self.prototype() class CPPConstructor(CPPMethod): @@ -462,7 +475,7 @@ return CPPMethod.call(self, cppthis, args_w) def __repr__(self): - return "CPPConstructor: %s" % self.signature() + return "CPPConstructor: %s" % self.prototype() class CPPSetItem(CPPMethod): @@ -549,12 +562,12 @@ w_exc_type = e.w_type elif all_same_type and not e.match(self.space, w_exc_type): all_same_type = False - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' '+e.errorstr(self.space) except Exception as e: # can not special case this for non-overloaded functions as we anyway need an # OperationError error down from here - errmsg += '\n '+cppyyfunc.signature()+' =>\n' + errmsg += '\n '+cppyyfunc.prototype()+' =>\n' errmsg += ' Exception: '+str(e) if all_same_type and w_exc_type is not None: @@ -562,20 +575,20 @@ else: raise OperationError(self.space.w_TypeError, self.space.newtext(errmsg)) - def signature(self): - sig = self.functions[0].signature() + def prototype(self): + sig = self.functions[0].prototype() for i in range(1, len(self.functions)): - sig += '\n'+self.functions[i].signature() + sig += '\n'+self.functions[i].prototype() return self.space.newtext(sig) def __repr__(self): - return "W_CPPOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPOverload(%s)" % [f.prototype() for f in self.functions] W_CPPOverload.typedef = TypeDef( 'CPPOverload', is_static = interp2app(W_CPPOverload.is_static), call = interp2app(W_CPPOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPOverload.prototype), ) @@ -591,24 +604,40 @@ @jit.unroll_safe @unwrap_spec(args_w='args_w') def call(self, w_cppinstance, args_w): + # TODO: factor out the following: + if capi.c_is_abstract(self.space, self.scope.handle): + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.scope.name) w_result = W_CPPOverload.call(self, w_cppinstance, args_w) newthis = rffi.cast(capi.C_OBJECT, self.space.uint_w(w_result)) cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if cppinstance is not None: cppinstance._rawobject = newthis memory_regulator.register(cppinstance) - return w_cppinstance - return wrap_cppobject(self.space, newthis, self.functions[0].scope, - do_cast=False, python_owns=True, fresh=True) def __repr__(self): - return "W_CPPConstructorOverload(%s)" % [f.signature() for f in self.functions] + return "W_CPPConstructorOverload(%s)" % [f.prototype() for f in self.functions] W_CPPConstructorOverload.typedef = TypeDef( 'CPPConstructorOverload', is_static = interp2app(W_CPPConstructorOverload.is_static), call = interp2app(W_CPPConstructorOverload.call), - signature = interp2app(W_CPPOverload.signature), + prototype = interp2app(W_CPPConstructorOverload.prototype), +) + + +class W_CPPTemplateOverload(W_CPPOverload): + @unwrap_spec(args_w='args_w') + def __getitem__(self, args_w): + pass + + def __repr__(self): + return "W_CPPTemplateOverload(%s)" % [f.prototype() for f in self.functions] + +W_CPPTemplateOverload.typedef = TypeDef( + 'CPPTemplateOverload', + __getitem__ = interp2app(W_CPPTemplateOverload.call), ) @@ -622,6 +651,9 @@ def __call__(self, args_w): return self.method.bound_call(self.cppthis, args_w) + def __repr__(self): + return "W_CPPBoundMethod(%s)" % [f.prototype() for f in self.functions] + W_CPPBoundMethod.typedef = TypeDef( 'CPPBoundMethod', __call__ = interp2app(W_CPPBoundMethod.__call__), @@ -643,8 +675,8 @@ def _get_offset(self, cppinstance): if cppinstance: - assert lltype.typeOf(cppinstance.cppclass.handle) == lltype.typeOf(self.scope.handle) - offset = self.offset + cppinstance.cppclass.get_base_offset(cppinstance, self.scope) + assert lltype.typeOf(cppinstance.clsdecl.handle) == lltype.typeOf(self.scope.handle) + offset = self.offset + cppinstance.clsdecl.get_base_offset(cppinstance, self.scope) else: offset = self.offset return offset @@ -652,7 +684,7 @@ def get(self, w_cppinstance, w_pycppclass): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) return self.converter.from_memory(self.space, w_cppinstance, w_pycppclass, offset) @@ -660,7 +692,7 @@ def set(self, w_cppinstance, w_value): cppinstance = self.space.interp_w(W_CPPClass, w_cppinstance, can_be_None=True) if not cppinstance: - raise oefmt(self.space.w_ReferenceError, + raise oefmt(self.space.w_AttributeError, "attribute access requires an instance") offset = self._get_offset(cppinstance) self.converter.to_memory(self.space, w_cppinstance, w_value, offset) @@ -705,12 +737,12 @@ return space.w_False class W_CPPScopeDecl(W_Root): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] _immutable_fields_ = ['handle', 'name'] - def __init__(self, space, name, opaque_handle): + def __init__(self, space, opaque_handle, final_scoped_name): self.space = space - self.name = name + self.name = final_scoped_name assert lltype.typeOf(opaque_handle) == capi.C_SCOPE self.handle = opaque_handle self.methods = {} @@ -753,7 +785,7 @@ overload = self.get_overload(name) sig = '(%s)' % signature for f in overload.functions: - if 0 < f.signature().find(sig): + if f.signature(False) == sig: return W_CPPOverload(self.space, self, [f]) raise oefmt(self.space.w_LookupError, "no overload matches signature") @@ -769,6 +801,9 @@ # classes for inheritance. Both are python classes, though, and refactoring # may be in order at some point. class W_CPPNamespaceDecl(W_CPPScopeDecl): + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name'] + def _make_cppfunction(self, pyname, index): num_args = capi.c_method_num_args(self.space, self, index) args_required = capi.c_method_req_args(self.space, self, index) @@ -779,9 +814,6 @@ arg_defs.append((arg_type, arg_dflt)) return CPPFunction(self.space, self, index, arg_defs, args_required) - def _build_methods(self): - pass # force lazy lookups in namespaces - def _make_datamember(self, dm_name, dm_idx): type_name = capi.c_datamember_type(self.space, self, dm_idx) offset = capi.c_datamember_offset(self.space, self, dm_idx) @@ -791,9 +823,6 @@ self.datamembers[dm_name] = datamember return datamember - def _find_datamembers(self): - pass # force lazy lookups in namespaces - def find_overload(self, meth_name): indices = capi.c_method_indices_from_name(self.space, self, meth_name) if not indices: @@ -855,18 +884,21 @@ class W_CPPClassDecl(W_CPPScopeDecl): - _attrs_ = ['space', 'name', 'handle', 'methods', 'datamembers'] - _immutable_fields_ = ['handle', 'constructor', 'methods[*]', 'datamembers[*]'] + _attrs_ = ['space', 'handle', 'name', 'methods', 'datamembers'] + _immutable_fields_ = ['handle', 'name', 'methods[*]', 'datamembers[*]'] def _build_methods(self): assert len(self.methods) == 0 methods_temp = {} for i in range(capi.c_num_methods(self.space, self)): idx = capi.c_method_index_at(self.space, self, i) - pyname = helper.map_operator_name(self.space, - capi.c_method_name(self.space, self, idx), - capi.c_method_num_args(self.space, self, idx), - capi.c_method_result_type(self.space, self, idx)) + if capi.c_is_constructor(self.space, self, idx): + pyname = '__init__' + else: + pyname = helper.map_operator_name(self.space, + capi.c_method_name(self.space, self, idx), + capi.c_method_num_args(self.space, self, idx), + capi.c_method_result_type(self.space, self, idx)) cppmethod = self._make_cppfunction(pyname, idx) methods_temp.setdefault(pyname, []).append(cppmethod) # the following covers the case where the only kind of operator[](idx) @@ -883,7 +915,7 @@ # create the overload methods from the method sets for pyname, methods in methods_temp.iteritems(): CPPMethodSort(methods).sort() - if pyname == self.name: + if pyname == '__init__': overload = W_CPPConstructorOverload(self.space, self, methods[:]) else: overload = W_CPPOverload(self.space, self, methods[:]) @@ -934,11 +966,11 @@ raise self.missing_attribute_error(name) def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return 0 def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl return cppinstance.get_rawobject() def is_namespace(self): @@ -973,13 +1005,13 @@ class W_CPPComplexClassDecl(W_CPPClassDecl): def get_base_offset(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = capi.c_base_offset(self.space, self, calling_scope, cppinstance.get_rawobject(), 1) return offset def get_cppthis(self, cppinstance, calling_scope): - assert self == cppinstance.cppclass + assert self == cppinstance.clsdecl offset = self.get_base_offset(cppinstance, calling_scope) return capi.direct_ptradd(cppinstance.get_rawobject(), offset) @@ -997,70 +1029,56 @@ W_CPPComplexClassDecl.typedef.acceptable_as_base_class = False -class W_CPPTemplateType(W_Root): - _attrs_ = ['space', 'name'] - _immutable_fields = ['name'] - - def __init__(self, space, name): - self.space = space - self.name = name - - @unwrap_spec(args_w='args_w') - def __call__(self, args_w): - # TODO: this is broken but unused (see pythonify.py) - fullname = "".join([self.name, '<', self.space.text_w(args_w[0]), '>']) - return scope_byname(self.space, fullname) - -W_CPPTemplateType.typedef = TypeDef( - 'CPPTemplateType', - __call__ = interp2app(W_CPPTemplateType.__call__), -) -W_CPPTemplateType.typedef.acceptable_as_base_class = False - - class W_CPPClass(W_Root): - _attrs_ = ['space', 'cppclass', '_rawobject', 'isref', 'python_owns', + _attrs_ = ['space', 'clsdecl', '_rawobject', 'flags', 'finalizer_registered'] - _immutable_fields_ = ["cppclass", "isref"] + _immutable_fields_ = ['clsdecl'] finalizer_registered = False - def __init__(self, space, cppclass, rawobject, isref, python_owns): + def __init__(self, space, decl, rawobject, isref, python_owns): self.space = space - self.cppclass = cppclass + self.clsdecl = decl assert lltype.typeOf(rawobject) == capi.C_OBJECT assert not isref or rawobject self._rawobject = rawobject assert not isref or not python_owns - self.isref = isref - self.python_owns = python_owns - self._opt_register_finalizer() + self.flags = 0 + if isref: + self.flags |= INSTANCE_FLAGS_IS_REF + if python_owns: + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() def _opt_register_finalizer(self): - if self.python_owns and not self.finalizer_registered \ - and not hasattr(self.space, "fake"): + if not self.finalizer_registered and not hasattr(self.space, "fake"): + assert self.flags & INSTANCE_FLAGS_PYTHON_OWNS self.register_finalizer(self.space) self.finalizer_registered = True def _nullcheck(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): raise oefmt(self.space.w_ReferenceError, "trying to access a NULL pointer") # allow user to determine ownership rules on a per object level def fget_python_owns(self, space): - return space.newbool(self.python_owns) + return space.newbool(bool(self.flags & INSTANCE_FLAGS_PYTHON_OWNS)) @unwrap_spec(value=bool) def fset_python_owns(self, space, value): - self.python_owns = space.is_true(value) - self._opt_register_finalizer() + if space.is_true(value): + self.flags |= INSTANCE_FLAGS_PYTHON_OWNS + self._opt_register_finalizer() + else: + self.flags &= ~INSTANCE_FLAGS_PYTHON_OWNS def get_cppthis(self, calling_scope): - return self.cppclass.get_cppthis(self, calling_scope) + return self.clsdecl.get_cppthis(self, calling_scope) def get_rawobject(self): - if not self.isref: + if not (self.flags & INSTANCE_FLAGS_IS_REF): return self._rawobject else: ptrptr = rffi.cast(rffi.VOIDPP, self._rawobject) @@ -1078,12 +1096,9 @@ return None def instance__init__(self, args_w): - if capi.c_is_abstract(self.space, self.cppclass.handle): - raise oefmt(self.space.w_TypeError, - "cannot instantiate abstract class '%s'", - self.cppclass.name) - constructor_overload = self.cppclass.get_overload(self.cppclass.name) - constructor_overload.call(self, args_w) + raise oefmt(self.space.w_TypeError, + "cannot instantiate abstract class '%s'", + self.clsdecl.name) def instance__eq__(self, w_other): # special case: if other is None, compare pointer-style @@ -1099,7 +1114,7 @@ for name in ["", "__gnu_cxx", "__1"]: nss = scope_byname(self.space, name) meth_idx = capi.c_get_global_operator( - self.space, nss, self.cppclass, other.cppclass, "operator==") + self.space, nss, self.clsdecl, other.clsdecl, "operator==") if meth_idx != -1: f = nss._make_cppfunction("operator==", meth_idx) ol = W_CPPOverload(self.space, nss, [f]) @@ -1118,14 +1133,15 @@ # fallback 2: direct pointer comparison (the class comparison is needed since # the first data member in a struct and the struct have the same address) other = self.space.interp_w(W_CPPClass, w_other, can_be_None=False) # TODO: factor out - iseq = (self._rawobject == other._rawobject) and (self.cppclass == other.cppclass) + iseq = (self._rawobject == other._rawobject) and (self.clsdecl == other.clsdecl) return self.space.newbool(iseq) def instance__ne__(self, w_other): return self.space.not_(self.instance__eq__(w_other)) def instance__nonzero__(self): - if not self._rawobject or (self.isref and not self.get_rawobject()): + if not self._rawobject or \ + ((self.flags & INSTANCE_FLAGS_IS_REF) and not self.get_rawobject()): return self.space.w_False return self.space.w_True @@ -1134,36 +1150,35 @@ if w_as_builtin is not None: return self.space.len(w_as_builtin) raise oefmt(self.space.w_TypeError, - "'%s' has no length", self.cppclass.name) + "'%s' has no length", self.clsdecl.name) def instance__cmp__(self, w_other): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.cmp(w_as_builtin, w_other) raise oefmt(self.space.w_AttributeError, - "'%s' has no attribute __cmp__", self.cppclass.name) + "'%s' has no attribute __cmp__", self.clsdecl.name) def instance__repr__(self): w_as_builtin = self._get_as_builtin() if w_as_builtin is not None: return self.space.repr(w_as_builtin) return self.space.newtext("<%s object at 0x%x>" % - (self.cppclass.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) + (self.clsdecl.name, rffi.cast(rffi.ULONG, self.get_rawobject()))) def destruct(self): - if self._rawobject and not self.isref: + if self._rawobject and not (self.flags & INSTANCE_FLAGS_IS_REF): memory_regulator.unregister(self) - capi.c_destruct(self.space, self.cppclass, self._rawobject) + capi.c_destruct(self.space, self.clsdecl, self._rawobject) self._rawobject = capi.C_NULL_OBJECT def _finalize_(self): - if self.python_owns: + if self.flags & INSTANCE_FLAGS_PYTHON_OWNS: self.destruct() W_CPPClass.typedef = TypeDef( 'CPPClass', - cppclass = interp_attrproperty_w('cppclass', cls=W_CPPClass), - _python_owns = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), + __python_owns__ = GetSetProperty(W_CPPClass.fget_python_owns, W_CPPClass.fset_python_owns), __init__ = interp2app(W_CPPClass.instance__init__), __eq__ = interp2app(W_CPPClass.instance__eq__), __ne__ = interp2app(W_CPPClass.instance__ne__), @@ -1220,21 +1235,21 @@ state = space.fromcache(State) return space.call_function(state.w_fngen_callback, w_callable, space.newint(npar)) -def wrap_cppobject(space, rawobject, cppclass, - do_cast=True, python_owns=False, is_ref=False, fresh=False): +def wrap_cppinstance(space, rawobject, clsdecl, + do_cast=True, python_owns=False, is_ref=False, fresh=False): rawobject = rffi.cast(capi.C_OBJECT, rawobject) # cast to actual if requested and possible w_pycppclass = None if do_cast and rawobject: - actual = capi.c_actual_class(space, cppclass, rawobject) - if actual != cppclass.handle: + actual = capi.c_actual_class(space, clsdecl, rawobject) + if actual != clsdecl.handle: try: w_pycppclass = get_pythonized_cppclass(space, actual) - offset = capi.c_base_offset1(space, actual, cppclass, rawobject, -1) + offset = capi.c_base_offset1(space, actual, clsdecl, rawobject, -1) rawobject = capi.direct_ptradd(rawobject, offset) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) + w_cppdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + clsdecl = space.interp_w(W_CPPClassDecl, w_cppdecl, can_be_None=False) except Exception: # failed to locate/build the derived class, so stick to the base (note # that only get_pythonized_cppclass is expected to raise, so none of @@ -1242,18 +1257,18 @@ pass if w_pycppclass is None: - w_pycppclass = get_pythonized_cppclass(space, cppclass.handle) + w_pycppclass = get_pythonized_cppclass(space, clsdecl.handle) # try to recycle existing object if this one is not newly created if not fresh and rawobject: obj = memory_regulator.retrieve(rawobject) - if obj is not None and obj.cppclass is cppclass: + if obj is not None and obj.clsdecl is clsdecl: return obj # fresh creation w_cppinstance = space.allocate_instance(W_CPPClass, w_pycppclass) cppinstance = space.interp_w(W_CPPClass, w_cppinstance, can_be_None=False) - cppinstance.__init__(space, cppclass, rawobject, is_ref, python_owns) + cppinstance.__init__(space, clsdecl, rawobject, is_ref, python_owns) memory_regulator.register(cppinstance) return w_cppinstance @@ -1264,7 +1279,7 @@ except TypeError: pass # attempt to get address of C++ instance - return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj)) + return rffi.cast(rffi.INTPTR_T, converter.get_rawobject(space, w_obj, False)) @unwrap_spec(w_obj=W_Root) def addressof(space, w_obj): @@ -1273,19 +1288,30 @@ return space.newlong(address) @unwrap_spec(owns=bool, cast=bool) -def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): - """Takes an address and a bound C++ class proxy, returns a bound instance.""" +def _bind_object(space, w_obj, w_clsdecl, owns=False, cast=False): try: # attempt address from array or C++ instance rawobject = rffi.cast(capi.C_OBJECT, _addressof(space, w_obj)) except Exception: # accept integer value as address rawobject = rffi.cast(capi.C_OBJECT, space.uint_w(w_obj)) - w_cppclass = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) - if not w_cppclass: - w_cppclass = scope_byname(space, space.text_w(w_pycppclass)) - if not w_cppclass: + decl = space.interp_w(W_CPPClassDecl, w_clsdecl, can_be_None=False) + return wrap_cppinstance(space, rawobject, decl, python_owns=owns, do_cast=cast) + + at unwrap_spec(owns=bool, cast=bool) +def bind_object(space, w_obj, w_pycppclass, owns=False, cast=False): + """Takes an address and a bound C++ class proxy, returns a bound instance.""" + w_clsdecl = space.findattr(w_pycppclass, space.newtext("__cppdecl__")) + if not w_clsdecl: + w_clsdecl = scope_byname(space, space.text_w(w_pycppclass)) + if not w_clsdecl: raise oefmt(space.w_TypeError, "no such class: %s", space.text_w(w_pycppclass)) - cppclass = space.interp_w(W_CPPClassDecl, w_cppclass, can_be_None=False) - return wrap_cppobject(space, rawobject, cppclass, do_cast=cast, python_owns=owns) + return _bind_object(space, w_obj, w_clsdecl, owns, cast) + +def move(space, w_obj): + """Casts the given instance into an C++-style rvalue.""" + obj = space.interp_w(W_CPPClass, w_obj, can_be_None=True) + if obj: + obj.flags |= INSTANCE_FLAGS_IS_R_VALUE + return w_obj diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py --- a/pypy/module/_cppyy/pythonify.py +++ b/pypy/module/_cppyy/pythonify.py @@ -10,7 +10,7 @@ class CPPMetaScope(type): def __getattr__(self, name): try: - return get_pycppitem(self, name) # will cache on self + return get_scoped_pycppitem(self, name) # will cache on self except Exception as e: raise AttributeError("%s object has no attribute '%s' (details: %s)" % (self, name, str(e))) @@ -36,11 +36,14 @@ self._scope = scope def _arg_to_str(self, arg): - if arg == str: - import _cppyy - arg = _cppyy._std_string_name() - elif type(arg) != str: - arg = arg.__name__ + try: + arg = arg.__cppname__ + except AttributeError: + if arg == str: + import _cppyy + arg = _cppyy._std_string_name() + elif type(arg) != str: + arg = arg.__name__ return arg def __call__(self, *args): @@ -58,8 +61,36 @@ return self.__call__(*args) -def clgen_callback(name): - return get_pycppclass(name) +def scope_splitter(name): + is_open_template, scope = 0, "" + for c in name: + if c == ':' and not is_open_template: + if scope: + yield scope + scope = "" + continue + elif c == '<': + is_open_template += 1 + elif c == '>': + is_open_template -= 1 + scope += c + yield scope + +def get_pycppitem(final_scoped_name): + # walk scopes recursively down from global namespace ("::") to get the + # actual (i.e. not typedef'ed) class, triggering all necessary creation + scope = gbl + for name in scope_splitter(final_scoped_name): + scope = getattr(scope, name) + return scope +get_pycppclass = get_pycppitem # currently no distinction, but might + # in future for performance + + +# callbacks (originating from interp_cppyy.py) to allow interp-level to +# initiate creation of app-level classes and function +def clgen_callback(final_scoped_name): + return get_pycppclass(final_scoped_name) def fngen_callback(func, npar): # todo, some kind of arg transform spec if npar == 0: @@ -75,20 +106,19 @@ return wrapper +# construction of namespaces and classes, and their helpers +def make_module_name(scope): + if scope: + return scope.__module__ + '.' + scope.__name__ + return 'cppyy' + def make_static_function(func_name, cppol): def function(*args): return cppol.call(None, *args) function.__name__ = func_name - function.__doc__ = cppol.signature() + function.__doc__ = cppol.prototype() return staticmethod(function) -def make_method(meth_name, cppol): - def method(self, *args): - return cppol.call(self, *args) - method.__name__ = meth_name - method.__doc__ = cppol.signature() - return method - def make_cppnamespace(scope, name, decl): # build up a representation of a C++ namespace (namespaces are classes) @@ -98,20 +128,19 @@ ns_meta = type(name+'_meta', (CPPMetaNamespace,), {}) # create the python-side C++ namespace representation, cache in scope if given - d = {"__cppdecl__" : decl, "__cppname__" : decl.__cppname__ } + d = {"__cppdecl__" : decl, + "__module__" : make_module_name(scope), + "__cppname__" : decl.__cppname__ } pyns = ns_meta(name, (CPPNamespace,), d) if scope: setattr(scope, name, pyns) # install as modules to allow importing from (note naming: cppyy) - modname = 'cppyy.gbl' - if scope: - modname = 'cppyy.gbl.'+pyns.__cppname__.replace('::', '.') - sys.modules[modname] = pyns + sys.modules[make_module_name(pyns)] = pyns return pyns def _drop_cycles(bases): - # TODO: figure this out, as it seems to be a PyPy bug?! + # TODO: figure out why this is necessary? for b1 in bases: for b2 in bases: if not (b1 is b2) and issubclass(b2, b1): @@ -119,27 +148,37 @@ break return tuple(bases) -def make_new(class_name): + +def make_new(decl): def __new__(cls, *args): # create a place-holder only as there may be a derived class defined + # TODO: get rid of the import and add user-land bind_object that uses + # _bind_object (see interp_cppyy.py) import _cppyy - instance = _cppyy.bind_object(0, class_name, True) + instance = _cppyy._bind_object(0, decl, True) if not instance.__class__ is cls: instance.__class__ = cls # happens for derived class return instance return __new__ -def make_cppclass(scope, class_name, final_class_name, decl): +def make_method(meth_name, cppol): + def method(self, *args): + return cppol.call(self, *args) + method.__name__ = meth_name + method.__doc__ = cppol.prototype() + return method + +def make_cppclass(scope, cl_name, decl): # get a list of base classes for class creation bases = [get_pycppclass(base) for base in decl.get_base_names()] if not bases: bases = [CPPClass,] else: - # it's technically possible that the required class now has been built - # if one of the base classes uses it in e.g. a function interface + # it's possible that the required class now has been built if one of + # the base classes uses it in e.g. a function interface try: - return scope.__dict__[final_class_name] + return scope.__dict__[cl_name] except KeyError: pass @@ -147,39 +186,41 @@ d_meta = {} # prepare dictionary for python-side C++ class representation - def dispatch(self, name, signature): - cppol = decl.dispatch(name, signature) - return types.MethodType(make_method(name, cppol), self, type(self)) + def dispatch(self, m_name, signature): + cppol = decl.__dispatch__(m_name, signature) + return types.MethodType(make_method(m_name, cppol), self, type(self)) d_class = {"__cppdecl__" : decl, + "__new__" : make_new(decl), + "__module__" : make_module_name(scope), "__cppname__" : decl.__cppname__, - "__new__" : make_new(class_name), + "__dispatch__" : dispatch, } # insert (static) methods into the class dictionary - for name in decl.get_method_names(): - cppol = decl.get_overload(name) + for m_name in decl.get_method_names(): + cppol = decl.get_overload(m_name) if cppol.is_static(): - d_class[name] = make_static_function(name, cppol) + d_class[m_name] = make_static_function(m_name, cppol) else: - d_class[name] = make_method(name, cppol) + d_class[m_name] = make_method(m_name, cppol) # add all data members to the dictionary of the class to be created, and # static ones also to the metaclass (needed for property setters) - for name in decl.get_datamember_names(): - cppdm = decl.get_datamember(name) - d_class[name] = cppdm + for d_name in decl.get_datamember_names(): + cppdm = decl.get_datamember(d_name) + d_class[d_name] = cppdm if cppdm.is_static(): - d_meta[name] = cppdm + d_meta[d_name] = cppdm # create a metaclass to allow properties (for static data write access) metabases = [type(base) for base in bases] - metacpp = type(CPPMetaScope)(class_name+'_meta', _drop_cycles(metabases), d_meta) + metacpp = type(CPPMetaScope)(cl_name+'_meta', _drop_cycles(metabases), d_meta) # create the python-side C++ class - pycls = metacpp(class_name, _drop_cycles(bases), d_class) + pycls = metacpp(cl_name, _drop_cycles(bases), d_class) # store the class on its outer scope - setattr(scope, final_class_name, pycls) + setattr(scope, cl_name, pycls) # the call to register will add back-end specific pythonizations and thus # needs to run first, so that the generic pythonizations can use them @@ -192,32 +233,32 @@ return CPPTemplate(template_name, scope) -def get_pycppitem(scope, name): +def get_scoped_pycppitem(scope, name): import _cppyy - # resolve typedefs/aliases - full_name = (scope == gbl) and name or (scope.__name__+'::'+name) - true_name = _cppyy._resolve_name(full_name) - if true_name != full_name: - return get_pycppclass(true_name) + # resolve typedefs/aliases: these may cross namespaces, in which case + # the lookup must trigger the creation of all necessary scopes + scoped_name = (scope == gbl) and name or (scope.__cppname__+'::'+name) + final_scoped_name = _cppyy._resolve_name(scoped_name) + if final_scoped_name != scoped_name: + pycppitem = get_pycppitem(final_scoped_name) + # also store on the requested scope (effectively a typedef or pointer copy) + setattr(scope, name, pycppitem) + return pycppitem pycppitem = None - # classes - cppitem = _cppyy._scope_byname(true_name) + # scopes (classes and namespaces) + cppitem = _cppyy._scope_byname(final_scoped_name) if cppitem: - name = true_name - if scope != gbl: - name = true_name[len(scope.__cppname__)+2:] if cppitem.is_namespace(): pycppitem = make_cppnamespace(scope, name, cppitem) - setattr(scope, name, pycppitem) else: - pycppitem = make_cppclass(scope, name, true_name, cppitem) + pycppitem = make_cppclass(scope, name, cppitem) # templates if not cppitem: - cppitem = _cppyy._template_byname(true_name) + cppitem = _cppyy._is_template(final_scoped_name) if cppitem: pycppitem = make_cpptemplatetype(scope, name) setattr(scope, name, pycppitem) @@ -249,29 +290,6 @@ raise AttributeError("'%s' has no attribute '%s'" % (str(scope), name)) -def scope_splitter(name): - is_open_template, scope = 0, "" - for c in name: - if c == ':' and not is_open_template: - if scope: - yield scope - scope = "" - continue - elif c == '<': - is_open_template += 1 - elif c == '>': - is_open_template -= 1 - scope += c - yield scope - -def get_pycppclass(name): - # break up the name, to walk the scopes and get the class recursively - scope = gbl - for part in scope_splitter(name): - scope = getattr(scope, part) - return scope - - # pythonization by decoration (move to their own file?) def python_style_getitem(self, idx): # python-style indexing: check for size and allow indexing from the back @@ -346,8 +364,8 @@ # also the fallback on the indexed __getitem__, but that is slower) if not 'vector' in pyclass.__name__[:11] and \ ('begin' in pyclass.__dict__ and 'end' in pyclass.__dict__): - if _cppyy._scope_byname(pyclass.__name__+'::iterator') or \ - _cppyy._scope_byname(pyclass.__name__+'::const_iterator'): + if _cppyy._scope_byname(pyclass.__cppname__+'::iterator') or \ + _cppyy._scope_byname(pyclass.__cppname__+'::const_iterator'): def __iter__(self): i = self.begin() while i != self.end(): @@ -416,17 +434,21 @@ # pre-create std to allow direct importing gbl.std = make_cppnamespace(gbl, 'std', _cppyy._scope_byname('std')) + # add move cast + gbl.std.move = _cppyy.move + # install a type for enums to refer to # TODO: this is correct for C++98, not for C++11 and in general there will # be the same issue for all typedef'd builtin types setattr(gbl, 'internal_enum_type_t', int) - # install nullptr as a unique reference - setattr(gbl, 'nullptr', _cppyy._get_nullptr()) - # install for user access _cppyy.gbl = gbl + # install nullptr as a unique reference + _cppyy.nullptr = _cppyy._get_nullptr() + + # user-defined pythonizations interface _pythonizations = {} def add_pythonization(class_name, callback): diff --git a/pypy/module/_cppyy/src/dummy_backend.cxx b/pypy/module/_cppyy/src/dummy_backend.cxx --- a/pypy/module/_cppyy/src/dummy_backend.cxx +++ b/pypy/module/_cppyy/src/dummy_backend.cxx @@ -955,7 +955,13 @@ return cppstring_to_cstring(""); } -char* cppyy_method_signature(cppyy_scope_t /* handle */, cppyy_index_t /* method_index */) { +char* cppyy_method_signature( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { + return cppstring_to_cstring(""); +} + +char* cppyy_method_prototype( + cppyy_scope_t /* handle */, cppyy_index_t /* method_index */, int /* show_formalargs */) { return cppstring_to_cstring(""); } diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile --- a/pypy/module/_cppyy/test/Makefile +++ b/pypy/module/_cppyy/test/Makefile @@ -1,12 +1,14 @@ dicts = advancedcppDict.so \ advancedcpp2Dict.so \ + cpp11featuresDict.so \ crossingDict.so \ datatypesDict.so \ example01Dict.so \ fragileDict.so \ operatorsDict.so \ overloadsDict.so \ - stltypesDict.so + stltypesDict.so \ + templatesDict.so all : $(dicts) diff --git a/pypy/module/_cppyy/test/advancedcpp.cxx b/pypy/module/_cppyy/test/advancedcpp.cxx --- a/pypy/module/_cppyy/test/advancedcpp.cxx +++ b/pypy/module/_cppyy/test/advancedcpp.cxx @@ -106,17 +106,6 @@ } -// more template testing -long my_templated_method_class::get_size() { return -1; } - -long my_templated_method_class::get_char_size() { return (long)sizeof(char); } -long my_templated_method_class::get_int_size() { return (long)sizeof(int); } -long my_templated_method_class::get_long_size() { return (long)sizeof(long); } -long my_templated_method_class::get_float_size() { return (long)sizeof(float); } -long my_templated_method_class::get_double_size() { return (long)sizeof(double); } -long my_templated_method_class::get_self_size() { return (long)sizeof(my_templated_method_class); } - - // overload order testing int overload_one_way::gime() const { return 1; } std::string overload_one_way::gime() { return "aap"; } diff --git a/pypy/module/_cppyy/test/advancedcpp.h b/pypy/module/_cppyy/test/advancedcpp.h --- a/pypy/module/_cppyy/test/advancedcpp.h +++ b/pypy/module/_cppyy/test/advancedcpp.h @@ -246,8 +246,6 @@ int m_i; }; -template class std::vector; - //=========================================================================== class some_convertible { // for math conversions testing @@ -275,6 +273,7 @@ extern double my_global_double; // a couple of globals for access testing extern double my_global_array[500]; extern double* my_global_ptr; +static const char my_global_string[] = "aap " " noot " " mies"; //=========================================================================== class some_class_with_data { // for life-line and identity testing @@ -387,37 +386,6 @@ template char my_templated_function(char); template double my_templated_function(double); -class my_templated_method_class { -public: - long get_size(); // to get around bug in genreflex - template long get_size(); - - long get_char_size(); - long get_int_size(); - long get_long_size(); - long get_float_size(); - long get_double_size(); - - long get_self_size(); - -private: - double m_data[3]; -}; - -template -inline long my_templated_method_class::get_size() { - return sizeof(B); -} - -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); -template long my_templated_method_class::get_size(); - -typedef my_templated_method_class my_typedef_t; -template long my_templated_method_class::get_size(); - //=========================================================================== class overload_one_way { // overload order testing diff --git a/pypy/module/_cppyy/test/advancedcpp.xml b/pypy/module/_cppyy/test/advancedcpp.xml --- a/pypy/module/_cppyy/test/advancedcpp.xml +++ b/pypy/module/_cppyy/test/advancedcpp.xml @@ -53,8 +53,6 @@ - - diff --git a/pypy/module/_cppyy/test/cpp11features.cxx b/pypy/module/_cppyy/test/cpp11features.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.cxx @@ -0,0 +1,18 @@ +#if __cplusplus >= 201103L + +#include "cpp11features.h" + + +// for std::shared_ptr<> testing +int TestSharedPtr::s_counter = 0; + +std::shared_ptr create_shared_ptr_instance() { + return std::shared_ptr(new TestSharedPtr); +} + + +// for move ctors etc. +int TestMoving1::s_move_counter = 0; +int TestMoving2::s_move_counter = 0; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.h b/pypy/module/_cppyy/test/cpp11features.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.h @@ -0,0 +1,45 @@ +#if __cplusplus >= 201103L + +#include + + +//=========================================================================== +class TestSharedPtr { // for std::shared_ptr<> testing +public: + static int s_counter; + +public: + TestSharedPtr() { ++s_counter; } + TestSharedPtr(const TestSharedPtr&) { ++s_counter; } + ~TestSharedPtr() { --s_counter; } +}; + +std::shared_ptr create_shared_ptr_instance(); + + +//=========================================================================== +class TestMoving1 { // for move ctors etc. +public: + static int s_move_counter; + +public: + TestMoving1() {} + TestMoving1(TestMoving1&&) { ++s_move_counter; } + TestMoving1(const TestMoving1&) {} + TestMoving1& operator=(TestMoving1&&) { ++s_move_counter; return *this; } + TestMoving1& operator=(TestMoving1&) { return *this; } +}; + +class TestMoving2 { // note opposite method order from TestMoving1 +public: + static int s_move_counter; + +public: + TestMoving2() {} + TestMoving2(const TestMoving2&) {} + TestMoving2(TestMoving2&& other) { ++s_move_counter; } + TestMoving2& operator=(TestMoving2&) { return *this; } + TestMoving2& operator=(TestMoving2&&) { ++s_move_counter; return *this; } +}; + +#endif // c++11 and later diff --git a/pypy/module/_cppyy/test/cpp11features.xml b/pypy/module/_cppyy/test/cpp11features.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/cpp11features.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/fragile.h b/pypy/module/_cppyy/test/fragile.h --- a/pypy/module/_cppyy/test/fragile.h +++ b/pypy/module/_cppyy/test/fragile.h @@ -30,9 +30,11 @@ void overload(int, no_such_class* p = 0) {} }; +static const int dummy_location = 0xdead; + class E { public: - E() : m_pp_no_such(0), m_pp_a(0) {} + E() : m_pp_no_such((no_such_class**)&dummy_location), m_pp_a(0) {} virtual int check() { return (int)'E'; } void overload(no_such_class**) {} diff --git a/pypy/module/_cppyy/test/templates.cxx b/pypy/module/_cppyy/test/templates.cxx new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.cxx @@ -0,0 +1,12 @@ +#include "templates.h" + + +// template methods +long MyTemplatedMethodClass::get_size() { return -1; } + +long MyTemplatedMethodClass::get_char_size() { return (long)sizeof(char); } +long MyTemplatedMethodClass::get_int_size() { return (long)sizeof(int); } +long MyTemplatedMethodClass::get_long_size() { return (long)42; /* "lying" */ } +long MyTemplatedMethodClass::get_float_size() { return (long)sizeof(float); } +long MyTemplatedMethodClass::get_double_size() { return (long)sizeof(double); } +long MyTemplatedMethodClass::get_self_size() { return (long)sizeof(MyTemplatedMethodClass); } diff --git a/pypy/module/_cppyy/test/templates.h b/pypy/module/_cppyy/test/templates.h new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.h @@ -0,0 +1,35 @@ +//=========================================================================== +class MyTemplatedMethodClass { // template methods +public: + long get_size(); // to get around bug in genreflex + template long get_size(); + + long get_char_size(); + long get_int_size(); + long get_long_size(); + long get_float_size(); + long get_double_size(); + + long get_self_size(); + +private: + double m_data[3]; +}; + +template +inline long MyTemplatedMethodClass::get_size() { + return sizeof(B); +} + +// +typedef MyTemplatedMethodClass MyTMCTypedef_t; + +// explicit instantiation +template long MyTemplatedMethodClass::get_size(); +template long MyTemplatedMethodClass::get_size(); + +// "lying" specialization +template<> +inline long MyTemplatedMethodClass::get_size() { + return 42; +} diff --git a/pypy/module/_cppyy/test/templates.xml b/pypy/module/_cppyy/test/templates.xml new file mode 100644 --- /dev/null +++ b/pypy/module/_cppyy/test/templates.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/pypy/module/_cppyy/test/test_advancedcpp.py b/pypy/module/_cppyy/test/test_advancedcpp.py --- a/pypy/module/_cppyy/test/test_advancedcpp.py +++ b/pypy/module/_cppyy/test/test_advancedcpp.py @@ -28,9 +28,9 @@ def test01_default_arguments(self): """Test usage of default arguments""" - import _cppyy + import _cppyy as cppyy def test_defaulter(n, t): - defaulter = getattr(_cppyy.gbl, '%s_defaulter' % n) + defaulter = getattr(cppyy.gbl, '%s_defaulter' % n) d = defaulter() assert d.m_a == t(11) @@ -55,23 +55,23 @@ assert d.m_b == t(4) assert d.m_c == t(5) d.__destruct__() - test_defaulter('short', int) + test_defaulter('short', int) test_defaulter('ushort', int) - test_defaulter('int', int) - test_defaulter('uint', int) - test_defaulter('long', long) - test_defaulter('ulong', long) - test_defaulter('llong', long) + test_defaulter('int', int) + test_defaulter('uint', int) + test_defaulter('long', long) + test_defaulter('ulong', long) + test_defaulter('llong', long) test_defaulter('ullong', long) - test_defaulter('float', float) + test_defaulter('float', float) test_defaulter('double', float) def test02_simple_inheritance(self): """Test binding of a basic inheritance structure""" - import _cppyy - base_class = _cppyy.gbl.base_class - derived_class = _cppyy.gbl.derived_class + import _cppyy as cppyy + base_class = cppyy.gbl.base_class + derived_class = cppyy.gbl.derived_class assert issubclass(derived_class, base_class) assert not issubclass(base_class, derived_class) @@ -123,8 +123,8 @@ def test03_namespaces(self): """Test access to namespaces and inner classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.a_ns is gbl.a_ns assert gbl.a_ns.d_ns is gbl.a_ns.d_ns @@ -150,8 +150,8 @@ def test03a_namespace_lookup_on_update(self): """Test whether namespaces can be shared across dictionaries.""" - import _cppyy, ctypes - gbl = _cppyy.gbl + import _cppyy as cppyy, ctypes + gbl = cppyy.gbl lib2 = ctypes.CDLL("./advancedcpp2Dict.so", ctypes.RTLD_GLOBAL) @@ -179,8 +179,8 @@ def test04_template_types(self): """Test bindings of templated types""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl assert gbl.T1 is gbl.T1 assert gbl.T2 is gbl.T2 @@ -245,8 +245,8 @@ def test05_abstract_classes(self): """Test non-instatiatability of abstract classes""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl raises(TypeError, gbl.a_class) raises(TypeError, gbl.some_abstract_class) @@ -260,12 +260,12 @@ def test06_datamembers(self): """Test data member access when using virtual inheritence""" - import _cppyy - a_class = _cppyy.gbl.a_class - b_class = _cppyy.gbl.b_class - c_class_1 = _cppyy.gbl.c_class_1 - c_class_2 = _cppyy.gbl.c_class_2 - d_class = _cppyy.gbl.d_class + import _cppyy as cppyy + a_class = cppyy.gbl.a_class + b_class = cppyy.gbl.b_class + c_class_1 = cppyy.gbl.c_class_1 + c_class_2 = cppyy.gbl.c_class_2 + d_class = cppyy.gbl.d_class assert issubclass(b_class, a_class) assert issubclass(c_class_1, a_class) @@ -354,8 +354,8 @@ def test07_pass_by_reference(self): """Test reference passing when using virtual inheritance""" - import _cppyy - gbl = _cppyy.gbl + import _cppyy as cppyy + gbl = cppyy.gbl b_class = gbl.b_class c_class = gbl.c_class_2 d_class = gbl.d_class @@ -387,71 +387,75 @@ def test08_void_pointer_passing(self): """Test passing of variants of void pointer arguments""" - import _cppyy - pointer_pass = _cppyy.gbl.pointer_pass - some_concrete_class = _cppyy.gbl.some_concrete_class + import _cppyy as cppyy + pointer_pass = cppyy.gbl.pointer_pass + some_concrete_class = cppyy.gbl.some_concrete_class pp = pointer_pass() o = some_concrete_class() - assert _cppyy.addressof(o) == pp.gime_address_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) - assert _cppyy.addressof(o) == pp.gime_address_ptr_ref(o) + assert cppyy.addressof(o) == pp.gime_address_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ptr(o) + assert cppyy.addressof(o) == pp.gime_address_ptr_ref(o) import array - addressofo = array.array('l', [_cppyy.addressof(o)]) - assert addressofo.buffer_info()[0] == pp.gime_address_ptr_ptr(addressofo) + addressofo = array.array('l', [cppyy.addressof(o)]) + assert addressofo[0] == pp.gime_address_ptr_ptr(addressofo) assert 0 == pp.gime_address_ptr(0) - assert 0 == pp.gime_address_ptr(None) + raises(TypeError, pp.gime_address_ptr, None) - ptr = _cppyy.bind_object(0, some_concrete_class) - assert _cppyy.addressof(ptr) == 0 + ptr = cppyy.bind_object(0, some_concrete_class) + assert cppyy.addressof(ptr) == 0 pp.set_address_ptr_ref(ptr) - assert _cppyy.addressof(ptr) == 0x1234 + assert cppyy.addressof(ptr) == 0x1234 pp.set_address_ptr_ptr(ptr) - assert _cppyy.addressof(ptr) == 0x4321 + assert cppyy.addressof(ptr) == 0x4321 + + assert cppyy.addressof(cppyy.nullptr) == 0 + raises(TypeError, cppyy.addressof, None) From pypy.commits at gmail.com Mon Oct 30 13:12:39 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 30 Oct 2017 10:12:39 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: fix merge quirks, document merged branch Message-ID: <59f75d87.0c181c0a.5626d.347a@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92886:81a7e5dcbc2d Date: 2017-10-30 19:10 +0200 http://bitbucket.org/pypy/pypy/changeset/81a7e5dcbc2d/ Log: fix merge quirks, document merged branch diff --git a/pypy/doc/whatsnew-pypy3-head.rst b/pypy/doc/whatsnew-pypy3-head.rst --- a/pypy/doc/whatsnew-pypy3-head.rst +++ b/pypy/doc/whatsnew-pypy3-head.rst @@ -11,3 +11,6 @@ .. branch: py3.5-appexec Raise if space.is_true(space.appexec()) used in app level tests, fix tests that did this + +.. branch: py3.5-mac-embedding +Download and patch dependencies when building cffi-based stdlib modules diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -1864,7 +1864,7 @@ def sched_yield(): return handle_posix_error('sched_yield', c_sched_yield()) - + #___________________________________________________________________ c_chroot = external('chroot', [rffi.CCHARP], rffi.INT, diff --git a/rpython/rlib/test/test_rposix.py b/rpython/rlib/test/test_rposix.py --- a/rpython/rlib/test/test_rposix.py +++ b/rpython/rlib/test/test_rposix.py @@ -810,11 +810,6 @@ assert isinstance(low, int) == True assert isinstance(high, int) == True assert high > low - - at rposix_requires('sched_yield') -def test_sched_yield(): - if sys.platform != 'win32': - rposix.sched_yield() @rposix_requires('sched_yield') def test_sched_yield(): From pypy.commits at gmail.com Mon Oct 30 13:12:33 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 30 Oct 2017 10:12:33 -0700 (PDT) Subject: [pypy-commit] pypy py3.5: merge py3.5-mac-embedding which can download stdlib cffi-module build depenencies Message-ID: <59f75d81.4a061c0a.23c83.f357@mx.google.com> Author: Matti Picus Branch: py3.5 Changeset: r92883:5a955fc2e868 Date: 2017-10-30 18:39 +0200 http://bitbucket.org/pypy/pypy/changeset/5a955fc2e868/ Log: merge py3.5-mac-embedding which can download stdlib cffi-module build depenencies diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -59,6 +59,7 @@ ^rpython/rlib/rvmprof/src/shared/libbacktrace/config.h$ ^rpython/rlib/rvmprof/src/shared/libbacktrace/config.log$ ^rpython/rlib/rvmprof/src/shared/libbacktrace/config.status$ +^pypy/tool/dest$ ^pypy/goal/pypy-translation-snapshot$ ^pypy/goal/pypy-c ^pypy/goal/pypy3-c diff --git a/lib_pypy/_cffi_ssl/_cffi_src/openssl/cryptography.py b/lib_pypy/_cffi_ssl/_cffi_src/openssl/cryptography.py --- a/lib_pypy/_cffi_ssl/_cffi_src/openssl/cryptography.py +++ b/lib_pypy/_cffi_ssl/_cffi_src/openssl/cryptography.py @@ -48,6 +48,9 @@ #else #define CRYPTOGRAPHY_IS_LIBRESSL 0 #endif + +#define CRYPTOGRAPHY_LIBRESSL_251_OR_GREATER \ + (CRYPTOGRAPHY_IS_LIBRESSL && LIBRESSL_VERSION_NUMBER >= 0x20501000) """ TYPES = """ diff --git a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py --- a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py +++ b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py @@ -244,6 +244,14 @@ static const long X509_V_FLAG_SUITEB_192_LOS = 0; static const long X509_V_FLAG_SUITEB_128_LOS = 0; +#if CRYPTOGRAPHY_LIBRESSL_251_OR_GREATER +int X509_VERIFY_PARAM_set1_host(X509_VERIFY_PARAM *, const char *, size_t); +int X509_VERIFY_PARAM_set1_email(X509_VERIFY_PARAM *, const char *, size_t); +int X509_VERIFY_PARAM_set1_ip(X509_VERIFY_PARAM *, const unsigned char *, + size_t); +int X509_VERIFY_PARAM_set1_ip_asc(X509_VERIFY_PARAM *, const char *); +void X509_VERIFY_PARAM_set_hostflags(X509_VERIFY_PARAM *, unsigned int); +#else int (*X509_VERIFY_PARAM_set1_host)(X509_VERIFY_PARAM *, const char *, size_t) = NULL; int (*X509_VERIFY_PARAM_set1_email)(X509_VERIFY_PARAM *, const char *, @@ -254,6 +262,7 @@ void (*X509_VERIFY_PARAM_set_hostflags)(X509_VERIFY_PARAM *, unsigned int) = NULL; #endif +#endif /* OpenSSL 1.0.2+ or Solaris's backport */ #ifdef X509_V_FLAG_PARTIAL_CHAIN diff --git a/lib_pypy/_cffi_ssl/osx-roots.diff b/lib_pypy/_cffi_ssl/osx-roots.diff new file mode 100644 --- /dev/null +++ b/lib_pypy/_cffi_ssl/osx-roots.diff @@ -0,0 +1,475 @@ +diff -Naur libressl-2.6.2.orig/crypto/Makefile.am libressl-2.6.2/crypto/Makefile.am +--- libressl-2.6.2.orig/crypto/Makefile.am 2017-09-02 01:49:55.000000000 +0200 ++++ libressl-2.6.2/crypto/Makefile.am 2017-10-07 14:05:16.000000000 +0200 +@@ -92,7 +92,7 @@ + -mv crypto_portable.sym.tmp crypto_portable.sym + endif + +-libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym ++libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym -framework Security -framework CoreFoundation + libcrypto_la_LIBADD = libcompat.la + if !HAVE_EXPLICIT_BZERO + libcrypto_la_LIBADD += libcompatnoopt.la +@@ -863,6 +863,7 @@ + libcrypto_la_SOURCES += x509/x509_txt.c + libcrypto_la_SOURCES += x509/x509_v3.c + libcrypto_la_SOURCES += x509/x509_vfy.c ++libcrypto_la_SOURCES += x509/x509_vfy_apple.c + libcrypto_la_SOURCES += x509/x509_vpm.c + libcrypto_la_SOURCES += x509/x509cset.c + libcrypto_la_SOURCES += x509/x509name.c +diff -Naur libressl-2.6.2.orig/crypto/Makefile.in libressl-2.6.2/crypto/Makefile.in +--- libressl-2.6.2.orig/crypto/Makefile.in 2017-09-26 06:07:03.000000000 +0200 ++++ libressl-2.6.2/crypto/Makefile.in 2017-10-07 14:05:24.000000000 +0200 +@@ -426,20 +426,20 @@ + x509/x509_err.c x509/x509_ext.c x509/x509_lu.c x509/x509_obj.c \ + x509/x509_r2x.c x509/x509_req.c x509/x509_set.c \ + x509/x509_trs.c x509/x509_txt.c x509/x509_v3.c x509/x509_vfy.c \ +- x509/x509_vpm.c x509/x509cset.c x509/x509name.c \ +- x509/x509rset.c x509/x509spki.c x509/x509type.c x509/x_all.c \ +- x509v3/pcy_cache.c x509v3/pcy_data.c x509v3/pcy_lib.c \ +- x509v3/pcy_map.c x509v3/pcy_node.c x509v3/pcy_tree.c \ +- x509v3/v3_akey.c x509v3/v3_akeya.c x509v3/v3_alt.c \ +- x509v3/v3_bcons.c x509v3/v3_bitst.c x509v3/v3_conf.c \ +- x509v3/v3_cpols.c x509v3/v3_crld.c x509v3/v3_enum.c \ +- x509v3/v3_extku.c x509v3/v3_genn.c x509v3/v3_ia5.c \ +- x509v3/v3_info.c x509v3/v3_int.c x509v3/v3_lib.c \ +- x509v3/v3_ncons.c x509v3/v3_ocsp.c x509v3/v3_pci.c \ +- x509v3/v3_pcia.c x509v3/v3_pcons.c x509v3/v3_pku.c \ +- x509v3/v3_pmaps.c x509v3/v3_prn.c x509v3/v3_purp.c \ +- x509v3/v3_skey.c x509v3/v3_sxnet.c x509v3/v3_utl.c \ +- x509v3/v3err.c ++ x509/x509_vfy_apple.c x509/x509_vpm.c x509/x509cset.c \ ++ x509/x509name.c x509/x509rset.c x509/x509spki.c \ ++ x509/x509type.c x509/x_all.c x509v3/pcy_cache.c \ ++ x509v3/pcy_data.c x509v3/pcy_lib.c x509v3/pcy_map.c \ ++ x509v3/pcy_node.c x509v3/pcy_tree.c x509v3/v3_akey.c \ ++ x509v3/v3_akeya.c x509v3/v3_alt.c x509v3/v3_bcons.c \ ++ x509v3/v3_bitst.c x509v3/v3_conf.c x509v3/v3_cpols.c \ ++ x509v3/v3_crld.c x509v3/v3_enum.c x509v3/v3_extku.c \ ++ x509v3/v3_genn.c x509v3/v3_ia5.c x509v3/v3_info.c \ ++ x509v3/v3_int.c x509v3/v3_lib.c x509v3/v3_ncons.c \ ++ x509v3/v3_ocsp.c x509v3/v3_pci.c x509v3/v3_pcia.c \ ++ x509v3/v3_pcons.c x509v3/v3_pku.c x509v3/v3_pmaps.c \ ++ x509v3/v3_prn.c x509v3/v3_purp.c x509v3/v3_skey.c \ ++ x509v3/v3_sxnet.c x509v3/v3_utl.c x509v3/v3err.c + am__objects_27 = aes/libcrypto_la-aes-elf-x86_64.lo \ + aes/libcrypto_la-bsaes-elf-x86_64.lo \ + aes/libcrypto_la-vpaes-elf-x86_64.lo \ +@@ -759,11 +759,12 @@ + x509/libcrypto_la-x509_r2x.lo x509/libcrypto_la-x509_req.lo \ + x509/libcrypto_la-x509_set.lo x509/libcrypto_la-x509_trs.lo \ + x509/libcrypto_la-x509_txt.lo x509/libcrypto_la-x509_v3.lo \ +- x509/libcrypto_la-x509_vfy.lo x509/libcrypto_la-x509_vpm.lo \ +- x509/libcrypto_la-x509cset.lo x509/libcrypto_la-x509name.lo \ +- x509/libcrypto_la-x509rset.lo x509/libcrypto_la-x509spki.lo \ +- x509/libcrypto_la-x509type.lo x509/libcrypto_la-x_all.lo \ +- x509v3/libcrypto_la-pcy_cache.lo \ ++ x509/libcrypto_la-x509_vfy.lo \ ++ x509/libcrypto_la-x509_vfy_apple.lo \ ++ x509/libcrypto_la-x509_vpm.lo x509/libcrypto_la-x509cset.lo \ ++ x509/libcrypto_la-x509name.lo x509/libcrypto_la-x509rset.lo \ ++ x509/libcrypto_la-x509spki.lo x509/libcrypto_la-x509type.lo \ ++ x509/libcrypto_la-x_all.lo x509v3/libcrypto_la-pcy_cache.lo \ + x509v3/libcrypto_la-pcy_data.lo x509v3/libcrypto_la-pcy_lib.lo \ + x509v3/libcrypto_la-pcy_map.lo x509v3/libcrypto_la-pcy_node.lo \ + x509v3/libcrypto_la-pcy_tree.lo x509v3/libcrypto_la-v3_akey.lo \ +@@ -1000,7 +1001,7 @@ + $(ASM_X86_64_ELF) $(ASM_X86_64_MACOSX) + BUILT_SOURCES = crypto_portable.sym + CLEANFILES = crypto_portable.sym +-libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym ++libcrypto_la_LDFLAGS = -version-info @LIBCRYPTO_VERSION@ -no-undefined -export-symbols crypto_portable.sym -framework Security -framework CoreFoundation + libcrypto_la_LIBADD = libcompat.la $(am__append_1) + libcrypto_la_CPPFLAGS = $(AM_CPPFLAGS) -DLIBRESSL_INTERNAL \ + -DOPENSSL_NO_HW_PADLOCK $(am__append_2) $(am__append_3) \ +@@ -1272,20 +1273,20 @@ + x509/x509_err.c x509/x509_ext.c x509/x509_lu.c x509/x509_obj.c \ + x509/x509_r2x.c x509/x509_req.c x509/x509_set.c \ + x509/x509_trs.c x509/x509_txt.c x509/x509_v3.c x509/x509_vfy.c \ +- x509/x509_vpm.c x509/x509cset.c x509/x509name.c \ +- x509/x509rset.c x509/x509spki.c x509/x509type.c x509/x_all.c \ +- x509v3/pcy_cache.c x509v3/pcy_data.c x509v3/pcy_lib.c \ +- x509v3/pcy_map.c x509v3/pcy_node.c x509v3/pcy_tree.c \ +- x509v3/v3_akey.c x509v3/v3_akeya.c x509v3/v3_alt.c \ +- x509v3/v3_bcons.c x509v3/v3_bitst.c x509v3/v3_conf.c \ +- x509v3/v3_cpols.c x509v3/v3_crld.c x509v3/v3_enum.c \ +- x509v3/v3_extku.c x509v3/v3_genn.c x509v3/v3_ia5.c \ +- x509v3/v3_info.c x509v3/v3_int.c x509v3/v3_lib.c \ +- x509v3/v3_ncons.c x509v3/v3_ocsp.c x509v3/v3_pci.c \ +- x509v3/v3_pcia.c x509v3/v3_pcons.c x509v3/v3_pku.c \ +- x509v3/v3_pmaps.c x509v3/v3_prn.c x509v3/v3_purp.c \ +- x509v3/v3_skey.c x509v3/v3_sxnet.c x509v3/v3_utl.c \ +- x509v3/v3err.c ++ x509/x509_vfy_apple.c x509/x509_vpm.c x509/x509cset.c \ ++ x509/x509name.c x509/x509rset.c x509/x509spki.c \ ++ x509/x509type.c x509/x_all.c x509v3/pcy_cache.c \ ++ x509v3/pcy_data.c x509v3/pcy_lib.c x509v3/pcy_map.c \ ++ x509v3/pcy_node.c x509v3/pcy_tree.c x509v3/v3_akey.c \ ++ x509v3/v3_akeya.c x509v3/v3_alt.c x509v3/v3_bcons.c \ ++ x509v3/v3_bitst.c x509v3/v3_conf.c x509v3/v3_cpols.c \ ++ x509v3/v3_crld.c x509v3/v3_enum.c x509v3/v3_extku.c \ ++ x509v3/v3_genn.c x509v3/v3_ia5.c x509v3/v3_info.c \ ++ x509v3/v3_int.c x509v3/v3_lib.c x509v3/v3_ncons.c \ ++ x509v3/v3_ocsp.c x509v3/v3_pci.c x509v3/v3_pcia.c \ ++ x509v3/v3_pcons.c x509v3/v3_pku.c x509v3/v3_pmaps.c \ ++ x509v3/v3_prn.c x509v3/v3_purp.c x509v3/v3_skey.c \ ++ x509v3/v3_sxnet.c x509v3/v3_utl.c x509v3/v3err.c + + # chacha + +@@ -2808,6 +2809,8 @@ + x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509_vfy.lo: x509/$(am__dirstamp) \ + x509/$(DEPDIR)/$(am__dirstamp) ++x509/libcrypto_la-x509_vfy_apple.lo: x509/$(am__dirstamp) \ ++ x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509_vpm.lo: x509/$(am__dirstamp) \ + x509/$(DEPDIR)/$(am__dirstamp) + x509/libcrypto_la-x509cset.lo: x509/$(am__dirstamp) \ +@@ -3583,6 +3586,7 @@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_txt.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_v3.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vfy.Plo at am__quote@ ++ at AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509_vpm.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509cset.Plo at am__quote@ + @AMDEP_TRUE@@am__include@ @am__quote at x509/$(DEPDIR)/libcrypto_la-x509name.Plo at am__quote@ +@@ -7460,6 +7464,13 @@ + @AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ + @am__fastdepCC_FALSE@ $(AM_V_CC at am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o x509/libcrypto_la-x509_vfy.lo `test -f 'x509/x509_vfy.c' || echo '$(srcdir)/'`x509/x509_vfy.c + ++x509/libcrypto_la-x509_vfy_apple.lo: x509/x509_vfy_apple.c ++ at am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT x509/libcrypto_la-x509_vfy_apple.lo -MD -MP -MF x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Tpo -c -o x509/libcrypto_la-x509_vfy_apple.lo `test -f 'x509/x509_vfy_apple.c' || echo '$(srcdir)/'`x509/x509_vfy_apple.c ++ at am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Tpo x509/$(DEPDIR)/libcrypto_la-x509_vfy_apple.Plo ++ at AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='x509/x509_vfy_apple.c' object='x509/libcrypto_la-x509_vfy_apple.lo' libtool=yes @AMDEPBACKSLASH@ ++ at AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ ++ at am__fastdepCC_FALSE@ $(AM_V_CC at am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o x509/libcrypto_la-x509_vfy_apple.lo `test -f 'x509/x509_vfy_apple.c' || echo '$(srcdir)/'`x509/x509_vfy_apple.c ++ + x509/libcrypto_la-x509_vpm.lo: x509/x509_vpm.c + @am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libcrypto_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT x509/libcrypto_la-x509_vpm.lo -MD -MP -MF x509/$(DEPDIR)/libcrypto_la-x509_vpm.Tpo -c -o x509/libcrypto_la-x509_vpm.lo `test -f 'x509/x509_vpm.c' || echo '$(srcdir)/'`x509/x509_vpm.c + @am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) x509/$(DEPDIR)/libcrypto_la-x509_vpm.Tpo x509/$(DEPDIR)/libcrypto_la-x509_vpm.Plo +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy.c libressl-2.6.2/crypto/x509/x509_vfy.c +--- libressl-2.6.2.orig/crypto/x509/x509_vfy.c 2017-09-02 14:01:08.000000000 +0200 ++++ libressl-2.6.2/crypto/x509/x509_vfy.c 2017-10-07 14:05:16.000000000 +0200 +@@ -115,6 +115,13 @@ + + #define CRL_SCORE_TIME_DELTA 0x002 + ++/* ++ * If we are using Trust Evaluation Agent, rename the original function ++ */ ++#ifdef __APPLE__ ++#define X509_verify_cert X509_verify_cert_orig ++#endif ++ + static int null_callback(int ok, X509_STORE_CTX *e); + static int check_issued(X509_STORE_CTX *ctx, X509 *x, X509 *issuer); + static X509 *find_issuer(X509_STORE_CTX *ctx, STACK_OF(X509) *sk, X509 *x); +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c libressl-2.6.2/crypto/x509/x509_vfy_apple.c +--- libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.c 1970-01-01 01:00:00.000000000 +0100 ++++ libressl-2.6.2/crypto/x509/x509_vfy_apple.c 2017-10-07 14:05:16.000000000 +0200 +@@ -0,0 +1,225 @@ ++/* ++ * Copyright (c) 2009 Apple Inc. All Rights Reserved. ++ * ++ * @APPLE_LICENSE_HEADER_START@ ++ * ++ * This file contains Original Code and/or Modifications of Original Code ++ * as defined in and that are subject to the Apple Public Source License ++ * Version 2.0 (the 'License'). You may not use this file except in ++ * compliance with the License. Please obtain a copy of the License at ++ * http://www.opensource.apple.com/apsl/ and read it before using this ++ * file. ++ * ++ * The Original Code and all software distributed under the License are ++ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER ++ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, ++ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, ++ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. ++ * Please see the License for the specific language governing rights and ++ * limitations under the License. ++ * ++ * @APPLE_LICENSE_HEADER_END@ ++ * ++ */ ++ ++#include ++#include ++#include ++ ++#include ++ ++#include ++ ++#include ++#include ++#include ++ ++#include "cryptlib.h" ++#include "vpm_int.h" ++#include "x509_vfy_apple.h" ++ ++#define TEA_might_correct_error(err) (err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY || err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT || err == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN) ++ ++ ++static bool add_cert_to_array(CFMutableArrayRef array, X509 *x509) ++{ ++ unsigned char *asn1_cert_data = NULL; ++ int asn1_cert_len = i2d_X509(x509, &asn1_cert_data); ++ ++ CFDataRef data = CFDataCreate(kCFAllocatorDefault, asn1_cert_data, asn1_cert_len); ++ ++ if (data == NULL) { ++ return false; ++ } ++ ++ SecCertificateRef cert = SecCertificateCreateWithData(NULL, data); ++ ++ free(asn1_cert_data); ++ ++ if (cert == NULL) { ++ CFRelease(data); ++ return false; ++ } ++ ++ CFArrayAppendValue(array, cert); ++ CFRelease(data); ++ ++ return true; ++} ++ ++static CFStringRef to_string(const char *s) { ++ if (s == NULL) ++ return NULL; ++ return CFStringCreateWithCString(kCFAllocatorDefault, s, ++ kCFStringEncodingASCII); ++} ++ ++static SecPolicyRef get_policy(X509_VERIFY_PARAM *param) { ++ switch (param->purpose) { ++ case X509_PURPOSE_SSL_CLIENT: ++ case X509_PURPOSE_SSL_SERVER: { ++ ++ if (!param->id) { ++ fprintf(stderr, "got no ID!\n"); ++ return NULL; ++ } ++ ++ CFStringRef hostname; ++ int nhosts = sk_OPENSSL_STRING_num(param->id->hosts); ++ ++ if (nhosts != 1) { ++ hostname = NULL; ++ ++ } else { ++ hostname = to_string(sk_OPENSSL_STRING_value(param->id->hosts, 0)); ++ CFShow(hostname); ++ } ++ ++ return SecPolicyCreateSSL(param->purpose == X509_PURPOSE_SSL_SERVER, ++ hostname); ++ } ++ ++ case X509_PURPOSE_NS_SSL_SERVER: ++ case X509_PURPOSE_SMIME_SIGN: ++ case X509_PURPOSE_SMIME_ENCRYPT: ++ case X509_PURPOSE_CRL_SIGN: ++ case X509_PURPOSE_ANY: ++ case X509_PURPOSE_OCSP_HELPER: ++ case X509_PURPOSE_TIMESTAMP_SIGN: ++ default: ++ fprintf(stderr, "unsupported purpose %d", param->purpose); ++ return NULL; ++ } ++} ++ ++/* ++ * Please see comment in x509_vfy_apple.h ++ */ ++int ++X509_verify_cert(X509_STORE_CTX *ctx) ++{ ++ uint64_t certLastIndex = 0; ++ uint64_t i = 0; ++ ++ /* Try OpenSSL, if we get a local certificate issue verify against trusted roots */ ++ int ret = X509_verify_cert_orig(ctx); ++ ++ /* Verify TEA is enabled and should be used. */ ++ if (0 == X509_TEA_is_enabled() || ++ ret == 1 || !TEA_might_correct_error(ctx->error)) { ++ return ret; ++ } ++ ++ /* Verify that the certificate chain exists, otherwise make it. */ ++ if (ctx->chain == NULL && (ctx->chain = sk_X509_new_null()) == NULL) { ++ fprintf(stderr, "Could not create the certificate chain"); ++ ctx->error = X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY; ++ return -1; ++ } ++ ++ /* Verify chain depth */ ++ certLastIndex = sk_X509_num(ctx->untrusted); ++ if (certLastIndex > ctx->param->depth) { ++ fprintf(stderr, "Pruning certificate chain to %" PRIu64, certLastIndex); ++ certLastIndex = ctx->param->depth; ++ } ++ ++ CFMutableArrayRef certArray = CFArrayCreateMutable(NULL, certLastIndex + 1, NULL); ++ CFRetain(certArray); ++ ++ if (!add_cert_to_array(certArray, ctx->cert)) { ++ fprintf(stderr, "Failed to add certificate to array"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return -1; ++ } ++ ++ for (i = 0; i < certLastIndex; ++i) { ++ X509 *t = sk_X509_value(ctx->untrusted, i); ++ if (!add_cert_to_array(certArray, t)) { ++ fprintf(stderr, "Failed to add chain certificate %lld to array", i); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return 0; ++ } ++ } ++ ++ // We put ASN.1 encoded X509 on the CertificateChain, so we don't call TEACertificateChainSetEncodingHandler ++ SecPolicyRef policy = get_policy(ctx->param); ++ ++ if (policy == NULL) { ++ fprintf(stderr, "Failed to create policy!\n"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_UNSPECIFIED; ++ return -1; ++ } ++ ++ SecTrustRef trust = NULL; ++ ++ if (SecTrustCreateWithCertificates(certArray, policy, &trust) != errSecSuccess) { ++ fprintf(stderr, "Failed to create trust!\n"); ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_CERT_UNTRUSTED; ++ return -1; ++ } ++ ++ if (ctx->param->flags & X509_V_FLAG_USE_CHECK_TIME) { ++ fprintf(stderr, "Setting time not supported yet?\n"); ++ SecTrustSetVerifyDate(trust, CFDateCreate(NULL, ctx->param->check_time)); ++ } ++ ++ SecTrustResultType result = 0; ++ ++ if (SecTrustEvaluate(trust, &result) != errSecSuccess || result != kSecTrustResultUnspecified) { ++ CFRelease(certArray); ++ ctx->error = X509_V_ERR_CERT_UNTRUSTED; ++ return 0; ++ } ++ ++ CFRelease(certArray); ++ ctx->error = 0; ++ return 1; ++} ++ ++#pragma mark Trust Evaluation Agent ++ ++/* -1: not set ++ * 0: set to false ++ * 1: set to true ++ */ ++static int tea_enabled = -1; ++ ++void ++X509_TEA_set_state(int change) ++{ ++ tea_enabled = (change) ? 1 : 0; ++} ++ ++int ++X509_TEA_is_enabled() ++{ ++ if (tea_enabled < 0) ++ tea_enabled = (NULL == getenv(X509_TEA_ENV_DISABLE)); ++ ++ return tea_enabled != 0; ++} +diff -Naur libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.h libressl-2.6.2/crypto/x509/x509_vfy_apple.h +--- libressl-2.6.2.orig/crypto/x509/x509_vfy_apple.h 1970-01-01 01:00:00.000000000 +0100 ++++ libressl-2.6.2/crypto/x509/x509_vfy_apple.h 2017-10-07 14:05:16.000000000 +0200 +@@ -0,0 +1,74 @@ ++/* ++ * Copyright (c) 2009 Apple Inc. All Rights Reserved. ++ * ++ * @APPLE_LICENSE_HEADER_START@ ++ * ++ * This file contains Original Code and/or Modifications of Original Code ++ * as defined in and that are subject to the Apple Public Source License ++ * Version 2.0 (the 'License'). You may not use this file except in ++ * compliance with the License. Please obtain a copy of the License at ++ * http://www.opensource.apple.com/apsl/ and read it before using this ++ * file. ++ * ++ * The Original Code and all software distributed under the License are ++ * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER ++ * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, ++ * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, ++ * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. ++ * Please see the License for the specific language governing rights and ++ * limitations under the License. ++ * ++ * @APPLE_LICENSE_HEADER_END@ ++ * ++ */ ++ ++#ifndef HEADER_X509_H ++#include ++#endif ++ ++#ifndef HEADER_X509_VFY_APPLE_H ++#define HEADER_X509_VFY_APPLE_H ++ ++/* Environment variable name to disable TEA. */ ++#define X509_TEA_ENV_DISABLE "OPENSSL_X509_TEA_DISABLE" ++ ++/* ++ * X509_verify_cert ++ * ++ * Originally located in x509_vfy.c. ++ * ++ * Verify certificate with OpenSSL created X509_verify_cert. If and only if ++ * OpenSSL cannot get certificate issuer locally then OS X security API will ++ * verify the certificate, using Trust Evaluation Agent. ++ * ++ * Return values: ++ * -------------- ++ * -1: Null was passed for either ctx or ctx->cert. ++ * 0: Certificate is trusted. ++ * 1: Certificate is not trusted. ++ */ ++int X509_verify_cert(X509_STORE_CTX *ctx); ++ ++/* ++ * X509_TEA_is_enabled ++ * ++ * Is the Trust Evaluation Agent (TEA) used for certificate verification when ++ * the issuer cannot be verified. ++ * ++ * Returns 0 if TEA is disabled and 1 if TEA is enabled. ++ */ ++int X509_TEA_is_enabled(); ++ ++/* ++ * X509_TEA_set_state ++ * ++ * Enables/disables certificate verification with Trust Evaluation Agent (TEA) ++ * when the issuer cannot be verified. ++ * ++ * Pass 0 to disable TEA and non-zero to enable TEA. ++ */ ++void X509_TEA_set_state(int change); ++ ++int X509_verify_cert_orig(X509_STORE_CTX *ctx); ++ ++#endif /* HEADER_X509_VFY_APPLE_H */ diff --git a/pypy/goal/targetpypystandalone.py b/pypy/goal/targetpypystandalone.py --- a/pypy/goal/targetpypystandalone.py +++ b/pypy/goal/targetpypystandalone.py @@ -352,8 +352,9 @@ def hack_for_cffi_modules(self, driver): # HACKHACKHACK - # ugly hack to modify target goal from compile_* to build_cffi_imports - # this should probably get cleaned up and merged with driver.create_exe + # ugly hack to modify target goal from compile_* to build_cffi_imports, + # as done in package.py + # this is needed by the benchmark buildbot run, maybe do it as a seperate step there? from rpython.tool.runsubprocess import run_subprocess from rpython.translator.driver import taskdef import types @@ -363,11 +364,14 @@ def task_build_cffi_imports(self): ''' Use cffi to compile cffi interfaces to modules''' filename = os.path.join(pypydir, 'tool', 'build_cffi_imports.py') + if sys.platform == 'darwin': + argv = [filename, '--embed-dependencies'] + else: + argv = [filename,] status, out, err = run_subprocess(str(driver.compute_exe_name()), - [filename]) + argv) sys.stdout.write(out) sys.stderr.write(err) - # otherwise, ignore errors driver.task_build_cffi_imports = types.MethodType(task_build_cffi_imports, driver) driver.tasks['build_cffi_imports'] = driver.task_build_cffi_imports, [compile_goal] driver.default_goal = 'build_cffi_imports' diff --git a/pypy/tool/build_cffi_imports.py b/pypy/tool/build_cffi_imports.py --- a/pypy/tool/build_cffi_imports.py +++ b/pypy/tool/build_cffi_imports.py @@ -1,5 +1,5 @@ from __future__ import print_function -import sys, shutil, os +import sys, shutil, os, tempfile, hashlib from os.path import join class MissingDependenciesError(Exception): @@ -22,12 +22,131 @@ "xx": None, # for testing: 'None' should be completely ignored } -def create_cffi_import_libraries(pypy_c, options, basedir, only=None): +# for distribution, we may want to fetch dependencies not provided by +# the OS, such as a recent openssl/libressl or liblzma/xz. +cffi_dependencies = { + 'lzma': ('https://tukaani.org/xz/xz-5.2.3.tar.gz', + '71928b357d0a09a12a4b4c5fafca8c31c19b0e7d3b8ebb19622e96f26dbf28cb', + []), + 'ssl': ('http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.6.2.tar.gz', + 'b029d2492b72a9ba5b5fcd9f3d602c9fd0baa087912f2aaecc28f52f567ec478', + ['--without-openssldir']), + '_gdbm': ('http://ftp.gnu.org/gnu/gdbm/gdbm-1.13.tar.gz', + '9d252cbd7d793f7b12bcceaddda98d257c14f4d1890d851c386c37207000a253', + ['--without-readline']), +} + + +def _unpack_tarfile(filename, extract_dir): + """Unpack tar/tar.gz/tar.bz2/tar.xz `filename` to `extract_dir` + """ + import tarfile # late import for breaking circular dependency + try: + tarobj = tarfile.open(filename) + except tarfile.TarError: + raise ReadError( + "%s is not a compressed or uncompressed tar file" % filename) + try: + tarobj.extractall(extract_dir) + finally: + tarobj.close() + +def _sha256(filename): + dgst = hashlib.sha256() + + with open(filename, 'rb') as fp: + dgst.update(fp.read()) + return dgst.hexdigest() + + +def _build_dependency(name, destdir, patches=[]): + import multiprocessing + import shutil + import subprocess + + from rpython.tool.runsubprocess import run_subprocess + + try: + from urllib.request import urlretrieve + except ImportError: + from urllib import urlretrieve + + try: + url, dgst, args = cffi_dependencies[name] + except KeyError: + return 0, None, None + + archive_dir = os.path.join(tempfile.gettempdir(), 'pypy-archives') + + if not os.path.isdir(archive_dir): + os.makedirs(archive_dir) + + archive = os.path.join(archive_dir, url.rsplit('/', 1)[-1]) + + # next, fetch the archive to disk, if needed + if not os.path.exists(archive) or _sha256(archive) != dgst: + print('fetching archive', url, file=sys.stderr) + urlretrieve(url, archive) + + # extract the archive into our destination directory + print('unpacking archive', archive, file=sys.stderr) + _unpack_tarfile(archive, destdir) + + sources = os.path.join( + destdir, + os.path.basename(archive)[:-7], + ) + + # apply any patches + if patches: + for patch in patches: + print('applying patch', patch, file=sys.stderr) + status, stdout, stderr = run_subprocess( + '/usr/bin/patch', ['-p1', '-i', patch], cwd=sources, + ) + + if status != 0: + return status, stdout, stderr + + print('configuring', sources, file=sys.stderr) + + # configure & build it + status, stdout, stderr = run_subprocess( + './configure', + [ + '--prefix=/usr', + '--disable-shared', + '--enable-silent-rules', + '--disable-dependency-tracking', + ] + args, + cwd=sources, + ) + + if status != 0: + return status, stdout, stderr + + print('building', sources, file=sys.stderr) + + status, stdout, stderr = run_subprocess( + 'make', + [ + '-s', '-j' + str(multiprocessing.cpu_count()), + 'install', 'DESTDIR={}/'.format(destdir), + ], + cwd=sources, + ) + + return status, stdout, stderr + + +def create_cffi_import_libraries(pypy_c, options, basedir, only=None, + embed_dependencies=False): from rpython.tool.runsubprocess import run_subprocess shutil.rmtree(str(join(basedir,'lib_pypy','__pycache__')), ignore_errors=True) failures = [] + for key, module in sorted(cffi_build_scripts.items()): if only and key not in only: print("* SKIPPING", key, '(not specified in --only)') @@ -40,9 +159,49 @@ else: args = ['-c', 'import ' + module] cwd = None + env = os.environ.copy() + print('*', ' '.join(args), file=sys.stderr) + if embed_dependencies: + curdir = os.path.abspath(os.path.dirname(__file__)) + destdir = os.path.join(curdir, 'dest') + + shutil.rmtree(destdir, ignore_errors=True) + os.makedirs(destdir) + + if key == 'ssl' and sys.platform == 'darwin': + # this patch is loosely inspired by an Apple and adds + # a fallback to the OS X roots when none are available + patches = [ + os.path.join(curdir, + '../../lib_pypy/_cffi_ssl/osx-roots.diff'), + ] + else: + patches = [] + + status, stdout, stderr = _build_dependency(key, destdir, + patches=patches) + + if status != 0: + failures.append((key, module)) + print("stdout:") + print(stdout.decode('utf-8')) + print("stderr:") + print(stderr.decode('utf-8')) + continue + + env['CPPFLAGS'] = \ + '-I{}/usr/include {}'.format(destdir, env.get('CPPFLAGS', '')) + env['LDFLAGS'] = \ + '-L{}/usr/lib {}'.format(destdir, env.get('LDFLAGS', '')) + + if key == 'ssl' and sys.platform == 'darwin': + # needed for our roots patch + env['LDFLAGS'] += ' -framework CoreFoundation -framework Security' + try: - status, stdout, stderr = run_subprocess(str(pypy_c), args, cwd=cwd) + status, stdout, stderr = run_subprocess(str(pypy_c), args, + cwd=cwd, env=env) if status != 0: failures.append((key, module)) print("stdout:") @@ -73,6 +232,8 @@ ' you can specify an alternative pypy vm here') parser.add_argument('--only', dest='only', default=None, help='Only build the modules delimited by a colon. E.g. ssl,sqlite') + parser.add_argument('--embed-dependencies', dest='embed_dependencies', action='store_true', + help='embed dependencies for distribution') args = parser.parse_args() exename = join(os.getcwd(), args.exefile) @@ -89,7 +250,8 @@ only = None else: only = set(args.only.split(',')) - failures = create_cffi_import_libraries(exename, options, basedir, only=only) + failures = create_cffi_import_libraries(exename, options, basedir, only=only, + embed_dependencies=args.embed_dependencies) if len(failures) > 0: print('*** failed to build the CFFI modules %r' % ( [f[1] for f in failures],), file=sys.stderr) diff --git a/pypy/tool/release/package.py b/pypy/tool/release/package.py --- a/pypy/tool/release/package.py +++ b/pypy/tool/release/package.py @@ -83,7 +83,11 @@ if not _fake and not pypy_runs(pypy_c): raise OSError("Running %r failed!" % (str(pypy_c),)) if not options.no_cffi: - failures = create_cffi_import_libraries(str(pypy_c), options, str(basedir)) + failures = create_cffi_import_libraries( + str(pypy_c), options, str(basedir), + embed_dependencies=options.embed_dependencies, + ) + for key, module in failures: print >>sys.stderr, """!!!!!!!!!!\nBuilding {0} bindings failed. You can either install development headers package, @@ -266,6 +270,15 @@ def package(*args, **kwds): import argparse + + class NegateAction(argparse.Action): + def __init__(self, option_strings, dest, nargs=0, **kwargs): + super(NegateAction, self).__init__(option_strings, dest, nargs, + **kwargs) + + def __call__(self, parser, ns, values, option): + setattr(ns, self.dest, option[2:4] != 'no') + if sys.platform == 'win32': pypy_exe = 'pypy3.exe' else: @@ -296,12 +309,20 @@ help='destination dir for archive') parser.add_argument('--override_pypy_c', type=str, default='', help='use as pypy3 exe instead of pypy/goal/pypy3-c') + parser.add_argument('--embedded-dependencies', '--no-embedded-dependencies', + dest='embed_dependencies', + action=NegateAction, + default=(sys.platform == 'darwin'), + help='whether to embed dependencies for distribution ' + '(default on OS X)') options = parser.parse_args(args) if os.environ.has_key("PYPY_PACKAGE_NOSTRIP"): options.nostrip = True if os.environ.has_key("PYPY_PACKAGE_WITHOUTTK"): options.no_tk = True + if os.environ.has_key("PYPY_EMBED_DEPENDENCIES"): + options.embed_dependencies = True if not options.builddir: # The import actually creates the udir directory from rpython.tool.udir import udir From pypy.commits at gmail.com Mon Oct 30 13:12:35 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 30 Oct 2017 10:12:35 -0700 (PDT) Subject: [pypy-commit] pypy default: graft rpython changes from py3.5, maybe we should enforce sycnronization with a test? Message-ID: <59f75d83.1bb3df0a.419be.a9af@mx.google.com> Author: Matti Picus Branch: Changeset: r92884:80602ff2f2d3 Date: 2017-10-30 18:51 +0200 http://bitbucket.org/pypy/pypy/changeset/80602ff2f2d3/ Log: graft rpython changes from py3.5, maybe we should enforce sycnronization with a test? diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py --- a/rpython/rlib/_rsocket_rffi.py +++ b/rpython/rlib/_rsocket_rffi.py @@ -162,7 +162,7 @@ IP_RECVRETOPTS IP_RETOPTS IP_TOS IP_TTL MSG_BTAG MSG_ETAG MSG_CTRUNC MSG_DONTROUTE MSG_DONTWAIT MSG_EOR MSG_OOB -MSG_PEEK MSG_TRUNC MSG_WAITALL +MSG_PEEK MSG_TRUNC MSG_WAITALL MSG_ERRQUEUE NI_DGRAM NI_MAXHOST NI_MAXSERV NI_NAMEREQD NI_NOFQDN NI_NUMERICHOST NI_NUMERICSERV @@ -348,7 +348,8 @@ ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))]) # insert handler for sendmsg / recvmsg here -if _POSIX: +HAVE_SENDMSG = bool(_POSIX) +if HAVE_SENDMSG: includes = ['stddef.h', 'sys/socket.h', 'unistd.h', @@ -534,7 +535,7 @@ int cmsg_status; struct iovec iov; struct recvmsg_info* retinfo; - int error_flag; // variable to be set in case of special errors. + int error_flag = 0; // variable to be set in case of special errors. int cmsgdatalen = 0; // variables that are set to 1, if the message charp has been allocated @@ -708,6 +709,7 @@ free(retinfo); } } + if (error_flag==0) error_flag = -1; return error_flag; err_closefds: diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py --- a/rpython/rlib/rposix.py +++ b/rpython/rlib/rposix.py @@ -472,7 +472,7 @@ def close(fd): validate_fd(fd) handle_posix_error('close', c_close(fd)) - + c_lseek = external('_lseeki64' if _WIN32 else 'lseek', [rffi.INT, rffi.LONGLONG, rffi.INT], rffi.LONGLONG, macro=_MACRO_ON_POSIX, save_err=rffi.RFFI_SAVE_ERRNO) @@ -505,7 +505,7 @@ with rffi.scoped_alloc_buffer(count) as buf: void_buf = rffi.cast(rffi.VOIDP, buf.raw) return buf.str(handle_posix_error('pread', c_pread(fd, void_buf, count, offset))) - + @enforceargs(int, None, None) def pwrite(fd, data, offset): count = len(data) @@ -1851,6 +1851,8 @@ rffi.INT, save_err=rffi.RFFI_FULL_ERRNO_ZERO) c_sched_get_priority_min = external('sched_get_priority_min', [rffi.INT], rffi.INT, save_err=rffi.RFFI_SAVE_ERRNO) + if not _WIN32: + c_sched_yield = external('sched_yield', [], rffi.INT) @enforceargs(int) def sched_get_priority_max(policy): @@ -1860,8 +1862,8 @@ def sched_get_priority_min(policy): return handle_posix_error('sched_get_priority_min', c_sched_get_priority_min(policy)) - - + def sched_yield(): + return handle_posix_error('sched_yield', c_sched_yield()) #___________________________________________________________________ diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py --- a/rpython/rlib/rsocket.py +++ b/rpython/rlib/rsocket.py @@ -1074,7 +1074,7 @@ if address is not None: address.unlock() - if _c.geterrno() == _c.EINTR: + if (_c.geterrno() == _c.EINTR) or (_c.geterrno() == 11): raise last_error() if (reply == -10000): raise RSocketError("Invalid message size") @@ -1393,7 +1393,7 @@ return (make_socket(fd0, family, type, proto, SocketClass), make_socket(fd1, family, type, proto, SocketClass)) -if _c._POSIX: +if _c.HAVE_SENDMSG: def CMSG_LEN( demanded_len): """ Socket method to determine the optimal byte size of the ancillary. diff --git a/rpython/rlib/test/test_rposix.py b/rpython/rlib/test/test_rposix.py --- a/rpython/rlib/test/test_rposix.py +++ b/rpython/rlib/test/test_rposix.py @@ -676,7 +676,7 @@ prio = rposix.getpriority(rposix.PRIO_PROCESS, 0) rposix.setpriority(rposix.PRIO_PROCESS, 0, prio) py.test.raises(OSError, rposix.getpriority, rposix.PRIO_PGRP, 123456789) - + if sys.platform != 'win32': def test_sendfile(): from rpython.rlib import rsocket @@ -723,7 +723,7 @@ os.close(fd) s2.close() s1.close() - + @rposix_requires('pread') def test_pread(): fname = str(udir.join('os_test.txt')) @@ -794,14 +794,14 @@ assert rposix.sched_get_priority_max(rposix.SCHED_RR) != -1 assert rposix.sched_get_priority_max(rposix.SCHED_OTHER) != -1 assert rposix.sched_get_priority_max(rposix.SCHED_BATCH) != -1 - + @rposix_requires('sched_get_priority_min') def test_sched_get_priority_min(): assert rposix.sched_get_priority_min(rposix.SCHED_FIFO) != -1 assert rposix.sched_get_priority_min(rposix.SCHED_RR) != -1 assert rposix.sched_get_priority_min(rposix.SCHED_OTHER) != -1 assert rposix.sched_get_priority_min(rposix.SCHED_BATCH) != -1 - + @rposix_requires('sched_get_priority_min') def test_os_sched_priority_max_greater_than_min(): policy = rposix.SCHED_RR @@ -811,3 +811,8 @@ assert isinstance(high, int) == True assert high > low + at rposix_requires('sched_yield') +def test_sched_yield(): + if sys.platform != 'win32': + rposix.sched_yield() + From pypy.commits at gmail.com Mon Oct 30 13:16:36 2017 From: pypy.commits at gmail.com (mattip) Date: Mon, 30 Oct 2017 10:16:36 -0700 (PDT) Subject: [pypy-commit] buildbot default: revert part of 2336f92eb562 that tries to run rpython tests on non-default branch Message-ID: <59f75e74.05d21c0a.82886.2d72@mx.google.com> Author: Matti Picus Branch: Changeset: r1036:bf6e08ac46ee Date: 2017-10-30 19:16 +0200 http://bitbucket.org/pypy/buildbot/changeset/bf6e08ac46ee/ Log: revert part of 2336f92eb562 that tries to run rpython tests on non- default branch Instead we should have a commit hook that rejects changes to rpython from py3.5 diff --git a/bot2/pypybuildbot/master.py b/bot2/pypybuildbot/master.py --- a/bot2/pypybuildbot/master.py +++ b/bot2/pypybuildbot/master.py @@ -346,15 +346,6 @@ JITWIN32, # on allegro_win32, SalsaSalsa ], branch="py3.5", hour=3, minute=0), - # XXX disable this if you are SURE py3.5 and default rpython are identical - Nightly("nightly-3-01-py3.5", [ - LINUX32RPYTHON, # on tannit32, uses all cores - LINUX64RPYTHON, # on bencher4, uses all cores - ], branch="py3.5", hour=3, minute=0, onlyIfChanged=True, - fileIsImportant=isRPython, - change_filter=filter.ChangeFilter(branch='py3.5'), - ), - # S390X vm (ibm-research) Nightly("nightly-4-00", [ LINUX_S390XOWN, From pypy.commits at gmail.com Tue Oct 31 12:06:19 2017 From: pypy.commits at gmail.com (fijal) Date: Tue, 31 Oct 2017 09:06:19 -0700 (PDT) Subject: [pypy-commit] pypy memory-accounting: add one more data point Message-ID: <59f89f7b.a8a0df0a.76977.068a@mx.google.com> Author: fijal Branch: memory-accounting Changeset: r92887:3c6f624e8ed0 Date: 2017-10-31 17:05 +0100 http://bitbucket.org/pypy/pypy/changeset/3c6f624e8ed0/ Log: add one more data point diff --git a/pypy/module/gc/referents.py b/pypy/module/gc/referents.py --- a/pypy/module/gc/referents.py +++ b/pypy/module/gc/referents.py @@ -174,10 +174,13 @@ class W_GcStats(W_Root): def __init__(self): self.total_memory_pressure = rgc.get_stats(rgc.TOTAL_MEMORY_PRESSURE) + self.total_gc_memory = rgc.get_stats(rgc.TOTAL_MEMORY) W_GcStats.typedef = TypeDef("GcStats", total_memory_pressure=interp_attrproperty("total_memory_pressure", cls=W_GcStats, wrapfn="newint"), + total_gc_memory=interp_attrproperty("total_gc_memory", + cls=W_GcStats, wrapfn="newint") ) def get_stats(space): diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py --- a/rpython/memory/gc/incminimark.py +++ b/rpython/memory/gc/incminimark.py @@ -2923,6 +2923,8 @@ from rpython.memory.gc import inspector if stats_no == rgc.TOTAL_MEMORY: + return intmask(self.get_total_memory_used() + self.nursery_size) + elif stats_no == rgc.TOTAL_ALLOCATED_MEMORY: return 0 elif stats_no == rgc.TOTAL_MEMORY_PRESSURE: return inspector.count_memory_pressure(self) diff --git a/rpython/memory/gc/minimarkpage.py b/rpython/memory/gc/minimarkpage.py --- a/rpython/memory/gc/minimarkpage.py +++ b/rpython/memory/gc/minimarkpage.py @@ -140,6 +140,7 @@ # the total memory used, counting every block in use, without # the additional bookkeeping stuff. self.total_memory_used = r_uint(0) + self.total_memory_alloced = r_uint(0) def _new_page_ptr_list(self, length): diff --git a/rpython/rlib/rgc.py b/rpython/rlib/rgc.py --- a/rpython/rlib/rgc.py +++ b/rpython/rlib/rgc.py @@ -650,7 +650,7 @@ else: return id(gcref._x) -TOTAL_MEMORY, TOTAL_MEMORY_PRESSURE = range(2) +TOTAL_MEMORY, TOTAL_ALLOCATED_MEMORY, TOTAL_MEMORY_PRESSURE = range(3) @not_rpython def get_stats(stat_no): From pypy.commits at gmail.com Tue Oct 31 13:18:31 2017 From: pypy.commits at gmail.com (fijal) Date: Tue, 31 Oct 2017 10:18:31 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: whack whack whack, fix UnicodeBuilder, so we can make json work Message-ID: <59f8b067.4fc7df0a.3841c.201f@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92888:1b909417f0e0 Date: 2017-10-31 18:17 +0100 http://bitbucket.org/pypy/pypy/changeset/1b909417f0e0/ Log: whack whack whack, fix UnicodeBuilder, so we can make json work diff --git a/pypy/module/__pypy__/__init__.py b/pypy/module/__pypy__/__init__.py --- a/pypy/module/__pypy__/__init__.py +++ b/pypy/module/__pypy__/__init__.py @@ -12,7 +12,7 @@ interpleveldefs = { "StringBuilder": "interp_builders.W_StringBuilder", - #"UnicodeBuilder": "interp_builders.W_UnicodeBuilder", + "UnicodeBuilder": "interp_builders.W_UnicodeBuilder", } class TimeModule(MixedModule): diff --git a/pypy/module/__pypy__/interp_builders.py b/pypy/module/__pypy__/interp_builders.py --- a/pypy/module/__pypy__/interp_builders.py +++ b/pypy/module/__pypy__/interp_builders.py @@ -2,60 +2,98 @@ from pypy.interpreter.error import oefmt from pypy.interpreter.gateway import interp2app, unwrap_spec from pypy.interpreter.typedef import TypeDef -from rpython.rlib.rstring import UnicodeBuilder, StringBuilder +from rpython.rlib.rstring import StringBuilder +from pypy.objspace.std.unicodeobject import W_UnicodeObject from rpython.tool.sourcetools import func_with_new_name -def create_builder(name, strtype, builder_cls, newmethod): - if strtype is str: - unwrap = 'bytes' - else: - unwrap = unicode - class W_Builder(W_Root): - def __init__(self, space, size): - if size < 0: - self.builder = builder_cls() - else: - self.builder = builder_cls(size) +class W_StringBuilder(W_Root): + def __init__(self, space, size): + if size < 0: + self.builder = StringBuilder() + else: + self.builder = StringBuilder(size) - @unwrap_spec(size=int) - def descr__new__(space, w_subtype, size=-1): - return W_Builder(space, size) + @unwrap_spec(size=int) + def descr__new__(space, w_subtype, size=-1): + return W_StringBuilder(space, size) - @unwrap_spec(s=unwrap) - def descr_append(self, space, s): - self.builder.append(s) + @unwrap_spec(s='bytes') + def descr_append(self, space, s): + self.builder.append(s) - @unwrap_spec(s=unwrap, start=int, end=int) - def descr_append_slice(self, space, s, start, end): - if not 0 <= start <= end <= len(s): - raise oefmt(space.w_ValueError, "bad start/stop") - self.builder.append_slice(s, start, end) + @unwrap_spec(s='bytes', start=int, end=int) + def descr_append_slice(self, space, s, start, end): + if not 0 <= start <= end <= len(s): + raise oefmt(space.w_ValueError, "bad start/stop") + self.builder.append_slice(s, start, end) - def descr_build(self, space): - w_s = getattr(space, newmethod)(self.builder.build()) - # after build(), we can continue to append more strings - # to the same builder. This is supported since - # 2ff5087aca28 in RPython. - return w_s + def descr_build(self, space): + w_s = space.newbytes(self.builder.build()) + # after build(), we can continue to append more strings + # to the same builder. This is supported since + # 2ff5087aca28 in RPython. + return w_s - def descr_len(self, space): - if self.builder is None: - raise oefmt(space.w_ValueError, "no length of built builder") - return space.newint(self.builder.getlength()) + def descr_len(self, space): + if self.builder is None: + raise oefmt(space.w_ValueError, "no length of built builder") + return space.newint(self.builder.getlength()) - W_Builder.__name__ = "W_%s" % name - W_Builder.typedef = TypeDef(name, - __new__ = interp2app(func_with_new_name( - W_Builder.descr__new__.im_func, - '%s_new' % (name,))), - append = interp2app(W_Builder.descr_append), - append_slice = interp2app(W_Builder.descr_append_slice), - build = interp2app(W_Builder.descr_build), - __len__ = interp2app(W_Builder.descr_len), - ) - W_Builder.typedef.acceptable_as_base_class = False - return W_Builder +W_StringBuilder.typedef = TypeDef("StringBuilder", + __new__ = interp2app(func_with_new_name( + W_StringBuilder.descr__new__.im_func, + 'StringBuilder_new')), + append = interp2app(W_StringBuilder.descr_append), + append_slice = interp2app(W_StringBuilder.descr_append_slice), + build = interp2app(W_StringBuilder.descr_build), + __len__ = interp2app(W_StringBuilder.descr_len), +) +W_StringBuilder.typedef.acceptable_as_base_class = False -W_StringBuilder = create_builder("StringBuilder", str, StringBuilder, "newbytes") -#W_UnicodeBuilder = create_builder("UnicodeBuilder", unicode, UnicodeBuilder, "newunicode") +class W_UnicodeBuilder(W_Root): + def __init__(self, space, size): + if size < 0: + self.builder = StringBuilder() + else: + self.builder = StringBuilder(size) + + @unwrap_spec(size=int) + def descr__new__(space, w_subtype, size=-1): + return W_UnicodeBuilder(space, size) + + @unwrap_spec(s='utf8') + def descr_append(self, space, s): + self.builder.append(s) + + @unwrap_spec(start=int, end=int) + def descr_append_slice(self, space, w_s, start, end): + w_unicode = W_UnicodeObject.convert_arg_to_w_unicode(space, w_s) + if not 0 <= start <= end <= w_unicode._len(): + raise oefmt(space.w_ValueError, "bad start/stop") + byte_start = w_unicode._index_to_byte(start) + byte_end = w_unicode._index_to_byte(end) + self.builder.append_slice(w_unicode._utf8, byte_start, byte_end) + + def descr_build(self, space): + w_s = space.newtext(self.builder.build()) + # after build(), we can continue to append more strings + # to the same builder. This is supported since + # 2ff5087aca28 in RPython. + return w_s + + def descr_len(self, space): + if self.builder is None: + raise oefmt(space.w_ValueError, "no length of built builder") + return space.newint(self.builder.getlength()) + +W_UnicodeBuilder.typedef = TypeDef("UnicodeBuilder", + __new__ = interp2app(func_with_new_name( + W_UnicodeBuilder.descr__new__.im_func, + 'UnicodeBuilder_new')), + append = interp2app(W_UnicodeBuilder.descr_append), + append_slice = interp2app(W_UnicodeBuilder.descr_append_slice), + build = interp2app(W_UnicodeBuilder.descr_build), + __len__ = interp2app(W_UnicodeBuilder.descr_len), +) +W_UnicodeBuilder.typedef.acceptable_as_base_class = False diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py --- a/pypy/module/cpyext/test/test_bytesobject.py +++ b/pypy/module/cpyext/test/test_bytesobject.py @@ -596,7 +596,7 @@ encoding = rffi.str2charp("latin-1") w_res = PyString_AsDecodedObject(space, w_str, encoding, None) rffi.free_charp(encoding) - assert space.unwrap(w_res) == u"caf\xe9" + assert w_res._utf8 == u"caf\xe9".encode('utf8') def test_eq(self, space): assert 1 == _PyString_Eq( From pypy.commits at gmail.com Tue Oct 31 13:31:55 2017 From: pypy.commits at gmail.com (fijal) Date: Tue, 31 Oct 2017 10:31:55 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: rewrite to_decimal Message-ID: <59f8b38b.508d1c0a.f0b96.c60c@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92889:81c556f8f8b4 Date: 2017-10-31 18:31 +0100 http://bitbucket.org/pypy/pypy/changeset/81c556f8f8b4/ Log: rewrite to_decimal diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -1746,21 +1746,22 @@ def unicode_to_decimal_w(space, w_unistr): if not isinstance(w_unistr, W_UnicodeObject): raise oefmt(space.w_TypeError, "expected unicode, got '%T'", w_unistr) - unistr = w_unistr._utf8.decode("utf8") - # XXX speed up + unistr = w_unistr._utf8 result = ['\0'] * len(unistr) digits = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] - for i in xrange(len(unistr)): - uchr = ord(unistr[i]) - if unicodedb.isspace(uchr): - result[i] = ' ' + i = 0 + res_pos = 0 + while i < len(unistr): + uchr = rutf8.codepoint_at_pos(unistr, i) + if rutf8.isspace(unistr, i): + result[res_pos] = ' ' continue try: - result[i] = digits[unicodedb.decimal(uchr)] + result[res_pos] = digits[unicodedb.decimal(uchr)] except KeyError: if 0 < uchr < 256: - result[i] = chr(uchr) + result[res_pos] = chr(uchr) else: w_encoding = space.newtext('decimal') w_start = space.newint(i) @@ -1770,6 +1771,8 @@ space.newtuple([w_encoding, w_unistr, w_start, w_end, w_reason])) + i = rutf8.next_codepoint_pos(unistr, i) + res_pos += 1 return ''.join(result) From pypy.commits at gmail.com Tue Oct 31 14:18:34 2017 From: pypy.commits at gmail.com (fijal) Date: Tue, 31 Oct 2017 11:18:34 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: fix readbuf_w Message-ID: <59f8be7a.cd5e1c0a.34b5a.99c7@mx.google.com> Author: fijal Branch: unicode-utf8 Changeset: r92890:f7f1771c68d3 Date: 2017-10-31 18:43 +0100 http://bitbucket.org/pypy/pypy/changeset/f7f1771c68d3/ Log: fix readbuf_w diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py --- a/pypy/objspace/std/unicodeobject.py +++ b/pypy/objspace/std/unicodeobject.py @@ -87,13 +87,15 @@ def readbuf_w(self, space): # XXX for now - from rpython.rlib.rstruct.unichar import pack_unichar, UNICODE_SIZE - v = self._utf8.decode("utf8") - builder = MutableStringBuffer(len(v) * UNICODE_SIZE) + from rpython.rlib.rstruct.unichar import pack_codepoint, UNICODE_SIZE + builder = MutableStringBuffer(self._len() * UNICODE_SIZE) pos = 0 - for unich in v: - pack_unichar(unich, builder, pos) + i = 0 + while i < len(self._utf8): + unich = rutf8.codepoint_at_pos(self._utf8, i) + pack_codepoint(unich, builder, pos) pos += UNICODE_SIZE + i = rutf8.next_codepoint_pos(self._utf8, i) return StringBuffer(builder.finish()) def writebuf_w(self, space): diff --git a/rpython/rlib/rstruct/unichar.py b/rpython/rlib/rstruct/unichar.py --- a/rpython/rlib/rstruct/unichar.py +++ b/rpython/rlib/rstruct/unichar.py @@ -12,24 +12,27 @@ BIGENDIAN = sys.byteorder == "big" def pack_unichar(unich, buf, pos): + pack_codepoint(ord(unich), buf, pos) + +def pack_codepoint(unich, buf, pos): if UNICODE_SIZE == 2: if BIGENDIAN: - buf.setitem(pos, chr(ord(unich) >> 8)) - buf.setitem(pos+1, chr(ord(unich) & 0xFF)) + buf.setitem(pos, chr(unich >> 8)) + buf.setitem(pos+1, chr(unich & 0xFF)) else: - buf.setitem(pos, chr(ord(unich) & 0xFF)) - buf.setitem(pos+1, chr(ord(unich) >> 8)) + buf.setitem(pos, chr(unich & 0xFF)) + buf.setitem(pos+1, chr(unich >> 8)) else: if BIGENDIAN: - buf.setitem(pos, chr(ord(unich) >> 24)) - buf.setitem(pos+1, chr((ord(unich) >> 16) & 0xFF)) - buf.setitem(pos+2, chr((ord(unich) >> 8) & 0xFF)) - buf.setitem(pos+3, chr(ord(unich) & 0xFF)) + buf.setitem(pos, chr(unich >> 24)) + buf.setitem(pos+1, chr((unich >> 16) & 0xFF)) + buf.setitem(pos+2, chr((unich >> 8) & 0xFF)) + buf.setitem(pos+3, chr(unich & 0xFF)) else: - buf.setitem(pos, chr(ord(unich) & 0xFF)) - buf.setitem(pos+1, chr((ord(unich) >> 8) & 0xFF)) - buf.setitem(pos+2, chr((ord(unich) >> 16) & 0xFF)) - buf.setitem(pos+3, chr(ord(unich) >> 24)) + buf.setitem(pos, chr(unich & 0xFF)) + buf.setitem(pos+1, chr((unich >> 8) & 0xFF)) + buf.setitem(pos+2, chr((unich >> 16) & 0xFF)) + buf.setitem(pos+3, chr(unich >> 24)) def unpack_unichar(rawstring): assert len(rawstring) == UNICODE_SIZE From pypy.commits at gmail.com Tue Oct 31 14:18:36 2017 From: pypy.commits at gmail.com (mattip) Date: Tue, 31 Oct 2017 11:18:36 -0700 (PDT) Subject: [pypy-commit] pypy unicode-utf8: move test summary to end on terminal, after the multiple pages of failure info, typos Message-ID: <59f8be7c.93111c0a.e334e.c728@mx.google.com> Author: Matti Picus Branch: unicode-utf8 Changeset: r92891:c77076692e78 Date: 2017-10-28 22:18 +0300 http://bitbucket.org/pypy/pypy/changeset/c77076692e78/ Log: move test summary to end on terminal, after the multiple pages of failure info, typos diff --git a/_pytest/terminal.py b/_pytest/terminal.py --- a/_pytest/terminal.py +++ b/_pytest/terminal.py @@ -366,11 +366,11 @@ EXIT_OK, EXIT_TESTSFAILED, EXIT_INTERRUPTED, EXIT_USAGEERROR, EXIT_NOTESTSCOLLECTED) if exitstatus in summary_exit_codes: - self.config.hook.pytest_terminal_summary(terminalreporter=self) self.summary_errors() self.summary_failures() self.summary_warnings() self.summary_passes() + self.config.hook.pytest_terminal_summary(terminalreporter=self) if exitstatus == EXIT_INTERRUPTED: self._report_keyboardinterrupt() del self._keyboardinterrupt_memo diff --git a/pypy/test_all.py b/pypy/test_all.py --- a/pypy/test_all.py +++ b/pypy/test_all.py @@ -27,7 +27,7 @@ sys.path.insert(0,os.path.dirname(os.path.dirname(os.path.realpath(__file__)))) import pytest if sys.platform == 'win32': - #Try to avoid opeing a dialog box if one of the tests causes a system error + #Try to avoid opening a dialog box if one of the tests causes a system error # We do this in runner.py, but buildbots run twisted which ruins inheritance # in windows subprocesses. import ctypes diff --git a/testrunner/runner.py b/testrunner/runner.py --- a/testrunner/runner.py +++ b/testrunner/runner.py @@ -21,7 +21,7 @@ win32api.CloseHandle(proch) except pywintypes.error, e: pass - #Try to avoid opeing a dialog box if one of the tests causes a system error + #Try to avoid opening a dialog box if one of the tests causes a system error import ctypes winapi = ctypes.windll.kernel32 SetErrorMode = winapi.SetErrorMode From pypy.commits at gmail.com Tue Oct 31 20:29:37 2017 From: pypy.commits at gmail.com (antocuni) Date: Tue, 31 Oct 2017 17:29:37 -0700 (PDT) Subject: [pypy-commit] pypy keep-debug-symbols: add a smartstrip tool, which can optionally keep the debug symbols in a separate file, instead of just stripping them away Message-ID: <59f91571.c4581c0a.5fcb4.65d3@mx.google.com> Author: Antonio Cuni Branch: keep-debug-symbols Changeset: r92892:005844269621 Date: 2017-11-01 01:19 +0100 http://bitbucket.org/pypy/pypy/changeset/005844269621/ Log: add a smartstrip tool, which can optionally keep the debug symbols in a separate file, instead of just stripping them away diff --git a/pypy/tool/release/smartstrip.py b/pypy/tool/release/smartstrip.py new file mode 100644 --- /dev/null +++ b/pypy/tool/release/smartstrip.py @@ -0,0 +1,32 @@ +""" +Strip symbols from an executable, but keep them in a .debug file +""" + +import sys +import os +import py + +def _strip(exe): + if sys.platform == 'win32': + pass + elif sys.platform == 'darwin': + # 'strip' fun: see issue #587 for why -x + os.system("strip -x " + str(exe)) # ignore errors + else: + os.system("strip " + str(exe)) # ignore errors + +def _extract_debug_symbols(exe, debug): + if sys.platform == 'linux2': + os.system("objcopy --only-keep-debug %s %s" % (exe, debug)) + os.system("objcopy --add-gnu-debuglink=%s %s" % (debug, exe)) + +def smartstrip(exe, keep_debug=True): + exe = py.path.local(exe) + debug = py.path.local(str(exe) + '.debug') + if keep_debug: + _extract_debug_symbols(exe, debug) + _strip(exe) + + +if __name__ == '__main__': + smartstrip(sys.argv[1]) diff --git a/pypy/tool/release/test/test_smartstrip.py b/pypy/tool/release/test/test_smartstrip.py new file mode 100644 --- /dev/null +++ b/pypy/tool/release/test/test_smartstrip.py @@ -0,0 +1,50 @@ +import pytest +import sys +import os +from commands import getoutput +from pypy.tool.release.smartstrip import smartstrip + + at pytest.fixture +def exe(tmpdir): + src = tmpdir.join("myprog.c") + src.write(""" + int foo(int a, int b) { + return a+b; + } + int main(void) { } + """) + exe = tmpdir.join("myprog") + ret = os.system("gcc -o %s %s" % (exe, src)) + assert ret == 0 + return exe + +def info_symbol(exe, symbol): + out = getoutput("gdb %s -ex 'info symbol %s' -ex 'quit'" % (exe, symbol)) + lines = out.splitlines() + return lines[-1] + + at pytest.mark.skipif(sys.platform == 'win32', + reason='strip not supported on windows') +class TestSmarStrip(object): + + def test_info_symbol(self, exe): + info = info_symbol(exe, "foo") + assert info == "foo in section .text" + + def test_strip(self, exe): + smartstrip(exe, keep_debug=False) + info = info_symbol(exe, "foo") + assert info.startswith("No symbol table is loaded") + + @pytest.mark.skipif(sys.platform != 'linux2', + reason='keep_debug not supported') + def test_keep_debug(self, exe, tmpdir): + smartstrip(exe, keep_debug=True) + debug = tmpdir.join("myprog.debug") + assert debug.check(file=True) + info = info_symbol(exe, "foo") + assert info == "foo in section .text of %s" % exe + # + debug.remove() + info = info_symbol(exe, "foo") + assert info.startswith("No symbol table is loaded") From pypy.commits at gmail.com Tue Oct 31 20:29:39 2017 From: pypy.commits at gmail.com (antocuni) Date: Tue, 31 Oct 2017 17:29:39 -0700 (PDT) Subject: [pypy-commit] pypy keep-debug-symbols: remove the nostrip option; introduce a --no-keep-debug option; use smartstrip(keep_debug=True) by default Message-ID: <59f91573.7cb8df0a.9f774.acea@mx.google.com> Author: Antonio Cuni Branch: keep-debug-symbols Changeset: r92893:7108fe737121 Date: 2017-11-01 01:28 +0100 http://bitbucket.org/pypy/pypy/changeset/7108fe737121/ Log: remove the nostrip option; introduce a --no-keep-debug option; use smartstrip(keep_debug=True) by default diff --git a/pypy/tool/release/package.py b/pypy/tool/release/package.py --- a/pypy/tool/release/package.py +++ b/pypy/tool/release/package.py @@ -21,6 +21,7 @@ import fnmatch import subprocess import glob +from pypy.tool.release.smartstrip import smartstrip if sys.version_info < (2,6): py.test.skip("requires 2.6 so far") @@ -212,15 +213,8 @@ old_dir = os.getcwd() try: os.chdir(str(builddir)) - if not options.nostrip: - for source, target in binaries: - if sys.platform == 'win32': - pass - elif sys.platform == 'darwin': - # 'strip' fun: see issue #587 for why -x - os.system("strip -x " + str(bindir.join(target))) # ignore errors - else: - os.system("strip " + str(bindir.join(target))) # ignore errors + for source, target in binaries: + smartstrip(bindir.join(target), keep_debug=options.keep_debug) # if USE_ZIPFILE_MODULE: import zipfile @@ -281,8 +275,8 @@ help='do not build and package the %r cffi module' % (key,)) parser.add_argument('--without-cffi', dest='no_cffi', action='store_true', help='skip building *all* the cffi modules listed above') - parser.add_argument('--nostrip', dest='nostrip', action='store_true', - help='do not strip the exe, making it ~10MB larger') + parser.add_argument('--no-keep-debug', dest='keep_debug', + action='store_false', help='do not keep debug symbols') parser.add_argument('--rename_pypy_c', dest='pypy_c', type=str, default=pypy_exe, help='target executable name, defaults to "pypy"') parser.add_argument('--archive-name', dest='name', type=str, default='', @@ -295,8 +289,8 @@ help='use as pypy exe instead of pypy/goal/pypy-c') options = parser.parse_args(args) - if os.environ.has_key("PYPY_PACKAGE_NOSTRIP"): - options.nostrip = True + if os.environ.has_key("PYPY_PACKAGE_NOKEEPDEBUG"): + options.keep_debug = False if os.environ.has_key("PYPY_PACKAGE_WITHOUTTK"): options.no_tk = True if not options.builddir:

Python2.7 compatible PyPy 5.8.0

Python2.7 compatible PyPy 5.9.0

Python 3.5.3 compatible PyPy3.5 v5.8

Python 3.5.3 compatible PyPy3.5 v5.9

Installing NumPy

1. NumPy via cpyext

1. Standard NumPy

2. NumPyPy