From pypy.commits at gmail.com  Tue Aug  1 05:13:41 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 01 Aug 2017 02:13:41 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2607 (jean-louis)
Message-ID: <59804645.d8a0df0a.9832.4e64@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92011:c8577e0e0932
Date: 2017-08-01 11:13 +0200
http://bitbucket.org/pypy/pypy/changeset/c8577e0e0932/

Log:	Issue #2607 (jean-louis)

	Instead of sys/poll.h, use poll.h, which is the officially correct
	header.

diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py
--- a/rpython/rlib/_rsocket_rffi.py
+++ b/rpython/rlib/_rsocket_rffi.py
@@ -20,7 +20,7 @@
     includes = ('sys/types.h',
                 'sys/socket.h',
                 'sys/un.h',
-                'sys/poll.h',
+                'poll.h',
                 'sys/select.h',
                 'sys/types.h',
                 'netinet/in.h',

From pypy.commits at gmail.com  Tue Aug  1 06:43:42 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 01 Aug 2017 03:43:42 -0700 (PDT)
Subject: [pypy-commit] pypy default: Skip unreliable test when testing with
 either -A or with a pypy host
Message-ID: <59805b5e.08891c0a.bacc6.6b10@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92012:c88684d6f4c1
Date: 2017-08-01 12:43 +0200
http://bitbucket.org/pypy/pypy/changeset/c88684d6f4c1/

Log:	Skip unreliable test when testing with either -A or with a pypy host

diff --git a/pypy/module/_vmprof/test/test__vmprof.py b/pypy/module/_vmprof/test/test__vmprof.py
--- a/pypy/module/_vmprof/test/test__vmprof.py
+++ b/pypy/module/_vmprof/test/test__vmprof.py
@@ -1,3 +1,4 @@
+import sys
 from rpython.tool.udir import udir
 from pypy.tool.pytest.objspace import gettestobjspace
 
@@ -7,6 +8,8 @@
     def setup_class(cls):
         cls.w_tmpfilename = cls.space.wrap(str(udir.join('test__vmprof.1')))
         cls.w_tmpfilename2 = cls.space.wrap(str(udir.join('test__vmprof.2')))
+        cls.w_plain = cls.space.wrap(not cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
 
     def test_import_vmprof(self):
         tmpfile = open(self.tmpfilename, 'wb')
@@ -117,6 +120,8 @@
         assert _vmprof.get_profile_path() is None
 
     def test_stop_sampling(self):
+        if not self.plain:
+            skip("unreliable test except on CPython without -A")
         import os
         import _vmprof
         tmpfile = open(self.tmpfilename, 'wb')

From pypy.commits at gmail.com  Tue Aug  1 06:49:25 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 01 Aug 2017 03:49:25 -0700 (PDT)
Subject: [pypy-commit] pypy default: Fix the description (there was never a
 --nopax option; there was only one temporarily inside that branch)
Message-ID: <59805cb5.b288df0a.254f8.243a@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92013:8a0f195540eb
Date: 2017-08-01 12:48 +0200
http://bitbucket.org/pypy/pypy/changeset/8a0f195540eb/

Log:	Fix the description (there was never a --nopax option; there was
	only one temporarily inside that branch)

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -43,7 +43,9 @@
 
 .. branch: nopax
 
-Deleted ``--nopax`` option
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
 
 .. branch: pypy_bytearray
 

From pypy.commits at gmail.com  Tue Aug  1 09:17:35 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 06:17:35 -0700 (PDT)
Subject: [pypy-commit] pypy default: improve cross-referencing in
 documentation
Message-ID: <59807f6f.d47d1c0a.29b66.87fc@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92016:54cc281fb04e
Date: 2017-08-01 16:16 +0300
http://bitbucket.org/pypy/pypy/changeset/54cc281fb04e/

Log:	improve cross-referencing in documentation

diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------

From pypy.commits at gmail.com  Tue Aug  1 09:17:33 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 06:17:33 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-refactor-tp_dealloc: fix translation by
 adding immortal to malloc in rpython
Message-ID: <59807f6d.44561c0a.34ce9.78b6@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-refactor-tp_dealloc
Changeset: r92015:e3219c61e7a8
Date: 2017-07-31 14:17 +0300
http://bitbucket.org/pypy/pypy/changeset/e3219c61e7a8/

Log:	fix translation by adding immortal to malloc in rpython

diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -204,7 +204,6 @@
 def getsetdescr_dealloc(space, obj):
     from pypy.module.cpyext.object import _dealloc
     py_getsetdescr = rffi.cast(PyGetSetDescrObject, obj)
-    xxx
     _dealloc(space, obj)
 
 def methoddescr_attach(space, py_obj, w_obj, w_userdata=None):
diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py
--- a/rpython/rtyper/lltypesystem/lltype.py
+++ b/rpython/rtyper/lltypesystem/lltype.py
@@ -2208,7 +2208,7 @@
     return _ptr(Ptr(T), o, solid)
 
 @analyzer_for(malloc)
-def ann_malloc(s_T, s_n=None, s_flavor=None, s_zero=None,
+def ann_malloc(s_T, s_n=None, s_flavor=None, s_immortal=None, s_zero=None,
                s_track_allocation=None, s_add_memory_pressure=None,
                s_nonmovable=None):
     assert (s_n is None or s_n.knowntype == int
diff --git a/rpython/rtyper/rbuiltin.py b/rpython/rtyper/rbuiltin.py
--- a/rpython/rtyper/rbuiltin.py
+++ b/rpython/rtyper/rbuiltin.py
@@ -347,19 +347,20 @@
 # annotation of low-level types
 
 @typer_for(lltype.malloc)
-def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None,
-                 i_add_memory_pressure=None, i_nonmovable=None):
+def rtype_malloc(hop, i_flavor=None, i_immortal=None, i_zero=None,
+        i_track_allocation=None, i_add_memory_pressure=None, i_nonmovable=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
     kwds_v = parse_kwds(
         hop,
         (i_flavor, lltype.Void),
+        (i_immortal, None),
         (i_zero, None),
         (i_track_allocation, None),
         (i_add_memory_pressure, None),
         (i_nonmovable, None))
-    (v_flavor, v_zero, v_track_allocation,
+    (v_flavor, v_immortal, v_zero, v_track_allocation,
      v_add_memory_pressure, v_nonmovable) = kwds_v
     flags = {'flavor': 'gc'}
     if v_flavor is not None:

From pypy.commits at gmail.com  Tue Aug  1 09:12:58 2017
From: pypy.commits at gmail.com (Dodan)
Date: Tue, 01 Aug 2017 06:12:58 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Fix for the test_EINTR
 errors on sendmsg and recvmsg
Message-ID: <59807e5a.10301c0a.a3247.6eeb@mx.google.com>

Author: Dodan Mihai <mihai.dodan at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92014:a8d5fdebbce7
Date: 2017-08-01 16:11 +0300
http://bitbucket.org/pypy/pypy/changeset/a8d5fdebbce7/

Log:	Fix for the test_EINTR errors on sendmsg and recvmsg

diff --git a/pypy/module/_socket/interp_socket.py b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -473,27 +473,27 @@
             raise oefmt(space.w_ValueError, "negative buffer size in recvmsg()")
         if ancbufsize < 0:
             raise oefmt(space.w_ValueError, "invalid ancillary data buffer length")
-        try:
-            tuple = self.sock.recvmsg(message_size, ancbufsize, flags)
-            message = space.newbytes(tuple[0])
-            # print(tuple[0])
-            list = []
-            for l in tuple[1]:
-                tup = space.newtuple([space.newint(l[0]), space.newint(l[1]), space.newbytes(l[2])])
-                list.append(tup)
+        while True:
+            try:
+                tuple = self.sock.recvmsg(message_size, ancbufsize, flags)
+                message = space.newbytes(tuple[0])
+                list = []
+                for l in tuple[1]:
+                    tup = space.newtuple([space.newint(l[0]), space.newint(l[1]), space.newbytes(l[2])])
+                    list.append(tup)
 
-            anc = space.newlist(list)
+                anc = space.newlist(list)
 
-            flag = space.newint(tuple[2])
-            if (tuple[3] is not None):
-                address = addr_as_object(tuple[3], self.sock.fd, space)
-            else:
-                address = space.w_None
-
-            rettup = space.newtuple([message, anc, flag, address])
-            return rettup
-        except SocketError as e:
-            converted_error(space, e, eintr_retry=True)
+                flag = space.newint(tuple[2])
+                if (tuple[3] is not None):
+                    address = addr_as_object(tuple[3], self.sock.fd, space)
+                else:
+                    address = space.w_None
+                rettup = space.newtuple([message, anc, flag, address])
+                break
+            except SocketError as e:
+                converted_error(space, e, eintr_retry=True)
+        return rettup
 
     @unwrap_spec(data='bufferstr', flags=int)
     def send_w(self, space, data, flags=0):
@@ -566,110 +566,117 @@
         :return: Bytes sent from the message
         """
         # Get the flag and address from the object space
-        flags = 0
-        if space.is_none(w_flags) is False:
-            flags = space.int_w(w_flags)
+        while True:
+            try:
+                flags = 0
+                if space.is_none(w_flags) is False:
+                    flags = space.int_w(w_flags)
 
-        address = None
-        if space.is_none(w_address) is False:
-            address = self.addr_from_object(space, w_address)
+                address = None
+                if space.is_none(w_address) is False:
+                    address = self.addr_from_object(space, w_address)
 
-        # find data's type in the ObjectSpace and get a list of string out of it.
-        data = []
-        if (w_data.typedef.name == 'list'):
-            for i in w_data.getitems():
-                if space.isinstance_w(i,space.w_bytes):
-                    data.append(space.bytes_w(i))
+                # find data's type in the ObjectSpace and get a list of string out of it.
+                data = []
+                if (w_data.typedef.name == 'list'):
+                    for i in w_data.getitems():
+                        if space.isinstance_w(i, space.w_bytes):
+                            data.append(space.bytes_w(i))
+                        else:
+                            if (i.typedef.name == 'array.array'):
+                                data.append(space.bytes_w(i.descr_tobytes(space)))
+                            else:
+                                if (i.typedef.name == 'memoryview'):
+                                    data.append(space.bytes_w(i.descr_tobytes(space)))
+                                else:
+                                    raise oefmt(space.w_TypeError, "a bytes-like object is required")
                 else:
-                    if (i.typedef.name == 'array.array'):
-                        data.append(space.bytes_w(i.descr_tobytes(space)))
+                    while True:
+                        try:
+                            if (space.is_generator(w_data) is False):
+                                raise oefmt(space.w_TypeError, "sendmsg(): argument 1 must be iterable")
+                            i = space.next(w_data)
+                            if space.isinstance_w(i, space.w_bytes):
+                                data.append(space.bytes_w(i))
+                            else:
+                                if (i.typedef.name == 'array.array'):
+                                    data.append(space.bytes_w(i.descr_tobytes(space)))
+                                else:
+                                    if (i.typedef.name == 'memoryview'):
+                                        data.append(space.bytes_w(i.descr_tobytes(space)))
+                                    else:
+                                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                        except OperationError as e:
+                            if not e.match(space, space.w_StopIteration):
+                                raise
+                            break
+
+                # find the ancillary's type in the ObjectSpace and get a list of tuples out of it.
+                ancillary = []
+                if w_ancillary is not None:
+                    if (space.isinstance_w(w_ancillary, space.w_list)):
+                        for i in w_ancillary.getitems():
+                            if (space.isinstance_w(i, space.w_tuple) is False):
+                                raise oefmt(space.w_TypeError, "[sendmsg() ancillary data items]() argument must be sequence")
+                            if (space.len_w(i) == 3):
+                                level = space.int_w(space.getitem(i, space.newint(0)))
+                                type = space.int_w(space.getitem(i, space.newint(1)))
+                                if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
+                                    cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
+                                else:
+                                    if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
+                                        cont = space.bytes_w(space.getitem(i, space.newint(2)))
+                                    else:
+                                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                                tup = (level, type, cont)
+                                ancillary.append(tup)
+                            else:
+                                raise oefmt(space.w_TypeError,
+                                            "[sendmsg() ancillary data items]() argument must be sequence of length 3")
+
                     else:
-                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
-        else:
-            while True:
-                try:
-                    if (space.is_generator(w_data) is False):
-                        raise oefmt(space.w_TypeError, "sendmsg(): argument 1 must be iterable")
-                    i = space.next(w_data)
-                    if space.isinstance_w(i, space.w_bytes):
-                        data.append(space.bytes_w(i))
-                    else:
-                        if (i.typedef.name == 'array.array'):
-                            data.append(space.bytes_w(i.descr_tobytes(space)))
-                        else:
-                            raise oefmt(space.w_TypeError, "a bytes-like object is required")
-                except OperationError as e:
-                    if not e.match(space,space.w_StopIteration):
-                        raise
-                    break
+                        while True:
+                            try:
+                                if (space.is_generator(w_ancillary) is False):
+                                    raise oefmt(space.w_TypeError,
+                                                "[sendmsg() ancillary data items]() argument must be sequence")
+                                i = space.next(w_ancillary)
+                                if (space.isinstance_w(i, space.w_tuple) is False):
+                                    raise oefmt(space.w_TypeError,
+                                                "[sendmsg() ancillary data items]() argument must be sequence of length 3")
+                                if (space.len_w(i) != 3):
+                                    raise oefmt(space.w_TypeError,
+                                                "[sendmsg() ancillary data items]() argument must be sequence of length 3")
+                            except OperationError as e:
+                                if not e.match(space, space.w_StopIteration):
+                                    raise
+                                break
+                            level = space.int_w(space.getitem(i, space.newint(0)))
+                            type = space.int_w(space.getitem(i, space.newint(1)))
+                            if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
+                                cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
+                            else:
+                                if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
+                                    cont = space.bytes_w(space.getitem(i, space.newint(2)))
+                                else:
+                                    raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                            tup = (level, type, cont)
+                            ancillary.append(tup)
 
-        # find the ancillary's type in the ObjectSpace and get a list of tuples out of it.
-        ancillary = []
-        if w_ancillary is not None:
-            if (space.isinstance_w(w_ancillary,space.w_list)):
-                for i in w_ancillary.getitems():
-                    if (space.isinstance_w(i, space.w_tuple) is False):
-                        raise oefmt(space.w_TypeError,"[sendmsg() ancillary data items]() argument must be sequence")
-                    if (space.len_w(i) == 3):
-                        level = space.int_w(space.getitem(i, space.newint(0)))
-                        type = space.int_w(space.getitem(i, space.newint(1)))
-                        if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
-                            cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
-                        else:
-                            if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
-                                cont = space.bytes_w(space.getitem(i, space.newint(2)))
-                            else:
-                                raise oefmt(space.w_TypeError,"a bytes-like object is required")
-                        tup = (level, type, cont)
-                        ancillary.append(tup)
-                    else:
-                        raise oefmt(space.w_TypeError,
-                                    "[sendmsg() ancillary data items]() argument must be sequence of length 3")
 
-            else:
-                while True:
-                    try:
-                        if (space.is_generator(w_ancillary) is False):
-                            raise oefmt(space.w_TypeError,
-                                        "[sendmsg() ancillary data items]() argument must be sequence")
-                        i = space.next(w_ancillary)
-                        if (space.isinstance_w(i, space.w_tuple) is False):
-                            raise oefmt(space.w_TypeError,
-                                        "[sendmsg() ancillary data items]() argument must be sequence of length 3")
-                        if (space.len_w(i) != 3):
-                            raise oefmt(space.w_TypeError,
-                                        "[sendmsg() ancillary data items]() argument must be sequence of length 3")
-                    except OperationError as e:
-                        if not e.match(space,space.w_StopIteration):
-                            raise
-                        break
-                    level = space.int_w(space.getitem(i, space.newint(0)))
-                    type = space.int_w(space.getitem(i, space.newint(1)))
-                    if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
-                        cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
-                    else:
-                        if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
-                            cont = space.bytes_w(space.getitem(i, space.newint(2)))
-                        else:
-                            raise oefmt(space.w_TypeError, "a bytes-like object is required")
-                    tup = (level, type, cont)
-                    ancillary.append(tup)
+                count = self.sock.sendmsg(data, ancillary, flags, address)
+                if count < 0:
+                    if (count == -1000):
+                        raise oefmt(space.w_OSError, "sending multiple control messages not supported")
+                    if (count == -1001):
+                        raise oefmt(space.w_OSError, "ancillary data item too large")
+                    if (count == -1002):
+                        raise oefmt(space.w_OSError, "too much ancillary data")
+                break
+            except SocketError as e:
+                converted_error(space, e, eintr_retry=True)
 
-        try:
-            count = self.sock.sendmsg(data, ancillary, flags, address)
-            if count < 0:
-                if (count == -1000):
-                    raise oefmt(space.w_OSError, "sending multiple control messages not supported")
-                if (count == -1001):
-                    raise oefmt(space.w_OSError, "ancillary data item too large")
-                if (count == -1002):
-                    raise oefmt(space.w_OSError, "too much ancillary data")
-
-            return space.newint(count)
-        except SocketError as e:
-            converted_error(space, e, eintr_retry=True)
-
-
+        return space.newint(count)
 
     @unwrap_spec(flag=int)
     def setblocking_w(self, flag):
diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py
--- a/rpython/rlib/_rsocket_rffi.py
+++ b/rpython/rlib/_rsocket_rffi.py
@@ -514,15 +514,15 @@
                                   int flags,
                                   struct sockaddr* address,
                                   socklen_t* addrlen,
-                                  int** length_of_messages,
+                                  long** length_of_messages,
                                   char** messages,
-                                  int* no_of_messages,
-                                  int* size_of_ancillary,
+                                  long* no_of_messages,
+                                  long* size_of_ancillary,
                                   long** levels,
                                   long** types,
                                   char** file_descr,
                                   long** descr_per_ancillary,
-                                  int* retflag)
+                                  long* retflag)
 
         {
 
@@ -629,13 +629,15 @@
             *addrlen = retinfo->addrlen;
 
             // Set the parameters of message
-            *no_of_messages = retinfo->no_of_messages;
-            *size_of_ancillary = retinfo->size_of_ancillary;
-            *length_of_messages = (int*) malloc (sizeof(int) * retinfo->no_of_messages);
-            memcpy(*length_of_messages, retinfo->length_of_messages, sizeof(int) * retinfo->no_of_messages);
+            no_of_messages[0] = retinfo->no_of_messages;
+            size_of_ancillary[0] = retinfo->size_of_ancillary;
+            *length_of_messages = (long*) malloc (sizeof(long) * retinfo->no_of_messages);
+            //memcpy(*length_of_messages, retinfo->length_of_messages, sizeof(int) * retinfo->no_of_messages);
             int counter = 0;
-            for (i=0; i< retinfo->no_of_messages; i++)
+            for (i=0; i< retinfo->no_of_messages; i++){
                 counter += retinfo->length_of_messages[i];
+                length_of_messages[0][i] = retinfo->length_of_messages[i];
+            }
             memset(*messages, 0, sizeof(char) * counter);
             counter = 0;
             for(i=0; i< retinfo->no_of_messages; i++){
@@ -664,7 +666,7 @@
             }
 
             // Set the retflag
-            *retflag = retinfo->retflag;
+            retflag[0] = retinfo->retflag;
 
             // Free the memory
             free(retinfo->address);
@@ -959,7 +961,7 @@
     post_include_bits =[ "RPY_EXTERN "
                          "int sendmsg_implementation(int socket, struct sockaddr* address, socklen_t addrlen, long* length_of_messages, char** messages, int no_of_messages, long* levels, long* types, char** file_descriptors, long* no_of_fds, int control_length, int flag );\n"
                          "RPY_EXTERN "
-                         "int recvmsg_implementation(int socket_fd, int message_size, int ancillary_size, int flags, struct sockaddr* address, socklen_t* addrlen, int** length_of_messages, char** messages, int* no_of_messages, int* size_of_ancillary, long** levels, long** types, char** file_descr, long** descr_per_ancillary, int* flag);\n"
+                         "int recvmsg_implementation(int socket_fd, int message_size, int ancillary_size, int flags, struct sockaddr* address, socklen_t* addrlen, long** length_of_messages, char** messages, long* no_of_messages, long* size_of_ancillary, long** levels, long** types, char** file_descr, long** descr_per_ancillary, long* flag);\n"
                          "static "
                          "int cmsg_min_space(struct msghdr *msg, struct cmsghdr *cmsgh, size_t space);\n"
                          "static "
diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py
--- a/rpython/rlib/rsocket.py
+++ b/rpython/rlib/rsocket.py
@@ -932,7 +932,6 @@
                     address.addrlen = addrlen
                 else:
                     address = None
-                print address
                 data = buf.str(read_bytes)
                 return (data, address)
         raise self.error_handler()
@@ -997,7 +996,7 @@
         retflag[0] = rffi.cast(rffi.SIGNED,0)
 
         # a mask for the SIGNEDP's that need to be cast to int. (long default)
-        LONG_MASK =  2**32 - 1
+        #LONG_MASK =  2**32 - 1
         reply = _c.recvmsg(self.fd, rffi.cast(lltype.Signed,message_size),
                            rffi.cast(lltype.Signed,ancbufsize),rffi.cast(lltype.Signed,flags),
                            addr_p, addrlen_p, len_of_msgs, messages, no_of_messages,size_of_anc,
@@ -1011,7 +1010,7 @@
 
             for i in range(msg_no):
                 x = rffi.cast(rffi.SIGNED,len_of_msgs[0][i])
-                x &= LONG_MASK
+                #x &= LONG_MASK
                 retmsg = rffi.charp2strn(messages[0],x)
 
             offset = 0
@@ -1078,6 +1077,8 @@
 
             if address is not None:
                 address.unlock()
+            if _c.geterrno() == _c.EINTR:
+                raise last_error()
             if (reply == -10000):
                 raise RSocketError("Invalid message size")
             if (reply == -10001):

From pypy.commits at gmail.com  Tue Aug  1 09:22:39 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 06:22:39 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-refactor-tp_dealloc: close dead-end branch
Message-ID: <5980809f.759adf0a.1d3b0.7360@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-refactor-tp_dealloc
Changeset: r92017:98384706fd2b
Date: 2017-08-01 16:21 +0300
http://bitbucket.org/pypy/pypy/changeset/98384706fd2b/

Log:	close dead-end branch


From pypy.commits at gmail.com  Tue Aug  1 14:11:17 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 11:11:17 -0700 (PDT)
Subject: [pypy-commit] pypy default: clean build warnings,
 expand building section
Message-ID: <5980c445.06931c0a.55c0e.e165@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92018:7a17b6d02bb1
Date: 2017-08-01 21:10 +0300
http://bitbucket.org/pypy/pypy/changeset/7a17b6d02bb1/

Log:	clean build warnings, expand building section

diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -152,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -181,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -217,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -64,9 +64,9 @@
 cppyy
 -----
 
-For C++, `cppyy`_ is an automated bindings generator available for both
+For C++, _cppyy_ is an automated bindings generator available for both
 PyPy and CPython.
-``cppyy`` relies on declarations from C++ header files to dynamically
+_cppyy_ relies on declarations from C++ header files to dynamically
 construct Python equivalent classes, functions, variables, etc.
 It is designed for use by large scale programs and supports modern C++.
 With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
@@ -75,8 +75,7 @@
 To install, run ``pip install cppyy``.
 Further details are available in the `full documentation`_.
 
-.. _cppyy: http://cppyy.readthedocs.org/
-.. _`full documentation`: http://cppyy.readthedocs.org/
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
 
 RPython Mixed Modules

From pypy.commits at gmail.com  Tue Aug  1 16:45:27 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 13:45:27 -0700 (PDT)
Subject: [pypy-commit] pypy vmprof-win32: fix compilation on win32
Message-ID: <5980e867.09de1c0a.ebdab.e8a2@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: vmprof-win32
Changeset: r92019:b3ef99751523
Date: 2017-08-01 23:32 +0300
http://bitbucket.org/pypy/pypy/changeset/b3ef99751523/

Log:	fix compilation on win32


From pypy.commits at gmail.com  Tue Aug  1 16:45:30 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 13:45:30 -0700 (PDT)
Subject: [pypy-commit] pypy vmprof-win32: fix compilation on win32
Message-ID: <5980e86a.04a6df0a.37a1b.a492@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: vmprof-win32
Changeset: r92020:16d18de7f580
Date: 2017-08-01 23:22 +0300
http://bitbucket.org/pypy/pypy/changeset/16d18de7f580/

Log:	fix compilation on win32

diff --git a/rpython/rlib/rvmprof/cintf.py b/rpython/rlib/rvmprof/cintf.py
--- a/rpython/rlib/rvmprof/cintf.py
+++ b/rpython/rlib/rvmprof/cintf.py
@@ -23,6 +23,8 @@
     SHARED.join('symboltable.c'),
     SHARED.join('vmprof_unix.c')
 ]
+include_dirs = [SRC, SHARED]
+
 if sys.platform.startswith('linux'):
     separate_module_files += [
        BACKTRACE.join('atomic.c'),
@@ -35,10 +37,12 @@
        BACKTRACE.join('mmapio.c'),
        BACKTRACE.join('posix.c'),
        BACKTRACE.join('sort.c'),
+       SHARED.join('vmprof_mt.c'),
     ]
     _libs = ['dl']
     compile_extra += ['-DVMPROF_UNIX']
     compile_extra += ['-DVMPROF_LINUX']
+    include_dirs.append(BACKTRACE)
 elif sys.platform == 'win32':
     compile_extra = ['-DRPYTHON_VMPROF', '-DVMPROF_WINDOWS']
     separate_module_files = [SHARED.join('vmprof_win.c')]
@@ -48,10 +52,12 @@
     compile_extra += ['-DVMPROF_UNIX']
     compile_extra += ['-DVMPROF_MAC']
     _libs = []
-
+    separate_module_files += [
+       SHARED.join('vmprof_mt.c'),
+    ]
 
 eci_kwds = dict(
-    include_dirs = [SRC, SHARED, BACKTRACE],
+    include_dirs = include_dirs,
     includes = ['rvmprof.h','vmprof_stack.h'],
     libraries = _libs,
     separate_module_files = [
@@ -59,7 +65,6 @@
         SHARED.join('compat.c'),
         SHARED.join('machine.c'),
         SHARED.join('vmp_stack.c'),
-        SHARED.join('vmprof_mt.c'),
         SHARED.join('vmprof_memory.c'),
         SHARED.join('vmprof_common.c'),
         # symbol table already in separate_module_files
diff --git a/rpython/rlib/rvmprof/src/rvmprof.h b/rpython/rlib/rvmprof/src/rvmprof.h
--- a/rpython/rlib/rvmprof/src/rvmprof.h
+++ b/rpython/rlib/rvmprof/src/rvmprof.h
@@ -7,9 +7,8 @@
 
 #ifdef VMPROF_WINDOWS
 #include <crtdefs.h>
-typedef __int64 int64_t;
-typedef unsigned __int64 uint64_t;
-typedef intptr_t ssize_t;
+#include "shared/msiinttypes/inttypes.h"
+#include "shared/msiinttypes/stdint.h"
 #else
 #include <inttypes.h>
 #include <stdint.h>
diff --git a/rpython/rlib/rvmprof/src/shared/machine.c b/rpython/rlib/rvmprof/src/shared/machine.c
--- a/rpython/rlib/rvmprof/src/shared/machine.c
+++ b/rpython/rlib/rvmprof/src/shared/machine.c
@@ -34,7 +34,7 @@
 #endif
 }
 
-long vmp_fd_to_path(int fd, char * buffer, long buffer_len)
+long vmp_fd_to_path(int fd, const char * buffer, long buffer_len)
 {
 #ifdef VMPROF_LINUX
     char proffs[24];
diff --git a/rpython/rlib/rvmprof/src/shared/machine.h b/rpython/rlib/rvmprof/src/shared/machine.h
--- a/rpython/rlib/rvmprof/src/shared/machine.h
+++ b/rpython/rlib/rvmprof/src/shared/machine.h
@@ -14,5 +14,5 @@
  * Writes the filename into buffer. Returns -1 if the platform is not
  * implemented.
  */
-long vmp_fd_to_path(int fd, char * buffer, long buffer_len);
+long vmp_fd_to_path(int fd, const char * buffer, long buffer_len);
 
diff --git a/rpython/rlib/rvmprof/src/shared/vmprof_common.c b/rpython/rlib/rvmprof/src/shared/vmprof_common.c
--- a/rpython/rlib/rvmprof/src/shared/vmprof_common.c
+++ b/rpython/rlib/rvmprof/src/shared/vmprof_common.c
@@ -2,6 +2,7 @@
 
 #include <assert.h>
 #include <errno.h>
+#include <string.h> /* for strerror */
 
 #ifdef RPYTHON_VMPROF
 #ifdef RPYTHON_LL2CTYPES
@@ -34,9 +35,11 @@
 static size_t threads_size_step = 8;
 #endif
 
+#ifndef VMPROF_WINDOWS
 int vmprof_get_itimer_type(void) {
     return itimer_type;
 }
+#endif
 
 int vmprof_is_enabled(void) {
     return is_enabled;
@@ -62,9 +65,11 @@
     profile_interval_usec = value;
 }
 
+#ifndef VMPROF_WINDOWS
 int vmprof_get_signal_type(void) {
     return signal_type;
 }
+#endif
 
 char *vmprof_init(int fd, double interval, int memory,
                   int proflines, const char *interp_name, int native, int real_time)
diff --git a/rpython/rlib/rvmprof/src/shared/vmprof_common.h b/rpython/rlib/rvmprof/src/shared/vmprof_common.h
--- a/rpython/rlib/rvmprof/src/shared/vmprof_common.h
+++ b/rpython/rlib/rvmprof/src/shared/vmprof_common.h
@@ -1,5 +1,6 @@
 #pragma once
 
+
 #include "vmprof.h"
 #include "machine.h"
 #include "compat.h"
@@ -8,6 +9,7 @@
 #include <time.h>
 #include <stdlib.h>
 
+
 #ifdef VMPROF_UNIX
 #include <sys/time.h>
 #include "vmprof_mt.h"
@@ -15,7 +17,9 @@
 #include <pthread.h>
 #endif
 
+#ifndef _MSC_VER
 #include "vmprof_getpc.h"
+#endif
 
 #ifdef VMPROF_LINUX
 #include <syscall.h>
@@ -109,3 +113,4 @@
 int broadcast_signal_for_threads(void);
 int is_main_thread(void);
 #endif
+
diff --git a/rpython/rlib/rvmprof/src/shared/vmprof_memory.c b/rpython/rlib/rvmprof/src/shared/vmprof_memory.c
--- a/rpython/rlib/rvmprof/src/shared/vmprof_memory.c
+++ b/rpython/rlib/rvmprof/src/shared/vmprof_memory.c
@@ -14,7 +14,11 @@
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <fcntl.h>
+#ifdef VMPROF_WINDOWS
+#include "shared/msiinttypes/stdint.h"
+#else
 #include <unistd.h>
+#endif
 #include <string.h>
 /* On '''normal''' Unices we can get RSS from '/proc/<pid>/status'. */
 static int proc_file = -1;
diff --git a/rpython/rlib/rvmprof/src/vmprof_stack.h b/rpython/rlib/rvmprof/src/vmprof_stack.h
--- a/rpython/rlib/rvmprof/src/vmprof_stack.h
+++ b/rpython/rlib/rvmprof/src/vmprof_stack.h
@@ -1,7 +1,7 @@
 #pragma once
 
 #ifdef _WIN32
-#define intptr_t long // XXX windows VC++ 2008 lacks stdint.h
+#include "msiinttypes/stdint.h"
 #else
 #include <unistd.h>
 #endif

From pypy.commits at gmail.com  Tue Aug  1 16:45:32 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 13:45:32 -0700 (PDT)
Subject: [pypy-commit] pypy vmprof-win32: close files
Message-ID: <5980e86c.3888df0a.6a4a5.6afc@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: vmprof-win32
Changeset: r92021:91789212fc18
Date: 2017-08-01 23:24 +0300
http://bitbucket.org/pypy/pypy/changeset/91789212fc18/

Log:	close files

diff --git a/pypy/module/_vmprof/test/test__vmprof.py b/pypy/module/_vmprof/test/test__vmprof.py
--- a/pypy/module/_vmprof/test/test__vmprof.py
+++ b/pypy/module/_vmprof/test/test__vmprof.py
@@ -1,6 +1,5 @@
 import sys
 from rpython.tool.udir import udir
-from pypy.tool.pytest.objspace import gettestobjspace
 
 class AppTestVMProf(object):
     spaceconfig = {'usemodules': ['_vmprof', 'struct']}
@@ -12,82 +11,90 @@
             '__pypy__' not in sys.builtin_module_names)
 
     def test_import_vmprof(self):
+        import _vmprof
         tmpfile = open(self.tmpfilename, 'wb')
-        tmpfileno = tmpfile.fileno()
         tmpfile2 = open(self.tmpfilename2, 'wb')
-        tmpfileno2 = tmpfile2.fileno()
+        try:
+            tmpfileno = tmpfile.fileno()
+            tmpfileno2 = tmpfile2.fileno()
 
-        import struct, sys, gc
+            import struct, gc
 
-        WORD = struct.calcsize('l')
+            WORD = struct.calcsize('l')
 
-        def count(s):
-            i = 0
-            count = 0
-            i += 5 * WORD # header
-            assert s[i    ] == '\x05'    # MARKER_HEADER
-            assert s[i + 1] == '\x00'    # 0
-            assert s[i + 2] == '\x06'    # VERSION_TIMESTAMP
-            assert s[i + 3] == '\x08'    # PROFILE_RPYTHON
-            assert s[i + 4] == chr(4)    # len('pypy')
-            assert s[i + 5: i + 9] == 'pypy'
-            i += 9
-            while i < len(s):
-                if s[i] == '\x03':
-                    break
-                elif s[i] == '\x01':
-                    i += 1
-                    _, size = struct.unpack("ll", s[i:i + 2 * WORD])
-                    i += 2 * WORD + size * struct.calcsize("P")
-                    i += WORD    # thread id
-                elif s[i] == '\x02':
-                    i += 1
-                    _, size = struct.unpack("ll", s[i:i + 2 * WORD])
-                    count += 1
-                    i += 2 * WORD + size
-                elif s[i] == '\x06':
-                    print(s[i:i+24])
-                    i += 1+8+8+8
-                elif s[i] == '\x07':
-                    i += 1
-                    # skip string
-                    size, = struct.unpack("l", s[i:i + WORD])
-                    i += WORD+size
-                    # skip string
-                    size, = struct.unpack("l", s[i:i + WORD])
-                    i += WORD+size
-                else:
-                    raise AssertionError(ord(s[i]))
-            return count
+            def count(s):
+                i = 0
+                count = 0
+                i += 5 * WORD # header
+                assert s[i    ] == '\x05'    # MARKER_HEADER
+                assert s[i + 1] == '\x00'    # 0
+                assert s[i + 2] == '\x06'    # VERSION_TIMESTAMP
+                assert s[i + 3] == '\x08'    # PROFILE_RPYTHON
+                assert s[i + 4] == chr(4)    # len('pypy')
+                assert s[i + 5: i + 9] == 'pypy'
+                i += 9
+                while i < len(s):
+                    if s[i] == '\x03':
+                        break
+                    elif s[i] == '\x01':
+                        i += 1
+                        _, size = struct.unpack("ll", s[i:i + 2 * WORD])
+                        i += 2 * WORD + size * struct.calcsize("P")
+                        i += WORD    # thread id
+                    elif s[i] == '\x02':
+                        i += 1
+                        _, size = struct.unpack("ll", s[i:i + 2 * WORD])
+                        count += 1
+                        i += 2 * WORD + size
+                    elif s[i] == '\x06':
+                        print(s[i:i+24])
+                        i += 1+8+8+8
+                    elif s[i] == '\x07':
+                        i += 1
+                        # skip string
+                        size, = struct.unpack("l", s[i:i + WORD])
+                        i += WORD+size
+                        # skip string
+                        size, = struct.unpack("l", s[i:i + WORD])
+                        i += WORD+size
+                    else:
+                        raise AssertionError(ord(s[i]))
+                return count
 
-        import _vmprof
-        gc.collect()  # try to make the weakref list deterministic
-        gc.collect()  # by freeing all dead code objects
-        _vmprof.enable(tmpfileno, 0.01, 0, 0, 0, 0)
-        _vmprof.disable()
-        s = open(self.tmpfilename, 'rb').read()
-        no_of_codes = count(s)
-        assert no_of_codes > 10
-        d = {}
+            gc.collect()  # try to make the weakref list deterministic
+            gc.collect()  # by freeing all dead code objects
+            _vmprof.enable(tmpfileno, 0.01, 0, 0, 0, 0)
+            _vmprof.disable()
+            with open(self.tmpfilename, 'rb') as fid:
+                s = fid.read()
+            no_of_codes = count(s)
+            assert no_of_codes > 10
+            d = {}
 
-        exec """def foo():
-            pass
-        """ in d
+            exec """def foo():
+                pass
+            """ in d
 
-        gc.collect()
-        gc.collect()
-        _vmprof.enable(tmpfileno2, 0.01, 0, 0, 0, 0)
+            gc.collect()
+            gc.collect()
+            _vmprof.enable(tmpfileno2, 0.01, 0, 0, 0, 0)
 
-        exec """def foo2():
-            pass
-        """ in d
+            exec """def foo2():
+                pass
+            """ in d
 
-        _vmprof.disable()
-        s = open(self.tmpfilename2, 'rb').read()
-        no_of_codes2 = count(s)
-        assert "py:foo:" in s
-        assert "py:foo2:" in s
-        assert no_of_codes2 >= no_of_codes + 2 # some extra codes from tests
+            _vmprof.disable()
+            with open(self.tmpfilename2, 'rb') as fid:
+                s = fid.read()
+            no_of_codes2 = count(s)
+            assert "py:foo:" in s
+            assert "py:foo2:" in s
+            assert no_of_codes2 >= no_of_codes + 2 # some extra codes from tests
+        finally:
+            if _vmprof.is_enabled():
+                _vmprof.disable()
+            tmpfile.close()
+            tmpfile2.close()
 
     def test_enable_ovf(self):
         import _vmprof
@@ -100,31 +107,34 @@
 
     def test_is_enabled(self):
         import _vmprof
-        tmpfile = open(self.tmpfilename, 'wb')
-        assert _vmprof.is_enabled() is False
-        _vmprof.enable(tmpfile.fileno(), 0.01, 0, 0, 0, 0)
-        assert _vmprof.is_enabled() is True
-        _vmprof.disable()
-        assert _vmprof.is_enabled() is False
+        with open(self.tmpfilename, 'wb') as tmpfile:
+            assert _vmprof.is_enabled() is False
+            _vmprof.enable(tmpfile.fileno(), 0.01, 0, 0, 0, 0)
+            assert _vmprof.is_enabled() is True
+            _vmprof.disable()
+            assert _vmprof.is_enabled() is False
 
     def test_get_profile_path(self):
         import _vmprof
-        tmpfile = open(self.tmpfilename, 'wb')
-        assert _vmprof.get_profile_path() is None
-        _vmprof.enable(tmpfile.fileno(), 0.01, 0, 0, 0, 0)
-        path = _vmprof.get_profile_path()
-        if path != tmpfile.name:
-            with open(path, "rb") as fd1:
-                assert fd1.read() == tmpfile.read()
-        _vmprof.disable()
-        assert _vmprof.get_profile_path() is None
+        try:
+            with open(self.tmpfilename, 'wb') as tmpfile:
+                assert _vmprof.get_profile_path() is None
+                _vmprof.enable(tmpfile.fileno(), 0.01, 0, 0, 0, 0)
+                path = _vmprof.get_profile_path()
+                if path != tmpfile.name:
+                    with open(path, "rb") as fd1:
+                        assert fd1.read() == tmpfile.read()
+                _vmprof.disable()
+                assert _vmprof.get_profile_path() is None
+        finally:
+            if _vmprof.is_enabled():
+                _vmprof.disable()
 
     def test_stop_sampling(self):
         if not self.plain:
             skip("unreliable test except on CPython without -A")
         import os
         import _vmprof
-        tmpfile = open(self.tmpfilename, 'wb')
         native = 1
         def f():
             import sys
@@ -132,19 +142,20 @@
             j = sys.maxsize
             for i in range(500):
                 j = math.sqrt(j)
-        _vmprof.enable(tmpfile.fileno(), 0.01, 0, native, 0, 0)
-        # get_vmprof_stack() always returns 0 here!
-        # see vmprof_common.c and assume RPYTHON_LL2CTYPES is defined!
-        f()
-        fileno = _vmprof.stop_sampling()
-        pos = os.lseek(fileno, 0, os.SEEK_CUR)
-        f()
-        pos2 = os.lseek(fileno, 0, os.SEEK_CUR)
-        assert pos == pos2
-        _vmprof.start_sampling()
-        f()
-        fileno = _vmprof.stop_sampling()
-        pos3 = os.lseek(fileno, 0, os.SEEK_CUR)
-        assert pos3 > pos
-        _vmprof.disable()
+        with open(self.tmpfilename, 'wb') as tmpfile:
+            _vmprof.enable(tmpfile.fileno(), 0.01, 0, native, 0, 0)
+            # get_vmprof_stack() always returns 0 here!
+            # see vmprof_common.c and assume RPYTHON_LL2CTYPES is defined!
+            f()
+            fileno = _vmprof.stop_sampling()
+            pos = os.lseek(fileno, 0, os.SEEK_CUR)
+            f()
+            pos2 = os.lseek(fileno, 0, os.SEEK_CUR)
+            assert pos == pos2
+            _vmprof.start_sampling()
+            f()
+            fileno = _vmprof.stop_sampling()
+            pos3 = os.lseek(fileno, 0, os.SEEK_CUR)
+            assert pos3 > pos
+            _vmprof.disable()
 

From pypy.commits at gmail.com  Tue Aug  1 22:35:48 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 01 Aug 2017 19:35:48 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Add space.appdef()
Message-ID: <59813a84.8e951c0a.e87d6.a2e2@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92022:93ef9a8f432f
Date: 2017-08-01 16:06 +0100
http://bitbucket.org/pypy/pypy/changeset/93ef9a8f432f/

Log:	Add space.appdef()

diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1,4 +1,5 @@
 import sys
+import py
 
 from rpython.rlib.cache import Cache
 from rpython.tool.uid import HUGEVAL_BYTES
@@ -1338,6 +1339,20 @@
             self.setitem(w_globals, w_key, self.builtin)
         return statement.exec_code(self, w_globals, w_locals)
 
+    @not_rpython
+    def appdef(self, source):
+        '''Create interp-level function object from app-level source.
+
+        The source should be in the same format as for space.appexec():
+            """(foo, bar): return 'baz'"""
+        '''
+        source = source.lstrip()
+        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
+        source = py.code.Source("def anonymous%s\n" % source)
+        w_glob = self.newdict(module=True)
+        self.exec_(str(source), w_glob, w_glob)
+        return self.getitem(w_glob, self.newtext('anonymous'))
+
     @specialize.arg(2)
     def appexec(self, posargs_w, source):
         """ return value from executing given source at applevel.
@@ -1926,15 +1941,7 @@
 class AppExecCache(SpaceCache):
     @not_rpython
     def build(cache, source):
-        space = cache.space
-        # XXX will change once we have our own compiler
-        import py
-        source = source.lstrip()
-        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
-        source = py.code.Source("def anonymous%s\n" % source)
-        w_glob = space.newdict(module=True)
-        space.exec_(str(source), w_glob, w_glob)
-        return space.getitem(w_glob, space.newtext('anonymous'))
+        return cache.space.appdef(source)
 
 
 # Table describing the regular part of the interface of object spaces,

From pypy.commits at gmail.com  Tue Aug  1 22:35:50 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 01 Aug 2017 19:35:50 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: fix translation by adding
 immortal to malloc in rpython
Message-ID: <59813a86.415f1c0a.c4ecf.ec03@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92023:c1d2c7124ad5
Date: 2017-07-31 14:17 +0300
http://bitbucket.org/pypy/pypy/changeset/c1d2c7124ad5/

Log:	fix translation by adding immortal to malloc in rpython (grafted
	from e3219c61e7a801f5257cba1a9df98aa1277cf102)

diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py
--- a/rpython/rtyper/lltypesystem/lltype.py
+++ b/rpython/rtyper/lltypesystem/lltype.py
@@ -2208,7 +2208,7 @@
     return _ptr(Ptr(T), o, solid)
 
 @analyzer_for(malloc)
-def ann_malloc(s_T, s_n=None, s_flavor=None, s_zero=None,
+def ann_malloc(s_T, s_n=None, s_flavor=None, s_immortal=None, s_zero=None,
                s_track_allocation=None, s_add_memory_pressure=None,
                s_nonmovable=None):
     assert (s_n is None or s_n.knowntype == int
diff --git a/rpython/rtyper/rbuiltin.py b/rpython/rtyper/rbuiltin.py
--- a/rpython/rtyper/rbuiltin.py
+++ b/rpython/rtyper/rbuiltin.py
@@ -347,19 +347,20 @@
 # annotation of low-level types
 
 @typer_for(lltype.malloc)
-def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None,
-                 i_add_memory_pressure=None, i_nonmovable=None):
+def rtype_malloc(hop, i_flavor=None, i_immortal=None, i_zero=None,
+        i_track_allocation=None, i_add_memory_pressure=None, i_nonmovable=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
     kwds_v = parse_kwds(
         hop,
         (i_flavor, lltype.Void),
+        (i_immortal, None),
         (i_zero, None),
         (i_track_allocation, None),
         (i_add_memory_pressure, None),
         (i_nonmovable, None))
-    (v_flavor, v_zero, v_track_allocation,
+    (v_flavor, v_immortal, v_zero, v_track_allocation,
      v_add_memory_pressure, v_nonmovable) = kwds_v
     flags = {'flavor': 'gc'}
     if v_flavor is not None:

From pypy.commits at gmail.com  Tue Aug  1 22:35:52 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 01 Aug 2017 19:35:52 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Add option to prevent
 caching in space.appexec() (for cpyext tests)
Message-ID: <59813a88.4588df0a.ca982.6d4e@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92024:d7afdd455feb
Date: 2017-08-02 01:55 +0100
http://bitbucket.org/pypy/pypy/changeset/d7afdd455feb/

Log:	Add option to prevent caching in space.appexec() (for cpyext tests)

diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1354,7 +1354,7 @@
         return self.getitem(w_glob, self.newtext('anonymous'))
 
     @specialize.arg(2)
-    def appexec(self, posargs_w, source):
+    def appexec(self, posargs_w, source, cache=True):
         """ return value from executing given source at applevel.
             The source must look like
                '''(x, y):
@@ -1362,7 +1362,11 @@
                        return result
                '''
         """
-        w_func = self.fromcache(AppExecCache).getorbuild(source)
+        if cache:
+            w_func = self.fromcache(AppExecCache).getorbuild(source)
+        else:
+            # NB: since appdef() is not-RPython, using cache=False also is.
+            w_func = self.appdef(source)
         args = Arguments(self, list(posargs_w))
         return self.call_args(w_func, args)
 

From pypy.commits at gmail.com  Tue Aug  1 22:35:54 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 01 Aug 2017 19:35:54 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Move 2 tests from app- to
 interp-level to fix refcounting issues
Message-ID: <59813a8a.93b5df0a.ff92c.2148@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92025:d893df676f94
Date: 2017-08-02 03:35 +0100
http://bitbucket.org/pypy/pypy/changeset/d893df676f94/

Log:	Move 2 tests from app- to interp-level to fix refcounting issues

diff --git a/pypy/module/cpyext/test/test_dictobject.py b/pypy/module/cpyext/test/test_dictobject.py
--- a/pypy/module/cpyext/test/test_dictobject.py
+++ b/pypy/module/cpyext/test/test_dictobject.py
@@ -111,70 +111,14 @@
             PyDict_Update(space, w_d, w_d2)
         assert space.unwrap(w_d) == dict(a='b') # unchanged
 
-    def test_iter(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        ppos[0] = 0
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        try:
-            w_copy = space.newdict()
-            while PyDict_Next(space, w_dict, ppos, pkey, pvalue):
-                w_key = from_ref(space, pkey[0])
-                w_value = from_ref(space, pvalue[0])
-                space.setitem(w_copy, w_key, w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.len(w_copy), space.len(w_dict))
-        assert space.eq_w(w_copy, w_dict)
-
-    def test_iterkeys(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        keys_w = []
-        values_w = []
-        try:
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, pkey, None):
-                w_key = from_ref(space, pkey[0])
-                keys_w.append(w_key)
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, None, pvalue):
-                w_value = from_ref(space, pvalue[0])
-                values_w.append(w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.newlist(keys_w),
-                          space.call_method(w_dict, "keys"))
-        assert space.eq_w(space.newlist(values_w),
-                          space.call_method(w_dict, "values"))
-
     def test_dictproxy(self, space):
-        w_dict = space.sys.get('modules')
+        w_dict = space.appexec([], """(): return {1: 2, 3: 4}""")
         w_proxy = PyDictProxy_New(space, w_dict)
-        assert space.contains_w(w_proxy, space.wrap('sys'))
+        assert space.contains_w(w_proxy, space.newint(1))
         raises(OperationError, space.setitem,
-               w_proxy, space.wrap('sys'), space.w_None)
+               w_proxy, space.newint(1), space.w_None)
         raises(OperationError, space.delitem,
-               w_proxy, space.wrap('sys'))
+               w_proxy, space.newint(1))
         raises(OperationError, space.call_method, w_proxy, 'clear')
         assert PyDictProxy_Check(space, w_proxy)
 
@@ -243,6 +187,59 @@
         d = {"a": 1}
         raises(AttributeError, module.update, d, [("c", 2)])
 
+    def test_iter(self):
+        module = self.import_extension('foo', [
+            ("copy", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value;
+             PyObject* copy = PyDict_New();
+             while (PyDict_Next(args, &pos, &key, &value))
+             {
+                if (PyDict_SetItem(copy, key, value) < 0)
+                {
+                    Py_DecRef(copy);
+                    return NULL;
+                }
+             }
+             return copy;
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        copy = module.copy(d)
+        assert len(copy) == len(d)
+        assert copy == d
+
+    def test_iterkeys(self):
+        module = self.import_extension('foo', [
+            ("keys_and_values", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value;
+             PyObject* keys = PyList_New(0);
+             while (PyDict_Next(args, &pos, &key, NULL))
+             {
+                if (PyList_Append(keys, key) < 0)
+                {
+                    Py_DecRef(keys);
+                    return NULL;
+                }
+             }
+             pos = 0;
+             PyObject* values = PyList_New(0);
+             while (PyDict_Next(args, &pos, NULL, &value))
+             {
+                if (PyList_Append(values, value) < 0)
+                {
+                    Py_DecRef(keys);
+                    Py_DecRef(values);
+                    return NULL;
+                }
+             }
+             return Py_BuildValue("(NN)", keys, values);
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        assert module.keys_and_values(d) == (d.keys(), d.values())
+
     def test_typedict2(self):
         module = self.import_extension('foo', [
             ("get_type_dict", "METH_O",
@@ -255,6 +252,7 @@
             ])
         d = module.get_type_dict(1)
         assert d['real'].__get__(1, 1) == 1
+
     def test_advanced(self):
         module = self.import_extension('foo', [
             ("dict_len", "METH_O",
@@ -266,7 +264,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 3 || !dict || 
+                if (PyTuple_Size(args) < 3 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);
@@ -279,7 +277,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 2 || !dict || 
+                if (PyTuple_Size(args) < 2 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);

From pypy.commits at gmail.com  Tue Aug  1 22:59:58 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 01 Aug 2017 19:59:58 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: fix some tests
Message-ID: <5981402e.4aa8df0a.13b04.fde0@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92026:398723725f36
Date: 2017-08-02 03:59 +0100
http://bitbucket.org/pypy/pypy/changeset/398723725f36/

Log:	fix some tests

diff --git a/pypy/module/cpyext/test/test_eval.py b/pypy/module/cpyext/test/test_eval.py
--- a/pypy/module/cpyext/test/test_eval.py
+++ b/pypy/module/cpyext/test/test_eval.py
@@ -279,6 +279,7 @@
         assert module.call_method("text") == 2
 
     def test_CompileString_and_Exec(self):
+        import sys
         module = self.import_extension('foo', [
             ("compile_string", "METH_NOARGS",
              """
@@ -313,6 +314,9 @@
         print mod.__dict__
         assert mod.f(42) == 47
 
+        # Clean-up
+        del sys.modules['cpyext_test_modname']
+
     def test_merge_compiler_flags(self):
         module = self.import_extension('foo', [
             ("get_flags", "METH_NOARGS",
@@ -357,4 +361,4 @@
         except RuntimeError as e:
             assert 'while calling recurse' in str(e)
         else:
-            assert False, "expected RuntimeError"  
+            assert False, "expected RuntimeError"
diff --git a/pypy/module/cpyext/test/test_floatobject.py b/pypy/module/cpyext/test/test_floatobject.py
--- a/pypy/module/cpyext/test/test_floatobject.py
+++ b/pypy/module/cpyext/test/test_floatobject.py
@@ -104,6 +104,7 @@
              PyFloatObject* pfo = (PyFloatObject*)pyobj;
              int res = PyFloat_Check(pyobj) && PyFloat_CheckExact(pyobj) &&
                 PyFloat_Check(pfo) && PyFloat_CheckExact(pfo);
+             Py_DecRef(pyobj);
              return PyLong_FromLong(res);"""),
             ])
         assert module.test() == 1
diff --git a/pypy/module/cpyext/test/test_funcobject.py b/pypy/module/cpyext/test/test_funcobject.py
--- a/pypy/module/cpyext/test/test_funcobject.py
+++ b/pypy/module/cpyext/test/test_funcobject.py
@@ -46,7 +46,7 @@
         w_function = space.appexec([], """():
             def func(x, y, z): return x
             return func
-        """)
+        """, cache=False)
         w_code = PyFunction_GetCode(space, w_function)
         assert w_code.co_name == "func"
 
@@ -63,7 +63,7 @@
             w_code = space.appexec([], """():
                 def func(%s): %s
                 return func.__code__
-            """ % (signature, body))
+            """ % (signature, body), cache=False)
             ref = make_ref(space, w_code)
             co_flags = rffi.cast(PyCodeObject, ref).c_co_flags
             decref(space, ref)
diff --git a/pypy/module/cpyext/test/test_longobject.py b/pypy/module/cpyext/test/test_longobject.py
--- a/pypy/module/cpyext/test/test_longobject.py
+++ b/pypy/module/cpyext/test/test_longobject.py
@@ -313,6 +313,7 @@
                     ret = obj->ob_type->tp_as_number->nb_power(obj, one, one);
                 else
                     ret = PyLong_FromLong(-1);
+                Py_DECREF(one);
                 Py_DECREF(obj);
                 return ret;
              """),
@@ -340,4 +341,3 @@
         assert module.has_pow() == 0
         assert module.has_hex() == '0x2aL'
         assert module.has_oct() == '052L'
-

From pypy.commits at gmail.com  Wed Aug  2 02:26:52 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 01 Aug 2017 23:26:52 -0700 (PDT)
Subject: [pypy-commit] pypy optinfo-into-bridges: close abandoned branch
 (everything useful was merged via the
Message-ID: <598170ac.3888df0a.6a4a5.b32c@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: optinfo-into-bridges
Changeset: r92027:596f167b7fd8
Date: 2017-08-02 08:26 +0200
http://bitbucket.org/pypy/pypy/changeset/596f167b7fd8/

Log:	close abandoned branch (everything useful was merged via the
	optinfo-into-bridges-3 branch)


From pypy.commits at gmail.com  Wed Aug  2 06:04:28 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 03:04:28 -0700 (PDT)
Subject: [pypy-commit] pypy arrays-force-less: a small improvement to
 getarrayitem caching
Message-ID: <5981a3ac.45091c0a.eba88.3d69@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: arrays-force-less
Changeset: r92029:7a124bb988d9
Date: 2017-08-02 12:02 +0200
http://bitbucket.org/pypy/pypy/changeset/7a124bb988d9/

Log:	a small improvement to getarrayitem caching

	when invalidating getarrayitem caches due to a setarrayitem, don't
	invalidate items that are at other indexes.

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -223,7 +223,10 @@
     def invalidate(self, descr):
         for opinfo in self.cached_infos:
             assert isinstance(opinfo, info.ArrayPtrInfo)
-            opinfo._items = None
+            # only invalidate those at self.index
+            if self.index < len(opinfo._items):
+                opinfo._items[self.index] = None
+            #opinfo._items = None #[self.index] = None
         self.cached_infos = []
         self.cached_structs = []
 
diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -1537,6 +1537,46 @@
         """
         self.optimize_loop(ops, expected)
 
+    def test_duplicate_getarrayitem_after_setarrayitem_and_guard(self):
+        ops = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p7 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p9 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        p10 = getarrayitem_gc_r(p1, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p7)
+        escape_n(p8)
+        escape_n(p9)
+        escape_n(p10)
+        jump(p0, p1, p2, p3, i1)
+        """
+        expected = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p4)
+        escape_n(p8)
+        escape_n(p6)
+        escape_n(p3)
+        jump(p0, p1, p2, p3, 1)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
         [p1, p2]

From pypy.commits at gmail.com  Wed Aug  2 06:04:25 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 03:04:25 -0700 (PDT)
Subject: [pypy-commit] pypy arrays-force-less: a branch to try to not force
 the whole array after a lazy setarrayitem is
Message-ID: <5981a3a9.6395df0a.29a7f.5f48@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: arrays-force-less
Changeset: r92028:da044fcdd451
Date: 2017-08-02 11:05 +0200
http://bitbucket.org/pypy/pypy/changeset/da044fcdd451/

Log:	a branch to try to not force the whole array after a lazy
	setarrayitem is forced, but only the affected indices


From pypy.commits at gmail.com  Wed Aug  2 07:54:59 2017
From: pypy.commits at gmail.com (exarkun)
Date: Wed, 02 Aug 2017 04:54:59 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Fix test_unflagged_non_text_codec_handling
Message-ID: <5981bd93.a3b0df0a.5f838.a76e@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92030:4499d3bd3e6f
Date: 2017-08-02 07:41 -0400
http://bitbucket.org/pypy/pypy/changeset/4499d3bd3e6f/

Log:	Fix test_unflagged_non_text_codec_handling

	test_unflagged_non_text_codec_handling
	(test.test_codecs.ExceptionChainingTest) expected a different
	exception message from misbehaving codecs. Update the message to
	reflect this.

diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -576,7 +576,9 @@
     w_retval = encode_text(space, w_object, encoding, errors)
     if not space.isinstance_w(w_retval, space.w_bytes):
         raise oefmt(space.w_TypeError,
-                    "encoder did not return a bytes object (type '%T')",
+                    "'%s' encoder returned '%T' instead of 'bytes'; "
+                    "use codecs.encode() to encode to arbitrary types",
+                    encoding,
                     w_retval)
     return w_retval
 
@@ -604,7 +606,9 @@
     w_retval = decode_text(space, w_obj, encoding, errors)
     if not space.isinstance_w(w_retval, space.w_unicode):
         raise oefmt(space.w_TypeError,
-                    "decoder did not return a bytes object (type '%T')",
+                    "'%s' decoder returned '%T' instead of 'str'; "
+                    "use codecs.decode() to decode to arbitrary types",
+                    encoding,
                     w_retval)
     return w_retval
 

From pypy.commits at gmail.com  Wed Aug  2 08:37:23 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 05:37:23 -0700 (PDT)
Subject: [pypy-commit] cffi default: Add an optional 'size' argument to
 ffi.gc(). So far, it has no effect
Message-ID: <5981c783.e6b2df0a.40f9c.ff2c@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r2998:120347b84c08
Date: 2017-08-02 14:37 +0200
http://bitbucket.org/cffi/cffi/changeset/120347b84c08/

Log:	Add an optional 'size' argument to ffi.gc(). So far, it has no
	effect on CPython.

diff --git a/c/_cffi_backend.c b/c/_cffi_backend.c
--- a/c/_cffi_backend.c
+++ b/c/_cffi_backend.c
@@ -6708,10 +6708,12 @@
     CDataObject *cd;
     CDataObject *origobj;
     PyObject *destructor;
-    static char *keywords[] = {"cdata", "destructor", NULL};
-
-    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O!O:gc", keywords,
-                                     &CData_Type, &origobj, &destructor))
+    Py_ssize_t ignored;   /* for pypy */
+    static char *keywords[] = {"cdata", "destructor", "size", NULL};
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O!O|n:gc", keywords,
+                                     &CData_Type, &origobj, &destructor,
+                                     &ignored))
         return NULL;
 
     if (destructor == Py_None) {
diff --git a/c/ffi_obj.c b/c/ffi_obj.c
--- a/c/ffi_obj.c
+++ b/c/ffi_obj.c
@@ -705,7 +705,12 @@
 PyDoc_STRVAR(ffi_gc_doc,
 "Return a new cdata object that points to the same data.\n"
 "Later, when this new cdata object is garbage-collected,\n"
-"'destructor(old_cdata_object)' will be called.");
+"'destructor(old_cdata_object)' will be called.\n"
+"\n"
+"The optional 'size' gives an estimate of the size, used to\n"
+"trigger the garbage collection more eagerly.  So far only used\n"
+"on PyPy.  It tells the GC that the returned object keeps alive\n"
+"roughly 'size' bytes of external memory.");
 
 #define ffi_gc  b_gcp     /* ffi_gc() => b_gcp()
                              from _cffi_backend.c */
diff --git a/cffi/api.py b/cffi/api.py
--- a/cffi/api.py
+++ b/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/cffi/backend_ctypes.py b/cffi/backend_ctypes.py
--- a/cffi/backend_ctypes.py
+++ b/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/testing/cffi0/test_verify.py b/testing/cffi0/test_verify.py
--- a/testing/cffi0/test_verify.py
+++ b/testing/cffi0/test_verify.py
@@ -2454,3 +2454,61 @@
     assert (pt.x, pt.y) == (-9*500*999, 9*500*999)
     pt = lib.call2(lib.cb2)
     assert (pt.x, pt.y) == (99*500*999, -99*500*999)
+
+def test_ffi_gc_size_arg():
+    # with PyPy's GC, these calls to ffi.gc() would rapidly consume
+    # 40 GB of RAM without the third argument
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/testing/cffi1/test_verify1.py b/testing/cffi1/test_verify1.py
--- a/testing/cffi1/test_verify1.py
+++ b/testing/cffi1/test_verify1.py
@@ -2290,3 +2290,61 @@
         expected = "unsigned int"
     assert ffi.typeof("UINT_PTR") is ffi.typeof(expected)
     assert ffi.typeof("PTSTR") is ffi.typeof("wchar_t *")
+
+def test_gc_pypy_size_arg():
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+        # with PyPy's GC, the above would rapidly consume 40 GB of RAM
+        # without the third argument to ffi.gc()
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x

From pypy.commits at gmail.com  Wed Aug  2 09:00:17 2017
From: pypy.commits at gmail.com (antocuni)
Date: Wed, 02 Aug 2017 06:00:17 -0700 (PDT)
Subject: [pypy-commit] pypy default: Add a workaround to make the distutils
 options 'runtime_library_dirs' working. Fixes #2593
Message-ID: <5981cce1.c98adf0a.b4cba.21a1@mx.google.com>

Author: Antonio Cuni <anto.cuni at gmail.com>
Branch: 
Changeset: r92031:620fefe235f3
Date: 2017-08-02 14:59 +0200
http://bitbucket.org/pypy/pypy/changeset/620fefe235f3/

Log:	Add a workaround to make the distutils options
	'runtime_library_dirs' working. Fixes #2593

diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:

From pypy.commits at gmail.com  Wed Aug  2 10:00:00 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 07:00:00 -0700 (PDT)
Subject: [pypy-commit] pypy default: Call add_memory_pressure(size) in
 response to ffi.gc(..., size)
Message-ID: <5981dae0.0f861c0a.e0962.b35d@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92032:53cb371f1828
Date: 2017-08-02 15:49 +0200
http://bitbucket.org/pypy/pypy/changeset/53cb371f1828/

Log:	Call add_memory_pressure(size) in response to ffi.gc(..., size)

diff --git a/pypy/module/_cffi_backend/cdataobj.py b/pypy/module/_cffi_backend/cdataobj.py
--- a/pypy/module/_cffi_backend/cdataobj.py
+++ b/pypy/module/_cffi_backend/cdataobj.py
@@ -433,17 +433,22 @@
     def _sizeof(self):
         return self.ctype.size
 
-    def with_gc(self, w_destructor):
+    def with_gc(self, w_destructor, size=0):
         space = self.space
         if space.is_none(w_destructor):
             if isinstance(self, W_CDataGCP):
                 self.detach_destructor()
-                return space.w_None
-            raise oefmt(space.w_TypeError,
-                        "Can remove destructor only on a object "
-                        "previously returned by ffi.gc()")
-        with self as ptr:
-            return W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+                w_res = space.w_None
+            else:
+                raise oefmt(space.w_TypeError,
+                            "Can remove destructor only on a object "
+                            "previously returned by ffi.gc()")
+        else:
+            with self as ptr:
+                w_res = W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+        if size != 0:
+            rgc.add_memory_pressure(size)
+        return w_res
 
     def unpack(self, length):
         from pypy.module._cffi_backend.ctypeptr import W_CTypePtrOrArray
diff --git a/pypy/module/_cffi_backend/ffi_obj.py b/pypy/module/_cffi_backend/ffi_obj.py
--- a/pypy/module/_cffi_backend/ffi_obj.py
+++ b/pypy/module/_cffi_backend/ffi_obj.py
@@ -351,14 +351,14 @@
         return handle.from_handle(self.space, w_arg)
 
 
-    @unwrap_spec(w_cdata=W_CData)
-    def descr_gc(self, w_cdata, w_destructor):
+    @unwrap_spec(w_cdata=W_CData, size=int)
+    def descr_gc(self, w_cdata, w_destructor, size=0):
         """\
 Return a new cdata object that points to the same data.
 Later, when this new cdata object is garbage-collected,
 'destructor(old_cdata_object)' will be called."""
         #
-        return w_cdata.with_gc(w_destructor)
+        return w_cdata.with_gc(w_destructor, size)
 
 
     @unwrap_spec(replace_with='text')
diff --git a/pypy/module/_cffi_backend/func.py b/pypy/module/_cffi_backend/func.py
--- a/pypy/module/_cffi_backend/func.py
+++ b/pypy/module/_cffi_backend/func.py
@@ -257,6 +257,6 @@
 
 # ____________________________________________________________
 
- at unwrap_spec(w_cdata=cdataobj.W_CData)
-def gcp(space, w_cdata, w_destructor):
-    return w_cdata.with_gc(w_destructor)
+ at unwrap_spec(w_cdata=cdataobj.W_CData, size=int)
+def gcp(space, w_cdata, w_destructor, size=0):
+    return w_cdata.with_gc(w_destructor, size)
diff --git a/pypy/module/_cffi_backend/test/test_ffi_obj.py b/pypy/module/_cffi_backend/test/test_ffi_obj.py
--- a/pypy/module/_cffi_backend/test/test_ffi_obj.py
+++ b/pypy/module/_cffi_backend/test/test_ffi_obj.py
@@ -377,7 +377,7 @@
         raises(TypeError, ffi.gc, p, None)
         seen = []
         q1 = ffi.gc(p, lambda p: seen.append(1))
-        q2 = ffi.gc(q1, lambda p: seen.append(2))
+        q2 = ffi.gc(q1, lambda p: seen.append(2), size=123)
         import gc; gc.collect()
         assert seen == []
         assert ffi.gc(q1, None) is None

From pypy.commits at gmail.com  Wed Aug  2 10:00:04 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 07:00:04 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge heads
Message-ID: <5981dae4.8dd81c0a.a72a9.c757@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92034:e4d42cef80b0
Date: 2017-08-02 15:59 +0200
http://bitbucket.org/pypy/pypy/changeset/e4d42cef80b0/

Log:	merge heads

diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:

From pypy.commits at gmail.com  Wed Aug  2 10:00:02 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 07:00:02 -0700 (PDT)
Subject: [pypy-commit] pypy default: import cffi/120347b84c08
Message-ID: <5981dae2.1babdf0a.4684e.dc81@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92033:1903d256e24b
Date: 2017-08-02 15:59 +0200
http://bitbucket.org/pypy/pypy/changeset/1903d256e24b/

Log:	import cffi/120347b84c08

diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
@@ -2455,3 +2455,61 @@
     assert (pt.x, pt.y) == (-9*500*999, 9*500*999)
     pt = lib.call2(lib.cb2)
     assert (pt.x, pt.y) == (99*500*999, -99*500*999)
+
+def test_ffi_gc_size_arg():
+    # with PyPy's GC, these calls to ffi.gc() would rapidly consume
+    # 40 GB of RAM without the third argument
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
@@ -2291,3 +2291,61 @@
         expected = "unsigned int"
     assert ffi.typeof("UINT_PTR") is ffi.typeof(expected)
     assert ffi.typeof("PTSTR") is ffi.typeof("wchar_t *")
+
+def test_gc_pypy_size_arg():
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+        # with PyPy's GC, the above would rapidly consume 40 GB of RAM
+        # without the third argument to ffi.gc()
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x

From pypy.commits at gmail.com  Wed Aug  2 10:14:06 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 07:14:06 -0700 (PDT)
Subject: [pypy-commit] cffi default: Document ffi.gc(..., size)
Message-ID: <5981de2e.129e1c0a.936d6.c5f4@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r2999:5b1214633c90
Date: 2017-08-02 16:13 +0200
http://bitbucket.org/cffi/cffi/changeset/5b1214633c90/

Log:	Document ffi.gc(..., size)

diff --git a/doc/source/ref.rst b/doc/source/ref.rst
--- a/doc/source/ref.rst
+++ b/doc/source/ref.rst
@@ -352,10 +352,13 @@
 .. __: #ffi-buffer
 
 
+.. _ffi-gc:
+
 ffi.gc()
 ++++++++
 
-**ffi.gc(cdata, destructor)**: return a new cdata object that points to the
+**ffi.gc(cdata, destructor, size=0)**:
+return a new cdata object that points to the
 same data.  Later, when this new cdata object is garbage-collected,
 ``destructor(old_cdata_object)`` will be called.  Example of usage:
 ``ptr = ffi.gc(lib.custom_malloc(42), lib.custom_free)``.
@@ -364,21 +367,38 @@
 which means the destructor is called as soon as *this* exact returned
 object is garbage-collected.
 
-**ffi.gc(ptr, None)**: removes the ownership on a object returned by a
+**ffi.gc(ptr, None, size=0)**:
+removes the ownership on a object returned by a
 regular call to ``ffi.gc``, and no destructor will be called when it
 is garbage-collected.  The object is modified in-place, and the
 function returns ``None``.  *New in version 1.7: ffi.gc(ptr, None)*
 
-Note that ``ffi.gc()`` should be avoided for large memory allocations or
-for limited resources.  This is particularly true on PyPy: its GC does
-not know how much memory or how many resources the returned ``ptr``
-holds.  It will only run its GC when enough memory it knows about has
-been allocated (and thus run the destructor possibly later than you
-would expect).  Moreover, the destructor is called in whatever thread
-PyPy is at that moment, which might be a problem for some C libraries.
-In these cases, consider writing a wrapper class with custom ``__enter__()``
-and ``__exit__()`` methods, allocating and freeing the C data at known
-points in time, and using it in a ``with`` statement.
+Note that ``ffi.gc()`` should be avoided for limited resources, or (with
+cffi below 1.11) for large memory allocations.  This is particularly
+true on PyPy: its GC does not know how much memory or how many resources
+the returned ``ptr`` holds.  It will only run its GC when enough memory
+it knows about has been allocated (and thus run the destructor possibly
+later than you would expect).  Moreover, the destructor is called in
+whatever thread PyPy is at that moment, which might be a problem for
+some C libraries.  In these cases, consider writing a wrapper class with
+custom ``__enter__()`` and ``__exit__()`` methods, allocating and
+freeing the C data at known points in time, and using it in a ``with``
+statement.
+
+*New in version 1.11:* the ``size`` argument.  If given, this should be
+an estimate of the size (in bytes) that ``ptr`` keeps alive.  This
+information is passed on to the garbage collector, fixing part of the
+problem described above.  The ``size`` argument is most important on
+PyPy; on CPython, it is ignored so far, but in the future it could be
+used to trigger more eagerly the cyclic reference GC, too (see CPython
+`issue 31105`__).
+
+The form ``ffi.gc(ptr, None, size=0)`` can be called with a negative
+``size``, to cancel the estimate.  It is not mandatory, though:
+nothing gets out of sync if the size estimates do not match.  It only
+makes the next GC start more or less early.
+
+.. __: http://bugs.python.org/issue31105
 
 
 .. _ffi-new-handle:
diff --git a/doc/source/whatsnew.rst b/doc/source/whatsnew.rst
--- a/doc/source/whatsnew.rst
+++ b/doc/source/whatsnew.rst
@@ -43,7 +43,16 @@
 * Functions returning booleans would in some case still return 0 or 1
   instead of False or True.  Fixed.
 
+* `ffi.gc()`__ now takes an optional third parameter, which gives an
+  estimate of the size (in bytes) of the object.  So far, this is only
+  used by PyPy, to make the next GC occur more quickly (`issue #320`__).
+  In the future, this might have an effect on CPython too (provided
+  the CPython `issue 31105`__ is addressed).
+
 .. __: https://bitbucket.org/cffi/cffi/issues/321/cffi-191-segmentation-fault-during-self
+.. __: ref.html#ffi-gc
+.. __: https://bitbucket.org/cffi/cffi/issues/320/improve-memory_pressure-management
+.. __: http://bugs.python.org/issue31105
 
 
 v1.10.1

From pypy.commits at gmail.com  Wed Aug  2 10:49:49 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 07:49:49 -0700 (PDT)
Subject: [pypy-commit] pypy arrays-force-less: document branch
Message-ID: <5981e68d.010d1c0a.f0778.5762@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: arrays-force-less
Changeset: r92035:5d7f91496e04
Date: 2017-08-02 16:48 +0200
http://bitbucket.org/pypy/pypy/changeset/5d7f91496e04/

Log:	document branch

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -55,3 +55,8 @@
 
 Fix the bounds in the GC when allocating a lot of objects with finalizers,
 fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.

From pypy.commits at gmail.com  Wed Aug  2 10:49:50 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 07:49:50 -0700 (PDT)
Subject: [pypy-commit] pypy arrays-force-less: close to-be-merged branch
Message-ID: <5981e68e.8f881c0a.356a8.d541@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: arrays-force-less
Changeset: r92036:82b3076c7887
Date: 2017-08-02 16:48 +0200
http://bitbucket.org/pypy/pypy/changeset/82b3076c7887/

Log:	close to-be-merged branch


From pypy.commits at gmail.com  Wed Aug  2 10:51:20 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 07:51:20 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge arrays-force-less
Message-ID: <5981e6e8.ccb21c0a.f4a20.cd07@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92037:a22d0969f40a
Date: 2017-08-02 16:50 +0200
http://bitbucket.org/pypy/pypy/changeset/a22d0969f40a/

Log:	merge arrays-force-less

	don't throw away all of the knowledge about an array when forcing a
	lazy setarrayitem. instead, only throw away the cache of the
	specific index.

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -55,3 +55,8 @@
 
 Fix the bounds in the GC when allocating a lot of objects with finalizers,
 fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -223,7 +223,10 @@
     def invalidate(self, descr):
         for opinfo in self.cached_infos:
             assert isinstance(opinfo, info.ArrayPtrInfo)
-            opinfo._items = None
+            # only invalidate those at self.index
+            if self.index < len(opinfo._items):
+                opinfo._items[self.index] = None
+            #opinfo._items = None #[self.index] = None
         self.cached_infos = []
         self.cached_structs = []
 
diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -1537,6 +1537,46 @@
         """
         self.optimize_loop(ops, expected)
 
+    def test_duplicate_getarrayitem_after_setarrayitem_and_guard(self):
+        ops = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p7 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p9 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        p10 = getarrayitem_gc_r(p1, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p7)
+        escape_n(p8)
+        escape_n(p9)
+        escape_n(p10)
+        jump(p0, p1, p2, p3, i1)
+        """
+        expected = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p4)
+        escape_n(p8)
+        escape_n(p6)
+        escape_n(p3)
+        jump(p0, p1, p2, p3, 1)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
         [p1, p2]

From pypy.commits at gmail.com  Wed Aug  2 11:32:27 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 08:32:27 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: merge
 arrays-force-less
Message-ID: <5981f08b.830a1c0a.82504.a945@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92039:e8af759e759d
Date: 2017-08-02 12:09 +0200
http://bitbucket.org/pypy/pypy/changeset/e8af759e759d/

Log:	merge arrays-force-less

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -224,7 +224,10 @@
     def invalidate(self, descr):
         for opinfo in self.cached_infos:
             assert isinstance(opinfo, info.ArrayPtrInfo)
-            opinfo._items[self.index] = None
+            # only invalidate those at self.index
+            if self.index < len(opinfo._items):
+                opinfo._items[self.index] = None
+            #opinfo._items = None #[self.index] = None
         self.cached_infos = []
         self.cached_structs = []
 
diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -1537,6 +1537,46 @@
         """
         self.optimize_loop(ops, expected)
 
+    def test_duplicate_getarrayitem_after_setarrayitem_and_guard(self):
+        ops = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p7 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p9 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        p10 = getarrayitem_gc_r(p1, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p7)
+        escape_n(p8)
+        escape_n(p9)
+        escape_n(p10)
+        jump(p0, p1, p2, p3, i1)
+        """
+        expected = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p4)
+        escape_n(p8)
+        escape_n(p6)
+        escape_n(p3)
+        jump(p0, p1, p2, p3, 1)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
         [p1, p2]

From pypy.commits at gmail.com  Wed Aug  2 11:32:29 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 08:32:29 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: finish support for
 serializing knowledge about arrays
Message-ID: <5981f08d.9c8bdf0a.4994a.9376@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92040:8885e06c04d6
Date: 2017-08-02 16:26 +0200
http://bitbucket.org/pypy/pypy/changeset/8885e06c04d6/

Log:	finish support for serializing knowledge about arrays

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -116,7 +116,6 @@
             return None
 
     def force_lazy_set(self, optheap, descr, can_cache=True):
-        import pdb; pdb.set_trace()
         op = self._lazy_set
         if op is not None:
             # This is the way _lazy_set is usually reset to None.
@@ -578,7 +577,6 @@
         cf.do_setfield(self, op)
 
     def optimize_GETARRAYITEM_GC_I(self, op):
-        import pdb; pdb.set_trace()
         arrayinfo = self.ensure_ptr_info_arg0(op)
         indexb = self.getintbound(op.getarg(1))
         cf = None
@@ -641,7 +639,6 @@
     optimize_GETARRAYITEM_GC_PURE_F = optimize_GETARRAYITEM_GC_PURE_I
 
     def optimize_SETARRAYITEM_GC(self, op):
-        import pdb; pdb.set_trace()
         #opnum = OpHelpers.getarrayitem_pure_for_descr(op.getdescr())
         #if self.has_pure_result(opnum, [op.getarg(0), op.getarg(1)],
         #                        op.getdescr()):
@@ -716,7 +713,6 @@
                 if isinstance(box2, Const) or box2 in available_boxes:
                     result_getfield.append((box1, descr, box2))
         result_array = []
-        import pdb; pdb.set_trace()
         for descr, indexdict in self.cached_arrayitems.iteritems():
             for index, cf in indexdict.iteritems():
                 if cf._lazy_set:
@@ -731,7 +727,7 @@
         return result_getfield, result_array
 
     def deserialize_optheap(self, triples_struct, triples_array):
-        for box1, descr, box2 in triples_struct[0]:
+        for box1, descr, box2 in triples_struct:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
             structinfo = box1.get_forwarded()
@@ -742,15 +738,14 @@
 
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
-        import pdb; pdb.set_trace()
-        for box1, index, descr, box2 in triples_array[0]:
-            structinfo = box1.get_forwarded()
-            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
-                structinfo = info.ArrayPtrInfo(descr)
-                box1.set_forwarded(structinfo)
+        for box1, index, descr, box2 in triples_array:
+            arrayinfo = box1.get_forwarded()
+            if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
+                arrayinfo = info.ArrayPtrInfo(descr)
+                box1.set_forwarded(arrayinfo)
 
-            cf = self.arrayitem_cache(index, descr)
-            structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
+            cf = self.arrayitem_cache(descr, index)
+            arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
 
 
 dispatch_opt = make_dispatcher_method(OptHeap, 'optimize_',
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -211,6 +211,5 @@
         assert res == f(6, 32, 16)
         self.check_trace_count(3)
         self.check_resops(guard_value=1)
-        self.check_resops(getarrayitem_gc_r=4) # 3x a.x, 1x a.n
-        self.check_resops(getfield_gc_r=1) # in main loop
+        self.check_resops(getarrayitem_gc_i=4)
 

From pypy.commits at gmail.com  Wed Aug  2 11:32:31 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 08:32:31 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: fix affected tests
Message-ID: <5981f08f.c98adf0a.b4cba.468e@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92041:a9a9281dad98
Date: 2017-08-02 17:31 +0200
http://bitbucket.org/pypy/pypy/changeset/a9a9281dad98/

Log:	fix affected tests

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -735,15 +735,14 @@
                 structinfo = info.InstancePtrInfo(parent_descr)
                 structinfo.init_fields(parent_descr, descr.get_index())
                 box1.set_forwarded(structinfo)
-
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
+
         for box1, index, descr, box2 in triples_array:
             arrayinfo = box1.get_forwarded()
             if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
                 arrayinfo = info.ArrayPtrInfo(descr)
                 box1.set_forwarded(arrayinfo)
-
             cf = self.arrayitem_cache(descr, index)
             arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
 
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -61,7 +61,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0]
+    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0, 0]
 
     rbox1 = InputArgRef()
     rbox2 = InputArgRef()
@@ -97,7 +97,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert len(numb_state.create_numbering().code) == 2 + math.ceil(len(refboxes) / 6.0)
+    assert len(numb_state.create_numbering().code) == 3 + math.ceil(len(refboxes) / 6.0)
 
     dct = {box: cls
               for box, known_class in boxes_known_classes

From pypy.commits at gmail.com  Wed Aug  2 11:32:24 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 08:32:24 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: in progress
Message-ID: <5981f088.0e951c0a.372ea.a7f4@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92038:046804fa21b3
Date: 2017-08-02 11:05 +0200
http://bitbucket.org/pypy/pypy/changeset/046804fa21b3/

Log:	in progress

diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -18,6 +18,10 @@
 # (<box1> <descr> <box2>) length times, if getfield(box1, descr) == box2
 #                         both boxes should be in the liveboxes
 #
+# <length>
+# (<box1> <index> <descr> <box2>) length times, if getarrayitem_gc(box1, index, descr) == box2
+#                                 both boxes should be in the liveboxes
+#
 # ----
 
 
@@ -82,18 +86,26 @@
     # structs
     # XXX could be extended to arrays
     if optimizer.optheap:
-        triples = optimizer.optheap.serialize_optheap(available_boxes)
+        triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
         # metainterp_sd.all_descrs
-        triples = [triple for triple in triples if triple[1].descr_index != -1]
-        numb_state.append_int(len(triples))
-        for box1, descr, box2 in triples:
-            index = descr.descr_index
+        triples_struct = [triple for triple in triples_struct if triple[1].descr_index != -1]
+        numb_state.append_int(len(triples_struct))
+        for box1, descr, box2 in triples_struct:
+            descr_index = descr.descr_index
+            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_int(descr_index)
+            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+        numb_state.append_int(len(triples_array))
+        for box1, index, descr, box2 in triples_array:
+            descr_index = descr.descr_index
             numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
             numb_state.append_int(index)
+            numb_state.append_int(descr_index)
             numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
     else:
         numb_state.append_int(0)
+        numb_state.append_int(0)
 
 def deserialize_optimizer_knowledge(optimizer, resumestorage, frontend_boxes, liveboxes):
     reader = resumecode.Reader(resumestorage.rd_numb)
@@ -123,13 +135,24 @@
     if not optimizer.optheap:
         return
     length = reader.next_item()
-    result = []
+    result_struct = []
+    for i in range(length):
+        tagged = reader.next_item()
+        box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
+        tagged = reader.next_item()
+        box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        result_struct.append((box1, descr, box2))
+    length = reader.next_item()
+    result_array = []
     for i in range(length):
         tagged = reader.next_item()
         box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
         index = reader.next_item()
-        descr = metainterp_sd.all_descrs[index]
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
         tagged = reader.next_item()
         box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
-        result.append((box1, descr, box2))
-    optimizer.optheap.deserialize_optheap(result)
+        result_array.append((box1, index, descr, box2))
+    optimizer.optheap.deserialize_optheap(result_struct, result_array)
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -116,6 +116,7 @@
             return None
 
     def force_lazy_set(self, optheap, descr, can_cache=True):
+        import pdb; pdb.set_trace()
         op = self._lazy_set
         if op is not None:
             # This is the way _lazy_set is usually reset to None.
@@ -223,7 +224,7 @@
     def invalidate(self, descr):
         for opinfo in self.cached_infos:
             assert isinstance(opinfo, info.ArrayPtrInfo)
-            opinfo._items = None
+            opinfo._items[self.index] = None
         self.cached_infos = []
         self.cached_structs = []
 
@@ -574,6 +575,7 @@
         cf.do_setfield(self, op)
 
     def optimize_GETARRAYITEM_GC_I(self, op):
+        import pdb; pdb.set_trace()
         arrayinfo = self.ensure_ptr_info_arg0(op)
         indexb = self.getintbound(op.getarg(1))
         cf = None
@@ -636,6 +638,7 @@
     optimize_GETARRAYITEM_GC_PURE_F = optimize_GETARRAYITEM_GC_PURE_I
 
     def optimize_SETARRAYITEM_GC(self, op):
+        import pdb; pdb.set_trace()
         #opnum = OpHelpers.getarrayitem_pure_for_descr(op.getdescr())
         #if self.has_pure_result(opnum, [op.getarg(0), op.getarg(1)],
         #                        op.getdescr()):
@@ -695,7 +698,7 @@
         return self.emit(op)
 
     def serialize_optheap(self, available_boxes):
-        result = []
+        result_getfield = []
         for descr, cf in self.cached_fields.iteritems():
             if cf._lazy_set:
                 continue # XXX safe default for now
@@ -708,11 +711,24 @@
                 structinfo = cf.cached_infos[i]
                 box2 = structinfo.getfield(descr).get_box_replacement()
                 if isinstance(box2, Const) or box2 in available_boxes:
-                    result.append((box1, descr, box2))
-        return result
+                    result_getfield.append((box1, descr, box2))
+        result_array = []
+        import pdb; pdb.set_trace()
+        for descr, indexdict in self.cached_arrayitems.iteritems():
+            for index, cf in indexdict.iteritems():
+                if cf._lazy_set:
+                    continue # XXX safe default for now
+                for i, box1 in enumerate(cf.cached_structs):
+                    if box1 not in available_boxes:
+                        continue
+                    arrayinfo = cf.cached_infos[i]
+                    box2 = arrayinfo.getitem(descr, index).get_box_replacement()
+                    if isinstance(box2, Const) or box2 in available_boxes:
+                        result_array.append((box1, index, descr, box2))
+        return result_getfield, result_array
 
-    def deserialize_optheap(self, triples):
-        for box1, descr, box2 in triples:
+    def deserialize_optheap(self, triples_struct, triples_array):
+        for box1, descr, box2 in triples_struct[0]:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
             structinfo = box1.get_forwarded()
@@ -723,6 +739,15 @@
 
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
+        import pdb; pdb.set_trace()
+        for box1, index, descr, box2 in triples_array[0]:
+            structinfo = box1.get_forwarded()
+            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
+                structinfo = info.ArrayPtrInfo(descr)
+                box1.set_forwarded(structinfo)
+
+            cf = self.arrayitem_cache(index, descr)
+            structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
 
 
 dispatch_opt = make_dispatcher_method(OptHeap, 'optimize_',
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -186,3 +186,31 @@
         self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
         self.check_resops(getfield_gc_r=1) # in main loop
 
+    def test_bridge_array_read(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
+        def f(x, y, n):
+            if x:
+                a = [1, n, 0]
+            else:
+                a = [2, n, 0]
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                n1 = a[1]
+                m = jit.promote(a[0])
+                res += m
+                a[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a[0])
+                res += m
+                res += n1 + a[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_r=4) # 3x a.x, 1x a.n
+        self.check_resops(getfield_gc_r=1) # in main loop
+

From pypy.commits at gmail.com  Wed Aug  2 11:34:38 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 08:34:38 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: merge default
Message-ID: <5981f10e.08891c0a.bacc6.a45d@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92042:d16b10f8ca91
Date: 2017-08-02 17:34 +0200
http://bitbucket.org/pypy/pypy/changeset/d16b10f8ca91/

Log:	merge default

diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -55,3 +55,8 @@
 
 Fix the bounds in the GC when allocating a lot of objects with finalizers,
 fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
diff --git a/pypy/module/_cffi_backend/cdataobj.py b/pypy/module/_cffi_backend/cdataobj.py
--- a/pypy/module/_cffi_backend/cdataobj.py
+++ b/pypy/module/_cffi_backend/cdataobj.py
@@ -433,17 +433,22 @@
     def _sizeof(self):
         return self.ctype.size
 
-    def with_gc(self, w_destructor):
+    def with_gc(self, w_destructor, size=0):
         space = self.space
         if space.is_none(w_destructor):
             if isinstance(self, W_CDataGCP):
                 self.detach_destructor()
-                return space.w_None
-            raise oefmt(space.w_TypeError,
-                        "Can remove destructor only on a object "
-                        "previously returned by ffi.gc()")
-        with self as ptr:
-            return W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+                w_res = space.w_None
+            else:
+                raise oefmt(space.w_TypeError,
+                            "Can remove destructor only on a object "
+                            "previously returned by ffi.gc()")
+        else:
+            with self as ptr:
+                w_res = W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+        if size != 0:
+            rgc.add_memory_pressure(size)
+        return w_res
 
     def unpack(self, length):
         from pypy.module._cffi_backend.ctypeptr import W_CTypePtrOrArray
diff --git a/pypy/module/_cffi_backend/ffi_obj.py b/pypy/module/_cffi_backend/ffi_obj.py
--- a/pypy/module/_cffi_backend/ffi_obj.py
+++ b/pypy/module/_cffi_backend/ffi_obj.py
@@ -351,14 +351,14 @@
         return handle.from_handle(self.space, w_arg)
 
 
-    @unwrap_spec(w_cdata=W_CData)
-    def descr_gc(self, w_cdata, w_destructor):
+    @unwrap_spec(w_cdata=W_CData, size=int)
+    def descr_gc(self, w_cdata, w_destructor, size=0):
         """\
 Return a new cdata object that points to the same data.
 Later, when this new cdata object is garbage-collected,
 'destructor(old_cdata_object)' will be called."""
         #
-        return w_cdata.with_gc(w_destructor)
+        return w_cdata.with_gc(w_destructor, size)
 
 
     @unwrap_spec(replace_with='text')
diff --git a/pypy/module/_cffi_backend/func.py b/pypy/module/_cffi_backend/func.py
--- a/pypy/module/_cffi_backend/func.py
+++ b/pypy/module/_cffi_backend/func.py
@@ -257,6 +257,6 @@
 
 # ____________________________________________________________
 
- at unwrap_spec(w_cdata=cdataobj.W_CData)
-def gcp(space, w_cdata, w_destructor):
-    return w_cdata.with_gc(w_destructor)
+ at unwrap_spec(w_cdata=cdataobj.W_CData, size=int)
+def gcp(space, w_cdata, w_destructor, size=0):
+    return w_cdata.with_gc(w_destructor, size)
diff --git a/pypy/module/_cffi_backend/test/test_ffi_obj.py b/pypy/module/_cffi_backend/test/test_ffi_obj.py
--- a/pypy/module/_cffi_backend/test/test_ffi_obj.py
+++ b/pypy/module/_cffi_backend/test/test_ffi_obj.py
@@ -377,7 +377,7 @@
         raises(TypeError, ffi.gc, p, None)
         seen = []
         q1 = ffi.gc(p, lambda p: seen.append(1))
-        q2 = ffi.gc(q1, lambda p: seen.append(2))
+        q2 = ffi.gc(q1, lambda p: seen.append(2), size=123)
         import gc; gc.collect()
         assert seen == []
         assert ffi.gc(q1, None) is None
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
@@ -2455,3 +2455,61 @@
     assert (pt.x, pt.y) == (-9*500*999, 9*500*999)
     pt = lib.call2(lib.cb2)
     assert (pt.x, pt.y) == (99*500*999, -99*500*999)
+
+def test_ffi_gc_size_arg():
+    # with PyPy's GC, these calls to ffi.gc() would rapidly consume
+    # 40 GB of RAM without the third argument
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
@@ -2291,3 +2291,61 @@
         expected = "unsigned int"
     assert ffi.typeof("UINT_PTR") is ffi.typeof(expected)
     assert ffi.typeof("PTSTR") is ffi.typeof("wchar_t *")
+
+def test_gc_pypy_size_arg():
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+        # with PyPy's GC, the above would rapidly consume 40 GB of RAM
+        # without the third argument to ffi.gc()
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x

From pypy.commits at gmail.com  Wed Aug  2 12:32:05 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 02 Aug 2017 09:32:05 -0700 (PDT)
Subject: [pypy-commit] pypy default: Fix the Makefile to produce the pypy-c
 inside pypy/goal/, like we
Message-ID: <5981fe85.47bf1c0a.d060a.3931@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92043:dd95b2eeb5ac
Date: 2017-08-02 18:31 +0200
http://bitbucket.org/pypy/pypy/changeset/dd95b2eeb5ac/

Log:	Fix the Makefile to produce the pypy-c inside pypy/goal/, like we
	normally expect and document

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true

From pypy.commits at gmail.com  Wed Aug  2 13:48:12 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 02 Aug 2017 10:48:12 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: maybe identify a possible
 leak in handling GetSetProperty?
Message-ID: <5982105c.56b71c0a.ea4c4.bbf9@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92044:c1786ec814d9
Date: 2017-08-02 20:47 +0300
http://bitbucket.org/pypy/pypy/changeset/c1786ec814d9/

Log:	maybe identify a possible leak in handling GetSetProperty?

diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -195,6 +195,8 @@
         py_getsetdef = make_GetSet(space, w_obj)
         assert space.isinstance_w(w_userdata, space.w_type)
         w_obj = W_GetSetPropertyEx(py_getsetdef, w_userdata)
+        # now w_obj.getset is py_getsetdef, which was freshly allocated
+        # XXX how is this ever released?
     # XXX assign to d_dname, d_type?
     assert isinstance(w_obj, W_GetSetPropertyEx)
     py_getsetdescr.c_d_getset = w_obj.getset

From pypy.commits at gmail.com  Wed Aug  2 13:50:56 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 02 Aug 2017 10:50:56 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: support for caching
 infos on constants, while I'm at it
Message-ID: <59821100.9fb6df0a.5de29.61c7@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92045:f105d8c2eae7
Date: 2017-08-02 18:04 +0200
http://bitbucket.org/pypy/pypy/changeset/f105d8c2eae7/

Log:	support for caching infos on constants, while I'm at it

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -706,11 +706,11 @@
             if not parent_descr.is_object():
                 continue # XXX could be extended to non-instance objects
             for i, box1 in enumerate(cf.cached_structs):
-                if box1 not in available_boxes:
+                if not box1.is_constant() and box1 not in available_boxes:
                     continue
                 structinfo = cf.cached_infos[i]
                 box2 = structinfo.getfield(descr).get_box_replacement()
-                if isinstance(box2, Const) or box2 in available_boxes:
+                if box2.is_constant() or box2 in available_boxes:
                     result_getfield.append((box1, descr, box2))
         result_array = []
         for descr, indexdict in self.cached_arrayitems.iteritems():
@@ -718,11 +718,11 @@
                 if cf._lazy_set:
                     continue # XXX safe default for now
                 for i, box1 in enumerate(cf.cached_structs):
-                    if box1 not in available_boxes:
+                    if not box1.is_constant() and box1 not in available_boxes:
                         continue
                     arrayinfo = cf.cached_infos[i]
                     box2 = arrayinfo.getitem(descr, index).get_box_replacement()
-                    if isinstance(box2, Const) or box2 in available_boxes:
+                    if box2.is_constant() or box2 in available_boxes:
                         result_array.append((box1, index, descr, box2))
         return result_getfield, result_array
 
@@ -730,19 +730,25 @@
         for box1, descr, box2 in triples_struct:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
-            structinfo = box1.get_forwarded()
-            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
-                structinfo = info.InstancePtrInfo(parent_descr)
-                structinfo.init_fields(parent_descr, descr.get_index())
-                box1.set_forwarded(structinfo)
+            if box1.is_constant():
+                structinfo = info.ConstPtrInfo(box1)
+            else:
+                structinfo = box1.get_forwarded()
+                if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
+                    structinfo = info.InstancePtrInfo(parent_descr)
+                    structinfo.init_fields(parent_descr, descr.get_index())
+                    box1.set_forwarded(structinfo)
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
 
         for box1, index, descr, box2 in triples_array:
-            arrayinfo = box1.get_forwarded()
-            if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
-                arrayinfo = info.ArrayPtrInfo(descr)
-                box1.set_forwarded(arrayinfo)
+            if box1.is_constant():
+                arrayinfo = info.ConstPtrInfo(box1)
+            else:
+                arrayinfo = box1.get_forwarded()
+                if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
+                    arrayinfo = info.ArrayPtrInfo(descr)
+                    box1.set_forwarded(arrayinfo)
             cf = self.arrayitem_cache(descr, index)
             arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
 
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -143,11 +143,7 @@
     def test_bridge_field_read(self):
         myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
         class A(object):
-            def f(self):
-                return 1
-        class B(A):
-            def f(self):
-                return 2
+            pass
         class M(object):
             _immutable_fields_ = ['x']
             def __init__(self, x):
@@ -156,18 +152,59 @@
         m1 = M(1)
         m2 = M(2)
         def f(x, y, n):
+            a = A()
+            a.n = n
             if x:
-                a = A()
+                a.m = m1
+            else:
+                a.m = m2
+            a.x = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                n1 = a.n
+                m = jit.promote(a.m)
+                res += m.x
+                a.x += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.m)
+                res += m.x
+                res += n1 + a.n
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
+        self.check_resops(getfield_gc_r=1) # in main loop
+
+    def test_bridge_field_read_constants(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        class M(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x):
+                self.x = x
+
+        m1 = M(1)
+        m2 = M(2)
+        a = A()
+        a.m = m1
+        a.n = 0
+        def f(x, y, n):
+            if x:
                 a.m = m1
                 a.n = n
             else:
-                a = B()
                 a.m = m2
                 a.n = n
             a.x = 0
             res = 0
             while y > 0:
-                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
                 n1 = a.n
                 m = jit.promote(a.m)
                 res += m.x
@@ -213,3 +250,35 @@
         self.check_resops(guard_value=1)
         self.check_resops(getarrayitem_gc_i=4)
 
+    def test_bridge_array_read_constant(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        a = A()
+        a.l = [1, -65, 0]
+        def f(x, y, n):
+            if x:
+                a.l[0] = 1
+            else:
+                a.l[0] = 2
+            a.l[1] = n
+            a.l[2] = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.l[1]
+                m = jit.promote(a.l[0])
+                res += m
+                a.l[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.l[0])
+                res += m
+                res += n1 + a.l[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=5)

From pypy.commits at gmail.com  Thu Aug  3 04:45:38 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 03 Aug 2017 01:45:38 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-debug-type_dealloc: pytest.py
 ../test_typeobject.py -k test_tp_getattro -s never deallocates "instance"
Message-ID: <5982e2b2.8a8edf0a.45db8.7e80@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-debug-type_dealloc
Changeset: r92046:1ce0da5af8e6
Date: 2017-08-03 11:44 +0300
http://bitbucket.org/pypy/pypy/changeset/1ce0da5af8e6/

Log:	pytest.py ../test_typeobject.py -k test_tp_getattro -s never
	deallocates "instance"

diff --git a/pypy/module/cpyext/state.py b/pypy/module/cpyext/state.py
--- a/pypy/module/cpyext/state.py
+++ b/pypy/module/cpyext/state.py
@@ -59,14 +59,17 @@
         if not self.space.config.translating:
             def dealloc_trigger():
                 from pypy.module.cpyext.pyobject import PyObject, decref
-                print 'dealloc_trigger...'
+                print >>sys.stderr, 'dealloc_trigger...'
                 while True:
                     ob = rawrefcount.next_dead(PyObject)
                     if not ob:
                         break
-                    print 'deallocating PyObject', ob
+                    if ob.c_ob_type.c_tp_name:
+                        print >>sys.stderr, 'deallocating PyObject', rffi.charp2str(ob.c_ob_type.c_tp_name), ob
+                    else:
+                        print >>sys.stderr, 'deallocating PyObject (without tp_name)', ob
                     decref(space, ob)
-                print 'dealloc_trigger DONE'
+                print >>sys.stderr, 'dealloc_trigger DONE'
                 return "RETRY"
             rawrefcount.init(dealloc_trigger)
         else:
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -540,22 +540,22 @@
     # w_obj is an instance of w_A or one of its subclasses. So climb up the
     # inheritance chain until base.c_tp_dealloc is exactly this_func, and then
     # continue on up until they differ.
-    #print 'subtype_dealloc, start from', rffi.charp2str(base.c_tp_name)
+    print 'subtype_dealloc, start from', rffi.charp2str(base.c_tp_name)
     while base.c_tp_dealloc != this_func_ptr:
         base = base.c_tp_base
         assert base
-        #print '                 ne move to', rffi.charp2str(base.c_tp_name)
+        print '                 ne move to', rffi.charp2str(base.c_tp_name)
         w_obj = from_ref(space, rffi.cast(PyObject, base))
     while base.c_tp_dealloc == this_func_ptr:
         base = base.c_tp_base
         assert base
-        #print '                 eq move to', rffi.charp2str(base.c_tp_name)
+        print '                 eq move to', rffi.charp2str(base.c_tp_name)
         w_obj = from_ref(space, rffi.cast(PyObject, base))
-    #print '                   end with', rffi.charp2str(base.c_tp_name)
+    print '                   end with', rffi.charp2str(base.c_tp_name)
     dealloc = base.c_tp_dealloc
     # XXX call tp_del if necessary
     generic_cpy_call(space, dealloc, obj)
-    # XXX cpy decrefs the pto here but we do it in the base-dealloc
+    # XXX cpy decrefs the pto on heaptypes here but we do it in PyObject_dealloc 
     # hopefully this does not clash with the memory model assumed in
     # extension modules
 
@@ -675,6 +675,7 @@
     from pypy.module.cpyext.object import _dealloc
     obj_pto = rffi.cast(PyTypeObjectPtr, obj)
     base_pyo = rffi.cast(PyObject, obj_pto.c_tp_base)
+    print 'decref tp_bases', from_ref(space, obj_pto.c_tp_bases)
     Py_DecRef(space, obj_pto.c_tp_bases)
     #Py_DecRef(space, obj_pto.c_tp_mro)
     Py_DecRef(space, obj_pto.c_tp_cache) # let's do it like cpython
@@ -927,6 +928,9 @@
         is_heaptype = bool(pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE)
         pto.c_tp_bases = make_ref(space, space.newtuple(bases_w),
                                   immortal=not is_heaptype)
+        name = rffi.charp2str(pto.c_tp_name)
+        if name == 'instance':
+            print 'incref tp_bases', from_ref(space, pto.c_tp_bases), 'on instance' 
 
 def finish_type_2(space, pto, w_obj):
     """

From pypy.commits at gmail.com  Thu Aug  3 05:19:23 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 03 Aug 2017 02:19:23 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-debug-type_dealloc: minimal failing test;
 __init__ causes leak-checker to fail
Message-ID: <5982ea9b.44691c0a.a0394.7913@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-debug-type_dealloc
Changeset: r92047:a12c2162e44e
Date: 2017-08-03 12:18 +0300
http://bitbucket.org/pypy/pypy/changeset/a12c2162e44e/

Log:	minimal failing test; __init__ causes leak-checker to fail

diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py
--- a/pypy/module/cpyext/test/test_typeobject.py
+++ b/pypy/module/cpyext/test/test_typeobject.py
@@ -569,6 +569,20 @@
             ])
         assert module.test_type(type(None))
 
+    def test_leaking(self):
+        module = self.import_extension('foo', [
+            ("test_leak", "METH_VARARGS",
+             '''
+                 Py_RETURN_TRUE;
+             '''
+             )
+            ])
+        class C:
+            def __init__(self):
+                pass
+        # leak checker should not report errors
+        assert module.test_leak(C())
+    
     def test_tp_getattro(self):
         module = self.import_extension('foo', [
             ("test_tp_getattro", "METH_VARARGS",

From pypy.commits at gmail.com  Thu Aug  3 05:35:18 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 03 Aug 2017 02:35:18 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-debug-type_dealloc: expand test, D(),
 E() fail leak check, C() passes. WHY?
Message-ID: <5982ee56.415f1c0a.6582d.6f5a@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-debug-type_dealloc
Changeset: r92048:18ec6be95b4a
Date: 2017-08-03 12:34 +0300
http://bitbucket.org/pypy/pypy/changeset/18ec6be95b4a/

Log:	expand test, D(), E() fail leak check, C() passes. WHY?

diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py
--- a/pypy/module/cpyext/test/test_typeobject.py
+++ b/pypy/module/cpyext/test/test_typeobject.py
@@ -577,11 +577,18 @@
              '''
              )
             ])
-        class C:
+        class C(object):
+            def dummymethod(self):
+                pass
+        class D(object):
             def __init__(self):
                 pass
-        # leak checker should not report errors
+        class E:
+            pass
+        # TODO C passes leak checker, D,E fails
         assert module.test_leak(C())
+        assert module.test_leak(D())
+        assert module.test_leak(E())
     
     def test_tp_getattro(self):
         module = self.import_extension('foo', [

From pypy.commits at gmail.com  Thu Aug  3 09:35:43 2017
From: pypy.commits at gmail.com (fijal)
Date: Thu, 03 Aug 2017 06:35:43 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: write a blog post draft
Message-ID: <598326af.d288df0a.452ff.e4a9@mx.google.com>

Author: fijal
Branch: extradoc
Changeset: r5823:f01eecaabd0f
Date: 2017-08-03 15:35 +0200
http://bitbucket.org/pypy/extradoc/changeset/f01eecaabd0f/

Log:	write a blog post draft

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
new file mode 100644
--- /dev/null
+++ b/blog/draft/remove-gil.rst
@@ -0,0 +1,28 @@
+GIL removal proposal
+--------------------
+
+Hello everyone.
+
+The topic of the infamous Global Interpreter Lock has been around for a while
+in the Python community. There has been various attempts at removing it
+(some successful ones, e.g. in Jython or IronPython with the help of the platform)
+and some yet to bear fruit, like `gilectomy`_. Since February sprint in Leysin,
+we've been on-and-off tackling the topic of GIL removal in the PyPy project.
+
+As of Europython announcement, we're able to run (very simple) programs with GIL-less
+PyPy that parallelizes nicely. The remaining 90% (and another 90%) of work
+is with putting locks in strategic places so PyPy does not segfault
+when you try to do a concurrent access to a data structure.
+
+Since such work would complicate the code base and our day to day work,
+we would like to judge the interest on the community and the commercial
+PyPy users.
+
+We would like to do it in a following way. We are looking for a contract
+with companies (individual donations did not work very well for us in the
+past). We put the total cost of doing the work at $50k, out of which we
+already have backing for about 1/3. If we can get a $100k contract, we would
+make it our priority to deliver before the end of the year.
+
+Best regards,
+Maciej Fijalkowski

From pypy.commits at gmail.com  Thu Aug  3 12:35:03 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 03 Aug 2017 09:35:03 -0700 (PDT)
Subject: [pypy-commit] pypy default: remove the withcelldict option
 (translating without withcelldict is really not
Message-ID: <598350b7.0591df0a.abbbc.cc78@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92049:6d1bf70e6214
Date: 2017-08-03 16:27 +0200
http://bitbucket.org/pypy/pypy/changeset/6d1bf70e6214/

Log:	remove the withcelldict option (translating without withcelldict is
	really not advisable, and many other related options have been
	removed already).

diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -224,11 +224,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -288,7 +283,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/objspace/std/dictmultiobject.py b/pypy/objspace/std/dictmultiobject.py
--- a/pypy/objspace/std/dictmultiobject.py
+++ b/pypy/objspace/std/dictmultiobject.py
@@ -56,7 +56,7 @@
     def allocate_and_init_instance(space, w_type=None, module=False,
                                    instance=False, strdict=False,
                                    kwargs=False):
-        if space.config.objspace.std.withcelldict and module:
+        if module:
             from pypy.objspace.std.celldict import ModuleDictStrategy
             assert w_type is None
             # every module needs its own strategy, because the strategy stores
diff --git a/pypy/objspace/std/test/test_celldict.py b/pypy/objspace/std/test/test_celldict.py
--- a/pypy/objspace/std/test/test_celldict.py
+++ b/pypy/objspace/std/test/test_celldict.py
@@ -58,7 +58,6 @@
         assert v2 is v3
 
 class AppTestModuleDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         cls.w_runappdirect = cls.space.wrap(cls.runappdirect)
@@ -116,7 +115,6 @@
 
 
 class AppTestCellDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         if cls.runappdirect:
diff --git a/pypy/objspace/std/test/test_dictmultiobject.py b/pypy/objspace/std/test/test_dictmultiobject.py
--- a/pypy/objspace/std/test/test_dictmultiobject.py
+++ b/pypy/objspace/std/test/test_dictmultiobject.py
@@ -1261,7 +1261,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 
@@ -1467,6 +1466,7 @@
 
 
 def test_module_uses_strdict():
+    from pypy.objspace.std.celldict import ModuleDictStrategy
     fakespace = FakeSpace()
     d = fakespace.newdict(module=True)
-    assert type(d.get_strategy()) is BytesDictStrategy
+    assert type(d.get_strategy()) is ModuleDictStrategy
diff --git a/pypy/objspace/std/test/test_mapdict.py b/pypy/objspace/std/test/test_mapdict.py
--- a/pypy/objspace/std/test/test_mapdict.py
+++ b/pypy/objspace/std/test/test_mapdict.py
@@ -4,7 +4,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 

From pypy.commits at gmail.com  Thu Aug  3 12:56:35 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 03 Aug 2017 09:56:35 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Don't incref the original
 object in PyMemoryView_FromObject()
Message-ID: <598355c3.01571c0a.1317f.f4c0@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92050:ff315452ce6f
Date: 2017-08-03 17:53 +0100
http://bitbucket.org/pypy/pypy/changeset/ff315452ce6f/

Log:	Don't incref the original object in PyMemoryView_FromObject()

diff --git a/pypy/module/cpyext/buffer.py b/pypy/module/cpyext/buffer.py
--- a/pypy/module/cpyext/buffer.py
+++ b/pypy/module/cpyext/buffer.py
@@ -93,6 +93,7 @@
                         lltype.free(pybuf, flavor='raw')
                 decref(self.space, self.pyobj)
             self.pyobj = lltype.nullptr(PyObject.TO)
+            self.w_obj = None
         else:
             #do not call twice
             return
diff --git a/pypy/module/cpyext/memoryobject.py b/pypy/module/cpyext/memoryobject.py
--- a/pypy/module/cpyext/memoryobject.py
+++ b/pypy/module/cpyext/memoryobject.py
@@ -72,7 +72,7 @@
                     readonly=widen(view.c_readonly))
     # Ensure view.c_buf is released upon object finalization
     fq.register_finalizer(buf)
-    # Allow subclassing W_MemeoryView
+    # Allow subclassing W_MemoryView
     w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type))
     w_obj = space.allocate_instance(W_MemoryView, w_type)
     w_obj.__init__(buf)
@@ -177,11 +177,9 @@
         return (_IsCContiguous(view) or _IsFortranContiguous(view))
     return 0
 
- at cpython_api([PyObject], PyObject, result_is_ll=True)
+ at cpython_api([PyObject], PyObject)
 def PyMemoryView_FromObject(space, w_obj):
-    w_memview = space.call_method(space.builtin, "memoryview", w_obj)
-    py_memview = make_ref(space, w_memview, w_obj)
-    return py_memview
+    return space.call_method(space.builtin, "memoryview", w_obj)
 
 @cpython_api([Py_bufferP], PyObject, result_is_ll=True)
 def PyMemoryView_FromBuffer(space, view):
@@ -193,6 +191,7 @@
     # copy view into obj.c_view, without creating a new view.c_obj
     typedescr = get_typedescr(W_MemoryView.typedef)
     py_obj = typedescr.allocate(space, space.w_memoryview)
+
     py_mem = rffi.cast(PyMemoryViewObject, py_obj)
     mview = py_mem.c_view
     mview.c_buf = view.c_buf
diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py
--- a/pypy/module/cpyext/test/test_memoryobject.py
+++ b/pypy/module/cpyext/test/test_memoryobject.py
@@ -4,7 +4,7 @@
 from pypy.module.cpyext.test.test_api import BaseApiTest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from rpython.rlib.buffer import StringBuffer
-from pypy.module.cpyext.pyobject import from_ref
+from pypy.module.cpyext.pyobject import make_ref, from_ref
 from pypy.module.cpyext.memoryobject import PyMemoryViewObject
 
 only_pypy ="config.option.runappdirect and '__pypy__' not in sys.builtin_module_names"
@@ -12,9 +12,9 @@
 class TestMemoryViewObject(BaseApiTest):
     def test_frombuffer(self, space, api):
         w_buf = space.newbuffer(StringBuffer("hello"))
+        w_memoryview = api.PyMemoryView_FromObject(w_buf)
         c_memoryview = rffi.cast(
-            PyMemoryViewObject, api.PyMemoryView_FromObject(w_buf))
-        w_memoryview = from_ref(space, c_memoryview)
+            PyMemoryViewObject, make_ref(space, w_memoryview))
         view = c_memoryview.c_view
         assert view.c_ndim == 1
         f = rffi.charp2str(view.c_format)
@@ -32,6 +32,7 @@
             assert space.eq_w(space.getattr(w_mv, w_f),
                               space.getattr(w_memoryview, w_f))
         api.Py_DecRef(ref)
+        api.Py_DecRef(w_memoryview)
 
 class AppTestPyBuffer_FillInfo(AppTestCpythonExtensionBase):
     def test_fillWithObject(self):
@@ -62,7 +63,6 @@
                  """)])
         result = module.fillinfo()
         assert b"hello, world." == result
-        del result
 
 class AppTestBufferProtocol(AppTestCpythonExtensionBase):
     def test_fromobject(self):
@@ -172,8 +172,6 @@
         #         in <bound method ConcreteArray.__del__ ...> ignored
 
     def test_releasebuffer(self):
-        if not self.runappdirect:
-            skip("Fails due to ll2ctypes nonsense")
         module = self.import_extension('foo', [
             ("create_test", "METH_NOARGS",
              """

From pypy.commits at gmail.com  Thu Aug  3 13:30:31 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 03 Aug 2017 10:30:31 -0700 (PDT)
Subject: [pypy-commit] pypy default: remove some more NOT_RPYTHON and
 replace them with the decorator
Message-ID: <59835db7.0e951c0a.b4e60.88f9@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92051:4cbf2f9fe1bc
Date: 2017-08-03 19:29 +0200
http://bitbucket.org/pypy/pypy/changeset/4cbf2f9fe1bc/

Log:	remove some more NOT_RPYTHON and replace them with the decorator

diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 
 from pypy.interpreter.error import OperationError, oefmt
@@ -46,8 +47,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,6 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import rstack, rstackovf
 
 from pypy.interpreter import debug
@@ -57,8 +58,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -107,15 +109,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -142,8 +145,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,7 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -423,8 +423,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -64,8 +64,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -380,8 +380,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -621,9 +621,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -969,10 +969,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1013,8 +1013,8 @@
             self._staticdefs = zip(argnames[-len(defaults):], defaults)
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         defs_w = []
         for name, defaultval in self._staticdefs:
             if name.startswith('w_'):
@@ -1070,8 +1070,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs = gateway._getdefaults(space) # needs to be implemented by subclass
         code = gateway._code
@@ -1141,8 +1141,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1179,15 +1179,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1198,8 +1199,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1245,6 +1247,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,16 +18,17 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         self.lazy = True
         self.__class__.buildloaders()
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -61,8 +65,8 @@
         self.w_initialdict = self.space.call_method(self.w_dict, 'items')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -130,8 +134,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -161,8 +165,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space':space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -202,8 +206,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -40,13 +40,15 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -144,8 +145,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (we_are_translated, always_inline,
-        dont_inline)
+        dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -20,8 +20,8 @@
 from pypy.interpreter.pycode import PyCode, BytecodeCorruption
 from pypy.tool.stdlib_opcode import bytecode_spec
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -31,8 +31,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -8,14 +8,15 @@
 
 from rpython.rlib.jit import promote
 from rpython.rlib.objectmodel import compute_identity_hash, specialize
-from rpython.rlib.objectmodel import instantiate
+from rpython.rlib.objectmodel import instantiate, not_rpython
 from rpython.tool.sourcetools import compile2, func_with_new_name
 
 
 class TypeDef(object):
+    @not_rpython
     def __init__(self, __name, __base=None, __total_ordering__=None,
                  __buffer=None, **rawdict):
-        "NOT_RPYTHON: initialization-time only"
+        "initialization-time only"
         self.name = __name
         if __base is None:
             bases = []
@@ -113,8 +114,9 @@
 # register_finalizer() or not.
 
 @specialize.memo()
+ at not_rpython
 def get_unique_interplevel_subclass(space, cls):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert cls.typedef.acceptable_as_base_class
     try:
         return _unique_subclass_cache[cls]
@@ -349,15 +351,17 @@
         return self
 
 
+ at not_rpython
 def interp_attrproperty(name, cls, doc=None, wrapfn=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert wrapfn is not None
     def fget(space, obj):
         return getattr(space, wrapfn)(getattr(obj, name))
     return GetSetProperty(fget, cls=cls, doc=doc)
 
+ at not_rpython
 def interp_attrproperty_w(name, cls, doc=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     def fget(space, obj):
         w_value = getattr(obj, name)
         if w_value is None:
diff --git a/pypy/module/_codecs/__init__.py b/pypy/module/_codecs/__init__.py
--- a/pypy/module/_codecs/__init__.py
+++ b/pypy/module/_codecs/__init__.py
@@ -1,5 +1,6 @@
 from pypy.interpreter.mixedmodule import MixedModule
 from rpython.rlib import runicode
+from rpython.rlib.objectmodel import not_rpython
 from pypy.module._codecs import interp_codecs
 
 class Module(MixedModule):
@@ -86,9 +87,8 @@
          'unicode_internal_encode'   :  'interp_codecs.unicode_internal_encode',
     }
 
+    @not_rpython
     def __init__(self, space, *args):
-        "NOT_RPYTHON"
-
         # mbcs codec is Windows specific, and based on rffi.
         if (hasattr(runicode, 'str_decode_mbcs')):
             self.interpleveldefs['mbcs_encode'] = 'interp_codecs.mbcs_encode'
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -1,5 +1,5 @@
 from rpython.rlib import jit
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rstring import UnicodeBuilder
 from rpython.rlib.runicode import code_to_unichr, MAXUNICODE
 
@@ -268,8 +268,8 @@
         raise oefmt(space.w_TypeError,
                     "don't know how to handle %T in error callback", w_exc)
 
+ at not_rpython
 def register_builtin_error_handlers(space):
-    "NOT_RPYTHON"
     state = space.fromcache(CodecState)
     for error in ("strict", "ignore", "replace", "xmlcharrefreplace",
                   "backslashreplace"):
diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py
--- a/pypy/module/posix/interp_posix.py
+++ b/pypy/module/posix/interp_posix.py
@@ -3,7 +3,7 @@
 
 from rpython.rlib import rposix, rposix_stat
 from rpython.rlib import objectmodel, rurandom
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, intmask, r_uint
 from rpython.rlib.unroll import unrolling_iterable
 
@@ -731,8 +731,8 @@
     else:
         assert False, "Unknown fork hook"
 
+ at not_rpython
 def add_fork_hook(where, hook):
-    "NOT_RPYTHON"
     get_fork_hooks(where).append(hook)
 
 add_fork_hook('child', ExecutionContext._mark_thread_disappeared)
diff --git a/pypy/objspace/std/objspace.py b/pypy/objspace/std/objspace.py
--- a/pypy/objspace/std/objspace.py
+++ b/pypy/objspace/std/objspace.py
@@ -181,8 +181,8 @@
 
         return self._wrap_not_rpython(x)
 
+    @not_rpython
     def _wrap_not_rpython(self, x):
-        "NOT_RPYTHON"
         # _____ this code is here to support testing only _____
 
         # wrap() of a container works on CPython, but the code is
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -11,7 +11,7 @@
 from rpython.rlib.jit import (promote, elidable_promote, we_are_jitted,
      elidable, dont_look_inside, unroll_safe)
 from rpython.rlib.objectmodel import current_object_addr_as_int, compute_hash
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 
 class MutableCell(W_Root):
@@ -212,8 +212,8 @@
         else:
             self.terminator = NoDictTerminator(space, self)
 
+    @not_rpython
     def __repr__(self):
-        "NOT_RPYTHON"
         return '<W_TypeObject %r at 0x%x>' % (self.name, id(self))
 
     def mutated(self, key):
@@ -492,8 +492,9 @@
                         self, w_subtype, w_subtype)
         return w_subtype
 
+    @not_rpython
     def _cleanup_(self):
-        "NOT_RPYTHON.  Forces the lazy attributes to be computed."
+        "Forces the lazy attributes to be computed."
         if 'lazyloaders' in self.__dict__:
             for attr in self.lazyloaders.keys():
                 self.getdictvalue(self.space, attr)
@@ -1317,8 +1318,9 @@
 
 
 class TypeCache(SpaceCache):
+    @not_rpython
     def build(self, typedef):
-        "NOT_RPYTHON: initialization-time only."
+        "initialization-time only."
         from pypy.objspace.std.objectobject import W_ObjectObject
         from pypy.interpreter.typedef import GetSetProperty
         from rpython.rlib.objectmodel import instantiate

From pypy.commits at gmail.com  Thu Aug  3 14:55:18 2017
From: pypy.commits at gmail.com (exarkun)
Date: Thu, 03 Aug 2017 11:55:18 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Mark files of unknown type with `?`
Message-ID: <59837196.43301c0a.ec078.c70f@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92052:1c67ea9ef466
Date: 2017-08-03 14:46 -0400
http://bitbucket.org/pypy/pypy/changeset/1c67ea9ef466/

Log:	Mark files of unknown type with `?`

	This mirrors CPython _stat.filemode behavior though it diverges from
	the (likely rarely used) stat.filemode behavior.

diff --git a/lib-python/3/stat.py b/lib-python/3/stat.py
--- a/lib-python/3/stat.py
+++ b/lib-python/3/stat.py
@@ -139,13 +139,21 @@
 def filemode(mode):
     """Convert a file's mode to a string of the form '-rwxrwxrwx'."""
     perm = []
+
+    # The first group gets a question mark if none of the bits match the mode.
+    empty = "?"
+
     for table in _filemode_table:
         for bit, char in table:
             if mode & bit == bit:
                 perm.append(char)
                 break
         else:
-            perm.append("-")
+            perm.append(empty)
+
+        # All the rest of the positions get a - if the bits don't match.
+        empty = "-"
+
     return "".join(perm)
 
 
diff --git a/lib-python/3/test/test_stat.py b/lib-python/3/test/test_stat.py
--- a/lib-python/3/test/test_stat.py
+++ b/lib-python/3/test/test_stat.py
@@ -138,6 +138,10 @@
             self.assertS_IS("REG", st_mode)
             self.assertEqual(modestr, '-r--r--r--')
             self.assertEqual(self.statmod.S_IMODE(st_mode), 0o444)
+
+            # If there are only permission bits, no type bytes, a question
+            # mark is rendered in the type field.
+            self.assertEqual(self.statmod.filemode(0o420), '?r---w----')
         else:
             os.chmod(TESTFN, 0o700)
             st_mode, modestr = self.get_mode()

From pypy.commits at gmail.com  Thu Aug  3 15:43:21 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 03 Aug 2017 12:43:21 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: a bug somewhere in
 heap.py that affects the new code that I wrote. I did not
Message-ID: <59837cd9.832d1c0a.91f84.d59d@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92053:6a7534bc78d2
Date: 2017-08-03 21:41 +0200
http://bitbucket.org/pypy/pypy/changeset/6a7534bc78d2/

Log:	a bug somewhere in heap.py that affects the new code that I wrote. I
	did not manage to write a test for this or fix it at all. deal with
	it gracefully, instead of crashing. (it's likely not really bad, but
	might still be good to figure out what's going on).

diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -709,7 +709,12 @@
                 if not box1.is_constant() and box1 not in available_boxes:
                     continue
                 structinfo = cf.cached_infos[i]
-                box2 = structinfo.getfield(descr).get_box_replacement()
+                box2 = structinfo.getfield(descr)
+                if box2 is None:
+                    # XXX this should not happen, as it is an invariant
+                    # violation! yet it does if box1 is a constant
+                    continue
+                box2 = box2.get_box_replacement()
                 if box2.is_constant() or box2 in available_boxes:
                     result_getfield.append((box1, descr, box2))
         result_array = []
@@ -721,7 +726,12 @@
                     if not box1.is_constant() and box1 not in available_boxes:
                         continue
                     arrayinfo = cf.cached_infos[i]
-                    box2 = arrayinfo.getitem(descr, index).get_box_replacement()
+                    box2 = arrayinfo.getitem(descr, index)
+                    if box2 is None:
+                        # XXX this should not happen, as it is an invariant
+                        # violation! yet it does if box1 is a constant
+                        continue
+                    box2 = box2.get_box_replacement()
                     if box2.is_constant() or box2 in available_boxes:
                         result_array.append((box1, index, descr, box2))
         return result_getfield, result_array

From pypy.commits at gmail.com  Thu Aug  3 17:17:10 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 03 Aug 2017 14:17:10 -0700 (PDT)
Subject: [pypy-commit] pypy default: add a jit driver for array.count and
 array.index
Message-ID: <598392d6.01571c0a.14ca8.37cf@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92054:5361e9bb4b15
Date: 2017-08-03 23:14 +0200
http://bitbucket.org/pypy/pypy/changeset/5361e9bb4b15/

Log:	add a jit driver for array.count and array.index

diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -118,6 +118,29 @@
         return space.w_True
     return space.w_False
 
+index_count_jd = jit.JitDriver(
+    greens = ['count', 'tp_item', 'arrclass'],
+    reds = 'auto', name = 'array.index_or_count')
+
+def index_count_array(arr, w_val, count=False):
+    space = arr.space
+    tp_item = space.type(w_val)
+    arrclass = arr.__class__
+    cnt = 0
+    for i in range(arr.len):
+        index_count_jd.jit_merge_point(
+            tp_item=tp_item, count=count,
+            arrclass=arrclass)
+        w_item = arr.w_getitem(space, i)
+        if space.eq_w(w_item, w_val):
+            if count:
+                cnt += 1
+            else:
+                return i
+    if count:
+        return cnt
+    return -1
+
 UNICODE_ARRAY = lltype.Ptr(lltype.Array(lltype.UniChar,
                                         hints={'nolength': True}))
 
@@ -257,17 +280,12 @@
         """
         self.extend(w_x)
 
-    def descr_count(self, space, w_val):
+    def descr_count(self, space, w_x):
         """ count(x)
 
         Return number of occurrences of x in the array.
         """
-        cnt = 0
-        for i in range(self.len):
-            # XXX jitdriver
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_val):
-                cnt += 1
+        cnt = index_count_array(self, w_x, count=True)
         return space.newint(cnt)
 
     def descr_index(self, space, w_x):
@@ -275,10 +293,9 @@
 
         Return index of first occurrence of x in the array.
         """
-        for i in range(self.len):
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_x):
-                return space.newint(i)
+        res = index_count_array(self, w_x, count=False)
+        if res >= 0:
+            return space.newint(res)
         raise oefmt(space.w_ValueError, "array.index(x): x not in list")
 
     def descr_reverse(self, space):

From pypy.commits at gmail.com  Thu Aug  3 20:16:16 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:16 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: merge default into branch
Message-ID: <5983bcd0.db85df0a.b03cf.c742@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92055:5fb13e8e0a8f
Date: 2017-08-01 09:23 -0700
http://bitbucket.org/pypy/pypy/changeset/5fb13e8e0a8f/

Log:	merge default into branch

diff too long, truncating to 2000 out of 5319 lines

diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py
--- a/lib_pypy/_tkinter/tklib_build.py
+++ b/lib_pypy/_tkinter/tklib_build.py
@@ -22,12 +22,27 @@
     linklibs = ['tcl', 'tk']
     libdirs = []
 else:
-    for _ver in ['', '8.6', '8.5', '']:
+    # On some Linux distributions, the tcl and tk libraries are
+    # stored in /usr/include, so we must check this case also
+    libdirs = []
+    found = False
+    for _ver in ['', '8.6', '8.5']:
         incdirs = ['/usr/include/tcl' + _ver]
         linklibs = ['tcl' + _ver, 'tk' + _ver]
-        libdirs = []
         if os.path.isdir(incdirs[0]):
+            found = True
             break
+    if not found:
+        for _ver in ['8.6', '8.5', '']:
+            incdirs = []
+            linklibs = ['tcl' + _ver, 'tk' + _ver]
+            if os.path.isfile(''.join(['/usr/lib/lib', linklibs[1], '.so'])):
+                found = True
+                break
+    if not found:
+        sys.stderr.write("*** TCL libraries not found!  Falling back...\n")
+        incdirs = []
+        linklibs = ['tcl', 'tk']
 
 config_ffi = FFI()
 config_ffi.cdef("""
diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h
--- a/lib_pypy/cffi/_cffi_include.h
+++ b/lib_pypy/cffi/_cffi_include.h
@@ -95,6 +95,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h
--- a/lib_pypy/cffi/_embedding.h
+++ b/lib_pypy/cffi/_embedding.h
@@ -1,7 +1,12 @@
 
 /***** Support code for embedding *****/
 
-#if defined(_MSC_VER)
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+#if defined(_WIN32)
 #  define CFFI_DLLEXPORT  __declspec(dllexport)
 #elif defined(__GNUC__)
 #  define CFFI_DLLEXPORT  __attribute__((visibility("default")))
@@ -525,3 +530,7 @@
 #undef cffi_compare_and_swap
 #undef cffi_write_barrier
 #undef cffi_read_barrier
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py
--- a/lib_pypy/cffi/recompiler.py
+++ b/lib_pypy/cffi/recompiler.py
@@ -412,6 +412,9 @@
             prnt('    }')
         prnt('    p[0] = (const void *)0x%x;' % self._version)
         prnt('    p[1] = &_cffi_type_context;')
+        prnt('#if PY_MAJOR_VERSION >= 3')
+        prnt('    return NULL;')
+        prnt('#endif')
         prnt('}')
         # on Windows, distutils insists on putting init_cffi_xyz in
         # 'export_symbols', so instead of fighting it, just give up and
@@ -578,7 +581,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.BasePrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif isinstance(tp, model.UnknownFloatType):
                 return '_cffi_from_c_double(%s)' % (var,)
diff --git a/lib_pypy/cffi/vengine_cpy.py b/lib_pypy/cffi/vengine_cpy.py
--- a/lib_pypy/cffi/vengine_cpy.py
+++ b/lib_pypy/cffi/vengine_cpy.py
@@ -296,7 +296,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.PrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif tp.name != 'long double':
                 return '_cffi_from_c_%s(%s)' % (tp.name.replace(' ', '_'), var)
@@ -872,6 +872,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst
--- a/pypy/doc/cpython_differences.rst
+++ b/pypy/doc/cpython_differences.rst
@@ -330,6 +330,8 @@
 
  - ``frozenset`` (empty frozenset only)
 
+ - unbound method objects (for Python 2 only)
+
 This change requires some changes to ``id`` as well. ``id`` fulfills the
 following condition: ``x is y <=> id(x) == id(y)``. Therefore ``id`` of the
 above types will return a value that is computed from the argument, and can
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------
diff --git a/pypy/doc/how-to-release.rst b/pypy/doc/how-to-release.rst
--- a/pypy/doc/how-to-release.rst
+++ b/pypy/doc/how-to-release.rst
@@ -40,6 +40,9 @@
   sure things are ported back to the trunk and to the branch as
   necessary.
 
+* Maybe bump the SOABI number in module/imp/importing. This has many
+  implications, so make sure the PyPy community agrees to the change.
+
 * Update and write documentation
 
   * update pypy/doc/contributor.rst (and possibly LICENSE)
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -5,6 +5,14 @@
 .. this is a revision shortly after release-pypy2.7-v5.8.0
 .. startrev: 558bd00b3dd8
 
+In previous versions of PyPy, ``instance.method`` would return always
+the same bound method object, when gotten out of the same instance (as
+far as ``is`` and ``id()`` can tell).  CPython doesn't do that.  Now
+PyPy, like CPython, returns a different bound method object every time.
+For ``type.method``, PyPy2 still returns always the same *unbound*
+method object; CPython does it for built-in types but not for
+user-defined types.
+
 .. branch: cffi-complex
 .. branch: cffi-char16-char32
 
@@ -30,3 +38,20 @@
 
 Renaming of ``cppyy`` to ``_cppyy``.
 The former is now an external package installable with ``pip install cppyy``.
+
+.. branch: Enable_PGO_for_clang
+
+.. branch: nopax
+
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
+
+.. branch: pypy_bytearray
+
+Improve ``bytearray`` performance (backported from py3.5)
+
+.. branch: gc-del-limit-growth
+
+Fix the bounds in the GC when allocating a lot of objects with finalizers,
+fixes issue #2590
diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -559,21 +559,29 @@
         return space.newbool(space.eq_w(self.w_function, w_other.w_function))
 
     def is_w(self, space, other):
+        if self.w_instance is not None:
+            return W_Root.is_w(self, space, other)
+        # The following special-case is only for *unbound* method objects.
+        # Motivation: in CPython, it seems that no strange internal type
+        # exists where the equivalent of ``x.method is x.method`` would
+        # return True.  This is unlike unbound methods, where e.g.
+        # ``list.append is list.append`` returns True.  The following code
+        # is here to emulate that behaviour.  Unlike CPython, we return
+        # True for all equal unbound methods, not just for built-in types.
         if not isinstance(other, Method):
             return False
-        return (self.w_instance is other.w_instance and
+        return (other.w_instance is None and
                 self.w_function is other.w_function and
                 self.w_class is other.w_class)
 
     def immutable_unique_id(self, space):
-        from pypy.objspace.std.util import IDTAG_METHOD as tag
+        if self.w_instance is not None:
+            return W_Root.immutable_unique_id(self, space)
+        # the special-case is only for *unbound* method objects
+        #
+        from pypy.objspace.std.util import IDTAG_UNBOUND_METHOD as tag
         from pypy.objspace.std.util import IDTAG_SHIFT
-        if self.w_instance is not None:
-            id = space.bigint_w(space.id(self.w_instance))
-            id = id.lshift(LONG_BIT)
-        else:
-            id = rbigint.fromint(0)
-        id = id.or_(space.bigint_w(space.id(self.w_function)))
+        id = space.bigint_w(space.id(self.w_function))
         id = id.lshift(LONG_BIT).or_(space.bigint_w(space.id(self.w_class)))
         id = id.lshift(IDTAG_SHIFT).int_or_(tag)
         return space.newlong_from_rbigint(id)
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -1,4 +1,4 @@
-import pytest
+import pytest, sys
 from pypy.interpreter import eval
 from pypy.interpreter.function import Function, Method, descr_function_get
 from pypy.interpreter.pycode import PyCode
@@ -342,6 +342,11 @@
         raises(ValueError, type(f).__setstate__, f, (1, 2, 3))
 
 class AppTestMethod:
+    def setup_class(cls):
+        cls.w_runappdirect_on_cpython = cls.space.wrap(
+            cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
+
     def test_simple_call(self):
         class A(object):
             def func(self, arg2):
@@ -572,7 +577,6 @@
         assert meth == meth
         assert meth == MethodType(func, object)
 
-    @pytest.mark.skipif("config.option.runappdirect")
     def test_method_identity(self):
         class A(object):
             def m(self):
@@ -589,19 +593,24 @@
 
         a = A()
         a2 = A()
-        assert a.m is a.m
-        assert id(a.m) == id(a.m)
-        assert a.m is not a.n
-        assert id(a.m) != id(a.n)
-        assert a.m is not a2.m
-        assert id(a.m) != id(a2.m)
+        x = a.m; y = a.m
+        assert x is not y
+        assert id(x) != id(y)
+        assert x == y
+        assert x is not a.n
+        assert id(x) != id(a.n)
+        assert x is not a2.m
+        assert id(x) != id(a2.m)
 
-        assert A.m is A.m
-        assert id(A.m) == id(A.m)
-        assert A.m is not A.n
-        assert id(A.m) != id(A.n)
-        assert A.m is not B.m
-        assert id(A.m) != id(B.m)
+        if not self.runappdirect_on_cpython:
+            assert A.m is A.m
+            assert id(A.m) == id(A.m)
+        assert A.m == A.m
+        x = A.m
+        assert x is not A.n
+        assert id(x) != id(A.n)
+        assert x is not B.m
+        assert id(x) != id(B.m)
 
 
 class TestMethod:
diff --git a/pypy/module/_cffi_backend/test/_backend_test_c.py b/pypy/module/_cffi_backend/test/_backend_test_c.py
--- a/pypy/module/_cffi_backend/test/_backend_test_c.py
+++ b/pypy/module/_cffi_backend/test/_backend_test_c.py
@@ -3838,6 +3838,7 @@
         assert result == samples
         for i in range(len(samples)):
             assert result[i] == p[i] and type(result[i]) is type(p[i])
+            assert (type(result[i]) is bool) == (type(samples[i]) is bool)
     #
     BInt = new_primitive_type("int")
     py.test.raises(TypeError, unpack, p)
diff --git a/pypy/module/_vmprof/__init__.py b/pypy/module/_vmprof/__init__.py
--- a/pypy/module/_vmprof/__init__.py
+++ b/pypy/module/_vmprof/__init__.py
@@ -11,7 +11,6 @@
     interpleveldefs = {
         'enable': 'interp_vmprof.enable',
         'disable': 'interp_vmprof.disable',
-        'write_all_code_objects': 'interp_vmprof.write_all_code_objects',
         'is_enabled': 'interp_vmprof.is_enabled',
         'get_profile_path': 'interp_vmprof.get_profile_path',
         'stop_sampling': 'interp_vmprof.stop_sampling',
diff --git a/pypy/module/_vmprof/interp_vmprof.py b/pypy/module/_vmprof/interp_vmprof.py
--- a/pypy/module/_vmprof/interp_vmprof.py
+++ b/pypy/module/_vmprof/interp_vmprof.py
@@ -70,11 +70,6 @@
     except rvmprof.VMProfError as e:
         raise VMProfError(space, e)
 
-def write_all_code_objects(space):
-    """ Needed on cpython, just empty function here
-    """
-    pass
-
 def disable(space):
     """Disable vmprof.  Remember to close the file descriptor afterwards
     if necessary.
diff --git a/pypy/module/_vmprof/test/test__vmprof.py b/pypy/module/_vmprof/test/test__vmprof.py
--- a/pypy/module/_vmprof/test/test__vmprof.py
+++ b/pypy/module/_vmprof/test/test__vmprof.py
@@ -1,3 +1,4 @@
+import sys
 from rpython.tool.udir import udir
 from pypy.tool.pytest.objspace import gettestobjspace
 
@@ -7,6 +8,8 @@
     def setup_class(cls):
         cls.w_tmpfilename = cls.space.wrap(str(udir.join('test__vmprof.1')))
         cls.w_tmpfilename2 = cls.space.wrap(str(udir.join('test__vmprof.2')))
+        cls.w_plain = cls.space.wrap(not cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
 
     def test_import_vmprof(self):
         tmpfile = open(self.tmpfilename, 'wb')
@@ -115,3 +118,33 @@
                 assert fd1.read() == tmpfile.read()
         _vmprof.disable()
         assert _vmprof.get_profile_path() is None
+
+    def test_stop_sampling(self):
+        if not self.plain:
+            skip("unreliable test except on CPython without -A")
+        import os
+        import _vmprof
+        tmpfile = open(self.tmpfilename, 'wb')
+        native = 1
+        def f():
+            import sys
+            import math
+            j = sys.maxsize
+            for i in range(500):
+                j = math.sqrt(j)
+        _vmprof.enable(tmpfile.fileno(), 0.01, 0, native, 0, 0)
+        # get_vmprof_stack() always returns 0 here!
+        # see vmprof_common.c and assume RPYTHON_LL2CTYPES is defined!
+        f()
+        fileno = _vmprof.stop_sampling()
+        pos = os.lseek(fileno, 0, os.SEEK_CUR)
+        f()
+        pos2 = os.lseek(fileno, 0, os.SEEK_CUR)
+        assert pos == pos2
+        _vmprof.start_sampling()
+        f()
+        fileno = _vmprof.stop_sampling()
+        pos3 = os.lseek(fileno, 0, os.SEEK_CUR)
+        assert pos3 > pos
+        _vmprof.disable()
+
diff --git a/pypy/module/cppyy/test/test_cint.py b/pypy/module/cppyy/test/test_cint.py
deleted file mode 100644
--- a/pypy/module/cppyy/test/test_cint.py
+++ /dev/null
@@ -1,710 +0,0 @@
-import py, os, sys
-
-# These tests are for the CINT backend only (they exercise ROOT features
-# and classes that are not loaded/available with the Reflex backend). At
-# some point, these tests are likely covered by the CLang/LLVM backend.
-from pypy.module.cppyy import capi
-if capi.identify() != 'CINT':
-    py.test.skip("backend-specific: CINT-only tests")
-
-# load _cffi_backend early, or its global vars are counted as leaks in the
-# test (note that the module is not otherwise used in the test itself)
-from pypy.module._cffi_backend import newtype
-
-currpath = py.path.local(__file__).dirpath()
-iotypes_dct = str(currpath.join("iotypesDict.so"))
-
-def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make CINT=t iotypesDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
-
-class AppTestCINT:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_globals(self):
-        """Test the availability of ROOT globals"""
-
-        import cppyy
-
-        assert cppyy.gbl.gROOT
-        assert cppyy.gbl.gApplication
-        assert cppyy.gbl.gSystem
-        assert cppyy.gbl.TInterpreter.Instance()           # compiled
-        assert cppyy.gbl.TInterpreter                      # interpreted
-        assert cppyy.gbl.TDirectory.CurrentDirectory()     # compiled
-        assert cppyy.gbl.TDirectory                        # interpreted
-
-    def test02_write_access_to_globals(self):
-        """Test overwritability of ROOT globals"""
-
-        import cppyy
-
-        oldval = cppyy.gbl.gDebug
-        assert oldval != 3
-
-        proxy = cppyy.gbl.__class__.__dict__['gDebug']
-        cppyy.gbl.gDebug = 3
-        assert proxy.__get__(proxy, None) == 3
-
-        # this is where this test differs from test03_write_access_to_globals
-        # in test_pythonify.py
-        cppyy.gbl.gROOT.ProcessLine('int gDebugCopy = gDebug;')
-        assert cppyy.gbl.gDebugCopy == 3
-
-        cppyy.gbl.gDebug = oldval
-
-    def test03_create_access_to_globals(self):
-        """Test creation and access of new ROOT globals"""
-
-        import cppyy
-
-        cppyy.gbl.gROOT.ProcessLine('double gMyOwnGlobal = 3.1415')
-        assert cppyy.gbl.gMyOwnGlobal == 3.1415
-
-        proxy = cppyy.gbl.__class__.__dict__['gMyOwnGlobal']
-        assert proxy.__get__(proxy, None) == 3.1415
-
-    def test04_auto_loading(self):
-        """Test auto-loading by retrieving a non-preloaded class"""
-
-        import cppyy
-
-        l = cppyy.gbl.TLorentzVector()
-        assert isinstance(l, cppyy.gbl.TLorentzVector)
-
-    def test05_macro_loading(self):
-        """Test accessibility to macro classes"""
-
-        import cppyy
-
-        loadres = cppyy.gbl.gROOT.LoadMacro('simple_class.C')
-        assert loadres == 0
-
-        base = cppyy.gbl.MySimpleBase
-        simple = cppyy.gbl.MySimpleDerived
-        simple_t = cppyy.gbl.MySimpleDerived_t
-
-        assert issubclass(simple, base)
-        assert simple is simple_t
-
-        c = simple()
-        assert isinstance(c, simple)
-        assert c.m_data == c.get_data()
-
-        c.set_data(13)
-        assert c.m_data == 13
-        assert c.get_data() == 13
-
-
-class AppTestCINTPYTHONIZATIONS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_strings(self):
-        """Test TString/TObjString compatibility"""
-
-        import cppyy
-
-        pyteststr = "aap noot mies"
-        def test_string(s1, s2):
-            assert len(s1) == len(s2)
-            assert s1 == s1
-            assert s1 == s2
-            assert s1 == str(s1)
-            assert s1 == pyteststr
-            assert s1 != "aap"
-            assert s1 != ""
-            assert s1 < "noot"
-            assert repr(s1) == repr(s2)
-
-        s1 = cppyy.gbl.TString(pyteststr)
-        test_string(s1, pyteststr)
-
-        s3 = cppyy.gbl.TObjString(pyteststr)
-        test_string(s3, pyteststr)
-
-    def test03_TVector(self):
-        """Test TVector2/3/T behavior"""
-
-        import cppyy, math
-
-        N = 51
-
-        # TVectorF is a typedef of floats
-        v = cppyy.gbl.TVectorF(N)
-        for i in range(N):
-            v[i] = i*i
-
-        assert len(v) == N
-        for j in v:
-            assert round(v[int(math.sqrt(j)+0.5)]-j, 5) == 0.
-
-    def test04_TStringTObjString(self):
-        """Test string/TString interchangebility"""
-
-        import cppyy
-
-        test = "aap noot mies"
-
-        s1 = cppyy.gbl.TString(test )
-        s2 = str(s1)
-
-        assert s1 == test
-        assert test == s2
-        assert s1 == s2
-
-        s3 = cppyy.gbl.TObjString(s2)
-        assert s3 == test
-        assert s2 == s3
-
-        # force use of: TNamed(const TString &name, const TString &title)
-        n = cppyy.gbl.TNamed(test, cppyy.gbl.TString("title"))
-        assert n.GetTitle() == "title"
-        assert n.GetName() == test
-
-
-class AppTestCINTTTREE:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def setup_class(cls):
-        cls.w_N = cls.space.newint(5)
-        cls.w_M = cls.space.newint(10)
-        cls.w_fname = cls.space.newtext("test.root")
-        cls.w_tname = cls.space.newtext("test")
-        cls.w_title = cls.space.newtext("test tree")
-        cls.w_iotypes = cls.space.appexec([], """():
-            import cppyy
-            return cppyy.load_reflection_info(%r)""" % (iotypes_dct,))
-
-    def test01_write_stdvector(self):
-        """Test writing of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        v = vector("double")()
-        raises(TypeError, TTree.Branch, None, "mydata", v.__class__.__name__, v)
-        raises(TypeError, TTree.Branch, v, "mydata", v.__class__.__name__, v)
-
-        mytree.Branch("mydata", v.__class__.__name__, v)
-
-        for i in range(self.N):
-            for j in range(self.M):
-                v.push_back(i*self.M+j)
-            mytree.Fill()
-            v.clear()
-        f.Write()
-        f.Close()
-
-    def test02_file_open(self):
-
-        from cppyy import gbl
-
-        f = gbl.TFile.Open(self.fname)
-        s = str(f)            # should not raise
-        r = repr(f)
-
-        f.Close()
-
-    def test03_read_stdvector(self):
-        """Test reading of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        i = 0
-        for event in mytree:
-            assert len(event.mydata) == self.M
-            for entry in event.mydata:
-                assert i == int(entry)
-                i += 1
-        assert i == self.N * self.M
-
-        f.Close()
-
-    def test04_write_some_data_object(self):
-        """Test writing of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree, IO
-        from cppyy.gbl.IO import SomeDataObject
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-
-        d = SomeDataObject()
-        b = mytree.Branch("data", d)
-        mytree._python_owns = False
-        assert b
-
-        for i in range(self.N):
-            for j in range(self.M):
-                d.add_float(i*self.M+j)
-            d.add_tuple(d.get_floats())
-
-            mytree.Fill()
-
-        f.Write()
-        f.Close()
-
-    def test05_read_some_data_object(self):
-        """Test reading of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        j = 1
-        for event in mytree:
-            i = 0
-            assert len(event.data.get_floats()) == j*self.M
-            for entry in event.data.get_floats():
-                assert i == int(entry)
-                i += 1
-
-            k = 1
-            assert len(event.data.get_tuples()) == j
-            for mytuple in event.data.get_tuples():
-                i = 0
-                assert len(mytuple) == k*self.M
-                for entry in mytuple:
-                    assert i == int(entry)
-                    i += 1
-                k += 1
-            j += 1
-        assert j-1 == self.N
-        #
-        f.Close()
-
-    def test06_branch_activation(self):
-        """Test of automatic branch activation"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        L = 5
-
-        # writing
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        for i in range(L):
-            v = vector("double")()
-            mytree.Branch("mydata_%d"%i, v.__class__.__name__, v)
-            mytree.__dict__["v_%d"%i] = v
-
-        for i in range(self.N):
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                for j in range(self.M):
-                    mytree.__dict__["v_%d"%k].push_back(i*self.M+j*L+k)
-            mytree.Fill()
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                v.clear()
-        f.Write()
-        f.Close()
-
-        del mytree, f
-        import gc
-        gc.collect()
-
-        # reading
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # force (initial) disabling of all branches
-        mytree.SetBranchStatus("*",0);
-
-        i = 0
-        for event in mytree:
-            for k in range(L):
-                j = 0
-                data = getattr(mytree, "mydata_%d"%k)
-                assert len(data) == self.M
-                for entry in data:
-                    assert entry == i*self.M+j*L+k
-                    j += 1
-                assert j == self.M
-            i += 1
-        assert i == self.N
-
-        f.Close()
-
-    def test07_write_builtin(self):
-        """Test writing of builtins"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        import array
-        mytree.ba = array.array('c', [chr(0)])
-        mytree.ia = array.array('i', [0])
-        mytree.da = array.array('d', [0.])
-
-        mytree.Branch("my_bool",   mytree.ba, "my_bool/O")
-        mytree.Branch("my_int",    mytree.ia, "my_int/I")
-        mytree.Branch("my_int2",   mytree.ia, "my_int2/I")
-        mytree.Branch("my_double", mytree.da, "my_double/D")
-
-        for i in range(self.N):
-            # make sure value is different from default (0)
-            mytree.ba[0] = i%2 and chr(0) or chr(1)
-            mytree.ia[0] = i+1
-            mytree.da[0] = (i+1)/2.
-            mytree.Fill()
-        f.Write()
-        f.Close()
-
-    def test08_read_builtin(self):
-        """Test reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        raises(AttributeError, getattr, mytree, "does_not_exist")
-
-        i = 1
-        for event in mytree:
-            assert event.my_bool   == (i-1)%2 and 0 or 1
-            assert event.my_int    == i
-            assert event.my_double == i/2.
-            i += 1
-        assert (i-1) == self.N
-
-        f.Close()
-
-    def test09_user_read_builtin(self):
-        """Test user-directed reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # note, this is an old, annoted tree from test08
-        for i in range(3, mytree.GetEntriesFast()):
-            mytree.GetEntry(i)
-            assert mytree.my_int  == i+1
-            assert mytree.my_int2 == i+1
-
-        f.Close()
-
-class AppTestCINTREGRESSION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that at some point in the past resulted in failures on
-    # PyROOT; kept here to confirm no regression from PyROOT
-
-    def test01_regression(self):
-        """TPaveText::AddText() used to result in KeyError"""
-
-        # This is where the original problem was discovered, and the test is
-        # left in. However, the detailed underlying problem, as well as the
-        # solution to it, is tested in test_fragile.py
-
-        from cppyy import gbl
-        from cppyy.gbl import TPaveText
-
-        hello = TPaveText( .1, .8, .9, .97 )
-        hello.AddText( 'Hello, World!' )
-
-
-class AppTestCINTFUNCTION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-    _pypytest_leaks = None   # TODO: figure out the false positives
-
-    # test the function callbacks; this does not work with Reflex, as it can
-    # not generate functions on the fly (it might with cffi?)
-
-    @py.test.mark.dont_track_allocations("TODO: understand; initialization left-over?")
-    def test01_global_function_callback(self):
-        """Test callback of a python global function"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        def identity(x):
-            return x[0]
-
-        f = TF1("pyf1", identity, -1., 1., 0)
-
-        assert f.Eval(0.5)  == 0.5
-        assert f.Eval(-10.) == -10.
-        assert f.Eval(1.0)  == 1.0
-
-        # check proper propagation of default value
-        f = TF1("pyf1d", identity, -1., 1.)
-
-        assert f.Eval(0.5) == 0.5
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test02_callable_object_callback(self):
-        """Test callback of a python callable object"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        class Linear:
-            def __call__(self, x, par):
-                return par[0] + x[0]*par[1]
-
-        f = TF1("pyf2", Linear(), -1., 1., 2)
-        f.SetParameters(5., 2.)
-
-        assert f.Eval(-0.1) == 4.8
-        assert f.Eval(1.3)  == 7.6
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test03_fit_with_python_gaussian(self):
-        """Test fitting with a python global function"""
-
-        # note: this function is dread-fully slow when running testing un-translated
-
-        import cppyy, gc, math
-        TF1, TH1F = cppyy.gbl.TF1, cppyy.gbl.TH1F
-
-        def pygaus(x, par):
-            arg1 = 0
-            scale1 = 0
-            ddx = 0.01
-
-            if (par[2] != 0.0):
-                arg1 = (x[0]-par[1])/par[2]
-                scale1 = (ddx*0.39894228)/par[2]
-                h1 = par[0]/(1+par[3])
-
-                gauss = h1*scale1*math.exp(-0.5*arg1*arg1)
-            else:
-                gauss = 0.
-            return gauss
-
-        f = TF1("pygaus", pygaus, -4, 4, 4)
-        f.SetParameters(600, 0.43, 0.35, 600)
-
-        h = TH1F("h", "test", 100, -4, 4)
-        h.FillRandom("gaus", 200000)
-        h.Fit(f, "0Q")
-
-        assert f.GetNDF() == 96
-        result = f.GetParameters()
-        assert round(result[1] - 0., 1) == 0  # mean
-        assert round(result[2] - 1., 1) == 0  # s.d.
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-
-class AppTestSURPLUS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that were historically exercised on ROOT classes and
-    # have twins on custom classes; kept here just in case differences crop
-    # up between the ROOT classes and the custom ones
-
-    def test01_class_enum(self):
-        """Test class enum access and values"""
-
-        import cppyy
-        TObject = cppyy.gbl.TObject
-        gROOT = cppyy.gbl.gROOT
-
-        assert TObject.kBitMask    == gROOT.ProcessLine("return TObject::kBitMask;")
-        assert TObject.kIsOnHeap   == gROOT.ProcessLine("return TObject::kIsOnHeap;")
-        assert TObject.kNotDeleted == gROOT.ProcessLine("return TObject::kNotDeleted;")
-        assert TObject.kZombie     == gROOT.ProcessLine("return TObject::kZombie;")
-
-        t = TObject()
-
-        assert TObject.kBitMask    == t.kBitMask
-        assert TObject.kIsOnHeap   == t.kIsOnHeap
-        assert TObject.kNotDeleted == t.kNotDeleted
-        assert TObject.kZombie     == t.kZombie
-
-    def test02_global_enum(self):
-        """Test global enums access and values"""
-
-        import cppyy
-        from cppyy import gbl
-
-        assert gbl.kRed   == gbl.gROOT.ProcessLine("return kRed;")
-        assert gbl.kGreen == gbl.gROOT.ProcessLine("return kGreen;")
-        assert gbl.kBlue  == gbl.gROOT.ProcessLine("return kBlue;")
-
-    def test03_copy_contructor(self):
-        """Test copy constructor"""
-
-        import cppyy
-        TLorentzVector = cppyy.gbl.TLorentzVector
-
-        t1 = TLorentzVector(1., 2., 3., -4.)
-        t2 = TLorentzVector(0., 0., 0.,  0.)
-        t3 = TLorentzVector(t1)
-
-        assert t1 == t3
-        assert t1 != t2
-
-        for i in range(4):
-            assert t1[i] == t3[i]
-
-    def test04_object_validity(self):
-        """Test object validity checking"""
-
-        import cppyy
-
-        t1 = cppyy.gbl.TObject()
-
-        assert t1
-        assert not not t1
-
-        t2 = cppyy.gbl.gROOT.FindObject("Nah, I don't exist")
-
-        assert not t2
-
-    def test05_element_access(self):
-        """Test access to elements in matrix and array objects."""
-
-        from cppyy import gbl
-
-        N = 3
-        v = gbl.TVectorF(N)
-        m = gbl.TMatrixD(N, N)
-
-        for i in range(N):
-            assert v[i] == 0.0
-
-            for j in range(N):
-                assert m[i][j] == 0.0
-
-    def test06_static_function_call( self ):
-        """Test call to static function."""
-
-        import cppyy
-        TROOT, gROOT = cppyy.gbl.TROOT, cppyy.gbl.gROOT
-
-        c1 = TROOT.Class()
-        assert not not c1
-
-        c2 = gROOT.Class()
-
-        assert c1 == c2
-
-        old = gROOT.GetDirLevel()
-        TROOT.SetDirLevel(2)
-        assert 2 == gROOT.GetDirLevel()
-        gROOT.SetDirLevel(old)
-
-        old = TROOT.GetDirLevel()
-        gROOT.SetDirLevel(3)
-        assert 3 == TROOT.GetDirLevel()
-        TROOT.SetDirLevel(old)
-
-    def test07_macro(self):
-        """Test access to cpp macro's"""
-
-        from cppyy import gbl
-
-        assert gbl.NULL == 0
-
-        gbl.gROOT.ProcessLine('#define aap "aap"')
-        gbl.gROOT.ProcessLine('#define noot 1')
-        gbl.gROOT.ProcessLine('#define mies 2.0')
-
-        # TODO: macro's assumed to always be of long type ...
-        #assert gbl.aap  == "aap"
-        assert gbl.noot == 1
-        #assert gbl.mies == 2.0
-
-    def test08_opaque_pointer_passing(self):
-        """Test passing around of opaque pointers"""
-
-        import cppyy
-
-        # TODO: figure out CObject (see also test_advanced.py)
-
-        s = cppyy.gbl.TString("Hello World!")
-        #cobj = cppyy.as_cobject(s)
-        addr = cppyy.addressof(s)
-
-        #assert s == cppyy.bind_object(cobj, s.__class__)
-        #assert s == cppyy.bind_object(cobj, "TString")
-        assert s == cppyy.bind_object(addr, s.__class__)
-        assert s == cppyy.bind_object(addr, "TString")
-
-    def test09_object_and_pointer_comparisons(self):
-        """Verify object and pointer comparisons"""
-
-        import cppyy
-        gbl = cppyy.gbl
-
-        c1 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == None
-        assert None == c1
-
-        c2 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == c2
-        assert c2 == c1
-
-        # TLorentzVector overrides operator==
-        l1 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == None
-        assert None == l1
-
-        assert c1 != l1
-        assert l1 != c1
-
-        l2 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == l2
-        assert l2 == l1 
-
-        l3 = gbl.TLorentzVector(1, 2, 3, 4)
-        l4 = gbl.TLorentzVector(1, 2, 3, 4)
-        l5 = gbl.TLorentzVector(4, 3, 2, 1)
-        assert l3 == l4
-        assert l4 == l3
-
-        assert l3 != None                 # like this to ensure __ne__ is called
-        assert None != l3                 # id.
-        assert l3 != l5
-        assert l5 != l3
-
-    def test10_recursive_remove(self):
-        """Verify that objects are recursively removed when destroyed"""
-
-        import cppyy
-
-        c = cppyy.gbl.TClass.GetClass("TObject")
-
-        o = cppyy.gbl.TObject()
-        assert o
-
-        o.SetBit(cppyy.gbl.TObject.kMustCleanup)
-        c.Destructor(o)
-        assert not o
diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py
--- a/pypy/module/cpyext/test/test_api.py
+++ b/pypy/module/cpyext/test/test_api.py
@@ -64,14 +64,7 @@
         except OperationError as e:
             print e.errorstr(self.space)
             raise
-
-        try:
-            self.space.getexecutioncontext().cleanup_cpyext_state()
-        except AttributeError:
-            pass
-
-        if self.check_and_print_leaks():
-            assert False, "Test leaks or loses object(s)."
+        self.cleanup()
 
 @slot_function([PyObject], lltype.Void)
 def PyPy_GetWrapped(space, w_arg):
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -1,16 +1,12 @@
 import sys
-import weakref
 
 import pytest
 
-from pypy.tool.cpyext.extbuild import (
-    SystemCompilationInfo, HERE, get_sys_info_app)
+from pypy.tool.cpyext.extbuild import SystemCompilationInfo, HERE
 from pypy.interpreter.gateway import unwrap_spec, interp2app
-from rpython.rtyper.lltypesystem import lltype, ll2ctypes
+from rpython.rtyper.lltypesystem import lltype
 from pypy.module.cpyext import api
 from pypy.module.cpyext.state import State
-from pypy.module.cpyext.pyobject import Py_DecRef
-from rpython.tool.identity_dict import identity_dict
 from rpython.tool import leakfinder
 from rpython.rlib import rawrefcount
 from rpython.tool.udir import udir
@@ -76,13 +72,6 @@
 
 def freeze_refcnts(self):
     rawrefcount._dont_free_any_more()
-    return #ZZZ
-    state = self.space.fromcache(RefcountState)
-    self.frozen_refcounts = {}
-    for w_obj, obj in state.py_objects_w2r.iteritems():
-        self.frozen_refcounts[w_obj] = obj.c_ob_refcnt
-    #state.print_refcounts()
-    self.frozen_ll2callocations = set(ll2ctypes.ALLOCATED.values())
 
 class LeakCheckingTest(object):
     """Base class for all cpyext tests."""
@@ -91,78 +80,13 @@
                                    'micronumpy', 'mmap'
                                    ])
 
-    enable_leak_checking = True
+    def cleanup(self):
+        self.space.getexecutioncontext().cleanup_cpyext_state()
+        rawrefcount._collect()
+        self.space.user_del_action._run_finalizers()
+        leakfinder.stop_tracking_allocations(check=False)
+        assert not self.space.finalizer_queue.next_dead()
 
-    @staticmethod
-    def cleanup_references(space):
-        return #ZZZ
-        state = space.fromcache(RefcountState)
-
-        import gc; gc.collect()
-        # Clear all lifelines, objects won't resurrect
-        for w_obj, obj in state.lifeline_dict._dict.items():
-            if w_obj not in state.py_objects_w2r:
-                state.lifeline_dict.set(w_obj, None)
-            del obj
-        import gc; gc.collect()
-
-
-        for w_obj in state.non_heaptypes_w:
-            Py_DecRef(space, w_obj)
-        state.non_heaptypes_w[:] = []
-        state.reset_borrowed_references()
-
-    def check_and_print_leaks(self):
-        rawrefcount._collect()
-        # check for sane refcnts
-        import gc
-
-        if 1:  #ZZZ  not self.enable_leak_checking:
-            leakfinder.stop_tracking_allocations(check=False)
-            return False
-
-        leaking = False
-        state = self.space.fromcache(RefcountState)
-        gc.collect()
-        lost_objects_w = identity_dict()
-        lost_objects_w.update((key, None) for key in self.frozen_refcounts.keys())
-
-        for w_obj, obj in state.py_objects_w2r.iteritems():
-            base_refcnt = self.frozen_refcounts.get(w_obj)
-            delta = obj.c_ob_refcnt
-            if base_refcnt is not None:
-                delta -= base_refcnt
-                lost_objects_w.pop(w_obj)
-            if delta != 0:
-                leaking = True
-                print >>sys.stderr, "Leaking %r: %i references" % (w_obj, delta)
-                try:
-                    weakref.ref(w_obj)
-                except TypeError:
-                    lifeline = None
-                else:
-                    lifeline = state.lifeline_dict.get(w_obj)
-                if lifeline is not None:
-                    refcnt = lifeline.pyo.c_ob_refcnt
-                    if refcnt > 0:
-                        print >>sys.stderr, "\tThe object also held by C code."
-                    else:
-                        referrers_repr = []
-                        for o in gc.get_referrers(w_obj):
-                            try:
-                                repr_str = repr(o)
-                            except TypeError as e:
-                                repr_str = "%s (type of o is %s)" % (str(e), type(o))
-                            referrers_repr.append(repr_str)
-                        referrers = ", ".join(referrers_repr)
-                        print >>sys.stderr, "\tThe object is referenced by these objects:", \
-                                referrers
-        for w_obj in lost_objects_w:
-            print >>sys.stderr, "Lost object %r" % (w_obj, )
-            leaking = True
-        # the actual low-level leak checking is done by pypy.tool.leakfinder,
-        # enabled automatically by pypy.conftest.
-        return leaking
 
 class AppTestApi(LeakCheckingTest):
     def setup_class(cls):
@@ -179,15 +103,7 @@
     def teardown_method(self, meth):
         if self.runappdirect:
             return
-        self.space.getexecutioncontext().cleanup_cpyext_state()
-        self.cleanup_references(self.space)
-        # XXX: like AppTestCpythonExtensionBase.teardown_method:
-        # find out how to disable check_and_print_leaks() if the
-        # test failed
-        assert not self.check_and_print_leaks(), (
-            "Test leaks or loses object(s).  You should also check if "
-            "the test actually passed in the first place; if it failed "
-            "it is likely to reach this place.")
+        self.cleanup()
 
     @pytest.mark.skipif(only_pypy, reason='pypy only test')
     def test_only_import(self):
@@ -355,7 +271,6 @@
         self.space.call_method(self.space.sys.get("stdout"), "flush")
 
         freeze_refcnts(self)
-        #self.check_and_print_leaks()
 
     def unimport_module(self, name):
         """
@@ -367,17 +282,12 @@
 
     def teardown_method(self, func):
         if self.runappdirect:
+            self.w_debug_collect()
             return
+        debug_collect(self.space)
         for name in self.imported_module_names:
             self.unimport_module(name)
-        self.space.getexecutioncontext().cleanup_cpyext_state()
-        self.cleanup_references(self.space)
-        # XXX: find out how to disable check_and_print_leaks() if the
-        # test failed...
-        assert not self.check_and_print_leaks(), (
-            "Test leaks or loses object(s).  You should also check if "
-            "the test actually passed in the first place; if it failed "
-            "it is likely to reach this place.")
+        self.cleanup()
 
 
 class AppTestCpythonExtension(AppTestCpythonExtensionBase):
@@ -415,7 +325,6 @@
 
 
     def test_export_docstring(self):
-        import sys
         init = """
         if (Py_IsInitialized())
             Py_InitModule("foo", methods);
@@ -534,7 +443,6 @@
 
 
     def test_export_function2(self):
-        import sys
         init = """
         if (Py_IsInitialized())
             Py_InitModule("foo", methods);
diff --git a/pypy/module/gc/referents.py b/pypy/module/gc/referents.py
--- a/pypy/module/gc/referents.py
+++ b/pypy/module/gc/referents.py
@@ -47,57 +47,6 @@
 
 # ____________________________________________________________
 
-class PathEntry(object):
-    # PathEntries are nodes of a complete tree of all objects, but
-    # built lazily (there is only one branch alive at any time).
-    # Each node has a 'gcref' and the list of referents from this gcref.
-    def __init__(self, prev, gcref, referents):
-        self.prev = prev
-        self.gcref = gcref
-        self.referents = referents
-        self.remaining = len(referents)
-
-    def get_most_recent_w_obj(self):
-        entry = self
-        while entry is not None:
-            if entry.gcref:
-                w_obj = try_cast_gcref_to_w_root(entry.gcref)
-                if w_obj is not None:
-                    return w_obj
-            entry = entry.prev
-        return None
-
-def do_get_referrers(w_arg):
-    result_w = []
-    gcarg = rgc.cast_instance_to_gcref(w_arg)
-    roots = [gcref for gcref in rgc.get_rpy_roots() if gcref]
-    head = PathEntry(None, rgc.NULL_GCREF, roots)
-    while True:
-        head.remaining -= 1
-        if head.remaining >= 0:
-            gcref = head.referents[head.remaining]
-            if not rgc.get_gcflag_extra(gcref):
-                # not visited so far
-                if gcref == gcarg:
-                    w_obj = head.get_most_recent_w_obj()
-                    if w_obj is not None:
-                        result_w.append(w_obj)   # found!
-                        rgc.toggle_gcflag_extra(gcref)  # toggle twice
-                rgc.toggle_gcflag_extra(gcref)
-                head = PathEntry(head, gcref, rgc.get_rpy_referents(gcref))
-        else:
-            # no more referents to visit
-            head = head.prev
-            if head is None:
-                break
-    # done.  Clear flags carefully
-    rgc.toggle_gcflag_extra(gcarg)
-    rgc.clear_gcflag_extra(roots)
-    rgc.clear_gcflag_extra([gcarg])
-    return result_w
-
-# ____________________________________________________________
-
 def _list_w_obj_referents(gcref, result_w):
     # Get all W_Root reachable directly from gcref, and add them to
     # the list 'result_w'.
@@ -184,9 +133,22 @@
     """Return the list of objects that directly refer to any of objs."""
     if not rgc.has_gcflag_extra():
         raise missing_operation(space)
+    # xxx uses a lot of memory to make the list of all W_Root objects,
+    # but it's simpler this way and more correct than the previous
+    # version of this code (issue #2612).  It is potentially very slow
+    # because each of the n calls to _list_w_obj_referents() could take
+    # O(n) time as well, in theory, but I hope in practice the whole
+    # thing takes much less than O(n^2).  We could re-add an algorithm
+    # that visits most objects only once, if needed...
+    all_objects_w = rgc.do_get_objects(try_cast_gcref_to_w_root)
     result_w = []
-    for w_arg in args_w:
-        result_w += do_get_referrers(w_arg)
+    for w_obj in all_objects_w:
+        refs_w = []
+        gcref = rgc.cast_instance_to_gcref(w_obj)
+        _list_w_obj_referents(gcref, refs_w)
+        for w_arg in args_w:
+            if w_arg in refs_w:
+                result_w.append(w_obj)
     rgc.assert_no_more_gcflags()
     return space.newlist(result_w)
 
diff --git a/pypy/module/gc/test/test_referents.py b/pypy/module/gc/test/test_referents.py
--- a/pypy/module/gc/test/test_referents.py
+++ b/pypy/module/gc/test/test_referents.py
@@ -116,3 +116,37 @@
                 break   # found
         else:
             assert 0, "the tuple (7,) is not found as gc.get_referrers(7)"
+
+
+class AppTestReferentsMore(object):
+
+    def setup_class(cls):
+        from rpython.rlib import rgc
+        cls._backup = [rgc.get_rpy_roots]
+        l4 = cls.space.newlist([])
+        cls.ALL_ROOTS = [l4]
+        cls.w_ALL_ROOTS = cls.space.newlist(cls.ALL_ROOTS)
+        rgc.get_rpy_roots = lambda: (
+            map(rgc._GcRef, cls.ALL_ROOTS) + [rgc.NULL_GCREF]*2)
+        cls.w_runappdirect = cls.space.wrap(option.runappdirect)
+
+    def teardown_class(cls):
+        from rpython.rlib import rgc
+        rgc.get_rpy_roots = cls._backup[0]
+
+    def test_get_referrers(self):
+        import gc
+        class A(object):
+            pass
+        a = A()
+        if not self.runappdirect:
+            l4 = self.ALL_ROOTS[0]
+            l4.append(a)              # add 'a' to the list which is in roots
+        lst = gc.get_referrers(A)
+        assert a in lst
+        lst = gc.get_referrers(A)
+        assert a in lst
+        lst = gc.get_referrers(A)
+        assert a in lst
+        lst = gc.get_referrers(A)
+        assert a in lst
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
@@ -1450,20 +1450,30 @@
         py.test.skip("_Bool not in MSVC")
     ffi = FFI()
     ffi.cdef("struct foo_s { _Bool x; };"
-             "_Bool foo(_Bool);")
+             "_Bool foo(_Bool); _Bool (*foop)(_Bool);")
     lib = ffi.verify("""
         struct foo_s { _Bool x; };
         int foo(int arg) {
             return !arg;
         }
+        _Bool _foofunc(_Bool x) {
+            return !x;
+        }
+        _Bool (*foop)(_Bool) = _foofunc;
     """)
     p = ffi.new("struct foo_s *")
     p.x = 1
-    assert p.x == 1
+    assert p.x is True
     py.test.raises(OverflowError, "p.x = -1")
     py.test.raises(TypeError, "p.x = 0.0")
-    assert lib.foo(1) == 0
-    assert lib.foo(0) == 1
+    assert lib.foop(1) is False
+    assert lib.foop(True) is False
+    assert lib.foop(0) is True
+    py.test.raises(OverflowError, lib.foop, 42)
+    py.test.raises(TypeError, lib.foop, 0.0)
+    assert lib.foo(1) is False
+    assert lib.foo(True) is False
+    assert lib.foo(0) is True
     py.test.raises(OverflowError, lib.foo, 42)
     py.test.raises(TypeError, lib.foo, 0.0)
     assert int(ffi.cast("_Bool", long(1))) == 1
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_recompiler.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_recompiler.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_recompiler.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_recompiler.py
@@ -1939,7 +1939,7 @@
     ffi = FFI()
     ffi.cdef("bool f(void);")
     lib = verify(ffi, "test_bool_in_cpp", "char f(void) { return 2; }")
-    assert lib.f() == 1
+    assert lib.f() is True
 
 def test_bool_in_cpp_2():
     ffi = FFI()
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
@@ -1419,20 +1419,30 @@
         py.test.skip("_Bool not in MSVC")
     ffi = FFI()
     ffi.cdef("struct foo_s { _Bool x; };"
-             "_Bool foo(_Bool);")
+             "_Bool foo(_Bool); _Bool (*foop)(_Bool);")
     lib = ffi.verify("""
         struct foo_s { _Bool x; };
         int foo(int arg) {
             return !arg;
         }
+        _Bool _foofunc(_Bool x) {
+            return !x;
+        }
+        _Bool (*foop)(_Bool) = _foofunc;
     """)
     p = ffi.new("struct foo_s *")
     p.x = 1
-    assert p.x == 1
+    assert p.x is True
     py.test.raises(OverflowError, "p.x = -1")
     py.test.raises(TypeError, "p.x = 0.0")
-    assert lib.foo(1) == 0
-    assert lib.foo(0) == 1
+    assert lib.foop(1) is False
+    assert lib.foop(True) is False
+    assert lib.foop(0) is True
+    py.test.raises(OverflowError, lib.foop, 42)
+    py.test.raises(TypeError, lib.foop, 0.0)
+    assert lib.foo(1) is False
+    assert lib.foo(True) is False
+    assert lib.foo(0) is True
     py.test.raises(OverflowError, lib.foo, 42)
     py.test.raises(TypeError, lib.foo, 0.0)
     assert int(ffi.cast("_Bool", long(1))) == 1
diff --git a/pypy/objspace/std/bytearrayobject.py b/pypy/objspace/std/bytearrayobject.py
--- a/pypy/objspace/std/bytearrayobject.py
+++ b/pypy/objspace/std/bytearrayobject.py
@@ -567,13 +567,38 @@
             raise
     else:
         return list(buf.as_str())
+    return _from_byte_sequence(space, w_source)
 
-    # sequence of bytes
+def _get_printable_location(w_type):
+    return ('bytearray_from_byte_sequence [w_type=%s]' %
+            w_type.getname(w_type.space))
+
+_byteseq_jitdriver = jit.JitDriver(
+    name='bytearray_from_byte_sequence',
+    greens=['w_type'],
+    reds=['w_iter', 'data'],
+    get_printable_location=_get_printable_location)
+
+def _from_byte_sequence(space, w_source):
+    # Split off in a separate function for the JIT's benefit
+    # and add a jitdriver with the type of w_iter as the green key
     w_iter = space.iter(w_source)
     length_hint = space.length_hint(w_source, 0)
     data = newlist_hint(length_hint)
-    extended = 0
+    #
+    _from_byte_sequence_loop(space, w_iter, data)
+    #
+    extended = len(data)
+    if extended < length_hint:
+        resizelist_hint(data, extended)
+    return data
+
+def _from_byte_sequence_loop(space, w_iter, data):
+    w_type = space.type(w_iter)
     while True:
+        _byteseq_jitdriver.jit_merge_point(w_type=w_type,
+                                           w_iter=w_iter,
+                                           data=data)
         try:
             w_item = space.next(w_iter)
         except OperationError as e:
@@ -581,10 +606,6 @@
                 raise
             break
         data.append(space.byte_w(w_item))
-        extended += 1
-    if extended < length_hint:
-        resizelist_hint(data, extended)
-    return data
 
 
 def _hex_digit_to_int(d):
diff --git a/pypy/objspace/std/test/test_bytearrayobject.py b/pypy/objspace/std/test/test_bytearrayobject.py
--- a/pypy/objspace/std/test/test_bytearrayobject.py
+++ b/pypy/objspace/std/test/test_bytearrayobject.py
@@ -448,6 +448,13 @@
         raises(TypeError, b.extend, [object()])
         raises(TypeError, b.extend, u"unicode")
 
+    def test_extend_calls_len_or_lengthhint(self):
+        class BadLen(object):
+            def __iter__(self): return iter(range(10))
+            def __len__(self): raise RuntimeError('hello')
+        b = bytearray()
+        raises(RuntimeError, b.extend, BadLen())
+
     def test_setitem_from_front(self):
         b = bytearray(b'abcdefghij')
         b[:2] = b''
diff --git a/pypy/objspace/std/util.py b/pypy/objspace/std/util.py
--- a/pypy/objspace/std/util.py
+++ b/pypy/objspace/std/util.py
@@ -8,7 +8,7 @@
 IDTAG_LONG    = 3
 IDTAG_FLOAT   = 5
 IDTAG_COMPLEX = 7
-IDTAG_METHOD  = 9
+IDTAG_UNBOUND_METHOD = 9
 IDTAG_SPECIAL = 11    # -1 - (-maxunicode-1): unichar
                       # 0 - 255: char
                       # 256: empty string
diff --git a/rpython/config/support.py b/rpython/config/support.py
--- a/rpython/config/support.py
+++ b/rpython/config/support.py
@@ -35,3 +35,15 @@
         return int(count)
     except (OSError, ValueError):
         return 1
+
+def detect_pax():
+    """
+    Function to determine if your system comes with PAX protection.
+    """
+    if sys.platform.startswith('linux'):
+        # we need a running process PID and 1 is always running
+        with open("/proc/1/status") as fd:
+            data = fd.read()
+        if 'PaX' in data:
+            return True
+    return False
diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py
--- a/rpython/memory/gc/incminimark.py
+++ b/rpython/memory/gc/incminimark.py
@@ -2308,6 +2308,7 @@
                 ll_assert(not (self.probably_young_objects_with_finalizers
                                .non_empty()),
                     "probably_young_objects_with_finalizers should be empty")
+                self.kept_alive_by_finalizer = r_uint(0)
                 if self.old_objects_with_finalizers.non_empty():
                     self.deal_with_objects_with_finalizers()
                 elif self.old_objects_with_weakrefs.non_empty():
@@ -2380,6 +2381,9 @@
                 # we currently have -- but no more than 'max_delta' more than
                 # we currently have.
                 total_memory_used = float(self.get_total_memory_used())
+                total_memory_used -= float(self.kept_alive_by_finalizer)
+                if total_memory_used < 0:
+                    total_memory_used = 0
                 bounded = self.set_major_threshold_from(
                     min(total_memory_used * self.major_collection_threshold,
                         total_memory_used + self.max_delta),
@@ -2418,7 +2422,7 @@
             self.execute_finalizers()
             #END FINALIZING
         else:
-            pass #XXX which exception to raise here. Should be unreachable.
+            ll_assert(False, "bogus gc_state")
 
         debug_print("stopping, now in gc state: ", GC_STATES[self.gc_state])
         debug_stop("gc-collect-step")
@@ -2784,8 +2788,17 @@
     def _bump_finalization_state_from_0_to_1(self, obj):
         ll_assert(self._finalization_state(obj) == 0,
                   "unexpected finalization state != 0")
+        size_gc_header = self.gcheaderbuilder.size_gc_header
+        totalsize = size_gc_header + self.get_size(obj)
         hdr = self.header(obj)
         hdr.tid |= GCFLAG_FINALIZATION_ORDERING
+        # A bit hackish, but we will not count these objects as "alive"
+        # for the purpose of computing when the next major GC should
+        # occur.  This is done for issue #2590: without this, if we
+        # allocate mostly objects with finalizers, the
+        # next_major_collection_threshold grows forever and actual
+        # memory usage is not bounded.
+        self.kept_alive_by_finalizer += raw_malloc_usage(totalsize)
 
     def _recursively_bump_finalization_state_from_2_to_3(self, obj):
         ll_assert(self._finalization_state(obj) == 2,
diff --git a/rpython/memory/gc/minimark.py b/rpython/memory/gc/minimark.py
--- a/rpython/memory/gc/minimark.py
+++ b/rpython/memory/gc/minimark.py
@@ -1636,6 +1636,7 @@
         # with a finalizer and all objects reachable from there (and also
         # moves some objects from 'objects_with_finalizers' to
         # 'run_finalizers').
+        self.kept_alive_by_finalizer = r_uint(0)
         if self.old_objects_with_finalizers.non_empty():
             self.deal_with_objects_with_finalizers()
         #
@@ -1678,6 +1679,9 @@
         # we currently have -- but no more than 'max_delta' more than
         # we currently have.
         total_memory_used = float(self.get_total_memory_used())
+        total_memory_used -= float(self.kept_alive_by_finalizer)
+        if total_memory_used < 0:
+            total_memory_used = 0
         bounded = self.set_major_threshold_from(
             min(total_memory_used * self.major_collection_threshold,
                 total_memory_used + self.max_delta),
@@ -1999,8 +2003,11 @@
     def _bump_finalization_state_from_0_to_1(self, obj):
         ll_assert(self._finalization_state(obj) == 0,
                   "unexpected finalization state != 0")
+        size_gc_header = self.gcheaderbuilder.size_gc_header
+        totalsize = size_gc_header + self.get_size(obj)
         hdr = self.header(obj)
         hdr.tid |= GCFLAG_FINALIZATION_ORDERING
+        self.kept_alive_by_finalizer += raw_malloc_usage(totalsize)
 
     def _recursively_bump_finalization_state_from_2_to_3(self, obj):
         ll_assert(self._finalization_state(obj) == 2,
diff --git a/rpython/memory/test/test_minimark_gc.py b/rpython/memory/test/test_minimark_gc.py
--- a/rpython/memory/test/test_minimark_gc.py
+++ b/rpython/memory/test/test_minimark_gc.py
@@ -1,3 +1,4 @@
+from rpython.rlib import rgc
 from rpython.rlib.rarithmetic import LONG_BIT
 
 from rpython.memory.test import test_semispace_gc
@@ -9,3 +10,39 @@
     GC_CAN_SHRINK_BIG_ARRAY = False
     GC_CAN_MALLOC_NONMOVABLE = True
     BUT_HOW_BIG_IS_A_BIG_STRING = 11*WORD
+
+    def test_bounded_memory_when_allocating_with_finalizers(self):
+        # Issue #2590: when allocating a lot of objects with a finalizer
+        # and little else, the bounds in the (inc)minimark GC are not
+        # set up reasonably and the total memory usage grows without
+        # limit.
+        class B(object):
+            pass
+        b = B()
+        b.num_deleted = 0
+        class A(object):
+            def __init__(self):
+                fq.register_finalizer(self)
+        class FQ(rgc.FinalizerQueue):
+            Class = A
+            def finalizer_trigger(self):
+                while True:
+                    a = self.next_dead()
+                    if a is None:
+                        break
+                    b.num_deleted += 1
+        fq = FQ()
+        def f(x, y):
+            i = 0
+            alive_max = 0
+            while i < x:
+                i += 1
+                a = A()
+                a.x = a.y = a.z = i
+                #print i - b.num_deleted, b.num_deleted
+                alive = i - b.num_deleted
+                assert alive >= 0
+                alive_max = max(alive_max, alive)
+            return alive_max
+        res = self.interpret(f, [1000, 0])
+        assert res < 100
diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py
--- a/rpython/rlib/_rsocket_rffi.py
+++ b/rpython/rlib/_rsocket_rffi.py
@@ -20,7 +20,7 @@
     includes = ('sys/types.h',
                 'sys/socket.h',
                 'sys/un.h',
-                'sys/poll.h',
+                'poll.h',
                 'sys/select.h',
                 'sys/types.h',
                 'netinet/in.h',
diff --git a/rpython/rlib/rsre/rpy/_sre.py b/rpython/rlib/rsre/rpy/_sre.py
--- a/rpython/rlib/rsre/rpy/_sre.py
+++ b/rpython/rlib/rsre/rpy/_sre.py
@@ -16,6 +16,8 @@
 
 
 def get_code(regexp, flags=0, allargs=False):
+    """NOT_RPYTHON: you can't compile new regexps in an RPython program,
+    you can only use precompiled ones"""
     from . import sre_compile
     try:
         sre_compile.compile(regexp, flags)
diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py
--- a/rpython/rlib/runicode.py
+++ b/rpython/rlib/runicode.py
@@ -16,6 +16,8 @@
     allow_surrogate_by_default = True
 
 BYTEORDER = sys.byteorder
+BYTEORDER2 = BYTEORDER[0] + 'e'      # either "le" or "be"
+assert BYTEORDER2 in ('le', 'be')
 
 # python 2.7 has a preview of py3k behavior, so those functions
 # are used either when we're testing wide pypy on narrow cpython
@@ -486,9 +488,31 @@
                                                          errorhandler, "little")
     return result, length
 
+def py3k_str_decode_utf_16(s, size, errors, final=True,
+                      errorhandler=None):
+    result, length, byteorder = str_decode_utf_16_helper(s, size, errors, final,
+                                                         errorhandler, "native",
+                                                         'utf-16-' + BYTEORDER2)
+    return result, length
+
+def py3k_str_decode_utf_16_be(s, size, errors, final=True,
+                         errorhandler=None):
+    result, length, byteorder = str_decode_utf_16_helper(s, size, errors, final,
+                                                         errorhandler, "big",
+                                                         'utf-16-be')
+    return result, length
+
+def py3k_str_decode_utf_16_le(s, size, errors, final=True,
+                         errorhandler=None):
+    result, length, byteorder = str_decode_utf_16_helper(s, size, errors, final,
+                                                         errorhandler, "little",
+                                                         'utf-16-le')
+    return result, length
+
 def str_decode_utf_16_helper(s, size, errors, final=True,
                              errorhandler=None,
-                             byteorder="native"):
+                             byteorder="native",
+                             public_encoding_name='utf16'):
     if errorhandler is None:
         errorhandler = default_unicode_error_decode
     bo = 0
@@ -546,7 +570,8 @@
         if len(s) - pos < 2:
             if not final:
                 break
-            r, pos = errorhandler(errors, 'utf16', "truncated data",
+            r, pos = errorhandler(errors, public_encoding_name,
+                                  "truncated data",
                                   s, pos, len(s))
             result.append(r)
             if len(s) - pos < 2:
@@ -562,7 +587,8 @@
             if not final:
                 break
             errmsg = "unexpected end of data"
-            r, pos = errorhandler(errors, 'utf16', errmsg, s, pos, len(s))
+            r, pos = errorhandler(errors, public_encoding_name,
+                                  errmsg, s, pos, len(s))
             result.append(r)
             if len(s) - pos < 2:
                 break
@@ -578,12 +604,12 @@
                                            (ch2 & 0x3FF)) + 0x10000))
                 continue
             else:
-                r, pos = errorhandler(errors, 'utf16',
+                r, pos = errorhandler(errors, public_encoding_name,
                                       "illegal UTF-16 surrogate",
                                       s, pos - 4, pos - 2)
                 result.append(r)
         else:
-            r, pos = errorhandler(errors, 'utf16',
+            r, pos = errorhandler(errors, public_encoding_name,
                                   "illegal encoding",
                                   s, pos - 2, pos)
             result.append(r)
@@ -592,7 +618,8 @@
 def unicode_encode_utf_16_helper(s, size, errors,
                                  errorhandler=None,
                                  allow_surrogates=True,
-                                 byteorder='little'):
+                                 byteorder='little',
+                                 public_encoding_name='utf16'):
     if errorhandler is None:
         errorhandler = default_unicode_error_encode
     if size == 0:
@@ -620,13 +647,13 @@
         elif ch >= 0xE000 or allow_surrogates:
             _STORECHAR(result, ch, byteorder)
         else:
-            ru, rs, pos = errorhandler(errors, 'utf16',
+            ru, rs, pos = errorhandler(errors, public_encoding_name,
                                        'surrogates not allowed',
                                        s, pos-1, pos)
             if rs is not None:
                 # py3k only
                 if len(rs) % 2 != 0:
-                    errorhandler('strict', 'utf16',
+                    errorhandler('strict', public_encoding_name,
                                  'surrogates not allowed',
                                  s, pos-1, pos)
                 result.append(rs)
@@ -635,7 +662,7 @@
                 if ord(ch) < 0xD800:
                     _STORECHAR(result, ord(ch), byteorder)
                 else:
-                    errorhandler('strict', 'utf16',
+                    errorhandler('strict', public_encoding_name,
                                  'surrogates not allowed',
                                  s, pos-1, pos)
             continue
@@ -648,20 +675,39 @@
     return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
                                         allow_surrogates, "native")
 
-
 def unicode_encode_utf_16_be(s, size, errors,
                              errorhandler=None,
                              allow_surrogates=True):
     return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
                                         allow_surrogates, "big")
 
-
 def unicode_encode_utf_16_le(s, size, errors,
                              errorhandler=None,
                              allow_surrogates=True):
     return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
                                         allow_surrogates, "little")
 
+def py3k_unicode_encode_utf_16(s, size, errors,
+                          errorhandler=None,
+                          allow_surrogates=True):
+    return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
+                                        allow_surrogates, "native",
+                                        'utf-16-' + BYTEORDER2)
+
+def py3k_unicode_encode_utf_16_be(s, size, errors,
+                             errorhandler=None,
+                             allow_surrogates=True):
+    return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
+                                        allow_surrogates, "big",
+                                        'utf-16-be')
+
+def py3k_unicode_encode_utf_16_le(s, size, errors,
+                             errorhandler=None,
+                             allow_surrogates=True):
+    return unicode_encode_utf_16_helper(s, size, errors, errorhandler,
+                                        allow_surrogates, "little",
+                                        'utf-16-le')
+
 
 # ____________________________________________________________
 # utf-32
@@ -684,12 +730,34 @@
                                                          errorhandler, "little")
     return result, length
 
+def py3k_str_decode_utf_32(s, size, errors, final=True,
+                           errorhandler=None):
+    result, length, byteorder = str_decode_utf_32_helper(s, size, errors, final,
+                                                         errorhandler, "native",
+                                                         'utf-32-' + BYTEORDER2)
+    return result, length
+
+def py3k_str_decode_utf_32_be(s, size, errors, final=True,
+                              errorhandler=None):
+    result, length, byteorder = str_decode_utf_32_helper(s, size, errors, final,
+                                                         errorhandler, "big",
+                                                         'utf-32-be')
+    return result, length
+
+def py3k_str_decode_utf_32_le(s, size, errors, final=True,
+                              errorhandler=None):
+    result, length, byteorder = str_decode_utf_32_helper(s, size, errors, final,
+                                                         errorhandler, "little",
+                                                         'utf-32-le')
+    return result, length
+
 BOM32_DIRECT  = intmask(0x0000FEFF)
 BOM32_REVERSE = intmask(0xFFFE0000)
 
 def str_decode_utf_32_helper(s, size, errors, final=True,

From pypy.commits at gmail.com  Thu Aug  3 20:16:20 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:20 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove test_aclassloader and
 test_streams as these are backend tests
Message-ID: <5983bcd4.c4b3df0a.7caab.314c@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92057:8a2ade4d76bf
Date: 2017-08-03 13:23 -0700
http://bitbucket.org/pypy/pypy/changeset/8a2ade4d76bf/

Log:	remove test_aclassloader and test_streams as these are backend tests

diff --git a/pypy/module/_cppyy/test/test_aclassloader.py b/pypy/module/_cppyy/test/test_aclassloader.py
deleted file mode 100644
--- a/pypy/module/_cppyy/test/test_aclassloader.py
+++ /dev/null
@@ -1,28 +0,0 @@
-import py, os, sys
-
-
-currpath = py.path.local(__file__).dirpath()
-
-def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make example01Dict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
-
-
-class AppTestACLASSLOADER:
-    spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
-
-    def setup_class(cls):
-        cls.space.appexec([], """():
-            import _cppyy""")
-
-    def test01_class_autoloading(self):
-        """Test whether a class can be found through .rootmap."""
-        import _cppyy
-        example01_class = _cppyy.gbl.example01
-        assert example01_class
-        cl2 = _cppyy.gbl.example01
-        assert cl2
-        assert example01_class is cl2
diff --git a/pypy/module/_cppyy/test/test_streams.py b/pypy/module/_cppyy/test/test_streams.py
deleted file mode 100644
--- a/pypy/module/_cppyy/test/test_streams.py
+++ /dev/null
@@ -1,38 +0,0 @@
-import py, os, sys
-
-
-currpath = py.path.local(__file__).dirpath()
-test_dct = str(currpath.join("std_streamsDict.so"))
-
-def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make std_streamsDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
-
-class AppTestSTDStreams:
-    spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
-
-    def setup_class(cls):
-        cls.w_test_dct  = cls.space.newtext(test_dct)
-        cls.w_streams = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
-
-    def test01_std_ostream(self):
-        """Test availability of std::ostream"""
-
-        import _cppyy
-
-        assert _cppyy.gbl.std is _cppyy.gbl.std
-        assert _cppyy.gbl.std.ostream is _cppyy.gbl.std.ostream
-
-        assert callable(_cppyy.gbl.std.ostream)
-
-    def test02_std_cout(self):
-        """Test access to std::cout"""
-
-        import _cppyy
-
-        assert not (_cppyy.gbl.std.cout is None)

From pypy.commits at gmail.com  Thu Aug  3 20:16:22 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:22 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove streams test support
 files
Message-ID: <5983bcd6.97a0df0a.b1ca5.abc5@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92058:110184c3d9d9
Date: 2017-08-03 13:24 -0700
http://bitbucket.org/pypy/pypy/changeset/110184c3d9d9/

Log:	remove streams test support files

diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile
--- a/pypy/module/_cppyy/test/Makefile
+++ b/pypy/module/_cppyy/test/Makefile
@@ -1,6 +1,6 @@
 dicts = example01Dict.so datatypesDict.so advancedcppDict.so advancedcpp2Dict.so \
 overloadsDict.so stltypesDict.so operatorsDict.so fragileDict.so crossingDict.so \
-std_streamsDict.so iotypesDict.so
+iotypesDict.so
 all : $(dicts)
 
 ifneq (${REFLEXHOME},)
@@ -62,12 +62,6 @@
 
 endif
 
-ifeq ($(DUMMY),)
-# TODO: methptrgetter causes these tests to crash, so don't use it for now
-std_streamsDict.so: std_streams.cxx std_streams.h std_streams.xml
-	$(genreflex) std_streams.h --selection=std_streams.xml
-	g++ -o $@ std_streams_rflx.cpp std_streams.cxx -shared -std=c++11 $(cppflags) $(cppflags2)
-endif
 
 .PHONY: clean
 clean:
diff --git a/pypy/module/_cppyy/test/std_streams.cxx b/pypy/module/_cppyy/test/std_streams.cxx
deleted file mode 100644
--- a/pypy/module/_cppyy/test/std_streams.cxx
+++ /dev/null
@@ -1,3 +0,0 @@
-#include "std_streams.h"
-
-template class std::basic_ios<char,std::char_traits<char> >;
diff --git a/pypy/module/_cppyy/test/std_streams.h b/pypy/module/_cppyy/test/std_streams.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/std_streams.h
+++ /dev/null
@@ -1,13 +0,0 @@
-#ifndef STD_STREAMS_H 
-#define STD_STREAMS_H 1
-
-#ifndef __CINT__
-#include <ios>
-#endif
-#include <iostream>
-
-#ifndef __CINT__
-extern template class std::basic_ios<char,std::char_traits<char> >;
-#endif
-
-#endif // STD_STREAMS_H
diff --git a/pypy/module/_cppyy/test/std_streams.xml b/pypy/module/_cppyy/test/std_streams.xml
deleted file mode 100644
--- a/pypy/module/_cppyy/test/std_streams.xml
+++ /dev/null
@@ -1,12 +0,0 @@
-<lcgdict>
-
-  <namespace name = "std" />
-
-  <class name = "std::ostream"/>
-  <class name = "std::ios_base"/>
-  <class name = "std::basic_ios<char,std::char_traits<char> >"/>
-
-  <class name = "std::basic_ostream<char,char_traits<char> >" />
-  <variable name="std::cout" />
-
-</lcgdict>

From pypy.commits at gmail.com  Thu Aug  3 20:16:18 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:18 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: add back the argument passing
 union for dummy backend
Message-ID: <5983bcd2.14d81c0a.8a8fe.3d5b@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92056:d66a61b5e9f9
Date: 2017-08-03 12:18 -0700
http://bitbucket.org/pypy/pypy/changeset/d66a61b5e9f9/

Log:	add back the argument passing union for dummy backend

diff --git a/pypy/module/_cppyy/src/dummy_backend.cxx b/pypy/module/_cppyy/src/dummy_backend.cxx
--- a/pypy/module/_cppyy/src/dummy_backend.cxx
+++ b/pypy/module/_cppyy/src/dummy_backend.cxx
@@ -1,4 +1,3 @@
-#include "cppyy.h"
 #include "capi.h"
 
 #include <map>
@@ -14,6 +13,27 @@
 
 #pragma GCC diagnostic ignored "-Winvalid-offsetof"
 
+// union for argument passing
+struct CPPYY_G__value {
+  union {
+    double d;
+    long    i; /* used to be int */
+    char ch;
+    short sh;
+    int in;
+    float fl;
+    unsigned char uch;
+    unsigned short ush;
+    unsigned int uin;
+    unsigned long ulo;
+    long long ll;
+    unsigned long long ull;
+    long double ld;
+  } obj;
+  long ref;
+  int type;
+};
+
 // add example01.cxx code
 int globalAddOneToInt(int a);
 

From pypy.commits at gmail.com  Thu Aug  3 20:16:28 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:28 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove now superfluous LinkDef
 files (were for CINT; cling can use xml)
Message-ID: <5983bcdc.db6f1c0a.3caf2.57a8@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92061:ad8862eba7b7
Date: 2017-08-03 15:08 -0700
http://bitbucket.org/pypy/pypy/changeset/ad8862eba7b7/

Log:	remove now superfluous LinkDef files (were for CINT; cling can use
	xml)

diff --git a/pypy/module/_cppyy/test/advancedcpp2_LinkDef.h b/pypy/module/_cppyy/test/advancedcpp2_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/advancedcpp2_LinkDef.h
+++ /dev/null
@@ -1,18 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ namespace a_ns;
-#pragma link C++ namespace a_ns::d_ns;
-#pragma link C++ struct a_ns::g_class;
-#pragma link C++ struct a_ns::g_class::h_class;
-#pragma link C++ struct a_ns::d_ns::i_class;
-#pragma link C++ struct a_ns::d_ns::i_class::j_class;
-#pragma link C++ variable a_ns::g_g;
-#pragma link C++ function a_ns::get_g_g;
-#pragma link C++ variable a_ns::d_ns::g_i;
-#pragma link C++ function a_ns::d_ns::get_g_i;
-
-#endif
diff --git a/pypy/module/_cppyy/test/advancedcpp_LinkDef.h b/pypy/module/_cppyy/test/advancedcpp_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/advancedcpp_LinkDef.h
+++ /dev/null
@@ -1,84 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ class short_defaulter;
-#pragma link C++ class ushort_defaulter;
-#pragma link C++ class int_defaulter;
-#pragma link C++ class uint_defaulter;
-#pragma link C++ class long_defaulter;
-#pragma link C++ class ulong_defaulter;
-#pragma link C++ class llong_defaulter;
-#pragma link C++ class ullong_defaulter;
-#pragma link C++ class float_defaulter;
-#pragma link C++ class double_defaulter;
-
-#pragma link C++ class base_class;
-#pragma link C++ class derived_class;
-
-#pragma link C++ class a_class;
-#pragma link C++ class b_class;
-#pragma link C++ class c_class;
-#pragma link C++ class c_class_1;
-#pragma link C++ class c_class_2;
-#pragma link C++ class d_class;
-
-#pragma link C++ function create_c1();
-#pragma link C++ function create_c2();
-
-#pragma link C++ function get_a(a_class&);
-#pragma link C++ function get_b(b_class&);
-#pragma link C++ function get_c(c_class&);
-#pragma link C++ function get_d(d_class&);
-
-#pragma link C++ class T1<int>;
-#pragma link C++ class T2<T1<int> >;
-#pragma link C++ class T3<int, double>;
-#pragma link C++ class T3<T1<int>, T2<T1<int> > >;
-#pragma link C++ class a_ns::T4<int>;
-#pragma link C++ class a_ns::T4<T3<int,double> >;
-#pragma link C++ class a_ns::T4<a_ns::T4<T3<int, double> > >;
-
-#pragma link C++ namespace a_ns;
-#pragma link C++ namespace a_ns::d_ns;
-#pragma link C++ struct a_ns::b_class;
-#pragma link C++ struct a_ns::b_class::c_class;
-#pragma link C++ struct a_ns::d_ns::e_class;
-#pragma link C++ struct a_ns::d_ns::e_class::f_class;
-#pragma link C++ variable a_ns::g_a;
-#pragma link C++ function a_ns::get_g_a;
-#pragma link C++ variable a_ns::d_ns::g_d;
-#pragma link C++ function a_ns::d_ns::get_g_d;
-
-#pragma link C++ class some_abstract_class;
-#pragma link C++ class some_concrete_class;
-#pragma link C++ class some_convertible;
-#pragma link C++ class some_class_with_data;
-#pragma link C++ class some_class_with_data::some_data;
-
-#pragma link C++ class some_comparable;
-#pragma link C++ function operator==(const some_comparable&, const some_comparable&);
-#pragma link C++ function operator!=(const some_comparable&, const some_comparable&);
-
-#pragma link C++ class ref_tester;
-#pragma link C++ class std::vector<ref_tester>;
-#pragma link C++ class pointer_pass;
-
-#pragma link C++ class multi1;
-#pragma link C++ class multi2;
-#pragma link C++ class multi;
-
-#pragma link C++ class new_overloader;
-
-#pragma link C++ class my_templated_class<std::vector<float> >;
-#pragma link C++ function my_templated_function<char>(char);
-#pragma link C++ function my_templated_function<double>(double);
-#pragma link C++ class my_templated_method_class;
-#pragma link C++ typedef my_typedef_t;
-
-#pragma link C++ class overload_one_way;
-#pragma link C++ class overload_the_other_way;
-
-#endif
diff --git a/pypy/module/_cppyy/test/crossing_LinkDef.h b/pypy/module/_cppyy/test/crossing_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/crossing_LinkDef.h
+++ /dev/null
@@ -1,11 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ namespace crossing;
-
-#pragma link C++ class crossing::A;
-
-#endif
diff --git a/pypy/module/_cppyy/test/datatypes_LinkDef.h b/pypy/module/_cppyy/test/datatypes_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/datatypes_LinkDef.h
+++ /dev/null
@@ -1,28 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ struct cppyy_test_pod;
-#pragma link C++ class cppyy_test_data;
-#pragma link C++ class four_vector;
-
-#pragma link C++ enum fruit;
-
-#pragma link C++ function get_pod_address(cppyy_test_data&);
-#pragma link C++ function get_int_address(cppyy_test_data&);
-#pragma link C++ function get_double_address(cppyy_test_data&);
-#pragma link C++ function set_global_int(int);
-#pragma link C++ function get_global_int();
-
-#pragma link C++ function is_global_pod(cppyy_test_pod*);
-#pragma link C++ function set_global_pod(cppyy_test_pod*);
-#pragma link C++ function get_global_pod();
-#pragma link C++ function get_null_pod();
-
-#pragma link C++ global N;
-#pragma link C++ global g_int;
-#pragma link C++ global g_pod;
-
-#endif
diff --git a/pypy/module/_cppyy/test/example01_LinkDef.h b/pypy/module/_cppyy/test/example01_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/example01_LinkDef.h
+++ /dev/null
@@ -1,22 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ class example01;
-#pragma link C++ typedef example01_t;
-#pragma link C++ class example01a;
-#pragma link C++ class payload;
-#pragma link C++ class ArgPasser;
-#pragma link C++ class z_;
-
-#pragma link C++ function globalAddOneToInt(int);
-#pragma link C++ function installableAddOneToInt(example01&, int);
-
-#pragma link C++ namespace ns_example01;
-#pragma link C++ function ns_example01::globalAddOneToInt(int);
-
-#pragma link C++ variable ns_example01::gMyGlobalInt;
-
-#endif
diff --git a/pypy/module/_cppyy/test/fragile_LinkDef.h b/pypy/module/_cppyy/test/fragile_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/fragile_LinkDef.h
+++ /dev/null
@@ -1,35 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ namespace fragile;
-#pragma link C++ namespace fragile::nested1;
-#pragma link C++ namespace fragile::nested1::nested2;
-#pragma link C++ namespace fragile::nested1::nested2::nested3;
-
-#pragma link C++ class fragile::A;
-#pragma link C++ class fragile::B;
-#pragma link C++ class fragile::C;
-#pragma link C++ class fragile::D;
-#pragma link C++ class fragile::E;
-#pragma link C++ class fragile::F;
-#pragma link C++ class fragile::G;
-#pragma link C++ class fragile::H;
-#pragma link C++ class fragile::I;
-#pragma link C++ class fragile::J;
-#pragma link C++ class fragile::K;
-#pragma link C++ class fragile::L;
-#pragma link C++ class fragile::M;
-#pragma link C++ class fragile::N;
-#pragma link C++ class fragile::O;
-#pragma link C++ class fragile::nested1::A;
-#pragma link C++ class fragile::nested1::nested2::A;
-#pragma link C++ class fragile::nested1::nested2::nested3::A;
-
-#pragma link C++ variable fragile::gI;
-
-#pragma link C++ function fragile::fglobal;
-
-#endif
diff --git a/pypy/module/_cppyy/test/iotypes_LinkDef.h b/pypy/module/_cppyy/test/iotypes_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/iotypes_LinkDef.h
+++ /dev/null
@@ -1,16 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-using namespace std;
-#pragma link C++ class vector<vector<float> >+;
-#pragma link C++ class vector<vector<float> >::iterator;
-#pragma link C++ class vector<vector<float> >::const_iterator;
-
-#pragma link C++ namespace IO;
-#pragma link C++ class IO::SomeDataObject+;
-#pragma link C++ class IO::SomeDataStruct+;
-
-#endif
diff --git a/pypy/module/_cppyy/test/operators_LinkDef.h b/pypy/module/_cppyy/test/operators_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/operators_LinkDef.h
+++ /dev/null
@@ -1,22 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ class number;
-
-#pragma link C++ struct operator_char_star;
-#pragma link C++ struct operator_const_char_star;
-#pragma link C++ struct operator_int;
-#pragma link C++ struct operator_long;
-#pragma link C++ struct operator_double;
-#pragma link C++ struct operator_short;
-#pragma link C++ struct operator_unsigned_int;
-#pragma link C++ struct operator_unsigned_long;
-#pragma link C++ struct operator_float;
-
-#pragma link C++ class v_opeq_base;
-#pragma link C++ class v_opeq_derived;
-
-#endif
diff --git a/pypy/module/_cppyy/test/overloads_LinkDef.h b/pypy/module/_cppyy/test/overloads_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/overloads_LinkDef.h
+++ /dev/null
@@ -1,27 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ class a_overload;
-#pragma link C++ class b_overload;
-#pragma link C++ class c_overload;
-#pragma link C++ class d_overload;
-
-#pragma link C++ namespace ns_a_overload;
-#pragma link C++ class ns_a_overload::a_overload;
-#pragma link C++ class ns_a_overload::b_overload;
-
-#pragma link C++ class ns_b_overload;
-#pragma link C++ class ns_b_overload::a_overload;
-
-#pragma link C++ class aa_ol;
-#pragma link C++ class cc_ol;
-
-#pragma link C++ class more_overloads;
-#pragma link C++ class more_overloads2;
-
-#pragma link C++ function calc_mean;
-
-#endif
diff --git a/pypy/module/_cppyy/test/stltypes_LinkDef.h b/pypy/module/_cppyy/test/stltypes_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/stltypes_LinkDef.h
+++ /dev/null
@@ -1,25 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#pragma link C++ class std::vector<just_a_class>;
-#pragma link C++ class std::vector<just_a_class>::iterator;
-#pragma link C++ class std::vector<just_a_class>::const_iterator;
-
-#pragma link C++ class map<std::string, unsigned int>;
-#pragma link C++ class map<std::string, unsigned int>::iterator;
-#pragma link C++ class map<std::string, unsigned int>::const_iterator;
-#pragma link C++ class pair<std::string, unsigned int>;
-
-#pragma link C++ class map<std::string, unsigned long>;
-#pragma link C++ class map<std::string, unsigned long>::iterator;
-#pragma link C++ class map<std::string, unsigned long>::const_iterator;
-#pragma link C++ class pair<std::string, unsigned long>;
-
-#pragma link C++ class just_a_class;
-#pragma link C++ class stringy_class;
-#pragma link C++ class stl_like_class<int>;
-
-#endif

From pypy.commits at gmail.com  Thu Aug  3 20:16:24 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:24 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: simplify Makefile (expects
 backend for full tests) and remove obsolete options
Message-ID: <5983bcd8.05861c0a.ba361.473a@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92059:21fdf4274169
Date: 2017-08-03 14:04 -0700
http://bitbucket.org/pypy/pypy/changeset/21fdf4274169/

Log:	simplify Makefile (expects backend for full tests) and remove
	obsolete options

diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile
--- a/pypy/module/_cppyy/test/Makefile
+++ b/pypy/module/_cppyy/test/Makefile
@@ -1,26 +1,23 @@
-dicts = example01Dict.so datatypesDict.so advancedcppDict.so advancedcpp2Dict.so \
-overloadsDict.so stltypesDict.so operatorsDict.so fragileDict.so crossingDict.so \
-iotypesDict.so
+dicts = advancedcppDict.so \
+        advancedcpp2Dict.so \
+        crossingDict.so \
+        datatypesDict.so \
+        example01Dict.so \
+        fragileDict.so \
+        operatorsDict.so \
+        overloadsDict.so \
+        stltypesDict.so
+
 all : $(dicts)
 
-ifneq (${REFLEXHOME},)
-  ROOTSYS := ${REFLEXHOME}
-else ifneq (${ROOTSYS},)
-  ROOTSYS := ${ROOTSYS}
+HASGENREFLEX:=$(shell command -v genreflex 2> /dev/null)
+
+cppflags=-std=c++11 -O3 -m64 -fPIC -rdynamic
+ifdef HASGENREFLEX
+  genreflex_flags:=$(shell genreflex --cppflags)
+  cppflags+=$(genreflex_flags)
 else
-  DUMMY := t
-endif
-
-ifeq ($(DUMMY),t)
-  cppflags=
-else
-  ifeq ($(ROOTSYS),)
-    genreflex=genreflex
-    cppflags=-pthread -std=c++11 -m64 -I./include -L./lib64 -L./lib
-  else
-    genreflex=$(ROOTSYS)/bin/genreflex
-    cppflags=$(shell $(ROOTSYS)/bin/root-config --cflags) $(shell $(ROOTSYS)/bin/root-config --ldflags) -L$(shell $(ROOTSYS)/bin/root-config --libdir) -lCore
-  endif
+  cppflags+=-DCPPYY_DUMMY_BACKEND
 endif
 
 PLATFORM := $(shell uname -s)
@@ -28,41 +25,22 @@
   cppflags+=-dynamiclib -single_module -arch x86_64 -undefined dynamic_lookup
 endif
 
-ifeq ($(DUMMY),t)
-  cppflags2=-O3 -fPIC -rdynamic -std=c++11 -DCPPYY_DUMMY_BACKEND
-else ifeq ($(CLING),t)
-  cppflags2=-O3 -fPIC -rdynamic
-else
-  ifeq ($(shell $(genreflex) --help | grep -- --with-methptrgetter),)
-    genreflexflags=
-    cppflags2=-O3 -fPIC
-  else
-    genreflexflags=--with-methptrgetter
-    cppflags2=-Wno-pmf-conversions -O3 -fPIC
-  endif
-endif
 
-ifeq ($(CLING),t)
-%Dict.so: %.h %.cxx %_cling.cxx
-	g++ -o $@ $*.cxx $*_cling.cxx -shared $(cppflags) $(cppflags2)
-
-%_cling.cxx: %.h %_LinkDef.h
-	rootcling -f $@ -rml $*Dict.so -rmf $*Dict.rootmap -c $*.h $*_LinkDef.h
-
-else ifeq ($(DUMMY),t)
+ifndef HASREFLEX
 %Dict.so: %.cxx
-	g++ -o $@ $^ -shared $(cppflags) $(cppflags2)
+	$(CXX) -shared $(cppflags) -o $@ $^
 
 else # reflex
 %Dict.so: %_rflx.cpp %.cxx
-	g++ -o $@ $^ -shared $(cppflags) $(cppflags2)
+	$(CXX) -shared $(cppflags) -o $@ $^
 
 %_rflx.cpp: %.h %.xml
-	$(genreflex) $< $(genreflexflags) --selection=$*.xml --rootmap=$*Dict.rootmap --rootmap-lib=$*Dict.so
+	genreflex $< --selection=$*.xml --rootmap=$*Dict.rootmap --rootmap-lib=$*Dict.so
 
 endif
 
 
 .PHONY: clean
+
 clean:
-	-rm -f $(dicts) $(subst .so,.rootmap,$(dicts)) $(subst Dict.so,_rflx_rdict.pcm,$(dicts)) $(subst Dict.so,_rflx.cpp,$(dicts)) $(subst Dict.so,_cling.h,$(dicts)) $(subst Dict.so,_cling.cxx,$(dicts)) $(subst Dict.so,_cling_rdict.pcm,$(dicts)) $(wildcard *.pyc)
+	-rm -f $(dicts) $(subst .so,.rootmap,$(dicts)) $(subst Dict.so,_rflx_rdict.pcm,$(dicts)) $(subst Dict.so,_rflx.cpp,$(dicts)) $(wildcard *.pyc)

From pypy.commits at gmail.com  Thu Aug  3 20:16:30 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:30 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: destruct -> __destruct__ in
 pythonification
Message-ID: <5983bcde.8b841c0a.bd9b0.638d@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92062:319df9089e80
Date: 2017-08-03 15:19 -0700
http://bitbucket.org/pypy/pypy/changeset/319df9089e80/

Log:	destruct -> __destruct__ in pythonification

diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py
--- a/pypy/module/_cppyy/pythonify.py
+++ b/pypy/module/_cppyy/pythonify.py
@@ -369,7 +369,7 @@
                 while i != self.end():
                     yield i.__deref__()
                     i.__preinc__()
-                i.destruct()
+                i.__destruct__()
                 raise StopIteration
             pyclass.__iter__ = __iter__
         # else: rely on numbered iteration

From pypy.commits at gmail.com  Thu Aug  3 20:16:31 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:31 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: fix comment
Message-ID: <5983bcdf.01571c0a.14ca8.4c78@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92063:d2b2427a84ce
Date: 2017-08-03 16:09 -0700
http://bitbucket.org/pypy/pypy/changeset/d2b2427a84ce/

Log:	fix comment

diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py
--- a/pypy/module/_cppyy/__init__.py
+++ b/pypy/module/_cppyy/__init__.py
@@ -1,9 +1,8 @@
 from pypy.interpreter.mixedmodule import MixedModule
 
 class Module(MixedModule):
-    "This module provides runtime bindings to C++ code for which reflection\n\
-    info has been generated. Current supported back-ends are Reflex and CINT.\n\
-    See http://doc.pypy.org/en/latest/cppyy.html for full details."
+    "This module brigdes the cppyy frontend with its backend, through PyPy.\n\
+    See http://cppyy.readthedocs.io/en/latest for full details."
 
     interpleveldefs = {
         '_resolve_name'          : 'interp_cppyy.resolve_name',

From pypy.commits at gmail.com  Thu Aug  3 20:16:26 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:26 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove load_reflection_info
 (lives in frontend)
Message-ID: <5983bcda.2e9ddf0a.f7c95.6fd2@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92060:27a381b2e6b9
Date: 2017-08-03 15:08 -0700
http://bitbucket.org/pypy/pypy/changeset/27a381b2e6b9/

Log:	remove load_reflection_info (lives in frontend)

diff --git a/pypy/module/_cppyy/__init__.py b/pypy/module/_cppyy/__init__.py
--- a/pypy/module/_cppyy/__init__.py
+++ b/pypy/module/_cppyy/__init__.py
@@ -6,7 +6,6 @@
     See http://doc.pypy.org/en/latest/cppyy.html for full details."
 
     interpleveldefs = {
-        '_load_dictionary'       : 'interp_cppyy.load_dictionary',
         '_resolve_name'          : 'interp_cppyy.resolve_name',
         '_scope_byname'          : 'interp_cppyy.scope_byname',
         '_template_byname'       : 'interp_cppyy.template_byname',
@@ -22,7 +21,6 @@
 
     appleveldefs = {
         '_init_pythonify'        : 'pythonify._init_pythonify',
-        'load_reflection_info'   : 'pythonify.load_reflection_info',
         'add_pythonization'      : 'pythonify.add_pythonization',
         'Template'               : 'pythonify.CPPTemplate',
     }
diff --git a/pypy/module/_cppyy/capi/loadable_capi.py b/pypy/module/_cppyy/capi/loadable_capi.py
--- a/pypy/module/_cppyy/capi/loadable_capi.py
+++ b/pypy/module/_cppyy/capi/loadable_capi.py
@@ -320,9 +320,6 @@
     ptr = _cdata_to_ptr(space, w_cdata)      # see above ... something better?
     return rffi.cast(rffi.CCHARP, ptr)
 
-def c_load_dictionary(name):
-    return libffi.CDLL(name)
-
 # name to opaque C++ scope representation ------------------------------------
 def c_num_scopes(space, cppscope):
     return space.int_w(call_capi(space, 'num_scopes', [_ArgH(cppscope.handle)]))
diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py
--- a/pypy/module/_cppyy/interp_cppyy.py
+++ b/pypy/module/_cppyy/interp_cppyy.py
@@ -31,21 +31,6 @@
     def lt(self, a, b):
         return a.priority() < b.priority()
 
- at unwrap_spec(name='text')
-def load_dictionary(space, name):
-    try:
-        cdll = capi.c_load_dictionary(name)
-        if not cdll:
-           raise OperationError(space.w_RuntimeError, space.newtext(str("could not load dictionary " + name)))
-
-    except rdynload.DLOpenError as e:
-        if hasattr(space, "fake"):      # FakeSpace fails e.msg (?!)
-            errmsg = "failed to load cdll"
-        else:
-            errmsg = e.msg
-        raise OperationError(space.w_RuntimeError, space.newtext(str(errmsg)))
-    return W_CPPLibrary(space, cdll)
-
 class State(object):
     def __init__(self, space):
         self.cppscope_cache = {
diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py
--- a/pypy/module/_cppyy/pythonify.py
+++ b/pypy/module/_cppyy/pythonify.py
@@ -403,17 +403,6 @@
         pyclass.__getitem__ = getitem
         pyclass.__len__     = return2
 
-_loaded_dictionaries = {}
-def load_reflection_info(name):
-    """Takes the name of a library containing reflection info, returns a handle
-    to the loaded library."""
-    try:
-        return _loaded_dictionaries[name]
-    except KeyError:
-        import _cppyy
-        lib = _cppyy._load_dictionary(name)
-        _loaded_dictionaries[name] = lib
-        return lib
 
 def _init_pythonify():
     # _cppyy should not be loaded at the module level, as that will trigger a
diff --git a/pypy/module/_cppyy/test/Makefile b/pypy/module/_cppyy/test/Makefile
--- a/pypy/module/_cppyy/test/Makefile
+++ b/pypy/module/_cppyy/test/Makefile
@@ -26,7 +26,7 @@
 endif
 
 
-ifndef HASREFLEX
+ifndef HASGENREFLEX
 %Dict.so: %.cxx
 	$(CXX) -shared $(cppflags) -o $@ $^
 
diff --git a/pypy/module/_cppyy/test/conftest.py b/pypy/module/_cppyy/test/conftest.py
--- a/pypy/module/_cppyy/test/conftest.py
+++ b/pypy/module/_cppyy/test/conftest.py
@@ -14,10 +14,10 @@
                 py.test.skip("genreflex is not installed")
             import re
             if tst == 'test_pythonify.py' and \
-                not re.search("AppTestPYTHONIFY.test0[1-6]", item.location[2]):
+                not re.search("AppTestPYTHONIFY.test0[1-5]", item.location[2]):
                 py.test.skip("genreflex is not installed")
             elif tst == 'test_datatypes.py' and \
-                not re.search("AppTestDATATYPES.test0[1-8]", item.location[2]):
+                not re.search("AppTestDATATYPES.test0[1-7]", item.location[2]):
                 py.test.skip("genreflex is not installed")
 
 def pytest_ignore_collect(path, config):
diff --git a/pypy/module/_cppyy/test/datatypes.h b/pypy/module/_cppyy/test/datatypes.h
--- a/pypy/module/_cppyy/test/datatypes.h
+++ b/pypy/module/_cppyy/test/datatypes.h
@@ -1,6 +1,3 @@
-#ifndef CPPYY_DUMMY_BACKEND
-#include "RtypesCore.h"
-#else
 // copied from RtypesCore.h ...
 #if defined(R__WIN32) && !defined(__CINT__)
 typedef __int64          Long64_t;  //Portable signed long integer 8 bytes
@@ -9,7 +6,7 @@
 typedef long long          Long64_t; //Portable signed long integer 8 bytes
 typedef unsigned long long ULong64_t;//Portable unsigned long integer 8 bytes
 #endif
-#endif
+
 #include <vector>
 
 const int N = 5;
diff --git a/pypy/module/_cppyy/test/std_streams_LinkDef.h b/pypy/module/_cppyy/test/std_streams_LinkDef.h
deleted file mode 100644
--- a/pypy/module/_cppyy/test/std_streams_LinkDef.h
+++ /dev/null
@@ -1,7 +0,0 @@
-#ifdef __CINT__
-
-#pragma link off all globals;
-#pragma link off all classes;
-#pragma link off all functions;
-
-#endif
diff --git a/pypy/module/_cppyy/test/test_advancedcpp.py b/pypy/module/_cppyy/test/test_advancedcpp.py
--- a/pypy/module/_cppyy/test/test_advancedcpp.py
+++ b/pypy/module/_cppyy/test/test_advancedcpp.py
@@ -1,8 +1,12 @@
 import py, os, sys
+from .support import setup_make
+
 
 currpath = py.path.local(__file__).dirpath()
 test_dct = str(currpath.join("advancedcppDict.so"))
 
+def setup_module(mod):
+    setup_make("advancedcppDict.so")
 
 def setup_module(mod):
     if sys.platform == 'win32':
@@ -18,8 +22,8 @@
     def setup_class(cls):
         cls.w_test_dct = cls.space.newtext(test_dct)
         cls.w_advanced = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_default_arguments(self):
         """Test usage of default arguments"""
@@ -146,10 +150,10 @@
     def test03a_namespace_lookup_on_update(self):
         """Test whether namespaces can be shared across dictionaries."""
 
-        import _cppyy
+        import _cppyy, ctypes
         gbl = _cppyy.gbl
 
-        lib2 = _cppyy.load_reflection_info("advancedcpp2Dict.so")
+        lib2 = ctypes.CDLL("./advancedcpp2Dict.so", ctypes.RTLD_GLOBAL)
 
         assert gbl.a_ns      is gbl.a_ns
         assert gbl.a_ns.d_ns is gbl.a_ns.d_ns
diff --git a/pypy/module/_cppyy/test/test_cppyy.py b/pypy/module/_cppyy/test/test_cppyy.py
--- a/pypy/module/_cppyy/test/test_cppyy.py
+++ b/pypy/module/_cppyy/test/test_cppyy.py
@@ -14,7 +14,8 @@
 class TestCPPYYImplementation:
     def test01_class_query(self, space):
         # NOTE: this test needs to run before test_pythonify.py
-        dct = interp_cppyy.load_dictionary(space, test_dct)
+        import ctypes
+        dct = ctypes.CDLL(test_dct)
         w_cppyyclass = interp_cppyy.scope_byname(space, "example01")
         w_cppyyclass2 = interp_cppyy.scope_byname(space, "example01")
         assert space.is_w(w_cppyyclass, w_cppyyclass2)
@@ -30,10 +31,12 @@
     spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
 
     def setup_class(cls):
-        cls.w_example01, cls.w_payload = cls.space.unpackiterable(cls.space.appexec([], """():
-            import _cppyy
-            _cppyy.load_reflection_info(%r)
-            return _cppyy._scope_byname('example01'), _cppyy._scope_byname('payload')""" % (test_dct, )))
+        cls.w_lib, cls.w_example01, cls.w_payload = \
+                   cls.space.unpackiterable(cls.space.appexec([], """():
+            import _cppyy, ctypes
+            lib = ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)
+            return lib, _cppyy._scope_byname('example01'), _cppyy._scope_byname('payload')"""\
+                                                              % (test_dct, )))
 
     def test01_static_int(self):
         """Test passing of an int, returning of an int, and overloading on a
diff --git a/pypy/module/_cppyy/test/test_crossing.py b/pypy/module/_cppyy/test/test_crossing.py
--- a/pypy/module/_cppyy/test/test_crossing.py
+++ b/pypy/module/_cppyy/test/test_crossing.py
@@ -1,4 +1,5 @@
 import py, os, sys
+from .support import setup_make
 
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from rpython.translator.tool.cbuild import ExternalCompilationInfo
@@ -14,11 +15,7 @@
 test_dct = str(currpath.join("crossingDict.so"))
 
 def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make crossingDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
+    setup_make("crossingDict.so")
 
 # from pypy/module/cpyext/test/test_cpyext.py; modified to accept more external
 # symbols and called directly instead of import_module
@@ -145,8 +142,8 @@
     def test02_crossing_dict(self):
         """Test availability of all needed classes in the dict"""
 
-        import _cppyy
-        _cppyy.load_reflection_info(self.test_dct)
+        import _cppyy, ctypes
+        lib = ctypes.CDLL(self.test_dct, ctypes.RTLD_GLOBAL)
 
         assert _cppyy.gbl.crossing == _cppyy.gbl.crossing
         crossing = _cppyy.gbl.crossing
diff --git a/pypy/module/_cppyy/test/test_datatypes.py b/pypy/module/_cppyy/test/test_datatypes.py
--- a/pypy/module/_cppyy/test/test_datatypes.py
+++ b/pypy/module/_cppyy/test/test_datatypes.py
@@ -14,17 +14,11 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_datatypes = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
         cls.w_N = cls.space.newint(5)  # should be imported from the dictionary
 
-    def test01_load_reflection_cache(self):
-        """Loading reflection info twice should result in the same object"""
-        import _cppyy
-        lib2 = _cppyy.load_reflection_info(self.test_dct)
-        assert self.datatypes is lib2
-
-    def test02_instance_data_read_access(self):
+    def test01_instance_data_read_access(self):
         """Read access to instance public data and verify values"""
 
         import _cppyy
@@ -112,7 +106,7 @@
 
         c.__destruct__()
 
-    def test03_instance_data_write_access(self):
+    def test02_instance_data_write_access(self):
         """Test write access to instance public data and verify values"""
 
         import _cppyy
@@ -199,7 +193,7 @@
 
         c.__destruct__()
 
-    def test04_array_passing(self):
+    def test03_array_passing(self):
         """Test passing of array arguments"""
 
         import _cppyy, array, sys
@@ -237,7 +231,7 @@
 
         c.__destruct__()
 
-    def test05_class_read_access(self):
+    def test04_class_read_access(self):
         """Test read access to class public data and verify values"""
 
         import _cppyy, sys
@@ -278,7 +272,7 @@
 
         c.__destruct__()
 
-    def test06_class_data_write_access(self):
+    def test05_class_data_write_access(self):
         """Test write access to class public data and verify values"""
 
         import _cppyy, sys
@@ -342,7 +336,7 @@
 
         c.__destruct__()
 
-    def test07_range_access(self):
+    def test06_range_access(self):
         """Test the ranges of integer types"""
 
         import _cppyy, sys
@@ -358,7 +352,7 @@
 
         c.__destruct__()
 
-    def test08_type_conversions(self):
+    def test07_type_conversions(self):
         """Test conversions between builtin types"""
 
         import _cppyy, sys
@@ -376,7 +370,7 @@
 
         c.__destruct__()
 
-    def test09_global_builtin_type(self):
+    def test08_global_builtin_type(self):
         """Test access to a global builtin type"""
 
         import _cppyy
@@ -392,7 +386,7 @@
         assert gbl.get_global_int() == 22
         assert gbl.g_int == 22
 
-    def test10_global_ptr(self):
+    def test09_global_ptr(self):
         """Test access of global objects through a pointer"""
 
         import _cppyy
@@ -423,7 +417,7 @@
         assert gbl.g_pod.m_int == 43
         assert gbl.g_pod.m_double == 2.14
 
-    def test11_enum(self):
+    def test10_enum(self):
         """Test access to enums"""
 
         import _cppyy
@@ -471,7 +465,7 @@
         assert gbl.kBanana == 29
         assert gbl.kCitrus == 34
 
-    def test12_string_passing(self):
+    def test11_string_passing(self):
         """Test passing/returning of a const char*"""
 
         import _cppyy
@@ -481,7 +475,7 @@
         assert c.get_valid_string('aap') == 'aap'
         #assert c.get_invalid_string() == ''
 
-    def test13_copy_contructor(self):
+    def test12_copy_contructor(self):
         """Test copy constructor"""
 
         import _cppyy
@@ -497,7 +491,7 @@
         for i in range(4):
             assert t1[i] == t3[i]
 
-    def test14_object_returns(self):
+    def test13_object_returns(self):
         """Test access to and return of PODs"""
 
         import _cppyy
@@ -524,7 +518,7 @@
         assert c.get_pod_ptrref().m_int == 666
         assert c.get_pod_ptrref().m_double == 3.14
 
-    def test15_object_arguments(self):
+    def test14_object_arguments(self):
         """Test setting and returning of a POD through arguments"""
 
         import _cppyy
@@ -592,7 +586,7 @@
         assert p.m_int == 888
         assert p.m_double == 3.14
 
-    def test16_nullptr_passing(self):
+    def test15_nullptr_passing(self):
         """Integer 0 ('NULL') and None allowed to pass through instance*"""
 
         import _cppyy
@@ -607,7 +601,7 @@
             assert not c.m_ppod
             assert not c.get_pod_ptr()
 
-    def test17_respect_privacy(self):
+    def test16_respect_privacy(self):
         """Test that privacy settings are respected"""
 
         import _cppyy
@@ -620,7 +614,7 @@
 
         c.__destruct__()
 
-    def test18_object_and_pointer_comparisons(self):
+    def test17_object_and_pointer_comparisons(self):
         """Verify object and pointer comparisons"""
 
         import _cppyy
@@ -657,7 +651,7 @@
         assert l3 != l5
         assert l5 != l3
 
-    def test19_object_validity(self):
+    def test18_object_validity(self):
         """Test object validity checking"""
 
         from _cppyy import gbl
@@ -671,7 +665,7 @@
 
         assert not d2
 
-    def test20_buffer_reshaping(self):
+    def test19_buffer_reshaping(self):
         """Test usage of buffer sizing"""
 
         import _cppyy
@@ -692,7 +686,7 @@
             for i in range(self.N):
                 assert arr[i] == l[i]
 
-    def test21_voidp(self):
+    def test20_voidp(self):
         """Test usage of void* data"""
 
         import _cppyy
diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py
--- a/pypy/module/_cppyy/test/test_fragile.py
+++ b/pypy/module/_cppyy/test/test_fragile.py
@@ -1,15 +1,12 @@
 import py, os, sys
+from .support import setup_make
+
 
 currpath = py.path.local(__file__).dirpath()
 test_dct = str(currpath.join("fragileDict.so"))
 
-
 def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make fragileDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
+    setup_make("fragileDict.so")
 
 class AppTestFRAGILE:
     spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
@@ -17,21 +14,10 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_fragile = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
-    def test01_load_failure(self):
-        """Test failure to load dictionary"""
-
-        import _cppyy
-        raises(RuntimeError, _cppyy.load_reflection_info, "does_not_exist.so")
-
-        try:
-            _cppyy.load_reflection_info("does_not_exist.so")
-        except RuntimeError as e:
-            assert "does_not_exist.so" in str(e)
-
-    def test02_missing_classes(self):
+    def test01_missing_classes(self):
         """Test (non-)access to missing classes"""
 
         import _cppyy
@@ -54,7 +40,7 @@
         assert fragile.C().check() == ord('C')
         raises(TypeError, fragile.C().use_no_such, None)
 
-    def test03_arguments(self):
+    def test02_arguments(self):
         """Test reporting when providing wrong arguments"""
 
         import _cppyy
@@ -72,7 +58,7 @@
         d.overload('a')
         d.overload(1)
 
-    def test04_unsupported_arguments(self):
+    def test03_unsupported_arguments(self):
         """Test arguments that are yet unsupported"""
 
         import _cppyy
@@ -87,7 +73,7 @@
         raises(TypeError, e.overload, None)
         raises(TypeError, getattr, e, 'm_pp_no_such')
 
-    def test05_wrong_arg_addressof(self):
+    def test04_wrong_arg_addressof(self):
         """Test addressof() error reporting"""
 
         import _cppyy
@@ -109,7 +95,7 @@
         assert _cppyy.addressof(None)               == 0
         assert _cppyy.addressof(_cppyy.gbl.nullptr) == 0
 
-    def test06_wrong_this(self):
+    def test05_wrong_this(self):
         """Test that using an incorrect self argument raises"""
 
         import _cppyy
@@ -130,7 +116,7 @@
         assert isinstance(a.gime_null(), fragile.A)
         raises(ReferenceError, fragile.A.check, a.gime_null())
 
-    def test07_unnamed_enum(self):
+    def test06_unnamed_enum(self):
         """Test that an unnamed enum does not cause infinite recursion"""
 
         import _cppyy
@@ -141,7 +127,7 @@
 
         g = fragile.G()
 
-    def test08_unhandled_scoped_datamember(self):
+    def test07_unhandled_scoped_datamember(self):
         """Test that an unhandled scoped data member does not cause infinite recursion"""
 
         import _cppyy
@@ -152,7 +138,7 @@
 
         h = fragile.H()
 
-    def test09_operator_bool(self):
+    def test08_operator_bool(self):
         """Access to global vars with an operator bool() returning False"""
 
         import _cppyy
@@ -163,7 +149,7 @@
         g = _cppyy.gbl.fragile.gI
         assert not g
 
-    def test10_documentation(self):
+    def test09_documentation(self):
         """Check contents of documentation"""
 
         import _cppyy
@@ -205,7 +191,7 @@
         except TypeError as e:
             assert "cannot instantiate abstract class 'O'" in str(e)
 
-    def test11_dir(self):
+    def test10_dir(self):
         """Test __dir__ method"""
 
         import _cppyy
@@ -223,7 +209,7 @@
         #assert 'fglobal' in members          # function
         #assert 'gI'in members                # variable
 
-    def test12_imports(self):
+    def test11_imports(self):
         """Test ability to import from namespace (or fail with ImportError)"""
 
         import _cppyy
@@ -258,7 +244,7 @@
         from _cppyy.gbl.fragile.nested1.nested2.nested3 import A
         assert _cppyy.gbl.fragile.nested1.nested2.nested3.A is nested3.A
 
-    def test13_missing_casts(self):
+    def test12_missing_casts(self):
         """Test proper handling when a hierarchy is not fully available"""
 
         import _cppyy
@@ -275,7 +261,7 @@
         l = k.GimeL()
         assert l is k.GimeL()
 
-    def test14_double_enum_trouble(self):
+    def test13_double_enum_trouble(self):
         """Test a redefinition of enum in a derived class"""
 
         return # don't bother; is fixed in cling-support
diff --git a/pypy/module/_cppyy/test/test_operators.py b/pypy/module/_cppyy/test/test_operators.py
--- a/pypy/module/_cppyy/test/test_operators.py
+++ b/pypy/module/_cppyy/test/test_operators.py
@@ -1,15 +1,12 @@
 import py, os, sys
+from .support import setup_make
 
 
 currpath = py.path.local(__file__).dirpath()
 test_dct = str(currpath.join("operatorsDict.so"))
 
 def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make operatorsDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
+    setup_make("operatorsDict.so")
 
 class AppTestOPERATORS:
     spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
@@ -18,8 +15,8 @@
         cls.w_N = cls.space.newint(5)  # should be imported from the dictionary
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_operators = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def teardown_method(self, meth):
         import gc
diff --git a/pypy/module/_cppyy/test/test_overloads.py b/pypy/module/_cppyy/test/test_overloads.py
--- a/pypy/module/_cppyy/test/test_overloads.py
+++ b/pypy/module/_cppyy/test/test_overloads.py
@@ -18,8 +18,8 @@
         env = os.environ
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_overloads = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_class_based_overloads(self):
         """Test functions overloaded on different C++ clases"""
diff --git a/pypy/module/_cppyy/test/test_pythonify.py b/pypy/module/_cppyy/test/test_pythonify.py
--- a/pypy/module/_cppyy/test/test_pythonify.py
+++ b/pypy/module/_cppyy/test/test_pythonify.py
@@ -1,7 +1,7 @@
 import py, os, sys
+from .support import setup_make
 
 from pypy.module._cppyy import interp_cppyy, executor
-from .support import setup_make
 
 
 currpath = py.path.local(__file__).dirpath()
@@ -16,16 +16,10 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_example01 = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
-    def test01_load_dictionary_cache(self):
-        """Test whether loading a dictionary twice results in the same object."""
-        import _cppyy
-        lib2 = _cppyy.load_reflection_info(self.test_dct)
-        assert self.example01 is lib2
-
-    def test02_finding_classes(self):
+    def test01_finding_classes(self):
         """Test the lookup of a class, and its caching."""
         import _cppyy
         example01_class = _cppyy.gbl.example01
@@ -34,7 +28,7 @@
 
         raises(AttributeError, '_cppyy.gbl.nonexistingclass')
 
-    def test03_calling_static_functions(self):
+    def test02_calling_static_functions(self):
         """Test calling of static methods."""
         import _cppyy, sys, math
         example01_class = _cppyy.gbl.example01
@@ -70,7 +64,7 @@
 
         raises(TypeError, 'example01_class.staticStrcpy(1.)')   # TODO: this leaks
 
-    def test04_constructing_and_calling(self):
+    def test03_constructing_and_calling(self):
         """Test object and method calls."""
         import _cppyy
         example01_class = _cppyy.gbl.example01
@@ -122,7 +116,7 @@
         instance.__destruct__()
         assert example01_class.getCount() == 0
 
-    def test05_passing_object_by_pointer(self):
+    def test04_passing_object_by_pointer(self):
         import _cppyy
         example01_class = _cppyy.gbl.example01
         payload_class = _cppyy.gbl.payload
@@ -145,7 +139,7 @@
         e.__destruct__()
         assert example01_class.getCount() == 0
 
-    def test06_returning_object_by_pointer(self):
+    def test05_returning_object_by_pointer(self):
         import _cppyy
         example01_class = _cppyy.gbl.example01
         payload_class = _cppyy.gbl.payload
@@ -165,7 +159,7 @@
         e.__destruct__()
         assert example01_class.getCount() == 0
 
-    def test07_returning_object_by_value(self):
+    def test06_returning_object_by_value(self):
         import _cppyy
         example01_class = _cppyy.gbl.example01
         payload_class = _cppyy.gbl.payload
@@ -187,7 +181,7 @@
         e.__destruct__()
         assert example01_class.getCount() == 0
 
-    def test08_global_functions(self):
+    def test07_global_functions(self):
         import _cppyy
 
         assert _cppyy.gbl.globalAddOneToInt(3) == 4    # creation lookup
@@ -196,7 +190,7 @@
         assert _cppyy.gbl.ns_example01.globalAddOneToInt(4) == 5
         assert _cppyy.gbl.ns_example01.globalAddOneToInt(4) == 5
 
-    def test09_memory(self):
+    def test08_memory(self):
         """Test proper C++ destruction by the garbage collector"""
 
         import _cppyy, gc
@@ -240,7 +234,7 @@
 
         # TODO: need ReferenceError on touching pl_a
 
-    def test10_default_arguments(self):
+    def test09_default_arguments(self):
         """Test propagation of default function arguments"""
 
         import _cppyy
@@ -275,7 +269,7 @@
             assert g(11., 2)           ==  2.
             assert g(11.)              == 11.
 
-    def test11_overload_on_arguments(self):
+    def test10_overload_on_arguments(self):
         """Test functions overloaded on arguments"""
 
         import _cppyy
@@ -286,14 +280,14 @@
         assert e.overloadedAddDataToInt(4, 5)    == 10
         assert e.overloadedAddDataToInt(6, 7, 8) == 22
 
-    def test12_typedefs(self):
+    def test11_typedefs(self):
         """Test access and use of typedefs"""
 
         import _cppyy
 
         assert _cppyy.gbl.example01 == _cppyy.gbl.example01_t
 
-    def test13_underscore_in_class_name(self):
+    def test12_underscore_in_class_name(self):
         """Test recognition of '_' as part of a valid class name"""
 
         import _cppyy
@@ -305,7 +299,7 @@
         assert hasattr(z, 'myint')
         assert z.gime_z_(z)
 
-    def test14_bound_unbound_calls(self):
+    def test13_bound_unbound_calls(self):
         """Test (un)bound method calls"""
 
         import _cppyy
@@ -319,7 +313,7 @@
         e = _cppyy.gbl.example01(2)
         assert 5 == meth(e, 3)
 
-    def test15_installable_function(self):
+    def test14_installable_function(self):
        """Test installing and calling global C++ function as python method"""
 
        import _cppyy
@@ -330,7 +324,7 @@
        assert 2 == e.fresh(1)
        assert 3 == e.fresh(2)
 
-    def test16_subclassing(self):
+    def test15_subclassing(self):
         """A sub-class on the python side should have that class as type"""
 
         import _cppyy
@@ -376,8 +370,8 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_example01 = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_pythonizations(self):
         """Test addition of user-defined pythonizations"""
diff --git a/pypy/module/_cppyy/test/test_stltypes.py b/pypy/module/_cppyy/test/test_stltypes.py
--- a/pypy/module/_cppyy/test/test_stltypes.py
+++ b/pypy/module/_cppyy/test/test_stltypes.py
@@ -1,15 +1,12 @@
 import py, os, sys
+from .support import setup_make
 
 
 currpath = py.path.local(__file__).dirpath()
 test_dct = str(currpath.join("stltypesDict.so"))
 
 def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make stltypesDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
+    setup_make("stltypesDict.so")
 
 class AppTestSTLVECTOR:
     spaceconfig = dict(usemodules=['_cppyy', '_rawffi', 'itertools'])
@@ -18,8 +15,8 @@
         cls.w_N = cls.space.newint(13)
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlvector = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_builtin_type_vector_types(self):
         """Test access to std::vector<int>/std::vector<double>"""
@@ -205,8 +202,8 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlstring = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_string_argument_passing(self):
         """Test mapping of python strings and std::string"""
@@ -284,8 +281,8 @@
         cls.w_N = cls.space.newint(13)
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlstring = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_builtin_list_type(self):
         """Test access to a list<int>"""
@@ -340,8 +337,8 @@
         cls.w_N = cls.space.newint(13)
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlstring = cls.space.appexec([], """():
-            import _cppyy
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_builtin_map_type(self):
         """Test access to a map<int,int>"""
@@ -448,8 +445,8 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlstring = cls.space.appexec([], """():
-            import _cppyy, sys
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_builtin_vector_iterators(self):
         """Test iterator comparison with operator== reflected"""
@@ -485,8 +482,8 @@
     def setup_class(cls):
         cls.w_test_dct  = cls.space.newtext(test_dct)
         cls.w_stlstring = cls.space.appexec([], """():
-            import _cppyy, sys
-            return _cppyy.load_reflection_info(%r)""" % (test_dct, ))
+            import ctypes
+            return ctypes.CDLL(%r, ctypes.RTLD_GLOBAL)""" % (test_dct, ))
 
     def test01_explicit_templates(self):
         """Explicit use of Template class"""
diff --git a/pypy/module/_cppyy/test/test_zjit.py b/pypy/module/_cppyy/test/test_zjit.py
--- a/pypy/module/_cppyy/test/test_zjit.py
+++ b/pypy/module/_cppyy/test/test_zjit.py
@@ -51,11 +51,7 @@
 test_dct = str(currpath.join("example01Dict.so"))
 
 def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make example01Dict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
+    setup_make("example01Dict.so")
 
 
 class FakeBase(W_Root):
@@ -263,7 +259,8 @@
         space = FakeSpace()
         drv = jit.JitDriver(greens=[], reds=["i", "inst", "cppmethod"])
         def f():
-            lib = interp_cppyy.load_dictionary(space, "./example01Dict.so")
+            import ctypes
+            lib = ctypes.CDLL("./example01Dict.so", ctypes.RTLD_GLOBAL)
             cls  = interp_cppyy.scope_byname(space, "example01")
             inst = cls.get_overload("example01").call(None, [FakeInt(0)])
             cppmethod = cls.get_overload(method_name)

From pypy.commits at gmail.com  Thu Aug  3 20:16:33 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:33 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: make test_zjit.py work with the
 loadable_capi
Message-ID: <5983bce1.d48bdf0a.2f446.52fb@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92064:270130f315a5
Date: 2017-08-03 16:43 -0700
http://bitbucket.org/pypy/pypy/changeset/270130f315a5/

Log:	make test_zjit.py work with the loadable_capi

diff --git a/pypy/module/_cppyy/test/test_zjit.py b/pypy/module/_cppyy/test/test_zjit.py
--- a/pypy/module/_cppyy/test/test_zjit.py
+++ b/pypy/module/_cppyy/test/test_zjit.py
@@ -1,4 +1,6 @@
 import py, os, sys
+from .support import setup_make
+
 from rpython.jit.metainterp.test.support import LLJitMixin
 from rpython.rlib.objectmodel import specialize, instantiate
 from rpython.rlib import rarithmetic, rbigint, jit
@@ -8,9 +10,9 @@
 
 from pypy.module._cppyy import interp_cppyy, capi, executor
 # These tests are for the backend that support the fast path only.
-if capi.identify() == 'loadable_capi':
-    py.test.skip("can not currently use FakeSpace with _cffi_backend")
-elif os.getenv("CPPYY_DISABLE_FASTPATH"):
+#if capi.identify() == 'loadable_capi':
+#    py.test.skip("can not currently use FakeSpace with _cffi_backend")
+if os.getenv("CPPYY_DISABLE_FASTPATH"):
     py.test.skip("fast path is disabled by CPPYY_DISABLE_FASTPATH envar")
 
 # load cpyext early, or its global vars are counted as leaks in the test
@@ -57,14 +59,11 @@
 class FakeBase(W_Root):
     typename = None
 
-class FakeBool(FakeBase):
-    typename = "bool"
-    def __init__(self, val):
-        self.val = val
 class FakeInt(FakeBase):
     typename = "int"
     def __init__(self, val):
         self.val = val
+FakeBool = FakeInt
 class FakeLong(FakeBase):
     typename = "long"
     def __init__(self, val):
@@ -110,6 +109,13 @@
     def __init__(self, space):
         self.slowcalls = 0
 
+class FakeFinalizerQueue(object):
+    def register_finalizer(self, obj):
+        pass
+
+class FakeConfig(object):
+    pass
+
 class FakeSpace(object):
     fake = True
 
@@ -119,10 +125,11 @@
     w_float = FakeType("float")
 
     def __init__(self):
+        self.finalizer_queue = FakeFinalizerQueue()
+
         self.fromcache = InternalSpaceCache(self).getorbuild
         self.user_del_action = FakeUserDelAction(self)
-        class dummy: pass
-        self.config = dummy()
+        self.config = FakeConfig()
         self.config.translating = False
 
         # kill calls to c_call_i (i.e. slow path)
@@ -200,6 +207,10 @@
     def is_w(self, w_one, w_two):
         return w_one is w_two
 
+    def bool_w(self, w_obj, allow_conversion=True):
+        assert isinstance(w_obj, FakeBool)
+        return w_obj.val
+
     def int_w(self, w_obj, allow_conversion=True):
         assert isinstance(w_obj, FakeInt)
         return w_obj.val
@@ -220,7 +231,12 @@
         assert isinstance(obj, str)
         return obj
 
+    def len_w(self, obj):
+        assert isinstance(obj, str)
+        return (obj)
+
     c_int_w = int_w
+    c_uint_w = uint_w
     r_longlong_w = int_w
     r_ulonglong_w = uint_w
 
@@ -254,13 +270,16 @@
     def _freeze_(self):
         return True
 
+
 class TestFastPathJIT(LLJitMixin):
+    def setup_class(cls):
+        import ctypes
+        return ctypes.CDLL(test_dct, ctypes.RTLD_GLOBAL)
+
     def _run_zjit(self, method_name):
         space = FakeSpace()
         drv = jit.JitDriver(greens=[], reds=["i", "inst", "cppmethod"])
         def f():
-            import ctypes
-            lib = ctypes.CDLL("./example01Dict.so", ctypes.RTLD_GLOBAL)
             cls  = interp_cppyy.scope_byname(space, "example01")
             inst = cls.get_overload("example01").call(None, [FakeInt(0)])
             cppmethod = cls.get_overload(method_name)
@@ -277,16 +296,19 @@
         self.check_jitcell_token_count(1)   # same for fast and slow path??
         # rely on replacement of capi calls to raise exception instead (see FakeSpace.__init__)
 
+    @py.test.mark.dont_track_allocations("cppmethod.cif_descr kept 'leaks'")
     def test01_simple(self):
         """Test fast path being taken for methods"""
 
         self._run_zjit("addDataToInt")
 
+    @py.test.mark.dont_track_allocations("cppmethod.cif_descr kept 'leaks'")
     def test02_overload(self):
         """Test fast path being taken for overloaded methods"""
 
         self._run_zjit("overloadedAddDataToInt")
 
+    @py.test.mark.dont_track_allocations("cppmethod.cif_descr kept 'leaks'")
     def test03_const_ref(self):
         """Test fast path being taken for methods with const ref arguments"""
 

From pypy.commits at gmail.com  Thu Aug  3 20:16:35 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:35 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove spurious references to
 CINT
Message-ID: <5983bce3.97a0df0a.b1ca5.abe4@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92065:701628812c28
Date: 2017-08-03 16:43 -0700
http://bitbucket.org/pypy/pypy/changeset/701628812c28/

Log:	remove spurious references to CINT

diff --git a/pypy/module/_cppyy/converter.py b/pypy/module/_cppyy/converter.py
--- a/pypy/module/_cppyy/converter.py
+++ b/pypy/module/_cppyy/converter.py
@@ -867,27 +867,3 @@
     for c_type, alias in aliases:
         _converters[alias] = _converters[c_type]
 _add_aliased_converters()
-
-# ROOT-specific converters (TODO: this is a general use case and should grow
-# an API; putting it here is done only to circumvent circular imports)
-if capi.identify() == "CINT":
-
-    class TStringConverter(InstanceConverter):
-        def __init__(self, space, extra):
-            from pypy.module._cppyy import interp_cppyy
-            cppclass = interp_cppyy.scope_byname(space, "TString")
-            InstanceConverter.__init__(self, space, cppclass)
-
-        def _unwrap_object(self, space, w_obj):
-            from pypy.module._cppyy import interp_cppyy
-            if isinstance(w_obj, interp_cppyy.W_CPPInstance):
-                arg = InstanceConverter._unwrap_object(self, space, w_obj)
-                return capi.backend.c_TString2TString(space, arg)
-            else:
-                return capi.backend.c_charp2TString(space, space.text_w(w_obj))
-
-        def free_argument(self, space, arg, call_local):
-            capi.c_destruct(space, self.cppclass, rffi.cast(capi.C_OBJECT, rffi.cast(rffi.VOIDPP, arg)[0]))
-
-    _converters["TString"]        = TStringConverter
-    _converters["const TString&"] = TStringConverter
diff --git a/pypy/module/_cppyy/test/advancedcpp.h b/pypy/module/_cppyy/test/advancedcpp.h
--- a/pypy/module/_cppyy/test/advancedcpp.h
+++ b/pypy/module/_cppyy/test/advancedcpp.h
@@ -246,9 +246,7 @@
     int m_i;
 };
 
-#ifndef __CINT__
 template class std::vector<ref_tester>;
-#endif
 
 
 //===========================================================================
diff --git a/pypy/module/_cppyy/test/datatypes.h b/pypy/module/_cppyy/test/datatypes.h
--- a/pypy/module/_cppyy/test/datatypes.h
+++ b/pypy/module/_cppyy/test/datatypes.h
@@ -1,5 +1,5 @@
 // copied from RtypesCore.h ...
-#if defined(R__WIN32) && !defined(__CINT__)
+#if defined(R__WIN32)
 typedef __int64          Long64_t;  //Portable signed long integer 8 bytes
 typedef unsigned __int64 ULong64_t; //Portable unsigned long integer 8 bytes
 #else
diff --git a/pypy/module/_cppyy/test/stltypes.h b/pypy/module/_cppyy/test/stltypes.h
--- a/pypy/module/_cppyy/test/stltypes.h
+++ b/pypy/module/_cppyy/test/stltypes.h
@@ -74,9 +74,7 @@
 
     struct _CppyyMapInstances {
 
-#ifndef __CINT__
         STLTYPE_INSTANTIATION2(map, int,         int,           1);
-#endif
         STLTYPE_INSTANTIATION2(map, std::string, int,           2);
         STLTYPE_INSTANTIATION2(map, std::string, unsigned int,  3);
         STLTYPE_INSTANTIATION2(map, std::string, unsigned long, 4);

From pypy.commits at gmail.com  Thu Aug  3 20:16:37 2017
From: pypy.commits at gmail.com (wlav)
Date: Thu, 03 Aug 2017 17:16:37 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove capi.identify() as
 that's the backend's job
Message-ID: <5983bce5.88aa1c0a.7f018.93e3@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92066:7e01e98a0155
Date: 2017-08-03 16:44 -0700
http://bitbucket.org/pypy/pypy/changeset/7e01e98a0155/

Log:	remove capi.identify() as that's the backend's job

diff --git a/pypy/module/_cppyy/capi/loadable_capi.py b/pypy/module/_cppyy/capi/loadable_capi.py
--- a/pypy/module/_cppyy/capi/loadable_capi.py
+++ b/pypy/module/_cppyy/capi/loadable_capi.py
@@ -17,9 +17,6 @@
 
 reflection_library = 'libcppyy_backend.so'
 
-def identify():
-    return 'loadable_capi'
-
 # this is not technically correct, but will do for now
 std_string_name = 'std::basic_string<char>'
 
diff --git a/pypy/module/_cppyy/test/test_zjit.py b/pypy/module/_cppyy/test/test_zjit.py
--- a/pypy/module/_cppyy/test/test_zjit.py
+++ b/pypy/module/_cppyy/test/test_zjit.py
@@ -9,9 +9,7 @@
 from pypy.interpreter.baseobjspace import InternalSpaceCache, W_Root
 
 from pypy.module._cppyy import interp_cppyy, capi, executor
-# These tests are for the backend that support the fast path only.
-#if capi.identify() == 'loadable_capi':
-#    py.test.skip("can not currently use FakeSpace with _cffi_backend")
+
 if os.getenv("CPPYY_DISABLE_FASTPATH"):
     py.test.skip("fast path is disabled by CPPYY_DISABLE_FASTPATH envar")
 

From pypy.commits at gmail.com  Fri Aug  4 03:03:25 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 00:03:25 -0700 (PDT)
Subject: [pypy-commit] pypy default: fix fallout from celldict
 defaultification (this test is anyway much closer now to what we really
 care about)
Message-ID: <59841c3d.5787df0a.babb5.c26a@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92067:1cf0fac8f2fe
Date: 2017-08-04 09:00 +0200
http://bitbucket.org/pypy/pypy/changeset/1cf0fac8f2fe/

Log:	fix fallout from celldict defaultification (this test is anyway much
	closer now to what we really care about)

diff --git a/pypy/module/__builtin__/test/test_classobj.py b/pypy/module/__builtin__/test/test_classobj.py
--- a/pypy/module/__builtin__/test/test_classobj.py
+++ b/pypy/module/__builtin__/test/test_classobj.py
@@ -1090,18 +1090,18 @@
     def setup_class(cls):
         if cls.runappdirect:
             py.test.skip("can only be run on py.py")
-        def is_strdict(space, w_class):
-            from pypy.objspace.std.dictmultiobject import BytesDictStrategy
+        def is_moduledict(space, w_class):
+            from pypy.objspace.std.celldict import ModuleDictStrategy
             w_d = w_class.getdict(space)
-            return space.wrap(isinstance(w_d.get_strategy(), BytesDictStrategy))
+            return space.wrap(isinstance(w_d.get_strategy(), ModuleDictStrategy))
 
-        cls.w_is_strdict = cls.space.wrap(gateway.interp2app(is_strdict))
+        cls.w_is_moduledict = cls.space.wrap(gateway.interp2app(is_moduledict))
 
-    def test_strdict(self):
+    def test_moduledict(self):
         class A:
             a = 1
             b = 2
-        assert self.is_strdict(A)
+        assert self.is_moduledict(A)
 
     def test_attr_slots(self):
         class C:

From pypy.commits at gmail.com  Fri Aug  4 03:03:27 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 00:03:27 -0700 (PDT)
Subject: [pypy-commit] pypy default: unbreak translation (bad me, no cookie)
Message-ID: <59841c3f.57b4df0a.475df.fb6a@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92068:7f9bc66e8749
Date: 2017-08-04 09:01 +0200
http://bitbucket.org/pypy/pypy/changeset/7f9bc66e8749/

Log:	unbreak translation (bad me, no cookie)

diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -119,7 +119,7 @@
     return space.w_False
 
 index_count_jd = jit.JitDriver(
-    greens = ['count', 'tp_item', 'arrclass'],
+    greens = ['count', 'arrclass', 'tp_item'],
     reds = 'auto', name = 'array.index_or_count')
 
 def index_count_array(arr, w_val, count=False):

From pypy.commits at gmail.com  Fri Aug  4 05:26:19 2017
From: pypy.commits at gmail.com (fijal)
Date: Fri, 04 Aug 2017 02:26:19 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: work on the blog post
Message-ID: <59843dbb.0e951c0a.b4e60.97a6@mx.google.com>

Author: fijal
Branch: extradoc
Changeset: r5824:155c9cbc30c7
Date: 2017-08-04 11:26 +0200
http://bitbucket.org/pypy/extradoc/changeset/155c9cbc30c7/

Log:	work on the blog post

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -24,5 +24,35 @@
 already have backing for about 1/3. If we can get a $100k contract, we would
 make it our priority to deliver before the end of the year.
 
+People asked several questions, so I'll try to answer the technical parts
+here.
+
+* What would the plan entail?
+
+We've already done the work on Garbage Collector to allow doing multi
+threaded programs in RPython. "all" that's left is adding locks on mutable
+data structures everywhere in PyPy codebase. Since it'll significantly complicated
+our workflow, we need to see real interest in that topic, backed up by
+commercial contracts, otherwise we're not going to do it.
+
+* Why the STM effort did not work out?
+
+STM was a research project that proved that the idea is possible. However,
+the amount of user effort that's required to make programs run nicely
+parallelizable is both significant and we never managed to develop tools
+that would help nicely. At the present time we're not sure if more manpower
+spent on tooling would improve the situation or idea is doomed. The whole
+approach also ended up being a significant overhead on single threaded programs,
+which means that it's very easy to make your programs slower.
+
+* Would subinterpreters not be a better idea?
+
+Python is a very mutable language - there is tons of mutable state and
+basic objects that are compile time in other languages, like classes and functions
+are mutable at runtime. That means that sharing things between subinterpreters would
+be restricted to basic immutable data structures, which defeats the point compared
+to multi-processing approach. We don't believe it's a viable approach without
+seriously impacting the semantics of the language.
+
 Best regards,
 Maciej Fijalkowski

From pypy.commits at gmail.com  Fri Aug  4 05:29:20 2017
From: pypy.commits at gmail.com (Dodan)
Date: Fri, 04 Aug 2017 02:29:20 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Fixed x86 32bit
 translation error and the multiprocessing_forkserver
Message-ID: <59843e70.935c1c0a.4cb2b.fc98@mx.google.com>

Author: Dodan Mihai <mihai.dodan at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92069:318fcd21d1de
Date: 2017-08-04 12:27 +0300
http://bitbucket.org/pypy/pypy/changeset/318fcd21d1de/

Log:	Fixed x86 32bit translation error and the multiprocessing_forkserver

diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py
--- a/rpython/rlib/rsocket.py
+++ b/rpython/rlib/rsocket.py
@@ -996,22 +996,16 @@
         retflag[0] = rffi.cast(rffi.SIGNED,0)
 
         # a mask for the SIGNEDP's that need to be cast to int. (long default)
-        #LONG_MASK =  2**32 - 1
         reply = _c.recvmsg(self.fd, rffi.cast(lltype.Signed,message_size),
                            rffi.cast(lltype.Signed,ancbufsize),rffi.cast(lltype.Signed,flags),
                            addr_p, addrlen_p, len_of_msgs, messages, no_of_messages,size_of_anc,
                            levels, types,file_descr,descr_per_anc,retflag)
         if reply >= 0:
-            msg_no = rffi.cast(rffi.SIGNED,no_of_messages[0])
             anc_size  = rffi.cast(rffi.SIGNED,size_of_anc[0])
             returnflag  = rffi.cast(rffi.SIGNED,retflag[0])
             addrlen = rffi.cast(rffi.SIGNED,addrlen_p[0])
-            retmsg = ""
 
-            for i in range(msg_no):
-                x = rffi.cast(rffi.SIGNED,len_of_msgs[0][i])
-                #x &= LONG_MASK
-                retmsg = rffi.charp2strn(messages[0],x)
+            retmsg = rffi.charpsize2str(messages[0],reply)
 
             offset = 0
             list_of_tuples = []

From pypy.commits at gmail.com  Fri Aug  4 05:29:56 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:29:56 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-tcp-style-trx-length: Change default
 transaction length to ~40 KB
Message-ID: <59843e94.4d86df0a.40611.5628@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-tcp-style-trx-length
Changeset: r2142:cdbb6dade13a
Date: 2017-08-02 17:10 +0200
http://bitbucket.org/pypy/stmgc/changeset/cdbb6dade13a/

Log:	Change default transaction length to ~40 KB

diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -20,8 +20,8 @@
 // corresponds to ~4 GB
 #define LARGE_FILL_MARK_NURSERY_BYTES   0x100000000L
 
-// corresponds to ~4 MB nursery fill
-#define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.001)
+// corresponds to ~40 KB nursery fill
+#define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.00001)
 // corresponds to ~4 KB nursery fill
 #define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.000001)
 

From pypy.commits at gmail.com  Fri Aug  4 05:29:59 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:29:59 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-tcp-style-trx-length: Increase min trx len
 to ~400KB
Message-ID: <59843e97.a1aedf0a.521b.daae@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-tcp-style-trx-length
Changeset: r2144:2d68598c75fd
Date: 2017-08-03 12:14 +0200
http://bitbucket.org/pypy/stmgc/changeset/2d68598c75fd/

Log:	Increase min trx len to ~400KB

diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -22,8 +22,8 @@
 
 // corresponds to ~4 MB nursery fill
 #define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.001)
-// corresponds to ~4 KB nursery fill
-#define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.000001)
+// corresponds to ~400 KB nursery fill
+#define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.0001)
 
 #define BACKOFF_COUNT (20)
 #define BACKOFF_MULTIPLIER (BACKOFF_COUNT / -log10(STM_MIN_RELATIVE_TRANSACTION_LENGTH))

From pypy.commits at gmail.com  Fri Aug  4 05:29:58 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:29:58 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-tcp-style-trx-length: Backed out changeset
 cdbb6dade13a
Message-ID: <59843e96.1db7df0a.41cce.aa22@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-tcp-style-trx-length
Changeset: r2143:7ffcfbd0993a
Date: 2017-08-03 12:13 +0200
http://bitbucket.org/pypy/stmgc/changeset/7ffcfbd0993a/

Log:	Backed out changeset cdbb6dade13a

diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -20,8 +20,8 @@
 // corresponds to ~4 GB
 #define LARGE_FILL_MARK_NURSERY_BYTES   0x100000000L
 
-// corresponds to ~40 KB nursery fill
-#define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.00001)
+// corresponds to ~4 MB nursery fill
+#define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.001)
 // corresponds to ~4 KB nursery fill
 #define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.000001)
 

From pypy.commits at gmail.com  Fri Aug  4 05:43:51 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:43:51 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Merge
 instrumentation updates
Message-ID: <598441d7.8baddf0a.9799c.9434@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2146:bcf327fad4ce
Date: 2017-08-04 11:42 +0200
http://bitbucket.org/pypy/stmgc/changeset/bcf327fad4ce/

Log:	Merge instrumentation updates

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -469,8 +469,8 @@
 #endif
 
     if (STM_PSEGMENT->last_commit_log_entry->next == INEV_RUNNING) {
-        pause_timer();
-        wait_for_inevitable(); // TODO may abort!! timing event lost
+        stop_timer_and_publish(STM_DURATION_VALIDATION);
+        wait_for_inevitable();
         continue_timer();
         goto retry_from_start;   /* redo _stm_validate() now */
     }
@@ -560,14 +560,13 @@
         OPT_ASSERT(yes);
 
         release_modification_lock_wr(STM_SEGMENT->segment_num);
+
+        stop_timer_and_publish(STM_DURATION_VALIDATION);
     }
     else {
-        pause_timer();
+        stop_timer_and_publish(STM_DURATION_VALIDATION);
         _validate_and_attach(new);
-        continue_timer();
     }
-
-    stop_timer_and_publish(STM_DURATION_VALIDATION);
 }
 
 /* ############# STM ############# */
@@ -1328,7 +1327,9 @@
        if there is an inevitable tx running) */
     bool was_inev = STM_PSEGMENT->transaction_state == TS_INEVITABLE;
 
-    pause_timer();
+    // publish here because the validation may abort
+    stop_timer_and_publish_for_thread(
+        thread_local_for_logging, STM_DURATION_COMMIT_EXCEPT_GC);
     _validate_and_add_to_commit_log();
     continue_timer();
 
@@ -1653,9 +1654,9 @@
             signal_commit_to_inevitable_transaction();
 
             s_mutex_lock();
-            if (any_soon_finished_or_inevitable_thread_segment() &&
-                    !safe_point_requested() &&
-                    num_waits <= NB_SEGMENTS) {
+            if (any_soon_finished_or_inevitable_thread_segment()
+                && !safe_point_requested()
+                && num_waits <= NB_SEGMENTS) {
 
                 /* wait until C_SEGMENT_FREE_OR_SAFE_POINT_REQ is signalled */
                 EMIT_WAIT(STM_WAIT_OTHER_INEVITABLE);
diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -759,16 +759,21 @@
 
 static void major_do_validation_and_minor_collections(void)
 {
+    start_timer();
+
     int original_num = STM_SEGMENT->segment_num;
     long i;
 
     assert(_has_mutex());
 
     /* including the sharing seg0 */
-    for (i = 0; i < NB_SEGMENTS; i++) {
+    for (i = 0; i < NB_SEGMENTS; i++) { // TODO why is this strictly smaller than?
         ensure_gs_register(i);
 
+        pause_timer();
         bool ok = _stm_validate();
+        continue_timer();
+
         assert(get_priv_segment(i)->last_commit_log_entry->next == NULL
                || get_priv_segment(i)->last_commit_log_entry->next == INEV_RUNNING);
         if (!ok) {
@@ -803,7 +808,9 @@
            Collecting might fail due to invalid state.
         */
         if (!must_abort()) {
+            pause_timer();
             _do_minor_collection(/*commit=*/ false);
+            continue_timer();
             assert(MINOR_NOTHING_TO_DO(STM_PSEGMENT));
         }
         else {
@@ -813,6 +820,8 @@
     }
 
     ensure_gs_register(original_num);
+
+    stop_timer_and_publish(STM_DURATION_MAJOR_GC_FULL);
 }
 
 
diff --git a/c8/stm/timing.h b/c8/stm/timing.h
--- a/c8/stm/timing.h
+++ b/c8/stm/timing.h
@@ -8,6 +8,8 @@
 #define start_timer() struct timespec start, stop;                             \
                       struct timespec duration = { .tv_sec = 0, .tv_nsec = 0 };\
                       uint32_t nanosec_diff, sec_diff;                         \
+                      stm_timing_event_payload_data_t stm_duration_data;       \
+                      stm_timing_event_payload_t stm_duration_payload;         \
                       continue_timer()
 
 /* Must use start_timer before using this macro. */
@@ -27,11 +29,12 @@
 #define pause_timer() clock_gettime(CLOCK_MONOTONIC_RAW, &stop);            \
                       get_duration()
 
+#define reset_timer() duration.tv_sec = 0; duration.tv_nsec = 0;
+
 #define stm_duration_payload(duration_data)                                 \
-    stm_timing_event_payload_data_t stm_duration_data =                     \
-        { .duration = &(duration_data) };                                   \
-    stm_timing_event_payload_t stm_duration_payload =                       \
-        { STM_EVENT_PAYLOAD_DURATION, stm_duration_data };
+    stm_duration_data.duration = &(duration_data);                          \
+    stm_duration_payload.type = STM_EVENT_PAYLOAD_DURATION;                 \
+    stm_duration_payload.data = stm_duration_data;
 
 #define publish_event(thread_local, event)                                  \
     (timing_enabled() ?                                                     \
@@ -42,7 +45,8 @@
     pause_timer()                                                           \
     stm_duration_payload(duration)                                          \
     assert((thread_local) != NULL);                                         \
-    publish_event((thread_local), (event))
+    publish_event((thread_local), (event))                                  \
+    reset_timer()
 
 #define stop_timer_and_publish(event)                                       \
     stop_timer_and_publish_for_thread(STM_SEGMENT->running_thread, (event))
diff --git a/c8/stmgc.h b/c8/stmgc.h
--- a/c8/stmgc.h
+++ b/c8/stmgc.h
@@ -9,7 +9,6 @@
 
 #include <stddef.h>
 #include <stdint.h>
-#include <stdbool.h>
 #include <assert.h>
 #include <limits.h>
 #include <unistd.h>
@@ -581,6 +580,8 @@
     STM_GC_MAJOR_DONE,
 
     /* execution duration profiling events */
+    STM_WARMUP_COMPLETE,
+
     STM_DURATION_START_TRX,
     STM_DURATION_WRITE_GC_ONLY,
     STM_DURATION_WRITE_SLOWPATH,
@@ -611,6 +612,7 @@
     "gc major start",                               \
     "gc major done",                                \
     /* names of duration events */                  \
+    "marks completion of benchmark warm up phase"   \
     "duration of transaction start",                \
     "duration of gc due to write",                  \
     "duration of write slowpath",                   \

From pypy.commits at gmail.com  Fri Aug  4 05:43:49 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:43:49 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Merge commit
 signalling
Message-ID: <598441d5.c49edf0a.66de4.8769@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2145:6cc14f08abf7
Date: 2017-08-04 11:35 +0200
http://bitbucket.org/pypy/stmgc/changeset/6cc14f08abf7/

Log:	Merge commit signalling

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -380,6 +380,14 @@
 static void readd_wb_executed_flags(void);
 static void check_all_write_barrier_flags(char *segbase, struct list_s *list);
 
+static void signal_commit_to_inevitable_transaction(void) {
+    struct stm_priv_segment_info_s* inevitable_segement = get_inevitable_thread_segment();
+    if (inevitable_segement != 0) {
+        // the inevitable thread is still running: set its "please commit" flag (is ignored by the inevitable thread if it is atomic)
+        inevitable_segement->commit_if_not_atomic = true;
+    }
+}
+
 static void wait_for_inevitable(void)
 {
     intptr_t detached = 0;
@@ -396,6 +404,8 @@
            try to detach an inevitable transaction regularly */
         detached = fetch_detached_transaction();
         if (detached == 0) {
+            // the inevitable trx was not detached or it was detached but is atomic
+            signal_commit_to_inevitable_transaction();
             EMIT_WAIT(STM_WAIT_OTHER_INEVITABLE);
             if (!cond_wait_timeout(C_SEGMENT_FREE_OR_SAFE_POINT_REQ, 0.00001))
                 goto wait_some_more;
@@ -1169,6 +1179,7 @@
     _do_start_transaction(tl);
     continue_timer();
 
+    STM_PSEGMENT->commit_if_not_atomic = false;
     STM_SEGMENT->nursery_mark = ((stm_char *)_stm_nursery_start +
                                         stm_get_transaction_length(tl));
 
@@ -1623,7 +1634,7 @@
 
 void _stm_become_inevitable(const char *msg)
 {
-    int num_waits = 0;
+    int num_waits = 1;
 
     timing_become_inevitable();
 
@@ -1634,50 +1645,48 @@
     if (msg != MSG_INEV_DONT_SLEEP) {
         dprintf(("become_inevitable: %s\n", msg));
 
-        if (any_soon_finished_or_inevitable_thread_segment() &&
-                num_waits <= NB_SEGMENTS) {
+        if (any_soon_finished_or_inevitable_thread_segment()) {
 #if STM_TESTS                           /* for tests: another transaction */
             stm_abort_transaction();    /*   is already inevitable, abort */
 #endif
 
-            bool timed_out = false;
+            signal_commit_to_inevitable_transaction();
 
             s_mutex_lock();
             if (any_soon_finished_or_inevitable_thread_segment() &&
-                    !safe_point_requested()) {
+                    !safe_point_requested() &&
+                    num_waits <= NB_SEGMENTS) {
 
                 /* wait until C_SEGMENT_FREE_OR_SAFE_POINT_REQ is signalled */
                 EMIT_WAIT(STM_WAIT_OTHER_INEVITABLE);
-                if (!cond_wait_timeout(C_SEGMENT_FREE_OR_SAFE_POINT_REQ,
-                                       0.000054321))
-                    timed_out = true;
+                if (cond_wait_timeout(C_SEGMENT_FREE_OR_SAFE_POINT_REQ, 0.00001)) {
+                    num_waits++;
+                }
             }
             s_mutex_unlock();
-
-            if (timed_out) {
-                /* try to detach another inevitable transaction, but
-                   only after waiting a bit.  This is necessary to avoid
-                   deadlocks in some situations, which are hopefully
-                   not too common.  We don't want two threads constantly
-                   detaching each other. */
-                intptr_t detached = fetch_detached_transaction();
-                if (detached != 0) {
-                    EMIT_WAIT_DONE();
-                    commit_fetched_detached_transaction(detached);
-                }
-            }
-            else {
-                num_waits++;
+            /* XXX try to detach another inevitable transaction, but
+              only after waiting a bit.  This is necessary to avoid
+              deadlocks in some situations, which are hopefully
+              not too common.  We don't want two threads constantly
+              detaching each other. */
+            intptr_t detached = fetch_detached_transaction();
+            if (detached != 0) {
+               EMIT_WAIT_DONE();
+               commit_fetched_detached_transaction(detached);
+               EMIT_WAIT(STM_WAIT_OTHER_INEVITABLE);
             }
             goto retry_from_start;
         }
-        EMIT_WAIT_DONE();
-        if (!_validate_and_turn_inevitable())
-            goto retry_from_start;
+        else {
+            EMIT_WAIT_DONE();
+            if (!_validate_and_turn_inevitable()) {
+                EMIT_WAIT(STM_WAIT_OTHER_INEVITABLE);
+                goto retry_from_start;
+            }
+        }
     }
-    else {
-        if (!_validate_and_turn_inevitable())
-            return;
+    else if (!_validate_and_turn_inevitable()) {
+        return;
     }
 
     /* There may be a concurrent commit of a detached Tx going on.
@@ -1689,6 +1698,7 @@
         stm_spin_loop();
     assert(_stm_detached_inevitable_from_thread == 0);
 
+    STM_PSEGMENT->commit_if_not_atomic = false;
     soon_finished_or_inevitable_thread_segment();
     STM_PSEGMENT->transaction_state = TS_INEVITABLE;
 
diff --git a/c8/stm/core.h b/c8/stm/core.h
--- a/c8/stm/core.h
+++ b/c8/stm/core.h
@@ -169,6 +169,9 @@
 
     /* For stm_enable_atomic() */
     uintptr_t atomic_nesting_levels;
+
+    // TODO signal flag that is checked in throw_away_nursery() for making immediate commit
+    bool commit_if_not_atomic;
 };
 
 enum /* safe_point */ {
diff --git a/c8/stm/detach.c b/c8/stm/detach.c
--- a/c8/stm/detach.c
+++ b/c8/stm/detach.c
@@ -215,6 +215,7 @@
     }
 }
 
+// TODO write tests, verify is working, verify no overflows with adaptive mode
 uintptr_t stm_is_atomic(stm_thread_local_t *tl)
 {
     assert(STM_SEGMENT->running_thread == tl);
@@ -228,14 +229,18 @@
     return STM_PSEGMENT->atomic_nesting_levels;
 }
 
+// max intptr_t value is 7FFFFFFFFFFFFFFF on 64-bit => larger than 2 * huge value
 #define HUGE_INTPTR_VALUE  0x3000000000000000L
 
 void stm_enable_atomic(stm_thread_local_t *tl)
 {
     if (!stm_is_atomic(tl)) {
+        // do for outermost atomic block only
         tl->self_or_0_if_atomic = 0;
         /* increment 'nursery_mark' by HUGE_INTPTR_VALUE, so that
-           stm_should_break_transaction() returns always false */
+           stm_should_break_transaction() returns always false.
+           preserves the previous nursery_mark, unless it is < 0
+           or >= huge value */
         intptr_t mark = (intptr_t)STM_SEGMENT->nursery_mark;
         if (mark < 0)
             mark = 0;
@@ -255,6 +260,7 @@
     STM_PSEGMENT->atomic_nesting_levels--;
 
     if (STM_PSEGMENT->atomic_nesting_levels == 0) {
+        // revert changes by stm_enable_atomic only if we left the outermost atomic block
         tl->self_or_0_if_atomic = (intptr_t)tl;
         /* decrement 'nursery_mark' by HUGE_INTPTR_VALUE, to cancel
            what was done in stm_enable_atomic() */
diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -527,6 +527,14 @@
     pseg->pub.nursery_current = (stm_char *)_stm_nursery_start;
     pseg->pub.nursery_mark -= nursery_used;
 
+    assert((pseg->transaction_state == TS_INEVITABLE) || !pseg->commit_if_not_atomic);
+    if (pseg->commit_if_not_atomic
+        && pseg->transaction_state == TS_INEVITABLE
+        && pseg->pub.running_thread->self_or_0_if_atomic != 0) {
+        // transaction is inevitable, not atomic, and commit has been signalled by waiting thread: commit immediately
+        pseg->pub.nursery_mark = 0;
+    }
+
     /* free any object left from 'young_outside_nursery' */
     if (!tree_is_cleared(pseg->young_outside_nursery)) {
         wlog_t *item;
diff --git a/c8/stm/sync.c b/c8/stm/sync.c
--- a/c8/stm/sync.c
+++ b/c8/stm/sync.c
@@ -302,6 +302,19 @@
     return false;
 }
 
+static struct stm_priv_segment_info_s* get_inevitable_thread_segment(void)
+{
+    struct stm_priv_segment_info_s* segment;
+    int num;
+    for (num = 1; num < NB_SEGMENTS; num++) {
+        segment = get_priv_segment(num);
+        if (segment->transaction_state == TS_INEVITABLE) {
+            return segment;
+        }
+    }
+    return 0;
+}
+
 __attribute__((unused))
 static bool _seems_to_be_running_transaction(void)
 {
diff --git a/c8/stm/sync.h b/c8/stm/sync.h
--- a/c8/stm/sync.h
+++ b/c8/stm/sync.h
@@ -30,6 +30,7 @@
 static void release_thread_segment(stm_thread_local_t *tl);
 static void soon_finished_or_inevitable_thread_segment(void);
 static bool any_soon_finished_or_inevitable_thread_segment(void);
+static struct stm_priv_segment_info_s* get_inevitable_thread_segment(void);
 
 enum sync_type_e {
     STOP_OTHERS_UNTIL_MUTEX_UNLOCK,

From pypy.commits at gmail.com  Fri Aug  4 05:53:47 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:53:47 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Update
 transaction lengths with learnings from TCP style
Message-ID: <5984442b.11331c0a.c5a14.c802@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2147:cac4878ee56a
Date: 2017-08-04 11:48 +0200
http://bitbucket.org/pypy/stmgc/changeset/cac4878ee56a/

Log:	Update transaction lengths with learnings from TCP style

diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -16,13 +16,17 @@
 static uintptr_t _stm_nursery_start;
 
 #define DEFAULT_FILL_MARK_NURSERY_BYTES (NURSERY_SIZE / 4)
-#define LARGE_FILL_MARK_NURSERY_BYTES   0x1000000000L
+
+// corresponds to ~4 GB
+#define LARGE_FILL_MARK_NURSERY_BYTES   0x100000000L
+// corresponds to ~400 KB nursery fill
+#define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.0001)
 
 static double get_new_transaction_length(stm_thread_local_t *tl, bool aborts) {
     double new = tl->relative_transaction_length;
     if (aborts) {
-        tl->transaction_length_backoff = 3;
-        new = 100.0 / LARGE_FILL_MARK_NURSERY_BYTES;
+        new = STM_MIN_RELATIVE_TRANSACTION_LENGTH;
+        tl->transaction_length_backoff = 20;
     } else if (tl->transaction_length_backoff == 0) {
         new = 1;
     } else { // not abort and backoff != 0

From pypy.commits at gmail.com  Fri Aug  4 05:53:50 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:53:50 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Move
 transaction length update on abort
Message-ID: <5984442e.99a3df0a.bfcf9.b962@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2149:aa5b73c18b88
Date: 2017-07-14 12:49 +0200
http://bitbucket.org/pypy/stmgc/changeset/aa5b73c18b88/

Log:	Move transaction length update on abort

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -1552,8 +1552,6 @@
     did_abort = 1;
 #endif
 
-    stm_transaction_length_handle_validation(pseg->pub.running_thread, true);
-
     list_clear(pseg->objects_pointing_to_nursery);
     list_clear(pseg->old_objects_with_cards_set);
     LIST_FOREACH_R(pseg->large_overflow_objects, uintptr_t /*item*/,
@@ -1584,6 +1582,8 @@
     tl->self_or_0_if_atomic = (intptr_t)tl;   /* clear the 'atomic' flag */
     STM_PSEGMENT->atomic_nesting_levels = 0;
 
+    stm_transaction_length_handle_validation(tl, true);
+
     if (tl->mem_clear_on_abort)
         memset(tl->mem_clear_on_abort, 0, tl->mem_bytes_to_clear_on_abort);
     if (tl->mem_reset_on_abort) {

From pypy.commits at gmail.com  Fri Aug  4 05:53:48 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 02:53:48 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Update trx
 length on commit and abort only
Message-ID: <5984442c.d47d1c0a.daced.73c9@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2148:114803b15227
Date: 2017-07-10 16:55 +0200
http://bitbucket.org/pypy/stmgc/changeset/114803b15227/

Log:	Update trx length on commit and abort only

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -347,7 +347,6 @@
     }
 
     if (thread_local_for_logging != NULL) {
-        stm_transaction_length_handle_validation(thread_local_for_logging, needs_abort);
         stop_timer_and_publish_for_thread(
             thread_local_for_logging, STM_DURATION_VALIDATION);
     }
@@ -1380,6 +1379,8 @@
 
     s_mutex_unlock();
 
+    stm_transaction_length_handle_validation(thread_local_for_logging, false);
+
     stop_timer_and_publish_for_thread(
         thread_local_for_logging, STM_DURATION_COMMIT_EXCEPT_GC);
 
@@ -1551,6 +1552,8 @@
     did_abort = 1;
 #endif
 
+    stm_transaction_length_handle_validation(pseg->pub.running_thread, true);
+
     list_clear(pseg->objects_pointing_to_nursery);
     list_clear(pseg->old_objects_with_cards_set);
     LIST_FOREACH_R(pseg->large_overflow_objects, uintptr_t /*item*/,

From pypy.commits at gmail.com  Fri Aug  4 05:58:06 2017
From: pypy.commits at gmail.com (antocuni)
Date: Fri, 04 Aug 2017 02:58:06 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: some reviews
Message-ID: <5984452e.a29adf0a.412a8.782b@mx.google.com>

Author: Antonio Cuni <anto.cuni at gmail.com>
Branch: extradoc
Changeset: r5825:103974b87e0f
Date: 2017-08-04 11:57 +0200
http://bitbucket.org/pypy/extradoc/changeset/103974b87e0f/

Log:	some reviews

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -3,26 +3,30 @@
 
 Hello everyone.
 
-The topic of the infamous Global Interpreter Lock has been around for a while
-in the Python community. There has been various attempts at removing it
-(some successful ones, e.g. in Jython or IronPython with the help of the platform)
-and some yet to bear fruit, like `gilectomy`_. Since February sprint in Leysin,
+Discussions about the infamous Global Interpreter Lock have been around for a while
+in the Python community. There has been various attempts at removing it:
+some were successful, like e.g. in Jython or IronPython with the help of the platform, and some yet to bear fruit, like `gilectomy`_. Since our `February sprint`_ in Leysin,
 we've been on-and-off tackling the topic of GIL removal in the PyPy project.
 
-As of Europython announcement, we're able to run (very simple) programs with GIL-less
-PyPy that parallelizes nicely. The remaining 90% (and another 90%) of work
-is with putting locks in strategic places so PyPy does not segfault
-when you try to do a concurrent access to a data structure.
+.. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html
 
-Since such work would complicate the code base and our day to day work,
-we would like to judge the interest on the community and the commercial
-PyPy users.
+As we announced at EuroPython, what we have got so far is a GIL-less PyPy
+which can to run **very simple** multi-threaded programs which are nicely
+parallelized.  At the moment, non-simple programs most likely segfaults: the
+remaining 90% (and another 90%) of work is with putting locks in strategic
+places so PyPy does not segfault when you try to do a concurrent access to a
+data structure.
 
-We would like to do it in a following way. We are looking for a contract
-with companies (individual donations did not work very well for us in the
-past). We put the total cost of doing the work at $50k, out of which we
-already have backing for about 1/3. If we can get a $100k contract, we would
-make it our priority to deliver before the end of the year.
+.. antocuni: I'd simply remove the following paragraph. It's redundant, IMHO
+..
+   Since such work would complicate the code base and our day to day work,
+   we would like to judge the interest on the community and the commercial
+   PyPy users.
+
+We are looking for commercial partners to make it happen (individual donations
+did not work very well for us in the past). We estimate a total cost of $50k,
+out of which we already have backing for about 1/3. If we can get a $100k
+contract, we would make it our priority to deliver before the end of the year.
 
 People asked several questions, so I'll try to answer the technical parts
 here.

From pypy.commits at gmail.com  Fri Aug  4 06:22:40 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 03:22:40 -0700 (PDT)
Subject: [pypy-commit] pypy default: fix error message to not contain the
 internal name 'bigint'
Message-ID: <59844af0.4ad71c0a.ece00.492b@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92070:76adf2cfc536
Date: 2017-08-04 12:22 +0200
http://bitbucket.org/pypy/pypy/changeset/76adf2cfc536/

Log:	fix error message to not contain the internal name 'bigint'

diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -769,7 +769,9 @@
 
 class TypeCode(object):
     def __init__(self, itemtype, unwrap, canoverflow=False, signed=False,
-                 method='__int__'):
+                 method='__int__', errorname=None):
+        if errorname is None:
+            errorname = unwrap[:-2]
         self.itemtype = itemtype
         self.bytes = rffi.sizeof(itemtype)
         self.arraytype = lltype.Array(itemtype, hints={'nolength': True})
@@ -779,6 +781,7 @@
         self.canoverflow = canoverflow
         self.w_class = None
         self.method = method
+        self.errorname = errorname
 
     def _freeze_(self):
         # hint for the annotator: track individual constant instances
@@ -802,8 +805,8 @@
     'i': TypeCode(rffi.INT,           'int_w', True, True),
     'I': _UINTTypeCode,
     'l': TypeCode(rffi.LONG,          'int_w', True, True),
-    'L': TypeCode(rffi.ULONG,         'bigint_w'),  # Overflow handled by
-                                                    # rbigint.touint() which
+    'L': TypeCode(rffi.ULONG,         'bigint_w',   # Overflow handled by
+                  errorname="integer"),             # rbigint.touint() which
                                                     # corresponds to the
                                                     # C-type unsigned long
     'f': TypeCode(lltype.SingleFloat, 'float_w', method='__float__'),
@@ -881,7 +884,7 @@
                         item = unwrap(space.call_method(w_item, mytype.method))
                     except OperationError:
                         raise oefmt(space.w_TypeError,
-                                    "array item must be " + mytype.unwrap[:-2])
+                                    "array item must be " + mytype.errorname)
                 else:
                     raise
             if mytype.unwrap == 'bigint_w':
diff --git a/pypy/module/array/test/test_array.py b/pypy/module/array/test/test_array.py
--- a/pypy/module/array/test/test_array.py
+++ b/pypy/module/array/test/test_array.py
@@ -162,6 +162,11 @@
             raises(OverflowError, a.append, -1)
             raises(OverflowError, a.append, 2 ** (8 * b))
 
+    def test_errormessage(self):
+        a = self.array("L", [1, 2, 3])
+        excinfo = raises(TypeError, "a[0] = 'abc'")
+        assert str(excinfo.value) == "array item must be integer"
+
     def test_fromstring(self):
         import sys
 

From pypy.commits at gmail.com  Fri Aug  4 06:40:29 2017
From: pypy.commits at gmail.com (mattip)
Date: Fri, 04 Aug 2017 03:40:29 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-debug-type_dealloc: add debug hacks to
 find gc referrers in leak test
Message-ID: <59844f1d.11331c0a.c5a14.d471@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-debug-type_dealloc
Changeset: r92071:c5b852756f17
Date: 2017-08-04 13:38 +0300
http://bitbucket.org/pypy/pypy/changeset/c5b852756f17/

Log:	add debug hacks to find gc referrers in leak test

diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -13,6 +13,7 @@
 from rpython.tool import leakfinder
 from rpython.rlib import rawrefcount
 from rpython.tool.udir import udir
+import gc
 
 only_pypy ="config.option.runappdirect and '__pypy__' not in sys.builtin_module_names"
 
@@ -113,7 +114,8 @@
     return is_interned_string(space, w_obj)
 
 def _get_w_obj(space, c_obj):
-    return from_ref(space, cts.cast('PyObject*', c_obj._as_ptr()))
+    py_obj = cts.cast('PyObject*', c_obj._as_ptr())
+    return from_ref(space, py_obj), py_obj
 
 class CpyextLeak(leakfinder.MallocMismatch):
     def __str__(self):
@@ -122,9 +124,21 @@
             "These objects are attached to the following W_Root objects:")
         for c_obj in self.args[0]:
             try:
-                lines.append("  %s" % (_get_w_obj(self.args[1], c_obj),))
-            except:
-                pass
+                w_obj, py_obj = _get_w_obj(self.args[1], c_obj)
+                if w_obj and py_obj.c_ob_refcnt == rawrefcount.REFCNT_FROM_PYPY:
+                    referrers = gc.get_referrers(w_obj)
+                    badboys = []
+                    for r in referrers:
+                        # trap the tuples?
+                        if isinstance(r, tuple):
+                            badboys.append(r)
+                    # look at badboys, why are there tuples of
+                    # (type, function) ?
+                    import pdb;pdb.set_trace()
+                lines.append("  %s" % (w_obj,))
+            except Exception as e:
+                lines.append(str(e))
+                lines.append("  (no W_Root object)")
         return '\n'.join(lines)
 
 
diff --git a/pypy/module/cpyext/test/test_typeobject.py b/pypy/module/cpyext/test/test_typeobject.py
--- a/pypy/module/cpyext/test/test_typeobject.py
+++ b/pypy/module/cpyext/test/test_typeobject.py
@@ -586,8 +586,14 @@
         class E:
             pass
         # TODO C passes leak checker, D,E fails
+        # in debugging code in CpyextLeak, badboys has tuple/list
+        # (<W_TypeObject 'instance' at 0x7f45d3d11410>, <FunctionWithFixedCode __getattribute__>)
+        # (<W_TypeObject 'D' at 0x7f45d31b95d0>, <Function __init__>)
+        # (<W_TypeObject 'instance' at 0x7f45d3d11410>, <FunctionWithFixedCode __setattr__>)
+        # (<W_TypeObject 'instance' at 0x7f45d3d11410>, <FunctionWithFixedCode __nonzero__>)
+        # ...
+        assert module.test_leak(D())
         assert module.test_leak(C())
-        assert module.test_leak(D())
         assert module.test_leak(E())
     
     def test_tp_getattro(self):
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -540,7 +540,8 @@
     # w_obj is an instance of w_A or one of its subclasses. So climb up the
     # inheritance chain until base.c_tp_dealloc is exactly this_func, and then
     # continue on up until they differ.
-    print 'subtype_dealloc, start from', rffi.charp2str(base.c_tp_name)
+    name = rffi.charp2str(base.c_tp_name)
+    print 'subtype_dealloc, start from', name
     while base.c_tp_dealloc != this_func_ptr:
         base = base.c_tp_base
         assert base
@@ -915,7 +916,6 @@
     base = pto.c_tp_base
     base_pyo = rffi.cast(PyObject, pto.c_tp_base)
     if base and not base.c_tp_flags & Py_TPFLAGS_READY:
-        name = rffi.charp2str(cts.cast('char*', base.c_tp_name))
         type_realize(space, base_pyo)
     if base and not pto.c_ob_type: # will be filled later
         pto.c_ob_type = base.c_ob_type

From pypy.commits at gmail.com  Fri Aug  4 06:40:31 2017
From: pypy.commits at gmail.com (mattip)
Date: Fri, 04 Aug 2017 03:40:31 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-debug-type_dealloc: maybe holding
 references from the SLOTS dictionary? it seems not ...
Message-ID: <59844f1f.cf97df0a.f8a92.bd74@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: cpyext-debug-type_dealloc
Changeset: r92072:7bd0d7e8d934
Date: 2017-08-04 13:39 +0300
http://bitbucket.org/pypy/pypy/changeset/7bd0d7e8d934/

Log:	maybe holding references from the SLOTS dictionary? it seems not ...

diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -61,12 +61,31 @@
 def _dealloc(space, obj):
     # This frees an object after its refcount dropped to zero, so we
     # assert that it is really zero here.
+    from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
     assert obj.c_ob_refcnt == 0
     pto = obj.c_ob_type
     obj_voidp = rffi.cast(rffi.VOIDP, obj)
     generic_cpy_call(space, pto.c_tp_free, obj_voidp)
     if pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE:
+        from pypy.module.cpyext.slotdefs import SLOTS
+        w_obj = from_ref(space, pto)
+        if pto.c_ob_refcnt <= REFCNT_FROM_PYPY + 1:
+            # remove the to-be-deleted type's functions from slotdef.SLOTS
+            toremove = []
+            typedef = w_obj.layout.typedef
+            for k in SLOTS.keys():
+                if k[0] is typedef:
+                    toremove.append(k)
+            for k in toremove:
+                SLOTS.pop(k)
+                print w_obj.name, k[1]
+        else:
+            print 'not releasing', w_obj.name, 'w/refcnt',pto.c_ob_refcnt - REFCNT_FROM_PYPY 
         Py_DecRef(space, rffi.cast(PyObject, pto))
+    else:
+        w_obj = from_ref(space, pto)
+        print 'no decref', w_obj.name, 'w/refcnt',pto.c_ob_refcnt - REFCNT_FROM_PYPY 
+        
 
 @cpython_api([PyTypeObjectPtr], PyObject, result_is_ll=True)
 def _PyObject_GC_New(space, type):
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -291,7 +291,7 @@
 
         if not slot_func_helper:
             if typedef is not None:
-                if slot_apifunc is None:
+                if slot_apifunc is None and slot_name in ('tp_new',):
                     slot_apifunc = get_slot_tp_function(space, typedef, slot_name)
             if not slot_apifunc:
                 if not we_are_translated():

From pypy.commits at gmail.com  Fri Aug  4 07:44:30 2017
From: pypy.commits at gmail.com (tobweber)
Date: Fri, 04 Aug 2017 04:44:30 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-binary-trx-length-per-thread: Fix missing
 type definitions for custom payload
Message-ID: <59845e1e.d1161c0a.22f23.c136@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-binary-trx-length-per-thread
Changeset: r2150:3394aed50b06
Date: 2017-07-14 20:27 +0200
http://bitbucket.org/pypy/stmgc/changeset/3394aed50b06/

Log:	Fix missing type definitions for custom payload

diff --git a/c8/stm/timing.h b/c8/stm/timing.h
--- a/c8/stm/timing.h
+++ b/c8/stm/timing.h
@@ -1,5 +1,9 @@
 #include <time.h>
 
+#define define_payload_types()                                              \
+    stm_timing_event_payload_data_t stm_duration_data;                      \
+    stm_timing_event_payload_t stm_duration_payload;
+
 #define continue_timer() clock_gettime(CLOCK_MONOTONIC_RAW, &start);
 
 /* Use raw monotonic time, i.e., solely based on local hardware (no NTP
@@ -8,8 +12,7 @@
 #define start_timer() struct timespec start, stop;                             \
                       struct timespec duration = { .tv_sec = 0, .tv_nsec = 0 };\
                       uint32_t nanosec_diff, sec_diff;                         \
-                      stm_timing_event_payload_data_t stm_duration_data;       \
-                      stm_timing_event_payload_t stm_duration_payload;         \
+                      define_payload_types()                                   \
                       continue_timer()
 
 /* Must use start_timer before using this macro. */
@@ -59,5 +62,6 @@
 
 #define publish_custom_value_event(double_value, event)                     \
     set_payload((double_value))                                             \
+    define_payload_types()                                                  \
     stm_duration_payload(payload_value);                                    \
     publish_event(STM_SEGMENT->running_thread, (event))

From pypy.commits at gmail.com  Fri Aug  4 08:51:44 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 05:51:44 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: merge
Message-ID: <59846de0.08811c0a.3cd95.84d3@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: py3.5
Changeset: r92074:28233e67ad80
Date: 2017-08-04 14:50 +0200
http://bitbucket.org/pypy/pypy/changeset/28233e67ad80/

Log:	merge

diff --git a/lib-python/3/stat.py b/lib-python/3/stat.py
--- a/lib-python/3/stat.py
+++ b/lib-python/3/stat.py
@@ -139,13 +139,21 @@
 def filemode(mode):
     """Convert a file's mode to a string of the form '-rwxrwxrwx'."""
     perm = []
+
+    # The first group gets a question mark if none of the bits match the mode.
+    empty = "?"
+
     for table in _filemode_table:
         for bit, char in table:
             if mode & bit == bit:
                 perm.append(char)
                 break
         else:
-            perm.append("-")
+            perm.append(empty)
+
+        # All the rest of the positions get a - if the bits don't match.
+        empty = "-"
+
     return "".join(perm)
 
 
diff --git a/lib-python/3/test/test_stat.py b/lib-python/3/test/test_stat.py
--- a/lib-python/3/test/test_stat.py
+++ b/lib-python/3/test/test_stat.py
@@ -138,6 +138,10 @@
             self.assertS_IS("REG", st_mode)
             self.assertEqual(modestr, '-r--r--r--')
             self.assertEqual(self.statmod.S_IMODE(st_mode), 0o444)
+
+            # If there are only permission bits, no type bytes, a question
+            # mark is rendered in the type field.
+            self.assertEqual(self.statmod.filemode(0o420), '?r---w----')
         else:
             os.chmod(TESTFN, 0o700)
             st_mode, modestr = self.get_mode()

From pypy.commits at gmail.com  Fri Aug  4 08:51:42 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 05:51:42 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: merge default
Message-ID: <59846dde.cc361c0a.19209.3a0f@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: py3.5
Changeset: r92073:5d4625159d48
Date: 2017-08-04 14:50 +0200
http://bitbucket.org/pypy/pypy/changeset/5d4625159d48/

Log:	merge default

diff too long, truncating to 2000 out of 2201 lines

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -227,11 +227,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -291,7 +286,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
@@ -140,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -169,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -205,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -64,9 +64,9 @@
 cppyy
 -----
 
-For C++, `cppyy`_ is an automated bindings generator available for both
+For C++, _cppyy_ is an automated bindings generator available for both
 PyPy and CPython.
-``cppyy`` relies on declarations from C++ header files to dynamically
+_cppyy_ relies on declarations from C++ header files to dynamically
 construct Python equivalent classes, functions, variables, etc.
 It is designed for use by large scale programs and supports modern C++.
 With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
@@ -75,8 +75,7 @@
 To install, run ``pip install cppyy``.
 Further details are available in the `full documentation`_.
 
-.. _cppyy: http://cppyy.readthedocs.org/
-.. _`full documentation`: http://cppyy.readthedocs.org/
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
 
 RPython Mixed Modules
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -38,3 +38,25 @@
 
 Renaming of ``cppyy`` to ``_cppyy``.
 The former is now an external package installable with ``pip install cppyy``.
+
+.. branch: Enable_PGO_for_clang
+
+.. branch: nopax
+
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
+
+.. branch: pypy_bytearray
+
+Improve ``bytearray`` performance (backported from py3.5)
+
+.. branch: gc-del-limit-growth
+
+Fix the bounds in the GC when allocating a lot of objects with finalizers,
+fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import enforceargs
 from rpython.rlib.rstring import StringBuilder
@@ -48,8 +49,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,7 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
-from rpython.rlib.objectmodel import dont_inline
+from rpython.rlib.objectmodel import dont_inline, not_rpython
 from rpython.rlib import rstack, rstackovf
 from rpython.rlib import rwin32
 from rpython.rlib import runicode
@@ -65,8 +65,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -119,15 +120,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -154,8 +156,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,6 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -410,8 +411,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -75,8 +75,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -399,8 +399,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -654,9 +654,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -1004,10 +1004,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1047,8 +1047,8 @@
 
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         alldefs_w = {}
         assert len(self._code._argnames) == len(self._code._unwrap_spec)
         for name, spec in zip(self._code._argnames, self._code._unwrap_spec):
@@ -1124,8 +1124,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs_w, kw_defs_w = gateway._getdefaults(space)
         code = gateway._code
@@ -1196,8 +1196,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1234,15 +1234,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1253,8 +1254,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1300,6 +1302,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,8 +18,8 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         init_extra_module_attrs(space, self)
         self.lazy = True
@@ -25,8 +28,9 @@
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -66,8 +70,8 @@
         self.w_initialdict = self.space.call_method(w_dict, 'copy')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -163,8 +167,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -194,8 +198,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space': space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -235,8 +239,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError, oefmt
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -35,8 +35,9 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         if modulename in self.space.builtin_modules:
             raise ValueError(
@@ -44,8 +45,9 @@
                 "app-level module %r" % (modulename,))
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -146,8 +147,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf, rstring
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (
-    we_are_translated, always_inline, dont_inline)
+    we_are_translated, always_inline, dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -23,8 +23,8 @@
 CANNOT_CATCH_MSG = ("catching classes that don't inherit from BaseException "
                     "is not allowed in 3.x")
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -34,8 +34,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -8,15 +8,16 @@
 
 from rpython.rlib.jit import promote
 from rpython.rlib.objectmodel import compute_identity_hash, specialize
-from rpython.rlib.objectmodel import instantiate
+from rpython.rlib.objectmodel import instantiate, not_rpython
 from rpython.tool.sourcetools import compile2, func_with_new_name
 
 
 class TypeDef(object):
+    @not_rpython
     def __init__(self, __name, __base=None, __total_ordering__=None,
                  __buffer=None, __confirm_applevel_del__=False,
                  variable_sized=False, **rawdict):
-        "NOT_RPYTHON: initialization-time only"
+        "initialization-time only"
         self.name = __name
         if __base is None:
             bases = []
@@ -116,8 +117,9 @@
 # register_finalizer() or not.
 
 @specialize.memo()
+ at not_rpython
 def get_unique_interplevel_subclass(space, cls):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert cls.typedef.acceptable_as_base_class
     try:
         return _unique_subclass_cache[cls]
@@ -367,15 +369,17 @@
         return self
 
 
+ at not_rpython
 def interp_attrproperty(name, cls, doc=None, wrapfn=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert wrapfn is not None
     def fget(space, obj):
         return getattr(space, wrapfn)(getattr(obj, name))
     return GetSetProperty(fget, cls=cls, doc=doc)
 
+ at not_rpython
 def interp_attrproperty_w(name, cls, doc=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     def fget(space, obj):
         w_value = getattr(obj, name)
         if w_value is None:
diff --git a/pypy/module/_cffi_backend/cdataobj.py b/pypy/module/_cffi_backend/cdataobj.py
--- a/pypy/module/_cffi_backend/cdataobj.py
+++ b/pypy/module/_cffi_backend/cdataobj.py
@@ -433,17 +433,22 @@
     def _sizeof(self):
         return self.ctype.size
 
-    def with_gc(self, w_destructor):
+    def with_gc(self, w_destructor, size=0):
         space = self.space
         if space.is_none(w_destructor):
             if isinstance(self, W_CDataGCP):
                 self.detach_destructor()
-                return space.w_None
-            raise oefmt(space.w_TypeError,
-                        "Can remove destructor only on a object "
-                        "previously returned by ffi.gc()")
-        with self as ptr:
-            return W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+                w_res = space.w_None
+            else:
+                raise oefmt(space.w_TypeError,
+                            "Can remove destructor only on a object "
+                            "previously returned by ffi.gc()")
+        else:
+            with self as ptr:
+                w_res = W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+        if size != 0:
+            rgc.add_memory_pressure(size)
+        return w_res
 
     def unpack(self, length):
         from pypy.module._cffi_backend.ctypeptr import W_CTypePtrOrArray
diff --git a/pypy/module/_cffi_backend/ffi_obj.py b/pypy/module/_cffi_backend/ffi_obj.py
--- a/pypy/module/_cffi_backend/ffi_obj.py
+++ b/pypy/module/_cffi_backend/ffi_obj.py
@@ -351,14 +351,14 @@
         return handle.from_handle(self.space, w_arg)
 
 
-    @unwrap_spec(w_cdata=W_CData)
-    def descr_gc(self, w_cdata, w_destructor):
+    @unwrap_spec(w_cdata=W_CData, size=int)
+    def descr_gc(self, w_cdata, w_destructor, size=0):
         """\
 Return a new cdata object that points to the same data.
 Later, when this new cdata object is garbage-collected,
 'destructor(old_cdata_object)' will be called."""
         #
-        return w_cdata.with_gc(w_destructor)
+        return w_cdata.with_gc(w_destructor, size)
 
 
     @unwrap_spec(replace_with='text')
diff --git a/pypy/module/_cffi_backend/func.py b/pypy/module/_cffi_backend/func.py
--- a/pypy/module/_cffi_backend/func.py
+++ b/pypy/module/_cffi_backend/func.py
@@ -257,6 +257,6 @@
 
 # ____________________________________________________________
 
- at unwrap_spec(w_cdata=cdataobj.W_CData)
-def gcp(space, w_cdata, w_destructor):
-    return w_cdata.with_gc(w_destructor)
+ at unwrap_spec(w_cdata=cdataobj.W_CData, size=int)
+def gcp(space, w_cdata, w_destructor, size=0):
+    return w_cdata.with_gc(w_destructor, size)
diff --git a/pypy/module/_cffi_backend/test/test_ffi_obj.py b/pypy/module/_cffi_backend/test/test_ffi_obj.py
--- a/pypy/module/_cffi_backend/test/test_ffi_obj.py
+++ b/pypy/module/_cffi_backend/test/test_ffi_obj.py
@@ -377,7 +377,7 @@
         raises(TypeError, ffi.gc, p, None)
         seen = []
         q1 = ffi.gc(p, lambda p: seen.append(1))
-        q2 = ffi.gc(q1, lambda p: seen.append(2))
+        q2 = ffi.gc(q1, lambda p: seen.append(2), size=123)
         import gc; gc.collect()
         assert seen == []
         assert ffi.gc(q1, None) is None
diff --git a/pypy/module/_codecs/__init__.py b/pypy/module/_codecs/__init__.py
--- a/pypy/module/_codecs/__init__.py
+++ b/pypy/module/_codecs/__init__.py
@@ -1,5 +1,6 @@
 from pypy.interpreter.mixedmodule import MixedModule
 from rpython.rlib import runicode
+from rpython.rlib.objectmodel import not_rpython
 from pypy.module._codecs import interp_codecs
 
 class Module(MixedModule):
@@ -85,9 +86,8 @@
          'unicode_internal_encode'   :  'interp_codecs.unicode_internal_encode',
     }
 
+    @not_rpython
     def __init__(self, space, *args):
-        "NOT_RPYTHON"
-
         # mbcs codec is Windows specific, and based on rffi.
         if (hasattr(runicode, 'str_decode_mbcs')):
             self.interpleveldefs['mbcs_encode'] = 'interp_codecs.mbcs_encode'
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -1,6 +1,6 @@
 import sys
 from rpython.rlib import jit
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rstring import UnicodeBuilder, StringBuilder
 from rpython.rlib.runicode import (
     code_to_unichr, MAXUNICODE,
@@ -477,8 +477,8 @@
         raise oefmt(space.w_TypeError,
                     "don't know how to handle %T in error callback", w_exc)
 
+ at not_rpython
 def register_builtin_error_handlers(space):
-    "NOT_RPYTHON"
     state = space.fromcache(CodecState)
     for error in ("strict", "ignore", "replace", "xmlcharrefreplace",
                   "backslashreplace", "surrogateescape", "surrogatepass",
diff --git a/pypy/module/_vmprof/test/test__vmprof.py b/pypy/module/_vmprof/test/test__vmprof.py
--- a/pypy/module/_vmprof/test/test__vmprof.py
+++ b/pypy/module/_vmprof/test/test__vmprof.py
@@ -1,3 +1,4 @@
+import sys
 from rpython.tool.udir import udir
 from pypy.tool.pytest.objspace import gettestobjspace
 
@@ -7,6 +8,8 @@
     def setup_class(cls):
         cls.w_tmpfilename = cls.space.wrap(str(udir.join('test__vmprof.1')))
         cls.w_tmpfilename2 = cls.space.wrap(str(udir.join('test__vmprof.2')))
+        cls.w_plain = cls.space.wrap(not cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
 
     def test_import_vmprof(self):
         tmpfile = open(self.tmpfilename, 'wb')
@@ -120,6 +123,8 @@
         assert _vmprof.get_profile_path() is None
 
     def test_stop_sampling(self):
+        if not self.plain:
+            skip("unreliable test except on CPython without -A")
         import os
         import _vmprof
         tmpfile = open(self.tmpfilename, 'wb')
diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -138,6 +138,29 @@
         return space.w_True
     return space.w_False
 
+index_count_jd = jit.JitDriver(
+    greens = ['count', 'arrclass', 'tp_item'],
+    reds = 'auto', name = 'array.index_or_count')
+
+def index_count_array(arr, w_val, count=False):
+    space = arr.space
+    tp_item = space.type(w_val)
+    arrclass = arr.__class__
+    cnt = 0
+    for i in range(arr.len):
+        index_count_jd.jit_merge_point(
+            tp_item=tp_item, count=count,
+            arrclass=arrclass)
+        w_item = arr.w_getitem(space, i)
+        if space.eq_w(w_item, w_val):
+            if count:
+                cnt += 1
+            else:
+                return i
+    if count:
+        return cnt
+    return -1
+
 UNICODE_ARRAY = lltype.Ptr(lltype.Array(lltype.UniChar,
                                         hints={'nolength': True}))
 
@@ -274,17 +297,12 @@
         """
         self.extend(w_x)
 
-    def descr_count(self, space, w_val):
+    def descr_count(self, space, w_x):
         """ count(x)
 
         Return number of occurrences of x in the array.
         """
-        cnt = 0
-        for i in range(self.len):
-            # XXX jitdriver
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_val):
-                cnt += 1
+        cnt = index_count_array(self, w_x, count=True)
         return space.newint(cnt)
 
     def descr_index(self, space, w_x):
@@ -292,10 +310,9 @@
 
         Return index of first occurrence of x in the array.
         """
-        for i in range(self.len):
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_x):
-                return space.newint(i)
+        res = index_count_array(self, w_x, count=False)
+        if res >= 0:
+            return space.newint(res)
         raise oefmt(space.w_ValueError, "array.index(x): x not in list")
 
     def descr_reverse(self, space):
@@ -801,7 +818,9 @@
 
 class TypeCode(object):
     def __init__(self, itemtype, unwrap, canoverflow=False, signed=False,
-                 method='__int__'):
+                 method='__int__', errorname=None):
+        if errorname is None:
+            errorname = unwrap[:-2]
         self.itemtype = itemtype
         self.bytes = rffi.sizeof(itemtype)
         self.arraytype = lltype.Array(itemtype, hints={'nolength': True})
@@ -812,6 +831,7 @@
         self.canoverflow = canoverflow
         self.w_class = None
         self.method = method
+        self.errorname = errorname
 
     def _freeze_(self):
         # hint for the annotator: track individual constant instances
@@ -838,9 +858,9 @@
     'i': TypeCode(rffi.INT,           'int_w', True, True),
     'I': _UINTTypeCode,
     'l': TypeCode(rffi.LONG,          'int_w', True, True),
-    'L': TypeCode(rffi.ULONG,         'bigint_w.touint'),
-    'q': TypeCode(rffi.LONGLONG,      'bigint_w.tolonglong', True, True),
-    'Q': TypeCode(rffi.ULONGLONG,     'bigint_w.toulonglong', True),
+    'L': TypeCode(rffi.ULONG,         'bigint_w.touint', errorname="integer"),
+    'q': TypeCode(rffi.LONGLONG,      'bigint_w.tolonglong', True, True, errorname="integer"),
+    'Q': TypeCode(rffi.ULONGLONG,     'bigint_w.toulonglong', True, errorname="integer"),
     'f': TypeCode(lltype.SingleFloat, 'float_w', method='__float__'),
     'd': TypeCode(lltype.Float,       'float_w', method='__float__'),
     }
@@ -970,7 +990,7 @@
                     except OperationError as e:
                         if e.async(space):
                             raise
-                        msg = "array item must be " + mytype.unwrap[:-2]
+                        msg = "array item must be " + mytype.errorname
                         raise OperationError(space.w_TypeError,
                                              space.newtext(msg))
                 else:
diff --git a/pypy/module/array/test/test_array.py b/pypy/module/array/test/test_array.py
--- a/pypy/module/array/test/test_array.py
+++ b/pypy/module/array/test/test_array.py
@@ -144,6 +144,11 @@
             raises(OverflowError, a.append, -1)
             raises(OverflowError, a.append, 2 ** (8 * b))
 
+    def test_errormessage(self):
+        a = self.array("L", [1, 2, 3])
+        excinfo = raises(TypeError, "a[0] = 'abc'")
+        assert str(excinfo.value) == "array item must be integer"
+
     def test_fromstring(self):
         a = self.array('b')
         a.fromstring('Hi!')
diff --git a/pypy/module/cppyy/test/test_cint.py b/pypy/module/cppyy/test/test_cint.py
deleted file mode 100644
--- a/pypy/module/cppyy/test/test_cint.py
+++ /dev/null
@@ -1,710 +0,0 @@
-import py, os, sys
-
-# These tests are for the CINT backend only (they exercise ROOT features
-# and classes that are not loaded/available with the Reflex backend). At
-# some point, these tests are likely covered by the CLang/LLVM backend.
-from pypy.module.cppyy import capi
-if capi.identify() != 'CINT':
-    py.test.skip("backend-specific: CINT-only tests")
-
-# load _cffi_backend early, or its global vars are counted as leaks in the
-# test (note that the module is not otherwise used in the test itself)
-from pypy.module._cffi_backend import newtype
-
-currpath = py.path.local(__file__).dirpath()
-iotypes_dct = str(currpath.join("iotypesDict.so"))
-
-def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make CINT=t iotypesDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
-
-class AppTestCINT:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_globals(self):
-        """Test the availability of ROOT globals"""
-
-        import cppyy
-
-        assert cppyy.gbl.gROOT
-        assert cppyy.gbl.gApplication
-        assert cppyy.gbl.gSystem
-        assert cppyy.gbl.TInterpreter.Instance()           # compiled
-        assert cppyy.gbl.TInterpreter                      # interpreted
-        assert cppyy.gbl.TDirectory.CurrentDirectory()     # compiled
-        assert cppyy.gbl.TDirectory                        # interpreted
-
-    def test02_write_access_to_globals(self):
-        """Test overwritability of ROOT globals"""
-
-        import cppyy
-
-        oldval = cppyy.gbl.gDebug
-        assert oldval != 3
-
-        proxy = cppyy.gbl.__class__.__dict__['gDebug']
-        cppyy.gbl.gDebug = 3
-        assert proxy.__get__(proxy, None) == 3
-
-        # this is where this test differs from test03_write_access_to_globals
-        # in test_pythonify.py
-        cppyy.gbl.gROOT.ProcessLine('int gDebugCopy = gDebug;')
-        assert cppyy.gbl.gDebugCopy == 3
-
-        cppyy.gbl.gDebug = oldval
-
-    def test03_create_access_to_globals(self):
-        """Test creation and access of new ROOT globals"""
-
-        import cppyy
-
-        cppyy.gbl.gROOT.ProcessLine('double gMyOwnGlobal = 3.1415')
-        assert cppyy.gbl.gMyOwnGlobal == 3.1415
-
-        proxy = cppyy.gbl.__class__.__dict__['gMyOwnGlobal']
-        assert proxy.__get__(proxy, None) == 3.1415
-
-    def test04_auto_loading(self):
-        """Test auto-loading by retrieving a non-preloaded class"""
-
-        import cppyy
-
-        l = cppyy.gbl.TLorentzVector()
-        assert isinstance(l, cppyy.gbl.TLorentzVector)
-
-    def test05_macro_loading(self):
-        """Test accessibility to macro classes"""
-
-        import cppyy
-
-        loadres = cppyy.gbl.gROOT.LoadMacro('simple_class.C')
-        assert loadres == 0
-
-        base = cppyy.gbl.MySimpleBase
-        simple = cppyy.gbl.MySimpleDerived
-        simple_t = cppyy.gbl.MySimpleDerived_t
-
-        assert issubclass(simple, base)
-        assert simple is simple_t
-
-        c = simple()
-        assert isinstance(c, simple)
-        assert c.m_data == c.get_data()
-
-        c.set_data(13)
-        assert c.m_data == 13
-        assert c.get_data() == 13
-
-
-class AppTestCINTPYTHONIZATIONS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_strings(self):
-        """Test TString/TObjString compatibility"""
-
-        import cppyy
-
-        pyteststr = "aap noot mies"
-        def test_string(s1, s2):
-            assert len(s1) == len(s2)
-            assert s1 == s1
-            assert s1 == s2
-            assert s1 == str(s1)
-            assert s1 == pyteststr
-            assert s1 != "aap"
-            assert s1 != ""
-            assert s1 < "noot"
-            assert repr(s1) == repr(s2)
-
-        s1 = cppyy.gbl.TString(pyteststr)
-        test_string(s1, pyteststr)
-
-        s3 = cppyy.gbl.TObjString(pyteststr)
-        test_string(s3, pyteststr)
-
-    def test03_TVector(self):
-        """Test TVector2/3/T behavior"""
-
-        import cppyy, math
-
-        N = 51
-
-        # TVectorF is a typedef of floats
-        v = cppyy.gbl.TVectorF(N)
-        for i in range(N):
-            v[i] = i*i
-
-        assert len(v) == N
-        for j in v:
-            assert round(v[int(math.sqrt(j)+0.5)]-j, 5) == 0.
-
-    def test04_TStringTObjString(self):
-        """Test string/TString interchangebility"""
-
-        import cppyy
-
-        test = "aap noot mies"
-
-        s1 = cppyy.gbl.TString(test )
-        s2 = str(s1)
-
-        assert s1 == test
-        assert test == s2
-        assert s1 == s2
-
-        s3 = cppyy.gbl.TObjString(s2)
-        assert s3 == test
-        assert s2 == s3
-
-        # force use of: TNamed(const TString &name, const TString &title)
-        n = cppyy.gbl.TNamed(test, cppyy.gbl.TString("title"))
-        assert n.GetTitle() == "title"
-        assert n.GetName() == test
-
-
-class AppTestCINTTTREE:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def setup_class(cls):
-        cls.w_N = cls.space.newint(5)
-        cls.w_M = cls.space.newint(10)
-        cls.w_fname = cls.space.newtext("test.root")
-        cls.w_tname = cls.space.newtext("test")
-        cls.w_title = cls.space.newtext("test tree")
-        cls.w_iotypes = cls.space.appexec([], """():
-            import cppyy
-            return cppyy.load_reflection_info(%r)""" % (iotypes_dct,))
-
-    def test01_write_stdvector(self):
-        """Test writing of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        v = vector("double")()
-        raises(TypeError, TTree.Branch, None, "mydata", v.__class__.__name__, v)
-        raises(TypeError, TTree.Branch, v, "mydata", v.__class__.__name__, v)
-
-        mytree.Branch("mydata", v.__class__.__name__, v)
-
-        for i in range(self.N):
-            for j in range(self.M):
-                v.push_back(i*self.M+j)
-            mytree.Fill()
-            v.clear()
-        f.Write()
-        f.Close()
-
-    def test02_file_open(self):
-
-        from cppyy import gbl
-
-        f = gbl.TFile.Open(self.fname)
-        s = str(f)            # should not raise
-        r = repr(f)
-
-        f.Close()
-
-    def test03_read_stdvector(self):
-        """Test reading of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        i = 0
-        for event in mytree:
-            assert len(event.mydata) == self.M
-            for entry in event.mydata:
-                assert i == int(entry)
-                i += 1
-        assert i == self.N * self.M
-
-        f.Close()
-
-    def test04_write_some_data_object(self):
-        """Test writing of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree, IO
-        from cppyy.gbl.IO import SomeDataObject
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-
-        d = SomeDataObject()
-        b = mytree.Branch("data", d)
-        mytree._python_owns = False
-        assert b
-
-        for i in range(self.N):
-            for j in range(self.M):
-                d.add_float(i*self.M+j)
-            d.add_tuple(d.get_floats())
-
-            mytree.Fill()
-
-        f.Write()
-        f.Close()
-
-    def test05_read_some_data_object(self):
-        """Test reading of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        j = 1
-        for event in mytree:
-            i = 0
-            assert len(event.data.get_floats()) == j*self.M
-            for entry in event.data.get_floats():
-                assert i == int(entry)
-                i += 1
-
-            k = 1
-            assert len(event.data.get_tuples()) == j
-            for mytuple in event.data.get_tuples():
-                i = 0
-                assert len(mytuple) == k*self.M
-                for entry in mytuple:
-                    assert i == int(entry)
-                    i += 1
-                k += 1
-            j += 1
-        assert j-1 == self.N
-        #
-        f.Close()
-
-    def test06_branch_activation(self):
-        """Test of automatic branch activation"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        L = 5
-
-        # writing
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        for i in range(L):
-            v = vector("double")()
-            mytree.Branch("mydata_%d"%i, v.__class__.__name__, v)
-            mytree.__dict__["v_%d"%i] = v
-
-        for i in range(self.N):
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                for j in range(self.M):
-                    mytree.__dict__["v_%d"%k].push_back(i*self.M+j*L+k)
-            mytree.Fill()
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                v.clear()
-        f.Write()
-        f.Close()
-
-        del mytree, f
-        import gc
-        gc.collect()
-
-        # reading
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # force (initial) disabling of all branches
-        mytree.SetBranchStatus("*",0);
-
-        i = 0
-        for event in mytree:
-            for k in range(L):
-                j = 0
-                data = getattr(mytree, "mydata_%d"%k)
-                assert len(data) == self.M
-                for entry in data:
-                    assert entry == i*self.M+j*L+k
-                    j += 1
-                assert j == self.M
-            i += 1
-        assert i == self.N
-
-        f.Close()
-
-    def test07_write_builtin(self):
-        """Test writing of builtins"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        import array
-        mytree.ba = array.array('c', [chr(0)])
-        mytree.ia = array.array('i', [0])
-        mytree.da = array.array('d', [0.])
-
-        mytree.Branch("my_bool",   mytree.ba, "my_bool/O")
-        mytree.Branch("my_int",    mytree.ia, "my_int/I")
-        mytree.Branch("my_int2",   mytree.ia, "my_int2/I")
-        mytree.Branch("my_double", mytree.da, "my_double/D")
-
-        for i in range(self.N):
-            # make sure value is different from default (0)
-            mytree.ba[0] = i%2 and chr(0) or chr(1)
-            mytree.ia[0] = i+1
-            mytree.da[0] = (i+1)/2.
-            mytree.Fill()
-        f.Write()
-        f.Close()
-
-    def test08_read_builtin(self):
-        """Test reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        raises(AttributeError, getattr, mytree, "does_not_exist")
-
-        i = 1
-        for event in mytree:
-            assert event.my_bool   == (i-1)%2 and 0 or 1
-            assert event.my_int    == i
-            assert event.my_double == i/2.
-            i += 1
-        assert (i-1) == self.N
-
-        f.Close()
-
-    def test09_user_read_builtin(self):
-        """Test user-directed reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # note, this is an old, annoted tree from test08
-        for i in range(3, mytree.GetEntriesFast()):
-            mytree.GetEntry(i)
-            assert mytree.my_int  == i+1
-            assert mytree.my_int2 == i+1
-
-        f.Close()
-
-class AppTestCINTREGRESSION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that at some point in the past resulted in failures on
-    # PyROOT; kept here to confirm no regression from PyROOT
-
-    def test01_regression(self):
-        """TPaveText::AddText() used to result in KeyError"""
-
-        # This is where the original problem was discovered, and the test is
-        # left in. However, the detailed underlying problem, as well as the
-        # solution to it, is tested in test_fragile.py
-
-        from cppyy import gbl
-        from cppyy.gbl import TPaveText
-
-        hello = TPaveText( .1, .8, .9, .97 )
-        hello.AddText( 'Hello, World!' )
-
-
-class AppTestCINTFUNCTION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-    _pypytest_leaks = None   # TODO: figure out the false positives
-
-    # test the function callbacks; this does not work with Reflex, as it can
-    # not generate functions on the fly (it might with cffi?)
-
-    @py.test.mark.dont_track_allocations("TODO: understand; initialization left-over?")
-    def test01_global_function_callback(self):
-        """Test callback of a python global function"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        def identity(x):
-            return x[0]
-
-        f = TF1("pyf1", identity, -1., 1., 0)
-
-        assert f.Eval(0.5)  == 0.5
-        assert f.Eval(-10.) == -10.
-        assert f.Eval(1.0)  == 1.0
-
-        # check proper propagation of default value
-        f = TF1("pyf1d", identity, -1., 1.)
-
-        assert f.Eval(0.5) == 0.5
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test02_callable_object_callback(self):
-        """Test callback of a python callable object"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        class Linear:
-            def __call__(self, x, par):
-                return par[0] + x[0]*par[1]
-
-        f = TF1("pyf2", Linear(), -1., 1., 2)
-        f.SetParameters(5., 2.)
-
-        assert f.Eval(-0.1) == 4.8
-        assert f.Eval(1.3)  == 7.6
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test03_fit_with_python_gaussian(self):
-        """Test fitting with a python global function"""
-
-        # note: this function is dread-fully slow when running testing un-translated
-
-        import cppyy, gc, math
-        TF1, TH1F = cppyy.gbl.TF1, cppyy.gbl.TH1F
-
-        def pygaus(x, par):
-            arg1 = 0
-            scale1 = 0
-            ddx = 0.01
-
-            if (par[2] != 0.0):
-                arg1 = (x[0]-par[1])/par[2]
-                scale1 = (ddx*0.39894228)/par[2]
-                h1 = par[0]/(1+par[3])
-
-                gauss = h1*scale1*math.exp(-0.5*arg1*arg1)
-            else:
-                gauss = 0.
-            return gauss
-
-        f = TF1("pygaus", pygaus, -4, 4, 4)
-        f.SetParameters(600, 0.43, 0.35, 600)
-
-        h = TH1F("h", "test", 100, -4, 4)
-        h.FillRandom("gaus", 200000)
-        h.Fit(f, "0Q")
-
-        assert f.GetNDF() == 96
-        result = f.GetParameters()
-        assert round(result[1] - 0., 1) == 0  # mean
-        assert round(result[2] - 1., 1) == 0  # s.d.
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-
-class AppTestSURPLUS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that were historically exercised on ROOT classes and
-    # have twins on custom classes; kept here just in case differences crop
-    # up between the ROOT classes and the custom ones
-
-    def test01_class_enum(self):
-        """Test class enum access and values"""
-
-        import cppyy
-        TObject = cppyy.gbl.TObject
-        gROOT = cppyy.gbl.gROOT
-
-        assert TObject.kBitMask    == gROOT.ProcessLine("return TObject::kBitMask;")
-        assert TObject.kIsOnHeap   == gROOT.ProcessLine("return TObject::kIsOnHeap;")
-        assert TObject.kNotDeleted == gROOT.ProcessLine("return TObject::kNotDeleted;")
-        assert TObject.kZombie     == gROOT.ProcessLine("return TObject::kZombie;")
-
-        t = TObject()
-
-        assert TObject.kBitMask    == t.kBitMask
-        assert TObject.kIsOnHeap   == t.kIsOnHeap
-        assert TObject.kNotDeleted == t.kNotDeleted
-        assert TObject.kZombie     == t.kZombie
-
-    def test02_global_enum(self):
-        """Test global enums access and values"""
-
-        import cppyy
-        from cppyy import gbl
-
-        assert gbl.kRed   == gbl.gROOT.ProcessLine("return kRed;")
-        assert gbl.kGreen == gbl.gROOT.ProcessLine("return kGreen;")
-        assert gbl.kBlue  == gbl.gROOT.ProcessLine("return kBlue;")
-
-    def test03_copy_contructor(self):
-        """Test copy constructor"""
-
-        import cppyy
-        TLorentzVector = cppyy.gbl.TLorentzVector
-
-        t1 = TLorentzVector(1., 2., 3., -4.)
-        t2 = TLorentzVector(0., 0., 0.,  0.)
-        t3 = TLorentzVector(t1)
-
-        assert t1 == t3
-        assert t1 != t2
-
-        for i in range(4):
-            assert t1[i] == t3[i]
-
-    def test04_object_validity(self):
-        """Test object validity checking"""
-
-        import cppyy
-
-        t1 = cppyy.gbl.TObject()
-
-        assert t1
-        assert not not t1
-
-        t2 = cppyy.gbl.gROOT.FindObject("Nah, I don't exist")
-
-        assert not t2
-
-    def test05_element_access(self):
-        """Test access to elements in matrix and array objects."""
-
-        from cppyy import gbl
-
-        N = 3
-        v = gbl.TVectorF(N)
-        m = gbl.TMatrixD(N, N)
-
-        for i in range(N):
-            assert v[i] == 0.0
-
-            for j in range(N):
-                assert m[i][j] == 0.0
-
-    def test06_static_function_call( self ):
-        """Test call to static function."""
-
-        import cppyy
-        TROOT, gROOT = cppyy.gbl.TROOT, cppyy.gbl.gROOT
-
-        c1 = TROOT.Class()
-        assert not not c1
-
-        c2 = gROOT.Class()
-
-        assert c1 == c2
-
-        old = gROOT.GetDirLevel()
-        TROOT.SetDirLevel(2)
-        assert 2 == gROOT.GetDirLevel()
-        gROOT.SetDirLevel(old)
-
-        old = TROOT.GetDirLevel()
-        gROOT.SetDirLevel(3)
-        assert 3 == TROOT.GetDirLevel()
-        TROOT.SetDirLevel(old)
-
-    def test07_macro(self):
-        """Test access to cpp macro's"""
-
-        from cppyy import gbl
-
-        assert gbl.NULL == 0
-
-        gbl.gROOT.ProcessLine('#define aap "aap"')
-        gbl.gROOT.ProcessLine('#define noot 1')
-        gbl.gROOT.ProcessLine('#define mies 2.0')
-
-        # TODO: macro's assumed to always be of long type ...
-        #assert gbl.aap  == "aap"
-        assert gbl.noot == 1
-        #assert gbl.mies == 2.0
-
-    def test08_opaque_pointer_passing(self):
-        """Test passing around of opaque pointers"""
-
-        import cppyy
-
-        # TODO: figure out CObject (see also test_advanced.py)
-
-        s = cppyy.gbl.TString("Hello World!")
-        #cobj = cppyy.as_cobject(s)
-        addr = cppyy.addressof(s)
-
-        #assert s == cppyy.bind_object(cobj, s.__class__)
-        #assert s == cppyy.bind_object(cobj, "TString")
-        assert s == cppyy.bind_object(addr, s.__class__)
-        assert s == cppyy.bind_object(addr, "TString")
-
-    def test09_object_and_pointer_comparisons(self):
-        """Verify object and pointer comparisons"""
-
-        import cppyy
-        gbl = cppyy.gbl
-
-        c1 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == None
-        assert None == c1
-
-        c2 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == c2
-        assert c2 == c1
-
-        # TLorentzVector overrides operator==
-        l1 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == None
-        assert None == l1
-
-        assert c1 != l1
-        assert l1 != c1
-
-        l2 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == l2
-        assert l2 == l1 
-
-        l3 = gbl.TLorentzVector(1, 2, 3, 4)
-        l4 = gbl.TLorentzVector(1, 2, 3, 4)
-        l5 = gbl.TLorentzVector(4, 3, 2, 1)
-        assert l3 == l4
-        assert l4 == l3
-
-        assert l3 != None                 # like this to ensure __ne__ is called
-        assert None != l3                 # id.
-        assert l3 != l5
-        assert l5 != l3
-
-    def test10_recursive_remove(self):
-        """Verify that objects are recursively removed when destroyed"""
-
-        import cppyy
-
-        c = cppyy.gbl.TClass.GetClass("TObject")
-
-        o = cppyy.gbl.TObject()
-        assert o
-
-        o.SetBit(cppyy.gbl.TObject.kMustCleanup)
-        c.Destructor(o)
-        assert not o
diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py
--- a/pypy/module/posix/interp_posix.py
+++ b/pypy/module/posix/interp_posix.py
@@ -10,7 +10,7 @@
 
 from rpython.rlib import rposix, rposix_stat, rfile
 from rpython.rlib import objectmodel, rurandom
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, intmask, r_uint, r_int
 from rpython.rlib.unroll import unrolling_iterable
 from rpython.rtyper.lltypesystem import lltype
@@ -1318,8 +1318,8 @@
     else:
         assert False, "Unknown fork hook"
 
+ at not_rpython
 def add_fork_hook(where, hook):
-    "NOT_RPYTHON"
     get_fork_hooks(where).append(hook)
 
 add_fork_hook('child', ExecutionContext._mark_thread_disappeared)
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
@@ -2455,3 +2455,61 @@
     assert (pt.x, pt.y) == (-9*500*999, 9*500*999)
     pt = lib.call2(lib.cb2)
     assert (pt.x, pt.y) == (99*500*999, -99*500*999)
+
+def test_ffi_gc_size_arg():
+    # with PyPy's GC, these calls to ffi.gc() would rapidly consume
+    # 40 GB of RAM without the third argument
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
@@ -2291,3 +2291,61 @@
         expected = "unsigned int"
     assert ffi.typeof("UINT_PTR") is ffi.typeof(expected)
     assert ffi.typeof("PTSTR") is ffi.typeof("wchar_t *")
+
+def test_gc_pypy_size_arg():
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+        # with PyPy's GC, the above would rapidly consume 40 GB of RAM
+        # without the third argument to ffi.gc()
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):

From pypy.commits at gmail.com  Fri Aug  4 11:26:12 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 08:26:12 -0700 (PDT)
Subject: [pypy-commit] pypy getarrayitem-into-bridges: close to-be-merged
 branch
Message-ID: <59849214.830f1c0a.b428c.f948@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: getarrayitem-into-bridges
Changeset: r92075:251f979f4bcc
Date: 2017-08-04 16:51 +0200
http://bitbucket.org/pypy/pypy/changeset/251f979f4bcc/

Log:	close to-be-merged branch


From pypy.commits at gmail.com  Fri Aug  4 11:26:14 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 08:26:14 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge getarrayitem-into-bridges:
Message-ID: <59849216.10e51c0a.32c6.e05e@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92076:43ff4a9015e3
Date: 2017-08-04 16:54 +0200
http://bitbucket.org/pypy/pypy/changeset/43ff4a9015e3/

Log:	merge getarrayitem-into-bridges:

	improvement on what information is retained into a bridge: in
	particular, knowledge about the content of arrays (at fixed indices)
	is stored in guards (and thus available at the beginning of
	bridges). also, some better handling of constants.

diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -18,6 +18,10 @@
 # (<box1> <descr> <box2>) length times, if getfield(box1, descr) == box2
 #                         both boxes should be in the liveboxes
 #
+# <length>
+# (<box1> <index> <descr> <box2>) length times, if getarrayitem_gc(box1, index, descr) == box2
+#                                 both boxes should be in the liveboxes
+#
 # ----
 
 
@@ -82,18 +86,26 @@
     # structs
     # XXX could be extended to arrays
     if optimizer.optheap:
-        triples = optimizer.optheap.serialize_optheap(available_boxes)
+        triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
         # metainterp_sd.all_descrs
-        triples = [triple for triple in triples if triple[1].descr_index != -1]
-        numb_state.append_int(len(triples))
-        for box1, descr, box2 in triples:
-            index = descr.descr_index
+        triples_struct = [triple for triple in triples_struct if triple[1].descr_index != -1]
+        numb_state.append_int(len(triples_struct))
+        for box1, descr, box2 in triples_struct:
+            descr_index = descr.descr_index
+            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_int(descr_index)
+            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+        numb_state.append_int(len(triples_array))
+        for box1, index, descr, box2 in triples_array:
+            descr_index = descr.descr_index
             numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
             numb_state.append_int(index)
+            numb_state.append_int(descr_index)
             numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
     else:
         numb_state.append_int(0)
+        numb_state.append_int(0)
 
 def deserialize_optimizer_knowledge(optimizer, resumestorage, frontend_boxes, liveboxes):
     reader = resumecode.Reader(resumestorage.rd_numb)
@@ -123,13 +135,24 @@
     if not optimizer.optheap:
         return
     length = reader.next_item()
-    result = []
+    result_struct = []
+    for i in range(length):
+        tagged = reader.next_item()
+        box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
+        tagged = reader.next_item()
+        box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        result_struct.append((box1, descr, box2))
+    length = reader.next_item()
+    result_array = []
     for i in range(length):
         tagged = reader.next_item()
         box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
         index = reader.next_item()
-        descr = metainterp_sd.all_descrs[index]
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
         tagged = reader.next_item()
         box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
-        result.append((box1, descr, box2))
-    optimizer.optheap.deserialize_optheap(result)
+        result_array.append((box1, index, descr, box2))
+    optimizer.optheap.deserialize_optheap(result_struct, result_array)
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -698,7 +698,7 @@
         return self.emit(op)
 
     def serialize_optheap(self, available_boxes):
-        result = []
+        result_getfield = []
         for descr, cf in self.cached_fields.iteritems():
             if cf._lazy_set:
                 continue # XXX safe default for now
@@ -706,27 +706,62 @@
             if not parent_descr.is_object():
                 continue # XXX could be extended to non-instance objects
             for i, box1 in enumerate(cf.cached_structs):
-                if box1 not in available_boxes:
+                if not box1.is_constant() and box1 not in available_boxes:
                     continue
                 structinfo = cf.cached_infos[i]
-                box2 = structinfo.getfield(descr).get_box_replacement()
-                if isinstance(box2, Const) or box2 in available_boxes:
-                    result.append((box1, descr, box2))
-        return result
+                box2 = structinfo.getfield(descr)
+                if box2 is None:
+                    # XXX this should not happen, as it is an invariant
+                    # violation! yet it does if box1 is a constant
+                    continue
+                box2 = box2.get_box_replacement()
+                if box2.is_constant() or box2 in available_boxes:
+                    result_getfield.append((box1, descr, box2))
+        result_array = []
+        for descr, indexdict in self.cached_arrayitems.iteritems():
+            for index, cf in indexdict.iteritems():
+                if cf._lazy_set:
+                    continue # XXX safe default for now
+                for i, box1 in enumerate(cf.cached_structs):
+                    if not box1.is_constant() and box1 not in available_boxes:
+                        continue
+                    arrayinfo = cf.cached_infos[i]
+                    box2 = arrayinfo.getitem(descr, index)
+                    if box2 is None:
+                        # XXX this should not happen, as it is an invariant
+                        # violation! yet it does if box1 is a constant
+                        continue
+                    box2 = box2.get_box_replacement()
+                    if box2.is_constant() or box2 in available_boxes:
+                        result_array.append((box1, index, descr, box2))
+        return result_getfield, result_array
 
-    def deserialize_optheap(self, triples):
-        for box1, descr, box2 in triples:
+    def deserialize_optheap(self, triples_struct, triples_array):
+        for box1, descr, box2 in triples_struct:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
-            structinfo = box1.get_forwarded()
-            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
-                structinfo = info.InstancePtrInfo(parent_descr)
-                structinfo.init_fields(parent_descr, descr.get_index())
-                box1.set_forwarded(structinfo)
-
+            if box1.is_constant():
+                structinfo = info.ConstPtrInfo(box1)
+            else:
+                structinfo = box1.get_forwarded()
+                if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
+                    structinfo = info.InstancePtrInfo(parent_descr)
+                    structinfo.init_fields(parent_descr, descr.get_index())
+                    box1.set_forwarded(structinfo)
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
 
+        for box1, index, descr, box2 in triples_array:
+            if box1.is_constant():
+                arrayinfo = info.ConstPtrInfo(box1)
+            else:
+                arrayinfo = box1.get_forwarded()
+                if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
+                    arrayinfo = info.ArrayPtrInfo(descr)
+                    box1.set_forwarded(arrayinfo)
+            cf = self.arrayitem_cache(descr, index)
+            arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
+
 
 dispatch_opt = make_dispatcher_method(OptHeap, 'optimize_',
                                       default=OptHeap.emit)
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -61,7 +61,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0]
+    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0, 0]
 
     rbox1 = InputArgRef()
     rbox2 = InputArgRef()
@@ -97,7 +97,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert len(numb_state.create_numbering().code) == 2 + math.ceil(len(refboxes) / 6.0)
+    assert len(numb_state.create_numbering().code) == 3 + math.ceil(len(refboxes) / 6.0)
 
     dct = {box: cls
               for box, known_class in boxes_known_classes
@@ -143,11 +143,7 @@
     def test_bridge_field_read(self):
         myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
         class A(object):
-            def f(self):
-                return 1
-        class B(A):
-            def f(self):
-                return 2
+            pass
         class M(object):
             _immutable_fields_ = ['x']
             def __init__(self, x):
@@ -156,14 +152,12 @@
         m1 = M(1)
         m2 = M(2)
         def f(x, y, n):
+            a = A()
+            a.n = n
             if x:
-                a = A()
                 a.m = m1
-                a.n = n
             else:
-                a = B()
                 a.m = m2
-                a.n = n
             a.x = 0
             res = 0
             while y > 0:
@@ -186,3 +180,105 @@
         self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
         self.check_resops(getfield_gc_r=1) # in main loop
 
+    def test_bridge_field_read_constants(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        class M(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x):
+                self.x = x
+
+        m1 = M(1)
+        m2 = M(2)
+        a = A()
+        a.m = m1
+        a.n = 0
+        def f(x, y, n):
+            if x:
+                a.m = m1
+                a.n = n
+            else:
+                a.m = m2
+                a.n = n
+            a.x = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.n
+                m = jit.promote(a.m)
+                res += m.x
+                a.x += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.m)
+                res += m.x
+                res += n1 + a.n
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
+        self.check_resops(getfield_gc_r=1) # in main loop
+
+    def test_bridge_array_read(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
+        def f(x, y, n):
+            if x:
+                a = [1, n, 0]
+            else:
+                a = [2, n, 0]
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                n1 = a[1]
+                m = jit.promote(a[0])
+                res += m
+                a[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a[0])
+                res += m
+                res += n1 + a[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=4)
+
+    def test_bridge_array_read_constant(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        a = A()
+        a.l = [1, -65, 0]
+        def f(x, y, n):
+            if x:
+                a.l[0] = 1
+            else:
+                a.l[0] = 2
+            a.l[1] = n
+            a.l[2] = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.l[1]
+                m = jit.promote(a.l[0])
+                res += m
+                a.l[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.l[0])
+                res += m
+                res += n1 + a.l[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=5)

From pypy.commits at gmail.com  Fri Aug  4 11:26:16 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Fri, 04 Aug 2017 08:26:16 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge
Message-ID: <59849218.11331c0a.c5a14.18e6@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92077:8c68d973f65b
Date: 2017-08-04 17:25 +0200
http://bitbucket.org/pypy/pypy/changeset/8c68d973f65b/

Log:	merge

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -224,11 +224,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -288,7 +283,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 
 from pypy.interpreter.error import OperationError, oefmt
@@ -46,8 +47,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,6 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import rstack, rstackovf
 
 from pypy.interpreter import debug
@@ -57,8 +58,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -107,15 +109,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -142,8 +145,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,7 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -423,8 +423,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -64,8 +64,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -380,8 +380,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -621,9 +621,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -969,10 +969,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1013,8 +1013,8 @@
             self._staticdefs = zip(argnames[-len(defaults):], defaults)
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         defs_w = []
         for name, defaultval in self._staticdefs:
             if name.startswith('w_'):
@@ -1070,8 +1070,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs = gateway._getdefaults(space) # needs to be implemented by subclass
         code = gateway._code
@@ -1141,8 +1141,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1179,15 +1179,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1198,8 +1199,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1245,6 +1247,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,16 +18,17 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         self.lazy = True
         self.__class__.buildloaders()
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -61,8 +65,8 @@
         self.w_initialdict = self.space.call_method(self.w_dict, 'items')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -130,8 +134,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -161,8 +165,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space':space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -202,8 +206,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -40,13 +40,15 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -144,8 +145,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (we_are_translated, always_inline,
-        dont_inline)
+        dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -20,8 +20,8 @@
 from pypy.interpreter.pycode import PyCode, BytecodeCorruption
 from pypy.tool.stdlib_opcode import bytecode_spec
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -31,8 +31,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -8,14 +8,15 @@
 
 from rpython.rlib.jit import promote
 from rpython.rlib.objectmodel import compute_identity_hash, specialize
-from rpython.rlib.objectmodel import instantiate
+from rpython.rlib.objectmodel import instantiate, not_rpython
 from rpython.tool.sourcetools import compile2, func_with_new_name
 
 
 class TypeDef(object):
+    @not_rpython
     def __init__(self, __name, __base=None, __total_ordering__=None,
                  __buffer=None, **rawdict):
-        "NOT_RPYTHON: initialization-time only"
+        "initialization-time only"
         self.name = __name
         if __base is None:
             bases = []
@@ -113,8 +114,9 @@
 # register_finalizer() or not.
 
 @specialize.memo()
+ at not_rpython
 def get_unique_interplevel_subclass(space, cls):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert cls.typedef.acceptable_as_base_class
     try:
         return _unique_subclass_cache[cls]
@@ -349,15 +351,17 @@
         return self
 
 
+ at not_rpython
 def interp_attrproperty(name, cls, doc=None, wrapfn=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert wrapfn is not None
     def fget(space, obj):
         return getattr(space, wrapfn)(getattr(obj, name))
     return GetSetProperty(fget, cls=cls, doc=doc)
 
+ at not_rpython
 def interp_attrproperty_w(name, cls, doc=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     def fget(space, obj):
         w_value = getattr(obj, name)
         if w_value is None:
diff --git a/pypy/module/__builtin__/test/test_classobj.py b/pypy/module/__builtin__/test/test_classobj.py
--- a/pypy/module/__builtin__/test/test_classobj.py
+++ b/pypy/module/__builtin__/test/test_classobj.py
@@ -1090,18 +1090,18 @@
     def setup_class(cls):
         if cls.runappdirect:
             py.test.skip("can only be run on py.py")
-        def is_strdict(space, w_class):
-            from pypy.objspace.std.dictmultiobject import BytesDictStrategy
+        def is_moduledict(space, w_class):
+            from pypy.objspace.std.celldict import ModuleDictStrategy
             w_d = w_class.getdict(space)
-            return space.wrap(isinstance(w_d.get_strategy(), BytesDictStrategy))
+            return space.wrap(isinstance(w_d.get_strategy(), ModuleDictStrategy))
 
-        cls.w_is_strdict = cls.space.wrap(gateway.interp2app(is_strdict))
+        cls.w_is_moduledict = cls.space.wrap(gateway.interp2app(is_moduledict))
 
-    def test_strdict(self):
+    def test_moduledict(self):
         class A:
             a = 1
             b = 2
-        assert self.is_strdict(A)
+        assert self.is_moduledict(A)
 
     def test_attr_slots(self):
         class C:
diff --git a/pypy/module/_codecs/__init__.py b/pypy/module/_codecs/__init__.py
--- a/pypy/module/_codecs/__init__.py
+++ b/pypy/module/_codecs/__init__.py
@@ -1,5 +1,6 @@
 from pypy.interpreter.mixedmodule import MixedModule
 from rpython.rlib import runicode
+from rpython.rlib.objectmodel import not_rpython
 from pypy.module._codecs import interp_codecs
 
 class Module(MixedModule):
@@ -86,9 +87,8 @@
          'unicode_internal_encode'   :  'interp_codecs.unicode_internal_encode',
     }
 
+    @not_rpython
     def __init__(self, space, *args):
-        "NOT_RPYTHON"
-
         # mbcs codec is Windows specific, and based on rffi.
         if (hasattr(runicode, 'str_decode_mbcs')):
             self.interpleveldefs['mbcs_encode'] = 'interp_codecs.mbcs_encode'
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -1,5 +1,5 @@
 from rpython.rlib import jit
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rstring import UnicodeBuilder
 from rpython.rlib.runicode import code_to_unichr, MAXUNICODE
 
@@ -268,8 +268,8 @@
         raise oefmt(space.w_TypeError,
                     "don't know how to handle %T in error callback", w_exc)
 
+ at not_rpython
 def register_builtin_error_handlers(space):
-    "NOT_RPYTHON"
     state = space.fromcache(CodecState)
     for error in ("strict", "ignore", "replace", "xmlcharrefreplace",
                   "backslashreplace"):
diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -118,6 +118,29 @@
         return space.w_True
     return space.w_False
 
+index_count_jd = jit.JitDriver(
+    greens = ['count', 'arrclass', 'tp_item'],
+    reds = 'auto', name = 'array.index_or_count')
+
+def index_count_array(arr, w_val, count=False):
+    space = arr.space
+    tp_item = space.type(w_val)
+    arrclass = arr.__class__
+    cnt = 0
+    for i in range(arr.len):
+        index_count_jd.jit_merge_point(
+            tp_item=tp_item, count=count,
+            arrclass=arrclass)
+        w_item = arr.w_getitem(space, i)
+        if space.eq_w(w_item, w_val):
+            if count:
+                cnt += 1
+            else:
+                return i
+    if count:
+        return cnt
+    return -1
+
 UNICODE_ARRAY = lltype.Ptr(lltype.Array(lltype.UniChar,
                                         hints={'nolength': True}))
 
@@ -257,17 +280,12 @@
         """
         self.extend(w_x)
 
-    def descr_count(self, space, w_val):
+    def descr_count(self, space, w_x):
         """ count(x)
 
         Return number of occurrences of x in the array.
         """
-        cnt = 0
-        for i in range(self.len):
-            # XXX jitdriver
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_val):
-                cnt += 1
+        cnt = index_count_array(self, w_x, count=True)
         return space.newint(cnt)
 
     def descr_index(self, space, w_x):
@@ -275,10 +293,9 @@
 
         Return index of first occurrence of x in the array.
         """
-        for i in range(self.len):
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_x):
-                return space.newint(i)
+        res = index_count_array(self, w_x, count=False)
+        if res >= 0:
+            return space.newint(res)
         raise oefmt(space.w_ValueError, "array.index(x): x not in list")
 
     def descr_reverse(self, space):
@@ -752,7 +769,9 @@
 
 class TypeCode(object):
     def __init__(self, itemtype, unwrap, canoverflow=False, signed=False,
-                 method='__int__'):
+                 method='__int__', errorname=None):
+        if errorname is None:
+            errorname = unwrap[:-2]
         self.itemtype = itemtype
         self.bytes = rffi.sizeof(itemtype)
         self.arraytype = lltype.Array(itemtype, hints={'nolength': True})
@@ -762,6 +781,7 @@
         self.canoverflow = canoverflow
         self.w_class = None
         self.method = method
+        self.errorname = errorname
 
     def _freeze_(self):
         # hint for the annotator: track individual constant instances
@@ -785,8 +805,8 @@
     'i': TypeCode(rffi.INT,           'int_w', True, True),
     'I': _UINTTypeCode,
     'l': TypeCode(rffi.LONG,          'int_w', True, True),
-    'L': TypeCode(rffi.ULONG,         'bigint_w'),  # Overflow handled by
-                                                    # rbigint.touint() which
+    'L': TypeCode(rffi.ULONG,         'bigint_w',   # Overflow handled by
+                  errorname="integer"),             # rbigint.touint() which
                                                     # corresponds to the
                                                     # C-type unsigned long
     'f': TypeCode(lltype.SingleFloat, 'float_w', method='__float__'),
@@ -864,7 +884,7 @@
                         item = unwrap(space.call_method(w_item, mytype.method))
                     except OperationError:
                         raise oefmt(space.w_TypeError,
-                                    "array item must be " + mytype.unwrap[:-2])
+                                    "array item must be " + mytype.errorname)
                 else:
                     raise
             if mytype.unwrap == 'bigint_w':
diff --git a/pypy/module/array/test/test_array.py b/pypy/module/array/test/test_array.py
--- a/pypy/module/array/test/test_array.py
+++ b/pypy/module/array/test/test_array.py
@@ -162,6 +162,11 @@
             raises(OverflowError, a.append, -1)
             raises(OverflowError, a.append, 2 ** (8 * b))
 
+    def test_errormessage(self):
+        a = self.array("L", [1, 2, 3])
+        excinfo = raises(TypeError, "a[0] = 'abc'")
+        assert str(excinfo.value) == "array item must be integer"
+
     def test_fromstring(self):
         import sys
 
diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py
--- a/pypy/module/posix/interp_posix.py
+++ b/pypy/module/posix/interp_posix.py
@@ -3,7 +3,7 @@
 
 from rpython.rlib import rposix, rposix_stat
 from rpython.rlib import objectmodel, rurandom
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, intmask, r_uint
 from rpython.rlib.unroll import unrolling_iterable
 
@@ -731,8 +731,8 @@
     else:
         assert False, "Unknown fork hook"
 
+ at not_rpython
 def add_fork_hook(where, hook):
-    "NOT_RPYTHON"
     get_fork_hooks(where).append(hook)
 
 add_fork_hook('child', ExecutionContext._mark_thread_disappeared)
diff --git a/pypy/objspace/std/dictmultiobject.py b/pypy/objspace/std/dictmultiobject.py
--- a/pypy/objspace/std/dictmultiobject.py
+++ b/pypy/objspace/std/dictmultiobject.py
@@ -56,7 +56,7 @@
     def allocate_and_init_instance(space, w_type=None, module=False,
                                    instance=False, strdict=False,
                                    kwargs=False):
-        if space.config.objspace.std.withcelldict and module:
+        if module:
             from pypy.objspace.std.celldict import ModuleDictStrategy
             assert w_type is None
             # every module needs its own strategy, because the strategy stores
diff --git a/pypy/objspace/std/objspace.py b/pypy/objspace/std/objspace.py
--- a/pypy/objspace/std/objspace.py
+++ b/pypy/objspace/std/objspace.py
@@ -181,8 +181,8 @@
 
         return self._wrap_not_rpython(x)
 
+    @not_rpython
     def _wrap_not_rpython(self, x):
-        "NOT_RPYTHON"
         # _____ this code is here to support testing only _____
 
         # wrap() of a container works on CPython, but the code is
diff --git a/pypy/objspace/std/test/test_celldict.py b/pypy/objspace/std/test/test_celldict.py
--- a/pypy/objspace/std/test/test_celldict.py
+++ b/pypy/objspace/std/test/test_celldict.py
@@ -58,7 +58,6 @@
         assert v2 is v3
 
 class AppTestModuleDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         cls.w_runappdirect = cls.space.wrap(cls.runappdirect)
@@ -116,7 +115,6 @@
 
 
 class AppTestCellDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         if cls.runappdirect:
diff --git a/pypy/objspace/std/test/test_dictmultiobject.py b/pypy/objspace/std/test/test_dictmultiobject.py
--- a/pypy/objspace/std/test/test_dictmultiobject.py
+++ b/pypy/objspace/std/test/test_dictmultiobject.py
@@ -1261,7 +1261,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 
@@ -1467,6 +1466,7 @@
 
 
 def test_module_uses_strdict():
+    from pypy.objspace.std.celldict import ModuleDictStrategy
     fakespace = FakeSpace()
     d = fakespace.newdict(module=True)
-    assert type(d.get_strategy()) is BytesDictStrategy
+    assert type(d.get_strategy()) is ModuleDictStrategy
diff --git a/pypy/objspace/std/test/test_mapdict.py b/pypy/objspace/std/test/test_mapdict.py
--- a/pypy/objspace/std/test/test_mapdict.py
+++ b/pypy/objspace/std/test/test_mapdict.py
@@ -4,7 +4,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -11,7 +11,7 @@
 from rpython.rlib.jit import (promote, elidable_promote, we_are_jitted,
      elidable, dont_look_inside, unroll_safe)
 from rpython.rlib.objectmodel import current_object_addr_as_int, compute_hash
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 
 class MutableCell(W_Root):
@@ -212,8 +212,8 @@
         else:
             self.terminator = NoDictTerminator(space, self)
 
+    @not_rpython
     def __repr__(self):
-        "NOT_RPYTHON"
         return '<W_TypeObject %r at 0x%x>' % (self.name, id(self))
 
     def mutated(self, key):
@@ -492,8 +492,9 @@
                         self, w_subtype, w_subtype)
         return w_subtype
 
+    @not_rpython
     def _cleanup_(self):
-        "NOT_RPYTHON.  Forces the lazy attributes to be computed."
+        "Forces the lazy attributes to be computed."
         if 'lazyloaders' in self.__dict__:
             for attr in self.lazyloaders.keys():
                 self.getdictvalue(self.space, attr)
@@ -1317,8 +1318,9 @@
 
 
 class TypeCache(SpaceCache):
+    @not_rpython
     def build(self, typedef):
-        "NOT_RPYTHON: initialization-time only."
+        "initialization-time only."
         from pypy.objspace.std.objectobject import W_ObjectObject
         from pypy.interpreter.typedef import GetSetProperty
         from rpython.rlib.objectmodel import instantiate

From pypy.commits at gmail.com  Fri Aug  4 18:51:04 2017
From: pypy.commits at gmail.com (wlav)
Date: Fri, 04 Aug 2017 15:51:04 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: remove (now unnecessary)
 explicit instantiations
Message-ID: <5984fa58.4b6b1c0a.2fc54.611f@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92078:1b6feb53dfb7
Date: 2017-08-04 09:58 -0700
http://bitbucket.org/pypy/pypy/changeset/1b6feb53dfb7/

Log:	remove (now unnecessary) explicit instantiations

diff --git a/pypy/module/_cppyy/test/stltypes.cxx b/pypy/module/_cppyy/test/stltypes.cxx
--- a/pypy/module/_cppyy/test/stltypes.cxx
+++ b/pypy/module/_cppyy/test/stltypes.cxx
@@ -1,15 +1,5 @@
 #include "stltypes.h"
 
-#define STLTYPES_EXPLICIT_INSTANTIATION_WITH_COMPS(STLTYPE, TTYPE)              \
-namespace __gnu_cxx {                                                           \
-template bool operator==(const std::STLTYPE< TTYPE >::iterator&,                \
-                         const std::STLTYPE< TTYPE >::iterator&);               \
-template bool operator!=(const std::STLTYPE< TTYPE >::iterator&,                \
-                         const std::STLTYPE< TTYPE >::iterator&);               \
-}
-
-//- explicit instantiations of used comparisons
-STLTYPES_EXPLICIT_INSTANTIATION_WITH_COMPS(vector, int)
 
 //- class with lots of std::string handling
 stringy_class::stringy_class(const char* s) : m_string(s) {}
diff --git a/pypy/module/_cppyy/test/stltypes.h b/pypy/module/_cppyy/test/stltypes.h
--- a/pypy/module/_cppyy/test/stltypes.h
+++ b/pypy/module/_cppyy/test/stltypes.h
@@ -37,62 +37,3 @@
    std::string operator[](double) { return "double"; }
    std::string operator[](const std::string&) { return "string"; }
 };      
-
-
-#define STLTYPE_INSTANTIATION(STLTYPE, TTYPE, N)                             \
-   std::STLTYPE<TTYPE > STLTYPE##_##N;                                       \
-   std::STLTYPE<TTYPE >::iterator STLTYPE##_##N##_i;                         \
-   std::STLTYPE<TTYPE >::const_iterator STLTYPE##_##N##_ci
-
-#define STLTYPE_INSTANTIATION2(STLTYPE, TTYPE1, TTYPE2, N)                   \
-   std::STLTYPE<TTYPE1, TTYPE2 > STLTYPE##_##N;                              \
-   std::pair<TTYPE1, TTYPE2 > STLTYPE##_##N##_p;                             \
-   std::pair<const TTYPE1, TTYPE2 > STLTYPE##_##N##_cp;                      \
-   std::STLTYPE<TTYPE1, TTYPE2 >::iterator STLTYPE##_##N##_i;                \
-   std::STLTYPE<TTYPE1, TTYPE2 >::const_iterator STLTYPE##_##N##_ci
-
-
-//- instantiations of used STL types
-namespace {
-
-    struct _CppyyVectorInstances {
-
-        STLTYPE_INSTANTIATION(vector, int,          1);
-        STLTYPE_INSTANTIATION(vector, float,        2);
-        STLTYPE_INSTANTIATION(vector, double,       3);
-        STLTYPE_INSTANTIATION(vector, just_a_class, 4);
-
-    };
-
-    struct _CppyyListInstances {
-
-        STLTYPE_INSTANTIATION(list, int,    1);
-        STLTYPE_INSTANTIATION(list, float,  2);
-        STLTYPE_INSTANTIATION(list, double, 3);
-
-    };
-
-    struct _CppyyMapInstances {
-
-        STLTYPE_INSTANTIATION2(map, int,         int,           1);
-        STLTYPE_INSTANTIATION2(map, std::string, int,           2);
-        STLTYPE_INSTANTIATION2(map, std::string, unsigned int,  3);
-        STLTYPE_INSTANTIATION2(map, std::string, unsigned long, 4);
-
-    };
-
-    stl_like_class<int> stlc_1;
-
-} // unnamed namespace
-
-#define STLTYPES_EXPLICIT_INSTANTIATION_DECL_COMPS(STLTYPE, TTYPE)           \
-namespace __gnu_cxx {                                                        \
-extern template bool operator==(const std::STLTYPE< TTYPE >::iterator&,      \
-                         const std::STLTYPE< TTYPE >::iterator&);            \
-extern template bool operator!=(const std::STLTYPE< TTYPE >::iterator&,      \
-                         const std::STLTYPE< TTYPE >::iterator&);            \
-}
-
-// comps for int only to allow testing: normal use of vector is looping over a
-// range-checked version of __getitem__
-STLTYPES_EXPLICIT_INSTANTIATION_DECL_COMPS(vector, int)
diff --git a/pypy/module/_cppyy/test/stltypes.xml b/pypy/module/_cppyy/test/stltypes.xml
--- a/pypy/module/_cppyy/test/stltypes.xml
+++ b/pypy/module/_cppyy/test/stltypes.xml
@@ -2,30 +2,9 @@
 
   <namespace name="std" />
 
-  <class pattern="std::vector<*>" />
-  <!-- <class pattern="std::vector<*>::iterator" />
-  <class pattern="std::vector<*>::const_iterator" /> -->
-
-  <class pattern="std::list<*>" />
-  <!-- <class pattern="std::list<*>::iterator" />
-  <class pattern="std::list<*>::const_iterator" />
-  <class name="std::__detail::_List_node_base" /> -->
-
-  <class pattern="std::map<*>" />
-  <!-- <class pattern="std::pair<*>" /> -->
-  <!-- <class pattern="std::map<*>::iterator" />
-  <class pattern="std::map<*>::const_iterator" /> -->
-
-  <!-- <class pattern="__gnu_cxx::__normal_iterator<*>" /> -->
-
-  <!-- the following are for testing, not for iteration -->
-  <function name="__gnu_cxx::operator=="/>
-  <function name="__gnu_cxx::operator!="/>
-
   <class name="just_a_class" />
 
   <class name="std::string" />
   <class name="stringy_class" />
-  <class pattern="stl_like_class<*>" />
 
 </lcgdict>

From pypy.commits at gmail.com  Fri Aug  4 18:51:06 2017
From: pypy.commits at gmail.com (wlav)
Date: Fri, 04 Aug 2017 15:51:06 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: add 'cppyy.gbl.namespace' to
 sys.modules, instead of '_cppyy.gbl.namespace' (to move to frontend, later)
Message-ID: <5984fa5a.84abdf0a.e3703.79ea@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92079:65f1498d698a
Date: 2017-08-04 15:19 -0700
http://bitbucket.org/pypy/pypy/changeset/65f1498d698a/

Log:	add 'cppyy.gbl.namespace' to sys.modules, instead of
	'_cppyy.gbl.namespace' (to move to frontend, later)

diff --git a/pypy/module/_cppyy/pythonify.py b/pypy/module/_cppyy/pythonify.py
--- a/pypy/module/_cppyy/pythonify.py
+++ b/pypy/module/_cppyy/pythonify.py
@@ -126,7 +126,7 @@
             setattr(metans, dm_name, cppdm)
 
         modname = pycppns.__name__.replace('::', '.')
-        sys.modules['_cppyy.gbl.'+modname] = pycppns
+        sys.modules['cppyy.gbl.'+modname] = pycppns   # note naming (cppyy)
     return pycppns
 
 def _drop_cycles(bases):
@@ -444,9 +444,9 @@
     # install for user access
     _cppyy.gbl = gbl
 
-    # install as modules to allow importing from
-    sys.modules['_cppyy.gbl'] = gbl
-    sys.modules['_cppyy.gbl.std'] = gbl.std
+    # install as modules to allow importing from (note naming: cppyy)
+    sys.modules['cppyy.gbl'] = gbl
+    sys.modules['cppyy.gbl.std'] = gbl.std
 
 # user-defined pythonizations interface
 _pythonizations = {}
diff --git a/pypy/module/_cppyy/test/test_fragile.py b/pypy/module/_cppyy/test/test_fragile.py
--- a/pypy/module/_cppyy/test/test_fragile.py
+++ b/pypy/module/_cppyy/test/test_fragile.py
@@ -216,13 +216,18 @@
 
         # TODO: namespaces aren't loaded (and thus not added to sys.modules)
         # with just the from ... import statement; actual use is needed
-        from _cppyy.gbl import fragile
+
+        # TODO: this is really front-end testing, but the code is still in
+        # _cppyy, so we test it here until pythonify.py can be moved
+        import sys
+        sys.modules['cppyy'] = sys.modules['_cppyy']
+        from cppyy.gbl import fragile
 
         def fail_import():
-            from _cppyy.gbl import does_not_exist
+            from cppyy.gbl import does_not_exist
         raises(ImportError, fail_import)
 
-        from _cppyy.gbl.fragile import A, B, C, D
+        from cppyy.gbl.fragile import A, B, C, D
         assert _cppyy.gbl.fragile.A is A
         assert _cppyy.gbl.fragile.B is B
         assert _cppyy.gbl.fragile.C is C
@@ -230,18 +235,18 @@
 
         # according to warnings, can't test "import *" ...
 
-        from _cppyy.gbl.fragile import nested1
+        from cppyy.gbl.fragile import nested1
         assert _cppyy.gbl.fragile.nested1 is nested1
 
-        from _cppyy.gbl.fragile.nested1 import A, nested2
+        from cppyy.gbl.fragile.nested1 import A, nested2
         assert _cppyy.gbl.fragile.nested1.A is A
         assert _cppyy.gbl.fragile.nested1.nested2 is nested2
 
-        from _cppyy.gbl.fragile.nested1.nested2 import A, nested3
+        from cppyy.gbl.fragile.nested1.nested2 import A, nested3
         assert _cppyy.gbl.fragile.nested1.nested2.A is A
         assert _cppyy.gbl.fragile.nested1.nested2.nested3 is nested3
 
-        from _cppyy.gbl.fragile.nested1.nested2.nested3 import A
+        from cppyy.gbl.fragile.nested1.nested2.nested3 import A
         assert _cppyy.gbl.fragile.nested1.nested2.nested3.A is nested3.A
 
     def test12_missing_casts(self):
diff --git a/pypy/module/_cppyy/test/test_stltypes.py b/pypy/module/_cppyy/test/test_stltypes.py
--- a/pypy/module/_cppyy/test/test_stltypes.py
+++ b/pypy/module/_cppyy/test/test_stltypes.py
@@ -288,7 +288,7 @@
         """Test access to a list<int>"""
 
         import _cppyy
-        from _cppyy.gbl import std
+        std = _cppyy.gbl.std
 
         type_info = (
             ("int",     int),
@@ -323,7 +323,7 @@
         """Test behavior of empty list<int>"""
 
         import _cppyy
-        from _cppyy.gbl import std
+        std = _cppyy.gbl.std
 
         a = std.list(int)()
         for arg in a:
@@ -452,7 +452,7 @@
         """Test iterator comparison with operator== reflected"""
 
         import _cppyy
-        from _cppyy.gbl import std
+        std = _cppyy.gbl.std
 
         v = std.vector(int)()
         v.resize(1)

From pypy.commits at gmail.com  Fri Aug  4 18:51:08 2017
From: pypy.commits at gmail.com (wlav)
Date: Fri, 04 Aug 2017 15:51:08 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: merge default into branch
Message-ID: <5984fa5c.db6f1c0a.3caf2.8033@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92080:3a13ca0c3f09
Date: 2017-08-04 15:37 -0700
http://bitbucket.org/pypy/pypy/changeset/3a13ca0c3f09/

Log:	merge default into branch

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -224,11 +224,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -288,7 +283,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -152,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -181,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -217,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -64,9 +64,9 @@
 cppyy
 -----
 
-For C++, `cppyy`_ is an automated bindings generator available for both
+For C++, _cppyy_ is an automated bindings generator available for both
 PyPy and CPython.
-``cppyy`` relies on declarations from C++ header files to dynamically
+_cppyy_ relies on declarations from C++ header files to dynamically
 construct Python equivalent classes, functions, variables, etc.
 It is designed for use by large scale programs and supports modern C++.
 With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
@@ -75,8 +75,7 @@
 To install, run ``pip install cppyy``.
 Further details are available in the `full documentation`_.
 
-.. _cppyy: http://cppyy.readthedocs.org/
-.. _`full documentation`: http://cppyy.readthedocs.org/
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
 
 RPython Mixed Modules
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -55,3 +55,8 @@
 
 Fix the bounds in the GC when allocating a lot of objects with finalizers,
 fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 
 from pypy.interpreter.error import OperationError, oefmt
@@ -46,8 +47,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,6 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import rstack, rstackovf
 
 from pypy.interpreter import debug
@@ -57,8 +58,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -107,15 +109,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -142,8 +145,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,7 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -423,8 +423,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -64,8 +64,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -380,8 +380,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -621,9 +621,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -969,10 +969,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1013,8 +1013,8 @@
             self._staticdefs = zip(argnames[-len(defaults):], defaults)
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         defs_w = []
         for name, defaultval in self._staticdefs:
             if name.startswith('w_'):
@@ -1070,8 +1070,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs = gateway._getdefaults(space) # needs to be implemented by subclass
         code = gateway._code
@@ -1141,8 +1141,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1179,15 +1179,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1198,8 +1199,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1245,6 +1247,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,16 +18,17 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         self.lazy = True
         self.__class__.buildloaders()
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -61,8 +65,8 @@
         self.w_initialdict = self.space.call_method(self.w_dict, 'items')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -130,8 +134,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -161,8 +165,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space':space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -202,8 +206,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -40,13 +40,15 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -144,8 +145,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (we_are_translated, always_inline,
-        dont_inline)
+        dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -20,8 +20,8 @@
 from pypy.interpreter.pycode import PyCode, BytecodeCorruption
 from pypy.tool.stdlib_opcode import bytecode_spec
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -31,8 +31,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -8,14 +8,15 @@
 
 from rpython.rlib.jit import promote
 from rpython.rlib.objectmodel import compute_identity_hash, specialize
-from rpython.rlib.objectmodel import instantiate
+from rpython.rlib.objectmodel import instantiate, not_rpython
 from rpython.tool.sourcetools import compile2, func_with_new_name
 
 
 class TypeDef(object):
+    @not_rpython
     def __init__(self, __name, __base=None, __total_ordering__=None,
                  __buffer=None, **rawdict):
-        "NOT_RPYTHON: initialization-time only"
+        "initialization-time only"
         self.name = __name
         if __base is None:
             bases = []
@@ -113,8 +114,9 @@
 # register_finalizer() or not.
 
 @specialize.memo()
+ at not_rpython
 def get_unique_interplevel_subclass(space, cls):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert cls.typedef.acceptable_as_base_class
     try:
         return _unique_subclass_cache[cls]
@@ -349,15 +351,17 @@
         return self
 
 
+ at not_rpython
 def interp_attrproperty(name, cls, doc=None, wrapfn=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert wrapfn is not None
     def fget(space, obj):
         return getattr(space, wrapfn)(getattr(obj, name))
     return GetSetProperty(fget, cls=cls, doc=doc)
 
+ at not_rpython
 def interp_attrproperty_w(name, cls, doc=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     def fget(space, obj):
         w_value = getattr(obj, name)
         if w_value is None:
diff --git a/pypy/module/__builtin__/test/test_classobj.py b/pypy/module/__builtin__/test/test_classobj.py
--- a/pypy/module/__builtin__/test/test_classobj.py
+++ b/pypy/module/__builtin__/test/test_classobj.py
@@ -1090,18 +1090,18 @@
     def setup_class(cls):
         if cls.runappdirect:
             py.test.skip("can only be run on py.py")
-        def is_strdict(space, w_class):
-            from pypy.objspace.std.dictmultiobject import BytesDictStrategy
+        def is_moduledict(space, w_class):
+            from pypy.objspace.std.celldict import ModuleDictStrategy
             w_d = w_class.getdict(space)
-            return space.wrap(isinstance(w_d.get_strategy(), BytesDictStrategy))
+            return space.wrap(isinstance(w_d.get_strategy(), ModuleDictStrategy))
 
-        cls.w_is_strdict = cls.space.wrap(gateway.interp2app(is_strdict))
+        cls.w_is_moduledict = cls.space.wrap(gateway.interp2app(is_moduledict))
 
-    def test_strdict(self):
+    def test_moduledict(self):
         class A:
             a = 1
             b = 2
-        assert self.is_strdict(A)
+        assert self.is_moduledict(A)
 
     def test_attr_slots(self):
         class C:
diff --git a/pypy/module/_cffi_backend/cdataobj.py b/pypy/module/_cffi_backend/cdataobj.py
--- a/pypy/module/_cffi_backend/cdataobj.py
+++ b/pypy/module/_cffi_backend/cdataobj.py
@@ -433,17 +433,22 @@
     def _sizeof(self):
         return self.ctype.size
 
-    def with_gc(self, w_destructor):
+    def with_gc(self, w_destructor, size=0):
         space = self.space
         if space.is_none(w_destructor):
             if isinstance(self, W_CDataGCP):
                 self.detach_destructor()
-                return space.w_None
-            raise oefmt(space.w_TypeError,
-                        "Can remove destructor only on a object "
-                        "previously returned by ffi.gc()")
-        with self as ptr:
-            return W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+                w_res = space.w_None
+            else:
+                raise oefmt(space.w_TypeError,
+                            "Can remove destructor only on a object "
+                            "previously returned by ffi.gc()")
+        else:
+            with self as ptr:
+                w_res = W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+        if size != 0:
+            rgc.add_memory_pressure(size)
+        return w_res
 
     def unpack(self, length):
         from pypy.module._cffi_backend.ctypeptr import W_CTypePtrOrArray
diff --git a/pypy/module/_cffi_backend/ffi_obj.py b/pypy/module/_cffi_backend/ffi_obj.py
--- a/pypy/module/_cffi_backend/ffi_obj.py
+++ b/pypy/module/_cffi_backend/ffi_obj.py
@@ -351,14 +351,14 @@
         return handle.from_handle(self.space, w_arg)
 
 
-    @unwrap_spec(w_cdata=W_CData)
-    def descr_gc(self, w_cdata, w_destructor):
+    @unwrap_spec(w_cdata=W_CData, size=int)
+    def descr_gc(self, w_cdata, w_destructor, size=0):
         """\
 Return a new cdata object that points to the same data.
 Later, when this new cdata object is garbage-collected,
 'destructor(old_cdata_object)' will be called."""
         #
-        return w_cdata.with_gc(w_destructor)
+        return w_cdata.with_gc(w_destructor, size)
 
 
     @unwrap_spec(replace_with='text')
diff --git a/pypy/module/_cffi_backend/func.py b/pypy/module/_cffi_backend/func.py
--- a/pypy/module/_cffi_backend/func.py
+++ b/pypy/module/_cffi_backend/func.py
@@ -257,6 +257,6 @@
 
 # ____________________________________________________________
 
- at unwrap_spec(w_cdata=cdataobj.W_CData)
-def gcp(space, w_cdata, w_destructor):
-    return w_cdata.with_gc(w_destructor)
+ at unwrap_spec(w_cdata=cdataobj.W_CData, size=int)
+def gcp(space, w_cdata, w_destructor, size=0):
+    return w_cdata.with_gc(w_destructor, size)
diff --git a/pypy/module/_cffi_backend/test/test_ffi_obj.py b/pypy/module/_cffi_backend/test/test_ffi_obj.py
--- a/pypy/module/_cffi_backend/test/test_ffi_obj.py
+++ b/pypy/module/_cffi_backend/test/test_ffi_obj.py
@@ -377,7 +377,7 @@
         raises(TypeError, ffi.gc, p, None)
         seen = []
         q1 = ffi.gc(p, lambda p: seen.append(1))
-        q2 = ffi.gc(q1, lambda p: seen.append(2))
+        q2 = ffi.gc(q1, lambda p: seen.append(2), size=123)
         import gc; gc.collect()
         assert seen == []
         assert ffi.gc(q1, None) is None
diff --git a/pypy/module/_codecs/__init__.py b/pypy/module/_codecs/__init__.py
--- a/pypy/module/_codecs/__init__.py
+++ b/pypy/module/_codecs/__init__.py
@@ -1,5 +1,6 @@
 from pypy.interpreter.mixedmodule import MixedModule
 from rpython.rlib import runicode
+from rpython.rlib.objectmodel import not_rpython
 from pypy.module._codecs import interp_codecs
 
 class Module(MixedModule):
@@ -86,9 +87,8 @@
          'unicode_internal_encode'   :  'interp_codecs.unicode_internal_encode',
     }
 
+    @not_rpython
     def __init__(self, space, *args):
-        "NOT_RPYTHON"
-
         # mbcs codec is Windows specific, and based on rffi.
         if (hasattr(runicode, 'str_decode_mbcs')):
             self.interpleveldefs['mbcs_encode'] = 'interp_codecs.mbcs_encode'
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -1,5 +1,5 @@
 from rpython.rlib import jit
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rstring import UnicodeBuilder
 from rpython.rlib.runicode import code_to_unichr, MAXUNICODE
 
@@ -268,8 +268,8 @@
         raise oefmt(space.w_TypeError,
                     "don't know how to handle %T in error callback", w_exc)
 
+ at not_rpython
 def register_builtin_error_handlers(space):
-    "NOT_RPYTHON"
     state = space.fromcache(CodecState)
     for error in ("strict", "ignore", "replace", "xmlcharrefreplace",
                   "backslashreplace"):
diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -118,6 +118,29 @@
         return space.w_True
     return space.w_False
 
+index_count_jd = jit.JitDriver(
+    greens = ['count', 'arrclass', 'tp_item'],
+    reds = 'auto', name = 'array.index_or_count')
+
+def index_count_array(arr, w_val, count=False):
+    space = arr.space
+    tp_item = space.type(w_val)
+    arrclass = arr.__class__
+    cnt = 0
+    for i in range(arr.len):
+        index_count_jd.jit_merge_point(
+            tp_item=tp_item, count=count,
+            arrclass=arrclass)
+        w_item = arr.w_getitem(space, i)
+        if space.eq_w(w_item, w_val):
+            if count:
+                cnt += 1
+            else:
+                return i
+    if count:
+        return cnt
+    return -1
+
 UNICODE_ARRAY = lltype.Ptr(lltype.Array(lltype.UniChar,
                                         hints={'nolength': True}))
 
@@ -257,17 +280,12 @@
         """
         self.extend(w_x)
 
-    def descr_count(self, space, w_val):
+    def descr_count(self, space, w_x):
         """ count(x)
 
         Return number of occurrences of x in the array.
         """
-        cnt = 0
-        for i in range(self.len):
-            # XXX jitdriver
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_val):
-                cnt += 1
+        cnt = index_count_array(self, w_x, count=True)
         return space.newint(cnt)
 
     def descr_index(self, space, w_x):
@@ -275,10 +293,9 @@
 
         Return index of first occurrence of x in the array.
         """
-        for i in range(self.len):
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_x):
-                return space.newint(i)
+        res = index_count_array(self, w_x, count=False)
+        if res >= 0:
+            return space.newint(res)
         raise oefmt(space.w_ValueError, "array.index(x): x not in list")
 
     def descr_reverse(self, space):
@@ -752,7 +769,9 @@
 
 class TypeCode(object):
     def __init__(self, itemtype, unwrap, canoverflow=False, signed=False,
-                 method='__int__'):
+                 method='__int__', errorname=None):
+        if errorname is None:
+            errorname = unwrap[:-2]
         self.itemtype = itemtype
         self.bytes = rffi.sizeof(itemtype)
         self.arraytype = lltype.Array(itemtype, hints={'nolength': True})
@@ -762,6 +781,7 @@
         self.canoverflow = canoverflow
         self.w_class = None
         self.method = method
+        self.errorname = errorname
 
     def _freeze_(self):
         # hint for the annotator: track individual constant instances
@@ -785,8 +805,8 @@
     'i': TypeCode(rffi.INT,           'int_w', True, True),
     'I': _UINTTypeCode,
     'l': TypeCode(rffi.LONG,          'int_w', True, True),
-    'L': TypeCode(rffi.ULONG,         'bigint_w'),  # Overflow handled by
-                                                    # rbigint.touint() which
+    'L': TypeCode(rffi.ULONG,         'bigint_w',   # Overflow handled by
+                  errorname="integer"),             # rbigint.touint() which
                                                     # corresponds to the
                                                     # C-type unsigned long
     'f': TypeCode(lltype.SingleFloat, 'float_w', method='__float__'),
@@ -864,7 +884,7 @@
                         item = unwrap(space.call_method(w_item, mytype.method))
                     except OperationError:
                         raise oefmt(space.w_TypeError,
-                                    "array item must be " + mytype.unwrap[:-2])
+                                    "array item must be " + mytype.errorname)
                 else:
                     raise
             if mytype.unwrap == 'bigint_w':
diff --git a/pypy/module/array/test/test_array.py b/pypy/module/array/test/test_array.py
--- a/pypy/module/array/test/test_array.py
+++ b/pypy/module/array/test/test_array.py
@@ -162,6 +162,11 @@
             raises(OverflowError, a.append, -1)
             raises(OverflowError, a.append, 2 ** (8 * b))
 
+    def test_errormessage(self):
+        a = self.array("L", [1, 2, 3])
+        excinfo = raises(TypeError, "a[0] = 'abc'")
+        assert str(excinfo.value) == "array item must be integer"
+
     def test_fromstring(self):
         import sys
 
diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py
--- a/pypy/module/posix/interp_posix.py
+++ b/pypy/module/posix/interp_posix.py
@@ -3,7 +3,7 @@
 
 from rpython.rlib import rposix, rposix_stat
 from rpython.rlib import objectmodel, rurandom
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, intmask, r_uint
 from rpython.rlib.unroll import unrolling_iterable
 
@@ -731,8 +731,8 @@
     else:
         assert False, "Unknown fork hook"
 
+ at not_rpython
 def add_fork_hook(where, hook):
-    "NOT_RPYTHON"
     get_fork_hooks(where).append(hook)
 
 add_fork_hook('child', ExecutionContext._mark_thread_disappeared)
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi0/test_verify.py
@@ -2455,3 +2455,61 @@
     assert (pt.x, pt.y) == (-9*500*999, 9*500*999)
     pt = lib.call2(lib.cb2)
     assert (pt.x, pt.y) == (99*500*999, -99*500*999)
+
+def test_ffi_gc_size_arg():
+    # with PyPy's GC, these calls to ffi.gc() would rapidly consume
+    # 40 GB of RAM without the third argument
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
--- a/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
+++ b/pypy/module/test_lib_pypy/cffi_tests/cffi1/test_verify1.py
@@ -2291,3 +2291,61 @@
         expected = "unsigned int"
     assert ffi.typeof("UINT_PTR") is ffi.typeof(expected)
     assert ffi.typeof("PTSTR") is ffi.typeof("wchar_t *")
+
+def test_gc_pypy_size_arg():
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    for i in range(2000):
+        p = lib.malloc(20*1024*1024)    # 20 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 20*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 20*1024*1024)
+        del p
+        # with PyPy's GC, the above would rapidly consume 40 GB of RAM
+        # without the third argument to ffi.gc()
+
+def test_ffi_gc_size_arg_2():
+    # a variant of the above: this "attack" works on cpython's cyclic gc too
+    # and I found no obvious way to prevent that.  So for now, this test
+    # is skipped on CPython, where it eats all the memory.
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("void *malloc(size_t); void free(void *);")
+    lib = ffi.verify(r"""
+        #include <stdlib.h>
+    """)
+    class X(object):
+        pass
+    for i in range(2000):
+        p = lib.malloc(50*1024*1024)    # 50 MB
+        p1 = ffi.cast("char *", p)
+        for j in xrange(0, 50*1024*1024, 4096):
+            p1[j] = '!'
+        p = ffi.gc(p, lib.free, 50*1024*1024)
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
+
+def test_ffi_new_with_cycles():
+    # still another variant, with ffi.new()
+    if '__pypy__' not in sys.builtin_module_names:
+        py.test.skip("find a way to tweak the cyclic GC of CPython")
+    ffi = FFI()
+    ffi.cdef("")
+    lib = ffi.verify("")
+    class X(object):
+        pass
+    for i in range(2000):
+        p = ffi.new("char[]", 50*1024*1024)    # 50 MB
+        for j in xrange(0, 50*1024*1024, 4096):
+            p[j] = '!'
+        x = X()
+        x.p = p
+        x.cyclic = x
+        del p, x
diff --git a/pypy/objspace/std/dictmultiobject.py b/pypy/objspace/std/dictmultiobject.py
--- a/pypy/objspace/std/dictmultiobject.py
+++ b/pypy/objspace/std/dictmultiobject.py
@@ -56,7 +56,7 @@
     def allocate_and_init_instance(space, w_type=None, module=False,
                                    instance=False, strdict=False,
                                    kwargs=False):
-        if space.config.objspace.std.withcelldict and module:
+        if module:
             from pypy.objspace.std.celldict import ModuleDictStrategy
             assert w_type is None
             # every module needs its own strategy, because the strategy stores
diff --git a/pypy/objspace/std/objspace.py b/pypy/objspace/std/objspace.py
--- a/pypy/objspace/std/objspace.py
+++ b/pypy/objspace/std/objspace.py
@@ -181,8 +181,8 @@
 
         return self._wrap_not_rpython(x)
 
+    @not_rpython
     def _wrap_not_rpython(self, x):
-        "NOT_RPYTHON"
         # _____ this code is here to support testing only _____
 
         # wrap() of a container works on CPython, but the code is
diff --git a/pypy/objspace/std/test/test_celldict.py b/pypy/objspace/std/test/test_celldict.py
--- a/pypy/objspace/std/test/test_celldict.py
+++ b/pypy/objspace/std/test/test_celldict.py
@@ -58,7 +58,6 @@
         assert v2 is v3
 
 class AppTestModuleDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         cls.w_runappdirect = cls.space.wrap(cls.runappdirect)
@@ -116,7 +115,6 @@
 
 
 class AppTestCellDict(object):
-    spaceconfig = {"objspace.std.withcelldict": True}
 
     def setup_class(cls):
         if cls.runappdirect:
diff --git a/pypy/objspace/std/test/test_dictmultiobject.py b/pypy/objspace/std/test/test_dictmultiobject.py
--- a/pypy/objspace/std/test/test_dictmultiobject.py
+++ b/pypy/objspace/std/test/test_dictmultiobject.py
@@ -1261,7 +1261,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 
@@ -1467,6 +1466,7 @@
 
 
 def test_module_uses_strdict():
+    from pypy.objspace.std.celldict import ModuleDictStrategy
     fakespace = FakeSpace()
     d = fakespace.newdict(module=True)
-    assert type(d.get_strategy()) is BytesDictStrategy
+    assert type(d.get_strategy()) is ModuleDictStrategy
diff --git a/pypy/objspace/std/test/test_mapdict.py b/pypy/objspace/std/test/test_mapdict.py
--- a/pypy/objspace/std/test/test_mapdict.py
+++ b/pypy/objspace/std/test/test_mapdict.py
@@ -4,7 +4,6 @@
 class Config:
     class objspace:
         class std:
-            withcelldict = False
             methodcachesizeexp = 11
             withmethodcachecounter = False
 
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -11,7 +11,7 @@
 from rpython.rlib.jit import (promote, elidable_promote, we_are_jitted,
      elidable, dont_look_inside, unroll_safe)
 from rpython.rlib.objectmodel import current_object_addr_as_int, compute_hash
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 
 class MutableCell(W_Root):
@@ -212,8 +212,8 @@
         else:
             self.terminator = NoDictTerminator(space, self)
 
+    @not_rpython
     def __repr__(self):
-        "NOT_RPYTHON"
         return '<W_TypeObject %r at 0x%x>' % (self.name, id(self))
 
     def mutated(self, key):
@@ -492,8 +492,9 @@
                         self, w_subtype, w_subtype)
         return w_subtype
 
+    @not_rpython
     def _cleanup_(self):
-        "NOT_RPYTHON.  Forces the lazy attributes to be computed."
+        "Forces the lazy attributes to be computed."
         if 'lazyloaders' in self.__dict__:
             for attr in self.lazyloaders.keys():
                 self.getdictvalue(self.space, attr)
@@ -1317,8 +1318,9 @@
 
 
 class TypeCache(SpaceCache):
+    @not_rpython
     def build(self, typedef):
-        "NOT_RPYTHON: initialization-time only."
+        "initialization-time only."
         from pypy.objspace.std.objectobject import W_ObjectObject
         from pypy.interpreter.typedef import GetSetProperty
         from rpython.rlib.objectmodel import instantiate
diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -18,6 +18,10 @@
 # (<box1> <descr> <box2>) length times, if getfield(box1, descr) == box2
 #                         both boxes should be in the liveboxes
 #
+# <length>
+# (<box1> <index> <descr> <box2>) length times, if getarrayitem_gc(box1, index, descr) == box2
+#                                 both boxes should be in the liveboxes
+#
 # ----
 
 
@@ -82,18 +86,26 @@
     # structs
     # XXX could be extended to arrays
     if optimizer.optheap:
-        triples = optimizer.optheap.serialize_optheap(available_boxes)
+        triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
         # metainterp_sd.all_descrs
-        triples = [triple for triple in triples if triple[1].descr_index != -1]
-        numb_state.append_int(len(triples))
-        for box1, descr, box2 in triples:
-            index = descr.descr_index
+        triples_struct = [triple for triple in triples_struct if triple[1].descr_index != -1]
+        numb_state.append_int(len(triples_struct))
+        for box1, descr, box2 in triples_struct:
+            descr_index = descr.descr_index
+            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_int(descr_index)
+            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+        numb_state.append_int(len(triples_array))
+        for box1, index, descr, box2 in triples_array:
+            descr_index = descr.descr_index
             numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
             numb_state.append_int(index)
+            numb_state.append_int(descr_index)
             numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
     else:
         numb_state.append_int(0)
+        numb_state.append_int(0)
 
 def deserialize_optimizer_knowledge(optimizer, resumestorage, frontend_boxes, liveboxes):
     reader = resumecode.Reader(resumestorage.rd_numb)
@@ -123,13 +135,24 @@
     if not optimizer.optheap:
         return
     length = reader.next_item()
-    result = []
+    result_struct = []
+    for i in range(length):
+        tagged = reader.next_item()
+        box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
+        tagged = reader.next_item()
+        box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        result_struct.append((box1, descr, box2))
+    length = reader.next_item()
+    result_array = []
     for i in range(length):
         tagged = reader.next_item()
         box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
         index = reader.next_item()
-        descr = metainterp_sd.all_descrs[index]
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
         tagged = reader.next_item()
         box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
-        result.append((box1, descr, box2))
-    optimizer.optheap.deserialize_optheap(result)
+        result_array.append((box1, index, descr, box2))
+    optimizer.optheap.deserialize_optheap(result_struct, result_array)
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -223,7 +223,10 @@
     def invalidate(self, descr):
         for opinfo in self.cached_infos:
             assert isinstance(opinfo, info.ArrayPtrInfo)
-            opinfo._items = None
+            # only invalidate those at self.index
+            if self.index < len(opinfo._items):
+                opinfo._items[self.index] = None
+            #opinfo._items = None #[self.index] = None
         self.cached_infos = []
         self.cached_structs = []
 
@@ -695,7 +698,7 @@
         return self.emit(op)
 
     def serialize_optheap(self, available_boxes):
-        result = []
+        result_getfield = []
         for descr, cf in self.cached_fields.iteritems():
             if cf._lazy_set:
                 continue # XXX safe default for now
@@ -703,27 +706,62 @@
             if not parent_descr.is_object():
                 continue # XXX could be extended to non-instance objects
             for i, box1 in enumerate(cf.cached_structs):
-                if box1 not in available_boxes:
+                if not box1.is_constant() and box1 not in available_boxes:
                     continue
                 structinfo = cf.cached_infos[i]
-                box2 = structinfo.getfield(descr).get_box_replacement()
-                if isinstance(box2, Const) or box2 in available_boxes:
-                    result.append((box1, descr, box2))
-        return result
+                box2 = structinfo.getfield(descr)
+                if box2 is None:
+                    # XXX this should not happen, as it is an invariant
+                    # violation! yet it does if box1 is a constant
+                    continue
+                box2 = box2.get_box_replacement()
+                if box2.is_constant() or box2 in available_boxes:
+                    result_getfield.append((box1, descr, box2))
+        result_array = []
+        for descr, indexdict in self.cached_arrayitems.iteritems():
+            for index, cf in indexdict.iteritems():
+                if cf._lazy_set:
+                    continue # XXX safe default for now
+                for i, box1 in enumerate(cf.cached_structs):
+                    if not box1.is_constant() and box1 not in available_boxes:
+                        continue
+                    arrayinfo = cf.cached_infos[i]
+                    box2 = arrayinfo.getitem(descr, index)
+                    if box2 is None:
+                        # XXX this should not happen, as it is an invariant
+                        # violation! yet it does if box1 is a constant
+                        continue
+                    box2 = box2.get_box_replacement()
+                    if box2.is_constant() or box2 in available_boxes:
+                        result_array.append((box1, index, descr, box2))
+        return result_getfield, result_array
 
-    def deserialize_optheap(self, triples):
-        for box1, descr, box2 in triples:
+    def deserialize_optheap(self, triples_struct, triples_array):
+        for box1, descr, box2 in triples_struct:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
-            structinfo = box1.get_forwarded()
-            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
-                structinfo = info.InstancePtrInfo(parent_descr)
-                structinfo.init_fields(parent_descr, descr.get_index())
-                box1.set_forwarded(structinfo)
-
+            if box1.is_constant():
+                structinfo = info.ConstPtrInfo(box1)
+            else:
+                structinfo = box1.get_forwarded()
+                if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
+                    structinfo = info.InstancePtrInfo(parent_descr)
+                    structinfo.init_fields(parent_descr, descr.get_index())
+                    box1.set_forwarded(structinfo)
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
 
+        for box1, index, descr, box2 in triples_array:
+            if box1.is_constant():
+                arrayinfo = info.ConstPtrInfo(box1)
+            else:
+                arrayinfo = box1.get_forwarded()
+                if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
+                    arrayinfo = info.ArrayPtrInfo(descr)
+                    box1.set_forwarded(arrayinfo)
+            cf = self.arrayitem_cache(descr, index)
+            arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
+
 
 dispatch_opt = make_dispatcher_method(OptHeap, 'optimize_',
                                       default=OptHeap.emit)
diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -1537,6 +1537,46 @@
         """
         self.optimize_loop(ops, expected)
 
+    def test_duplicate_getarrayitem_after_setarrayitem_and_guard(self):
+        ops = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p7 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p9 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        p10 = getarrayitem_gc_r(p1, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p7)
+        escape_n(p8)
+        escape_n(p9)
+        escape_n(p10)
+        jump(p0, p1, p2, p3, i1)
+        """
+        expected = """
+        [p0, p1, p2, p3, i1]
+        p4 = getarrayitem_gc_r(p0, 0, descr=arraydescr2)
+        p5 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        p6 = getarrayitem_gc_r(p1, 0, descr=arraydescr2)
+        setarrayitem_gc(p1, 1, p3, descr=arraydescr2)
+        guard_true(i1) [i1]
+        p8 = getarrayitem_gc_r(p0, 1, descr=arraydescr2)
+        escape_n(p4)
+        escape_n(p5)
+        escape_n(p6)
+        escape_n(p4)
+        escape_n(p8)
+        escape_n(p6)
+        escape_n(p3)
+        jump(p0, p1, p2, p3, 1)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
         [p1, p2]
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -61,7 +61,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0]
+    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0, 0]
 
     rbox1 = InputArgRef()
     rbox2 = InputArgRef()
@@ -97,7 +97,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert len(numb_state.create_numbering().code) == 2 + math.ceil(len(refboxes) / 6.0)
+    assert len(numb_state.create_numbering().code) == 3 + math.ceil(len(refboxes) / 6.0)
 
     dct = {box: cls
               for box, known_class in boxes_known_classes
@@ -143,11 +143,7 @@
     def test_bridge_field_read(self):
         myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
         class A(object):
-            def f(self):
-                return 1
-        class B(A):
-            def f(self):
-                return 2
+            pass
         class M(object):
             _immutable_fields_ = ['x']
             def __init__(self, x):
@@ -156,14 +152,12 @@
         m1 = M(1)
         m2 = M(2)
         def f(x, y, n):
+            a = A()
+            a.n = n
             if x:
-                a = A()
                 a.m = m1
-                a.n = n
             else:
-                a = B()
                 a.m = m2
-                a.n = n
             a.x = 0
             res = 0
             while y > 0:
@@ -186,3 +180,105 @@
         self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
         self.check_resops(getfield_gc_r=1) # in main loop
 
+    def test_bridge_field_read_constants(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        class M(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x):
+                self.x = x
+
+        m1 = M(1)
+        m2 = M(2)
+        a = A()
+        a.m = m1
+        a.n = 0
+        def f(x, y, n):
+            if x:
+                a.m = m1
+                a.n = n
+            else:
+                a.m = m2
+                a.n = n
+            a.x = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.n
+                m = jit.promote(a.m)
+                res += m.x
+                a.x += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.m)
+                res += m.x
+                res += n1 + a.n
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
+        self.check_resops(getfield_gc_r=1) # in main loop
+
+    def test_bridge_array_read(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
+        def f(x, y, n):
+            if x:
+                a = [1, n, 0]
+            else:
+                a = [2, n, 0]
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                n1 = a[1]
+                m = jit.promote(a[0])
+                res += m
+                a[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a[0])
+                res += m
+                res += n1 + a[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=4)
+
+    def test_bridge_array_read_constant(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        a = A()
+        a.l = [1, -65, 0]
+        def f(x, y, n):
+            if x:
+                a.l[0] = 1
+            else:
+                a.l[0] = 2
+            a.l[1] = n
+            a.l[2] = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.l[1]
+                m = jit.promote(a.l[0])
+                res += m
+                a.l[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.l[0])
+                res += m
+                res += n1 + a.l[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=5)

From pypy.commits at gmail.com  Fri Aug  4 23:27:59 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 04 Aug 2017 20:27:59 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Remove temporary changes
 and disable leakfinder checks, so that the tests pass
Message-ID: <59853b3f.935c1c0a.4cb2b.d0db@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92082:d66ca989779b
Date: 2017-08-05 04:27 +0100
http://bitbucket.org/pypy/pypy/changeset/d66ca989779b/

Log:	Remove temporary changes and disable leakfinder checks, so that the
	tests pass

diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -1,3 +1,4 @@
+import pytest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from pypy.conftest import option
 
@@ -111,6 +112,7 @@
         res = [1, 2, 3] * arr
         assert res == [2, 4, 6]
 
+    @pytest.mark.xfail
     def test_subclass_dealloc(self):
         module = self.import_module(name='array')
         class Sub(module.array):
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -154,7 +154,8 @@
         rawrefcount._collect()
         self.space.user_del_action._run_finalizers()
         try:
-            leakfinder.stop_tracking_allocations(check=True)
+            # set check=True to actually enable leakfinder
+            leakfinder.stop_tracking_allocations(check=False)
         except leakfinder.MallocMismatch as e:
             result = e.args[0]
             filtered_result = {}
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -676,9 +676,9 @@
     obj_pto = rffi.cast(PyTypeObjectPtr, obj)
     base_pyo = rffi.cast(PyObject, obj_pto.c_tp_base)
     Py_DecRef(space, obj_pto.c_tp_bases)
-    #Py_DecRef(space, obj_pto.c_tp_mro)
+    Py_DecRef(space, obj_pto.c_tp_mro)
     Py_DecRef(space, obj_pto.c_tp_cache) # let's do it like cpython
-    #Py_DecRef(space, obj_pto.c_tp_dict)
+    Py_DecRef(space, obj_pto.c_tp_dict)
     if obj_pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE:
         heaptype = rffi.cast(PyHeapTypeObject, obj)
         Py_DecRef(space, heaptype.c_ht_name)
@@ -932,7 +932,7 @@
     """
     Sets up other attributes, when the interpreter type has been created.
     """
-    #pto.c_tp_mro = make_ref(space, space.newtuple(w_obj.mro_w))
+    pto.c_tp_mro = make_ref(space, space.newtuple(w_obj.mro_w))
     base = pto.c_tp_base
     if base:
         inherit_special(space, pto, base)
@@ -952,10 +952,10 @@
 
     if w_obj.is_cpytype():
         Py_DecRef(space, pto.c_tp_dict)
-    #w_dict = w_obj.getdict(space)
+    w_dict = w_obj.getdict(space)
     # pass in the w_obj to convert any values that are
     # unbound GetSetProperty into bound PyGetSetDescrObject
-    #pto.c_tp_dict = make_ref(space, w_dict, w_obj)
+    pto.c_tp_dict = make_ref(space, w_dict, w_obj)
 
 @cpython_api([PyTypeObjectPtr, PyTypeObjectPtr], rffi.INT_real, error=CANNOT_FAIL)
 def PyType_IsSubtype(space, a, b):

From pypy.commits at gmail.com  Fri Aug  4 23:27:57 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 04 Aug 2017 20:27:57 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Remove useless 'reference
 warm-up'
Message-ID: <59853b3d.500a1c0a.12bf2.62ea@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92081:c65274990272
Date: 2017-08-03 21:01 +0100
http://bitbucket.org/pypy/pypy/changeset/c65274990272/

Log:	Remove useless 'reference warm-up'

diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py
--- a/pypy/module/cpyext/test/test_api.py
+++ b/pypy/module/cpyext/test/test_api.py
@@ -22,17 +22,6 @@
 class BaseApiTest(LeakCheckingTest):
     def setup_class(cls):
         space = cls.space
-        # warm up reference counts:
-        # - the posix module allocates a HCRYPTPROV on Windows
-        # - writing to stdout and stderr allocates a file lock
-        space.getbuiltinmodule("cpyext")
-        space.getbuiltinmodule(os.name)
-        space.call_function(space.getattr(space.sys.get("stderr"),
-                                          space.wrap("write")),
-                            space.wrap(""))
-        space.call_function(space.getattr(space.sys.get("stdout"),
-                                          space.wrap("write")),
-                            space.wrap(""))
         cls.preload_builtins(space)
 
         class CAPI:
@@ -41,9 +30,6 @@
         cls.api = CAPI()
         CAPI.__dict__.update(INTERPLEVEL_API)
 
-        print 'DONT_FREE_ANY_MORE'
-        rawrefcount._dont_free_any_more()
-
     def raises(self, space, api, expected_exc, f, *args):
         if not callable(f):
             raise Exception("%s is not callable" % (f,))
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -141,10 +141,6 @@
         Eagerly create pyobjs for various builtins so they don't look like
         leaks.
         """
-        space.getbuiltinmodule("cpyext")
-        # 'import os' to warm up reference counts
-        w_import = space.builtin.getdictvalue(space, '__import__')
-        space.call_function(w_import, space.wrap("os"))
         for name in [
                 'buffer', 'mmap.mmap',
                 'types.FunctionType', 'types.CodeType',

From pypy.commits at gmail.com  Sat Aug  5 00:24:24 2017
From: pypy.commits at gmail.com (wlav)
Date: Fri, 04 Aug 2017 21:24:24 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: raise LookupError when failing
 the dispatch lookup (consistency with CPyCppyy)
Message-ID: <59854878.4283df0a.4f05e.abc7@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92083:17bcb20012f1
Date: 2017-08-04 20:52 -0700
http://bitbucket.org/pypy/pypy/changeset/17bcb20012f1/

Log:	raise LookupError when failing the dispatch lookup (consistency with
	CPyCppyy)

diff --git a/pypy/module/_cppyy/interp_cppyy.py b/pypy/module/_cppyy/interp_cppyy.py
--- a/pypy/module/_cppyy/interp_cppyy.py
+++ b/pypy/module/_cppyy/interp_cppyy.py
@@ -788,7 +788,7 @@
         for f in overload.functions:
             if 0 < f.signature().find(sig):
                 return W_CPPOverload(self.space, self, [f])
-        raise oefmt(self.space.w_TypeError, "no overload matches signature")
+        raise oefmt(self.space.w_LookupError, "no overload matches signature")
 
     def missing_attribute_error(self, name):
         return oefmt(self.space.w_AttributeError,
diff --git a/pypy/module/_cppyy/test/test_overloads.py b/pypy/module/_cppyy/test/test_overloads.py
--- a/pypy/module/_cppyy/test/test_overloads.py
+++ b/pypy/module/_cppyy/test/test_overloads.py
@@ -57,7 +57,7 @@
 
         c = c_overload()
         raises(TypeError, c.__dispatch__, 'get_int', 12)
-        raises(TypeError, c.__dispatch__, 'get_int', 'does_not_exist')
+        raises(LookupError, c.__dispatch__, 'get_int', 'does_not_exist')
         assert c.__dispatch__('get_int', 'a_overload*')(a_overload()) == 42
         assert c.__dispatch__('get_int', 'b_overload*')(b_overload()) == 13
 

From pypy.commits at gmail.com  Sat Aug  5 00:24:25 2017
From: pypy.commits at gmail.com (wlav)
Date: Fri, 04 Aug 2017 21:24:25 -0700 (PDT)
Subject: [pypy-commit] pypy cppyy-packaging: (re-)enable some more tests and
 add a collect cycle for consistency
Message-ID: <59854879.6b99df0a.b0ebe.ec32@mx.google.com>

Author: Wim Lavrijsen <WLavrijsen at lbl.gov>
Branch: cppyy-packaging
Changeset: r92084:5f12167c0f08
Date: 2017-08-04 21:10 -0700
http://bitbucket.org/pypy/pypy/changeset/5f12167c0f08/

Log:	(re-)enable some more tests and add a collect cycle for consistency

diff --git a/pypy/module/_cppyy/test/test_pythonify.py b/pypy/module/_cppyy/test/test_pythonify.py
--- a/pypy/module/_cppyy/test/test_pythonify.py
+++ b/pypy/module/_cppyy/test/test_pythonify.py
@@ -68,18 +68,16 @@
         """Test object and method calls."""
         import _cppyy
         example01_class = _cppyy.gbl.example01
-        #assert example01_class.getCount() == 0
+        assert example01_class.getCount() == 0
         instance = example01_class(7)
-        #assert example01_class.getCount() == 1
+        assert example01_class.getCount() == 1
         res = instance.addDataToInt(4)
-        return
         assert res == 11
         res = instance.addDataToInt(-4)
         assert res == 3
         instance.__destruct__()
         assert example01_class.getCount() == 0
         raises(ReferenceError, 'instance.addDataToInt(4)')
-        return
 
         instance = example01_class(7)
         instance2 = example01_class(8)
@@ -89,7 +87,6 @@
         instance2.__destruct__()
         assert example01_class.getCount() == 0
 
-        t = self.example01
         instance = example01_class(13)
         res = instance.addDataToDouble(16)
         assert round(res-29, 8) == 0.
@@ -97,10 +94,10 @@
         instance = example01_class(-13)
         res = instance.addDataToDouble(16)
         assert round(res-3, 8) == 0.
+        instance.__destruct__()
 
-
-        t = self.example01
         instance = example01_class(42)
+        assert example01_class.getCount() == 1
 
         res = instance.addDataToAtoi("13")
         assert res == 55
@@ -327,7 +324,9 @@
     def test15_subclassing(self):
         """A sub-class on the python side should have that class as type"""
 
-        import _cppyy
+        import _cppyy, gc
+        gc.collect()
+
         example01 = _cppyy.gbl.example01
 
         assert example01.getCount() == 0

From pypy.commits at gmail.com  Sat Aug  5 02:58:59 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 04 Aug 2017 23:58:59 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2623
Message-ID: <59856cb3.6596df0a.aa15.f242@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92085:2d3c081aa72d
Date: 2017-08-05 08:58 +0200
http://bitbucket.org/pypy/pypy/changeset/2d3c081aa72d/

Log:	Issue #2623

	Tweak the signals logic to avoid the possibly-harmful optimization
	of pypysig_occurred.

diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -31,11 +31,11 @@
 # endif
 #endif
 
+#define N_LONGBITS  (8 * sizeof(long))
+#define N_LONGSIG   ((NSIG - 1) / N_LONGBITS + 1)
+
 struct pypysig_long_struct pypysig_counter = {0};
-static char volatile pypysig_flags[NSIG] = {0};
-static int volatile pypysig_occurred = 0;
-/* pypysig_occurred is only an optimization: it tells if any
-   pypysig_flags could be set. */
+static long volatile pypysig_flags_bits[N_LONGSIG];
 static int wakeup_fd = -1;
 static int wakeup_with_nul_byte = 1;
 
@@ -73,12 +73,28 @@
 #endif
 }
 
+#ifdef _WIN32
+#define atomic_cas(ptr, oldv, newv)   (InterlockedCompareExchange(ptr, \
+                                            newv, oldv) == (oldv))
+#else
+#define atomic_cas(ptr, oldv, newv)    __sync_bool_compare_and_swap(ptr, \
+                                            oldv, newv)
+#endif
+
 void pypysig_pushback(int signum)
 {
     if (0 <= signum && signum < NSIG)
       {
-        pypysig_flags[signum] = 1;
-        pypysig_occurred = 1;
+        int ok, index = signum / N_LONGBITS;
+        unsigned long bitmask = 1UL << (signum % N_LONGBITS);
+        do
+        {
+            long value = pypysig_flags_bits[index];
+            if (value & bitmask)
+                break;   /* already set */
+            ok = atomic_cas(&pypysig_flags_bits[index], value, value | bitmask);
+        } while (!ok);
+
         pypysig_counter.value = -1;
       }
 }
@@ -161,19 +177,22 @@
 
 int pypysig_poll(void)
 {
-  if (pypysig_occurred)
-    {
-      int i;
-      pypysig_occurred = 0;
-      for (i=0; i<NSIG; i++)
-        if (pypysig_flags[i])
-          {
-            pypysig_flags[i] = 0;
-            pypysig_occurred = 1;   /* maybe another signal is pending */
-            return i;
-          }
+    int index;
+    for (index = 0; index < N_LONGSIG; index++) {
+        long value;
+      retry:
+        value = pypysig_flags_bits[index];
+        if (value != 0L) {
+            int j = 0;
+            while ((value & (1UL << j)) == 0)
+                j++;
+            if (!atomic_cas(&pypysig_flags_bits[index], value,
+                            value & ~(1UL << j)))
+                goto retry;
+            return index * N_LONGBITS + j;
+        }
     }
-  return -1;  /* no pending signal */
+    return -1;  /* no pending signal */
 }
 
 int pypysig_set_wakeup_fd(int fd, int with_nul_byte)

From pypy.commits at gmail.com  Sat Aug  5 03:19:36 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 05 Aug 2017 00:19:36 -0700 (PDT)
Subject: [pypy-commit] pypy default: Fix: unclear why this was done,
 but the end result was that even "make
Message-ID: <59857188.93071c0a.1bb73.305e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92086:758dbcd2983e
Date: 2017-08-05 09:19 +0200
http://bitbucket.org/pypy/pypy/changeset/758dbcd2983e/

Log:	Fix: unclear why this was done, but the end result was that even
	"make lldebug" in the PyPy sources would compile everything in -O3
	instead of
	-O1, which was most likely not intended.

diff --git a/rpython/rlib/rvmprof/cintf.py b/rpython/rlib/rvmprof/cintf.py
--- a/rpython/rlib/rvmprof/cintf.py
+++ b/rpython/rlib/rvmprof/cintf.py
@@ -18,7 +18,7 @@
 SHARED = SRC.join('shared')
 BACKTRACE = SHARED.join('libbacktrace')
 
-compile_extra = ['-DRPYTHON_VMPROF', '-O3']
+compile_extra = ['-DRPYTHON_VMPROF']
 separate_module_files = [
     SHARED.join('symboltable.c'),
     SHARED.join('vmprof_unix.c')

From pypy.commits at gmail.com  Sat Aug  5 04:12:45 2017
From: pypy.commits at gmail.com (fijal)
Date: Sat, 05 Aug 2017 01:12:45 -0700 (PDT)
Subject: [pypy-commit] pypy default: whack until things stop exploding.
 can't reproduce in a test
Message-ID: <59857dfd.9c8bdf0a.9ad09.eb33@mx.google.com>

Author: fijal
Branch: 
Changeset: r92087:2c6fe9073d99
Date: 2017-08-05 10:12 +0200
http://bitbucket.org/pypy/pypy/changeset/2c6fe9073d99/

Log:	whack until things stop exploding. can't reproduce in a test

diff --git a/rpython/rlib/rpath.py b/rpython/rlib/rpath.py
--- a/rpython/rlib/rpath.py
+++ b/rpython/rlib/rpath.py
@@ -5,6 +5,7 @@
 import os, stat
 from rpython.rlib import rposix
 from rpython.rlib.signature import signature
+from rpython.rlib.rstring import assert_str0
 from rpython.annotator.model import s_Str0
 
 
@@ -31,9 +32,11 @@
     """Test whether a path is absolute"""
     return s.startswith('/')
 
+ at signature(s_Str0, returns=s_Str0)
 def _posix_rnormpath(path):
     """Normalize path, eliminating double slashes, etc."""
     slash, dot = '/', '.'
+    assert_str0(dot)
     if path == '':
         return dot
     initial_slashes = path.startswith('/')
@@ -56,6 +59,7 @@
     path = slash.join(comps)
     if initial_slashes:
         path = slash*initial_slashes + path
+    assert_str0(path)
     return path or dot
 
 @signature(s_Str0, returns=s_Str0)
@@ -66,6 +70,7 @@
         if not _posix_risabs(path):
             cwd = os.getcwd()
             path = _posix_rjoin(cwd, path)
+        assert path is not None
         return _posix_rnormpath(path)
     except OSError:
         return path

From pypy.commits at gmail.com  Sat Aug  5 04:21:51 2017
From: pypy.commits at gmail.com (fijal)
Date: Sat, 05 Aug 2017 01:21:51 -0700 (PDT)
Subject: [pypy-commit] pypy default: disable logging that's unneccessary
Message-ID: <5985801f.34a9df0a.bc750.3f45@mx.google.com>

Author: fijal
Branch: 
Changeset: r92088:dd32912dbb65
Date: 2017-08-05 10:21 +0200
http://bitbucket.org/pypy/pypy/changeset/dd32912dbb65/

Log:	disable logging that's unneccessary

diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -545,7 +545,7 @@
 
     def question(self, ask_gcc):
         try:
-            ask_gcc(self.name + ';')
+            ask_gcc('(void)' + self.name + ';')
             return True
         except CompilationError:
             return False
diff --git a/rpython/translator/platform/__init__.py b/rpython/translator/platform/__init__.py
--- a/rpython/translator/platform/__init__.py
+++ b/rpython/translator/platform/__init__.py
@@ -129,7 +129,7 @@
     # some helpers which seem to be cross-platform enough
 
     def _execute_c_compiler(self, cc, args, outname, cwd=None):
-        log.execute(cc + ' ' + ' '.join(args))
+        #log.execute(cc + ' ' + ' '.join(args))
         # 'cc' can also contain some options for the C compiler;
         # e.g. it can be "gcc -m32".  We handle it by splitting on ' '.
         cclist = cc.split()

From pypy.commits at gmail.com  Sat Aug  5 09:39:02 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 05 Aug 2017 06:39:02 -0700 (PDT)
Subject: [pypy-commit] pypy install-rpython: hg merge default
Message-ID: <5985ca76.0eae1c0a.f13ac.a2b8@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: install-rpython
Changeset: r92089:a1f3fe63fa46
Date: 2017-08-05 14:02 +0100
http://bitbucket.org/pypy/pypy/changeset/a1f3fe63fa46/

Log:	hg merge default

diff too long, truncating to 2000 out of 3400 lines

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -224,11 +224,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -288,7 +283,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
@@ -140,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -169,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -205,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst
--- a/pypy/doc/cpython_differences.rst
+++ b/pypy/doc/cpython_differences.rst
@@ -330,6 +330,8 @@
 
  - ``frozenset`` (empty frozenset only)
 
+ - unbound method objects (for Python 2 only)
+
 This change requires some changes to ``id`` as well. ``id`` fulfills the
 following condition: ``x is y <=> id(x) == id(y)``. Therefore ``id`` of the
 above types will return a value that is computed from the argument, and can
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -64,9 +64,9 @@
 cppyy
 -----
 
-For C++, `cppyy`_ is an automated bindings generator available for both
+For C++, _cppyy_ is an automated bindings generator available for both
 PyPy and CPython.
-``cppyy`` relies on declarations from C++ header files to dynamically
+_cppyy_ relies on declarations from C++ header files to dynamically
 construct Python equivalent classes, functions, variables, etc.
 It is designed for use by large scale programs and supports modern C++.
 With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
@@ -75,8 +75,7 @@
 To install, run ``pip install cppyy``.
 Further details are available in the `full documentation`_.
 
-.. _cppyy: http://cppyy.readthedocs.org/
-.. _`full documentation`: http://cppyy.readthedocs.org/
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
 
 RPython Mixed Modules
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -5,6 +5,14 @@
 .. this is a revision shortly after release-pypy2.7-v5.8.0
 .. startrev: 558bd00b3dd8
 
+In previous versions of PyPy, ``instance.method`` would return always
+the same bound method object, when gotten out of the same instance (as
+far as ``is`` and ``id()`` can tell).  CPython doesn't do that.  Now
+PyPy, like CPython, returns a different bound method object every time.
+For ``type.method``, PyPy2 still returns always the same *unbound*
+method object; CPython does it for built-in types but not for
+user-defined types.
+
 .. branch: cffi-complex
 .. branch: cffi-char16-char32
 
@@ -30,3 +38,25 @@
 
 Renaming of ``cppyy`` to ``_cppyy``.
 The former is now an external package installable with ``pip install cppyy``.
+
+.. branch: Enable_PGO_for_clang
+
+.. branch: nopax
+
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
+
+.. branch: pypy_bytearray
+
+Improve ``bytearray`` performance (backported from py3.5)
+
+.. branch: gc-del-limit-growth
+
+Fix the bounds in the GC when allocating a lot of objects with finalizers,
+fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 
 from pypy.interpreter.error import OperationError, oefmt
@@ -46,8 +47,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,6 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import rstack, rstackovf
 
 from pypy.interpreter import debug
@@ -57,8 +58,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -107,15 +109,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -142,8 +145,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,7 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
-from rpython.rlib.objectmodel import specialize
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -423,8 +423,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -559,21 +559,29 @@
         return space.newbool(space.eq_w(self.w_function, w_other.w_function))
 
     def is_w(self, space, other):
+        if self.w_instance is not None:
+            return W_Root.is_w(self, space, other)
+        # The following special-case is only for *unbound* method objects.
+        # Motivation: in CPython, it seems that no strange internal type
+        # exists where the equivalent of ``x.method is x.method`` would
+        # return True.  This is unlike unbound methods, where e.g.
+        # ``list.append is list.append`` returns True.  The following code
+        # is here to emulate that behaviour.  Unlike CPython, we return
+        # True for all equal unbound methods, not just for built-in types.
         if not isinstance(other, Method):
             return False
-        return (self.w_instance is other.w_instance and
+        return (other.w_instance is None and
                 self.w_function is other.w_function and
                 self.w_class is other.w_class)
 
     def immutable_unique_id(self, space):
-        from pypy.objspace.std.util import IDTAG_METHOD as tag
+        if self.w_instance is not None:
+            return W_Root.immutable_unique_id(self, space)
+        # the special-case is only for *unbound* method objects
+        #
+        from pypy.objspace.std.util import IDTAG_UNBOUND_METHOD as tag
         from pypy.objspace.std.util import IDTAG_SHIFT
-        if self.w_instance is not None:
-            id = space.bigint_w(space.id(self.w_instance))
-            id = id.lshift(LONG_BIT)
-        else:
-            id = rbigint.fromint(0)
-        id = id.or_(space.bigint_w(space.id(self.w_function)))
+        id = space.bigint_w(space.id(self.w_function))
         id = id.lshift(LONG_BIT).or_(space.bigint_w(space.id(self.w_class)))
         id = id.lshift(IDTAG_SHIFT).int_or_(tag)
         return space.newlong_from_rbigint(id)
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -64,8 +64,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -380,8 +380,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -621,9 +621,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -969,10 +969,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1013,8 +1013,8 @@
             self._staticdefs = zip(argnames[-len(defaults):], defaults)
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         defs_w = []
         for name, defaultval in self._staticdefs:
             if name.startswith('w_'):
@@ -1070,8 +1070,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs = gateway._getdefaults(space) # needs to be implemented by subclass
         code = gateway._code
@@ -1141,8 +1141,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1179,15 +1179,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1198,8 +1199,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1245,6 +1247,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,16 +18,17 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         self.lazy = True
         self.__class__.buildloaders()
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -61,8 +65,8 @@
         self.w_initialdict = self.space.call_method(self.w_dict, 'items')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -130,8 +134,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -161,8 +165,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space':space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -202,8 +206,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -40,13 +40,15 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -144,8 +145,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (we_are_translated, always_inline,
-        dont_inline)
+        dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -20,8 +20,8 @@
 from pypy.interpreter.pycode import PyCode, BytecodeCorruption
 from pypy.tool.stdlib_opcode import bytecode_spec
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -31,8 +31,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -1,4 +1,4 @@
-import pytest
+import pytest, sys
 from pypy.interpreter import eval
 from pypy.interpreter.function import Function, Method, descr_function_get
 from pypy.interpreter.pycode import PyCode
@@ -342,6 +342,11 @@
         raises(ValueError, type(f).__setstate__, f, (1, 2, 3))
 
 class AppTestMethod:
+    def setup_class(cls):
+        cls.w_runappdirect_on_cpython = cls.space.wrap(
+            cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
+
     def test_simple_call(self):
         class A(object):
             def func(self, arg2):
@@ -572,7 +577,6 @@
         assert meth == meth
         assert meth == MethodType(func, object)
 
-    @pytest.mark.skipif("config.option.runappdirect")
     def test_method_identity(self):
         class A(object):
             def m(self):
@@ -589,19 +593,24 @@
 
         a = A()
         a2 = A()
-        assert a.m is a.m
-        assert id(a.m) == id(a.m)
-        assert a.m is not a.n
-        assert id(a.m) != id(a.n)
-        assert a.m is not a2.m
-        assert id(a.m) != id(a2.m)
+        x = a.m; y = a.m
+        assert x is not y
+        assert id(x) != id(y)
+        assert x == y
+        assert x is not a.n
+        assert id(x) != id(a.n)
+        assert x is not a2.m
+        assert id(x) != id(a2.m)
 
-        assert A.m is A.m
-        assert id(A.m) == id(A.m)
-        assert A.m is not A.n
-        assert id(A.m) != id(A.n)
-        assert A.m is not B.m
-        assert id(A.m) != id(B.m)
+        if not self.runappdirect_on_cpython:
+            assert A.m is A.m
+            assert id(A.m) == id(A.m)
+        assert A.m == A.m
+        x = A.m
+        assert x is not A.n
+        assert id(x) != id(A.n)
+        assert x is not B.m
+        assert id(x) != id(B.m)
 
 
 class TestMethod:
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -8,14 +8,15 @@
 
 from rpython.rlib.jit import promote
 from rpython.rlib.objectmodel import compute_identity_hash, specialize
-from rpython.rlib.objectmodel import instantiate
+from rpython.rlib.objectmodel import instantiate, not_rpython
 from rpython.tool.sourcetools import compile2, func_with_new_name
 
 
 class TypeDef(object):
+    @not_rpython
     def __init__(self, __name, __base=None, __total_ordering__=None,
                  __buffer=None, **rawdict):
-        "NOT_RPYTHON: initialization-time only"
+        "initialization-time only"
         self.name = __name
         if __base is None:
             bases = []
@@ -113,8 +114,9 @@
 # register_finalizer() or not.
 
 @specialize.memo()
+ at not_rpython
 def get_unique_interplevel_subclass(space, cls):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert cls.typedef.acceptable_as_base_class
     try:
         return _unique_subclass_cache[cls]
@@ -349,15 +351,17 @@
         return self
 
 
+ at not_rpython
 def interp_attrproperty(name, cls, doc=None, wrapfn=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     assert wrapfn is not None
     def fget(space, obj):
         return getattr(space, wrapfn)(getattr(obj, name))
     return GetSetProperty(fget, cls=cls, doc=doc)
 
+ at not_rpython
 def interp_attrproperty_w(name, cls, doc=None):
-    "NOT_RPYTHON: initialization-time only"
+    "initialization-time only"
     def fget(space, obj):
         w_value = getattr(obj, name)
         if w_value is None:
diff --git a/pypy/module/__builtin__/test/test_classobj.py b/pypy/module/__builtin__/test/test_classobj.py
--- a/pypy/module/__builtin__/test/test_classobj.py
+++ b/pypy/module/__builtin__/test/test_classobj.py
@@ -1090,18 +1090,18 @@
     def setup_class(cls):
         if cls.runappdirect:
             py.test.skip("can only be run on py.py")
-        def is_strdict(space, w_class):
-            from pypy.objspace.std.dictmultiobject import BytesDictStrategy
+        def is_moduledict(space, w_class):
+            from pypy.objspace.std.celldict import ModuleDictStrategy
             w_d = w_class.getdict(space)
-            return space.wrap(isinstance(w_d.get_strategy(), BytesDictStrategy))
+            return space.wrap(isinstance(w_d.get_strategy(), ModuleDictStrategy))
 
-        cls.w_is_strdict = cls.space.wrap(gateway.interp2app(is_strdict))
+        cls.w_is_moduledict = cls.space.wrap(gateway.interp2app(is_moduledict))
 
-    def test_strdict(self):
+    def test_moduledict(self):
         class A:
             a = 1
             b = 2
-        assert self.is_strdict(A)
+        assert self.is_moduledict(A)
 
     def test_attr_slots(self):
         class C:
diff --git a/pypy/module/_cffi_backend/cdataobj.py b/pypy/module/_cffi_backend/cdataobj.py
--- a/pypy/module/_cffi_backend/cdataobj.py
+++ b/pypy/module/_cffi_backend/cdataobj.py
@@ -433,17 +433,22 @@
     def _sizeof(self):
         return self.ctype.size
 
-    def with_gc(self, w_destructor):
+    def with_gc(self, w_destructor, size=0):
         space = self.space
         if space.is_none(w_destructor):
             if isinstance(self, W_CDataGCP):
                 self.detach_destructor()
-                return space.w_None
-            raise oefmt(space.w_TypeError,
-                        "Can remove destructor only on a object "
-                        "previously returned by ffi.gc()")
-        with self as ptr:
-            return W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+                w_res = space.w_None
+            else:
+                raise oefmt(space.w_TypeError,
+                            "Can remove destructor only on a object "
+                            "previously returned by ffi.gc()")
+        else:
+            with self as ptr:
+                w_res = W_CDataGCP(space, ptr, self.ctype, self, w_destructor)
+        if size != 0:
+            rgc.add_memory_pressure(size)
+        return w_res
 
     def unpack(self, length):
         from pypy.module._cffi_backend.ctypeptr import W_CTypePtrOrArray
diff --git a/pypy/module/_cffi_backend/ffi_obj.py b/pypy/module/_cffi_backend/ffi_obj.py
--- a/pypy/module/_cffi_backend/ffi_obj.py
+++ b/pypy/module/_cffi_backend/ffi_obj.py
@@ -351,14 +351,14 @@
         return handle.from_handle(self.space, w_arg)
 
 
-    @unwrap_spec(w_cdata=W_CData)
-    def descr_gc(self, w_cdata, w_destructor):
+    @unwrap_spec(w_cdata=W_CData, size=int)
+    def descr_gc(self, w_cdata, w_destructor, size=0):
         """\
 Return a new cdata object that points to the same data.
 Later, when this new cdata object is garbage-collected,
 'destructor(old_cdata_object)' will be called."""
         #
-        return w_cdata.with_gc(w_destructor)
+        return w_cdata.with_gc(w_destructor, size)
 
 
     @unwrap_spec(replace_with='text')
diff --git a/pypy/module/_cffi_backend/func.py b/pypy/module/_cffi_backend/func.py
--- a/pypy/module/_cffi_backend/func.py
+++ b/pypy/module/_cffi_backend/func.py
@@ -257,6 +257,6 @@
 
 # ____________________________________________________________
 
- at unwrap_spec(w_cdata=cdataobj.W_CData)
-def gcp(space, w_cdata, w_destructor):
-    return w_cdata.with_gc(w_destructor)
+ at unwrap_spec(w_cdata=cdataobj.W_CData, size=int)
+def gcp(space, w_cdata, w_destructor, size=0):
+    return w_cdata.with_gc(w_destructor, size)
diff --git a/pypy/module/_cffi_backend/test/test_ffi_obj.py b/pypy/module/_cffi_backend/test/test_ffi_obj.py
--- a/pypy/module/_cffi_backend/test/test_ffi_obj.py
+++ b/pypy/module/_cffi_backend/test/test_ffi_obj.py
@@ -377,7 +377,7 @@
         raises(TypeError, ffi.gc, p, None)
         seen = []
         q1 = ffi.gc(p, lambda p: seen.append(1))
-        q2 = ffi.gc(q1, lambda p: seen.append(2))
+        q2 = ffi.gc(q1, lambda p: seen.append(2), size=123)
         import gc; gc.collect()
         assert seen == []
         assert ffi.gc(q1, None) is None
diff --git a/pypy/module/_codecs/__init__.py b/pypy/module/_codecs/__init__.py
--- a/pypy/module/_codecs/__init__.py
+++ b/pypy/module/_codecs/__init__.py
@@ -1,5 +1,6 @@
 from pypy.interpreter.mixedmodule import MixedModule
 from rpython.rlib import runicode
+from rpython.rlib.objectmodel import not_rpython
 from pypy.module._codecs import interp_codecs
 
 class Module(MixedModule):
@@ -86,9 +87,8 @@
          'unicode_internal_encode'   :  'interp_codecs.unicode_internal_encode',
     }
 
+    @not_rpython
     def __init__(self, space, *args):
-        "NOT_RPYTHON"
-
         # mbcs codec is Windows specific, and based on rffi.
         if (hasattr(runicode, 'str_decode_mbcs')):
             self.interpleveldefs['mbcs_encode'] = 'interp_codecs.mbcs_encode'
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -1,5 +1,5 @@
 from rpython.rlib import jit
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rstring import UnicodeBuilder
 from rpython.rlib.runicode import code_to_unichr, MAXUNICODE
 
@@ -268,8 +268,8 @@
         raise oefmt(space.w_TypeError,
                     "don't know how to handle %T in error callback", w_exc)
 
+ at not_rpython
 def register_builtin_error_handlers(space):
-    "NOT_RPYTHON"
     state = space.fromcache(CodecState)
     for error in ("strict", "ignore", "replace", "xmlcharrefreplace",
                   "backslashreplace"):
diff --git a/pypy/module/_vmprof/test/test__vmprof.py b/pypy/module/_vmprof/test/test__vmprof.py
--- a/pypy/module/_vmprof/test/test__vmprof.py
+++ b/pypy/module/_vmprof/test/test__vmprof.py
@@ -1,3 +1,4 @@
+import sys
 from rpython.tool.udir import udir
 from pypy.tool.pytest.objspace import gettestobjspace
 
@@ -7,6 +8,8 @@
     def setup_class(cls):
         cls.w_tmpfilename = cls.space.wrap(str(udir.join('test__vmprof.1')))
         cls.w_tmpfilename2 = cls.space.wrap(str(udir.join('test__vmprof.2')))
+        cls.w_plain = cls.space.wrap(not cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
 
     def test_import_vmprof(self):
         tmpfile = open(self.tmpfilename, 'wb')
@@ -117,6 +120,8 @@
         assert _vmprof.get_profile_path() is None
 
     def test_stop_sampling(self):
+        if not self.plain:
+            skip("unreliable test except on CPython without -A")
         import os
         import _vmprof
         tmpfile = open(self.tmpfilename, 'wb')
diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
--- a/pypy/module/array/interp_array.py
+++ b/pypy/module/array/interp_array.py
@@ -118,6 +118,29 @@
         return space.w_True
     return space.w_False
 
+index_count_jd = jit.JitDriver(
+    greens = ['count', 'arrclass', 'tp_item'],
+    reds = 'auto', name = 'array.index_or_count')
+
+def index_count_array(arr, w_val, count=False):
+    space = arr.space
+    tp_item = space.type(w_val)
+    arrclass = arr.__class__
+    cnt = 0
+    for i in range(arr.len):
+        index_count_jd.jit_merge_point(
+            tp_item=tp_item, count=count,
+            arrclass=arrclass)
+        w_item = arr.w_getitem(space, i)
+        if space.eq_w(w_item, w_val):
+            if count:
+                cnt += 1
+            else:
+                return i
+    if count:
+        return cnt
+    return -1
+
 UNICODE_ARRAY = lltype.Ptr(lltype.Array(lltype.UniChar,
                                         hints={'nolength': True}))
 
@@ -257,17 +280,12 @@
         """
         self.extend(w_x)
 
-    def descr_count(self, space, w_val):
+    def descr_count(self, space, w_x):
         """ count(x)
 
         Return number of occurrences of x in the array.
         """
-        cnt = 0
-        for i in range(self.len):
-            # XXX jitdriver
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_val):
-                cnt += 1
+        cnt = index_count_array(self, w_x, count=True)
         return space.newint(cnt)
 
     def descr_index(self, space, w_x):
@@ -275,10 +293,9 @@
 
         Return index of first occurrence of x in the array.
         """
-        for i in range(self.len):
-            w_item = self.w_getitem(space, i)
-            if space.eq_w(w_item, w_x):
-                return space.newint(i)
+        res = index_count_array(self, w_x, count=False)
+        if res >= 0:
+            return space.newint(res)
         raise oefmt(space.w_ValueError, "array.index(x): x not in list")
 
     def descr_reverse(self, space):
@@ -752,7 +769,9 @@
 
 class TypeCode(object):
     def __init__(self, itemtype, unwrap, canoverflow=False, signed=False,
-                 method='__int__'):
+                 method='__int__', errorname=None):
+        if errorname is None:
+            errorname = unwrap[:-2]
         self.itemtype = itemtype
         self.bytes = rffi.sizeof(itemtype)
         self.arraytype = lltype.Array(itemtype, hints={'nolength': True})
@@ -762,6 +781,7 @@
         self.canoverflow = canoverflow
         self.w_class = None
         self.method = method
+        self.errorname = errorname
 
     def _freeze_(self):
         # hint for the annotator: track individual constant instances
@@ -785,8 +805,8 @@
     'i': TypeCode(rffi.INT,           'int_w', True, True),
     'I': _UINTTypeCode,
     'l': TypeCode(rffi.LONG,          'int_w', True, True),
-    'L': TypeCode(rffi.ULONG,         'bigint_w'),  # Overflow handled by
-                                                    # rbigint.touint() which
+    'L': TypeCode(rffi.ULONG,         'bigint_w',   # Overflow handled by
+                  errorname="integer"),             # rbigint.touint() which
                                                     # corresponds to the
                                                     # C-type unsigned long
     'f': TypeCode(lltype.SingleFloat, 'float_w', method='__float__'),
@@ -864,7 +884,7 @@
                         item = unwrap(space.call_method(w_item, mytype.method))
                     except OperationError:
                         raise oefmt(space.w_TypeError,
-                                    "array item must be " + mytype.unwrap[:-2])
+                                    "array item must be " + mytype.errorname)
                 else:
                     raise
             if mytype.unwrap == 'bigint_w':
diff --git a/pypy/module/array/test/test_array.py b/pypy/module/array/test/test_array.py
--- a/pypy/module/array/test/test_array.py
+++ b/pypy/module/array/test/test_array.py
@@ -162,6 +162,11 @@
             raises(OverflowError, a.append, -1)
             raises(OverflowError, a.append, 2 ** (8 * b))
 
+    def test_errormessage(self):
+        a = self.array("L", [1, 2, 3])
+        excinfo = raises(TypeError, "a[0] = 'abc'")
+        assert str(excinfo.value) == "array item must be integer"
+
     def test_fromstring(self):
         import sys
 
diff --git a/pypy/module/cppyy/test/test_cint.py b/pypy/module/cppyy/test/test_cint.py
deleted file mode 100644
--- a/pypy/module/cppyy/test/test_cint.py
+++ /dev/null
@@ -1,710 +0,0 @@
-import py, os, sys
-
-# These tests are for the CINT backend only (they exercise ROOT features
-# and classes that are not loaded/available with the Reflex backend). At
-# some point, these tests are likely covered by the CLang/LLVM backend.
-from pypy.module.cppyy import capi
-if capi.identify() != 'CINT':
-    py.test.skip("backend-specific: CINT-only tests")
-
-# load _cffi_backend early, or its global vars are counted as leaks in the
-# test (note that the module is not otherwise used in the test itself)
-from pypy.module._cffi_backend import newtype
-
-currpath = py.path.local(__file__).dirpath()
-iotypes_dct = str(currpath.join("iotypesDict.so"))
-
-def setup_module(mod):
-    if sys.platform == 'win32':
-        py.test.skip("win32 not supported so far")
-    err = os.system("cd '%s' && make CINT=t iotypesDict.so" % currpath)
-    if err:
-        raise OSError("'make' failed (see stderr)")
-
-class AppTestCINT:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_globals(self):
-        """Test the availability of ROOT globals"""
-
-        import cppyy
-
-        assert cppyy.gbl.gROOT
-        assert cppyy.gbl.gApplication
-        assert cppyy.gbl.gSystem
-        assert cppyy.gbl.TInterpreter.Instance()           # compiled
-        assert cppyy.gbl.TInterpreter                      # interpreted
-        assert cppyy.gbl.TDirectory.CurrentDirectory()     # compiled
-        assert cppyy.gbl.TDirectory                        # interpreted
-
-    def test02_write_access_to_globals(self):
-        """Test overwritability of ROOT globals"""
-
-        import cppyy
-
-        oldval = cppyy.gbl.gDebug
-        assert oldval != 3
-
-        proxy = cppyy.gbl.__class__.__dict__['gDebug']
-        cppyy.gbl.gDebug = 3
-        assert proxy.__get__(proxy, None) == 3
-
-        # this is where this test differs from test03_write_access_to_globals
-        # in test_pythonify.py
-        cppyy.gbl.gROOT.ProcessLine('int gDebugCopy = gDebug;')
-        assert cppyy.gbl.gDebugCopy == 3
-
-        cppyy.gbl.gDebug = oldval
-
-    def test03_create_access_to_globals(self):
-        """Test creation and access of new ROOT globals"""
-
-        import cppyy
-
-        cppyy.gbl.gROOT.ProcessLine('double gMyOwnGlobal = 3.1415')
-        assert cppyy.gbl.gMyOwnGlobal == 3.1415
-
-        proxy = cppyy.gbl.__class__.__dict__['gMyOwnGlobal']
-        assert proxy.__get__(proxy, None) == 3.1415
-
-    def test04_auto_loading(self):
-        """Test auto-loading by retrieving a non-preloaded class"""
-
-        import cppyy
-
-        l = cppyy.gbl.TLorentzVector()
-        assert isinstance(l, cppyy.gbl.TLorentzVector)
-
-    def test05_macro_loading(self):
-        """Test accessibility to macro classes"""
-
-        import cppyy
-
-        loadres = cppyy.gbl.gROOT.LoadMacro('simple_class.C')
-        assert loadres == 0
-
-        base = cppyy.gbl.MySimpleBase
-        simple = cppyy.gbl.MySimpleDerived
-        simple_t = cppyy.gbl.MySimpleDerived_t
-
-        assert issubclass(simple, base)
-        assert simple is simple_t
-
-        c = simple()
-        assert isinstance(c, simple)
-        assert c.m_data == c.get_data()
-
-        c.set_data(13)
-        assert c.m_data == 13
-        assert c.get_data() == 13
-
-
-class AppTestCINTPYTHONIZATIONS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def test01_strings(self):
-        """Test TString/TObjString compatibility"""
-
-        import cppyy
-
-        pyteststr = "aap noot mies"
-        def test_string(s1, s2):
-            assert len(s1) == len(s2)
-            assert s1 == s1
-            assert s1 == s2
-            assert s1 == str(s1)
-            assert s1 == pyteststr
-            assert s1 != "aap"
-            assert s1 != ""
-            assert s1 < "noot"
-            assert repr(s1) == repr(s2)
-
-        s1 = cppyy.gbl.TString(pyteststr)
-        test_string(s1, pyteststr)
-
-        s3 = cppyy.gbl.TObjString(pyteststr)
-        test_string(s3, pyteststr)
-
-    def test03_TVector(self):
-        """Test TVector2/3/T behavior"""
-
-        import cppyy, math
-
-        N = 51
-
-        # TVectorF is a typedef of floats
-        v = cppyy.gbl.TVectorF(N)
-        for i in range(N):
-            v[i] = i*i
-
-        assert len(v) == N
-        for j in v:
-            assert round(v[int(math.sqrt(j)+0.5)]-j, 5) == 0.
-
-    def test04_TStringTObjString(self):
-        """Test string/TString interchangebility"""
-
-        import cppyy
-
-        test = "aap noot mies"
-
-        s1 = cppyy.gbl.TString(test )
-        s2 = str(s1)
-
-        assert s1 == test
-        assert test == s2
-        assert s1 == s2
-
-        s3 = cppyy.gbl.TObjString(s2)
-        assert s3 == test
-        assert s2 == s3
-
-        # force use of: TNamed(const TString &name, const TString &title)
-        n = cppyy.gbl.TNamed(test, cppyy.gbl.TString("title"))
-        assert n.GetTitle() == "title"
-        assert n.GetName() == test
-
-
-class AppTestCINTTTREE:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    def setup_class(cls):
-        cls.w_N = cls.space.newint(5)
-        cls.w_M = cls.space.newint(10)
-        cls.w_fname = cls.space.newtext("test.root")
-        cls.w_tname = cls.space.newtext("test")
-        cls.w_title = cls.space.newtext("test tree")
-        cls.w_iotypes = cls.space.appexec([], """():
-            import cppyy
-            return cppyy.load_reflection_info(%r)""" % (iotypes_dct,))
-
-    def test01_write_stdvector(self):
-        """Test writing of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        v = vector("double")()
-        raises(TypeError, TTree.Branch, None, "mydata", v.__class__.__name__, v)
-        raises(TypeError, TTree.Branch, v, "mydata", v.__class__.__name__, v)
-
-        mytree.Branch("mydata", v.__class__.__name__, v)
-
-        for i in range(self.N):
-            for j in range(self.M):
-                v.push_back(i*self.M+j)
-            mytree.Fill()
-            v.clear()
-        f.Write()
-        f.Close()
-
-    def test02_file_open(self):
-
-        from cppyy import gbl
-
-        f = gbl.TFile.Open(self.fname)
-        s = str(f)            # should not raise
-        r = repr(f)
-
-        f.Close()
-
-    def test03_read_stdvector(self):
-        """Test reading of a single branched TTree with an std::vector<double>"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        i = 0
-        for event in mytree:
-            assert len(event.mydata) == self.M
-            for entry in event.mydata:
-                assert i == int(entry)
-                i += 1
-        assert i == self.N * self.M
-
-        f.Close()
-
-    def test04_write_some_data_object(self):
-        """Test writing of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree, IO
-        from cppyy.gbl.IO import SomeDataObject
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-
-        d = SomeDataObject()
-        b = mytree.Branch("data", d)
-        mytree._python_owns = False
-        assert b
-
-        for i in range(self.N):
-            for j in range(self.M):
-                d.add_float(i*self.M+j)
-            d.add_tuple(d.get_floats())
-
-            mytree.Fill()
-
-        f.Write()
-        f.Close()
-
-    def test05_read_some_data_object(self):
-        """Test reading of a complex data object"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        j = 1
-        for event in mytree:
-            i = 0
-            assert len(event.data.get_floats()) == j*self.M
-            for entry in event.data.get_floats():
-                assert i == int(entry)
-                i += 1
-
-            k = 1
-            assert len(event.data.get_tuples()) == j
-            for mytuple in event.data.get_tuples():
-                i = 0
-                assert len(mytuple) == k*self.M
-                for entry in mytuple:
-                    assert i == int(entry)
-                    i += 1
-                k += 1
-            j += 1
-        assert j-1 == self.N
-        #
-        f.Close()
-
-    def test06_branch_activation(self):
-        """Test of automatic branch activation"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        L = 5
-
-        # writing
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        for i in range(L):
-            v = vector("double")()
-            mytree.Branch("mydata_%d"%i, v.__class__.__name__, v)
-            mytree.__dict__["v_%d"%i] = v
-
-        for i in range(self.N):
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                for j in range(self.M):
-                    mytree.__dict__["v_%d"%k].push_back(i*self.M+j*L+k)
-            mytree.Fill()
-            for k in range(L):
-                v = mytree.__dict__["v_%d"%k]
-                v.clear()
-        f.Write()
-        f.Close()
-
-        del mytree, f
-        import gc
-        gc.collect()
-
-        # reading
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # force (initial) disabling of all branches
-        mytree.SetBranchStatus("*",0);
-
-        i = 0
-        for event in mytree:
-            for k in range(L):
-                j = 0
-                data = getattr(mytree, "mydata_%d"%k)
-                assert len(data) == self.M
-                for entry in data:
-                    assert entry == i*self.M+j*L+k
-                    j += 1
-                assert j == self.M
-            i += 1
-        assert i == self.N
-
-        f.Close()
-
-    def test07_write_builtin(self):
-        """Test writing of builtins"""
-
-        from cppyy import gbl               # bootstraps, only needed for tests
-        from cppyy.gbl import TFile, TTree
-        from cppyy.gbl.std import vector
-
-        f = TFile(self.fname, "RECREATE")
-        mytree = TTree(self.tname, self.title)
-        mytree._python_owns = False
-
-        import array
-        mytree.ba = array.array('c', [chr(0)])
-        mytree.ia = array.array('i', [0])
-        mytree.da = array.array('d', [0.])
-
-        mytree.Branch("my_bool",   mytree.ba, "my_bool/O")
-        mytree.Branch("my_int",    mytree.ia, "my_int/I")
-        mytree.Branch("my_int2",   mytree.ia, "my_int2/I")
-        mytree.Branch("my_double", mytree.da, "my_double/D")
-
-        for i in range(self.N):
-            # make sure value is different from default (0)
-            mytree.ba[0] = i%2 and chr(0) or chr(1)
-            mytree.ia[0] = i+1
-            mytree.da[0] = (i+1)/2.
-            mytree.Fill()
-        f.Write()
-        f.Close()
-
-    def test08_read_builtin(self):
-        """Test reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        raises(AttributeError, getattr, mytree, "does_not_exist")
-
-        i = 1
-        for event in mytree:
-            assert event.my_bool   == (i-1)%2 and 0 or 1
-            assert event.my_int    == i
-            assert event.my_double == i/2.
-            i += 1
-        assert (i-1) == self.N
-
-        f.Close()
-
-    def test09_user_read_builtin(self):
-        """Test user-directed reading of builtins"""
-
-        from cppyy import gbl
-        from cppyy.gbl import TFile
-
-        f = TFile(self.fname)
-        mytree = f.Get(self.tname)
-
-        # note, this is an old, annoted tree from test08
-        for i in range(3, mytree.GetEntriesFast()):
-            mytree.GetEntry(i)
-            assert mytree.my_int  == i+1
-            assert mytree.my_int2 == i+1
-
-        f.Close()
-
-class AppTestCINTREGRESSION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that at some point in the past resulted in failures on
-    # PyROOT; kept here to confirm no regression from PyROOT
-
-    def test01_regression(self):
-        """TPaveText::AddText() used to result in KeyError"""
-
-        # This is where the original problem was discovered, and the test is
-        # left in. However, the detailed underlying problem, as well as the
-        # solution to it, is tested in test_fragile.py
-
-        from cppyy import gbl
-        from cppyy.gbl import TPaveText
-
-        hello = TPaveText( .1, .8, .9, .97 )
-        hello.AddText( 'Hello, World!' )
-
-
-class AppTestCINTFUNCTION:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-    _pypytest_leaks = None   # TODO: figure out the false positives
-
-    # test the function callbacks; this does not work with Reflex, as it can
-    # not generate functions on the fly (it might with cffi?)
-
-    @py.test.mark.dont_track_allocations("TODO: understand; initialization left-over?")
-    def test01_global_function_callback(self):
-        """Test callback of a python global function"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        def identity(x):
-            return x[0]
-
-        f = TF1("pyf1", identity, -1., 1., 0)
-
-        assert f.Eval(0.5)  == 0.5
-        assert f.Eval(-10.) == -10.
-        assert f.Eval(1.0)  == 1.0
-
-        # check proper propagation of default value
-        f = TF1("pyf1d", identity, -1., 1.)
-
-        assert f.Eval(0.5) == 0.5
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test02_callable_object_callback(self):
-        """Test callback of a python callable object"""
-
-        import cppyy, gc
-        TF1 = cppyy.gbl.TF1
-
-        class Linear:
-            def __call__(self, x, par):
-                return par[0] + x[0]*par[1]
-
-        f = TF1("pyf2", Linear(), -1., 1., 2)
-        f.SetParameters(5., 2.)
-
-        assert f.Eval(-0.1) == 4.8
-        assert f.Eval(1.3)  == 7.6
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-    def test03_fit_with_python_gaussian(self):
-        """Test fitting with a python global function"""
-
-        # note: this function is dread-fully slow when running testing un-translated
-
-        import cppyy, gc, math
-        TF1, TH1F = cppyy.gbl.TF1, cppyy.gbl.TH1F
-
-        def pygaus(x, par):
-            arg1 = 0
-            scale1 = 0
-            ddx = 0.01
-
-            if (par[2] != 0.0):
-                arg1 = (x[0]-par[1])/par[2]
-                scale1 = (ddx*0.39894228)/par[2]
-                h1 = par[0]/(1+par[3])
-
-                gauss = h1*scale1*math.exp(-0.5*arg1*arg1)
-            else:
-                gauss = 0.
-            return gauss
-
-        f = TF1("pygaus", pygaus, -4, 4, 4)
-        f.SetParameters(600, 0.43, 0.35, 600)
-
-        h = TH1F("h", "test", 100, -4, 4)
-        h.FillRandom("gaus", 200000)
-        h.Fit(f, "0Q")
-
-        assert f.GetNDF() == 96
-        result = f.GetParameters()
-        assert round(result[1] - 0., 1) == 0  # mean
-        assert round(result[2] - 1., 1) == 0  # s.d.
-
-        del f      # force here, to prevent leak-check complaints
-        gc.collect()
-
-
-class AppTestSURPLUS:
-    spaceconfig = dict(usemodules=['cppyy', '_rawffi', 'itertools'])
-
-    # these are tests that were historically exercised on ROOT classes and
-    # have twins on custom classes; kept here just in case differences crop
-    # up between the ROOT classes and the custom ones
-
-    def test01_class_enum(self):
-        """Test class enum access and values"""
-
-        import cppyy
-        TObject = cppyy.gbl.TObject
-        gROOT = cppyy.gbl.gROOT
-
-        assert TObject.kBitMask    == gROOT.ProcessLine("return TObject::kBitMask;")
-        assert TObject.kIsOnHeap   == gROOT.ProcessLine("return TObject::kIsOnHeap;")
-        assert TObject.kNotDeleted == gROOT.ProcessLine("return TObject::kNotDeleted;")
-        assert TObject.kZombie     == gROOT.ProcessLine("return TObject::kZombie;")
-
-        t = TObject()
-
-        assert TObject.kBitMask    == t.kBitMask
-        assert TObject.kIsOnHeap   == t.kIsOnHeap
-        assert TObject.kNotDeleted == t.kNotDeleted
-        assert TObject.kZombie     == t.kZombie
-
-    def test02_global_enum(self):
-        """Test global enums access and values"""
-
-        import cppyy
-        from cppyy import gbl
-
-        assert gbl.kRed   == gbl.gROOT.ProcessLine("return kRed;")
-        assert gbl.kGreen == gbl.gROOT.ProcessLine("return kGreen;")
-        assert gbl.kBlue  == gbl.gROOT.ProcessLine("return kBlue;")
-
-    def test03_copy_contructor(self):
-        """Test copy constructor"""
-
-        import cppyy
-        TLorentzVector = cppyy.gbl.TLorentzVector
-
-        t1 = TLorentzVector(1., 2., 3., -4.)
-        t2 = TLorentzVector(0., 0., 0.,  0.)
-        t3 = TLorentzVector(t1)
-
-        assert t1 == t3
-        assert t1 != t2
-
-        for i in range(4):
-            assert t1[i] == t3[i]
-
-    def test04_object_validity(self):
-        """Test object validity checking"""
-
-        import cppyy
-
-        t1 = cppyy.gbl.TObject()
-
-        assert t1
-        assert not not t1
-
-        t2 = cppyy.gbl.gROOT.FindObject("Nah, I don't exist")
-
-        assert not t2
-
-    def test05_element_access(self):
-        """Test access to elements in matrix and array objects."""
-
-        from cppyy import gbl
-
-        N = 3
-        v = gbl.TVectorF(N)
-        m = gbl.TMatrixD(N, N)
-
-        for i in range(N):
-            assert v[i] == 0.0
-
-            for j in range(N):
-                assert m[i][j] == 0.0
-
-    def test06_static_function_call( self ):
-        """Test call to static function."""
-
-        import cppyy
-        TROOT, gROOT = cppyy.gbl.TROOT, cppyy.gbl.gROOT
-
-        c1 = TROOT.Class()
-        assert not not c1
-
-        c2 = gROOT.Class()
-
-        assert c1 == c2
-
-        old = gROOT.GetDirLevel()
-        TROOT.SetDirLevel(2)
-        assert 2 == gROOT.GetDirLevel()
-        gROOT.SetDirLevel(old)
-
-        old = TROOT.GetDirLevel()
-        gROOT.SetDirLevel(3)
-        assert 3 == TROOT.GetDirLevel()
-        TROOT.SetDirLevel(old)
-
-    def test07_macro(self):
-        """Test access to cpp macro's"""
-
-        from cppyy import gbl
-
-        assert gbl.NULL == 0
-
-        gbl.gROOT.ProcessLine('#define aap "aap"')
-        gbl.gROOT.ProcessLine('#define noot 1')
-        gbl.gROOT.ProcessLine('#define mies 2.0')
-
-        # TODO: macro's assumed to always be of long type ...
-        #assert gbl.aap  == "aap"
-        assert gbl.noot == 1
-        #assert gbl.mies == 2.0
-
-    def test08_opaque_pointer_passing(self):
-        """Test passing around of opaque pointers"""
-
-        import cppyy
-
-        # TODO: figure out CObject (see also test_advanced.py)
-
-        s = cppyy.gbl.TString("Hello World!")
-        #cobj = cppyy.as_cobject(s)
-        addr = cppyy.addressof(s)
-
-        #assert s == cppyy.bind_object(cobj, s.__class__)
-        #assert s == cppyy.bind_object(cobj, "TString")
-        assert s == cppyy.bind_object(addr, s.__class__)
-        assert s == cppyy.bind_object(addr, "TString")
-
-    def test09_object_and_pointer_comparisons(self):
-        """Verify object and pointer comparisons"""
-
-        import cppyy
-        gbl = cppyy.gbl
-
-        c1 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == None
-        assert None == c1
-
-        c2 = cppyy.bind_object(0, gbl.TCanvas)
-        assert c1 == c2
-        assert c2 == c1
-
-        # TLorentzVector overrides operator==
-        l1 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == None
-        assert None == l1
-
-        assert c1 != l1
-        assert l1 != c1
-
-        l2 = cppyy.bind_object(0, gbl.TLorentzVector)
-        assert l1 == l2
-        assert l2 == l1 
-
-        l3 = gbl.TLorentzVector(1, 2, 3, 4)
-        l4 = gbl.TLorentzVector(1, 2, 3, 4)
-        l5 = gbl.TLorentzVector(4, 3, 2, 1)
-        assert l3 == l4
-        assert l4 == l3
-
-        assert l3 != None                 # like this to ensure __ne__ is called
-        assert None != l3                 # id.
-        assert l3 != l5
-        assert l5 != l3
-
-    def test10_recursive_remove(self):
-        """Verify that objects are recursively removed when destroyed"""
-
-        import cppyy
-
-        c = cppyy.gbl.TClass.GetClass("TObject")
-

From pypy.commits at gmail.com  Sat Aug  5 14:18:04 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 05 Aug 2017 11:18:04 -0700 (PDT)
Subject: [pypy-commit] pypy install-rpython: Fix MANIFEST.in again,
 bump version number
Message-ID: <59860bdc.4aa8df0a.79410.7c1d@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: install-rpython
Changeset: r92090:9842d8147b5a
Date: 2017-08-05 14:15 +0100
http://bitbucket.org/pypy/pypy/changeset/9842d8147b5a/

Log:	Fix MANIFEST.in again, bump version number

diff --git a/MANIFEST.in b/MANIFEST.in
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,2 +1,4 @@
 include README-rpython.rst
 exclude README.rst
+recursive-include rpython *.c *.h
+recursive-include rpython/rlib/vmprof/src/
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,7 @@
 
 setup(
     name='rpython',
-    version='0.2.0',
+    version='0.2.1',
     description='RPython',
     long_description=long_description,
 

From pypy.commits at gmail.com  Sat Aug  5 15:11:39 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 05 Aug 2017 12:11:39 -0700 (PDT)
Subject: [pypy-commit] pypy default: jit-log-noopt logging was broken by
 2bf0191fb21d
Message-ID: <5986186b.4f821c0a.36e7a.88c8@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92091:eef485998ca1
Date: 2017-08-05 21:11 +0200
http://bitbucket.org/pypy/pypy/changeset/eef485998ca1/

Log:	jit-log-noopt logging was broken by 2bf0191fb21d

	(the have_debug_prints needs to be *inside* the debug_start,
	otherwise stuff is never logged.)

diff --git a/rpython/jit/metainterp/logger.py b/rpython/jit/metainterp/logger.py
--- a/rpython/jit/metainterp/logger.py
+++ b/rpython/jit/metainterp/logger.py
@@ -13,10 +13,11 @@
         self.guard_number = guard_number
 
     def log_loop_from_trace(self, trace, memo):
+        debug_start("jit-log-noopt")
         if not have_debug_prints():
+            debug_stop("jit-log-noopt")
             return
         inputargs, ops = self._unpack_trace(trace)
-        debug_start("jit-log-noopt")
         debug_print("# Traced loop or bridge with", len(ops), "ops")
         logops = self._log_operations(inputargs, ops, None, memo)
         debug_stop("jit-log-noopt")

From pypy.commits at gmail.com  Sat Aug  5 16:33:20 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 05 Aug 2017 13:33:20 -0700 (PDT)
Subject: [pypy-commit] pypy default: remove bridge potential in hashing (int,
 int) tuples
Message-ID: <59862b90.56bf1c0a.fe576.bc0f@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92092:b235c624a167
Date: 2017-08-05 22:32 +0200
http://bitbucket.org/pypy/pypy/changeset/b235c624a167/

Log:	remove bridge potential in hashing (int, int) tuples

diff --git a/pypy/objspace/std/specialisedtupleobject.py b/pypy/objspace/std/specialisedtupleobject.py
--- a/pypy/objspace/std/specialisedtupleobject.py
+++ b/pypy/objspace/std/specialisedtupleobject.py
@@ -74,8 +74,7 @@
                 elif typetuple[i] == int:
                     # mimic cpythons behavior of a hash value of -2 for -1
                     y = value
-                    if y == -1:
-                        y = -2
+                    y -= (y == -1)  # No explicit condition, to avoid JIT bridges
                 elif typetuple[i] == float:
                     # get the correct hash for float which is an
                     # integer & other less frequent cases
diff --git a/pypy/objspace/std/test/test_specialisedtupleobject.py b/pypy/objspace/std/test/test_specialisedtupleobject.py
--- a/pypy/objspace/std/test/test_specialisedtupleobject.py
+++ b/pypy/objspace/std/test/test_specialisedtupleobject.py
@@ -37,6 +37,7 @@
                     self.space.eq(self.space.hash(N_w_tuple),
                                   self.space.hash(S_w_tuple)))
 
+        hash_test([-1, -1])
         hash_test([1, 2])
         hash_test([1.5, 2.8])
         hash_test([1.0, 2.0])

From pypy.commits at gmail.com  Sat Aug  5 16:50:05 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 05 Aug 2017 13:50:05 -0700 (PDT)
Subject: [pypy-commit] pypy default: document merged branch
Message-ID: <59862f7d.5ba4df0a.7e1b3.1799@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92093:694a045a39eb
Date: 2017-08-05 22:49 +0200
http://bitbucket.org/pypy/pypy/changeset/694a045a39eb/

Log:	document merged branch

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -60,3 +60,11 @@
 
 Small improvement to optimize list accesses with constant indexes better by
 throwing away information about them less eagerly.
+
+
+.. branch: getarrayitem-into-bridges:
+
+More information is retained into a bridge: knowledge about the content of
+arrays (at fixed indices) is stored in guards (and thus available at the
+beginning of bridges). Also, some better feeding of information about known
+fields of constant objects into bridges.

From pypy.commits at gmail.com  Sat Aug  5 16:50:39 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 05 Aug 2017 13:50:39 -0700 (PDT)
Subject: [pypy-commit] pypy default: fix hashing of float tuples
Message-ID: <59862f9f.c1a41c0a.c943c.e51e@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92094:5c090931a660
Date: 2017-08-05 22:50 +0200
http://bitbucket.org/pypy/pypy/changeset/5c090931a660/

Log:	fix hashing of float tuples

diff --git a/pypy/objspace/std/specialisedtupleobject.py b/pypy/objspace/std/specialisedtupleobject.py
--- a/pypy/objspace/std/specialisedtupleobject.py
+++ b/pypy/objspace/std/specialisedtupleobject.py
@@ -80,8 +80,9 @@
                     # integer & other less frequent cases
                     from pypy.objspace.std.floatobject import _hash_float
                     y = _hash_float(space, value)
+                    y -= (y == -1)
                 else:
-                    y = compute_hash(value)
+                    assert 0, "unreachable"
                 x = (x ^ y) * mult
                 z -= 1
                 mult += 82520 + z + z
diff --git a/pypy/objspace/std/test/test_specialisedtupleobject.py b/pypy/objspace/std/test/test_specialisedtupleobject.py
--- a/pypy/objspace/std/test/test_specialisedtupleobject.py
+++ b/pypy/objspace/std/test/test_specialisedtupleobject.py
@@ -38,6 +38,7 @@
                                   self.space.hash(S_w_tuple)))
 
         hash_test([-1, -1])
+        hash_test([-1.0, -1.0])
         hash_test([1, 2])
         hash_test([1.5, 2.8])
         hash_test([1.0, 2.0])

From pypy.commits at gmail.com  Sat Aug  5 21:19:04 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 05 Aug 2017 18:19:04 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-leakchecking: Fix tests for MSVC
Message-ID: <59866e88.e681df0a.700d1.739e@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: cpyext-leakchecking
Changeset: r92095:7b5f5263c9de
Date: 2017-08-06 02:18 +0100
http://bitbucket.org/pypy/pypy/changeset/7b5f5263c9de/

Log:	Fix tests for MSVC

diff --git a/pypy/module/cpyext/test/array.c b/pypy/module/cpyext/test/array.c
--- a/pypy/module/cpyext/test/array.c
+++ b/pypy/module/cpyext/test/array.c
@@ -1864,11 +1864,12 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject *v = getarrayitem(obj2, 0);
         int i = ((PyIntObject*)v)->ob_ival;
         Py_DECREF(v);
-        PyObject * ret = PyList_New(n*i);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -1881,11 +1882,12 @@
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject *v = getarrayitem(obj1, 0);
         int i = ((PyIntObject*)v)->ob_ival;
         Py_DECREF(v);
-        PyObject * ret = PyList_New(n*i);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -1918,11 +1920,12 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject *v = getarrayitem(obj2, 0);
         int i = ((PyIntObject*)v)->ob_ival;
         Py_DECREF(v);
-        PyObject * ret = PyList_New(n);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             v = PyList_GetItem(obj1, nn);
@@ -1939,11 +1942,12 @@
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject *v = getarrayitem(obj1, 0);
         int i = ((PyIntObject*)v)->ob_ival;
         Py_DECREF(v);
-        PyObject * ret = PyList_New(n);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             v = PyList_GetItem(obj2, nn);
diff --git a/pypy/module/cpyext/test/test_dictobject.py b/pypy/module/cpyext/test/test_dictobject.py
--- a/pypy/module/cpyext/test/test_dictobject.py
+++ b/pypy/module/cpyext/test/test_dictobject.py
@@ -214,7 +214,7 @@
             ("keys_and_values", "METH_O",
              '''
              Py_ssize_t pos = 0;
-             PyObject *key, *value;
+             PyObject *key, *value, *values;
              PyObject* keys = PyList_New(0);
              while (PyDict_Next(args, &pos, &key, NULL))
              {
@@ -225,7 +225,7 @@
                 }
              }
              pos = 0;
-             PyObject* values = PyList_New(0);
+             values = PyList_New(0);
              while (PyDict_Next(args, &pos, NULL, &value))
              {
                 if (PyList_Append(values, value) < 0)

From pypy.commits at gmail.com  Sat Aug  5 22:09:59 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 05 Aug 2017 19:09:59 -0700 (PDT)
Subject: [pypy-commit] pypy default: document merged branch
Message-ID: <59867a77.97a0df0a.b1ca5.d84d@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: 
Changeset: r92098:8c10f7297290
Date: 2017-08-06 03:09 +0100
http://bitbucket.org/pypy/pypy/changeset/8c10f7297290/

Log:	document merged branch

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -68,3 +68,8 @@
 arrays (at fixed indices) is stored in guards (and thus available at the
 beginning of bridges). Also, some better feeding of information about known
 fields of constant objects into bridges.
+
+.. branch: cpyext-leakchecking
+
+Add support for leakfinder in cpyext tests (disabled for now, due to too many
+failures).

From pypy.commits at gmail.com  Sun Aug  6 03:20:15 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 06 Aug 2017 00:20:15 -0700 (PDT)
Subject: [pypy-commit] pypy default: this was done
Message-ID: <5986c32f.9086df0a.894eb.7195@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92099:a186a6bccf3f
Date: 2017-08-06 09:19 +0200
http://bitbucket.org/pypy/pypy/changeset/a186a6bccf3f/

Log:	this was done

diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -84,7 +84,6 @@
 
     # heap knowledge: we store triples of known heap fields in non-virtual
     # structs
-    # XXX could be extended to arrays
     if optimizer.optheap:
         triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into

From pypy.commits at gmail.com  Sun Aug  6 10:38:25 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sun, 06 Aug 2017 07:38:25 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <598729e1.4f821c0a.36e7a.320a@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92100:45907003511b
Date: 2017-08-06 15:37 +0100
http://bitbucket.org/pypy/pypy/changeset/45907003511b/

Log:	hg merge default

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -60,3 +60,16 @@
 
 Small improvement to optimize list accesses with constant indexes better by
 throwing away information about them less eagerly.
+
+
+.. branch: getarrayitem-into-bridges:
+
+More information is retained into a bridge: knowledge about the content of
+arrays (at fixed indices) is stored in guards (and thus available at the
+beginning of bridges). Also, some better feeding of information about known
+fields of constant objects into bridges.
+
+.. branch: cpyext-leakchecking
+
+Add support for leakfinder in cpyext tests (disabled for now, due to too many
+failures).
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1,4 +1,5 @@
 import sys
+import py
 
 from rpython.rlib.cache import Cache
 from rpython.tool.uid import HUGEVAL_BYTES
@@ -1271,8 +1272,22 @@
             self.setitem(w_globals, w_key, self.builtin)
         return statement.exec_code(self, w_globals, w_locals)
 
+    @not_rpython
+    def appdef(self, source):
+        '''Create interp-level function object from app-level source.
+
+        The source should be in the same format as for space.appexec():
+            """(foo, bar): return 'baz'"""
+        '''
+        source = source.lstrip()
+        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
+        source = py.code.Source("def anonymous%s\n" % source)
+        w_glob = self.newdict(module=True)
+        self.exec_(str(source), w_glob, w_glob)
+        return self.getitem(w_glob, self.newtext('anonymous'))
+
     @specialize.arg(2)
-    def appexec(self, posargs_w, source):
+    def appexec(self, posargs_w, source, cache=True):
         """ return value from executing given source at applevel.
             The source must look like
                '''(x, y):
@@ -1280,7 +1295,11 @@
                        return result
                '''
         """
-        w_func = self.fromcache(AppExecCache).getorbuild(source)
+        if cache:
+            w_func = self.fromcache(AppExecCache).getorbuild(source)
+        else:
+            # NB: since appdef() is not-RPython, using cache=False also is.
+            w_func = self.appdef(source)
         args = Arguments(self, list(posargs_w))
         return self.call_args(w_func, args)
 
@@ -1817,15 +1836,7 @@
 class AppExecCache(SpaceCache):
     @not_rpython
     def build(cache, source):
-        space = cache.space
-        # XXX will change once we have our own compiler
-        import py
-        source = source.lstrip()
-        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
-        source = py.code.Source("def anonymous%s\n" % source)
-        w_glob = space.newdict(module=True)
-        space.exec_(str(source), w_glob, w_glob)
-        return space.getitem(w_glob, space.newtext('anonymous'))
+        return cache.space.appdef(source)
 
 
 # Table describing the regular part of the interface of object spaces,
diff --git a/pypy/module/cpyext/buffer.py b/pypy/module/cpyext/buffer.py
--- a/pypy/module/cpyext/buffer.py
+++ b/pypy/module/cpyext/buffer.py
@@ -84,22 +84,27 @@
             if self.needs_decref:
                 if self.releasebufferproc:
                     func_target = rffi.cast(releasebufferproc, self.releasebufferproc)
-                    with lltype.scoped_alloc(Py_buffer) as pybuf:
-                        pybuf.c_buf = self.ptr
-                        pybuf.c_len = self.size
-                        pybuf.c_ndim = cts.cast('int', self.ndim)
-                        pybuf.c_shape = cts.cast('Py_ssize_t*', pybuf.c__shape)
-                        pybuf.c_strides = cts.cast('Py_ssize_t*', pybuf.c__strides)
-                        for i in range(self.ndim):
-                            pybuf.c_shape[i] = self.shape[i]
-                            pybuf.c_strides[i] = self.strides[i]
-                        if self.format:
-                            pybuf.c_format = rffi.str2charp(self.format)
-                        else:
-                            pybuf.c_format = rffi.str2charp("B")
+                    size = rffi.sizeof(cts.gettype('Py_buffer'))
+                    pybuf = lltype.malloc(rffi.VOIDP.TO, size, flavor='raw', zero=True)
+                    pybuf = cts.cast('Py_buffer*', pybuf)
+                    pybuf.c_buf = self.ptr
+                    pybuf.c_len = self.size
+                    pybuf.c_ndim = cts.cast('int', self.ndim)
+                    pybuf.c_shape = cts.cast('Py_ssize_t*', pybuf.c__shape)
+                    pybuf.c_strides = cts.cast('Py_ssize_t*', pybuf.c__strides)
+                    for i in range(self.ndim):
+                        pybuf.c_shape[i] = self.shape[i]
+                        pybuf.c_strides[i] = self.strides[i]
+                    fmt = rffi.str2charp(self.format if self.format else "B")
+                    try:
+                        pybuf.c_format = fmt
                         generic_cpy_call(self.space, func_target, self.pyobj, pybuf)
+                    finally:
+                        lltype.free(fmt, flavor='raw')
+                        lltype.free(pybuf, flavor='raw')
                 decref(self.space, self.pyobj)
             self.pyobj = lltype.nullptr(PyObject.TO)
+            self.w_obj = None
         else:
             #do not call twice
             return
@@ -211,6 +216,7 @@
     view.c_suboffsets = lltype.nullptr(Py_ssize_tP.TO)
     view.c_internal = lltype.nullptr(rffi.VOIDP.TO)
 
+DEFAULT_FMT = rffi.str2charp("B")
 
 @cpython_api([lltype.Ptr(Py_buffer), PyObject, rffi.VOIDP, Py_ssize_t,
               lltype.Signed, lltype.Signed], rffi.INT, error=-1)
@@ -233,7 +239,8 @@
     rffi.setintfield(view, 'c_ndim', 1)
     view.c_format = lltype.nullptr(rffi.CCHARP.TO)
     if (flags & PyBUF_FORMAT) == PyBUF_FORMAT:
-        view.c_format = rffi.str2charp("B")
+        # NB: this needs to be a static string, because nothing frees it
+        view.c_format = DEFAULT_FMT
     view.c_shape = lltype.nullptr(Py_ssize_tP.TO)
     if (flags & PyBUF_ND) == PyBUF_ND:
         view.c_shape = rffi.cast(Py_ssize_tP, view.c__shape)
diff --git a/pypy/module/cpyext/memoryobject.py b/pypy/module/cpyext/memoryobject.py
--- a/pypy/module/cpyext/memoryobject.py
+++ b/pypy/module/cpyext/memoryobject.py
@@ -73,7 +73,7 @@
                     readonly=widen(view.c_readonly))
     # Ensure view.c_buf is released upon object finalization
     fq.register_finalizer(buf)
-    # Allow subclassing W_MemeoryView
+    # Allow subclassing W_MemoryView
     w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type))
     w_obj = space.allocate_instance(W_MemoryView, w_type)
     w_obj.__init__(buf)
@@ -178,11 +178,9 @@
         return (_IsCContiguous(view) or _IsFortranContiguous(view))
     return 0
 
- at cpython_api([PyObject], PyObject, result_is_ll=True)
+ at cpython_api([PyObject], PyObject)
 def PyMemoryView_FromObject(space, w_obj):
-    w_memview = space.call_method(space.builtin, "memoryview", w_obj)
-    py_memview = make_ref(space, w_memview, w_obj)
-    return py_memview
+    return space.call_method(space.builtin, "memoryview", w_obj)
 
 @cts.decl("""PyObject *
     PyMemoryView_FromMemory(char *mem, Py_ssize_t size, int flags)""")
@@ -207,6 +205,7 @@
     # copy view into obj.c_view, without creating a new view.c_obj
     typedescr = get_typedescr(W_MemoryView.typedef)
     py_obj = typedescr.allocate(space, space.w_memoryview)
+
     py_mem = rffi.cast(PyMemoryViewObject, py_obj)
     mview = py_mem.c_view
     mview.c_buf = view.c_buf
diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py
--- a/pypy/module/cpyext/pyobject.py
+++ b/pypy/module/cpyext/pyobject.py
@@ -29,7 +29,7 @@
         from pypy.module.cpyext.typeobject import subtype_dealloc
         return subtype_dealloc.api_func
 
-    def allocate(self, space, w_type, itemcount=0):
+    def allocate(self, space, w_type, itemcount=0, immortal=False):
         # typically called from PyType_GenericAlloc via typedescr.allocate
         # this returns a PyObject with ob_refcnt == 1.
 
@@ -50,7 +50,7 @@
         assert size >= rffi.sizeof(PyObject.TO)
         buf = lltype.malloc(rffi.VOIDP.TO, size,
                             flavor='raw', zero=True,
-                            add_memory_pressure=True)
+                            add_memory_pressure=True, immortal=immortal)
         pyobj = rffi.cast(PyObject, buf)
         if pytype.c_tp_itemsize:
             pyvarobj = rffi.cast(PyVarObject, pyobj)
@@ -102,7 +102,7 @@
         basestruct = tp_basestruct
 
         if tp_alloc:
-            def allocate(self, space, w_type, itemcount=0):
+            def allocate(self, space, w_type, itemcount=0, immortal=False):
                 return tp_alloc(space, w_type, itemcount)
 
         if tp_dealloc:
@@ -151,7 +151,7 @@
 class InvalidPointerException(Exception):
     pass
 
-def create_ref(space, w_obj, w_userdata=None):
+def create_ref(space, w_obj, w_userdata=None, immortal=False):
     """
     Allocates a PyObject, and fills its fields with info from the given
     interpreter object.
@@ -163,7 +163,7 @@
         itemcount = space.len_w(w_obj) # PyBytesObject and subclasses
     else:
         itemcount = 0
-    py_obj = typedescr.allocate(space, w_type, itemcount=itemcount)
+    py_obj = typedescr.allocate(space, w_type, itemcount=itemcount, immortal=immortal)
     track_reference(space, py_obj, w_obj)
     #
     # py_obj.c_ob_refcnt should be exactly REFCNT_FROM_PYPY + 1 here,
@@ -227,7 +227,7 @@
     assert isinstance(w_type, W_TypeObject)
     return get_typedescr(w_type.layout.typedef).realize(space, ref)
 
-def as_pyobj(space, w_obj, w_userdata=None):
+def as_pyobj(space, w_obj, w_userdata=None, immortal=False):
     """
     Returns a 'PyObject *' representing the given intepreter object.
     This doesn't give a new reference, but the returned 'PyObject *'
@@ -239,7 +239,7 @@
         assert not is_pyobj(w_obj)
         py_obj = rawrefcount.from_obj(PyObject, w_obj)
         if not py_obj:
-            py_obj = create_ref(space, w_obj, w_userdata)
+            py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal)
         return py_obj
     else:
         return lltype.nullptr(PyObject.TO)
@@ -270,7 +270,7 @@
         return hop.inputconst(lltype.Bool, hop.s_result.const)
 
 @specialize.ll()
-def make_ref(space, obj, w_userdata=None):
+def make_ref(space, obj, w_userdata=None, immortal=False):
     """Increment the reference counter of the PyObject and return it.
     Can be called with either a PyObject or a W_Root.
     """
@@ -278,7 +278,7 @@
         pyobj = rffi.cast(PyObject, obj)
         at_least = 1
     else:
-        pyobj = as_pyobj(space, obj, w_userdata)
+        pyobj = as_pyobj(space, obj, w_userdata, immortal=immortal)
         at_least = rawrefcount.REFCNT_FROM_PYPY
     if pyobj:
         assert pyobj.c_ob_refcnt >= at_least
diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -48,7 +48,7 @@
     m as the message text. If the conversion otherwise, fails, reraise the
     original exception"""
     if isinstance(w_obj, W_ListObject):
-        # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM    
+        # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM
         w_obj.convert_to_cpy_strategy(space)
         return w_obj
     try:
@@ -313,7 +313,7 @@
                                                          self)
         w_clone.switch_to_object_strategy()
         return w_clone
-        
+
     def copy_into(self, w_list, w_other):
         w_list.switch_to_object_strategy()
         w_list.strategy.copy_into(w_list, w_other)
@@ -378,7 +378,7 @@
 
     def is_empty_strategy(self):
         return False
-        
+
 
 PyObjectList = lltype.Ptr(lltype.Array(PyObject, hints={'nolength': True}))
 
diff --git a/pypy/module/cpyext/test/array.c b/pypy/module/cpyext/test/array.c
--- a/pypy/module/cpyext/test/array.c
+++ b/pypy/module/cpyext/test/array.c
@@ -2322,10 +2322,12 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject *v = getarrayitem(obj2, 0);
         long i = PyLong_AsLong(v);  // XXX: error checking?
-        PyObject * ret = PyList_New(n*i);
+        Py_DECREF(v);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -2338,10 +2340,12 @@
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject *v = getarrayitem(obj1, 0);
         long i = PyLong_AsLong(v);
-        PyObject * ret = PyList_New(n*i);
+        Py_DECREF(v);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -2374,31 +2378,35 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject * lhs, * out;
         PyObject * rhs = getarrayitem(obj2, 0);
-        PyObject * ret = PyList_New(n);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             lhs = PyList_GetItem(obj1, nn);
             out = lhs->ob_type->tp_as_number->nb_multiply(lhs, rhs);
             PyList_SetItem(ret, nn, out);
         }
+        Py_DECREF(rhs);
         return ret;
     }
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject * rhs, * out;
         PyObject * lhs = getarrayitem(obj1, 0);
-        PyObject * ret = PyList_New(n);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             rhs = PyList_GetItem(obj2, nn);
             out = lhs->ob_type->tp_as_number->nb_multiply(lhs, rhs);
             PyList_SetItem(ret, nn, out);
         }
+        Py_DECREF(lhs);
         return ret;
     }
     else if(obj1->ob_type == &Arraytype)
@@ -2904,6 +2912,16 @@
     return PyLong_FromLong(buf_len);
 }
 
+static PyObject *
+same_dealloc(PyObject *self, PyObject *args)
+{
+    PyObject *obj1, *obj2;
+    if (!PyArg_ParseTuple(args, "OO", &obj1, &obj2)) {
+        return NULL;
+    }
+    return PyLong_FromLong(obj1->ob_type->tp_dealloc == obj2->ob_type->tp_dealloc);
+}
+
 /*********************** Install Module **************************/
 
 static PyMethodDef a_methods[] = {
@@ -2914,6 +2932,7 @@
     {"get_releasebuffer_cnt",   (PyCFunction)get_releasebuffer_cnt, METH_NOARGS, NULL},
     {"create_and_release_buffer",   (PyCFunction)create_and_release_buffer, METH_O, NULL},
     {"write_buffer_len", write_buffer_len, METH_O, NULL},
+    {"same_dealloc",   (PyCFunction)same_dealloc, METH_VARARGS, NULL},
     {NULL, NULL, 0, NULL}        /* Sentinel */
 };
 
diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py
--- a/pypy/module/cpyext/test/test_api.py
+++ b/pypy/module/cpyext/test/test_api.py
@@ -6,7 +6,8 @@
 from pypy.module.cpyext.api import (
     slot_function, cpython_api, copy_header_files, INTERPLEVEL_API,
     Py_ssize_t, Py_ssize_tP, PyObject, cts)
-from pypy.module.cpyext.test.test_cpyext import freeze_refcnts, LeakCheckingTest
+from pypy.module.cpyext.test.test_cpyext import (
+    freeze_refcnts, LeakCheckingTest)
 from pypy.interpreter.error import OperationError
 from rpython.rlib import rawrefcount
 import os
@@ -21,17 +22,7 @@
 class BaseApiTest(LeakCheckingTest):
     def setup_class(cls):
         space = cls.space
-        # warm up reference counts:
-        # - the posix module allocates a HCRYPTPROV on Windows
-        # - writing to stdout and stderr allocates a file lock
-        space.getbuiltinmodule("cpyext")
-        space.getbuiltinmodule(os.name)
-        space.call_function(space.getattr(space.sys.get("stderr"),
-                                          space.wrap("write")),
-                            space.wrap(""))
-        space.call_function(space.getattr(space.sys.get("stdout"),
-                                          space.wrap("write")),
-                            space.wrap(""))
+        cls.preload_builtins(space)
 
         class CAPI:
             def __getattr__(self, name):
@@ -39,9 +30,6 @@
         cls.api = CAPI()
         CAPI.__dict__.update(INTERPLEVEL_API)
 
-        print 'DONT_FREE_ANY_MORE'
-        rawrefcount._dont_free_any_more()
-
     def raises(self, space, api, expected_exc, f, *args):
         if not callable(f):
             raise Exception("%s is not callable" % (f,))
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -1,3 +1,4 @@
+import pytest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from pypy.conftest import option
 
@@ -99,6 +100,19 @@
         res = [1, 2, 3] * arr
         assert res == [2, 4, 6]
 
+    @pytest.mark.xfail
+    def test_subclass_dealloc(self):
+        module = self.import_module(name='array')
+        class Sub(module.array):
+            pass
+
+        arr = Sub('i', [2])
+        module.readbuffer_as_string(arr)
+        class A(object):
+            pass
+        assert not module.same_dealloc(arr, module.array('i', [2]))
+        assert module.same_dealloc(arr, A())
+
     def test_string_buf(self):
         module = self.import_module(name='array')
         arr = module.array('u', '123')
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -4,8 +4,11 @@
 
 from pypy.tool.cpyext.extbuild import SystemCompilationInfo, HERE
 from pypy.interpreter.gateway import unwrap_spec, interp2app
+from pypy.interpreter.error import OperationError
 from rpython.rtyper.lltypesystem import lltype
 from pypy.module.cpyext import api
+from pypy.module.cpyext.api import cts
+from pypy.module.cpyext.pyobject import from_ref
 from pypy.module.cpyext.state import State
 from rpython.tool import leakfinder
 from rpython.rlib import rawrefcount
@@ -75,18 +78,94 @@
 def freeze_refcnts(self):
     rawrefcount._dont_free_any_more()
 
+def preload(space, name):
+    from pypy.module.cpyext.pyobject import make_ref
+    if '.' not in name:
+        w_obj = space.builtin.getdictvalue(space, name)
+    else:
+        module, localname = name.rsplit('.', 1)
+        code = "(): import {module}; return {module}.{localname}"
+        code = code.format(**locals())
+        w_obj = space.appexec([], code)
+    make_ref(space, w_obj)
+
+def preload_expr(space, expr):
+    from pypy.module.cpyext.pyobject import make_ref
+    code = "(): return {}".format(expr)
+    w_obj = space.appexec([], code)
+    make_ref(space, w_obj)
+
+def is_interned_string(space, w_obj):
+    try:
+        s = space.str_w(w_obj)
+    except OperationError:
+        return False
+    return space.is_interned_str(s)
+
+def is_allowed_to_leak(space, obj):
+    from pypy.module.cpyext.methodobject import W_PyCFunctionObject
+    try:
+        w_obj = from_ref(space, cts.cast('PyObject*', obj._as_ptr()))
+    except:
+        return False
+    if isinstance(w_obj, W_PyCFunctionObject):
+        return True
+    # It's OK to "leak" some interned strings: if the pyobj is created by
+    # the test, but the w_obj is referred to from elsewhere.
+    return is_interned_string(space, w_obj)
+
+def _get_w_obj(space, c_obj):
+    return from_ref(space, cts.cast('PyObject*', c_obj._as_ptr()))
+
+class CpyextLeak(leakfinder.MallocMismatch):
+    def __str__(self):
+        lines = [leakfinder.MallocMismatch.__str__(self), '']
+        lines.append(
+            "These objects are attached to the following W_Root objects:")
+        for c_obj in self.args[0]:
+            try:
+                lines.append("  %s" % (_get_w_obj(self.args[1], c_obj),))
+            except:
+                pass
+        return '\n'.join(lines)
+
+
 class LeakCheckingTest(object):
     """Base class for all cpyext tests."""
     spaceconfig = dict(usemodules=['cpyext', 'thread', 'struct', 'array',
                                    'itertools', 'time', 'binascii', 'mmap',
                                    ])
 
+    @classmethod
+    def preload_builtins(cls, space):
+        """
+        Eagerly create pyobjs for various builtins so they don't look like
+        leaks.
+        """
+        for name in [
+                'buffer', 'mmap.mmap',
+                'types.FunctionType', 'types.CodeType',
+                'types.TracebackType', 'types.FrameType']:
+            preload(space, name)
+        for expr in ['type(str.join)']:
+            preload_expr(space, expr)
+
     def cleanup(self):
         self.space.getexecutioncontext().cleanup_cpyext_state()
         for _ in range(5):
             rawrefcount._collect()
             self.space.user_del_action._run_finalizers()
-        leakfinder.stop_tracking_allocations(check=False)
+        try:
+            # set check=True to actually enable leakfinder
+            leakfinder.stop_tracking_allocations(check=False)
+        except leakfinder.MallocMismatch as e:
+            result = e.args[0]
+            filtered_result = {}
+            for obj, value in result.iteritems():
+                if not is_allowed_to_leak(self.space, obj):
+                    filtered_result[obj] = value
+            if filtered_result:
+                raise CpyextLeak(filtered_result, self.space)
         assert not self.space.finalizer_queue.next_dead()
 
 
@@ -127,6 +206,7 @@
 def debug_collect(space):
     rawrefcount._collect()
 
+
 class AppTestCpythonExtensionBase(LeakCheckingTest):
 
     def setup_class(cls):
@@ -136,13 +216,8 @@
         cls.w_runappdirect = space.wrap(cls.runappdirect)
         if not cls.runappdirect:
             cls.sys_info = get_cpyext_info(space)
-            space.getbuiltinmodule("cpyext")
-            # 'import os' to warm up reference counts
-            w_import = space.builtin.getdictvalue(space, '__import__')
-            space.call_function(w_import, space.wrap("os"))
-            #state = cls.space.fromcache(RefcountState) ZZZ
-            #state.non_heaptypes_w[:] = []
             cls.w_debug_collect = space.wrap(interp2app(debug_collect))
+            cls.preload_builtins(space)
         else:
             def w_import_module(self, name, init=None, body='', filename=None,
                     include_dirs=None, PY_SSIZE_T_CLEAN=False):
diff --git a/pypy/module/cpyext/test/test_dictobject.py b/pypy/module/cpyext/test/test_dictobject.py
--- a/pypy/module/cpyext/test/test_dictobject.py
+++ b/pypy/module/cpyext/test/test_dictobject.py
@@ -112,74 +112,14 @@
             PyDict_Update(space, w_d, w_d2)
         assert space.unwrap(w_d) == dict(a='b') # unchanged
 
-    def test_iter(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        ppos[0] = 0
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        try:
-            w_copy = space.newdict()
-            while PyDict_Next(space, w_dict, ppos, pkey, pvalue):
-                w_key = from_ref(space, pkey[0])
-                w_value = from_ref(space, pvalue[0])
-                space.setitem(w_copy, w_key, w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.len(w_copy), space.len(w_dict))
-        assert space.eq_w(w_copy, w_dict)
-
-    def test_iterkeys(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        keys_w = []
-        values_w = []
-        try:
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, pkey, None):
-                w_key = from_ref(space, pkey[0])
-                keys_w.append(w_key)
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, None, pvalue):
-                w_value = from_ref(space, pvalue[0])
-                values_w.append(w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.newlist(keys_w),
-                          space.call_function(
-                             space.w_list,
-                             space.call_method(w_dict, "keys")))
-        assert space.eq_w(space.newlist(values_w),
-                          space.call_function(
-                             space.w_list,
-                             space.call_method(w_dict, "values")))
-
     def test_dictproxy(self, space):
-        w_dict = space.sys.get('modules')
+        w_dict = space.appexec([], """(): return {1: 2, 3: 4}""")
         w_proxy = PyDictProxy_New(space, w_dict)
-        assert space.contains_w(w_proxy, space.wrap('sys'))
+        assert space.contains_w(w_proxy, space.newint(1))
         raises(OperationError, space.setitem,
-               w_proxy, space.wrap('sys'), space.w_None)
+               w_proxy, space.newint(1), space.w_None)
         raises(OperationError, space.delitem,
-               w_proxy, space.wrap('sys'))
+               w_proxy, space.newint(1))
         raises(OperationError, space.call_method, w_proxy, 'clear')
         assert PyDictProxy_Check(space, w_proxy)
 
@@ -248,6 +188,59 @@
         d = {"a": 1}
         raises(AttributeError, module.update, d, [("c", 2)])
 
+    def test_iter(self):
+        module = self.import_extension('foo', [
+            ("copy", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value;
+             PyObject* copy = PyDict_New();
+             while (PyDict_Next(args, &pos, &key, &value))
+             {
+                if (PyDict_SetItem(copy, key, value) < 0)
+                {
+                    Py_DecRef(copy);
+                    return NULL;
+                }
+             }
+             return copy;
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        copy = module.copy(d)
+        assert len(copy) == len(d)
+        assert copy == d
+
+    def test_iterkeys(self):
+        module = self.import_extension('foo', [
+            ("keys_and_values", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value, *values;
+             PyObject* keys = PyList_New(0);
+             while (PyDict_Next(args, &pos, &key, NULL))
+             {
+                if (PyList_Append(keys, key) < 0)
+                {
+                    Py_DecRef(keys);
+                    return NULL;
+                }
+             }
+             pos = 0;
+             values = PyList_New(0);
+             while (PyDict_Next(args, &pos, NULL, &value))
+             {
+                if (PyList_Append(values, value) < 0)
+                {
+                    Py_DecRef(keys);
+                    Py_DecRef(values);
+                    return NULL;
+                }
+             }
+             return Py_BuildValue("(NN)", keys, values);
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        assert module.keys_and_values(d) == (list(d.keys()), list(d.values()))
+
     def test_typedict2(self):
         module = self.import_extension('foo', [
             ("get_type_dict", "METH_O",
@@ -260,6 +253,7 @@
             ])
         d = module.get_type_dict(1)
         assert d['real'].__get__(1, 1) == 1
+
     def test_advanced(self):
         module = self.import_extension('foo', [
             ("dict_len", "METH_O",
@@ -271,7 +265,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 3 || !dict || 
+                if (PyTuple_Size(args) < 3 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);
@@ -284,7 +278,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 2 || !dict || 
+                if (PyTuple_Size(args) < 2 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);
diff --git a/pypy/module/cpyext/test/test_eval.py b/pypy/module/cpyext/test/test_eval.py
--- a/pypy/module/cpyext/test/test_eval.py
+++ b/pypy/module/cpyext/test/test_eval.py
@@ -279,6 +279,7 @@
         assert module.call_method("text") == 2
 
     def test_CompileString_and_Exec(self):
+        import sys
         module = self.import_extension('foo', [
             ("compile_string", "METH_NOARGS",
              """
@@ -313,6 +314,9 @@
         print(mod.__dict__)
         assert mod.f(42) == 47
 
+        # Clean-up
+        del sys.modules['cpyext_test_modname']
+
     def test_merge_compiler_flags(self):
         import sys
         module = self.import_extension('foo', [
diff --git a/pypy/module/cpyext/test/test_floatobject.py b/pypy/module/cpyext/test/test_floatobject.py
--- a/pypy/module/cpyext/test/test_floatobject.py
+++ b/pypy/module/cpyext/test/test_floatobject.py
@@ -104,6 +104,7 @@
              PyFloatObject* pfo = (PyFloatObject*)pyobj;
              int res = PyFloat_Check(pyobj) && PyFloat_CheckExact(pyobj) &&
                 PyFloat_Check(pfo) && PyFloat_CheckExact(pfo);
+             Py_DecRef(pyobj);
              return PyLong_FromLong(res);"""),
             ])
         assert module.test() == 1
diff --git a/pypy/module/cpyext/test/test_funcobject.py b/pypy/module/cpyext/test/test_funcobject.py
--- a/pypy/module/cpyext/test/test_funcobject.py
+++ b/pypy/module/cpyext/test/test_funcobject.py
@@ -44,7 +44,7 @@
         w_function = space.appexec([], """():
             def func(x, y, z): return x
             return func
-        """)
+        """, cache=False)
         w_code = PyFunction_GetCode(space, w_function)
         assert w_code.co_name == "func"
 
@@ -61,7 +61,7 @@
             w_code = space.appexec([], """():
                 def func(%s): %s
                 return func.__code__
-            """ % (signature, body))
+            """ % (signature, body), cache=False)
             ref = make_ref(space, w_code)
             co_flags = rffi.cast(PyCodeObject, ref).c_co_flags
             decref(space, ref)
diff --git a/pypy/module/cpyext/test/test_longobject.py b/pypy/module/cpyext/test/test_longobject.py
--- a/pypy/module/cpyext/test/test_longobject.py
+++ b/pypy/module/cpyext/test/test_longobject.py
@@ -329,6 +329,7 @@
                     ret = obj->ob_type->tp_as_number->nb_power(obj, one, one);
                 else
                     ret = PyLong_FromLong(-1);
+                Py_DECREF(one);
                 Py_DECREF(obj);
                 return ret;
              """)])
diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py
--- a/pypy/module/cpyext/test/test_memoryobject.py
+++ b/pypy/module/cpyext/test/test_memoryobject.py
@@ -6,7 +6,7 @@
 from pypy.module.cpyext.test.test_api import BaseApiTest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from pypy.interpreter.buffer import SimpleView
-from pypy.module.cpyext.pyobject import from_ref
+from pypy.module.cpyext.pyobject import make_ref, from_ref
 from pypy.module.cpyext.memoryobject import PyMemoryViewObject
 
 only_pypy ="config.option.runappdirect and '__pypy__' not in sys.builtin_module_names"
@@ -14,9 +14,9 @@
 class TestMemoryViewObject(BaseApiTest):
     def test_frombuffer(self, space, api):
         w_view = SimpleView(StringBuffer("hello")).wrap(space)
+        w_memoryview = api.PyMemoryView_FromObject(w_view)
         c_memoryview = rffi.cast(
-            PyMemoryViewObject, api.PyMemoryView_FromObject(w_view))
-        w_memoryview = from_ref(space, c_memoryview)
+            PyMemoryViewObject, make_ref(space, w_memoryview))
         view = c_memoryview.c_view
         assert view.c_ndim == 1
         f = rffi.charp2str(view.c_format)
@@ -34,6 +34,7 @@
             assert space.eq_w(space.getattr(w_mv, w_f),
                               space.getattr(w_memoryview, w_f))
         api.Py_DecRef(ref)
+        api.Py_DecRef(w_memoryview)
 
 class AppTestPyBuffer_FillInfo(AppTestCpythonExtensionBase):
     def test_fillWithObject(self):
@@ -64,7 +65,6 @@
                  """)])
         result = module.fillinfo()
         assert b"hello, world." == result
-        del result
 
     def test_0d(self):
         module = self.import_extension('foo', [
@@ -195,8 +195,6 @@
         #         in <bound method ConcreteArray.__del__ ...> ignored
 
     def test_releasebuffer(self):
-        if not self.runappdirect:
-            skip("Fails due to ll2ctypes nonsense")
         module = self.import_extension('foo', [
             ("create_test", "METH_NOARGS",
              """
diff --git a/pypy/module/cpyext/test/test_traceback.py b/pypy/module/cpyext/test/test_traceback.py
--- a/pypy/module/cpyext/test/test_traceback.py
+++ b/pypy/module/cpyext/test/test_traceback.py
@@ -3,17 +3,19 @@
 from pypy.module.cpyext.pyobject import PyObject, make_ref, from_ref
 from pypy.module.cpyext.pytraceback import PyTracebackObject
 from pypy.interpreter.pytraceback import PyTraceback
-from pypy.interpreter.pyframe import PyFrame
+from pypy.interpreter.baseobjspace import AppExecCache
 
 class TestPyTracebackObject(BaseApiTest):
     def test_traceback(self, space, api):
-        w_traceback = space.appexec([], """():
+        src = """():
             import sys
             try:
                 1/0
             except:
                 return sys.exc_info()[2]
-        """)
+        """
+        w_traceback = space.appexec([], src)
+
         py_obj = make_ref(space, w_traceback)
         py_traceback = rffi.cast(PyTracebackObject, py_obj)
         assert (from_ref(space, rffi.cast(PyObject, py_traceback.c_ob_type)) is
@@ -38,3 +40,5 @@
         assert lltype.normalizeptr(py_traceback) is None
 
         api.Py_DecRef(py_obj)
+        # hack to allow the code object to be freed
+        del space.fromcache(AppExecCache).content[src]
diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py
--- a/pypy/module/cpyext/test/test_tupleobject.py
+++ b/pypy/module/cpyext/test/test_tupleobject.py
@@ -24,6 +24,7 @@
     def test_tuple_realize_refuses_nulls(self, space, api):
         py_tuple = api.PyTuple_New(1)
         py.test.raises(FatalError, from_ref, space, py_tuple)
+        api.Py_DecRef(py_tuple)
 
     def test_tuple_resize(self, space, api):
         w_42 = space.wrap(42)
@@ -70,6 +71,7 @@
         w_tuple = from_ref(space, py_tuple)
         assert space.eq_w(w_tuple, space.newtuple([space.wrap(42),
                                                    space.wrap(43)]))
+        api.Py_DecRef(py_tuple)
 
     def test_getslice(self, space, api):
         w_tuple = space.newtuple([space.wrap(i) for i in range(10)])
@@ -174,6 +176,7 @@
                 res = PyTuple_SetItem(tuple, 0, one);
                 if (res != 0)
                 {
+                    Py_DECREF(one);
                     Py_DECREF(tuple);
                     return NULL;
                 }
@@ -187,14 +190,13 @@
                 /* Do something that uses the tuple, but does not incref */
                 t2 = PyTuple_GetSlice(tuple, 0, 1);
                 Py_DECREF(t2);
-                Py_INCREF(one);
                 res = PyTuple_SetItem(tuple, 0, one);
-                Py_DECREF(tuple);
                 if (res != 0)
                 {
-                    Py_DECREF(one);
+                    Py_DECREF(tuple);
                     return NULL;
                 }
+                Py_DECREF(tuple);
                 Py_INCREF(Py_None);
                 return Py_None;
              """),
@@ -205,4 +207,3 @@
             raises(SystemError, module.set_after_use, s)
         else:
             module.set_after_use(s)
-
diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py
--- a/pypy/module/cpyext/tupleobject.py
+++ b/pypy/module/cpyext/tupleobject.py
@@ -143,6 +143,7 @@
     old_ref = tupleobj.c_ob_item[index]
     if pyobj_has_w_obj(ref):
         # similar but not quite equal to ref.c_ob_refcnt != 1 on CPython
+        decref(space, py_obj)
         raise oefmt(space.w_SystemError, "PyTuple_SetItem called on tuple after"
                                         " use of tuple")
     tupleobj.c_ob_item[index] = py_obj    # consumes a reference
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -194,6 +194,8 @@
         py_getsetdef = make_GetSet(space, w_obj)
         assert space.isinstance_w(w_userdata, space.w_type)
         w_obj = W_GetSetPropertyEx(py_getsetdef, w_userdata)
+        # now w_obj.getset is py_getsetdef, which was freshly allocated
+        # XXX how is this ever released?
     # XXX assign to d_dname, d_type?
     assert isinstance(w_obj, W_GetSetPropertyEx)
     py_getsetdescr.c_d_getset = w_obj.getset
@@ -823,7 +825,9 @@
                 bases_w = []
             else:
                 bases_w = [from_ref(space, base_pyo)]
-        pto.c_tp_bases = make_ref(space, space.newtuple(bases_w))
+        is_heaptype = bool(pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE)
+        pto.c_tp_bases = make_ref(space, space.newtuple(bases_w),
+                                  immortal=not is_heaptype)
 
 def finish_type_2(space, pto, w_obj):
     """
diff --git a/pypy/objspace/std/specialisedtupleobject.py b/pypy/objspace/std/specialisedtupleobject.py
--- a/pypy/objspace/std/specialisedtupleobject.py
+++ b/pypy/objspace/std/specialisedtupleobject.py
@@ -74,8 +74,7 @@
                 elif typetuple[i] == int:
                     # mimic cpythons behavior of a hash value of -2 for -1
                     y = value
-                    if y == -1:
-                        y = -2
+                    y -= (y == -1)  # No explicit condition, to avoid JIT bridges
                 elif typetuple[i] == float:
                     # get the correct hash for float which is an
                     # integer & other less frequent cases
diff --git a/pypy/objspace/std/test/test_specialisedtupleobject.py b/pypy/objspace/std/test/test_specialisedtupleobject.py
--- a/pypy/objspace/std/test/test_specialisedtupleobject.py
+++ b/pypy/objspace/std/test/test_specialisedtupleobject.py
@@ -37,6 +37,8 @@
                     self.space.eq(self.space.hash(N_w_tuple),
                                   self.space.hash(S_w_tuple)))
 
+        hash_test([-1, -1])
+        hash_test([-1.0, -1.0])
         hash_test([1, 2])
         hash_test([1.5, 2.8])
         hash_test([1.0, 2.0])
diff --git a/rpython/jit/metainterp/logger.py b/rpython/jit/metainterp/logger.py
--- a/rpython/jit/metainterp/logger.py
+++ b/rpython/jit/metainterp/logger.py
@@ -13,10 +13,11 @@
         self.guard_number = guard_number
 
     def log_loop_from_trace(self, trace, memo):
+        debug_start("jit-log-noopt")
         if not have_debug_prints():
+            debug_stop("jit-log-noopt")
             return
         inputargs, ops = self._unpack_trace(trace)
-        debug_start("jit-log-noopt")
         debug_print("# Traced loop or bridge with", len(ops), "ops")
         logops = self._log_operations(inputargs, ops, None, memo)
         debug_stop("jit-log-noopt")
diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -18,6 +18,10 @@
 # (<box1> <descr> <box2>) length times, if getfield(box1, descr) == box2
 #                         both boxes should be in the liveboxes
 #
+# <length>
+# (<box1> <index> <descr> <box2>) length times, if getarrayitem_gc(box1, index, descr) == box2
+#                                 both boxes should be in the liveboxes
+#
 # ----
 
 
@@ -82,18 +86,26 @@
     # structs
     # XXX could be extended to arrays
     if optimizer.optheap:
-        triples = optimizer.optheap.serialize_optheap(available_boxes)
+        triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
         # metainterp_sd.all_descrs
-        triples = [triple for triple in triples if triple[1].descr_index != -1]
-        numb_state.append_int(len(triples))
-        for box1, descr, box2 in triples:
-            index = descr.descr_index
+        triples_struct = [triple for triple in triples_struct if triple[1].descr_index != -1]
+        numb_state.append_int(len(triples_struct))
+        for box1, descr, box2 in triples_struct:
+            descr_index = descr.descr_index
+            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_int(descr_index)
+            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+        numb_state.append_int(len(triples_array))
+        for box1, index, descr, box2 in triples_array:
+            descr_index = descr.descr_index
             numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
             numb_state.append_int(index)
+            numb_state.append_int(descr_index)
             numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
     else:
         numb_state.append_int(0)
+        numb_state.append_int(0)
 
 def deserialize_optimizer_knowledge(optimizer, resumestorage, frontend_boxes, liveboxes):
     reader = resumecode.Reader(resumestorage.rd_numb)
@@ -123,13 +135,24 @@
     if not optimizer.optheap:
         return
     length = reader.next_item()
-    result = []
+    result_struct = []
+    for i in range(length):
+        tagged = reader.next_item()
+        box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
+        tagged = reader.next_item()
+        box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        result_struct.append((box1, descr, box2))
+    length = reader.next_item()
+    result_array = []
     for i in range(length):
         tagged = reader.next_item()
         box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
         index = reader.next_item()
-        descr = metainterp_sd.all_descrs[index]
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
         tagged = reader.next_item()
         box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
-        result.append((box1, descr, box2))
-    optimizer.optheap.deserialize_optheap(result)
+        result_array.append((box1, index, descr, box2))
+    optimizer.optheap.deserialize_optheap(result_struct, result_array)
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -698,7 +698,7 @@
         return self.emit(op)
 
     def serialize_optheap(self, available_boxes):
-        result = []
+        result_getfield = []
         for descr, cf in self.cached_fields.iteritems():
             if cf._lazy_set:
                 continue # XXX safe default for now
@@ -706,27 +706,62 @@
             if not parent_descr.is_object():
                 continue # XXX could be extended to non-instance objects
             for i, box1 in enumerate(cf.cached_structs):
-                if box1 not in available_boxes:
+                if not box1.is_constant() and box1 not in available_boxes:
                     continue
                 structinfo = cf.cached_infos[i]
-                box2 = structinfo.getfield(descr).get_box_replacement()
-                if isinstance(box2, Const) or box2 in available_boxes:
-                    result.append((box1, descr, box2))
-        return result
+                box2 = structinfo.getfield(descr)
+                if box2 is None:
+                    # XXX this should not happen, as it is an invariant
+                    # violation! yet it does if box1 is a constant
+                    continue
+                box2 = box2.get_box_replacement()
+                if box2.is_constant() or box2 in available_boxes:
+                    result_getfield.append((box1, descr, box2))
+        result_array = []
+        for descr, indexdict in self.cached_arrayitems.iteritems():
+            for index, cf in indexdict.iteritems():
+                if cf._lazy_set:
+                    continue # XXX safe default for now
+                for i, box1 in enumerate(cf.cached_structs):
+                    if not box1.is_constant() and box1 not in available_boxes:
+                        continue
+                    arrayinfo = cf.cached_infos[i]
+                    box2 = arrayinfo.getitem(descr, index)
+                    if box2 is None:
+                        # XXX this should not happen, as it is an invariant
+                        # violation! yet it does if box1 is a constant
+                        continue
+                    box2 = box2.get_box_replacement()
+                    if box2.is_constant() or box2 in available_boxes:
+                        result_array.append((box1, index, descr, box2))
+        return result_getfield, result_array
 
-    def deserialize_optheap(self, triples):
-        for box1, descr, box2 in triples:
+    def deserialize_optheap(self, triples_struct, triples_array):
+        for box1, descr, box2 in triples_struct:
             parent_descr = descr.get_parent_descr()
             assert parent_descr.is_object()
-            structinfo = box1.get_forwarded()
-            if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
-                structinfo = info.InstancePtrInfo(parent_descr)
-                structinfo.init_fields(parent_descr, descr.get_index())
-                box1.set_forwarded(structinfo)
-
+            if box1.is_constant():
+                structinfo = info.ConstPtrInfo(box1)
+            else:
+                structinfo = box1.get_forwarded()
+                if not isinstance(structinfo, info.AbstractVirtualPtrInfo):
+                    structinfo = info.InstancePtrInfo(parent_descr)
+                    structinfo.init_fields(parent_descr, descr.get_index())
+                    box1.set_forwarded(structinfo)
             cf = self.field_cache(descr)
             structinfo.setfield(descr, box1, box2, optheap=self, cf=cf)
 
+        for box1, index, descr, box2 in triples_array:
+            if box1.is_constant():
+                arrayinfo = info.ConstPtrInfo(box1)
+            else:
+                arrayinfo = box1.get_forwarded()
+                if not isinstance(arrayinfo, info.AbstractVirtualPtrInfo):
+                    arrayinfo = info.ArrayPtrInfo(descr)
+                    box1.set_forwarded(arrayinfo)
+            cf = self.arrayitem_cache(descr, index)
+            arrayinfo.setitem(descr, index, box1, box2, optheap=self, cf=cf)
+
 
 dispatch_opt = make_dispatcher_method(OptHeap, 'optimize_',
                                       default=OptHeap.emit)
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -61,7 +61,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0]
+    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0, 0]
 
     rbox1 = InputArgRef()
     rbox2 = InputArgRef()
@@ -97,7 +97,7 @@
 
     serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
 
-    assert len(numb_state.create_numbering().code) == 2 + math.ceil(len(refboxes) / 6.0)
+    assert len(numb_state.create_numbering().code) == 3 + math.ceil(len(refboxes) / 6.0)
 
     dct = {box: cls
               for box, known_class in boxes_known_classes
@@ -143,11 +143,7 @@
     def test_bridge_field_read(self):
         myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
         class A(object):
-            def f(self):
-                return 1
-        class B(A):
-            def f(self):
-                return 2
+            pass
         class M(object):
             _immutable_fields_ = ['x']
             def __init__(self, x):
@@ -156,14 +152,12 @@
         m1 = M(1)
         m2 = M(2)
         def f(x, y, n):
+            a = A()
+            a.n = n
             if x:
-                a = A()
                 a.m = m1
-                a.n = n
             else:
-                a = B()
                 a.m = m2
-                a.n = n
             a.x = 0
             res = 0
             while y > 0:
@@ -186,3 +180,105 @@
         self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
         self.check_resops(getfield_gc_r=1) # in main loop
 
+    def test_bridge_field_read_constants(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        class M(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x):
+                self.x = x
+
+        m1 = M(1)
+        m2 = M(2)
+        a = A()
+        a.m = m1
+        a.n = 0
+        def f(x, y, n):
+            if x:
+                a.m = m1
+                a.n = n
+            else:
+                a.m = m2
+                a.n = n
+            a.x = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.n
+                m = jit.promote(a.m)
+                res += m.x
+                a.x += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.m)
+                res += m.x
+                res += n1 + a.n
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getfield_gc_i=4) # 3x a.x, 1x a.n
+        self.check_resops(getfield_gc_r=1) # in main loop
+
+    def test_bridge_array_read(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
+        def f(x, y, n):
+            if x:
+                a = [1, n, 0]
+            else:
+                a = [2, n, 0]
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res, a=a)
+                n1 = a[1]
+                m = jit.promote(a[0])
+                res += m
+                a[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a[0])
+                res += m
+                res += n1 + a[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=4)
+
+    def test_bridge_array_read_constant(self):
+        myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n'])
+        class A(object):
+            pass
+        a = A()
+        a.l = [1, -65, 0]
+        def f(x, y, n):
+            if x:
+                a.l[0] = 1
+            else:
+                a.l[0] = 2
+            a.l[1] = n
+            a.l[2] = 0
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point(y=y, n=n, res=res)
+                n1 = a.l[1]
+                m = jit.promote(a.l[0])
+                res += m
+                a.l[2] += 1
+                if y > n:
+                    res += 1
+                m = jit.promote(a.l[0])
+                res += m
+                res += n1 + a.l[1]
+                y -= 1
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_value=1)
+        self.check_resops(getarrayitem_gc_i=5)
diff --git a/rpython/rlib/rpath.py b/rpython/rlib/rpath.py
--- a/rpython/rlib/rpath.py
+++ b/rpython/rlib/rpath.py
@@ -5,6 +5,7 @@
 import os, stat
 from rpython.rlib import rposix
 from rpython.rlib.signature import signature
+from rpython.rlib.rstring import assert_str0
 from rpython.annotator.model import s_Str0
 
 
@@ -31,9 +32,11 @@
     """Test whether a path is absolute"""
     return s.startswith('/')
 
+ at signature(s_Str0, returns=s_Str0)
 def _posix_rnormpath(path):
     """Normalize path, eliminating double slashes, etc."""
     slash, dot = '/', '.'
+    assert_str0(dot)
     if path == '':
         return dot
     initial_slashes = path.startswith('/')
@@ -56,6 +59,7 @@
     path = slash.join(comps)
     if initial_slashes:
         path = slash*initial_slashes + path
+    assert_str0(path)
     return path or dot
 
 @signature(s_Str0, returns=s_Str0)
@@ -66,6 +70,7 @@
         if not _posix_risabs(path):
             cwd = os.getcwd()
             path = _posix_rjoin(cwd, path)
+        assert path is not None
         return _posix_rnormpath(path)
     except OSError:
         return path
diff --git a/rpython/rlib/rvmprof/cintf.py b/rpython/rlib/rvmprof/cintf.py
--- a/rpython/rlib/rvmprof/cintf.py
+++ b/rpython/rlib/rvmprof/cintf.py
@@ -18,7 +18,7 @@
 SHARED = SRC.join('shared')
 BACKTRACE = SHARED.join('libbacktrace')
 
-compile_extra = ['-DRPYTHON_VMPROF', '-O3']
+compile_extra = ['-DRPYTHON_VMPROF']
 separate_module_files = [
     SHARED.join('symboltable.c'),
     SHARED.join('vmprof_unix.c')
diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py
--- a/rpython/rtyper/lltypesystem/lltype.py
+++ b/rpython/rtyper/lltypesystem/lltype.py
@@ -2208,7 +2208,7 @@
     return _ptr(Ptr(T), o, solid)
 
 @analyzer_for(malloc)
-def ann_malloc(s_T, s_n=None, s_flavor=None, s_zero=None,
+def ann_malloc(s_T, s_n=None, s_flavor=None, s_immortal=None, s_zero=None,
                s_track_allocation=None, s_add_memory_pressure=None,
                s_nonmovable=None):
     assert (s_n is None or s_n.knowntype == int
diff --git a/rpython/rtyper/rbuiltin.py b/rpython/rtyper/rbuiltin.py
--- a/rpython/rtyper/rbuiltin.py
+++ b/rpython/rtyper/rbuiltin.py
@@ -347,19 +347,20 @@
 # annotation of low-level types
 
 @typer_for(lltype.malloc)
-def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None,
-                 i_add_memory_pressure=None, i_nonmovable=None):
+def rtype_malloc(hop, i_flavor=None, i_immortal=None, i_zero=None,
+        i_track_allocation=None, i_add_memory_pressure=None, i_nonmovable=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
     kwds_v = parse_kwds(
         hop,
         (i_flavor, lltype.Void),
+        (i_immortal, None),
         (i_zero, None),
         (i_track_allocation, None),
         (i_add_memory_pressure, None),
         (i_nonmovable, None))
-    (v_flavor, v_zero, v_track_allocation,
+    (v_flavor, v_immortal, v_zero, v_track_allocation,
      v_add_memory_pressure, v_nonmovable) = kwds_v
     flags = {'flavor': 'gc'}
     if v_flavor is not None:
diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -545,7 +545,7 @@
 
     def question(self, ask_gcc):
         try:
-            ask_gcc(self.name + ';')
+            ask_gcc('(void)' + self.name + ';')
             return True
         except CompilationError:
             return False
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -31,11 +31,11 @@
 # endif
 #endif
 
+#define N_LONGBITS  (8 * sizeof(long))
+#define N_LONGSIG   ((NSIG - 1) / N_LONGBITS + 1)
+
 struct pypysig_long_struct pypysig_counter = {0};
-static char volatile pypysig_flags[NSIG] = {0};
-static int volatile pypysig_occurred = 0;
-/* pypysig_occurred is only an optimization: it tells if any
-   pypysig_flags could be set. */
+static long volatile pypysig_flags_bits[N_LONGSIG];
 static int wakeup_fd = -1;
 static int wakeup_with_nul_byte = 1;
 
@@ -73,12 +73,28 @@
 #endif
 }
 
+#ifdef _WIN32
+#define atomic_cas(ptr, oldv, newv)   (InterlockedCompareExchange(ptr, \
+                                            newv, oldv) == (oldv))
+#else
+#define atomic_cas(ptr, oldv, newv)    __sync_bool_compare_and_swap(ptr, \
+                                            oldv, newv)
+#endif
+
 void pypysig_pushback(int signum)
 {
     if (0 <= signum && signum < NSIG)
       {
-        pypysig_flags[signum] = 1;
-        pypysig_occurred = 1;
+        int ok, index = signum / N_LONGBITS;
+        unsigned long bitmask = 1UL << (signum % N_LONGBITS);
+        do
+        {
+            long value = pypysig_flags_bits[index];
+            if (value & bitmask)
+                break;   /* already set */
+            ok = atomic_cas(&pypysig_flags_bits[index], value, value | bitmask);
+        } while (!ok);
+
         pypysig_counter.value = -1;
       }
 }
@@ -161,19 +177,22 @@
 
 int pypysig_poll(void)
 {
-  if (pypysig_occurred)
-    {
-      int i;
-      pypysig_occurred = 0;
-      for (i=0; i<NSIG; i++)
-        if (pypysig_flags[i])
-          {
-            pypysig_flags[i] = 0;
-            pypysig_occurred = 1;   /* maybe another signal is pending */
-            return i;
-          }
+    int index;
+    for (index = 0; index < N_LONGSIG; index++) {
+        long value;
+      retry:
+        value = pypysig_flags_bits[index];
+        if (value != 0L) {
+            int j = 0;
+            while ((value & (1UL << j)) == 0)
+                j++;
+            if (!atomic_cas(&pypysig_flags_bits[index], value,
+                            value & ~(1UL << j)))
+                goto retry;
+            return index * N_LONGBITS + j;
+        }
     }
-  return -1;  /* no pending signal */
+    return -1;  /* no pending signal */
 }
 
 int pypysig_set_wakeup_fd(int fd, int with_nul_byte)
diff --git a/rpython/translator/platform/__init__.py b/rpython/translator/platform/__init__.py
--- a/rpython/translator/platform/__init__.py
+++ b/rpython/translator/platform/__init__.py
@@ -129,7 +129,7 @@
     # some helpers which seem to be cross-platform enough
 
     def _execute_c_compiler(self, cc, args, outname, cwd=None):
-        log.execute(cc + ' ' + ' '.join(args))
+        #log.execute(cc + ' ' + ' '.join(args))
         # 'cc' can also contain some options for the C compiler;
         # e.g. it can be "gcc -m32".  We handle it by splitting on ' '.
         cclist = cc.split()

From pypy.commits at gmail.com  Sun Aug  6 12:44:27 2017
From: pypy.commits at gmail.com (arigo)
Date: Sun, 06 Aug 2017 09:44:27 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2621
Message-ID: <5987476b.0e951c0a.b4e60.12b5@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92101:f58c6966c54d
Date: 2017-08-06 18:43 +0200
http://bitbucket.org/pypy/pypy/changeset/f58c6966c54d/

Log:	Issue #2621

	Hack around until we support duplicate field names like ctypes

diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -562,3 +562,13 @@
 
         x = X()
         assert x.x == 0
+
+    def test_duplicate_names(self):
+        class S(Structure):
+            _fields_ = [('a', c_int),
+                        ('b', c_int),
+                        ('a', c_byte)]
+        s = S(260, -123)
+        assert sizeof(s) == 3 * sizeof(c_int)
+        assert s.a == 4     # 256 + 4
+        assert s.b == -123

From pypy.commits at gmail.com  Sun Aug  6 18:11:49 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 06 Aug 2017 15:11:49 -0700 (PDT)
Subject: [pypy-commit] pypy bridgeopt-improvements: only one subclass
 nowadays
Message-ID: <59879425.b288df0a.9f77f.64a2@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: bridgeopt-improvements
Changeset: r92103:b3eb7293ac01
Date: 2017-08-07 00:09 +0200
http://bitbucket.org/pypy/pypy/changeset/b3eb7293ac01/

Log:	only one subclass nowadays

diff --git a/rpython/jit/metainterp/test/test_call.py b/rpython/jit/metainterp/test/test_call.py
--- a/rpython/jit/metainterp/test/test_call.py
+++ b/rpython/jit/metainterp/test/test_call.py
@@ -2,7 +2,7 @@
 from rpython.jit.metainterp.test.support import LLJitMixin, noConst
 from rpython.rlib import jit
 
-class CallTest(object):
+class TestCall(LLJitMixin):
     def test_indirect_call(self):
         @jit.dont_look_inside
         def f1(x):
@@ -330,6 +330,3 @@
         res = self.meta_interp(f, [21, 0, 0])
         assert res == -2011
 
-
-class TestCall(LLJitMixin, CallTest):
-    pass

From pypy.commits at gmail.com  Sun Aug  6 18:11:47 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 06 Aug 2017 15:11:47 -0700 (PDT)
Subject: [pypy-commit] pypy bridgeopt-improvements: experimental attempt to
 reduce the cost of call_loopinvariant in every bridge
Message-ID: <59879423.9987df0a.fcbea.64f1@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: bridgeopt-improvements
Changeset: r92102:3ab4221d7876
Date: 2017-08-07 00:09 +0200
http://bitbucket.org/pypy/pypy/changeset/3ab4221d7876/

Log:	experimental attempt to reduce the cost of call_loopinvariant in
	every bridge that calls a method.

	approach: pass call_loopinvariant results into failargs (a bit
	everywhere) and then reuse the result in the bridge.

diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -2,6 +2,7 @@
 optimizer of the bridge attached to a guard. """
 
 from rpython.jit.metainterp import resumecode
+from rpython.rlib.objectmodel import we_are_translated
 
 
 # adds the following sections at the end of the resume code:
@@ -22,17 +23,22 @@
 # (<box1> <index> <descr> <box2>) length times, if getarrayitem_gc(box1, index, descr) == box2
 #                                 both boxes should be in the liveboxes
 #
+# <length>
+# (<const> <descr> <box1>) length times, if call_loop_invariant(const, descr) == box1
+#                          the box should be in the liveboxes
 # ----
 
 
 # maybe should be delegated to the optimization classes?
 
-def tag_box(box, liveboxes_from_env, memo):
+def tag_box(box, adder):
     from rpython.jit.metainterp.history import Const
     if isinstance(box, Const):
-        return memo.getconst(box)
+        return adder.memo.getconst(box)
     else:
-        return liveboxes_from_env[box] # has to exist
+        if box in adder.liveboxes_from_env:
+            return adder.liveboxes_from_env[box]
+        return adder.liveboxes[box] # has to exist
 
 def decode_box(resumestorage, tagged, liveboxes, cpu):
     from rpython.jit.metainterp.resume import untag, TAGCONST, TAGINT, TAGBOX
@@ -54,10 +60,13 @@
         raise AssertionError("unreachable")
     return box
 
-def serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, liveboxes_from_env, memo):
+def serialize_optimizer_knowledge(adder, numb_state, liveboxes):
+    optimizer = adder.optimizer
+    liveboxes_from_env = adder.liveboxes_from_env
     available_boxes = {}
     for box in liveboxes:
-        if box is not None and box in liveboxes_from_env:
+        if box is not None and (
+                box in adder.liveboxes_from_env or box in adder.liveboxes):
             available_boxes[box] = None
     metainterp_sd = optimizer.metainterp_sd
 
@@ -84,7 +93,6 @@
 
     # heap knowledge: we store triples of known heap fields in non-virtual
     # structs
-    # XXX could be extended to arrays
     if optimizer.optheap:
         triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
@@ -93,20 +101,32 @@
         numb_state.append_int(len(triples_struct))
         for box1, descr, box2 in triples_struct:
             descr_index = descr.descr_index
-            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_short(tag_box(box1, adder))
             numb_state.append_int(descr_index)
-            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+            numb_state.append_short(tag_box(box2, adder))
         numb_state.append_int(len(triples_array))
         for box1, index, descr, box2 in triples_array:
             descr_index = descr.descr_index
-            numb_state.append_short(tag_box(box1, liveboxes_from_env, memo))
+            numb_state.append_short(tag_box(box1, adder))
             numb_state.append_int(index)
             numb_state.append_int(descr_index)
-            numb_state.append_short(tag_box(box2, liveboxes_from_env, memo))
+            numb_state.append_short(tag_box(box2, adder))
     else:
         numb_state.append_int(0)
         numb_state.append_int(0)
 
+    # loop invariant calls
+    if optimizer.optrewrite:
+        triples = optimizer.optrewrite.serialize_optrewrite(available_boxes)
+        numb_state.append_int(len(triples))
+        for const, descr, box in triples:
+            descr_index = descr.descr_index
+            numb_state.append_short(tag_box(const, adder))
+            numb_state.append_int(descr_index)
+            numb_state.append_short(tag_box(box, adder))
+    else:
+        numb_state.append_int(0)
+
 def deserialize_optimizer_knowledge(optimizer, resumestorage, frontend_boxes, liveboxes):
     reader = resumecode.Reader(resumestorage.rd_numb)
     assert len(frontend_boxes) == len(liveboxes)
@@ -115,6 +135,8 @@
     # skip resume section
     startcount = reader.next_item()
     reader.jump(startcount - 1)
+    extracount = reader.next_item()
+    reader.jump(extracount)
 
     # class knowledge
     bitfield = 0
@@ -132,8 +154,6 @@
             optimizer.make_constant_class(box, cls)
 
     # heap knowledge
-    if not optimizer.optheap:
-        return
     length = reader.next_item()
     result_struct = []
     for i in range(length):
@@ -155,4 +175,59 @@
         tagged = reader.next_item()
         box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
         result_array.append((box1, index, descr, box2))
-    optimizer.optheap.deserialize_optheap(result_struct, result_array)
+    if result_struct or result_array:
+        optimizer.optheap.deserialize_optheap(result_struct, result_array)
+
+    # loop_invariant knowledge
+    length = reader.next_item()
+    results = []
+    for i in range(length):
+        tagged = reader.next_item()
+        box1 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        descr_index = reader.next_item()
+        descr = metainterp_sd.all_descrs[descr_index]
+        tagged = reader.next_item()
+        box2 = decode_box(resumestorage, tagged, liveboxes, metainterp_sd.cpu)
+        results.append((box1, descr, box2))
+    if results:
+        optimizer.optrewrite.deserialize_optrewrite(results)
+
+def consistency_checking_numbering(numb, liveboxes):
+    if we_are_translated():
+        return
+    # very much a "does not crash" kind of affair
+    reader = resumecode.Reader(numb)
+
+    # skip resume section
+    startcount = reader.next_item()
+    reader.jump(startcount - 1)
+    extracount = reader.next_item()
+    reader.jump(extracount)
+
+    mask = 0
+    for i, box in enumerate(liveboxes):
+        if box.type != "r":
+            continue
+        if not mask:
+            bitfield = reader.next_item()
+            mask = 0b100000
+        mask >>= 1
+
+    length = reader.next_item()
+    for i in range(length):
+        tagged = reader.next_item()
+        descr_index = reader.next_item()
+        tagged = reader.next_item()
+    length = reader.next_item()
+    for i in range(length):
+        tagged = reader.next_item()
+        index = reader.next_item()
+        descr_index = reader.next_item()
+        tagged = reader.next_item()
+
+    # loop_invariant knowledge
+    length = reader.next_item()
+    for i in range(length):
+        tagged = reader.next_item()
+        descr_index = reader.next_item()
+        tagged = reader.next_item()
diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py b/rpython/jit/metainterp/optimizeopt/optimizer.py
--- a/rpython/jit/metainterp/optimizeopt/optimizer.py
+++ b/rpython/jit/metainterp/optimizeopt/optimizer.py
@@ -620,7 +620,12 @@
                 del self.replaces_guard[orig_op]
                 return
             else:
-                op = self.emit_guard_operation(op, pendingfields)
+                extra_liveboxes = []
+                # hack, but probably a good one
+                if len(self.optrewrite.loop_invariant_results) == 1:
+                    extra_liveboxes = [
+                        self.optrewrite.loop_invariant_results.values()[0][0].get_box_replacement()]
+                op = self.emit_guard_operation(op, pendingfields, extra_liveboxes)
         elif op.can_raise():
             self.exception_might_have_happened = True
         opnum = op.opnum
@@ -633,7 +638,7 @@
         self._really_emitted_operation = op
         self._newoperations.append(op)
 
-    def emit_guard_operation(self, op, pendingfields):
+    def emit_guard_operation(self, op, pendingfields, extra_liveboxes):
         guard_op = op # self.replace_op_with(op, op.getopnum())
         opnum = guard_op.getopnum()
         # If guard_(no)_exception is merged with another previous guard, then
@@ -653,7 +658,8 @@
             op = self._copy_resume_data_from(guard_op,
                                              self._last_guard_op)
         else:
-            op = self.store_final_boxes_in_guard(guard_op, pendingfields)
+            op = self.store_final_boxes_in_guard(
+                    guard_op, pendingfields, extra_liveboxes)
             self._last_guard_op = op
             # for unrolling
             for farg in op.getfailargs():
@@ -723,7 +729,7 @@
         new_descr.copy_all_attributes_from(old_descr)
         self._newoperations[old_op_pos] = new_op
 
-    def store_final_boxes_in_guard(self, op, pendingfields):
+    def store_final_boxes_in_guard(self, op, pendingfields, extra_liveboxes):
         assert pendingfields is not None
         if op.getdescr() is not None:
             descr = op.getdescr()
@@ -736,7 +742,7 @@
         modifier = resume.ResumeDataVirtualAdder(self, descr, op, self.trace,
                                                  self.resumedata_memo)
         try:
-            newboxes = modifier.finish(pendingfields)
+            newboxes = modifier.finish(pendingfields, extra_liveboxes)
             if (newboxes is not None and
                 len(newboxes) > self.metainterp_sd.options.failargs_limit):
                 raise resume.TagOverflow
diff --git a/rpython/jit/metainterp/optimizeopt/rewrite.py b/rpython/jit/metainterp/optimizeopt/rewrite.py
--- a/rpython/jit/metainterp/optimizeopt/rewrite.py
+++ b/rpython/jit/metainterp/optimizeopt/rewrite.py
@@ -28,7 +28,7 @@
     def _callback(self, op, old_op):
         key = make_hashable_int(op.getarg(0).getint())
         self.opt.loop_invariant_producer[key] = self.opt.optimizer.getlastop()
-        self.opt.loop_invariant_results[key] = old_op
+        self.opt.loop_invariant_results[key] = old_op, op.getarg(0), old_op.getdescr()
 
 
 class OptRewrite(Optimization):
@@ -568,13 +568,15 @@
         arg = op.getarg(0)
         # 'arg' must be a Const, because residual_call in codewriter
         # expects a compile-time constant
+        # XXX the descr is ignored! let's hope there are no different
+        # call_loop_invariant around
         assert isinstance(arg, Const)
         key = make_hashable_int(arg.getint())
 
-        resvalue = self.loop_invariant_results.get(key, None)
+        resvalue, arg0, descr = self.loop_invariant_results.get(key, (None, None, None))
         if resvalue is not None:
             resvalue = self.optimizer.force_op_from_preamble(resvalue)
-            self.loop_invariant_results[key] = resvalue
+            self.loop_invariant_results[key] = resvalue, arg0, descr
             self.make_equal_to(op, resvalue)
             self.last_emitted_operation = REMOVED
             return
@@ -867,6 +869,18 @@
     optimize_SAME_AS_R = optimize_SAME_AS_I
     optimize_SAME_AS_F = optimize_SAME_AS_I
 
+    def serialize_optrewrite(self, available_boxes):
+        triples = []
+        for box, arg0, descr in self.loop_invariant_results.values():
+            triples.append((arg0, descr, box.get_box_replacement()))
+        return triples
+
+    def deserialize_optrewrite(self, triples):
+        for arg, descr, resvalue in triples:
+            assert isinstance(arg, Const)
+            key = make_hashable_int(arg.getint())
+            self.loop_invariant_results[key] = resvalue, arg, descr
+
 dispatch_opt = make_dispatcher_method(OptRewrite, 'optimize_',
                                       default=OptRewrite.emit)
 optimize_guards = _findall(OptRewrite, 'optimize_', 'GUARD')
diff --git a/rpython/jit/metainterp/optimizeopt/shortpreamble.py b/rpython/jit/metainterp/optimizeopt/shortpreamble.py
--- a/rpython/jit/metainterp/optimizeopt/shortpreamble.py
+++ b/rpython/jit/metainterp/optimizeopt/shortpreamble.py
@@ -154,8 +154,11 @@
             return
         op = self.res
         key = make_hashable_int(op.getarg(0).getint())
-        optrewrite.loop_invariant_results[key] = PreambleOp(op, preamble_op,
-                                                            invented_name)
+        optrewrite.loop_invariant_results[key] = (
+            PreambleOp(op, preamble_op, invented_name),
+            op.getarg(0),
+            op.getdescr()
+        )
 
     def add_op_to_short(self, sb):
         op = self.res
diff --git a/rpython/jit/metainterp/resume.py b/rpython/jit/metainterp/resume.py
--- a/rpython/jit/metainterp/resume.py
+++ b/rpython/jit/metainterp/resume.py
@@ -412,7 +412,8 @@
         _, tagbits = untag(tagged)
         return tagbits == TAGVIRTUAL
 
-    def finish(self, pending_setfields=[]):
+    def finish(self, pending_setfields=[], extra_liveboxes=[]):
+        from rpython.jit.metainterp.optimizeopt.bridgeopt import consistency_checking_numbering
         optimizer = self.optimizer
         # compute the numbering
         storage = self.storage
@@ -458,17 +459,31 @@
             info = optimizer.getptrinfo(fieldbox)
             assert info is not None and info.is_virtual()
             info.visitor_walk_recursive(fieldbox, self, optimizer)
+        for box in extra_liveboxes:
+            box = optimizer.get_box_replacement(box)
+            self.register_box(box)
+            info = optimizer.getptrinfo(box)
+            assert info is None or not info.is_virtual()
 
         self._number_virtuals(liveboxes, optimizer, num_virtuals)
         self._add_pending_fields(optimizer, pending_setfields)
 
         numb_state.patch(1, len(liveboxes))
 
-        self._add_optimizer_sections(numb_state, liveboxes, liveboxes_from_env)
-        storage.rd_numb = numb_state.create_numbering()
+        self._add_extra_box_section(extra_liveboxes, numb_state)
+
+        self._add_optimizer_sections(numb_state, liveboxes)
+        rd_numb = numb_state.create_numbering()
+        consistency_checking_numbering(rd_numb, liveboxes)
+        storage.rd_numb = rd_numb
         storage.rd_consts = self.memo.consts
         return liveboxes[:]
 
+    def _add_extra_box_section(self, extra_liveboxes, numb_state):
+        numb_state.append_int(len(extra_liveboxes))
+        for box in extra_liveboxes:
+            numb_state.append_short(self._gettagged(box.get_box_replacement()))
+
     def _number_virtuals(self, liveboxes, optimizer, num_env_virtuals):
         from rpython.jit.metainterp.optimizeopt.info import AbstractVirtualPtrInfo
         
@@ -584,11 +599,10 @@
                 return self.liveboxes_from_env[box]
             return self.liveboxes[box]
 
-    def _add_optimizer_sections(self, numb_state, liveboxes, liveboxes_from_env):
+    def _add_optimizer_sections(self, numb_state, liveboxes):
         # add extra information about things the optimizer learned
         from rpython.jit.metainterp.optimizeopt.bridgeopt import serialize_optimizer_knowledge
-        serialize_optimizer_knowledge(
-            self.optimizer, numb_state, liveboxes, liveboxes_from_env, self.memo)
+        serialize_optimizer_knowledge(self, numb_state, liveboxes)
 
 class AbstractVirtualInfo(object):
     kind = REF
@@ -1067,6 +1081,7 @@
         resumereader.consume_boxes(f.get_current_position_info(),
                                    f.registers_i, f.registers_r, f.registers_f)
         f.handle_rvmprof_enter_on_resume()
+    resumereader.consume_extra_boxes()
     return resumereader.liveboxes, virtualizable_boxes, virtualref_boxes
 
 
@@ -1113,6 +1128,11 @@
         virtualref_boxes = self.consume_virtualref_boxes()
         return virtualizable_boxes, virtualref_boxes
 
+    def consume_extra_boxes(self):
+        extra_boxes_size = self.resumecodereader.next_item()
+        for i in range(extra_boxes_size):
+            self.next_ref() # does nothing but read the box!
+
     def allocate_with_vtable(self, descr=None):
         return self.metainterp.execute_new_with_vtable(descr=descr)
 
diff --git a/rpython/jit/metainterp/resumecode.py b/rpython/jit/metainterp/resumecode.py
--- a/rpython/jit/metainterp/resumecode.py
+++ b/rpython/jit/metainterp/resumecode.py
@@ -18,6 +18,8 @@
 
   until the size of the resume section
 
+  [<length> <numb> <numb> ... <numb>]              more boxes for the optimizer section
+
   # ----- optimization section
   <more code>                                      further sections according to bridgeopt.py
 """
diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -27,6 +27,7 @@
 class FakeOptimizer(object):
     metainterp_sd = None
     optheap = None
+    optrewrite = None
 
     def __init__(self, dct={}, cpu=None):
         self.dct = dct
@@ -46,6 +47,13 @@
     def __init__(self, numb):
         self.rd_numb = numb
 
+class FakeAdder(object):
+    def __init__(self, optimizer, liveboxes_from_env, liveboxes, memo):
+        self.optimizer = optimizer
+        self.liveboxes_from_env = liveboxes_from_env
+        self.liveboxes = liveboxes
+        self.memo = memo
+
 def test_known_classes():
     box1 = InputArgRef()
     box2 = InputArgRef()
@@ -57,11 +65,13 @@
 
     numb_state = NumberingState(4)
     numb_state.append_int(1) # size of resume block
+    numb_state.append_int(0) # size of extra arg block
     liveboxes = [InputArgInt(), box2, box1, box3]
+    adder = FakeAdder(optimizer, {}, {}, None)
 
-    serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
+    serialize_optimizer_knowledge(adder, numb_state, liveboxes)
 
-    assert unpack_numbering(numb_state.create_numbering()) == [1, 0b010000, 0, 0]
+    assert unpack_numbering(numb_state.create_numbering()) == [1, 0, 0b010000, 0, 0, 0]
 
     rbox1 = InputArgRef()
     rbox2 = InputArgRef()
@@ -93,11 +103,14 @@
 
     numb_state = NumberingState(1)
     numb_state.append_int(1) # size of resume block
+    numb_state.append_int(0) # size of extra arg block
     liveboxes = [box for (box, _) in boxes_known_classes]
 
-    serialize_optimizer_knowledge(optimizer, numb_state, liveboxes, {}, None)
+    adder = FakeAdder(optimizer, {}, {}, None)
 
-    assert len(numb_state.create_numbering().code) == 3 + math.ceil(len(refboxes) / 6.0)
+    serialize_optimizer_knowledge(adder, numb_state, liveboxes)
+
+    assert len(numb_state.create_numbering().code) == 5 + math.ceil(len(refboxes) / 6.0)
 
     dct = {box: cls
               for box, known_class in boxes_known_classes
@@ -140,6 +153,40 @@
         self.check_trace_count(3)
         self.check_resops(guard_class=1)
 
+    def Xtest_bridge_guard_class_virtual(self):
+        myjitdriver = jit.JitDriver(greens=[], reds='auto')
+        class A(object):
+            def f(self):
+                return 1
+        class B(A):
+            def f(self):
+                return 2
+        class Box(object):
+            def __init__(self, a):
+                self.a = a
+        def f(x, y, n):
+            if x:
+                a = A()
+            else:
+                a = B()
+            a.x = 0
+            box = Box(a)
+            res = 0
+            while y > 0:
+                myjitdriver.jit_merge_point()
+                res += box.a.f()
+                a.x += 1
+                if y > n:
+                    res += 1
+                res += box.a.f()
+                y -= 1
+                box = Box(box.a)
+            return res
+        res = self.meta_interp(f, [6, 32, 16])
+        assert res == f(6, 32, 16)
+        self.check_trace_count(3)
+        self.check_resops(guard_class=1)
+
     def test_bridge_field_read(self):
         myjitdriver = jit.JitDriver(greens=[], reds=['y', 'res', 'n', 'a'])
         class A(object):
@@ -282,3 +329,33 @@
         self.check_trace_count(3)
         self.check_resops(guard_value=1)
         self.check_resops(getarrayitem_gc_i=5)
+
+    def test_loop_invariant_bridge(self):
+        myjitdriver = jit.JitDriver(greens = [], reds = ['x', 'res'])
+        class A(object):
+            pass
+        a = A()
+        a.current_a = A()
+        a.current_a.x = 12
+        @jit.loop_invariant
+        def f():
+            return a.current_a
+
+        def g(x):
+            res = 0
+            while x > 0:
+                myjitdriver.can_enter_jit(x=x, res=res)
+                myjitdriver.jit_merge_point(x=x, res=res)
+                res += jit.promote(f().x)
+                if x % 5 == 1:
+                    res += 5
+                res += jit.promote(f().x)
+                res += jit.promote(f().x)
+                x -= 1
+            a.current_a = A()
+            a.current_a.x = 2
+            return res
+        res = self.meta_interp(g, [21])
+        assert res == g(21)
+        self.check_resops(call_r=1)
+

From pypy.commits at gmail.com  Sun Aug  6 19:24:25 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sun, 06 Aug 2017 16:24:25 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: adapt is_interned_string() for py3
Message-ID: <5987a529.6596df0a.aa15.9bbc@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92104:e622a373b04c
Date: 2017-08-07 00:23 +0100
http://bitbucket.org/pypy/pypy/changeset/e622a373b04c/

Log:	adapt is_interned_string() for py3

diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -97,10 +97,10 @@
 
 def is_interned_string(space, w_obj):
     try:
-        s = space.str_w(w_obj)
+        u = space.unicode_w(w_obj)
     except OperationError:
         return False
-    return space.is_interned_str(s)
+    return space.interned_strings.get(u) is not None
 
 def is_allowed_to_leak(space, obj):
     from pypy.module.cpyext.methodobject import W_PyCFunctionObject

From pypy.commits at gmail.com  Mon Aug  7 08:24:04 2017
From: pypy.commits at gmail.com (fijal)
Date: Mon, 07 Aug 2017 05:24:04 -0700 (PDT)
Subject: [pypy-commit] pypy default: make sure dict != dict is complained
 about early
Message-ID: <59885be4.05861c0a.ba361.3ee6@mx.google.com>

Author: fijal
Branch: 
Changeset: r92105:fd96c60cfb19
Date: 2017-08-07 14:17 +0200
http://bitbucket.org/pypy/pypy/changeset/fd96c60cfb19/

Log:	make sure dict != dict is complained about early

diff --git a/rpython/annotator/binaryop.py b/rpython/annotator/binaryop.py
--- a/rpython/annotator/binaryop.py
+++ b/rpython/annotator/binaryop.py
@@ -529,6 +529,8 @@
         assert dic1.__class__ == dic2.__class__
         return dic1.__class__(dic1.dictdef.union(dic2.dictdef))
 
+    def ne((dic1, dic2)):
+        raise AnnotatorError("dict != dict not implemented")
 
 def _dict_can_only_throw_keyerror(s_dct, *ignore):
     if s_dct.dictdef.dictkey.custom_eq_hash:

From pypy.commits at gmail.com  Mon Aug  7 08:24:06 2017
From: pypy.commits at gmail.com (fijal)
Date: Mon, 07 Aug 2017 05:24:06 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge
Message-ID: <59885be6.c6c31c0a.6078.746b@mx.google.com>

Author: fijal
Branch: 
Changeset: r92106:bddc7d672228
Date: 2017-08-07 14:23 +0200
http://bitbucket.org/pypy/pypy/changeset/bddc7d672228/

Log:	merge

diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -60,3 +60,16 @@
 
 Small improvement to optimize list accesses with constant indexes better by
 throwing away information about them less eagerly.
+
+
+.. branch: getarrayitem-into-bridges:
+
+More information is retained into a bridge: knowledge about the content of
+arrays (at fixed indices) is stored in guards (and thus available at the
+beginning of bridges). Also, some better feeding of information about known
+fields of constant objects into bridges.
+
+.. branch: cpyext-leakchecking
+
+Add support for leakfinder in cpyext tests (disabled for now, due to too many
+failures).
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1,4 +1,5 @@
 import sys
+import py
 
 from rpython.rlib.cache import Cache
 from rpython.tool.uid import HUGEVAL_BYTES
@@ -1338,8 +1339,22 @@
             self.setitem(w_globals, w_key, self.builtin)
         return statement.exec_code(self, w_globals, w_locals)
 
+    @not_rpython
+    def appdef(self, source):
+        '''Create interp-level function object from app-level source.
+
+        The source should be in the same format as for space.appexec():
+            """(foo, bar): return 'baz'"""
+        '''
+        source = source.lstrip()
+        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
+        source = py.code.Source("def anonymous%s\n" % source)
+        w_glob = self.newdict(module=True)
+        self.exec_(str(source), w_glob, w_glob)
+        return self.getitem(w_glob, self.newtext('anonymous'))
+
     @specialize.arg(2)
-    def appexec(self, posargs_w, source):
+    def appexec(self, posargs_w, source, cache=True):
         """ return value from executing given source at applevel.
             The source must look like
                '''(x, y):
@@ -1347,7 +1362,11 @@
                        return result
                '''
         """
-        w_func = self.fromcache(AppExecCache).getorbuild(source)
+        if cache:
+            w_func = self.fromcache(AppExecCache).getorbuild(source)
+        else:
+            # NB: since appdef() is not-RPython, using cache=False also is.
+            w_func = self.appdef(source)
         args = Arguments(self, list(posargs_w))
         return self.call_args(w_func, args)
 
@@ -1926,15 +1945,7 @@
 class AppExecCache(SpaceCache):
     @not_rpython
     def build(cache, source):
-        space = cache.space
-        # XXX will change once we have our own compiler
-        import py
-        source = source.lstrip()
-        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
-        source = py.code.Source("def anonymous%s\n" % source)
-        w_glob = space.newdict(module=True)
-        space.exec_(str(source), w_glob, w_glob)
-        return space.getitem(w_glob, space.newtext('anonymous'))
+        return cache.space.appdef(source)
 
 
 # Table describing the regular part of the interface of object spaces,
diff --git a/pypy/module/cpyext/buffer.py b/pypy/module/cpyext/buffer.py
--- a/pypy/module/cpyext/buffer.py
+++ b/pypy/module/cpyext/buffer.py
@@ -73,22 +73,27 @@
             if self.needs_decref:
                 if self.releasebufferproc:
                     func_target = rffi.cast(releasebufferproc, self.releasebufferproc)
-                    with lltype.scoped_alloc(Py_buffer) as pybuf:
-                        pybuf.c_buf = self.ptr
-                        pybuf.c_len = self.size
-                        pybuf.c_ndim = cts.cast('int', self.ndim)
-                        pybuf.c_shape = cts.cast('Py_ssize_t*', pybuf.c__shape)
-                        pybuf.c_strides = cts.cast('Py_ssize_t*', pybuf.c__strides)
-                        for i in range(self.ndim):
-                            pybuf.c_shape[i] = self.shape[i]
-                            pybuf.c_strides[i] = self.strides[i]
-                        if self.format:
-                            pybuf.c_format = rffi.str2charp(self.format)
-                        else:
-                            pybuf.c_format = rffi.str2charp("B")
+                    size = rffi.sizeof(cts.gettype('Py_buffer'))
+                    pybuf = lltype.malloc(rffi.VOIDP.TO, size, flavor='raw', zero=True)
+                    pybuf = cts.cast('Py_buffer*', pybuf)
+                    pybuf.c_buf = self.ptr
+                    pybuf.c_len = self.size
+                    pybuf.c_ndim = cts.cast('int', self.ndim)
+                    pybuf.c_shape = cts.cast('Py_ssize_t*', pybuf.c__shape)
+                    pybuf.c_strides = cts.cast('Py_ssize_t*', pybuf.c__strides)
+                    for i in range(self.ndim):
+                        pybuf.c_shape[i] = self.shape[i]
+                        pybuf.c_strides[i] = self.strides[i]
+                    fmt = rffi.str2charp(self.format if self.format else "B")
+                    try:
+                        pybuf.c_format = fmt
                         generic_cpy_call(self.space, func_target, self.pyobj, pybuf)
+                    finally:
+                        lltype.free(fmt, flavor='raw')
+                        lltype.free(pybuf, flavor='raw')
                 decref(self.space, self.pyobj)
             self.pyobj = lltype.nullptr(PyObject.TO)
+            self.w_obj = None
         else:
             #do not call twice
             return
@@ -167,6 +172,8 @@
     sizep[0] = size
     return 0
 
+DEFAULT_FMT = rffi.str2charp("B")
+
 @cpython_api([lltype.Ptr(Py_buffer), PyObject, rffi.VOIDP, Py_ssize_t,
               lltype.Signed, lltype.Signed], rffi.INT, error=-1)
 def PyBuffer_FillInfo(space, view, obj, buf, length, readonly, flags):
@@ -187,7 +194,8 @@
     rffi.setintfield(view, 'c_ndim', 1)
     view.c_format = lltype.nullptr(rffi.CCHARP.TO)
     if (flags & PyBUF_FORMAT) == PyBUF_FORMAT:
-        view.c_format = rffi.str2charp("B")
+        # NB: this needs to be a static string, because nothing frees it
+        view.c_format = DEFAULT_FMT
     view.c_shape = lltype.nullptr(Py_ssize_tP.TO)
     if (flags & PyBUF_ND) == PyBUF_ND:
         view.c_shape = rffi.cast(Py_ssize_tP, view.c__shape)
diff --git a/pypy/module/cpyext/memoryobject.py b/pypy/module/cpyext/memoryobject.py
--- a/pypy/module/cpyext/memoryobject.py
+++ b/pypy/module/cpyext/memoryobject.py
@@ -72,7 +72,7 @@
                     readonly=widen(view.c_readonly))
     # Ensure view.c_buf is released upon object finalization
     fq.register_finalizer(buf)
-    # Allow subclassing W_MemeoryView
+    # Allow subclassing W_MemoryView
     w_type = from_ref(space, rffi.cast(PyObject, obj.c_ob_type))
     w_obj = space.allocate_instance(W_MemoryView, w_type)
     w_obj.__init__(buf)
@@ -177,11 +177,9 @@
         return (_IsCContiguous(view) or _IsFortranContiguous(view))
     return 0
 
- at cpython_api([PyObject], PyObject, result_is_ll=True)
+ at cpython_api([PyObject], PyObject)
 def PyMemoryView_FromObject(space, w_obj):
-    w_memview = space.call_method(space.builtin, "memoryview", w_obj)
-    py_memview = make_ref(space, w_memview, w_obj)
-    return py_memview
+    return space.call_method(space.builtin, "memoryview", w_obj)
 
 @cpython_api([Py_bufferP], PyObject, result_is_ll=True)
 def PyMemoryView_FromBuffer(space, view):
@@ -193,6 +191,7 @@
     # copy view into obj.c_view, without creating a new view.c_obj
     typedescr = get_typedescr(W_MemoryView.typedef)
     py_obj = typedescr.allocate(space, space.w_memoryview)
+
     py_mem = rffi.cast(PyMemoryViewObject, py_obj)
     mview = py_mem.c_view
     mview.c_buf = view.c_buf
diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py
--- a/pypy/module/cpyext/pyobject.py
+++ b/pypy/module/cpyext/pyobject.py
@@ -29,7 +29,7 @@
         from pypy.module.cpyext.typeobject import subtype_dealloc
         return subtype_dealloc.api_func
 
-    def allocate(self, space, w_type, itemcount=0):
+    def allocate(self, space, w_type, itemcount=0, immortal=False):
         # typically called from PyType_GenericAlloc via typedescr.allocate
         # this returns a PyObject with ob_refcnt == 1.
 
@@ -50,7 +50,7 @@
         assert size >= rffi.sizeof(PyObject.TO)
         buf = lltype.malloc(rffi.VOIDP.TO, size,
                             flavor='raw', zero=True,
-                            add_memory_pressure=True)
+                            add_memory_pressure=True, immortal=immortal)
         pyobj = rffi.cast(PyObject, buf)
         if pytype.c_tp_itemsize:
             pyvarobj = rffi.cast(PyVarObject, pyobj)
@@ -102,7 +102,7 @@
         basestruct = tp_basestruct
 
         if tp_alloc:
-            def allocate(self, space, w_type, itemcount=0):
+            def allocate(self, space, w_type, itemcount=0, immortal=False):
                 return tp_alloc(space, w_type, itemcount)
 
         if tp_dealloc:
@@ -151,7 +151,7 @@
 class InvalidPointerException(Exception):
     pass
 
-def create_ref(space, w_obj, w_userdata=None):
+def create_ref(space, w_obj, w_userdata=None, immortal=False):
     """
     Allocates a PyObject, and fills its fields with info from the given
     interpreter object.
@@ -163,7 +163,7 @@
         itemcount = space.len_w(w_obj) # PyBytesObject and subclasses
     else:
         itemcount = 0
-    py_obj = typedescr.allocate(space, w_type, itemcount=itemcount)
+    py_obj = typedescr.allocate(space, w_type, itemcount=itemcount, immortal=immortal)
     track_reference(space, py_obj, w_obj)
     #
     # py_obj.c_ob_refcnt should be exactly REFCNT_FROM_PYPY + 1 here,
@@ -227,7 +227,7 @@
     assert isinstance(w_type, W_TypeObject)
     return get_typedescr(w_type.layout.typedef).realize(space, ref)
 
-def as_pyobj(space, w_obj, w_userdata=None):
+def as_pyobj(space, w_obj, w_userdata=None, immortal=False):
     """
     Returns a 'PyObject *' representing the given intepreter object.
     This doesn't give a new reference, but the returned 'PyObject *'
@@ -239,7 +239,7 @@
         assert not is_pyobj(w_obj)
         py_obj = rawrefcount.from_obj(PyObject, w_obj)
         if not py_obj:
-            py_obj = create_ref(space, w_obj, w_userdata)
+            py_obj = create_ref(space, w_obj, w_userdata, immortal=immortal)
         return py_obj
     else:
         return lltype.nullptr(PyObject.TO)
@@ -270,7 +270,7 @@
         return hop.inputconst(lltype.Bool, hop.s_result.const)
 
 @specialize.ll()
-def make_ref(space, obj, w_userdata=None):
+def make_ref(space, obj, w_userdata=None, immortal=False):
     """Increment the reference counter of the PyObject and return it.
     Can be called with either a PyObject or a W_Root.
     """
@@ -278,7 +278,7 @@
         pyobj = rffi.cast(PyObject, obj)
         at_least = 1
     else:
-        pyobj = as_pyobj(space, obj, w_userdata)
+        pyobj = as_pyobj(space, obj, w_userdata, immortal=immortal)
         at_least = rawrefcount.REFCNT_FROM_PYPY
     if pyobj:
         assert pyobj.c_ob_refcnt >= at_least
diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -48,7 +48,7 @@
     m as the message text. If the conversion otherwise, fails, reraise the
     original exception"""
     if isinstance(w_obj, W_ListObject):
-        # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM    
+        # make sure we can return a borrowed obj from PySequence_Fast_GET_ITEM
         w_obj.convert_to_cpy_strategy(space)
         return w_obj
     try:
@@ -313,7 +313,7 @@
                                                          self)
         w_clone.switch_to_object_strategy()
         return w_clone
-        
+
     def copy_into(self, w_list, w_other):
         w_list.switch_to_object_strategy()
         w_list.strategy.copy_into(w_list, w_other)
@@ -378,7 +378,7 @@
 
     def is_empty_strategy(self):
         return False
-        
+
 
 PyObjectList = lltype.Ptr(lltype.Array(PyObject, hints={'nolength': True}))
 
diff --git a/pypy/module/cpyext/test/array.c b/pypy/module/cpyext/test/array.c
--- a/pypy/module/cpyext/test/array.c
+++ b/pypy/module/cpyext/test/array.c
@@ -1864,10 +1864,12 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject *v = getarrayitem(obj2, 0);
         int i = ((PyIntObject*)v)->ob_ival;
-        PyObject * ret = PyList_New(n*i);
+        Py_DECREF(v);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -1880,10 +1882,12 @@
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int ii, nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject *v = getarrayitem(obj1, 0);
         int i = ((PyIntObject*)v)->ob_ival;
-        PyObject * ret = PyList_New(n*i);
+        Py_DECREF(v);
+        ret = PyList_New(n*i);
         for (ii = 0; ii < i; ii++)
             for (nn = 0; nn < n; nn++)
             {
@@ -1916,34 +1920,44 @@
     if (PyList_Check(obj1) && ((arrayobject*)obj2)->ob_descr->typecode == 'i' && Py_SIZE(obj2) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj1);
         PyObject *v = getarrayitem(obj2, 0);
         int i = ((PyIntObject*)v)->ob_ival;
-        PyObject * ret = PyList_New(n);
+        Py_DECREF(v);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             v = PyList_GetItem(obj1, nn);
             if (PyInt_Check(v))
                 PyList_SetItem(ret, nn, PyLong_FromLong(i * ((PyIntObject*)v)->ob_ival));
             else
+            {
+                Py_INCREF(v);
                 PyList_SetItem(ret, nn, v);
+            }
         }
         return ret;
     }
     else if (PyList_Check(obj2) && ((arrayobject*)obj1)->ob_descr->typecode == 'i' && Py_SIZE(obj1) == 1)
     {
         int nn;
+        PyObject *ret;
         int n = PyList_Size(obj2);
         PyObject *v = getarrayitem(obj1, 0);
         int i = ((PyIntObject*)v)->ob_ival;
-        PyObject * ret = PyList_New(n);
+        Py_DECREF(v);
+        ret = PyList_New(n);
         for (nn = 0; nn < n; nn++)
         {
             v = PyList_GetItem(obj2, nn);
             if (PyInt_Check(v))
                 PyList_SetItem(ret, nn, PyLong_FromLong(i * ((PyIntObject*)v)->ob_ival));
             else
+            {
+                Py_INCREF(v);
                 PyList_SetItem(ret, nn, v);
+            }
         }
         return ret;
     }
@@ -2458,6 +2472,15 @@
     Py_RETURN_NONE;
 }
 
+static PyObject *
+same_dealloc(PyObject *self, PyObject *args)
+{
+    PyObject *obj1, *obj2;
+    if (!PyArg_ParseTuple(args, "OO", &obj1, &obj2)) {
+        return NULL;
+    }
+    return PyLong_FromLong(obj1->ob_type->tp_dealloc == obj2->ob_type->tp_dealloc);
+}
 
 /*********************** Install Module **************************/
 
@@ -2467,6 +2490,7 @@
     {"readbuffer_as_string",   (PyCFunction)readbuffer_as_string, METH_VARARGS, NULL},
     {"get_releasebuffer_cnt",   (PyCFunction)get_releasebuffer_cnt, METH_NOARGS, NULL},
     {"create_and_release_buffer",   (PyCFunction)create_and_release_buffer, METH_O, NULL},
+    {"same_dealloc",   (PyCFunction)same_dealloc, METH_VARARGS, NULL},
     {NULL, NULL, 0, NULL}        /* Sentinel */
 };
 
diff --git a/pypy/module/cpyext/test/test_api.py b/pypy/module/cpyext/test/test_api.py
--- a/pypy/module/cpyext/test/test_api.py
+++ b/pypy/module/cpyext/test/test_api.py
@@ -6,7 +6,8 @@
 from pypy.module.cpyext.api import (
     slot_function, cpython_api, copy_header_files, INTERPLEVEL_API,
     Py_ssize_t, Py_ssize_tP, PyObject, cts)
-from pypy.module.cpyext.test.test_cpyext import freeze_refcnts, LeakCheckingTest
+from pypy.module.cpyext.test.test_cpyext import (
+    freeze_refcnts, LeakCheckingTest)
 from pypy.interpreter.error import OperationError
 from rpython.rlib import rawrefcount
 import os
@@ -21,17 +22,7 @@
 class BaseApiTest(LeakCheckingTest):
     def setup_class(cls):
         space = cls.space
-        # warm up reference counts:
-        # - the posix module allocates a HCRYPTPROV on Windows
-        # - writing to stdout and stderr allocates a file lock
-        space.getbuiltinmodule("cpyext")
-        space.getbuiltinmodule(os.name)
-        space.call_function(space.getattr(space.sys.get("stderr"),
-                                          space.wrap("write")),
-                            space.wrap(""))
-        space.call_function(space.getattr(space.sys.get("stdout"),
-                                          space.wrap("write")),
-                            space.wrap(""))
+        cls.preload_builtins(space)
 
         class CAPI:
             def __getattr__(self, name):
@@ -39,9 +30,6 @@
         cls.api = CAPI()
         CAPI.__dict__.update(INTERPLEVEL_API)
 
-        print 'DONT_FREE_ANY_MORE'
-        rawrefcount._dont_free_any_more()
-
     def raises(self, space, api, expected_exc, f, *args):
         if not callable(f):
             raise Exception("%s is not callable" % (f,))
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -1,3 +1,4 @@
+import pytest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from pypy.conftest import option
 
@@ -111,6 +112,19 @@
         res = [1, 2, 3] * arr
         assert res == [2, 4, 6]
 
+    @pytest.mark.xfail
+    def test_subclass_dealloc(self):
+        module = self.import_module(name='array')
+        class Sub(module.array):
+            pass
+
+        arr = Sub('i', [2])
+        module.readbuffer_as_string(arr)
+        class A(object):
+            pass
+        assert not module.same_dealloc(arr, module.array('i', [2]))
+        assert module.same_dealloc(arr, A())
+
     def test_subclass(self):
         import struct
         module = self.import_module(name='array')
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -4,8 +4,11 @@
 
 from pypy.tool.cpyext.extbuild import SystemCompilationInfo, HERE
 from pypy.interpreter.gateway import unwrap_spec, interp2app
+from pypy.interpreter.error import OperationError
 from rpython.rtyper.lltypesystem import lltype
 from pypy.module.cpyext import api
+from pypy.module.cpyext.api import cts
+from pypy.module.cpyext.pyobject import from_ref
 from pypy.module.cpyext.state import State
 from rpython.tool import leakfinder
 from rpython.rlib import rawrefcount
@@ -73,6 +76,58 @@
 def freeze_refcnts(self):
     rawrefcount._dont_free_any_more()
 
+def preload(space, name):
+    from pypy.module.cpyext.pyobject import make_ref
+    if '.' not in name:
+        w_obj = space.builtin.getdictvalue(space, name)
+    else:
+        module, localname = name.rsplit('.', 1)
+        code = "(): import {module}; return {module}.{localname}"
+        code = code.format(**locals())
+        w_obj = space.appexec([], code)
+    make_ref(space, w_obj)
+
+def preload_expr(space, expr):
+    from pypy.module.cpyext.pyobject import make_ref
+    code = "(): return {}".format(expr)
+    w_obj = space.appexec([], code)
+    make_ref(space, w_obj)
+
+def is_interned_string(space, w_obj):
+    try:
+        s = space.str_w(w_obj)
+    except OperationError:
+        return False
+    return space.is_interned_str(s)
+
+def is_allowed_to_leak(space, obj):
+    from pypy.module.cpyext.methodobject import W_PyCFunctionObject
+    try:
+        w_obj = from_ref(space, cts.cast('PyObject*', obj._as_ptr()))
+    except:
+        return False
+    if isinstance(w_obj, W_PyCFunctionObject):
+        return True
+    # It's OK to "leak" some interned strings: if the pyobj is created by
+    # the test, but the w_obj is referred to from elsewhere.
+    return is_interned_string(space, w_obj)
+
+def _get_w_obj(space, c_obj):
+    return from_ref(space, cts.cast('PyObject*', c_obj._as_ptr()))
+
+class CpyextLeak(leakfinder.MallocMismatch):
+    def __str__(self):
+        lines = [leakfinder.MallocMismatch.__str__(self), '']
+        lines.append(
+            "These objects are attached to the following W_Root objects:")
+        for c_obj in self.args[0]:
+            try:
+                lines.append("  %s" % (_get_w_obj(self.args[1], c_obj),))
+            except:
+                pass
+        return '\n'.join(lines)
+
+
 class LeakCheckingTest(object):
     """Base class for all cpyext tests."""
     spaceconfig = dict(usemodules=['cpyext', 'thread', 'struct', 'array',
@@ -80,11 +135,35 @@
                                    'micronumpy', 'mmap'
                                    ])
 
+    @classmethod
+    def preload_builtins(cls, space):
+        """
+        Eagerly create pyobjs for various builtins so they don't look like
+        leaks.
+        """
+        for name in [
+                'buffer', 'mmap.mmap',
+                'types.FunctionType', 'types.CodeType',
+                'types.TracebackType', 'types.FrameType']:
+            preload(space, name)
+        for expr in ['type(str.join)']:
+            preload_expr(space, expr)
+
     def cleanup(self):
         self.space.getexecutioncontext().cleanup_cpyext_state()
         rawrefcount._collect()
         self.space.user_del_action._run_finalizers()
-        leakfinder.stop_tracking_allocations(check=False)
+        try:
+            # set check=True to actually enable leakfinder
+            leakfinder.stop_tracking_allocations(check=False)
+        except leakfinder.MallocMismatch as e:
+            result = e.args[0]
+            filtered_result = {}
+            for obj, value in result.iteritems():
+                if not is_allowed_to_leak(self.space, obj):
+                    filtered_result[obj] = value
+            if filtered_result:
+                raise CpyextLeak(filtered_result, self.space)
         assert not self.space.finalizer_queue.next_dead()
 
 
@@ -131,6 +210,7 @@
 def debug_collect(space):
     rawrefcount._collect()
 
+
 class AppTestCpythonExtensionBase(LeakCheckingTest):
 
     def setup_class(cls):
@@ -140,13 +220,8 @@
         cls.w_runappdirect = space.wrap(cls.runappdirect)
         if not cls.runappdirect:
             cls.sys_info = get_cpyext_info(space)
-            space.getbuiltinmodule("cpyext")
-            # 'import os' to warm up reference counts
-            w_import = space.builtin.getdictvalue(space, '__import__')
-            space.call_function(w_import, space.wrap("os"))
-            #state = cls.space.fromcache(RefcountState) ZZZ
-            #state.non_heaptypes_w[:] = []
             cls.w_debug_collect = space.wrap(interp2app(debug_collect))
+            cls.preload_builtins(space)
         else:
             def w_import_module(self, name, init=None, body='', filename=None,
                     include_dirs=None, PY_SSIZE_T_CLEAN=False):
diff --git a/pypy/module/cpyext/test/test_dictobject.py b/pypy/module/cpyext/test/test_dictobject.py
--- a/pypy/module/cpyext/test/test_dictobject.py
+++ b/pypy/module/cpyext/test/test_dictobject.py
@@ -111,70 +111,14 @@
             PyDict_Update(space, w_d, w_d2)
         assert space.unwrap(w_d) == dict(a='b') # unchanged
 
-    def test_iter(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        ppos[0] = 0
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        try:
-            w_copy = space.newdict()
-            while PyDict_Next(space, w_dict, ppos, pkey, pvalue):
-                w_key = from_ref(space, pkey[0])
-                w_value = from_ref(space, pvalue[0])
-                space.setitem(w_copy, w_key, w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.len(w_copy), space.len(w_dict))
-        assert space.eq_w(w_copy, w_dict)
-
-    def test_iterkeys(self, space):
-        w_dict = space.sys.getdict(space)
-        py_dict = make_ref(space, w_dict)
-
-        ppos = lltype.malloc(Py_ssize_tP.TO, 1, flavor='raw')
-        pkey = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-        pvalue = lltype.malloc(PyObjectP.TO, 1, flavor='raw')
-
-        keys_w = []
-        values_w = []
-        try:
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, pkey, None):
-                w_key = from_ref(space, pkey[0])
-                keys_w.append(w_key)
-            ppos[0] = 0
-            while PyDict_Next(space, w_dict, ppos, None, pvalue):
-                w_value = from_ref(space, pvalue[0])
-                values_w.append(w_value)
-        finally:
-            lltype.free(ppos, flavor='raw')
-            lltype.free(pkey, flavor='raw')
-            lltype.free(pvalue, flavor='raw')
-
-        decref(space, py_dict) # release borrowed references
-
-        assert space.eq_w(space.newlist(keys_w),
-                          space.call_method(w_dict, "keys"))
-        assert space.eq_w(space.newlist(values_w),
-                          space.call_method(w_dict, "values"))
-
     def test_dictproxy(self, space):
-        w_dict = space.sys.get('modules')
+        w_dict = space.appexec([], """(): return {1: 2, 3: 4}""")
         w_proxy = PyDictProxy_New(space, w_dict)
-        assert space.contains_w(w_proxy, space.wrap('sys'))
+        assert space.contains_w(w_proxy, space.newint(1))
         raises(OperationError, space.setitem,
-               w_proxy, space.wrap('sys'), space.w_None)
+               w_proxy, space.newint(1), space.w_None)
         raises(OperationError, space.delitem,
-               w_proxy, space.wrap('sys'))
+               w_proxy, space.newint(1))
         raises(OperationError, space.call_method, w_proxy, 'clear')
         assert PyDictProxy_Check(space, w_proxy)
 
@@ -243,6 +187,59 @@
         d = {"a": 1}
         raises(AttributeError, module.update, d, [("c", 2)])
 
+    def test_iter(self):
+        module = self.import_extension('foo', [
+            ("copy", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value;
+             PyObject* copy = PyDict_New();
+             while (PyDict_Next(args, &pos, &key, &value))
+             {
+                if (PyDict_SetItem(copy, key, value) < 0)
+                {
+                    Py_DecRef(copy);
+                    return NULL;
+                }
+             }
+             return copy;
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        copy = module.copy(d)
+        assert len(copy) == len(d)
+        assert copy == d
+
+    def test_iterkeys(self):
+        module = self.import_extension('foo', [
+            ("keys_and_values", "METH_O",
+             '''
+             Py_ssize_t pos = 0;
+             PyObject *key, *value, *values;
+             PyObject* keys = PyList_New(0);
+             while (PyDict_Next(args, &pos, &key, NULL))
+             {
+                if (PyList_Append(keys, key) < 0)
+                {
+                    Py_DecRef(keys);
+                    return NULL;
+                }
+             }
+             pos = 0;
+             values = PyList_New(0);
+             while (PyDict_Next(args, &pos, NULL, &value))
+             {
+                if (PyList_Append(values, value) < 0)
+                {
+                    Py_DecRef(keys);
+                    Py_DecRef(values);
+                    return NULL;
+                }
+             }
+             return Py_BuildValue("(NN)", keys, values);
+             ''')])
+        d = {1: 'xyz', 3: 'abcd'}
+        assert module.keys_and_values(d) == (d.keys(), d.values())
+
     def test_typedict2(self):
         module = self.import_extension('foo', [
             ("get_type_dict", "METH_O",
@@ -255,6 +252,7 @@
             ])
         d = module.get_type_dict(1)
         assert d['real'].__get__(1, 1) == 1
+
     def test_advanced(self):
         module = self.import_extension('foo', [
             ("dict_len", "METH_O",
@@ -266,7 +264,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 3 || !dict || 
+                if (PyTuple_Size(args) < 3 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);
@@ -279,7 +277,7 @@
             '''
                 int ret;
                 PyObject * dict = PyTuple_GetItem(args, 0);
-                if (PyTuple_Size(args) < 2 || !dict || 
+                if (PyTuple_Size(args) < 2 || !dict ||
                         !dict->ob_type->tp_as_mapping ||
                         !dict->ob_type->tp_as_mapping->mp_ass_subscript)
                     return PyLong_FromLong(-1);
diff --git a/pypy/module/cpyext/test/test_eval.py b/pypy/module/cpyext/test/test_eval.py
--- a/pypy/module/cpyext/test/test_eval.py
+++ b/pypy/module/cpyext/test/test_eval.py
@@ -279,6 +279,7 @@
         assert module.call_method("text") == 2
 
     def test_CompileString_and_Exec(self):
+        import sys
         module = self.import_extension('foo', [
             ("compile_string", "METH_NOARGS",
              """
@@ -313,6 +314,9 @@
         print mod.__dict__
         assert mod.f(42) == 47
 
+        # Clean-up
+        del sys.modules['cpyext_test_modname']
+
     def test_merge_compiler_flags(self):
         module = self.import_extension('foo', [
             ("get_flags", "METH_NOARGS",
@@ -357,4 +361,4 @@
         except RuntimeError as e:
             assert 'while calling recurse' in str(e)
         else:
-            assert False, "expected RuntimeError"  
+            assert False, "expected RuntimeError"
diff --git a/pypy/module/cpyext/test/test_floatobject.py b/pypy/module/cpyext/test/test_floatobject.py
--- a/pypy/module/cpyext/test/test_floatobject.py
+++ b/pypy/module/cpyext/test/test_floatobject.py
@@ -104,6 +104,7 @@
              PyFloatObject* pfo = (PyFloatObject*)pyobj;
              int res = PyFloat_Check(pyobj) && PyFloat_CheckExact(pyobj) &&
                 PyFloat_Check(pfo) && PyFloat_CheckExact(pfo);
+             Py_DecRef(pyobj);
              return PyLong_FromLong(res);"""),
             ])
         assert module.test() == 1
diff --git a/pypy/module/cpyext/test/test_funcobject.py b/pypy/module/cpyext/test/test_funcobject.py
--- a/pypy/module/cpyext/test/test_funcobject.py
+++ b/pypy/module/cpyext/test/test_funcobject.py
@@ -46,7 +46,7 @@
         w_function = space.appexec([], """():
             def func(x, y, z): return x
             return func
-        """)
+        """, cache=False)
         w_code = PyFunction_GetCode(space, w_function)
         assert w_code.co_name == "func"
 
@@ -63,7 +63,7 @@
             w_code = space.appexec([], """():
                 def func(%s): %s
                 return func.__code__
-            """ % (signature, body))
+            """ % (signature, body), cache=False)
             ref = make_ref(space, w_code)
             co_flags = rffi.cast(PyCodeObject, ref).c_co_flags
             decref(space, ref)
diff --git a/pypy/module/cpyext/test/test_longobject.py b/pypy/module/cpyext/test/test_longobject.py
--- a/pypy/module/cpyext/test/test_longobject.py
+++ b/pypy/module/cpyext/test/test_longobject.py
@@ -313,6 +313,7 @@
                     ret = obj->ob_type->tp_as_number->nb_power(obj, one, one);
                 else
                     ret = PyLong_FromLong(-1);
+                Py_DECREF(one);
                 Py_DECREF(obj);
                 return ret;
              """),
@@ -340,4 +341,3 @@
         assert module.has_pow() == 0
         assert module.has_hex() == '0x2aL'
         assert module.has_oct() == '052L'
-
diff --git a/pypy/module/cpyext/test/test_memoryobject.py b/pypy/module/cpyext/test/test_memoryobject.py
--- a/pypy/module/cpyext/test/test_memoryobject.py
+++ b/pypy/module/cpyext/test/test_memoryobject.py
@@ -4,7 +4,7 @@
 from pypy.module.cpyext.test.test_api import BaseApiTest
 from pypy.module.cpyext.test.test_cpyext import AppTestCpythonExtensionBase
 from rpython.rlib.buffer import StringBuffer
-from pypy.module.cpyext.pyobject import from_ref
+from pypy.module.cpyext.pyobject import make_ref, from_ref
 from pypy.module.cpyext.memoryobject import PyMemoryViewObject
 
 only_pypy ="config.option.runappdirect and '__pypy__' not in sys.builtin_module_names"
@@ -12,9 +12,9 @@
 class TestMemoryViewObject(BaseApiTest):
     def test_frombuffer(self, space, api):
         w_buf = space.newbuffer(StringBuffer("hello"))
+        w_memoryview = api.PyMemoryView_FromObject(w_buf)
         c_memoryview = rffi.cast(
-            PyMemoryViewObject, api.PyMemoryView_FromObject(w_buf))
-        w_memoryview = from_ref(space, c_memoryview)
+            PyMemoryViewObject, make_ref(space, w_memoryview))
         view = c_memoryview.c_view
         assert view.c_ndim == 1
         f = rffi.charp2str(view.c_format)
@@ -32,6 +32,7 @@
             assert space.eq_w(space.getattr(w_mv, w_f),
                               space.getattr(w_memoryview, w_f))
         api.Py_DecRef(ref)
+        api.Py_DecRef(w_memoryview)
 
 class AppTestPyBuffer_FillInfo(AppTestCpythonExtensionBase):
     def test_fillWithObject(self):
@@ -62,7 +63,6 @@
                  """)])
         result = module.fillinfo()
         assert b"hello, world." == result
-        del result
 
 class AppTestBufferProtocol(AppTestCpythonExtensionBase):
     def test_fromobject(self):
@@ -172,8 +172,6 @@
         #         in <bound method ConcreteArray.__del__ ...> ignored
 
     def test_releasebuffer(self):
-        if not self.runappdirect:
-            skip("Fails due to ll2ctypes nonsense")
         module = self.import_extension('foo', [
             ("create_test", "METH_NOARGS",
              """
diff --git a/pypy/module/cpyext/test/test_traceback.py b/pypy/module/cpyext/test/test_traceback.py
--- a/pypy/module/cpyext/test/test_traceback.py
+++ b/pypy/module/cpyext/test/test_traceback.py
@@ -3,17 +3,19 @@
 from pypy.module.cpyext.pyobject import PyObject, make_ref, from_ref
 from pypy.module.cpyext.pytraceback import PyTracebackObject
 from pypy.interpreter.pytraceback import PyTraceback
-from pypy.interpreter.pyframe import PyFrame
+from pypy.interpreter.baseobjspace import AppExecCache
 
 class TestPyTracebackObject(BaseApiTest):
     def test_traceback(self, space, api):
-        w_traceback = space.appexec([], """():
+        src = """():
             import sys
             try:
                 1/0
             except:
                 return sys.exc_info()[2]
-        """)
+        """
+        w_traceback = space.appexec([], src)
+
         py_obj = make_ref(space, w_traceback)
         py_traceback = rffi.cast(PyTracebackObject, py_obj)
         assert (from_ref(space, rffi.cast(PyObject, py_traceback.c_ob_type)) is
@@ -38,3 +40,5 @@
         assert lltype.normalizeptr(py_traceback) is None
 
         api.Py_DecRef(py_obj)
+        # hack to allow the code object to be freed
+        del space.fromcache(AppExecCache).content[src]
diff --git a/pypy/module/cpyext/test/test_tupleobject.py b/pypy/module/cpyext/test/test_tupleobject.py
--- a/pypy/module/cpyext/test/test_tupleobject.py
+++ b/pypy/module/cpyext/test/test_tupleobject.py
@@ -24,6 +24,7 @@
     def test_tuple_realize_refuses_nulls(self, space, api):
         py_tuple = api.PyTuple_New(1)
         py.test.raises(FatalError, from_ref, space, py_tuple)
+        api.Py_DecRef(py_tuple)
 
     def test_tuple_resize(self, space, api):
         w_42 = space.wrap(42)
@@ -70,6 +71,7 @@
         w_tuple = from_ref(space, py_tuple)
         assert space.eq_w(w_tuple, space.newtuple([space.wrap(42),
                                                    space.wrap(43)]))
+        api.Py_DecRef(py_tuple)
 
     def test_getslice(self, space, api):
         w_tuple = space.newtuple([space.wrap(i) for i in range(10)])
@@ -174,6 +176,7 @@
                 res = PyTuple_SetItem(tuple, 0, one);
                 if (res != 0)
                 {
+                    Py_DECREF(one);
                     Py_DECREF(tuple);
                     return NULL;
                 }
@@ -187,14 +190,13 @@
                 /* Do something that uses the tuple, but does not incref */
                 t2 = PyTuple_GetSlice(tuple, 0, 1);
                 Py_DECREF(t2);
-                Py_INCREF(one);
                 res = PyTuple_SetItem(tuple, 0, one);
-                Py_DECREF(tuple);
                 if (res != 0)
                 {
-                    Py_DECREF(one);
+                    Py_DECREF(tuple);
                     return NULL;
                 }
+                Py_DECREF(tuple);
                 Py_INCREF(Py_None);
                 return Py_None;
              """),
@@ -205,4 +207,3 @@
             raises(SystemError, module.set_after_use, s)
         else:
             module.set_after_use(s)
-
diff --git a/pypy/module/cpyext/tupleobject.py b/pypy/module/cpyext/tupleobject.py
--- a/pypy/module/cpyext/tupleobject.py
+++ b/pypy/module/cpyext/tupleobject.py
@@ -143,6 +143,7 @@
     old_ref = tupleobj.c_ob_item[index]
     if pyobj_has_w_obj(ref):
         # similar but not quite equal to ref.c_ob_refcnt != 1 on CPython
+        decref(space, py_obj)
         raise oefmt(space.w_SystemError, "PyTuple_SetItem called on tuple after"
                                         " use of tuple")
     tupleobj.c_ob_item[index] = py_obj    # consumes a reference
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -195,6 +195,8 @@
         py_getsetdef = make_GetSet(space, w_obj)
         assert space.isinstance_w(w_userdata, space.w_type)
         w_obj = W_GetSetPropertyEx(py_getsetdef, w_userdata)
+        # now w_obj.getset is py_getsetdef, which was freshly allocated
+        # XXX how is this ever released?
     # XXX assign to d_dname, d_type?
     assert isinstance(w_obj, W_GetSetPropertyEx)
     py_getsetdescr.c_d_getset = w_obj.getset
@@ -922,7 +924,9 @@
                 bases_w = []
             else:
                 bases_w = [from_ref(space, base_pyo)]
-        pto.c_tp_bases = make_ref(space, space.newtuple(bases_w))
+        is_heaptype = bool(pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE)
+        pto.c_tp_bases = make_ref(space, space.newtuple(bases_w),
+                                  immortal=not is_heaptype)
 
 def finish_type_2(space, pto, w_obj):
     """
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -562,3 +562,13 @@
 
         x = X()
         assert x.x == 0
+
+    def test_duplicate_names(self):
+        class S(Structure):
+            _fields_ = [('a', c_int),
+                        ('b', c_int),
+                        ('a', c_byte)]
+        s = S(260, -123)
+        assert sizeof(s) == 3 * sizeof(c_int)
+        assert s.a == 4     # 256 + 4
+        assert s.b == -123
diff --git a/pypy/objspace/std/specialisedtupleobject.py b/pypy/objspace/std/specialisedtupleobject.py
--- a/pypy/objspace/std/specialisedtupleobject.py
+++ b/pypy/objspace/std/specialisedtupleobject.py
@@ -74,15 +74,15 @@
                 elif typetuple[i] == int:
                     # mimic cpythons behavior of a hash value of -2 for -1
                     y = value
-                    if y == -1:
-                        y = -2
+                    y -= (y == -1)  # No explicit condition, to avoid JIT bridges
                 elif typetuple[i] == float:
                     # get the correct hash for float which is an
                     # integer & other less frequent cases
                     from pypy.objspace.std.floatobject import _hash_float
                     y = _hash_float(space, value)
+                    y -= (y == -1)
                 else:
-                    y = compute_hash(value)
+                    assert 0, "unreachable"
                 x = (x ^ y) * mult
                 z -= 1
                 mult += 82520 + z + z
diff --git a/pypy/objspace/std/test/test_specialisedtupleobject.py b/pypy/objspace/std/test/test_specialisedtupleobject.py
--- a/pypy/objspace/std/test/test_specialisedtupleobject.py
+++ b/pypy/objspace/std/test/test_specialisedtupleobject.py
@@ -37,6 +37,8 @@
                     self.space.eq(self.space.hash(N_w_tuple),
                                   self.space.hash(S_w_tuple)))
 
+        hash_test([-1, -1])
+        hash_test([-1.0, -1.0])
         hash_test([1, 2])
         hash_test([1.5, 2.8])
         hash_test([1.0, 2.0])
diff --git a/rpython/jit/metainterp/logger.py b/rpython/jit/metainterp/logger.py
--- a/rpython/jit/metainterp/logger.py
+++ b/rpython/jit/metainterp/logger.py
@@ -13,10 +13,11 @@
         self.guard_number = guard_number
 
     def log_loop_from_trace(self, trace, memo):
+        debug_start("jit-log-noopt")
         if not have_debug_prints():
+            debug_stop("jit-log-noopt")
             return
         inputargs, ops = self._unpack_trace(trace)
-        debug_start("jit-log-noopt")
         debug_print("# Traced loop or bridge with", len(ops), "ops")
         logops = self._log_operations(inputargs, ops, None, memo)
         debug_stop("jit-log-noopt")
diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -84,7 +84,6 @@
 
     # heap knowledge: we store triples of known heap fields in non-virtual
     # structs
-    # XXX could be extended to arrays
     if optimizer.optheap:
         triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py
--- a/rpython/rtyper/lltypesystem/lltype.py
+++ b/rpython/rtyper/lltypesystem/lltype.py
@@ -2208,7 +2208,7 @@
     return _ptr(Ptr(T), o, solid)
 
 @analyzer_for(malloc)
-def ann_malloc(s_T, s_n=None, s_flavor=None, s_zero=None,
+def ann_malloc(s_T, s_n=None, s_flavor=None, s_immortal=None, s_zero=None,
                s_track_allocation=None, s_add_memory_pressure=None,
                s_nonmovable=None):
     assert (s_n is None or s_n.knowntype == int
diff --git a/rpython/rtyper/rbuiltin.py b/rpython/rtyper/rbuiltin.py
--- a/rpython/rtyper/rbuiltin.py
+++ b/rpython/rtyper/rbuiltin.py
@@ -347,19 +347,20 @@
 # annotation of low-level types
 
 @typer_for(lltype.malloc)
-def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None,
-                 i_add_memory_pressure=None, i_nonmovable=None):
+def rtype_malloc(hop, i_flavor=None, i_immortal=None, i_zero=None,
+        i_track_allocation=None, i_add_memory_pressure=None, i_nonmovable=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
     kwds_v = parse_kwds(
         hop,
         (i_flavor, lltype.Void),
+        (i_immortal, None),
         (i_zero, None),
         (i_track_allocation, None),
         (i_add_memory_pressure, None),
         (i_nonmovable, None))
-    (v_flavor, v_zero, v_track_allocation,
+    (v_flavor, v_immortal, v_zero, v_track_allocation,
      v_add_memory_pressure, v_nonmovable) = kwds_v
     flags = {'flavor': 'gc'}
     if v_flavor is not None:

From pypy.commits at gmail.com  Mon Aug  7 10:05:59 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 07 Aug 2017 07:05:59 -0700 (PDT)
Subject: [pypy-commit] pypy default: add get_config_h_filename for
 cygwincompiler, which justifiably fails later anyway
Message-ID: <598873c7.57b61c0a.ae42a.288e@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92107:a85c3651973e
Date: 2017-08-07 17:05 +0300
http://bitbucket.org/pypy/pypy/changeset/a85c3651973e/

Log:	add get_config_h_filename for cygwincompiler, which justifiably
	fails later anyway

diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)

From pypy.commits at gmail.com  Mon Aug  7 10:28:28 2017
From: pypy.commits at gmail.com (Dodan)
Date: Mon, 07 Aug 2017 07:28:28 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Corrected sendmsg &
 recvmsg. Thanks Ronan Lamy
Message-ID: <5988790c.28afdf0a.f8853.6d91@mx.google.com>

Author: Dodan Mihai <mihai.dodan at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92108:528a9fe09ebd
Date: 2017-08-07 17:27 +0300
http://bitbucket.org/pypy/pypy/changeset/528a9fe09ebd/

Log:	Corrected sendmsg & recvmsg. Thanks Ronan Lamy

diff --git a/pypy/module/_socket/interp_socket.py b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -469,27 +469,27 @@
         :param flags: Receive flag. For more details, please check the Unix manual
         :return: a tuple consisting of the message, the ancillary data, return flag and the address.
         """
-        if (message_size < 0):
+        if message_size < 0:
             raise oefmt(space.w_ValueError, "negative buffer size in recvmsg()")
         if ancbufsize < 0:
             raise oefmt(space.w_ValueError, "invalid ancillary data buffer length")
         while True:
             try:
-                tuple = self.sock.recvmsg(message_size, ancbufsize, flags)
-                message = space.newbytes(tuple[0])
-                list = []
-                for l in tuple[1]:
+                recvtup = self.sock.recvmsg(message_size, ancbufsize, flags)
+                w_message = space.newbytes(recvtup[0])
+                anclist = []
+                for l in recvtup[1]:
                     tup = space.newtuple([space.newint(l[0]), space.newint(l[1]), space.newbytes(l[2])])
-                    list.append(tup)
+                    anclist.append(tup)
 
-                anc = space.newlist(list)
+                w_anc = space.newlist(anclist)
 
-                flag = space.newint(tuple[2])
-                if (tuple[3] is not None):
-                    address = addr_as_object(tuple[3], self.sock.fd, space)
+                w_flag = space.newint(recvtup[2])
+                if (recvtup[3] is not None):
+                    w_address = addr_as_object(recvtup[3], self.sock.fd, space)
                 else:
-                    address = space.w_None
-                rettup = space.newtuple([message, anc, flag, address])
+                    w_address = space.w_None
+                rettup = space.newtuple([w_message, w_anc, w_flag, w_address])
                 break
             except SocketError as e:
                 converted_error(space, e, eintr_retry=True)
@@ -550,7 +550,8 @@
                 converted_error(space, e, eintr_retry=True)
         return space.newint(count)
 
-    def sendmsg_w(self, space, w_data, w_ancillary=None, w_flags=None ,w_address=None):
+    @unwrap_spec(flags=int)
+    def sendmsg_w(self, space, w_data, w_ancillary=None, flags=0 ,w_address=None):
         """
         sendmsg(data[,ancillary[,flags[,address]]]) -> bytes_sent
         Send normal and ancillary data to the socket, gathering the non-ancillary data
@@ -568,101 +569,44 @@
         # Get the flag and address from the object space
         while True:
             try:
-                flags = 0
-                if space.is_none(w_flags) is False:
-                    flags = space.int_w(w_flags)
-
                 address = None
-                if space.is_none(w_address) is False:
+                if not space.is_none(w_address):
                     address = self.addr_from_object(space, w_address)
 
                 # find data's type in the ObjectSpace and get a list of string out of it.
                 data = []
-                if (w_data.typedef.name == 'list'):
-                    for i in w_data.getitems():
-                        if space.isinstance_w(i, space.w_bytes):
-                            data.append(space.bytes_w(i))
-                        else:
-                            if (i.typedef.name == 'array.array'):
-                                data.append(space.bytes_w(i.descr_tobytes(space)))
-                            else:
-                                if (i.typedef.name == 'memoryview'):
-                                    data.append(space.bytes_w(i.descr_tobytes(space)))
-                                else:
-                                    raise oefmt(space.w_TypeError, "a bytes-like object is required")
-                else:
-                    while True:
-                        try:
-                            if (space.is_generator(w_data) is False):
-                                raise oefmt(space.w_TypeError, "sendmsg(): argument 1 must be iterable")
-                            i = space.next(w_data)
-                            if space.isinstance_w(i, space.w_bytes):
-                                data.append(space.bytes_w(i))
-                            else:
-                                if (i.typedef.name == 'array.array'):
-                                    data.append(space.bytes_w(i.descr_tobytes(space)))
-                                else:
-                                    if (i.typedef.name == 'memoryview'):
-                                        data.append(space.bytes_w(i.descr_tobytes(space)))
-                                    else:
-                                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
-                        except OperationError as e:
-                            if not e.match(space, space.w_StopIteration):
-                                raise
-                            break
+                data_iter = space.unpackiterable(w_data)
+                for i in data_iter:
+                    if space.isinstance_w(i, space.w_bytes):
+                        data.append(space.bytes_w(i))
+                    elif (i.typedef.name == 'array.array'):
+                        data.append(space.bytes_w(i.descr_tobytes(space)))
+                    elif (i.typedef.name == 'memoryview'):
+                        data.append(space.bytes_w(i.descr_tobytes(space)))
+                    else:
+                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
 
                 # find the ancillary's type in the ObjectSpace and get a list of tuples out of it.
                 ancillary = []
                 if w_ancillary is not None:
-                    if (space.isinstance_w(w_ancillary, space.w_list)):
-                        for i in w_ancillary.getitems():
-                            if (space.isinstance_w(i, space.w_tuple) is False):
-                                raise oefmt(space.w_TypeError, "[sendmsg() ancillary data items]() argument must be sequence")
-                            if (space.len_w(i) == 3):
-                                level = space.int_w(space.getitem(i, space.newint(0)))
-                                type = space.int_w(space.getitem(i, space.newint(1)))
-                                if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
-                                    cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
-                                else:
-                                    if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
-                                        cont = space.bytes_w(space.getitem(i, space.newint(2)))
-                                    else:
-                                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
-                                tup = (level, type, cont)
-                                ancillary.append(tup)
-                            else:
-                                raise oefmt(space.w_TypeError,
-                                            "[sendmsg() ancillary data items]() argument must be sequence of length 3")
-
-                    else:
-                        while True:
-                            try:
-                                if (space.is_generator(w_ancillary) is False):
-                                    raise oefmt(space.w_TypeError,
-                                                "[sendmsg() ancillary data items]() argument must be sequence")
-                                i = space.next(w_ancillary)
-                                if (space.isinstance_w(i, space.w_tuple) is False):
-                                    raise oefmt(space.w_TypeError,
-                                                "[sendmsg() ancillary data items]() argument must be sequence of length 3")
-                                if (space.len_w(i) != 3):
-                                    raise oefmt(space.w_TypeError,
-                                                "[sendmsg() ancillary data items]() argument must be sequence of length 3")
-                            except OperationError as e:
-                                if not e.match(space, space.w_StopIteration):
-                                    raise
-                                break
+                    anc_iter = space.unpackiterable(w_ancillary)
+                    for i in anc_iter:
+                        if (not space.isinstance_w(i, space.w_tuple)):
+                            raise oefmt(space.w_TypeError, "[sendmsg() ancillary data items]() argument must be sequence")
+                        if (space.len_w(i) == 3):
                             level = space.int_w(space.getitem(i, space.newint(0)))
                             type = space.int_w(space.getitem(i, space.newint(1)))
                             if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
                                 cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
+                            elif (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
+                                cont = space.bytes_w(space.getitem(i, space.newint(2)))
                             else:
-                                if (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
-                                    cont = space.bytes_w(space.getitem(i, space.newint(2)))
-                                else:
-                                    raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                                raise oefmt(space.w_TypeError, "a bytes-like object is required")
                             tup = (level, type, cont)
                             ancillary.append(tup)
-
+                        else:
+                            raise oefmt(space.w_TypeError,
+                                        "[sendmsg() ancillary data items]() argument must be sequence of length 3")
 
                 count = self.sock.sendmsg(data, ancillary, flags, address)
                 if count < 0:

From pypy.commits at gmail.com  Mon Aug  7 10:28:47 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 07 Aug 2017 07:28:47 -0700 (PDT)
Subject: [pypy-commit] pypy default: typo
Message-ID: <5988791f.8c941c0a.2a8bd.ec92@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92109:8a997c17d016
Date: 2017-08-07 17:26 +0300
http://bitbucket.org/pypy/pypy/changeset/8a997c17d016/

Log:	typo

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -62,7 +62,7 @@
 throwing away information about them less eagerly.
 
 
-.. branch: getarrayitem-into-bridges:
+.. branch: getarrayitem-into-bridges
 
 More information is retained into a bridge: knowledge about the content of
 arrays (at fixed indices) is stored in guards (and thus available at the

From pypy.commits at gmail.com  Tue Aug  8 07:35:42 2017
From: pypy.commits at gmail.com (Dodan)
Date: Tue, 08 Aug 2017 04:35:42 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Simplified the sendmsg_w
 function in interp_socket
Message-ID: <5989a20e.4bd51c0a.58f7e.7ef1@mx.google.com>

Author: Dodan Mihai <mihai.dodan at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92110:59c1cbda22f6
Date: 2017-08-08 14:34 +0300
http://bitbucket.org/pypy/pypy/changeset/59c1cbda22f6/

Log:	Simplified the sendmsg_w function in interp_socket

diff --git a/pypy/module/_socket/interp_socket.py b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -577,31 +577,20 @@
                 data = []
                 data_iter = space.unpackiterable(w_data)
                 for i in data_iter:
-                    if space.isinstance_w(i, space.w_bytes):
-                        data.append(space.bytes_w(i))
-                    elif (i.typedef.name == 'array.array'):
-                        data.append(space.bytes_w(i.descr_tobytes(space)))
-                    elif (i.typedef.name == 'memoryview'):
-                        data.append(space.bytes_w(i.descr_tobytes(space)))
-                    else:
-                        raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                    data.append(space.readbuf_w(i).as_str())
 
                 # find the ancillary's type in the ObjectSpace and get a list of tuples out of it.
                 ancillary = []
                 if w_ancillary is not None:
                     anc_iter = space.unpackiterable(w_ancillary)
-                    for i in anc_iter:
-                        if (not space.isinstance_w(i, space.w_tuple)):
+                    for w_i in anc_iter:
+                        if not space.isinstance_w(w_i, space.w_tuple):
                             raise oefmt(space.w_TypeError, "[sendmsg() ancillary data items]() argument must be sequence")
-                        if (space.len_w(i) == 3):
-                            level = space.int_w(space.getitem(i, space.newint(0)))
-                            type = space.int_w(space.getitem(i, space.newint(1)))
-                            if (space.getitem(i, space.newint(2)).typedef.name == 'array.array'):
-                                cont = space.bytes_w(space.getitem(i, space.newint(2)).descr_tobytes(space))
-                            elif (space.isinstance_w(space.getitem(i, space.newint(2)), space.w_bytes)):
-                                cont = space.bytes_w(space.getitem(i, space.newint(2)))
-                            else:
-                                raise oefmt(space.w_TypeError, "a bytes-like object is required")
+                        if space.len_w(w_i) == 3:
+                            intemtup = space.unpackiterable(w_i)
+                            level = space.int_w(intemtup[0])
+                            type = space.int_w(intemtup[1])
+                            cont = space.readbuf_w(intemtup[2]).as_str()
                             tup = (level, type, cont)
                             ancillary.append(tup)
                         else:

From pypy.commits at gmail.com  Tue Aug  8 11:33:56 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 08 Aug 2017 08:33:56 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <5989d9e4.6ea0df0a.d3eb2.c464@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92111:474e84c737d0
Date: 2017-08-08 17:32 +0200
http://bitbucket.org/pypy/pypy/changeset/474e84c737d0/

Log:	hg merge py3.5

diff too long, truncating to 2000 out of 23586 lines

diff --git a/.hgignore b/.hgignore
--- a/.hgignore
+++ b/.hgignore
@@ -27,16 +27,17 @@
 ^pypy/module/cpyext/test/.+\.manifest$
 ^pypy/module/test_lib_pypy/ctypes_tests/.+\.o$
 ^pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test\.o$
-^pypy/module/cppyy/src/.+\.o$
-^pypy/module/cppyy/bench/.+\.so$
-^pypy/module/cppyy/bench/.+\.root$
-^pypy/module/cppyy/bench/.+\.d$
-^pypy/module/cppyy/src/.+\.errors$
-^pypy/module/cppyy/test/.+_rflx\.cpp$
-^pypy/module/cppyy/test/.+\.so$
-^pypy/module/cppyy/test/.+\.rootmap$
-^pypy/module/cppyy/test/.+\.exe$
-^pypy/module/cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/src/.+\.o$
+^pypy/module/_cppyy/bench/.+\.so$
+^pypy/module/_cppyy/bench/.+\.root$
+^pypy/module/_cppyy/bench/.+\.d$
+^pypy/module/_cppyy/src/.+\.errors$
+^pypy/module/_cppyy/test/.+_rflx\.cpp$
+^pypy/module/_cppyy/test/.+\.so$
+^pypy/module/_cppyy/test/.+\.rootmap$
+^pypy/module/_cppyy/test/.+\.exe$
+^pypy/module/_cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/.+/*\.pcm$
 ^pypy/module/test_lib_pypy/cffi_tests/__pycache__.+$
 ^pypy/doc/.+\.html$
 ^pypy/doc/config/.+\.rst$
@@ -93,6 +94,3 @@
 ^release/
 ^rpython/_cache$
 
-pypy/module/cppyy/.+/*\.pcm
-
-
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib-python/3/stat.py b/lib-python/3/stat.py
--- a/lib-python/3/stat.py
+++ b/lib-python/3/stat.py
@@ -139,13 +139,21 @@
 def filemode(mode):
     """Convert a file's mode to a string of the form '-rwxrwxrwx'."""
     perm = []
+
+    # The first group gets a question mark if none of the bits match the mode.
+    empty = "?"
+
     for table in _filemode_table:
         for bit, char in table:
             if mode & bit == bit:
                 perm.append(char)
                 break
         else:
-            perm.append("-")
+            perm.append(empty)
+
+        # All the rest of the positions get a - if the bits don't match.
+        empty = "-"
+
     return "".join(perm)
 
 
diff --git a/lib-python/3/test/test_stat.py b/lib-python/3/test/test_stat.py
--- a/lib-python/3/test/test_stat.py
+++ b/lib-python/3/test/test_stat.py
@@ -138,6 +138,10 @@
             self.assertS_IS("REG", st_mode)
             self.assertEqual(modestr, '-r--r--r--')
             self.assertEqual(self.statmod.S_IMODE(st_mode), 0o444)
+
+            # If there are only permission bits, no type bytes, a question
+            # mark is rendered in the type field.
+            self.assertEqual(self.statmod.filemode(0o420), '?r---w----')
         else:
             os.chmod(TESTFN, 0o700)
             st_mode, modestr = self.get_mode()
diff --git a/lib_pypy/_cffi_ssl/_stdssl/certificate.py b/lib_pypy/_cffi_ssl/_stdssl/certificate.py
--- a/lib_pypy/_cffi_ssl/_stdssl/certificate.py
+++ b/lib_pypy/_cffi_ssl/_stdssl/certificate.py
@@ -173,14 +173,13 @@
 
     return tuple(dn)
 
-STATIC_BIO_BUF = ffi.new("char[]", 2048)
-
 def _bio_get_str(biobuf):
-    length = lib.BIO_gets(biobuf, STATIC_BIO_BUF, len(STATIC_BIO_BUF)-1)
+    bio_buf = ffi.new("char[]", 2048)
+    length = lib.BIO_gets(biobuf, bio_buf, len(bio_buf)-1)
     if length < 0:
         if biobuf: lib.BIO_free(biobuf)
         raise ssl_error(None)
-    return _str_with_len(STATIC_BIO_BUF, length)
+    return _str_with_len(bio_buf, length)
 
 def _decode_certificate(certificate):
     retval = {}
diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py
--- a/lib_pypy/_curses.py
+++ b/lib_pypy/_curses.py
@@ -411,7 +411,7 @@
             val = lib.mvwget_wch(self._win, *args, wch)
         else:
             raise error("get_wch requires 0 or 2 arguments")
-        _check_ERR(val, "get_wch"):
+        _check_ERR(val, "get_wch")
         return wch[0]
 
     def getkey(self, *args):
diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py
--- a/lib_pypy/_tkinter/tklib_build.py
+++ b/lib_pypy/_tkinter/tklib_build.py
@@ -22,12 +22,27 @@
     linklibs = ['tcl', 'tk']
     libdirs = []
 else:
-    for _ver in ['', '8.6', '8.5', '']:
+    # On some Linux distributions, the tcl and tk libraries are
+    # stored in /usr/include, so we must check this case also
+    libdirs = []
+    found = False
+    for _ver in ['', '8.6', '8.5']:
         incdirs = ['/usr/include/tcl' + _ver]
         linklibs = ['tcl' + _ver, 'tk' + _ver]
-        libdirs = []
         if os.path.isdir(incdirs[0]):
+            found = True
             break
+    if not found:
+        for _ver in ['8.6', '8.5', '']:
+            incdirs = []
+            linklibs = ['tcl' + _ver, 'tk' + _ver]
+            if os.path.isfile(''.join(['/usr/lib/lib', linklibs[1], '.so'])):
+                found = True
+                break
+    if not found:
+        sys.stderr.write("*** TCL libraries not found!  Falling back...\n")
+        incdirs = []
+        linklibs = ['tcl', 'tk']
 
 config_ffi = FFI()
 config_ffi.cdef("""
diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h
--- a/lib_pypy/cffi/_cffi_include.h
+++ b/lib_pypy/cffi/_cffi_include.h
@@ -95,6 +95,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h
--- a/lib_pypy/cffi/_embedding.h
+++ b/lib_pypy/cffi/_embedding.h
@@ -1,7 +1,12 @@
 
 /***** Support code for embedding *****/
 
-#if defined(_MSC_VER)
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+#if defined(_WIN32)
 #  define CFFI_DLLEXPORT  __declspec(dllexport)
 #elif defined(__GNUC__)
 #  define CFFI_DLLEXPORT  __attribute__((visibility("default")))
@@ -525,3 +530,7 @@
 #undef cffi_compare_and_swap
 #undef cffi_write_barrier
 #undef cffi_read_barrier
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py
--- a/lib_pypy/cffi/recompiler.py
+++ b/lib_pypy/cffi/recompiler.py
@@ -412,6 +412,9 @@
             prnt('    }')
         prnt('    p[0] = (const void *)0x%x;' % self._version)
         prnt('    p[1] = &_cffi_type_context;')
+        prnt('#if PY_MAJOR_VERSION >= 3')
+        prnt('    return NULL;')
+        prnt('#endif')
         prnt('}')
         # on Windows, distutils insists on putting init_cffi_xyz in
         # 'export_symbols', so instead of fighting it, just give up and
@@ -578,7 +581,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.BasePrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif isinstance(tp, model.UnknownFloatType):
                 return '_cffi_from_c_double(%s)' % (var,)
diff --git a/lib_pypy/cffi/vengine_cpy.py b/lib_pypy/cffi/vengine_cpy.py
--- a/lib_pypy/cffi/vengine_cpy.py
+++ b/lib_pypy/cffi/vengine_cpy.py
@@ -296,7 +296,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.PrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif tp.name != 'long double':
                 return '_cffi_from_c_%s(%s)' % (tp.name.replace(' ', '_'), var)
@@ -872,6 +872,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -39,7 +39,7 @@
     "thread", "itertools", "pyexpat", "cpyext", "array",
     "binascii", "_multiprocessing", '_warnings', "_collections",
     "_multibytecodec", "_continuation", "_cffi_backend",
-    "_csv", "_pypyjson", "_posixsubprocess", # "cppyy", "micronumpy"
+    "_csv", "_pypyjson", "_posixsubprocess", # "_cppyy", "micronumpy"
     "_jitlog",
 ])
 
@@ -71,8 +71,8 @@
         if name in translation_modules:
             translation_modules.remove(name)
 
-    if "cppyy" in working_modules:
-        working_modules.remove("cppyy")  # not tested on win32
+    if "_cppyy" in working_modules:
+        working_modules.remove("_cppyy")  # not tested on win32
 
     # The _locale module is needed by site.py on Windows
     default_modules.add("_locale")
@@ -81,8 +81,8 @@
     working_modules.remove('fcntl')  # LOCK_NB not defined
     working_modules.remove("_minimal_curses")
     working_modules.remove("termios")
-    if "cppyy" in working_modules:
-        working_modules.remove("cppyy")  # depends on ctypes
+    if "_cppyy" in working_modules:
+        working_modules.remove("_cppyy")  # depends on ctypes
 
 #if sys.platform.startswith("linux"):
 #    _mach = os.popen('uname -m', 'r').read().strip()
@@ -94,7 +94,7 @@
     '_multiprocessing': [('objspace.usemodules.time', True),
                          ('objspace.usemodules.thread', True)],
     'cpyext': [('objspace.usemodules.array', True)],
-    'cppyy': [('objspace.usemodules.cpyext', True)],
+    '_cppyy': [('objspace.usemodules.cpyext', True)],
     'faulthandler': [('objspace.usemodules._vmprof', True)],
     }
 module_suggests = {
@@ -227,11 +227,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -291,7 +286,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
@@ -140,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -169,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -205,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy.rst b/pypy/doc/cppyy.rst
deleted file mode 100644
--- a/pypy/doc/cppyy.rst
+++ /dev/null
@@ -1,672 +0,0 @@
-cppyy: C++ bindings for PyPy
-============================
-
-The cppyy module delivers dynamic Python-C++ bindings.
-It is designed for automation, high performance, scale, interactivity, and
-handling all of modern C++ (11, 14, etc.).
-It is based on `Cling`_ which, through `LLVM`_/`clang`_, provides C++
-reflection and interactivity.
-Reflection information is extracted from C++ header files.
-Cppyy itself is built into PyPy (an alternative exists for CPython), but
-it requires a `backend`_, installable through pip, to interface with Cling.
-
-.. _Cling: https://root.cern.ch/cling
-.. _LLVM: http://llvm.org/
-.. _clang: http://clang.llvm.org/
-.. _backend: https://pypi.python.org/pypi/PyPy-cppyy-backend
-
-
-Installation
-------------
-
-This assumes PyPy2.7 v5.7 or later; earlier versions use a Reflex-based cppyy
-module, which is no longer supported.
-Both the tooling and user-facing Python codes are very backwards compatible,
-however.
-Further dependencies are cmake (for general build), Python2.7 (for LLVM), and
-a modern C++ compiler (one that supports at least C++11).
-
-Assuming you have a recent enough version of PyPy installed, use pip to
-complete the installation of cppyy::
-
- $ MAKE_NPROCS=4 pypy-c -m pip install --verbose PyPy-cppyy-backend
-
-Set the number of parallel builds ('4' in this example, through the MAKE_NPROCS
-environment variable) to a number appropriate for your machine.
-The building process may take quite some time as it includes a customized
-version of LLVM as part of Cling, which is why --verbose is recommended so that
-you can see the build progress.
-
-The default installation will be under
-$PYTHONHOME/site-packages/cppyy_backend/lib,
-which needs to be added to your dynamic loader path (LD_LIBRARY_PATH).
-If you need the dictionary and class map generation tools (used in the examples
-below), you need to add $PYTHONHOME/site-packages/cppyy_backend/bin to your
-executable path (PATH).
-
-
-Basic bindings example
-----------------------
-
-These examples assume that cppyy_backend is pointed to by the environment
-variable CPPYYHOME, and that CPPYYHOME/lib is added to LD_LIBRARY_PATH and
-CPPYYHOME/bin to PATH.
-
-Let's first test with a trivial example whether all packages are properly
-installed and functional.
-Create a C++ header file with some class in it (all functions are made inline
-for convenience; if you have out-of-line code, link with it as appropriate)::
-
-    $ cat MyClass.h
-    class MyClass {
-    public:
-        MyClass(int i = -99) : m_myint(i) {}
-
-        int GetMyInt() { return m_myint; }
-        void SetMyInt(int i) { m_myint = i; }
-
-    public:
-        int m_myint;
-    };
-
-Then, generate the bindings using ``genreflex`` (installed under
-cppyy_backend/bin in site_packages), and compile the code::
-
-    $ genreflex MyClass.h
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling
-
-Next, make sure that the library can be found through the dynamic lookup path
-(the ``LD_LIBRARY_PATH`` environment variable on Linux, ``PATH`` on Windows),
-for example by adding ".".
-Now you're ready to use the bindings.
-Since the bindings are designed to look pythonistic, it should be
-straightforward::
-
-    $ pypy-c
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libMyClassDict.so")
-    <CPPLibrary object at 0xb6fd7c4c>
-    >>>> myinst = cppyy.gbl.MyClass(42)
-    >>>> print myinst.GetMyInt()
-    42
-    >>>> myinst.SetMyInt(33)
-    >>>> print myinst.m_myint
-    33
-    >>>> myinst.m_myint = 77
-    >>>> print myinst.GetMyInt()
-    77
-    >>>> help(cppyy.gbl.MyClass)   # shows that normal python introspection works
-
-That's all there is to it!
-
-
-Automatic class loader
-----------------------
-
-There is one big problem in the code above, that prevents its use in a (large
-scale) production setting: the explicit loading of the reflection library.
-Clearly, if explicit load statements such as these show up in code downstream
-from the ``MyClass`` package, then that prevents the ``MyClass`` author from
-repackaging or even simply renaming the dictionary library.
-
-The solution is to make use of an automatic class loader, so that downstream
-code never has to call ``load_reflection_info()`` directly.
-The class loader makes use of so-called rootmap files, which ``genreflex``
-can produce.
-These files contain the list of available C++ classes and specify the library
-that needs to be loaded for their use (as an aside, this listing allows for a
-cross-check to see whether reflection info is generated for all classes that
-you expect).
-By convention, the rootmap files should be located next to the reflection info
-libraries, so that they can be found through the normal shared library search
-path.
-They can be concatenated together, or consist of a single rootmap file per
-library.
-For example::
-
-    $ genreflex MyClass.h --rootmap=libMyClassDict.rootmap --rootmap-lib=libMyClassDict.so
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling
-
-where the first option (``--rootmap``) specifies the output file name, and the
-second option (``--rootmap-lib``) the name of the reflection library where
-``MyClass`` will live.
-It is necessary to provide that name explicitly, since it is only in the
-separate linking step where this name is fixed.
-If the second option is not given, the library is assumed to be libMyClass.so,
-a name that is derived from the name of the header file.
-
-With the rootmap file in place, the above example can be rerun without explicit
-loading of the reflection info library::
-
-    $ pypy-c
-    >>>> import cppyy
-    >>>> myinst = cppyy.gbl.MyClass(42)
-    >>>> print myinst.GetMyInt()
-    42
-    >>>> # etc. ...
-
-As a caveat, note that the class loader is currently limited to classes only.
-
-
-Advanced example
-----------------
-
-The following snippet of C++ is very contrived, to allow showing that such
-pathological code can be handled and to show how certain features play out in
-practice::
-
-    $ cat MyAdvanced.h
-    #include <string>
-
-    class Base1 {
-    public:
-        Base1(int i) : m_i(i) {}
-        virtual ~Base1() {}
-        int m_i;
-    };
-
-    class Base2 {
-    public:
-        Base2(double d) : m_d(d) {}
-        virtual ~Base2() {}
-        double m_d;
-    };
-
-    class C;
-
-    class Derived : public virtual Base1, public virtual Base2 {
-    public:
-        Derived(const std::string& name, int i, double d) : Base1(i), Base2(d), m_name(name) {}
-        virtual C* gimeC() { return (C*)0; }
-        std::string m_name;
-    };
-
-    Base2* BaseFactory(const std::string& name, int i, double d) {
-        return new Derived(name, i, d);
-    }
-
-This code is still only in a header file, with all functions inline, for
-convenience of the example.
-If the implementations live in a separate source file or shared library, the
-only change needed is to link those in when building the reflection library.
-
-If you were to run ``genreflex`` like above in the basic example, you will
-find that not all classes of interest will be reflected, nor will be the
-global factory function.
-In particular, ``std::string`` will be missing, since it is not defined in
-this header file, but in a header file that is included.
-In practical terms, general classes such as ``std::string`` should live in a
-core reflection set, but for the moment assume we want to have it in the
-reflection library that we are building for this example.
-
-The ``genreflex`` script can be steered using a so-called `selection file`_
-(see "Generating Reflex Dictionaries")
-which is a simple XML file specifying, either explicitly or by using a
-pattern, which classes, variables, namespaces, etc. to select from the given
-header file.
-With the aid of a selection file, a large project can be easily managed:
-simply ``#include`` all relevant headers into a single header file that is
-handed to ``genreflex``.
-In fact, if you hand multiple header files to ``genreflex``, then a selection
-file is almost obligatory: without it, only classes from the last header will
-be selected.
-Then, apply a selection file to pick up all the relevant classes.
-For our purposes, the following rather straightforward selection will do
-(the name ``lcgdict`` for the root is historical, but required)::
-
-    $ cat MyAdvanced.xml
-    <lcgdict>
-        <class pattern="Base?" />
-        <class name="Derived" />
-        <class name="std::string" />
-        <function name="BaseFactory" />
-    </lcgdict>
-
-.. _selection file: https://root.cern.ch/how/how-use-reflex
-
-Now the reflection info can be generated and compiled::
-
-    $ genreflex MyAdvanced.h --selection=MyAdvanced.xml
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyAdvanced_rflx.cpp -o libAdvExDict.so -L$CPPYYHOME/lib -lCling
-
-and subsequently be used from PyPy::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libAdvExDict.so")
-    <CPPLibrary object at 0x00007fdb48fc8120>
-    >>>> d = cppyy.gbl.BaseFactory("name", 42, 3.14)
-    >>>> type(d)
-    <class '__main__.Derived'>
-    >>>> isinstance(d, cppyy.gbl.Base1)
-    True
-    >>>> isinstance(d, cppyy.gbl.Base2)
-    True
-    >>>> d.m_i, d.m_d
-    (42, 3.14)
-    >>>> d.m_name == "name"
-    True
-    >>>>
-
-Again, that's all there is to it!
-
-A couple of things to note, though.
-If you look back at the C++ definition of the ``BaseFactory`` function,
-you will see that it declares the return type to be a ``Base2``, yet the
-bindings return an object of the actual type ``Derived``?
-This choice is made for a couple of reasons.
-First, it makes method dispatching easier: if bound objects are always their
-most derived type, then it is easy to calculate any offsets, if necessary.
-Second, it makes memory management easier: the combination of the type and
-the memory address uniquely identifies an object.
-That way, it can be recycled and object identity can be maintained if it is
-entered as a function argument into C++ and comes back to PyPy as a return
-value.
-Last, but not least, casting is decidedly unpythonistic.
-By always providing the most derived type known, casting becomes unnecessary.
-For example, the data member of ``Base2`` is simply directly available.
-Note also that the unreflected ``gimeC`` method of ``Derived`` does not
-preclude its use.
-It is only the ``gimeC`` method that is unusable as long as class ``C`` is
-unknown to the system.
-
-
-Features
---------
-
-The following is not meant to be an exhaustive list, since cppyy is still
-under active development.
-Furthermore, the intention is that every feature is as natural as possible on
-the python side, so if you find something missing in the list below, simply
-try it out.
-It is not always possible to provide exact mapping between python and C++
-(active memory management is one such case), but by and large, if the use of a
-feature does not strike you as obvious, it is more likely to simply be a bug.
-That is a strong statement to make, but also a worthy goal.
-For the C++ side of the examples, refer to this :doc:`example code <cppyy_example>`, which was
-bound using::
-
-    $ genreflex example.h --deep --rootmap=libexampleDict.rootmap --rootmap-lib=libexampleDict.so
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include example_rflx.cpp -o libexampleDict.so -L$CPPYYHOME/lib -lCling
-
-* **abstract classes**: Are represented as python classes, since they are
-  needed to complete the inheritance hierarchies, but will raise an exception
-  if an attempt is made to instantiate from them.
-  Example::
-
-    >>>> from cppyy.gbl import AbstractClass, ConcreteClass
-    >>>> a = AbstractClass()
-    Traceback (most recent call last):
-      File "<console>", line 1, in <module>
-    TypeError: cannot instantiate abstract class 'AbstractClass'
-    >>>> issubclass(ConcreteClass, AbstractClass)
-    True
-    >>>> c = ConcreteClass()
-    >>>> isinstance(c, AbstractClass)
-    True
-    >>>>
-
-* **arrays**: Supported for builtin data types only, as used from module
-  ``array``.
-  Out-of-bounds checking is limited to those cases where the size is known at
-  compile time (and hence part of the reflection info).
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> from array import array
-    >>>> c = ConcreteClass()
-    >>>> c.array_method(array('d', [1., 2., 3., 4.]), 4)
-    1 2 3 4
-    >>>>
-
-* **builtin data types**: Map onto the expected equivalent python types, with
-  the caveat that there may be size differences, and thus it is possible that
-  exceptions are raised if an overflow is detected.
-
-* **casting**: Is supposed to be unnecessary.
-  Object pointer returns from functions provide the most derived class known
-  in the hierarchy of the object being returned.
-  This is important to preserve object identity as well as to make casting,
-  a pure C++ feature after all, superfluous.
-  Example::
-
-    >>>> from cppyy.gbl import AbstractClass, ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> ConcreteClass.show_autocast.__doc__
-    'AbstractClass* ConcreteClass::show_autocast()'
-    >>>> d = c.show_autocast()
-    >>>> type(d)
-    <class '__main__.ConcreteClass'>
-    >>>>
-
-  However, if need be, you can perform C++-style reinterpret_casts (i.e.
-  without taking offsets into account), by taking and rebinding the address
-  of an object::
-
-    >>>> from cppyy import addressof, bind_object
-    >>>> e = bind_object(addressof(d), AbstractClass)
-    >>>> type(e)
-    <class '__main__.AbstractClass'>
-    >>>>
-
-* **classes and structs**: Get mapped onto python classes, where they can be
-  instantiated as expected.
-  If classes are inner classes or live in a namespace, their naming and
-  location will reflect that.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass, Namespace
-    >>>> ConcreteClass == Namespace.ConcreteClass
-    False
-    >>>> n = Namespace.ConcreteClass.NestedClass()
-    >>>> type(n)
-    <class '__main__.Namespace::ConcreteClass::NestedClass'>
-    >>>>
-
-* **data members**: Public data members are represented as python properties
-  and provide read and write access on instances as expected.
-  Private and protected data members are not accessible.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> c.m_int
-    42
-    >>>>
-
-* **default arguments**: C++ default arguments work as expected, but python
-  keywords are not supported.
-  It is technically possible to support keywords, but for the C++ interface,
-  the formal argument names have no meaning and are not considered part of the
-  API, hence it is not a good idea to use keywords.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()       # uses default argument
-    >>>> c.m_int
-    42
-    >>>> c = ConcreteClass(13)
-    >>>> c.m_int
-    13
-    >>>>
-
-* **doc strings**: The doc string of a method or function contains the C++
-  arguments and return types of all overloads of that name, as applicable.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> print ConcreteClass.array_method.__doc__
-    void ConcreteClass::array_method(int*, int)
-    void ConcreteClass::array_method(double*, int)
-    >>>>
-
-* **enums**: Are translated as ints with no further checking.
-
-* **functions**: Work as expected and live in their appropriate namespace
-  (which can be the global one, ``cppyy.gbl``).
-
-* **inheritance**: All combinations of inheritance on the C++ (single,
-  multiple, virtual) are supported in the binding.
-  However, new python classes can only use single inheritance from a bound C++
-  class.
-  Multiple inheritance would introduce two "this" pointers in the binding.
-  This is a current, not a fundamental, limitation.
-  The C++ side will not see any overridden methods on the python side, as
-  cross-inheritance is planned but not yet supported.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> help(ConcreteClass)
-    Help on class ConcreteClass in module __main__:
-
-    class ConcreteClass(AbstractClass)
-     |  Method resolution order:
-     |      ConcreteClass
-     |      AbstractClass
-     |      cppyy.CPPObject
-     |      __builtin__.CPPInstance
-     |      __builtin__.object
-     |
-     |  Methods defined here:
-     |
-     |  ConcreteClass(self, *args)
-     |      ConcreteClass::ConcreteClass(const ConcreteClass&)
-     |      ConcreteClass::ConcreteClass(int)
-     |      ConcreteClass::ConcreteClass()
-     |
-     etc. ....
-
-* **memory**: C++ instances created by calling their constructor from python
-  are owned by python.
-  You can check/change the ownership with the _python_owns flag that every
-  bound instance carries.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> c._python_owns            # True: object created in Python
-    True
-    >>>>
-
-* **methods**: Are represented as python methods and work as expected.
-  They are first class objects and can be bound to an instance.
-  Virtual C++ methods work as expected.
-  To select a specific virtual method, do like with normal python classes
-  that override methods: select it from the class that you need, rather than
-  calling the method on the instance.
-  To select a specific overload, use the __dispatch__ special function, which
-  takes the name of the desired method and its signature (which can be
-  obtained from the doc string) as arguments.
-
-* **namespaces**: Are represented as python classes.
-  Namespaces are more open-ended than classes, so sometimes initial access may
-  result in updates as data and functions are looked up and constructed
-  lazily.
-  Thus the result of ``dir()`` on a namespace shows the classes available,
-  even if they may not have been created yet.
-  It does not show classes that could potentially be loaded by the class
-  loader.
-  Once created, namespaces are registered as modules, to allow importing from
-  them.
-  Namespace currently do not work with the class loader.
-  Fixing these bootstrap problems is on the TODO list.
-  The global namespace is ``cppyy.gbl``.
-
-* **NULL**: Is represented as ``cppyy.gbl.nullptr``.
-  In C++11, the keyword ``nullptr`` is used to represent ``NULL``.
-  For clarity of intent, it is recommended to use this instead of ``None``
-  (or the integer ``0``, which can serve in some cases), as ``None`` is better
-  understood as ``void`` in C++.
-
-* **operator conversions**: If defined in the C++ class and a python
-  equivalent exists (i.e. all builtin integer and floating point types, as well
-  as ``bool``), it will map onto that python conversion.
-  Note that ``char*`` is mapped onto ``__str__``.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> print ConcreteClass()
-    Hello operator const char*!
-    >>>>
-
-* **operator overloads**: If defined in the C++ class and if a python
-  equivalent is available (not always the case, think e.g. of ``operator||``),
-  then they work as expected.
-  Special care needs to be taken for global operator overloads in C++: first,
-  make sure that they are actually reflected, especially for the global
-  overloads for ``operator==`` and ``operator!=`` of STL vector iterators in
-  the case of gcc (note that they are not needed to iterate over a vector).
-  Second, make sure that reflection info is loaded in the proper order.
-  I.e. that these global overloads are available before use.
-
-* **pointers**: For builtin data types, see arrays.
-  For objects, a pointer to an object and an object looks the same, unless
-  the pointer is a data member.
-  In that case, assigning to the data member will cause a copy of the pointer
-  and care should be taken about the object's life time.
-  If a pointer is a global variable, the C++ side can replace the underlying
-  object and the python side will immediately reflect that.
-
-* **PyObject***: Arguments and return types of ``PyObject*`` can be used, and
-  passed on to CPython API calls.
-  Since these CPython-like objects need to be created and tracked (this all
-  happens through ``cpyext``) this interface is not particularly fast.
-
-* **static data members**: Are represented as python property objects on the
-  class and the meta-class.
-  Both read and write access is as expected.
-
-* **static methods**: Are represented as python's ``staticmethod`` objects
-  and can be called both from the class as well as from instances.
-
-* **strings**: The std::string class is considered a builtin C++ type and
-  mixes quite well with python's str.
-  Python's str can be passed where a ``const char*`` is expected, and an str
-  will be returned if the return type is ``const char*``.
-
-* **templated classes**: Are represented in a meta-class style in python.
-  This may look a little bit confusing, but conceptually is rather natural.
-  For example, given the class ``std::vector<int>``, the meta-class part would
-  be ``std.vector``.
-  Then, to get the instantiation on ``int``, do ``std.vector(int)`` and to
-  create an instance of that class, do ``std.vector(int)()``::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info('libexampleDict.so')
-    >>>> cppyy.gbl.std.vector                # template metatype
-    <cppyy.CppyyTemplateType object at 0x00007fcdd330f1a0>
-    >>>> cppyy.gbl.std.vector(int)           # instantiates template -> class
-    <class '__main__.std::vector<int>'>
-    >>>> cppyy.gbl.std.vector(int)()         # instantiates class -> object
-    <__main__.std::vector<int> object at 0x00007fe480ba4bc0>
-    >>>>
-
-  Note that templates can be build up by handing actual types to the class
-  instantiation (as done in this vector example), or by passing in the list of
-  template arguments as a string.
-  The former is a lot easier to work with if you have template instantiations
-  using classes that themselves are templates in  the arguments (think e.g a
-  vector of vectors).
-  All template classes must already exist in the loaded reflection info, they
-  do not work (yet) with the class loader.
-
-  For compatibility with other bindings generators, use of square brackets
-  instead of parenthesis to instantiate templates is supported as well.
-
-* **templated functions**: Automatically participate in overloading and are
-  used in the same way as other global functions.
-
-* **templated methods**: For now, require an explicit selection of the
-  template parameters.
-  This will be changed to allow them to participate in overloads as expected.
-
-* **typedefs**: Are simple python references to the actual classes to which
-  they refer.
-
-* **unary operators**: Are supported if a python equivalent exists, and if the
-  operator is defined in the C++ class.
-
-You can always find more detailed examples and see the full of supported
-features by looking at the tests in pypy/module/cppyy/test.
-
-If a feature or reflection info is missing, this is supposed to be handled
-gracefully.
-In fact, there are unit tests explicitly for this purpose (even as their use
-becomes less interesting over time, as the number of missing features
-decreases).
-Only when a missing feature is used, should there be an exception.
-For example, if no reflection info is available for a return type, then a
-class that has a method with that return type can still be used.
-Only that one specific method can not be used.
-
-
-Templates
----------
-
-Templates can be automatically instantiated, assuming the appropriate header
-files have been loaded or are accessible to the class loader.
-This is the case for example for all of STL.
-For example::
-
-    $ cat MyTemplate.h
-    #include <vector>
-
-    class MyClass {
-    public:
-        MyClass(int i = -99) : m_i(i) {}
-        MyClass(const MyClass& s) : m_i(s.m_i) {}
-        MyClass& operator=(const MyClass& s) { m_i = s.m_i; return *this; }
-        ~MyClass() {}
-        int m_i;
-    };
-
-Run the normal ``genreflex`` and compilation steps::
-
-    $ genreflex MyTemplate.h --selection=MyTemplate.xml
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyTemplate_rflx.cpp -o libTemplateDict.so -L$CPPYYHOME/lib -lCling
-
-Subsequent use should be as expected.
-Note the meta-class style of "instantiating" the template::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libTemplateDict.so")
-    >>>> std = cppyy.gbl.std
-    >>>> MyClass = cppyy.gbl.MyClass
-    >>>> v = std.vector(MyClass)()
-    >>>> v += [MyClass(1), MyClass(2), MyClass(3)]
-    >>>> for m in v:
-    ....     print m.m_i,
-    ....
-    1 2 3
-    >>>>
-
-The arguments to the template instantiation can either be a string with the
-full list of arguments, or the explicit classes.
-The latter makes for easier code writing if the classes passed to the
-instantiation are themselves templates.
-
-
-The fast lane
--------------
-
-By default, cppyy will use direct function pointers through `CFFI`_ whenever
-possible. If this causes problems for you, you can disable it by setting the
-CPPYY_DISABLE_FASTPATH environment variable.
-
-.. _CFFI: https://cffi.readthedocs.io/en/latest/
-
-
-CPython
--------
-
-Most of the ideas in cppyy come originally from the `PyROOT`_ project, which
-contains a CPython-based cppyy.py module (with similar dependencies as the
-one that comes with PyPy).
-A standalone pip-installable version is planned, but for now you can install
-ROOT through your favorite distribution installer (available in the science
-section).
-
-.. _PyROOT: https://root.cern.ch/pyroot
-
-There are a couple of minor differences between the two versions of cppyy
-(the CPython version has a few more features).
-Work is on-going to integrate the nightly tests of both to make sure their
-feature sets are equalized.
-
-
-Python3
--------
-
-The CPython version of cppyy supports Python3, assuming your packager has
-build the backend for it.
-The cppyy module has not been tested with the `Py3k`_ version of PyPy.
-Note that the generated reflection information (from ``genreflex``) is fully
-independent of Python, and does not need to be rebuild when switching versions
-or interpreters.
-
-.. _Py3k: https://bitbucket.org/pypy/pypy/src/py3k
-
-
-.. toctree::
-   :hidden:
-
-   cppyy_example
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst
--- a/pypy/doc/cpython_differences.rst
+++ b/pypy/doc/cpython_differences.rst
@@ -337,6 +337,8 @@
 
  - ``frozenset`` (empty frozenset only)
 
+ - unbound method objects (for Python 2 only)
+
 This change requires some changes to ``id`` as well. ``id`` fulfills the
 following condition: ``x is y <=> id(x) == id(y)``. Therefore ``id`` of the
 above types will return a value that is computed from the argument, and can
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -61,29 +61,22 @@
 .. _libffi: http://sourceware.org/libffi/
 
 
-Cling and cppyy
----------------
+cppyy
+-----
 
-The builtin :doc:`cppyy <cppyy>` module uses reflection information, provided by
-`Cling`_ (which needs to be `installed separately`_), of C/C++ code to
-automatically generate bindings at runtime.
-In Python, classes and functions are always runtime structures, so when they
-are generated matters not for performance.
-However, if the backend itself is capable of dynamic behavior, it is a much
-better functional match, allowing tighter integration and more natural
-language mappings.
+For C++, _cppyy_ is an automated bindings generator available for both
+PyPy and CPython.
+_cppyy_ relies on declarations from C++ header files to dynamically
+construct Python equivalent classes, functions, variables, etc.
+It is designed for use by large scale programs and supports modern C++.
+With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
+remove most of the cross-language overhead.
 
-The :doc:`cppyy <cppyy>` module is written in RPython, thus PyPy's JIT is able to remove
-most cross-language call overhead.
+To install, run ``pip install cppyy``.
+Further details are available in the `full documentation`_.
 
-:doc:Full details are `available here <cppyy>`.
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
-.. _installed separately: https://pypi.python.org/pypi/PyPy-cppyy-backend
-.. _Cling: https://root.cern.ch/cling
-
-.. toctree::
-
-   cppyy
 
 RPython Mixed Modules
 ---------------------
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------
diff --git a/pypy/doc/how-to-release.rst b/pypy/doc/how-to-release.rst
--- a/pypy/doc/how-to-release.rst
+++ b/pypy/doc/how-to-release.rst
@@ -40,6 +40,9 @@
   sure things are ported back to the trunk and to the branch as
   necessary.
 
+* Maybe bump the SOABI number in module/imp/importing. This has many
+  implications, so make sure the PyPy community agrees to the change.
+
 * Update and write documentation
 
   * update pypy/doc/contributor.rst (and possibly LICENSE)
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -5,6 +5,14 @@
 .. this is a revision shortly after release-pypy2.7-v5.8.0
 .. startrev: 558bd00b3dd8
 
+In previous versions of PyPy, ``instance.method`` would return always
+the same bound method object, when gotten out of the same instance (as
+far as ``is`` and ``id()`` can tell).  CPython doesn't do that.  Now
+PyPy, like CPython, returns a different bound method object every time.
+For ``type.method``, PyPy2 still returns always the same *unbound*
+method object; CPython does it for built-in types but not for
+user-defined types.
+
 .. branch: cffi-complex
 .. branch: cffi-char16-char32
 
@@ -25,3 +33,43 @@
 .. branch: cpyext-hash_notimpl
 
 If ``tp_hash`` is ``PyObject_HashNotImplemented``, set ``obj.__dict__['__hash__']`` to None
+
+.. branch: cppyy-packaging
+
+Renaming of ``cppyy`` to ``_cppyy``.
+The former is now an external package installable with ``pip install cppyy``.
+
+.. branch: Enable_PGO_for_clang
+
+.. branch: nopax
+
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
+
+.. branch: pypy_bytearray
+
+Improve ``bytearray`` performance (backported from py3.5)
+
+.. branch: gc-del-limit-growth
+
+Fix the bounds in the GC when allocating a lot of objects with finalizers,
+fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
+
+
+.. branch: getarrayitem-into-bridges:
+
+More information is retained into a bridge: knowledge about the content of
+arrays (at fixed indices) is stored in guards (and thus available at the
+beginning of bridges). Also, some better feeding of information about known
+fields of constant objects into bridges.
+
+.. branch: cpyext-leakchecking
+
+Add support for leakfinder in cpyext tests (disabled for now, due to too many
+failures).
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import enforceargs
 from rpython.rlib.rstring import StringBuilder
@@ -48,8 +49,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1,4 +1,5 @@
 import sys
+import py
 
 from rpython.rlib.cache import Cache
 from rpython.tool.uid import HUGEVAL_BYTES
@@ -1271,8 +1272,22 @@
             self.setitem(w_globals, w_key, self.builtin)
         return statement.exec_code(self, w_globals, w_locals)
 
+    @not_rpython
+    def appdef(self, source):
+        '''Create interp-level function object from app-level source.
+
+        The source should be in the same format as for space.appexec():
+            """(foo, bar): return 'baz'"""
+        '''
+        source = source.lstrip()
+        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
+        source = py.code.Source("def anonymous%s\n" % source)
+        w_glob = self.newdict(module=True)
+        self.exec_(str(source), w_glob, w_glob)
+        return self.getitem(w_glob, self.newtext('anonymous'))
+
     @specialize.arg(2)
-    def appexec(self, posargs_w, source):
+    def appexec(self, posargs_w, source, cache=True):
         """ return value from executing given source at applevel.
             The source must look like
                '''(x, y):
@@ -1280,7 +1295,11 @@
                        return result
                '''
         """
-        w_func = self.fromcache(AppExecCache).getorbuild(source)
+        if cache:
+            w_func = self.fromcache(AppExecCache).getorbuild(source)
+        else:
+            # NB: since appdef() is not-RPython, using cache=False also is.
+            w_func = self.appdef(source)
         args = Arguments(self, list(posargs_w))
         return self.call_args(w_func, args)
 
@@ -1817,15 +1836,7 @@
 class AppExecCache(SpaceCache):
     @not_rpython
     def build(cache, source):
-        space = cache.space
-        # XXX will change once we have our own compiler
-        import py
-        source = source.lstrip()
-        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
-        source = py.code.Source("def anonymous%s\n" % source)
-        w_glob = space.newdict(module=True)
-        space.exec_(str(source), w_glob, w_glob)
-        return space.getitem(w_glob, space.newtext('anonymous'))
+        return cache.space.appdef(source)
 
 
 # Table describing the regular part of the interface of object spaces,
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,7 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
-from rpython.rlib.objectmodel import dont_inline
+from rpython.rlib.objectmodel import dont_inline, not_rpython
 from rpython.rlib import rstack, rstackovf
 from rpython.rlib import rwin32
 from rpython.rlib import runicode
@@ -65,8 +65,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -119,15 +120,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -154,8 +156,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,6 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -410,8 +411,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -516,8 +516,9 @@
 
     def __init__(self, space, w_function, w_instance):
         self.space = space
+        assert w_instance is not None   # unbound methods only exist in Python 2
         self.w_function = w_function
-        self.w_instance = w_instance   # or None
+        self.w_instance = w_instance
 
     def descr_method__new__(space, w_subtype, w_function, w_instance):
         if space.is_w(w_instance, space.w_None):
@@ -577,24 +578,6 @@
             return space.w_False
         return space.newbool(space.eq_w(self.w_function, w_other.w_function))
 
-    def is_w(self, space, other):
-        if not isinstance(other, Method):
-            return False
-        return (self.w_instance is other.w_instance and
-                self.w_function is other.w_function)
-
-    def immutable_unique_id(self, space):
-        from pypy.objspace.std.util import IDTAG_METHOD as tag
-        from pypy.objspace.std.util import IDTAG_SHIFT
-        if self.w_instance is not None:
-            id = space.bigint_w(space.id(self.w_instance))
-            id = id.lshift(LONG_BIT)
-        else:
-            id = rbigint.fromint(0)
-        id = id.or_(space.bigint_w(space.id(self.w_function)))
-        id = id.lshift(IDTAG_SHIFT).int_or_(tag)
-        return space.newlong_from_rbigint(id)
-
     def descr_method_hash(self):
         space = self.space
         w_result = space.hash(self.w_function)
@@ -606,7 +589,7 @@
         from pypy.interpreter.gateway import BuiltinCode
         w_mod    = space.getbuiltinmodule('_pickle_support')
         mod      = space.interp_w(MixedModule, w_mod)
-        w_instance = self.w_instance or space.w_None
+        w_instance = self.w_instance
         w_function = self.w_function
         if (isinstance(w_function, Function) and
                 isinstance(w_function.code, BuiltinCode)):
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -75,8 +75,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -399,8 +399,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):
-        """NOT_RPYTHON"""
         self.behavior = behavior
 
     def _run(self, space, scope_w):
@@ -654,9 +654,9 @@
     # When a BuiltinCode is stored in a Function object,
     # you get the functionality of CPython's built-in function type.
 
+    @not_rpython
     def __init__(self, func, unwrap_spec=None, self_type=None,
                  descrmismatch=None, doc=None):
-        "NOT_RPYTHON"
         # 'implfunc' is the interpreter-level function.
         # Note that this uses a lot of (construction-time) introspection.
         Code.__init__(self, func.__name__)
@@ -1004,10 +1004,10 @@
 
     instancecache = {}
 
+    @not_rpython
     def __new__(cls, f, app_name=None, unwrap_spec=None, descrmismatch=None,
                 as_classmethod=False, doc=None):
 
-        "NOT_RPYTHON"
         # f must be a function whose name does NOT start with 'app_'
         self_type = None
         if hasattr(f, 'im_func'):
@@ -1047,8 +1047,8 @@
 
         return self
 
+    @not_rpython
     def _getdefaults(self, space):
-        "NOT_RPYTHON"
         alldefs_w = {}
         assert len(self._code._argnames) == len(self._code._unwrap_spec)
         for name, spec in zip(self._code._argnames, self._code._unwrap_spec):
@@ -1124,8 +1124,8 @@
 
 
 class GatewayCache(SpaceCache):
+    @not_rpython
     def build(cache, gateway):
-        "NOT_RPYTHON"
         space = cache.space
         defs_w, kw_defs_w = gateway._getdefaults(space)
         code = gateway._code
@@ -1196,8 +1196,8 @@
         w_globals = self.getwdict(space)
         return space.getitem(w_globals, space.newtext(name))
 
+    @not_rpython
     def interphook(self, name):
-        "NOT_RPYTHON"
         def appcaller(space, *args_w):
             if not isinstance(space, ObjSpace):
                 raise TypeError("first argument must be a space instance.")
@@ -1234,15 +1234,16 @@
     """NOT_RPYTHON
     The cache mapping each applevel instance to its lazily built w_dict"""
 
+    @not_rpython
     def build(self, app):
-        "NOT_RPYTHON.  Called indirectly by Applevel.getwdict()."
+        "Called indirectly by Applevel.getwdict()."
         return build_applevel_dict(app, self.space)
 
 
 # __________ pure applevel version __________
 
+ at not_rpython
 def build_applevel_dict(self, space):
-    "NOT_RPYTHON"
     w_glob = space.newdict(module=True)
     space.setitem(w_glob, space.newtext('__name__'), space.newtext(self.modname))
     space.exec_(self.source, w_glob, w_glob,
@@ -1253,8 +1254,9 @@
 # ____________________________________________________________
 
 
+ at not_rpython
 def appdef(source, applevel=ApplevelClass, filename=None):
-    """ NOT_RPYTHON: build an app-level helper function, like for example:
+    """ build an app-level helper function, like for example:
     myfunc = appdef('''myfunc(x, y):
                            return x+y
                     ''')
@@ -1300,6 +1302,6 @@
 
 
 # app2interp_temp is used for testing mainly
+ at not_rpython
 def app2interp_temp(func, applevel_temp=applevel_temp, filename=None):
-    """ NOT_RPYTHON """
     return appdef(func, applevel_temp, filename=filename)
diff --git a/pypy/interpreter/miscutils.py b/pypy/interpreter/miscutils.py
--- a/pypy/interpreter/miscutils.py
+++ b/pypy/interpreter/miscutils.py
@@ -3,6 +3,7 @@
 """
 
 from rpython.rlib.listsort import make_timsort_class
+from rpython.rlib.objectmodel import not_rpython
 
 
 class ThreadLocals:
@@ -41,9 +42,8 @@
         # but in some corner cases it is not...  unsure why
         self._value = None
 
-
+ at not_rpython
 def make_weak_value_dictionary(space, keytype, valuetype):
-    "NOT_RPYTHON"
     if space.config.translation.rweakref:
         from rpython.rlib.rweakref import RWeakValueDictionary
         return RWeakValueDictionary(keytype, valuetype)
diff --git a/pypy/interpreter/mixedmodule.py b/pypy/interpreter/mixedmodule.py
--- a/pypy/interpreter/mixedmodule.py
+++ b/pypy/interpreter/mixedmodule.py
@@ -3,6 +3,9 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root
+
+from rpython.rlib.objectmodel import not_rpython
+
 import sys
 
 class MixedModule(Module):
@@ -15,8 +18,8 @@
     lazy = False
     submodule_name = None
 
+    @not_rpython
     def __init__(self, space, w_name):
-        """ NOT_RPYTHON """
         Module.__init__(self, space, w_name)
         init_extra_module_attrs(space, self)
         self.lazy = True
@@ -25,8 +28,9 @@
         self.loaders = self.loaders.copy()    # copy from the class to the inst
         self.submodules_w = []
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: install this module, and it's submodules into
+        """install this module, and it's submodules into
         space.builtin_modules"""
         Module.install(self)
         if hasattr(self, "submodules"):
@@ -66,8 +70,8 @@
         self.w_initialdict = self.space.call_method(w_dict, 'copy')
 
     @classmethod
+    @not_rpython
     def get_applevel_name(cls):
-        """ NOT_RPYTHON """
         if cls.applevel_name is not None:
             return cls.applevel_name
         else:
@@ -163,8 +167,8 @@
         self._frozen = True
 
     @classmethod
+    @not_rpython
     def buildloaders(cls):
-        """ NOT_RPYTHON """
         if not hasattr(cls, 'loaders'):
             # build a constant dictionary out of
             # applevel/interplevel definitions
@@ -194,8 +198,8 @@
         return space.newtext_or_none(cls.__doc__)
 
 
+ at not_rpython
 def getinterpevalloader(pkgroot, spec):
-    """ NOT_RPYTHON """
     def ifileloader(space):
         d = {'space': space}
         # EVIL HACK (but it works, and this is not RPython :-)
@@ -235,8 +239,8 @@
     return ifileloader
 
 applevelcache = {}
+ at not_rpython
 def getappfileloader(pkgroot, appname, spec):
-    """ NOT_RPYTHON """
     # hum, it's a bit more involved, because we usually
     # want the import at applevel
     modname, attrname = spec.split('.')
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -4,7 +4,7 @@
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.interpreter.error import OperationError, oefmt
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 
 
 class Module(W_Root):
@@ -35,8 +35,9 @@
         except OperationError:
             pass
 
+    @not_rpython
     def install(self):
-        """NOT_RPYTHON: installs this module into space.builtin_modules"""
+        """installs this module into space.builtin_modules"""
         modulename = self.space.text0_w(self.w_name)
         if modulename in self.space.builtin_modules:
             raise ValueError(
@@ -44,8 +45,9 @@
                 "app-level module %r" % (modulename,))
         self.space.builtin_modules[modulename] = self
 
+    @not_rpython
     def setup_after_space_initialization(self):
-        """NOT_RPYTHON: to allow built-in modules to do some more setup
+        """to allow built-in modules to do some more setup
         after the space is fully initialized."""
 
     def init(self, space):
diff --git a/pypy/interpreter/pyframe.py b/pypy/interpreter/pyframe.py
--- a/pypy/interpreter/pyframe.py
+++ b/pypy/interpreter/pyframe.py
@@ -7,6 +7,7 @@
 from rpython.rlib.debug import ll_assert_not_none
 from rpython.rlib.jit import hint
 from rpython.rlib.objectmodel import instantiate, specialize, we_are_translated
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib.rarithmetic import intmask, r_uint
 from rpython.tool.pairtype import extendabletype
 
@@ -146,8 +147,9 @@
             return None
         return d.w_locals
 
+    @not_rpython
     def __repr__(self):
-        # NOT_RPYTHON: useful in tracebacks
+        # useful in tracebacks
         return "<%s.%s executing %s at line %s" % (
             self.__class__.__module__, self.__class__.__name__,
             self.pycode, self.get_last_lineno())
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -7,7 +7,7 @@
 from rpython.rlib import jit, rstackovf, rstring
 from rpython.rlib.debug import check_nonneg
 from rpython.rlib.objectmodel import (
-    we_are_translated, always_inline, dont_inline)
+    we_are_translated, always_inline, dont_inline, not_rpython)
 from rpython.rlib.rarithmetic import r_uint, intmask
 from rpython.tool.sourcetools import func_with_new_name
 
@@ -23,8 +23,8 @@
 CANNOT_CATCH_MSG = ("catching classes that don't inherit from BaseException "
                     "is not allowed in 3.x")
 
+ at not_rpython
 def unaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_1 = self.popvalue()
@@ -34,8 +34,8 @@
 
     return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname)
 
+ at not_rpython
 def binaryoperation(operationname):
-    """NOT_RPYTHON"""
     def opimpl(self, *ignored):
         operation = getattr(self.space, operationname)
         w_2 = self.popvalue()
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -1,5 +1,5 @@
 # encoding: utf-8
-import pytest
+import pytest, sys
 from pypy.interpreter import eval
 from pypy.interpreter.function import Function, Method, descr_function_get
 from pypy.interpreter.pycode import PyCode
@@ -416,6 +416,11 @@
             raises(ValueError, FunctionType.__setstate__, f, (1, 2, 3))
 
 class AppTestMethod:
+    def setup_class(cls):
+        cls.w_runappdirect_on_cpython = cls.space.wrap(
+            cls.runappdirect and
+            '__pypy__' not in sys.builtin_module_names)
+
     def test_simple_call(self):
         class A(object):
             def func(self, arg2):
@@ -586,7 +591,6 @@
         assert meth == meth
         assert meth == MethodType(func, object)
 
-    @pytest.mark.skipif("config.option.runappdirect")
     def test_method_identity(self):
         class A(object):
             def m(self):
@@ -603,19 +607,24 @@
 
         a = A()
         a2 = A()
-        assert a.m is a.m
-        assert id(a.m) == id(a.m)
-        assert a.m is not a.n
-        assert id(a.m) != id(a.n)
-        assert a.m is not a2.m
-        assert id(a.m) != id(a2.m)
+        x = a.m; y = a.m
+        assert x is not y
+        assert id(x) != id(y)
+        assert x == y
+        assert x is not a.n
+        assert id(x) != id(a.n)
+        assert x is not a2.m
+        assert id(x) != id(a2.m)
 
-        assert A.m is A.m
-        assert id(A.m) == id(A.m)
-        assert A.m is not A.n
-        assert id(A.m) != id(A.n)
-        assert A.m is B.m
-        assert id(A.m) == id(B.m)
+        if not self.runappdirect_on_cpython:
+            assert A.m is A.m

From pypy.commits at gmail.com  Tue Aug  8 12:46:06 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 08 Aug 2017 09:46:06 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: uncomment disabled code in
 multiphase2.c
Message-ID: <5989eace.0e951c0a.f71e3.7460@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92112:ac2fc4b50086
Date: 2017-08-08 18:45 +0200
http://bitbucket.org/pypy/pypy/changeset/ac2fc4b50086/

Log:	uncomment disabled code in multiphase2.c

diff --git a/pypy/module/cpyext/test/multiphase2.c b/pypy/module/cpyext/test/multiphase2.c
--- a/pypy/module/cpyext/test/multiphase2.c
+++ b/pypy/module/cpyext/test/multiphase2.c
@@ -249,51 +249,51 @@
 
 /**** Importing a non-module object ****/
 
-//static PyModuleDef def_nonmodule;
-//static PyModuleDef def_nonmodule_with_methods;
-//
-///* Create a SimpleNamespace(three=3) */
-//static PyObject*
-//createfunc_nonmodule(PyObject *spec, PyModuleDef *def)
-//{
-//    PyObject *dct, *ns, *three;
-//
-//    if (def != &def_nonmodule && def != &def_nonmodule_with_methods) {
-//        PyErr_SetString(PyExc_SystemError, "def does not match");
-//        return NULL;
-//    }
-//
-//    dct = PyDict_New();
-//    if (dct == NULL)
-//        return NULL;
-//
-//    three = PyLong_FromLong(3);
-//    if (three == NULL) {
-//        Py_DECREF(dct);
-//        return NULL;
-//    }
-//    PyDict_SetItemString(dct, "three", three);
-//    Py_DECREF(three);
-//
-//    ns = _PyNamespace_New(dct);
-//    Py_DECREF(dct);
-//    return ns;
-//}
-//
-//static PyModuleDef_Slot slots_create_nonmodule[] = {
-//    {Py_mod_create, createfunc_nonmodule},
-//    {0, NULL},
-//};
-//
-//static PyModuleDef def_nonmodule = TEST_MODULE_DEF(
-//    "_testmultiphase_nonmodule", slots_create_nonmodule, NULL);
-//
-//PyMODINIT_FUNC
-//PyInit__testmultiphase_nonmodule(PyObject *spec)
-//{
-//    return PyModuleDef_Init(&def_nonmodule);
-//}
-/*
+static PyModuleDef def_nonmodule;
+static PyModuleDef def_nonmodule_with_methods;
+
+/* Create a SimpleNamespace(three=3) */
+static PyObject*
+createfunc_nonmodule(PyObject *spec, PyModuleDef *def)
+{
+    PyObject *dct, *ns, *three;
+
+    if (def != &def_nonmodule && def != &def_nonmodule_with_methods) {
+        PyErr_SetString(PyExc_SystemError, "def does not match");
+        return NULL;
+    }
+
+    dct = PyDict_New();
+    if (dct == NULL)
+        return NULL;
+
+    three = PyLong_FromLong(3);
+    if (three == NULL) {
+        Py_DECREF(dct);
+        return NULL;
+    }
+    PyDict_SetItemString(dct, "three", three);
+    Py_DECREF(three);
+
+    ns = _PyNamespace_New(dct);
+    Py_DECREF(dct);
+    return ns;
+}
+
+static PyModuleDef_Slot slots_create_nonmodule[] = {
+    {Py_mod_create, createfunc_nonmodule},
+    {0, NULL},
+};
+
+static PyModuleDef def_nonmodule = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule", slots_create_nonmodule, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule);
+}
+
 PyDoc_STRVAR(nonmodule_bar_doc,
 "bar(i,j)\n\
 \n\
@@ -309,20 +309,20 @@
     res = i - j;
     return PyLong_FromLong(res);
 }
-*/
-//static PyMethodDef nonmodule_methods[] = {
-//    {"bar", nonmodule_bar, METH_VARARGS, nonmodule_bar_doc},
-//    {NULL, NULL}           /* sentinel */
-//};
-//
-//static PyModuleDef def_nonmodule_with_methods = TEST_MODULE_DEF(
-//    "_testmultiphase_nonmodule_with_methods", slots_create_nonmodule, nonmodule_methods);
-//
-//PyMODINIT_FUNC
-//PyInit__testmultiphase_nonmodule_with_methods(PyObject *spec)
-//{
-//    return PyModuleDef_Init(&def_nonmodule_with_methods);
-//}
+
+static PyMethodDef nonmodule_methods[] = {
+    {"bar", nonmodule_bar, METH_VARARGS, nonmodule_bar_doc},
+    {NULL, NULL}           /* sentinel */
+};
+
+static PyModuleDef def_nonmodule_with_methods = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule_with_methods", slots_create_nonmodule, nonmodule_methods);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule_with_methods(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule_with_methods);
+}
 
 /**** Non-ASCII-named modules ****/
 
@@ -539,20 +539,20 @@
     return PyModuleDef_Init(&def_create_unreported_exception);
 }
 
-//static PyModuleDef_Slot slots_nonmodule_with_exec_slots[] = {
-//    {Py_mod_create, createfunc_nonmodule},
-//    {Py_mod_exec, execfunc},
-//    {0, NULL},
-//};
-//
-//static PyModuleDef def_nonmodule_with_exec_slots = TEST_MODULE_DEF(
-//    "_testmultiphase_nonmodule_with_exec_slots", slots_nonmodule_with_exec_slots, NULL);
-//
-//PyMODINIT_FUNC
-//PyInit__testmultiphase_nonmodule_with_exec_slots(PyObject *spec)
-//{
-//    return PyModuleDef_Init(&def_nonmodule_with_exec_slots);
-//}
+static PyModuleDef_Slot slots_nonmodule_with_exec_slots[] = {
+    {Py_mod_create, createfunc_nonmodule},
+    {Py_mod_exec, execfunc},
+    {0, NULL},
+};
+
+static PyModuleDef def_nonmodule_with_exec_slots = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule_with_exec_slots", slots_nonmodule_with_exec_slots, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule_with_exec_slots(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule_with_exec_slots);
+}
 
 static int
 execfunc_err(PyObject *mod)

From pypy.commits at gmail.com  Tue Aug  8 21:41:10 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 08 Aug 2017 18:41:10 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Add test
Message-ID: <598a6836.4f8f1c0a.91e7.9a63@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92113:c0c2ccfe883f
Date: 2017-08-09 03:40 +0200
http://bitbucket.org/pypy/pypy/changeset/c0c2ccfe883f/

Log:	Add test

diff --git a/pypy/module/cpyext/test/multiphase2.c b/pypy/module/cpyext/test/multiphase2.c
--- a/pypy/module/cpyext/test/multiphase2.c
+++ b/pypy/module/cpyext/test/multiphase2.c
@@ -417,7 +417,7 @@
     PyDoc_STR("Not a PyModuleObject object, but requests per-module state"),
     10,                                         /* m_size */
     NULL,                                       /* m_methods */
-    //slots_create_nonmodule,                     /* m_slots */
+    slots_create_nonmodule,                     /* m_slots */
     NULL,                                       /* m_traverse */
     NULL,                                       /* m_clear */
     NULL,                                       /* m_free */
@@ -436,7 +436,7 @@
     PyDoc_STR("PyModuleDef with negative m_size"),
     -1,                                         /* m_size */
     NULL,                                       /* m_methods */
-    NULL, //slots_create_nonmodule,                     /* m_slots */
+    slots_create_nonmodule,                     /* m_slots */
     NULL,                                       /* m_traverse */
     NULL,                                       /* m_clear */
     NULL,                                       /* m_free */
diff --git a/pypy/module/cpyext/test/test_module.py b/pypy/module/cpyext/test/test_module.py
--- a/pypy/module/cpyext/test/test_module.py
+++ b/pypy/module/cpyext/test/test_module.py
@@ -175,3 +175,38 @@
         ex_class = module.Example
         importlib.reload(module)
         assert ex_class is module.Example
+
+    def w_load_from_spec(self, loader, spec):
+        from importlib import util
+        module = util.module_from_spec(spec)
+        loader.exec_module(module)
+        return module
+
+    def test_bad_modules(self):
+        # XXX: not a very good test, since most internal issues in cpyext
+        # cause SystemErrors.
+        from importlib import machinery, util
+        NAME = 'multiphase2'
+        module = self.import_module(name=NAME)
+        origin = module.__loader__.path
+        for name_base in [
+                'bad_slot_large',
+                'bad_slot_negative',
+                'create_int_with_state',
+                'negative_size',
+                'export_null',
+                'export_uninitialized',
+                'export_raise',
+                'export_unreported_exception',
+                'create_null',
+                'create_raise',
+                'create_unreported_exception',
+                'nonmodule_with_exec_slots',
+                'exec_err',
+                'exec_raise',
+                'exec_unreported_exception',
+                ]:
+            name = '_testmultiphase_' + name_base
+            loader = machinery.ExtensionFileLoader(name, origin)
+            spec = util.spec_from_loader(name, loader)
+            raises(SystemError, self.load_from_spec, loader, spec)

From pypy.commits at gmail.com  Wed Aug  9 06:27:48 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 09 Aug 2017 03:27:48 -0700 (PDT)
Subject: [pypy-commit] pypy default: test,
 fix calling unbound object method with no args like
 np.int32.__array__()
Message-ID: <598ae3a4.6596df0a.b8ed8.f40a@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92114:dcea72970920
Date: 2017-08-09 13:26 +0300
http://bitbucket.org/pypy/pypy/changeset/dcea72970920/

Log:	test, fix calling unbound object method with no args like
	np.int32.__array__()

diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -210,6 +210,10 @@
 def cmethod_descr_call(space, w_self, __args__):
     self = space.interp_w(W_PyCFunctionObject, w_self)
     args_w, kw_w = __args__.unpack()
+    if len(args_w) < 1:
+        raise oefmt(space.w_TypeError,
+            "descriptor '%s' of '%s' object needs an argument",
+            self.name, self.w_objclass.getname(space))
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -21,6 +21,9 @@
         assert arr.itemsize == 4
         assert arr[2] == 3
         assert len(arr.buffer_info()) == 2
+        exc = raises(TypeError, module.array.append)
+        errstr = str(exc.value)
+        assert errstr.startswith("descriptor 'append' of")
         arr.append(4)
         assert arr.tolist() == [1, 2, 3, 4]
         assert len(arr) == 4

From pypy.commits at gmail.com  Wed Aug  9 07:20:55 2017
From: pypy.commits at gmail.com (fijal)
Date: Wed, 09 Aug 2017 04:20:55 -0700 (PDT)
Subject: [pypy-commit] pypy default: warn only if mercurial explodes
Message-ID: <598af017.6b99df0a.e9f8d.b3b3@mx.google.com>

Author: fijal
Branch: 
Changeset: r92115:1a9ef5428d4e
Date: 2017-08-09 13:19 +0200
http://bitbucket.org/pypy/pypy/changeset/1a9ef5428d4e/

Log:	warn only if mercurial explodes

diff --git a/rpython/tool/version.py b/rpython/tool/version.py
--- a/rpython/tool/version.py
+++ b/rpython/tool/version.py
@@ -59,8 +59,8 @@
     p = Popen([str(hgexe), 'id', '-i', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgid = p.stdout.read().strip()
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgid = '?'
 
     p = Popen([str(hgexe), 'id', '-t', root],

From pypy.commits at gmail.com  Wed Aug  9 07:20:57 2017
From: pypy.commits at gmail.com (fijal)
Date: Wed, 09 Aug 2017 04:20:57 -0700 (PDT)
Subject: [pypy-commit] pypy default: merge
Message-ID: <598af019.e3afdf0a.fc2eb.208b@mx.google.com>

Author: fijal
Branch: 
Changeset: r92116:e3868e494742
Date: 2017-08-09 13:20 +0200
http://bitbucket.org/pypy/pypy/changeset/e3868e494742/

Log:	merge

diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -62,7 +62,7 @@
 throwing away information about them less eagerly.
 
 
-.. branch: getarrayitem-into-bridges:
+.. branch: getarrayitem-into-bridges
 
 More information is retained into a bridge: knowledge about the content of
 arrays (at fixed indices) is stored in guards (and thus available at the
diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -210,6 +210,10 @@
 def cmethod_descr_call(space, w_self, __args__):
     self = space.interp_w(W_PyCFunctionObject, w_self)
     args_w, kw_w = __args__.unpack()
+    if len(args_w) < 1:
+        raise oefmt(space.w_TypeError,
+            "descriptor '%s' of '%s' object needs an argument",
+            self.name, self.w_objclass.getname(space))
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -21,6 +21,9 @@
         assert arr.itemsize == 4
         assert arr[2] == 3
         assert len(arr.buffer_info()) == 2
+        exc = raises(TypeError, module.array.append)
+        errstr = str(exc.value)
+        assert errstr.startswith("descriptor 'append' of")
         arr.append(4)
         assert arr.tolist() == [1, 2, 3, 4]
         assert len(arr) == 4

From pypy.commits at gmail.com  Wed Aug  9 07:25:11 2017
From: pypy.commits at gmail.com (fijal)
Date: Wed, 09 Aug 2017 04:25:11 -0700 (PDT)
Subject: [pypy-commit] pypy default: more of the same
Message-ID: <598af117.0591df0a.45edf.1176@mx.google.com>

Author: fijal
Branch: 
Changeset: r92117:85d7ced5c2a8
Date: 2017-08-09 13:24 +0200
http://bitbucket.org/pypy/pypy/changeset/85d7ced5c2a8/

Log:	more of the same

diff --git a/rpython/tool/version.py b/rpython/tool/version.py
--- a/rpython/tool/version.py
+++ b/rpython/tool/version.py
@@ -66,8 +66,8 @@
     p = Popen([str(hgexe), 'id', '-t', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgtags = [t for t in p.stdout.read().strip().split() if t != 'tip']
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgtags = ['?']
 
     if hgtags:
@@ -77,7 +77,8 @@
         p = Popen([str(hgexe), 'id', '-b', root],
                   stdout=PIPE, stderr=PIPE, env=env)
         hgbranch = p.stdout.read().strip()
-        maywarn(p.stderr.read())
+        if p.wait() != 0:
+            maywarn(p.stderr.read())
 
         return hgbranch, hgid
 

From pypy.commits at gmail.com  Wed Aug  9 11:27:24 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 09 Aug 2017 08:27:24 -0700 (PDT)
Subject: [pypy-commit] pypy default: test,
 fix for cpython compatibility when PyObject_RichCompareBool(a, a, ...)
Message-ID: <598b29dc.09de1c0a.33d61.5cf2@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92118:d6b37b7c15ee
Date: 2017-08-09 18:25 +0300
http://bitbucket.org/pypy/pypy/changeset/d6b37b7c15ee/

Log:	test, fix for cpython compatibility when PyObject_RichCompareBool(a,
	a, ...)

diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -305,7 +305,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid):
+def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -313,7 +313,15 @@
     0 if the result is false, 1 otherwise. This is the equivalent of the
     Python expression o1 op o2, where op is the operator corresponding to
     opid."""
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid)
+    # Quick result when objects are the same.
+    # Guarantees that identity implies equality.
+    if ref1 is ref2:
+        opid = rffi.cast(lltype.Signed, opid_int)
+        if opid == Py_EQ:
+            return 1
+        if opid == Py_NE:
+            return 0 
+    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -8,7 +8,7 @@
 from pypy.module.cpyext.object import (
     PyObject_IsTrue, PyObject_Not, PyObject_GetAttrString,
     PyObject_DelAttrString, PyObject_GetAttr, PyObject_DelAttr,
-    PyObject_GetItem, PyObject_RichCompareBool,
+    PyObject_GetItem, 
     PyObject_IsInstance, PyObject_IsSubclass, PyObject_AsFileDescriptor,
     PyObject_Hash, PyObject_Cmp, PyObject_Unicode
 )
@@ -136,7 +136,18 @@
 
         w_i = space.wrap(1)
         with raises_w(space, SystemError):
-            PyObject_RichCompareBool(space, w_i, w_i, 123456)
+            api.PyObject_RichCompareBool(w_i, w_i, 123456)
+
+    def test_RichCompareNanlike(self, space,api):
+        w_obj = space.appexec([], """():
+            class Nanlike(object):
+                def __eq__(self, other):
+                    raise RuntimeError('unreachable')
+            return Nanlike()""")
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_EQ)
+        assert res == 1
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_NE)
+        assert res == 0
 
     def test_IsInstance(self, space, api):
         assert api.PyObject_IsInstance(space.wrap(1), space.w_int) == 1

From pypy.commits at gmail.com  Thu Aug 10 16:37:16 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 10 Aug 2017 13:37:16 -0700 (PDT)
Subject: [pypy-commit] pypy default: char * -> const char *,
 fixes issue #2626
Message-ID: <598cc3fc.e484df0a.a3223.7951@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92120:4b21dcb59a60
Date: 2017-08-10 23:36 +0300
http://bitbucket.org/pypy/pypy/changeset/4b21dcb59a60/

Log:	char * -> const char *, fixes issue #2626

diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py
--- a/pypy/module/cpyext/unicodeobject.py
+++ b/pypy/module/cpyext/unicodeobject.py
@@ -520,7 +520,7 @@
 if sys.platform == 'win32':
     make_conversion_functions('MBCS', 'mbcs')
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF16(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-16 encoded buffer string and return the
     corresponding Unicode object.  errors (if non-NULL) defines the error
@@ -574,7 +574,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, rffi.CCHARP, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF32(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-32 encoded buffer string and
     return the corresponding Unicode object.  errors (if non-NULL)
@@ -630,7 +630,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, rffi.CCHARP],
+ at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, CONST_STRING],
              rffi.INT_real, error=-1)
 def PyUnicode_EncodeDecimal(space, s, length, output, llerrors):
     """Takes a Unicode string holding a decimal value and writes it

From pypy.commits at gmail.com  Thu Aug 10 16:37:14 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 10 Aug 2017 13:37:14 -0700 (PDT)
Subject: [pypy-commit] pypy default: quiet a gcc warning by adding
 RPY_UNUSED __attribute__ ((__unused__))
Message-ID: <598cc3fa.5ba4df0a.eb3e2.922a@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92119:ad807f62258a
Date: 2017-08-10 23:35 +0300
http://bitbucket.org/pypy/pypy/changeset/ad807f62258a/

Log:	quiet a gcc warning by adding RPY_UNUSED __attribute__
	((__unused__))

diff --git a/rpython/translator/c/src/precommondefs.h b/rpython/translator/c/src/precommondefs.h
--- a/rpython/translator/c/src/precommondefs.h
+++ b/rpython/translator/c/src/precommondefs.h
@@ -63,9 +63,11 @@
 #ifdef __GNUC__
 #  define RPY_EXPORTED extern __attribute__((visibility("default")))
 #  define _RPY_HIDDEN  __attribute__((visibility("hidden")))
+#  define RPY_UNUSED __attribute__ ((__unused__))
 #else
 #  define RPY_EXPORTED extern __declspec(dllexport)
 #  define _RPY_HIDDEN  /* nothing */
+#  define RPY_UNUSED   /*nothing */
 #endif
 #ifndef RPY_EXTERN
 #  define RPY_EXTERN   extern _RPY_HIDDEN
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -102,9 +102,10 @@
 static void write_str(int fd, const char *p)
 {
     int i = 0;
+    int res RPY_UNUSED;
     while (p[i] != '\x00')
         i++;
-    (void)write(fd, p, i);
+    res = write(fd, p, i);
 }
 
 static void signal_setflag_handler(int signum)

From pypy.commits at gmail.com  Fri Aug 11 06:14:27 2017
From: pypy.commits at gmail.com (mattip)
Date: Fri, 11 Aug 2017 03:14:27 -0700 (PDT)
Subject: [pypy-commit] pypy default: do not change preexisting error when
 calling c-api functions
Message-ID: <598d8383.d48bdf0a.a1a69.3aac@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92121:9ddefd44f80d
Date: 2017-08-11 12:37 +0300
http://bitbucket.org/pypy/pypy/changeset/9ddefd44f80d/

Log:	do not change preexisting error when calling c-api functions

	unsuccessfully tried to write a test, since testing does not go
	through this path

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1575,6 +1575,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
+        preexist_error = PyErr_Occurred(space) is not None
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1598,15 +1599,15 @@
             # Check for exception consistency
             has_error = PyErr_Occurred(space) is not None
             has_result = ret is not None
-            if has_error and has_result:
-                raise oefmt(space.w_SystemError,
-                            "An exception was set, but function returned a "
-                            "value")
-            elif not expect_null and not has_error and not has_result:
-                raise oefmt(space.w_SystemError,
-                            "Function returned a NULL result without setting "
-                            "an exception")
-
+            if not preexist_error:
+                if has_error and has_result:
+                    raise oefmt(space.w_SystemError,
+                                "An exception was set, but function returned a "
+                                "value")
+                elif not expect_null and not has_error and not has_result:
+                    raise oefmt(space.w_SystemError,
+                                "Function returned a NULL result without setting "
+                                "an exception")
             if has_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()

From pypy.commits at gmail.com  Fri Aug 11 08:48:50 2017
From: pypy.commits at gmail.com (mattip)
Date: Fri, 11 Aug 2017 05:48:50 -0700 (PDT)
Subject: [pypy-commit] pypy default: document 'default' argument to
 sys.getsizeof
Message-ID: <598da7b2.cf371c0a.5da21.61be@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92122:37e8eeb5775a
Date: 2017-08-11 15:47 +0300
http://bitbucket.org/pypy/pypy/changeset/37e8eeb5775a/

Log:	document 'default' argument to sys.getsizeof

diff --git a/pypy/module/sys/vm.py b/pypy/module/sys/vm.py
--- a/pypy/module/sys/vm.py
+++ b/pypy/module/sys/vm.py
@@ -253,7 +253,13 @@
     from rpython.rtyper.lltypesystem import lltype, rffi
     return space.newint(rffi.cast(lltype.Signed, handle))
 
-getsizeof_missing = """sys.getsizeof() is not implemented on PyPy.
+getsizeof_missing = """getsizeof(...)
+    getsizeof(object, default) -> int
+    
+    Return the size of object in bytes.
+
+sys.getsizeof(object, default) will always return default on PyPy, and
+raise a TypeError if default is not provided.
 
 First note that the CPython documentation says that this function may
 raise a TypeError, so if you are seeing it, it means that the program

From pypy.commits at gmail.com  Fri Aug 11 11:30:30 2017
From: pypy.commits at gmail.com (fijal)
Date: Fri, 11 Aug 2017 08:30:30 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: work on paragraphs
Message-ID: <598dcd96.8ba61c0a.abdb0.8bc1@mx.google.com>

Author: fijal
Branch: extradoc
Changeset: r5826:3cd0a7835f19
Date: 2017-08-11 17:30 +0200
http://bitbucket.org/pypy/extradoc/changeset/3cd0a7835f19/

Log:	work on paragraphs

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -17,16 +17,14 @@
 places so PyPy does not segfault when you try to do a concurrent access to a
 data structure.
 
-.. antocuni: I'd simply remove the following paragraph. It's redundant, IMHO
-..
-   Since such work would complicate the code base and our day to day work,
-   we would like to judge the interest on the community and the commercial
-   PyPy users.
-
-We are looking for commercial partners to make it happen (individual donations
+Since such work would complicate the code base and our day to day work,
+we would like to judge the interest on the community and the commercial
+ partners to make it happen (individual donations
 did not work very well for us in the past). We estimate a total cost of $50k,
-out of which we already have backing for about 1/3. If we can get a $100k
-contract, we would make it our priority to deliver before the end of the year.
+out of which we already have backing for about 1/3. This would give us a good
+shot at delivering a good proof of concept of working PyPy no-GIL. If we can get a $100k
+contract, we will deliver fully working PyPy interpreter with no GIL as a release,
+possibly separate from the default PyPy release.
 
 People asked several questions, so I'll try to answer the technical parts
 here.
@@ -54,9 +52,20 @@
 Python is a very mutable language - there is tons of mutable state and
 basic objects that are compile time in other languages, like classes and functions
 are mutable at runtime. That means that sharing things between subinterpreters would
-be restricted to basic immutable data structures, which defeats the point compared
-to multi-processing approach. We don't believe it's a viable approach without
+be restricted to basic immutable data structures, which defeats the point,
+because it has the same problems as the approach of having multiple processes and
+no additional benefits.
+We don't believe it's a viable approach without
 seriously impacting the semantics of the language.
 
+* Why is it easier to do in PyPy than CPython?
+
+Removing the GIL in CPython has two problems - how do we guard access to mutable
+data structures with locks and what do we do with reference counting that needs
+to be guarded. PyPy only has the former problem, with latter being non-existance
+due to a different garbage collector approach. We believe that fixing reference
+counting would be quite a bit bigger undertaking than "just" providing locks around
+mutable data structures.
+
 Best regards,
 Maciej Fijalkowski

From pypy.commits at gmail.com  Fri Aug 11 11:35:15 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 11 Aug 2017 08:35:15 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: Tweaks
Message-ID: <598dceb3.85961c0a.a6b32.8baa@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: extradoc
Changeset: r5827:9e0ef6516b6f
Date: 2017-08-11 17:35 +0200
http://bitbucket.org/pypy/extradoc/changeset/9e0ef6516b6f/

Log:	Tweaks

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -6,12 +6,12 @@
 Discussions about the infamous Global Interpreter Lock have been around for a while
 in the Python community. There has been various attempts at removing it:
 some were successful, like e.g. in Jython or IronPython with the help of the platform, and some yet to bear fruit, like `gilectomy`_. Since our `February sprint`_ in Leysin,
-we've been on-and-off tackling the topic of GIL removal in the PyPy project.
+we've been on-and-off tackling directly the topic of GIL removal in the PyPy project.
 
 .. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html
 
 As we announced at EuroPython, what we have got so far is a GIL-less PyPy
-which can to run **very simple** multi-threaded programs which are nicely
+which can run **very simple** multi-threaded programs which are nicely
 parallelized.  At the moment, non-simple programs most likely segfaults: the
 remaining 90% (and another 90%) of work is with putting locks in strategic
 places so PyPy does not segfault when you try to do a concurrent access to a
@@ -62,10 +62,9 @@
 
 Removing the GIL in CPython has two problems - how do we guard access to mutable
 data structures with locks and what do we do with reference counting that needs
-to be guarded. PyPy only has the former problem, with latter being non-existance
-due to a different garbage collector approach. We believe that fixing reference
-counting would be quite a bit bigger undertaking than "just" providing locks around
-mutable data structures.
+to be guarded. PyPy only has the former problem; the latter doesn't exist,
+due to a different garbage collector approach.  Of course the first problem
+is a mess too, but at least we are already half-way there.
 
 Best regards,
 Maciej Fijalkowski

From pypy.commits at gmail.com  Fri Aug 11 11:42:42 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 11 Aug 2017 08:42:42 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <598dd072.15b81c0a.9bd01.8040@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92123:ca42f6a90f18
Date: 2017-08-11 17:36 +0200
http://bitbucket.org/pypy/pypy/changeset/ca42f6a90f18/

Log:	hg merge default

diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -62,7 +62,7 @@
 throwing away information about them less eagerly.
 
 
-.. branch: getarrayitem-into-bridges:
+.. branch: getarrayitem-into-bridges
 
 More information is retained into a bridge: knowledge about the content of
 arrays (at fixed indices) is stored in guards (and thus available at the
diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1632,6 +1632,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
+        preexist_error = PyErr_Occurred(space) is not None
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1656,15 +1657,15 @@
 
             # Check for exception consistency
             has_error = PyErr_Occurred(space) is not None
-            if has_error and has_result:
-                raise oefmt(space.w_SystemError,
-                            "An exception was set, but function returned a "
-                            "value")
-            elif not expect_null and not has_error and not has_result:
-                raise oefmt(space.w_SystemError,
-                            "Function returned a NULL result without setting "
-                            "an exception")
-
+            if not preexist_error:
+                if has_error and has_result:
+                    raise oefmt(space.w_SystemError,
+                                "An exception was set, but function returned a "
+                                "value")
+                elif not expect_null and not has_error and not has_result:
+                    raise oefmt(space.w_SystemError,
+                                "Function returned a NULL result without setting "
+                                "an exception")
             if has_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()
diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -203,6 +203,10 @@
 def cmethod_descr_call(space, w_self, __args__):
     self = space.interp_w(W_PyCFunctionObject, w_self)
     args_w, kw_w = __args__.unpack()
+    if len(args_w) < 1:
+        raise oefmt(space.w_TypeError,
+            "descriptor '%s' of '%s' object needs an argument",
+            self.name, self.w_objclass.getname(space))
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -311,7 +311,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid):
+def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -319,7 +319,15 @@
     0 if the result is false, 1 otherwise. This is the equivalent of the
     Python expression o1 op o2, where op is the operator corresponding to
     opid."""
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid)
+    # Quick result when objects are the same.
+    # Guarantees that identity implies equality.
+    if ref1 is ref2:
+        opid = rffi.cast(lltype.Signed, opid_int)
+        if opid == Py_EQ:
+            return 1
+        if opid == Py_NE:
+            return 0 
+    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -12,6 +12,9 @@
         assert arr.itemsize == 4
         assert arr[2] == 3
         assert len(arr.buffer_info()) == 2
+        exc = raises(TypeError, module.array.append)
+        errstr = str(exc.value)
+        assert errstr.startswith("descriptor 'append' of")
         arr.append(4)
         assert arr.tolist() == [1, 2, 3, 4]
         assert len(arr) == 4
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -8,7 +8,7 @@
 from pypy.module.cpyext.object import (
     PyObject_IsTrue, PyObject_Not, PyObject_GetAttrString,
     PyObject_DelAttrString, PyObject_GetAttr, PyObject_DelAttr,
-    PyObject_GetItem, PyObject_RichCompareBool,
+    PyObject_GetItem, 
     PyObject_IsInstance, PyObject_IsSubclass, PyObject_AsFileDescriptor,
     PyObject_Hash)
 
@@ -135,7 +135,18 @@
 
         w_i = space.wrap(1)
         with raises_w(space, SystemError):
-            PyObject_RichCompareBool(space, w_i, w_i, 123456)
+            api.PyObject_RichCompareBool(w_i, w_i, 123456)
+
+    def test_RichCompareNanlike(self, space,api):
+        w_obj = space.appexec([], """():
+            class Nanlike(object):
+                def __eq__(self, other):
+                    raise RuntimeError('unreachable')
+            return Nanlike()""")
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_EQ)
+        assert res == 1
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_NE)
+        assert res == 0
 
     def test_IsInstance(self, space, api):
         assert api.PyObject_IsInstance(space.wrap(1), space.w_int) == 1
diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py
--- a/pypy/module/cpyext/unicodeobject.py
+++ b/pypy/module/cpyext/unicodeobject.py
@@ -730,7 +730,7 @@
 if sys.platform == 'win32':
     make_conversion_functions('MBCS', 'mbcs')
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF16(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-16 encoded buffer string and return the
     corresponding Unicode object.  errors (if non-NULL) defines the error
@@ -784,7 +784,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, rffi.CCHARP, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF32(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-32 encoded buffer string and
     return the corresponding Unicode object.  errors (if non-NULL)
@@ -840,7 +840,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, rffi.CCHARP],
+ at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, CONST_STRING],
              rffi.INT_real, error=-1)
 def PyUnicode_EncodeDecimal(space, s, length, output, llerrors):
     """Takes a Unicode string holding a decimal value and writes it
diff --git a/pypy/module/sys/vm.py b/pypy/module/sys/vm.py
--- a/pypy/module/sys/vm.py
+++ b/pypy/module/sys/vm.py
@@ -279,7 +279,13 @@
     from rpython.rtyper.lltypesystem import lltype, rffi
     return space.newint(rffi.cast(lltype.Signed, handle))
 
-getsizeof_missing = """sys.getsizeof() is not implemented on PyPy.
+getsizeof_missing = """getsizeof(...)
+    getsizeof(object, default) -> int
+    
+    Return the size of object in bytes.
+
+sys.getsizeof(object, default) will always return default on PyPy, and
+raise a TypeError if default is not provided.
 
 First note that the CPython documentation says that this function may
 raise a TypeError, so if you are seeing it, it means that the program
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -556,3 +556,13 @@
 
         x = X()
         assert x.x == 0
+
+    def test_duplicate_names(self):
+        class S(Structure):
+            _fields_ = [('a', c_int),
+                        ('b', c_int),
+                        ('a', c_byte)]
+        s = S(260, -123)
+        assert sizeof(s) == 3 * sizeof(c_int)
+        assert s.a == 4     # 256 + 4
+        assert s.b == -123
diff --git a/rpython/annotator/binaryop.py b/rpython/annotator/binaryop.py
--- a/rpython/annotator/binaryop.py
+++ b/rpython/annotator/binaryop.py
@@ -529,6 +529,8 @@
         assert dic1.__class__ == dic2.__class__
         return dic1.__class__(dic1.dictdef.union(dic2.dictdef))
 
+    def ne((dic1, dic2)):
+        raise AnnotatorError("dict != dict not implemented")
 
 def _dict_can_only_throw_keyerror(s_dct, *ignore):
     if s_dct.dictdef.dictkey.custom_eq_hash:
diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -84,7 +84,6 @@
 
     # heap knowledge: we store triples of known heap fields in non-virtual
     # structs
-    # XXX could be extended to arrays
     if optimizer.optheap:
         triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
diff --git a/rpython/tool/version.py b/rpython/tool/version.py
--- a/rpython/tool/version.py
+++ b/rpython/tool/version.py
@@ -59,15 +59,15 @@
     p = Popen([str(hgexe), 'id', '-i', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgid = p.stdout.read().strip()
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgid = '?'
 
     p = Popen([str(hgexe), 'id', '-t', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgtags = [t for t in p.stdout.read().strip().split() if t != 'tip']
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgtags = ['?']
 
     if hgtags:
@@ -77,7 +77,8 @@
         p = Popen([str(hgexe), 'id', '-b', root],
                   stdout=PIPE, stderr=PIPE, env=env)
         hgbranch = p.stdout.read().strip()
-        maywarn(p.stderr.read())
+        if p.wait() != 0:
+            maywarn(p.stderr.read())
 
         return hgbranch, hgid
 
diff --git a/rpython/translator/c/src/precommondefs.h b/rpython/translator/c/src/precommondefs.h
--- a/rpython/translator/c/src/precommondefs.h
+++ b/rpython/translator/c/src/precommondefs.h
@@ -63,9 +63,11 @@
 #ifdef __GNUC__
 #  define RPY_EXPORTED extern __attribute__((visibility("default")))
 #  define _RPY_HIDDEN  __attribute__((visibility("hidden")))
+#  define RPY_UNUSED __attribute__ ((__unused__))
 #else
 #  define RPY_EXPORTED extern __declspec(dllexport)
 #  define _RPY_HIDDEN  /* nothing */
+#  define RPY_UNUSED   /*nothing */
 #endif
 #ifndef RPY_EXTERN
 #  define RPY_EXTERN   extern _RPY_HIDDEN
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -102,9 +102,10 @@
 static void write_str(int fd, const char *p)
 {
     int i = 0;
+    int res RPY_UNUSED;
     while (p[i] != '\x00')
         i++;
-    (void)write(fd, p, i);
+    res = write(fd, p, i);
 }
 
 static void signal_setflag_handler(int signum)

From pypy.commits at gmail.com  Fri Aug 11 11:44:39 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 11 Aug 2017 08:44:39 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Check the result of module init
 functions more carefully; improve tests for this
Message-ID: <598dd0e7.85961c0a.a6b32.8cff@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92124:fe04b833579b
Date: 2017-08-11 17:41 +0200
http://bitbucket.org/pypy/pypy/changeset/fe04b833579b/

Log:	Check the result of module init functions more carefully; improve
	tests for this

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1545,7 +1545,17 @@
     try:
         initfunc = rffi.cast(initfunctype, initptr)
         initret = generic_cpy_call_dont_convert_result(space, initfunc)
-        state.check_and_raise_exception()
+        if not initret:
+            if state.operror:
+                state.check_and_raise_exception()
+            raise oefmt(space.w_SystemError,
+                "initialization of %s failed without raising an exception",
+                name)
+        else:
+            if state.clear_exception():
+                raise oefmt(space.w_SystemError,
+                    "initialization of %s raised unreported exception",
+                    name)
         if not initret.c_ob_type:
             raise oefmt(space.w_SystemError,
                         "init function of %s returned uninitialized object",
@@ -1588,7 +1598,7 @@
 @specialize.ll()
 def generic_cpy_call_dont_convert_result(space, func, *args):
     FT = lltype.typeOf(func).TO
-    return make_generic_cpy_call(FT, False, False)(space, func, *args)
+    return make_generic_cpy_call(FT, True, False)(space, func, *args)
 
 @specialize.memo()
 def make_generic_cpy_call(FT, expect_null, convert_result):
diff --git a/pypy/module/cpyext/test/test_module.py b/pypy/module/cpyext/test/test_module.py
--- a/pypy/module/cpyext/test/test_module.py
+++ b/pypy/module/cpyext/test/test_module.py
@@ -135,25 +135,28 @@
         raises(SystemError, self.import_module, name='multiphase', body=body,
                init=init)
 
+class AppTestMultiPhase2(AppTestCpythonExtensionBase):
+    def setup_class(cls):
+        cls.w_name = cls.space.wrap('multiphase2')
+        AppTestCpythonExtensionBase.setup_class.im_func(cls)
+
     def test_multiphase2(self):
         import sys
         from importlib import machinery, util
-        NAME = 'multiphase2'
-        module = self.import_module(name=NAME)
+        module = self.import_module(name=self.name)
         finder = machinery.FileFinder(None)
-        spec = util.find_spec(NAME)
+        spec = util.find_spec(self.name)
         assert spec
-        assert module.__name__ == NAME
+        assert module.__name__ == self.name
         #assert module.__file__ == spec.origin
         assert module.__package__ == ''
         raises(AttributeError, 'module.__path__')
-        assert module is sys.modules[NAME]
+        assert module is sys.modules[self.name]
         assert isinstance(module.__loader__, machinery.ExtensionFileLoader)
 
     def test_functionality(self):
         import types
-        NAME = 'multiphase2'
-        module = self.import_module(name=NAME)
+        module = self.import_module(name=self.name)
         assert isinstance(module, types.ModuleType)
         ex = module.Example()
         assert ex.demo('abcd') == 'abcd'
@@ -170,14 +173,19 @@
 
     def test_reload(self):
         import importlib
-        NAME = 'multiphase2'
-        module = self.import_module(name=NAME)
+        module = self.import_module(name=self.name)
         ex_class = module.Example
         importlib.reload(module)
         assert ex_class is module.Example
 
-    def w_load_from_spec(self, loader, spec):
-        from importlib import util
+    def w_load_from_name(self, name, origin=None):
+        from importlib import machinery, util
+        if not origin:
+            module = self.import_module(name=self.name)
+            origin = module.__loader__.path
+        name = '_testmultiphase_' + name
+        loader = machinery.ExtensionFileLoader(name, origin)
+        spec = util.spec_from_loader(name, loader)
         module = util.module_from_spec(spec)
         loader.exec_module(module)
         return module
@@ -185,19 +193,13 @@
     def test_bad_modules(self):
         # XXX: not a very good test, since most internal issues in cpyext
         # cause SystemErrors.
-        from importlib import machinery, util
-        NAME = 'multiphase2'
-        module = self.import_module(name=NAME)
+        module = self.import_module(name=self.name)
         origin = module.__loader__.path
-        for name_base in [
+        for name in [
                 'bad_slot_large',
                 'bad_slot_negative',
                 'create_int_with_state',
                 'negative_size',
-                'export_null',
-                'export_uninitialized',
-                'export_raise',
-                'export_unreported_exception',
                 'create_null',
                 'create_raise',
                 'create_unreported_exception',
@@ -206,7 +208,23 @@
                 'exec_raise',
                 'exec_unreported_exception',
                 ]:
-            name = '_testmultiphase_' + name_base
-            loader = machinery.ExtensionFileLoader(name, origin)
-            spec = util.spec_from_loader(name, loader)
-            raises(SystemError, self.load_from_spec, loader, spec)
+            raises(SystemError, self.load_from_name, name, origin)
+
+    def test_export_null(self):
+        excinfo = raises(SystemError, self.load_from_name, 'export_null')
+        assert "initialization" in excinfo.value.args[0]
+        assert "without raising" in excinfo.value.args[0]
+
+    def test_export_uninit(self):
+        excinfo = raises(SystemError, self.load_from_name, 'export_uninitialized')
+        assert "init function" in excinfo.value.args[0]
+        assert "uninitialized object" in excinfo.value.args[0]
+
+    def test_export_raise(self):
+        excinfo = raises(SystemError, self.load_from_name, 'export_raise')
+        assert "bad export function" == excinfo.value.args[0]
+
+    def test_export_unreported(self):
+        excinfo = raises(SystemError, self.load_from_name, 'export_unreported_exception')
+        assert "initialization" in excinfo.value.args[0]
+        assert "unreported exception" in excinfo.value.args[0]

From pypy.commits at gmail.com  Fri Aug 11 12:02:58 2017
From: pypy.commits at gmail.com (fijal)
Date: Fri, 11 Aug 2017 09:02:58 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: expand
Message-ID: <598dd532.4588df0a.dd3fd.996d@mx.google.com>

Author: fijal
Branch: extradoc
Changeset: r5828:a37e8e51e1bc
Date: 2017-08-11 18:02 +0200
http://bitbucket.org/pypy/extradoc/changeset/a37e8e51e1bc/

Log:	expand

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -7,6 +7,10 @@
 in the Python community. There has been various attempts at removing it:
 some were successful, like e.g. in Jython or IronPython with the help of the platform, and some yet to bear fruit, like `gilectomy`_. Since our `February sprint`_ in Leysin,
 we've been on-and-off tackling directly the topic of GIL removal in the PyPy project.
+We believe that the work done in IronPython or Jython can be reproduced with
+only a bit more effort. Compared to that, removing the GIL in CPython is a much
+harder topic, since it requires tackling the problem of multi-threaded reference
+counting. See the section below for further discussions.
 
 .. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html
 
@@ -64,7 +68,11 @@
 data structures with locks and what do we do with reference counting that needs
 to be guarded. PyPy only has the former problem; the latter doesn't exist,
 due to a different garbage collector approach.  Of course the first problem
-is a mess too, but at least we are already half-way there.
+is a mess too, but at least we are already half-way there. Compare to Jython
+or IronPython, PyPy lacks some data structures that are provided by JVM or .NET,
+which we would need to implement, hence the problem is a little harder
+than on an existing multithreaded platform. However, there is good research
+and we know how the problem can be solved.
 
 Best regards,
 Maciej Fijalkowski

From pypy.commits at gmail.com  Fri Aug 11 12:06:47 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 11 Aug 2017 09:06:47 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <598dd617.935c1c0a.98700.bc12@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92125:dce989c3370d
Date: 2017-08-11 17:47 +0200
http://bitbucket.org/pypy/pypy/changeset/dce989c3370d/

Log:	hg merge py3.5

diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -62,7 +62,7 @@
 throwing away information about them less eagerly.
 
 
-.. branch: getarrayitem-into-bridges:
+.. branch: getarrayitem-into-bridges
 
 More information is retained into a bridge: knowledge about the content of
 arrays (at fixed indices) is stored in guards (and thus available at the
diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1645,6 +1645,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
+        preexist_error = PyErr_Occurred(space) is not None
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1669,15 +1670,15 @@
 
             # Check for exception consistency
             has_error = PyErr_Occurred(space) is not None
-            if has_error and has_result:
-                raise oefmt(space.w_SystemError,
-                            "An exception was set, but function returned a "
-                            "value")
-            elif not expect_null and not has_error and not has_result:
-                raise oefmt(space.w_SystemError,
-                            "Function returned a NULL result without setting "
-                            "an exception")
-
+            if not preexist_error:
+                if has_error and has_result:
+                    raise oefmt(space.w_SystemError,
+                                "An exception was set, but function returned a "
+                                "value")
+                elif not expect_null and not has_error and not has_result:
+                    raise oefmt(space.w_SystemError,
+                                "Function returned a NULL result without setting "
+                                "an exception")
             if has_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()
diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -203,6 +203,10 @@
 def cmethod_descr_call(space, w_self, __args__):
     self = space.interp_w(W_PyCFunctionObject, w_self)
     args_w, kw_w = __args__.unpack()
+    if len(args_w) < 1:
+        raise oefmt(space.w_TypeError,
+            "descriptor '%s' of '%s' object needs an argument",
+            self.name, self.w_objclass.getname(space))
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -311,7 +311,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid):
+def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -319,7 +319,15 @@
     0 if the result is false, 1 otherwise. This is the equivalent of the
     Python expression o1 op o2, where op is the operator corresponding to
     opid."""
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid)
+    # Quick result when objects are the same.
+    # Guarantees that identity implies equality.
+    if ref1 is ref2:
+        opid = rffi.cast(lltype.Signed, opid_int)
+        if opid == Py_EQ:
+            return 1
+        if opid == Py_NE:
+            return 0 
+    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/test/test_arraymodule.py b/pypy/module/cpyext/test/test_arraymodule.py
--- a/pypy/module/cpyext/test/test_arraymodule.py
+++ b/pypy/module/cpyext/test/test_arraymodule.py
@@ -12,6 +12,9 @@
         assert arr.itemsize == 4
         assert arr[2] == 3
         assert len(arr.buffer_info()) == 2
+        exc = raises(TypeError, module.array.append)
+        errstr = str(exc.value)
+        assert errstr.startswith("descriptor 'append' of")
         arr.append(4)
         assert arr.tolist() == [1, 2, 3, 4]
         assert len(arr) == 4
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -8,7 +8,7 @@
 from pypy.module.cpyext.object import (
     PyObject_IsTrue, PyObject_Not, PyObject_GetAttrString,
     PyObject_DelAttrString, PyObject_GetAttr, PyObject_DelAttr,
-    PyObject_GetItem, PyObject_RichCompareBool,
+    PyObject_GetItem, 
     PyObject_IsInstance, PyObject_IsSubclass, PyObject_AsFileDescriptor,
     PyObject_Hash)
 
@@ -135,7 +135,18 @@
 
         w_i = space.wrap(1)
         with raises_w(space, SystemError):
-            PyObject_RichCompareBool(space, w_i, w_i, 123456)
+            api.PyObject_RichCompareBool(w_i, w_i, 123456)
+
+    def test_RichCompareNanlike(self, space,api):
+        w_obj = space.appexec([], """():
+            class Nanlike(object):
+                def __eq__(self, other):
+                    raise RuntimeError('unreachable')
+            return Nanlike()""")
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_EQ)
+        assert res == 1
+        res = api.PyObject_RichCompareBool(w_obj, w_obj, Py_NE)
+        assert res == 0
 
     def test_IsInstance(self, space, api):
         assert api.PyObject_IsInstance(space.wrap(1), space.w_int) == 1
diff --git a/pypy/module/cpyext/unicodeobject.py b/pypy/module/cpyext/unicodeobject.py
--- a/pypy/module/cpyext/unicodeobject.py
+++ b/pypy/module/cpyext/unicodeobject.py
@@ -730,7 +730,7 @@
 if sys.platform == 'win32':
     make_conversion_functions('MBCS', 'mbcs')
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF16(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-16 encoded buffer string and return the
     corresponding Unicode object.  errors (if non-NULL) defines the error
@@ -784,7 +784,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CCHARP, Py_ssize_t, rffi.CCHARP, rffi.INTP], PyObject)
+ at cpython_api([CONST_STRING, Py_ssize_t, CONST_STRING, rffi.INTP], PyObject)
 def PyUnicode_DecodeUTF32(space, s, size, llerrors, pbyteorder):
     """Decode length bytes from a UTF-32 encoded buffer string and
     return the corresponding Unicode object.  errors (if non-NULL)
@@ -840,7 +840,7 @@
 
     return space.newunicode(result)
 
- at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, rffi.CCHARP],
+ at cpython_api([rffi.CWCHARP, Py_ssize_t, rffi.CCHARP, CONST_STRING],
              rffi.INT_real, error=-1)
 def PyUnicode_EncodeDecimal(space, s, length, output, llerrors):
     """Takes a Unicode string holding a decimal value and writes it
diff --git a/pypy/module/sys/vm.py b/pypy/module/sys/vm.py
--- a/pypy/module/sys/vm.py
+++ b/pypy/module/sys/vm.py
@@ -279,7 +279,13 @@
     from rpython.rtyper.lltypesystem import lltype, rffi
     return space.newint(rffi.cast(lltype.Signed, handle))
 
-getsizeof_missing = """sys.getsizeof() is not implemented on PyPy.
+getsizeof_missing = """getsizeof(...)
+    getsizeof(object, default) -> int
+    
+    Return the size of object in bytes.
+
+sys.getsizeof(object, default) will always return default on PyPy, and
+raise a TypeError if default is not provided.
 
 First note that the CPython documentation says that this function may
 raise a TypeError, so if you are seeing it, it means that the program
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -556,3 +556,13 @@
 
         x = X()
         assert x.x == 0
+
+    def test_duplicate_names(self):
+        class S(Structure):
+            _fields_ = [('a', c_int),
+                        ('b', c_int),
+                        ('a', c_byte)]
+        s = S(260, -123)
+        assert sizeof(s) == 3 * sizeof(c_int)
+        assert s.a == 4     # 256 + 4
+        assert s.b == -123
diff --git a/rpython/annotator/binaryop.py b/rpython/annotator/binaryop.py
--- a/rpython/annotator/binaryop.py
+++ b/rpython/annotator/binaryop.py
@@ -529,6 +529,8 @@
         assert dic1.__class__ == dic2.__class__
         return dic1.__class__(dic1.dictdef.union(dic2.dictdef))
 
+    def ne((dic1, dic2)):
+        raise AnnotatorError("dict != dict not implemented")
 
 def _dict_can_only_throw_keyerror(s_dct, *ignore):
     if s_dct.dictdef.dictkey.custom_eq_hash:
diff --git a/rpython/jit/metainterp/optimizeopt/bridgeopt.py b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
--- a/rpython/jit/metainterp/optimizeopt/bridgeopt.py
+++ b/rpython/jit/metainterp/optimizeopt/bridgeopt.py
@@ -84,7 +84,6 @@
 
     # heap knowledge: we store triples of known heap fields in non-virtual
     # structs
-    # XXX could be extended to arrays
     if optimizer.optheap:
         triples_struct, triples_array = optimizer.optheap.serialize_optheap(available_boxes)
         # can only encode descrs that have a known index into
diff --git a/rpython/tool/version.py b/rpython/tool/version.py
--- a/rpython/tool/version.py
+++ b/rpython/tool/version.py
@@ -59,15 +59,15 @@
     p = Popen([str(hgexe), 'id', '-i', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgid = p.stdout.read().strip()
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgid = '?'
 
     p = Popen([str(hgexe), 'id', '-t', root],
               stdout=PIPE, stderr=PIPE, env=env)
     hgtags = [t for t in p.stdout.read().strip().split() if t != 'tip']
-    maywarn(p.stderr.read())
     if p.wait() != 0:
+        maywarn(p.stderr.read())
         hgtags = ['?']
 
     if hgtags:
@@ -77,7 +77,8 @@
         p = Popen([str(hgexe), 'id', '-b', root],
                   stdout=PIPE, stderr=PIPE, env=env)
         hgbranch = p.stdout.read().strip()
-        maywarn(p.stderr.read())
+        if p.wait() != 0:
+            maywarn(p.stderr.read())
 
         return hgbranch, hgid
 
diff --git a/rpython/translator/c/src/precommondefs.h b/rpython/translator/c/src/precommondefs.h
--- a/rpython/translator/c/src/precommondefs.h
+++ b/rpython/translator/c/src/precommondefs.h
@@ -63,9 +63,11 @@
 #ifdef __GNUC__
 #  define RPY_EXPORTED extern __attribute__((visibility("default")))
 #  define _RPY_HIDDEN  __attribute__((visibility("hidden")))
+#  define RPY_UNUSED __attribute__ ((__unused__))
 #else
 #  define RPY_EXPORTED extern __declspec(dllexport)
 #  define _RPY_HIDDEN  /* nothing */
+#  define RPY_UNUSED   /*nothing */
 #endif
 #ifndef RPY_EXTERN
 #  define RPY_EXTERN   extern _RPY_HIDDEN
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -102,9 +102,10 @@
 static void write_str(int fd, const char *p)
 {
     int i = 0;
+    int res RPY_UNUSED;
     while (p[i] != '\x00')
         i++;
-    (void)write(fd, p, i);
+    res = write(fd, p, i);
 }
 
 static void signal_setflag_handler(int signum)

From pypy.commits at gmail.com  Fri Aug 11 12:15:25 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 11 Aug 2017 09:15:25 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: More tweaks
Message-ID: <598dd81d.8fb91c0a.caff.b193@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: extradoc
Changeset: r5829:20b8371b4615
Date: 2017-08-11 18:15 +0200
http://bitbucket.org/pypy/extradoc/changeset/20b8371b4615/

Log:	More tweaks

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -9,7 +9,7 @@
 we've been on-and-off tackling directly the topic of GIL removal in the PyPy project.
 We believe that the work done in IronPython or Jython can be reproduced with
 only a bit more effort. Compared to that, removing the GIL in CPython is a much
-harder topic, since it requires tackling the problem of multi-threaded reference
+harder topic, since it also requires tackling the problem of multi-threaded reference
 counting. See the section below for further discussions.
 
 .. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html
@@ -18,16 +18,17 @@
 which can run **very simple** multi-threaded programs which are nicely
 parallelized.  At the moment, non-simple programs most likely segfaults: the
 remaining 90% (and another 90%) of work is with putting locks in strategic
-places so PyPy does not segfault when you try to do a concurrent access to a
-data structure.
+places so PyPy does not segfault when you try to do concurrent accesses to
+data structures.
 
-Since such work would complicate the code base and our day to day work,
+Since such work would complicate the code base and our day-to-day work,
 we would like to judge the interest on the community and the commercial
- partners to make it happen (individual donations
-did not work very well for us in the past). We estimate a total cost of $50k,
-out of which we already have backing for about 1/3. This would give us a good
+partners to make it happen (we are not looking for individual
+donations at this point).  We estimate a total cost of $50k,
+out of which we already have backing for about 1/3 (with a possible 1/3
+extra from the STM money, see below).  This would give us a good
 shot at delivering a good proof of concept of working PyPy no-GIL. If we can get a $100k
-contract, we will deliver fully working PyPy interpreter with no GIL as a release,
+contract, we will deliver a fully working PyPy interpreter with no GIL as a release,
 possibly separate from the default PyPy release.
 
 People asked several questions, so I'll try to answer the technical parts
@@ -35,32 +36,36 @@
 
 * What would the plan entail?
 
-We've already done the work on Garbage Collector to allow doing multi
-threaded programs in RPython. "all" that's left is adding locks on mutable
-data structures everywhere in PyPy codebase. Since it'll significantly complicated
+We've already done the work on Garbage Collector to allow doing multi-
+threaded programs in RPython.  "All" that is left is adding locks on mutable
+data structures everywhere in the PyPy codebase. Since it'll significantly complicate
 our workflow, we need to see real interest in that topic, backed up by
 commercial contracts, otherwise we're not going to do it.
 
 * Why the STM effort did not work out?
 
 STM was a research project that proved that the idea is possible. However,
-the amount of user effort that's required to make programs run nicely
-parallelizable is both significant and we never managed to develop tools
-that would help nicely. At the present time we're not sure if more manpower
-spent on tooling would improve the situation or idea is doomed. The whole
-approach also ended up being a significant overhead on single threaded programs,
-which means that it's very easy to make your programs slower.
+the amount of user effort that is required to make programs run in a
+parallelizable way is significant, and we never managed to develop tools
+that would help in doing so.  At the moment we're not sure if more manpower
+spent on tooling would improve the situation or if the whole idea is really doomed.
+The approach also ended up being a significant overhead on single threaded programs,
+so in the end it is very easy to make your programs slower.  (We have some money
+left in the donation pot for STM which we are not using; according to the rules, we
+could declare the STM attempt failed and channel that money towards the present
+GIL removal proposal.)
 
 * Would subinterpreters not be a better idea?
 
-Python is a very mutable language - there is tons of mutable state and
-basic objects that are compile time in other languages, like classes and functions
-are mutable at runtime. That means that sharing things between subinterpreters would
-be restricted to basic immutable data structures, which defeats the point,
-because it has the same problems as the approach of having multiple processes and
-no additional benefits.
-We don't believe it's a viable approach without
-seriously impacting the semantics of the language.
+Python is a very mutable language - there are tons of mutable state and
+basic objects (classes, functions,...) that are compile-time in other
+language but runtime and fully mutable in Python.  In the end, sharing
+things between subinterpreters would be restricted to basic immutable
+data structures, which defeats the point: it has the same problems as
+the approach of having multiple processes and no additional benefits.
+We believe that this is not viable without seriously impacting the
+semantics of the language (a conclusion which applies to many other
+approaches too).
 
 * Why is it easier to do in PyPy than CPython?
 
@@ -68,11 +73,11 @@
 data structures with locks and what do we do with reference counting that needs
 to be guarded. PyPy only has the former problem; the latter doesn't exist,
 due to a different garbage collector approach.  Of course the first problem
-is a mess too, but at least we are already half-way there. Compare to Jython
+is a mess too, but at least we are already half-way there. Compared to Jython
 or IronPython, PyPy lacks some data structures that are provided by JVM or .NET,
 which we would need to implement, hence the problem is a little harder
 than on an existing multithreaded platform. However, there is good research
-and we know how the problem can be solved.
+and we know how that problem can be solved.
 
 Best regards,
 Maciej Fijalkowski

From pypy.commits at gmail.com  Fri Aug 11 16:18:18 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 11 Aug 2017 13:18:18 -0700 (PDT)
Subject: [pypy-commit] pypy default: Mirror CPython classmethod __reduce__
Message-ID: <598e110a.2f98df0a.61c54.3cac@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: 
Changeset: r92126:c38befdc824e
Date: 2017-08-11 16:17 -0400
http://bitbucket.org/pypy/pypy/changeset/c38befdc824e/

Log:	Mirror CPython classmethod __reduce__

	This makes classmethods pickleable and should fix lib-
	python/3/test_pickle.py.

diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -596,8 +596,9 @@
             new_inst = mod.get('builtin_method_new')
             tup = [w_instance, space.newtext(w_function.name)]
         else:
-            new_inst = mod.get('method_new')
-            tup = [self.w_function, w_instance]
+            w_builtins = space.getbuiltinmodule('builtins')
+            new_inst = space.getattr(w_builtins, space.newtext('getattr'))
+            tup = [w_instance, space.newtext(self.w_function.name)]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -49,6 +49,15 @@
         assert f().__qualname__ == 'inner_global'
         assert f()().__qualname__ == 'inner_global.<locals>.inner_function2'
 
+    def test_classmethod_reduce(self):
+        class X(object):
+            @classmethod
+            def y(cls):
+                pass
+
+        f, args = X.y.__reduce__()
+        assert f(*args) == X.y
+
     def test_annotations(self):
         def f(): pass
         ann = f.__annotations__

From pypy.commits at gmail.com  Fri Aug 11 16:29:36 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 11 Aug 2017 13:29:36 -0700 (PDT)
Subject: [pypy-commit] pypy default: meant for py3.5 branch
Message-ID: <598e13b0.c8a1df0a.84fad.b68d@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: 
Changeset: r92127:4578b8104495
Date: 2017-08-11 16:28 -0400
http://bitbucket.org/pypy/pypy/changeset/4578b8104495/

Log:	meant for py3.5 branch

diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -596,9 +596,8 @@
             new_inst = mod.get('builtin_method_new')
             tup = [w_instance, space.newtext(w_function.name)]
         else:
-            w_builtins = space.getbuiltinmodule('builtins')
-            new_inst = space.getattr(w_builtins, space.newtext('getattr'))
-            tup = [w_instance, space.newtext(self.w_function.name)]
+            new_inst = mod.get('method_new')
+            tup = [self.w_function, w_instance]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -49,15 +49,6 @@
         assert f().__qualname__ == 'inner_global'
         assert f()().__qualname__ == 'inner_global.<locals>.inner_function2'
 
-    def test_classmethod_reduce(self):
-        class X(object):
-            @classmethod
-            def y(cls):
-                pass
-
-        f, args = X.y.__reduce__()
-        assert f(*args) == X.y
-
     def test_annotations(self):
         def f(): pass
         ann = f.__annotations__

From pypy.commits at gmail.com  Fri Aug 11 16:42:23 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 11 Aug 2017 13:42:23 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Mirror CPython classmethod __reduce__
Message-ID: <598e16af.8f371c0a.2e0b3.05c8@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92128:d5f42df20932
Date: 2017-08-11 16:17 -0400
http://bitbucket.org/pypy/pypy/changeset/d5f42df20932/

Log:	Mirror CPython classmethod __reduce__

	This makes classmethods pickleable and should fix lib-
	python/3/test_pickle.py.

diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -596,8 +596,9 @@
             new_inst = mod.get('builtin_method_new')
             tup = [w_instance, space.newtext(w_function.name)]
         else:
-            new_inst = mod.get('method_new')
-            tup = [self.w_function, w_instance]
+            w_builtins = space.getbuiltinmodule('builtins')
+            new_inst = space.getattr(w_builtins, space.newtext('getattr'))
+            tup = [w_instance, space.newtext(self.w_function.name)]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -49,6 +49,15 @@
         assert f().__qualname__ == 'inner_global'
         assert f()().__qualname__ == 'inner_global.<locals>.inner_function2'
 
+    def test_classmethod_reduce(self):
+        class X(object):
+            @classmethod
+            def y(cls):
+                pass
+
+        f, args = X.y.__reduce__()
+        assert f(*args) == X.y
+
     def test_annotations(self):
         def f(): pass
         ann = f.__annotations__

From pypy.commits at gmail.com  Fri Aug 11 16:42:25 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 11 Aug 2017 13:42:25 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: A more precise assertion.
Message-ID: <598e16b1.47e51c0a.d7d98.04af@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92129:2586878e1a3c
Date: 2017-08-11 16:40 -0400
http://bitbucket.org/pypy/pypy/changeset/2586878e1a3c/

Log:	A more precise assertion.

diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -57,6 +57,10 @@
 
         f, args = X.y.__reduce__()
         assert f(*args) == X.y
+        # This is perhaps overly specific.  It's an attempt to be certain that
+        # pickle will actually work with this implementation.
+        assert f == getattr
+        assert args == (X, "y")
 
     def test_annotations(self):
         def f(): pass

From pypy.commits at gmail.com  Sat Aug 12 05:45:40 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 02:45:40 -0700 (PDT)
Subject: [pypy-commit] pypy default: Close wrongly created branch head
Message-ID: <598ece44.2f98df0a.61c54.8daf@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92130:328306bbabd7
Date: 2017-08-12 11:45 +0200
http://bitbucket.org/pypy/pypy/changeset/328306bbabd7/

Log:	Close wrongly created branch head


From pypy.commits at gmail.com  Sat Aug 12 05:58:03 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 02:58:03 -0700 (PDT)
Subject: [pypy-commit] pypy default: Add missing header
Message-ID: <598ed12b.09de1c0a.caf3b.3fe9@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92131:e55de3a1ccc9
Date: 2017-08-12 11:57 +0200
http://bitbucket.org/pypy/pypy/changeset/e55de3a1ccc9/

Log:	Add missing header

diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -74,6 +74,7 @@
 }
 
 #ifdef _WIN32
+#include <Windows.h>
 #define atomic_cas(ptr, oldv, newv)   (InterlockedCompareExchange(ptr, \
                                             newv, oldv) == (oldv))
 #else

From pypy.commits at gmail.com  Sat Aug 12 06:15:24 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 03:15:24 -0700 (PDT)
Subject: [pypy-commit] pypy default: Add a FIXME for _vmprof on Windows
Message-ID: <598ed53c.a799df0a.ad850.5c09@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92132:351e1a58f5ac
Date: 2017-08-12 12:14 +0200
http://bitbucket.org/pypy/pypy/changeset/351e1a58f5ac/

Log:	Add a FIXME for _vmprof on Windows

diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -71,6 +71,8 @@
         working_modules.remove("_cppyy")  # not tested on win32
     if "faulthandler" in working_modules:
         working_modules.remove("faulthandler")  # missing details
+    if "_vmprof" in working_modules:
+        working_modules.remove("_vmprof")  # FIXME: missing details
 
     # The _locale module is needed by site.py on Windows
     default_modules.add("_locale")

From pypy.commits at gmail.com  Sat Aug 12 09:12:42 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 06:12:42 -0700 (PDT)
Subject: [pypy-commit] cffi default: Mention that ffi.dlopen(None) does not
 work on Python 3 on Windows.
Message-ID: <598efeca.4588df0a.dd3fd.b89e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3000:50aa734652c8
Date: 2017-08-12 15:12 +0200
http://bitbucket.org/cffi/cffi/changeset/50aa734652c8/

Log:	Mention that ffi.dlopen(None) does not work on Python 3 on Windows.

diff --git a/doc/source/overview.rst b/doc/source/overview.rst
--- a/doc/source/overview.rst
+++ b/doc/source/overview.rst
@@ -44,6 +44,10 @@
 arguments.  In the above example it would be ``b"world"`` and ``b"hi
 there, %s!\n"``.  In general it is ``somestring.encode(myencoding)``.
 
+*Python 3 on Windows:* ``ffi.dlopen(None)`` does not work.  This problem
+is messy and not really fixable.  The example above could be fixed by
+calling another function from a specific DLL that exists on your system.
+
 *This example does not call any C compiler.  It works in the so-called
 ABI mode, which means that it will crash if you call some function or
 access some fields of a structure that was slightly misdeclared in the

From pypy.commits at gmail.com  Sat Aug 12 09:23:01 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 06:23:01 -0700 (PDT)
Subject: [pypy-commit] cffi default: Another note about performance
Message-ID: <598f0135.e484df0a.f8bd.47a9@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3001:281124d85f28
Date: 2017-08-12 15:22 +0200
http://bitbucket.org/cffi/cffi/changeset/281124d85f28/

Log:	Another note about performance

diff --git a/doc/source/cdef.rst b/doc/source/cdef.rst
--- a/doc/source/cdef.rst
+++ b/doc/source/cdef.rst
@@ -314,7 +314,9 @@
 Let me state it again: this gives ABI-level access to the library, so
 you need to have all types declared manually exactly as they were
 while the library was made.  No checking is done.  Mismatches can
-cause random crashes.
+cause random crashes.  API-level access, on the other hand, is safer.
+Speed-wise, API-level access is much faster (it is common to have
+the opposite misconception about performance).
 
 Note that only functions and global variables live in library objects;
 the types exist in the ``ffi`` instance independently of library objects.

From pypy.commits at gmail.com  Sat Aug 12 09:30:50 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 06:30:50 -0700 (PDT)
Subject: [pypy-commit] cffi default: Another note about performance,
 this time in whatsnew.rst
Message-ID: <598f030a.2e9ddf0a.15d48.f415@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3002:8a874e1b7a32
Date: 2017-08-12 15:30 +0200
http://bitbucket.org/cffi/cffi/changeset/8a874e1b7a32/

Log:	Another note about performance, this time in whatsnew.rst

diff --git a/doc/source/whatsnew.rst b/doc/source/whatsnew.rst
--- a/doc/source/whatsnew.rst
+++ b/doc/source/whatsnew.rst
@@ -49,6 +49,10 @@
   In the future, this might have an effect on CPython too (provided
   the CPython `issue 31105`__ is addressed).
 
+* Add a note to the documentation: the ABI mode gives function objects
+  that are *slower* to call than the API mode does.  For some reason it
+  is often thought to be faster.  It is not!
+
 .. __: https://bitbucket.org/cffi/cffi/issues/321/cffi-191-segmentation-fault-during-self
 .. __: ref.html#ffi-gc
 .. __: https://bitbucket.org/cffi/cffi/issues/320/improve-memory_pressure-management

From pypy.commits at gmail.com  Sat Aug 12 10:39:27 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 12 Aug 2017 07:39:27 -0700 (PDT)
Subject: [pypy-commit] pypy reverse-debugger: Update the ASLR failure message
Message-ID: <598f131f.2e9ddf0a.15d48.000e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: reverse-debugger
Changeset: r92133:136c8d30b2d7
Date: 2017-08-12 16:38 +0200
http://bitbucket.org/pypy/pypy/changeset/136c8d30b2d7/

Log:	Update the ASLR failure message

diff --git a/rpython/translator/revdb/src-revdb/revdb.c b/rpython/translator/revdb/src-revdb/revdb.c
--- a/rpython/translator/revdb/src-revdb/revdb.c
+++ b/rpython/translator/revdb/src-revdb/revdb.c
@@ -1031,6 +1031,12 @@
                 "command disables it manually:\n"
                 "\n"
                 "    echo 0 | sudo tee /proc/sys/kernel/randomize_va_space\n"
+                "\n"
+                "It has been reported that on Linux kernel 4.12.4-1-ARCH,\n"
+                "ASLR cannot be disabled at all for libpypy-c.so.  For now\n"
+                "there is no good solution.  Either you downgrade the\n"
+                "kernel, or you translate with --no-shared (and you loose\n"
+                "PyPy's cpyext ability).\n"
                 "\n", argv[0]);
         exit(1);
     }

From pypy.commits at gmail.com  Sat Aug 12 12:50:05 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 09:50:05 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Allow non-ascii extension names (PEP
 489)
Message-ID: <598f31bd.1aa4df0a.b6169.9793@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92134:9adc44d5f032
Date: 2017-08-12 18:49 +0200
http://bitbucket.org/pypy/pypy/changeset/9adc44d5f032/

Log:	Allow non-ascii extension names (PEP 489)

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -26,6 +26,7 @@
 from pypy.interpreter.module import Module
 from pypy.interpreter.function import StaticMethod
 from pypy.objspace.std.sliceobject import W_SliceObject
+from pypy.objspace.std.unicodeobject import encode_object
 from pypy.module.__builtin__.descriptor import W_Property
 #from pypy.module.micronumpy.base import W_NDimArray
 from rpython.rlib.entrypoint import entrypoint_lowlevel
@@ -1476,12 +1477,11 @@
     # order of things here.
     from rpython.rlib import rdynload
 
-    name = space.text_w(space.getattr(w_spec, space.newtext("name")))
+    w_name = space.getattr(w_spec, space.newtext("name"))
     path = space.text_w(space.getattr(w_spec, space.newtext("origin")))
 
     if os.sep not in path:
         path = os.curdir + os.sep + path      # force a '/' in the path
-    basename = name.split('.')[-1]
     try:
         ll_libname = rffi.str2charp(path)
         try:
@@ -1489,13 +1489,14 @@
         finally:
             lltype.free(ll_libname, flavor='raw')
     except rdynload.DLOpenError as e:
-        w_name = space.newunicode(name.decode('ascii'))
         w_path = space.newfilename(path)
         raise raise_import_error(space,
             space.newfilename(e.msg), w_name, w_path)
     look_for = None
+    name = space.text_w(w_name)
     #
     if space.config.objspace.usemodules._cffi_backend:
+        basename = name.split('.')[-1]
         look_for = '_cffi_pypyinit_%s' % (basename,)
         try:
             initptr = rdynload.dlsym(dll, look_for)
@@ -1510,7 +1511,7 @@
                 raise
     #
     if space.config.objspace.usemodules.cpyext:
-        also_look_for = 'PyInit_%s' % (basename,)
+        also_look_for = get_init_name(space, w_name)
         try:
             initptr = rdynload.dlsym(dll, also_look_for)
         except KeyError:
@@ -1523,10 +1524,21 @@
             look_for = also_look_for
     msg = u"function %s not found in library %s" % (
         unicode(look_for), space.unicode_w(space.newfilename(path)))
-    w_name = space.newunicode(name.decode('ascii'))
     w_path = space.newfilename(path)
     raise_import_error(space, space.newunicode(msg), w_name, w_path)
 
+def get_init_name(space, w_name):
+    name_u = space.unicode_w(w_name)
+    basename_u = name_u.split(u'.')[-1]
+    try:
+        basename = basename_u.encode('ascii')
+        return 'PyInit_%s' % (basename,)
+    except UnicodeEncodeError:
+        basename = space.bytes_w(encode_object(
+            space, space.newunicode(basename_u), 'punycode', None))
+        basename = basename.replace('-', '_')
+        return 'PyInitU_%s' % (basename,)
+
 
 initfunctype = lltype.Ptr(lltype.FuncType([], PyObject))
 
@@ -1570,6 +1582,7 @@
                                                    name)
     finally:
         state.package_context = old_context
+    # XXX: should disable single-step init for non-ascii module names
     w_mod = get_w_obj_and_decref(space, initret)
     state.fixup_extension(w_mod, name, path)
     return w_mod
diff --git a/pypy/module/cpyext/test/test_module.py b/pypy/module/cpyext/test/test_module.py
--- a/pypy/module/cpyext/test/test_module.py
+++ b/pypy/module/cpyext/test/test_module.py
@@ -178,12 +178,13 @@
         importlib.reload(module)
         assert ex_class is module.Example
 
-    def w_load_from_name(self, name, origin=None):
+    def w_load_from_name(self, name, origin=None, use_prefix=True):
         from importlib import machinery, util
         if not origin:
             module = self.import_module(name=self.name)
             origin = module.__loader__.path
-        name = '_testmultiphase_' + name
+        if use_prefix:
+            name = '_testmultiphase_' + name
         loader = machinery.ExtensionFileLoader(name, origin)
         spec = util.spec_from_loader(name, loader)
         module = util.module_from_spec(spec)
@@ -228,3 +229,21 @@
         excinfo = raises(SystemError, self.load_from_name, 'export_unreported_exception')
         assert "initialization" in excinfo.value.args[0]
         assert "unreported exception" in excinfo.value.args[0]
+
+    def test_unloadable_nonascii(self):
+        name = u"fo\xf3"
+        excinfo = raises(ImportError, self.load_from_name, name)
+        assert excinfo.value.name == '_testmultiphase_' + name
+
+    def test_nonascii(self):
+        module = self.import_module(name=self.name)
+        origin = module.__loader__.path
+        cases = [
+            ('_testmultiphase_zkou\u0161ka_na\u010dten\xed', 'Czech'),
+            ('\uff3f\u30a4\u30f3\u30dd\u30fc\u30c8\u30c6\u30b9\u30c8',
+             'Japanese'),
+            ]
+        for name, lang in cases:
+            module = self.load_from_name(name, origin=origin, use_prefix=False)
+            assert module.__name__ == name
+            assert module.__doc__ == "Module named in %s" % lang

From pypy.commits at gmail.com  Sat Aug 12 14:23:50 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 11:23:50 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Ensure that reloading an extension
 is a no-op
Message-ID: <598f47b6.41931c0a.90df6.f384@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92135:b086a8161ab6
Date: 2017-08-12 20:23 +0200
http://bitbucket.org/pypy/pypy/changeset/b086a8161ab6/

Log:	Ensure that reloading an extension is a no-op

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1595,6 +1595,8 @@
         return
     space.getbuiltinmodule("cpyext")
     mod_as_pyobj = rawrefcount.from_obj(PyObject, w_mod)
+    if cts.cast('PyModuleObject*', mod_as_pyobj).c_md_state:
+        return
     if mod_as_pyobj:
         return exec_def(space, w_mod, mod_as_pyobj)
 
diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py
--- a/pypy/module/cpyext/modsupport.py
+++ b/pypy/module/cpyext/modsupport.py
@@ -142,6 +142,11 @@
     mod = rffi.cast(PyModuleObject, mod_as_pyobj)
     moddef = mod.c_md_def
     cur_slot = rffi.cast(rffi.CArrayPtr(PyModuleDef_Slot), moddef.c_m_slots)
+    if moddef.c_m_size >= 0 and not mod.c_md_state:
+        # Always set md_state, to use as marker for exec_extension_module()
+        # (cf. CPython's PyModule_ExecDef)
+        mod.c_md_state = lltype.malloc(
+            rffi.VOIDP.TO, moddef.c_m_size, flavor='raw', zero=True)
     while cur_slot and rffi.cast(lltype.Signed, cur_slot[0].c_slot):
         if rffi.cast(lltype.Signed, cur_slot[0].c_slot) == 2:
             execf = rffi.cast(execfunctype, cur_slot[0].c_value)
diff --git a/pypy/module/cpyext/test/test_module.py b/pypy/module/cpyext/test/test_module.py
--- a/pypy/module/cpyext/test/test_module.py
+++ b/pypy/module/cpyext/test/test_module.py
@@ -175,7 +175,8 @@
         import importlib
         module = self.import_module(name=self.name)
         ex_class = module.Example
-        importlib.reload(module)
+        # Simulate what importlib.reload() does, without recomputing the spec
+        module.__spec__.loader.exec_module(module)
         assert ex_class is module.Example
 
     def w_load_from_name(self, name, origin=None, use_prefix=True):

From pypy.commits at gmail.com  Sat Aug 12 14:39:35 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 11:39:35 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <598f4b67.15b81c0a.9bd01.05b3@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92136:02ab6e8e42e8
Date: 2017-08-12 20:35 +0200
http://bitbucket.org/pypy/pypy/changeset/02ab6e8e42e8/

Log:	hg merge default

diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -73,6 +73,8 @@
 
     if "_cppyy" in working_modules:
         working_modules.remove("_cppyy")  # not tested on win32
+    if "_vmprof" in working_modules:
+        working_modules.remove("_vmprof")  # FIXME: missing details
 
     # The _locale module is needed by site.py on Windows
     default_modules.add("_locale")
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -74,6 +74,7 @@
 }
 
 #ifdef _WIN32
+#include <Windows.h>
 #define atomic_cas(ptr, oldv, newv)   (InterlockedCompareExchange(ptr, \
                                             newv, oldv) == (oldv))
 #else

From pypy.commits at gmail.com  Sat Aug 12 14:39:37 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 11:39:37 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <598f4b69.5ba4df0a.f8c11.d2e0@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92137:154db7d0b56e
Date: 2017-08-12 20:37 +0200
http://bitbucket.org/pypy/pypy/changeset/154db7d0b56e/

Log:	hg merge py3.5

diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -73,6 +73,8 @@
 
     if "_cppyy" in working_modules:
         working_modules.remove("_cppyy")  # not tested on win32
+    if "_vmprof" in working_modules:
+        working_modules.remove("_vmprof")  # FIXME: missing details
 
     # The _locale module is needed by site.py on Windows
     default_modules.add("_locale")
diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -596,8 +596,9 @@
             new_inst = mod.get('builtin_method_new')
             tup = [w_instance, space.newtext(w_function.name)]
         else:
-            new_inst = mod.get('method_new')
-            tup = [self.w_function, w_instance]
+            w_builtins = space.getbuiltinmodule('builtins')
+            new_inst = space.getattr(w_builtins, space.newtext('getattr'))
+            tup = [w_instance, space.newtext(self.w_function.name)]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/interpreter/test/test_function.py b/pypy/interpreter/test/test_function.py
--- a/pypy/interpreter/test/test_function.py
+++ b/pypy/interpreter/test/test_function.py
@@ -49,6 +49,19 @@
         assert f().__qualname__ == 'inner_global'
         assert f()().__qualname__ == 'inner_global.<locals>.inner_function2'
 
+    def test_classmethod_reduce(self):
+        class X(object):
+            @classmethod
+            def y(cls):
+                pass
+
+        f, args = X.y.__reduce__()
+        assert f(*args) == X.y
+        # This is perhaps overly specific.  It's an attempt to be certain that
+        # pickle will actually work with this implementation.
+        assert f == getattr
+        assert args == (X, "y")
+
     def test_annotations(self):
         def f(): pass
         ann = f.__annotations__
diff --git a/rpython/translator/c/src/signals.c b/rpython/translator/c/src/signals.c
--- a/rpython/translator/c/src/signals.c
+++ b/rpython/translator/c/src/signals.c
@@ -74,6 +74,7 @@
 }
 
 #ifdef _WIN32
+#include <Windows.h>
 #define atomic_cas(ptr, oldv, newv)   (InterlockedCompareExchange(ptr, \
                                             newv, oldv) == (oldv))
 #else

From pypy.commits at gmail.com  Sat Aug 12 15:35:01 2017
From: pypy.commits at gmail.com (mattip)
Date: Sat, 12 Aug 2017 12:35:01 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: suggest some wording/editorial
 changes
Message-ID: <598f5865.1aa4df0a.b6169.bc82@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: extradoc
Changeset: r5830:c40625c68b87
Date: 2017-08-12 22:34 +0300
http://bitbucket.org/pypy/extradoc/changeset/c40625c68b87/

Log:	suggest some wording/editorial changes

diff --git a/blog/draft/remove-gil.rst b/blog/draft/remove-gil.rst
--- a/blog/draft/remove-gil.rst
+++ b/blog/draft/remove-gil.rst
@@ -3,31 +3,33 @@
 
 Hello everyone.
 
-Discussions about the infamous Global Interpreter Lock have been around for a while
-in the Python community. There has been various attempts at removing it:
-some were successful, like e.g. in Jython or IronPython with the help of the platform, and some yet to bear fruit, like `gilectomy`_. Since our `February sprint`_ in Leysin,
-we've been on-and-off tackling directly the topic of GIL removal in the PyPy project.
+The Python community has been discussing removing the Global Interpreter Lock for
+a long time.
+There have been various attempts at removing it:
+Jython or IronPython successfully removed it with the help of the underlying 
+platform, and some have yet to bear fruit, like `gilectomy`_. Since our `February sprint`_ in Leysin,
+we have experimented with the topic of GIL removal in the PyPy project.
 We believe that the work done in IronPython or Jython can be reproduced with
-only a bit more effort. Compared to that, removing the GIL in CPython is a much
+only a bit more effort in PyPy. Compared to that, removing the GIL in CPython is a much
 harder topic, since it also requires tackling the problem of multi-threaded reference
-counting. See the section below for further discussions.
+counting. See the section below for further details.
 
 .. _`February sprint`: https://morepypy.blogspot.it/2017/03/leysin-winter-sprint-summary.html
 
-As we announced at EuroPython, what we have got so far is a GIL-less PyPy
-which can run **very simple** multi-threaded programs which are nicely
-parallelized.  At the moment, non-simple programs most likely segfaults: the
+As we announced at EuroPython, what we have so far is a GIL-less PyPy
+which can run **very simple** multi-threaded, nicely parallelized, programs.
+At the moment, more complicated programs probably segfault. The
 remaining 90% (and another 90%) of work is with putting locks in strategic
-places so PyPy does not segfault when you try to do concurrent accesses to
+places so PyPy does not segfault during concurrent accesses to
 data structures.
 
-Since such work would complicate the code base and our day-to-day work,
-we would like to judge the interest on the community and the commercial
+Since such work would complicate the PyPy code base and our day-to-day work,
+we would like to judge the interest of the community and the commercial
 partners to make it happen (we are not looking for individual
 donations at this point).  We estimate a total cost of $50k,
 out of which we already have backing for about 1/3 (with a possible 1/3
 extra from the STM money, see below).  This would give us a good
-shot at delivering a good proof of concept of working PyPy no-GIL. If we can get a $100k
+shot at delivering a good proof-of-concept working PyPy with no GIL. If we can get a $100k
 contract, we will deliver a fully working PyPy interpreter with no GIL as a release,
 possibly separate from the default PyPy release.
 
@@ -36,42 +38,45 @@
 
 * What would the plan entail?
 
-We've already done the work on Garbage Collector to allow doing multi-
+We've already done the work on the Garbage Collector to allow doing multi-
 threaded programs in RPython.  "All" that is left is adding locks on mutable
-data structures everywhere in the PyPy codebase. Since it'll significantly complicate
-our workflow, we need to see real interest in that topic, backed up by
-commercial contracts, otherwise we're not going to do it.
+data structures everywhere in the PyPy codebase. Since it would significantly complicate
+our workflow, we require real interest in that topic, backed up by
+commercial contracts in order to justify the added maintenance burden.
 
-* Why the STM effort did not work out?
+* Why did the STM effort not work out?
 
 STM was a research project that proved that the idea is possible. However,
 the amount of user effort that is required to make programs run in a
 parallelizable way is significant, and we never managed to develop tools
-that would help in doing so.  At the moment we're not sure if more manpower
+that would help in doing so.  At the moment we're not sure if more work
 spent on tooling would improve the situation or if the whole idea is really doomed.
-The approach also ended up being a significant overhead on single threaded programs,
+The approach also ended up adding significant overhead on single threaded programs,
 so in the end it is very easy to make your programs slower.  (We have some money
 left in the donation pot for STM which we are not using; according to the rules, we
 could declare the STM attempt failed and channel that money towards the present
 GIL removal proposal.)
 
-* Would subinterpreters not be a better idea?
+* Wouldn't subinterpreters be a better idea?
 
 Python is a very mutable language - there are tons of mutable state and
 basic objects (classes, functions,...) that are compile-time in other
 language but runtime and fully mutable in Python.  In the end, sharing
 things between subinterpreters would be restricted to basic immutable
-data structures, which defeats the point: it has the same problems as
-the approach of having multiple processes and no additional benefits.
-We believe that this is not viable without seriously impacting the
+data structures, which defeats the point. Subinterpreters suffers from the same problems as
+multiprocessing with no additional benefits.
+We believe that reducing mutability to implement subinterpreters is not viable without seriously impacting the
 semantics of the language (a conclusion which applies to many other
 approaches too).
 
 * Why is it easier to do in PyPy than CPython?
 
-Removing the GIL in CPython has two problems - how do we guard access to mutable
-data structures with locks and what do we do with reference counting that needs
-to be guarded. PyPy only has the former problem; the latter doesn't exist,
+Removing the GIL in CPython has two problems:
+
+- how do we guard access to mutable  data structures with locks and  
+- what to do with reference counting that needs to be guarded. 
+
+PyPy only has the former problem; the latter doesn't exist,
 due to a different garbage collector approach.  Of course the first problem
 is a mess too, but at least we are already half-way there. Compared to Jython
 or IronPython, PyPy lacks some data structures that are provided by JVM or .NET,

From pypy.commits at gmail.com  Sat Aug 12 18:01:59 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 15:01:59 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: fix translation
Message-ID: <598f7ad7.aa8adf0a.b0db5.aa25@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92138:4c0d6aa834a4
Date: 2017-08-13 00:01 +0200
http://bitbucket.org/pypy/pypy/changeset/4c0d6aa834a4/

Log:	fix translation

diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -598,7 +598,7 @@
         else:
             w_builtins = space.getbuiltinmodule('builtins')
             new_inst = space.getattr(w_builtins, space.newtext('getattr'))
-            tup = [w_instance, space.newtext(self.w_function.name)]
+            tup = [w_instance, space.newunicode(w_function.getname(space))]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -46,7 +46,7 @@
 class W_PyCFunctionObject(W_Root):
     def __init__(self, space, ml, w_self, w_module=None):
         self.ml = ml
-        self.name = rffi.charp2str(rffi.cast(rffi.CCHARP,self.ml.c_ml_name))
+        self.name = rffi.charp2str(rffi.cast(rffi.CCHARP, self.ml.c_ml_name))
         self.w_self = w_self
         self.w_module = w_module
 
@@ -205,8 +205,8 @@
     args_w, kw_w = __args__.unpack()
     if len(args_w) < 1:
         raise oefmt(space.w_TypeError,
-            "descriptor '%s' of '%s' object needs an argument",
-            self.name, self.w_objclass.getname(space))
+            "descriptor '%8' of '%N' object needs an argument",
+            self.name, self.w_objclass)
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
@@ -328,8 +328,8 @@
             if not method.c_ml_name:
                 break
             if name == "__methods__":
-                method_list_w.append(
-                    space.newtext(rffi.charp2str(rffi.cast(rffi.CCHARP, method.c_ml_name))))
+                method_list_w.append(space.newtext(rffi.charp2str(
+                    rffi.cast(rffi.CCHARP, method.c_ml_name))))
             elif rffi.charp2str(rffi.cast(rffi.CCHARP, method.c_ml_name)) == name: # XXX expensive copy
                 return W_PyCFunctionObject(space, method, w_obj)
     if name == "__methods__":

From pypy.commits at gmail.com  Sat Aug 12 18:03:26 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sat, 12 Aug 2017 15:03:26 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <598f7b2e.9398df0a.48c3d.f9cc@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92139:3ddef501ba8e
Date: 2017-08-13 00:02 +0200
http://bitbucket.org/pypy/pypy/changeset/3ddef501ba8e/

Log:	hg merge py3.5

diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -598,7 +598,7 @@
         else:
             w_builtins = space.getbuiltinmodule('builtins')
             new_inst = space.getattr(w_builtins, space.newtext('getattr'))
-            tup = [w_instance, space.newtext(self.w_function.name)]
+            tup = [w_instance, space.newunicode(w_function.getname(space))]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/module/cpyext/methodobject.py b/pypy/module/cpyext/methodobject.py
--- a/pypy/module/cpyext/methodobject.py
+++ b/pypy/module/cpyext/methodobject.py
@@ -46,7 +46,7 @@
 class W_PyCFunctionObject(W_Root):
     def __init__(self, space, ml, w_self, w_module=None):
         self.ml = ml
-        self.name = rffi.charp2str(rffi.cast(rffi.CCHARP,self.ml.c_ml_name))
+        self.name = rffi.charp2str(rffi.cast(rffi.CCHARP, self.ml.c_ml_name))
         self.w_self = w_self
         self.w_module = w_module
 
@@ -205,8 +205,8 @@
     args_w, kw_w = __args__.unpack()
     if len(args_w) < 1:
         raise oefmt(space.w_TypeError,
-            "descriptor '%s' of '%s' object needs an argument",
-            self.name, self.w_objclass.getname(space))
+            "descriptor '%8' of '%N' object needs an argument",
+            self.name, self.w_objclass)
     w_instance = args_w[0] # XXX typecheck missing
     w_args = space.newtuple(args_w[1:])
     w_kw = space.newdict()
@@ -328,8 +328,8 @@
             if not method.c_ml_name:
                 break
             if name == "__methods__":
-                method_list_w.append(
-                    space.newtext(rffi.charp2str(rffi.cast(rffi.CCHARP, method.c_ml_name))))
+                method_list_w.append(space.newtext(rffi.charp2str(
+                    rffi.cast(rffi.CCHARP, method.c_ml_name))))
             elif rffi.charp2str(rffi.cast(rffi.CCHARP, method.c_ml_name)) == name: # XXX expensive copy
                 return W_PyCFunctionObject(space, method, w_obj)
     if name == "__methods__":

From pypy.commits at gmail.com  Sun Aug 13 10:07:22 2017
From: pypy.commits at gmail.com (rlamy)
Date: Sun, 13 Aug 2017 07:07:22 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: fix exec_extension_module()
Message-ID: <59905d1a.a799df0a.ad850.e7e2@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92140:0385d595a7df
Date: 2017-08-13 16:06 +0200
http://bitbucket.org/pypy/pypy/changeset/0385d595a7df/

Log:	fix exec_extension_module()

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1595,9 +1595,10 @@
         return
     space.getbuiltinmodule("cpyext")
     mod_as_pyobj = rawrefcount.from_obj(PyObject, w_mod)
-    if cts.cast('PyModuleObject*', mod_as_pyobj).c_md_state:
-        return
     if mod_as_pyobj:
+        if cts.cast('PyModuleObject*', mod_as_pyobj).c_md_state:
+            # already initialised
+            return
         return exec_def(space, w_mod, mod_as_pyobj)
 
 @specialize.ll()

From pypy.commits at gmail.com  Sun Aug 13 14:46:46 2017
From: pypy.commits at gmail.com (minrk)
Date: Sun, 13 Aug 2017 11:46:46 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: add darwin patches to sysconfig
Message-ID: <59909e96.b785df0a.3b8b7.8c94@mx.google.com>

Author: Min RK <benjaminrk at gmail.com>
Branch: py3.5
Changeset: r92141:92ede31b23c6
Date: 2017-08-13 12:11 +0000
http://bitbucket.org/pypy/pypy/changeset/92ede31b23c6/

Log:	add darwin patches to sysconfig

	copied patch from py2.7 adds flags such as -arch and -undefined
	dynamic_lookup

diff --git a/lib-python/3/distutils/sysconfig_pypy.py b/lib-python/3/distutils/sysconfig_pypy.py
--- a/lib-python/3/distutils/sysconfig_pypy.py
+++ b/lib-python/3/distutils/sysconfig_pypy.py
@@ -81,6 +81,19 @@
     g['LIBDIR'] = os.path.join(sys.prefix, 'lib')
     g['VERSION'] = get_python_version()
 
+    if sys.platform[:6] == "darwin":
+        import platform
+        if platform.machine() == 'i386':
+            if platform.architecture()[0] == '32bit':
+                arch = 'i386'
+            else:
+                arch = 'x86_64'
+        else:
+            # just a guess
+            arch = platform.machine()
+        g['LDSHARED'] += ' -undefined dynamic_lookup'
+        g['CC'] += ' -arch %s' % (arch,)
+
     global _config_vars
     _config_vars = g
 

From pypy.commits at gmail.com  Mon Aug 14 08:17:02 2017
From: pypy.commits at gmail.com (exarkun)
Date: Mon, 14 Aug 2017 05:17:02 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Replace fragile upstream expat exception
 test with a more reasonable one
Message-ID: <599194be.5597df0a.e9f1e.4ca6@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92142:e2a71b4f385a
Date: 2017-08-14 08:16 -0400
http://bitbucket.org/pypy/pypy/changeset/e2a71b4f385a/

Log:	Replace fragile upstream expat exception test with a more reasonable
	one

diff --git a/lib-python/3/test/test_pyexpat.py b/lib-python/3/test/test_pyexpat.py
--- a/lib-python/3/test/test_pyexpat.py
+++ b/lib-python/3/test/test_pyexpat.py
@@ -11,7 +11,7 @@
 from xml.parsers import expat
 from xml.parsers.expat import errors
 
-from test.support import sortdict
+from test.support import sortdict, impl_detail
 
 
 class SetAttributeTest(unittest.TestCase):
@@ -446,6 +446,7 @@
         self.assertEqual(os.path.basename(entry[0]), filename)
         self.assertEqual(entry[2], funcname)
 
+    @impl_detail("PyPy does not have pyexpat.c", pypy=False)
     def test_exception(self):
         parser = expat.ParserCreate()
         parser.StartElementHandler = self.StartElementHandler
@@ -458,6 +459,7 @@
                              " found %r" % e.args[0])
             # Check that the traceback contains the relevant line in pyexpat.c
             entries = traceback.extract_tb(e.__traceback__)
+            print(entries)
             self.assertEqual(len(entries), 3)
             self.check_traceback_entry(entries[0],
                                        "test_pyexpat.py", "test_exception")
diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py
--- a/pypy/module/pyexpat/test/test_parser.py
+++ b/pypy/module/pyexpat/test/test_parser.py
@@ -210,6 +210,30 @@
             p.ParseFile(fake_reader)
             assert fake_reader.read_count == 4
 
+
+    def test_exception(self):
+        """
+        lib-python/3/test_pyexpat.py:HandlerExceptionTest.test_exception port
+        without the fragile traceback inspection.
+        """
+        import pyexpat as expat
+
+        def StartElementHandler(name, attrs):
+            raise RuntimeError(name)
+
+        parser = expat.ParserCreate()
+        parser.StartElementHandler = StartElementHandler
+
+        try:
+            parser.Parse(b"<a><b><c/></b></a>", 1)
+            self.fail()
+        except RuntimeError as e:
+            assert e.args[0] == 'a', (
+                "Expected RuntimeError for element 'a', but" + \
+                " found %r" % e.args[0]
+            )
+
+
 class AppTestPyexpat2:
     spaceconfig = dict(usemodules=['_rawffi', 'pyexpat', 'itertools',
                                    '_socket', 'time', 'struct', 'binascii',

From pypy.commits at gmail.com  Mon Aug 14 08:18:33 2017
From: pypy.commits at gmail.com (exarkun)
Date: Mon, 14 Aug 2017 05:18:33 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Remove debug print
Message-ID: <59919519.97a0df0a.e200a.1e17@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92143:dc0a896dbbd6
Date: 2017-08-14 08:17 -0400
http://bitbucket.org/pypy/pypy/changeset/dc0a896dbbd6/

Log:	Remove debug print

diff --git a/lib-python/3/test/test_pyexpat.py b/lib-python/3/test/test_pyexpat.py
--- a/lib-python/3/test/test_pyexpat.py
+++ b/lib-python/3/test/test_pyexpat.py
@@ -459,7 +459,6 @@
                              " found %r" % e.args[0])
             # Check that the traceback contains the relevant line in pyexpat.c
             entries = traceback.extract_tb(e.__traceback__)
-            print(entries)
             self.assertEqual(len(entries), 3)
             self.check_traceback_entry(entries[0],
                                        "test_pyexpat.py", "test_exception")

From pypy.commits at gmail.com  Mon Aug 14 10:39:56 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 14 Aug 2017 07:39:56 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: fix crash when non-ascii extension
 cannot be found
Message-ID: <5991b63c.4bd51c0a.6fc82.724d@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92144:8c278da05b4c
Date: 2017-08-14 16:39 +0200
http://bitbucket.org/pypy/pypy/changeset/8c278da05b4c/

Log:	fix crash when non-ascii extension cannot be found

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1523,7 +1523,7 @@
         else:
             look_for = also_look_for
     msg = u"function %s not found in library %s" % (
-        unicode(look_for), space.unicode_w(space.newfilename(path)))
+        look_for.decode('utf-8'), space.unicode_w(space.newfilename(path)))
     w_path = space.newfilename(path)
     raise_import_error(space, space.newunicode(msg), w_name, w_path)
 

From pypy.commits at gmail.com  Mon Aug 14 11:24:48 2017
From: pypy.commits at gmail.com (exarkun)
Date: Mon, 14 Aug 2017 08:24:48 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Use utf-8 for readline history file.
Message-ID: <5991c0c0.43491c0a.4c481.ddc0@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92145:759e514cd5ab
Date: 2017-08-14 11:23 -0400
http://bitbucket.org/pypy/pypy/changeset/759e514cd5ab/

Log:	Use utf-8 for readline history file.

diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -297,10 +297,7 @@
         line = line.rstrip('\n')
         if isinstance(line, unicode):
             return line # on py3k
-        try:
-            return unicode(line, ENCODING)
-        except UnicodeDecodeError:   # bah, silently fall back...
-            return unicode(line, 'utf-8', 'replace')
+        return unicode(line, 'utf-8', 'replace')
 
     def get_history_length(self):
         return self.saved_history_length
@@ -317,7 +314,7 @@
         # history item: we use \r\n instead of just \n.  If the history
         # file is passed to GNU readline, the extra \r are just ignored.
         history = self.get_reader().history
-        f = open(os.path.expanduser(filename), 'r')
+        f = open(os.path.expanduser(filename), 'r', encoding='utf-8')
         buffer = []
         for line in f:
             if line.endswith('\r\n'):
@@ -334,15 +331,12 @@
     def write_history_file(self, filename='~/.history'):
         maxlength = self.saved_history_length
         history = self.get_reader().get_trimmed_history(maxlength)
-        f = open(os.path.expanduser(filename), 'w')
+        f = open(os.path.expanduser(filename), 'w', encoding='utf-8')
         for entry in history:
             # if we are on py3k, we don't need to encode strings before
             # writing it to a file
             if isinstance(entry, unicode) and sys.version_info < (3,):
-                try:
-                    entry = entry.encode(ENCODING)
-                except UnicodeEncodeError:   # bah, silently fall back...
-                    entry = entry.encode('utf-8')
+                entry = entry.encode('utf-8')
             entry = entry.replace('\n', '\r\n')   # multiline history support
             f.write(entry + '\n')
         f.close()
diff --git a/pypy/module/readline/test/test_readline.py b/pypy/module/readline/test/test_readline.py
new file mode 100644
--- /dev/null
+++ b/pypy/module/readline/test/test_readline.py
@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+
+class AppTestReadline:
+    spaceconfig = dict(usemodules={
+        'unicodedata', 'termios', 'select', 'signal', 'fcntl',
+        '_minimal_curses', 'faulthandler', '_socket', 'binascii',
+        '_posixsubprocess',
+    })
+
+    def test_nonascii_history(self):
+        import os, readline
+        TESTFN = "{}_{}_tmp".format("@test", os.getpid())
+
+        is_editline = readline.__doc__ and "libedit" in readline.__doc__
+
+        readline.clear_history()
+        try:
+            readline.add_history("entr&#233;e 1")
+        except UnicodeEncodeError as err:
+            skip("Locale cannot encode test data: " + format(err))
+        readline.add_history("entr&#233;e 2")
+        readline.replace_history_item(1, "entr&#233;e 22")
+        readline.write_history_file(TESTFN)
+        readline.clear_history()
+        readline.read_history_file(TESTFN)
+        if is_editline:
+            # An add_history() call seems to be required for get_history_
+            # item() to register items from the file
+            readline.add_history("dummy")
+        assert readline.get_history_item(1) ==  "entr&#233;e 1"
+        assert readline.get_history_item(2) == "entr&#233;e 22"

From pypy.commits at gmail.com  Mon Aug 14 13:15:44 2017
From: pypy.commits at gmail.com (arigo)
Date: Mon, 14 Aug 2017 10:15:44 -0700 (PDT)
Subject: [pypy-commit] pypy default: A corner case for super.__get__
Message-ID: <5991dac0.2cacdf0a.e83e2.419b@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92146:35cb17be8aae
Date: 2017-08-14 19:12 +0200
http://bitbucket.org/pypy/pypy/changeset/35cb17be8aae/

Log:	A corner case for super.__get__

diff --git a/pypy/module/__builtin__/descriptor.py b/pypy/module/__builtin__/descriptor.py
--- a/pypy/module/__builtin__/descriptor.py
+++ b/pypy/module/__builtin__/descriptor.py
@@ -16,12 +16,12 @@
     def descr_init(self, space, w_starttype, w_obj_or_type=None):
         if space.is_none(w_obj_or_type):
             w_type = None  # unbound super object
-            w_obj_or_type = space.w_None
+            w_obj_or_type = None
         else:
             w_type = _super_check(space, w_starttype, w_obj_or_type)
         self.w_starttype = w_starttype
         self.w_objtype = w_type
-        self.w_self = w_obj_or_type
+        self.w_self = w_obj_or_type      # may be None
 
     def descr_repr(self, space):
         if self.w_objtype is not None:
@@ -32,7 +32,7 @@
             self.w_starttype.getname(space), objtype_name))
 
     def get(self, space, w_obj, w_type=None):
-        if self.w_self is None or space.is_w(w_obj, space.w_None):
+        if self.w_self is not None or space.is_w(w_obj, space.w_None):
             return self
         else:
             # if type(self) is W_Super:
@@ -54,10 +54,9 @@
                     return w_value
                 # Only pass 'obj' param if this is instance-mode super
                 # (see CPython sourceforge id #743627)
-                if self.w_self is self.w_objtype:
+                w_obj = self.w_self
+                if w_obj is None or w_obj is self.w_objtype:
                     w_obj = space.w_None
-                else:
-                    w_obj = self.w_self
                 return space.get_and_call_function(w_get, w_value,
                                                    w_obj, self.w_objtype)
         # fallback to object.__getattribute__()
diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py
--- a/pypy/module/__builtin__/test/test_descriptor.py
+++ b/pypy/module/__builtin__/test/test_descriptor.py
@@ -280,6 +280,14 @@
         assert repr(A()).endswith('>!')
         assert repr(super(A, A())) == "<super: <class 'A'>, <A object>>"
 
+    def test_super_get_corner_case(self):
+        class A(object):
+            pass
+        s = super(A, A())
+        assert s.__get__(42) is s
+        s = super(A)
+        assert s.__get__(None, "anything") is s
+
     def test_property_docstring(self):
         assert property.__doc__.startswith('property')
 

From pypy.commits at gmail.com  Mon Aug 14 13:15:46 2017
From: pypy.commits at gmail.com (arigo)
Date: Mon, 14 Aug 2017 10:15:46 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2629
Message-ID: <5991dac2.e681df0a.246b.7c79@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92147:8775f1e1bf8b
Date: 2017-08-14 19:14 +0200
http://bitbucket.org/pypy/pypy/changeset/8775f1e1bf8b/

Log:	Issue #2629

	Uninitialized instances of 'property' or 'super' should not cause
	segfaults

diff --git a/pypy/module/__builtin__/descriptor.py b/pypy/module/__builtin__/descriptor.py
--- a/pypy/module/__builtin__/descriptor.py
+++ b/pypy/module/__builtin__/descriptor.py
@@ -28,12 +28,19 @@
             objtype_name = "<%s object>" % self.w_objtype.getname(space)
         else:
             objtype_name = 'NULL'
+        if self.w_starttype is not None:
+            starttype_name = self.w_starttype.getname(space)
+        else:
+            starttype_name = 'NULL'
         return space.newtext("<super: <class '%s'>, %s>" % (
-            self.w_starttype.getname(space), objtype_name))
+            starttype_name, objtype_name))
 
     def get(self, space, w_obj, w_type=None):
         if self.w_self is not None or space.is_w(w_obj, space.w_None):
             return self
+        if self.w_starttype is None:
+            raise oefmt(space.w_TypeError,
+                "__get__(x) is invalid on an uninitialized instance of 'super'")
         else:
             # if type(self) is W_Super:
             #     XXX write a fast path for this common case
@@ -45,6 +52,7 @@
         # only use a special logic for bound super objects and not for
         # getting the __class__ of the super object itself.
         if self.w_objtype is not None and name != '__class__':
+            assert self.w_starttype is not None
             w_value = space.lookup_in_type_starting_at(self.w_objtype,
                                                        self.w_starttype,
                                                        name)
@@ -114,7 +122,11 @@
     _immutable_fields_ = ["w_fget", "w_fset", "w_fdel"]
 
     def __init__(self, space):
-        pass
+        self.w_fget = space.w_None
+        self.w_fset = space.w_None
+        self.w_fdel = space.w_None
+        self.w_doc = space.w_None
+        self.getter_doc = False
 
     @unwrap_spec(w_fget=WrappedDefault(None),
                  w_fset=WrappedDefault(None),
diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py
--- a/pypy/module/__builtin__/test/test_descriptor.py
+++ b/pypy/module/__builtin__/test/test_descriptor.py
@@ -419,3 +419,35 @@
         assert x.y == 42
         del x.x
         assert x.z == 42
+
+    def test_uninitialized_property(self):
+        p = property.__new__(property)
+        raises(AttributeError, p.__get__, 42)
+        raises(AttributeError, p.__set__, 42, None)
+        raises(AttributeError, p.__delete__, 42)
+        assert repr(p).startswith("<property object at ")
+        assert p.fget is p.fset is p.fdel is p.__doc__ is None
+        #
+        lst = []
+        p.deleter(lst.append).__delete__(42)
+        assert lst == [42]
+        #
+        lst = []
+        p.getter(lst.append).__get__(43)
+        assert lst == [43]
+        #
+        lst = []
+        p.setter(lambda x, y: lst.append((x, y))).__set__(44, 45)
+        assert lst == [(44, 45)]
+
+    def test_uninitialized_super(self):
+        s = super.__new__(super)
+        assert repr(s) == "<super: <class 'NULL'>, NULL>"
+        assert s.__thisclass__ is s.__self__ is s.__self_class__ is None
+        assert s.__get__(None, "anything") is s
+        raises(TypeError, s.__get__, 42)
+        raises(TypeError, s.__get__, int)
+        raises(TypeError, s.__get__, type(None))
+        raises(AttributeError, "s.abcde")
+        raises(AttributeError, "s.abcde = 42")
+        raises(AttributeError, "del s.abcde")

From pypy.commits at gmail.com  Mon Aug 14 13:21:58 2017
From: pypy.commits at gmail.com (arigo)
Date: Mon, 14 Aug 2017 10:21:58 -0700 (PDT)
Subject: [pypy-commit] pypy default: More tests
Message-ID: <5991dc36.50131c0a.8570b.f81f@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92148:bffaeddee9f5
Date: 2017-08-14 19:21 +0200
http://bitbucket.org/pypy/pypy/changeset/bffaeddee9f5/

Log:	More tests

diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py
--- a/pypy/module/__builtin__/test/test_descriptor.py
+++ b/pypy/module/__builtin__/test/test_descriptor.py
@@ -283,10 +283,18 @@
     def test_super_get_corner_case(self):
         class A(object):
             pass
-        s = super(A, A())
-        assert s.__get__(42) is s
-        s = super(A)
-        assert s.__get__(None, "anything") is s
+        s1 = super(A, A())
+        assert s1.__get__(42) is s1
+        assert s1.__get__(42, int) is s1
+        s2 = super(A)
+        assert s2.__get__(None, "anything") is s2
+        #
+        assert s1.__get__(None, "anything") is s1
+        raises(TypeError, s2.__get__, 42)
+        raises(TypeError, s2.__get__, 42, int)
+        a = A()
+        assert s2.__get__(a).__self__ is a
+        assert s1.__get__(a) is s1
 
     def test_property_docstring(self):
         assert property.__doc__.startswith('property')

From pypy.commits at gmail.com  Mon Aug 14 13:24:08 2017
From: pypy.commits at gmail.com (arigo)
Date: Mon, 14 Aug 2017 10:24:08 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <5991dcb8.0e951c0a.449d1.5944@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92149:90985eb27be1
Date: 2017-08-14 19:23 +0200
http://bitbucket.org/pypy/pypy/changeset/90985eb27be1/

Log:	hg merge default

diff --git a/pypy/module/__builtin__/descriptor.py b/pypy/module/__builtin__/descriptor.py
--- a/pypy/module/__builtin__/descriptor.py
+++ b/pypy/module/__builtin__/descriptor.py
@@ -21,24 +21,31 @@
 
         if space.is_none(w_obj_or_type):
             w_type = None  # unbound super object
-            w_obj_or_type = space.w_None
+            w_obj_or_type = None
         else:
             w_type = _super_check(space, w_starttype, w_obj_or_type)
         self.w_starttype = w_starttype
         self.w_objtype = w_type
-        self.w_self = w_obj_or_type
+        self.w_self = w_obj_or_type      # may be None
 
     def descr_repr(self, space):
         if self.w_objtype is not None:
             objtype_name = u"<%s object>" % self.w_objtype.getname(space)
         else:
             objtype_name = u'NULL'
+        if self.w_starttype is not None:
+            starttype_name = self.w_starttype.getname(space)
+        else:
+            starttype_name = u'NULL'
         return space.newunicode(u"<super: <class '%s'>, %s>" % (
-            self.w_starttype.getname(space), objtype_name))
+            starttype_name, objtype_name))
 
     def get(self, space, w_obj, w_type=None):
-        if self.w_self is None or space.is_w(w_obj, space.w_None):
+        if self.w_self is not None or space.is_w(w_obj, space.w_None):
             return self
+        if self.w_starttype is None:
+            raise oefmt(space.w_TypeError,
+                "__get__(x) is invalid on an uninitialized instance of 'super'")
         else:
             # if type(self) is W_Super:
             #     XXX write a fast path for this common case
@@ -50,6 +57,7 @@
         # only use a special logic for bound super objects and not for
         # getting the __class__ of the super object itself.
         if self.w_objtype is not None and name != '__class__':
+            assert self.w_starttype is not None
             w_value = space.lookup_in_type_starting_at(self.w_objtype,
                                                        self.w_starttype,
                                                        name)
@@ -59,10 +67,9 @@
                     return w_value
                 # Only pass 'obj' param if this is instance-mode super
                 # (see CPython sourceforge id #743627)
-                if self.w_self is self.w_objtype:
+                w_obj = self.w_self
+                if w_obj is None or w_obj is self.w_objtype:
                     w_obj = space.w_None
-                else:
-                    w_obj = self.w_self
                 return space.get_and_call_function(w_get, w_value,
                                                    w_obj, self.w_objtype)
         # fallback to object.__getattribute__()
@@ -160,7 +167,11 @@
     _immutable_fields_ = ["w_fget", "w_fset", "w_fdel"]
 
     def __init__(self, space):
-        pass
+        self.w_fget = space.w_None
+        self.w_fset = space.w_None
+        self.w_fdel = space.w_None
+        self.w_doc = space.w_None
+        self.getter_doc = False
 
     @unwrap_spec(w_fget=WrappedDefault(None),
                  w_fset=WrappedDefault(None),
diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py
--- a/pypy/module/__builtin__/test/test_descriptor.py
+++ b/pypy/module/__builtin__/test/test_descriptor.py
@@ -333,6 +333,22 @@
         assert repr(A()).endswith('>!')
         assert repr(super(A, A())) == "<super: <class 'A'>, <A object>>"
 
+    def test_super_get_corner_case(self):
+        class A(object):
+            pass
+        s1 = super(A, A())
+        assert s1.__get__(42) is s1
+        assert s1.__get__(42, int) is s1
+        s2 = super(A)
+        assert s2.__get__(None, "anything") is s2
+        #
+        assert s1.__get__(None, "anything") is s1
+        raises(TypeError, s2.__get__, 42)
+        raises(TypeError, s2.__get__, 42, int)
+        a = A()
+        assert s2.__get__(a).__self__ is a
+        assert s1.__get__(a) is s1
+
     def test_property_docstring(self):
         assert property.__doc__.startswith('property')
 
@@ -520,3 +536,35 @@
                 super()
         raises(RuntimeError, X().f)
         """
+
+    def test_uninitialized_property(self):
+        p = property.__new__(property)
+        raises(AttributeError, p.__get__, 42)
+        raises(AttributeError, p.__set__, 42, None)
+        raises(AttributeError, p.__delete__, 42)
+        assert repr(p).startswith("<property object at ")
+        assert p.fget is p.fset is p.fdel is p.__doc__ is None
+        #
+        lst = []
+        p.deleter(lst.append).__delete__(42)
+        assert lst == [42]
+        #
+        lst = []
+        p.getter(lst.append).__get__(43)
+        assert lst == [43]
+        #
+        lst = []
+        p.setter(lambda x, y: lst.append((x, y))).__set__(44, 45)
+        assert lst == [(44, 45)]
+
+    def test_uninitialized_super(self):
+        s = super.__new__(super)
+        assert repr(s) == "<super: <class 'NULL'>, NULL>"
+        assert s.__thisclass__ is s.__self__ is s.__self_class__ is None
+        assert s.__get__(None, "anything") is s
+        raises(TypeError, s.__get__, 42)
+        raises(TypeError, s.__get__, int)
+        raises(TypeError, s.__get__, type(None))
+        raises(AttributeError, "s.abcde")
+        raises(AttributeError, "s.abcde = 42")
+        raises(AttributeError, "del s.abcde")

From pypy.commits at gmail.com  Mon Aug 14 14:50:34 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 14 Aug 2017 11:50:34 -0700 (PDT)
Subject: [pypy-commit] pypy default: Do not leak reference when using
 PyModule_AddObject (reverts 5e783074ef82)
Message-ID: <5991f0fa.b785df0a.3b8b7.d4db@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: 
Changeset: r92150:9ac0b00b959a
Date: 2017-08-14 20:49 +0200
http://bitbucket.org/pypy/pypy/changeset/9ac0b00b959a/

Log:	Do not leak reference when using PyModule_AddObject (reverts
	5e783074ef82)

diff --git a/pypy/module/cpyext/src/modsupport.c b/pypy/module/cpyext/src/modsupport.c
--- a/pypy/module/cpyext/src/modsupport.c
+++ b/pypy/module/cpyext/src/modsupport.c
@@ -516,12 +516,10 @@
     return res;
 }
 
-/* returns -1 in case of error, 0 if a new key was added, 1 if the key
-   was already there (and replaced) */
-static int
-_PyModule_AddObject_NoConsumeRef(PyObject *m, const char *name, PyObject *o)
+int
+PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
 {
-    PyObject *dict, *prev;
+    PyObject *dict;
     if (!PyModule_Check(m)) {
         PyErr_SetString(PyExc_TypeError,
                     "PyModule_AddObject() needs module as first arg");
@@ -541,47 +539,32 @@
                      PyModule_GetName(m));
         return -1;
     }
-    prev = PyDict_GetItemString(dict, name);
     if (PyDict_SetItemString(dict, name, o))
         return -1;
-    return prev != NULL;
-}
-
-int
-PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
-{
-    int result = _PyModule_AddObject_NoConsumeRef(m, name, o);
-    /* XXX WORKAROUND for a common misusage of PyModule_AddObject:
-       for the common case of adding a new key, we don't consume a
-       reference, but instead just leak it away.  The issue is that
-       people generally don't realize that this function consumes a
-       reference, because on CPython the reference is still stored
-       on the dictionary. */
-    if (result != 0)
-        Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    Py_DECREF(o);
+    return 0;
 }
 
 int
 PyModule_AddIntConstant(PyObject *m, const char *name, long value)
 {
-    int result;
     PyObject *o = PyInt_FromLong(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
 
 int
 PyModule_AddStringConstant(PyObject *m, const char *name, const char *value)
 {
-    int result;
     PyObject *o = PyString_FromString(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
diff --git a/pypy/module/cpyext/test/test_capsule.py b/pypy/module/cpyext/test/test_capsule.py
--- a/pypy/module/cpyext/test/test_capsule.py
+++ b/pypy/module/cpyext/test/test_capsule.py
@@ -12,9 +12,6 @@
                  if (PyErr_Occurred()) return NULL;
                  module = PyImport_ImportModule("foo");
                  PyModule_AddObject(module, "_ptr", capsule);
-                 #ifdef PYPY_VERSION
-                   Py_DECREF(capsule);  /* XXX <--- anti-workaround */
-                 #endif
                  Py_DECREF(module);
                  if (PyErr_Occurred()) return NULL;
                  Py_RETURN_NONE;
diff --git a/pypy/module/cpyext/test/test_pycobject.py b/pypy/module/cpyext/test/test_pycobject.py
--- a/pypy/module/cpyext/test/test_pycobject.py
+++ b/pypy/module/cpyext/test/test_pycobject.py
@@ -3,8 +3,6 @@
 
 class AppTestStringObject(AppTestCpythonExtensionBase):
     def test_pycobject_import(self):
-        if self.runappdirect:
-            py.test.xfail('segfault')
         module = self.import_extension('foo', [
             ("set_ptr", "METH_O",
              """
@@ -15,7 +13,6 @@
                  if (PyErr_Occurred()) return NULL;
                  module = PyImport_ImportModule("foo");
                  PyModule_AddObject(module, "_ptr", pointer);
-                 Py_DECREF(pointer);  /* XXX <--- anti-workaround */
                  Py_DECREF(module);
                  if (PyErr_Occurred()) return NULL;
                  Py_RETURN_NONE;

From pypy.commits at gmail.com  Mon Aug 14 14:54:05 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 14 Aug 2017 11:54:05 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <5991f1cd.9398df0a.48c3d.a401@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92151:fee5f0e18b17
Date: 2017-08-14 20:52 +0200
http://bitbucket.org/pypy/pypy/changeset/fee5f0e18b17/

Log:	hg merge default

diff --git a/pypy/module/cpyext/src/modsupport.c b/pypy/module/cpyext/src/modsupport.c
--- a/pypy/module/cpyext/src/modsupport.c
+++ b/pypy/module/cpyext/src/modsupport.c
@@ -523,12 +523,10 @@
     return res;
 }
 
-/* returns -1 in case of error, 0 if a new key was added, 1 if the key
-   was already there (and replaced) */
-static int
-_PyModule_AddObject_NoConsumeRef(PyObject *m, const char *name, PyObject *o)
+int
+PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
 {
-    PyObject *dict, *prev;
+    PyObject *dict;
     if (!PyModule_Check(m)) {
         PyErr_SetString(PyExc_TypeError,
                     "PyModule_AddObject() needs module as first arg");
@@ -548,49 +546,34 @@
                      PyModule_GetName(m));
         return -1;
     }
-    prev = PyDict_GetItemString(dict, name);
     if (PyDict_SetItemString(dict, name, o))
         return -1;
-    return prev != NULL;
-}
-
-int
-PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
-{
-    int result = _PyModule_AddObject_NoConsumeRef(m, name, o);
-    /* XXX WORKAROUND for a common misusage of PyModule_AddObject:
-       for the common case of adding a new key, we don't consume a
-       reference, but instead just leak it away.  The issue is that
-       people generally don't realize that this function consumes a
-       reference, because on CPython the reference is still stored
-       on the dictionary. */
-    if (result != 0)
-        Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    Py_DECREF(o);
+    return 0;
 }
 
 int
 PyModule_AddIntConstant(PyObject *m, const char *name, long value)
 {
-    int result;
     PyObject *o = PyLong_FromLong(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
 
 int
 PyModule_AddStringConstant(PyObject *m, const char *name, const char *value)
 {
-    int result;
     PyObject *o = PyUnicode_FromString(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
 
 PyModuleDef*
diff --git a/pypy/module/cpyext/test/test_capsule.py b/pypy/module/cpyext/test/test_capsule.py
--- a/pypy/module/cpyext/test/test_capsule.py
+++ b/pypy/module/cpyext/test/test_capsule.py
@@ -12,9 +12,6 @@
                  if (PyErr_Occurred()) return NULL;
                  module = PyImport_ImportModule("foo");
                  PyModule_AddObject(module, "_ptr", capsule);
-                 #ifdef PYPY_VERSION
-                   Py_DECREF(capsule);  /* XXX <--- anti-workaround */
-                 #endif
                  Py_DECREF(module);
                  if (PyErr_Occurred()) return NULL;
                  Py_RETURN_NONE;

From pypy.commits at gmail.com  Mon Aug 14 14:54:07 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 14 Aug 2017 11:54:07 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <5991f1cf.6395df0a.f5534.9604@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92152:4babb75613f0
Date: 2017-08-14 20:53 +0200
http://bitbucket.org/pypy/pypy/changeset/4babb75613f0/

Log:	hg merge py3.5

diff --git a/lib-python/3/distutils/sysconfig_pypy.py b/lib-python/3/distutils/sysconfig_pypy.py
--- a/lib-python/3/distutils/sysconfig_pypy.py
+++ b/lib-python/3/distutils/sysconfig_pypy.py
@@ -81,6 +81,19 @@
     g['LIBDIR'] = os.path.join(sys.prefix, 'lib')
     g['VERSION'] = get_python_version()
 
+    if sys.platform[:6] == "darwin":
+        import platform
+        if platform.machine() == 'i386':
+            if platform.architecture()[0] == '32bit':
+                arch = 'i386'
+            else:
+                arch = 'x86_64'
+        else:
+            # just a guess
+            arch = platform.machine()
+        g['LDSHARED'] += ' -undefined dynamic_lookup'
+        g['CC'] += ' -arch %s' % (arch,)
+
     global _config_vars
     _config_vars = g
 
diff --git a/lib-python/3/test/test_pyexpat.py b/lib-python/3/test/test_pyexpat.py
--- a/lib-python/3/test/test_pyexpat.py
+++ b/lib-python/3/test/test_pyexpat.py
@@ -11,7 +11,7 @@
 from xml.parsers import expat
 from xml.parsers.expat import errors
 
-from test.support import sortdict
+from test.support import sortdict, impl_detail
 
 
 class SetAttributeTest(unittest.TestCase):
@@ -446,6 +446,7 @@
         self.assertEqual(os.path.basename(entry[0]), filename)
         self.assertEqual(entry[2], funcname)
 
+    @impl_detail("PyPy does not have pyexpat.c", pypy=False)
     def test_exception(self):
         parser = expat.ParserCreate()
         parser.StartElementHandler = self.StartElementHandler
diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -297,10 +297,7 @@
         line = line.rstrip('\n')
         if isinstance(line, unicode):
             return line # on py3k
-        try:
-            return unicode(line, ENCODING)
-        except UnicodeDecodeError:   # bah, silently fall back...
-            return unicode(line, 'utf-8', 'replace')
+        return unicode(line, 'utf-8', 'replace')
 
     def get_history_length(self):
         return self.saved_history_length
@@ -317,7 +314,7 @@
         # history item: we use \r\n instead of just \n.  If the history
         # file is passed to GNU readline, the extra \r are just ignored.
         history = self.get_reader().history
-        f = open(os.path.expanduser(filename), 'r')
+        f = open(os.path.expanduser(filename), 'r', encoding='utf-8')
         buffer = []
         for line in f:
             if line.endswith('\r\n'):
@@ -334,15 +331,12 @@
     def write_history_file(self, filename='~/.history'):
         maxlength = self.saved_history_length
         history = self.get_reader().get_trimmed_history(maxlength)
-        f = open(os.path.expanduser(filename), 'w')
+        f = open(os.path.expanduser(filename), 'w', encoding='utf-8')
         for entry in history:
             # if we are on py3k, we don't need to encode strings before
             # writing it to a file
             if isinstance(entry, unicode) and sys.version_info < (3,):
-                try:
-                    entry = entry.encode(ENCODING)
-                except UnicodeEncodeError:   # bah, silently fall back...
-                    entry = entry.encode('utf-8')
+                entry = entry.encode('utf-8')
             entry = entry.replace('\n', '\r\n')   # multiline history support
             f.write(entry + '\n')
         f.close()
diff --git a/pypy/module/__builtin__/descriptor.py b/pypy/module/__builtin__/descriptor.py
--- a/pypy/module/__builtin__/descriptor.py
+++ b/pypy/module/__builtin__/descriptor.py
@@ -21,24 +21,31 @@
 
         if space.is_none(w_obj_or_type):
             w_type = None  # unbound super object
-            w_obj_or_type = space.w_None
+            w_obj_or_type = None
         else:
             w_type = _super_check(space, w_starttype, w_obj_or_type)
         self.w_starttype = w_starttype
         self.w_objtype = w_type
-        self.w_self = w_obj_or_type
+        self.w_self = w_obj_or_type      # may be None
 
     def descr_repr(self, space):
         if self.w_objtype is not None:
             objtype_name = u"<%s object>" % self.w_objtype.getname(space)
         else:
             objtype_name = u'NULL'
+        if self.w_starttype is not None:
+            starttype_name = self.w_starttype.getname(space)
+        else:
+            starttype_name = u'NULL'
         return space.newunicode(u"<super: <class '%s'>, %s>" % (
-            self.w_starttype.getname(space), objtype_name))
+            starttype_name, objtype_name))
 
     def get(self, space, w_obj, w_type=None):
-        if self.w_self is None or space.is_w(w_obj, space.w_None):
+        if self.w_self is not None or space.is_w(w_obj, space.w_None):
             return self
+        if self.w_starttype is None:
+            raise oefmt(space.w_TypeError,
+                "__get__(x) is invalid on an uninitialized instance of 'super'")
         else:
             # if type(self) is W_Super:
             #     XXX write a fast path for this common case
@@ -50,6 +57,7 @@
         # only use a special logic for bound super objects and not for
         # getting the __class__ of the super object itself.
         if self.w_objtype is not None and name != '__class__':
+            assert self.w_starttype is not None
             w_value = space.lookup_in_type_starting_at(self.w_objtype,
                                                        self.w_starttype,
                                                        name)
@@ -59,10 +67,9 @@
                     return w_value
                 # Only pass 'obj' param if this is instance-mode super
                 # (see CPython sourceforge id #743627)
-                if self.w_self is self.w_objtype:
+                w_obj = self.w_self
+                if w_obj is None or w_obj is self.w_objtype:
                     w_obj = space.w_None
-                else:
-                    w_obj = self.w_self
                 return space.get_and_call_function(w_get, w_value,
                                                    w_obj, self.w_objtype)
         # fallback to object.__getattribute__()
@@ -160,7 +167,11 @@
     _immutable_fields_ = ["w_fget", "w_fset", "w_fdel"]
 
     def __init__(self, space):
-        pass
+        self.w_fget = space.w_None
+        self.w_fset = space.w_None
+        self.w_fdel = space.w_None
+        self.w_doc = space.w_None
+        self.getter_doc = False
 
     @unwrap_spec(w_fget=WrappedDefault(None),
                  w_fset=WrappedDefault(None),
diff --git a/pypy/module/__builtin__/test/test_descriptor.py b/pypy/module/__builtin__/test/test_descriptor.py
--- a/pypy/module/__builtin__/test/test_descriptor.py
+++ b/pypy/module/__builtin__/test/test_descriptor.py
@@ -333,6 +333,22 @@
         assert repr(A()).endswith('>!')
         assert repr(super(A, A())) == "<super: <class 'A'>, <A object>>"
 
+    def test_super_get_corner_case(self):
+        class A(object):
+            pass
+        s1 = super(A, A())
+        assert s1.__get__(42) is s1
+        assert s1.__get__(42, int) is s1
+        s2 = super(A)
+        assert s2.__get__(None, "anything") is s2
+        #
+        assert s1.__get__(None, "anything") is s1
+        raises(TypeError, s2.__get__, 42)
+        raises(TypeError, s2.__get__, 42, int)
+        a = A()
+        assert s2.__get__(a).__self__ is a
+        assert s1.__get__(a) is s1
+
     def test_property_docstring(self):
         assert property.__doc__.startswith('property')
 
@@ -520,3 +536,35 @@
                 super()
         raises(RuntimeError, X().f)
         """
+
+    def test_uninitialized_property(self):
+        p = property.__new__(property)
+        raises(AttributeError, p.__get__, 42)
+        raises(AttributeError, p.__set__, 42, None)
+        raises(AttributeError, p.__delete__, 42)
+        assert repr(p).startswith("<property object at ")
+        assert p.fget is p.fset is p.fdel is p.__doc__ is None
+        #
+        lst = []
+        p.deleter(lst.append).__delete__(42)
+        assert lst == [42]
+        #
+        lst = []
+        p.getter(lst.append).__get__(43)
+        assert lst == [43]
+        #
+        lst = []
+        p.setter(lambda x, y: lst.append((x, y))).__set__(44, 45)
+        assert lst == [(44, 45)]
+
+    def test_uninitialized_super(self):
+        s = super.__new__(super)
+        assert repr(s) == "<super: <class 'NULL'>, NULL>"
+        assert s.__thisclass__ is s.__self__ is s.__self_class__ is None
+        assert s.__get__(None, "anything") is s
+        raises(TypeError, s.__get__, 42)
+        raises(TypeError, s.__get__, int)
+        raises(TypeError, s.__get__, type(None))
+        raises(AttributeError, "s.abcde")
+        raises(AttributeError, "s.abcde = 42")
+        raises(AttributeError, "del s.abcde")
diff --git a/pypy/module/cpyext/src/modsupport.c b/pypy/module/cpyext/src/modsupport.c
--- a/pypy/module/cpyext/src/modsupport.c
+++ b/pypy/module/cpyext/src/modsupport.c
@@ -523,12 +523,10 @@
     return res;
 }
 
-/* returns -1 in case of error, 0 if a new key was added, 1 if the key
-   was already there (and replaced) */
-static int
-_PyModule_AddObject_NoConsumeRef(PyObject *m, const char *name, PyObject *o)
+int
+PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
 {
-    PyObject *dict, *prev;
+    PyObject *dict;
     if (!PyModule_Check(m)) {
         PyErr_SetString(PyExc_TypeError,
                     "PyModule_AddObject() needs module as first arg");
@@ -548,49 +546,34 @@
                      PyModule_GetName(m));
         return -1;
     }
-    prev = PyDict_GetItemString(dict, name);
     if (PyDict_SetItemString(dict, name, o))
         return -1;
-    return prev != NULL;
-}
-
-int
-PyModule_AddObject(PyObject *m, const char *name, PyObject *o)
-{
-    int result = _PyModule_AddObject_NoConsumeRef(m, name, o);
-    /* XXX WORKAROUND for a common misusage of PyModule_AddObject:
-       for the common case of adding a new key, we don't consume a
-       reference, but instead just leak it away.  The issue is that
-       people generally don't realize that this function consumes a
-       reference, because on CPython the reference is still stored
-       on the dictionary. */
-    if (result != 0)
-        Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    Py_DECREF(o);
+    return 0;
 }
 
 int
 PyModule_AddIntConstant(PyObject *m, const char *name, long value)
 {
-    int result;
     PyObject *o = PyLong_FromLong(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
 
 int
 PyModule_AddStringConstant(PyObject *m, const char *name, const char *value)
 {
-    int result;
     PyObject *o = PyUnicode_FromString(value);
     if (!o)
         return -1;
-    result = _PyModule_AddObject_NoConsumeRef(m, name, o);
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
     Py_DECREF(o);
-    return result < 0 ? -1 : 0;
+    return -1;
 }
 
 PyModuleDef*
diff --git a/pypy/module/cpyext/test/test_capsule.py b/pypy/module/cpyext/test/test_capsule.py
--- a/pypy/module/cpyext/test/test_capsule.py
+++ b/pypy/module/cpyext/test/test_capsule.py
@@ -12,9 +12,6 @@
                  if (PyErr_Occurred()) return NULL;
                  module = PyImport_ImportModule("foo");
                  PyModule_AddObject(module, "_ptr", capsule);
-                 #ifdef PYPY_VERSION
-                   Py_DECREF(capsule);  /* XXX <--- anti-workaround */
-                 #endif
                  Py_DECREF(module);
                  if (PyErr_Occurred()) return NULL;
                  Py_RETURN_NONE;
diff --git a/pypy/module/pyexpat/test/test_parser.py b/pypy/module/pyexpat/test/test_parser.py
--- a/pypy/module/pyexpat/test/test_parser.py
+++ b/pypy/module/pyexpat/test/test_parser.py
@@ -210,6 +210,30 @@
             p.ParseFile(fake_reader)
             assert fake_reader.read_count == 4
 
+
+    def test_exception(self):
+        """
+        lib-python/3/test_pyexpat.py:HandlerExceptionTest.test_exception port
+        without the fragile traceback inspection.
+        """
+        import pyexpat as expat
+
+        def StartElementHandler(name, attrs):
+            raise RuntimeError(name)
+
+        parser = expat.ParserCreate()
+        parser.StartElementHandler = StartElementHandler
+
+        try:
+            parser.Parse(b"<a><b><c/></b></a>", 1)
+            self.fail()
+        except RuntimeError as e:
+            assert e.args[0] == 'a', (
+                "Expected RuntimeError for element 'a', but" + \
+                " found %r" % e.args[0]
+            )
+
+
 class AppTestPyexpat2:
     spaceconfig = dict(usemodules=['_rawffi', 'pyexpat', 'itertools',
                                    '_socket', 'time', 'struct', 'binascii',
diff --git a/pypy/module/readline/test/test_readline.py b/pypy/module/readline/test/test_readline.py
new file mode 100644
--- /dev/null
+++ b/pypy/module/readline/test/test_readline.py
@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+
+class AppTestReadline:
+    spaceconfig = dict(usemodules={
+        'unicodedata', 'termios', 'select', 'signal', 'fcntl',
+        '_minimal_curses', 'faulthandler', '_socket', 'binascii',
+        '_posixsubprocess',
+    })
+
+    def test_nonascii_history(self):
+        import os, readline
+        TESTFN = "{}_{}_tmp".format("@test", os.getpid())
+
+        is_editline = readline.__doc__ and "libedit" in readline.__doc__
+
+        readline.clear_history()
+        try:
+            readline.add_history("entr&#233;e 1")
+        except UnicodeEncodeError as err:
+            skip("Locale cannot encode test data: " + format(err))
+        readline.add_history("entr&#233;e 2")
+        readline.replace_history_item(1, "entr&#233;e 22")
+        readline.write_history_file(TESTFN)
+        readline.clear_history()
+        readline.read_history_file(TESTFN)
+        if is_editline:
+            # An add_history() call seems to be required for get_history_
+            # item() to register items from the file
+            readline.add_history("dummy")
+        assert readline.get_history_item(1) ==  "entr&#233;e 1"
+        assert readline.get_history_item(2) == "entr&#233;e 22"

From pypy.commits at gmail.com  Mon Aug 14 15:30:52 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 14 Aug 2017 12:30:52 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Fix leaks in PyType_FromSpec
Message-ID: <5991fa6c.091f1c0a.a3c1d.082e@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92153:14b50017ab08
Date: 2017-08-14 21:29 +0200
http://bitbucket.org/pypy/pypy/changeset/14b50017ab08/

Log:	Fix leaks in PyType_FromSpec

diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -597,6 +597,7 @@
     if obj_pto.c_tp_flags & Py_TPFLAGS_HEAPTYPE:
         heaptype = rffi.cast(PyHeapTypeObject, obj)
         Py_DecRef(space, heaptype.c_ht_name)
+        Py_DecRef(space, heaptype.c_ht_qualname)
         Py_DecRef(space, base_pyo)
         _dealloc(space, obj)
 
@@ -925,8 +926,7 @@
         name = specname
     else:
         name = specname[dotpos + 1:]
-    res.c_ht_name = make_ref(
-        space, PyUnicode_FromString(space, rffi.str2charp(name)))
+    res.c_ht_name = make_ref(space, space.newtext(name))
     res.c_ht_qualname = res.c_ht_name
     incref(space, res.c_ht_qualname)
     typ.c_tp_name = spec.c_name

From pypy.commits at gmail.com  Tue Aug 15 12:29:25 2017
From: pypy.commits at gmail.com (exarkun)
Date: Tue, 15 Aug 2017 09:29:25 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Allow readline text insertion without
 preparation.
Message-ID: <59932165.959adf0a.34ae4.d660@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92154:9f2963dca939
Date: 2017-08-15 12:28 -0400
http://bitbucket.org/pypy/pypy/changeset/9f2963dca939/

Log:	Allow readline text insertion without preparation.

diff --git a/lib_pypy/pyrepl/reader.py b/lib_pypy/pyrepl/reader.py
--- a/lib_pypy/pyrepl/reader.py
+++ b/lib_pypy/pyrepl/reader.py
@@ -239,6 +239,10 @@
 
     def __init__(self, console):
         self.buffer = []
+        # Enable the use of `insert` without a `prepare` call - necessary to
+        # facilitate the tab completion hack implemented for
+        # <https://bugs.python.org/issue25660>.
+        self.pos = 0
         self.ps1 = "->> "
         self.ps2 = "/>> "
         self.ps3 = "|.. "
diff --git a/pypy/module/readline/test/test_readline.py b/pypy/module/readline/test/test_readline.py
--- a/pypy/module/readline/test/test_readline.py
+++ b/pypy/module/readline/test/test_readline.py
@@ -29,3 +29,14 @@
             readline.add_history("dummy")
         assert readline.get_history_item(1) ==  "entr&#233;e 1"
         assert readline.get_history_item(2) == "entr&#233;e 22"
+
+
+    def test_insert_text_leading_tab(self):
+        """
+        A literal tab can be inserted at the beginning of a line.
+
+        See <https://bugs.python.org/issue25660>
+        """
+        import readline
+        readline.insert_text("\t")
+        assert readline.get_line_buffer() == b"\t"

From pypy.commits at gmail.com  Tue Aug 15 13:35:22 2017
From: pypy.commits at gmail.com (exarkun)
Date: Tue, 15 Aug 2017 10:35:22 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Only expose OSError.winerror on Win32.
Message-ID: <599330da.c9a61c0a.6abc9.92c2@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92155:cce9378e5bb7
Date: 2017-08-15 13:34 -0400
http://bitbucket.org/pypy/pypy/changeset/cce9378e5bb7/

Log:	Only expose OSError.winerror on Win32.

diff --git a/pypy/module/exceptions/interp_exceptions.py b/pypy/module/exceptions/interp_exceptions.py
--- a/pypy/module/exceptions/interp_exceptions.py
+++ b/pypy/module/exceptions/interp_exceptions.py
@@ -636,6 +636,14 @@
 else:
     WINERROR_TO_ERRNO, DEFAULT_WIN32_ERRNO = {}, 22 # EINVAL
 
+if rwin32.WIN32:
+    _winerror_property = dict(
+        winerror = readwrite_attrproperty_w('w_winerror', W_OSError),
+    )
+else:
+    _winerror_property = dict()
+
+
 W_OSError.typedef = TypeDef(
     'OSError',
     W_Exception.typedef,
@@ -648,9 +656,9 @@
     strerror = readwrite_attrproperty_w('w_strerror', W_OSError),
     filename = readwrite_attrproperty_w('w_filename', W_OSError),
     filename2= readwrite_attrproperty_w('w_filename2',W_OSError),
-    winerror = readwrite_attrproperty_w('w_winerror', W_OSError),
     characters_written = GetSetProperty(W_OSError.descr_get_written,
                                         W_OSError.descr_set_written),
+    **_winerror_property
     )
 
 W_BlockingIOError = _new_exception(

From pypy.commits at gmail.com  Wed Aug 16 02:06:34 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 15 Aug 2017 23:06:34 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: Make the current stack limits a
 thread-local, too
Message-ID: <5993e0ea.97a0df0a.cf3ca.0274@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92158:13c93572cf88
Date: 2017-08-16 10:03 +0400
http://bitbucket.org/pypy/pypy/changeset/13c93572cf88/

Log:	Make the current stack limits a thread-local, too

diff --git a/rpython/translator/c/src/stack.c b/rpython/translator/c/src/stack.c
--- a/rpython/translator/c/src/stack.c
+++ b/rpython/translator/c/src/stack.c
@@ -10,16 +10,15 @@
    stack that grows downward here. */
 
 /* (stored in a struct to ensure that stack_end and stack_length are
-   close together; used e.g. by the ppc jit backend) */
-rpy_stacktoobig_t rpy_stacktoobig = {
-    NULL,             /* stack_end */
-    MAX_STACK_SIZE,   /* stack_length */
-    1                 /* report_error */
-};
+   close together; used e.g. by the ppc jit backend)
+   XXX this is no longer the case in the nogil-unsafe-2 branch */
+__thread rpy_stacktoobig_t rpy_stacktoobig;
+long rpy_stack_length = MAX_STACK_SIZE;
+
 
 void LL_stack_set_length_fraction(double fraction)
 {
-	rpy_stacktoobig.stack_length = (long)(MAX_STACK_SIZE * fraction);
+	rpy_stack_length = (long)(MAX_STACK_SIZE * fraction);
 }
 
 char LL_stack_too_big_slowpath(long current)
@@ -33,12 +32,14 @@
 	   if it is still 0 or if we later find a 'curptr' position
 	   that is above it.  The real stack_end pointer is stored in
 	   thread-local storage, but we try to minimize its overhead by
-	   keeping a local copy in rpy_stacktoobig.stack_end. */
+	   keeping a local copy in rpy_stacktoobig.stack_end.
+
+       XXX no point in having another thread-local copy */
 
 	OP_THREADLOCALREF_ADDR(tl);
 	tl1 = (struct pypy_threadlocal_s *)tl;
 	baseptr = tl1->stack_end;
-	max_stack_size = rpy_stacktoobig.stack_length;
+	max_stack_size = rpy_stack_length;
 	if (baseptr == NULL) {
 		/* first time we see this thread */
 	}
@@ -54,7 +55,7 @@
 			   the stack base must be revised */
 		}
 		else {	/* stack overflow (probably) */
-			return rpy_stacktoobig.report_error;
+			return !rpy_stacktoobig.dont_report_error;
 		}
 	}
 
diff --git a/rpython/translator/c/src/stack.h b/rpython/translator/c/src/stack.h
--- a/rpython/translator/c/src/stack.h
+++ b/rpython/translator/c/src/stack.h
@@ -22,11 +22,11 @@
 
 typedef struct {
     char *stack_end;
-    long stack_length;
-    char report_error;
+    char dont_report_error;
 } rpy_stacktoobig_t;
 
-RPY_EXTERN rpy_stacktoobig_t rpy_stacktoobig;
+RPY_EXTERN __thread rpy_stacktoobig_t rpy_stacktoobig;
+RPY_EXTERN long rpy_stack_length;
 
 RPY_EXTERN
 char LL_stack_too_big_slowpath(long);    /* returns 0 (ok) or 1 (too big) */
@@ -35,12 +35,12 @@
 
 /* some macros referenced from rpython.rlib.rstack */
 #define LL_stack_get_end() ((long)rpy_stacktoobig.stack_end)
-#define LL_stack_get_length() rpy_stacktoobig.stack_length
-#define LL_stack_get_end_adr()    ((long)&rpy_stacktoobig.stack_end)   /* JIT */
-#define LL_stack_get_length_adr() ((long)&rpy_stacktoobig.stack_length)/* JIT */
+#define LL_stack_get_length() rpy_stack_length
+#define LL_stack_get_end_adr()    FIXME ((long)&rpy_stacktoobig.stack_end)   /* JIT */
+#define LL_stack_get_length_adr()   ((long)&rpy_stack_length)     /* JIT */
 
-#define LL_stack_criticalcode_start()  (rpy_stacktoobig.report_error = 0)
-#define LL_stack_criticalcode_stop()   (rpy_stacktoobig.report_error = 1)
+#define LL_stack_criticalcode_start()  (rpy_stacktoobig.dont_report_error = 1)
+#define LL_stack_criticalcode_stop()   (rpy_stacktoobig.dont_report_error = 0)
 
 
 #ifdef __GNUC__

From pypy.commits at gmail.com  Wed Aug 16 02:06:29 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 15 Aug 2017 23:06:29 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: With nogil,
 it doesn't make sense to not have threads
Message-ID: <5993e0e5.cda2df0a.94a79.0e45@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92156:55fb6aff1863
Date: 2017-08-15 20:36 +0200
http://bitbucket.org/pypy/pypy/changeset/55fb6aff1863/

Log:	With nogil, it doesn't make sense to not have threads (and it fails
	translation right now, FIXME)

diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -23,7 +23,9 @@
 default_modules.update([
     "_codecs", "gc", "_weakref", "marshal", "errno", "imp", "math", "cmath",
     "_sre", "_pickle_support", "operator", "parser", "symbol", "token", "_ast",
-    "_io", "_random", "__pypy__", "_testing", "time"
+    "_io", "_random", "__pypy__", "_testing", "time",
+    "thread"     # XXX with nogil, it doesn't make sense to not have threads
+                 # (and it fails translation right now, FIXME)
 ])
 
 

From pypy.commits at gmail.com  Wed Aug 16 02:06:32 2017
From: pypy.commits at gmail.com (arigo)
Date: Tue, 15 Aug 2017 23:06:32 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: Fixes. Now the branch seems to
 "work" again
Message-ID: <5993e0e8.cc151c0a.143df.2f13@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92157:cd60a593d1b4
Date: 2017-08-16 09:44 +0400
http://bitbucket.org/pypy/pypy/changeset/cd60a593d1b4/

Log:	Fixes. Now the branch seems to "work" again

diff --git a/rpython/rlib/rgil.py b/rpython/rlib/rgil.py
--- a/rpython/rlib/rgil.py
+++ b/rpython/rlib/rgil.py
@@ -3,7 +3,7 @@
 from rpython.translator.tool.cbuild import ExternalCompilationInfo
 from rpython.rtyper.lltypesystem import lltype, llmemory, rffi
 from rpython.rtyper.extregistry import ExtRegistryEntry
-from rpython.rlib.objectmodel import not_rpython
+from rpython.rlib.objectmodel import not_rpython, we_are_translated
 
 # these functions manipulate directly the GIL, whose definition does not
 # escape the C code itself
@@ -122,21 +122,24 @@
     _gil_allocate()
 
 def release():
-    return
     # this function must not raise, in such a way that the exception
     # transformer knows that it cannot raise!
-    _gil_release()
-#release._gctransformer_hint_cannot_collect_ = True
-#release._dont_reach_me_in_del_ = True
+    if we_are_translated():
+        _gil_release()
+release._gctransformer_hint_cannot_collect_ = True
+release._dont_reach_me_in_del_ = True
 
 def acquire():
-    return
-    from rpython.rlib import rthread
-    _gil_acquire()
-    rthread.gc_thread_run()
-    _after_thread_switch()
-#acquire._gctransformer_hint_cannot_collect_ = True
-#acquire._dont_reach_me_in_del_ = True
+    ###XXX commented some lines out for nogil-unsafe-2
+    ###XXX but note that _gil_acquire() does not acquire any GIL there!
+
+    ###from rpython.rlib import rthread
+    if we_are_translated():
+        _gil_acquire()
+    ###rthread.gc_thread_run()
+    ###_after_thread_switch()
+acquire._gctransformer_hint_cannot_collect_ = True
+acquire._dont_reach_me_in_del_ = True
 
 # The _gctransformer_hint_cannot_collect_ hack is needed for
 # translations in which the *_external_call() functions are not inlined.
diff --git a/rpython/translator/c/src/thread.h b/rpython/translator/c/src/thread.h
--- a/rpython/translator/c/src/thread.h
+++ b/rpython/translator/c/src/thread.h
@@ -75,22 +75,21 @@
       SLOWPATH: signal "now at safepoint"; 111 -> 110
  */
 
-#define _RPyGilAcquire() /*do {                                           \
-        assert((__sync_fetch_and_add(                                   \
-                    &RPY_THREADLOCALREF_GET(synclock), 0)               \
-                & 0b001) == 0b0);                                       \
+
+/* in the nogil-unsafe-2 branch,
+      !! THIS IS NOT A GIL !!    It's the logic to do global safe-points
+*/
+#define _RPyGilAcquire() do {                                           \
     if (!__sync_bool_compare_and_swap(                                  \
             &RPY_THREADLOCALREF_GET(synclock), 0b100L, 0b101L))         \
         RPyGilAcquireSlowPath();                                        \
-            } while (0)*/
+            } while (0)
 
-#define _RPyGilRelease() /*do {                                           \
-        assert((__sync_fetch_and_add(                                   \
-                    &RPY_THREADLOCALREF_GET(synclock), 0) & 0b101) == 0b101); \
+#define _RPyGilRelease() do {                                           \
     if (!__sync_bool_compare_and_swap(                                  \
             &RPY_THREADLOCALREF_GET(synclock), 0b101L, 0b100L))         \
         RPyGilReleaseSlowPath();                                        \
-        } while (0)*/
+        } while (0)
 
 static inline long *_RPyFetchFastGil(void) {
     abort();
diff --git a/rpython/translator/c/src/thread_gil.c b/rpython/translator/c/src/thread_gil.c
--- a/rpython/translator/c/src/thread_gil.c
+++ b/rpython/translator/c/src/thread_gil.c
@@ -45,6 +45,10 @@
 
 void RPyGilAcquireSlowPath(void)
 {
+    assert((__sync_fetch_and_add(
+                &RPY_THREADLOCALREF_GET(synclock), 0)
+            & 0b001) == 0b0);
+
     /* wait until the master leaves the safe point */
     pthread_mutex_lock(&master_mutex);
 
@@ -63,6 +67,9 @@
 
 void RPyGilReleaseSlowPath(void)
 {
+    assert((__sync_fetch_and_add(
+                &RPY_THREADLOCALREF_GET(synclock), 0) & 0b101) == 0b101);
+
     pthread_mutex_lock(&sync_mutex);
     assert(RPY_THREADLOCALREF_GET(synclock) == 0b111L);
 

From pypy.commits at gmail.com  Wed Aug 16 04:44:50 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 16 Aug 2017 01:44:50 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: Don't decrement the ticker at
 all in the nogil-unsafe-2 branch
Message-ID: <59940602.d288df0a.80f5.261d@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92159:b03810fcb5c1
Date: 2017-08-16 10:43 +0200
http://bitbucket.org/pypy/pypy/changeset/b03810fcb5c1/

Log:	Don't decrement the ticker at all in the nogil-unsafe-2 branch

diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -436,6 +436,8 @@
         # hack to put the release-the-GIL one at the end of the list,
         # and the report-the-signals one at the start of the list.
         if use_bytecode_counter:
+            assert False, ("nogil-unsafe-2: shouldn't register a "
+                           "gil-releasing periodic action")
             self._periodic_actions.append(action)
             self.has_bytecode_counter = True
         else:
diff --git a/pypy/module/thread/gil.py b/pypy/module/thread/gil.py
--- a/pypy/module/thread/gil.py
+++ b/pypy/module/thread/gil.py
@@ -19,6 +19,7 @@
 
     def initialize(self, space):
         # add the GIL-releasing callback as an action on the space
+        return # XXX nogil-unsafe-2
         space.actionflag.register_periodic_action(GILReleaseAction(space),
                                                   use_bytecode_counter=True)
 

From pypy.commits at gmail.com  Wed Aug 16 15:57:05 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 16 Aug 2017 12:57:05 -0700 (PDT)
Subject: [pypy-commit] pypy default: refactor 9ddefd44f80d handling
 pre-existing exceptions, add tests, still not bulletproof
Message-ID: <5994a391.c98adf0a.54101.d9c2@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92160:40ee3c492e28
Date: 2017-08-16 22:56 +0300
http://bitbucket.org/pypy/pypy/changeset/40ee3c492e28/

Log:	refactor 9ddefd44f80d handling pre-existing exceptions, add tests,
	still not bulletproof

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1575,7 +1575,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
-        preexist_error = PyErr_Occurred(space) is not None
+        preexist_error = PyErr_Occurred(space)
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1597,18 +1597,20 @@
                     ret = None
 
             # Check for exception consistency
-            has_error = PyErr_Occurred(space) is not None
+            # XXX best attempt, will miss preexisting error that is
+            # overwritten with a new error of the same type
+            error = PyErr_Occurred(space)
+            has_new_error = (error is not None) and (error is not preexist_error)
             has_result = ret is not None
-            if not preexist_error:
-                if has_error and has_result:
-                    raise oefmt(space.w_SystemError,
-                                "An exception was set, but function returned a "
-                                "value")
-                elif not expect_null and not has_error and not has_result:
-                    raise oefmt(space.w_SystemError,
-                                "Function returned a NULL result without setting "
-                                "an exception")
-            if has_error:
+            if not expect_null and has_new_error and has_result:
+                raise oefmt(space.w_SystemError,
+                            "An exception was set, but function returned a "
+                            "value")
+            elif not expect_null and not has_new_error and not has_result:
+                raise oefmt(space.w_SystemError,
+                            "Function returned a NULL result without setting "
+                            "an exception")
+            elif has_new_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()
 
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -24,6 +24,10 @@
 def PyPy_Crash2(space):
     1/0
 
+ at api.cpython_api([api.PyObject], api.PyObject, result_is_ll=True)
+def PyPy_Noop(space, pyobj):
+    return pyobj
+
 class TestApi:
     def test_signature(self):
         common_functions = api.FUNCTIONS_BY_HEADER[api.pypy_decl]
@@ -685,6 +689,7 @@
         body = """
         PyAPI_FUNC(PyObject*) PyPy_Crash1(void);
         PyAPI_FUNC(long) PyPy_Crash2(void);
+        PyAPI_FUNC(PyObject*) PyPy_Noop(PyObject*);
         static PyObject* foo_crash1(PyObject* self, PyObject *args)
         {
             return PyPy_Crash1();
@@ -708,9 +713,27 @@
             int a = PyPy_Crash2();
             return PyFloat_FromDouble(a);
         }
+        static PyObject* foo_noop(PyObject* self, PyObject* args)
+        {
+            Py_INCREF(args);
+            return PyPy_Noop(args);
+        }
+        static PyObject* foo_set(PyObject* self, PyObject *args)
+        {
+            PyErr_SetString(PyExc_TypeError, "clear called with no error");
+            if (PyInt_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
+            return NULL;
+        }
         static PyObject* foo_clear(PyObject* self, PyObject *args)
         {
             PyErr_Clear();
+            if (PyInt_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
             return NULL;
         }
         static PyMethodDef methods[] = {
@@ -718,20 +741,53 @@
             { "crash2", foo_crash2, METH_NOARGS },
             { "crash3", foo_crash3, METH_NOARGS },
             { "crash4", foo_crash4, METH_NOARGS },
-            { "clear",  foo_clear, METH_NOARGS },
+            { "clear",  foo_clear,  METH_O },
+            { "set",    foo_set,    METH_O },
+            { "noop",   foo_noop,   METH_O },
             { NULL }
         };
         """
         module = self.import_module(name='foo', init=init, body=body)
+
         # uncaught interplevel exceptions are turned into SystemError
-        raises(SystemError, module.crash1)
-        raises(SystemError, module.crash2)
-        # caught exception
+        expected = "ZeroDivisionError('integer division or modulo by zero',)"
+        exc = raises(SystemError, module.crash1)
+        assert exc.value[0] == expected
+
+        exc = raises(SystemError, module.crash2)
+        assert exc.value[0] == expected
+
+        # caught exception, api.cpython_api return value works
         assert module.crash3() == -1
-        # An exception was set, but function returned a value
-        raises(SystemError, module.crash4)
-        # No exception set, but NULL returned
-        raises(SystemError, module.clear)
+
+        expected = 'An exception was set, but function returned a value'
+        # PyPy only incompatibility/extension
+        exc = raises(SystemError, module.crash4)
+        assert exc.value[0] == expected
+
+        # An exception was set by the previous call, it can pass
+        # cleanly through a call that doesn't check error state
+        assert module.noop(1) == 1
+
+        # clear the exception but return NULL, signalling an error
+        expected = 'Function returned a NULL result without setting an exception'
+        exc = raises(SystemError, module.clear, None)
+        assert exc.value[0] == expected
+
+        # Set an exception and return NULL
+        raises(TypeError, module.set, None)
+
+        # clear any exception and return a value 
+        assert module.clear(1) == 1
+
+        # Set an exception, but return non-NULL
+        expected = 'An exception was set, but function returned a value'
+        exc = raises(SystemError, module.set, 1)
+        assert exc.value[0] == expected
+        
+
+        # Clear the exception and return a value, all is OK
+        assert module.clear(1) == 1
 
     def test_new_exception(self):
         mod = self.import_extension('foo', [

From pypy.commits at gmail.com  Thu Aug 17 05:39:37 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 17 Aug 2017 02:39:37 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: Attempt to reduce false sharing
 between threads. Unclear results
Message-ID: <59956459.8583df0a.47840.55a5@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92161:e40f8472eb81
Date: 2017-08-17 11:38 +0200
http://bitbucket.org/pypy/pypy/changeset/e40f8472eb81/

Log:	Attempt to reduce false sharing between threads. Unclear results

diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py
--- a/rpython/memory/gc/incminimark.py
+++ b/rpython/memory/gc/incminimark.py
@@ -280,7 +280,7 @@
         # "cache_line_min" is used to round the actual thread-local
         # blocks to a cache line, to avoid pointless cache conflicts.
         "tl_block_size": 131072,
-        "cache_line_min": 256,  # why not 64b?
+        "cache_line_min": 128,  # two cache lines on x86
         }
 
     def __init__(self, config,
@@ -313,6 +313,7 @@
         self.max_heap_size_already_raised = False
         self.max_delta = float(r_uint(-1))
         self.max_number_of_pinned_objects = 0      # computed later
+        self.collecting_roots_in_nursery = False
         #
         self.card_page_indices = card_page_indices
         if self.card_page_indices > 0:
@@ -1983,13 +1984,20 @@
         # see them.
         use_jit_frame_stoppers = not any_pinned_object_from_earlier
         #
+        self.collecting_roots_in_nursery = True
         self.root_walker.walk_roots(
             callback,     # stack roots
             callback,     # static in prebuilt non-gc
             None,         # static in prebuilt gc
             is_minor=use_jit_frame_stoppers)
+        self.collecting_roots_in_nursery = False
         debug_stop("gc-minor-walkroots")
 
+    def collected_roots_for_one_thread(self):
+        if self.collecting_roots_in_nursery:
+            self.collect_oldrefs_to_nursery()
+            self.ac.force_non_sharing_by_dummy_allocation(self.cache_line_min)
+
     def collect_cardrefs_to_nursery(self):
         size_gc_header = self.gcheaderbuilder.size_gc_header
         oldlist = self.old_objects_with_cards_set
diff --git a/rpython/memory/gc/minimarkpage.py b/rpython/memory/gc/minimarkpage.py
--- a/rpython/memory/gc/minimarkpage.py
+++ b/rpython/memory/gc/minimarkpage.py
@@ -191,6 +191,30 @@
         return result
 
 
+    def force_non_sharing_by_dummy_allocation(self, alignment):
+        """Force a few bytes of memory to be lost, to ensure that
+        a CPU cache of size "alignment" would not cause false sharing
+        between objects allocated just before and objects allocated
+        just after the call to the present function.
+        """
+        size_class_max = self.small_request_threshold >> WORD_POWER_2
+        size_class = 1
+        while size_class <= size_class_max:
+            page = self.page_for_size[size_class]
+            if page != PAGE_NULL:
+                next_alloc = page.freeblock
+                allocation_start = llmemory.cast_ptr_to_adr(page) + self.hdrsize
+                if next_alloc != allocation_start:
+                    next_alloc = rffi.cast(lltype.Signed, next_alloc)
+                    rounded_up = (next_alloc + (alignment-1)) & ~(alignment-1)
+                    while next_alloc < rounded_up:
+                        self.malloc(size_class << WORD_POWER_2)
+                        if self.page_for_size[size_class] != page:
+                            break
+                        next_alloc = rffi.cast(lltype.Signed, page.freeblock)
+            size_class += 1
+
+
     def allocate_new_page(self, size_class):
         """Allocate and return a new page for the given size_class."""
         #
diff --git a/rpython/memory/gctransform/shadowstack.py b/rpython/memory/gctransform/shadowstack.py
--- a/rpython/memory/gctransform/shadowstack.py
+++ b/rpython/memory/gctransform/shadowstack.py
@@ -113,6 +113,7 @@
             debug_print("walk_stack", base, top)
             walk_stack_root(self.invoke_collect_stack_root, collect_stack_root,
                 None, base, top, is_minor=False)
+            self.gcdata.gc.collected_roots_for_one_thread()
 
         self._walk_thread_stack = walk_thread_stack
 
diff --git a/rpython/translator/c/src/threadlocal.c b/rpython/translator/c/src/threadlocal.c
--- a/rpython/translator/c/src/threadlocal.c
+++ b/rpython/translator/c/src/threadlocal.c
@@ -11,32 +11,36 @@
 #include "src/thread.h"
 
 
-/* this is a spin-lock that must be acquired around each doubly-linked-list
+/* this is a reentrant lock that must be acquired around each doubly-linked-list
    manipulation (because such manipulations can occur without the GIL) */
-static long pypy_threadlocal_lock = 0;
+static pthread_mutex_t _rpy_threadlocal_lock;
 
 static int check_valid(void);
 
-int _RPython_ThreadLocals_AcquireTimeout(int max_wait_iterations) {
-    while (1) {
-        long old_value = pypy_lock_test_and_set(&pypy_threadlocal_lock, 1);
-        if (old_value == 0)
-            break;
-        /* busy loop */
-        if (max_wait_iterations == 0)
-            return -1;
-        if (max_wait_iterations > 0)
-            --max_wait_iterations;
+static void do_check(int result)
+{
+    if (result != 0) {
+        fprintf(stderr, "threadlocal.c got an unexpected mutex error\n");
+        exit(1);
     }
+}
+
+static void init_lock(void)
+{
+    pthread_mutexattr_t attr;
+    do_check(pthread_mutexattr_init(&attr)
+          || pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE)
+          || pthread_mutex_init(&_rpy_threadlocal_lock, &attr)
+          || pthread_mutexattr_destroy(&attr));
+}
+
+void _RPython_ThreadLocals_Acquire(void) {
+    do_check(pthread_mutex_lock(&_rpy_threadlocal_lock));
     assert(check_valid());
-    return 0;
-}
-void _RPython_ThreadLocals_Acquire(void) {
-    _RPython_ThreadLocals_AcquireTimeout(-1);
 }
 void _RPython_ThreadLocals_Release(void) {
     assert(check_valid());
-    pypy_lock_release(&pypy_threadlocal_lock);
+    do_check(pthread_mutex_unlock(&_rpy_threadlocal_lock));
 }
 
 
@@ -73,6 +77,7 @@
 {
     /* assume that at most one pypy_threadlocal_s survived, the current one */
     struct pypy_threadlocal_s *cur;
+    init_lock();
     cur = (struct pypy_threadlocal_s *)_RPy_ThreadLocals_Get();
     if (cur && cur->ready == 42) {
         cur->next = cur->prev = &linkedlist_head;
@@ -81,7 +86,6 @@
     else {
         linkedlist_head.next = linkedlist_head.prev = &linkedlist_head;
     }
-    _RPython_ThreadLocals_Release();
 }
 
 
@@ -188,7 +192,7 @@
        a non-null thread-local value).  This is needed even in the
        case where we use '__thread' below, for the destructor.
     */
-    assert(pypy_threadlocal_lock == 0);
+    init_lock();
 #ifdef _WIN32
     pypy_threadlocal_key = TlsAlloc();
     if (pypy_threadlocal_key == TLS_OUT_OF_INDEXES)
diff --git a/rpython/translator/c/src/threadlocal.h b/rpython/translator/c/src/threadlocal.h
--- a/rpython/translator/c/src/threadlocal.h
+++ b/rpython/translator/c/src/threadlocal.h
@@ -21,7 +21,6 @@
 
 RPY_EXTERN void _RPython_ThreadLocals_Acquire(void);
 RPY_EXTERN void _RPython_ThreadLocals_Release(void);
-RPY_EXTERN int _RPython_ThreadLocals_AcquireTimeout(int max_wait_iterations);
 
 /* Must acquire/release the thread-local lock around a series of calls
    to the following function */

From pypy.commits at gmail.com  Thu Aug 17 08:28:55 2017
From: pypy.commits at gmail.com (exarkun)
Date: Thu, 17 Aug 2017 05:28:55 -0700 (PDT)
Subject: [pypy-commit] pypy default: Fix imp module test_suffixes so that it
 runs its intended assertions.
Message-ID: <59958c07.94891c0a.e35f9.09c6@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: 
Changeset: r92162:727bbd9f3a14
Date: 2017-08-17 08:28 -0400
http://bitbucket.org/pypy/pypy/changeset/727bbd9f3a14/

Log:	Fix imp module test_suffixes so that it runs its intended
	assertions.

	Also change one of the assertions to reflect the changed
	implementation for how source files are read (with U instead of r,
	now).

diff --git a/pypy/module/imp/test/test_app.py b/pypy/module/imp/test/test_app.py
--- a/pypy/module/imp/test/test_app.py
+++ b/pypy/module/imp/test/test_app.py
@@ -45,15 +45,17 @@
 
     def test_suffixes(self):
         for suffix, mode, type in self.imp.get_suffixes():
-            if mode == self.imp.PY_SOURCE:
+            if type == self.imp.PY_SOURCE:
                 assert suffix == '.py'
-                assert type == 'r'
-            elif mode == self.imp.PY_COMPILED:
+                assert mode == 'U'
+            elif type == self.imp.PY_COMPILED:
                 assert suffix in ('.pyc', '.pyo')
-                assert type == 'rb'
-            elif mode == self.imp.C_EXTENSION:
+                assert mode == 'rb'
+            elif type == self.imp.C_EXTENSION:
                 assert suffix.endswith(('.pyd', '.so'))
-                assert type == 'rb'
+                assert mode == 'rb'
+            else:
+                assert False, ("Unknown type", suffix, mode, type)
 
 
     def test_obscure_functions(self):

From pypy.commits at gmail.com  Thu Aug 17 10:29:28 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 17 Aug 2017 07:29:28 -0700 (PDT)
Subject: [pypy-commit] pypy nogil-unsafe-2: Really need a read-write lock
 here, not a reentrant mutex
Message-ID: <5995a848.6395df0a.214ab.fabc@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: nogil-unsafe-2
Changeset: r92163:3f8621fd99f2
Date: 2017-08-17 16:28 +0200
http://bitbucket.org/pypy/pypy/changeset/3f8621fd99f2/

Log:	Really need a read-write lock here, not a reentrant mutex

diff --git a/rpython/translator/c/src/threadlocal.c b/rpython/translator/c/src/threadlocal.c
--- a/rpython/translator/c/src/threadlocal.c
+++ b/rpython/translator/c/src/threadlocal.c
@@ -11,9 +11,10 @@
 #include "src/thread.h"
 
 
-/* this is a reentrant lock that must be acquired around each doubly-linked-list
-   manipulation (because such manipulations can occur without the GIL) */
-static pthread_mutex_t _rpy_threadlocal_lock;
+/* this is a read-write lock that must be acquired around each
+   doubly-linked-list access or manipulation (because such manipulations
+   can occur without the GIL) */
+static pthread_rwlockattr_t _rpy_threadlocal_lock;
 
 static int check_valid(void);
 
@@ -27,20 +28,23 @@
 
 static void init_lock(void)
 {
-    pthread_mutexattr_t attr;
-    do_check(pthread_mutexattr_init(&attr)
-          || pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE)
-          || pthread_mutex_init(&_rpy_threadlocal_lock, &attr)
-          || pthread_mutexattr_destroy(&attr));
+    do_check(pthread_rwlock_init(&_rpy_threadlocal_lock, NULL));
 }
 
-void _RPython_ThreadLocals_Acquire(void) {
-    do_check(pthread_mutex_lock(&_rpy_threadlocal_lock));
+void _RPython_ThreadLocals_Acquire(void)
+{
+    do_check(pthread_rwlock_wrlock(&_rpy_threadlocal_lock));
     assert(check_valid());
 }
-void _RPython_ThreadLocals_Release(void) {
+void _RPython_ThreadLocals_ReadOnlyAcquire(void)
+{
+    do_check(pthread_rwlock_rdlock(&_rpy_threadlocal_lock));
     assert(check_valid());
-    do_check(pthread_mutex_unlock(&_rpy_threadlocal_lock));
+}
+void _RPython_ThreadLocals_Release(void)
+{
+    assert(check_valid());
+    do_check(pthread_rwlock_unlock(&_rpy_threadlocal_lock));
 }
 
 
diff --git a/rpython/translator/c/src/threadlocal.h b/rpython/translator/c/src/threadlocal.h
--- a/rpython/translator/c/src/threadlocal.h
+++ b/rpython/translator/c/src/threadlocal.h
@@ -20,6 +20,7 @@
 RPY_EXTERN char *_RPython_ThreadLocals_Build(void);
 
 RPY_EXTERN void _RPython_ThreadLocals_Acquire(void);
+RPY_EXTERN void _RPython_ThreadLocals_ReadOnlyAcquire(void);
 RPY_EXTERN void _RPython_ThreadLocals_Release(void);
 
 /* Must acquire/release the thread-local lock around a series of calls
@@ -30,7 +31,7 @@
 /* will return the head of the list */
 RPY_EXTERN struct pypy_threadlocal_s *_RPython_ThreadLocals_Head();
 
-#define OP_THREADLOCALREF_ACQUIRE(r)   _RPython_ThreadLocals_Acquire()
+#define OP_THREADLOCALREF_ACQUIRE(r)   _RPython_ThreadLocals_ReadOnlyAcquire()
 #define OP_THREADLOCALREF_RELEASE(r)   _RPython_ThreadLocals_Release()
 #define OP_THREADLOCALREF_ENUM(p, r)   r = _RPython_ThreadLocals_Enum(p)
 

From pypy.commits at gmail.com  Thu Aug 17 13:23:28 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 17 Aug 2017 10:23:28 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Issue #2635
Message-ID: <5995d110.56b71c0a.2520f.f755@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92164:f2d17e80c68d
Date: 2017-08-17 19:22 +0200
http://bitbucket.org/pypy/pypy/changeset/f2d17e80c68d/

Log:	Issue #2635

	{datetime,date,time).replace() returns an instance of the specific
	subclass in CPython's C '_datetime' module.

diff --git a/lib-python/3/datetime.py b/lib-python/3/datetime.py
--- a/lib-python/3/datetime.py
+++ b/lib-python/3/datetime.py
@@ -810,7 +810,8 @@
             month = self._month
         if day is None:
             day = self._day
-        return date(year, month, day)
+        # PyPy fix: returns type(self)() instead of date()
+        return type(self)(year, month, day)
 
     # Comparisons of date objects with other.
 
@@ -1285,7 +1286,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return time(hour, minute, second, microsecond, tzinfo)
+        # PyPy fix: returns type(self)() instead of time()
+        return type(self)(hour, minute, second, microsecond, tzinfo)
 
     # Pickle support.
 
@@ -1497,7 +1499,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return datetime(year, month, day, hour, minute, second, microsecond,
+        # PyPy fix: returns type(self)() instead of datetime()
+        return type(self)(year, month, day, hour, minute, second, microsecond,
                         tzinfo)
 
     def astimezone(self, tz=None):

From pypy.commits at gmail.com  Thu Aug 17 14:05:44 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 17 Aug 2017 11:05:44 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2632
Message-ID: <5995daf8.89e61c0a.8e83c.18e7@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92165:141ba627dc5f
Date: 2017-08-17 20:05 +0200
http://bitbucket.org/pypy/pypy/changeset/141ba627dc5f/

Log:	Issue #2632

	Try even harder to get 'CDLL._handle' work like CPython

diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -361,17 +361,20 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self._pypy_dll = pypy_dll
+            handle = int(pypy_dll)
+            if _sys.maxint > 2 ** 32:
+                handle = int(handle)   # long -> int
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxint * 2 + 1))
-
+        return "<%s '%s', handle %x at %x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxint*2 + 1)),
+                id(self) & (_sys.maxint*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._handle.getaddressindll(name))
+        return self.from_address(dll._pypy_dll.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._handle
+        cdll = self.dll._pypy_dll
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
@@ -43,6 +43,12 @@
                 cdll.LoadLibrary(lib)
                 CDLL(lib)
 
+    def test__handle(self):
+        lib = find_library("c")
+        if lib:
+            cdll = CDLL(lib)
+            assert type(cdll._handle) in (int, long)
+
     if os.name in ("nt", "ce"):
         def test_load_library(self):
             if is_resource_enabled("printing"):

From pypy.commits at gmail.com  Fri Aug 18 10:28:53 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 18 Aug 2017 07:28:53 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <5996f9a5.919bdf0a.c7a5.e09c@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92166:90593ef2b17c
Date: 2017-08-18 16:28 +0200
http://bitbucket.org/pypy/pypy/changeset/90593ef2b17c/

Log:	hg merge default

diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -361,17 +361,20 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self._pypy_dll = pypy_dll
+            handle = int(pypy_dll)
+            if _sys.maxint > 2 ** 32:
+                handle = int(handle)   # long -> int
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxint * 2 + 1))
-
+        return "<%s '%s', handle %x at %x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxint*2 + 1)),
+                id(self) & (_sys.maxint*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._handle.getaddressindll(name))
+        return self.from_address(dll._pypy_dll.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._handle
+        cdll = self.dll._pypy_dll
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1632,7 +1632,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
-        preexist_error = PyErr_Occurred(space) is not None
+        preexist_error = PyErr_Occurred(space)
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1656,17 +1656,20 @@
                 has_result = ret is not None
 
             # Check for exception consistency
-            has_error = PyErr_Occurred(space) is not None
-            if not preexist_error:
-                if has_error and has_result:
-                    raise oefmt(space.w_SystemError,
-                                "An exception was set, but function returned a "
-                                "value")
-                elif not expect_null and not has_error and not has_result:
-                    raise oefmt(space.w_SystemError,
-                                "Function returned a NULL result without setting "
-                                "an exception")
-            if has_error:
+            # XXX best attempt, will miss preexisting error that is
+            # overwritten with a new error of the same type
+            error = PyErr_Occurred(space)
+            has_new_error = (error is not None) and (error is not preexist_error)
+            has_result = ret is not None
+            if not expect_null and has_new_error and has_result:
+                raise oefmt(space.w_SystemError,
+                            "An exception was set, but function returned a "
+                            "value")
+            elif not expect_null and not has_new_error and not has_result:
+                raise oefmt(space.w_SystemError,
+                            "Function returned a NULL result without setting "
+                            "an exception")
+            elif has_new_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()
 
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -24,6 +24,10 @@
 def PyPy_Crash2(space):
     1/0
 
+ at api.cpython_api([api.PyObject], api.PyObject, result_is_ll=True)
+def PyPy_Noop(space, pyobj):
+    return pyobj
+
 class TestApi:
     def test_signature(self):
         common_functions = api.FUNCTIONS_BY_HEADER[api.pypy_decl]
@@ -665,6 +669,7 @@
         body = """
         PyAPI_FUNC(PyObject*) PyPy_Crash1(void);
         PyAPI_FUNC(long) PyPy_Crash2(void);
+        PyAPI_FUNC(PyObject*) PyPy_Noop(PyObject*);
         static PyObject* foo_crash1(PyObject* self, PyObject *args)
         {
             return PyPy_Crash1();
@@ -688,9 +693,27 @@
             int a = PyPy_Crash2();
             return PyFloat_FromDouble(a);
         }
+        static PyObject* foo_noop(PyObject* self, PyObject* args)
+        {
+            Py_INCREF(args);
+            return PyPy_Noop(args);
+        }
+        static PyObject* foo_set(PyObject* self, PyObject *args)
+        {
+            PyErr_SetString(PyExc_TypeError, "clear called with no error");
+            if (PyLong_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
+            return NULL;
+        }
         static PyObject* foo_clear(PyObject* self, PyObject *args)
         {
             PyErr_Clear();
+            if (PyLong_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
             return NULL;
         }
         static PyMethodDef methods[] = {
@@ -698,7 +721,9 @@
             { "crash2", foo_crash2, METH_NOARGS },
             { "crash3", foo_crash3, METH_NOARGS },
             { "crash4", foo_crash4, METH_NOARGS },
-            { "clear",  foo_clear, METH_NOARGS },
+            { "clear",  foo_clear,  METH_O },
+            { "set",    foo_set,    METH_O },
+            { "noop",   foo_noop,   METH_O },
             { NULL }
         };
         static struct PyModuleDef moduledef = {
@@ -710,15 +735,46 @@
         };
         """
         module = self.import_module(name='foo', body=body)
+
         # uncaught interplevel exceptions are turned into SystemError
-        raises(SystemError, module.crash1)
-        raises(SystemError, module.crash2)
-        # caught exception
+        expected = "ZeroDivisionError('integer division or modulo by zero',)"
+        exc = raises(SystemError, module.crash1)
+        assert exc.value[0] == expected
+
+        exc = raises(SystemError, module.crash2)
+        assert exc.value[0] == expected
+
+        # caught exception, api.cpython_api return value works
         assert module.crash3() == -1
-        # An exception was set, but function returned a value
-        raises(SystemError, module.crash4)
-        # No exception set, but NULL returned
-        raises(SystemError, module.clear)
+
+        expected = 'An exception was set, but function returned a value'
+        # PyPy only incompatibility/extension
+        exc = raises(SystemError, module.crash4)
+        assert exc.value[0] == expected
+
+        # An exception was set by the previous call, it can pass
+        # cleanly through a call that doesn't check error state
+        assert module.noop(1) == 1
+
+        # clear the exception but return NULL, signalling an error
+        expected = 'Function returned a NULL result without setting an exception'
+        exc = raises(SystemError, module.clear, None)
+        assert exc.value[0] == expected
+
+        # Set an exception and return NULL
+        raises(TypeError, module.set, None)
+
+        # clear any exception and return a value
+        assert module.clear(1) == 1
+
+        # Set an exception, but return non-NULL
+        expected = 'An exception was set, but function returned a value'
+        exc = raises(SystemError, module.set, 1)
+        assert exc.value[0] == expected
+
+
+        # Clear the exception and return a value, all is OK
+        assert module.clear(1) == 1
 
     def test_new_exception(self):
         mod = self.import_extension('foo', [
diff --git a/pypy/module/imp/test/test_app.py b/pypy/module/imp/test/test_app.py
--- a/pypy/module/imp/test/test_app.py
+++ b/pypy/module/imp/test/test_app.py
@@ -81,15 +81,17 @@
     def test_suffixes(self):
         import imp
         for suffix, mode, type in imp.get_suffixes():
-            if mode == imp.PY_SOURCE:
+            if type == imp.PY_SOURCE:
                 assert suffix == '.py'
-                assert type == 'r'
-            elif mode == imp.PY_COMPILED:
+                assert mode == 'U'
+            elif type == imp.PY_COMPILED:
                 assert suffix in ('.pyc', '.pyo')
-                assert type == 'rb'
-            elif mode == imp.C_EXTENSION:
+                assert mode == 'rb'
+            elif type == imp.C_EXTENSION:
                 assert suffix.endswith(('.pyd', '.so'))
-                assert type == 'rb'
+                assert mode == 'rb'
+            else:
+                assert False, ("Unknown type", suffix, mode, type)
 
     def test_ext_suffixes(self):
         import _imp
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
@@ -43,6 +43,12 @@
                 cdll.LoadLibrary(lib)
                 CDLL(lib)
 
+    def test__handle(self):
+        lib = find_library("c")
+        if lib:
+            cdll = CDLL(lib)
+            assert type(cdll._handle) in (int, long)
+
     if os.name in ("nt", "ce"):
         def test_load_library(self):
             if is_resource_enabled("printing"):

From pypy.commits at gmail.com  Fri Aug 18 11:23:03 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 18 Aug 2017 08:23:03 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2636
Message-ID: <59970657.44051c0a.a9ffb.abce@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92167:eaa386e4970b
Date: 2017-08-18 17:22 +0200
http://bitbucket.org/pypy/pypy/changeset/eaa386e4970b/

Log:	Issue #2636

	Add the same workaround as CPython for os.write() on Windows on a
	tty

diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py
--- a/rpython/rlib/rposix.py
+++ b/rpython/rlib/rposix.py
@@ -205,6 +205,18 @@
         if not is_valid_fd(fd):
             from errno import EBADF
             raise OSError(EBADF, 'Bad file descriptor')
+
+    def _bound_for_write(fd, count):
+        if count > 32767 and c_isatty(fd):
+            # CPython Issue #11395, PyPy Issue #2636: the Windows console
+            # returns an error (12: not enough space error) on writing into
+            # stdout if stdout mode is binary and the length is greater than
+            # 66,000 bytes (or less, depending on heap usage).  Can't easily
+            # test that, because we need 'fd' to be non-redirected...
+            count = 32767
+        elif count > 0x7fffffff:
+            count = 0x7fffffff
+        return count
 else:
     def is_valid_fd(fd):
         return 1
@@ -213,6 +225,9 @@
     def validate_fd(fd):
         pass
 
+    def _bound_for_write(fd, count):
+        return count
+
 def closerange(fd_low, fd_high):
     # this behaves like os.closerange() from Python 2.6.
     for fd in xrange(fd_low, fd_high):
@@ -449,6 +464,7 @@
 def write(fd, data):
     count = len(data)
     validate_fd(fd)
+    count = _bound_for_write(fd, count)
     with rffi.scoped_nonmovingbuffer(data) as buf:
         return handle_posix_error('write', c_write(fd, buf, count))
 

From pypy.commits at gmail.com  Fri Aug 18 11:31:16 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 18 Aug 2017 08:31:16 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: The `U` mode is deprecated on Python 3;
 use r instead.
Message-ID: <59970844.03081c0a.5d75b.7a24@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92168:6ee00ca3924e
Date: 2017-08-18 11:30 -0400
http://bitbucket.org/pypy/pypy/changeset/6ee00ca3924e/

Log:	The `U` mode is deprecated on Python 3; use r instead.

diff --git a/pypy/module/imp/test/test_app.py b/pypy/module/imp/test/test_app.py
--- a/pypy/module/imp/test/test_app.py
+++ b/pypy/module/imp/test/test_app.py
@@ -83,7 +83,7 @@
         for suffix, mode, type in imp.get_suffixes():
             if type == imp.PY_SOURCE:
                 assert suffix == '.py'
-                assert mode == 'U'
+                assert mode == 'r'
             elif type == imp.PY_COMPILED:
                 assert suffix in ('.pyc', '.pyo')
                 assert mode == 'rb'

From pypy.commits at gmail.com  Fri Aug 18 15:21:30 2017
From: pypy.commits at gmail.com (mjacob)
Date: Fri, 18 Aug 2017 12:21:30 -0700 (PDT)
Subject: [pypy-commit] pypy py3.6: hg merge py3.5
Message-ID: <59973e3a.1a9cdf0a.9ec2.adaf@mx.google.com>

Author: Manuel Jacob <me at manueljacob.de>
Branch: py3.6
Changeset: r92170:3cbf980069f5
Date: 2017-07-22 03:37 +0200
http://bitbucket.org/pypy/pypy/changeset/3cbf980069f5/

Log:	hg merge py3.5

diff --git a/lib_pypy/_cffi_ssl/README.md b/lib_pypy/_cffi_ssl/README.md
--- a/lib_pypy/_cffi_ssl/README.md
+++ b/lib_pypy/_cffi_ssl/README.md
@@ -5,9 +5,15 @@
 it renames the compiled shared object to _pypy_openssl.so (which means
 that cryptography can ship their own cffi backend)
 
-NOTE: currently, we have changed ``_cffi_src/openssl/callbacks.py`` to
-not rely on the CPython C API, and ``_cffi_src/utils.py`` for issue #2575
-(29c9a89359e4).  (The first change is now backported.)
+NOTE: currently, we have the following changes:
+
+* ``_cffi_src/openssl/callbacks.py`` to not rely on the CPython C API
+  (this change is now backported)
+
+* ``_cffi_src/utils.py`` for issue #2575 (29c9a89359e4)
+
+* ``_cffi_src/openssl/x509_vfy.py`` for issue #2605 (ca4d0c90f5a1)
+
 
 # Tests?
 
diff --git a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
--- a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
+++ b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
@@ -221,10 +221,16 @@
 static const long X509_V_ERR_SUITE_B_INVALID_SIGNATURE_ALGORITHM = 0;
 static const long X509_V_ERR_SUITE_B_LOS_NOT_ALLOWED = 0;
 static const long X509_V_ERR_SUITE_B_CANNOT_SIGN_P_384_WITH_P_256 = 0;
+#ifndef X509_V_ERR_HOSTNAME_MISMATCH
 static const long X509_V_ERR_HOSTNAME_MISMATCH = 0;
+#endif
+#ifndef X509_V_ERR_EMAIL_MISMATCH
 static const long X509_V_ERR_EMAIL_MISMATCH = 0;
+#endif
+#ifndef X509_V_ERR_IP_ADDRESS_MISMATCH
 static const long X509_V_ERR_IP_ADDRESS_MISMATCH = 0;
 #endif
+#endif
 
 /* OpenSSL 1.0.2beta2+ verification parameters */
 #if CRYPTOGRAPHY_OPENSSL_102BETA2_OR_GREATER && \
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -462,6 +462,12 @@
         fwrite(buf, 1, count, fp)
     return 0
 
+ at cts.decl("""
+    Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t defaultvalue)""",
+    error=-1)
+def PyObject_LengthHint(space, w_o, defaultvalue):
+    return space.length_hint(w_o, defaultvalue)
+
 @cpython_api([lltype.Signed], lltype.Void)
 def _PyPyGC_AddMemoryPressure(space, report):
     from rpython.rlib import rgc
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -349,6 +349,27 @@
         assert type(module.asbytes(sub1(b''))) is bytes
         assert type(module.asbytes(sub2(b''))) is sub2
 
+    def test_LengthHint(self):
+        import operator
+        class WithLen:
+            def __len__(self):
+                return 1
+            def __length_hint__(self):
+                return 42
+        class NoLen:
+            def __length_hint__(self):
+                return 2
+        module = self.import_extension('test_LengthHint', [
+            ('length_hint', 'METH_VARARGS',
+             """
+                 PyObject *obj = PyTuple_GET_ITEM(args, 0);
+                 Py_ssize_t i = PyLong_AsSsize_t(PyTuple_GET_ITEM(args, 1));
+                 return PyLong_FromSsize_t(PyObject_LengthHint(obj, i));
+             """)])
+        assert module.length_hint(WithLen(), 5) == operator.length_hint(WithLen(), 5) == 1
+        assert module.length_hint(NoLen(), 5) == operator.length_hint(NoLen(), 5) == 2
+        assert module.length_hint(object(), 5) == operator.length_hint(object(), 5) == 5
+
     def test_add_memory_pressure(self):
         self.reset_memory_pressure()    # for the potential skip
         module = self.import_extension('foo', [
@@ -528,4 +549,3 @@
     Py_RETURN_NONE;
                  """)])
         assert module.release() is None
-
diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -365,8 +365,8 @@
         characters, all remaining cased characters have lowercase.
         """
 
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         """B.translate(table[, deletechars]) -> copy of B
 
         Return a copy of the string B, where all characters occurring
diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py
--- a/pypy/objspace/std/stringmethods.py
+++ b/pypy/objspace/std/stringmethods.py
@@ -742,8 +742,8 @@
     DEFAULT_NOOP_TABLE = ''.join([chr(i) for i in range(256)])
 
     # for bytes and bytearray, overridden by unicode
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         if space.is_w(w_table, space.w_None):
             table = self.DEFAULT_NOOP_TABLE
         else:
@@ -753,7 +753,7 @@
                             "translation table must be 256 characters long")
 
         string = self._val(space)
-        deletechars = self._op_val(space, w_deletechars)
+        deletechars = self._op_val(space, w_delete)
         if len(deletechars) == 0:
             buf = self._builder(len(string))
             for char in string:
diff --git a/rpython/translator/c/genc.py b/rpython/translator/c/genc.py
--- a/rpython/translator/c/genc.py
+++ b/rpython/translator/c/genc.py
@@ -382,6 +382,80 @@
         if self.config.translation.profopt:
             if self.config.translation.profoptargs is None:
                 raise Exception("No profoptargs specified, neither in the command line, nor in the target. If the target is not PyPy, please specify profoptargs")
+
+            # Set the correct PGO params based on OS and CC
+            profopt_gen_flag = ""
+            profopt_use_flag = ""
+            profopt_merger = ""
+            profopt_file = ""
+            llvm_profdata = ""
+
+            cc = self.translator.platform.cc
+
+            # Locate llvm-profdata
+            if "clang" in cc:
+                clang_bin = cc
+                path = os.environ.get("PATH").split(":")
+                profdata_found = False
+
+                # Try to find it in $PATH (Darwin and Linux)
+                for dir in path:
+                    bin = "%s/llvm-profdata" % dir
+                    if os.path.isfile(bin):
+                        llvm_profdata = bin
+                        profdata_found = True
+                        break
+
+                # If not found, try to find it where clang is actually installed (Darwin and Linux)
+                if not profdata_found:
+                    # If the full path is not given, find where clang is located
+                    if not os.path.isfile(clang_bin):
+                        for dir in path:
+                            bin = "%s/%s" % (dir, cc)
+                            if os.path.isfile(bin):
+                                clang_bin = bin
+                                break
+                    # Some systems install clang elsewhere as a symlink to the real path,
+                    # which is where the related llvm tools are located.
+                    if os.path.islink(clang_bin):
+                        clang_bin = os.path.realpath(clang_bin)  # the real clang binary
+                    # llvm-profdata must be in the same directory as clang
+                    llvm_profdata = "%s/llvm-profdata" % os.path.dirname(clang_bin)
+                    profdata_found = os.path.isfile(llvm_profdata)
+
+                # If not found, and Darwin is used, try to find it in the development environment
+                # More: https://apple.stackexchange.com/questions/197053/
+                if not profdata_found and sys.platform == 'darwin':
+                    code = os.system("/usr/bin/xcrun -find llvm-profdata 2>/dev/null")
+                    if code == 0:
+                        llvm_profdata = "/usr/bin/xcrun llvm-profdata"
+                        profdata_found = True
+
+                # If everything failed, throw Exception, sorry
+                if not profdata_found:
+                    raise Exception(
+                        "Error: Cannot perform profopt build because llvm-profdata was not found in PATH. "
+                        "Please add it to PATH and run the translation again.")
+
+            # Set the PGO flags
+            if "clang" in cc:
+                # Any changes made here should be reflected in the GCC+Darwin case below
+                profopt_gen_flag = "-fprofile-instr-generate"
+                profopt_use_flag = "-fprofile-instr-use=code.profclangd"
+                profopt_merger = "%s merge -output=code.profclangd *.profclangr" % llvm_profdata
+                profopt_file = 'LLVM_PROFILE_FILE="code-%p.profclangr"'
+            elif "gcc" in cc:
+                if sys.platform == 'darwin':
+                    profopt_gen_flag = "-fprofile-instr-generate"
+                    profopt_use_flag = "-fprofile-instr-use=code.profclangd"
+                    profopt_merger = "%s merge -output=code.profclangd *.profclangr" % llvm_profdata
+                    profopt_file = 'LLVM_PROFILE_FILE="code-%p.profclangr"'
+                else:
+                    profopt_gen_flag = "-fprofile-generate"
+                    profopt_use_flag = "-fprofile-use -fprofile-correction"
+                    profopt_merger = "true"
+                    profopt_file = ""
+
             if self.config.translation.shared:
                 mk.rule('$(PROFOPT_TARGET)', '$(TARGET) main.o',
                          '$(CC_LINK) $(LDFLAGS_LINK) main.o -L. -l$(SHARED_IMPORT_LIB) -o $@ $(RPATH_FLAGS) -lgcov')
@@ -390,10 +464,11 @@
 
             rules.append(
                 ('profopt', '', [
-                    '$(MAKE) CFLAGS="-fprofile-generate -fPIC $(CFLAGS) -fno-lto"  LDFLAGS="-fprofile-generate $(LDFLAGS) -fno-lto" $(PROFOPT_TARGET)',
-                    '%s %s ' % (exe_name, self.config.translation.profoptargs),
+                    '$(MAKE) CFLAGS="%s -fPIC $(CFLAGS)"  LDFLAGS="%s $(LDFLAGS)" $(PROFOPT_TARGET)' % (profopt_gen_flag, profopt_gen_flag),
+                    '%s %s %s ' % (profopt_file, exe_name, self.config.translation.profoptargs),
+                    '%s' % (profopt_merger),
                     '$(MAKE) clean_noprof',
-                    '$(MAKE) CFLAGS="-fprofile-use -fprofile-correction -fPIC $(CFLAGS) -fno-lto"  LDFLAGS="-fprofile-use $(LDFLAGS) -fno-lto" $(PROFOPT_TARGET)',
+                    '$(MAKE) CFLAGS="%s -fPIC $(CFLAGS)"  LDFLAGS="%s $(LDFLAGS)" $(PROFOPT_TARGET)' % (profopt_use_flag, profopt_use_flag),
                 ]))
 
         for rule in rules:

From pypy.commits at gmail.com  Fri Aug 18 15:21:32 2017
From: pypy.commits at gmail.com (mjacob)
Date: Fri, 18 Aug 2017 12:21:32 -0700 (PDT)
Subject: [pypy-commit] pypy py3.6: Test and fix type of result of <subclass
 of bytes/bytearray>.fromhex().
Message-ID: <59973e3c.c98adf0a.54101.7eb6@mx.google.com>

Author: Manuel Jacob <me at manueljacob.de>
Branch: py3.6
Changeset: r92171:ac3e33369ba0
Date: 2017-07-22 03:54 +0200
http://bitbucket.org/pypy/pypy/changeset/ac3e33369ba0/

Log:	Test and fix type of result of <subclass of
	bytes/bytearray>.fromhex().

diff --git a/pypy/objspace/std/bytearrayobject.py b/pypy/objspace/std/bytearrayobject.py
--- a/pypy/objspace/std/bytearrayobject.py
+++ b/pypy/objspace/std/bytearrayobject.py
@@ -204,7 +204,10 @@
         data = _hexstring_to_array(space, hexstring)
         # in CPython bytearray.fromhex is a staticmethod, so
         # we ignore w_type and always return a bytearray
-        return new_bytearray(space, space.w_bytearray, data)
+        w_result = new_bytearray(space, space.w_bytearray, data)
+        if w_bytearraytype is not space.w_bytearray:
+            w_result = space.call_function(w_bytearraytype, w_result)
+        return w_result
 
     @unwrap_spec(encoding='text_or_none', errors='text_or_none')
     def descr_init(self, space, w_source=None, encoding=None, errors=None):
diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -564,7 +564,10 @@
         from pypy.objspace.std.bytearrayobject import _hexstring_to_array
         hexstring = space.unicode_w(w_hexstring)
         bytes = ''.join(_hexstring_to_array(space, hexstring))
-        return W_BytesObject(bytes)
+        w_result = W_BytesObject(bytes)
+        if w_type is not space.w_bytes:
+            w_result = space.call_function(w_type, w_result)
+        return w_result
 
     def descr_repr(self, space):
         return space.newtext(string_escape_encode(self._value, True))
diff --git a/pypy/objspace/std/test/test_bytearrayobject.py b/pypy/objspace/std/test/test_bytearrayobject.py
--- a/pypy/objspace/std/test/test_bytearrayobject.py
+++ b/pypy/objspace/std/test/test_bytearrayobject.py
@@ -439,6 +439,11 @@
         raises(ValueError, bytearray.fromhex, '12   \x00   34')
         raises(ValueError, bytearray.fromhex, '\u1234')
 
+    def test_fromhex_subclass(self):
+        class Sub(bytearray):
+            pass
+        assert type(Sub.fromhex("abcd")) is Sub
+
     def test_extend(self):
         b = bytearray(b'abc')
         b.extend(bytearray(b'def'))
diff --git a/pypy/objspace/std/test/test_bytesobject.py b/pypy/objspace/std/test/test_bytesobject.py
--- a/pypy/objspace/std/test/test_bytesobject.py
+++ b/pypy/objspace/std/test/test_bytesobject.py
@@ -135,6 +135,11 @@
         raises(TypeError, bytes.fromhex, True)
         raises(ValueError, bytes.fromhex, "hello world")
 
+    def test_fromhex_subclass(self):
+        class Sub(bytes):
+            pass
+        assert type(Sub.fromhex("abcd")) is Sub
+
     def test_format(self):
         raises(TypeError, "foo".__mod__, "bar")
         raises(TypeError, u"foo".__mod__, "bar")

From pypy.commits at gmail.com  Fri Aug 18 15:21:28 2017
From: pypy.commits at gmail.com (mjacob)
Date: Fri, 18 Aug 2017 12:21:28 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Rename bytes.translate's 'deletechars'
 argument to 'delete' to match CPython.
Message-ID: <59973e38.90c5df0a.7e079.e5a0@mx.google.com>

Author: Manuel Jacob <me at manueljacob.de>
Branch: py3.5
Changeset: r92169:995cc5bdbad5
Date: 2017-07-22 03:36 +0200
http://bitbucket.org/pypy/pypy/changeset/995cc5bdbad5/

Log:	Rename bytes.translate's 'deletechars' argument to 'delete' to match
	CPython.

diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -365,8 +365,8 @@
         characters, all remaining cased characters have lowercase.
         """
 
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         """B.translate(table[, deletechars]) -> copy of B
 
         Return a copy of the string B, where all characters occurring
diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py
--- a/pypy/objspace/std/stringmethods.py
+++ b/pypy/objspace/std/stringmethods.py
@@ -742,8 +742,8 @@
     DEFAULT_NOOP_TABLE = ''.join([chr(i) for i in range(256)])
 
     # for bytes and bytearray, overridden by unicode
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         if space.is_w(w_table, space.w_None):
             table = self.DEFAULT_NOOP_TABLE
         else:
@@ -753,7 +753,7 @@
                             "translation table must be 256 characters long")
 
         string = self._val(space)
-        deletechars = self._op_val(space, w_deletechars)
+        deletechars = self._op_val(space, w_delete)
         if len(deletechars) == 0:
             buf = self._builder(len(string))
             for char in string:

From pypy.commits at gmail.com  Fri Aug 18 15:21:34 2017
From: pypy.commits at gmail.com (mjacob)
Date: Fri, 18 Aug 2017 12:21:34 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge
Message-ID: <59973e3e.cda2df0a.94a79.0e57@mx.google.com>

Author: Manuel Jacob <me at manueljacob.de>
Branch: py3.5
Changeset: r92172:9872266a277f
Date: 2017-08-18 21:20 +0200
http://bitbucket.org/pypy/pypy/changeset/9872266a277f/

Log:	hg merge

diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -364,8 +364,8 @@
         characters, all remaining cased characters have lowercase.
         """
 
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         """B.translate(table[, deletechars]) -> copy of B
 
         Return a copy of the string B, where all characters occurring
diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py
--- a/pypy/objspace/std/stringmethods.py
+++ b/pypy/objspace/std/stringmethods.py
@@ -742,8 +742,8 @@
     DEFAULT_NOOP_TABLE = ''.join([chr(i) for i in range(256)])
 
     # for bytes and bytearray, overridden by unicode
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         if space.is_w(w_table, space.w_None):
             table = self.DEFAULT_NOOP_TABLE
         else:
@@ -753,7 +753,7 @@
                             "translation table must be 256 characters long")
 
         string = self._val(space)
-        deletechars = self._op_val(space, w_deletechars)
+        deletechars = self._op_val(space, w_delete)
         if len(deletechars) == 0:
             buf = self._builder(len(string))
             for char in string:

From pypy.commits at gmail.com  Fri Aug 18 15:44:47 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 18 Aug 2017 12:44:47 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: If the machine can be [3-6],
 allow the suffix to be [3-6].
Message-ID: <599743af.42a7df0a.b750.fd50@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92173:6eb780259953
Date: 2017-08-18 15:43 -0400
http://bitbucket.org/pypy/pypy/changeset/6eb780259953/

Log:	If the machine can be [3-6], allow the suffix to be [3-6].

	 It's hard to tell but the intent
	(https://www.python.org/dev/peps/pep-3149/) seems to be to embed the
	machine identifier into the suffix. Therefore, the old version of
	the test was broken in asserting that i[4-6]86 machines all turn
	into i386 in the suffix. Change the test to accept the other machine
	identifiers as well.

diff --git a/lib-python/3/test/test_sysconfig.py b/lib-python/3/test/test_sysconfig.py
--- a/lib-python/3/test/test_sysconfig.py
+++ b/lib-python/3/test/test_sysconfig.py
@@ -397,9 +397,16 @@
             self.assertTrue('linux' in suffix, suffix)
         if re.match('(i[3-6]86|x86_64)$', machine):
             if ctypes.sizeof(ctypes.c_char_p()) == 4:
-                self.assertTrue(suffix.endswith('i386-linux-gnu.so') \
-                                or suffix.endswith('x86_64-linux-gnux32.so'),
-                                suffix)
+                self.assertTrue(
+                    suffix.endswith((
+                        'i386-linux-gnu.so',
+                        'i486-linux-gnu.so',
+                        'i586-linux-gnu.so',
+                        'i686-linux-gnu.so',
+                        'x86_64-linux-gnux32.so',
+                    )),
+                    suffix,
+                )
             else: # 8 byte pointer size
                 self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
 

From pypy.commits at gmail.com  Fri Aug 18 15:44:49 2017
From: pypy.commits at gmail.com (exarkun)
Date: Fri, 18 Aug 2017 12:44:49 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: merge upstream
Message-ID: <599743b1.71b8df0a.a1e87.f243@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: py3.5
Changeset: r92174:38134ba71d3c
Date: 2017-08-18 15:44 -0400
http://bitbucket.org/pypy/pypy/changeset/38134ba71d3c/

Log:	merge upstream

diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -364,8 +364,8 @@
         characters, all remaining cased characters have lowercase.
         """
 
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         """B.translate(table[, deletechars]) -> copy of B
 
         Return a copy of the string B, where all characters occurring
diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py
--- a/pypy/objspace/std/stringmethods.py
+++ b/pypy/objspace/std/stringmethods.py
@@ -742,8 +742,8 @@
     DEFAULT_NOOP_TABLE = ''.join([chr(i) for i in range(256)])
 
     # for bytes and bytearray, overridden by unicode
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         if space.is_w(w_table, space.w_None):
             table = self.DEFAULT_NOOP_TABLE
         else:
@@ -753,7 +753,7 @@
                             "translation table must be 256 characters long")
 
         string = self._val(space)
-        deletechars = self._op_val(space, w_deletechars)
+        deletechars = self._op_val(space, w_delete)
         if len(deletechars) == 0:
             buf = self._builder(len(string))
             for char in string:

From pypy.commits at gmail.com  Sat Aug 19 03:14:08 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 19 Aug 2017 00:14:08 -0700 (PDT)
Subject: [pypy-commit] pypy default: fix comment
Message-ID: <5997e540.90aadf0a.e530a.1904@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92175:46ce5f43e718
Date: 2017-08-19 09:13 +0200
http://bitbucket.org/pypy/pypy/changeset/46ce5f43e718/

Log:	fix comment

diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -1305,7 +1305,7 @@
         self.rm.possibly_free_var(tmpbox_high)
 
     def compute_hint_frame_locations(self, operations):
-        # optimization only: fill in the 'hint_frame_locations' dictionary
+        # optimization only: fill in the 'hint_frame_pos' dictionary
         # of 'fm' based on the JUMP at the end of the loop, by looking
         # at where we would like the boxes to be after the jump.
         op = operations[-1]
@@ -1320,7 +1320,7 @@
             self._compute_hint_frame_locations_from_descr(descr)
         #else:
         #   The loop ends in a JUMP going back to a LABEL in the same loop.
-        #   We cannot fill 'hint_frame_locations' immediately, but we can
+        #   We cannot fill 'hint_frame_pos' immediately, but we can
         #   wait until the corresponding consider_label() to know where the
         #   we would like the boxes to be after the jump.
 

From pypy.commits at gmail.com  Sat Aug 19 08:04:26 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 19 Aug 2017 05:04:26 -0700 (PDT)
Subject: [pypy-commit] pypy default: Apply part of PR #546
Message-ID: <5998294a.90addf0a.4572c.e0c5@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92176:2a268685e180
Date: 2017-08-19 14:03 +0200
http://bitbucket.org/pypy/pypy/changeset/2a268685e180/

Log:	Apply part of PR #546

diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -710,7 +710,8 @@
         size, _ = expected_size_and_sign
         return lltype.FixedSizeArray(fieldtype.OF, size/_sizeof(fieldtype.OF))
     raise TypeError("conflict between translating python and compiler field"
-                    " type %r for %r" % (fieldtype, fieldname))
+                    " type %r for symbol %r, expected size+sign %r" % (
+                        fieldtype, fieldname, expected_size_and_sign))
 
 def expose_value_as_rpython(value):
     if intmask(value) == value:

From pypy.commits at gmail.com  Sun Aug 20 04:15:27 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 01:15:27 -0700 (PDT)
Subject: [pypy-commit] pypy default: remove remnants of getfield support
 (now done via gc_load)
Message-ID: <5999451f.c6c31c0a.16a35.58fc@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92177:0d71d69924c8
Date: 2017-08-20 10:03 +0200
http://bitbucket.org/pypy/pypy/changeset/0d71d69924c8/

Log:	remove remnants of getfield support (now done via gc_load)

diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -1618,18 +1618,6 @@
         else:
             not_implemented("save_into_mem size = %d" % size)
 
-    def _genop_getfield(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc, sign_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
-        source_addr = AddressLoc(base_loc, ofs_loc)
-        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
-
-    genop_getfield_gc_i = _genop_getfield
-    genop_getfield_gc_r = _genop_getfield
-    genop_getfield_gc_f = _genop_getfield
-    genop_getfield_raw_i = _genop_getfield
-    genop_getfield_raw_f = _genop_getfield
-
     def _genop_gc_load(self, op, arglocs, resloc):
         base_loc, ofs_loc, size_loc, sign_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)

From pypy.commits at gmail.com  Sun Aug 20 06:38:22 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 03:38:22 -0700 (PDT)
Subject: [pypy-commit] pypy default: issue #2637 testing: implement
 CPyListStrategy.getstorage_copy
Message-ID: <5999669e.87badf0a.5c587.4844@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92178:06465478f8df
Date: 2017-08-20 12:37 +0200
http://bitbucket.org/pypy/pypy/changeset/06465478f8df/

Log:	issue #2637 testing: implement CPyListStrategy.getstorage_copy

diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -330,7 +330,11 @@
         return w_list.strategy.getitems_copy(w_list)
 
     def getstorage_copy(self, w_list):
-        raise NotImplementedError
+        storage = self.unerase(w_list.lstorage)
+        lst = [None] * storage._length
+        for i in range(storage._length):
+            lst[i] = from_ref(w_list.space, storage._elems[i])
+        return self.erase(CPyListStorage(w_list.space, lst))
 
     def append(self, w_list, w_item):
         w_list.switch_to_object_strategy()
diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py
--- a/pypy/module/cpyext/test/test_sequence.py
+++ b/pypy/module/cpyext/test/test_sequence.py
@@ -226,6 +226,15 @@
         w_l.inplace_mul(2)
         assert space.int_w(space.len(w_l)) == 10
 
+    def test_getstorage_copy(self, space, api):
+        w = space.wrap
+        w_l = w([1, 2, 3, 4])
+        api.PySequence_Fast(w_l, "foo") # converts
+
+        w_l1 = w([])
+        space.setitem(w_l1, space.newslice(w(0), w(0), w(1)), w_l)
+        assert map(space.unwrap, space.unpackiterable(w_l1)) == [1, 2, 3, 4]
+
 
 class AppTestSequenceObject(AppTestCpythonExtensionBase):
     def test_fast(self):

From pypy.commits at gmail.com  Sun Aug 20 06:55:23 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 03:55:23 -0700 (PDT)
Subject: [pypy-commit] pypy default: a bit of cleanup, remove some copies
Message-ID: <59996a9b.06981c0a.98175.f1c4@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92179:190bc6072d6d
Date: 2017-08-20 12:54 +0200
http://bitbucket.org/pypy/pypy/changeset/190bc6072d6d/

Log:	a bit of cleanup, remove some copies

diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -294,6 +294,23 @@
     def getitems_fixedsize(self, w_list):
         return self.getitems_unroll(w_list)
 
+    def copy_into(self, w_list, w_other):
+        w_other.strategy = self
+        w_other.lstorage = self.getstorage_copy(w_list)
+
+    def clone(self, w_list):
+        storage = self.getstorage_copy(w_list)
+        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
+                                                         self)
+        return w_clone
+
+    def getitems_copy(self, w_list):
+        return self.getitems(w_list) # getitems copies anyway
+
+    def getstorage_copy(self, w_list):
+        lst = self.getitems(w_list)
+        return self.erase(CPyListStorage(w_list.space, lst))
+
     #------------------------------------------
     # all these methods fail or switch strategy and then call ListObjectStrategy's method
 
@@ -301,23 +318,9 @@
         w_list.switch_to_object_strategy()
         w_list.strategy.setslice(w_list, start, stop, step, length)
 
-    def get_sizehint(self):
-        return -1
-
     def init_from_list_w(self, w_list, list_w):
         raise NotImplementedError
 
-    def clone(self, w_list):
-        storage = w_list.lstorage  # lstorage is tuple, no need to clone
-        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
-                                                         self)
-        w_clone.switch_to_object_strategy()
-        return w_clone
-
-    def copy_into(self, w_list, w_other):
-        w_list.switch_to_object_strategy()
-        w_list.strategy.copy_into(w_list, w_other)
-
     def _resize_hint(self, w_list, hint):
         pass
 
@@ -325,17 +328,6 @@
         w_list.switch_to_object_strategy()
         return w_list.strategy.find(w_list, w_item, start, stop)
 
-    def getitems_copy(self, w_list):
-        w_list.switch_to_object_strategy()
-        return w_list.strategy.getitems_copy(w_list)
-
-    def getstorage_copy(self, w_list):
-        storage = self.unerase(w_list.lstorage)
-        lst = [None] * storage._length
-        for i in range(storage._length):
-            lst[i] = from_ref(w_list.space, storage._elems[i])
-        return self.erase(CPyListStorage(w_list.space, lst))
-
     def append(self, w_list, w_item):
         w_list.switch_to_object_strategy()
         w_list.strategy.append(w_list, w_item)

From pypy.commits at gmail.com  Sun Aug 20 10:31:30 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 07:31:30 -0700 (PDT)
Subject: [pypy-commit] pypy default: make jit-backend-counts not be empty if
 it's the only section asked for
Message-ID: <59999d42.41bd1c0a.9f84.a648@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92180:0f01249125f8
Date: 2017-08-20 16:30 +0200
http://bitbucket.org/pypy/pypy/changeset/0f01249125f8/

Log:	make jit-backend-counts not be empty if it's the only section asked
	for

diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -504,7 +504,7 @@
         clt.frame_info = rffi.cast(jitframe.JITFRAMEINFOPTR, frame_info)
         clt.frame_info.clear() # for now
 
-        if log:
+        if log or self._debug:
             number = looptoken.number
             operations = self._inject_debugging_code(looptoken, operations,
                                                      'e', number)
@@ -589,7 +589,7 @@
                 faildescr.adr_jump_offset)
         self.mc.force_frame_size(DEFAULT_FRAME_BYTES)
         descr_number = compute_unique_id(faildescr)
-        if log:
+        if log or self._debug:
             operations = self._inject_debugging_code(faildescr, operations,
                                                      'b', descr_number)
         arglocs = self.rebuild_faillocs_from_descr(faildescr, inputargs)

From pypy.commits at gmail.com  Sun Aug 20 11:07:22 2017
From: pypy.commits at gmail.com (arigo)
Date: Sun, 20 Aug 2017 08:07:22 -0700 (PDT)
Subject: [pypy-commit] cffi default: Should call GC_UnTrack before
 Py_DECREF()
Message-ID: <5999a5aa.07c41c0a.64f3e.9b92@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3003:2763112dfca7
Date: 2017-08-20 17:07 +0200
http://bitbucket.org/cffi/cffi/changeset/2763112dfca7/

Log:	Should call GC_UnTrack before Py_DECREF()

diff --git a/c/lib_obj.c b/c/lib_obj.c
--- a/c/lib_obj.c
+++ b/c/lib_obj.c
@@ -89,6 +89,7 @@
 
 static void lib_dealloc(LibObject *lib)
 {
+    PyObject_GC_UnTrack(lib);
     cdlopen_close_ignore_errors(lib->l_libhandle);
     Py_DECREF(lib->l_dict);
     Py_DECREF(lib->l_libname);

From pypy.commits at gmail.com  Sun Aug 20 13:05:27 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:05:27 -0700 (PDT)
Subject: [pypy-commit] pypy default: simplify the logic
Message-ID: <5999c157.883f1c0a.dd59b.6c0b@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: 
Changeset: r92181:f59f82d01a6d
Date: 2017-08-20 19:04 +0200
http://bitbucket.org/pypy/pypy/changeset/f59f82d01a6d/

Log:	simplify the logic

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -552,10 +552,11 @@
             self.reg_bindings[result_v] = loc
             return loc
         if v not in self.reg_bindings:
+            # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(v, forbidden_vars)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-        assert v in self.reg_bindings
+            return loc
         if self.longevity[v][1] > self.position:
             # we need to find a new place for variable v and
             # store result in the same place

From pypy.commits at gmail.com  Sun Aug 20 13:09:26 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:26 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: a branch to try out some
 register allocation ideas
Message-ID: <5999c246.6395df0a.214ab.0010@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92182:32c23f8fb859
Date: 2017-08-18 21:36 +0200
http://bitbucket.org/pypy/pypy/changeset/32c23f8fb859/

Log:	a branch to try out some register allocation ideas

	first step: refactor lifetime storage to not be tuple-based

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -293,12 +293,12 @@
     def is_still_alive(self, v):
         # Check if 'v' is alive at the current position.
         # Return False if the last usage is strictly before.
-        return self.longevity[v][1] >= self.position
+        return self.longevity[v].last_usage >= self.position
 
     def stays_alive(self, v):
         # Check if 'v' stays alive after the current position.
         # Return False if the last usage is before or at position.
-        return self.longevity[v][1] > self.position
+        return self.longevity[v].last_usage > self.position
 
     def next_instruction(self, incr=1):
         self.position += incr
@@ -315,7 +315,7 @@
         self._check_type(v)
         if isinstance(v, Const):
             return
-        if v not in self.longevity or self.longevity[v][1] <= self.position:
+        if v not in self.longevity or self.longevity[v].last_usage <= self.position:
             if v in self.reg_bindings:
                 self.free_regs.append(self.reg_bindings[v])
                 del self.reg_bindings[v]
@@ -351,7 +351,7 @@
             for v in self.reg_bindings:
                 if v not in self.longevity:
                     llop.debug_print(lltype.Void, "variable %s not in longevity\n" % v.repr({}))
-                assert self.longevity[v][1] > self.position
+                assert self.longevity[v].last_usage > self.position
 
     def try_allocate_reg(self, v, selected_reg=None, need_lower_byte=False):
         """ Try to allocate a register, if we have one free.
@@ -427,7 +427,7 @@
                     continue
             if need_lower_byte and reg in self.no_lower_byte_regs:
                 continue
-            max_age = self.longevity[next][1]
+            max_age = self.longevity[next].last_usage
             if cur_max_age < max_age:
                 cur_max_age = max_age
                 candidate = next
@@ -446,7 +446,7 @@
         """
         self._check_type(v)
         if isinstance(v, TempVar):
-            self.longevity[v] = (self.position, self.position)
+            self.longevity[v] = Lifetime(self.position, self.position)
         loc = self.try_allocate_reg(v, selected_reg,
                                     need_lower_byte=need_lower_byte)
         if loc:
@@ -556,7 +556,7 @@
             loc = self.force_allocate_reg(v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
         assert v in self.reg_bindings
-        if self.longevity[v][1] > self.position:
+        if self.longevity[v].last_usage > self.position:
             # we need to find a new place for variable v and
             # store result in the same place
             loc = self.reg_bindings[v]
@@ -643,7 +643,7 @@
         move_or_spill = []
 
         for v, reg in self.reg_bindings.items():
-            max_age = self.longevity[v][1]
+            max_age = self.longevity[v].last_usage
             if v not in force_store and max_age <= self.position:
                 # variable dies
                 del self.reg_bindings[v]
@@ -765,7 +765,7 @@
         # of COND_CALL don't accept a cc as input
         if next_op.getarg(0) is not op:
             return False
-        if self.longevity[op][1] > i + 1:
+        if self.longevity[op].last_usage > i + 1:
             return False
         if opnum != rop.COND_CALL:
             if op in operations[i + 1].getfailargs():
@@ -785,6 +785,10 @@
             assert op.numargs() == 1
             return [self.loc(op.getarg(0))]
 
+class Lifetime(object):
+    def __init__(self, definition_pos, last_usage):
+        self.definition_pos = definition_pos
+        self.last_usage = last_usage
 
 def compute_vars_longevity(inputargs, operations):
     # compute a dictionary that maps variables to index in
@@ -824,14 +828,14 @@
         if arg.type != 'v' and arg in last_used:
             assert not isinstance(arg, Const)
             assert i < last_used[arg]
-            longevity[arg] = (i, last_used[arg])
+            longevity[arg] = Lifetime(i, last_used[arg])
             del last_used[arg]
     for arg in inputargs:
         assert not isinstance(arg, Const)
         if arg not in last_used:
-            longevity[arg] = (-1, -1)
+            longevity[arg] = Lifetime(-1, -1)
         else:
-            longevity[arg] = (0, last_used[arg])
+            longevity[arg] = Lifetime(0, last_used[arg])
             del last_used[arg]
     assert len(last_used) == 0
 
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1,7 +1,7 @@
 import py
 from rpython.jit.metainterp.history import ConstInt, INT, FLOAT
 from rpython.jit.backend.llsupport.regalloc import FrameManager, LinkedList
-from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan
+from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan, Lifetime
 from rpython.jit.metainterp.resoperation import InputArgInt, InputArgRef,\
      InputArgFloat
 
@@ -17,7 +17,7 @@
     for i in range(num):
         box = InputArgInt(0)
         res.append(box)
-        longevity[box] = (0, 1)
+        longevity[box] = Lifetime(0, 1)
     return res, longevity
 
 class FakeReg(object):
@@ -76,7 +76,7 @@
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)
-        longevity = {b0: (0, 1), b1: (0, 2), b2: (0, 2)}
+        longevity = {b0: Lifetime(0, 1), b1: Lifetime(0, 2), b2: Lifetime(0, 2)}
         rm = RegisterManager(longevity)
         rm.next_instruction()
         for b in b0, b1, b2:
@@ -190,7 +190,7 @@
         
     def test_force_result_in_reg_1(self):
         b0, b1 = newboxes(0, 0)
-        longevity = {b0: (0, 1), b1: (1, 3)}
+        longevity = {b0: Lifetime(0, 1), b1: Lifetime(1, 3)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -206,7 +206,7 @@
 
     def test_force_result_in_reg_2(self):
         b0, b1 = newboxes(0, 0)
-        longevity = {b0: (0, 2), b1: (1, 3)}
+        longevity = {b0: Lifetime(0, 2), b1: Lifetime(1, 3)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -222,7 +222,9 @@
 
     def test_force_result_in_reg_3(self):
         b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
-        longevity = {b0: (0, 2), b1: (0, 2), b3: (0, 2), b2: (0, 2), b4: (1, 3)}
+        longevity = {b0: Lifetime(0, 2), b1: Lifetime(0, 2),
+                     b3: Lifetime(0, 2), b2: Lifetime(0, 2),
+                     b4: Lifetime(1, 3)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -238,7 +240,7 @@
 
     def test_force_result_in_reg_4(self):
         b0, b1 = newboxes(0, 0)
-        longevity = {b0: (0, 1), b1: (0, 1)}
+        longevity = {b0: Lifetime(0, 1), b1: Lifetime(0, 1)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -254,7 +256,7 @@
 
     def test_bogus_make_sure_var_in_reg(self):
         b0, = newboxes(0)
-        longevity = {b0: (0, 1)}
+        longevity = {b0: Lifetime(0, 1)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -350,11 +352,11 @@
 
         fm = TFrameManager()
         b0 = InputArgInt()
-        longevity = {b0: (0, 1)}
+        longevity = {b0: Lifetime(0, 1)}
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
         f0 = InputArgFloat()
-        longevity = {f0: (0, 1)}
+        longevity = {f0: Lifetime(0, 1)}
         xrm = XRegisterManager(longevity, frame_manager=fm, assembler=asm)
         xrm.loc(f0)
         rm.loc(b0)
@@ -362,7 +364,9 @@
                 
     def test_spilling(self):
         b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
-        longevity = {b0: (0, 3), b1: (0, 3), b3: (0, 5), b2: (0, 2), b4: (1, 4), b5: (1, 3)}
+        longevity = {b0: Lifetime(0, 3), b1: Lifetime(0, 3),
+                     b3: Lifetime(0, 5), b2: Lifetime(0, 2),
+                     b4: Lifetime(1, 4), b5: Lifetime(1, 3)}
         fm = TFrameManager()
         asm = MockAsm()
         rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
@@ -378,7 +382,6 @@
         assert spilled2 is loc
         rm._check_invariants()
 
-
     def test_hint_frame_locations_1(self):
         for hint_value in range(11):
             b0, = newboxes(0)
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -511,7 +511,7 @@
         # and won't be used after the current operation finishes,
         # then swap the role of 'x' and 'y'
         if (symm and isinstance(argloc, RegLoc) and
-                self.rm.longevity[y][1] == self.rm.position):
+                self.rm.longevity[y].last_usage == self.rm.position):
             x, y = y, x
             argloc = self.loc(y)
         #
diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py
--- a/rpython/jit/backend/zarch/regalloc.py
+++ b/rpython/jit/backend/zarch/regalloc.py
@@ -1,6 +1,7 @@
 from rpython.jit.backend.llsupport.regalloc import (RegisterManager, FrameManager,
                                                     TempVar, compute_vars_longevity,
-                                                    BaseRegalloc, NoVariableToSpill)
+                                                    BaseRegalloc, NoVariableToSpill,
+                                                    Lifetime)
 from rpython.jit.backend.llsupport.jump import remap_frame_layout_mixed
 from rpython.jit.backend.zarch.arch import WORD
 from rpython.jit.codewriter import longlong
@@ -255,9 +256,9 @@
         self._check_type(even_var)
         self._check_type(odd_var)
         if isinstance(even_var, TempVar):
-            self.longevity[even_var] = (self.position, self.position)
+            self.longevity[even_var] = Lifetime(self.position, self.position)
         if isinstance(odd_var, TempVar):
-            self.longevity[odd_var] = (self.position, self.position)
+            self.longevity[odd_var] = Lifetime(self.position, self.position)
 
         # this function steps through the following:
         # 1) maybe there is an even/odd pair that is always

From pypy.commits at gmail.com  Sun Aug 20 13:09:29 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:29 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: fix test
Message-ID: <5999c249.a588df0a.56f4c.05a2@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92183:39a4798dc769
Date: 2017-08-19 08:56 +0200
http://bitbucket.org/pypy/pypy/changeset/39a4798dc769/

Log:	fix test

diff --git a/rpython/jit/backend/x86/test/test_x86vector.py b/rpython/jit/backend/x86/test/test_x86vector.py
--- a/rpython/jit/backend/x86/test/test_x86vector.py
+++ b/rpython/jit/backend/x86/test/test_x86vector.py
@@ -1,5 +1,6 @@
 import py
 from rpython.jit.backend.x86.regloc import *
+from rpython.jit.backend.llsupport.regalloc import Lifetime
 from rpython.jit.backend.x86.regalloc import (RegAlloc,
         X86FrameManager, X86XMMRegisterManager, X86RegisterManager)
 from rpython.jit.backend.x86.vector_ext import TempVector
@@ -146,9 +147,9 @@
         xrm = self.regalloc.xrm
         xrm.reg_bindings[arg1] = xmm0
         xrm.reg_bindings[arg2] = xmm1
-        xrm.longevity[arg1] = (0,1)
-        xrm.longevity[arg2] = (0,2)
-        xrm.longevity[arg] = (0,3)
+        xrm.longevity[arg1] = Lifetime(0,1)
+        xrm.longevity[arg2] = Lifetime(0,2)
+        xrm.longevity[arg] = Lifetime(0,3)
         fr = xrm.free_regs
         xrm.free_regs = []
         self.regalloc.fm.bindings[arg] = FrameLoc(0, 64, 'f')

From pypy.commits at gmail.com  Sun Aug 20 13:09:31 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:31 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: some ideas what to improve
Message-ID: <5999c24b.90b6df0a.b932.966e@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92184:beea904fa30e
Date: 2017-08-19 08:59 +0200
http://bitbucket.org/pypy/pypy/changeset/beea904fa30e/

Log:	some ideas what to improve

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -394,6 +394,7 @@
         try:
             return self.reg_bindings[v]
         except KeyError:
+            # YYY here we should chose the free variable a bit more carefully
             if self.free_regs:
                 loc = self.free_regs.pop()
                 self.reg_bindings[v] = loc
@@ -416,6 +417,8 @@
         """
         cur_max_age = -1
         candidate = None
+        # YYY we should pick a variable to spill that is only used in failargs
+        # from now on
         for next in self.reg_bindings:
             reg = self.reg_bindings[next]
             if next in forbidden_vars:
@@ -528,6 +531,8 @@
         self.reg_bindings[to_v] = reg
 
     def _move_variable_away(self, v, prev_loc):
+        # YYY here we should not move it to another reg, if all uses are in
+        # failargs
         if self.free_regs:
             loc = self.free_regs.pop()
             self.reg_bindings[v] = loc

From pypy.commits at gmail.com  Sun Aug 20 13:09:32 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:32 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: store the last_real_usage
 on the Lifetimes as well
Message-ID: <5999c24c.d24a1c0a.92acd.c1ac@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92185:b4d4200219d6
Date: 2017-08-19 09:00 +0200
http://bitbucket.org/pypy/pypy/changeset/b4d4200219d6/

Log:	store the last_real_usage on the Lifetimes as well

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -791,9 +791,15 @@
             return [self.loc(op.getarg(0))]
 
 class Lifetime(object):
-    def __init__(self, definition_pos, last_usage):
+    def __init__(self, definition_pos, last_usage, last_real_usage=-42):
         self.definition_pos = definition_pos
         self.last_usage = last_usage
+        if last_real_usage == -42:
+            last_real_usage = last_usage
+        self.last_real_usage = last_real_usage
+
+    def is_last_real_use_before(self, position):
+        return self.last_real_usage <= position
 
 def compute_vars_longevity(inputargs, operations):
     # compute a dictionary that maps variables to index in
@@ -833,14 +839,17 @@
         if arg.type != 'v' and arg in last_used:
             assert not isinstance(arg, Const)
             assert i < last_used[arg]
-            longevity[arg] = Lifetime(i, last_used[arg])
+            longevity[arg] = Lifetime(
+                i, last_used[arg], last_real_usage.get(arg, -1))
             del last_used[arg]
     for arg in inputargs:
         assert not isinstance(arg, Const)
         if arg not in last_used:
-            longevity[arg] = Lifetime(-1, -1)
+            longevity[arg] = Lifetime(
+                -1, -1, -1)
         else:
-            longevity[arg] = Lifetime(0, last_used[arg])
+            longevity[arg] = Lifetime(
+                0, last_used[arg], last_real_usage.get(arg, -1))
             del last_used[arg]
     assert len(last_used) == 0
 
@@ -853,8 +862,8 @@
                 if not isinstance(arg, Const):
                     assert arg in produced
             produced[op] = None
-    
-    return longevity, last_real_usage
+
+    return longevity
 
 def is_comparison_or_ovf_op(opnum):
     return rop.is_comparison(opnum) or rop.is_ovf(opnum)
diff --git a/rpython/jit/backend/ppc/regalloc.py b/rpython/jit/backend/ppc/regalloc.py
--- a/rpython/jit/backend/ppc/regalloc.py
+++ b/rpython/jit/backend/ppc/regalloc.py
@@ -197,10 +197,8 @@
         operations = cpu.gc_ll_descr.rewrite_assembler(cpu, operations,
                                                        allgcrefs)
         # compute longevity of variables
-        longevity, last_real_usage = compute_vars_longevity(
-                                                    inputargs, operations)
+        longevity = compute_vars_longevity(inputargs, operations)
         self.longevity = longevity
-        self.last_real_usage = last_real_usage
         self.rm = PPCRegisterManager(self.longevity,
                                      frame_manager = self.fm,
                                      assembler = self.assembler)
@@ -949,7 +947,7 @@
         position = self.rm.position
         for arg in inputargs:
             assert not isinstance(arg, Const)
-            if self.last_real_usage.get(arg, -1) <= position:
+            if self.longevity[arg].is_last_real_use_before(position):
                 self.force_spill_var(arg)
         #
         # we need to make sure that no variable is stored in spp (=r31)
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -165,10 +165,8 @@
         operations = cpu.gc_ll_descr.rewrite_assembler(cpu, operations,
                                                        allgcrefs)
         # compute longevity of variables
-        longevity, last_real_usage = compute_vars_longevity(
-                                                    inputargs, operations)
+        longevity = compute_vars_longevity(inputargs, operations)
         self.longevity = longevity
-        self.last_real_usage = last_real_usage
         self.rm = gpr_reg_mgr_cls(self.longevity,
                                   frame_manager = self.fm,
                                   assembler = self.assembler)
@@ -1404,7 +1402,7 @@
         position = self.rm.position
         for arg in inputargs:
             assert not isinstance(arg, Const)
-            if self.last_real_usage.get(arg, -1) <= position:
+            if self.longevity[arg].is_last_real_use_before(position):
                 self.force_spill_var(arg)
         #
         # we need to make sure that no variable is stored in ebp
diff --git a/rpython/jit/backend/zarch/regalloc.py b/rpython/jit/backend/zarch/regalloc.py
--- a/rpython/jit/backend/zarch/regalloc.py
+++ b/rpython/jit/backend/zarch/regalloc.py
@@ -452,10 +452,8 @@
         operations = cpu.gc_ll_descr.rewrite_assembler(cpu, operations,
                                                        allgcrefs)
         # compute longevity of variables
-        longevity, last_real_usage = compute_vars_longevity(
-                                                    inputargs, operations)
+        longevity = compute_vars_longevity(inputargs, operations)
         self.longevity = longevity
-        self.last_real_usage = last_real_usage
         self.rm = ZARCHRegisterManager(self.longevity,
                                      frame_manager = self.fm,
                                      assembler = self.assembler)
@@ -1307,7 +1305,7 @@
         position = self.rm.position
         for arg in inputargs:
             assert not isinstance(arg, Const)
-            if self.last_real_usage.get(arg, -1) <= position:
+            if self.longevity[arg].is_last_real_use_before(position):
                 self.force_spill_var(arg)
         #
         # we need to make sure that no variable is stored in spp (=r31)

From pypy.commits at gmail.com  Sun Aug 20 13:09:39 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:39 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: fix tests
Message-ID: <5999c253.6ea9df0a.a0568.60e8@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92188:cc329ecab757
Date: 2017-08-20 16:26 +0200
http://bitbucket.org/pypy/pypy/changeset/cc329ecab757/

Log:	fix tests

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -69,7 +69,20 @@
 class MockAsm(object):
     def __init__(self):
         self.moves = []
-    
+
+        # XXX register allocation statistics to be removed later
+        self.num_moves_calls = 0
+        self.num_moves_jump = 0
+        self.num_spills = 0
+        self.num_spills_to_existing = 0
+        self.num_reloads = 0
+
+        self.preamble_num_moves_calls = 0
+        self.preamble_num_moves_jump = 0
+        self.preamble_num_spills = 0
+        self.preamble_num_spills_to_existing = 0
+        self.preamble_num_reloads = 0
+
     def regalloc_mov(self, from_loc, to_loc):
         self.moves.append((from_loc, to_loc))
 

From pypy.commits at gmail.com  Sun Aug 20 13:09:35 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:35 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: improve comments
Message-ID: <5999c24f.52831c0a.35b43.03fe@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92186:fc6b711bad77
Date: 2017-08-19 09:39 +0200
http://bitbucket.org/pypy/pypy/changeset/fc6b711bad77/

Log:	improve comments

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -792,23 +792,27 @@
 
 class Lifetime(object):
     def __init__(self, definition_pos, last_usage, last_real_usage=-42):
+        # all positions are indexes into the operations list
+
+        # the position where the variable is defined
         self.definition_pos = definition_pos
+        # the position where the variable is last used. this includes failargs
+        # and jumps
         self.last_usage = last_usage
         if last_real_usage == -42:
             last_real_usage = last_usage
+        # last *real* usage, ie as an argument to an operation
+        # after last_real_usage and last_usage it does not matter whether the
+        # variable is stored on the stack
         self.last_real_usage = last_real_usage
 
     def is_last_real_use_before(self, position):
         return self.last_real_usage <= position
 
 def compute_vars_longevity(inputargs, operations):
-    # compute a dictionary that maps variables to index in
-    # operations that is a "last-time-seen"
-
-    # returns a pair longevity/useful. Non-useful variables are ones that
-    # never appear in the assembler or it does not matter if they appear on
-    # stack or in registers. Main example is loop arguments that go
-    # only to guard operations or to jump or to finish
+    # compute a dictionary that maps variables to Lifetime information
+    # if a variable is not in the dictionary, it's operation is dead because
+    # it's side-effect-free and the result is unused
     last_used = {}
     last_real_usage = {}
     for i in range(len(operations)-1, -1, -1):

From pypy.commits at gmail.com  Sun Aug 20 13:09:41 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:41 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: spill variables that are no
 longer "really" used
Message-ID: <5999c255.41aedf0a.8ab32.d990@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92189:c5b3232137fe
Date: 2017-08-20 16:27 +0200
http://bitbucket.org/pypy/pypy/changeset/c5b3232137fe/

Log:	spill variables that are no longer "really" used (ie appear only in
	failargs or jumps)

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -433,7 +433,13 @@
                     continue
             if need_lower_byte and reg in self.no_lower_byte_regs:
                 continue
-            max_age = self.longevity[next].last_usage
+            lifetime = self.longevity[next]
+            if lifetime.is_last_real_use_before(self.position):
+                # this variable has no "real" use as an argument to an op left
+                # it is only used in failargs, and maybe in a jump. spilling is
+                # fine
+                return next
+            max_age = lifetime.last_usage
             if cur_max_age < max_age:
                 cur_max_age = max_age
                 candidate = next
@@ -814,6 +820,9 @@
     def is_last_real_use_before(self, position):
         return self.last_real_usage <= position
 
+    def __repr__(self):
+        return "%s:%s(%s)" % (self.definition_pos, self.last_real_usage, self.last_usage)
+
 def compute_vars_longevity(inputargs, operations):
     # compute a dictionary that maps variables to Lifetime information
     # if a variable is not in the dictionary, it's operation is dead because
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -395,6 +395,28 @@
         assert spilled2 is loc
         rm._check_invariants()
 
+    def test_spill_useless_vars_first(self):
+        b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
+        longevity = {b0: Lifetime(0, 5), b1: Lifetime(0, 5),
+                     # b3 becomes useless but b2 lives longer
+                     b3: Lifetime(0, 5, 3), b2: Lifetime(0, 6),
+                     b4: Lifetime(4, 5), b5: Lifetime(4, 7)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        for b in b0, b1, b2, b3:
+            rm.force_allocate_reg(b)
+        rm.position = 4
+        assert len(rm.free_regs) == 0
+        loc = rm.loc(b3)
+        spilled = rm.force_allocate_reg(b4)
+        assert spilled is loc
+        loc = rm.loc(b2)
+        spilled2 = rm.force_allocate_reg(b5)
+        assert spilled2 is loc
+        rm._check_invariants()
+
     def test_hint_frame_locations_1(self):
         for hint_value in range(11):
             b0, = newboxes(0)

From pypy.commits at gmail.com  Sun Aug 20 13:09:37 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:37 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: collect some statitics
 about the register allocator
Message-ID: <5999c251.8691df0a.e1dd9.9e3a@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92187:9ee8caf969f5
Date: 2017-08-20 16:04 +0200
http://bitbucket.org/pypy/pypy/changeset/9ee8caf969f5/

Log:	collect some statitics about the register allocator

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -406,9 +406,12 @@
                                selected_reg, need_lower_byte=need_lower_byte)
         loc = self.reg_bindings[v_to_spill]
         del self.reg_bindings[v_to_spill]
+        self.assembler.num_spills += 1
         if self.frame_manager.get(v_to_spill) is None:
             newloc = self.frame_manager.loc(v_to_spill)
             self.assembler.regalloc_mov(loc, newloc)
+        else:
+            self.assembler.num_spills_to_existing += 1
         return loc
 
     def _pick_variable_to_spill(self, v, forbidden_vars, selected_reg=None,
@@ -522,6 +525,7 @@
         loc = self.force_allocate_reg(v, forbidden_vars, selected_reg,
                                       need_lower_byte=need_lower_byte)
         if prev_loc is not loc:
+            self.assembler.num_reloads += 1
             self.assembler.regalloc_mov(prev_loc, loc)
         return loc
 
@@ -576,6 +580,7 @@
 
     def _sync_var(self, v):
         if not self.frame_manager.get(v):
+            self.num_moves_calls += 1
             reg = self.reg_bindings[v]
             to = self.frame_manager.loc(v)
             self.assembler.regalloc_mov(reg, to)
diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -90,6 +90,20 @@
         self.target_tokens_currently_compiling = {}
         self.frame_depth_to_patch = []
 
+        # XXX register allocation statistics to be removed later
+        self.num_moves_calls = 0
+        self.num_moves_jump = 0
+        self.num_spills = 0
+        self.num_spills_to_existing = 0
+        self.num_reloads = 0
+
+        self.preamble_num_moves_calls = 0
+        self.preamble_num_moves_jump = 0
+        self.preamble_num_spills = 0
+        self.preamble_num_spills_to_existing = 0
+        self.preamble_num_reloads = 0
+
+
     def teardown(self):
         self.pending_guard_tokens = None
         if WORD == 8:
@@ -545,6 +559,25 @@
                                                  size_excluding_failure_stuff))
         debug_print("            end: 0x%x" % r_uint(rawstart + full_size))
         debug_stop("jit-backend-addr")
+        debug_start("jit-regalloc-stats")
+        debug_print("Loop %d (%s) has address 0x%x to 0x%x (bootstrap 0x%x)" % (
+            looptoken.number, loopname,
+            r_uint(rawstart + looppos),
+            r_uint(rawstart + size_excluding_failure_stuff),
+            r_uint(rawstart + functionpos)))
+        debug_print("assembler size: ", size_excluding_failure_stuff)
+        debug_print("number ops: ", len(operations))
+        debug_print("preamble num moves calls: ", self.preamble_num_moves_calls)
+        debug_print("preamble num moves jump:", self.preamble_num_moves_jump)
+        debug_print("preamble num moves spills:", self.preamble_num_spills)
+        debug_print("preamble num moves spills to existing:", self.preamble_num_spills_to_existing)
+        debug_print("preamble num register reloads:", self.preamble_num_reloads)
+        debug_print("num moves calls: ", self.num_moves_calls)
+        debug_print("num moves jump:", self.num_moves_jump)
+        debug_print("num moves spills:", self.num_spills)
+        debug_print("num moves spills to existing:", self.num_spills_to_existing)
+        debug_print("num moves register reloads:", self.num_reloads)
+        debug_stop("jit-regalloc-stats")
         self.patch_pending_failure_recoveries(rawstart)
         #
         ops_offset = self.mc.ops_offset
@@ -624,6 +657,24 @@
         debug_print("       failures: 0x%x" % r_uint(rawstart + codeendpos))
         debug_print("            end: 0x%x" % r_uint(rawstart + fullsize))
         debug_stop("jit-backend-addr")
+        debug_start("jit-regalloc-stats")
+        debug_print("bridge out of Guard 0x%x has address 0x%x to 0x%x" %
+                    (r_uint(descr_number), r_uint(rawstart + startpos),
+                        r_uint(rawstart + codeendpos)))
+
+        debug_print("assembler size: ", fullsize)
+        debug_print("number ops: ", len(operations))
+        debug_print("preamble num moves calls: ", self.preamble_num_moves_calls)
+        debug_print("preamble num moves jump:", self.preamble_num_moves_jump)
+        debug_print("preamble num moves spills:", self.preamble_num_spills)
+        debug_print("preamble num moves spills to existing:", self.preamble_num_spills_to_existing)
+        debug_print("preamble num register reloads:", self.preamble_num_reloads)
+        debug_print("num moves calls: ", self.num_moves_calls)
+        debug_print("num moves jump:", self.num_moves_jump)
+        debug_print("num moves spills:", self.num_spills)
+        debug_print("num moves spills to existing:", self.num_spills_to_existing)
+        debug_print("num moves register reloads:", self.num_reloads)
+        debug_stop("jit-regalloc-stats")
         self.patch_pending_failure_recoveries(rawstart)
         # patch the jump from original guard
         self.patch_jump_for_descr(faildescr, rawstart + startpos)
@@ -1291,10 +1342,12 @@
                                      result_loc, result_type,
                                      result_size)
         cb.emit()
+        self.num_moves_calls += cb.num_moves
 
     def simple_call_no_collect(self, fnloc, arglocs):
         cb = callbuilder.CallBuilder(self, fnloc, arglocs)
         cb.emit_no_collect()
+        self.num_moves_calls += cb.num_moves
 
     def _reload_frame_if_necessary(self, mc, shadowstack_reg=None):
         gcrootmap = self.cpu.gc_ll_descr.gcrootmap
@@ -2144,6 +2197,7 @@
                 cb.emit()
             else:
                 cb.emit_no_collect()
+        self.num_moves_calls += cb.num_moves
 
     def _store_force_index(self, guard_op):
         assert (guard_op.getopnum() == rop.GUARD_NOT_FORCED or
@@ -2422,6 +2476,16 @@
             self.mc.JMP(imm(target))
 
     def label(self):
+        self.preamble_num_moves_calls += self.num_moves_calls
+        self.preamble_num_moves_jump += self.num_moves_jump
+        self.preamble_num_spills += self.num_spills
+        self.preamble_num_spills_to_existing += self.num_spills_to_existing
+        self.preamble_num_reloads += self.num_reloads
+        self.num_moves_calls = 0
+        self.num_moves_jump = 0
+        self.num_spills = 0
+        self.num_spills_to_existing = 0
+        self.num_reloads = 0
         self._check_frame_depth_debug(self.mc)
 
     class CondCallSlowPath(codebuf.SlowPath):
diff --git a/rpython/jit/backend/x86/callbuilder.py b/rpython/jit/backend/x86/callbuilder.py
--- a/rpython/jit/backend/x86/callbuilder.py
+++ b/rpython/jit/backend/x86/callbuilder.py
@@ -632,25 +632,30 @@
         self.subtract_esp_aligned(on_stack - self.stack_max)
 
         # Handle register arguments: first remap the xmm arguments
-        remap_frame_layout(self.asm, xmm_src_locs, xmm_dst_locs,
-                           X86_64_XMM_SCRATCH_REG)
+        num_moves = remap_frame_layout(self.asm, xmm_src_locs, xmm_dst_locs,
+                                       X86_64_XMM_SCRATCH_REG)
         # Load the singlefloat arguments from main regs or stack to xmm regs
         if singlefloats is not None:
             for src, dst in singlefloats:
                 if isinstance(dst, RawEspLoc):
                     # XXX too much special logic
                     if isinstance(src, RawEbpLoc):
+                        num_moves += 2
                         self.mc.MOV32(X86_64_SCRATCH_REG, src)
                         self.mc.MOV32(dst, X86_64_SCRATCH_REG)
                     else:
+                        num_moves += 1
                         self.mc.MOV32(dst, src)
                     continue
                 if isinstance(src, ImmedLoc):
+                    num_moves += 1
                     self.mc.MOV(X86_64_SCRATCH_REG, src)
                     src = X86_64_SCRATCH_REG
+                num_moves += 1
                 self.mc.MOVD32(dst, src)
         # Finally remap the arguments in the main regs
-        remap_frame_layout(self.asm, src_locs, dst_locs, X86_64_SCRATCH_REG)
+        num_moves += remap_frame_layout(self.asm, src_locs, dst_locs, X86_64_SCRATCH_REG)
+        self.num_moves = num_moves
 
 
     def emit_raw_call(self):
diff --git a/rpython/jit/backend/x86/jump.py b/rpython/jit/backend/x86/jump.py
--- a/rpython/jit/backend/x86/jump.py
+++ b/rpython/jit/backend/x86/jump.py
@@ -6,6 +6,7 @@
     pending_dests = len(dst_locations)
     srccount = {}    # maps dst_locations to how many times the same
                      # location appears in src_locations
+    num_moves = 0
     for dst in dst_locations:
         key = dst._getregkey()
         assert key not in srccount, "duplicate value in dst_locations!"
@@ -39,6 +40,7 @@
                     if key in srccount:
                         srccount[key] -= 1
                 _move(assembler, src, dst, tmpreg)
+                num_moves += 1
                 progress = True
         if not progress:
             # we are left with only pure disjoint cycles
@@ -53,6 +55,7 @@
                 originalkey = dst._getregkey()
                 if srccount[originalkey] >= 0:
                     assembler.regalloc_push(dst)
+                    num_moves += 1
                     while True:
                         key = dst._getregkey()
                         assert srccount[key] == 1
@@ -63,9 +66,12 @@
                         if src._getregkey() == originalkey:
                             break
                         _move(assembler, src, dst, tmpreg)
+                        num_moves += 1
                         dst = src
                     assembler.regalloc_pop(dst)
+                    num_moves += 1
             assert pending_dests == 0
+    return num_moves
 
 def _move(assembler, src, dst, tmpreg):
     if dst.is_memory_reference() and src.is_memory_reference():
@@ -93,6 +99,7 @@
         dst_keys[loc._getregkey()] = None
     src_locations2red = []
     dst_locations2red = []
+    num_moves = 0
     for i in range(len(src_locations2)):
         loc    = src_locations2[i]
         dstloc = dst_locations2[i]
@@ -100,6 +107,7 @@
             key = loc._getregkey()
             if (key in dst_keys or (loc.get_width() > WORD and
                                     (key + WORD) in dst_keys)):
+                num_moves += 1
                 assembler.regalloc_push(loc)
                 extrapushes.append(dstloc)
                 continue
@@ -109,12 +117,14 @@
     dst_locations2 = dst_locations2red
     #
     # remap the integer and pointer registers and stack locations
-    remap_frame_layout(assembler, src_locations1, dst_locations1, tmpreg1)
+    num_moves += remap_frame_layout(assembler, src_locations1, dst_locations1, tmpreg1)
     #
     # remap the xmm registers and stack locations
-    remap_frame_layout(assembler, src_locations2, dst_locations2, tmpreg2)
+    num_moves += remap_frame_layout(assembler, src_locations2, dst_locations2, tmpreg2)
     #
     # finally, pop the extra xmm stack locations
     while len(extrapushes) > 0:
         loc = extrapushes.pop()
         assembler.regalloc_pop(loc)
+        num_moves += 1
+    return num_moves
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -1365,11 +1365,12 @@
             tmpreg = None
             xmmtmp = None
         # Do the remapping
-        remap_frame_layout_mixed(assembler,
+        num_moves = remap_frame_layout_mixed(assembler,
                                  src_locations1, dst_locations1, tmpreg,
                                  src_locations2, dst_locations2, xmmtmp)
         self.possibly_free_vars_for_op(op)
         assembler.closing_jump(self.jump_target_descr)
+        assembler.num_moves_jump += num_moves
 
     def consider_enter_portal_frame(self, op):
         self.assembler.enter_portal_frame(op)

From pypy.commits at gmail.com  Sun Aug 20 13:09:42 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:42 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: merge default
Message-ID: <5999c256.93071c0a.b049.f964@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92190:7a89232328d8
Date: 2017-08-20 16:31 +0200
http://bitbucket.org/pypy/pypy/changeset/7a89232328d8/

Log:	merge default

diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -294,6 +294,23 @@
     def getitems_fixedsize(self, w_list):
         return self.getitems_unroll(w_list)
 
+    def copy_into(self, w_list, w_other):
+        w_other.strategy = self
+        w_other.lstorage = self.getstorage_copy(w_list)
+
+    def clone(self, w_list):
+        storage = self.getstorage_copy(w_list)
+        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
+                                                         self)
+        return w_clone
+
+    def getitems_copy(self, w_list):
+        return self.getitems(w_list) # getitems copies anyway
+
+    def getstorage_copy(self, w_list):
+        lst = self.getitems(w_list)
+        return self.erase(CPyListStorage(w_list.space, lst))
+
     #------------------------------------------
     # all these methods fail or switch strategy and then call ListObjectStrategy's method
 
@@ -301,23 +318,9 @@
         w_list.switch_to_object_strategy()
         w_list.strategy.setslice(w_list, start, stop, step, length)
 
-    def get_sizehint(self):
-        return -1
-
     def init_from_list_w(self, w_list, list_w):
         raise NotImplementedError
 
-    def clone(self, w_list):
-        storage = w_list.lstorage  # lstorage is tuple, no need to clone
-        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
-                                                         self)
-        w_clone.switch_to_object_strategy()
-        return w_clone
-
-    def copy_into(self, w_list, w_other):
-        w_list.switch_to_object_strategy()
-        w_list.strategy.copy_into(w_list, w_other)
-
     def _resize_hint(self, w_list, hint):
         pass
 
@@ -325,13 +328,6 @@
         w_list.switch_to_object_strategy()
         return w_list.strategy.find(w_list, w_item, start, stop)
 
-    def getitems_copy(self, w_list):
-        w_list.switch_to_object_strategy()
-        return w_list.strategy.getitems_copy(w_list)
-
-    def getstorage_copy(self, w_list):
-        raise NotImplementedError
-
     def append(self, w_list, w_item):
         w_list.switch_to_object_strategy()
         w_list.strategy.append(w_list, w_item)
diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py
--- a/pypy/module/cpyext/test/test_sequence.py
+++ b/pypy/module/cpyext/test/test_sequence.py
@@ -226,6 +226,15 @@
         w_l.inplace_mul(2)
         assert space.int_w(space.len(w_l)) == 10
 
+    def test_getstorage_copy(self, space, api):
+        w = space.wrap
+        w_l = w([1, 2, 3, 4])
+        api.PySequence_Fast(w_l, "foo") # converts
+
+        w_l1 = w([])
+        space.setitem(w_l1, space.newslice(w(0), w(0), w(1)), w_l)
+        assert map(space.unwrap, space.unpackiterable(w_l1)) == [1, 2, 3, 4]
+
 
 class AppTestSequenceObject(AppTestCpythonExtensionBase):
     def test_fast(self):
diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -1671,18 +1671,6 @@
         else:
             not_implemented("save_into_mem size = %d" % size)
 
-    def _genop_getfield(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc, sign_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
-        source_addr = AddressLoc(base_loc, ofs_loc)
-        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
-
-    genop_getfield_gc_i = _genop_getfield
-    genop_getfield_gc_r = _genop_getfield
-    genop_getfield_gc_f = _genop_getfield
-    genop_getfield_raw_i = _genop_getfield
-    genop_getfield_raw_f = _genop_getfield
-
     def _genop_gc_load(self, op, arglocs, resloc):
         base_loc, ofs_loc, size_loc, sign_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -1303,7 +1303,7 @@
         self.rm.possibly_free_var(tmpbox_high)
 
     def compute_hint_frame_locations(self, operations):
-        # optimization only: fill in the 'hint_frame_locations' dictionary
+        # optimization only: fill in the 'hint_frame_pos' dictionary
         # of 'fm' based on the JUMP at the end of the loop, by looking
         # at where we would like the boxes to be after the jump.
         op = operations[-1]
@@ -1318,7 +1318,7 @@
             self._compute_hint_frame_locations_from_descr(descr)
         #else:
         #   The loop ends in a JUMP going back to a LABEL in the same loop.
-        #   We cannot fill 'hint_frame_locations' immediately, but we can
+        #   We cannot fill 'hint_frame_pos' immediately, but we can
         #   wait until the corresponding consider_label() to know where the
         #   we would like the boxes to be after the jump.
 
diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py
--- a/rpython/rlib/rposix.py
+++ b/rpython/rlib/rposix.py
@@ -205,6 +205,18 @@
         if not is_valid_fd(fd):
             from errno import EBADF
             raise OSError(EBADF, 'Bad file descriptor')
+
+    def _bound_for_write(fd, count):
+        if count > 32767 and c_isatty(fd):
+            # CPython Issue #11395, PyPy Issue #2636: the Windows console
+            # returns an error (12: not enough space error) on writing into
+            # stdout if stdout mode is binary and the length is greater than
+            # 66,000 bytes (or less, depending on heap usage).  Can't easily
+            # test that, because we need 'fd' to be non-redirected...
+            count = 32767
+        elif count > 0x7fffffff:
+            count = 0x7fffffff
+        return count
 else:
     def is_valid_fd(fd):
         return 1
@@ -213,6 +225,9 @@
     def validate_fd(fd):
         pass
 
+    def _bound_for_write(fd, count):
+        return count
+
 def closerange(fd_low, fd_high):
     # this behaves like os.closerange() from Python 2.6.
     for fd in xrange(fd_low, fd_high):
@@ -449,6 +464,7 @@
 def write(fd, data):
     count = len(data)
     validate_fd(fd)
+    count = _bound_for_write(fd, count)
     with rffi.scoped_nonmovingbuffer(data) as buf:
         return handle_posix_error('write', c_write(fd, buf, count))
 
diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -710,7 +710,8 @@
         size, _ = expected_size_and_sign
         return lltype.FixedSizeArray(fieldtype.OF, size/_sizeof(fieldtype.OF))
     raise TypeError("conflict between translating python and compiler field"
-                    " type %r for %r" % (fieldtype, fieldname))
+                    " type %r for symbol %r, expected size+sign %r" % (
+                        fieldtype, fieldname, expected_size_and_sign))
 
 def expose_value_as_rpython(value):
     if intmask(value) == value:

From pypy.commits at gmail.com  Sun Aug 20 13:09:44 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:44 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: merge default
Message-ID: <5999c258.53e81c0a.90271.e492@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92191:d88eaeb16464
Date: 2017-08-20 16:31 +0200
http://bitbucket.org/pypy/pypy/changeset/d88eaeb16464/

Log:	merge default

diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -518,7 +518,7 @@
         clt.frame_info = rffi.cast(jitframe.JITFRAMEINFOPTR, frame_info)
         clt.frame_info.clear() # for now
 
-        if log:
+        if log or self._debug:
             number = looptoken.number
             operations = self._inject_debugging_code(looptoken, operations,
                                                      'e', number)
@@ -622,7 +622,7 @@
                 faildescr.adr_jump_offset)
         self.mc.force_frame_size(DEFAULT_FRAME_BYTES)
         descr_number = compute_unique_id(faildescr)
-        if log:
+        if log or self._debug:
             operations = self._inject_debugging_code(faildescr, operations,
                                                      'b', descr_number)
         arglocs = self.rebuild_faillocs_from_descr(faildescr, inputargs)

From pypy.commits at gmail.com  Sun Aug 20 13:09:46 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:46 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: typo
Message-ID: <5999c25a.919bdf0a.c7a5.f22f@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92192:9b96099898a8
Date: 2017-08-20 16:33 +0200
http://bitbucket.org/pypy/pypy/changeset/9b96099898a8/

Log:	typo

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -586,7 +586,7 @@
 
     def _sync_var(self, v):
         if not self.frame_manager.get(v):
-            self.num_moves_calls += 1
+            self.assembler.num_moves_calls += 1
             reg = self.reg_bindings[v]
             to = self.frame_manager.loc(v)
             self.assembler.regalloc_mov(reg, to)

From pypy.commits at gmail.com  Sun Aug 20 13:09:50 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:50 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: merge default
Message-ID: <5999c25e.4d86df0a.1fabc.89da@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92194:385e5a5df482
Date: 2017-08-20 19:07 +0200
http://bitbucket.org/pypy/pypy/changeset/385e5a5df482/

Log:	merge default

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -567,10 +567,11 @@
             self.reg_bindings[result_v] = loc
             return loc
         if v not in self.reg_bindings:
+            # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(v, forbidden_vars)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-        assert v in self.reg_bindings
+            return loc
         if self.longevity[v].last_usage > self.position:
             # we need to find a new place for variable v and
             # store result in the same place

From pypy.commits at gmail.com  Sun Aug 20 13:09:48 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sun, 20 Aug 2017 10:09:48 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: test for longevity
 computation
Message-ID: <5999c25c.4d86df0a.1fabc.89d8@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92193:7f318c6a860c
Date: 2017-08-20 18:50 +0200
http://bitbucket.org/pypy/pypy/changeset/7f318c6a860c/

Log:	test for longevity computation

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc_integration.py b/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
@@ -114,6 +114,7 @@
 
     def prepare_loop(self, ops):
         loop = self.parse(ops)
+        self.loop = loop
         regalloc = self.cpu.build_regalloc()
         regalloc.prepare_loop(loop.inputargs, loop.operations,
                               loop.original_jitcell_token, [])
@@ -406,6 +407,30 @@
         assert len(regalloc.rm.reg_bindings) == 0
         assert len(regalloc.fm.bindings) == 4
 
+    def test_longevity(self):
+        ops = """
+        [i0, i1, i2, i3, i4]
+        i5 = int_add(i0, i1)
+        i6 = int_is_true(i5)
+        guard_true(i6) [i0, i4]
+        jump(i5, i1, i2, i3, i5)
+        """
+        regalloc = self.prepare_loop(ops)
+        i0, i1, i2, i3, i4 = self.loop.inputargs
+        i5 = self.loop.operations[0]
+        longevity = regalloc.longevity
+        longevity[i0].last_usage == 2
+        longevity[i0].last_real_usage == 0
+        longevity[i1].last_usage == 3
+        longevity[i1].last_real_usage == 0
+        longevity[i2].last_usage == 3
+        longevity[i2].last_real_usage == 0
+        longevity[i3].last_usage == 3
+        longevity[i3].last_real_usage == 0
+        longevity[i4].last_usage == 3
+        longevity[i4].last_real_usage == -1
+        longevity[i5].last_usage == 3
+        longevity[i5].last_real_usage == 2
 
 class TestRegallocCompOps(BaseTestRegalloc):
 

From pypy.commits at gmail.com  Mon Aug 21 06:46:09 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 21 Aug 2017 03:46:09 -0700 (PDT)
Subject: [pypy-commit] pypy default: make error message compliant with
 cPickle, low hanging fruit
Message-ID: <599ab9f1.81131c0a.74668.4223@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92196:52ae4608d26f
Date: 2017-08-21 13:43 +0300
http://bitbucket.org/pypy/pypy/changeset/52ae4608d26f/

Log:	make error message compliant with cPickle, low hanging fruit

diff --git a/lib_pypy/cPickle.py b/lib_pypy/cPickle.py
--- a/lib_pypy/cPickle.py
+++ b/lib_pypy/cPickle.py
@@ -116,10 +116,20 @@
 
 @builtinify
 def dump(obj, file, protocol=None):
+    if protocol > HIGHEST_PROTOCOL:
+        # use cPickle error message, not pickle.py one
+        raise ValueError("pickle protocol %d asked for; "
+                     "the highest available protocol is %d" % (
+                     protocol, HIGHEST_PROTOCOL))
     Pickler(file, protocol).dump(obj)
 
 @builtinify
 def dumps(obj, protocol=None):
+    if protocol > HIGHEST_PROTOCOL:
+        # use cPickle error message, not pickle.py one
+        raise ValueError("pickle protocol %d asked for; "
+                     "the highest available protocol is %d" % (
+                     protocol, HIGHEST_PROTOCOL))
     file = StringIO()
     Pickler(file, protocol).dump(obj)
     return file.getvalue()

From pypy.commits at gmail.com  Mon Aug 21 06:46:11 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 21 Aug 2017 03:46:11 -0700 (PDT)
Subject: [pypy-commit] pypy default: test,
 fix for Py_RichCompareBool(float('nan'), float('nan'))
Message-ID: <599ab9f3.57b61c0a.cf76d.9b88@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92197:3e2341208fe6
Date: 2017-08-21 13:44 +0300
http://bitbucket.org/pypy/pypy/changeset/3e2341208fe6/

Log:	test, fix for Py_RichCompareBool(float('nan'), float('nan'))

diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -305,7 +305,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
+def PyObject_RichCompareBool(space, w_o1, w_o2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -315,13 +315,13 @@
     opid."""
     # Quick result when objects are the same.
     # Guarantees that identity implies equality.
-    if ref1 is ref2:
+    if space.is_w(w_o1, w_o2):
         opid = rffi.cast(lltype.Signed, opid_int)
         if opid == Py_EQ:
             return 1
         if opid == Py_NE:
             return 0 
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
+    w_res = PyObject_RichCompare(space, w_o1, w_o2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -408,7 +408,7 @@
         Py_buffer passed to it.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyString_FromString("hello, world.");
@@ -460,7 +460,7 @@
         object.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyString_FromString("hello, world.");
@@ -506,7 +506,7 @@
         PyBuffer_FillInfo fails if WRITABLE is passed but object is readonly.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyString_FromString("hello, world.");
@@ -533,7 +533,7 @@
         decremented by PyBuffer_Release.
         """
         module = self.import_extension('foo', [
-                ("release", "METH_VARARGS",
+                ("release", "METH_NOARGS",
                  """
     Py_buffer buf;
     buf.obj = PyString_FromString("release me!");
@@ -553,3 +553,19 @@
                  """)])
         assert module.release() is None
 
+
+class AppTestPyBuffer_Release(AppTestCpythonExtensionBase):
+    def test_richcomp_nan(self):
+        module = self.import_extension('foo', [
+               ("comp_eq", "METH_VARARGS",
+                """
+                PyObject *a = PyTuple_GetItem(args, 0);
+                PyObject *b = PyTuple_GetItem(args, 1);
+                int res = PyObject_RichCompareBool(a, b, Py_EQ);
+                return PyLong_FromLong(res);  
+                """),])
+        a = float('nan')
+        b = float('nan')
+        assert a is b
+        res = module.comp_eq(a, b)
+        assert res == 1

From pypy.commits at gmail.com  Mon Aug 21 06:46:07 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 21 Aug 2017 03:46:07 -0700 (PDT)
Subject: [pypy-commit] pypy default: delete implemented function from unused
 stubs.py
Message-ID: <599ab9ef.6ea9df0a.a0568.3cb4@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92195:0f254609cb4f
Date: 2017-08-20 17:39 +0300
http://bitbucket.org/pypy/pypy/changeset/0f254609cb4f/

Log:	delete implemented function from unused stubs.py

diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -226,11 +226,6 @@
     representation."""
     raise NotImplementedError
 
- at cpython_api([Py_complex], PyObject)
-def PyComplex_FromCComplex(space, v):
-    """Create a new Python complex number object from a C Py_complex value."""
-    raise NotImplementedError
-
 @cpython_api([rffi.CCHARP, rffi.CCHARPP], rffi.DOUBLE, error=CANNOT_FAIL)
 def PyOS_ascii_strtod(space, nptr, endptr):
     """Convert a string to a double. This function behaves like the Standard C

From pypy.commits at gmail.com  Mon Aug 21 06:52:40 2017
From: pypy.commits at gmail.com (mattip)
Date: Mon, 21 Aug 2017 03:52:40 -0700 (PDT)
Subject: [pypy-commit] pypy.org extradoc: remove outdated mention of
 pypy/numpy, regenerate
Message-ID: <599abb78.c69edf0a.51976.ad62@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: extradoc
Changeset: r897:df872ae704e1
Date: 2017-08-21 13:51 +0300
http://bitbucket.org/pypy/pypy.org/changeset/df872ae704e1/

Log:	remove outdated mention of pypy/numpy, regenerate

diff --git a/compat.html b/compat.html
--- a/compat.html
+++ b/compat.html
@@ -97,7 +97,6 @@
 <li><tt class="docutils literal">cPickle, ctypes, datetime, dbm, _functools, grp, readline, resource, sqlite3, syslog</tt></li>
 </ul>
 <p>All modules that are pure python in CPython of course work.</p>
-<p>Numpy support is not complete. We maintain our own fork of numpy for now, further instructions can be found at <a class="reference external" href="https://bitbucket.org/pypy/numpy.git">https://bitbucker.org/pypy/numpy.git</a>.</p>
 <p>Python libraries known to work under PyPy (the list is not exhaustive).
 A <a class="reference external" href="http://packages.pypy.org">fuller list</a> is available.</p>
 <ul class="simple">
diff --git a/download.html b/download.html
--- a/download.html
+++ b/download.html
@@ -120,8 +120,8 @@
 <div class="section" id="python2-7-compatible-pypy-5-8-0">
 <span id="release"></span><h1>Python2.7 compatible PyPy 5.8.0</h1>
 <ul class="simple">
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux32.tar.bz2">Linux x86 binary (32bit, tar.bz2 built on Ubuntu 12.04 - 14.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux64.tar.bz2">Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 14.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
+<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux32.tar.bz2">Linux x86 binary (32bit, tar.bz2 built on Ubuntu 12.04 - 16.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
+<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux64.tar.bz2">Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 16.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux-armhf-raspbian.tar.bz2">ARM Hardfloat Linux binary (ARMHF/gnueabihf, tar.bz2, Raspbian)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux-armhf-raring.tar.bz2">ARM Hardfloat Linux binary (ARMHF/gnueabihf, tar.bz2, Ubuntu Raring)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux-armel.tar.bz2">ARM Softfloat Linux binary (ARMEL/gnueabi, tar.bz2,  Ubuntu Precise)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
@@ -144,7 +144,7 @@
 PyPy 2.  You are welcome to use it anyway; if you're lucky it will
 be fast in your case.</p>
 <ul class="simple">
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.8.0-linux64.tar.bz2">Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 14.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
+<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.8.0-linux64.tar.bz2">Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 16.04)</a> (see <tt class="docutils literal">[1]</tt> below)</li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.8.0-src.tar.bz2">Source (tar.bz2)</a></li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.8.0-src.zip">Source (zip)</a></li>
 <li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads">All our downloads,</a> including previous versions.  We also have a
@@ -166,7 +166,7 @@
 <p>This is a special version of PyPy!  See the <a class="reference external" href="http://doc.pypy.org/en/latest/stm.html">Software Transactional
 Memory</a> (STM) documentation.</p>
 <ul class="simple">
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-stm-2.5.1-linux64.tar.bz2">PyPy-STM Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 14.04)</a></li>
+<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-stm-2.5.1-linux64.tar.bz2">PyPy-STM Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 16.04)</a></li>
 </ul>
 </div>
 <div class="section" id="other-versions">
diff --git a/source/compat.txt b/source/compat.txt
--- a/source/compat.txt
+++ b/source/compat.txt
@@ -39,10 +39,6 @@
 
 All modules that are pure python in CPython of course work.
 
-Numpy support is not complete. We maintain our own fork of numpy for now, further instructions can be found at `https://bitbucker.org/pypy/numpy.git`__.
-
-.. __: https://bitbucket.org/pypy/numpy.git
-
 Python libraries known to work under PyPy (the list is not exhaustive).
 A `fuller list`_ is available.
 

From pypy.commits at gmail.com  Mon Aug 21 06:58:41 2017
From: pypy.commits at gmail.com (tobweber)
Date: Mon, 21 Aug 2017 03:58:41 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-efficient-serial-execution-master: Merge
 timing events enum so that all branches share the same interface with
 print_stm_log.py
Message-ID: <599abce1.579c1c0a.492e0.d8b1@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-efficient-serial-execution-master
Changeset: r2151:4a71ee20626e
Date: 2017-08-05 14:17 +0200
http://bitbucket.org/pypy/stmgc/changeset/4a71ee20626e/

Log:	Merge timing events enum so that all branches share the same
	interface with print_stm_log.py

diff --git a/c8/stmgc.h b/c8/stmgc.h
--- a/c8/stmgc.h
+++ b/c8/stmgc.h
@@ -201,7 +201,7 @@
 /* ==================== PUBLIC API ==================== */
 
 /* Number of segments (i.e. how many transactions can be executed in
-   parallel, in maximum).  If you try to start transactions in more
+   parallel, at maximum).  If you try to start transactions in more
    threads than the number of segments, it will block, waiting for the
    next segment to become free.
 */
@@ -574,21 +574,49 @@
     STM_GC_MAJOR_START,
     STM_GC_MAJOR_DONE,
 
+    /* execution duration profiling events */
+    STM_WARMUP_COMPLETE,
+
+    STM_DURATION_START_TRX,
+    STM_DURATION_WRITE_GC_ONLY,
+    STM_DURATION_WRITE_SLOWPATH,
+    STM_DURATION_VALIDATION,
+    STM_DURATION_CREATE_CLE,
+    STM_DURATION_COMMIT_EXCEPT_GC,
+    STM_DURATION_MINOR_GC,
+    STM_DURATION_MAJOR_GC_LOG_ONLY,
+    STM_DURATION_MAJOR_GC_FULL,
+
+    STM_SINGLE_THREAD_MODE_ON,
+    STM_SINGLE_THREAD_MODE_OFF,
+    STM_SINGLE_THREAD_MODE_ADAPTIVE,
+
     _STM_EVENT_N
 };
 
-#define STM_EVENT_NAMES                         \
-    "transaction start",                        \
-    "transaction commit",                       \
-    "transaction abort",                        \
-    "contention write read",                    \
-    "wait free segment",                        \
-    "wait other inevitable",                    \
-    "wait done",                                \
-    "gc minor start",                           \
-    "gc minor done",                            \
-    "gc major start",                           \
-    "gc major done"
+#define STM_EVENT_NAMES                             \
+    "transaction start",                            \
+    "transaction commit",                           \
+    "transaction abort",                            \
+    "contention write read",                        \
+    "wait free segment",                            \
+    "wait other inevitable",                        \
+    "wait done",                                    \
+    "gc minor start",                               \
+    "gc minor done",                                \
+    "gc major start",                               \
+    "gc major done",                                \
+    /* names of duration events */                  \
+    "marks completion of benchmark warm up phase"   \
+    "duration of transaction start",                \
+    "duration of gc due to write",                  \
+    "duration of write slowpath",                   \
+    "duration of validation",                       \
+    "duration of commit log entry creation",        \
+    "duration of commit except gc",                 \
+    "duration of minor gc",                         \
+    "duration of major gc doing log clean up only", \
+    "duration of full major gc"
 
 /* The markers pushed in the shadowstack are an odd number followed by a
    regular object pointer. */

From pypy.commits at gmail.com  Mon Aug 21 06:58:43 2017
From: pypy.commits at gmail.com (tobweber)
Date: Mon, 21 Aug 2017 03:58:43 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-efficient-serial-execution-master: Merge TCP
 style optimization
Message-ID: <599abce3.4981df0a.601c0.7fa6@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-efficient-serial-execution-master
Changeset: r2152:d56fd821ed46
Date: 2017-08-21 12:10 +0200
http://bitbucket.org/pypy/stmgc/changeset/d56fd821ed46/

Log:	Merge TCP style optimization

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -1117,11 +1117,9 @@
     _do_start_transaction(tl);
 
     STM_PSEGMENT->commit_if_not_atomic = false;
-    if (repeat_count == 0) {  /* else, 'nursery_mark' was already set
-                                 in abort_data_structures_from_segment_num() */
-        STM_SEGMENT->nursery_mark = ((stm_char *)_stm_nursery_start +
-                                     stm_fill_mark_nursery_bytes);
-    }
+    STM_SEGMENT->nursery_mark = ((stm_char *)_stm_nursery_start +
+                                        stm_get_transaction_length(tl));
+
     return repeat_count;
 }
 
@@ -1304,6 +1302,8 @@
 
     s_mutex_unlock();
 
+    stm_transaction_length_handle_validation(thread_local_for_logging, false);
+
     /* between transactions, call finalizers. this will execute
        a transaction itself */
     if (tl != NULL)
@@ -1468,22 +1468,6 @@
     if (pseg->active_queues)
         queues_deactivate_all(pseg, /*at_commit=*/false);
 
-
-    /* Set the next nursery_mark: first compute the value that
-       nursery_mark must have had at the start of the aborted transaction */
-    stm_char *old_mark =pseg->pub.nursery_mark + pseg->total_throw_away_nursery;
-
-    /* This means that the limit, in term of bytes, was: */
-    uintptr_t old_limit = old_mark - (stm_char *)_stm_nursery_start;
-
-    /* If 'total_throw_away_nursery' is smaller than old_limit, use that */
-    if (pseg->total_throw_away_nursery < old_limit)
-        old_limit = pseg->total_throw_away_nursery;
-
-    /* Now set the new limit to 90% of the old limit */
-    pseg->pub.nursery_mark = ((stm_char *)_stm_nursery_start +
-                              (uintptr_t)(old_limit * 0.9));
-
 #ifdef STM_NO_AUTOMATIC_SETJMP
     did_abort = 1;
 #endif
@@ -1518,6 +1502,8 @@
     tl->self_or_0_if_atomic = (intptr_t)tl;   /* clear the 'atomic' flag */
     STM_PSEGMENT->atomic_nesting_levels = 0;
 
+    stm_transaction_length_handle_validation(tl, true);
+
     if (tl->mem_clear_on_abort)
         memset(tl->mem_clear_on_abort, 0, tl->mem_bytes_to_clear_on_abort);
     if (tl->mem_reset_on_abort) {
diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -4,6 +4,8 @@
 #endif
 
 #include "finalizer.h"
+#include <math.h>
+#include <inttypes.h>
 
 /************************************************************/
 
@@ -13,14 +15,79 @@
 
 static uintptr_t _stm_nursery_start;
 
+#define DEFAULT_FILL_MARK_NURSERY_BYTES (NURSERY_SIZE / 4)
 
-#define DEFAULT_FILL_MARK_NURSERY_BYTES   (NURSERY_SIZE / 4)
+// corresponds to ~4 GB
+#define LARGE_FILL_MARK_NURSERY_BYTES   0x100000000L
 
-uintptr_t stm_fill_mark_nursery_bytes = DEFAULT_FILL_MARK_NURSERY_BYTES;
+// corresponds to ~4 MB nursery fill
+#define STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH (0.001)
+// corresponds to ~400 KB nursery fill
+#define STM_MIN_RELATIVE_TRANSACTION_LENGTH (0.0001)
+
+#define BACKOFF_COUNT (20)
+#define BACKOFF_MULTIPLIER (BACKOFF_COUNT / -log10(STM_MIN_RELATIVE_TRANSACTION_LENGTH))
+
+static inline void set_backoff(stm_thread_local_t *tl, double rel_trx_len) {
+    /* the shorter the trx, the more backoff:
+    think a*x + b = backoff, x := -log(rel-trx-len),
+    backoff is <BACKOFF_COUNT> + b at default trx length,
+    linear decrease to b at max trx length */
+    const int b = 5;
+    int new_backoff = (int)((BACKOFF_MULTIPLIER * -log10(rel_trx_len)) + b);
+    tl->transaction_length_backoff = new_backoff;
+    // printf("thread %d, backoff %d\n", tl->thread_local_counter, tl->transaction_length_backoff);
+    tl->linear_transaction_length_increment = rel_trx_len / new_backoff;
+}
+
+static inline double get_new_transaction_length(stm_thread_local_t *tl, bool aborts) {
+    const int multiplier = 2;
+    double previous = tl->relative_transaction_length;
+    double new;
+    if (aborts) {
+        new = previous / multiplier;
+        if (new < STM_MIN_RELATIVE_TRANSACTION_LENGTH) {
+            new = STM_MIN_RELATIVE_TRANSACTION_LENGTH;
+        }
+        set_backoff(tl, new);
+    } else if (tl->transaction_length_backoff == 0) {
+        // backoff counter is zero, exponential increase up to 1
+        new = previous * multiplier;
+        if (new > 1) {
+            new = 1;
+        }
+        if (tl->linear_transaction_length_increment != 0) {
+            // thread had to abort before: slow start
+            set_backoff(tl, new);
+        }
+    } else { // not abort and backoff != 0
+        // in backoff, linear increase up to 1
+        new = previous + tl->linear_transaction_length_increment;
+        if (new > 1) {
+            new = 1;
+        }
+        tl->transaction_length_backoff -= 1;
+    }
+    return new;
+}
+
+static inline void stm_transaction_length_handle_validation(stm_thread_local_t *tl, bool aborts) {
+    tl->relative_transaction_length = get_new_transaction_length(tl, aborts);
+}
+
+static inline uintptr_t stm_get_transaction_length(stm_thread_local_t *tl) {
+    double relative_additional_length = tl->relative_transaction_length;
+    publish_custom_value_event(
+        relative_additional_length, STM_SINGLE_THREAD_MODE_ADAPTIVE);
+    uintptr_t result =
+        (uintptr_t)(LARGE_FILL_MARK_NURSERY_BYTES * relative_additional_length);
+    // printf("%020" PRIxPTR "\n", result);
+    return result;
+}
+
 
 /************************************************************/
 
-
 static void setup_nursery(void)
 {
     assert(_STM_FAST_ALLOC <= NURSERY_SIZE);
diff --git a/c8/stm/nursery.h b/c8/stm/nursery.h
--- a/c8/stm/nursery.h
+++ b/c8/stm/nursery.h
@@ -56,4 +56,7 @@
 static inline struct object_s *mark_loc(object_t *obj);
 static inline bool _is_from_same_transaction(object_t *obj);
 
+static inline void stm_transaction_length_handle_validation(stm_thread_local_t *tl, bool aborts);
+static inline uintptr_t stm_get_transaction_length(stm_thread_local_t *tl);
+
 #endif
diff --git a/c8/stm/setup.c b/c8/stm/setup.c
--- a/c8/stm/setup.c
+++ b/c8/stm/setup.c
@@ -245,6 +245,12 @@
        numbers automatically. */
     tl->last_associated_segment_num = num + 1;
     tl->thread_local_counter = ++thread_local_counters;
+
+    /* init adaptive transaction length mode */
+    tl->relative_transaction_length = STM_DEFAULT_RELATIVE_TRANSACTION_LENGTH;
+    tl->transaction_length_backoff = 0;
+    tl->linear_transaction_length_increment = 0;
+
     *_get_cpth(tl) = pthread_self();
     _init_shadow_stack(tl);
     set_gs_register(get_segment_base(num + 1));
diff --git a/c8/stm/sync.c b/c8/stm/sync.c
--- a/c8/stm/sync.c
+++ b/c8/stm/sync.c
@@ -176,6 +176,16 @@
 
 /************************************************************/
 
+static uint8_t number_of_segments_in_use(void) {
+    uint8_t result = 0;
+    int num;
+    for (num = 1; num < NB_SEGMENTS; num++) {
+        if (sync_ctl.in_use1[num] > 0) {
+            result++;
+        }
+    }
+    return result;
+}
 
 #if 0
 void stm_wait_for_current_inevitable_transaction(void)
@@ -202,7 +212,6 @@
 }
 #endif
 
-
 static void acquire_thread_segment(stm_thread_local_t *tl)
 {
     /* This function acquires a segment for the currently running thread,
diff --git a/c8/stm/sync.h b/c8/stm/sync.h
--- a/c8/stm/sync.h
+++ b/c8/stm/sync.h
@@ -22,6 +22,7 @@
 static void set_gs_register(char *value);
 static void ensure_gs_register(long segnum);
 
+static uint8_t number_of_segments_in_use(void);
 
 /* acquire and release one of the segments for running the given thread
    (must have the mutex acquired!) */
diff --git a/c8/stmgc.h b/c8/stmgc.h
--- a/c8/stmgc.h
+++ b/c8/stmgc.h
@@ -88,6 +88,13 @@
     struct stm_thread_local_s *prev, *next;
     intptr_t self_or_0_if_atomic;
     void *creating_pthread[2];
+    /* == adaptive single thread mode == */
+    /* factor that is multiplied with max transaction length before the start of the next transaction on this thread */
+    double relative_transaction_length;
+    /* when zero, transaction length may increase exponentially, otherwise transaction length may only increase linearly. is (re-)set to some value upon abort and counted down until zero upon successful validation. */
+    int transaction_length_backoff;
+    /* during the backoff, transaction length may increase linearly by this increment on every successful validation */
+    double linear_transaction_length_increment;
 } stm_thread_local_t;
 
 

From pypy.commits at gmail.com  Mon Aug 21 06:58:45 2017
From: pypy.commits at gmail.com (tobweber)
Date: Mon, 21 Aug 2017 03:58:45 -0700 (PDT)
Subject: [pypy-commit] stmgc c8-efficient-serial-execution-master: Fix trx
 length update depends on instrumentation for thread local reference and
 remove logging of trx length
Message-ID: <599abce5.8c8f1c0a.fcf57.08c6@mx.google.com>

Author: Tobias Weber <tobias_weber89 at gmx.de>
Branch: c8-efficient-serial-execution-master
Changeset: r2153:cadbddf81079
Date: 2017-08-21 12:40 +0200
http://bitbucket.org/pypy/stmgc/changeset/cadbddf81079/

Log:	Fix trx length update depends on instrumentation for thread local
	reference and remove logging of trx length

diff --git a/c8/stm/core.c b/c8/stm/core.c
--- a/c8/stm/core.c
+++ b/c8/stm/core.c
@@ -1255,6 +1255,8 @@
     bool was_inev = STM_PSEGMENT->transaction_state == TS_INEVITABLE;
     _validate_and_add_to_commit_log();
 
+
+    stm_thread_local_t *tl_for_trx_len = STM_SEGMENT->running_thread;
     if (external) {
         /* from this point on, unlink the original 'stm_thread_local_t *'
            from its segment.  Better do it as soon as possible, because
@@ -1302,7 +1304,7 @@
 
     s_mutex_unlock();
 
-    stm_transaction_length_handle_validation(thread_local_for_logging, false);
+    stm_transaction_length_handle_validation(tl_for_trx_len, false);
 
     /* between transactions, call finalizers. this will execute
        a transaction itself */
diff --git a/c8/stm/nursery.c b/c8/stm/nursery.c
--- a/c8/stm/nursery.c
+++ b/c8/stm/nursery.c
@@ -77,8 +77,6 @@
 
 static inline uintptr_t stm_get_transaction_length(stm_thread_local_t *tl) {
     double relative_additional_length = tl->relative_transaction_length;
-    publish_custom_value_event(
-        relative_additional_length, STM_SINGLE_THREAD_MODE_ADAPTIVE);
     uintptr_t result =
         (uintptr_t)(LARGE_FILL_MARK_NURSERY_BYTES * relative_additional_length);
     // printf("%020" PRIxPTR "\n", result);

From pypy.commits at gmail.com  Mon Aug 21 09:28:00 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 21 Aug 2017 06:28:00 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: tweak: pick the
 longest-living useless variable
Message-ID: <599adfe0.86acdf0a.99dd0.295f@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92198:dfada6cd5c1a
Date: 2017-08-21 15:27 +0200
http://bitbucket.org/pypy/pypy/changeset/dfada6cd5c1a/

Log:	tweak: pick the longest-living useless variable

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -420,8 +420,8 @@
         """
         cur_max_age = -1
         candidate = None
-        # YYY we should pick a variable to spill that is only used in failargs
-        # from now on
+        cur_max_age_failargs = -1
+        candidate_from_failargs = None
         for next in self.reg_bindings:
             reg = self.reg_bindings[next]
             if next in forbidden_vars:
@@ -434,18 +434,22 @@
             if need_lower_byte and reg in self.no_lower_byte_regs:
                 continue
             lifetime = self.longevity[next]
+            max_age = lifetime.last_usage
             if lifetime.is_last_real_use_before(self.position):
                 # this variable has no "real" use as an argument to an op left
                 # it is only used in failargs, and maybe in a jump. spilling is
                 # fine
-                return next
-            max_age = lifetime.last_usage
+                if cur_max_age_failargs < max_age:
+                    cur_max_age_failargs = max_age
+                    candidate_from_failargs = next
             if cur_max_age < max_age:
                 cur_max_age = max_age
                 candidate = next
-        if candidate is None:
-            raise NoVariableToSpill
-        return candidate
+        if candidate_from_failargs is not None:
+            return candidate_from_failargs
+        if candidate is not None:
+            return candidate
+        raise NoVariableToSpill
 
     def force_allocate_reg(self, v, forbidden_vars=[], selected_reg=None,
                            need_lower_byte=False):
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -397,9 +397,9 @@
 
     def test_spill_useless_vars_first(self):
         b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
-        longevity = {b0: Lifetime(0, 5), b1: Lifetime(0, 5),
-                     # b3 becomes useless but b2 lives longer
-                     b3: Lifetime(0, 5, 3), b2: Lifetime(0, 6),
+        longevity = {b0: Lifetime(0, 5), b1: Lifetime(0, 10),
+                     # b2 and b3 become useless but b3 lives longer
+                     b3: Lifetime(0, 7, 3), b2: Lifetime(0, 6, 3),
                      b4: Lifetime(4, 5), b5: Lifetime(4, 7)}
         fm = TFrameManager()
         asm = MockAsm()

From pypy.commits at gmail.com  Mon Aug 21 14:55:59 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 21 Aug 2017 11:55:59 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: refactoring: compute
 Lifetime objects directly
Message-ID: <599b2cbf.42a7df0a.b750.6883@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92199:a163b0deca41
Date: 2017-08-21 16:24 +0200
http://bitbucket.org/pypy/pypy/changeset/a163b0deca41/

Log:	refactoring: compute Lifetime objects directly

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -806,8 +806,11 @@
             assert op.numargs() == 1
             return [self.loc(op.getarg(0))]
 
+UNDEF_POS = -42
+
 class Lifetime(object):
-    def __init__(self, definition_pos, last_usage, last_real_usage=-42):
+    def __init__(self, definition_pos=UNDEF_POS, last_usage=UNDEF_POS,
+                 last_real_usage=UNDEF_POS):
         # all positions are indexes into the operations list
 
         # the position where the variable is defined
@@ -815,7 +818,7 @@
         # the position where the variable is last used. this includes failargs
         # and jumps
         self.last_usage = last_usage
-        if last_real_usage == -42:
+        if last_real_usage == UNDEF_POS:
             last_real_usage = last_usage
         # last *real* usage, ie as an argument to an operation
         # after last_real_usage and last_usage it does not matter whether the
@@ -832,49 +835,41 @@
     # compute a dictionary that maps variables to Lifetime information
     # if a variable is not in the dictionary, it's operation is dead because
     # it's side-effect-free and the result is unused
-    last_used = {}
-    last_real_usage = {}
+    longevity = {}
     for i in range(len(operations)-1, -1, -1):
         op = operations[i]
-        if op.type != 'v':
-            if op not in last_used and rop.has_no_side_effect(op.opnum):
+        opnum = op.getopnum()
+        if op not in longevity:
+            if op.type != 'v' and rop.has_no_side_effect(opnum):
+                # result not used, operation has no side-effect, it can be
+                # removed
                 continue
-        opnum = op.getopnum()
+            longevity[op] = Lifetime(definition_pos=i, last_usage=i)
+        else:
+            longevity[op].definition_pos = i
         for j in range(op.numargs()):
             arg = op.getarg(j)
             if isinstance(arg, Const):
                 continue
-            if arg not in last_used:
-                last_used[arg] = i
+            if arg not in longevity:
+                lifetime = longevity[arg] = Lifetime(last_usage=i)
+            else:
+                lifetime = longevity[arg]
             if opnum != rop.JUMP and opnum != rop.LABEL:
-                if arg not in last_real_usage:
-                    last_real_usage[arg] = i
+                if lifetime.last_real_usage == UNDEF_POS:
+                    lifetime.last_real_usage = i
         if rop.is_guard(op.opnum):
             for arg in op.getfailargs():
                 if arg is None: # hole
                     continue
                 assert not isinstance(arg, Const)
-                if arg not in last_used:
-                    last_used[arg] = i
+                if arg not in longevity:
+                    longevity[arg] = Lifetime(last_usage=i)
     #
-    longevity = {}
-    for i, arg in enumerate(operations):
-        if arg.type != 'v' and arg in last_used:
-            assert not isinstance(arg, Const)
-            assert i < last_used[arg]
-            longevity[arg] = Lifetime(
-                i, last_used[arg], last_real_usage.get(arg, -1))
-            del last_used[arg]
     for arg in inputargs:
         assert not isinstance(arg, Const)
-        if arg not in last_used:
-            longevity[arg] = Lifetime(
-                -1, -1, -1)
-        else:
-            longevity[arg] = Lifetime(
-                0, last_used[arg], last_real_usage.get(arg, -1))
-            del last_used[arg]
-    assert len(last_used) == 0
+        if arg not in longevity:
+            longevity[arg] = Lifetime(-1, -1, -1)
 
     if not we_are_translated():
         produced = {}

From pypy.commits at gmail.com  Mon Aug 21 14:56:02 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 21 Aug 2017 11:56:02 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: move this case to the tests,
 where it belongs
Message-ID: <599b2cc2.0a561c0a.90a96.2373@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92200:eeca1d43c304
Date: 2017-08-21 16:32 +0200
http://bitbucket.org/pypy/pypy/changeset/eeca1d43c304/

Log:	move this case to the tests, where it belongs

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -818,8 +818,7 @@
         # the position where the variable is last used. this includes failargs
         # and jumps
         self.last_usage = last_usage
-        if last_real_usage == UNDEF_POS:
-            last_real_usage = last_usage
+
         # last *real* usage, ie as an argument to an operation
         # after last_real_usage and last_usage it does not matter whether the
         # variable is stored on the stack
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1,7 +1,8 @@
 import py
 from rpython.jit.metainterp.history import ConstInt, INT, FLOAT
 from rpython.jit.backend.llsupport.regalloc import FrameManager, LinkedList
-from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan, Lifetime
+from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan,\
+     Lifetime as RealLifetime, UNDEF_POS
 from rpython.jit.metainterp.resoperation import InputArgInt, InputArgRef,\
      InputArgFloat
 
@@ -11,6 +12,13 @@
 def newrefboxes(count):
     return [InputArgRef() for _ in range(count)]
 
+def Lifetime(definition_pos=UNDEF_POS, last_usage=UNDEF_POS,
+             last_real_usage=UNDEF_POS):
+    if last_real_usage == UNDEF_POS:
+        last_real_usage = last_usage
+    return RealLifetime(definition_pos, last_usage, last_real_usage)
+
+
 def boxes_and_longevity(num):
     res = []
     longevity = {}

From pypy.commits at gmail.com  Mon Aug 21 14:56:04 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 21 Aug 2017 11:56:04 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: implement the most common
 spilling heuristic used in linear scan
Message-ID: <599b2cc4.0187df0a.26ae1.c7fb@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92201:a961fe5b9c4a
Date: 2017-08-21 20:55 +0200
http://bitbucket.org/pypy/pypy/changeset/a961fe5b9c4a/

Log:	implement the most common spilling heuristic used in linear scan
	implementations:

	spill the variable where the next use as as argument is the furthest
	away from the current position.

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -416,9 +416,13 @@
 
     def _pick_variable_to_spill(self, v, forbidden_vars, selected_reg=None,
                                 need_lower_byte=False):
-        """ Slightly less silly algorithm.
-        """
-        cur_max_age = -1
+        # try to spill a variable that has no further real usages, ie that only
+        # appears in failargs or in a jump
+        # if that doesn't exist, spill the variable that has a real_usage that
+        # is the furthest away from the current position
+
+        cur_max_use_distance = -1
+        position = self.position
         candidate = None
         cur_max_age_failargs = -1
         candidate_from_failargs = None
@@ -434,17 +438,19 @@
             if need_lower_byte and reg in self.no_lower_byte_regs:
                 continue
             lifetime = self.longevity[next]
-            max_age = lifetime.last_usage
-            if lifetime.is_last_real_use_before(self.position):
+            if lifetime.is_last_real_use_before(position):
                 # this variable has no "real" use as an argument to an op left
                 # it is only used in failargs, and maybe in a jump. spilling is
                 # fine
+                max_age = lifetime.last_usage
                 if cur_max_age_failargs < max_age:
                     cur_max_age_failargs = max_age
                     candidate_from_failargs = next
-            if cur_max_age < max_age:
-                cur_max_age = max_age
-                candidate = next
+            else:
+                use_distance = lifetime.next_real_usage(position) - position
+                if cur_max_use_distance < use_distance:
+                    cur_max_use_distance = use_distance
+                    candidate = next
         if candidate_from_failargs is not None:
             return candidate_from_failargs
         if candidate is not None:
@@ -809,8 +815,7 @@
 UNDEF_POS = -42
 
 class Lifetime(object):
-    def __init__(self, definition_pos=UNDEF_POS, last_usage=UNDEF_POS,
-                 last_real_usage=UNDEF_POS):
+    def __init__(self, definition_pos=UNDEF_POS, last_usage=UNDEF_POS):
         # all positions are indexes into the operations list
 
         # the position where the variable is defined
@@ -819,16 +824,38 @@
         # and jumps
         self.last_usage = last_usage
 
-        # last *real* usage, ie as an argument to an operation
-        # after last_real_usage and last_usage it does not matter whether the
-        # variable is stored on the stack
-        self.last_real_usage = last_real_usage
+        # *real* usages, ie as an argument to an operation (as opposed to jump
+        # arguments or in failargs)
+        self.real_usages = None
 
     def is_last_real_use_before(self, position):
-        return self.last_real_usage <= position
+        if self.real_usages is None:
+            return True
+        return self.real_usages[-1] <= position
+
+    def next_real_usage(self, position):
+        assert position >= self.definition_pos
+        # binary search
+        l = self.real_usages
+        low = 0
+        high = len(l)
+        while low < high:
+            mid = low + (high - low) // 2 # no overflow ;-)
+            if position < l[mid]:
+                high = mid
+            else:
+                low = mid + 1
+        return l[low]
+
+    def _check_invariants(self):
+        assert self.definition_pos <= self.last_usage
+        if self.real_usages is not None:
+            assert sorted(self.real_usages) == self.real_usages
+            assert self.last_usage >= max(self.real_usages)
+            assert self.definition_pos < min(self.real_usages)
 
     def __repr__(self):
-        return "%s:%s(%s)" % (self.definition_pos, self.last_real_usage, self.last_usage)
+        return "%s:%s(%s)" % (self.definition_pos, self.real_usages, self.last_usage)
 
 def compute_vars_longevity(inputargs, operations):
     # compute a dictionary that maps variables to Lifetime information
@@ -855,8 +882,9 @@
             else:
                 lifetime = longevity[arg]
             if opnum != rop.JUMP and opnum != rop.LABEL:
-                if lifetime.last_real_usage == UNDEF_POS:
-                    lifetime.last_real_usage = i
+                if lifetime.real_usages is None:
+                    lifetime.real_usages = []
+                lifetime.real_usages.append(i)
         if rop.is_guard(op.opnum):
             for arg in op.getfailargs():
                 if arg is None: # hole
@@ -868,7 +896,7 @@
     for arg in inputargs:
         assert not isinstance(arg, Const)
         if arg not in longevity:
-            longevity[arg] = Lifetime(-1, -1, -1)
+            longevity[arg] = Lifetime(-1, -1)
 
     if not we_are_translated():
         produced = {}
@@ -879,6 +907,11 @@
                 if not isinstance(arg, Const):
                     assert arg in produced
             produced[op] = None
+    for lifetime in longevity.itervalues():
+        if lifetime.real_usages is not None:
+            lifetime.real_usages.reverse()
+        if not we_are_translated():
+            lifetime._check_invariants()
 
     return longevity
 
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -13,10 +13,14 @@
     return [InputArgRef() for _ in range(count)]
 
 def Lifetime(definition_pos=UNDEF_POS, last_usage=UNDEF_POS,
-             last_real_usage=UNDEF_POS):
-    if last_real_usage == UNDEF_POS:
-        last_real_usage = last_usage
-    return RealLifetime(definition_pos, last_usage, last_real_usage)
+             real_usages=UNDEF_POS):
+    if real_usages == UNDEF_POS:
+        real_usages = last_usage
+    lifetime = RealLifetime(definition_pos, last_usage)
+    if isinstance(real_usages, int):
+        real_usages = [real_usages]
+    lifetime.real_usages = real_usages
+    return lifetime
 
 
 def boxes_and_longevity(num):
@@ -94,6 +98,16 @@
     def regalloc_mov(self, from_loc, to_loc):
         self.moves.append((from_loc, to_loc))
 
+
+def test_lifetime_next_real_usage():
+    lt = RealLifetime(0, 1000)
+    lt.real_usages = [0, 1, 5, 10, 24, 35, 55, 56, 57, 90, 92, 100]
+    for i in range(100):
+        next = lt.next_real_usage(i)
+        assert next in lt.real_usages
+        assert next > i
+        assert lt.real_usages[lt.real_usages.index(next) - 1] <= i
+
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)
@@ -382,7 +396,7 @@
         xrm.loc(f0)
         rm.loc(b0)
         assert fm.get_frame_depth() == 3
-                
+
     def test_spilling(self):
         b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
         longevity = {b0: Lifetime(0, 3), b1: Lifetime(0, 3),
@@ -403,6 +417,27 @@
         assert spilled2 is loc
         rm._check_invariants()
 
+    def test_spilling_furthest_next_real_use(self):
+        b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
+        longevity = {b0: Lifetime(0, 3, [1, 2, 3]), b1: Lifetime(0, 3, [3]),
+                     b3: Lifetime(0, 4, [1, 2, 3, 4]), b2: Lifetime(0, 2),
+                     b4: Lifetime(1, 4), b5: Lifetime(1, 3)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        for b in b0, b1, b2, b3:
+            rm.force_allocate_reg(b)
+        assert len(rm.free_regs) == 0
+        rm.next_instruction()
+        loc = rm.loc(b1)
+        spilled = rm.force_allocate_reg(b4)
+        assert spilled is loc
+        spilled2 = rm.force_allocate_reg(b5)
+        assert spilled2 is loc
+        rm._check_invariants()
+
+
     def test_spill_useless_vars_first(self):
         b0, b1, b2, b3, b4, b5 = newboxes(0, 1, 2, 3, 4, 5)
         longevity = {b0: Lifetime(0, 5), b1: Lifetime(0, 10),
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc_integration.py b/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc_integration.py
@@ -409,28 +409,35 @@
 
     def test_longevity(self):
         ops = """
-        [i0, i1, i2, i3, i4]
-        i5 = int_add(i0, i1)
-        i6 = int_is_true(i5)
-        guard_true(i6) [i0, i4]
-        jump(i5, i1, i2, i3, i5)
+        [i0, i1, i2, i3, i4, i10]
+        i5 = int_add(i0, i1)     # 0
+        i8 = int_add(i0, i1)     # 1 unused result, so not in real_usages
+        i6 = int_is_true(i5)     # 2
+        i11 = int_add(i5, i10)   # 3
+        guard_true(i6) [i0, i4]  # 4
+        jump(i5, i1, i2, i3, i5, i11) # 5
         """
         regalloc = self.prepare_loop(ops)
-        i0, i1, i2, i3, i4 = self.loop.inputargs
+        i0, i1, i2, i3, i4, i10 = self.loop.inputargs
         i5 = self.loop.operations[0]
+        i6 = self.loop.operations[2]
         longevity = regalloc.longevity
-        longevity[i0].last_usage == 2
-        longevity[i0].last_real_usage == 0
-        longevity[i1].last_usage == 3
-        longevity[i1].last_real_usage == 0
-        longevity[i2].last_usage == 3
-        longevity[i2].last_real_usage == 0
-        longevity[i3].last_usage == 3
-        longevity[i3].last_real_usage == 0
-        longevity[i4].last_usage == 3
-        longevity[i4].last_real_usage == -1
-        longevity[i5].last_usage == 3
-        longevity[i5].last_real_usage == 2
+        assert longevity[i0].last_usage == 4
+        assert longevity[i0].real_usages == [0]
+        assert longevity[i1].last_usage == 5
+        assert longevity[i1].real_usages == [0]
+        assert longevity[i2].last_usage == 5
+        assert longevity[i2].real_usages is None
+        assert longevity[i3].last_usage == 5
+        assert longevity[i3].real_usages is None
+        assert longevity[i4].last_usage == 4
+        assert longevity[i4].real_usages is None
+        assert longevity[i5].last_usage == 5
+        assert longevity[i5].real_usages == [2, 3]
+        assert longevity[i6].last_usage == 4
+        assert longevity[i6].real_usages == [4]
+        assert longevity[i10].last_usage == 3
+        assert longevity[i10].real_usages == [3]
 
 class TestRegallocCompOps(BaseTestRegalloc):
 

From pypy.commits at gmail.com  Tue Aug 22 11:05:47 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:47 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: - implement __contains__
 support in rpython
Message-ID: <599c484b.ce8bdf0a.71ef2.790c@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92203:d8e2a043e8eb
Date: 2017-08-22 12:43 +0200
http://bitbucket.org/pypy/pypy/changeset/d8e2a043e8eb/

Log:	- implement __contains__ support in rpython
	- wrap a class around the Lifetime dict

diff --git a/rpython/annotator/binaryop.py b/rpython/annotator/binaryop.py
--- a/rpython/annotator/binaryop.py
+++ b/rpython/annotator/binaryop.py
@@ -729,6 +729,10 @@
     return [get_setitem,
             op.simple_call(get_setitem.result, v_idx, v_value)]
 
+ at op.contains.register_transform(SomeInstance)
+def contains_SomeInstance(annotator, v_ins, v_idx):
+    get_contains = op.getattr(v_ins, const('__contains__'))
+    return [get_contains, op.simple_call(get_contains.result, v_idx)]
 
 class __extend__(pairtype(SomeIterator, SomeIterator)):
 
diff --git a/rpython/annotator/test/test_annrpython.py b/rpython/annotator/test/test_annrpython.py
--- a/rpython/annotator/test/test_annrpython.py
+++ b/rpython/annotator/test/test_annrpython.py
@@ -4077,6 +4077,20 @@
         assert len(a.translator.graphs) == 2 # fn, __setitem__
         assert isinstance(s, annmodel.SomeInteger)
 
+    def test_instance_contains(self):
+        class A(object):
+            def __contains__(self, i):
+                return i & 1 == 0
+
+        def fn(i):
+            a = A()
+            return 0 in a and 1 not in a
+
+        a = self.RPythonAnnotator()
+        s = a.build_types(fn, [int])
+        assert len(a.translator.graphs) == 2 # fn, __contains__
+        assert isinstance(s, annmodel.SomeBool)
+
     def test_instance_getslice(self):
         class A(object):
             def __getslice__(self, stop, start):
diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -856,6 +856,22 @@
     def __repr__(self):
         return "%s:%s(%s)" % (self.definition_pos, self.real_usages, self.last_usage)
 
+class LifetimeManager(object):
+    def __init__(self, longevity):
+        self.longevity = longevity
+
+    def register_hint(self, opindex, var, register):
+        raise NotImplementedError
+
+    def __contains__(self, var):
+        return var in self.longevity
+
+    def __getitem__(self, var):
+        return self.longevity[var]
+
+    def __setitem__(self, var, val):
+        self.longevity[var] = val
+
 def compute_vars_longevity(inputargs, operations):
     # compute a dictionary that maps variables to Lifetime information
     # if a variable is not in the dictionary, it's operation is dead because
@@ -912,7 +928,7 @@
         if not we_are_translated():
             lifetime._check_invariants()
 
-    return longevity
+    return LifetimeManager(longevity)
 
 def is_comparison_or_ovf_op(opnum):
     return rop.is_comparison(opnum) or rop.is_ovf(opnum)

From pypy.commits at gmail.com  Tue Aug 22 11:05:45 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:45 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: a kludgy and lengthy
 explicit way to test the register allocator with a fake
Message-ID: <599c4849.24addf0a.bcb4.b628@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92202:7ffc9b6f6e75
Date: 2017-08-22 12:14 +0200
http://bitbucket.org/pypy/pypy/changeset/7ffc9b6f6e75/

Log:	a kludgy and lengthy explicit way to test the register allocator
	with a fake set or registers

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -1,4 +1,3 @@
-import os
 from rpython.jit.metainterp.history import Const, REF, JitCellToken
 from rpython.rlib.objectmodel import we_are_translated, specialize
 from rpython.jit.metainterp.resoperation import rop, AbstractValue
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1,10 +1,17 @@
 import py
 from rpython.jit.metainterp.history import ConstInt, INT, FLOAT
+from rpython.jit.metainterp.history import BasicFailDescr, TargetToken
+from rpython.jit.metainterp.resoperation import rop
+from rpython.jit.metainterp.resoperation import InputArgInt, InputArgRef,\
+     InputArgFloat
+from rpython.jit.backend.detect_cpu import getcpuclass
 from rpython.jit.backend.llsupport.regalloc import FrameManager, LinkedList
 from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan,\
-     Lifetime as RealLifetime, UNDEF_POS
-from rpython.jit.metainterp.resoperation import InputArgInt, InputArgRef,\
-     InputArgFloat
+     Lifetime as RealLifetime, UNDEF_POS, BaseRegalloc, compute_vars_longevity
+from rpython.jit.tool.oparser import parse
+from rpython.jit.codewriter.effectinfo import EffectInfo
+from rpython.rtyper.lltypesystem import lltype
+from rpython.rtyper.annlowlevel import llhelper
 
 def newboxes(*values):
     return [InputArgInt(v) for v in values]
@@ -49,6 +56,7 @@
 class FakeFramePos(object):
     def __init__(self, pos, box_type):
         self.pos = pos
+        self.value = pos
         self.box_type = box_type
     def __repr__(self):
         return 'FramePos<%d,%s>' % (self.pos, self.box_type)
@@ -78,9 +86,15 @@
         assert isinstance(loc, FakeFramePos)
         return loc.pos
 
+class FakeCPU(object):
+    def get_baseofs_of_frame_field(self):
+        return 0
+
 class MockAsm(object):
     def __init__(self):
         self.moves = []
+        self.emitted = []
+        self.cpu = FakeCPU()
 
         # XXX register allocation statistics to be removed later
         self.num_moves_calls = 0
@@ -97,6 +111,7 @@
 
     def regalloc_mov(self, from_loc, to_loc):
         self.moves.append((from_loc, to_loc))
+        self.emitted.append(("move", to_loc, from_loc))
 
 
 def test_lifetime_next_real_usage():
@@ -649,3 +664,171 @@
         assert fm.get_loc_index(floc) == 0
         for box in fm.bindings.keys():
             fm.mark_as_free(box)
+
+# _____________________________________________________
+# tests that assign registers in a mocked way for a fake CPU
+
+r4, r5, r6, r7, r8, r9 = [FakeReg(i) for i in range(4, 10)]
+
+class RegisterManager2(BaseRegMan):
+    all_regs = [r0, r1, r2, r3, r4, r5, r6, r7]
+
+    save_around_call_regs = [r0, r1, r2, r3]
+
+    frame_reg = r8
+
+    # calling conventions: r0 is result
+    # r1 r2 r3 are arguments and callee-saved registers
+    # r4 r5 r6 r7 are caller-saved registers
+
+    def convert_to_imm(self, v):
+        return v.value
+
+
+class FakeRegalloc(BaseRegalloc):
+    def __init__(self):
+        self.assembler = MockAsm()
+
+    def prepare_loop(self, inputargs, operations, looptoken, allgcrefs):
+        operations = self._prepare(inputargs, operations, allgcrefs)
+        self.operations = operations
+        self._set_initial_bindings(inputargs, looptoken)
+        # note: we need to make a copy of inputargs because possibly_free_vars
+        # is also used on op args, which is a non-resizable list
+        self.possibly_free_vars(list(inputargs))
+        return operations
+
+    def _prepare(self, inputargs, operations, allgcrefs):
+        self.fm = TFrameManager()
+        # compute longevity of variables
+        longevity = compute_vars_longevity(inputargs, operations)
+        self.longevity = longevity
+        self.rm = RegisterManager2(
+            longevity, assembler=self.assembler, frame_manager=self.fm)
+        return operations
+
+    def possibly_free_var(self, var):
+        self.rm.possibly_free_var(var)
+
+    def possibly_free_vars(self, vars):
+        for var in vars:
+            if var is not None: # xxx kludgy
+                self.possibly_free_var(var)
+
+    def loc(self, x):
+        return self.rm.loc(x)
+
+    def force_allocate_reg_or_cc(self, var):
+        assert var.type == INT
+        if self.next_op_can_accept_cc(self.operations, self.rm.position):
+            # hack: return the ebp location to mean "lives in CC".  This
+            # ebp will not actually be used, and the location will be freed
+            # after the next op as usual.
+            self.rm.force_allocate_frame_reg(var)
+            return r8
+        else:
+            # else, return a regular register (not ebp).
+            return self.rm.force_allocate_reg(var, need_lower_byte=True)
+
+    def fake_allocate(self, loop):
+        emit = self.assembler.emitted.append
+        for i, op in enumerate(loop.operations):
+            self.rm.position = i
+            if rop.is_comparison(op.getopnum()):
+                locs = [self.loc(x) for x in op.getarglist()]
+                loc = self.force_allocate_reg_or_cc(op)
+                emit((op.getopname(), loc, locs))
+            elif op.getopname().startswith("int_"):
+                locs = [self.loc(x) for x in op.getarglist()]
+                loc = self.rm.force_result_in_reg(
+                    op, op.getarg(0), op.getarglist())
+                emit((op.getopname(), loc, locs[1:]))
+            elif op.is_guard():
+                emit((op.getopname(), self.loc(op.getarg(0))))
+            else:
+                locs = [self.loc(x) for x in op.getarglist()]
+                if op.type != "v":
+                    loc = self.rm.force_allocate_reg(op)
+                    emit((op.getopname(), loc, locs))
+                else:
+                    emit((op.getopname(), locs))
+        return self.assembler.emitted
+
+CPU = getcpuclass()
+class TestFullRegallocFakeCPU(object):
+    # XXX copy-paste from test_regalloc_integration
+    cpu = CPU(None, None)
+    cpu.setup_once()
+
+    targettoken = TargetToken()
+    targettoken2 = TargetToken()
+    fdescr1 = BasicFailDescr(1)
+    fdescr2 = BasicFailDescr(2)
+    fdescr3 = BasicFailDescr(3)
+
+    def setup_method(self, meth):
+        self.targettoken._ll_loop_code = 0
+        self.targettoken2._ll_loop_code = 0
+
+    def f1(x):
+        return x+1
+
+    def f2(x, y):
+        return x*y
+
+    def f10(*args):
+        assert len(args) == 10
+        return sum(args)
+
+    F1PTR = lltype.Ptr(lltype.FuncType([lltype.Signed], lltype.Signed))
+    F2PTR = lltype.Ptr(lltype.FuncType([lltype.Signed]*2, lltype.Signed))
+    F10PTR = lltype.Ptr(lltype.FuncType([lltype.Signed]*10, lltype.Signed))
+    f1ptr = llhelper(F1PTR, f1)
+    f2ptr = llhelper(F2PTR, f2)
+    f10ptr = llhelper(F10PTR, f10)
+
+    f1_calldescr = cpu.calldescrof(F1PTR.TO, F1PTR.TO.ARGS, F1PTR.TO.RESULT,
+                                   EffectInfo.MOST_GENERAL)
+    f2_calldescr = cpu.calldescrof(F2PTR.TO, F2PTR.TO.ARGS, F2PTR.TO.RESULT,
+                                   EffectInfo.MOST_GENERAL)
+    f10_calldescr = cpu.calldescrof(F10PTR.TO, F10PTR.TO.ARGS, F10PTR.TO.RESULT,
+                                    EffectInfo.MOST_GENERAL)
+
+    namespace = locals().copy()
+
+    def parse(self, s, boxkinds=None, namespace=None):
+        return parse(s, self.cpu, namespace or self.namespace,
+                     boxkinds=boxkinds)
+
+    def allocate(self, s):
+        loop = self.parse(s)
+        self.loop = loop
+        regalloc = FakeRegalloc()
+        regalloc.prepare_loop(loop.inputargs, loop.operations,
+                              loop.original_jitcell_token, [])
+        return regalloc.fake_allocate(loop)
+
+    def _consider_binop(self, op):
+        loc, argloc = self._consider_binop_part(op)
+        self.perform(op, [loc, argloc], loc)
+
+
+    def test_simple(self):
+        ops = '''
+        [i0]
+        label(i0, descr=targettoken)
+        i1 = int_add(i0, 1)
+        i2 = int_lt(i1, 20)
+        guard_true(i2) [i1]
+        jump(i1, descr=targettoken)
+        '''
+        emitted = self.allocate(ops)
+        fp0 = FakeFramePos(0, INT)
+        assert emitted == [
+            ("label", [fp0]),
+            ("move", r0, fp0),
+            ("int_add", r0, [1]),
+            ("int_lt", r8, [r0, 20]),
+            ("guard_true", r8),
+            ("jump", [r0]),
+            ]

From pypy.commits at gmail.com  Tue Aug 22 11:05:51 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:51 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: some fundamental data
 structures for supporting putting boxes into fixed
Message-ID: <599c484f.52d31c0a.72835.b46b@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92205:f6baf7f14279
Date: 2017-08-22 15:01 +0200
http://bitbucket.org/pypy/pypy/changeset/f6baf7f14279/

Log:	some fundamental data structures for supporting putting boxes into
	fixed registers

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -1,3 +1,4 @@
+import sys
 from rpython.jit.metainterp.history import Const, REF, JitCellToken
 from rpython.rlib.objectmodel import we_are_translated, specialize
 from rpython.jit.metainterp.resoperation import rop, AbstractValue
@@ -827,6 +828,10 @@
         # arguments or in failargs)
         self.real_usages = None
 
+        # fixed registers are positions where the variable *needs* to be in a
+        # specific register
+        self.fixed_positions = None
+
     def is_last_real_use_before(self, position):
         if self.real_usages is None:
             return True
@@ -846,6 +851,14 @@
                 low = mid + 1
         return l[low]
 
+    def fixed_register(self, position, reg):
+        assert self.definition_pos <= position <= self.last_usage
+        if self.fixed_positions is None:
+            self.fixed_positions = []
+        else:
+            assert position > self.fixed_positions[-1][0]
+        self.fixed_positions.append((position, reg))
+
     def _check_invariants(self):
         assert self.definition_pos <= self.last_usage
         if self.real_usages is not None:
@@ -856,12 +869,65 @@
     def __repr__(self):
         return "%s:%s(%s)" % (self.definition_pos, self.real_usages, self.last_usage)
 
+
+class FixedRegisterPositions(object):
+    def __init__(self, register):
+        self.register = register
+
+        self.index_lifetimes = []
+
+    def fixed_register(self, opindex, varlifetime):
+        if self.index_lifetimes:
+            assert opindex > self.index_lifetimes[-1][0]
+        self.index_lifetimes.append((opindex, varlifetime))
+
+    def compute_free_until_pos(self, opindex):
+        for (index, varlifetime) in self.index_lifetimes:
+            if opindex <= index:
+                if varlifetime.definition_pos >= opindex:
+                    return varlifetime.definition_pos
+                else:
+                    # the variable didn't make it into the register despite
+                    # being defined already. so we don't care too much, and can
+                    # say that the variable is free until index
+                    return index
+        return sys.maxint
+
 class LifetimeManager(object):
     def __init__(self, longevity):
         self.longevity = longevity
 
-    def register_hint(self, opindex, var, register):
-        raise NotImplementedError
+        # dictionary maps register to FixedRegisterPositions
+        self.fixed_register_use = {}
+
+    def fixed_register(self, opindex, register, var=None):
+        """ Tell the LifetimeManager that variable var *must* be in register at
+        operation opindex. var can be None, if no variable at all can be in
+        that register at the point."""
+        varlifetime = self.longevity[var]
+        if register not in self.fixed_register_use:
+            self.fixed_register_use[register] = FixedRegisterPositions(register)
+        self.fixed_register_use[register].fixed_register(opindex, varlifetime)
+        varlifetime.fixed_register(opindex, register)
+
+    def compute_longest_free_reg(self, position, free_regs):
+        """ for every register in free_regs, compute how far into the
+        future that register can remain free, according to the constraints of
+        the fixed registers. Find the register that is free the longest. Return a tuple
+        (reg, free_until_pos). """
+        free_until_pos = {}
+        max_free_pos = -1
+        best_reg = None
+        for reg in free_regs:
+            fixed_reg_pos = self.fixed_register_use.get(reg, None)
+            if fixed_reg_pos is None:
+                return reg, sys.maxint
+            else:
+                free_until_pos = fixed_reg_pos.compute_free_until_pos(position)
+                if free_until_pos > max_free_pos:
+                    best_reg = reg
+                    max_free_pos = free_until_pos
+        return best_reg, max_free_pos
 
     def __contains__(self, var):
         return var in self.longevity
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1,4 +1,5 @@
 import py
+import sys
 from rpython.jit.metainterp.history import ConstInt, INT, FLOAT
 from rpython.jit.metainterp.history import BasicFailDescr, TargetToken
 from rpython.jit.metainterp.resoperation import rop
@@ -7,7 +8,8 @@
 from rpython.jit.backend.detect_cpu import getcpuclass
 from rpython.jit.backend.llsupport.regalloc import FrameManager, LinkedList
 from rpython.jit.backend.llsupport.regalloc import RegisterManager as BaseRegMan,\
-     Lifetime as RealLifetime, UNDEF_POS, BaseRegalloc, compute_vars_longevity
+     Lifetime as RealLifetime, UNDEF_POS, BaseRegalloc, compute_vars_longevity,\
+     LifetimeManager
 from rpython.jit.tool.oparser import parse
 from rpython.jit.codewriter.effectinfo import EffectInfo
 from rpython.rtyper.lltypesystem import lltype
@@ -123,6 +125,73 @@
         assert next > i
         assert lt.real_usages[lt.real_usages.index(next) - 1] <= i
 
+def test_fixed_position():
+    b0, b1, b2 = newboxes(0, 0, 0)
+    l0 = Lifetime(0, 5)
+    l1 = Lifetime(2, 9)
+    l2 = Lifetime(0, 9)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(1, r0, b0)
+    longevity.fixed_register(4, r2, b0)
+    longevity.fixed_register(5, r1, b1)
+    longevity.fixed_register(8, r1, b1)
+
+    assert l0.fixed_positions == [(1, r0), (4, r2)]
+    assert l1.fixed_positions == [(5, r1), (8, r1)]
+    assert l2.fixed_positions is None
+
+    fpr0 = longevity.fixed_register_use[r0]
+    fpr1 = longevity.fixed_register_use[r1]
+    fpr2 = longevity.fixed_register_use[r2]
+    assert r3 not in longevity.fixed_register_use
+    assert fpr0.index_lifetimes == [(1, l0)]
+    assert fpr1.index_lifetimes == [(5, l1), (8, l1)]
+    assert fpr2.index_lifetimes == [(4, l0)]
+
+
+def test_compute_free_until_pos():
+    b0, b1, b2 = newboxes(0, 0, 0)
+    l0 = Lifetime(0, 5)
+    l1 = Lifetime(2, 9)
+    l2 = Lifetime(30, 40)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(1, r0, b0)
+    longevity.fixed_register(4, r2, b0)
+    longevity.fixed_register(5, r1, b1)
+    longevity.fixed_register(8, r1, b1)
+    longevity.fixed_register(35, r1, b2)
+
+    fpr1 = longevity.fixed_register_use[r1]
+
+    # simple cases: we are before the beginning of the lifetime of the variable
+    # in the fixed register, then it's free until the definition of the
+    # variable
+    assert fpr1.compute_free_until_pos(0) == 2
+    assert fpr1.compute_free_until_pos(1) == 2
+    assert fpr1.compute_free_until_pos(2) == 2
+    assert fpr1.compute_free_until_pos(10) == 30
+    assert fpr1.compute_free_until_pos(20) == 30
+    assert fpr1.compute_free_until_pos(30) == 30
+
+    # after the fixed use, we are fined anyway
+    assert fpr1.compute_free_until_pos(36) == sys.maxint
+    assert fpr1.compute_free_until_pos(50) == sys.maxint
+
+    # asking for a position *after* the definition of the variable in the fixed
+    # register means the variable didn't make it into the fixed register, but
+    # at the latest by the use point it will have to go there
+    assert fpr1.compute_free_until_pos(3) == 5
+    assert fpr1.compute_free_until_pos(4) == 5
+    assert fpr1.compute_free_until_pos(5) == 5
+    assert fpr1.compute_free_until_pos(6) == 8
+    assert fpr1.compute_free_until_pos(7) == 8
+    assert fpr1.compute_free_until_pos(8) == 8
+    assert fpr1.compute_free_until_pos(31) == 35
+    assert fpr1.compute_free_until_pos(32) == 35
+    assert fpr1.compute_free_until_pos(33) == 35
+    assert fpr1.compute_free_until_pos(34) == 35
+    assert fpr1.compute_free_until_pos(35) == 35
+
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)
@@ -223,7 +292,7 @@
         assert isinstance(loc, FakeReg)
         assert loc not in [r2, r3]
         rm._check_invariants()
-    
+
     def test_make_sure_var_in_reg(self):
         boxes, longevity = boxes_and_longevity(5)
         fm = TFrameManager()
@@ -237,7 +306,7 @@
         loc = rm.make_sure_var_in_reg(b0)
         assert isinstance(loc, FakeReg)
         rm._check_invariants()
-        
+
     def test_force_result_in_reg_1(self):
         b0, b1 = newboxes(0, 0)
         longevity = {b0: Lifetime(0, 1), b1: Lifetime(1, 3)}

From pypy.commits at gmail.com  Tue Aug 22 11:05:53 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:53 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: support not specifying a
 variable (eg for caller-saved regs)
Message-ID: <599c4851.4c0e1c0a.f723c.7e2a@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92206:85eb224c8d6c
Date: 2017-08-22 15:06 +0200
http://bitbucket.org/pypy/pypy/changeset/85eb224c8d6c/

Log:	support not specifying a variable (eg for caller-saved regs)

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -884,12 +884,13 @@
     def compute_free_until_pos(self, opindex):
         for (index, varlifetime) in self.index_lifetimes:
             if opindex <= index:
-                if varlifetime.definition_pos >= opindex:
+                if varlifetime is not None and varlifetime.definition_pos >= opindex:
                     return varlifetime.definition_pos
                 else:
-                    # the variable didn't make it into the register despite
-                    # being defined already. so we don't care too much, and can
-                    # say that the variable is free until index
+                    # the variable doesn't exist or didn't make it into the
+                    # register despite being defined already. so we don't care
+                    # too much, and can say that the variable is free until
+                    # index
                     return index
         return sys.maxint
 
@@ -904,11 +905,14 @@
         """ Tell the LifetimeManager that variable var *must* be in register at
         operation opindex. var can be None, if no variable at all can be in
         that register at the point."""
-        varlifetime = self.longevity[var]
+        if var is None:
+            varlifetime = None
+        else:
+            varlifetime = self.longevity[var]
+            varlifetime.fixed_register(opindex, register)
         if register not in self.fixed_register_use:
             self.fixed_register_use[register] = FixedRegisterPositions(register)
         self.fixed_register_use[register].fixed_register(opindex, varlifetime)
-        varlifetime.fixed_register(opindex, register)
 
     def compute_longest_free_reg(self, position, free_regs):
         """ for every register in free_regs, compute how far into the
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -148,6 +148,46 @@
     assert fpr1.index_lifetimes == [(5, l1), (8, l1)]
     assert fpr2.index_lifetimes == [(4, l0)]
 
+def test_fixed_position_none():
+    b0, b1, b2 = newboxes(0, 0, 0)
+    l0 = Lifetime(0, 5)
+    l1 = Lifetime(2, 9)
+    l2 = Lifetime(0, 9)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(1, r0)
+    longevity.fixed_register(4, r2)
+    longevity.fixed_register(5, r1)
+    longevity.fixed_register(8, r1)
+
+    fpr0 = longevity.fixed_register_use[r0]
+    fpr1 = longevity.fixed_register_use[r1]
+    fpr2 = longevity.fixed_register_use[r2]
+    assert r3 not in longevity.fixed_register_use
+    assert fpr0.index_lifetimes == [(1, None)]
+    assert fpr1.index_lifetimes == [(5, None), (8, None)]
+    assert fpr2.index_lifetimes == [(4, None)]
+
+
+def test_compute_free_until_pos_none():
+    longevity = LifetimeManager({})
+    longevity.fixed_register(1, r0, None)
+    longevity.fixed_register(4, r2, None)
+    longevity.fixed_register(5, r1, None)
+    longevity.fixed_register(8, r1, None)
+    longevity.fixed_register(35, r1, None)
+
+    fpr1 = longevity.fixed_register_use[r1]
+
+    assert fpr1.compute_free_until_pos(0) == 5
+    assert fpr1.compute_free_until_pos(1) == 5
+    assert fpr1.compute_free_until_pos(2) == 5
+    assert fpr1.compute_free_until_pos(3) == 5
+    assert fpr1.compute_free_until_pos(4) == 5
+    assert fpr1.compute_free_until_pos(5) == 5
+    assert fpr1.compute_free_until_pos(10) == 35
+    assert fpr1.compute_free_until_pos(20) == 35
+    assert fpr1.compute_free_until_pos(30) == 35
+    assert fpr1.compute_free_until_pos(36) == sys.maxint
 
 def test_compute_free_until_pos():
     b0, b1, b2 = newboxes(0, 0, 0)

From pypy.commits at gmail.com  Tue Aug 22 11:05:49 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:49 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: refactoring
Message-ID: <599c484d.8b841c0a.91b1c.4f31@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92204:b4a1f3830c25
Date: 2017-08-22 13:55 +0200
http://bitbucket.org/pypy/pypy/changeset/b4a1f3830c25/

Log:	refactoring

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -734,24 +734,31 @@
         emit = self.assembler.emitted.append
         for i, op in enumerate(loop.operations):
             self.rm.position = i
-            if rop.is_comparison(op.getopnum()):
+            opnum = op.getopnum()
+            opname = op.getopname()
+            if rop.is_comparison(opnum):
                 locs = [self.loc(x) for x in op.getarglist()]
                 loc = self.force_allocate_reg_or_cc(op)
-                emit((op.getopname(), loc, locs))
-            elif op.getopname().startswith("int_"):
+                emit((opname, loc, locs))
+            elif opname.startswith("int_"):
                 locs = [self.loc(x) for x in op.getarglist()]
                 loc = self.rm.force_result_in_reg(
                     op, op.getarg(0), op.getarglist())
-                emit((op.getopname(), loc, locs[1:]))
+                emit((opname, loc, locs[1:]))
             elif op.is_guard():
-                emit((op.getopname(), self.loc(op.getarg(0))))
+                emit((opname, self.loc(op.getarg(0))))
+            elif opname == "label":
+                descr = op.getdescr()
+                locs = [self.loc(x) for x in op.getarglist()]
+                emit((opname, locs))
+                descr._fake_arglocs = locs
             else:
                 locs = [self.loc(x) for x in op.getarglist()]
                 if op.type != "v":
                     loc = self.rm.force_allocate_reg(op)
-                    emit((op.getopname(), loc, locs))
+                    emit((opname, loc, locs))
                 else:
-                    emit((op.getopname(), locs))
+                    emit((opname, locs))
         return self.assembler.emitted
 
 CPU = getcpuclass()

From pypy.commits at gmail.com  Tue Aug 22 11:05:55 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:55 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: simplify
Message-ID: <599c4853.43491c0a.aea41.a068@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92207:50dd22a26d67
Date: 2017-08-22 15:10 +0200
http://bitbucket.org/pypy/pypy/changeset/50dd22a26d67/

Log:	simplify

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -881,7 +881,7 @@
             assert opindex > self.index_lifetimes[-1][0]
         self.index_lifetimes.append((opindex, varlifetime))
 
-    def compute_free_until_pos(self, opindex):
+    def free_until_pos(self, opindex):
         for (index, varlifetime) in self.index_lifetimes:
             if opindex <= index:
                 if varlifetime is not None and varlifetime.definition_pos >= opindex:
@@ -914,7 +914,7 @@
             self.fixed_register_use[register] = FixedRegisterPositions(register)
         self.fixed_register_use[register].fixed_register(opindex, varlifetime)
 
-    def compute_longest_free_reg(self, position, free_regs):
+    def longest_free_reg(self, position, free_regs):
         """ for every register in free_regs, compute how far into the
         future that register can remain free, according to the constraints of
         the fixed registers. Find the register that is free the longest. Return a tuple
@@ -927,7 +927,7 @@
             if fixed_reg_pos is None:
                 return reg, sys.maxint
             else:
-                free_until_pos = fixed_reg_pos.compute_free_until_pos(position)
+                free_until_pos = fixed_reg_pos.free_until_pos(position)
                 if free_until_pos > max_free_pos:
                     best_reg = reg
                     max_free_pos = free_until_pos
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -168,35 +168,31 @@
     assert fpr2.index_lifetimes == [(4, None)]
 
 
-def test_compute_free_until_pos_none():
+def test_free_until_pos_none():
     longevity = LifetimeManager({})
-    longevity.fixed_register(1, r0, None)
-    longevity.fixed_register(4, r2, None)
     longevity.fixed_register(5, r1, None)
     longevity.fixed_register(8, r1, None)
     longevity.fixed_register(35, r1, None)
 
     fpr1 = longevity.fixed_register_use[r1]
 
-    assert fpr1.compute_free_until_pos(0) == 5
-    assert fpr1.compute_free_until_pos(1) == 5
-    assert fpr1.compute_free_until_pos(2) == 5
-    assert fpr1.compute_free_until_pos(3) == 5
-    assert fpr1.compute_free_until_pos(4) == 5
-    assert fpr1.compute_free_until_pos(5) == 5
-    assert fpr1.compute_free_until_pos(10) == 35
-    assert fpr1.compute_free_until_pos(20) == 35
-    assert fpr1.compute_free_until_pos(30) == 35
-    assert fpr1.compute_free_until_pos(36) == sys.maxint
+    assert fpr1.free_until_pos(0) == 5
+    assert fpr1.free_until_pos(1) == 5
+    assert fpr1.free_until_pos(2) == 5
+    assert fpr1.free_until_pos(3) == 5
+    assert fpr1.free_until_pos(4) == 5
+    assert fpr1.free_until_pos(5) == 5
+    assert fpr1.free_until_pos(10) == 35
+    assert fpr1.free_until_pos(20) == 35
+    assert fpr1.free_until_pos(30) == 35
+    assert fpr1.free_until_pos(36) == sys.maxint
 
-def test_compute_free_until_pos():
+def test_free_until_pos():
     b0, b1, b2 = newboxes(0, 0, 0)
     l0 = Lifetime(0, 5)
     l1 = Lifetime(2, 9)
     l2 = Lifetime(30, 40)
     longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
-    longevity.fixed_register(1, r0, b0)
-    longevity.fixed_register(4, r2, b0)
     longevity.fixed_register(5, r1, b1)
     longevity.fixed_register(8, r1, b1)
     longevity.fixed_register(35, r1, b2)
@@ -206,31 +202,32 @@
     # simple cases: we are before the beginning of the lifetime of the variable
     # in the fixed register, then it's free until the definition of the
     # variable
-    assert fpr1.compute_free_until_pos(0) == 2
-    assert fpr1.compute_free_until_pos(1) == 2
-    assert fpr1.compute_free_until_pos(2) == 2
-    assert fpr1.compute_free_until_pos(10) == 30
-    assert fpr1.compute_free_until_pos(20) == 30
-    assert fpr1.compute_free_until_pos(30) == 30
+    assert fpr1.free_until_pos(0) == 2
+    assert fpr1.free_until_pos(1) == 2
+    assert fpr1.free_until_pos(2) == 2
+    assert fpr1.free_until_pos(10) == 30
+    assert fpr1.free_until_pos(20) == 30
+    assert fpr1.free_until_pos(30) == 30
 
     # after the fixed use, we are fined anyway
-    assert fpr1.compute_free_until_pos(36) == sys.maxint
-    assert fpr1.compute_free_until_pos(50) == sys.maxint
+    assert fpr1.free_until_pos(36) == sys.maxint
+    assert fpr1.free_until_pos(50) == sys.maxint
 
     # asking for a position *after* the definition of the variable in the fixed
     # register means the variable didn't make it into the fixed register, but
     # at the latest by the use point it will have to go there
-    assert fpr1.compute_free_until_pos(3) == 5
-    assert fpr1.compute_free_until_pos(4) == 5
-    assert fpr1.compute_free_until_pos(5) == 5
-    assert fpr1.compute_free_until_pos(6) == 8
-    assert fpr1.compute_free_until_pos(7) == 8
-    assert fpr1.compute_free_until_pos(8) == 8
-    assert fpr1.compute_free_until_pos(31) == 35
-    assert fpr1.compute_free_until_pos(32) == 35
-    assert fpr1.compute_free_until_pos(33) == 35
-    assert fpr1.compute_free_until_pos(34) == 35
-    assert fpr1.compute_free_until_pos(35) == 35
+    assert fpr1.free_until_pos(3) == 5
+    assert fpr1.free_until_pos(4) == 5
+    assert fpr1.free_until_pos(5) == 5
+    assert fpr1.free_until_pos(6) == 8
+    assert fpr1.free_until_pos(7) == 8
+    assert fpr1.free_until_pos(8) == 8
+    assert fpr1.free_until_pos(31) == 35
+    assert fpr1.free_until_pos(32) == 35
+    assert fpr1.free_until_pos(33) == 35
+    assert fpr1.free_until_pos(34) == 35
+    assert fpr1.free_until_pos(35) == 35
+
 
 class TestRegalloc(object):
     def test_freeing_vars(self):

From pypy.commits at gmail.com  Tue Aug 22 11:05:57 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:05:57 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: a special case for repeated
 uses of the same fixed register (with different vars). a test for
 longest_free_reg
Message-ID: <599c4855.4981df0a.601c0.a29a@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92208:ffb5755ee817
Date: 2017-08-22 15:41 +0200
http://bitbucket.org/pypy/pypy/changeset/ffb5755ee817/

Log:	a special case for repeated uses of the same fixed register (with
	different vars). a test for longest_free_reg

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -852,12 +852,18 @@
         return l[low]
 
     def fixed_register(self, position, reg):
+        """ registers a fixed register use for the variable at position in
+        register reg. returns the position from where on the register should be
+        held free. """
         assert self.definition_pos <= position <= self.last_usage
         if self.fixed_positions is None:
             self.fixed_positions = []
+            res = self.definition_pos
         else:
             assert position > self.fixed_positions[-1][0]
+            res = self.fixed_positions[-1][0]
         self.fixed_positions.append((position, reg))
+        return res
 
     def _check_invariants(self):
         assert self.definition_pos <= self.last_usage
@@ -876,16 +882,17 @@
 
         self.index_lifetimes = []
 
-    def fixed_register(self, opindex, varlifetime):
+    def fixed_register(self, opindex, definition_pos):
         if self.index_lifetimes:
             assert opindex > self.index_lifetimes[-1][0]
-        self.index_lifetimes.append((opindex, varlifetime))
+        self.index_lifetimes.append((opindex, definition_pos))
 
     def free_until_pos(self, opindex):
-        for (index, varlifetime) in self.index_lifetimes:
+        # XXX could use binary search
+        for (index, definition_pos) in self.index_lifetimes:
             if opindex <= index:
-                if varlifetime is not None and varlifetime.definition_pos >= opindex:
-                    return varlifetime.definition_pos
+                if definition_pos >= opindex:
+                    return definition_pos
                 else:
                     # the variable doesn't exist or didn't make it into the
                     # register despite being defined already. so we don't care
@@ -894,6 +901,7 @@
                     return index
         return sys.maxint
 
+
 class LifetimeManager(object):
     def __init__(self, longevity):
         self.longevity = longevity
@@ -906,19 +914,19 @@
         operation opindex. var can be None, if no variable at all can be in
         that register at the point."""
         if var is None:
-            varlifetime = None
+            definition_pos = opindex
         else:
             varlifetime = self.longevity[var]
-            varlifetime.fixed_register(opindex, register)
+            definition_pos = varlifetime.fixed_register(opindex, register)
         if register not in self.fixed_register_use:
             self.fixed_register_use[register] = FixedRegisterPositions(register)
-        self.fixed_register_use[register].fixed_register(opindex, varlifetime)
+        self.fixed_register_use[register].fixed_register(opindex, definition_pos)
 
     def longest_free_reg(self, position, free_regs):
-        """ for every register in free_regs, compute how far into the
-        future that register can remain free, according to the constraints of
-        the fixed registers. Find the register that is free the longest. Return a tuple
-        (reg, free_until_pos). """
+        """ for every register in free_regs, compute how far into the future
+        that register can remain free, according to the constraints of the
+        fixed registers. Find the register that is free the longest. Return a
+        tuple (reg, free_until_pos). """
         free_until_pos = {}
         max_free_pos = -1
         best_reg = None
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -144,9 +144,9 @@
     fpr1 = longevity.fixed_register_use[r1]
     fpr2 = longevity.fixed_register_use[r2]
     assert r3 not in longevity.fixed_register_use
-    assert fpr0.index_lifetimes == [(1, l0)]
-    assert fpr1.index_lifetimes == [(5, l1), (8, l1)]
-    assert fpr2.index_lifetimes == [(4, l0)]
+    assert fpr0.index_lifetimes == [(1, 0)]
+    assert fpr1.index_lifetimes == [(5, 2), (8, 5)]
+    assert fpr2.index_lifetimes == [(4, 1)]
 
 def test_fixed_position_none():
     b0, b1, b2 = newboxes(0, 0, 0)
@@ -163,9 +163,9 @@
     fpr1 = longevity.fixed_register_use[r1]
     fpr2 = longevity.fixed_register_use[r2]
     assert r3 not in longevity.fixed_register_use
-    assert fpr0.index_lifetimes == [(1, None)]
-    assert fpr1.index_lifetimes == [(5, None), (8, None)]
-    assert fpr2.index_lifetimes == [(4, None)]
+    assert fpr0.index_lifetimes == [(1, 1)]
+    assert fpr1.index_lifetimes == [(5, 5), (8, 8)]
+    assert fpr2.index_lifetimes == [(4, 4)]
 
 
 def test_free_until_pos_none():
@@ -228,6 +228,33 @@
     assert fpr1.free_until_pos(34) == 35
     assert fpr1.free_until_pos(35) == 35
 
+def test_free_until_pos_different_regs():
+    b0, b1, b2 = newboxes(0, 0, 0)
+    l0 = Lifetime(0, 5)
+    l1 = Lifetime(2, 9)
+    l2 = Lifetime(30, 40)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(1, r0, b0)
+    longevity.fixed_register(4, r2, b0)
+    fpr2 = longevity.fixed_register_use[r2]
+    # the definition of b0 is before the other fixed register use of r0, so the
+    # earliest b0 can be in r2 is that use point at index 1
+    assert fpr2.free_until_pos(0) == 1
+
+
+def test_longest_free_reg():
+    b0, b1, b2 = newboxes(0, 0, 0)
+    l0 = Lifetime(0, 5)
+    l1 = Lifetime(2, 9)
+    l2 = Lifetime(30, 40)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(1, r0, b0)
+    longevity.fixed_register(4, r2, b0)
+    longevity.fixed_register(5, r1, b1)
+    longevity.fixed_register(8, r1, b1)
+    longevity.fixed_register(35, r1, b2)
+
+    assert longevity.longest_free_reg(0, [r0, r1, r2]) == (r2, 2)
 
 class TestRegalloc(object):
     def test_freeing_vars(self):

From pypy.commits at gmail.com  Tue Aug 22 11:06:06 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 08:06:06 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: enough mocking for a simple
 call test (but with wrong results!)
Message-ID: <599c485e.03081c0a.5d75b.0b95@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92209:84209398e70f
Date: 2017-08-22 17:04 +0200
http://bitbucket.org/pypy/pypy/changeset/84209398e70f/

Log:	enough mocking for a simple call test (but with wrong results!)

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -579,9 +579,9 @@
         if v not in self.reg_bindings:
             # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(result_v, forbidden_vars)
+            loc = self.force_allocate_reg(v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-            return loc
+        assert v in self.reg_bindings
         if self.longevity[v].last_usage > self.position:
             # we need to find a new place for variable v and
             # store result in the same place
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -44,6 +44,10 @@
 class FakeReg(object):
     def __init__(self, i):
         self.n = i
+    def _getregkey(self):
+        return self.n
+    def is_memory_reference(self):
+        return False
     def __repr__(self):
         return 'r%d' % self.n
 
@@ -60,6 +64,10 @@
         self.pos = pos
         self.value = pos
         self.box_type = box_type
+    def _getregkey(self):
+        return ~self.value
+    def is_memory_reference(self):
+        return True
     def __repr__(self):
         return 'FramePos<%d,%s>' % (self.pos, self.box_type)
     def __eq__(self, other):
@@ -254,7 +262,7 @@
     longevity.fixed_register(8, r1, b1)
     longevity.fixed_register(35, r1, b2)
 
-    assert longevity.longest_free_reg(0, [r0, r1, r2]) == (r2, 2)
+    assert longevity.longest_free_reg(0, [r0, r1, r2]) == (r1, 2)
 
 class TestRegalloc(object):
     def test_freeing_vars(self):
@@ -811,12 +819,15 @@
     frame_reg = r8
 
     # calling conventions: r0 is result
-    # r1 r2 r3 are arguments and callee-saved registers
-    # r4 r5 r6 r7 are caller-saved registers
+    # r1 r2 r3 are arguments and caller-saved registers
+    # r4 r5 r6 r7 are callee-saved registers
 
     def convert_to_imm(self, v):
         return v.value
 
+    def call_result_location(self, v):
+        return r0
+
 
 class FakeRegalloc(BaseRegalloc):
     def __init__(self):
@@ -864,6 +875,8 @@
             return self.rm.force_allocate_reg(var, need_lower_byte=True)
 
     def fake_allocate(self, loop):
+        from rpython.jit.backend.x86.jump import remap_frame_layout
+
         emit = self.assembler.emitted.append
         for i, op in enumerate(loop.operations):
             self.rm.position = i
@@ -879,7 +892,16 @@
                     op, op.getarg(0), op.getarglist())
                 emit((opname, loc, locs[1:]))
             elif op.is_guard():
-                emit((opname, self.loc(op.getarg(0))))
+                fail_locs = [self.loc(x) for x in op.getfailargs()]
+                emit((opname, self.loc(op.getarg(0)), fail_locs))
+            elif rop.is_call(opnum):
+                # calling convention!
+                src_locs = [self.loc(x) for x in op.getarglist()[1:]]
+                self.rm.before_call()
+                loc = self.rm.after_call(op)
+                dst_locs = [r1, r2, r3][:len(src_locs)]
+                remap_frame_layout(self.assembler, src_locs, dst_locs, r8)
+                emit((opname, loc, dst_locs))
             elif opname == "label":
                 descr = op.getdescr()
                 locs = [self.loc(x) for x in op.getarglist()]
@@ -946,6 +968,7 @@
         regalloc = FakeRegalloc()
         regalloc.prepare_loop(loop.inputargs, loop.operations,
                               loop.original_jitcell_token, [])
+        self.regalloc = regalloc
         return regalloc.fake_allocate(loop)
 
     def _consider_binop(self, op):
@@ -969,6 +992,23 @@
             ("move", r0, fp0),
             ("int_add", r0, [1]),
             ("int_lt", r8, [r0, 20]),
-            ("guard_true", r8),
+            ("guard_true", r8, [r0]),
             ("jump", [r0]),
-            ]
+        ]
+
+    def test_call(self):
+        ops = '''
+        [i0]
+        i1 = int_mul(i0, 2)
+        i2 = call_i(ConstClass(f1ptr), i1, descr=f1_calldescr)
+        guard_false(i2) []
+        '''
+        emitted = self.allocate(ops)
+        fp0 = FakeFramePos(0, INT)
+        assert emitted == [
+            ("move", r0, fp0),
+            ("int_mul", r0, [2]),
+            ("move", r1, r0),
+            ("call_i", r0, [r1]),
+            ("guard_false", r0, []),
+        ]
diff --git a/rpython/jit/backend/tool/viewcode.py b/rpython/jit/backend/tool/viewcode.py
--- a/rpython/jit/backend/tool/viewcode.py
+++ b/rpython/jit/backend/tool/viewcode.py
@@ -223,7 +223,7 @@
             addr = addrs[-1]
             final = '\tjmp' in line
             yield i, addr, final
-        if self.fallthrough and '\tret' not in line:
+        if self.fallthrough and '\tret' not in line and "\tjmp    r11" not in line:
             yield len(lines), self.addr + len(self.data), True
 
 

From pypy.commits at gmail.com  Tue Aug 22 13:28:42 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 22 Aug 2017 10:28:42 -0700 (PDT)
Subject: [pypy-commit] pypy default: cextension types should have the short
 names and long module for pickling
Message-ID: <599c69ca.3198df0a.49111.bcb1@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92210:0087987731d8
Date: 2017-08-22 20:27 +0300
http://bitbucket.org/pypy/pypy/changeset/0087987731d8/

Log:	cextension types should have the short names and long module for
	pickling

diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -536,23 +536,27 @@
         space = self.space
         if self.is_heaptype():
             return self.getdictvalue(space, '__module__')
+        elif self.is_cpytype():
+            dot = self.name.rfind('.')
         else:
             dot = self.name.find('.')
-            if dot >= 0:
-                mod = self.name[:dot]
-            else:
-                mod = "__builtin__"
-            return space.newtext(mod)
+        if dot >= 0:
+            mod = self.name[:dot]
+        else:
+            mod = "__builtin__"
+        return space.newtext(mod)
 
     def getname(self, space):
         if self.is_heaptype():
             return self.name
+        elif self.is_cpytype():
+            dot = self.name.rfind('.')
         else:
             dot = self.name.find('.')
-            if dot >= 0:
-                return self.name[dot+1:]
-            else:
-                return self.name
+        if dot >= 0:
+            return self.name[dot+1:]
+        else:
+            return self.name
 
     def add_subclass(self, w_subclass):
         space = self.space

From pypy.commits at gmail.com  Tue Aug 22 15:21:33 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 12:21:33 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: chose the fixed register if
 it is available
Message-ID: <599c843d.08e61c0a.e6ed1.0459@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92211:bb3d99077725
Date: 2017-08-22 21:07 +0200
http://bitbucket.org/pypy/pypy/changeset/bb3d99077725/

Log:	chose the fixed register if it is available

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -381,24 +381,27 @@
             loc = self.reg_bindings.get(v, None)
             if loc is not None and loc not in self.no_lower_byte_regs:
                 return loc
-            for i in range(len(self.free_regs) - 1, -1, -1):
-                reg = self.free_regs[i]
-                if reg not in self.no_lower_byte_regs:
-                    if loc is not None:
-                        self.free_regs[i] = loc
-                    else:
-                        del self.free_regs[i]
-                    self.reg_bindings[v] = reg
-                    return reg
-            return None
+            free_regs = [reg for reg in self.free_regs
+                         if reg not in self.no_lower_byte_regs]
+            newloc = self.longevity.try_pick_free_reg(
+                self.position, v, free_regs)
+            if newloc is None:
+                return None
+            self.free_regs.remove(newloc)
+            if loc is not None:
+                self.free_regs.append(loc)
+            self.reg_bindings[v] = newloc
+            return newloc
         try:
             return self.reg_bindings[v]
         except KeyError:
-            # YYY here we should chose the free variable a bit more carefully
-            if self.free_regs:
-                loc = self.free_regs.pop()
-                self.reg_bindings[v] = loc
-                return loc
+            loc = self.longevity.try_pick_free_reg(
+                self.position, v, self.free_regs)
+            if loc is None:
+                return None
+            self.reg_bindings[v] = loc
+            self.free_regs.remove(loc)
+            return loc
 
     def _spill_var(self, v, forbidden_vars, selected_reg,
                    need_lower_byte=False):
@@ -421,6 +424,8 @@
         # if that doesn't exist, spill the variable that has a real_usage that
         # is the furthest away from the current position
 
+        # YYY check for fixed variable usages
+
         cur_max_use_distance = -1
         position = self.position
         candidate = None
@@ -579,9 +584,9 @@
         if v not in self.reg_bindings:
             # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(v, forbidden_vars)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-        assert v in self.reg_bindings
+            return loc
         if self.longevity[v].last_usage > self.position:
             # we need to find a new place for variable v and
             # store result in the same place
@@ -865,6 +870,14 @@
         self.fixed_positions.append((position, reg))
         return res
 
+    def find_fixed_register(self, opindex):
+        # XXX could use binary search
+        if self.fixed_positions is None:
+            return None
+        for (index, reg) in self.fixed_positions:
+            if opindex <= index:
+                return reg
+
     def _check_invariants(self):
         assert self.definition_pos <= self.last_usage
         if self.real_usages is not None:
@@ -928,9 +941,11 @@
         fixed registers. Find the register that is free the longest. Return a
         tuple (reg, free_until_pos). """
         free_until_pos = {}
-        max_free_pos = -1
+        max_free_pos = position
         best_reg = None
-        for reg in free_regs:
+        # reverse for compatibility with old code
+        for i in range(len(free_regs) - 1, -1, -1):
+            reg = free_regs[i]
             fixed_reg_pos = self.fixed_register_use.get(reg, None)
             if fixed_reg_pos is None:
                 return reg, sys.maxint
@@ -941,6 +956,26 @@
                     max_free_pos = free_until_pos
         return best_reg, max_free_pos
 
+    def try_pick_free_reg(self, position, v, free_regs):
+        if not free_regs:
+            return None
+        longevityvar = self[v]
+        reg = longevityvar.find_fixed_register(position)
+        if reg is not None and reg in free_regs:
+            return reg
+        return free_regs[-1]
+        # more advanced stuff below, needs tests
+
+
+
+        loc, free_until = self.longevity.longest_free_reg(
+                self.position, free_regs)
+        if loc is None:
+            return None
+        # YYY could check whether it's best to spill v here, but hard
+        # to do in the current system
+        return loc
+
     def __contains__(self, var):
         return var in self.longevity
 
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -56,6 +56,12 @@
 
 class RegisterManager(BaseRegMan):
     all_regs = regs
+
+    def __init__(self, longevity, frame_manager=None, assembler=None):
+        if isinstance(longevity, dict):
+            longevity = LifetimeManager(longevity)
+        BaseRegMan.__init__(self, longevity, frame_manager, assembler)
+
     def convert_to_imm(self, v):
         return v
 
@@ -840,6 +846,7 @@
         # note: we need to make a copy of inputargs because possibly_free_vars
         # is also used on op args, which is a non-resizable list
         self.possibly_free_vars(list(inputargs))
+        self._add_fixed_registers()
         return operations
 
     def _prepare(self, inputargs, operations, allgcrefs):
@@ -916,6 +923,16 @@
                     emit((opname, locs))
         return self.assembler.emitted
 
+    def _add_fixed_registers(self):
+        for i, op in enumerate(self.operations):
+            if rop.is_call(op.getopnum()):
+                # calling convention!
+                arglist = op.getarglist()[1:]
+                for arg, reg in zip(arglist + [None] * (3 - len(arglist)), [r1, r2, r3]):
+                    self.longevity.fixed_register(i, reg, arg)
+                self.longevity.fixed_register(i, r0, op)
+
+
 CPU = getcpuclass()
 class TestFullRegallocFakeCPU(object):
     # XXX copy-paste from test_regalloc_integration
@@ -1006,9 +1023,8 @@
         emitted = self.allocate(ops)
         fp0 = FakeFramePos(0, INT)
         assert emitted == [
-            ("move", r0, fp0),
-            ("int_mul", r0, [2]),
-            ("move", r1, r0),
+            ("move", r1, fp0),
+            ("int_mul", r1, [2]),
             ("call_i", r0, [r1]),
             ("guard_false", r0, []),
         ]

From pypy.commits at gmail.com  Tue Aug 22 15:21:35 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 12:21:35 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: a variable that survives a
 call gets put into a callee-saved register
Message-ID: <599c843f.53e81c0a.e51bb.042f@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92212:ce386eba1dfa
Date: 2017-08-22 21:20 +0200
http://bitbucket.org/pypy/pypy/changeset/ce386eba1dfa/

Log:	a variable that survives a call gets put into a callee-saved
	register

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -960,16 +960,13 @@
         if not free_regs:
             return None
         longevityvar = self[v]
+        # check whether there is a fixed register and whether it's free
         reg = longevityvar.find_fixed_register(position)
         if reg is not None and reg in free_regs:
             return reg
-        return free_regs[-1]
-        # more advanced stuff below, needs tests
 
-
-
-        loc, free_until = self.longevity.longest_free_reg(
-                self.position, free_regs)
+        # pick the register that's free the longest
+        loc, free_until = self.longest_free_reg(position, free_regs)
         if loc is None:
             return None
         # YYY could check whether it's best to spill v here, but hard
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1028,3 +1028,25 @@
             ("call_i", r0, [r1]),
             ("guard_false", r0, []),
         ]
+
+    def test_call_2(self):
+        ops = '''
+        [i0, i1]
+        i2 = int_mul(i0, 2)
+        i3 = int_add(i1, 1)
+        i4 = call_i(ConstClass(f1ptr), i2, descr=f1_calldescr)
+        guard_false(i4) [i3]
+        '''
+        emitted = self.allocate(ops)
+        fp0 = FakeFramePos(0, INT)
+        fp1 = FakeFramePos(1, INT)
+        assert emitted == [
+            ("move", r1, fp0),
+            ("int_mul", r1, [2]),
+            ("move", r4, fp1), # r4 gets picked since it's callee-saved
+            ("int_add", r4, [1]),
+            ("call_i", r0, [r1]),
+            ("guard_false", r0, [r4]),
+        ]
+
+

From pypy.commits at gmail.com  Tue Aug 22 16:55:42 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 13:55:42 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: another heuristic: if
 there's a fixed register around and the current
Message-ID: <599c9a4e.248fdf0a.23227.ceb4@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92213:e8eede93629e
Date: 2017-08-22 22:54 +0200
http://bitbucket.org/pypy/pypy/changeset/e8eede93629e/

Log:	another heuristic: if there's a fixed register around and the
	current to-be-allocated one fits before the next fixed use, use that
	(and use the smallest lifetime hole)

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -956,6 +956,30 @@
                     max_free_pos = free_until_pos
         return best_reg, max_free_pos
 
+    def free_reg_whole_lifetime(self, position, v, free_regs):
+        """ try to find a register from free_regs for v at position that's
+        free for the whole lifetime of v. pick the one that is blocked first
+        *after* the lifetime of v. """
+        longevityvar = self[v]
+        min_fixed_use_after = sys.maxint
+        best_reg = None
+        unfixed_reg = None
+        for reg in free_regs:
+            fixed_reg_pos = self.fixed_register_use.get(reg, None)
+            if fixed_reg_pos is None:
+                unfixed_reg = reg
+                continue
+            use_after = fixed_reg_pos.free_until_pos(longevityvar.last_usage)
+            assert use_after >= longevityvar.last_usage
+            if use_after < min_fixed_use_after:
+                best_reg = reg
+                min_fixed_use_after = use_after
+        if best_reg is not None:
+            return best_reg
+
+        # no fitting fixed registers. pick a non-fixed one
+        return unfixed_reg
+
     def try_pick_free_reg(self, position, v, free_regs):
         if not free_regs:
             return None
@@ -965,13 +989,19 @@
         if reg is not None and reg in free_regs:
             return reg
 
-        # pick the register that's free the longest
+        # try to find a register that's free for the whole lifetime of v
+        # pick the one that is blocked first *after* the lifetime of v
+        loc = self.free_reg_whole_lifetime(position, v, free_regs)
+        if loc is not None:
+            return loc
+
+        # can't fit v completely, so pick the register that's free the longest
         loc, free_until = self.longest_free_reg(position, free_regs)
-        if loc is None:
-            return None
+        if loc is not None:
+            return loc
         # YYY could check whether it's best to spill v here, but hard
         # to do in the current system
-        return loc
+        return None
 
     def __contains__(self, var):
         return var in self.longevity
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -52,6 +52,8 @@
         return 'r%d' % self.n
 
 r0, r1, r2, r3 = [FakeReg(i) for i in range(4)]
+r4, r5, r6, r7, r8, r9 = [FakeReg(i) for i in range(4, 10)]
+
 regs = [r0, r1, r2, r3]
 
 class RegisterManager(BaseRegMan):
@@ -270,6 +272,27 @@
 
     assert longevity.longest_free_reg(0, [r0, r1, r2]) == (r1, 2)
 
+def test_try_pick_free_reg():
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(0, 4)
+    l1 = Lifetime(2, 20)
+    l2 = Lifetime(6, 20)
+    l3 = Lifetime(8, 20)
+    l4 = Lifetime(0, 10)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2, b3: l3, b4: l4})
+    longevity.fixed_register(3, r1, b1)
+    longevity.fixed_register(7, r2, b2)
+    longevity.fixed_register(9, r3, b3)
+
+    # a best fit
+    loc = longevity.try_pick_free_reg(0, b0, [r1, r2, r3, r4, r5])
+    assert loc is r2
+
+    # does not fit into any of the fixed regs, use a non-fixed one
+    loc = longevity.try_pick_free_reg(0, b4, [r5, r2, r3, r4, r1])
+    assert loc in [r4, r5]
+
+
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)
@@ -815,7 +838,6 @@
 # _____________________________________________________
 # tests that assign registers in a mocked way for a fake CPU
 
-r4, r5, r6, r7, r8, r9 = [FakeReg(i) for i in range(4, 10)]
 
 class RegisterManager2(BaseRegMan):
     all_regs = [r0, r1, r2, r3, r4, r5, r6, r7]

From pypy.commits at gmail.com  Tue Aug 22 16:55:44 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 13:55:44 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: harder test for later
Message-ID: <599c9a50.c6141c0a.f616d.1461@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92214:f6f7d81e72e9
Date: 2017-08-22 22:55 +0200
http://bitbucket.org/pypy/pypy/changeset/f6f7d81e72e9/

Log:	harder test for later

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1071,4 +1071,21 @@
             ("guard_false", r0, [r4]),
         ]
 
-
+    def test_coalescing(self):
+        py.test.skip("hard - later")
+        ops = '''
+        [i0]
+        i2 = int_mul(i0, 2)
+        i3 = int_add(i2, 1) # i2 and i3 need to be coalesced
+        i4 = call_i(ConstClass(f1ptr), i3, descr=f1_calldescr)
+        guard_false(i4) []
+        '''
+        emitted = self.allocate(ops)
+        fp0 = FakeFramePos(0, INT)
+        assert emitted == [
+            ("move", r1, fp0),
+            ("int_mul", r1, [2]),
+            ("int_add", r1, [1]),
+            ("call_i", r0, [r1]),
+            ("guard_false", r0, []),
+        ]

From pypy.commits at gmail.com  Wed Aug 23 02:42:19 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 23:42:19 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: a test about the remaining
 case, and a comment
Message-ID: <599d23cb.db85df0a.c853d.6a56@mx.google.com>

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92215:358d42ebcdea
Date: 2017-08-23 07:04 +0200
http://bitbucket.org/pypy/pypy/changeset/358d42ebcdea/

Log:	a test about the remaining case, and a comment

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -700,6 +700,7 @@
 
         if len(move_or_spill) > 0:
             while len(self.free_regs) > 0:
+                # YYY here we need to use the new information to pick stuff
                 new_reg = self.free_regs.pop()
                 if new_reg in self.save_around_call_regs:
                     new_free_regs.append(new_reg)    # not this register...
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -292,6 +292,11 @@
     loc = longevity.try_pick_free_reg(0, b4, [r5, r2, r3, r4, r1])
     assert loc in [r4, r5]
 
+    # all available are fixed but var doesn't fit completely into any of these.
+    # pick the biggest interval
+    loc = longevity.try_pick_free_reg(0, b4, [r1, r2, r3])
+    assert loc is r3
+
 
 class TestRegalloc(object):
     def test_freeing_vars(self):

From pypy.commits at gmail.com  Wed Aug 23 02:42:21 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 23:42:21 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: first stab at coalescing
 support
Message-ID: <599d23cd.45b01c0a.2278b.4565@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92216:5c8cbd8502e7
Date: 2017-08-23 07:41 +0200
http://bitbucket.org/pypy/pypy/changeset/5c8cbd8502e7/

Log:	first stab at coalescing support

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -838,6 +838,11 @@
         # specific register
         self.fixed_positions = None
 
+        # another Lifetime that lives after the current one that would like to
+        # share a register with this variable
+        self.share_with = None
+
+
     def is_last_real_use_before(self, position):
         if self.real_usages is None:
             return True
@@ -873,11 +878,12 @@
 
     def find_fixed_register(self, opindex):
         # XXX could use binary search
-        if self.fixed_positions is None:
-            return None
-        for (index, reg) in self.fixed_positions:
-            if opindex <= index:
-                return reg
+        if self.fixed_positions is not None:
+            for (index, reg) in self.fixed_positions:
+                if opindex <= index:
+                    return reg
+        if self.share_with is not None:
+            return self.share_with.find_fixed_register(opindex)
 
     def _check_invariants(self):
         assert self.definition_pos <= self.last_usage
@@ -936,6 +942,17 @@
             self.fixed_register_use[register] = FixedRegisterPositions(register)
         self.fixed_register_use[register].fixed_register(opindex, definition_pos)
 
+    def try_use_same_register(self, v0, v1):
+        """ Try to arrange things to put v0 and v1 into the same register.
+        v0 must be defined before v1"""
+        # only works in limited situations now
+        longevityvar0 = self[v0]
+        longevityvar1 = self[v1]
+        assert longevityvar0.definition_pos < longevityvar1.definition_pos
+        if longevityvar0.last_usage != longevityvar1.definition_pos:
+            return # not supported for now
+        longevityvar0.share_with = longevityvar1
+
     def longest_free_reg(self, position, free_regs):
         """ for every register in free_regs, compute how far into the future
         that register can remain free, according to the constraints of the
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -297,6 +297,18 @@
     loc = longevity.try_pick_free_reg(0, b4, [r1, r2, r3])
     assert loc is r3
 
+def test_simple_coalescing():
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(0, 4)
+    l1 = Lifetime(4, 20)
+    l2 = Lifetime(4, 20)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2})
+    longevity.fixed_register(10, r1, b1)
+    longevity.fixed_register(10, r2, b2)
+    longevity.try_use_same_register(b0, b2)
+
+    loc = longevity.try_pick_free_reg(0, b0, [r0, r1, r2, r3, r4])
+    assert loc is r2
 
 class TestRegalloc(object):
     def test_freeing_vars(self):

From pypy.commits at gmail.com  Wed Aug 23 02:42:23 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 23:42:23 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: block the fixed register
 earlier after coalescing
Message-ID: <599d23cf.4692df0a.f2949.2b20@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92217:666fb2557e24
Date: 2017-08-23 07:52 +0200
http://bitbucket.org/pypy/pypy/changeset/666fb2557e24/

Log:	block the fixed register earlier after coalescing

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -842,6 +842,8 @@
         # share a register with this variable
         self.share_with = None
 
+        # the other lifetime will have this variable set to self.definition_pos
+        self.definition_pos_shared = UNDEF_POS
 
     def is_last_real_use_before(self, position):
         if self.real_usages is None:
@@ -869,7 +871,10 @@
         assert self.definition_pos <= position <= self.last_usage
         if self.fixed_positions is None:
             self.fixed_positions = []
-            res = self.definition_pos
+            if self.definition_pos_shared != UNDEF_POS:
+                res = self.definition_pos_shared
+            else:
+                res = self.definition_pos
         else:
             assert position > self.fixed_positions[-1][0]
             res = self.fixed_positions[-1][0]
@@ -952,6 +957,7 @@
         if longevityvar0.last_usage != longevityvar1.definition_pos:
             return # not supported for now
         longevityvar0.share_with = longevityvar1
+        longevityvar1.definition_pos_shared = longevityvar0.definition_pos
 
     def longest_free_reg(self, position, free_regs):
         """ for every register in free_regs, compute how far into the future
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -310,6 +310,26 @@
     loc = longevity.try_pick_free_reg(0, b0, [r0, r1, r2, r3, r4])
     assert loc is r2
 
+def test_coalescing_blocks_regs_correctly():
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(10, 30)
+    l1 = Lifetime(30, 40)
+    l2 = Lifetime(30, 40)
+    l3 = Lifetime(0, 15)
+    l4 = Lifetime(0, 5)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2, b3: l3, b4: l4})
+    longevity.try_use_same_register(b0, b1)
+    longevity.fixed_register(35, r1, b1)
+    longevity.fixed_register(35, r2, b2)
+
+    loc = longevity.try_pick_free_reg(0, b3, [r1, r2])
+    # r2 is picked, otherwise b0 can't b0 can't end up in r1
+    assert loc is r2
+
+    loc = longevity.try_pick_free_reg(0, b4, [r1, r2])
+    # r1 is picked, because b4 fits before b0
+    assert loc is r1
+
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)

From pypy.commits at gmail.com  Wed Aug 23 02:42:25 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Tue, 22 Aug 2017 23:42:25 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: implement chained
 coalescing. fix a bug in free_reg_whole_lifetime
Message-ID: <599d23d1.59451c0a.b46a8.9917@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92218:0c3b2571af1c
Date: 2017-08-23 08:41 +0200
http://bitbucket.org/pypy/pypy/changeset/0c3b2571af1c/

Log:	implement chained coalescing. fix a bug in free_reg_whole_lifetime

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -843,7 +843,7 @@
         self.share_with = None
 
         # the other lifetime will have this variable set to self.definition_pos
-        self.definition_pos_shared = UNDEF_POS
+        self._definition_pos_shared = UNDEF_POS
 
     def is_last_real_use_before(self, position):
         if self.real_usages is None:
@@ -864,6 +864,12 @@
                 low = mid + 1
         return l[low]
 
+    def definition_pos_shared(self):
+        if self._definition_pos_shared != UNDEF_POS:
+            return self._definition_pos_shared
+        else:
+            return self.definition_pos
+
     def fixed_register(self, position, reg):
         """ registers a fixed register use for the variable at position in
         register reg. returns the position from where on the register should be
@@ -871,10 +877,7 @@
         assert self.definition_pos <= position <= self.last_usage
         if self.fixed_positions is None:
             self.fixed_positions = []
-            if self.definition_pos_shared != UNDEF_POS:
-                res = self.definition_pos_shared
-            else:
-                res = self.definition_pos
+            res = self.definition_pos_shared()
         else:
             assert position > self.fixed_positions[-1][0]
             res = self.fixed_positions[-1][0]
@@ -957,7 +960,7 @@
         if longevityvar0.last_usage != longevityvar1.definition_pos:
             return # not supported for now
         longevityvar0.share_with = longevityvar1
-        longevityvar1.definition_pos_shared = longevityvar0.definition_pos
+        longevityvar1._definition_pos_shared = longevityvar0.definition_pos_shared()
 
     def longest_free_reg(self, position, free_regs):
         """ for every register in free_regs, compute how far into the future
@@ -993,7 +996,10 @@
             if fixed_reg_pos is None:
                 unfixed_reg = reg
                 continue
-            use_after = fixed_reg_pos.free_until_pos(longevityvar.last_usage)
+            use_after = fixed_reg_pos.free_until_pos(position)
+            if use_after < longevityvar.last_usage:
+                # can't fit
+                continue
             assert use_after >= longevityvar.last_usage
             if use_after < min_fixed_use_after:
                 best_reg = reg
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -297,6 +297,17 @@
     loc = longevity.try_pick_free_reg(0, b4, [r1, r2, r3])
     assert loc is r3
 
+def test_try_pick_free_reg_bug():
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(10, 30)
+    l1 = Lifetime(0, 15)
+    longevity = LifetimeManager({b0: l0, b1: l1})
+    longevity.fixed_register(20, r0, b0)
+
+    # does not fit into r0, use r1
+    loc = longevity.try_pick_free_reg(0, b1, [r0, r1])
+    assert loc == r1
+
 def test_simple_coalescing():
     b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
     l0 = Lifetime(0, 4)
@@ -330,6 +341,42 @@
     # r1 is picked, because b4 fits before b0
     assert loc is r1
 
+
+def test_chained_coalescing():
+    #              5 + b4
+    #                |
+    #  10  + b0      |
+    #      |         |
+    #      |      15 +
+    #      |
+    #      +
+    #  20
+    #      + b1
+    #      |
+    #      |
+    #      |
+    #      +
+    #  30
+    #      + b2
+    #      |
+    #  r1  *
+    #      |
+    #      +
+    #  40
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(10, 20)
+    l1 = Lifetime(20, 30)
+    l2 = Lifetime(30, 40)
+    l4 = Lifetime(5, 15)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2, b4: l4})
+    longevity.try_use_same_register(b0, b1)
+    longevity.try_use_same_register(b1, b2)
+    longevity.fixed_register(35, r1, b2)
+
+    loc = longevity.try_pick_free_reg(5, b4, [r0, r1])
+    assert loc is r0
+
+
 class TestRegalloc(object):
     def test_freeing_vars(self):
         b0, b1, b2 = newboxes(0, 0, 0)
@@ -356,7 +403,7 @@
         rm._check_invariants()
         assert len(rm.free_regs) == 4
         assert len(rm.reg_bindings) == 0
-        
+
     def test_register_exhaustion(self):
         boxes, longevity = boxes_and_longevity(5)
         rm = RegisterManager(longevity)

From pypy.commits at gmail.com  Wed Aug 23 02:56:41 2017
From: pypy.commits at gmail.com (mattip)
Date: Tue, 22 Aug 2017 23:56:41 -0700 (PDT)
Subject: [pypy-commit] pypy default: dummy implementation to make cython
 happier
Message-ID: <599d2729.83671c0a.13bc1.375f@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92219:bd4073222df1
Date: 2017-08-23 09:55 +0300
http://bitbucket.org/pypy/pypy/changeset/bd4073222df1/

Log:	dummy implementation to make cython happier

diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -94,6 +94,13 @@
     the fields used by the tp_traverse handler become invalid."""
     pass
 
+ at cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL)
+def PyType_IS_GC(space, o):
+    """Return true if the type object includes support for the cycle detector; this
+    tests the type flag Py_TPFLAGS_HAVE_GC.
+    """
+    return False
+
 @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL)
 def _PyObject_GetDictPtr(space, op):
     return lltype.nullptr(PyObjectP.TO)

From pypy.commits at gmail.com  Wed Aug 23 03:05:57 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 23 Aug 2017 00:05:57 -0700 (PDT)
Subject: [pypy-commit] pypy default: remove implemented function from stubs
Message-ID: <599d2955.24addf0a.5f478.4318@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92220:d91e29b93f3a
Date: 2017-08-23 10:04 +0300
http://bitbucket.org/pypy/pypy/changeset/d91e29b93f3a/

Log:	remove implemented function from stubs

diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -1506,13 +1506,6 @@
     """
     raise NotImplementedError
 
- at cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL)
-def PyType_IS_GC(space, o):
-    """Return true if the type object includes support for the cycle detector; this
-    tests the type flag Py_TPFLAGS_HAVE_GC.
-    """
-    raise NotImplementedError
-
 @cpython_api([], rffi.INT_real, error=CANNOT_FAIL)
 def PyUnicode_ClearFreeList(space):
     """Clear the free list. Return the total number of freed items.

From pypy.commits at gmail.com  Wed Aug 23 09:15:48 2017
From: pypy.commits at gmail.com (exarkun)
Date: Wed, 23 Aug 2017 06:15:48 -0700 (PDT)
Subject: [pypy-commit] buildbot cleanup-hg-bookmarks: Optionally delete
 .hg/bookmarks
Message-ID: <599d8004.11331c0a.b27fa.9bba@mx.google.com>

Author: Jean-Paul Calderone <exarkun at twistedmatrix.com>
Branch: cleanup-hg-bookmarks
Changeset: r1023:aea6e451355f
Date: 2017-08-23 09:15 -0400
http://bitbucket.org/pypy/buildbot/changeset/aea6e451355f/

Log:	Optionally delete .hg/bookmarks

diff --git a/bot2/pypybuildbot/builds.py b/bot2/pypybuildbot/builds.py
--- a/bot2/pypybuildbot/builds.py
+++ b/bot2/pypybuildbot/builds.py
@@ -333,11 +333,24 @@
            workdir=workdir))
 
 def update_hg(platform, factory, repourl, workdir, use_branch,
-              force_branch=None):
+              force_branch=None, wipe_bookmarks=False):
     if not use_branch:
         assert force_branch is None
         update_hg_old_method(platform, factory, repourl, workdir)
         return
+
+    if wipe_bookmarks:
+        # We don't use bookmarks at all.  If a bookmark accidentally gets
+        # created and pushed to the server and we pull it down, it gets stuck
+        # here.  Deleting it from the server doesn't seem to delete it from
+        # the local checkout.  So, manually clean it up.
+        factory.addStep(ShellCmd(
+            description="cleanup bookmarks",
+            command=["rm", "-f", ".hg/bookmarks"],
+            workdir=workdir,
+            haltOnFailure=False,
+        ))
+
     factory.addStep(
             Mercurial(
                 repourl=repourl,
@@ -374,7 +387,7 @@
                                   doStepIf=ParseRevision.doStepIf))
     #
     update_hg(platform, factory, repourl, workdir, use_branch=True,
-              force_branch=force_branch)
+              force_branch=force_branch, wipe_bookmarks=True)
     #
     factory.addStep(CheckGotRevision(workdir=workdir))
 
@@ -410,7 +423,7 @@
     # If target_tmpdir is empty, crash.
     tmp_or_crazy = '%(prop:target_tmpdir:-crazy/name/so/mkdir/fails/)s'
     pytest = "pytest"
-    factory.addStep(ShellCmd( 
+    factory.addStep(ShellCmd(
         description="mkdir for tests",
         command=['python', '-c', Interpolate("import os;  os.mkdir(r'" + \
                     tmp_or_crazy + pytest + "') if not os.path.exists(r'" + \
@@ -424,7 +437,7 @@
                    '/D', '-' + nDays, '/c', "cmd /c rmdir /q /s @path"]
     else:
         command = ['find', Interpolate(tmp_or_crazy + pytest), '-mtime',
-                   '+' + nDays, '-exec', 'rm', '-r', '{}', ';'] 
+                   '+' + nDays, '-exec', 'rm', '-r', '{}', ';']
     factory.addStep(SuccessAlways(
         description="cleanout old test files",
         command = command,
@@ -481,7 +494,7 @@
         # If target_tmpdir is empty, crash.
         tmp_or_crazy = '%(prop:target_tmpdir:-crazy/name/so/mkdir/fails/)s'
         pytest = "pytest"
-        self.addStep(ShellCmd( 
+        self.addStep(ShellCmd(
             description="mkdir for tests",
             command=['python', '-c', Interpolate("import os;  os.mkdir(r'" + \
                         tmp_or_crazy + pytest + "') if not os.path.exists(r'" + \
@@ -495,7 +508,7 @@
                        '/D', '-' + nDays, '/c', "cmd /c rmdir /q /s @path"]
         else:
             command = ['find', Interpolate(tmp_or_crazy + pytest), '-mtime',
-                       '+' + nDays, '-exec', 'rm', '-r', '{}', ';'] 
+                       '+' + nDays, '-exec', 'rm', '-r', '{}', ';']
         self.addStep(SuccessAlways(
             description="cleanout old test files",
             command = command,
@@ -976,7 +989,7 @@
             workdir='pypy-c',
             haltOnFailure=True,
             ))
-        
+
         if platform == 'win32':
             self.addStep(ShellCmd(
                 description='move decompressed dir',
@@ -1010,7 +1023,7 @@
         # obtain a pypy-compatible branch of numpy
         numpy_url = 'https://www.bitbucket.org/pypy/numpy'
         update_git(platform, self, numpy_url, 'numpy_src', branch='master',
-                   alwaysUseLatest=True, # ignore pypy rev number when 
+                   alwaysUseLatest=True, # ignore pypy rev number when
                                          # triggered by a pypy build
                    )
 

From pypy.commits at gmail.com  Wed Aug 23 09:40:00 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 06:40:00 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Issue #2638
Message-ID: <599d85b0.090b1c0a.41761.a788@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92221:b4d19ffb3525
Date: 2017-08-23 15:39 +0200
http://bitbucket.org/pypy/pypy/changeset/b4d19ffb3525/

Log:	Issue #2638

	Workaround for .python_history files that have non-utf-8 chars.

	But ideally we should encode/decode the content in some locale-aware
	format, instead of using UTF-8 all the time.

diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -314,7 +314,8 @@
         # history item: we use \r\n instead of just \n.  If the history
         # file is passed to GNU readline, the extra \r are just ignored.
         history = self.get_reader().history
-        f = open(os.path.expanduser(filename), 'r', encoding='utf-8')
+        f = open(os.path.expanduser(filename), 'r', encoding='utf-8',
+                 errors='replace')
         buffer = []
         for line in f:
             if line.endswith('\r\n'):

From pypy.commits at gmail.com  Wed Aug 23 10:49:30 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 07:49:30 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Fix error message
Message-ID: <599d95fa.11421c0a.729dd.e4cd@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92222:d459f7e19937
Date: 2017-08-23 16:48 +0200
http://bitbucket.org/pypy/pypy/changeset/d459f7e19937/

Log:	Fix error message

diff --git a/pypy/objspace/std/objectobject.py b/pypy/objspace/std/objectobject.py
--- a/pypy/objspace/std/objectobject.py
+++ b/pypy/objspace/std/objectobject.py
@@ -143,7 +143,7 @@
     from pypy.objspace.std.typeobject import W_TypeObject
     if not isinstance(w_newcls, W_TypeObject):
         raise oefmt(space.w_TypeError,
-                    "__class__ must be set to new-style class, not '%T' "
+                    "__class__ must be set to a class, not '%T' "
                     "object", w_newcls)
     if not w_newcls.is_heaptype():
         raise oefmt(space.w_TypeError,

From pypy.commits at gmail.com  Wed Aug 23 11:31:55 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 08:31:55 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Simplification
Message-ID: <599d9feb.4b6b1c0a.4ff12.b7a9@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92223:34f222bdfa0b
Date: 2017-08-23 17:29 +0200
http://bitbucket.org/pypy/pypy/changeset/34f222bdfa0b/

Log:	Simplification

diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -130,7 +130,7 @@
         return subcls
 _unique_subclass_cache = {}
 
-def _getusercls(cls, reallywantdict=False):
+def _getusercls(cls):
     from rpython.rlib import objectmodel
     from pypy.objspace.std.objectobject import W_ObjectObject
     from pypy.objspace.std.mapdict import (BaseUserClassMapdict,
@@ -144,7 +144,7 @@
     else:
         base_mixin = MapdictStorageMixin
     copy_methods = [BaseUserClassMapdict]
-    if reallywantdict or not typedef.hasdict:
+    if not typedef.hasdict:
         # the type has no dict, mapdict to provide the dict
         copy_methods.append(MapdictDictSupport)
         name += "Dict"

From pypy.commits at gmail.com  Wed Aug 23 11:31:58 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 08:31:58 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Issue #2639
Message-ID: <599d9fee.9aa4df0a.99c3f.9044@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92224:8fabef083b67
Date: 2017-08-23 17:31 +0200
http://bitbucket.org/pypy/pypy/changeset/8fabef083b67/

Log:	Issue #2639

	Assigning '__class__' between ModuleType and subclasses

diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -10,9 +10,10 @@
 class Module(W_Root):
     """A module."""
 
-    _immutable_fields_ = ["w_dict?"]
+    _immutable_fields_ = ["w_dict?", "w_userclass?"]
 
     _frozen = False
+    w_userclass = None
 
     def __init__(self, space, w_name, w_dict=None):
         self.space = space
@@ -148,6 +149,26 @@
                         self)
         return space.call_function(space.w_list, w_dict)
 
+    # These three methods are needed to implement '__class__' assignment
+    # between a module and a subclass of module.  They give every module
+    # the ability to have its '__class__' set, manually.  Note that if
+    # you instantiate a subclass of ModuleType in the first place, then
+    # you get an RPython instance of a subclass of Module created in the
+    # normal way by typedef.py.  That instance has got its own
+    # getclass(), getslotvalue(), etc. but provided it has no __slots__,
+    # it is compatible with ModuleType for '__class__' assignment.
+
+    def getclass(self, space):
+        if self.w_userclass is None:
+            return W_Root.getclass(self, space)
+        return self.w_userclass
+
+    def setclass(self, space, w_cls):
+        self.w_userclass = w_cls
+
+    def user_setup(self, space, w_subtype):
+        self.w_userclass = w_subtype
+
 
 def init_extra_module_attrs(space, w_mod):
     w_dict = w_mod.getdict(space)
diff --git a/pypy/interpreter/test/test_module.py b/pypy/interpreter/test/test_module.py
--- a/pypy/interpreter/test/test_module.py
+++ b/pypy/interpreter/test/test_module.py
@@ -220,3 +220,45 @@
         import sys
         m = type(sys).__new__(type(sys))
         assert not m.__dict__
+
+    def test_class_assignment_for_module(self):
+        import sys
+        modtype = type(sys)
+        class X(modtype):
+            _foobar_ = 42
+
+        m = X("yytest_moduleyy")
+        assert type(m) is m.__class__ is X
+        assert m._foobar_ == 42
+        m.__class__ = modtype
+        assert type(m) is m.__class__ is modtype
+        assert not hasattr(m, '_foobar_')
+
+        m = modtype("xxtest_modulexx")
+        assert type(m) is m.__class__ is modtype
+        m.__class__ = X
+        assert m._foobar_ == 42
+        assert type(m) is m.__class__ is X
+
+        sys.__class__ = modtype
+        assert type(sys) is sys.__class__ is modtype
+        sys.__class__ = X
+        assert sys._foobar_ == 42
+        sys.__class__ = modtype
+
+        class XX(modtype):
+            __slots__ = ['a', 'b']
+
+        x = XX("zztest_modulezz")
+        assert x.__class__ is XX
+        raises(AttributeError, "x.a")
+        x.a = 42
+        assert x.a == 42
+        x.a = 43
+        assert x.a == 43
+        assert 'a' not in x.__dict__
+        del x.a
+        raises(AttributeError, "x.a")
+        raises(AttributeError, "del x.a")
+        raises(TypeError, "x.__class__ = X")
+        raises(TypeError, "sys.__class__ = XX")
diff --git a/pypy/objspace/std/objectobject.py b/pypy/objspace/std/objectobject.py
--- a/pypy/objspace/std/objectobject.py
+++ b/pypy/objspace/std/objectobject.py
@@ -141,13 +141,17 @@
 
 def descr_set___class__(space, w_obj, w_newcls):
     from pypy.objspace.std.typeobject import W_TypeObject
+    from pypy.interpreter.module import Module
+    #
     if not isinstance(w_newcls, W_TypeObject):
         raise oefmt(space.w_TypeError,
                     "__class__ must be set to a class, not '%T' "
                     "object", w_newcls)
-    if not w_newcls.is_heaptype():
+    if not (w_newcls.is_heaptype() or
+            w_newcls is space.gettypeobject(Module.typedef)):
         raise oefmt(space.w_TypeError,
-                    "__class__ assignment: only for heap types")
+                    "__class__ assignment only supported for heap types "
+                    "or ModuleType subclasses")
     w_oldcls = space.type(w_obj)
     assert isinstance(w_oldcls, W_TypeObject)
     if (w_oldcls.get_full_instance_layout() ==

From pypy.commits at gmail.com  Wed Aug 23 11:56:45 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 08:56:45 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Test and fix
 (lib-python/3/test/test_descr)
Message-ID: <599da5bd.c35c1c0a.55d8.b680@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92225:c1fb69da92dc
Date: 2017-08-23 17:56 +0200
http://bitbucket.org/pypy/pypy/changeset/c1fb69da92dc/

Log:	Test and fix (lib-python/3/test/test_descr)

diff --git a/pypy/objspace/std/test/test_typeobject.py b/pypy/objspace/std/test/test_typeobject.py
--- a/pypy/objspace/std/test/test_typeobject.py
+++ b/pypy/objspace/std/test/test_typeobject.py
@@ -1284,6 +1284,25 @@
         raises(ValueError, type, 'A\x00B', (), {})
         raises(TypeError, type, b'A', (), {})
 
+    def test_incomplete_extend(self): """
+        # Extending an unitialized type with type.__mro__ is None must
+        # throw a reasonable TypeError exception, instead of failing
+        # with a segfault.
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ != 'X':
+                    try:
+                        class X(cls):
+                            pass
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class A(metaclass=M):
+            pass
+        assert found == [1]
+        """
+
 
 class AppTestWithMethodCacheCounter:
     spaceconfig = {"objspace.std.withmethodcachecounter": True}
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -1055,6 +1055,9 @@
     if w_bestbase is None:
         raise oefmt(space.w_TypeError,
                     "a new-style class can't have only classic bases")
+    if not w_bestbase.hasmro:
+        raise oefmt(space.w_TypeError,
+                    "Cannot extend an incomplete type '%N'", w_bestbase)
     if not w_bestbase.layout.typedef.acceptable_as_base_class:
         raise oefmt(space.w_TypeError,
                     "type '%N' is not an acceptable base class", w_bestbase)

From pypy.commits at gmail.com  Wed Aug 23 12:01:34 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 09:01:34 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Refinement of c1fb69da92dc
Message-ID: <599da6de.1db7df0a.98fc4.897f@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92226:e8a566e9fcd9
Date: 2017-08-23 18:00 +0200
http://bitbucket.org/pypy/pypy/changeset/e8a566e9fcd9/

Log:	Refinement of c1fb69da92dc

diff --git a/pypy/objspace/std/test/test_typeobject.py b/pypy/objspace/std/test/test_typeobject.py
--- a/pypy/objspace/std/test/test_typeobject.py
+++ b/pypy/objspace/std/test/test_typeobject.py
@@ -1303,6 +1303,27 @@
         assert found == [1]
         """
 
+    def test_incomplete_extend_2(self): """
+        # Same as test_incomplete_extend, with multiple inheritance
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ == 'Second':
+                    try:
+                        class X(First, cls):
+                            pass
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class Base(metaclass=M):
+            pass
+        class First(Base):
+            pass
+        class Second(Base):
+            pass
+        assert found == [1]
+        """
+
 
 class AppTestWithMethodCacheCounter:
     spaceconfig = {"objspace.std.withmethodcachecounter": True}
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -1036,6 +1036,9 @@
     for w_candidate in bases_w:
         if not isinstance(w_candidate, W_TypeObject):
             continue
+        if not w_candidate.hasmro:
+            raise oefmt(w_candidate.space.w_TypeError,
+                        "Cannot extend an incomplete type '%N'", w_candidate)
         if w_bestbase is None:
             w_bestbase = w_candidate   # for now
             continue
@@ -1055,9 +1058,6 @@
     if w_bestbase is None:
         raise oefmt(space.w_TypeError,
                     "a new-style class can't have only classic bases")
-    if not w_bestbase.hasmro:
-        raise oefmt(space.w_TypeError,
-                    "Cannot extend an incomplete type '%N'", w_bestbase)
     if not w_bestbase.layout.typedef.acceptable_as_base_class:
         raise oefmt(space.w_TypeError,
                     "type '%N' is not an acceptable base class", w_bestbase)

From pypy.commits at gmail.com  Wed Aug 23 12:10:45 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 09:10:45 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: This case "works",
 but gives a slightly strange error message
Message-ID: <599da905.491a1c0a.b390c.1e46@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92227:c30f2d4e721f
Date: 2017-08-23 18:10 +0200
http://bitbucket.org/pypy/pypy/changeset/c30f2d4e721f/

Log:	This case "works", but gives a slightly strange error message on
	both CPython and PyPy

diff --git a/pypy/objspace/std/test/test_typeobject.py b/pypy/objspace/std/test/test_typeobject.py
--- a/pypy/objspace/std/test/test_typeobject.py
+++ b/pypy/objspace/std/test/test_typeobject.py
@@ -1324,6 +1324,25 @@
         assert found == [1]
         """
 
+    def test_incomplete_extend_3(self): """
+        # this case "works", but gives a slightly strange error message
+        # on both CPython and PyPy
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ == 'A':
+                    try:
+                        Base.__new__(cls)
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class Base(metaclass=M):
+            pass
+        class A(Base):
+            pass
+        assert found == [1]
+        """
+
 
 class AppTestWithMethodCacheCounter:
     spaceconfig = {"objspace.std.withmethodcachecounter": True}

From pypy.commits at gmail.com  Wed Aug 23 12:23:47 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 09:23:47 -0700 (PDT)
Subject: [pypy-commit] pypy default: (fijal, arigo)
Message-ID: <599dac13.4ad61c0a.c9047.bb68@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92228:18b10a6743c4
Date: 2017-08-23 18:21 +0200
http://bitbucket.org/pypy/pypy/changeset/18b10a6743c4/

Log:	(fijal, arigo)

	Rename _pypy_dll to __pypy_dll__, so that the custom __getattr__
	would raise if the attribute doesn't exist

diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -364,7 +364,7 @@
                 pypy_dll = _ffi.CDLL(name, mode)
             else:
                 pypy_dll = _ffi.WinDLL(name, mode)
-            self._pypy_dll = pypy_dll
+            self.__pypy_dll__ = pypy_dll
             handle = int(pypy_dll)
             if _sys.maxint > 2 ** 32:
                 handle = int(handle)   # long -> int
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._pypy_dll.getaddressindll(name))
+        return self.from_address(dll.__pypy_dll__.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._pypy_dll
+        cdll = self.dll.__pypy_dll__
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()

From pypy.commits at gmail.com  Wed Aug 23 12:23:49 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 09:23:49 -0700 (PDT)
Subject: [pypy-commit] pypy default: (fijal, arigo)
Message-ID: <599dac15.81131c0a.37075.e37e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92229:b02f6ce0d05b
Date: 2017-08-23 18:23 +0200
http://bitbucket.org/pypy/pypy/changeset/b02f6ce0d05b/

Log:	(fijal, arigo)

	Improve the explanation a bit

diff --git a/pypy/module/test_lib_pypy/README.txt b/pypy/module/test_lib_pypy/README.txt
--- a/pypy/module/test_lib_pypy/README.txt
+++ b/pypy/module/test_lib_pypy/README.txt
@@ -1,4 +1,7 @@
 This directory contains app-level tests are supposed to be run *after*
 translation. So you run them by saying:
 
-pypy pytest.py <testfile.py>
+../../goal/pypy-c pytest.py <testfile.py>
+
+Note that if you run it with a PyPy from elsewhere, it will not pick
+up the changes to lib-python and lib_pypy.

From pypy.commits at gmail.com  Wed Aug 23 12:34:31 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 23 Aug 2017 09:34:31 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: more realistic jump faking
Message-ID: <599dae97.6ea9df0a.97a44.4572@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92231:3ba71498fe77
Date: 2017-08-23 11:07 +0200
http://bitbucket.org/pypy/pypy/changeset/3ba71498fe77/

Log:	more realistic jump faking

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -997,8 +997,8 @@
 
     def fake_allocate(self, loop):
         from rpython.jit.backend.x86.jump import remap_frame_layout
-
-        emit = self.assembler.emitted.append
+        def emit(*args):
+            self.assembler.emitted.append(args)
         for i, op in enumerate(loop.operations):
             self.rm.position = i
             opnum = op.getopnum()
@@ -1006,15 +1006,15 @@
             if rop.is_comparison(opnum):
                 locs = [self.loc(x) for x in op.getarglist()]
                 loc = self.force_allocate_reg_or_cc(op)
-                emit((opname, loc, locs))
+                emit(opname, loc, locs)
             elif opname.startswith("int_"):
                 locs = [self.loc(x) for x in op.getarglist()]
                 loc = self.rm.force_result_in_reg(
                     op, op.getarg(0), op.getarglist())
-                emit((opname, loc, locs[1:]))
+                emit(opname, loc, locs[1:])
             elif op.is_guard():
                 fail_locs = [self.loc(x) for x in op.getfailargs()]
-                emit((opname, self.loc(op.getarg(0)), fail_locs))
+                emit(opname, self.loc(op.getarg(0)), fail_locs)
             elif rop.is_call(opnum):
                 # calling convention!
                 src_locs = [self.loc(x) for x in op.getarglist()[1:]]
@@ -1022,19 +1022,24 @@
                 loc = self.rm.after_call(op)
                 dst_locs = [r1, r2, r3][:len(src_locs)]
                 remap_frame_layout(self.assembler, src_locs, dst_locs, r8)
-                emit((opname, loc, dst_locs))
+                emit(opname, loc, dst_locs)
             elif opname == "label":
                 descr = op.getdescr()
                 locs = [self.loc(x) for x in op.getarglist()]
-                emit((opname, locs))
+                emit(opname, locs)
                 descr._fake_arglocs = locs
+            elif opname == "jump":
+                src_locs = [self.loc(x) for x in op.getarglist()]
+                dst_locs = op.getdescr()._fake_arglocs
+                remap_frame_layout(self.assembler, src_locs, dst_locs, r8)
+                emit("jump", dst_locs)
             else:
                 locs = [self.loc(x) for x in op.getarglist()]
                 if op.type != "v":
                     loc = self.rm.force_allocate_reg(op)
-                    emit((opname, loc, locs))
+                    emit(opname, loc, locs)
                 else:
-                    emit((opname, locs))
+                    emit(opname, locs)
             self.possibly_free_vars_for_op(op)
         return self.assembler.emitted
 
@@ -1131,7 +1136,8 @@
             ("int_add", r0, [1]),
             ("int_lt", r8, [r0, 20]),
             ("guard_true", r8, [r0]),
-            ("jump", [r0]),
+            ("move", fp0, r0),
+            ("jump", [fp0]),
         ]
 
     def test_call(self):

From pypy.commits at gmail.com  Wed Aug 23 12:34:32 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 23 Aug 2017 09:34:32 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: make it possible to specify
 the locs of the inputargs in the tests. a skipped
Message-ID: <599dae98.6ea9df0a.97a44.4579@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92232:3cde3ad8b828
Date: 2017-08-23 11:53 +0200
http://bitbucket.org/pypy/pypy/changeset/3cde3ad8b828/

Log:	make it possible to specify the locs of the inputargs in the tests.
	a skipped test that's messy to fix.

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -945,12 +945,15 @@
     def __init__(self):
         self.assembler = MockAsm()
 
-    def prepare_loop(self, inputargs, operations, looptoken, allgcrefs):
-        operations = self._prepare(inputargs, operations, allgcrefs)
+    def fake_prepare_loop(self, inputargs, operations, looptoken, inputarg_locs=None):
+        operations = self._prepare(inputargs, operations, [])
         self.operations = operations
-        self._set_initial_bindings(inputargs, looptoken)
-        # note: we need to make a copy of inputargs because possibly_free_vars
-        # is also used on op args, which is a non-resizable list
+        if inputarg_locs is None:
+            self._set_initial_bindings(inputargs, looptoken)
+        else:
+            for v, loc in zip(inputargs, inputarg_locs):
+                self.rm.reg_bindings[v] = loc
+                self.rm.free_regs.remove(loc)
         self.possibly_free_vars(list(inputargs))
         self._add_fixed_registers()
         return operations
@@ -1105,12 +1108,12 @@
         return parse(s, self.cpu, namespace or self.namespace,
                      boxkinds=boxkinds)
 
-    def allocate(self, s):
+    def allocate(self, s, inputarg_locs=None):
         loop = self.parse(s)
         self.loop = loop
         regalloc = FakeRegalloc()
-        regalloc.prepare_loop(loop.inputargs, loop.operations,
-                              loop.original_jitcell_token, [])
+        regalloc.fake_prepare_loop(loop.inputargs, loop.operations,
+                                   loop.original_jitcell_token, inputarg_locs)
         self.regalloc = regalloc
         return regalloc.fake_allocate(loop)
 
@@ -1200,3 +1203,36 @@
             ('call_i', r0, [r1]),
             ('guard_false', r0, [])
         ]
+
+    def test_specify_inputarg_locs(self):
+        ops = '''
+        [i0]
+        i1 = int_mul(i0, 5)
+        i5 = int_is_true(i1)
+        guard_true(i5) []
+        '''
+        emitted = self.allocate(ops, [r0])
+        assert emitted == [
+            ('int_mul', r0, [5]),
+            ('int_is_true', r8, [r0]),
+            ('guard_true', r8, [])
+        ]
+
+    def test_coalescing_first_var_already_in_different_reg(self):
+        py.test.skip("messy - later")
+        ops = '''
+        [i0]
+        i2 = int_mul(i0, 2)
+        i3 = int_add(i2, 1) # i2 and i3 need to be coalesced
+        i4 = call_i(ConstClass(f1ptr), i3, descr=f1_calldescr)
+        guard_false(i4) [i0]
+        '''
+        emitted = self.allocate(ops, [r5])
+        assert emitted == [
+            ('move', r1, r5),
+            ('int_mul', r1, [2]),
+            ('int_add', r1, [1]),
+            ('call_i', r0, [r1]),
+            ('guard_false', r0, [])
+        ]
+

From pypy.commits at gmail.com  Wed Aug 23 12:34:29 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 23 Aug 2017 09:34:29 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: now the coalescing
 integration test works
Message-ID: <599dae95.54871c0a.dcbf8.e94d@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92230:c02b584c919d
Date: 2017-08-23 10:42 +0200
http://bitbucket.org/pypy/pypy/changeset/c02b584c919d/

Log:	now the coalescing integration test works

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -972,6 +972,14 @@
             if var is not None: # xxx kludgy
                 self.possibly_free_var(var)
 
+    def possibly_free_vars_for_op(self, op):
+        for i in range(op.numargs()):
+            var = op.getarg(i)
+            if var is not None: # xxx kludgy
+                self.possibly_free_var(var)
+        if op.type != 'v':
+            self.possibly_free_var(op)
+
     def loc(self, x):
         return self.rm.loc(x)
 
@@ -1027,16 +1035,23 @@
                     emit((opname, loc, locs))
                 else:
                     emit((opname, locs))
+            self.possibly_free_vars_for_op(op)
         return self.assembler.emitted
 
     def _add_fixed_registers(self):
         for i, op in enumerate(self.operations):
-            if rop.is_call(op.getopnum()):
+            opnum = op.getopnum()
+            opname = op.getopname()
+            args = op.getarglist()
+            if rop.is_call(opnum):
                 # calling convention!
                 arglist = op.getarglist()[1:]
                 for arg, reg in zip(arglist + [None] * (3 - len(arglist)), [r1, r2, r3]):
                     self.longevity.fixed_register(i, reg, arg)
                 self.longevity.fixed_register(i, r0, op)
+            elif opname.startswith("int_"):
+                if not args[0].is_constant():
+                    self.longevity.try_use_same_register(args[0], op)
 
 
 CPU = getcpuclass()
@@ -1156,9 +1171,11 @@
         ]
 
     def test_coalescing(self):
-        py.test.skip("hard - later")
         ops = '''
         [i0]
+        i1 = int_mul(i0, 5)
+        i5 = int_is_true(i1)
+        guard_true(i5) []
         i2 = int_mul(i0, 2)
         i3 = int_add(i2, 1) # i2 and i3 need to be coalesced
         i4 = call_i(ConstClass(f1ptr), i3, descr=f1_calldescr)
@@ -1167,9 +1184,13 @@
         emitted = self.allocate(ops)
         fp0 = FakeFramePos(0, INT)
         assert emitted == [
-            ("move", r1, fp0),
-            ("int_mul", r1, [2]),
-            ("int_add", r1, [1]),
-            ("call_i", r0, [r1]),
-            ("guard_false", r0, []),
+            ('move', r1, fp0),
+            ('int_mul', r1, [5]),
+            ('int_is_true', r8, [r1]),
+            ('guard_true', r8, []),
+            ('move', r1, fp0),
+            ('int_mul', r1, [2]),
+            ('int_add', r1, [1]),
+            ('call_i', r0, [r1]),
+            ('guard_false', r0, [])
         ]

From pypy.commits at gmail.com  Wed Aug 23 12:34:34 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 23 Aug 2017 09:34:34 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: use the regular spill
 heuristics to chose the variables to spill before a call
Message-ID: <599dae9a.c7181c0a.7d20a.321d@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92233:f68f729a80cf
Date: 2017-08-23 18:33 +0200
http://bitbucket.org/pypy/pypy/changeset/f68f729a80cf/

Log:	use the regular spill heuristics to chose the variables to spill
	before a call

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -418,20 +418,24 @@
         return loc
 
     def _pick_variable_to_spill(self, v, forbidden_vars, selected_reg=None,
-                                need_lower_byte=False):
+                                need_lower_byte=False, vars=None):
+        # YYY v is unused, remove
+
         # try to spill a variable that has no further real usages, ie that only
         # appears in failargs or in a jump
         # if that doesn't exist, spill the variable that has a real_usage that
         # is the furthest away from the current position
 
         # YYY check for fixed variable usages
+        if vars is None:
+            vars = self.reg_bindings.keys()
 
         cur_max_use_distance = -1
         position = self.position
         candidate = None
         cur_max_age_failargs = -1
         candidate_from_failargs = None
-        for next in self.reg_bindings:
+        for next in vars:
             reg = self.reg_bindings[next]
             if next in forbidden_vars:
                 continue
@@ -696,46 +700,31 @@
             else:
                 # this is a register like eax/rax, which needs either
                 # spilling or moving.
-                move_or_spill.append((v, max_age))
+                move_or_spill.append(v)
 
         if len(move_or_spill) > 0:
-            while len(self.free_regs) > 0:
-                # YYY here we need to use the new information to pick stuff
-                new_reg = self.free_regs.pop()
-                if new_reg in self.save_around_call_regs:
-                    new_free_regs.append(new_reg)    # not this register...
-                    continue
-                # This 'new_reg' is suitable for moving a candidate to.
-                # Pick the one with the smallest max_age.  (This
-                # is one step of a naive sorting algo, slow in theory,
-                # but the list should always be very small so it
-                # doesn't matter.)
-                best_i = 0
-                smallest_max_age = move_or_spill[0][1]
-                for i in range(1, len(move_or_spill)):
-                    max_age = move_or_spill[i][1]
-                    if max_age < smallest_max_age:
-                        best_i = i
-                        smallest_max_age = max_age
-                v, max_age = move_or_spill.pop(best_i)
-                # move from 'reg' to 'new_reg'
+            free_regs = [reg for reg in self.free_regs
+                             if reg not in self.save_around_call_regs]
+            # chose which to spill using the usual spill heuristics
+            while len(move_or_spill) > len(free_regs):
+                v = self._pick_variable_to_spill(None, [], vars=move_or_spill)
+                self._bc_spill(v, new_free_regs)
+                move_or_spill.remove(v)
+            assert len(move_or_spill) <= len(free_regs)
+            for v in move_or_spill:
+                # search next good reg
+                new_reg = None
+                while True:
+                    new_reg = self.free_regs.pop()
+                    if new_reg in self.save_around_call_regs:
+                        new_free_regs.append(new_reg)    # not this register...
+                        continue
+                    break
+                assert new_reg is not None # must succeed
                 reg = self.reg_bindings[v]
-                if not we_are_translated():
-                    if move_or_spill:
-                        assert max_age <= min([_a for _, _a in move_or_spill])
-                    assert reg in save_sublist
-                    assert reg in self.save_around_call_regs
-                    assert new_reg not in self.save_around_call_regs
                 self.assembler.regalloc_mov(reg, new_reg)
                 self.reg_bindings[v] = new_reg    # change the binding
                 new_free_regs.append(reg)
-                #
-                if len(move_or_spill) == 0:
-                    break
-            else:
-                # no more free registers to move to, spill the rest
-                for v, max_age in move_or_spill:
-                    self._bc_spill(v, new_free_regs)
 
         # re-add registers in 'new_free_regs', but in reverse order,
         # so that the last ones (added just above, from
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -617,6 +617,7 @@
                               assembler=asm)
         for b in boxes[:-1]:
             rm.force_allocate_reg(b)
+        rm.position = 0
         rm.before_call()
         assert len(rm.reg_bindings) == 2
         assert fm.get_frame_depth() == 2
@@ -1236,3 +1237,35 @@
             ('guard_false', r0, [])
         ]
 
+    def test_call_spill_furthest_use(self):
+        # here, i2 should be spilled, because its use is farther away
+        ops = '''
+        [i0, i1, i2, i3, i4, i5, i6]
+        i8 = call_i(ConstClass(f2ptr), i0, i1, descr=f2_calldescr)
+        escape_i(i3)
+        escape_i(i2)
+        guard_false(i8) [i2, i3, i4, i5, i6]
+        '''
+        emitted = self.allocate(ops, [r1, r2, r0, r3, r4, r5, r6])
+        fp0 = FakeFramePos(0, INT)
+        assert emitted == [
+            ('move', fp0, r0),
+            ('move', r7, r3),
+            ('call_i', r0, [r1, r2]),
+            ('escape_i', r1, [r7]),
+            ('escape_i', r1, [fp0]),
+            ('guard_false', r0, [fp0, r7, r4, r5, r6])
+        ]
+
+    def test_call_spill(self):
+        py.test.skip("also messy")
+        # i0 dies, i1 is the argument, the other fight for caller-saved regs
+        # all_regs = [r0, r1, r2, r3, r4, r5, r6, r7]
+        # save_around_call_regs = [r0, r1, r2, r3]
+        ops = '''
+        [i0, i1, i2, i3, i4, i5, i6]
+        i8 = call_i(ConstClass(f2ptr), i1, i0, descr=f2_calldescr)
+        guard_false(i8) [i2, i3, i4, i5, i6]
+        '''
+        emitted = self.allocate(ops, [r5, r1, r0, r2, r3, r6, r7])
+        assert emitted == ["???"]

From pypy.commits at gmail.com  Wed Aug 23 12:45:26 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Wed, 23 Aug 2017 09:45:26 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: just to check that jump
 hinting would work too
Message-ID: <599db126.6ea0df0a.5700f.a1e1@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92234:ac35a22e47df
Date: 2017-08-23 18:44 +0200
http://bitbucket.org/pypy/pypy/changeset/ac35a22e47df/

Log:	just to check that jump hinting would work too

diff --git a/pytest.ini b/pytest.ini
--- a/pytest.ini
+++ b/pytest.ini
@@ -1,2 +0,0 @@
-[pytest]
-addopts = --assert=reinterp -rf
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -1032,6 +1032,14 @@
                 locs = [self.loc(x) for x in op.getarglist()]
                 emit(opname, locs)
                 descr._fake_arglocs = locs
+                lastop = loop.operations[-1]
+                if lastop.getopname() == "jump" and lastop.getdescr() is descr:
+                    # now we know the places, add hints
+                    for i, r in enumerate(locs):
+                        if isinstance(r, FakeReg):
+                            self.longevity.fixed_register(
+                                len(loop.operations) - 1, r, lastop.getarg(i))
+
             elif opname == "jump":
                 src_locs = [self.loc(x) for x in op.getarglist()]
                 dst_locs = op.getdescr()._fake_arglocs
@@ -1269,3 +1277,24 @@
         '''
         emitted = self.allocate(ops, [r5, r1, r0, r2, r3, r6, r7])
         assert emitted == ["???"]
+
+    def test_jump_hinting(self):
+        ops = '''
+        [i0, i1]
+        i2 = escape_i()
+        i3 = escape_i()
+        label(i2, i3, descr=targettoken)
+        i4 = escape_i()
+        i5 = escape_i()
+        jump(i4, i5, descr=targettoken)
+        '''
+        self.targettoken._fake_arglocs = [r5, r6]
+        emitted = self.allocate(ops)
+        assert emitted == [
+            ('escape_i', r0, []),
+            ('escape_i', r1, []),
+            ('label', [r0, r1]),
+            ('escape_i', r0, []),
+            ('escape_i', r1, []),
+            ('jump', [r0, r1])
+        ]
diff --git a/rpython/pytest.ini b/rpython/pytest.ini
--- a/rpython/pytest.ini
+++ b/rpython/pytest.ini
@@ -1,2 +0,0 @@
-[pytest]
-addopts = --assert=reinterp -rf

From pypy.commits at gmail.com  Wed Aug 23 12:55:53 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 09:55:53 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <599db399.91a5df0a.9f412.9f24@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92235:d4535b1f44cd
Date: 2017-08-23 18:55 +0200
http://bitbucket.org/pypy/pypy/changeset/d4535b1f44cd/

Log:	hg merge default

diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -364,7 +364,7 @@
                 pypy_dll = _ffi.CDLL(name, mode)
             else:
                 pypy_dll = _ffi.WinDLL(name, mode)
-            self._pypy_dll = pypy_dll
+            self.__pypy_dll__ = pypy_dll
             handle = int(pypy_dll)
             if _sys.maxint > 2 ** 32:
                 handle = int(handle)   # long -> int
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._pypy_dll.getaddressindll(name))
+        return self.from_address(dll.__pypy_dll__.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._pypy_dll
+        cdll = self.dll.__pypy_dll__
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -311,7 +311,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
+def PyObject_RichCompareBool(space, w_o1, w_o2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -321,13 +321,13 @@
     opid."""
     # Quick result when objects are the same.
     # Guarantees that identity implies equality.
-    if ref1 is ref2:
+    if space.is_w(w_o1, w_o2):
         opid = rffi.cast(lltype.Signed, opid_int)
         if opid == Py_EQ:
             return 1
         if opid == Py_NE:
             return 0 
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
+    w_res = PyObject_RichCompare(space, w_o1, w_o2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -294,6 +294,23 @@
     def getitems_fixedsize(self, w_list):
         return self.getitems_unroll(w_list)
 
+    def copy_into(self, w_list, w_other):
+        w_other.strategy = self
+        w_other.lstorage = self.getstorage_copy(w_list)
+
+    def clone(self, w_list):
+        storage = self.getstorage_copy(w_list)
+        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
+                                                         self)
+        return w_clone
+
+    def getitems_copy(self, w_list):
+        return self.getitems(w_list) # getitems copies anyway
+
+    def getstorage_copy(self, w_list):
+        lst = self.getitems(w_list)
+        return self.erase(CPyListStorage(w_list.space, lst))
+
     #------------------------------------------
     # all these methods fail or switch strategy and then call ListObjectStrategy's method
 
@@ -301,23 +318,9 @@
         w_list.switch_to_object_strategy()
         w_list.strategy.setslice(w_list, start, stop, step, length)
 
-    def get_sizehint(self):
-        return -1
-
     def init_from_list_w(self, w_list, list_w):
         raise NotImplementedError
 
-    def clone(self, w_list):
-        storage = w_list.lstorage  # lstorage is tuple, no need to clone
-        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
-                                                         self)
-        w_clone.switch_to_object_strategy()
-        return w_clone
-
-    def copy_into(self, w_list, w_other):
-        w_list.switch_to_object_strategy()
-        w_list.strategy.copy_into(w_list, w_other)
-
     def _resize_hint(self, w_list, hint):
         pass
 
@@ -325,13 +328,6 @@
         w_list.switch_to_object_strategy()
         return w_list.strategy.find(w_list, w_item, start, stop)
 
-    def getitems_copy(self, w_list):
-        w_list.switch_to_object_strategy()
-        return w_list.strategy.getitems_copy(w_list)
-
-    def getstorage_copy(self, w_list):
-        raise NotImplementedError
-
     def append(self, w_list, w_item):
         w_list.switch_to_object_strategy()
         w_list.strategy.append(w_list, w_item)
diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -1516,13 +1516,6 @@
     raise NotImplementedError
 
 
- at cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL)
-def PyType_IS_GC(space, o):
-    """Return true if the type object includes support for the cycle detector; this
-    tests the type flag Py_TPFLAGS_HAVE_GC."""
-    raise NotImplementedError
-
-
 @cpython_api([], rffi.INT_real, error=-1)
 def PyUnicode_ClearFreeList(space, ):
     """Clear the free list. Return the total number of freed items."""
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -416,7 +416,7 @@
         Py_buffer passed to it.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -468,7 +468,7 @@
         object.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -514,7 +514,7 @@
         PyBuffer_FillInfo fails if WRITABLE is passed but object is readonly.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -541,7 +541,7 @@
         decremented by PyBuffer_Release.
         """
         module = self.import_extension('foo', [
-                ("release", "METH_VARARGS",
+                ("release", "METH_NOARGS",
                  """
     Py_buffer buf;
     buf.obj = PyBytes_FromString("release me!");
@@ -560,3 +560,20 @@
     Py_RETURN_NONE;
                  """)])
         assert module.release() is None
+
+
+class AppTestPyBuffer_Release(AppTestCpythonExtensionBase):
+    def test_richcomp_nan(self):
+        module = self.import_extension('foo', [
+               ("comp_eq", "METH_VARARGS",
+                """
+                PyObject *a = PyTuple_GetItem(args, 0);
+                PyObject *b = PyTuple_GetItem(args, 1);
+                int res = PyObject_RichCompareBool(a, b, Py_EQ);
+                return PyLong_FromLong(res);  
+                """),])
+        a = float('nan')
+        b = float('nan')
+        assert a is b
+        res = module.comp_eq(a, b)
+        assert res == 1
diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py
--- a/pypy/module/cpyext/test/test_sequence.py
+++ b/pypy/module/cpyext/test/test_sequence.py
@@ -226,6 +226,15 @@
         w_l.inplace_mul(2)
         assert space.int_w(space.len(w_l)) == 10
 
+    def test_getstorage_copy(self, space, api):
+        w = space.wrap
+        w_l = w([1, 2, 3, 4])
+        api.PySequence_Fast(w_l, "foo") # converts
+
+        w_l1 = w([])
+        space.setitem(w_l1, space.newslice(w(0), w(0), w(1)), w_l)
+        assert map(space.unwrap, space.unpackiterable(w_l1)) == [1, 2, 3, 4]
+
 
 class AppTestSequenceObject(AppTestCpythonExtensionBase):
     def test_fast(self):
diff --git a/pypy/module/test_lib_pypy/README.txt b/pypy/module/test_lib_pypy/README.txt
--- a/pypy/module/test_lib_pypy/README.txt
+++ b/pypy/module/test_lib_pypy/README.txt
@@ -1,4 +1,7 @@
 This directory contains app-level tests are supposed to be run *after*
 translation. So you run them by saying:
 
-pypy pytest.py <testfile.py>
+../../goal/pypy-c pytest.py <testfile.py>
+
+Note that if you run it with a PyPy from elsewhere, it will not pick
+up the changes to lib-python and lib_pypy.
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -546,19 +546,24 @@
         space = self.space
         if self.is_heaptype():
             return self.getdictvalue(space, '__module__')
+        elif self.is_cpytype():
+            dot = self.name.rfind('.')
         else:
             dot = self.name.find('.')
-            if dot >= 0:
-                mod = self.name[:dot]
-            else:
-                mod = "builtins"
-            return space.newtext(mod)
+        if dot >= 0:
+            mod = self.name[:dot]
+        else:
+            mod = "builtins"
+        return space.newtext(mod)
 
     def getname(self, space):
         if self.is_heaptype():
             result = self.name
         else:
-            dot = self.name.find('.')
+            if self.is_cpytype():
+                dot = self.name.rfind('.')
+            else:
+                dot = self.name.find('.')
             if dot >= 0:
                 result = self.name[dot+1:]
             else:
diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -552,10 +552,11 @@
             self.reg_bindings[result_v] = loc
             return loc
         if v not in self.reg_bindings:
+            # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(v, forbidden_vars)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-        assert v in self.reg_bindings
+            return loc
         if self.longevity[v][1] > self.position:
             # we need to find a new place for variable v and
             # store result in the same place
diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -504,7 +504,7 @@
         clt.frame_info = rffi.cast(jitframe.JITFRAMEINFOPTR, frame_info)
         clt.frame_info.clear() # for now
 
-        if log:
+        if log or self._debug:
             number = looptoken.number
             operations = self._inject_debugging_code(looptoken, operations,
                                                      'e', number)
@@ -589,7 +589,7 @@
                 faildescr.adr_jump_offset)
         self.mc.force_frame_size(DEFAULT_FRAME_BYTES)
         descr_number = compute_unique_id(faildescr)
-        if log:
+        if log or self._debug:
             operations = self._inject_debugging_code(faildescr, operations,
                                                      'b', descr_number)
         arglocs = self.rebuild_faillocs_from_descr(faildescr, inputargs)
@@ -1618,18 +1618,6 @@
         else:
             not_implemented("save_into_mem size = %d" % size)
 
-    def _genop_getfield(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc, sign_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
-        source_addr = AddressLoc(base_loc, ofs_loc)
-        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
-
-    genop_getfield_gc_i = _genop_getfield
-    genop_getfield_gc_r = _genop_getfield
-    genop_getfield_gc_f = _genop_getfield
-    genop_getfield_raw_i = _genop_getfield
-    genop_getfield_raw_f = _genop_getfield
-
     def _genop_gc_load(self, op, arglocs, resloc):
         base_loc, ofs_loc, size_loc, sign_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -1305,7 +1305,7 @@
         self.rm.possibly_free_var(tmpbox_high)
 
     def compute_hint_frame_locations(self, operations):
-        # optimization only: fill in the 'hint_frame_locations' dictionary
+        # optimization only: fill in the 'hint_frame_pos' dictionary
         # of 'fm' based on the JUMP at the end of the loop, by looking
         # at where we would like the boxes to be after the jump.
         op = operations[-1]
@@ -1320,7 +1320,7 @@
             self._compute_hint_frame_locations_from_descr(descr)
         #else:
         #   The loop ends in a JUMP going back to a LABEL in the same loop.
-        #   We cannot fill 'hint_frame_locations' immediately, but we can
+        #   We cannot fill 'hint_frame_pos' immediately, but we can
         #   wait until the corresponding consider_label() to know where the
         #   we would like the boxes to be after the jump.
 
diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py
--- a/rpython/rlib/rposix.py
+++ b/rpython/rlib/rposix.py
@@ -205,6 +205,18 @@
         if not is_valid_fd(fd):
             from errno import EBADF
             raise OSError(EBADF, 'Bad file descriptor')
+
+    def _bound_for_write(fd, count):
+        if count > 32767 and c_isatty(fd):
+            # CPython Issue #11395, PyPy Issue #2636: the Windows console
+            # returns an error (12: not enough space error) on writing into
+            # stdout if stdout mode is binary and the length is greater than
+            # 66,000 bytes (or less, depending on heap usage).  Can't easily
+            # test that, because we need 'fd' to be non-redirected...
+            count = 32767
+        elif count > 0x7fffffff:
+            count = 0x7fffffff
+        return count
 else:
     def is_valid_fd(fd):
         return 1
@@ -213,6 +225,9 @@
     def validate_fd(fd):
         pass
 
+    def _bound_for_write(fd, count):
+        return count
+
 def closerange(fd_low, fd_high):
     # this behaves like os.closerange() from Python 2.6.
     for fd in xrange(fd_low, fd_high):
@@ -449,6 +464,7 @@
 def write(fd, data):
     count = len(data)
     validate_fd(fd)
+    count = _bound_for_write(fd, count)
     with rffi.scoped_nonmovingbuffer(data) as buf:
         return handle_posix_error('write', c_write(fd, buf, count))
 
diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -710,7 +710,8 @@
         size, _ = expected_size_and_sign
         return lltype.FixedSizeArray(fieldtype.OF, size/_sizeof(fieldtype.OF))
     raise TypeError("conflict between translating python and compiler field"
-                    " type %r for %r" % (fieldtype, fieldname))
+                    " type %r for symbol %r, expected size+sign %r" % (
+                        fieldtype, fieldname, expected_size_and_sign))
 
 def expose_value_as_rpython(value):
     if intmask(value) == value:

From pypy.commits at gmail.com  Wed Aug 23 13:00:28 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 10:00:28 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Manual merge of 141ba627dc5f+18b10a6743c4
Message-ID: <599db4ac.84a3df0a.55d2e.5b0c@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92236:9b4bd629bb44
Date: 2017-08-23 18:59 +0200
http://bitbucket.org/pypy/pypy/changeset/9b4bd629bb44/

Log:	Manual merge of 141ba627dc5f+18b10a6743c4

diff --git a/lib-python/3/ctypes/__init__.py b/lib-python/3/ctypes/__init__.py
--- a/lib-python/3/ctypes/__init__.py
+++ b/lib-python/3/ctypes/__init__.py
@@ -346,16 +346,18 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxsize * 2 + 1))
+        return "<%s '%s', handle %x at 0x%x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxsize*2 + 1)),
+                id(self) & (_sys.maxsize*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):

From pypy.commits at gmail.com  Wed Aug 23 13:05:08 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 10:05:08 -0700 (PDT)
Subject: [pypy-commit] pypy default: (fijal, arigo)
Message-ID: <599db5c4.12badf0a.92cf.057b@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92237:8cff23e10359
Date: 2017-08-23 19:04 +0200
http://bitbucket.org/pypy/pypy/changeset/8cff23e10359/

Log:	(fijal, arigo)

	Turn functions that do nothing into macros that do the same

diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h
--- a/pypy/module/cpyext/include/object.h
+++ b/pypy/module/cpyext/include/object.h
@@ -277,6 +277,15 @@
     char dummy;
 } PyGC_Head;
 
+/* dummy GC macros */
+#define _PyGC_FINALIZED(o) 1
+#define PyType_IS_GC(tp) 1
+
+#define PyObject_GC_Track(o)      do { } while(0)
+#define PyObject_GC_UnTrack(o)    do { } while(0)
+#define _PyObject_GC_TRACK(o)     do { } while(0)
+#define _PyObject_GC_UNTRACK(o)   do { } while(0)
+
 /* Utility macro to help write tp_traverse functions.
  * To use this macro, the tp_traverse function must name its arguments
  * "visit" and "arg".  This is intended to keep tp_traverse functions
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -76,31 +76,6 @@
 def PyObject_GC_Del(space, obj):
     PyObject_Free(space, obj)
 
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_Track(space, op):
-    """Adds the object op to the set of container objects tracked by the
-    collector.  The collector can run at unexpected times so objects must be
-    valid while being tracked.  This should be called once all the fields
-    followed by the tp_traverse handler become valid, usually near the
-    end of the constructor."""
-    pass
-
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_UnTrack(space, op):
-    """Remove the object op from the set of container objects tracked by the
-    collector.  Note that PyObject_GC_Track() can be called again on
-    this object to add it back to the set of tracked objects.  The deallocator
-    (tp_dealloc handler) should call this for the object before any of
-    the fields used by the tp_traverse handler become invalid."""
-    pass
-
- at cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL)
-def PyType_IS_GC(space, o):
-    """Return true if the type object includes support for the cycle detector; this
-    tests the type flag Py_TPFLAGS_HAVE_GC.
-    """
-    return False
-
 @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL)
 def _PyObject_GetDictPtr(space, op):
     return lltype.nullptr(PyObjectP.TO)
diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -655,18 +655,6 @@
     require changes in your code for properly supporting 64-bit systems."""
     raise NotImplementedError
 
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_TRACK(space, op):
-    """A macro version of PyObject_GC_Track().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_UNTRACK(space, op):
-    """A macro version of PyObject_GC_UnTrack().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
 @cpython_api([PyFrameObject], PyObject)
 def PyGen_New(space, frame):
     """Create and return a new generator object based on the frame object. A

From pypy.commits at gmail.com  Wed Aug 23 13:06:36 2017
From: pypy.commits at gmail.com (arigo)
Date: Wed, 23 Aug 2017 10:06:36 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: hg merge default
Message-ID: <599db61c.8d6d1c0a.dc421.1e22@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: py3.5
Changeset: r92238:89b8fc097575
Date: 2017-08-23 19:05 +0200
http://bitbucket.org/pypy/pypy/changeset/89b8fc097575/

Log:	hg merge default

diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h
--- a/pypy/module/cpyext/include/object.h
+++ b/pypy/module/cpyext/include/object.h
@@ -269,6 +269,11 @@
 #define _PyGC_FINALIZED(o) 1
 #define PyType_IS_GC(tp) 1
 
+#define PyObject_GC_Track(o)      do { } while(0)
+#define PyObject_GC_UnTrack(o)    do { } while(0)
+#define _PyObject_GC_TRACK(o)     do { } while(0)
+#define _PyObject_GC_UNTRACK(o)   do { } while(0)
+
 /* Utility macro to help write tp_traverse functions.
  * To use this macro, the tp_traverse function must name its arguments
  * "visit" and "arg".  This is intended to keep tp_traverse functions
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -80,24 +80,6 @@
 def PyObject_GC_Del(space, obj):
     PyObject_Free(space, obj)
 
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_Track(space, op):
-    """Adds the object op to the set of container objects tracked by the
-    collector.  The collector can run at unexpected times so objects must be
-    valid while being tracked.  This should be called once all the fields
-    followed by the tp_traverse handler become valid, usually near the
-    end of the constructor."""
-    pass
-
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_UnTrack(space, op):
-    """Remove the object op from the set of container objects tracked by the
-    collector.  Note that PyObject_GC_Track() can be called again on
-    this object to add it back to the set of tracked objects.  The deallocator
-    (tp_dealloc handler) should call this for the object before any of
-    the fields used by the tp_traverse handler become invalid."""
-    pass
-
 @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL)
 def _PyObject_GetDictPtr(space, op):
     return lltype.nullptr(PyObjectP.TO)
diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -625,18 +625,6 @@
     resized object or NULL on failure."""
     raise NotImplementedError
 
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_TRACK(space, op):
-    """A macro version of PyObject_GC_Track().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_UNTRACK(space, op):
-    """A macro version of PyObject_GC_UnTrack().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
 @cpython_api([PyFrameObject], PyObject)
 def PyGen_New(space, frame):
     """Create and return a new generator object based on the frame object. A

From pypy.commits at gmail.com  Wed Aug 23 15:54:41 2017
From: pypy.commits at gmail.com (mattip)
Date: Wed, 23 Aug 2017 12:54:41 -0700 (PDT)
Subject: [pypy-commit] pypy default: test, fix for missing userslot tp_iter,
 tp_iternext, this time via PyObject_Call
Message-ID: <599ddd81.85921c0a.d803d.14f3@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92239:eb3baada82b7
Date: 2017-08-23 21:28 +0300
http://bitbucket.org/pypy/pypy/changeset/eb3baada82b7/

Log:	test, fix for missing userslot tp_iter, tp_iternext, this time via
	PyObject_Call

diff --git a/pypy/module/cpyext/test/test_eval.py b/pypy/module/cpyext/test/test_eval.py
--- a/pypy/module/cpyext/test/test_eval.py
+++ b/pypy/module/cpyext/test/test_eval.py
@@ -362,3 +362,56 @@
             assert 'while calling recurse' in str(e)
         else:
             assert False, "expected RuntimeError"
+
+    def test_build_class(self):
+            # make sure PyObject_Call generates a proper PyTypeObject,
+            # along the way verify that userslot has iter and next
+            module = self.import_extension('foo', [
+                ("object_call", "METH_O",
+                 '''
+                    return PyObject_Call((PyObject*)&PyType_Type, args, NULL);
+                 '''),
+                ('iter', "METH_O",
+                 '''
+                    if (NULL == args->ob_type->tp_iter)
+                    {
+                        PyErr_SetString(PyExc_TypeError, "NULL tp_iter");
+                        return NULL;
+                    }
+                    return args->ob_type->tp_iter(args);
+                 '''),
+                ('next', "METH_O",
+                 '''
+                    if (NULL == args->ob_type->tp_iternext)
+                    {
+                        PyErr_SetString(PyExc_TypeError, "NULL tp_iternext");
+                        return NULL;
+                    }
+                    return args->ob_type->tp_iternext(args);
+                 '''),])
+            def __init__(self, N):
+                self.N = N
+                self.i = 0
+
+            def __iter__(self):
+                return self
+
+            def __next__(self):
+                if self.i < self.N:
+                    i = self.i
+                    self.i += 1
+                    return i
+                raise StopIteration
+
+            d = {'__init__': __init__, '__iter__': __iter__, 'next': __next__,
+                 '__next__': next}
+            C = module.object_call(('Iterable', (object,), d))
+            c = C(5)
+            i = module.iter(c)
+            out = []
+            try:
+                while 1:
+                    out.append(module.next(i))
+            except StopIteration:
+                pass
+            assert out == [0, 1, 2, 3, 4]
diff --git a/pypy/module/cpyext/userslot.py b/pypy/module/cpyext/userslot.py
--- a/pypy/module/cpyext/userslot.py
+++ b/pypy/module/cpyext/userslot.py
@@ -122,3 +122,11 @@
     else:
         space.delete(w_self, w_obj)
     return 0
+
+ at slot_function([PyObject], PyObject)
+def slot_tp_iter(space, w_self):
+    return space.iter(w_self)
+
+ at slot_function([PyObject], PyObject)
+def slot_tp_iternext(space, w_self):
+    return space.next(w_self)

From pypy.commits at gmail.com  Thu Aug 24 04:49:03 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 01:49:03 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Backed out changeset 3e38274ddd35
Message-ID: <599e92ff.491a1c0a.ebdea.2740@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92240:7655999bf67f
Date: 2017-08-24 10:48 +0200
http://bitbucket.org/pypy/pypy/changeset/7655999bf67f/

Log:	Backed out changeset 3e38274ddd35

	(fijal)

diff --git a/pypy/module/_pypyjson/interp_decoder.py b/pypy/module/_pypyjson/interp_decoder.py
--- a/pypy/module/_pypyjson/interp_decoder.py
+++ b/pypy/module/_pypyjson/interp_decoder.py
@@ -295,7 +295,7 @@
             i += 1
             bits |= ord(ch)
             if ch == '"':
-                if 1 or bits & 0x80:
+                if bits & 0x80:
                     # the 8th bit is set, it's an utf8 strnig
                     content_utf8 = self.getslice(start, i-1)
                     content_unicode = unicodehelper.decode_utf8(self.space, content_utf8)
diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -20,7 +20,6 @@
 from pypy.objspace.std.formatting import mod_format
 from pypy.objspace.std.stringmethods import StringMethods
 from pypy.objspace.std.util import IDTAG_SPECIAL, IDTAG_SHIFT
-from pypy.objspace.std.sliceobject import unwrap_start_stop
 
 __all__ = ['W_UnicodeObject', 'wrapunicode', 'plain_str2unicode',
            'encode_object', 'decode_object', 'unicode_from_object',
@@ -76,13 +75,6 @@
             uid = (base << IDTAG_SHIFT) | IDTAG_SPECIAL
         return space.newint(uid)
 
-    def _convert_idx_params_unicode(self, space, w_start, w_end):
-        """ Specialcase this for unicode - one less element in the tuple
-        """
-        lenself = self._len()
-        start, end = unwrap_start_stop(space, lenself, w_start, w_end)
-        return start, end
-
     def str_w(self, space):
         return space.text_w(space.str(self))
 
@@ -133,8 +125,8 @@
         return rutf8.compute_length_utf8(self._utf8)
 
     def _val(self, space):
-        import pdb
-        pdb.set_trace()
+        #import pdb
+        #pdb.set_trace()
         return self._utf8.decode('utf8')
 
     @staticmethod
@@ -454,6 +446,9 @@
             i = rutf8.next_codepoint_pos(val, i)
         return space.newbool(cased)
 
+    def _starts_ends_overflow(self, prefix):
+        return len(prefix) == 0
+
     def descr_add(self, space, w_other):
         try:
             w_other = self.convert_arg_to_w_unicode(space, w_other)
@@ -727,26 +722,6 @@
         assert rpos >= lpos    # annotator hint, don't remove
         return self._utf8_sliced(lpos, rpos, lgt)
 
-    def descr_startswith(self, space, w_prefix, w_start=None, w_end=None):
-        (start, end) = self._convert_idx_params_unicode(space, w_start, w_end)
-        if space.isinstance_w(w_prefix, space.w_tuple):
-            return self._startswith_tuple(space, w_prefix, start, end)
-        return space.newbool(self._startswith(space, w_prefix, start, end))
-
-    def _startswith_tuple(self, space, w_prefix, start, end):
-        for w_prefix in space.fixedview(w_prefix):
-            if self._startswith(space, w_prefix, start, end):
-                return space.w_True
-        return space.w_False
-
-    def _startswith(self, space, w_prefix, start, end):
-        prefix = self.convert_arg_to_w_unicode(space, w_prefix)._utf8
-        if start > self._len():
-            return len(prefix) == 0 # bug-to-bug cpython compatibility
-        xxx
-        return startswith(self._utf8, prefix, start, end)
-
-
     def descr_getnewargs(self, space):
         return space.newtuple([W_UnicodeObject(self._utf8, self._length)])
 

From pypy.commits at gmail.com  Thu Aug 24 05:11:12 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 02:11:12 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599e9830.a799df0a.9ed31.3fbd@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92241:41e0c8d46641
Date: 2017-08-24 11:10 +0200
http://bitbucket.org/pypy/pypy/changeset/41e0c8d46641/

Log:	(fijal, arigo)

	Fix

diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py
--- a/pypy/interpreter/unicodehelper.py
+++ b/pypy/interpreter/unicodehelper.py
@@ -44,7 +44,8 @@
         string, len(string), "strict",
         final=True, errorhandler=DecodeWrapper(decode_error_handler(space)).handle,
         unicodedata_handler=unicodedata_handler)
-    return result_u.encode('utf8'), len(result_u)
+    # XXX argh.  we want each surrogate to be encoded separately
+    return ''.join([u.encode('utf8') for u in result_u]), len(result_u)
 
 def decode_raw_unicode_escape(space, string):
     # XXX pick better length, maybe
@@ -52,7 +53,8 @@
     result_u, consumed = runicode.str_decode_raw_unicode_escape(
         string, len(string), "strict",
         final=True, errorhandler=DecodeWrapper(decode_error_handler(space)).handle)
-    return result_u.encode('utf8'), len(result_u)
+    # XXX argh.  we want each surrogate to be encoded separately
+    return ''.join([u.encode('utf8') for u in result_u]), len(result_u)
 
 def check_utf8(space, string):
     # Surrogates are accepted and not treated specially at all.

From pypy.commits at gmail.com  Thu Aug 24 05:44:54 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 02:44:54 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: hg merge default
Message-ID: <599ea016.05371c0a.c2fa0.a95e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92242:367afaf4ad3a
Date: 2017-08-24 11:43 +0200
http://bitbucket.org/pypy/pypy/changeset/367afaf4ad3a/

Log:	hg merge default

	Manual merges may go wrong

diff too long, truncating to 2000 out of 88216 lines

diff --git a/.hgignore b/.hgignore
--- a/.hgignore
+++ b/.hgignore
@@ -1,6 +1,6 @@
 syntax: glob
 *.py[co]
-*.sw[po]
+*.sw[pon]
 *~
 .*.swp
 .idea
@@ -8,6 +8,8 @@
 .pydevproject
 __pycache__
 
+.cache/
+.gdb_history
 syntax: regexp
 ^testresult$
 ^site-packages$
@@ -23,16 +25,17 @@
 ^pypy/module/cpyext/test/.+\.manifest$
 ^pypy/module/test_lib_pypy/ctypes_tests/.+\.o$
 ^pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test\.o$
-^pypy/module/cppyy/src/.+\.o$
-^pypy/module/cppyy/bench/.+\.so$
-^pypy/module/cppyy/bench/.+\.root$
-^pypy/module/cppyy/bench/.+\.d$
-^pypy/module/cppyy/src/.+\.errors$
-^pypy/module/cppyy/test/.+_rflx\.cpp$
-^pypy/module/cppyy/test/.+\.so$
-^pypy/module/cppyy/test/.+\.rootmap$
-^pypy/module/cppyy/test/.+\.exe$
-^pypy/module/cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/src/.+\.o$
+^pypy/module/_cppyy/bench/.+\.so$
+^pypy/module/_cppyy/bench/.+\.root$
+^pypy/module/_cppyy/bench/.+\.d$
+^pypy/module/_cppyy/src/.+\.errors$
+^pypy/module/_cppyy/test/.+_rflx\.cpp$
+^pypy/module/_cppyy/test/.+\.so$
+^pypy/module/_cppyy/test/.+\.rootmap$
+^pypy/module/_cppyy/test/.+\.exe$
+^pypy/module/_cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/.+/*\.pcm$
 ^pypy/module/test_lib_pypy/cffi_tests/__pycache__.+$
 ^pypy/doc/.+\.html$
 ^pypy/doc/config/.+\.rst$
@@ -49,6 +52,11 @@
 ^rpython/translator/goal/target.+-c$
 ^rpython/translator/goal/.+\.exe$
 ^rpython/translator/goal/.+\.dll$
+^rpython/rlib/rvmprof/src/shared/libbacktrace/Makefile$
+^rpython/rlib/rvmprof/src/shared/libbacktrace/config.guess$
+^rpython/rlib/rvmprof/src/shared/libbacktrace/config.h$
+^rpython/rlib/rvmprof/src/shared/libbacktrace/config.log$
+^rpython/rlib/rvmprof/src/shared/libbacktrace/config.status$
 ^pypy/goal/pypy-translation-snapshot$
 ^pypy/goal/pypy-c
 ^pypy/goal/.+\.exe$
@@ -60,6 +68,9 @@
 ^lib_pypy/ctypes_config_cache/_.+_cache\.py$
 ^lib_pypy/ctypes_config_cache/_.+_.+_\.py$
 ^lib_pypy/_libmpdec/.+.o$
+^lib_pypy/.+.c$
+^lib_pypy/.+.o$
+^lib_pypy/.+.so$
 ^pypy/doc/discussion/.+\.html$
 ^include/.+\.h$
 ^include/.+\.inl$
@@ -74,8 +85,7 @@
 ^rpython/doc/_build/.*$
 ^compiled
 ^.git/
-^.hypothesis/
+.hypothesis/
 ^release/
 ^rpython/_cache$
 
-pypy/module/cppyy/.+/*\.pcm
diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -34,3 +34,9 @@
 050d84dd78997f021acf0e133934275d63547cc0 release-pypy2.7-v5.4.1
 0e2d9a73f5a1818d0245d75daccdbe21b2d5c3ef release-pypy2.7-v5.4.1
 aff251e543859ce4508159dd9f1a82a2f553de00 release-pypy2.7-v5.6.0
+fa3249d55d15b9829e1be69cdf45b5a44cec902d release-pypy2.7-v5.7.0
+b16a4363e930f6401bceb499b9520955504c6cb0 release-pypy3.5-v5.7.0
+1aa2d8e03cdfab54b7121e93fda7e98ea88a30bf release-pypy2.7-v5.7.1
+2875f328eae2216a87f3d6f335092832eb031f56 release-pypy3.5-v5.7.1
+c925e73810367cd960a32592dd7f728f436c125c release-pypy2.7-v5.8.0
+a37ecfe5f142bc971a86d17305cc5d1d70abec64 release-pypy3.5-v5.8.0
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -1,3 +1,5 @@
+#encoding utf-8
+
 License
 =======
 
@@ -37,14 +39,14 @@
 
   Armin Rigo
   Maciej Fijalkowski
-  Carl Friedrich Bolz
+  Carl Friedrich Bolz-Tereick
   Amaury Forgeot d'Arc
   Antonio Cuni
+  Matti Picus
   Samuele Pedroni
-  Matti Picus
+  Ronan Lamy
   Alex Gaynor
   Philip Jenvey
-  Ronan Lamy
   Brian Kearns
   Richard Plangger
   Michael Hudson
@@ -55,12 +57,12 @@
   Hakan Ardo
   Benjamin Peterson
   Anders Chrigstrom
+  Wim Lavrijsen
   Eric van Riet Paap
-  Wim Lavrijsen
   Richard Emslie
   Alexander Schremmer
+  Remi Meier
   Dan Villiom Podlaski Christiansen
-  Remi Meier
   Lukas Diekmann
   Sven Hager
   Anders Lehmann
@@ -83,8 +85,8 @@
   Lawrence Oluyede
   Bartosz Skowron
   Daniel Roberts
+  Adrien Di Mascio
   Niko Matsakis
-  Adrien Di Mascio
   Alexander Hesse
   Ludovic Aubry
   Jacob Hallen
@@ -99,278 +101,288 @@
   Vincent Legoll
   Michael Foord
   Stephan Diehl
+  Stefano Rivera
   Stefan Schwarzer
+  Tomek Meka
   Valentino Volonghi
-  Tomek Meka
-  Stefano Rivera
   Patrick Maupin
   Devin Jeanpierre
   Bob Ippolito
   Bruno Gola
   David Malcolm
   Jean-Paul Calderone
+  Squeaky
+  Edd Barrett
   Timo Paulssen
-  Edd Barrett
-  Squeaky
   Marius Gedminas
   Alexandre Fayolle
   Simon Burton
+  Nicolas Truessel
   Martin Matusiak
-  Nicolas Truessel
+  Laurence Tratt
+  Wenzhu Man
   Konstantin Lopuhin
-  Wenzhu Man
   John Witulski
-  Laurence Tratt
+  Greg Price
   Ivan Sichmann Freitas
-  Greg Price
   Dario Bertini
+  Jeremy Thurgood
   Mark Pearse
   Simon Cross
-  Jeremy Thurgood
+  Tobias Pape
   Andreas St&#252;hrk
-  Tobias Pape
   Jean-Philippe St. Pierre
   Guido van Rossum
   Pavel Vinogradov
   Pawe&#322; Piotr Przeradowski
+  William Leslie
+  marky1991
+  Ilya Osadchiy
+  Tobias Oberstein
   Paul deGrandis
-  Ilya Osadchiy
-  marky1991
-  Tobias Oberstein
+  Boris Feigin
+  Taavi Burns
   Adrian Kuhn
-  Boris Feigin
   tav
-  Taavi Burns
   Georg Brandl
   Bert Freudenberg
   Stian Andreassen
   Wanja Saatkamp
+  Mike Blume
+  Joannah Nanjekye
   Gerald Klix
-  Mike Blume
   Oscar Nierstrasz
+  Rami Chowdhury
   Stefan H. Muller
-  Rami Chowdhury
+  Tim Felgentreff
   Eugene Oden
+  Jeff Terrace
   Henry Mason
   Vasily Kuznetsov
   Preston Timmons
   David Ripton
-  Jeff Terrace
-  Tim Felgentreff
   Dusty Phillips
   Lukas Renggli
   Guenter Jantzen
-  William Leslie
+  Jasper Schulz
   Ned Batchelder
+  Amit Regmi
   Anton Gulenko
-  Amit Regmi
-  Ben Young
-  Jasper Schulz
+  Sergey Matyunin
+  Andrew Chambers
   Nicolas Chauvat
   Andrew Durdin
-  Andrew Chambers
-  Sergey Matyunin
+  Ben Young
   Michael Schneider
   Nicholas Riley
   Jason Chu
   Igor Trindade Oliveira
   Yichao Yu
+  Michael Twomey
   Rocco Moretti
   Gintautas Miliauskas
-  Michael Twomey
   Lucian Branescu Mihaila
   anatoly techtonik
+  Dodan Mihai
+  Karl Bartel
   Gabriel Lavoie
+  Jared Grubb
   Olivier Dormond
-  Jared Grubb
-  Karl Bartel
   Wouter van Heyst
+  Sebastian Pawlu&#347;
   Brian Dorsey
   Victor Stinner
   Andrews Medina
-  Sebastian Pawlu&#347;
-  Stuart Williams
-  Daniel Patrick
   Aaron Iles
   Toby Watson
+  Daniel Patrick
+  Stuart Williams
   Antoine Pitrou
   Christian Hudon
+  Justas Sadzevicius
+  Neil Shepperd
   Michael Cheng
-  Justas Sadzevicius
+  Mikael Sch&#246;nenberg
+  Stanislaw Halik
+  Berkin Ilbeyi
   Gasper Zejn
-  Neil Shepperd
-  Stanislaw Halik
-  Mikael Sch&#246;nenberg
-  Berkin Ilbeyi
   Faye Zhao
   Elmo M&#228;ntynen
-  Jonathan David Riehl
   Anders Qvist
   Corbin Simpson
   Chirag Jadwani
+  Jonathan David Riehl
   Beatrice During
   Alex Perry
+  p_zieschang at yahoo.de
+  Robert Zaremba
+  Alan McIntyre
+  Alexander Sedov
   Vaibhav Sood
-  Alan McIntyre
   Reuben Cummings
-  Alexander Sedov
-  p_zieschang at yahoo.de
   Attila Gobi
+  Alecsandru Patrascu
   Christopher Pope
-  Aaron Gallagher
+  Tristan Arthur
+  Christian Tismer 
+  Dan Stromberg
+  Carl Meyer
   Florin Papa
-  Christian Tismer 
-  Marc Abramowitz
-  Dan Stromberg
-  Arjun Naik
+  Jens-Uwe Mager
   Valentina Mukhamedzhanova
   Stefano Parmesan
   touilleMan
+  Marc Abramowitz
+  Arjun Naik
+  Aaron Gallagher
   Alexis Daboville
-  Jens-Uwe Mager
-  Carl Meyer
+  Pieter Zieschang
   Karl Ramm
-  Pieter Zieschang
-  Gabriel
   Lukas Vacek
-  Kunal Grover
-  Andrew Dalke
+  Omer Katz
+  Jacek Generowicz
   Sylvain Thenault
   Jakub Stasiak
+  Stefan Beyer
+  Andrew Dalke
+  Alejandro J. Cura
+  Vladimir Kryachko
+  Gabriel
+  Mark Williams
+  Kunal Grover
   Nathan Taylor
-  Vladimir Kryachko
-  Omer Katz
-  Mark Williams
-  Jacek Generowicz
-  Alejandro J. Cura
+  Travis Francis Athougies
+  Yasir Suhail
+  Sergey Kishchenko
+  Martin Blais
+  Lutz Paelike
+  Ian Foote
+  Philipp Rustemeuer
+  Catalin Gabriel Manciu
   Jacob Oscarson
-  Travis Francis Athougies
   Ryan Gonzalez
-  Ian Foote
   Kristjan Valur Jonsson
+  Lucio Torre
+  Richard Lancaster
+  Dan Buch
+  Lene Wagner
+  Tomo Cocoa
   David Lievens
   Neil Blakey-Milner
-  Lutz Paelike
-  Lucio Torre
+  Henrik Vendelbo
   Lars Wassermann
-  Philipp Rustemeuer
-  Henrik Vendelbo
-  Richard Lancaster
-  Yasir Suhail
-  Dan Buch
+  Ignas Mikalajunas
+  Christoph Gerum
   Miguel de Val Borro
   Artur Lisiecki
-  Sergey Kishchenko
-  Ignas Mikalajunas
-  Alecsandru Patrascu
-  Christoph Gerum
-  Martin Blais
-  Lene Wagner
-  Catalin Gabriel Manciu
-  Tomo Cocoa
-  Kim Jin Su
-  rafalgalczynski at gmail.com
   Toni Mattis
-  Amber Brown
+  Laurens Van Houtven
+  Bobby Impollonia
+  Roberto De Ioris
+  Jeong YunWon
+  Christopher Armstrong
+  Aaron Tubbs
+  Vasantha Ganesh K
+  Jason Michalski
+  Markus Holtermann
+  Andrew Thompson
+  Yusei Tahara
+  Ruochen Huang
+  Fabio Niephaus
+  Akira Li
+  Gustavo Niemeyer
+  Rafa&#322; Ga&#322;czy&#324;ski
+  Logan Chien
   Lucas Stadler
-  Julian Berman
-  Markus Holtermann
   roberto at goyle
+  Matt Bogosian
   Yury V. Zaytsev
-  Anna Katrina Dominguez
-  Bobby Impollonia
-  Vasantha Ganesh K
-  Andrew Thompson
   florinpapa
-  Yusei Tahara
-  Aaron Tubbs
-  Ben Darnell
-  Roberto De Ioris
-  Logan Chien
-  Juan Francisco Cantero Hurtado
-  Ruochen Huang
-  Jeong YunWon
-  Godefroid Chappelle
-  Joshua Gilbert
-  Dan Colish
-  Christopher Armstrong
-  Michael Hudson-Doyle
   Anders Sigfridsson
   Nikolay Zinov
-  Jason Michalski
+  rafalgalczynski at gmail.com
+  Joshua Gilbert
+  Anna Katrina Dominguez
+  Kim Jin Su
+  Amber Brown
+  Nate Bragg
+  Ben Darnell
+  Juan Francisco Cantero Hurtado
+  Godefroid Chappelle
+  Julian Berman
+  Michael Hudson-Doyle
   Floris Bruynooghe
-  Laurens Van Houtven
-  Akira Li
-  Gustavo Niemeyer
   Stephan Busemann
-  Rafa&#322; Ga&#322;czy&#324;ski
-  Matt Bogosian
+  Dan Colish
   timo
-  Christian Muirhead
-  Berker Peksag
-  James Lan
   Volodymyr Vladymyrov
-  shoma hosaka
-  Ben Mather
-  Niclas Olofsson
-  Matthew Miller
-  Rodrigo Ara&#250;jo
+  Daniel Neuh&#228;user
+  Flavio Percoco
   halgari
-  Boglarka Vezer
-  Chris Pressey
-  Buck Golemon
-  Diana Popa
-  Konrad Delong
-  Dinu Gherman
+  Jim Baker
   Chris Lambacher
   coolbutuseless at gmail.com
+  Mike Bayer
+  Rodrigo Ara&#250;jo
   Daniil Yarancev
-  Jim Baker
+  OlivierBlanvillain
+  Jonas Pfannschmidt
+  Zearin
+  Andrey Churin
   Dan Crosta
-  Nikolaos-Digenis Karagiannis
-  James Robert
-  Armin Ronacher
-  Brett Cannon
-  Donald Stufft
-  yrttyr
-  aliceinwire
-  OlivierBlanvillain
-  Dan Sanders
-  Zooko Wilcox-O Hearn
+  reubano at gmail.com
+  Julien Phalip
+  Roman Podoliaka
+  Eli Stevens
+  Boglarka Vezer
+  PavloKapyshin
   Tomer Chachamu
   Christopher Groskopf
   Asmo Soinio
-  jiaaro
-  Mads Kiilerich
   Antony Lee
-  Jason Madden
-  Daniel Neuh&#65533;user
-  reubano at gmail.com
-  Yaroslav Fedevych
   Jim Hunziker
-  Markus Unterwaditzer
-  Even Wiik Thomassen
-  jbs
-  squeaky
-  soareschen
-  Jonas Pfannschmidt
-  Kurt Griffiths
-  Mike Bayer
-  Stefan Marr
-  Flavio Percoco
-  Kristoffer Kleine
+  shoma hosaka
+  Buck Golemon
+  Iraklis D.
+  JohnDoe
+  yrttyr
   Michael Chermside
   Anna Ravencroft
+  remarkablerocket
+  Petre Vijiac
+  Berker Peksag
+  Christian Muirhead
+  soareschen
+  Matthew Miller
+  Konrad Delong
+  Dinu Gherman
   pizi
-  remarkablerocket
-  Andrey Churin
-  Zearin
-  Eli Stevens
-  Tobias Diaz
-  Julien Phalip
-  Roman Podoliaka
+  James Robert
+  Armin Ronacher
+  Diana Popa
+  Mads Kiilerich
+  Brett Cannon
+  aliceinwire
+  Zooko Wilcox-O Hearn
+  James Lan
+  jiaaro
+  Markus Unterwaditzer
+  Kristoffer Kleine
+  Graham Markall
   Dan Loewenherz
   werat
+  Niclas Olofsson
+  Chris Pressey
+  Tobias Diaz
+  Nikolaos-Digenis Karagiannis
+  Kurt Griffiths
+  Ben Mather
+  Donald Stufft
+  Dan Sanders
+  Jason Madden
+  Yaroslav Fedevych
+  Even Wiik Thomassen
+  Stefan Marr
 
   Heinrich-Heine University, Germany 
   Open End AB (formerly AB Strakt), Sweden
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/README.rst b/README.rst
--- a/README.rst
+++ b/README.rst
@@ -27,14 +27,19 @@
 Building
 ========
 
-build with:
+First switch to or download the correct branch.  The basic choices are
+``default`` for Python 2.7 and, for Python 3.X, the corresponding py3.X
+branch (e.g. ``py3.5``).
+
+Build with:
 
 .. code-block:: console
 
     $ rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
 
-This ends up with ``pypy-c`` binary in the main pypy directory. We suggest
-to use virtualenv with the resulting pypy-c as the interpreter; you can
-find more details about various installation schemes here:
+This ends up with a ``pypy-c`` or ``pypy3-c`` binary in the main pypy
+directory.  We suggest to use virtualenv with the resulting
+pypy-c/pypy3-c as the interpreter; you can find more details about
+various installation schemes here:
 
     http://doc.pypy.org/en/latest/install.html
diff --git a/extra_tests/README.txt b/extra_tests/README.txt
new file mode 100644
--- /dev/null
+++ b/extra_tests/README.txt
@@ -0,0 +1,5 @@
+The tests in this directory are a complement to lib-python/3/test/.
+
+They are meant to run on top of a compiled pypy3 or CPython3.5 in an
+environment containing at least pytest and hypothesis, using a command like
+'pytest extra_tests/'.
diff --git a/extra_tests/pytest.ini b/extra_tests/pytest.ini
new file mode 100644
diff --git a/extra_tests/test_unicode.py b/extra_tests/test_unicode.py
new file mode 100644
--- /dev/null
+++ b/extra_tests/test_unicode.py
@@ -0,0 +1,34 @@
+import pytest
+from hypothesis import strategies as st
+from hypothesis import given, settings, example
+
+from unicodedata import normalize
+
+# For every (n1, n2, n3) triple, applying n1 then n2 must be the same
+# as applying n3.
+# Reference: http://unicode.org/reports/tr15/#Design_Goals
+compositions = [
+    ('NFC', 'NFC', 'NFC'),
+    ('NFC', 'NFD', 'NFD'),
+    ('NFC', 'NFKC', 'NFKC'),
+    ('NFC', 'NFKD', 'NFKD'),
+    ('NFD', 'NFC', 'NFC'),
+    ('NFD', 'NFD', 'NFD'),
+    ('NFD', 'NFKC', 'NFKC'),
+    ('NFD', 'NFKD', 'NFKD'),
+    ('NFKC', 'NFC', 'NFKC'),
+    ('NFKC', 'NFD', 'NFKD'),
+    ('NFKC', 'NFKC', 'NFKC'),
+    ('NFKC', 'NFKD', 'NFKD'),
+    ('NFKD', 'NFC', 'NFKC'),
+    ('NFKD', 'NFD', 'NFKD'),
+    ('NFKD', 'NFKC', 'NFKC'),
+    ('NFKD', 'NFKD', 'NFKD'),
+]
+
+ at pytest.mark.parametrize('norm1, norm2, norm3', compositions)
+ at settings(max_examples=1000)
+ at example(s=u'---\uafb8\u11a7---')  # issue 2289
+ at given(s=st.text())
+def test_composition(s, norm1, norm2, norm3):
+    assert normalize(norm2, normalize(norm1, s)) == normalize(norm3, s)
diff --git a/include/README b/include/README
--- a/include/README
+++ b/include/README
@@ -1,7 +1,11 @@
 This directory contains all the include files needed to build cpython
 extensions with PyPy.  Note that these are just copies of the original headers
-that are in pypy/module/cpyext/include: they are automatically copied from
-there during translation.
+that are in pypy/module/cpyext/{include,parse}: they are automatically copied
+from there during translation.
 
-Moreover, pypy_decl.h and pypy_macros.h are automatically generated, also
-during translation.
+Moreover, some pypy-specific files are automatically generated, also during
+translation. Currently they are:
+* pypy_decl.h
+* pypy_macros.h
+* pypy_numpy.h
+* pypy_structmember_decl.h
diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -361,17 +361,20 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+            if _sys.maxint > 2 ** 32:
+                handle = int(handle)   # long -> int
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxint * 2 + 1))
-
+        return "<%s '%s', handle %x at %x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxint*2 + 1)),
+                id(self) & (_sys.maxint*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib-python/2.7/ctypes/test/test_unaligned_structures.py b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
--- a/lib-python/2.7/ctypes/test/test_unaligned_structures.py
+++ b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
@@ -37,7 +37,10 @@
         for typ in byteswapped_structures:
 ##            print >> sys.stderr, typ.value
             self.assertEqual(typ.value.offset, 1)
-            o = typ()
+            try:
+                o = typ()
+            except NotImplementedError as e:
+                self.skipTest(str(e))    # for PyPy
             o.value = 4
             self.assertEqual(o.value, 4)
 
diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -61,12 +61,12 @@
 def _init_posix():
     """Initialize the module as appropriate for POSIX systems."""
     g = {}
-    g['CC'] = "gcc -pthread"
-    g['CXX'] = "g++ -pthread"
+    g['CC'] = "cc -pthread"
+    g['CXX'] = "c++ -pthread"
     g['OPT'] = "-DNDEBUG -O2"
     g['CFLAGS'] = "-DNDEBUG -O2"
     g['CCSHARED'] = "-fPIC"
-    g['LDSHARED'] = "gcc -pthread -shared"
+    g['LDSHARED'] = "cc -pthread -shared"
     g['SO'] = [s[0] for s in imp.get_suffixes() if s[2] == imp.C_EXTENSION][0]
     g['AR'] = "ar"
     g['ARFLAGS'] = "rc"
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib-python/2.7/sysconfig.py b/lib-python/2.7/sysconfig.py
--- a/lib-python/2.7/sysconfig.py
+++ b/lib-python/2.7/sysconfig.py
@@ -29,8 +29,8 @@
     'pypy': {
         'stdlib': '{base}/lib-{implementation_lower}/{py_version_short}',
         'platstdlib': '{base}/lib-{implementation_lower}/{py_version_short}',
-        'purelib': '{base}/lib-{implementation_lower}/{py_version_short}',
-        'platlib': '{base}/lib-{implementation_lower}/{py_version_short}',
+        'purelib': '{base}/site-packages',
+        'platlib': '{base}/site-packages',
         'include': '{base}/include',
         'platinclude': '{base}/include',
         'scripts': '{base}/bin',
@@ -369,11 +369,8 @@
 
 def _init_posix(vars):
     """Initialize the module as appropriate for POSIX systems."""
-    # in cPython, _sysconfigdata is generated at build time, see _generate_posix_vars()
-    # in PyPy no such module exists
-    #from _sysconfigdata import build_time_vars
-    #vars.update(build_time_vars)
-    return
+    from _sysconfigdata import build_time_vars
+    vars.update(build_time_vars)
 
 def _init_non_posix(vars):
     """Initialize the module as appropriate for NT"""
@@ -529,7 +526,9 @@
         for suffix, mode, type_ in imp.get_suffixes():
             if type_ == imp.C_EXTENSION:
                 _CONFIG_VARS['SOABI'] = suffix.split('.')[1]
-                break        
+                break
+        _CONFIG_VARS['INCLUDEPY'] = os.path.join(_CONFIG_VARS['prefix'],
+                                                 'include')
 
     if args:
         vals = []
diff --git a/lib-python/2.7/test/test_os.py b/lib-python/2.7/test/test_os.py
--- a/lib-python/2.7/test/test_os.py
+++ b/lib-python/2.7/test/test_os.py
@@ -580,6 +580,7 @@
                  "getentropy() does not use a file descriptor")
 class URandomFDTests(unittest.TestCase):
     @unittest.skipUnless(resource, "test requires the resource module")
+    @test_support.impl_detail(pypy=False)    # on Linux, may use getrandom()
     def test_urandom_failure(self):
         # Check urandom() failing when it is not able to open /dev/random.
         # We spawn a new process to make the test more robust (if getrlimit()
diff --git a/lib-python/2.7/warnings.py b/lib-python/2.7/warnings.py
--- a/lib-python/2.7/warnings.py
+++ b/lib-python/2.7/warnings.py
@@ -309,9 +309,12 @@
 
     def __init__(self, message, category, filename, lineno, file=None,
                     line=None):
-        local_values = locals()
-        for attr in self._WARNING_DETAILS:
-            setattr(self, attr, local_values[attr])
+        self.message = message
+        self.category = category
+        self.filename = filename
+        self.lineno = lineno
+        self.file = file
+        self.line = line
         self._category_name = category.__name__ if category else None
 
     def __str__(self):
diff --git a/lib-python/2.7/weakref.py b/lib-python/2.7/weakref.py
--- a/lib-python/2.7/weakref.py
+++ b/lib-python/2.7/weakref.py
@@ -36,9 +36,9 @@
 except ImportError:
     def _delitem_if_value_is(d, key, value):
         try:
-            if self.data[key] is value:  # fall-back: there is a potential
+            if d[key] is value:  # fall-back: there is a potential
                 #             race condition in multithreaded programs HERE
-                del self.data[key]
+                del d[key]
         except KeyError:
             pass
 
diff --git a/lib-python/2.7/zipfile.py b/lib-python/2.7/zipfile.py
--- a/lib-python/2.7/zipfile.py
+++ b/lib-python/2.7/zipfile.py
@@ -622,19 +622,23 @@
         """Read and return up to n bytes.
         If the argument is omitted, None, or negative, data is read and returned until EOF is reached..
         """
-        buf = ''
+        # PyPy modification: don't do repeated string concatenation
+        buf = []
+        lenbuf = 0
         if n is None:
             n = -1
         while True:
             if n < 0:
                 data = self.read1(n)
-            elif n > len(buf):
-                data = self.read1(n - len(buf))
+            elif n > lenbuf:
+                data = self.read1(n - lenbuf)
             else:
-                return buf
+                break
             if len(data) == 0:
-                return buf
-            buf += data
+                break
+            lenbuf += len(data)
+            buf.append(data)
+        return "".join(buf)
 
     def _update_crc(self, newdata, eof):
         # Update the CRC using the given data.
diff --git a/lib_pypy/_ctypes/array.py b/lib_pypy/_ctypes/array.py
--- a/lib_pypy/_ctypes/array.py
+++ b/lib_pypy/_ctypes/array.py
@@ -76,17 +76,22 @@
         return self._type_._alignmentofinstances()
 
     def _CData_output(self, resarray, base=None, index=-1):
-        # this seems to be a string if we're array of char, surprise!
-        from ctypes import c_char, c_wchar
-        if self._type_ is c_char:
-            return _rawffi.charp2string(resarray.buffer, self._length_)
-        if self._type_ is c_wchar:
-            return _rawffi.wcharp2unicode(resarray.buffer, self._length_)
+        from _rawffi.alt import types
+        # If a char_p or unichar_p is received, skip the string interpretation
+        if base._ffiargtype != types.Pointer(types.char_p) and \
+           base._ffiargtype != types.Pointer(types.unichar_p):
+            # this seems to be a string if we're array of char, surprise!
+            from ctypes import c_char, c_wchar
+            if self._type_ is c_char:
+                return _rawffi.charp2string(resarray.buffer, self._length_)
+            if self._type_ is c_wchar:
+                return _rawffi.wcharp2unicode(resarray.buffer, self._length_)
         res = self.__new__(self)
         ffiarray = self._ffiarray.fromaddress(resarray.buffer, self._length_)
         res._buffer = ffiarray
-        res._base = base
-        res._index = index
+        if base is not None:
+            res._base = base
+            res._index = index
         return res
 
     def _CData_retval(self, resbuffer):
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -64,8 +64,9 @@
         res = object.__new__(self)
         res.__class__ = self
         res.__dict__['_buffer'] = resbuffer
-        res.__dict__['_base'] = base
-        res.__dict__['_index'] = index
+        if base is not None:
+            res.__dict__['_base'] = base
+            res.__dict__['_index'] = index
         return res
 
     def _CData_retval(self, resbuffer):
@@ -81,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._handle.getaddressindll(name))
+        return self.from_address(dll.__pypy_dll__.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -1,4 +1,3 @@
-
 from _ctypes.basics import _CData, _CDataMeta, cdata_from_address
 from _ctypes.primitive import SimpleType, _SimpleCData
 from _ctypes.basics import ArgumentError, keepalive_key
@@ -9,13 +8,16 @@
 import sys
 import traceback
 
-try: from __pypy__ import builtinify
-except ImportError: builtinify = lambda f: f
+
+try:
+    from __pypy__ import builtinify
+except ImportError:
+    builtinify = lambda f: f
 
 # XXX this file needs huge refactoring I fear
 
-PARAMFLAG_FIN   = 0x1
-PARAMFLAG_FOUT  = 0x2
+PARAMFLAG_FIN = 0x1
+PARAMFLAG_FOUT = 0x2
 PARAMFLAG_FLCID = 0x4
 PARAMFLAG_COMBINED = PARAMFLAG_FIN | PARAMFLAG_FOUT | PARAMFLAG_FLCID
 
@@ -24,9 +26,9 @@
     PARAMFLAG_FIN,
     PARAMFLAG_FIN | PARAMFLAG_FOUT,
     PARAMFLAG_FIN | PARAMFLAG_FLCID
-    )
+)
 
-WIN64 = sys.platform == 'win32' and sys.maxint == 2**63 - 1
+WIN64 = sys.platform == 'win32' and sys.maxint == 2 ** 63 - 1
 
 
 def get_com_error(errcode, riid, pIunk):
@@ -35,6 +37,7 @@
     from _ctypes import COMError
     return COMError(errcode, None, None)
 
+
 @builtinify
 def call_function(func, args):
     "Only for debugging so far: So that we can call CFunction instances"
@@ -94,14 +97,9 @@
                         "item %d in _argtypes_ has no from_param method" % (
                             i + 1,))
             self._argtypes_ = list(argtypes)
-            self._check_argtypes_for_fastpath()
+
     argtypes = property(_getargtypes, _setargtypes)
 
-    def _check_argtypes_for_fastpath(self):
-        if all([hasattr(argtype, '_ffiargshape_') for argtype in self._argtypes_]):
-            fastpath_cls = make_fastpath_subclass(self.__class__)
-            fastpath_cls.enable_fastpath_maybe(self)
-
     def _getparamflags(self):
         return self._paramflags
 
@@ -126,27 +124,26 @@
                 raise TypeError(
                     "paramflags must be a sequence of (int [,string [,value]]) "
                     "tuples"
-                    )
+                )
             if not isinstance(flag, int):
                 raise TypeError(
                     "paramflags must be a sequence of (int [,string [,value]]) "
                     "tuples"
-                    )
+                )
             _flag = flag & PARAMFLAG_COMBINED
             if _flag == PARAMFLAG_FOUT:
                 typ = self._argtypes_[idx]
                 if getattr(typ, '_ffiargshape_', None) not in ('P', 'z', 'Z'):
                     raise TypeError(
                         "'out' parameter %d must be a pointer type, not %s"
-                        % (idx+1, type(typ).__name__)
-                        )
+                        % (idx + 1, type(typ).__name__)
+                    )
             elif _flag not in VALID_PARAMFLAGS:
                 raise TypeError("paramflag value %d not supported" % flag)
         self._paramflags = paramflags
 
     paramflags = property(_getparamflags, _setparamflags)
 
-
     def _getrestype(self):
         return self._restype_
 
@@ -156,7 +153,7 @@
             from ctypes import c_int
             restype = c_int
         if not (isinstance(restype, _CDataMeta) or restype is None or
-                callable(restype)):
+                    callable(restype)):
             raise TypeError("restype must be a type, a callable, or None")
         self._restype_ = restype
 
@@ -168,15 +165,18 @@
 
     def _geterrcheck(self):
         return getattr(self, '_errcheck_', None)
+
     def _seterrcheck(self, errcheck):
         if not callable(errcheck):
             raise TypeError("The errcheck attribute must be callable")
         self._errcheck_ = errcheck
+
     def _delerrcheck(self):
         try:
             del self._errcheck_
         except AttributeError:
             pass
+
     errcheck = property(_geterrcheck, _seterrcheck, _delerrcheck)
 
     def _ffishapes(self, args, restype):
@@ -188,7 +188,7 @@
                 raise TypeError("invalid result type for callback function")
             restype = restype._ffiargshape_
         else:
-            restype = 'O' # void
+            restype = 'O'  # void
         return argtypes, restype
 
     def _set_address(self, address):
@@ -201,7 +201,7 @@
 
     def __init__(self, *args):
         self.name = None
-        self._objects = {keepalive_key(0):self}
+        self._objects = {keepalive_key(0): self}
         self._needs_free = True
 
         # Empty function object -- this is needed for casts
@@ -222,10 +222,8 @@
             if self._argtypes_ is None:
                 self._argtypes_ = []
             self._ptr = self._getfuncptr_fromaddress(self._argtypes_, restype)
-            self._check_argtypes_for_fastpath()
             return
 
-
         # A callback into python
         if callable(argument) and not argsl:
             self.callable = argument
@@ -259,7 +257,7 @@
         if (sys.platform == 'win32' and isinstance(argument, (int, long))
             and argsl):
             ffiargs, ffires = self._ffishapes(self._argtypes_, self._restype_)
-            self._com_index =  argument + 0x1000
+            self._com_index = argument + 0x1000
             self.name = argsl.pop(0)
             if argsl:
                 self.paramflags = argsl.pop(0)
@@ -281,6 +279,7 @@
             except SystemExit as e:
                 handle_system_exit(e)
                 raise
+
         return f
 
     def __call__(self, *args, **kwargs):
@@ -317,7 +316,7 @@
             except:
                 exc_info = sys.exc_info()
                 traceback.print_tb(exc_info[2], file=sys.stderr)
-                print >>sys.stderr, "%s: %s" % (exc_info[0].__name__, exc_info[1])
+                print >> sys.stderr, "%s: %s" % (exc_info[0].__name__, exc_info[1])
                 return 0
             if self._restype_ is not None:
                 return res
@@ -328,7 +327,7 @@
             # really slow".  Now we don't worry that much about slowness
             # of ctypes, and it's strange to get warnings for perfectly-
             # legal code.
-            #warnings.warn('C function without declared arguments called',
+            # warnings.warn('C function without declared arguments called',
             #              RuntimeWarning, stacklevel=2)
             argtypes = []
 
@@ -337,7 +336,7 @@
             if not args:
                 raise ValueError(
                     "native COM method call without 'this' parameter"
-                    )
+                )
             thisvalue = args[0]
             thisarg = cast(thisvalue, POINTER(POINTER(c_void_p)))
             keepalives, newargs, argtypes, outargs, errcheckargs = (
@@ -366,7 +365,6 @@
         return tuple(outargs)
 
     def _call_funcptr(self, funcptr, *newargs):
-
         if self._flags_ & _rawffi.FUNCFLAG_USE_ERRNO:
             tmp = _rawffi.get_errno()
             _rawffi.set_errno(get_errno())
@@ -431,8 +429,8 @@
             ffiargs = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
-        
-        cdll = self.dll._handle
+
+        cdll = self.dll.__pypy_dll__
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
@@ -450,7 +448,7 @@
             # funcname -> _funcname@<n>
             # where n is 0, 4, 8, 12, ..., 128
             for i in range(33):
-                mangled_name = "_%s@%d" % (self.name, i*4)
+                mangled_name = "_%s@%d" % (self.name, i * 4)
                 try:
                     return cdll.getfunc(mangled_name,
                                         ffi_argtypes, ffi_restype,
@@ -492,7 +490,7 @@
         for argtype, arg in zip(argtypes, args):
             param = argtype.from_param(arg)
             _type_ = getattr(argtype, '_type_', None)
-            if _type_ == 'P': # special-case for c_void_p
+            if _type_ == 'P':  # special-case for c_void_p
                 param = param._get_buffer_value()
             elif self._is_primitive(argtype):
                 param = param.value
@@ -668,69 +666,11 @@
             self._needs_free = False
 
 
-def make_fastpath_subclass(CFuncPtr):
-    if CFuncPtr._is_fastpath:
-        return CFuncPtr
-    #
-    try:
-        return make_fastpath_subclass.memo[CFuncPtr]
-    except KeyError:
-        pass
-
-    class CFuncPtrFast(CFuncPtr):
-
-        _is_fastpath = True
-        _slowpath_allowed = True # set to False by tests
-
-        @classmethod
-        def enable_fastpath_maybe(cls, obj):
-            if (obj.callable is None and
-                obj._com_index is None):
-                obj.__class__ = cls
-
-        def __rollback(self):
-            assert self._slowpath_allowed
-            self.__class__ = CFuncPtr
-
-        # disable the fast path if we reset argtypes
-        def _setargtypes(self, argtypes):
-            self.__rollback()
-            self._setargtypes(argtypes)
-        argtypes = property(CFuncPtr._getargtypes, _setargtypes)
-
-        def _setcallable(self, func):
-            self.__rollback()
-            self.callable = func
-        callable = property(lambda x: None, _setcallable)
-
-        def _setcom_index(self, idx):
-            self.__rollback()
-            self._com_index = idx
-        _com_index = property(lambda x: None, _setcom_index)
-
-        def __call__(self, *args):
-            thisarg = None
-            argtypes = self._argtypes_
-            restype = self._restype_
-            funcptr = self._getfuncptr(argtypes, restype, thisarg)
-            try:
-                result = self._call_funcptr(funcptr, *args)
-                result, _ = self._do_errcheck(result, args)
-            except (TypeError, ArgumentError, UnicodeDecodeError):
-                assert self._slowpath_allowed
-                return CFuncPtr.__call__(self, *args)
-            return result
-
-    make_fastpath_subclass.memo[CFuncPtr] = CFuncPtrFast
-    return CFuncPtrFast
-make_fastpath_subclass.memo = {}
-
-
 def handle_system_exit(e):
     # issue #1194: if we get SystemExit here, then exit the interpreter.
     # Highly obscure imho but some people seem to depend on it.
     if sys.flags.inspect:
-        return   # Don't exit if -i flag was given.
+        return  # Don't exit if -i flag was given.
     else:
         code = e.code
         if isinstance(code, int):
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
@@ -234,6 +250,9 @@
         if ('_abstract_' in cls.__dict__ or cls is Structure 
                                          or cls is union.Union):
             raise TypeError("abstract class")
+        if hasattr(cls, '_swappedbytes_'):
+            raise NotImplementedError("missing in PyPy: structure/union with "
+                                      "swapped (non-native) byte ordering")
         if hasattr(cls, '_ffistruct_'):
             self.__dict__['_buffer'] = self._ffistruct_(autofree=True)
         return self
diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py
--- a/lib_pypy/_curses.py
+++ b/lib_pypy/_curses.py
@@ -8,6 +8,9 @@
 
 from _curses_cffi import ffi, lib
 
+version = b"2.2"
+__version__ = b"2.2"
+
 def _copy_to_globals(name):
     globals()[name] = getattr(lib, name)
 
@@ -60,10 +63,6 @@
 
 _setup()
 
-# Do we want this?
-# version = "2.2"
-# __version__ = "2.2"
-
 
 # ____________________________________________________________
 
@@ -913,101 +912,29 @@
     return None
 
 
-# XXX: Do something about the following?
-# /* Internal helper used for updating curses.LINES, curses.COLS, _curses.LINES
-#  * and _curses.COLS */
-# #if defined(HAVE_CURSES_RESIZETERM) || defined(HAVE_CURSES_RESIZE_TERM)
-# static int
-# update_lines_cols(void)
-# {
-#     PyObject *o;
-#     PyObject *m = PyImport_ImportModuleNoBlock("curses");
+# Internal helper used for updating curses.LINES, curses.COLS, _curses.LINES
+# and _curses.COLS
+def update_lines_cols():
+    globals()["LINES"] = lib.LINES
+    globals()["COLS"] = lib.COLS
+    try:
+        m = sys.modules["curses"]
+        m.LINES = lib.LINES
+        m.COLS = lib.COLS
+    except (KeyError, AttributeError):
+        pass
 
-#     if (!m)
-#         return 0;
 
-#     o = PyInt_FromLong(LINES);
-#     if (!o) {
-#         Py_DECREF(m);
-#         return 0;
-#     }
-#     if (PyObject_SetAttrString(m, "LINES", o)) {
-#         Py_DECREF(m);
-#         Py_DECREF(o);
-#         return 0;
-#     }
-#     if (PyDict_SetItemString(ModDict, "LINES", o)) {
-#         Py_DECREF(m);
-#         Py_DECREF(o);
-#         return 0;
-#     }
-#     Py_DECREF(o);
-#     o = PyInt_FromLong(COLS);
-#     if (!o) {
-#         Py_DECREF(m);
-#         return 0;
-#     }
-#     if (PyObject_SetAttrString(m, "COLS", o)) {
-#         Py_DECREF(m);
-#         Py_DECREF(o);
-#         return 0;
-#     }
-#     if (PyDict_SetItemString(ModDict, "COLS", o)) {
-#         Py_DECREF(m);
-#         Py_DECREF(o);
-#         return 0;
-#     }
-#     Py_DECREF(o);
-#     Py_DECREF(m);
-#     return 1;
-# }
-# #endif
+def resizeterm(lines, columns):
+    _ensure_initialised()
+    _check_ERR(lib.resizeterm(lines, columns), "resizeterm")
+    update_lines_cols()
 
-# #ifdef HAVE_CURSES_RESIZETERM
-# static PyObject *
-# PyCurses_ResizeTerm(PyObject *self, PyObject *args)
-# {
-#     int lines;
-#     int columns;
-#     PyObject *result;
 
-#     PyCursesInitialised;
-
-#     if (!PyArg_ParseTuple(args,"ii:resizeterm", &lines, &columns))
-#         return NULL;
-
-#     result = PyCursesCheckERR(resizeterm(lines, columns), "resizeterm");
-#     if (!result)
-#         return NULL;
-#     if (!update_lines_cols())
-#         return NULL;
-#     return result;
-# }
-
-# #endif
-
-# #ifdef HAVE_CURSES_RESIZE_TERM
-# static PyObject *
-# PyCurses_Resize_Term(PyObject *self, PyObject *args)
-# {
-#     int lines;
-#     int columns;
-
-#     PyObject *result;
-
-#     PyCursesInitialised;
-
-#     if (!PyArg_ParseTuple(args,"ii:resize_term", &lines, &columns))
-#         return NULL;
-
-#     result = PyCursesCheckERR(resize_term(lines, columns), "resize_term");
-#     if (!result)
-#         return NULL;
-#     if (!update_lines_cols())
-#         return NULL;
-#     return result;
-# }
-# #endif /* HAVE_CURSES_RESIZE_TERM */
+def resize_term(lines, columns):
+    _ensure_initialised()
+    _check_ERR(lib.resize_term(lines, columns), "resize_term")
+    update_lines_cols()
 
 
 def setsyx(y, x):
diff --git a/lib_pypy/_curses_build.py b/lib_pypy/_curses_build.py
--- a/lib_pypy/_curses_build.py
+++ b/lib_pypy/_curses_build.py
@@ -87,6 +87,13 @@
 static const chtype A_CHARTEXT;
 static const chtype A_COLOR;
 
+static const chtype A_HORIZONTAL;
+static const chtype A_LEFT;
+static const chtype A_LOW;
+static const chtype A_RIGHT;
+static const chtype A_TOP;
+static const chtype A_VERTICAL;
+
 static const int BUTTON1_RELEASED;
 static const int BUTTON1_PRESSED;
 static const int BUTTON1_CLICKED;
@@ -202,6 +209,8 @@
 int resetty(void);
 int reset_prog_mode(void);
 int reset_shell_mode(void);
+int resizeterm(int, int);
+int resize_term(int, int);
 int savetty(void);
 int scroll(WINDOW *);
 int scrollok(WINDOW *, bool);
diff --git a/lib_pypy/_pypy_winbase_build.py b/lib_pypy/_pypy_winbase_build.py
--- a/lib_pypy/_pypy_winbase_build.py
+++ b/lib_pypy/_pypy_winbase_build.py
@@ -79,10 +79,20 @@
 BOOL WINAPI CreateProcessA(char *, char *, void *,
                            void *, BOOL, DWORD, char *,
                            char *, LPSTARTUPINFO, LPPROCESS_INFORMATION);
+BOOL WINAPI CreateProcessW(wchar_t *, wchar_t *, void *,
+                           void *, BOOL, DWORD, wchar_t *,
+                           wchar_t *, LPSTARTUPINFO, LPPROCESS_INFORMATION);
 DWORD WINAPI WaitForSingleObject(HANDLE, DWORD);
 BOOL WINAPI GetExitCodeProcess(HANDLE, LPDWORD);
 BOOL WINAPI TerminateProcess(HANDLE, UINT);
 HANDLE WINAPI GetStdHandle(DWORD);
+DWORD WINAPI GetModuleFileNameW(HANDLE, wchar_t *, DWORD);
+
+UINT WINAPI SetErrorMode(UINT);
+#define SEM_FAILCRITICALERRORS     0x0001
+#define SEM_NOGPFAULTERRORBOX      0x0002
+#define SEM_NOALIGNMENTFAULTEXCEPT 0x0004
+#define SEM_NOOPENFILEERRORBOX     0x8000
 """)
 
 # --------------------
diff --git a/lib_pypy/_pypy_winbase_cffi.py b/lib_pypy/_pypy_winbase_cffi.py
--- a/lib_pypy/_pypy_winbase_cffi.py
+++ b/lib_pypy/_pypy_winbase_cffi.py
@@ -3,8 +3,8 @@
 
 ffi = _cffi_backend.FFI('_pypy_winbase_cffi',
     _version = 0x2601,
-    _types = b'\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x07\x01\x00\x00\x09\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x19\x01\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x50\x03\x00\x00\x13\x11\x00\x00\x53\x03\x00\x00\x15\x11\x00\x00\x07\x01\x00\x00\x0A\x01\x00\x00\x13\x11\x00\x00\x13\x11\x00\x00\x4F\x03\x00\x00\x4E\x03\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x03\x00\x00\x1F\x11\x00\x00\x15\x11\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x08\x01\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x18\x03\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x15\x11\x00\x00\x15\x11\x00\x00\x1F\x11\x00\x00\x0A\x01\x00\x00\x07\x01\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x0D\x0D\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x18\x0D\x00\x00\x15\x11\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x18\x0D\x00\x00\x02\x0F\x00\x00\x42\x0D\x00\x00\x06\x01\x00\x00\x00\x0F\x00\x00\x42\x0D\x00\x00\x00\x0F\x00\x00\x42\x0D\x00\x00\x10\x01\x00\x00\x00\x0F\x00\x00\x15\x0D\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x15\x0D\x00\x00\x02\x0F\x00\x00\x00\x09\x00\x00\x01\x09\x00\x00\x02\x01\x00\x00\x52\x03\x00\x00\x04\x01\x00\x00\x00\x01',
-    _globals = (b'\x00\x00\x24\x23CloseHandle',0,b'\x00\x00\x1E\x23CreatePipe',0,b'\x00\x00\x12\x23CreateProcessA',0,b'\x00\x00\x2F\x23DuplicateHandle',0,b'\x00\x00\x4C\x23GetCurrentProcess',0,b'\x00\x00\x2B\x23GetExitCodeProcess',0,b'\x00\x00\x49\x23GetStdHandle',0,b'\x00\x00\x3F\x23GetVersion',0,b'\x00\x00\x27\x23TerminateProcess',0,b'\x00\x00\x3B\x23WaitForSingleObject',0,b'\x00\x00\x38\x23_get_osfhandle',0,b'\x00\x00\x10\x23_getch',0,b'\x00\x00\x10\x23_getche',0,b'\x00\x00\x44\x23_getwch',0,b'\x00\x00\x44\x23_getwche',0,b'\x00\x00\x10\x23_kbhit',0,b'\x00\x00\x07\x23_locking',0,b'\x00\x00\x0C\x23_open_osfhandle',0,b'\x00\x00\x00\x23_putch',0,b'\x00\x00\x46\x23_putwch',0,b'\x00\x00\x03\x23_setmode',0,b'\x00\x00\x00\x23_ungetch',0,b'\x00\x00\x41\x23_ungetwch',0),
-    _struct_unions = ((b'\x00\x00\x00\x4E\x00\x00\x00\x02$PROCESS_INFORMATION',b'\x00\x00\x15\x11hProcess',b'\x00\x00\x15\x11hThread',b'\x00\x00\x18\x11dwProcessId',b'\x00\x00\x18\x11dwThreadId'),(b'\x00\x00\x00\x4F\x00\x00\x00\x02$STARTUPINFO',b'\x00\x00\x18\x11cb',b'\x00\x00\x13\x11lpReserved',b'\x00\x00\x13\x11lpDesktop',b'\x00\x00\x13\x11lpTitle',b'\x00\x00\x18\x11dwX',b'\x00\x00\x18\x11dwY',b'\x00\x00\x18\x11dwXSize',b'\x00\x00\x18\x11dwYSize',b'\x00\x00\x18\x11dwXCountChars',b'\x00\x00\x18\x11dwYCountChars',b'\x00\x00\x18\x11dwFillAttribute',b'\x00\x00\x18\x11dwFlags',b'\x00\x00\x42\x11wShowWindow',b'\x00\x00\x42\x11cbReserved2',b'\x00\x00\x51\x11lpReserved2',b'\x00\x00\x15\x11hStdInput',b'\x00\x00\x15\x11hStdOutput',b'\x00\x00\x15\x11hStdError')),
-    _typenames = (b'\x00\x00\x00\x1CLPPROCESS_INFORMATION',b'\x00\x00\x00\x1BLPSTARTUPINFO',b'\x00\x00\x00\x4EPROCESS_INFORMATION',b'\x00\x00\x00\x4FSTARTUPINFO',b'\x00\x00\x00\x42wint_t'),
+    _types = b'\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x07\x01\x00\x00\x07\x01\x00\x00\x09\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x19\x01\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x00\x0F\x00\x00\x01\x0D\x00\x00\x64\x03\x00\x00\x13\x11\x00\x00\x67\x03\x00\x00\x15\x11\x00\x00\x07\x01\x00\x00\x0A\x01\x00\x00\x13\x11\x00\x00\x13\x11\x00\x00\x63\x03\x00\x00\x62\x03\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x03\x00\x00\x1F\x11\x00\x00\x15\x11\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x08\x01\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x18\x03\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x15\x11\x00\x00\x15\x11\x00\x00\x15\x11\x00\x00\x1F\x11\x00\x00\x0A\x01\x00\x00\x07\x01\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x01\x0D\x00\x00\x5B\x03\x00\x00\x39\x11\x00\x00\x15\x11\x00\x00\x15\x11\x00\x00\x07\x01\x00\x00\x0A\x01\x00\x00\x39\x11\x00\x00\x39\x11\x00\x00\x1B\x11\x00\x00\x1C\x11\x00\x00\x02\x0F\x00\x00\x0D\x0D\x00\x00\x07\x01\x00\x00\x00\x0F\x00\x00\x29\x0D\x00\x00\x08\x01\x00\x00\x02\x0F\x00\x00\x18\x0D\x00\x00\x15\x11\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x18\x0D\x00\x00\x15\x11\x00\x00\x39\x11\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x18\x0D\x00\x00\x02\x0F\x00\x00\x56\x0D\x00\x00\x06\x01\x00\x00\x00\x0F\x00\x00\x56\x0D\x00\x00\x00\x0F\x00\x00\x56\x0D\x00\x00\x10\x01\x00\x00\x00\x0F\x00\x00\x15\x0D\x00\x00\x0A\x01\x00\x00\x02\x0F\x00\x00\x15\x0D\x00\x00\x02\x0F\x00\x00\x00\x09\x00\x00\x01\x09\x00\x00\x02\x01\x00\x00\x66\x03\x00\x00\x04\x01\x00\x00\x00\x01',
+    _globals = (b'\x00\x00\x24\x23CloseHandle',0,b'\x00\x00\x1E\x23CreatePipe',0,b'\x00\x00\x12\x23CreateProcessA',0,b'\x00\x00\x38\x23CreateProcessW',0,b'\x00\x00\x2F\x23DuplicateHandle',0,b'\x00\x00\x60\x23GetCurrentProcess',0,b'\x00\x00\x2B\x23GetExitCodeProcess',0,b'\x00\x00\x4E\x23GetModuleFileNameW',0,b'\x00\x00\x5D\x23GetStdHandle',0,b'\x00\x00\x53\x23GetVersion',0,b'\xFF\xFF\xFF\x1FSEM_FAILCRITICALERRORS',1,b'\xFF\xFF\xFF\x1FSEM_NOALIGNMENTFAULTEXCEPT',4,b'\xFF\xFF\xFF\x1FSEM_NOGPFAULTERRORBOX',2,b'\xFF\xFF\xFF\x1FSEM_NOOPENFILEERRORBOX',32768,b'\x00\x00\x47\x23SetErrorMode',0,b'\x00\x00\x27\x23TerminateProcess',0,b'\x00\x00\x4A\x23WaitForSingleObject',0,b'\x00\x00\x44\x23_get_osfhandle',0,b'\x00\x00\x10\x23_getch',0,b'\x00\x00\x10\x23_getche',0,b'\x00\x00\x58\x23_getwch',0,b'\x00\x00\x58\x23_getwche',0,b'\x00\x00\x10\x23_kbhit',0,b'\x00\x00\x07\x23_locking',0,b'\x00\x00\x0C\x23_open_osfhandle',0,b'\x00\x00\x00\x23_putch',0,b'\x00\x00\x5A\x23_putwch',0,b'\x00\x00\x03\x23_setmode',0,b'\x00\x00\x00\x23_ungetch',0,b'\x00\x00\x55\x23_ungetwch',0),
+    _struct_unions = ((b'\x00\x00\x00\x62\x00\x00\x00\x02$PROCESS_INFORMATION',b'\x00\x00\x15\x11hProcess',b'\x00\x00\x15\x11hThread',b'\x00\x00\x18\x11dwProcessId',b'\x00\x00\x18\x11dwThreadId'),(b'\x00\x00\x00\x63\x00\x00\x00\x02$STARTUPINFO',b'\x00\x00\x18\x11cb',b'\x00\x00\x13\x11lpReserved',b'\x00\x00\x13\x11lpDesktop',b'\x00\x00\x13\x11lpTitle',b'\x00\x00\x18\x11dwX',b'\x00\x00\x18\x11dwY',b'\x00\x00\x18\x11dwXSize',b'\x00\x00\x18\x11dwYSize',b'\x00\x00\x18\x11dwXCountChars',b'\x00\x00\x18\x11dwYCountChars',b'\x00\x00\x18\x11dwFillAttribute',b'\x00\x00\x18\x11dwFlags',b'\x00\x00\x56\x11wShowWindow',b'\x00\x00\x56\x11cbReserved2',b'\x00\x00\x65\x11lpReserved2',b'\x00\x00\x15\x11hStdInput',b'\x00\x00\x15\x11hStdOutput',b'\x00\x00\x15\x11hStdError')),
+    _typenames = (b'\x00\x00\x00\x1CLPPROCESS_INFORMATION',b'\x00\x00\x00\x1BLPSTARTUPINFO',b'\x00\x00\x00\x62PROCESS_INFORMATION',b'\x00\x00\x00\x63STARTUPINFO',b'\x00\x00\x00\x56wint_t'),
 )
diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py
--- a/lib_pypy/_sqlite3.py
+++ b/lib_pypy/_sqlite3.py
@@ -31,10 +31,11 @@
 import weakref
 from threading import _get_ident as _thread_get_ident
 try:
-    from __pypy__ import newlist_hint
+    from __pypy__ import newlist_hint, add_memory_pressure
 except ImportError:
     assert '__pypy__' not in sys.builtin_module_names
     newlist_hint = lambda sizehint: []
+    add_memory_pressure = lambda size: None
 
 if sys.version_info[0] >= 3:
     StandardError = Exception
@@ -150,6 +151,9 @@
 def connect(database, timeout=5.0, detect_types=0, isolation_level="",
                  check_same_thread=True, factory=None, cached_statements=100):
     factory = Connection if not factory else factory
+    # an sqlite3 db seems to be around 100 KiB at least (doesn't matter if
+    # backed by :memory: or a file)
+    add_memory_pressure(100 * 1024)
     return factory(database, timeout, detect_types, isolation_level,
                     check_same_thread, factory, cached_statements)
 
diff --git a/lib_pypy/_sysconfigdata.py b/lib_pypy/_sysconfigdata.py
new file mode 100644
--- /dev/null
+++ b/lib_pypy/_sysconfigdata.py
@@ -0,0 +1,5 @@
+import imp
+
+build_time_vars = {
+    "SO": [s[0] for s in imp.get_suffixes() if s[2] == imp.C_EXTENSION][0]
+}
diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py
--- a/lib_pypy/_tkinter/tklib_build.py
+++ b/lib_pypy/_tkinter/tklib_build.py
@@ -22,12 +22,27 @@
     linklibs = ['tcl', 'tk']
     libdirs = []
 else:
-    for _ver in ['', '8.6', '8.5', '']:
+    # On some Linux distributions, the tcl and tk libraries are
+    # stored in /usr/include, so we must check this case also
+    libdirs = []
+    found = False
+    for _ver in ['', '8.6', '8.5']:
         incdirs = ['/usr/include/tcl' + _ver]
         linklibs = ['tcl' + _ver, 'tk' + _ver]
-        libdirs = []
         if os.path.isdir(incdirs[0]):
+            found = True
             break
+    if not found:
+        for _ver in ['8.6', '8.5', '']:
+            incdirs = []
+            linklibs = ['tcl' + _ver, 'tk' + _ver]
+            if os.path.isfile(''.join(['/usr/lib/lib', linklibs[1], '.so'])):
+                found = True
+                break
+    if not found:
+        sys.stderr.write("*** TCL libraries not found!  Falling back...\n")
+        incdirs = []
+        linklibs = ['tcl', 'tk']
 
 config_ffi = FFI()
 config_ffi.cdef("""
diff --git a/lib_pypy/cPickle.py b/lib_pypy/cPickle.py
--- a/lib_pypy/cPickle.py
+++ b/lib_pypy/cPickle.py
@@ -116,10 +116,20 @@
 
 @builtinify
 def dump(obj, file, protocol=None):
+    if protocol > HIGHEST_PROTOCOL:
+        # use cPickle error message, not pickle.py one
+        raise ValueError("pickle protocol %d asked for; "
+                     "the highest available protocol is %d" % (
+                     protocol, HIGHEST_PROTOCOL))
     Pickler(file, protocol).dump(obj)
 
 @builtinify
 def dumps(obj, protocol=None):
+    if protocol > HIGHEST_PROTOCOL:
+        # use cPickle error message, not pickle.py one
+        raise ValueError("pickle protocol %d asked for; "
+                     "the highest available protocol is %d" % (
+                     protocol, HIGHEST_PROTOCOL))
     file = StringIO()
     Pickler(file, protocol).dump(obj)
     return file.getvalue()
@@ -431,7 +441,14 @@
         self.append(obj)
 
     def find_class(self, module, name):
-        # Subclasses may override this
+        if self.find_global is None:
+            raise UnpicklingError(
+                "Global and instance pickles are not supported.")
+        return self.find_global(module, name)
+
+    def find_global(self, module, name):
+        # This can officially be patched directly in the Unpickler
+        # instance, according to the docs
         __import__(module)
         mod = sys.modules[module]
         klass = getattr(mod, name)
diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO
--- a/lib_pypy/cffi.egg-info/PKG-INFO
+++ b/lib_pypy/cffi.egg-info/PKG-INFO
@@ -1,6 +1,6 @@
 Metadata-Version: 1.1
 Name: cffi
-Version: 1.10.0
+Version: 1.11.0
 Summary: Foreign Function Interface for Python calling C code.
 Home-page: http://cffi.readthedocs.org
 Author: Armin Rigo, Maciej Fijalkowski
diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py
--- a/lib_pypy/cffi/__init__.py
+++ b/lib_pypy/cffi/__init__.py
@@ -4,8 +4,8 @@
 from .api import FFI
 from .error import CDefError, FFIError, VerificationError, VerificationMissing
 
-__version__ = "1.10.0"
-__version_info__ = (1, 10, 0)
+__version__ = "1.11.0"
+__version_info__ = (1, 11, 0)
 
 # The verifier module file names are based on the CRC32 of a string that
 # contains the following version number.  It may be older than __version__
diff --git a/lib_pypy/cffi/_cffi_errors.h b/lib_pypy/cffi/_cffi_errors.h
new file mode 100644
--- /dev/null
+++ b/lib_pypy/cffi/_cffi_errors.h
@@ -0,0 +1,145 @@
+#ifndef CFFI_MESSAGEBOX
+# ifdef _MSC_VER
+#  define CFFI_MESSAGEBOX  1
+# else
+#  define CFFI_MESSAGEBOX  0
+# endif
+#endif
+
+
+#if CFFI_MESSAGEBOX
+/* Windows only: logic to take the Python-CFFI embedding logic
+   initialization errors and display them in a background thread
+   with MessageBox.  The idea is that if the whole program closes
+   as a result of this problem, then likely it is already a console
+   program and you can read the stderr output in the console too.
+   If it is not a console program, then it will likely show its own
+   dialog to complain, or generally not abruptly close, and for this
+   case the background thread should stay alive.
+*/
+static void *volatile _cffi_bootstrap_text;
+
+static PyObject *_cffi_start_error_capture(void)
+{
+    PyObject *result = NULL;
+    PyObject *x, *m, *bi;
+
+    if (InterlockedCompareExchangePointer(&_cffi_bootstrap_text,
+            (void *)1, NULL) != NULL)
+        return (PyObject *)1;
+
+    m = PyImport_AddModule("_cffi_error_capture");
+    if (m == NULL)
+        goto error;
+
+    result = PyModule_GetDict(m);
+    if (result == NULL)
+        goto error;
+
+#if PY_MAJOR_VERSION >= 3
+    bi = PyImport_ImportModule("builtins");
+#else
+    bi = PyImport_ImportModule("__builtin__");
+#endif
+    if (bi == NULL)
+        goto error;
+    PyDict_SetItemString(result, "__builtins__", bi);
+    Py_DECREF(bi);
+
+    x = PyRun_String(
+        "import sys\n"
+        "class FileLike:\n"
+        "  def write(self, x):\n"
+        "    of.write(x)\n"
+        "    self.buf += x\n"
+        "fl = FileLike()\n"
+        "fl.buf = ''\n"
+        "of = sys.stderr\n"
+        "sys.stderr = fl\n"
+        "def done():\n"
+        "  sys.stderr = of\n"
+        "  return fl.buf\n",   /* make sure the returned value stays alive */
+        Py_file_input,
+        result, result);
+    Py_XDECREF(x);
+
+ error:
+    if (PyErr_Occurred())
+    {
+        PyErr_WriteUnraisable(Py_None);
+        PyErr_Clear();
+    }
+    return result;
+}
+
+#pragma comment(lib, "user32.lib")
+
+static DWORD WINAPI _cffi_bootstrap_dialog(LPVOID ignored)
+{
+    Sleep(666);    /* may be interrupted if the whole process is closing */
+#if PY_MAJOR_VERSION >= 3
+    MessageBoxW(NULL, (wchar_t *)_cffi_bootstrap_text,
+                L"Python-CFFI error",
+                MB_OK | MB_ICONERROR);
+#else
+    MessageBoxA(NULL, (char *)_cffi_bootstrap_text,
+                "Python-CFFI error",
+                MB_OK | MB_ICONERROR);
+#endif
+    _cffi_bootstrap_text = NULL;
+    return 0;
+}
+
+static void _cffi_stop_error_capture(PyObject *ecap)
+{
+    PyObject *s;
+    void *text;
+
+    if (ecap == (PyObject *)1)
+        return;
+
+    if (ecap == NULL)
+        goto error;
+
+    s = PyRun_String("done()", Py_eval_input, ecap, ecap);
+    if (s == NULL)
+        goto error;
+
+    /* Show a dialog box, but in a background thread, and
+       never show multiple dialog boxes at once. */
+#if PY_MAJOR_VERSION >= 3
+    text = PyUnicode_AsWideCharString(s, NULL);
+#else
+    text = PyString_AsString(s);
+#endif
+
+    _cffi_bootstrap_text = text;
+
+    if (text != NULL)
+    {
+        HANDLE h;
+        h = CreateThread(NULL, 0, _cffi_bootstrap_dialog,
+                         NULL, 0, NULL);
+        if (h != NULL)
+            CloseHandle(h);
+    }
+    /* decref the string, but it should stay alive as 'fl.buf'
+       in the small module above.  It will really be freed only if
+       we later get another similar error.  So it's a leak of at
+       most one copy of the small module.  That's fine for this
+       situation which is usually a "fatal error" anyway. */
+    Py_DECREF(s);
+    PyErr_Clear();
+    return;
+
+  error:
+    _cffi_bootstrap_text = NULL;
+    PyErr_Clear();
+}
+
+#else
+
+static PyObject *_cffi_start_error_capture(void) { return NULL; }
+static void _cffi_stop_error_capture(PyObject *ecap) { }
+
+#endif
diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h
--- a/lib_pypy/cffi/_cffi_include.h
+++ b/lib_pypy/cffi/_cffi_include.h
@@ -8,7 +8,7 @@
    the same works for the other two macros.  Py_DEBUG implies them,
    but not the other way around.
 */
-#ifndef _CFFI_USE_EMBEDDING
+#if !defined(_CFFI_USE_EMBEDDING) && !defined(Py_LIMITED_API)
 #  include <pyconfig.h>
 #  if !defined(Py_DEBUG) && !defined(Py_TRACE_REFS) && !defined(Py_REF_DEBUG)
 #    define Py_LIMITED_API
@@ -95,6 +95,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
@@ -159,9 +160,9 @@
 #define _cffi_from_c_struct                                              \
     ((PyObject *(*)(char *, struct _cffi_ctypedescr *))_cffi_exports[18])
 #define _cffi_to_c_wchar_t                                               \
-    ((wchar_t(*)(PyObject *))_cffi_exports[19])
+    ((_cffi_wchar_t(*)(PyObject *))_cffi_exports[19])
 #define _cffi_from_c_wchar_t                                             \
-    ((PyObject *(*)(wchar_t))_cffi_exports[20])
+    ((PyObject *(*)(_cffi_wchar_t))_cffi_exports[20])
 #define _cffi_to_c_long_double                                           \
     ((long double(*)(PyObject *))_cffi_exports[21])
 #define _cffi_to_c__Bool                                                 \
@@ -174,7 +175,11 @@
 #define _CFFI_CPIDX  25
 #define _cffi_call_python                                                \
     ((void(*)(struct _cffi_externpy_s *, char *))_cffi_exports[_CFFI_CPIDX])
-#define _CFFI_NUM_EXPORTS 26
+#define _cffi_to_c_wchar3216_t                                           \
+    ((int(*)(PyObject *))_cffi_exports[26])
+#define _cffi_from_c_wchar3216_t                                         \
+    ((PyObject *(*)(int))_cffi_exports[27])
+#define _CFFI_NUM_EXPORTS 28
 
 struct _cffi_ctypedescr;
 
@@ -215,6 +220,46 @@
     return NULL;
 }
 
+
+#ifdef HAVE_WCHAR_H
+typedef wchar_t _cffi_wchar_t;
+#else
+typedef uint16_t _cffi_wchar_t;   /* same random pick as _cffi_backend.c */
+#endif
+
+_CFFI_UNUSED_FN static uint16_t _cffi_to_c_char16_t(PyObject *o)
+{
+    if (sizeof(_cffi_wchar_t) == 2)
+        return (uint16_t)_cffi_to_c_wchar_t(o);
+    else
+        return (uint16_t)_cffi_to_c_wchar3216_t(o);
+}
+
+_CFFI_UNUSED_FN static PyObject *_cffi_from_c_char16_t(uint16_t x)
+{
+    if (sizeof(_cffi_wchar_t) == 2)
+        return _cffi_from_c_wchar_t(x);
+    else
+        return _cffi_from_c_wchar3216_t(x);
+}
+
+_CFFI_UNUSED_FN static int _cffi_to_c_char32_t(PyObject *o)
+{
+    if (sizeof(_cffi_wchar_t) == 4)
+        return (int)_cffi_to_c_wchar_t(o);
+    else
+        return (int)_cffi_to_c_wchar3216_t(o);
+}
+
+_CFFI_UNUSED_FN static PyObject *_cffi_from_c_char32_t(int x)
+{
+    if (sizeof(_cffi_wchar_t) == 4)
+        return _cffi_from_c_wchar_t(x);
+    else
+        return _cffi_from_c_wchar3216_t(x);
+}
+
+
 /**********  end CPython-specific section  **********/
 #else
 _CFFI_UNUSED_FN
diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h
--- a/lib_pypy/cffi/_embedding.h
+++ b/lib_pypy/cffi/_embedding.h
@@ -1,7 +1,12 @@
 
 /***** Support code for embedding *****/
 
-#if defined(_MSC_VER)
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+#if defined(_WIN32)
 #  define CFFI_DLLEXPORT  __declspec(dllexport)
 #elif defined(__GNUC__)
 #  define CFFI_DLLEXPORT  __attribute__((visibility("default")))
@@ -109,6 +114,8 @@
 /**********  CPython-specific section  **********/
 #ifndef PYPY_VERSION
 
+#include "_cffi_errors.h"
+
 
 #define _cffi_call_python_org  _cffi_exports[_CFFI_CPIDX]
 
@@ -220,8 +227,16 @@
         /* Print as much information as potentially useful.
            Debugging load-time failures with embedding is not fun
         */
+        PyObject *ecap;
         PyObject *exception, *v, *tb, *f, *modules, *mod;
         PyErr_Fetch(&exception, &v, &tb);
+        ecap = _cffi_start_error_capture();
+        f = PySys_GetObject((char *)"stderr");
+        if (f != NULL && f != Py_None) {
+            PyFile_WriteString(
+                "Failed to initialize the Python-CFFI embedding logic:\n\n", f);
+        }
+
         if (exception != NULL) {
             PyErr_NormalizeException(&exception, &v, &tb);
             PyErr_Display(exception, v, tb);
@@ -230,10 +245,9 @@
         Py_XDECREF(v);
         Py_XDECREF(tb);
 
-        f = PySys_GetObject((char *)"stderr");
         if (f != NULL && f != Py_None) {
             PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME
-                               "\ncompiled with cffi version: 1.10.0"
+                               "\ncompiled with cffi version: 1.11.0"
                                "\n_cffi_backend module: ", f);
             modules = PyImport_GetModuleDict();
             mod = PyDict_GetItemString(modules, "_cffi_backend");
@@ -249,6 +263,7 @@
             PyFile_WriteObject(PySys_GetObject((char *)"path"), f, 0);
             PyFile_WriteString("\n\n", f);
         }
+        _cffi_stop_error_capture(ecap);
     }
     result = -1;
     goto done;
@@ -515,3 +530,7 @@
 #undef cffi_compare_and_swap
 #undef cffi_write_barrier
 #undef cffi_read_barrier
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -75,9 +75,10 @@
         self._init_once_cache = {}
         self._cdef_version = None
         self._embedding = None
+        self._typecache = model.get_typecache(backend)
         if hasattr(backend, 'set_ffi'):
             backend.set_ffi(self)
-        for name in backend.__dict__:
+        for name in list(backend.__dict__):
             if name.startswith('RTLD_'):
                 setattr(self, name, getattr(backend, name))
         #
@@ -393,12 +394,17 @@

From pypy.commits at gmail.com  Thu Aug 24 05:57:00 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 24 Aug 2017 02:57:00 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: improve allocation choice
 for coalesced variables
Message-ID: <599ea2ec.4692df0a.f2949.d204@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92244:2e31b6e3f902
Date: 2017-08-24 10:19 +0200
http://bitbucket.org/pypy/pypy/changeset/2e31b6e3f902/

Log:	improve allocation choice for coalesced variables

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -482,7 +482,7 @@
                                     need_lower_byte=need_lower_byte)
         if loc:
             return loc
-        loc = self._spill_var(v, forbidden_vars, selected_reg,
+        loc = self._spill_var(forbidden_vars, selected_reg,
                               need_lower_byte=need_lower_byte)
         prev_loc = self.reg_bindings.get(v, None)
         if prev_loc is not None:
@@ -707,7 +707,7 @@
                              if reg not in self.save_around_call_regs]
             # chose which to spill using the usual spill heuristics
             while len(move_or_spill) > len(free_regs):
-                v = self._pick_variable_to_spill(None, [], vars=move_or_spill)
+                v = self._pick_variable_to_spill([], vars=move_or_spill)
                 self._bc_spill(v, new_free_regs)
                 move_or_spill.remove(v)
             assert len(move_or_spill) <= len(free_regs)
@@ -807,6 +807,11 @@
             assert op.numargs() == 1
             return [self.loc(op.getarg(0))]
 
+
+# ____________________________________________________________
+
+
+
 UNDEF_POS = -42
 
 class Lifetime(object):
@@ -834,6 +839,11 @@
         # the other lifetime will have this variable set to self.definition_pos
         self._definition_pos_shared = UNDEF_POS
 
+    def last_usage_including_sharing(self):
+        while self.share_with is not None:
+            self = self.share_with
+        return self.last_usage
+
     def is_last_real_use_before(self, position):
         if self.real_usages is None:
             return True
@@ -918,6 +928,9 @@
                     return index
         return sys.maxint
 
+    def __repr__(self):
+        return "%s: fixed at %s" % (self.register, self.index_lifetimes)
+
 
 class LifetimeManager(object):
     def __init__(self, longevity):
@@ -986,10 +999,10 @@
                 unfixed_reg = reg
                 continue
             use_after = fixed_reg_pos.free_until_pos(position)
-            if use_after < longevityvar.last_usage:
+            if use_after < longevityvar.last_usage_including_sharing():
                 # can't fit
                 continue
-            assert use_after >= longevityvar.last_usage
+            assert use_after >= longevityvar.last_usage_including_sharing()
             if use_after < min_fixed_use_after:
                 best_reg = reg
                 min_fixed_use_after = use_after
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -341,6 +341,21 @@
     # r1 is picked, because b4 fits before b0
     assert loc is r1
 
+def test_coalescing_non_fixed_regs():
+    b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+    l0 = Lifetime(0, 10)
+    l1 = Lifetime(10, 20)
+    l2 = Lifetime(25, 40)
+    l3 = Lifetime(15, 40)
+    longevity = LifetimeManager({b0: l0, b1: l1, b2: l2, b3: l3})
+    longevity.try_use_same_register(b0, b1)
+    longevity.fixed_register(35, r2, b2)
+    longevity.fixed_register(35, r3, b3)
+
+    loc = longevity.try_pick_free_reg(0, b0, [r1, r2, r3])
+    # r2 is picked, otherwise b1 can't end up in the same reg as b0
+    assert loc is r2
+
 
 def test_chained_coalescing():
     #              5 + b4
@@ -419,7 +434,7 @@
 
         class XRegisterManager(RegisterManager):
             no_lower_byte_regs = [r2, r3]
-        
+
         rm = XRegisterManager(longevity)
         rm.next_instruction()
         loc0 = rm.try_allocate_reg(b0, need_lower_byte=True)
@@ -454,7 +469,7 @@
 
         class XRegisterManager(RegisterManager):
             no_lower_byte_regs = [r2, r3]
-        
+
         rm = XRegisterManager(longevity,
                               frame_manager=fm,
                               assembler=MockAsm())
@@ -649,7 +664,7 @@
         rm.after_call(boxes[-1])
         assert len(rm.reg_bindings) == 1
         rm._check_invariants()
-        
+
 
     def test_different_frame_width(self):
         class XRegisterManager(RegisterManager):

From pypy.commits at gmail.com  Thu Aug 24 05:57:02 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 24 Aug 2017 02:57:02 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: cleanups
Message-ID: <599ea2ee.08e61c0a.e6ed1.7a78@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92245:f19f3e988faf
Date: 2017-08-24 10:20 +0200
http://bitbucket.org/pypy/pypy/changeset/f19f3e988faf/

Log:	cleanups

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -403,23 +403,18 @@
             self.free_regs.remove(loc)
             return loc
 
-    def _spill_var(self, v, forbidden_vars, selected_reg,
+    def _spill_var(self, forbidden_vars, selected_reg,
                    need_lower_byte=False):
-        v_to_spill = self._pick_variable_to_spill(v, forbidden_vars,
+        v_to_spill = self._pick_variable_to_spill(forbidden_vars,
                                selected_reg, need_lower_byte=need_lower_byte)
         loc = self.reg_bindings[v_to_spill]
+        self.assembler.num_spills += 1
+        self._sync_var(v_to_spill)
         del self.reg_bindings[v_to_spill]
-        self.assembler.num_spills += 1
-        if self.frame_manager.get(v_to_spill) is None:
-            newloc = self.frame_manager.loc(v_to_spill)
-            self.assembler.regalloc_mov(loc, newloc)
-        else:
-            self.assembler.num_spills_to_existing += 1
         return loc
 
-    def _pick_variable_to_spill(self, v, forbidden_vars, selected_reg=None,
+    def _pick_variable_to_spill(self, forbidden_vars, selected_reg=None,
                                 need_lower_byte=False, vars=None):
-        # YYY v is unused, remove
 
         # try to spill a variable that has no further real usages, ie that only
         # appears in failargs or in a jump
@@ -529,7 +524,7 @@
             if selected_reg in self.free_regs:
                 self.assembler.regalloc_mov(immloc, selected_reg)
                 return selected_reg
-            loc = self._spill_var(v, forbidden_vars, selected_reg)
+            loc = self._spill_var(forbidden_vars, selected_reg)
             self.free_regs.append(loc)
             self.assembler.regalloc_mov(immloc, loc)
             return loc
@@ -578,12 +573,8 @@
         self._check_type(result_v)
         self._check_type(v)
         if isinstance(v, Const):
-            if self.free_regs:
-                loc = self.free_regs.pop()
-            else:
-                loc = self._spill_var(v, forbidden_vars, None)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(self.convert_to_imm(v), loc)
-            self.reg_bindings[result_v] = loc
             return loc
         if v not in self.reg_bindings:
             # v not in a register. allocate one for result_v and move v there
@@ -605,11 +596,13 @@
         return loc
 
     def _sync_var(self, v):
+        self.assembler.num_spills += 1
         if not self.frame_manager.get(v):
-            self.assembler.num_moves_calls += 1
             reg = self.reg_bindings[v]
             to = self.frame_manager.loc(v)
             self.assembler.regalloc_mov(reg, to)
+        else:
+            self.assembler.num_spills_to_existing += 1
         # otherwise it's clean
 
     def _bc_spill(self, v, new_free_regs):
@@ -1102,6 +1095,7 @@
 
     return LifetimeManager(longevity)
 
+# YYY unused?
 def is_comparison_or_ovf_op(opnum):
     return rop.is_comparison(opnum) or rop.is_ovf(opnum)
 
diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -334,7 +334,7 @@
     longevity.fixed_register(35, r2, b2)
 
     loc = longevity.try_pick_free_reg(0, b3, [r1, r2])
-    # r2 is picked, otherwise b0 can't b0 can't end up in r1
+    # r2 is picked, otherwise b0 can't end up in r1
     assert loc is r2
 
     loc = longevity.try_pick_free_reg(0, b4, [r1, r2])
@@ -1303,7 +1303,6 @@
         i5 = escape_i()
         jump(i4, i5, descr=targettoken)
         '''
-        self.targettoken._fake_arglocs = [r5, r6]
         emitted = self.allocate(ops)
         assert emitted == [
             ('escape_i', r0, []),

From pypy.commits at gmail.com  Thu Aug 24 05:56:58 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 24 Aug 2017 02:56:58 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: oops
Message-ID: <599ea2ea.85961c0a.140bf.68d5@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92243:0600554dc7ef
Date: 2017-08-24 09:36 +0200
http://bitbucket.org/pypy/pypy/changeset/0600554dc7ef/

Log:	oops

diff --git a/pytest.ini b/pytest.ini
--- a/pytest.ini
+++ b/pytest.ini
@@ -0,0 +1,2 @@
+[pytest]
+addopts = --assert=reinterp -rf
diff --git a/rpython/pytest.ini b/rpython/pytest.ini
--- a/rpython/pytest.ini
+++ b/rpython/pytest.ini
@@ -0,0 +1,2 @@
+[pytest]
+addopts = --assert=reinterp -rf

From pypy.commits at gmail.com  Thu Aug 24 05:57:04 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 24 Aug 2017 02:57:04 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: rename argument
Message-ID: <599ea2f0.c6581c0a.5283d.668a@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92246:7a0afd3f4cdd
Date: 2017-08-24 11:08 +0200
http://bitbucket.org/pypy/pypy/changeset/7a0afd3f4cdd/

Log:	rename argument

diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -414,7 +414,7 @@
         return loc
 
     def _pick_variable_to_spill(self, forbidden_vars, selected_reg=None,
-                                need_lower_byte=False, vars=None):
+                                need_lower_byte=False, regs=None):
 
         # try to spill a variable that has no further real usages, ie that only
         # appears in failargs or in a jump
@@ -422,15 +422,15 @@
         # is the furthest away from the current position
 
         # YYY check for fixed variable usages
-        if vars is None:
-            vars = self.reg_bindings.keys()
+        if regs is None:
+            regs = self.reg_bindings.keys()
 
         cur_max_use_distance = -1
         position = self.position
         candidate = None
         cur_max_age_failargs = -1
         candidate_from_failargs = None
-        for next in vars:
+        for next in regs:
             reg = self.reg_bindings[next]
             if next in forbidden_vars:
                 continue
@@ -700,7 +700,7 @@
                              if reg not in self.save_around_call_regs]
             # chose which to spill using the usual spill heuristics
             while len(move_or_spill) > len(free_regs):
-                v = self._pick_variable_to_spill([], vars=move_or_spill)
+                v = self._pick_variable_to_spill([], regs=move_or_spill)
                 self._bc_spill(v, new_free_regs)
                 move_or_spill.remove(v)
             assert len(move_or_spill) <= len(free_regs)

From pypy.commits at gmail.com  Thu Aug 24 05:57:06 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Thu, 24 Aug 2017 02:57:06 -0700 (PDT)
Subject: [pypy-commit] pypy regalloc-playground: move tests for
 force_result_in_regs to their own class, since there are so
Message-ID: <599ea2f2.862e1c0a.a65c7.100b@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: regalloc-playground
Changeset: r92247:52e02755ad6a
Date: 2017-08-24 11:08 +0200
http://bitbucket.org/pypy/pypy/changeset/52e02755ad6a/

Log:	move tests for force_result_in_regs to their own class, since there
	are so many cases (and I am about to add more)

diff --git a/rpython/jit/backend/llsupport/test/test_regalloc.py b/rpython/jit/backend/llsupport/test/test_regalloc.py
--- a/rpython/jit/backend/llsupport/test/test_regalloc.py
+++ b/rpython/jit/backend/llsupport/test/test_regalloc.py
@@ -507,72 +507,6 @@
         assert isinstance(loc, FakeReg)
         rm._check_invariants()
 
-    def test_force_result_in_reg_1(self):
-        b0, b1 = newboxes(0, 0)
-        longevity = {b0: Lifetime(0, 1), b1: Lifetime(1, 3)}
-        fm = TFrameManager()
-        asm = MockAsm()
-        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
-        rm.next_instruction()
-        # first path, var is already in reg and dies
-        loc0 = rm.force_allocate_reg(b0)
-        rm._check_invariants()
-        rm.next_instruction()
-        loc = rm.force_result_in_reg(b1, b0)
-        assert loc is loc0
-        assert len(asm.moves) == 0
-        rm._check_invariants()
-
-    def test_force_result_in_reg_2(self):
-        b0, b1 = newboxes(0, 0)
-        longevity = {b0: Lifetime(0, 2), b1: Lifetime(1, 3)}
-        fm = TFrameManager()
-        asm = MockAsm()
-        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
-        rm.next_instruction()
-        loc0 = rm.force_allocate_reg(b0)
-        rm._check_invariants()
-        rm.next_instruction()
-        loc = rm.force_result_in_reg(b1, b0)
-        assert loc is loc0
-        assert rm.loc(b0) is not loc0
-        assert len(asm.moves) == 1
-        rm._check_invariants()
-
-    def test_force_result_in_reg_3(self):
-        b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
-        longevity = {b0: Lifetime(0, 2), b1: Lifetime(0, 2),
-                     b3: Lifetime(0, 2), b2: Lifetime(0, 2),
-                     b4: Lifetime(1, 3)}
-        fm = TFrameManager()
-        asm = MockAsm()
-        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
-        rm.next_instruction()
-        for b in b0, b1, b2, b3:
-            rm.force_allocate_reg(b)
-        assert not len(rm.free_regs)
-        rm._check_invariants()
-        rm.next_instruction()
-        rm.force_result_in_reg(b4, b0)
-        rm._check_invariants()
-        assert len(asm.moves) == 1
-
-    def test_force_result_in_reg_4(self):
-        b0, b1 = newboxes(0, 0)
-        longevity = {b0: Lifetime(0, 1), b1: Lifetime(0, 1)}
-        fm = TFrameManager()
-        asm = MockAsm()
-        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
-        rm.next_instruction()
-        fm.loc(b0)
-        rm.force_result_in_reg(b1, b0)
-        rm._check_invariants()
-        loc = rm.loc(b1)
-        assert isinstance(loc, FakeReg)
-        loc = rm.loc(b0)
-        assert isinstance(loc, FakeFramePos)
-        assert len(asm.moves) == 1
-
     def test_bogus_make_sure_var_in_reg(self):
         b0, = newboxes(0)
         longevity = {b0: Lifetime(0, 1)}
@@ -602,17 +536,6 @@
         assert len(rm.reg_bindings) == 4
         rm._check_invariants()
 
-    def test_force_result_in_reg_const(self):
-        boxes, longevity = boxes_and_longevity(2)
-        fm = TFrameManager()
-        asm = MockAsm()
-        rm = RegisterManager(longevity, frame_manager=fm,
-                             assembler=asm)
-        rm.next_instruction()
-        c = ConstInt(0)
-        rm.force_result_in_reg(boxes[0], c)
-        rm._check_invariants()
-
     def test_loc_of_const(self):
         rm = RegisterManager({})
         rm.next_instruction()
@@ -935,6 +858,87 @@
         for box in fm.bindings.keys():
             fm.mark_as_free(box)
 
+
+class TestForceResultInReg(object):
+    # use it's own class since there are so many cases
+
+    def test_force_result_in_reg_1(self):
+        b0, b1 = newboxes(0, 0)
+        longevity = {b0: Lifetime(0, 1), b1: Lifetime(1, 3)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        # first path, var is already in reg and dies
+        loc0 = rm.force_allocate_reg(b0)
+        rm._check_invariants()
+        rm.next_instruction()
+        loc = rm.force_result_in_reg(b1, b0)
+        assert loc is loc0
+        assert len(asm.moves) == 0
+        rm._check_invariants()
+
+    def test_force_result_in_reg_2(self):
+        b0, b1 = newboxes(0, 0)
+        longevity = {b0: Lifetime(0, 2), b1: Lifetime(1, 3)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        loc0 = rm.force_allocate_reg(b0)
+        rm._check_invariants()
+        rm.next_instruction()
+        loc = rm.force_result_in_reg(b1, b0)
+        assert loc is loc0
+        assert rm.loc(b0) is not loc0
+        assert len(asm.moves) == 1
+        rm._check_invariants()
+
+    def test_force_result_in_reg_3(self):
+        b0, b1, b2, b3, b4 = newboxes(0, 0, 0, 0, 0)
+        longevity = {b0: Lifetime(0, 2), b1: Lifetime(0, 2),
+                     b3: Lifetime(0, 2), b2: Lifetime(0, 2),
+                     b4: Lifetime(1, 3)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        for b in b0, b1, b2, b3:
+            rm.force_allocate_reg(b)
+        assert not len(rm.free_regs)
+        rm._check_invariants()
+        rm.next_instruction()
+        rm.force_result_in_reg(b4, b0)
+        rm._check_invariants()
+        assert len(asm.moves) == 1
+
+    def test_force_result_in_reg_4(self):
+        b0, b1 = newboxes(0, 0)
+        longevity = {b0: Lifetime(0, 1), b1: Lifetime(0, 1)}
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm, assembler=asm)
+        rm.next_instruction()
+        fm.loc(b0)
+        rm.force_result_in_reg(b1, b0)
+        rm._check_invariants()
+        loc = rm.loc(b1)
+        assert isinstance(loc, FakeReg)
+        loc = rm.loc(b0)
+        assert isinstance(loc, FakeFramePos)
+        assert len(asm.moves) == 1
+
+    def test_force_result_in_reg_const(self):
+        boxes, longevity = boxes_and_longevity(2)
+        fm = TFrameManager()
+        asm = MockAsm()
+        rm = RegisterManager(longevity, frame_manager=fm,
+                             assembler=asm)
+        rm.next_instruction()
+        c = ConstInt(0)
+        rm.force_result_in_reg(boxes[0], c)
+        rm._check_invariants()
+
 # _____________________________________________________
 # tests that assign registers in a mocked way for a fake CPU
 
@@ -1243,7 +1247,6 @@
         ]
 
     def test_coalescing_first_var_already_in_different_reg(self):
-        py.test.skip("messy - later")
         ops = '''
         [i0]
         i2 = int_mul(i0, 2)

From pypy.commits at gmail.com  Thu Aug 24 08:40:00 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 05:40:00 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (arigo, fijal climbing)
Message-ID: <599ec920.e8aedf0a.28f79.7ee0@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92248:bf0d9ddd4a6e
Date: 2017-08-24 14:39 +0200
http://bitbucket.org/pypy/pypy/changeset/bf0d9ddd4a6e/

Log:	(arigo, fijal climbing)

	Clean up rutf8.py

diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py
--- a/rpython/rlib/rutf8.py
+++ b/rpython/rlib/rutf8.py
@@ -1,60 +1,70 @@
+""" This file is about supporting unicode strings in RPython,
+represented by a byte string that is exactly the UTF-8 version
+(for some definition of UTF-8).
 
+This doesn't support Python 2's unicode characters beyond 0x10ffff,
+which are theoretically possible to obtain using strange tricks like
+the array or ctypes modules.
+
+Fun comes from surrogates.  Various functions don't normally accept
+any unicode character betwen 0xd800 and 0xdfff, but do if you give
+the 'allow_surrogates = True' flag.
+"""
+
+from rpython.rlib.objectmodel import enforceargs
 from rpython.rlib.rstring import StringBuilder
-from rpython.rlib import runicode, jit
+from rpython.rlib import jit
+from rpython.rlib.rarithmetic import r_uint
 
-def unichr_as_utf8(code):
-    """ Encode code (numeric value) as utf8 encoded string
+
+def unichr_as_utf8(code, allow_surrogates=False):
+    """Encode code (numeric value) as utf8 encoded string
     """
-    if code < 0:
-        raise ValueError
-    lgt = 1
-    if code >= runicode.MAXUNICODE:
-        lgt = 2
-    if code < 0x80:
+    code = r_uint(code)
+    if code <= r_uint(0x7F):
         # Encode ASCII
-        return chr(code), 1
-    if code < 0x0800:
-        # Encode Latin-1
-        return chr((0xc0 | (code >> 6))) + chr((0x80 | (code & 0x3f))), lgt
-    if code < 0x10000:
+        return chr(code)
+    if code <= r_uint(0x07FF):
+        return chr((0xc0 | (code >> 6))) + chr((0x80 | (code & 0x3f)))
+    if code <= r_uint(0xFFFF):
+        if not allow_surrogates and 0xD800 <= code <= 0xDfff:
+            raise ValueError
         return (chr((0xe0 | (code >> 12))) +
                 chr((0x80 | ((code >> 6) & 0x3f))) +
-                chr((0x80 | (code & 0x3f)))), lgt
-    if code < 0x10ffff:
+                chr((0x80 | (code & 0x3f))))
+    if code <= r_uint(0x10FFFF):
         return (chr((0xf0 | (code >> 18))) +
                 chr((0x80 | ((code >> 12) & 0x3f))) +
                 chr((0x80 | ((code >> 6) & 0x3f))) +
-                chr((0x80 | (code & 0x3f)))), lgt
+                chr((0x80 | (code & 0x3f))))
     raise ValueError
 
-def unichr_as_utf8_append(builder, code):
-    """ Encode code (numeric value) as utf8 encoded string
+def unichr_as_utf8_append(builder, code, allow_surrogates=False):
+    """Encode code (numeric value) as utf8 encoded string
+    and emit the result into the given StringBuilder.
     """
-    if code < 0:
-        raise ValueError
-    lgt = 1
-    if code >= runicode.MAXUNICODE:
-        lgt = 2
-    if code < 0x80:
+    code = r_uint(code)
+    if code <= r_uint(0x7F):
         # Encode ASCII
         builder.append(chr(code))
-        return 1
-    if code < 0x0800:
-        # Encode Latin-1
+        return
+    if code <= r_uint(0x07FF):
         builder.append(chr((0xc0 | (code >> 6))))
         builder.append(chr((0x80 | (code & 0x3f))))
-        return lgt
-    if code < 0x10000:
+        return
+    if code <= r_uint(0xFFFF):
+        if not allow_surrogates and 0xd800 <= code <= 0xdfff:
+            raise ValueError
         builder.append(chr((0xe0 | (code >> 12))))
         builder.append(chr((0x80 | ((code >> 6) & 0x3f))))
         builder.append(chr((0x80 | (code & 0x3f))))
-        return lgt
-    if code < 0x10ffff:
+        return
+    if code <= r_uint(0x10FFFF):
         builder.append(chr((0xf0 | (code >> 18))))
         builder.append(chr((0x80 | ((code >> 12) & 0x3f))))
         builder.append(chr((0x80 | ((code >> 6) & 0x3f))))
         builder.append(chr((0x80 | (code & 0x3f))))
-        return lgt
+        return
     raise ValueError
 
 # note - table lookups are really slow. Measured on various elements of obama
@@ -62,61 +72,64 @@
 #        In extreme cases (small, only chinese text), they're 40% slower
 
 def next_codepoint_pos(code, pos):
-    """ Gives the position of the next codepoint after pos, -1
-    if it's the last one (assumes valid utf8)
+    """Gives the position of the next codepoint after pos.
+    Assumes valid utf8.  'pos' must be before the end of the string.
     """
     chr1 = ord(code[pos])
-    if chr1 < 0x80:
+    if chr1 <= 0x7F:
         return pos + 1
-    if 0xC2 <= chr1 <= 0xDF:
+    if chr1 <= 0xDF:
         return pos + 2
-    if chr1 >= 0xE0 and chr1 <= 0xEF:
+    if chr1 <= 0xEF:
         return pos + 3
     return pos + 4
 
 def prev_codepoint_pos(code, pos):
-    """ Gives the position of the previous codepoint
+    """Gives the position of the previous codepoint.
+    'pos' must not be zero.
     """
     pos -= 1
     chr1 = ord(code[pos])
-    if chr1 < 0x80:
+    if chr1 <= 0x7F:
         return pos
-    while ord(code[pos]) & 0xC0 == 0x80:
-        pos -= 1
+    pos -= 1
+    if ord(code[pos]) >= 0xC0:
+        return pos
+    pos -= 1
+    if ord(code[pos]) >= 0xC0:
+        return pos
+    pos -= 1
     return pos
 
 def compute_length_utf8(s):
-    pos = 0
-    lgt = 0
-    while pos < len(s):
-        pos = next_codepoint_pos(s, pos)
-        lgt += 1
-    return lgt
+    continuation_bytes = 0
+    for i in range(len(s)):
+        if 0x80 <= ord(s[i]) <= 0xBF:    # count the continuation bytes
+            continuation_bytes += 1
+    return len(s) - continuation_bytes
 
 def codepoint_at_pos(code, pos):
     """ Give a codepoint in code at pos - assumes valid utf8, no checking!
     """
     ordch1 = ord(code[pos])
-    if ordch1 < 0x80:
+    if ordch1 <= 0x7F:
         return ordch1
 
-    n = ord(runicode._utf8_code_length[ordch1 - 0x80])
-    if n == 2:
-        ordch2 = ord(code[pos+1])
+    ordch2 = ord(code[pos+1])
+    if ordch1 <= 0xDF:
         # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz
         return (((ordch1 & 0x1F) << 6) +    # 0b00011111
                  (ordch2 & 0x3F))           # 0b00111111
-    elif n == 3:
-        ordch2 = ord(code[pos+1])
-        ordch3 = ord(code[pos+2])
+
+    ordch3 = ord(code[pos+2])
+    if ordch1 <= 0xEF:
         # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz
         return (((ordch1 & 0x0F) << 12) +     # 0b00001111
                 ((ordch2 & 0x3F) << 6) +      # 0b00111111
                 (ordch3 & 0x3F))              # 0b00111111
-    elif n == 4:
-        ordch2 = ord(code[pos+1])
-        ordch3 = ord(code[pos+2])
-        ordch4 = ord(code[pos+3])
+
+    ordch4 = ord(code[pos+3])
+    if True:
         # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz
         return (((ordch1 & 0x07) << 18) +      # 0b00000111
                 ((ordch2 & 0x3F) << 12) +      # 0b00111111
@@ -124,46 +137,44 @@
                 (ordch4 & 0x3F))               # 0b00111111
     assert False, "unreachable"
 
-class AsciiCheckError(Exception):
-    def __init__(self, pos):
-        self.pos = pos
+class CheckError(Exception):
+    pass
 
-def check_ascii(s, size=-1):
-    if size == -1:
-        size = len(s)
-    for i in range(0, size):
-        if ord(s[i]) & 0x80:
-            raise AsciiCheckError(i)
+ at jit.elidable
+def check_ascii(s):
+    for i in range(len(s)):
+        if ord(s[i]) > 0x7F:
+            raise CheckError
 
-def utf8_encode_ascii(s, errors, encoding, msg, errorhandler):
-    res = StringBuilder(len(s))
-    u_pos = 0
-    pos = 0
-    while pos < len(s):
-        chr1 = s[pos]
-        if ord(chr1) < 0x80:
-            res.append(chr1)
-        else:
-            repl, _, _, _ = errorhandler(errors, encoding, msg, s, u_pos, u_pos + 1)
-            res.append(repl)
-        u_pos += 1
-        pos = next_codepoint_pos(s, pos)
-    return res.build()
+#def utf8_encode_ascii(s, errors, encoding, msg, errorhandler):
+#    res = StringBuilder(len(s))
+#    u_pos = 0
+#    pos = 0
+#    while pos < len(s):
+#        chr1 = s[pos]
+#        if ord(chr1) < 0x80:
+#            res.append(chr1)
+#        else:
+#            repl, _, _, _ = errorhandler(errors, encoding, msg, s, u_pos, u_pos + 1)
+#            res.append(repl)
+#        u_pos += 1
+#        pos = next_codepoint_pos(s, pos)
+#    return res.build()
 
-def str_decode_ascii(s, size, errors, errorhandler):
-    # ASCII is equivalent to the first 128 ordinals in Unicode.
-    result = StringBuilder(size)
-    pos = 0
-    while pos < size:
-        c = s[pos]
-        if ord(c) < 128:
-            result.append(c)
-        else:
-            r, _, _ = errorhandler(errors, "ascii", "ordinal not in range(128)",
-                                   s,  pos, pos + 1)
-            result.append(r)
-        pos += 1
-    return result.build(), pos, -1
+#def str_decode_ascii(s, size, errors, errorhandler):
+#    # ASCII is equivalent to the first 128 ordinals in Unicode.
+#    result = StringBuilder(size)
+#    pos = 0
+#    while pos < size:
+#        c = s[pos]
+#        if ord(c) < 128:
+#            result.append(c)
+#        else:
+#            r, _, _ = errorhandler(errors, "ascii", "ordinal not in range(128)",
+#                                   s,  pos, pos + 1)
+#            result.append(r)
+#        pos += 1
+#    return result.build(), pos, -1
 
 def islinebreak(s, pos):
     chr1 = ord(s[pos])
@@ -217,149 +228,92 @@
         return True
     return False
 
-def utf8_in_chars(value, pos, chars):
-    """ equivalent of u'x' in u'xyz', just done in utf8
-    """
-    lgt = next_codepoint_pos(value, pos) - pos
-    i = 0
-    while i < len(chars):
-        j = next_codepoint_pos(chars, i)
-        if j - i != lgt:
-            i = j
-            continue
-        for k in range(lgt):
-            if value[k + pos] != chars[i + k]:
-                break
-        else:
-            return True
-        i = j
-    return False
 
-class Utf8CheckError(Exception):
-    def __init__(self, msg, startpos, endpos):
-        self.msg = msg
-        self.startpos = startpos
-        self.endpos = endpos
+def _invalid_cont_byte(ordch):
+    return ordch>>6 != 0x2    # 0b10
+
+_invalid_byte_2_of_2 = _invalid_cont_byte
+_invalid_byte_3_of_3 = _invalid_cont_byte
+_invalid_byte_3_of_4 = _invalid_cont_byte
+_invalid_byte_4_of_4 = _invalid_cont_byte
+
+ at enforceargs(allow_surrogates=bool)
+def _invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates):
+    return (ordch2>>6 != 0x2 or    # 0b10
+            (ordch1 == 0xe0 and ordch2 < 0xa0)
+            # surrogates shouldn't be valid UTF-8!
+            or (ordch1 == 0xed and ordch2 > 0x9f and not allow_surrogates))
+
+def _invalid_byte_2_of_4(ordch1, ordch2):
+    return (ordch2>>6 != 0x2 or    # 0b10
+            (ordch1 == 0xf0 and ordch2 < 0x90) or
+            (ordch1 == 0xf4 and ordch2 > 0x8f))
+
 
 @jit.elidable
-def str_check_utf8(s, size, final=False,
-                   allow_surrogates=runicode.allow_surrogate_by_default):
-    """ A simplified version of utf8 encoder - it only works with 'strict'
-    error handling.
+def check_utf8(s, allow_surrogates=False):
+    """Check that 's' is a utf-8-encoded byte string.
+    Returns the length (number of chars) or raise CheckError.
+    Note that surrogates are not handled specially here.
     """
-    # XXX do the following in a cleaner way, e.g. via signature
-    # NB. a bit messy because rtyper/rstr.py also calls the same
-    # function.  Make sure we annotate for the args it passes, too
-    #if NonConstant(False):
-    #    s = NonConstant('?????')
-    #    size = NonConstant(12345)
-    #    errors = NonConstant('strict')
-    #    final = NonConstant(True)
-    #    errorhandler = ll_unicode_error_decode
-    #    allow_surrogates = NonConstant(True)
-    if size == 0:
-        return 0, 0
-
     pos = 0
-    lgt = 0
-    while pos < size:
+    continuation_bytes = 0
+    while pos < len(s):
         ordch1 = ord(s[pos])
+        pos += 1
         # fast path for ASCII
-        # XXX maybe use a while loop here
-        if ordch1 < 0x80:
-            lgt += 1
-            pos += 1
+        if ordch1 <= 0x7F:
             continue
 
-        n = ord(runicode._utf8_code_length[ordch1 - 0x80])
-        if pos + n > size:
-            if not final:
-                break
-            # argh, this obscure block of code is mostly a copy of
-            # what follows :-(
-            charsleft = size - pos - 1 # either 0, 1, 2
-            # note: when we get the 'unexpected end of data' we need
-            # to care about the pos returned; it can be lower than size,
-            # in case we need to continue running this loop
-            if not charsleft:
-                # there's only the start byte and nothing else
-                raise Utf8CheckError('unexpected end of data', pos, pos + 1)
-            ordch2 = ord(s[pos+1])
-            if n == 3:
-                # 3-bytes seq with only a continuation byte
-                if runicode._invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates):
-                    # second byte invalid, take the first and continue
-                    raise Utf8CheckError('invalid continuation byte', pos,
-                                         pos + 1)
-                else:
-                    # second byte valid, but third byte missing
-                    raise Utf8CheckError('unexpected end of data', pos, pos + 2)
-            elif n == 4:
-                # 4-bytes seq with 1 or 2 continuation bytes
-                if runicode._invalid_byte_2_of_4(ordch1, ordch2):
-                    # second byte invalid, take the first and continue
-                    raise Utf8CheckError('invalid continuation byte', pos,
-                                         pos + 1)
-                elif charsleft == 2 and runicode._invalid_byte_3_of_4(ord(s[pos+2])):
-                    # third byte invalid, take the first two and continue
-                    raise Utf8CheckError('invalid continuation byte', pos,
-                                         pos + 2)
-                else:
-                    # there's only 1 or 2 valid cb, but the others are missing
-                    raise Utf8CheckError('unexpected end of data', pos,
-                                         pos + charsleft + 1)
-            raise AssertionError("unreachable")
+        if ordch1 <= 0xC1:
+            raise CheckError
 
-        if n == 0:
-            raise Utf8CheckError('invalid start byte', pos, pos + 1)
-        elif n == 1:
-            assert 0, "ascii should have gone through the fast path"
+        if ordch1 <= 0xDF:
+            continuation_bytes += 1
+            if pos >= len(s):
+                raise CheckError
+            ordch2 = ord(s[pos])
+            pos += 1
 
-        elif n == 2:
-            ordch2 = ord(s[pos+1])
-            if runicode._invalid_byte_2_of_2(ordch2):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 2)
+            if _invalid_byte_2_of_2(ordch2):
+                raise CheckError
             # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz
-            lgt += 1
+            continue
+
+        if ordch1 <= 0xEF:
+            continuation_bytes += 2
+            if (pos + 2) > len(s):
+                raise CheckError
+            ordch2 = ord(s[pos])
+            ordch3 = ord(s[pos + 1])
             pos += 2
 
-        elif n == 3:
-            ordch2 = ord(s[pos+1])
-            ordch3 = ord(s[pos+2])
-            if runicode._invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 1)
-            elif runicode._invalid_byte_3_of_3(ordch3):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 2)
+            if _invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates):
+                raise CheckError
+            elif _invalid_byte_3_of_3(ordch3):
+                raise CheckError
             # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz
-            lgt += 1
+            continue
+
+        if ordch1 <= 0xF4:
+            continuation_bytes += 3
+            if (pos + 3) > len(s):
+                raise CheckError
+            ordch2 = ord(s[pos])
+            ordch3 = ord(s[pos + 1])
+            ordch4 = ord(s[pos + 2])
             pos += 3
 
-        elif n == 4:
-            ordch2 = ord(s[pos+1])
-            ordch3 = ord(s[pos+2])
-            ordch4 = ord(s[pos+3])
-            if runicode._invalid_byte_2_of_4(ordch1, ordch2):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 1)
-            elif runicode._invalid_byte_3_of_4(ordch3):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 2)
-            elif runicode._invalid_byte_4_of_4(ordch4):
-                raise Utf8CheckError('invalid continuation byte', pos,
-                                     pos + 3)
+            if _invalid_byte_2_of_4(ordch1, ordch2):
+                raise CheckError
+            elif _invalid_byte_3_of_4(ordch3):
+                raise CheckError
+            elif _invalid_byte_4_of_4(ordch4):
+                raise CheckError
             # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz
-            c = (((ordch1 & 0x07) << 18) +      # 0b00000111
-                 ((ordch2 & 0x3F) << 12) +      # 0b00111111
-                 ((ordch3 & 0x3F) << 6) +       # 0b00111111
-                 (ordch4 & 0x3F))               # 0b00111111
-            if c <= runicode.MAXUNICODE:
-                lgt += 1
-            else:
-                # append the two surrogates:
-                lgt += 2
-            pos += 4
+            continue
 
-    return pos, lgt
+        raise CheckError
+
+    assert pos == len(s)
+    return pos - continuation_bytes
diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py
--- a/rpython/rlib/test/test_rutf8.py
+++ b/rpython/rlib/test/test_rutf8.py
@@ -1,14 +1,18 @@
-
+import py
 import sys
 from hypothesis import given, strategies, settings, example
 
 from rpython.rlib import rutf8, runicode
 
- at given(strategies.integers(min_value=0, max_value=runicode.MAXUNICODE))
-def test_unichr_as_utf8(i):
-    u, lgt = rutf8.unichr_as_utf8(i)
-    r = runicode.UNICHR(i)
-    assert u == r.encode('utf8')
+
+ at given(strategies.characters(), strategies.booleans())
+def test_unichr_as_utf8(c, allow_surrogates):
+    i = ord(c)
+    if not allow_surrogates and 0xD800 <= i <= 0xDFFF:
+        py.test.raises(ValueError, rutf8.unichr_as_utf8, i, allow_surrogates)
+    else:
+        u = rutf8.unichr_as_utf8(i, allow_surrogates)
+        assert u == c.encode('utf8')
 
 @given(strategies.binary())
 def test_check_ascii(s):
@@ -19,28 +23,32 @@
         raised = True
     try:
         rutf8.check_ascii(s)
-    except rutf8.AsciiCheckError as a:
+    except rutf8.CheckError:
         assert raised
-        assert a.pos == e.start
     else:
         assert not raised
 
- at given(strategies.binary())
-def test_str_check_utf8(s):
+ at given(strategies.binary(), strategies.booleans())
+def test_check_utf8(s, allow_surrogates):
+    _test_check_utf8(s, allow_surrogates)
+
+ at given(strategies.text(), strategies.booleans())
+def test_check_utf8_valid(u, allow_surrogates):
+    _test_check_utf8(u.encode('utf-8'), allow_surrogates)
+
+def _test_check_utf8(s, allow_surrogates):
     try:
-        u, _ = runicode.str_decode_utf_8(s, len(s), None, final=True)
+        u, _ = runicode.str_decode_utf_8(s, len(s), None, final=True,
+                                         allow_surrogates=allow_surrogates)
         valid = True
     except UnicodeDecodeError as e:
         valid = False
     try:
-        consumed, length = rutf8.str_check_utf8(s, len(s), final=True)
-    except rutf8.Utf8CheckError as a:
+        length = rutf8.check_utf8(s, allow_surrogates)
+    except rutf8.CheckError:
         assert not valid
-        assert a.startpos == e.start
-        # assert a.end == e.end, ideally
     else:
         assert valid
-        assert consumed == len(s)
         assert length == len(u)
 
 @given(strategies.characters())
@@ -80,5 +88,5 @@
         response = True
     else:
         response = False
-    r = rutf8.utf8_in_chars(unichr(i).encode('utf8'), 0, uni.encode('utf8'))
+    r = unichr(i).encode('utf8') in uni.encode('utf8')
     assert r == response

From pypy.commits at gmail.com  Thu Aug 24 08:50:48 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 05:50:48 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599ecba8.db85df0a.c853d.e306@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92249:c9a84142d1e3
Date: 2017-08-24 14:50 +0200
http://bitbucket.org/pypy/pypy/changeset/c9a84142d1e3/

Log:	(fijal, arigo)

	Fix the gateway logic: we can now pass 'utf8' to get just a
	utf-8-encoded string

diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -160,6 +160,9 @@
     def visit_text0(self, el, app_sig):
         self.checked_space_method(el, app_sig)
 
+    def visit_utf8(self, el, app_sig):
+        self.checked_space_method(el, app_sig)
+
     def visit_fsencode(self, el, app_sig):
         self.checked_space_method(el, app_sig)
 
@@ -244,7 +247,6 @@
     def __init__(self):
         UnwrapSpecEmit.__init__(self)
         self.run_args = []
-        self.extracode = []
 
     def scopenext(self):
         return "scope_w[%d]" % self.succ()
@@ -305,6 +307,9 @@
     def visit_text0(self, typ):
         self.run_args.append("space.text0_w(%s)" % (self.scopenext(),))
 
+    def visit_utf8(self, typ):
+        self.run_args.append("space.utf8_w(%s)" % (self.scopenext(),))
+
     def visit_fsencode(self, typ):
         self.run_args.append("space.fsencode_w(%s)" % (self.scopenext(),))
 
@@ -359,9 +364,8 @@
             d = {}
             source = """if 1:
                 def _run(self, space, scope_w):
-                    %s
                     return self.behavior(%s)
-                \n""" % ("\n".join(self.extracode), ', '.join(self.run_args))
+                \n""" % (', '.join(self.run_args),)
             exec compile2(source) in self.miniglobals, d
 
             activation_cls = type("BuiltinActivation_UwS_%s" % label,
@@ -402,7 +406,6 @@
         UnwrapSpecEmit.__init__(self)
         self.args = []
         self.unwrap = []
-        self.extracode = []
         self.finger = 0
 
     def dispatch(self, el, *args):
@@ -472,6 +475,9 @@
     def visit_text0(self, typ):
         self.unwrap.append("space.text0_w(%s)" % (self.nextarg(),))
 
+    def visit_utf8(self, typ):
+        self.unwrap.append("space.utf8_w(%s)" % (self.nextarg(),))
+
     def visit_fsencode(self, typ):
         self.unwrap.append("space.fsencode_w(%s)" % (self.nextarg(),))
 
@@ -526,10 +532,9 @@
             unwrap_info.miniglobals['func'] = func
             source = """if 1:
                 def fastfunc_%s_%d(%s):
-                    %s
                     return func(%s)
                 \n""" % (func.__name__.replace('-', '_'), narg,
-                         ', '.join(args), '\n'.join(unwrap_info.extracode),
+                         ', '.join(args),
                          ', '.join(unwrap_info.unwrap))
             exec compile2(source) in unwrap_info.miniglobals, d
             fastfunc = d['fastfunc_%s_%d' % (func.__name__.replace('-', '_'), narg)]
diff --git a/pypy/interpreter/test/test_gateway.py b/pypy/interpreter/test/test_gateway.py
--- a/pypy/interpreter/test/test_gateway.py
+++ b/pypy/interpreter/test/test_gateway.py
@@ -538,8 +538,8 @@
     def test_interp2app_unwrap_spec_utf8(self):
         space = self.space
         w = space.wrap
-        def g3_u(space, utf8, utf8len):
-            return space.newtuple([space.wrap(len(utf8)), space.wrap(utf8len)])
+        def g3_u(space, utf8):
+            return space.wrap(utf8)
         app_g3_u = gateway.interp2app_temp(g3_u,
                                          unwrap_spec=[gateway.ObjSpace,
                                                       'utf8'])
@@ -547,14 +547,20 @@
         encoded = u"g&#281;&#347;&#263;".encode('utf8')
         assert self.space.eq_w(
             space.call_function(w_app_g3_u, w(u"g&#281;&#347;&#263;")),
-            space.newtuple([w(len(encoded)), w(4)]))
+            w(encoded))
         assert self.space.eq_w(
             space.call_function(w_app_g3_u, w("foo")),
-            space.newtuple([w(3), w(3)]))
+            w("foo"))
         raises(gateway.OperationError, space.call_function, w_app_g3_u,
                w(None))
         raises(gateway.OperationError, space.call_function, w_app_g3_u,
                w(42))
+        w_ascii = space.appexec([], """():
+            import sys
+            return sys.getdefaultencoding() == 'ascii'""")
+        if space.is_true(w_ascii):
+            raises(gateway.OperationError, space.call_function, w_app_g3_u,
+                   w("\x80"))
 
     def test_interp2app_unwrap_spec_unwrapper(self):
         space = self.space
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -374,10 +374,10 @@
 
 def make_encoder_wrapper(name):
     rname = "utf8_encode_%s" % (name.replace("_encode", ""), )
-    XXX
     @unwrap_spec(utf8='utf8', errors='text_or_none')
     def wrap_encoder(space, utf8, utf8len, errors="strict"):
         from pypy.interpreter import unicodehelper
+        XXX
 
         if errors is None:
             errors = 'strict'
diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -831,7 +831,8 @@
             s = space.charbuf_w(w_obj)
             try:
                 rutf8.check_ascii(s)
-            except rutf8.AsciiCheckError as e:
+            except rutf8.CheckError:
+                XXX
                 unicodehelper.decode_error_handler(space)(None,
                     'ascii', "ordinal not in range(128)", s, e.pos, e.pos+1)
                 assert False
@@ -842,7 +843,8 @@
             try:
                 _, lgt = rutf8.str_check_utf8(s, len(s), final=True,
                                               allow_surrogates=True)
-            except rutf8.Utf8CheckError as e:
+            except rutf8.CheckError:
+                XXX
                 eh(None, 'utf8', e.msg, s, e.startpos, e.endpos)
                 assert False, "has to raise"
             return space.newutf8(s, lgt)

From pypy.commits at gmail.com  Thu Aug 24 08:55:20 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 05:55:20 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Fix
Message-ID: <599eccb8.93b5df0a.b63c6.d03a@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92250:86b689eb4f9f
Date: 2017-08-24 14:54 +0200
http://bitbucket.org/pypy/pypy/changeset/86b689eb4f9f/

Log:	Fix

diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -831,8 +831,7 @@
             s = space.charbuf_w(w_obj)
             try:
                 rutf8.check_ascii(s)
-            except rutf8.CheckError:
-                XXX
+            except rutf8.CheckError as e:
                 unicodehelper.decode_error_handler(space)(None,
                     'ascii', "ordinal not in range(128)", s, e.pos, e.pos+1)
                 assert False
diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py
--- a/rpython/rlib/rutf8.py
+++ b/rpython/rlib/rutf8.py
@@ -9,6 +9,10 @@
 Fun comes from surrogates.  Various functions don't normally accept
 any unicode character betwen 0xd800 and 0xdfff, but do if you give
 the 'allow_surrogates = True' flag.
+
+This is a minimal reference implementation.  A lot of interpreters
+need their own copy-pasted copy of some of the logic here, with
+extra code in the middle for error handlers and so on.
 """
 
 from rpython.rlib.objectmodel import enforceargs
@@ -138,43 +142,14 @@
     assert False, "unreachable"
 
 class CheckError(Exception):
-    pass
+    def __init__(self, pos):
+        self.pos = pos
 
 @jit.elidable
 def check_ascii(s):
     for i in range(len(s)):
         if ord(s[i]) > 0x7F:
-            raise CheckError
-
-#def utf8_encode_ascii(s, errors, encoding, msg, errorhandler):
-#    res = StringBuilder(len(s))
-#    u_pos = 0
-#    pos = 0
-#    while pos < len(s):
-#        chr1 = s[pos]
-#        if ord(chr1) < 0x80:
-#            res.append(chr1)
-#        else:
-#            repl, _, _, _ = errorhandler(errors, encoding, msg, s, u_pos, u_pos + 1)
-#            res.append(repl)
-#        u_pos += 1
-#        pos = next_codepoint_pos(s, pos)
-#    return res.build()
-
-#def str_decode_ascii(s, size, errors, errorhandler):
-#    # ASCII is equivalent to the first 128 ordinals in Unicode.
-#    result = StringBuilder(size)
-#    pos = 0
-#    while pos < size:
-#        c = s[pos]
-#        if ord(c) < 128:
-#            result.append(c)
-#        else:
-#            r, _, _ = errorhandler(errors, "ascii", "ordinal not in range(128)",
-#                                   s,  pos, pos + 1)
-#            result.append(r)
-#        pos += 1
-#    return result.build(), pos, -1
+            raise CheckError(i)
 
 def islinebreak(s, pos):
     chr1 = ord(s[pos])
@@ -266,54 +241,51 @@
             continue
 
         if ordch1 <= 0xC1:
-            raise CheckError
+            raise CheckError(pos - 1)
 
         if ordch1 <= 0xDF:
-            continuation_bytes += 1
             if pos >= len(s):
-                raise CheckError
+                raise CheckError(pos - 1)
             ordch2 = ord(s[pos])
             pos += 1
 
             if _invalid_byte_2_of_2(ordch2):
-                raise CheckError
+                raise CheckError(pos - 2)
             # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz
+            continuation_bytes += 1
             continue
 
         if ordch1 <= 0xEF:
-            continuation_bytes += 2
             if (pos + 2) > len(s):
-                raise CheckError
+                raise CheckError(pos - 1)
             ordch2 = ord(s[pos])
             ordch3 = ord(s[pos + 1])
             pos += 2
 
-            if _invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates):
-                raise CheckError
-            elif _invalid_byte_3_of_3(ordch3):
-                raise CheckError
+            if (_invalid_byte_2_of_3(ordch1, ordch2, allow_surrogates) or
+                _invalid_byte_3_of_3(ordch3)):
+                raise CheckError(pos - 3)
             # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz
+            continuation_bytes += 2
             continue
 
         if ordch1 <= 0xF4:
-            continuation_bytes += 3
             if (pos + 3) > len(s):
-                raise CheckError
+                raise CheckError(pos - 1)
             ordch2 = ord(s[pos])
             ordch3 = ord(s[pos + 1])
             ordch4 = ord(s[pos + 2])
             pos += 3
 
-            if _invalid_byte_2_of_4(ordch1, ordch2):
-                raise CheckError
-            elif _invalid_byte_3_of_4(ordch3):
-                raise CheckError
-            elif _invalid_byte_4_of_4(ordch4):
-                raise CheckError
+            if (_invalid_byte_2_of_4(ordch1, ordch2) or
+                _invalid_byte_3_of_4(ordch3) or
+                _invalid_byte_4_of_4(ordch4)):
+                raise CheckError(pos - 4)
             # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz
+            continuation_bytes += 3
             continue
 
-        raise CheckError
+        raise CheckError(pos - 1)
 
     assert pos == len(s)
     return pos - continuation_bytes

From pypy.commits at gmail.com  Thu Aug 24 09:03:41 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 06:03:41 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Tweaks tweaks,
 test_unicodeobject starts to pass again
Message-ID: <599ecead.0b99df0a.cbefd.7193@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92251:d602bc94d49f
Date: 2017-08-24 15:03 +0200
http://bitbucket.org/pypy/pypy/changeset/d602bc94d49f/

Log:	Tweaks tweaks, test_unicodeobject starts to pass again

diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py
--- a/pypy/interpreter/unicodehelper.py
+++ b/pypy/interpreter/unicodehelper.py
@@ -63,8 +63,9 @@
     # you still get two surrogate unicode characters in the result.
     # These are the Python2 rules; Python3 differs.
     try:
-        consumed, length = rutf8.str_check_utf8(string, len(string), True)
-    except rutf8.Utf8CheckError as e:
+        length = rutf8.check_utf8(string, allow_surrogates=True)
+    except rutf8.CheckError as e:
+        XXX
         decode_error_handler(space)('strict', 'utf8', e.msg, string, e.startpos,
                                     e.endpos)
         raise False, "unreachable"
diff --git a/pypy/module/__builtin__/operation.py b/pypy/module/__builtin__/operation.py
--- a/pypy/module/__builtin__/operation.py
+++ b/pypy/module/__builtin__/operation.py
@@ -5,9 +5,8 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.gateway import unwrap_spec, WrappedDefault
-from rpython.rlib.rutf8 import unichr_as_utf8
 from rpython.rlib.rfloat import isfinite, isinf, round_double, round_away
-from rpython.rlib import rfloat
+from rpython.rlib import rfloat, rutf8
 import __builtin__
 
 def abs(space, w_val):
@@ -25,12 +24,11 @@
 @unwrap_spec(code=int)
 def unichr(space, code):
     "Return a Unicode string of one character with the given ordinal."
-    # XXX this assumes unichr would be happy to return you surrogates
     try:
-        s, lgt = unichr_as_utf8(code)
+        s = rutf8.unichr_as_utf8(code, allow_surrogates=True)
     except ValueError:
         raise oefmt(space.w_ValueError, "unichr() arg out of range")
-    return space.newutf8(s, lgt)
+    return space.newutf8(s, 1)
 
 def len(space, w_obj):
     "len(object) -> integer\n\nReturn the number of items of a sequence or mapping."
diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -34,12 +34,13 @@
     @enforceargs(utf8str=str)
     def __init__(self, utf8str, length, ucs4str=None):
         assert isinstance(utf8str, str)
+        assert length >= 0
         if ucs4str is not None:
             assert isinstance(ucs4str, unicode)
         self._utf8 = utf8str
         self._length = length
         self._ucs4 = ucs4str
-        if not we_are_translated() and length != -1:
+        if not we_are_translated():
             assert rutf8.compute_length_utf8(utf8str) == length
 
     def __repr__(self):
@@ -133,8 +134,8 @@
         return W_UnicodeObject.EMPTY
 
     def _len(self):
-        if self._length == -1:
-            self._length = self._compute_length()
+        #if self._length == -1:
+        #    self._length = self._compute_length()
         return self._length
 
     def _compute_length(self):
@@ -902,7 +903,7 @@
     s = space.bytes_w(w_bytes)
     try:
         rutf8.check_ascii(s)
-    except rutf8.AsciiCheckError:
+    except rutf8.CheckError:
         # raising UnicodeDecodeError is messy, "please crash for me"
         return unicode_from_encoded_object(space, w_bytes, "ascii", "strict")
     return W_UnicodeObject(s, len(s))

From pypy.commits at gmail.com  Thu Aug 24 09:14:47 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 06:14:47 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Tweak unicode.splitlines()
Message-ID: <599ed147.f688df0a.2d174.28f3@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92252:2a8ae058f62e
Date: 2017-08-24 15:14 +0200
http://bitbucket.org/pypy/pypy/changeset/2a8ae058f62e/

Log:	Tweak unicode.splitlines()

diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -522,17 +522,18 @@
                 lgt += 1
             eol = pos
             if pos < length:
-                pos = rutf8.next_codepoint_pos(value, pos)
-            # read CRLF as one line break
-            if pos < length and value[eol] == '\r' and value[pos] == '\n':
-                pos += 1
+                # read CRLF as one line break
+                if (value[pos] == '\r' and pos + 1 < length
+                                       and value[pos + 1] == '\n'):
+                    pos += 2
+                    line_end_chars = 2
+                else:
+                    pos = rutf8.next_codepoint_pos(value, pos)
+                    line_end_chars = 1
                 if keepends:
-                    lgt += 1
-            if keepends:
-                eol = pos
-                lgt += 1
-            # XXX find out why lgt calculation is off
-            strs_w.append(W_UnicodeObject(value[sol:eol], -1))
+                    eol = pos
+                    lgt += line_end_chars
+            strs_w.append(W_UnicodeObject(value[sol:eol], lgt))
         return space.newlist(strs_w)
 
     @unwrap_spec(width=int)

From pypy.commits at gmail.com  Thu Aug 24 09:20:30 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 06:20:30 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Fixes
Message-ID: <599ed29e.50131c0a.a6220.449a@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92253:07a16357501d
Date: 2017-08-24 15:19 +0200
http://bitbucket.org/pypy/pypy/changeset/07a16357501d/

Log:	Fixes

diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -375,7 +375,7 @@
 def make_encoder_wrapper(name):
     rname = "utf8_encode_%s" % (name.replace("_encode", ""), )
     @unwrap_spec(utf8='utf8', errors='text_or_none')
-    def wrap_encoder(space, utf8, utf8len, errors="strict"):
+    def wrap_encoder(space, utf8, errors="strict"):
         from pypy.interpreter import unicodehelper
         XXX
 
@@ -446,7 +446,8 @@
 # utf-8 functions are not regular, because we have to pass
 # "allow_surrogates=True"
 @unwrap_spec(utf8='utf8', errors='text_or_none')
-def utf_8_encode(space, utf8, utf8len, errors="strict"):
+def utf_8_encode(space, utf8, errors="strict"):
+    XXXX
     return space.newtuple([space.newbytes(utf8), space.newint(utf8len)])
 #@unwrap_spec(uni=unicode, errors='text_or_none')
 #def utf_8_encode(space, uni, errors="strict"):
@@ -472,29 +473,17 @@
     state = space.fromcache(CodecState)
     # call the fast version for checking
     try:
-        consumed, lgt = rutf8.str_check_utf8(string, len(string), final)
-    except rutf8.Utf8CheckError as e:
-        if errors == 'strict':
-            # just raise
-            state.decode_error_handler(errors, 'utf8', e.msg, string,
-                                       e.startpos, e.endpos)
-            assert False, "raises"
-        # XXX do the way aroun runicode - we can optimize it later if we
+        lgt = rutf8.check_utf8(string)
+    except rutf8.CheckError as e:
+        # XXX do the way around runicode - we can optimize it later if we
         # decide we care about obscure cases
         res, consumed, lgt = unicodehelper.str_decode_utf8(string, len(string),
             errors, final, state.decode_error_handler)
         return space.newtuple([space.newutf8(res, lgt),
-                           space.newint(consumed)])
-    #result, consumed = runicode.str_decode_utf_8_impl(
-    #    string, len(string), errors,
-    #    final, state.decode_error_handler,
-    #    allow_surrogates=True)
-    if final or consumed == len(string):
+                               space.newint(consumed)])
+    else:
         return space.newtuple([space.newutf8(string, lgt),
-                               space.newint(consumed)])
-
-    return space.newtuple([space.newutf8(string[:consumed], lgt),
-                           space.newint(consumed)])
+                               space.newint(len(string))])
 
 @unwrap_spec(data='bufferstr', errors='text_or_none', byteorder=int,
              w_final=WrappedDefault(False))
@@ -639,8 +628,9 @@
     return space.newtuple([space.newunicode(result), space.newint(consumed)])
 
 @unwrap_spec(utf8='utf8', errors='text_or_none')
-def charmap_encode(space, utf8, utf8len, errors="strict", w_mapping=None):
+def charmap_encode(space, utf8, errors="strict", w_mapping=None):
     from pypy.interpreter.unicodehelper import EncodeWrapper
+    XXXXX
 
     if errors is None:
         errors = 'strict'
@@ -658,8 +648,9 @@
 
 
 @unwrap_spec(chars='utf8')
-def charmap_build(space, chars, charslen):
+def charmap_build(space, chars):
     # XXX CPython sometimes uses a three-level trie
+    XXXXXX
     w_charmap = space.newdict()
     pos = 0
     num = 0

From pypy.commits at gmail.com  Thu Aug 24 11:01:38 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 08:01:38 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599eea52.830a1c0a.ebb7.78c6@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92254:d4bde635e3a9
Date: 2017-08-24 17:00 +0200
http://bitbucket.org/pypy/pypy/changeset/d4bde635e3a9/

Log:	(fijal, arigo)

	General progress

diff --git a/pypy/objspace/std/objspace.py b/pypy/objspace/std/objspace.py
--- a/pypy/objspace/std/objspace.py
+++ b/pypy/objspace/std/objspace.py
@@ -9,7 +9,7 @@
 from rpython.rlib.debug import make_sure_not_resized
 from rpython.rlib.rarithmetic import base_int, widen, is_valid_int
 from rpython.rlib.objectmodel import import_from_mixin, enforceargs, not_rpython
-from rpython.rlib import jit
+from rpython.rlib import jit, rutf8
 
 # Object imports
 from pypy.objspace.std.basestringtype import basestring_typedef
@@ -312,11 +312,12 @@
         return self.newlist(list_u)
         return W_ListObject.newlist_unicode(self, list_u)
 
-    def newlist_from_unicode(self, lst):
+    def newlist_utf8(self, lst):
         res_w = []
-        for u in lst:
-            assert u is not None
-            res_w.append(self.newutf8(u, -1))
+        for utf in lst:
+            assert utf is not None
+            assert isinstance(utf, str)
+            res_w.append(self.newutf8(utf, rutf8.check_utf8(utf)))
         return self.newlist(res_w)
 
     def newlist_int(self, list_i):
diff --git a/pypy/objspace/std/test/test_unicodeobject.py b/pypy/objspace/std/test/test_unicodeobject.py
--- a/pypy/objspace/std/test/test_unicodeobject.py
+++ b/pypy/objspace/std/test/test_unicodeobject.py
@@ -315,6 +315,16 @@
         assert u'one!two!three!'.replace('x', '@') == u'one!two!three!'
         assert u'one!two!three!'.replace(u'x', '@', 2) == u'one!two!three!'
         assert u'abc'.replace('', u'-') == u'-a-b-c-'
+        assert u'\u1234'.replace(u'', '-') == u'-\u1234-'
+        assert u'\u0234\u5678'.replace('', u'-') == u'-\u0234-\u5678-'
+        assert u'\u0234\u5678'.replace('', u'-', 0) == u'\u0234\u5678'
+        assert u'\u0234\u5678'.replace('', u'-', 1) == u'-\u0234\u5678'
+        assert u'\u0234\u5678'.replace('', u'-', 2) == u'-\u0234-\u5678'
+        assert u'\u0234\u5678'.replace('', u'-', 3) == u'-\u0234-\u5678-'
+        assert u'\u0234\u5678'.replace('', u'-', 4) == u'-\u0234-\u5678-'
+        assert u'\u0234\u5678'.replace('', u'-', 700) == u'-\u0234-\u5678-'
+        assert u'\u0234\u5678'.replace('', u'-', -1) == u'-\u0234-\u5678-'
+        assert u'\u0234\u5678'.replace('', u'-', -42) == u'-\u0234-\u5678-'
         assert u'abc'.replace(u'', u'-', 3) == u'-a-b-c'
         assert u'abc'.replace('', '-', 0) == u'abc'
         assert u''.replace(u'', '') == u''
diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -6,7 +6,7 @@
 from rpython.rlib.buffer import StringBuffer
 from rpython.rlib.mutbuffer import MutableStringBuffer
 from rpython.rlib.rstring import StringBuilder, split, rsplit, UnicodeBuilder,\
-     replace
+     replace_count
 from rpython.rlib.runicode import make_unicode_escape_function
 from rpython.rlib import rutf8, jit
 
@@ -41,7 +41,7 @@
         self._length = length
         self._ucs4 = ucs4str
         if not we_are_translated():
-            assert rutf8.compute_length_utf8(utf8str) == length
+            assert rutf8.check_utf8(utf8str) == length
 
     def __repr__(self):
         """representation for debugging purposes"""
@@ -561,30 +561,30 @@
         res = []
         value = self._utf8
         if space.is_none(w_sep):
-            res = split(value, maxsplit=maxsplit, isutf8=1)
-            return space.newlist_from_unicode(res)
+            res = split(value, maxsplit=maxsplit, isutf8=True)
+            return space.newlist_utf8(res)
 
         by = self.convert_arg_to_w_unicode(space, w_sep)._utf8
         if len(by) == 0:
             raise oefmt(space.w_ValueError, "empty separator")
-        res = split(value, by, maxsplit, isutf8=1)
+        res = split(value, by, maxsplit, isutf8=True)
 
-        return space.newlist_from_unicode(res)
+        return space.newlist_utf8(res)
 
     @unwrap_spec(maxsplit=int)
     def descr_rsplit(self, space, w_sep=None, maxsplit=-1):
         res = []
         value = self._utf8
         if space.is_none(w_sep):
-            res = rsplit(value, maxsplit=maxsplit, isutf8=1)
-            return space.newlist_from_unicode(res)
+            res = rsplit(value, maxsplit=maxsplit, isutf8=True)
+            return space.newlist_utf8(res)
 
         by = self.convert_arg_to_w_unicode(space, w_sep)._utf8
         if len(by) == 0:
             raise oefmt(space.w_ValueError, "empty separator")
-        res = rsplit(value, by, maxsplit, isutf8=1)
+        res = rsplit(value, by, maxsplit, isutf8=True)
 
-        return space.newlist_from_unicode(res)
+        return space.newlist_utf8(res)
 
     @unwrap_spec(width=int, w_fillchar=WrappedDefault(' '))
     def descr_center(self, space, width, w_fillchar):
@@ -622,11 +622,13 @@
         if count >= 0 and len(input) == 0:
             return self._empty()
         try:
-            res = replace(input, w_sub._utf8, w_by._utf8, count)
+            res, replacements = replace_count(input, w_sub._utf8, w_by._utf8,
+                                              count, isutf8=True)
         except OverflowError:
             raise oefmt(space.w_OverflowError, "replace string is too long")
 
-        return W_UnicodeObject(res, -1)
+        newlength = self._length + replacements * (w_by._length - w_sub._length)
+        return W_UnicodeObject(res, newlength)
 
     def descr_mul(self, space, w_times):
         try:
diff --git a/rpython/rlib/rstring.py b/rpython/rlib/rstring.py
--- a/rpython/rlib/rstring.py
+++ b/rpython/rlib/rstring.py
@@ -16,42 +16,38 @@
 # -------------- public API for string functions -----------------------
 
 @specialize.ll_and_arg(2)
-def _isspace(s, pos, isutf8=0):
+def _isspace(s, pos, isutf8=False):
     if isutf8:
         from rpython.rlib import rutf8
         return rutf8.isspace(s, pos)
+    char = s[pos]
+    if isinstance(char, str):
+        return char.isspace()
     else:
-        char = s[pos]
-        if isinstance(char, str):
-            return char.isspace()
-        else:
-            assert isinstance(char, unicode)
-            return unicodedb.isspace(ord(char))
+        assert isinstance(char, unicode)
+        return unicodedb.isspace(ord(char))
 
 @specialize.ll_and_arg(2)
 def _incr(s, pos, isutf8):
-    from rpython.rlib.rutf8 import next_codepoint_pos
-
     if isutf8:
-        if pos == -1:
-            return 0
+        from rpython.rlib.rutf8 import next_codepoint_pos
+        assert pos >= 0
         return next_codepoint_pos(s, pos)
     else:
         return pos + 1
 
 @specialize.ll_and_arg(2)
 def _decr(s, pos, isutf8):
-    from rpython.rlib.rutf8 import prev_codepoint_pos
-
     if isutf8:
-        if pos == 0:
+        from rpython.rlib.rutf8 import prev_codepoint_pos
+        if pos <= 0:
             return -1
         return prev_codepoint_pos(s, pos)
     else:
         return pos - 1
 
 @specialize.ll_and_arg(3)
-def split(value, by=None, maxsplit=-1, isutf8=0):
+def split(value, by=None, maxsplit=-1, isutf8=False):
     if by is None:
         length = len(value)
         i = 0
@@ -83,7 +79,11 @@
             else:
                 break
         return res
+    else:
+        return _split_by(value, by, maxsplit)
 
+ at specialize.argtype(0)
+def _split_by(value, by, maxsplit):
     if isinstance(value, unicode):
         assert isinstance(by, unicode)
     if isinstance(value, str):
@@ -133,7 +133,7 @@
 
 
 @specialize.ll_and_arg(3)
-def rsplit(value, by=None, maxsplit=-1, isutf8=0):
+def rsplit(value, by=None, maxsplit=-1, isutf8=False):
     if by is None:
         res = []
 
@@ -147,30 +147,34 @@
             else:
                 break  # end of string, finished
 
-            # find the start of the word
-            # (more precisely, 'j' will be the space character before the word)
+            # find the start of the word as 'j1'
             if maxsplit == 0:
-                j = -1   # take all the rest of the string
+                j1 = 0   # take all the rest of the string
+                j = -1
             else:
-                j = _decr(value, i, isutf8)
-                while j >= 0 and not _isspace(value, j, isutf8):
-                    j = _decr(value, j, isutf8)
+                j1 = i
+                while True:
+                    j = _decr(value, j1, isutf8)
+                    if j < 0 or _isspace(value, j, isutf8):
+                        break
+                    j1 = j
                 maxsplit -= 1   # NB. if it's already < 0, it stays < 0
 
-            # the word is value[j+1:i+1]
-            j1 = _incr(value, j, isutf8)
+            # the word is value[j1:i+1]
             assert j1 >= 0
             i1 = _incr(value, i, isutf8)
             res.append(value[j1:i1])
-            if j < 0:
-                break
 
             # continue to look from the character before the space before the word
             i = _decr(value, j, isutf8)
 
         res.reverse()
         return res
+    else:
+        return _rsplit_by(value, by, maxsplit)
 
+ at specialize.argtype(0)
+def _rsplit_by(value, by, maxsplit):
     if isinstance(value, unicode):
         assert isinstance(by, unicode)
     if isinstance(value, str):
@@ -203,6 +207,11 @@
 @specialize.argtype(0, 1)
 @jit.elidable
 def replace(input, sub, by, maxsplit=-1):
+    return replace_count(input, sub, by, maxsplit)[0]
+
+ at specialize.ll_and_arg(4)
+ at jit.elidable
+def replace_count(input, sub, by, maxsplit=-1, isutf8=False):
     if isinstance(input, str):
         Builder = StringBuilder
     elif isinstance(input, unicode):
@@ -211,10 +220,10 @@
         assert isinstance(input, list)
         Builder = ByteListBuilder
     if maxsplit == 0:
-        return input
+        return input, 0
 
 
-    if not sub:
+    if not sub and not isutf8:
         upper = len(input)
         if maxsplit > 0 and maxsplit < upper + 2:
             upper = maxsplit - 1
@@ -234,9 +243,16 @@
             builder.append(input[i])
         builder.append(by)
         builder.append_slice(input, upper, len(input))
+        replacements = upper + 1
     else:
         # First compute the exact result size
-        cnt = count(input, sub, 0, len(input))
+        if sub:
+            cnt = count(input, sub, 0, len(input))
+        else:
+            assert isutf8
+            from rpython.rlib import rutf8
+            cnt = rutf8.compute_length_utf8(input) + 1
+
         if cnt > maxsplit and maxsplit > 0:
             cnt = maxsplit
         diff_len = len(by) - len(sub)
@@ -245,23 +261,36 @@
             result_size = ovfcheck(result_size + len(input))
         except OverflowError:
             raise
+        replacements = cnt
 
         builder = Builder(result_size)
         start = 0
         sublen = len(sub)
 
-        while maxsplit != 0:
-            next = find(input, sub, start, len(input))
-            if next < 0:
-                break
-            builder.append_slice(input, start, next)
-            builder.append(by)
-            start = next + sublen
-            maxsplit -= 1   # NB. if it's already < 0, it stays < 0
+        if sublen == 0:
+            assert isutf8
+            from rpython.rlib import rutf8
+            while True:
+                builder.append(by)
+                maxsplit -= 1
+                if start == len(input) or maxsplit == 0:
+                    break
+                next = rutf8.next_codepoint_pos(input, start)
+                builder.append_slice(input, start, next)
+                start = next
+        else:
+            while maxsplit != 0:
+                next = find(input, sub, start, len(input))
+                if next < 0:
+                    break
+                builder.append_slice(input, start, next)
+                builder.append(by)
+                start = next + sublen
+                maxsplit -= 1   # NB. if it's already < 0, it stays < 0
 
         builder.append_slice(input, start, len(input))
 
-    return builder.build()
+    return builder.build(), replacements
 
 def _normalize_start_end(length, start, end):
     if start < 0:
diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py
--- a/rpython/rlib/rutf8.py
+++ b/rpython/rlib/rutf8.py
@@ -203,6 +203,24 @@
         return True
     return False
 
+def utf8_in_chars(value, pos, chars):
+    """Equivalent of u'x' in u'xyz', where the left-hand side is
+    a single UTF-8 character extracted from the string 'value' at 'pos'.
+    Only works if both 'value' and 'chars' are correctly-formed UTF-8
+    strings.
+    """
+    end = next_codepoint_pos(value, pos)
+    i = 0
+    while i < len(chars):
+        k = pos
+        while value[k] == chars[i]:
+            k += 1
+            i += 1
+            if k == end:
+                return True
+        i += 1
+    return False
+
 
 def _invalid_cont_byte(ordch):
     return ordch>>6 != 0x2    # 0b10
diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py
--- a/rpython/rlib/test/test_rutf8.py
+++ b/rpython/rlib/test/test_rutf8.py
@@ -79,14 +79,8 @@
         else:
             assert not rutf8.isspace(unichr(i).encode('utf8'), 0)
 
- at given(strategies.integers(min_value=0, max_value=sys.maxunicode),
-       strategies.characters())
-def test_utf8_in_chars(i, uni):
-    if not uni:
-        return
-    if unichr(i) in uni:
-        response = True
-    else:
-        response = False
-    r = unichr(i).encode('utf8') in uni.encode('utf8')
+ at given(strategies.characters(), strategies.text())
+def test_utf8_in_chars(ch, txt):
+    response = rutf8.utf8_in_chars(ch.encode('utf8'), 0, txt.encode('utf8'))
+    r = (ch in txt)
     assert r == response

From pypy.commits at gmail.com  Thu Aug 24 11:22:45 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 08:22:45 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599eef45.54871c0a.dcbf8.d7bb@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92255:8e029544ca2f
Date: 2017-08-24 17:21 +0200
http://bitbucket.org/pypy/pypy/changeset/8e029544ca2f/

Log:	(fijal, arigo)

	Fix unicode.translate()

diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -365,36 +365,39 @@
         return mod_format(space, w_values, self, do_unicode=True)
 
     def descr_translate(self, space, w_table):
-        selfvalue = self._utf8.decode("utf8")
-        w_sys = space.getbuiltinmodule('sys')
-        maxunicode = space.int_w(space.getattr(w_sys,
-                                               space.newtext("maxunicode")))
-        result = []
-        for unichar in selfvalue:
+        input = self._utf8
+        result = StringBuilder(len(input))
+        result_length = 0
+        i = 0
+        while i < len(input):
+            codepoint = rutf8.codepoint_at_pos(input, i)
+            i = rutf8.next_codepoint_pos(input, i)
             try:
-                w_newval = space.getitem(w_table, space.newint(ord(unichar)))
+                w_newval = space.getitem(w_table, space.newint(codepoint))
             except OperationError as e:
-                if e.match(space, space.w_LookupError):
-                    result.append(unichar)
-                else:
+                if not e.match(space, space.w_LookupError):
                     raise
             else:
                 if space.is_w(w_newval, space.w_None):
                     continue
                 elif space.isinstance_w(w_newval, space.w_int):
-                    newval = space.int_w(w_newval)
-                    if newval < 0 or newval > maxunicode:
-                        raise oefmt(space.w_TypeError,
-                                    "character mapping must be in range(%s)",
-                                    hex(maxunicode + 1))
-                    result.append(unichr(newval))
-                elif space.isinstance_w(w_newval, space.w_unicode):
-                    result.append(space.utf8_w(w_newval).decode("utf8"))
+                    codepoint = space.int_w(w_newval)
+                elif isinstance(w_newval, W_UnicodeObject):
+                    result.append(w_newval._utf8)
+                    result_length += w_newval._length
+                    continue
                 else:
                     raise oefmt(space.w_TypeError,
                                 "character mapping must return integer, None "
                                 "or unicode")
-        return W_UnicodeObject(u''.join(result).encode("utf8"), -1)
+            try:
+                rutf8.unichr_as_utf8_append(result, codepoint,
+                                            allow_surrogates=True)
+                result_length += 1
+            except ValueError:
+                raise oefmt(space.w_TypeError,
+                            "character mapping must be in range(0x110000)")
+        return W_UnicodeObject(result.build(), result_length)
 
     def descr_encode(self, space, w_encoding=None, w_errors=None):
         encoding, errors = _get_encoding_and_errors(space, w_encoding,

From pypy.commits at gmail.com  Thu Aug 24 11:25:22 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 08:25:22 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599eefe2.41931c0a.d1c9b.3796@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92256:bc374de6e273
Date: 2017-08-24 17:24 +0200
http://bitbucket.org/pypy/pypy/changeset/bc374de6e273/

Log:	(fijal, arigo)

	Tweaks

diff --git a/pypy/objspace/std/unicodeobject.py b/pypy/objspace/std/unicodeobject.py
--- a/pypy/objspace/std/unicodeobject.py
+++ b/pypy/objspace/std/unicodeobject.py
@@ -41,7 +41,7 @@
         self._length = length
         self._ucs4 = ucs4str
         if not we_are_translated():
-            assert rutf8.check_utf8(utf8str) == length
+            assert rutf8.check_utf8(utf8str, allow_surrogates=True) == length
 
     def __repr__(self):
         """representation for debugging purposes"""
@@ -845,12 +845,11 @@
             return space.newutf8(s, len(s))
         if encoding == 'utf-8':
             s = space.charbuf_w(w_obj)
-            eh = unicodehelper.decode_error_handler(space)
             try:
-                _, lgt = rutf8.str_check_utf8(s, len(s), final=True,
-                                              allow_surrogates=True)
+                lgt = rutf8.check_utf8(s, allow_surrogates=True)
             except rutf8.CheckError:
                 XXX
+                eh = unicodehelper.decode_error_handler(space)
                 eh(None, 'utf8', e.msg, s, e.startpos, e.endpos)
                 assert False, "has to raise"
             return space.newutf8(s, lgt)

From pypy.commits at gmail.com  Thu Aug 24 11:38:08 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 24 Aug 2017 08:38:08 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (fijal, arigo)
Message-ID: <599ef2e0.9aa4df0a.99c3f.437b@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92257:2cc3cf290fbf
Date: 2017-08-24 17:37 +0200
http://bitbucket.org/pypy/pypy/changeset/2cc3cf290fbf/

Log:	(fijal, arigo)

	Hack hack hack

diff --git a/pypy/interpreter/unicodehelper.py b/pypy/interpreter/unicodehelper.py
--- a/pypy/interpreter/unicodehelper.py
+++ b/pypy/interpreter/unicodehelper.py
@@ -29,6 +29,10 @@
                                              space.newtext(msg)]))
     return raise_unicode_exception_encode
 
+def convert_arg_to_w_unicode(space, w_arg, strict=None):
+    from pypy.objspace.std.unicodeobject import W_UnicodeObject
+    return W_UnicodeObject.convert_arg_to_w_unicode(space, w_arg, strict)
+
 # ____________________________________________________________
 
 def encode(space, w_data, encoding=None, errors='strict'):
diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py
--- a/pypy/module/_codecs/interp_codecs.py
+++ b/pypy/module/_codecs/interp_codecs.py
@@ -374,16 +374,17 @@
 
 def make_encoder_wrapper(name):
     rname = "utf8_encode_%s" % (name.replace("_encode", ""), )
-    @unwrap_spec(utf8='utf8', errors='text_or_none')
-    def wrap_encoder(space, utf8, errors="strict"):
+    @unwrap_spec(errors='text_or_none')
+    def wrap_encoder(space, w_arg, errors="strict"):
         from pypy.interpreter import unicodehelper
-        XXX
 
+        w_arg = unicodehelper.convert_arg_to_w_unicode(space, w_arg, rname)
         if errors is None:
             errors = 'strict'
         state = space.fromcache(CodecState)
         func = getattr(unicodehelper, rname)
-        result = func(utf8, utf8len,
+        utf8len = w_arg._length
+        result = func(w_arg._utf8, utf8len,
             errors, state.encode_error_handler)
         return space.newtuple([space.newbytes(result), space.newint(utf8len)])
     wrap_encoder.func_name = rname

From pypy.commits at gmail.com  Thu Aug 24 12:40:22 2017
From: pypy.commits at gmail.com (fijal)
Date: Thu, 24 Aug 2017 09:40:22 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: (arigo,
 fijal) implement fast skipping technique in RPython
Message-ID: <599f0176.f688df0a.2d174.664a@mx.google.com>

Author: fijal
Branch: unicode-utf8
Changeset: r92258:d2735187e72f
Date: 2017-08-24 18:39 +0200
http://bitbucket.org/pypy/pypy/changeset/d2735187e72f/

Log:	(arigo, fijal) implement fast skipping technique in RPython

diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py
--- a/rpython/rlib/rutf8.py
+++ b/rpython/rlib/rutf8.py
@@ -19,6 +19,7 @@
 from rpython.rlib.rstring import StringBuilder
 from rpython.rlib import jit
 from rpython.rlib.rarithmetic import r_uint
+from rpython.rtyper.lltypesystem import lltype
 
 
 def unichr_as_utf8(code, allow_surrogates=False):
@@ -307,3 +308,65 @@
 
     assert pos == len(s)
     return pos - continuation_bytes
+
+
+UTF8_INDEX_STORAGE = lltype.GcArray(lltype.Struct(
+    'utf8_loc',
+    ('index', lltype.Signed),
+    ('ofs', lltype.FixedSizeArray(lltype.Char, 16))
+    ))
+
+EMPTY_INDEX_STORAGE = lltype.malloc(UTF8_INDEX_STORAGE, 0, immortal=True)
+
+def create_utf8_index_storage(utf8, utf8len):
+    """ Create an index storage which stores index of each 4th character
+    in utf8 encoded unicode string.
+    """
+    if utf8len == 0:
+        return EMPTY_INDEX_STORAGE
+    arraysize = (utf8len + 63) // 64
+    storage = lltype.malloc(UTF8_INDEX_STORAGE, arraysize)
+    baseindex = 0
+    current = 0
+    next = 0
+    while True:
+        storage[current].index = baseindex
+        for i in range(16):
+            next = next_codepoint_pos(utf8, next)
+            storage[current].ofs[i] = chr(next - baseindex)
+            utf8len -= 4
+            if utf8len <= 0:
+                break
+            next = next_codepoint_pos(utf8, next)
+            next = next_codepoint_pos(utf8, next)
+            next = next_codepoint_pos(utf8, next)            
+        else:
+            current += 1
+            baseindex = next
+            continue
+        break
+    return storage
+
+def codepoint_position_at_index(utf8, storage, index):
+    """ Return byte index of a character inside utf8 encoded string, given
+    storage of type UTF8_INDEX_STORAGE
+    """
+    current = index >> 6
+    ofs = ord(storage[current].ofs[(index >> 2) & 15])
+    bytepos = storage[current].index + ofs
+    index &= 0x3
+    if index == 0:
+        return prev_codepoint_pos(utf8, bytepos)
+    elif index == 1:
+        return bytepos
+    elif index == 2:
+        return next_codepoint_pos(utf8, bytepos)
+    else:
+        return next_codepoint_pos(utf8, next_codepoint_pos(utf8, bytepos))
+
+def codepoint_at_index(utf8, storage, index):
+    """ Return codepoint of a character inside utf8 encoded string, given
+    storage of type UTF8_INDEX_STORAGE
+    """
+    bytepos = codepoint_position_at_index(utf8, storage, index)
+    return codepoint_at_pos(utf8, bytepos)
diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py
--- a/rpython/rlib/test/test_rutf8.py
+++ b/rpython/rlib/test/test_rutf8.py
@@ -84,3 +84,9 @@
     response = rutf8.utf8_in_chars(ch.encode('utf8'), 0, txt.encode('utf8'))
     r = (ch in txt)
     assert r == response
+
+ at given(strategies.text())
+def test_utf8_index_storage(u):
+    index = rutf8.create_utf8_index_storage(u.encode('utf8'), len(u))
+    for i, item in enumerate(u):
+        rutf8.codepoint_at_index(u.encode('utf8'), index, i) == item.encode('utf8')

From pypy.commits at gmail.com  Thu Aug 24 16:08:49 2017
From: pypy.commits at gmail.com (mattip)
Date: Thu, 24 Aug 2017 13:08:49 -0700 (PDT)
Subject: [pypy-commit] pypy default: test, fix for missing __ne__ on slices
Message-ID: <599f3251.f28ddf0a.56b74.7959@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92259:5a7118bbfee9
Date: 2017-08-24 23:07 +0300
http://bitbucket.org/pypy/pypy/changeset/5a7118bbfee9/

Log:	test, fix for missing __ne__ on slices

diff --git a/pypy/objspace/std/sliceobject.py b/pypy/objspace/std/sliceobject.py
--- a/pypy/objspace/std/sliceobject.py
+++ b/pypy/objspace/std/sliceobject.py
@@ -132,6 +132,18 @@
         else:
             return space.w_False
 
+    def descr_ne(self, space, w_other):
+        if space.is_w(self, w_other):
+            return space.w_False
+        if not isinstance(w_other, W_SliceObject):
+            return space.w_NotImplemented
+        if space.eq_w(self.w_start, w_other.w_start) and \
+           space.eq_w(self.w_stop, w_other.w_stop) and \
+           space.eq_w(self.w_step, w_other.w_step):
+            return space.w_False
+        else:
+            return space.w_True
+
     def descr_lt(self, space, w_other):
         if space.is_w(self, w_other):
             return space.w_False   # see comments in descr_eq()
@@ -177,6 +189,7 @@
     __reduce__ = gateway.interp2app(W_SliceObject.descr__reduce__),
 
     __eq__ = gateway.interp2app(W_SliceObject.descr_eq),
+    __ne__ = gateway.interp2app(W_SliceObject.descr_ne),
     __lt__ = gateway.interp2app(W_SliceObject.descr_lt),
 
     start = slicewprop('w_start'),
diff --git a/pypy/objspace/std/test/test_sliceobject.py b/pypy/objspace/std/test/test_sliceobject.py
--- a/pypy/objspace/std/test/test_sliceobject.py
+++ b/pypy/objspace/std/test/test_sliceobject.py
@@ -94,6 +94,7 @@
         slice1 = slice(1, 2, 3)
         slice2 = slice(1, 2, 3)
         assert slice1 == slice2
+        assert not slice1 != slice2
         slice2 = slice(1, 2)
         assert slice1 != slice2
 

From pypy.commits at gmail.com  Fri Aug 25 03:53:24 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 25 Aug 2017 00:53:24 -0700 (PDT)
Subject: [pypy-commit] pypy.org extradoc: Update the section about
 sandboxing again, to make it sound even less supported
Message-ID: <599fd774.0b99df0a.cbefd.ca88@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: extradoc
Changeset: r898:4d64c0df4967
Date: 2017-08-25 09:52 +0200
http://bitbucket.org/pypy/pypy.org/changeset/4d64c0df4967/

Log:	Update the section about sandboxing again, to make it sound even
	less supported

diff --git a/download.html b/download.html
--- a/download.html
+++ b/download.html
@@ -182,16 +182,12 @@
 </ul>
 <ul class="simple">
 <li>Sandboxing: A special safe version.  Read the docs about <a class="reference external" href="features.html#sandboxing">sandboxing</a>.
-(It is also possible to <a class="reference internal" href="#translate">translate</a> a version that includes both
-sandboxing and the JIT compiler, although as the JIT is relatively
-complicated, this reduces a bit the level of confidence we can put in
-the result.) <strong>Note that the sandboxed binary needs a full pypy checkout
-to work</strong>. Consult the <a class="reference external" href="http://doc.pypy.org/en/latest/sandbox.html">sandbox docs</a> for details. (These are old,
-PyPy 1.8.)<ul>
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux64.tar.bz2">Linux binary (64bit)</a></li>
-<li><a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux.tar.bz2">Linux binary (32bit)</a></li>
-</ul>
-</li>
+This version is <strong>not supported</strong> and not actively maintained.  You
+will likely have to fix some issues yourself, or checkout an old
+version, or otherwise play around on your own.  We provide this
+documentation only for historical reasons.  Please do not use in
+production.  For reference, there are some very old, unmaintained
+binaries for Linux (<a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux64.tar.bz2">32bit</a>, <a class="reference external" href="https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux.tar.bz2">64bit</a>).</li>
 </ul>
 </div>
 <div class="section" id="installing">
diff --git a/source/download.txt b/source/download.txt
--- a/source/download.txt
+++ b/source/download.txt
@@ -179,16 +179,12 @@
 .. __: https://bitbucket.org/pypy/revdb/
 
 * Sandboxing: A special safe version.  Read the docs about sandboxing_.
-  (It is also possible to translate_ a version that includes both
-  sandboxing and the JIT compiler, although as the JIT is relatively
-  complicated, this reduces a bit the level of confidence we can put in
-  the result.) **Note that the sandboxed binary needs a full pypy checkout
-  to work**. Consult the `sandbox docs`_ for details. (These are old,
-  PyPy 1.8.)
-
-  * `Linux binary (64bit)`__
-
-  * `Linux binary (32bit)`__
+  This version is **not supported** and not actively maintained.  You
+  will likely have to fix some issues yourself, or checkout an old
+  version, or otherwise play around on your own.  We provide this
+  documentation only for historical reasons.  Please do not use in
+  production.  For reference, there are some very old, unmaintained
+  binaries for Linux (32bit__, 64bit__).
 
 .. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux64.tar.bz2
 .. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux.tar.bz2

From pypy.commits at gmail.com  Fri Aug 25 04:36:11 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 25 Aug 2017 01:36:11 -0700 (PDT)
Subject: [pypy-commit] cffi default: Improve error message
Message-ID: <599fe17b.e9a9df0a.ea3c.c3c3@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3004:def116eeded6
Date: 2017-08-25 10:35 +0200
http://bitbucket.org/cffi/cffi/changeset/def116eeded6/

Log:	Improve error message

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -44,11 +44,14 @@
 
 def no_working_compiler_found():
     sys.stderr.write("""
-    No working compiler found, or bogus compiler options
-    passed to the compiler from Python's distutils module.
-    See the error messages above.
-    (If they are about -mno-fused-madd and you are on OS/X 10.8,
-    see http://stackoverflow.com/questions/22313407/ .)\n""")
+    No working compiler found, or bogus compiler options passed to
+    the compiler from Python's standard "distutils" module.  See
+    the error messages above.  Likely, the problem is not related
+    to CFFI but generic to the setup.py of any Python package that
+    tries to compile C code.  (Hints: on OS/X 10.8, for errors about
+    -mno-fused-madd see http://stackoverflow.com/questions/22313407/
+    Otherwise, see https://wiki.python.org/moin/CompLangPython or
+    the IRC channel #python on irc.freenode.net.)\n""")
     sys.exit(1)
 
 def get_config():

From pypy.commits at gmail.com  Fri Aug 25 06:32:29 2017
From: pypy.commits at gmail.com (arigo)
Date: Fri, 25 Aug 2017 03:32:29 -0700 (PDT)
Subject: [pypy-commit] pypy unicode-utf8: Fix test, improve logic
Message-ID: <599ffcbd.11b2df0a.d2919.ab37@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: unicode-utf8
Changeset: r92260:7193602c9384
Date: 2017-08-25 12:30 +0200
http://bitbucket.org/pypy/pypy/changeset/7193602c9384/

Log:	Fix test, improve logic

diff --git a/rpython/rlib/rutf8.py b/rpython/rlib/rutf8.py
--- a/rpython/rlib/rutf8.py
+++ b/rpython/rlib/rutf8.py
@@ -93,6 +93,7 @@
     """Gives the position of the previous codepoint.
     'pos' must not be zero.
     """
+    pos = r_uint(pos)
     pos -= 1
     chr1 = ord(code[pos])
     if chr1 <= 0x7F:
@@ -142,6 +143,43 @@
                 (ordch4 & 0x3F))               # 0b00111111
     assert False, "unreachable"
 
+def codepoint_before_pos(code, pos):
+    """Give a codepoint in code at the position immediately before pos
+    - assumes valid utf8, no checking!
+    """
+    pos = r_uint(pos)
+    ordch1 = ord(code[pos-1])
+    if ordch1 <= 0x7F:
+        return ordch1
+
+    ordch2 = ordch1
+    ordch1 = ord(code[pos-2])
+    if ordch1 >= 0xC0:
+        # 110yyyyy 10zzzzzz -> 00000000 00000yyy yyzzzzzz
+        return (((ordch1 & 0x1F) << 6) +    # 0b00011111
+                 (ordch2 & 0x3F))           # 0b00111111
+
+    ordch3 = ordch2
+    ordch2 = ordch1
+    ordch1 = ord(code[pos-3])
+    if ordch1 >= 0xC0:
+        # 1110xxxx 10yyyyyy 10zzzzzz -> 00000000 xxxxyyyy yyzzzzzz
+        return (((ordch1 & 0x0F) << 12) +     # 0b00001111
+                ((ordch2 & 0x3F) << 6) +      # 0b00111111
+                (ordch3 & 0x3F))              # 0b00111111
+
+    ordch4 = ordch3
+    ordch3 = ordch2
+    ordch2 = ordch1
+    ordch1 = ord(code[pos-4])
+    if True:
+        # 11110www 10xxxxxx 10yyyyyy 10zzzzzz -> 000wwwxx xxxxyyyy yyzzzzzz
+        return (((ordch1 & 0x07) << 18) +      # 0b00000111
+                ((ordch2 & 0x3F) << 12) +      # 0b00111111
+                ((ordch3 & 0x3F) << 6) +       # 0b00111111
+                (ordch4 & 0x3F))               # 0b00111111
+    assert False, "unreachable"
+
 class CheckError(Exception):
     def __init__(self, pos):
         self.pos = pos
@@ -312,25 +350,32 @@
 
 UTF8_INDEX_STORAGE = lltype.GcArray(lltype.Struct(
     'utf8_loc',
-    ('index', lltype.Signed),
+    ('baseindex', lltype.Signed),
     ('ofs', lltype.FixedSizeArray(lltype.Char, 16))
     ))
 
-EMPTY_INDEX_STORAGE = lltype.malloc(UTF8_INDEX_STORAGE, 0, immortal=True)
+ASCII_INDEX_STORAGE_BLOCKS = 5
+ASCII_INDEX_STORAGE = lltype.malloc(UTF8_INDEX_STORAGE,
+                                    ASCII_INDEX_STORAGE_BLOCKS,
+                                    immortal=True)
+for _i in range(ASCII_INDEX_STORAGE_BLOCKS):
+    ASCII_INDEX_STORAGE[_i].baseindex = _i * 64
+    for _j in range(16):
+        ASCII_INDEX_STORAGE[_i].ofs[_j] = chr(_j * 4 + 1)
 
 def create_utf8_index_storage(utf8, utf8len):
     """ Create an index storage which stores index of each 4th character
     in utf8 encoded unicode string.
     """
-    if utf8len == 0:
-        return EMPTY_INDEX_STORAGE
+    if len(utf8) == utf8len <= ASCII_INDEX_STORAGE_BLOCKS * 64:
+        return ASCII_INDEX_STORAGE
     arraysize = (utf8len + 63) // 64
     storage = lltype.malloc(UTF8_INDEX_STORAGE, arraysize)
     baseindex = 0
     current = 0
-    next = 0
     while True:
-        storage[current].index = baseindex
+        storage[current].baseindex = baseindex
+        next = baseindex
         for i in range(16):
             next = next_codepoint_pos(utf8, next)
             storage[current].ofs[i] = chr(next - baseindex)
@@ -339,7 +384,7 @@
                 break
             next = next_codepoint_pos(utf8, next)
             next = next_codepoint_pos(utf8, next)
-            next = next_codepoint_pos(utf8, next)            
+            next = next_codepoint_pos(utf8, next)
         else:
             current += 1
             baseindex = next
@@ -349,11 +394,13 @@
 
 def codepoint_position_at_index(utf8, storage, index):
     """ Return byte index of a character inside utf8 encoded string, given
-    storage of type UTF8_INDEX_STORAGE
+    storage of type UTF8_INDEX_STORAGE.  The index must be smaller than
+    the utf8 length: if needed, check explicitly before calling this
+    function.
     """
     current = index >> 6
-    ofs = ord(storage[current].ofs[(index >> 2) & 15])
-    bytepos = storage[current].index + ofs
+    ofs = ord(storage[current].ofs[(index >> 2) & 0x0F])
+    bytepos = storage[current].baseindex + ofs
     index &= 0x3
     if index == 0:
         return prev_codepoint_pos(utf8, bytepos)
@@ -368,5 +415,15 @@
     """ Return codepoint of a character inside utf8 encoded string, given
     storage of type UTF8_INDEX_STORAGE
     """
-    bytepos = codepoint_position_at_index(utf8, storage, index)
+    current = index >> 6
+    ofs = ord(storage[current].ofs[(index >> 2) & 0x0F])
+    bytepos = storage[current].baseindex + ofs
+    index &= 0x3
+    if index == 0:
+        return codepoint_before_pos(utf8, bytepos)
+    if index == 3:
+        bytepos = next_codepoint_pos(utf8, bytepos)
+        index = 2     # fall-through to the next case
+    if index == 2:
+        bytepos = next_codepoint_pos(utf8, bytepos)
     return codepoint_at_pos(utf8, bytepos)
diff --git a/rpython/rlib/test/test_rutf8.py b/rpython/rlib/test/test_rutf8.py
--- a/rpython/rlib/test/test_rutf8.py
+++ b/rpython/rlib/test/test_rutf8.py
@@ -89,4 +89,12 @@
 def test_utf8_index_storage(u):
     index = rutf8.create_utf8_index_storage(u.encode('utf8'), len(u))
     for i, item in enumerate(u):
-        rutf8.codepoint_at_index(u.encode('utf8'), index, i) == item.encode('utf8')
+        assert (rutf8.codepoint_at_index(u.encode('utf8'), index, i) ==
+                ord(item))
+
+ at given(strategies.text())
+def test_codepoint_position_at_index(u):
+    index = rutf8.create_utf8_index_storage(u.encode('utf8'), len(u))
+    for i in range(len(u)):
+        assert (rutf8.codepoint_position_at_index(u.encode('utf8'), index, i) ==
+                len(u[:i].encode('utf8')))

From pypy.commits at gmail.com  Fri Aug 25 10:19:02 2017
From: pypy.commits at gmail.com (rlamy)
Date: Fri, 25 Aug 2017 07:19:02 -0700 (PDT)
Subject: [pypy-commit] pypy default: Imperative py.test.skip() is evil
Message-ID: <59a031d6.91a5df0a.9f412.ff05@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: 
Changeset: r92261:2b1b6c5545d0
Date: 2017-08-25 16:18 +0200
http://bitbucket.org/pypy/pypy/changeset/2b1b6c5545d0/

Log:	Imperative py.test.skip() is evil

diff --git a/rpython/flowspace/test/test_objspace.py b/rpython/flowspace/test/test_objspace.py
--- a/rpython/flowspace/test/test_objspace.py
+++ b/rpython/flowspace/test/test_objspace.py
@@ -839,15 +839,15 @@
             return x[s]
         graph = self.codetest(myfunc)
 
+    @py.test.mark.xfail
     def test_unichr_constfold(self):
-        py.test.skip("not working")
         def myfunc():
             return unichr(1234)
         graph = self.codetest(myfunc)
         assert graph.startblock.exits[0].target is graph.returnblock
 
+    @py.test.mark.xfail
     def test_unicode_constfold(self):
-        py.test.skip("not working for now")
         def myfunc():
             return unicode("1234")
         graph = self.codetest(myfunc)

From pypy.commits at gmail.com  Sat Aug 26 16:12:22 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Sat, 26 Aug 2017 13:12:22 -0700 (PDT)
Subject: [pypy-commit] extradoc extradoc: add a link to the license so I
 don't need to continuously re-hunt the mailing
Message-ID: <59a1d626.55281c0a.66f93.067a@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: extradoc
Changeset: r5831:e59c6c8223b2
Date: 2017-08-26 22:12 +0200
http://bitbucket.org/pypy/extradoc/changeset/e59c6c8223b2/

Log:	add a link to the license so I don't need to continuously re-hunt
	the mailing list post

diff --git a/logo/LICENSE b/logo/LICENSE
new file mode 100644
--- /dev/null
+++ b/logo/LICENSE
@@ -0,0 +1,4 @@
+The logo is (c) Samuel Reis under an CC-BY-SA license according to
+https://mail.python.org/pipermail/pypy-dev/2011-April/007224.html
+
+https://creativecommons.org/licenses/by-sa/3.0/

From pypy.commits at gmail.com  Sun Aug 27 02:54:05 2017
From: pypy.commits at gmail.com (smihnea)
Date: Sat, 26 Aug 2017 23:54:05 -0700 (PDT)
Subject: [pypy-commit] pypy pypy_swappedbytes: Added _swappedbytes_ support
 for ctypes.Structure
Message-ID: <59a26c8d.41931c0a.6854a.9c3f@mx.google.com>

Author: Mihnea Saracin <mihnea.saracin at rinftech.com>
Branch: pypy_swappedbytes
Changeset: r92262:7ec88773f8b9
Date: 2017-07-27 17:29 +0300
http://bitbucket.org/pypy/pypy/changeset/7ec88773f8b9/

Log:	Added _swappedbytes_ support for ctypes.Structure

diff --git a/lib-python/2.7/ctypes/test/test_unaligned_structures.py b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
--- a/lib-python/2.7/ctypes/test/test_unaligned_structures.py
+++ b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
@@ -37,10 +37,7 @@
         for typ in byteswapped_structures:
 ##            print >> sys.stderr, typ.value
             self.assertEqual(typ.value.offset, 1)
-            try:
-                o = typ()
-            except NotImplementedError as e:
-                self.skipTest(str(e))    # for PyPy
+            o = typ()
             o.value = 4
             self.assertEqual(o.value, 4)
 
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -130,6 +130,7 @@
             obj._buffer.__setattr__(self.name, arg)
 
 
+
 def _set_shape(tp, rawfields, is_union=False):
     tp._ffistruct_ = _rawffi.Structure(rawfields, is_union,
                                       getattr(tp, '_pack_', 0))
@@ -224,7 +225,6 @@
         res.__dict__['_index'] = -1
         return res
 
-
 class StructOrUnion(_CData):
     __metaclass__ = StructOrUnionMeta
 
@@ -234,9 +234,6 @@
         if ('_abstract_' in cls.__dict__ or cls is Structure 
                                          or cls is union.Union):
             raise TypeError("abstract class")
-        if hasattr(cls, '_swappedbytes_'):
-            raise NotImplementedError("missing in PyPy: structure/union with "
-                                      "swapped (non-native) byte ordering")
         if hasattr(cls, '_ffistruct_'):
             self.__dict__['_buffer'] = self._ffistruct_(autofree=True)
         return self
@@ -253,6 +250,17 @@
         for name, arg in kwds.items():
             self.__setattr__(name, arg)
 
+    def __getattribute__(self, item):
+        if item in (field[0] for field in object.__getattribute__(self, "_fields_"))\
+                and hasattr(self.__class__, '_swappedbytes_'):
+            self._swap_bytes(item, 'get')
+        return object.__getattribute__(self, item)
+
+    def __setattr__(self, key, value):
+        object.__setattr__(self,  key, value)
+        if key in (field[0] for field in self._fields_) and hasattr(self.__class__, '_swappedbytes_'):
+            self._swap_bytes(key, 'set')
+
     def _subarray(self, fieldtype, name):
         """Return a _rawffi array of length 1 whose address is the same as
         the address of the field 'name' of self."""
@@ -269,6 +277,63 @@
     def _to_ffi_param(self):
         return self._buffer
 
+    def _swap_bytes(self, field, get_or_set):
+        def swap_2(v):
+            return ((v >> 8) & 0x00FF) | ((v << 8) & 0xFF00)
+
+        def swap_4(v):
+            return ((v & 0x000000FF) << 24) | \
+                   ((v & 0x0000FF00) << 8) | \
+                   ((v & 0x00FF0000) >> 8) | \
+                   ((v >> 24) & 0xFF)
+
+        def swap_8(v):
+            return ((v & 0x00000000000000FFL) << 56) | \
+                   ((v & 0x000000000000FF00L) << 40) | \
+                   ((v & 0x0000000000FF0000L) << 24) | \
+                   ((v & 0x00000000FF000000L) << 8) | \
+                   ((v & 0x000000FF00000000L) >> 8) | \
+                   ((v & 0x0000FF0000000000L) >> 24) | \
+                   ((v & 0x00FF000000000000L) >> 40) | \
+                   ((v >> 56) & 0xFF)
+
+        def swap_double_float(v, typ):
+            from struct import pack, unpack
+            st = ''
+            if get_or_set == 'set':
+                if sys.byteorder == 'little':
+                    st = pack(''.join(['>', typ]), v)
+                else:
+                    st = pack(''.join(['<', typ]), v)
+                return unpack(typ, st)[0]
+            else:
+                packed = pack(typ, v)
+                if sys.byteorder == 'little':
+                    st = unpack(''.join(['>', typ]), packed)
+                else:
+                    st = unpack(''.join(['<', typ]), packed)
+                return st[0]
+
+        from ctypes import sizeof, c_double, c_float
+        sizeof_field = 0
+        typeof_field = None
+        for i in self._fields_:
+            if i[0] == field:
+                sizeof_field = sizeof(i[1])
+                typeof_field = i[1]
+        field_value = object.__getattribute__(self, field)
+        if typeof_field == c_float:
+            object.__setattr__(self, field, swap_double_float(field_value, 'f'))
+        elif typeof_field == c_double:
+            object.__setattr__(self, field, swap_double_float(field_value, 'd'))
+        else:
+            if sizeof_field == 2:
+                object.__setattr__(self, field, swap_2(field_value))
+            elif sizeof_field == 4:
+                object.__setattr__(self, field, swap_4(field_value))
+            elif sizeof_field == 8:
+                object.__setattr__(self, field, swap_8(field_value))
+
 
 class StructureMeta(StructOrUnionMeta):
     _is_union = False
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -460,6 +460,38 @@
         class X(Structure):
             _fields_ = [(u"i", c_int)]
 
+    def test_swapped_bytes(self):
+        import sys
+
+        for i in [c_short, c_int, c_long, c_longlong,
+                  c_float, c_double, c_ushort, c_uint,
+                  c_ulong, c_ulonglong]:
+            FIELDS = [
+                ('n', i)
+            ]
+
+            class Native(Structure):
+                _fields_ = FIELDS
+
+            class Big(BigEndianStructure):
+                _fields_ = FIELDS
+
+            class Little(LittleEndianStructure):
+                _fields_ = FIELDS
+
+            def dostruct(c):
+                ba = create_string_buffer(sizeof(c))
+                ms = c.from_buffer(ba)
+                ms.n = 0xff00
+                return repr(ba[:])
+
+            if sys.byteorder == 'little':
+                assert dostruct(Native) == dostruct(Little)
+                assert dostruct(Native) != dostruct(Big)
+            else:
+                assert dostruct(Native) == dostruct(Big)
+                assert dostruct(Native) != dostruct(Little)
+
 
 class TestPointerMember(BaseCTypesTestChecker):
     def test_1(self):

From pypy.commits at gmail.com  Sun Aug 27 02:54:07 2017
From: pypy.commits at gmail.com (smihnea)
Date: Sat, 26 Aug 2017 23:54:07 -0700 (PDT)
Subject: [pypy-commit] pypy pypy_swappedbytes: Final modifications ,
 1 test still unskipped in test_byteswap.py
Message-ID: <59a26c8f.f886df0a.782c.bf60@mx.google.com>

Author: Mihnea Saracin <mihnea.saracin at rinftech.com>
Branch: pypy_swappedbytes
Changeset: r92263:f611791f1958
Date: 2017-08-10 15:10 +0300
http://bitbucket.org/pypy/pypy/changeset/f611791f1958/

Log:	Final modifications , 1 test still unskipped in test_byteswap.py

diff --git a/lib-python/2.7/ctypes/test/test_byteswap.py b/lib-python/2.7/ctypes/test/test_byteswap.py
--- a/lib-python/2.7/ctypes/test/test_byteswap.py
+++ b/lib-python/2.7/ctypes/test/test_byteswap.py
@@ -23,7 +23,6 @@
             setattr(bits, "i%s" % i, 1)
             dump(bits)
 
-    @xfail
     def test_endian_short(self):
         if sys.byteorder == "little":
             self.assertIs(c_short.__ctype_le__, c_short)
@@ -51,7 +50,6 @@
         self.assertEqual(bin(s), "3412")
         self.assertEqual(s.value, 0x1234)
 
-    @xfail
     def test_endian_int(self):
         if sys.byteorder == "little":
             self.assertIs(c_int.__ctype_le__, c_int)
@@ -80,7 +78,6 @@
         self.assertEqual(bin(s), "78563412")
         self.assertEqual(s.value, 0x12345678)
 
-    @xfail
     def test_endian_longlong(self):
         if sys.byteorder == "little":
             self.assertIs(c_longlong.__ctype_le__, c_longlong)
@@ -109,7 +106,6 @@
         self.assertEqual(bin(s), "EFCDAB9078563412")
         self.assertEqual(s.value, 0x1234567890ABCDEF)
 
-    @xfail
     def test_endian_float(self):
         if sys.byteorder == "little":
             self.assertIs(c_float.__ctype_le__, c_float)
@@ -128,7 +124,6 @@
         self.assertAlmostEqual(s.value, math.pi, 6)
         self.assertEqual(bin(struct.pack(">f", math.pi)), bin(s))
 
-    @xfail
     def test_endian_double(self):
         if sys.byteorder == "little":
             self.assertIs(c_double.__ctype_le__, c_double)
@@ -156,7 +151,6 @@
         self.assertIs(c_char.__ctype_le__, c_char)
         self.assertIs(c_char.__ctype_be__, c_char)
 
-    @xfail
     def test_struct_fields_1(self):
         if sys.byteorder == "little":
             base = BigEndianStructure
@@ -221,7 +215,6 @@
                 self.assertEqual(s.point.x, 1)
                 self.assertEqual(s.point.y, 2)
 
-    @xfail
     def test_struct_fields_2(self):
         # standard packing in struct uses no alignment.
         # So, we have to align using pad bytes.
@@ -245,7 +238,6 @@
         s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14)
         self.assertEqual(bin(s1), bin(s2))
 
-    @xfail
     def test_unaligned_nonnative_struct_fields(self):
         if sys.byteorder == "little":
             base = BigEndianStructure
diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py
--- a/lib_pypy/_ctypes/primitive.py
+++ b/lib_pypy/_ctypes/primitive.py
@@ -61,6 +61,54 @@
 
 pyobj_container = GlobalPyobjContainer()
 
+def swap_bytes(value, sizeof, typeof, get_or_set):
+    def swap_2():
+        return ((value >> 8) & 0x00FF) | ((value << 8) & 0xFF00)
+
+    def swap_4():
+        return ((value & 0x000000FF) << 24) | \
+               ((value & 0x0000FF00) << 8) | \
+               ((value & 0x00FF0000) >> 8) | \
+               ((value >> 24) & 0xFF)
+
+    def swap_8():
+        return ((value & 0x00000000000000FFL) << 56) | \
+               ((value & 0x000000000000FF00L) << 40) | \
+               ((value & 0x0000000000FF0000L) << 24) | \
+               ((value & 0x00000000FF000000L) << 8) | \
+               ((value & 0x000000FF00000000L) >> 8) | \
+               ((value & 0x0000FF0000000000L) >> 24) | \
+               ((value & 0x00FF000000000000L) >> 40) | \
+               ((value >> 56) & 0xFF)
+
+    def swap_double_float(typ):
+        from struct import pack, unpack
+        if get_or_set == 'set':
+            if sys.byteorder == 'little':
+                st = pack(''.join(['>', typ]), value)
+            else:
+                st = pack(''.join(['<', typ]), value)
+            return unpack(typ, st)[0]
+        else:
+            packed = pack(typ, value)
+            if sys.byteorder == 'little':
+                st = unpack(''.join(['>', typ]), packed)
+            else:
+                st = unpack(''.join(['<', typ]), packed)
+            return st[0]
+
+    if typeof in ('c_float', 'c_float_le', 'c_float_be'):
+        return swap_double_float('f')
+    elif typeof in ('c_double', 'c_double_le', 'c_double_be'):
+        return swap_double_float('d')
+    else:
+        if sizeof == 2:
+            return swap_2()
+        elif sizeof == 4:
+            return swap_4()
+        elif sizeof == 8:
+            return swap_8()
+
 def generic_xxx_p_from_param(cls, value):
     if value is None:
         return cls(None)
@@ -271,6 +319,31 @@
             def _as_ffi_pointer_(self, ffitype):
                 return as_ffi_pointer(self, ffitype)
             result._as_ffi_pointer_ = _as_ffi_pointer_
+        if name[-2:] != '_p' and name[-3:] not in ('_le', '_be') \
+                and name not in ('c_wchar', '_SimpleCData', 'c_longdouble', 'c_bool', 'py_object'):
+            from sys import byteorder
+            if byteorder == 'big':
+                name += '_le'
+                swapped = self.__new__(self, name, bases, dct)
+                result.__ctype_le__ = swapped
+                result.__ctype_be__ = result
+                swapped.__ctype_be__ = result
+                swapped.__ctype_le__ = swapped
+            else:
+                name += '_be'
+                swapped = self.__new__(self, name, bases, dct)
+                result.__ctype_be__ = swapped
+                result.__ctype_le__ = result
+                swapped.__ctype_le__ = result
+                swapped.__ctype_be__ = swapped
+            from _ctypes import sizeof
+            def _getval(self):
+                return swap_bytes(self._buffer[0], sizeof(self), name, 'get')
+            def _setval(self, value):
+                d = result()
+                d.value = value
+                self._buffer[0] = swap_bytes(d.value, sizeof(self), name, 'set')
+            swapped.value = property(_getval, _setval)
 
         return result
 
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
@@ -230,10 +246,22 @@
 
     def __new__(cls, *args, **kwds):
         from _ctypes import union
-        self = super(_CData, cls).__new__(cls)
-        if ('_abstract_' in cls.__dict__ or cls is Structure 
+        if ('_abstract_' in cls.__dict__ or cls is Structure
                                          or cls is union.Union):
             raise TypeError("abstract class")
+        if hasattr(cls, '_swappedbytes_'):
+            fields = [None] * len(cls._fields_)
+            for i in range(len(cls._fields_)):
+                if cls._fields_[i][1] == cls._fields_[i][1].__dict__.get('__ctype_be__', None):
+                    swapped = cls._fields_[i][1].__dict__.get('__ctype_le__', cls._fields_[i][1])
+                else:
+                    swapped = cls._fields_[i][1].__dict__.get('__ctype_be__', cls._fields_[i][1])
+                if len(cls._fields_[i]) < 3:
+                    fields[i] = (cls._fields_[i][0], swapped)
+                else:
+                    fields[i] = (cls._fields_[i][0], swapped, cls._fields_[i][2])
+            names_and_fields(cls, fields, _CData, cls.__dict__.get('_anonymous_', None))
+        self = super(_CData, cls).__new__(cls)
         if hasattr(cls, '_ffistruct_'):
             self.__dict__['_buffer'] = self._ffistruct_(autofree=True)
         return self
@@ -250,17 +278,6 @@
         for name, arg in kwds.items():
             self.__setattr__(name, arg)
 
-    def __getattribute__(self, item):
-        if item in (field[0] for field in object.__getattribute__(self, "_fields_"))\
-                and hasattr(self.__class__, '_swappedbytes_'):
-            self._swap_bytes(item, 'get')
-        return object.__getattribute__(self, item)
-
-    def __setattr__(self, key, value):
-        object.__setattr__(self,  key, value)
-        if key in (field[0] for field in self._fields_) and hasattr(self.__class__, '_swappedbytes_'):
-            self._swap_bytes(key, 'set')
-
     def _subarray(self, fieldtype, name):
         """Return a _rawffi array of length 1 whose address is the same as
         the address of the field 'name' of self."""
@@ -277,63 +294,6 @@
     def _to_ffi_param(self):
         return self._buffer
 
-    def _swap_bytes(self, field, get_or_set):
-        def swap_2(v):
-            return ((v >> 8) & 0x00FF) | ((v << 8) & 0xFF00)
-
-        def swap_4(v):
-            return ((v & 0x000000FF) << 24) | \
-                   ((v & 0x0000FF00) << 8) | \
-                   ((v & 0x00FF0000) >> 8) | \
-                   ((v >> 24) & 0xFF)
-
-        def swap_8(v):
-            return ((v & 0x00000000000000FFL) << 56) | \
-                   ((v & 0x000000000000FF00L) << 40) | \
-                   ((v & 0x0000000000FF0000L) << 24) | \
-                   ((v & 0x00000000FF000000L) << 8) | \
-                   ((v & 0x000000FF00000000L) >> 8) | \
-                   ((v & 0x0000FF0000000000L) >> 24) | \
-                   ((v & 0x00FF000000000000L) >> 40) | \
-                   ((v >> 56) & 0xFF)
-
-        def swap_double_float(v, typ):
-            from struct import pack, unpack
-            st = ''
-            if get_or_set == 'set':
-                if sys.byteorder == 'little':
-                    st = pack(''.join(['>', typ]), v)
-                else:
-                    st = pack(''.join(['<', typ]), v)
-                return unpack(typ, st)[0]
-            else:
-                packed = pack(typ, v)
-                if sys.byteorder == 'little':
-                    st = unpack(''.join(['>', typ]), packed)
-                else:
-                    st = unpack(''.join(['<', typ]), packed)
-                return st[0]
-
-        from ctypes import sizeof, c_double, c_float
-        sizeof_field = 0
-        typeof_field = None
-        for i in self._fields_:
-            if i[0] == field:
-                sizeof_field = sizeof(i[1])
-                typeof_field = i[1]
-        field_value = object.__getattribute__(self, field)
-        if typeof_field == c_float:
-            object.__setattr__(self, field, swap_double_float(field_value, 'f'))
-        elif typeof_field == c_double:
-            object.__setattr__(self, field, swap_double_float(field_value, 'd'))
-        else:
-            if sizeof_field == 2:
-                object.__setattr__(self, field, swap_2(field_value))
-            elif sizeof_field == 4:
-                object.__setattr__(self, field, swap_4(field_value))
-            elif sizeof_field == 8:
-                object.__setattr__(self, field, swap_8(field_value))
-
 
 class StructureMeta(StructOrUnionMeta):
     _is_union = False
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -22,7 +22,6 @@
         assert X._fields_ == [("a", c_int)]
         assert Y._fields_ == [("b", c_int)]
         assert Z._fields_ == [("a", c_int)]
-
         assert Y._names_ == ['a', 'b']
 
     def test_subclass_delayed(self):
@@ -594,3 +593,13 @@
 
         x = X()
         assert x.x == 0
+
+    def test_duplicate_names(self):
+        class S(Structure):
+            _fields_ = [('a', c_int),
+                        ('b', c_int),
+                        ('a', c_byte)]
+        s = S(260, -123)
+        assert sizeof(s) == 3 * sizeof(c_int)
+        assert s.a == 4     # 256 + 4
+        assert s.b == -123

From pypy.commits at gmail.com  Sun Aug 27 02:54:11 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 26 Aug 2017 23:54:11 -0700 (PDT)
Subject: [pypy-commit] pypy default: hg merge pypy_swappedbytes
Message-ID: <59a26c93.45b01c0a.8eb03.884c@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92265:c64a2e680657
Date: 2017-08-27 08:52 +0200
http://bitbucket.org/pypy/pypy/changeset/c64a2e680657/

Log:	hg merge pypy_swappedbytes

	PR #560

	Adding swappedbytes support for ctypes.Structure

diff --git a/lib-python/2.7/ctypes/test/test_byteswap.py b/lib-python/2.7/ctypes/test/test_byteswap.py
--- a/lib-python/2.7/ctypes/test/test_byteswap.py
+++ b/lib-python/2.7/ctypes/test/test_byteswap.py
@@ -23,7 +23,6 @@
             setattr(bits, "i%s" % i, 1)
             dump(bits)
 
-    @xfail
     def test_endian_short(self):
         if sys.byteorder == "little":
             self.assertIs(c_short.__ctype_le__, c_short)
@@ -51,7 +50,6 @@
         self.assertEqual(bin(s), "3412")
         self.assertEqual(s.value, 0x1234)
 
-    @xfail
     def test_endian_int(self):
         if sys.byteorder == "little":
             self.assertIs(c_int.__ctype_le__, c_int)
@@ -80,7 +78,6 @@
         self.assertEqual(bin(s), "78563412")
         self.assertEqual(s.value, 0x12345678)
 
-    @xfail
     def test_endian_longlong(self):
         if sys.byteorder == "little":
             self.assertIs(c_longlong.__ctype_le__, c_longlong)
@@ -109,7 +106,6 @@
         self.assertEqual(bin(s), "EFCDAB9078563412")
         self.assertEqual(s.value, 0x1234567890ABCDEF)
 
-    @xfail
     def test_endian_float(self):
         if sys.byteorder == "little":
             self.assertIs(c_float.__ctype_le__, c_float)
@@ -128,7 +124,6 @@
         self.assertAlmostEqual(s.value, math.pi, 6)
         self.assertEqual(bin(struct.pack(">f", math.pi)), bin(s))
 
-    @xfail
     def test_endian_double(self):
         if sys.byteorder == "little":
             self.assertIs(c_double.__ctype_le__, c_double)
@@ -156,7 +151,6 @@
         self.assertIs(c_char.__ctype_le__, c_char)
         self.assertIs(c_char.__ctype_be__, c_char)
 
-    @xfail
     def test_struct_fields_1(self):
         if sys.byteorder == "little":
             base = BigEndianStructure
@@ -221,7 +215,6 @@
                 self.assertEqual(s.point.x, 1)
                 self.assertEqual(s.point.y, 2)
 
-    @xfail
     def test_struct_fields_2(self):
         # standard packing in struct uses no alignment.
         # So, we have to align using pad bytes.
@@ -245,7 +238,6 @@
         s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14)
         self.assertEqual(bin(s1), bin(s2))
 
-    @xfail
     def test_unaligned_nonnative_struct_fields(self):
         if sys.byteorder == "little":
             base = BigEndianStructure
diff --git a/lib-python/2.7/ctypes/test/test_unaligned_structures.py b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
--- a/lib-python/2.7/ctypes/test/test_unaligned_structures.py
+++ b/lib-python/2.7/ctypes/test/test_unaligned_structures.py
@@ -37,10 +37,7 @@
         for typ in byteswapped_structures:
 ##            print >> sys.stderr, typ.value
             self.assertEqual(typ.value.offset, 1)
-            try:
-                o = typ()
-            except NotImplementedError as e:
-                self.skipTest(str(e))    # for PyPy
+            o = typ()
             o.value = 4
             self.assertEqual(o.value, 4)
 
diff --git a/lib_pypy/_ctypes/primitive.py b/lib_pypy/_ctypes/primitive.py
--- a/lib_pypy/_ctypes/primitive.py
+++ b/lib_pypy/_ctypes/primitive.py
@@ -61,6 +61,54 @@
 
 pyobj_container = GlobalPyobjContainer()
 
+def swap_bytes(value, sizeof, typeof, get_or_set):
+    def swap_2():
+        return ((value >> 8) & 0x00FF) | ((value << 8) & 0xFF00)
+
+    def swap_4():
+        return ((value & 0x000000FF) << 24) | \
+               ((value & 0x0000FF00) << 8) | \
+               ((value & 0x00FF0000) >> 8) | \
+               ((value >> 24) & 0xFF)
+
+    def swap_8():
+        return ((value & 0x00000000000000FFL) << 56) | \
+               ((value & 0x000000000000FF00L) << 40) | \
+               ((value & 0x0000000000FF0000L) << 24) | \
+               ((value & 0x00000000FF000000L) << 8) | \
+               ((value & 0x000000FF00000000L) >> 8) | \
+               ((value & 0x0000FF0000000000L) >> 24) | \
+               ((value & 0x00FF000000000000L) >> 40) | \
+               ((value >> 56) & 0xFF)
+
+    def swap_double_float(typ):
+        from struct import pack, unpack
+        if get_or_set == 'set':
+            if sys.byteorder == 'little':
+                st = pack(''.join(['>', typ]), value)
+            else:
+                st = pack(''.join(['<', typ]), value)
+            return unpack(typ, st)[0]
+        else:
+            packed = pack(typ, value)
+            if sys.byteorder == 'little':
+                st = unpack(''.join(['>', typ]), packed)
+            else:
+                st = unpack(''.join(['<', typ]), packed)
+            return st[0]
+
+    if typeof in ('c_float', 'c_float_le', 'c_float_be'):
+        return swap_double_float('f')
+    elif typeof in ('c_double', 'c_double_le', 'c_double_be'):
+        return swap_double_float('d')
+    else:
+        if sizeof == 2:
+            return swap_2()
+        elif sizeof == 4:
+            return swap_4()
+        elif sizeof == 8:
+            return swap_8()
+
 def generic_xxx_p_from_param(cls, value):
     if value is None:
         return cls(None)
@@ -271,6 +319,31 @@
             def _as_ffi_pointer_(self, ffitype):
                 return as_ffi_pointer(self, ffitype)
             result._as_ffi_pointer_ = _as_ffi_pointer_
+        if name[-2:] != '_p' and name[-3:] not in ('_le', '_be') \
+                and name not in ('c_wchar', '_SimpleCData', 'c_longdouble', 'c_bool', 'py_object'):
+            from sys import byteorder
+            if byteorder == 'big':
+                name += '_le'
+                swapped = self.__new__(self, name, bases, dct)
+                result.__ctype_le__ = swapped
+                result.__ctype_be__ = result
+                swapped.__ctype_be__ = result
+                swapped.__ctype_le__ = swapped
+            else:
+                name += '_be'
+                swapped = self.__new__(self, name, bases, dct)
+                result.__ctype_be__ = swapped
+                result.__ctype_le__ = result
+                swapped.__ctype_le__ = result
+                swapped.__ctype_be__ = swapped
+            from _ctypes import sizeof
+            def _getval(self):
+                return swap_bytes(self._buffer[0], sizeof(self), name, 'get')
+            def _setval(self, value):
+                d = result()
+                d.value = value
+                self._buffer[0] = swap_bytes(d.value, sizeof(self), name, 'set')
+            swapped.value = property(_getval, _setval)
 
         return result
 
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -146,6 +146,7 @@
             obj._buffer.__setattr__(self.name, arg)
 
 
+
 def _set_shape(tp, rawfields, is_union=False):
     tp._ffistruct_ = _rawffi.Structure(rawfields, is_union,
                                       getattr(tp, '_pack_', 0))
@@ -240,19 +241,27 @@
         res.__dict__['_index'] = -1
         return res
 
-
 class StructOrUnion(_CData):
     __metaclass__ = StructOrUnionMeta
 
     def __new__(cls, *args, **kwds):
         from _ctypes import union
-        self = super(_CData, cls).__new__(cls)
-        if ('_abstract_' in cls.__dict__ or cls is Structure 
+        if ('_abstract_' in cls.__dict__ or cls is Structure
                                          or cls is union.Union):
             raise TypeError("abstract class")
         if hasattr(cls, '_swappedbytes_'):
-            raise NotImplementedError("missing in PyPy: structure/union with "
-                                      "swapped (non-native) byte ordering")
+            fields = [None] * len(cls._fields_)
+            for i in range(len(cls._fields_)):
+                if cls._fields_[i][1] == cls._fields_[i][1].__dict__.get('__ctype_be__', None):
+                    swapped = cls._fields_[i][1].__dict__.get('__ctype_le__', cls._fields_[i][1])
+                else:
+                    swapped = cls._fields_[i][1].__dict__.get('__ctype_be__', cls._fields_[i][1])
+                if len(cls._fields_[i]) < 3:
+                    fields[i] = (cls._fields_[i][0], swapped)
+                else:
+                    fields[i] = (cls._fields_[i][0], swapped, cls._fields_[i][2])
+            names_and_fields(cls, fields, _CData, cls.__dict__.get('_anonymous_', None))
+        self = super(_CData, cls).__new__(cls)
         if hasattr(cls, '_ffistruct_'):
             self.__dict__['_buffer'] = self._ffistruct_(autofree=True)
         return self
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_structures.py
@@ -22,7 +22,6 @@
         assert X._fields_ == [("a", c_int)]
         assert Y._fields_ == [("b", c_int)]
         assert Z._fields_ == [("a", c_int)]
-
         assert Y._names_ == ['a', 'b']
 
     def test_subclass_delayed(self):
@@ -460,6 +459,38 @@
         class X(Structure):
             _fields_ = [(u"i", c_int)]
 
+    def test_swapped_bytes(self):
+        import sys
+
+        for i in [c_short, c_int, c_long, c_longlong,
+                  c_float, c_double, c_ushort, c_uint,
+                  c_ulong, c_ulonglong]:
+            FIELDS = [
+                ('n', i)
+            ]
+
+            class Native(Structure):
+                _fields_ = FIELDS
+
+            class Big(BigEndianStructure):
+                _fields_ = FIELDS
+
+            class Little(LittleEndianStructure):
+                _fields_ = FIELDS
+
+            def dostruct(c):
+                ba = create_string_buffer(sizeof(c))
+                ms = c.from_buffer(ba)
+                ms.n = 0xff00
+                return repr(ba[:])
+
+            if sys.byteorder == 'little':
+                assert dostruct(Native) == dostruct(Little)
+                assert dostruct(Native) != dostruct(Big)
+            else:
+                assert dostruct(Native) == dostruct(Big)
+                assert dostruct(Native) != dostruct(Little)
+
 
 class TestPointerMember(BaseCTypesTestChecker):
     def test_1(self):

From pypy.commits at gmail.com  Sun Aug 27 02:54:09 2017
From: pypy.commits at gmail.com (arigo)
Date: Sat, 26 Aug 2017 23:54:09 -0700 (PDT)
Subject: [pypy-commit] pypy pypy_swappedbytes: close branch, ready to merge
Message-ID: <59a26c91.248fdf0a.7befb.352e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: pypy_swappedbytes
Changeset: r92264:1f217e18c43f
Date: 2017-08-27 08:47 +0200
http://bitbucket.org/pypy/pypy/changeset/1f217e18c43f/

Log:	close branch, ready to merge


From pypy.commits at gmail.com  Sun Aug 27 03:11:29 2017
From: pypy.commits at gmail.com (arigo)
Date: Sun, 27 Aug 2017 00:11:29 -0700 (PDT)
Subject: [pypy-commit] pypy default: Issue #2625
Message-ID: <59a270a1.c6c7df0a.797ca.c88c@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92266:99fff483bea8
Date: 2017-08-27 09:10 +0200
http://bitbucket.org/pypy/pypy/changeset/99fff483bea8/

Log:	Issue #2625

	ctypes.cast("some-string", TP)

diff --git a/lib_pypy/_ctypes/pointer.py b/lib_pypy/_ctypes/pointer.py
--- a/lib_pypy/_ctypes/pointer.py
+++ b/lib_pypy/_ctypes/pointer.py
@@ -142,6 +142,10 @@
         ptr._buffer = tp._ffiarray(1, autofree=True)
         ptr._buffer[0] = obj._buffer
         result = ptr
+    elif isinstance(obj, bytes):
+        result = tp()
+        result._buffer[0] = buffer(obj)._pypy_raw_address()
+        return result
     elif not (isinstance(obj, _CData) and type(obj)._is_pointer_like()):
         raise TypeError("cast() argument 1 must be a pointer, not %s"
                         % (type(obj),))

From pypy.commits at gmail.com  Sun Aug 27 03:12:37 2017
From: pypy.commits at gmail.com (arigo)
Date: Sun, 27 Aug 2017 00:12:37 -0700 (PDT)
Subject: [pypy-commit] pypy default: test_struct_struct now passes
Message-ID: <59a270e5.4b6b1c0a.c006f.b347@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92267:c4f7ee9d1f93
Date: 2017-08-27 09:12 +0200
http://bitbucket.org/pypy/pypy/changeset/c4f7ee9d1f93/

Log:	test_struct_struct now passes

diff --git a/lib-python/2.7/ctypes/test/test_byteswap.py b/lib-python/2.7/ctypes/test/test_byteswap.py
--- a/lib-python/2.7/ctypes/test/test_byteswap.py
+++ b/lib-python/2.7/ctypes/test/test_byteswap.py
@@ -186,7 +186,6 @@
                 pass
             self.assertRaises(TypeError, setattr, T, "_fields_", [("x", typ)])
 
-    @xfail
     def test_struct_struct(self):
         # nested structures with different byteorders
 

From pypy.commits at gmail.com  Sun Aug 27 13:22:23 2017
From: pypy.commits at gmail.com (mattip)
Date: Sun, 27 Aug 2017 10:22:23 -0700 (PDT)
Subject: [pypy-commit] pypy default: failing test ' ' + np.string_('abc')
 should not call np.string_.__radd__
Message-ID: <59a2ffcf.4f821c0a.b7c68.59e3@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92269:fc115074c233
Date: 2017-08-27 19:20 +0300
http://bitbucket.org/pypy/pypy/changeset/fc115074c233/

Log:	failing test ' ' + np.string_('abc') should not call
	np.string_.__radd__

diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py
--- a/pypy/module/cpyext/test/test_bytesobject.py
+++ b/pypy/module/cpyext/test/test_bytesobject.py
@@ -387,6 +387,16 @@
                     0                             /* tp_itemsize */
                     };
 
+                PyTypeObject PyGenArrType_Type = {
+                    PyObject_HEAD_INIT(NULL)
+                    0,                            /* ob_size */
+                    "bar.generic",                /* tp_name*/
+                    };
+
+                static PyObject*
+                gentype_add(PyObject* self, PyObject*other) {
+                    return PyString_FromString("gentype_add");
+                }
                     static PyObject *
                     stringtype_repr(PyObject *self)
                     {
@@ -443,15 +453,25 @@
                         memcpy(destptr, data, itemsize);
                         return obj;
                     }
+                static PyNumberMethods gentype_as_number;
             """, more_init = '''
+                long flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_CHECKTYPES;
+                PyGenArrType_Type.tp_flags = flags;
+                gentype_as_number.nb_add = gentype_add;
+                PyGenArrType_Type.tp_as_number = &gentype_as_number;
+                if (PyType_Ready(&PyGenArrType_Type) < 0) INITERROR;
+
                 PyStringArrType_Type.tp_alloc = NULL;
                 PyStringArrType_Type.tp_free = NULL;
-
                 PyStringArrType_Type.tp_repr = stringtype_repr;
                 PyStringArrType_Type.tp_str = stringtype_str;
-                PyStringArrType_Type.tp_flags = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
+                PyStringArrType_Type.tp_flags = flags;
                 PyStringArrType_Type.tp_itemsize = sizeof(char);
                 PyStringArrType_Type.tp_base = &PyString_Type;
+                Py_INCREF(&PyString_Type);
+                Py_INCREF(&PyGenArrType_Type);
+                PyStringArrType_Type.tp_bases = PyTuple_Pack(2,
+                                                &PyString_Type, &PyGenArrType_Type);
                 PyStringArrType_Type.tp_hash = PyString_Type.tp_hash;
                 if (PyType_Ready(&PyStringArrType_Type) < 0) INITERROR;
             ''')
@@ -461,6 +481,8 @@
         assert module.has_nb_add(a) is False
         assert type(a).__name__ == 'string_'
         assert a == 'abc'
+        ret = ' ' + a
+        assert ret == ' abc'
         assert 3 == module.get_len(a)
         b = module.newsubstr('')
         assert 0 == module.get_len(b)

From pypy.commits at gmail.com  Sun Aug 27 13:22:20 2017
From: pypy.commits at gmail.com (mattip)
Date: Sun, 27 Aug 2017 10:22:20 -0700 (PDT)
Subject: [pypy-commit] pypy default: test,
 fix - any str subtype should never have tp_as_a_number.* functions
 set
Message-ID: <59a2ffcc.521a1c0a.60bd.9d23@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92268:e45fdeb7813a
Date: 2017-08-26 20:02 +0300
http://bitbucket.org/pypy/pypy/changeset/e45fdeb7813a/

Log:	test, fix - any str subtype should never have tp_as_a_number.*
	functions set

diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py
--- a/pypy/module/cpyext/test/test_bytesobject.py
+++ b/pypy/module/cpyext/test/test_bytesobject.py
@@ -367,6 +367,16 @@
              """
                 return PyLong_FromLong(PyObject_Size(args));
              """),
+            ('has_nb_add', "METH_O",
+             '''
+                if (args->ob_type->tp_as_number == NULL) {
+                    Py_RETURN_FALSE;
+                }
+                if (args->ob_type->tp_as_number->nb_add == NULL) {
+                    Py_RETURN_FALSE;
+                }
+                Py_RETURN_TRUE;
+             '''),
             ], prologue="""
                 #include <Python.h>
                 PyTypeObject PyStringArrType_Type = {
@@ -447,6 +457,8 @@
             ''')
 
         a = module.newsubstr('abc')
+        assert module.has_nb_add('a') is False
+        assert module.has_nb_add(a) is False
         assert type(a).__name__ == 'string_'
         assert a == 'abc'
         assert 3 == module.get_len(a)
diff --git a/pypy/module/cpyext/typeobject.py b/pypy/module/cpyext/typeobject.py
--- a/pypy/module/cpyext/typeobject.py
+++ b/pypy/module/cpyext/typeobject.py
@@ -309,13 +309,17 @@
                 setattr(pto, slot_names[0], slot_func_helper)
         elif ((w_type is space.w_list or w_type is space.w_tuple) and
               slot_names[0] == 'c_tp_as_number'):
-            # XXX hack - hwo can we generalize this? The problem is method
+            # XXX hack - how can we generalize this? The problem is method
             # names like __mul__ map to more than one slot, and we have no
             # convenient way to indicate which slots CPython have filled
             #
             # We need at least this special case since Numpy checks that
             # (list, tuple) do __not__ fill tp_as_number
             pass
+        elif (space.issubtype_w(w_type, space.w_basestring) and
+                slot_names[0] == 'c_tp_as_number'):
+            # like above but for any str type
+            pass
         else:
             assert len(slot_names) == 2
             struct = getattr(pto, slot_names[0])

From pypy.commits at gmail.com  Sun Aug 27 15:18:43 2017
From: pypy.commits at gmail.com (mattip)
Date: Sun, 27 Aug 2017 12:18:43 -0700 (PDT)
Subject: [pypy-commit] pypy default: Backed out: fc115074c233,
 not clear if this is a PyPy bug or a CPython bug
Message-ID: <59a31b13.8ac4df0a.32ba9.2bb7@mx.google.com>

Author: Matti Picus <matti.picus at gmail.com>
Branch: 
Changeset: r92270:23392d66a346
Date: 2017-08-27 22:17 +0300
http://bitbucket.org/pypy/pypy/changeset/23392d66a346/

Log:	Backed out: fc115074c233, not clear if this is a PyPy bug or a
	CPython bug

diff --git a/pypy/module/cpyext/test/test_bytesobject.py b/pypy/module/cpyext/test/test_bytesobject.py
--- a/pypy/module/cpyext/test/test_bytesobject.py
+++ b/pypy/module/cpyext/test/test_bytesobject.py
@@ -387,16 +387,6 @@
                     0                             /* tp_itemsize */
                     };
 
-                PyTypeObject PyGenArrType_Type = {
-                    PyObject_HEAD_INIT(NULL)
-                    0,                            /* ob_size */
-                    "bar.generic",                /* tp_name*/
-                    };
-
-                static PyObject*
-                gentype_add(PyObject* self, PyObject*other) {
-                    return PyString_FromString("gentype_add");
-                }
                     static PyObject *
                     stringtype_repr(PyObject *self)
                     {
@@ -453,25 +443,15 @@
                         memcpy(destptr, data, itemsize);
                         return obj;
                     }
-                static PyNumberMethods gentype_as_number;
             """, more_init = '''
-                long flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_CHECKTYPES;
-                PyGenArrType_Type.tp_flags = flags;
-                gentype_as_number.nb_add = gentype_add;
-                PyGenArrType_Type.tp_as_number = &gentype_as_number;
-                if (PyType_Ready(&PyGenArrType_Type) < 0) INITERROR;
-
                 PyStringArrType_Type.tp_alloc = NULL;
                 PyStringArrType_Type.tp_free = NULL;
+
                 PyStringArrType_Type.tp_repr = stringtype_repr;
                 PyStringArrType_Type.tp_str = stringtype_str;
-                PyStringArrType_Type.tp_flags = flags;
+                PyStringArrType_Type.tp_flags = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
                 PyStringArrType_Type.tp_itemsize = sizeof(char);
                 PyStringArrType_Type.tp_base = &PyString_Type;
-                Py_INCREF(&PyString_Type);
-                Py_INCREF(&PyGenArrType_Type);
-                PyStringArrType_Type.tp_bases = PyTuple_Pack(2,
-                                                &PyString_Type, &PyGenArrType_Type);
                 PyStringArrType_Type.tp_hash = PyString_Type.tp_hash;
                 if (PyType_Ready(&PyStringArrType_Type) < 0) INITERROR;
             ''')
@@ -481,8 +461,6 @@
         assert module.has_nb_add(a) is False
         assert type(a).__name__ == 'string_'
         assert a == 'abc'
-        ret = ' ' + a
-        assert ret == ' abc'
         assert 3 == module.get_len(a)
         b = module.newsubstr('')
         assert 0 == module.get_len(b)

From pypy.commits at gmail.com  Sun Aug 27 16:35:54 2017
From: pypy.commits at gmail.com (arigo)
Date: Sun, 27 Aug 2017 13:35:54 -0700 (PDT)
Subject: [pypy-commit] pypy default: "Fix" the most glaring issue about
 sandboxing
Message-ID: <59a32d2a.05371c0a.6f0d0.5cc7@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r92271:2032a21a8320
Date: 2017-08-27 22:35 +0200
http://bitbucket.org/pypy/pypy/changeset/2032a21a8320/

Log:	"Fix" the most glaring issue about sandboxing

diff --git a/rpython/translator/sandbox/rsandbox.py b/rpython/translator/sandbox/rsandbox.py
--- a/rpython/translator/sandbox/rsandbox.py
+++ b/rpython/translator/sandbox/rsandbox.py
@@ -27,17 +27,20 @@
 ll_read_not_sandboxed = rposix.external('read',
                                         [rffi.INT, rffi.CCHARP, rffi.SIZE_T],
                                         rffi.SIZE_T,
-                                        sandboxsafe=True)
+                                        sandboxsafe=True,
+                                        _nowrapper=True)
 
 ll_write_not_sandboxed = rposix.external('write',
                                          [rffi.INT, rffi.CCHARP, rffi.SIZE_T],
                                          rffi.SIZE_T,
-                                         sandboxsafe=True)
+                                         sandboxsafe=True,
+                                         _nowrapper=True)
 
 
 @signature(types.int(), types.ptr(rffi.CCHARP.TO), types.int(),
     returns=types.none())
 def writeall_not_sandboxed(fd, buf, length):
+    fd = rffi.cast(rffi.INT, fd)
     while length > 0:
         size = rffi.cast(rffi.SIZE_T, length)
         count = rffi.cast(lltype.Signed, ll_write_not_sandboxed(fd, buf, size))
@@ -58,7 +61,8 @@
         buflen = self.buflen
         with lltype.scoped_alloc(rffi.CCHARP.TO, buflen) as buf:
             buflen = rffi.cast(rffi.SIZE_T, buflen)
-            count = ll_read_not_sandboxed(self.fd, buf, buflen)
+            fd = rffi.cast(rffi.INT, self.fd)
+            count = ll_read_not_sandboxed(fd, buf, buflen)
             count = rffi.cast(lltype.Signed, count)
             if count <= 0:
                 raise IOError

From pypy.commits at gmail.com  Mon Aug 28 05:51:53 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 28 Aug 2017 02:51:53 -0700 (PDT)
Subject: [pypy-commit] pypy default: generate tuples more efficiently to
 stop the occasional FailedHealthCheck
Message-ID: <59a3e7b9.a481df0a.1319b.03a8@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: 
Changeset: r92272:7153657512df
Date: 2017-08-28 11:45 +0200
http://bitbucket.org/pypy/pypy/changeset/7153657512df/

Log:	generate tuples more efficiently to stop the occasional
	FailedHealthCheck

diff --git a/rpython/jit/metainterp/test/test_bridgeopt.py b/rpython/jit/metainterp/test/test_bridgeopt.py
--- a/rpython/jit/metainterp/test/test_bridgeopt.py
+++ b/rpython/jit/metainterp/test/test_bridgeopt.py
@@ -76,8 +76,11 @@
 
 
 box_strategy = strategies.builds(InputArgInt) | strategies.builds(InputArgRef)
-tuples = strategies.tuples(box_strategy, strategies.booleans()).filter(
-        lambda (box, known_class): isinstance(box, InputArgRef) or not known_class)
+def _make_tup(box, known_class):
+    if isinstance(box, InputArgInt):
+        known_class = False
+    return box, known_class
+tuples = strategies.builds(_make_tup, box_strategy, strategies.booleans())
 boxes_known_classes = strategies.lists(tuples, min_size=1)
 
 @given(boxes_known_classes)

From pypy.commits at gmail.com  Mon Aug 28 05:51:57 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 28 Aug 2017 02:51:57 -0700 (PDT)
Subject: [pypy-commit] pypy default: optimize this sequence:
Message-ID: <59a3e7bd.0c181c0a.40289.8fb3@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: 
Changeset: r92274:c6568dda3266
Date: 2017-08-28 11:50 +0200
http://bitbucket.org/pypy/pypy/changeset/c6568dda3266/

Log:	optimize this sequence:

	 i2 = int_is_zero(i0) guard_false(i2) i1 = int_is_true(i0)
	guard_true(i1)

	(happens quite often in rpython list code, it seems)

diff --git a/rpython/jit/metainterp/optimizeopt/rewrite.py b/rpython/jit/metainterp/optimizeopt/rewrite.py
--- a/rpython/jit/metainterp/optimizeopt/rewrite.py
+++ b/rpython/jit/metainterp/optimizeopt/rewrite.py
@@ -10,7 +10,7 @@
 from rpython.jit.metainterp.optimizeopt.info import INFO_NONNULL, INFO_NULL
 from rpython.jit.metainterp.optimizeopt.util import _findall, make_dispatcher_method
 from rpython.jit.metainterp.resoperation import rop, ResOperation, opclasses,\
-     OpHelpers
+     OpHelpers, AbstractResOp
 from rpython.rlib.rarithmetic import highest_bit
 from rpython.rtyper.lltypesystem import llmemory
 from rpython.rtyper import rclass
@@ -490,6 +490,11 @@
 
     def postprocess_GUARD_TRUE(self, op):
         box = self.get_box_replacement(op.getarg(0))
+        if (isinstance(box, AbstractResOp) and
+                box.getopnum() == rop.INT_IS_TRUE):
+            # we can't use the (current) range analysis for this because
+            # "anything but 0" is not a valid range
+            self.pure_from_args(rop.INT_IS_ZERO, [box.getarg(0)], CONST_0)
         self.make_constant(box, CONST_1)
 
     def optimize_GUARD_FALSE(self, op):
@@ -497,6 +502,11 @@
 
     def postprocess_GUARD_FALSE(self, op):
         box = self.get_box_replacement(op.getarg(0))
+        if (isinstance(box, AbstractResOp) and
+                box.getopnum() == rop.INT_IS_ZERO):
+            # we can't use the (current) range analysis for this because
+            # "anything but 0" is not a valid range
+            self.pure_from_args(rop.INT_IS_TRUE, [box.getarg(0)], CONST_1)
         self.make_constant(box, CONST_0)
 
     def optimize_ASSERT_NOT_NONE(self, op):
diff --git a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -288,7 +288,6 @@
         self.optimize_loop(ops, expected)
 
     def test_int_is_true_is_zero(self):
-        py.test.skip("XXX implement me")
         ops = """
         [i0]
         i1 = int_is_true(i0)
@@ -305,6 +304,22 @@
         """
         self.optimize_loop(ops, expected)
 
+        ops = """
+        [i0]
+        i2 = int_is_zero(i0)
+        guard_false(i2) []
+        i1 = int_is_true(i0)
+        guard_true(i1) []
+        jump(i0)
+        """
+        expected = """
+        [i0]
+        i2 = int_is_zero(i0)
+        guard_false(i2) []
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_int_is_zero_int_is_true(self):
         ops = """
         [i0]

From pypy.commits at gmail.com  Mon Aug 28 05:51:55 2017
From: pypy.commits at gmail.com (cfbolz)
Date: Mon, 28 Aug 2017 02:51:55 -0700 (PDT)
Subject: [pypy-commit] pypy default: typo (the method is unused)
Message-ID: <59a3e7bb.d1c5df0a.53f6b.0167@mx.google.com>

Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
Branch: 
Changeset: r92273:3868025e1ee1
Date: 2017-08-28 11:49 +0200
http://bitbucket.org/pypy/pypy/changeset/3868025e1ee1/

Log:	typo (the method is unused)

diff --git a/rpython/jit/metainterp/resoperation.py b/rpython/jit/metainterp/resoperation.py
--- a/rpython/jit/metainterp/resoperation.py
+++ b/rpython/jit/metainterp/resoperation.py
@@ -400,7 +400,7 @@
         return rop.can_raise(self.getopnum())
 
     def is_foldable_guard(self):
-        return rop.is_foldable_guard(self.getopnun())
+        return rop.is_foldable_guard(self.getopnum())
 
     def is_primitive_array_access(self):
         """ Indicates that this operations loads/stores a

From pypy.commits at gmail.com  Mon Aug 28 13:29:18 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 28 Aug 2017 10:29:18 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: fix merge
Message-ID: <59a452ee.0783df0a.f01e4.4fc0@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92275:f3328bccb6b2
Date: 2017-08-28 18:28 +0100
http://bitbucket.org/pypy/pypy/changeset/f3328bccb6b2/

Log:	fix merge

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1660,7 +1660,6 @@
             # overwritten with a new error of the same type
             error = PyErr_Occurred(space)
             has_new_error = (error is not None) and (error is not preexist_error)
-            has_result = ret is not None
             if not expect_null and has_new_error and has_result:
                 raise oefmt(space.w_SystemError,
                             "An exception was set, but function returned a "
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -739,10 +739,10 @@
         # uncaught interplevel exceptions are turned into SystemError
         expected = "ZeroDivisionError('integer division or modulo by zero',)"
         exc = raises(SystemError, module.crash1)
-        assert exc.value[0] == expected
+        assert exc.value.args[0] == expected
 
         exc = raises(SystemError, module.crash2)
-        assert exc.value[0] == expected
+        assert exc.value.args[0] == expected
 
         # caught exception, api.cpython_api return value works
         assert module.crash3() == -1
@@ -750,7 +750,7 @@
         expected = 'An exception was set, but function returned a value'
         # PyPy only incompatibility/extension
         exc = raises(SystemError, module.crash4)
-        assert exc.value[0] == expected
+        assert exc.value.args[0] == expected
 
         # An exception was set by the previous call, it can pass
         # cleanly through a call that doesn't check error state
@@ -759,7 +759,7 @@
         # clear the exception but return NULL, signalling an error
         expected = 'Function returned a NULL result without setting an exception'
         exc = raises(SystemError, module.clear, None)
-        assert exc.value[0] == expected
+        assert exc.value.args[0] == expected
 
         # Set an exception and return NULL
         raises(TypeError, module.set, None)
@@ -770,7 +770,7 @@
         # Set an exception, but return non-NULL
         expected = 'An exception was set, but function returned a value'
         exc = raises(SystemError, module.set, 1)
-        assert exc.value[0] == expected
+        assert exc.value.args[0] == expected
 
 
         # Clear the exception and return a value, all is OK

From pypy.commits at gmail.com  Mon Aug 28 17:52:53 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 28 Aug 2017 14:52:53 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: hg merge py3.5
Message-ID: <59a490b5.d1471c0a.b22bc.1500@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92276:fdd1b3877173
Date: 2017-08-28 18:33 +0100
http://bitbucket.org/pypy/pypy/changeset/fdd1b3877173/

Log:	hg merge py3.5

diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -361,17 +361,20 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+            if _sys.maxint > 2 ** 32:
+                handle = int(handle)   # long -> int
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxint * 2 + 1))
-
+        return "<%s '%s', handle %x at %x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxint*2 + 1)),
+                id(self) & (_sys.maxint*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib-python/3/ctypes/__init__.py b/lib-python/3/ctypes/__init__.py
--- a/lib-python/3/ctypes/__init__.py
+++ b/lib-python/3/ctypes/__init__.py
@@ -346,16 +346,18 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxsize * 2 + 1))
+        return "<%s '%s', handle %x at 0x%x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxsize*2 + 1)),
+                id(self) & (_sys.maxsize*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib-python/3/datetime.py b/lib-python/3/datetime.py
--- a/lib-python/3/datetime.py
+++ b/lib-python/3/datetime.py
@@ -810,7 +810,8 @@
             month = self._month
         if day is None:
             day = self._day
-        return date(year, month, day)
+        # PyPy fix: returns type(self)() instead of date()
+        return type(self)(year, month, day)
 
     # Comparisons of date objects with other.
 
@@ -1285,7 +1286,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return time(hour, minute, second, microsecond, tzinfo)
+        # PyPy fix: returns type(self)() instead of time()
+        return type(self)(hour, minute, second, microsecond, tzinfo)
 
     # Pickle support.
 
@@ -1497,7 +1499,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return datetime(year, month, day, hour, minute, second, microsecond,
+        # PyPy fix: returns type(self)() instead of datetime()
+        return type(self)(year, month, day, hour, minute, second, microsecond,
                         tzinfo)
 
     def astimezone(self, tz=None):
diff --git a/lib-python/3/test/test_sysconfig.py b/lib-python/3/test/test_sysconfig.py
--- a/lib-python/3/test/test_sysconfig.py
+++ b/lib-python/3/test/test_sysconfig.py
@@ -397,9 +397,16 @@
             self.assertTrue('linux' in suffix, suffix)
         if re.match('(i[3-6]86|x86_64)$', machine):
             if ctypes.sizeof(ctypes.c_char_p()) == 4:
-                self.assertTrue(suffix.endswith('i386-linux-gnu.so') \
-                                or suffix.endswith('x86_64-linux-gnux32.so'),
-                                suffix)
+                self.assertTrue(
+                    suffix.endswith((
+                        'i386-linux-gnu.so',
+                        'i486-linux-gnu.so',
+                        'i586-linux-gnu.so',
+                        'i686-linux-gnu.so',
+                        'x86_64-linux-gnux32.so',
+                    )),
+                    suffix,
+                )
             else: # 8 byte pointer size
                 self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
 
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._handle.getaddressindll(name))
+        return self.from_address(dll.__pypy_dll__.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._handle
+        cdll = self.dll.__pypy_dll__
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
diff --git a/lib_pypy/pyrepl/reader.py b/lib_pypy/pyrepl/reader.py
--- a/lib_pypy/pyrepl/reader.py
+++ b/lib_pypy/pyrepl/reader.py
@@ -239,6 +239,10 @@
 
     def __init__(self, console):
         self.buffer = []
+        # Enable the use of `insert` without a `prepare` call - necessary to
+        # facilitate the tab completion hack implemented for
+        # <https://bugs.python.org/issue25660>.
+        self.pos = 0
         self.ps1 = "->> "
         self.ps2 = "/>> "
         self.ps3 = "|.. "
diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -314,7 +314,8 @@
         # history item: we use \r\n instead of just \n.  If the history
         # file is passed to GNU readline, the extra \r are just ignored.
         history = self.get_reader().history
-        f = open(os.path.expanduser(filename), 'r', encoding='utf-8')
+        f = open(os.path.expanduser(filename), 'r', encoding='utf-8',
+                 errors='replace')
         buffer = []
         for line in f:
             if line.endswith('\r\n'):
diff --git a/pypy/interpreter/module.py b/pypy/interpreter/module.py
--- a/pypy/interpreter/module.py
+++ b/pypy/interpreter/module.py
@@ -10,9 +10,10 @@
 class Module(W_Root):
     """A module."""
 
-    _immutable_fields_ = ["w_dict?"]
+    _immutable_fields_ = ["w_dict?", "w_userclass?"]
 
     _frozen = False
+    w_userclass = None
 
     def __init__(self, space, w_name, w_dict=None):
         self.space = space
@@ -148,6 +149,26 @@
                         self)
         return space.call_function(space.w_list, w_dict)
 
+    # These three methods are needed to implement '__class__' assignment
+    # between a module and a subclass of module.  They give every module
+    # the ability to have its '__class__' set, manually.  Note that if
+    # you instantiate a subclass of ModuleType in the first place, then
+    # you get an RPython instance of a subclass of Module created in the
+    # normal way by typedef.py.  That instance has got its own
+    # getclass(), getslotvalue(), etc. but provided it has no __slots__,
+    # it is compatible with ModuleType for '__class__' assignment.
+
+    def getclass(self, space):
+        if self.w_userclass is None:
+            return W_Root.getclass(self, space)
+        return self.w_userclass
+
+    def setclass(self, space, w_cls):
+        self.w_userclass = w_cls
+
+    def user_setup(self, space, w_subtype):
+        self.w_userclass = w_subtype
+
 
 def init_extra_module_attrs(space, w_mod):
     w_dict = w_mod.getdict(space)
diff --git a/pypy/interpreter/test/test_module.py b/pypy/interpreter/test/test_module.py
--- a/pypy/interpreter/test/test_module.py
+++ b/pypy/interpreter/test/test_module.py
@@ -220,3 +220,45 @@
         import sys
         m = type(sys).__new__(type(sys))
         assert not m.__dict__
+
+    def test_class_assignment_for_module(self):
+        import sys
+        modtype = type(sys)
+        class X(modtype):
+            _foobar_ = 42
+
+        m = X("yytest_moduleyy")
+        assert type(m) is m.__class__ is X
+        assert m._foobar_ == 42
+        m.__class__ = modtype
+        assert type(m) is m.__class__ is modtype
+        assert not hasattr(m, '_foobar_')
+
+        m = modtype("xxtest_modulexx")
+        assert type(m) is m.__class__ is modtype
+        m.__class__ = X
+        assert m._foobar_ == 42
+        assert type(m) is m.__class__ is X
+
+        sys.__class__ = modtype
+        assert type(sys) is sys.__class__ is modtype
+        sys.__class__ = X
+        assert sys._foobar_ == 42
+        sys.__class__ = modtype
+
+        class XX(modtype):
+            __slots__ = ['a', 'b']
+
+        x = XX("zztest_modulezz")
+        assert x.__class__ is XX
+        raises(AttributeError, "x.a")
+        x.a = 42
+        assert x.a == 42
+        x.a = 43
+        assert x.a == 43
+        assert 'a' not in x.__dict__
+        del x.a
+        raises(AttributeError, "x.a")
+        raises(AttributeError, "del x.a")
+        raises(TypeError, "x.__class__ = X")
+        raises(TypeError, "sys.__class__ = XX")
diff --git a/pypy/interpreter/typedef.py b/pypy/interpreter/typedef.py
--- a/pypy/interpreter/typedef.py
+++ b/pypy/interpreter/typedef.py
@@ -130,7 +130,7 @@
         return subcls
 _unique_subclass_cache = {}
 
-def _getusercls(cls, reallywantdict=False):
+def _getusercls(cls):
     from rpython.rlib import objectmodel
     from pypy.objspace.std.objectobject import W_ObjectObject
     from pypy.objspace.std.mapdict import (BaseUserClassMapdict,
@@ -144,7 +144,7 @@
     else:
         base_mixin = MapdictStorageMixin
     copy_methods = [BaseUserClassMapdict]
-    if reallywantdict or not typedef.hasdict:
+    if not typedef.hasdict:
         # the type has no dict, mapdict to provide the dict
         copy_methods.append(MapdictDictSupport)
         name += "Dict"
diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1661,7 +1661,7 @@
         assert cpyext_glob_tid_ptr[0] == 0
         cpyext_glob_tid_ptr[0] = tid
 
-        preexist_error = PyErr_Occurred(space) is not None
+        preexist_error = PyErr_Occurred(space)
         try:
             # Call the function
             result = call_external_function(func, *boxed_args)
@@ -1685,17 +1685,19 @@
                 has_result = ret is not None
 
             # Check for exception consistency
-            has_error = PyErr_Occurred(space) is not None
-            if not preexist_error:
-                if has_error and has_result:
-                    raise oefmt(space.w_SystemError,
-                                "An exception was set, but function returned a "
-                                "value")
-                elif not expect_null and not has_error and not has_result:
-                    raise oefmt(space.w_SystemError,
-                                "Function returned a NULL result without setting "
-                                "an exception")
-            if has_error:
+            # XXX best attempt, will miss preexisting error that is
+            # overwritten with a new error of the same type
+            error = PyErr_Occurred(space)
+            has_new_error = (error is not None) and (error is not preexist_error)
+            if not expect_null and has_new_error and has_result:
+                raise oefmt(space.w_SystemError,
+                            "An exception was set, but function returned a "
+                            "value")
+            elif not expect_null and not has_new_error and not has_result:
+                raise oefmt(space.w_SystemError,
+                            "Function returned a NULL result without setting "
+                            "an exception")
+            elif has_new_error:
                 state = space.fromcache(State)
                 state.check_and_raise_exception()
 
diff --git a/pypy/module/cpyext/include/object.h b/pypy/module/cpyext/include/object.h
--- a/pypy/module/cpyext/include/object.h
+++ b/pypy/module/cpyext/include/object.h
@@ -273,6 +273,11 @@
 #define _PyGC_FINALIZED(o) 1
 #define PyType_IS_GC(tp) 1
 
+#define PyObject_GC_Track(o)      do { } while(0)
+#define PyObject_GC_UnTrack(o)    do { } while(0)
+#define _PyObject_GC_TRACK(o)     do { } while(0)
+#define _PyObject_GC_UNTRACK(o)   do { } while(0)
+
 /* Utility macro to help write tp_traverse functions.
  * To use this macro, the tp_traverse function must name its arguments
  * "visit" and "arg".  This is intended to keep tp_traverse functions
diff --git a/pypy/module/cpyext/object.py b/pypy/module/cpyext/object.py
--- a/pypy/module/cpyext/object.py
+++ b/pypy/module/cpyext/object.py
@@ -80,24 +80,6 @@
 def PyObject_GC_Del(space, obj):
     PyObject_Free(space, obj)
 
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_Track(space, op):
-    """Adds the object op to the set of container objects tracked by the
-    collector.  The collector can run at unexpected times so objects must be
-    valid while being tracked.  This should be called once all the fields
-    followed by the tp_traverse handler become valid, usually near the
-    end of the constructor."""
-    pass
-
- at cpython_api([rffi.VOIDP], lltype.Void)
-def PyObject_GC_UnTrack(space, op):
-    """Remove the object op from the set of container objects tracked by the
-    collector.  Note that PyObject_GC_Track() can be called again on
-    this object to add it back to the set of tracked objects.  The deallocator
-    (tp_dealloc handler) should call this for the object before any of
-    the fields used by the tp_traverse handler become invalid."""
-    pass
-
 @cpython_api([PyObject], PyObjectP, error=CANNOT_FAIL)
 def _PyObject_GetDictPtr(space, op):
     return lltype.nullptr(PyObjectP.TO)
@@ -311,7 +293,7 @@
     PyErr_BadInternalCall(space)
 
 @cpython_api([PyObject, PyObject, rffi.INT_real], rffi.INT_real, error=-1)
-def PyObject_RichCompareBool(space, ref1, ref2, opid_int):
+def PyObject_RichCompareBool(space, w_o1, w_o2, opid_int):
     """Compare the values of o1 and o2 using the operation specified by opid,
     which must be one of Py_LT, Py_LE, Py_EQ,
     Py_NE, Py_GT, or Py_GE, corresponding to <,
@@ -321,13 +303,13 @@
     opid."""
     # Quick result when objects are the same.
     # Guarantees that identity implies equality.
-    if ref1 is ref2:
+    if space.is_w(w_o1, w_o2):
         opid = rffi.cast(lltype.Signed, opid_int)
         if opid == Py_EQ:
             return 1
         if opid == Py_NE:
             return 0 
-    w_res = PyObject_RichCompare(space, ref1, ref2, opid_int)
+    w_res = PyObject_RichCompare(space, w_o1, w_o2, opid_int)
     return int(space.is_true(w_res))
 
 @cpython_api([PyObject], PyObject, result_is_ll=True)
diff --git a/pypy/module/cpyext/sequence.py b/pypy/module/cpyext/sequence.py
--- a/pypy/module/cpyext/sequence.py
+++ b/pypy/module/cpyext/sequence.py
@@ -294,6 +294,23 @@
     def getitems_fixedsize(self, w_list):
         return self.getitems_unroll(w_list)
 
+    def copy_into(self, w_list, w_other):
+        w_other.strategy = self
+        w_other.lstorage = self.getstorage_copy(w_list)
+
+    def clone(self, w_list):
+        storage = self.getstorage_copy(w_list)
+        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
+                                                         self)
+        return w_clone
+
+    def getitems_copy(self, w_list):
+        return self.getitems(w_list) # getitems copies anyway
+
+    def getstorage_copy(self, w_list):
+        lst = self.getitems(w_list)
+        return self.erase(CPyListStorage(w_list.space, lst))
+
     #------------------------------------------
     # all these methods fail or switch strategy and then call ListObjectStrategy's method
 
@@ -301,23 +318,9 @@
         w_list.switch_to_object_strategy()
         w_list.strategy.setslice(w_list, start, stop, step, length)
 
-    def get_sizehint(self):
-        return -1
-
     def init_from_list_w(self, w_list, list_w):
         raise NotImplementedError
 
-    def clone(self, w_list):
-        storage = w_list.lstorage  # lstorage is tuple, no need to clone
-        w_clone = W_ListObject.from_storage_and_strategy(self.space, storage,
-                                                         self)
-        w_clone.switch_to_object_strategy()
-        return w_clone
-
-    def copy_into(self, w_list, w_other):
-        w_list.switch_to_object_strategy()
-        w_list.strategy.copy_into(w_list, w_other)
-
     def _resize_hint(self, w_list, hint):
         pass
 
@@ -325,13 +328,6 @@
         w_list.switch_to_object_strategy()
         return w_list.strategy.find(w_list, w_item, start, stop)
 
-    def getitems_copy(self, w_list):
-        w_list.switch_to_object_strategy()
-        return w_list.strategy.getitems_copy(w_list)
-
-    def getstorage_copy(self, w_list):
-        raise NotImplementedError
-
     def append(self, w_list, w_item):
         w_list.switch_to_object_strategy()
         w_list.strategy.append(w_list, w_item)
diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py
--- a/pypy/module/cpyext/stubs.py
+++ b/pypy/module/cpyext/stubs.py
@@ -625,18 +625,6 @@
     resized object or NULL on failure."""
     raise NotImplementedError
 
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_TRACK(space, op):
-    """A macro version of PyObject_GC_Track().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
- at cpython_api([PyObject], lltype.Void)
-def _PyObject_GC_UNTRACK(space, op):
-    """A macro version of PyObject_GC_UnTrack().  It should not be used for
-    extension modules."""
-    raise NotImplementedError
-
 @cpython_api([PyFrameObject], PyObject)
 def PyGen_New(space, frame):
     """Create and return a new generator object based on the frame object. A
@@ -1516,13 +1504,6 @@
     raise NotImplementedError
 
 
- at cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL)
-def PyType_IS_GC(space, o):
-    """Return true if the type object includes support for the cycle detector; this
-    tests the type flag Py_TPFLAGS_HAVE_GC."""
-    raise NotImplementedError
-
-
 @cpython_api([], rffi.INT_real, error=-1)
 def PyUnicode_ClearFreeList(space, ):
     """Clear the free list. Return the total number of freed items."""
diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -24,6 +24,10 @@
 def PyPy_Crash2(space):
     1/0
 
+ at api.cpython_api([api.PyObject], api.PyObject, result_is_ll=True)
+def PyPy_Noop(space, pyobj):
+    return pyobj
+
 class TestApi:
     def test_signature(self):
         common_functions = api.FUNCTIONS_BY_HEADER[api.pypy_decl]
@@ -665,6 +669,7 @@
         body = """
         PyAPI_FUNC(PyObject*) PyPy_Crash1(void);
         PyAPI_FUNC(long) PyPy_Crash2(void);
+        PyAPI_FUNC(PyObject*) PyPy_Noop(PyObject*);
         static PyObject* foo_crash1(PyObject* self, PyObject *args)
         {
             return PyPy_Crash1();
@@ -688,9 +693,27 @@
             int a = PyPy_Crash2();
             return PyFloat_FromDouble(a);
         }
+        static PyObject* foo_noop(PyObject* self, PyObject* args)
+        {
+            Py_INCREF(args);
+            return PyPy_Noop(args);
+        }
+        static PyObject* foo_set(PyObject* self, PyObject *args)
+        {
+            PyErr_SetString(PyExc_TypeError, "clear called with no error");
+            if (PyLong_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
+            return NULL;
+        }
         static PyObject* foo_clear(PyObject* self, PyObject *args)
         {
             PyErr_Clear();
+            if (PyLong_Check(args)) {
+                Py_INCREF(args);
+                return args;
+            }
             return NULL;
         }
         static PyMethodDef methods[] = {
@@ -698,7 +721,9 @@
             { "crash2", foo_crash2, METH_NOARGS },
             { "crash3", foo_crash3, METH_NOARGS },
             { "crash4", foo_crash4, METH_NOARGS },
-            { "clear",  foo_clear, METH_NOARGS },
+            { "clear",  foo_clear,  METH_O },
+            { "set",    foo_set,    METH_O },
+            { "noop",   foo_noop,   METH_O },
             { NULL }
         };
         static struct PyModuleDef moduledef = {
@@ -710,15 +735,46 @@
         };
         """
         module = self.import_module(name='foo', body=body)
+
         # uncaught interplevel exceptions are turned into SystemError
-        raises(SystemError, module.crash1)
-        raises(SystemError, module.crash2)
-        # caught exception
+        expected = "ZeroDivisionError('integer division or modulo by zero',)"
+        exc = raises(SystemError, module.crash1)
+        assert exc.value.args[0] == expected
+
+        exc = raises(SystemError, module.crash2)
+        assert exc.value.args[0] == expected
+
+        # caught exception, api.cpython_api return value works
         assert module.crash3() == -1
-        # An exception was set, but function returned a value
-        raises(SystemError, module.crash4)
-        # No exception set, but NULL returned
-        raises(SystemError, module.clear)
+
+        expected = 'An exception was set, but function returned a value'
+        # PyPy only incompatibility/extension
+        exc = raises(SystemError, module.crash4)
+        assert exc.value.args[0] == expected
+
+        # An exception was set by the previous call, it can pass
+        # cleanly through a call that doesn't check error state
+        assert module.noop(1) == 1
+
+        # clear the exception but return NULL, signalling an error
+        expected = 'Function returned a NULL result without setting an exception'
+        exc = raises(SystemError, module.clear, None)
+        assert exc.value.args[0] == expected
+
+        # Set an exception and return NULL
+        raises(TypeError, module.set, None)
+
+        # clear any exception and return a value
+        assert module.clear(1) == 1
+
+        # Set an exception, but return non-NULL
+        expected = 'An exception was set, but function returned a value'
+        exc = raises(SystemError, module.set, 1)
+        assert exc.value.args[0] == expected
+
+
+        # Clear the exception and return a value, all is OK
+        assert module.clear(1) == 1
 
     def test_new_exception(self):
         mod = self.import_extension('foo', [
diff --git a/pypy/module/cpyext/test/test_object.py b/pypy/module/cpyext/test/test_object.py
--- a/pypy/module/cpyext/test/test_object.py
+++ b/pypy/module/cpyext/test/test_object.py
@@ -416,7 +416,7 @@
         Py_buffer passed to it.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -468,7 +468,7 @@
         object.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -514,7 +514,7 @@
         PyBuffer_FillInfo fails if WRITABLE is passed but object is readonly.
         """
         module = self.import_extension('foo', [
-                ("fillinfo", "METH_VARARGS",
+                ("fillinfo", "METH_NOARGS",
                  """
     Py_buffer buf;
     PyObject *str = PyBytes_FromString("hello, world.");
@@ -541,7 +541,7 @@
         decremented by PyBuffer_Release.
         """
         module = self.import_extension('foo', [
-                ("release", "METH_VARARGS",
+                ("release", "METH_NOARGS",
                  """
     Py_buffer buf;
     buf.obj = PyBytes_FromString("release me!");
@@ -560,3 +560,20 @@
     Py_RETURN_NONE;
                  """)])
         assert module.release() is None
+
+
+class AppTestPyBuffer_Release(AppTestCpythonExtensionBase):
+    def test_richcomp_nan(self):
+        module = self.import_extension('foo', [
+               ("comp_eq", "METH_VARARGS",
+                """
+                PyObject *a = PyTuple_GetItem(args, 0);
+                PyObject *b = PyTuple_GetItem(args, 1);
+                int res = PyObject_RichCompareBool(a, b, Py_EQ);
+                return PyLong_FromLong(res);  
+                """),])
+        a = float('nan')
+        b = float('nan')
+        assert a is b
+        res = module.comp_eq(a, b)
+        assert res == 1
diff --git a/pypy/module/cpyext/test/test_sequence.py b/pypy/module/cpyext/test/test_sequence.py
--- a/pypy/module/cpyext/test/test_sequence.py
+++ b/pypy/module/cpyext/test/test_sequence.py
@@ -226,6 +226,15 @@
         w_l.inplace_mul(2)
         assert space.int_w(space.len(w_l)) == 10
 
+    def test_getstorage_copy(self, space, api):
+        w = space.wrap
+        w_l = w([1, 2, 3, 4])
+        api.PySequence_Fast(w_l, "foo") # converts
+
+        w_l1 = w([])
+        space.setitem(w_l1, space.newslice(w(0), w(0), w(1)), w_l)
+        assert map(space.unwrap, space.unpackiterable(w_l1)) == [1, 2, 3, 4]
+
 
 class AppTestSequenceObject(AppTestCpythonExtensionBase):
     def test_fast(self):
diff --git a/pypy/module/exceptions/interp_exceptions.py b/pypy/module/exceptions/interp_exceptions.py
--- a/pypy/module/exceptions/interp_exceptions.py
+++ b/pypy/module/exceptions/interp_exceptions.py
@@ -636,6 +636,14 @@
 else:
     WINERROR_TO_ERRNO, DEFAULT_WIN32_ERRNO = {}, 22 # EINVAL
 
+if rwin32.WIN32:
+    _winerror_property = dict(
+        winerror = readwrite_attrproperty_w('w_winerror', W_OSError),
+    )
+else:
+    _winerror_property = dict()
+
+
 W_OSError.typedef = TypeDef(
     'OSError',
     W_Exception.typedef,
@@ -648,9 +656,9 @@
     strerror = readwrite_attrproperty_w('w_strerror', W_OSError),
     filename = readwrite_attrproperty_w('w_filename', W_OSError),
     filename2= readwrite_attrproperty_w('w_filename2',W_OSError),
-    winerror = readwrite_attrproperty_w('w_winerror', W_OSError),
     characters_written = GetSetProperty(W_OSError.descr_get_written,
                                         W_OSError.descr_set_written),
+    **_winerror_property
     )
 
 W_BlockingIOError = _new_exception(
diff --git a/pypy/module/imp/test/test_app.py b/pypy/module/imp/test/test_app.py
--- a/pypy/module/imp/test/test_app.py
+++ b/pypy/module/imp/test/test_app.py
@@ -81,15 +81,17 @@
     def test_suffixes(self):
         import imp
         for suffix, mode, type in imp.get_suffixes():
-            if mode == imp.PY_SOURCE:
+            if type == imp.PY_SOURCE:
                 assert suffix == '.py'
-                assert type == 'r'
-            elif mode == imp.PY_COMPILED:
+                assert mode == 'r'
+            elif type == imp.PY_COMPILED:
                 assert suffix in ('.pyc', '.pyo')
-                assert type == 'rb'
-            elif mode == imp.C_EXTENSION:
+                assert mode == 'rb'
+            elif type == imp.C_EXTENSION:
                 assert suffix.endswith(('.pyd', '.so'))
-                assert type == 'rb'
+                assert mode == 'rb'
+            else:
+                assert False, ("Unknown type", suffix, mode, type)
 
     def test_ext_suffixes(self):
         import _imp
diff --git a/pypy/module/readline/test/test_readline.py b/pypy/module/readline/test/test_readline.py
--- a/pypy/module/readline/test/test_readline.py
+++ b/pypy/module/readline/test/test_readline.py
@@ -29,3 +29,14 @@
             readline.add_history("dummy")
         assert readline.get_history_item(1) ==  "entr&#233;e 1"
         assert readline.get_history_item(2) == "entr&#233;e 22"
+
+
+    def test_insert_text_leading_tab(self):
+        """
+        A literal tab can be inserted at the beginning of a line.
+
+        See <https://bugs.python.org/issue25660>
+        """
+        import readline
+        readline.insert_text("\t")
+        assert readline.get_line_buffer() == b"\t"
diff --git a/pypy/module/test_lib_pypy/README.txt b/pypy/module/test_lib_pypy/README.txt
--- a/pypy/module/test_lib_pypy/README.txt
+++ b/pypy/module/test_lib_pypy/README.txt
@@ -1,4 +1,7 @@
 This directory contains app-level tests are supposed to be run *after*
 translation. So you run them by saying:
 
-pypy pytest.py <testfile.py>
+../../goal/pypy-c pytest.py <testfile.py>
+
+Note that if you run it with a PyPy from elsewhere, it will not pick
+up the changes to lib-python and lib_pypy.
diff --git a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
--- a/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
+++ b/pypy/module/test_lib_pypy/ctypes_tests/test_loading.py
@@ -43,6 +43,12 @@
                 cdll.LoadLibrary(lib)
                 CDLL(lib)
 
+    def test__handle(self):
+        lib = find_library("c")
+        if lib:
+            cdll = CDLL(lib)
+            assert type(cdll._handle) in (int, long)
+
     if os.name in ("nt", "ce"):
         def test_load_library(self):
             if is_resource_enabled("printing"):
diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py
--- a/pypy/objspace/std/bytesobject.py
+++ b/pypy/objspace/std/bytesobject.py
@@ -364,8 +364,8 @@
         characters, all remaining cased characters have lowercase.
         """
 
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         """B.translate(table[, deletechars]) -> copy of B
 
         Return a copy of the string B, where all characters occurring
diff --git a/pypy/objspace/std/objectobject.py b/pypy/objspace/std/objectobject.py
--- a/pypy/objspace/std/objectobject.py
+++ b/pypy/objspace/std/objectobject.py
@@ -141,13 +141,17 @@
 
 def descr_set___class__(space, w_obj, w_newcls):
     from pypy.objspace.std.typeobject import W_TypeObject
+    from pypy.interpreter.module import Module
+    #
     if not isinstance(w_newcls, W_TypeObject):
         raise oefmt(space.w_TypeError,
-                    "__class__ must be set to new-style class, not '%T' "
+                    "__class__ must be set to a class, not '%T' "
                     "object", w_newcls)
-    if not w_newcls.is_heaptype():
+    if not (w_newcls.is_heaptype() or
+            w_newcls is space.gettypeobject(Module.typedef)):
         raise oefmt(space.w_TypeError,
-                    "__class__ assignment: only for heap types")
+                    "__class__ assignment only supported for heap types "
+                    "or ModuleType subclasses")
     w_oldcls = space.type(w_obj)
     assert isinstance(w_oldcls, W_TypeObject)
     if (w_oldcls.get_full_instance_layout() ==
diff --git a/pypy/objspace/std/stringmethods.py b/pypy/objspace/std/stringmethods.py
--- a/pypy/objspace/std/stringmethods.py
+++ b/pypy/objspace/std/stringmethods.py
@@ -742,8 +742,8 @@
     DEFAULT_NOOP_TABLE = ''.join([chr(i) for i in range(256)])
 
     # for bytes and bytearray, overridden by unicode
-    @unwrap_spec(w_deletechars=WrappedDefault(''))
-    def descr_translate(self, space, w_table, w_deletechars):
+    @unwrap_spec(w_delete=WrappedDefault(''))
+    def descr_translate(self, space, w_table, w_delete):
         if space.is_w(w_table, space.w_None):
             table = self.DEFAULT_NOOP_TABLE
         else:
@@ -753,7 +753,7 @@
                             "translation table must be 256 characters long")
 
         string = self._val(space)
-        deletechars = self._op_val(space, w_deletechars)
+        deletechars = self._op_val(space, w_delete)
         if len(deletechars) == 0:
             buf = self._builder(len(string))
             for char in string:
diff --git a/pypy/objspace/std/test/test_typeobject.py b/pypy/objspace/std/test/test_typeobject.py
--- a/pypy/objspace/std/test/test_typeobject.py
+++ b/pypy/objspace/std/test/test_typeobject.py
@@ -1284,6 +1284,65 @@
         raises(ValueError, type, 'A\x00B', (), {})
         raises(TypeError, type, b'A', (), {})
 
+    def test_incomplete_extend(self): """
+        # Extending an unitialized type with type.__mro__ is None must
+        # throw a reasonable TypeError exception, instead of failing
+        # with a segfault.
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ != 'X':
+                    try:
+                        class X(cls):
+                            pass
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class A(metaclass=M):
+            pass
+        assert found == [1]
+        """
+
+    def test_incomplete_extend_2(self): """
+        # Same as test_incomplete_extend, with multiple inheritance
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ == 'Second':
+                    try:
+                        class X(First, cls):
+                            pass
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class Base(metaclass=M):
+            pass
+        class First(Base):
+            pass
+        class Second(Base):
+            pass
+        assert found == [1]
+        """
+
+    def test_incomplete_extend_3(self): """
+        # this case "works", but gives a slightly strange error message
+        # on both CPython and PyPy
+        class M(type):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ == 'A':
+                    try:
+                        Base.__new__(cls)
+                    except TypeError:
+                        found.append(1)
+                return type.mro(cls)
+        found = []
+        class Base(metaclass=M):
+            pass
+        class A(Base):
+            pass
+        assert found == [1]
+        """
+
 
 class AppTestWithMethodCacheCounter:
     spaceconfig = {"objspace.std.withmethodcachecounter": True}
diff --git a/pypy/objspace/std/typeobject.py b/pypy/objspace/std/typeobject.py
--- a/pypy/objspace/std/typeobject.py
+++ b/pypy/objspace/std/typeobject.py
@@ -546,19 +546,24 @@
         space = self.space
         if self.is_heaptype():
             return self.getdictvalue(space, '__module__')
+        elif self.is_cpytype():
+            dot = self.name.rfind('.')
         else:
             dot = self.name.find('.')
-            if dot >= 0:
-                mod = self.name[:dot]
-            else:
-                mod = "builtins"
-            return space.newtext(mod)
+        if dot >= 0:
+            mod = self.name[:dot]
+        else:
+            mod = "builtins"
+        return space.newtext(mod)
 
     def getname(self, space):
         if self.is_heaptype():
             result = self.name
         else:
-            dot = self.name.find('.')
+            if self.is_cpytype():
+                dot = self.name.rfind('.')
+            else:
+                dot = self.name.find('.')
             if dot >= 0:
                 result = self.name[dot+1:]
             else:
@@ -1036,6 +1041,9 @@
     for w_candidate in bases_w:
         if not isinstance(w_candidate, W_TypeObject):
             continue
+        if not w_candidate.hasmro:
+            raise oefmt(w_candidate.space.w_TypeError,
+                        "Cannot extend an incomplete type '%N'", w_candidate)
         if w_bestbase is None:
             w_bestbase = w_candidate   # for now
             continue
diff --git a/rpython/jit/backend/llsupport/regalloc.py b/rpython/jit/backend/llsupport/regalloc.py
--- a/rpython/jit/backend/llsupport/regalloc.py
+++ b/rpython/jit/backend/llsupport/regalloc.py
@@ -552,10 +552,11 @@
             self.reg_bindings[result_v] = loc
             return loc
         if v not in self.reg_bindings:
+            # v not in a register. allocate one for result_v and move v there
             prev_loc = self.frame_manager.loc(v)
-            loc = self.force_allocate_reg(v, forbidden_vars)
+            loc = self.force_allocate_reg(result_v, forbidden_vars)
             self.assembler.regalloc_mov(prev_loc, loc)
-        assert v in self.reg_bindings
+            return loc
         if self.longevity[v][1] > self.position:
             # we need to find a new place for variable v and
             # store result in the same place
diff --git a/rpython/jit/backend/x86/assembler.py b/rpython/jit/backend/x86/assembler.py
--- a/rpython/jit/backend/x86/assembler.py
+++ b/rpython/jit/backend/x86/assembler.py
@@ -504,7 +504,7 @@
         clt.frame_info = rffi.cast(jitframe.JITFRAMEINFOPTR, frame_info)
         clt.frame_info.clear() # for now
 
-        if log:
+        if log or self._debug:
             number = looptoken.number
             operations = self._inject_debugging_code(looptoken, operations,
                                                      'e', number)
@@ -589,7 +589,7 @@
                 faildescr.adr_jump_offset)
         self.mc.force_frame_size(DEFAULT_FRAME_BYTES)
         descr_number = compute_unique_id(faildescr)
-        if log:
+        if log or self._debug:
             operations = self._inject_debugging_code(faildescr, operations,
                                                      'b', descr_number)
         arglocs = self.rebuild_faillocs_from_descr(faildescr, inputargs)
@@ -1618,18 +1618,6 @@
         else:
             not_implemented("save_into_mem size = %d" % size)
 
-    def _genop_getfield(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc, sign_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
-        source_addr = AddressLoc(base_loc, ofs_loc)
-        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
-
-    genop_getfield_gc_i = _genop_getfield
-    genop_getfield_gc_r = _genop_getfield
-    genop_getfield_gc_f = _genop_getfield
-    genop_getfield_raw_i = _genop_getfield
-    genop_getfield_raw_f = _genop_getfield
-
     def _genop_gc_load(self, op, arglocs, resloc):
         base_loc, ofs_loc, size_loc, sign_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
diff --git a/rpython/jit/backend/x86/regalloc.py b/rpython/jit/backend/x86/regalloc.py
--- a/rpython/jit/backend/x86/regalloc.py
+++ b/rpython/jit/backend/x86/regalloc.py
@@ -1305,7 +1305,7 @@
         self.rm.possibly_free_var(tmpbox_high)
 
     def compute_hint_frame_locations(self, operations):
-        # optimization only: fill in the 'hint_frame_locations' dictionary
+        # optimization only: fill in the 'hint_frame_pos' dictionary
         # of 'fm' based on the JUMP at the end of the loop, by looking
         # at where we would like the boxes to be after the jump.
         op = operations[-1]
@@ -1320,7 +1320,7 @@
             self._compute_hint_frame_locations_from_descr(descr)
         #else:
         #   The loop ends in a JUMP going back to a LABEL in the same loop.
-        #   We cannot fill 'hint_frame_locations' immediately, but we can
+        #   We cannot fill 'hint_frame_pos' immediately, but we can
         #   wait until the corresponding consider_label() to know where the
         #   we would like the boxes to be after the jump.
 
diff --git a/rpython/rlib/rposix.py b/rpython/rlib/rposix.py
--- a/rpython/rlib/rposix.py
+++ b/rpython/rlib/rposix.py
@@ -205,6 +205,18 @@
         if not is_valid_fd(fd):
             from errno import EBADF
             raise OSError(EBADF, 'Bad file descriptor')
+
+    def _bound_for_write(fd, count):
+        if count > 32767 and c_isatty(fd):
+            # CPython Issue #11395, PyPy Issue #2636: the Windows console
+            # returns an error (12: not enough space error) on writing into
+            # stdout if stdout mode is binary and the length is greater than
+            # 66,000 bytes (or less, depending on heap usage).  Can't easily
+            # test that, because we need 'fd' to be non-redirected...
+            count = 32767
+        elif count > 0x7fffffff:
+            count = 0x7fffffff
+        return count
 else:
     def is_valid_fd(fd):
         return 1
@@ -213,6 +225,9 @@
     def validate_fd(fd):
         pass
 
+    def _bound_for_write(fd, count):
+        return count
+
 def closerange(fd_low, fd_high):
     # this behaves like os.closerange() from Python 2.6.
     for fd in xrange(fd_low, fd_high):
@@ -449,6 +464,7 @@
 def write(fd, data):
     count = len(data)
     validate_fd(fd)
+    count = _bound_for_write(fd, count)
     with rffi.scoped_nonmovingbuffer(data) as buf:
         return handle_posix_error('write', c_write(fd, buf, count))
 
diff --git a/rpython/rtyper/tool/rffi_platform.py b/rpython/rtyper/tool/rffi_platform.py
--- a/rpython/rtyper/tool/rffi_platform.py
+++ b/rpython/rtyper/tool/rffi_platform.py
@@ -710,7 +710,8 @@
         size, _ = expected_size_and_sign
         return lltype.FixedSizeArray(fieldtype.OF, size/_sizeof(fieldtype.OF))
     raise TypeError("conflict between translating python and compiler field"
-                    " type %r for %r" % (fieldtype, fieldname))
+                    " type %r for symbol %r, expected size+sign %r" % (
+                        fieldtype, fieldname, expected_size_and_sign))
 
 def expose_value_as_rpython(value):
     if intmask(value) == value:

From pypy.commits at gmail.com  Mon Aug 28 17:52:56 2017
From: pypy.commits at gmail.com (rlamy)
Date: Mon, 28 Aug 2017 14:52:56 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: fix translation
Message-ID: <59a490b8.83341c0a.89b94.6c75@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92277:4581a4a84e6f
Date: 2017-08-28 22:52 +0100
http://bitbucket.org/pypy/pypy/changeset/4581a4a84e6f/

Log:	fix translation

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -1522,6 +1522,7 @@
             look_for += ' or ' + also_look_for
         else:
             look_for = also_look_for
+    assert look_for is not None
     msg = u"function %s not found in library %s" % (
         look_for.decode('utf-8'), space.unicode_w(space.newfilename(path)))
     w_path = space.newfilename(path)

From pypy.commits at gmail.com  Tue Aug 29 22:56:36 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 29 Aug 2017 19:56:36 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: clear exception when module exec
 sets one without reporting failure
Message-ID: <59a62964.50901c0a.cc0d0.3bf8@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92279:7a20909775d3
Date: 2017-08-30 03:55 +0100
http://bitbucket.org/pypy/pypy/changeset/7a20909775d3/

Log:	clear exception when module exec sets one without reporting failure

diff --git a/pypy/module/cpyext/modsupport.py b/pypy/module/cpyext/modsupport.py
--- a/pypy/module/cpyext/modsupport.py
+++ b/pypy/module/cpyext/modsupport.py
@@ -152,15 +152,16 @@
             execf = rffi.cast(execfunctype, cur_slot[0].c_value)
             res = generic_cpy_call(space, execf, w_mod)
             has_error = PyErr_Occurred(space) is not None
+            state = space.fromcache(State)
             if rffi.cast(lltype.Signed, res):
                 if has_error:
-                    state = space.fromcache(State)
                     state.check_and_raise_exception()
                 else:
                     raise oefmt(space.w_SystemError,
                                 "execution of module %S failed without "
                                 "setting an exception", w_mod.w_name)
             if has_error:
+                state.clear_exception()
                 raise oefmt(space.w_SystemError,
                             "execution of module %S raised unreported "
                             "exception", w_mod.w_name)

From pypy.commits at gmail.com  Tue Aug 29 22:56:33 2017
From: pypy.commits at gmail.com (rlamy)
Date: Tue, 29 Aug 2017 19:56:33 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: check for stray exceptions at test
 teardown
Message-ID: <59a62961.8fbadf0a.19fb5.6542@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92278:48e754e6916d
Date: 2017-08-30 03:50 +0100
http://bitbucket.org/pypy/pypy/changeset/48e754e6916d/

Log:	check for stray exceptions at test teardown

diff --git a/pypy/module/cpyext/test/test_cpyext.py b/pypy/module/cpyext/test/test_cpyext.py
--- a/pypy/module/cpyext/test/test_cpyext.py
+++ b/pypy/module/cpyext/test/test_cpyext.py
@@ -363,6 +363,8 @@
         for name in self.imported_module_names:
             self.unimport_module(name)
         self.cleanup()
+        state = self.space.fromcache(State)
+        assert not state.operror
 
 
 class AppTestCpythonExtension(AppTestCpythonExtensionBase):

From pypy.commits at gmail.com  Wed Aug 30 07:48:34 2017
From: pypy.commits at gmail.com (rlamy)
Date: Wed, 30 Aug 2017 04:48:34 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Merged py3.5 into
 py3.5-sendmsg-recvmsg
Message-ID: <59a6a612.0ac41c0a.f4fc1.5d71@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92280:05eff6c713c4
Date: 2017-08-30 12:48 +0100
http://bitbucket.org/pypy/pypy/changeset/05eff6c713c4/

Log:	Merged py3.5 into py3.5-sendmsg-recvmsg

diff too long, truncating to 2000 out of 25457 lines

diff --git a/.hgignore b/.hgignore
--- a/.hgignore
+++ b/.hgignore
@@ -27,16 +27,17 @@
 ^pypy/module/cpyext/test/.+\.manifest$
 ^pypy/module/test_lib_pypy/ctypes_tests/.+\.o$
 ^pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test\.o$
-^pypy/module/cppyy/src/.+\.o$
-^pypy/module/cppyy/bench/.+\.so$
-^pypy/module/cppyy/bench/.+\.root$
-^pypy/module/cppyy/bench/.+\.d$
-^pypy/module/cppyy/src/.+\.errors$
-^pypy/module/cppyy/test/.+_rflx\.cpp$
-^pypy/module/cppyy/test/.+\.so$
-^pypy/module/cppyy/test/.+\.rootmap$
-^pypy/module/cppyy/test/.+\.exe$
-^pypy/module/cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/src/.+\.o$
+^pypy/module/_cppyy/bench/.+\.so$
+^pypy/module/_cppyy/bench/.+\.root$
+^pypy/module/_cppyy/bench/.+\.d$
+^pypy/module/_cppyy/src/.+\.errors$
+^pypy/module/_cppyy/test/.+_rflx\.cpp$
+^pypy/module/_cppyy/test/.+\.so$
+^pypy/module/_cppyy/test/.+\.rootmap$
+^pypy/module/_cppyy/test/.+\.exe$
+^pypy/module/_cppyy/test/.+_cint.h$
+^pypy/module/_cppyy/.+/*\.pcm$
 ^pypy/module/test_lib_pypy/cffi_tests/__pycache__.+$
 ^pypy/doc/.+\.html$
 ^pypy/doc/config/.+\.rst$
@@ -93,6 +94,3 @@
 ^release/
 ^rpython/_cache$
 
-pypy/module/cppyy/.+/*\.pcm
-
-
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@
 RUNINTERP = $(PYPY_EXECUTABLE)
 endif
 
-.PHONY: cffi_imports
+.PHONY: pypy-c cffi_imports
 
 pypy-c:
 	@echo
@@ -32,7 +32,7 @@
 	@echo "===================================================================="
 	@echo
 	@sleep 5
-	$(RUNINTERP) rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+	cd pypy/goal && $(RUNINTERP) ../../rpython/bin/rpython -Ojit targetpypystandalone.py
 
 # Note: the -jN option, or MAKEFLAGS=-jN, are not usable.  They are
 # replaced with an opaque --jobserver option by the time this Makefile
@@ -40,4 +40,4 @@
 # http://lists.gnu.org/archive/html/help-make/2010-08/msg00106.html
 
 cffi_imports: pypy-c
-	PYTHONPATH=. ./pypy-c pypy/tool/build_cffi_imports.py || /bin/true
+	PYTHONPATH=. pypy/goal/pypy-c pypy/tool/build_cffi_imports.py || /bin/true
diff --git a/lib-python/2.7/ctypes/__init__.py b/lib-python/2.7/ctypes/__init__.py
--- a/lib-python/2.7/ctypes/__init__.py
+++ b/lib-python/2.7/ctypes/__init__.py
@@ -361,17 +361,20 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+            if _sys.maxint > 2 ** 32:
+                handle = int(handle)   # long -> int
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxint * 2 + 1))
-
+        return "<%s '%s', handle %x at %x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxint*2 + 1)),
+                id(self) & (_sys.maxint*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py
--- a/lib-python/2.7/distutils/sysconfig_pypy.py
+++ b/lib-python/2.7/distutils/sysconfig_pypy.py
@@ -218,6 +218,10 @@
 
         compiler.shared_lib_extension = so_ext
 
+def get_config_h_filename():
+    """Returns the path of pyconfig.h."""
+    inc_dir = get_python_inc(plat_specific=1)
+    return os.path.join(inc_dir, 'pyconfig.h')
 
 from sysconfig_cpython import (
     parse_makefile, _variable_rx, expand_makefile_vars)
diff --git a/lib-python/2.7/distutils/unixccompiler.py b/lib-python/2.7/distutils/unixccompiler.py
--- a/lib-python/2.7/distutils/unixccompiler.py
+++ b/lib-python/2.7/distutils/unixccompiler.py
@@ -226,7 +226,19 @@
         return "-L" + dir
 
     def _is_gcc(self, compiler_name):
-        return "gcc" in compiler_name or "g++" in compiler_name
+        # XXX PyPy workaround, look at the big comment below for more
+        # context. On CPython, the hack below works fine because
+        # `compiler_name` contains the name of the actual compiler which was
+        # used at compile time (e.g. 'x86_64-linux-gnu-gcc' on my machine).
+        # PyPy hardcodes it to 'cc', so the hack doesn't work, and the end
+        # result is that we pass the wrong option to the compiler.
+        #
+        # The workaround is to *always* pretend to be GCC if we are on Linux:
+        # this should cover the vast majority of real systems, including the
+        # ones which use clang (which understands the '-Wl,-rpath' syntax as
+        # well)
+        return (sys.platform == "linux2" or
+                "gcc" in compiler_name or "g++" in compiler_name)
 
     def runtime_library_dir_option(self, dir):
         # XXX Hackish, at the very least.  See Python bug #445902:
diff --git a/lib-python/3/ctypes/__init__.py b/lib-python/3/ctypes/__init__.py
--- a/lib-python/3/ctypes/__init__.py
+++ b/lib-python/3/ctypes/__init__.py
@@ -346,16 +346,18 @@
 
         if handle is None:
             if flags & _FUNCFLAG_CDECL:
-                self._handle = _ffi.CDLL(name, mode)
+                pypy_dll = _ffi.CDLL(name, mode)
             else:
-                self._handle = _ffi.WinDLL(name, mode)
-        else:
-            self._handle = handle
+                pypy_dll = _ffi.WinDLL(name, mode)
+            self.__pypy_dll__ = pypy_dll
+            handle = int(pypy_dll)
+        self._handle = handle
 
     def __repr__(self):
-        return "<%s '%s', handle %r at 0x%x>" % (
-            self.__class__.__name__, self._name, self._handle,
-            id(self) & (_sys.maxsize * 2 + 1))
+        return "<%s '%s', handle %x at 0x%x>" % \
+               (self.__class__.__name__, self._name,
+                (self._handle & (_sys.maxsize*2 + 1)),
+                id(self) & (_sys.maxsize*2 + 1))
 
     def __getattr__(self, name):
         if name.startswith('__') and name.endswith('__'):
diff --git a/lib-python/3/datetime.py b/lib-python/3/datetime.py
--- a/lib-python/3/datetime.py
+++ b/lib-python/3/datetime.py
@@ -810,7 +810,8 @@
             month = self._month
         if day is None:
             day = self._day
-        return date(year, month, day)
+        # PyPy fix: returns type(self)() instead of date()
+        return type(self)(year, month, day)
 
     # Comparisons of date objects with other.
 
@@ -1285,7 +1286,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return time(hour, minute, second, microsecond, tzinfo)
+        # PyPy fix: returns type(self)() instead of time()
+        return type(self)(hour, minute, second, microsecond, tzinfo)
 
     # Pickle support.
 
@@ -1497,7 +1499,8 @@
             microsecond = self.microsecond
         if tzinfo is True:
             tzinfo = self.tzinfo
-        return datetime(year, month, day, hour, minute, second, microsecond,
+        # PyPy fix: returns type(self)() instead of datetime()
+        return type(self)(year, month, day, hour, minute, second, microsecond,
                         tzinfo)
 
     def astimezone(self, tz=None):
diff --git a/lib-python/3/distutils/sysconfig_pypy.py b/lib-python/3/distutils/sysconfig_pypy.py
--- a/lib-python/3/distutils/sysconfig_pypy.py
+++ b/lib-python/3/distutils/sysconfig_pypy.py
@@ -81,6 +81,19 @@
     g['LIBDIR'] = os.path.join(sys.prefix, 'lib')
     g['VERSION'] = get_python_version()
 
+    if sys.platform[:6] == "darwin":
+        import platform
+        if platform.machine() == 'i386':
+            if platform.architecture()[0] == '32bit':
+                arch = 'i386'
+            else:
+                arch = 'x86_64'
+        else:
+            # just a guess
+            arch = platform.machine()
+        g['LDSHARED'] += ' -undefined dynamic_lookup'
+        g['CC'] += ' -arch %s' % (arch,)
+
     global _config_vars
     _config_vars = g
 
diff --git a/lib-python/3/stat.py b/lib-python/3/stat.py
--- a/lib-python/3/stat.py
+++ b/lib-python/3/stat.py
@@ -139,13 +139,21 @@
 def filemode(mode):
     """Convert a file's mode to a string of the form '-rwxrwxrwx'."""
     perm = []
+
+    # The first group gets a question mark if none of the bits match the mode.
+    empty = "?"
+
     for table in _filemode_table:
         for bit, char in table:
             if mode & bit == bit:
                 perm.append(char)
                 break
         else:
-            perm.append("-")
+            perm.append(empty)
+
+        # All the rest of the positions get a - if the bits don't match.
+        empty = "-"
+
     return "".join(perm)
 
 
diff --git a/lib-python/3/test/test_pyexpat.py b/lib-python/3/test/test_pyexpat.py
--- a/lib-python/3/test/test_pyexpat.py
+++ b/lib-python/3/test/test_pyexpat.py
@@ -11,7 +11,7 @@
 from xml.parsers import expat
 from xml.parsers.expat import errors
 
-from test.support import sortdict
+from test.support import sortdict, impl_detail
 
 
 class SetAttributeTest(unittest.TestCase):
@@ -446,6 +446,7 @@
         self.assertEqual(os.path.basename(entry[0]), filename)
         self.assertEqual(entry[2], funcname)
 
+    @impl_detail("PyPy does not have pyexpat.c", pypy=False)
     def test_exception(self):
         parser = expat.ParserCreate()
         parser.StartElementHandler = self.StartElementHandler
diff --git a/lib-python/3/test/test_stat.py b/lib-python/3/test/test_stat.py
--- a/lib-python/3/test/test_stat.py
+++ b/lib-python/3/test/test_stat.py
@@ -138,6 +138,10 @@
             self.assertS_IS("REG", st_mode)
             self.assertEqual(modestr, '-r--r--r--')
             self.assertEqual(self.statmod.S_IMODE(st_mode), 0o444)
+
+            # If there are only permission bits, no type bytes, a question
+            # mark is rendered in the type field.
+            self.assertEqual(self.statmod.filemode(0o420), '?r---w----')
         else:
             os.chmod(TESTFN, 0o700)
             st_mode, modestr = self.get_mode()
diff --git a/lib-python/3/test/test_sysconfig.py b/lib-python/3/test/test_sysconfig.py
--- a/lib-python/3/test/test_sysconfig.py
+++ b/lib-python/3/test/test_sysconfig.py
@@ -397,9 +397,16 @@
             self.assertTrue('linux' in suffix, suffix)
         if re.match('(i[3-6]86|x86_64)$', machine):
             if ctypes.sizeof(ctypes.c_char_p()) == 4:
-                self.assertTrue(suffix.endswith('i386-linux-gnu.so') \
-                                or suffix.endswith('x86_64-linux-gnux32.so'),
-                                suffix)
+                self.assertTrue(
+                    suffix.endswith((
+                        'i386-linux-gnu.so',
+                        'i486-linux-gnu.so',
+                        'i586-linux-gnu.so',
+                        'i686-linux-gnu.so',
+                        'x86_64-linux-gnux32.so',
+                    )),
+                    suffix,
+                )
             else: # 8 byte pointer size
                 self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
 
diff --git a/lib_pypy/_cffi_ssl/README.md b/lib_pypy/_cffi_ssl/README.md
--- a/lib_pypy/_cffi_ssl/README.md
+++ b/lib_pypy/_cffi_ssl/README.md
@@ -5,9 +5,15 @@
 it renames the compiled shared object to _pypy_openssl.so (which means
 that cryptography can ship their own cffi backend)
 
-NOTE: currently, we have changed ``_cffi_src/openssl/callbacks.py`` to
-not rely on the CPython C API, and ``_cffi_src/utils.py`` for issue #2575
-(29c9a89359e4).  (The first change is now backported.)
+NOTE: currently, we have the following changes:
+
+* ``_cffi_src/openssl/callbacks.py`` to not rely on the CPython C API
+  (this change is now backported)
+
+* ``_cffi_src/utils.py`` for issue #2575 (29c9a89359e4)
+
+* ``_cffi_src/openssl/x509_vfy.py`` for issue #2605 (ca4d0c90f5a1)
+
 
 # Tests?
 
diff --git a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
--- a/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
+++ b/lib_pypy/_cffi_ssl/_cffi_src/openssl/x509_vfy.py
@@ -221,10 +221,16 @@
 static const long X509_V_ERR_SUITE_B_INVALID_SIGNATURE_ALGORITHM = 0;
 static const long X509_V_ERR_SUITE_B_LOS_NOT_ALLOWED = 0;
 static const long X509_V_ERR_SUITE_B_CANNOT_SIGN_P_384_WITH_P_256 = 0;
+#ifndef X509_V_ERR_HOSTNAME_MISMATCH
 static const long X509_V_ERR_HOSTNAME_MISMATCH = 0;
+#endif
+#ifndef X509_V_ERR_EMAIL_MISMATCH
 static const long X509_V_ERR_EMAIL_MISMATCH = 0;
+#endif
+#ifndef X509_V_ERR_IP_ADDRESS_MISMATCH
 static const long X509_V_ERR_IP_ADDRESS_MISMATCH = 0;
 #endif
+#endif
 
 /* OpenSSL 1.0.2beta2+ verification parameters */
 #if CRYPTOGRAPHY_OPENSSL_102BETA2_OR_GREATER && \
diff --git a/lib_pypy/_cffi_ssl/_stdssl/certificate.py b/lib_pypy/_cffi_ssl/_stdssl/certificate.py
--- a/lib_pypy/_cffi_ssl/_stdssl/certificate.py
+++ b/lib_pypy/_cffi_ssl/_stdssl/certificate.py
@@ -173,14 +173,13 @@
 
     return tuple(dn)
 
-STATIC_BIO_BUF = ffi.new("char[]", 2048)
-
 def _bio_get_str(biobuf):
-    length = lib.BIO_gets(biobuf, STATIC_BIO_BUF, len(STATIC_BIO_BUF)-1)
+    bio_buf = ffi.new("char[]", 2048)
+    length = lib.BIO_gets(biobuf, bio_buf, len(bio_buf)-1)
     if length < 0:
         if biobuf: lib.BIO_free(biobuf)
         raise ssl_error(None)
-    return _str_with_len(STATIC_BIO_BUF, length)
+    return _str_with_len(bio_buf, length)
 
 def _decode_certificate(certificate):
     retval = {}
diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py
--- a/lib_pypy/_ctypes/basics.py
+++ b/lib_pypy/_ctypes/basics.py
@@ -82,7 +82,7 @@
         return False
 
     def in_dll(self, dll, name):
-        return self.from_address(dll._handle.getaddressindll(name))
+        return self.from_address(dll.__pypy_dll__.getaddressindll(name))
 
     def from_buffer(self, obj, offset=0):
         size = self._sizeofinstances()
diff --git a/lib_pypy/_ctypes/function.py b/lib_pypy/_ctypes/function.py
--- a/lib_pypy/_ctypes/function.py
+++ b/lib_pypy/_ctypes/function.py
@@ -430,7 +430,7 @@
             ffires = restype.get_ffi_argtype()
             return _ffi.FuncPtr.fromaddr(ptr, '', ffiargs, ffires, self._flags_)
 
-        cdll = self.dll._handle
+        cdll = self.dll.__pypy_dll__
         try:
             ffi_argtypes = [argtype.get_ffi_argtype() for argtype in argtypes]
             ffi_restype = restype.get_ffi_argtype()
diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py
--- a/lib_pypy/_ctypes/structure.py
+++ b/lib_pypy/_ctypes/structure.py
@@ -40,6 +40,22 @@
         else:
             rawfields.append((f[0], f[1]._ffishape_))
 
+    # hack for duplicate field names
+    already_seen = set()
+    names1 = names
+    names = []
+    for f in names1:
+        if f not in already_seen:
+            names.append(f)
+            already_seen.add(f)
+    already_seen = set()
+    for i in reversed(range(len(rawfields))):
+        if rawfields[i][0] in already_seen:
+            rawfields[i] = (('$DUP%d$%s' % (i, rawfields[i][0]),)
+                            + rawfields[i][1:])
+        already_seen.add(rawfields[i][0])
+    # /hack
+
     _set_shape(self, rawfields, self._is_union)
 
     fields = {}
diff --git a/lib_pypy/_curses.py b/lib_pypy/_curses.py
--- a/lib_pypy/_curses.py
+++ b/lib_pypy/_curses.py
@@ -411,7 +411,7 @@
             val = lib.mvwget_wch(self._win, *args, wch)
         else:
             raise error("get_wch requires 0 or 2 arguments")
-        _check_ERR(val, "get_wch"):
+        _check_ERR(val, "get_wch")
         return wch[0]
 
     def getkey(self, *args):
diff --git a/lib_pypy/_tkinter/tklib_build.py b/lib_pypy/_tkinter/tklib_build.py
--- a/lib_pypy/_tkinter/tklib_build.py
+++ b/lib_pypy/_tkinter/tklib_build.py
@@ -22,12 +22,27 @@
     linklibs = ['tcl', 'tk']
     libdirs = []
 else:
-    for _ver in ['', '8.6', '8.5', '']:
+    # On some Linux distributions, the tcl and tk libraries are
+    # stored in /usr/include, so we must check this case also
+    libdirs = []
+    found = False
+    for _ver in ['', '8.6', '8.5']:
         incdirs = ['/usr/include/tcl' + _ver]
         linklibs = ['tcl' + _ver, 'tk' + _ver]
-        libdirs = []
         if os.path.isdir(incdirs[0]):
+            found = True
             break
+    if not found:
+        for _ver in ['8.6', '8.5', '']:
+            incdirs = []
+            linklibs = ['tcl' + _ver, 'tk' + _ver]
+            if os.path.isfile(''.join(['/usr/lib/lib', linklibs[1], '.so'])):
+                found = True
+                break
+    if not found:
+        sys.stderr.write("*** TCL libraries not found!  Falling back...\n")
+        incdirs = []
+        linklibs = ['tcl', 'tk']
 
 config_ffi = FFI()
 config_ffi.cdef("""
diff --git a/lib_pypy/cffi/_cffi_include.h b/lib_pypy/cffi/_cffi_include.h
--- a/lib_pypy/cffi/_cffi_include.h
+++ b/lib_pypy/cffi/_cffi_include.h
@@ -95,6 +95,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h
--- a/lib_pypy/cffi/_embedding.h
+++ b/lib_pypy/cffi/_embedding.h
@@ -1,7 +1,12 @@
 
 /***** Support code for embedding *****/
 
-#if defined(_MSC_VER)
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+#if defined(_WIN32)
 #  define CFFI_DLLEXPORT  __declspec(dllexport)
 #elif defined(__GNUC__)
 #  define CFFI_DLLEXPORT  __attribute__((visibility("default")))
@@ -525,3 +530,7 @@
 #undef cffi_compare_and_swap
 #undef cffi_write_barrier
 #undef cffi_read_barrier
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -394,12 +394,17 @@
             replace_with = ' ' + replace_with
         return self._backend.getcname(cdecl, replace_with)
 
-    def gc(self, cdata, destructor):
+    def gc(self, cdata, destructor, size=0):
         """Return a new cdata object that points to the same
         data.  Later, when this new cdata object is garbage-collected,
         'destructor(old_cdata_object)' will be called.
+
+        The optional 'size' gives an estimate of the size, used to
+        trigger the garbage collection more eagerly.  So far only used
+        on PyPy.  It tells the GC that the returned object keeps alive
+        roughly 'size' bytes of external memory.
         """
-        return self._backend.gcp(cdata, destructor)
+        return self._backend.gcp(cdata, destructor, size)
 
     def _get_cached_btype(self, type):
         assert self._lock.acquire(False) is False
diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py
--- a/lib_pypy/cffi/backend_ctypes.py
+++ b/lib_pypy/cffi/backend_ctypes.py
@@ -1002,7 +1002,7 @@
 
     _weakref_cache_ref = None
 
-    def gcp(self, cdata, destructor):
+    def gcp(self, cdata, destructor, size=0):
         if self._weakref_cache_ref is None:
             import weakref
             class MyRef(weakref.ref):
diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py
--- a/lib_pypy/cffi/recompiler.py
+++ b/lib_pypy/cffi/recompiler.py
@@ -412,6 +412,9 @@
             prnt('    }')
         prnt('    p[0] = (const void *)0x%x;' % self._version)
         prnt('    p[1] = &_cffi_type_context;')
+        prnt('#if PY_MAJOR_VERSION >= 3')
+        prnt('    return NULL;')
+        prnt('#endif')
         prnt('}')
         # on Windows, distutils insists on putting init_cffi_xyz in
         # 'export_symbols', so instead of fighting it, just give up and
@@ -578,7 +581,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.BasePrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif isinstance(tp, model.UnknownFloatType):
                 return '_cffi_from_c_double(%s)' % (var,)
diff --git a/lib_pypy/cffi/vengine_cpy.py b/lib_pypy/cffi/vengine_cpy.py
--- a/lib_pypy/cffi/vengine_cpy.py
+++ b/lib_pypy/cffi/vengine_cpy.py
@@ -296,7 +296,7 @@
 
     def _convert_expr_from_c(self, tp, var, context):
         if isinstance(tp, model.PrimitiveType):
-            if tp.is_integer_type():
+            if tp.is_integer_type() and tp.name != '_Bool':
                 return '_cffi_from_c_int(%s, %s)' % (var, tp.name)
             elif tp.name != 'long double':
                 return '_cffi_from_c_%s(%s)' % (tp.name.replace(' ', '_'), var)
@@ -872,6 +872,7 @@
 #define _cffi_from_c_ulong PyLong_FromUnsignedLong
 #define _cffi_from_c_longlong PyLong_FromLongLong
 #define _cffi_from_c_ulonglong PyLong_FromUnsignedLongLong
+#define _cffi_from_c__Bool PyBool_FromLong
 
 #define _cffi_to_c_double PyFloat_AsDouble
 #define _cffi_to_c_float PyFloat_AsDouble
diff --git a/lib_pypy/pyrepl/reader.py b/lib_pypy/pyrepl/reader.py
--- a/lib_pypy/pyrepl/reader.py
+++ b/lib_pypy/pyrepl/reader.py
@@ -239,6 +239,10 @@
 
     def __init__(self, console):
         self.buffer = []
+        # Enable the use of `insert` without a `prepare` call - necessary to
+        # facilitate the tab completion hack implemented for
+        # <https://bugs.python.org/issue25660>.
+        self.pos = 0
         self.ps1 = "->> "
         self.ps2 = "/>> "
         self.ps3 = "|.. "
diff --git a/lib_pypy/pyrepl/readline.py b/lib_pypy/pyrepl/readline.py
--- a/lib_pypy/pyrepl/readline.py
+++ b/lib_pypy/pyrepl/readline.py
@@ -297,10 +297,7 @@
         line = line.rstrip('\n')
         if isinstance(line, unicode):
             return line # on py3k
-        try:
-            return unicode(line, ENCODING)
-        except UnicodeDecodeError:   # bah, silently fall back...
-            return unicode(line, 'utf-8', 'replace')
+        return unicode(line, 'utf-8', 'replace')
 
     def get_history_length(self):
         return self.saved_history_length
@@ -317,7 +314,8 @@
         # history item: we use \r\n instead of just \n.  If the history
         # file is passed to GNU readline, the extra \r are just ignored.
         history = self.get_reader().history
-        f = open(os.path.expanduser(filename), 'r')
+        f = open(os.path.expanduser(filename), 'r', encoding='utf-8',
+                 errors='replace')
         buffer = []
         for line in f:
             if line.endswith('\r\n'):
@@ -334,15 +332,12 @@
     def write_history_file(self, filename='~/.history'):
         maxlength = self.saved_history_length
         history = self.get_reader().get_trimmed_history(maxlength)
-        f = open(os.path.expanduser(filename), 'w')
+        f = open(os.path.expanduser(filename), 'w', encoding='utf-8')
         for entry in history:
             # if we are on py3k, we don't need to encode strings before
             # writing it to a file
             if isinstance(entry, unicode) and sys.version_info < (3,):
-                try:
-                    entry = entry.encode(ENCODING)
-                except UnicodeEncodeError:   # bah, silently fall back...
-                    entry = entry.encode('utf-8')
+                entry = entry.encode('utf-8')
             entry = entry.replace('\n', '\r\n')   # multiline history support
             f.write(entry + '\n')
         f.close()
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -39,7 +39,7 @@
     "thread", "itertools", "pyexpat", "cpyext", "array",
     "binascii", "_multiprocessing", '_warnings', "_collections",
     "_multibytecodec", "_continuation", "_cffi_backend",
-    "_csv", "_pypyjson", "_posixsubprocess", # "cppyy", "micronumpy"
+    "_csv", "_pypyjson", "_posixsubprocess", # "_cppyy", "micronumpy"
     "_jitlog",
 ])
 
@@ -71,8 +71,10 @@
         if name in translation_modules:
             translation_modules.remove(name)
 
-    if "cppyy" in working_modules:
-        working_modules.remove("cppyy")  # not tested on win32
+    if "_cppyy" in working_modules:
+        working_modules.remove("_cppyy")  # not tested on win32
+    if "_vmprof" in working_modules:
+        working_modules.remove("_vmprof")  # FIXME: missing details
 
     # The _locale module is needed by site.py on Windows
     default_modules.add("_locale")
@@ -81,8 +83,8 @@
     working_modules.remove('fcntl')  # LOCK_NB not defined
     working_modules.remove("_minimal_curses")
     working_modules.remove("termios")
-    if "cppyy" in working_modules:
-        working_modules.remove("cppyy")  # depends on ctypes
+    if "_cppyy" in working_modules:
+        working_modules.remove("_cppyy")  # depends on ctypes
 
 #if sys.platform.startswith("linux"):
 #    _mach = os.popen('uname -m', 'r').read().strip()
@@ -94,7 +96,7 @@
     '_multiprocessing': [('objspace.usemodules.time', True),
                          ('objspace.usemodules.thread', True)],
     'cpyext': [('objspace.usemodules.array', True)],
-    'cppyy': [('objspace.usemodules.cpyext', True)],
+    '_cppyy': [('objspace.usemodules.cpyext', True)],
     'faulthandler': [('objspace.usemodules._vmprof', True)],
     }
 module_suggests = {
@@ -227,11 +229,6 @@
                    "use specialised tuples",
                    default=False),
 
-        BoolOption("withcelldict",
-                   "use dictionaries that are optimized for being used as module dicts",
-                   default=False,
-                   requires=[("objspace.honor__builtins__", False)]),
-
         BoolOption("withliststrategies",
                    "enable optimized ways to store lists of primitives ",
                    default=True),
@@ -291,7 +288,7 @@
 
     # extra optimizations with the JIT
     if level == 'jit':
-        config.objspace.std.suggest(withcelldict=True)
+        pass # none at the moment
 
 
 def enable_allworkingmodules(config):
diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst
--- a/pypy/doc/build.rst
+++ b/pypy/doc/build.rst
@@ -10,6 +10,18 @@
 minutes on a fast machine -- and RAM-hungry.  You will need **at least** 2 GB
 of memory on a 32-bit machine and 4GB on a 64-bit machine.
 
+Before you start
+----------------
+
+Our normal development workflow avoids a full translation by using test-driven
+development. You can read more about how to develop PyPy here_, and latest
+translated (hopefully functional) binary packages are available on our
+buildbot's `nightly builds`_
+
+.. _here: getting-started-dev.html
+.. _`nightly builds`: http://buildbot.pypy.org/nightly
+
+You will need the build dependencies below to run the tests.
 
 Clone the repository
 --------------------
@@ -140,22 +152,61 @@
 Run the translation
 -------------------
 
+We usually translate in the ``pypy/goal`` directory, so all the following
+commands assume your ``$pwd`` is there.
+
 Translate with JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=jit
 
 Translate without JIT::
 
-    cd pypy/goal
     pypy ../../rpython/bin/rpython --opt=2
 
+Note this translates pypy via the ``targetpypystandalone.py`` file, so these
+are shorthand for::
+
+    pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>
+
+More help is availabe via ``--help`` at either option position, and more info
+can be found in the :doc:`config/index` section.
+
 (You can use ``python`` instead of ``pypy`` here, which will take longer
 but works too.)
 
-If everything works correctly this will create an executable ``pypy-c`` in the
-current directory. The executable behaves mostly like a normal Python
-interpreter (see :doc:`cpython_differences`).
+If everything works correctly this will:
+
+1. Run the rpython `translation chain`_, producing a database of the
+   entire pypy interpreter. This step is currently singe threaded, and RAM
+   hungry. As part of this step,  the chain creates a large number of C code
+   files and a Makefile to compile them in a
+   directory controlled by the ``PYPY_USESSION_DIR`` environment variable.  
+2. Create an executable ``pypy-c`` by running the Makefile. This step can
+   utilize all possible cores on the machine.  
+3. Copy the needed binaries to the current directory.  
+4. Generate c-extension modules for any cffi-based stdlib modules.  
+
+
+The resulting executable behaves mostly like a normal Python
+interpreter (see :doc:`cpython_differences`), and is ready for testing, for
+use as a base interpreter for a new virtualenv, or for packaging into a binary
+suitable for installation on another machine running the same OS as the build
+machine. 
+
+Note that step 4 is merely done as a convenience, any of the steps may be rerun
+without rerunning the previous steps.
+
+.. _`translation chain`: https://rpython.readthedocs.io/en/latest/translation.html
+
+
+Making a debug build of PyPy
+----------------------------
+
+If the Makefile is rerun with the lldebug or lldebug0 target, appropriate
+compilation flags are added to add debug info and reduce compiler optimizations
+to ``-O0`` respectively. If you stop in a debugger, you will see the
+very wordy machine-generated C code from the rpython translation step, which
+takes a little bit of reading to relate back to the rpython code.
 
 Build cffi import libraries for the stdlib
 ------------------------------------------
@@ -169,14 +220,6 @@
 
 .. _`out-of-line API mode`: http://cffi.readthedocs.org/en/latest/overview.html#real-example-api-level-out-of-line
 
-Translating with non-standard options
--------------------------------------
-
-It is possible to have non-standard features enabled for translation,
-but they are not really tested any more.  Look, for example, at the
-:doc:`objspace proxies <objspace-proxies>` document.
-
-
 Packaging (preparing for installation)
 --------------------------------------
 
@@ -205,14 +248,16 @@
 
 * PyPy 2.5.1 or earlier: normal users would see permission errors.
   Installers need to run ``pypy -c "import gdbm"`` and other similar
-  commands at install time; the exact list is in `package.py`_.  Users
+  commands at install time; the exact list is in 
+  :source:`pypy/tool/release/package.py <package.py>`.  Users
   seeing a broken installation of PyPy can fix it after-the-fact if they
   have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``.
 
 * PyPy 2.6 and later: anyone would get ``ImportError: no module named
   _gdbm_cffi``.  Installers need to run ``pypy _gdbm_build.py`` in the
   ``lib_pypy`` directory during the installation process (plus others;
-  see the exact list in `package.py`_).  Users seeing a broken
+  see the exact list in :source:`pypy/tool/release/package.py <package.py>`).
+  Users seeing a broken
   installation of PyPy can fix it after-the-fact, by running ``pypy
   /path/to/lib_pypy/_gdbm_build.py``.  This command produces a file
   called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension
diff --git a/pypy/doc/config/objspace.std.withcelldict.txt b/pypy/doc/config/objspace.std.withcelldict.txt
deleted file mode 100644
--- a/pypy/doc/config/objspace.std.withcelldict.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-Enable cell-dicts. This optimization is not helpful without the JIT. In the
-presence of the JIT, it greatly helps looking up globals.
diff --git a/pypy/doc/configuration.rst b/pypy/doc/configuration.rst
--- a/pypy/doc/configuration.rst
+++ b/pypy/doc/configuration.rst
@@ -188,4 +188,6 @@
 can be found on the ``config`` attribute of all ``TranslationContext``
 instances and are described in :source:`rpython/config/translationoption.py`. The interpreter options
 are attached to the object space, also under the name ``config`` and are
-described in :source:`pypy/config/pypyoption.py`.
+described in :source:`pypy/config/pypyoption.py`. Both set of options are
+documented in the :doc:`config/index` section.
+
diff --git a/pypy/doc/cppyy.rst b/pypy/doc/cppyy.rst
deleted file mode 100644
--- a/pypy/doc/cppyy.rst
+++ /dev/null
@@ -1,672 +0,0 @@
-cppyy: C++ bindings for PyPy
-============================
-
-The cppyy module delivers dynamic Python-C++ bindings.
-It is designed for automation, high performance, scale, interactivity, and
-handling all of modern C++ (11, 14, etc.).
-It is based on `Cling`_ which, through `LLVM`_/`clang`_, provides C++
-reflection and interactivity.
-Reflection information is extracted from C++ header files.
-Cppyy itself is built into PyPy (an alternative exists for CPython), but
-it requires a `backend`_, installable through pip, to interface with Cling.
-
-.. _Cling: https://root.cern.ch/cling
-.. _LLVM: http://llvm.org/
-.. _clang: http://clang.llvm.org/
-.. _backend: https://pypi.python.org/pypi/PyPy-cppyy-backend
-
-
-Installation
-------------
-
-This assumes PyPy2.7 v5.7 or later; earlier versions use a Reflex-based cppyy
-module, which is no longer supported.
-Both the tooling and user-facing Python codes are very backwards compatible,
-however.
-Further dependencies are cmake (for general build), Python2.7 (for LLVM), and
-a modern C++ compiler (one that supports at least C++11).
-
-Assuming you have a recent enough version of PyPy installed, use pip to
-complete the installation of cppyy::
-
- $ MAKE_NPROCS=4 pypy-c -m pip install --verbose PyPy-cppyy-backend
-
-Set the number of parallel builds ('4' in this example, through the MAKE_NPROCS
-environment variable) to a number appropriate for your machine.
-The building process may take quite some time as it includes a customized
-version of LLVM as part of Cling, which is why --verbose is recommended so that
-you can see the build progress.
-
-The default installation will be under
-$PYTHONHOME/site-packages/cppyy_backend/lib,
-which needs to be added to your dynamic loader path (LD_LIBRARY_PATH).
-If you need the dictionary and class map generation tools (used in the examples
-below), you need to add $PYTHONHOME/site-packages/cppyy_backend/bin to your
-executable path (PATH).
-
-
-Basic bindings example
-----------------------
-
-These examples assume that cppyy_backend is pointed to by the environment
-variable CPPYYHOME, and that CPPYYHOME/lib is added to LD_LIBRARY_PATH and
-CPPYYHOME/bin to PATH.
-
-Let's first test with a trivial example whether all packages are properly
-installed and functional.
-Create a C++ header file with some class in it (all functions are made inline
-for convenience; if you have out-of-line code, link with it as appropriate)::
-
-    $ cat MyClass.h
-    class MyClass {
-    public:
-        MyClass(int i = -99) : m_myint(i) {}
-
-        int GetMyInt() { return m_myint; }
-        void SetMyInt(int i) { m_myint = i; }
-
-    public:
-        int m_myint;
-    };
-
-Then, generate the bindings using ``genreflex`` (installed under
-cppyy_backend/bin in site_packages), and compile the code::
-
-    $ genreflex MyClass.h
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling
-
-Next, make sure that the library can be found through the dynamic lookup path
-(the ``LD_LIBRARY_PATH`` environment variable on Linux, ``PATH`` on Windows),
-for example by adding ".".
-Now you're ready to use the bindings.
-Since the bindings are designed to look pythonistic, it should be
-straightforward::
-
-    $ pypy-c
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libMyClassDict.so")
-    <CPPLibrary object at 0xb6fd7c4c>
-    >>>> myinst = cppyy.gbl.MyClass(42)
-    >>>> print myinst.GetMyInt()
-    42
-    >>>> myinst.SetMyInt(33)
-    >>>> print myinst.m_myint
-    33
-    >>>> myinst.m_myint = 77
-    >>>> print myinst.GetMyInt()
-    77
-    >>>> help(cppyy.gbl.MyClass)   # shows that normal python introspection works
-
-That's all there is to it!
-
-
-Automatic class loader
-----------------------
-
-There is one big problem in the code above, that prevents its use in a (large
-scale) production setting: the explicit loading of the reflection library.
-Clearly, if explicit load statements such as these show up in code downstream
-from the ``MyClass`` package, then that prevents the ``MyClass`` author from
-repackaging or even simply renaming the dictionary library.
-
-The solution is to make use of an automatic class loader, so that downstream
-code never has to call ``load_reflection_info()`` directly.
-The class loader makes use of so-called rootmap files, which ``genreflex``
-can produce.
-These files contain the list of available C++ classes and specify the library
-that needs to be loaded for their use (as an aside, this listing allows for a
-cross-check to see whether reflection info is generated for all classes that
-you expect).
-By convention, the rootmap files should be located next to the reflection info
-libraries, so that they can be found through the normal shared library search
-path.
-They can be concatenated together, or consist of a single rootmap file per
-library.
-For example::
-
-    $ genreflex MyClass.h --rootmap=libMyClassDict.rootmap --rootmap-lib=libMyClassDict.so
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling
-
-where the first option (``--rootmap``) specifies the output file name, and the
-second option (``--rootmap-lib``) the name of the reflection library where
-``MyClass`` will live.
-It is necessary to provide that name explicitly, since it is only in the
-separate linking step where this name is fixed.
-If the second option is not given, the library is assumed to be libMyClass.so,
-a name that is derived from the name of the header file.
-
-With the rootmap file in place, the above example can be rerun without explicit
-loading of the reflection info library::
-
-    $ pypy-c
-    >>>> import cppyy
-    >>>> myinst = cppyy.gbl.MyClass(42)
-    >>>> print myinst.GetMyInt()
-    42
-    >>>> # etc. ...
-
-As a caveat, note that the class loader is currently limited to classes only.
-
-
-Advanced example
-----------------
-
-The following snippet of C++ is very contrived, to allow showing that such
-pathological code can be handled and to show how certain features play out in
-practice::
-
-    $ cat MyAdvanced.h
-    #include <string>
-
-    class Base1 {
-    public:
-        Base1(int i) : m_i(i) {}
-        virtual ~Base1() {}
-        int m_i;
-    };
-
-    class Base2 {
-    public:
-        Base2(double d) : m_d(d) {}
-        virtual ~Base2() {}
-        double m_d;
-    };
-
-    class C;
-
-    class Derived : public virtual Base1, public virtual Base2 {
-    public:
-        Derived(const std::string& name, int i, double d) : Base1(i), Base2(d), m_name(name) {}
-        virtual C* gimeC() { return (C*)0; }
-        std::string m_name;
-    };
-
-    Base2* BaseFactory(const std::string& name, int i, double d) {
-        return new Derived(name, i, d);
-    }
-
-This code is still only in a header file, with all functions inline, for
-convenience of the example.
-If the implementations live in a separate source file or shared library, the
-only change needed is to link those in when building the reflection library.
-
-If you were to run ``genreflex`` like above in the basic example, you will
-find that not all classes of interest will be reflected, nor will be the
-global factory function.
-In particular, ``std::string`` will be missing, since it is not defined in
-this header file, but in a header file that is included.
-In practical terms, general classes such as ``std::string`` should live in a
-core reflection set, but for the moment assume we want to have it in the
-reflection library that we are building for this example.
-
-The ``genreflex`` script can be steered using a so-called `selection file`_
-(see "Generating Reflex Dictionaries")
-which is a simple XML file specifying, either explicitly or by using a
-pattern, which classes, variables, namespaces, etc. to select from the given
-header file.
-With the aid of a selection file, a large project can be easily managed:
-simply ``#include`` all relevant headers into a single header file that is
-handed to ``genreflex``.
-In fact, if you hand multiple header files to ``genreflex``, then a selection
-file is almost obligatory: without it, only classes from the last header will
-be selected.
-Then, apply a selection file to pick up all the relevant classes.
-For our purposes, the following rather straightforward selection will do
-(the name ``lcgdict`` for the root is historical, but required)::
-
-    $ cat MyAdvanced.xml
-    <lcgdict>
-        <class pattern="Base?" />
-        <class name="Derived" />
-        <class name="std::string" />
-        <function name="BaseFactory" />
-    </lcgdict>
-
-.. _selection file: https://root.cern.ch/how/how-use-reflex
-
-Now the reflection info can be generated and compiled::
-
-    $ genreflex MyAdvanced.h --selection=MyAdvanced.xml
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyAdvanced_rflx.cpp -o libAdvExDict.so -L$CPPYYHOME/lib -lCling
-
-and subsequently be used from PyPy::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libAdvExDict.so")
-    <CPPLibrary object at 0x00007fdb48fc8120>
-    >>>> d = cppyy.gbl.BaseFactory("name", 42, 3.14)
-    >>>> type(d)
-    <class '__main__.Derived'>
-    >>>> isinstance(d, cppyy.gbl.Base1)
-    True
-    >>>> isinstance(d, cppyy.gbl.Base2)
-    True
-    >>>> d.m_i, d.m_d
-    (42, 3.14)
-    >>>> d.m_name == "name"
-    True
-    >>>>
-
-Again, that's all there is to it!
-
-A couple of things to note, though.
-If you look back at the C++ definition of the ``BaseFactory`` function,
-you will see that it declares the return type to be a ``Base2``, yet the
-bindings return an object of the actual type ``Derived``?
-This choice is made for a couple of reasons.
-First, it makes method dispatching easier: if bound objects are always their
-most derived type, then it is easy to calculate any offsets, if necessary.
-Second, it makes memory management easier: the combination of the type and
-the memory address uniquely identifies an object.
-That way, it can be recycled and object identity can be maintained if it is
-entered as a function argument into C++ and comes back to PyPy as a return
-value.
-Last, but not least, casting is decidedly unpythonistic.
-By always providing the most derived type known, casting becomes unnecessary.
-For example, the data member of ``Base2`` is simply directly available.
-Note also that the unreflected ``gimeC`` method of ``Derived`` does not
-preclude its use.
-It is only the ``gimeC`` method that is unusable as long as class ``C`` is
-unknown to the system.
-
-
-Features
---------
-
-The following is not meant to be an exhaustive list, since cppyy is still
-under active development.
-Furthermore, the intention is that every feature is as natural as possible on
-the python side, so if you find something missing in the list below, simply
-try it out.
-It is not always possible to provide exact mapping between python and C++
-(active memory management is one such case), but by and large, if the use of a
-feature does not strike you as obvious, it is more likely to simply be a bug.
-That is a strong statement to make, but also a worthy goal.
-For the C++ side of the examples, refer to this :doc:`example code <cppyy_example>`, which was
-bound using::
-
-    $ genreflex example.h --deep --rootmap=libexampleDict.rootmap --rootmap-lib=libexampleDict.so
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include example_rflx.cpp -o libexampleDict.so -L$CPPYYHOME/lib -lCling
-
-* **abstract classes**: Are represented as python classes, since they are
-  needed to complete the inheritance hierarchies, but will raise an exception
-  if an attempt is made to instantiate from them.
-  Example::
-
-    >>>> from cppyy.gbl import AbstractClass, ConcreteClass
-    >>>> a = AbstractClass()
-    Traceback (most recent call last):
-      File "<console>", line 1, in <module>
-    TypeError: cannot instantiate abstract class 'AbstractClass'
-    >>>> issubclass(ConcreteClass, AbstractClass)
-    True
-    >>>> c = ConcreteClass()
-    >>>> isinstance(c, AbstractClass)
-    True
-    >>>>
-
-* **arrays**: Supported for builtin data types only, as used from module
-  ``array``.
-  Out-of-bounds checking is limited to those cases where the size is known at
-  compile time (and hence part of the reflection info).
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> from array import array
-    >>>> c = ConcreteClass()
-    >>>> c.array_method(array('d', [1., 2., 3., 4.]), 4)
-    1 2 3 4
-    >>>>
-
-* **builtin data types**: Map onto the expected equivalent python types, with
-  the caveat that there may be size differences, and thus it is possible that
-  exceptions are raised if an overflow is detected.
-
-* **casting**: Is supposed to be unnecessary.
-  Object pointer returns from functions provide the most derived class known
-  in the hierarchy of the object being returned.
-  This is important to preserve object identity as well as to make casting,
-  a pure C++ feature after all, superfluous.
-  Example::
-
-    >>>> from cppyy.gbl import AbstractClass, ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> ConcreteClass.show_autocast.__doc__
-    'AbstractClass* ConcreteClass::show_autocast()'
-    >>>> d = c.show_autocast()
-    >>>> type(d)
-    <class '__main__.ConcreteClass'>
-    >>>>
-
-  However, if need be, you can perform C++-style reinterpret_casts (i.e.
-  without taking offsets into account), by taking and rebinding the address
-  of an object::
-
-    >>>> from cppyy import addressof, bind_object
-    >>>> e = bind_object(addressof(d), AbstractClass)
-    >>>> type(e)
-    <class '__main__.AbstractClass'>
-    >>>>
-
-* **classes and structs**: Get mapped onto python classes, where they can be
-  instantiated as expected.
-  If classes are inner classes or live in a namespace, their naming and
-  location will reflect that.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass, Namespace
-    >>>> ConcreteClass == Namespace.ConcreteClass
-    False
-    >>>> n = Namespace.ConcreteClass.NestedClass()
-    >>>> type(n)
-    <class '__main__.Namespace::ConcreteClass::NestedClass'>
-    >>>>
-
-* **data members**: Public data members are represented as python properties
-  and provide read and write access on instances as expected.
-  Private and protected data members are not accessible.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> c.m_int
-    42
-    >>>>
-
-* **default arguments**: C++ default arguments work as expected, but python
-  keywords are not supported.
-  It is technically possible to support keywords, but for the C++ interface,
-  the formal argument names have no meaning and are not considered part of the
-  API, hence it is not a good idea to use keywords.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()       # uses default argument
-    >>>> c.m_int
-    42
-    >>>> c = ConcreteClass(13)
-    >>>> c.m_int
-    13
-    >>>>
-
-* **doc strings**: The doc string of a method or function contains the C++
-  arguments and return types of all overloads of that name, as applicable.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> print ConcreteClass.array_method.__doc__
-    void ConcreteClass::array_method(int*, int)
-    void ConcreteClass::array_method(double*, int)
-    >>>>
-
-* **enums**: Are translated as ints with no further checking.
-
-* **functions**: Work as expected and live in their appropriate namespace
-  (which can be the global one, ``cppyy.gbl``).
-
-* **inheritance**: All combinations of inheritance on the C++ (single,
-  multiple, virtual) are supported in the binding.
-  However, new python classes can only use single inheritance from a bound C++
-  class.
-  Multiple inheritance would introduce two "this" pointers in the binding.
-  This is a current, not a fundamental, limitation.
-  The C++ side will not see any overridden methods on the python side, as
-  cross-inheritance is planned but not yet supported.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> help(ConcreteClass)
-    Help on class ConcreteClass in module __main__:
-
-    class ConcreteClass(AbstractClass)
-     |  Method resolution order:
-     |      ConcreteClass
-     |      AbstractClass
-     |      cppyy.CPPObject
-     |      __builtin__.CPPInstance
-     |      __builtin__.object
-     |
-     |  Methods defined here:
-     |
-     |  ConcreteClass(self, *args)
-     |      ConcreteClass::ConcreteClass(const ConcreteClass&)
-     |      ConcreteClass::ConcreteClass(int)
-     |      ConcreteClass::ConcreteClass()
-     |
-     etc. ....
-
-* **memory**: C++ instances created by calling their constructor from python
-  are owned by python.
-  You can check/change the ownership with the _python_owns flag that every
-  bound instance carries.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> c = ConcreteClass()
-    >>>> c._python_owns            # True: object created in Python
-    True
-    >>>>
-
-* **methods**: Are represented as python methods and work as expected.
-  They are first class objects and can be bound to an instance.
-  Virtual C++ methods work as expected.
-  To select a specific virtual method, do like with normal python classes
-  that override methods: select it from the class that you need, rather than
-  calling the method on the instance.
-  To select a specific overload, use the __dispatch__ special function, which
-  takes the name of the desired method and its signature (which can be
-  obtained from the doc string) as arguments.
-
-* **namespaces**: Are represented as python classes.
-  Namespaces are more open-ended than classes, so sometimes initial access may
-  result in updates as data and functions are looked up and constructed
-  lazily.
-  Thus the result of ``dir()`` on a namespace shows the classes available,
-  even if they may not have been created yet.
-  It does not show classes that could potentially be loaded by the class
-  loader.
-  Once created, namespaces are registered as modules, to allow importing from
-  them.
-  Namespace currently do not work with the class loader.
-  Fixing these bootstrap problems is on the TODO list.
-  The global namespace is ``cppyy.gbl``.
-
-* **NULL**: Is represented as ``cppyy.gbl.nullptr``.
-  In C++11, the keyword ``nullptr`` is used to represent ``NULL``.
-  For clarity of intent, it is recommended to use this instead of ``None``
-  (or the integer ``0``, which can serve in some cases), as ``None`` is better
-  understood as ``void`` in C++.
-
-* **operator conversions**: If defined in the C++ class and a python
-  equivalent exists (i.e. all builtin integer and floating point types, as well
-  as ``bool``), it will map onto that python conversion.
-  Note that ``char*`` is mapped onto ``__str__``.
-  Example::
-
-    >>>> from cppyy.gbl import ConcreteClass
-    >>>> print ConcreteClass()
-    Hello operator const char*!
-    >>>>
-
-* **operator overloads**: If defined in the C++ class and if a python
-  equivalent is available (not always the case, think e.g. of ``operator||``),
-  then they work as expected.
-  Special care needs to be taken for global operator overloads in C++: first,
-  make sure that they are actually reflected, especially for the global
-  overloads for ``operator==`` and ``operator!=`` of STL vector iterators in
-  the case of gcc (note that they are not needed to iterate over a vector).
-  Second, make sure that reflection info is loaded in the proper order.
-  I.e. that these global overloads are available before use.
-
-* **pointers**: For builtin data types, see arrays.
-  For objects, a pointer to an object and an object looks the same, unless
-  the pointer is a data member.
-  In that case, assigning to the data member will cause a copy of the pointer
-  and care should be taken about the object's life time.
-  If a pointer is a global variable, the C++ side can replace the underlying
-  object and the python side will immediately reflect that.
-
-* **PyObject***: Arguments and return types of ``PyObject*`` can be used, and
-  passed on to CPython API calls.
-  Since these CPython-like objects need to be created and tracked (this all
-  happens through ``cpyext``) this interface is not particularly fast.
-
-* **static data members**: Are represented as python property objects on the
-  class and the meta-class.
-  Both read and write access is as expected.
-
-* **static methods**: Are represented as python's ``staticmethod`` objects
-  and can be called both from the class as well as from instances.
-
-* **strings**: The std::string class is considered a builtin C++ type and
-  mixes quite well with python's str.
-  Python's str can be passed where a ``const char*`` is expected, and an str
-  will be returned if the return type is ``const char*``.
-
-* **templated classes**: Are represented in a meta-class style in python.
-  This may look a little bit confusing, but conceptually is rather natural.
-  For example, given the class ``std::vector<int>``, the meta-class part would
-  be ``std.vector``.
-  Then, to get the instantiation on ``int``, do ``std.vector(int)`` and to
-  create an instance of that class, do ``std.vector(int)()``::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info('libexampleDict.so')
-    >>>> cppyy.gbl.std.vector                # template metatype
-    <cppyy.CppyyTemplateType object at 0x00007fcdd330f1a0>
-    >>>> cppyy.gbl.std.vector(int)           # instantiates template -> class
-    <class '__main__.std::vector<int>'>
-    >>>> cppyy.gbl.std.vector(int)()         # instantiates class -> object
-    <__main__.std::vector<int> object at 0x00007fe480ba4bc0>
-    >>>>
-
-  Note that templates can be build up by handing actual types to the class
-  instantiation (as done in this vector example), or by passing in the list of
-  template arguments as a string.
-  The former is a lot easier to work with if you have template instantiations
-  using classes that themselves are templates in  the arguments (think e.g a
-  vector of vectors).
-  All template classes must already exist in the loaded reflection info, they
-  do not work (yet) with the class loader.
-
-  For compatibility with other bindings generators, use of square brackets
-  instead of parenthesis to instantiate templates is supported as well.
-
-* **templated functions**: Automatically participate in overloading and are
-  used in the same way as other global functions.
-
-* **templated methods**: For now, require an explicit selection of the
-  template parameters.
-  This will be changed to allow them to participate in overloads as expected.
-
-* **typedefs**: Are simple python references to the actual classes to which
-  they refer.
-
-* **unary operators**: Are supported if a python equivalent exists, and if the
-  operator is defined in the C++ class.
-
-You can always find more detailed examples and see the full of supported
-features by looking at the tests in pypy/module/cppyy/test.
-
-If a feature or reflection info is missing, this is supposed to be handled
-gracefully.
-In fact, there are unit tests explicitly for this purpose (even as their use
-becomes less interesting over time, as the number of missing features
-decreases).
-Only when a missing feature is used, should there be an exception.
-For example, if no reflection info is available for a return type, then a
-class that has a method with that return type can still be used.
-Only that one specific method can not be used.
-
-
-Templates
----------
-
-Templates can be automatically instantiated, assuming the appropriate header
-files have been loaded or are accessible to the class loader.
-This is the case for example for all of STL.
-For example::
-
-    $ cat MyTemplate.h
-    #include <vector>
-
-    class MyClass {
-    public:
-        MyClass(int i = -99) : m_i(i) {}
-        MyClass(const MyClass& s) : m_i(s.m_i) {}
-        MyClass& operator=(const MyClass& s) { m_i = s.m_i; return *this; }
-        ~MyClass() {}
-        int m_i;
-    };
-
-Run the normal ``genreflex`` and compilation steps::
-
-    $ genreflex MyTemplate.h --selection=MyTemplate.xml
-    $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyTemplate_rflx.cpp -o libTemplateDict.so -L$CPPYYHOME/lib -lCling
-
-Subsequent use should be as expected.
-Note the meta-class style of "instantiating" the template::
-
-    >>>> import cppyy
-    >>>> cppyy.load_reflection_info("libTemplateDict.so")
-    >>>> std = cppyy.gbl.std
-    >>>> MyClass = cppyy.gbl.MyClass
-    >>>> v = std.vector(MyClass)()
-    >>>> v += [MyClass(1), MyClass(2), MyClass(3)]
-    >>>> for m in v:
-    ....     print m.m_i,
-    ....
-    1 2 3
-    >>>>
-
-The arguments to the template instantiation can either be a string with the
-full list of arguments, or the explicit classes.
-The latter makes for easier code writing if the classes passed to the
-instantiation are themselves templates.
-
-
-The fast lane
--------------
-
-By default, cppyy will use direct function pointers through `CFFI`_ whenever
-possible. If this causes problems for you, you can disable it by setting the
-CPPYY_DISABLE_FASTPATH environment variable.
-
-.. _CFFI: https://cffi.readthedocs.io/en/latest/
-
-
-CPython
--------
-
-Most of the ideas in cppyy come originally from the `PyROOT`_ project, which
-contains a CPython-based cppyy.py module (with similar dependencies as the
-one that comes with PyPy).
-A standalone pip-installable version is planned, but for now you can install
-ROOT through your favorite distribution installer (available in the science
-section).
-
-.. _PyROOT: https://root.cern.ch/pyroot
-
-There are a couple of minor differences between the two versions of cppyy
-(the CPython version has a few more features).
-Work is on-going to integrate the nightly tests of both to make sure their
-feature sets are equalized.
-
-
-Python3
--------
-
-The CPython version of cppyy supports Python3, assuming your packager has
-build the backend for it.
-The cppyy module has not been tested with the `Py3k`_ version of PyPy.
-Note that the generated reflection information (from ``genreflex``) is fully
-independent of Python, and does not need to be rebuild when switching versions
-or interpreters.
-
-.. _Py3k: https://bitbucket.org/pypy/pypy/src/py3k
-
-
-.. toctree::
-   :hidden:
-
-   cppyy_example
diff --git a/pypy/doc/cppyy_example.rst b/pypy/doc/cppyy_example.rst
deleted file mode 100644
--- a/pypy/doc/cppyy_example.rst
+++ /dev/null
@@ -1,59 +0,0 @@
-File example.h
-==============
-
-::
-
-    #include <iostream>
-    #include <vector>
-
-    class AbstractClass {
-    public:
-        virtual ~AbstractClass() {}
-        virtual void abstract_method() = 0;
-    };
-
-    class ConcreteClass : AbstractClass {
-    public:
-        ConcreteClass(int n=42) : m_int(n) {}
-        ~ConcreteClass() {}
-
-        virtual void abstract_method() {
-            std::cout << "called concrete method" << std::endl;
-        }
-
-        void array_method(int* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        void array_method(double* ad, int size) {
-            for (int i=0; i < size; ++i)
-                std::cout << ad[i] << ' ';
-            std::cout << std::endl;
-        }
-
-        AbstractClass* show_autocast() {
-            return this;
-        }
-
-        operator const char*() {
-            return "Hello operator const char*!";
-        }
-
-    public:
-        int m_int;
-    };
-
-    namespace Namespace {
-
-       class ConcreteClass {
-       public:
-          class NestedClass {
-          public:
-             std::vector<int> m_v;
-          };
-
-       };
-
-    } // namespace Namespace
diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst
--- a/pypy/doc/cpython_differences.rst
+++ b/pypy/doc/cpython_differences.rst
@@ -337,6 +337,8 @@
 
  - ``frozenset`` (empty frozenset only)
 
+ - unbound method objects (for Python 2 only)
+
 This change requires some changes to ``id`` as well. ``id`` fulfills the
 following condition: ``x is y <=> id(x) == id(y)``. Therefore ``id`` of the
 above types will return a value that is computed from the argument, and can
diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst
--- a/pypy/doc/extending.rst
+++ b/pypy/doc/extending.rst
@@ -12,7 +12,7 @@
 
 * Write them in pure Python and use ctypes_.
 
-* Write them in C++ and bind them through  :doc:`cppyy <cppyy>` using Cling.
+* Write them in C++ and bind them through  cppyy_ using Cling.
 
 * Write them as `RPython mixed modules`_.
 
@@ -61,29 +61,22 @@
 .. _libffi: http://sourceware.org/libffi/
 
 
-Cling and cppyy
----------------
+cppyy
+-----
 
-The builtin :doc:`cppyy <cppyy>` module uses reflection information, provided by
-`Cling`_ (which needs to be `installed separately`_), of C/C++ code to
-automatically generate bindings at runtime.
-In Python, classes and functions are always runtime structures, so when they
-are generated matters not for performance.
-However, if the backend itself is capable of dynamic behavior, it is a much
-better functional match, allowing tighter integration and more natural
-language mappings.
+For C++, _cppyy_ is an automated bindings generator available for both
+PyPy and CPython.
+_cppyy_ relies on declarations from C++ header files to dynamically
+construct Python equivalent classes, functions, variables, etc.
+It is designed for use by large scale programs and supports modern C++.
+With PyPy, it leverages the built-in ``_cppyy`` module, allowing the JIT to
+remove most of the cross-language overhead.
 
-The :doc:`cppyy <cppyy>` module is written in RPython, thus PyPy's JIT is able to remove
-most cross-language call overhead.
+To install, run ``pip install cppyy``.
+Further details are available in the `full documentation`_.
 
-:doc:Full details are `available here <cppyy>`.
+.. _`full documentation`: https://cppyy.readthedocs.org/
 
-.. _installed separately: https://pypi.python.org/pypi/PyPy-cppyy-backend
-.. _Cling: https://root.cern.ch/cling
-
-.. toctree::
-
-   cppyy
 
 RPython Mixed Modules
 ---------------------
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
--- a/pypy/doc/getting-started-dev.rst
+++ b/pypy/doc/getting-started-dev.rst
@@ -35,8 +35,8 @@
 
 * Edit things.  Use ``hg diff`` to see what you changed.  Use ``hg add``
   to make Mercurial aware of new files you added, e.g. new test files.
-  Use ``hg status`` to see if there are such files.  Run tests!  (See
-  the rest of this page.)
+  Use ``hg status`` to see if there are such files.  Write and run tests!
+  (See the rest of this page.)
 
 * Commit regularly with ``hg commit``.  A one-line commit message is
   fine.  We love to have tons of commits; make one as soon as you have
@@ -113,6 +113,10 @@
 make sure you have the correct version installed which
 you can find out with the ``--version`` switch.
 
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
 Now on to running some tests.  PyPy has many different test directories
 and you can use shell completion to point at directories or files::
 
@@ -141,7 +145,7 @@
 
 .. _py.test testing tool: http://pytest.org
 .. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-
+.. _`build requirements`: build.html#install-build-time-dependencies
 
 Special Introspection Features of the Untranslated Python Interpreter
 ---------------------------------------------------------------------
diff --git a/pypy/doc/how-to-release.rst b/pypy/doc/how-to-release.rst
--- a/pypy/doc/how-to-release.rst
+++ b/pypy/doc/how-to-release.rst
@@ -40,6 +40,9 @@
   sure things are ported back to the trunk and to the branch as
   necessary.
 
+* Maybe bump the SOABI number in module/imp/importing. This has many
+  implications, so make sure the PyPy community agrees to the change.
+
 * Update and write documentation
 
   * update pypy/doc/contributor.rst (and possibly LICENSE)
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -5,6 +5,14 @@
 .. this is a revision shortly after release-pypy2.7-v5.8.0
 .. startrev: 558bd00b3dd8
 
+In previous versions of PyPy, ``instance.method`` would return always
+the same bound method object, when gotten out of the same instance (as
+far as ``is`` and ``id()`` can tell).  CPython doesn't do that.  Now
+PyPy, like CPython, returns a different bound method object every time.
+For ``type.method``, PyPy2 still returns always the same *unbound*
+method object; CPython does it for built-in types but not for
+user-defined types.
+
 .. branch: cffi-complex
 .. branch: cffi-char16-char32
 
@@ -25,3 +33,43 @@
 .. branch: cpyext-hash_notimpl
 
 If ``tp_hash`` is ``PyObject_HashNotImplemented``, set ``obj.__dict__['__hash__']`` to None
+
+.. branch: cppyy-packaging
+
+Renaming of ``cppyy`` to ``_cppyy``.
+The former is now an external package installable with ``pip install cppyy``.
+
+.. branch: Enable_PGO_for_clang
+
+.. branch: nopax
+
+At the end of translation, run ``attr -q -s pax.flags -V m`` on
+PAX-enabled systems on the produced binary.  This seems necessary
+because PyPy uses a JIT.
+
+.. branch: pypy_bytearray
+
+Improve ``bytearray`` performance (backported from py3.5)
+
+.. branch: gc-del-limit-growth
+
+Fix the bounds in the GC when allocating a lot of objects with finalizers,
+fixes issue #2590
+
+.. branch: arrays-force-less
+
+Small improvement to optimize list accesses with constant indexes better by
+throwing away information about them less eagerly.
+
+
+.. branch: getarrayitem-into-bridges
+
+More information is retained into a bridge: knowledge about the content of
+arrays (at fixed indices) is stored in guards (and thus available at the
+beginning of bridges). Also, some better feeding of information about known
+fields of constant objects into bridges.
+
+.. branch: cpyext-leakchecking
+
+Add support for leakfinder in cpyext tests (disabled for now, due to too many
+failures).
diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
--- a/pypy/interpreter/argument.py
+++ b/pypy/interpreter/argument.py
@@ -2,6 +2,7 @@
 Arguments objects.
 """
 from rpython.rlib.debug import make_sure_not_resized
+from rpython.rlib.objectmodel import not_rpython
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import enforceargs
 from rpython.rlib.rstring import StringBuilder
@@ -48,8 +49,8 @@
         # behaviour but produces better error messages
         self.methodcall = methodcall
 
+    @not_rpython
     def __repr__(self):
-        """ NOT_RPYTHON """
         name = self.__class__.__name__
         if not self.keywords:
             return '%s(%s)' % (name, self.arguments_w,)
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1,4 +1,5 @@
 import sys
+import py
 
 from rpython.rlib.cache import Cache
 from rpython.tool.uid import HUGEVAL_BYTES
@@ -1271,8 +1272,22 @@
             self.setitem(w_globals, w_key, self.builtin)
         return statement.exec_code(self, w_globals, w_locals)
 
+    @not_rpython
+    def appdef(self, source):
+        '''Create interp-level function object from app-level source.
+
+        The source should be in the same format as for space.appexec():
+            """(foo, bar): return 'baz'"""
+        '''
+        source = source.lstrip()
+        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
+        source = py.code.Source("def anonymous%s\n" % source)
+        w_glob = self.newdict(module=True)
+        self.exec_(str(source), w_glob, w_glob)
+        return self.getitem(w_glob, self.newtext('anonymous'))
+
     @specialize.arg(2)
-    def appexec(self, posargs_w, source):
+    def appexec(self, posargs_w, source, cache=True):
         """ return value from executing given source at applevel.
             The source must look like
                '''(x, y):
@@ -1280,7 +1295,11 @@
                        return result
                '''
         """
-        w_func = self.fromcache(AppExecCache).getorbuild(source)
+        if cache:
+            w_func = self.fromcache(AppExecCache).getorbuild(source)
+        else:
+            # NB: since appdef() is not-RPython, using cache=False also is.
+            w_func = self.appdef(source)
         args = Arguments(self, list(posargs_w))
         return self.call_args(w_func, args)
 
@@ -1817,15 +1836,7 @@
 class AppExecCache(SpaceCache):
     @not_rpython
     def build(cache, source):
-        space = cache.space
-        # XXX will change once we have our own compiler
-        import py
-        source = source.lstrip()
-        assert source.startswith('('), "incorrect header in:\n%s" % (source,)
-        source = py.code.Source("def anonymous%s\n" % source)
-        w_glob = space.newdict(module=True)
-        space.exec_(str(source), w_glob, w_glob)
-        return space.getitem(w_glob, space.newtext('anonymous'))
+        return cache.space.appdef(source)
 
 
 # Table describing the regular part of the interface of object spaces,
diff --git a/pypy/interpreter/error.py b/pypy/interpreter/error.py
--- a/pypy/interpreter/error.py
+++ b/pypy/interpreter/error.py
@@ -7,7 +7,7 @@
 
 from rpython.rlib import jit
 from rpython.rlib.objectmodel import we_are_translated, specialize
-from rpython.rlib.objectmodel import dont_inline
+from rpython.rlib.objectmodel import dont_inline, not_rpython
 from rpython.rlib import rstack, rstackovf
 from rpython.rlib import rwin32
 from rpython.rlib import runicode
@@ -65,8 +65,9 @@
                 self.match(space, space.w_KeyboardInterrupt))
         # note: an extra case is added in OpErrFmtNoArgs
 
+    @not_rpython
     def __str__(self):
-        "NOT_RPYTHON: Convenience for tracebacks."
+        "Convenience for tracebacks."
         s = self._w_value
         space = getattr(self.w_type, 'space', None)
         if space is not None:
@@ -119,15 +120,16 @@
             if RECORD_INTERPLEVEL_TRACEBACK:
                 self.debug_excs.append(sys.exc_info())
 
+    @not_rpython
     def print_application_traceback(self, space, file=None):
-        "NOT_RPYTHON: Dump a standard application-level traceback."
+        "Dump a standard application-level traceback."
         if file is None:
             file = sys.stderr
         self.print_app_tb_only(file)
         print >> file, self.errorstr(space)
 
+    @not_rpython
     def print_app_tb_only(self, file):
-        "NOT_RPYTHON"
         tb = self._application_traceback
         if tb:
             import linecache
@@ -154,8 +156,9 @@
                     print >> file, l
                 tb = tb.next
 
+    @not_rpython
     def print_detailed_traceback(self, space=None, file=None):
-        """NOT_RPYTHON: Dump a nice detailed interpreter- and
+        """Dump a nice detailed interpreter- and
         application-level traceback, useful to debug the interpreter."""
         if file is None:
             file = sys.stderr
diff --git a/pypy/interpreter/executioncontext.py b/pypy/interpreter/executioncontext.py
--- a/pypy/interpreter/executioncontext.py
+++ b/pypy/interpreter/executioncontext.py
@@ -1,6 +1,7 @@
 import sys
 from pypy.interpreter.error import OperationError, get_cleared_operation_error
 from rpython.rlib.unroll import unrolling_iterable
+from rpython.rlib.objectmodel import specialize, not_rpython
 from rpython.rlib import jit, rgc, objectmodel
 
 TICK_COUNTER_STEP = 100
@@ -410,8 +411,9 @@
             # to run at the next possible bytecode
             self.reset_ticker(-1)
 
+    @not_rpython
     def register_periodic_action(self, action, use_bytecode_counter):
-        """NOT_RPYTHON:
+        """
         Register the PeriodicAsyncAction action to be called whenever the
         tick counter becomes smaller than 0.  If 'use_bytecode_counter' is
         True, make sure that we decrease the tick counter at every bytecode.
diff --git a/pypy/interpreter/function.py b/pypy/interpreter/function.py
--- a/pypy/interpreter/function.py
+++ b/pypy/interpreter/function.py
@@ -516,8 +516,9 @@
 
     def __init__(self, space, w_function, w_instance):
         self.space = space
+        assert w_instance is not None   # unbound methods only exist in Python 2
         self.w_function = w_function
-        self.w_instance = w_instance   # or None
+        self.w_instance = w_instance
 
     def descr_method__new__(space, w_subtype, w_function, w_instance):
         if space.is_w(w_instance, space.w_None):
@@ -577,24 +578,6 @@
             return space.w_False
         return space.newbool(space.eq_w(self.w_function, w_other.w_function))
 
-    def is_w(self, space, other):
-        if not isinstance(other, Method):
-            return False
-        return (self.w_instance is other.w_instance and
-                self.w_function is other.w_function)
-
-    def immutable_unique_id(self, space):
-        from pypy.objspace.std.util import IDTAG_METHOD as tag
-        from pypy.objspace.std.util import IDTAG_SHIFT
-        if self.w_instance is not None:
-            id = space.bigint_w(space.id(self.w_instance))
-            id = id.lshift(LONG_BIT)
-        else:
-            id = rbigint.fromint(0)
-        id = id.or_(space.bigint_w(space.id(self.w_function)))
-        id = id.lshift(IDTAG_SHIFT).int_or_(tag)
-        return space.newlong_from_rbigint(id)
-
     def descr_method_hash(self):
         space = self.space
         w_result = space.hash(self.w_function)
@@ -606,15 +589,16 @@
         from pypy.interpreter.gateway import BuiltinCode
         w_mod    = space.getbuiltinmodule('_pickle_support')
         mod      = space.interp_w(MixedModule, w_mod)
-        w_instance = self.w_instance or space.w_None
+        w_instance = self.w_instance
         w_function = self.w_function
         if (isinstance(w_function, Function) and
                 isinstance(w_function.code, BuiltinCode)):
             new_inst = mod.get('builtin_method_new')
             tup = [w_instance, space.newtext(w_function.name)]
         else:
-            new_inst = mod.get('method_new')
-            tup = [self.w_function, w_instance]
+            w_builtins = space.getbuiltinmodule('builtins')
+            new_inst = space.getattr(w_builtins, space.newtext('getattr'))
+            tup = [w_instance, space.newunicode(w_function.getname(space))]
         return space.newtuple([new_inst, space.newtuple(tup)])
 
 
diff --git a/pypy/interpreter/gateway.py b/pypy/interpreter/gateway.py
--- a/pypy/interpreter/gateway.py
+++ b/pypy/interpreter/gateway.py
@@ -23,7 +23,7 @@
     DescrMismatch)
 from pypy.interpreter.error import OperationError, oefmt
 from pypy.interpreter.function import ClassMethod, FunctionWithFixedCode
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, not_rpython
 from rpython.rlib.rarithmetic import r_longlong, r_int, r_ulonglong, r_uint
 from rpython.tool.sourcetools import func_with_new_name, compile2
 
@@ -75,8 +75,8 @@
     def _freeze_(self):
         return True
 
+    @not_rpython
     def unwrap(self, space, w_value):
-        """NOT_RPYTHON"""
         raise NotImplementedError
 
 
@@ -399,8 +399,8 @@
 class BuiltinActivation(object):
     _immutable_ = True
 
+    @not_rpython
     def __init__(self, behavior):

From pypy.commits at gmail.com  Wed Aug 30 07:54:53 2017
From: pypy.commits at gmail.com (stevie_92)
Date: Wed, 30 Aug 2017 04:54:53 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-gc-trialdeletion: Implemented
 non-incremental cycle detection, removed simple trial deletion
Message-ID: <59a6a78d.48badf0a.f1519.e126@mx.google.com>

Author: Stefan Beyer <home at sbeyer.at>
Branch: cpyext-gc-trialdeletion
Changeset: r92281:2fb71fc31ef4
Date: 2017-08-06 13:48 +0200
http://bitbucket.org/pypy/pypy/changeset/2fb71fc31ef4/

Log:	Implemented non-incremental cycle detection, removed simple trial
	deletion

diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py
--- a/pypy/module/cpyext/pyobject.py
+++ b/pypy/module/cpyext/pyobject.py
@@ -15,6 +15,10 @@
 from rpython.rlib.objectmodel import keepalive_until_here
 from rpython.rtyper.annlowlevel import llhelper
 from rpython.rlib import rawrefcount
+from rpython.rlib.rawrefcount import (
+    REFCNT_MASK, REFCNT_FROM_PYPY, REFCNT_OVERFLOW, REFCNT_CYCLE_BUFFERED,
+    REFCNT_CLR_MASK, REFCNT_CLR_GREEN, REFCNT_CLR_PURPLE, 
+    W_MARKER_DEALLOCATING)
 from rpython.rlib.debug import fatalerror, debug_print
 from pypy.module.cpyext.api import slot_function
 from pypy.module.cpyext.typeobjectdefs import visitproc
@@ -190,9 +194,6 @@
     py_obj.c_ob_refcnt += rawrefcount.REFCNT_FROM_PYPY
     rawrefcount.create_link_pypy(w_obj, py_obj)
 
-
-w_marker_deallocating = W_Root()
-
 def from_ref(space, ref):
     """
     Finds the interpreter object corresponding to the given reference.  If the
@@ -203,7 +204,7 @@
         return None
     w_obj = rawrefcount.to_obj(W_Root, ref)
     if w_obj is not None:
-        if w_obj is not w_marker_deallocating:
+        if w_obj is not W_MARKER_DEALLOCATING:
             return w_obj
         fatalerror(
             "*** Invalid usage of a dying CPython object ***\n"
@@ -250,7 +251,7 @@
 
 def pyobj_has_w_obj(pyobj):
     w_obj = rawrefcount.to_obj(W_Root, pyobj)
-    return w_obj is not None and w_obj is not w_marker_deallocating
+    return w_obj is not None and w_obj is not W_MARKER_DEALLOCATING
 
 
 def is_pyobj(x):
@@ -270,27 +271,6 @@
         hop.exception_cannot_occur()
         return hop.inputconst(lltype.Bool, hop.s_result.const)
 
-def _decref(pyobj):
-    if pyobj.c_ob_refcnt & rawrefcount.REFCNT_OVERFLOW == 0:
-        pyobj.c_ob_refcnt -= 1
-    else:
-        if pyobj.c_ob_refcnt & rawrefcount.REFCNT_MASK \
-           == rawrefcount.REFCNT_OVERFLOW:
-            pyobj.c_ob_refcnt -= 1
-        elif rawrefcount.overflow_sub(pyobj):
-            pyobj.c_ob_refcnt -= 1
-
-def _incref(pyobj):
-    if pyobj.c_ob_refcnt & rawrefcount.REFCNT_OVERFLOW == 0:
-        pyobj.c_ob_refcnt += 1
-    else:
-        if pyobj.c_ob_refcnt & rawrefcount.REFCNT_MASK \
-           == rawrefcount.REFCNT_OVERFLOW:
-            pyobj.c_ob_refcnt += 1
-            rawrefcount.overflow_new(pyobj)
-        else:
-            rawrefcount.overflow_add(pyobj)
-
 @specialize.ll()
 def make_ref(space, obj, w_userdata=None):
     """Increment the reference counter of the PyObject and return it.
@@ -301,7 +281,7 @@
     else:
         pyobj = as_pyobj(space, obj, w_userdata)
     if pyobj:
-        _incref(pyobj)
+        rawrefcount.incref(pyobj)
         if not is_pyobj(obj):
             keepalive_until_here(obj)
     return pyobj
@@ -321,7 +301,7 @@
         w_obj = obj
         pyobj = as_pyobj(space, w_obj)
     if pyobj:
-        _decref(pyobj)
+        rawrefcount.decref(pyobj)
         keepalive_until_here(w_obj)
     return w_obj
 
@@ -334,122 +314,30 @@
     if is_pyobj(obj):
         obj = rffi.cast(PyObject, obj)
         if obj:
-            _decref(obj)
-
-            if obj.c_ob_refcnt & rawrefcount.REFCNT_MASK == 0 and \
-               rawrefcount.get_trialdeletion_phase() != 1:
-                if obj.c_ob_refcnt & rawrefcount.REFCNT_FROM_PYPY == 0:
+            rawrefcount.decref(obj)
+            rc = obj.c_ob_refcnt
+            if (rc & REFCNT_MASK == 0):
+                if (rc & REFCNT_FROM_PYPY == 0 and
+                   rc & REFCNT_CLR_MASK != REFCNT_CLR_PURPLE):
                     _Py_Dealloc(space, obj)
-            elif obj.c_ob_refcnt & rawrefcount.REFCNT_CLR_GREEN == 0:
-                if rawrefcount.get_trialdeletion_phase() == 0:
-                    trial_delete(space, obj)
+            elif (rc & REFCNT_CLR_MASK != REFCNT_CLR_GREEN):
+                possible_root(space, obj)
     else:
         get_w_obj_and_decref(space, obj)
 
- at specialize.ll()
-def refcnt_overflow(space, obj):
-    if is_pyobj(obj):
-        pyobj = rffi.cast(PyObject, obj)
-    else:
-        pyobj = as_pyobj(space, obj, None)
-    if pyobj:
-        if (pyobj.c_ob_refcnt & rawrefcount.REFCNT_MASK ==
-           rawrefcount.REFCNT_OVERFLOW):
-            return rawrefcount.REFCNT_OVERFLOW
-        else:
-            return (pyobj.c_ob_refcnt & rawrefcount.REFCNT_MASK) \
-                + rawrefcount.overflow_get(pyobj)
-    return 0
-
-def traverse(space, obj, visit):
-    from pypy.module.cpyext.api import generic_cpy_call
-    if obj.c_ob_type and obj.c_ob_type.c_tp_traverse:
-        generic_cpy_call(space, obj.c_ob_type.c_tp_traverse, obj, visit,
-                         rffi.cast(rffi.VOIDP, obj))
-
-def clear(space, obj):
-    from pypy.module.cpyext.api import generic_cpy_call
-    if obj.c_ob_type:
-        generic_cpy_call(space, obj.c_ob_type.c_tp_clear, obj)
-
- at slot_function([PyObject, rffi.VOIDP], rffi.INT_real, error=-1)
-def visit_decref(space, obj, args):
-    _decref(obj)
-    debug_print("visited dec", obj, "new refcnt", obj.c_ob_refcnt)
-    if (obj not in rawrefcount.get_visited()):
-        rawrefcount.add_visited(obj)
-        from pypy.module.cpyext.slotdefs import llslot
-        traverse(space, obj, rffi.cast(visitproc, llslot(space, visit_decref)))
-    return 0
-
- at slot_function([PyObject, rffi.VOIDP], rffi.INT_real, error=-1)
-def visit_incref(space, obj, args):
-    _incref(obj)
-    debug_print("visited inc", obj, "new refcnt", obj.c_ob_refcnt)
-    if (obj not in rawrefcount.get_visited()):
-        rawrefcount.add_visited(obj)
-        from pypy.module.cpyext.slotdefs import llslot
-        traverse(space, obj, rffi.cast(visitproc, llslot(space, visit_incref)))
-    return 0
-
- at specialize.ll()
-def trial_delete(space, obj):
+def possible_root(space, obj):
+    debug_print("possible root", obj)
+    rc = obj.c_ob_refcnt
     if not obj.c_ob_type or not obj.c_ob_type.c_tp_traverse:
-        obj.c_ob_refcnt = obj.c_ob_refcnt | rawrefcount.REFCNT_CLR_GREEN
-        return
-
-    from pypy.module.cpyext.slotdefs import llslot
-    visitproc_incref = rffi.cast(visitproc, llslot(space, visit_incref))
-    visitproc_decref = rffi.cast(visitproc, llslot(space, visit_decref))
-
-    rawrefcount.set_trialdeletion_phase(1)
-
-    debug_print("trial_delete", obj, "refct after decref", obj.c_ob_refcnt)
-
-    debug_print("decref phase")
-    rawrefcount.clear_visited()
-    rawrefcount.add_visited(obj)
-    traverse(space, obj, visitproc_decref)
-
-    debug_print("checkref phase")
-    visited = []
-    alive = []
-    for visited_obj in rawrefcount.get_visited():
-        visited.append(visited_obj)
-        if visited_obj.c_ob_refcnt != 0 and \
-           visited_obj.c_ob_refcnt != rawrefcount.REFCNT_FROM_PYPY:
-            alive.append(visited_obj)
-            debug_print("alive", visited_obj)
-
-    debug_print("incref phase")
-    rawrefcount.clear_visited()
-    for alive_obj in alive:
-        if alive_obj not in rawrefcount.get_visited():
-            rawrefcount.add_visited(alive_obj)
-            traverse(space, alive_obj, visitproc_incref)
-
-    alive = []
-    for alive_obj in rawrefcount.get_visited():
-        debug_print("alive", alive_obj, alive_obj.c_ob_refcnt)
-        alive.append(alive_obj)
-
-    for reachable_obj in visited:
-        if reachable_obj not in rawrefcount.get_visited():
-            rawrefcount.add_visited(reachable_obj)
-            traverse(space, reachable_obj, visitproc_incref)
-
-    debug_print("clear phase")
-    rawrefcount.set_trialdeletion_phase(2)
-
-    for reachable_obj in visited:
-        if reachable_obj not in alive:
-            if reachable_obj.c_ob_refcnt < rawrefcount.REFCNT_FROM_PYPY \
-               and reachable_obj.c_ob_refcnt > 0:
-                debug_print("clear", reachable_obj)
-                clear(space, reachable_obj)
-
-    rawrefcount.set_trialdeletion_phase(0)
-    rawrefcount.clear_visited()
+        debug_print("mark green", obj)
+        rc = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_GREEN
+    elif rc & REFCNT_CLR_MASK != REFCNT_CLR_PURPLE:
+        rc = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_PURPLE
+        if rc & REFCNT_CYCLE_BUFFERED == 0:
+            debug_print("mark purple", obj)
+            rawrefcount.buffer_pyobj(obj)
+            rc = rc | REFCNT_CYCLE_BUFFERED
+    obj.c_ob_refcnt = rc
 
 @cpython_api([PyObject], lltype.Void)
 def Py_IncRef(space, obj):
@@ -463,6 +351,20 @@
 def _Py_RefCnt_Overflow(space, obj):
     return refcnt_overflow(space, obj)
 
+ at specialize.ll()
+def refcnt_overflow(space, obj):
+    if is_pyobj(obj):
+        pyobj = rffi.cast(PyObject, obj)
+    else:
+        pyobj = as_pyobj(space, obj, None)
+    if pyobj:
+        if (pyobj.c_ob_refcnt & REFCNT_MASK == REFCNT_OVERFLOW):
+            return REFCNT_OVERFLOW
+        else:
+            return (pyobj.c_ob_refcnt & REFCNT_MASK) + \
+                rawrefcount.overflow_get(pyobj)
+    return 0
+
 @cpython_api([PyObject], lltype.Void)
 def _Py_NewReference(space, obj):
     obj.c_ob_refcnt = 1
@@ -477,7 +379,7 @@
     pto = obj.c_ob_type
     #print >>sys.stderr, "Calling dealloc slot", pto.c_tp_dealloc, "of", obj, \
     #      "'s type which is", rffi.charp2str(pto.c_tp_name)
-    rawrefcount.mark_deallocating(w_marker_deallocating, obj)
+    rawrefcount.mark_deallocating(W_MARKER_DEALLOCATING, obj)
     generic_cpy_call(space, pto.c_tp_dealloc, obj)
 
 @cpython_api([rffi.VOIDP], lltype.Signed, error=CANNOT_FAIL)
diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py
--- a/rpython/memory/gc/incminimark.py
+++ b/rpython/memory/gc/incminimark.py
@@ -73,6 +73,8 @@
 from rpython.rlib.debug import ll_assert, debug_print, debug_start, debug_stop
 from rpython.rlib.objectmodel import specialize
 from rpython.memory.gc.minimarkpage import out_of_memory
+from pypy.module.cpyext.api import slot_function, PyObject
+from rpython.rtyper.lltypesystem import rffi
 
 #
 # Handles the objects in 2 generations:
@@ -187,6 +189,105 @@
                               ('forw', llmemory.Address))
 FORWARDSTUBPTR = lltype.Ptr(FORWARDSTUB)
 NURSARRAY = lltype.Array(llmemory.Address)
+VISIT_FUNCTYPE = rffi.CCallback([PyObject, rffi.VOIDP],
+                                rffi.INT_real)
+
+def traverse(obj, func_ptr):
+    from pypy.module.cpyext.api import generic_cpy_call
+    from pypy.module.cpyext.typeobjectdefs import visitproc
+    if obj.c_ob_type and obj.c_ob_type.c_tp_traverse:
+        visitproc_ptr = rffi.cast(visitproc, func_ptr)
+        generic_cpy_call(True, obj.c_ob_type.c_tp_traverse, obj,
+                         visitproc_ptr, rffi.cast(rffi.VOIDP, obj))
+
+def visit_mark_gray(obj, args):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_GREEN,
+                                          REFCNT_CLR_MASK,
+                                          decref)
+    decref(obj)
+    rc = obj.c_ob_refcnt
+    if rc & REFCNT_CLR_MASK != REFCNT_CLR_GREEN:
+        mark_gray_recursive(obj)
+    return rffi.cast(rffi.INT_real, 0)
+
+def mark_gray_recursive(obj):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_GRAY,
+                                          REFCNT_CLR_MASK)
+    from rpython.rtyper.annlowlevel import llhelper
+    debug_print("mark_gray_recursive", obj)
+    rc = obj.c_ob_refcnt
+    if rc & REFCNT_CLR_MASK != REFCNT_CLR_GRAY:
+        obj.c_ob_refcnt = obj.c_ob_refcnt & ~REFCNT_CLR_MASK | REFCNT_CLR_GRAY
+        func_ptr = llhelper(VISIT_FUNCTYPE, visit_mark_gray)
+        traverse(obj, func_ptr)
+
+def visit_scan_black(obj, args):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_BLACK,
+                                          REFCNT_CLR_MASK,
+                                          REFCNT_CLR_GREEN,
+                                          incref)
+    incref(obj)
+    rc = obj.c_ob_refcnt
+    if (rc & REFCNT_CLR_MASK != REFCNT_CLR_BLACK and
+       rc & REFCNT_CLR_MASK != REFCNT_CLR_GREEN):
+        scan_black_recursive(obj)
+    return rffi.cast(rffi.INT_real, 0)
+
+def scan_black_recursive(obj):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_BLACK,
+                                          REFCNT_CLR_MASK)
+    from rpython.rtyper.annlowlevel import llhelper
+    debug_print("scan_black_recursive", obj)
+    rc = obj.c_ob_refcnt
+    obj.c_ob_refcnt = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_BLACK
+    func_ptr = llhelper(VISIT_FUNCTYPE, visit_scan_black)
+    traverse(obj, func_ptr)
+
+def visit_scan(obj, args):
+    scan_recursive(obj)
+    return rffi.cast(rffi.INT_real, 0)
+
+def scan_recursive(obj):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_WHITE,
+                                          REFCNT_CLR_GRAY,
+                                          REFCNT_CLR_GREEN,
+                                          REFCNT_CLR_MASK,
+                                          REFCNT_MASK)
+    from rpython.rtyper.annlowlevel import llhelper
+    debug_print("scan_recursive", obj)
+    rc = obj.c_ob_refcnt
+    if (rc & REFCNT_CLR_MASK == REFCNT_CLR_GRAY or
+       rc & REFCNT_CLR_MASK == REFCNT_CLR_GREEN):
+        if rc & REFCNT_MASK > 0 and rc & REFCNT_CLR_MASK != REFCNT_CLR_GREEN:
+            scan_black_recursive(obj)
+        else:
+            obj.c_ob_refcnt = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_WHITE
+            func_ptr = llhelper(VISIT_FUNCTYPE, visit_scan)
+            traverse(obj, func_ptr)
+
+def visit_collect_white(obj, args):
+    collect_white_recursive(obj)
+    return rffi.cast(rffi.INT_real, 0)
+
+def collect_white_recursive(obj):
+    from rpython.rlib.rawrefcount import (REFCNT_CLR_WHITE,
+                                          REFCNT_CLR_BLACK,
+                                          REFCNT_CLR_MASK,
+                                          REFCNT_CYCLE_BUFFERED,
+                                          REFCNT_FROM_PYPY)
+    from pypy.module.cpyext.api import generic_cpy_call
+    from rpython.rtyper.annlowlevel import llhelper
+    debug_print("collect_white_recursive", obj)
+    rc = obj.c_ob_refcnt
+    if (rc & REFCNT_CLR_MASK == REFCNT_CLR_WHITE and
+       rc & REFCNT_CYCLE_BUFFERED == 0):
+        obj.c_ob_refcnt = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_BLACK
+        func_ptr = llhelper(VISIT_FUNCTYPE, visit_collect_white)
+        traverse(obj, func_ptr)
+        if (rc & REFCNT_FROM_PYPY == 0 and
+           obj.c_ob_type and obj.c_ob_type.c_tp_free):
+            debug_print("free", obj)
+            generic_cpy_call(True, obj.c_ob_type.c_tp_free, obj)
 
 # ____________________________________________________________
 
@@ -1685,6 +1786,7 @@
         #
         # visit the P list from rawrefcount, if enabled.
         if self.rrc_enabled:
+            self.rrc_collect_cycles()  # TODO only for testing
             self.rrc_minor_collection_trace()
         #
         # visit the "probably young" objects with finalizers.  They
@@ -2303,6 +2405,7 @@
                 self.visit_all_objects()
                 #
                 if self.rrc_enabled:
+                    self.rrc_collect_cycles()
                     self.rrc_major_collection_trace()
                 #
                 ll_assert(not (self.probably_young_objects_with_finalizers
@@ -2901,13 +3004,13 @@
 
     _ADDRARRAY = lltype.Array(llmemory.Address, hints={'nolength': True})
     PYOBJ_HDR = lltype.Struct('GCHdr_PyObject',
-                              ('ob_refcnt', lltype.Signed),
-                              ('ob_pypy_link', lltype.Signed))
+                              ('c_ob_refcnt', lltype.Signed),
+                              ('c_ob_pypy_link', lltype.Signed))
     PYOBJ_HDR_PTR = lltype.Ptr(PYOBJ_HDR)
     RAWREFCOUNT_DEALLOC_TRIGGER = lltype.Ptr(lltype.FuncType([], lltype.Void))
 
     def _pyobj(self, pyobjaddr):
-        return llmemory.cast_adr_to_ptr(pyobjaddr, self.PYOBJ_HDR_PTR)
+        return llmemory.cast_adr_to_ptr(pyobjaddr, lltype.Ptr(PyObject.TO))
 
     def rawrefcount_init(self, dealloc_trigger_callback):
         # see pypy/doc/discussion/rawrefcount.rst
@@ -2916,6 +3019,7 @@
             self.rrc_p_list_old   = self.AddressStack()
             self.rrc_o_list_young = self.AddressStack()
             self.rrc_o_list_old   = self.AddressStack()
+            self.rrc_buffered     = self.AddressStack()
             self.rrc_p_dict       = self.AddressDict()  # non-nursery keys only
             self.rrc_p_dict_nurs  = self.AddressDict()  # nursery keys only
             self.rrc_dealloc_trigger_callback = dealloc_trigger_callback
@@ -2937,7 +3041,7 @@
         ll_assert(self.rrc_enabled, "rawrefcount.init not called")
         obj = llmemory.cast_ptr_to_adr(gcobj)
         objint = llmemory.cast_adr_to_int(obj, "symbolic")
-        self._pyobj(pyobject).ob_pypy_link = objint
+        self._pyobj(pyobject).c_ob_pypy_link = objint
         #
         lst = self.rrc_p_list_young
         if self.is_in_nursery(obj):
@@ -2957,14 +3061,17 @@
         else:
             self.rrc_o_list_old.append(pyobject)
         objint = llmemory.cast_adr_to_int(obj, "symbolic")
-        self._pyobj(pyobject).ob_pypy_link = objint
+        self._pyobj(pyobject).c_ob_pypy_link = objint
         # there is no rrc_o_dict
 
+    def rawrefcount_buffer_pyobj(self, pyobject):
+        self.rrc_buffered.append(pyobject)
+
     def rawrefcount_mark_deallocating(self, gcobj, pyobject):
         ll_assert(self.rrc_enabled, "rawrefcount.init not called")
         obj = llmemory.cast_ptr_to_adr(gcobj)   # should be a prebuilt obj
         objint = llmemory.cast_adr_to_int(obj, "symbolic")
-        self._pyobj(pyobject).ob_pypy_link = objint
+        self._pyobj(pyobject).c_ob_pypy_link = objint
 
     def rawrefcount_from_obj(self, gcobj):
         obj = llmemory.cast_ptr_to_adr(gcobj)
@@ -2975,7 +3082,7 @@
         return dct.get(obj)
 
     def rawrefcount_to_obj(self, pyobject):
-        obj = llmemory.cast_int_to_adr(self._pyobj(pyobject).ob_pypy_link)
+        obj = llmemory.cast_int_to_adr(self._pyobj(pyobject).c_ob_pypy_link)
         return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF)
 
     def rawrefcount_next_dead(self):
@@ -2996,15 +3103,13 @@
                                       self.singleaddr)
 
     def _rrc_minor_trace(self, pyobject, singleaddr):
-        from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
-        from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
+        from rpython.rlib.rawrefcount import REFCNT_MASK
         #
-        rc = self._pyobj(pyobject).ob_refcnt
-        if rc == REFCNT_FROM_PYPY or rc == REFCNT_FROM_PYPY_LIGHT:
+        rc = self._pyobj(pyobject).c_ob_refcnt
+        if rc & REFCNT_MASK == 0:
             pass     # the corresponding object may die
         else:
-            # force the corresponding object to be alive
-            intobj = self._pyobj(pyobject).ob_pypy_link
+            intobj = self._pyobj(pyobject).c_ob_pypy_link
             singleaddr.address[0] = llmemory.cast_int_to_adr(intobj)
             self._trace_drag_out1(singleaddr)
 
@@ -3021,14 +3126,14 @@
                                             no_o_dict)
 
     def _rrc_minor_free(self, pyobject, surviving_list, surviving_dict):
-        intobj = self._pyobj(pyobject).ob_pypy_link
+        intobj = self._pyobj(pyobject).c_ob_pypy_link
         obj = llmemory.cast_int_to_adr(intobj)
         if self.is_in_nursery(obj):
             if self.is_forwarded(obj):
                 # Common case: survives and moves
                 obj = self.get_forwarding_address(obj)
                 intobj = llmemory.cast_adr_to_int(obj, "symbolic")
-                self._pyobj(pyobject).ob_pypy_link = intobj
+                self._pyobj(pyobject).c_ob_pypy_link = intobj
                 surviving = True
                 if surviving_dict:
                     # Surviving nursery object: was originally in
@@ -3059,23 +3164,24 @@
     def _rrc_free(self, pyobject):
         from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
         from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
+        from rpython.rlib.rawrefcount import REFCNT_MASK
         #
-        rc = self._pyobj(pyobject).ob_refcnt
+        rc = self._pyobj(pyobject).c_ob_refcnt
         if rc >= REFCNT_FROM_PYPY_LIGHT:
             rc -= REFCNT_FROM_PYPY_LIGHT
-            if rc == 0:
+            if rc & REFCNT_MASK == 0:
                 lltype.free(self._pyobj(pyobject), flavor='raw')
             else:
                 # can only occur if LIGHT is used in create_link_pyobj()
-                self._pyobj(pyobject).ob_refcnt = rc
-                self._pyobj(pyobject).ob_pypy_link = 0
+                self._pyobj(pyobject).c_ob_refcnt = rc
+                self._pyobj(pyobject).c_ob_pypy_link = 0
         else:
             ll_assert(rc >= REFCNT_FROM_PYPY, "refcount underflow?")
             ll_assert(rc < int(REFCNT_FROM_PYPY_LIGHT * 0.99),
                       "refcount underflow from REFCNT_FROM_PYPY_LIGHT?")
             rc -= REFCNT_FROM_PYPY
-            self._pyobj(pyobject).ob_pypy_link = 0
-            if rc == 0:
+            self._pyobj(pyobject).c_ob_pypy_link = 0
+            if rc & REFCNT_MASK == 0:
                 self.rrc_dealloc_pending.append(pyobject)
                 # an object with refcnt == 0 cannot stay around waiting
                 # for its deallocator to be called.  Some code (lxml)
@@ -3086,22 +3192,62 @@
                 # because after a Py_INCREF()/Py_DECREF() on it, its
                 # tp_dealloc is also called!
                 rc = 1
-            self._pyobj(pyobject).ob_refcnt = rc
+            self._pyobj(pyobject).c_ob_refcnt = rc
     _rrc_free._always_inline_ = True
 
+    def rrc_collect_cycles(self):
+        self.rrc_buffered.foreach(self._rrc_cycle_mark_roots, None)
+        self.rrc_buffered.foreach(self._rrc_cycle_scan_roots, None)
+        self.rrc_buffered.foreach(self._rrc_cycle_collect_roots, None)
+
+    def _rrc_cycle_mark_roots(self, pyobject, ignore):
+        from pypy.module.cpyext.api import generic_cpy_call
+        from rpython.rlib.rawrefcount import (REFCNT_CYCLE_BUFFERED,
+                                              REFCNT_CLR_MASK,
+                                              REFCNT_CLR_PURPLE,
+                                              REFCNT_MASK,
+                                              W_MARKER_DEALLOCATING,
+                                              mark_deallocating)
+        obj = self._pyobj(pyobject)
+        rc = obj.c_ob_refcnt
+        debug_print("_rrc_cycle_mark_roots", obj)
+        if rc & REFCNT_CLR_MASK == REFCNT_CLR_PURPLE and \
+           rc & REFCNT_MASK > 0:
+            mark_gray_recursive(obj)
+        else:
+            obj.c_ob_refcnt = rc & ~REFCNT_CYCLE_BUFFERED
+            self.rrc_buffered.remove(pyobject)
+            if rc & REFCNT_MASK == 0:
+                mark_deallocating(W_MARKER_DEALLOCATING, obj)
+                generic_cpy_call(True, obj.c_ob_type.c_tp_dealloc, obj)
+
+    def _rrc_cycle_scan_roots(self, pyobject, ignore):
+        obj = self._pyobj(pyobject)
+        debug_print("_rrc_cycle_scan_roots", obj)
+        scan_recursive(obj)
+
+    def _rrc_cycle_collect_roots(self, pyobject, ignore):
+        from rpython.rlib.rawrefcount import REFCNT_CYCLE_BUFFERED
+        obj = self._pyobj(pyobject)
+        debug_print("_rrc_cycle_collect_roots", obj)
+        self.rrc_buffered.remove(pyobject)
+        obj.c_ob_refcnt = obj.c_ob_refcnt & ~REFCNT_CYCLE_BUFFERED
+        collect_white_recursive(obj)
+
     def rrc_major_collection_trace(self):
         self.rrc_p_list_old.foreach(self._rrc_major_trace, None)
 
     def _rrc_major_trace(self, pyobject, ignore):
-        from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
-        from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
+        from rpython.rlib.rawrefcount import (REFCNT_FROM_PYPY,
+                                              REFCNT_FROM_PYPY_LIGHT,
+                                              REFCNT_MASK)
         #
-        rc = self._pyobj(pyobject).ob_refcnt
-        if rc == REFCNT_FROM_PYPY or rc == REFCNT_FROM_PYPY_LIGHT:
+        rc = self._pyobj(pyobject).c_ob_refcnt
+        if rc & REFCNT_MASK == 0:
             pass     # the corresponding object may die
         else:
             # force the corresponding object to be alive
-            intobj = self._pyobj(pyobject).ob_pypy_link
+            intobj = self._pyobj(pyobject).c_ob_pypy_link
             obj = llmemory.cast_int_to_adr(intobj)
             self.objects_to_trace.append(obj)
             self.visit_all_objects()
@@ -3131,7 +3277,7 @@
         # This is true if the obj has one of the following two flags:
         #  * GCFLAG_VISITED: was seen during tracing
         #  * GCFLAG_NO_HEAP_PTRS: immortal object never traced (so far)
-        intobj = self._pyobj(pyobject).ob_pypy_link
+        intobj = self._pyobj(pyobject).c_ob_pypy_link
         obj = llmemory.cast_int_to_adr(intobj)
         if self.header(obj).tid & (GCFLAG_VISITED | GCFLAG_NO_HEAP_PTRS):
             surviving_list.append(pyobject)
diff --git a/rpython/memory/gc/test/test_rawrefcount.py b/rpython/memory/gc/test/test_rawrefcount.py
--- a/rpython/memory/gc/test/test_rawrefcount.py
+++ b/rpython/memory/gc/test/test_rawrefcount.py
@@ -4,6 +4,7 @@
 from rpython.memory.gc.test.test_direct import BaseDirectGCTest
 from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
 from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
+from pypy.module.cpyext.api import PyObject, PyTypeObject
 
 PYOBJ_HDR = IncrementalMiniMarkGC.PYOBJ_HDR
 PYOBJ_HDR_PTR = IncrementalMiniMarkGC.PYOBJ_HDR_PTR
@@ -56,21 +57,22 @@
             self._collect(major=False)
             p1 = self.stackroots.pop()
         p1ref = lltype.cast_opaque_ptr(llmemory.GCREF, p1)
-        r1 = lltype.malloc(PYOBJ_HDR, flavor='raw', immortal=create_immortal)
-        r1.ob_refcnt = rc
-        r1.ob_pypy_link = 0
+        r1 = lltype.malloc(PyObject.TO, flavor='raw', immortal=create_immortal)
+        r1.c_ob_refcnt = rc
+        r1.c_ob_pypy_link = 0
+        r1.c_ob_type = lltype.nullptr(PyTypeObject)
         r1addr = llmemory.cast_ptr_to_adr(r1)
         if is_pyobj:
             assert not is_light
             self.gc.rawrefcount_create_link_pyobj(p1ref, r1addr)
         else:
             self.gc.rawrefcount_create_link_pypy(p1ref, r1addr)
-        assert r1.ob_refcnt == rc
-        assert r1.ob_pypy_link != 0
+        assert r1.c_ob_refcnt == rc
+        assert r1.c_ob_pypy_link != 0
 
         def check_alive(extra_refcount):
-            assert r1.ob_refcnt == rc + extra_refcount
-            assert r1.ob_pypy_link != 0
+            assert r1.c_ob_refcnt == rc + extra_refcount
+            assert r1.c_ob_pypy_link != 0
             p1ref = self.gc.rawrefcount_to_obj(r1addr)
             p1 = lltype.cast_opaque_ptr(lltype.Ptr(S), p1ref)
             assert p1.x == intval
@@ -87,13 +89,13 @@
         p2 = self.malloc(S)
         p2.x = 84
         p2ref = lltype.cast_opaque_ptr(llmemory.GCREF, p2)
-        r2 = lltype.malloc(PYOBJ_HDR, flavor='raw')
-        r2.ob_refcnt = 1
-        r2.ob_pypy_link = 0
+        r2 = lltype.malloc(PyObject.TO, flavor='raw')
+        r2.c_ob_refcnt = 1
+        r2.c_ob_pypy_link = 0
         r2addr = llmemory.cast_ptr_to_adr(r2)
         # p2 and r2 are not linked
-        assert r1.ob_pypy_link != 0
-        assert r2.ob_pypy_link == 0
+        assert r1.c_ob_pypy_link != 0
+        assert r2.c_ob_pypy_link == 0
         assert self.gc.rawrefcount_from_obj(p1ref) == r1addr
         assert self.gc.rawrefcount_from_obj(p2ref) == llmemory.NULL
         assert self.gc.rawrefcount_to_obj(r1addr) == p1ref
@@ -106,16 +108,16 @@
         p1, p1ref, r1, r1addr, check_alive = (
             self._rawrefcount_pair(42, is_light=True, create_old=old))
         check_alive(0)
-        r1.ob_refcnt += 1
+        r1.c_ob_refcnt += 1
         self._collect(major=False)
         check_alive(+1)
         self._collect(major=True)
         check_alive(+1)
-        r1.ob_refcnt -= 1
+        r1.c_ob_refcnt -= 1
         self._collect(major=False)
         p1 = check_alive(0)
         self._collect(major=True)
-        py.test.raises(RuntimeError, "r1.ob_refcnt")    # dead
+        py.test.raises(RuntimeError, "r1.c_ob_refcnt")    # dead
         py.test.raises(RuntimeError, "p1.x")            # dead
         self.gc.check_no_more_rawrefcount_state()
         assert self.trigger == []
@@ -129,7 +131,7 @@
         if old:
             check_alive(0)
             self._collect(major=True)
-        py.test.raises(RuntimeError, "r1.ob_refcnt")    # dead
+        py.test.raises(RuntimeError, "r1.c_ob_refcnt")    # dead
         py.test.raises(RuntimeError, "p1.x")            # dead
         self.gc.check_no_more_rawrefcount_state()
 
@@ -147,7 +149,7 @@
         check_alive(0)
         assert p1.x == 42
         self._collect(major=True)
-        py.test.raises(RuntimeError, "r1.ob_refcnt")    # dead
+        py.test.raises(RuntimeError, "r1.c_ob_refcnt")    # dead
         py.test.raises(RuntimeError, "p1.x")            # dead
         self.gc.check_no_more_rawrefcount_state()
 
@@ -164,18 +166,18 @@
         p1, p1ref, r1, r1addr, check_alive = (
             self._rawrefcount_pair(42, is_light=False, create_old=old))
         check_alive(0)
-        r1.ob_refcnt += 1
+        r1.c_ob_refcnt += 1
         self._collect(major=False)
         check_alive(+1)
         self._collect(major=True)
         check_alive(+1)
-        r1.ob_refcnt -= 1
+        r1.c_ob_refcnt -= 1
         self._collect(major=False)
         p1 = check_alive(0)
         self._collect(major=True, expected_trigger=1)
         py.test.raises(RuntimeError, "p1.x")            # dead
-        assert r1.ob_refcnt == 1       # in the pending list
-        assert r1.ob_pypy_link == 0
+        assert r1.c_ob_refcnt == 1       # in the pending list
+        assert r1.c_ob_pypy_link == 0
         assert self.gc.rawrefcount_next_dead() == r1addr
         assert self.gc.rawrefcount_next_dead() == llmemory.NULL
         assert self.gc.rawrefcount_next_dead() == llmemory.NULL
@@ -197,8 +199,8 @@
         assert p1.x == 42
         self._collect(major=True, expected_trigger=1)
         py.test.raises(RuntimeError, "p1.x")            # dead
-        assert r1.ob_refcnt == 1
-        assert r1.ob_pypy_link == 0
+        assert r1.c_ob_refcnt == 1
+        assert r1.c_ob_pypy_link == 0
         assert self.gc.rawrefcount_next_dead() == r1addr
         self.gc.check_no_more_rawrefcount_state()
         lltype.free(r1, flavor='raw')
@@ -214,8 +216,8 @@
         else:
             self._collect(major=False, expected_trigger=1)
         py.test.raises(RuntimeError, "p1.x")            # dead
-        assert r1.ob_refcnt == 1
-        assert r1.ob_pypy_link == 0
+        assert r1.c_ob_refcnt == 1
+        assert r1.c_ob_pypy_link == 0
         assert self.gc.rawrefcount_next_dead() == r1addr
         self.gc.check_no_more_rawrefcount_state()
         lltype.free(r1, flavor='raw')
@@ -232,10 +234,10 @@
         p1, p1ref, r1, r1addr, check_alive = (
             self._rawrefcount_pair(42, is_pyobj=True, force_external=external))
         check_alive(0)
-        r1.ob_refcnt += 1            # the pyobject is kept alive
+        r1.c_ob_refcnt += 1            # the pyobject is kept alive
         self._collect(major=False)
-        assert r1.ob_refcnt == 1     # refcnt dropped to 1
-        assert r1.ob_pypy_link == 0  # detached
+        assert r1.c_ob_refcnt == 1     # refcnt dropped to 1
+        assert r1.c_ob_pypy_link == 0  # detached
         self.gc.check_no_more_rawrefcount_state()
         lltype.free(r1, flavor='raw')
 
@@ -252,8 +254,8 @@
             self._collect(major=True, expected_trigger=1)
         else:
             self._collect(major=False, expected_trigger=1)
-        assert r1.ob_refcnt == 1     # refcnt 1, in the pending list
-        assert r1.ob_pypy_link == 0  # detached
+        assert r1.c_ob_refcnt == 1     # refcnt 1, in the pending list
+        assert r1.c_ob_pypy_link == 0  # detached
         assert self.gc.rawrefcount_next_dead() == r1addr
         self.gc.check_no_more_rawrefcount_state()
         lltype.free(r1, flavor='raw')
@@ -277,8 +279,8 @@
         assert self.trigger == []
         self._collect(major=True, expected_trigger=1)
         py.test.raises(RuntimeError, "p1.x")            # dead
-        assert r1.ob_refcnt == 1
-        assert r1.ob_pypy_link == 0
+        assert r1.c_ob_refcnt == 1
+        assert r1.c_ob_pypy_link == 0
         assert self.gc.rawrefcount_next_dead() == r1addr
         self.gc.check_no_more_rawrefcount_state()
         lltype.free(r1, flavor='raw')
@@ -289,3 +291,10 @@
         check_alive(0)
         self._collect(major=True)
         check_alive(0)
+
+    def test_cycle(self):
+        p1, p1ref, r1, r1addr, check_alive = (
+            self._rawrefcount_pair(42, is_pyobj=True))
+        self.gc.rawrefcount_buffer_pyobj(r1addr)
+        self.gc.rrc_collect_cycles()
+        lltype.free(r1, flavor='raw')
diff --git a/rpython/memory/gctransform/framework.py b/rpython/memory/gctransform/framework.py
--- a/rpython/memory/gctransform/framework.py
+++ b/rpython/memory/gctransform/framework.py
@@ -482,6 +482,10 @@
                 GCClass.rawrefcount_mark_deallocating,
                 [s_gc, s_gcref, SomeAddress()],
                 annmodel.s_None)
+            self.rawrefcount_buffer_pyobj = getfn(
+                GCClass.rawrefcount_buffer_pyobj,
+                [s_gc, SomeAddress()],
+                annmodel.s_None)
             self.rawrefcount_from_obj_ptr = getfn(
                 GCClass.rawrefcount_from_obj, [s_gc, s_gcref], SomeAddress(),
                 inline = True)
@@ -1292,6 +1296,13 @@
                   [self.rawrefcount_mark_deallocating, self.c_const_gc,
                    v_gcobj, v_pyobject])
 
+    def gct_gc_rawrefcount_buffer_pyobj(self, hop):
+        [v_pyobject] = hop.spaceop.args
+        assert v_pyobject.concretetype == llmemory.Address
+        hop.genop("direct_call",
+                  [self.rawrefcount_buffer_pyobj, self.c_const_gc,
+                   v_pyobject])
+
     def gct_gc_rawrefcount_from_obj(self, hop):
         [v_gcobj] = hop.spaceop.args
         assert v_gcobj.concretetype == llmemory.GCREF
diff --git a/rpython/rlib/rawrefcount.py b/rpython/rlib/rawrefcount.py
--- a/rpython/rlib/rawrefcount.py
+++ b/rpython/rlib/rawrefcount.py
@@ -9,7 +9,8 @@
 from rpython.rlib.objectmodel import we_are_translated, specialize, not_rpython
 from rpython.rtyper.extregistry import ExtRegistryEntry
 from rpython.translator.tool.cbuild import ExternalCompilationInfo
-from rpython.rlib import rgc
+from rpython.rlib import rgc, objectmodel
+from pypy.interpreter.baseobjspace import W_Root
 
 
 MAX_BIT = int(math.log(sys.maxint, 2))
@@ -33,6 +34,7 @@
 REFCNT_CLR_GREEN = 4 << REFCNT_CLR_OFFS   # Acyclic
 REFCNT_CLR_RED = 5 << REFCNT_CLR_OFFS     # Cand cycle undergoing SIGMA-comp.
 REFCNT_CLR_ORANGE = 6 << REFCNT_CLR_OFFS  # Cand cycle awaiting epoch boundary
+REFCNT_CLR_MASK = 7 << REFCNT_CLR_OFFS
 
 # Cyclic reference count with overflow bit
 REFCNT_CRC_OVERFLOW = 1 << REFCNT_CRC_OFFS + REFCNT_BITS
@@ -46,6 +48,8 @@
 
 RAWREFCOUNT_DEALLOC_TRIGGER = lltype.Ptr(lltype.FuncType([], lltype.Void))
 
+W_MARKER_DEALLOCATING = W_Root()
+
 
 def _build_pypy_link(p):
     res = len(_adr2pypy)
@@ -70,21 +74,41 @@
 
 _refcount_overflow = dict()
 
+def incref(pyobj):
+    if pyobj.c_ob_refcnt & REFCNT_OVERFLOW == 0:
+        pyobj.c_ob_refcnt += 1
+    else:
+        if pyobj.c_ob_refcnt & REFCNT_MASK == REFCNT_OVERFLOW:
+            pyobj.c_ob_refcnt += 1
+            overflow_new(pyobj)
+        else:
+            overflow_add(pyobj)
+
+def decref(pyobj):
+    if pyobj.c_ob_refcnt & REFCNT_OVERFLOW == 0:
+        pyobj.c_ob_refcnt -= 1
+    else:
+        if pyobj.c_ob_refcnt & REFCNT_MASK == REFCNT_OVERFLOW:
+            pyobj.c_ob_refcnt -= 1
+        elif overflow_sub(pyobj):
+            pyobj.c_ob_refcnt -= 1
+
 # TODO: if object moves, address changes!
 def overflow_new(obj):
-    _refcount_overflow[id(obj)] = 0
+    _refcount_overflow[objectmodel.current_object_addr_as_int(obj)] = 0
 def overflow_add(obj):
-    _refcount_overflow[id(obj)] += 1
+    _refcount_overflow[objectmodel.current_object_addr_as_int(obj)] += 1
 def overflow_sub(obj):
-    c = _refcount_overflow[id(obj)]
+    addr = objectmodel.current_object_addr_as_int(obj)
+    c = _refcount_overflow[addr]
     if c > 0:
-        _refcount_overflow[id(obj)] = c - 1
+        _refcount_overflow[addr] = c - 1
         return False
     else:
-        _refcount_overflow.pop(id(obj))
+        _refcount_overflow.pop(addr)
         return True
 def overflow_get(obj):
-    return _refcount_overflow[id(obj)]
+    return _refcount_overflow[objectmodel.current_object_addr_as_int(obj)]
 
 # TODO: _cyclic_refcount_overflow = dict()
 
@@ -136,6 +160,10 @@
     ob.c_ob_pypy_link = _build_pypy_link(marker)
 
 @not_rpython
+def buffer_pyobj(ob):
+    pass  # TODO: implement?
+
+ at not_rpython
 def from_obj(OB_PTR_TYPE, p):
     ob = _pypy2ob.get(p)
     if ob is None:
@@ -321,6 +349,19 @@
 
 
 class Entry(ExtRegistryEntry):
+    _about_ = buffer_pyobj
+
+    def compute_result_annotation(self, s_ob):
+        pass
+
+    def specialize_call(self, hop):
+        name = 'gc_rawrefcount_buffer_pyobj'
+        hop.exception_cannot_occur()
+        v_ob = hop.inputarg(hop.args_r[0], arg=0)
+        hop.genop(name, [_unspec_ob(hop, v_ob)])
+
+
+class Entry(ExtRegistryEntry):
     _about_ = from_obj
 
     def compute_result_annotation(self, s_OB_PTR_TYPE, s_p):
diff --git a/rpython/rlib/rgc.py b/rpython/rlib/rgc.py
--- a/rpython/rlib/rgc.py
+++ b/rpython/rlib/rgc.py
@@ -522,8 +522,8 @@
         translator = hop.rtyper.annotator.translator
         fq = hop.args_s[0].const
         graph = translator._graphof(fq.finalizer_trigger.im_func)
-        #InstanceRepr.check_graph_of_del_does_not_call_too_much(hop.rtyper,
-        #                                                       graph)
+        InstanceRepr.check_graph_of_del_does_not_call_too_much(hop.rtyper,
+                                                               graph)
         hop.exception_cannot_occur()
         return hop.inputconst(lltype.Signed, hop.s_result.const)
 
diff --git a/rpython/rtyper/llinterp.py b/rpython/rtyper/llinterp.py
--- a/rpython/rtyper/llinterp.py
+++ b/rpython/rtyper/llinterp.py
@@ -966,6 +966,9 @@
     def op_gc_rawrefcount_mark_deallocating(self, *args):
         raise NotImplementedError("gc_rawrefcount_mark_deallocating")
 
+    def op_gc_rawrefcount_buffer_pyobj(self, *args):
+        raise NotImplementedError("gc_rawrefcount_buffer_pyobj")
+
     def op_gc_rawrefcount_next_dead(self, *args):
         raise NotImplementedError("gc_rawrefcount_next_dead")
 
diff --git a/rpython/rtyper/lltypesystem/lloperation.py b/rpython/rtyper/lltypesystem/lloperation.py
--- a/rpython/rtyper/lltypesystem/lloperation.py
+++ b/rpython/rtyper/lltypesystem/lloperation.py
@@ -492,6 +492,7 @@
     'gc_rawrefcount_create_link_pypy':  LLOp(),
     'gc_rawrefcount_create_link_pyobj': LLOp(),
     'gc_rawrefcount_mark_deallocating': LLOp(),
+    'gc_rawrefcount_buffer_pyobj': LLOp(),
     'gc_rawrefcount_from_obj':          LLOp(sideeffects=False),
     'gc_rawrefcount_to_obj':            LLOp(sideeffects=False),
     'gc_rawrefcount_next_dead':         LLOp(),
diff --git a/rpython/rtyper/lltypesystem/lltype.py b/rpython/rtyper/lltypesystem/lltype.py
--- a/rpython/rtyper/lltypesystem/lltype.py
+++ b/rpython/rtyper/lltypesystem/lltype.py
@@ -564,7 +564,7 @@
     def _container_example(self):
         def ex(*args):
             return self.RESULT._defl()
-        return _func(self, _callable=ex)
+        return _func(self, {'_callable': ex})
 
     def _trueargs(self):
         return [arg for arg in self.ARGS if arg is not Void]
@@ -2094,7 +2094,7 @@
 
 
 class _func(_container):
-    def __init__(self, TYPE, **attrs):
+    def __init__(self, TYPE, attrs):
         attrs.setdefault('_TYPE', TYPE)
         attrs.setdefault('_name', '?')
         attrs.setdefault('_callable', None)
@@ -2303,7 +2303,8 @@
         hash(tuple(attrs.items()))
     except TypeError:
         raise TypeError("'%r' must be hashable"%attrs)
-    o = _func(TYPE, _name=name, **attrs)
+    attrs['_name'] = name
+    o = _func(TYPE, attrs)
     return _ptr(Ptr(TYPE), o)
 
 def _getconcretetype(v):
diff --git a/rpython/rtyper/rclass.py b/rpython/rtyper/rclass.py
--- a/rpython/rtyper/rclass.py
+++ b/rpython/rtyper/rclass.py
@@ -585,8 +585,8 @@
                 assert len(s_func.descriptions) == 1
                 funcdesc, = s_func.descriptions
                 graph = funcdesc.getuniquegraph()
-                #self.check_graph_of_del_does_not_call_too_much(self.rtyper,
-                #                                               graph)
+                self.check_graph_of_del_does_not_call_too_much(self.rtyper,
+                                                               graph)
                 FUNCTYPE = FuncType([Ptr(source_repr.object_type)], Void)
                 destrptr = functionptr(FUNCTYPE, graph.name,
                                        graph=graph,

From pypy.commits at gmail.com  Wed Aug 30 08:18:36 2017
From: pypy.commits at gmail.com (Dodan)
Date: Wed, 30 Aug 2017 05:18:36 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Fixed code formatting
 issues. Thanks rlamy!
Message-ID: <59a6ad1c.02d71c0a.c8276.a93d@mx.google.com>

Author: Dodan Mihai <mihai.dodan at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92282:7d4e4f3183e5
Date: 2017-08-30 15:17 +0300
http://bitbucket.org/pypy/pypy/changeset/7d4e4f3183e5/

Log:	Fixed code formatting issues. Thanks rlamy!

diff --git a/pypy/module/_socket/interp_socket.py b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -285,8 +285,6 @@
         except SocketError as e:
             raise converted_error(space, e)
 
-
-
     def close_w(self, space):
         """close()
 
@@ -449,10 +447,9 @@
         return space.newtuple([space.newbytes(data), w_addr])
 
     @unwrap_spec(message_size=int, ancbufsize=int, flags=int)
-    def recvmsg_w(self,space,message_size, ancbufsize = 0, flags = 0):
+    def recvmsg_w(self, space, message_size, ancbufsize=0, flags=0):
         """
-        recvfrom(message_size[, ancbufsize[, flags]]) -> (message, ancillary, flags, address)
-        recvmsg(message_size, [ancbufsize,[flags]]) -> (message, ancillary, flags, address)
+        recvmsg(message_size[, ancbufsize[, flags]]) -> (message, ancillary, flags, address)
         Receive normal data (up to bufsize bytes) and ancillary data from the socket.
         The ancbufsize argument sets the size in bytes of the internal buffer used to receive the ancillary data;
         it defaults to 0, meaning that no ancillary data will be received.

From pypy.commits at gmail.com  Wed Aug 30 14:26:05 2017
From: pypy.commits at gmail.com (stevie_92)
Date: Wed, 30 Aug 2017 11:26:05 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-gc-trialdeletion: Added simple tests for
 cycle detection
Message-ID: <59a7033d.5496df0a.d406b.5fa9@mx.google.com>

Author: Stefan Beyer <home at sbeyer.at>
Branch: cpyext-gc-trialdeletion
Changeset: r92283:f1659c93bbd0
Date: 2017-08-30 20:25 +0200
http://bitbucket.org/pypy/pypy/changeset/f1659c93bbd0/

Log:	Added simple tests for cycle detection

diff --git a/rpython/memory/gc/test/test_rawrefcount.py b/rpython/memory/gc/test/test_rawrefcount.py
--- a/rpython/memory/gc/test/test_rawrefcount.py
+++ b/rpython/memory/gc/test/test_rawrefcount.py
@@ -4,7 +4,9 @@
 from rpython.memory.gc.test.test_direct import BaseDirectGCTest
 from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
 from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
-from pypy.module.cpyext.api import PyObject, PyTypeObject
+from pypy.module.cpyext.api import (PyObject, PyTypeObject, PyTypeObjectPtr,
+                                    PyObjectFields, cpython_struct)
+from pypy.module.cpyext.complexobject import PyComplexObject
 
 PYOBJ_HDR = IncrementalMiniMarkGC.PYOBJ_HDR
 PYOBJ_HDR_PTR = IncrementalMiniMarkGC.PYOBJ_HDR_PTR
@@ -83,6 +85,49 @@
             return p1
         return p1, p1ref, r1, r1addr, check_alive
 
+    def _rawrefcount_cycle_obj(self):
+        from pypy.module.cpyext.typeobjectdefs import visitproc, traverseproc
+        from rpython.rtyper.lltypesystem import rffi
+        from rpython.rtyper.annlowlevel import llhelper
+        from rpython.rlib.rawrefcount import (REFCNT_CLR_PURPLE)
+        from rpython.rtyper.tool import rffi_platform
+
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+
+        # construct test type
+        TEST_P = lltype.Ptr(lltype.ForwardReference())
+        TEST_P.TO.become(lltype.Struct('test',
+                                       ('base', PyObject.TO),
+                                       ('next', TEST_P),
+                                       ('value', lltype.Signed)))
+
+        def test_tp_traverse(obj, visit, args):
+            from pypy.module.cpyext.api import generic_cpy_call
+            test = rffi.cast(TEST_P, obj)
+            vret = 0
+            if test.next is not None:
+                next = rffi.cast(PyObject, test.next)
+                vret = visit(next, args)
+                if vret != 0:
+                    return vret
+            return vret
+
+        TRAVERSE_FUNCTYPE = rffi.CCallback([PyObject, visitproc, rffi.VOIDP],
+                                           rffi.INT_real)
+        func_ptr = llhelper(TRAVERSE_FUNCTYPE, test_tp_traverse)
+        rffi_func_ptr = rffi.cast(traverseproc, func_ptr)
+        t1 = lltype.malloc(PyTypeObject, flavor='raw', immortal=True)
+        t1.c_tp_traverse = rffi_func_ptr
+
+        # initialize object
+        r1 = lltype.malloc(TEST_P.TO, flavor='raw', immortal=True)
+        r1.base.c_ob_refcnt = 1 | REFCNT_CLR_PURPLE
+        r1.base.c_ob_pypy_link = 0
+        r1.base.c_ob_type = t1
+        r1addr = llmemory.cast_ptr_to_adr(r1)
+
+        return r1, r1addr
+
     def test_rawrefcount_objects_basic(self, old=False):
         p1, p1ref, r1, r1addr, check_alive = (
             self._rawrefcount_pair(42, is_light=True, create_old=old))
@@ -292,9 +337,17 @@
         self._collect(major=True)
         check_alive(0)
 
-    def test_cycle(self):
-        p1, p1ref, r1, r1addr, check_alive = (
-            self._rawrefcount_pair(42, is_pyobj=True))
+    def test_cycle_self_reference_free(self):
+        r1, r1addr = self._rawrefcount_cycle_obj()
+        r1.next = r1
         self.gc.rawrefcount_buffer_pyobj(r1addr)
         self.gc.rrc_collect_cycles()
-        lltype.free(r1, flavor='raw')
+        assert r1.base.c_ob_refcnt == 0
+
+    def test_cycle_self_reference_not_free(self):
+        r1, r1addr = self._rawrefcount_cycle_obj()
+        r1.base.c_ob_refcnt += 1
+        r1.next = r1
+        self.gc.rawrefcount_buffer_pyobj(r1addr)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt == 2

From pypy.commits at gmail.com  Thu Aug 31 03:16:29 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 31 Aug 2017 00:16:29 -0700 (PDT)
Subject: [pypy-commit] cffi default: Expand the docs about
 wchar_t/char16_t/char32_t
Message-ID: <59a7b7cd.90c5df0a.63f4f.3f1f@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3005:4a39df6e9df5
Date: 2017-08-31 09:16 +0200
http://bitbucket.org/cffi/cffi/changeset/4a39df6e9df5/

Log:	Expand the docs about wchar_t/char16_t/char32_t

diff --git a/doc/source/ref.rst b/doc/source/ref.rst
--- a/doc/source/ref.rst
+++ b/doc/source/ref.rst
@@ -796,20 +796,8 @@
 
 `[8]` ``wchar_t``, ``char16_t`` and ``char32_t``
 
-   The ``wchar_t`` type has the same signedness as the underlying
-   platform's.  For example, on Linux, it is a signed 32-bit integer.
-   However, the types ``char16_t`` and ``char32_t`` (*new in version
-   1.11*) are always unsigned.  **Warning:** for now, if you use
-   ``char16_t`` and ``char32_t`` with ``cdef()`` and ``set_source()``,
-   you have to make sure yourself that the types are declared by the C
-   source you provide to ``set_source()``.  They would be declared if
-   you ``#include`` a library that explicitly uses them, for example,
-   or when using C++11.  Otherwise, you need ``#include <uchar.h>`` on
-   Linux, or more generally something like ``typedef uint_least16_t
-   char16_t;``.  This is not done automatically by CFFI because
-   ``uchar.h`` is not standard across platforms, and writing a
-   ``typedef`` like above would crash if the type happens to be
-   already defined.
+   See `Unicode character types`_ below.
+
 
 .. _file:
 
@@ -842,3 +830,66 @@
 The special support for ``FILE *`` is anyway implemented in a similar manner
 on CPython 3.x and on PyPy, because these Python implementations' files are
 not natively based on ``FILE *``.  Doing it explicity offers more control.
+
+
+.. _unichar:
+
+Unicode character types
++++++++++++++++++++++++
+
+The ``wchar_t`` type has the same signedness as the underlying
+platform's.  For example, on Linux, it is a signed 32-bit integer.
+However, the types ``char16_t`` and ``char32_t`` (*new in version 1.11*)
+are always unsigned.
+
+Note that CFFI assumes that these types are meant to contain UTF-16 or
+UTF-32 characters in the native endianness.  More precisely:
+
+* ``char32_t`` is assumed to contain UTF-32, or UCS4, which is just the
+  unicode codepoint;
+
+* ``char16_t`` is assumed to contain UTF-16, i.e. UCS2 plus surrogates;
+
+* ``wchar_t`` is assumed to contain either UTF-32 or UTF-16 based on its
+  actual platform-defined size of 4 or 2 bytes.
+
+Whether this assumption is true or not is unspecified by the C language.
+In theory, the C library you are interfacing with could use one of these
+types with a different meaning.  You would then need to handle it
+yourself---for example, by using ``uint32_t`` instead of ``char32_t`` in
+the ``cdef()``, and building the expected arrays of ``uint32_t``
+manually.
+
+Python itself can be compiled with ``sys.maxunicode == 65535`` or
+``sys.maxunicode == 1114111`` (Python >= 3.3 is always 1114111).  This
+changes the handling of surrogates (which are pairs of 16-bit
+"characters" which actually stand for a single codepoint whose value is
+greater than 65535).  If your Python is ``sys.maxunicode == 1114111``,
+then it can store arbitrary unicode codepoints; surrogates are
+automatically inserted when converting from Python unicodes to UTF-16,
+and automatically removed when converting back.   On the other hand, if
+your Python is ``sys.maxunicode == 65535``, then it is the other way
+around: surrogates are removed when converting from Python unicodes
+to UTF-32, and added when converting back.  In other words, surrogate
+conversion is done only when there is a size mismatch.
+
+Note that Python's internal representations is not specified.  For
+example, on CPython >= 3.3, it will use 1- or 2- or 4-bytes arrays
+depending on what the string actually contains.  With CFFI, when you
+pass a Python byte string to a C function expecting a ``char*``, then
+we pass directly a pointer to the existing data without needing a
+temporary buffer; however, the same cannot cleanly be done with
+*unicode* string arguments and the ``wchar_t*`` / ``char16_t*`` /
+``char32_t*`` types, because of the changing internal
+representation.  As a result, and for consistency, CFFI always allocates
+a temporary buffer for unicode strings.
+
+**Warning:** for now, if you use ``char16_t`` and ``char32_t`` with
+``set_source()``, you have to make sure yourself that the types are
+declared by the C source you provide to ``set_source()``.  They would be
+declared if you ``#include`` a library that explicitly uses them, for
+example, or when using C++11.  Otherwise, you need ``#include
+<uchar.h>`` on Linux, or more generally something like ``typedef
+uint16_t char16_t;``.  This is not done automatically by CFFI because
+``uchar.h`` is not standard across platforms, and writing a ``typedef``
+like above would crash if the type happens to be already defined.
diff --git a/doc/source/using.rst b/doc/source/using.rst
--- a/doc/source/using.rst
+++ b/doc/source/using.rst
@@ -206,6 +206,9 @@
 from a unicode string,
 and calling ``ffi.string()`` on the cdata object returns the current unicode
 string stored in the source array (adding surrogates if necessary).
+See the `Unicode character types`__ section for more details.
+
+__: ref.html#unichar
 
 Note that unlike Python lists or tuples, but like C, you *cannot* index in
 a C array from the end using negative numbers.

From pypy.commits at gmail.com  Thu Aug 31 05:40:10 2017
From: pypy.commits at gmail.com (stevie_92)
Date: Thu, 31 Aug 2017 02:40:10 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-gc-trialdeletion: Added tests
Message-ID: <59a7d97a.052f1c0a.49af8.0de0@mx.google.com>

Author: Stefan Beyer <home at sbeyer.at>
Branch: cpyext-gc-trialdeletion
Changeset: r92284:a478bda34d52
Date: 2017-08-31 11:38 +0200
http://bitbucket.org/pypy/pypy/changeset/a478bda34d52/

Log:	Added tests Fixed bug in generic_cpy_call if called recursively
	Fixed bug in cycle detection if object is buffered twice

diff --git a/pypy/module/cpyext/api.py b/pypy/module/cpyext/api.py
--- a/pypy/module/cpyext/api.py
+++ b/pypy/module/cpyext/api.py
@@ -200,10 +200,11 @@
 # executes.  In non-cpyext-related code, it will thus always be 0.
 #
 # **make_generic_cpy_call():** RPython to C, with the GIL held.  Before
-# the call, must assert that the global variable is 0 and set the
-# current thread identifier into the global variable.  After the call,
-# assert that the global variable still contains the current thread id,
-# and reset it to 0.
+# the call, must assert that the global variable is 0 or the current
+# thread identifier (recursive call) and set the current thread identifier
+# into the global variable.  After the call, assert that the global variable
+# still contains the current thread id, and reset it to the value it held
+# before the call.
 #
 # **make_wrapper():** C to RPython; by default assume that the GIL is
 # held, but accepts gil="acquire", "release", "around",
@@ -1598,7 +1599,8 @@
 
         # see "Handling of the GIL" above
         tid = rthread.get_ident()
-        assert cpyext_glob_tid_ptr[0] == 0
+        tid_before = cpyext_glob_tid_ptr[0]
+        assert tid_before == 0 or tid_before == tid
         cpyext_glob_tid_ptr[0] = tid
 
         try:
@@ -1606,7 +1608,7 @@
             result = call_external_function(func, *boxed_args)
         finally:
             assert cpyext_glob_tid_ptr[0] == tid
-            cpyext_glob_tid_ptr[0] = 0
+            cpyext_glob_tid_ptr[0] = tid_before
             keepalive_until_here(*keepalives)
 
         if is_PyObject(RESULT_TYPE):
diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py
--- a/rpython/memory/gc/incminimark.py
+++ b/rpython/memory/gc/incminimark.py
@@ -3205,9 +3205,7 @@
         from rpython.rlib.rawrefcount import (REFCNT_CYCLE_BUFFERED,
                                               REFCNT_CLR_MASK,
                                               REFCNT_CLR_PURPLE,
-                                              REFCNT_MASK,
-                                              W_MARKER_DEALLOCATING,
-                                              mark_deallocating)
+                                              REFCNT_MASK)
         obj = self._pyobj(pyobject)
         rc = obj.c_ob_refcnt
         debug_print("_rrc_cycle_mark_roots", obj)
@@ -3218,8 +3216,8 @@
             obj.c_ob_refcnt = rc & ~REFCNT_CYCLE_BUFFERED
             self.rrc_buffered.remove(pyobject)
             if rc & REFCNT_MASK == 0:
-                mark_deallocating(W_MARKER_DEALLOCATING, obj)
-                generic_cpy_call(True, obj.c_ob_type.c_tp_dealloc, obj)
+                if obj.c_ob_type.c_tp_dealloc:
+                    generic_cpy_call(True, obj.c_ob_type.c_tp_dealloc, obj)
 
     def _rrc_cycle_scan_roots(self, pyobject, ignore):
         obj = self._pyobj(pyobject)
diff --git a/rpython/memory/gc/test/test_rawrefcount.py b/rpython/memory/gc/test/test_rawrefcount.py
--- a/rpython/memory/gc/test/test_rawrefcount.py
+++ b/rpython/memory/gc/test/test_rawrefcount.py
@@ -2,11 +2,15 @@
 from rpython.rtyper.lltypesystem import lltype, llmemory
 from rpython.memory.gc.incminimark import IncrementalMiniMarkGC
 from rpython.memory.gc.test.test_direct import BaseDirectGCTest
-from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY
-from rpython.rlib.rawrefcount import REFCNT_FROM_PYPY_LIGHT
+from rpython.rlib.rawrefcount import (REFCNT_FROM_PYPY, REFCNT_FROM_PYPY_LIGHT,
+                                      REFCNT_MASK)
 from pypy.module.cpyext.api import (PyObject, PyTypeObject, PyTypeObjectPtr,
                                     PyObjectFields, cpython_struct)
 from pypy.module.cpyext.complexobject import PyComplexObject
+from rpython.rtyper.lltypesystem import rffi
+from pypy.module.cpyext.typeobjectdefs import visitproc, traverseproc
+from rpython.rtyper.annlowlevel import llhelper
+from rpython.rtyper.tool import rffi_platform
 
 PYOBJ_HDR = IncrementalMiniMarkGC.PYOBJ_HDR
 PYOBJ_HDR_PTR = IncrementalMiniMarkGC.PYOBJ_HDR_PTR
@@ -17,6 +21,17 @@
                          ('prev', lltype.Ptr(S)),
                          ('next', lltype.Ptr(S))))
 
+T = lltype.Ptr(lltype.ForwardReference())
+T.TO.become(lltype.Struct('test',
+                          ('base', PyObject.TO),
+                          ('next', T),
+                          ('prev', T),
+                          ('value', lltype.Signed)))
+
+TRAVERSE_FUNCTYPE = rffi.CCallback([PyObject, visitproc, rffi.VOIDP],
+                                   rffi.INT_real)
+t1 = lltype.malloc(PyTypeObject, flavor='raw', immortal=True)
+
 
 class TestRawRefCount(BaseDirectGCTest):
     GCClass = IncrementalMiniMarkGC
@@ -86,47 +101,38 @@
         return p1, p1ref, r1, r1addr, check_alive
 
     def _rawrefcount_cycle_obj(self):
-        from pypy.module.cpyext.typeobjectdefs import visitproc, traverseproc
-        from rpython.rtyper.lltypesystem import rffi
-        from rpython.rtyper.annlowlevel import llhelper
-        from rpython.rlib.rawrefcount import (REFCNT_CLR_PURPLE)
-        from rpython.rtyper.tool import rffi_platform
-
-        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
-
-        # construct test type
-        TEST_P = lltype.Ptr(lltype.ForwardReference())
-        TEST_P.TO.become(lltype.Struct('test',
-                                       ('base', PyObject.TO),
-                                       ('next', TEST_P),
-                                       ('value', lltype.Signed)))
 
         def test_tp_traverse(obj, visit, args):
-            from pypy.module.cpyext.api import generic_cpy_call
-            test = rffi.cast(TEST_P, obj)
+            test = rffi.cast(T, obj)
             vret = 0
-            if test.next is not None:
+            if llmemory.cast_ptr_to_adr(test.next).ptr is not None:
                 next = rffi.cast(PyObject, test.next)
                 vret = visit(next, args)
                 if vret != 0:
                     return vret
+            if llmemory.cast_ptr_to_adr(test.prev).ptr is not None:
+                next = rffi.cast(PyObject, test.prev)
+                vret = visit(next, args)
+                if vret != 0:
+                    return vret
             return vret
 
-        TRAVERSE_FUNCTYPE = rffi.CCallback([PyObject, visitproc, rffi.VOIDP],
-                                           rffi.INT_real)
         func_ptr = llhelper(TRAVERSE_FUNCTYPE, test_tp_traverse)
         rffi_func_ptr = rffi.cast(traverseproc, func_ptr)
-        t1 = lltype.malloc(PyTypeObject, flavor='raw', immortal=True)
         t1.c_tp_traverse = rffi_func_ptr
 
-        # initialize object
-        r1 = lltype.malloc(TEST_P.TO, flavor='raw', immortal=True)
-        r1.base.c_ob_refcnt = 1 | REFCNT_CLR_PURPLE
+        r1 = lltype.malloc(T.TO, flavor='raw', immortal=True)
         r1.base.c_ob_pypy_link = 0
         r1.base.c_ob_type = t1
-        r1addr = llmemory.cast_ptr_to_adr(r1)
+        r1.base.c_ob_refcnt = 1
+        return r1
 
-        return r1, r1addr
+    def _rawrefcount_buffer_obj(self, obj):
+        from rpython.rlib.rawrefcount import REFCNT_CLR_MASK, REFCNT_CLR_PURPLE
+        rc = obj.base.c_ob_refcnt
+        obj.base.c_ob_refcnt = rc & ~REFCNT_CLR_MASK | REFCNT_CLR_PURPLE
+        objaddr = llmemory.cast_ptr_to_adr(obj)
+        self.gc.rawrefcount_buffer_pyobj(objaddr)
 
     def test_rawrefcount_objects_basic(self, old=False):
         p1, p1ref, r1, r1addr, check_alive = (
@@ -338,16 +344,100 @@
         check_alive(0)
 
     def test_cycle_self_reference_free(self):
-        r1, r1addr = self._rawrefcount_cycle_obj()
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
         r1.next = r1
-        self.gc.rawrefcount_buffer_pyobj(r1addr)
+        self._rawrefcount_buffer_obj(r1)
         self.gc.rrc_collect_cycles()
-        assert r1.base.c_ob_refcnt == 0
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
 
     def test_cycle_self_reference_not_free(self):
-        r1, r1addr = self._rawrefcount_cycle_obj()
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
         r1.base.c_ob_refcnt += 1
         r1.next = r1
-        self.gc.rawrefcount_buffer_pyobj(r1addr)
+        self._rawrefcount_buffer_obj(r1)
         self.gc.rrc_collect_cycles()
-        assert r1.base.c_ob_refcnt == 2
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 2
+
+    def test_simple_cycle_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.next = r1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 0
+
+    def test_simple_cycle_not_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.next = r1
+        r2.base.c_ob_refcnt += 1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 1
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 2
+
+    def test_complex_cycle_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r3 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r1.prev = r2
+        r2.base.c_ob_refcnt += 1
+        r2.next = r3
+        r3.prev = r1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r3.base.c_ob_refcnt & REFCNT_MASK == 0
+
+    def test_complex_cycle_not_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r3 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r1.prev = r2
+        r2.base.c_ob_refcnt += 1
+        r2.next = r3
+        r3.prev = r1
+        r3.base.c_ob_refcnt += 1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 1
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 2
+        assert r3.base.c_ob_refcnt & REFCNT_MASK == 2
+
+    def test_cycle_2_buffered_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.prev = r1
+        self._rawrefcount_buffer_obj(r1)
+        self._rawrefcount_buffer_obj(r2)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 0
+
+    def test_cycle_2_buffered_not_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.prev = r1
+        r1.base.c_ob_refcnt += 1
+        self._rawrefcount_buffer_obj(r1)
+        self._rawrefcount_buffer_obj(r2)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 2
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 1
+

From pypy.commits at gmail.com  Thu Aug 31 07:51:16 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 31 Aug 2017 04:51:16 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5: Merged in py3.5-sendmsg-recvmsg (pull
 request #562)
Message-ID: <59a7f834.44c11c0a.9dcf8.b445@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5
Changeset: r92286:d12c25571050
Date: 2017-08-31 11:50 +0000
http://bitbucket.org/pypy/pypy/changeset/d12c25571050/

Log:	Merged in py3.5-sendmsg-recvmsg (pull request #562)

	Implement socket.sendmsg()/.recvmsg()

diff --git a/pypy/module/_socket/__init__.py b/pypy/module/_socket/__init__.py
--- a/pypy/module/_socket/__init__.py
+++ b/pypy/module/_socket/__init__.py
@@ -34,6 +34,7 @@
             ntohs ntohl htons htonl inet_aton inet_ntoa inet_pton inet_ntop
             getaddrinfo getnameinfo
             getdefaulttimeout setdefaulttimeout
+            CMSG_SPACE CMSG_LEN
             """.split():
 
             if (name in ('inet_pton', 'inet_ntop', 'socketpair') and
diff --git a/pypy/module/_socket/interp_func.py b/pypy/module/_socket/interp_func.py
--- a/pypy/module/_socket/interp_func.py
+++ b/pypy/module/_socket/interp_func.py
@@ -327,6 +327,42 @@
             for (family, socktype, protocol, canonname, addr) in lst]
     return space.newlist(lst1)
 
+ at unwrap_spec(size=int)
+def CMSG_SPACE(space, size):
+    """
+    Socket method to determine the optimal byte size of the ancillary.
+    Recommended to be used when computing the ancillary size for recvmsg.
+    :param space:
+    :param size: an integer with the minimum size required.
+    :return: an integer with the minimum memory needed for the required size. The value is memory alligned
+    """
+    if size < 0:
+        raise oefmt(space.w_OverflowError,
+                    "CMSG_SPACE() argument out of range")
+    retval = rsocket.CMSG_SPACE(size)
+    if retval == 0:
+        raise oefmt(space.w_OverflowError,
+                    "CMSG_SPACE() argument out of range")
+    return space.newint(retval)
+
+ at unwrap_spec(len=int)
+def CMSG_LEN(space, len):
+    """
+    Socket method to determine the optimal byte size of the ancillary.
+    Recommended to be used when computing the ancillary size for recvmsg.
+    :param space:
+    :param len: an integer with the minimum size required.
+    :return: an integer with the minimum memory needed for the required size. The value is not mem alligned.
+    """
+    if len < 0:
+        raise oefmt(space.w_OverflowError,
+                    "CMSG_LEN() argument out of range")
+    retval = rsocket.CMSG_LEN(len)
+    if retval == 0:
+        raise oefmt(space.w_OverflowError,
+                    "CMSG_LEN() argument out of range")
+    return space.newint(retval)
+
 def getdefaulttimeout(space):
     """getdefaulttimeout() -> timeout
 
diff --git a/pypy/module/_socket/interp_socket.py b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -446,6 +446,52 @@
                 converted_error(space, e, eintr_retry=True)
         return space.newtuple([space.newbytes(data), w_addr])
 
+    @unwrap_spec(message_size=int, ancbufsize=int, flags=int)
+    def recvmsg_w(self, space, message_size, ancbufsize=0, flags=0):
+        """
+        recvmsg(message_size[, ancbufsize[, flags]]) -> (message, ancillary, flags, address)
+        Receive normal data (up to bufsize bytes) and ancillary data from the socket.
+        The ancbufsize argument sets the size in bytes of the internal buffer used to receive the ancillary data;
+        it defaults to 0, meaning that no ancillary data will be received.
+        Appropriate buffer sizes for ancillary data can be calculated using CMSG_SPACE() or CMSG_LEN(),
+        and items which do not fit into the buffer might be truncated or discarded.
+        The flags argument defaults to 0 and has the same meaning as for recv().
+        The ancdata item is a list of zero or more tuples (cmsg_level, cmsg_type, cmsg_data):
+        cmsg_level and cmsg_type are integers specifying the protocol level and protocol-specific type respectively,
+        and cmsg_data is a bytes object holding the associated data.
+
+        :param space: Non useable parameter. It represents the object space.
+        :param message_size: Maximum size of the message to be received
+        :param ancbufsize:  Maximum size of the ancillary data to be received
+        :param flags: Receive flag. For more details, please check the Unix manual
+        :return: a tuple consisting of the message, the ancillary data, return flag and the address.
+        """
+        if message_size < 0:
+            raise oefmt(space.w_ValueError, "negative buffer size in recvmsg()")
+        if ancbufsize < 0:
+            raise oefmt(space.w_ValueError, "invalid ancillary data buffer length")
+        while True:
+            try:
+                recvtup = self.sock.recvmsg(message_size, ancbufsize, flags)
+                w_message = space.newbytes(recvtup[0])
+                anclist = []
+                for l in recvtup[1]:
+                    tup = space.newtuple([space.newint(l[0]), space.newint(l[1]), space.newbytes(l[2])])
+                    anclist.append(tup)
+
+                w_anc = space.newlist(anclist)
+
+                w_flag = space.newint(recvtup[2])
+                if (recvtup[3] is not None):
+                    w_address = addr_as_object(recvtup[3], self.sock.fd, space)
+                else:
+                    w_address = space.w_None
+                rettup = space.newtuple([w_message, w_anc, w_flag, w_address])
+                break
+            except SocketError as e:
+                converted_error(space, e, eintr_retry=True)
+        return rettup
+
     @unwrap_spec(data='bufferstr', flags=int)
     def send_w(self, space, data, flags=0):
         """send(data[, flags]) -> count
@@ -501,6 +547,67 @@
                 converted_error(space, e, eintr_retry=True)
         return space.newint(count)
 
+    @unwrap_spec(flags=int)
+    def sendmsg_w(self, space, w_data, w_ancillary=None, flags=0 ,w_address=None):
+        """
+        sendmsg(data[,ancillary[,flags[,address]]]) -> bytes_sent
+        Send normal and ancillary data to the socket, gathering the non-ancillary data
+        from a series of buffers and concatenating it into a single message.
+        The ancdata argument specifies the ancillary data (control messages) as an iterable of zero or more tuples
+        (cmsg_level, cmsg_type, cmsg_data), where cmsg_level and cmsg_type are integers specifying the protocol level
+        and protocol-specific type respectively, and cmsg_data is a bytes-like object holding the associated data.
+        :param space: Represents the object space.
+        :param w_data: The message(s). needs to be a bytes like object
+        :param w_ancillary: needs to be a sequence object Can remain unspecified.
+        :param w_flags: needs to be an integer. Can remain unspecified.
+        :param w_address: needs to be a bytes-like object Can remain unspecified.
+        :return: Bytes sent from the message
+        """
+        # Get the flag and address from the object space
+        while True:
+            try:
+                address = None
+                if not space.is_none(w_address):
+                    address = self.addr_from_object(space, w_address)
+
+                # find data's type in the ObjectSpace and get a list of string out of it.
+                data = []
+                data_iter = space.unpackiterable(w_data)
+                for i in data_iter:
+                    data.append(space.readbuf_w(i).as_str())
+
+                # find the ancillary's type in the ObjectSpace and get a list of tuples out of it.
+                ancillary = []
+                if w_ancillary is not None:
+                    anc_iter = space.unpackiterable(w_ancillary)
+                    for w_i in anc_iter:
+                        if not space.isinstance_w(w_i, space.w_tuple):
+                            raise oefmt(space.w_TypeError, "[sendmsg() ancillary data items]() argument must be sequence")
+                        if space.len_w(w_i) == 3:
+                            intemtup = space.unpackiterable(w_i)
+                            level = space.int_w(intemtup[0])
+                            type = space.int_w(intemtup[1])
+                            cont = space.readbuf_w(intemtup[2]).as_str()
+                            tup = (level, type, cont)
+                            ancillary.append(tup)
+                        else:
+                            raise oefmt(space.w_TypeError,
+                                        "[sendmsg() ancillary data items]() argument must be sequence of length 3")
+
+                count = self.sock.sendmsg(data, ancillary, flags, address)
+                if count < 0:
+                    if (count == -1000):
+                        raise oefmt(space.w_OSError, "sending multiple control messages not supported")
+                    if (count == -1001):
+                        raise oefmt(space.w_OSError, "ancillary data item too large")
+                    if (count == -1002):
+                        raise oefmt(space.w_OSError, "too much ancillary data")
+                break
+            except SocketError as e:
+                converted_error(space, e, eintr_retry=True)
+
+        return space.newint(count)
+
     @unwrap_spec(flag=int)
     def setblocking_w(self, flag):
         """setblocking(flag)
@@ -772,7 +879,7 @@
 socketmethodnames = """
 _accept bind close connect connect_ex fileno detach
 getpeername getsockname getsockopt gettimeout listen
-recv recvfrom send sendall sendto setblocking
+recv recvfrom recvmsg send sendall sendto sendmsg setblocking
 setsockopt settimeout shutdown _reuse _drop recv_into recvfrom_into
 """.split()
 if hasattr(rsocket._c, 'WSAIoctl'):
@@ -813,6 +920,8 @@
 sendall(data[, flags]) -- send all data
 send(data[, flags]) -- send data, may not send all of it
 sendto(data[, flags], addr) -- send data to a given address
+sendmsg(messages[, ancillary[, flags[, address]]]) -- send data and ancillary payload in a packet. May specifiy flags or the address
+recvmsg(message_size,[ ancillary_size,[ flags]]) -- receive data and ancillary payload. Return a tup of message, ancdata, flags and address
 setblocking(0 | 1) -- set or clear the blocking I/O flag
 setsockopt(level, optname, value) -- set socket options
 settimeout(None | float) -- set or clear the timeout
diff --git a/rpython/rlib/_rsocket_rffi.py b/rpython/rlib/_rsocket_rffi.py
--- a/rpython/rlib/_rsocket_rffi.py
+++ b/rpython/rlib/_rsocket_rffi.py
@@ -2,6 +2,7 @@
 from rpython.rtyper.lltypesystem import lltype
 from rpython.rtyper.tool import rffi_platform as platform
 from rpython.rtyper.lltypesystem.rffi import CCHARP
+from rpython.rlib import jit
 from rpython.translator.tool.cbuild import ExternalCompilationInfo
 from rpython.translator.platform import platform as target_platform
 
@@ -190,6 +191,8 @@
 
 IPX_TYPE
 
+SCM_RIGHTS
+
 POLLIN POLLPRI POLLOUT POLLERR POLLHUP POLLNVAL
 POLLRDNORM POLLRDBAND POLLWRNORM POLLWEBAND POLLMSG
 
@@ -260,6 +263,7 @@
 sockaddr_ptr = lltype.Ptr(lltype.ForwardReference())
 addrinfo_ptr = lltype.Ptr(lltype.ForwardReference())
 
+
 # struct types
 CConfig.sockaddr = platform.Struct('struct sockaddr',
                                              [('sa_family', rffi.INT),
@@ -343,6 +347,650 @@
                                 [('ifr_ifindex', rffi.INT),
                                  ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))])
 
+# insert handler for sendmsg / recvmsg here
+if _POSIX:
+    includes = ['stddef.h',
+                'sys/socket.h',
+                'unistd.h',
+                'string.h',
+                'stdlib.h',
+                'errno.h',
+                'limits.h',
+                'stdio.h',
+                'sys/types.h',
+                'netinet/in.h',
+                'arpa/inet.h']
+    separate_module_sources = ['''
+
+        // special defines for returning from recvmsg
+        #define BAD_MSG_SIZE_GIVEN -10000
+        #define BAD_ANC_SIZE_GIVEN -10001
+        #define MAL_ANC -10002
+
+        // special defines for returning from sendmsg
+        #define MUL_MSGS_NOT_SUP -1000
+        #define ANC_DATA_TOO_LARGE -1001
+        #define ANC_DATA_TOO_LARGEX -1002
+
+        /*
+            Even though you could, theoretically, receive more than one message, IF you set the socket option,
+            CPython has hardcoded the message number to 1, and implemented the option to receive more then 1 in a
+            different socket method: recvmsg_into
+        */
+        #define MSG_IOVLEN 1 // CPython has hardcoded this as well.
+        #if INT_MAX > 0x7fffffff
+            #define SOCKLEN_T_LIMIT 0x7fffffff
+        #else
+        #define SOCKLEN_T_LIMIT INT_MAX
+        #endif
+
+        // ################################################################################################
+        // Recvmsg implementation and associated functions
+
+        // Taken from CPython. Determines the minimum memory space required for the ancillary data.
+        #ifdef CMSG_SPACE
+        static int
+        cmsg_min_space(struct msghdr *msg, struct cmsghdr *cmsgh, size_t space)
+        {
+            size_t cmsg_offset;
+            static const size_t cmsg_len_end = (offsetof(struct cmsghdr, cmsg_len) +
+                                                sizeof(cmsgh->cmsg_len));
+
+            /* Note that POSIX allows msg_controllen to be of signed type. */
+            if (cmsgh == NULL || msg->msg_control == NULL)
+                return 0;
+            /* Note that POSIX allows msg_controllen to be of a signed type. This is
+               annoying under OS X as it's unsigned there and so it triggers a
+               tautological comparison warning under Clang when compared against 0.
+               Since the check is valid on other platforms, silence the warning under
+               Clang. */
+            #ifdef __clang__
+            #pragma clang diagnostic push
+            #pragma clang diagnostic ignored "-Wtautological-compare"
+            #endif
+            #if defined(__GNUC__) && ((__GNUC__ > 4) || ((__GNUC__ == 4) && (__GNUC_MINOR__ > 5)))
+            #pragma GCC diagnostic push
+            #pragma GCC diagnostic ignored "-Wtype-limits"
+            #endif
+            if (msg->msg_controllen < 0)
+                return 0;
+            #if defined(__GNUC__) && ((__GNUC__ > 4) || ((__GNUC__ == 4) && (__GNUC_MINOR__ > 5)))
+            #pragma GCC diagnostic pop
+            #endif
+            #ifdef __clang__
+            #pragma clang diagnostic pop
+            #endif
+            if (space < cmsg_len_end)
+                space = cmsg_len_end;
+            cmsg_offset = (char *)cmsgh - (char *)msg->msg_control;
+            return (cmsg_offset <= (size_t)-1 - space &&
+                    cmsg_offset + space <= msg->msg_controllen);
+        }
+        #endif
+
+        // Taken from CPython.
+        #ifdef CMSG_LEN
+        /* If pointer CMSG_DATA(cmsgh) is in buffer msg->msg_control, set
+           *space to number of bytes following it in the buffer and return
+           true; otherwise, return false.  Assumes cmsgh, msg->msg_control and
+           msg->msg_controllen are valid. */
+        static int
+        get_cmsg_data_space(struct msghdr *msg, struct cmsghdr *cmsgh, size_t *space)
+        {
+            size_t data_offset;
+            char *data_ptr;
+
+            if ((data_ptr = (char *)CMSG_DATA(cmsgh)) == NULL)
+                return 0;
+            data_offset = data_ptr - (char *)msg->msg_control;
+            if (data_offset > msg->msg_controllen)
+                return 0;
+            *space = msg->msg_controllen - data_offset;
+            return 1;
+        }
+
+        // Taken from CPython.
+        /* If cmsgh is invalid or not contained in the buffer pointed to by
+           msg->msg_control, return -1.  If cmsgh is valid and its associated
+           data is entirely contained in the buffer, set *data_len to the
+           length of the associated data and return 0.  If only part of the
+           associated data is contained in the buffer but cmsgh is otherwise
+           valid, set *data_len to the length contained in the buffer and
+           return 1. */
+        static int
+        get_cmsg_data_len(struct msghdr *msg, struct cmsghdr *cmsgh, size_t *data_len)
+        {
+            size_t space, cmsg_data_len;
+
+            if (!cmsg_min_space(msg, cmsgh, CMSG_LEN(0)) ||
+                cmsgh->cmsg_len < CMSG_LEN(0))
+                return -1;
+            cmsg_data_len = cmsgh->cmsg_len - CMSG_LEN(0);
+            if (!get_cmsg_data_space(msg, cmsgh, &space))
+                return -1;
+            if (space >= cmsg_data_len) {
+                *data_len = cmsg_data_len;
+                return 0;
+            }
+            *data_len = space;
+            return 1;
+        }
+        #endif    /* CMSG_LEN */
+
+        /*
+            Structure meant to hold the information received after a recvmsg is performed.
+            Essentially it holds: the address, the message, the ancillary data and the return flags.
+            I use this structure for 2 main reasons:
+               - keep things ordered
+               - some of the ancillary parameters need to be int not long (rffi SignedP is actually long*),
+                 therefore I cannot use the parameters directly
+        */
+        struct recvmsg_info
+        {
+            struct sockaddr* address; // address fields
+            socklen_t addrlen;
+            int* length_of_messages; // message fields
+            char** messages;
+            int no_of_messages;
+            int size_of_ancillary; // ancillary fields
+            int* levels;
+            int* types;
+            char** file_descr;
+            int* descr_per_ancillary;
+            int retflag; // return flag field
+        };
+
+        /*
+            Wrapper function over recvmsg. Since it returns a lot of data,
+            in a structure that is hard to parse in rffi, it was implemented in C.
+            All the parameters, save the socket fd, message_size, ancillary_size
+            will be malloc'd and/or modified.
+        */
+        RPY_EXTERN
+        int recvmsg_implementation(
+                                  int socket_fd,
+                                  int message_size,
+                                  int ancillary_size,
+                                  int flags,
+                                  struct sockaddr* address,
+                                  socklen_t* addrlen,
+                                  long** length_of_messages,
+                                  char** messages,
+                                  long* no_of_messages,
+                                  long* size_of_ancillary,
+                                  long** levels,
+                                  long** types,
+                                  char** file_descr,
+                                  long** descr_per_ancillary,
+                                  long* retflag)
+
+        {
+
+            struct sockaddr* recvd_address;
+            socklen_t recvd_addrlen;
+            struct msghdr msg = {0};
+            void *controlbuf = NULL;
+            struct cmsghdr *cmsgh;
+            int cmsg_status;
+            struct iovec iov;
+            struct recvmsg_info* retinfo;
+            int error_flag;   // variable to be set in case of special errors.
+            int cmsgdatalen = 0;
+
+            // variables that are set to 1, if the message charp has been allocated
+            // and if the ancillary variables have been allocated. To be used in case of failure.
+            int iov_alloc = 0;
+            int anc_alloc = 0;
+
+            retinfo = (struct recvmsg_info*) malloc(sizeof(struct recvmsg_info));
+
+            if (ancillary_size > SOCKLEN_T_LIMIT){
+                error_flag = BAD_ANC_SIZE_GIVEN;
+                goto fail;
+            }
+
+            // Setup the messages iov struct memory
+            iov.iov_base = (char*) malloc(message_size);
+            memset(iov.iov_base, 0, message_size);
+            iov.iov_len = message_size;
+
+            // Setup the ancillary buffer memory
+            controlbuf = malloc(ancillary_size);
+
+            // Setup the recv address memory
+            recvd_addrlen = sizeof(struct sockaddr_storage);
+            recvd_address = (struct sockaddr*) malloc(recvd_addrlen);
+
+            memset(recvd_address, 0,recvd_addrlen);
+
+            // Setup the msghdr struct
+            msg.msg_name = recvd_address;
+            msg.msg_namelen = recvd_addrlen;
+            msg.msg_iov = &iov;
+            msg.msg_iovlen = MSG_IOVLEN;
+            msg.msg_control = controlbuf;
+            msg.msg_controllen = ancillary_size;
+
+            // Link my structure to the msghdr fields
+            retinfo->address = msg.msg_name;
+            retinfo->length_of_messages = (int*) malloc (MSG_IOVLEN * sizeof(int));
+            retinfo->no_of_messages = MSG_IOVLEN;
+            retinfo->messages = (char**) malloc (MSG_IOVLEN * sizeof(char*));
+            retinfo->messages[0] = msg.msg_iov->iov_base;
+
+            iov_alloc = 1;
+            ssize_t bytes_recvd = 0;
+
+            bytes_recvd = recvmsg(socket_fd, &msg, flags);
+
+            if (bytes_recvd < 0){
+                 goto fail;
+            }
+
+            retinfo->addrlen = (socklen_t) msg.msg_namelen;
+            retinfo->length_of_messages[0] = msg.msg_iov->iov_len;
+
+            // Count the ancillary items & allocate the memory
+            int anc_counter = 0;
+            for (cmsgh = ((msg.msg_controllen > 0) ? CMSG_FIRSTHDR(&msg) : NULL);
+                 cmsgh != NULL; cmsgh = CMSG_NXTHDR(&msg, cmsgh)) {
+
+                 anc_counter++;
+            }
+            retinfo->size_of_ancillary = anc_counter;
+            retinfo->file_descr = (char**) malloc (anc_counter * sizeof(char*));
+            retinfo->levels = (int*) malloc(anc_counter * sizeof(int));
+            retinfo->types = (int*) malloc(anc_counter * sizeof(int));
+            retinfo->descr_per_ancillary = (int*) malloc(anc_counter * sizeof(int));
+            anc_alloc = 1;
+
+            // Extract the ancillary items
+            int i=0;
+            for (cmsgh = ((msg.msg_controllen > 0) ? CMSG_FIRSTHDR(&msg) : NULL);
+                 cmsgh != NULL; cmsgh = CMSG_NXTHDR(&msg, cmsgh)) {
+                 size_t local_size = 0;
+                 cmsg_status = get_cmsg_data_len(&msg, cmsgh, &local_size);
+                 if (cmsg_status !=0 ){
+                    error_flag = MAL_ANC;
+                    goto err_closefds;
+                 }
+                 retinfo->file_descr[i] = (char*) malloc(local_size);
+                 memcpy(retinfo->file_descr[i], CMSG_DATA(cmsgh), local_size);
+                 retinfo->levels[i] = cmsgh->cmsg_level;
+                 retinfo->types[i] = cmsgh->cmsg_type;
+                 retinfo->descr_per_ancillary[i] =local_size;
+                 i++;
+
+            }
+            retinfo->retflag = msg.msg_flags;
+
+            // Set the parameters of address
+            memcpy(address,retinfo->address,retinfo->addrlen);
+            *addrlen = retinfo->addrlen;
+
+            // Set the parameters of message
+            no_of_messages[0] = retinfo->no_of_messages;
+            size_of_ancillary[0] = retinfo->size_of_ancillary;
+            *length_of_messages = (long*) malloc (sizeof(long) * retinfo->no_of_messages);
+            //memcpy(*length_of_messages, retinfo->length_of_messages, sizeof(int) * retinfo->no_of_messages);
+            int counter = 0;
+            for (i=0; i< retinfo->no_of_messages; i++){
+                counter += retinfo->length_of_messages[i];
+                length_of_messages[0][i] = retinfo->length_of_messages[i];
+            }
+            memset(*messages, 0, sizeof(char) * counter);
+            counter = 0;
+            for(i=0; i< retinfo->no_of_messages; i++){
+                memcpy(*messages+counter,retinfo->messages[i],retinfo->length_of_messages[i]);
+                counter += retinfo->length_of_messages[i];
+            }
+
+            // Set the parameters of ancillary
+            *levels = (long*) malloc (sizeof(long) * retinfo->size_of_ancillary);
+            *types = (long*) malloc (sizeof(long) * retinfo->size_of_ancillary);
+            *descr_per_ancillary = (long*) malloc (sizeof(long) * retinfo->size_of_ancillary);
+            counter = 0;
+            for (i=0; i < retinfo->size_of_ancillary; i++){
+                counter += retinfo->descr_per_ancillary[i];
+                // Convert the int* to long*
+                levels[0][i] = (long) retinfo->levels[i];
+                types[0][i] = (long) retinfo->types[i];
+                descr_per_ancillary[0][i] = (long) retinfo->descr_per_ancillary[i];
+            }
+            *file_descr = (char*) malloc (sizeof(char) * counter);
+            memset(*file_descr, 0, sizeof(char) * counter);
+            counter = 0;
+            for (i=0; i<retinfo->size_of_ancillary; i++){
+                memcpy(*file_descr+counter,retinfo->file_descr[i], retinfo->descr_per_ancillary[i]);
+                counter += retinfo->descr_per_ancillary[i];
+            }
+
+            // Set the retflag
+            retflag[0] = retinfo->retflag;
+
+            // Free the memory
+            free(retinfo->address);
+            free(retinfo->length_of_messages);
+            free(retinfo->levels);
+            free(retinfo->types);
+            free(retinfo->descr_per_ancillary);
+            for(i = 0; i<retinfo->no_of_messages; i++)
+                free(retinfo->messages[i]);
+            for (i = 0; i < retinfo->size_of_ancillary; i++)
+                free(retinfo->file_descr[i]);
+            free(retinfo->file_descr);
+            free(retinfo->messages);
+            free(retinfo);
+            free(controlbuf);
+
+            return bytes_recvd;
+
+        fail:
+            if (anc_alloc){
+                free(retinfo->file_descr);
+                free(retinfo->levels);
+                free(retinfo->types);
+                free(retinfo->descr_per_ancillary);
+                free(retinfo->length_of_messages);
+                free(retinfo->messages[0]);
+                free(retinfo->messages);
+                free(retinfo->address);
+                free(retinfo);
+                free(controlbuf);
+
+            }else{
+                if (iov_alloc){
+                    free(retinfo->length_of_messages);
+                    free(retinfo->messages[0]);
+                    free(retinfo->messages);
+                    free(retinfo->address);
+                    free(controlbuf);
+                    free(retinfo);
+                }
+            }
+            return error_flag;
+
+        err_closefds:
+        // Special case for UNIX sockets. In case file descriptors are received, they need to be closed.
+        // Taken from CPython
+        #ifdef SCM_RIGHTS
+            /* Close all descriptors coming from SCM_RIGHTS, so they don't leak. */
+            for (cmsgh = ((msg.msg_controllen > 0) ? CMSG_FIRSTHDR(&msg) : NULL);
+                 cmsgh != NULL; cmsgh = CMSG_NXTHDR(&msg, cmsgh)) {
+                size_t dataleng;
+                cmsg_status = get_cmsg_data_len(&msg, cmsgh, &dataleng);
+                cmsgdatalen = (int) dataleng;
+                if (cmsg_status < 0)
+                    break;
+                if (cmsgh->cmsg_level == SOL_SOCKET &&
+                    cmsgh->cmsg_type == SCM_RIGHTS) {
+                    size_t numfds;
+                    int *fdp;
+
+                    numfds = cmsgdatalen / sizeof(int);
+                    fdp = (int *)CMSG_DATA(cmsgh);
+                    while (numfds-- > 0)
+                        close(*fdp++);
+                }
+                if (cmsg_status != 0)
+                    break;
+            }
+        #endif /* SCM_RIGHTS */
+            goto fail;
+        }
+
+
+        // ################################################################################################
+        // Sendmsg implementation and associated functions
+
+        #ifdef CMSG_LEN
+        static int
+        get_CMSG_LEN(size_t length, size_t *result)
+        {
+            size_t tmp;
+
+            if (length > (SOCKLEN_T_LIMIT - CMSG_LEN(0)))
+                return 0;
+            tmp = CMSG_LEN(length);
+            if ((tmp > SOCKLEN_T_LIMIT) || (tmp < length))
+                return 0;
+            *result = tmp;
+            return 1;
+        }
+        #endif
+
+        #ifdef CMSG_SPACE
+        /* If length is in range, set *result to CMSG_SPACE(length) and return
+           true; otherwise, return false. */
+        static int
+        get_CMSG_SPACE(size_t length, size_t *result)
+        {
+            size_t tmp;
+
+            /* Use CMSG_SPACE(1) here in order to take account of the padding
+               necessary before *and* after the data. */
+            if (length > (SOCKLEN_T_LIMIT - CMSG_SPACE(1)))
+                return 0;
+            tmp = CMSG_SPACE(length);
+            if ((tmp > SOCKLEN_T_LIMIT) || (tmp < length))
+                return 0;
+            *result = tmp;
+            return 1;
+        }
+        #endif
+
+        /*
+            sendmsg_implementation is a wrapper over sendmsg of the API.
+            It was inspired from the way CPython did their implementation of this.
+            The main reason that it was written in C, is the struct msghdr,
+            which contains the ancillary data in a linked list of cmsghdr structures.
+            It was simpler to use it in C, and then push the simpler types of data via rffi.
+        */
+        RPY_EXTERN
+        int sendmsg_implementation
+                            (int socket,
+                            struct sockaddr* address,
+                            socklen_t addrlen,
+                            long* length_of_messages,
+                            char** messages,
+                            int no_of_messages,
+                            long* levels,
+                            long* types,
+                            char** file_descriptors,
+                            long* no_of_fds,
+                            int control_length,
+                            int flag
+                            )
+        {
+
+            struct msghdr        msg = {0};
+            struct cmsghdr       *cmsg;
+            void* controlbuf = NULL;
+            int retval;
+            size_t i;
+
+            // Prepare the msghdr structure for the send:
+
+            // Add the address
+            if (address != NULL) {
+                msg.msg_name = address;
+                msg.msg_namelen = addrlen;
+            }
+
+            // Add the message
+            struct iovec *iovs = NULL;
+            if (no_of_messages > 0){
+
+                iovs = (struct iovec*) malloc(no_of_messages * sizeof(struct iovec));
+                memset(iovs, 0, no_of_messages * sizeof(struct iovec));
+                msg.msg_iov = iovs;
+                msg.msg_iovlen = no_of_messages;
+
+                for (i=0; i< no_of_messages; i++){
+                    iovs[i].iov_base = messages[i];
+                    iovs[i].iov_len = length_of_messages[i];
+                }
+            }
+
+            // Add the ancillary
+            #ifndef CMSG_SPACE
+                if (control_length > 1){
+                    free(iovs);
+                    return MUL_MSGS_NOT_SUP;
+                }
+            #endif
+            if (control_length > 0){
+
+                //compute the total size of the ancillary
+                //getting the exact amount of space can be tricky and os dependent.
+                size_t total_size_of_ancillary = 0;
+                size_t space;
+                size_t controllen = 0, controllen_last = 0;
+                for (i = 0; i< control_length; i++){
+                    total_size_of_ancillary = no_of_fds[i];
+                    #ifdef CMSG_SPACE
+                        if (!get_CMSG_SPACE(total_size_of_ancillary, &space)) {
+                    #else
+                        if (!get_CMSG_LEN(total_size_of_ancillary, &space)) {
+                    #endif
+                            if (iovs != NULL)
+                                free(iovs);
+                            return ANC_DATA_TOO_LARGE;
+                        }
+                    controllen +=space;
+                    if ((controllen > SOCKLEN_T_LIMIT) || (controllen < controllen_last)) {
+                        if (iovs != NULL)
+                                free(iovs);
+                        return ANC_DATA_TOO_LARGEX;
+                    }
+                    controllen_last = controllen;
+                }
+
+                controlbuf = malloc(controllen);
+                msg.msg_control= controlbuf;
+                msg.msg_controllen = controllen;
+
+                // memset controlbuf to 0 to avoid trash in the ancillary
+                memset(controlbuf, 0, controllen);
+                cmsg = NULL;
+                for (i = 0; i< control_length; i++){
+                    cmsg = (i == 0) ? CMSG_FIRSTHDR(&msg) : CMSG_NXTHDR(&msg, cmsg);
+
+                    cmsg->cmsg_level = (int) levels[i];
+                    cmsg->cmsg_type = (int) types[i];
+                    cmsg->cmsg_len = CMSG_LEN(sizeof(char) * no_of_fds[i]);
+                    memcpy(CMSG_DATA(cmsg), file_descriptors[i], sizeof(char) * no_of_fds[i]);
+                }
+
+
+            }
+            // Add the flags
+            msg.msg_flags = flag;
+
+            // Send the data
+            retval = sendmsg(socket, &msg, flag);
+
+            // free everything that was allocated here, and we would not need in rsocket
+            if (iovs != NULL)
+                free(iovs);
+            if (controlbuf !=NULL)
+               free(controlbuf);
+
+            return retval;
+        }
+
+        // ################################################################################################
+        // Wrappers for CMSG_SPACE and CMSG_LEN
+
+        /*
+            These 2 functions are wrappers over sys/socket.h's CMSG_SPACE and CMSG_LEN.
+            They are identical to CPython's.
+        */
+        #ifdef CMSG_SPACE
+        RPY_EXTERN
+        size_t CMSG_SPACE_wrapper(size_t desired_space){
+            size_t result;
+            if (!get_CMSG_SPACE(desired_space, &result)){
+                return 0;
+            }
+            return result;
+        }
+        #endif
+
+        #ifdef CMSG_LEN
+        RPY_EXTERN
+        size_t CMSG_LEN_wrapper(size_t desired_len){
+            size_t result;
+            if (!get_CMSG_LEN(desired_len, &result)){
+                return 0;
+            }
+            return result;
+        }
+        #endif
+
+        // ################################################################################################
+        // Extra functions that I needed
+
+        /*
+           This function is used to memcpy from a char* at an offset.
+           Could not get rffi.c_memcpy to do it at an offset, so I made my own.
+        */
+        RPY_EXTERN
+        int memcpy_from_CCHARP_at_offset_and_size(char* stringfrom, char** stringto, int offset, int size){
+            *stringto = memcpy(*stringto, stringfrom + offset, size);
+            return 0;
+        }
+
+        /*
+            These functions free memory that was allocated in C (sendmsg or recvmsg) was used in rsocket and now needs cleanup
+        */
+        RPY_EXTERN
+        int free_pointer_to_signedp(int** ptrtofree){
+            free(*ptrtofree);
+            return 0;
+        }
+
+        RPY_EXTERN
+        int free_ptr_to_charp(char** ptrtofree){
+            free(*ptrtofree);
+            return 0;
+        }
+
+    ''',]
+
+    post_include_bits =[ "RPY_EXTERN "
+                         "int sendmsg_implementation(int socket, struct sockaddr* address, socklen_t addrlen, long* length_of_messages, char** messages, int no_of_messages, long* levels, long* types, char** file_descriptors, long* no_of_fds, int control_length, int flag );\n"
+                         "RPY_EXTERN "
+                         "int recvmsg_implementation(int socket_fd, int message_size, int ancillary_size, int flags, struct sockaddr* address, socklen_t* addrlen, long** length_of_messages, char** messages, long* no_of_messages, long* size_of_ancillary, long** levels, long** types, char** file_descr, long** descr_per_ancillary, long* flag);\n"
+                         "static "
+                         "int cmsg_min_space(struct msghdr *msg, struct cmsghdr *cmsgh, size_t space);\n"
+                         "static "
+                         "int get_cmsg_data_space(struct msghdr *msg, struct cmsghdr *cmsgh, size_t *space);\n"
+                         "static "
+                         "int get_cmsg_data_len(struct msghdr *msg, struct cmsghdr *cmsgh, size_t *data_len);\n"
+                         "static "
+                         "int get_CMSG_LEN(size_t length, size_t *result);\n"
+                         "static "
+                         "int get_CMSG_SPACE(size_t length, size_t *result);\n"
+                         "RPY_EXTERN "
+                         "size_t CMSG_LEN_wrapper(size_t desired_len);\n"
+                         "RPY_EXTERN "
+                         "size_t CMSG_SPACE_wrapper(size_t desired_space);\n"
+                         "RPY_EXTERN "
+                         "int memcpy_from_CCHARP_at_offset_and_size(char* stringfrom, char** stringto, int offset, int size);\n"
+                         "RPY_EXTERN "
+                         "int free_pointer_to_signedp(int** ptrtofree);\n"
+                         "RPY_EXTERN "
+                         "int free_ptr_to_charp(char** ptrtofree);\n"
+                         ]
+
+
+    compilation_info = ExternalCompilationInfo(
+                                    includes=includes,
+                                    separate_module_sources=separate_module_sources,
+                                    post_include_bits=post_include_bits,
+                               )
+
 if _WIN32:
     CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP)
     CConfig.WSANETWORKEVENTS = platform.Struct(
@@ -387,6 +1035,7 @@
 sockaddr_ptr.TO.become(cConfig.sockaddr)
 addrinfo_ptr.TO.become(cConfig.addrinfo)
 
+
 # fill in missing constants with reasonable defaults
 cConfig.NI_MAXHOST = cConfig.NI_MAXHOST or 1025
 cConfig.NI_MAXSERV = cConfig.NI_MAXSERV or 32
@@ -571,11 +1220,32 @@
 recvfrom = external('recvfrom', [socketfd_type, rffi.VOIDP, size_t,
                            rffi.INT, sockaddr_ptr, socklen_t_ptr], rffi.INT,
                     save_err=SAVE_ERR)
+recvmsg = jit.dont_look_inside(rffi.llexternal("recvmsg_implementation",
+                                               [rffi.INT, rffi.INT, rffi.INT, rffi.INT,sockaddr_ptr, socklen_t_ptr, rffi.SIGNEDPP, rffi.CCHARPP,
+                                                rffi.SIGNEDP,rffi.SIGNEDP, rffi.SIGNEDPP, rffi.SIGNEDPP, rffi.CCHARPP, rffi.SIGNEDPP, rffi.SIGNEDP],
+                                               rffi.INT, save_err=SAVE_ERR,
+                                               compilation_info=compilation_info))
+
+memcpy_from_CCHARP_at_offset = jit.dont_look_inside(rffi.llexternal("memcpy_from_CCHARP_at_offset_and_size",
+                                    [rffi.CCHARP, rffi.CCHARPP,rffi.INT,rffi.INT],rffi.INT,save_err=SAVE_ERR,compilation_info=compilation_info))
+freeccharp = jit.dont_look_inside(rffi.llexternal("free_ptr_to_charp",
+                                    [rffi.CCHARPP],rffi.INT,save_err=SAVE_ERR,compilation_info=compilation_info))
+freesignedp = jit.dont_look_inside(rffi.llexternal("free_pointer_to_signedp",
+                                    [rffi.SIGNEDPP],rffi.INT,save_err=SAVE_ERR,compilation_info=compilation_info))
+
 send = external('send', [socketfd_type, rffi.CCHARP, size_t, rffi.INT],
                        ssize_t, save_err=SAVE_ERR)
 sendto = external('sendto', [socketfd_type, rffi.VOIDP, size_t, rffi.INT,
                                     sockaddr_ptr, socklen_t], ssize_t,
                   save_err=SAVE_ERR)
+sendmsg = jit.dont_look_inside(rffi.llexternal("sendmsg_implementation",
+                               [rffi.INT, sockaddr_ptr, socklen_t, rffi.SIGNEDP, rffi.CCHARPP, rffi.INT,
+                                rffi.SIGNEDP, rffi.SIGNEDP, rffi.CCHARPP, rffi.SIGNEDP, rffi.INT, rffi.INT],
+                               rffi.INT, save_err=SAVE_ERR,
+                               compilation_info=compilation_info))
+CMSG_SPACE = jit.dont_look_inside(rffi.llexternal("CMSG_SPACE_wrapper",[size_t], size_t, save_err=SAVE_ERR,compilation_info=compilation_info))
+CMSG_LEN = jit.dont_look_inside(rffi.llexternal("CMSG_LEN_wrapper",[size_t], size_t, save_err=SAVE_ERR,compilation_info=compilation_info))
+
 socketshutdown = external('shutdown', [socketfd_type, rffi.INT], rffi.INT,
                           save_err=SAVE_ERR)
 gethostname = external('gethostname', [rffi.CCHARP, rffi.INT], rffi.INT,
diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py
--- a/rpython/rlib/rsocket.py
+++ b/rpython/rlib/rsocket.py
@@ -963,6 +963,126 @@
                 return (read_bytes, address)
             raise self.error_handler()
 
+    @jit.dont_look_inside
+    def recvmsg(self, message_size, ancbufsize = 0, flags = 0):
+        """
+        Receive up to message_size bytes from a message. Also receives ancillary data.
+        Returns the message, ancillary, flag and address of the sender.
+        :param message_size: Maximum size of the message to be received
+        :param ancbufsize:  Maximum size of the ancillary data to be received
+        :param flags: Receive flag. For more details, please check the Unix manual
+        :return: a tuple consisting of the message, the ancillary data, return flag and the address.
+        """
+        if message_size < 0:
+            raise RSocketError("Invalid message size")
+        if ancbufsize < 0:
+            raise RSocketError("invalid ancillary data buffer length")
+
+        self.wait_for_data(False)
+        address, addr_p, addrlen_p  = self._addrbuf()
+        len_of_msgs = lltype.malloc(rffi.SIGNEDPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False)
+        messages = lltype.malloc(rffi.CCHARPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False )
+        messages[0] = lltype.malloc(rffi.CCHARP.TO, message_size,flavor='raw',track_allocation=True,nonmovable=False)
+        rffi.c_memset(messages[0], 0, message_size)
+        no_of_messages  = lltype.malloc(rffi.SIGNEDP.TO,1,flavor='raw',track_allocation=True,nonmovable=False )
+        no_of_messages[0] = rffi.cast(rffi.SIGNED, 0)
+        size_of_anc = lltype.malloc(rffi.SIGNEDP.TO,1,flavor='raw',track_allocation=True,nonmovable=False )
+        size_of_anc[0] = rffi.cast(rffi.SIGNED,0)
+        levels = lltype.malloc(rffi.SIGNEDPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False)
+        types = lltype.malloc(rffi.SIGNEDPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False)
+        file_descr = lltype.malloc(rffi.CCHARPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False )
+        descr_per_anc = lltype.malloc(rffi.SIGNEDPP.TO,1,flavor='raw',track_allocation=True,nonmovable=False)
+        retflag = lltype.malloc(rffi.SIGNEDP.TO,1,flavor='raw',track_allocation=True,nonmovable=False )
+        retflag[0] = rffi.cast(rffi.SIGNED,0)
+
+        # a mask for the SIGNEDP's that need to be cast to int. (long default)
+        reply = _c.recvmsg(self.fd, rffi.cast(lltype.Signed,message_size),
+                           rffi.cast(lltype.Signed,ancbufsize),rffi.cast(lltype.Signed,flags),
+                           addr_p, addrlen_p, len_of_msgs, messages, no_of_messages,size_of_anc,
+                           levels, types,file_descr,descr_per_anc,retflag)
+        if reply >= 0:
+            anc_size  = rffi.cast(rffi.SIGNED,size_of_anc[0])
+            returnflag  = rffi.cast(rffi.SIGNED,retflag[0])
+            addrlen = rffi.cast(rffi.SIGNED,addrlen_p[0])
+
+            retmsg = rffi.charpsize2str(messages[0],reply)
+
+            offset = 0
+            list_of_tuples = []
+
+            pre_anc = lltype.malloc(rffi.CCHARPP.TO, 1, flavor='raw', track_allocation=True, nonmovable=False)
+            for i in range(anc_size):
+                level = rffi.cast(rffi.SIGNED, levels[0][i])
+                type = rffi.cast(rffi.SIGNED, types[0][i])
+                bytes_in_anc = rffi.cast(rffi.SIGNED, descr_per_anc[0][i])
+                pre_anc[0] = lltype.malloc(rffi.CCHARP.TO, bytes_in_anc,flavor='raw',track_allocation=True,nonmovable=False)
+                _c.memcpy_from_CCHARP_at_offset(file_descr[0], pre_anc,rffi.cast(rffi.SIGNED,offset), bytes_in_anc)
+                anc = rffi.charpsize2str(pre_anc[0],bytes_in_anc)
+                tup = (level,type, anc)
+                list_of_tuples.append(tup)
+                offset += bytes_in_anc
+                lltype.free(pre_anc[0], flavor='raw')
+
+            if addrlen:
+                address.addrlen = addrlen
+            else:
+                address.unlock()
+                address = None
+
+            rettup = (retmsg,list_of_tuples,returnflag,address)
+
+            if address is not None:
+                address.unlock()
+            # free underlying complexity first
+            _c.freeccharp(file_descr)
+            _c.freesignedp(len_of_msgs)
+            _c.freesignedp(levels)
+            _c.freesignedp(types)
+            _c.freesignedp(descr_per_anc)
+
+            lltype.free(messages[0], flavor='raw')
+            lltype.free(pre_anc,flavor='raw')
+            lltype.free(messages,flavor='raw')
+            lltype.free(file_descr,flavor='raw')
+            lltype.free(len_of_msgs,flavor='raw')
+            lltype.free(no_of_messages, flavor='raw')
+            lltype.free(size_of_anc, flavor='raw')
+            lltype.free(levels, flavor='raw')
+            lltype.free(types, flavor='raw')
+            lltype.free(descr_per_anc, flavor='raw')
+            lltype.free(retflag, flavor='raw')
+            lltype.free(addrlen_p,flavor='raw')
+
+            return rettup
+        else:
+
+            #in case of failure the underlying complexity has already been freed
+            lltype.free(messages[0], flavor='raw')
+            lltype.free(messages, flavor='raw')
+            lltype.free(file_descr, flavor='raw')
+            lltype.free(len_of_msgs, flavor='raw')
+            lltype.free(no_of_messages, flavor='raw')
+            lltype.free(size_of_anc, flavor='raw')
+            lltype.free(levels, flavor='raw')
+            lltype.free(types, flavor='raw')
+            lltype.free(descr_per_anc, flavor='raw')
+            lltype.free(retflag, flavor='raw')
+            lltype.free(addrlen_p, flavor='raw')
+
+            if address is not None:
+                address.unlock()
+            if _c.geterrno() == _c.EINTR:
+                raise last_error()
+            if (reply == -10000):
+                raise RSocketError("Invalid message size")
+            if (reply == -10001):
+                raise RSocketError("Invalid ancillary data buffer length")
+            if (reply == -10002):
+                raise RSocketError("received malformed or improperly truncated ancillary data")
+            raise last_error()
+
+
+
     def send_raw(self, dataptr, length, flags=0):
         """Send data from a CCHARP buffer."""
         self.wait_for_data(True)
@@ -1009,6 +1129,86 @@
             raise self.error_handler()
         return res
 
+    @jit.dont_look_inside
+    def sendmsg(self, messages, ancillary=None, flags=0, address=None):
+        """
+        Send data and ancillary on a socket. For use of ancillary data, please check the Unix manual.
+        Work on connectionless sockets via the address parameter.
+        :param messages: a message that is a list of strings
+        :param ancillary: data to be sent separate from the message body. Needs to be a list of tuples.
+                            E.g. [(level,type, bytes),...]. Default None.
+        :param flags: the flag to be set for sendmsg. Please check the Unix manual regarding values. Default 0
+        :param address: address of the recepient. Useful for when sending on connectionless sockets. Default None
+        :return: Bytes sent from the message
+        """
+        need_to_free_address = True
+        if address is None:
+            need_to_free_address = False
+            addr = lltype.nullptr(_c.sockaddr)
+            addrlen = 0
+        else:
+            addr = address.lock()
+            addrlen = address.addrlen
+
+        no_of_messages = len(messages)
+        messages_ptr = lltype.malloc(rffi.CCHARPP.TO,no_of_messages+1,flavor='raw',track_allocation=True,nonmovable=False)
+        messages_length_ptr = lltype.malloc(rffi.SIGNEDP.TO,no_of_messages,flavor='raw',zero=True, track_allocation=True,nonmovable=False)
+        counter = 0
+        for message in messages:
+            messages_ptr[counter] = rffi.str2charp(message)
+            messages_length_ptr[counter] = rffi.cast(rffi.SIGNED, len(message))
+            counter += 1
+        messages_ptr[counter] = lltype.nullptr(rffi.CCHARP.TO)
+        if ancillary is not None:
+            size_of_ancillary = len(ancillary)
+        else:
+            size_of_ancillary = 0
+        levels = lltype.malloc(rffi.SIGNEDP.TO, size_of_ancillary,flavor='raw',zero=True, track_allocation=True,nonmovable=False)
+        types = lltype.malloc(rffi.SIGNEDP.TO, size_of_ancillary,flavor='raw',zero=True, track_allocation=True,nonmovable=False)
+        desc_per_ancillary = lltype.malloc(rffi.SIGNEDP.TO, size_of_ancillary,flavor='raw',zero=True, track_allocation=True,nonmovable=False)
+        file_descr = lltype.malloc(rffi.CCHARPP.TO, size_of_ancillary,flavor='raw', track_allocation=True,nonmovable=False)
+        if ancillary is not None:
+            counter = 0
+            for level, type, content in ancillary:
+                assert isinstance(type,int)
+                assert isinstance(level, int)
+                levels[counter] = rffi.cast(rffi.SIGNED,level)
+                types[counter] = rffi.cast(rffi.SIGNED,type)
+                desc_per_ancillary[counter] = rffi.cast(rffi.SIGNED, (len(content)))
+                file_descr[counter] = rffi.str2charp(content, track_allocation=True)
+                counter +=1
+        else:
+            size_of_ancillary = 0
+        snd_no_msgs = rffi.cast(rffi.SIGNED, no_of_messages)
+        snd_anc_size =rffi.cast(rffi.SIGNED, size_of_ancillary)
+
+
+        bytes_sent = _c.sendmsg(self.fd, addr, addrlen, messages_length_ptr, messages_ptr, snd_no_msgs,levels,types,file_descr,desc_per_ancillary,snd_anc_size,flags)
+
+
+        if need_to_free_address:
+            address.unlock()
+        for i in range(len(messages)):
+            lltype.free(messages_ptr[i], flavor='raw', track_allocation=True)
+        lltype.free(messages_ptr, flavor='raw', track_allocation=True)
+        lltype.free(messages_length_ptr, flavor='raw', track_allocation=True)
+
+        if size_of_ancillary > 0:
+            for i in range(len(ancillary)):
+                lltype.free(file_descr[i], flavor='raw', track_allocation=True)
+        lltype.free(desc_per_ancillary, flavor='raw', track_allocation=True)
+        lltype.free(types, flavor='raw', track_allocation=True)
+        lltype.free(levels, flavor='raw', track_allocation=True)
+        lltype.free(file_descr, flavor='raw', track_allocation=True)
+
+        self.wait_for_data(True)
+        if (bytes_sent < 0) and (bytes_sent!=-1000) and (bytes_sent!=-1001) and (bytes_sent!=-1002):
+            raise last_error()
+
+        return bytes_sent
+
+
+
     def setblocking(self, block):
         if block:
             timeout = -1.0
@@ -1190,6 +1390,31 @@
         return (make_socket(fd0, family, type, proto, SocketClass),
                 make_socket(fd1, family, type, proto, SocketClass))
 
+if _c._POSIX:
+    def CMSG_LEN( demanded_len):
+        """
+        Socket method to determine the optimal byte size of the ancillary.
+        Recommended to be used when computing the ancillary size for recvmsg.
+        :param demanded_len: an integer with the minimum size required.
+        :return: an integer with the minimum memory needed for the required size. The value is not memory alligned
+        """
+        if demanded_len < 0:
+            return 0
+        result = _c.CMSG_LEN(demanded_len)
+        return result
+
+    def CMSG_SPACE( demanded_size):
+        """
+        Socket method to determine the optimal byte size of the ancillary.
+        Recommended to be used when computing the ancillary size for recvmsg.
+        :param demanded_size: an integer with the minimum size required.
+        :return: an integer with the minimum memory needed for the required size. The value is memory alligned
+        """
+        if demanded_size < 0:
+            return 0
+        result = _c.CMSG_SPACE(demanded_size)
+        return result
+
 if _c.WIN32:
     def dup(fd, inheritable=True):
         with lltype.scoped_alloc(_c.WSAPROTOCOL_INFO, zero=True) as info:
diff --git a/rpython/rtyper/lltypesystem/rffi.py b/rpython/rtyper/lltypesystem/rffi.py
--- a/rpython/rtyper/lltypesystem/rffi.py
+++ b/rpython/rtyper/lltypesystem/rffi.py
@@ -752,7 +752,8 @@
 
 # Signed, Signed *
 SIGNED = lltype.Signed
-SIGNEDP = lltype.Ptr(lltype.Array(SIGNED, hints={'nolength': True}))
+SIGNEDP = lltype.Ptr(lltype.Array(lltype.Signed, hints={'nolength': True}))
+SIGNEDPP = lltype.Ptr(lltype.Array(SIGNEDP, hints={'nolength': True}))
 
 
 # various type mapping

From pypy.commits at gmail.com  Thu Aug 31 07:50:55 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 31 Aug 2017 04:50:55 -0700 (PDT)
Subject: [pypy-commit] pypy py3.5-sendmsg-recvmsg: Close branch
 py3.5-sendmsg-recvmsg
Message-ID: <59a7f81f.358edf0a.db521.1dcb@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: py3.5-sendmsg-recvmsg
Changeset: r92285:ffb1e878bd5f
Date: 2017-08-31 11:50 +0000
http://bitbucket.org/pypy/pypy/changeset/ffb1e878bd5f/

Log:	Close branch py3.5-sendmsg-recvmsg


From pypy.commits at gmail.com  Thu Aug 31 08:46:36 2017
From: pypy.commits at gmail.com (stevie_92)
Date: Thu, 31 Aug 2017 05:46:36 -0700 (PDT)
Subject: [pypy-commit] pypy cpyext-gc-trialdeletion: Added more tests
Message-ID: <59a8052c.02da1c0a.194b9.0570@mx.google.com>

Author: Stefan Beyer <home at sbeyer.at>
Branch: cpyext-gc-trialdeletion
Changeset: r92287:74fa1f758dc8
Date: 2017-08-31 14:45 +0200
http://bitbucket.org/pypy/pypy/changeset/74fa1f758dc8/

Log:	Added more tests

diff --git a/rpython/memory/gc/test/test_rawrefcount.py b/rpython/memory/gc/test/test_rawrefcount.py
--- a/rpython/memory/gc/test/test_rawrefcount.py
+++ b/rpython/memory/gc/test/test_rawrefcount.py
@@ -441,3 +441,47 @@
         assert r1.base.c_ob_refcnt & REFCNT_MASK == 2
         assert r2.base.c_ob_refcnt & REFCNT_MASK == 1
 
+    def test_multiple_cycles_partial_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r3 = self._rawrefcount_cycle_obj()
+        r4 = self._rawrefcount_cycle_obj()
+        r5 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.next = r3
+        r3.next = r1
+        r2.prev = r5
+        r5.next = r4
+        r4.next = r5
+        r5.base.c_ob_refcnt += 1
+        r4.base.c_ob_refcnt += 1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r3.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r4.base.c_ob_refcnt & REFCNT_MASK == 2
+        assert r5.base.c_ob_refcnt & REFCNT_MASK == 1
+
+    def test_multiple_cycles_all_free(self):
+        self.gc.rawrefcount_init(lambda: self.trigger.append(1))
+        r1 = self._rawrefcount_cycle_obj()
+        r2 = self._rawrefcount_cycle_obj()
+        r3 = self._rawrefcount_cycle_obj()
+        r4 = self._rawrefcount_cycle_obj()
+        r5 = self._rawrefcount_cycle_obj()
+        r1.next = r2
+        r2.next = r3
+        r3.next = r1
+        r2.prev = r5
+        r5.next = r4
+        r4.next = r5
+        r5.base.c_ob_refcnt += 1
+        self._rawrefcount_buffer_obj(r1)
+        self.gc.rrc_collect_cycles()
+        assert r1.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r2.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r3.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r4.base.c_ob_refcnt & REFCNT_MASK == 0
+        assert r5.base.c_ob_refcnt & REFCNT_MASK == 0

From pypy.commits at gmail.com  Thu Aug 31 11:44:20 2017
From: pypy.commits at gmail.com (arigo)
Date: Thu, 31 Aug 2017 08:44:20 -0700 (PDT)
Subject: [pypy-commit] cffi default: Write down an explicit example of what
 not to do
Message-ID: <59a82ed4.10301c0a.1cc6a.342e@mx.google.com>

Author: Armin Rigo <arigo at tunes.org>
Branch: 
Changeset: r3006:3d609382a4b8
Date: 2017-08-31 17:44 +0200
http://bitbucket.org/cffi/cffi/changeset/3d609382a4b8/

Log:	Write down an explicit example of what not to do

diff --git a/doc/source/ref.rst b/doc/source/ref.rst
--- a/doc/source/ref.rst
+++ b/doc/source/ref.rst
@@ -51,6 +51,9 @@
 data can be used as long as this object is kept alive, but must not be
 used for a longer time.  Be careful about that when copying the
 pointer to the memory somewhere else, e.g. into another structure.
+Also, this means that a line like ``x = ffi.new(...)[0]`` is *always
+wrong:* the newly allocated object goes out of scope instantly, and so
+is freed immediately, and ``x`` is garbage.
 
 The returned memory is initially cleared (filled with zeroes), before
 the optional initializer is applied.  For performance, see

From pypy.commits at gmail.com  Thu Aug 31 12:29:31 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 31 Aug 2017 09:29:31 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Add _testmultiphase module,
 for test_importlib
Message-ID: <59a8396b.010b1c0a.4403c.35ac@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92288:236f54a091d0
Date: 2017-08-31 17:02 +0100
http://bitbucket.org/pypy/pypy/changeset/236f54a091d0/

Log:	Add _testmultiphase module, for test_importlib

diff --git a/lib-python/3/test/test_importlib/extension/test_loader.py b/lib-python/3/test/test_importlib/extension/test_loader.py
--- a/lib-python/3/test/test_importlib/extension/test_loader.py
+++ b/lib-python/3/test/test_importlib/extension/test_loader.py
@@ -88,6 +88,7 @@
 
     def setUp(self):
         self.name = '_testmultiphase'
+        __import__(self.name)  # PyPy hack
         finder = self.machinery.FileFinder(None)
         self.spec = importlib.util.find_spec(self.name)
         assert self.spec
diff --git a/lib_pypy/_testmultiphase.c b/lib_pypy/_testmultiphase.c
new file mode 100644
--- /dev/null
+++ b/lib_pypy/_testmultiphase.c
@@ -0,0 +1,627 @@
+/* Copied from CPython's Modules/_testmultiphase.c */
+/***************************************************/
+
+/* Testing module for multi-phase initialization of extension modules (PEP 489)
+ */
+
+#include "Python.h"
+
+/* Example objects */
+typedef struct {
+    PyObject_HEAD
+    PyObject            *x_attr;        /* Attributes dictionary */
+} ExampleObject;
+
+/* Example methods */
+
+static int
+Example_traverse(ExampleObject *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->x_attr);
+    return 0;
+}
+
+static int
+Example_finalize(ExampleObject *self)
+{
+    Py_CLEAR(self->x_attr);
+    return 0;
+}
+
+static PyObject *
+Example_demo(ExampleObject *self, PyObject *args)
+{
+    PyObject *o = NULL;
+    if (!PyArg_ParseTuple(args, "|O:demo", &o))
+        return NULL;
+    if (o != NULL && PyUnicode_Check(o)) {
+        Py_INCREF(o);
+        return o;
+    }
+    Py_INCREF(Py_None);
+    return Py_None;
+}
+
+
+static PyMethodDef Example_methods[] = {
+    {"demo",            (PyCFunction)Example_demo,  METH_VARARGS,
+        PyDoc_STR("demo() -> None")},
+    {NULL,              NULL}           /* sentinel */
+};
+
+static PyObject *
+Example_getattro(ExampleObject *self, PyObject *name)
+{
+    if (self->x_attr != NULL) {
+        PyObject *v = PyDict_GetItem(self->x_attr, name);
+        if (v != NULL) {
+            Py_INCREF(v);
+            return v;
+        }
+    }
+    return PyObject_GenericGetAttr((PyObject *)self, name);
+}
+
+static int
+Example_setattr(ExampleObject *self, char *name, PyObject *v)
+{
+    if (self->x_attr == NULL) {
+        self->x_attr = PyDict_New();
+        if (self->x_attr == NULL)
+            return -1;
+    }
+    if (v == NULL) {
+        int rv = PyDict_DelItemString(self->x_attr, name);
+        if (rv < 0)
+            PyErr_SetString(PyExc_AttributeError,
+                "delete non-existing Example attribute");
+        return rv;
+    }
+    else
+        return PyDict_SetItemString(self->x_attr, name, v);
+}
+
+static PyType_Slot Example_Type_slots[] = {
+    {Py_tp_doc, "The Example type"},
+    {Py_tp_finalize, Example_finalize},
+    {Py_tp_traverse, Example_traverse},
+    {Py_tp_getattro, Example_getattro},
+    {Py_tp_setattr, Example_setattr},
+    {Py_tp_methods, Example_methods},
+    {0, 0},
+};
+
+static PyType_Spec Example_Type_spec = {
+    "_testimportexec.Example",
+    sizeof(ExampleObject),
+    0,
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE,
+    Example_Type_slots
+};
+
+/* Function of two integers returning integer */
+
+PyDoc_STRVAR(testexport_foo_doc,
+"foo(i,j)\n\
+\n\
+Return the sum of i and j.");
+
+static PyObject *
+testexport_foo(PyObject *self, PyObject *args)
+{
+    long i, j;
+    long res;
+    if (!PyArg_ParseTuple(args, "ll:foo", &i, &j))
+        return NULL;
+    res = i + j;
+    return PyLong_FromLong(res);
+}
+
+/* Test that PyState registration fails  */
+
+PyDoc_STRVAR(call_state_registration_func_doc,
+"register_state(0): call PyState_FindModule()\n\
+register_state(1): call PyState_AddModule()\n\
+register_state(2): call PyState_RemoveModule()");
+
+static PyObject *
+call_state_registration_func(PyObject *mod, PyObject *args)
+{
+    int i, ret;
+    PyModuleDef *def = PyModule_GetDef(mod);
+    if (def == NULL) {
+        return NULL;
+    }
+    if (!PyArg_ParseTuple(args, "i:call_state_registration_func", &i))
+        return NULL;
+    switch (i) {
+        case 0:
+            mod = PyState_FindModule(def);
+            if (mod == NULL) {
+                Py_RETURN_NONE;
+            }
+            return mod;
+        case 1:
+            ret = PyState_AddModule(mod, def);
+            if (ret != 0) {
+                return NULL;
+            }
+            break;
+        case 2:
+            ret = PyState_RemoveModule(def);
+            if (ret != 0) {
+                return NULL;
+            }
+            break;
+    }
+    Py_RETURN_NONE;
+}
+
+
+static PyType_Slot Str_Type_slots[] = {
+    {Py_tp_base, NULL}, /* filled out in module exec function */
+    {0, 0},
+};
+
+static PyType_Spec Str_Type_spec = {
+    "_testimportexec.Str",
+    0,
+    0,
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
+    Str_Type_slots
+};
+
+static PyMethodDef testexport_methods[] = {
+    {"foo",             testexport_foo,         METH_VARARGS,
+        testexport_foo_doc},
+    {"call_state_registration_func",  call_state_registration_func,
+        METH_VARARGS, call_state_registration_func_doc},
+    {NULL,              NULL}           /* sentinel */
+};
+
+static int execfunc(PyObject *m)
+{
+    PyObject *temp = NULL;
+
+    /* Due to cross platform compiler issues the slots must be filled
+     * here. It's required for portability to Windows without requiring
+     * C++. */
+    Str_Type_slots[0].pfunc = &PyUnicode_Type;
+
+    /* Add a custom type */
+    temp = PyType_FromSpec(&Example_Type_spec);
+    if (temp == NULL)
+        goto fail;
+    if (PyModule_AddObject(m, "Example", temp) != 0)
+        goto fail;
+
+    /* Add an exception type */
+    temp = PyErr_NewException("_testimportexec.error", NULL, NULL);
+    if (temp == NULL)
+        goto fail;
+    if (PyModule_AddObject(m, "error", temp) != 0)
+        goto fail;
+
+    /* Add Str */
+    temp = PyType_FromSpec(&Str_Type_spec);
+    if (temp == NULL)
+        goto fail;
+    if (PyModule_AddObject(m, "Str", temp) != 0)
+        goto fail;
+
+    if (PyModule_AddIntConstant(m, "int_const", 1969) != 0)
+        goto fail;
+
+    if (PyModule_AddStringConstant(m, "str_const", "something different") != 0)
+        goto fail;
+
+    return 0;
+ fail:
+    return -1;
+}
+
+/* Helper for module definitions; there'll be a lot of them */
+#define TEST_MODULE_DEF(name, slots, methods) { \
+    PyModuleDef_HEAD_INIT,                      /* m_base */ \
+    name,                                       /* m_name */ \
+    PyDoc_STR("Test module " name),             /* m_doc */ \
+    0,                                          /* m_size */ \
+    methods,                                    /* m_methods */ \
+    slots,                                      /* m_slots */ \
+    NULL,                                       /* m_traverse */ \
+    NULL,                                       /* m_clear */ \
+    NULL,                                       /* m_free */ \
+}
+
+PyModuleDef_Slot main_slots[] = {
+    {Py_mod_exec, execfunc},
+    {0, NULL},
+};
+
+static PyModuleDef main_def = TEST_MODULE_DEF("main", main_slots, testexport_methods);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase(PyObject *spec)
+{
+    return PyModuleDef_Init(&main_def);
+}
+
+
+/**** Importing a non-module object ****/
+
+static PyModuleDef def_nonmodule;
+static PyModuleDef def_nonmodule_with_methods;
+
+/* Create a SimpleNamespace(three=3) */
+static PyObject*
+createfunc_nonmodule(PyObject *spec, PyModuleDef *def)
+{
+    PyObject *dct, *ns, *three;
+
+    if (def != &def_nonmodule && def != &def_nonmodule_with_methods) {
+        PyErr_SetString(PyExc_SystemError, "def does not match");
+        return NULL;
+    }
+
+    dct = PyDict_New();
+    if (dct == NULL)
+        return NULL;
+
+    three = PyLong_FromLong(3);
+    if (three == NULL) {
+        Py_DECREF(dct);
+        return NULL;
+    }
+    PyDict_SetItemString(dct, "three", three);
+    Py_DECREF(three);
+
+    ns = _PyNamespace_New(dct);
+    Py_DECREF(dct);
+    return ns;
+}
+
+static PyModuleDef_Slot slots_create_nonmodule[] = {
+    {Py_mod_create, createfunc_nonmodule},
+    {0, NULL},
+};
+
+static PyModuleDef def_nonmodule = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule", slots_create_nonmodule, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule);
+}
+
+PyDoc_STRVAR(nonmodule_bar_doc,
+"bar(i,j)\n\
+\n\
+Return the difference of i - j.");
+
+static PyObject *
+nonmodule_bar(PyObject *self, PyObject *args)
+{
+    long i, j;
+    long res;
+    if (!PyArg_ParseTuple(args, "ll:bar", &i, &j))
+        return NULL;
+    res = i - j;
+    return PyLong_FromLong(res);
+}
+
+static PyMethodDef nonmodule_methods[] = {
+    {"bar", nonmodule_bar, METH_VARARGS, nonmodule_bar_doc},
+    {NULL, NULL}           /* sentinel */
+};
+
+static PyModuleDef def_nonmodule_with_methods = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule_with_methods", slots_create_nonmodule, nonmodule_methods);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule_with_methods(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule_with_methods);
+}
+
+/**** Non-ASCII-named modules ****/
+
+static PyModuleDef def_nonascii_latin = { \
+    PyModuleDef_HEAD_INIT,                      /* m_base */
+    "_testmultiphase_nonascii_latin",           /* m_name */
+    PyDoc_STR("Module named in Czech"),         /* m_doc */
+    0,                                          /* m_size */
+    NULL,                                       /* m_methods */
+    NULL,                                       /* m_slots */
+    NULL,                                       /* m_traverse */
+    NULL,                                       /* m_clear */
+    NULL,                                       /* m_free */
+};
+
+PyMODINIT_FUNC
+PyInitU__testmultiphase_zkouka_naten_evc07gi8e(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonascii_latin);
+}
+
+static PyModuleDef def_nonascii_kana = { \
+    PyModuleDef_HEAD_INIT,                      /* m_base */
+    "_testmultiphase_nonascii_kana",            /* m_name */
+    PyDoc_STR("Module named in Japanese"),      /* m_doc */
+    0,                                          /* m_size */
+    NULL,                                       /* m_methods */
+    NULL,                                       /* m_slots */
+    NULL,                                       /* m_traverse */
+    NULL,                                       /* m_clear */
+    NULL,                                       /* m_free */
+};
+
+PyMODINIT_FUNC
+PyInitU_eckzbwbhc6jpgzcx415x(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonascii_kana);
+}
+
+/*** Module with a single-character name ***/
+
+PyMODINIT_FUNC
+PyInit_x(PyObject *spec)
+{
+    return PyModuleDef_Init(&main_def);
+}
+
+/**** Testing NULL slots ****/
+
+static PyModuleDef null_slots_def = TEST_MODULE_DEF(
+    "_testmultiphase_null_slots", NULL, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_null_slots(PyObject *spec)
+{
+    return PyModuleDef_Init(&null_slots_def);
+}
+
+/**** Problematic modules ****/
+
+static PyModuleDef_Slot slots_bad_large[] = {
+    {_Py_mod_LAST_SLOT + 1, NULL},
+    {0, NULL},
+};
+
+static PyModuleDef def_bad_large = TEST_MODULE_DEF(
+    "_testmultiphase_bad_slot_large", slots_bad_large, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_bad_slot_large(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_bad_large);
+}
+
+static PyModuleDef_Slot slots_bad_negative[] = {
+    {-1, NULL},
+    {0, NULL},
+};
+
+static PyModuleDef def_bad_negative = TEST_MODULE_DEF(
+    "_testmultiphase_bad_slot_negative", slots_bad_negative, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_bad_slot_negative(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_bad_negative);
+}
+
+static PyModuleDef def_create_int_with_state = { \
+    PyModuleDef_HEAD_INIT,                      /* m_base */
+    "create_with_state",                        /* m_name */
+    PyDoc_STR("Not a PyModuleObject object, but requests per-module state"),
+    10,                                         /* m_size */
+    NULL,                                       /* m_methods */
+    slots_create_nonmodule,                     /* m_slots */
+    NULL,                                       /* m_traverse */
+    NULL,                                       /* m_clear */
+    NULL,                                       /* m_free */
+};
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_create_int_with_state(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_create_int_with_state);
+}
+
+
+static PyModuleDef def_negative_size = { \
+    PyModuleDef_HEAD_INIT,                      /* m_base */
+    "negative_size",                            /* m_name */
+    PyDoc_STR("PyModuleDef with negative m_size"),
+    -1,                                         /* m_size */
+    NULL,                                       /* m_methods */
+    slots_create_nonmodule,                     /* m_slots */
+    NULL,                                       /* m_traverse */
+    NULL,                                       /* m_clear */
+    NULL,                                       /* m_free */
+};
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_negative_size(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_negative_size);
+}
+
+
+static PyModuleDef uninitialized_def = TEST_MODULE_DEF("main", main_slots, testexport_methods);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_export_uninitialized(PyObject *spec)
+{
+    return (PyObject*) &uninitialized_def;
+}
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_export_null(PyObject *spec)
+{
+    return NULL;
+}
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_export_raise(PyObject *spec)
+{
+    PyErr_SetString(PyExc_SystemError, "bad export function");
+    return NULL;
+}
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_export_unreported_exception(PyObject *spec)
+{
+    PyErr_SetString(PyExc_SystemError, "bad export function");
+    return PyModuleDef_Init(&main_def);
+}
+
+static PyObject*
+createfunc_null(PyObject *spec, PyModuleDef *def)
+{
+    return NULL;
+}
+
+PyModuleDef_Slot slots_create_null[] = {
+    {Py_mod_create, createfunc_null},
+    {0, NULL},
+};
+
+static PyModuleDef def_create_null = TEST_MODULE_DEF(
+    "_testmultiphase_create_null", slots_create_null, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_create_null(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_create_null);
+}
+
+static PyObject*
+createfunc_raise(PyObject *spec, PyModuleDef *def)
+{
+    PyErr_SetString(PyExc_SystemError, "bad create function");
+    return NULL;
+}
+
+static PyModuleDef_Slot slots_create_raise[] = {
+    {Py_mod_create, createfunc_raise},
+    {0, NULL},
+};
+
+static PyModuleDef def_create_raise = TEST_MODULE_DEF(
+    "_testmultiphase_create_null", slots_create_raise, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_create_raise(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_create_raise);
+}
+
+static PyObject*
+createfunc_unreported_exception(PyObject *spec, PyModuleDef *def)
+{
+    PyErr_SetString(PyExc_SystemError, "bad create function");
+    return PyModule_New("foo");
+}
+
+static PyModuleDef_Slot slots_create_unreported_exception[] = {
+    {Py_mod_create, createfunc_unreported_exception},
+    {0, NULL},
+};
+
+static PyModuleDef def_create_unreported_exception = TEST_MODULE_DEF(
+    "_testmultiphase_create_unreported_exception", slots_create_unreported_exception, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_create_unreported_exception(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_create_unreported_exception);
+}
+
+static PyModuleDef_Slot slots_nonmodule_with_exec_slots[] = {
+    {Py_mod_create, createfunc_nonmodule},
+    {Py_mod_exec, execfunc},
+    {0, NULL},
+};
+
+static PyModuleDef def_nonmodule_with_exec_slots = TEST_MODULE_DEF(
+    "_testmultiphase_nonmodule_with_exec_slots", slots_nonmodule_with_exec_slots, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_nonmodule_with_exec_slots(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_nonmodule_with_exec_slots);
+}
+
+static int
+execfunc_err(PyObject *mod)
+{
+    return -1;
+}
+
+static PyModuleDef_Slot slots_exec_err[] = {
+    {Py_mod_exec, execfunc_err},
+    {0, NULL},
+};
+
+static PyModuleDef def_exec_err = TEST_MODULE_DEF(
+    "_testmultiphase_exec_err", slots_exec_err, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_exec_err(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_exec_err);
+}
+
+static int
+execfunc_raise(PyObject *spec)
+{
+    PyErr_SetString(PyExc_SystemError, "bad exec function");
+    return -1;
+}
+
+static PyModuleDef_Slot slots_exec_raise[] = {
+    {Py_mod_exec, execfunc_raise},
+    {0, NULL},
+};
+
+static PyModuleDef def_exec_raise = TEST_MODULE_DEF(
+    "_testmultiphase_exec_raise", slots_exec_raise, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_exec_raise(PyObject *mod)
+{
+    return PyModuleDef_Init(&def_exec_raise);
+}
+
+static int
+execfunc_unreported_exception(PyObject *mod)
+{
+    PyErr_SetString(PyExc_SystemError, "bad exec function");
+    return 0;
+}
+
+static PyModuleDef_Slot slots_exec_unreported_exception[] = {
+    {Py_mod_exec, execfunc_unreported_exception},
+    {0, NULL},
+};
+
+static PyModuleDef def_exec_unreported_exception = TEST_MODULE_DEF(
+    "_testmultiphase_exec_unreported_exception", slots_exec_unreported_exception, NULL);
+
+PyMODINIT_FUNC
+PyInit__testmultiphase_exec_unreported_exception(PyObject *spec)
+{
+    return PyModuleDef_Init(&def_exec_unreported_exception);
+}
+
+/*** Helper for imp test ***/
+
+static PyModuleDef imp_dummy_def = TEST_MODULE_DEF("imp_dummy", main_slots, testexport_methods);
+
+PyMODINIT_FUNC
+PyInit_imp_dummy(PyObject *spec)
+{
+    return PyModuleDef_Init(&imp_dummy_def);
+}
diff --git a/lib_pypy/_testmultiphase.py b/lib_pypy/_testmultiphase.py
new file mode 100644
--- /dev/null
+++ b/lib_pypy/_testmultiphase.py
@@ -0,0 +1,18 @@
+import imp
+import os
+
+try:
+    import cpyext
+except ImportError:
+    raise ImportError("No module named '_testmultiphase'")
+import _pypy_testcapi
+cfile = '_testmultiphase.c'
+thisdir = os.path.dirname(__file__)
+output_dir = _pypy_testcapi.get_hashed_dir(os.path.join(thisdir, cfile))
+try:
+    fp, filename, description = imp.find_module('_test_multiphase', path=[output_dir])
+    with fp:
+        imp.load_module('_testmultiphase', fp, filename, description)
+except ImportError:
+    print('could not find _testmultiphase in %s' % output_dir)
+    _pypy_testcapi.compile_shared('_testmultiphase.c', '_testmultiphase', output_dir)

From pypy.commits at gmail.com  Thu Aug 31 12:29:33 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 31 Aug 2017 09:29:33 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: disable
 _testmultiphase.call_state_registration_func()
Message-ID: <59a8396d.d8aadf0a.71600.06c6@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92289:8de4e0b09ce1
Date: 2017-08-31 17:23 +0100
http://bitbucket.org/pypy/pypy/changeset/8de4e0b09ce1/

Log:	disable _testmultiphase.call_state_registration_func()

diff --git a/lib-python/3/test/test_importlib/extension/test_loader.py b/lib-python/3/test/test_importlib/extension/test_loader.py
--- a/lib-python/3/test/test_importlib/extension/test_loader.py
+++ b/lib-python/3/test/test_importlib/extension/test_loader.py
@@ -146,7 +146,8 @@
             importlib.reload(module)
             self.assertIs(ex_class, module.Example)
 
-    def test_try_registration(self):
+    # XXX: PyPy doesn't support the PyState_* functions yet
+    def XXXtest_try_registration(self):
         '''Assert that the PyState_{Find,Add,Remove}Module C API doesn't work'''
         module = self.load_module()
         with self.subTest('PyState_FindModule'):
diff --git a/lib_pypy/_testmultiphase.c b/lib_pypy/_testmultiphase.c
--- a/lib_pypy/_testmultiphase.c
+++ b/lib_pypy/_testmultiphase.c
@@ -119,43 +119,43 @@
 
 /* Test that PyState registration fails  */
 
-PyDoc_STRVAR(call_state_registration_func_doc,
-"register_state(0): call PyState_FindModule()\n\
-register_state(1): call PyState_AddModule()\n\
-register_state(2): call PyState_RemoveModule()");
-
-static PyObject *
-call_state_registration_func(PyObject *mod, PyObject *args)
-{
-    int i, ret;
-    PyModuleDef *def = PyModule_GetDef(mod);
-    if (def == NULL) {
-        return NULL;
-    }
-    if (!PyArg_ParseTuple(args, "i:call_state_registration_func", &i))
-        return NULL;
-    switch (i) {
-        case 0:
-            mod = PyState_FindModule(def);
-            if (mod == NULL) {
-                Py_RETURN_NONE;
-            }
-            return mod;
-        case 1:
-            ret = PyState_AddModule(mod, def);
-            if (ret != 0) {
-                return NULL;
-            }
-            break;
-        case 2:
-            ret = PyState_RemoveModule(def);
-            if (ret != 0) {
-                return NULL;
-            }
-            break;
-    }
-    Py_RETURN_NONE;
-}
+//PyDoc_STRVAR(call_state_registration_func_doc,
+//"register_state(0): call PyState_FindModule()\n\
+//register_state(1): call PyState_AddModule()\n\
+//register_state(2): call PyState_RemoveModule()");
+//
+//static PyObject *
+//call_state_registration_func(PyObject *mod, PyObject *args)
+//{
+//    int i, ret;
+//    PyModuleDef *def = PyModule_GetDef(mod);
+//    if (def == NULL) {
+//        return NULL;
+//    }
+//    if (!PyArg_ParseTuple(args, "i:call_state_registration_func", &i))
+//        return NULL;
+//    switch (i) {
+//        case 0:
+//            mod = PyState_FindModule(def);
+//            if (mod == NULL) {
+//                Py_RETURN_NONE;
+//            }
+//            return mod;
+//        case 1:
+//            ret = PyState_AddModule(mod, def);
+//            if (ret != 0) {
+//                return NULL;
+//            }
+//            break;
+//        case 2:
+//            ret = PyState_RemoveModule(def);
+//            if (ret != 0) {
+//                return NULL;
+//            }
+//            break;
+//    }
+//    Py_RETURN_NONE;
+//}
 
 
 static PyType_Slot Str_Type_slots[] = {
@@ -174,8 +174,8 @@
 static PyMethodDef testexport_methods[] = {
     {"foo",             testexport_foo,         METH_VARARGS,
         testexport_foo_doc},
-    {"call_state_registration_func",  call_state_registration_func,
-        METH_VARARGS, call_state_registration_func_doc},
+//    {"call_state_registration_func",  call_state_registration_func,
+//        METH_VARARGS, call_state_registration_func_doc},
     {NULL,              NULL}           /* sentinel */
 };
 

From pypy.commits at gmail.com  Thu Aug 31 12:29:35 2017
From: pypy.commits at gmail.com (rlamy)
Date: Thu, 31 Aug 2017 09:29:35 -0700 (PDT)
Subject: [pypy-commit] pypy multiphase: Disable not-yet-supported
 tp_finalize slot in _testmultiphase
Message-ID: <59a8396f.89c4df0a.72ff5.0a3d@mx.google.com>

Author: Ronan Lamy <ronan.lamy at gmail.com>
Branch: multiphase
Changeset: r92290:9edde9447569
Date: 2017-08-31 17:28 +0100
http://bitbucket.org/pypy/pypy/changeset/9edde9447569/

Log:	Disable not-yet-supported tp_finalize slot in _testmultiphase

diff --git a/lib_pypy/_testmultiphase.c b/lib_pypy/_testmultiphase.c
--- a/lib_pypy/_testmultiphase.c
+++ b/lib_pypy/_testmultiphase.c
@@ -83,7 +83,7 @@
 
 static PyType_Slot Example_Type_slots[] = {
     {Py_tp_doc, "The Example type"},
-    {Py_tp_finalize, Example_finalize},
+//    {Py_tp_finalize, Example_finalize},
     {Py_tp_traverse, Example_traverse},
     {Py_tp_getattro, Example_getattro},
     {Py_tp_setattr, Example_setattr},