From rougered4 at gmail.com  Sat Jun  1 23:59:19 2013
From: rougered4 at gmail.com (Riccardo Rossi)
Date: Sat, 01 Jun 2013 21:59:19 -0000
Subject: [pypy-dev] Compatibilidad with boost python
Message-ID: <1597854793122323073@unknownmsgid>

Dear list,
              I am writing since I would like to understand if it is
possible to use pypy together with modulus and types exported by
boost_python.

My python scripts may be slow( due to a lot of loops), but they are rather
elementary. The key issue is that All the datatype I work with are exported
to python via boost python...

By googling, I could not figure out if this is supposed to work or not. If
it should work, which lib should I compile boost against?

Thx in advance for any hint

Riccardo

Sent from my BlackBerry? PlayBook?
www.blackberry.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130601/fcf5248e/attachment.html>

From dataroid at gmail.com  Sun Jun  2 01:51:50 2013
From: dataroid at gmail.com (David Roid)
Date: Sun, 2 Jun 2013 07:51:50 +0800
Subject: [pypy-dev] pypy-2.0 package issue
Message-ID: <CAKN1bq5GBq9VZ4+divNiP_KRTAMYkR+WupSUoCQkZScqd6QrVA@mail.gmail.com>

Greetings,

As prebuilt pypy Linux binaries only available for Ubuntu, I'm following
this guide (http://pypy.org/download.html) to build pypy-2.0 from source
for CentOS 5.8 x64. The translation was successful with pypy-c generated,
but subsequent packaging went wrong with following error:

<snip>
    /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c: In
function ?_cffi_const_SQLITE_CREATE_VTABLE?:

/root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
error: ?SQLITE_CREATE_VTABLE? undeclared (first use in this function)

/root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
error: (Each undeclared identifier is reported only once

/root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
error: for each function it appears in.)
    /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c: In
function ?_cffi_const_SQLITE_DROP_VTABLE?:

/root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:577:
error: ?SQLITE_DROP_VTABLE? undeclared (first use in this function)
    /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c: In
function ?_cffi_const_SQLITE_FUNCTION?:

/root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:613:
error: ?SQLITE_FUNCTION? undeclared (first use in this function)
    Traceback (most recent call last):
      File "app_main.py", line 72, in run_toplevel
    ...omitting some info...
      File "/root/pypy-2.0-src/lib_pypy/cffi/ffiplatform.py", line 50, in
_build
        raise VerificationError('%s: %s' % (e.__class__.__name__, e))
    VerificationError: CompileError: command 'cc' failed with exit status 1
    Traceback (most recent call last):
      File "./package.py", line 185, in <module>
        package(*args, **kw)
      File "./package.py", line 69, in package
        subprocess.check_call([str(pypy_c), '-c', 'import _sqlite3'])
      File "/usr/local/lib/python2.7/subprocess.py", line 511, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command
'['/root/pypy-2.0-src/pypy/goal/pypy-c', '-c', 'import _sqlite3']' returned
non-zero exit status 1
<snip>

As far as I can tell this has to do with sqlite3 but I already have both
sqlite3 and sqlite3-devel packages installed. So what do I miss here?

p.s. This might not be the best list for such issues but I don't see any
other pypy maillist.

Regards
-David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130602/0c0390cf/attachment.html>

From arigo at tunes.org  Sun Jun  2 09:59:11 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 2 Jun 2013 09:59:11 +0200
Subject: [pypy-dev] pypy-2.0 package issue
In-Reply-To: <CAKN1bq5GBq9VZ4+divNiP_KRTAMYkR+WupSUoCQkZScqd6QrVA@mail.gmail.com>
References: <CAKN1bq5GBq9VZ4+divNiP_KRTAMYkR+WupSUoCQkZScqd6QrVA@mail.gmail.com>
Message-ID: <CAMSv6X1v05kOGsHH0SAOKbFL-Z=1Wj5yeJa+9Qu5xhaTweG=kQ@mail.gmail.com>

Hi David,

On Sun, Jun 2, 2013 at 1:51 AM, David Roid <dataroid at gmail.com> wrote:
> CentOS 5.8 x64. The translation was successful with pypy-c generated, but
> subsequent packaging went wrong with following error:
>
> <snip>
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c: In
> function ?_cffi_const_SQLITE_CREATE_VTABLE?:
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> error: ?SQLITE_CREATE_VTABLE? undeclared (first use in this function)
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> error: (Each undeclared identifier is reported only once
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> error: for each function it appears in.)
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c: In
> function ?_cffi_const_SQLITE_DROP_VTABLE?:
>     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:577:
> error: ?SQLITE_DROP_VTABLE? undeclared (first use in this function)

Look a bit above (in the <snip> part :-) what the first error message
produced by GCC is.  It may likely be: cannot find #include
<sqlite3.h>.  Either that or you have a broken or very out-of-date
version of sqlite3.h around.  Either way I don't really understand,
given that you said it's not the case.  For example in my old
sqlite3.h there is a line "#define SQLITE_DROP_VTABLE 30" which should
prevent the above error.


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Jun  2 18:00:36 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 02 Jun 2013 19:00:36 +0300
Subject: [pypy-dev] why does jit.backend.x86.test.test_fficall crash python
	on win32?
Message-ID: <51AB6C24.3090807@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130602/af7fad44/attachment.html>

From amauryfa at gmail.com  Sun Jun  2 18:32:50 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 2 Jun 2013 18:32:50 +0200
Subject: [pypy-dev] why does jit.backend.x86.test.test_fficall crash
 python on win32?
In-Reply-To: <51AB6C24.3090807@gmail.com>
References: <51AB6C24.3090807@gmail.com>
Message-ID: <CAGmFida04euVz1NqBczUY+wo7i=OY6F003_8NbcB3W4BFs2XLw@mail.gmail.com>

2013/6/2 Matti Picus <matti.picus at gmail.com>

>  I have been trying to whip win32 own tests into passing, or at least to
> an understanding of why they fail.
> I am seeing the interpreter crash on the
> jit.backend.x86.test.test_fficall.py, here is a traceback
> http://bpaste.net/show/cM9YptahKH0PkZjp5gtN/
> and the original fail is here (not very helpful, sorry)
>
> http://buildbot.pypy.org/summary/longrepr?testname=&builder=own-win-x86-32&build=39&mod=jit%2Fbackend%2Fx86%2Ftest%2Ftest_fficall.py
>

That page has a hint though: the signal number 1073741783 corresponds to
0xC0000029,
which is STATUS_INVALID_UNWIND_TARGET, and suggests a corruption of the
stack.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130602/2bca0d58/attachment.html>

From arigo at tunes.org  Sun Jun  2 18:33:18 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 2 Jun 2013 18:33:18 +0200
Subject: [pypy-dev] why does jit.backend.x86.test.test_fficall crash
 python on win32?
In-Reply-To: <51AB6C24.3090807@gmail.com>
References: <51AB6C24.3090807@gmail.com>
Message-ID: <CAMSv6X2++_1qmsfJafwOQfJyQ-8antxMMLED7Tu44OtET1yw3A@mail.gmail.com>

Hi Matti,

On Sun, Jun 2, 2013 at 6:00 PM, Matti Picus <matti.picus at gmail.com> wrote:
> I have been trying to whip win32 own tests into passing, or at least to an
> understanding of why they fail.

Fixed in 663474f13069; it turned out to be just a stupid mistake (the
random value abi=42 in the metainterp test was used all the way down
in the x86 test).


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Jun  2 22:58:17 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 02 Jun 2013 23:58:17 +0300
Subject: [pypy-dev] cpython changed _sre between 2.7.3 to 2.7.4,
 now intmask(pattern) is negative
Message-ID: <51ABB1E9.4080305@gmail.com>

Running own tests on win32 gives many errors in the _sre module:
rlib/rsre/rsre_core.py:111: AssertionError
which is checking that all values of the pattern are >=0
Here is one of many such failures:
http://buildbot.pypy.org/summary/longrepr?testname=TestMatch.%28%29.test_repeated_set&builder=own-win-x86-32&build=40&mod=rlib.rsre.test.test_match

It turns out that the buildbot has been updated to use python 2.7.4, and 
in cpython's changeset c67b7e0c818a they changed the type of SRE_CODE 
from int to long.
We demand the pattern values be int via a call to intmask in 
rpython/rlib/rsre/rsre_core.py line 37-8:
def my_compile(pattern, flags, code, *args):
     raise GotIt(*[intmask*(i) for i in code], flags, args)

This call flips the sign of long values to <0, hence the tests fail.

Note this would also happen on Linux32, except that the cpython used in 
the tests is 2.6.5
http://buildbot.pypy.org/builders/own-linux-x86-32/builds/3044/steps/shell_7/logs/stdio

So far the detective work, can someone suggest a fix?
Matti

From fijall at gmail.com  Sun Jun  2 23:05:30 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 3 Jun 2013 05:05:30 +0800
Subject: [pypy-dev] cpython changed _sre between 2.7.3 to 2.7.4,
 now intmask(pattern) is negative
In-Reply-To: <51ABB1E9.4080305@gmail.com>
References: <51ABB1E9.4080305@gmail.com>
Message-ID: <CAK5idxS8PBLWRi8SBEbK+_Hpu9r_N6CJNWJhGjf6RmZhSUuVnA@mail.gmail.com>

pfff, just make it r_uint maybe? Also, what are legal values on CPython?

On Mon, Jun 3, 2013 at 4:58 AM, Matti Picus <matti.picus at gmail.com> wrote:
> Running own tests on win32 gives many errors in the _sre module:
> rlib/rsre/rsre_core.py:111: AssertionError
> which is checking that all values of the pattern are >=0
> Here is one of many such failures:
> http://buildbot.pypy.org/summary/longrepr?testname=TestMatch.%28%29.test_repeated_set&builder=own-win-x86-32&build=40&mod=rlib.rsre.test.test_match
>
> It turns out that the buildbot has been updated to use python 2.7.4, and in
> cpython's changeset c67b7e0c818a they changed the type of SRE_CODE from int
> to long.
> We demand the pattern values be int via a call to intmask in
> rpython/rlib/rsre/rsre_core.py line 37-8:
> def my_compile(pattern, flags, code, *args):
>     raise GotIt(*[intmask*(i) for i in code], flags, args)
>
> This call flips the sign of long values to <0, hence the tests fail.
>
> Note this would also happen on Linux32, except that the cpython used in the
> tests is 2.6.5
> http://buildbot.pypy.org/builders/own-linux-x86-32/builds/3044/steps/shell_7/logs/stdio
>
> So far the detective work, can someone suggest a fix?
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From matti.picus at gmail.com  Mon Jun  3 00:52:48 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 03 Jun 2013 01:52:48 +0300
Subject: [pypy-dev] cpython changed _sre between 2.7.3 to 2.7.4,
 now intmask(pattern) is negative
In-Reply-To: <CAK5idxS8PBLWRi8SBEbK+_Hpu9r_N6CJNWJhGjf6RmZhSUuVnA@mail.gmail.com>
References: <51ABB1E9.4080305@gmail.com>
	<CAK5idxS8PBLWRi8SBEbK+_Hpu9r_N6CJNWJhGjf6RmZhSUuVnA@mail.gmail.com>
Message-ID: <51ABCCC0.8060504@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130603/efa3cd35/attachment.html>

From dataroid at gmail.com  Mon Jun  3 01:18:37 2013
From: dataroid at gmail.com (David Roid)
Date: Mon, 3 Jun 2013 07:18:37 +0800
Subject: [pypy-dev] pypy-2.0 package issue
In-Reply-To: <CAMSv6X1v05kOGsHH0SAOKbFL-Z=1Wj5yeJa+9Qu5xhaTweG=kQ@mail.gmail.com>
References: <CAKN1bq5GBq9VZ4+divNiP_KRTAMYkR+WupSUoCQkZScqd6QrVA@mail.gmail.com>
	<CAMSv6X1v05kOGsHH0SAOKbFL-Z=1Wj5yeJa+9Qu5xhaTweG=kQ@mail.gmail.com>
Message-ID: <CAKN1bq70JG1HW18q_XzW3+H9vGwwhjjYFC=_8sxQrtcj=FM6LQ@mail.gmail.com>

Thanks Armin! An upgrade from sqlite 3.3 to 3.6 solved this problem.

Regards
-David


2013/6/2 Armin Rigo <arigo at tunes.org>

> Hi David,
>
> On Sun, Jun 2, 2013 at 1:51 AM, David Roid <dataroid at gmail.com> wrote:
> > CentOS 5.8 x64. The translation was successful with pypy-c generated, but
> > subsequent packaging went wrong with following error:
> >
> > <snip>
> >     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:
> In
> > function ?_cffi_const_SQLITE_CREATE_VTABLE?:
> >
> /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> > error: ?SQLITE_CREATE_VTABLE? undeclared (first use in this function)
> >
> /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> > error: (Each undeclared identifier is reported only once
> >
> /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:499:
> > error: for each function it appears in.)
> >     /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:
> In
> > function ?_cffi_const_SQLITE_DROP_VTABLE?:
> >
> /root/pypy-2.0-src/lib_pypy/__pycache__/_cffi__gb0f282f4xe752c2be.c:577:
> > error: ?SQLITE_DROP_VTABLE? undeclared (first use in this function)
>
> Look a bit above (in the <snip> part :-) what the first error message
> produced by GCC is.  It may likely be: cannot find #include
> <sqlite3.h>.  Either that or you have a broken or very out-of-date
> version of sqlite3.h around.  Either way I don't really understand,
> given that you said it's not the case.  For example in my old
> sqlite3.h there is a line "#define SQLITE_DROP_VTABLE 30" which should
> prevent the above error.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130603/ef440e7c/attachment-0001.html>

From arigo at tunes.org  Mon Jun  3 10:00:21 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Jun 2013 10:00:21 +0200
Subject: [pypy-dev] cpython changed _sre between 2.7.3 to 2.7.4,
 now intmask(pattern) is negative
In-Reply-To: <51ABB1E9.4080305@gmail.com>
References: <51ABB1E9.4080305@gmail.com>
Message-ID: <CAMSv6X3DPY3uPURCV2kj5BhWVMMk_k6ANixyVeD8CwtuGWEKEA@mail.gmail.com>

Hi Matti,

On Sun, Jun 2, 2013 at 10:58 PM, Matti Picus <matti.picus at gmail.com> wrote:
> Running own tests on win32 gives many errors in the _sre module:
> rlib/rsre/rsre_core.py:111: AssertionError
> which is checking that all values of the pattern are >=0

Is this only about the untranslated tests?  If so that's easily fixed,
but I wonder if we're also going to hit the same issues if we upgrade
our stdlib to that of CPython 2.7.4.


A bient?t,

Armin.

From arigo at tunes.org  Mon Jun  3 10:15:04 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Jun 2013 10:15:04 +0200
Subject: [pypy-dev] cpython changed _sre between 2.7.3 to 2.7.4,
 now intmask(pattern) is negative
In-Reply-To: <CAMSv6X3DPY3uPURCV2kj5BhWVMMk_k6ANixyVeD8CwtuGWEKEA@mail.gmail.com>
References: <51ABB1E9.4080305@gmail.com>
	<CAMSv6X3DPY3uPURCV2kj5BhWVMMk_k6ANixyVeD8CwtuGWEKEA@mail.gmail.com>
Message-ID: <CAMSv6X3djLhArqfrYw_0x7ss+wkxaiVCT7Xr_s7NUcskbQqQYw@mail.gmail.com>

Re-Hi,

On Mon, Jun 3, 2013 at 10:00 AM, Armin Rigo <arigo at tunes.org> wrote:
> On Sun, Jun 2, 2013 at 10:58 PM, Matti Picus <matti.picus at gmail.com> wrote:
>> Running own tests on win32 gives many errors in the _sre module:
>> rlib/rsre/rsre_core.py:111: AssertionError
>> which is checking that all values of the pattern are >=0

Ok, it's going to be tons of fun.  CPython changed in c1b3d25882ca a
detail in both the pure Python sre compiler and in the _sre.c module.
This means that now regexps created with Python 2.7.3 don't work with
Python 2.7.4's built-in _sre module and vice-versa.  Argh!  Of course
PyPy still has a 2.7.3-compatible _sre module.  So the own tests need
to be skipped when we are running CPython 2.7.4 or higher.
Conversely, once we do the upgrade of PyPy to 2.7.4, they will need to
be skipped if running CPython 2.7.3 or lower.

The alternative is to include the complete sre*.py from CPython 2.7.3
into PyPy's rpython/rlib/rsre/, for the purposes of running the tests
--- but also for the purposes of some other non-PyPy RPython
interpreter that uses rsre to get regexps at translation time.  This
means basically fixing get_hacked_sre_compile() in rlib/rsre/rpy.py to
not include the standard module's sre_compile and sre_constants, but
instead our own version of it.


A bient?t,

Armin.

From svpcom at gmail.com  Mon Jun  3 12:08:57 2013
From: svpcom at gmail.com (Vasily Evseenko)
Date: Mon, 03 Jun 2013 14:08:57 +0400
Subject: [pypy-dev] JIT slowdown on creating types
Message-ID: <51AC6B39.3060703@gmail.com>

Hi,

When i dynamically create new type (for example classes nested to
functions) I've got linear code slowdown:

-----
def test():
    obj = type('foo' , (object,), {'attr1': 0})()

    for i in xrange(1000000):
        obj.attr1 += 1


for j in xrange(100):
    ts = time.time()
    test()
    print time.time() - ts
------
# pypy ./test.py
0.0529999732971
0.0519850254059
0.0545370578766
0.0622198581696
...
0.678023099899
0.686202049255
0.68180680275
0.703506946564

[PyPy 2.0.2 with GCC 4.4.5] on linux2

Is it bug or a feature?

From tbaldridge at gmail.com  Mon Jun  3 20:55:20 2013
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Mon, 3 Jun 2013 12:55:20 -0600
Subject: [pypy-dev] Best FFI interface
Message-ID: <CAL36E+sKEFAugw9FJZE-9A+aHsm5fj-p6GNPpvYmx=Grgb2QRg@mail.gmail.com>

I'm attempting to integrate my rpython interpreter with libgmp, and I'm a
bit confused in how to call the library from my rpython code. it seems we
have three ways?

rlib.libffi (deprecated?)
rlib.jit_libffi
rlib.clibffi

Which of these is prefered for a from-scratch rpython interpreter? In
addition, I've looked at the code for each of these and it's not terribly
clear how to go about something like this (in psudeocode):

mpz = malloc(sizeof(size_t) * 3)
__gmpz_init_set_si(mpz, (signed long)val)

the struct passed around in mpz can be opaque to the interpreter as it will
only ever be access via gmp functions.

Thanks for the help,

Timothy Baldridge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130603/1f7dd168/attachment.html>

From arigo at tunes.org  Mon Jun  3 22:11:03 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Jun 2013 22:11:03 +0200
Subject: [pypy-dev] Best FFI interface
In-Reply-To: <CAL36E+sKEFAugw9FJZE-9A+aHsm5fj-p6GNPpvYmx=Grgb2QRg@mail.gmail.com>
References: <CAL36E+sKEFAugw9FJZE-9A+aHsm5fj-p6GNPpvYmx=Grgb2QRg@mail.gmail.com>
Message-ID: <CAMSv6X2gGLdq-odOfk-CtsGVqp04mpf1PwdDrmQwaq8hA5hHTA@mail.gmail.com>

Hi Timothy,

On Mon, Jun 3, 2013 at 8:55 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> I'm attempting to integrate my rpython interpreter with libgmp, and I'm a
> bit confused in how to call the library from my rpython code. it seems we
> have three ways?
>
> rlib.libffi (deprecated?)
> rlib.jit_libffi
> rlib.clibffi

You're looking at the wrong place.  You should instead look at
rpython.rtyper.lltypesystem.rffi, which is minimally documented in
pypy/doc/rffi.rst.  What you list above are ways to invoke libffi,
which is a C library to do dynamic calls.  (Additionally, if the goal
is only to have ints of unlimited size in RPython, there is
rpython.rlib.rbigint.)


A bient?t,

Armin.

From wdct at msn.com  Wed Jun  5 06:32:39 2013
From: wdct at msn.com (cuitao)
Date: Wed, 5 Jun 2013 04:32:39 +0000
Subject: [pypy-dev] pypy-dev Digest, Vol 26, Issue 5
In-Reply-To: <mailman.15.1370340002.14810.pypy-dev@python.org>
References: <mailman.15.1370340002.14810.pypy-dev@python.org>
Message-ID: <BAY157-W23B67555486AF49B679409C19F0@phx.gbl>

hello, can you consider adding the blist support in pypy ?
blist: a list-like type with better asymptotic performance and similar performance on small lists

> From: pypy-dev-request at python.org
> Subject: pypy-dev Digest, Vol 26, Issue 5
> To: pypy-dev at python.org
> Date: Tue, 4 Jun 2013 12:00:02 +0200
> 
> Send pypy-dev mailing list submissions to
> 	pypy-dev at python.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mail.python.org/mailman/listinfo/pypy-dev
> or, via email, send a message with subject or body 'help' to
> 	pypy-dev-request at python.org
> 
> You can reach the person managing the list at
> 	pypy-dev-owner at python.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of pypy-dev digest..."
> 
> 
> Today's Topics:
> 
>    1. JIT slowdown on creating types (Vasily Evseenko)
>    2. Best FFI interface (Timothy Baldridge)
>    3. Re: Best FFI interface (Armin Rigo)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 03 Jun 2013 14:08:57 +0400
> From: Vasily Evseenko <svpcom at gmail.com>
> To: PyPy Developer Mailing List <pypy-dev at python.org>
> Subject: [pypy-dev] JIT slowdown on creating types
> Message-ID: <51AC6B39.3060703 at gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> Hi,
> 
> When i dynamically create new type (for example classes nested to
> functions) I've got linear code slowdown:
> 
> -----
> def test():
>     obj = type('foo' , (object,), {'attr1': 0})()
> 
>     for i in xrange(1000000):
>         obj.attr1 += 1
> 
> 
> for j in xrange(100):
>     ts = time.time()
>     test()
>     print time.time() - ts
> ------
> # pypy ./test.py
> 0.0529999732971
> 0.0519850254059
> 0.0545370578766
> 0.0622198581696
> ...
> 0.678023099899
> 0.686202049255
> 0.68180680275
> 0.703506946564
> 
> [PyPy 2.0.2 with GCC 4.4.5] on linux2
> 
> Is it bug or a feature?
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 3 Jun 2013 12:55:20 -0600
> From: Timothy Baldridge <tbaldridge at gmail.com>
> To: PyPy Developer Mailing List <pypy-dev at python.org>
> Subject: [pypy-dev] Best FFI interface
> Message-ID:
> 	<CAL36E+sKEFAugw9FJZE-9A+aHsm5fj-p6GNPpvYmx=Grgb2QRg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> I'm attempting to integrate my rpython interpreter with libgmp, and I'm a
> bit confused in how to call the library from my rpython code. it seems we
> have three ways?
> 
> rlib.libffi (deprecated?)
> rlib.jit_libffi
> rlib.clibffi
> 
> Which of these is prefered for a from-scratch rpython interpreter? In
> addition, I've looked at the code for each of these and it's not terribly
> clear how to go about something like this (in psudeocode):
> 
> mpz = malloc(sizeof(size_t) * 3)
> __gmpz_init_set_si(mpz, (signed long)val)
> 
> the struct passed around in mpz can be opaque to the interpreter as it will
> only ever be access via gmp functions.
> 
> Thanks for the help,
> 
> Timothy Baldridge
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130603/1f7dd168/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 3
> Date: Mon, 3 Jun 2013 22:11:03 +0200
> From: Armin Rigo <arigo at tunes.org>
> To: Timothy Baldridge <tbaldridge at gmail.com>
> Cc: PyPy Developer Mailing List <pypy-dev at python.org>
> Subject: Re: [pypy-dev] Best FFI interface
> Message-ID:
> 	<CAMSv6X2gGLdq-odOfk-CtsGVqp04mpf1PwdDrmQwaq8hA5hHTA at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> Hi Timothy,
> 
> On Mon, Jun 3, 2013 at 8:55 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> > I'm attempting to integrate my rpython interpreter with libgmp, and I'm a
> > bit confused in how to call the library from my rpython code. it seems we
> > have three ways?
> >
> > rlib.libffi (deprecated?)
> > rlib.jit_libffi
> > rlib.clibffi
> 
> You're looking at the wrong place.  You should instead look at
> rpython.rtyper.lltypesystem.rffi, which is minimally documented in
> pypy/doc/rffi.rst.  What you list above are ways to invoke libffi,
> which is a C library to do dynamic calls.  (Additionally, if the goal
> is only to have ints of unlimited size in RPython, there is
> rpython.rlib.rbigint.)
> 
> 
> A bient?t,
> 
> Armin.
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 
> 
> ------------------------------
> 
> End of pypy-dev Digest, Vol 26, Issue 5
> ***************************************
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130605/a2d7c5d0/attachment.html>

From fijall at gmail.com  Wed Jun  5 07:26:51 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 5 Jun 2013 13:26:51 +0800
Subject: [pypy-dev] fastjson module
Message-ID: <CAK5idxQrpkGZD6iP7BwDfRkcxbRJJeWgyt0BwuAaWpyBt4ckmQ@mail.gmail.com>

Hi anto

if this is for speeding up json, call the module _json, as CPython
does (I don't care if the API is similar or not, as long as it's used
by json lib)

If it's not for json, then it does not belong in PyPy IMO.

Cheers,
fijal

From njh at njhurst.com  Wed Jun  5 08:30:16 2013
From: njh at njhurst.com (Nathan Hurst)
Date: Wed, 5 Jun 2013 16:30:16 +1000
Subject: [pypy-dev] blist, was Re:  pypy-dev Digest, Vol 26, Issue 5
In-Reply-To: <BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
References: <mailman.15.1370340002.14810.pypy-dev@python.org>
	<BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
Message-ID: <20130605063016.GA4604@ajhurst.org>

On Wed, Jun 05, 2013 at 04:32:39AM +0000, cuitao wrote:
> hello, can you consider adding the blist support in pypy ?
> blist: a list-like type with better asymptotic performance and similar performance on small lists

blist is written in C though, and pypy is written in python.  Perhaps
you would like to port it to python?

njh

From anto.cuni at gmail.com  Wed Jun  5 10:58:17 2013
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 05 Jun 2013 10:58:17 +0200
Subject: [pypy-dev] fastjson module
In-Reply-To: <CAK5idxQrpkGZD6iP7BwDfRkcxbRJJeWgyt0BwuAaWpyBt4ckmQ@mail.gmail.com>
References: <CAK5idxQrpkGZD6iP7BwDfRkcxbRJJeWgyt0BwuAaWpyBt4ckmQ@mail.gmail.com>
Message-ID: <51AEFDA9.8040308@gmail.com>

Hi,

On 06/05/2013 07:26 AM, Maciej Fijalkowski wrote:
> Hi anto
>
> if this is for speeding up json, call the module _json, as CPython
> does (I don't care if the API is similar or not, as long as it's used
> by json lib)


yes, it is for speeding up json, but I called it differently precisely because 
the API is different (basically, my _fastjson offers a drop-in replacement for 
"json.loads", while cpython's _json offers only some of the functions to 
implement it)

I think it's better to name it differently because I can imagine there are 
programs around which do things like "from _json import scanstring" for their 
own purposes, and they would be broken by our _json-which-is-not-really-_json 
module.

What do the others think?

From fijall at gmail.com  Wed Jun  5 12:16:13 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 5 Jun 2013 18:16:13 +0800
Subject: [pypy-dev] fastjson module
In-Reply-To: <51AEFDA9.8040308@gmail.com>
References: <CAK5idxQrpkGZD6iP7BwDfRkcxbRJJeWgyt0BwuAaWpyBt4ckmQ@mail.gmail.com>
	<51AEFDA9.8040308@gmail.com>
Message-ID: <CAK5idxS7YLvTuyfSo68yauDEznOM23wUM8TOYMCo5Ai3B-c3ZQ@mail.gmail.com>

On Wed, Jun 5, 2013 at 4:58 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Hi,
>
>
> On 06/05/2013 07:26 AM, Maciej Fijalkowski wrote:
>>
>> Hi anto
>>
>> if this is for speeding up json, call the module _json, as CPython
>> does (I don't care if the API is similar or not, as long as it's used
>> by json lib)
>
>
>
> yes, it is for speeding up json, but I called it differently precisely
> because the API is different (basically, my _fastjson offers a drop-in
> replacement for "json.loads", while cpython's _json offers only some of the
> functions to implement it)
>
> I think it's better to name it differently because I can imagine there are
> programs around which do things like "from _json import scanstring" for
> their own purposes, and they would be broken by our
> _json-which-is-not-really-_json module.
>
> What do the others think?

maybe they should stop doing that instead ;-) (note that from json
import scanstring also works and in fact is preferred. from _json
import scanstring would fail on pypy regardless)

From phyo.arkarlwin at gmail.com  Wed Jun  5 13:03:15 2013
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Wed, 5 Jun 2013 17:33:15 +0630
Subject: [pypy-dev] blist, was Re: pypy-dev Digest, Vol 26, Issue 5
In-Reply-To: <20130605063016.GA4604@ajhurst.org>
References: <mailman.15.1370340002.14810.pypy-dev@python.org>
	<BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
	<20130605063016.GA4604@ajhurst.org>
Message-ID: <CA+HjJrgsvWB0wc0tFg2DNC+XYMAimKW92w_6Re_-NmNwK1-gVA@mail.gmail.com>

or write CFFI version?


On Wed, Jun 5, 2013 at 1:00 PM, Nathan Hurst <njh at njhurst.com> wrote:

> On Wed, Jun 05, 2013 at 04:32:39AM +0000, cuitao wrote:
> > hello, can you consider adding the blist support in pypy ?
> > blist: a list-like type with better asymptotic performance and similar
> performance on small lists
>
> blist is written in C though, and pypy is written in python.  Perhaps
> you would like to port it to python?
>
> njh
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130605/ad072614/attachment.html>

From andrewsmedina at gmail.com  Wed Jun  5 14:47:35 2013
From: andrewsmedina at gmail.com (Andrews Medina)
Date: Wed, 5 Jun 2013 09:47:35 -0300
Subject: [pypy-dev] comparing strings using is does not works
Message-ID: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>

Hi,

In the code below:

>>>> x = "a"
>>>> b = "a"
>>>> x is b
False

It is a bug or a behaviour?

Thanks.

--
Andrews Medina
www.andrewsmedina.com

From skip at pobox.com  Wed Jun  5 14:54:10 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 5 Jun 2013 07:54:10 -0500
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
Message-ID: <CANc-5UxR2=p0Rx4RkHjzWUUZ5=ARquJWHZB_qcfugY0E0YyVkw@mail.gmail.com>

On Wed, Jun 5, 2013 at 7:47 AM, Andrews Medina <andrewsmedina at gmail.com> wrote:
> Hi,
>
> In the code below:
>
>>>> x = "a"
>>>> b = "a"
>>>> x is b
> False
>
> It is a bug or a behaviour?

Behavior.  Interning strings so different bindings point to the same
memory location is an optimization.  There is no guarantee that any
given Python implementation (or even the same implementation in
different versions) will do what you expect.  FWIW, Python 2.7 seems
to "get it right" (for certain user-defined definition of "right"):

>>> x = "a"
>>> y = "a"
>>> id(x)
135958064
>>> id(y)
135958064
>>> x is y
True

Skip

From sparks.m at gmail.com  Wed Jun  5 14:54:51 2013
From: sparks.m at gmail.com (Michael)
Date: Wed, 5 Jun 2013 13:54:51 +0100
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
Message-ID: <CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>

On 5 June 2013 13:47, Andrews Medina <andrewsmedina at gmail.com> wrote:

> Hi,
>
> In the code below:
>
> >>>> x = "a"
> >>>> b = "a"
> >>>> x is b
> False
>

It's more consistent than cpython - where the result of "x is b" is
dependent on the length of the string :

Python 2.7.3 (default, Apr 10 2013, 05:46:21)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> x="a"*20
>>> b="a"*20
>>> x is b
True
>>> x="a"*21
>>> b="a"*21
>>> x is b
False

Relying on "is" for comparing strings is generally a bug in your code
waiting to be triggered.


Michael.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130605/61c176c6/attachment.html>

From svpcom at gmail.com  Wed Jun  5 14:54:53 2013
From: svpcom at gmail.com (Vasily Evseenko)
Date: Wed, 05 Jun 2013 16:54:53 +0400
Subject: [pypy-dev] Measure time inside of JIT'ed code
Message-ID: <51AF351D.4060606@gmail.com>

Hi,

I want to profile my program and calculate which portion of code is
JIT'ed and what it interpreted.
Is there any way to dump into PYPYLOG entries when JIT'ed code sections
running?
I found that only JIT tracing and generation events are logged.

From arigo at tunes.org  Wed Jun  5 15:28:24 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 5 Jun 2013 15:28:24 +0200
Subject: [pypy-dev] pypy-dev Digest, Vol 26, Issue 5
In-Reply-To: <BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
References: <mailman.15.1370340002.14810.pypy-dev@python.org>
	<BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
Message-ID: <CAMSv6X0yeSkwKWE3NZ7cFNhWZJ9R9-60H3pes3eoY+sodAOCpA@mail.gmail.com>

Hi,

On Wed, Jun 5, 2013 at 6:32 AM, cuitao <wdct at msn.com> wrote:
> hello, can you consider adding the blist support in pypy ?

I take it you are talking about https://pypi.python.org/pypi/blist/ .
Sorry, the PyPy team is a finite small number of people, so we cannot
port all C extension modules out there.  Please consider if using it
over cpyext gives enough performance (but I doubt it, in this case).
Otherwise, you need to find someone who is willing to port it from a C
extension to become a regular pure Python module, which should
hopefully not be too complicated.


A bient?t,

Armin.

From arigo at tunes.org  Wed Jun  5 15:31:42 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 5 Jun 2013 15:31:42 +0200
Subject: [pypy-dev] Measure time inside of JIT'ed code
In-Reply-To: <51AF351D.4060606@gmail.com>
References: <51AF351D.4060606@gmail.com>
Message-ID: <CAMSv6X3w4AnFNbqFA_m=sBREBDQ+Fftw24Wtars2s=Uhmp2+2A@mail.gmail.com>

Hi,

On Wed, Jun 5, 2013 at 2:54 PM, Vasily Evseenko <svpcom at gmail.com> wrote:
> I want to profile my program and calculate which portion of code is
> JIT'ed and what it interpreted.

You can use callgrind/cachegrind to get a rough estimate.  But it's
unclear to me the reason you are interested in this particular piece
of data.  Basically any code that runs repeatedly in the interpreter
is eventually going to be JITted.


A bient?t,

Armin.

From amauryfa at gmail.com  Wed Jun  5 18:30:49 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 5 Jun 2013 18:30:49 +0200
Subject: [pypy-dev] blist, was Re: pypy-dev Digest, Vol 26, Issue 5
In-Reply-To: <CA+HjJrgsvWB0wc0tFg2DNC+XYMAimKW92w_6Re_-NmNwK1-gVA@mail.gmail.com>
References: <mailman.15.1370340002.14810.pypy-dev@python.org>
	<BAY157-W23B67555486AF49B679409C19F0@phx.gbl>
	<20130605063016.GA4604@ajhurst.org>
	<CA+HjJrgsvWB0wc0tFg2DNC+XYMAimKW92w_6Re_-NmNwK1-gVA@mail.gmail.com>
Message-ID: <CAGmFidYTBJ78QvZjmTeHpOUJurYYsOZ3j0mmrFxK85=svMJxcw@mail.gmail.com>

2013/6/5 Phyo Arkar <phyo.arkarlwin at gmail.com>

> or write CFFI version?


Unfortunately there is no library to wrap.
All the C code is deeply integrated with the CPython API.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130605/d6cc49b6/attachment.html>

From fijall at gmail.com  Thu Jun  6 05:24:18 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Jun 2013 11:24:18 +0800
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
	<CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>
Message-ID: <CAK5idxQFhFPEhs=7Q5E+apdbKfNtrh2j7UMim+-RUEQ7D8AGrA@mail.gmail.com>

it *also* depends where the string comes from. I think encodings don't
respect caching small strings for example

On Wed, Jun 5, 2013 at 8:54 PM, Michael <sparks.m at gmail.com> wrote:
>
> On 5 June 2013 13:47, Andrews Medina <andrewsmedina at gmail.com> wrote:
>>
>> Hi,
>>
>> In the code below:
>>
>> >>>> x = "a"
>> >>>> b = "a"
>> >>>> x is b
>> False
>
>
> It's more consistent than cpython - where the result of "x is b" is
> dependent on the length of the string :
>
> Python 2.7.3 (default, Apr 10 2013, 05:46:21)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> x="a"*20
>>>> b="a"*20
>>>> x is b
> True
>>>> x="a"*21
>>>> b="a"*21
>>>> x is b
> False
>
> Relying on "is" for comparing strings is generally a bug in your code
> waiting to be triggered.
>
>
> Michael.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From amauryfa at gmail.com  Thu Jun  6 08:09:49 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 6 Jun 2013 08:09:49 +0200
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAK5idxQFhFPEhs=7Q5E+apdbKfNtrh2j7UMim+-RUEQ7D8AGrA@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
	<CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>
	<CAK5idxQFhFPEhs=7Q5E+apdbKfNtrh2j7UMim+-RUEQ7D8AGrA@mail.gmail.com>
Message-ID: <CAGmFidb_8qXDKWxTLKJ9yA=euQFRfJZbg_05_E6V+jXK6ffj=w@mail.gmail.com>

2013/6/6 Maciej Fijalkowski <fijall at gmail.com>

> it *also* depends where the string comes from. I think encodings don't
> respect caching small strings for example


It's really a detail of the C implementation:
PyString_FromString() returns a prebuilt value when the length is 0 or 1,
but _PyString_Resize() won't replace the string being built with a shared
object.
Moreover those details completely changed with Python 3.3 and the new
Unicode implementation.

Don't use "is" with immutable objects (except with the singletons: None,
True, False)

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130606/6ae3020b/attachment.html>

From william.leslie.ttg at gmail.com  Thu Jun  6 08:14:22 2013
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Thu, 6 Jun 2013 16:14:22 +1000
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAGmFidb_8qXDKWxTLKJ9yA=euQFRfJZbg_05_E6V+jXK6ffj=w@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
	<CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>
	<CAK5idxQFhFPEhs=7Q5E+apdbKfNtrh2j7UMim+-RUEQ7D8AGrA@mail.gmail.com>
	<CAGmFidb_8qXDKWxTLKJ9yA=euQFRfJZbg_05_E6V+jXK6ffj=w@mail.gmail.com>
Message-ID: <CAHgd1hFPZD2QJ33iibxHjRHtPJjeJdEAqhj0n5cUQkGVqpNgzw@mail.gmail.com>

On 6 June 2013 16:09, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Don't use "is" with immutable objects (except with the singletons: None,
> True, False)

Never with True or False either.  It's required to work by the
language definition, but it's still nonsense.

--
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely may reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to deny you those rights would be illegal without
prior contractual agreement.

From arigo at tunes.org  Thu Jun  6 09:55:13 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Jun 2013 09:55:13 +0200
Subject: [pypy-dev] comparing strings using is does not works
In-Reply-To: <CAHgd1hFPZD2QJ33iibxHjRHtPJjeJdEAqhj0n5cUQkGVqpNgzw@mail.gmail.com>
References: <CAGpeaCmsJiTscHf=HAzT0u8BP=00rxr6LOqHvSywXobQKTiJZw@mail.gmail.com>
	<CAB+QZVANahuL7X6JvQOcLdLvf5DXgu3jxk9hbEoxzqWnXm7_pQ@mail.gmail.com>
	<CAK5idxQFhFPEhs=7Q5E+apdbKfNtrh2j7UMim+-RUEQ7D8AGrA@mail.gmail.com>
	<CAGmFidb_8qXDKWxTLKJ9yA=euQFRfJZbg_05_E6V+jXK6ffj=w@mail.gmail.com>
	<CAHgd1hFPZD2QJ33iibxHjRHtPJjeJdEAqhj0n5cUQkGVqpNgzw@mail.gmail.com>
Message-ID: <CAMSv6X18juJ9TcmTJY4AeDtLah2=SyjMWk4C=9qJQ6KS+jR4uA@mail.gmail.com>

Hi William,

On Thu, Jun 6, 2013 at 8:14 AM, William ML Leslie
<william.leslie.ttg at gmail.com> wrote:
> On 6 June 2013 16:09, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
>> Don't use "is" with immutable objects (except with the singletons: None,
>> True, False)
>
> Never with True or False either.  It's required to work by the
> language definition, but it's still nonsense.

You can use "x is True" or "x is False", but it means really that x
must not be e.g. an integer 0 or 1 but really the boolean you test
for.  In this sense, it's not most useful but not wrong.  But any
place where "x is 3" appears is bogus.  I vaguely suggested at some
point to emit a SyntaxWarning for these cases...


A bient?t,

Armin.

From rrossi at cimne.upc.edu  Thu Jun  6 16:15:13 2013
From: rrossi at cimne.upc.edu (Riccardo Rossi)
Date: Thu, 6 Jun 2013 16:15:13 +0200
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
References: <1597854793122323073@unknownmsgid>
	<CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
Message-ID: <CAOVdALxOx9M=KKnFEYWN_CWfqzR27N7ON5Pi-f70kR5bzt9SqA@mail.gmail.com>

Hi Armin,
        thx for the answer (shall i say, merci beaucoup? :-) )

Riccardo


On Thu, Jun 6, 2013 at 3:30 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Riccardo,
>
> On Sat, Jun 1, 2013 at 11:59 PM, Riccardo Rossi <rougered4 at gmail.com>
> wrote:
> >               I am writing since I would like to understand if it is
> > possible to use pypy together with modulus and types exported by
> > boost_python.
>
> I am sorry not to be able to answer you.  Hopefully others here have a
> better idea about boost_python than what I do.  Hi there, please speak
> up if you do :-)
>
>
> A bient?t,
>
> Armin.
>



-- 

Dr. Riccardo Rossi, Civil Engineer

Member of Kratos Team

International Center for Numerical Methods in Engineering - CIMNE
Campus Norte, Edificio C1

c/ Gran Capit?n s/n

08034 Barcelona, Espa?a

Tel:        (+34) 93 401 56 96

Fax:       (+34) 93.401.6517
web:       www.cimne.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130606/50a096bc/attachment.html>

From arigo at tunes.org  Thu Jun  6 15:30:04 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Jun 2013 15:30:04 +0200
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <1597854793122323073@unknownmsgid>
References: <1597854793122323073@unknownmsgid>
Message-ID: <CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>

Hi Riccardo,

On Sat, Jun 1, 2013 at 11:59 PM, Riccardo Rossi <rougered4 at gmail.com> wrote:
>               I am writing since I would like to understand if it is
> possible to use pypy together with modulus and types exported by
> boost_python.

I am sorry not to be able to answer you.  Hopefully others here have a
better idea about boost_python than what I do.  Hi there, please speak
up if you do :-)


A bient?t,

Armin.

From amauryfa at gmail.com  Thu Jun  6 16:35:44 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 6 Jun 2013 16:35:44 +0200
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
References: <1597854793122323073@unknownmsgid>
	<CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
Message-ID: <CAGmFidanWepR6BgfMcgmcGdStABZ9m583V_NObCyxXzO1-0rkw@mail.gmail.com>

Hi Riccardo,

2013/6/6 Armin Rigo <arigo at tunes.org>

> On Sat, Jun 1, 2013 at 11:59 PM, Riccardo Rossi <rougered4 at gmail.com>
> wrote:
> >               I am writing since I would like to understand if it is
> > possible to use pypy together with modulus and types exported by
> > boost_python.
>
> I am sorry not to be able to answer you.  Hopefully others here have a
> better idea about boost_python than what I do.  Hi there, please speak
> up if you do :-)
>

There was some work a while ago to support boost::python in PyPy.
In any case you have to recompile the boost::python library with pypy
headers
(and .lib on Windows) and your C++ code as well.

I can't remember the details, but I know modified some user-config.jam file
(or was it site-config.jam?) to set the correct Python paths.
And be sure to name your extension module foo.pypy-20.so (or
foo.pypy-20.pyd on Windows)

Then, the usual warnings about cpyext apply:
The interface between C/C++ and Python is very slow,
and not all of the CPython API is implemented.
But it did pass a lot of the test suite.

Sorry I don't have any turnkey solution.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130606/3923206c/attachment.html>

From rrossi at cimne.upc.edu  Thu Jun  6 16:51:54 2013
From: rrossi at cimne.upc.edu (Riccardo Rossi)
Date: Thu, 6 Jun 2013 16:51:54 +0200
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <CAGmFidanWepR6BgfMcgmcGdStABZ9m583V_NObCyxXzO1-0rkw@mail.gmail.com>
References: <1597854793122323073@unknownmsgid>
	<CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
	<CAGmFidanWepR6BgfMcgmcGdStABZ9m583V_NObCyxXzO1-0rkw@mail.gmail.com>
Message-ID: <CAOVdALypcJhdGwWFR+vrVdsFKYJRzZGE9gNzQx_nyV5GLgfNfQ@mail.gmail.com>

Dear Amaury,
         first of all thank you for your answer. I think i could hack the
boost to compile with a different python, however your last warning is
actuall the answer i was interested in getting.

if you tell me that the interface will be actually slower than in cpython,
than most probably it will not fit my needs.

i will keep monitoring your project (i really believe you do a very
interesting job) in the hope the link to C/C++ will get better (or that
boost_python will be ported based on cffi)

thank you
Riccardo


On Thu, Jun 6, 2013 at 4:35 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> Hi Riccardo,
>
> 2013/6/6 Armin Rigo <arigo at tunes.org>
>
>> On Sat, Jun 1, 2013 at 11:59 PM, Riccardo Rossi <rougered4 at gmail.com>
>> wrote:
>> >               I am writing since I would like to understand if it is
>> > possible to use pypy together with modulus and types exported by
>> > boost_python.
>>
>> I am sorry not to be able to answer you.  Hopefully others here have a
>> better idea about boost_python than what I do.  Hi there, please speak
>> up if you do :-)
>>
>
> There was some work a while ago to support boost::python in PyPy.
> In any case you have to recompile the boost::python library with pypy
> headers
> (and .lib on Windows) and your C++ code as well.
>
> I can't remember the details, but I know modified some user-config.jam file
> (or was it site-config.jam?) to set the correct Python paths.
> And be sure to name your extension module foo.pypy-20.so (or
> foo.pypy-20.pyd on Windows)
>
> Then, the usual warnings about cpyext apply:
> The interface between C/C++ and Python is very slow,
> and not all of the CPython API is implemented.
> But it did pass a lot of the test suite.
>
> Sorry I don't have any turnkey solution.
>
> --
> Amaury Forgeot d'Arc
>



-- 

Dr. Riccardo Rossi, Civil Engineer

Member of Kratos Team

International Center for Numerical Methods in Engineering - CIMNE
Campus Norte, Edificio C1

c/ Gran Capit?n s/n

08034 Barcelona, Espa?a

Tel:        (+34) 93 401 56 96

Fax:       (+34) 93.401.6517
web:       www.cimne.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130606/c4ce113c/attachment-0001.html>

From wlavrijsen at lbl.gov  Thu Jun  6 17:25:22 2013
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 6 Jun 2013 08:25:22 -0700 (PDT)
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <CAOVdALypcJhdGwWFR+vrVdsFKYJRzZGE9gNzQx_nyV5GLgfNfQ@mail.gmail.com>
References: <1597854793122323073@unknownmsgid>
	<CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
	<CAGmFidanWepR6BgfMcgmcGdStABZ9m583V_NObCyxXzO1-0rkw@mail.gmail.com>
	<CAOVdALypcJhdGwWFR+vrVdsFKYJRzZGE9gNzQx_nyV5GLgfNfQ@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1306060824020.5810@pc3a-wlav.lbl.gov>

Hi,

On Thu, 6 Jun 2013, Riccardo Rossi wrote:
> i will keep monitoring your project (i really believe you do a very
> interesting job) in the hope the link to C/C++ will get better (or that
> boost_python will be ported based on cffi)

for general C++ (i.e. if the boost::python portion does not offer further
functionality other than the binding), you can use:

   http://doc.pypy.org/en/latest/cppyy.html

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From amauryfa at gmail.com  Fri Jun  7 10:59:17 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 7 Jun 2013 10:59:17 +0200
Subject: [pypy-dev] Compatibilidad with boost python
In-Reply-To: <CAOVdALypcJhdGwWFR+vrVdsFKYJRzZGE9gNzQx_nyV5GLgfNfQ@mail.gmail.com>
References: <1597854793122323073@unknownmsgid>
	<CAMSv6X3WxtmDJ7K8Y53kbWYykbL9aSAyAPuvDnkR=+F6arKNbA@mail.gmail.com>
	<CAGmFidanWepR6BgfMcgmcGdStABZ9m583V_NObCyxXzO1-0rkw@mail.gmail.com>
	<CAOVdALypcJhdGwWFR+vrVdsFKYJRzZGE9gNzQx_nyV5GLgfNfQ@mail.gmail.com>
Message-ID: <CAGmFidYnDbTkVt6hSmkAzQ2zq1h9rHNHQ3Y=Ckty3rBRBW52WA@mail.gmail.com>

2013/6/6 Riccardo Rossi <rrossi at cimne.upc.edu>

> Dear Amaury,
>          first of all thank you for your answer. I think i could hack the
> boost to compile with a different python, however your last warning is
> actuall the answer i was interested in getting.
>
> if you tell me that the interface will be actually slower than in cpython,
> than most probably it will not fit my needs.
>
> i will keep monitoring your project (i really believe you do a very
> interesting job) in the hope the link to C/C++ will get better (or that
> boost_python will be ported based on cffi)
>

There are fundamental reasons why the CPython API is slow with PyPy
(one of them is that PyPy objects move in memory, so cannot be represented
as simple pointers)

Porting boost::python to cffi is... a challenging idea.
First, a cffi module is written in Python, when a boost::python module is
C++ code.
Then, cffi is currently designed to pass only simple types: int, float,
pointers, struct.
No C++ class instance, and no Python object.

A boost::python::object could be emulated with the help of ffi.new_handle(),
but I don't know whether this integrates easily with C++ memory management
(the "handle" object
needs to stay alive, and this can only be done from Python code)

It's worth exploring though -- and no specific PyPy knowledge is needed for
this.
Help is welcome!

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130607/594bd2ae/attachment.html>

From abrenon at wyplay.com  Fri Jun  7 16:33:49 2013
From: abrenon at wyplay.com (Alexis BRENON)
Date: Fri, 07 Jun 2013 16:33:49 +0200
Subject: [pypy-dev] BlockBuilderMixin.get_relative_pos()
Message-ID: <51B1EF4D.6040800@wyplay.com>

Hi,

I see in the ARM tests many call to InstructionBuilder.curpos() which 
returns BlockBuilderMixin.get_relative_pos(). Then, the value returned 
is used to make some branches and some jumps.
I try to do the same for my MIPS port, but all my branch and jump fail. 
When I print the value returned I get value under 1000 that let me think 
that it's not a PC value.

Can anyone explain me what's the value returned by 
BlockBuilderMixin.get_relative_pos().

Friendly,
Alexis BRENON

From roberto at unbit.it  Fri Jun  7 18:25:16 2013
From: roberto at unbit.it (Roberto De Ioris)
Date: Fri, 7 Jun 2013 18:25:16 +0200
Subject: [pypy-dev] uWSGI PyPy improvements and support for continulets (and
	greenlets)
Message-ID: <b9e7760684498fc0bb86b193185584ae.squirrel@manage.unbit.it>


Hi everyone, the current development tree of uWSGI includes a new set of
patches aimed at improving performances of the pypy plugin.

The new code removes the need of using c helpers for calling a bunch of
uWSGI api functions, the improvements is about 5% in respect to the
current stable plugin.

In addition to this, support for continulets have been added:

https://github.com/unbit/uwsgi/blob/master/plugins/pypy/pypy_setup.py#L725

that means you can use the async/nonblocking (raw) api of uWSGI:

https://github.com/unbit/uwsgi/tree/master/t/pypy

Finally (just as a proof of concept) i have added greenlets support too
(this time in an external file you can load on startup):

https://github.com/unbit/uwsgi/blob/master/contrib/pypy/uwsgi_pypy_greenlets.py


I would like to point the fact that this is a really "extreme" usage of an
interpreter and PyPy is surviving to basically all of the tests i have
made
(for example the uwsgi internal routing subsystem that is completely in c
is automatically wrapped to a continulet when the server is configured in
async mode)

As always many thanks to Maciej Fijalkowski for his help

-- 
Roberto De Ioris
http://unbit.it

From fijall at gmail.com  Sat Jun  8 03:41:46 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 8 Jun 2013 09:41:46 +0800
Subject: [pypy-dev] BlockBuilderMixin.get_relative_pos()
In-Reply-To: <51B1EF4D.6040800@wyplay.com>
References: <51B1EF4D.6040800@wyplay.com>
Message-ID: <CAK5idxTqY0-RVm4f4bv01tVBnicHNZsMKxyasZR-JQTZtBvOxQ@mail.gmail.com>

On Fri, Jun 7, 2013 at 10:33 PM, Alexis BRENON <abrenon at wyplay.com> wrote:
> Hi,
>
> I see in the ARM tests many call to InstructionBuilder.curpos() which
> returns BlockBuilderMixin.get_relative_pos(). Then, the value returned is
> used to make some branches and some jumps.
> I try to do the same for my MIPS port, but all my branch and jump fail. When
> I print the value returned I get value under 1000 that let me think that
> it's not a PC value.
>
> Can anyone explain me what's the value returned by
> BlockBuilderMixin.get_relative_pos().

it's a relative pos to the start of the block and it's used for
relative jumps. if you want to do absolute jumps, you need to patch
them later using the value that's returned from materialize() (that
writes it to memory). At least x86 backend has some jump patching done

Cheers,
fijal

From santagada at gmail.com  Sat Jun  8 18:29:13 2013
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 8 Jun 2013 13:29:13 -0300
Subject: [pypy-dev] uWSGI PyPy improvements and support for continulets
 (and greenlets)
In-Reply-To: <b9e7760684498fc0bb86b193185584ae.squirrel@manage.unbit.it>
References: <b9e7760684498fc0bb86b193185584ae.squirrel@manage.unbit.it>
Message-ID: <CA+HFc87MqFirXaAnAm2HnYytCuGKj+bw0P7+p1jBugLuL_+jBg@mail.gmail.com>

On Fri, Jun 7, 2013 at 1:25 PM, Roberto De Ioris <roberto at unbit.it> wrote:

>
> Hi everyone, the current development tree of uWSGI includes a new set of
> patches aimed at improving performances of the pypy plugin.
>
> The new code removes the need of using c helpers for calling a bunch of
> uWSGI api functions, the improvements is about 5% in respect to the
> current stable plugin.
>
> In addition to this, support for continulets have been added:
>
> https://github.com/unbit/uwsgi/blob/master/plugins/pypy/pypy_setup.py#L725
>
> that means you can use the async/nonblocking (raw) api of uWSGI:
>
> https://github.com/unbit/uwsgi/tree/master/t/pypy
>
> Finally (just as a proof of concept) i have added greenlets support too
> (this time in an external file you can load on startup):
>
>
> https://github.com/unbit/uwsgi/blob/master/contrib/pypy/uwsgi_pypy_greenlets.py
>
>
> I would like to point the fact that this is a really "extreme" usage of an
> interpreter and PyPy is surviving to basically all of the tests i have
> made
> (for example the uwsgi internal routing subsystem that is completely in c
> is automatically wrapped to a continulet when the server is configured in
> async mode)
>
> As always many thanks to Maciej Fijalkowski for his help
>

Awesome


-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130608/6305cac2/attachment.html>

From miglanz at gmail.com  Mon Jun 10 18:28:16 2013
From: miglanz at gmail.com (Kamil Klimkiewicz)
Date: Mon, 10 Jun 2013 18:28:16 +0200
Subject: [pypy-dev] Problems with translating JITted interpreters
Message-ID: <CAAHE_TEkMFa3XqDyUX_ww7yY1AbL9mdEO9w31=x1rX5ZOBoDJw@mail.gmail.com>

Hi guys,

I'm playing with PyPy/RPython toolchain to create JITted interpreters
for some toy languages. Unfortunately I'm getting some strange errors
when performing the actual translation with:

  $ pypy ~/dev/pypy/rpython/bin/rpython -Ojit example4

The file I'm trying to translate is a modified example from the tutorial
from morepypy blog. Please take a look at the 3 files on the following
gist:

https://gist.github.com/miGlanz/5749914

The files in question are:

- example4.py - modified file from the tutorial; I get an error which is
  written in rpython.log
- example4-ok.py - this is a file that get translated correctly - please
  note the only difference is in Tape.advance()

I have more complicated code but I'm getting exactly the same error
there. Could you, please, tell me what I'm doing wrong? I am a total
noob when it comes to PyPy toolchain so please forgive my ignorance.

Thanks in advance.

-- 
Best regards,
Kamil Klimkiewicz

From fijall at gmail.com  Tue Jun 11 00:19:52 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 11 Jun 2013 06:19:52 +0800
Subject: [pypy-dev] Problems with translating JITted interpreters
In-Reply-To: <CAAHE_TEkMFa3XqDyUX_ww7yY1AbL9mdEO9w31=x1rX5ZOBoDJw@mail.gmail.com>
References: <CAAHE_TEkMFa3XqDyUX_ww7yY1AbL9mdEO9w31=x1rX5ZOBoDJw@mail.gmail.com>
Message-ID: <CAK5idxTShA_0QzHvk_U7_Br7fto8NpK4j2e_sRoL8Jm7dozC1Q@mail.gmail.com>

Antonio Cuni was supposed to fix this problem ages ago :)

On Tue, Jun 11, 2013 at 12:28 AM, Kamil Klimkiewicz <miglanz at gmail.com> wrote:
> Hi guys,
>
> I'm playing with PyPy/RPython toolchain to create JITted interpreters
> for some toy languages. Unfortunately I'm getting some strange errors
> when performing the actual translation with:
>
>   $ pypy ~/dev/pypy/rpython/bin/rpython -Ojit example4
>
> The file I'm trying to translate is a modified example from the tutorial
> from morepypy blog. Please take a look at the 3 files on the following
> gist:
>
> https://gist.github.com/miGlanz/5749914
>
> The files in question are:
>
> - example4.py - modified file from the tutorial; I get an error which is
>   written in rpython.log
> - example4-ok.py - this is a file that get translated correctly - please
>   note the only difference is in Tape.advance()
>
> I have more complicated code but I'm getting exactly the same error
> there. Could you, please, tell me what I'm doing wrong? I am a total
> noob when it comes to PyPy toolchain so please forgive my ignorance.
>
> Thanks in advance.
>
> --
> Best regards,
> Kamil Klimkiewicz
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From alex.gaynor at gmail.com  Tue Jun 11 00:29:04 2013
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 10 Jun 2013 17:29:04 -0500
Subject: [pypy-dev] Problems with translating JITted interpreters
In-Reply-To: <CAK5idxTShA_0QzHvk_U7_Br7fto8NpK4j2e_sRoL8Jm7dozC1Q@mail.gmail.com>
References: <CAAHE_TEkMFa3XqDyUX_ww7yY1AbL9mdEO9w31=x1rX5ZOBoDJw@mail.gmail.com>
	<CAK5idxTShA_0QzHvk_U7_Br7fto8NpK4j2e_sRoL8Jm7dozC1Q@mail.gmail.com>
Message-ID: <CAFRnB2UMY19xNqwjr4cSfYLzT5HMJtNgJ-VfDBhF1Y9a5d0Gpg@mail.gmail.com>

Armin ended up comitting a fix today. :)

Alex


On Mon, Jun 10, 2013 at 5:19 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> Antonio Cuni was supposed to fix this problem ages ago :)
>
> On Tue, Jun 11, 2013 at 12:28 AM, Kamil Klimkiewicz <miglanz at gmail.com>
> wrote:
> > Hi guys,
> >
> > I'm playing with PyPy/RPython toolchain to create JITted interpreters
> > for some toy languages. Unfortunately I'm getting some strange errors
> > when performing the actual translation with:
> >
> >   $ pypy ~/dev/pypy/rpython/bin/rpython -Ojit example4
> >
> > The file I'm trying to translate is a modified example from the tutorial
> > from morepypy blog. Please take a look at the 3 files on the following
> > gist:
> >
> > https://gist.github.com/miGlanz/5749914
> >
> > The files in question are:
> >
> > - example4.py - modified file from the tutorial; I get an error which is
> >   written in rpython.log
> > - example4-ok.py - this is a file that get translated correctly - please
> >   note the only difference is in Tape.advance()
> >
> > I have more complicated code but I'm getting exactly the same error
> > there. Could you, please, tell me what I'm doing wrong? I am a total
> > noob when it comes to PyPy toolchain so please forgive my ignorance.
> >
> > Thanks in advance.
> >
> > --
> > Best regards,
> > Kamil Klimkiewicz
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130610/521cb05f/attachment.html>

From arigo at tunes.org  Wed Jun 12 20:36:41 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 12 Jun 2013 20:36:41 +0200
Subject: [pypy-dev] JIT slowdown on creating types
In-Reply-To: <51AC6B39.3060703@gmail.com>
References: <51AC6B39.3060703@gmail.com>
Message-ID: <CAMSv6X1rEw_APKjk8Y-UMikPXV2BNzn46C4uGJQD5R_J=TH-Ww@mail.gmail.com>

Hi Vasily,

Sorry for the delay.

On Mon, Jun 3, 2013 at 12:08 PM, Vasily Evseenko <svpcom at gmail.com> wrote:
> When i dynamically create new type (for example classes nested to
> functions) I've got linear code slowdown:

It's a known bug, related to the approach we take for JITting.
Technically, it occurs because we write a guard about the exact class
of 'obj'.  When test() is called the 2nd time, the guard fails, and we
compile a new "bridge" (i.e. a new portion of machine code) that
starts with a guard that the class is, this time, the 2nd version.
This fails again at the next test() call, and so on.  In the end we
have 99 guards that fails (so 99 conditional jumps in machine code)
before we reach the working one.  It basically ends up as a switch,
but a very bad variant of a switch, doing linear search.

This bad case can occur in a lot of situations, but all of them
relatively rare.  We should still improve it.


A bient?t,

Armin.

From mike at tradeworx.com  Fri Jun 14 16:26:02 2013
From: mike at tradeworx.com (Mike Beller)
Date: Fri, 14 Jun 2013 10:26:02 -0400
Subject: [pypy-dev] Memap in numpypy
Message-ID: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>

Hello

Do you think it is likely that the memap capabilities of numpy will find
their way in to numpypy any time soon?

It seems to me that some people think memap is a relatively unimportant
aspect of numpy.  But I do not think so.  Because of the way the linux IO
subsystem and virtual memory systems interact, memap enables numpy to have
high performance access to very large data sets -- it helps make numpy
relevant to "Big Data".

The code to allow numpy to support memap doesn't seem very large.  But,
while I have tried reading through the code, I really can not tell whether
the same is true for numpypy, or whether it is a large endeavor (for
example, due to some kind of pypy memory management architectural issue).

I'm interested in any input on this.

Mike Beller
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130614/357757d4/attachment.html>

From amauryfa at gmail.com  Fri Jun 14 17:08:37 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 14 Jun 2013 17:08:37 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
Message-ID: <CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>

Hi, this is definitely doable, but needs some work.
Here are the tasks I identified:

- copy "memmap.py" from the official numpy, and write a unit test (there is
a nice docstring in the previous file)

- add support in numpy for buffers with a fixed address
  (in interp-level terms: a RWBuffer with a get_raw_address() method)

- have buffer(mmap) return such a buffer, very very similar to
array.ArrayType.

The three tasks are quite independent, not too difficult, and could be a
nice start for newcomers...
I'll be happy to help.



2013/6/14 Mike Beller <mike at tradeworx.com>

> Hello
>
> Do you think it is likely that the memap capabilities of numpy will find
> their way in to numpypy any time soon?
>
> It seems to me that some people think memap is a relatively unimportant
> aspect of numpy.  But I do not think so.  Because of the way the linux IO
> subsystem and virtual memory systems interact, memap enables numpy to have
> high performance access to very large data sets -- it helps make numpy
> relevant to "Big Data".
>
> The code to allow numpy to support memap doesn't seem very large.  But,
> while I have tried reading through the code, I really can not tell whether
> the same is true for numpypy, or whether it is a large endeavor (for
> example, due to some kind of pypy memory management architectural issue).
>
> I'm interested in any input on this.
>
> Mike Beller
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130614/19305b6a/attachment.html>

From mike at tradeworx.com  Fri Jun 14 23:22:43 2013
From: mike at tradeworx.com (Mike Beller)
Date: Fri, 14 Jun 2013 17:22:43 -0400
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
Message-ID: <CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>

Thank you for responding Amaury.

I looked at the code for memmap.py.  The core of it comes down to these two
lines of code:

       mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
       self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,
            offset=offset, order=order)

It seems to me one needs to modify ndarray.__new__ so it can take a named
buffer= argument, where that argument can be an mmap object.   Currently
the python ndarray.__new__ can accept such an argument, but the numpypy one
does not appear to.  When I try to figure out the changes to
interp_numarray.py to make it work, it goes way over my head.

Thoughts?

Mike



On Fri, Jun 14, 2013 at 11:08 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com>wrote:

> Hi, this is definitely doable, but needs some work.
> Here are the tasks I identified:
>
> - copy "memmap.py" from the official numpy, and write a unit test (there
> is a nice docstring in the previous file)
>
> - add support in numpy for buffers with a fixed address
>   (in interp-level terms: a RWBuffer with a get_raw_address() method)
>
> - have buffer(mmap) return such a buffer, very very similar to
> array.ArrayType.
>
> The three tasks are quite independent, not too difficult, and could be a
> nice start for newcomers...
> I'll be happy to help.
>
>
>
> 2013/6/14 Mike Beller <mike at tradeworx.com>
>
>> Hello
>>
>> Do you think it is likely that the memap capabilities of numpy will find
>> their way in to numpypy any time soon?
>>
>> It seems to me that some people think memap is a relatively unimportant
>> aspect of numpy.  But I do not think so.  Because of the way the linux IO
>> subsystem and virtual memory systems interact, memap enables numpy to have
>> high performance access to very large data sets -- it helps make numpy
>> relevant to "Big Data".
>>
>> The code to allow numpy to support memap doesn't seem very large.  But,
>> while I have tried reading through the code, I really can not tell whether
>> the same is true for numpypy, or whether it is a large endeavor (for
>> example, due to some kind of pypy memory management architectural issue).
>>
>> I'm interested in any input on this.
>>
>> Mike Beller
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130614/3f23d777/attachment.html>

From amauryfa at gmail.com  Fri Jun 14 23:43:58 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 14 Jun 2013 23:43:58 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
Message-ID: <CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>

2013/6/14 Mike Beller <mike at tradeworx.com>

> It seems to me one needs to modify ndarray.__new__ so it can take a named
> buffer= argument, where that argument can be an mmap object.   Currently
> the python ndarray.__new__ can accept such an argument, but the numpypy one
> does not appear to.


Correct.


> When I try to figure out the changes to interp_numarray.py to make it
> work, it goes way over my head.


Hey, you need to get used to our RPython language. At least you opened the
correct file, that's a great first step :-). In interp_numarray.py:

- the block starting with "W_NDimArray.typedef" describe the type and the
methods. There is a "__new__" entry, which describes the constructor. The
implementation is descr_new_array().

- in descr_new_array(), the variables that start with "w_" are Wrapped
objects, i.e. visible to Python (similar to PyObject* in the CPython
implementation). At the moment there is a check for
"space.is_none(w_buffer)", we'll have to remove it of course.

- The goal is to use the memory allocated for w_buffer memory instead of
allocating a new one.
So don't call W_NDimArray.from_shape(),
call W_NDimArray.from_shape_and_storage() instead, it takes the raw address
of the buffer. That's it!

Of course I skipped over all the details, so the first thing to do is to
write a unit test to be sure that everything works correctly before
building a new PyPy.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130614/38925a2d/attachment-0001.html>

From albzey at googlemail.com  Sat Jun 15 20:32:15 2013
From: albzey at googlemail.com (Albert Zeyer)
Date: Sat, 15 Jun 2013 20:32:15 +0200
Subject: [pypy-dev] RPython: write generic native library / write CPython C
	module
Message-ID: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>

Hi,

I got the idea to use RPython to use my Python code to write a generic
native library.

I have looked roughly at the RPython translator code and it seems like
there are ways to export custom symbols.

Has that already been done? Or is it possible to do that?

---

Then, I got the further idea to write a CPython C extension module
with RPython. (Somehow the opposite of cpyext.)

I thought I could just export the PyMODINIT_FUNC init*() symbol and do
the common logic there. I.e., export the functions and other
objects/classes.

The next step would be to define wrapper functions which get native
CPython PyObject structures and transform them into the RPython
equivalents. Maybe cpyext can be useful for this task.

Maybe someone could point me to the direction whether this is feasible
and where to start?

Thanks and regards,
Albert

From arigo at tunes.org  Sun Jun 16 08:45:56 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 16 Jun 2013 08:45:56 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
Message-ID: <CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>

Hi Albert,

You may or may not get help with your questions: see
http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython


Armin

From albzey at googlemail.com  Sun Jun 16 13:37:37 2013
From: albzey at googlemail.com (Albert Zeyer)
Date: Sun, 16 Jun 2013 13:37:37 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
Message-ID: <CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>

Hi Armin,

Well, not quite. I think I know about the limitations and restrictions
of RPython. But that is why I think it is especially useful in some
cases where I have written some library in Python with some strict
subset of Python (or already mostly RPython). I need a native version
of the same library and don't really see the point in translating it
1:1 to C by hand.

Also, this is maybe more for playing around and just testing it. Maybe
I stumble upon other problems which hinder the whole idea.

-Albert



On Sun, Jun 16, 2013 at 8:45 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Albert,
>
> You may or may not get help with your questions: see
> http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython
>
>
> Armin

From mike at tradeworx.com  Sun Jun 16 19:38:35 2013
From: mike at tradeworx.com (Mike Beller)
Date: Sun, 16 Jun 2013 13:38:35 -0400
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
Message-ID: <CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>

Ok -- made some progress.  Built a unit test -- it's similar to the doctest
from memmap.py, but rewritten in py.test format (uses BaseNumpyAppTest from
pypy.module.micronumpy.test.test_base, just like test_numeric.py does).

Here is the next problem I face:  To run my unit test, I added in an import
of my new memmap.py, which I have placed in
lib_pypy/numpypy/core/memmap.py, and linked into core/__init__.py.  This
module imports mmap.py for obvious reasons (it wants to use mmap.py to
create the mmap object).  That import fails when I run the test: python2.7
pytest.py pypy/module/test_lib_pypy/numpypy/core/test_memmap.py ("Import
error: no module named mmap")

It fails because it can not import the module mmap.  Whereas if I just fire
up a normal interpreter-level pypy I have no problem importing mmap.
 Clearly it does exist in pypy.

I don't understand which modules will be loadable and which modules will
not be loadable when I run in pytest.py.  Somehow, clearly, the
BaseNumpyAppTest inheritance mechanism has allowed all the numpypy stuff to
show up (but only inside the class definition, not at module scope), but
other pypy application-level modules are not available.  I need to somehow
invoke the same magic for mmap if I want to be able to use the mmap module
to implement the numpypy memmap functionality.    I read coding-guide.rst
but it's still not obvious to me.  (Alternatively I suppose the answer
could be full translation, but testing this way would take a full 90-minute
translation every time I wanted to change a line of code.)

If you feel this detailed emailing is wasting your time, just let me know
and I can drop it.   (Or if you want to take the conversation off pypy-dev
that's ok too.)

Regards

Mike




On Fri, Jun 14, 2013 at 5:43 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> 2013/6/14 Mike Beller <mike at tradeworx.com>
>
>> It seems to me one needs to modify ndarray.__new__ so it can take a named
>> buffer= argument, where that argument can be an mmap object.   Currently
>> the python ndarray.__new__ can accept such an argument, but the numpypy one
>> does not appear to.
>
>
> Correct.
>
>
>> When I try to figure out the changes to interp_numarray.py to make it
>> work, it goes way over my head.
>
>
> Hey, you need to get used to our RPython language. At least you opened the
> correct file, that's a great first step :-). In interp_numarray.py:
>
> - the block starting with "W_NDimArray.typedef" describe the type and the
> methods. There is a "__new__" entry, which describes the constructor. The
> implementation is descr_new_array().
>
> - in descr_new_array(), the variables that start with "w_" are Wrapped
> objects, i.e. visible to Python (similar to PyObject* in the CPython
> implementation). At the moment there is a check for
> "space.is_none(w_buffer)", we'll have to remove it of course.
>
> - The goal is to use the memory allocated for w_buffer memory instead of
> allocating a new one.
> So don't call W_NDimArray.from_shape(),
> call W_NDimArray.from_shape_and_storage() instead, it takes the raw address
> of the buffer. That's it!
>
> Of course I skipped over all the details, so the first thing to do is to
> write a unit test to be sure that everything works correctly before
> building a new PyPy.
>
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130616/c8d22847/attachment.html>

From amauryfa at gmail.com  Sun Jun 16 21:29:06 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 16 Jun 2013 21:29:06 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
Message-ID: <CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>

2013/6/16 Mike Beller <mike at tradeworx.com>

> Ok -- made some progress.  Built a unit test -- it's similar to the
> doctest from memmap.py, but rewritten in py.test format (uses
> BaseNumpyAppTest from pypy.module.micronumpy.test.test_base, just like
> test_numeric.py does).
>
> Here is the next problem I face:  To run my unit test, I added in an
> import of my new memmap.py, which I have placed in
> lib_pypy/numpypy/core/memmap.py, and linked into core/__init__.py.  This
> module imports mmap.py for obvious reasons (it wants to use mmap.py to
> create the mmap object).  That import fails when I run the test: python2.7
> pytest.py pypy/module/test_lib_pypy/numpypy/core/test_memmap.py ("Import
> error: no module named mmap")
>

great progress anyway!


> It fails because it can not import the module mmap.  Whereas if I just
> fire up a normal interpreter-level pypy I have no problem importing mmap.
>  Clearly it does exist in pypy.
>

> I don't understand which modules will be loadable and which modules will
> not be loadable when I run in pytest.py.  Somehow, clearly, the
> BaseNumpyAppTest inheritance mechanism has allowed all the numpypy stuff to
> show up (but only inside the class definition, not at module scope), but
> other pypy application-level modules are not available.  I need to somehow
> invoke the same magic for mmap if I want to be able to use the mmap module
> to implement the numpypy memmap functionality.    I read coding-guide.rst
> but it's still not obvious to me.  (Alternatively I suppose the answer
> could be full translation, but testing this way would take a full 90-minute
> translation every time I wanted to change a line of code.)
>

That's because most built-in modules are not made available by default. You
need to pass the equivalent of "--withmod-mmap" to the interpreter. In a
test, this is done in a statement like:
    spaceconfig = dict(usemodules=["mmap"])
at the top of the test class. There certainly is already one (in order to
import numpy), you could add "mmap" to the existing list.

>

>
> If you feel this detailed emailing is wasting your time, just let me know
> and I can drop it.   (Or if you want to take the conversation off pypy-dev
> that's ok too.)
>

You are welcome. PyPy is a wonderful platform with powerful tools, but they
are often not documented enough.
Pypy-dev traffic is not that high these days; to get immediate answers
there many of us are hanging on IRC: #pypy at freenode.net.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130616/f9759947/attachment.html>

From fijall at gmail.com  Mon Jun 17 03:50:36 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 17 Jun 2013 09:50:36 +0800
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
Message-ID: <CAK5idxTLOcpHFKn8_W2-qVdfL0yiBU_r=SM7__=6YwvpSYFALw@mail.gmail.com>

On Sun, Jun 16, 2013 at 7:37 PM, Albert Zeyer <albzey at googlemail.com> wrote:
> Hi Armin,
>
> Well, not quite. I think I know about the limitations and restrictions
> of RPython. But that is why I think it is especially useful in some
> cases where I have written some library in Python with some strict
> subset of Python (or already mostly RPython). I need a native version
> of the same library and don't really see the point in translating it
> 1:1 to C by hand.
>
> Also, this is maybe more for playing around and just testing it. Maybe
> I stumble upon other problems which hinder the whole idea.
>
> -Albert

so let's reverse the question. Why do you need such a library?

From arigo at tunes.org  Mon Jun 17 09:23:46 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jun 2013 09:23:46 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
Message-ID: <CAMSv6X2sUCajvf4Lvqc5Ruc9AKFG6ecyprX8G3JLfbyd4g+aNA@mail.gmail.com>

Hi Albert,

On Sun, Jun 16, 2013 at 1:37 PM, Albert Zeyer <albzey at googlemail.com> wrote:
>> You may or may not get help with your questions: see
>> http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython
>
> Well, not quite.

No, sorry for the misunderstanding :-)  What I meant is that your
questions may be possibly valid, but I was pointing you to this FAQ
entry that explains why we (= the general PyPy developers) are not
really interested in this direction, and which lists some of our
reasons for that.  There is notably the fact that RPython is not meant
as a general language used to produce stand-alone C libraries.  It's
certainly possible to make one in theory, but cumbersome.  That's what
I meant with "you may not get help with your questions".


A bient?t,

Armin.

From vincent.legoll at gmail.com  Mon Jun 17 11:10:28 2013
From: vincent.legoll at gmail.com (Vincent Legoll)
Date: Mon, 17 Jun 2013 11:10:28 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
Message-ID: <CAEwRq=pvAPZHgoNX73gbX+ytOLVu=B0YNMSjm8bkaz1kbT0b5Q@mail.gmail.com>

On Sun, Jun 16, 2013 at 9:29 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

>
> If you feel this detailed emailing is wasting your time, just let me know
>> and I can drop it.   (Or if you want to take the conversation off pypy-dev
>> that's ok too.)
>>
>
> You are welcome. PyPy is a wonderful platform with powerful tools, but
> they are often not documented enough.
> Pypy-dev traffic is not that high these days; to get immediate answers
> there many of us are hanging on IRC: #pypy at freenode.net.
>

Exactly, and you're not alone following the discussion. I personally
welcome all those detailed explanations
about how to do something inside pypy, because I've tried myself and it was
hard to understand the big picture
and also hard to get all the little details that escaped me at that time.

So please continue this on-list, if deemed appropriate. There are people
learning through it.

-- 
Vincent Legoll
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130617/8e347eb0/attachment.html>

From albzey at googlemail.com  Mon Jun 17 11:20:41 2013
From: albzey at googlemail.com (Albert Zeyer)
Date: Mon, 17 Jun 2013 11:20:41 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAMSv6X2sUCajvf4Lvqc5Ruc9AKFG6ecyprX8G3JLfbyd4g+aNA@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
	<CAMSv6X2sUCajvf4Lvqc5Ruc9AKFG6ecyprX8G3JLfbyd4g+aNA@mail.gmail.com>
Message-ID: <CAO1Q+jferwkEP1WM+=s3sarrmc98wjDk909GKPNzyWJLQeKTyQ@mail.gmail.com>

Hi Armin,

Thanks for the clarification!

I was wondering: Why is it that RPython is not a good general purpose
language? In the original paper
(http://rpython.googlecode.com/svn/trunk/doc/rpython-DLS08.pdf), it is
said:

> The result is a language that is more expressive than C# and Java, but
> which does not compromise runtime efficiency. RPython was initially
> designed for the specific purpose of implementing PyPy [25] (a Python
> interpreter written in Python), but it has grown into a full-fledged
> language in its own right.
>
> Currently, RPython can be used in many contexts: to develop
> stand-alone programs, such as the Standard Interpreter itself; to
> write highly efficient extension modules for CPython, which could only
> be written in C in the past; to develop dynamic web applications
> without the need to write JavaScript code; to produce efficient
> libraries of classes and functions to be used by other .NET and Java
> programs. In particular, RPython can be the ideal companion for all
> those CPython, IronPython and Jython developers that so far have been
> forced to write the parts of their programs that need high performance
> in C, C# or Java.

I think, in general, RPython looks quite nice. I have static types via
automatic type inference. I could use C or C++ for the same tasks but
I like the Python syntax. I could also use Cython but I don't really
see much the difference between Cython and RPython, except that
RPython does automatic type inference and is a strict subset of
Python, while Cython is explicit manifest typing.

(Note that I also asked the same question here:
http://stackoverflow.com/questions/17134479/why-do-people-say-that-rpython-is-an-unpleasant-language-to-program-in
)

Regards,
Albert


On Mon, Jun 17, 2013 at 9:23 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Albert,
>
> On Sun, Jun 16, 2013 at 1:37 PM, Albert Zeyer <albzey at googlemail.com> wrote:
>>> You may or may not get help with your questions: see
>>> http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython
>>
>> Well, not quite.
>
> No, sorry for the misunderstanding :-)  What I meant is that your
> questions may be possibly valid, but I was pointing you to this FAQ
> entry that explains why we (= the general PyPy developers) are not
> really interested in this direction, and which lists some of our
> reasons for that.  There is notably the fact that RPython is not meant
> as a general language used to produce stand-alone C libraries.  It's
> certainly possible to make one in theory, but cumbersome.  That's what
> I meant with "you may not get help with your questions".
>
>
> A bient?t,
>
> Armin.

From cfbolz at gmx.de  Mon Jun 17 11:38:35 2013
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 17 Jun 2013 11:38:35 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
Message-ID: <51BED91B.8060901@gmx.de>

On 06/16/2013 01:37 PM, Albert Zeyer wrote:
> Hi Armin,
>
> Well, not quite. I think I know about the limitations and restrictions
> of RPython. But that is why I think it is especially useful in some
> cases where I have written some library in Python with some strict
> subset of Python (or already mostly RPython). I need a native version
> of the same library and don't really see the point in translating it
> 1:1 to C by hand.

One of the reasons why the RPython project does not care much about
this use case anymore is that we *had* a way to make CPython
extensions. It was called the "extension compiler". It turned out to be
a) extremely annoying to implement b) not very useful in practice. Thus
a totally frustrating experience overall.

The reasons for slowness was mainly: when compiling RPython code to be
a C extension module, reference counting is used to reclaim garbage.
That is extremely slow. Therefore the modules never got the performance
anybody expected.

Cheers,

Carl Friedrich

From albzey at googlemail.com  Mon Jun 17 12:47:53 2013
From: albzey at googlemail.com (Albert Zeyer)
Date: Mon, 17 Jun 2013 12:47:53 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <51BED91B.8060901@gmx.de>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
	<51BED91B.8060901@gmx.de>
Message-ID: <CAO1Q+jfbNmaZj3nYcrmj_PXGJ6ZaSjpHRBZ2ND-80kDtLowE-Q@mail.gmail.com>

Oh, that is interesting. I searched a bit around but did not find too
much. Is that what I can see in test_wrapping.py for example
[here](https://bitbucket.org/william_ml_leslie/pypy-effect-analysis/src/4ba135c9c31ec7e256202f8aadf1f0dce24188e0/pypy/translator/c/test/test_wrapping.py?at=0.9.x)?
I also read about the [RPythonic
project](https://code.google.com/p/rpythonic/) which seems to do
something similar.

In my case, I think it would already be fast if for every call with
attributes of immutable objects (strings, numbers, tuples), it would
just copy them and otherwise, it would use some wrapper objects. That
way, RPython would not have to rely on reference counting (except that
the wrapper objects would hold one reference to a native PyObject).
For some other objects, e.g. dicts, it maybe could also copy them if
they are not going to be modified.

I'm interested in trying it out. Or do you think it would be a waste of time?

(Btw., I saw that you have worked at Heinrich Heine D?sseldorf. I'm
just sitting there right now in the library. :))


On Mon, Jun 17, 2013 at 11:38 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> On 06/16/2013 01:37 PM, Albert Zeyer wrote:
>>
>> Hi Armin,
>>
>> Well, not quite. I think I know about the limitations and restrictions
>> of RPython. But that is why I think it is especially useful in some
>> cases where I have written some library in Python with some strict
>> subset of Python (or already mostly RPython). I need a native version
>> of the same library and don't really see the point in translating it
>> 1:1 to C by hand.
>
>
> One of the reasons why the RPython project does not care much about
> this use case anymore is that we *had* a way to make CPython
> extensions. It was called the "extension compiler". It turned out to be
> a) extremely annoying to implement b) not very useful in practice. Thus
> a totally frustrating experience overall.
>
> The reasons for slowness was mainly: when compiling RPython code to be
> a C extension module, reference counting is used to reclaim garbage.
> That is extremely slow. Therefore the modules never got the performance
> anybody expected.
>
> Cheers,
>
> Carl Friedrich
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From arigo at tunes.org  Mon Jun 17 21:47:24 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jun 2013 21:47:24 +0200
Subject: [pypy-dev] RPython: write generic native library / write
 CPython C module
In-Reply-To: <CAO1Q+jferwkEP1WM+=s3sarrmc98wjDk909GKPNzyWJLQeKTyQ@mail.gmail.com>
References: <CAO1Q+je2h2_7_5hbqJZ0tJa+DqHkyrkod-Ug2x29OsBRCCnXOQ@mail.gmail.com>
	<CAMSv6X08DO3r9tEPzTMqd1k=kVX71ph7=NhB18Yw62XFo7g09Q@mail.gmail.com>
	<CAO1Q+jdpYyBeFipfQ3PhCYamu93iHTvB5apiTMOAy7GGeW-XMg@mail.gmail.com>
	<CAMSv6X2sUCajvf4Lvqc5Ruc9AKFG6ecyprX8G3JLfbyd4g+aNA@mail.gmail.com>
	<CAO1Q+jferwkEP1WM+=s3sarrmc98wjDk909GKPNzyWJLQeKTyQ@mail.gmail.com>
Message-ID: <CAMSv6X21=kNNE=iEOXgdejfGtO+Ckie-RWOD5T2KZBFeTxzKxA@mail.gmail.com>

Hi Albert,

On Mon, Jun 17, 2013 at 11:20 AM, Albert Zeyer <albzey at googlemail.com> wrote:
> I was wondering: Why is it that RPython is not a good general purpose
> language? In the original paper
> (http://rpython.googlecode.com/svn/trunk/doc/rpython-DLS08.pdf), it is
> said:

RPython dates further back.  It is perhaps first documented in the
2006 paper "PyPy's Approach to Virtual Machine Construction".  In
there you'll already find paragraphs like this:

"""
We should mention that restricting oneself to write RPython code instead
of Python is still a burden, and that we are not proposing in any way
that the Python language itself should evolve in this direction, nor
even that RPython usage should become widespread.  It is a tool designed
with a specific goal in mind, which is the ability to produce reasonably
efficient, stand-alone code in a large variety of environment.
"""

The paper you cite was written by an academic group that was focusing
on OO languages (Java and CLI), building on the "variety of
environment" concept.  However the two paragraphs you cite feel a bit
like advertisement today, given that interest for Java and CLI dried
out in PyPy...

Anyway, let's try to answer your questions in more details.

RPython is great at one thing: being a language in which you express
just the right amount of details in your interpreter for the purpose
of turning this interpreter into a full VM automatically.  For any
other "more traditional" usage --- whatever the programming level ---
there is another language out there that already exists and is much
more well-supported than RPython in terms of features, tools,
debuggers, group of people, etc.

When people look at RPython, an obvious feature is that it is
syntactically identical to Python.  "RPython must be an easy language,
given that it has got the syntax of Python, which is easy".  This is a
common misconception.  In fact, pleasing the automatic type inference
process can be difficult.  It requires the programmer keeping in his
head the global types of his whole program, and carefully writing code
according to these implicit types.  The process is much harder for
newcomers, which don't have any written-down example to learn how to
manipulate the types --- precisely because they are implicit.

So this is the reason we are recommending against RPython now (and for
many years now).  Anybody who thinks RPython is as easy as Python
is someone who will happily try out RPython and be burned alive by it.
So "by default" we answer negatively to people who might be at the
level of newcomers, to keep them away from this in-house,
not-easy-to-program, not-easy-to-debug subset of Python which we
call RPython.

Obviously, we don't think that RPython is completely bogus.  It is a
language with use cases, and probably the one language with the most
advanced meta-JIT framework :-)  But, at least, I don't want to spend
three years of my time trying to turn it into an good language for
everybody (and, likely, getting out only an ok-ish language).
Personally, I find it far more interesting to spend three years of my
time (or 10!) on other things, like giving a good alternative
implementation of the full unmodified Python language :-)


A bient?t,

Armin.

PS: if you go to the STUPS Group in Softwaretechnik und
Programmiersprachen, you might find David Schneider still in
D?sseldorf :-)

From cfbolz at gmx.de  Wed Jun 19 18:25:31 2013
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 19 Jun 2013 18:25:31 +0200
Subject: [pypy-dev] London sprint pre-announcement
Message-ID: <51C1DB7B.7080301@gmx.de>

Hi all,

we are planning a sprint in London the last week of August (26th August
to 1st of September, both included). It will be a public PyPy sprint,
suitable for newcomers. The actual sprinting will happen either at
King's College Strand Campus or at some other place, we are still
evaluating options. We will do a proper announcement soon, just wanted
to give everybody some advance warning.

Cheers,

Carl Friedrich

From mike at tradeworx.com  Thu Jun 20 03:00:57 2013
From: mike at tradeworx.com (Mike Beller)
Date: Wed, 19 Jun 2013 21:00:57 -0400
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
Message-ID: <CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>

Thank you both, for your encouragement.

So I have made progress.  I now have some unit tests, and they can compile
and run.  I have imported the memmap.py from numpy, and modified it so it
gets the needed items from micronumpy.  I can run the unit tests and they
fail for the correct reason -- that reason being the buffer attribute is
not supported in interp_numarray.descr_new_array() .  So that's great.

Here are my next questions:

1) The way the numpy mmap module works, it  calls ndarray.__new__(), and
monkey-patches the return value to add  _mmap, offset, and mode attributes.
 This, for example, ensures the mmap object is kept around until the array
is deleted.  However, I can't monkey-patch a numpy ndarray object.  I
presume this is because it is an interpreter level object rather than an
app level one?  Anyway -- not sure how to deal with this situation.

2) Secondly, the mmap object itself doesn't really provide a usable buffer
implementation.  The implementation of buffer(mmap) is currently
W_MMap.descr_buffer(), (found in interp_mmap.py), which returns a
StringLikeBuffer object.  This object (implemented in
pypy/interpreter/buffer.py) is a subclass of Buffer, which does not
implement get_raw_address().  Our current plan clearly requires the buffer
object to implement get_raw_address so it can be used by
ndarray.from_shape_and_storage().  Interestingly, it seems as if the
interp_mmap author anticipated this shortcoming -- there is a comment:
 "improve to work directly on low-level address" right in the descr_buffer
method.

So -- am I on the wrong path?  Should I not even bother trying to use the
mmap?  (since I can't monkey patch it and it doesn't do what I want?)  This
would mean perhaps using the underlying rffi mmap to build my own memmap
module.  Alternatively, can I fix the monkey-patching problem some other
way, and then take the advice of interp_mmap's author to "improve to work
directly on low-level address" by returning something better than a
StringLikeBuffer object.

Thoughts?

Mike



On Sun, Jun 16, 2013 at 3:29 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

>
> 2013/6/16 Mike Beller <mike at tradeworx.com>
>
>> Ok -- made some progress.  Built a unit test -- it's similar to the
>> doctest from memmap.py, but rewritten in py.test format (uses
>> BaseNumpyAppTest from pypy.module.micronumpy.test.test_base, just like
>> test_numeric.py does).
>>
>> Here is the next problem I face:  To run my unit test, I added in an
>> import of my new memmap.py, which I have placed in
>> lib_pypy/numpypy/core/memmap.py, and linked into core/__init__.py.  This
>> module imports mmap.py for obvious reasons (it wants to use mmap.py to
>> create the mmap object).  That import fails when I run the test: python2.7
>> pytest.py pypy/module/test_lib_pypy/numpypy/core/test_memmap.py ("Import
>> error: no module named mmap")
>>
>
> great progress anyway!
>
>
>> It fails because it can not import the module mmap.  Whereas if I just
>> fire up a normal interpreter-level pypy I have no problem importing mmap.
>>  Clearly it does exist in pypy.
>>
>
>> I don't understand which modules will be loadable and which modules will
>> not be loadable when I run in pytest.py.  Somehow, clearly, the
>> BaseNumpyAppTest inheritance mechanism has allowed all the numpypy stuff to
>> show up (but only inside the class definition, not at module scope), but
>> other pypy application-level modules are not available.  I need to somehow
>> invoke the same magic for mmap if I want to be able to use the mmap module
>> to implement the numpypy memmap functionality.    I read coding-guide.rst
>> but it's still not obvious to me.  (Alternatively I suppose the answer
>> could be full translation, but testing this way would take a full 90-minute
>> translation every time I wanted to change a line of code.)
>>
>
> That's because most built-in modules are not made available by default.
> You need to pass the equivalent of "--withmod-mmap" to the interpreter. In
> a test, this is done in a statement like:
>     spaceconfig = dict(usemodules=["mmap"])
> at the top of the test class. There certainly is already one (in order to
> import numpy), you could add "mmap" to the existing list.
>
>>
>
>>
>> If you feel this detailed emailing is wasting your time, just let me know
>> and I can drop it.   (Or if you want to take the conversation off pypy-dev
>> that's ok too.)
>>
>
> You are welcome. PyPy is a wonderful platform with powerful tools, but
> they are often not documented enough.
> Pypy-dev traffic is not that high these days; to get immediate answers
> there many of us are hanging on IRC: #pypy at freenode.net.
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130619/cc97a8e6/attachment.html>

From amauryfa at gmail.com  Thu Jun 20 11:07:38 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 20 Jun 2013 11:07:38 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
Message-ID: <CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>

2013/6/20 Mike Beller <mike at tradeworx.com>

> Thank you both, for your encouragement.
>
> So I have made progress.  I now have some unit tests, and they can compile
> and run.  I have imported the memmap.py from numpy, and modified it so it
> gets the needed items from micronumpy.  I can run the unit tests and they
> fail for the correct reason -- that reason being the buffer attribute is
> not supported in interp_numarray.descr_new_array() .  So that's great.
>

Great indeed!


>
> Here are my next questions:
>
> 1) The way the numpy mmap module works, it  calls ndarray.__new__(), and
> monkey-patches the return value to add  _mmap, offset, and mode attributes.
>  This, for example, ensures the mmap object is kept around until the array
> is deleted.  However, I can't monkey-patch a numpy ndarray object.  I
> presume this is because it is an interpreter level object rather than an
> app level one?  Anyway -- not sure how to deal with this situation.
>

No, that's because ndarray.__new__(subtype, ...) returns a ndarray. This is
wrong, it should return a instance of the subtype (the memmap.memmap class
in this case).
ndarray has no __dict__, OTOH memmap is a Python-defined class and has a
__dict__ where you can store attributes.
(It's also possible to have a __dict__ on ndarray, but it's not necessary
here)

In interp_numarray.py, descr_new_array() does not use w_subtype at all!
This means that ndarray cannot be subclassed in Python...
To make the necessary changes, you can pick one module and see how it's
done there.
For example, in the bz2 module, __new__ could simply have written "return
W_BZ2File(space)", but instead it handles subclasses correctly:

    def descr_bz2file__new__(space, w_subtype, __args__):
        bz2file = space.allocate_instance(W_BZ2File, w_subtype)
        W_BZ2File.__init__(bz2file, space)
        return space.wrap(bz2file)

ndarray should do the same, maybe by changing W_NDimArray.from_shape() (and
friends) this way:

    @classmethod
    def from_shape(w_subtype, shape, dtype, order='C'):
        ...
        if w_subtype is None:
            return W_NDimArray(impl)
        else:
            ...allocate_instance, __init__ and wrap...

This subclassing feature should have its own unit tests, btw.


> 2) Secondly, the mmap object itself doesn't really provide a usable buffer
> implementation.  The implementation of buffer(mmap) is currently
> W_MMap.descr_buffer(), (found in interp_mmap.py), which returns a
> StringLikeBuffer object.  This object (implemented in
> pypy/interpreter/buffer.py) is a subclass of Buffer, which does not
> implement get_raw_address().  Our current plan clearly requires the buffer
> object to implement get_raw_address so it can be used by
> ndarray.from_shape_and_storage().  Interestingly, it seems as if the
> interp_mmap author anticipated this shortcoming -- there is a comment:
>  "improve to work directly on low-level address" right in the descr_buffer
> method.
>

> So -- am I on the wrong path?  Should I not even bother trying to use the
> mmap?  (since I can't monkey patch it and it doesn't do what I want?)  This
> would mean perhaps using the underlying rffi mmap to build my own memmap
> module.  Alternatively, can I fix the monkey-patching problem some other
> way, and then take the advice of interp_mmap's author to "improve to work
> directly on low-level address" by returning something better than a
> StringLikeBuffer object.
>

This was the third task I mentioned earlier. It turns out that Armin
implemented it just this morning, thanks! :-)

Mike, you are doing well. Please keep going.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130620/0df27a46/attachment-0001.html>

From arigo at tunes.org  Thu Jun 20 11:22:30 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 20 Jun 2013 11:22:30 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
	<CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
Message-ID: <CAMSv6X2zS4NQ0rbXeo5Ktp9hjiQnKPQCgjaCuphe+ziOcudCwQ@mail.gmail.com>

Hi,

On Thu, Jun 20, 2013 at 11:07 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> This was the third task I mentioned earlier. It turns out that Armin
> implemented it just this morning, thanks! :-)

Yes, but still running translation tests :-)


A bient?t,

Armin.

From radek.falhar at hotmail.com  Thu Jun 20 17:43:55 2013
From: radek.falhar at hotmail.com (Radek Falhar)
Date: Thu, 20 Jun 2013 15:43:55 +0000
Subject: [pypy-dev] =?utf-8?q?How_does_RPython_read_=22live_objects=22_fro?=
	=?utf-8?q?m_Python?=
Message-ID: <DUB402-EAS1774E4BB02CDF596A85F0F5888E0@phx.gbl>

I started studying RPython and I find it's way of metaprogramming through first running Python interpret over source and then compiling from internally generated objects interesting.

I would like to ask how does RPython do it exactly? Where can I find it in RPython's compiler and if there is more detailed information about this practice?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130620/cdeff1f2/attachment.html>

From albzey at googlemail.com  Thu Jun 20 18:00:09 2013
From: albzey at googlemail.com (Albert Zeyer)
Date: Thu, 20 Jun 2013 18:00:09 +0200
Subject: [pypy-dev] How does RPython read "live objects" from Python
In-Reply-To: <DUB402-EAS1774E4BB02CDF596A85F0F5888E0@phx.gbl>
References: <DUB402-EAS1774E4BB02CDF596A85F0F5888E0@phx.gbl>
Message-ID: <CAO1Q+jfUm5tXynk_Z0toa8WRAFfXkFRA1qPGPAEaPvm6dzdbaw@mail.gmail.com>

When you run Python code, once the Python interpreter encounters a
function definition, it automatically compiles it to Python bytecode.
That's nothing specific to RPython. Also, decorators are only
syntactic sugar and are also just executed by Python itself (refer to
the Python docs).

The bytecode of a function can also be accessed (read-only) via
`func.func_code.co_code` - it's just a binary string. `func.func_code`
itself is an code-object (refer to the Python docs). You can use the
`dis` module to get a human-readable version of the bytecode.



On Thu, Jun 20, 2013 at 5:43 PM, Radek Falhar <radek.falhar at hotmail.com> wrote:
> I started studying RPython and I find it's way of metaprogramming through
> first running Python interpret over source and then compiling from
> internally generated objects interesting.
> I would like to ask how does RPython do it exactly? Where can I find it in
> RPython's compiler and if there is more detailed information about this
> practice?
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From ddvento at ucar.edu  Thu Jun 20 21:32:55 2013
From: ddvento at ucar.edu (Davide Del Vento)
Date: Thu, 20 Jun 2013 13:32:55 -0600
Subject: [pypy-dev] installing after translating
Message-ID: <51C358E7.9040709@ucar.edu>

Folks,

I've successfully translated pypy 2.0.2 as described here:
http://pypy.org/download.html#building-from-source (which is slightly 
different from running make).

Now I want to install it in a global location for others to use (it's on 
a machine shared by many users). The problem is, I've done this before, 
but I don't remember what I did and I did not write in my notes about 
translating.

Do I just rename pypy-c to pypy, place it in a bin directory somewhere 
in the PATH and do the same with site-packages and PYTHONPATH (and 
corresponding actions to include, lib_pypy and lib-python directories)?

Is there any reason why the makefile does not have an install target, 
possibly with prefix option (other than lack of interest or time to 
write it)?

Thanks and sorry for the stupidity of the question :-)

Dav

From amauryfa at gmail.com  Thu Jun 20 22:17:06 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 20 Jun 2013 22:17:06 +0200
Subject: [pypy-dev] installing after translating
In-Reply-To: <51C358E7.9040709@ucar.edu>
References: <51C358E7.9040709@ucar.edu>
Message-ID: <CAGmFidZV0jDMEcQa+6Lb4jXBY32b3HvFGEE381N4LTqCAte+KA@mail.gmail.com>

Hi,

2013/6/20 Davide Del Vento <ddvento at ucar.edu>

> Folks,
>
> I've successfully translated pypy 2.0.2 as described here:
> http://pypy.org/download.html#**building-from-source<http://pypy.org/download.html#building-from-source>(which is slightly different from running make).
>
> Now I want to install it in a global location for others to use (it's on a
> machine shared by many users). The problem is, I've done this before, but I
> don't remember what I did and I did not write in my notes about translating.
>
> Do I just rename pypy-c to pypy, place it in a bin directory somewhere in
> the PATH and do the same with site-packages and PYTHONPATH (and
> corresponding actions to include, lib_pypy and lib-python directories)?
>
> Is there any reason why the makefile does not have an install target,
> possibly with prefix option (other than lack of interest or time to write
> it)?
>

The makefile is only to build the C binary. It is generated, by the way.

To build a pypy distribution, after translation you can call
pypy/tool/release/package.py; this is how the nightly downloads are made.

PyPy does not use any prefix or anything: to install, just unpack the .tgz
somewhere and eventually add a link in /usr/local/bin:
http://doc.pypy.org/en/latest/getting-started.html#installing-pypy


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130620/bbbbd3cb/attachment.html>

From svpcom at gmail.com  Mon Jun 24 14:28:55 2013
From: svpcom at gmail.com (Vasily Evseenko)
Date: Mon, 24 Jun 2013 16:28:55 +0400
Subject: [pypy-dev] PyPy + setuptools issue
Message-ID: <51C83B87.7040705@gmail.com>

Hi,
I've got strange issue. Its sometimes produced on PyPy + setuptools
while creating virtualenv environment. On CPython (on the same hosts)
its not occurs. All recent versions of PyPy (1.9,  2.0b{1,2}, 2.0.2) are
affected, but frequency is decreased with version grow (on 2.0.2 we got
it only several times when on 2.0b1 it was frequent).

Symptoms: On heavy loaded linux hosts (tests build machines with Debian
GNU/Linux 6.0 x86_64) sometimes os.listdir for PyPy base directory
(/usr/pypy-2.0/lib-python/2.7) produces OSError: [Errno 11] Resource
temporarily unavailable. It is strange because all system calls (which
os.listdir called) unable to emit (due to man pages) EAGAIN error. Host
filesystem is ext4 (not nfs).

How to reproduce: we have no idea how to reliably reproduce it, but it
sometimes occurs in heavy loaded environment and only with os.listdir
inside of setuptools.

-----------------------------------------------------------------------------------------------------------------
13:01:26,594 INFO  - Exception:
13:01:26,594 INFO  - Traceback (most recent call last):
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py",
line 139, in main
13:01:26,594 INFO  - status = self.run(options, args)
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py",
line 266, in run
13:01:26,594 INFO  - requirement_set.prepare_files(finder,
force_root_egg_info=self.bundle, bundle=self.bundle)
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/pip-1.3.1-py2.7.egg/pip/req.py",
line 954, in prepare_files
13:01:26,594 INFO  - req_to_install.check_if_exists()
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/pip-1.3.1-py2.7.egg/pip/req.py",
line 686, in check_if_exists
13:01:26,594 INFO  - self.satisfied_by =
pkg_resources.get_distribution(self.req)
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 311, in get_distribution
13:01:26,594 INFO  - if isinstance(dist,Requirement): dist =
get_provider(dist)
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 197, in get_provider
13:01:26,594 INFO  - return working_set.find(moduleOrReq) or
require(str(moduleOrReq))[0]
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 666, in require
13:01:26,594 INFO  - needed = self.resolve(parse_requirements(requirements))
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 562, in resolve
13:01:26,594 INFO  - env = Environment(self.entries)
13:01:26,594 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 722, in __init__
13:01:26,594 INFO  - self.scan(search_path)
13:01:26,595 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 751, in scan
13:01:26,595 INFO  - for dist in find_distributions(item):
13:01:26,595 INFO  - File
"/var/lib/buildagent/workspace/52609/env/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py",
line 1700, in find_on_path
13:01:26,595 INFO  - for entry in os.listdir(path_item):
13:01:26,595 INFO  - OSError: [Errno 11] Resource temporarily
unavailable: '/usr/pypy-2.0/lib-python/2.7'


From arigo at tunes.org  Mon Jun 24 18:05:39 2013
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jun 2013 18:05:39 +0200
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <51C83B87.7040705@gmail.com>
References: <51C83B87.7040705@gmail.com>
Message-ID: <CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>

Hi Vasily,

On Mon, Jun 24, 2013 at 2:28 PM, Vasily Evseenko <svpcom at gmail.com> wrote:
> 13:01:26,595 INFO  - for entry in os.listdir(path_item):
> 13:01:26,595 INFO  - OSError: [Errno 11] Resource temporarily
> unavailable: '/usr/pypy-2.0/lib-python/2.7'

Shooting in the dark, but one possible reading of the man page shows
that when readdir() returns a non-null pointer, the value of errno
might be randomly modified.  It could be the case that an ultimately
successful readdir() was done by two system calls, the first one being
interrupted with EAGAIN.  If that's correct, it's fixed in 83f89665175e,
but you have to try it out...


A bient?t,

Armin.

From laurie at tratt.net  Tue Jun 25 11:14:16 2013
From: laurie at tratt.net (Laurence Tratt)
Date: Tue, 25 Jun 2013 10:14:16 +0100
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
Message-ID: <20130625091441.GB651@phase.tratt.net>

On Mon, Jun 24, 2013 at 06:05:39PM +0200, Armin Rigo wrote:

> Shooting in the dark, but one possible reading of the man page shows
> that when readdir() returns a non-null pointer, the value of errno
> might be randomly modified.

Yes, this is true and bit me in a different context just over a year ago:

  http://comments.gmane.org/gmane.os.openbsd.tech/27909

Philip Guenther's message is worth reading to get some idea of the
particular, quasi-legal language, that POSiX uses (and that confused me).


Laurie
-- 
Personal                                             http://tratt.net/laurie/
Software Development Team                                http://soft-dev.org/
   https://github.com/ltratt              http://twitter.com/laurencetratt

From arigo at tunes.org  Tue Jun 25 13:51:48 2013
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 25 Jun 2013 13:51:48 +0200
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <20130625091441.GB651@phase.tratt.net>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
	<20130625091441.GB651@phase.tratt.net>
Message-ID: <CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>

Hi Laurence,

On Tue, Jun 25, 2013 at 11:14 AM, Laurence Tratt <laurie at tratt.net> wrote:
>> Shooting in the dark, but one possible reading of the man page shows
>> that when readdir() returns a non-null pointer, the value of errno
>> might be randomly modified.
>
> Yes, this is true and bit me in a different context just over a year ago:

According to these mails, the issue you've hit is different: what
occurs to errno if readdir() returns NULL.  This is indeed strange but
documented.  But what occurs to errno if readdir() does *not* return
NULL?  I thought that errno would not be modified in this case, but as
far as I know the Posix man page is silent on the subject, and it
might be the cause of Vasily's report.


A bient?t,

Armin.

From laurie at tratt.net  Tue Jun 25 15:24:56 2013
From: laurie at tratt.net (Laurence Tratt)
Date: Tue, 25 Jun 2013 14:23:56 +0059
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
	<20130625091441.GB651@phase.tratt.net>
	<CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>
Message-ID: <20130625132421.GD20031@phase.tratt.net>

On Tue, Jun 25, 2013 at 01:51:48PM +0200, Armin Rigo wrote:

> According to these mails, the issue you've hit is different: what occurs to
> errno if readdir() returns NULL.  This is indeed strange but documented.
> But what occurs to errno if readdir() does *not* return NULL?  I thought
> that errno would not be modified in this case, but as far as I know the
> Posix man page is silent on the subject, and it might be the cause of
> Vasily's report.

The point that eventually became clear to me is that returning NULL and
setting errno are not necessarily connected, which was the original mistake I
made. errors can happen (and errno set) even if readdir hasn't hit the end of
the directory (so NULL need not be returned). Linux's man page is, unusually,
clearer than OpenBSD's in this regard, though it's so long and lawyerly that
it's still hard to work out what's going on! I had to read OpenBSD's source
code to work this out last year :/


Laurie
-- 
Personal                                             http://tratt.net/laurie/
Software Development Team                                http://soft-dev.org/
   https://github.com/ltratt              http://twitter.com/laurencetratt

From ddvento at ucar.edu  Tue Jun 25 17:21:02 2013
From: ddvento at ucar.edu (Davide Del Vento)
Date: Tue, 25 Jun 2013 09:21:02 -0600
Subject: [pypy-dev] installing after translating
In-Reply-To: <CAGmFidZV0jDMEcQa+6Lb4jXBY32b3HvFGEE381N4LTqCAte+KA@mail.gmail.com>
References: <51C358E7.9040709@ucar.edu>
	<CAGmFidZV0jDMEcQa+6Lb4jXBY32b3HvFGEE381N4LTqCAte+KA@mail.gmail.com>
Message-ID: <51C9B55E.8000203@ucar.edu>

Thanks that makes sense.
For me, I made sure to add it to my notes. To avoid confusion for 
others, it may be good to either add this info on the website right 
after the building from source, or adding a Makefile target named 
"tarball" or something.

Cheers,

Davide Del Vento,
NCAR Computational & Information Services Laboratory
Consulting Services Software Engineer

On 06/20/2013 02:17 PM, Amaury Forgeot d'Arc wrote:
> Hi,
>
> 2013/6/20 Davide Del Vento <ddvento at ucar.edu <mailto:ddvento at ucar.edu>>
>
>     Folks,
>
>     I've successfully translated pypy 2.0.2 as described here:
>     http://pypy.org/download.html#__building-from-source
>     <http://pypy.org/download.html#building-from-source> (which is
>     slightly different from running make).
>
>     Now I want to install it in a global location for others to use
>     (it's on a machine shared by many users). The problem is, I've done
>     this before, but I don't remember what I did and I did not write in
>     my notes about translating.
>
>     Do I just rename pypy-c to pypy, place it in a bin directory
>     somewhere in the PATH and do the same with site-packages and
>     PYTHONPATH (and corresponding actions to include, lib_pypy and
>     lib-python directories)?
>
>     Is there any reason why the makefile does not have an install
>     target, possibly with prefix option (other than lack of interest or
>     time to write it)?
>
>
> The makefile is only to build the C binary. It is generated, by the way.
>
> To build a pypy distribution, after translation you can call
> pypy/tool/release/package.py; this is how the nightly downloads are made.
>
> PyPy does not use any prefix or anything: to install, just unpack the
> .tgz somewhere and eventually add a link in /usr/local/bin:
> http://doc.pypy.org/en/latest/getting-started.html#installing-pypy
>
>
> --
> Amaury Forgeot d'Arc

From arigo at tunes.org  Tue Jun 25 23:34:03 2013
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 25 Jun 2013 23:34:03 +0200
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <20130625132421.GD20031@phase.tratt.net>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
	<20130625091441.GB651@phase.tratt.net>
	<CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>
	<20130625132421.GD20031@phase.tratt.net>
Message-ID: <CAMSv6X1FpdwDK_s_hC0SgEThToObinNAa=VVgNjhv73UOgkErQ@mail.gmail.com>

Hi Laurence,

On Tue, Jun 25, 2013 at 3:24 PM, Laurence Tratt <laurie at tratt.net> wrote:
> The point that eventually became clear to me is that returning NULL and
> setting errno are not necessarily connected, which was the original mistake I
> made. errors can happen (and errno set) even if readdir hasn't hit the end of
> the directory (so NULL need not be returned). Linux's man page is, unusually,
> clearer than OpenBSD's in this regard, though it's so long and lawyerly that
> it's still hard to work out what's going on! I had to read OpenBSD's source
> code to work this out last year :/

Sorry, I don't follow you.  As far as I know the Linux man page is not
clear about what occurs to errno if readdir() doesn't return NULL...


A bient?t,

Armin.

From laurie at tratt.net  Wed Jun 26 00:03:01 2013
From: laurie at tratt.net (Laurence Tratt)
Date: Tue, 25 Jun 2013 23:03:01 +0100
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <CAMSv6X1FpdwDK_s_hC0SgEThToObinNAa=VVgNjhv73UOgkErQ@mail.gmail.com>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
	<20130625091441.GB651@phase.tratt.net>
	<CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>
	<20130625132421.GD20031@phase.tratt.net>
	<CAMSv6X1FpdwDK_s_hC0SgEThToObinNAa=VVgNjhv73UOgkErQ@mail.gmail.com>
Message-ID: <20130625220326.GQ20031@phase.tratt.net>

On Tue, Jun 25, 2013 at 11:34:03PM +0200, Armin Rigo wrote:

> Sorry, I don't follow you.  As far as I know the Linux man page is not
> clear about what occurs to errno if readdir() doesn't return NULL...

Exactly :) My reading of it is that readdir returns: NULL and sets errno if
an error occurred; NULL and doesn't change errno if end of directory was
reached; non-NULL and no guarantees about errno if it's half-way through
reading a directory and no errors occurred. I came to the conclusion that I
had to be defensive and do the following: errno must be set to 0 before
readdir; only if NULL is returned can errno be checked to see its value.

[When I read through the OpenBSD source, the implementation happens to define
that if non-NULL is returned, errno isn't fiddled about with (unless
getdirentries is buggy, which I didn't check!). The documentation does not
mention anything about this, so I think one has to assume that all bets are
off, particularly on other OS's.]

This all seems a big mess to me, but since I changed my code, I've had no
problems with readdir (on OpenBSD or Linux), so I assume my defensive action
was successful...


Laurie
-- 
Personal                                             http://tratt.net/laurie/
Software Development Team                                http://soft-dev.org/
   https://github.com/ltratt              http://twitter.com/laurencetratt

From cfbolz at gmx.de  Wed Jun 26 09:06:01 2013
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 26 Jun 2013 09:06:01 +0200
Subject: [pypy-dev] PyPy doesn't make code written in C faster
In-Reply-To: <20130530164102.GA13899@ajhurst.org>
References: <CAMSv6X2f-_rGTb4aNNPA2BEOt3iV9Jk9A7saOJfeP3WmUZ81sQ@mail.gmail.com>
	<20130530164102.GA13899@ajhurst.org>
Message-ID: <51CA92D9.8040007@gmx.de>

On 30.05.2013 18:41, Nathan Hurst wrote:
> It doesn't have to be quadratic, it's easy to come up with a splitting
> algorithm:

FWIW, I started turning this code into RPython code on the
faster-str-of-bigint branch.

Cheers,

Carl Friedrich

From arigo at tunes.org  Wed Jun 26 10:52:25 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 26 Jun 2013 10:52:25 +0200
Subject: [pypy-dev] PyPy + setuptools issue
In-Reply-To: <20130625220326.GQ20031@phase.tratt.net>
References: <51C83B87.7040705@gmail.com>
	<CAMSv6X0eb-KAuL2vFtpEcdzs4NFJO99=YiFv7MtdEZgBMkZWqQ@mail.gmail.com>
	<20130625091441.GB651@phase.tratt.net>
	<CAMSv6X28rrDAkxpEaM2xU-zoc0=xR21e7dDuhKPKL+Dh2=VrJw@mail.gmail.com>
	<20130625132421.GD20031@phase.tratt.net>
	<CAMSv6X1FpdwDK_s_hC0SgEThToObinNAa=VVgNjhv73UOgkErQ@mail.gmail.com>
	<20130625220326.GQ20031@phase.tratt.net>
Message-ID: <CAMSv6X1ZR=Or16BZwXVFZZHhtRELrvr_QiQrK-J845NO-Frc3w@mail.gmail.com>

Hi Laurence,

On Wed, Jun 26, 2013 at 12:03 AM, Laurence Tratt <laurie at tratt.net> wrote:
>> Sorry, I don't follow you.  As far as I know the Linux man page is not
>> clear about what occurs to errno if readdir() doesn't return NULL...
>
> Exactly :)
> (...)
> This all seems a big mess to me, but since I changed my code, I've had no
> problems with readdir (on OpenBSD or Linux), so I assume my defensive action
> was successful...

Ok, thanks :-)  I really have to wonder why Posix didn't say that
readdir() returning NULL on end-of-directory simply *sets* errno to 0.
 Or even why it wouldn't allow both behaviors (it doesn't change
anything if you set errno to 0 yourself before, and setting it to 0 is
what you need in all practical cases), recommend the saner one, and so
gradually phase out the old behavior.  I even wonder why some systems
don't implement a "readdir1()" function that does "errno=0;
readdir()"...  But enough pointless ranting :-)


A bient?t,

Armin.

From matti.picus at gmail.com  Fri Jun 28 10:55:54 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 28 Jun 2013 11:55:54 +0300
Subject: [pypy-dev] adding numpy test target to buildbot
Message-ID: <51CD4F9A.5030804@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130628/e5166611/attachment.html>

From fijall at gmail.com  Fri Jun 28 11:07:45 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 28 Jun 2013 11:07:45 +0200
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <51CD4F9A.5030804@gmail.com>
References: <51CD4F9A.5030804@gmail.com>
Message-ID: <CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>

On Fri, Jun 28, 2013 at 10:55 AM, Matti Picus <matti.picus at gmail.com> wrote:
> Using Ronan's git repository and a few modifications we pass about 30% of
> the numpy test suite.
> It seems like now is the time to add a build target to our nightly builds
> that would install numpy and test a build:
> 1) download a pre-built version of pypy from nightlies
> 2) untar it, optionally create a virtualenv
> 3) install nose
> 4) git clone or git fetch the modified numpy repository, i.e.
> https://github.com/rlamy/numpy
> 5) checkout the correct branch "git checkout pypy_hack"
> 6) pypy setup.py install (in the numpy directory)
> 7) pypy -c "import numpy;numpy.test()" and collect the results
> 8) create an output page with the results of the nose tests
>
> I started this a bit on the numpy-tests branch of the buildbot , but it
> seems to be turning into something too big:
> - Are the steps I outlined above reasonable, can we do some shortcuts?

sounds good

> - buildbot has no support for git, I wrote a git_update function but it
> needs test

i don't believe you http://docs.buildbot.net/0.8.1/Git.html

> - how do I get nose installed?

pip install nose?

> - can buildbot parse output from nose?

buildbot does not parse output "summary" is our custom hack.

>
> And most importantly:
> - anyone want to take lead on this :) ?

not me ;-)

>
> Matti
>
> N.B. - I have some modifications to Ronan's github here,
> https://github.com/mattip/numpy, on the pypy-hack branch, that branch is
> actually the source of my 30% figure.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From matti.picus at gmail.com  Fri Jun 28 11:13:04 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 28 Jun 2013 12:13:04 +0300
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
References: <51CD4F9A.5030804@gmail.com>
	<CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
Message-ID: <51CD53A0.7080708@gmail.com>


On 06/28/2013 12:07 PM, Maciej Fijalkowski wrote:
>> - buildbot has no support for git, I wrote a git_update function but it
>> >needs test
> i don't believe youhttp://docs.buildbot.net/0.8.1/Git.html
>
cool, thanks! I guess we have our own update_hg for historical reasons?
Matti

From amauryfa at gmail.com  Fri Jun 28 11:14:31 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 28 Jun 2013 11:14:31 +0200
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
References: <51CD4F9A.5030804@gmail.com>
	<CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
Message-ID: <CAGmFidb676ryDOkSPHE1hyhZNydAwW2gFhiGgwevq9GuEp_1Fw@mail.gmail.com>

2013/6/28 Maciej Fijalkowski <fijall at gmail.com>

> > - can buildbot parse output from nose?
>
> buildbot does not parse output "summary" is our custom hack.


How are these summaries implemented btw? A custom pytest reporter?

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130628/e17e7ec7/attachment.html>

From fijall at gmail.com  Fri Jun 28 11:24:00 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 28 Jun 2013 11:24:00 +0200
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <CAGmFidb676ryDOkSPHE1hyhZNydAwW2gFhiGgwevq9GuEp_1Fw@mail.gmail.com>
References: <51CD4F9A.5030804@gmail.com>
	<CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
	<CAGmFidb676ryDOkSPHE1hyhZNydAwW2gFhiGgwevq9GuEp_1Fw@mail.gmail.com>
Message-ID: <CAK5idxSMbzwF8NCHovRfVnRarR8VDAkagMRGEYidxPttf9Fj2A@mail.gmail.com>

On Fri, Jun 28, 2013 at 11:14 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
>
> 2013/6/28 Maciej Fijalkowski <fijall at gmail.com>
>>
>> > - can buildbot parse output from nose?
>>
>> buildbot does not parse output "summary" is our custom hack.
>
>
> How are these summaries implemented btw? A custom pytest reporter?

either custom or something in pytest + code in our buildbot

From anto.cuni at gmail.com  Fri Jun 28 11:26:27 2013
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 28 Jun 2013 11:26:27 +0200
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <51CD53A0.7080708@gmail.com>
References: <51CD4F9A.5030804@gmail.com>
	<CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
	<51CD53A0.7080708@gmail.com>
Message-ID: <51CD56C3.9060605@gmail.com>

On 06/28/2013 11:13 AM, Matti Picus wrote:
>
> On 06/28/2013 12:07 PM, Maciej Fijalkowski wrote:
>>> - buildbot has no support for git, I wrote a git_update function but it
>>> >needs test
>> i don't believe youhttp://docs.buildbot.net/0.8.1/Git.html
>>
> cool, thanks! I guess we have our own update_hg for historical reasons?

IIRC, the hg support in buildbot had some strange features and it was just 
easier to write our own function instead of trying to convince buildbot to do 
what we meant. The situation might have changed since then, I don't know.

ciao,
Anto


From arigo at tunes.org  Fri Jun 28 14:14:37 2013
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 28 Jun 2013 14:14:37 +0200
Subject: [pypy-dev] adding numpy test target to buildbot
In-Reply-To: <51CD56C3.9060605@gmail.com>
References: <51CD4F9A.5030804@gmail.com>
	<CAK5idxSuL-SNPmEbGYsiOm3WQ_ZsTjefoQHoqdLeM9KX+Lh_1Q@mail.gmail.com>
	<51CD53A0.7080708@gmail.com> <51CD56C3.9060605@gmail.com>
Message-ID: <CAMSv6X3xjza0YjbRdAPc-sKwegi33SoN8C+M0SDC13T1MnOULg@mail.gmail.com>

Hi,

On Fri, Jun 28, 2013 at 11:26 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> cool, thanks! I guess we have our own update_hg for historical reasons?
>
> IIRC, the hg support in buildbot had some strange features and it was just
> easier to write our own function instead of trying to convince buildbot to
> do what we meant. The situation might have changed since then, I don't know.

Yes, the main issue is that it insisted on doing fresh clones from
scratch, for example whenever we tested a different branch.  That's
not a problem for small projects, but for PyPy that's a "definitely
not".  That's why it's written as a series of shell commands that call
hg.  (And based on the results, I'm not sure I understand why buildbot
needs special "hg support" or "git support"; it's just more flexible
to call the shell commands you want.)


A bient?t,

Armin.

From skip at pobox.com  Fri Jun 28 21:08:40 2013
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 28 Jun 2013 14:08:40 -0500
Subject: [pypy-dev] VerificationError after build
Message-ID: <CANc-5UyhkVt71EO=E90xU2DmBTcFhoyDHE8Ntcw-JRFut2nbgQ@mail.gmail.com>

My Linux machine at work is OpenSUSE 12.2, so I couldn't use a
pre-built binary.  I built PyPy 2.0.2 yesterday, then tried the
interpreter.  I got a couple VerificationError exceptions and never
got a prompt.  Looking more closely, I realized I was missing some
prerequisites I thought I had.  I installed everything I could find
from the list and fired up the build process again before heading
home.

It was done when I got in this morning.  I tried again.  Same result.
Since it was a compilation error, I thought I should mention what I
have available:

hornet% type cc
cc is /usr/bin/cc
hornet% type gcc
gcc is /opt/TWWfsw/bin/gcc
hornet% cc --version
cc (SUSE Linux) 4.7.1 20120723 [gcc-4_7-branch revision 189773]
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

hornet% gcc --version
gcc (GCC) 4.4.6 [TWW]
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For compatibility with our other bits of software, and with the PyPy
build itself, I would prefer to guarantee it uses gcc.  The error
message wasn't clear about what "cc" failed.  I skimmed the --help
output of rpython, then built it again with --cc=/.../.../gcc.  Third
try is a charm?  Not so much.  Can someone give me some suggestions
about debugging these tracebacks?

hornet% ./pypy-c
Python 2.7.3 (5acfe049a5b0cd0de158f62553a98f5ef364fd29, Jun 28 2013, 15:39:43)
[PyPy 2.0.2] on linux3
Type "help", "copyright", "credits" or "license" for more information.
/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/__pycache__/_cffi__g85535e98x310af836.c:27:19:
fatal error: panel.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "app_main.py", line 598, in run_it
  File "/home/skipm/.python.py", line 27, in <module>
    from save_session import mark, save, load
  File "/home/skipm/misc/python/save_session.py", line 19, in <module>
    import readline
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/readline.py",
line 9, in <module>
    from pyrepl.readline import *
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/readline.py",
line 33, in <module>
    from pyrepl.unix_console import UnixConsole, _error
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/unix_console.py",
line 25, in <module>
    from pyrepl import curses
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/curses.py",
line 28, in <module>
    import _curses
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/_curses.py",
line 319, in <module>
    """, libraries=['ncurses', 'panel'])
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/api.py",
line 311, in verify
    lib = self.verifier.load_library()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 68, in load_library
    self.compile_module()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 56, in compile_module
    self._compile_module()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 142, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/ffiplatform.py",
line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/ffiplatform.py",
line 50, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
VerificationError: CompileError: command 'cc' failed with exit status 1
And now for something completely different: ``it is the expected behavior,
except when you don't expect it''
/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/__pycache__/_cffi__g85535e98x310af836.c:27:19:
fatal error: panel.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/_pypy_interact.py",
line 34, in interactive_console
    from pyrepl.simple_interact import check
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/simple_interact.py",
line 27, in <module>
    from pyrepl.readline import multiline_input, _error, _get_reader
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/readline.py",
line 33, in <module>
    from pyrepl.unix_console import UnixConsole, _error
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/unix_console.py",
line 25, in <module>
    from pyrepl import curses
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/pyrepl/curses.py",
line 28, in <module>
    import _curses
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/_curses.py",
line 319, in <module>
    """, libraries=['ncurses', 'panel'])
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/api.py",
line 311, in verify
    lib = self.verifier.load_library()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 68, in load_library
    self.compile_module()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 56, in compile_module
    self._compile_module()
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/verifier.py",
line 142, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/ffiplatform.py",
line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "/net/udesktop267.wacker/export/home/skipm/3rdParty/pypy-2.0.2-src/lib_pypy/cffi/ffiplatform.py",
line 50, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
VerificationError: CompileError: command 'cc' failed with exit status 1

Thx,

Skip

From wlavrijsen at lbl.gov  Fri Jun 28 21:19:12 2013
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 28 Jun 2013 12:19:12 -0700 (PDT)
Subject: [pypy-dev] VerificationError after build
In-Reply-To: <CANc-5UyhkVt71EO=E90xU2DmBTcFhoyDHE8Ntcw-JRFut2nbgQ@mail.gmail.com>
References: <CANc-5UyhkVt71EO=E90xU2DmBTcFhoyDHE8Ntcw-JRFut2nbgQ@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1306281213130.30321@pc3a-wlav.lbl.gov>

Skip,

SuSE has a somewhat different packaging of curses than do other installation.
It'd be ideal if pypy-c would be immune to that, but so lacking I did:

   1) symlink tinfo to ncurses:
   /usr/lib64/libtinfo.so.5 -> libncurses.so

   2) symlink panel.h to ncurses/panel.h
   /usr/include/panel.h -> ncurses/panel.h

Live's been good ever since. :)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From skip at pobox.com  Fri Jun 28 21:30:29 2013
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 28 Jun 2013 14:30:29 -0500
Subject: [pypy-dev] VerificationError after build
In-Reply-To: <alpine.LNX.2.00.1306281213130.30321@pc3a-wlav.lbl.gov>
References: <CANc-5UyhkVt71EO=E90xU2DmBTcFhoyDHE8Ntcw-JRFut2nbgQ@mail.gmail.com>
	<alpine.LNX.2.00.1306281213130.30321@pc3a-wlav.lbl.gov>
Message-ID: <CANc-5Uy4nObHDYFBpVH1rrCa+zc2hyZTZvW9nPKB4oZqF+=sfA@mail.gmail.com>

> SuSE has a somewhat different packaging of curses than do other
> installation.
> It'd be ideal if pypy-c would be immune to that, but so lacking I did:
>
>   1) symlink tinfo to ncurses:
>   /usr/lib64/libtinfo.so.5 -> libncurses.so
>
>   2) symlink panel.h to ncurses/panel.h
>   /usr/include/panel.h -> ncurses/panel.h

Thanks.  That seems to have worked.  I now have a prompt and many pystones:

% ./pypy-c
Python 2.7.3 (5acfe049a5b0cd0de158f62553a98f5ef364fd29, Jun 28 2013, 15:39:43)
[PyPy 2.0.2] on linux3
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``PyPy and CPython: they are mortal
enemies intent on killing each other''
>>>> from test import pystone
>>>> pystone.main()
Pystone(1.1) time for 50000 passes = 0.082987
This machine benchmarks at 602504 pystones/second
>>>> pystone.main()
Pystone(1.1) time for 50000 passes = 0.027996
This machine benchmarks at 1.78597e+06 pystones/second
>>>> pystone.main()
Pystone(1.1) time for 50000 passes = 0.028995
This machine benchmarks at 1.72444e+06 pystones/second


Skip

From skip.montanaro at gmail.com  Fri Jun 28 22:41:49 2013
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 28 Jun 2013 15:41:49 -0500
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
Message-ID: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>

PyPy doesn't seem to handle PYTHONPATH the same way Python does.
Consider this:

% echo $PWD:$PYTHONPATH
/home/skipm/src/git/models/lenin:/home/skipm/src/tl_v11/prefix/lib/python2.7/site-packages:/opt/TWWfsw/libgtk+222/lib/python27/gtk-2.0

Now, run Python with that set as PYTHONPATH and see where directories
land.  I've trimmed out all the other elements which don't
matter. Note that

/home/skipm/src/tl_v11/prefix/lib/python2.7/site-packages

appears in sys.path ahead of

/opt/local/lib/python2.7/site-packages

hornet% PYTHONPATH=$PWD:$PYTHONPATH python
Python 2.7.2 (default, Nov 14 2012, 05:07:35)
[GCC 4.4.6 [TWW]] on linux3
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[..., '/home/skipm/src/tl_v11/prefix/lib/python2.7/site-packages',
..., '/opt/local/lib/python2.7/site-packages', ...]

Now, try the same experiment with PyPy.  The relative order of those
two directories is reversed:

hornet% PYTHONPATH=$PWD:$PYTHONPATH pypy
Python 2.7.3 (5acfe049a5b0cd0de158f62553a98f5ef364fd29, Jun 28 2013, 15:39:43)
[PyPy 2.0.2] on linux3
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``"there should be one and only one
obvious way to do it". PyPy variant: "there can be N half-buggy ways to do
it"''
>>>> import sys
>>>> sys.path
[..., '/opt/local/lib/python2.7/site-packages', ...,
'/home/skipm/src/tl_v11/prefix/lib/python2.7/site-packages', ...]

They each hold the copies of the internal development platform I work
on.  Since I am actively developing software using that platform, I
prefer to have my development version ahead of our production install.
 Is there a way to make that work?

Thx,

Skip

From arigo at tunes.org  Sat Jun 29 10:24:25 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jun 2013 10:24:25 +0200
Subject: [pypy-dev] VerificationError after build
In-Reply-To: <CANc-5Uy4nObHDYFBpVH1rrCa+zc2hyZTZvW9nPKB4oZqF+=sfA@mail.gmail.com>
References: <CANc-5UyhkVt71EO=E90xU2DmBTcFhoyDHE8Ntcw-JRFut2nbgQ@mail.gmail.com>
	<alpine.LNX.2.00.1306281213130.30321@pc3a-wlav.lbl.gov>
	<CANc-5Uy4nObHDYFBpVH1rrCa+zc2hyZTZvW9nPKB4oZqF+=sfA@mail.gmail.com>
Message-ID: <CAMSv6X2mCeiHPxCbnWtO40kd_X1YfhZxGvYGWBiaRmLyiTW0Nw@mail.gmail.com>

Hi Skip,

On Fri, Jun 28, 2013 at 9:30 PM, Skip Montanaro <skip at pobox.com> wrote:
>> SuSE has a somewhat different packaging of curses than do other
>> installation.
>> It'd be ideal if pypy-c would be immune to that, but so lacking I did:

I tried yesterday to have it refuse to translate when some
dependencies are missing (or just not found, like in this case
curses).

I also fixed yesterday a bug that prevents the prompt from showing up
if curses is missing --- it would just try to fall back to the cffi
version of curses, which would fail "spectacularly" like in your
example.

It seems you've been bitten by the same issues reported yesterday :-)


A bient?t,

Armin.

From arigo at tunes.org  Sat Jun 29 10:34:07 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jun 2013 10:34:07 +0200
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
In-Reply-To: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
References: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
Message-ID: <CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>

Hi Skip,

On Fri, Jun 28, 2013 at 10:41 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> Now, try the same experiment with PyPy.  The relative order of those
> two directories is reversed:

Sorry, I can't reproduce the problem you describe.  More importantly I
don't understand why '/opt/local/lib/python2.7/site-packages' ends up
in sys.path at all on top of pypy.  There is no reason to: pypy has no
notion of '.../lib/python2.7', and certainly ending up with the same
path as CPython looks very wrong to me (like it would be wrong to have
CPython 2.6 and 2.7 both use
'/opt/local/lib/python2.7/site-packages').

It's a custom-compiled pypy, so did you install it using the official
script pypy/tool/release/package.py?  If not, please give the paths
where you copied things, starting with the pypy executable.  Note that
pypy is *not* supposed to be installed exactly like CPython.


A bient?t,

Armin.

From cfbolz at gmx.de  Sat Jun 29 16:27:02 2013
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sat, 29 Jun 2013 16:27:02 +0200
Subject: [pypy-dev] PyPy doesn't make code written in C faster
In-Reply-To: <20130629141907.GA4844@ajhurst.org>
References: <CAMSv6X2f-_rGTb4aNNPA2BEOt3iV9Jk9A7saOJfeP3WmUZ81sQ@mail.gmail.com>
	<20130530164102.GA13899@ajhurst.org> <51CA92D9.8040007@gmx.de>
	<20130629141907.GA4844@ajhurst.org>
Message-ID: <51CEEEB6.8040803@gmx.de>

On 29.06.2013 16:19, Nathan Hurst wrote:
> On Wed, Jun 26, 2013 at 09:06:01AM +0200, Carl Friedrich Bolz wrote:
>> On 30.05.2013 18:41, Nathan Hurst wrote:
>>> It doesn't have to be quadratic, it's easy to come up with a splitting
>>> algorithm:
>>
>> FWIW, I started turning this code into RPython code on the
>> faster-str-of-bigint branch.
>
> Do you have a github gist or similar?

I already merged it to the main branch:

https://bitbucket.org/pypy/pypy/commits/902241cca7dc4cb76bf65732c3d134543fb433a0

Thanks very much :-).

The main changes I did from your version was generalize it to work for
any 3 <= n <= 36, optimize the handling of leading zeros and use a
StringBuilder instead of a list to build the resulting string. Plus I
kept the already existing fast version for powers of two alive.

Cheers,

Carl Friedrich

From njh at njhurst.com  Sat Jun 29 16:19:07 2013
From: njh at njhurst.com (Nathan Hurst)
Date: Sun, 30 Jun 2013 00:19:07 +1000
Subject: [pypy-dev] PyPy doesn't make code written in C faster
In-Reply-To: <51CA92D9.8040007@gmx.de>
References: <CAMSv6X2f-_rGTb4aNNPA2BEOt3iV9Jk9A7saOJfeP3WmUZ81sQ@mail.gmail.com>
	<20130530164102.GA13899@ajhurst.org> <51CA92D9.8040007@gmx.de>
Message-ID: <20130629141907.GA4844@ajhurst.org>

On Wed, Jun 26, 2013 at 09:06:01AM +0200, Carl Friedrich Bolz wrote:
> On 30.05.2013 18:41, Nathan Hurst wrote:
> >It doesn't have to be quadratic, it's easy to come up with a splitting
> >algorithm:
> 
> FWIW, I started turning this code into RPython code on the
> faster-str-of-bigint branch.

Do you have a github gist or similar?

njh

From njh at njhurst.com  Sat Jun 29 20:06:26 2013
From: njh at njhurst.com (Nathan Hurst)
Date: Sun, 30 Jun 2013 04:06:26 +1000
Subject: [pypy-dev] PyPy doesn't make code written in C faster
In-Reply-To: <51CEEEB6.8040803@gmx.de>
References: <CAMSv6X2f-_rGTb4aNNPA2BEOt3iV9Jk9A7saOJfeP3WmUZ81sQ@mail.gmail.com>
	<20130530164102.GA13899@ajhurst.org> <51CA92D9.8040007@gmx.de>
	<20130629141907.GA4844@ajhurst.org> <51CEEEB6.8040803@gmx.de>
Message-ID: <20130629180625.GB4844@ajhurst.org>

On Sat, Jun 29, 2013 at 04:27:02PM +0200, Carl Friedrich Bolz wrote:
> On 29.06.2013 16:19, Nathan Hurst wrote:
> >On Wed, Jun 26, 2013 at 09:06:01AM +0200, Carl Friedrich Bolz wrote:
> >>On 30.05.2013 18:41, Nathan Hurst wrote:
> >>>It doesn't have to be quadratic, it's easy to come up with a splitting
> >>>algorithm:
> >>
> >>FWIW, I started turning this code into RPython code on the
> >>faster-str-of-bigint branch.


I don't actually see any difference between that code and normal
python.  What makes it rpython?


> >Do you have a github gist or similar?
> 
> I already merged it to the main branch:
> 
> https://bitbucket.org/pypy/pypy/commits/902241cca7dc4cb76bf65732c3d134543fb433a0
> 
> Thanks very much :-).
> 
> The main changes I did from your version was generalize it to work for
> any 3 <= n <= 36, optimize the handling of leading zeros and use a
> StringBuilder instead of a list to build the resulting string. Plus I
> kept the already existing fast version for powers of two alive.

All good decisions.  Is there a nice way to profile this with
something like line_profiler?  It's possible to improve the
asymptotics in various places, but they almost certainly aren't worth
it.

Of course the real question is whether pypy now beats python2.7 on the
all important factorial benchmark.  (And if so should we rush out a
new release? ;)

njh

From skip.montanaro at gmail.com  Sat Jun 29 23:24:50 2013
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sat, 29 Jun 2013 16:24:50 -0500
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
In-Reply-To: <CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>
References: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
	<CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>
Message-ID: <CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>

> On Fri, Jun 28, 2013 at 10:41 PM, Skip Montanaro
> <skip.montanaro at gmail.com> wrote:
>> Now, try the same experiment with PyPy.  The relative order of those
>> two directories is reversed:
>
> Sorry, I can't reproduce the problem you describe.  More importantly I
> don't understand why '/opt/local/lib/python2.7/site-packages' ends up
> in sys.path at all on top of pypy.  There is no reason to: pypy has no
> notion of '.../lib/python2.7', and certainly ending up with the same
> path as CPython looks very wrong to me (like it would be wrong to have
> CPython 2.6 and 2.7 both use
> '/opt/local/lib/python2.7/site-packages').
>
> It's a custom-compiled pypy, so did you install it using the official
> script pypy/tool/release/package.py?  If not, please give the paths
> where you copied things, starting with the pypy executable.  Note that
> pypy is *not* supposed to be installed exactly like CPython.

Armin,

Thanks for the response.  We have a lot of open source software
packaged and provided for us by a company called The Written Word.
That includes Python 2.7.  Everything is installed with /opt/local as
a replacement for /usr/local.  For example:

% PYTHONPATH= python -S -E
Python 2.7.2 (default, Nov 14 2012, 05:07:35)
[GCC 4.4.6 [TWW]] on linux3
>>> import sys
>>> sys.path
['', '/opt/TWWfsw/curl722/lib/python27',
'/opt/TWWfsw/cython017/lib/python27',
'/opt/TWWfsw/distribute06/lib/python27',
'/opt/TWWfsw/docutils08/lib/python27',
'/opt/TWWfsw/git18/lib/python27',
'/opt/TWWfsw/gnuplot44/lib/python27',
'/opt/TWWfsw/jinja26/lib/python27',
'/opt/TWWfsw/libcairo110/lib/python27',
'/opt/TWWfsw/libgd20/lib/python27',
'/opt/TWWfsw/libglib226/lib/python27',
'/opt/TWWfsw/libgnutls210/lib/python27',
'/opt/TWWfsw/libgtk+222/lib/python27',
'/opt/TWWfsw/libgtk+222/lib/python27/gtk-2.0',
'/opt/TWWfsw/libqt47/lib/python27',
'/opt/TWWfsw/libxml27/lib/python27',
'/opt/TWWfsw/libyaml01/lib/python27',
'/opt/TWWfsw/libzeromq22/lib/python27',
'/opt/TWWfsw/lxml22/lib/python27',
'/opt/TWWfsw/matplotlib12/lib/python27',
'/opt/TWWfsw/mysql5515r/lib/python27',
'/opt/TWWfsw/nose11/lib/python27', '/opt/TWWfsw/numpy16/lib/python27',
'/opt/TWWfsw/openldap24/lib/python27',
'/opt/TWWfsw/pycrypto23/lib/python27',
'/opt/TWWfsw/pygments14/lib/python27',
'/opt/TWWfsw/scipy09/lib/python27',
'/opt/TWWfsw/sphinx10/lib/python27',
'/opt/TWWfsw/subversion17/lib/python27',
'/opt/local/lib/python2.7/site-packages',
'/opt/local/lib/python2.7/site-packages',
'/opt/local/lib/python2.7/site-packages',
'/opt/local/lib/python2.7/site-packages/snakemodels',
'/opt/local/lib/python2.7/site-packages',
'/opt/TWWfsw/python27/lib/python27.zip',
'/opt/TWWfsw/python27/lib/python2.7/',
'/opt/TWWfsw/python27/lib/python2.7/plat-linux3',
'/opt/TWWfsw/python27/lib/python2.7/lib-tk',
'/opt/TWWfsw/python27/lib/python2.7/lib-old',
'/opt/TWWfsw/python27/lib/python2.7/lib-dynload']

Kind of weird that /opt/local/lib/python2.7/site-packages shows up four times.

When I ran

  python ../../rpython/bin/rpython -Ojit targetpypystandalone

that version of Python was executed.  Accordingly,
/opt/local/lib/python2.7/site-packages was in sys.path, as it should
have been.  It appears that the generated pypy-c wound up with that
directory in its sys.path as well.  I wasn't executing it from an
installed location.  I just set up an alias to execute it from the
goal directory. OTOH, perhaps I should build it using /usr/bin/python:

% PYTHONPATH= /usr/bin/python -S -E
Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
>>> import sys
>>> sys.path
['', '/usr/lib/python27.zip', '/usr/lib64/python2.7/',
'/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk',
'/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload']

(though I don't understand why /usr/lib/python2.7/site-packages isn't
in sys.path here.  Maybe site.py does that?)

Skip

From arigo at tunes.org  Sun Jun 30 00:25:54 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 30 Jun 2013 00:25:54 +0200
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
In-Reply-To: <CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>
References: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
	<CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>
	<CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>
Message-ID: <CAMSv6X2V2ydLNODxF-Csr_sniFxTZg=bRJgyQ1Yd++iLvgNKZw@mail.gmail.com>

Hi Skip,

On Sat, Jun 29, 2013 at 11:24 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> that version of Python was executed.  Accordingly,
> /opt/local/lib/python2.7/site-packages was in sys.path, as it should
> have been.  It appears that the generated pypy-c wound up with that
> directory in its sys.path as well.

It doesn't matter much which version of Python is used to run the
translation of PyPy, in the same way than it doesn't matter much which
version of GCC is used to create CPython.  It's certainly not meant to
copy sys.path from the Python used for translation.

I can't help you much more than this, however.  If anyone else has
anything to suggest, feel free.


A bient?t,

Armin.

From anto.cuni at gmail.com  Sun Jun 30 01:30:40 2013
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 30 Jun 2013 01:30:40 +0200
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
In-Reply-To: <CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>
References: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
	<CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>
	<CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>
Message-ID: <51CF6E20.7090309@gmail.com>

On 06/29/2013 11:24 PM, Skip Montanaro wrote:

[cut]
> that version of Python was executed.  Accordingly,
> /opt/local/lib/python2.7/site-packages was in sys.path, as it should
> have been.  It appears that the generated pypy-c wound up with that
> directory in its sys.path as well.  I wasn't executing it from an
> installed location.  I just set up an alias to execute it from the
> goal directory. OTOH, perhaps I should build it using /usr/bin/python:

Do you by chance see this warning message when you start your pypy?

debug: WARNING: Library path not found, using compiled-in sys.path.
debug: WARNING: 'sys.prefix' will not be set.
debug: WARNING: Make sure the pypy binary is kept inside its tree of files.
debug: WARNING: It is ok to create a symlink to it from somewhere else.
'import site' failed

If so, it means that somehow pypy does not find its stdlib and it just uses 
the builtin sys.path. Note that if you run pypy from within the hg checkout, 
this should not happen.




From skip.montanaro at gmail.com  Sun Jun 30 02:40:42 2013
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sat, 29 Jun 2013 19:40:42 -0500
Subject: [pypy-dev] PYTHONPATH handling doesn't seem to match Python
In-Reply-To: <51CF6E20.7090309@gmail.com>
References: <CANc-5UyiOG6gXZpT=vMKoU4fLrhZh7gWz8OZW7DyBvF0ZCeC+g@mail.gmail.com>
	<CAMSv6X2Aqtcas7_BjEjjp1ECvuouRm+32DFNLp5shB8stOocMg@mail.gmail.com>
	<CANc-5Uy5=EeMsVBUjgqY9GXy_uUbffHei5h6r_7+FbQADqv9ig@mail.gmail.com>
	<51CF6E20.7090309@gmail.com>
Message-ID: <CANc-5UzRzyW7JwV86YJCW=eXZ1cUwcjPNU5nRhc=e0c=4B056Q@mail.gmail.com>

> Do you by chance see this warning message when you start your pypy?
>
> debug: WARNING: Library path not found, using compiled-in sys.path.
> debug: WARNING: 'sys.prefix' will not be set.
> debug: WARNING: Make sure the pypy binary is kept inside its tree of files.
> debug: WARNING: It is ok to create a symlink to it from somewhere else.
> 'import site' failed

No, nothing odd at all:

% type pypy
pypy is aliased to `/home/skipm/3rdParty/pypy-2.0.2-src/pypy/goal/pypy-c'
% pypy
Python 2.7.3 (5acfe049a5b0cd0de158f62553a98f5ef364fd29, Jun 29 2013, 21:26:14)
[PyPy 2.0.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``A pink glittery rotating lambda''
>>>>

From mike at tradeworx.com  Sun Jun 30 19:15:19 2013
From: mike at tradeworx.com (Mike Beller)
Date: Sun, 30 Jun 2013 13:15:19 -0400
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
	<CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
Message-ID: <CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>

Amaury

Sorry for the high latency.  Day job intervenes.  (This may take some time
:-) )

I have reviewed your advice (below) and I can certainly modify class
W_NDimArray as proposed.  Here is my question.  It seems from looking at
other similar modules, the convention in pypy/module is to have w_subtype
as the second positional parameter to the factory method.   Changing the 4
factory methods of W_NDimArray to follow this convention would involve
changing the following references (found using egrep/awk/uniq so I may not
be exactly right but you get the right order of magnitude):

      2 arrayimpl/concrete.py:
      2 arrayimpl/scalar.py:
      1 arrayimpl/sort.py:
      1 base.py:
      3 compile.py:
      5 interp_arrayops.py:
      1 interp_dtype.py:
      1 interp_flatiter.py:
     19 interp_numarray.py:
      2 interp_support.py:
      3 interp_ufuncs.py:
      2 iter.py:
      3 loop.py:
      2 test/test_numarray.py:

47 references in 14 files.   I'm a bit hesitant to change code spewed all
over micronumpy unless someone with more ownership over the code tells me
for sure that's the right thing to do.  So I can also propose 2
alternatives:

1) Create a parallel set of methods:  from_shape_and_subtype,
from_shape_and_storage_and_subtype, for example, which take w_subtype, but
this would cause method proliferation.

2) Add a named parameter to the existing factory methods, so they are like
  from_shape(shape, subtype=None....), but this would be kind of
nonstandard. On a related matter, I found other modules in the code base
where w_subtype is ignored (e.g. select/interp_epoll/W_EPoll, and others)
-- seems a common problem -- so maybe "kind of nonstandard" is not that
important -- or perhaps this is an area which was simply left to be cleaned
up later.

Thoughts about best approach?

Mike



On Thu, Jun 20, 2013 at 5:07 AM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

>
>
> In interp_numarray.py, descr_new_array() does not use w_subtype at all!
> This means that ndarray cannot be subclassed in Python...
> To make the necessary changes, you can pick one module and see how it's
> done there.
> For example, in the bz2 module, __new__ could simply have written "return
> W_BZ2File(space)", but instead it handles subclasses correctly:
>
>     def descr_bz2file__new__(space, w_subtype, __args__):
>         bz2file = space.allocate_instance(W_BZ2File, w_subtype)
>         W_BZ2File.__init__(bz2file, space)
>         return space.wrap(bz2file)
>
> ndarray should do the same, maybe by changing W_NDimArray.from_shape()
> (and friends) this way:
>
>     @classmethod
>     def from_shape(w_subtype, shape, dtype, order='C'):
>         ...
>         if w_subtype is None:
>             return W_NDimArray(impl)
>         else:
>             ...allocate_instance, __init__ and wrap...
>
> This subclassing feature should have its own unit tests, btw.
>
>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130630/2ac271cf/attachment.html>

From arigo at tunes.org  Sun Jun 30 21:07:28 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 30 Jun 2013 21:07:28 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
	<CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
	<CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>
Message-ID: <CAMSv6X1HOGLaNFHJBYxhquAKzP-LhLBmEd4fXUuWoj_-CNgkuw@mail.gmail.com>

Hi Mike,

On Sun, Jun 30, 2013 at 7:15 PM, Mike Beller <mike at tradeworx.com> wrote:
> is ignored (e.g. select/interp_epoll/W_EPoll, and others) -- seems a common
> problem

At the end of that file we have
"W_Epoll.typedef.acceptable_as_base_class = False", meaning that it
cannot be subclassed as app-level; so ignoring w_subtype is correct.
Are there other examples which don't have "acceptable_as_base_class =
False" ?


A bient?t,

Armin.

From amauryfa at gmail.com  Sun Jun 30 21:42:46 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 30 Jun 2013 21:42:46 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMSv6X1HOGLaNFHJBYxhquAKzP-LhLBmEd4fXUuWoj_-CNgkuw@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
	<CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
	<CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>
	<CAMSv6X1HOGLaNFHJBYxhquAKzP-LhLBmEd4fXUuWoj_-CNgkuw@mail.gmail.com>
Message-ID: <CAGmFidZ-Bc+Yk0pbLqDd_+hZcUN7qkgYvzHdByTzBHn0_YTibw@mail.gmail.com>

2013/6/30 Armin Rigo <arigo at tunes.org>

> Hi Mike,
>
> On Sun, Jun 30, 2013 at 7:15 PM, Mike Beller <mike at tradeworx.com> wrote:
> > is ignored (e.g. select/interp_epoll/W_EPoll, and others) -- seems a
> common
> > problem
>
> At the end of that file we have
> "W_Epoll.typedef.acceptable_as_base_class = False", meaning that it
> cannot be subclassed as app-level; so ignoring w_subtype is correct.
> Are there other examples which don't have "acceptable_as_base_class =
> False" ?
>

I tried to make a list. In the "module/" subdirectory, here are the
__new__ methods that don't use "allocate_instance" and don't define
"acceptable_as_base_class":

clr._CliObject_internal
_ffi.CDLL
_ffi.WinDLL
_ffi.Field
_ffi._StructDescr
numpypy.string_
numpypy.unicode_
numpypy.dtype
numpypy.ndarray
pypyjit.Box
_rawffi.CallbackPtr
_rawffi.CDLL
_winreg.HKEYType
zipimport.zipimporter

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130630/8d917057/attachment.html>

From amauryfa at gmail.com  Sun Jun 30 22:15:14 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 30 Jun 2013 22:15:14 +0200
Subject: [pypy-dev] Memap in numpypy
In-Reply-To: <CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>
References: <CAMhNvtG-xwK4X_gbrTQgkwRoTkkgRY41J-dbUUh1Awo3AvvPYA@mail.gmail.com>
	<CAGmFidb=fmcN9D6YQGwsHuyk4kS9OLv_frdVNbsbdFAsce+7ww@mail.gmail.com>
	<CAMhNvtHe+12U+rqz=namMtqouq3=A_Ntvwss6zC0-T0_tjm5xw@mail.gmail.com>
	<CAGmFidYjjPgM5Zc5CNJ3tRzB4bV10bSvgbe=7=DQxcAevm4kWw@mail.gmail.com>
	<CAMhNvtGefukSTKAdbz+X_=uoFXE0dvzQG_VBTkcuDKo0mXhWjw@mail.gmail.com>
	<CAGmFidbO_AG_epyCZOxtnUGQ6-hPwUHnQm=1ktheTkR9RD+ghw@mail.gmail.com>
	<CAMhNvtHQvXj285gJPQa0xZaiQm7=sQvdEAo-Qf7Qjv7in5+eWg@mail.gmail.com>
	<CAGmFidYApsUqBU_0ERC9JZ76+MstvPYJuegU3eGu0zVe7EBYkQ@mail.gmail.com>
	<CAMhNvtEKp6XWZYGUr6otr08WK9QMnH7v1M9_hJOWFjnA8G7e3w@mail.gmail.com>
Message-ID: <CAGmFidbr0aUV3ZDQNOK4YiEb=sEwPb2RiayhatUwxZ95y4=HLw@mail.gmail.com>

Hello,

2013/6/30 Mike Beller <mike at tradeworx.com>

> Amaury
>
> Sorry for the high latency.  Day job intervenes.  (This may take some time
> :-) )
>

No problem, we are all volunteers here. There is no pressure.
The worst that can happen is someone may need the feature and implement it
before you :-)


> I have reviewed your advice (below) and I can certainly modify class
> W_NDimArray as proposed.  Here is my question.  It seems from looking at
> other similar modules, the convention in pypy/module is to have w_subtype
> as the second positional parameter to the factory method.   Changing the 4
> factory methods of W_NDimArray to follow this convention would involve
> changing the following references (found using egrep/awk/uniq so I may not
> be exactly right but you get the right order of magnitude):
>
>       2 arrayimpl/concrete.py:
>       2 arrayimpl/scalar.py:
>       1 arrayimpl/sort.py:
>       1 base.py:
>       3 compile.py:
>       5 interp_arrayops.py:
>       1 interp_dtype.py:
>       1 interp_flatiter.py:
>      19 interp_numarray.py:
>       2 interp_support.py:
>       3 interp_ufuncs.py:
>       2 iter.py:
>       3 loop.py:
>       2 test/test_numarray.py:
>
> 47 references in 14 files.
>

Ouch!


> I'm a bit hesitant to change code spewed all over micronumpy unless
> someone with more ownership over the code tells me for sure that's the
> right thing to do.  So I can also propose 2 alternatives:
>
> 1) Create a parallel set of methods:  from_shape_and_subtype,
> from_shape_and_storage_and_subtype, for example, which take w_subtype, but
> this would cause method proliferation.
>
> 2) Add a named parameter to the existing factory methods, so they are like
>   from_shape(shape, subtype=None....), but this would be kind of
> nonstandard. On a related matter, I found other modules in the code base
> where w_subtype is ignored (e.g. select/interp_epoll/W_EPoll, and others)
> -- seems a common problem -- so maybe "kind of nonstandard" is not that
> important -- or perhaps this is an area which was simply left to be cleaned
> up later.
>

Both approach can work IMO:
for 1), these are only two new functions, and I'm sure they can share most
of the code with existing functions.
method 2) would work as well, and is proably even easier to implement.
And there is no standard that forbids named parameter! Some calls to
from_shape() already use them...

Do as you prefer, and what makes the code more natural to you.

Cheers,

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130630/de5c81c7/attachment-0001.html>