From jezreel at gmail.com  Sun Jul  1 02:45:52 2012
From: jezreel at gmail.com (Jez)
Date: Sat, 30 Jun 2012 17:45:52 -0700
Subject: [pypy-dev] Building the jvm-improvements branch
Message-ID: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>

Hi,

I am thinking about contributing to the JVM backend, and I'm trying to
build the basic translator for a start. I'm trying out the jvm-improvements
branch since there has been some recent work done there, but I can't seem
to get it to build. Here's my setup: 32-bit Ubuntu, OpenJDK 6, building
with `python translate.py --opt=0 --backend=jvm targetpypystandalone.py`.
The translation fails with:

[translation:ERROR]    File
"/mnt/hgfs/jez/src/pypy/pypy/translator/jvm/database.py", line 493, in
lltype_to_cts
[translation:ERROR]     raise AssertionError("Untranslatable type %s!" %
OOT)
[translation:ERROR]  AssertionError: Untranslatable type * Struct timespec
{ c_tv_sec, c_tv_nsec }!

Am I doing something wrong, or is this failure expected?

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120630/7a0c7e5f/attachment.html>

From fijall at gmail.com  Sun Jul  1 10:35:31 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 1 Jul 2012 10:35:31 +0200
Subject: [pypy-dev] Building the jvm-improvements branch
In-Reply-To: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>
References: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>
Message-ID: <CAK5idxQcoN3-H5mPY_Ui6zjbm5JYKTjDbJMtHsVNMRWaqRttqQ@mail.gmail.com>

On Sun, Jul 1, 2012 at 2:45 AM, Jez <jezreel at gmail.com> wrote:

> Hi,
>
> I am thinking about contributing to the JVM backend, and I'm trying to
> build the basic translator for a start. I'm trying out the jvm-improvements
> branch since there has been some recent work done there, but I can't seem
> to get it to build. Here's my setup: 32-bit Ubuntu, OpenJDK 6, building
> with `python translate.py --opt=0 --backend=jvm targetpypystandalone.py`.
> The translation fails with:
>
> [translation:ERROR]    File
> "/mnt/hgfs/jez/src/pypy/pypy/translator/jvm/database.py", line 493, in
> lltype_to_cts
> [translation:ERROR]     raise AssertionError("Untranslatable type %s!" %
> OOT)
> [translation:ERROR]  AssertionError: Untranslatable type * Struct timespec
> { c_tv_sec, c_tv_nsec }!
>
> Am I doing something wrong, or is this failure expected?
>
> Jez
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
Hi Jez

As a general note - the branches might fail. As a specific note - I can
probably help you with this particular problem in a few days. This is some
timing thing (not sure what) exposed in an ootype unfriendly way. Do you
want more info about this problem to try to fix yourself?

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120701/6b5e5ae7/attachment.html>

From jezreel at gmail.com  Sun Jul  1 11:09:49 2012
From: jezreel at gmail.com (Jez)
Date: Sun, 1 Jul 2012 02:09:49 -0700
Subject: [pypy-dev] Building the jvm-improvements branch
In-Reply-To: <CAK5idxQcoN3-H5mPY_Ui6zjbm5JYKTjDbJMtHsVNMRWaqRttqQ@mail.gmail.com>
References: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>
	<CAK5idxQcoN3-H5mPY_Ui6zjbm5JYKTjDbJMtHsVNMRWaqRttqQ@mail.gmail.com>
Message-ID: <CABVpgLWoFBnWVsgbv1KsOp-ioKNA_-ZjafaoBjwJ_pyF-33EfQ@mail.gmail.com>

On Sun, Jul 1, 2012 at 1:35 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Sun, Jul 1, 2012 at 2:45 AM, Jez <jezreel at gmail.com> wrote:
>
>> Hi,
>>
>> I am thinking about contributing to the JVM backend, and I'm trying to
>> build the basic translator for a start. I'm trying out the jvm-improvements
>> branch since there has been some recent work done there, but I can't seem
>> to get it to build. Here's my setup: 32-bit Ubuntu, OpenJDK 6, building
>> with `python translate.py --opt=0 --backend=jvm targetpypystandalone.py`.
>> The translation fails with:
>>
>> [translation:ERROR]    File
>> "/mnt/hgfs/jez/src/pypy/pypy/translator/jvm/database.py", line 493, in
>> lltype_to_cts
>> [translation:ERROR]     raise AssertionError("Untranslatable type %s!" %
>> OOT)
>> [translation:ERROR]  AssertionError: Untranslatable type * Struct
>> timespec { c_tv_sec, c_tv_nsec }!
>>
>> Am I doing something wrong, or is this failure expected?
>>
>> Jez
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
> Hi Jez
>
> As a general note - the branches might fail. As a specific note - I can
> probably help you with this particular problem in a few days. This is some
> timing thing (not sure what) exposed in an ootype unfriendly way. Do you
> want more info about this problem to try to fix yourself?
>
> Cheers,
> fijal
>

Hey Fijal,

Yeah, I don't mind taking a stab at it myself if you could tell me where to
start :)

Cheers,
Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120701/61e42955/attachment-0001.html>

From paulo.koch at gmail.com  Sun Jul  1 16:40:55 2012
From: paulo.koch at gmail.com (=?UTF-8?Q?Paulo_K=C3=B6ch?=)
Date: Sun, 1 Jul 2012 15:40:55 +0100
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
Message-ID: <CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>

On Sat, Jun 30, 2012 at 6:10 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Just make up a random password, we'll get it from you later. There is no
> pypy user as far as I know

YouTube now requires me to create a Google account to create a YouTube
user. We also don't have one of those, right?

From chleiberqqq at yahoo.com  Mon Jul  2 00:02:22 2012
From: chleiberqqq at yahoo.com (Louanne_ezzell)
Date: Sun, 1 Jul 2012 23:02:22 +0100 (BST)
Subject: [pypy-dev] Do u like coffee? Me too and I'm pretty sure we've got
	much more in common...
Message-ID: <1341180142.22832.YahooMailNeo@web132105.mail.ird.yahoo.com>

Hey, it is Louanne.
I'm looking for some fun in here, I mean  I need somebody to spend time together.
Shortly about myself. I love playing some music on guitar, partying, watching sci-fi movies, and I also really enjoy snowboarding!
It'd a lie if I didn't mention sex. I do love it and I think that there is nothing wrong with that. You do love sex, don't you?) Maybe we will like each other, right?)
Write me back and we will see what we can do about it)

From fijall at gmail.com  Mon Jul  2 08:36:13 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 2 Jul 2012 08:36:13 +0200
Subject: [pypy-dev] Building the jvm-improvements branch
In-Reply-To: <CABVpgLWoFBnWVsgbv1KsOp-ioKNA_-ZjafaoBjwJ_pyF-33EfQ@mail.gmail.com>
References: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>
	<CAK5idxQcoN3-H5mPY_Ui6zjbm5JYKTjDbJMtHsVNMRWaqRttqQ@mail.gmail.com>
	<CABVpgLWoFBnWVsgbv1KsOp-ioKNA_-ZjafaoBjwJ_pyF-33EfQ@mail.gmail.com>
Message-ID: <CAK5idxTi0VT=D3OkeeH2HvjSbtWva-TGFQsM+KsPZPPiLj=S0Q@mail.gmail.com>

On Sun, Jul 1, 2012 at 11:09 AM, Jez <jezreel at gmail.com> wrote:

> On Sun, Jul 1, 2012 at 1:35 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:
>
>> On Sun, Jul 1, 2012 at 2:45 AM, Jez <jezreel at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am thinking about contributing to the JVM backend, and I'm trying to
>>> build the basic translator for a start. I'm trying out the jvm-improvements
>>> branch since there has been some recent work done there, but I can't seem
>>> to get it to build. Here's my setup: 32-bit Ubuntu, OpenJDK 6, building
>>> with `python translate.py --opt=0 --backend=jvm targetpypystandalone.py`.
>>> The translation fails with:
>>>
>>> [translation:ERROR]    File
>>> "/mnt/hgfs/jez/src/pypy/pypy/translator/jvm/database.py", line 493, in
>>> lltype_to_cts
>>> [translation:ERROR]     raise AssertionError("Untranslatable type %s!" %
>>> OOT)
>>> [translation:ERROR]  AssertionError: Untranslatable type * Struct
>>> timespec { c_tv_sec, c_tv_nsec }!
>>>
>>> Am I doing something wrong, or is this failure expected?
>>>
>>> Jez
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>> Hi Jez
>>
>> As a general note - the branches might fail. As a specific note - I can
>> probably help you with this particular problem in a few days. This is some
>> timing thing (not sure what) exposed in an ootype unfriendly way. Do you
>> want more info about this problem to try to fix yourself?
>>
>> Cheers,
>> fijal
>>
>
> Hey Fijal,
>
> Yeah, I don't mind taking a stab at it myself if you could tell me where
> to start :)
>
> Cheers,
> Jez
>
>
Start at finding where the thing gets created (in graphs and in source
code). Look at which level you can use the same hacks as in
rpython/module/ll_os.py. Expect europython related lags in replies,
otherwise I would recommend IRC :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120702/2313335e/attachment.html>

From blbyeh at winning.net.cn  Mon Jul  2 13:19:34 2012
From: blbyeh at winning.net.cn (Awwxyz)
Date: Mon, 2 Jul 2012 19:19:34 +0800
Subject: [pypy-dev] =?gb2312?b?z97Bv7DmyPDKv8P7se2jrMzYvNvHwLm6o6zLzcDx?=
 =?gb2312?b?19TTw7a8zOXD5sW2o6EoQUQpIHB5cHktZGV2QGNvZGVzcGVhay5uZXQ=?=
Message-ID: <bad5ff0a-f596-4fa1-8990-50bfa6463670@winmail.winning.net.cn>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120702/078abc45/attachment.html>

From rokujyouhitomajp at gmail.com  Mon Jul  2 14:05:40 2012
From: rokujyouhitomajp at gmail.com (Tohru Ike)
Date: Mon, 2 Jul 2012 21:05:40 +0900
Subject: [pypy-dev] =?gb2312?b?z97Bv7DmyPDKv8P7se2jrMzYvNvHwLm6o6zLzcDx?=
	=?gb2312?b?19TTw7a8zOXD5sW2o6EoQUQpIHB5cHktZGV2QGNvZGVzcGVhay5u?=
	=?gb2312?b?ZXQ=?=
In-Reply-To: <bad5ff0a-f596-4fa1-8990-50bfa6463670@winmail.winning.net.cn>
References: <bad5ff0a-f596-4fa1-8990-50bfa6463670@winmail.winning.net.cn>
Message-ID: <CABp0Y4Fau4qdEgXA7PGUNfKa_7XNapDdXjKaPxeMBrvOsuufhg@mail.gmail.com>

I think that the following message are spam in Chinese.

Regards,
Tohru Ike @rokujyouhitoma

2012/7/2 Awwxyz <blbyeh at winning.net.cn>

>  pypy-dev at codespeak.net
>
> *?7?P?b??L?8?9?d? ?*
>
> * ???????????????????*
>
> *?**S**?**9**?**8**?**O**???**i**?**7**?**d**?**R**?**9**??**0**?**d**?**a
> **?**7**?**j**?**e**?**7**?**d**?**9**??*
>
> * ????????>>>>>>> <http://163.fm/3FmG15m>*
>
>      pypy-dev at codespeak.net
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120702/6f3b036d/attachment.html>

From holger at merlinux.eu  Tue Jul  3 08:59:19 2012
From: holger at merlinux.eu (holger krekel)
Date: Tue, 3 Jul 2012 06:59:19 +0000
Subject: [pypy-dev] pypy ML aliases at codespeak to be deleted
Message-ID: <20120703065918.GG11942@merlinux.eu>

Hi all,

in the future you must use pypy-*@python.org addresses, not @codespeak.net
anymore.

It's been a while since Ralf Hildebrandt and I migrated the pypy
mailing lists to @python.org.  The old addresses @codespeak.net forward
mail to the new location but this implicitely disables some spam checks
that python.org implements.  Therefore, in a few days the codespeak
aliases will be deleted, namely pypy-dev, pypy-z, pypy-svn. I assume
also that pypy-issue can be switched off.

best,
holger


From fijall at gmail.com  Tue Jul  3 11:01:23 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 3 Jul 2012 11:01:23 +0200
Subject: [pypy-dev] pypy ML aliases at codespeak to be deleted
In-Reply-To: <20120703065918.GG11942@merlinux.eu>
References: <20120703065918.GG11942@merlinux.eu>
Message-ID: <CAK5idxSTeeAHEKHnS0HVFqa_Qw=HgYkpLnp-+FiTPwpJo1iFMQ@mail.gmail.com>

On Tue, Jul 3, 2012 at 8:59 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi all,
>
> in the future you must use pypy-*@python.org addresses, not @codespeak.net
> anymore.
>
> It's been a while since Ralf Hildebrandt and I migrated the pypy
> mailing lists to @python.org.  The old addresses @codespeak.net forward
> mail to the new location but this implicitely disables some spam checks
> that python.org implements.  Therefore, in a few days the codespeak
> aliases will be deleted, namely pypy-dev, pypy-z, pypy-svn. I assume
> also that pypy-issue can be switched off.
>
> best,
> holger
>

Yes, please.

Thank you!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/165b4a91/attachment.html>

From stefan_ml at behnel.de  Tue Jul  3 11:33:16 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 11:33:16 +0200
Subject: [pypy-dev] cpyext performance
Message-ID: <jsue8t$nrh$1@dough.gmane.org>

Hi,

I've been working with one of the modules in the Python benchmark suite,
namely nbody, and tried to make it run a little faster when compiled with
Cython in PyPy. I managed to get a massive speed-up by avoiding some
borrowed references during list iteration and using PySequence_GetItem()
instead, but now some 50-60% of the runtime are spent in Py_DecRef().

I tried debugging into it, but so far, all my attempts to get anything
useful out of PyPy or cpyext have failed.

I applied this obvious change

"""
diff -r 20258fbf10d0 pypy/module/cpyext/pyobject.py
--- a/pypy/module/cpyext/pyobject.py    Sun Jul 01 10:02:26 2012 +0200
+++ b/pypy/module/cpyext/pyobject.py    Tue Jul 03 11:18:22 2012 +0200
@@ -340,13 +340,13 @@
     if obj.c_ob_refcnt == 0:
         state = space.fromcache(RefcountState)
         ptr = rffi.cast(ADDR, obj)
-        if ptr not in state.py_objects_r2w:
+        try:
+            w_obj = state.py_objects_r2w.pop(ptr)
+        except KeyError:
             # this is a half-allocated object, lets call the deallocator
             # without modifying the r2w/w2r dicts
             _Py_Dealloc(space, obj)
         else:
-            w_obj = state.py_objects_r2w[ptr]
-            del state.py_objects_r2w[ptr]
             w_type = space.type(w_obj)
             if not w_type.is_cpytype():
                 _Py_Dealloc(space, obj)
"""

and it gave me a couple of percent, but beyond that, I'd have to know what
paths are actually taken here and where most of the time is spent.

Running it through callgrind yields nothing helpful, so I tried running it
in plain Python. That failed with an OverflowError in ll2ctypes.py:

"""
Traceback (most recent call last):
  File "pypy-1.9/pypy/bin/py.py", line 187, in <module>
    sys.exit(main_(sys.argv))
  File "pypy-1.9/pypy/bin/py.py", line 86, in main_
    space = option.make_objspace(config)
  File "pypy-1.9/pypy/tool/option.py", line 45, in make_objspace
    None, None, ['Space'])
  File "pypy-1.9/pypy/objspace/std/__init__.py", line 1, in <module>
    from pypy.objspace.std.objspace import StdObjSpace
  File "pypy-1.9/pypy/objspace/std/objspace.py", line 18, in <module>
    from pypy.objspace.std.complexobject import W_ComplexObject
  File "pypy-1.9/pypy/objspace/std/complexobject.py", line 6, in <module>
    from pypy.objspace.std.floatobject import W_FloatObject, _hash_float
  File "pypy-1.9/pypy/objspace/std/floatobject.py", line 48, in <module>
    class W_FloatObject(W_AbstractFloatObject):
  File "pypy-1.9/pypy/objspace/std/floatobject.py", line 51, in W_FloatObject
    from pypy.objspace.std.floattype import float_typedef as typedef
  File "pypy-1.9/pypy/objspace/std/floattype.py", line 87, in <module>
    _double_format, _float_format = detect_floatformat()
  File "pypy-1.9/pypy/objspace/std/floattype.py", line 64, in
detect_floatformat
    rffi.cast(rffi.DOUBLEP, buf)[0] = 9006104071832581.0
  File "pypy-1.9/pypy/rpython/lltypesystem/lltype.py", line 1219, in
__setitem__
    self._obj.setitem(i, val)
  File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 620, in setitem
    self._storage.contents._setitem(index, value, boundscheck=False)
  File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 279, in _setitem
    items = self._indexable(index)
  File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 259, in
_indexable
    PtrType = self._get_ptrtype()
  File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 255, in
_get_ptrtype
    raise e
OverflowError: array too large
"""

No idea what this is supposed to tell me, except that I apparently hit yet
another bug in PyPy.

Now, having wasted enough time with this, I'd be happy if someone else
could take over. I put the C code here that I used:

http://cython.org/nbodybench.tar.bz2

You can build it without Cython, just run the included setupdu.py script
over it and call

    pypy -c 'import nbody; print(nbody.test_nbody(1))'

Then try to profile that.

Stefan


From amauryfa at gmail.com  Tue Jul  3 14:55:40 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 3 Jul 2012 14:55:40 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jsue8t$nrh$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
Message-ID: <CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>

Hi,

2012/7/3 Stefan Behnel <stefan_ml at behnel.de>

> I've been working with one of the modules in the Python benchmark suite,
> namely nbody, and tried to make it run a little faster when compiled with
> Cython in PyPy. I managed to get a massive speed-up by avoiding some
> borrowed references during list iteration and using PySequence_GetItem()
> instead, but now some 50-60% of the runtime are spent in Py_DecRef().
>

Don't forget that cpyext reference counts are quite different from CPython:
PySequence_GetItem() needs to *create* a PyObject structure, and the
returned object has a refcount of 1.
Then Py_DECREF() will really *deallocate* the PyObject structure...

This is quite more expensive than the simple refcount increment/decrement
done by CPython.


> OverflowError: array too large
>

Looks like a ctypes bug to me. Which OS, Python, etc. are you using?


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/d436c80e/attachment.html>

From stefan_ml at behnel.de  Tue Jul  3 16:59:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 16:59:51 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
Message-ID: <jsv1d8$oi9$1@dough.gmane.org>

Amaury Forgeot d'Arc, 03.07.2012 14:55:
> 2012/7/3 Stefan Behnel
>> I've been working with one of the modules in the Python benchmark suite,
>> namely nbody, and tried to make it run a little faster when compiled with
>> Cython in PyPy. I managed to get a massive speed-up by avoiding some
>> borrowed references during list iteration and using PySequence_GetItem()
>> instead, but now some 50-60% of the runtime are spent in Py_DecRef().
> 
> Don't forget that cpyext reference counts are quite different from CPython:
> PySequence_GetItem() needs to *create* a PyObject structure, and the
> returned object has a refcount of 1.
> Then Py_DECREF() will really *deallocate* the PyObject structure...

Sure. And using PySequence_GetItem() is still some 10x faster in PyPy than
taking a borrowed reference using PyList_GET_ITEM() and then calling
Py_INCREF() on it, which is what Cython does in CPython because it's way
faster there. The overhead of borrowed references is seriously huge in PyPy.

BTW, are PyObject structures currently cached in a free-list somewhere?
That would be really helpful for the iteration performance.


> This is quite more expensive than the simple refcount increment/decrement
> done by CPython.
> 
>> OverflowError: array too large
> 
> Looks like a ctypes bug to me. Which OS, Python, etc. are you using?

Ah - totally, sure. I accidentally ran the system Py2.5 on 64bit Linux.
Running it with Py2.7 fixes this specific problem, thanks for the hint!
Although it now names the extension module "nbody.so" instead of
"nbody.pypy-19.so". Comprend qui peut ...

After figuring out that I was supposed to enable cpyext manually and
running strace to see what extension module name it is actually looking
for, I failed to make it load the module it just built regardless of how I
named it, so I tried building it within the same run as follows:

  pypy/bin/py.py --withmod-cpyext -c 'import setup; import nbody; \
                                      nbody.test_nbody(1)'  build_ext -i

Doing this, it then fails with the following error:

"""Traceback (most recent call last):
  File ".../pypy/bin/py.py", line 187, in <module>
    sys.exit(main_(sys.argv))
  File ".../pypy/bin/py.py", line 158, in main_
    verbose=interactiveconfig.verbose):
  File ".../pypy/interpreter/main.py", line 103, in run_toplevel
    f()
  File ".../pypy/bin/py.py", line 133, in doit
    main.run_string(command, space=space)
  File ".../pypy/interpreter/main.py", line 59, in run_string
    _run_eval_string(source, filename, space, False)
  File ".../pypy/interpreter/main.py", line 48, in _run_eval_string
    retval = pycode.exec_code(space, w_globals, w_globals)
  File ".../pypy/interpreter/eval.py", line 33, in exec_code
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 834, in IMPORT_NAME
    w_locals, w_fromlist)
  File ".../pypy/interpreter/baseobjspace.py", line 986, in call_function
    return w_func.funccall(*args_w)
  File ".../pypy/interpreter/function.py", line 111, in funccall
    return self.call_args(Arguments(self.space, list(args_w)))
  File ".../pypy/interpreter/function.py", line 59, in call_args
    return self.getcode().funcrun(self, args)
  File ".../pypy/interpreter/gateway.py", line 614, in funcrun
    return BuiltinCode.funcrun_obj(self, func, None, args)
  File ".../pypy/interpreter/gateway.py", line 631, in funcrun_obj
    self.handle_exception(space, e)
  File ".../pypy/interpreter/gateway.py", line 648, in handle_exception
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/gateway.py", line 622, in funcrun_obj
    w_result = activation._run(space, scope_w)
  File "<6076-codegen .../pypy/tool/sourcetools.py:174>", line 3, in _run
  File ".../pypy/module/imp/importing.py", line 271, in importhook
    fromlist_w, tentative=True)
  File ".../pypy/module/imp/importing.py", line 297, in absolute_import
    fromlist_w, tentative)
  File ".../pypy/module/imp/importing.py", line 306, in
absolute_import_with_lock
    fromlist_w, tentative)
  File ".../pypy/module/imp/importing.py", line 367, in _absolute_import
    tentative=tentative)
  File ".../pypy/module/imp/importing.py", line 646, in load_part
    w_mod = load_module(space, w_modulename, find_info)
  File ".../pypy/module/imp/importing.py", line 601, in load_module
    find_info.stream.readall())
  File ".../pypy/module/imp/importing.py", line 911, in load_source_module
    exec_code_module(space, w_mod, code_w)
  File ".../pypy/module/imp/importing.py", line 872, in exec_code_module
    code_w.exec_code(space, w_dict, w_dict)
  File ".../pypy/interpreter/eval.py", line 33, in exec_code
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1029, in CALL_FUNCTION
    self.call_function(oparg)
  File ".../pypy/interpreter/pyopcode.py", line 1012, in call_function
    w_result = self.space.call_args(w_function, args)
  File ".../pypy/objspace/descroperation.py", line 158, in call_args
    return w_obj.call_args(args)
  File ".../pypy/interpreter/function.py", line 59, in call_args
    return self.getcode().funcrun(self, args)
  File ".../pypy/interpreter/pycode.py", line 210, in funcrun
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1029, in CALL_FUNCTION
    self.call_function(oparg)
  File ".../pypy/interpreter/pyopcode.py", line 1012, in call_function
    w_result = self.space.call_args(w_function, args)
  File ".../pypy/objspace/descroperation.py", line 160, in call_args
    return w_obj.call_args(args)
  File ".../pypy/interpreter/function.py", line 471, in call_args
    return space.call_obj_args(self.w_function, self.w_instance, args)
  File ".../pypy/interpreter/baseobjspace.py", line 961, in call_obj_args
    return w_callable.call_obj_args(w_obj, args)
  File ".../pypy/interpreter/function.py", line 63, in call_obj_args
    return self.getcode().funcrun_obj(self, w_obj, args)
  File ".../pypy/interpreter/pycode.py", line 223, in funcrun_obj
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 147, in funccall_valuestack
    return self._flat_pycall(code, nargs, frame)
  File ".../pypy/interpreter/function.py", line 172, in _flat_pycall
    return new_frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1029, in CALL_FUNCTION
    self.call_function(oparg)
  File ".../pypy/interpreter/pyopcode.py", line 1012, in call_function
    w_result = self.space.call_args(w_function, args)
  File ".../pypy/objspace/descroperation.py", line 158, in call_args
    return w_obj.call_args(args)
  File ".../pypy/interpreter/function.py", line 59, in call_args
    return self.getcode().funcrun(self, args)
  File ".../pypy/interpreter/pycode.py", line 210, in funcrun
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1029, in CALL_FUNCTION
    self.call_function(oparg)
  File ".../pypy/interpreter/pyopcode.py", line 1012, in call_function
    w_result = self.space.call_args(w_function, args)
  File ".../pypy/objspace/descroperation.py", line 158, in call_args
    return w_obj.call_args(args)
  File ".../pypy/interpreter/function.py", line 59, in call_args
    return self.getcode().funcrun(self, args)
  File ".../pypy/interpreter/pycode.py", line 210, in funcrun
    return frame.run()
  File ".../pypy/interpreter/pyframe.py", line 141, in run
    return self.execute_frame()
  File ".../pypy/interpreter/pyframe.py", line 175, in execute_frame
    executioncontext)
  File ".../pypy/interpreter/pyopcode.py", line 84, in dispatch
    next_instr = self.handle_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 107, in handle_bytecode
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/pyopcode.py", line 90, in handle_bytecode
    next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
  File ".../pypy/interpreter/pyopcode.py", line 265, in dispatch_bytecode
    res = meth(oparg, next_instr)
  File ".../pypy/interpreter/pyopcode.py", line 1022, in CALL_FUNCTION
    w_result = self.space.call_valuestack(w_function, nargs, self)
  File ".../pypy/interpreter/baseobjspace.py", line 1015, in call_valuestack
    return w_func.funccall_valuestack(nargs, frame)
  File ".../pypy/interpreter/function.py", line 128, in funccall_valuestack
    return code.fastcall_0(self.space, self)
  File ".../pypy/interpreter/gateway.py", line 705, in fastcall_0
    self.handle_exception(space, e)
  File ".../pypy/interpreter/gateway.py", line 648, in handle_exception
    rstackovf.check_stack_overflow()
  File ".../pypy/interpreter/gateway.py", line 700, in fastcall_0
    w_result = self.fastfunc_0(space)
  File ".../pypy/module/posix/interp_posix.py", line 706, in fork
    run_fork_hooks('child', space)
  File ".../pypy/module/posix/interp_posix.py", line 690, in run_fork_hooks
    hook(space)
  File ".../pypy/rpython/lltypesystem/rffi.py", line 241, in wrapper
    res = call_external_function(*real_args)
  File "<6063-codegen .../pypy/rpython/lltypesystem/rffi.py:168>", line 6,
in call_external_function
  File ".../pypy/rpython/lltypesystem/lltype.py", line 1283, in __call__
    return callb(*args)
  File ".../pypy/rpython/lltypesystem/ll2ctypes.py", line 1164, in __call__
    cfunc = get_ctypes_callable(self.funcptr, self.calling_conv)
  File ".../pypy/rpython/lltypesystem/ll2ctypes.py", line 1138, in
get_ctypes_callable
    funcname, place))
NotImplementedError: function 'PyThread_ReInitTLS' not found in library
'/tmp/usession-default-21/module_cache/pypyapi.so'
"""

I have no clue why it thinks it needs to call that function. Any idea?

Stefan


From amauryfa at gmail.com  Tue Jul  3 18:26:33 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 3 Jul 2012 18:26:33 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jsv1d8$oi9$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
Message-ID: <CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>

2012/7/3 Stefan Behnel <stefan_ml at behnel.de>

> > Don't forget that cpyext reference counts are quite different from
> CPython:
> > PySequence_GetItem() needs to *create* a PyObject structure, and the
> > returned object has a refcount of 1.
> > Then Py_DECREF() will really *deallocate* the PyObject structure...
>
> Sure. And using PySequence_GetItem() is still some 10x faster in PyPy than
> taking a borrowed reference using PyList_GET_ITEM() and then calling
> Py_INCREF() on it, which is what Cython does in CPython because it's way
> faster there. The overhead of borrowed references is seriously huge in
> PyPy.
>
> BTW, are PyObject structures currently cached in a free-list somewhere?
> That would be really helpful for the iteration performance.
>

No optimization of any kind have been done in cpyext (it's difficult enough
to get it right...)
A freelist would be a nice thing, but there would still be the cost of
attaching the PyObject to the pypy w_object.

Maybe we should use a weak dictionary to cache the PyObject structure.
This already exists for objects defined and created from C...


> > This is quite more expensive than the simple refcount increment/decrement
> > done by CPython.
> >
> >> OverflowError: array too large
> >
> > Looks like a ctypes bug to me. Which OS, Python, etc. are you using?
>
> Ah - totally, sure. I accidentally ran the system Py2.5 on 64bit Linux.
> Running it with Py2.7 fixes this specific problem, thanks for the hint!
> Although it now names the extension module "nbody.so" instead of
> "nbody.pypy-19.so". Comprend qui peut ...
>
> After figuring out that I was supposed to enable cpyext manually and
> running strace to see what extension module name it is actually looking
> for, I failed to make it load the module it just built regardless of how I
> named it, so I tried building it within the same run as follows:
>
>   pypy/bin/py.py --withmod-cpyext -c 'import setup; import nbody; \
>                                       nbody.test_nbody(1)'  build_ext -i


Ah, but this won't work!
py.py runs on top of CPython, so the PyString_AsString symbol is already
defined by your CPython interpreter!

There is a workaround though: compile your extension module with
   python2.7 pypy/module/cpyext/presetup.py setup.py build_ext -i

presetup.py will patch distutils, and create a module "nbody.pypy-19i.so"
(note the i) which works on top of an *interpreted* pypy.
Among the hacks, all symbols are renamed: #define PyString_AsString
PyPyString_AsString.

Then this should work:
   pypy/bin/py.py --withmod-cpyext -c "import nbody"
*very* slowly of course, but I was able to debug pygames this way!

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/d2bdd608/attachment.html>

From paulo.koch at gmail.com  Tue Jul  3 19:54:28 2012
From: paulo.koch at gmail.com (=?UTF-8?Q?Paulo_K=C3=B6ch?=)
Date: Tue, 3 Jul 2012 18:54:28 +0100
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
Message-ID: <CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>

On Sun, Jul 1, 2012 at 3:40 PM, Paulo K?ch <paulo.koch at gmail.com> wrote:
>
> YouTube now requires me to create a Google account to create a YouTube
> user. We also don't have one of those, right?

Bump.

If I don't get stopped in the next two days, I'll assume we don't have
a Google Account for PyPy yet and proceed to creating it.

From stefan_ml at behnel.de  Tue Jul  3 19:56:31 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 19:56:31 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
Message-ID: <jsvbog$j7c$1@dough.gmane.org>

Amaury Forgeot d'Arc, 03.07.2012 18:26:
> 2012/7/3 Stefan Behnel
>> BTW, are PyObject structures currently cached in a free-list somewhere?
>> That would be really helpful for the iteration performance.
> 
> No optimization of any kind have been done in cpyext (it's difficult enough
> to get it right...)

I'm sure it is.


> A freelist would be a nice thing, but there would still be the cost of
> attaching the PyObject to the pypy w_object.

Any reduction in the cost of passing and cleaning up objects would
dramatically improve the overall performance of the interface.


> Maybe we should use a weak dictionary to cache the PyObject structure.
> This already exists for objects defined and created from C...

That would be really helpful. In particular, it would solve one of the most
annoying problems that extensions currently have: even if you keep a Python
reference to an object, e.g. in a list, its PyObject structure will die
once the last C reference to it is gone. That is really hard to work around
in some cases. It's very common to keep e.g. a Python list (or set) of byte
strings and pass their char* buffer pointers into a C library. That doesn't
currently work with cpyext.


>>>> OverflowError: array too large
>>>
>>> Looks like a ctypes bug to me. Which OS, Python, etc. are you using?
>>
>> Ah - totally, sure. I accidentally ran the system Py2.5 on 64bit Linux.
>> Running it with Py2.7 fixes this specific problem, thanks for the hint!
>> Although it now names the extension module "nbody.so" instead of
>> "nbody.pypy-19.so". Comprend qui peut ...
>>
>> After figuring out that I was supposed to enable cpyext manually and
>> running strace to see what extension module name it is actually looking
>> for, I failed to make it load the module it just built regardless of how I
>> named it, so I tried building it within the same run as follows:
>>
>>   pypy/bin/py.py --withmod-cpyext -c 'import setup; import nbody; \
>>                                       nbody.test_nbody(1)'  build_ext -i
> 
> Ah, but this won't work!
> py.py runs on top of CPython, so the PyString_AsString symbol is already
> defined by your CPython interpreter!

Right. I keep forgetting that. This inherent indirection in PyPy makes
things seriously complicated. (And no, that's not a good thing.)

It would be helpful if it printed an error message giving a hint of why it
failed, instead of just stating that it failed to load the extension (I can
see that it failed, dammit!).


> There is a workaround though: compile your extension module with
>    python2.7 pypy/module/cpyext/presetup.py setup.py build_ext -i
> 
> presetup.py will patch distutils, and create a module "nbody.pypy-19i.so"
> (note the i) which works on top of an *interpreted* pypy.

Is there a reason why an interpreted PyPy cannot always do this? I mean, it
can't work without this, can it?


> Among the hacks, all symbols are renamed: #define PyString_AsString
> PyPyString_AsString.
> 
> Then this should work:
>    pypy/bin/py.py --withmod-cpyext -c "import nbody"

Ok, that did the trick.

This should be in a tutorial somewhere. Or maybe I should just dump it into
a blog post.


> *very* slowly of course, but I was able to debug pygames this way!

The problem is not so much that it's generally slow but that the
performance characteristics of the Python code are likely way different
than those of the translated C code. That's certainly the case for Cython
code, running cProfile over Python code, running it over the compiled
module and running callgrind over it often yields totally different
results. That's why I would prefer running this through callgrind instead
of Python+profile (I noticed that cProfile doesn't work either).

Is there a way to get readable debugging symbols in a translated PyPy that
would tell me what is being executed?

Stefan


From amauryfa at gmail.com  Tue Jul  3 20:08:31 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 3 Jul 2012 20:08:31 +0200
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
	<CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
Message-ID: <CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>

2012/7/3 Paulo K?ch <paulo.koch at gmail.com>

> On Sun, Jul 1, 2012 at 3:40 PM, Paulo K?ch <paulo.koch at gmail.com> wrote:
> >
> > YouTube now requires me to create a Google account to create a YouTube
> > user. We also don't have one of those, right?
>
> Bump.
>
> If I don't get stopped in the next two days, I'll assume we don't have
> a Google Account for PyPy yet and proceed to creating it.


Actually two days ago I created a pypy account and uploaded the videos:
https://www.youtube.com/morepypy
(and yes, there is a already a pypy user)
Since I happen to work for YouTube, I thought I had to do it...

I'll shortly send the password to other core developers.
In the meantime, I still need help to get better title/descriptions for the
videos.


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/4f42269d/attachment-0001.html>

From paulo.koch at gmail.com  Tue Jul  3 20:16:47 2012
From: paulo.koch at gmail.com (=?UTF-8?Q?Paulo_K=C3=B6ch?=)
Date: Tue, 3 Jul 2012 19:16:47 +0100
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
	<CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
	<CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
Message-ID: <CAM_z3tFS_OpdWAzRiBPV_cL1hK5rhzj=YYw9pJE6hGnLARg=aQ@mail.gmail.com>

On Tue, Jul 3, 2012 at 7:08 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Actually two days ago I created a pypy account and uploaded the videos:
> https://www.youtube.com/morepypy
> (and yes, there is a already a pypy user)
> Since I happen to work for YouTube, I thought I had to do it...

Alright! :D Thank you!

> I'll shortly send the password to other core developers.
> In the meantime, I still need help to get better title/descriptions for the
> videos.

I'll leave this to those who actually watched the videos.


We also have to update the wiki. I'll do some sweeping for new videos, also.

From amauryfa at gmail.com  Tue Jul  3 20:35:49 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 3 Jul 2012 20:35:49 +0200
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAM_z3tFS_OpdWAzRiBPV_cL1hK5rhzj=YYw9pJE6hGnLARg=aQ@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
	<CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
	<CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
	<CAM_z3tFS_OpdWAzRiBPV_cL1hK5rhzj=YYw9pJE6hGnLARg=aQ@mail.gmail.com>
Message-ID: <CAGmFidaZZo8VPfG3iU=hkR2egGqdyhMGL6tc524xmKo9mNk-4A@mail.gmail.com>

2012/7/3 Paulo K?ch <paulo.koch at gmail.com>

> On Tue, Jul 3, 2012 at 7:08 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>
> wrote:
> > Actually two days ago I created a pypy account and uploaded the videos:
> > https://www.youtube.com/morepypy
> > (and yes, there is a already a pypy user)
> > Since I happen to work for YouTube, I thought I had to do it...
>
> Alright! :D Thank you!
>
> > I'll shortly send the password to other core developers.
> > In the meantime, I still need help to get better title/descriptions for
> the
> > videos.
>
> I'll leave this to those who actually watched the videos.
>

Ok, I found some descriptions on
http://doc.pypy.org/en/latest/video-index.html



>
>
> We also have to update the wiki. I'll do some sweeping for new videos,
> also.
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/61f2e351/attachment.html>

From michal at bendowski.pl  Tue Jul  3 20:51:37 2012
From: michal at bendowski.pl (Michal Bendowski)
Date: Tue, 3 Jul 2012 20:51:37 +0200
Subject: [pypy-dev] Building the jvm-improvements branch
In-Reply-To: <CABVpgLWoFBnWVsgbv1KsOp-ioKNA_-ZjafaoBjwJ_pyF-33EfQ@mail.gmail.com>
References: <CABVpgLXqtN+XEgrjpn6m64Mhd7yd_v95EWm6i2B2cprAZY-tOw@mail.gmail.com>
	<CAK5idxQcoN3-H5mPY_Ui6zjbm5JYKTjDbJMtHsVNMRWaqRttqQ@mail.gmail.com>
	<CABVpgLWoFBnWVsgbv1KsOp-ioKNA_-ZjafaoBjwJ_pyF-33EfQ@mail.gmail.com>
Message-ID: <CAFKZzE-KNGDSyuSEt_GckXngoAO91Q2rz960f1o0L9DW5rmitw@mail.gmail.com>

Hello Jez,

First of all ? I should have looked into this problem a long time ago and
still haven't found the time. I will be away until next week, but then we
can work this out together (unless you fix it before that time).

As for your setup, I remember having problems with --opt=0 and so I'm using
--opt=2 for now. You can test your setup on the de6baf124c33 changeset
(last one before merging with default, still based on sources from around
march).

Until I come back next week, the only tip I can give is
that RPythonGetThreadIdent is being called, when it should not be with
ootype typesystem (i.e. the jvm backend).

Good luck!

Micha?

On Sun, Jul 1, 2012 at 11:09 AM, Jez <jezreel at gmail.com> wrote:

> On Sun, Jul 1, 2012 at 1:35 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:
>
>> On Sun, Jul 1, 2012 at 2:45 AM, Jez <jezreel at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am thinking about contributing to the JVM backend, and I'm trying to
>>> build the basic translator for a start. I'm trying out the jvm-improvements
>>> branch since there has been some recent work done there, but I can't seem
>>> to get it to build. Here's my setup: 32-bit Ubuntu, OpenJDK 6, building
>>> with `python translate.py --opt=0 --backend=jvm targetpypystandalone.py`.
>>> The translation fails with:
>>>
>>> [translation:ERROR]    File
>>> "/mnt/hgfs/jez/src/pypy/pypy/translator/jvm/database.py", line 493, in
>>> lltype_to_cts
>>> [translation:ERROR]     raise AssertionError("Untranslatable type %s!" %
>>> OOT)
>>> [translation:ERROR]  AssertionError: Untranslatable type * Struct
>>> timespec { c_tv_sec, c_tv_nsec }!
>>>
>>> Am I doing something wrong, or is this failure expected?
>>>
>>> Jez
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>> Hi Jez
>>
>> As a general note - the branches might fail. As a specific note - I can
>> probably help you with this particular problem in a few days. This is some
>> timing thing (not sure what) exposed in an ootype unfriendly way. Do you
>> want more info about this problem to try to fix yourself?
>>
>> Cheers,
>> fijal
>>
>
> Hey Fijal,
>
> Yeah, I don't mind taking a stab at it myself if you could tell me where
> to start :)
>
> Cheers,
> Jez
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/717c0bae/attachment.html>

From papito.dit at gmail.com  Tue Jul  3 23:28:32 2012
From: papito.dit at gmail.com (Michael Sioutis)
Date: Wed, 4 Jul 2012 00:28:32 +0300
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
Message-ID: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>

Dear PyPy team,

I am in the process of writing a paper that will target some AI conference,
and I would like
to ask if there are any relevant publications of yours or in general that
showcase the possible advantages
of trace-based JIT compilation over method-based JIT compilation or static
compilation.

I just want to use 2 or 3 of them as references when I explain why my
implementation is more scalable
and robust when compared to some C/C++/Java implementations (appart from
different data structures and
algorithms).

Thank you :)

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/4ac9d258/attachment.html>

From stefan_ml at behnel.de  Wed Jul  4 08:27:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 08:27:26 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
Message-ID: <jt0nof$uh$1@dough.gmane.org>

Amaury Forgeot d'Arc, 03.07.2012 18:26:
> No optimization of any kind have been done in cpyext (it's difficult enough
> to get it right...)
> A freelist would be a nice thing, but there would still be the cost of
> attaching the PyObject to the pypy w_object.
> 
> Maybe we should use a weak dictionary to cache the PyObject structure.
> This already exists for objects defined and created from C...

Is there a PyPy Wiki somewhere where we could collect this kind of ideas?

There's also stuff like reusing the PyObject on things like
PyNumber_InPlaceAdd() if the refcount is 1, for example.

Stefan


From wlavrijsen at lbl.gov  Wed Jul  4 08:29:52 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 3 Jul 2012 23:29:52 -0700 (PDT)
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
 compiler?
In-Reply-To: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207032255570.24513@pc3-wlav.lbl.gov>

Hi Michael,

> I am in the process of writing a paper that will target some AI conference,
> and I would like to ask if there are any relevant publications of yours or
> in general that showcase the possible advantages of trace-based JIT
> compilation over method-based JIT compilation or static compilation.

this publication has a nice listing of benefits over static compilation:

    http://www.hpl.hp.com/techreports/1999/HPL-1999-78.html

Some of it is outdated, as I find in particular that compiled traces are
very nice on contemporary speculative, out-of-order executing, branch
predicting, hyper-threading CPUs in ways that made no difference on deep
pipeline CPUs of old. It's a great read nonetheless.

Method-based JIT compilation does not play as well with modern CPUs, as
the greatest benefits are had from the inlining of functions and removal
of branches.

Even for C++, vtable indirection and the trampolines for calls across
shared libraries are tough on modern CPUs. Inlining and finalizing calls
helps, but with static profiling you only have one choice of organizing
the code, on one "representative" data set.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From cfbolz at gmx.de  Tue Jul  3 20:45:15 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 03 Jul 2012 20:45:15 +0200
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
	<CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
	<CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
Message-ID: <ae266724-35cc-422a-97f9-b85b9f99a0a9@email.android.com>



Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
>
>Actually two days ago I created a pypy account and uploaded the videos:
>https://www.youtube.com/morepypy
>(and yes, there is a already a pypy user)
>Since I happen to work for YouTube, I thought I had to do it...
>
>I'll shortly send the password to other core developers.
>In the meantime, I still need help to get better title/descriptions for
>the
>videos.

Thanks a lot, Amaury. Mostly of historical interest now, but still very cool to have the videos around. 

Cheers, 

Carl Friedrich 

From fijall at gmail.com  Wed Jul  4 10:48:50 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Jul 2012 10:48:50 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt0nof$uh$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jt0nof$uh$1@dough.gmane.org>
Message-ID: <CAK5idxQEDXte_qV9dY3O_ZduVgHUht4iNpDtWUBG09jWcT7Ypg@mail.gmail.com>

On Wed, Jul 4, 2012 at 8:27 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Amaury Forgeot d'Arc, 03.07.2012 18:26:
> > No optimization of any kind have been done in cpyext (it's difficult
> enough
> > to get it right...)
> > A freelist would be a nice thing, but there would still be the cost of
> > attaching the PyObject to the pypy w_object.
> >
> > Maybe we should use a weak dictionary to cache the PyObject structure.
> > This already exists for objects defined and created from C...
>
> Is there a PyPy Wiki somewhere where we could collect this kind of ideas?
>
> There's also stuff like reusing the PyObject on things like
> PyNumber_InPlaceAdd() if the refcount is 1, for example.
>
> Stefan
>
>
Feel free to use the wiki on bitbucket.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/f858d960/attachment.html>

From fijall at gmail.com  Wed Jul  4 10:49:41 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Jul 2012 10:49:41 +0200
Subject: [pypy-dev] Hello! and Video Tutorials
In-Reply-To: <ae266724-35cc-422a-97f9-b85b9f99a0a9@email.android.com>
References: <CAM_z3tEowiBtxnGjDBgisHZ5v4-4-bEKiqZ6H1PMBtiYPjh+HQ@mail.gmail.com>
	<CAMSv6X35mpdqG4wSRMJnz4T08ymLTxRbv+Hs9iSCPO8hAMRqwQ@mail.gmail.com>
	<20120629173918.GX11942@merlinux.eu>
	<CAM_z3tF56ENatjJNe+1NYA+p+oTMPXb+oDsdFJivhEdzB-FFcQ@mail.gmail.com>
	<CAK5idxRFEGOvHNq0fsNpo46QuEyg3Zz-jr08XbKb8JJAWQF-fg@mail.gmail.com>
	<CAM_z3tHLPEcba-BQojoxAvMpGmuHA-Q8TQ-Tx_dj2vXKHEVc7w@mail.gmail.com>
	<CAM_z3tE=YQSUGxnid3d32uiXESgysktk_ydMOpUv==4=HYOr9g@mail.gmail.com>
	<CAGmFidY=CxnFimLWnQrvOe6dtZJyx90Y1AK6G5F2Q+MnevwgLA@mail.gmail.com>
	<ae266724-35cc-422a-97f9-b85b9f99a0a9@email.android.com>
Message-ID: <CAK5idxTjp_SHqnAK9NwAgGnZXEijgJLB9+Qc88CQng+9RUT1fA@mail.gmail.com>

On Tue, Jul 3, 2012 at 8:45 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

>
>
> Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> >
> >Actually two days ago I created a pypy account and uploaded the videos:
> >https://www.youtube.com/morepypy
> >(and yes, there is a already a pypy user)
> >Since I happen to work for YouTube, I thought I had to do it...
> >
> >I'll shortly send the password to other core developers.
> >In the meantime, I still need help to get better title/descriptions for
> >the
> >videos.
>
> Thanks a lot, Amaury. Mostly of historical interest now, but still very
> cool to have the videos around.
>
> Cheers,
>
> Carl Friedrich
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Thanks amaury!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/31826716/attachment.html>

From stefan_ml at behnel.de  Wed Jul  4 11:21:17 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 11:21:17 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxQEDXte_qV9dY3O_ZduVgHUht4iNpDtWUBG09jWcT7Ypg@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jt0nof$uh$1@dough.gmane.org>
	<CAK5idxQEDXte_qV9dY3O_ZduVgHUht4iNpDtWUBG09jWcT7Ypg@mail.gmail.com>
Message-ID: <jt11ud$cnl$1@dough.gmane.org>

Maciej Fijalkowski, 04.07.2012 10:48:
> On Wed, Jul 4, 2012 at 8:27 AM, Stefan Behnel wrote:
> 
>> Amaury Forgeot d'Arc, 03.07.2012 18:26:
>>> No optimization of any kind have been done in cpyext (it's difficult
>>> enough to get it right...)
>>> A freelist would be a nice thing, but there would still be the cost of
>>> attaching the PyObject to the pypy w_object.
>>>
>>> Maybe we should use a weak dictionary to cache the PyObject structure.
>>> This already exists for objects defined and created from C...
>>
>> Is there a PyPy Wiki somewhere where we could collect this kind of ideas?
>>
>> There's also stuff like reusing the PyObject on things like
>> PyNumber_InPlaceAdd() if the refcount is 1, for example.
>
> Feel free to use the wiki on bitbucket.

Done.

https://bitbucket.org/pypy/pypy/wiki/Speeding%20up%20cpyext

Stefan


From papito.dit at gmail.com  Wed Jul  4 12:43:16 2012
From: papito.dit at gmail.com (Michael Sioutis)
Date: Wed, 4 Jul 2012 13:43:16 +0300
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <alpine.LNX.2.00.1207032255570.24513@pc3-wlav.lbl.gov>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<alpine.LNX.2.00.1207032255570.24513@pc3-wlav.lbl.gov>
Message-ID: <CAC1wwBdHnWGbYC_h_i3TyNKh9a30RgLei9G4MmTxs_4FVCDkgw@mail.gmail.com>

Thank you Vim,

I will use it. I have also found "Trace-based compilation in execution
environments without interpreters" that seems kind of relevant.
http://www.ics.uci.edu/~mbebenit/pubs/pppj-2010.pdf

Regards,
Mike

On Wed, Jul 4, 2012 at 9:29 AM, <wlavrijsen at lbl.gov> wrote:

> Hi Michael,
>
>
>  I am in the process of writing a paper that will target some AI
>> conference,
>> and I would like to ask if there are any relevant publications of yours or
>> in general that showcase the possible advantages of trace-based JIT
>> compilation over method-based JIT compilation or static compilation.
>>
>
> this publication has a nice listing of benefits over static compilation:
>
>    http://www.hpl.hp.com/**techreports/1999/HPL-1999-78.**html<http://www.hpl.hp.com/techreports/1999/HPL-1999-78.html>
>
> Some of it is outdated, as I find in particular that compiled traces are
> very nice on contemporary speculative, out-of-order executing, branch
> predicting, hyper-threading CPUs in ways that made no difference on deep
> pipeline CPUs of old. It's a great read nonetheless.
>
> Method-based JIT compilation does not play as well with modern CPUs, as
> the greatest benefits are had from the inlining of functions and removal
> of branches.
>
> Even for C++, vtable indirection and the trampolines for calls across
> shared libraries are tough on modern CPUs. Inlining and finalizing calls
> helps, but with static profiling you only have one choice of organizing
> the code, on one "representative" data set.
>
> Best regards,
>            Wim
> --
> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/2ecb541a/attachment.html>

From cfbolz at gmx.de  Wed Jul  4 14:44:12 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 04 Jul 2012 14:44:12 +0200
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
 compiler?
In-Reply-To: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
Message-ID: <4FF43A9C.5000203@gmx.de>

On 07/03/2012 11:28 PM, Michael Sioutis wrote:
> Dear PyPy team,
>
> I am in the process of writing a paper that will target some AI
> conference, and I would like
> to ask if there are any relevant publications of yours or in general
> that showcase the possible advantages
> of trace-based JIT compilation over method-based JIT compilation or
> static compilation.
>
> I just want to use 2 or 3 of them as references when I explain why my
> implementation is more scalable
> and robust when compared to some C/C++/Java implementations (appart from
> different data structures and
> algorithms).
>
> Thank you :)

The two PyPy papers about our tracing JIT are these:

http://dl.acm.org/citation.cfm?id=1565827
http://dl.acm.org/citation.cfm?id=2069172.2069181

However, they don't really precisely answer you question, why tracing 
JITs are better.

Cheers,

Carl Friedrich



From fijall at gmail.com  Wed Jul  4 15:20:03 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Jul 2012 15:20:03 +0200
Subject: [pypy-dev] PyPy sprint in Cape Town
Message-ID: <CAK5idxQmjfwsD6JAtVNSjSy=v6J3AOOymHv6ooZreV4EEfrbFQ@mail.gmail.com>

Hello everyone.

There is a Pycon South Africa happening in Cape Town on Oct 4th-5th and I
would like to encourage everyone to come it's a really awesome destination!

As an extra addon, there is a high chance we'll host a full PyPy sprint in
the following week (7th - 13th of October 2012) accompanying the
conference. Consider this a pre-announcement, since there are quite a few
details to figure out.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/bcd3854e/attachment.html>

From dje.gcc at gmail.com  Wed Jul  4 16:19:33 2012
From: dje.gcc at gmail.com (David Edelsohn)
Date: Wed, 4 Jul 2012 10:19:33 -0400
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <4FF43A9C.5000203@gmx.de>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
Message-ID: <CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>

On Wed, Jul 4, 2012 at 8:44 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> On 07/03/2012 11:28 PM, Michael Sioutis wrote:
>>
>> Dear PyPy team,
>>
>> I am in the process of writing a paper that will target some AI
>> conference, and I would like
>> to ask if there are any relevant publications of yours or in general
>> that showcase the possible advantages
>> of trace-based JIT compilation over method-based JIT compilation or
>> static compilation.
>>
>> I just want to use 2 or 3 of them as references when I explain why my
>> implementation is more scalable
>> and robust when compared to some C/C++/Java implementations (appart from
>> different data structures and
>> algorithms).
>>
>> Thank you :)
>
>
> The two PyPy papers about our tracing JIT are these:
>
> http://dl.acm.org/citation.cfm?id=1565827
> http://dl.acm.org/citation.cfm?id=2069172.2069181
>
> However, they don't really precisely answer you question, why tracing JITs
> are better.

I don't think it is clear that tracing JITs are better.

Tracing often implies some other characteristics and features, such as
level of inlining, specialization and guards versus arbitrary control
flow, but method JITs are not prevented from adapting these
techniques.

The advantages and benefits are a lot subtler than tracing versus method JITs.

- David

From cfbolz at gmx.de  Wed Jul  4 16:32:13 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 04 Jul 2012 16:32:13 +0200
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
 compiler?
In-Reply-To: <CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
Message-ID: <4FF453ED.8020308@gmx.de>


On 07/04/2012 04:19 PM, David Edelsohn wrote:
 > On Wed, Jul 4, 2012 at 8:44 AM, Carl Friedrich Bolz <cfbolz at gmx.de> 
wrote:
 >> However, they don't really precisely answer you question, why 
tracing JITs
 >> are better.
 >
 > I don't think it is clear that tracing JITs are better.
 >
 > Tracing often implies some other characteristics and features, such as
 > level of inlining, specialization and guards versus arbitrary control
 > flow, but method JITs are not prevented from adapting these
 > techniques.
 >
 > The advantages and benefits are a lot subtler than tracing versus 
method JITs.

Indeed, I don't think anybody really knows the answer yet, under which
circumstances which of the two methods is better. Somebody should try
to do a controlled study .

Cheers,

Carl Friedrich

From papito.dit at gmail.com  Wed Jul  4 17:29:30 2012
From: papito.dit at gmail.com (Michael Sioutis)
Date: Wed, 4 Jul 2012 18:29:30 +0300
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <4FF453ED.8020308@gmx.de>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
Message-ID: <CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>

Thank you all very much!

Of course I don't want to present trace-based JITs as the best choice, but
leave some hints on the "possible"
advantages that it "possibly" offers, just to put an additional comment on
my graphs.

So it has come down to this:
Trace-based JIT compilation allows more specialized optimizations on code
and is potentially
more well suited for modern CPUs \cite{PyPy,Dynamo}
...
here we also see the possible advantages of tracing JIT since ... scalable
... robust.

So I think I'm not lying to anyone who reads it :)

Thank you all,
Mike

On Wed, Jul 4, 2012 at 5:32 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

>
> On 07/04/2012 04:19 PM, David Edelsohn wrote:
> > On Wed, Jul 4, 2012 at 8:44 AM, Carl Friedrich Bolz <cfbolz at gmx.de>
> wrote:
> >> However, they don't really precisely answer you question, why tracing
> JITs
> >> are better.
> >
> > I don't think it is clear that tracing JITs are better.
> >
> > Tracing often implies some other characteristics and features, such as
> > level of inlining, specialization and guards versus arbitrary control
> > flow, but method JITs are not prevented from adapting these
> > techniques.
> >
> > The advantages and benefits are a lot subtler than tracing versus method
> JITs.
>
> Indeed, I don't think anybody really knows the answer yet, under which
> circumstances which of the two methods is better. Somebody should try
> to do a controlled study .
>
> Cheers,
>
> Carl Friedrich
>
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/f799ccb5/attachment-0001.html>

From dje.gcc at gmail.com  Wed Jul  4 17:49:52 2012
From: dje.gcc at gmail.com (David Edelsohn)
Date: Wed, 4 Jul 2012 11:49:52 -0400
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
	<CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
Message-ID: <CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>

On Wed, Jul 4, 2012 at 11:29 AM, Michael Sioutis <papito.dit at gmail.com> wrote:
> Thank you all very much!
>
> Of course I don't want to present trace-based JITs as the best choice, but
> leave some hints on the "possible"
> advantages that it "possibly" offers, just to put an additional comment on
> my graphs.
>
> So it has come down to this:
> Trace-based JIT compilation allows more specialized optimizations on code
> and is potentially
> more well suited for modern CPUs \cite{PyPy,Dynamo}
> ...
> here we also see the possible advantages of tracing JIT since ... scalable
> ... robust.
>
> So I think I'm not lying to anyone who reads it :)

Michael,

Personally, I would not cite Dynamo to assert anything about the
benefits of tracing JITs. Some of Dynamo's success was due to the
quirkiness of the HP PA-RISC architecture, which also certainly would
not be a good reference for modern CPUs.

Also, I believe that Firefox, Chrome and Safari JavaScript JITs now
are all method-based.

The Meta-tracing JITs is another design feature of PyPy and provides
some early optimization for dynamic languages.

But as far as tracing JITs in general, the "advantage" may be more
about the JIT design decisions that a tracing JIT encourages versus
the default design of a method JIT.  One really needs to consider a
tracing JIT versus a method JIT with extensive profiling,
specialization, etc.  In other words, it's not a dichotomy, but a
distribution.

- David

From paulo.koch at gmail.com  Wed Jul  4 17:59:37 2012
From: paulo.koch at gmail.com (=?UTF-8?Q?Paulo_K=C3=B6ch?=)
Date: Wed, 4 Jul 2012 16:59:37 +0100
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
	<CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
	<CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>
Message-ID: <CAM_z3tEnE+jPyf+AhFkuYUNMFE3V2nKbMKoJE9k5QueexJ7vJw@mail.gmail.com>

On Wed, Jul 4, 2012 at 4:49 PM, David Edelsohn <dje.gcc at gmail.com> wrote:
> Also, I believe that Firefox, Chrome and Safari JavaScript JITs now
> are all method-based.

This needs to be confirmed but, I really think Firefox's TraceMonkey
is a tracing JIT and Chrome's v8 is a method JIT.

From alex.gaynor at gmail.com  Wed Jul  4 18:06:50 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 4 Jul 2012 09:06:50 -0700
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <CAM_z3tEnE+jPyf+AhFkuYUNMFE3V2nKbMKoJE9k5QueexJ7vJw@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
	<CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
	<CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>
	<CAM_z3tEnE+jPyf+AhFkuYUNMFE3V2nKbMKoJE9k5QueexJ7vJw@mail.gmail.com>
Message-ID: <CAFRnB2Un1o1HZiPWhAbfB9-PHz_b2rr6RriJYcGnknwx5izobg@mail.gmail.com>

On Wed, Jul 4, 2012 at 8:59 AM, Paulo K?ch <paulo.koch at gmail.com> wrote:

> On Wed, Jul 4, 2012 at 4:49 PM, David Edelsohn <dje.gcc at gmail.com> wrote:
> > Also, I believe that Firefox, Chrome and Safari JavaScript JITs now
> > are all method-based.
>
> This needs to be confirmed but, I really think Firefox's TraceMonkey
> is a tracing JIT and Chrome's v8 is a method JIT.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Tracemonkey *was* a tracing JIT, at this point it's being (if not has
already been) removed in favor of IonMonkey which is a method JIT.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/63554937/attachment.html>

From berker.peksag at gmail.com  Wed Jul  4 18:21:57 2012
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Wed, 4 Jul 2012 19:21:57 +0300
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <CAFRnB2Un1o1HZiPWhAbfB9-PHz_b2rr6RriJYcGnknwx5izobg@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
	<CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
	<CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>
	<CAM_z3tEnE+jPyf+AhFkuYUNMFE3V2nKbMKoJE9k5QueexJ7vJw@mail.gmail.com>
	<CAFRnB2Un1o1HZiPWhAbfB9-PHz_b2rr6RriJYcGnknwx5izobg@mail.gmail.com>
Message-ID: <CAF4280++3kFtFxrZ-1=DSaVsv=vEQ6=MboV18dkE7NooyJNtXw@mail.gmail.com>

On Wed, Jul 4, 2012 at 7:06 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>
> On Wed, Jul 4, 2012 at 8:59 AM, Paulo K?ch <paulo.koch at gmail.com> wrote:
>>
>> On Wed, Jul 4, 2012 at 4:49 PM, David Edelsohn <dje.gcc at gmail.com> wrote:
>> > Also, I believe that Firefox, Chrome and Safari JavaScript JITs now
>> > are all method-based.
>>
>> This needs to be confirmed but, I really think Firefox's TraceMonkey
>> is a tracing JIT and Chrome's v8 is a method JIT.
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
> Tracemonkey *was* a tracing JIT, at this point it's being (if not has
> already been) removed in favor of IonMonkey which is a method JIT.

Yes, it's removed from SpiderMonkey:

https://bugzilla.mozilla.org/show_bug.cgi?id=698201
http://blog.mozilla.org/nnethercote/2011/11/23/memshrink-progress-report-week-23/

--Berker

>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From stefan_ml at behnel.de  Wed Jul  4 20:53:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 20:53:07 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jsvbog$j7c$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org>
Message-ID: <jt23ej$rq1$1@dough.gmane.org>

Stefan Behnel, 03.07.2012 19:56:
> Amaury Forgeot d'Arc, 03.07.2012 18:26:
>> Then this should work:
>>    pypy/bin/py.py --withmod-cpyext -c "import nbody"
>>
>> *very* slowly of course, but I was able to debug pygames this way!
> 
> The problem is not so much that it's generally slow but that the
> performance characteristics of the Python code are likely way different
> than those of the translated C code. That's certainly the case for Cython
> code, running cProfile over Python code, running it over the compiled
> module and running callgrind over it often yields totally different
> results. That's why I would prefer running this through callgrind instead
> of Python+profile (I noticed that cProfile doesn't work either).

Actually, it did work. I just had to enable the _lsprof module.

However, it now prints a trace of every C-API function that it calls, e.g.

"""
<function PyTuple_CheckExact at 0x42da5a0>  DONE
<function PyList_CheckExact at 0x43cc5a0>  DONE
<function PySequence_Size at 0x43e1300>  DONE
<function PySequence_ITEM at 0x43ef300>  DONE
<function PySequence_ITEM at 0x43ef300>  DONE
<function PySequence_ITEM at 0x43ef300>  DONE
<function Py_DecRef at 0x40aea38> <function subtype_dealloc at 0x4322990>
<function PyObject_dealloc at 0x43466f0> <function PyObject_Del at
0x43464f8>  DONE
 DONE
 DONE
 DONE
"""

Is there a way to disable that? That level of verbosity could be a bit costly.

Stefan


From amauryfa at gmail.com  Wed Jul  4 21:12:31 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 4 Jul 2012 21:12:31 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt23ej$rq1$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt23ej$rq1$1@dough.gmane.org>
Message-ID: <CAGmFidYTXyRkbR9Kka=KB7_sehwJ_zadjF-i-H4OKe=OSrmTsw@mail.gmail.com>

2012/7/4 Stefan Behnel <stefan_ml at behnel.de>:
> However, it now prints a trace of every C-API function that it calls, e.g.
>
> """
> <function PyTuple_CheckExact at 0x42da5a0>  DONE
> <function PyList_CheckExact at 0x43cc5a0>  DONE
> <function PySequence_Size at 0x43e1300>  DONE
> <function PySequence_ITEM at 0x43ef300>  DONE
> <function PySequence_ITEM at 0x43ef300>  DONE
> <function PySequence_ITEM at 0x43ef300>  DONE
> <function Py_DecRef at 0x40aea38> <function subtype_dealloc at 0x4322990>
> <function PyObject_dealloc at 0x43466f0> <function PyObject_Del at
> 0x43464f8>  DONE
>  DONE
>  DONE
>  DONE
> """
>
> Is there a way to disable that? That level of verbosity could be a bit costly.

in module/cpyext/api.py, just set DEBUG_WRAPPER to False.

-- 
Amaury Forgeot d'Arc

From carlosmf.pt at gmail.com  Thu Jul  5 00:32:27 2012
From: carlosmf.pt at gmail.com (Carlos Ferreira)
Date: Wed, 4 Jul 2012 23:32:27 +0100
Subject: [pypy-dev] AttributeError: 'Message' object has no attribute
	'__sizeof__'
Message-ID: <CAJpYY6UA9hPz6LmPbfem9equqNXM9nYkFAzG7Qh_St4HTTFKRA@mail.gmail.com>

Hello all!
I'm having a simple problem when using PyPy in windows 7.

My program which only uses 2 external packages (SimPy and networkx) and
which also works well under CPython 2.7.2, is throwing an AttributeError
Exception, when run in PyPy 1.9.

*Traceback (most recent call last):
  File "C:\Languages\pypy-1.9\lib-python\2.7\multiprocessing\process.py",
line 258, in _bootstrap
    self.run()
  File "C:\Languages\pypy-1.9\lib-python\2.7\multiprocessing\process.py",
line 114, in run
    self._target(*self._args, **self._kwargs)
  File "E:\Users\Claymore\Desktop\SimPy-Sims\Main.py", line 1197, in
ProcessMain
    main(SCENARIO_SIZE = size, maximumDepthSearch = depth, trial = trial)
  File "E:\Users\Claymore\Desktop\SimPy-Sims\Main.py", line 1139, in main
    simulate(until = 999999999)
  File "C:\Languages\pypy-1.9\site-packages\SimPy\Globals.py", line 61, in
simulate
    return sim.simulate(until = until)
  File "C:\Languages\pypy-1.9\site-packages\SimPy\Simulation.py", line 581,
in simulate
    step()
  File "C:\Languages\pypy-1.9\site-packages\SimPy\Simulation.py", line 525,
in step
    resultTuple = proc._nextpoint.next()
  File "E:\Users\Claymore\Desktop\SimPy-Sims\Main.py", line 700, in run

agentOwnerRef.networkInterfaces[fromNetworkInterface].TransmitMessage(message)
  File "E:\Users\Claymore\Desktop\SimPy-Sims\Main.py", line 184, in
TransmitMessage

GlobalShelve["GlobalNetwork"][str(self.agentName)]["NetworkInterfacesStats"][self.interfaceName]["MessagesTraceInput"].append({"arrivalTime":now(),
"messageID":message.messageID, "sizeInBytes":message.__sizeof__()})
AttributeError: 'Message' object has no attribute '__sizeof__'
*

Can anyone help me here ?

-- 

Carlos Miguel Ferreira
Researcher at Telecommunications Institute
Aveiro - Portugal
Work E-mail - cmf at av.it.pt
University of Aveiro E-mail -> cmf at ua.pt
MSN Contact -> carlosmf.pt at gmail.com
Skype & GTalk -> carlosmf.pt at gmail.com
LinkedIn -> http://www.linkedin.com/in/carlosmferreira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120704/c4c2c7db/attachment.html>

From stefan_ml at behnel.de  Thu Jul  5 09:44:18 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 09:44:18 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jsvbog$j7c$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org>
Message-ID: <jt3gki$r2$1@dough.gmane.org>

Stefan Behnel, 03.07.2012 19:56:
> Amaury Forgeot d'Arc, 03.07.2012 18:26:
>> Then this should work:
>>    pypy/bin/py.py --withmod-cpyext -c "import nbody"
>>
>> *very* slowly of course, but I was able to debug pygames this way!
> 
> The problem is not so much that it's generally slow but that the
> performance characteristics of the Python code are likely way different
> than those of the translated C code. That's certainly the case for Cython
> code, running cProfile over Python code, running it over the compiled
> module and running callgrind over it often yields totally different
> results. That's why I would prefer running this through callgrind instead
> of Python+profile.

Ok, I finally managed to get a profile and it turned out to be completely
useless. I tried profiling inside of PyPy, but that doesn't tell me
anything about cpyext. I then tried profiling the interpreted PyPy itself
from CPython, but that only gives me a profile of the interpreted code,
which is clearly different from what the compiled code does. It spends lots
of time doing calls, for example, and the top time consuming function is
"getattr", followed by "build_ctypes_array". At least "api.py:wrapper" is
also somewhat up on the list, but it's just drowning in the noise.

Back to that question then:

> Is there a way to get readable debugging symbols in a translated PyPy that
> would tell me what is being executed?

Stefan


From amauryfa at gmail.com  Thu Jul  5 10:26:41 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 5 Jul 2012 10:26:41 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt3gki$r2$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
Message-ID: <CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>

2012/7/5 Stefan Behnel <stefan_ml at behnel.de>

> Back to that question then:
>
> > Is there a way to get readable debugging symbols in a translated PyPy
> that
> > would tell me what is being executed?
>

I fear that pypy standard distribution calls "strip" on the resulting
binary.
You could translate pypy yourself, I'm quite sure it contains debug info
already and it's quite easy to call "make debug" anyway.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/0b744e61/attachment.html>

From amauryfa at gmail.com  Thu Jul  5 10:36:39 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 5 Jul 2012 10:36:39 +0200
Subject: [pypy-dev] AttributeError: 'Message' object has no attribute
	'__sizeof__'
In-Reply-To: <CAJpYY6UA9hPz6LmPbfem9equqNXM9nYkFAzG7Qh_St4HTTFKRA@mail.gmail.com>
References: <CAJpYY6UA9hPz6LmPbfem9equqNXM9nYkFAzG7Qh_St4HTTFKRA@mail.gmail.com>
Message-ID: <CAGmFidYm7oRjTAhtwypxrfcF7VMfc8wphE0VmpPnyJZ1VnLnPw@mail.gmail.com>

2012/7/5 Carlos Ferreira <carlosmf.pt at gmail.com>

> *  File "E:\Users\Claymore\Desktop\SimPy-Sims\Main.py", line 184, in
> TransmitMessage
>
> GlobalShelve["GlobalNetwork"][str(self.agentName)]["NetworkInterfacesStats"][self.interfaceName]["MessagesTraceInput"].append({"arrivalTime":now(),
> "messageID":message.messageID, "sizeInBytes":message.__sizeof__()})
> AttributeError: 'Message' object has no attribute '__sizeof__'
> *
>
> Can anyone help me here ?


PyPy does not implement __sizeof__.
You have two choices here:
remove the call from the code (it's seems to be only for information
purpose),
or implement the __sizeof__ method in the Message class, by returning some
random value.


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/e270e22c/attachment.html>

From holger at merlinux.eu  Thu Jul  5 11:09:09 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 5 Jul 2012 09:09:09 +0000
Subject: [pypy-dev] pypy ML aliases at codespeak to be deleted
In-Reply-To: <CAK5idxSTeeAHEKHnS0HVFqa_Qw=HgYkpLnp-+FiTPwpJo1iFMQ@mail.gmail.com>
References: <20120703065918.GG11942@merlinux.eu>
	<CAK5idxSTeeAHEKHnS0HVFqa_Qw=HgYkpLnp-+FiTPwpJo1iFMQ@mail.gmail.com>
Message-ID: <20120705090909.GH18336@merlinux.eu>

On Tue, Jul 03, 2012 at 11:01 +0200, Maciej Fijalkowski wrote:
> On Tue, Jul 3, 2012 at 8:59 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi all,
> >
> > in the future you must use pypy-*@python.org addresses, not @codespeak.net
> > anymore.
> >
> > It's been a while since Ralf Hildebrandt and I migrated the pypy
> > mailing lists to @python.org.  The old addresses @codespeak.net forward
> > mail to the new location but this implicitely disables some spam checks
> > that python.org implements.  Therefore, in a few days the codespeak
> > aliases will be deleted, namely pypy-dev, pypy-z, pypy-svn. I assume
> > also that pypy-issue can be switched off.
> >
> > best,
> > holger
> >
> 
> Yes, please.

done.  Let's watch out for any missing mail.

holger

From fijall at gmail.com  Thu Jul  5 11:01:51 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 5 Jul 2012 11:01:51 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
Message-ID: <CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>

On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

>
> 2012/7/5 Stefan Behnel <stefan_ml at behnel.de>
>
>> Back to that question then:
>>
>> > Is there a way to get readable debugging symbols in a translated PyPy
>> that
>> > would tell me what is being executed?
>>
>
> I fear that pypy standard distribution calls "strip" on the resulting
> binary.
> You could translate pypy yourself, I'm quite sure it contains debug info
> already and it's quite easy to call "make debug" anyway.
>
>
Default build (not the distribution or nightly, you have to trasnlate
yourself), contains debug info. You can do make lldebug to have even more,
but even without it it's usable with valgrind.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/52607e1d/attachment-0001.html>

From papito.dit at gmail.com  Thu Jul  5 11:54:41 2012
From: papito.dit at gmail.com (Michael Sioutis)
Date: Thu, 5 Jul 2012 12:54:41 +0300
Subject: [pypy-dev] Any publications regarding PyPy / trace-based JIT
	compiler?
In-Reply-To: <CAF4280++3kFtFxrZ-1=DSaVsv=vEQ6=MboV18dkE7NooyJNtXw@mail.gmail.com>
References: <CAC1wwBdE2SEDjn+CvSytAAN+eVeaR5fP9RTg9KPOJjRT6SOTCg@mail.gmail.com>
	<4FF43A9C.5000203@gmx.de>
	<CAGWvnykRMOjyss__Zw1r64xyJ=VhtLL6JfB40at4snmNc=m4eA@mail.gmail.com>
	<4FF453ED.8020308@gmx.de>
	<CAC1wwBe5xcoL7EBotU7ymZ9K17krWKcU1JN27jXfmPNddbA7jg@mail.gmail.com>
	<CAGWvnynW=y_zniPWJFUHB_tuKdjAx685V+LudbWC6GPPCbUZVw@mail.gmail.com>
	<CAM_z3tEnE+jPyf+AhFkuYUNMFE3V2nKbMKoJE9k5QueexJ7vJw@mail.gmail.com>
	<CAFRnB2Un1o1HZiPWhAbfB9-PHz_b2rr6RriJYcGnknwx5izobg@mail.gmail.com>
	<CAF4280++3kFtFxrZ-1=DSaVsv=vEQ6=MboV18dkE7NooyJNtXw@mail.gmail.com>
Message-ID: <CAC1wwBfA2GWwuodk3LrzzdVcjaAE91G4f9jbih11J5ZU67jVNg@mail.gmail.com>

So, after discussing with a Professor of our Department with a very good
knowledge on
the issue, he basically was in the line of "...JIT design decisions that a
tracing JIT encourages versus
the default design of a method JIT...", but to be more specific on the
advantages he claimed that:
"tracing JITs can discover optimization opportunities in common dynamic
execution paths that are not apparent to a static compiler or a
method-based JIT compiler".

I think this is a nice comment overall, that the PyPy publications and the
features they describe can
back up nicely.

Thank you :)
Mike

On Wed, Jul 4, 2012 at 7:21 PM, Berker Peksa? <berker.peksag at gmail.com>wrote:

> On Wed, Jul 4, 2012 at 7:06 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> >
> >
> > On Wed, Jul 4, 2012 at 8:59 AM, Paulo K?ch <paulo.koch at gmail.com> wrote:
> >>
> >> On Wed, Jul 4, 2012 at 4:49 PM, David Edelsohn <dje.gcc at gmail.com>
> wrote:
> >> > Also, I believe that Firefox, Chrome and Safari JavaScript JITs now
> >> > are all method-based.
> >>
> >> This needs to be confirmed but, I really think Firefox's TraceMonkey
> >> is a tracing JIT and Chrome's v8 is a method JIT.
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >
> >
> > Tracemonkey *was* a tracing JIT, at this point it's being (if not has
> > already been) removed in favor of IonMonkey which is a method JIT.
>
> Yes, it's removed from SpiderMonkey:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=698201
>
> http://blog.mozilla.org/nnethercote/2011/11/23/memshrink-progress-report-week-23/
>
> --Berker
>
> >
> > Alex
> >
> > --
> > "I disapprove of what you say, but I will defend to the death your right
> to
> > say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> > "The people's good is the highest law." -- Cicero
> >
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/c27aad62/attachment.html>

From stefan_ml at behnel.de  Thu Jul  5 14:35:29 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 14:35:29 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
	<CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
Message-ID: <jt41mh$85s$1@dough.gmane.org>

Maciej Fijalkowski, 05.07.2012 11:01:
> On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote:
>> 2012/7/5 Stefan Behnel
>>> Back to that question then:
>>>
>>>> Is there a way to get readable debugging symbols in a translated PyPy
>>>> that would tell me what is being executed?
>>
>> I fear that pypy standard distribution calls "strip" on the resulting
>> binary.
>> You could translate pypy yourself, I'm quite sure it contains debug info
>> already and it's quite easy to call "make debug" anyway.
>
> Default build (not the distribution or nightly, you have to trasnlate
> yourself), contains debug info.

Ah, yes. Given a suitably large machine and enough time to do other stuff,
that did the trick for me. Here's the result:

http://cython.org/callgrind-pypy-nbody.png

As you can see, make_ref() and Py_DecRef() combined account for almost 80%
of the runtime. So the expected gain from optimising the PyObject handling
is *huge*.

The first thing that pops up from the graph is the different calls through
generic_cpy_call(). That looks way to generic for something as performance
critical as Py_DecRef().

Also, what's that recursive "stackwalk()" thing doing?

Stefan


From fijall at gmail.com  Thu Jul  5 14:50:04 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 5 Jul 2012 14:50:04 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt41mh$85s$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
	<CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
	<jt41mh$85s$1@dough.gmane.org>
Message-ID: <CAK5idxSewMM1YuYoLzWwp6mSG=ShHP6er96cYS0WvF3BQUCcHQ@mail.gmail.com>

On Thu, Jul 5, 2012 at 2:35 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Maciej Fijalkowski, 05.07.2012 11:01:
> > On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote:
> >> 2012/7/5 Stefan Behnel
> >>> Back to that question then:
> >>>
> >>>> Is there a way to get readable debugging symbols in a translated PyPy
> >>>> that would tell me what is being executed?
> >>
> >> I fear that pypy standard distribution calls "strip" on the resulting
> >> binary.
> >> You could translate pypy yourself, I'm quite sure it contains debug info
> >> already and it's quite easy to call "make debug" anyway.
> >
> > Default build (not the distribution or nightly, you have to trasnlate
> > yourself), contains debug info.
>
> Ah, yes. Given a suitably large machine and enough time to do other stuff,
> that did the trick for me. Here's the result:
>
> http://cython.org/callgrind-pypy-nbody.png
>
> As you can see, make_ref() and Py_DecRef() combined account for almost 80%
> of the runtime. So the expected gain from optimising the PyObject handling
> is *huge*.
>
> The first thing that pops up from the graph is the different calls through
> generic_cpy_call(). That looks way to generic for something as performance
> critical as Py_DecRef().
>
> Also, what's that recursive "stackwalk()" thing doing?
>
> Stefan
>

Haha :)

This is related to garbage collection - it scans the stack for GC pointers
to save them I think. We might think that it's a bit too much to do it for
every single call there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/2b4974fd/attachment.html>

From fijall at gmail.com  Thu Jul  5 14:55:06 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 5 Jul 2012 14:55:06 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt41mh$85s$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
	<CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
	<jt41mh$85s$1@dough.gmane.org>
Message-ID: <CAK5idxTSC1Eu_=OtqH+pzHhrRWPJOg-f5DQCF7+Y8asgqP+59w@mail.gmail.com>

On Thu, Jul 5, 2012 at 2:35 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Maciej Fijalkowski, 05.07.2012 11:01:
> > On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote:
> >> 2012/7/5 Stefan Behnel
> >>> Back to that question then:
> >>>
> >>>> Is there a way to get readable debugging symbols in a translated PyPy
> >>>> that would tell me what is being executed?
> >>
> >> I fear that pypy standard distribution calls "strip" on the resulting
> >> binary.
> >> You could translate pypy yourself, I'm quite sure it contains debug info
> >> already and it's quite easy to call "make debug" anyway.
> >
> > Default build (not the distribution or nightly, you have to trasnlate
> > yourself), contains debug info.
>
> Ah, yes. Given a suitably large machine and enough time to do other stuff,
> that did the trick for me. Here's the result:
>
> http://cython.org/callgrind-pypy-nbody.png
>
> As you can see, make_ref() and Py_DecRef() combined account for almost 80%
> of the runtime. So the expected gain from optimising the PyObject handling
> is *huge*.
>
> The first thing that pops up from the graph is the different calls through
> generic_cpy_call(). That looks way to generic for something as performance
> critical as Py_DecRef().
>

Note that this is RPython not C. This is not a "generic_call" - this is a
specialized version of generic call based on some arguments. That means
that whatever could have been determined by those attributes have been
constant folded away. In RPython this is very typical - you write a generic
version that specializes based on some values to arrive at a specific
version.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120705/b65aec5e/attachment.html>

From stefan_ml at behnel.de  Thu Jul  5 20:23:15 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 20:23:15 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt41mh$85s$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
	<CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
	<jt41mh$85s$1@dough.gmane.org>
Message-ID: <jt4m2k$i7o$1@dough.gmane.org>

Stefan Behnel, 05.07.2012 14:35:
> http://cython.org/callgrind-pypy-nbody.png

I've set up a job on our build server to run a couple of (simple)
benchmarks comparing Cython's current performance under CPython and PyPy.
Note that these are C-API intensive benchmarks by design (they are compiled
from Python code), even though they use static type annotations for
optimisation. Despite of what you might think about these benchmarks in
general, I think they are quite suitable for cpyext.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-pypy/

The latest results are here:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-pypy/lastSuccessfulBuild/artifact/bench_chart.html

They currently run 100-200x slower through cpyext than in CPython 2.7. The
build job always uses the latest nightly build of PyPy, so any changes in
Cython or PyPy will usually show up within the next 24 hours.

The build job also archives the generated .c files (and the original
sources, including the HTML version). If anyone wants to play with them,
you can just download the C file, build it with distutils, import it and
then call its "main(number_of_iterations)" function. The C code works in
both PyPy and CPython, although the actual C-API calls differ somewhat
between the two.

Stefan


From n210241048576 at gmail.com  Fri Jul  6 07:07:27 2012
From: n210241048576 at gmail.com (Robert Grosse)
Date: Fri, 6 Jul 2012 01:07:27 -0400
Subject: [pypy-dev] pypy translation failures
Message-ID: <CALJjGitcfhmxv0+_xiV6b0VjNhNzdJHpm8Mwq+DL06qGZJ+-uA@mail.gmail.com>

Hello, I am having difficulties translating Pypy, but googling searching
did not reveal any mentions of similar problems, so I am not sure what to
do.

I am not able to successfully translate Pypy, using any combination of
options I have tried. Here are my results so far.

Ubuntu 12.04 64 bit
Trying to translate pypy sandbox from latest source release (I can find the
actual hash number if necessary).
I used PYPY_GC_MAX_DELTA=200MB and loop_longevity=300 as suggested to
reduce memory usage, along with -O2 and --sandbox
The translator produced an executable, but attempting to run this
executable with or without pypy_interact.py leads to an immediate
termination with the message
Fatalerror during initialization out of memory

Windows 7 64 bit
Same steps as above except without max delta, and using the 1.9 source
release.
The sandbox again fails with the Fatalerror out of memory. It also produces
some weird characters before terminating which I'm guessing are the
marshalled stdout from the sandbox.

Windows 7 64 bit
Attempt to translate plain Pypy, with -Ojit but no loop longevity.
Translation fails partway through with a MemoryError despite several GB of
free memory.

I have tried both CPython 2.7 and prebuilt Pypy binaries to do the
translating, and nothing I do will work. I can't find any mentions of
others experiencing similar problems, but the fact that the problems show
up for me with multiple versions on multiple machines with different OSs
indicates that something serious is wrong. Does anyone have any idea what
the problem might be?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120706/2c24d1de/attachment-0001.html>

From matti.picus at gmail.com  Fri Jul  6 08:06:09 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 06 Jul 2012 09:06:09 +0300
Subject: [pypy-dev] pypy translation failures
In-Reply-To: <CALJjGitcfhmxv0+_xiV6b0VjNhNzdJHpm8Mwq+DL06qGZJ+-uA@mail.gmail.com>
References: <CALJjGitcfhmxv0+_xiV6b0VjNhNzdJHpm8Mwq+DL06qGZJ+-uA@mail.gmail.com>
Message-ID: <4FF68051.8060107@gmail.com>

On 6/07/2012 8:07 AM, Robert Grosse wrote:
> Hello, I am having difficulties translating Pypy, but googling 
> searching did not reveal any mentions of similar problems, so I am not 
> sure what to do.
>
> I am not able to successfully translate Pypy, using any combination of 
> options I have tried. Here are my results so far.
>
> Ubuntu 12.04 64 bit
> Trying to translate pypy sandbox from latest source release (I can 
> find the actual hash number if necessary).
> I used PYPY_GC_MAX_DELTA=200MB and loop_longevity=300 as suggested to 
> reduce memory usage, along with -O2 and --sandbox
> The translator produced an executable, but attempting to run this 
> executable with or without pypy_interact.py leads to an immediate 
> termination with the message
> Fatalerror during initialization out of memory
>
> Windows 7 64 bit
> Same steps as above except without max delta, and using the 1.9 source 
> release.
> The sandbox again fails with the Fatalerror out of memory. It also 
> produces some weird characters before terminating which I'm guessing 
> are the marshalled stdout from the sandbox.
>
> Windows 7 64 bit
> Attempt to translate plain Pypy, with -Ojit but no loop longevity. 
> Translation fails partway through with a MemoryError despite several 
> GB of free memory.
>
> I have tried both CPython 2.7 and prebuilt Pypy binaries to do the 
> translating, and nothing I do will work. I can't find any mentions of 
> others experiencing similar problems, but the fact that the problems 
> show up for me with multiple versions on multiple machines with 
> different OSs indicates that something serious is wrong. Does anyone 
> have any idea what the problem might be?
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
What is the actual command line you are using?
Note that the last trunk known to complete translation correctly was 
around July 3, version 20258fbf10d0. You can see the pypy buildbot 
history at http://buildbot.pypy.org/nightly/trunk 
<http://buildbot.pypy.org/nightly/trunk/> The "tip" version does not 
translate, someone should fix it soonish.

To translate on windows, you must use pypy (only 32 bit is available)  
or 32 bit cpython and follow the directions here 
http://pypy.readthedocs.org/en/latest/windows.html?highlight=windows
especially this bit: editbin /largeaddressaware <translating python>

Matti


From fijall at gmail.com  Fri Jul  6 14:24:49 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 6 Jul 2012 14:24:49 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jt4m2k$i7o$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org>
	<CAGmFidbmPu32f9TQuX7_Z8QUQUXygMfadDnCPHgJbUyUzXF26A@mail.gmail.com>
	<jsv1d8$oi9$1@dough.gmane.org>
	<CAGmFidbpXXrY3F8_=QgYP5ANHC4hoh9x7soqMo234BopPZD0NA@mail.gmail.com>
	<jsvbog$j7c$1@dough.gmane.org> <jt3gki$r2$1@dough.gmane.org>
	<CAGmFidb-=gL6SaJdwX-6gc2tOHyYzEWLXXtpN6pLE6ogeNH_1w@mail.gmail.com>
	<CAK5idxTnHtaewh_V5tHnH44cveuoo=0Dm9Y6Lnf6WBVwbapCGA@mail.gmail.com>
	<jt41mh$85s$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
Message-ID: <CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>

Thanks!

i might think bad about those benchmarks representing python workloads,
howecer they are very likely good for cpyext. good job.

On Thursday, July 5, 2012, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 05.07.2012 14:35:
>> http://cython.org/callgrind-pypy-nbody.png
>
> I've set up a job on our build server to run a couple of (simple)
> benchmarks comparing Cython's current performance under CPython and PyPy.
> Note that these are C-API intensive benchmarks by design (they are
compiled
> from Python code), even though they use static type annotations for
> optimisation. Despite of what you might think about these benchmarks in
> general, I think they are quite suitable for cpyext.
>
>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-pypy/
>
> The latest results are here:
>
>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-pypy/lastSuccessfulBuild/artifact/bench_chart.html
>
> They currently run 100-200x slower through cpyext than in CPython 2.7. The
> build job always uses the latest nightly build of PyPy, so any changes in
> Cython or PyPy will usually show up within the next 24 hours.
>
> The build job also archives the generated .c files (and the original
> sources, including the HTML version). If anyone wants to play with them,
> you can just download the C file, build it with distutils, import it and
> then call its "main(number_of_iterations)" function. The C code works in
> both PyPy and CPython, although the actual C-API calls differ somewhat
> between the two.
>
> Stefan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120706/743c32af/attachment.html>

From jonathan at slenders.be  Sat Jul  7 17:25:22 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Sat, 7 Jul 2012 17:25:22 +0200
Subject: [pypy-dev] Circular reference when importing struct from a pypy
	sandbox.
Message-ID: <CAKfyG3z5=z4p5=yVe4ScjM6bAezw1-dcxF14qTpRszup2eL0Qg@mail.gmail.com>

Hi everyone,

Not sure whether this is sandbox related, but I get a circular reference
when I want to import struct.py
The problem seems to be in the MixedModule import which get to import
himself somehow...
Anyone here who likes to take a look at the trackback below?

Actually, I'm not entirely sure whether I have the correct sys.path inside
the sandbox.
But anyway, I already had a great experience with the sandboxing mode.
(Right now, I'm rewriting sandlib to something usable in Twisted Matrix,
amazing technologies they are.)

Cheers,
Jonathan


Traceback (most recent call last):

  File "app_main.py", line 51, in run_toplevel

  File "/application/main.py", line 35, in <module>

    import struct

  File "/bin/pypy/module/struct/__init__.py", line 8, in <module>

    from pypy.interpreter.mixedmodule import MixedModule

  File "/bin/pypy/interpreter/mixedmodule.py", line 1, in <module>

    from pypy.interpreter.module import Module

  File "/bin/pypy/interpreter/module.py", line 5, in <module>

    from pypy.interpreter.baseobjspace import Wrappable

  File "/bin/pypy/interpreter/baseobjspace.py", line 2, in <module>

    from pypy.interpreter.executioncontext import ExecutionContext,
ActionFlag
  File "/bin/pypy/interpreter/executioncontext.py", line 2, in <module>

    from pypy.interpreter.error import OperationError

  File "/bin/pypy/interpreter/error.py", line 2, in <module>

    from pypy.rlib import jit

  File "/bin/pypy/rlib/jit.py", line 3, in <module>

    import py

  File "/bin/pypy/bin/py.py", line 15, in <module>

    from pypy.tool import option

  File "/bin/pypy/tool/option.py", line 4, in <module>

    from pypy.config.pypyoption import get_pypy_config

  File "/bin/pypy/config/pypyoption.py", line 7, in <module>

    from pypy.config.translationoption import IS_64_BITS

  File "/bin/pypy/config/translationoption.py", line 6, in <module>

    from pypy.config.support import detect_number_of_processors

  File "/bin/pypy/config/support.py", line 5, in <module>

    import re, sys, os, subprocess

  File "/bin/lib-python/2.7/subprocess.py", line 398, in <module>

    import signal

  File "/bin/pypy/module/signal/__init__.py", line 2, in <module>

    from pypy.interpreter.mixedmodule import MixedModule

ImportError: cannot import name 'MixedModule'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120707/db9befff/attachment.html>

From amauryfa at gmail.com  Sat Jul  7 18:00:17 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 7 Jul 2012 18:00:17 +0200
Subject: [pypy-dev] Circular reference when importing struct from a pypy
	sandbox.
In-Reply-To: <CAKfyG3z5=z4p5=yVe4ScjM6bAezw1-dcxF14qTpRszup2eL0Qg@mail.gmail.com>
References: <CAKfyG3z5=z4p5=yVe4ScjM6bAezw1-dcxF14qTpRszup2eL0Qg@mail.gmail.com>
Message-ID: <CAGmFidavSfrfW5oteDAqAQYgG2mNgnVdOi2Z4xDJNY9zmCCaBA@mail.gmail.com>

2012/7/7 Jonathan Slenders <jonathan at slenders.be>:
> Not sure whether this is sandbox related, but I get a circular reference
> when I want to import struct.py
> The problem seems to be in the MixedModule import which get to import
> himself somehow...
> Anyone here who likes to take a look at the trackback below?
>
> Actually, I'm not entirely sure whether I have the correct sys.path inside
> the sandbox.
> But anyway, I already had a great experience with the sandboxing mode.
> (Right now, I'm rewriting sandlib to something usable in Twisted Matrix,
> amazing technologies they are.)

It seems you have the /bin/pypy/module directory in your sys.path.
Did you set it manually?
Could you remove it and try again?


-- 
Amaury Forgeot d'Arc

From jonathan at slenders.be  Sat Jul  7 19:28:21 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Sat, 7 Jul 2012 19:28:21 +0200
Subject: [pypy-dev] Circular reference when importing struct from a pypy
	sandbox.
In-Reply-To: <CAGmFidavSfrfW5oteDAqAQYgG2mNgnVdOi2Z4xDJNY9zmCCaBA@mail.gmail.com>
References: <CAKfyG3z5=z4p5=yVe4ScjM6bAezw1-dcxF14qTpRszup2eL0Qg@mail.gmail.com>
	<CAGmFidavSfrfW5oteDAqAQYgG2mNgnVdOi2Z4xDJNY9zmCCaBA@mail.gmail.com>
Message-ID: <CAKfyG3zMCexLqV-DYm=HTN0OW5M1wiYqUQPmUbzuHObx2q2hcg@mail.gmail.com>

Thanks Amaury,

That's right. I had /bin/pypy/module in my path.

Am I correct in assuming that there does not exist a _struct.py yet for the
pypy sandbox?
/lib-python/2.7/struct.py includes _struct.

The problem was that I'd like to use a json library for passing messages
between the sandbox and it's hypervisor.
Now I patched simplejson to not use the struct and decimal libraries. And
that works wonderful, but it would be even better if I could use the plain
original simplejson. :)


Jonathan



2012/7/7 Amaury Forgeot d'Arc <amauryfa at gmail.com>

> 2012/7/7 Jonathan Slenders <jonathan at slenders.be>:
> > Not sure whether this is sandbox related, but I get a circular reference
> > when I want to import struct.py
> > The problem seems to be in the MixedModule import which get to import
> > himself somehow...
> > Anyone here who likes to take a look at the trackback below?
> >
> > Actually, I'm not entirely sure whether I have the correct sys.path
> inside
> > the sandbox.
> > But anyway, I already had a great experience with the sandboxing mode.
> > (Right now, I'm rewriting sandlib to something usable in Twisted Matrix,
> > amazing technologies they are.)
>
> It seems you have the /bin/pypy/module directory in your sys.path.
> Did you set it manually?
> Could you remove it and try again?
>
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120707/b1ae40fe/attachment.html>

From alex.pyattaev at gmail.com  Mon Jul  9 20:53:19 2012
From: alex.pyattaev at gmail.com (Alex Pyattaev)
Date: Mon, 09 Jul 2012 11:53:19 -0700
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
Message-ID: <134193137.B9fqzCHM9S@hunter-laptop>

And now for something completely different:
Would not it make dramatically more sense to just finish the cppyy branch and 
get a working extension-making scheme that actually works? I have a project 
now that uses C++ extensions and python quite extensively, and the only thing 
stopping me from migrating to cppyy from SWIG is the fact that cppyy is still 
so unfinished and requires rebuild of pypy with root libraries and all the 
other tedious things, that slow down deployment.
BR,
Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120709/e9eb05ee/attachment.html>

From stefan_ml at behnel.de  Mon Jul  9 20:58:58 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 09 Jul 2012 20:58:58 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <134193137.B9fqzCHM9S@hunter-laptop>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
Message-ID: <jtf9li$hlc$1@dough.gmane.org>

Alex Pyattaev, 09.07.2012 20:53:
> And now for something completely different:
> Would not it make dramatically more sense to just finish the cppyy branch and 
> get a working extension-making scheme that actually works? I have a project 
> now that uses C++ extensions and python quite extensively, and the only thing 
> stopping me from migrating to cppyy from SWIG is the fact that cppyy is still 
> so unfinished and requires rebuild of pypy with root libraries and all the 
> other tedious things, that slow down deployment.

Different use cases.

Stefan


From wlavrijsen at lbl.gov  Tue Jul 10 00:04:07 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 9 Jul 2012 15:04:07 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <134193137.B9fqzCHM9S@hunter-laptop>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
Message-ID: <alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>

Hi Alex,

> cppyy is still so unfinished

provide a prioritized list of what's still missing for you? I'm following a
more or less random walk otherwise, with most work going into the CINT backend
at the moment.

> and requires rebuild of pypy with root libraries and all the other tedious
> things, that slow down deployment.

Just Reflex w/o ROOT should work now, but rebuilding pypy-c is going to remain
needed for a while, I'm afraid. The only idea that I have for another method
that allows deferring to runtime, is to use ctypes instead of rffi for the CAPI.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From fijall at gmail.com  Tue Jul 10 00:11:58 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 10 Jul 2012 00:11:58 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
Message-ID: <CAK5idxTYE-qPRwzsde-iPyjCnxJMYDTS9hx35LWd4mf24-mpuA@mail.gmail.com>

On Tue, Jul 10, 2012 at 12:04 AM, <wlavrijsen at lbl.gov> wrote:

> Hi Alex,
>
>
>  cppyy is still so unfinished
>>
>
> provide a prioritized list of what's still missing for you? I'm following a
> more or less random walk otherwise, with most work going into the CINT
> backend
> at the moment.
>
>
>  and requires rebuild of pypy with root libraries and all the other tedious
>> things, that slow down deployment.
>>
>
> Just Reflex w/o ROOT should work now, but rebuilding pypy-c is going to
> remain
> needed for a while, I'm afraid. The only idea that I have for another
> method
> that allows deferring to runtime, is to use ctypes instead of rffi for the
> CAPI.
>
> Best regards,
>            Wim


Or CFFI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120710/2ba7f83e/attachment.html>

From wlavrijsen at lbl.gov  Tue Jul 10 00:17:21 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 9 Jul 2012 15:17:21 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxTYE-qPRwzsde-iPyjCnxJMYDTS9hx35LWd4mf24-mpuA@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<CAK5idxTYE-qPRwzsde-iPyjCnxJMYDTS9hx35LWd4mf24-mpuA@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207091513320.23262@pc3-wlav.lbl.gov>

Hi Maciej,

> Or CFFI

did I miss an announcement and has it been added to PyPy? I can't find it ...

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From fijall at gmail.com  Tue Jul 10 00:29:11 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 10 Jul 2012 00:29:11 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207091513320.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<CAK5idxTYE-qPRwzsde-iPyjCnxJMYDTS9hx35LWd4mf24-mpuA@mail.gmail.com>
	<alpine.LNX.2.00.1207091513320.23262@pc3-wlav.lbl.gov>
Message-ID: <CAK5idxQ+ADaAMpigSKZe6YyKTMSBYL72WVcnnooM1as6yqXamg@mail.gmail.com>

On Tue, Jul 10, 2012 at 12:17 AM, <wlavrijsen at lbl.gov> wrote:

> Hi Maciej,
>
>  Or CFFI
>>
>
> did I miss an announcement and has it been added to PyPy? I can't find it
> ...
>
> Thanks,
>
>      Wim
>

it's in the process. look on ffi-backend branch


>  --
> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120710/8cf59bcb/attachment-0001.html>

From wlavrijsen at lbl.gov  Tue Jul 10 01:05:14 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 9 Jul 2012 16:05:14 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxQ+ADaAMpigSKZe6YyKTMSBYL72WVcnnooM1as6yqXamg@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<CAK5idxTYE-qPRwzsde-iPyjCnxJMYDTS9hx35LWd4mf24-mpuA@mail.gmail.com>
	<alpine.LNX.2.00.1207091513320.23262@pc3-wlav.lbl.gov>
	<CAK5idxQ+ADaAMpigSKZe6YyKTMSBYL72WVcnnooM1as6yqXamg@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207091536310.23262@pc3-wlav.lbl.gov>

Hi Maciej,

> it's in the process. look on ffi-backend branch

looking good!

Any chance of getting a W_CTypeFunc.call() taking args instead of args_w? I'd
like to pull in the function pointer from app-level, so that loading in the
.so's is deferred to run-time, but then do the call from interpreter level
from an unwrapped function pointer with interpreter-level args, so the args
need no re-wrapping.

As an aside, and I know this is hard to do portably (Windows ...), but could
load_library() take dl flags for dlopen?

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From matti.picus at gmail.com  Tue Jul 10 10:13:59 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 10 Jul 2012 18:13:59 +1000
Subject: [pypy-dev] Developer selection for Py3k
In-Reply-To: <4FECE3EC.7030000@gmail.com>
References: <4FECE3EC.7030000@gmail.com>
Message-ID: <4FFBE447.30205@gmail.com>

I would be honored for consideration, however there are at least two 
complications:

I am still awaiting formal approval from my full-time employer to allow 
me to take on a second
     part-time job for the Software Conservancy Foundation, a process 
that will take
     until mid-August at the earliest.
I am still working out the tax and other implications of issuing an 
invoice for the work
    to an American tax entity. This too should be clear within a month 
or so.
Matti Picus

On 29/06/2012 9:08 AM, Antonio Cuni wrote:
> Hi all,
>
> as you probably know, the Py3k [1] proposal is getting funded thanks to our
> generous donors.
>
> During the first round, three of use were selected: me, Alex Gaynor and
> Benjamin Peterson.  However, due to unforeseen unavailability of Alex and
> Benjamin, we are now looking for one more developer to help with the py3k work.
>
> If you are interested in getting paid to work on the Py3k proposal, please
> apply by replying to this email.
>
> To be applicable you need to be an experienced PyPy developer, preferably with
> a previous experience on the Python Interpreter part of PyPy.
>
> ciao,
> Anto
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev



From estama at gmail.com  Tue Jul 10 11:46:04 2012
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Tue, 10 Jul 2012 12:46:04 +0300
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
Message-ID: <4FFBF9DC.3080200@gmail.com>

On 10/07/12 01:04, wlavrijsen at lbl.gov wrote:
> provide a prioritized list of what's still missing for you? I'm following a
> more or less random walk otherwise, with most work going into the CINT
> backend
> at the moment.

Do jitted callbacks work?

My main use case has to do with sqlite UDFs written in python that are 
called from sqlite's side.

Thank you all for your work on pypy.

l.

From amauryfa at gmail.com  Tue Jul 10 12:45:13 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Jul 2012 12:45:13 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <4FFBF9DC.3080200@gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
Message-ID: <CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>

2012/7/10 Eleytherios Stamatogiannakis <estama at gmail.com>:
> On 10/07/12 01:04, wlavrijsen at lbl.gov wrote:
>>
>> provide a prioritized list of what's still missing for you? I'm following
>> a
>> more or less random walk otherwise, with most work going into the CINT
>> backend
>> at the moment.
>
>
> Do jitted callbacks work?
>
> My main use case has to do with sqlite UDFs written in python that are
> called from sqlite's side.

A callback called from C cannot be jitted, unless it has a loop in Python code.
The loop in the sqlite library does not count. The JIT needs a
complete understanding of all operations in the loop...

-- 
Amaury Forgeot d'Arc

From estama at gmail.com  Tue Jul 10 12:48:51 2012
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Tue, 10 Jul 2012 13:48:51 +0300
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
Message-ID: <4FFC0893.2000605@gmail.com>

On 10/07/12 13:45, Amaury Forgeot d'Arc wrote:
> A callback called from C cannot be jitted, unless it has a loop in Python code.
> The loop in the sqlite library does not count. The JIT needs a
> complete understanding of all operations in the loop...
>
> -- Amaury Forgeot d'Arc

 From what i understand you need to hit a certain path some times before 
jitting it.

Would it be possible to define that this function will be externally 
called (from C) millions of times so keep it jitted at all times?

l.


From fijall at gmail.com  Tue Jul 10 12:55:30 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 10 Jul 2012 12:55:30 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
Message-ID: <CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>

On Tue, Jul 10, 2012 at 12:45 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com>wrote:

> 2012/7/10 Eleytherios Stamatogiannakis <estama at gmail.com>:
> > On 10/07/12 01:04, wlavrijsen at lbl.gov wrote:
> >>
> >> provide a prioritized list of what's still missing for you? I'm
> following
> >> a
> >> more or less random walk otherwise, with most work going into the CINT
> >> backend
> >> at the moment.
> >
> >
> > Do jitted callbacks work?
> >
> > My main use case has to do with sqlite UDFs written in python that are
> > called from sqlite's side.
>
> A callback called from C cannot be jitted, unless it has a loop in Python
> code.
> The loop in the sqlite library does not count. The JIT needs a
> complete understanding of all operations in the loop...
>
> --
> Amaury Forgeot d'Arc
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Amaury - not true, we can JIT functions from the start. However, they would
still accept wrapped arguments. I can think about a simple way to enable
jitting from the start without the need though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120710/02bd26a1/attachment.html>

From naveen.garg at gmail.com  Tue Jul 10 17:55:10 2012
From: naveen.garg at gmail.com (Naveen Garg)
Date: Tue, 10 Jul 2012 10:55:10 -0500
Subject: [pypy-dev] translation on RHEL 6.x
Message-ID: <CABipEAtPEtWjVBjQA9Wsdd0X8WRB2eCTDkOTNotGK8vRffx8pw@mail.gmail.com>

I am trying to translate a pypy sandbox on RHEL 6.
I am having problems because I think libffi-devel is not available in the
EPEL repo.
Has anyone had any success / build instructions on RHEL 6 ? Or do I really
have to revert to RHEL 5 ?
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120710/16562d99/attachment.html>

From dmalcolm at redhat.com  Tue Jul 10 19:10:19 2012
From: dmalcolm at redhat.com (David Malcolm)
Date: Tue, 10 Jul 2012 13:10:19 -0400
Subject: [pypy-dev] translation on RHEL 6.x
In-Reply-To: <CABipEAtPEtWjVBjQA9Wsdd0X8WRB2eCTDkOTNotGK8vRffx8pw@mail.gmail.com>
References: <CABipEAtPEtWjVBjQA9Wsdd0X8WRB2eCTDkOTNotGK8vRffx8pw@mail.gmail.com>
Message-ID: <1341940219.31178.17.camel@surprise>

On Tue, 2012-07-10 at 10:55 -0500, Naveen Garg wrote:
> I am trying to translate a pypy sandbox on RHEL 6.
> I am having problems because I think libffi-devel is not available in
> the EPEL repo.
> Has anyone had any success / build instructions on RHEL 6 ?
I build PyPy for RHEL 6 within the EPEL repo:
  http://koji.fedoraproject.org/koji/buildinfo?buildID=325048

IIRC libffi-devel is a core part of RHEL (as opposed to being part of
EPEL).

However, I'm not building with --sandbox, if that's what you're
referring to.

You can see the specfile for my pypy packages here:
http://pkgs.fedoraproject.org/gitweb/?p=pypy.git;a=shortlog;h=refs/heads/el6

I do some patching FWIW; in particular, I'm applying this patch:
http://pkgs.fedoraproject.org/gitweb/?p=pypy.git;a=blob;f=config.patch;h=a11f397710bde5ed285c7360c70375d0452274eb;hb=615fd77fe04457d552feff9d10986589b35a6148

Hope this is helpful
Dave



From wlavrijsen at lbl.gov  Tue Jul 10 19:38:22 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 10:38:22 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>

Hi all,

On Tue, 10 Jul 2012, Maciej Fijalkowski wrote:
> Amaury - not true, we can JIT functions from the start. However, they would
> still accept wrapped arguments. I can think about a simple way to enable
> jitting from the start without the need though.

if the JITted function still requires wrapped arguments, then an additional
C function needs to be generated to be passed as the actual pointer, to receive
the arguments and wrap them.

Generating such a C function by the backend then requires a C-API for the calls
to do the wrapping. That PyPy C-API is a long awaited project. :)

As an aside, would it be sufficient to compile the callback (rather than JIT
it)? I.e. to get code that doesn't have guards or the ability to fallback? The
input args are coming from C, so the types are guaranteed, and if the code is
purely numeric, with no fallback to the interpreter anywhere, then acquiring
the GIL shouldn't be needed.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From amauryfa at gmail.com  Tue Jul 10 19:57:43 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Jul 2012 19:57:43 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
Message-ID: <CAGmFidY2FCsHVVpm7sv7qus9yaZBE=QNj1HY9kVuW09S1n5xLg@mail.gmail.com>

2012/7/10  <wlavrijsen at lbl.gov>:
> Generating such a C function by the backend then requires a C-API for the
> calls
> to do the wrapping. That PyPy C-API is a long awaited project. :)

Do you have an idea what this API would look like?
Then I can help with the implementation :)

-- 
Amaury Forgeot d'Arc

From stefan_ml at behnel.de  Tue Jul 10 20:01:18 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jul 2012 20:01:18 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
Message-ID: <jthqle$rs7$1@dough.gmane.org>

wlavrijsen at lbl.gov, 10.07.2012 19:38:
> On Tue, 10 Jul 2012, Maciej Fijalkowski wrote:
>> Amaury - not true, we can JIT functions from the start. However, they would
>> still accept wrapped arguments. I can think about a simple way to enable
>> jitting from the start without the need though.
> 
> if the JITted function still requires wrapped arguments

There's no reason it would require wrapped arguments. The input types are
known from the static low-level function type, so the JIT compiler can just
work with them and adapt the function that is being called.

These things are trickier in a non-JIT environment, but we are currently
working on a general framework to support low-level calls through the
CPython ecosystem (I mentioned that before).

Stefan


From wlavrijsen at lbl.gov  Tue Jul 10 20:10:46 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 11:10:46 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jthqle$rs7$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
Message-ID: <alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>

Hi Stefan,

> There's no reason it would require wrapped arguments.

that completely depends on how it is generated, of course, and in the context
of calls within Python (pypy-c), it makes sense to have the entry point of
the function expect wrapped arguments, and have the exit point re-wrap.

> The input types are
> known from the static low-level function type, so the JIT compiler can just
> work with them and adapt the function that is being called.

Yes, except that the above only follows after this:

> These things are trickier in a non-JIT environment, but we are currently
> working on a general framework to support low-level calls through the
> CPython ecosystem (I mentioned that before).

I.e., there should first be a good method of delivering the low-level info.
Right now, that delivery method is the act of unwrapping itself (that is,
the wrapped types carry the low-level info).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From stefan_ml at behnel.de  Tue Jul 10 20:23:18 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jul 2012 20:23:18 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
Message-ID: <jthrum$681$1@dough.gmane.org>

wlavrijsen at lbl.gov, 10.07.2012 20:10:
>> There's no reason it would require wrapped arguments.
> 
> that completely depends on how it is generated, of course, and in the context
> of calls within Python (pypy-c), it makes sense to have the entry point of
> the function expect wrapped arguments, and have the exit point re-wrap.

There's no reason you can't have multiple entry points to the same code.
Cython has been doing this for a while, and I'm sure PyPy's JIT does it, too.


>> The input types are
>> known from the static low-level function type, so the JIT compiler can just
>> work with them and adapt the function that is being called.
> 
> Yes, except that the above only follows after this:
> 
>> These things are trickier in a non-JIT environment, but we are currently
>> working on a general framework to support low-level calls through the
>> CPython ecosystem (I mentioned that before).
> 
> I.e., there should first be a good method of delivering the low-level info.

Sure, that's why we've been working on a specification for a protocol here.
Basically, the caller and the callee would have to agree on a common
signature at runtime out of the given N:M set of choices. A JIT compiler
would obviously prefer generating a suitable signature, given the set of
signatures that the other side presents.


> Right now, that delivery method is the act of unwrapping itself (that is,
> the wrapped types carry the low-level info).

I have no idea what you mean, but you make it sound backwards for the case
of a callback.

Stefan


From wlavrijsen at lbl.gov  Tue Jul 10 20:37:54 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 11:37:54 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jthrum$681$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
Message-ID: <alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>

Hi Stefan,

On Tue, 10 Jul 2012, Stefan Behnel wrote:
> wlavrijsen at lbl.gov, 10.07.2012 20:10:
>> that completely depends on how it is generated, of course, and in the context
>> of calls within Python (pypy-c), it makes sense to have the entry point of
>> the function expect wrapped arguments, and have the exit point re-wrap.
> There's no reason you can't have multiple entry points to the same code.
> Cython has been doing this for a while, and I'm sure PyPy's JIT does it, too.

I'm not sure whether the PyPy JIT does that, as an entry point somewhere in the
middle of a compiled trace would bypass guards, and as said re-wrap at an exit
point is needed (not to mention if a guard fails half-way during the trace and
the code drops in the black hole).

But a JIT expert should provide the details of what's possible. :)

>> Right now, that delivery method is the act of unwrapping itself (that is,
>> the wrapped types carry the low-level info).
> I have no idea what you mean, but you make it sound backwards for the case
> of a callback.

See Anto's talk at EuroPython for a better explanation of what I'm saying (or
trying to say anyway):

    https://ep2012.europython.eu/conference/talks/pypy-jit-under-the-hood

The type information going into the traces comes from the type information
that the interpreter has. I.e. from the wrapped types. This isn't backwards
AFAICS, since a general Python function (callback or otherwise) can receive
any type (and of course, is allowed to fail by raising an exception if it
can't deal with the types). It's only once the trace has been generated,
that the types are fixed for the trace (with a fallback option, or course).

Now, retrofitting the callback mechanism on top of this, may very well be
backwards, which is why I think we all do agree that a better handshake is
needed. And if somebody could pretty please code that up for PyPy, so that
I can use it. :)

As-is, I could even live with CFFI funcptrs taking a wrapped tuple of args.
After all, wrapping is easy and fast at the interpreter level. However, the
JIT will be blind to it.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From stefan_ml at behnel.de  Tue Jul 10 20:48:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jul 2012 20:48:51 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
Message-ID: <jthtej$jde$1@dough.gmane.org>

wlavrijsen at lbl.gov, 10.07.2012 20:37:
> On Tue, 10 Jul 2012, Stefan Behnel wrote:
>> wlavrijsen at lbl.gov, 10.07.2012 20:10:
>>> that completely depends on how it is generated, of course, and in the
>>> context
>>> of calls within Python (pypy-c), it makes sense to have the entry point of
>>> the function expect wrapped arguments, and have the exit point re-wrap.
>> There's no reason you can't have multiple entry points to the same code.
>> Cython has been doing this for a while, and I'm sure PyPy's JIT does it,
>> too.
> 
> I'm not sure whether the PyPy JIT does that, as an entry point somewhere in
> the
> middle of a compiled trace would bypass guards, and as said re-wrap at an exit
> point is needed (not to mention if a guard fails half-way during the trace and
> the code drops in the black hole).
> 
> But a JIT expert should provide the details of what's possible. :)
> 
>>> Right now, that delivery method is the act of unwrapping itself (that is,
>>> the wrapped types carry the low-level info).
>> I have no idea what you mean, but you make it sound backwards for the case
>> of a callback.
> 
> See Anto's talk at EuroPython for a better explanation of what I'm saying (or
> trying to say anyway):
> 
>    https://ep2012.europython.eu/conference/talks/pypy-jit-under-the-hood
> 
> The type information going into the traces comes from the type information
> that the interpreter has. I.e. from the wrapped types. This isn't backwards
> AFAICS, since a general Python function (callback or otherwise) can receive
> any type (and of course, is allowed to fail by raising an exception if it
> can't deal with the types). It's only once the trace has been generated,
> that the types are fixed for the trace (with a fallback option, or course).

Ok, then in the case of a callback, the runtime could simply start with
totally stupid code that packs the low-level arguments into Python
arguments, attaches the low-level type information to them and then passes
them into the function. The JIT should then see in its trace what types
originally came in and optimise the argument packing away, shouldn't it?

Stefan


From wlavrijsen at lbl.gov  Tue Jul 10 21:02:54 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 12:02:54 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jthtej$jde$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
	<jthtej$jde$1@dough.gmane.org>
Message-ID: <alpine.LNX.2.00.1207101154350.23262@pc3-wlav.lbl.gov>

Hi Stefan,

> Ok, then in the case of a callback, the runtime could simply start with
> totally stupid code that packs the low-level arguments into Python
> arguments, attaches the low-level type information to them and then passes
> them into the function. The JIT should then see in its trace what types
> originally came in and optimise the argument packing away, shouldn't it?

no, don't think so: the JIT works by seeing code execute (the "tracing" part
of it), so with the above recipe, there's still a chicken-and-egg problem.
That is, in order to JIT, the code in the callback needs to be executed, but
to execute, the trace entry point is needed, but for that, the JIT needs to
run ... circle. :/

I was more thinking of an automated equivalent of the toolchain:

   http://doc.pypy.org/en/latest/getting-started-dev.html#id13

with annotations provided programmatically. But if that worked cleanly on
pypy-c, I'd have expected separable "builtin" modules already. :)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From stefan_ml at behnel.de  Tue Jul 10 21:19:50 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jul 2012 21:19:50 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101154350.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
	<jthtej$jde$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101154350.23262@pc3-wlav.lbl.gov>
Message-ID: <jthv8m$1t0$1@dough.gmane.org>

wlavrijsen at lbl.gov, 10.07.2012 21:02:
>> Ok, then in the case of a callback, the runtime could simply start with
>> totally stupid code that packs the low-level arguments into Python
>> arguments, attaches the low-level type information to them and then passes
>> them into the function. The JIT should then see in its trace what types
>> originally came in and optimise the argument packing away, shouldn't it?
> 
> no, don't think so: the JIT works by seeing code execute (the "tracing" part
> of it), so with the above recipe, there's still a chicken-and-egg problem.
> That is, in order to JIT, the code in the callback needs to be executed, but
> to execute, the trace entry point is needed, but for that, the JIT needs to
> run ... circle. :/

Depends. The initial (stupid) entry point would be generated in the moment
where you assign a Python function to a C function pointer. You'd basically
pass a pointer to a wrapper function that calls your Python function.

The question is if a conversion from low-level types to Python types can be
seen and handled by the JIT, but I'd be surprised if it couldn't, since it
generates this kind of code itself anyway. It's the same as calling a
previously unoptimised (Python) branch from an optimised (low-level) one.

Stefan


From wlavrijsen at lbl.gov  Tue Jul 10 21:20:07 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 12:20:07 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidY2FCsHVVpm7sv7qus9yaZBE=QNj1HY9kVuW09S1n5xLg@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<CAGmFidY2FCsHVVpm7sv7qus9yaZBE=QNj1HY9kVuW09S1n5xLg@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207101115411.23262@pc3-wlav.lbl.gov>

Hi Amaury,

On Tue, 10 Jul 2012, Amaury Forgeot d'Arc wrote:
> 2012/7/10  <wlavrijsen at lbl.gov>:
>> Generating such a C function by the backend then requires a C-API for the
>> calls to do the wrapping. That PyPy C-API is a long awaited project. :)
> Do you have an idea what this API would look like?
> Then I can help with the implementation :)

I'm trying to resist the urge to say that it should be the Python C-API, with
all PyObject*'s replaced by PyPyObject*'s. :)

For callbacks, something simpler could do. For example, the PyPyObject*'s
passed in (i.e. the wrapped callback arguments), are going to be fully owned
by the callback function. That way, the JIT is not blind to what happens to
the objects, which should lead to better traces.

Myself, I have two needs for callbacks: GUIs and fitting. For the former,
performance is a non-issue. These callbacks are things like "button clicked",
and its thread safety and clean error handling that's important. As an API,
something like PyPyRun_SimpleString or PyPyRun_AnyFile will already do fine.

The latter, which is closer to what Eleytherios wants, I think, could be
self-contained. More likely, (our) users will want to implement a __call__
member function and pass the callable instance. The idea is to have access
to instance-specific data. Since the instance must live somewhere before
being send down the wire, its data are basically global. In terms of API, to
generate the C stub, a simple PyPyObject_Call() would be enough (the tracking
of the relevant PyPyObject* would be done in the bindings layer at the
interpreter level; this is how it currently works in PyROOT/CPython as well).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From amauryfa at gmail.com  Tue Jul 10 21:27:45 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Jul 2012 21:27:45 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101115411.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<CAGmFidY2FCsHVVpm7sv7qus9yaZBE=QNj1HY9kVuW09S1n5xLg@mail.gmail.com>
	<alpine.LNX.2.00.1207101115411.23262@pc3-wlav.lbl.gov>
Message-ID: <CAGmFidaVFicqiUcS42ZpFpS20QN=BG-U1oJurAxqpcnaXAKGEw@mail.gmail.com>

2012/7/10  <wlavrijsen at lbl.gov>:
> I'm trying to resist the urge to say that it should be the Python C-API,
> with all PyObject*'s replaced by PyPyObject*'s. :)

I don't think it's a good idea to use pointers with a moving GC.
But I may be wrong. In any case, we need a memory model compatible with C.

-- 
Amaury Forgeot d'Arc

From wlavrijsen at lbl.gov  Tue Jul 10 21:32:13 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 12:32:13 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <CAGmFidaVFicqiUcS42ZpFpS20QN=BG-U1oJurAxqpcnaXAKGEw@mail.gmail.com>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<CAGmFidY2FCsHVVpm7sv7qus9yaZBE=QNj1HY9kVuW09S1n5xLg@mail.gmail.com>
	<alpine.LNX.2.00.1207101115411.23262@pc3-wlav.lbl.gov>
	<CAGmFidaVFicqiUcS42ZpFpS20QN=BG-U1oJurAxqpcnaXAKGEw@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1207101230050.23262@pc3-wlav.lbl.gov>

Hi Amaury,

> I don't think it's a good idea to use pointers with a moving GC.

then make it (the moral equivalent of) a PyPyObject**. Like in Smalltalk.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From wlavrijsen at lbl.gov  Tue Jul 10 21:58:24 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 10 Jul 2012 12:58:24 -0700 (PDT)
Subject: [pypy-dev] cpyext performance
In-Reply-To: <jthv8m$1t0$1@dough.gmane.org>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
	<jthtej$jde$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101154350.23262@pc3-wlav.lbl.gov>
	<jthv8m$1t0$1@dough.gmane.org>
Message-ID: <alpine.LNX.2.00.1207101240180.23262@pc3-wlav.lbl.gov>

Hi Stefan,

> The question is if a conversion from low-level types to Python types can be
> seen and handled by the JIT, but I'd be surprised if it couldn't, since it
> generates this kind of code itself anyway.

this is beyond my expertise, but one way of making the wrapping visible, is
to generate the C stub with types, transform those in a memory block with the
needed annotations, then push that block and annotations into PyPy, and do
the wrapping and python call in RPython. This also by-passes the need for a
C-API (other than that one API to push the memory + annotations).

Sounds like something I could do today, but still feels like a square peg /
round hole kind of solution, since the code to trigger that C stub generation
would (in cppyy's case) start in RPython to begin with. Makes me wish I could
generate RPython (after translation, that is).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From amauryfa at gmail.com  Tue Jul 10 22:07:11 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Jul 2012 22:07:11 +0200
Subject: [pypy-dev] cpyext performance
In-Reply-To: <alpine.LNX.2.00.1207101240180.23262@pc3-wlav.lbl.gov>
References: <jsue8t$nrh$1@dough.gmane.org> <jt4m2k$i7o$1@dough.gmane.org>
	<CAK5idxTds8t0kDdEoUDDoWg7qFe_Ypufqu7ECP0S93n=BKeQyQ@mail.gmail.com>
	<134193137.B9fqzCHM9S@hunter-laptop>
	<alpine.LNX.2.00.1207091202480.23262@pc3-wlav.lbl.gov>
	<4FFBF9DC.3080200@gmail.com>
	<CAGmFidb1WiTbw-fsT8hu06HHrZE-Cb2j8MHsKBrq=05YPzDPkg@mail.gmail.com>
	<CAK5idxSVT_cS=Tprh=d0LydDFOZ7BXZA-HM+z++Zr5gXeT6D=w@mail.gmail.com>
	<alpine.LNX.2.00.1207101025070.23262@pc3-wlav.lbl.gov>
	<jthqle$rs7$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101104410.23262@pc3-wlav.lbl.gov>
	<jthrum$681$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101126270.23262@pc3-wlav.lbl.gov>
	<jthtej$jde$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101154350.23262@pc3-wlav.lbl.gov>
	<jthv8m$1t0$1@dough.gmane.org>
	<alpine.LNX.2.00.1207101240180.23262@pc3-wlav.lbl.gov>
Message-ID: <CAGmFidakBXLg194=Ao8=3EZe3+GhbTrvwV9Y3WRoPPinie3g9g@mail.gmail.com>

2012/7/10  <wlavrijsen at lbl.gov>:
> Sounds like something I could do today, but still feels like a square peg /
> round hole kind of solution, since the code to trigger that C stub
> generation would (in cppyy's case) start in RPython to begin with.
> Makes me wish I could generate RPython (after translation, that is).

If we could, we would write extension modules in RPython to start with...



-- 
Amaury Forgeot d'Arc

From tismer at stackless.com  Sat Jul 14 00:47:20 2012
From: tismer at stackless.com (Christian Tismer)
Date: Sat, 14 Jul 2012 00:47:20 +0200
Subject: [pypy-dev] Developer selection for Py3k
In-Reply-To: <4FEE3B5E.4060206@stackless.com>
References: <4FECE3EC.7030000@gmail.com> <4FEE3B5E.4060206@stackless.com>
Message-ID: <5000A578.70109@stackless.com>

On 6/30/12 1:33 AM, Christian Tismer wrote:
> On 29.06.12 01:08, Antonio Cuni wrote:
>> Hi all,
>>
>> as you probably know, the Py3k [1] proposal is getting funded thanks 
>> to our
>> generous donors.
>>
>> During the first round, three of use were selected: me, Alex Gaynor and
>> Benjamin Peterson.  However, due to unforeseen unavailability of Alex 
>> and
>> Benjamin, we are now looking for one more developer to help with the 
>> py3k work.
>>
>> If you are interested in getting paid to work on the Py3k proposal, 
>> please
>> apply by replying to this email.
>>
>> To be applicable you need to be an experienced PyPy developer, 
>> preferably with
>> a previous experience on the Python Interpreter part of PyPy.
>>
>
> Hi Anto,
>
> Starting from August, I have some time that I could spend on this.

Hi,

has the selection process revealed a result, and where can I see it?

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
work +49 173 24 18 776  mobile +49 173 24 18 776  fax n.a.
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From arigo at tunes.org  Sat Jul 14 10:35:48 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 14 Jul 2012 10:35:48 +0200
Subject: [pypy-dev] PyPy C API
Message-ID: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>

Hi Amaury,

On Tue, Jul 10, 2012 at 7:57 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> Do you have an idea what this API would look like?
> Then I can help with the implementation :)

If we want to go down this path without caring for compatibility with
CPython's C API, but instead focusing on what gives the best
performance, then I would think about something like this:

pypyobj pypy_wraplong(long value);
pypyobj pypy_add(pypyobj x, pypyobj y);
void pypy_close(pypyobj x);
pypyobj pypy_dup(pypyobj x);

using handles of type "pypyobj", which are basically opaque stuff (or
even just integers, with "-1" meaning "exception").  Instead of the
refcounting approach of CPython, it would be similar to file
descriptors: a file descriptor refers to a file, but most files don't
have any open file descriptor, and some files may have more than one.
Any "object descriptor" must be closed with pypy_close().  pypy_dup()
just duplicates the object descriptor, so that both descriptors refer
to the same object but will be pypy_close()d independently.

This can be implemented efficiently: the C->PyPy direction is just
doing one array lookup (this minimal indirection is hard to avoid
anyway with a moving GC); and the PyPy->C direction (like the return
value from pypy_add) just creates a new object descriptor anyway,
without needing to look if one already exists.

(1) Do I make some sense, and (2) is there any real use case for such
an API?  E.g. would the expected performance gains of Cython justify
the rewrite needed to handle such an API, which is quite different
from CPython's?


A bient?t,

Armin.

From arigo at tunes.org  Sat Jul 14 10:42:04 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 14 Jul 2012 10:42:04 +0200
Subject: [pypy-dev] pypy translation failures
In-Reply-To: <4FF68051.8060107@gmail.com>
References: <CALJjGitcfhmxv0+_xiV6b0VjNhNzdJHpm8Mwq+DL06qGZJ+-uA@mail.gmail.com>
	<4FF68051.8060107@gmail.com>
Message-ID: <CAMSv6X0aM3kOgBrdHaB3U6J0Xw_CViiKyq_=FLnSndrwgbJPwg@mail.gmail.com>

Hi Robert,

If this is still relevant (sorry for the delay, EuroPython happened):

On Fri, Jul 6, 2012 at 8:06 AM, Matti Picus <matti.picus at gmail.com> wrote:
>> --sandbox

The sandbox version may or may not work (usually for some small
detail); the point is that we don't test it regularly.

If you want I can try to give it a look again.


A bient?t,

Armin.

From Ronny.Pfannschmidt at gmx.de  Sat Jul 14 12:09:21 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sat, 14 Jul 2012 12:09:21 +0200
Subject: [pypy-dev] PyPy C API
In-Reply-To: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
References: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
Message-ID: <50014551.8020800@gmx.de>

On 07/14/2012 10:35 AM, Armin Rigo wrote:
> Hi Amaury,
>
> On Tue, Jul 10, 2012 at 7:57 PM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com>  wrote:
>> Do you have an idea what this API would look like?
>> Then I can help with the implementation :)
>
> If we want to go down this path without caring for compatibility with
> CPython's C API, but instead focusing on what gives the best
> performance, then I would think about something like this:
>
> pypyobj pypy_wraplong(long value);
> pypyobj pypy_add(pypyobj x, pypyobj y);
> void pypy_close(pypyobj x);
> pypyobj pypy_dup(pypyobj x);
>
> using handles of type "pypyobj", which are basically opaque stuff (or
> even just integers, with "-1" meaning "exception").  Instead of the
> refcounting approach of CPython, it would be similar to file
> descriptors: a file descriptor refers to a file, but most files don't
> have any open file descriptor, and some files may have more than one.
> Any "object descriptor" must be closed with pypy_close().  pypy_dup()
> just duplicates the object descriptor, so that both descriptors refer
> to the same object but will be pypy_close()d independently.
>
> This can be implemented efficiently: the C->PyPy direction is just
> doing one array lookup (this minimal indirection is hard to avoid
> anyway with a moving GC); and the PyPy->C direction (like the return
> value from pypy_add) just creates a new object descriptor anyway,
> without needing to look if one already exists.
>
> (1) Do I make some sense, and (2) is there any real use case for such
> an API?  E.g. would the expected performance gains of Cython justify
> the rewrite needed to handle such an API, which is quite different
> from CPython's?
>
(1) yes
(2)
   (a)
     cython could target that with some added primitives like
     new_ref and drop_rev which in cpython would incref/decref
     and on pypy would dup/close
   (b)
      this can be the basis of creating/generating higher level apis to
      interface with pypy

      imagine a gobject+gobject introspection based wrapper
      that would allow stuff like the following
      in vala/javascript/anything else for example::

        import PyPy/ PyPy = require('pypy')
        var long1 = PyPy.Int.from_long(123)
        var long2 = PyPy.Int.from_long(123)
        var result = long1.add(long2)
        var text = result.to_string()
        print(text)
    (c)
      (slightly crazy)
      it might be possible to create a pure c lib
      that can be used as compat layer
      for pypy and cpython extensions
      which would allow gradual adoption even
      in cases where cpyext would have been too slow
      and ctypes/cffi aren't a option

      (but i have a feeling that using such a lib would require
       a linter to help avoiding all those easy misstakes)

-- Ronny
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From janpeterr at freenet.de  Mon Jul 16 06:42:35 2012
From: janpeterr at freenet.de (Jan Riechers)
Date: Mon, 16 Jul 2012 07:42:35 +0300
Subject: [pypy-dev] Compiling for pypy
Message-ID: <50039BBB.2060604@freenet.de>

Hello,

I got a hard time trying to get pymongo running in Pypy - I simple can't 
figure out how to build/compile the driver for Pypy.

After searching I coudn't find any answer on how to translate and or 
compile the latest branch of the driver and I'm quite desperate on how 
to make this work as I really would like to make use of Pypy in my 
present project..

I have a 32bit Windows here, running also Visual C++ Express Edition 
2008 - so a compiler is available.

Can anyone provide me a hint on how to get started with compiling 
modules for Pypy?
I successfully translated Pypy itself so far, but that's about it. Any 
other tips are highly appreciated on how to compile 3rd party modules.

Also note: I tried the latest pymongo branch download from Github with 
"setup.py build install" - I still receive an error like this:

"Fatal Python error: PyThreadState_Get: no current thread

This application has requested the Runtime to terminate it in an unusual 
way.
Please contact the application's support team for more information."

Which should be out from the pymongo driver as it seems not to be 
compile for pypy usage, right guess?

Regards,
Jan

From fijall at gmail.com  Mon Jul 16 10:02:39 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 16 Jul 2012 10:02:39 +0200
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <50039BBB.2060604@freenet.de>
References: <50039BBB.2060604@freenet.de>
Message-ID: <CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>

On Mon, Jul 16, 2012 at 6:42 AM, Jan Riechers <janpeterr at freenet.de> wrote:

> Hello,
>
> I got a hard time trying to get pymongo running in Pypy - I simple can't
> figure out how to build/compile the driver for Pypy.
>
> After searching I coudn't find any answer on how to translate and or
> compile the latest branch of the driver and I'm quite desperate on how to
> make this work as I really would like to make use of Pypy in my present
> project..
>
> I have a 32bit Windows here, running also Visual C++ Express Edition 2008
> - so a compiler is available.
>
> Can anyone provide me a hint on how to get started with compiling modules
> for Pypy?
> I successfully translated Pypy itself so far, but that's about it. Any
> other tips are highly appreciated on how to compile 3rd party modules.
>
> Also note: I tried the latest pymongo branch download from Github with
> "setup.py build install" - I still receive an error like this:
>
> "Fatal Python error: PyThreadState_Get: no current thread
>
> This application has requested the Runtime to terminate it in an unusual
> way.
> Please contact the application's support team for more information."
>
> Which should be out from the pymongo driver as it seems not to be compile
> for pypy usage, right guess?
>
> Regards,
> Jan
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>

Hi

The CPython C API compatibility layer is incomplete (and also a big hack).
This particular error *might* be related to already reported problems with
pyopenssl, I think there is even a patch. I suppose the best thing is to
wait a bit until we pull in patches for threads and then see if it helps
you.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120716/79d62931/attachment.html>

From stefan_ml at behnel.de  Tue Jul 17 14:17:36 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 14:17:36 +0200
Subject: [pypy-dev] PyPy C API
In-Reply-To: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
References: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
Message-ID: <ju3l50$sad$1@dough.gmane.org>

Armin Rigo, 14.07.2012 10:35:
> On Tue, Jul 10, 2012 at 7:57 PM, Amaury Forgeot d'Arc wrote:
>> Do you have an idea what this API would look like?
>> Then I can help with the implementation :)
> 
> If we want to go down this path without caring for compatibility with
> CPython's C API, but instead focusing on what gives the best
> performance, then I would think about something like this:
> 
> pypyobj pypy_wraplong(long value);
> pypyobj pypy_add(pypyobj x, pypyobj y);
> void pypy_close(pypyobj x);
> pypyobj pypy_dup(pypyobj x);
> 
> using handles of type "pypyobj", which are basically opaque stuff (or
> even just integers, with "-1" meaning "exception").  Instead of the
> refcounting approach of CPython, it would be similar to file
> descriptors: a file descriptor refers to a file, but most files don't
> have any open file descriptor, and some files may have more than one.
> Any "object descriptor" must be closed with pypy_close().  pypy_dup()
> just duplicates the object descriptor, so that both descriptors refer
> to the same object but will be pypy_close()d independently.
> 
> This can be implemented efficiently: the C->PyPy direction is just
> doing one array lookup (this minimal indirection is hard to avoid
> anyway with a moving GC); and the PyPy->C direction (like the return
> value from pypy_add) just creates a new object descriptor anyway,
> without needing to look if one already exists.

Assuming that the PyPy->C path takes its object descriptors from a
free-list (because we'll need a lot of them, and usually just for a very
short time), I think this would be acceptable.


> (1) Do I make some sense

Sounds like a really straight forward C-API to me, and also quite efficient.


> and (2) is there any real use case for such
> an API?  E.g. would the expected performance gains of Cython justify
> the rewrite needed to handle such an API, which is quite different
> from CPython's?

Very different, yes. I can't fully estimate the amount of work it would
take to port Cython to it, but it's definitely not small, and there's no
quick win along the path. The API is so different that we can only port it
completely or not at all, no mixing possible.

If Cython wants to continue to support both C-APIs in the same C code file
(and switch at C compile time), this will add a lot of C code overhead.
Basically all Python related code would have to be duplicated, including
object variables and error tests. It will also mean a lot of code overhead
in the code generation part of the compiler.

I'm not convinced that this is worth it solely for the purpose of
interfacing with Cython, but if PyPy wants a suitable C-API for itself (and
you should seriously consider that), then this would be a suitable
long-term solution IMHO.

A more immediate solution would obviously be to bring cpyext up to speed...

Stefan


From rayvroberts at gmail.com  Thu Jul 19 22:10:36 2012
From: rayvroberts at gmail.com (Raymond Roberts)
Date: Thu, 19 Jul 2012 16:10:36 -0400
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
Message-ID: <CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>

Have you tried building the pymongo driver without the C extensions?

On Mon, Jul 16, 2012 at 4:02 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Mon, Jul 16, 2012 at 6:42 AM, Jan Riechers <janpeterr at freenet.de>wrote:
>
>> Hello,
>>
>> I got a hard time trying to get pymongo running in Pypy - I simple can't
>> figure out how to build/compile the driver for Pypy.
>>
>> After searching I coudn't find any answer on how to translate and or
>> compile the latest branch of the driver and I'm quite desperate on how to
>> make this work as I really would like to make use of Pypy in my present
>> project..
>>
>> I have a 32bit Windows here, running also Visual C++ Express Edition 2008
>> - so a compiler is available.
>>
>> Can anyone provide me a hint on how to get started with compiling modules
>> for Pypy?
>> I successfully translated Pypy itself so far, but that's about it. Any
>> other tips are highly appreciated on how to compile 3rd party modules.
>>
>> Also note: I tried the latest pymongo branch download from Github with
>> "setup.py build install" - I still receive an error like this:
>>
>> "Fatal Python error: PyThreadState_Get: no current thread
>>
>> This application has requested the Runtime to terminate it in an unusual
>> way.
>> Please contact the application's support team for more information."
>>
>> Which should be out from the pymongo driver as it seems not to be compile
>> for pypy usage, right guess?
>>
>> Regards,
>> Jan
>> ______________________________**_________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>>
>
> Hi
>
> The CPython C API compatibility layer is incomplete (and also a big hack).
> This particular error *might* be related to already reported problems with
> pyopenssl, I think there is even a patch. I suppose the best thing is to
> wait a bit until we pull in patches for threads and then see if it helps
> you.
>
> Cheers,
> fijal
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120719/b2707839/attachment.html>

From janpeterr at freenet.de  Fri Jul 20 10:11:49 2012
From: janpeterr at freenet.de (Jan Riechers)
Date: Fri, 20 Jul 2012 11:11:49 +0300
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
Message-ID: <500912C5.2030304@freenet.de>

On 19.07.2012 23:10, Raymond Roberts wrote:
> Have you tried building the pymongo driver without the C extensions?
>

Hello Raymond
thank you for coming back, I just tried once more and the pymongo driver 
works. Also the latest version.

I caused some trouble by having also greenlet and eventlet installed, 
seems that those did not like to come along and causing the:
"Fatal Python error: PyThreadState_Get: no current thread" error

Actually I also use pil besides eventlet and greenlet, but this time I 
can't get pil compiled correctly and raises me a compiler error during 
start of the c module compilation. But this is another topic I guess 
plus I haven't searched for any answer yet.

But if you have any information in matter of the above modules - PIL is 
assumed to work with Pypy it states in the wiki, please let me know!

Thank you in advance,
Jan

From fijall at gmail.com  Fri Jul 20 10:36:45 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 20 Jul 2012 10:36:45 +0200
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <500912C5.2030304@freenet.de>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
	<500912C5.2030304@freenet.de>
Message-ID: <CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>

On Fri, Jul 20, 2012 at 10:11 AM, Jan Riechers <janpeterr at freenet.de> wrote:

> On 19.07.2012 23:10, Raymond Roberts wrote:
>
>> Have you tried building the pymongo driver without the C extensions?
>>
>>
> Hello Raymond
> thank you for coming back, I just tried once more and the pymongo driver
> works. Also the latest version.
>
> I caused some trouble by having also greenlet and eventlet installed,
> seems that those did not like to come along and causing the:
> "Fatal Python error: PyThreadState_Get: no current thread" error
>
> Actually I also use pil besides eventlet and greenlet, but this time I
> can't get pil compiled correctly and raises me a compiler error during
> start of the c module compilation. But this is another topic I guess plus I
> haven't searched for any answer yet.
>
> But if you have any information in matter of the above modules - PIL is
> assumed to work with Pypy it states in the wiki, please let me know!
>
>
> Thank you in advance,
> Jan
>

Indeed, PIL should work, what's the error? Are you using recent PyPy?

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120720/595eca61/attachment.html>

From janpeterr at freenet.de  Fri Jul 20 11:35:28 2012
From: janpeterr at freenet.de (Jan Riechers)
Date: Fri, 20 Jul 2012 12:35:28 +0300
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
	<500912C5.2030304@freenet.de>
	<CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
Message-ID: <50092660.8040702@freenet.de>

On 20.07.2012 11:36, Maciej Fijalkowski wrote:> On Fri, Jul 20, 2012 at 
10:11 AM, Jan Riechers <janpeterr at freenet.de
 > <mailto:janpeterr at freenet.de>> wrote:
 >
 >     I caused some trouble by having also greenlet and eventlet
 >     installed, seems that those did not like to come along and 
causing the:
 >     "Fatal Python error: PyThreadState_Get: no current thread" error
 >
 >     Actually I also use pil besides eventlet and greenlet, but this time
 >     I can't get pil compiled correctly and raises me a compiler error
 >     during start of the c module compilation. But this is another topic
 >     I guess plus I haven't searched for any answer yet.
 >
 >     But if you have any information in matter of the above modules - PIL
 >     is assumed to work with Pypy it states in the wiki, please let me 
know!
 >
 >
 >     Thank you in advance,
 >     Jan
 >
 >
 > Indeed, PIL should work, what's the error? Are you using recent PyPy?
 >
 > Cheers,
 > fijal

Hi,

yes, I assume I use the most recent version 1.9, but official windows 
build, not a own compilation. I saw there was already a similiar issue 
posted here (for reference, my error below):
http://www.mail-archive.com/pypy-dev at python.org/msg02117.html

I receive a quite long error message from command prompt, I shorted
the "copy" statements of the build as they run fine so:

---- SNIP -----------------------

D:\pypy-1.9\bin>pypy pip-2.7-script.py install pil

Downloading/unpacking pil
   Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
   Running setup.py egg_info for package pil
     WARNING: '' not a valid package name; please use only.-separated 
package names in setup.py

Installing collected packages: pil
   Running setup.py install for pil
     WARNING: '' not a valid package name; please use only.-separated 
package names in setup.py
     building '_imaging' extension
     D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 
/GS- /DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tc_imaging.c 
/Fobuild\temp.win32-2.7\Release\_imaging.obj
     _imaging.c
     _imaging.c(1230) : warning C4133: 'function' : incompatible types - 
from 'PyStringObject *' to 'PyObject *'
     _imaging.c(1384) : warning C4996: 'strcpy': This function or 
variable may be unsafe. Consider using strcpy_s instead. To disable 
deprecation, use _CRT_SECURE_NO_WARNINGS. See o
nline help for details.
     _imaging.c(1412) : warning C4996: 'strcpy': This function or 
variable may be unsafe. Consider using strcpy_s instead. To disable 
deprecation, use _CRT_SECURE_NO_WARNINGS. See o
nline help for details.
     _imaging.c(1550) : warning C4996: 'strcpy': This function or 
variable may be unsafe. Consider using strcpy_s instead. To disable 
deprecation, use _CRT_SECURE_NO_WARNINGS. See o
nline help for details.
     D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 
/GS- /DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcdecode.c 
/Fobuild\temp.win32-2.7\Release\decode.obj
     decode.c
     D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 
/GS- /DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcencode.c 
/Fobuild\temp.win32-2.7\Release\encode.obj
     encode.c
     encode.c(161) : warning C4996: 'write': The POSIX name for this 
item is deprecated. Instead, use the ISO C++ conformant name: _write. 
See online help for details.
             D:\Microsoft Visual Studio 9.0\VC\INCLUDE\io.h(322) : see 
declaration of 'write'
     D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 
/GS- /DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcmap.c 
/Fobuild\temp.win32-2.7\Release\map.obj
     map.c
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) 
: warning C4114: same type qualifier used more than once
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) 
: error C2632: 'char' followed by 'char' is illegal
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) 
: error C2059: syntax error : ','
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(77) 
: error C2632: 'short' followed by 'short' is illegal
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(77) 
: error C2059: syntax error : ','
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) 
: warning C4114: same type qualifier used more than once
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) 
: error C2632: 'char' followed by 'char' is illegal
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) 
: error C2059: syntax error : ','
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) 
: warning C4114: same type qualifier used more than once
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) 
: error C2632: 'short' followed by 'short' is illegal
     C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) 
: error C2059: syntax error : ','
     error: command 'D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe' 
failed with exit status 2
     Complete output from command D:\pypy-1.9\pypy.exe -c "import 
setuptools;__file__='D:\\pypy-1.9\\bin\\build\\pil\\setup.py';exec(compile(open(__file__).read().replace('\r\n', 
'\
n'), __file__, 'exec'))" install --single-version-externally-managed 
--record e:\_tmp\pip-zvf7ry-record\install-record.txt:
     WARNING: '' not a valid package name; please use only.-separated 
package names in setup.py

running install
running build
running build_py
creating build
creating build\lib.win32-2.7

a lot of copying like :
copying PIL\ArgImagePlugin.py -> build\lib.win32-2.7
.. (removed) ..
copying PIL\__init__.py -> build\lib.win32-2.7

running build_ext
building '_imaging' extension
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\libImaging

D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- 
/DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tc_imaging.c 
/Fobuild\temp.win32-2.7\Release\_imaging.obj

_imaging.c

_imaging.c(1230) : warning C4133: 'function' : incompatible types - from 
'PyStringObject *' to 'PyObject *'

_imaging.c(1384) : warning C4996: 'strcpy': This function or variable 
may be unsafe. Consider using strcpy_s instead. To disable deprecation, 
use _CRT_SECURE_NO_WARNINGS. See onlin
e help for details.

_imaging.c(1412) : warning C4996: 'strcpy': This function or variable 
may be unsafe. Consider using strcpy_s instead. To disable deprecation, 
use _CRT_SECURE_NO_WARNINGS. See onlin
e help for details.

_imaging.c(1550) : warning C4996: 'strcpy': This function or variable 
may be unsafe. Consider using strcpy_s instead. To disable deprecation, 
use _CRT_SECURE_NO_WARNINGS. See onlin
e help for details.

D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- 
/DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcdecode.c 
/Fobuild\temp.win32-2.7\Release\decode.obj

decode.c

D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- 
/DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcencode.c 
/Fobuild\temp.win32-2.7\Release\encode.obj

encode.c

encode.c(161) : warning C4996: 'write': The POSIX name for this item is 
deprecated. Instead, use the ISO C++ conformant name: _write. See online 
help for details.

         D:\Microsoft Visual Studio 9.0\VC\INCLUDE\io.h(322) : see 
declaration of 'write'

D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- 
/DNDEBUG -IlibImaging -ID:\pypy-1.9\include /Tcmap.c 
/Fobuild\temp.win32-2.7\Release\map.obj

map.c

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) : 
warning C4114: same type qualifier used more than once

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) : 
error C2632: 'char' followed by 'char' is illegal

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(76) : 
error C2059: syntax error : ','

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(77) : 
error C2632: 'short' followed by 'short' is illegal

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(77) : 
error C2059: syntax error : ','

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) : 
warning C4114: same type qualifier used more than once

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) : 
error C2632: 'char' followed by 'char' is illegal

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(80) : 
error C2059: syntax error : ','

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) : 
warning C4114: same type qualifier used more than once

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) : 
error C2632: 'short' followed by 'short' is illegal

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\basetsd.h(81) : 
error C2059: syntax error : ','

error: command 'D:\Microsoft Visual Studio 9.0\VC\BIN\cl.exe' failed 
with exit status 2

----------------------------------------
Command D:\pypy-1.9\pypy.exe -c "import 
setuptools;__file__='D:\\pypy-1.9\\bin\\build\\pil\\setup.py';exec(compile(open(__file__).read().replace('\r\n', 
'\n'), __file__, 'exec'))"
install --single-version-externally-managed --record 
e:\_tmp\pip-zvf7ry-record\install-record.txt failed with error code 1 in 
D:\pypy-1.9\bin\build\pil
Storing complete log in 
C:\Users\mydecentusername\AppData\Roaming\pip\pip.log

D:\pypy-1.9\bin>


---- SNAP -----------------------

From amauryfa at gmail.com  Fri Jul 20 12:34:20 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 20 Jul 2012 12:34:20 +0200
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <50092660.8040702@freenet.de>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
	<500912C5.2030304@freenet.de>
	<CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
	<50092660.8040702@freenet.de>
Message-ID: <CAGmFidanktazTo67N49EEWhAYQmbvyiUKeg45Kp0r4cz_ritng@mail.gmail.com>

2012/7/20 Jan Riechers <janpeterr at freenet.de>:
> yes, I assume I use the most recent version 1.9, but official windows build,
> not a own compilation. I saw there was already a similiar issue posted here
> (for reference, my error below):
> http://www.mail-archive.com/pypy-dev at python.org/msg02117.html

And the answer was:
http://www.mail-archive.com/pypy-dev at python.org/msg02119.html
Does PIL compiles with CPython on your system?

-- 
Amaury Forgeot d'Arc

From janpeterr at freenet.de  Fri Jul 20 12:53:22 2012
From: janpeterr at freenet.de (Jan Riechers)
Date: Fri, 20 Jul 2012 13:53:22 +0300
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <CAGmFidanktazTo67N49EEWhAYQmbvyiUKeg45Kp0r4cz_ritng@mail.gmail.com>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
	<500912C5.2030304@freenet.de>
	<CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
	<50092660.8040702@freenet.de>
	<CAGmFidanktazTo67N49EEWhAYQmbvyiUKeg45Kp0r4cz_ritng@mail.gmail.com>
Message-ID: <500938A2.4020204@freenet.de>




On 20.07.2012 13:34, Amaury Forgeot d'Arc wrote:
> 2012/7/20 Jan Riechers <janpeterr at freenet.de>:
>> yes, I assume I use the most recent version 1.9, but official windows build,
>> not a own compilation. I saw there was already a similiar issue posted here
>> (for reference, my error below):
>> http://www.mail-archive.com/pypy-dev at python.org/msg02117.html
>
> And the answer was:
> http://www.mail-archive.com/pypy-dev at python.org/msg02119.html
> Does PIL compiles with CPython on your system?
>


On 20.07.2012 11:36, Maciej Fijalkowski wrote:
 > On Fri, Jul 20, 2012 at 10:11 AM, Jan Riechers <janpeterr at freenet.de
 > <mailto:janpeterr at freenet.de>> wrote:
 >
 >     On 19.07.2012 23:10, Raymond Roberts wrote:
 >
 >         Have you tried building the pymongo driver without the C 
extensions?
 >
 >
 >     Hello Raymond
 >     thank you for coming back, I just tried once more and the pymongo
 >     driver works. Also the latest version.
 >
 >     I caused some trouble by having also greenlet and eventlet
 >     installed, seems that those did not like to come along and 
causing the:
 >     "Fatal Python error: PyThreadState_Get: no current thread" error
 >
 >     Actually I also use pil besides eventlet and greenlet, but this time
 >     I can't get pil compiled correctly and raises me a compiler error
 >     during start of the c module compilation. But this is another topic
 >     I guess plus I haven't searched for any answer yet.
 >
 >     But if you have any information in matter of the above modules - PIL
 >     is assumed to work with Pypy it states in the wiki, please let me 
know!
 >
 >
 >     Thank you in advance,
 >     Jan
 >
 >
 > Indeed, PIL should work, what's the error? Are you using recent PyPy?
 >
 > Cheers,
 > fijal

[SNIP]
Sorry for posting twice on this PIL issue, but perhaps I should use 
"Pillow" as proposed here:
http://www.mail-archive.com/pypy-dev at python.org/msg02257.html

I just tried the latest branch from github at:
https://github.com/mattip/Pillow#introduction

and it works

But PIL is definetly broken it seems, its also on python Package index 
and also on the gitHub notes.

Thank you Fijal for helping out, hope this message reachs you before 
digging too much.

Jan

-[SNAP]------------------------------


For some reason my message did not appear, but I tried Pillow as noted 
above and also it run with the support libraries fine (jpeg, zlib).

Pil, as stated, does not compile with the errors noted earlier.

Jan

From janpeterr at freenet.de  Fri Jul 20 11:58:21 2012
From: janpeterr at freenet.de (Jan Riechers)
Date: Fri, 20 Jul 2012 12:58:21 +0300
Subject: [pypy-dev] Compiling for pypy
In-Reply-To: <CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
References: <50039BBB.2060604@freenet.de>
	<CAK5idxQiyeL21ED18=O0hv72KwZ2=7H3dr-nF7DG1vBk0hdJOw@mail.gmail.com>
	<CABN7w2vF9t15J-ShXMDnFAKM41KaMcaRVAbrdr5G_KHc4A6Z-w@mail.gmail.com>
	<500912C5.2030304@freenet.de>
	<CAK5idxTG+FcQo=e1iTfHwCF=VB_U+XT8+cgGWqPk2-HUpuSfCQ@mail.gmail.com>
Message-ID: <50092BBD.1070603@freenet.de>

On 20.07.2012 11:36, Maciej Fijalkowski wrote:
> On Fri, Jul 20, 2012 at 10:11 AM, Jan Riechers <janpeterr at freenet.de
> <mailto:janpeterr at freenet.de>> wrote:
>
>     On 19.07.2012 23:10, Raymond Roberts wrote:
>
>         Have you tried building the pymongo driver without the C extensions?
>
>
>     Hello Raymond
>     thank you for coming back, I just tried once more and the pymongo
>     driver works. Also the latest version.
>
>     I caused some trouble by having also greenlet and eventlet
>     installed, seems that those did not like to come along and causing the:
>     "Fatal Python error: PyThreadState_Get: no current thread" error
>
>     Actually I also use pil besides eventlet and greenlet, but this time
>     I can't get pil compiled correctly and raises me a compiler error
>     during start of the c module compilation. But this is another topic
>     I guess plus I haven't searched for any answer yet.
>
>     But if you have any information in matter of the above modules - PIL
>     is assumed to work with Pypy it states in the wiki, please let me know!
>
>
>     Thank you in advance,
>     Jan
>
>
> Indeed, PIL should work, what's the error? Are you using recent PyPy?
>
> Cheers,
> fijal

Sorry for posting twice on this PIL issue, but perhaps I should use 
"Pillow" as proposed here:
http://www.mail-archive.com/pypy-dev at python.org/msg02257.html

I just tried the latest branch from github at:
https://github.com/mattip/Pillow#introduction

and it works :)

But PIL is definetly broken it seems, its also on python Package index 
and also on the gitHub notes.

Thank you Fijal for helping out, hope this message reachs you before 
digging too much. ;)

Jan

From alex.pyattaev at gmail.com  Fri Jul 20 21:55:43 2012
From: alex.pyattaev at gmail.com (Alexander Pyattaev)
Date: Fri, 20 Jul 2012 12:55:43 -0700
Subject: [pypy-dev] PyPy C API
In-Reply-To: <ju3l50$sad$1@dough.gmane.org>
References: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
	<ju3l50$sad$1@dough.gmane.org>
Message-ID: <CAHPsuBNsrLwJD8jDxeaTd+Hxin19CQmv2_6HuHNZxQZkhb=vNA@mail.gmail.com>

I think that most of the end-users would not care much and would use SWIG
anyway, because it is simple to do and scalable. So whatever API you decide
to go with, the only real problem will be to make a swig wrapper library
for that and about 80% of applications that use SWIG would not even need
any tweaks to work. I'd be able to help with those wrappers, did quite a
few of them so far.

On Tue, Jul 17, 2012 at 5:17 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Armin Rigo, 14.07.2012 10:35:
> > On Tue, Jul 10, 2012 at 7:57 PM, Amaury Forgeot d'Arc wrote:
> >> Do you have an idea what this API would look like?
> >> Then I can help with the implementation :)
> >
> > If we want to go down this path without caring for compatibility with
> > CPython's C API, but instead focusing on what gives the best
> > performance, then I would think about something like this:
> >
> > pypyobj pypy_wraplong(long value);
> > pypyobj pypy_add(pypyobj x, pypyobj y);
> > void pypy_close(pypyobj x);
> > pypyobj pypy_dup(pypyobj x);
> >
> > using handles of type "pypyobj", which are basically opaque stuff (or
> > even just integers, with "-1" meaning "exception").  Instead of the
> > refcounting approach of CPython, it would be similar to file
> > descriptors: a file descriptor refers to a file, but most files don't
> > have any open file descriptor, and some files may have more than one.
> > Any "object descriptor" must be closed with pypy_close().  pypy_dup()
> > just duplicates the object descriptor, so that both descriptors refer
> > to the same object but will be pypy_close()d independently.
> >
> > This can be implemented efficiently: the C->PyPy direction is just
> > doing one array lookup (this minimal indirection is hard to avoid
> > anyway with a moving GC); and the PyPy->C direction (like the return
> > value from pypy_add) just creates a new object descriptor anyway,
> > without needing to look if one already exists.
>
> Assuming that the PyPy->C path takes its object descriptors from a
> free-list (because we'll need a lot of them, and usually just for a very
> short time), I think this would be acceptable.
>
>
> > (1) Do I make some sense
>
> Sounds like a really straight forward C-API to me, and also quite
> efficient.
>
>
> > and (2) is there any real use case for such
> > an API?  E.g. would the expected performance gains of Cython justify
> > the rewrite needed to handle such an API, which is quite different
> > from CPython's?
>
> Very different, yes. I can't fully estimate the amount of work it would
> take to port Cython to it, but it's definitely not small, and there's no
> quick win along the path. The API is so different that we can only port it
> completely or not at all, no mixing possible.
>
> If Cython wants to continue to support both C-APIs in the same C code file
> (and switch at C compile time), this will add a lot of C code overhead.
> Basically all Python related code would have to be duplicated, including
> object variables and error tests. It will also mean a lot of code overhead
> in the code generation part of the compiler.
>
> I'm not convinced that this is worth it solely for the purpose of
> interfacing with Cython, but if PyPy wants a suitable C-API for itself (and
> you should seriously consider that), then this would be a suitable
> long-term solution IMHO.
>
> A more immediate solution would obviously be to bring cpyext up to speed...
>
> Stefan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120720/3d22a565/attachment-0001.html>

From amauryfa at gmail.com  Fri Jul 20 22:28:39 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 20 Jul 2012 22:28:39 +0200
Subject: [pypy-dev] PyPy C API
In-Reply-To: <CAHPsuBNsrLwJD8jDxeaTd+Hxin19CQmv2_6HuHNZxQZkhb=vNA@mail.gmail.com>
References: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
	<ju3l50$sad$1@dough.gmane.org>
	<CAHPsuBNsrLwJD8jDxeaTd+Hxin19CQmv2_6HuHNZxQZkhb=vNA@mail.gmail.com>
Message-ID: <CAGmFida6_chHSwk4=gPdCCPbVnPMuWGHaznNC0zP_L8g=uPhGg@mail.gmail.com>

2012/7/20 Alexander Pyattaev <alex.pyattaev at gmail.com>:
> I think that most of the end-users would not care much and would use SWIG
> anyway, because it is simple to do and scalable. So whatever API you decide
> to go with, the only real problem will be to make a swig wrapper library for
> that and about 80% of applications that use SWIG would not even need any
> tweaks to work. I'd be able to help with those wrappers, did quite a few of
> them so far.

I also worked a lot with SWIG, and it does not hide the interpreter API at all;
most of the time you write a %typemap, you'll need functions like
PyString_FromString.
(For example, the one for char**argv)

And advanced uses (callbacks, threads, directors for C++) will need advanced API
functions (like PyThreadState_New) to properly integrate the library
with Python.

-- 
Amaury Forgeot d'Arc

From stefan_ml at behnel.de  Sun Jul 22 22:05:44 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jul 2012 22:05:44 +0200
Subject: [pypy-dev] PyPy C API
In-Reply-To: <CAGmFida6_chHSwk4=gPdCCPbVnPMuWGHaznNC0zP_L8g=uPhGg@mail.gmail.com>
References: <CAMSv6X2U9cowkHeGQaXhjEBFrrjk6w22BzHzyoVeig28nAkU6w@mail.gmail.com>
	<ju3l50$sad$1@dough.gmane.org>
	<CAHPsuBNsrLwJD8jDxeaTd+Hxin19CQmv2_6HuHNZxQZkhb=vNA@mail.gmail.com>
	<CAGmFida6_chHSwk4=gPdCCPbVnPMuWGHaznNC0zP_L8g=uPhGg@mail.gmail.com>
Message-ID: <juhmeo$hsd$1@dough.gmane.org>

Amaury Forgeot d'Arc, 20.07.2012 22:28:
> 2012/7/20 Alexander Pyattaev:
>> I think that most of the end-users would not care much and would use SWIG
>> anyway, because it is simple to do and scalable. So whatever API you decide
>> to go with, the only real problem will be to make a swig wrapper library for
>> that and about 80% of applications that use SWIG would not even need any
>> tweaks to work. I'd be able to help with those wrappers, did quite a few of
>> them so far.
> 
> I also worked a lot with SWIG, and it does not hide the interpreter API at all;
> most of the time you write a %typemap, you'll need functions like
> PyString_FromString.
> (For example, the one for char**argv)
> 
> And advanced uses (callbacks, threads, directors for C++) will need advanced API
> functions (like PyThreadState_New) to properly integrate the library
> with Python.

And even if you managed to get it working, you'd still have to figure out
how to make your wrapper fast enough for what you do with it, which may be
anywhere between "don't care as long as it works" and "can't use SWIG for
it at all".

Stefan



From drsalists at gmail.com  Wed Jul 25 00:34:32 2012
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 24 Jul 2012 22:34:32 +0000
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
Message-ID: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>

The subject pretty much says it, but here it is again:

Is there a concurrent, nosql, opensource key-value store that works with
Pypy, CPython 2.x, CPython 3.x and Jython?

It doesn't need to be highly concurrent.  In fact, locking the entire table
for the duration of operations would be acceptable if the others block
until the lock is released.

TIA!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120724/43589e32/attachment.html>

From fijall at gmail.com  Wed Jul 25 00:56:05 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Jul 2012 00:56:05 +0200
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
Message-ID: <CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>

On Wed, Jul 25, 2012 at 12:34 AM, Dan Stromberg <drsalists at gmail.com> wrote:
>
> The subject pretty much says it, but here it is again:
>
> Is there a concurrent, nosql, opensource key-value store that works with
> Pypy, CPython 2.x, CPython 3.x and Jython?
>
> It doesn't need to be highly concurrent.  In fact, locking the entire table
> for the duration of operations would be acceptable if the others block until
> the lock is released.

I think people get some of the stuff that works on python 2 working on
PyPy, but I don't remember details. Jython might be massively more
tricky though.

Cheers,
fijal

From alex.gaynor at gmail.com  Wed Jul 25 00:58:03 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 24 Jul 2012 15:58:03 -0700
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
	<CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
Message-ID: <CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>

On Tue, Jul 24, 2012 at 3:56 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Wed, Jul 25, 2012 at 12:34 AM, Dan Stromberg <drsalists at gmail.com>
> wrote:
> >
> > The subject pretty much says it, but here it is again:
> >
> > Is there a concurrent, nosql, opensource key-value store that works with
> > Pypy, CPython 2.x, CPython 3.x and Jython?
> >
> > It doesn't need to be highly concurrent.  In fact, locking the entire
> table
> > for the duration of operations would be acceptable if the others block
> until
> > the lock is released.
>
> I think people get some of the stuff that works on python 2 working on
> PyPy, but I don't remember details. Jython might be massively more
> tricky though.
>
> Cheers,
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Are you looking for a database actually written in Python, or just one with
a python client?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120724/294f4f06/attachment.html>

From drsalists at gmail.com  Wed Jul 25 01:19:53 2012
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 24 Jul 2012 23:19:53 +0000
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
	<CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
	<CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>
Message-ID: <CAGGBd_omU7TW6LCOXh0UiQQk7SSPaVmr3tTBiNAKLzvX4FnXwg@mail.gmail.com>

On Tue, Jul 24, 2012 at 10:58 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:

>
>
>
> Are you looking for a database actually written in Python, or just one
> with a python client?


Either would be good.  But I'm guessing Jython ctypes isn't quite there yet
(wasn't last time I checked), so maybe something pure python would be
easiest to find.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120724/a09ca4a9/attachment.html>

From alex.gaynor at gmail.com  Wed Jul 25 01:20:40 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 24 Jul 2012 16:20:40 -0700
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAGGBd_omU7TW6LCOXh0UiQQk7SSPaVmr3tTBiNAKLzvX4FnXwg@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
	<CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
	<CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>
	<CAGGBd_omU7TW6LCOXh0UiQQk7SSPaVmr3tTBiNAKLzvX4FnXwg@mail.gmail.com>
Message-ID: <CAFRnB2U24jM0Jmda9YtKy-S9QZR8TtTZ906Jd5EjadcDdpRPjQ@mail.gmail.com>

On Tue, Jul 24, 2012 at 4:19 PM, Dan Stromberg <drsalists at gmail.com> wrote:

>
> On Tue, Jul 24, 2012 at 10:58 PM, Alex Gaynor <alex.gaynor at gmail.com>wrote:
>
>>
>>
>>
>> Are you looking for a database actually written in Python, or just one
>> with a python client?
>
>
> Either would be good.  But I'm guessing Jython ctypes isn't quite there
> yet (wasn't last time I checked), so maybe something pure python would be
> easiest to find.
>
>
Both redis and memcached have pure python clients.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120724/0563bddb/attachment.html>

From wickedgrey at gmail.com  Wed Jul 25 01:46:22 2012
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Tue, 24 Jul 2012 16:46:22 -0700
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAFRnB2U24jM0Jmda9YtKy-S9QZR8TtTZ906Jd5EjadcDdpRPjQ@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
	<CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
	<CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>
	<CAGGBd_omU7TW6LCOXh0UiQQk7SSPaVmr3tTBiNAKLzvX4FnXwg@mail.gmail.com>
	<CAFRnB2U24jM0Jmda9YtKy-S9QZR8TtTZ906Jd5EjadcDdpRPjQ@mail.gmail.com>
Message-ID: <CADa34LCLWhyBQsXoKFoCahMHhhicVnpa6VEUSSkWfX7Jh8C0gg@mail.gmail.com>

On Tue, Jul 24, 2012 at 4:20 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Both redis and memcached have pure python clients.

So does Apache CouchDB (two of them, in fact - couchdb-python, which
seems a bit dusty, and couchdbkit, which I haven't used).

Eli

From dynamicgl at gmail.com  Wed Jul 25 06:27:15 2012
From: dynamicgl at gmail.com (gelin yan)
Date: Wed, 25 Jul 2012 12:27:15 +0800
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CADa34LCLWhyBQsXoKFoCahMHhhicVnpa6VEUSSkWfX7Jh8C0gg@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
	<CAK5idxSe1zbisPPw79--aFNrnKZ6WYZV4CZb4dPzdPTdFs5Scg@mail.gmail.com>
	<CAFRnB2VL=LS7NNwAzTkeGGNNQQc+DAXf6uwrU95bY1_ymDYqbg@mail.gmail.com>
	<CAGGBd_omU7TW6LCOXh0UiQQk7SSPaVmr3tTBiNAKLzvX4FnXwg@mail.gmail.com>
	<CAFRnB2U24jM0Jmda9YtKy-S9QZR8TtTZ906Jd5EjadcDdpRPjQ@mail.gmail.com>
	<CADa34LCLWhyBQsXoKFoCahMHhhicVnpa6VEUSSkWfX7Jh8C0gg@mail.gmail.com>
Message-ID: <CABkOF6T2ytUvrZnvRgDDD=-trnNePiNBr=YT=qnfyxqg3osqAg@mail.gmail.com>

On Wed, Jul 25, 2012 at 7:46 AM, Eli Stevens (Gmail)
<wickedgrey at gmail.com>wrote:

> On Tue, Jul 24, 2012 at 4:20 PM, Alex Gaynor <alex.gaynor at gmail.com>
> wrote:
> > Both redis and memcached have pure python clients.
>
> So does Apache CouchDB (two of them, in fact - couchdb-python, which
> seems a bit dusty, and couchdbkit, which I haven't used).
>
> Eli
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>


Hi

   Mongodb probably works too...You also may try Kyoto cabinet which has a
python wrapper.

Regards

gelin yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120725/a62f88be/attachment.html>

From stefan_ml at behnel.de  Wed Jul 25 08:01:00 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 08:01:00 +0200
Subject: [pypy-dev] Concurrent, nosql, opensource,
 key-value datastore that works with CPython, Pypy and Jython?
In-Reply-To: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
References: <CAGGBd_p4y4zy0durR+9iBd4hk9gMRkJ1ZLnmCtH-XcrGS4CMmA@mail.gmail.com>
Message-ID: <juo22s$d0c$1@dough.gmane.org>

Dan Stromberg, 25.07.2012 00:34:
> The subject pretty much says it, but here it is again:
> 
> Is there a concurrent, nosql, opensource key-value store that works with
> Pypy, CPython 2.x, CPython 3.x and Jython?
> 
> It doesn't need to be highly concurrent.  In fact, locking the entire table
> for the duration of operations would be acceptable if the others block
> until the lock is released.

Doesn't any of the *dbm modules work in all three? Or at least one of them
in each? They have the same interfaces, after all, and I doubt that you'd
have to exchange data between the three platforms by accessing the same
database store.

Stefan



From chirag.jadwani at gmail.com  Wed Jul 25 17:30:55 2012
From: chirag.jadwani at gmail.com (Chirag Jadwani)
Date: Wed, 25 Jul 2012 21:00:55 +0530
Subject: [pypy-dev] Any easy tasks for newcomers?
Message-ID: <CAKBn7ZBQ88VN=_v+AOwOkOdyVeVC61KgQ+LNBLT+b5CwkqsS=A@mail.gmail.com>

Hi all,

I am interested in contributing to PyPy.  I have a total of ~2 years of
experience as a Django dev and some (read-only) knowledge of C.  I can be
available 12 hours/week, possibly more if I get addicted.  Any thing (other
than documentation; bad English) for me to do?

Best

- Chirag
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120725/8bcc36da/attachment.html>

From fijall at gmail.com  Wed Jul 25 17:48:40 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Jul 2012 17:48:40 +0200
Subject: [pypy-dev] Any easy tasks for newcomers?
In-Reply-To: <CAKBn7ZBQ88VN=_v+AOwOkOdyVeVC61KgQ+LNBLT+b5CwkqsS=A@mail.gmail.com>
References: <CAKBn7ZBQ88VN=_v+AOwOkOdyVeVC61KgQ+LNBLT+b5CwkqsS=A@mail.gmail.com>
Message-ID: <CAK5idxQ5vr-MdyWdt-n5eiTxMFj45sivi0SbzeK4cuAZZzvPnQ@mail.gmail.com>

On Wed, Jul 25, 2012 at 5:30 PM, Chirag Jadwani
<chirag.jadwani at gmail.com> wrote:
> Hi all,
>
> I am interested in contributing to PyPy.  I have a total of ~2 years of
> experience as a Django dev and some (read-only) knowledge of C.  I can be
> available 12 hours/week, possibly more if I get addicted.  Any thing (other
> than documentation; bad English) for me to do?
>
> Best

Hi Chirag

PyPy is a vast project and there are various tasks for various levels
of involvement. It would be cool if you can identify some areas where
you are interested. Ideally come to #pypy on IRC and we can talk.

Cheers,
fijal

From dynamicgl at gmail.com  Mon Jul 30 11:09:46 2012
From: dynamicgl at gmail.com (gelin yan)
Date: Mon, 30 Jul 2012 17:09:46 +0800
Subject: [pypy-dev] any tutorial for compiling pypy on windows?
Message-ID: <CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w@mail.gmail.com>

Hi All

     I spent almost three hours for compiling pypy on my windows 7 and got
nothing. During the period of translating, I saw many errors about no file
"expat.h", "zlib.h" sth like that. I have tried to located those files on
different paths but it still failed.

    I have searched on google a bit however there is only one post which
give a brief introduction how to compile pypy 1.4 on windows. It is kinda
outdated and obscure.

     At least I want to know where I can locate those third-party
dependencies correctly so that translator can find them.Thanks.

Regards

gelin yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/1a78a601/attachment.html>

From n210241048576 at gmail.com  Mon Jul 30 12:45:36 2012
From: n210241048576 at gmail.com (Robert Grosse)
Date: Mon, 30 Jul 2012 06:45:36 -0400
Subject: [pypy-dev] pypy-dev Digest, Vol 15, Issue 33
In-Reply-To: <mailman.21.1343642402.14375.pypy-dev@python.org>
References: <mailman.21.1343642402.14375.pypy-dev@python.org>
Message-ID: <CALJjGiuEHVvF1g=zsnqRY07SxDM1PBcnApUK-rCW0LnE3aErpg@mail.gmail.com>

Are you trying to compile it on 64bit Windows? As I understand it, 64 bit
Windows isn't supported. I had issues with this too.

On Mon, Jul 30, 2012 at 6:00 AM, <pypy-dev-request at python.org> wrote:

> Send pypy-dev mailing list submissions to
>         pypy-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.python.org/mailman/listinfo/pypy-dev
> or, via email, send a message with subject or body 'help' to
>         pypy-dev-request at python.org
>
> You can reach the person managing the list at
>         pypy-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of pypy-dev digest..."
>
>
> Today's Topics:
>
>    1. any tutorial for compiling pypy on windows? (gelin yan)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 30 Jul 2012 17:09:46 +0800
> From: gelin yan <dynamicgl at gmail.com>
> To: PyPy Developer Mailing List <pypy-dev at python.org>
> Subject: [pypy-dev] any tutorial for compiling pypy on windows?
> Message-ID:
>         <
> CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi All
>
>      I spent almost three hours for compiling pypy on my windows 7 and got
> nothing. During the period of translating, I saw many errors about no file
> "expat.h", "zlib.h" sth like that. I have tried to located those files on
> different paths but it still failed.
>
>     I have searched on google a bit however there is only one post which
> give a brief introduction how to compile pypy 1.4 on windows. It is kinda
> outdated and obscure.
>
>      At least I want to know where I can locate those third-party
> dependencies correctly so that translator can find them.Thanks.
>
> Regards
>
> gelin yan
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/pypy-dev/attachments/20120730/1a78a601/attachment-0001.html
> >
>
> ------------------------------
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
> End of pypy-dev Digest, Vol 15, Issue 33
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/cb955ce5/attachment.html>

From dynamicgl at gmail.com  Mon Jul 30 12:52:02 2012
From: dynamicgl at gmail.com (gelin yan)
Date: Mon, 30 Jul 2012 18:52:02 +0800
Subject: [pypy-dev] pypy-dev Digest, Vol 15, Issue 33
In-Reply-To: <CALJjGiuEHVvF1g=zsnqRY07SxDM1PBcnApUK-rCW0LnE3aErpg@mail.gmail.com>
References: <mailman.21.1343642402.14375.pypy-dev@python.org>
	<CALJjGiuEHVvF1g=zsnqRY07SxDM1PBcnApUK-rCW0LnE3aErpg@mail.gmail.com>
Message-ID: <CABkOF6S+og3_UeXcekk9dLtoQ5ZbLsBX7AXv3tZOYyWF6+AiRg@mail.gmail.com>

On Mon, Jul 30, 2012 at 6:45 PM, Robert Grosse <n210241048576 at gmail.com>wrote:

> Are you trying to compile it on 64bit Windows? As I understand it, 64 bit
> Windows isn't supported. I had issues with this too.
>
> On Mon, Jul 30, 2012 at 6:00 AM, <pypy-dev-request at python.org> wrote:
>
>> Send pypy-dev mailing list submissions to
>>         pypy-dev at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://mail.python.org/mailman/listinfo/pypy-dev
>> or, via email, send a message with subject or body 'help' to
>>         pypy-dev-request at python.org
>>
>> You can reach the person managing the list at
>>         pypy-dev-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of pypy-dev digest..."
>>
>>
>> Today's Topics:
>>
>>    1. any tutorial for compiling pypy on windows? (gelin yan)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 30 Jul 2012 17:09:46 +0800
>> From: gelin yan <dynamicgl at gmail.com>
>> To: PyPy Developer Mailing List <pypy-dev at python.org>
>> Subject: [pypy-dev] any tutorial for compiling pypy on windows?
>> Message-ID:
>>         <
>> CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi All
>>
>>      I spent almost three hours for compiling pypy on my windows 7 and got
>> nothing. During the period of translating, I saw many errors about no file
>> "expat.h", "zlib.h" sth like that. I have tried to located those files on
>> different paths but it still failed.
>>
>>     I have searched on google a bit however there is only one post which
>> give a brief introduction how to compile pypy 1.4 on windows. It is kinda
>> outdated and obscure.
>>
>>      At least I want to know where I can locate those third-party
>> dependencies correctly so that translator can find them.Thanks.
>>
>> Regards
>>
>> gelin yan
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.python.org/pipermail/pypy-dev/attachments/20120730/1a78a601/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>> End of pypy-dev Digest, Vol 15, Issue 33
>> ****************************************
>>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>

Hi All

   more detail here (sorry forget that)

Os: win7 64 bit
compiler: VS2008 full version

python version: 2.7.3 (32 bit)

I want to build a pypy 32 bit only. I know pypy doesn't support 64 bit on
windows currently.

Regards

gelin yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/851015c7/attachment.html>

From matti.picus at gmail.com  Mon Jul 30 18:01:21 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 31 Jul 2012 02:01:21 +1000
Subject: [pypy-dev] pypy-dev Digest, Vol 15, Issue 33
In-Reply-To: <CABkOF6S+og3_UeXcekk9dLtoQ5ZbLsBX7AXv3tZOYyWF6+AiRg@mail.gmail.com>
References: <mailman.21.1343642402.14375.pypy-dev@python.org>
	<CALJjGiuEHVvF1g=zsnqRY07SxDM1PBcnApUK-rCW0LnE3aErpg@mail.gmail.com>
	<CABkOF6S+og3_UeXcekk9dLtoQ5ZbLsBX7AXv3tZOYyWF6+AiRg@mail.gmail.com>
Message-ID: <5016AFD1.1040500@gmail.com>

Searching for "pypy windows compile" on google gave me this link as the 
top result, which is indeed the latest and greatest instructions for 
compiling on Windows:
http://pypy.readthedocs.org/en/latest/windows.html
Please let us know if the instructions are unclear, suggested 
modifications would be apprectiated.

Note that you do not need to translate (compile) pypy in order to use 
it, in fact the recommended way to use it is to unzip a binary and use 
it in a virtual environment, as documented here:
http://pypy.readthedocs.org/en/latest/getting-started.html#download-a-pre-built-pypy
however these are translated with studio .net 2005

Feedback is welcome
Matti

On 30/07/2012 8:52 PM, gelin yan wrote:
>
> Hi All
>
>    more detail here (sorry forget that)
>
> Os: win7 64 bit
> compiler: VS2008 full version
>
> python version: 2.7.3 (32 bit)
>
> I want to build a pypy 32 bit only. I know pypy doesn't support 64 bit 
> on windows currently.
>
> Regards
>
> gelin yan

From chris at kateandchris.net  Tue Jul 31 01:52:58 2012
From: chris at kateandchris.net (Chris Lambacher)
Date: Mon, 30 Jul 2012 19:52:58 -0400
Subject: [pypy-dev] any tutorial for compiling pypy on windows?
In-Reply-To: <CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w@mail.gmail.com>
References: <CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w@mail.gmail.com>
Message-ID: <CAAXXHg+Ky01Xj1SvoTfJLrUakU0_UtnnwxS+Oq1-KzKS13SfXA@mail.gmail.com>

It's been a while since I set this up, so I'm hazy on some of the details.

Pypy builds on windows may look for dependencies in other locations but the
way I had recommended to me and which worked was parallel to the pypy
checkout. For example I have:
c:\work\pypy
      |
      |- bzip2-1.0.5
      |- expat-2.0.1
      |- gc-7.1
      |- openssl-0.9.8q
      |-pypy ( this is the pypy working tree )
      |-zlib-1.2.3

It has been a while since I tried so the exact versions may be incorrect.
The other thing that I had trouble with was the version of the Visual C++
compiler. You need to use the same version that was used to compile the
python interpreter you use to do the translation. For python 2.6 and 2.7
this is visual C++ 2008 which is difficult to obtain. You can get the
express version (which worked for me) here:
http://www.microsoft.com/visualstudio/en-us/products/2008-editions/express

Good luck and try asking on freenode in #pypy, the one or two windows guys
are helpful when they are around.

-Chris


On Mon, Jul 30, 2012 at 5:09 AM, gelin yan <dynamicgl at gmail.com> wrote:

> Hi All
>
>      I spent almost three hours for compiling pypy on my windows 7 and got
> nothing. During the period of translating, I saw many errors about no file
> "expat.h", "zlib.h" sth like that. I have tried to located those files on
> different paths but it still failed.
>
>     I have searched on google a bit however there is only one post which
> give a brief introduction how to compile pypy 1.4 on windows. It is kinda
> outdated and obscure.
>
>      At least I want to know where I can locate those third-party
> dependencies correctly so that translator can find them.Thanks.
>
> Regards
>
> gelin yan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Christopher Lambacher
chris at kateandchris.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/2e256842/attachment-0001.html>

From qwcode at gmail.com  Tue Jul 31 05:04:21 2012
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 30 Jul 2012 20:04:21 -0700
Subject: [pypy-dev] __pycache__ folders
Message-ID: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>

Hello:

I'm working on trying to get the pip test suite passing for pypy.
I need to figure out the mechanism that turns on/off  the use of
 __pycache__ folders.
locally in my pypy v1.9 install, I only see normal *.pyc files for
installed distributions.
In Travis service builds which also reports v1.9, there are __pycache__
folders with *.pypy-19.pyc files.

How is this set or determined in pypy?

any help appreciated,
thanks
Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/c226ab9a/attachment.html>

From benjamin at python.org  Tue Jul 31 05:28:18 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 30 Jul 2012 20:28:18 -0700
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
Message-ID: <CAPZV6o-P=DMzWo-KSX+ksV8W11g2DWkJFmVJ465KHQQeCcfLLg@mail.gmail.com>

2012/7/30 Marcus Smith <qwcode at gmail.com>:
> Hello:
>
> I'm working on trying to get the pip test suite passing for pypy.
> I need to figure out the mechanism that turns on/off  the use of
> __pycache__ folders.
> locally in my pypy v1.9 install, I only see normal *.pyc files for installed
> distributions.
> In Travis service builds which also reports v1.9, there are __pycache__
> folders with *.pypy-19.pyc files.
>
> How is this set or determined in pypy?

py.test is probably creating them.



-- 
Regards,
Benjamin

From matti.picus at gmail.com  Tue Jul 31 05:34:46 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 31 Jul 2012 13:34:46 +1000
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAPZV6o-P=DMzWo-KSX+ksV8W11g2DWkJFmVJ465KHQQeCcfLLg@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPZV6o-P=DMzWo-KSX+ksV8W11g2DWkJFmVJ465KHQQeCcfLLg@mail.gmail.com>
Message-ID: <50175256.5010509@gmail.com>

IMHO, *.pyc files should not be distributed with a binary pypy. This is 
probably an issue for the team who supply the pypy for travis-ci.org 
(since the __pycache__ files do not appear in the nightly or release 
builds on pypy.org) - I think they live at https://launchpad.net/~pypy 
<https://launchpad.net/%7Epypy>
Matti

On 31/07/2012 1:28 PM, Benjamin Peterson wrote:
> 2012/7/30 Marcus Smith <qwcode at gmail.com>:
>> Hello:
>>
>> I'm working on trying to get the pip test suite passing for pypy.
>> I need to figure out the mechanism that turns on/off  the use of
>> __pycache__ folders.
>> locally in my pypy v1.9 install, I only see normal *.pyc files for installed
>> distributions.
>> In Travis service builds which also reports v1.9, there are __pycache__
>> folders with *.pypy-19.pyc files.
>>
>> How is this set or determined in pypy?
> py.test is probably creating them.
>
>   
>


From qwcode at gmail.com  Tue Jul 31 06:27:13 2012
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 30 Jul 2012 21:27:13 -0700
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
Message-ID: <CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>

let me explain some more.  I have may have thrown people with my use of
"distribution".
I just meant installed packages that you get from pypi or wherever, not the
pypy distribution itself.

e.g. one of our nose test cases installs and then uninstalls INITools (from
pypi)
it's not expecting to find a __pycache__ folder when doing an uninstall,
but it's there in the travis build,
despite the "installed-files.txt" manifest (in the egg-info dir) listing
normal inline *.pyc files

that's not so bad, except that locally, I don't know how to "turn on"
__pycache__ folders for pypy so I can troubleshoot and get a workaround.
I only ever get inline *.pyc files.

here's the travis log for that one test. It's got some of my custom
logging,
so not so clear what's going on, but you can see me logging the contents of
the __pycache__ dir,
as the code attempts to clear the initools pkg directory.
http://travis-ci.org/#!/pypa/pip/builds/1996370

thanks
Marcus



On Mon, Jul 30, 2012 at 8:04 PM, Marcus Smith <qwcode at gmail.com> wrote:

> Hello:
>
> I'm working on trying to get the pip test suite passing for pypy.
> I need to figure out the mechanism that turns on/off  the use of
>  __pycache__ folders.
> locally in my pypy v1.9 install, I only see normal *.pyc files for
> installed distributions.
> In Travis service builds which also reports v1.9, there are __pycache__
> folders with *.pypy-19.pyc files.
>
> How is this set or determined in pypy?
>
> any help appreciated,
> thanks
> Marcus
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120730/125dad09/attachment.html>

From dynamicgl at gmail.com  Tue Jul 31 08:09:35 2012
From: dynamicgl at gmail.com (gelin yan)
Date: Tue, 31 Jul 2012 14:09:35 +0800
Subject: [pypy-dev] any tutorial for compiling pypy on windows?
In-Reply-To: <CAAXXHg+Ky01Xj1SvoTfJLrUakU0_UtnnwxS+Oq1-KzKS13SfXA@mail.gmail.com>
References: <CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w@mail.gmail.com>
	<CAAXXHg+Ky01Xj1SvoTfJLrUakU0_UtnnwxS+Oq1-KzKS13SfXA@mail.gmail.com>
Message-ID: <CABkOF6TxNB7-PGOM4CzNnUMUXAn_hDaZu4YASPvxrfnD1AvmVQ@mail.gmail.com>

On Tue, Jul 31, 2012 at 7:52 AM, Chris Lambacher <chris at kateandchris.net>wrote:

> It's been a while since I set this up, so I'm hazy on some of the details.
>
> Pypy builds on windows may look for dependencies in other locations but
> the way I had recommended to me and which worked was parallel to the pypy
> checkout. For example I have:
> c:\work\pypy
>       |
>       |- bzip2-1.0.5
>       |- expat-2.0.1
>       |- gc-7.1
>       |- openssl-0.9.8q
>       |-pypy ( this is the pypy working tree )
>       |-zlib-1.2.3
>
> It has been a while since I tried so the exact versions may be incorrect.
> The other thing that I had trouble with was the version of the Visual C++
> compiler. You need to use the same version that was used to compile the
> python interpreter you use to do the translation. For python 2.6 and 2.7
> this is visual C++ 2008 which is difficult to obtain. You can get the
> express version (which worked for me) here:
> http://www.microsoft.com/visualstudio/en-us/products/2008-editions/express
>
> Good luck and try asking on freenode in #pypy, the one or two windows guys
> are helpful when they are around.
>
> -Chris
>
>
> On Mon, Jul 30, 2012 at 5:09 AM, gelin yan <dynamicgl at gmail.com> wrote:
>
>> Hi All
>>
>>      I spent almost three hours for compiling pypy on my windows 7 and
>> got nothing. During the period of translating, I saw many errors about no
>> file "expat.h", "zlib.h" sth like that. I have tried to located those files
>> on different paths but it still failed.
>>
>>     I have searched on google a bit however there is only one post which
>> give a brief introduction how to compile pypy 1.4 on windows. It is kinda
>> outdated and obscure.
>>
>>      At least I want to know where I can locate those third-party
>> dependencies correctly so that translator can find them.Thanks.
>>
>> Regards
>>
>> gelin yan
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
>
> --
> Christopher Lambacher
> chris at kateandchris.ne <chris at kateandchris.net>t
>

Hi Chris

     Thanks for your info. I tried the structure as you guided but the c
compiler still complained "can't open include file "expat.h"
What exact path do I need to set?

Regards

gelin yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120731/846ebece/attachment-0001.html>

From arigo at tunes.org  Tue Jul 31 09:25:08 2012
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 31 Jul 2012 09:25:08 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
Message-ID: <CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>

Hi Marcus,

PyPy contains no logic to create or search for __pycache__.  A grep
tells me that neither does CPython 2.7.  You are probably confused by
py.test creating them.


A bient?t,

Armin.

From holger at merlinux.eu  Tue Jul 31 10:27:40 2012
From: holger at merlinux.eu (holger krekel)
Date: Tue, 31 Jul 2012 08:27:40 +0000
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
Message-ID: <20120731082740.GN26019@merlinux.eu>

Hi Marcus, Armin, 

On Tue, Jul 31, 2012 at 09:25 +0200, Armin Rigo wrote:
> Hi Marcus,
> 
> PyPy contains no logic to create or search for __pycache__.  A grep
> tells me that neither does CPython 2.7.  You are probably confused by
> py.test creating them.

... creating them during a pypy test run - the artificats (*.pyc files) of 
such runs should not be distributed as Matti points out.  Not sure it helps
in this case but in general you can set PYTHONDONTWRITEBYTECODE=1 to
prevent writing of such files.

best,
holger

> 
> A bient?t,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 

From stefano at rivera.za.net  Tue Jul 31 11:44:19 2012
From: stefano at rivera.za.net (Stefano Rivera)
Date: Tue, 31 Jul 2012 11:44:19 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
Message-ID: <20120731094419.GT1912@bach.rivera.co.za>

Hi Armin (2012.07.31_09:25:08_+0200)
> PyPy contains no logic to create or search for __pycache__.  A grep
> tells me that neither does CPython 2.7.  You are probably confused by
> py.test creating them.

A Debian/Ubuntu pypy does, though. And IIRC, that's what travis CI is
using.

It's the pep3147 patches:
http://anonscm.debian.org/gitweb/?p=collab-maint/pypy.git;a=tree;f=debian/patches

I thought it would be a useful feature, to make transition between pypy
versions in Debian easier. Proposed the patches in #pypy, and the
reception was lukewarm, so I never pushed harder.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

From arigo at tunes.org  Tue Jul 31 12:10:01 2012
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 31 Jul 2012 12:10:01 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <20120731094419.GT1912@bach.rivera.co.za>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
Message-ID: <CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>

Hi Stefano,

On Tue, Jul 31, 2012 at 11:44 AM, Stefano Rivera <stefano at rivera.za.net> wrote:
>> PyPy contains no logic to create or search for __pycache__.  A grep
>> tells me that neither does CPython 2.7.
>
> A Debian/Ubuntu pypy does, though.

I think that PyPy shouldn't do that, because CPython doesn't.  If
Debian/Ubuntu also hacked CPython to do that, then fine; we are then
free to point users to them (blame them?) for issues :-)


A bient?t,

Armin.

From stefano at rivera.za.net  Tue Jul 31 12:35:56 2012
From: stefano at rivera.za.net (Stefano Rivera)
Date: Tue, 31 Jul 2012 12:35:56 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
Message-ID: <20120731103556.GU1911@bach.rivera.co.za>

Hi Armin (2012.07.31_12:10:01_+0200)
> I think that PyPy shouldn't do that, because CPython doesn't.  If
> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
> free to point users to them (blame them?) for issues :-)

We did seriously consider hacking cpython to do that, before introducing
2.7. But by that point, there was no real benefit, as 2.7 is the last of
the 2.x series.
https://lists.debian.org/debian-python/2010/04/msg00046.html

I did it for pypy, as I thought it fairly likely that there would be a
benefit (there's no guarantee that pypy's bytecode format won't change
between pypy releases).

Only very esoteric code would break as a result of this change.

PyPy already has slightly different .pyc semantics to cpython, and has
PEP3149, a vaguely similar change, for C extensions.

It's worth stating that I did ask if there was any objection to the
patches I was applying, more than once, before the first upload to
Debian, and received none. The general opinion seemed to be that .pyc
files aren't liked much in the pypy community, at all, and this was
vaguely inline

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

From fijall at gmail.com  Tue Jul 31 12:46:23 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 31 Jul 2012 12:46:23 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <20120731103556.GU1911@bach.rivera.co.za>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
Message-ID: <CAK5idxRPmC=h+xO_0s+jZm8Z_1Umqt-tP0Nq1etAg-wwV9mKRw@mail.gmail.com>

On Tue, Jul 31, 2012 at 12:35 PM, Stefano Rivera <stefano at rivera.za.net> wrote:
> Hi Armin (2012.07.31_12:10:01_+0200)
>> I think that PyPy shouldn't do that, because CPython doesn't.  If
>> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
>> free to point users to them (blame them?) for issues :-)
>
> We did seriously consider hacking cpython to do that, before introducing
> 2.7. But by that point, there was no real benefit, as 2.7 is the last of
> the 2.x series.
> https://lists.debian.org/debian-python/2010/04/msg00046.html
>
> I did it for pypy, as I thought it fairly likely that there would be a
> benefit (there's no guarantee that pypy's bytecode format won't change
> between pypy releases).
>
> Only very esoteric code would break as a result of this change.

Never underestimate users. Just saying

>
> PyPy already has slightly different .pyc semantics to cpython, and has
> PEP3149, a vaguely similar change, for C extensions.
>
> It's worth stating that I did ask if there was any objection to the
> patches I was applying, more than once, before the first upload to
> Debian, and received none. The general opinion seemed to be that .pyc
> files aren't liked much in the pypy community, at all, and this was
> vaguely inline
>

From tismer at stackless.com  Tue Jul 31 13:05:48 2012
From: tismer at stackless.com (Christian Tismer)
Date: Tue, 31 Jul 2012 13:05:48 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <20120731103556.GU1911@bach.rivera.co.za>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
Message-ID: <5017BC0C.3030602@stackless.com>

On 31.07.12 12:35, Stefano Rivera wrote:
> Hi Armin (2012.07.31_12:10:01_+0200)
>> I think that PyPy shouldn't do that, because CPython doesn't.  If
>> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
>> free to point users to them (blame them?) for issues :-)
> We did seriously consider hacking cpython to do that, before introducing
> 2.7. But by that point, there was no real benefit, as 2.7 is the last of
> the 2.x series.
> https://lists.debian.org/debian-python/2010/04/msg00046.html
>
> I did it for pypy, as I thought it fairly likely that there would be a
> benefit (there's no guarantee that pypy's bytecode format won't change
> between pypy releases).
>
> Only very esoteric code would break as a result of this change.
>
> PyPy already has slightly different .pyc semantics to cpython, and has
> PEP3149, a vaguely similar change, for C extensions.
>
> It's worth stating that I did ask if there was any objection to the
> patches I was applying, more than once, before the first upload to
> Debian, and received none. The general opinion seemed to be that .pyc
> files aren't liked much in the pypy community, at all, and this was
> vaguely inline

Well, good to know.

Your backport makes sense, but the reasoning should be identical
between cpython and pypy.
If cpython does not do it, pypy does not do it.

The bytecode compatibility is also identical between cpython and pypy,
there is no issue here that is not there.

pypy currently shows itself as python 2.7.2, and that is the truth.
Different behavior introduced by a distribution is IMHO not a
benefit, but goes into an incompatible direction.

If a backport happens, then in CPython, and PyPy will adjusted, accordingly.
One of the incompatibility issues _is_ the unexpected behavior of tests.

To Marcus:
Now that the issue is clear, it probably makes sense to add some extra check
to setup.py that does not inquire the python version but does some probing.
Better to be defensive ;-)

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From stevenjackson121 at gmail.com  Tue Jul 31 13:25:03 2012
From: stevenjackson121 at gmail.com (Steven Jackson)
Date: Tue, 31 Jul 2012 07:25:03 -0400
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature Request)
Message-ID: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>

I know there is no current plan to implement scipy in pypy, but searching
the PyPy website, I was not able to find the reason.

If it is not to be included as a feature due to lack of interest or
developer time, I am offering to begin rewriting scipy in pure python (I
have quite a bit of free time to do this).
If it has been discussed and actively excluded from pypy, I would like to
know that before I waste too much time rewriting it.

I have an active interest in being able to use some modules which depend on
scipy while using the PyPy interpreter (especially pyBrain). I have seen
the hack on the pypy blog:

http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html

I could simply use the hack myself, but unless there's a reason not too, I
think it'd be nice to have scipy available as a pure python module (perhaps
called scipypy in the same manner as numpy).

-- 
Steven Jackson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120731/374957d1/attachment.html>

From stefano at rivera.za.net  Tue Jul 31 13:34:31 2012
From: stefano at rivera.za.net (Stefano Rivera)
Date: Tue, 31 Jul 2012 13:34:31 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <5017BC0C.3030602@stackless.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
	<5017BC0C.3030602@stackless.com>
Message-ID: <20120731113431.GV1911@bach.rivera.co.za>

Hi Christian (2012.07.31_13:05:48_+0200)
> >vaguely inline

Hrm, I never finished that sentance, because I went back and made the
point earlier in the mail.

> Your backport makes sense, but the reasoning should be identical
> between cpython and pypy.
> If cpython does not do it, pypy does not do it.

cpython has a long legacy in Debian, and many existing packages that
depend on particular semantics. PyPy can afford to make changes that
cpython can't. Using PEP3147 + 3149 means that I only need to put python
modules in one place, for all versions of PyPy to consume.

I don't want to end up having to carry multiple versions of pypy in the
archive, as we have to with cpython 2.x. With 3.x, we've already seen
the benefit of PEP3147 + 3149.

> The bytecode compatibility is also identical between cpython and pypy,
> there is no issue here that is not there.

I believe that isn't the case:
https://bitbucket.org/pypy/pypy/src/6bb09a17e2aa/pypy/module/imp/importing.py#cl-832

> pypy currently shows itself as python 2.7.2, and that is the truth.
> Different behavior introduced by a distribution is IMHO not a
> benefit, but goes into an incompatible direction.

And I do my best to minimize that. In this case, I thought it was worth
the delta.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

From fijall at gmail.com  Tue Jul 31 14:36:13 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 31 Jul 2012 14:36:13 +0200
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature
	Request)
In-Reply-To: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
References: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
Message-ID: <CAK5idxQsBMK1Q5FrCtABJ9ZeFpoYcXmDaeYRpA6aEEi+qJZyVw@mail.gmail.com>

Hi Steven.

Thanks for getting to us about it.

On Tue, Jul 31, 2012 at 1:25 PM, Steven Jackson
<stevenjackson121 at gmail.com> wrote:
> I know there is no current plan to implement scipy in pypy, but searching
> the PyPy website, I was not able to find the reason.
>
> If it is not to be included as a feature due to lack of interest or
> developer time, I am offering to begin rewriting scipy in pure python (I
> have quite a bit of free time to do this).
> If it has been discussed and actively excluded from pypy, I would like to
> know that before I waste too much time rewriting it.
>
> I have an active interest in being able to use some modules which depend on
> scipy while using the PyPy interpreter (especially pyBrain). I have seen the
> hack on the pypy blog:
>
> http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html
>
> I could simply use the hack myself, but unless there's a reason not too, I
> think it'd be nice to have scipy available as a pure python module (perhaps
> called scipypy in the same manner as numpy).
>
> --
> Steven Jackson

I would *think* that extending such a hack is the best way forward,
but I'm not 100% sure about it. Numpy also requires finishing and the
work has stalled recently (it's almost exclusively my fault, I have
been doing other stuff recently), but we're definitely looking forward
to implement a way of having scipy work one way or another.

Btw - I'm not aware, but how hard would it be to port scipy to pure
python & pure C with cffi in between?

Cheers,
fijal

From davidmenhur at gmail.com  Tue Jul 31 15:24:02 2012
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Tue, 31 Jul 2012 14:24:02 +0100
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature
	Request)
In-Reply-To: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
References: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
Message-ID: <CAJhcF=3SRExRzmLQwioGHJb3Y+aW=gdMZG6BP=otuXqThmJL=w@mail.gmail.com>

I believe the reason is that they prefer to focus the development
effort into the more "basic" Numpy and from that go on. Numpy is not
fully implemented

I am working (actually, right now I am only in the "thinking" state)
on some modules of NumPy, maybe we could coordinate and give this a
boost.

On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson
<stevenjackson121 at gmail.com> wrote:
> I know there is no current plan to implement scipy in pypy, but searching
> the PyPy website, I was not able to find the reason.
>
> If it is not to be included as a feature due to lack of interest or
> developer time, I am offering to begin rewriting scipy in pure python (I
> have quite a bit of free time to do this).
> If it has been discussed and actively excluded from pypy, I would like to
> know that before I waste too much time rewriting it.
>
> I have an active interest in being able to use some modules which depend on
> scipy while using the PyPy interpreter (especially pyBrain). I have seen the
> hack on the pypy blog:
>
> http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html
>
> I could simply use the hack myself, but unless there's a reason not too, I
> think it'd be nice to have scipy available as a pure python module (perhaps
> called scipypy in the same manner as numpy).
>
> --
> Steven Jackson
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From stevenjackson121 at gmail.com  Tue Jul 31 15:41:45 2012
From: stevenjackson121 at gmail.com (Steven Jackson)
Date: Tue, 31 Jul 2012 09:41:45 -0400
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature
	Request)
In-Reply-To: <CAJhcF=3SRExRzmLQwioGHJb3Y+aW=gdMZG6BP=otuXqThmJL=w@mail.gmail.com>
References: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
	<CAJhcF=3SRExRzmLQwioGHJb3Y+aW=gdMZG6BP=otuXqThmJL=w@mail.gmail.com>
Message-ID: <CAPTidvin2E4W5rRwXRtv4QF4Zu1Df9_4dN_UdJYd8=0B09jLKg@mail.gmail.com>

I am definitely open to working on Numpy first, my only hesitation was that
it might be hard to get spun up, and scipy seemed like something I could
slice off and do (some of) by myself without accidentally conflicting with
ongoing development efforts.

If, as your replies have indicated, Numpy isn't being very actively
developed, and it would be easy to get caught up and avoid conflicting with
other developers, then I'd be glad to help there instead of independently
trying to get scipy going.


On Tue, Jul 31, 2012 at 9:24 AM, Da?id <davidmenhur at gmail.com> wrote:

> I believe the reason is that they prefer to focus the development
> effort into the more "basic" Numpy and from that go on. Numpy is not
> fully implemented
>
> I am working (actually, right now I am only in the "thinking" state)
> on some modules of NumPy, maybe we could coordinate and give this a
> boost.
>
> On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson
> <stevenjackson121 at gmail.com> wrote:
> > I know there is no current plan to implement scipy in pypy, but searching
> > the PyPy website, I was not able to find the reason.
> >
> > If it is not to be included as a feature due to lack of interest or
> > developer time, I am offering to begin rewriting scipy in pure python (I
> > have quite a bit of free time to do this).
> > If it has been discussed and actively excluded from pypy, I would like to
> > know that before I waste too much time rewriting it.
> >
> > I have an active interest in being able to use some modules which depend
> on
> > scipy while using the PyPy interpreter (especially pyBrain). I have seen
> the
> > hack on the pypy blog:
> >
> >
> http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html
> >
> > I could simply use the hack myself, but unless there's a reason not too,
> I
> > think it'd be nice to have scipy available as a pure python module
> (perhaps
> > called scipypy in the same manner as numpy).
> >
> > --
> > Steven Jackson
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
>



-- 
Steven Jackson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120731/c85eae30/attachment.html>

From fijall at gmail.com  Tue Jul 31 15:55:21 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 31 Jul 2012 15:55:21 +0200
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature
	Request)
In-Reply-To: <CAPTidvin2E4W5rRwXRtv4QF4Zu1Df9_4dN_UdJYd8=0B09jLKg@mail.gmail.com>
References: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
	<CAJhcF=3SRExRzmLQwioGHJb3Y+aW=gdMZG6BP=otuXqThmJL=w@mail.gmail.com>
	<CAPTidvin2E4W5rRwXRtv4QF4Zu1Df9_4dN_UdJYd8=0B09jLKg@mail.gmail.com>
Message-ID: <CAK5idxTReTPY_tkb6fHdz_UmeLS+7RZ-CTMYw7q_4Hrg5nh7tQ@mail.gmail.com>

On Tue, Jul 31, 2012 at 3:41 PM, Steven Jackson
<stevenjackson121 at gmail.com> wrote:
> I am definitely open to working on Numpy first, my only hesitation was that
> it might be hard to get spun up, and scipy seemed like something I could
> slice off and do (some of) by myself without accidentally conflicting with
> ongoing development efforts.
>
> If, as your replies have indicated, Numpy isn't being very actively
> developed, and it would be easy to get caught up and avoid conflicting with
> other developers, then I'd be glad to help there instead of independently
> trying to get scipy going.

That sounds very good to me. The thing I started (and I'm not actively
working on it for at least a week or two more) is to clean up stuff
and try to reuse pure-python numpy part without changes

>
>
>
> On Tue, Jul 31, 2012 at 9:24 AM, Da?id <davidmenhur at gmail.com> wrote:
>>
>> I believe the reason is that they prefer to focus the development
>> effort into the more "basic" Numpy and from that go on. Numpy is not
>> fully implemented
>>
>> I am working (actually, right now I am only in the "thinking" state)
>> on some modules of NumPy, maybe we could coordinate and give this a
>> boost.
>>
>> On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson
>> <stevenjackson121 at gmail.com> wrote:
>> > I know there is no current plan to implement scipy in pypy, but
>> > searching
>> > the PyPy website, I was not able to find the reason.
>> >
>> > If it is not to be included as a feature due to lack of interest or
>> > developer time, I am offering to begin rewriting scipy in pure python (I
>> > have quite a bit of free time to do this).
>> > If it has been discussed and actively excluded from pypy, I would like
>> > to
>> > know that before I waste too much time rewriting it.
>> >
>> > I have an active interest in being able to use some modules which depend
>> > on
>> > scipy while using the PyPy interpreter (especially pyBrain). I have seen
>> > the
>> > hack on the pypy blog:
>> >
>> >
>> > http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html
>> >
>> > I could simply use the hack myself, but unless there's a reason not too,
>> > I
>> > think it'd be nice to have scipy available as a pure python module
>> > (perhaps
>> > called scipypy in the same manner as numpy).
>> >
>> > --
>> > Steven Jackson
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > http://mail.python.org/mailman/listinfo/pypy-dev
>> >
>
>
>
>
> --
> Steven Jackson
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Tue Jul 31 16:26:50 2012
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 31 Jul 2012 16:26:50 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <20120731113431.GV1911@bach.rivera.co.za>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
	<5017BC0C.3030602@stackless.com>
	<20120731113431.GV1911@bach.rivera.co.za>
Message-ID: <CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>

Hi Stefano,

On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera <stefano at rivera.za.net> wrote:
> And I do my best to minimize that. In this case, I thought it was worth
> the delta.

My point of view is that this goes in the same pack of
incompatibilities introduced by some distributions as splitting the
install path (which we recommend to install as one block in
/opt/pypy-1.9).  It may be useful to do that according to some
distribution-specific rules, but either you do it as a distribution
hack and we don't care about it, or you try to come here and discuss
more in details the problems that you have and how we could solve it
together.  As far as I know the __pycache__ is another example of
distribution-specific hack that we were was not aware of.  I don't
think we are by principle necessarily completely opposed to it:
whereas it still looks like a good idea to stick to CPython's
behavior, failing that, if there are good reasons, the next best
solution is probably to come with something that we can integrate into
the "real" pypy, rather than keep on your own.  If you don't, users
complain here and you are just making us hate .pyc files even more.
:-)


A bient?t,

Armin.

From stefano at rivera.za.net  Tue Jul 31 17:05:03 2012
From: stefano at rivera.za.net (Stefano Rivera)
Date: Tue, 31 Jul 2012 17:05:03 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
	<5017BC0C.3030602@stackless.com>
	<20120731113431.GV1911@bach.rivera.co.za>
	<CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>
Message-ID: <20120731150503.GV1912@bach.rivera.co.za>

Hi Armin (2012.07.31_16:26:50_+0200)
> My point of view is that this goes in the same pack of
> incompatibilities introduced by some distributions as splitting the
> install path (which we recommend to install as one block in
> /opt/pypy-1.9).

Most distros won't use /opt. We tend to follow the FHS.
/opt makes a lot of sense for the binary pypy you distribute. Just as it
makes sense for the binary junk that Adobe et al. distribute.

> It may be useful to do that according to some distribution-specific
> rules, but either you do it as a distribution hack and we don't care
> about it, or you try to come here and discuss more in details the
> problems that you have and how we could solve it together.

Having the same installation layout on all distros would be fantastic.
Practically, that gets non-trivial, though. Debian has the weird
dist-packages layout for cpython, that I copied for pypy, but that
approach hasn't been documented in upstream cpython or adopted by any
other distributions.

> As far as I know the __pycache__ is another example of
> distribution-specific hack that we were was not aware of.

I did my best to make you all aware of it, and get some feedback.
I mentioned the PEP3147 support here [0], and a zillion times on IRC.
But I have no problem having that conversation now that people seem to
be aware of it :)

[0]: http://mail.python.org/pipermail/pypy-dev/2012-January/009104.html

> I don't think we are by principle necessarily completely opposed to it:
> whereas it still looks like a good idea to stick to CPython's
> behavior, failing that, if there are good reasons, the next best
> solution is probably to come with something that we can integrate into
> the "real" pypy, rather than keep on your own.

Yup, that's the opinion I felt before. And the approach I'm taking. But
the patch is fairly gnarly due to test-suite changes, so I haven't
forwarded it up to you.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

From Ronny.Pfannschmidt at gmx.de  Tue Jul 31 17:12:15 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 31 Jul 2012 17:12:15 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
	<5017BC0C.3030602@stackless.com>
	<20120731113431.GV1911@bach.rivera.co.za>
	<CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>
Message-ID: <5017F5CF.5080704@gmx.de>

Hi,

<rant>
i would dare to say that Distributions that manage semi-compatible,
but distro specific "forks" of open source projects
instead of aiming for a distro compatible upstream
create quite some grief for developers.

after all users don't tend to notice those incompatibilities,
but it breaks the tooling of developers

I'm at a point where i need to run self-compiled pythons,
since the ones in Debian for example are broken
for some of my use-cases by Debian specific patches/changes

what makes it worse is that the error reporting paths for the distro 
specific "forks" aren't made clear
</rant>

-- Ronny

On 07/31/2012 04:26 PM, Armin Rigo wrote:
> Hi Stefano,
>
> On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera<stefano at rivera.za.net>  wrote:
>> And I do my best to minimize that. In this case, I thought it was worth
>> the delta.
>
> My point of view is that this goes in the same pack of
> incompatibilities introduced by some distributions as splitting the
> install path (which we recommend to install as one block in
> /opt/pypy-1.9).  It may be useful to do that according to some
> distribution-specific rules, but either you do it as a distribution
> hack and we don't care about it, or you try to come here and discuss
> more in details the problems that you have and how we could solve it
> together.  As far as I know the __pycache__ is another example of
> distribution-specific hack that we were was not aware of.  I don't
> think we are by principle necessarily completely opposed to it:
> whereas it still looks like a good idea to stick to CPython's
> behavior, failing that, if there are good reasons, the next best
> solution is probably to come with something that we can integrate into
> the "real" pypy, rather than keep on your own.  If you don't, users
> complain here and you are just making us hate .pyc files even more.
> :-)
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From stefano at rivera.za.net  Tue Jul 31 17:21:32 2012
From: stefano at rivera.za.net (Stefano Rivera)
Date: Tue, 31 Jul 2012 17:21:32 +0200
Subject: [pypy-dev] __pycache__ folders
In-Reply-To: <5017F5CF.5080704@gmx.de>
References: <CAPYWazrrbFD_BRmy4owdyfXq3VZ0qc8yZYzRq0vzf-c91aDNVA@mail.gmail.com>
	<CAPYWazrrXW-hCGpvohHd8eA8tRAxpVH6TOJFVkSTHoj8cSSKeQ@mail.gmail.com>
	<CAMSv6X23NbYHfsQQat2i5-w9OZpRADFgSwCubv0AyhrYxZX8Tg@mail.gmail.com>
	<20120731094419.GT1912@bach.rivera.co.za>
	<CAMSv6X2ZZ3Hir4pmzcbWUv9Hjd_BkJ6ojebx3XdfXGevWYmDKg@mail.gmail.com>
	<20120731103556.GU1911@bach.rivera.co.za>
	<5017BC0C.3030602@stackless.com>
	<20120731113431.GV1911@bach.rivera.co.za>
	<CAMSv6X3Q3k9=TKE0vb7RakoSO7WvDP5F9wGtKPJ-pkfTnoQLug@mail.gmail.com>
	<5017F5CF.5080704@gmx.de>
Message-ID: <20120731152132.GW1912@bach.rivera.co.za>

Hi Ronny (2012.07.31_17:12:15_+0200)
> i would dare to say that Distributions that manage semi-compatible,
> but distro specific "forks" of open source projects
> instead of aiming for a distro compatible upstream
> create quite some grief for developers.

Weighed up against making grief for users.

> after all users don't tend to notice those incompatibilities,
> but it breaks the tooling of developers

We really try not to make changes that break tooling. And when we do, to
fix the tools, upstream.

> I'm at a point where i need to run self-compiled pythons,
> since the ones in Debian for example are broken
> for some of my use-cases by Debian specific patches/changes

And that was the point of the dist-packages change.

> what makes it worse is that the error reporting paths for the distro
> specific "forks" aren't made clear

Distros have bug trackers.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

From stevenjackson121 at gmail.com  Tue Jul 31 21:05:34 2012
From: stevenjackson121 at gmail.com (Steven Jackson)
Date: Tue, 31 Jul 2012 15:05:34 -0400
Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature
	Request)
In-Reply-To: <CAK5idxTReTPY_tkb6fHdz_UmeLS+7RZ-CTMYw7q_4Hrg5nh7tQ@mail.gmail.com>
References: <CAPTidvibJ=pBK83+13ZV=FyiVbABEBs+ixVOWF0qPhpecTK+6Q@mail.gmail.com>
	<CAJhcF=3SRExRzmLQwioGHJb3Y+aW=gdMZG6BP=otuXqThmJL=w@mail.gmail.com>
	<CAPTidvin2E4W5rRwXRtv4QF4Zu1Df9_4dN_UdJYd8=0B09jLKg@mail.gmail.com>
	<CAK5idxTReTPY_tkb6fHdz_UmeLS+7RZ-CTMYw7q_4Hrg5nh7tQ@mail.gmail.com>
Message-ID: <CAPTidvgbEk-TnpM=8DbBMiAzrKDbMZ=b0UFd95jQcbPmz_NLwg@mail.gmail.com>

Which source files/functions/line numbers should I take a look at, as
specifically as possible?

On Tue, Jul 31, 2012 at 9:55 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Tue, Jul 31, 2012 at 3:41 PM, Steven Jackson
> <stevenjackson121 at gmail.com> wrote:
> > I am definitely open to working on Numpy first, my only hesitation was
> that
> > it might be hard to get spun up, and scipy seemed like something I could
> > slice off and do (some of) by myself without accidentally conflicting
> with
> > ongoing development efforts.
> >
> > If, as your replies have indicated, Numpy isn't being very actively
> > developed, and it would be easy to get caught up and avoid conflicting
> with
> > other developers, then I'd be glad to help there instead of independently
> > trying to get scipy going.
>
> That sounds very good to me. The thing I started (and I'm not actively
> working on it for at least a week or two more) is to clean up stuff
> and try to reuse pure-python numpy part without changes
>
> >
> >
> >
> > On Tue, Jul 31, 2012 at 9:24 AM, Da?id <davidmenhur at gmail.com> wrote:
> >>
> >> I believe the reason is that they prefer to focus the development
> >> effort into the more "basic" Numpy and from that go on. Numpy is not
> >> fully implemented
> >>
> >> I am working (actually, right now I am only in the "thinking" state)
> >> on some modules of NumPy, maybe we could coordinate and give this a
> >> boost.
> >>
> >> On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson
> >> <stevenjackson121 at gmail.com> wrote:
> >> > I know there is no current plan to implement scipy in pypy, but
> >> > searching
> >> > the PyPy website, I was not able to find the reason.
> >> >
> >> > If it is not to be included as a feature due to lack of interest or
> >> > developer time, I am offering to begin rewriting scipy in pure python
> (I
> >> > have quite a bit of free time to do this).
> >> > If it has been discussed and actively excluded from pypy, I would like
> >> > to
> >> > know that before I waste too much time rewriting it.
> >> >
> >> > I have an active interest in being able to use some modules which
> depend
> >> > on
> >> > scipy while using the PyPy interpreter (especially pyBrain). I have
> seen
> >> > the
> >> > hack on the pypy blog:
> >> >
> >> >
> >> >
> http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html
> >> >
> >> > I could simply use the hack myself, but unless there's a reason not
> too,
> >> > I
> >> > think it'd be nice to have scipy available as a pure python module
> >> > (perhaps
> >> > called scipypy in the same manner as numpy).
> >> >
> >> > --
> >> > Steven Jackson
> >> >
> >> > _______________________________________________
> >> > pypy-dev mailing list
> >> > pypy-dev at python.org
> >> > http://mail.python.org/mailman/listinfo/pypy-dev
> >> >
> >
> >
> >
> >
> > --
> > Steven Jackson
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
>



-- 
Steven Jackson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120731/aed656c7/attachment-0001.html>