From fijall at gmail.com  Sun Oct  5 16:22:43 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 5 Oct 2008 07:22:43 -0700
Subject: [pypy-dev] sprint
Message-ID: <693bc9ab0810050722n62818c9eh3fe32edf6b7727c7@mail.gmail.com>

Hello from the third world.

I failed to embark the train today. It seems that these days you need
to buy ticket at least 3 days in advance in order to get to berlin. I
managed though to get some place on ukrainian train, which goes 3h
longer, but at least have some places. I hope to be in ddorf tomorrow
around lunch.

Cheers,
fijal


From ondrej at certik.cz  Thu Oct  9 02:19:35 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 9 Oct 2008 02:19:35 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
Message-ID: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>

Hi,

if you want to get many bugreports, here's how. :)

$ git clone git://git.sympy.org/sympy.git
$ cd sympy
$ python
>>> import sympy
>>> sympy.test("sympy/core")
[...]
============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
True
>>>


This works in python2.4, python2.5 (there are some failures in
python2.6) and it *doesn't* require py.test, just pure python.

Now try pypy:

$ ../pypy-dist/pypy-c    # compile pypy to C and then to executable
Python 2.4.1 (pypy 1.0.0 build 58853) on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``don't post links to patents
here''
>>>> import sympy
>>>> sympy.test("sympy/core")
[ dozens pages of errors ]


To ease debugging, use bin/test:

$ ../pypy-dist/pypy-c bin/test sympy/core/

and use it like py.test.

Basically, most of them reduce to this problem:

$ python
Python 2.4.5 (#2, Sep 29 2008, 15:31:42)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sympy import Function
>>> f = Function("f")
>>>

$ ../pypy-dist/pypy-c
Python 2.4.1 (pypy 1.0.0 build 58853) on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``things will be nice and stuff''
>>>> from sympy import Function
>>>> f = Function("f")
[boom ...]
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
43, in iter_copy
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
43, in iter_copy
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
42, in iter_copy
    if hasattr(i, "__iter__"):
RuntimeError: internal error: <RuntimeError object at 0xb03b60>
>>>>




Ondrej


From ondrej at certik.cz  Fri Oct 10 13:00:12 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 13:00:12 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
Message-ID: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>

On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I'm working on 2.5 compatibility branch right now. Due to incompatible
> changes, I suggest that we say we don't provide --pyversion option any
> more and simply compile 2.5 compatible interpreter. I don't really see
> benefits of providing 2.4 right now (after a brief discussion with
> Armin).
>
> If anyone objects, please do that now.

Just a curious question --- what kind of features are python2.4 only
that doesn't work in python2.5? I though that except hashing,
python2.5 is backwards compatible.

Ondrej


From ondrej at certik.cz  Fri Oct 10 13:12:19 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 13:12:19 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
Message-ID: <85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>

On Thu, Oct 9, 2008 at 2:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> if you want to get many bugreports, here's how. :)
>
> $ git clone git://git.sympy.org/sympy.git
> $ cd sympy
> $ python
>>>> import sympy
>>>> sympy.test("sympy/core")
> [...]
> ============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
> True
>>>>
>
>
> This works in python2.4, python2.5 (there are some failures in
> python2.6) and it *doesn't* require py.test, just pure python.

Holger asked me why we are not using py.test anymore and any
particular feedback. I am not subscribed to py-dev, so I just post it
here.

We still use py.test. But I also want to have something simple
(current implementation has 335 lines) that we can include with sympy,
because py.test is not so easy to install --- you need to install
setuptools, which in particular require python to be compiled with
zlib. So while py.test is fine let's say in Debian or Ubuntu, it was
quite a pain for me to install it 6x on our buildbots:

http://buildbot.sympy.org/waterfall

(E.g. I compiled python2.4, 2.5, 2.6 from python.org, then I realized
that setuptools cannot be installed, because python didn't compile
with the zlib module, because I didn't have the zlib-dev Debian
package installed, so I fixed that, recompiled python, installed
setuptools 6x and then installed py.test 6x).

And we are also porting sympy to jython and it's just easier to just
install sympy and do "sympy.test()" to test that all is ok.

We also thought we could just include py.test in sympy, but py.test
depends on the py lib and it is big. So I just rewrote the necessary
thing from scratch and it can do what I want.

Ondrej


From cfbolz at gmx.de  Fri Oct 10 13:23:21 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 10 Oct 2008 13:23:21 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
Message-ID: <48EF3B29.8040301@gmx.de>

Ondrej Certik wrote:
> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>> changes, I suggest that we say we don't provide --pyversion option any
>> more and simply compile 2.5 compatible interpreter. I don't really see
>> benefits of providing 2.4 right now (after a brief discussion with
>> Armin).
>>
>> If anyone objects, please do that now.
> 
> Just a curious question --- what kind of features are python2.4 only
> that doesn't work in python2.5? I though that except hashing,
> python2.5 is backwards compatible.

I don't think there are any (big ones, at least). But that's just a
further argument for removing the --pyversion option, isn't it?

Cheers,

Carl Friedrich


From fijall at gmail.com  Fri Oct 10 14:05:20 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 10 Oct 2008 14:05:20 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <48EF3B29.8040301@gmx.de>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
	<48EF3B29.8040301@gmx.de>
Message-ID: <693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>

On Fri, Oct 10, 2008 at 1:23 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Ondrej Certik wrote:
>> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>>> changes, I suggest that we say we don't provide --pyversion option any
>>> more and simply compile 2.5 compatible interpreter. I don't really see
>>> benefits of providing 2.4 right now (after a brief discussion with
>>> Armin).
>>>
>>> If anyone objects, please do that now.
>>
>> Just a curious question --- what kind of features are python2.4 only
>> that doesn't work in python2.5? I though that except hashing,
>> python2.5 is backwards compatible.
>
> I don't think there are any (big ones, at least). But that's just a
> further argument for removing the --pyversion option, isn't it?
>
> Cheers,
>
> Carl Friedrich

For example incompatible module APIs. They're kind of the same from
user level, but the way they interact inside (for example sre and
_sre) has changed.

Cheers,
fijal


From ondrej at certik.cz  Fri Oct 10 14:10:54 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 14:10:54 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
	<48EF3B29.8040301@gmx.de>
	<693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>
Message-ID: <85b5c3130810100510l737996cax3689d78f42eb4b1@mail.gmail.com>

On Fri, Oct 10, 2008 at 2:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 10, 2008 at 1:23 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> Ondrej Certik wrote:
>>> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>>>> changes, I suggest that we say we don't provide --pyversion option any
>>>> more and simply compile 2.5 compatible interpreter. I don't really see
>>>> benefits of providing 2.4 right now (after a brief discussion with
>>>> Armin).
>>>>
>>>> If anyone objects, please do that now.
>>>
>>> Just a curious question --- what kind of features are python2.4 only
>>> that doesn't work in python2.5? I though that except hashing,
>>> python2.5 is backwards compatible.
>>
>> I don't think there are any (big ones, at least). But that's just a
>> further argument for removing the --pyversion option, isn't it?
>>
>> Cheers,
>>
>> Carl Friedrich
>
> For example incompatible module APIs. They're kind of the same from
> user level, but the way they interact inside (for example sre and
> _sre) has changed.

I see. Thanks!

Ondrej


From holger at merlinux.eu  Fri Oct 10 15:04:47 2008
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Oct 2008 15:04:47 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
Message-ID: <20081010130447.GN6829@trillke.net>

Hi Ondrej, 

On Fri, Oct 10, 2008 at 13:12 +0200, Ondrej Certik wrote:
> On Thu, Oct 9, 2008 at 2:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> > Hi,
> >
> > if you want to get many bugreports, here's how. :)
> >
> > $ git clone git://git.sympy.org/sympy.git
> > $ cd sympy
> > $ python
> >>>> import sympy
> >>>> sympy.test("sympy/core")
> > [...]
> > ============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
> > True
> >>>>
> >
> >
> > This works in python2.4, python2.5 (there are some failures in
> > python2.6) and it *doesn't* require py.test, just pure python.
> 
> Holger asked me why we are not using py.test anymore and any
> particular feedback. I am not subscribed to py-dev, so I just post it
> here.

thanks - i am cross-posting this to py-dev. 
 
> We still use py.test. But I also want to have something simple
> (current implementation has 335 lines) that we can include with sympy,
> because py.test is not so easy to install --- you need to install
> setuptools, which in particular require python to be compiled with
> zlib. So while py.test is fine let's say in Debian or Ubuntu, it was
> quite a pain for me to install it 6x on our buildbots:
> 
> http://buildbot.sympy.org/waterfall

> (E.g. I compiled python2.4, 2.5, 2.6 from python.org, then I realized
> that setuptools cannot be installed, because python didn't compile
> with the zlib module, because I didn't have the zlib-dev Debian
> package installed, so I fixed that, recompiled python, installed
> setuptools 6x and then installed py.test 6x).

Hum, that's kind of bothersome, i agree.  I wonder if going
away from setuptools makes sense, shouldn't be hard. 

> And we are also porting sympy to jython and it's just easier to just
> install sympy and do "sympy.test()" to test that all is ok.

right, makes sense. 
 
> We also thought we could just include py.test in sympy, but py.test
> depends on the py lib and it is big. So I just rewrote the necessary
> thing from scratch and it can do what I want.

Raymond Hettinger at some point also wrote a pytest one-file
script with a subset of features.  Probably providing 
something like that officially makes sense.   Also 
i'd like to work a bit on minimizing py lib (i plan
to factor some parts out into plugins soon) - most of it
is actually used from py.test. 

best,

holger

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 


From ondrej at certik.cz  Fri Oct 10 15:24:34 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 15:24:34 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <20081010130447.GN6829@trillke.net>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
	<20081010130447.GN6829@trillke.net>
Message-ID: <85b5c3130810100624v59f9385eg282ea2864d624618@mail.gmail.com>

>> We also thought we could just include py.test in sympy, but py.test
>> depends on the py lib and it is big. So I just rewrote the necessary
>> thing from scratch and it can do what I want.
>
> Raymond Hettinger at some point also wrote a pytest one-file
> script with a subset of features.  Probably providing
> something like that officially makes sense.   Also
> i'd like to work a bit on minimizing py lib (i plan
> to factor some parts out into plugins soon) - most of it
> is actually used from py.test.


Thanks for the work on py.test. The user interface is still I think
the best from all the testing frameworks around. I think it's cool to
have something small & simple that projects can just copy verbatim,
and then full featured py.test, that one can use when it is installed.
In fact, that's how I use it.

Ondrej


From arigo at tunes.org  Sat Oct 11 17:14:57 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Oct 2008 17:14:57 +0200
Subject: [pypy-dev] switched dist -> trunk
Message-ID: <20081011151456.GA19291@code0.codespeak.net>

Hi all,

We finally created the path http://codespeak.net/svn/pypy/trunk, which
contains the in-development 2.5-compatible version of PyPy.  This closes
our many-year-old issue 30 :-)

The http://codespeak.net/svn/pypy/dist path is still available, and it's
fine to continue checking in there for some more days, but please switch
to using 'trunk' for development when convenient.

The plan is to merge the changes from 'dist' into 'trunk' in a few days,
after which we'll use 'dist' only as a kind of mirror of 'trunk'
containing read-only snapshots of stable revisions.


A bientot,

Armin


From arigo at tunes.org  Sat Oct 11 19:34:26 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Oct 2008 19:34:26 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
Message-ID: <20081011173426.GA1699@code0.codespeak.net>

Hi,

On Thu, Oct 09, 2008 at 02:19:35AM +0200, Ondrej Certik wrote:
>     if hasattr(i, "__iter__"):
> RuntimeError: internal error: <RuntimeError object at 0xb03b60>

This occurs because our 'str' type has an '__iter__' special method.  It
turns out that CPython's does not.  I suppose we could remove our
__iter__, but I would rather suggest that sympy's code is fragile there:
as soon as someone realizes that in CPython it is possible to speed up
iteration over strings by creating a stringiterator type, then CPython
will grow a 'str.__iter__' as well, and the same infinite recursion will
occur in sympy...

A cleaner way to say "is x iterable?" would be to try to call iter(x)
and see if it raises TypeError or not.


A bientot,

Armin


From ondrej at certik.cz  Sat Oct 11 19:50:25 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 11 Oct 2008 19:50:25 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <20081011173426.GA1699@code0.codespeak.net>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
Message-ID: <85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>

On Sat, Oct 11, 2008 at 7:34 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Thu, Oct 09, 2008 at 02:19:35AM +0200, Ondrej Certik wrote:
>>     if hasattr(i, "__iter__"):
>> RuntimeError: internal error: <RuntimeError object at 0xb03b60>
>
> This occurs because our 'str' type has an '__iter__' special method.  It
> turns out that CPython's does not.  I suppose we could remove our
> __iter__, but I would rather suggest that sympy's code is fragile there:
> as soon as someone realizes that in CPython it is possible to speed up
> iteration over strings by creating a stringiterator type, then CPython
> will grow a 'str.__iter__' as well, and the same infinite recursion will
> occur in sympy...
>
> A cleaner way to say "is x iterable?" would be to try to call iter(x)
> and see if it raises TypeError or not.

Indeed, thanks very much for the tip. We'll fix that, that's
definitely something that should be fixed in sympy, I created a new
issue for that:

http://code.google.com/p/sympy/issues/detail?id=1153

Ondrej


From ondrej at certik.cz  Sat Oct 11 23:42:47 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 11 Oct 2008 23:42:47 +0200
Subject: [pypy-dev] [sympy] Re: sympy tests: pypy vs cpython
	incompatibilities
In-Reply-To: <4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
	<85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>
	<4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
Message-ID: <85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>

On Sat, Oct 11, 2008 at 8:13 PM, didier deshommes <dfdeshom at gmail.com> wrote:
>
> On Sat, Oct 11, 2008 at 1:50 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>>> as soon as someone realizes that in CPython it is possible to speed up
>>> iteration over strings by creating a stringiterator type, then CPython
>>> will grow a 'str.__iter__' as well, and the same infinite recursion will
>>> occur in sympy...
>>>
>>> A cleaner way to say "is x iterable?" would be to try to call iter(x)
>>> and see if it raises TypeError or not.
>>
>> Indeed, thanks very much for the tip. We'll fix that, that's
>> definitely something that should be fixed in sympy, I created a new
>> issue for that:
>
> Hi,
> I've faced the same problem and checking if x has the __getitem__
> attribute has worked for me. The rationale being that any iterable has
> a way to access its individual items.

Thanks Didier for the tip. Both iter("p") and "p".__getitem__ behaves
the same in both python and pypy. But "p".__iter__ behaves differently
and our code assumed that "p" is not iterable, so I just implemented a
function:

http://code.google.com/p/sympy/issues/detail?id=1153#c1

+def is_iterable(x):
+    try:
+        iter(x)
+        if isinstance(x, str) and len(x) == 1:
+            return False
+        return True
+    except TypeError:
+        return False


Which fixes the problem above. Unfortunately, then there are other
unrelated problems.

Armin, is there some way to tell pypy-c to stop on ctrl-C? I am
getting hangups and the only way out is to kill the pypy-c program.
It'd be much more convenient if I could just do ctrl-C and it would
print the stacktrace.

I'll be continuing porting sympy to pypy some other time, so anyone
feel free to take over it. You just need to apply the patch in the
issue 1153 and then try to make all tests to pass.

Thanks,
Ondrej


From micahel at gmail.com  Sun Oct 12 12:13:53 2008
From: micahel at gmail.com (Michael Hudson)
Date: Sun, 12 Oct 2008 23:13:53 +1300
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
Message-ID: <aa0dfd860810120313u63119a34g7a4b50bdb7e0418f@mail.gmail.com>

2008/10/11 Ondrej Certik <ondrej at certik.cz>:
> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>> changes, I suggest that we say we don't provide --pyversion option any
>> more and simply compile 2.5 compatible interpreter. I don't really see
>> benefits of providing 2.4 right now (after a brief discussion with
>> Armin).
>>
>> If anyone objects, please do that now.
>
> Just a curious question --- what kind of features are python2.4 only
> that doesn't work in python2.5? I though that except hashing,
> python2.5 is backwards compatible.

New style exceptions break a few things.  A few deprecated things were
removed -- OverflowWarning, for example.

The idea that Python 3 was the first release of Python that isn't
backwards compatible is slightly misleading :)

Cheers,
mwh


From dw at botanicus.net  Tue Oct 14 18:39:56 2008
From: dw at botanicus.net (David Wilson)
Date: Tue, 14 Oct 2008 17:39:56 +0100
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
Message-ID: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>

Hi there,

I just read about this on the PyPy status blog and find myself
wondering about the strategies you considered before settling on the
Reflex approach.

I've had an extension to ctypes in mind for quite some time that
involves simply leaving the debugging segments present in a shared
library in order to be able to introspect them and provide type-safe
access to the code within.

The idea is simple: implement a DWARF/stabs/etc. interpreter that can
construct Python types and function proxies to represent what is
present in the shared library. Initially I wanted this for C, so that
a Python programmer needn't manually declare structs using quirky
ctypes class definitions, but I can't see why a (rather more complex)
implementation couldn't reconstruct C++ classes with their associated
class/member functions.

The obvious benefit is that a C/C++ library need not be modified at
all in order to benefit, other than not being stripped of debugging
information during its build. Linux distributions are slowly gaining
infrastructure that makes this almost zero cost (e.g. the Debian
python-dbg package).

I've had the DWARF specification sitting on my desktop for about 2
months now, waiting for me to validate my idea.

I'm wondering if you've considered this approach, and if so, why you
discarded it.

Thanks,


David.


From fijall at gmail.com  Wed Oct 15 23:53:53 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 15 Oct 2008 23:53:53 +0200
Subject: [pypy-dev] buildbot problem/misunderstanding.
Message-ID: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>

I've been starring at report (here:
http://codespeak.net:8099/builders/own-linux-x86-32/builds/16/steps/shell_2/logs/stdio)
and I don't understand why there is only one test in translator/c. Am
I missing something? (search for translator/c/test/test_support to see
what I mean). I'm confused.

Cheers,
fijal


From fijall at gmail.com  Wed Oct 15 23:54:48 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 15 Oct 2008 23:54:48 +0200
Subject: [pypy-dev] buildbot problem/misunderstanding.
In-Reply-To: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>
References: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>
Message-ID: <693bc9ab0810151454r17564304t756a52688854064e@mail.gmail.com>

Ah, sorry for spam, got it.

On Wed, Oct 15, 2008 at 11:53 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I've been starring at report (here:
> http://codespeak.net:8099/builders/own-linux-x86-32/builds/16/steps/shell_2/logs/stdio)
> and I don't understand why there is only one test in translator/c. Am
> I missing something? (search for translator/c/test/test_support to see
> what I mean). I'm confused.
>
> Cheers,
> fijal
>


From arigo at tunes.org  Thu Oct 16 12:53:37 2008
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Oct 2008 12:53:37 +0200
Subject: [pypy-dev] [sympy] Re: sympy tests: pypy vs cpython
	incompatibilities
In-Reply-To: <85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
	<85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>
	<4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
	<85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>
Message-ID: <20081016105337.GA9516@code0.codespeak.net>

Hi Ondrej,

On Sat, Oct 11, 2008 at 11:42:47PM +0200, Ondrej Certik wrote:
> +def is_iterable(x):
> +    try:
> +        iter(x)
> +        if isinstance(x, str) and len(x) == 1:
> +            return False
> +        return True
> +    except TypeError:
> +        return False

I don't see why there is a check for len(x); the same difference exists
for strings of any length, not just characters.

> Armin, is there some way to tell pypy-c to stop on ctrl-C? I am
> getting hangups and the only way out is to kill the pypy-c program.
> It'd be much more convenient if I could just do ctrl-C and it would
> print the stacktrace.

That's what it should do.  Do you have a way to reproduce the problem?
Just running the sympy tests with the above patch?


A bientot,

Armin


From irving at naml.us  Fri Oct 17 05:50:36 2008
From: irving at naml.us (Geoffrey Irving)
Date: Thu, 16 Oct 2008 20:50:36 -0700
Subject: [pypy-dev] C++ library bindings
Message-ID: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>

Hello,

I posted a response to your blog post on C++ library bindings, and
wanted to continue the discussion further via email if anyone's
interested.  I just signed up for the mailing list, so apologies if I
missed a lot of previous discussion.  I'll say up front that it's
unlikely that I'll be able to devote any actual coding effort to this,
so feel free to tell me to get lost if you have plenty of ideas and
not enough manpower. :)

I started out writing C++ bindings using Boost.Python, and was very
happy with it for a long time.  It's strongest point is the ability to
wrap libraries that were never designed with python in mind,
specifically code with poor and inflexible ownership semantics.
Internally, this means that C++ objects are exposed indirectly through
a holder object containing either an inline copy of the C++ object or
any type of pointer holding the object.  Every access to the object
has to go through runtime dispatch in order to work with any possible
holder type.  The holder also contains the logic for ownership and
finalization.  For example, Boost.Python can return a reference to a
field inside another object, in which case the holder will keep a
reference to the parent object to keep it alive as long as the field
reference lives.

The problem with this generality is that it produces a huge amount of
object code (wrapping a single function in Boost.Python can add 10k to
the object file), and adds a lot of runtime indirection.

Assuming that one is writing C++ bindings because of speed issues,
it'd be nice if this extra layer of memory indirection and runtime
dispatch was exposed to the (eventual) JIT.  In order to do that, pypy
would have to be capable of handling pointers to raw memory containing
non-python objects (is already true due to ctypes stuff?), with
separate information about type and ownership.  For example, if you
have bindings for a C++ vector class and a C++ array containing the
vectors, a "reference" to an individual vector in the array is really
three different pieces:

1. The actual pointer to the vector.
2. A type structure containing functions to be called with the pointer
(1) as an argument.
3. A list of references to other objects that need to stay alive while
this reference lives.

If pypy and the JIT ends up able to treat these pieces separately,
it'd be a significant performance win over libraries wrapped with
CPython.

The other main source of slowness and complexity in Boost.Python is
overloading support, but I think that part is fairly straightforward
to handle in the python level.  All Boost.Python does internally is
loop over the set of functions registered for a given name, and for
each one loop over the arguments calling into its converter registry
to see if the python object can be converted to the C++ type.

As I mentioned in the blog comment, a lot of these issues come up in
contexts outside C++, like numpy.  Internally numpy represents
operations like addition as a big list of optimized routines to call
depending on the stored data type.  Functions in these tables are
called on raw pointers to memory, which is fundamental since numpy
arrays can refer to memory inside objects from C++, Fortran, mmap,
etc.  It'd be really awesome if the type dispatch step could be
written in python but still call into optimized C code for the final
arithmetic.

The other major issue is safety: if a lot of overloading and dispatch
code is going to be written in python, it'd be nice to shield that
code from segfaults.  I think you can get a long way there just by
having a consistent scheme for boxing the three components above
(pointer, type, and reference info), a way to label C function
pointers with type information, a small RPython layer that did simple
type-checked calls (with no support for overloading or type
conversion).  I just wrote a C++ analogue to this last part as a
minimal replacement for Boost.Python, so I could try to formulate what
I mean in pseudocode if there's interest.  There'd be some amount of
duplicate type checking if higher level layers such as overload
resolution were written in application level python, but that
duplication should be amenable to elimination by the JIT.

That's enough for now.  I'll look forward to the discussion.  Most of
my uses of python revolve heavily around C++ bindings, so it's
exciting to see that you're starting to think about it even if it's a
long way off.

Geoffrey


From fijall at gmail.com  Sat Oct 18 18:07:34 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 18 Oct 2008 18:07:34 +0200
Subject: [pypy-dev] [pypy-svn] r59181 -
	pypy/branch/cbuild-refactor/pypy/translator/platform
In-Reply-To: <20081017131928.2B0D316A151@codespeak.net>
References: <20081017131928.2B0D316A151@codespeak.net>
Message-ID: <693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>

On Fri, Oct 17, 2008 at 3:19 PM,  <afa at codespeak.net> wrote:
> Author: afa
> Date: Fri Oct 17 15:19:27 2008
> New Revision: 59181
>
> Modified:
>   pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
> Log:
> Log message about updated environment
>
>
> Modified: pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
> ==============================================================================
> --- pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     (original)
> +++ pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     Fri Oct 17 15:19:27 2008
> @@ -38,6 +38,7 @@
>         key, value = line.split('=', 1)
>         if key.upper() in ['PATH', 'INCLUDE', 'LIB']:
>             os.environ[key] = value
> +    log.msg("Updated environment with %s" % (vcvars,))
>
>  try:
>     _install_msvc_env()
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>

Can you please make sure that os.environ is cleaned afterwards? thanks


From fijall at gmail.com  Sat Oct 18 18:42:02 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 18 Oct 2008 18:42:02 +0200
Subject: [pypy-dev] [pypy-svn] r59181 -
	pypy/branch/cbuild-refactor/pypy/translator/platform
In-Reply-To: <693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>
References: <20081017131928.2B0D316A151@codespeak.net>
	<693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>
Message-ID: <693bc9ab0810180942m34e3a6d9qfc3c557713ec9764@mail.gmail.com>

On Sat, Oct 18, 2008 at 6:07 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 17, 2008 at 3:19 PM,  <afa at codespeak.net> wrote:
>> Author: afa
>> Date: Fri Oct 17 15:19:27 2008
>> New Revision: 59181
>>
>> Modified:
>>   pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
>> Log:
>> Log message about updated environment
>>
>>
>> Modified: pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
>> ==============================================================================
>> --- pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     (original)
>> +++ pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     Fri Oct 17 15:19:27 2008
>> @@ -38,6 +38,7 @@
>>         key, value = line.split('=', 1)
>>         if key.upper() in ['PATH', 'INCLUDE', 'LIB']:
>>             os.environ[key] = value
>> +    log.msg("Updated environment with %s" % (vcvars,))
>>
>>  try:
>>     _install_msvc_env()
>> _______________________________________________
>> pypy-svn mailing list
>> pypy-svn at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-svn
>>
>
> Can you please make sure that os.environ is cleaned afterwards? thanks
>

Ah, xoraxax just pointed out that this is once-per-import, so it's
completely fine.


From jurgis.pralgauskis at gmail.com  Mon Oct 20 13:32:23 2008
From: jurgis.pralgauskis at gmail.com (Jurgis Pralgauskis)
Date: Mon, 20 Oct 2008 14:32:23 +0300
Subject: [pypy-dev] py2php
Message-ID: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>

Hello,

I'd like to make life easier for writing simple php scripts
(as I am teaching python and web technologies, but my students (and I)
complain, that python web hosting is still not widelly affordable).

I imagine, that php is mostly similar to javascript for what's in pypy now
first I need very basic things: variables, loops, some string_methods,
echo and read/write to file.

I looked around https://codespeak.net/viewvc/pypy/dist/pypy/translator/js/

as I see, things would be easy to change as in  jts.py  and sbuiltin.py,
but I dont' see where loop structure representation is generated...

maybe smb could construct a bare php translator structure,
which I could start filling up and experimenting
if this smb is busy right now, I could wait :)

Thanks in advance

ps.: another idea:
can pypy quite easily help to visualize the list (or other structure)
manipulation for teaching purposes
it would be kind of watchvariables functionality
but more the way of https://launchpad.net/algorithms-animator

-- 
Jurgis Pralgauskis
tel: 8-616 77613;
jabber: jurgis at akl.lt; skype: dz0rdzas;
Don't worry, be happy and make things better ;)


From arigo at tunes.org  Wed Oct 22 18:51:41 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 22 Oct 2008 18:51:41 +0200
Subject: [pypy-dev] switched dist -> trunk
In-Reply-To: <20081011151456.GA19291@code0.codespeak.net>
References: <20081011151456.GA19291@code0.codespeak.net>
Message-ID: <20081022165141.GA14520@code0.codespeak.net>

Hi all,

On Sat, Oct 11, 2008 at 05:14:57PM +0200, Armin Rigo wrote:
> The plan is to merge the changes from 'dist' into 'trunk' in a few days,
> after which we'll use 'dist' only as a kind of mirror of 'trunk'
> containing read-only snapshots of stable revisions.

Done.  There is now a pre-commit hook that rejects commits inside
pypy/dist.  Please continue development on pypy/trunk.

SVN reminder on how to switch your existing working copy:

    $ svn info
    URL: http://codespeak.net/svn/pypy/dist
    # ^^^ check this URL! if it's not exactly .../pypy/dist, your
    #     current working directory is wrong!

    $ svn switch http://codespeak.net/svn/pypy/trunk


A bientot,

Armin.


From fijall at gmail.com  Thu Oct 23 14:19:06 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:19:06 +0200
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
In-Reply-To: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
References: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
Message-ID: <693bc9ab0810230519o1f6880c0qcb893ae5d8437f06@mail.gmail.com>

On Tue, Oct 14, 2008 at 6:39 PM, David Wilson <dw at botanicus.net> wrote:
> Hi there,
>
> I just read about this on the PyPy status blog and find myself
> wondering about the strategies you considered before settling on the
> Reflex approach.

We're not settling on it. It's rather "first to try".

>
> I've had an extension to ctypes in mind for quite some time that
> involves simply leaving the debugging segments present in a shared
> library in order to be able to introspect them and provide type-safe
> access to the code within.
>
> The idea is simple: implement a DWARF/stabs/etc. interpreter that can
> construct Python types and function proxies to represent what is
> present in the shared library. Initially I wanted this for C, so that
> a Python programmer needn't manually declare structs using quirky
> ctypes class definitions, but I can't see why a (rather more complex)
> implementation couldn't reconstruct C++ classes with their associated
> class/member functions.

The problem with such approach is usually that this is too low level.
Gaining back lost abstraction level is usually hard and sometimes
impossible. I cannot say exactly that this approach is not worth
following - I'll just try to use reflex first as it seems people are
using it to do complex tasks. Of course feel free to prove that I'm
wrong, right now I'm more thinking about possibilities (hence
questions on blog to people to provide input), than settling on
anything.

>
> The obvious benefit is that a C/C++ library need not be modified at
> all in order to benefit, other than not being stripped of debugging
> information during its build. Linux distributions are slowly gaining
> infrastructure that makes this almost zero cost (e.g. the Debian
> python-dbg package).
>
> I've had the DWARF specification sitting on my desktop for about 2
> months now, waiting for me to validate my idea.
>
> I'm wondering if you've considered this approach, and if so, why you
> discarded it.

The answer is simple: we didn't consider it yet :-)


From fijall at gmail.com  Thu Oct 23 14:25:20 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:25:20 +0200
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
Message-ID: <693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>

Hey.

First sorry for late response, we're kind of busy doing other things
now (ie working on 2.5-compatible release). That doesn't mean we don't
appreciate input about our problems.

On Fri, Oct 17, 2008 at 5:50 AM, Geoffrey Irving <irving at naml.us> wrote:
> Hello,
>
> I posted a response to your blog post on C++ library bindings, and
> wanted to continue the discussion further via email if anyone's
> interested.  I just signed up for the mailing list, so apologies if I
> missed a lot of previous discussion.  I'll say up front that it's
> unlikely that I'll be able to devote any actual coding effort to this,
> so feel free to tell me to get lost if you have plenty of ideas and
> not enough manpower. :)

That's fine. We don't have enough manpower to work on this now, but
knowing what people do in this area is very valuable once we get to
it.

>
> I started out writing C++ bindings using Boost.Python, and was very
> happy with it for a long time.  It's strongest point is the ability to
> wrap libraries that were never designed with python in mind,
> specifically code with poor and inflexible ownership semantics.
> Internally, this means that C++ objects are exposed indirectly through
> a holder object containing either an inline copy of the C++ object or
> any type of pointer holding the object.  Every access to the object
> has to go through runtime dispatch in order to work with any possible
> holder type.  The holder also contains the logic for ownership and
> finalization.  For example, Boost.Python can return a reference to a
> field inside another object, in which case the holder will keep a
> reference to the parent object to keep it alive as long as the field
> reference lives.
>
> The problem with this generality is that it produces a huge amount of
> object code (wrapping a single function in Boost.Python can add 10k to
> the object file), and adds a lot of runtime indirection.
>
> Assuming that one is writing C++ bindings because of speed issues,
> it'd be nice if this extra layer of memory indirection and runtime
> dispatch was exposed to the (eventual) JIT.  In order to do that, pypy
> would have to be capable of handling pointers to raw memory containing
> non-python objects (is already true due to ctypes stuff?)

That's true. PyPy is able to handle pointers to any C place.

> .. with
> separate information about type and ownership.

We don't provide this, since C has no notion of that at all.

> For example, if you
> have bindings for a C++ vector class and a C++ array containing the
> vectors, a "reference" to an individual vector in the array is really
> three different pieces:
>
> 1. The actual pointer to the vector.
> 2. A type structure containing functions to be called with the pointer
> (1) as an argument.
> 3. A list of references to other objects that need to stay alive while
> this reference lives.
>
> If pypy and the JIT ends up able to treat these pieces separately,
> it'd be a significant performance win over libraries wrapped with
> CPython.
>
> The other main source of slowness and complexity in Boost.Python is
> overloading support, but I think that part is fairly straightforward
> to handle in the python level.  All Boost.Python does internally is
> loop over the set of functions registered for a given name, and for
> each one loop over the arguments calling into its converter registry
> to see if the python object can be converted to the C++ type.
>
> As I mentioned in the blog comment, a lot of these issues come up in
> contexts outside C++, like numpy.  Internally numpy represents
> operations like addition as a big list of optimized routines to call
> depending on the stored data type.  Functions in these tables are
> called on raw pointers to memory, which is fundamental since numpy
> arrays can refer to memory inside objects from C++, Fortran, mmap,
> etc.  It'd be really awesome if the type dispatch step could be
> written in python but still call into optimized C code for the final
> arithmetic.

That's the goal. Well, not exactly - point is that you write this code
in Python/RPython and JIT is able to generate efficient assembler out
of it. That's a very far-reaching goal though to have nice integration
between yet-non-existant JIT and yet-non-existant PyPy's numpy :-)

>
> The other major issue is safety: if a lot of overloading and dispatch
> code is going to be written in python, it'd be nice to shield that
> code from segfaults.  I think you can get a long way there just by
> having a consistent scheme for boxing the three components above
> (pointer, type, and reference info), a way to label C function
> pointers with type information, a small RPython layer that did simple
> type-checked calls (with no support for overloading or type
> conversion).  I just wrote a C++ analogue to this last part as a
> minimal replacement for Boost.Python, so I could try to formulate what
> I mean in pseudocode if there's interest.  There'd be some amount of
> duplicate type checking if higher level layers such as overload
> resolution were written in application level python, but that
> duplication should be amenable to elimination by the JIT.

I think for now we're happy with extra overhead. We would like to have
*any* working C++ bindings first and then eventually think about
speeding it up.

>
> That's enough for now.  I'll look forward to the discussion.  Most of
> my uses of python revolve heavily around C++ bindings, so it's
> exciting to see that you're starting to think about it even if it's a
> long way off.

Thank you :)

Cheers,
fijal


From fijall at gmail.com  Thu Oct 23 14:27:34 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:27:34 +0200
Subject: [pypy-dev] py2php
In-Reply-To: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>
References: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>
Message-ID: <693bc9ab0810230527m7b5f1418v6ee7c411aeecbf5@mail.gmail.com>

On Mon, Oct 20, 2008 at 1:32 PM, Jurgis Pralgauskis
<jurgis.pralgauskis at gmail.com> wrote:
> Hello,
>
> I'd like to make life easier for writing simple php scripts
> (as I am teaching python and web technologies, but my students (and I)
> complain, that python web hosting is still not widelly affordable).
>
> I imagine, that php is mostly similar to javascript for what's in pypy now
> first I need very basic things: variables, loops, some string_methods,
> echo and read/write to file.
>
> I looked around https://codespeak.net/viewvc/pypy/dist/pypy/translator/js/

This is going to be moved somewhere else as it's not working as we
would expect. Besides, it's translating rpython, not python into JS.

> ps.: another idea:
> can pypy quite easily help to visualize the list (or other structure)
> manipulation for teaching purposes
> it would be kind of watchvariables functionality
> but more the way of https://launchpad.net/algorithms-animator

I think it's easier to do such things in pypy, since you'll write code
that does in in python and not in C. But it's still work - it doesn't
come for free at all. If you want to do it, you can expect to get some
help.

Cheers,
fijal


From cfbolz at gmx.de  Thu Oct 23 14:36:00 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 23 Oct 2008 14:36:00 +0200
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
In-Reply-To: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
References: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
Message-ID: <49006FB0.9010004@gmx.de>

Hi David,

David Wilson wrote:
> I just read about this on the PyPy status blog and find myself
> wondering about the strategies you considered before settling on the
> Reflex approach.
> 
> I've had an extension to ctypes in mind for quite some time that
> involves simply leaving the debugging segments present in a shared
> library in order to be able to introspect them and provide type-safe
> access to the code within.
> 
> The idea is simple: implement a DWARF/stabs/etc. interpreter that can
> construct Python types and function proxies to represent what is
> present in the shared library. Initially I wanted this for C, so that
> a Python programmer needn't manually declare structs using quirky
> ctypes class definitions, but I can't see why a (rather more complex)
> implementation couldn't reconstruct C++ classes with their associated
> class/member functions.
> 
> The obvious benefit is that a C/C++ library need not be modified at
> all in order to benefit, other than not being stripped of debugging
> information during its build. Linux distributions are slowly gaining
> infrastructure that makes this almost zero cost (e.g. the Debian
> python-dbg package).
> 
> I've had the DWARF specification sitting on my desktop for about 2
> months now, waiting for me to validate my idea.
> 
> I'm wondering if you've considered this approach, and if so, why you
> discarded it.

One problem of this approach (in addition to what Maciek mentioned) is
that the debug information is quite large. It contains much more
information that what is needed for doing language bindings, e.g. also
detailed information about the inside of functions ("when the PC is
this, these variables are in those registers"). The Reflex information
should be rather small in size, compared to that.

Cheers,

Carl Friedrich


From irving at naml.us  Thu Oct 23 19:30:33 2008
From: irving at naml.us (Geoffrey Irving)
Date: Thu, 23 Oct 2008 10:30:33 -0700
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
	<693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
Message-ID: <7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>

On Thu, Oct 23, 2008 at 5:25 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hey.
>
> First sorry for late response, we're kind of busy doing other things
> now (ie working on 2.5-compatible release). That doesn't mean we don't
> appreciate input about our problems.
>
> On Fri, Oct 17, 2008 at 5:50 AM, Geoffrey Irving <irving at naml.us> wrote:
> <snip>
>
> That's true. PyPy is able to handle pointers to any C place.
>
>> .. with
>>
>> separate information about type and ownership.
>
> We don't provide this, since C has no notion of that at all.

At the lowest level the type is just a hashable identifier object, so
it can probably implemented at the RPython level.  E.g.,

# RPython type-safety layer

class CppObject:
    def __init__(ptr, type):
        self.ptr = ptr # pointer to the actual C++ instance
        self.type = type # represents the C++ type
        self.destructor = type.destructor # function pointer to destructor

    def __traverse__(self):
        ... traverse through list of contained python object pointers ...

    def __del__(self):
        CCall(self.destructor, self.ptr)

class CppFunc:
    def __init__(ptr, resulttype, argtypes):
        self.ptr = ptr
        self.resulttype = resulttype
        self.argtypes = argtypes

    def __call__(self, *args):
        if len(args) != len(self.argtypes):
            raise TypeError(...)
        argptrs = []
        for a,t in zip(args,self.argtypes):
            if not isinstance(a, CppObject) or a.type != t:
                raise TypeError(...)
            argptrs.append(a.ptr)
        resultptr = Alloc(self.resulttype.size)
        try:
            CppCall(self.ptr, resultptr, *argptrs) # assumes specific
calling convention
        except CppException, e: # CppCall would have to generate this
            Dealloc(resultptr)
            raise CppToPythonException(e)
        return CppObject(resultptr, self.resulttype)

If this layer is written in RPython, features like overload resolution
and C++ methods can be written in application-level python without
worring about safety.

>> <snip>
>>
>> As I mentioned in the blog comment, a lot of these issues come up in
>> contexts outside C++, like numpy.  Internally numpy represents
>> operations like addition as a big list of optimized routines to call
>> depending on the stored data type.  Functions in these tables are
>> called on raw pointers to memory, which is fundamental since numpy
>> arrays can refer to memory inside objects from C++, Fortran, mmap,
>> etc.  It'd be really awesome if the type dispatch step could be
>> written in python but still call into optimized C code for the final
>> arithmetic.
>
> That's the goal. Well, not exactly - point is that you write this code
> in Python/RPython and JIT is able to generate efficient assembler out
> of it. That's a very far-reaching goal though to have nice integration
> between yet-non-existant JIT and yet-non-existant PyPy's numpy :-)

Asking the JIT to generate to generate efficient code might be
sufficient in this case, but in terms of this discussion it just
removes numpy as a useful thought experiment towards C++ bindings. :)
Also for maximum speed I doubt the JIT will be able to match custom
code such as BLAS, given that C++ compilers usually don't get there
either.

>> The other major issue is safety: if a lot of overloading and dispatch
>> code is going to be written in python, it'd be nice to shield that
>> code from segfaults.  I think you can get a long way there just by
>> having a consistent scheme for boxing the three components above
>> (pointer, type, and reference info), a way to label C function
>> pointers with type information, a small RPython layer that did simple
>> type-checked calls (with no support for overloading or type
>> conversion).  I just wrote a C++ analogue to this last part as a
>> minimal replacement for Boost.Python, so I could try to formulate what
>> I mean in pseudocode if there's interest.  There'd be some amount of
>> duplicate type checking if higher level layers such as overload
>> resolution were written in application level python, but that
>> duplication should be amenable to elimination by the JIT.
>
> I think for now we're happy with extra overhead. We would like to have
> *any* working C++ bindings first and then eventually think about
> speeding it up.

Another advantage of splitting the code into an RPython type-safety
layer and application-level code is that the latter could be shared
with between pypy and cpython.  I haven't looked at reflex at all, but
in Boost.Python most of the complexity goes into code that could exist
at the application-level.

Geoffrey


From anto.cuni at gmail.com  Mon Oct 27 12:13:20 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 27 Oct 2008 12:13:20 +0100
Subject: [pypy-dev] bugday
Message-ID: <4905A250.5050403@gmail.com>

Hi pypy-dev,

we are trying to organize the first "PyPy Bugday", but it seems it's 
hard to find a proper date for it; so, we've set up a poll on doodle:

http://doodle.com/73tepx6ktg9cyd29

the poll is mainly targeted at core pypy developers, but of course 
everyone can vote.
Please spend two minutes for voting, so that we can pick the best date 
for it.

ciao & a bientot,
Anto & Armin


From arigo at tunes.org  Mon Oct 27 15:09:16 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 27 Oct 2008 15:09:16 +0100
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
	<693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
	<7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>
Message-ID: <20081027140916.GA30825@code0.codespeak.net>

Hi Geoffrey,

On Thu, Oct 23, 2008 at 10:30:33AM -0700, Geoffrey Irving wrote:
> If this layer is written in RPython, features like overload resolution
> and C++ methods can be written in application-level python without
> worring about safety.

I think that in this area, any worries of safety went out of the window
when CPython officially adopted ctypes.  So it's completely fine if the
application-level dispatching layer has access to unsafe features, as
long as user applications that only use the dispatching layer are
shielded from crashes.


A bientot,

Armin.


From brunogola at gmail.com  Mon Oct 27 15:21:25 2008
From: brunogola at gmail.com (Bruno Gola)
Date: Mon, 27 Oct 2008 12:21:25 -0200
Subject: [pypy-dev] bugday
In-Reply-To: <4905A250.5050403@gmail.com>
References: <4905A250.5050403@gmail.com>
Message-ID: <b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>

Hi,

On Mon, Oct 27, 2008 at 9:13 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> we are trying to organize the first "PyPy Bugday", but it seems it's
> hard to find a proper date for it; so, we've set up a poll on doodle:

Cool!

Is there any information available about how to help in a PyPy Bugday?

S?o Paulo Python User Group always participate of the Python Bugday
and I'm sure they will be happy to help PyPy.

Cheers,
--
Bruno Fialho Marques Gola <brunogola at gmail.com>
http://www.brunogola.com.br
Cel: (11) 9294-5883


From fijall at gmail.com  Mon Oct 27 15:54:46 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 27 Oct 2008 15:54:46 +0100
Subject: [pypy-dev] bugday
In-Reply-To: <b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>
References: <4905A250.5050403@gmail.com>
	<b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>
Message-ID: <693bc9ab0810270754t15e1a198u2a47aeac5051b37c@mail.gmail.com>

On Mon, Oct 27, 2008 at 3:21 PM, Bruno Gola <brunogola at gmail.com> wrote:
> Hi,
>
> On Mon, Oct 27, 2008 at 9:13 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> we are trying to organize the first "PyPy Bugday", but it seems it's
>> hard to find a proper date for it; so, we've set up a poll on doodle:
>
> Cool!
>
> Is there any information available about how to help in a PyPy Bugday?
>
> S?o Paulo Python User Group always participate of the Python Bugday
> and I'm sure they will be happy to help PyPy.
>
> Cheers,
> --
> Bruno Fialho Marques Gola <brunogola at gmail.com>
> http://www.brunogola.com.br
> Cel: (11) 9294-5883

Of course! By fixing bugs :-)

Details of failures are here:
https://codespeak.net/issue/pypy-dev/index and here:
http://codespeak.net:8099/summary

Cheers,
fijal

From p.giarrusso at gmail.com  Tue Oct 28 20:20:19 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 28 Oct 2008 20:20:19 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
Message-ID: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>

Hi, for a student project, we are evaluating the possibility to
experiment with some ideas on PyPy.

Even if it is a student project, we have high expectations (our aim is
to really improve PyPy's speed, if possible). But we are still trying
to choose which features to implement (we started with some nice
ideas, but part of them is already implemented).

We will work for something like 2 months on a separate branch - maybe
we can give readonly access to the source code repository, but write
access would be a problem for our exam, for obvious reasons.

Do you have any possibility to host a development branch for this project?

To let you know who we are, I'll present us:

- I, Paolo 'Blaisorblade' Giarrusso, am a past Linux kernel hacker, I
worked also with Java, C++, and a bit of Python, and I am currently a
graduate student at Aarhus University, Denmark;
- Sigurd is a PhD candidate at Aarhus University, Denmark, currently
working, among other things, on a cryptography research project in
Python (viff.dk).

- our professor is Lars Bak, the lead architect of the Google V8
Javascript engine, on which we implemented various optimizations in
the previous months.

We are obviously open to suggestions, and we have been looking at the
status blog and at various blogs.

It seems that there is still space for improvement in the space of
garbage collectors, as mentioned here:
- http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt
working on that could be interesting.

The main idea we wanted to apply to Python, based on a suggestion from
Lars, is the main peculiar V8 idea, the one of hidden classes with
transitions. That allows avoiding dictionary lookups for property
accesses on objects, using instead a Java-like data representation in
memory, without any visible effects on the Python semantics.

While revising this email, we saw that this is already partially
implemented: for what the blog says, the memory savings where done but
not the memory speedups. Also, the blog does not mention whether and
how class transitions are used. Some of the initial ideas below may
already have been implemented, so point out what is there and what can
be interesting.
I've also read the discussion about the JIT, which will be missing in
the next release.

Obviously, this would require partial reworking the core object model of PyPy.

Still, we think that specialization will especially benefit from this,
even because this allows to specialize better also on class types, not
only on primitive types. It is not clear to me how do you handle
changes (like addition of a property) to a single object - is it
possible to specialize the code on the (anonymous) type of this new
object? Can you give me pointers inside the documentation, if this is
already explained? I've read the EU reports about this and partial
compilation, but it is still unclear to us how things work.

Btw, the first thing to do is studying the object model and reflection
capability of Python. Since Lars said that the target of a VM designer
is to allow programmers work in the most pleasant way and use advanced
features without incurring any cost, our development plan should try
to optimize also reflective features as much as possible.

Just to make an example, in Python you can register an handler for
unknown method calls on an object (I don't remember how, but I'm
pretty sure something like this exists). This idea means that our
design should be able to do inline caching by storing a call to this
handler (maybe we'll defer the implementation, but the initial design
should allow doing that IMHO).

Any comments?
--
Paolo Giarrusso


From arigo at tunes.org  Wed Oct 29 11:29:29 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 29 Oct 2008 11:29:29 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
	<a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
Message-ID: <20081029102929.GA8041@code0.codespeak.net>

Hi Paolo,

Thanks for the introduction e-mail.  I'd like to answer it more
directly, but maybe I should first give a quick overview of what I think
are possible misunderstandings.

The main issue is that unlike Google V8, in PyPy there is no machine
code generation at all (ignoring the JIT compiler generator for the
first part of this e-mail).  That means that the notion of inline cache
simply doesn't make sense.  Let me explain this in more details.

What we implemented long ago in PyPy is completely similar to the "maps"
of the Self language.  It is a dictionary implementation that stores the
keys in a separate, shared "map" object, and that stores the values in a
flat list.  In PyPy this thing still presents the interface of a
full-featured dictionary, so it supports adding and removing arbitrary
elements, which is done by changing the dictionary's current map.

This saves memory, as you know.  But I don't see how this can provide
much speed-up in a pure interpreter (as opposed to in generated machine
code with inline caches).  It seems to me that whatever you do, you need
to perform a lookup *somewhere* in order to implement the "getattr"
operation.  The lookup can be done directly in the object's dictionary,
or it can be done in the shadow "map" object, but the point is that in
both cases it needs to be done at runtime for every attribute access.

There are probably advanced ways to cache recent lookups, which we
didn't try so far, like attaching to each GETATTR opcode in the Python
bytecode a tiny structure that caches the result of the most recent
lookup performed at that source code position.  I don't know how much
speed-ups this would give; in fact it is rather unclear to me why people
think that dictionary lookups are seriously bad.  Of course they are
more expensive than just reading an item out of a list, but I don't
think it's worse than roughly 3 times slower; this number looks big in
the context of machine code, but insignificant in the context of a
complicated bytecode interpreter.  I'm ready to be proven wrong, of
course.

However, this is only half of the story.  Another very important point
is that the mid-term goal for PyPy is to have again a JIT compiler
generator (it did so in the 1.0 release, but it was far from providing
consistent speed-ups, which is the reason that we are re-experimenting
at the moment).  In the context of our JIT, the inline cache technique
you describe -- as well as the polymorphic inline caches of Java VMs --
are special cases of a generic construct called "promotion" in [1] or
"unlifting" in Psyco [2].  Using this construct, the JIT compiler
generator takes as input our "dictionary-with-maps"-enabled interpreter
and produces as output a JIT compiler that itself emits inline caches
similar to Google V8's.  Another way to put it is that in PyPy the real
purpose of our "maps", besides saving memory, is to be
JIT-generator-friendly, enabling good speed-ups in the machine code
generated by the JIT (and not in the pure interpreter).  Thus, what we
implemented in the interpreter is enough already; the main missing piece
is a (completely generic) JIT generator producing a consistent JIT.
Note that this is not just an abstract wish: such inline caches is what
Psyco emits already; and the automatically generated JIT of PyPy 1.0
definitely did that too.

To repeat a point that we tried to make a lot of times but apparently
keep failing to: in the context of the PyPy JIT there is no point in
thinking deeply about the various semantics of Python specifically, like
reflection, operator overloading, the handlers for sends to unsupported
methods (that's __getattr__ in Python), and so on.  These semantics are
encoded in the interpreter already, where the JIT compiler generator can
see them and use them to produce a JIT that emits reasonably efficient
machine code for a given user program.  The relevant question in this
context is not how to encode some complicated Python semantics in the
machine code; the relevant question is to identify the cases where the
generated JIT emits definitely inefficient code, and how to tweak the
interpreter in order to be more JIT-friendly (e.g. by adding maps into
the interpreter).

In an attempt to make it clearer that thinking about advanced Python
semantics is not useful in this context, the "Python" in the previous
paragraph can be replaced by any language, preferably dynamic, and we
still get a JIT for it (e.g. we did it for Prolog, and we have partial
or in-progress interpreters for JavaScript and Smalltalk and other
languages).

Does the above make any sense?  There are three relatively distinct
areas to work on -- tweaking the interpreter to maximize its speed,
tweaking it to maximize its JIT-friendlyness (better done after the JIT
generator is ready), or helping develop the JIT generator itself; I hope
the above helped clarify a bit this aspect of PyPy.  In any case, we
would definitely be interested in working with you in these areas, or in
the other areas that you mentioned in your e-mail (GC...).  Feel free to
discuss matters further, here or on irc (#pypy on irc.freenode.net).


    [1] http://codespeak.net/pypy/dist/pypy/doc/jit/
    [2] http://psyco.sourceforge.net/doc.html [PEPM'04 paper]


A bientot,

Armin.


From cfbolz at gmx.de  Wed Oct 29 15:40:15 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 29 Oct 2008 15:40:15 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
	<a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
Message-ID: <490875CF.6010303@gmx.de>

Paolo Giarrusso wrote:
> Hi, for a student project, we are evaluating the possibility to
> experiment with some ideas on PyPy.
> 
> Even if it is a student project, we have high expectations (our aim is
> to really improve PyPy's speed, if possible). But we are still trying
> to choose which features to implement (we started with some nice
> ideas, but part of them is already implemented).
> 
> We will work for something like 2 months on a separate branch - maybe
> we can give readonly access to the source code repository, but write
> access would be a problem for our exam, for obvious reasons.
> 
> Do you have any possibility to host a development branch for this project?
> 
> To let you know who we are, I'll present us:
> 
> - I, Paolo 'Blaisorblade' Giarrusso, am a past Linux kernel hacker, I
> worked also with Java, C++, and a bit of Python, and I am currently a
> graduate student at Aarhus University, Denmark;
> - Sigurd is a PhD candidate at Aarhus University, Denmark, currently
> working, among other things, on a cryptography research project in
> Python (viff.dk).
> 
> - our professor is Lars Bak, the lead architect of the Google V8
> Javascript engine, on which we implemented various optimizations in
> the previous months.
> 
> We are obviously open to suggestions, and we have been looking at the
> status blog and at various blogs.
> 
> It seems that there is still space for improvement in the space of
> garbage collectors, as mentioned here:
> - http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt
> working on that could be interesting.
> 

Just to go a bit in a different direction than what Armin already wrote:
Yes, we would be very interested in having somebody work on our GCs.
They currently work, but they are not very advanced in many respects. So
if somebody is interested in having a go at them, that would probably be
quite useful for PyPy. I can't really say how interesting it would be
research-wise, though.

Cheers,

Carl Friedrich


From andrewfr_ice at yahoo.com  Fri Oct 31 20:45:00 2008
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Fri, 31 Oct 2008 12:45:00 -0700 (PDT)
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
Message-ID: <361057.6962.qm@web34203.mail.mud.yahoo.com>

Hello Colleagues:

I am not sure if this is the right place to ask....

I am trying to get RPython to work. I get the following problem:

[translation:ERROR]     from compiler import parse, ast, pycodegen
[translation:ERROR]  ImportError: cannot import name parse

I also installed Pylint including the Logilab-astng package. I get the error:
 
no module called common.compat.

I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.

Cheers,
Andrew








      


From fijall at gmail.com  Fri Oct 31 20:51:15 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 31 Oct 2008 20:51:15 +0100
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <361057.6962.qm@web34203.mail.mud.yahoo.com>
References: <361057.6962.qm@web34203.mail.mud.yahoo.com>
Message-ID: <693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>

On Fri, Oct 31, 2008 at 8:45 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:
> Hello Colleagues:
>
> I am not sure if this is the right place to ask....
>
> I am trying to get RPython to work. I get the following problem:
>
> [translation:ERROR]     from compiler import parse, ast, pycodegen
> [translation:ERROR]  ImportError: cannot import name parse

You cannot use a compiler package when writing rpython code. I'm not
sure what you're trying to do, but if your goal is to translate
existing python program using rpython toolchain, this will not work
(almost certainly).

>
> I also installed Pylint including the Logilab-astng package. I get the error:
>
> no module called common.compat.
>
> I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.

I think it's not the right place to ask (at least i have no clue).


From adimasci at gmail.com  Fri Oct 31 22:23:46 2008
From: adimasci at gmail.com (Adrien Di Mascio)
Date: Fri, 31 Oct 2008 22:23:46 +0100
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>
References: <361057.6962.qm@web34203.mail.mud.yahoo.com>
	<693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>
Message-ID: <437de8500810311423v19f49e8fvc47428711ac58da2@mail.gmail.com>

Hi there,

On 10/31/08, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 31, 2008 at 8:45 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:
>  > I also installed Pylint including the Logilab-astng package. I get the error:
>  >
>  > no module called common.compat.
>  >
>  > I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.
>
>
> I think it's not the right place to ask (at least i have no clue).

This is indeed not the right place to ask :-). Try
python-projects at logilab.org instead for any pylint related question.
Juste to make a quick answer, your problem is probably just
that you didn't install the logilab.common package
(http://www.logilab.org/project/logilab-common)

Cheers,
Adrien.