From arigo at tunes.org  Fri Dec  1 15:52:37 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 1 Dec 2006 15:52:37 +0100
Subject: [pypy-dev] A new idea?
In-Reply-To: <9c0bb8a00611291503i59b2e570o2d8d828d12d31e05@mail.gmail.com>
References: <9c0bb8a00611050946o713a970di4ef8b1165d78255d@mail.gmail.com>
	<20061129213747.GA26408@code0.codespeak.net>
	<9c0bb8a00611291503i59b2e570o2d8d828d12d31e05@mail.gmail.com>
Message-ID: <20061201145237.GA7669@code0.codespeak.net>

Hi Paul,

On Wed, Nov 29, 2006 at 06:03:24PM -0500, Paul deGrandis wrote:
> I'm interested in general interesting applications of emerging technology
> like Pypy.
> Pypy has so many amazing features and aspects that there are a pool of
> interesting applications to create.  Alternatively, there are also boring
> applications
> that when done with PyPy, offer an elegant solution.

RuntimeError: infinite recursion :-)

Sorry for the bad joke - I asked if you had some examples in mind about
applications that would benefit from PyPy's features, and the answer is
quite self-recursive.  Thanks for the background, though :-)

Do you have anything more precise in mind about which PyPy features you'd
like to see used in which applications?


A bientot,

Armin


From paul.degrandis at gmail.com  Fri Dec  1 21:17:00 2006
From: paul.degrandis at gmail.com (Paul deGrandis)
Date: Fri, 1 Dec 2006 15:17:00 -0500
Subject: [pypy-dev] A new idea?
In-Reply-To: <20061201145237.GA7669@code0.codespeak.net>
References: <9c0bb8a00611050946o713a970di4ef8b1165d78255d@mail.gmail.com>
	<20061129213747.GA26408@code0.codespeak.net>
	<9c0bb8a00611291503i59b2e570o2d8d828d12d31e05@mail.gmail.com>
	<20061201145237.GA7669@code0.codespeak.net>
Message-ID: <9c0bb8a00612011217o2d4aec8bs76f887a04320670e@mail.gmail.com>

Hi Armin,

On 12/1/06, Armin Rigo <arigo at tunes.org> wrote:
>
>
> RuntimeError: infinite recursion :-)



haha.  I'm so sorry.
I really wanted to leave it up to the core PyPy developer's for their input.
I also wanted to generate "tutorials" for the applications I developed.
This way we could build up a wealth of documentation that showed
practical examples of how to apply the techonology in PyPy.

I think compiler, JIT, proxies, lazily computed objects, stackless features
all have interesting applications.

I thought the original idea submitted was a good one and will work towards
that unless someone has a better idea.

Also, I see you're involved in TUNES.  I love that project, a lot.  I also
really liked parts of the Slate language.

Paul

Sorry for the bad joke - I asked if you had some examples in mind about
> applications that would benefit from PyPy's features, and the answer is
> quite self-recursive.  Thanks for the background, though :-)
>
> Do you have anything more precise in mind about which PyPy features you'd
> like to see used in which applications?
>
>
> A bientot,
>
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20061201/3d907496/attachment.html>

From arigo at tunes.org  Mon Dec  4 12:05:08 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 4 Dec 2006 12:05:08 +0100
Subject: [pypy-dev] test_logicobjspace
Message-ID: <20061204110507.GA30618@code0.codespeak.net>

Hi,

There are these 3 failures in test_logicobjspace that have been around
for, possibly, a few months now...  Anybody feeling like something needs
to be done about it?

    AppTest_LogicFutures.test_lazy_producer_consummer
    AppTest_LogicFutures.test_stacklet
    AppTest_LogicFutures.test_wait_two 


Thanks!

Armin


From aurelien.campeas at logilab.fr  Mon Dec  4 12:48:12 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 4 Dec 2006 12:48:12 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <20061204110507.GA30618@code0.codespeak.net>
References: <20061204110507.GA30618@code0.codespeak.net>
Message-ID: <20061204114812.GB7093@crater.logilab.fr>

On Mon, Dec 04, 2006 at 12:05:08PM +0100, Armin Rigo wrote:
> Hi,
> 
> There are these 3 failures in test_logicobjspace that have been around
> for, possibly, a few months now...  Anybody feeling like something needs
> to be done about it?
> 
>     AppTest_LogicFutures.test_lazy_producer_consummer
>     AppTest_LogicFutures.test_stacklet
>     AppTest_LogicFutures.test_wait_two 
> 
> 
> Thanks!
> 
> Armin

Last time I had a look, they worked perfectly fine. I believe Anders
also checked positively that they worked on his machine. We both tried
on a fresh check-out. They pass. Locally.

So something might be happening on snake. I don't know. This is
clearly out of scope for me right now. Feel free to do something about
it. Well, I don't think simply disabling them would be satisfying.

Aur?lien.



From cfbolz at gmx.de  Mon Dec  4 12:52:52 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 04 Dec 2006 12:52:52 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <20061204114812.GB7093@crater.logilab.fr>
References: <20061204110507.GA30618@code0.codespeak.net>
	<20061204114812.GB7093@crater.logilab.fr>
Message-ID: <45740C14.2050606@gmx.de>

Aur?lien Camp?as wrote:
> On Mon, Dec 04, 2006 at 12:05:08PM +0100, Armin Rigo wrote:
>> Hi,
>>
>> There are these 3 failures in test_logicobjspace that have been around
>> for, possibly, a few months now...  Anybody feeling like something needs
>> to be done about it?
>>
>>     AppTest_LogicFutures.test_lazy_producer_consummer
>>     AppTest_LogicFutures.test_stacklet
>>     AppTest_LogicFutures.test_wait_two
>>
>>
>> Thanks!
>>
>> Armin
>
> Last time I had a look, they worked perfectly fine. I believe Anders
> also checked positively that they worked on his machine. We both tried
> on a fresh check-out. They pass. Locally.
>
> So something might be happening on snake. I don't know. This is
> clearly out of scope for me right now. Feel free to do something about
> it. Well, I don't think simply disabling them would be satisfying.

They fail quite consistently for me too, on all the various machines I
am developing on. I would actually prefer it if they are fixed, since
you never know if something you changed caused the problem. If nobody
can work on this currently, they should be skipped and fixed later.

Cheers,

Carl Friedrich


From aurelien.campeas at logilab.fr  Mon Dec  4 13:00:37 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 4 Dec 2006 13:00:37 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <45740C14.2050606@gmx.de>
References: <20061204110507.GA30618@code0.codespeak.net>
	<20061204114812.GB7093@crater.logilab.fr> <45740C14.2050606@gmx.de>
Message-ID: <20061204120034.GC7093@crater.logilab.fr>

On Mon, Dec 04, 2006 at 12:52:52PM +0100, Carl Friedrich Bolz wrote:
> Aur?lien Camp?as wrote:
> > On Mon, Dec 04, 2006 at 12:05:08PM +0100, Armin Rigo wrote:
> >> Hi,
> >>
> >> There are these 3 failures in test_logicobjspace that have been around
> >> for, possibly, a few months now...  Anybody feeling like something needs
> >> to be done about it?
> >>
> >>     AppTest_LogicFutures.test_lazy_producer_consummer
> >>     AppTest_LogicFutures.test_stacklet
> >>     AppTest_LogicFutures.test_wait_two
> >>
> >>
> >> Thanks!
> >>
> >> Armin
> >
> > Last time I had a look, they worked perfectly fine. I believe Anders
> > also checked positively that they worked on his machine. We both tried
> > on a fresh check-out. They pass. Locally.
> >
> > So something might be happening on snake. I don't know. This is
> > clearly out of scope for me right now. Feel free to do something about
> > it. Well, I don't think simply disabling them would be satisfying.
> 
> They fail quite consistently for me too, on all the various machines I
> am developing on. I would actually prefer it if they are fixed, since
> you never know if something you changed caused the problem. If
> nobody

I tested them on a freshly checked-out pypy. 

> can work on this currently, they should be skipped and fixed later.
> 

How do you invoke the tests ?

I do, for instance (working dir being objspace/test) :

~/devel/pypy-dist/py/bin/py.test test_logicobjspace.py -s -k
test_stacklet

Does that trigger a failure on the machines you develop on ?

> Cheers,
> 
> Carl Friedrich
> 


From arigo at tunes.org  Mon Dec  4 13:02:54 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 4 Dec 2006 13:02:54 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <20061204114812.GB7093@crater.logilab.fr>
References: <20061204110507.GA30618@code0.codespeak.net>
	<20061204114812.GB7093@crater.logilab.fr>
Message-ID: <20061204120254.GA4310@code0.codespeak.net>

Hi Aurelien,

On Mon, Dec 04, 2006 at 12:48:12PM +0100, Aur?lien Camp?as wrote:
> So something might be happening on snake. I don't know.

I've seen the same 3 failures on various machines, though admittedly not
on tuatara, our PowerPC server.  It fails on pypy2, for example, so you
can investigate over there if needed (ssh codespeak.net; ssh pypy2).


A bientot,

Armin


From aurelien.campeas at logilab.fr  Mon Dec  4 13:08:36 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 4 Dec 2006 13:08:36 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <20061204120254.GA4310@code0.codespeak.net>
References: <20061204110507.GA30618@code0.codespeak.net>
	<20061204114812.GB7093@crater.logilab.fr>
	<20061204120254.GA4310@code0.codespeak.net>
Message-ID: <20061204120836.GD7093@crater.logilab.fr>

On Mon, Dec 04, 2006 at 01:02:54PM +0100, Armin Rigo wrote:
> Hi Aurelien,
> 
> On Mon, Dec 04, 2006 at 12:48:12PM +0100, Aur?lien Camp?as wrote:
> > So something might be happening on snake. I don't know.
> 
> I've seen the same 3 failures on various machines, though admittedly not
> on tuatara, our PowerPC server.  It fails on pypy2, for example, so you
> can investigate over there if needed (ssh codespeak.net; ssh pypy2).

they fail indeed

i'll have a look after lunch

> 
> 
> A bientot,
> 
> Armin
> 


From scott+pypy-dev at scottdial.com  Mon Dec  4 15:01:56 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Mon, 04 Dec 2006 09:01:56 -0500
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <20061204120254.GA4310@code0.codespeak.net>
References: <20061204110507.GA30618@code0.codespeak.net>	<20061204114812.GB7093@crater.logilab.fr>
	<20061204120254.GA4310@code0.codespeak.net>
Message-ID: <45742A54.7020001@scottdial.com>

Armin Rigo wrote:
> On Mon, Dec 04, 2006 at 12:48:12PM +0100, Aur?lien Camp?as wrote:
>> So something might be happening on snake. I don't know.
> 
> I've seen the same 3 failures on various machines, though admittedly not
> on tuatara, our PowerPC server.  It fails on pypy2, for example, so you
> can investigate over there if needed (ssh codespeak.net; ssh pypy2).
> 

Also for consideration is that the test has consistently passed on my 
win32 tester.

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu


From aurelien.campeas at logilab.fr  Mon Dec  4 15:05:30 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 4 Dec 2006 15:05:30 +0100
Subject: [pypy-dev] test_logicobjspace
In-Reply-To: <45742A54.7020001@scottdial.com>
References: <20061204110507.GA30618@code0.codespeak.net>
	<20061204114812.GB7093@crater.logilab.fr>
	<20061204120254.GA4310@code0.codespeak.net>
	<45742A54.7020001@scottdial.com>
Message-ID: <20061204140530.GE7093@crater.logilab.fr>

On Mon, Dec 04, 2006 at 09:01:56AM -0500, Scott Dial wrote:
> Armin Rigo wrote:
> >On Mon, Dec 04, 2006 at 12:48:12PM +0100, Aur?lien Camp?as wrote:
> >>So something might be happening on snake. I don't know.
> >
> >I've seen the same 3 failures on various machines, though admittedly not
> >on tuatara, our PowerPC server.  It fails on pypy2, for example, so you
> >can investigate over there if needed (ssh codespeak.net; ssh pypy2).
> >
> 
> Also for consideration is that the test has consistently passed on my 
> win32 tester.

It was a dict ordering issue: the test depended on
somedict.values()[0] always giving the right thing ...

> 
> -Scott
> 
> -- 
> Scott Dial
> scott at scottdial.com
> scodial at cs.indiana.edu
> 


From arigo at tunes.org  Tue Dec  5 18:10:05 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 5 Dec 2006 18:10:05 +0100
Subject: [pypy-dev] Leysin sprint: 8-14th January 2007
Message-ID: <20061205171003.GA10831@code0.codespeak.net>

=====================================================================
         PyPy Leysin Winter Sports Sprint (8-14th January 2007)
=====================================================================
..                             image:: http://www.ermina.ch/002.JPG

The next PyPy sprint will be in Leysin, Switzerland, for the fourth
time.  This sprint will be the final public sprint of our EU-funded
period, and a kick-off for the final work on the upcoming PyPy 1.0
release (scheduled for mid-February).

The sprint is the last chance for students looking for a "summer" job
with PyPy this winter! If you have a proposal and would like to 
work with us in the mountains please send it in before 15th December 
to "pypy-tb at codespeak.net" and cross-read this page: 

   http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html

------------------------------
Goals and topics of the sprint
------------------------------

* Like previous winter, the main side goal is to have fun in winter
  sports :-) We can take a couple of days off for ski; at this time of
  year, ski days end before 4pm, which still leaves plenty of time
  to recover (er, I mean hack).

* Progress on the JIT compiler, which we are just starting to scale to
  the whole of PyPy. `[1]`_

* Polish the code and documentation of the py lib, and eventually
  release it.

* Work on prototypes that use the new features that PyPy enables:
  distribution `[2]`_ (based on transparent proxying `[3]`_),
  security `[4]`_, persistence `[5]`_, etc. `[6]`_.

.. _[1]: http://codespeak.net/pypy/dist/pypy/doc/jit.html
.. _[2]: http://codespeak.net/svn/pypy/dist/pypy/lib/distributed/
.. _[3]: http://codespeak.net/pypy/dist/pypy/doc/proxy.html
.. _[4]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#security
.. _[5]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#persistence
.. _[6]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html

-----------------------
Location & Accomodation
-----------------------

Leysin, Switzerland, "same place as before".  Let me refresh your
memory: both the sprint venue and the lodging will be in a very spacious
pair of chalets built specifically for bed & breakfast:
http://www.ermina.ch/.  The place has a baseline ADSL Internet connexion
(600Kb) with wireless installed.  You can of course arrange your own
lodging anywhere (so long as you are in Leysin, you cannot be more than a
15 minute walk away from the sprint venue), but I definitely recommend
lodging there too -- you won't find a better view anywhere else (though you
probably won't get much worse ones easily, either :-)

I made pre-reservations in the Chalet, so please *confirm* quickly that
you are coming so that we can adjust the reservations as appropriate.
The rate so far has been 60 CHF a night all included in 2-person rooms,
with breakfast.  There are larger rooms too (less expensive, possibly
more available too) and the possibility to get a single room if you
really want to.

Please register by svn:

  http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2007/people.txt

or on the pypy-sprint mailing list if you do not yet have check-in rights:

  http://codespeak.net/mailman/listinfo/pypy-sprint

You need a Swiss-to-(insert country here) power adapter.  There will be
some Swiss-to-EU adapters around - bring a EU-format power strip if you
have one.

-----------
Exact times
-----------

Officially, 8th-14th January 2007.  Both dates are flexible, you can
arrive or leave earlier or later, though it is recommended to arrive on
the 7th (if many people arrive on the 6th we need to check for
accomodation availability first).  We will give introductions and
tutorials on the 8th, in the morning if everybody is there, or in the
afternoon otherwise.


From stephan.diehl at gmx.net  Tue Dec  5 19:12:34 2006
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Tue, 05 Dec 2006 19:12:34 +0100
Subject: [pypy-dev] coroutine interface for CPython
Message-ID: <4575B692.9090206@gmx.net>

Hi all,
I've just hacked together a coroutine interface for CPython. Basicly, 
I've stripped out all references to pypy from christians 
interp_coroutine.py module.
Amazingly, even pypy's stackless module works with this coroutine 
implementation.
At the moment, all of this can be found here:
https://codespeak.net/viewvc/user/stephan/hacks/coroutine/

Should I put this somewhere into the pypy repository? And if yes, where 
exactly?

Cheers

Stephan


From alexandre.fayolle at logilab.fr  Tue Dec  5 20:29:03 2006
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Tue, 5 Dec 2006 20:29:03 +0100
Subject: [pypy-dev] problem with build tool
Message-ID: <20061205192903.GE20280@crater.logilab.fr>

Hi,

I will probably have no network access tomorrow, so I post this as a
reminder, and also because the error takes time to reproduce. 

When finishing a build, the client process crashes with the following
exception:

<snip>
[translation:info] created: ./pypy-c
Traceback (most recent call last):
  File "bin/client", line 65, in ?
    zip.writestr(fpath.relto(udir), fpath.read())
  File "/home/alf/dev/pypy/dist/py/path/common.py", line 312, in read
    f = self.open(mode)
  File "/home/alf/dev/pypy/dist/py/path/local/local.py", line 200, in
open
    return self._callex(open, self.strpath, mode)
  File "/home/alf/dev/pypy/dist/py/path/common.py", line 229, in _callex
    raise cls, value
py.error.ENOENT: [No such file or directory]:
file('/tmp/usession-16/.lock', 'rb')
Exception exceptions.AttributeError: "'ChannelWrapper' object has no
attribute 'tell'" in <bound method ZipFile.__del__ of <zipfile.ZipFile
instance at 0x2aaaac415fc8>> ignored

The error is a bit mysterious to me, so if people in the assistance have
an idea of what is happening but no time to fix it, their opinion is
very welcome. 

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
Reprise et maintenance de sites CPS:     http://www.migration-cms.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20061205/04526b72/attachment.pgp>

From arigo at tunes.org  Tue Dec  5 20:35:28 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 5 Dec 2006 20:35:28 +0100
Subject: [pypy-dev] problem with build tool
In-Reply-To: <20061205192903.GE20280@crater.logilab.fr>
References: <20061205192903.GE20280@crater.logilab.fr>
Message-ID: <20061205193528.GA28030@code0.codespeak.net>

Hi,

On Tue, Dec 05, 2006 at 08:29:03PM +0100, Alexandre Fayolle wrote:
> py.error.ENOENT: [No such file or directory]:
> file('/tmp/usession-16/.lock', 'rb')

This .lock file is a symlink whose content is the pid of the process, so
uncareful places (as here) find it with listdir() but crash when opening
it.

Not sure why the build tool tries to zip it all, instead of just the
'testing_1' subdirectory.  Not sure why it didn't crash before, either.


A bientot,

Armin


From alexandre.fayolle at logilab.fr  Wed Dec  6 15:42:04 2006
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Wed, 6 Dec 2006 15:42:04 +0100
Subject: [pypy-dev] problem with build tool
In-Reply-To: <20061205193528.GA28030@code0.codespeak.net>
References: <20061205192903.GE20280@crater.logilab.fr>
	<20061205193528.GA28030@code0.codespeak.net>
Message-ID: <20061206144204.GA15969@crater.logilab.fr>

On Tue, Dec 05, 2006 at 08:35:28PM +0100, Armin Rigo wrote:
> Hi,
> 
> On Tue, Dec 05, 2006 at 08:29:03PM +0100, Alexandre Fayolle wrote:
> > py.error.ENOENT: [No such file or directory]:
> > file('/tmp/usession-16/.lock', 'rb')
> 
> This .lock file is a symlink whose content is the pid of the process, so
> uncareful places (as here) find it with listdir() but crash when opening
> it.
> 
> Not sure why the build tool tries to zip it all, instead of just the
> 'testing_1' subdirectory.  Not sure why it didn't crash before, either.

Well, before changes I checked in yesterday, it crashed before reaching
this line, so this is why ;-)

I fixed this locally in the train this morning, and will check in once
I've been able to run a successful translation, which I cannot do
reasonably while running on battery. 
 

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
Reprise et maintenance de sites CPS:     http://www.migration-cms.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20061206/f8b2ddd1/attachment.pgp>

From arigo at tunes.org  Fri Dec  8 03:19:50 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 8 Dec 2006 03:19:50 +0100
Subject: [pypy-dev] The first JITing pypy-c!
Message-ID: <20061208021946.GA1900@code0.codespeak.net>

Hi all,

Two days ago, we got our first JITing version of pypy-c!

This was mostly thanks to Arre, who compiled a version even though we
knew it would segfault due to missing support for C structures with
non-int-sized fields.  Writing to a 1-byte-sized Bool field would
overwrite the next 3 bytes of memory with zeroes...  Nevertheless, the
result managed to successfully run one function (and indeed segfault on
many other functions).  Our first JIT-run function!  A recursive
factorial.

Today the field size problem is fixed.  Playing around seems to show
that it's harder to provoke a segfault now.  The generated machine code
is completely incredible, in size and complexity, but the following
example runs.  It could even be said to run faster with the JIT (8.2
seconds versus 11.4 seconds) but that's unfair, as all normal
optimizations are turned off in this example (a regular pypy-c runs this
example in 2.8 seconds).  It still shows that our JIT already gives an
improvement over completely-unoptimized C code, which is already some
kind of success!

    def f(n):
        while n > 0:
            n -= 2
        return n

Many other examples give an UnboundLocalError, due probably to some
minor bug somewhere either in the JIT transformation or in the back-end
(along the lines of a == compiled as a !=).

If you want to try for yourself:

- check out or switch to the branch
  http://codespeak.net/svn/pypy/branch/jit-real-world

- run "translate.py /path/to/pypy/jit/goal/targetjit.py"
  (that's the usual translate.py from translator/goal)

- you get uncompiled C sources for now; copy it safely away
  from /tmp/usession-yourname/testing_1/ before it gets deleted,
  and compile it ("make" or "make debug").

- run "PYPYJITLOG=log ./testing_1"

    import pypyjit; pypyjit.enable(f.func_code)
    f(7000000)  # see f above

- the above PYPYJITLOG env var causes a file called 'log' to be
  produced, containing the generated assembler code.  It can be viewed
  in a flowgraph-like fashion with pypy/jit/codegen/i386/viewcode.py.
  Don't ask me yet to describe the result, nor where the while loop is
  :-)  If you are familiar with i386 assembler, you'll laugh at the
  obviously bad code, too.  That's where your help would be appreciated!
  Making the backend produce more reasonable code, starting with some
  basic register allocation, is a mostly-independent project.  The PPC
  backend, btw, has already got this kind of techniques (I wonder what
  speed-ups we get on PPC from a jitting pypy-c).


Have fun,

Armin


From niko at alum.mit.edu  Fri Dec  8 07:49:08 2006
From: niko at alum.mit.edu (Niko Matsakis)
Date: Fri, 08 Dec 2006 07:49:08 +0100
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <20061208021946.GA1900@code0.codespeak.net>
References: <20061208021946.GA1900@code0.codespeak.net>
Message-ID: <45790AE4.3030506@alum.mit.edu>


> Two days ago, we got our first JITing version of pypy-c!
>   
That's very exciting!  Congratulations.


Niko


From mwh at python.net  Fri Dec  8 11:23:26 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 08 Dec 2006 11:23:26 +0100
Subject: [pypy-dev] The first JITing pypy-c!
References: <20061208021946.GA1900@code0.codespeak.net>
Message-ID: <878xhifzz5.fsf@starship.python.net>

Armin Rigo <arigo at tunes.org> writes:

> Hi all,
>
> Two days ago, we got our first JITing version of pypy-c!
>
> This was mostly thanks to Arre, who compiled a version even though we
> knew it would segfault due to missing support for C structures with
> non-int-sized fields.  Writing to a 1-byte-sized Bool field would
> overwrite the next 3 bytes of memory with zeroes...  Nevertheless, the
> result managed to successfully run one function (and indeed segfault on
> many other functions).  Our first JIT-run function!  A recursive
> factorial.

Hooray!

> - the above PYPYJITLOG env var causes a file called 'log' to be
>   produced, containing the generated assembler code.  It can be viewed
>   in a flowgraph-like fashion with pypy/jit/codegen/i386/viewcode.py.
>   Don't ask me yet to describe the result, nor where the while loop is
>   :-)  If you are familiar with i386 assembler, you'll laugh at the
>   obviously bad code, too.  That's where your help would be appreciated!
>   Making the backend produce more reasonable code, starting with some
>   basic register allocation, is a mostly-independent project.  The PPC
>   backend, btw, has already got this kind of techniques (I wonder what
>   speed-ups we get on PPC from a jitting pypy-c).

Well, I guess so far it just plain doesn't work, but also the PPC
register allocation is pretty horrible, especially around function
calls.  We've talked about the backend change needed to make this
sensible (giving the list of live values to the block-ending builder
methods), I guess it just has to be done at some point...

Cheers,
mwh

-- 
  You can lead an idiot to knowledge but you cannot make him 
  think.  You can, however, rectally insert the information, 
  printed on stone tablets, using a sharpened poker.        -- Nicolai
               -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html



From rxe at ukshells.co.uk  Fri Dec  8 18:20:29 2006
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Fri, 8 Dec 2006 17:20:29 +0000 (GMT)
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <20061208021946.GA1900@code0.codespeak.net>
References: <20061208021946.GA1900@code0.codespeak.net>
Message-ID: <Pine.LNX.4.62.0612081718200.14085@echo.ukshells.co.uk>


Wow - that's awesome!  Congratulations to all involved.

Cheers,
Richard

On Fri, 8 Dec 2006, Armin Rigo wrote:

> Hi all,
>
> Two days ago, we got our first JITing version of pypy-c!
>
> This was mostly thanks to Arre, who compiled a version even though we
> knew it would segfault due to missing support for C structures with
> non-int-sized fields.  Writing to a 1-byte-sized Bool field would
> overwrite the next 3 bytes of memory with zeroes...  Nevertheless, the
> result managed to successfully run one function (and indeed segfault on
> many other functions).  Our first JIT-run function!  A recursive
> factorial.
>
> Today the field size problem is fixed.  Playing around seems to show
> that it's harder to provoke a segfault now.  The generated machine code
> is completely incredible, in size and complexity, but the following
> example runs.  It could even be said to run faster with the JIT (8.2
> seconds versus 11.4 seconds) but that's unfair, as all normal
> optimizations are turned off in this example (a regular pypy-c runs this
> example in 2.8 seconds).  It still shows that our JIT already gives an
> improvement over completely-unoptimized C code, which is already some
> kind of success!
>
>    def f(n):
>        while n > 0:
>            n -= 2
>        return n
>
> Many other examples give an UnboundLocalError, due probably to some
> minor bug somewhere either in the JIT transformation or in the back-end
> (along the lines of a == compiled as a !=).
>
> If you want to try for yourself:
>
> - check out or switch to the branch
>  http://codespeak.net/svn/pypy/branch/jit-real-world
>
> - run "translate.py /path/to/pypy/jit/goal/targetjit.py"
>  (that's the usual translate.py from translator/goal)
>
> - you get uncompiled C sources for now; copy it safely away
>  from /tmp/usession-yourname/testing_1/ before it gets deleted,
>  and compile it ("make" or "make debug").
>
> - run "PYPYJITLOG=log ./testing_1"
>
>    import pypyjit; pypyjit.enable(f.func_code)
>    f(7000000)  # see f above
>
> - the above PYPYJITLOG env var causes a file called 'log' to be
>  produced, containing the generated assembler code.  It can be viewed
>  in a flowgraph-like fashion with pypy/jit/codegen/i386/viewcode.py.
>  Don't ask me yet to describe the result, nor where the while loop is
>  :-)  If you are familiar with i386 assembler, you'll laugh at the
>  obviously bad code, too.  That's where your help would be appreciated!
>  Making the backend produce more reasonable code, starting with some
>  basic register allocation, is a mostly-independent project.  The PPC
>  backend, btw, has already got this kind of techniques (I wonder what
>  speed-ups we get on PPC from a jitting pypy-c).
>
>
> Have fun,
>
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Sat Dec  9 03:03:33 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 9 Dec 2006 03:03:33 +0100
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <20061208021946.GA1900@code0.codespeak.net>
References: <20061208021946.GA1900@code0.codespeak.net>
Message-ID: <20061209020332.GA14456@code0.codespeak.net>

Re-hi all,

On Fri, Dec 08, 2006 at 03:19:50AM +0100, Armin Rigo wrote:
> Many other examples give an UnboundLocalError

Fixed now.  Jitting execution seems to work as well as the normal one.
We tried various things, including stuff not supported by Psyco
(generators, nested scopes...), with success.

Now it's only a matter to make it produce useful code, as opposed to
just slow, big and incredible code...

> If you want to try for yourself:
> 
> - check out or switch to the branch
>   http://codespeak.net/svn/pypy/branch/jit-real-world

The branch is now merged into the trunk!  The rest of the instructions
still apply.  (The merging was a bit of fun, as usual; if someone thinks
that he might have lost any changes, don't hesitate to blame us before
checking in detail :-)


A bientot,

Armin.


From niko at alum.mit.edu  Sat Dec  9 12:17:21 2006
From: niko at alum.mit.edu (Niko Matsakis)
Date: Sat, 9 Dec 2006 12:17:21 +0100
Subject: [pypy-dev] Problem with subversion?
Message-ID: <45ACBD2E-D300-4285-B8F7-6FD0BA334092@alum.mit.edu>

When I attempt to do a "svn up", I get the following error:

;svn up
svnserve: error while loading shared libraries: libaprutil-0.so.0:  
cannot open shared object file: No such file or directory
svn: Connection closed unexpectedly
;

I get the error on multiple machines, both linux and OS/X, and I am  
able to work with other remote repositories, so I don't *think* it's  
my machine's fault...


Niko


From hpk at trillke.net  Sat Dec  9 15:36:27 2006
From: hpk at trillke.net (holger krekel)
Date: Sat, 9 Dec 2006 15:36:27 +0100
Subject: [pypy-dev] Problem with subversion?
In-Reply-To: <45ACBD2E-D300-4285-B8F7-6FD0BA334092@alum.mit.edu>
References: <45ACBD2E-D300-4285-B8F7-6FD0BA334092@alum.mit.edu>
Message-ID: <20061209143626.GH11948@solar.trillke.net>

Hi Niko,

On Sat, Dec 09, 2006 at 12:17 +0100, Niko Matsakis wrote:
> When I attempt to do a "svn up", I get the following error:
> 
> ;svn up
> svnserve: error while loading shared libraries: libaprutil-0.so.0:  
> cannot open shared object file: No such file or directory
> svn: Connection closed unexpectedly
> ;
> 
> I get the error on multiple machines, both linux and OS/X, and I am  
> able to work with other remote repositories, so I don't *think* it's  
> my machine's fault...

is the problem still there?  Today there was an update
of the apache server on the codespeak.net machine because 
the old one caused memory problems and had to be 
restarted daily.   If you still have problems,
please talk to Martin who did the upgrade (CCed
in this mail). 

best, 

    holger


From arigo at tunes.org  Sun Dec 10 01:38:02 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 10 Dec 2006 01:38:02 +0100
Subject: [pypy-dev] RuPy conference
Message-ID: <20061210003802.GA17355@code0.codespeak.net>

Hi all,

Should we try to get a PyPy talk at the upcoming RuPy conference in
Poznan (Poland)?

    http://rupy.wmid.amu.edu.pl/

With luck it could trigger the start of a translation frontend for the
restricted variant of Ruby that one of the Ruby-in-Ruby projects is
based on, if we can convince people that this way leads to a full Ruby
JIT :-)


A bientot,

Armin


From fijal at genesilico.pl  Sun Dec 10 11:24:58 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Sun, 10 Dec 2006 11:24:58 +0100
Subject: [pypy-dev] RuPy conference
In-Reply-To: <20061210003802.GA17355@code0.codespeak.net>
References: <20061210003802.GA17355@code0.codespeak.net>
Message-ID: <457BE07A.4000904@genesilico.pl>

Armin Rigo wrote:

>Hi all,
>
>Should we try to get a PyPy talk at the upcoming RuPy conference in
>Poznan (Poland)?
>  
>
Yes, it's definitely Poland ;-) About 3h train ride from my place.

>    http://rupy.wmid.amu.edu.pl/
>
>With luck it could trigger the start of a translation frontend for the
>restricted variant of Ruby that one of the Ruby-in-Ruby projects is
>based on, if we can convince people that this way leads to a full Ruby
>JIT :-)
>
>
>A bientot,
>
>Armin
>_______________________________________________
>pypy-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/pypy-dev
>  
>
The date is quite convinient (it's just after EU funding period, 
right?), I can take care about hotels there and such.

Cheers,
Fijal



From faassen at startifact.com  Mon Dec 11 20:40:08 2006
From: faassen at startifact.com (Martijn Faassen)
Date: Mon, 11 Dec 2006 20:40:08 +0100
Subject: [pypy-dev] nightly builds and benchmarks page down
Message-ID: <elkc6o$744$1@sea.gmane.org>

Hi there,

I've noticed for a while now that the following page is down:

http://snake.cs.uni-duesseldorf.de/pypytest/benchmark.html

It's listed here:

http://codespeak.net/pypy/dist/pypy/doc/index.html

I thought I'd post a message here in case that this isn't deliberate. :)

Regards,

Martijn



From elmo13 at jippii.fi  Mon Dec 11 22:01:44 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Mon, 11 Dec 2006 23:01:44 +0200
Subject: [pypy-dev] nightly builds and benchmarks page down
In-Reply-To: <elkc6o$744$1@sea.gmane.org>
References: <elkc6o$744$1@sea.gmane.org>
Message-ID: <457DC738.4070906@jippii.fi>

Martijn Faassen wrote:
> Hi there,
>
> I've noticed for a while now that the following page is down:
>
> http://snake.cs.uni-duesseldorf.de/pypytest/benchmark.html
>
> It's listed here:
>
> http://codespeak.net/pypy/dist/pypy/doc/index.html
>
> I thought I'd post a message here in case that this isn't deliberate. :)
>
> Regards,
>
> Martijn
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>   
That particular server ("snake") has been acting weird lately, and no
solution at the moment. So, not deliberate. Luckily, autotesting has
been set up on a nother server currently:
http://www.strakt.com/~pedronis/autotest/summary.html

Elmo


From fijal at genesilico.pl  Tue Dec 12 12:06:47 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Tue, 12 Dec 2006 12:06:47 +0100
Subject: [pypy-dev] nightly builds and benchmarks page down
In-Reply-To: <457DC738.4070906@jippii.fi>
References: <elkc6o$744$1@sea.gmane.org> <457DC738.4070906@jippii.fi>
Message-ID: <457E8D47.1070604@genesilico.pl>

Elmo M?ntynen wrote:

>Martijn Faassen wrote:
>  
>
>>Hi there,
>>
>>I've noticed for a while now that the following page is down:
>>
>>http://snake.cs.uni-duesseldorf.de/pypytest/benchmark.html
>>
>>It's listed here:
>>
>>http://codespeak.net/pypy/dist/pypy/doc/index.html
>>
>>I thought I'd post a message here in case that this isn't deliberate. :)
>>
>>Regards,
>>
>>Martijn
>>
>>_______________________________________________
>>pypy-dev at codespeak.net
>>http://codespeak.net/mailman/listinfo/pypy-dev
>>  
>>    
>>
>That particular server ("snake") has been acting weird lately, and no
>solution at the moment. So, not deliberate. Luckily, autotesting has
>been set up on a nother server currently:
>http://www.strakt.com/~pedronis/autotest/summary.html
>
>Elmo
>_______________________________________________
>pypy-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/pypy-dev
>  
>
And benchmarks are on tuatara.

http://tuatara.cs.uni-duesseldorf.de/benchmark.html




From mwh at python.net  Tue Dec 12 12:38:36 2006
From: mwh at python.net (Michael Hudson)
Date: Tue, 12 Dec 2006 12:38:36 +0100
Subject: [pypy-dev] nightly builds and benchmarks page down
References: <elkc6o$744$1@sea.gmane.org> <457DC738.4070906@jippii.fi>
Message-ID: <87irghe43n.fsf@starship.python.net>

Elmo M?ntynen <elmo13 at jippii.fi> writes:

> Martijn Faassen wrote:
>> Hi there,
>>
>> I've noticed for a while now that the following page is down:
>>
>> http://snake.cs.uni-duesseldorf.de/pypytest/benchmark.html
>>
>> It's listed here:
>>
>> http://codespeak.net/pypy/dist/pypy/doc/index.html
>>
>> I thought I'd post a message here in case that this isn't deliberate. :)
>>
>> Regards,
>>
>> Martijn
>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>   
> That particular server ("snake") has been acting weird lately, and no
> solution at the moment. So, not deliberate. Luckily, autotesting has
> been set up on a nother server currently:
> http://www.strakt.com/~pedronis/autotest/summary.html

But if it's a benchmark.html you're looking for, a smaller set of
benchmarks has been being run on 'tuatara' (a PowerMac G5 in the same
server room as snake):

    http://tuatara.cs.uni-duesseldorf.de/benchmark.html

Cheers,
mwh

-- 
  When physicists speak of a TOE, they don't really mean a theory
  of *everything*.  Taken literally, "Everything" covers a lot of
  ground, including biology, art, decoherence and the best way to
  barbecue ribs.                    -- John Baez, sci.physics.research



From len-l at telus.net  Thu Dec 14 00:50:21 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Wed, 13 Dec 2006 15:50:21 -0800
Subject: [pypy-dev] Confusion with meaning of "Stackless"
Message-ID: <4580213D.9801.19B2FC9@len-l.telus.net>

Tasklets are nice, but I just wish to update my understanding of 
"Stackless". Does the old motto "A Python Implementation That Does 
Not Use The C Stack" [1] still apply? Is it relevant to PyPy?

[1] http://stackless.com/index_old.htm
Lenard Lindstrom
<len-l at telus.net>




From rxe at ukshells.co.uk  Thu Dec 14 02:22:15 2006
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Thu, 14 Dec 2006 01:22:15 +0000 (GMT)
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <20061209020332.GA14456@code0.codespeak.net>
References: <20061208021946.GA1900@code0.codespeak.net>
	<20061209020332.GA14456@code0.codespeak.net>
Message-ID: <Pine.LNX.4.62.0612140119400.13608@echo.ukshells.co.uk>



Hi,

On Sat, 9 Dec 2006, Armin Rigo wrote:

> Re-hi all,
>
> On Fri, Dec 08, 2006 at 03:19:50AM +0100, Armin Rigo wrote:
>
> Fixed now.  Jitting execution seems to work as well as the normal one.
> We tried various things, including stuff not supported by Psyco
> (generators, nested scopes...), with success.

Very cool!  But do recursive intepreters work now? :-)

Cheers,
Richard



From arigo at tunes.org  Thu Dec 14 04:22:05 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 14 Dec 2006 04:22:05 +0100
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <Pine.LNX.4.62.0612140119400.13608@echo.ukshells.co.uk>
References: <20061208021946.GA1900@code0.codespeak.net>
	<20061209020332.GA14456@code0.codespeak.net>
	<Pine.LNX.4.62.0612140119400.13608@echo.ukshells.co.uk>
Message-ID: <20061214032205.GA28797@code0.codespeak.net>

Hi Richard,

On Thu, Dec 14, 2006 at 01:22:15AM +0000, Richard Emslie wrote:
> > Fixed now.  Jitting execution seems to work as well as the normal one.
> > We tried various things, including stuff not supported by Psyco
> > (generators, nested scopes...), with success.
> 
> Very cool!  But do recursive intepreters work now? :-)

Yes, everything works :-)  Except the bytecode trace hook.  The machine
code is really terrible at all points of view, but you can JIT whatever
piece of Python code you like - generators, nested scopes, class:
statement bodies, all these cases where Psyco give up.  That's the point
of the approach, really :-)

We have a minor detail to solve before it can be tested on larger
examples, though - it exhausts the 32-bit address space far too early
(without actually consuming much of the reserved pages) and then we get
a MemoryError.  It's a back-end problem; Arre started working on that
today.


A bientot,

Armin.


From arigo at tunes.org  Thu Dec 14 07:48:58 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 14 Dec 2006 07:48:58 +0100
Subject: [pypy-dev] Confusion with meaning of "Stackless"
In-Reply-To: <4580213D.9801.19B2FC9@len-l.telus.net>
References: <4580213D.9801.19B2FC9@len-l.telus.net>
Message-ID: <20061214064857.GA14933@code0.codespeak.net>

Hi Lenard,

On Wed, Dec 13, 2006 at 03:50:21PM -0800, Lenard Lindstrom wrote:
> Tasklets are nice, but I just wish to update my understanding of 
> "Stackless". Does the old motto "A Python Implementation That Does 
> Not Use The C Stack" [1] still apply? Is it relevant to PyPy?

Yes and no.  A bit confusingly, the usage of the term has evolved, along
with the Stackless Python project itself.  Now, "a Stackless feature"
means a feature that cannot be implemented without some form of explicit
stack control, either by being very careful about the C code (e.g.
avoiding recursive calls) or by saving and restoring pieces of the C
stack by a brute force 'memcpy' approach.  In PyPy, we use a variant of
the former approach: a "stackless build" of pypy-c is a compiled version
of some C code that was generated with careful systematic tweaks.

The motto of a Stackless PyPy is probably: "A Python implementation that
uses the C Stack as a cache".  Indeed, the C code supports saving its
own stack of frames away into the heap, and restoring it from there,
frame by frame.  So the C stack is really just a cache for the heap.  If
the cache is full (stack overflow), we save some more frames into the
heap to free some cache space.  If a context switch occurs (coroutine
switch) the cache is invalidated (entierely flushed to the heap) and the
new context's frames are copied from the heap to the cache as needed
(the function's frames are resumed one by one).


A bientot,

Armin.


From mwh at python.net  Thu Dec 14 10:11:36 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 14 Dec 2006 10:11:36 +0100
Subject: [pypy-dev] Confusion with meaning of "Stackless"
References: <4580213D.9801.19B2FC9@len-l.telus.net>
	<20061214064857.GA14933@code0.codespeak.net>
Message-ID: <873b7i6dvb.fsf@starship.python.net>

Armin Rigo <arigo at tunes.org> writes:

> Hi Lenard,
>
> On Wed, Dec 13, 2006 at 03:50:21PM -0800, Lenard Lindstrom wrote:
>> Tasklets are nice, but I just wish to update my understanding of 
>> "Stackless". Does the old motto "A Python Implementation That Does 
>> Not Use The C Stack" [1] still apply? Is it relevant to PyPy?
>
> Yes and no.  A bit confusingly, the usage of the term has evolved, along
> with the Stackless Python project itself.  Now, "a Stackless feature"
> means a feature that cannot be implemented without some form of explicit
> stack control, either by being very careful about the C code (e.g.
> avoiding recursive calls) or by saving and restoring pieces of the C
> stack by a brute force 'memcpy' approach.  In PyPy, we use a variant of
> the former approach: a "stackless build" of pypy-c is a compiled version
> of some C code that was generated with careful systematic tweaks.
>
> The motto of a Stackless PyPy is probably: "A Python implementation that
> uses the C Stack as a cache".  Indeed, the C code supports saving its
> own stack of frames away into the heap, and restoring it from there,
> frame by frame.  So the C stack is really just a cache for the heap.  If
> the cache is full (stack overflow), we save some more frames into the
> heap to free some cache space.  If a context switch occurs (coroutine
> switch) the cache is invalidated (entierely flushed to the heap) and the
> new context's frames are copied from the heap to the cache as needed
> (the function's frames are resumed one by one).

Do you feel like adding this explanation to the D07.1 report? :-)

Cheers,
mwh

-- 
  <radix> A VoIP server "powered entirely by stabbing, that I made
          out of this gun I had"                -- from Twisted.Quotes



From rxe at ukshells.co.uk  Thu Dec 14 19:02:34 2006
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Thu, 14 Dec 2006 18:02:34 +0000 (GMT)
Subject: [pypy-dev] The first JITing pypy-c!
In-Reply-To: <20061214032205.GA28797@code0.codespeak.net>
References: <20061208021946.GA1900@code0.codespeak.net>
	<20061209020332.GA14456@code0.codespeak.net>
	<Pine.LNX.4.62.0612140119400.13608@echo.ukshells.co.uk>
	<20061214032205.GA28797@code0.codespeak.net>
Message-ID: <Pine.LNX.4.62.0612141720360.31944@echo.ukshells.co.uk>


Hi Armin!

Wow!  So at risk of coming across really stupid, I am going to hazard a 
guess at what is going on.  :-)

As timeshifting is a translation time thing, we can handle mutiple or 
recursive merge points as a one off atomic operation before we run.  So 
now when we run we have two modes of execution - 'compile mode' and 
'normal mode'.  (i hope at least this part is right!)

So now we are running our timeshifted translated code, where the original 
rpython code defined a recursive interpreter, which in turn defined a 
recursive function with a mergepoint.  When we hit a flexiswitch (the 
merge point?), we go from normal mode to compile mode and may promote / 
partially evaluate run time variables for each new instance of a run time 
variable (the promotion hinted ones).  So what if a 'promotion' in compile 
mode triggers another promotion down the line (ie during the timeshifted 
residualizing code) and we go into a infinite loop of promoting?

Ok - so more questions than statements - and a high probabilty that 
it made no sense to all - and maybe I should just wait for the docs! :-)

Cheers,
Richard

On Thu, 14 Dec 2006, Armin Rigo wrote:

> Hi Richard,
>
> On Thu, Dec 14, 2006 at 01:22:15AM +0000, Richard Emslie wrote:
>>> Fixed now.  Jitting execution seems to work as well as the normal one.
>>> We tried various things, including stuff not supported by Psyco
>>> (generators, nested scopes...), with success.
>>
>> Very cool!  But do recursive intepreters work now? :-)
>
> Yes, everything works :-)  Except the bytecode trace hook.  The machine
> code is really terrible at all points of view, but you can JIT whatever
> piece of Python code you like - generators, nested scopes, class:
> statement bodies, all these cases where Psyco give up.  That's the point
> of the approach, really :-)
>
> We have a minor detail to solve before it can be tested on larger
> examples, though - it exhausts the 32-bit address space far too early
> (without actually consuming much of the reserved pages) and then we get
> a MemoryError.  It's a back-end problem; Arre started working on that
> today.
>
>
> A bientot,
>
> Armin.
>


From len-l at telus.net  Thu Dec 14 23:12:57 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Thu, 14 Dec 2006 14:12:57 -0800
Subject: [pypy-dev] Confusion with meaning of "Stackless"
In-Reply-To: <20061214064857.GA14933@code0.codespeak.net>
References: <4580213D.9801.19B2FC9@len-l.telus.net>
Message-ID: <45815BE9.10807.12AF266@len-l.telus.net>

On 14 Dec 2006 at 7:48, Armin Rigo wrote:

> Hi Lenard,
> 
> On Wed, Dec 13, 2006 at 03:50:21PM -0800, Lenard Lindstrom wrote:
> > Tasklets are nice, but I just wish to update my understanding of 
> > "Stackless". Does the old motto "A Python Implementation That Does 
> > Not Use The C Stack" [1] still apply? Is it relevant to PyPy?
> 
> Yes and no.  A bit confusingly, the usage of the term has evolved, along
> with the Stackless Python project itself.  Now, "a Stackless feature"
> means a feature that cannot be implemented without some form of explicit
> stack control, either by being very careful about the C code (e.g.
> avoiding recursive calls) or by saving and restoring pieces of the C
> stack by a brute force 'memcpy' approach.  In PyPy, we use a variant of
> the former approach: a "stackless build" of pypy-c is a compiled version
> of some C code that was generated with careful systematic tweaks.
> 
> The motto of a Stackless PyPy is probably: "A Python implementation that
> uses the C Stack as a cache".  Indeed, the C code supports saving its
> own stack of frames away into the heap, and restoring it from there,
> frame by frame.  So the C stack is really just a cache for the heap.  If
> the cache is full (stack overflow), we save some more frames into the
> heap to free some cache space.  If a context switch occurs (coroutine
> switch) the cache is invalidated (entierely flushed to the heap) and the
> new context's frames are copied from the heap to the cache as needed
> (the function's frames are resumed one by one).
> 
Hi Armin,

Thanks for the explanation. I find that "Stackless" is still 
associated with Scheme like continuations and unlimited recursion 
[1]. But yes, the PyPy documentation clearly states that  "Stackless" 
means greenlets, coroutines and tasklets at the application level.


[1] http://en.wikipedia.org/wiki/Stackless

Lenard Lindstrom
<len-l at telus.net>




From arigo at tunes.org  Fri Dec 15 10:17:28 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 15 Dec 2006 10:17:28 +0100
Subject: [pypy-dev] [stephan.diehl@gmx.net: Re: [pypy-svn] r35720 -
	pypy/dist/pypy/lib]
Message-ID: <20061215091728.GA25265@code0.codespeak.net>

Hi all,

With his permission...  (Note that user/stephan/hacks/coroutines is by no
way stable, according to Stephan.)

Armin

----- Forwarded message from Stephan Diehl <stephan.diehl at gmx.net> -----

Date: Thu, 14 Dec 2006 08:59:36 +0100
From: Stephan Diehl <stephan.diehl at gmx.net>
To: arigo at tunes.org
Subject: Re: [pypy-svn] r35720 - pypy/dist/pypy/lib

Hi Armin,

arigo at codespeak.net wrote:
>Author: arigo
>Date: Thu Dec 14 07:31:17 2006
>New Revision: 35720
>
>Added:
>   pypy/dist/pypy/lib/stackless.py
>      - copied unchanged from r35688, pypy/dist/pypy/lib/stackless.py
>   pypy/dist/pypy/lib/stackless_new.py
>      - copied unchanged from r35719, pypy/dist/pypy/lib/stackless.py
>Removed:
>   pypy/dist/pypy/lib/stackless_old.py
>Log:
>Temporarily keep the old stackless.py.  The new one seems to break all
>tests depending on it (at least its own test_stackless, and
>test_distributed).
>
>Stephan: I suspect that you only tested it on top of CPython with your
>greenlet-based wrapper (could you check it in somewhere too, btw?).  The
>behavior of this wrapper probably differs from the behavior of the
>module/_stackless coroutine in some way.

Arghhh. Actually, I ran ../../../test_all.py test_stackless.py from 
within the pypy/module/_stackless/test directory and it passed 
everything. Otherwise I wouldn't have replaced the stuff. Sigh.
Anyway. All the stuff can be found in
http://codespeak.net/svn/user/stephan/hacks/coroutine/
(the new stackless is called stackless2 :-)
The tests I have there (test_stackless2.py) are passing when running on 
top of CPython, CStackless and pypy-c.
(The secret plan is of course, to have the stackless interface available 
to CPython users --- minus the pickling, of course)

The coroutine module is just the (old) interp_coroutine minus the parts 
that depend on pypy. I suspect that it could be made much, much thinner.

Whenever this (greenlet based) coroutine is ready, I'd like to place it 
in py.magic.

I'm really sorry about the breakage. pypy is really a difficult beast :-)

Cheers

Stephan

P.S.: Unfortunatelly, I don't have any time left for this week, but I'll 
continue next week, to hunt down the bugs and complete the interface 
(and write much, much more tests as most features of stackless are not 
really covert yet)
>
>_______________________________________________
>pypy-svn mailing list
>pypy-svn at codespeak.net
>http://codespeak.net/mailman/listinfo/pypy-svn
>

----- End forwarded message -----


From jcea at argo.es  Fri Dec 15 12:48:03 2006
From: jcea at argo.es (Jesus Cea)
Date: Fri, 15 Dec 2006 12:48:03 +0100
Subject: [pypy-dev] Interesting technology to look at: Strongtalk
Message-ID: <45828B73.4050004@argo.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Since both smalltalk and python are highly dynamic language without
static type information, efficient execution is a challenge.

This smaltalk virtual machine efficiency is stunning. Maybe could give
some ideas to pypy effort.

http://www.strongtalk.org/

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRYKLc5lgi5GaxT1NAQKaogQAiKTVixiPa5q/94XE6EN+fr+lJ8+lTB+j
nkui/g4i0juLcyHzPY3hIcNxDSw5oDF/ljHQaLs1y8z2nwwQwI/TUAXWuxqCzmP8
h/t9BSKlywG4yieKToLXR4YEoDZvsW9640Crx28CNsJUP8fFhvjh6rvmuKSjbQmh
iyiyR1MCmys=
=seEN
-----END PGP SIGNATURE-----


From stephan.diehl at gmx.net  Sat Dec 16 10:28:18 2006
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Sat, 16 Dec 2006 10:28:18 +0100
Subject: [pypy-dev] update on CPython coroutine
Message-ID: <4583BC32.7070303@gmx.net>

Hi all,

after my (unsuccessful:-) attempt to place CPython runnable coroutines 
into py.magic (they might end up there just after the next release, 
we'll see), they live now in
http://codespeak.net/svn/user/stephan/hacks/coroutine/coroutine.py
In order to use them, on must place that file into the python path.
I've fixed a couple of things, Armin pointed out to me. I'd be happy to 
get some feedback.

Background: the pypy.lib.stackless_new implementation relies only on 
coroutines as a special requirement.
If one had coroutines that were running on CPython, one could run have 
the stackless interface on CPython as well (minus the pickling, of course).
It turned out that the coroutine interface is so similar to the greenlet 
one, that just a few lines of code are enough to provide the coroutine 
interface to greenlets.

Have a nice weekend

Stephan


From david at ar.media.kyoto-u.ac.jp  Mon Dec 18 04:32:45 2006
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 18 Dec 2006 12:32:45 +0900
Subject: [pypy-dev] Newcommer in pypy, numpy user
Message-ID: <45860BDD.4020004@ar.media.kyoto-u.ac.jp>

Hi there,

    I am a recently convert of python, which I started using for my 
numerical computing needs (I am a PhD student in signal processing) to 
replace matlab.

    For those who do not know matlab, it is a big (and expensive) 
software which implements a 'language' optimized for linear algebra, and 
with time becomes one of the most used software for numerical 
computation. In my field of research (signal processing), matlab is 
almost a standard (by standard, I mean everybody knows it and uses it). 
I think python with numpy/scipy is better in almost any way if you are 
ready to invest some time.

    Now, concerning pypy. The main idea of numpy is to give an array 
class to python, so that most inner loops are not interpreted, but run 
in Cpython or through highly optimized fortran libraries. In those 
cases, python is fast enough for most cases. But there are some cases 
where this paradigm of using linear algebra to speed things up does not 
work really well (recursive algorithms); in those case, the loop + 
function call cost of python makes any implementation for non toy 
problems really slow. Right now, the only choice is to code the thing in 
C, with a big loss on the flexibility side.

    I was wondering if pypy has some solution/new approaches to this 
problem. For example, when using numpy, I would suspect that many 
functions calls are 'static', that is always expect the same type of 
arguments; also, for simple loop on integers, my understanding is that 
JIT compilation has some nice solution to give to have much better 
performances (matlab has a JIT compiler to make loop faster for 
interpreted code). It looks like at some point, there was some work done 
in pypy relatively to numpy arrays, but I didn't find any documentation 
on that.

    cheers,

    David




From mwh at python.net  Mon Dec 18 17:39:36 2006
From: mwh at python.net (Michael Hudson)
Date: Mon, 18 Dec 2006 17:39:36 +0100
Subject: [pypy-dev] Newcommer in pypy, numpy user
References: <45860BDD.4020004@ar.media.kyoto-u.ac.jp>
Message-ID: <87ejqx17lj.fsf@starship.python.net>

David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:

> Hi there,
>
>     I am a recently convert of python, which I started using for my 
> numerical computing needs (I am a PhD student in signal processing) to 
> replace matlab.
>
>     For those who do not know matlab, it is a big (and expensive) 
> software which implements a 'language' optimized for linear algebra, and 
> with time becomes one of the most used software for numerical 
> computation. In my field of research (signal processing), matlab is 
> almost a standard (by standard, I mean everybody knows it and uses it). 
> I think python with numpy/scipy is better in almost any way if you are 
> ready to invest some time.
>
>     Now, concerning pypy. The main idea of numpy is to give an array 
> class to python, so that most inner loops are not interpreted, but run 
> in Cpython or through highly optimized fortran libraries. In those 
> cases, python is fast enough for most cases. But there are some cases 
> where this paradigm of using linear algebra to speed things up does not 
> work really well (recursive algorithms); in those case, the loop + 
> function call cost of python makes any implementation for non toy 
> problems really slow. Right now, the only choice is to code the thing in 
> C, with a big loss on the flexibility side.
>
>     I was wondering if pypy has some solution/new approaches to this 
> problem. For example, when using numpy, I would suspect that many 
> functions calls are 'static', that is always expect the same type of 
> arguments; also, for simple loop on integers, my understanding is that 
> JIT compilation has some nice solution to give to have much better 
> performances (matlab has a JIT compiler to make loop faster for 
> interpreted code).

It may be that the JIT is good for this sort of code.  It's not yet
though :-)

> It looks like at some point, there was some work done in pypy
> relatively to numpy arrays, but I didn't find any documentation on
> that.

As far as I'm aware this was something else: teaching PyPy's annotator
to recognise code that uses Numeric arrays and the code generator how
to compile this to equivalent code that manipulates arrays at a lower
level.  This could be seen as an alternative to rewriting your code in
C.  I'm not sure what the state of this code is, but I don't think
it's very advanced.

Cheers,
mwh

-- 
  The Internet is full.  Go away.
                      -- http://www.disobey.com/devilshat/ds011101.htm



From tismer at stackless.com  Mon Dec 18 21:15:24 2006
From: tismer at stackless.com (Christian Tismer)
Date: Mon, 18 Dec 2006 21:15:24 +0100
Subject: [pypy-dev] Confusion with meaning of "Stackless"
In-Reply-To: <45815BE9.10807.12AF266@len-l.telus.net>
References: <4580213D.9801.19B2FC9@len-l.telus.net>
	<45815BE9.10807.12AF266@len-l.telus.net>
Message-ID: <4586F6DC.40903@stackless.com>

Lenard Lindstrom wrote:
> On 14 Dec 2006 at 7:48, Armin Rigo wrote:

...
> Thanks for the explanation. I find that "Stackless" is still 
> associated with Scheme like continuations and unlimited recursion 
> [1]. But yes, the PyPy documentation clearly states that  "Stackless" 
> means greenlets, coroutines and tasklets at the application level.

Well, I should try to update that page.
While it is still true, the basic machinery
can support continuations, and infinite recursion
was available in every Stackless. The "tamed" continuations
which are greenlets, coroutines or tasklets seem to be
what is much more acceptable for the users.

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 177 402 25 57  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      where do you want to jump today?   http://www.stackless.com/


From len-l at telus.net  Tue Dec 19 01:15:48 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Mon, 18 Dec 2006 16:15:48 -0800
Subject: [pypy-dev] Confusion with meaning of "Stackless"
In-Reply-To: <4586F6DC.40903@stackless.com>
References: <45815BE9.10807.12AF266@len-l.telus.net>
Message-ID: <4586BEB4.13298.19981EC@len-l.telus.net>

On 18 Dec 2006 at 21:15, Christian Tismer wrote:

> Lenard Lindstrom wrote:
> > On 14 Dec 2006 at 7:48, Armin Rigo wrote:
> 
> ...
> > Thanks for the explanation. I find that "Stackless" is still 
> > associated with Scheme like continuations and unlimited recursion 
> > [1]. But yes, the PyPy documentation clearly states that  "Stackless" 
> > means greenlets, coroutines and tasklets at the application level.
> 
> Well, I should try to update that page.
> While it is still true, the basic machinery
> can support continuations, and infinite recursion
> was available in every Stackless. The "tamed" continuations
> which are greenlets, coroutines or tasklets seem to be
> what is much more acceptable for the users.
> 
Taskets interest people. But so did infinite recursion. That it is 
still a part of Stackless Python is worth mentioning. And if 
Stackless PyPy supports it then infinite recursion should be 
documented as a user level feature along with tasklets and such. It 
does effect the choices a coder makes.

As for continuations, I have only seen them advertised as a  language 
feature when the language provides a call/cc or something. Tasklets, 
coroutines, generators and such are basically treated as independent 
features.

Lenard Lindstrom
<len-l at telus.net>




From arigo at tunes.org  Tue Dec 19 13:36:00 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 19 Dec 2006 13:36:00 +0100
Subject: [pypy-dev] Confusion with meaning of "Stackless"
In-Reply-To: <4586BEB4.13298.19981EC@len-l.telus.net>
References: <45815BE9.10807.12AF266@len-l.telus.net>
	<4586BEB4.13298.19981EC@len-l.telus.net>
Message-ID: <20061219123600.GA17671@code0.codespeak.net>

Hi Lenard,

On Mon, Dec 18, 2006 at 04:15:48PM -0800, Lenard Lindstrom wrote:
> Taskets interest people. But so did infinite recursion. That it is 
> still a part of Stackless Python is worth mentioning. And if 
> Stackless PyPy supports it then infinite recursion should be 
> documented as a user level feature along with tasklets and such. It 
> does effect the choices a coder makes.

Good point.  I've added it to stackless.html.


A bientot,

Armin


From aurelien.campeas at logilab.fr  Tue Dec 19 18:50:46 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Tue, 19 Dec 2006 18:50:46 +0100
Subject: [pypy-dev] some questions
Message-ID: <20061219175046.GI28906@crater.logilab.fr>

Dear pypyers,

I've resumed work on WP9, with the intent to finish it ASAP. Right now
I face a little difficulty. I'd like to have your input (essentially
Armin's input I guess) on this.

Back when I was thinking about the requirements of WP9, esp. the
non-deterministic aspects, I thought clonable coroutines/threads would
do, fuzzily. Yet obviously these features have been provided and could
still be used to achieve my goals, but that is not what I have decided
to do since then...

What I'd like to know if I have painted myself in a corner with the
current design of the logic objectspace. Right now, a computation
space is defined as a set of concurrent activities (thread group); in
the case of constraint solving, there is one 'main' thread doing the
choices, and an army of propagator threads trying to empty the
domains. The idea is to associate a garbage collector pool to such a
set. I wrote the pool handling code and the clone stuff following as
closely as possible what's in modules/_stackless/interp_clonable and
friends (most recent versions I think); without a clear understanding
of how it works ... (see in objspace/cclp/types.py the Pool class,
some client code in space.py and scheduler.py)

Then, what happens now is that the cloning routine returns the
original thread group.

My question is the following :

* is there a chance that a peer review and eventual correction/advice
on that non-working code can yield a working solution in a reasonnable
amount of time ? or 

* is it time for me to simplify considerably the computation space
down to one single thread, so as to directly peruse the (working)
coroutine cloning facility ?

The later option would entail throwing away ten days of work, and
spending a more days to reorganize and rewrite stuff, but can be done
while retaining a reasonnable amount of functionnality.

(Well, in truth I expect a quick code review by Armin to expose some
blunders of mine... Just don't hit too hard my poor head.)

Regards,
Aur?lien.


From david at ar.media.kyoto-u.ac.jp  Wed Dec 20 03:50:37 2006
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 20 Dec 2006 11:50:37 +0900
Subject: [pypy-dev] Newcommer in pypy, numpy user
In-Reply-To: <87ejqx17lj.fsf@starship.python.net>
References: <45860BDD.4020004@ar.media.kyoto-u.ac.jp>
	<87ejqx17lj.fsf@starship.python.net>
Message-ID: <4588A4FD.90009@ar.media.kyoto-u.ac.jp>

Michael Hudson wrote:
> David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:
>
>> Hi there,
>>
>>     I am a recently convert of python, which I started using for my 
>> numerical computing needs (I am a PhD student in signal processing) to 
>> replace matlab.
>>
>>     For those who do not know matlab, it is a big (and expensive) 
>> software which implements a 'language' optimized for linear algebra, and 
>> with time becomes one of the most used software for numerical 
>> computation. In my field of research (signal processing), matlab is 
>> almost a standard (by standard, I mean everybody knows it and uses it). 
>> I think python with numpy/scipy is better in almost any way if you are 
>> ready to invest some time.
>>
>>     Now, concerning pypy. The main idea of numpy is to give an array 
>> class to python, so that most inner loops are not interpreted, but run 
>> in Cpython or through highly optimized fortran libraries. In those 
>> cases, python is fast enough for most cases. But there are some cases 
>> where this paradigm of using linear algebra to speed things up does not 
>> work really well (recursive algorithms); in those case, the loop + 
>> function call cost of python makes any implementation for non toy 
>> problems really slow. Right now, the only choice is to code the thing in 
>> C, with a big loss on the flexibility side.
>>
>>     I was wondering if pypy has some solution/new approaches to this 
>> problem. For example, when using numpy, I would suspect that many 
>> functions calls are 'static', that is always expect the same type of 
>> arguments; also, for simple loop on integers, my understanding is that 
>> JIT compilation has some nice solution to give to have much better 
>> performances (matlab has a JIT compiler to make loop faster for 
>> interpreted code).
>
> It may be that the JIT is good for this sort of code.  It's not yet
> though :-)
I understand that pypy is still in (relative) infancy, but I have to 
confess I am more and more interested in all those concepts of JIT, 
etc... even if I don't know much outside the concept and the big picture.
>
> As far as I'm aware this was something else: teaching PyPy's annotator
> to recognise code that uses Numeric arrays and the code generator how
> to compile this to equivalent code that manipulates arrays at a lower
> level.  This could be seen as an alternative to rewriting your code in
> C.  I'm not sure what the state of this code is, but I don't think
> it's very advanced.
>
I am not familiar with the pypy vocabulary yet, so let me rephrase to be 
sure I understand: the idea is to detect numpy arrays, and instead of 
using C extension for fast computation, it would generate automatically 
code  in the target language (let's say C for C generation) ?

So for example, if a and b are numpy arrays, a python expression b *= a 
would be translated in C by something like for(i = 0; i < b.size; ++i) 
{b[i] *= a[i];} ?

Is anyone working on that ?

cheers,

David


From mwh at python.net  Wed Dec 20 11:12:03 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 20 Dec 2006 11:12:03 +0100
Subject: [pypy-dev] Newcommer in pypy, numpy user
References: <45860BDD.4020004@ar.media.kyoto-u.ac.jp>
	<87ejqx17lj.fsf@starship.python.net>
	<4588A4FD.90009@ar.media.kyoto-u.ac.jp>
Message-ID: <87wt4myiz0.fsf@starship.python.net>

David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:

> Michael Hudson wrote:
>> David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:
>>>     I was wondering if pypy has some solution/new approaches to this 
>>> problem. For example, when using numpy, I would suspect that many 
>>> functions calls are 'static', that is always expect the same type of 
>>> arguments; also, for simple loop on integers, my understanding is that 
>>> JIT compilation has some nice solution to give to have much better 
>>> performances (matlab has a JIT compiler to make loop faster for 
>>> interpreted code).
>>
>> It may be that the JIT is good for this sort of code.  It's not yet
>> though :-)
> I understand that pypy is still in (relative) infancy, but I have to 
> confess I am more and more interested in all those concepts of JIT, 
> etc... even if I don't know much outside the concept and the big picture.

It's definitely interesting stuff :-)

>> As far as I'm aware this was something else: teaching PyPy's annotator
>> to recognise code that uses Numeric arrays and the code generator how
>> to compile this to equivalent code that manipulates arrays at a lower
>> level.  This could be seen as an alternative to rewriting your code in
>> C.  I'm not sure what the state of this code is, but I don't think
>> it's very advanced.
>>
> I am not familiar with the pypy vocabulary yet, so let me rephrase to be 
> sure I understand: the idea is to detect numpy arrays, and instead of 
> using C extension for fast computation, it would generate automatically 
> code  in the target language (let's say C for C generation) ?
>
> So for example, if a and b are numpy arrays, a python expression b *= a 
> would be translated in C by something like for(i = 0; i < b.size; ++i) 
> {b[i] *= a[i];} ?

Yes, I think that's a pretty good description.

> Is anyone working on that ?

Not that I know of.

Cheers,
mwh

-- 
  > say-hi-to-the-flying-pink-elephants-for-me-ly y'rs,
  No way, the flying pink elephants are carrying MACHINE GUNS!
  Aiiee!!  Time for a kinder, gentler hallucinogen...
                               -- Barry Warsaw & Greg Ward, python-dev



From faassen at startifact.com  Wed Dec 20 14:43:52 2006
From: faassen at startifact.com (Martijn Faassen)
Date: Wed, 20 Dec 2006 14:43:52 +0100
Subject: [pypy-dev] r35918 - pypy/dist/pypy/config
In-Reply-To: <20061220132226.ED5D21007E@code0.codespeak.net>
References: <20061220132226.ED5D21007E@code0.codespeak.net>
Message-ID: <embemm$9qf$1@sea.gmane.org>

mwh at codespeak.net wrote:
> Author: mwh
> Date: Wed Dec 20 14:22:23 2006
> New Revision: 35918
> 
> Modified:
>    pypy/dist/pypy/config/pypyoption.py
> Log:
> add a --faassen command line optimization that enables all optimizations.

I saw that. :)

Regards,

Martijn



From mwh at python.net  Wed Dec 20 15:24:33 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 20 Dec 2006 15:24:33 +0100
Subject: [pypy-dev] r35918 - pypy/dist/pypy/config
References: <20061220132226.ED5D21007E@code0.codespeak.net>
	<embemm$9qf$1@sea.gmane.org>
Message-ID: <87zm9iwspq.fsf@starship.python.net>

Martijn Faassen <faassen at startifact.com> writes:

> mwh at codespeak.net wrote:
>> Author: mwh
>> Date: Wed Dec 20 14:22:23 2006
>> New Revision: 35918
>> 
>> Modified:
>>    pypy/dist/pypy/config/pypyoption.py
>> Log:
>> add a --faassen command line optimization that enables all optimizations.
>
> I saw that. :)

Good :-)

Cheers,
mwh

-- 
  <shawn> the highest calling of technical book writers is to
          destroy the sun                       -- from Twisted.Quotes



From guido at merlinux.de  Wed Dec 20 15:50:44 2006
From: guido at merlinux.de (Guido Wesdorp)
Date: Wed, 20 Dec 2006 15:50:44 +0100
Subject: [pypy-dev] r35918 - pypy/dist/pypy/config
In-Reply-To: <embemm$9qf$1@sea.gmane.org>
References: <20061220132226.ED5D21007E@code0.codespeak.net>
	<embemm$9qf$1@sea.gmane.org>
Message-ID: <45894DC4.8080405@merlinux.de>

Martijn Faassen wrote:
> mwh at codespeak.net wrote:
>   
>> Author: mwh
>> Date: Wed Dec 20 14:22:23 2006
>> New Revision: 35918
>>
>> Modified:
>>    pypy/dist/pypy/config/pypyoption.py
>> Log:
>> add a --faassen command line optimization that enables all optimizations.
>>     
>
> I saw that. :)
>
>   
Hahaha!

Cheers,

Guido


From cfbolz at gmx.de  Wed Dec 20 15:51:51 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 20 Dec 2006 15:51:51 +0100
Subject: [pypy-dev] r35918 - pypy/dist/pypy/config
In-Reply-To: <45894DC4.8080405@merlinux.de>
References: <20061220132226.ED5D21007E@code0.codespeak.net>	<embemm$9qf$1@sea.gmane.org>
	<45894DC4.8080405@merlinux.de>
Message-ID: <45894E07.3040509@gmx.de>

Guido Wesdorp wrote:
> Martijn Faassen wrote:
>> mwh at codespeak.net wrote:
>>   
>>> Author: mwh
>>> Date: Wed Dec 20 14:22:23 2006
>>> New Revision: 35918
>>>
>>> Modified:
>>>    pypy/dist/pypy/config/pypyoption.py
>>> Log:
>>> add a --faassen command line optimization that enables all optimizations.
>>>     
>> I saw that. :)
>>
>>   
> Hahaha!
> 
> Cheers,
> 
> Guido

Admit it, Guido, it's all your fault!

:-)

Carl Friedrich


From simon at arrowtheory.com  Wed Dec 20 17:08:24 2006
From: simon at arrowtheory.com (Simon Burton)
Date: Wed, 20 Dec 2006 08:08:24 -0800
Subject: [pypy-dev] Newcommer in pypy, numpy user
In-Reply-To: <4588A4FD.90009@ar.media.kyoto-u.ac.jp>
References: <45860BDD.4020004@ar.media.kyoto-u.ac.jp>
	<87ejqx17lj.fsf@starship.python.net>
	<4588A4FD.90009@ar.media.kyoto-u.ac.jp>
Message-ID: <20061220080824.4cf6bd1d.simon@arrowtheory.com>

On Wed, 20 Dec 2006 11:50:37 +0900
David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:

> 
> So for example, if a and b are numpy arrays, a python expression b *= a 
> would be translated in C by something like for(i = 0; i < b.size; ++i) 
> {b[i] *= a[i];} ?

Yes, but numpy already does this. The ultimate goal was to be able to
collapse many such operations into an "optimal" (minimal) number of loops, eg.

a = b + c + d => for(...) {a[i]=b[i]+c[i]+d[i]}

which is much more cache friendly.

Simon.


From guido at merlinux.de  Wed Dec 20 21:39:31 2006
From: guido at merlinux.de (Guido Wesdorp)
Date: Wed, 20 Dec 2006 21:39:31 +0100
Subject: [pypy-dev] r35918 - pypy/dist/pypy/config
In-Reply-To: <45894E07.3040509@gmx.de>
References: <20061220132226.ED5D21007E@code0.codespeak.net>	<embemm$9qf$1@sea.gmane.org>
	<45894DC4.8080405@merlinux.de> <45894E07.3040509@gmx.de>
Message-ID: <45899F83.1000403@merlinux.de>

Carl Friedrich Bolz wrote:
> Admit it, Guido, it's all your fault!
>
>   
Fault? It's an honour, isn't it? :)

Cheers,

Guido


From arigo at tunes.org  Thu Dec 21 17:44:00 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 21 Dec 2006 17:44:00 +0100
Subject: [pypy-dev] some questions
In-Reply-To: <20061219175046.GI28906@crater.logilab.fr>
References: <20061219175046.GI28906@crater.logilab.fr>
Message-ID: <20061221164400.GA8604@code0.codespeak.net>

Hi Aurelien,

On Tue, Dec 19, 2006 at 06:50:46PM +0100, Aur?lien Camp?as wrote:
> * is there a chance that a peer review and eventual correction/advice
> on that non-working code can yield a working solution in a reasonnable
> amount of time ?

No.  Getting the original coroutine cloning code to really work
correctly took us quite some time, and certainly more than 10 days,
including a final night following data structures at the C level with
ddd.  (Btw, that's a gdb front-end that I really recommend for Debugging
and Displaying Data :-)  It looks innocent enough but it is really
subtle code, so you should try to build on top of that and not reinvent
it from scratch.

Note also that if you need sets of several coroutines that are all
cloned together, it's possible to put each set in a single
InterpClonableCoroutine that acts as a "big" thread; and it would itself
contain several regular Coroutines for its own local purposes.  This
works nicely if in each InterpClonableCoroutine you first create a new
CoState() instance, and use it as the 'state' argument of all the local
Coroutines.  This local CoState has a current and a main that are local
too.  If you clone one of the "big" InterpClonableCoroutine, everything
will be cloned - the local CoState and its Coroutines.

Hum, actually for now that's theory only.  The local Coroutines have a
__del__ and objects with a __del__ are never cloned.  I guess that my
point is that we could make this work with minimal efforts (e.g. with a
variety of Coroutines that doesn't need a __del__).

> The later option would entail throwing away ten days of work, and
> spending a more days to reorganize and rewrite stuff, but can be done
> while retaining a reasonnable amount of functionnality.

Well, I would have given you the same answer 10 days ago, or even a few
months ago, if only I had known.  I should point out that we ("the
technical board") tried to ask Logilab in general to tell us what plans
they had, and we tried quite a few times now, so I fail to feel bad
about telling you now to throw away just 10 days of work, sorry.


A bientot,

Armin


From aurelien.campeas at logilab.fr  Thu Dec 21 18:13:25 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Thu, 21 Dec 2006 18:13:25 +0100
Subject: [pypy-dev] some questions
In-Reply-To: <20061221164400.GA8604@code0.codespeak.net>
References: <20061219175046.GI28906@crater.logilab.fr>
	<20061221164400.GA8604@code0.codespeak.net>
Message-ID: <20061221171325.GB20558@crater.logilab.fr>

On Thu, Dec 21, 2006 at 05:44:00PM +0100, Armin Rigo wrote:
> Hi Aurelien,
> 
> On Tue, Dec 19, 2006 at 06:50:46PM +0100, Aur?lien Camp?as wrote:
> > * is there a chance that a peer review and eventual correction/advice
> > on that non-working code can yield a working solution in a reasonnable
> > amount of time ?
> 
> No.  Getting the original coroutine cloning code to really work
> correctly took us quite some time, and certainly more than 10 days,
> including a final night following data structures at the C level with
> ddd.  (Btw, that's a gdb front-end that I really recommend for Debugging
> and Displaying Data :-)  It looks innocent enough but it is really
> subtle code, so you should try to build on top of that and not reinvent
> it from scratch.

I never meant to reinvent this from scratch. You also said to me
recently that the local pooling was not especially tied to coroutines,
which matched the intuition I had from reading the interp_clonable
stuff.

> 
> Note also that if you need sets of several coroutines that are all
> cloned together, it's possible to put each set in a single
> InterpClonableCoroutine that acts as a "big" thread; and it would itself
> contain several regular Coroutines for its own local purposes.  This
> works nicely if in each InterpClonableCoroutine you first create a new
> CoState() instance, and use it as the 'state' argument of all the local
> Coroutines.  This local CoState has a current and a main that are local
> too.  If you clone one of the "big" InterpClonableCoroutine, everything
> will be cloned - the local CoState and its Coroutines.

Some time ago I tried that road but abandonned it because it yielded
non-working builds. But at that time I missed the "this works nicely
if ..." part. Thanks for taking the time to talk about it.

> 
> Hum, actually for now that's theory only.  The local Coroutines have a
> __del__ and objects with a __del__ are never cloned.  I guess that my
> point is that we could make this work with minimal efforts (e.g. with a
> variety of Coroutines that doesn't need a __del__).

I don't think in my usage of Coroutines I need a __del__. So maybe
there is a tiny hope.

> 
> > The later option would entail throwing away ten days of work, and
> > spending a more days to reorganize and rewrite stuff, but can be done
> > while retaining a reasonnable amount of functionnality.
> 
> Well, I would have given you the same answer 10 days ago, or even a few
> months ago, if only I had known.  I should point out that we ("the
> technical board") tried to ask Logilab in general to tell us what plans
> they had, and we tried quite a few times now, 

Logilab's plan wrt WP9 have always been clear I think, but
whatever. Talking about logilab relationships with the PyPy group is
for another day.

> so I fail to feel bad
> about telling you now to throw away just 10 days of work, sorry.

Well, sorry if you interpreted my poor wording about days of work
being thrown away as an attempt to culpabilize you. That was not my
intent. Moreover I highly respect what you have done for the cloning
facility and everything in general the core PyPy developpers have
achieved so far.

Thanks,
Aur?lien.


From arigo at tunes.org  Wed Dec 27 11:44:34 2006
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 27 Dec 2006 11:44:34 +0100
Subject: [pypy-dev] Machines at uni-duesseldorf
Message-ID: <20061227104434.GA10138@code0.codespeak.net>

Hi all,

Here's an update on the computers that we have for PyPy at the HHU.

Snake is back, but nothing wrong was found with its hardware, which
means that it'll probably continue to hang randomly from time to time...
Maybe we'll reinstall the machine from scratch.  For all I know it could
be corrupted system files.

However, there are now two other machines.  Just like snake they are
machines whose performances can be a bit strange - hyperthreaded Xeon
cores.  Each core has roughly snake-like performance, but each machine
has got 4 cores.  The software setup is: SuSE Linux base, which I'm
hiding as much as possible behind a Gentoo Linux installation in the
/gentoo directory.  When you log in, you reach a chrooted environment,
so you should not see that at all unless you dig in special files.

I've copied some accounts from codespeak, so you must use the same public
key to log in (no password logins).  Just ask me if you want to use the
machines but I forgot to copy your account over.

To login, you must use port 922 instead of 22.  The latter would reach
the SSH daemon of the outer SuSE installation (where you don't have an
account).  I recommend adding the following lines to your file
~/.ssh/config:

Host wyvern
    HostName wyvern.cs.uni-duesseldorf.de
    Port 922
Host cobra
    HostName cobra.cs.uni-duesseldorf.de
    Port 922

Then you can log in with 'ssh wyvern' or 'ssh cobra'.  Note that
'wyvern' is the fastest machine so feel free to prefer it.

A great way to use them is for distributed testing.  Put in your file
~/conftest.py:

disthosts = ['wyvern'] * 4 + ['cobra'] * 4

Then you can do 'py.test --session=R'.  Try also the --runbrowser
option.


A bientot,

Armin


From arigo at tunes.org  Sat Dec 30 16:05:49 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 30 Dec 2006 16:05:49 +0100
Subject: [pypy-dev] Automated tests
Message-ID: <20061230150546.GA17431@code0.codespeak.net>

Hi all,

The automated tests are running on wyvern nightly:

    http://wyvern.cs.uni-duesseldorf.de/pypytest/summary.html

As the tests are parallelized it takes on the order of two hours to run
them all, so we could consider running them more than once per day.

Something else I'd like to consider is a tool that checks new failures
shown by these test runs, finding the exact revision that seemed to
broke it, and actively signals the problem somewhere - maybe as a notice
by elbowtone on #pypy and as an e-mail to e.g. the author of the
revision?  What do you think about it?

Longer term, we could also have a way to ask for a complete test run on
wyvern, so that we can check within two hours if we broke something
after a delicate check-in, or just after we fixed a couple of problems.
If we manage 0-failure runs at least every few days we could then have
an URL to which we copy the trunk whenever all tests pass, so that
"outside" people have the option to follow with 'svn up' an URL that is
at least fully tested instead of the bleeding edge trunk.


A bientot,

Armin


From santagada at gmail.com  Sat Dec 30 17:39:01 2006
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 30 Dec 2006 14:39:01 -0200
Subject: [pypy-dev] Automated tests
In-Reply-To: <20061230150546.GA17431@code0.codespeak.net>
References: <20061230150546.GA17431@code0.codespeak.net>
Message-ID: <24D9047B-1393-44E9-927C-1AA786EC1CC8@gmail.com>


Em 30/12/2006, ?s 13:05, Armin Rigo escreveu:

> Hi all,
>
> The automated tests are running on wyvern nightly:
>
>     http://wyvern.cs.uni-duesseldorf.de/pypytest/summary.html
>
> As the tests are parallelized it takes on the order of two hours to  
> run
> them all, so we could consider running them more than once per day.
>
> Something else I'd like to consider is a tool that checks new failures
> shown by these test runs, finding the exact revision that seemed to
> broke it, and actively signals the problem somewhere - maybe as a  
> notice
> by elbowtone on #pypy and as an e-mail to e.g. the author of the
> revision?  What do you think about it?
>
Does buildbot do something like this or you will have to implement  
all yourself?

> Longer term, we could also have a way to ask for a complete test  
> run on
> wyvern, so that we can check within two hours if we broke something
> after a delicate check-in, or just after we fixed a couple of  
> problems.
> If we manage 0-failure runs at least every few days we could then have
> an URL to which we copy the trunk whenever all tests pass, so that
> "outside" people have the option to follow with 'svn up' an URL  
> that is
> at least fully tested instead of the bleeding edge trunk.
>
We could just have a moving tag named "latest_tested" and people  
could be following it (or a branch, I really don't know that much  
about svn)

>
> A bientot,
>
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

Leonardo Santagada
santagada at gmail.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20061230/8eaff1e7/attachment.html>