From p.giarrusso at gmail.com  Thu Jan  1 06:38:39 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 1 Jan 2009 06:38:39 +0100
Subject: [pypy-dev] [Fwd: [Fwd: Re: Threaded interpretation]]
In-Reply-To: <495AA2C0.9050105@gmail.com>
References: <495AA2C0.9050105@gmail.com>
Message-ID: <a4d0c86c0812312138p4742f398mf2017fc5af282ad9@mail.gmail.com>

On Tue, Dec 30, 2008 at 23:37, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Hi,
> Antoine Pitrou told me that his mail got rejected by the mailing list, so I'm
> forwarding it.

> -------- Message transf?r? --------
> De: Antoine Pitrou <solipsis at pitrou.net>
> ?: pypy-dev at codespeak.net
> Sujet: Re: Threaded interpretation
> Date: Fri, 26 Dec 2008 21:16:36 +0000 (UTC)
>
> Hi people,
>
> By reading this thread I had the idea to write a threaded interpretation patch
> for py3k. Speedup on pybench and pystone is 15-20%.
> http://bugs.python.org/issue4753

> Regards

> Antoine.

That's nice!

I didn't have time until now (Erasmus life is pretty demanding), but
it was nice anyway, also because the code is a bit different, and a
bit better than mine (I didn't realize I could fetch the operand after
dispatch instead of before, and this makes a difference, because after
the jump the result of HAS_ARG() becomes known at compile time).

Regards
-- 
Paolo Giarrusso

From arigo at tunes.org  Fri Jan  2 14:23:38 2009
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 2 Jan 2009 14:23:38 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<20081223150113.GA1522@code0.codespeak.net>
	<a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>
Message-ID: <20090102132338.GA17873@code0.codespeak.net>

Hi Paolo,

On Thu, Dec 25, 2008 at 12:42:18AM +0100, Paolo Giarrusso wrote:
> If I'll want to try something without refcounting, I'll guess I'd turn
> to PyPy, but don't hold your breath for that. The fact that indirect
> threading didn't work, that you're 1.5-2x slower than CPython, and
> that you store locals in frame objects, they all show that the
> abstraction overhead of the interpret is too high.

True, but a 1.5x slowdown is not a big deal on many application; the
blocker is mostly elsewhere.  And on the other hand, we've been working
on the JIT generator -- since a while now, so I cannot make any promise
-- and the goal is to turn this slowish interpreter into a good JITting
one "for free".  As far as I know, this cannot be done so easily if you
start from a lower-level interpreter like CPython.  This is our "real"
goal somehow, or one of our real goals: a JITting virtual machine for
any language that we care to write an interpreter for.

> 3) still, I do believe that working on it was interesting to get
> experience about how to optimize an interpreter.

Sure, it makes some sense if you think about it in this way.  I doesn't
if you think about "I want to give the world a fast Python interpreter",
but you corrected me already: this is not your goal, so it's fine :-)

> And the original idea was to show that real multithreading (without a
> global interpreter lock) cannot be done in Python just because of the
> big design mistakes of CPython.

I disagree on this point.  Jython shows that without redesign of the
language, they can implement a real multithreading model without a GIL.
About PyPy, the lesson that we learned is different: it's that
implementing a GIL-free model requires a bit of tweaking of the whole
interpreter -- as opposed to doing it in a nicely separable manner.  So
far we only implemented threads-with-a-GIL, which can be done in this
separable way (i.e. without changing a single line of most of the
interpreter).  Given enough interest we can implement full GIL-free
multithreading the slow way: by going over most features of the
interpreter and thinking.  Note that it would also be possible to do
that in CPython (but it would be even more work).

It's just that doing so is not really "in the spirit" of PyPy :-)
In our style, we would rather start by thinking harder about some
unpolished ideas that float around about doing it in a separable
manner anyway -- but this has not been tried at all.


A bientot,

Armin.


From p.giarrusso at gmail.com  Sat Jan  3 05:59:57 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 3 Jan 2009 05:59:57 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <20090102132338.GA17873@code0.codespeak.net>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<20081223150113.GA1522@code0.codespeak.net>
	<a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>
	<20090102132338.GA17873@code0.codespeak.net>
Message-ID: <a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>

On Fri, Jan 2, 2009 at 14:23, Armin Rigo <arigo at tunes.org> wrote:
> Hi Paolo,

> On Thu, Dec 25, 2008 at 12:42:18AM +0100, Paolo Giarrusso wrote:
>> If I'll want to try something without refcounting, I'll guess I'd turn
>> to PyPy, but don't hold your breath for that. The fact that indirect
>> threading didn't work, that you're 1.5-2x slower than CPython, and
>> that you store locals in frame objects, they all show that the
>> abstraction overhead of the interpret is too high.

> True, but a 1.5x slowdown is not a big deal on many application; the
> blocker is mostly elsewhere.

Let's say 1.5x * 2x = 3x, since CPython is not as fast as it could be,
because of refcounting for instance. The 2x is taken from PyVM
performance reports (see http://www.python.org/dev/implementations/).
And for PyPy a slower interpreter means a slower JIT output, isn't it?
See below.

Also, according to CS literature, interpreter performance makes more
difference than JIT. According to the paper about efficient
interpreters I'm mentioning endless times, an inefficient interpreter
is 1000x slower than C, while an efficient one is only 10x slower.
Python is not that slow, but what you wrote about Psyco seems to imply
that there is still lots of room for improvement.

"You might get 10x to 100x speed-ups. It is theoretically possible to
actually speed up this kind of code up to the performance of C
itself."

At some point, I should repeat these comparison with the OcaML interpreter:
  http://mail.python.org/pipermail/python-list/2000-December/063212.html
to see how faster it got :-).

> And on the other hand, we've been working
> on the JIT generator -- since a while now, so I cannot make any promise
> -- and the goal is to turn this slowish interpreter into a good JITting
> one "for free".  As far as I know, this cannot be done so easily if you
> start from a lower-level interpreter like CPython.  This is our "real"
> goal somehow, or one of our real goals: a JITting virtual machine for
> any language that we care to write an interpreter for.

That's amazing - I knew that, but your explanation is much nicer,
since it gives the emphasis to the right thing.
>From a native interpreter you can only do code copying, which removes
dispatch costs but is still suboptimal.

And well, the current overhead can be removed with further
improvements on the interpreter and/or the RPython translator. And it
is important, for two reasons:
1) an interpreter is still needed to profile the code; experience with
Java showed that only having a JIT gives too much latency, and people
expect even less latency from Python.
2) partially different translators are used I guess, but since the
compiler and the interpreter are built from the same source, part of
the 3x slowdown would have effect also on the JIT-ed code.

In any case, even if it may be unclear, I do like your plans.
Believe me, lack of time until now is the only thing which prevented
me from actually joining you and coding.

>> And the original idea was to show that real multithreading (without a
>> global interpreter lock) cannot be done in Python just because of the
>> big design mistakes of CPython.

> I disagree on this point.  Jython shows that without redesign of the
> language, they can implement a real multithreading model without a GIL.

Ok, first I have to rephrase what I said. Python in itself is a Good
Thing. BUT, there are some big design mistakes about some semantic
details of __del__, which make a difference for code using files
without calling close() for instance. To be polite, those semantics
are totally brain-dead, unless one worships reference counting and
considers GC as the devil.

Having clarified that, I can state that indeed, Jython may only show
the opposite of what you say; indeed I like to say that they didn't
bother to be bug-to-bug compatible with Python _specification bugs_,
and that's why it seems they proved your point.

Need I point out that reference cycles can't be reclaimed in Python if
any involved object implements __del__?
I know the Python gc module allows to fix that, but in really clumsy
ways. I was going to bash Python specs a lot more, but I'll spare your
time for now :-).

Regards
-- 
Paolo Giarrusso


From p.giarrusso at gmail.com  Sat Jan  3 07:18:22 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 3 Jan 2009 07:18:22 +0100
Subject: [pypy-dev] GIL removal in PyPy (was: Re: Threaded interpretation)
Message-ID: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>

On Fri, Jan 2, 2009 at 14:23, Armin Rigo <arigo at tunes.org> wrote:

> About PyPy, the lesson that we learned is different: it's that
> implementing a GIL-free model requires a bit of tweaking of the whole
> interpreter -- as opposed to doing it in a nicely separable manner.  So
> far we only implemented threads-with-a-GIL, which can be done in this
> separable way (i.e. without changing a single line of most of the
> interpreter).

> Given enough interest we can implement full GIL-free
> multithreading the slow way: by going over most features of the
> interpreter and thinking.

That'll take more time now than it would have taken to do it in the first place.

IMHO, introducing fine-grained locking is one of the things costing x
if done from the beginning and costing 10x to do afterwards. Switching
from refcounting to GC is another of those things (the factor here is
maybe even higher), and your EU report on GC acknowledges this when
talking about CPython's choice of refcounting. The location of all
pointers on the C stack would have to be registered to convert
CPython.
So, I can understand what you mean by "nicely separable".

> Note that it would also be possible to do
> that in CPython (but it would be even more work).

That has been proven to be false :-).
It has been tried in 2004 (look for "Python free threading", I guess),
but free (i.e. GIL-less) threading requires atomic manipulation of
reference counts, and that gave a 2x overall slowdown; nowadays, the
relative impact would be even higher, just because . The reaction from
Guido van Rossum and/or the list (IIRC) was "ok, it can't be done,
we're not gonna have real multithreading, and there are better
programming models anyway".
There was, luckily, somebody on the mailing list who said "maybe we
should drop refcounting", but people didn't listen for some reason.

> It's just that doing so is not really "in the spirit" of PyPy :-)
> In our style, we would rather start by thinking harder about some
> unpolished ideas that float around about doing it in a separable
> manner anyway -- but this has not been tried at all.

That's interesting, can I get some pointers to these ideas?
I've heard about some work on this on the AOP front, but to me it
looks like their work is still not relevant, except for explaining us
that locking is a crosscutting concern, so I'd be interested about
something that could actually work.

And an _interpreter_ is not special at all for this, I think, it is
just another concurrent application (a compiler might be a different
case) - if you want to keep it simple, making everything in the data
space a monitor is the best bet. The problem is, that anything simple
will probably perform horribly and need to be optimized. And for me,
it's easier to write code in a much more sensible way from day one,
than optimizing it afterward.

=== Why AOP is broken for locking ===
I'm simply too used to advanced (and even not advanced) locking
pattern that can't be expressed in AOP. Given the following
independent snippets in an application:

LOCK(obj)
obj.a()
obj.b()
UNLOCK(obj)

LOCK(obj)
obj.b()
obj.c()
UNLOCK(obj)

how can an aspect introduce the calls to locks if they are omitted
from the source? I can't add an advice on a() or b(), the a()-b()
sequence must be atomic. And often I can't advice the containing
methods, unless it begins and ends with a locking call; obj might not
even be available at method entry. The only solution is to split each
critical sections out of the containing method into another one, but
that's not cheaper than AOP usage.

But the real challenge is how to implement, for instance, the
Collection.addAll(Collection b) Java method, if it has to lock both
dictionaries. Thread A might call a.addAll(b) while thread B calls
b.addAll(a) and any thread might attempt any concurrent modification
on any of a and b. So, which is the correct locking order to avoid
deadlock?

If I had really to face this problem (this particular example is
partially unrealistic - locking for addAll should be a responsibility
of the caller, probably), the only solution is to find some total
order on all collections and lock first the smaller object. In C, I
would compare their pointers, here I'd have to find some trick (like
comparing their identityHashCode() in Java).

I've been used to similar issues while coding in the Linux kernel, and
that's why I think that the locking concern should not be separated
from the application code. Unless one uses message passing, but I
can't see any way to implement multithreading by using message passing
that is not horribly inefficient.

The first Google hit on "AOP locking" is this article:
http://www.voelter.de/data/articles/aop/aop.html
and the example it makes is just too simple and unrealistic - that
pattern might apply to a minority of real-world cases. The author
seems unaware of real issues in concurrent code.

Moreover, I'm concerned about the quality of the article. When
discussing the "classic" OOP solution, it makes a really basic mistake
about the usage of inheritance. What follows is a copy-n-paste, with
added emphasis:
"The traditional approach is to introduce an (abstract) base class,
from which ALL "WORKER" CLASSES INHERIT. This class defines a method
lock() and a method unlock() which must be called before and after the
actual work is done (semaphores, basically)."
In fact, the lock() and unlock() method are not part of the API of the
Worker classes, so one would just use an instance of such a Lock
class, not inherit from it. And even writing "semaphore" is slightly
incorrect.

This article is much better:
http://www.kentlai.me/articles-journal/locks-via-spring-aop.html
still, it shares the same fundamental limitations. All method in a
class share the same lock, and locking only happens on method
boundaries. In fact, the article seems to acknowledge that AOP is just
a cheap way to introduce really simplistic locking.

Having said that, I'm really interested to see how JBoss uses AOP for
locking. But those problems seem really fundamental to me, so probably
just the simple cases are handled through AOP.

Regards
-- 
Paolo Giarrusso


From santagada at gmail.com  Sat Jan  3 12:05:25 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 3 Jan 2009 09:05:25 -0200
Subject: [pypy-dev] GIL removal in PyPy (was: Re: Threaded
	interpretation)
In-Reply-To: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
References: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
Message-ID: <26B655DE-C3D2-4BA3-851E-29BA0B9CB728@gmail.com>


On Jan 3, 2009, at 4:18 AM, Paolo Giarrusso wrote:

> There was, luckily, somebody on the mailing list who said "maybe we
> should drop refcounting", but people didn't listen for some reason.


You repeated this meme many times in your emails, so I thought that  
maybe you really didn't see the full picture. This is what I  
understand from the reasoning behind it.

Dropping refcounting and move to free threading would completely break  
all C modules so they would have to be rewritten and would make the  
CPython API much more complex and integration with C libraries hard.  
That's why no one took it seriously. Think like this, breaking all c  
modules would make CPython as usable as haskell :), or just look at  
the number of libraries not available right now for Python 3.0.

It is not some retarded choice made by GvR, but a pragmatic one.  
Python as a language used by millions of people can't completely  
change semantics from version to version.

--
Leonardo Santagada
santagada at gmail.com





From cfbolz at gmx.de  Sat Jan  3 13:23:03 2009
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sat, 03 Jan 2009 13:23:03 +0100
Subject: [pypy-dev] GIL removal in PyPy (was: Re:
	Threaded	interpretation)
In-Reply-To: <26B655DE-C3D2-4BA3-851E-29BA0B9CB728@gmail.com>
References: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
	<26B655DE-C3D2-4BA3-851E-29BA0B9CB728@gmail.com>
Message-ID: <495F58A7.6080303@gmx.de>

Hi Leonardo,

Leonardo Santagada wrote:
> On Jan 3, 2009, at 4:18 AM, Paolo Giarrusso wrote:
> 
>> There was, luckily, somebody on the mailing list who said "maybe we
>> should drop refcounting", but people didn't listen for some reason.
> 
> 
> You repeated this meme many times in your emails, so I thought that  
> maybe you really didn't see the full picture. This is what I  
> understand from the reasoning behind it.
> 
> Dropping refcounting and move to free threading would completely break  
> all C modules so they would have to be rewritten and would make the  
> CPython API much more complex and integration with C libraries hard.  
> That's why no one took it seriously. Think like this, breaking all c  
> modules would make CPython as usable as haskell :), or just look at  
> the number of libraries not available right now for Python 3.0.
> 
> It is not some retarded choice made by GvR, but a pragmatic one.  
> Python as a language used by millions of people can't completely  
> change semantics from version to version.

While I understand the reasoning above, I don't agree with the last
paragraph. Switching to a different GC doesn't lead to that huge a
change in semantics. There are subtle difference about finalizers (see
the blog posts about that:

http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-1.html
http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-2.html

It definitely leads to a different C API, of course. Thus your above
points remain valid.

Cheers,

Carl Friedrich


From arigo at tunes.org  Sat Jan  3 15:12:44 2009
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 3 Jan 2009 15:12:44 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<20081223150113.GA1522@code0.codespeak.net>
	<a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>
	<20090102132338.GA17873@code0.codespeak.net>
	<a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>
Message-ID: <20090103141244.GA6999@code0.codespeak.net>

Hi,

Your writing style is too annoying to answer properly :-(
I will not try to answer all of the points you mention;
instead, let's just point out (as others have done) that
the picture is more complex that you suppose, because it
includes a lot of aspects that you completely ignore.  I
know definitely that with changes to the semantics of the
language, Python could be made seriously faster.  PyPy is
not about this; it's a collection of people who either
don't care too much about performance, or believe that
even with these semantics Python *can* be made seriously
fast -- it's just harder, but not impossible.


A bientot,

Armin.


From arigo at tunes.org  Sat Jan  3 16:33:07 2009
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 3 Jan 2009 16:33:07 +0100
Subject: [pypy-dev] GIL removal in PyPy (was: Re: Threaded
	interpretation)
In-Reply-To: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
References: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
Message-ID: <20090103153306.GA13032@code0.codespeak.net>

Hi Paolo,

On Sat, Jan 03, 2009 at 07:18:22AM +0100, Paolo Giarrusso wrote:
> > Given enough interest we can implement full GIL-free
> > multithreading the slow way: by going over most features of the
> > interpreter and thinking.
> 
> That'll take more time now than it would have taken to do it in the first place.

Yes, so we probably won't do it (except if we can find another
orthogonal technique to try, i.e. a separable way, which is only at the
state of vague discussion so far).

> > Note that it would also be possible to do
> > that in CPython (but it would be even more work).
> 
> That has been proven to be false :-).

Wrong.  For example, Jython is such an example, showing that we "can" do
it even in C (becauce anything possible in Java is possible in C, given
enough efforts).  I know it has been tried several times over the course
of the CPython development, but never succeeded.  That's why I'm saying
it would be even more work in CPython than in PyPy.


A bientot,

Armin.


From p.giarrusso at gmail.com  Sat Jan  3 19:36:29 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 3 Jan 2009 19:36:29 +0100
Subject: [pypy-dev] GIL removal in PyPy (was: Re: Threaded
	interpretation)
In-Reply-To: <20090103153306.GA13032@code0.codespeak.net>
References: <a4d0c86c0901022218y5ef828f6ja00b04fd0b4da0d7@mail.gmail.com>
	<20090103153306.GA13032@code0.codespeak.net>
Message-ID: <a4d0c86c0901031036x44292f0bp4d5d9892dfa6e577@mail.gmail.com>

Hi Armin,

On Sat, Jan 3, 2009 at 16:33, Armin Rigo <arigo at tunes.org> wrote:
>> > Note that it would also be possible to do
>> > that in CPython (but it would be even more work).

>> That has been proven to be false :-).

> Wrong.  For example, Jython is such an example, showing that we "can" do
> it even in C (becauce anything possible in Java is possible in C, given
> enough efforts).  I know it has been tried several times over the course
> of the CPython development, but never succeeded.  That's why I'm saying
> it would be even more work in CPython than in PyPy.

Obviously what can be done in Java can be done in C, but atomic
refcount manipulation is too expensive, so that, again, would require
dropping refcounting; in fact Jython has no refcounting.

Regards
-- 
Paolo Giarrusso


From p.giarrusso at gmail.com  Sun Jan  4 01:23:12 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sun, 4 Jan 2009 01:23:12 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <20090103141244.GA6999@code0.codespeak.net>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<20081223150113.GA1522@code0.codespeak.net>
	<a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>
	<20090102132338.GA17873@code0.codespeak.net>
	<a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>
	<20090103141244.GA6999@code0.codespeak.net>
Message-ID: <a4d0c86c0901031623h22875581i5013dea57ef7bce7@mail.gmail.com>

On Sat, Jan 3, 2009 at 15:12, Armin Rigo <arigo at tunes.org> wrote:
> Hi,

> Your writing style is too annoying to answer properly :-(

Sorry for that, I don't usually do it. I'm trying to do advocacy of my
position, but sometimes I realize there's a subtle boundary with
flaming; and most of that flaming is actually addressed to CPython
developers.

> I will not try to answer all of the points you mention;
> instead, let's just point out (as others have done) that
> the picture is more complex that you suppose, because it
> includes a lot of aspects that you completely ignore.

Tradeoffs are always involved, but in lots of cases, in the Python
community, misaddressed concerns about simplicity of the Python VM,
and ignorance of literature on Virtual Machines, lead to mistakes. And
I think we mostly agree on the facts underlying this statement, even
if we express that with different harshness.

I don't claim to be an expert of VM development, not at all.
But since there is a big disagreement between what I learned during 3
months of lessons (which was a first course on VM development) and
CPython, either one is wrong; my professor suggests that the Python
side is wrong and that implementers of Smalltalk, Self, and other
languages have some point. And all the evidence I could find
(including your project) agrees with that, on topics like:

- threading in the interpreter
- garbage collection vs reference counting
- tagged pointers
- hidden classes (i.e. V8 maps, which mostly date back to the early '90s).

If anybody is interested, this is the homepage of the course, with the
needed references (most of the links to papers are not directly
accessible, but the papers can be downloaded from other sources):

http://www.daimi.au.dk/~amoeller/VM/

== __DEL__ SEMANTICS ===

> I
> know definitely that with changes to the semantics of the
> language, Python could be made seriously faster.

> PyPy is
> not about this; it's a collection of people who either
> don't care too much about performance, or believe that
> even with these semantics Python *can* be made seriously
> fast -- it's just harder, but not impossible.

Just to make sure: I don't advocate changing semantics which have
value for the programmer; I advocate changing semantics when they have
problems anyway, like for this case, or for the int/long distinction,
which created problems both for tagged integers and for implementers.

Those two posts pointed out by Carl are interesting:

http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-1.html
http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-2.html

I'm interested about the costs of guaranteeing destructor calls in
topological order, I might reconsider my position on that particular
point; however, "topological-order-unless-a-cycle-is-found" is IMHO a
worse semantics than "no-topological-order", and that's what actually
matters, because it complicates implementation without being easier to
use correctly, because using fields requires proving that the object
cannot be part of a cycle, and that requires in turn an analysis of
the internals of other objects; this shouldn't be required to the
programmer since data hiding mostly forbids relying on this.

It would maybe be more useful to implement "reverse topological order"
and to clear all pointers to children of an object, to make sure that
nobody tries to rely on undefined behaviour, at least for Java, but
maybe also for Python.

=== SUMMARY ===
So, the full Python semantics (i.e. __del__ is like a C++ destructor)
are surely easier to use than Java/.NET finalizers, and are safe,
because one cannot get NULL pointers; but they have for the huge
problem of memory leaks; I'm undecided about
"topological-order-unless-a-cycle-is-found"; Java finalizers are
something that nobody should use; Python weakrefs still allow
resurrection, which causes slowdown as documented by Boehm; Java
weakrefs involve a fair amount of boilerplate code and they also
require the programmer to decide where the reference queue should be
polled, but at least they are quite difficult (if not impossible) to
misuse.

Regards
-- 
Paolo Giarrusso


From cfbolz at gmx.de  Sun Jan  4 11:19:13 2009
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 04 Jan 2009 11:19:13 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re:
 compiler	optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>	<494EC7EE.7000205@gmail.com>	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>	<20081223150113.GA1522@code0.codespeak.net>	<a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>	<20090102132338.GA17873@code0.codespeak.net>
	<a4d0c86c0901022059v7821bf13n481ff2839a1384d2@mail.gmail.com>
Message-ID: <49608D21.5080301@gmx.de>

Hi Paolo,

Happy new year!

I am getting back to one of the points that Anto has made which you
dismissed. I still think that it is the central one to explain why we
don't think that threaded interpretation is the panacea that you think
it is.

Paolo Giarrusso wrote:
> On Fri, Jan 2, 2009 at 14:23, Armin Rigo <arigo at tunes.org> wrote:
>> On Thu, Dec 25, 2008 at 12:42:18AM +0100, Paolo Giarrusso wrote:
>>> If I'll want to try something without refcounting, I'll guess I'd
>>>  turn to PyPy, but don't hold your breath for that. The fact that
>>>  indirect threading didn't work, that you're 1.5-2x slower than 
>>> CPython, and that you store locals in frame objects, they all 
>>> show that the abstraction overhead of the interpret is too high.
> 
>> True, but a 1.5x slowdown is not a big deal on many application; 
>> the blocker is mostly elsewhere.
> 
> Let's say 1.5x * 2x = 3x, since CPython is not as fast as it could 
> be, because of refcounting for instance. The 2x is taken from PyVM 
> performance reports (see http://www.python.org/dev/implementations/).
>  And for PyPy a slower interpreter means a slower JIT output, isn't 
> it? See below.
> 
> Also, according to CS literature, interpreter performance makes more
>  difference than JIT. According to the paper about efficient 
> interpreters I'm mentioning endless times, an inefficient interpreter
>  is 1000x slower than C, while an efficient one is only 10x slower. 
> Python is not that slow, but what you wrote about Psyco seems to 
> imply that there is still lots of room for improvement.

I finally looked at the paper. I think that one of the basic premises of
the paper does not apply to Python. This is the premise that a VM
instruction itself is a simple operation. For example the paper states:

"Most VM instructions are quite simple (e.g., add the top two numbers on
the stack), and can be implemented in a few native instructions plus
dispatch code. Complex VM instructions occur relatively rarely, and the
setup for them typically takes many simple VM instructions; [...]"

This is just nonsense in Python, and indeed in many other dynamic
languages as well. Take a simple fibonacci function:

>>> def fibo(n):
...     if n <= 2:
...         return 1
...     return fibo(n - 1) + fibo(n - 2)
...
>>> import dis dis.dis(fibo)
  2           0 LOAD_FAST                0 (n)
              3 LOAD_CONST               1 (2)
              6 COMPARE_OP               1 (<=)
              9 JUMP_IF_FALSE            8 (to 20)
             12 POP_TOP

  3          13 LOAD_CONST               2 (1)
             16 RETURN_VALUE
             17 JUMP_FORWARD             1 (to 21)
        >>   20 POP_TOP

  4     >>   21 LOAD_GLOBAL              0 (fibo)
             24 LOAD_FAST                0 (n)
             27 LOAD_CONST               2 (1)
             30 BINARY_SUBTRACT
             31 CALL_FUNCTION            1
             34 LOAD_GLOBAL              0 (fibo)
             37 LOAD_FAST                0 (n)
             40 LOAD_CONST               1 (2)
             43 BINARY_SUBTRACT
             44 CALL_FUNCTION            1
             47 BINARY_ADD
             48 RETURN_VALUE


Of the opcodes present here, some are indeed simple (like LOAD_CONST and
JUMP_FORWARD, etc). However, most of them perform potentially complex
operations and a disproportionate amount of time is spent in them. E.g.
COMPARE_OP, BINARY_SUBTRACT, BINARY_ADD, CALL_FUNCTION
are implemented using special methods, so are essentially method calls,
which need a full method lookup (which can take a lot of time, given
complex class hierarchies). Then the method can be implemented in Python
which leads to even more overhead (yes, one could try to speed up common
cases like integers for binary operations, but those turn out to not be
all that common in normal Python code).

Even a seemingly harmless opcode like LOAD_GLOBAL is rather complex to
implement. First you need to check the global dictionary and then
potentially the builtins dictionary. I guess to be really conclusive one
would have to do measurements about which bytecodes take the most time
at runtime. However, there is some evidence from PyPy that what I said
is right: Implementing a method cache gave PyPy much larger speedups
than any of the experiments we ever did for speeding up dispatching.

For these reasons I argue that the paper you keep citing vastly
exaggerates the effect of bytecode dispatch for Python, and I would
guess also for other dynamic languages. I am entering pure speculation
here, but I could guess that the reason the Perl interpreter appears so
inefficient in the paper is because it has similar characteristics.
Being a bit mean, the paper seems to be a bit of a self-fulfilling
prophecy: "Efficient interpreters are those that have a large number of
indirect branches.  Therefore, efficient interpreters will be helped by
our techniques. If our techniques don't help, well, then the interpreter
just wasn't efficient enough in the first place."

Note that I am not saying that techniques like a threaded interpreter
loop cannot lead to a faster Python interpreter. I am pretty sure that
they can help. My main point is that they might not help as much as you
seem to hope, and that one needs additional things to get really great
performance results (and with great I mean basically anything that is
more than an order of magnitude better).

[snip]

Cheers,

Carl Friedrich



From guillem at torroja.dmt.upm.es  Fri Jan  9 10:54:51 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Fri, 9 Jan 2009 10:54:51 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
Message-ID: <200901091054.51758.guillem@torroja.dmt.upm.es>

Hello everyone

My name is Guillem Borrell and I am a researcher in the Computational Fluid 
Dynamics Laboratory in the School of Aeronautics, Universidad Polit?cnica de 
Madrid.

This year I'm in charge of organising the Supercomputing Day. Its aim is to 
present in form of talks the recent and future developments in high performance 
computing in science and engineering. It holds five talks, open to anyone, on 
one day and we expect that it will take place on Agust the 1st of 2009.

Most of supercomputing software use static languages such as Fortran or C but 
postprocessing is usually done in dynamic  languages, mostly matlab. While it 
uses dynamic languages on a daily basis the scientific community has almost 
never taken a look at what the virtual machines can offer in terms of 
versatility, ease of use and performance. 

Pypy is now at the bleeding edge of dynamic language implementation and I think 
that some of you can give some insight on what dynamic languages (and pypy, of 
course) can offer in the field of supercomputing.  I know that pypy is not only 
focused in performance but our intention is to get an overview, not only to talk 
about a single project.

My question is ?Is some of the pypy developers interested in give such a talk? 
If the answer is yes  ther's still plenty of time to think about the details.

I personall think that pypy is the most mind challenging software project and 
this can be a little effort to present some of its achievements to a broader 
audience.

We have two confirmed talks at the moment.  The first one will be on future 
microprocessor architecture development and the Cell architecture in particular 
and the second one will be about distributed file systems and Lustre.

Sincerely yours.
-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From holger at merlinux.eu  Fri Jan  9 19:34:21 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 9 Jan 2009 19:34:21 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901091054.51758.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
Message-ID: <20090109183421.GA2219@trillke.net>

Hi Guillemi!

thanks for mailing - this definitely sounds interesting to me.  
I discussed earlier today with Maciej Fijalkowski who is also
interested to come.  So let's stay in contact and discuss details. 

cheers,
holger


On Fri, Jan 09, 2009 at 10:54 +0100, Guillem Borrell i Nogueras wrote:
> Hello everyone
> 
> My name is Guillem Borrell and I am a researcher in the Computational Fluid 
> Dynamics Laboratory in the School of Aeronautics, Universidad Polit?cnica de 
> Madrid.
> 
> This year I'm in charge of organising the Supercomputing Day. Its aim is to 
> present in form of talks the recent and future developments in high performance 
> computing in science and engineering. It holds five talks, open to anyone, on 
> one day and we expect that it will take place on Agust the 1st of 2009.
> 
> Most of supercomputing software use static languages such as Fortran or C but 
> postprocessing is usually done in dynamic  languages, mostly matlab. While it 
> uses dynamic languages on a daily basis the scientific community has almost 
> never taken a look at what the virtual machines can offer in terms of 
> versatility, ease of use and performance. 
> 
> Pypy is now at the bleeding edge of dynamic language implementation and I think 
> that some of you can give some insight on what dynamic languages (and pypy, of 
> course) can offer in the field of supercomputing.  I know that pypy is not only 
> focused in performance but our intention is to get an overview, not only to talk 
> about a single project.
> 
> My question is ?Is some of the pypy developers interested in give such a talk? 
> If the answer is yes  ther's still plenty of time to think about the details.
> 
> I personall think that pypy is the most mind challenging software project and 
> this can be a little effort to present some of its achievements to a broader 
> audience.
> 
> We have two confirmed talks at the moment.  The first one will be on future 
> microprocessor architecture development and the Cell architecture in particular 
> and the second one will be about distributed file systems and Lustre.
> 
> Sincerely yours.
> -- 
> guillem
> 
> Guillem Borrell i Nogueras
> Laboratorio de Mec?nica de Fluidos Computacional
> Escuela T?cnica Superior de Ingenieros Aeron?uticos
> guillem at torroja.dmt.upm.es
> Web: http://torroja.dmt.upm.es/guillem/blog
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 


From inhahe at gmail.com  Sun Jan 11 12:57:21 2009
From: inhahe at gmail.com (inhahe)
Date: Sun, 11 Jan 2009 06:57:21 -0500
Subject: [pypy-dev] fast
Message-ID: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>

With trace trees, hidden classes, aggressive type speculation, etc.,
javascript engines run code about as fast as unoptimized C.
And I know that Python isn't any more intractable than Javascript
because pyjamas automatically translates from one to the other.

So however you're doing it now.. you're doing it wrong. ;)

If CPython won't do it, maybe you will..

HTH,
inhahe


From fijall at gmail.com  Sun Jan 11 13:26:22 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 11 Jan 2009 13:26:22 +0100
Subject: [pypy-dev] fast
In-Reply-To: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
Message-ID: <693bc9ab0901110426s1476ff29yd78e84009219e1d2@mail.gmail.com>

We had this discussion couple of times already, but I'll try to recap.

On Sun, Jan 11, 2009 at 12:57 PM, inhahe <inhahe at gmail.com> wrote:
> With trace trees, hidden classes, aggressive type speculation, etc.,
> javascript engines run code about as fast as unoptimized C.
> And I know that Python isn't any more intractable than Javascript
> because pyjamas automatically translates from one to the other.

Well, wrong:

1. pyjamas is translating python syntax to js syntax (even not all of
them). Which mean mostly that semantics are python. That said, you
cannot run the same program under python and under js after pyjamas
and expect the same result. Simple example would be as follow:

d[index]

In python, if d is a dictionary, this will raise KeyError, if d is a
list this will raise IndexError, if d is a custom object, it'll try to
call it's __getitem__. In JS however (after pyjamas) this will simply
be translated to d[index], which will return undefined if index is not
there (and not raise any exception).

2. python is far more complex than javascript, someone tried to count
LOC of our python implementation vs our javascript implementation and
it was something in order of 6-10x more code for python. It's also far
more dynamic.

3. Psyco runs programs (let's say integer computations) as far as
unoptimized C. Our prototype JIT does as well and we're working hard
on improving things (in fact we work a lot on JIT these days).

>
> So however you're doing it now.. you're doing it wrong. ;)

Well, tell us how to do it better/faster? Your points are rather
shallow as it seems you didn't look into anything in detail. Look
deeper, maybe you can give us a useful advice.

>
> If CPython won't do it, maybe you will..

Thanks for trust :)

cheers,
fijal


From jewel at subvert-the-dominant-paradigm.net  Sun Jan 11 17:20:20 2009
From: jewel at subvert-the-dominant-paradigm.net (John Leuner)
Date: Sun, 11 Jan 2009 18:20:20 +0200
Subject: [pypy-dev] fast
In-Reply-To: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
Message-ID: <1231690820.13208.5.camel@cmalu.WAG54GS>

On Sun, 2009-01-11 at 06:57 -0500, inhahe wrote:
> With trace trees, hidden classes, aggressive type speculation, etc.,
> javascript engines run code about as fast as unoptimized C.

What benchmarks/tests have you seen that show this?

John




From p.giarrusso at gmail.com  Sun Jan 11 20:44:54 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sun, 11 Jan 2009 20:44:54 +0100
Subject: [pypy-dev] fast
In-Reply-To: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
Message-ID: <a4d0c86c0901111144s297a115p5ab76ab948c9fb0c@mail.gmail.com>

On Sun, Jan 11, 2009 at 12:57, inhahe <inhahe at gmail.com> wrote:
> With trace trees, hidden classes, aggressive type speculation, etc.,
> javascript engines run code about as fast as unoptimized C.

> And I know that Python isn't any more intractable than Javascript
> because pyjamas automatically translates from one to the other.

This is not a valid reasoning, as discussed in the other mail.

Having said that, in the end we all agree that even Python can get
that speed, the only thing we argue about is how complex is doing
that.

> So however you're doing it now.. you're doing it wrong. ;)

> If CPython won't do it, maybe you will..

I don't see the point of your mail, is it just trolling? IMHO, one can
only write "you're doing it wrong" if he's able to do better, and has
proven that in practice.

Btw, I'm not (yet) part of the PyPy project, and I've been discussing
the same issues as you. That doesn't make me agree with such mails.
-- 
Paolo Giarrusso


From p.giarrusso at gmail.com  Sun Jan 11 21:09:05 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sun, 11 Jan 2009 21:09:05 +0100
Subject: [pypy-dev] fast
In-Reply-To: <693bc9ab0901110426s1476ff29yd78e84009219e1d2@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
	<693bc9ab0901110426s1476ff29yd78e84009219e1d2@mail.gmail.com>
Message-ID: <a4d0c86c0901111209x3bbf3528uc0ec9e0e3f6c5a04@mail.gmail.com>

On Sun, Jan 11, 2009 at 13:26, Maciej Fijalkowski <fijall at gmail.com> wrote:
> We had this discussion couple of times already, but I'll try to recap.

> On Sun, Jan 11, 2009 at 12:57 PM, inhahe <inhahe at gmail.com> wrote:
>> With trace trees, hidden classes, aggressive type speculation, etc.,
>> javascript engines run code about as fast as unoptimized C.
>> And I know that Python isn't any more intractable than Javascript
>> because pyjamas automatically translates from one to the other.

> Well, wrong:

> 1. pyjamas is translating python syntax to js syntax (even not all of
> them). Which mean mostly that semantics are python. That said, you
> cannot run the same program under python and under js after pyjamas
> and expect the same result. Simple example would be as follow:

> d[index]

> In python, if d is a dictionary, this will raise KeyError, if d is a
> list this will raise IndexError, if d is a custom object, it'll try to
> call it's __getitem__. In JS however (after pyjamas) this will simply
> be translated to d[index], which will return undefined if index is not
> there (and not raise any exception).

You have a point on semantic differences. However, such a problem does
not make any of the two implementations faster than the other, and the
error handling can be as slow as needed to make the common case fast.

> 2. python is far more complex than javascript, someone tried to count
> LOC of our python implementation vs our javascript implementation and
> it was something in order of 6-10x more code for python.

Well, I expect your Python implementation to be far more optimized
than your Javascript one, isn't it? And the 80-20% (or 90-10%) rule
still applies. If the language allows fancier things, one would
probably focus on standard practices. Take __slots__. Once one
implements V8 maps, __slots__ becomes useless, and it's implementation
would simply reuse the one for normal objects.

Also, if extra complexity is needed for weird stuff, most of the times
it shouldn't affect the fast path. The existence of sys._getframe() is
one of the exceptions, and f_lasti a particularly bad one (it has
10-20% impact on execution runtime).

> It's also far
> more dynamic.

Can you explain why, or point to the archives where it was discussed?
I made a quick search but I couldn't find the right keywords to use.

> 3. Psyco runs programs (let's say integer computations) as far as
> unoptimized C. Our prototype JIT does as well and we're working hard
> on improving things (in fact we work a lot on JIT these days).

-- 
Paolo Giarrusso


From fijall at gmail.com  Sun Jan 11 21:16:15 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 11 Jan 2009 21:16:15 +0100
Subject: [pypy-dev] fast
In-Reply-To: <a4d0c86c0901111209x3bbf3528uc0ec9e0e3f6c5a04@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
	<693bc9ab0901110426s1476ff29yd78e84009219e1d2@mail.gmail.com>
	<a4d0c86c0901111209x3bbf3528uc0ec9e0e3f6c5a04@mail.gmail.com>
Message-ID: <693bc9ab0901111216s61787a7dybaeac54b9c430c27@mail.gmail.com>

On Sun, Jan 11, 2009 at 9:09 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Sun, Jan 11, 2009 at 13:26, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> We had this discussion couple of times already, but I'll try to recap.
>
>> On Sun, Jan 11, 2009 at 12:57 PM, inhahe <inhahe at gmail.com> wrote:
>>> With trace trees, hidden classes, aggressive type speculation, etc.,
>>> javascript engines run code about as fast as unoptimized C.
>>> And I know that Python isn't any more intractable than Javascript
>>> because pyjamas automatically translates from one to the other.
>
>> Well, wrong:
>
>> 1. pyjamas is translating python syntax to js syntax (even not all of
>> them). Which mean mostly that semantics are python. That said, you
>> cannot run the same program under python and under js after pyjamas
>> and expect the same result. Simple example would be as follow:
>
>> d[index]
>
>> In python, if d is a dictionary, this will raise KeyError, if d is a
>> list this will raise IndexError, if d is a custom object, it'll try to
>> call it's __getitem__. In JS however (after pyjamas) this will simply
>> be translated to d[index], which will return undefined if index is not
>> there (and not raise any exception).
>
> You have a point on semantic differences. However, such a problem does
> not make any of the two implementations faster than the other, and the
> error handling can be as slow as needed to make the common case fast.

I was just pointing out that argument is completely void since pyjamas
is not transforming semantics.

>
>> 2. python is far more complex than javascript, someone tried to count
>> LOC of our python implementation vs our javascript implementation and
>> it was something in order of 6-10x more code for python.
>
> Well, I expect your Python implementation to be far more optimized
> than your Javascript one, isn't it? And the 80-20% (or 90-10%) rule
> still applies. If the language allows fancier things, one would
> probably focus on standard practices. Take __slots__. Once one
> implements V8 maps, __slots__ becomes useless, and it's implementation
> would simply reuse the one for normal objects.

That's true.

>
> Also, if extra complexity is needed for weird stuff, most of the times
> it shouldn't affect the fast path. The existence of sys._getframe() is
> one of the exceptions, and f_lasti a particularly bad one (it has
> 10-20% impact on execution runtime).
>
>> It's also far
>> more dynamic.
>
> Can you explain why, or point to the archives where it was discussed?
> I made a quick search but I couldn't find the right keywords to use.

Well for various reasons. Take for example descriptor semantics or
metaclasses. I'm not up to discuss exactly which language is more
dynamic, because this is a valid point for static compilation. The
point is that python is more complex and require more effort to make
it fast.

cheers,
fijal


From jewel at subvert-the-dominant-paradigm.net  Sun Jan 11 22:02:14 2009
From: jewel at subvert-the-dominant-paradigm.net (John Leuner)
Date: Sun, 11 Jan 2009 23:02:14 +0200
Subject: [pypy-dev] fast
In-Reply-To: <a4d0c86c0901111206o1e3a4dcya4af6cd65a542aec@mail.gmail.com>
References: <da776a8c0901110357v36017268r76dcef55cc1d74b4@mail.gmail.com>
	<1231690820.13208.5.camel@cmalu.WAG54GS>
	<a4d0c86c0901111206o1e3a4dcya4af6cd65a542aec@mail.gmail.com>
Message-ID: <1231707734.13208.16.camel@cmalu.WAG54GS>

On Sun, 2009-01-11 at 21:06 +0100, Paolo Giarrusso wrote:
> On Sun, Jan 11, 2009 at 17:20, John Leuner
> <jewel at subvert-the-dominant-paradigm.net> wrote:
> > On Sun, 2009-01-11 at 06:57 -0500, inhahe wrote:
> >> With trace trees, hidden classes, aggressive type speculation, etc.,
> >> javascript engines run code about as fast as unoptimized C.
> >
> > What benchmarks/tests have you seen that show this?
> 
> That's first a claim from this post, from TraceMonkey author
> http://andreasgal.com/2008/08/22/tracing-the-web/
> 
> This is the benchmark saying that:
> http://shaver.off.net/diary/2008/08/22/the-birth-of-a-faster-monkey/
> 
> And TraceMonkey is quite young.

Quoting from that site:

"The goal of the TraceMonkey project ? which is still in its early
stages ? is to take JavaScript performance to another level, where
instead of competing against other interpreters, we start to compete
against native code. Even with this very, very early version we?re
already seeing some promising results: a simple ?for loop? is getting
close to unoptimized gcc:"

This claim about unoptimized C is only about a "for loop". We all know
that javascript and python execution involves much more than simple for
loops.

John






From guillem at torroja.dmt.upm.es  Mon Jan 12 12:10:50 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Mon, 12 Jan 2009 12:10:50 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <20090109183421.GA2219@trillke.net>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
Message-ID: <200901121210.50200.guillem@torroja.dmt.upm.es>

Hi again

Let's discuss the details.

I'll try to explain why I've thought about pypy when planning the conference 
sessions.

My work as Computational Fluid Dynamics researcher is intimately related to 
supercomputing for obvious reasons. Most of applications we work on are fine 
tuned codes with pieces that are more than twenty years old.  They are rock 
solid, fortan implemented, and run for hours in clusters of thousand of 
computing nodes.

Since the last couple of years computer architectures are becoming more and more 
complex. I'm playing with the cell processor lately and that little bastard is 
causing me real pain. While programming is easier every day, supercomputing is 
harder and harder.  Think about an arhitecture like the roadrunner, AMD Opteron 
PPU with PowerPC SPU... Two assemblers in one chip!

Talking with Stanley Ahalt (Ohio Supercomputing Center) about a year ago he 
called that the "software gap". In computing, as times goes, low performance is 
easier but high performance is harder. And that gap gets wider. Platform SDK are 
helpful but they are not a huge leap.

I've always thought that virtual machines could help supercomputing like they 
have helped grid and cloud computing.  This is the point where I need someone to 
proof if I am wright or wrong.  Pypy is the most versatile, albeit complex, 
dynamic language implementation. I've been following the project during the last 
year and a half or so and I am impressed. I've thought that you could have a 
vision on how interpreted languages and virtual machines can help managing 
complexity.

In addition, most of postprocessing tools are written in matlab, an interpreted 
language.  Running not-so-high performance tasks in a workstation efficiently is 
sometimes as important as running a simulation in a 12000 node supercomputer. It 
yould be nice if someone would remind the audience that matlab is not the only 
suitable (or best) tool for that job.

I hope that I managed to explain the topics I'm interested in.

Those are the speakers that accepted the invitation at the moment:

* J.M?. Cela, Computer Architecture Department, Universitat Polit?cnica de 
Catalunya and Barcelona Supercomputing Center. Future supercomputers, CFD 
applications and the Cell processor.

* Sun Microsystems. Distributed Filesystems and Lustre.

* S. Hoyas. School of Aeronautics. Universitat Polit?cnica de Val?ncia. Running 
a supercompuingr simulation in more than 2000 processors.

I'm very interested in your comments.

cheers.

guillem

El Friday 09 January 2009 19:34:21 holger krekel escribi?:
> Hi Guillemi!
>
> thanks for mailing - this definitely sounds interesting to me.
> I discussed earlier today with Maciej Fijalkowski who is also
> interested to come.  So let's stay in contact and discuss details.
>
> cheers,
> holger
>
> On Fri, Jan 09, 2009 at 10:54 +0100, Guillem Borrell i Nogueras wrote:
> > Hello everyone
> >
> > My name is Guillem Borrell and I am a researcher in the Computational
> > Fluid Dynamics Laboratory in the School of Aeronautics, Universidad
> > Polit?cnica de Madrid.
> >
> > This year I'm in charge of organising the Supercomputing Day. Its aim is
> > to present in form of talks the recent and future developments in high
> > performance computing in science and engineering. It holds five talks,
> > open to anyone, on one day and we expect that it will take place on Agust
> > the 1st of 2009.
> >
> > Most of supercomputing software use static languages such as Fortran or C
> > but postprocessing is usually done in dynamic  languages, mostly matlab.
> > While it uses dynamic languages on a daily basis the scientific community
> > has almost never taken a look at what the virtual machines can offer in
> > terms of versatility, ease of use and performance.
> >
> > Pypy is now at the bleeding edge of dynamic language implementation and I
> > think that some of you can give some insight on what dynamic languages
> > (and pypy, of course) can offer in the field of supercomputing.  I know
> > that pypy is not only focused in performance but our intention is to get
> > an overview, not only to talk about a single project.
> >
> > My question is ?Is some of the pypy developers interested in give such a
> > talk? If the answer is yes  ther's still plenty of time to think about
> > the details.
> >
> > I personall think that pypy is the most mind challenging software project
> > and this can be a little effort to present some of its achievements to a
> > broader audience.
> >
> > We have two confirmed talks at the moment.  The first one will be on
> > future microprocessor architecture development and the Cell architecture
> > in particular and the second one will be about distributed file systems
> > and Lustre.
> >
> > Sincerely yours.
> > --
> > guillem
> >
> > Guillem Borrell i Nogueras
> > Laboratorio de Mec?nica de Fluidos Computacional
> > Escuela T?cnica Superior de Ingenieros Aeron?uticos
> > guillem at torroja.dmt.upm.es
> > Web: http://torroja.dmt.upm.es/guillem/blog
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From p.giarrusso at gmail.com  Mon Jan 12 12:50:35 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 12 Jan 2009 12:50:35 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901121210.50200.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
Message-ID: <a4d0c86c0901120350p296e5a2cn85c4d8634c42f6c9@mail.gmail.com>

Hi, first I'd like to qualify myself as a student of Virtual Machine
implementations, not working (yet) on PyPy itself, and aware of some
HPC issues at a basic level. Still, I'd like to help pinpointing the
discussion.

On Mon, Jan 12, 2009 at 12:10, Guillem Borrell i Nogueras
<guillem at torroja.dmt.upm.es> wrote:
> Hi again

> Let's discuss the details.

> I'll try to explain why I've thought about pypy when planning the conference
> sessions.

> My work as Computational Fluid Dynamics researcher is intimately related to
> supercomputing for obvious reasons. Most of applications we work on are fine
> tuned codes with pieces that are more than twenty years old.  They are rock
> solid, fortan implemented, and run for hours in clusters of thousand of
> computing nodes.

> Since the last couple of years computer architectures are becoming more and more
> complex. I'm playing with the cell processor lately and that little bastard is
> causing me real pain. While programming is easier every day, supercomputing is
> harder and harder.  Think about an arhitecture like the roadrunner, AMD Opteron
> PPU with PowerPC SPU... Two assemblers in one chip!

> Talking with Stanley Ahalt (Ohio Supercomputing Center) about a year ago he
> called that the "software gap". In computing, as times goes, low performance is
> easier but high performance is harder. And that gap gets wider. Platform SDK are
> helpful but they are not a huge leap.

> I've always thought that virtual machines could help supercomputing like they
> have helped grid and cloud computing.  This is the point where I need someone to
> proof if I am wright or wrong.  Pypy is the most versatile, albeit complex,
> dynamic language implementation. I've been following the project during the last
> year and a half or so and I am impressed. I've thought that you could have a
> vision on how interpreted languages and virtual machines can help managing
> complexity.

> In addition, most of postprocessing tools are written in matlab, an interpreted
> language.  Running not-so-high performance tasks in a workstation efficiently is
> sometimes as important as running a simulation in a 12000 node supercomputer. It
> yould be nice if someone would remind the audience that matlab is not the only
> suitable (or best) tool for that job.

This is IMHO an issue with the expressivity of Matlab & Python. There
are Python libraries for that, but Python has not a Domain-Specific
syntax I guess - you'd need to spell out method names sometimes,
instead of using / and ./. Slicing is something already existing,
though.
However, JIT-compiled Python would have the huge advantage to make
also loops more efficient, instead of forcing the user to write all
loops in terms of matrix parallel operations. That was the biggest
slowdown in Matlab development I experienced. And I remember fixing
that in somebody else's program, getting from 12 hours to 1 minute of
runtime. 720x speedup. Interpreters can be much faster than that, even
CPython is already faster in that area.

The advantage of vectorized loops is that they easily use optimized
BLAS routines, maybe SSE/Cell processor based ones.

> I'm very interested in your comments.

Now, the first question is: do you have Python to be faster than
Matlab or faster than Fortran? VMs can be faster than C++ (for
instance, static C++ can't inline virtual methods).

That makes a huge difference. I'm unaware of why one can't use its
Fortran source on Cell. OK, I can wildly guess that having two
specialized microprocessors, one might like having an automatic
parallelizer to split certain operations to one and certain to
another? Can you make some better example?

The obvious ones are automatic vectorization and automatic
parallelization, but I'm not aware of any VM-specific research on that
topic. And automatic parallelization is a quite difficult topic
anyway, as far as I know.

In other words: is there any special advantage given by adaptive
optimization (even profile-based one) that static optimizations (like
the ones done by ICC) cannot match?
None is obvious for vectorization, automatic tuning of sizes comes to
mind for auto parallelization.

In general it is well known that the more a language is high-level,
the more informations the compiler has to optimize it, but also the
more the language has fancy features not trivial to optimize.
Actually, Fortran is better than C exactly because is more high-level.
Most C code assumes for instance GCC option -fno-strict-aliasing,
which is contrary to the language semantics and forbids many
interesting optimizations.

See this website for information (note: I found the link somewhere
else, but be careful that according to Google&Firefox, the server is
under control of hackers and is spreading viruses; I run Linux and I'm
safe, YMMV):
http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_aliasing.html

Having said that, the point is to understand which optimizations do
you need to perform manually and could be performed automatically by a
VM? Note that your Fortran compiler probably has a far better static
code optimizer. If you write plain Fortran code in Python, it's going
to be much slower (like if no optimization were present) until
dataflow analysis, register allocation, instruction scheduling etc.
will be implemented in PyPy, after all the rest is finished. It's just
a matter of implementation costs, but it is huge.
Where VM shine is when they can do optimizations unavailable to static
compilers, like adaptive optimizations. Inlining of virtual method is
an example, but automatic prefetching from memory into cache (by
adding SSE prefetch instructions) is another on which research has
been done, for instance, just to make an example.

Regards
-- 
Paolo Giarrusso


From fijall at gmail.com  Mon Jan 12 17:05:15 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 12 Jan 2009 17:05:15 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <a4d0c86c0901120350p296e5a2cn85c4d8634c42f6c9@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<a4d0c86c0901120350p296e5a2cn85c4d8634c42f6c9@mail.gmail.com>
Message-ID: <693bc9ab0901120805l74e95faan2d0117fe826a2021@mail.gmail.com>

On Mon, Jan 12, 2009 at 12:50 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> Hi, first I'd like to qualify myself as a student of Virtual Machine
> implementations, not working (yet) on PyPy itself, and aware of some
> HPC issues at a basic level. Still, I'd like to help pinpointing the
> discussion.
> ...

Sorry, but I would like to point out that this thread is about
possible conference appearance, not general discussion about HPC.

Things that you mentioned are interesting, but off topic and are
possibly well known to Guillem I suppose.

Paolo, please feel free to use pypy-dev as a discussion place,
especially regarding implementing dynamic VMs, but we would be
grateful if you stick to some guidelines:

* Stick to the topic of current discussion.

* Discuss one issue at a time, and not a complete flow of buzzwords.

* Try to provide more sources/details rather than general statements.

Thanks and cheers,
fijal


From van.lindberg at gmail.com  Mon Jan 12 18:12:39 2009
From: van.lindberg at gmail.com (VanL)
Date: Mon, 12 Jan 2009 11:12:39 -0600
Subject: [pypy-dev] Non-circular path through pypy module imports
Message-ID: <gkftm6$qfk$1@ger.gmane.org>

Hello,

I have been going through the pypy sources recently and I wanted to start from
the leaf modules first (in terms of import dependencies)... but there don't
appear to be any leaves.

For example, starting with baseobjspace in the interpreter, it has 30 or so
non-stdlib imports, many of them delayed by placing them in functions. Even
something like rarithmetic eventually pulls in most of the lltype hierarchy. The
closest I found to a leaf module was extregistry.py.

Is there a reason for the circularities that I just don't get? If not, why are
there so many?

Thanks,

Van



From holger at merlinux.eu  Tue Jan 13 13:31:07 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 13 Jan 2009 13:31:07 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901121210.50200.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
Message-ID: <20090113123107.GL2219@trillke.net>

Hi Guillem! 

On Mon, Jan 12, 2009 at 12:10 +0100, Guillem Borrell i Nogueras wrote:
> Hi again
> 
> Let's discuss the details.
> 
> I'll try to explain why I've thought about pypy when planning the conference 
> sessions.
> 
> My work as Computational Fluid Dynamics researcher is intimately related to 
> supercomputing for obvious reasons. Most of applications we work on are fine 
> tuned codes with pieces that are more than twenty years old.  They are rock 
> solid, fortan implemented, and run for hours in clusters of thousand of 
> computing nodes.
> 
> Since the last couple of years computer architectures are becoming more and more 
> complex. I'm playing with the cell processor lately and that little bastard is 
> causing me real pain. While programming is easier every day, supercomputing is 
> harder and harder.  Think about an arhitecture like the roadrunner, AMD Opteron 
> PPU with PowerPC SPU... Two assemblers in one chip!

Could you give examples on the concrete challenges you face?

> Talking with Stanley Ahalt (Ohio Supercomputing Center) about a year ago he 
> called that the "software gap". In computing, as times goes, low performance is 
> easier but high performance is harder. And that gap gets wider. Platform SDK are 
> helpful but they are not a huge leap.
> 
> I've always thought that virtual machines could help
> supercomputing like they have helped grid and cloud
> computing.  This is the point where I need someone to proof
> if I am wright or wrong.  Pypy is the most versatile, albeit
> complex, dynamic language implementation. I've been
> following the project during the last year and a half or so
> and I am impressed. I've thought that you could have a
> vision on how interpreted languages and virtual machines can
> help managing complexity.

The PyPy project is mostly concerned with reducing the complexity 
of writing efficient dynamic language interpreters. 
PyPy's JIT and its flexible backend implementation 
should help with bridging the software gap IIUC. 
It probably makes sense to stress and elaborate
on this point and to discuss examples in the talk. 

Managing the complexity of software deployment and integration
is not PyPy's primary concern but it provides some interesting
building blocks.  For example, i am interested in using using
PyPy's sandboxing and virtualization capabilities to build an
open "cloud at home" network: people would run a bare-bones
executable, configure & control CPU, RAM, Network, Disk
resources and access rights. On this network e.g. open source
developers or scientists could launch programs and
computations or start os-level virtual machines where
non-python code runs.  

> In addition, most of postprocessing tools are written in matlab, an interpreted 
> language.  Running not-so-high performance tasks in a workstation efficiently is 
> sometimes as important as running a simulation in a 12000 node supercomputer. It 
> yould be nice if someone would remind the audience that matlab is not the only 
> suitable (or best) tool for that job.

Right, I guess that something like Numpy combined with PyPy's JIT 
would could make a killer combination.  That's work, though. 

> I hope that I managed to explain the topics I'm interested in.

sure, let us know what you think makes most sense for
the audience - who will be the audience btw? 

thanks & best,

holger


 
> Those are the speakers that accepted the invitation at the moment:
> 
> * J.M?. Cela, Computer Architecture Department, Universitat Polit?cnica de 
> Catalunya and Barcelona Supercomputing Center. Future supercomputers, CFD 
> applications and the Cell processor.
> 
> * Sun Microsystems. Distributed Filesystems and Lustre.
> 
> * S. Hoyas. School of Aeronautics. Universitat Polit?cnica de Val?ncia. Running 
> a supercompuingr simulation in more than 2000 processors.
> 
> I'm very interested in your comments.

> cheers.
> 
> guillem
> 
> El Friday 09 January 2009 19:34:21 holger krekel escribi?:
> > Hi Guillemi!
> >
> > thanks for mailing - this definitely sounds interesting to me.
> > I discussed earlier today with Maciej Fijalkowski who is also
> > interested to come.  So let's stay in contact and discuss details.
> >
> > cheers,
> > holger
> >
> > On Fri, Jan 09, 2009 at 10:54 +0100, Guillem Borrell i Nogueras wrote:
> > > Hello everyone
> > >
> > > My name is Guillem Borrell and I am a researcher in the Computational
> > > Fluid Dynamics Laboratory in the School of Aeronautics, Universidad
> > > Polit?cnica de Madrid.
> > >
> > > This year I'm in charge of organising the Supercomputing Day. Its aim is
> > > to present in form of talks the recent and future developments in high
> > > performance computing in science and engineering. It holds five talks,
> > > open to anyone, on one day and we expect that it will take place on Agust
> > > the 1st of 2009.
> > >
> > > Most of supercomputing software use static languages such as Fortran or C
> > > but postprocessing is usually done in dynamic  languages, mostly matlab.
> > > While it uses dynamic languages on a daily basis the scientific community
> > > has almost never taken a look at what the virtual machines can offer in
> > > terms of versatility, ease of use and performance.
> > >
> > > Pypy is now at the bleeding edge of dynamic language implementation and I
> > > think that some of you can give some insight on what dynamic languages
> > > (and pypy, of course) can offer in the field of supercomputing.  I know
> > > that pypy is not only focused in performance but our intention is to get
> > > an overview, not only to talk about a single project.
> > >
> > > My question is ?Is some of the pypy developers interested in give such a
> > > talk? If the answer is yes  ther's still plenty of time to think about
> > > the details.
> > >
> > > I personall think that pypy is the most mind challenging software project
> > > and this can be a little effort to present some of its achievements to a
> > > broader audience.
> > >
> > > We have two confirmed talks at the moment.  The first one will be on
> > > future microprocessor architecture development and the Cell architecture
> > > in particular and the second one will be about distributed file systems
> > > and Lustre.
> > >
> > > Sincerely yours.
> > > --
> > > guillem
> > >
> > > Guillem Borrell i Nogueras
> > > Laboratorio de Mec?nica de Fluidos Computacional
> > > Escuela T?cnica Superior de Ingenieros Aeron?uticos
> > > guillem at torroja.dmt.upm.es
> > > Web: http://torroja.dmt.upm.es/guillem/blog
> > > _______________________________________________
> > > pypy-dev at codespeak.net
> > > http://codespeak.net/mailman/listinfo/pypy-dev
> 
> -- 
> guillem
> 
> Guillem Borrell i Nogueras
> Laboratorio de Mec?nica de Fluidos Computacional
> Escuela T?cnica Superior de Ingenieros Aeron?uticos
> guillem at torroja.dmt.upm.es
> Web: http://torroja.dmt.upm.es/guillem/blog
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 


From guillem at torroja.dmt.upm.es  Tue Jan 13 16:17:53 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Tue, 13 Jan 2009 16:17:53 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <20090113123107.GL2219@trillke.net>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<20090113123107.GL2219@trillke.net>
Message-ID: <200901131617.54395.guillem@torroja.dmt.upm.es>

Hi!

El Tuesday 13 January 2009 13:31:07 holger krekel escribi?:

>
> Could you give examples on the concrete challenges you face?
>

Multicore architectures are killing supercomputing because two kinds of 
parallelism are needed at the same time. A couple of years ago the computer 
architecture departments started suggesting multiple OpenMP threads in every MPI 
processes. This required a huge amount of work and never worked as expected 
(maybe compilers are to blame?).

Now the trend seems to be using MPI for processes and a superscalar compiler 
that uses all the cores and SPU for just a few kind of operations. This is not 
more useful than using vectorized versions of Blas, levels 1 and 2.

Some years ago I started experimenting with python and MPI. I coded a very 
limited MPI bindings collection for Python and launched python interpreters as 
MPI processes. Instantly thought that a JIT-accelerated, multicore-aware 
interpreter and a fast MPI-like message passing library would lower the 
complexity of using parallel computers. That would not give the highest 
performance but one has to be realistic. There's no need to distribute a VM, 
message passing is not that hard.

Maybe Java would be more suitable but I am contrary to Java from an almost 
religious point of view. I just hate it.

That's what I mean with narrowing the software gap. I've think that pypy has all 
the blocks to build this kind of interpeter, but maybe I am wrong. This is 
because I cant' really understand what pypy is ;) (maybe that's the reason why I 
find it so interesting).

>
> The PyPy project is mostly concerned with reducing the complexity
> of writing efficient dynamic language interpreters.
> PyPy's JIT and its flexible backend implementation
> should help with bridging the software gap IIUC.
> It probably makes sense to stress and elaborate
> on this point and to discuss examples in the talk.
>
> Managing the complexity of software deployment and integration
> is not PyPy's primary concern but it provides some interesting
> building blocks.  For example, i am interested in using using
> PyPy's sandboxing and virtualization capabilities to build an
> open "cloud at home" network: people would run a bare-bones
> executable, configure & control CPU, RAM, Network, Disk
> resources and access rights. On this network e.g. open source
> developers or scientists could launch programs and
> computations or start os-level virtual machines where
> non-python code runs.
>
> > In addition, most of postprocessing tools are written in matlab, an
> > interpreted language.  Running not-so-high performance tasks in a
> > workstation efficiently is sometimes as important as running a simulation
> > in a 12000 node supercomputer. It yould be nice if someone would remind
> > the audience that matlab is not the only suitable (or best) tool for that
> > job.
>
> Right, I guess that something like Numpy combined with PyPy's JIT
> would could make a killer combination.  That's work, though.

I think that you'll have to start explaining what a JIT is. See the note on the 
audience below. I've been using Python+Numpy+Scipy+... as a complete replacement 
for matlab for the last two years. I suggested all of my colleagues to do the 
same but they all believe that python is a toy language (!). I think that you 
could give a more authorized opinion on this.

> > I hope that I managed to explain the topics I'm interested in.
>
> sure, let us know what you think makes most sense for
> the audience - who will be the audience btw?

The talks will be in the school of aeronautics. Most of people in the audience 
will be engineers, mathematicians and physicists users of supercomputing without 
any academic background on Computer Science. 

Thany you so much!
-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From p.giarrusso at gmail.com  Tue Jan 13 19:04:24 2009
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 13 Jan 2009 19:04:24 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <693bc9ab0901120805l74e95faan2d0117fe826a2021@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<a4d0c86c0901120350p296e5a2cn85c4d8634c42f6c9@mail.gmail.com>
	<693bc9ab0901120805l74e95faan2d0117fe826a2021@mail.gmail.com>
Message-ID: <a4d0c86c0901131004t72f5c94eof5610330a29ee4ea@mail.gmail.com>

On Mon, Jan 12, 2009 at 17:05, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Jan 12, 2009 at 12:50 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>> Hi, first I'd like to qualify myself as a student of Virtual Machine
>> implementations, not working (yet) on PyPy itself, and aware of some
>> HPC issues at a basic level. Still, I'd like to help pinpointing the
>> discussion.

> Sorry, but I would like to point out that this thread is about
> possible conference appearance, not general discussion about HPC.

> Things that you mentioned are interesting, but off topic and are
> possibly well known to Guillem I suppose.

I introduced them thinking they were on topic, as general ideas to
maybe discuss in the talk. Or rather, to suggest discussing facts
about them in the talk.

== MORE REFERENCES ==
Anwyay, some googling found a couple of interesting papers on
(dynamic) auto-parallelization, which are for Java.

http://portal.acm.org/citation.cfm?id=859668
http://cat.inist.fr/?aModele=afficheN&cpsidt=991008

>From the abstracts, I expect the second to be more challenging to
transport to Python. Existing static automatic parallelizing compilers
are already not satisfactory, as pointed out by Guillem, but the 1st
paper seem to suggest that a JIT can better tune the introduced
parallelism.

Anyway, this seems still research and not production-level technology.
Is the latter available?
> Paolo, please feel free to use pypy-dev as a discussion place,
> especially regarding implementing dynamic VMs, but we would be
> grateful if you stick to some guidelines:

> * Stick to the topic of current discussion.

> * Discuss one issue at a time, and not a complete flow of buzzwords.

> * Try to provide more sources/details rather than general statements

Maciej, I first want to apologize if my mail was not indeed useful,
and if I seemed to use pypy-dev as a forum.
But I was trying to discuss the topic of the conference appearance,
given that the topics (of the talk itself) are probably going to be
"Ideas for a JIT for numerical computing in Python" and "Is/Will be
Python better than alternatives, as of today/in the future?". The
"future" part is where all the "buzzwords" came from, even if more
experience could have detailed some of them. And I did want to provide
an overview, so that one can then focus on specific points.

Given the interest and resource on HPC computing, implementing in a
JIT optimizations for static compilers which are suited to HPC could
be useful and could find enough resources, if the optimizations are
worth their cost, and given somebody willing to solve the formidable
challenges I expect of this idea.

Regards
-- 
Paolo Giarrusso


From renesd at gmail.com  Tue Jan 13 23:41:47 2009
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Wed, 14 Jan 2009 09:41:47 +1100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <20090113123107.GL2219@trillke.net>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<20090113123107.GL2219@trillke.net>
Message-ID: <64ddb72c0901131441w98b1e12mfb3d872ebc2c20fd@mail.gmail.com>

On Tue, Jan 13, 2009 at 11:31 PM, holger krekel <holger at merlinux.eu> wrote:
>
> Managing the complexity of software deployment and integration
> is not PyPy's primary concern but it provides some interesting
> building blocks.  For example, i am interested in using using
> PyPy's sandboxing and virtualization capabilities to build an
> open "cloud at home" network: people would run a bare-bones
> executable, configure & control CPU, RAM, Network, Disk
> resources and access rights. On this network e.g. open source
> developers or scientists could launch programs and
> computations or start os-level virtual machines where
> non-python code runs.
>

You could call it -- Vapour-ware ?


    ... sorry!   I'll go back to lurking now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20090114/03eb2957/attachment.html>

From fijall at gmail.com  Wed Jan 14 09:49:14 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 14 Jan 2009 09:49:14 +0100
Subject: [pypy-dev] wroclaw 2009 sprint announcement
Message-ID: <693bc9ab0901140049t34174e7l9a059e034589615b@mail.gmail.com>

========================================================
Wroc?aw PyPy sprint 7-14th February, 2009
========================================================

The next PyPy sprint will be held in Wroc?aw, Poland. This is fully public
sprint and all newcomers are welcomed. Preceeding the sprint there
will be a talk at University of Technology in Wroc?aw held at 22nd of January.

Worth noting is that this is a perfect time and place to go cross country
skiing either before or after sprint, so if anyone wants to participate in
such activity, we can organize something.

--------
Location
--------

Wroc?aw University of Technology, Poland
Wybrze?e Wyspia?skiego 23-25, Wroc?aw
building: C-13 (picture: http://tinyurl.com/6vvzwq)
room: 3.01 (3rd floor)
opening-hours: 8-21 (but the building is opened 24h/7)
campus map: http://www.pwr.wroc.pl/upload_module/images/english/campus_map.png
city map: http://tinyurl.com/7q3tfx

---------------
Native speakers
---------------

Official language in Poland is polish. There shouldn't be any problem with
english at all (especially during hotel booking etc) but if you need any help,
here is a list of native speakers who participate in the sprint. Feel free
to contact with any of them:

 * Maciej Fija?kowski ( fijall at gmail.com )
 * Bartosz Skowron ( getxsick at gmail.com )
 * Jakub Gustak ( jgustak at gmail.com )

------------
Registration
------------

If you'd like to come, please subscribe to the `pypy-sprint mailing list`_
and drop a note about your interests and post any questions.  More
organisational information will be sent to that list.

Please register by adding yourself on the following list (via svn):

  http://codespeak.net/svn/pypy/extradoc/sprintinfo/wroclaw2009/people.txt

or announce yourself on the pypy-sprint mailing list if you do not
yet have check-in rights:

  http://codespeak.net/mailman/listinfo/pypy-sprint

-------
Flights
-------

You can either fly directly to Wroc?aw, with many international connections
(like one to Dortmund and one to Duesseldorf). Details are available on
airport's webpage: http://www.airport.wroclaw.pl/en/. You can also fly to:
Warsaw (5h by train), Berlin (6h by train), Poznan (2h by train),
Katowice (3h by train) or Krak?w (4h by train).

--------------------------------------
Preparation (if you feel it is needed)
--------------------------------------

* read the `getting-started`_ pages on http://codespeak.net/pypy

* for inspiration, overview and technical status you are welcome to
  read `the technical reports available and other relevant documentation`_

* please direct any technical and/or development oriented questions to
  pypy-dev at codespeak.net and any sprint organizing/logistical
  questions to pypy-sprint at codespeak.net

We are looking forward to meet you at the Wroc?aw PyPy sprint!

The PyPy team

.. See also ..

.. _getting-started:
http://codespeak.net/pypy/dist/pypy/doc/getting-started.html
.. _`pypy-sprint mailing list`:
http://codespeak.net/mailman/listinfo/pypy-sprint
.. _`the technical reports available and other relevant
documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html


From fijall at gmail.com  Wed Jan 14 10:10:36 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 14 Jan 2009 10:10:36 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901131617.54395.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<20090113123107.GL2219@trillke.net>
	<200901131617.54395.guillem@torroja.dmt.upm.es>
Message-ID: <693bc9ab0901140110l73656338na16626271863ab5d@mail.gmail.com>

Hi Guillem

> I think that you'll have to start explaining what a JIT is. See the note on the
> audience below. I've been using Python+Numpy+Scipy+... as a complete replacement
> for matlab for the last two years. I suggested all of my colleagues to do the
> same but they all believe that python is a toy language (!). I think that you
> could give a more authorized opinion on this.
>

The JIT is a just-in-time compiler that compiles some intermediate
representation (for example python bytecode), to native code (for
example x86 assembler). The only existing JIT for python, psyco
(http://psyco.sourceforge.net) also massively specializes operations,
which means it gives much greater speedups than just getting read of
bytecode dispatch. It also gets read of type dispatching. For simple
loops with integer operations it can give speedups up to non-optimized
C (gcc -O0). PyPy is in some sense a followup project to psyco, which
should give you much larger benefits at some point in the future.

The exact gap that you might be interested in is the difference of
performance between numpy operations (which are rather fast, since
they're calling to years old fortran code) and a situation where you
need to for example iterate over the whole array and do some
computations in python (which is deadly slow).

I cannot really judge how python compares to matlab in terms of
availability of mathematic routines (scipy + numpy), but it's surely
nicer language and if it's fast enough, it could be a real win.

Now regarding pypy, the status is that we don't even have a numpy
bindings. And while we're surely working a lot on JIT these days, it's
still not ready for production use. So speaking about what we could
possibly do with jit & numpy is a bit speculative (but just a bit,
it's rather obvious how it could work, it's just not there yet).

Well, regarding toyness of a language, it's really hard to discuss
with such vague argument. If you provide us a reasoning why python is
a toy language we can try to provide you with some argumentation :)
>From my POV I can say that I wouldn't write a serious project size of
pypy in a toy language.

Cheers,
Maciej Fijalkowski


From holger at merlinux.eu  Wed Jan 14 10:12:04 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 14 Jan 2009 10:12:04 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <64ddb72c0901131441w98b1e12mfb3d872ebc2c20fd@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<20090109183421.GA2219@trillke.net>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<20090113123107.GL2219@trillke.net>
	<64ddb72c0901131441w98b1e12mfb3d872ebc2c20fd@mail.gmail.com>
Message-ID: <20090114091204.GM2219@trillke.net>

Hi Rene,

On Wed, Jan 14, 2009 at 09:41 +1100, Ren? Dudfield wrote:
> On Tue, Jan 13, 2009 at 11:31 PM, holger krekel <holger at merlinux.eu> wrote:
> >
> > Managing the complexity of software deployment and integration
> > is not PyPy's primary concern but it provides some interesting
> > building blocks.  For example, i am interested in using using
> > PyPy's sandboxing and virtualization capabilities to build an
> > open "cloud at home" network: people would run a bare-bones
> > executable, configure & control CPU, RAM, Network, Disk
> > resources and access rights. On this network e.g. open source
> > developers or scientists could launch programs and
> > computations or start os-level virtual machines where
> > non-python code runs.
> 
> You could call it -- Vapour-ware ?

I don't think that stating my interests amounts to pre-announcing
a release of non-existent software.  

Regarding this interest I am actually looking for like-minded
developers because i find it more fun to tackle this in a group. 

cheers,
holger


From holger at merlinux.eu  Wed Jan 14 10:44:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 14 Jan 2009 10:44:29 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901131617.54395.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901121210.50200.guillem@torroja.dmt.upm.es>
	<20090113123107.GL2219@trillke.net>
	<200901131617.54395.guillem@torroja.dmt.upm.es>
Message-ID: <20090114094429.GN2219@trillke.net>

Hi Guillem, 

On Tue, Jan 13, 2009 at 16:17 +0100, Guillem Borrell i Nogueras wrote:
> Hi!
> 
> El Tuesday 13 January 2009 13:31:07 holger krekel escribi?:
> 
> >
> > Could you give examples on the concrete challenges you face?
> >
> 
> Multicore architectures are killing supercomputing because two kinds of 
> parallelism are needed at the same time. A couple of years ago the computer 
> architecture departments started suggesting multiple OpenMP threads in every MPI 
> processes. This required a huge amount of work and never worked as expected 
> (maybe compilers are to blame?).
> 
> Now the trend seems to be using MPI for processes and a superscalar compiler 
> that uses all the cores and SPU for just a few kind of operations. This is not 
> more useful than using vectorized versions of Blas, levels 1 and 2.
> 
> Some years ago I started experimenting with python and MPI. I coded a very 
> limited MPI bindings collection for Python and launched python interpreters as 
> MPI processes. Instantly thought that a JIT-accelerated, multicore-aware 
> interpreter and a fast MPI-like message passing library would lower the 
> complexity of using parallel computers. That would not give the highest 
> performance but one has to be realistic. There's no need to distribute a VM, 
> message passing is not that hard.

For that scenario PyPy is currently missing the JIT, multicore-awareness and bindings 
to numeric libraries.  
 
> > > In addition, most of postprocessing tools are written in matlab, an
> > > interpreted language.  Running not-so-high performance tasks in a
> > > workstation efficiently is sometimes as important as running a simulation
> > > in a 12000 node supercomputer. It yould be nice if someone would remind
> > > the audience that matlab is not the only suitable (or best) tool for that
> > > job.
> >
> > Right, I guess that something like Numpy combined with PyPy's JIT
> > would could make a killer combination.  That's work, though.
> 
> I think that you'll have to start explaining what a JIT is. See the note on the 
> audience below. I've been using Python+Numpy+Scipy+... as a complete replacement 
> for matlab for the last two years. I suggested all of my colleagues to do the 
> same but they all believe that python is a toy language (!). I think that you 
> could give a more authorized opinion on this.

If that is your main idea - and considering the audience - 
i'd rather suggest to get someone authoritative from the
Numpy/Scipy community - they did two dedicated conferences
last year, the November issue of the Python Magazine was about
Python in scientific environments.   So it shouldn't be hard
to get someone who authentically confirms that Python is
a seriously used language for scientific computing.  

Of course, we'd be happy to discuss and incorporate a "PyPy"
perspective in such a talk.  

cheers,
holger

> > > I hope that I managed to explain the topics I'm interested in.
> >
> > sure, let us know what you think makes most sense for
> > the audience - who will be the audience btw?
> 
> The talks will be in the school of aeronautics. Most of people in the audience 
> will be engineers, mathematicians and physicists users of supercomputing without 
> any academic background on Computer Science. 
> 
> Thany you so much!
> -- 
> guillem
> 
> Guillem Borrell i Nogueras
> Laboratorio de Mec?nica de Fluidos Computacional
> Escuela T?cnica Superior de Ingenieros Aeron?uticos
> guillem at torroja.dmt.upm.es
> Web: http://torroja.dmt.upm.es/guillem/blog
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 


From guillem at torroja.dmt.upm.es  Wed Jan 14 11:46:06 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Wed, 14 Jan 2009 11:46:06 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <693bc9ab0901140110l73656338na16626271863ab5d@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901131617.54395.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140110l73656338na16626271863ab5d@mail.gmail.com>
Message-ID: <200901141146.07064.guillem@torroja.dmt.upm.es>

Hi!

I'll try to sort things out a bit.

I think that there are two separate but related topics.

* What can dynamic languages can offer to massive parallel computing in. Ease of 
use gain vs. performance loss. How to handle the multicore problem. How pypy and 
python could help. Total vaporware (sic.) but the trends and ideas are important 
too. 

* Interpreted languages in workstations. Interpreters and VM are becoming 
smarter (JIT or why using loops is not a crime anymore). Ease of use gain vs. 
performance loss (again). What can happen in the future and the pypy point of 
view.

(The audience will not have any knowledge about VM implementation.  They just 
know that from Matlab v6.5 simple loops are faster by some kind of unexplainable 
reason. They also believe that fortran and MPI is the only path to parallel 
computing.)

I asked you to give such a talk because, as pypy is a general VM implementation 
you have a wider vision of such things. I know that one could organize hundreds 
of congresses on this but the point here is to lighten the audience's interest. 
I know that *today* pypy lacks lots of technologies to be the definitive answer 
but I never asked for a solution to any problem.

I am more interested in what you would do than in what you have done (that is 
already impressive). The goal of the supercomputing day is to talk about 
interesting technologies, no matter if they are not ready yet.

Numpy support and bindings to numerical libraries is a problem I'd like to 
tackle myself in the future but I'm afraid I don't quite understand how to 
extend pypy.  Maybe it's just a matter of time or that I'm dumb. However I'm 
sure you could help me to understand lots of things if you come to Madrid in 
April.

cheers

PS: I'm not trying to tell you what to say in the talk. I'm just trying to 
explain what could be more interesting to the audience. You are free to 
completely ignore anything I suggest.

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From fijall at gmail.com  Wed Jan 14 11:59:18 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 14 Jan 2009 11:59:18 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901141146.07064.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901131617.54395.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140110l73656338na16626271863ab5d@mail.gmail.com>
	<200901141146.07064.guillem@torroja.dmt.upm.es>
Message-ID: <693bc9ab0901140259k6a9a6ffds275d36321f6fc84@mail.gmail.com>

> Numpy support and bindings to numerical libraries is a problem I'd like to
> tackle myself in the future but I'm afraid I don't quite understand how to
> extend pypy.  Maybe it's just a matter of time or that I'm dumb. However I'm
> sure you could help me to understand lots of things if you come to Madrid in
> April.
>

Confusion. Is it April or August? (you said August in your first mail)

Adding numpy to pypy shouldn't be that hard. It's work though and it
requires a bit of knowledge that is not documented anywhere. We're up
to help though.

Cheers,
fijal


From guillem at torroja.dmt.upm.es  Wed Jan 14 12:25:56 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Wed, 14 Jan 2009 12:25:56 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <693bc9ab0901140259k6a9a6ffds275d36321f6fc84@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901141146.07064.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140259k6a9a6ffds275d36321f6fc84@mail.gmail.com>
Message-ID: <200901141225.56173.guillem@torroja.dmt.upm.es>

Ooops April the 1st

sorry

El Wednesday 14 January 2009 11:59:18 Maciej Fijalkowski escribi?:
> > Numpy support and bindings to numerical libraries is a problem I'd like
> > to tackle myself in the future but I'm afraid I don't quite understand
> > how to extend pypy.  Maybe it's just a matter of time or that I'm dumb.
> > However I'm sure you could help me to understand lots of things if you
> > come to Madrid in April.
>
> Confusion. Is it April or August? (you said August in your first mail)
>
> Adding numpy to pypy shouldn't be that hard. It's work though and it
> requires a bit of knowledge that is not documented anywhere. We're up
> to help though.
>
> Cheers,
> fijal

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From fijall at gmail.com  Wed Jan 14 13:59:31 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 14 Jan 2009 13:59:31 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901141225.56173.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901141146.07064.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140259k6a9a6ffds275d36321f6fc84@mail.gmail.com>
	<200901141225.56173.guillem@torroja.dmt.upm.es>
Message-ID: <693bc9ab0901140459i66accf43ja116f564b682333d@mail.gmail.com>

On Wed, Jan 14, 2009 at 12:25 PM, Guillem Borrell i Nogueras
<guillem at torroja.dmt.upm.es> wrote:
> Ooops April the 1st
>
> sorry
>

Then, it's completely out of scope, at least for some of us. Pycon US
(http://us.pycon.org) ends 2nd of April in Chicago.

Cheers,
fijal


From bea at changemaker.nu  Wed Jan 14 14:10:43 2009
From: bea at changemaker.nu (Bea During)
Date: Wed, 14 Jan 2009 14:10:43 +0100
Subject: [pypy-dev] wroclaw 2009 sprint announcement
In-Reply-To: <693bc9ab0901140049t34174e7l9a059e034589615b@mail.gmail.com>
References: <693bc9ab0901140049t34174e7l9a059e034589615b@mail.gmail.com>
Message-ID: <496DE453.2080605@changemaker.nu>

Hi all

Just a tip when booking your flights - be prepared for the
flight search/booking instances to call Wroclaw:

- Strachowice
- Breslavia (german name for the city is Breslau)

Thanks Maciek for the help!

Cheers

Bea

Maciej Fijalkowski wrote:
> ========================================================
> Wroc?aw PyPy sprint 7-14th February, 2009
> ========================================================
>
> The next PyPy sprint will be held in Wroc?aw, Poland. This is fully public
> sprint and all newcomers are welcomed. Preceeding the sprint there
> will be a talk at University of Technology in Wroc?aw held at 22nd of January.
>
> Worth noting is that this is a perfect time and place to go cross country
> skiing either before or after sprint, so if anyone wants to participate in
> such activity, we can organize something.
>
> --------
> Location
> --------
>
> Wroc?aw University of Technology, Poland
> Wybrze?e Wyspia?skiego 23-25, Wroc?aw
> building: C-13 (picture: http://tinyurl.com/6vvzwq)
> room: 3.01 (3rd floor)
> opening-hours: 8-21 (but the building is opened 24h/7)
> campus map: http://www.pwr.wroc.pl/upload_module/images/english/campus_map.png
> city map: http://tinyurl.com/7q3tfx
>
> ---------------
> Native speakers
> ---------------
>
> Official language in Poland is polish. There shouldn't be any problem with
> english at all (especially during hotel booking etc) but if you need any help,
> here is a list of native speakers who participate in the sprint. Feel free
> to contact with any of them:
>
>  * Maciej Fija?kowski ( fijall at gmail.com )
>  * Bartosz Skowron ( getxsick at gmail.com )
>  * Jakub Gustak ( jgustak at gmail.com )
>
> ------------
> Registration
> ------------
>
> If you'd like to come, please subscribe to the `pypy-sprint mailing list`_
> and drop a note about your interests and post any questions.  More
> organisational information will be sent to that list.
>
> Please register by adding yourself on the following list (via svn):
>
>   http://codespeak.net/svn/pypy/extradoc/sprintinfo/wroclaw2009/people.txt
>
> or announce yourself on the pypy-sprint mailing list if you do not
> yet have check-in rights:
>
>   http://codespeak.net/mailman/listinfo/pypy-sprint
>
> -------
> Flights
> -------
>
> You can either fly directly to Wroc?aw, with many international connections
> (like one to Dortmund and one to Duesseldorf). Details are available on
> airport's webpage: http://www.airport.wroclaw.pl/en/. You can also fly to:
> Warsaw (5h by train), Berlin (6h by train), Poznan (2h by train),
> Katowice (3h by train) or Krak?w (4h by train).
>
> --------------------------------------
> Preparation (if you feel it is needed)
> --------------------------------------
>
> * read the `getting-started`_ pages on http://codespeak.net/pypy
>
> * for inspiration, overview and technical status you are welcome to
>   read `the technical reports available and other relevant documentation`_
>
> * please direct any technical and/or development oriented questions to
>   pypy-dev at codespeak.net and any sprint organizing/logistical
>   questions to pypy-sprint at codespeak.net
>
> We are looking forward to meet you at the Wroc?aw PyPy sprint!
>
> The PyPy team
>
> .. See also ..
>
> .. _getting-started:
> http://codespeak.net/pypy/dist/pypy/doc/getting-started.html
> .. _`pypy-sprint mailing list`:
> http://codespeak.net/mailman/listinfo/pypy-sprint
> .. _`the technical reports available and other relevant
> documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



From guillem at torroja.dmt.upm.es  Wed Jan 14 15:22:54 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Wed, 14 Jan 2009 15:22:54 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <693bc9ab0901140459i66accf43ja116f564b682333d@mail.gmail.com>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901141225.56173.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140459i66accf43ja116f564b682333d@mail.gmail.com>
Message-ID: <200901141522.54790.guillem@torroja.dmt.upm.es>


Ok, I'll try to move it to a later day.

guillem

El Wednesday 14 January 2009 13:59:31 Maciej Fijalkowski escribi?:
> On Wed, Jan 14, 2009 at 12:25 PM, Guillem Borrell i Nogueras
>
> <guillem at torroja.dmt.upm.es> wrote:
> > Ooops April the 1st
> >
> > sorry
>
> Then, it's completely out of scope, at least for some of us. Pycon US
> (http://us.pycon.org) ends 2nd of April in Chicago.
>
> Cheers,
> fijal

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From guillem at torroja.dmt.upm.es  Wed Jan 14 16:23:22 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Wed, 14 Jan 2009 16:23:22 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901141522.54790.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140459i66accf43ja116f564b682333d@mail.gmail.com>
	<200901141522.54790.guillem@torroja.dmt.upm.es>
Message-ID: <200901141623.22706.guillem@torroja.dmt.upm.es>

I have a new date

What about April 14th or 15th?

guillem

El Wednesday 14 January 2009 15:22:54 Guillem Borrell i Nogueras escribi?:
> Ok, I'll try to move it to a later day.
>
> guillem
>
> El Wednesday 14 January 2009 13:59:31 Maciej Fijalkowski escribi?:
> > On Wed, Jan 14, 2009 at 12:25 PM, Guillem Borrell i Nogueras
> >
> > <guillem at torroja.dmt.upm.es> wrote:
> > > Ooops April the 1st
> > >
> > > sorry
> >
> > Then, it's completely out of scope, at least for some of us. Pycon US
> > (http://us.pycon.org) ends 2nd of April in Chicago.
> >
> > Cheers,
> > fijal

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From hakan at debian.org  Thu Jan 15 13:26:14 2009
From: hakan at debian.org (Hakan Ardo)
Date: Thu, 15 Jan 2009 13:26:14 +0100
Subject: [pypy-dev] Index checking for setitem
Message-ID: <e7e642fc0901150426v2277bf2aqa3ff4e14c61972e2@mail.gmail.com>

Hi,
getitem operations gets converted into getitem_idx if they are
surrounded by try: except IndexError. I'v duplicated that
implementation to make setitem work in the same way. A patch is
available here:

  http://hakan.ardoe.net/pypy/setitem.patch

Also, I've updated my special_methods implementation to map
getitem_idx to the special method __getitem_idx__ and likewise for
setitem.

Would it be possible to build support for constructs such as:

  def get(lst,i):
    return lst[i]

  try:
    get([1,2,3],10)
  excpt IndexError:
    print "OK"

I suppose the convention from getitem to getitem_idx would have to
happen during annotation as different callsites are discovered and
each conversion would have to force a reflow of the converted
operation?

Thanx!

-- 
H?kan Ard?


From holger at merlinux.eu  Thu Jan 15 14:54:54 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 15 Jan 2009 14:54:54 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <200901141623.22706.guillem@torroja.dmt.upm.es>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<693bc9ab0901140459i66accf43ja116f564b682333d@mail.gmail.com>
	<200901141522.54790.guillem@torroja.dmt.upm.es>
	<200901141623.22706.guillem@torroja.dmt.upm.es>
Message-ID: <20090115135454.GP2219@trillke.net>

Hi Guillem,

On Wed, Jan 14, 2009 at 16:23 +0100, Guillem Borrell i Nogueras wrote:
> I have a new date
>
> What about April 14th or 15th?

Maciej and me discussed a bit.  April in general does not fit
well, unfortunately.  Apart from Pycon we'll be away and busy 
with pypy release works and moving the JIT forward.

In general we feel it's a bit early to talk about PyPy's
potential regarding super-computing, particularly to people
not primarily interested in compilers or Python.  

If you happen to go for such a day second half 2009
or in 2010 we'd be very happy to attend.

cheers & many thanks for your interest!
holger

> El Wednesday 14 January 2009 15:22:54 Guillem Borrell i Nogueras escribi?:
> > Ok, I'll try to move it to a later day.
> >
> > guillem
> >
> > El Wednesday 14 January 2009 13:59:31 Maciej Fijalkowski escribi?:
> > > On Wed, Jan 14, 2009 at 12:25 PM, Guillem Borrell i Nogueras
> > >
> > > <guillem at torroja.dmt.upm.es> wrote:
> > > > Ooops April the 1st
> > > >
> > > > sorry
> > >
> > > Then, it's completely out of scope, at least for some of us. Pycon US
> > > (http://us.pycon.org) ends 2nd of April in Chicago.
> > >
> > > Cheers,
> > > fijal
> 
> -- 
> guillem
> 
> Guillem Borrell i Nogueras
> Laboratorio de Mec?nica de Fluidos Computacional
> Escuela T?cnica Superior de Ingenieros Aeron?uticos
> guillem at torroja.dmt.upm.es
> Web: http://torroja.dmt.upm.es/guillem/blog
> 

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 


From guillem at torroja.dmt.upm.es  Thu Jan 15 17:28:59 2009
From: guillem at torroja.dmt.upm.es (Guillem Borrell i Nogueras)
Date: Thu, 15 Jan 2009 17:28:59 +0100
Subject: [pypy-dev] Talk in the Supercomputing Day, Madrid
In-Reply-To: <20090115135454.GP2219@trillke.net>
References: <200901091054.51758.guillem@torroja.dmt.upm.es>
	<200901141623.22706.guillem@torroja.dmt.upm.es>
	<20090115135454.GP2219@trillke.net>
Message-ID: <200901151728.59724.guillem@torroja.dmt.upm.es>

Hi Holger!

I can understand your decision. We plan to organize this supercomputing day 
yearly, this means that probably you will recieve the same invitation for the 
2010 event.  I'm afraid that it will be in April too, hopefully not the same day 
as Pycon US.

I plan to attend one pypy-sprint this year so we can discuss this things in 
person.

Thank you very much for your time!

Yours

guillem

El Thursday 15 January 2009 14:54:54 holger krekel escribi?:
> Hi Guillem,
>
> On Wed, Jan 14, 2009 at 16:23 +0100, Guillem Borrell i Nogueras wrote:
> > I have a new date
> >
> > What about April 14th or 15th?
>
> Maciej and me discussed a bit.  April in general does not fit
> well, unfortunately.  Apart from Pycon we'll be away and busy
> with pypy release works and moving the JIT forward.
>
> In general we feel it's a bit early to talk about PyPy's
> potential regarding super-computing, particularly to people
> not primarily interested in compilers or Python.
>
> If you happen to go for such a day second half 2009
> or in 2010 we'd be very happy to attend.
>
> cheers & many thanks for your interest!
> holger
>
> > El Wednesday 14 January 2009 15:22:54 Guillem Borrell i Nogueras escribi?:
> > > Ok, I'll try to move it to a later day.
> > >
> > > guillem
> > >
> > > El Wednesday 14 January 2009 13:59:31 Maciej Fijalkowski escribi?:
> > > > On Wed, Jan 14, 2009 at 12:25 PM, Guillem Borrell i Nogueras
> > > >
> > > > <guillem at torroja.dmt.upm.es> wrote:
> > > > > Ooops April the 1st
> > > > >
> > > > > sorry
> > > >
> > > > Then, it's completely out of scope, at least for some of us. Pycon US
> > > > (http://us.pycon.org) ends 2nd of April in Chicago.
> > > >
> > > > Cheers,
> > > > fijal
> >
> > --
> > guillem
> >
> > Guillem Borrell i Nogueras
> > Laboratorio de Mec?nica de Fluidos Computacional
> > Escuela T?cnica Superior de Ingenieros Aeron?uticos
> > guillem at torroja.dmt.upm.es
> > Web: http://torroja.dmt.upm.es/guillem/blog

-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mec?nica de Fluidos Computacional
Escuela T?cnica Superior de Ingenieros Aeron?uticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog


From getxsick at gmail.com  Thu Jan 15 19:27:45 2009
From: getxsick at gmail.com (Bartosz SKOWRON)
Date: Thu, 15 Jan 2009 19:27:45 +0100
Subject: [pypy-dev] wroclaw 2009 sprint announcement
In-Reply-To: <693bc9ab0901140049t34174e7l9a059e034589615b@mail.gmail.com>
References: <693bc9ab0901140049t34174e7l9a059e034589615b@mail.gmail.com>
Message-ID: <77887e110901151027o60e8eb6fl249228ed94b78d06@mail.gmail.com>

On Wed, Jan 14, 2009 at 9:49 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> ========================================================
> Wroc?aw PyPy sprint 7-14th February, 2009
> ========================================================

I added short info about accomodation.

Index: announcement.txt
===================================================================
--- announcement.txt	(wersja 60905)
+++ announcement.txt	(kopia robocza)
@@ -62,6 +62,20 @@
 Warsaw (5h by train), Berlin (6h by train), Poznan (2h by train),
 Katowice (3h by train) or Krak?w (4h by train).

+------------
+Accomodation
+------------
+
+Unfortunately, there aren't any hotels in really close area to sprint venue.
+There is a lot of hotels in city centre area and it's like 10-20 mins far away.
+List of possible accomodations:
+
+* http://www.wroclaw.pl/m6874/p7318.aspx (hotels)
+* http://www.wroclaw.pl/m6875/p7137.aspx (hostels)
+
+You can simple check the location at http://maps.google.com
+If you have any additional questions or advice, drop a line to Bartosz Skowron.
+
 --------------------------------------
 Preparation (if you feel it is needed)
 --------------------------------------

From pedronis at openend.se  Sun Jan 18 14:16:37 2009
From: pedronis at openend.se (Samuele Pedroni)
Date: Sun, 18 Jan 2009 14:16:37 +0100
Subject: [pypy-dev] forcing builds Re: another issue with buildbot
In-Reply-To: <693bc9ab0901180241n112cf28ck47883cb84c83e9dc@mail.gmail.com>
References: <693bc9ab0901180241n112cf28ck47883cb84c83e9dc@mail.gmail.com>
Message-ID: <49732BB5.4070405@openend.se>

Maciej Fijalkowski wrote:
> I found out that I'm unable to force run by hand. It checks out the
> revision from last build. Do you know how to bump it?
>   
you need to use the rebuild from the builder page, the rebuild from the 
build page will reuse the got_revision



From fijall at gmail.com  Sun Jan 18 15:17:10 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 18 Jan 2009 15:17:10 +0100
Subject: [pypy-dev] forcing builds Re: another issue with buildbot
In-Reply-To: <49732BB5.4070405@openend.se>
References: <693bc9ab0901180241n112cf28ck47883cb84c83e9dc@mail.gmail.com>
	<49732BB5.4070405@openend.se>
Message-ID: <693bc9ab0901180617x142fe532mce6b5eaa093013c4@mail.gmail.com>

On Sun, Jan 18, 2009 at 2:16 PM, Samuele Pedroni <pedronis at openend.se> wrote:
> Maciej Fijalkowski wrote:
>>
>> I found out that I'm unable to force run by hand. It checks out the
>> revision from last build. Do you know how to bump it?
>>
>
> you need to use the rebuild from the builder page, the rebuild from the
> build page will reuse the got_revision
>
>

I did. The thing is it does not use svn trunk, but instead uses
got_revision. How to bump got_revision?


From pedronis at openend.se  Sun Jan 18 16:15:08 2009
From: pedronis at openend.se (Samuele Pedroni)
Date: Sun, 18 Jan 2009 16:15:08 +0100
Subject: [pypy-dev] forcing builds Re: another issue with buildbot
In-Reply-To: <49732BB5.4070405@openend.se>
References: <693bc9ab0901180241n112cf28ck47883cb84c83e9dc@mail.gmail.com>
	<49732BB5.4070405@openend.se>
Message-ID: <4973477C.20905@openend.se>

Samuele Pedroni wrote:
> Maciej Fijalkowski wrote:
>   
>> I found out that I'm unable to force run by hand. It checks out the
>> revision from last build. Do you know how to bump it?
>>   
>>     
> you need to use the rebuild from the builder page, the rebuild from the 
> build page will reuse the got_revision
>   
to be clear (for anybody interested/following this):

http://codespeak.net:8099/builders/pypy-c-app-level-linux-x86-32

is a Builder page and the right one.

http://codespeak.net:8099/builders/pypy-c-app-level-linux-x86-32/builds/11

is a Build page and not so useful for forcing runs unless one is 
debugging a failure that is about the buildbot environment and not the 
code itself.


From nvetoshkin at naumen.ru  Tue Jan 20 19:55:49 2009
From: nvetoshkin at naumen.ru (=?UTF-8?B?0JLQtdGC0L7RiNC60LjQvSDQndC40LrQuNGC0LA=?=)
Date: Tue, 20 Jan 2009 23:55:49 +0500
Subject: [pypy-dev] Translate failes =(
Message-ID: <49761E35.3020307@naumen.ru>

Hi, all!

I'm trying to translate PyPy on Windows Vista x64 with Python2.4 32 bit 
and Visual Studio 2008 Express Edition. 32 bit - cause Express Edition 
isn't shipped with x64 compiler.
Translation failes: http://pastebin.com/m70de1376.
Fixing Visual Studio version recognition ( warning about KeyError 
VS71COMNTOOLS) doesn't help, translation failes with the same error.
Any suggestions?

P.S. In answers, please, put my mail in copy, cause I'm subscribed to 
digest =)


From nvetoshkin at naumen.ru  Tue Jan 20 22:13:51 2009
From: nvetoshkin at naumen.ru (=?UTF-8?B?0JLQtdGC0L7RiNC60LjQvSDQndC40LrQuNGC0LA=?=)
Date: Wed, 21 Jan 2009 02:13:51 +0500
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49761E35.3020307@naumen.ru>
References: <49761E35.3020307@naumen.ru>
Message-ID: <49763E8F.4010304@naumen.ru>

Thanks to fijal, I've installed ctypes and now I have MessageBox saying 
that system can't find "MSVCR90.DLL". After putting this library into 
current directory, I have this error:
---------------------------
Microsoft Visual C++ Runtime Library
---------------------------
Runtime Error!

Program: C:\Python24\python.exe

R6034

An application has made an attempt to load the C runtime library 
incorrectly.
Please contact the application's support team for more information.


---------------------------
OK
---------------------------

Google found this: http://msdn.microsoft.com/en-us/library/ms235560.aspx
As I understand - we need to include manifest into resulting binary or 
perhaps link externmod_2.dll statically.
The resulting stacktrace: http://pastebin.com/m6965b0f7
Please help =)

20.01.2009 23:55, ???????? ?????? ?????:
> Hi, all!
>
> I'm trying to translate PyPy on Windows Vista x64 with Python2.4 32 bit
> and Visual Studio 2008 Express Edition. 32 bit - cause Express Edition
> isn't shipped with x64 compiler.
> Translation failes: http://pastebin.com/m70de1376.
> Fixing Visual Studio version recognition ( warning about KeyError
> VS71COMNTOOLS) doesn't help, translation failes with the same error.
> Any suggestions?
>
> P.S. In answers, please, put my mail in copy, cause I'm subscribed to
> digest =)


From fijall at gmail.com  Tue Jan 20 22:23:38 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 20 Jan 2009 22:23:38 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49763E8F.4010304@naumen.ru>
References: <49761E35.3020307@naumen.ru> <49763E8F.4010304@naumen.ru>
Message-ID: <693bc9ab0901201323i53626928j38af1020536eca16@mail.gmail.com>

> Thanks to fijal, I've installed ctypes and now I have MessageBox saying
> that system can't find "MSVCR90.DLL". After putting this library into
> current directory, I have this error:
> ---------------------------
> Microsoft Visual C++ Runtime Library
> ---------------------------
> Runtime Error!
>
> Program: C:\Python24\python.exe
>
> R6034
>
> An application has made an attempt to load the C runtime library
> incorrectly.
> Please contact the application's support team for more information.
>

Heh :)

I suppose that's us, but I have no clue, sorry ;-)

Cheers,
fijal


From lutz_p at gmx.net  Tue Jan 20 22:56:05 2009
From: lutz_p at gmx.net (Lutz Paelike)
Date: Tue, 20 Jan 2009 22:56:05 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49763E8F.4010304@naumen.ru>
References: <49761E35.3020307@naumen.ru> <49763E8F.4010304@naumen.ru>
Message-ID: <B6537249-DE6D-4686-B022-49F56A7312D7@gmx.net>

I have a guess how that could happen.

The reason is probably that two different versions of  MSVCR dll are  
loaded into the process that
result in a name clash of the symbols provided by the library.

Now *why* this happens is not entirely clear but i have also a wild  
guess here.

The standard Python2.4 distribution is using MSVCR71.dll an older  
version (7.1)
that is used when compiled with e.g. Visual Studio 2005

If you use python all installed  modules are  linked against  the same  
version of the MSVCR dll
and thus it works without a problem (provided you have MSVCR71.dll on  
your system).

Now the problem arises if you compile an extension yourself with a  
different compiler version
which links your compiled extension against a different  version of  
the library (in this case MSVCR90.dll = Version 9.0).

Solution:
1. You should use the same compiler version (VS 2005) as the one used  
of your python installation
or
2. Compile Python and all modules you use yourself from source and use  
that instead of an precompiled distribution

Another variable in this setup might be ctypes.
If ctypes loads the MSVCR library without providing the exact  
requested version it might
determine the best (newest) version itself and that might not be right  
one.

Hope this helps.
Cheers,

Lutz


Am 20.01.2009 um 22:13 schrieb ???????? ??????:

> Thanks to fijal, I've installed ctypes and now I have MessageBox  
> saying
> that system can't find "MSVCR90.DLL". After putting this library into
> current directory, I have this error:
> ---------------------------
> Microsoft Visual C++ Runtime Library
> ---------------------------
> Runtime Error!
>
> Program: C:\Python24\python.exe
>
> R6034
>
> An application has made an attempt to load the C runtime library
> incorrectly.
> Please contact the application's support team for more information.
>
>
> ---------------------------
> OK
> ---------------------------
>
> Google found this: http://msdn.microsoft.com/en-us/library/ms235560.aspx
> As I understand - we need to include manifest into resulting binary or
> perhaps link externmod_2.dll statically.
> The resulting stacktrace: http://pastebin.com/m6965b0f7
> Please help =)
>
> 20.01.2009 23:55, ???????? ?????? ?????:
>> Hi, all!
>>
>> I'm trying to translate PyPy on Windows Vista x64 with Python2.4 32  
>> bit
>> and Visual Studio 2008 Express Edition. 32 bit - cause Express  
>> Edition
>> isn't shipped with x64 compiler.
>> Translation failes: http://pastebin.com/m70de1376.
>> Fixing Visual Studio version recognition ( warning about KeyError
>> VS71COMNTOOLS) doesn't help, translation failes with the same error.
>> Any suggestions?
>>
>> P.S. In answers, please, put my mail in copy, cause I'm subscribed to
>> digest =)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



From lutz_p at gmx.net  Tue Jan 20 22:57:30 2009
From: lutz_p at gmx.net (Lutz Paelike)
Date: Tue, 20 Jan 2009 22:57:30 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49763E8F.4010304@naumen.ru>
References: <49761E35.3020307@naumen.ru> <49763E8F.4010304@naumen.ru>
Message-ID: <DC3EFDC5-F48E-41CC-B49A-A9B68CECC29D@gmx.net>

I have a guess how that could happen.

The reason is probably that two different versions of  MSVCR dll are  
loaded into the process that
result in a name clash of the symbols provided by the library.

Now *why* this happens is not entirely clear but i have also a wild  
guess here.

The standard Python2.4 distribution is using MSVCR71.dll an older  
version (7.1)
that is used when compiled with e.g. Visual Studio 2005

If you use python all installed  modules are  linked against  the same  
version of the MSVCR dll
and thus it works without a problem (provided you have MSVCR71.dll on  
your system).

Now the problem arises if you compile an extension yourself with a  
different compiler version
which links your compiled extension against a different  version of  
the library (in this case MSVCR90.dll = Version 9.0).

Solution:
1. You should use the same compiler version (VS 2005) as the one used  
of your python installation
or
2. Compile Python and all modules you use yourself from source and use  
that instead of an precompiled distribution

Another variable in this setup might be ctypes.
If ctypes loads the MSVCR library without providing the exact  
requested version it might
determine the best (newest) version itself and that might not be right  
one.

Hope this helps.
Cheers,

Lutz


Am 20.01.2009 um 22:13 schrieb ???????? ??????:

> Thanks to fijal, I've installed ctypes and now I have MessageBox  
> saying
> that system can't find "MSVCR90.DLL". After putting this library into
> current directory, I have this error:
> ---------------------------
> Microsoft Visual C++ Runtime Library
> ---------------------------
> Runtime Error!
>
> Program: C:\Python24\python.exe
>
> R6034
>
> An application has made an attempt to load the C runtime library
> incorrectly.
> Please contact the application's support team for more information.
>
>
> ---------------------------
> OK
> ---------------------------
>
> Google found this: http://msdn.microsoft.com/en-us/library/ms235560.aspx
> As I understand - we need to include manifest into resulting binary or
> perhaps link externmod_2.dll statically.
> The resulting stacktrace: http://pastebin.com/m6965b0f7
> Please help =)
>
> 20.01.2009 23:55, ???????? ?????? ?????:
>> Hi, all!
>>
>> I'm trying to translate PyPy on Windows Vista x64 with Python2.4 32  
>> bit
>> and Visual Studio 2008 Express Edition. 32 bit - cause Express  
>> Edition
>> isn't shipped with x64 compiler.
>> Translation failes: http://pastebin.com/m70de1376.
>> Fixing Visual Studio version recognition ( warning about KeyError
>> VS71COMNTOOLS) doesn't help, translation failes with the same error.
>> Any suggestions?
>>
>> P.S. In answers, please, put my mail in copy, cause I'm subscribed to
>> digest =)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



From nvetoshkin at naumen.ru  Tue Jan 20 23:24:34 2009
From: nvetoshkin at naumen.ru (=?UTF-8?B?0JLQtdGC0L7RiNC60LjQvSDQndC40LrQuNGC0LA=?=)
Date: Wed, 21 Jan 2009 03:24:34 +0500
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <DC3EFDC5-F48E-41CC-B49A-A9B68CECC29D@gmx.net>
References: <49761E35.3020307@naumen.ru> <49763E8F.4010304@naumen.ru>
	<DC3EFDC5-F48E-41CC-B49A-A9B68CECC29D@gmx.net>
Message-ID: <49764F22.7080606@naumen.ru>

I used /MT flag - that is static linking (in 
pypy\translator\platform\windows.py). Then installed zlib, added zlib's 
include dir to my INCLUDE env variable. And... now it's building =)

21.01.2009 2:57, Lutz Paelike ?????:
> I have a guess how that could happen.
>
> The reason is probably that two different versions of MSVCR dll are
> loaded into the process that
> result in a name clash of the symbols provided by the library.
>
> Now *why* this happens is not entirely clear but i have also a wild
> guess here.
>
> The standard Python2.4 distribution is using MSVCR71.dll an older
> version (7.1)
> that is used when compiled with e.g. Visual Studio 2005
>
> If you use python all installed modules are linked against the same
> version of the MSVCR dll
> and thus it works without a problem (provided you have MSVCR71.dll on
> your system).
>
> Now the problem arises if you compile an extension yourself with a
> different compiler version
> which links your compiled extension against a different version of the
> library (in this case MSVCR90.dll = Version 9.0).
>
> Solution:
> 1. You should use the same compiler version (VS 2005) as the one used of
> your python installation
> or
> 2. Compile Python and all modules you use yourself from source and use
> that instead of an precompiled distribution
>
> Another variable in this setup might be ctypes.
> If ctypes loads the MSVCR library without providing the exact requested
> version it might
> determine the best (newest) version itself and that might not be right one.
>
> Hope this helps.
> Cheers,
>
> Lutz
>
>
> Am 20.01.2009 um 22:13 schrieb ???????? ??????:
>
>> Thanks to fijal, I've installed ctypes and now I have MessageBox saying
>> that system can't find "MSVCR90.DLL". After putting this library into
>> current directory, I have this error:
>> ---------------------------
>> Microsoft Visual C++ Runtime Library
>> ---------------------------
>> Runtime Error!
>>
>> Program: C:\Python24\python.exe
>>
>> R6034
>>
>> An application has made an attempt to load the C runtime library
>> incorrectly.
>> Please contact the application's support team for more information.
>>
>>
>> ---------------------------
>> OK
>> ---------------------------
>>
>> Google found this: http://msdn.microsoft.com/en-us/library/ms235560.aspx
>> As I understand - we need to include manifest into resulting binary or
>> perhaps link externmod_2.dll statically.
>> The resulting stacktrace: http://pastebin.com/m6965b0f7
>> Please help =)
>>
>> 20.01.2009 23:55, ???????? ?????? ?????:
>>> Hi, all!
>>>
>>> I'm trying to translate PyPy on Windows Vista x64 with Python2.4 32 bit
>>> and Visual Studio 2008 Express Edition. 32 bit - cause Express Edition
>>> isn't shipped with x64 compiler.
>>> Translation failes: http://pastebin.com/m70de1376.
>>> Fixing Visual Studio version recognition ( warning about KeyError
>>> VS71COMNTOOLS) doesn't help, translation failes with the same error.
>>> Any suggestions?
>>>
>>> P.S. In answers, please, put my mail in copy, cause I'm subscribed to
>>> digest =)
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>


From amauryfa at gmail.com  Tue Jan 20 23:53:31 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 20 Jan 2009 23:53:31 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49764F22.7080606@naumen.ru>
References: <49761E35.3020307@naumen.ru> <49763E8F.4010304@naumen.ru>
	<DC3EFDC5-F48E-41CC-B49A-A9B68CECC29D@gmx.net>
	<49764F22.7080606@naumen.ru>
Message-ID: <e27efe130901201453q3003adb2x9360518855bac1aa@mail.gmail.com>

Hello,

On Tue, Jan 20, 2009 at 23:24, ???????? ?????? <nvetoshkin at naumen.ru> wrote:
> I used /MT flag - that is static linking (in
> pypy\translator\platform\windows.py). Then installed zlib, added zlib's
> include dir to my INCLUDE env variable. And... now it's building =)

The /MT flag is a nice trick.
But the ctypes module of the generated pypy-c will not work correctly:
get_libc() is supposed to return the MSVCRT.dll linked with the
executable, this function won't work if the c runtime is linked
statically.

The best would be to use /MT (=static C runtime) to build extension
modules (during the "configure" phase) and /MD (=shared C runtime)
when building the pypy-c executable.

-- 
Amaury Forgeot d'Arc

From nvetoshkin at naumen.ru  Wed Jan 21 00:12:37 2009
From: nvetoshkin at naumen.ru (Vetoshkin Nikita)
Date: Wed, 21 Jan 2009 04:12:37 +0500
Subject: [pypy-dev] Translate failes =(
Message-ID: <49765A65.6040102@naumen.ru>

Could you please point me source files where to look/digg:
"configure" phase building and
pypy-c executable building.
As I see now, /MT flag I used applies to all compiled binaries and you 
suggest using it only during first building process?
Any ideas about building manifest file to proper msvcrt.dll linking?

 >The /MT flag is a nice trick.
 >But the ctypes module of the generated pypy-c will not work correctly:
 >get_libc() is supposed to return the MSVCRT.dll linked with the
 >executable, this function won't work if the c runtime is linked
 >statically.

 >The best would be to use /MT (=static C runtime) to build extension
 >modules (during the "configure" phase) and /MD (=shared C runtime)
 >when building the pypy-c executable.
 >
 >-- Amaury Forgeot d'Arc


From fijall at gmail.com  Wed Jan 21 00:17:25 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 21 Jan 2009 00:17:25 +0100
Subject: [pypy-dev] windows buildbot
Message-ID: <693bc9ab0901201517ma487febub82414bdfe4174e0@mail.gmail.com>

I just triggered translation here:

http://codespeak.net:8099/builders/pypy-c-lib-python-win-32/builds/23/steps/translate/logs/stdio

and I noticed that there are 48 implement files. Is it because files
are split to smaller chunks for some reason,
or something went wrong along the lines?

Cheers,
fijal


From amauryfa at gmail.com  Wed Jan 21 00:40:45 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 21 Jan 2009 00:40:45 +0100
Subject: [pypy-dev] windows buildbot
In-Reply-To: <693bc9ab0901201517ma487febub82414bdfe4174e0@mail.gmail.com>
References: <693bc9ab0901201517ma487febub82414bdfe4174e0@mail.gmail.com>
Message-ID: <e27efe130901201540i4c23df8bpba1b1fb3b6afe561@mail.gmail.com>

On Wed, Jan 21, 2009 at 00:17, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I just triggered translation here:
>
> http://codespeak.net:8099/builders/pypy-c-lib-python-win-32/builds/23/steps/translate/logs/stdio
>
> and I noticed that there are 48 implement files. Is it because files
> are split to smaller chunks for some reason,
> or something went wrong along the lines?

It's normal if you consider this code in translator/c/genc.py:

SPLIT_CRITERIA = 65535 # support VC++ 7.2
[...]
        if py.std.sys.platform != "win32":
            split_criteria_big = SPLIT_CRITERIA * 4
        else:
            split_criteria_big = SPLIT_CRITERIA


Old versions of Visual Studio had problems with large files.
Even today, debug information is wrong when lineno>65536.

-- 
Amaury Forgeot d'Arc


From renesd at gmail.com  Wed Jan 21 00:43:42 2009
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Wed, 21 Jan 2009 10:43:42 +1100
Subject: [pypy-dev] windows buildbot
In-Reply-To: <e27efe130901201540i4c23df8bpba1b1fb3b6afe561@mail.gmail.com>
References: <693bc9ab0901201517ma487febub82414bdfe4174e0@mail.gmail.com>
	<e27efe130901201540i4c23df8bpba1b1fb3b6afe561@mail.gmail.com>
Message-ID: <64ddb72c0901201543k133d1d7dm11c02e41052cda48@mail.gmail.com>

hi,

I guess you already know that newer versions of gcc are slower with
larger files.  It can be faster even for gcc to split up files into
smaller pieces.

cheers,


On Wed, Jan 21, 2009 at 10:40 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> On Wed, Jan 21, 2009 at 00:17, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I just triggered translation here:
>>
>> http://codespeak.net:8099/builders/pypy-c-lib-python-win-32/builds/23/steps/translate/logs/stdio
>>
>> and I noticed that there are 48 implement files. Is it because files
>> are split to smaller chunks for some reason,
>> or something went wrong along the lines?
>
> It's normal if you consider this code in translator/c/genc.py:
>
> SPLIT_CRITERIA = 65535 # support VC++ 7.2
> [...]
>        if py.std.sys.platform != "win32":
>            split_criteria_big = SPLIT_CRITERIA * 4
>        else:
>            split_criteria_big = SPLIT_CRITERIA
>
>
> Old versions of Visual Studio had problems with large files.
> Even today, debug information is wrong when lineno>65536.
>
> --
> Amaury Forgeot d'Arc


From amauryfa at gmail.com  Wed Jan 21 01:26:54 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 21 Jan 2009 01:26:54 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49765A65.6040102@naumen.ru>
References: <49765A65.6040102@naumen.ru>
Message-ID: <e27efe130901201626l7bbfd9dch95557ebd00e2166c@mail.gmail.com>

On Wed, Jan 21, 2009 at 00:12, Vetoshkin Nikita <nvetoshkin at naumen.ru> wrote:
> Could you please point me source files where to look/digg:
> "configure" phase building and
> pypy-c executable building.
> As I see now, /MT flag I used applies to all compiled binaries and you
> suggest using it only during first building process?

Yes. My suggestion was that "standalone" builds should use /MD and
shared libraries ("not standalone")
use /MT.
But I realize that this would not solve all problems.

> Any ideas about building manifest file to proper msvcrt.dll linking?

I am currently trying to to it, borrowing ideas from cpython's code:
http://svn.python.org/view/python/trunk/Lib/distutils/msvc9compiler.py?rev=68081&view=markup
look for "manifest"

-- 
Amaury Forgeot d'Arc


From amauryfa at gmail.com  Wed Jan 21 02:14:34 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 21 Jan 2009 02:14:34 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49765A65.6040102@naumen.ru>
References: <49765A65.6040102@naumen.ru>
Message-ID: <e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>

On Wed, Jan 21, 2009 at 00:12, Vetoshkin Nikita <nvetoshkin at naumen.ru> wrote:
> Any ideas about building manifest file to proper msvcrt.dll linking?

I just changed the compilation tools in windows.py to build and embed
this manifest.
Can you try again?

-- 
Amaury Forgeot d'Arc


From nvetoshkin at naumen.ru  Wed Jan 21 02:42:02 2009
From: nvetoshkin at naumen.ru (Vetoshkin Nikita)
Date: Wed, 21 Jan 2009 06:42:02 +0500
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
References: <49765A65.6040102@naumen.ru>
	<e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
Message-ID: <49767D6A.8080607@naumen.ru>

I've got this now: http://pastebin.com/d4f3dfefd

21.01.2009 6:14, Amaury Forgeot d'Arc ?????:
> On Wed, Jan 21, 2009 at 00:12, Vetoshkin Nikita<nvetoshkin at naumen.ru>  wrote:
>> Any ideas about building manifest file to proper msvcrt.dll linking?
>
> I just changed the compilation tools in windows.py to build and embed
> this manifest.
> Can you try again?
>


From amauryfa at gmail.com  Wed Jan 21 02:50:03 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 21 Jan 2009 02:50:03 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49767D6A.8080607@naumen.ru>
References: <49765A65.6040102@naumen.ru>
	<e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
	<49767D6A.8080607@naumen.ru>
Message-ID: <e27efe130901201750oaf47a25s2cff92abb73afad0@mail.gmail.com>

On Wed, Jan 21, 2009 at 02:42, Vetoshkin Nikita <nvetoshkin at naumen.ru> wrote:
> I've got this now: http://pastebin.com/d4f3dfefd

This output is missing the few lines that can be important.
I would like to see the (long) lines starting with
    "[platform:execute] link.exe..."
and
    "[platform:execute] mt.exe..."
and which contain the manifest file name

-- 
Amaury Forgeot d'Arc


From amauryfa at gmail.com  Wed Jan 21 02:50:48 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 21 Jan 2009 02:50:48 +0100
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <49767D6A.8080607@naumen.ru>
References: <49765A65.6040102@naumen.ru>
	<e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
	<49767D6A.8080607@naumen.ru>
Message-ID: <e27efe130901201750s1b17c12cqa8489d958d377a7c@mail.gmail.com>

On Wed, Jan 21, 2009 at 02:42, Vetoshkin Nikita <nvetoshkin at naumen.ru> wrote:
> I've got this now: http://pastebin.com/d4f3dfefd

Oh, and did you reset the /MD option? /MT is wrong now.

-- 
Amaury Forgeot d'Arc


From nvetoshkin at naumen.ru  Wed Jan 21 02:52:07 2009
From: nvetoshkin at naumen.ru (Vetoshkin Nikita)
Date: Wed, 21 Jan 2009 06:52:07 +0500
Subject: [pypy-dev] Translate failes =(
In-Reply-To: <e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
References: <49765A65.6040102@naumen.ru>
	<e27efe130901201714w4ec2f4bcyc7b7f499ec55a03d@mail.gmail.com>
Message-ID: <49767FC7.9010902@naumen.ru>

Sorry, I was wrong. It works fine... awaiting translation to complete =)

21.01.2009 6:14, Amaury Forgeot d'Arc ?????:
> On Wed, Jan 21, 2009 at 00:12, Vetoshkin Nikita<nvetoshkin at naumen.ru>  wrote:
>> Any ideas about building manifest file to proper msvcrt.dll linking?
>
> I just changed the compilation tools in windows.py to build and embed
> this manifest.
> Can you try again?
>


From anto.cuni at gmail.com  Wed Jan 21 14:07:58 2009
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 21 Jan 2009 14:07:58 +0100
Subject: [pypy-dev] Non-circular path through pypy module imports
In-Reply-To: <gkftm6$qfk$1@ger.gmane.org>
References: <gkftm6$qfk$1@ger.gmane.org>
Message-ID: <49771E2E.4030301@gmail.com>

VanL wrote:
> Hello,

Hi!
Sorry for the late response, everyone thought that someone else would have 
been answered, but nobody did at the end :-)

> I have been going through the pypy sources recently and I wanted to start from
> the leaf modules first (in terms of import dependencies)... but there don't
> appear to be any leaves.
> 
> For example, starting with baseobjspace in the interpreter, it has 30 or so
> non-stdlib imports, many of them delayed by placing them in functions. Even
> something like rarithmetic eventually pulls in most of the lltype hierarchy. The
> closest I found to a leaf module was extregistry.py.
> 
> Is there a reason for the circularities that I just don't get? If not, why are
> there so many?

Well, for at least some of circularities is partly our fault, we know that the 
import relationships between our modules are a mess currently. Sorry for that.

Anyway, I don't think that starting from the leaves is the best option to 
grasp pypy: probably it's better to pick a topic you are interested in, and 
start from there.

If you are more interested in the translation toolchain (i.e., the 
rpython-to-{c,cli,jvm} compiler), you can start from either the beginning or 
the end of the chain; look for example at the annotation/ directory or at the 
various backends in translator/{c,cli,jvm}.

On the other hand, if you are more interested in the interpreter (i.e., the 
python implementation written in rpython) you want to have a look to 
interpreter/ and objspace/std. The former directory contains the core of the 
interpreter, while the latter contains the implementation of all the standard 
builtin types such as lists, dictionaries, classes, etc.

A good starting point could be the main loop of the interpreter: look at the 
dispatch_bytecode function inside interpreter/pyopcode.py.

I assume that you have already read the pypy documentation online; if not, you 
are strongly encouraged to do that before looking at the source; in 
particular, this document describes the high level architecture of pypy:
http://codespeak.net/pypy/dist/pypy/doc/architecture.html

More generally, all the pypy documentation is available here:
http://codespeak.net/pypy/dist/pypy/doc/

ciao,
Anto


From anto.cuni at gmail.com  Thu Jan 22 11:16:45 2009
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 22 Jan 2009 11:16:45 +0100
Subject: [pypy-dev] [pypy-svn] r61220 - pypy/trunk/lib-python
In-Reply-To: <20090122093417.85D5A168460@codespeak.net>
References: <20090122093417.85D5A168460@codespeak.net>
Message-ID: <4978478D.1030909@gmail.com>

fijal at codespeak.net wrote:

> we need to have RegrTest for each file starting with test_ anyway...
> 
> 
> Modified: pypy/trunk/lib-python/conftest.py
> ==============================================================================
> --- pypy/trunk/lib-python/conftest.py	(original)
> +++ pypy/trunk/lib-python/conftest.py	Thu Jan 22 10:34:15 2009
> @@ -294,6 +294,7 @@
>      RegrTest('test_mmap.py'),
>      RegrTest('test_module.py', core=True),
>      RegrTest('test_multibytecodec.py', skip="unsupported codecs"),
> +    RegrTest('test_multibytecodec.py_support', skip="not a test"),

uhm... did you want to write test_multibytecodec_support.py, maybe?


From fijall at gmail.com  Thu Jan 22 11:47:27 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 22 Jan 2009 11:47:27 +0100
Subject: [pypy-dev] [pypy-svn] r61220 - pypy/trunk/lib-python
In-Reply-To: <4978478D.1030909@gmail.com>
References: <20090122093417.85D5A168460@codespeak.net>
	<4978478D.1030909@gmail.com>
Message-ID: <693bc9ab0901220247y1303d009vaaf23e36057f2dfc@mail.gmail.com>

argh, yes

On 1/22/09, Antonio Cuni <anto.cuni at gmail.com> wrote:
> fijal at codespeak.net wrote:
>
>> we need to have RegrTest for each file starting with test_ anyway...
>>
>>
>> Modified: pypy/trunk/lib-python/conftest.py
>> ==============================================================================
>> --- pypy/trunk/lib-python/conftest.py	(original)
>> +++ pypy/trunk/lib-python/conftest.py	Thu Jan 22 10:34:15 2009
>> @@ -294,6 +294,7 @@
>>      RegrTest('test_mmap.py'),
>>      RegrTest('test_module.py', core=True),
>>      RegrTest('test_multibytecodec.py', skip="unsupported codecs"),
>> +    RegrTest('test_multibytecodec.py_support', skip="not a test"),
>
> uhm... did you want to write test_multibytecodec_support.py, maybe?
>


From fijall at gmail.com  Sat Jan 24 20:42:55 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 24 Jan 2009 20:42:55 +0100
Subject: [pypy-dev] parser compiler and _ast
Message-ID: <693bc9ab0901241142j50a5dd96s591b219620ba8b60@mail.gmail.com>

I did a bit of research on a matter of different python interfaces to
parsing. This is a bit of a mess, but it is as follows:

a) there is a parser module which gives you very low-level interface
    (so-called concrete syntax tree), which is essentially
    a list of tuples with numbers inside. you can compile those tuples
    to bytecode afterwards. It's slowly being deprecated.

b) there is a compiler module, which is a wrapper around parser. it
    does not let you compile ast, but it gives you ast. It's deprecated since
    2.6.

c) there is an ast module which lets you have ast *and* compile it. very neat.

Now the state of the art is that we have parser, compiler is pure
app-level and we
don't provide ast. We already discussed a bit whether to have ast module for 1.1
and the answer is "very likely no", raise hands if you want to have it.

Now I want to know how much do we care about all of this and how much do we care
about any of those? My suggestions are as follows:

1. We keep parser as it is and don't try to fix bugs. Bugs (as
reported by CPython's
test failures) are that we don't do extra walk around the tree to
detect syntax errors.
those errors are only detected when we generated the code

- or -

1a. We just always generate the code and throw it away. We loose
performance, but we don't care
(and we pass tests)

2. We slowly deprecate parser & compiler, with some removal in future

3. If we develop a new parser, we drop compatibility with exact
concrete syntax trees and
we only keep ast around (as ast module) if people want to interact
with it (this is one less
worry for writing parser).

What do you think?

Cheers,
fijal


From fijall at gmail.com  Mon Jan 26 11:05:20 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 26 Jan 2009 11:05:20 +0100
Subject: [pypy-dev] resource module for pypy
Message-ID: <693bc9ab0901260205o5a144cfcx368a992a372b96a5@mail.gmail.com>

Hi Victor, hi pypy-dev

I think you're the one who implemented the resource module using
ctypes. Right now it timouts/fails when
running cpython's test suite (1.) - last run. Our policy is to either
fix it, or remove it for 1.1 release.
Do you eventually would like to take a look?

Cheers,
fijal

1. http://codespeak.net:8099/summary/longrepr?testname=unmodified&builder=pypy-c-lib-python-linux-x86-32&build=129&mod=lib-python.test_resource


From anto.cuni at gmail.com  Mon Jan 26 14:24:14 2009
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 26 Jan 2009 14:24:14 +0100
Subject: [pypy-dev] parser compiler and _ast
In-Reply-To: <693bc9ab0901241142j50a5dd96s591b219620ba8b60@mail.gmail.com>
References: <693bc9ab0901241142j50a5dd96s591b219620ba8b60@mail.gmail.com>
Message-ID: <497DB97E.6070103@gmail.com>

Maciej Fijalkowski wrote:
> I did a bit of research on a matter of different python interfaces to
> parsing. This is a bit of a mess, but it is as follows:
> 
[cut]
> 1a. We just always generate the code and throw it away. We loose
> performance, but we don't care
> (and we pass tests)

a quick google code search reveals that there are a couple of projects using 
parser, like pychecker, epydoc and quixote. If it's not too time consuming, I 
would go for this option.

> 2. We slowly deprecate parser & compiler, with some removal in future

+1

> 3. If we develop a new parser, we drop compatibility with exact
> concrete syntax trees and
> we only keep ast around (as ast module) if people want to interact
> with it (this is one less
> worry for writing parser).

That's an option, but not for 1.1, I think.

ciao,
Anto


From jacob at openend.se  Tue Jan 27 22:24:20 2009
From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=)
Date: Tue, 27 Jan 2009 22:24:20 +0100
Subject: [pypy-dev] Europython sprint
Message-ID: <200901272224.20283.jacob@openend.se>

Hi everyone,

the Europython organisers would like us to settle on dates for a sprint around 
Europython as soon as possible. They are very eager to do well in the 
sprinting department, as sprints are something that hasn't taken off in the 
UK so far. It is something they very much want to change.

PyPy is naturally considered to be the leader in the field of sprints and the 
organisers would like to accomdate our needs as best they can. This means 
that they have to book rooms in the next week or so.

The outine of the programme is as follows:

    *  Sunday 28th June and Monday 29th June : Tutorial Days
    * Tuesday 30th June to Thursday 2nd July : Conference
    * Friday 3rd July, as long as needed : Sprints

We could have a sprint(an internal one?)  in parallel with the tutorials if we 
want.

How long would we like to sprint after the conference? Friday-Sunday?

Birmingham is quite a nice environment and the rate of the British pound is 
not so outrageous, that we probably can afford a decent length of stay.

Jacob


From anto.cuni at gmail.com  Wed Jan 28 13:27:25 2009
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 28 Jan 2009 13:27:25 +0100
Subject: [pypy-dev] Europython sprint
In-Reply-To: <200901272224.20283.jacob@openend.se>
References: <200901272224.20283.jacob@openend.se>
Message-ID: <49804F2D.3060005@gmail.com>

Hi Jacob, hi all,

Jacob Hall?n wrote:

> The outine of the programme is as follows:
> 
>     *  Sunday 28th June and Monday 29th June : Tutorial Days
>     * Tuesday 30th June to Thursday 2nd July : Conference
>     * Friday 3rd July, as long as needed : Sprints
> 
> We could have a sprint(an internal one?)  in parallel with the tutorials if we 
> want.
> 
> How long would we like to sprint after the conference? Friday-Sunday?

Just a quick input from my side: when deciding the sprint dates, we should 
keep in mind that ECOOP 2009 starts on Monday 6th July. I will surely be 
there, and possibly other pypyers as well (both Carl Friedrich and Armin 
showed interest e.g.).

ciao,
Anto


From fijall at gmail.com  Wed Jan 28 13:35:25 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 28 Jan 2009 13:35:25 +0100
Subject: [pypy-dev] official announcement of snakebite
Message-ID: <693bc9ab0901280435h7b0ca1d8mc007cd4f5216d197@mail.gmail.com>

Some very good news for us:

http://groups.google.com/group/snakebite-list/browse_thread/thread/6402c23808063cbe

seems like going towards end of windows problems :)

Cheers and thanks to Trent,
fijal