From arigo at tunes.org  Wed Jan  2 12:55:11 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 2 Jan 2008 12:55:11 +0100
Subject: [pypy-dev] llvm.gcroot
Message-ID: <20080102115511.GA24609@code0.codespeak.net>

Hi Eric, hi all,

A field report about the 'llvmgcroot' branch...

I've been playing with ways to remove the "shadow stack" of GC roots.
More precisely, with our own framework GCs, the problem is that in C
there is no portable way to walk the current stack and locate the local
variables containing pointers to GC-managed objects (let alone modify
these pointers safely, in case of a moving GC).  So what we do at the
moment is to push/pop all such pointers in a custom shadow stack around
each call, everywhere.

In LLVM there is a very recently implemented primitive called
llvm.gcroot() which allows us to declare local variables which contain
pointers to GC-managed objects.  Then with the help of a few pages of
custom C++ code to dynamically link with LLVM's native code backend
'llc', we can put extra data into the assembler that describes where
these local variables are and how big each function's stack frame is.
This is not platform-specific so far, in the sense that the same C++
code works on any platform.  There is still a bit of platform-specific
code, which is how exactly to interpret this data to walk the stack from
our GC's collect() methods.

For more info:
  C++ plugin for llc (see README.txt):
    http://codespeak.net/svn/user/arigo/hack/pypy-hack/stackrootwalker
  PyPy branch:
    http://codespeak.net/svn/pypy/branch/llvmgcroot
  with the platform-specific stack walking code:
    http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/rpython/memory/gctransform/llvmgcroot.py

This gives some modest speed-up over a plain llvm translation (which is
slower than a C translation, which is itself slower than a --llvm-via-c,
but the latter doesn't support llvm.gcroot(), obviously, as it goes
through C as well).  To activate this in the branch, use translate.py
--llvmgcroot.

Then I started playing with something similar for genc by parsing the .s
files produced by gcc.  Completely platform-specific, obviously.  To
active this in the branch, use translate.py --llvmgcroot --backend=c
(sorry if it's a bit confusing).  This uses the special 'asm' keyword of
gcc to generate markers that are found by
http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/translator/c/trackgcroot.py
and we get data added to the .s file in the same format as the custom
C++ GC plugin for LLVM.

Definitely an obscure hack, though it's kind of safe in the sense that
if gcc does too advanced optimizations it will not give obscure
segfaults, but clean crashes in trackgcroot.py itself.  It gives the
same modest speed-ups for pypy-c.

In both the C and the LLVM case, the presence of the gcroot markers
currently prevent some low-level optimizations by, respecitvely, llc and
gcc.  The reason is that the local variables that are marked as GC roots
cannot be simply "fast" variables that the compiler is free to put in
any register and move around as needed.  I'm not sure how to fix this
for llc, but for gcc I have the project to try to extend trackgcroot.py.

Finally, if we're talking about maximum performance, it might be
possible (given enough time and obscure hacking) to use trackgcroot.py
on the assembler files produced by gcc during a --llvm-via-c
translation...  by convincing LLVM to put in the C code things that look
like calls, but that are really macros that expand to the asm()
markers...


A bientot,

Armin.


From santagada at gmail.com  Thu Jan  3 22:34:42 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 3 Jan 2008 19:34:42 -0200
Subject: [pypy-dev] Fwd: [pypy-sprint] AVM2 / Tamarin backend at the sprint
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
Message-ID: <2DB03D38-8277-488D-9F37-CD5C150C63DF@gmail.com>

The right place to discuss this is pypy-dev

Begin forwarded message:

> From: Toby Watson <toby at thetobe.com>
> Date: 3 de janeiro de 2008 19h18min11s GMT-02:00
> To: pypy-sprint at codespeak.net
> Subject: [pypy-sprint] AVM2 / Tamarin backend at the sprint
>
> Hi Everyone,
>
> I have a PyPy related idea - one I'd like to explore at the upcoming
> sprint in Leysin.
>
> Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual
> machine shipped in Flash and donated as Tamarin (to be the Mozilla/
> Firefox JS engine). This in turn would open a number of possibilities,
> not least would be seeing a >>>> prompt sitting in a Flash app on a
> web page.
>
> As I understand it from some of the PyPy CLI/JVM docs I have been
> reading it is possible certainly to generate an interpreter, but
> possibly also a compiler for any particular platform, in this case
> using the oobackend.
>
> Just quickly, because I know many people do not know about (or
> care! ;-) what is happening in Flash, the recent version 9 release
> introduced:
>
> * A new virtual machine (AVM2)
> * With JITter compilation
> * Support for prototype and class-based ECMAscript styles (should be
> suitable as Python target)
> * Dynamic bytecode generation _and loading_ is possible (should be
> handy)
> * ActionScript 3/ECMAScript has strong typing and other more mature
> language features
>
> FYI:
> * Bytecode is called Actionscript Byte Code (ABC)
> * Another language (HaXe) has already targeted this VM successfully.
>
> With such a 'Flython' interpreter I would hope in time to be able to
> bind to Adobe's native and library classes and hence by able to :
>
> * Compile swf files from Python (for Flash) source
> * Build Flex apps (Adobe's GUI framework)
> * Target AIR (Flash desktop apps)
>
> About myself, my interests:
>
> 30-something developer of many, many interactive systems using a lot
> of different technology from web Python/PHP, through ActionScript to
> OpenGL/C++, C#, etc. I have written many small parts of compilers and
> language transformers, but never a whole compiler. I have experience
> with Agile development environments and Test-driven development.
>
> I've started digging around using Adobe's flex compiler, the open-
> source Tamarin engine, plus my own small swf decompiler and lots of
> public information to understand how the AVM2 VM works and Adobe's
> compiler infrastructure - separate compilation, linking, the bytecode
> etc.
>
> I am actually interested in developing new language/programming styles
> atop PyPy/Flython (if it did exist) and I like the Flash VM because it
> is web-borne and so widely distributed - essentially people could try
> out experiments in their browser without having to download/build
> anything.
>
> I don't know how much is possible in a week and of course much depends
> on me, but my goal would be to get a skeleton for the interpreter/
> translator and to sound out what all the issues are in targetting the
> AVM, binding to native classes and so on.
>
> I was considering writing another compiler specifically for this task,
> but after looking at Jython, IronPython, Boo and PyPy it became
> apparent that PyPy is quite mature now, may save a lot of time, and
> ultimately produce a more interesting result.
>
> I would hope that by adding another backend I could contribute to the
> fleshing-out of PyPy, help participate in the general maturing of the
> project and hopefully give the group a cool demo too.
>
> So...I hope to see some of you at the sprint and look forward to some
> interesting discussions!
>
> Kind regards,
> Toby


--
Leonardo Santagada





From cfbolz at gmx.de  Thu Jan  3 22:40:35 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 03 Jan 2008 22:40:35 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
Message-ID: <477D5653.1020007@gmx.de>

Hi Toby,

(ccing pypy-dev)

Toby Watson wrote:
> I have a PyPy related idea - one I'd like to explore at the upcoming  
> sprint in Leysin.
> 
> Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual  
> machine shipped in Flash and donated as Tamarin (to be the Mozilla/ 
> Firefox JS engine). This in turn would open a number of possibilities,  
> not least would be seeing a >>>> prompt sitting in a Flash app on a  
> web page.
> 
> As I understand it from some of the PyPy CLI/JVM docs I have been  
> reading it is possible certainly to generate an interpreter, but  
> possibly also a compiler for any particular platform, in this case  
> using the oobackend.
> 
> Just quickly, because I know many people do not know about (or  
> care! ;-) what is happening in Flash, the recent version 9 release  
> introduced:
> 
> * A new virtual machine (AVM2)
> * With JITter compilation
> * Support for prototype and class-based ECMAscript styles (should be  
> suitable as Python target)
> * Dynamic bytecode generation _and loading_ is possible (should be  
> handy)
> * ActionScript 3/ECMAScript has strong typing and other more mature  
> language features
> 
> FYI:
> * Bytecode is called Actionscript Byte Code (ABC)
> * Another language (HaXe) has already targeted this VM successfully.
> 
> With such a 'Flython' interpreter I would hope in time to be able to  
> bind to Adobe's native and library classes and hence by able to :
> 
> * Compile swf files from Python (for Flash) source
> * Build Flex apps (Adobe's GUI framework)
> * Target AIR (Flash desktop apps)
> 
> About myself, my interests:
> 
> 30-something developer of many, many interactive systems using a lot  
> of different technology from web Python/PHP, through ActionScript to  
> OpenGL/C++, C#, etc. I have written many small parts of compilers and  
> language transformers, but never a whole compiler. I have experience  
> with Agile development environments and Test-driven development.
> 
> I've started digging around using Adobe's flex compiler, the open- 
> source Tamarin engine, plus my own small swf decompiler and lots of  
> public information to understand how the AVM2 VM works and Adobe's  
> compiler infrastructure - separate compilation, linking, the bytecode  
> etc.
> 
> I am actually interested in developing new language/programming styles  
> atop PyPy/Flython (if it did exist) and I like the Flash VM because it  
> is web-borne and so widely distributed - essentially people could try  
> out experiments in their browser without having to download/build  
> anything.
> 
> I don't know how much is possible in a week and of course much depends  
> on me, but my goal would be to get a skeleton for the interpreter/ 
> translator and to sound out what all the issues are in targetting the  
> AVM, binding to native classes and so on.
> 
> I was considering writing another compiler specifically for this task,  
> but after looking at Jython, IronPython, Boo and PyPy it became  
> apparent that PyPy is quite mature now, may save a lot of time, and  
> ultimately produce a more interesting result.
> 
> I would hope that by adding another backend I could contribute to the  
> fleshing-out of PyPy, help participate in the general maturing of the  
> project and hopefully give the group a cool demo too.
> 
> So...I hope to see some of you at the sprint and look forward to some  
> interesting discussions!

I think this task is probably sprint-sized and relevant to PyPy. The 
question is whether some of the core-devs would like to work on it, so 
you can pair with that person.

Then there are technical questions too, for example whether the tamarin 
VM is good enough to actually run all the full PyPy Python interpreter 
on top of it.

Cheers,

Carl Friedrich


From santagada at gmail.com  Thu Jan  3 22:54:18 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 3 Jan 2008 19:54:18 -0200
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5653.1020007@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
Message-ID: <F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>


>> Flash, the recent version 9 release
>> introduced:
>>
>> * A new virtual machine (AVM2)
>> * With JITter compilation
>> * Support for prototype and class-based ECMAscript styles (should be
>> suitable as Python target)
>> * Dynamic bytecode generation _and loading_ is possible (should be
>> handy)
>> * ActionScript 3/ECMAScript has strong typing and other more mature
>> language features

Also the latest version does a form of cache of libs, so in the future  
you go to a site with a pypy interpreter and then go to another one  
you download pypy only once if the library was corretly built.

>> FYI:
>> * Bytecode is called Actionscript Byte Code (ABC)
>> * Another language (HaXe) has already targeted this VM successfully.

Is there any assembler for ABC? maybe haxe has one (do they target  
avm2? I always thought it was only the old flash vm).

>> With such a 'Flython' interpreter I would hope in time to be able to
>> bind to Adobe's native and library classes and hence by able to :
>>
>> * Compile swf files from Python (for Flash) source
>> * Build Flex apps (Adobe's GUI framework)
>> * Target AIR (Flash desktop apps)
>>


I think this is a good idea, probably avm2 can run at least RPython  
and maybe this could be the idea, just like the javascript backend.  
This could be quite easier than supporting full python and pypy  
interpreter.

(I don't know if you have read any docs on the pypy site, but RPyton  
is the restricted python in wich the pypy interpreter is made on,  
suporting only it on avm2 would make programing for flex fun).

[]'s
--
Leonardo Santagada




From toby at thetobe.com  Thu Jan  3 23:07:29 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:07:29 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5653.1020007@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
Message-ID: <312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>

>> [much snippage of intro to Tamarin/AVM2]

> I think this task is probably sprint-sized and relevant to PyPy. The  
> question is whether some of the core-devs would like to work on it,  
> so you can pair with that person.
>
> Then there are technical questions too, for example whether the  
> tamarin VM is good enough to actually run all the full PyPy Python  
> interpreter on top of it.
>
> Cheers,
>
> Carl Friedrich

Hi,

What do you think may be the tripping points?

- some datatype support problems? surely this is mostly a case of  
emulating the correct types modulo gross problems with the 'native'  
AVM types.

- speed. The new VM is a lot faster than the old one and has a JITer.  
I know that this has a go at all non-constructor methods as classes  
are loaded and can be very fast. AS3 is now running things like the  
Papervision3D library which is pretty CPU intensive, though I admit  
making particular type of requirements on the VM.

- some piece of machinery like the bytecode verifier getting in the  
way? AFAIK this just does relatively simple overflow checks and the  
like.

- I've also seen small schemes/LISPs running on the AVM so there's  
some confidence there.
Infact I've generated my own images using a basic stack machine  
interpreter (a bit like the PyPy JIT test toy). Additionally I've  
generated code at runtime, which is then dynamically loaded, bound to  
and executed to make the same images.

With regard to support I was mainly not expecting any unless this  
effort started to show some progress on my part and interest from  
others. However I expected to benefit from any intro talks (I think  
some were muted on 13/14?) and talking to other developers who have  
written backends. Not to mention general environment of  
industriousness ;-) Of course I'd be delighted if anyone else did  
become interested in it!

cheers,
Toby


From cfbolz at gmx.de  Thu Jan  3 23:11:42 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 03 Jan 2008 23:11:42 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
Message-ID: <477D5D9E.9060505@gmx.de>

Toby Watson wrote:
>>> [much snippage of intro to Tamarin/AVM2]
> 
>> I think this task is probably sprint-sized and relevant to PyPy. The 
>> question is whether some of the core-devs would like to work on it, so 
>> you can pair with that person.
>>
>> Then there are technical questions too, for example whether the 
>> tamarin VM is good enough to actually run all the full PyPy Python 
>> interpreter on top of it.
>>
>> Cheers,
>>
>> Carl Friedrich
> 
> Hi,
> 
> What do you think may be the tripping points?
> 
> - some datatype support problems? surely this is mostly a case of 
> emulating the correct types modulo gross problems with the 'native' AVM 
> types.
> 
> - speed. The new VM is a lot faster than the old one and has a JITer. I 
> know that this has a go at all non-constructor methods as classes are 
> loaded and can be very fast. AS3 is now running things like the 
> Papervision3D library which is pretty CPU intensive, though I admit 
> making particular type of requirements on the VM.

My gut feeling is the pure size of our Python interpreter, but maybe I 
am pessimistic. I think in terms of features we should be fine.

> - some piece of machinery like the bytecode verifier getting in the way? 
> AFAIK this just does relatively simple overflow checks and the like.
> 
> - I've also seen small schemes/LISPs running on the AVM so there's some 
> confidence there.
> Infact I've generated my own images using a basic stack machine 
> interpreter (a bit like the PyPy JIT test toy). Additionally I've 
> generated code at runtime, which is then dynamically loaded, bound to 
> and executed to make the same images.
> 
> With regard to support I was mainly not expecting any unless this effort 
> started to show some progress on my part and interest from others. 
> However I expected to benefit from any intro talks (I think some were 
> muted on 13/14?) and talking to other developers who have written 
> backends. Not to mention general environment of industriousness ;-) Of 
> course I'd be delighted if anyone else did become interested in it!

Well, the usual mode of programming at a sprint (and one of the main 
points of doing one) is to work on a problem in pairs. That's why it 
would be nice if we found somebody else who is interested in this. Of 
course the general environment of industriousness and being able to ask 
questions all the time already helps :-).

Cheers,

Carl Friedrich


From toby at thetobe.com  Thu Jan  3 23:42:26 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:42:26 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
Message-ID: <A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>

> Also the latest version does a form of cache of libs, so in the  
> future you go to a site with a pypy interpreter and then go to  
> another one you download pypy only once if the library was corretly  
> built.

Yup! Could be cool. Although my software often changes too much for  
caching to help ;-)

>>> FYI:
>>> * Bytecode is called Actionscript Byte Code (ABC)
>>> * Another language (HaXe) has already targeted this VM successfully.
>
> Is there any assembler for ABC? maybe haxe has one (do they target  
> avm2? I always thought it was only the old flash vm).

Certainly haXe has a generator library which I've used before, I've  
rewritten that lib in AS3 before so a Python port is not out of the  
question.

haXe does target AVM2: http://tinyurl.com/33t2ox . Certainly there's  
plenty of reference to it in the compiler source (O'Caml) that I've  
been looking at today.

Also the Adobe Flex 3 (will be OSS at some point) compiler can be  
coerced to emit a kind of Intermediate Representation, though I don't  
think it will assemble from it. Looks like this:

// ++StartProgram
// ++StartMethod $init
AllocTemp 1
LoadThis [1]
PushScope [0]
FindProperty print 3 { (package-internal) public test.as$0(private) }  
is_strict=true is_qualified=false is_attr=false [1]
PushString hello world [2]
CallProperty print 3 { (package-internal) public test.as$0(private) }  
1 is_qualified=false is_super=false is_attr=false [1]
CheckType * [1]
StoreRegister 1, * [0]
LoadRegister 1, * [1]
Return [0]
FreeTemp 1
// --FinishMethod $init
// --FinishProgram

> I think this is a good idea, probably avm2 can run at least RPython  
> and maybe this could be the idea, just like the javascript backend.  
> This could be quite easier than supporting full python and pypy  
> interpreter.
>
> (I don't know if you have read any docs on the pypy site, but RPyton  
> is the restricted python in wich the pypy interpreter is made on,  
> suporting only it on avm2 would make programing for flex fun).

I'm busy reading them now. I started with the recent some of the  
Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
  and Antonio Cuni's thesis on the CLI backend. I'll happily look at  
any other recommendations.

I know of the idea of RPython and am familar with similar in SmallTalk  
from their squeak translator. That said I haven't yet stumbled across  
any document saying what the Restrictions are...

oh here, "2.1   RPython Definition, not"

> suporting only it [RPython] on avm2 would make programing for flex  
> fun).


I think you've hit on a key point. This aspect for me would be about  
producing something that is fun...if it turns into something  
productive and worthy later on then so be it. I'm also optimistic that  
the VM could - in time - support some decent optimizations.

But baby steps first...

Toby





From toby at thetobe.com  Thu Jan  3 23:42:52 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:42:52 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5D9E.9060505@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
	<477D5D9E.9060505@gmx.de>
Message-ID: <88DAF02E-E783-4E40-A3B2-CF700B0BB83D@thetobe.com>

>> - speed. The new VM is a lot faster than the old one and has a  
>> JITer. I know that this has a go at all non-constructor methods as  
>> classes are loaded and can be very fast. AS3 is now running things  
>> like the Papervision3D library which is pretty CPU intensive,  
>> though I admit making particular type of requirements on the VM.
>
> My gut feeling is the pure size of our Python interpreter, but maybe  
> I am pessimistic. I think in terms of features we should be fine.

I did once have to change a few of the SWF defaults like  
max_stack_depth and is_this_script_dead_timeout? for a client project  
involving 100's of classes. But that was with the old VM ;-)

> Well, the usual mode of programming at a sprint (and one of the main  
> points of doing one) is to work on a problem in pairs. That's why it  
> would be nice if we found somebody else who is interested in this.  
> Of course the general environment of industriousness and being able  
> to ask questions all the time already helps :-).

Agreed that pairing is often super-productive.

Toby


From toby at thetobe.com  Fri Jan  4 00:12:01 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 23:12:01 +0000
Subject: [pypy-dev] AVM2 / Tamarin backend at the sprint
Message-ID: <21849469-E463-4860-9462-394787EDC786@thetobe.com>

Since there seems to be a little discussion here, may I ask:

What would be a useful code / piece of understanding to bring along?

Things I've been thinking about:

* A Python lib for generating ABC byte code, ABC byte code files (to  
run on Tamarin) and SWFs (for Flash).

* An understanding of how class linking / loading environment works in  
Tamarin / AVM2.

* ??? What else to figure out on the target side ???

I'm thinking that I want to generate ABC and produce ABC files. They  
are executable by the avmplus test runtime which is part of Tamarin.  
There is also an Actionscript 3 unit test framework, so some  
arrangement of these could be used to run a test suite.

That said I'd like to go further to wrappering the ABC bytecode in a  
SWF because I think it will have a lot more impact / is cuter at this  
time to be able to see something in a browser.

For anyone who's curious:

AVM2 overview:
www.adobe.com/devnet/actionscript/ articles/avm2overview.pdf

hxasm - assembler library (with example)
http://haxe.org/hxasm

ABC bytecodes here: (referenced from above)
http://www.anotherbigidea.com/javaswf/avm2/AVM2Instructions.html

cheers,
Toby


From anto.cuni at gmail.com  Fri Jan  4 00:20:04 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 00:20:04 +0100
Subject: [pypy-dev] JVM integration
In-Reply-To: <9c0bb8a00712310914k17265838w26666b09b29fce0d@mail.gmail.com>
References: <9c0bb8a00712310914k17265838w26666b09b29fce0d@mail.gmail.com>
Message-ID: <477D6DA4.6040308@gmail.com>

Paul deGrandis wrote:
> Anto, Niko, and All,

HI Paul,

> I have begun working on PyPy again and am starting on jvm integration.
>  After talking briefly with Carl on IRC, I looked at the code in CLI.
> 
> I have a few questions:
> It appears as though the general approach is to identify entry points
> into CLI code (via dlls), use reflection, and make calls.  This is
> also later used to make (r)python modules into dlls themselves
> (allowing for bidirectional integration).  Is this correct?  

I'm not sure to understand what you mean, so I briefly explain how 
things work for CLI:

   1. (interp-level) you can use .NET libraries from RPython; this means 
that in theory you can do in RPython everything you do in C# (modulo the 
fact that the support is not complete right now, but it could be in the 
future)

   2. (app-level) you can use .NET libraries from pypy-cli; to do this, 
you go through the clr module: part of such a module is written in 
RPython and contains calls to .NET reflection API, which are used to 
dynamically access .NET libraries from python

Obviously, point 2 depends on point 1 to work.

In the source code, point 1 is implemented by translator/cli/dotnet.py, 
while point 2 is implemented by module/clr/*.py.

In both cases, probably a lot of code can be shared between the CLI and 
JVM versions; so, the first thing to do would be to split dotnet.py into 
a backend-independent part (probably to be put in translator/oosupport/) 
and the CLI specific part, then write the JVM specific code. The same 
for the code in module/clr/.

 > Also, how
 > does the notion of PythonNet play into this (ie: where do you get the
 > CLR module from)?

Look again at the point (1) above; ok, we can use .NET libraries in 
RPython; consider this RPython snippet:

def main():
     System.Console.WriteLine("Hello World")

when we translate it trough gencli, we get a .NET exe whose bytecode 
contains a call to System.Console::WriteLine, and everything is fine.

On the other hand, one of the greatest features of RPython is that we 
can test RPython programs directly in CPython, without the need to 
translate them; as you probably already know, if you type e.g. py.py you 
run the PyPy interpreter on top of CPython, which is extremely good for 
testing.

Now, suppose you want to run the snippet above on top of CPython, i.e. 
*without* translating it; how can it work? CPython can't call 
System.Console.WriteLine, can it?

This is where PythonNet come to the action: it allows CPython to access 
.NET libraries; so, even if we can't run that snippet directly on 
CPython, we can run it on PythonNet (which is nothing else than a thin 
wrapper of CPython).

To be clearer, the only role that PythonNet plays is to allow testing 
without the need of a translation. If you look at 
translator/cli/test/test_dotnet.py, you can see the TestPythonnet class: 
this class runs all the tests in TestDotnetRtyping to ensure that every 
snippet behave the same when translated and when interpreted.

There are equivalent of PythonNet for JVM; I think the most promising is 
JPype, which Carl Friedrich tried some time ago.

> If this is not the current approach, is there a document that someone
> can point me to that talks about this in further detail?
> Lastly, as work for integration on the JVM begins, a larger goal is
> the generalize the approach.  Has any work been done in the CLI/CLR to
> generalize the approach so far?

I wrote that code with JVM in mind, so I think it wouldn't be too hard 
to split the CLR specific parts from the general purpose parts.


ciao Anto


From anto.cuni at gmail.com  Fri Jan  4 00:29:18 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 00:29:18 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
Message-ID: <477D6FCE.9060700@gmail.com>

Hi Toby,

Toby Watson wrote:

> I'm busy reading them now. I started with the recent some of the  
> Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
>   and Antonio Cuni's thesis on the CLI backend. I'll happily look at  
> any other recommendations.

I was just writing you a mail pointing to that documents, but I see you 
already found them :-).

Be warn that my thesis is a bit outdated, because a lot of things have 
been improved since then. In particular, we added a new backend for the 
JVM which shares most of the code with the CLI backend; the good news is 
that almost certainly such code can be reused also for a new tamarin 
backend, meaning that writing it would be much easier than what was 
writing gencli and genjvm.

If you want to look at the source, the backend independent code is 
located in translator/oosupport/; each of the files there contains 
classes that are subclassed by the code in translator/{cli,jvm}.

Feel free to ask questions either here or on #pypy, I'll be glad to answer.

ciao Anto


From toby at thetobe.com  Fri Jan  4 01:06:18 2008
From: toby at thetobe.com (Toby Watson)
Date: Fri, 4 Jan 2008 00:06:18 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D6FCE.9060700@gmail.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
	<477D6FCE.9060700@gmail.com>
Message-ID: <CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>

Hi Antonio,

Thanks for the advice and pointers into the code.

Would you say this is still a fair assessment of the tasks that have  
to be done to target a new backend? : (pulled from PyPy[cli-backend])

	? map ootypesystem's types to CLI Common Type System's types;

	? map ootypesystem's low level operation to CLI instructions;

	? map Python exceptions to CLI exceptions;

	? write a code generator that translates a flow graph into a list of  
CLI instructions;

	? write a class generator that translates ootypesystem classes into  
CLI classes.

Was this pre or post the refactoring you describe?

cheers,
Toby


On 3 Jan 2008, at 23:29, Antonio Cuni wrote:

> Hi Toby,
>
> Toby Watson wrote:
>
>> I'm busy reading them now. I started with the recent some of the   
>> Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
>>    and Antonio Cuni's thesis on the CLI backend. I'll happily look  
>> at  any other recommendations.
>
> I was just writing you a mail pointing to that documents, but I see  
> you already found them :-).
>
> Be warn that my thesis is a bit outdated, because a lot of things  
> have been improved since then. In particular, we added a new backend  
> for the JVM which shares most of the code with the CLI backend; the  
> good news is that almost certainly such code can be reused also for  
> a new tamarin backend, meaning that writing it would be much easier  
> than what was writing gencli and genjvm.
>
> If you want to look at the source, the backend independent code is  
> located in translator/oosupport/; each of the files there contains  
> classes that are subclassed by the code in translator/{cli,jvm}.
>
> Feel free to ask questions either here or on #pypy, I'll be glad to  
> answer.
>
> ciao Anto



From drfarina at gmail.com  Fri Jan  4 10:46:40 2008
From: drfarina at gmail.com (Daniel Farina)
Date: Fri, 04 Jan 2008 01:46:40 -0800
Subject: [pypy-dev] interrupts, preemption, threads
Message-ID: <1199440000.8858.9.camel@tenacity>

Hello list,

I wasn't able to locate any of the keywords ['interrupt',
'preemption', 'thread'] in the web documentation, so I come to
the list to ask what is roughly the status of these things in the
generated PyPy virtual machines on whatever applicable platforms.

What I am not asking about is PyPy's Python implemention of
threads, but rather support for those writing their own
interpreters.

Example scenarios:

- timeouts on arbitrary code fragments (conflated by the fact
that non-terminating interrupt-disabling code will allow code to
ignore the timeout, but I digress)

- writing a preemptive scheduler of closures

- making use of two processors 

- triggering garbage collections

fdr




From arigo at tunes.org  Fri Jan  4 12:35:50 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 4 Jan 2008 12:35:50 +0100
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <1199440000.8858.9.camel@tenacity>
References: <1199440000.8858.9.camel@tenacity>
Message-ID: <20080104113550.GA26548@code0.codespeak.net>

Hi Daniel,

On Fri, Jan 04, 2008 at 01:46:40AM -0800, Daniel Farina wrote:
> I wasn't able to locate any of the keywords ['interrupt',
> 'preemption', 'thread'] in the web documentation, so I come to
> the list to ask what is roughly the status of these things in the
> generated PyPy virtual machines on whatever applicable platforms.

We only have limited support for threads at the moment.  I think we have
no specific support for high-level backends (JVM, CLI).  On the
low-level backends (C, LLVM), you can use thread primitives to start
threads and use locks as you would do in C.  In addition there is
support in the translator for automatically inserting custom code before
and after every external function call.  We use this in PyPy's Python
interpreter: it uses a global interpreter lock (GIL), which is released
and reacquired around each external function call.  (This is similar to
CPython but we don't have to put the release/reacquire code everywhere
by hand.)

> - timeouts on arbitrary code fragments
> - writing a preemptive scheduler of closures
> - making use of two processors 
> - triggering garbage collections

We have no specific support for all this.  Multiple threads will run on
multiple processors just like in C.  Timeouts can be done in a
platform-specific way with signals.  For the rest, our custom garbage
collectors are not thread-safe, at least yet; only the Boehm collector
is, but its performance is not very good in multithreaded programs.


A bientot,

Armin.


From drfarina at gmail.com  Fri Jan  4 13:03:55 2008
From: drfarina at gmail.com (Daniel Farina)
Date: Fri, 04 Jan 2008 04:03:55 -0800
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <20080104113550.GA26548@code0.codespeak.net>
References: <1199440000.8858.9.camel@tenacity>
	<20080104113550.GA26548@code0.codespeak.net>
Message-ID: <1199448235.8858.16.camel@tenacity>


On Fri, 2008-01-04 at 12:35 +0100, Armin Rigo wrote:
> Hi Daniel,
> 
> On Fri, Jan 04, 2008 at 01:46:40AM -0800, Daniel Farina wrote:
> > I wasn't able to locate any of the keywords ['interrupt',
> > 'preemption', 'thread'] in the web documentation, so I come to
> > the list to ask what is roughly the status of these things in the
> > generated PyPy virtual machines on whatever applicable platforms.
> 
> We only have limited support for threads at the moment.  I think we have
> no specific support for high-level backends (JVM, CLI).  On the
> low-level backends (C, LLVM), you can use thread primitives to start
> threads and use locks as you would do in C.

That seems like a pretty useful subset already...

>   In addition there is
> support in the translator for automatically inserting custom code before
> and after every external function call.  We use this in PyPy's Python
> interpreter: it uses a global interpreter lock (GIL), which is released
> and reacquired around each external function call.  (This is similar to
> CPython but we don't have to put the release/reacquire code everywhere
> by hand.)
> 

Nifty. Noted.

> > - timeouts on arbitrary code fragments
> > - writing a preemptive scheduler of closures
> > - making use of two processors 
> > - triggering garbage collections
> 
> We have no specific support for all this.  Multiple threads will run on
> multiple processors just like in C.  Timeouts can be done in a
> platform-specific way with signals.  For the rest, our custom garbage
> collectors are not thread-safe, at least yet; only the Boehm collector
> is, but its performance is not very good in multithreaded programs.
> 

Is there going to be some cross-platform support for this? 

As for examples, I imagine pypy/module/signal[/test] is the place to go,
correct?
 
Thanks,
fdr




From arigo at tunes.org  Fri Jan  4 15:24:34 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 4 Jan 2008 15:24:34 +0100
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <1199448235.8858.16.camel@tenacity>
References: <1199440000.8858.9.camel@tenacity>
	<20080104113550.GA26548@code0.codespeak.net>
	<1199448235.8858.16.camel@tenacity>
Message-ID: <20080104142434.GA7682@code0.codespeak.net>

Hi Daniel,

On Fri, Jan 04, 2008 at 04:03:55AM -0800, Daniel Farina wrote:
> > > - timeouts on arbitrary code fragments
> > > - writing a preemptive scheduler of closures
> > > - making use of two processors 
> > > - triggering garbage collections
> > 
> > We have no specific support for all this.  Multiple threads will run on
> > multiple processors just like in C.  Timeouts can be done in a
> > platform-specific way with signals.  For the rest, our custom garbage
> > collectors are not thread-safe, at least yet; only the Boehm collector
> > is, but its performance is not very good in multithreaded programs.
> > 
> 
> Is there going to be some cross-platform support for this? 

Thread-safe GCs are probably the biggest blocker.  To some extend we
will also need something for PyPy, although at first we can probably
just rely on the GIL to avoid concurrent GC calls.  So all the more
advanced stuff - including thread-safe or even truly concurrent GC - is
not really likely to be done any time soon by any of us, I imagine.  On
the other hand we're always open to contributions :-)

> As for examples, I imagine pypy/module/signal[/test] is the place to go,
> correct?

Yes, interp_signal.py contains all the necessary declaration, although
they are mixed up with gluing code specific to PyPy's Python
interpreter.  The first step would be to split this file and move the
general RPython-only code to pypy/rlib/rsignal.py, and keep only the
gluing in interp_signal.py; this is what we have already done with other
modules like _socket.


A bientot,

Armin.


From anto.cuni at gmail.com  Fri Jan  4 16:04:44 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 16:04:44 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
	<477D6FCE.9060700@gmail.com>
	<CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>
Message-ID: <477E4B0C.2020008@gmail.com>

Toby Watson wrote:
> Hi Antonio,

Hi Toby,

> Thanks for the advice and pointers into the code.
> 
> Would you say this is still a fair assessment of the tasks that have to 
> be done to target a new backend? : (pulled from PyPy[cli-backend])
> 
>     ? map ootypesystem's types to CLI Common Type System's types;
> 
>     ? map ootypesystem's low level operation to CLI instructions;
> 
>     ? map Python exceptions to CLI exceptions;
> 
>     ? write a code generator that translates a flow graph into a list of 
> CLI instructions;
> 
>     ? write a class generator that translates ootypesystem classes into 
> CLI classes.
> 
> Was this pre or post the refactoring you describe?

this list was written before the refactoring, but I think it's more or 
less still valid.

First of all, you need a strong understanding of both ootypesystem and 
the type system of the VM you want to targets: then you can think how to 
map the types; for CLI and JVM it was mostly straightforward, but maybe 
it wouldn't be so for AVM2, e.g. if it doesn't support classes in the 
same way ootypesystem expects.

After you have mapped types, mapping instructions would be just a matter 
of coding but it shouldn't be too hard.

The paragraph about exceptions is mostly historical; at least at the 
beginning, you shouldn't need to do anything special about exceptions as 
long as you keep the hierarchy of RPython exceptions separated from the 
hierarchy of the VM exceptions.

The cool thing is that the hardest point (code generator) have been 
already implemented in oosupport, and it's very easy to subclass it for 
targeting yet another VM.

If you want to have a closer look to how each point is implemented, here 
are some pointers in the code:

   * mapping types
       - for cli, it's cli/cts.py; the main entry-point is the function 
lltype_to_cts

       - for jvm, it's jvm/typesystem.py; however, the main entry-point 
is the function lltype_to_cts in jvm/database.py


   * mapping operations: see cli/opcodes.py and jvm/opcodes.py

   * code generator: the base class is in oosupport/function.py, 
subclassed in cli/function.py and jvm/nodes.py (class GraphFunction); 
the Function class knows how to deal with graphs, but it delegates the 
actual code generation to something specific to each backend; for cli, 
it's in cli/ilgenerator.py, for jvm it's the GraphFunction class itself.

   * class generator: cli/class_.py and the Class class in jvm/node.py

ciao Anto


From exarkun at divmod.com  Thu Jan 10 15:15:13 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 10 Jan 2008 09:15:13 -0500
Subject: [pypy-dev] CLI on Windows
In-Reply-To: 0
Message-ID: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>

Hello all,

I've played around a little bit with PyPy and Mono on Linux.  Now I'm
wondering what the requirements are to use/test the CLI features of PyPy
on Windows.

Can anyone give a list of the dependencies PyPy/CLI has on Windows?

Jean-Paul


From anto.cuni at gmail.com  Thu Jan 10 18:13:48 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 10 Jan 2008 18:13:48 +0100
Subject: [pypy-dev] CLI on Windows
In-Reply-To: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>
References: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>
Message-ID: <4786524C.90707@gmail.com>

Jean-Paul Calderone wrote:
> Hello all,
> 
> I've played around a little bit with PyPy and Mono on Linux.  Now I'm
> wondering what the requirements are to use/test the CLI features of PyPy
> on Windows.
> 
> Can anyone give a list of the dependencies PyPy/CLI has on Windows?

Hi Jean-Paul,

the only requirement is that .NET 2.0 is installed and that csc.exe and 
ilasm.exe are on the path. I've never tried with newer versions of .NET, 
it might work or not.

Moreover, some time ago I encountered one problem with microsoft ilasm 
which prevented pypy-cli to be compiled; I don't know if the problem 
it's still there. The workaround is to use mono's ilasm to assemble the 
bytecode, then run the resulting exe as usual; if pypy finds that mono 
is installed, it will automatically use mono's ilasm instead of 
microsoft's one.

ciao Anto


From toby at thetobe.com  Fri Jan 11 12:19:08 2008
From: toby at thetobe.com (Toby Watson)
Date: Fri, 11 Jan 2008 11:19:08 +0000
Subject: [pypy-dev] Blog / Parser
Message-ID: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>

Hi,

I've just read the blog post, "Visualizing a Python tokenizer" and it  
reminded me of this:

"OMeta: an object oriented language for pattern matching"
http://www.cs.ucla.edu/~awarth/papers/dls07.pdf

OMeta is an extension and generalisation of the idea of PEGs*. It  
provides a nice way to describe a language both at the character level  
(tokens), the grammar itself and productions into the AST. Finally the  
grammars are extensible (possibly from within the language itself).

The implementation is discussed in "Packrat Parsers Can Support Left  
Recursion" and there is some discussion of the performance there. http://www.vpri.org/pdf/packrat_TR-2007-002.pdf

I wonder whether the same idea behind PyPy can be applied to the  
grammar. Write a program in some language (a python version of OMeta  
for instance) which is then transformed by the translator, compiler,  
or JIT into something that runs fast.

What could be nice about this is bringing the tokenising and parsing  
closer in spirit to the heart of PyPy, writing 'nicer' code, and  
providing a (I think tantalising) way to try new syntax going forward.

And there are things to play with on this page:
http://www.cs.ucla.edu/~awarth/ometa/ometa-js/

* Parsing Expression Grammar


With regard to railroad diagrams (I think that's what they're called):

There used to be a script that generated them - it's mentioned at the  
top of the python grammar file, and here http://www.python.org/search/hypermail/python-1994q3/0294.html

But I've seen discussion elsewhere that it has been lost :(

How about this? http://www.informatik.uni-freiburg.de/~thiemann/haskell/ebnf2ps/README


cheers,
Toby


From cfbolz at gmx.de  Fri Jan 11 22:22:13 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 11 Jan 2008 22:22:13 +0100
Subject: [pypy-dev] Blog / Parser
In-Reply-To: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>
References: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>
Message-ID: <4787DE05.30601@gmx.de>

Hi Toby,

Toby Watson wrote:
 > I've just read the blog post, "Visualizing a Python tokenizer" and it
 > reminded me of this:
 >
 > "OMeta: an object oriented language for pattern matching"
 > http://www.cs.ucla.edu/~awarth/papers/dls07.pdf
 >
 > OMeta is an extension and generalisation of the idea of PEGs*. It
 > provides a nice way to describe a language both at the character level
 > (tokens), the grammar itself and productions into the AST. Finally the
 > grammars are extensible (possibly from within the language itself).

Yes, I know of OMeta and this paper. It contains some nice things, but
in my opinion there is quite some hype around it too (and the paper is
at least missing a good "Prior work" section):

  - semantic-wise (apart from supporting left-recursion) their parsers
    are not more powerful than simple recursive-descent parsers with
    backtracking

  - it's not clear that the memoizing that packrat parsers do is a good
    idea for practical parsers in general. See this paper:
http://www.cs.mu.oz.au/research/mercury/information/papers.html#packrat-padl08

  - Prolog supported matching on non-stream data structures long ago
    _and_ is more powerful. It is also as efficient as packrat parsers if
    you use tabling (which is not a standard Prolog feature admittedly).

  - while I have not actually tried OMeta I somehow fear that debugging
    these programs if they behave unexpectedly is extremely annoying.
    Backtracking with first-match semantics can lead to unpleasant
    surprises and it's not always easy to see what is going on.

So what is left is some nice syntax, the inheritance support (which you
can rather easily get with a manually written recursive descent parser
too) and left recursion support.

 > The implementation is discussed in "Packrat Parsers Can Support Left
 > Recursion" and there is some discussion of the performance there. 
http://www.vpri.org/pdf/packrat_TR-2007-002.pdf

I have only glanced at it, I guess I should read it in more detail.
FWIW, the left-recursion support is basically exactly what a tabling
Prolog does, just less general.

 > I wonder whether the same idea behind PyPy can be applied to the
 > grammar. Write a program in some language (a python version of OMeta
 > for instance) which is then transformed by the translator, compiler,
 > or JIT into something that runs fast.

Maybe, yes. FWIW, we support simple packrat parsers, see here:

http://codespeak.net/pypy/dist/pypy/doc/rlib.html#parsing

It's far from OMeta, just a fairly boring parser generator. It contains
good support for regular expressions, though, which is where the diagram
from the blog post comes from.

I also experimented a bit with something OMeta-like in PyPy, you can
look at the tests here:

http://codespeak.net/svn/pypy/dist/pypy/rlib/parsing/test/test_pypackrat.py

It supports the left recursion and the composability. I am not sure it
is a good idea, though. The resulting code is RPython and can be
translated to mostly efficient C.

 > What could be nice about this is bringing the tokenising and parsing
 > closer in spirit to the heart of PyPy, writing 'nicer' code, and
 > providing a (I think tantalising) way to try new syntax going forward.

We don't really use any advanced stuff for our Python parser because we
want to be able to use the CPython's grammar files, which contain LL(1)
grammars anyway, so why bother with anything advanced? If someone wants
to support user-modifiable grammars or something like that he can
probably do so easily with PyPy. The PyPy core team is usually keeping
out of such "language-designy" topics (although I guess I would be
interested at least on the implementation-side of things if somebody
stepped forward to go in such a direction).

 >
 > And there are things to play with on this page:
 > http://www.cs.ucla.edu/~awarth/ometa/ometa-js/
 >
 > * Parsing Expression Grammar
 >
 >
 > With regard to railroad diagrams (I think that's what they're called):
 >
 > There used to be a script that generated them - it's mentioned at the
 > top of the python grammar file, and here 
http://www.python.org/search/hypermail/python-1994q3/0294.html
 >
 > But I've seen discussion elsewhere that it has been lost :(
 >
 > How about this? 
http://www.informatik.uni-freiburg.de/~thiemann/haskell/ebnf2ps/README

ebnf2ps seems to be using haskell that is no longer liked by modern 
haskell compilers. I don't know any haskell but whacking at the source a 
bit produced some sort of executable, haven't tried to use it yet.

Cheers,

Carl Friedrich


From michaelschneider at current.net  Sat Jan 12 21:12:17 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sat, 12 Jan 2008 15:12:17 -0500
Subject: [pypy-dev] Performance Regression with SVN version 50525
Message-ID: <47891F21.7080100@current.net>

Hello All,


The pypy c translation time doubled for me with SVN revision 50525



(2:13:48 PM) *bigdog:* 50524:
(2:13:48 PM) *bigdog:* Timer] Timings:
(2:13:48 PM) *bigdog:* Timer] annotate - 318.9 s
(2:13:48 PM) *bigdog:* Timer] rtype_lltype - 174.2 s
(2:13:48 PM) *bigdog:* Timer] backendopt_lltype - 183.3 s
(2:13:48 PM) *bigdog:* Timer] stackcheckinsertion_lltype - 62.6 s
(2:13:48 PM) *bigdog:* Timer] database_c - 128.2 s
(2:13:48 PM) *bigdog:* Timer] source_c - 69.3 s
(2:13:48 PM) *bigdog:* Timer] compile_c - 0.1 s
(2:13:48 PM) *bigdog:* Timer] ========================================
(2:13:48 PM) *bigdog:* Timer] Total: - 936.6 s


(2:55:29 PM) *bigdog:* 50525 timings:
(2:55:32 PM) *bigdog:* [Timer] Timings:
(2:55:32 PM) *bigdog:* [Timer] annotate - 335.2 s
(2:55:32 PM) *bigdog:* [Timer] rtype_lltype - 174.3 s
(2:55:32 PM) *bigdog:* [Timer] backendopt_lltype - 183.5 s
(2:55:32 PM) *bigdog:* [Timer] stackcheckinsertion_lltype - 1066.0 s
(2:55:32 PM) *bigdog:* [Timer] database_c - 126.3 s
(2:55:32 PM) *bigdog:* [Timer] source_c - 70.4 s
(2:55:32 PM) *bigdog:* [Timer] compile_c - 0.1 s
(2:55:32 PM) *bigdog:* [Timer] =========================================
(2:55:32 PM) *bigdog:* [Timer] Total: - 1955.8 s



The translation spent inordinate amount of time in:
(2:37:52 PM) *bigdog:* 
..+*%+................................*%%..................................... 

(2:37:52 PM) *bigdog:* [backendopt:malloc] removed 3922 simple mallocs 
in total
(2:37:52 PM) *bigdog:* [backendopt:mergeifblocks] starting to merge if 
blocks
(2:37:52 PM) *bigdog:* 
..+*%+................................*%%..............
(2:37:52 PM) *bigdog:* [translation:info] inserting stack checks...
(2:37:52 PM) *bigdog:* ..
(2:37:52 PM) *bigdog:* [rtyper] -=- specialized 6 more blocks -=-



Translation memory use went from ~350 meg to ~1.3 gb  (memory use are 
not exact, I just looked at the perfomrnace monitor when runnning)

Hope this is helpful,
Mike


From michaelschneider at current.net  Sun Jan 13 16:26:12 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sun, 13 Jan 2008 10:26:12 -0500
Subject: [pypy-dev] Performance Regression with SVN version 50525
In-Reply-To: <47891F21.7080100@current.net>
References: <47891F21.7080100@current.net>
Message-ID: <478A2D94.5070207@current.net>

Looks like Carls' fix took care of the issue:

(10:23:58 AM) *bigdog:* cfbolts: Translation timings for 50557
(10:24:01 AM) *bigdog:* [Timer] Timings:
(10:24:01 AM) *bigdog:* [Timer] annotate - 329.5 s
(10:24:01 AM) *bigdog:* [Timer] rtype_lltype - 172.7 s
(10:24:01 AM) *bigdog:* [Timer] backendopt_lltype - 182.5 s
(10:24:01 AM) *bigdog:* [Timer] stackcheckinsertion_lltype - 12.3 s
(10:24:01 AM) *bigdog:* [Timer] database_c - 124.8 s
(10:24:01 AM) *bigdog:* [Timer] source_c - 67.7 s
(10:24:01 AM) *bigdog:* [Timer] compile_c - 0.1 s
(10:24:01 AM) *bigdog:* [Timer] ========================================
(10:24:01 AM) *bigdog:* [Timer] Total: - 889.5 s


Very good work, amazing turnaround time.

Thanks,
Mike

Michael Schneider wrote:
> Hello All,
>
>
> The pypy c translation time doubled for me with SVN revision 50525
>
>
>
> (2:13:48 PM) *bigdog:* 50524:
> (2:13:48 PM) *bigdog:* Timer] Timings:
> (2:13:48 PM) *bigdog:* Timer] annotate - 318.9 s
> (2:13:48 PM) *bigdog:* Timer] rtype_lltype - 174.2 s
> (2:13:48 PM) *bigdog:* Timer] backendopt_lltype - 183.3 s
> (2:13:48 PM) *bigdog:* Timer] stackcheckinsertion_lltype - 62.6 s
> (2:13:48 PM) *bigdog:* Timer] database_c - 128.2 s
> (2:13:48 PM) *bigdog:* Timer] source_c - 69.3 s
> (2:13:48 PM) *bigdog:* Timer] compile_c - 0.1 s
> (2:13:48 PM) *bigdog:* Timer] ========================================
> (2:13:48 PM) *bigdog:* Timer] Total: - 936.6 s
>
>
> (2:55:29 PM) *bigdog:* 50525 timings:
> (2:55:32 PM) *bigdog:* [Timer] Timings:
> (2:55:32 PM) *bigdog:* [Timer] annotate - 335.2 s
> (2:55:32 PM) *bigdog:* [Timer] rtype_lltype - 174.3 s
> (2:55:32 PM) *bigdog:* [Timer] backendopt_lltype - 183.5 s
> (2:55:32 PM) *bigdog:* [Timer] stackcheckinsertion_lltype - 1066.0 s
> (2:55:32 PM) *bigdog:* [Timer] database_c - 126.3 s
> (2:55:32 PM) *bigdog:* [Timer] source_c - 70.4 s
> (2:55:32 PM) *bigdog:* [Timer] compile_c - 0.1 s
> (2:55:32 PM) *bigdog:* [Timer] =========================================
> (2:55:32 PM) *bigdog:* [Timer] Total: - 1955.8 s
>
>
>
> The translation spent inordinate amount of time in:
> (2:37:52 PM) *bigdog:* 
> ..+*%+................................*%%..................................... 
>
> (2:37:52 PM) *bigdog:* [backendopt:malloc] removed 3922 simple mallocs 
> in total
> (2:37:52 PM) *bigdog:* [backendopt:mergeifblocks] starting to merge if 
> blocks
> (2:37:52 PM) *bigdog:* 
> ..+*%+................................*%%..............
> (2:37:52 PM) *bigdog:* [translation:info] inserting stack checks...
> (2:37:52 PM) *bigdog:* ..
> (2:37:52 PM) *bigdog:* [rtyper] -=- specialized 6 more blocks -=-
>
>
>
> Translation memory use went from ~350 meg to ~1.3 gb  (memory use are 
> not exact, I just looked at the perfomrnace monitor when runnning)
>
> Hope this is helpful,
> Mike
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>   



From michaelschneider at current.net  Mon Jan 14 05:34:20 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sun, 13 Jan 2008 23:34:20 -0500
Subject: [pypy-dev] windows C translation progress - complete with patches
 for 2/3 of the problems
In-Reply-To: <47891F21.7080100@current.net>
References: <47891F21.7080100@current.net>
Message-ID: <478AE64C.1080706@current.net>

Hello All,

Hope you are having fun at the sprint.

I have been working on getting the windows translation going again.

There are 3 issues, and I have fixes for 2 of them

1) translator/c/src/stack.h  includes a unix only include file without 
guards,  this is an easy fix.  This include is not used, simply delete 
this #include

#include <unistd.h>

2) translator/c/src/ll_strtod.h   apply this patch to add 
#include<winsock2.h> during a windows compile.  The patch at then end of 
this email will fix
the winsoc2 related compile errors


3) The generated code module_0.c  #includes  <sys/wait.h>   rumor has a 
fix is close for this, so I did not work this.

                                module_0.c
                                module_0.c(859) : fatal error C1083: 
Cannot open include file: 'sys/wait.h': No
                                such file or directory



Can some kind soul please review and commit these changes?


Thanks, and happy skiing/snowboarding/lounging in a nice place,
Mike

----------------------------------------------------------------------------------------------------------

m
#------------------------------------------ start 
patch-------------------------------------------
Index: ll_strtod.h

===================================================================

--- ll_strtod.h    (revision 50566)

+++ ll_strtod.h    (working copy)

@@ -1,6 +1,10 @@

 #ifndef LL_STRTOD_H
 #define LL_STRTOD_H
 
+#ifdef MS_WINDOWS
+#include <winsock2.h>
+#endif
+
 #include <locale.h>
 #include <ctype.h>
 #include <string.h>











From alexandre.fayolle at logilab.fr  Mon Jan 14 09:52:37 2008
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Mon, 14 Jan 2008 09:52:37 +0100
Subject: [pypy-dev] pylib patch to enable mips and mipsel greenlets
Message-ID: <20080114085237.GD11580@logilab.fr>

Hi,

I've received a patch by Thiemo Seufer <ths at networkno.de> to port pylib
greenlets to mips and mipsel linux. 

It is available for examination at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=459520

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20080114/deeba165/attachment.pgp>

From cfbolz at gmx.de  Mon Jan 14 13:33:11 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 14 Jan 2008 13:33:11 +0100
Subject: [pypy-dev] windows C translation progress - complete with
 patches for 2/3 of the problems
In-Reply-To: <478AE64C.1080706@current.net>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
Message-ID: <478B5687.9010705@gmx.de>

Hi Mike,

Michael Schneider wrote:
> Hope you are having fun at the sprint.

yes, apart from the usual head-wall interactions :-).

> I have been working on getting the windows translation going again.
> 
> There are 3 issues, and I have fixes for 2 of them
> 
> 1) translator/c/src/stack.h  includes a unix only include file without 
> guards,  this is an easy fix.  This include is not used, simply delete 
> this #include
> 
> #include <unistd.h>
> 
> 2) translator/c/src/ll_strtod.h   apply this patch to add 
> #include<winsock2.h> during a windows compile.  The patch at then end of 
> this email will fix
> the winsoc2 related compile errors

I committed both of these, thanks a lot!


> 3) The generated code module_0.c  #includes  <sys/wait.h>   rumor has a 
> fix is close for this, so I did not work this.
> 
>                                 module_0.c
>                                 module_0.c(859) : fatal error C1083: 
> Cannot open include file: 'sys/wait.h': No
>                                 such file or directory

According to Maciek this should be fixed as of revision 50588

Thanks again,

Carl Friedrich


From michaelschneider at current.net  Tue Jan 15 05:29:53 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Mon, 14 Jan 2008 23:29:53 -0500
Subject: [pypy-dev] windows pypy C translation status - 3 compile classes
 fixe - simple link issue remaining
In-Reply-To: <478B5687.9010705@gmx.de>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de>
Message-ID: <478C36C1.5040400@current.net>

Good News, things are progressing nicely.

1) the 3 classes are compile errors have been address, and the source 
files now compile (note: header cleanup may still occur)

2) The link is very close, the emmited link command includes a m.lib 
command which is an error on visual studio links..


I am sorry, I don't understand how the link line is emmited.  Can 
someone familiar with this please remove the m.lib from the generated 
link line for visual studio 2003?

Thanks
Mike


PS.  I will be happy to test the fix



---------------------------------------- compile and link line from 
simple Visual Studio Project 2003, that includes a sin(90.0) ------------

 
Command Lines
 
 
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000005.rsp" with contents
[
/Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /D "_MBCS" /Gm /EHsc /RTC1 /MLd /Yu"stdafx.h" /Fp"Debug/simplemath.pch" /Fo"Debug/" /Fd"Debug/vc70.pdb" /W3 /c /Wp64 /ZI /TP
".\simplemath.cpp"
]
Creating command line "cl.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000005.rsp" /nologo"
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000006.rsp" with contents
[
/Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /D "_MBCS" /Gm /EHsc /RTC1 /MLd /Yc"stdafx.h" /Fp"Debug/simplemath.pch" /Fo"Debug/" /Fd"Debug/vc70.pdb" /W3 /c /Wp64 /ZI /TP
".\stdafx.cpp"
]
Creating command line "cl.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000006.rsp" /nologo"
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000007.rsp" with contents
[
/OUT:"Debug/simplemath.exe" /INCREMENTAL /NOLOGO /DEBUG /PDB:"Debug/simplemath.pdb" /SUBSYSTEM:CONSOLE /MACHINE:X86  kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib
".\debug\stdafx.obj"
".\debug\simplemath.obj"
]
Creating command line "link.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000007.rsp""
 
Output Window
 
 
Compiling...
stdafx.cpp
Compiling...
simplemath.cpp
Linking...
 
Results
 
 
Build log was saved at "file://c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\BuildLog.htm"
simplemath - 0 error(s), 0 warning(s)



-------------------------------------end simple visual studio project 
-----------------------------------------------------------------------------------------------



------------------------------------- current pypy compile and link line 
---------------------------------------------------------------------

.... compile other .obj's cut   \pypy-dist\pypy\translator\c 
/Tcmodule_2.c /Fomodule_2.obj [cbuild:execute] C:\Program 
Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox 
/MD /W3 /GX /DNDEBUG -Ic:\python25\include 
-IH:\pypy\pypy-dist\pypy\translator\c /Tcmodule_3.c /Fomodule_3.obj 
[cbuild:execute] C:\Program Files\Microsoft Visual Studio .NET 
2003\Vc7\bin\link.exe /DEBUG /nologo /INCREMENTAL:NO gc_pypy.lib m.lib 
h:\temp\usession-27\testing_1\testing_1.obj 
h:\temp\usession-27\testing_1\structimpl.obj 
h:\temp\usession-27\testing_1\nonfuncnodes.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_1.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_2.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_3.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_4.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_5.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_6.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_7.obj 
h:\temp\usession-27\testing_1\implement.obj 
h:\temp\usession-27\testing_1\implement_1.obj 
h:\temp\usession-27\testing_1\implement_2.obj 
h:\temp\usession-27\testing_1\implement_3.obj 
h:\temp\usession-27\testing_1\implement_4.obj 
h:\temp\usession-27\testing_1\implement_5.obj 
h:\temp\usession-27\testing_1\implement_6.obj 
h:\temp\usession-27\testing_1\implement_7.obj 
h:\temp\usession-27\testing_1\implement_8.obj 
h:\temp\usession-27\testing_1\implement_9.obj 
h:\temp\usession-27\testing_1\implement_10.obj 
h:\temp\usession-27\testing_1\implement_11.obj 
h:\temp\usession-27\testing_1\implement_12.obj 
h:\temp\usession-27\testing_1\implement_13.obj 
h:\temp\usession-27\testing_1\implement_14.obj 
h:\temp\usession-27\module_cache\module_0.obj 
h:\temp\usession-27\module_cache\module_1.obj 
h:\temp\usession-27\module_cache\module_2.obj 
h:\temp\usession-27\module_cache\module_3.obj 
/OUT:h:\temp\usession-27\testing_1\testing_1.exe -=- specialized 0 more 
blocks -=- [backendopt:inlining] phase with threshold factor: 32.

--------------------------------- end pypy link line 
-----------------------------------------------------------------


From michaelschneider at current.net  Tue Jan 15 21:29:00 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Tue, 15 Jan 2008 15:29:00 -0500
Subject: [pypy-dev] windows C translation update
In-Reply-To: <478B5687.9010705@gmx.de>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de>
Message-ID: <478D178C.9020202@current.net>

Wow,

Thanks to everyone for their efforts.   The compile errors have been 
addressed, and the first couple of layers of the link-issue onion
have been peeled.

We are down to this link issue on windows

Mike

Note: this translation does not include the allmodules flag,  we did the 
first trial balloon of that today, and there are some issues in the mmap 
area. 
Those issues can until we finish this first level of success.
----------------------------------------------------------------------------------------------

module_0.c
module_0.c(885) : warning C4013: 'WCOREDUMP' undefined; assuming extern 
returnin
g int
module_0.c(890) : warning C4013: 'WIFCONTINUED' undefined; assuming 
extern retur
ning int
module_0.c(895) : warning C4013: 'WIFSTOPPED' undefined; assuming extern 
returni
ng int
module_0.c(900) : warning C4013: 'WIFSIGNALED' undefined; assuming 
extern return
ing int
module_0.c(905) : warning C4013: 'WIFEXITED' undefined; assuming extern 
returnin
g int
module_0.c(910) : warning C4013: 'WEXITSTATUS' undefined; assuming 
extern return
ing int
module_0.c(915) : warning C4013: 'WSTOPSIG' undefined; assuming extern 
returning
 int
module_0.c(920) : warning C4013: 'WTERMSIG' undefined; assuming extern 
returning
 int
module_1.c
module_1.c(860) : warning C4273: '_errno' : inconsistent dll linkage
module_2.c
module_3.c
module_3.c(859) : warning C4273: '__p__environ' : inconsistent dll linkage
   Creating library h:\temp\usession-36\testing_1\testing_1.lib and 
object h:\te
mp\usession-36\testing_1\testing_1.exp
module_0.obj : error LNK2019: unresolved external symbol _WCOREDUMP 
referenced i
n function _pypy_macro_wrapper_WCOREDUMP
module_0.obj : error LNK2019: unresolved external symbol _WIFCONTINUED 
reference
d in function _pypy_macro_wrapper_WIFCONTINUED
module_0.obj : error LNK2019: unresolved external symbol _WIFSTOPPED 
referenced
in function _pypy_macro_wrapper_WIFSTOPPED
module_0.obj : error LNK2019: unresolved external symbol _WIFSIGNALED 
referenced
 in function _pypy_macro_wrapper_WIFSIGNALED
module_0.obj : error LNK2019: unresolved external symbol _WIFEXITED 
referenced i
n function _pypy_macro_wrapper_WIFEXITED
module_0.obj : error LNK2019: unresolved external symbol _WEXITSTATUS 
referenced
 in function _pypy_macro_wrapper_WEXITSTATUS
module_0.obj : error LNK2019: unresolved external symbol _WSTOPSIG 
referenced in
 function _pypy_macro_wrapper_WSTOPSIG
module_0.obj : error LNK2019: unresolved external symbol _WTERMSIG 
referenced in
 function _pypy_macro_wrapper_WTERMSIG
h:\temp\usession-36\testing_1\testing_1.exe : fatal error LNK1120: 8 
unresolved
externals



From fijall at gmail.com  Tue Jan 15 22:35:36 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 15 Jan 2008 22:35:36 +0100
Subject: [pypy-dev] windows C translation update
In-Reply-To: <478D178C.9020202@current.net>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de> <478D178C.9020202@current.net>
Message-ID: <693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>

On Jan 15, 2008 9:29 PM, Michael Schneider <michaelschneider at current.net>
wrote:

> Wow,
>
> Thanks to everyone for their efforts.   The compile errors have been
> addressed, and the first couple of layers of the link-issue onion
> have been peeled.
>
> We are down to this link issue on windows
>
> This one should be fixed by 50653.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20080115/627ac684/attachment.html>

From michaelschneider at current.net  Wed Jan 16 04:56:01 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Tue, 15 Jan 2008 22:56:01 -0500
Subject: [pypy-dev] windows C translation update
In-Reply-To: <693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>	
	<478B5687.9010705@gmx.de> <478D178C.9020202@current.net>
	<693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>
Message-ID: <478D8051.4020506@current.net>

Maciej ,

Good work,  I updated to trunk, and the translation completes 
successfully :-)

Thanks
Mike

Maciej Fijalkowski wrote:
>
>
> On Jan 15, 2008 9:29 PM, Michael Schneider 
> <michaelschneider at current.net <mailto:michaelschneider at current.net>> 
> wrote:
>
>     Wow,
>
>     Thanks to everyone for their efforts.   The compile errors have been
>     addressed, and the first couple of layers of the link-issue onion
>     have been peeled.
>
>     We are down to this link issue on windows
>
> This one should be fixed by 50653.



From Martin.vonLoewis at hpi.uni-potsdam.de  Tue Jan 22 13:26:32 2008
From: Martin.vonLoewis at hpi.uni-potsdam.de (=?iso-8859-1?Q?Martin_von_L=F6wis?=)
Date: Tue, 22 Jan 2008 13:26:32 +0100
Subject: [pypy-dev] Workshop on Self-sustaining Systems (S3)
Message-ID: <5AC0F51A926CD442BC08D55DA7DAB1D40868182F@3mxma1r.hpi.uni-potsdam.de>

*** Workshop on Self-sustaining Systems (S3) 2008 ***

May 15-16, 2008
Potsdam, Germany
http://www.swa.hpi.uni-potsdam.de/s3/

-- Call for papers:

The Workshop on Self-sustaining Systems (S3) is a forum for discussion
of topics relating to computer systems and languages that are able to
bootstrap, implement, modify, and maintain themselves. One property of
these systems is that their implementation is based on small but
powerful abstractions; examples include (amongst others)
Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and
Lisp. Such systems are the engines of their own replacement, giving
researchers and developers great power to experiment with, and explore
future directions from within, their own small language kernels.

S3 will be take place May 15-16, 2008 at the Hasso-Plattner-Institute in
Potsdam, Germany. It is an exciting opportunity for researchers and
practitioners interested in self-sustaining systems to meet and share
their knowledge, experience, and ideas for future research and development.

-- Invited talk:

Ian Piumarta: Late-bound Object Lambda Architectures (Viewpoints
Research Institute, USA)

-- Submissions and proceedings:

S3 invites submissions of high-quality papers reporting original
research, or describing innovative contributions to, or experience with,
self-sustaining systems, their implementation, and their application.
Papers that depart significantly from established ideas and practices
are particularly welcome.

Submissions must not have been published previously and must not be
under review for any another refereed event or publication. The program
committee will evaluate each contributed paper based on its relevance,
significance, clarity, and originality. Revised papers will be published
as post-proceedings in the Springer LNCS series.

Papers should be submitted electronically via EasyChair at
http://www.easychair.org/conferences/?conf=s3 in PDF format. Submissions
must be written in English (the official language of the workshop) and
must not exceed 20 pages. They should use the LNCS format, templates for
which are available at http://www.springer.de/comp/lncs/authors.html.

-- Venue:

Hasso-Plattner-Institut (Potsdam, Germany)

-- Important dates:

Submission of papers: February 15, 2008
Author notification: April 11, 2008
Revised papers due: April 25, 2008

S3 workshop: May 15-16, 2008

Final papers for LNCS post-proceedings due: June 6, 2008

-- Chairs:

* Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany)
* Kim Rose (Viewpoints Research Institute, USA)

-- Program committee:

* Johan Brichau, Universite Catholique de Louvain, Belgium
* Pascal Costanza, Vrije Universiteit Brussel, Belgium
* Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium
* Stephane Ducasse, INRIA Lille, France
* Michael Haupt, Hasso-Plattner-Institut, Germany
* Robert Hirschfeld, Hasso-Plattner-Institut, Germany
* Dan Ingalls, Sun Microsystems Laboratories, USA
* Martin von L?wis, Hasso-Plattner-Institut, Germany
* Hidehiko Masuhara, University of Tokyo, Japan
* Ian Piumarta, Viewpoints Research Institute, USA
* David Ungar, IBM, USA

-- Registration fees:

Early (until April 18, 2008)
* Regular participants: EUR 160
* Students: EUR 80

Late (after April 18, 2008)
* Regular participants: EUR 170
* Students: EUR 90 


From arigo at tunes.org  Wed Jan 23 18:54:10 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 23 Jan 2008 18:54:10 +0100
Subject: [pypy-dev] Loops for gcc
Message-ID: <20080123175409.GA29424@code0.codespeak.net>

Hi all,

Some results seem to show that GCC is more happy optimizing "for" and
"while" loops than the equivalent loops written with a bunch of goto's
like we produce in our C backend.  Interestingly, it seems that what
makes GCC unhappy is not the bunch of goto's, but only the absence of
the explicit C-level loop.  Example:

For this piece of C code as someone would write by hand:

        while (n > 0) {
            ...body...
        }

we generate the following instead, which defeats GCC's loop-specific
optimizations:

      block1:
        cond = n > 0;
        if (cond) goto block2;
        goto block3;

      block2:
        ...body...
        goto block1;

The smallest change I could find that make GCC produce good code again
is to modify block1 and block2 like this:

      block1:
        cond = n > 0;
        while (cond) {      /* trick */
            goto block2;
          block1_back:
            cond = n > 0;   /* the operations in block1 are repeated */
        }
        goto block3;

      block2:
        ...body...
        goto block1_back;   /* changed */

GCC doesn't seem to mind that the real body of the loop is not in the
loop at all but somewhere else; I guess it just follows the chain of
goto's and treat the result as equivalent to having the source code
directly within the loop's { }.

So I wrote a quick hack that detects (for now only the innermost) loops
in graphs.  More specifically it finds "loop head" blocks like block1,
and "loop-closing links" like the link from block2 to block1.  It then
produces code like above.  It's in the loops-for-gcc branch.  It's quite
obscure but seems to give a consistent 5% speed-up on pypy-c running
richards...


A bientot,

Armin.


From martin at martincmartin.com  Fri Jan 25 16:30:17 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:30:17 -0500
Subject: [pypy-dev] Two problems
Message-ID: <479A0089.3000400@martincmartin.com>

Hi,

I'm really impressed with the C code generated for some functions I'd 
like to use in my project.  You guys rock!

I've run into two problems this morning.  In pypy-1.0.0, this:

import re
import os, sys
from pypy.translator.interactive import Translation
import py

def mytest():
    x = re.search('bar', 'foobar')
    return x.start()

t = Translation(mytest)
t.annotate()

throws an exception:

   File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
567, in simple_call
     return bltn.analyser(bltn.s_self, *args)
   File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
19, in __call__
     return self.im_func(firstarg, *args, **kwds)
TypeError': method_get() takes exactly 3 arguments (2 given)
         .. v1 = simple_call((builtin_function_or_method get), v0)
         .. '(sre:213)_compile_star2'


This is an InstanceMethod where im_self is a SomeDict and args is a 
tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
It seems to be calling a method_get() with two args, SomeDict and 
SomeTuple, whereas it wants 3.  Any idea how to fix this?

Thinking it might be fixed in svn, I got the latest code using:

svn co http://codespeak.net/svn/pypy/dist pypy-dist

Then tried to start pypy using:

python pypy-dist/pypy/bin/py.py

But got:

   File "/ita/downloaded/pypy-dist/pypy/module/posix/__init__.py", line 
3, in ?
     from pypy.rpython.module.ll_os import RegisterOs
   File "/ita/downloaded/pypy-dist/pypy/rpython/module/ll_os.py", line 127
     @registering_if(os, 'execv')
     ^
SyntaxError: invalid syntax

This is all on 64 bit Linux, python 2.4.1.

Any idea how I can get my simple function to compile?

Best,
Martin



From fijal at genesilico.pl  Fri Jan 25 16:40:34 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 16:40:34 +0100
Subject: [pypy-dev] Two problems
In-Reply-To: <479A0089.3000400@martincmartin.com>
References: <479A0089.3000400@martincmartin.com>
Message-ID: <479A02F2.8030109@genesilico.pl>

Martin C. Martin wrote:
> Hi,
>
> I'm really impressed with the C code generated for some functions I'd 
> like to use in my project.  You guys rock!
>
> I've run into two problems this morning.  In pypy-1.0.0, this:
>
> import re
> import os, sys
> from pypy.translator.interactive import Translation
> import py
>
> def mytest():
>     x = re.search('bar', 'foobar')
>     return x.start()
>
> t = Translation(mytest)
> t.annotate()
>
> throws an exception:
>
>    File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
> 567, in simple_call
>      return bltn.analyser(bltn.s_self, *args)
>    File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
> 19, in __call__
>      return self.im_func(firstarg, *args, **kwds)
> TypeError': method_get() takes exactly 3 arguments (2 given)
>          .. v1 = simple_call((builtin_function_or_method get), v0)
>          .. '(sre:213)_compile_star2'
>
>   
re is not RPythonic, you cannot use it.
> This is an InstanceMethod where im_self is a SomeDict and args is a 
> tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
> 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
> It seems to be calling a method_get() with two args, SomeDict and 
> SomeTuple, whereas it wants 3.  Any idea how to fix this?
>
> Thinking it might be fixed in svn, I got the latest code using:
>
> svn co http://codespeak.net/svn/pypy/dist pypy-dist
>
> Then tried to start pypy using:
>
> python pypy-dist/pypy/bin/py.py
>
> But got:
>
>    File "/ita/downloaded/pypy-dist/pypy/module/posix/__init__.py", line 
> 3, in ?
>      from pypy.rpython.module.ll_os import RegisterOs
>    File "/ita/downloaded/pypy-dist/pypy/rpython/module/ll_os.py", line 127
>      @registering_if(os, 'execv')
>      ^
> SyntaxError: invalid syntax
>
> This is all on 64 bit Linux, python 2.4.1.
>
> Any idea how I can get my simple function to compile?
>
> Best,
> Martin
>   
this is valid 2.4 syntax (it does not work on 2.3 though), are you sure 
you're using 2.4?

Also, 64 bit might only work on svn (and only *might*)

Cheers,
fijal


:.



From martin at martincmartin.com  Fri Jan 25 16:40:44 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:40:44 -0500
Subject: [pypy-dev] Two problems
In-Reply-To: <20080125073621.d6b88167.sburton@ewtllc.com>
References: <479A0089.3000400@martincmartin.com>
	<20080125073621.d6b88167.sburton@ewtllc.com>
Message-ID: <479A02FC.7010403@martincmartin.com>

So it's not possible for RPython to call regular, non-RPython?  Is there 
some wrapper I could write to let RPython call regular python code?

Best,
Martin

Simon Burton wrote:
> On Fri, 25 Jan 2008 10:30:17 -0500
> "Martin C. Martin" <martin at martincmartin.com> wrote:
> 
>> Hi,
>>
>> I'm really impressed with the C code generated for some functions I'd 
>> like to use in my project.  You guys rock!
>>
>> I've run into two problems this morning.  In pypy-1.0.0, this:
>>
>> import re
>> import os, sys
>> from pypy.translator.interactive import Translation
>> import py
>>
>> def mytest():
>>     x = re.search('bar', 'foobar')
>>     return x.start()
>>
>> t = Translation(mytest)
>> t.annotate()
>>
>> throws an exception:
>>
>>    File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
>> 567, in simple_call
>>      return bltn.analyser(bltn.s_self, *args)
>>    File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
>> 19, in __call__
>>      return self.im_func(firstarg, *args, **kwds)
>> TypeError': method_get() takes exactly 3 arguments (2 given)
>>          .. v1 = simple_call((builtin_function_or_method get), v0)
>>          .. '(sre:213)_compile_star2'
>>
>>
>> This is an InstanceMethod where im_self is a SomeDict and args is a 
>> tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
>> 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
>> It seems to be calling a method_get() with two args, SomeDict and 
>> SomeTuple, whereas it wants 3.  Any idea how to fix this?
> 
> I'm not sure you can use module "re" in rpython code.
> 
> The error is because somewhere there is a .get(XX) method call on a dict.
> In rpython the default arg for dict.get must be specified.
> 
> Simon.
> 
> 


From fijal at genesilico.pl  Fri Jan 25 16:46:27 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 16:46:27 +0100
Subject: [pypy-dev] Two problems
In-Reply-To: <479A02FC.7010403@martincmartin.com>
References: <479A0089.3000400@martincmartin.com>	<20080125073621.d6b88167.sburton@ewtllc.com>
	<479A02FC.7010403@martincmartin.com>
Message-ID: <479A0453.2060400@genesilico.pl>

Martin C. Martin wrote:
> So it's not possible for RPython to call regular, non-RPython?  Is there 
> some wrapper I could write to let RPython call regular python code?
>   
Not easily. You can call normal python code, provided that you run pypy 
interpreter, not cpython. This means that stuff like re are working, but 
stuff like pygtk or any 3rd party modules doesn't.

Also calling is very different. You can take a look at module/_demo if 
you like, but there are obstacles, like you cannot compile only your 
module, you need each time to compile whole interpreter (which takes 
~30minutes on decent machine). It's very unlikely you want to use it 
that way :)


:.



From martin at martincmartin.com  Fri Jan 25 16:48:42 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:48:42 -0500
Subject: [pypy-dev] Two problems
In-Reply-To: <479A0453.2060400@genesilico.pl>
References: <479A0089.3000400@martincmartin.com>	<20080125073621.d6b88167.sburton@ewtllc.com>
	<479A02FC.7010403@martincmartin.com>
	<479A0453.2060400@genesilico.pl>
Message-ID: <479A04DA.8070401@martincmartin.com>



Maciek Fijalkowski wrote:
> Martin C. Martin wrote:
>> So it's not possible for RPython to call regular, non-RPython?  Is 
>> there some wrapper I could write to let RPython call regular python code?
>>   
> Not easily. You can call normal python code, provided that you run pypy 
> interpreter, not cpython. This means that stuff like re are working, but 
> stuff like pygtk or any 3rd party modules doesn't.
> 
> Also calling is very different. You can take a look at module/_demo if 
> you like, but there are obstacles, like you cannot compile only your 
> module, you need each time to compile whole interpreter (which takes 
> ~30minutes on decent machine). It's very unlikely you want to use it 
> that way :)

:)  Ok, thanks.

Best,
Martin


From martin at martincmartin.com  Fri Jan 25 21:32:38 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 15:32:38 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
Message-ID: <479A4766.1060006@martincmartin.com>

Hi,

There seems to be a lot of overhead when passing a large string (23 Meg) 
to C compiled RPython code.  For example, this code:

def small(text):
    return 3

t = Translation(small)
t.annotate()
t.rtype()
f3 = t.compile_c()

st = time.time()
z = f3(xml)
print time.time() - st

Whereas parsing the 16,000 XML elements using a regular expression only 
took 22 msec.  Even reading the text file inside the compiled RPython is 
faster than passing it.

Here's the code from rstr.py that (seems to be) doing the conversion. 
Any idea how I'd put timing code in there to see what's taking all the 
time?  Any idea how to speed it up?

class __extend__(pairtype(PyObjRepr, AbstractStringRepr)):
     def convert_from_to((r_from, r_to), v, llops):
         v_len = llops.gencapicall('PyString_Size', [v], resulttype=Signed)
         cstr = inputconst(Void, STR)
         v_result = llops.genop('malloc_varsize', [cstr, v_len],
                                resulttype=Ptr(STR))
         cchars = inputconst(Void, "chars")
         v_chars = llops.genop('getsubstruct', [v_result, cchars],
                               resulttype=Ptr(STR.chars))
         llops.gencapicall('PyString_ToLLCharArray', [v, v_chars])
         string_repr = llops.rtyper.type_system.rstr.string_repr
         v_result = llops.convertvar(v_result, string_repr, r_to)
         return v_result

Best,
Martin




From fijal at genesilico.pl  Fri Jan 25 22:04:38 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 22:04:38 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A4766.1060006@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>
Message-ID: <479A4EE6.3030609@genesilico.pl>

Martin C. Martin wrote:
> Hi,
>
> There seems to be a lot of overhead when passing a large string (23 Meg) 
> to C compiled RPython code.  For example, this code:
>
> def small(text):
>     return 3
>
> t = Translation(small)
> t.annotate()
> t.rtype()
> f3 = t.compile_c()
>
> st = time.time()
> z = f3(xml)
> print time.time() - st
>
>   
This is wrong. You should even get a warning, the proper command is 
t.annotate([str]).

Besides, this is not the official way of writing rpython standalone 
programs. The official way is to go to pypy/translator/goal and for 
example modify targetnopstandalone for a standalone target (the entry 
function is entry_point).

You should translate this by ./translate.py targetnopstandalone.py (or 
whatever target you choose). You can even use some fancy options (like 
different gcs). In your example the xml was converted to python object, 
which will never happen the official way.

Cheers,
fijal


:.



From martin at martincmartin.com  Fri Jan 25 22:40:49 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 16:40:49 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A4EE6.3030609@genesilico.pl>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
Message-ID: <479A5761.2030903@martincmartin.com>



Maciek Fijalkowski wrote:
> Martin C. Martin wrote:
>> Hi,
>>
>> There seems to be a lot of overhead when passing a large string (23 
>> Meg) to C compiled RPython code.  For example, this code:
>>
>> def small(text):
>>     return 3
>>
>> t = Translation(small)
>> t.annotate()
>> t.rtype()
>> f3 = t.compile_c()
>>
>> st = time.time()
>> z = f3(xml)
>> print time.time() - st
>>
>>   
> This is wrong. You should even get a warning, the proper command is 
> t.annotate([str]).

Oops, yes, I've been working with variations of this all day, and I 
hadn't actually compiled & run the example in the email, although I'd 
done something equivalent.

> Besides, this is not the official way of writing rpython standalone 
> programs.

Thanks, but I'm not trying to write a standalone program, I need to call 
some 3rd party libraries.  For example, the string comes from one of a 
couple dozen of socket connections, managed by Twisted.  So I just want 
my inner loop in RPython.  The inner loop turns XML into a MySQL 
statement, which the main python program can then send to a database.

So I need to get a big string into RPython, and a smaller (but still 
pretty big) string out of it.

I see some other targets in there for shared libraries.  The docs 
mention that translate.py takes a --gc=generation argument, but when I 
try that I get:

$ python ./translator/goal/translate.py --gc=generation fun3.py
Usage
=====
   translate [options] [target] [target-specific-options]

translate: error: invalid value generation for option gc

Am I specifying it wrong?

Thanks,
Martin

> The official way is to go to pypy/translator/goal and for 
> example modify targetnopstandalone for a standalone target (the entry 
> function is entry_point).
> 
> You should translate this by ./translate.py targetnopstandalone.py (or 
> whatever target you choose). You can even use some fancy options (like 
> different gcs). In your example the xml was converted to python object, 
> which will never happen the official way.
> 
> Cheers,
> fijal
> 
> 
> :.
> 


From fijal at genesilico.pl  Fri Jan 25 22:45:35 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 22:45:35 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A5761.2030903@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
Message-ID: <479A587F.7040807@genesilico.pl>


> Thanks, but I'm not trying to write a standalone program, I need to 
> call some 3rd party libraries.  For example, the string comes from one 
> of a couple dozen of socket connections, managed by Twisted.  So I 
> just want my inner loop in RPython.  The inner loop turns XML into a 
> MySQL statement, which the main python program can then send to a 
> database.
You can do it that way, true. But it's pretty unofficial and 
unsupported. Also you can only pass very simple types (ie list or dict 
will not work).
>
> Am I specifying it wrong?
>
You're using too old version (you should just check it out from svn)

:.



From martin at martincmartin.com  Fri Jan 25 22:53:49 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 16:53:49 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A587F.7040807@genesilico.pl>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
	<479A587F.7040807@genesilico.pl>
Message-ID: <479A5A6D.7030906@martincmartin.com>



Maciek Fijalkowski wrote:
> 
>> Thanks, but I'm not trying to write a standalone program, I need to 
>> call some 3rd party libraries.  For example, the string comes from one 
>> of a couple dozen of socket connections, managed by Twisted.  So I 
>> just want my inner loop in RPython.  The inner loop turns XML into a 
>> MySQL statement, which the main python program can then send to a 
>> database.
> You can do it that way, true. But it's pretty unofficial and 
> unsupported. Also you can only pass very simple types (ie list or dict 
> will not work).

Ah, thanks for letting me know.  And thanks for all your help today!

Best,
Martin



From jgilbert.python at gmail.com  Fri Jan 25 23:06:03 2008
From: jgilbert.python at gmail.com (Josh Gilbert)
Date: Fri, 25 Jan 2008 17:06:03 -0500
Subject: [pypy-dev] How to learn RPython as a general purpose language
Message-ID: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>

I tried downloading the archives of this list and grepping, but
couldn't find an answer to this question. If it's been discussed to
death and I missed it I apologize.

I realize that it is not recommended to use RPython as a general
purpose language, a front end to C if you will. My question is how one
might learn how to do so, regardless. I'm also interested in reasons
why it shouldn't be done (preferably ones that can be resolved).

## References
I've found a comment on LtU suggesting that it is extremely hard to
learn, yet a valuable skill if you can cut it:
http://lambda-the-ultimate.org/node/1260#comment-13972

I've also found a hello world example from 2006 (humorously labeled part 1):
http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-part-1.html


Josh Gilbert.


From martin at martincmartin.com  Fri Jan 25 23:19:15 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 17:19:15 -0500
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
Message-ID: <479A6063.6020001@martincmartin.com>

In RPython, you can't call into other libraries, be they C or Python. 
Also, calling RPython from Python is "pretty unofficial and 
unsupported."  Given that much of modern programming centers around 
connecting various libraries together, this makes RPython a poor choice 
for most practical work.

At the moment, the most straight forward way to learn RPython is to 
learn Python, then learn the restrictions of RPython, since RPython is a 
subset of Python.  The easiest way to do this is, basically, keep 
writing Python and seeing what the translator will and won't accept.

Please correct me if I'm wrong; I've only spent a few days looking at 
RPython.

Best,
Martin

Josh Gilbert wrote:
> I tried downloading the archives of this list and grepping, but
> couldn't find an answer to this question. If it's been discussed to
> death and I missed it I apologize.
> 
> I realize that it is not recommended to use RPython as a general
> purpose language, a front end to C if you will. My question is how one
> might learn how to do so, regardless. I'm also interested in reasons
> why it shouldn't be done (preferably ones that can be resolved).
> 
> ## References
> I've found a comment on LtU suggesting that it is extremely hard to
> learn, yet a valuable skill if you can cut it:
> http://lambda-the-ultimate.org/node/1260#comment-13972
> 
> I've also found a hello world example from 2006 (humorously labeled part 1):
> http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-part-1.html
> 
> 
> Josh Gilbert.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From fijal at genesilico.pl  Fri Jan 25 23:29:15 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 23:29:15 +0100
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <479A6063.6020001@martincmartin.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
	<479A6063.6020001@martincmartin.com>
Message-ID: <479A62BB.3090108@genesilico.pl>

Martin C. Martin wrote:
> In RPython, you can't call into other libraries, be they C or Python. 
> Also, calling RPython from Python is "pretty unofficial and 
> unsupported."  Given that much of modern programming centers around 
> connecting various libraries together, this makes RPython a poor choice 
> for most practical work.
>   
You can call C libraries from rpython. There is even a doc here 
http://codespeak.net/pypy/dist/pypy/doc/rffi.html. Calling from Python 
to RPython is indeed pretty troublesome.
> At the moment, the most straight forward way to learn RPython is to 
> learn Python, then learn the restrictions of RPython, since RPython is a 
> subset of Python.  The easiest way to do this is, basically, keep 
> writing Python and seeing what the translator will and won't accept.
>
> Please correct me if I'm wrong; I've only spent a few days looking at 
> RPython.
>
> Best,
> Martin
>   
I think this is bad decision. Especially that translator might accept 
something and start behaving differently. Some docs are here 
http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#rpython, but 
it's indeed hard to deal with. RPython is similiar to C++ in a sense 
that one can write books about tricks. The main strangeness is that 
RPython is constructed out of live objects, which means that during 
initialization all python dynamism is allowed, but not after. This is 
sometimes hard to follow and also it's evolving a bit in time. I would 
suggest looking at various smaller language implementations (lang/ 
subdirectory in svn) and come to #pypy on freenode and ask. But it would 
be better if you have good reason why exactly you want to use it.

Cheers,
fijal

:.



From 2007b at usenet.alexanderweb.de  Fri Jan 25 23:52:05 2008
From: 2007b at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 25 Jan 2008 23:52:05 +0100
Subject: [pypy-dev] How to learn RPython as a general purpose language
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
	<479A6063.6020001@martincmartin.com>
Message-ID: <mp5q65-nb8.ln1@tuxpaddy.alexanderweb.homeip.net>

Martin C. Martin wrote:

> At the moment, the most straight forward way to learn RPython is to
> learn Python, then learn the restrictions of RPython, since RPython is a
> subset of Python.  The easiest way to do this is, basically, keep
> writing Python and seeing what the translator will and won't accept.
> 
> Please correct me if I'm wrong; I've only spent a few days looking at
> RPython.

No, that's pretty much correct. Another way to describe RPython is this
analogy:

Syntax and semantics of Python, speed of C, restrictions of Java and
compiler error messages as penetrable as MUMPS (*)

Kind regards,
Alexander


(*) http://en.wikipedia.org/wiki/MUMPS#Sample_Programs



From richardemslie at gmail.com  Sat Jan 26 00:10:32 2008
From: richardemslie at gmail.com (Richard Emslie)
Date: Fri, 25 Jan 2008 23:10:32 +0000
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
Message-ID: <479A6C68.70300@gmail.com>

Hi Josh,

Josh Gilbert wrote:
> 
> ## References
> I've found a comment on LtU suggesting that it is extremely hard to
> learn, yet a valuable skill if you can cut it:
> http://lambda-the-ultimate.org/node/1260#comment-13972
> 

I'm not sure I said it is 'extremely hard' or it is a 'valuable skill' - 
but i definitely said:

"""
Python is a subset of Python. The rules as to what the subset is vague 
indeed, but in short writing code without using dynamic language 
features post import time (ie the static-ness is based on code objects 
not python code) should be enough. These rules are fairly well 
documented and unlikely to change significantly in the future. The 
problem (IMHO) of promoting it is a language is that the tool chain is 
extremely non-user friendly and it is necessary to have a good 
understanding of how the flow graphs are created and how the several 
annotation passes work. All you will get is a stack trace in some 
obscure area of the annotation code for a compile error. :-) If one 
takes the time to learn how the above works then you have pretty useful 
language at your disposal.
"""

Well I wrote that many moons ago.  And is more or less still true - 
except I understood pypy's tool chain better than I do now.  After 
spending about 4-5 months rewriting a quite large application from 
python to rpython - I can say this about having a 'pretty useful 
language at your disposal':

(a) the translated rpython code was faster than python

(b) some parts were pretty much a copy and paste of the original python 
code and just a few tweaks to make it compile

(c) some parts were a complete rewrite

(d) the time taken for (b) and (c) was no less than it would of taken to 
write the code in say java or c++.

(e) the end result is still some nice python code that was still 
readable and easy to modify.

(f) a lot of niceties of using python go out the window such as no long 
waits for compiles, nice stack traces, run time introspection, no 
segmentation faults, etc etc

(a), (b) and (e) are advantages (c), (d) and (f) are disadvantages.  Of 
course if your application is interpretative and can make use of say the 
jit (and similar arguments for other features like the stackless 
transformation), then (d) would be a massive (colossal) effort that you 
get for free via pypy.  (f) does also have the advantage than rpython is 
python and during development one can run with some interpreter.

YMMV

Cheers - Richard





From cfbolz at gmx.de  Sat Jan 26 11:27:27 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sat, 26 Jan 2008 11:27:27 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A5761.2030903@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
Message-ID: <479B0B0F.608@gmx.de>

Martin C. Martin wrote:
> 
> Maciek Fijalkowski wrote:
>> Martin C. Martin wrote:
>>> Hi,
>>>
>>> There seems to be a lot of overhead when passing a large string (23 
>>> Meg) to C compiled RPython code.  For example, this code:
>>>
>>> def small(text):
>>>     return 3
>>>
>>> t = Translation(small)
>>> t.annotate()
>>> t.rtype()
>>> f3 = t.compile_c()
>>>
>>> st = time.time()
>>> z = f3(xml)
>>> print time.time() - st
>>>
>>>   
>> This is wrong. You should even get a warning, the proper command is 
>> t.annotate([str]).
> 
> Oops, yes, I've been working with variations of this all day, and I 
> hadn't actually compiled & run the example in the email, although I'd 
> done something equivalent.
> 
>> Besides, this is not the official way of writing rpython standalone 
>> programs.
> 
> Thanks, but I'm not trying to write a standalone program, I need to call 
> some 3rd party libraries.  For example, the string comes from one of a 
> couple dozen of socket connections, managed by Twisted.  So I just want 
> my inner loop in RPython.  The inner loop turns XML into a MySQL 
> statement, which the main python program can then send to a database.
> 
> So I need to get a big string into RPython, and a smaller (but still 
> pretty big) string out of it.

Couldn't you just use a subprocess, read the string from stdin and write 
the result to stdout? It's quite likely that this is not slower than the 
way strings are passed in and out now and has many advantages. You would 
need to use os.read and os.write, since sys.stdin/stdout is not 
supported in RPython, but apart from that it should work fine.

One of them is that if you use the Translation class, your RPython 
program will use reference counting, which is our slowest GC. If you use 
a subprocess you get the benefits of our much better generational GC.

Cheers,

Carl Friedrich



From martin at martincmartin.com  Sun Jan 27 09:51:46 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Sun, 27 Jan 2008 03:51:46 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479B0B0F.608@gmx.de>
References: <479A4766.1060006@martincmartin.com>	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com> <479B0B0F.608@gmx.de>
Message-ID: <479C4622.1090409@martincmartin.com>



Carl Friedrich Bolz wrote:
> Martin C. Martin wrote:
>>
>> Thanks, but I'm not trying to write a standalone program, I need to 
>> call some 3rd party libraries.  For example, the string comes from one 
>> of a couple dozen of socket connections, managed by Twisted.  So I 
>> just want my inner loop in RPython.  The inner loop turns XML into a 
>> MySQL statement, which the main python program can then send to a 
>> database.
>>
>> So I need to get a big string into RPython, and a smaller (but still 
>> pretty big) string out of it.
> 
> Couldn't you just use a subprocess, read the string from stdin and write 
> the result to stdout? It's quite likely that this is not slower than the 
> way strings are passed in and out now and has many advantages. You would 
> need to use os.read and os.write, since sys.stdin/stdout is not 
> supported in RPython, but apart from that it should work fine.
> 
> One of them is that if you use the Translation class, your RPython 
> program will use reference counting, which is our slowest GC. If you use 
> a subprocess you get the benefits of our much better generational GC.

What I'm really looking for is a way to write most of my applications in 
a dynamic language (because its more productive to write & maintain), 
then if and when performance is a problem, have some way to speed it up. 
  PyPy promises to do this even before performance is a problem, which 
will be great!

Until that comes, I was hoping for a language where I could give some 
hints to the compiler or runtime to speed it up.  Things like "although 
this binding could change each time through the loop, it doesn't 
actually change, so there's no need to do a hash lookup for every 
access."  Or "this variable is always an int."

The only language I know of that can do that is Lisp, which is a strong 
possibility.  But Lisp's syntax is more verbose and low level than 
modern dynamic languages, it doesn't have as many libraries, it doesn't 
have an IDE with auto completion, or a good source level debugger.  I 
had hoped Groovy would be like that, with its optional typing and Java 
inspired syntax and semantics, but sadly, the developers valued 
dynamism, however rarely used, over performance.

So the next best thing is to rewrite the performance critical parts in 
some other language.  I had hoped RPython would be that language for 
Python, but it turns out not to be.  I could rewrite in C++, but the 
semantics of C++ are very different than Python, so interfacing the two 
becomes verbose and awkward.  The ctypes module looks good for calling C 
libraries that weren't originally designed to work with Python.  But it 
doesn't have a good way (or any way?) to manipulate Python objects from 
C.  Even Java's JNI makes for a lot of boilerplate code to translate 
back and forth.

So it looks like my best bet may be Groovy, which interacts with Java 
seamlessly.  A year ago, when I last checked, the IDEs weren't up to the 
job, but may that's changed.

And once PyPy is done, that may be an even better solution.

Best,
Martin


From simon at arrowtheory.com  Tue Jan 29 18:06:22 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Tue, 29 Jan 2008 09:06:22 -0800
Subject: [pypy-dev] real-time gc ?
Message-ID: <20080129090622.351039ab.simon@arrowtheory.com>


I've taken some statistics on how the boehm gc performs with our current application,
it has terrible worst case performance, often stopping for hundreds of mS at a time.

I tried calling GC_enable_incremental() but that seems to cause the code to explode.

I don't have any idea how to obtain similar statistics for the refcounting gc, which
is also non-realtime. Perhaps it is in fact worse than boehm in this regard.

Over-all CPU usage is significantly less with boehm, but a lot can happen in 300mS of
a garbage collection.

I don't think it's feasable compiling the pypy interpreter with refcounting, it takes
a truly vast amount of time (some hours...)

It would be good using the pypy interpreter to script our rpython codebase, but this
gc issue seems to be a blocker.

Simon.


From atveit at gmail.com  Tue Jan 29 19:31:16 2008
From: atveit at gmail.com (Amund Tveit)
Date: Tue, 29 Jan 2008 19:31:16 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
Message-ID: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>

Hi, I wrote a blog entry yesterday about using rpython for the great
computer language shootout recursive program, and got a RunTime error
when I input on the commandline 11 (1 to 10 goes fine). Just did an
svn update of pypy to version 51115 and still get the error, here is
the program:

blog entry:
http://amundblog.blogspot.com/2008/01/rpython-gclb-benchmark-recursive.html

source file:
recursive.py
# The Computer Language Shootout
# http://shootout.alioth.debian.org/
# based on bearophile's psyco program
# slightly modified by Isaac Gouy

def Ack(x, y):
    if x == 0: return y+1
    if y == 0: return Ack(x-1, 1)
    return Ack(x-1, Ack(x, y-1))

def Fib(n):
    if n < 2: return 1
    return Fib(n-2) + Fib(n-1)

def FibFP(n):
    if n < 2.0: return 1.0
    return FibFP(n-2.0) + FibFP(n-1.0)

def Tak(x, y, z):
    if y < x: return Tak( Tak(x-1,y,z), Tak(y-1,z,x), Tak(z-1,x,y) )
    return z

def TakFP(x, y, z):
    if y < x: return TakFP( TakFP(x-1.0,y,z), TakFP(y-1.0,z,x),
TakFP(z-1.0,x,y) )
    return z

from sys import argv, setrecursionlimit
setrecursionlimit(20000)

def main(argv):
    n = int(argv[1]) - 1
    print "Ack(3,%d):" % (n+1), Ack(3, n+1)
    print "Fib(" + str(28.0+n) + "," + str(FibFP(28.0+n))
    print "Tak(%d,%d,%d): %d" % (3*n, 2*n, n, Tak(3*n, 2*n, n))
    print "Fib(3):", Fib(3)
    print "Tak(3.0,2.0,1.0):", TakFP(3.0, 2.0, 1.0)
    return 0

from pypy.translator.interactive import Translation
t = Translation(main, standalone=True, gc='ref')
t.source(backend='c')
path = t.compile()
print path

Best regards,
Amund


From amauryfa at gmail.com  Tue Jan 29 19:57:43 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 29 Jan 2008 19:57:43 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
Message-ID: <e27efe130801291057j7c831a8fwcd4d990203e74c35@mail.gmail.com>

Hello,

Amund Tveit wrote:
> Hi, I wrote a blog entry yesterday about using rpython for the great
> computer language shootout recursive program, and got a RunTime error
> when I input on the commandline 11 (1 to 10 goes fine). Just did an
> svn update of pypy to version 51115 and still get the error, here is
> the program:
>
> blog entry:
> http://amundblog.blogspot.com/2008/01/rpython-gclb-benchmark-recursive.html
>
> source file:
> recursive.py
> # The Computer Language Shootout
> # http://shootout.alioth.debian.org/
> # based on bearophile's psyco program
> # slightly modified by Isaac Gouy
>
> def Ack(x, y):
>     if x == 0: return y+1
>     if y == 0: return Ack(x-1, 1)
>     return Ack(x-1, Ack(x, y-1))
...
>
> from sys import argv, setrecursionlimit
> setrecursionlimit(20000)
>
> def main(argv):
>     n = int(argv[1]) - 1
>     print "Ack(3,%d):" % (n+1), Ack(3, n+1)
>     print "Fib(" + str(28.0+n) + "," + str(FibFP(28.0+n))
>     print "Tak(%d,%d,%d): %d" % (3*n, 2*n, n, Tak(3*n, 2*n, n))
>     print "Fib(3):", Fib(3)
>     print "Tak(3.0,2.0,1.0):", TakFP(3.0, 2.0, 1.0)
>     return 0
>
> from pypy.translator.interactive import Translation
> t = Translation(main, standalone=True, gc='ref')
> t.source(backend='c')
> path = t.compile()
> print path

This has nothing to do with pypy, and I get the same error with a
normal python interpreter: the Ackermann function is gigantically
recursive, and 20000 is probably not enough for Ack(3, 11).

-- 
Amaury Forgeot d'Arc


From cfbolz at gmx.de  Tue Jan 29 20:37:02 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 29 Jan 2008 20:37:02 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
Message-ID: <479F805E.20901@gmx.de>

Hi Amund,

Amund Tveit wrote:
> Hi, I wrote a blog entry yesterday about using rpython for the great
> computer language shootout recursive program, and got a RunTime error
> when I input on the commandline 11 (1 to 10 goes fine). Just did an
> svn update of pypy to version 51115 and still get the error, here is
> the program:

The runtime error is a stack overflow. RPython is checking for stack 
overflows when a recursive call is occuring (which is probably also the 
reason why the RPython version is slower). It is a bit conservative in 
doing that, which is why C still works for 11, but not the RPython version.

To simulate an infinite stack you could additionally pass in the option 
stackless=True and gc="generation" instead of ="ref" to the Translation 
class. This will make the program slightly slower, but should allow you 
very large arguments (argument sizes that probably make the normal C 
version segfault).

Cheers,

Carl Friedirch


From ae at amundtveit.info  Tue Jan 29 20:44:36 2008
From: ae at amundtveit.info (Amund Tveit)
Date: Tue, 29 Jan 2008 20:44:36 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <479F805E.20901@gmx.de>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
	<479F805E.20901@gmx.de>
Message-ID: <be52387e0801291144t1b11479ex8a0a63293a05b42b@mail.gmail.com>

thanks, will try that.

(used stackless python a while back for simulation -
http://amundtveit.info/publications/2003/zereal.pdf )

Amund


2008/1/29, Carl Friedrich Bolz <cfbolz at gmx.de>:
>
> Hi Amund,
>
> Amund Tveit wrote:
> > Hi, I wrote a blog entry yesterday about using rpython for the great
> > computer language shootout recursive program, and got a RunTime error
> > when I input on the commandline 11 (1 to 10 goes fine). Just did an
> > svn update of pypy to version 51115 and still get the error, here is
> > the program:
>
> The runtime error is a stack overflow. RPython is checking for stack
> overflows when a recursive call is occuring (which is probably also the
> reason why the RPython version is slower). It is a bit conservative in
> doing that, which is why C still works for 11, but not the RPython
> version.
>
> To simulate an infinite stack you could additionally pass in the option
> stackless=True and gc="generation" instead of ="ref" to the Translation
> class. This will make the program slightly slower, but should allow you
> very large arguments (argument sizes that probably make the normal C
> version segfault).
>
> Cheers,
>
> Carl Friedirch
>



-- 
Amund Tveit
http://amundtveit.info/ - +47 416 26 572
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20080129/7898a851/attachment.html>