From elmo.mantynen at iki.fi  Wed Sep  1 11:01:34 2010
From: elmo.mantynen at iki.fi (Elmo)
Date: Wed, 01 Sep 2010 12:01:34 +0300
Subject: [pypy-dev] gpgpu and pypy
In-Reply-To: <AANLkTik-fs3hf_Scn+4gyeoe+ZM6G93MEYfrYA8MoMqp@mail.gmail.com>
References: <AANLkTik9E=EKnqkWz_Re9hmG5hcUY2WTKFk=rH=kGJc4@mail.gmail.com>	<AANLkTin=hnD7TEhtk7=wRdJi-pDhxRc1DHSoZHn60xsp@mail.gmail.com>	<AANLkTi=hnpVxTPtRDjD-_wfC7PWhCDCVRX_ddpo+1xEU@mail.gmail.com>
	<AANLkTik-fs3hf_Scn+4gyeoe+ZM6G93MEYfrYA8MoMqp@mail.gmail.com>
Message-ID: <4C7E166E.9030008@iki.fi>

This seems similar to what MyHDL does. It's a python framework to be 
used as an HDL (hardware description language) for describing gate array 
configurations (outputs Verilog and VHDL for FPGAs or ASICs). It uses a 
similar approach to RPython as the compilation works on objects created 
by running the python code, with restrictions applying mostly to the 
code inside the generators that are used to encode the intended behavior 
(which is intrinsically parallel).

A bit on a different level, since you could use MyHDL to describe (and 
implement) a GPU, but I thought it would be interesting :)

Elmo

On 08/21/2010 10:06 AM, Hakan Ardo wrote:
> Hi,
> here is a another effort allowing you to write GPU kernels using
> python, targeted at gpgpu. The programmer has to explicitly state the
> parallelism and there are restrictions on what kind of constructs are
> allowed in the kernels, but it's pretty cool:
>
>    http://www.cs.lth.se/home/Calle_Lejdfors/pygpu/
>
> On Sat, Aug 21, 2010 at 12:46 AM, Nick Bray<ncbray at gmail.com>  wrote:
>> I can't speak for GPGPU, but I have compiled a subset of Python onto
>> the GPU for real-time rendering.  The subset is a little broader than
>> RPython in some ways (for example, attributes are semantically
>> identical to Python) and a little narrower in some ways (many forms of
>> recursion are disallowed.)  This big idea is that it allows you to
>> create a real-time rendering system with a single code base, and
>> transparently share functions and data structures between the CPU and
>> GPU.
>>
>> http://www.ncbray.com/pystream.html
>> http://www.ncbray.com/ncbray-dissertation.pdf
>>
>> It's at least ~100,000x faster than interpreting Python on the CPU.
>> "At least" because the measurements neglect doing things on the CPU
>> like texture sampling.  This speedup is pretty obscene, but if you
>> break it down it isn't too unbelievable... 100x for interpreted ->
>> compiled, 10x for abstraction overhead of using floats instead of
>> doubles, 100x for using the GPU and using it for a task it was built
>> for.
>>
>> Parallelism issues are sidestepped by explicitly identifying the
>> parallel sections (one function processes every vertex, one function
>> processes every fragment), requiring the parallel sections have no
>> global side effects, and that certain I/O conventions are followed.
>> Sorry, no big answers here - it's essentially Pythonic stream
>> programming.
>>
>> The biggest issues with getting Python onto the GPU is memory.  I was
>> actually targeting GLSL, not CUDA (it can't access the full rendering
>> pipeline), so pointers were not available.  To work around this, the
>> code is optimized to an extreme degree to remove as many memory
>> operations as possible.  The remaining memory operations are emulated
>> by splitting the heap into regions, indirecting through arrays, and
>> copying constant data wherever possible.  From what I've seen this is
>> where PyPy would have the most trouble: its analysis algorithms are
>> good enough for inferring types and  allowing compilation /
>> translation... they aren't designed to enable aggressive optimization
>> of memory operations (there's not a huge reason to do this if you're
>> translating RPython into C... the C compiler will do it for you).  In
>> general, GPU programming doesn't work well with memory access (too
>> many functional units, too little bandwidth).  Most of the "C-like"
>> GPU languages are designed to they can easily boil down into code
>> operating out of registers.  Python, on the other hand, is addicted to
>> heap memory.  Even if you target CUDA, eliminating memory operations
>> will be a huge win.
>>
>> I'll freely admit there's some ugly things going on, such as the lack
>> of recursion, reliance on exhaustive inlining, requiring GPU code
>> follow a specific form, and not working well with container objects in
>> certain situations (it needs to bound the size of the heap).  In the
>> end, however, it's a talking dog... the grammar may not be perfect,
>> but the dog talks!  If anyone has questions, either private or on the
>> list, I'd be happy to answer them.  I have not done enough to
>> advertise my project, and this seems like a good place to start.
>>
>> - Nick Bray
>>
>> 2010/8/20 Paolo Giarrusso<p.giarrusso at gmail.com>:
>>> 2010/8/20 Jorge Tim?n<timon.elviejo at gmail.com>:
>>>> Hi, I'm just curious about the feasibility of running python code
in a gpu
>>>> by extending pypy.
>>> Disclaimer: I am not a PyPy developer, even if I've been following the
>>> project with interest. Nor am I an expert of GPU - I provide links to
>>> the literature I've read.
>>> Yet, I believe that such an attempt is unlikely to be interesting.
>>> Quoting Wikipedia's synthesis:
>>> "Unlike CPUs however, GPUs have a parallel throughput architecture
>>> that emphasizes executing many concurrent threads slowly, rather than
>>> executing a single thread very fast."
>>> And significant optimizations are needed anyway to get performance for
>>> GPU code (and if you don't need the last bit of performance, why
>>> bother with a GPU?), so I think that the need to use a C-like language
>>> is the smallest problem.
>>>
>>>> I don't have the time (and probably the knowledge neither) to
develop that
>>>> pypy extension, but I just want to know if it's possible.
>>>> I'm interested in languages like openCL and nvidia's CUDA because
I think
>>>> the future of supercomputing is going to be GPGPU.
>>>
>>> I would like to point out that while for some cases it might be right,
>>> the importance of GPGPU is probably often exaggerated:
>>>
>>>
http://portal.acm.org/citation.cfm?id=1816021&coll=GUIDE&dl=GUIDE&CFID=11111111&CFTOKEN=2222222&ret=1#
>>>
>>> Researchers in the field are mostly aware of the fact that GPGPU is
>>> the way to go only for a very restricted category of code. For that
>>> code, fine.
>>> Thus, instead of running Python code in a GPU, designing from scratch
>>> an easy way to program a GPU efficiently, for those task, is better,
>>> and projects for that already exist (i.e. what you cite).
>>>
>>> Additionally, it would take probably a different kind of JIT to
>>> exploit GPUs. No branch prediction, very small non-coherent caches, no
>>> efficient synchronization primitives, as I read from this paper... I'm
>>> no expert, but I guess you'd need to rearchitecture from scratch the
>>> needed optimizations.
>>> And it took 20-30 years to get from the first, slow Lisp (1958) to,
>>> say, Self (1991), a landmark in performant high-level languages,
>>> derived from SmallTalk. Most of that would have to be redone.
>>>
>>> So, I guess that the effort to compile Python code for a GPU is not
>>> worth it. There might be further reasons due to the kind of code a JIT
>>> generates, since a GPU has no branch predictor, no caches, and so on,
>>> but I'm no GPU expert and I would have to check again.
>>>
>>> Finally, for general purpose code, exploiting the big expected number
>>> of CPUs on our desktop systems is already a challenge.
>>>
>>>> There's people working in
>>>> bringing GPGPU to python:
>>>>
>>>> http://mathema.tician.de/software/pyopencl
>>>> http://mathema.tician.de/software/pycuda
>>>>
>>>> Would it be possible to run python code in parallel without the
need (for
>>>> the developer) of actively parallelizing the code?
>>>
>>> I would say that Python is not yet the language to use to write
>>> efficient parallel code, because of the Global Interpreter Lock
>>> (Google for "Python GIL"). The two implementations having no GIL are
>>> IronPython (as slow as CPython) and Jython (slower). PyPy has a GIL,
>>> and the current focus is not on removing it.
>>> Scientific computing uses external libraries (like NumPy) - for the
>>> supported algorithms, one could introduce parallelism at that level.
>>> If that's enough for your application, good.
>>> If you want to write a parallel algorithm in Python, we're not
there yet.
>>>
>>>> I'm not talking about code of hard concurrency, but of code with
intrinsic
>>>> parallelism (let's say matrix multiplication).
>>>
>>> Automatic parallelization is hard, see:
>>> http://en.wikipedia.org/wiki/Automatic_parallelization
>>>
>>> Lots of scientists have tried, lots of money has been invested, but
>>> it's still hard.
>>> The only practical approaches still require the programmer to
>>> introduce parallelism, but in ways much simpler than using
>>> multithreading directly. Google OpenMP and Cilk.
>>>
>>>> Would a JIT compilation be capable of detecting parallelism?
>>> Summing up what is above, probably not.
>>>
>>> Moreover, matrix multiplication may not be so easy as one might think.
>>> I do not know how to write it for a GPU, but in the end I reference
>>> some suggestions from that paper (where it is one of the benchmarks).
>>> But here, I explain why writing it for a CPU is complicated. You can
>>> multiply two matrixes with a triply nested for, but such an algorithm
>>> has poor performance for big matrixes because of bad cache locality.
>>> GPUs, according to the above mentioned paper, provide no caches and
>>> hides latency in other ways.
>>>
>>> See here for the two main alternative ideas which allow solving this
>>> problem of writing an efficient matrix multiplication algorithm:
>>> http://en.wikipedia.org/wiki/Cache_blocking
>>> http://en.wikipedia.org/wiki/Cache-oblivious_algorithm
>>>
>>> Then, you need to parallelize the resulting code yourself, which might
>>> or might not be easy (depending on the interactions between the
>>> parallel blocks that are found there).
>>> In that paper, where matrix multiplication is called as SGEMM (the
>>> BLAS routine implementing it), they suggest using a cache-blocked
>>> version of matrix multiplication for both CPUs and GPUs, and argue
>>> that parallelization is then easy.
>>>
>>> Cheers,
>>> --
>>> Paolo Giarrusso - Ph.D. Student
>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>
>
>


From cfbolz at gmx.de  Wed Sep  1 15:41:19 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 01 Sep 2010 15:41:19 +0200
Subject: [pypy-dev] Registration for S3 - A Workshop on Self-Sustaining
 Systems is now open.
Message-ID: <4C7E57FF.3000601@gmx.de>

Registration for S3 - A Workshop on Self-Sustaining Systems is now open.

     http://www.hpi.uni-potsdam.de/hirschfeld/s3/s3-10/

We hope you will join us on September 27-28 in Tokyo.

Invited speakers:
     ? Takashi Ikegami: Sustainable Autonomy and Designing Mind Time 
(The University of Tokyo)
     ? Yukihiro Matsumoto: From Lisp to Ruby to Rubinius (Rakuten 
Institute of Technology)
     ? Vishal Sikka: On Sustainable Business Solutions (SAP)

You can find the workshop at:
http://www.hpi.uni-potsdam.de/hirschfeld/s3/s3-10/program/index.html

Kim Rose, Hidehiko Masuhara, and Robert Hirschfeld


From hakan at debian.org  Thu Sep  2 07:32:54 2010
From: hakan at debian.org (Hakan Ardo)
Date: Thu, 2 Sep 2010 07:32:54 +0200
Subject: [pypy-dev] jit-bounds branch (was: Loop invaraints)
In-Reply-To: <AANLkTi=EOAmdiV0u5fdYFnL=xcBrLsCHtHAY=KqT1ees@mail.gmail.com>
References: <AANLkTi=kXbXMXJh63B07ZJFajA8uQGv2y6NjsTkbUO88@mail.gmail.com>
	<4C7A3737.50902@gmx.de>
	<AANLkTinwz=71akbSFGpDpOg5Vfq3SnE=dokfgs_i5f+4@mail.gmail.com>
	<4C7A4C99.2050803@gmx.de>
	<AANLkTiknjSwX7vY69hjcmZ1OLZq03QYAvxx41DQZA_+f@mail.gmail.com>
	<AANLkTi=EOAmdiV0u5fdYFnL=xcBrLsCHtHAY=KqT1ees@mail.gmail.com>
Message-ID: <AANLkTinj0tRvkwnt5Rkg+8Bc=MVRDU1AgZvtE8v4BdKf@mail.gmail.com>

Hi,
I've checked in a version of optimizeopt that is a package with
support for building a chain of optimizations and passing instructions
down this chain. Does this design make sens? If so I'll start moving
the different optimization to the different files. It will require
some refactoring, but not too much I hope...

On Tue, Aug 31, 2010 at 9:25 AM, Hakan Ardo <hakan at debian.org> wrote:
> Ok, so we split it up into a set of Optimization classes in separate
> files. Each containing a subset of the optimize_... methods. Then we
> have the propagate_forward method iterate over the instructions
> passing them to one Optimization after the other? That way we keep the
> single iteration over the instructions. Would it be preferable to
> separate them even more and have each Optimization contain it's own
> loop over the instructions?
>
> On Sun, Aug 29, 2010 at 10:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Sun, Aug 29, 2010 at 2:03 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>>> On 08/29/2010 01:49 PM, Hakan Ardo wrote:
>>>>> P.S.: A bit unrelated, but a comment on the jit-bounds branch: I think
>>>>> it would be good if the bounds-related optimizations could move out of
>>>>> optimizeopt.py to their own file, because otherwise optimizeopt.py is
>>>>> getting really unwieldy. Does that make sense?
>>>>
>>>> Well, class IntBound and the propagate_bounds_ methods could probably
>>>> be moved elsewhere, but a lot of the work is done in optimize_...
>>>> methods, which I'm not so sure it would make sens to split up.
>>>
>>> I guess then the things that can be sanely moved should move. The file
>>> is nearly 2000 lines, which is way too big. I guess also the heap
>>> optimizations could go to their own file.
>>>
>>> Carl Friedrich
>>
>> How about a couple of files (preferably small) each containing a
>> contained optimization if possible? (maybe a package?)
>>
>
>
>
> --
> H?kan Ard?
>



-- 
H?kan Ard?


From sarvi at yahoo.com  Thu Sep  2 07:54:02 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Wed, 1 Sep 2010 22:54:02 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
Message-ID: <525425.61205.qm@web53702.mail.re2.yahoo.com>


I understand from various threads here,  that RPython is not for general purpose 
use.
Why this lack of Focus on general use.

I am looking at this and I am thinking and comparing this to a corporation that 
is working on this awesome product. 

They are so focused on this awesome final product vision that they fail to 
realize the awesome potential of some if its intermediate side deliverables.

PyPy is definitely gaining momentum.  
But as a strategy to build that momentum, and gain new converts it should put 
some focus on some of its niche strengths.
Things other python implementions cannot do. 

One such niche is its RPython and RPython Compiler.  
No other python implementation can convert python programs to executables.
I am seeing growing interest in writing Rpython code for performance critical 
code and even potentially compiling it to binaries.

http://olliwang.com/2009/12/20/aes-implementation-in-rpython/ 
http://alexgaynor.net/2010/may/15/pypy-future-python/


Is it possible the PyPy team may be understating the significance of RPython?
Am I crazy to think this way? :-)

Sarvi 


      


From mcneil at hku.hk  Thu Sep  2 08:27:32 2010
From: mcneil at hku.hk (Douglas McNeil)
Date: Thu, 2 Sep 2010 14:27:32 +0800
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <525425.61205.qm@web53702.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
Message-ID: <AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>

> No other python implementation can convert python programs to executables.

There's shedskin, which is actually very good as these things go:

http://code.google.com/p/shedskin/

Like RPython, you have to write in a small subset of python which can
be a little frustrating once you've gotten used to pythonic freedom.
But I've found it very useful for some short numerical codes (putting
on my OEIS associate editor hat).  And Cython is pretty powerful these
days.

ObPyPy: the other day I had cause to run a very short, unoptimized,
mostly integer-arithmetic code.  With shedskin, it took between ~42s
(with ints) and ~1m43 (with longs), as compared with only ~3m30 or so
to run under pypy.  That's only a factor of two (if I'd needed longs).
 Both could be much improved, and a lower-level version in C would
beat them both, but I was very impressed by how little difference
there was.  Major props!

For numerics it'd be interesting to have a JIT option which didn't
care about compilation times, and instead of generating assembly
itself generated assembly-like C which was then delegated to an
external compiler.


Doug

-- 
Department of Earth Sciences
University of Hong Kong


From p.giarrusso at gmail.com  Thu Sep  2 09:02:49 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 2 Sep 2010 09:02:49 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
Message-ID: <AANLkTim-w8MmhyOtaarY2FwJBT7bwkJY7cRePoKbDTuU@mail.gmail.com>

On Thu, Sep 2, 2010 at 08:27, Douglas McNeil <mcneil at hku.hk> wrote:
>> No other python implementation can convert python programs to executables.
>
> There's shedskin, which is actually very good as these things go:
>
> http://code.google.com/p/shedskin/
>
> Like RPython, you have to write in a small subset of python which can
> be a little frustrating once you've gotten used to pythonic freedom.
> But I've found it very useful for some short numerical codes (putting
> on my OEIS associate editor hat). ?And Cython is pretty powerful these
> days.

> ObPyPy: the other day I had cause to run a very short, unoptimized,
> mostly integer-arithmetic code. ?With shedskin, it took between ~42s
> (with ints) and ~1m43 (with longs), as compared with only ~3m30 or so
> to run under pypy. ?That's only a factor of two (if I'd needed longs).
> ?Both could be much improved, and a lower-level version in C would
> beat them both, but I was very impressed by how little difference
> there was. ?Major props!

> For numerics it'd be interesting to have a JIT option which didn't
> care about compilation times, and instead of generating assembly
> itself generated assembly-like C which was then delegated to an
> external compiler.

A more interesting road (which is mentioned somewhere in the PyPy
blog) is to use LLVM in place of this "external JIT compiler", so that
you generate "assembly-like LLVM Intermediate Representation". A bit
like UnladenSwallow is doing, with the difference of having a saner
runtime model to start with (say, no reference counting). Once you
start with LLVM, you are free to choose which optimization passes to
run, from very little to -O3 to even more ones.
The other C compilers incur huge startup costs for no good, and don't
usually allow being used as a library, if just for engineering
problems. LLVM is so much cooler anyway, especially now that say
_everybody_ is switching to it.

About the compilation times tradeoff, you can look for "tiered
compilation", which is a general strategy for doing it automatically,
possibly allowing different tunings (say, like java -server, which is
tuned for performance rather than responsiveness). My authoritative
reference is Cliff Click's blog [1], but you probably want to stop
reading it after the introduction, as I did in this case.

[1] http://www.azulsystems.com/blog/cliff-click/2010-07-16-tiered-compilation
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From william.leslie.ttg at gmail.com  Thu Sep  2 09:09:03 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Thu, 2 Sep 2010 17:09:03 +1000
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <525425.61205.qm@web53702.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
Message-ID: <AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>

On 2 September 2010 15:54, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
>
> I understand from various threads here, ?that RPython is not for general purpose
> use.
> Why this lack of Focus on general use.

Because then we would have to support that general use.

Python benefits from being reasonably standardised, you can be sure
that most python you write will run on any implementation that
supports the version you are targetting. On the other hand, if you are
mangling cpython or pypy bytecode, you are asking for trouble. Rpython
is an example of such an implementation detail - we* might like to
change features of it here or there to better support some needed
pattern.

Introducing yet another incompatable and complicated language to the
python ecosystem is not a worthwhile goal in itself.

* Just my opinion. Others might feel like standardising some amount of
rpython is a worthwhile idea.

> They are so focused on this awesome final product vision that they fail to
> realize the awesome potential of some if its intermediate side deliverables.
>
> PyPy is definitely gaining momentum.
> But as a strategy to build that momentum, and gain new converts it should put
> some focus on some of its niche strengths.
> Things other python implementions cannot do.
>
> One such niche is its RPython and RPython Compiler.
> No other python implementation can convert python programs to executables.

I can't see why you would ever want to do this - if you use py2exe or
the like instead, you get a large standard library and a great
language to work in, neither of which you get if you use rpython.

> I am seeing growing interest in writing Rpython code for performance critical
> code and even potentially compiling it to binaries.

The intention is to get almost the same performance out of the JIT.
For those that actually care about the last few percent, it would be
nicer to provide hints to generate specialised code at module compile
time, that way you can still work at python level.

> Is it possible the PyPy team may be understating the significance of RPython?
> Am I crazy to think this way? :-)

Supporting better integration between app-level python and other
languages that interact with interpreter level would be nice. CLI
integration is good, and JVM integration is lagging just a little. But
once you can interact with that level, there are much saner languages
that you could use for your low-level code than rpython - languages
/designed/ to be general purpose languages.

At the moment, the lack of separate compilation is a real issue
standing in the way of using rpython as a general purpose language, or
even as an extension language. Having to re-translate *everything*
every time you want to install an extension module is not on. Even C
doesn't require that.

The other is that type inference is global and changes you make to one
function can have far-reaching consequences. The error messages when
you do screw up aren't very friendly either.

If you want a low-level general purpose language with type inference
and garbage collection that has implementations for every platform
pypy targets, there are already plenty of options.

-- 
William Leslie


From p.giarrusso at gmail.com  Thu Sep  2 09:56:24 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 2 Sep 2010 09:56:24 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
Message-ID: <AANLkTimUU013WRhXgEi=pgEEQRUBJBkaQj8kB9aHZOnR@mail.gmail.com>

Hi,
I was curious about the interplay between type inference and separate
compilation.

On Thu, Sep 2, 2010 at 09:09, William Leslie
<william.leslie.ttg at gmail.com> wrote:
> At the moment, the lack of separate compilation is a real issue
> standing in the way of using rpython as a general purpose language, or
> even as an extension language. Having to re-translate *everything*
> every time you want to install an extension module is not on. Even C
> doesn't require that.

> The other is that type inference is global and changes you make to one
> function can have far-reaching consequences.
Is it module-global or is it performed on the whole program?
I guess you'd need modular type inference before allowing separate
compilation, and of course lots of implementation work.
Functional languages allow separate compilation - is there any
RPython-specific problem for that? I've omitted my guesses here.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From sarvi at yahoo.com  Thu Sep  2 09:57:33 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 00:57:33 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
Message-ID: <69412.54494.qm@web53706.mail.re2.yahoo.com>

I afraid people are missing the point here.
For an average engineer its better to be an expert of 1 language than be an 
average at 4. 
Thats my take on things.

Take Merurial(an SCM) 95% python 5%C and gives GIT a run for its money
This could be 95%Python and 5%RPython.

>From what I can tell writing RPython is still simpler than writing C/C++. 
Garbage collection alone justifies its use in my opinion.
The option of the Interpreter during development is just huge amounts of icing 
on the cake.

>From my reading on PyPy, thats why yall chose to write the PyPy in RPython. 
Yall could have done in this C/C++ right?

So far as I can tell RPython is a strict subset of Python. I don't see why it 
shouldn't continue to be.
And even if yall needed to make a very small set of static extension to RPython, 
you wouldn't any worse that Cython and Shedskin.

I would still rather work with just one interpreter/compiler, say PyPy. Better 
than PyPy/CPython for interpreter and Cython/Shedkin for compiling, with 
interpreter support during development.

I am just seeing Cython/Shedskin as fragmentation of resources. 
A lot more could accomplished if these projects came together.

Sarvi


----- Original Message ----
From: William Leslie <william.leslie.ttg at gmail.com>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: pypy-dev at codespeak.net
Sent: Thu, September 2, 2010 12:09:03 AM
Subject: Re: [pypy-dev] Question on the future of RPython

On 2 September 2010 15:54, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
>
> I understand from various threads here,  that RPython is not for general 
>purpose
> use.
> Why this lack of Focus on general use.

Because then we would have to support that general use.

Python benefits from being reasonably standardised, you can be sure
that most python you write will run on any implementation that
supports the version you are targetting. On the other hand, if you are
mangling cpython or pypy bytecode, you are asking for trouble. Rpython
is an example of such an implementation detail - we* might like to
change features of it here or there to better support some needed
pattern.

Introducing yet another incompatable and complicated language to the
python ecosystem is not a worthwhile goal in itself.

* Just my opinion. Others might feel like standardising some amount of
rpython is a worthwhile idea.

> They are so focused on this awesome final product vision that they fail to
> realize the awesome potential of some if its intermediate side deliverables.
>
> PyPy is definitely gaining momentum.
> But as a strategy to build that momentum, and gain new converts it should put
> some focus on some of its niche strengths.
> Things other python implementions cannot do.
>
> One such niche is its RPython and RPython Compiler.
> No other python implementation can convert python programs to executables.

I can't see why you would ever want to do this - if you use py2exe or
the like instead, you get a large standard library and a great
language to work in, neither of which you get if you use rpython.

> I am seeing growing interest in writing Rpython code for performance critical
> code and even potentially compiling it to binaries.

The intention is to get almost the same performance out of the JIT.
For those that actually care about the last few percent, it would be
nicer to provide hints to generate specialised code at module compile
time, that way you can still work at python level.

> Is it possible the PyPy team may be understating the significance of RPython?
> Am I crazy to think this way? :-)

Supporting better integration between app-level python and other
languages that interact with interpreter level would be nice. CLI
integration is good, and JVM integration is lagging just a little. But
once you can interact with that level, there are much saner languages
that you could use for your low-level code than rpython - languages
/designed/ to be general purpose languages.

At the moment, the lack of separate compilation is a real issue
standing in the way of using rpython as a general purpose language, or
even as an extension language. Having to re-translate *everything*
every time you want to install an extension module is not on. Even C
doesn't require that.

The other is that type inference is global and changes you make to one
function can have far-reaching consequences. The error messages when
you do screw up aren't very friendly either.

If you want a low-level general purpose language with type inference
and garbage collection that has implementations for every platform
pypy targets, there are already plenty of options.

-- 
William Leslie



      


From sarvi at yahoo.com  Thu Sep  2 10:10:06 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 01:10:06 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
Message-ID: <569927.57152.qm@web53706.mail.re2.yahoo.com>

If PyPy is using RPython for its compiler implementation, it should and will be 
optimized eventually for its compiler/JIT to to be fast.
Which just tells me that the performance gap between Shedskin and PyPy will be 
narrowed/beat pretty soon.

I would still rather work with just one interpreter/compiler, say PyPy. 
Better than using PyPy/CPython for an interpreter and Cython/Shedkin for 
compiling, without interpreter support during development.

I am just seeing Cython/Shedskin as fragmentation of resources. 
A lot more could accomplished if these projects came together with PyPy.


If you ask this question on the Shedskin/Cython alias as to why they shouldn't 
pool resources into making the PyPy RPython compiler into a First class 
citizen/goal of PyPy.
They will immediately tell you its not a goal PyPy.

Why not officially make it so. 
Formalize RPython and its compiler. 

Obviate the need for Cython/Shedskin and get them on board.

Like the example I quoated. Mercurial is 95% python 5% C for peformance. It 
should be 95% python and 5% RPython.

We have Pickle and cPickle for performance. The Pickle could have simply been 
rewritten in RPython and probably compiled and we don't need  different versons 
:-))

Sarvi


----- Original Message ----
From: Douglas McNeil <mcneil at hku.hk>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: "pypy-dev at codespeak.net" <pypy-dev at codespeak.net>
Sent: Wed, September 1, 2010 11:27:32 PM
Subject: Re: [pypy-dev] Question on the future of RPython

> No other python implementation can convert python programs to executables.

There's shedskin, which is actually very good as these things go:

http://code.google.com/p/shedskin/

Like RPython, you have to write in a small subset of python which can
be a little frustrating once you've gotten used to pythonic freedom.
But I've found it very useful for some short numerical codes (putting
on my OEIS associate editor hat).  And Cython is pretty powerful these
days.

ObPyPy: the other day I had cause to run a very short, unoptimized,
mostly integer-arithmetic code.  With shedskin, it took between ~42s
(with ints) and ~1m43 (with longs), as compared with only ~3m30 or so
to run under pypy.  That's only a factor of two (if I'd needed longs).
Both could be much improved, and a lower-level version in C would
beat them both, but I was very impressed by how little difference
there was.  Major props!

For numerics it'd be interesting to have a JIT option which didn't
care about compilation times, and instead of generating assembly
itself generated assembly-like C which was then delegated to an
external compiler.


Doug

-- 
Department of Earth Sciences
University of Hong Kong



      


From stefan_ml at behnel.de  Thu Sep  2 10:11:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Sep 2010 10:11:10 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <69412.54494.qm@web53706.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
	<69412.54494.qm@web53706.mail.re2.yahoo.com>
Message-ID: <i5nm6u$slk$1@dough.gmane.org>

Saravanan Shanmugham, 02.09.2010 09:57:
> I afraid people are missing the point here.
> For an average engineer its better to be an expert of 1 language than be an
> average at 4.

Well, it's certainly better to be an almost-expert in two, than a 
no-left-no-right expert in only one.


> I am just seeing Cython/Shedskin as fragmentation of resources.

You might want to closer look at the projects and their goals before 
judging that way.

Stefan



From sarvi at yahoo.com  Thu Sep  2 10:18:43 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 01:18:43 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <i5nm6u$slk$1@dough.gmane.org>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
	<69412.54494.qm@web53706.mail.re2.yahoo.com>
	<i5nm6u$slk$1@dough.gmane.org>
Message-ID: <694746.77392.qm@web53706.mail.re2.yahoo.com>

I have researched these projects quite extensively.
Quite similar beasts as far as I can tell.

Cython/Pyrex used to write python extensions. They use statically typed variants 
of Python which gets compiled into C which can then be compiled.

Shedskin is slightly more general purpose Restricted Python to C++ compiler.

PyPy as I understand can convert RPython into C code

Am I missing something here?

Sarvi




----- Original Message ----
From: Stefan Behnel <stefan_ml at behnel.de>
To: pypy-dev at codespeak.net
Sent: Thu, September 2, 2010 1:11:10 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Saravanan Shanmugham, 02.09.2010 09:57:
> I afraid people are missing the point here.
> For an average engineer its better to be an expert of 1 language than be an
> average at 4.

Well, it's certainly better to be an almost-expert in two, than a 
no-left-no-right expert in only one.


> I am just seeing Cython/Shedskin as fragmentation of resources.

You might want to closer look at the projects and their goals before 
judging that way.

Stefan

_______________________________________________
pypy-dev at codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev



      


From p.giarrusso at gmail.com  Thu Sep  2 10:22:24 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 2 Sep 2010 10:22:24 +0200
Subject: [pypy-dev] jit-bounds branch (was: Loop invaraints)
In-Reply-To: <AANLkTi=EOAmdiV0u5fdYFnL=xcBrLsCHtHAY=KqT1ees@mail.gmail.com>
References: <AANLkTi=kXbXMXJh63B07ZJFajA8uQGv2y6NjsTkbUO88@mail.gmail.com>
	<4C7A3737.50902@gmx.de>
	<AANLkTinwz=71akbSFGpDpOg5Vfq3SnE=dokfgs_i5f+4@mail.gmail.com>
	<4C7A4C99.2050803@gmx.de>
	<AANLkTiknjSwX7vY69hjcmZ1OLZq03QYAvxx41DQZA_+f@mail.gmail.com>
	<AANLkTi=EOAmdiV0u5fdYFnL=xcBrLsCHtHAY=KqT1ees@mail.gmail.com>
Message-ID: <AANLkTinnKQWZkzVQaGK+vqJxGOPupTRWVhte=9sh3fdA@mail.gmail.com>

On Tue, Aug 31, 2010 at 09:25, Hakan Ardo <hakan at debian.org> wrote:
> Ok, so we split it up into a set of Optimization classes in separate
> files. Each containing a subset of the optimize_... methods. Then we
> have the propagate_forward method iterate over the instructions
> passing them to one Optimization after the other? That way we keep the
> single iteration over the instructions. Would it be preferable to
> separate them even more and have each Optimization contain it's own
> loop over the instructions?

But won't this affect performance? Which is very important in a JIT compiler.
When compiling traces bigger than a cacheline, it might even affect
locality, i.e. be an important performance problem.
Unless your RPython compiler can join the loops. If they are just
loops, it could. If they are tree visits, it likely can't; it's done
by the Haskell compiler (google Haskell, stream fusion, shortcut
deforestation, I guess), but the techniques are unlikely to generalize
to languages with side effects; it's also done/doable in some
Domain-Specific Languages for tree visitors.

> On Sun, Aug 29, 2010 at 10:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Sun, Aug 29, 2010 at 2:03 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>>> On 08/29/2010 01:49 PM, Hakan Ardo wrote:
>>>>> P.S.: A bit unrelated, but a comment on the jit-bounds branch: I think
>>>>> it would be good if the bounds-related optimizations could move out of
>>>>> optimizeopt.py to their own file, because otherwise optimizeopt.py is
>>>>> getting really unwieldy. Does that make sense?
>>>>
>>>> Well, class IntBound and the propagate_bounds_ methods could probably
>>>> be moved elsewhere, but a lot of the work is done in optimize_...
>>>> methods, which I'm not so sure it would make sens to split up.
>>>
>>> I guess then the things that can be sanely moved should move. The file
>>> is nearly 2000 lines, which is way too big. I guess also the heap
>>> optimizations could go to their own file.
>>>
>>> Carl Friedrich
>>
>> How about a couple of files (preferably small) each containing a
>> contained optimization if possible? (maybe a package?)
>>
>
>
>
> --
> H?kan Ard?
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From amauryfa at gmail.com  Thu Sep  2 10:28:14 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 2 Sep 2010 10:28:14 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <569927.57152.qm@web53706.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
Message-ID: <AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>

Hi,

2010/9/2 Saravanan Shanmugham <sarvi at yahoo.com>:
> We have Pickle and cPickle for performance. The Pickle could have simply been
> rewritten in RPython and probably compiled and we don't need ?different versons
> :-))

The PyPy way is much simpler:
there is only the original pickle.py, written in plain full Python,
and it's as fast as a C or RPython implementation.

-- 
Amaury Forgeot d'Arc


From sarvi at yahoo.com  Thu Sep  2 10:37:39 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 01:37:39 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
	<AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
Message-ID: <527977.40072.qm@web53701.mail.re2.yahoo.com>

awesome.

The point I was making is that RPython(a static subset of Python) will be faster 
than Dynamic Python code on a JIT or compiled to machine code.

Sarvi


----- Original Message ----
From: Amaury Forgeot d'Arc <amauryfa at gmail.com>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: Douglas McNeil <mcneil at hku.hk>; "pypy-dev at codespeak.net" 
<pypy-dev at codespeak.net>
Sent: Thu, September 2, 2010 1:28:14 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Hi,

2010/9/2 Saravanan Shanmugham <sarvi at yahoo.com>:
> We have Pickle and cPickle for performance. The Pickle could have simply been
> rewritten in RPython and probably compiled and we don't need  different 
versons
> :-))

The PyPy way is much simpler:
there is only the original pickle.py, written in plain full Python,
and it's as fast as a C or RPython implementation.

-- 
Amaury Forgeot d'Arc



      


From william.leslie.ttg at gmail.com  Thu Sep  2 10:40:03 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Thu, 2 Sep 2010 18:40:03 +1000
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <527977.40072.qm@web53701.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
	<AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
	<527977.40072.qm@web53701.mail.re2.yahoo.com>
Message-ID: <AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>

But what makes you think that? A dynamic compiler has more information, so
it should be able to produce better code.

On 02/09/2010 6:37 PM, "Saravanan Shanmugham" <sarvi at yahoo.com> wrote:

awesome.

The point I was making is that RPython(a static subset of Python) will be
faster
than Dynamic Python code on a JIT or compiled to machine code.

Sarvi



----- Original Message ----
From: Amaury Forgeot d'Arc <amauryfa at gmail.com>
To: Saravanan Shanmugh...

Sent: Thu, September 2, 2010 1:28:14 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Hi,

2010/9/2 Saravanan Shanmugham <sarvi at yahoo.com>:
> We have Pickle and cPickle for performance. ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100902/d4f04e18/attachment.html>

From p.giarrusso at gmail.com  Thu Sep  2 10:56:14 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 2 Sep 2010 10:56:14 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
	<AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
	<527977.40072.qm@web53701.mail.re2.yahoo.com>
	<AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>
Message-ID: <AANLkTi=VrKcD8gM1MQUv1xEri_cvggQTF++9bfQT7ZLa@mail.gmail.com>

On Thu, Sep 2, 2010 at 10:40, William Leslie
<william.leslie.ttg at gmail.com>wrote:

> But what makes you think that? A dynamic compiler has more information, so
> it should be able to produce better code.
>
Note that he's not arguing about a static compiler for the same code, which
has no type information, and where you are obviously right. He's arguing
about a statically typed language, where the type information is already
there in the source, e.g. C - there is much less information missing.
Actually, your point can still be made, but it becomes much less obvious.
For this case, it's much more contended what's best - see the "java faster
than C" debate. Nobody has yet given a proof convincing enough to close the
debate.

I would say that there's a tradeoff between JIT and Ahead-Of-Time
compilation, when AOT makes sense (not in Python, SmallTalk, Self...).

On 02/09/2010 6:37 PM, "Saravanan Shanmugham" <sarvi at yahoo.com> wrote:
>
> The point I was making is that RPython(a static subset of Python) will be
> faster
> than Dynamic Python code on a JIT or compiled to machine code.
>
> Note that, with enough implementation effort, that doesn't need to be true.
Run-time specialization would allow exactly the same code to be generated,
without any extra guards in the inner loop. Java can do that at times, and
can even be better than C, but not always (see above). You'd need a static
compiler with Profile-Guided Optimization and have a profile which matches
runtime, to guarantee superior results.

Cheers
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100902/d33bb6f2/attachment.html>

From sarvi at yahoo.com  Thu Sep  2 11:00:27 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 02:00:27 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
	<AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
	<527977.40072.qm@web53701.mail.re2.yahoo.com>
	<AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>
Message-ID: <996190.16331.qm@web53703.mail.re2.yahoo.com>

So far as I can tell from Unladed Swallow and PyPy, it is some of these Dynamic 
features of Python, such as Dynamic Typing that make it hard to compile/optimize 
and hit C like speeds.

Hence the need for RPython in PyPy or Restricted Python in Shedskin?

Sarvi



________________________________
From: William Leslie <william.leslie.ttg at gmail.com>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: "pypy-dev at codespeak.net" <pypy-dev at codespeak.net>; Amaury Forgeot d'Arc 
<amauryfa at gmail.com>
Sent: Thu, September 2, 2010 1:40:03 AM
Subject: Re: [pypy-dev] Question on the future of RPython


But what makes you think that? A dynamic compiler has more information, so it 
should be able to produce better code.
On 02/09/2010 6:37 PM, "Saravanan Shanmugham" <sarvi at yahoo.com> wrote:
>
>awesome.
>
>The point I was making is that RPython(a static subset of Python) will be 
faster
>than Dynamic Python code on a JIT or compiled to machine code.
>
>Sarvi
>
>
>
>----- Original Message ----
>From: Amaury Forgeot d'Arc <amauryfa at gmail.com>
>To: Saravanan Shanmugh...
>Sent: Thu, September 2, 2010 1:28:14 AM
>Subject: Re: [pypy-dev] Question on the future of RPython
>
>Hi,
>
>2010/9/2 Saravanan Shanmugham <sarvi at yahoo.com>:
>> We have Pickle and cPickle for performance. ...


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100902/84eb7267/attachment.html>

From fijall at gmail.com  Thu Sep  2 11:35:05 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 2 Sep 2010 11:35:05 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTimUU013WRhXgEi=pgEEQRUBJBkaQj8kB9aHZOnR@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
	<AANLkTimUU013WRhXgEi=pgEEQRUBJBkaQj8kB9aHZOnR@mail.gmail.com>
Message-ID: <AANLkTinxk84=8jO81zQ+s7GhiiM5hzKYnYBS8kFTtmYJ@mail.gmail.com>

On Thu, Sep 2, 2010 at 9:56 AM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> Hi,
> I was curious about the interplay between type inference and separate
> compilation.
>
> On Thu, Sep 2, 2010 at 09:09, William Leslie
> <william.leslie.ttg at gmail.com> wrote:
>> At the moment, the lack of separate compilation is a real issue
>> standing in the way of using rpython as a general purpose language, or
>> even as an extension language. Having to re-translate *everything*
>> every time you want to install an extension module is not on. Even C
>> doesn't require that.
>
>> The other is that type inference is global and changes you make to one
>> function can have far-reaching consequences.
> Is it module-global or is it performed on the whole program?
> I guess you'd need modular type inference before allowing separate
> compilation, and of course lots of implementation work.
> Functional languages allow separate compilation - is there any
> RPython-specific problem for that? I've omitted my guesses here.

There is no notion of a "module" in RPython. RPython is compiled from
live python objects (hence python is a metaprogramming language for
RPython). There is a bunch of technical problems, but it's generally
possible to implement separate compilation (it's work though).

Cheers,
fijal

> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From jacob at openend.se  Thu Sep  2 11:40:45 2010
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Thu, 2 Sep 2010 11:40:45 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <525425.61205.qm@web53702.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
Message-ID: <201009021140.53415.jacob@openend.se>

Thursday 02 September 2010 you wrote:
> I understand from various threads here,  that RPython is not for general
> purpose use.
> Why this lack of Focus on general use.
> 
> I am looking at this and I am thinking and comparing this to a corporation
> that is working on this awesome product.
> 
> They are so focused on this awesome final product vision that they fail to
> realize the awesome potential of some if its intermediate side
> deliverables.
> 
> PyPy is definitely gaining momentum.
> But as a strategy to build that momentum, and gain new converts it should
> put some focus on some of its niche strengths.
> Things other python implementions cannot do.
> 
> One such niche is its RPython and RPython Compiler.
> No other python implementation can convert python programs to executables.
> I am seeing growing interest in writing Rpython code for performance
> critical code and even potentially compiling it to binaries.
> 
> http://olliwang.com/2009/12/20/aes-implementation-in-rpython/
> http://alexgaynor.net/2010/may/15/pypy-future-python/
> 
> 
> Is it possible the PyPy team may be understating the significance of
> RPython? Am I crazy to think this way? :-)

RPython was tried in a production environment some years ago and while it 
produced some very nice results, it was quite difficult to work with. Dealing 
with those difficulties requires a group of people who are willing to build 
RPython code for general applications, run the code and identify what the 
difficulties actually are. Then they need to come up with strategies for how 
to remedy the problems and implement them in code. This is a very large 
undertaking for which Pypy does not have the manpower.. It also reqires people 
who are interested in building support for compiled programming languages. 
Pypy is a volunteer effort and the only person who was interested in this has 
retired from the project.

Jacob Hall?n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100902/d69f8a19/attachment.pgp>

From william.leslie.ttg at gmail.com  Thu Sep  2 11:52:56 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Thu, 2 Sep 2010 19:52:56 +1000
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <996190.16331.qm@web53703.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTindMzKH+Oufi5qbUBODnODk3UwnqLQXFNKAc4M=@mail.gmail.com>
	<569927.57152.qm@web53706.mail.re2.yahoo.com>
	<AANLkTimbYiL5+28PSdJ+PNGKez2i=xHY+S_SX6Aod4=B@mail.gmail.com>
	<527977.40072.qm@web53701.mail.re2.yahoo.com>
	<AANLkTinhDTcrsLvzErXD_kKQZU9tGcRmcfMjXvBv=QmC@mail.gmail.com>
	<996190.16331.qm@web53703.mail.re2.yahoo.com>
Message-ID: <AANLkTi=OaUytAKfzSqo=a0U8TS3xYnKLx1SBbVtLR5Wz@mail.gmail.com>

On 2 September 2010 19:00, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> So far as I can tell from Unladed Swallow and PyPy, it is some of these
> Dynamic features of Python, such as Dynamic Typing that make it hard to
> compile/optimize and hit C like speeds.
> Hence the need for RPython in PyPy or Restricted Python in Shedskin?

Hence the need for the JIT, not rpython. Rpython is an implementation
detail, to support translation easily to C as well as CLI and JVM
bytecode, and to support translation aspects such as stackless, and
testing on top of a full python environment. Rewriting things in
rpython for performance is a hack that should stop happening as the
JIT matures.

Dynamic typing means you need to do more work to produce code of the
same performance, but it's not impossible.


On 2 September 2010 18:56, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Thu, Sep 2, 2010 at 10:40, William Leslie <william.leslie.ttg at gmail.com>
> wrote:
>>
>> But what makes you think that? A dynamic compiler has more information, so
>> it should be able to produce better code.
>
> Note that he's not arguing about a static compiler for the same code, which
> has no type information, and where you are obviously right. He's arguing
> about a statically typed language, where the type information is already
> there in the source, e.g. C - there is much less information missing.
> Actually, your point can still be made, but it becomes much less obvious.
> For this case, it's much more contended what's best - see the "java faster
> than C" debate. Nobody has yet given a proof convincing enough to close the
> debate.

Sure - having static type guarantees is another case of "more information".

There is a little more room for discussion here, because there are
cases where a dynamic compiler for a safe runtime can do better at
considering certain optimisations, too. We have been talking about our
stock-standard type systems here, which ensure that our object will
have the field or method that we are interested in at runtime, and
perhaps (as long as it isn't an interface method, which we don't have
in rpython anyway) the offset into the instance or vtable
respectively. That makes for a pretty decent optimisation, but type
systems can infer much more than this, including which objects may
escape (via region typing a-la cyclone), which fields may be None, and
which instructions are loop invariant. The point is that some of these
type systems work fine with separate compilation, and some do
significantly better with runtime or linktime specialisation.

On 2 September 2010 17:56, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Thu, Sep 2, 2010 at 09:09, William Leslie
> <william.leslie.ttg at gmail.com> wrote:
>> The other is that type inference is global and changes you make to one
>> function can have far-reaching consequences.
> Is it module-global or is it performed on the whole program?

Rtyping is whole-program.

> Functional languages allow separate compilation - is there any
> RPython-specific problem for that? I've omitted my guesses here.

Many do, yes. To use ML derivatives as an example, you require the
signature of any modules you directly import. I was recently reading
about MLKit's module system, which is quite interesting (it has region
typing, and the way it unifies these types at the module boundary is
cute - carrying region information around in the source text is
fragile, so must be inferred). Haskell is kind of a special case,
requiring dictionaries to be passed around at runtime to determine
which method of some typeclass to call.

For OCaml (most MLs are similar) see section 2.5: "Modules and
separate compilation" of
http://pauillac.inria.fr/ocaml/htmlman/manual004.html

On MLKit's module implementation and region inference:
http://www.itu.dk/research/mlkit/index.php/Static_Interpretation

-- 
William Leslie


From p.giarrusso at gmail.com  Thu Sep  2 16:10:07 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 2 Sep 2010 16:10:07 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <694746.77392.qm@web53706.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<AANLkTims0Ngf2ez01DPisizO8oT6Z0VJZGDUaGLBmnvJ@mail.gmail.com>
	<69412.54494.qm@web53706.mail.re2.yahoo.com>
	<i5nm6u$slk$1@dough.gmane.org>
	<694746.77392.qm@web53706.mail.re2.yahoo.com>
Message-ID: <AANLkTimvxtDxrWV702+9YNY2-hHK15SzMwzUYTK9123=@mail.gmail.com>

On Thu, Sep 2, 2010 at 10:18, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> I have researched these projects quite extensively.
> Quite similar beasts as far as I can tell.
>
> Cython/Pyrex used to write python extensions. They use statically typed variants
> of Python which gets compiled into C which can then be compiled.
>
> Shedskin is slightly more general purpose Restricted Python to C++ compiler.
>
> PyPy as I understand can convert RPython into C code
>
> Am I missing something here?

Maybe you have done extensive research, but the above is not enough
for the conclusion, which might still be valid.
There could be some cool way to reuse each other's code, and that
would be cool given the available manpower.

The question is:
=> Do different goals cause _incompatible_ design/implementation choices?
Currently, static typing versus global type inference seems to be
already a fundamental difference. Modular type inference, if
polymorphic (and I guess it has to be), would require using boxed or
tagged integers more often, as far as I can see.

RPython is intended to be compiled to various environments, with
different variations (choose GC/stackful or stackless/heaps of other
choices), and its programmers are somewhat OK with its limitations; it
has type inference, with its set of tradeoffs. This for instance
prevents reusing shedskin, and probably even prevents reusing any of
its code.

Cython/Shedskin are intended to be used by more people and to be simpler.

==> Would making RPython usable for people harm its usability for PyPy?

I see no trivial answer to the above questions which allows merging,
but I don't develop any of them.
However, a discussion of this could probably end in a PyPy FAQ.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From sarvi at yahoo.com  Thu Sep  2 19:18:56 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 10:18:56 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <201009021140.53415.jacob@openend.se>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
Message-ID: <933850.84117.qm@web53705.mail.re2.yahoo.com>

one response inline


----- Original Message ----
From: Jacob Hall?n <jacob at openend.se>
To: pypy-dev at codespeak.net
Sent: Thu, September 2, 2010 2:40:45 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Thursday 02 September 2010 you wrote:
> I understand from various threads here,  that RPython is not for general
> purpose use.
> Why this lack of Focus on general use.
> 
> I am looking at this and I am thinking and comparing this to a corporation
> that is working on this awesome product.
> 
> They are so focused on this awesome final product vision that they fail to
> realize the awesome potential of some if its intermediate side
> deliverables.
> 
> PyPy is definitely gaining momentum.
> But as a strategy to build that momentum, and gain new converts it should
> put some focus on some of its niche strengths.
> Things other python implementions cannot do.
> 
> One such niche is its RPython and RPython Compiler.
> No other python implementation can convert python programs to executables.
> I am seeing growing interest in writing Rpython code for performance
> critical code and even potentially compiling it to binaries.
> 
> http://olliwang.com/2009/12/20/aes-implementation-in-rpython/
> http://alexgaynor.net/2010/may/15/pypy-future-python/
> 
> 
> Is it possible the PyPy team may be understating the significance of
> RPython? Am I crazy to think this way? :-)

RPython was tried in a production environment some years ago and while it 
produced some very nice results, it was quite difficult to work with. Dealing 
with those difficulties requires a group of people who are willing to build 
RPython code for general applications, run the code and identify what the 
difficulties actually are. Then they need to come up with strategies for how 
to remedy the problems and implement them in code. This is a very large 
undertaking for which Pypy does not have the manpower.. It also reqires people 
who are interested in building support for compiled programming languages. 
Pypy is a volunteer effort and the only person who was interested in this has 
retired from the project.

Sarvi>>>>
This makes sense. 
But wouldn't the answer to this problem be to invite people like the 
Shedskin/Cython developers to join forces with PyPy?
So that they can pursue the general RPython usecase you mention above while the 
others focus on JIT and stuff on a common code base?

Wouldn't that be a win-win for everybody?

This collaboration feels so obvious to me, that I am confused why it isn't to 
others.
Considering that Shedskin's goals feel almost like a strict subset of PyPy.


Sarvi



Jacob Hall?n



      


From santagada at gmail.com  Thu Sep  2 20:18:32 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 2 Sep 2010 15:18:32 -0300
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <933850.84117.qm@web53705.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
Message-ID: <AANLkTim4NBm4LL7sZYQg6FZfvfvSGJn4FPy-k9r_b9oR@mail.gmail.com>

On Thu, Sep 2, 2010 at 2:18 PM, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> one response inline
>
>
> ----- Original Message ----
> From: Jacob Hall?n <jacob at openend.se>
> To: pypy-dev at codespeak.net
> Sent: Thu, September 2, 2010 2:40:45 AM
> Subject: Re: [pypy-dev] Question on the future of RPython
>
> Thursday 02 September 2010 you wrote:
>> I understand from various threads here, ?that RPython is not for general
>> purpose use.
>> Why this lack of Focus on general use.
>>
>> I am looking at this and I am thinking and comparing this to a corporation
>> that is working on this awesome product.
>>
>> They are so focused on this awesome final product vision that they fail to
>> realize the awesome potential of some if its intermediate side
>> deliverables.
>>
>> PyPy is definitely gaining momentum.
>> But as a strategy to build that momentum, and gain new converts it should
>> put some focus on some of its niche strengths.
>> Things other python implementions cannot do.
>>
>> One such niche is its RPython and RPython Compiler.
>> No other python implementation can convert python programs to executables.
>> I am seeing growing interest in writing Rpython code for performance
>> critical code and even potentially compiling it to binaries.
>>
>> http://olliwang.com/2009/12/20/aes-implementation-in-rpython/
>> http://alexgaynor.net/2010/may/15/pypy-future-python/
>>
>>
>> Is it possible the PyPy team may be understating the significance of
>> RPython? Am I crazy to think this way? :-)
>
> RPython was tried in a production environment some years ago and while it
> produced some very nice results, it was quite difficult to work with. Dealing
> with those difficulties requires a group of people who are willing to build
> RPython code for general applications, run the code and identify what the
> difficulties actually are. Then they need to come up with strategies for how
> to remedy the problems and implement them in code. This is a very large
> undertaking for which Pypy does not have the manpower.. It also reqires people
> who are interested in building support for compiled programming languages.
> Pypy is a volunteer effort and the only person who was interested in this has
> retired from the project.
>
> Sarvi>>>>
> This makes sense.
> But wouldn't the answer to this problem be to invite people like the
> Shedskin/Cython developers to join forces with PyPy?
> So that they can pursue the general RPython usecase you mention above while the
> others focus on JIT and stuff on a common code base?
>
> Wouldn't that be a win-win for everybody?
>
> This collaboration feels so obvious to me, that I am confused why it isn't to
> others.
> Considering that Shedskin's goals feel almost like a strict subset of PyPy.

I think what you don't get is how open source works, there is always
ten projects doing almost the same thing. Everyone at least once
thought "why does linux has this many media players/text editors/flash
implementations/jvm when all we need is a really good one with lots of
support". I does get me depressed sometimes, but this is the way it
is.

Cython has a big user base that they have to support and lots of
programs that are in production today, shedskin is looking for pure
performance and the pypy guys want to have a faster python. Although I
also think that maybe RPython and the pypy python interpreter could
solve all this problems someday it doesn't do so right now.

I have used RPython in the past and the error messages alone would
drive some people away. Some group of people could work to fix this,
but I doubt it will happen soon.

What I think could be done to make pypy more visible to people would
be to have a killer app running on pypy way faster/better than on
cpython. For me this app is either mercurial or django.

-- 
Leonardo Santagada


From sarvi at yahoo.com  Thu Sep  2 22:03:31 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Thu, 2 Sep 2010 13:03:31 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTim4NBm4LL7sZYQg6FZfvfvSGJn4FPy-k9r_b9oR@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<AANLkTim4NBm4LL7sZYQg6FZfvfvSGJn4FPy-k9r_b9oR@mail.gmail.com>
Message-ID: <56902.77920.qm@web53708.mail.re2.yahoo.com>





----- Original Message ----
From: Leonardo Santagada <santagada at gmail.com>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: Jacob Hall?n <jacob at openend.se>; pypy-dev at codespeak.net
Sent: Thu, September 2, 2010 11:18:32 AM
Subject: Re: [pypy-dev] Question on the future of RPython

On Thu, Sep 2, 2010 at 2:18 PM, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> one response inline
>
>
> ----- Original Message ----
> From: Jacob Hall?n <jacob at openend.se>
> To: pypy-dev at codespeak.net
> Sent: Thu, September 2, 2010 2:40:45 AM
> Subject: Re: [pypy-dev] Question on the future of RPython
>
> Thursday 02 September 2010 you wrote:
>> I understand from various threads here,  that RPython is not for general
>> purpose use.
>> Why this lack of Focus on general use.
>>
>> I am looking at this and I am thinking and comparing this to a corporation
>> that is working on this awesome product.
>>
>> They are so focused on this awesome final product vision that they fail to
>> realize the awesome potential of some if its intermediate side
>> deliverables.
>>
>> PyPy is definitely gaining momentum.
>> But as a strategy to build that momentum, and gain new converts it should
>> put some focus on some of its niche strengths.
>> Things other python implementions cannot do.
>>
>> One such niche is its RPython and RPython Compiler.
>> No other python implementation can convert python programs to executables.
>> I am seeing growing interest in writing Rpython code for performance
>> critical code and even potentially compiling it to binaries.
>>
>> http://olliwang.com/2009/12/20/aes-implementation-in-rpython/
>> http://alexgaynor.net/2010/may/15/pypy-future-python/
>>
>>
>> Is it possible the PyPy team may be understating the significance of
>> RPython? Am I crazy to think this way? :-)
>
> RPython was tried in a production environment some years ago and while it
> produced some very nice results, it was quite difficult to work with. Dealing
> with those difficulties requires a group of people who are willing to build
> RPython code for general applications, run the code and identify what the
> difficulties actually are. Then they need to come up with strategies for how
> to remedy the problems and implement them in code. This is a very large
> undertaking for which Pypy does not have the manpower.. It also reqires people
> who are interested in building support for compiled programming languages.
> Pypy is a volunteer effort and the only person who was interested in this has
> retired from the project.
>
> Sarvi>>>>
> This makes sense.
> But wouldn't the answer to this problem be to invite people like the
> Shedskin/Cython developers to join forces with PyPy?
> So that they can pursue the general RPython usecase you mention above while 
the
> others focus on JIT and stuff on a common code base?
>
> Wouldn't that be a win-win for everybody?
>
> This collaboration feels so obvious to me, that I am confused why it isn't to
> others.
> Considering that Shedskin's goals feel almost like a strict subset of PyPy.

I think what you don't get is how open source works, there is always
ten projects doing almost the same thing. Everyone at least once
thought "why does linux has this many media players/text editors/flash
implementations/jvm when all we need is a really good one with lots of
support". I does get me depressed sometimes, but this is the way it
is.

Cython has a big user base that they have to support and lots of
programs that are in production today, shedskin is looking for pure
performance and the pypy guys want to have a faster python. Although I
also think that maybe RPython and the pypy python interpreter could
solve all this problems someday it doesn't do so right now.

I have used RPython in the past and the error messages alone would
drive some people away. Some group of people could work to fix this,
but I doubt it will happen soon.



Sarvi>> Yeah having this thread of conversation on 4 separate aliases, 
Python.org, Unladen Swallow, Shedskn and PyPy was just my attempt 
at seeing if these can come together.
Oh well.



What I think could be done to make pypy more visible to people would
be to have a killer app running on pypy way faster/better than on
cpython. For me this app is either mercurial or django.

Sarvi>>> Very True. 

Sarvi




-- 
Leonardo Santagada



      


From jacob at openend.se  Thu Sep  2 22:54:01 2010
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Thu, 2 Sep 2010 22:54:01 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <933850.84117.qm@web53705.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
Message-ID: <201009022254.07665.jacob@openend.se>

Thursday 02 September 2010 you wrote:
> This makes sense.
> But wouldn't the answer to this problem be to invite people like the
> Shedskin/Cython developers to join forces with PyPy?
> So that they can pursue the general RPython usecase you mention above while
> the others focus on JIT and stuff on a common code base?
> 
> Wouldn't that be a win-win for everybody?
> 
> This collaboration feels so obvious to me, that I am confused why it isn't
> to others.
> Considering that Shedskin's goals feel almost like a strict subset of PyPy.

It is a matter of personal pride, I think. If we made the invitation to the 
Shedskin people they would see this as "Pypy thinks they are way cooler than 
us, so they invite us to be part of their project". This would naturally 
generate a refusal, because even though we don't make such value statements, 
it would be viewed that way.

So, we don't make such invitations, even if they make sense.

What we hope is that some people examine the Pypy project and find that it 
actually is a really cool piece of technology with lots of possible side 
projects and expansion possibilities. If they decide to join the project we 
will give them all the help we are capable of.

Most people have actually joined Pypy in this way. The most recent example is 
H?kan Ard? who wanted to expand Pypy in the direction of numeric calculations. 
The learning curve is fairly steep, but there are quite a few people on the 
IRC channel who are ready to help you overcome the hurdles.

Jacob Hall?n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100902/39455c75/attachment.pgp>

From sarvi at yahoo.com  Fri Sep  3 11:11:13 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Fri, 3 Sep 2010 02:11:13 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <201009022254.07665.jacob@openend.se>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
Message-ID: <810929.87945.qm@web53702.mail.re2.yahoo.com>

I have heard repeatedly in this alias that PyPy's RPython is very difficult to 
use.

I have also heard here and elsewhere that Shedskin fast and is great for what it 
does i.e. translate its version of Restricted Python to C++.

Which then begs the question, would it make sense for PyPy to adopt Shedskin to 
compile its PyPy RPython code into C++/binary.

Sarvi


----- Original Message ----
From: Jacob Hall?n <jacob at openend.se>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: pypy-dev at codespeak.net
Sent: Thu, September 2, 2010 1:54:01 PM
Subject: Re: [pypy-dev] Question on the future of RPython

Thursday 02 September 2010 you wrote:
> This makes sense.
> But wouldn't the answer to this problem be to invite people like the
> Shedskin/Cython developers to join forces with PyPy?
> So that they can pursue the general RPython usecase you mention above while
> the others focus on JIT and stuff on a common code base?
> 
> Wouldn't that be a win-win for everybody?
> 
> This collaboration feels so obvious to me, that I am confused why it isn't
> to others.
> Considering that Shedskin's goals feel almost like a strict subset of PyPy.

It is a matter of personal pride, I think. If we made the invitation to the 
Shedskin people they would see this as "Pypy thinks they are way cooler than 
us, so they invite us to be part of their project". This would naturally 
generate a refusal, because even though we don't make such value statements, 
it would be viewed that way.

So, we don't make such invitations, even if they make sense.

What we hope is that some people examine the Pypy project and find that it 
actually is a really cool piece of technology with lots of possible side 
projects and expansion possibilities. If they decide to join the project we 
will give them all the help we are capable of.

Most people have actually joined Pypy in this way. The most recent example is 
H?kan Ard? who wanted to expand Pypy in the direction of numeric calculations. 
The learning curve is fairly steep, but there are quite a few people on the 
IRC channel who are ready to help you overcome the hurdles.

Jacob Hall?n



      


From stefan_ml at behnel.de  Fri Sep  3 11:30:32 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Sep 2010 11:30:32 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <810929.87945.qm@web53702.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<201009021140.53415.jacob@openend.se>	<933850.84117.qm@web53705.mail.re2.yahoo.com>	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
Message-ID: <i5qf7p$g4h$2@dough.gmane.org>

Saravanan Shanmugham, 03.09.2010 11:11:
> From: Jacob Hall?n
>> It is a matter of personal pride, I think. If we made the invitation to the
>> Shedskin people they would see this as "Pypy thinks they are way cooler than
>> us, so they invite us to be part of their project". This would naturally
>> generate a refusal, because even though we don't make such value statements,
>> it would be viewed that way.
>>
>> So, we don't make such invitations, even if they make sense.
>
> I have heard repeatedly in this alias that PyPy's RPython is very difficult to
> use.
>
> I have also heard here and elsewhere that Shedskin fast and is great for what it
> does i.e. translate its version of Restricted Python to C++.
>
> Which then begs the question, would it make sense for PyPy to adopt Shedskin to
> compile its PyPy RPython code into C++/binary.

You should seriously read and try to understand the e-mails that you reply 
to, instead of top-posting them away.

Stefan



From sarvi at yahoo.com  Fri Sep  3 19:22:58 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Fri, 3 Sep 2010 10:22:58 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <i5qf7p$g4h$2@dough.gmane.org>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
Message-ID: <609691.54456.qm@web53705.mail.re2.yahoo.com>

Lets not be a little presumptious shall we.
This is the second time you seem to be claiming that I haven't done my 
research/reading.

I have been following the progress of PyPy over 2 years. Its great work. So is 
Shedskin.
Just for the record, I have used PyRex, Cython.
And have read the documentation and/or sample code for both Shedskin and PyPy

If people say that there are emotional and pragmatic reasons for the 2 projects 
not coming together. 
That makes sense.

I just don't see any logical reasons, thats all. And I haven't heard any on this 
thread either.

BTW, Just because I top post for "readability" doesn't mean I haven't read all 
the threads in detail.

Sarvi


----- Original Message ----
From: Stefan Behnel <stefan_ml at behnel.de>
To: pypy-dev at codespeak.net
Sent: Fri, September 3, 2010 2:30:32 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Saravanan Shanmugham, 03.09.2010 11:11:
> From: Jacob Hall?n
>> It is a matter of personal pride, I think. If we made the invitation to the
>> Shedskin people they would see this as "Pypy thinks they are way cooler than
>> us, so they invite us to be part of their project". This would naturally
>> generate a refusal, because even though we don't make such value statements,
>> it would be viewed that way.
>>
>> So, we don't make such invitations, even if they make sense.
>
> I have heard repeatedly in this alias that PyPy's RPython is very difficult to
> use.
>
> I have also heard here and elsewhere that Shedskin fast and is great for what 
>it
> does i.e. translate its version of Restricted Python to C++.
>
> Which then begs the question, would it make sense for PyPy to adopt Shedskin 
to
> compile its PyPy RPython code into C++/binary.

You should seriously read and try to understand the e-mails that you reply 
to, instead of top-posting them away.

Stefan

_______________________________________________
pypy-dev at codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev


      


From stefan_ml at behnel.de  Fri Sep  3 19:52:41 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Sep 2010 19:52:41 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <609691.54456.qm@web53705.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<201009021140.53415.jacob@openend.se>	<933850.84117.qm@web53705.mail.re2.yahoo.com>	<201009022254.07665.jacob@openend.se>	<810929.87945.qm@web53702.mail.re2.yahoo.com>	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
Message-ID: <i5rcl9$ip2$1@dough.gmane.org>

Saravanan Shanmugham, 03.09.2010 19:22:
> Lets not be a little presumptious shall we.
> This is the second time you seem to be claiming that I haven't done my
> research/reading.

That's just the impression that I get from what you write and how you write it.


> I just don't see any logical reasons, thats all. And I haven't heard any on this
> thread either.

Well, you are talking to people who know a lot more about what you are 
talking about than you do. It's normal that they are not equally 
enthusiastic about pie-in-the-sky ideas that someone throws at them.


> BTW, Just because I top post for "readability" doesn't mean I haven't read all
> the threads in detail.

I like the fact that you put marks of irony around the word "readability".

Stefan



From amauryfa at gmail.com  Fri Sep  3 20:16:11 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 3 Sep 2010 20:16:11 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <810929.87945.qm@web53702.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
Message-ID: <AANLkTim5qE1bUOM33uwPuVJuYT=qQ7nPPsbSNQpjEBJN@mail.gmail.com>

Hi,
2010/9/3 Saravanan Shanmugham <sarvi at yahoo.com>
>
> I have heard repeatedly in this alias that PyPy's RPython is very difficult to
> use.
>
> I have also heard here and elsewhere that Shedskin fast and is great for what it
> does i.e. translate its version of Restricted Python to C++.
>
> Which then begs the question, would it make sense for PyPy to adopt Shedskin to
> compile its PyPy RPython code into C++/binary.

But PyPy does not translate RPython code to C++.
Or before doing so, it performs transformations to the code that
require the analysis
of the program as a whole and that a C++ compiler cannot do, like the choice
of a garbage collector, the stackless mode, and most of all the
generation of a tracing JIT.

It also operates on the bytecode, which offers interesting
metaprogramming techniques
that are used throughout the code (similar to C++ templates, for example,
except that it's written in Python :-) )

Shedskin on the other hand performs a more direct translation of
Python code (it uses the ast)

Both projects don't have the same goals.

--
Amaury Forgeot d'Arc


From sarvi at yahoo.com  Fri Sep  3 21:06:14 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Fri, 3 Sep 2010 12:06:14 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <i5rcl9$ip2$1@dough.gmane.org>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
Message-ID: <294101.49986.qm@web53704.mail.re2.yahoo.com>

Stefan, 
    If I were to go with my impressions, based on you being the lead developer 
of Cython, I could have claimed you have an ulterior motive on this thread.
But then I didn't because, inspite of first impressions/scepticism I believe we 
are all here with a genuine interest to improve the Python environment and get 
more visibiity and momentum on PyPy. Personally I cant wait to see PyPy become 
the default Python. :-)

Lets start with an understanding that we are all smart people with good ideas 
and lets also not get too cocky enough to think we have all the answers.

I saw some genuine synergies that I was calling out.

And I have heard some pragmatic arguments from others though no one has 
necessarily claimed logical impossibility on why this may not work.
And I can understand that.

Sarvi



----- Original Message ----
From: Stefan Behnel <stefan_ml at behnel.de>
To: pypy-dev at codespeak.net
Sent: Fri, September 3, 2010 10:52:41 AM
Subject: Re: [pypy-dev] Question on the future of RPython

Saravanan Shanmugham, 03.09.2010 19:22:
> Lets not be a little presumptious shall we.
> This is the second time you seem to be claiming that I haven't done my
> research/reading.

That's just the impression that I get from what you write and how you write it.


> I just don't see any logical reasons, thats all. And I haven't heard any on 
>this
> thread either.

Well, you are talking to people who know a lot more about what you are 
talking about than you do. It's normal that they are not equally 
enthusiastic about pie-in-the-sky ideas that someone throws at them.


> BTW, Just because I top post for "readability" doesn't mean I haven't read all
> the threads in detail.

I like the fact that you put marks of irony around the word "readability".

Stefan

_______________________________________________
pypy-dev at codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev



      


From stefan_ml at behnel.de  Fri Sep  3 21:13:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Sep 2010 21:13:26 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <294101.49986.qm@web53704.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<201009021140.53415.jacob@openend.se>	<933850.84117.qm@web53705.mail.re2.yahoo.com>	<201009022254.07665.jacob@openend.se>	<810929.87945.qm@web53702.mail.re2.yahoo.com>	<i5qf7p$g4h$2@dough.gmane.org>	<609691.54456.qm@web53705.mail.re2.yahoo.com>	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
Message-ID: <i5rhcm$61g$1@dough.gmane.org>

Saravanan Shanmugham, 03.09.2010 21:06:
>      If I were to go with my impressions, based on you being the lead developer
> of Cython, I could have claimed you have an ulterior motive on this thread.

*shrug*

Stefan



From p.giarrusso at gmail.com  Sat Sep  4 01:34:24 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 4 Sep 2010 01:34:24 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <i5qf7p$g4h$2@dough.gmane.org>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
Message-ID: <AANLkTi=zW4k57TmUjE0+DtAO-AMDQZgYBnMVe2EFiofc@mail.gmail.com>

> You should seriously read and try to understand the e-mails that you reply
> to, instead of top-posting them away.

Stefan, there are different ways to argue the same valid thing, and
the way you chose is IMHO counterproductive for you - the only result
is offensive comments.
Also, while I seldom top-post, especially in public forums/MLs, IIRC
several PyPy contributors routinely top-post, and I see some sensible
arguments (see http://en.wikipedia.org/wiki/Posting_style#Top-posting).

Saravanan, a small part of the issue is that many people consider top
posting inappropriate and/or lame (for instance
http://www.caliburn.nl/topposting.html). Be aware of that risk if you
top-post. And please, never claim it increases readability - it makes
your post only readable if you read the whole thread. Non-crappy email
clients highlight differently the new text from the quoted text,
making the former easy to find.

In particular, by top-posting you never address the comments which
explain why merging does not necessarily make sense (like some of
mine), or the ones which argue it's a bad idea (like last Amaury's
mail). Interleaved replying brings instead to point-by-point answers.

See other comments below.
On Fri, Sep 3, 2010 at 11:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Saravanan Shanmugham, 03.09.2010 11:11:
>> From: Jacob Hall?n
>> I have heard repeatedly in this alias that PyPy's RPython is very difficult to
>> use.
This alias?? You mean this _thread_, don't you?
>> I have also heard here and elsewhere that Shedskin fast and is great for what it
>> does i.e. translate its version of Restricted Python to C++.

>> Which then begs the question, would it make sense for PyPy to adopt Shedskin to
>> compile its PyPy RPython code into C++/binary.

The answer is already implicit in one of my previous emails, and is a
very clear "no, unless considerable extra merging effort is done,
which might be more than the effort to make the RPython compiler
better than Shedskin". I paste a relevant subset of that mail at the
end; while I can believe that you have read it, I often do not
understand all the implications of what I read the first time, if
that's complex, like it is for everybody, so do not be offended if I
suggest you to re-read it again.

A similar, more detailed argument is discussed by Amaury Forgeot d'Arc
in an email where he replies to you.

In other mails, you write that:

> I just don't see any logical reasons [against the merger], thats all. And I haven't heard any on this
> thread either.

> no one has necessarily claimed logical impossibility on why this may not work.

which strikes me as _wrong_. The mails I mentioned explain why there
are different design goals - Amaury, who knows more about Shedskin
than me, explained why it is less general. That's already an answer
for me.
Of course, this does not prove impossibility, it only suggests that it
may be not a good idea to merge the projects. You shouldn't care about
logical impossibility, which makes _NO SENSE_ in such questions in
software engineering; what is possible and bad makes little sense.

If you meant "nobody claimed that this is necessarily a bad idea",
then I agree. We believe there's no obvious way to combine the
projects; anybody, including you, is welcome to address the specific
issues and find some clever solution. You didn't even scratch them,
yet. And while you claimed experience with using the projects, or
reading their documentation, it is not clear at all that you
understand their internals, and this is required to address these
problems.

The only idea which makes some sense is that instead of starting the
development of Shedskin, the author could have tried achieving the
same results improving RPython, fixing its error messages and so on.
However, I can imagine a ton of possible reasons for which he might
have consciously decided to do something else. The keyword here is
"design tradeoff": a design choice can make a product better in some
respects and worse in other ones. Shedskin is less flexible, but
possibly this gives technical advantages which are important. That's
the same thing as explained below.

Best regards

=====

=> Do different goals cause _incompatible_ design/implementation choices?
Currently, static typing versus global type inference seems to be
already a fundamental difference. Modular type inference, if
polymorphic (and I guess it has to be), would require using boxed or
tagged integers more often, as far as I can see.

RPython is intended to be compiled to various environments, with
different variations (choose GC/stackful or stackless/heaps of other
choices), and its programmers are somewhat OK with its limitations; it
has type inference, with its set of tradeoffs. This for instance
prevents reusing shedskin, and probably even prevents reusing any of
its code.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From p.giarrusso at gmail.com  Sat Sep  4 02:15:47 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 4 Sep 2010 02:15:47 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <294101.49986.qm@web53704.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
Message-ID: <AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>

On Fri, Sep 3, 2010 at 21:06, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> Stefan,
> ? ?If I were to go with my impressions, based on you being the lead developer
> of Cython, I could have claimed you have an ulterior motive on this thread.

> But then I didn't because, inspite of first impressions/scepticism I believe we
> are all here with a genuine interest to improve the Python environment

While you didn't initiate the flame, I think that's totally
inappropriate, and I can say so even without knowing Stefan.
You wrote:
> Lets not be a little presumptious shall we.
Well, rereading previous comments it is clear that:
a) you don't know well some basics of virtual machines which have been explained
Which is fine, but then you shouldn't consider yourself a peer to
developers of these projects. And you shouldn't claim you have done
proper research if you just used the projects.
You are welcome to be curious, but with such a comment you are the
presumptuous one.
Note that I already remarked that Stefan's comment was not appropriate in style.

b) your email client _is_ crappy, given the way you reply inline (I
was mentioning crappy clients in my previous email).
Socially speaking, in an Open Source community, not using a decent
email client can look as bad as dressing very very wrong. I'm not so
picky, but it does mean you're not a hacker.

Note I'm not a developer of PyPy, and I don't claim being an expert,
but I have some technical knowledge of its documentation about
internals and of some literature, and some small experience with a
Python implementation.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From sarvi at yahoo.com  Sat Sep  4 04:33:16 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Fri, 3 Sep 2010 19:33:16 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTi=zW4k57TmUjE0+DtAO-AMDQZgYBnMVe2EFiofc@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<AANLkTi=zW4k57TmUjE0+DtAO-AMDQZgYBnMVe2EFiofc@mail.gmail.com>
Message-ID: <597924.69989.qm@web53705.mail.re2.yahoo.com>





----- Original Message ----
From: Paolo Giarrusso <p.giarrusso at gmail.com>
To: Stefan Behnel <stefan_ml at behnel.de>; Saravanan Shanmugham <sarvi at yahoo.com>
Cc: pypy-dev at codespeak.net
Sent: Fri, September 3, 2010 4:34:24 PM
Subject: Re: [pypy-dev] Question on the future of RPython

> You should seriously read and try to understand the e-mails that you reply
> to, instead of top-posting them away.

Stefan, there are different ways to argue the same valid thing, and
the way you chose is IMHO counterproductive for you - the only result
is offensive comments.
Also, while I seldom top-post, especially in public forums/MLs, IIRC
several PyPy contributors routinely top-post, and I see some sensible
arguments (see http://en.wikipedia.org/wiki/Posting_style#Top-posting).

Saravanan, a small part of the issue is that many people consider top
posting inappropriate and/or lame (for instance
http://www.caliburn.nl/topposting.html). Be aware of that risk if you
top-post. And please, never claim it increases readability - it makes
your post only readable if you read the whole thread. Non-crappy email
clients highlight differently the new text from the quoted text,
making the former easy to find.

Sarvi>> Point taken. will keep that in mind.
It was misguided notion of what would be readable.

Sarvi 

In particular, by top-posting you never address the comments which
explain why merging does not necessarily make sense (like some of
mine), or the ones which argue it's a bad idea (like last Amaury's
mail). Interleaved replying brings instead to point-by-point answers.

See other comments below.
On Fri, Sep 3, 2010 at 11:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Saravanan Shanmugham, 03.09.2010 11:11:
>> From: Jacob Hall?n
>> I have heard repeatedly in this alias that PyPy's RPython is very difficult 
to
>> use.
This alias?? You mean this _thread_, don't you?
>> I have also heard here and elsewhere that Shedskin fast and is great for what 
>>it
>> does i.e. translate its version of Restricted Python to C++.

>> Which then begs the question, would it make sense for PyPy to adopt Shedskin 
>to
>> compile its PyPy RPython code into C++/binary.

The answer is already implicit in one of my previous emails, and is a
very clear "no, unless considerable extra merging effort is done,
which might be more than the effort to make the RPython compiler
better than Shedskin". I paste a relevant subset of that mail at the
end; while I can believe that you have read it, I often do not
understand all the implications of what I read the first time, if
that's complex, like it is for everybody, so do not be offended if I
suggest you to re-read it again.

A similar, more detailed argument is discussed by Amaury Forgeot d'Arc
in an email where he replies to you.

In other mails, you write that:

> I just don't see any logical reasons [against the merger], thats all. And I 
>haven't heard any on this
> thread either.

> no one has necessarily claimed logical impossibility on why this may not work.

which strikes me as _wrong_. The mails I mentioned explain why there
are different design goals - Amaury, who knows more about Shedskin
than me, explained why it is less general. That's already an answer
for me.
Of course, this does not prove impossibility, it only suggests that it
may be not a good idea to merge the projects. You shouldn't care about
logical impossibility, which makes _NO SENSE_ in such questions in
software engineering; what is possible and bad makes little sense.

If you meant "nobody claimed that this is necessarily a bad idea",
then I agree. We believe there's no obvious way to combine the
projects; anybody, including you, is welcome to address the specific
issues and find some clever solution. You didn't even scratch them,
yet. And while you claimed experience with using the projects, or
reading their documentation, it is not clear at all that you
understand their internals, and this is required to address these
problems.

The only idea which makes some sense is that instead of starting the
development of Shedskin, the author could have tried achieving the
same results improving RPython, fixing its error messages and so on.
However, I can imagine a ton of possible reasons for which he might
have consciously decided to do something else. The keyword here is
"design tradeoff": a design choice can make a product better in some
respects and worse in other ones. Shedskin is less flexible, but
possibly this gives technical advantages which are important. That's
the same thing as explained below.

Best regards

=====

=> Do different goals cause _incompatible_ design/implementation choices?
Currently, static typing versus global type inference seems to be
already a fundamental difference. Modular type inference, if
polymorphic (and I guess it has to be), would require using boxed or
tagged integers more often, as far as I can see.

RPython is intended to be compiled to various environments, with
different variations (choose GC/stackful or stackless/heaps of other
choices), and its programmers are somewhat OK with its limitations; it
has type inference, with its set of tradeoffs. This for instance
prevents reusing shedskin, and probably even prevents reusing any of
its code.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/



      


From sarvi at yahoo.com  Sat Sep  4 04:56:49 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Fri, 3 Sep 2010 19:56:49 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
Message-ID: <860977.83657.qm@web53705.mail.re2.yahoo.com>





----- Original Message ----
From: Paolo Giarrusso <p.giarrusso at gmail.com>
To: Saravanan Shanmugham <sarvi at yahoo.com>
Cc: Stefan Behnel <stefan_ml at behnel.de>; pypy-dev at codespeak.net
Sent: Fri, September 3, 2010 5:15:47 PM
Subject: Re: [pypy-dev] Question on the future of RPython

On Fri, Sep 3, 2010 at 21:06, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> Stefan,
>    If I were to go with my impressions, based on you being the lead developer
> of Cython, I could have claimed you have an ulterior motive on this thread.

> But then I didn't because, inspite of first impressions/scepticism I believe 
we
> are all here with a genuine interest to improve the Python environment

While you didn't initiate the flame, I think that's totally
inappropriate, and I can say so even without knowing Stefan.

Sarvi>> I believe it is an appropriate response to the flame bait.
BTW, I was very careful not to make the accusation. 
No real offense meant.
It was just a what if argument to drive the point that if every one responded 
like that, based on impression and presumptions,  it would be wrong.
So I standby that.

You wrote:
> Lets not be a little presumptious shall we.
Well, rereading previous comments it is clear that:
a) you don't know well some basics of virtual machines which have been explained
Which is fine, but then you shouldn't consider yourself a peer to
developers of these projects. And you shouldn't claim you have done
proper research if you just used the projects.
You are welcome to be curious, but with such a comment you are the
presumptuous one.
Note that I already remarked that Stefan's comment was not appropriate in style.

Sarvi>> We may have to agree to disagree here. 
I don't believe my thread of discussion has anything to do with Virtual Machines 
at al. 
What I have been saying has more to do with compiling plain RPython code into 
C/C++/ASM executables.

Shedskin uses a statically typed restricted version of Python that gets 
converted to C++
PyPy does convert a statically typed restricted version of Python to C that can 
then be compiled to an executable.
So though with different approachs the final goal is to produce an compiled 
binary executable for the RPython code.

Agreed PyPy does additionally allow using Language/JIT hints to help 
write/generate JIT compilers as well.

That does not remove the possibility that the statically typed version of 
Restricted Python used by Shedskin cannot be full subset of the PyPy RPython.
Nor that there is a possibility of using PyPy as just a plain/pure Restricted 
Python compiler. pure and simple.

This thought angle has nothing to do with Virtual Machines, really.


b) your email client _is_ crappy, given the way you reply inline (I
was mentioning crappy clients in my previous email).
Socially speaking, in an Open Source community, not using a decent
email client can look as bad as dressing very very wrong. I'm not so
picky, but it does mean you're not a hacker.

Sarvi>>> Point taken. 
I use plain Yahoo Web Mail. Do you have any suggestion how I could do better 
with the Yahoo Web Mail client???
I am open learning a better way. :-) Will look into it.


Thanks,
Sarvi


Note I'm not a developer of PyPy, and I don't claim being an expert,
but I have some technical knowledge of its documentation about
internals and of some literature, and some small experience with a
Python implementation.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/



      


From arigo at tunes.org  Sat Sep  4 11:03:24 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 4 Sep 2010 11:03:24 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <860977.83657.qm@web53705.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
Message-ID: <AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>

Hi,

Can we please close this thread?  The basic answer you will get from
anybody that actually worked at least a bit with PyPy is that all your
discussions are moving air around and nothing else.

There is no one working with PyPy that is interested in using RPython
for the purpose of compiling some RPython programs to C code
statically (except interpreters).  If anyone is really interested in
this topic he can (again) give it a try.  He would get some help from
us, i.e. the rest of the PyPy team, but it would be a fork for now.  I
say "again" because there are some previous attempts at doing that,
which all failed.  As long as no such project exists and is successful
-- and I have some doubts about it -- I will not believe in the nice
(and, to me, completely bogus) claims made on this thread, like "let's
bring RPython and Shedskin together".


A bientot,

Armin.


From bhartsho at yahoo.com  Sun Sep  5 06:50:44 2010
From: bhartsho at yahoo.com (Hart's Antler)
Date: Sat, 4 Sep 2010 21:50:44 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
Message-ID: <579650.12542.qm@web114018.mail.gq1.yahoo.com>

RPython is a nice language even right now, its very useful, i'm using it all the time.  Too many people downplay its potentials saying its too hard or complex.  I like the error messages, they help you understand how it all works; however it would be nice to see an official FAQ where the common errors are explained in plain english. RPython is only lacking a version number.
-brett





From sarvi at yahoo.com  Sun Sep  5 18:51:26 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Sun, 5 Sep 2010 09:51:26 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython - My Donation to
	the Cause
In-Reply-To: <579650.12542.qm@web114018.mail.gq1.yahoo.com>
References: <579650.12542.qm@web114018.mail.gq1.yahoo.com>
Message-ID: <333806.72573.qm@web53704.mail.re2.yahoo.com>

I think the work PyPy and Shedskin is doing is excellent for the python 
community.
And I think RPython has an excellent future if will give it a little bit of a 
push.

I am just a private citizen.
So here is a small bounty/price of motivation to the PyPy team.
I understand people do open source work for pride and not money.
And Yes $200 may sound like small. 
 
I woud contribute code if I had the time, but I already have my hands on too 
many side projects. 
So this is me helping another way.

Think of this as me hosting Pizza and Coke for one of the endless Sprints yall 
do :-))

Also I am hoping this is just seed money to motivate the RPython cause.
And I am sincerely hoping others interested in seeing RPython and the C backend 
of PyPy develop more completely will also add to this prize pool and show 
support for the cause.


Thx,
Sarvi


----- Original Message ----
> From: Hart's Antler <bhartsho at yahoo.com>
> To: pypy-dev at codespeak.net
> Sent: Sat, September 4, 2010 9:50:44 PM
> Subject: Re: [pypy-dev] Question on the future of RPython
> 
> RPython is a nice language even right now, its very useful, i'm using it all 
>the  time.  Too many people downplay its potentials saying its too hard or  
>complex.  I like the error messages, they help you understand how it all  works; 
>however it would be nice to see an official FAQ where the common errors  are 
>explained in plain english. RPython is only lacking a version  number.
> -brett
> 
> 
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


      


From sarvi at yahoo.com  Mon Sep  6 20:27:08 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Mon, 6 Sep 2010 11:27:08 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
Message-ID: <792921.22517.qm@web53703.mail.re2.yahoo.com>

Hi Armin,
   Could you point me to some of these previous attempts at improving 
RPython-to-Executable capability. 
I would like to understand what was attempted.

Hart's Antler, who seems to be working on RPython quite extensively contacted me 
privately about dong some work in the RPython area.
I am considering sponsoring him in to do some work on PyPy,only  if it is done 
with the PyPy teams blessings and will help help PyPy as a whole.

Is there a wish list of RPython enhancements somewhere that the PyPy team might 
be considering? 
Stuff that would benefit RPython users in general.

Sarvi



----- Original Message ----
> From: Armin Rigo <arigo at tunes.org>
> To: Saravanan Shanmugham <sarvi at yahoo.com>
> Cc: Paolo Giarrusso <p.giarrusso at gmail.com>; pypy-dev at codespeak.net; Stefan 
>Behnel <stefan_ml at behnel.de>
> Sent: Sat, September 4, 2010 2:03:24 AM
> Subject: Re: [pypy-dev] Question on the future of RPython
> 
> Hi,
> 
> Can we please close this thread?  The basic answer you will get  from
> anybody that actually worked at least a bit with PyPy is that all  your
> discussions are moving air around and nothing else.
> 
> There is no  one working with PyPy that is interested in using RPython
> for the purpose of  compiling some RPython programs to C code
> statically (except  interpreters).  If anyone is really interested in
> this topic he can  (again) give it a try.  He would get some help from
> us, i.e. the rest of  the PyPy team, but it would be a fork for now.  I
> say "again" because  there are some previous attempts at doing that,
> which all failed.  As  long as no such project exists and is successful
> -- and I have some doubts  about it -- I will not believe in the nice
> (and, to me, completely bogus)  claims made on this thread, like "let's
> bring RPython and Shedskin  together".
> 
> 
> A bientot,
> 
> Armin.
> 


      


From hakan at debian.org  Tue Sep  7 10:51:00 2010
From: hakan at debian.org (Hakan Ardo)
Date: Tue, 7 Sep 2010 10:51:00 +0200
Subject: [pypy-dev] jit-bounds branch (was: Loop invaraints)
In-Reply-To: <AANLkTinnKQWZkzVQaGK+vqJxGOPupTRWVhte=9sh3fdA@mail.gmail.com>
References: <AANLkTi=kXbXMXJh63B07ZJFajA8uQGv2y6NjsTkbUO88@mail.gmail.com>
	<4C7A3737.50902@gmx.de>
	<AANLkTinwz=71akbSFGpDpOg5Vfq3SnE=dokfgs_i5f+4@mail.gmail.com>
	<4C7A4C99.2050803@gmx.de>
	<AANLkTiknjSwX7vY69hjcmZ1OLZq03QYAvxx41DQZA_+f@mail.gmail.com>
	<AANLkTi=EOAmdiV0u5fdYFnL=xcBrLsCHtHAY=KqT1ees@mail.gmail.com>
	<AANLkTinnKQWZkzVQaGK+vqJxGOPupTRWVhte=9sh3fdA@mail.gmail.com>
Message-ID: <AANLkTik5F3KoH9y4+CVQFyLxU3Bv5Lg-Pr4LcGo05bfm@mail.gmail.com>

Hi,
there is now a package-version of optimizeopt in the jit-bounds
branch. In optimizeopt/__init__.py a chain of optimizers is created:

??? optimizations = [OptIntBounds(),
???????????????????? OptRewrite(),
???????????????????? OptVirtualize(),
???????????????????? OptHeap(),
??????????????????? ]

The opperations are passed from one optimizer to the next, which means
we keep the single loop over the iterations. Each optimazation is
located in it's own file, and it should be straight forward to add
more optimization and even make them optional using runtime arguments
if that is of interest.

I believe this branch is ready to be merged now.

On Thu, Sep 2, 2010 at 10:22 AM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>
> On Tue, Aug 31, 2010 at 09:25, Hakan Ardo <hakan at debian.org> wrote:
> > Ok, so we split it up into a set of Optimization classes in separate
> > files. Each containing a subset of the optimize_... methods. Then we
> > have the propagate_forward method iterate over the instructions
> > passing them to one Optimization after the other? That way we keep the
> > single iteration over the instructions. Would it be preferable to
> > separate them even more and have each Optimization contain it's own
> > loop over the instructions?
>
> But won't this affect performance? Which is very important in a JIT compiler.
> When compiling traces bigger than a cacheline, it might even affect
> locality, i.e. be an important performance problem.
> Unless your RPython compiler can join the loops. If they are just
> loops, it could. If they are tree visits, it likely can't; it's done
> by the Haskell compiler (google Haskell, stream fusion, shortcut
> deforestation, I guess), but the techniques are unlikely to generalize
> to languages with side effects; it's also done/doable in some
> Domain-Specific Languages for tree visitors.
>
> > On Sun, Aug 29, 2010 at 10:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >> On Sun, Aug 29, 2010 at 2:03 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> >>> On 08/29/2010 01:49 PM, Hakan Ardo wrote:
> >>>>> P.S.: A bit unrelated, but a comment on the jit-bounds branch: I think
> >>>>> it would be good if the bounds-related optimizations could move out of
> >>>>> optimizeopt.py to their own file, because otherwise optimizeopt.py is
> >>>>> getting really unwieldy. Does that make sense?
> >>>>
> >>>> Well, class IntBound and the propagate_bounds_ methods could probably
> >>>> be moved elsewhere, but a lot of the work is done in optimize_...
> >>>> methods, which I'm not so sure it would make sens to split up.
> >>>
> >>> I guess then the things that can be sanely moved should move. The file
> >>> is nearly 2000 lines, which is way too big. I guess also the heap
> >>> optimizations could go to their own file.
> >>>
> >>> Carl Friedrich
> >>
> >> How about a couple of files (preferably small) each containing a
> >> contained optimization if possible? (maybe a package?)
> >>
> >
> >
> >
> > --
> > H?kan Ard?
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
>
>
>
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/



--
H?kan Ard?


From arigo at tunes.org  Tue Sep  7 10:57:15 2010
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 7 Sep 2010 10:57:15 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <792921.22517.qm@web53703.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
Message-ID: <AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>

Hi,

On Mon, Sep 6, 2010 at 8:27 PM, Saravanan Shanmugham
> Is there a wish list of RPython enhancements somewhere that the
> PyPy team might be considering?
> Stuff that would benefit RPython users in general.

I feel like I am repeating myself so that's my last mail to this
thread.  There are no enhancements we are considering to benefit other
RPython users because *there* *are* *no* *other* *RPython* *users.*
There is only us and RPython suits us just fine for the purpose for
which it was designed.

Again, feel free to make a fork or a branch of PyPy and try to develop
a version of RPython that is more suited to writing general programs
in.  I don't know if there is a wish list of what is missing, but
certainly I haven't given it much thoughts myself.  Personally, I
think that writing RPython programs is kind of fun, but in a perverse
way -- if I could just write plain Python that was as fast or mostly
as fast, it would be perfect.


A bient?t,

Armin.


From stefan_ml at behnel.de  Tue Sep  7 11:07:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Sep 2010 11:07:42 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<201009021140.53415.jacob@openend.se>	<933850.84117.qm@web53705.mail.re2.yahoo.com>	<201009022254.07665.jacob@openend.se>	<810929.87945.qm@web53702.mail.re2.yahoo.com>	<i5qf7p$g4h$2@dough.gmane.org>	<609691.54456.qm@web53705.mail.re2.yahoo.com>	<i5rcl9$ip2$1@dough.gmane.org>	<294101.49986.qm@web53704.mail.re2.yahoo.com>	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>	<860977.83657.qm@web53705.mail.re2.yahoo.com>	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
Message-ID: <i64vcv$ckn$1@dough.gmane.org>

Armin Rigo, 07.09.2010 10:57:
> On Mon, Sep 6, 2010 at 8:27 PM, Saravanan Shanmugham
>> Is there a wish list of RPython enhancements somewhere that the
>> PyPy team might be considering?
>> Stuff that would benefit RPython users in general.
>
> Again, feel free to make a fork or a branch of PyPy and try to develop
> a version of RPython that is more suited to writing general programs
> in.

In that case, I suggest working on Shedskin or Cython instead.

Stefan



From angelflow at yahoo.com  Thu Sep  9 20:04:11 2010
From: angelflow at yahoo.com (Andy)
Date: Thu, 9 Sep 2010 11:04:11 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
Message-ID: <599423.93719.qm@web111315.mail.gq1.yahoo.com>

Hi,

I'm interested in using PyPy JIT with async greenlet-based frameworks such as gevent or meinheld.

Is there any plan to make PyPy JIT work with greenlet and C extensions? If so when may that be available?

Thanks.
Andy


      


From angelflow at yahoo.com  Thu Sep  9 20:06:20 2010
From: angelflow at yahoo.com (Andy)
Date: Thu, 9 Sep 2010 11:06:20 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT and Django
Message-ID: <215409.51875.qm@web111306.mail.gq1.yahoo.com>

I'd like to run Django on PyPy JIT. 

Could you give me some instructions on how to do that? I couldn't really find any documentation in that area.

Thanks.

Andy


      


From amauryfa at gmail.com  Thu Sep  9 20:18:22 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 9 Sep 2010 20:18:22 +0200
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <215409.51875.qm@web111306.mail.gq1.yahoo.com>
References: <215409.51875.qm@web111306.mail.gq1.yahoo.com>
Message-ID: <AANLkTimu9yibhq5BEFrX7fdFa5VYpJxfpXhyBzmXoz2Q@mail.gmail.com>

Hi,

2010/9/9 Andy <angelflow at yahoo.com>:
> I'd like to run Django on PyPy JIT.
>
> Could you give me some instructions on how to do that? I couldn't really find any documentation in that area.

I suggest to start with the obvious:
- Install PyPy
- download Django, unpack the archive
- in the Django directory, run "pypy setup.py install"
And tell us how it behaves!

-- 
Amaury Forgeot d'Arc


From angelflow at yahoo.com  Thu Sep  9 20:32:55 2010
From: angelflow at yahoo.com (Andy)
Date: Thu, 9 Sep 2010 11:32:55 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <AANLkTimu9yibhq5BEFrX7fdFa5VYpJxfpXhyBzmXoz2Q@mail.gmail.com>
Message-ID: <515725.20046.qm@web111303.mail.gq1.yahoo.com>


--- On Thu, 9/9/10, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:

> I suggest to start with the obvious:
> - Install PyPy
> - download Django, unpack the archive
> - in the Django directory, run "pypy setup.py install"
> And tell us how it behaves!
> 

Would this replace my existing Python interpretor with PyPy? I want to keep my existing Python.

I'd like to have the option to choose either the standard CPython or PyPy. I could set up 2 vrtualenvs I suppose - 1 for CPython and 1 for PyPy. Does PyPy work with virtualenv?


      


From fuzzyman at voidspace.org.uk  Thu Sep  9 20:35:14 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 9 Sep 2010 19:35:14 +0100
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <515725.20046.qm@web111303.mail.gq1.yahoo.com>
References: <AANLkTimu9yibhq5BEFrX7fdFa5VYpJxfpXhyBzmXoz2Q@mail.gmail.com>
	<515725.20046.qm@web111303.mail.gq1.yahoo.com>
Message-ID: <AANLkTimc1Ah13UDRnSZj-8GdnmRE9V2ewi4so31LzZqD@mail.gmail.com>

On 9 September 2010 19:32, Andy <angelflow at yahoo.com> wrote:

>
> --- On Thu, 9/9/10, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
>
> > I suggest to start with the obvious:
> > - Install PyPy
> > - download Django, unpack the archive
> > - in the Django directory, run "pypy setup.py install"
> > And tell us how it behaves!
> >
>
> Would this replace my existing Python interpretor with PyPy? I want to keep
> my existing Python.
>
>
Nope, the pypy interpreter is called pypy not python.


> I'd like to have the option to choose either the standard CPython or PyPy.
> I could set up 2 vrtualenvs I suppose - 1 for CPython and 1 for PyPy. Does
> PyPy work with virtualenv?
>
>
There is definitely a version of virtualenv that works with pypy. It may
even be included in pypy.

All the best,

Michael



>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
http://www.voidspace.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100909/ceca28bf/attachment.html>

From anto.cuni at gmail.com  Thu Sep  9 20:42:10 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 09 Sep 2010 20:42:10 +0200
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <515725.20046.qm@web111303.mail.gq1.yahoo.com>
References: <515725.20046.qm@web111303.mail.gq1.yahoo.com>
Message-ID: <4C892A82.5080804@gmail.com>

On 09/09/10 20:32, Andy wrote:
> Would this replace my existing Python interpretor with PyPy? I want to keep my existing Python.

well, no: if you run pypy setup.py install on whatever package, what happens
is that you install this package in pypy's site-package directory instead of
cpython's one.

> I'd like to have the option to choose either the standard CPython or PyPy. I could set up 2 vrtualenvs I suppose - 1 for CPython and 1 for PyPy. Does PyPy work with virtualenv?

yes, pypy supports virtualenv but:

1) you need a pypy newer than pypy 1.3: you can build one by yourself from
svn, or download one of our nightly builds:
http://buildbot.pypy.org/nightly/trunk/

note that you probably want the one with -jit and -linux in it, which is a 32
bit executable. There is no pypy-jit for linux 64 bit yet (but it might be
there soon).

2) you need a recent version of virtualenv, as pypy support has been added
only recently (http://bitbucket.org/ianb/virtualenv/changeset/a03cb042dd81).
AFAIK, no released version supports it, so you need to install/run it from the
mercurial repository.

ciao,
Anto


From angelflow at yahoo.com  Thu Sep  9 22:29:53 2010
From: angelflow at yahoo.com (Andy)
Date: Thu, 9 Sep 2010 13:29:53 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <4C892A82.5080804@gmail.com>
Message-ID: <392170.19756.qm@web111313.mail.gq1.yahoo.com>

Thanks Antonio.

So right now there's no way to run PyPy JIT on Linux 64 bit? 

--- On Thu, 9/9/10, Antonio Cuni <anto.cuni at gmail.com> wrote:

> From: Antonio Cuni <anto.cuni at gmail.com>
> Subject: Re: [pypy-dev] PyPy JIT and Django
> To: "Andy" <angelflow at yahoo.com>
> Cc: "Amaury Forgeot d'Arc" <amauryfa at gmail.com>, pypy-dev at codespeak.net
> Date: Thursday, September 9, 2010, 2:42 PM
> On 09/09/10 20:32, Andy wrote:
> > Would this replace my existing Python interpretor with
> PyPy? I want to keep my existing Python.
> 
> well, no: if you run pypy setup.py install on whatever
> package, what happens
> is that you install this package in pypy's site-package
> directory instead of
> cpython's one.
> 
> > I'd like to have the option to choose either the
> standard CPython or PyPy. I could set up 2 vrtualenvs I
> suppose - 1 for CPython and 1 for PyPy. Does PyPy work with
> virtualenv?
> 
> yes, pypy supports virtualenv but:
> 
> 1) you need a pypy newer than pypy 1.3: you can build one
> by yourself from
> svn, or download one of our nightly builds:
> http://buildbot.pypy.org/nightly/trunk/
> 
> note that you probably want the one with -jit and -linux in
> it, which is a 32
> bit executable. There is no pypy-jit for linux 64 bit yet
> (but it might be
> there soon).
> 
> 2) you need a recent version of virtualenv, as pypy support
> has been added
> only recently (http://bitbucket.org/ianb/virtualenv/changeset/a03cb042dd81).
> AFAIK, no released version supports it, so you need to
> install/run it from the
> mercurial repository.
> 
> ciao,
> Anto
> 


      


From amauryfa at gmail.com  Thu Sep  9 22:37:15 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 9 Sep 2010 22:37:15 +0200
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <392170.19756.qm@web111313.mail.gq1.yahoo.com>
References: <4C892A82.5080804@gmail.com>
	<392170.19756.qm@web111313.mail.gq1.yahoo.com>
Message-ID: <AANLkTimNPnbtRWrOEUUmcRmcQCW8B2dN=r7v9Byo=N=2@mail.gmail.com>

2010/9/9 Andy <angelflow at yahoo.com>:
> Thanks Antonio.
>
> So right now there's no way to run PyPy JIT on Linux 64 bit?

Yes there is!
Armin merged the asmgcc-64 branch yesterday.
You have to use trunk version of PyPy, and build it yourself.

-- 
Amaury Forgeot d'Arc


From fijall at gmail.com  Thu Sep  9 22:55:55 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 9 Sep 2010 22:55:55 +0200
Subject: [pypy-dev] PyPy JIT and Django
In-Reply-To: <AANLkTimNPnbtRWrOEUUmcRmcQCW8B2dN=r7v9Byo=N=2@mail.gmail.com>
References: <4C892A82.5080804@gmail.com>
	<392170.19756.qm@web111313.mail.gq1.yahoo.com>
	<AANLkTimNPnbtRWrOEUUmcRmcQCW8B2dN=r7v9Byo=N=2@mail.gmail.com>
Message-ID: <AANLkTinx04pU2_hXiMS7y4R+=orURMoEweaK2wL8X4ZN@mail.gmail.com>

On Thu, Sep 9, 2010 at 10:37 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2010/9/9 Andy <angelflow at yahoo.com>:
>> Thanks Antonio.
>>
>> So right now there's no way to run PyPy JIT on Linux 64 bit?
>
> Yes there is!
> Armin merged the asmgcc-64 branch yesterday.
> You have to use trunk version of PyPy, and build it yourself.

Although I would mark this support as "experimental" for now, it works :-)

>
> --
> Amaury Forgeot d'Arc
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Sat Sep 11 16:57:41 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Sep 2010 16:57:41 +0200
Subject: [pypy-dev] External RPython mailing list
Message-ID: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>

Hi,

To anyone interested, Sarvi(?) created an RPython mailing list (Thanks
Bea for spotting this):

    http://pyppet.blogspot.com/2010/09/rpython-mailing-list.html

The following paragraph should have been posted as a comment to that
blog post, but it doesn't record my post no matter how much I try, so
I'll put it here:

"""
Ah, sorry about the money issue.  I didn't realize that you already
sent it to us; I misunderstood that you would not send it at all after
we told you that we don't have resources and motivation to make
RPython more user-friendly (even with $200).  Now I suppose that we
can arrange for you to get the money back if you like, or else thank
you properly for it if it's ours to keep anyway :-)
"""

About the non-money issue, I end up looking like the bad guy.  I
suppose I should not have tried to say and repeat "no" so many times
in the previous thread in increasingly bad tones; now Sarvi points
only to my most negative e-mail.


A bient?t,

Armin


From bea at changemaker.nu  Sun Sep 12 21:58:14 2010
From: bea at changemaker.nu (Bea During)
Date: Sun, 12 Sep 2010 21:58:14 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
Message-ID: <4C8D30D6.10802@changemaker.nu>

Hi there

Armin Rigo skrev:
> Hi,
>
> To anyone interested, Sarvi(?) created an RPython mailing list (Thanks
> Bea for spotting this):
>
>     http://pyppet.blogspot.com/2010/09/rpython-mailing-list.html
>
> The following paragraph should have been posted as a comment to that
> blog post, but it doesn't record my post no matter how much I try, so
> I'll put it here:
>
> """
> Ah, sorry about the money issue.  I didn't realize that you already
> sent it to us; I misunderstood that you would not send it at all after
> we told you that we don't have resources and motivation to make
> RPython more user-friendly (even with $200).  Now I suppose that we
> can arrange for you to get the money back if you like, or else thank
> you properly for it if it's ours to keep anyway :-)
> """
>
> About the non-money issue, I end up looking like the bad guy.  I
> suppose I should not have tried to say and repeat "no" so many times
> in the previous thread in increasingly bad tones; now Sarvi points
> only to my most negative e-mail.
>
>
> A bient?t,
>
> Armin
>   

Thanks for posting this Armin. Let?s focus on the future then.

Maybe we should be clear in our documentation somewhere on
where we stand regarding RPython and maybe give some friendly
advice on how to get started with experimenting (because that is
what it means trying out RPython for other purposes than what Pypy
uses it for). And if more questions like these pop up we can refer the
inquiries there?

That way it?s the core dev team who expresses the views, in one voice
so to say.

Just a suggestion.

Cheers

Bea
(

> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
>   



From angelflow at yahoo.com  Mon Sep 13 07:21:03 2010
From: angelflow at yahoo.com (Andy)
Date: Sun, 12 Sep 2010 22:21:03 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <AANLkTi=hBYyhr-h+U8u44-5wMk6PX5rxnd15t_TQ9s-8@mail.gmail.com>
Message-ID: <173287.30107.qm@web111312.mail.gq1.yahoo.com>



--- On Fri, 9/10/10, Armin Rigo <arigo at tunes.org> wrote:

> On Thu, Sep 9, 2010 at 8:04 PM, Andy <angelflow at yahoo.com>
> wrote:
> > Is there any plan to make PyPy JIT work with greenlet
> and C extensions?
> > If so when may that be available?
> 
> As far as I can tell it works already.? Just download
> (http://buildbot.pypy.org/nightly/trunk/)
> or build yourself a
> stackless version of PyPy; it should contain "cpyext",
> which means
> that it should support C extension modules.
> 
> Note that callbacks from C to Python do not play nicely
> with stackless
> features: if you try e.g. to switch to another greenlet
> while you are
> in such a callback, you get a fatal error.? Hopefully
> that's a rare
> case, but still, it should at least be improved to give you
> a regular
> catchable Python exception.


Thanks Armin.

Does that mean PyPy will not work with greenlet/gevent/etc?

Is there any plan to make PyPy support greenlet? Or is there some fundamental obstacle that would prevent it from doing so?

Andy


      


From arigo at tunes.org  Mon Sep 13 09:57:28 2010
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Sep 2010 09:57:28 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <4C8D30D6.10802@changemaker.nu>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
Message-ID: <AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>

Hi,

On Sun, Sep 12, 2010 at 9:58 PM, Bea During <bea at changemaker.nu> wrote:
> Maybe we should be clear in our documentation somewhere on
> where we stand regarding RPython

What about renaming it first?  There is at least one other project
that uses the name RPython.  What about something like InterpPy or
InterpPython to make it clear that it's supposed to be used to write
interpreters?  It doesn't sound terrific but I don't really care --
so, comments welcome, but please no infinite discussion on the pros
and cons of various names.


A bient?t,

Armin.


From fijall at gmail.com  Mon Sep 13 10:03:01 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 13 Sep 2010 10:03:01 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
Message-ID: <AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>

On Mon, Sep 13, 2010 at 9:57 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Sun, Sep 12, 2010 at 9:58 PM, Bea During <bea at changemaker.nu> wrote:
>> Maybe we should be clear in our documentation somewhere on
>> where we stand regarding RPython
>
> What about renaming it first? ?There is at least one other project
> that uses the name RPython. ?What about something like InterpPy or
> InterpPython to make it clear that it's supposed to be used to write
> interpreters? ?It doesn't sound terrific but I don't really care --
> so, comments welcome, but please no infinite discussion on the pros
> and cons of various names.
>

While we're at it, how about splitting the translation toolchain from
pypy interpreter? I don't mean on technical merits, it can still be
the same or mostly the same source codebase, but more on the
conceptual level, to have 2 different websites names etc.


From arigo at tunes.org  Mon Sep 13 10:10:40 2010
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Sep 2010 10:10:40 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <173287.30107.qm@web111312.mail.gq1.yahoo.com>
References: <AANLkTi=hBYyhr-h+U8u44-5wMk6PX5rxnd15t_TQ9s-8@mail.gmail.com>
	<173287.30107.qm@web111312.mail.gq1.yahoo.com>
Message-ID: <AANLkTinXwXS0jZ4zPNUNwUa7bAjdBu0MtMNKjExGuGfD@mail.gmail.com>

Hi Andy,

On Mon, Sep 13, 2010 at 7:21 AM, Andy <angelflow at yahoo.com> wrote:
> Does that mean PyPy will not work with greenlet/gevent/etc?

Sorry if I wasn't clear.  PyPy contains greenlet support (since
2005-6).  It's part of the same package that we call "pypy-stackless".


Armin


From fijall at gmail.com  Mon Sep 13 10:11:42 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 13 Sep 2010 10:11:42 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
Message-ID: <AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>

On Mon, Sep 13, 2010 at 10:08 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Maciej,
>
> On Mon, Sep 13, 2010 at 10:03 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> While we're at it, how about splitting the translation toolchain from
>> pypy interpreter? I don't mean on technical merits, it can still be
>> the same or mostly the same source codebase, but more on the
>> conceptual level, to have 2 different websites names etc.
>
> I don't care too much right now. ?My motivation was to make RPython
> *less* visible, not create a second website for the translation
> toolchain (which would make RPython more visible).
>

I don't think it's hideable. What we can do instead is to leave some
kind of description why it is like it is and what it is. Trying to
hide it means to some people that we have an awesome tool that we
don't want to share. Instead it's worth explaining why we don't share
this (because it's eg hard to use)

>
> A bient?t,
>
> Armin.
>


From arigo at tunes.org  Mon Sep 13 10:14:01 2010
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Sep 2010 10:14:01 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
Message-ID: <AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>

Hi,

On Mon, Sep 13, 2010 at 10:11 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I don't think it's hideable.

Sorry, I wasn't clear.  I'm not really trying to hide it.  But I'm
also not really trying to push it forward (which seems to be what
creating a website for it would do).


Armin


From arigo at tunes.org  Mon Sep 13 10:08:51 2010
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Sep 2010 10:08:51 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
Message-ID: <AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>

Hi Maciej,

On Mon, Sep 13, 2010 at 10:03 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> While we're at it, how about splitting the translation toolchain from
> pypy interpreter? I don't mean on technical merits, it can still be
> the same or mostly the same source codebase, but more on the
> conceptual level, to have 2 different websites names etc.

I don't care too much right now.  My motivation was to make RPython
*less* visible, not create a second website for the translation
toolchain (which would make RPython more visible).


A bient?t,

Armin.


From anto.cuni at gmail.com  Mon Sep 13 10:20:07 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 13 Sep 2010 10:20:07 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <AANLkTinXwXS0jZ4zPNUNwUa7bAjdBu0MtMNKjExGuGfD@mail.gmail.com>
References: <AANLkTi=hBYyhr-h+U8u44-5wMk6PX5rxnd15t_TQ9s-8@mail.gmail.com>	<173287.30107.qm@web111312.mail.gq1.yahoo.com>
	<AANLkTinXwXS0jZ4zPNUNwUa7bAjdBu0MtMNKjExGuGfD@mail.gmail.com>
Message-ID: <4C8DDEB7.3080006@gmail.com>

On 13/09/10 10:10, Armin Rigo wrote:
> Hi Andy,
> 
> On Mon, Sep 13, 2010 at 7:21 AM, Andy <angelflow at yahoo.com> wrote:
>> Does that mean PyPy will not work with greenlet/gevent/etc?
> 
> Sorry if I wasn't clear.  PyPy contains greenlet support (since
> 2005-6).  It's part of the same package that we call "pypy-stackless".

yes, but it must also be said that at the moment, pypy-stackless and pypy-jit
do not work together.

ciao,
Anto


From anto.cuni at gmail.com  Mon Sep 13 10:22:54 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 13 Sep 2010 10:22:54 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>	<4C8D30D6.10802@changemaker.nu>	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
Message-ID: <4C8DDF5E.8060303@gmail.com>

On 13/09/10 10:14, Armin Rigo wrote:
> Hi,
> 
> On Mon, Sep 13, 2010 at 10:11 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I don't think it's hideable.
> 
> Sorry, I wasn't clear.  I'm not really trying to hide it.  But I'm
> also not really trying to push it forward (which seems to be what
> creating a website for it would do).

well, I don't think that hiding it or pushing it backward is a good idea.  In
theory, we would like if other people start to use rpython to write interpreters.

What we don't like is to use rpython as a general purpose language, but that's
a slightly different issue, IMHO.

ciao,
Anto


From fijall at gmail.com  Mon Sep 13 10:27:27 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 13 Sep 2010 10:27:27 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <4C8DDF5E.8060303@gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
	<4C8DDF5E.8060303@gmail.com>
Message-ID: <AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>

On Mon, Sep 13, 2010 at 10:22 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 13/09/10 10:14, Armin Rigo wrote:
>> Hi,
>>
>> On Mon, Sep 13, 2010 at 10:11 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> I don't think it's hideable.
>>
>> Sorry, I wasn't clear. ?I'm not really trying to hide it. ?But I'm
>> also not really trying to push it forward (which seems to be what
>> creating a website for it would do).
>
> well, I don't think that hiding it or pushing it backward is a good idea. ?In
> theory, we would like if other people start to use rpython to write interpreters.
>
> What we don't like is to use rpython as a general purpose language, but that's
> a slightly different issue, IMHO.

Is it really about interpreters? (what's interpreter-specific after
all in RPython) or is it just that it's hard to use and does not
integrate with CPython well?

>
> ciao,
> Anto
>


From arigo at tunes.org  Mon Sep 13 10:40:02 2010
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Sep 2010 10:40:02 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <4C8DDEB7.3080006@gmail.com>
References: <AANLkTi=hBYyhr-h+U8u44-5wMk6PX5rxnd15t_TQ9s-8@mail.gmail.com>
	<173287.30107.qm@web111312.mail.gq1.yahoo.com>
	<AANLkTinXwXS0jZ4zPNUNwUa7bAjdBu0MtMNKjExGuGfD@mail.gmail.com>
	<4C8DDEB7.3080006@gmail.com>
Message-ID: <AANLkTikLBQW=DXxnRdnYq-gW5nr+hjce_OSdG8WGUW-g@mail.gmail.com>

Hi,

On Mon, Sep 13, 2010 at 10:20 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> yes, but it must also be said that at the moment, pypy-stackless and pypy-jit
> do not work together.

Oups, sorry. I missed the word "JIT" in the original message of this
thread :-(  Sorry for the confusion.

To answer the original question: it would be nice if someone would
show up and help contribute JIT support for Stackless builds of PyPy.
I think that the status is that no-one of us is ready to invest a lot
of time there, but we can definitely give pointers and get people
started and follow their progress.


A bient?t,

Armin.


From anto.cuni at gmail.com  Mon Sep 13 10:50:34 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 13 Sep 2010 10:50:34 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
	<4C8DDF5E.8060303@gmail.com>
	<AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>
Message-ID: <4C8DE5DA.9050909@gmail.com>

On 13/09/10 10:27, Maciej Fijalkowski wrote:

> Is it really about interpreters? (what's interpreter-specific after
> all in RPython) or is it just that it's hard to use and does not
> integrate with CPython well?

my point if that it's definitely good enough for writing interpreters. For the
rest, it's a bit unknown (in the sense that nobody has ever tried), and we
don't care about knowing :-)


From holger at merlinux.eu  Mon Sep 13 11:24:16 2010
From: holger at merlinux.eu (holger krekel)
Date: Mon, 13 Sep 2010 11:24:16 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <4C8DE5DA.9050909@gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
	<4C8DDF5E.8060303@gmail.com>
	<AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>
	<4C8DE5DA.9050909@gmail.com>
Message-ID: <20100913092416.GH32478@trillke.net>

On Mon, Sep 13, 2010 at 10:50 +0200, Antonio Cuni wrote:
> On 13/09/10 10:27, Maciej Fijalkowski wrote:
> 
> > Is it really about interpreters? (what's interpreter-specific after
> > all in RPython) or is it just that it's hard to use and does not
> > integrate with CPython well?
> 
> my point if that it's definitely good enough for writing interpreters. For the
> rest, it's a bit unknown (in the sense that nobody has ever tried), and we
> don't care about knowing :-)

People have written apps and libs in RPython at several points in its
history.  And while i find it perfectly acceptable and fine for PyPy core devs 
to not want to care for usage of RPython for non-interpreter purposes i 
am a bit tired of this ever ongoing competition of expressing dis-interest
and uttering discouraging statements. 

best,
holger


From benjamin at python.org  Mon Sep 13 17:42:07 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 13 Sep 2010 10:42:07 -0500
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
Message-ID: <AANLkTinqMYs2ikvzG1m3ogGR805b=_=qXeaoLUxdyi7q@mail.gmail.com>

2010/9/13 Maciej Fijalkowski <fijall at gmail.com>:
> On Mon, Sep 13, 2010 at 9:57 AM, Armin Rigo <arigo at tunes.org> wrote:
>> Hi,
>>
>> On Sun, Sep 12, 2010 at 9:58 PM, Bea During <bea at changemaker.nu> wrote:
>>> Maybe we should be clear in our documentation somewhere on
>>> where we stand regarding RPython
>>
>> What about renaming it first? ?There is at least one other project
>> that uses the name RPython. ?What about something like InterpPy or
>> InterpPython to make it clear that it's supposed to be used to write
>> interpreters? ?It doesn't sound terrific but I don't really care --
>> so, comments welcome, but please no infinite discussion on the pros
>> and cons of various names.
>>
>
> While we're at it, how about splitting the translation toolchain from
> pypy interpreter? I don't mean on technical merits, it can still be
> the same or mostly the same source codebase, but more on the
> conceptual level, to have 2 different websites names etc.

-0. We don't need more websites/trees to maintain. Anyway, it's not
clear to me where the split would be, since the translator and the
python interpreter are very interdependent.

-- 
Regards,
Benjamin


From angelflow at yahoo.com  Mon Sep 13 18:18:15 2010
From: angelflow at yahoo.com (Andy)
Date: Mon, 13 Sep 2010 09:18:15 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <AANLkTikLBQW=DXxnRdnYq-gW5nr+hjce_OSdG8WGUW-g@mail.gmail.com>
Message-ID: <122423.28497.qm@web111302.mail.gq1.yahoo.com>


--- On Mon, 9/13/10, Armin Rigo <arigo at tunes.org> wrote:

> > yes, but it must also be said that at the moment,
> pypy-stackless and pypy-jit
> > do not work together.
> 
> Oups, sorry. I missed the word "JIT" in the original
> message of this
> thread :-(? Sorry for the confusion.
> 
> To answer the original question: it would be nice if
> someone would
> show up and help contribute JIT support for Stackless
> builds of PyPy.
> I think that the status is that no-one of us is ready to
> invest a lot

OK let me make sure I got it right:

PyPy-JIT does not work with pypy-stackless. I'm mostly interested in greenlet, not stackless python. Is pypy-stackless required for greenlet support?

Looks like you're saying PyPy-JIT doesn't support greenlet and there's no plan to do so, correct?






      


From cfbolz at gmx.de  Mon Sep 13 19:15:38 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 13 Sep 2010 19:15:38 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <122423.28497.qm@web111302.mail.gq1.yahoo.com>
References: <122423.28497.qm@web111302.mail.gq1.yahoo.com>
Message-ID: <4C8E5C3A.7080302@gmx.de>

Hi Andy,

On 09/13/2010 06:18 PM, Andy wrote:
> OK let me make sure I got it right:
>
> PyPy-JIT does not work with pypy-stackless. I'm mostly interested in
> greenlet, not stackless python. Is pypy-stackless required for
> greenlet support?
>
> Looks like you're saying PyPy-JIT doesn't support greenlet

That's correct.

> there's no plan to do so, correct?

I think it's more a case of "no manpower". If somebody is interested in 
implementing it and shows up in the channel, we can give help. We have 
currently no time to do it ourselves.

Cheers,

Carl Friedrich



From sarvi at yahoo.com  Tue Sep 14 01:10:59 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Mon, 13 Sep 2010 16:10:59 -0700 (PDT)
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
Message-ID: <157589.10631.qm@web53701.mail.re2.yahoo.com>

No I didn't create the mailing list. Possibly Hart's Antler who did.

And the money to PyPy had no strings attached.
I really hope PyPy replaces CPython as the standard python sooner, rather than 
later.

True. 
That yall are not interested in standardizing an implicitly static subset of 
Python that can be used to create compiled executables or python extension 
libraries.

But if PyPy gains momentum, I am pretty sure this idea will gain momentum 
eventually.

Keep the good work.

Sarvi




----- Original Message ----
> From: Armin Rigo <arigo at tunes.org>
> To: pypy-dev at codespeak.net
> Sent: Sat, September 11, 2010 7:57:41 AM
> Subject: [pypy-dev] External RPython mailing list
> 
> Hi,
> 
> To anyone interested, Sarvi(?) created an RPython mailing list  (Thanks
> Bea for spotting this):
> 
>      http://pyppet.blogspot.com/2010/09/rpython-mailing-list.html
> 
> The  following paragraph should have been posted as a comment to that
> blog post,  but it doesn't record my post no matter how much I try, so
> I'll put it  here:
> 
> """
> Ah, sorry about the money issue.  I didn't realize that  you already
> sent it to us; I misunderstood that you would not send it at all  after
> we told you that we don't have resources and motivation to  make
> RPython more user-friendly (even with $200).  Now I suppose that  we
> can arrange for you to get the money back if you like, or else  thank
> you properly for it if it's ours to keep anyway :-)
> """
> 
> About  the non-money issue, I end up looking like the bad guy.  I
> suppose I  should not have tried to say and repeat "no" so many times
> in the previous  thread in increasingly bad tones; now Sarvi points
> only to my most negative  e-mail.
> 
> 
> A  bient?t,
> 
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


      


From sarvi at yahoo.com  Tue Sep 14 01:36:07 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Mon, 13 Sep 2010 16:36:07 -0700 (PDT)
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <4C8DE5DA.9050909@gmail.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
	<4C8DDF5E.8060303@gmail.com>
	<AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>
	<4C8DE5DA.9050909@gmail.com>
Message-ID: <328327.91624.qm@web53705.mail.re2.yahoo.com>

I believe Hart's Antler has done quite bit of work in RPython.

Yeah, that said, more work to do there.

My goal with my RPython thread, is that I believe that there is an implicitly 
static subset of Python that can be compiled into standalone executables and 
DLLs without needing JIT or VMs.
Can serve 2 purposes.
   1. Make standalone executables just like C/C++ code.
   2. Write Python Extension modules that can be compiled into shared DLL 
modules for CPython and PyPy

Looking through the various threads on PyPy, Shedskin and Cython, I believe its 
just a matter of time.

Sarvi



  

----- Original Message ----
> From: Antonio Cuni <anto.cuni at gmail.com>
> To: Maciej Fijalkowski <fijall at gmail.com>
> Cc: pypy-dev at codespeak.net; Armin Rigo <arigo at tunes.org>
> Sent: Mon, September 13, 2010 1:50:34 AM
> Subject: Re: [pypy-dev] External RPython mailing list
> 
> On 13/09/10 10:27, Maciej Fijalkowski wrote:
> 
> > Is it really about  interpreters? (what's interpreter-specific after
> > all in RPython) or is  it just that it's hard to use and does not
> > integrate with CPython  well?
> 
> my point if that it's definitely good enough for writing  interpreters. For 
the
> rest, it's a bit unknown (in the sense that nobody has  ever tried), and we
> don't care about knowing  :-)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


      


From fijall at gmail.com  Tue Sep 14 11:39:30 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Sep 2010 11:39:30 +0200
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <328327.91624.qm@web53705.mail.re2.yahoo.com>
References: <AANLkTi=qu4o-_qms13ENUTuUiQD6W4wfR0ezdTmOMkvK@mail.gmail.com>
	<4C8D30D6.10802@changemaker.nu>
	<AANLkTi=zufrgfvHeheUE73J-iXX1QfU8m-YQEdL6mXZU@mail.gmail.com>
	<AANLkTimSDCtXCiNyeF7dyNXaY3kBh1A2ccW_EU53UQwM@mail.gmail.com>
	<AANLkTi=KczTMki5Dn+5cyQkTBpCj6N4Wje2U1_Kp6J-r@mail.gmail.com>
	<AANLkTik+jvdv6uSAtCNcjpfOpuFy6xX2pUPG8ix_cVop@mail.gmail.com>
	<AANLkTikKDRhdtwyzVHaJVtDt5vnnrBsgKj2h_f4sbHai@mail.gmail.com>
	<4C8DDF5E.8060303@gmail.com>
	<AANLkTik2cMCvfFz43m3R-+NrSZqEviu030kVuBDqhwXA@mail.gmail.com>
	<4C8DE5DA.9050909@gmail.com>
	<328327.91624.qm@web53705.mail.re2.yahoo.com>
Message-ID: <AANLkTinfDpeszDzfQeKAVgcnuLPY+kFckQX2ZOtgiKt_@mail.gmail.com>

Hi.

Speaking from a personal perspective here, I would help people write
standalone executables using RPython. This has been tried (even with
success) for small examples and works. However, for most places where
it was tried, it was an ill-chosen tool for that purpose (where slight
python optimizations or using JIT would work equally well) with
RPython having sometimes bizarre limitations that we're not willing to
work on.

The second part (writing Python extensions) I think is not a very good
target, but can be done. However, I don't want people telling me that
I should work on it. If some people want to implement this, they can
get my help.

I think this defines rough outline where I'm (personally) willing to
help or not to help people using RPython.

On Tue, Sep 14, 2010 at 1:36 AM, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> I believe Hart's Antler has done quite bit of work in RPython.
>
> Yeah, that said, more work to do there.
>
> My goal with my RPython thread, is that I believe that there is an implicitly
> static subset of Python that can be compiled into standalone executables and
> DLLs without needing JIT or VMs.
> Can serve 2 purposes.
> ? 1. Make standalone executables just like C/C++ code.
> ? 2. Write Python Extension modules that can be compiled into shared DLL
> modules for CPython and PyPy
>
> Looking through the various threads on PyPy, Shedskin and Cython, I believe its
> just a matter of time.
>
> Sarvi
>
>
>
>
>
> ----- Original Message ----
>> From: Antonio Cuni <anto.cuni at gmail.com>
>> To: Maciej Fijalkowski <fijall at gmail.com>
>> Cc: pypy-dev at codespeak.net; Armin Rigo <arigo at tunes.org>
>> Sent: Mon, September 13, 2010 1:50:34 AM
>> Subject: Re: [pypy-dev] External RPython mailing list
>>
>> On 13/09/10 10:27, Maciej Fijalkowski wrote:
>>
>> > Is it really about ?interpreters? (what's interpreter-specific after
>> > all in RPython) or is ?it just that it's hard to use and does not
>> > integrate with CPython ?well?
>>
>> my point if that it's definitely good enough for writing ?interpreters. For
> the
>> rest, it's a bit unknown (in the sense that nobody has ?ever tried), and we
>> don't care about knowing ?:-)
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>
>
>
>


From sarvi at yahoo.com  Tue Sep 14 21:32:27 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Tue, 14 Sep 2010 12:32:27 -0700 (PDT)
Subject: [pypy-dev] PyPy to generate C/C++ code
Message-ID: <553714.16055.qm@web53703.mail.re2.yahoo.com>

To be very clear this is not a question on PyPY RPython itself. :-))

But I had another thought and wanted to run it by PyPy team. 


As I understand it PyPy is foremost a language development framework. 
It is about implementing the python interpreter in RPython, plus 
additional hints to assist in JIT generation. 

If the Python language implementation in RPython has enough 
information to create a python interpreter and do JIT compilation. 
I am thinking it should have enough information to generate C/C++ code.

The kind that shedskin has under shedskin/lib/ 

Basically port the type inference engine from shedskin over to PyPy and use the 
bulk of Shedksin C++ code but use PyPy Language 
Framework to implement the Python Compiler that shedksin implements? 


In otherwords, can PyPy be the language framework in which Shedskin is 
implemented/ported onto?

Looking Shedskin and PyPy do yall have a rough feel for how difficult this would 
be.

Why the question?
I am planning to fund some prize money for an Under/Graduate school project back 
in India and am looking for ideas. 

This means we would able to motivate a team of 2-5 smart young engineers for 
about 6 months into doing something interesting for them but  beneficial for the 
python community. 


One area I am obviously looking at is compiling Python code. 


I was thinking the project could be to 
   1. take your C++ code under shedskin/lib as is 
   2. Have them implement/port the shedskin type inference engine onto the PyPy 
framework and create a PyPy backend that generates  the C++ code 
from the shedskin/lib 


What would yall think of such an idea. 
Estimates? Feasibility? 
Do you see any benefits to this work for Shedskin or PyPy or both? 

Sarvi 



      


From benjamin at python.org  Tue Sep 14 23:26:31 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 14 Sep 2010 16:26:31 -0500
Subject: [pypy-dev] PyPy to generate C/C++ code
In-Reply-To: <553714.16055.qm@web53703.mail.re2.yahoo.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
Message-ID: <AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>

2010/9/14 Saravanan Shanmugham <sarvi at yahoo.com>:
> To be very clear this is not a question on PyPY RPython itself. :-))
>
> But I had another thought and wanted to run it by PyPy team.
>
>
> As I understand it PyPy is foremost a language development framework.
> It is about implementing the python interpreter in RPython, plus
> additional hints to assist in JIT generation.
>
> If the Python language implementation in RPython has enough
> information to create a python interpreter and do JIT compilation.
> I am thinking it should have enough information to generate C/C++ code.

Creating a JIT compiler is completely different from statically
compiling code. In a JIT, you use runtime information to optimize the
code. You can't do anything about this in C.



-- 
Regards,
Benjamin


From sarvi at yahoo.com  Wed Sep 15 00:15:36 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Tue, 14 Sep 2010 15:15:36 -0700 (PDT)
Subject: [pypy-dev] PyPy to generate C/C++ code
In-Reply-To: <AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
	<AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>
Message-ID: <708334.39926.qm@web53705.mail.re2.yahoo.com>

I don't expect this python compiler to be for full python but just a Restricted 
statically typed subset of python as defined by Shedskin.

Yes. JIT annotation may not serve the purpose of generating a compiler.
Hence the porting of the type inference engine and may be use JIT notations if 
it can be . 

Sarvi



----- Original Message ----
> From: Benjamin Peterson <benjamin at python.org>
> To: Saravanan Shanmugham <sarvi at yahoo.com>
> Cc: pypy-dev at codespeak.net
> Sent: Tue, September 14, 2010 2:26:31 PM
> Subject: Re: [pypy-dev] PyPy to generate C/C++ code
> 
> 2010/9/14 Saravanan Shanmugham <sarvi at yahoo.com>:
> > To be very clear  this is not a question on PyPY RPython itself. :-))
> >
> > But I had  another thought and wanted to run it by PyPy team.
> >
> >
> > As I  understand it PyPy is foremost a language development framework.
> > It is  about implementing the python interpreter in RPython, plus
> > additional  hints to assist in JIT generation.
> >
> > If the Python language  implementation in RPython has enough
> > information to create a python  interpreter and do JIT compilation.
> > I am thinking it should have enough  information to generate C/C++ code.
> 
> Creating a JIT compiler is completely  different from statically
> compiling code. In a JIT, you use runtime  information to optimize the
> code. You can't do anything about this in  C.
> 
> 
> 
> -- 
> Regards,
> Benjamin
> 


      


From benjamin at python.org  Wed Sep 15 00:19:36 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 14 Sep 2010 17:19:36 -0500
Subject: [pypy-dev] Fwd:  PyPy to generate C/C++ code
In-Reply-To: <AANLkTi=vb11UM1VerzpYZoRERcxXBpTJPdL3QcB_+a18@mail.gmail.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
	<AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>
	<708334.39926.qm@web53705.mail.re2.yahoo.com>
	<AANLkTi=vb11UM1VerzpYZoRERcxXBpTJPdL3QcB_+a18@mail.gmail.com>
Message-ID: <AANLkTimJWAQNCKb7X5DwQ8N3hFL9MwuGVB5r4WfFHMOb@mail.gmail.com>

---------- Forwarded message ----------
From: Benjamin Peterson <benjamin at python.org>
Date: 2010/9/14
Subject: Re: [pypy-dev] PyPy to generate C/C++ code
To: Saravanan Shanmugham <sarvi at yahoo.com>


2010/9/14 Saravanan Shanmugham <sarvi at yahoo.com>:
> I don't expect this python compiler to be for full python but just a Restricted
> statically typed subset of python as defined by Shedskin.

So how is that any different than the existing RPython?



--
Regards,
Benjamin



-- 
Regards,
Benjamin


From sarvi at yahoo.com  Wed Sep 15 00:45:11 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Tue, 14 Sep 2010 15:45:11 -0700 (PDT)
Subject: [pypy-dev] Fwd:  PyPy to generate C/C++ code
In-Reply-To: <AANLkTimJWAQNCKb7X5DwQ8N3hFL9MwuGVB5r4WfFHMOb@mail.gmail.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
	<AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>
	<708334.39926.qm@web53705.mail.re2.yahoo.com>
	<AANLkTi=vb11UM1VerzpYZoRERcxXBpTJPdL3QcB_+a18@mail.gmail.com>
	<AANLkTimJWAQNCKb7X5DwQ8N3hFL9MwuGVB5r4WfFHMOb@mail.gmail.com>
Message-ID: <313654.95430.qm@web53701.mail.re2.yahoo.com>

This gives us an opportunity to define a more general purpose definition of 
RPython.

Based on previous threads, RPython as currently defined in PyPy is up for 
general use.
And the team seems to have no interest in exploring this RPython for general 
use.
And to be honest I can see some extent why. 
Keeps it simple and domain specific for the purpose of language definition which 
is PyPy's goals.

So my intent is to instead go for more general purpose RPython, by starting with 
Shedskin's definition of Restricted Python, which is pretty close to PyPy's 
RPython
Refer to Hart's comparison 
here. http://groups.google.com/group/shedskin-discuss/browse_thread/thread/a8b473f0b4b52217


Why port shedskin onto PyPy. 
   1. See if PyPy as a language definition framework can be used generate 
compilers and not just Interpreters.
   2. Leverage the C++ code under shedskin/lib to quickly get to that goal. 
   3. I am thinking bringing them together will bring more interest and momentum 
to both PyPy but more importantly a general purpose Restricted Python Compiler.

Sarvi 


----- Original Message ----
> From: Benjamin Peterson <benjamin at python.org>
> To: PyPy Dev <pypy-dev at codespeak.net>
> Sent: Tue, September 14, 2010 3:19:36 PM
> Subject: [pypy-dev] Fwd:  PyPy to generate C/C++ code
> 
> ---------- Forwarded message ----------
> From: Benjamin Peterson <benjamin at python.org>
> Date:  2010/9/14
> Subject: Re: [pypy-dev] PyPy to generate C/C++ code
> To:  Saravanan Shanmugham <sarvi at yahoo.com>
> 
> 
> 2010/9/14  Saravanan Shanmugham <sarvi at yahoo.com>:
> > I don't expect  this python compiler to be for full python but just a 
>Restricted
> >  statically typed subset of python as defined by Shedskin.
> 
> So how is that  any different than the existing  RPython?
> 
> 
> 
> --
> Regards,
> Benjamin
> 
> 
> 
> -- 
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


      


From amauryfa at gmail.com  Wed Sep 15 01:11:28 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 15 Sep 2010 01:11:28 +0200
Subject: [pypy-dev] PyPy to generate C/C++ code
In-Reply-To: <708334.39926.qm@web53705.mail.re2.yahoo.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
	<AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>
	<708334.39926.qm@web53705.mail.re2.yahoo.com>
Message-ID: <AANLkTi=XMxACmjXUF0t9Vnpzfikrf84s4pDT=z4q7927@mail.gmail.com>

2010/9/15 Saravanan Shanmugham <sarvi at yahoo.com>:
> I don't expect this python compiler to be for full python but just a Restricted
> statically typed subset of python as defined by Shedskin.
>
> Yes. JIT annotation may not serve the purpose of generating a compiler.
> Hence the porting of the type inference engine and may be use JIT notations if
> it can be.

I've downloaded and read the source code of shedskin.
>From what I understand, here are some differences between PyPy and Shedksin.

- Shedskin analyses and generates code directly by walking the AST of a python
module.  (there are two passes: the first to grab information about global types
and functions, the second to emit code)

- Shedskin does very little type inference. Shedskin's type system is based on
C++ templates, and once a variable's type has been determined, generic code is
emitted and the C++ compiler will select the correct implementation.  Other
inference engines also work on the AST; Logilab's pylint, for example, works
much harder to check all instructions and the type of all variables.  Shedskin
does not seem to need such power.

- On the other hand, PyPy analyzes imported modules, and works on the bytecode
of functions living in memory.  It does a complete type inference and emits
low-level C code or Java intermediate representation.

- PyPy has its own way to write generic code and templates, the language for
meta-programming is Python itself!  [I'm referring to loops that
generate classes and functions, and things like "specialize:argtype(0)",
"unrolling_iterable" combined with constant propagation].

In most cases, PyPy does not generate better code than Shedskin. When Shedskin
compiles code, it does it well.  And its restrictions are easier to work with;
RPython is really tricky to get right sometimes.

Of course, PyPy goal is different: it does not only generate low-level C code,
it also generates a JIT compiler that can optimize calls at runtime - in the
context of an interpreter.  I can't see which computations made there could be
applied to static code.

Bottom line: if you want to generate efficient C code from python, use (and
improve) Shedskin.  If you want python code to run faster, don't translate
anything, and use the PyPy interpreter.

-- 
Amaury Forgeot d'Arc


From cfbolz at gmx.de  Wed Sep 15 10:05:17 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 15 Sep 2010 10:05:17 +0200
Subject: [pypy-dev] PyPy to generate C/C++ code
In-Reply-To: <AANLkTi=XMxACmjXUF0t9Vnpzfikrf84s4pDT=z4q7927@mail.gmail.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>	<AANLkTi=vhMrsEgEKFBtv7LmiCwiX5ah8=uYdYxW5JCMr@mail.gmail.com>	<708334.39926.qm@web53705.mail.re2.yahoo.com>
	<AANLkTi=XMxACmjXUF0t9Vnpzfikrf84s4pDT=z4q7927@mail.gmail.com>
Message-ID: <4C907E3D.10104@gmx.de>

Hi Amaury,

On 09/15/2010 01:11 AM, Amaury Forgeot d'Arc wrote:
> 2010/9/15 Saravanan Shanmugham<sarvi at yahoo.com>:
>> I don't expect this python compiler to be for full python but just a Restricted
>> statically typed subset of python as defined by Shedskin.
>>
>> Yes. JIT annotation may not serve the purpose of generating a compiler.
>> Hence the porting of the type inference engine and may be use JIT notations if
>> it can be.
>
> I've downloaded and read the source code of shedskin.
>> From what I understand, here are some differences between PyPy and Shedksin.
>
> - Shedskin analyses and generates code directly by walking the AST of a python
> module.  (there are two passes: the first to grab information about global types
> and functions, the second to emit code)
>
> - Shedskin does very little type inference. Shedskin's type system is based on
> C++ templates, and once a variable's type has been determined, generic code is
> emitted and the C++ compiler will select the correct implementation.  Other
> inference engines also work on the AST; Logilab's pylint, for example, works
> much harder to check all instructions and the type of all variables.  Shedskin
> does not seem to need such power.
>
> - On the other hand, PyPy analyzes imported modules, and works on the bytecode
> of functions living in memory.  It does a complete type inference and emits
> low-level C code or Java intermediate representation.
>
> - PyPy has its own way to write generic code and templates, the language for
> meta-programming is Python itself!  [I'm referring to loops that
> generate classes and functions, and things like "specialize:argtype(0)",
> "unrolling_iterable" combined with constant propagation].
>
> In most cases, PyPy does not generate better code than Shedskin. When Shedskin
> compiles code, it does it well.  And its restrictions are easier to work with;
> RPython is really tricky to get right sometimes.

Nice analysis and description, thank you!

Carl Friedrich


From jacob at openend.se  Wed Sep 15 11:01:26 2010
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 15 Sep 2010 11:01:26 +0200
Subject: [pypy-dev] PyPy to generate C/C++ code
In-Reply-To: <AANLkTi=XMxACmjXUF0t9Vnpzfikrf84s4pDT=z4q7927@mail.gmail.com>
References: <553714.16055.qm@web53703.mail.re2.yahoo.com>
	<708334.39926.qm@web53705.mail.re2.yahoo.com>
	<AANLkTi=XMxACmjXUF0t9Vnpzfikrf84s4pDT=z4q7927@mail.gmail.com>
Message-ID: <201009151101.27281.jacob@openend.se>

At a higher level of abstraction, Python is a dynamic language. The dynamicity 
is what makes it slow. There are simply so many things that might occur at 
runtime that have to be taken into account in the code. The JIT is designed to 
find cases where the dynamic properties of the language are not being used in 
that particular instance of execution, and generate faster code for that bit 
of the program.
This has almost nothing in common with trying to generate C or machine code 
from a static language that superficially looks like Python. Square Peg, Round 
Hole.

Jacob Hall?n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100915/a5225d57/attachment.pgp>

From fijall at gmail.com  Wed Sep 15 18:18:05 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 15 Sep 2010 18:18:05 +0200
Subject: [pypy-dev] [pypy-svn] r77083 - pypy/branch/jitffi
In-Reply-To: <20100915110721.C94F0282C16@codespeak.net>
References: <20100915110721.C94F0282C16@codespeak.net>
Message-ID: <AANLkTimwv_SmQEu4XdiPqBke55GYUW7wAFQk41PA3HWC@mail.gmail.com>

Hey anto.

There was a SoC about that, I guess it would be good to chat about it
at least (personally I think jitting rlib/libffi is exactly bad layer
to be jitted and some experiments were done).

Cheers,
fijal

On Wed, Sep 15, 2010 at 1:07 PM,  <antocuni at codespeak.net> wrote:
> Author: antocuni
> Date: Wed Sep 15 13:07:20 2010
> New Revision: 77083
>
> Added:
> ? pypy/branch/jitffi/ ? (props changed)
> ? ? ?- copied from r77082, pypy/trunk/
> Log:
> a branch in which to try to jit rlib/libffi.py
>
>
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>


From bhartsho at yahoo.com  Thu Sep 16 02:44:35 2010
From: bhartsho at yahoo.com (Hart's Antler)
Date: Wed, 15 Sep 2010 17:44:35 -0700 (PDT)
Subject: [pypy-dev] External RPython mailing list
Message-ID: <463721.60230.qm@web114014.mail.gq1.yahoo.com>

Porting ShedSkin to use the PyPy translation toolchain on the surface sounds like a good idea, but its not if we look at the details.  The first issue is legal, PyPy is MIT licensed, which works very well when integrated by commerical software.  But Shedskin uses the GNU GPL3, so importing any of its code (or code that imports GPL code, etc) into the user's compiled program also binds it to the GPL - which is no good for commerical software.  Shedskin within the PyPy toolchain may taint the users program with GPL code because some RPython programs will import from rlib (which may in someway depend on Mark's GPL code).

The second issue is technical, not much is gained for the likely great amount of effort it would take to merge ShedSkin.  Lets look at what is gained:
	1. muteable globals
	2. None and (int,float) are intermixable as attributes on an instance because ShedSkin has some limited support for dynamic-sub-types.  (PyPy can not mix None with int and float)
	3. operator overloading (except for __iter__ and __call__), PyPy only allows overloading __init__ and __del__

#1, would be nice to have but its an easy workaround to use singleton instances.
#2, no big advantage.
#3, this is a big advantage, i wish i could at least overload __getattr__ in PyPy Rpy.

ShedSkin is behind PyPy Rpy in the following areas:
	1. no getattr, hasattr etc.
	2. *args		(Mark says he can bring it back but only for homogenous types (PyPy supports *args with non-homogenous types))
	3. passing method references
	4. no interface to C
	5. mixed-type tuples are limited to length two (PyPy allows for any length)
	6. multiple inheritance

The ShedSkin readme itself states that " the type inference techniques employed by Shed Skin currently do not scale very well beyond several hundred lines of code" and recommends ShedSkin only for small programs.  Not having the 6 items above are additional reasons why ShedSkin is not ideal for writting large programs, these are serious limitations for a large program, especially #4 - not having a easy way to interface with C is a huge show-stopper; PyPy Rpy has an amazingly simple way to interface with C (rffi).  The other dissadvantages of ShedSkin are: slow object allocation (Phil Hassey did a test showing ShedSkin 30% slower than RPython), and it only translates to C++ while PyPy can translate to C, Java, and #C.

There are simple ways to improve PyPy Rpy that will benifit both the PyPy project and those who strictly want to use the translation toolchain.  I've been following the progress of this years Google Summer of Code projects, and i see a big stumbling block for everybody was RPython.  PyPy today would have a better 64bit JIT, faster ctypes, and better numpy support if RPython was itself better.

RPython Wishlist:
	. documentation
	. iteration over tuples of any length (with any mixed types)
	. overloading __getattr__, __setattr__
	. pickle support, if its limited thats ok.
	. rstruct is incomplete
	. llvm backend, what happened to llvm support?
	. not having to define dummy functions on the base class to prevent 'demotion'
	. not having to use the hack `assert isinstance(a,MySubClass)` to call methods with incompatible signatures.
	. we already have the decorator: @specialize.argtype(1), why can't we have @specialize.argtype(*) so that all arguments can have flexible types?
	. methods stored in a list for easy dispatch can not have mismatched signatures.

I aggree with Fijal, CPython module extension should be a low priority.  There is a big speed overhead when passing data back and forth from CPython, and speed is the whole point of going through the trouble of writting in RPython.  Those who are less concerned with speed and want CPython module extensions can use Cython, which is well tested.  Those who are interested in RPython and want a simple way to get started can use ShedSkin to make an extension module, and then migrate their module to a standalone app with PyPy if they choose.

-hart





From william.leslie.ttg at gmail.com  Thu Sep 16 04:10:29 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Thu, 16 Sep 2010 12:10:29 +1000
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
References: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
Message-ID: <AANLkTi=QxfdMrP0f2RcNq2LP=SmLaBuk2ouBNZxqzE6Z@mail.gmail.com>

On 16 September 2010 10:44, Hart's Antler <bhartsho at yahoo.com> wrote:
> [the GPL] is no good for commerical software.

Please don't go there.

> ... what is gained:
> ? ? ? ?2. None and (int,float) are intermixable as attributes on an instance because ShedSkin has some limited support for dynamic-sub-types. ?(PyPy can not mix None with int and float)

And it's not obvious we would want this anyway, as such type
information has to live somewhere, and having to carry that around
makes using ints and floats more expensive. In pypy-c, small applevel
ints are already tagged in the obvious way (the lowest bit of a
pointer is always unset, so setting it tags an app-level integer), but
rpython ints shouldn't have to live with the performance hit and
additional accuracy loss where tagged ints are cast to native ints.

> ? ? ? ?3. operator overloading (except for __iter__ and __call__), PyPy only allows overloading __init__ and __del__

Makes rtyping more work. It takes long enough as it is.

> ShedSkin is behind PyPy Rpy in the following areas:
> ? ? ? ?1. no getattr, hasattr etc.

Does rpython really have this? As it appears in, say, the main python
eval loop, it's constant folded. This is a good example of python
usage in metaprogramming rpython.

> ? ? ? ?6. multiple inheritance

That would limit rpython's ability to target the CLI and JVM if
implemented naeively. It's doable, but means we need to use interfaces
on ootype targets and need to do more work to look up methods in the C
backend - possibly implementing hotspot-style itables or virtual
inheritance or passing a dictionary around. It could get ugly quickly,
not only for the backends, but also for the JIT generator, which
already has too much to think about in terms of the model it is
implementing.

> ? ? ? ?. not having to define dummy functions on the base class to prevent 'demotion'

Some concept of an interface would be handy.

-- 
William Leslie


From santagada at gmail.com  Thu Sep 16 05:01:50 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 16 Sep 2010 00:01:50 -0300
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
References: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
Message-ID: <AANLkTi=2Y+1=2CAKZD9eBr+Z19qDDEdN7Ocj-0+U8fx9@mail.gmail.com>

If anyone wants to pay developers to work on rpython it should
probably follow this wishlist and not focus on trying to merge
shedskin into pypy (for the all the reasons given above).

Here is my version of the list

On Wed, Sep 15, 2010 at 9:44 PM, Hart's Antler <bhartsho at yahoo.com> wrote:
> RPython Wishlist:
> ? ? ? ?. documentation
Yes, this should be the highest priority. I could add documentation,
doctests explaining stuff, templates for new modules and other things,
better error messages.

> ? ? ? ?. iteration over tuples of any length (with any mixed types)
> ? ? ? ?. overloading __getattr__, __setattr__
> ? ? ? ?. pickle support, if its limited thats ok.
did anyone ever really needed this? Would be cool I think, but unnecessary.

> ? ? ? ?. rstruct is incomplete
> ? ? ? ?. llvm backend, what happened to llvm support?
also I don't know about the utility of this for rpython itself, would
be interesting if there was a jit backend that used llvm to better jit
everything.

> ? ? ? ?. not having to define dummy functions on the base class to prevent 'demotion'
> ? ? ? ?. not having to use the hack `assert isinstance(a,MySubClass)` to call methods with incompatible signatures.
both are nice to have in a language with type inferences. Without this
explicit in the code maybe the error messages would be even more
complex to deal with.

> ? ? ? ?. we already have the decorator: @specialize.argtype(1), why can't we have @specialize.argtype(*) so that all arguments can have flexible types?
could be @specilize.argtype(42)

> ? ? ? ?. methods stored in a list for easy dispatch can not have mismatched signatures.

I would add:
        . tool to automatically generate stubs for c libraries
        . c++ support (to be able to use c++ libs directly on rpython)
        . better java support (because the jvm matters).
        . better tooling (profiling, debugging etc)
        . separate compilation and/or some cache to speed up compilation.


-- 
Leonardo Santagada


From sarvi at yahoo.com  Thu Sep 16 05:03:43 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Wed, 15 Sep 2010 20:03:43 -0700 (PDT)
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
References: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
Message-ID: <119208.74110.qm@web53706.mail.re2.yahoo.com>





----- Original Message ----
> From: Hart's Antler <bhartsho at yahoo.com>
> To: pypy-dev at codespeak.net
> Cc: Saravanan Shanmugham <sarvi at yahoo.com>
> Sent: Wed, September 15, 2010 5:44:35 PM
> Subject: Re: External RPython mailing list
> 
> Porting ShedSkin to use the PyPy translation toolchain on the surface sounds  
>like a good idea, but its not if we look at the details.  The first issue  is 
>legal, PyPy is MIT licensed, which works very well when integrated by  
>commerical software.  But Shedskin uses the GNU GPL3, so importing any of  its 
>code (or code that imports GPL code, etc) into the user's compiled program  also 
>binds it to the GPL - which is no good for commerical software.   Shedskin 
>within the PyPy toolchain may taint the users program with GPL code  because 
>some RPython programs will import from rlib (which may in someway depend  on 
>Mark's GPL code).

Sarvi: Good point. I hadn't noticed that the generated C++ code was GPL.
I think Mark might be open to MIT licesing the generated C++ code. Coz, without 
it Shedskin as a tool chain would make no sense.
But then thats a different story.

Either way, it looks like there i not much enthusiasm for porting Shedskin on 
PyPy and have pypy generate a compiler instead of an interpreter.

>From various threads on python.org as well pypy itself, I see a lot of interest 
in a compiler for a staticaly typed subset of python.
I also feel that a statically typed subset of python can be faster than the 
dynamic superset. 

Which is why I have exploring options to spur some interest to drive some 
momentum in this area. Hasn't been easy.

I can see why some might feel that RPython is not for general use and only for 
language development. 

But what totally surprises me though is that as a language developer, I would 
want RPython to be as flexible as possibile within feasibility of course.

Anyway, I am going to keep it simple and just start exploring just expanding on 
PyPy's RPython compiler to make it more general purpose.

yes, on a separate branch :-))  of ofcourse. :-)

I'll start with some of the items below.

Lets see where that can go.

Sarvi


> 
> The second issue is technical, not much is gained  for the likely great amount 
>of effort it would take to merge ShedSkin.   Lets look at what is gained:
>     1. muteable  globals
>     2. None and (int,float) are intermixable as  attributes on an instance 
>because ShedSkin has some limited support for  dynamic-sub-types.  (PyPy can not 
>mix None with int and  float)
>     3. operator overloading (except for __iter__ and  __call__), PyPy only 
>allows overloading __init__ and __del__
> 
> #1, would be  nice to have but its an easy workaround to use singleton 
>instances.
> #2, no  big advantage.
> #3, this is a big advantage, i wish i could at least overload  __getattr__ in 
>PyPy Rpy.
> 
> ShedSkin is behind PyPy Rpy in the following  areas:
>     1. no getattr, hasattr etc.
>      2. *args        (Mark says he can bring it back  but only for homogenous 
>types (PyPy supports *args with non-homogenous  types))
>     3. passing method references
>      4. no interface to C
>     5. mixed-type tuples are limited to  length two (PyPy allows for any 
>length)
>     6. multiple  inheritance
> 
> The ShedSkin readme itself states that " the type inference  techniques 
>employed by Shed Skin currently do not scale very well beyond several  hundred 
>lines of code" and recommends ShedSkin only for small programs.   Not having the 
>6 items above are additional reasons why ShedSkin is not ideal  for writting 
>large programs, these are serious limitations for a large program,  especially 
>#4 - not having a easy way to interface with C is a huge  show-stopper; PyPy Rpy 
>has an amazingly simple way to interface with C  (rffi).  The other 
>dissadvantages of ShedSkin are: slow object allocation  (Phil Hassey did a test 
>showing ShedSkin 30% slower than RPython), and it only  translates to C++ while 
>PyPy can translate to C, Java, and #C.
> 
> There are  simple ways to improve PyPy Rpy that will benifit both the PyPy 
>project and  those who strictly want to use the translation toolchain.  I've 
>been  following the progress of this years Google Summer of Code projects, and i 
>see a  big stumbling block for everybody was RPython.  PyPy today would have a  
>better 64bit JIT, faster ctypes, and better numpy support if RPython was itself  
>better.
> 
> RPython Wishlist:
>     .  documentation
>     . iteration over tuples of any length (with  any mixed types)
>     . overloading __getattr__,  __setattr__
>     . pickle support, if its limited thats  ok.
>     . rstruct is incomplete
>     . llvm  backend, what happened to llvm support?
>     . not having to  define dummy functions on the base class to prevent  
>'demotion'
>     . not having to use the hack `assert  isinstance(a,MySubClass)` to call 
>methods with incompatible  signatures.
>     . we already have the decorator:  @specialize.argtype(1), why can't we have 
>@specialize.argtype(*) so that all  arguments can have flexible types?
>     . methods stored in a  list for easy dispatch can not have mismatched 
>signatures.
> 
> I aggree with  Fijal, CPython module extension should be a low priority.  There 
>is a big  speed overhead when passing data back and forth from CPython, and 
>speed is the  whole point of going through the trouble of writting in RPython.  
>Those who  are less concerned with speed and want CPython module extensions can 
>use Cython,  which is well tested.  Those who are interested in RPython and want 
>a  simple way to get started can use ShedSkin to make an extension module, and 
>then  migrate their module to a standalone app with PyPy if they  choose.
> 
> -hart
> 
> 
> 
> 


      


From william.leslie.ttg at gmail.com  Thu Sep 16 05:42:11 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Thu, 16 Sep 2010 13:42:11 +1000
Subject: [pypy-dev] External RPython mailing list
In-Reply-To: <119208.74110.qm@web53706.mail.re2.yahoo.com>
References: <463721.60230.qm@web114014.mail.gq1.yahoo.com>
	<119208.74110.qm@web53706.mail.re2.yahoo.com>
Message-ID: <AANLkTi=qb=djR2=4CrZTqjEUQie4=XHfHm7p3Kxe5vQ-@mail.gmail.com>

On 16 September 2010 13:03, Saravanan Shanmugham <sarvi at yahoo.com> wrote:
> Either way, it looks like there i not much enthusiasm for porting Shedskin on
> PyPy and have pypy generate a compiler instead of an interpreter.

In a sense, it already does :). And of course translation is compilation, too.

> >From various threads on python.org as well pypy itself, I see a lot of interest
> in a compiler for a staticaly typed subset of python.
> I also feel that a statically typed subset of python can be faster than the
> dynamic superset.

It can be, there's nothing stopping you from dynamically compiling a
static language, and feeding back profiling information is easy too.
It's just probably not going to be *that* much faster to be of value
at the end of the day. Then again, the end of the day could be a long
way away.

> I can see why some might feel that RPython is not for general use and only for
> language development.
>
> But what totally surprises me though is that as a language developer, I would
> want RPython to be as flexible as possibile within feasibility of course.

You also have to look at it from the other perspective - that of
someone implementing a backend or translation aspect, such as a
garbage collector or a JIT compiler generator. This is my perspective
coming to pypy - I am experimenting with a range of optimisations
based on extensive region and effect analysis, and I fear rpython
already makes this difficult.

For example, the use of abstract interpretation to generate the
flowgraph IR means that you now have no information about which loop
is the 'outer' one, and that information can be useful in generating
heuristics. Similar things could be said about the JIT and generators,
which is not something I have looked at extensively, but dealing with
the generator case would have been implicit from the start if the IR
used a CPS transform to represent all instruction flow. In short:
rpython is complicated enough already.

It happens to do the job it was created for, but not a whole lot more
than that. It happens to be well suited to my experiments for two
unrelated reasons*. But I can't imagine choosing it to write extension
modules or inner loops - there are plenty of languages that do it
better, like cython, pyrex, D, cyclone, SML, etc.

* it's (memory) safe and the rffi is sane, particularly about letting
native code deal with rpython objects. And thanks to the pypy python
interpreter, there's a large body of code to test it on.

-- 
William Leslie


From anto.cuni at gmail.com  Thu Sep 16 08:57:12 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 16 Sep 2010 08:57:12 +0200
Subject: [pypy-dev] [pypy-svn] r77083 - pypy/branch/jitffi
In-Reply-To: <AANLkTimwv_SmQEu4XdiPqBke55GYUW7wAFQk41PA3HWC@mail.gmail.com>
References: <20100915110721.C94F0282C16@codespeak.net>
	<AANLkTimwv_SmQEu4XdiPqBke55GYUW7wAFQk41PA3HWC@mail.gmail.com>
Message-ID: <4C91BFC8.9060600@gmail.com>

Hi,

On 15/09/10 18:18, Maciej Fijalkowski wrote:
> Hey anto.
>
> There was a SoC about that, I guess it would be good to chat about it
> at least (personally I think jitting rlib/libffi is exactly bad layer
> to be jitted and some experiments were done).

yes, I read the code in the fast-ctypes branch but I wanted to take another 
(simpler) approach.  Note that my goal is not only to speed up ctypes, but 
also to provide a useful building block for cppyy (the module to call c++ 
functions that we started at the cern sprint).

My basic idea was to mark libffi.FuncPtr.{push_arg,call} in a special way, so 
that the backend can recognize the pattern (i.e. push* + call) and emit a 
single assembler call.

I even started to write a bit of code, but then I realized that libffi.FuncPtr 
is not used at all, as _rawffi uses RawFuncPtr: the bad news is that 
RawFuncPtr uses a different interface, as it does not have push_arg but passes 
the arguments already packed in a list, so my easy solution above cannot work. 
  Note however that doing it at the level of FuncPtr might still be useful for 
cppyy.

Question: why does _rawffi use RawFuncPtr instead of FuncPtr? Would it be 
possible/easy/hard/whatever to switch to FuncPtr?

ciao,
Anto


From anto.cuni at gmail.com  Thu Sep 16 09:03:32 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 16 Sep 2010 09:03:32 +0200
Subject: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl
 module/__builtin__ module/__builtin__/test module/pypyjit/test
In-Reply-To: <20100916052748.2421F282C23@codespeak.net>
References: <20100916052748.2421F282C23@codespeak.net>
Message-ID: <4C91C144.3010100@gmail.com>

Hi,

On 16/09/10 07:27, hakanardo at codespeak.net wrote:

> Log:
> Allow jit to unroll calls to max() and min() with more than one argument.
>
[cut]
> + at unroll_safe
>   @specialize.arg(2)
>   def min_max(space, args, implementation_of):
>       if implementation_of == "max":
>           compare = space.gt
>       else:
>           compare = space.lt
> +
> +    args_w = args.arguments_w
> +    if len(args_w)>  1 and not args.keywords: # Unrollable case
> +        w_max_item = None
> +        for w_item in args_w:
> +            if w_max_item is None or \
> +                   space.is_true(compare(w_item, w_max_item)):
> +                w_max_item = w_item
> +        return w_max_item
> +    else:
> +        return min_max_loop(space, args, implementation_of)


I don't think it's a good idea. What happens if I call max() over a list of 1 
million of elements? We obviously don't want the jit to unroll 1 million of 
iterations. Or am I missing something?

ciao,
Anto


From hakan at debian.org  Thu Sep 16 09:18:57 2010
From: hakan at debian.org (Hakan Ardo)
Date: Thu, 16 Sep 2010 09:18:57 +0200
Subject: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl
 module/__builtin__ module/__builtin__/test module/pypyjit/test
In-Reply-To: <4C91C144.3010100@gmail.com>
References: <20100916052748.2421F282C23@codespeak.net>
	<4C91C144.3010100@gmail.com>
Message-ID: <AANLkTimn-MWH9ONi6QpGuejjPhdi+yDBYB02tOBnbdPO@mail.gmail.com>

2010 at 9:03 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> +
>> + ? ?args_w = args.arguments_w
>> + ? ?if len(args_w)> ?1 and not args.keywords: # Unrollable case
>> + ? ? ? ?w_max_item = None
>> + ? ? ? ?for w_item in args_w:
>> + ? ? ? ? ? ?if w_max_item is None or \
>> + ? ? ? ? ? ? ? ? ? space.is_true(compare(w_item, w_max_item)):
>> + ? ? ? ? ? ? ? ?w_max_item = w_item
>> + ? ? ? ?return w_max_item
>> + ? ?else:
>> + ? ? ? ?return min_max_loop(space, args, implementation_of)
>
>
> I don't think it's a good idea. What happens if I call max() over a list of 1
> million of elements? We obviously don't want the jit to unroll 1 million of
> iterations. Or am I missing something?

If lst is your list, the call max(lst) has a single argument, the
list, and it will be passed to the old implementation now called
min_max_loop. However if you call max(*lst) the jit will unroll it.
But why would you do that? The idea here was to optimize the case
max(i,0) where you typically only have a few arguments. Anyway, how
about calling min_max_loop() as soon as len(args_w)  > 10 to be on the
safe side?

-- 
H?kan Ard?


From arigo at tunes.org  Thu Sep 16 17:44:05 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Sep 2010 17:44:05 +0200
Subject: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl
 module/__builtin__ module/__builtin__/test module/pypyjit/test
In-Reply-To: <AANLkTimn-MWH9ONi6QpGuejjPhdi+yDBYB02tOBnbdPO@mail.gmail.com>
References: <20100916052748.2421F282C23@codespeak.net>
	<4C91C144.3010100@gmail.com>
	<AANLkTimn-MWH9ONi6QpGuejjPhdi+yDBYB02tOBnbdPO@mail.gmail.com>
Message-ID: <AANLkTinoD_-QdqkzYJeRvMb-pZV-jQek+swhhz+uYjzN@mail.gmail.com>

Hi,

Can you maybe make a branch, and move the checkin on the branch?
That's the kind of change that needs careful consideration...


Armin


From garyrob at me.com  Thu Sep 23 17:51:13 2010
From: garyrob at me.com (Gary Robinson)
Date: Thu, 23 Sep 2010 11:51:13 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
Message-ID: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>

Hi,

I saw the PyPy Status Blog post mentioning that there is a working asmgcc for x86_64 linux. I wonder if you could clarify the status of it a bit further. The last thing Jason Creighton wrote on the subject that I can find, from Aug 13, was: "...the bottom line is that the main goal of my GSoC was accomplished: A working 64-bit PyPy JIT. Hopefully I'll be able to complete asmgcc-64, and make the JIT even faster..."

But the new Status Blog post says " It not only includes working 64bit JIT (merged into PyPy trunk), but also a working asmgcc for x86_64 linux platform, that makes it possible to run the JIT on this architecture with our advanced garbage collectors"

So it sounds like he (or someone) DID get the Linux version of it working. Has it been merged into the trunk? Does it seem stable? You say: "Expect this to be a major selling point for the next PyPy release :-)"  Do you have an estimate of when that'll come out?

I'm looking forward to testing PyPy for some of our music recommendation code. The main thing holding me back so far is the lack of 64-bit support.

The other thing in the way is that I need to use multiple cores. I can home-grow a solution for my needs, but it would be great if the python multiprocessing library were to be supported. I see "r77223 - in	pypy/branch/fast-forward/pypy/module/_multiprocessing: . test" in the svn commit log, dated Tuesday of this week (http://permalink.gmane.org/gmane.comp.python.pypy.cvs/29865)... I'm hoping that means it's going to be supported soon? That would be really great.

Thanks,
Gary

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From alex.gaynor at gmail.com  Thu Sep 23 18:54:16 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 23 Sep 2010 12:54:16 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
Message-ID: <AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>

On Thu, Sep 23, 2010 at 11:51 AM, Gary Robinson <garyrob at me.com> wrote:
> Hi,
>
> I saw the PyPy Status Blog post mentioning that there is a working asmgcc for x86_64 linux. I wonder if you could clarify the status of it a bit further. The last thing Jason Creighton wrote on the subject that I can find, from Aug 13, was: "...the bottom line is that the main goal of my GSoC was accomplished: A working 64-bit PyPy JIT. Hopefully I'll be able to complete asmgcc-64, and make the JIT even faster..."
>
> But the new Status Blog post says " It not only includes working 64bit JIT (merged into PyPy trunk), but also a working asmgcc for x86_64 linux platform, that makes it possible to run the JIT on this architecture with our advanced garbage collectors"
>
> So it sounds like he (or someone) DID get the Linux version of it working. Has it been merged into the trunk? Does it seem stable? You say: "Expect this to be a major selling point for the next PyPy release :-)" ?Do you have an estimate of when that'll come out?
>
> I'm looking forward to testing PyPy for some of our music recommendation code. The main thing holding me back so far is the lack of 64-bit support.
>
> The other thing in the way is that I need to use multiple cores. I can home-grow a solution for my needs, but it would be great if the python multiprocessing library were to be supported. I see "r77223 - in ?pypy/branch/fast-forward/pypy/module/_multiprocessing: . test" in the svn commit log, dated Tuesday of this week (http://permalink.gmane.org/gmane.comp.python.pypy.cvs/29865)... I'm hoping that means it's going to be supported soon? That would be really great.
>
> Thanks,
> Gary
>
> --
>
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog: ? ?http://www.garyrobinson.net
>
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Yes, 64-bit support for asmgcc as merged, however there appears to be
a performance issue with it, it's not nearly as fast as it should be.

multiproccessing was added to the stdlib in 2.6, we have a
fast-forward branch that's aiming to implement 2.7, so when it's
released it will contain a multiprocessing module.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me


From garyrob at me.com  Thu Sep 23 23:28:23 2010
From: garyrob at me.com (Gary Robinson)
Date: Thu, 23 Sep 2010 17:28:23 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
Message-ID: <885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>

Thanks for your response Alex... I have a couple follow-up questions:

> Yes, 64-bit support for asmgcc as merged, however there appears to be
> a performance issue with it, it's not nearly as fast as it should be.

Is this a matter that is getting PyPy developer attention, or is expected to in the relatively near future?

> multiproccessing was added to the stdlib in 2.6, we have a
> fast-forward branch that's aiming to implement 2.7, so when it's
> released it will contain a multiprocessing module.

That's great news. Is there any estimate of when a fairly stable beta will be available?

Thanks!
Gary

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Sep 23, 2010, at 12:54 PM, Alex Gaynor wrote:

> On Thu, Sep 23, 2010 at 11:51 AM, Gary Robinson <garyrob at me.com> wrote:
>> Hi,
>> 
>> I saw the PyPy Status Blog post mentioning that there is a working asmgcc for x86_64 linux. I wonder if you could clarify the status of it a bit further. The last thing Jason Creighton wrote on the subject that I can find, from Aug 13, was: "...the bottom line is that the main goal of my GSoC was accomplished: A working 64-bit PyPy JIT. Hopefully I'll be able to complete asmgcc-64, and make the JIT even faster..."
>> 
>> But the new Status Blog post says " It not only includes working 64bit JIT (merged into PyPy trunk), but also a working asmgcc for x86_64 linux platform, that makes it possible to run the JIT on this architecture with our advanced garbage collectors"
>> 
>> So it sounds like he (or someone) DID get the Linux version of it working. Has it been merged into the trunk? Does it seem stable? You say: "Expect this to be a major selling point for the next PyPy release :-)"  Do you have an estimate of when that'll come out?
>> 
>> I'm looking forward to testing PyPy for some of our music recommendation code. The main thing holding me back so far is the lack of 64-bit support.
>> 
>> The other thing in the way is that I need to use multiple cores. I can home-grow a solution for my needs, but it would be great if the python multiprocessing library were to be supported. I see "r77223 - in  pypy/branch/fast-forward/pypy/module/_multiprocessing: . test" in the svn commit log, dated Tuesday of this week (http://permalink.gmane.org/gmane.comp.python.pypy.cvs/29865)... I'm hoping that means it's going to be supported soon? That would be really great.
>> 
>> Thanks,
>> Gary
>> 
>> --
>> 
>> Gary Robinson
>> CTO
>> Emergent Discovery, LLC
>> personal email: garyrob at me.com
>> work email: grobinson at emergentdiscovery.com
>> Company: http://www.emergentdiscovery.com
>> Blog:    http://www.garyrobinson.net
>> 
>> 
>> 
>> 
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>> 
> 
> Yes, 64-bit support for asmgcc as merged, however there appears to be
> a performance issue with it, it's not nearly as fast as it should be.
> 
> multiproccessing was added to the stdlib in 2.6, we have a
> fast-forward branch that's aiming to implement 2.7, so when it's
> released it will contain a multiprocessing module.
> 
> Alex
> 
> -- 
> "I disapprove of what you say, but I will defend to the death your
> right to say it." -- Voltaire
> "The people's good is the highest law." -- Cicero
> "Code can always be simpler than you think, but never as simple as you
> want" -- Me



From alex.gaynor at gmail.com  Thu Sep 23 23:34:54 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 23 Sep 2010 17:34:54 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
Message-ID: <AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>

On Thu, Sep 23, 2010 at 5:28 PM, Gary Robinson <garyrob at me.com> wrote:
> Thanks for your response Alex... I have a couple follow-up questions:
>
>> Yes, 64-bit support for asmgcc as merged, however there appears to be
>> a performance issue with it, it's not nearly as fast as it should be.
>
> Is this a matter that is getting PyPy developer attention, or is expected to in the relatively near future?
>

We're aware of it, and it will definitely happen before we do any sort
of release.

>> multiproccessing was added to the stdlib in 2.6, we have a
>> fast-forward branch that's aiming to implement 2.7, so when it's
>> released it will contain a multiprocessing module.
>
> That's great news. Is there any estimate of when a fairly stable beta will be available?
>

Amaury or Benjamin could better say.

> Thanks!
> Gary
>
> --
>
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog: ? ?http://www.garyrobinson.net
>
>
>
>
> On Sep 23, 2010, at 12:54 PM, Alex Gaynor wrote:
>
>> On Thu, Sep 23, 2010 at 11:51 AM, Gary Robinson <garyrob at me.com> wrote:
>>> Hi,
>>>
>>> I saw the PyPy Status Blog post mentioning that there is a working asmgcc for x86_64 linux. I wonder if you could clarify the status of it a bit further. The last thing Jason Creighton wrote on the subject that I can find, from Aug 13, was: "...the bottom line is that the main goal of my GSoC was accomplished: A working 64-bit PyPy JIT. Hopefully I'll be able to complete asmgcc-64, and make the JIT even faster..."
>>>
>>> But the new Status Blog post says " It not only includes working 64bit JIT (merged into PyPy trunk), but also a working asmgcc for x86_64 linux platform, that makes it possible to run the JIT on this architecture with our advanced garbage collectors"
>>>
>>> So it sounds like he (or someone) DID get the Linux version of it working. Has it been merged into the trunk? Does it seem stable? You say: "Expect this to be a major selling point for the next PyPy release :-)" ?Do you have an estimate of when that'll come out?
>>>
>>> I'm looking forward to testing PyPy for some of our music recommendation code. The main thing holding me back so far is the lack of 64-bit support.
>>>
>>> The other thing in the way is that I need to use multiple cores. I can home-grow a solution for my needs, but it would be great if the python multiprocessing library were to be supported. I see "r77223 - in ?pypy/branch/fast-forward/pypy/module/_multiprocessing: . test" in the svn commit log, dated Tuesday of this week (http://permalink.gmane.org/gmane.comp.python.pypy.cvs/29865)... I'm hoping that means it's going to be supported soon? That would be really great.
>>>
>>> Thanks,
>>> Gary
>>>
>>> --
>>>
>>> Gary Robinson
>>> CTO
>>> Emergent Discovery, LLC
>>> personal email: garyrob at me.com
>>> work email: grobinson at emergentdiscovery.com
>>> Company: http://www.emergentdiscovery.com
>>> Blog: ? ?http://www.garyrobinson.net
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>
>> Yes, 64-bit support for asmgcc as merged, however there appears to be
>> a performance issue with it, it's not nearly as fast as it should be.
>>
>> multiproccessing was added to the stdlib in 2.6, we have a
>> fast-forward branch that's aiming to implement 2.7, so when it's
>> released it will contain a multiprocessing module.
>>
>> Alex
>>
>> --
>> "I disapprove of what you say, but I will defend to the death your
>> right to say it." -- Voltaire
>> "The people's good is the highest law." -- Cicero
>> "Code can always be simpler than you think, but never as simple as you
>> want" -- Me
>
>

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me


From benjamin at python.org  Thu Sep 23 23:55:55 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 23 Sep 2010 16:55:55 -0500
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
Message-ID: <AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>

2010/9/23 Alex Gaynor <alex.gaynor at gmail.com>:
> On Thu, Sep 23, 2010 at 5:28 PM, Gary Robinson <garyrob at me.com> wrote:
>> Thanks for your response Alex... I have a couple follow-up questions:
>>
>>> Yes, 64-bit support for asmgcc as merged, however there appears to be
>>> a performance issue with it, it's not nearly as fast as it should be.
>>
>> Is this a matter that is getting PyPy developer attention, or is expected to in the relatively near future?
>>
>
> We're aware of it, and it will definitely happen before we do any sort
> of release.
>
>>> multiproccessing was added to the stdlib in 2.6, we have a
>>> fast-forward branch that's aiming to implement 2.7, so when it's
>>> released it will contain a multiprocessing module.
>>
>> That's great news. Is there any estimate of when a fairly stable beta will be available?
>>
>
> Amaury or Benjamin could better say.

"Longer than you want"


-- 
Regards,
Benjamin


From alex.gaynor at gmail.com  Fri Sep 24 04:36:16 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 23 Sep 2010 22:36:16 -0400
Subject: [pypy-dev] PyCon Call for Proposals -- PyCon 2011
Message-ID: <AANLkTindmEgtGBop=E2q6Z8ZF8Xzy-G+jhM45uLsyJPg@mail.gmail.com>

Call for proposals -- PyCon 2011 -- <http://us.pycon.org/2011/>
===============================================================

Proposal Due date: November 1st, 2010

PyCon is back! With a rocking new website, a great location and
more Python hackers and luminaries under one roof than you could
possibly shake a stick at. We've also added an "Extreme" talk
track this year - no introduction, no fluff - only the pure
technical meat!

PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta,
Georgia. (Home of some of the best southern food you can possibly
find on Earth!) The PyCon conference days will be March 11-13,
preceded by two tutorial days (March 9-10), and followed by four
days of development sprints (March 14-17).

PyCon 2011 is looking for proposals for the formal presentation
tracks (this includes "extreme talks"). A request for proposals for
poster sessions and tutorials will come separately.

Want to showcase your skills as a Python Hacker? Want to have
hundreds of people see your talk on the subject of your choice? Have
some hot button issue you think the community needs to address, or have
some package, code or project you simply love talking about? Want to
launch your master plan to take over the world with Python?

PyCon is your platform for getting the word out and teaching something
new to hundreds of people, face to face.

In the past, PyCon has had a broad range of presentations, from reports
on academic and commercial projects, tutorials on a broad range of
subjects, and case studies. All conference speakers are volunteers and
come from a myriad of backgrounds: some are new speakers, some have been
speaking for years. Everyone is welcome, so bring your passion and your
code! We've had some incredible past PyCons, and we're looking to you to
help us top them!

Online proposal submission is open now! Proposals  will be accepted
through November 10th, with acceptance notifications coming out by
January 20th. To get started, please see:

   <http://us.pycon.org/2011/speaker/>

For videos of talks from previous years - check out:

   <http://python.mirocommunity.org/category/conferences>

For more information on "Extreme Talks" see:

   <http://us.pycon.org/2011/speaker/extreme/>

We look forward to seeing you in Atlanta!

Please also note - registration for PyCon 2011 will also be capped at a
maximum of 1,500 delegates, including speakers. When registration opens
(soon), you're going to want to make sure you register early! Speakers
with accepted talks will have a guaranteed slot.

Important Dates:
   * November 1st, 2010: Talk proposals due.
   * December 15th, 2010: Acceptance emails sent.
   * January 19th, 2010: Early bird registration closes.
   * March 9-10th, 2011: Tutorial days at PyCon.
   * March 11-13th, 2011: PyCon main conference.
   * March 14-17th, 2011: PyCon sprints days.

Contact Emails:
   Van Lindberg (Conference Chair) - van at python.org
   Jesse Noller (Co-Chair) - jnoller at python.org
   PyCon Organizers list: pycon-organizers at python.org

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me


From garyrob at me.com  Fri Sep 24 18:40:22 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 24 Sep 2010 12:40:22 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
	<AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
Message-ID: <E8202367-ADF8-4D05-91D9-28F471697227@me.com>



-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net


> 
> "Longer than you want"

Ain't that the truth! :)

And yet, if you felt like one of the following blocks if time was more likely than the others, I'd be very interested in knowing:

   < 6 months
   < 1 year
   < 2 years

Or if you'd rather not say anything even at that level, I understand.

Thanks!
Gary


On Sep 23, 2010, at 5:55 PM, Benjamin Peterson wrote:

> 2010/9/23 Alex Gaynor <alex.gaynor at gmail.com>:
>> On Thu, Sep 23, 2010 at 5:28 PM, Gary Robinson <garyrob at me.com> wrote:
>>> Thanks for your response Alex... I have a couple follow-up questions:
>>> 
>>>> Yes, 64-bit support for asmgcc as merged, however there appears to be
>>>> a performance issue with it, it's not nearly as fast as it should be.
>>> 
>>> Is this a matter that is getting PyPy developer attention, or is expected to in the relatively near future?
>>> 
>> 
>> We're aware of it, and it will definitely happen before we do any sort
>> of release.
>> 
>>>> multiproccessing was added to the stdlib in 2.6, we have a
>>>> fast-forward branch that's aiming to implement 2.7, so when it's
>>>> released it will contain a multiprocessing module.
>>> 
>>> That's great news. Is there any estimate of when a fairly stable beta will be available?
>>> 
>> 
>> Amaury or Benjamin could better say.
> 
> "Longer than you want"
> 
> 
> -- 
> Regards,
> Benjamin



From amauryfa at gmail.com  Fri Sep 24 18:56:15 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 24 Sep 2010 18:56:15 +0200
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <E8202367-ADF8-4D05-91D9-28F471697227@me.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
	<AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
	<E8202367-ADF8-4D05-91D9-28F471697227@me.com>
Message-ID: <AANLkTinRtE=S9-HDVin6fPa4Rb4-xDYVa8Lh6vppm_yY@mail.gmail.com>

Hi,
2010/9/24 Gary Robinson <garyrob at me.com>:
>> "Longer than you want"
>
> Ain't that the truth! :)
>
> And yet, if you felt like one of the following blocks if time was more likely than the others, I'd be very interested in knowing:
>
> ? < 6 months
> ? < 1 year
> ? < 2 years

I will certainly give up before this delay.
At the moment, 1/3 of the files in the test suite pass without error,
they were zero yesterday.
We need volunteers to help us and implement the failing/missing parts!

-- 
Amaury Forgeot d'Arc


From garyrob at me.com  Fri Sep 24 19:16:01 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 24 Sep 2010 13:16:01 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <AANLkTinRtE=S9-HDVin6fPa4Rb4-xDYVa8Lh6vppm_yY@mail.gmail.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
	<AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
	<E8202367-ADF8-4D05-91D9-28F471697227@me.com>
	<AANLkTinRtE=S9-HDVin6fPa4Rb4-xDYVa8Lh6vppm_yY@mail.gmail.com>
Message-ID: <C9EB184B-A388-422F-B0E4-D7AD22B7814D@me.com>

> I will certainly give up before this delay.
> At the moment, 1/3 of the files in the test suite pass without error,
> they were zero yesterday.
> We need volunteers to help us and implement the failing/missing parts!

0 to 1/3 is big progress. Sounds to me like the <6 mo time frame might be the right one.

I unfortunately can't contribute code/patches at this point (I don't have the time-freedom or competence in JIT's or C++ to try), but when you have 64-bit code that supports the multiprocessing module ready for end-user testing, I'll be eager to install it and try it against my code, and of course I'll let you know of any problems. Sounds like it's not quite there yet, but I'll continue to follow developments. Perhaps popping up here sometimes to ask again about progress. I'm excited about the work on 64-bit and multiprocessing...

Best,
Gary

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Sep 24, 2010, at 12:56 PM, Amaury Forgeot d'Arc wrote:

> Hi,
> 2010/9/24 Gary Robinson <garyrob at me.com>:
>>> "Longer than you want"
>> 
>> Ain't that the truth! :)
>> 
>> And yet, if you felt like one of the following blocks if time was more likely than the others, I'd be very interested in knowing:
>> 
>>   < 6 months
>>   < 1 year
>>   < 2 years
> 
> I will certainly give up before this delay.
> At the moment, 1/3 of the files in the test suite pass without error,
> they were zero yesterday.
> We need volunteers to help us and implement the failing/missing parts!
> 
> -- 
> Amaury Forgeot d'Arc



From santagada at gmail.com  Fri Sep 24 20:29:14 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Fri, 24 Sep 2010 15:29:14 -0300
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <C9EB184B-A388-422F-B0E4-D7AD22B7814D@me.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
	<AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
	<E8202367-ADF8-4D05-91D9-28F471697227@me.com>
	<AANLkTinRtE=S9-HDVin6fPa4Rb4-xDYVa8Lh6vppm_yY@mail.gmail.com>
	<C9EB184B-A388-422F-B0E4-D7AD22B7814D@me.com>
Message-ID: <AANLkTikjU_HBNuNKXxULtX_=+N7r-6jWVe41pxssUS2q@mail.gmail.com>

On Fri, Sep 24, 2010 at 2:16 PM, Gary Robinson <garyrob at me.com> wrote:
> I unfortunately can't contribute code/patches at this point (I don't have the time-freedom or competence in JIT's or C++ to try),

Just to be clear, the pypy python interpreter is written in RPython,
and for most of the stuff that is missing on the fast-forward branch
you don't need to know nothing about how the JIT work. Its a pain to
learn RPython, but just to be clear on what you need to know.

Are there any tasks that you can do with pure python?


-- 
Leonardo Santagada


From garyrob at me.com  Fri Sep 24 20:43:56 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 24 Sep 2010 14:43:56 -0400
Subject: [pypy-dev] Readiness of asmgcc for x86_64 linux?
In-Reply-To: <AANLkTikjU_HBNuNKXxULtX_=+N7r-6jWVe41pxssUS2q@mail.gmail.com>
References: <740FA798-2388-47B2-BF54-5F274DED752F@me.com>
	<AANLkTinxdyNSzVaVjoVJZxr9qd5bQ-EG_p6458vP3iZH@mail.gmail.com>
	<885F1724-7CBB-4EF9-BBCE-0A81DA4A8A7D@me.com>
	<AANLkTi=3D73J7z6VygefKTmJ-GLgqkWfgS8JjBYnzUvr@mail.gmail.com>
	<AANLkTinpaj1JNHUEXM7WAbkgOZjXLJ=QEyU1UryrcZmK@mail.gmail.com>
	<E8202367-ADF8-4D05-91D9-28F471697227@me.com>
	<AANLkTinRtE=S9-HDVin6fPa4Rb4-xDYVa8Lh6vppm_yY@mail.gmail.com>
	<C9EB184B-A388-422F-B0E4-D7AD22B7814D@me.com>
	<AANLkTikjU_HBNuNKXxULtX_=+N7r-6jWVe41pxssUS2q@mail.gmail.com>
Message-ID: <3A450FE3-CDF4-428C-A8ED-AEDE92FA83FA@me.com>

> Just to be clear, the pypy python interpreter is written in RPython,
> and for most of the stuff that is missing on the fast-forward branch
> you don't need to know nothing about how the JIT work. Its a pain to
> learn RPython, but just to be clear on what you need to know.
> 
> Are there any tasks that you can do with pure python?

I would love to do it if I had the time-freedom. The information you give above tells me that I probably do have the skills -- I've written a ton of python code over the last 12 years or so. But I'm working to the point that I'm depriving my family already, because my company needs my full attention now. There will come a time when I'll be able to do it, and I suspect PyPy will still be able to benefit from code and patch contributors then. But, of course, that's easy to say. The point is that PyPy would can use more contributors right now, and I, unfortunately, can't be among them.

But I will be an active tester when it's ready for end-user testing.

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Sep 24, 2010, at 2:29 PM, Leonardo Santagada wrote:

> On Fri, Sep 24, 2010 at 2:16 PM, Gary Robinson <garyrob at me.com> wrote:
>> I unfortunately can't contribute code/patches at this point (I don't have the time-freedom or competence in JIT's or C++ to try),
> 
> Just to be clear, the pypy python interpreter is written in RPython,
> and for most of the stuff that is missing on the fast-forward branch
> you don't need to know nothing about how the JIT work. Its a pain to
> learn RPython, but just to be clear on what you need to know.
> 
> Are there any tasks that you can do with pure python?
> 
> 
> -- 
> Leonardo Santagada



From horace3d at gmail.com  Sat Sep 25 17:47:20 2010
From: horace3d at gmail.com (horace grant)
Date: Sat, 25 Sep 2010 17:47:20 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <i64vcv$ckn$1@dough.gmane.org>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
Message-ID: <AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>

i just had a (probably) silly idea. :)

if some people like rpython so much, how about writing a rpython
interpreter in rpython? wouldn't it be much easier for the jit to
optimize rpython code? couldn't jitted rpython code theoretically be
as fast as a program that got compiled to c from rpython?

hm... but i wonder if this would make sense at all. maybe if you ran
rpython code with pypy-c-jit, it already could be jitted as well as
with a special rpython interpreter? ...if there were a special rpython
interpreter, would the current jit generator have to be changed to
take advantage of the more simple language?

just curious...


On Tue, Sep 7, 2010 at 11:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Armin Rigo, 07.09.2010 10:57:
>> On Mon, Sep 6, 2010 at 8:27 PM, Saravanan Shanmugham
>>> Is there a wish list of RPython enhancements somewhere that the
>>> PyPy team might be considering?
>>> Stuff that would benefit RPython users in general.
>>
>> Again, feel free to make a fork or a branch of PyPy and try to develop
>> a version of RPython that is more suited to writing general programs
>> in.
>
> In that case, I suggest working on Shedskin or Cython instead.
>
> Stefan
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From william.leslie.ttg at gmail.com  Sun Sep 26 01:45:37 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Sun, 26 Sep 2010 09:45:37 +1000
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
Message-ID: <AANLkTiko0DJFJRfHDWjv8bRi9jwV+CNMWnjtFsx-4LoJ@mail.gmail.com>

The current JIT generator creates a tracing jit, which gives very different
performance profile to static compilation. For tight loops etc this might be
ok, but might be different for the specific use case people are interested
in (I admit I still don't know what that is).

On 26/09/2010 1:47 AM, "horace grant" <horace3d at gmail.com> wrote:

i just had a (probably) silly idea. :)

if some people like rpython so much, how about writing a rpython
interpreter in rpython? wouldn't it be much easier for the jit to
optimize rpython code? couldn't jitted rpython code theoretically be
as fast as a program that got compiled to c from rpython?

hm... but i wonder if this would make sense at all. maybe if you ran
rpython code with pypy-c-jit, it already could be jitted as well as
with a special rpython interpreter? ...if there were a special rpython
interpreter, would the current jit generator have to be changed to
take advantage of the more simple language?

just curious...



On Tue, Sep 7, 2010 at 11:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Armin Rigo, 07.09.20...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100926/f123db1d/attachment.html>

From list-sink at trainedmonkeystudios.org  Sun Sep 26 23:28:12 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Sun, 26 Sep 2010 14:28:12 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
Message-ID: <1285536492.8752.28.camel@localhost>

On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> i just had a (probably) silly idea. :)
> 
> if some people like rpython so much, how about writing a rpython
> interpreter in rpython? wouldn't it be much easier for the jit to
> optimize rpython code? couldn't jitted rpython code theoretically be
> as fast as a program that got compiled to c from rpython?
>
> hm... but i wonder if this would make sense at all. maybe if you ran
> rpython code with pypy-c-jit, it already could be jitted as well as
> with a special rpython interpreter? ...if there were a special rpython
> interpreter, would the current jit generator have to be changed to
> take advantage of the more simple language?

An excellent question at least.  

A better idea, I think, would be to ask what subset of full-python will
jit well.  What I'd really like to see is a static analyzer that can
display (e.g. by coloring names or lines) how "jit friendly" a piece of
python code is.  This would allow a programmer to get an idea of what
help the jit is going to be when running their code and, hopefully, help
people avoid tragic performance results.  Naturally, for performance
intensive code, you would still need to profile, but for a lot of uses,
simply not having catastrophically bad performance is more than enough
for a good user experience.  

With such a tool, it wouldn't really matter if the answer to "what is
faster" is RPython -- it would be whatever python language subset
happens to work well in a particular case.  I've started working on
something like this [1], but given that I'm doing a startup, I don't
have nearly the time I would need to make this useful in the near-term.

-Terrence

[1] http://github.com/terrence2/melano

> just curious...
> 
> 
> On Tue, Sep 7, 2010 at 11:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> > Armin Rigo, 07.09.2010 10:57:
> >> On Mon, Sep 6, 2010 at 8:27 PM, Saravanan Shanmugham
> >>> Is there a wish list of RPython enhancements somewhere that the
> >>> PyPy team might be considering?
> >>> Stuff that would benefit RPython users in general.
> >>
> >> Again, feel free to make a fork or a branch of PyPy and try to develop
> >> a version of RPython that is more suited to writing general programs
> >> in.
> >
> > In that case, I suggest working on Shedskin or Cython instead.
> >
> > Stefan
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev





From sarvi at yahoo.com  Mon Sep 27 08:57:16 2010
From: sarvi at yahoo.com (Saravanan Shanmugham)
Date: Sun, 26 Sep 2010 23:57:16 -0700 (PDT)
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285536492.8752.28.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
	<1285536492.8752.28.camel@localhost>
Message-ID: <827221.6704.qm@web53704.mail.re2.yahoo.com>

Well, I am happy to see that the my interest in a general purpose RPython is not 
as isolated as I was lead to believe :-))
Thx,

Sarvi


----- Original Message ----
> From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> To: pypy-dev at codespeak.net
> Sent: Sun, September 26, 2010 2:28:12 PM
> Subject: Re: [pypy-dev] Question on the future of RPython
> 
> On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> > i just had a  (probably) silly idea. :)
> > 
> > if some people like rpython so much,  how about writing a rpython
> > interpreter in rpython? wouldn't it be much  easier for the jit to
> > optimize rpython code? couldn't jitted rpython  code theoretically be
> > as fast as a program that got compiled to c from  rpython?
> >
> > hm... but i wonder if this would make sense at all.  maybe if you ran
> > rpython code with pypy-c-jit, it already could be  jitted as well as
> > with a special rpython interpreter? ...if there were a  special rpython
> > interpreter, would the current jit generator have to be  changed to
> > take advantage of the more simple language?
> 
> An  excellent question at least.  
> 
> A better idea, I think, would be to  ask what subset of full-python will
> jit well.  What I'd really like to  see is a static analyzer that can
> display (e.g. by coloring names or lines)  how "jit friendly" a piece of
> python code is.  This would allow a  programmer to get an idea of what
> help the jit is going to be when running  their code and, hopefully, help
> people avoid tragic performance  results.  Naturally, for performance
> intensive code, you would still  need to profile, but for a lot of uses,
> simply not having catastrophically  bad performance is more than enough
> for a good user experience.  
> 
> With such a tool, it wouldn't really matter if the answer to "what  is
> faster" is RPython -- it would be whatever python language  subset
> happens to work well in a particular case.  I've started working  on
> something like this [1], but given that I'm doing a startup, I  don't
> have nearly the time I would need to make this useful in the  near-term.
> 
> -Terrence
> 
> [1]  http://github.com/terrence2/melano
> 
> > just curious...
> > 
> > 
> > On Tue, Sep 7, 2010 at 11:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> >  > Armin Rigo, 07.09.2010 10:57:
> > >> On Mon, Sep 6, 2010 at 8:27  PM, Saravanan Shanmugham
> > >>> Is there a wish list of RPython  enhancements somewhere that the
> > >>> PyPy team might be  considering?
> > >>> Stuff that would benefit RPython users in  general.
> > >>
> > >> Again, feel free to make a fork or a  branch of PyPy and try to develop
> > >> a version of RPython that is  more suited to writing general programs
> > >> in.
> >  >
> > > In that case, I suggest working on Shedskin or Cython  instead.
> > >
> > > Stefan
> > >
> > >  _______________________________________________
> > > pypy-dev at codespeak.net
> > >  http://codespeak.net/mailman/listinfo/pypy-dev
> > >
> >  _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


      


From ipc at srand.net  Mon Sep 27 13:49:11 2010
From: ipc at srand.net (Ian P. Cooke)
Date: Mon, 27 Sep 2010 06:49:11 -0500
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
Message-ID: <4CA084B7.3030106@srand.net>


There was a recent thread with the same subject and I would like to look 
into this a bit more.
I knew pypy-stackless wouldn't work after I built a working 64-bit pypy 
w/ JIT, well, now I'm intrigued.

I will look at the code more closely soon.  Armin, Carl Friedrich, would 
you answer a couple of questions in the mean-time?

What is the largest roadblock to making pypy-stackless work on pypy w/ JIT?
Would it be possible/easier to port the greenlet module?

Having built-in support for co-routines would be very nice but my own 
goal is to get greenlet working in any manner.
If I could build a 64-bit pypy w/ JIT and then easy_install greenlet, 
that would work for me.

Thanks,
     Ian

P.S. congratulations on all your recent progress!  I always look forward 
for the next pypy blog update :)


From fijall at gmail.com  Mon Sep 27 14:23:14 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 27 Sep 2010 14:23:14 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <4CA084B7.3030106@srand.net>
References: <4CA084B7.3030106@srand.net>
Message-ID: <AANLkTinYC+Orf9RXfyST3ZJPX0ZM0nAXYsh=tH8PtAxK@mail.gmail.com>

Hey.

greenlet C module is quite incompatible with pypy and won't work.
However making pypy work with jit and stackless is something that
requires a bit of work only (teaching jit how to unroll the stack
mostly) and I plan to look into it in the very near future.

Cheers,
fijal

On Mon, Sep 27, 2010 at 1:49 PM, Ian P. Cooke <ipc at srand.net> wrote:
>
> There was a recent thread with the same subject and I would like to look
> into this a bit more.
> I knew pypy-stackless wouldn't work after I built a working 64-bit pypy
> w/ JIT, well, now I'm intrigued.
>
> I will look at the code more closely soon. ?Armin, Carl Friedrich, would
> you answer a couple of questions in the mean-time?
>
> What is the largest roadblock to making pypy-stackless work on pypy w/ JIT?
> Would it be possible/easier to port the greenlet module?
>
> Having built-in support for co-routines would be very nice but my own
> goal is to get greenlet working in any manner.
> If I could build a 64-bit pypy w/ JIT and then easy_install greenlet,
> that would work for me.
>
> Thanks,
> ? ? Ian
>
> P.S. congratulations on all your recent progress! ?I always look forward
> for the next pypy blog update :)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From angelflow at yahoo.com  Mon Sep 27 14:29:49 2010
From: angelflow at yahoo.com (Andy)
Date: Mon, 27 Sep 2010 05:29:49 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <AANLkTinYC+Orf9RXfyST3ZJPX0ZM0nAXYsh=tH8PtAxK@mail.gmail.com>
Message-ID: <187982.37806.qm@web111302.mail.gq1.yahoo.com>

Why wouldn't pypy work with greenlet but would work with Stackless? greenlet calls itself a spin-off of Stackless. Isn't greenlet a subset of Stackless without the scheduling? Could you explain a bit more?

Thanks.

--- On Mon, 9/27/10, Maciej Fijalkowski <fijall at gmail.com> wrote:

> From: Maciej Fijalkowski <fijall at gmail.com>
> Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet
> To: "Ian P. Cooke" <ipc at srand.net>
> Cc: pypy-dev at codespeak.net
> Date: Monday, September 27, 2010, 8:23 AM
> Hey.
> 
> greenlet C module is quite incompatible with pypy and won't
> work.
> However making pypy work with jit and stackless is
> something that
> requires a bit of work only (teaching jit how to unroll the
> stack
> mostly) and I plan to look into it in the very near
> future.
> 
> Cheers,
> fijal
> 
> On Mon, Sep 27, 2010 at 1:49 PM, Ian P. Cooke <ipc at srand.net>
> wrote:
> >
> > There was a recent thread with the same subject and I
> would like to look
> > into this a bit more.
> > I knew pypy-stackless wouldn't work after I built a
> working 64-bit pypy
> > w/ JIT, well, now I'm intrigued.
> >
> > I will look at the code more closely soon. ?Armin,
> Carl Friedrich, would
> > you answer a couple of questions in the mean-time?
> >
> > What is the largest roadblock to making pypy-stackless
> work on pypy w/ JIT?
> > Would it be possible/easier to port the greenlet
> module?
> >
> > Having built-in support for co-routines would be very
> nice but my own
> > goal is to get greenlet working in any manner.
> > If I could build a 64-bit pypy w/ JIT and then
> easy_install greenlet,
> > that would work for me.
> >
> > Thanks,
> > ? ? Ian
> >
> > P.S. congratulations on all your recent progress! ?I
> always look forward
> > for the next pypy blog update :)
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


      


From fijall at gmail.com  Mon Sep 27 14:30:50 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 27 Sep 2010 14:30:50 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <187982.37806.qm@web111302.mail.gq1.yahoo.com>
References: <AANLkTinYC+Orf9RXfyST3ZJPX0ZM0nAXYsh=tH8PtAxK@mail.gmail.com>
	<187982.37806.qm@web111302.mail.gq1.yahoo.com>
Message-ID: <AANLkTimuaU+2JiCGN4yyOHptcHQHbWKk5ViaMgn1JZj=@mail.gmail.com>

PyPy stackless does support greenlets. PyPy would not work with
CPython C module called "greenlets".

On Mon, Sep 27, 2010 at 2:29 PM, Andy <angelflow at yahoo.com> wrote:
> Why wouldn't pypy work with greenlet but would work with Stackless? greenlet calls itself a spin-off of Stackless. Isn't greenlet a subset of Stackless without the scheduling? Could you explain a bit more?
>
> Thanks.
>
> --- On Mon, 9/27/10, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> From: Maciej Fijalkowski <fijall at gmail.com>
>> Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet
>> To: "Ian P. Cooke" <ipc at srand.net>
>> Cc: pypy-dev at codespeak.net
>> Date: Monday, September 27, 2010, 8:23 AM
>> Hey.
>>
>> greenlet C module is quite incompatible with pypy and won't
>> work.
>> However making pypy work with jit and stackless is
>> something that
>> requires a bit of work only (teaching jit how to unroll the
>> stack
>> mostly) and I plan to look into it in the very near
>> future.
>>
>> Cheers,
>> fijal
>>
>> On Mon, Sep 27, 2010 at 1:49 PM, Ian P. Cooke <ipc at srand.net>
>> wrote:
>> >
>> > There was a recent thread with the same subject and I
>> would like to look
>> > into this a bit more.
>> > I knew pypy-stackless wouldn't work after I built a
>> working 64-bit pypy
>> > w/ JIT, well, now I'm intrigued.
>> >
>> > I will look at the code more closely soon. ?Armin,
>> Carl Friedrich, would
>> > you answer a couple of questions in the mean-time?
>> >
>> > What is the largest roadblock to making pypy-stackless
>> work on pypy w/ JIT?
>> > Would it be possible/easier to port the greenlet
>> module?
>> >
>> > Having built-in support for co-routines would be very
>> nice but my own
>> > goal is to get greenlet working in any manner.
>> > If I could build a 64-bit pypy w/ JIT and then
>> easy_install greenlet,
>> > that would work for me.
>> >
>> > Thanks,
>> > ? ? Ian
>> >
>> > P.S. congratulations on all your recent progress! ?I
>> always look forward
>> > for the next pypy blog update :)
>> > _______________________________________________
>> > pypy-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/pypy-dev
>> >
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
>


From list-sink at trainedmonkeystudios.org  Mon Sep 27 21:44:51 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Mon, 27 Sep 2010 12:44:51 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <827221.6704.qm@web53704.mail.re2.yahoo.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
	<1285536492.8752.28.camel@localhost>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
Message-ID: <1285616691.5954.34.camel@localhost>

On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> Well, I am happy to see that the my interest in a general purpose RPython is not 
> as isolated as I was lead to believe :-))
> Thx,

What I wrote has apparently been widely misunderstood, so let me explain
what I mean in more detail.  What I want is _not_ RPython and it is
_not_ Shedskin.  What I want is not a compiler at all.  What I want is a
visual tool, for example, a plugin to an IDE.  This tool would perform
static analysis on a piece of python code.  Instead of generating code
with this information, it would mark up the python code in the text
display with colors, weights, etc in order to show properties from the
static analysis.  This would be something like semantic highlighting, as
opposed to syntax highlighting.  

I think it possible that this information would, if created and
presented in the correct way, represent the sort of optimizations that
pypy-c-jit -- a full python implementation, not a language subset --
would likely perform on the code if run.  Given this sort of feedback,
it would be much easier for a python coder to write code that works well
with the jit: for example, moving a declaration inside a loop to avoid
boxing, based on the information presented.

Ideally, such a tool would perform instantaneous syntax highlighting
while editing and do full parsing and analysis in the background to
update the semantic highlighting as frequently as possible.  Obviously,
detailed static analysis will provide far more information than it would
be possible to display on the code at once, so I see this gui as having
several modes -- like predator vision -- that show different information
from the analysis.  Naturally, what those modes are will depend strongly
on the details of how pypy-c-jit works internally, what sort of
information can be sanely collected through static analysis, and,
naturally, user testing.

I was somewhat baffled at first as to how what I wrote before was
interpreted as interest in a static python.  I think the disconnect here
is the assumption on many people's part that a static language will
always be faster than a dynamic one.  Given the existing tools that
provide basically no feedback from the compiler / interpreter / jitter,
this is inevitably true at the moment.  I foresee a future, however,
where better tools let us use the full power of a dynamic python AND let
us tighten up our code for speed to get the full advantages of jit
compilation as well.  I believe that in the end, this combination will
prove superior to any fully static compiler.

-Terrence


> Sarvi
> 
> 
> ----- Original Message ----
> > From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> > To: pypy-dev at codespeak.net
> > Sent: Sun, September 26, 2010 2:28:12 PM
> > Subject: Re: [pypy-dev] Question on the future of RPython
> > 
> > On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> > > i just had a  (probably) silly idea. :)
> > > 
> > > if some people like rpython so much,  how about writing a rpython
> > > interpreter in rpython? wouldn't it be much  easier for the jit to
> > > optimize rpython code? couldn't jitted rpython  code theoretically be
> > > as fast as a program that got compiled to c from  rpython?
> > >
> > > hm... but i wonder if this would make sense at all.  maybe if you ran
> > > rpython code with pypy-c-jit, it already could be  jitted as well as
> > > with a special rpython interpreter? ...if there were a  special rpython
> > > interpreter, would the current jit generator have to be  changed to
> > > take advantage of the more simple language?
> > 
> > An  excellent question at least.  
> > 
> > A better idea, I think, would be to  ask what subset of full-python will
> > jit well.  What I'd really like to  see is a static analyzer that can
> > display (e.g. by coloring names or lines)  how "jit friendly" a piece of
> > python code is.  This would allow a  programmer to get an idea of what
> > help the jit is going to be when running  their code and, hopefully, help
> > people avoid tragic performance  results.  Naturally, for performance
> > intensive code, you would still  need to profile, but for a lot of uses,
> > simply not having catastrophically  bad performance is more than enough
> > for a good user experience.  
> > 
> > With such a tool, it wouldn't really matter if the answer to "what  is
> > faster" is RPython -- it would be whatever python language  subset
> > happens to work well in a particular case.  I've started working  on
> > something like this [1], but given that I'm doing a startup, I  don't
> > have nearly the time I would need to make this useful in the  near-term.
> > 
> > -Terrence
> > 
> > [1]  http://github.com/terrence2/melano
> > 
> > > just curious...
> > > 
> > > 
> > > On Tue, Sep 7, 2010 at 11:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> > >  > Armin Rigo, 07.09.2010 10:57:
> > > >> On Mon, Sep 6, 2010 at 8:27  PM, Saravanan Shanmugham
> > > >>> Is there a wish list of RPython  enhancements somewhere that the
> > > >>> PyPy team might be  considering?
> > > >>> Stuff that would benefit RPython users in  general.
> > > >>
> > > >> Again, feel free to make a fork or a  branch of PyPy and try to develop
> > > >> a version of RPython that is  more suited to writing general programs
> > > >> in.
> > >  >
> > > > In that case, I suggest working on Shedskin or Cython  instead.
> > > >
> > > > Stefan
> > > >
> > > >  _______________________________________________
> > > > pypy-dev at codespeak.net
> > > >  http://codespeak.net/mailman/listinfo/pypy-dev
> > > >
> > >  _______________________________________________
> > > pypy-dev at codespeak.net
> > > http://codespeak.net/mailman/listinfo/pypy-dev
> > 
> > 
> > 
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> > 
> 
> 
>       




From santagada at gmail.com  Mon Sep 27 21:58:51 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Mon, 27 Sep 2010 16:58:51 -0300
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285616691.5954.34.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
	<1285536492.8752.28.camel@localhost>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
Message-ID: <AANLkTi=_AGr5omqTH9MpEc7phWS7qmVB45cct_qvvz0B@mail.gmail.com>

On Mon, Sep 27, 2010 at 4:44 PM, Terrence Cole
<list-sink at trainedmonkeystudios.org> wrote:
> On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
>> Well, I am happy to see that the my interest in a general purpose RPython is not
>> as isolated as I was lead to believe :-))
>> Thx,
>
> What I wrote has apparently been widely misunderstood, so let me explain
> what I mean in more detail. ?What I want is _not_ RPython and it is
> _not_ Shedskin. ?What I want is not a compiler at all. ?What I want is a
> visual tool, for example, a plugin to an IDE. ?This tool would perform
> static analysis on a piece of python code. ?Instead of generating code
> with this information, it would mark up the python code in the text
> display with colors, weights, etc in order to show properties from the
> static analysis. ?This would be something like semantic highlighting, as
> opposed to syntax highlighting.
>
> I think it possible that this information would, if created and
> presented in the correct way, represent the sort of optimizations that
> pypy-c-jit -- a full python implementation, not a language subset --
> would likely perform on the code if run. ?Given this sort of feedback,
> it would be much easier for a python coder to write code that works well
> with the jit: for example, moving a declaration inside a loop to avoid
> boxing, based on the information presented.
>
> Ideally, such a tool would perform instantaneous syntax highlighting
> while editing and do full parsing and analysis in the background to
> update the semantic highlighting as frequently as possible. ?Obviously,
> detailed static analysis will provide far more information than it would
> be possible to display on the code at once, so I see this gui as having
> several modes -- like predator vision -- that show different information
> from the analysis. ?Naturally, what those modes are will depend strongly
> on the details of how pypy-c-jit works internally, what sort of
> information can be sanely collected through static analysis, and,
> naturally, user testing.
>
> I was somewhat baffled at first as to how what I wrote before was
> interpreted as interest in a static python. ?I think the disconnect here
> is the assumption on many people's part that a static language will
> always be faster than a dynamic one. ?Given the existing tools that
> provide basically no feedback from the compiler / interpreter / jitter,
> this is inevitably true at the moment. ?I foresee a future, however,
> where better tools let us use the full power of a dynamic python AND let
> us tighten up our code for speed to get the full advantages of jit
> compilation as well. ?I believe that in the end, this combination will
> prove superior to any fully static compiler.

This all looks interesting, and if you can plug that on emacs or
textmate I would be really happy, but it is not what I want. I would
settle for a tool that generates at runtime information about what the
jit is doing in a simple text format (json, yaml or something even
simpler?) and a tool to visualize this so you can optimize python
programs to run on pypy easily. The biggest difference is that just
collecting this info from the JIT appears to be much much easier than
somehow implement a static processor for python code that do some form
of analysis.

I think that fijal is at least thinking about doing such a tool right?

-- 
Leonardo Santagada


From p.giarrusso at gmail.com  Tue Sep 28 00:52:48 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 28 Sep 2010 00:52:48 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTi=_AGr5omqTH9MpEc7phWS7qmVB45cct_qvvz0B@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
	<1285536492.8752.28.camel@localhost>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<AANLkTi=_AGr5omqTH9MpEc7phWS7qmVB45cct_qvvz0B@mail.gmail.com>
Message-ID: <AANLkTi=q=MacCxdQQ17DGuxViVugBQvnb76vFS=Myn_F@mail.gmail.com>

On Mon, Sep 27, 2010 at 21:58, Leonardo Santagada <santagada at gmail.com> wrote:
> On Mon, Sep 27, 2010 at 4:44 PM, Terrence Cole
> <list-sink at trainedmonkeystudios.org> wrote:
>> On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
>>> Well, I am happy to see that the my interest in a general purpose RPython is not
>>> as isolated as I was lead to believe :-))
>>> Thx,
>>
>> What I wrote has apparently been widely misunderstood, so let me explain
>> what I mean in more detail. ?What I want is _not_ RPython and it is
>> _not_ Shedskin. ?What I want is not a compiler at all. ?What I want is a
>> visual tool, for example, a plugin to an IDE. ?This tool would perform
>> static analysis on a piece of python code. ?Instead of generating code
>> with this information, it would mark up the python code in the text
>> display with colors, weights, etc in order to show properties from the
>> static analysis. ?This would be something like semantic highlighting, as
>> opposed to syntax highlighting.
>>
>> I think it possible that this information would, if created and
>> presented in the correct way, represent the sort of optimizations that
>> pypy-c-jit -- a full python implementation, not a language subset --
>> would likely perform on the code if run. ?Given this sort of feedback,
>> it would be much easier for a python coder to write code that works well
>> with the jit: for example, moving a declaration inside a loop to avoid
>> boxing, based on the information presented.
>>
>> Ideally, such a tool would perform instantaneous syntax highlighting
>> while editing and do full parsing and analysis in the background to
>> update the semantic highlighting as frequently as possible. ?Obviously,
>> detailed static analysis will provide far more information than it would
>> be possible to display on the code at once, so I see this gui as having
>> several modes -- like predator vision -- that show different information
>> from the analysis. ?Naturally, what those modes are will depend strongly
>> on the details of how pypy-c-jit works internally, what sort of
>> information can be sanely collected through static analysis, and,
>> naturally, user testing.
>>
>> I was somewhat baffled at first as to how what I wrote before was
>> interpreted as interest in a static python. ?I think the disconnect here
>> is the assumption on many people's part that a static language will
>> always be faster than a dynamic one. ?Given the existing tools that
>> provide basically no feedback from the compiler / interpreter / jitter,
>> this is inevitably true at the moment. ?I foresee a future, however,
>> where better tools let us use the full power of a dynamic python AND let
>> us tighten up our code for speed to get the full advantages of jit
>> compilation as well. ?I believe that in the end, this combination will
>> prove superior to any fully static compiler.
>
> This all looks interesting, and if you can plug that on emacs or
> textmate I would be really happy, but it is not what I want. I would
> settle for a tool that generates at runtime information about what the
> jit is doing in a simple text format (json, yaml or something even
> simpler?) and a tool to visualize this so you can optimize python
> programs to run on pypy easily. The biggest difference is that just
> collecting this info from the JIT appears to be much much easier than
> somehow implement a static processor for python code that do some form
> of analysis.

Have you looked at what the Azul Java VM supports for Java, in
particular RTPM (Real Time Performance Monitoring)?

Academic accounts are available, and from Cliff Click's presentations,
it seems to be a production-quality solution for this (for Java),
which could give interesting ideas. Azul business is exclusively
centered around Java optimization at the JVM level, so while
not-so-famous they are quite relevant.

See slide 28 of: www.azulsystems.com/events/vee_2009/2009_VEE.pdf for
some more details.
See also wiki.jvmlangsummit.com/pdf/36_Click_fastbcs.pdf, and the
account about JRuby's slowness (caused by unreliable performance
analysis tools).

Given that JIT can beat static compilation only through forms of
profile-directed optimization, I also believe that the interesting
information should be obtained through logs from the JIT. A static
analyser can't do something better than a static compiler - not
reliably at least.

_However_, static semantic highlighting might still be interesting:
while it does not help understanding profile-directed optimizations
done by the JIT, it might help understanding the consequences of the
execution model of the language itself, where it has a weird impact on
performance.
E.g., for CPython, it might be very useful simply highlighting usages
of global variables, that require a dict lookup, as "bad", especially
in tight loops. OTOH, that kind of optimization should be done by a
JIT like PyPy, not by the programmer.
I believe that CALL_LIKELY_BUILTIN and hidden classes already allow
PyPy to fix the problem without changing the source code.

The question then is: which kinds of constructs are unexpectedly slow
in Python, even with a good JIT?

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From jacob at openend.se  Tue Sep 28 01:57:11 2010
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Tue, 28 Sep 2010 01:57:11 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285616691.5954.34.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
Message-ID: <201009280157.17264.jacob@openend.se>

Monday 27 September 2010 you wrote:
> On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> > Well, I am happy to see that the my interest in a general purpose RPython
> > is not as isolated as I was lead to believe :-))
> > Thx,
> 
> What I wrote has apparently been widely misunderstood, so let me explain
> what I mean in more detail.  What I want is _not_ RPython and it is
> _not_ Shedskin.  What I want is not a compiler at all.  What I want is a
> visual tool, for example, a plugin to an IDE.  This tool would perform
> static analysis on a piece of python code.  Instead of generating code
> with this information, it would mark up the python code in the text
> display with colors, weights, etc in order to show properties from the
> static analysis.  This would be something like semantic highlighting, as
> opposed to syntax highlighting.
> 
> I think it possible that this information would, if created and
> presented in the correct way, represent the sort of optimizations that
> pypy-c-jit -- a full python implementation, not a language subset --
> would likely perform on the code if run.  Given this sort of feedback,
> it would be much easier for a python coder to write code that works well
> with the jit: for example, moving a declaration inside a loop to avoid
> boxing, based on the information presented.
> 
> Ideally, such a tool would perform instantaneous syntax highlighting
> while editing and do full parsing and analysis in the background to
> update the semantic highlighting as frequently as possible.  Obviously,
> detailed static analysis will provide far more information than it would
> be possible to display on the code at once, so I see this gui as having
> several modes -- like predator vision -- that show different information
> from the analysis.  Naturally, what those modes are will depend strongly
> on the details of how pypy-c-jit works internally, what sort of
> information can be sanely collected through static analysis, and,
> naturally, user testing.
> 
> I was somewhat baffled at first as to how what I wrote before was
> interpreted as interest in a static python.  I think the disconnect here
> is the assumption on many people's part that a static language will
> always be faster than a dynamic one.  Given the existing tools that
> provide basically no feedback from the compiler / interpreter / jitter,
> this is inevitably true at the moment.  I foresee a future, however,
> where better tools let us use the full power of a dynamic python AND let
> us tighten up our code for speed to get the full advantages of jit
> compilation as well.  I believe that in the end, this combination will
> prove superior to any fully static compiler.
> 
> -Terrence
> 
> > Sarvi
> > 
> > 
> > ----- Original Message ----
> > 
> > > From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> > > To: pypy-dev at codespeak.net
> > > Sent: Sun, September 26, 2010 2:28:12 PM
> > > Subject: Re: [pypy-dev] Question on the future of RPython
> > > 
> > > On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> > > > i just had a  (probably) silly idea. :)
> > > > 
> > > > if some people like rpython so much,  how about writing a rpython
> > > > interpreter in rpython? wouldn't it be much  easier for the jit to
> > > > optimize rpython code? couldn't jitted rpython  code theoretically be
> > > > as fast as a program that got compiled to c from  rpython?
> > > > 
> > > > hm... but i wonder if this would make sense at all.  maybe if you ran
> > > > rpython code with pypy-c-jit, it already could be  jitted as well as
> > > > with a special rpython interpreter? ...if there were a  special
> > > > rpython interpreter, would the current jit generator have to be 
> > > > changed to take advantage of the more simple language?
> > > 
> > > An  excellent question at least.
> > > 
> > > A better idea, I think, would be to  ask what subset of full-python
> > > will jit well.  What I'd really like to  see is a static analyzer that
> > > can display (e.g. by coloring names or lines)  how "jit friendly" a
> > > piece of python code is.  This would allow a  programmer to get an
> > > idea of what help the jit is going to be when running  their code and,
> > > hopefully, help people avoid tragic performance  results.  Naturally,
> > > for performance intensive code, you would still  need to profile, but
> > > for a lot of uses, simply not having catastrophically  bad performance
> > > is more than enough for a good user experience.
> > > 
> > > With such a tool, it wouldn't really matter if the answer to "what  is
> > > faster" is RPython -- it would be whatever python language  subset
> > > happens to work well in a particular case.  I've started working  on
> > > something like this [1], but given that I'm doing a startup, I  don't
> > > have nearly the time I would need to make this useful in the 
> > > near-term.

The JIT works because it has more information at runtime than what is 
available at compile time. If the information was available at compile time we 
could do the optimizations then and not have to invoke the extra complexity 
required by the JIT. Examples of  the extra information include things like 
knowing that introspection will not be used in the current evaluation of a 
loop, specific argument types will be used in calls and that some arguments 
will be known to be constant over part of the program execution.. Knowing 
these bits allows you to optimize away large chunks o f the code that 
otherwise would have been executed.

Static analysis assumes that none of the above mentioned possibilities can 
actually take place. It is impossible to make such assumptions at compile time 
in a dynamic language. Therefore PyPy is a bad match for people wanting to 
staically compile subsets of Python. Applying the JIT to RPython code is not 
workable, because the JIT is optimized to remove bits of generated assembler 
code that never shows up in the compilation of RPython code.

These are very basic first principle concepts, and it is a mystery to me why 
people can't work them out for themselves.

Jacob Hall?n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20100928/d1212fae/attachment.pgp>

From list-sink at trainedmonkeystudios.org  Tue Sep 28 02:03:08 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Mon, 27 Sep 2010 17:03:08 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTi=q=MacCxdQQ17DGuxViVugBQvnb76vFS=Myn_F@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<201009021140.53415.jacob@openend.se>
	<933850.84117.qm@web53705.mail.re2.yahoo.com>
	<201009022254.07665.jacob@openend.se>
	<810929.87945.qm@web53702.mail.re2.yahoo.com>
	<i5qf7p$g4h$2@dough.gmane.org>
	<609691.54456.qm@web53705.mail.re2.yahoo.com>
	<i5rcl9$ip2$1@dough.gmane.org>
	<294101.49986.qm@web53704.mail.re2.yahoo.com>
	<AANLkTine1vca4HLALdELzSK2aeL-zQLCnx8g5iyV3mA_@mail.gmail.com>
	<860977.83657.qm@web53705.mail.re2.yahoo.com>
	<AANLkTinrJBsRwBaRj_wH34sFM8bpjiXiryoQMdXUqrps@mail.gmail.com>
	<792921.22517.qm@web53703.mail.re2.yahoo.com>
	<AANLkTimfHBcBi_YnLQ1Ya2m12OBT2sfLgyyH_-cu6fXf@mail.gmail.com>
	<i64vcv$ckn$1@dough.gmane.org>
	<AANLkTinrdmpFsfmiv_Ai=8vhm1oB=_dXznnL_iDyzj6n@mail.gmail.com>
	<1285536492.8752.28.camel@localhost>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<AANLkTi=_AGr5omqTH9MpEc7phWS7qmVB45cct_qvvz0B@mail.gmail.com>
	<AANLkTi=q=MacCxdQQ17DGuxViVugBQvnb76vFS=Myn_F@mail.gmail.com>
Message-ID: <1285632188.5954.94.camel@localhost>

On Tue, 2010-09-28 at 00:52 +0200, Paolo Giarrusso wrote:
> On Mon, Sep 27, 2010 at 21:58, Leonardo Santagada <santagada at gmail.com> wrote:
> > On Mon, Sep 27, 2010 at 4:44 PM, Terrence Cole
> > <list-sink at trainedmonkeystudios.org> wrote:
> >> On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> >>> Well, I am happy to see that the my interest in a general purpose RPython is not
> >>> as isolated as I was lead to believe :-))
> >>> Thx,
> >>
> >> What I wrote has apparently been widely misunderstood, so let me explain
> >> what I mean in more detail.  What I want is _not_ RPython and it is
> >> _not_ Shedskin.  What I want is not a compiler at all.  What I want is a
> >> visual tool, for example, a plugin to an IDE.  This tool would perform
> >> static analysis on a piece of python code.  Instead of generating code
> >> with this information, it would mark up the python code in the text
> >> display with colors, weights, etc in order to show properties from the
> >> static analysis.  This would be something like semantic highlighting, as
> >> opposed to syntax highlighting.
> >>
> >> I think it possible that this information would, if created and
> >> presented in the correct way, represent the sort of optimizations that
> >> pypy-c-jit -- a full python implementation, not a language subset --
> >> would likely perform on the code if run.  Given this sort of feedback,
> >> it would be much easier for a python coder to write code that works well
> >> with the jit: for example, moving a declaration inside a loop to avoid
> >> boxing, based on the information presented.
> >>
> >> Ideally, such a tool would perform instantaneous syntax highlighting
> >> while editing and do full parsing and analysis in the background to
> >> update the semantic highlighting as frequently as possible.  Obviously,
> >> detailed static analysis will provide far more information than it would
> >> be possible to display on the code at once, so I see this gui as having
> >> several modes -- like predator vision -- that show different information
> >> from the analysis.  Naturally, what those modes are will depend strongly
> >> on the details of how pypy-c-jit works internally, what sort of
> >> information can be sanely collected through static analysis, and,
> >> naturally, user testing.
> >>
> >> I was somewhat baffled at first as to how what I wrote before was
> >> interpreted as interest in a static python.  I think the disconnect here
> >> is the assumption on many people's part that a static language will
> >> always be faster than a dynamic one.  Given the existing tools that
> >> provide basically no feedback from the compiler / interpreter / jitter,
> >> this is inevitably true at the moment.  I foresee a future, however,
> >> where better tools let us use the full power of a dynamic python AND let
> >> us tighten up our code for speed to get the full advantages of jit
> >> compilation as well.  I believe that in the end, this combination will
> >> prove superior to any fully static compiler.
> >
> > This all looks interesting, and if you can plug that on emacs or
> > textmate I would be really happy, but it is not what I want. I would
> > settle for a tool that generates at runtime information about what the
> > jit is doing in a simple text format (json, yaml or something even
> > simpler?) and a tool to visualize this so you can optimize python
> > programs to run on pypy easily. The biggest difference is that just
> > collecting this info from the JIT appears to be much much easier than
> > somehow implement a static processor for python code that do some form
> > of analysis.
> 
> Have you looked at what the Azul Java VM supports for Java, in
> particular RTPM (Real Time Performance Monitoring)?

Briefly, but it's not open source, and it's a Java thing, so it didn't
pique my interest significantly.

> Academic accounts are available, and from Cliff Click's presentations,
> it seems to be a production-quality solution for this (for Java),
> which could give interesting ideas. Azul business is exclusively
> centered around Java optimization at the JVM level, so while
> not-so-famous they are quite relevant.
> 
> See slide 28 of: www.azulsystems.com/events/vee_2009/2009_VEE.pdf for
> some more details.
> See also wiki.jvmlangsummit.com/pdf/36_Click_fastbcs.pdf, and the
> account about JRuby's slowness (caused by unreliable performance
> analysis tools).
> 
> Given that JIT can beat static compilation only through forms of
> profile-directed optimization, I also believe that the interesting
> information should be obtained through logs from the JIT. A static
> analyser can't do something better than a static compiler - not
> reliably at least.

I'd be pursuing the jit logging approach much more aggressively if I
cared at all about Python2 anymore.  All of the source I care about
analyzing is in Python3.  However, considering the rate I'm going, pypy
will doubtless support python3 by the time I get a half-way descent
static analyzer working anyway, so it's probably worth considering.

> _However_, static semantic highlighting might still be interesting:
> while it does not help understanding profile-directed optimizations
> done by the JIT, it might help understanding the consequences of the
> execution model of the language itself, where it has a weird impact on
> performance.
> E.g., for CPython, it might be very useful simply highlighting usages
> of global variables, that require a dict lookup, as "bad", especially
> in tight loops. OTOH, that kind of optimization should be done by a
> JIT like PyPy, not by the programmer.
> I believe that CALL_LIKELY_BUILTIN and hidden classes already allow
> PyPy to fix the problem without changing the source code.
> 
> The question then is: which kinds of constructs are unexpectedly slow
> in Python, even with a good JIT?

Precisely.  I'd love a good answer to that question.

In addition to jitting, although it would not technically be python
anymore, I see a place for something like SPUR or Jaegermonkey --
combined compilation and jitting.  Naturally, the performance of such a
beast over a jit alone would be dependent on how much boxing the
compiler could remove.  My goal for this work is about half geared
towards answering that single question, just so I'll know if I should
stop dreaming about python eventually having performance parity with C/C
++.

I tend to think that having a solid (if never perfect) static analyzer
for python could help in many areas.   I had thought that helping coders
help the jit out would be a good first use, but as you say, there will
be problems with that.  Regardless, my hope is that a library for static
analysis of python will be more generally useful than my own
hare-brained schemes.

In any case, I'm working on this in the form of a code editor first
because, regardless of what the answer to the previous question is, I
know from experience that highlighting for python like what
SourceInsight does for C++ will be extremely useful. 


Thank you for the kind feedback, your comments are much appreciated.
-Terrence

> Best regards




From list-sink at trainedmonkeystudios.org  Tue Sep 28 02:43:33 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Mon, 27 Sep 2010 17:43:33 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <201009280157.17264.jacob@openend.se>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
Message-ID: <1285634613.5954.112.camel@localhost>

On Tue, 2010-09-28 at 01:57 +0200, Jacob Hall?n wrote:
> Monday 27 September 2010 you wrote:
> > On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> > > Well, I am happy to see that the my interest in a general purpose RPython
> > > is not as isolated as I was lead to believe :-))
> > > Thx,
> > 
> > What I wrote has apparently been widely misunderstood, so let me explain
> > what I mean in more detail.  What I want is _not_ RPython and it is
> > _not_ Shedskin.  What I want is not a compiler at all.  What I want is a
> > visual tool, for example, a plugin to an IDE.  This tool would perform
> > static analysis on a piece of python code.  Instead of generating code
> > with this information, it would mark up the python code in the text
> > display with colors, weights, etc in order to show properties from the
> > static analysis.  This would be something like semantic highlighting, as
> > opposed to syntax highlighting.
> > 
> > I think it possible that this information would, if created and
> > presented in the correct way, represent the sort of optimizations that
> > pypy-c-jit -- a full python implementation, not a language subset --
> > would likely perform on the code if run.  Given this sort of feedback,
> > it would be much easier for a python coder to write code that works well
> > with the jit: for example, moving a declaration inside a loop to avoid
> > boxing, based on the information presented.
> > 
> > Ideally, such a tool would perform instantaneous syntax highlighting
> > while editing and do full parsing and analysis in the background to
> > update the semantic highlighting as frequently as possible.  Obviously,
> > detailed static analysis will provide far more information than it would
> > be possible to display on the code at once, so I see this gui as having
> > several modes -- like predator vision -- that show different information
> > from the analysis.  Naturally, what those modes are will depend strongly
> > on the details of how pypy-c-jit works internally, what sort of
> > information can be sanely collected through static analysis, and,
> > naturally, user testing.
> > 
> > I was somewhat baffled at first as to how what I wrote before was
> > interpreted as interest in a static python.  I think the disconnect here
> > is the assumption on many people's part that a static language will
> > always be faster than a dynamic one.  Given the existing tools that
> > provide basically no feedback from the compiler / interpreter / jitter,
> > this is inevitably true at the moment.  I foresee a future, however,
> > where better tools let us use the full power of a dynamic python AND let
> > us tighten up our code for speed to get the full advantages of jit
> > compilation as well.  I believe that in the end, this combination will
> > prove superior to any fully static compiler.
> > 
> > -Terrence
> > 
> > > Sarvi
> > > 
> > > 
> > > ----- Original Message ----
> > > 
> > > > From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> > > > To: pypy-dev at codespeak.net
> > > > Sent: Sun, September 26, 2010 2:28:12 PM
> > > > Subject: Re: [pypy-dev] Question on the future of RPython
> > > > 
> > > > On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> > > > > i just had a  (probably) silly idea. :)
> > > > > 
> > > > > if some people like rpython so much,  how about writing a rpython
> > > > > interpreter in rpython? wouldn't it be much  easier for the jit to
> > > > > optimize rpython code? couldn't jitted rpython  code theoretically be
> > > > > as fast as a program that got compiled to c from  rpython?
> > > > > 
> > > > > hm... but i wonder if this would make sense at all.  maybe if you ran
> > > > > rpython code with pypy-c-jit, it already could be  jitted as well as
> > > > > with a special rpython interpreter? ...if there were a  special
> > > > > rpython interpreter, would the current jit generator have to be 
> > > > > changed to take advantage of the more simple language?
> > > > 
> > > > An  excellent question at least.
> > > > 
> > > > A better idea, I think, would be to  ask what subset of full-python
> > > > will jit well.  What I'd really like to  see is a static analyzer that
> > > > can display (e.g. by coloring names or lines)  how "jit friendly" a
> > > > piece of python code is.  This would allow a  programmer to get an
> > > > idea of what help the jit is going to be when running  their code and,
> > > > hopefully, help people avoid tragic performance  results.  Naturally,
> > > > for performance intensive code, you would still  need to profile, but
> > > > for a lot of uses, simply not having catastrophically  bad performance
> > > > is more than enough for a good user experience.
> > > > 
> > > > With such a tool, it wouldn't really matter if the answer to "what  is
> > > > faster" is RPython -- it would be whatever python language  subset
> > > > happens to work well in a particular case.  I've started working  on
> > > > something like this [1], but given that I'm doing a startup, I  don't
> > > > have nearly the time I would need to make this useful in the 
> > > > near-term.
> 
> The JIT works because it has more information at runtime than what is 
> available at compile time. If the information was available at compile time we 
> could do the optimizations then and not have to invoke the extra complexity 
> required by the JIT. Examples of  the extra information include things like 
> knowing that introspection will not be used in the current evaluation of a 
> loop, specific argument types will be used in calls and that some arguments 
> will be known to be constant over part of the program execution.. Knowing 
> these bits allows you to optimize away large chunks o f the code that 
> otherwise would have been executed.
> 
> Static analysis assumes that none of the above mentioned possibilities can 
> actually take place. It is impossible to make such assumptions at compile time 
> in a dynamic language. Therefore PyPy is a bad match for people wanting to 
> staically compile subsets of Python. Applying the JIT to RPython code

Yes, that idea is just dumb.  It's also not what I suggested at all.  I
can see now that what I said would be easy to misinterpret, but on
re-reading it, it clearly doesn't say what you think it does.

>  is not 
> workable, because the JIT is optimized to remove bits of generated assembler 
> code that never shows up in the compilation of RPython code.
> 
> These are very basic first principle concepts, and it is a mystery to me why 
> people can't work them out for themselves.

You are quite right that static analysis will be able to do little to
help an optimal jit.  However, I doubt that in the near term pypy's jit
will cover all the dark corners of python equally well -- C has been
around for 38 years and its still got room for optimization.

-Terrence

> Jacob Hall?n




From william.leslie.ttg at gmail.com  Tue Sep 28 03:55:06 2010
From: william.leslie.ttg at gmail.com (William Leslie)
Date: Tue, 28 Sep 2010 11:55:06 +1000
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285634613.5954.112.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
Message-ID: <AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>

On 28 September 2010 10:43, Terrence Cole
<list-sink at trainedmonkeystudios.org> wrote:
> On Tue, 2010-09-28 at 01:57 +0200, Jacob Hall?n wrote:
>> Monday 27 September 2010 you wrote:
>> > On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
>> > > Well, I am happy to see that the my interest in a general purpose RPython
>> > > is not as isolated as I was lead to believe :-))
>> > > Thx,
>> >
>> > What I wrote has apparently been widely misunderstood, so let me explain
>> > what I mean in more detail. ?What I want is _not_ RPython and it is
>> > _not_ Shedskin. ?What I want is not a compiler at all. ?What I want is a
>> > visual tool, for example, a plugin to an IDE. ?This tool would perform
>> > static analysis on a piece of python code. ?Instead of generating code
>> > with this information, it would mark up the python code in the text
>> > display with colors, weights, etc in order to show properties from the
>> > static analysis. ?This would be something like semantic highlighting, as
>> > opposed to syntax highlighting.
>> >
>> > I think it possible that this information would, if created and
>> > presented in the correct way, represent the sort of optimizations that
>> > pypy-c-jit -- a full python implementation, not a language subset --
>> > would likely perform on the code if run. ?Given this sort of feedback,
>> > it would be much easier for a python coder to write code that works well
>> > with the jit: for example, moving a declaration inside a loop to avoid
>> > boxing, based on the information presented.
>> >
>> > Ideally, such a tool would perform instantaneous syntax highlighting
>> > while editing and do full parsing and analysis in the background to
>> > update the semantic highlighting as frequently as possible. ?Obviously,
>> > detailed static analysis will provide far more information than it would
>> > be possible to display on the code at once, so I see this gui as having
>> > several modes -- like predator vision -- that show different information
>> > from the analysis. ?Naturally, what those modes are will depend strongly
>> > on the details of how pypy-c-jit works internally, what sort of
>> > information can be sanely collected through static analysis, and,
>> > naturally, user testing.
>> >
>> > I was somewhat baffled at first as to how what I wrote before was
>> > interpreted as interest in a static python. ?I think the disconnect here
>> > is the assumption on many people's part that a static language will
>> > always be faster than a dynamic one. ?Given the existing tools that
>> > provide basically no feedback from the compiler / interpreter / jitter,
>> > this is inevitably true at the moment. ?I foresee a future, however,
>> > where better tools let us use the full power of a dynamic python AND let
>> > us tighten up our code for speed to get the full advantages of jit
>> > compilation as well. ?I believe that in the end, this combination will
>> > prove superior to any fully static compiler.
>>
>> The JIT works because it has more information at runtime than what is
>> available at compile time. If the information was available at compile time we
>> could do the optimizations then and not have to invoke the extra complexity
>> required by the JIT. Examples of ?the extra information include things like
>> knowing that introspection will not be used in the current evaluation of a
>> loop, specific argument types will be used in calls and that some arguments
>> will be known to be constant over part of the program execution.. Knowing
>> these bits allows you to optimize away large chunks o f the code that
>> otherwise would have been executed.
>>
>> Static analysis assumes that none of the above mentioned possibilities can
>> actually take place. It is impossible to make such assumptions at compile time
>> in a dynamic language. Therefore PyPy is a bad match for people wanting to
>> staically compile subsets of Python. Applying the JIT to RPython code
>
> Yes, that idea is just dumb. ?It's also not what I suggested at all. ?I
> can see now that what I said would be easy to misinterpret, but on
> re-reading it, it clearly doesn't say what you think it does.

It does make /some/ sense, I think. From the perspective of the JIT,
operating at interp-level, the app-level python program *is the
biggest part of* the "stuff you don't know about until runtime". That
is, you don't know the program source at translation time, and most of
the information the JIT is supposed to find are app-level constructs
(eg app-level loops).

Of course any such analysis will fall flat in certain cases, like
eval(raw_input(...)). But you should still be able to gather enough
information for most fairly hygenic code.

What sort of analyses did you have in mind?

>> ?is not
>> workable, because the JIT is optimized to remove bits of generated assembler
>> code that never shows up in the compilation of RPython code.
>>
>> These are very basic first principle concepts, and it is a mystery to me why
>> people can't work them out for themselves.
>
> You are quite right that static analysis will be able to do little to
> help an optimal jit. ?However, I doubt that in the near term pypy's jit
> will cover all the dark corners of python equally well -- C has been
> around for 38 years and its still got room for optimization.

There are some undesirable things about static analysis, but it can
sure be useful from optimisation, security and reliability
perspectives. There's also code browsing, too; IDEs require a
different (fuzzier) parser, but the question of 'what types does this
object probably have' makes more sense with a little dependent region
analysis. Optimising when you can be fairly confident of the types
involved could be useful. That doesn't really sound like pypy at that
point, though.

-- 
William Leslie


From fijall at gmail.com  Tue Sep 28 15:20:46 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 28 Sep 2010 15:20:46 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285634613.5954.112.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
Message-ID: <AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>

On Tue, Sep 28, 2010 at 2:43 AM, Terrence Cole
<list-sink at trainedmonkeystudios.org> wrote:
> On Tue, 2010-09-28 at 01:57 +0200, Jacob Hall?n wrote:
>> Monday 27 September 2010 you wrote:
>> > On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
>> > > Well, I am happy to see that the my interest in a general purpose RPython
>> > > is not as isolated as I was lead to believe :-))
>> > > Thx,
>> >
>> > What I wrote has apparently been widely misunderstood, so let me explain
>> > what I mean in more detail. ?What I want is _not_ RPython and it is
>> > _not_ Shedskin. ?What I want is not a compiler at all. ?What I want is a
>> > visual tool, for example, a plugin to an IDE. ?This tool would perform
>> > static analysis on a piece of python code. ?Instead of generating code
>> > with this information, it would mark up the python code in the text
>> > display with colors, weights, etc in order to show properties from the
>> > static analysis. ?This would be something like semantic highlighting, as
>> > opposed to syntax highlighting.
>> >
>> > I think it possible that this information would, if created and
>> > presented in the correct way, represent the sort of optimizations that
>> > pypy-c-jit -- a full python implementation, not a language subset --
>> > would likely perform on the code if run. ?Given this sort of feedback,
>> > it would be much easier for a python coder to write code that works well
>> > with the jit: for example, moving a declaration inside a loop to avoid
>> > boxing, based on the information presented.
>> >
>> > Ideally, such a tool would perform instantaneous syntax highlighting
>> > while editing and do full parsing and analysis in the background to
>> > update the semantic highlighting as frequently as possible. ?Obviously,
>> > detailed static analysis will provide far more information than it would
>> > be possible to display on the code at once, so I see this gui as having
>> > several modes -- like predator vision -- that show different information
>> > from the analysis. ?Naturally, what those modes are will depend strongly
>> > on the details of how pypy-c-jit works internally, what sort of
>> > information can be sanely collected through static analysis, and,
>> > naturally, user testing.
>> >
>> > I was somewhat baffled at first as to how what I wrote before was
>> > interpreted as interest in a static python. ?I think the disconnect here
>> > is the assumption on many people's part that a static language will
>> > always be faster than a dynamic one. ?Given the existing tools that
>> > provide basically no feedback from the compiler / interpreter / jitter,
>> > this is inevitably true at the moment. ?I foresee a future, however,
>> > where better tools let us use the full power of a dynamic python AND let
>> > us tighten up our code for speed to get the full advantages of jit
>> > compilation as well. ?I believe that in the end, this combination will
>> > prove superior to any fully static compiler.
>> >
>> > -Terrence
>> >
>> > > Sarvi
>> > >
>> > >
>> > > ----- Original Message ----
>> > >
>> > > > From: Terrence Cole <list-sink at trainedmonkeystudios.org>
>> > > > To: pypy-dev at codespeak.net
>> > > > Sent: Sun, September 26, 2010 2:28:12 PM
>> > > > Subject: Re: [pypy-dev] Question on the future of RPython
>> > > >
>> > > > On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
>> > > > > i just had a ?(probably) silly idea. :)
>> > > > >
>> > > > > if some people like rpython so much, ?how about writing a rpython
>> > > > > interpreter in rpython? wouldn't it be much ?easier for the jit to
>> > > > > optimize rpython code? couldn't jitted rpython ?code theoretically be
>> > > > > as fast as a program that got compiled to c from ?rpython?
>> > > > >
>> > > > > hm... but i wonder if this would make sense at all. ?maybe if you ran
>> > > > > rpython code with pypy-c-jit, it already could be ?jitted as well as
>> > > > > with a special rpython interpreter? ...if there were a ?special
>> > > > > rpython interpreter, would the current jit generator have to be
>> > > > > changed to take advantage of the more simple language?
>> > > >
>> > > > An ?excellent question at least.
>> > > >
>> > > > A better idea, I think, would be to ?ask what subset of full-python
>> > > > will jit well. ?What I'd really like to ?see is a static analyzer that
>> > > > can display (e.g. by coloring names or lines) ?how "jit friendly" a
>> > > > piece of python code is. ?This would allow a ?programmer to get an
>> > > > idea of what help the jit is going to be when running ?their code and,
>> > > > hopefully, help people avoid tragic performance ?results. ?Naturally,
>> > > > for performance intensive code, you would still ?need to profile, but
>> > > > for a lot of uses, simply not having catastrophically ?bad performance
>> > > > is more than enough for a good user experience.
>> > > >
>> > > > With such a tool, it wouldn't really matter if the answer to "what ?is
>> > > > faster" is RPython -- it would be whatever python language ?subset
>> > > > happens to work well in a particular case. ?I've started working ?on
>> > > > something like this [1], but given that I'm doing a startup, I ?don't
>> > > > have nearly the time I would need to make this useful in the
>> > > > near-term.
>>
>> The JIT works because it has more information at runtime than what is
>> available at compile time. If the information was available at compile time we
>> could do the optimizations then and not have to invoke the extra complexity
>> required by the JIT. Examples of ?the extra information include things like
>> knowing that introspection will not be used in the current evaluation of a
>> loop, specific argument types will be used in calls and that some arguments
>> will be known to be constant over part of the program execution.. Knowing
>> these bits allows you to optimize away large chunks o f the code that
>> otherwise would have been executed.
>>
>> Static analysis assumes that none of the above mentioned possibilities can
>> actually take place. It is impossible to make such assumptions at compile time
>> in a dynamic language. Therefore PyPy is a bad match for people wanting to
>> staically compile subsets of Python. Applying the JIT to RPython code
>
> Yes, that idea is just dumb. ?It's also not what I suggested at all. ?I
> can see now that what I said would be easy to misinterpret, but on
> re-reading it, it clearly doesn't say what you think it does.
>
>> ?is not
>> workable, because the JIT is optimized to remove bits of generated assembler
>> code that never shows up in the compilation of RPython code.
>>
>> These are very basic first principle concepts, and it is a mystery to me why
>> people can't work them out for themselves.
>
> You are quite right that static analysis will be able to do little to
> help an optimal jit. ?However, I doubt that in the near term pypy's jit
> will cover all the dark corners of python equally well -- C has been
> around for 38 years and its still got room for optimization.
>
> -Terrence
>
>> Jacob Hall?n
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

Hey.

I'm really interested in having jit feedback displayed as text info
(say for profiling purposes). Do you have any particular ideas in mind
or just a general one?


From tismer at stackless.com  Tue Sep 28 16:39:38 2010
From: tismer at stackless.com (Christian Tismer)
Date: Tue, 28 Sep 2010 16:39:38 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <187982.37806.qm@web111302.mail.gq1.yahoo.com>
References: <187982.37806.qm@web111302.mail.gq1.yahoo.com>
Message-ID: <4CA1FE2A.5060105@stackless.com>

  On 9/27/10 2:29 PM, Andy wrote:
> Why wouldn't pypy work with greenlet but would work with Stackless? greenlet calls itself a spin-off of Stackless. Isn't greenlet a subset of Stackless without the scheduling? Could you explain a bit more?
>

This is a deep misconception. Neither stackless nor greenlets
work with PyPy. Instead, a special coroutine version was written
for PyP's RPython, and then Stackless was written as an application
module. There is a greenlet implementation as well.

They both rely on the stack unwinding, which is not yet implemented
for the Jit.

The original greenlets and stackless have some similarities, since
they use the same tricks to modify the stack in assembly. This is
not related to PyPy, this reasoning is just the improper level.

ciao - chris

-- 
Christian Tismer             :^)<mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key ->  http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/



From list-sink at trainedmonkeystudios.org  Tue Sep 28 21:49:08 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Tue, 28 Sep 2010 12:49:08 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
Message-ID: <1285703348.5276.56.camel@localhost>

On Tue, 2010-09-28 at 11:55 +1000, William Leslie wrote:
> On 28 September 2010 10:43, Terrence Cole
> <list-sink at trainedmonkeystudios.org> wrote:
> > On Tue, 2010-09-28 at 01:57 +0200, Jacob Hall?n wrote:
> >> Monday 27 September 2010 you wrote:
> >> > On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> >> > > Well, I am happy to see that the my interest in a general purpose RPython
> >> > > is not as isolated as I was lead to believe :-))
> >> > > Thx,
> >> >
> >> > What I wrote has apparently been widely misunderstood, so let me explain
> >> > what I mean in more detail.  What I want is _not_ RPython and it is
> >> > _not_ Shedskin.  What I want is not a compiler at all.  What I want is a
> >> > visual tool, for example, a plugin to an IDE.  This tool would perform
> >> > static analysis on a piece of python code.  Instead of generating code
> >> > with this information, it would mark up the python code in the text
> >> > display with colors, weights, etc in order to show properties from the
> >> > static analysis.  This would be something like semantic highlighting, as
> >> > opposed to syntax highlighting.
> >> >
> >> > I think it possible that this information would, if created and
> >> > presented in the correct way, represent the sort of optimizations that
> >> > pypy-c-jit -- a full python implementation, not a language subset --
> >> > would likely perform on the code if run.  Given this sort of feedback,
> >> > it would be much easier for a python coder to write code that works well
> >> > with the jit: for example, moving a declaration inside a loop to avoid
> >> > boxing, based on the information presented.
> >> >
> >> > Ideally, such a tool would perform instantaneous syntax highlighting
> >> > while editing and do full parsing and analysis in the background to
> >> > update the semantic highlighting as frequently as possible.  Obviously,
> >> > detailed static analysis will provide far more information than it would
> >> > be possible to display on the code at once, so I see this gui as having
> >> > several modes -- like predator vision -- that show different information
> >> > from the analysis.  Naturally, what those modes are will depend strongly
> >> > on the details of how pypy-c-jit works internally, what sort of
> >> > information can be sanely collected through static analysis, and,
> >> > naturally, user testing.
> >> >
> >> > I was somewhat baffled at first as to how what I wrote before was
> >> > interpreted as interest in a static python.  I think the disconnect here
> >> > is the assumption on many people's part that a static language will
> >> > always be faster than a dynamic one.  Given the existing tools that
> >> > provide basically no feedback from the compiler / interpreter / jitter,
> >> > this is inevitably true at the moment.  I foresee a future, however,
> >> > where better tools let us use the full power of a dynamic python AND let
> >> > us tighten up our code for speed to get the full advantages of jit
> >> > compilation as well.  I believe that in the end, this combination will
> >> > prove superior to any fully static compiler.
> >>
> >> The JIT works because it has more information at runtime than what is
> >> available at compile time. If the information was available at compile time we
> >> could do the optimizations then and not have to invoke the extra complexity
> >> required by the JIT. Examples of  the extra information include things like
> >> knowing that introspection will not be used in the current evaluation of a
> >> loop, specific argument types will be used in calls and that some arguments
> >> will be known to be constant over part of the program execution.. Knowing
> >> these bits allows you to optimize away large chunks o f the code that
> >> otherwise would have been executed.
> >>
> >> Static analysis assumes that none of the above mentioned possibilities can
> >> actually take place. It is impossible to make such assumptions at compile time
> >> in a dynamic language. Therefore PyPy is a bad match for people wanting to
> >> staically compile subsets of Python. Applying the JIT to RPython code
> >
> > Yes, that idea is just dumb.  It's also not what I suggested at all.  I
> > can see now that what I said would be easy to misinterpret, but on
> > re-reading it, it clearly doesn't say what you think it does.
> 
> It does make /some/ sense, I think. From the perspective of the JIT,
> operating at interp-level, 

I think this is a disconnect.  Applying a jit to a non-interpretted
language -- Jacob here seems to think I was talking about a static,
compiled subset of python -- makes little sense.  Static analysis to
provide help to an interpreter does, as you say, make some sense, and
not to just me.  Brett Cannon applied static type analysis to the
CPython interpreter for his PHD thesis [1], looking for a speed boost by
removing some typing abstraction.  Unfortunately, it was not
spectacularly helpful for CPython.  I think for pypy-jit, however, it
has much greater potential because of the possibility of full unboxing.
Given past results however, it's not the first place I'd go looking for
speedups.  Others may have better ideas in this area than I do though.

> the app-level python program *is the
> biggest part of* the "stuff you don't know about until runtime". That
> is, you don't know the program source at translation time, and most of
> the information the JIT is supposed to find are app-level constructs
> (eg app-level loops).

This is one of the reasons that I had to pull together my own parsing
(largely borrowed from pypy, actually) and analysis infrastructure,
rather than just using pypy's off-the-shelf.  Even without pypy's neat
analysis code, the fact that it ditches character-level info when making
an ast means you can't apply highlighting with it without groping about
half-blindly in the source.

> Of course any such analysis will fall flat in certain cases, like
> eval(raw_input(...)). But you should still be able to gather enough
> information for most fairly hygenic code.

Given the choice between the status quo and an extremely slow eval, but
much faster python overall, I think most people would pick the second.

> What sort of analyses did you have in mind?

As this is a side project, for the moment I am focusing on simple stuff,
mostly things I need/want for work.  In the short term these include
Python3 linting (which is almost working) and static type analysis.  The
second will be particularly interesting because we have (at work)
annotated most of our interfaces with type data, so this will probably
net much more specific and helpful data than it would in many projects.
I am also, specifically, as I mentioned to Paolo yesterday, trying to
find out how much of our code could be fully unboxed, given that we have
extensive type contracts at our interfaces.  If the answer is "most of
it", then it may make sense for us to build something like Jaegermonkey
for python someday.

> >>  is not
> >> workable, because the JIT is optimized to remove bits of generated assembler
> >> code that never shows up in the compilation of RPython code.
> >>
> >> These are very basic first principle concepts, and it is a mystery to me why
> >> people can't work them out for themselves.
> >
> > You are quite right that static analysis will be able to do little to
> > help an optimal jit.  However, I doubt that in the near term pypy's jit
> > will cover all the dark corners of python equally well -- C has been
> > around for 38 years and its still got room for optimization.
> 
> There are some undesirable things about static analysis, but it can
> sure be useful from optimisation, security and reliability
> perspectives. 

Brendan Eich agrees [2].  This is heartening, because javascript has
much in common with python.

I agree too, for that matter, but that's probably a lot less
heartening :-).

> There's also code browsing, too; IDEs require a
> different (fuzzier) parser, 

Reason number two that I have to maintain a separate parser/analyzer.

> but the question of 'what types does this
> object probably have' makes more sense with a little dependent region
> analysis. Optimising when you can be fairly confident of the types
> involved could be useful. That doesn't really sound like pypy at that
> point, though.

Given that I want to work with Python3 anyway (and that I'd never be
able to beat pypy's performance before it supports Python3), I'm
focusing mostly on a tool to help make reliable and correct code.  

However, performance is always in the back of my mind these days.  It
seems from this thread that I won't be able to do much in that regard
with my current approach, unfortunately.  Maybe by the time I can focus
on it, pypy will support python3 and I can work on providing real-time
jit feedback.

-Terrence

[1] http://www.ocf.berkeley.edu/~bac/thesis.pdf
[2] http://brendaneich.com/2010/08/static-analysis-ftw/




From list-sink at trainedmonkeystudios.org  Tue Sep 28 22:33:14 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Tue, 28 Sep 2010 13:33:14 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
Message-ID: <1285705994.5276.99.camel@localhost>

On Tue, 2010-09-28 at 15:20 +0200, Maciej Fijalkowski wrote:
> On Tue, Sep 28, 2010 at 2:43 AM, Terrence Cole
> <list-sink at trainedmonkeystudios.org> wrote:
> > On Tue, 2010-09-28 at 01:57 +0200, Jacob Hall?n wrote:
> >> Monday 27 September 2010 you wrote:
> >> > On Sun, 2010-09-26 at 23:57 -0700, Saravanan Shanmugham wrote:
> >> > > Well, I am happy to see that the my interest in a general purpose RPython
> >> > > is not as isolated as I was lead to believe :-))
> >> > > Thx,
> >> >
> >> > What I wrote has apparently been widely misunderstood, so let me explain
> >> > what I mean in more detail.  What I want is _not_ RPython and it is
> >> > _not_ Shedskin.  What I want is not a compiler at all.  What I want is a
> >> > visual tool, for example, a plugin to an IDE.  This tool would perform
> >> > static analysis on a piece of python code.  Instead of generating code
> >> > with this information, it would mark up the python code in the text
> >> > display with colors, weights, etc in order to show properties from the
> >> > static analysis.  This would be something like semantic highlighting, as
> >> > opposed to syntax highlighting.
> >> >
> >> > I think it possible that this information would, if created and
> >> > presented in the correct way, represent the sort of optimizations that
> >> > pypy-c-jit -- a full python implementation, not a language subset --
> >> > would likely perform on the code if run.  Given this sort of feedback,
> >> > it would be much easier for a python coder to write code that works well
> >> > with the jit: for example, moving a declaration inside a loop to avoid
> >> > boxing, based on the information presented.
> >> >
> >> > Ideally, such a tool would perform instantaneous syntax highlighting
> >> > while editing and do full parsing and analysis in the background to
> >> > update the semantic highlighting as frequently as possible.  Obviously,
> >> > detailed static analysis will provide far more information than it would
> >> > be possible to display on the code at once, so I see this gui as having
> >> > several modes -- like predator vision -- that show different information
> >> > from the analysis.  Naturally, what those modes are will depend strongly
> >> > on the details of how pypy-c-jit works internally, what sort of
> >> > information can be sanely collected through static analysis, and,
> >> > naturally, user testing.
> >> >
> >> > I was somewhat baffled at first as to how what I wrote before was
> >> > interpreted as interest in a static python.  I think the disconnect here
> >> > is the assumption on many people's part that a static language will
> >> > always be faster than a dynamic one.  Given the existing tools that
> >> > provide basically no feedback from the compiler / interpreter / jitter,
> >> > this is inevitably true at the moment.  I foresee a future, however,
> >> > where better tools let us use the full power of a dynamic python AND let
> >> > us tighten up our code for speed to get the full advantages of jit
> >> > compilation as well.  I believe that in the end, this combination will
> >> > prove superior to any fully static compiler.
> >> >
> >> > -Terrence
> >> >
> >> > > Sarvi
> >> > >
> >> > >
> >> > > ----- Original Message ----
> >> > >
> >> > > > From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> >> > > > To: pypy-dev at codespeak.net
> >> > > > Sent: Sun, September 26, 2010 2:28:12 PM
> >> > > > Subject: Re: [pypy-dev] Question on the future of RPython
> >> > > >
> >> > > > On Sat, 2010-09-25 at 17:47 +0200, horace grant wrote:
> >> > > > > i just had a  (probably) silly idea. :)
> >> > > > >
> >> > > > > if some people like rpython so much,  how about writing a rpython
> >> > > > > interpreter in rpython? wouldn't it be much  easier for the jit to
> >> > > > > optimize rpython code? couldn't jitted rpython  code theoretically be
> >> > > > > as fast as a program that got compiled to c from  rpython?
> >> > > > >
> >> > > > > hm... but i wonder if this would make sense at all.  maybe if you ran
> >> > > > > rpython code with pypy-c-jit, it already could be  jitted as well as
> >> > > > > with a special rpython interpreter? ...if there were a  special
> >> > > > > rpython interpreter, would the current jit generator have to be
> >> > > > > changed to take advantage of the more simple language?
> >> > > >
> >> > > > An  excellent question at least.
> >> > > >
> >> > > > A better idea, I think, would be to  ask what subset of full-python
> >> > > > will jit well.  What I'd really like to  see is a static analyzer that
> >> > > > can display (e.g. by coloring names or lines)  how "jit friendly" a
> >> > > > piece of python code is.  This would allow a  programmer to get an
> >> > > > idea of what help the jit is going to be when running  their code and,
> >> > > > hopefully, help people avoid tragic performance  results.  Naturally,
> >> > > > for performance intensive code, you would still  need to profile, but
> >> > > > for a lot of uses, simply not having catastrophically  bad performance
> >> > > > is more than enough for a good user experience.
> >> > > >
> >> > > > With such a tool, it wouldn't really matter if the answer to "what  is
> >> > > > faster" is RPython -- it would be whatever python language  subset
> >> > > > happens to work well in a particular case.  I've started working  on
> >> > > > something like this [1], but given that I'm doing a startup, I  don't
> >> > > > have nearly the time I would need to make this useful in the
> >> > > > near-term.
> >>
> >> The JIT works because it has more information at runtime than what is
> >> available at compile time. If the information was available at compile time we
> >> could do the optimizations then and not have to invoke the extra complexity
> >> required by the JIT. Examples of  the extra information include things like
> >> knowing that introspection will not be used in the current evaluation of a
> >> loop, specific argument types will be used in calls and that some arguments
> >> will be known to be constant over part of the program execution.. Knowing
> >> these bits allows you to optimize away large chunks o f the code that
> >> otherwise would have been executed.
> >>
> >> Static analysis assumes that none of the above mentioned possibilities can
> >> actually take place. It is impossible to make such assumptions at compile time
> >> in a dynamic language. Therefore PyPy is a bad match for people wanting to
> >> staically compile subsets of Python. Applying the JIT to RPython code
> >
> > Yes, that idea is just dumb.  It's also not what I suggested at all.  I
> > can see now that what I said would be easy to misinterpret, but on
> > re-reading it, it clearly doesn't say what you think it does.
> >
> >>  is not
> >> workable, because the JIT is optimized to remove bits of generated assembler
> >> code that never shows up in the compilation of RPython code.
> >>
> >> These are very basic first principle concepts, and it is a mystery to me why
> >> people can't work them out for themselves.
> >
> > You are quite right that static analysis will be able to do little to
> > help an optimal jit.  However, I doubt that in the near term pypy's jit
> > will cover all the dark corners of python equally well -- C has been
> > around for 38 years and its still got room for optimization.
> >
> > -Terrence
> >
> >> Jacob Hall?n
> >
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> 
> Hey.
> 
> I'm really interested in having jit feedback displayed as text info
> (say for profiling purposes). Do you have any particular ideas in mind
> or just a general one?

Lots.  They're almost all probably wrong though, so be warned :-).  I'm
also not entirely clear on what you mean, so let me tell you what I have
in mind and you can tell me if I'm way off base.

I assume workflow would go like this:  1) run pypy on a bunch of code in
profiling mode, 2) pypy spits out lots of data about what happened in
the jit when the program exits, 3) start up external analysis program
pointing it at this data, 4) browse the python source with the data from
the jit overlayed as color, formatting, etc on top of the source.
Potentially there would be several separate modes for viewing different
aspects of the jit info.  This could also include the ability to select
different program elements (loops, variables, functions, etc) and get
detailed information about their runtime usage in a side-pane.  Ideally,
this workflow would be taken care of automatically by pushing the run
button in your IDE.

As a more specific example of what the gui would do in, for instance,
escape analysis mode:  display local variables that do not escape any
loops in green, others in red.  Hovering over a red variable would show
information about how, why, and where it escapes the loop in a tooltip
or bubble.  Selecting a red variable show the same info in a pane and
would draw arrows on the source showing where it escapes from a
loop/function etc.

In my ideal world, this profiling data analysis would sit side-by-side
with various display modes that show useful static analysis feedback,
all inside a full-fledged python IDE.

This is all, of course, a long way off still.  What I'm working on right
now is basic linting for python3 so that I can add a lint step to our
hudson server and start to get some graphs up.

What I _really_ would like to work on, if I had the time, is making pypy
support Python3 so that I could use it at work.  However, I think I'd
mostly just get in the way if I tried that, given my other time
commitments.

I hope there was something helpful in that brain-dump, but I suspect I
may be way off target at this point.

-Terrence





From anto.cuni at gmail.com  Wed Sep 29 11:37:05 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 29 Sep 2010 11:37:05 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285705994.5276.99.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	<827221.6704.qm@web53704.mail.re2.yahoo.com>	<1285616691.5954.34.camel@localhost>	<201009280157.17264.jacob@openend.se>	<1285634613.5954.112.camel@localhost>	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost>
Message-ID: <4CA308C1.5030200@gmail.com>

Hi Terrence, hi all

On 28/09/10 22:33, Terrence Cole wrote:
> I assume workflow would go like this:  1) run pypy on a bunch of code in
> profiling mode, 2) pypy spits out lots of data about what happened in
> the jit when the program exits, 3) start up external analysis program
> pointing it at this data, 4) browse the python source with the data from
> the jit overlayed as color, formatting, etc on top of the source.

You can already to it (partially) by using the PYPYLOG environment variable 
like this:

PYPYLOG=jit-log-opt:mylog ./pypy -m test.pystone

then, mylog contains all the loops and bridges produced by the jit. The 
interesting point is that there are also special operations called 
"debug_merge_point" that are emitted for each python bytecode, so you can 
easily map the low-level jit instructions back to the original python source.

E.g., take lines 214 of pystone:
     Array1Par[IntLoc+30] = IntLoc

The corresponding python bytecode is this:
214          38 LOAD_FAST                4 (IntLoc)
              41 LOAD_FAST                0 (Array1Par)
              44 LOAD_FAST                4 (IntLoc)
              47 LOAD_CONST               3 (30)
              50 BINARY_ADD
              51 STORE_SUBSCR


By searching in the logs, you find the following (I edited it a bit to improve 
readability):

debug_merge_point('<code object Proc8, line 208> #38 LOAD_FAST')
debug_merge_point('<code object Proc8, line 208> #41 LOAD_FAST')
debug_merge_point('<code object Proc8, line 208> #44 LOAD_FAST')
debug_merge_point('<code object Proc8, line 208> #47 LOAD_CONST')
debug_merge_point('<code object Proc8, line 208> #50 BINARY_ADD')
debug_merge_point('<code object Proc8, line 208> #51 STORE_SUBSCR')
p345 = new_with_vtable(ConstClass(W_IntObject))
setfield_gc(p345, 8, descr=<SignedFieldDescr 
pypy.objspace.std.intobject.W_IntObject.inst_intval 8>)
call(ConstClass(ll_setitem__dum_checkidxConst_listPtr_Signed_objectPtr),
      p333, 38, p345, descr=<VoidCallDescr>)
guard_no_exception(, descr=<Guard146>) [p1, p0, p71, p345, p312, p3, p4, p6, 
p308, p315, p335, p12, p13, p14, p15, p16, p18, p19, p178, p26, p320, p328, 
i124, p25, i329]

Here, you can see that most opcodes are "empty" (i.e., no operations between 
one debug_merge_point and the next). In general, all the opcodes that 
manipulate the python stack are optimized away by the jit, because all the 
python variables on the stack become "local variables" in the assembler.

Moreover, you can see that BINARY_ADD is also empty: this probably means that 
the loop was specialized for the specific value of IntLoc, so the addition has 
been constant-folded away.  Indeed, the only opcode that do real work is 
STORE_SUBSCR.  What it does it to allocate a new W_IntObject whose value is 8 
(i.e., boxing IntLoc on the fly, because it's escaping), and store it into the 
element 38 of the list stored in p333.

Finally, we check that no exception was raised.

Obviously, when presenting these information to the user you must consider 
that there is not a 1-to-1 mapping from python source to jit loops.  In the 
example above, the very same opcodes are compiled also in another loop (which 
by chance it has the same jit-operations, but they might also be very 
different, depending on the cases).

As you can see, there is already lot of information that can be useful to the 
user.  However, don't ask me how to present it visually :-)

ciao,
anto


From fijall at gmail.com  Wed Sep 29 12:35:00 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 29 Sep 2010 12:35:00 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <4CA308C1.5030200@gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost> <4CA308C1.5030200@gmail.com>
Message-ID: <AANLkTikk+uxjETJ5bHYkJWKuHjq1wroZoJwenD2dmUNc@mail.gmail.com>

>
> As you can see, there is already lot of information that can be useful to
> the user. ?However, don't ask me how to present it visually :-)
>

As you've probably noticed, it takes quite a bit of skill to actually
read it and say which variables are unescaped locals for example


From list-sink at trainedmonkeystudios.org  Wed Sep 29 22:40:59 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Wed, 29 Sep 2010 13:40:59 -0700
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <4CA308C1.5030200@gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost>  <4CA308C1.5030200@gmail.com>
Message-ID: <1285792859.5129.79.camel@localhost>

On Wed, 2010-09-29 at 11:37 +0200, Antonio Cuni wrote:
> Hi Terrence, hi all
> 
> On 28/09/10 22:33, Terrence Cole wrote:
> > I assume workflow would go like this:  1) run pypy on a bunch of code in
> > profiling mode, 2) pypy spits out lots of data about what happened in
> > the jit when the program exits, 3) start up external analysis program
> > pointing it at this data, 4) browse the python source with the data from
> > the jit overlayed as color, formatting, etc on top of the source.
> 
> You can already to it (partially) by using the PYPYLOG environment variable 
> like this:
> 
> PYPYLOG=jit-log-opt:mylog ./pypy -m test.pystone
> 
> then, mylog contains all the loops and bridges produced by the jit. The 
> interesting point is that there are also special operations called 
> "debug_merge_point" that are emitted for each python bytecode, so you can 
> easily map the low-level jit instructions back to the original python source.

I think that 'easily' in that last sentence is missing scare-quotes. :-)

> E.g., take lines 214 of pystone:
>      Array1Par[IntLoc+30] = IntLoc
> 
> The corresponding python bytecode is this:
> 214          38 LOAD_FAST                4 (IntLoc)
>               41 LOAD_FAST                0 (Array1Par)
>               44 LOAD_FAST                4 (IntLoc)
>               47 LOAD_CONST               3 (30)
>               50 BINARY_ADD
>               51 STORE_SUBSCR
> 
> 
> By searching in the logs, you find the following (I edited it a bit to improve 
> readability):
> 
> debug_merge_point('<code object Proc8, line 208> #38 LOAD_FAST')
> debug_merge_point('<code object Proc8, line 208> #41 LOAD_FAST')
> debug_merge_point('<code object Proc8, line 208> #44 LOAD_FAST')
> debug_merge_point('<code object Proc8, line 208> #47 LOAD_CONST')
> debug_merge_point('<code object Proc8, line 208> #50 BINARY_ADD')
> debug_merge_point('<code object Proc8, line 208> #51 STORE_SUBSCR')
> p345 = new_with_vtable(ConstClass(W_IntObject))
> setfield_gc(p345, 8, descr=<SignedFieldDescr 
> pypy.objspace.std.intobject.W_IntObject.inst_intval 8>)
> call(ConstClass(ll_setitem__dum_checkidxConst_listPtr_Signed_objectPtr),
>       p333, 38, p345, descr=<VoidCallDescr>)
> guard_no_exception(, descr=<Guard146>) [p1, p0, p71, p345, p312, p3, p4, p6, 
> p308, p315, p335, p12, p13, p14, p15, p16, p18, p19, p178, p26, p320, p328, 
> i124, p25, i329]
> 
> Here, you can see that most opcodes are "empty" (i.e., no operations between 
> one debug_merge_point and the next). In general, all the opcodes that 
> manipulate the python stack are optimized away by the jit, because all the 
> python variables on the stack become "local variables" in the assembler.
> 
> Moreover, you can see that BINARY_ADD is also empty: this probably means that 
> the loop was specialized for the specific value of IntLoc, so the addition has 
> been constant-folded away.  Indeed, the only opcode that do real work is 
> STORE_SUBSCR.  What it does it to allocate a new W_IntObject whose value is 8 
> (i.e., boxing IntLoc on the fly, because it's escaping), and store it into the 
> element 38 of the list stored in p333.
> 
> Finally, we check that no exception was raised.

Wow, thank you for the awesome explanation.  I think the only surprising
thing in there is that I actually understood all of that.

> Obviously, when presenting these information to the user you must consider 
> that there is not a 1-to-1 mapping from python source to jit loops.  In the 
> example above, the very same opcodes are compiled also in another loop (which 
> by chance it has the same jit-operations, but they might also be very 
> different, depending on the cases).

Currently, in my hacked together parsing chain, the low-level parser
keeps a reference to the underlying token when it creates a new node and
subsequently the ast builder keeps a references to the low-level parse
node when it creates an ast node.  This way, I can easily map down to
the individual source chars and full context when walking the AST to do
highlighting, linting, etc.  

My first inclination would be to continue this chain and add a bytecode
compiler on top of the ast builder.  This would keep ast node references
in the instructions it creates.  If the algorithms don't diverge too
much, I think this would allow the debug output to be mapped all the way
back to the source chars with minimal effort.  I'm not terrifically
familiar with the specifics of how python emits bytecode from an ast, so
I'd appreciate any feedback if you think this is crazy-talk.

> As you can see, there is already lot of information that can be useful to the 
> user.  However, don't ask me how to present it visually :-)

Neither do I, but finding out is going to be the fun part.  

> ciao,
> anto

I'm excited to try some of this out, but unfortunately, there is an
annoying problem in the way.  All of my work in the last year has been
on python3.  Having worked in python3 for awhile now, my opinion is that
it's just a much better language -- surprisingly so, considering how
little it changed from python2.  If pypy supported python3, then I could
maintain my parser as a diff against pypy (you are switching to
Mercurial at some point, right?), which would make it much easier to
avoid divergence.

So what I'm getting at is: what is the pypy story for python3 support?
I haven't seen anything on pypy-dev, or in my occasional looks at the
repository, to suggest that it is being worked on but I'm sure you have
a plan of some sort.  I'm willing to help out with python3 support, if I
can do so without getting in anyone's way.  It seems like the sort of
thing that will be disruptive, however, so I have been leery of jumping
in, considering how little time I have to contribute, at the moment.

In my mind, the python3 picture is something like:
At the compilation level, it's easy enough to dump Grammar3.2 in
pypy/interpreter/pyparser/data and to modify astbuilder for python3 --
I'll backport the changes I made, if you want.  Interpreter support for
the new language features will be harder, but that's probably already
done since 2.7 is almost working.  The only potential problems I see are
the big string/unicode switch and the management of the fairly large
changes to astbuilder -- I'm sure you want to continue supporting
python2 into the future.  I don't know how much the bytecode changed
between 2 and 3, so I'm not sure if there are jit issues to worry about.
Am I missing anything big?


-Terrence




From andrewfr_ice at yahoo.com  Wed Sep 29 23:39:55 2010
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Wed, 29 Sep 2010 14:39:55 -0700 (PDT)
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <mailman.1036.1285616701.6189.pypy-dev@codespeak.net>
Message-ID: <69092.15439.qm@web120710.mail.ne1.yahoo.com>

Hi Maciej:

Message: 4
Date: Mon, 27 Sep 2010 14:23:14 +0200
From: Maciej Fijalkowski <fijall at gmail.com>
Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet
To: "Ian P. Cooke" <ipc at srand.net>
Cc: pypy-dev at codespeak.net
Message-ID:
    <AANLkTinYC+Orf9RXfyST3ZJPX0ZM0nAXYsh=tH8PtAxK at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

>greenlet C module is quite incompatible with pypy and won't work.
>However making pypy work with jit and stackless is something that
>requires a bit of work only (teaching jit how to unroll the stack
>mostly) and I plan to look into it in the very near future.

I talked briefly with Armin at EuroPython about Stackless and JIT. 
I poke around pypy-dev and use stackless.py. However I am very interested
in learning about how the stackless transform works and how pypy works so
I could help. I have been at Stackless for about five years now and I wouldn't mind spending a year learning pypy.

Cheers,
Andrew





      


From p.giarrusso at gmail.com  Thu Sep 30 08:35:22 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 30 Sep 2010 08:35:22 +0200
Subject: [pypy-dev] Fwd:  Question on the future of RPython
In-Reply-To: <1285809701.5129.150.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
Message-ID: <AANLkTimhRRVVMSQFmPvrZP=wXD5kCmtJ2FO0QxqP+=Lp@mail.gmail.com>

I'm forwarding this mail that Terrence Cole sent me privately for no
reason apparent to me, because without it my mail makes less sense.

---------- Forwarded message ----------
From: Terrence Cole <list-sink at trainedmonkeystudios.org>
Date: Thu, Sep 30, 2010 at 03:21
Subject: Re: [pypy-dev] Question on the future of RPython
To: Paolo Giarrusso <p.giarrusso at gmail.com>


On Wed, 2010-09-29 at 23:50 +0200, Paolo Giarrusso wrote:
> The uselessness of static type analysis in dynamic languages was shown
> concretely in 1991 on a SmallTalk derivative, Self, the father of
> JavaScript, but I've seen the result only in the PhD thesis* of U.
> H?lzle, one of Self authors - and I'm pointing it out because I'm not
> sure how well known it is.

I've read parts of that at some point in the past.

> That might explain why Brett Canon's PhD thesis was not that
> successful. Of course, after reading his thesis you might be able in
> theory to address his concerns, and he acknowledges that type analysis
> might help with a few specific cases. I'm not sure about Brendan
> Eich's points, but I don't have the time, unfortunately, to
> investigate them.

Dr. Canon achieved an ~1% performance improvement. ?From his conclusion,
the most significant problem he found was that most program data is
stored in object attributes and object attributes were not
type-inferable with the methods he used.

> * Title: "Adaptive optimization for Self: Reconciling High Performance
> with Exploratory Programming". See, among others, sec. 7.2, "Type
> analysis does not improve performance of object-oriented programs",
> sec. 7.4.1, "type analysis exhibits unstable performance". The
> conclusion is that in most cases it helps more to use ?"type feedback"
> , ?indicates profile-guided optimization applied

Scanning through the 7.2, they say:

"In general, type analysis cannot infer the result types of non-inlined
message sends, of arguments of non-inlined sends, or of assignable slots
(i.e., instance variables). Since sends to such values are fairly
frequent, a large fraction of message sends does not benefit from type
analysis."

So they seem to have run into similar problems: the type inferencing
algorithm doesn't get to sink its teeth into any of the bits that most
need it.

Chapter3 of Cannon05 details what specific parts of python can't be
type-inferred. ?It turns out this is most of python. ?For example, type
inferencing can't cross module boundaries because the module used at
runtime might be different from the one present at compile time. ?While
perfectly true, this is probably going to be quite rare in practice.

I think it makes sense to treat the sorts of optimizations you can do
with static analysis of a dynamic language with the same sort of guarded
optimism that is used in pypy's jit compiler: run with the assumption
that it will all work out, but watch for failure and fallback to a safe
slow-path nicely when things go off the rails.

> See below further inline replies.
>
> On Tue, Sep 28, 2010 at 21:49, Terrence Cole
> <list-sink at trainedmonkeystudios.org> wrote:
> > I think this is a disconnect. ?Applying a jit to a non-interpretted
> > language -- Jacob here seems to think I was talking about a static,
> > compiled subset of python -- makes little sense.
>
> JIT just makes much _more_ sense for dynamic languages. Profile-guided
> optimization (PGO) give impressive performance improvements for C, but
> can also worsen performance when the execution profile changes
> (because of different inputs), and JIT would solve this problem. Since
> we are talking about optimized C, "impressive performance
> improvements" is more like 20% more (so to say) than like 10x-100x,
> and it's mostly about things like static branch prediction.
> For RPython, it makes more sense to reuse the support from the
> translation backend (JVM and .NET should do this by default, in C you
> can use PGO on the translation output.
>
> >> There are some undesirable things about static analysis, but it can
> >> sure be useful from optimisation, security and reliability
> >> perspectives.
> >
> > Brendan Eich agrees [2]. ?This is heartening, because javascript has
> > much in common with python.
>
> > [1] http://www.ocf.berkeley.edu/~bac/thesis.pdf
> > [2] http://brendaneich.com/2010/08/static-analysis-ftw/
>





-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From p.giarrusso at gmail.com  Thu Sep 30 08:33:02 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 30 Sep 2010 08:33:02 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285809701.5129.150.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
Message-ID: <AANLkTinWMUs-xPYxsDiNMTSX13YZ7TLYr+xmZ5K=59+V@mail.gmail.com>

On Thu, Sep 30, 2010 at 03:21, Terrence Cole
<list-sink at trainedmonkeystudios.org> wrote:
> On Wed, 2010-09-29 at 23:50 +0200, Paolo Giarrusso wrote:

> I think it makes sense to treat the sorts of optimizations you can do
> with static analysis of a dynamic language with the same sort of guarded
> optimism that is used in pypy's jit compiler: run with the assumption
> that it will all work out, but watch for failure and fallback to a safe
> slow-path nicely when things go off the rails.

Agreed, but "watch out for failure" is done by guards, and one of the
further advantages of type inference, on top of "type feedback",
described there, is being able to remove some of those guards, because
that might help in some inner loops.
In some cases, that's still possible, by "watching out" not during
code execution but during execution of infrequent events. In Java
loading a class might invalidate some optimization assumptions (like
"this class has no subclass", useful for inlining without guards), but
that's checked by the classloading fast-path.

Same things could apply to allow Python cross-module type-inferencing.
I don't know exactly how modules at JIT compile-time and runtime can
be different, but I guess that invalidation at module loading should
catch that. And invalidate lots of compiled code, which is usually
fine. The interaction of this with tracing is actually interesting: in
a Python tracing JIT, you could keep the traces and restore omitted
guards, but when your JIT traces the Python interpreter, I wonder how
do you express any of this. One can insert a guard only if needed, but
telling "hey, this is invalid" requires some special API.

My proposal, here, would be a "virtual guard", which is recorded in
the guard but omitted from the output. The "omission" is what can be
invalidated, but the trace itself (not its compiled version) is kept
(because it can still be executed).
If some form of this makes actually sense (I've just thought on it 5
minutes), this is something worth publishing, which would allow me to
take some time of my PhD to work on it - if it's not already done by
papers on tracing JITs.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From holger at merlinux.eu  Thu Sep 30 09:37:46 2010
From: holger at merlinux.eu (holger krekel)
Date: Thu, 30 Sep 2010 09:37:46 +0200
Subject: [pypy-dev] Python3,
	Python2.7 - fast-forward (was Re: Question on	the future of RPython)
In-Reply-To: <1285792859.5129.79.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost> <4CA308C1.5030200@gmail.com>
	<1285792859.5129.79.camel@localhost>
Message-ID: <20100930073746.GL20695@trillke.net>

Hi Terence, all,

On Wed, Sep 29, 2010 at 13:40 -0700, Terrence Cole wrote:
> So what I'm getting at is: what is the pypy story for python3 support?
> I haven't seen anything on pypy-dev, or in my occasional looks at the
> repository, to suggest that it is being worked on but I'm sure you have
> a plan of some sort.  I'm willing to help out with python3 support, if I
> can do so without getting in anyone's way.  It seems like the sort of
> thing that will be disruptive, however, so I have been leery of jumping
> in, considering how little time I have to contribute, at the moment.

In fact, there has been work from Benjamin Peterson and is some 
work from Amaury and Alex to complete the

    http://codespeak.net/svn/pypy/branch/fast-forward/

branch.  It aims at offering Python2.7 compatibility.  This is a
good intermediate step to jump to Python3 at some point.  Most
PyPy core devs are focusing on JIT related tasks so this is 
a good place to help out in general.  

If you like to help you can drop by at #pypy on freenode and/or maybe
some of the involved persons can point to some tasks here. 

cheers,
holger

> In my mind, the python3 picture is something like:
> At the compilation level, it's easy enough to dump Grammar3.2 in
> pypy/interpreter/pyparser/data and to modify astbuilder for python3 --
> I'll backport the changes I made, if you want.  Interpreter support for
> the new language features will be harder, but that's probably already
> done since 2.7 is almost working.  The only potential problems I see are
> the big string/unicode switch and the management of the fairly large
> changes to astbuilder -- I'm sure you want to continue supporting
> python2 into the future.  I don't know how much the bytecode changed
> between 2 and 3, so I'm not sure if there are jit issues to worry about.
> Am I missing anything big?
> 
> 
> -Terrence
> 
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 


From fijall at gmail.com  Thu Sep 30 10:51:01 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 30 Sep 2010 10:51:01 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285792859.5129.79.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost> <4CA308C1.5030200@gmail.com>
	<1285792859.5129.79.camel@localhost>
Message-ID: <AANLkTikgTrOeoKS1Ysdx+KMNRQGavxQD1eP=X2ZGiTYz@mail.gmail.com>

[not answering the rest]

> Am I missing anything big?

Standard library is usually the biggest thing when porting from one
version of python to another. Other big issues are about RPython
itself. Do we want RPython to be python3 compatible? How?


From arigo at tunes.org  Thu Sep 30 12:05:10 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Sep 2010 12:05:10 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTikgTrOeoKS1Ysdx+KMNRQGavxQD1eP=X2ZGiTYz@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>
	<1285705994.5276.99.camel@localhost> <4CA308C1.5030200@gmail.com>
	<1285792859.5129.79.camel@localhost>
	<AANLkTikgTrOeoKS1Ysdx+KMNRQGavxQD1eP=X2ZGiTYz@mail.gmail.com>
Message-ID: <AANLkTi=+Q0Qmcu=CMF0eyZUko4wUAC-urzf0V+CO92ac@mail.gmail.com>

Hi Maciej,

On Thu, Sep 30, 2010 at 10:51 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Other big issues are about RPython
> itself. Do we want RPython to be python3 compatible? How?

No, I'm pretty sure that even if we want to support python3 at some
point, RPython will remain what it is now, and translate.py will
remain a python2 tool.


Armin


From arigo at tunes.org  Thu Sep 30 13:01:59 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Sep 2010 13:01:59 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTinWMUs-xPYxsDiNMTSX13YZ7TLYr+xmZ5K=59+V@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
	<AANLkTinWMUs-xPYxsDiNMTSX13YZ7TLYr+xmZ5K=59+V@mail.gmail.com>
Message-ID: <AANLkTikAeDUhD34M0Xxaoavr+DMYt63DhGcgRNg9oQZF@mail.gmail.com>

Hi Paolo,

On Thu, Sep 30, 2010 at 8:33 AM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> My proposal, here, would be a "virtual guard", (...)

Yes, this proposal makes sense.  It's an optimization that is
definitely done in regular JITs, and we have a "to-do" task about it
in http://codespeak.net/svn/pypy/extradoc/planning/jit.txt (where they
are called "out-of-line guards").


Armin.


From arigo at tunes.org  Thu Sep 30 13:23:24 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Sep 2010 13:23:24 +0200
Subject: [pypy-dev] Fwd: Question on the future of RPython
In-Reply-To: <AANLkTimhRRVVMSQFmPvrZP=wXD5kCmtJ2FO0QxqP+=Lp@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
	<AANLkTimhRRVVMSQFmPvrZP=wXD5kCmtJ2FO0QxqP+=Lp@mail.gmail.com>
Message-ID: <AANLkTi==Cd9FWok6GEKkjvscHu=FXBFPDH7c9XKuoYSm@mail.gmail.com>

Hi Terrence,

I think that what you are describing is found in informal discussions
about LLVM/HLVM, and more formally in the plans for Unladen Swallow at
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan .  See in
particular the section about Feedback-Directed Optimization.  Maybe
you want to discuss these ideas with the Unladen Swallow guys instead
:-)


Armin.


From arigo at tunes.org  Thu Sep 30 14:25:53 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Sep 2010 14:25:53 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <69092.15439.qm@web120710.mail.ne1.yahoo.com>
References: <mailman.1036.1285616701.6189.pypy-dev@codespeak.net>
	<69092.15439.qm@web120710.mail.ne1.yahoo.com>
Message-ID: <AANLkTinpPFfWQyZHb0QMUCw3tXdFMgmdp4FE3xyBRzt3@mail.gmail.com>

Hi,

On Wed, Sep 29, 2010 at 11:39 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:
> I talked briefly with Armin at EuroPython about Stackless and JIT.
> I poke around pypy-dev and use stackless.py. However I am very interested
> in learning about how the stackless transform works and how pypy works so
> I could help. I have been at Stackless for about five years now and I wouldn't mind
> spending a year learning pypy.

Maybe I should expand on an idea posted on #pypy by fijal.  He
mentioned that he would like to try to support Stackless in PyPy
without using the stackless transform, just by using the same
low-level stack hacks that are done by greenlet.c and optionally by
Stackless Python.  This means that there would be two different
approaches we can consider to support Stackless in PyPy:

    stackless transform (done)          C-level stack switching
    --------------------------------    --------------------------------

    approach from Stackless Python 1    approach from Stackless Python 2

    10-20% speed penalty in the         no speed penalty
    whole interpreter

    JIT support needed                  JIT supports comes for free
    (missing so far)

    fully portable                      needs a little bit of assembler

    some issues to integrate with       easy to integrate with C code
    non-PyPy C code

    tasklet-switching Python code       tasklet-switching Python code
    becomes a single loop in            becomes N loops with residual calls
    machine code                        to switch() functions
    (potentially very good)             (less good)


As you can see from the above summary, the main issue with the 2nd
approach would be that Python tasklet-switching loops do not turn into
a form that is as efficient as what we would get in the 1st approach.
Nevertheless it is an interesting approach because it makes basic JIT
support and integration easier.


A bient?t,

Armin.


From anto.cuni at gmail.com  Thu Sep 30 14:32:01 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 30 Sep 2010 14:32:01 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <1285792859.5129.79.camel@localhost>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>	
	<827221.6704.qm@web53704.mail.re2.yahoo.com>	
	<1285616691.5954.34.camel@localhost>	<201009280157.17264.jacob@openend.se>	
	<1285634613.5954.112.camel@localhost>	
	<AANLkTinyNYSS_ho-SnSXJdv7KiAx47PYvO0dsoqvmOw8@mail.gmail.com>	
	<1285705994.5276.99.camel@localhost> <4CA308C1.5030200@gmail.com>
	<1285792859.5129.79.camel@localhost>
Message-ID: <4CA48341.3020106@gmail.com>

On 29/09/10 22:40, Terrence Cole wrote:

>> then, mylog contains all the loops and bridges produced by the jit. The
>> interesting point is that there are also special operations called
>> "debug_merge_point" that are emitted for each python bytecode, so you can
>> easily map the low-level jit instructions back to the original python source.
>
> I think that 'easily' in that last sentence is missing scare-quotes. :-)

well, it's easy as long as you have a bytecode-compiled version around. With 
only the AST I agree that it might be a bit trickier.

[cut]
> My first inclination would be to continue this chain and add a bytecode
> compiler on top of the ast builder.  This would keep ast node references
> in the instructions it creates.  If the algorithms don't diverge too
> much, I think this would allow the debug output to be mapped all the way
> back to the source chars with minimal effort.  I'm not terrifically
> familiar with the specifics of how python emits bytecode from an ast, so
> I'd appreciate any feedback if you think this is crazy-talk.

Are you using your custom-made AST or the one from the standard library? In 
the latter case, you can just pass the ast to the compile() builtin function 
to get the corresponding bytecode.



From arigo at tunes.org  Thu Sep 30 14:55:15 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Sep 2010 14:55:15 +0200
Subject: [pypy-dev] PyPy JIT & C extensions, greenlet
In-Reply-To: <AANLkTinpPFfWQyZHb0QMUCw3tXdFMgmdp4FE3xyBRzt3@mail.gmail.com>
References: <mailman.1036.1285616701.6189.pypy-dev@codespeak.net>
	<69092.15439.qm@web120710.mail.ne1.yahoo.com>
	<AANLkTinpPFfWQyZHb0QMUCw3tXdFMgmdp4FE3xyBRzt3@mail.gmail.com>
Message-ID: <AANLkTikb0QXiDy3Mu3=_K7s7UN8Aen0iN09Z6ZeOiitz@mail.gmail.com>

Re-hi,

I forgot to mention that the improvement over the "stackless
transform" approach for PyPy might be to not apply the stackless
transform on the interpreter, but replace it with resuming C code
using the blackhole interpreter that we have anyway in the JIT.

Also, any "final" long-term approach that anyone should at least
consider if taking all of this seriously might be a mix of the two
approaches, similar to Stackless Python 3 which combines both
Stackless Python 1 and Stackless Python 2 features.


Armin


From renesd at gmail.com  Thu Sep 30 16:35:05 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Thu, 30 Sep 2010 16:35:05 +0200
Subject: [pypy-dev] Question on the future of RPython
In-Reply-To: <AANLkTikAeDUhD34M0Xxaoavr+DMYt63DhGcgRNg9oQZF@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
	<AANLkTinWMUs-xPYxsDiNMTSX13YZ7TLYr+xmZ5K=59+V@mail.gmail.com>
	<AANLkTikAeDUhD34M0Xxaoavr+DMYt63DhGcgRNg9oQZF@mail.gmail.com>
Message-ID: <AANLkTikPxqO3+bQLFJnQLw_h9xTw6WtBZ6QSspr=xU_J@mail.gmail.com>

Hi,

on the topic of optimizations, since I should get these out of my
brain, but you probably don't want them in yours... but I'll write
them anyway.

Considered Duffs device for loop unwinding?? For dividing the number
of loop counter ++ calls by 8 times or more:
http://en.wikipedia.org/wiki/Duff%27s_Device? I'm guessing pypy
already does this or something better.

Outputting gcc extensions, and pragmas, for example using
__builtin_expect to help branch prediction:
#define likely(x)       __builtin_expect((x),1)
#define unlikely(x)     __builtin_expect((x),0)

As well as rectrict to tell it about pointer aliasing?
http://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Restricted-Pointers.html


Also, have you considered using PREFETCH* (or gccs __builtin_prefetch)
instructions when you are iterating over sequences?  It might be a win
If you know there is some memory coming, and can slip in some of these
instructions, it's usually a win.
    http://gcc.gnu.org/projects/prefetch.html

__builtin_constant_p for constant detection?

SSE2 optimized hash functions?  It seems this is a big speedup for
interpreters when the hash function is sped up... I guess pypy already
uses an inline cache, but I hope it would still speed things up.



... almost deleted this email, since I hate suggesting things that
people might like to work on... but I didn't. oops, sorry.



On Thu, Sep 30, 2010 at 1:01 PM, Armin Rigo <arigo at tunes.org> wrote:
>
> Hi Paolo,
>
> On Thu, Sep 30, 2010 at 8:33 AM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> > My proposal, here, would be a "virtual guard", (...)
>
> Yes, this proposal makes sense. ?It's an optimization that is
> definitely done in regular JITs, and we have a "to-do" task about it
> in http://codespeak.net/svn/pypy/extradoc/planning/jit.txt (where they
> are called "out-of-line guards").
>
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From list-sink at trainedmonkeystudios.org  Thu Sep 30 22:01:11 2010
From: list-sink at trainedmonkeystudios.org (Terrence Cole)
Date: Thu, 30 Sep 2010 13:01:11 -0700
Subject: [pypy-dev] Fwd:  Question on the future of RPython
In-Reply-To: <AANLkTimhRRVVMSQFmPvrZP=wXD5kCmtJ2FO0QxqP+=Lp@mail.gmail.com>
References: <525425.61205.qm@web53702.mail.re2.yahoo.com>
	<827221.6704.qm@web53704.mail.re2.yahoo.com>
	<1285616691.5954.34.camel@localhost>
	<201009280157.17264.jacob@openend.se>
	<1285634613.5954.112.camel@localhost>
	<AANLkTi=JGkkZOB0uTmD1rsD9zN=qd-YrT8pTkycE9KMF@mail.gmail.com>
	<1285703348.5276.56.camel@localhost>
	<AANLkTinWUGEcGtzmAPbPmREBUDtBVpwYy3OpCdFTa0We@mail.gmail.com>
	<1285809701.5129.150.camel@localhost>
	<AANLkTimhRRVVMSQFmPvrZP=wXD5kCmtJ2FO0QxqP+=Lp@mail.gmail.com>
Message-ID: <1285876871.9991.7.camel@localhost>

On Thu, 2010-09-30 at 08:35 +0200, Paolo Giarrusso wrote:
> I'm forwarding this mail that Terrence Cole sent me privately for no
> reason apparent to me, because without it my mail makes less sense.

I replied to you privately because the mail from you that I was replying
to did not have the list cc'd.  I checked the headers several times to
be sure that my mail client was not lying to me.  I thought that you
wanted to take the ensuing discussion off list, as it is starting to
range even farther off topic for pypy-dev than it was before.

Oh well.  Sorry for the noise.

> ---------- Forwarded message ----------
> From: Terrence Cole <list-sink at trainedmonkeystudios.org>
> Date: Thu, Sep 30, 2010 at 03:21
> Subject: Re: [pypy-dev] Question on the future of RPython
> To: Paolo Giarrusso <p.giarrusso at gmail.com>
> 
> 
> On Wed, 2010-09-29 at 23:50 +0200, Paolo Giarrusso wrote:
> > The uselessness of static type analysis in dynamic languages was shown
> > concretely in 1991 on a SmallTalk derivative, Self, the father of
> > JavaScript, but I've seen the result only in the PhD thesis* of U.
> > H?lzle, one of Self authors - and I'm pointing it out because I'm not
> > sure how well known it is.
> 
> I've read parts of that at some point in the past.
> 
> > That might explain why Brett Canon's PhD thesis was not that
> > successful. Of course, after reading his thesis you might be able in
> > theory to address his concerns, and he acknowledges that type analysis
> > might help with a few specific cases. I'm not sure about Brendan
> > Eich's points, but I don't have the time, unfortunately, to
> > investigate them.
> 
> Dr. Canon achieved an ~1% performance improvement.  From his conclusion,
> the most significant problem he found was that most program data is
> stored in object attributes and object attributes were not
> type-inferable with the methods he used.
> 
> > * Title: "Adaptive optimization for Self: Reconciling High Performance
> > with Exploratory Programming". See, among others, sec. 7.2, "Type
> > analysis does not improve performance of object-oriented programs",
> > sec. 7.4.1, "type analysis exhibits unstable performance". The
> > conclusion is that in most cases it helps more to use  "type feedback"
> > ,  indicates profile-guided optimization applied
> 
> Scanning through the 7.2, they say:
> 
> "In general, type analysis cannot infer the result types of non-inlined
> message sends, of arguments of non-inlined sends, or of assignable slots
> (i.e., instance variables). Since sends to such values are fairly
> frequent, a large fraction of message sends does not benefit from type
> analysis."
> 
> So they seem to have run into similar problems: the type inferencing
> algorithm doesn't get to sink its teeth into any of the bits that most
> need it.
> 
> Chapter3 of Cannon05 details what specific parts of python can't be
> type-inferred.  It turns out this is most of python.  For example, type
> inferencing can't cross module boundaries because the module used at
> runtime might be different from the one present at compile time.  While
> perfectly true, this is probably going to be quite rare in practice.
> 
> I think it makes sense to treat the sorts of optimizations you can do
> with static analysis of a dynamic language with the same sort of guarded
> optimism that is used in pypy's jit compiler: run with the assumption
> that it will all work out, but watch for failure and fallback to a safe
> slow-path nicely when things go off the rails.
> 
> > See below further inline replies.
> >
> > On Tue, Sep 28, 2010 at 21:49, Terrence Cole
> > <list-sink at trainedmonkeystudios.org> wrote:
> > > I think this is a disconnect.  Applying a jit to a non-interpretted
> > > language -- Jacob here seems to think I was talking about a static,
> > > compiled subset of python -- makes little sense.
> >
> > JIT just makes much _more_ sense for dynamic languages. Profile-guided
> > optimization (PGO) give impressive performance improvements for C, but
> > can also worsen performance when the execution profile changes
> > (because of different inputs), and JIT would solve this problem. Since
> > we are talking about optimized C, "impressive performance
> > improvements" is more like 20% more (so to say) than like 10x-100x,
> > and it's mostly about things like static branch prediction.
> > For RPython, it makes more sense to reuse the support from the
> > translation backend (JVM and .NET should do this by default, in C you
> > can use PGO on the translation output.
> >
> > >> There are some undesirable things about static analysis, but it can
> > >> sure be useful from optimisation, security and reliability
> > >> perspectives.
> > >
> > > Brendan Eich agrees [2].  This is heartening, because javascript has
> > > much in common with python.
> >
> > > [1] http://www.ocf.berkeley.edu/~bac/thesis.pdf
> > > [2] http://brendaneich.com/2010/08/static-analysis-ftw/
> >
> 
> 
> 
> 
> 
> -- 
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev