From andrewfr_ice at yahoo.com  Mon Oct  1 20:32:37 2012
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Mon, 1 Oct 2012 11:32:37 -0700 (PDT)
Subject: [pypy-dev] PyPy STM
In-Reply-To: <50685A78.50908@gmx.de>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<50685A78.50908@gmx.de>
Message-ID: <1349116357.80146.YahooMailNeo@web140703.mail.bf1.yahoo.com>

Hi Ronny:



________________________________
 From: Ronny Pfannschmidt <Ronny.Pfannschmidt at gmx.de>
To: Armin Rigo <arigo at tunes.org> 
Cc: Sarah Mount <s.mount at wlv.ac.uk>; PyPy_Developers <pypy-dev at python.org> 
Sent: Sunday, September 30, 2012 10:43 AM
Subject: Re: [pypy-dev] PyPy STM
 
>the reason why i start with generators instead of green-lets is simply 
>cause they cannot ever be nested.

I would hold the reverse view since to the best of my knowledge, generators
don't lend themselves to composability. I suspect greenlets (or continuelets?)
are easier to work with. 

>i think for most purposes simple sequential communicating programs
>are way more easy to reason about than anything else.

That is why I use Stackless Python.... And I would love to try to incorporate STM into Stackless.

I recently read a book by John Reppy on Concurrent ML. In it, Reppy gives the theoretical reasons 
about the virtues of synchronous channels with buffers. 

You may want to look at John Reppy's paper "Parallel Concurrent ML." ( http://people.cs.uchicago.edu/~jhr/papers/2009/icfp-parallel-cml.pdf). Under the hood, Reppy and company use optimistic locking to implement efficient message passing on a multicore system. Another paper you should check out is "Scalable Join Patterns" by Russo and Turon ( I exchanged e-mails with these guys). 

I'll admit, I don't understand all the ins-and-outs of the papers. However my takeaway is that STM and STMish mechanisms are used in a very small place: the message passing system's implementation. Given that message passing systems typically share information through messages :-), this ought to create a very small footprint for transaction conflicts. 

>the transaction module approach already seems to require to chunk up 
>programs in semi-small transactions,
>that may cause other transactions to be scheduled

>which seems more and more like twisted's defereds.
>to my eyes twisted style code is a kind of spaghetti
>that is very hard to reason about.

>which is why i want to experiment executing multiple longer sequential 
>programs in chunks that may be interleaved and/or parallel

As I understand it, a properly functioning STM implementation is going to make transactions
appear that they are serialised. As for interleaving. That is happening below the hood and is an 
efficiency and implementation? issue. One may use the scheduler to directly enforce serialisation.
(I have to review my notes for how STM aware is the Haskell runtime scheduler). 

Cheers,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121001/0f58e5f6/attachment.html>

From Ronny.Pfannschmidt at gmx.de  Mon Oct  1 20:51:05 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Mon, 01 Oct 2012 20:51:05 +0200
Subject: [pypy-dev] PyPy STM
In-Reply-To: <1349116357.80146.YahooMailNeo@web140703.mail.bf1.yahoo.com>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<50685A78.50908@gmx.de>
	<1349116357.80146.YahooMailNeo@web140703.mail.bf1.yahoo.com>
Message-ID: <5069E619.4080505@gmx.de>

On 10/01/2012 08:32 PM, Andrew Francis wrote:
> Hi Ronny:
>
> ------------------------------------------------------------------------
> *From:* Ronny Pfannschmidt <Ronny.Pfannschmidt at gmx.de>
> *To:* Armin Rigo <arigo at tunes.org>
> *Cc:* Sarah Mount <s.mount at wlv.ac.uk>; PyPy_Developers
> <pypy-dev at python.org>
> *Sent:* Sunday, September 30, 2012 10:43 AM
> *Subject:* Re: [pypy-dev] PyPy STM
>
>  >the reason why i start with generators instead of green-lets is simply
>  >cause they cannot ever be nested.
>
> I would hold the reverse view since to the best of my knowledge, generators
> don't lend themselves to composability. I suspect greenlets (or
> continuelets?)
> are easier to work with.

practically there is no semantic difference between
yielding a request for completion of "another process"
and calling into a function

its practically equivalent, the "calls" just look different

>
>  >i think for most purposes simple sequential communicating programs
>  >are way more easy to reason about than anything else.
>
> That is why I use Stackless Python.... And I would love to try to
> incorporate STM into Stackless.
no point in pushing that into old stackless
once continulet-jit-3 works as we hope to, "continuations"
will have distinct stacks (Armin please correct if i missed )
>
> I recently read a book by John Reppy on Concurrent ML. In it, Reppy
> gives the theoretical reasons
> about the virtues of synchronous channels with buffers.

channels with buffers will be one of my primitives

>
> You may want to look at John Reppy's paper "Parallel Concurrent ML." (
> http://people.cs.uchicago.edu/~jhr/papers/2009/icfp-parallel-cml.pdf).
> Under the hood, Reppy and company use optimistic locking to implement
> efficient message passing on a multicore system. Another paper you
> should check out is "Scalable Join Patterns" by Russo and Turon ( I
> exchanged e-mails with these guys).
>
> I'll admit, I don't understand all the ins-and-outs of the papers.
> However my takeaway is that STM and STMish mechanisms are used in a very
> small place: the message passing system's implementation. Given that
> message passing systems typically share information through messages
> :-), this ought to create a very small footprint for transaction conflicts.

stm will be everywhere in pypy because we need it everywhere
to have it appear as if we still have the gil

>
>  >the transaction module approach already seems to require to chunk up
>  >programs in semi-small transactions,
>  >that may cause other transactions to be scheduled
>
>  >which seems more and more like twisted's defereds.
>  >to my eyes twisted style code is a kind of spaghetti
>  >that is very hard to reason about.
>
>  >which is why i want to experiment executing multiple longer sequential
>  >programs in chunks that may be interleaved and/or parallel
>
> As I understand it, a properly functioning STM implementation is going
> to make transactions
> appear that they are serialised. As for interleaving. That is happening
> below the hood and is an
> efficiency and implementation issue. One may use the scheduler to
> directly enforce serialisation.
> (I have to review my notes for how STM aware is the Haskell runtime
> scheduler).
>

see Armin's latest blog entry on stm
its no longer completely under the hood,
but the transaction module will be
a convenient interface to it for some kinds of program

the base primitives will be thread.atomic/thread.exclusive_atomic

my experiment will touch it from a different angle

-- Ronny

From bokr at oz.net  Tue Oct  2 13:40:09 2012
From: bokr at oz.net (Bengt Richter)
Date: Tue, 02 Oct 2012 13:40:09 +0200
Subject: [pypy-dev] PyPy STM
In-Reply-To: <CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
Message-ID: <506AD299.8010301@oz.net>

On 09/30/2012 04:22 PM Armin Rigo wrote:
> Hi Ronny,
>
> On Sun, Sep 30, 2012 at 3:50 PM, Ronny Pfannschmidt
> <Ronny.Pfannschmidt at gmx.de>  wrote:
>> after my thesis i'll be experimenting with a relaxed csp-ish model
>> based on python native generator based continuations as well as
>> the new continulet-jit-3 based greenlets.
>>
>> my basic assumption is that having limited amount
>> of shared memory is acceptable.
>
> What you are thinking about is to start from the naturally multicore
> model of separate address spaces, and add some amount of shared
> memory.  You would use STM to handle the result.  It is the opposite
> of what I'm thinking about, which is to start with a non-multithread,
> non-tasklet-based program and add multicore capability to it.  I would
> be using STM to "create" multicore capability, whereas you would be
> using it to "create" shared memory.  I am more interested in the first
> approach than the second because I think it is closer to what
> untrained programmers start with, but both approaches are potentially
> valid.
>
> Russel: STM is a powerful tool that makes sense of shared memory in
> multicore situations.  I fail to understand why you are looking at it
> in the absence of shared memory...
>
>
> A bient?t,
>
> Armin.

Hi Armin,

Just a triggered thought: I am wondering if Conway's Game of Life
     http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
might be be an interesting/fun basis for experiments and maybe a benchmark
for STM use in the parallel update of the life grid.

I am thinking of in-place update, since old-frame -> new-frame would
not create any conflicts.

A naive single-thread in-place update would obviously require
local neighbor "old-frame" values to calculate correct updates,
so a programmer would naturally have to provide for that somehow.

What kinds of design patterns wrt STM would emerge from solving the problem in various
loops that create different interferences when parallelized, e.g., looping by rows
vs looping recursively through tiles and subtiles, vs maybe using a quadtree representation
or whatever?

I am wondering how you would visualize STM helping in bringing either manual or
automated parallelism into this update problem, say with a goal of maximizing
frame rate.

And what do you expect the "untrained programmers" to think of, and to be able to avoid
thinking about, because of STM, in their designs.

Regards,
Bengt Richter



From arigo at tunes.org  Thu Oct  4 14:53:19 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 4 Oct 2012 14:53:19 +0200
Subject: [pypy-dev] PyPy STM
In-Reply-To: <506AD299.8010301@oz.net>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<506AD299.8010301@oz.net>
Message-ID: <CAMSv6X3j+hGpeE+dSTqTR6=XzN_AMc8LFbv7J2H=GFyn6HOmrA@mail.gmail.com>

Hi Bengt,

On Tue, Oct 2, 2012 at 1:40 PM, Bengt Richter <bokr at oz.net> wrote:
> Just a triggered thought: I am wondering if Conway's Game of Life
>     http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
> might be be an interesting/fun basis for experiments and maybe a benchmark
> for STM use in the parallel update of the life grid.
>
> I am thinking of in-place update, since old-frame -> new-frame would
> not create any conflicts.

I suppose the boring answer is that old-frame -> new-frame looks more
natural.  Even if not, I can think about different answers, but it
doesn't seem that STM as I think about it is really related to them.
STM in PyPy is merely an "implementation detail" to speed up a
GIL-like user experience.


Armin

From andrewfr_ice at yahoo.com  Thu Oct  4 18:14:56 2012
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Thu, 4 Oct 2012 09:14:56 -0700 (PDT)
Subject: [pypy-dev] PyPy STM
In-Reply-To: <5069E619.4080505@gmx.de>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<50685A78.50908@gmx.de>
	<1349116357.80146.YahooMailNeo@web140703.mail.bf1.yahoo.com>
	<5069E619.4080505@gmx.de>
Message-ID: <1349367296.75039.YahooMailNeo@web140702.mail.bf1.yahoo.com>

Hi Ronny:


________________________________
 From: Ronny Pfannschmidt <Ronny.Pfannschmidt at gmx.de>
To: Andrew Francis <andrewfr_ice at yahoo.com> 
Cc: Armin Rigo <arigo at tunes.org>; Sarah Mount <s.mount at wlv.ac.uk>; PyPy_Developers <pypy-dev at python.org> 
Sent: Monday, October 1, 2012 2:51 PM
Subject: Re: [pypy-dev] PyPy STM
 
On 10/01/2012 08:32 PM, Andrew Francis wrote:

AF> That is why I use Stackless Python.... And I would love to try to
AF> incorporate STM into Stackless.


>no point in pushing that into old stackless
>once continulet-jit-3 works as we hope to, "continuations"
>will have distinct stacks (Armin please correct if i missed )

I view Stackless Python and its?API as functioning at a higher?level?than continuelets?or some other low?
level concurrency?mechanism. As an example, look at the old stackless.py module that supported both greenlets
and coroutines.?

Maybe Stackless a few years from now, will be Stackless in API only rather than under the hood
mechanisms?


AF> I'll admit, I don't understand all the ins-and-outs of the papers.
AF> However my takeaway is that STM and STMish mechanisms are used in a very
AF> small place: the message passing system's implementation. Given that
AF> message passing systems typically share information through messages
AF> :-), this ought to create a very small footprint for transaction conflicts.

>stm will be everywhere in pypy because we need it everywhere
>to have it appear as if we still have the gil

Yes the STM mechanism will be everywhere. ?However I would suspect that one would still want
to write programmes that would result in fewer transaction conflicts and redo/undo work. It would be nice
to have programming constructs to do this (I guess atomic is one of them).?Message passing is another mechanism.?
Message passing is a fairly easy?model to understand.?

Again, my point is that the literature points to using STM and STM like features to build more efficient?
message passing systems. It is this line of thinking that I feel is worth pursuing. I don't know how the PyCSP
folks feel about this?

If you want to see an example of the headaches of using fine-grained concurrency control to implement
channels, look at Go source code (i.e, you will see stuff like sorting locks on channel data structures).

Cheers,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121004/387f1b49/attachment.html>

From Ronny.Pfannschmidt at gmx.de  Thu Oct  4 19:29:29 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Thu, 04 Oct 2012 19:29:29 +0200
Subject: [pypy-dev] PyPy STM
In-Reply-To: <1349367296.75039.YahooMailNeo@web140702.mail.bf1.yahoo.com>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<50685A78.50908@gmx.de>
	<1349116357.80146.YahooMailNeo@web140703.mail.bf1.yahoo.com>
	<5069E619.4080505@gmx.de>
	<1349367296.75039.YahooMailNeo@web140702.mail.bf1.yahoo.com>
Message-ID: <506DC779.7080206@gmx.de>

On 10/04/2012 06:14 PM, Andrew Francis wrote:
> Hi Ronny:
>
> ------------------------------------------------------------------------
> *From:* Ronny Pfannschmidt <Ronny.Pfannschmidt at gmx.de>
> *To:* Andrew Francis <andrewfr_ice at yahoo.com>
> *Cc:* Armin Rigo <arigo at tunes.org>; Sarah Mount <s.mount at wlv.ac.uk>;
> PyPy_Developers <pypy-dev at python.org>
> *Sent:* Monday, October 1, 2012 2:51 PM
> *Subject:* Re: [pypy-dev] PyPy STM
>
> On 10/01/2012 08:32 PM, Andrew Francis wrote:
>
> AF> That is why I use Stackless Python.... And I would love to try to
> AF> incorporate STM into Stackless.
>
>  >no point in pushing that into old stackless
>  >once continulet-jit-3 works as we hope to, "continuations"
>  >will have distinct stacks (Armin please correct if i missed )
>
> I view Stackless Python and its API as functioning at a higher level
> than continuelets or some other low
> level concurrency mechanism. As an example, look at the old stackless.py
> module that supported both greenlets
> and coroutines.
>
> Maybe Stackless a few years from now, will be Stackless in API only
> rather than under the hood
> mechanisms?
>
as far as i understood, that will be the state once the continulet-jit-3 
branch is done
it just gives each continuation a own stack, the rest is api
>
> AF> I'll admit, I don't understand all the ins-and-outs of the papers.
> AF> However my takeaway is that STM and STMish mechanisms are used in a very
> AF> small place: the message passing system's implementation. Given that
> AF> message passing systems typically share information through messages
> AF> :-), this ought to create a very small footprint for transaction
> conflicts.
>
>>stm will be everywhere in pypy because we need it everywhere
>  >to have it appear as if we still have the gil
>
> Yes the STM mechanism will be everywhere. However I would suspect that
> one would still want
> to write programmes that would result in fewer transaction conflicts and
> redo/undo work. It would be nice
> to have programming constructs to do this (I guess atomic is one of
> them).Message passing is another mechanism.
> Message passing is a fairly easy model to understand.

i think that some fundamentals (aka channels/queues) will be enough
the rest is a question of the programming models/patterns
it will be interesting to figure good/new ones

>
> Again, my point is that the literature points to using STM and STM like
> features to build more efficient
> message passing systems. It is this line of thinking that I feel is
> worth pursuing. I don't know how the PyCSP
> folks feel about this?
im under the impression that messages are to be used
to help with low conflict rates, but [[citation needed]]
i haven?t yet experimented with that enough
>
> If you want to see an example of the headaches of using fine-grained
> concurrency control to implement
> channels, look at Go source code (i.e, you will see stuff like sorting
> locks on channel data structures).


noted for in 2 months

-- Ronny

>
> Cheers,
> Andrew
>
>
>


From randall.leeds at gmail.com  Thu Oct  4 19:58:15 2012
From: randall.leeds at gmail.com (Randall Leeds)
Date: Thu, 4 Oct 2012 10:58:15 -0700
Subject: [pypy-dev] PyPy STM
In-Reply-To: <CAMSv6X3j+hGpeE+dSTqTR6=XzN_AMc8LFbv7J2H=GFyn6HOmrA@mail.gmail.com>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<506AD299.8010301@oz.net>
	<CAMSv6X3j+hGpeE+dSTqTR6=XzN_AMc8LFbv7J2H=GFyn6HOmrA@mail.gmail.com>
Message-ID: <CAAL6JQhJ=G0RJOSHRiB1vH+HwFSjxqU7pseN7PM2xn-kcEbYBg@mail.gmail.com>

On Thu, Oct 4, 2012 at 5:53 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Bengt,
>
> On Tue, Oct 2, 2012 at 1:40 PM, Bengt Richter <bokr at oz.net> wrote:
>> Just a triggered thought: I am wondering if Conway's Game of Life
>>     http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
>> might be be an interesting/fun basis for experiments and maybe a benchmark
>> for STM use in the parallel update of the life grid.
>>
>> I am thinking of in-place update, since old-frame -> new-frame would
>> not create any conflicts.
>
> I suppose the boring answer is that old-frame -> new-frame looks more
> natural.  Even if not, I can think about different answers, but it
> doesn't seem that STM as I think about it is really related to them.
> STM in PyPy is merely an "implementation detail" to speed up a
> GIL-like user experience.

In other words, the GIL is a point of contention even when the
application code is not sharing data between threads. Correct?

If we're looking for benchmark problems which are not trivially
parallelized, STAMP[0] is probably a great place to start.

[0] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.139.472

From smith195 at illinois.edu  Thu Oct  4 20:17:55 2012
From: smith195 at illinois.edu (Adam R. Smith)
Date: Thu, 4 Oct 2012 13:17:55 -0500
Subject: [pypy-dev] Research with PyPy Translation Toolchain
Message-ID: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>

I am a PhD candidate at the University of Illinois. I am interested in
learning more about how to use the PyPy Translation Toolchain as part of a
research project. We would like to write some code in Python and generate C
code from that. When using the Translation Toolchain, a *.c source file is
generated and then compiled with GCC. How do I find where the source files
are stored before GCC is called? I want to interface with those *.c files.

Adam Smith
Ph.D. Candidate
University of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121004/59ab7704/attachment.html>

From alex0player at gmail.com  Thu Oct  4 20:46:49 2012
From: alex0player at gmail.com (Alexander Sedov)
Date: Thu, 4 Oct 2012 22:46:49 +0400
Subject: [pypy-dev] Research with PyPy Translation Toolchain
In-Reply-To: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
References: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
Message-ID: <CAL7ARbRc=rFnGe1Ro7=sGw5x6tTpvFiKY7BYwgj-P9NAs3046w@mail.gmail.com>

2012/10/4 Adam R. Smith <smith195 at illinois.edu>:
> I am a PhD candidate at the University of Illinois. I am interested in
> learning more about how to use the PyPy Translation Toolchain as part of a
> research project. We would like to write some code in Python and generate C
> code from that. When using the Translation Toolchain, a *.c source file is
> generated and then compiled with GCC. How do I find where the source files
> are stored before GCC is called? I want to interface with those *.c files.
Look at /tmp/<some semirandom directory name>.
They're there, nicely named after your modules.

From fijall at gmail.com  Thu Oct  4 23:05:00 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 4 Oct 2012 23:05:00 +0200
Subject: [pypy-dev] Research with PyPy Translation Toolchain
In-Reply-To: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
References: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
Message-ID: <CAK5idxSMtJ5ZmRTpCqNgQgbNK8o51wjwPxn=YtLXWexH+8xU0g@mail.gmail.com>

On Thu, Oct 4, 2012 at 8:17 PM, Adam R. Smith <smith195 at illinois.edu> wrote:
> I am a PhD candidate at the University of Illinois. I am interested in
> learning more about how to use the PyPy Translation Toolchain as part of a
> research project. We would like to write some code in Python and generate C
> code from that. When using the Translation Toolchain, a *.c source file is
> generated and then compiled with GCC. How do I find where the source files
> are stored before GCC is called? I want to interface with those *.c files.
>
> Adam Smith
> Ph.D. Candidate
> University of Illinois at Urbana-Champaign

Hi Adam

Note that PyPy translation toolchain does not compile Python. It
compiles RPython, which is a restricted subset of Python. However, we
have a Python interpreter written in rPython.

From mradamsmith at gmail.com  Thu Oct  4 23:22:13 2012
From: mradamsmith at gmail.com (Adam R. Smith)
Date: Thu, 4 Oct 2012 16:22:13 -0500
Subject: [pypy-dev] Research with PyPy Translation Toolchain
In-Reply-To: <b4bafb1689bc4413baa9989050630cbc@CITESHT3.ad.uillinois.edu>
References: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
	<b4bafb1689bc4413baa9989050630cbc@CITESHT3.ad.uillinois.edu>
Message-ID: <CAK57ttromTh_na_6VXnstL08S_An+4kpuyYGa46VtDjXhvfH9Q@mail.gmail.com>

Is there any documentation available on what RPython consists of? I can't
seem to find a manual.


On Thu, Oct 4, 2012 at 4:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Thu, Oct 4, 2012 at 8:17 PM, Adam R. Smith <smith195 at illinois.edu>
> wrote:
> > I am a PhD candidate at the University of Illinois. I am interested in
> > learning more about how to use the PyPy Translation Toolchain as part of
> a
> > research project. We would like to write some code in Python and
> generate C
> > code from that. When using the Translation Toolchain, a *.c source file
> is
> > generated and then compiled with GCC. How do I find where the source
> files
> > are stored before GCC is called? I want to interface with those *.c
> files.
> >
> > Adam Smith
> > Ph.D. Candidate
> > University of Illinois at Urbana-Champaign
>
> Hi Adam
>
> Note that PyPy translation toolchain does not compile Python. It
> compiles RPython, which is a restricted subset of Python. However, we
> have a Python interpreter written in rPython.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121004/e034d925/attachment.html>

From fijall at gmail.com  Thu Oct  4 23:25:33 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 4 Oct 2012 23:25:33 +0200
Subject: [pypy-dev] Research with PyPy Translation Toolchain
In-Reply-To: <CAK57ttromTh_na_6VXnstL08S_An+4kpuyYGa46VtDjXhvfH9Q@mail.gmail.com>
References: <CAK57ttrSVM8h0Vnv=Frw=cTgLgLQjoeeNSX2P6YjusCZngEtLA@mail.gmail.com>
	<b4bafb1689bc4413baa9989050630cbc@CITESHT3.ad.uillinois.edu>
	<CAK57ttromTh_na_6VXnstL08S_An+4kpuyYGa46VtDjXhvfH9Q@mail.gmail.com>
Message-ID: <CAK5idxQ505Wu+_ScN8auD3sRES=sKLt6BbQ4myFTtLJ1ZgBRCA@mail.gmail.com>

On Thu, Oct 4, 2012 at 11:22 PM, Adam R. Smith <mradamsmith at gmail.com> wrote:
> Is there any documentation available on what RPython consists of? I can't
> seem to find a manual.

http://doc.pypy.org/en/latest/coding-guide.html#id1

>
>
> On Thu, Oct 4, 2012 at 4:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>> On Thu, Oct 4, 2012 at 8:17 PM, Adam R. Smith <smith195 at illinois.edu>
>> wrote:
>> > I am a PhD candidate at the University of Illinois. I am interested in
>> > learning more about how to use the PyPy Translation Toolchain as part of
>> > a
>> > research project. We would like to write some code in Python and
>> > generate C
>> > code from that. When using the Translation Toolchain, a *.c source file
>> > is
>> > generated and then compiled with GCC. How do I find where the source
>> > files
>> > are stored before GCC is called? I want to interface with those *.c
>> > files.
>> >
>> > Adam Smith
>> > Ph.D. Candidate
>> > University of Illinois at Urbana-Champaign
>>
>> Hi Adam
>>
>> Note that PyPy translation toolchain does not compile Python. It
>> compiles RPython, which is a restricted subset of Python. However, we
>> have a Python interpreter written in rPython.
>
>

From arigo at tunes.org  Fri Oct  5 12:31:14 2012
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 5 Oct 2012 12:31:14 +0200
Subject: [pypy-dev] PyPy STM
In-Reply-To: <CAAL6JQhJ=G0RJOSHRiB1vH+HwFSjxqU7pseN7PM2xn-kcEbYBg@mail.gmail.com>
References: <1348996219.9072.20.camel@lionors.winder.org.uk>
	<50684E13.9070104@gmx.de>
	<CAMSv6X20KAs78z=yBtqeC4L3LKgL1fytikwy5mx+xG-a4BHdfg@mail.gmail.com>
	<506AD299.8010301@oz.net>
	<CAMSv6X3j+hGpeE+dSTqTR6=XzN_AMc8LFbv7J2H=GFyn6HOmrA@mail.gmail.com>
	<CAAL6JQhJ=G0RJOSHRiB1vH+HwFSjxqU7pseN7PM2xn-kcEbYBg@mail.gmail.com>
Message-ID: <CAMSv6X2XvNn9q7+PBmW356=oJQxVw5sKMEjaU=NnB=sOAW+odQ@mail.gmail.com>

Hi,

On Thu, Oct 4, 2012 at 7:58 PM, Randall Leeds <randall.leeds at gmail.com> wrote:
> If we're looking for benchmark problems which are not trivially
> parallelized, STAMP[0] is probably a great place to start.
>
> [0] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.139.472

You're welcome to contribute by rewriting them to Python.  As long as
they are in C, of course, they are not immediately useful to us.


A bient?t,

Armin.

From julien.rioux at gmail.com  Sat Oct  6 10:16:12 2012
From: julien.rioux at gmail.com (Julien Rioux)
Date: Sat, 6 Oct 2012 04:16:12 -0400
Subject: [pypy-dev] failure in SymPy's test suite,
	isprime gives RPython AssertionError
Message-ID: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>

Hi,

Running sympy's test suite would be a good test for pypy. Sympy is a
python module for symbolic mathematics (www.sympy.org). The release
0.7.2 is being prepared and there is a single test failure remaining
when running under pypy nightly (from a couple days ago). It boils
down to this small example:

dirac:~/git/sympy  $ cat test_arnault.py
from sympy.ntheory import isprime
# Arnault's number
a = int('''
803837457453639491257079614341942108138837688287558145837488917522297\
427376533365218650233616396004545791504202360320876656996676098728404\
396540823292873879185086916685732826776177102938969773947016708230428\
687109997439976544144845341155872450633409279022275296229414984230688\
1685404326457534018329786111298960644845216191652872597534901''')
print type(a)
print a
print isprime(a)
dirac:~/git/sympy  $ python test_arnault.py
<type 'long'>
8038374574536394912570796143419421081388376882875581458374889175222974273765333652186502336163960045457915042023603208766569966760987284043965408232928738791850869166857328267761771029389697739470167082304286871099974399765441448453411558724506334092790222752962294149842306881685404326457534018329786111298960644845216191652872597534901
True
dirac:~/git/sympy  $ python -V
Python 2.7.2
dirac:~/git/sympy  $ pypy test_arnault.py
<type 'long'>
8038374574536394912570796143419421081388376882875581458374889175222974273765333652186502336163960045457915042023603208766569966760987284043965408232928738791850869166857328267761771029389697739470167082304286871099974399765441448453411558724506334092790222752962294149842306881685404326457534018329786111298960644845216191652872597534901
RPython traceback:
  File "objspace_std_longobject.c", line 4316, in pow__Long_Long_Long
  File "rlib_rbigint.c", line 23606, in rbigint_pow
Fatal RPython error: AssertionError
Aborted
dirac:~/git/sympy  $ pypy --version
Python 2.7.3 (7f6d5c878b908afec8a643b6d6166dfbd0af2bdf, Oct 03 2012, 11:42:40)
[PyPy 1.9.1-dev0 with GCC 4.6.2]


This code (isprime) is part of previous sympy releases, and this
failure did not happen with pypy 1.9 but happens with the nightly from
Oct 3 (self built with translate.py -Ojit). Can anyone reproduce?

Thanks,
Cheers,
Julien

From bokr at oz.net  Sun Oct  7 10:49:34 2012
From: bokr at oz.net (Bengt Richter)
Date: Sun, 07 Oct 2012 10:49:34 +0200
Subject: [pypy-dev] failure in SymPy's test suite,
	isprime gives RPython AssertionError
In-Reply-To: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
References: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
Message-ID: <5071421E.10200@oz.net>

On 10/06/2012 10:16 AM Julien Rioux wrote:
> Hi,
>
> Running sympy's test suite would be a good test for pypy. Sympy is a
> python module for symbolic mathematics (www.sympy.org). The release
> 0.7.2 is being prepared and there is a single test failure remaining
> when running under pypy nightly (from a couple days ago). It boils
> down to this small example:
>
> dirac:~/git/sympy  $ cat test_arnault.py
> from sympy.ntheory import isprime
> # Arnault's number
> a = int('''
> 803837457453639491257079614341942108138837688287558145837488917522297\
> 427376533365218650233616396004545791504202360320876656996676098728404\
> 396540823292873879185086916685732826776177102938969773947016708230428\
> 687109997439976544144845341155872450633409279022275296229414984230688\
> 1685404326457534018329786111298960644845216191652872597534901''')
> print type(a)
> print a
> print isprime(a)
> dirac:~/git/sympy  $ python test_arnault.py
> <type 'long'>
> 8038374574536394912570796143419421081388376882875581458374889175222974273765333652186502336163960045457915042023603208766569966760987284043965408232928738791850869166857328267761771029389697739470167082304286871099974399765441448453411558724506334092790222752962294149842306881685404326457534018329786111298960644845216191652872597534901
> True
> dirac:~/git/sympy  $ python -V
> Python 2.7.2
> dirac:~/git/sympy  $ pypy test_arnault.py
> <type 'long'>
> 8038374574536394912570796143419421081388376882875581458374889175222974273765333652186502336163960045457915042023603208766569966760987284043965408232928738791850869166857328267761771029389697739470167082304286871099974399765441448453411558724506334092790222752962294149842306881685404326457534018329786111298960644845216191652872597534901
> RPython traceback:
>    File "objspace_std_longobject.c", line 4316, in pow__Long_Long_Long
>    File "rlib_rbigint.c", line 23606, in rbigint_pow
> Fatal RPython error: AssertionError
> Aborted
> dirac:~/git/sympy  $ pypy --version
> Python 2.7.3 (7f6d5c878b908afec8a643b6d6166dfbd0af2bdf, Oct 03 2012, 11:42:40)
> [PyPy 1.9.1-dev0 with GCC 4.6.2]
>
>
> This code (isprime) is part of previous sympy releases, and this
> failure did not happen with pypy 1.9 but happens with the nightly from
> Oct 3 (self built with translate.py -Ojit). Can anyone reproduce?
>
> Thanks,
> Cheers,
> Julien

Hi Julien,
I installed an unshared copy of sympy and changed your test_arnault.py
to an equivalent ta2.py, to make it importable, with the code in main,
so I could try an old trace routine I wrote long ago, which surprisingly
still seems to sort of work:
______________________________________________________________________
[10:10 ~/wk/sympy]$ cat ta2.py
from sympy.ntheory import isprime
# Arnault's number
def main():
     a = int('''
803837457453639491257079614341942108138837688287558145837488917522297\
427376533365218650233616396004545791504202360320876656996676098728404\
396540823292873879185086916685732826776177102938969773947016708230428\
687109997439976544144845341155872450633409279022275296229414984230688\
1685404326457534018329786111298960644845216191652872597534901''')
     print type(a)
     print a
     print isprime(a)

if __name__ == '__main__':
     main()
[10:10 ~/wk/sympy]$
______________________________________________________________________

Here's the run, diffing the results of _test calls grepped out of a traces
for pypy and python:

[10:23 ~/wk/sympy]$ diff -u <(pypy $(which tw) ta2 main _test |grep 'R: _test\|^File') <(python $(which tw) ta2 main 
_test|grep 'R: _test\|^File')
pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy)
pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy)
--- /dev/fd/63  2012-10-07 10:24:30.108533498 +0200
+++ /dev/fd/62  2012-10-07 10:24:30.108533498 +0200
@@ -1,2 +1,47 @@
  File: "/home/bokr/lib/python2.7/site-packages/sympy/ntheory/primetest.py"
-  72 [_test]:           R: _test(...) => False
+  71 [_test]:           R: _test(...) => True
+  66 [_test]:           R: _test(...) => True
+  66 [_test]:           R: _test(...) => True
+  71 [_test]:           R: _test(...) => True
                    ... 41 repeats of same as above and below ...
+  71 [_test]:           R: _test(...) => True
[10:24 ~/wk/sympy]$

The 72, 71, 66 represent the line numbers of the returns from _test
I didn't grep out the calls, as the args are such long wrapping long integers.

I wrote out some failing args to pow (which I can't trace into as it's builtin)
and made a tpow.py, seen below.

HTH,
Regards,
Bengt Richter
______________________________________________________________________

[09:56 ~/wk/sympy]$ python --version
Python 2.7.2
[09:56 ~/wk/sympy]$ pypy --version
pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy)
pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy)
Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:38)
[PyPy 1.5.0-alpha0 with GCC 4.4.3]
[09:56 ~/wk/sympy]$ cat tpow.py
#!/usr/bin/python
#
def main(opts):
     expected = int('205308233964764741474065440722548147152837487'
             '8436244152161245680082478906739003837537707602796'
             '2523882678514957414514288325968530357544873923327'
             '2560600119280228812360239622859389835106130303819'
             '8812608395321192472488554180795988393259013286827'
             '6127496893617704776708371167151813294028632749297'
             '63810769866878488336371858543186166507924901737')

     actual = pow(2, int('200959364363409872814269903585485527034709'
         '4220718895364593722293805743568441333413046625584040990011'
         '3644787605059008021916424916902468210109913520582321846979'
         '6271729171433206694044275734742443486754177057607171777499'
         '3599941360362113352889681126583523197555688240573537460576'
         '720421351081614383504582446527824740161211304047913218149383725'),
         int('803837457453639491257079614341942108138837688287558145'
             '837488917522297427376533365218650233616396004545791504'
             '202360320876656996676098728404396540823292873879185086'
             '916685732826776177102938969773947016708230428687109997'
             '439976544144845341155872450633409279022275296229414984'
             '230688168540432645753401832978611129896064484521619165'
             '2872597534901'))

     text =  'expected:\n    %s\nactual:\n    %s'%(
             tuple('\n    '.join(s[i:i+50] for i in range(0,len(s),50))
                for s in [repr(v) for v in (expected, actual)]))

     assert actual==expected, text

     if opts and opts[0]=='-print': print text
     if actual == expected:
         print '... as expected ;-)'

if __name__ == '__main__':
     import sys
     try: main(sys.argv[1:])
     except Exception,e: print '%s:\n%s'%(e.__class__.__name__, e)

Bear in mind my python and pypy versions are out of date,
but you can easily try tpow.py as follows and see what
happens for you...

[09:56 ~/wk/sympy]$ python tpow.py
... as expected ;-)
[09:57 ~/wk/sympy]$ pypy tpow.py
pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy)
pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy)
AssertionError:
expected:
     20530823396476474147406544072254814715283748784362
     44152161245680082478906739003837537707602796252388
     26785149574145142883259685303575448739233272560600
     11928022881236023962285938983510613030381988126083
     95321192472488554180795988393259013286827612749689
     36177047767083711671518132940286327492976381076986
     6878488336371858543186166507924901737L
actual:
     54907080201776472107337034411173408169461451059492
     19340520622621434519462362497005098074867506892669
     64669150197877920364556131878818528130569072864509
     78537992136333962220382281410469047624566423572707
     23947703936700498584279881397667810995651289825448
     34510224055185249725794168117352710937820049011975
     6120237666889918094387505100251368387L
[09:57 ~/wk/sympy]$ python tpow.py -print
expected:
     20530823396476474147406544072254814715283748784362
     44152161245680082478906739003837537707602796252388
     26785149574145142883259685303575448739233272560600
     11928022881236023962285938983510613030381988126083
     95321192472488554180795988393259013286827612749689
     36177047767083711671518132940286327492976381076986
     6878488336371858543186166507924901737L
actual:
     20530823396476474147406544072254814715283748784362
     44152161245680082478906739003837537707602796252388
     26785149574145142883259685303575448739233272560600
     11928022881236023962285938983510613030381988126083
     95321192472488554180795988393259013286827612749689
     36177047767083711671518132940286327492976381076986
     6878488336371858543186166507924901737L
... as expected ;-)
[09:57 ~/wk/sympy]$


From arigo at tunes.org  Sun Oct  7 19:27:08 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 7 Oct 2012 19:27:08 +0200
Subject: [pypy-dev] failure in SymPy's test suite,
	isprime gives RPython AssertionError
In-Reply-To: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
References: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
Message-ID: <CAMSv6X2H-08geswVEpozyb+1C0+Yb-v2CWFO+10exe6w6qLdfg@mail.gmail.com>

Hi Julien,

On Sat, Oct 6, 2012 at 10:16 AM, Julien Rioux <julien.rioux at gmail.com> wrote:
> Fatal RPython error: AssertionError

This should be fixed now (2fd0ae547d86).  Thanks!


Armin

From julien.rioux at gmail.com  Mon Oct  8 16:14:57 2012
From: julien.rioux at gmail.com (Julien Rioux)
Date: Mon, 8 Oct 2012 10:14:57 -0400
Subject: [pypy-dev] failure in SymPy's test suite,
	isprime gives RPython AssertionError
In-Reply-To: <CAMSv6X2H-08geswVEpozyb+1C0+Yb-v2CWFO+10exe6w6qLdfg@mail.gmail.com>
References: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
	<CAMSv6X2H-08geswVEpozyb+1C0+Yb-v2CWFO+10exe6w6qLdfg@mail.gmail.com>
Message-ID: <CAG4vuP0PSicVodhvLLmN1ticxSa+B-HOzQQ7m_Bj-x+ciG0Lvw@mail.gmail.com>

On Sun, Oct 7, 2012 at 1:27 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Julien,
>
> On Sat, Oct 6, 2012 at 10:16 AM, Julien Rioux <julien.rioux at gmail.com> wrote:
>> Fatal RPython error: AssertionError
>
> This should be fixed now (2fd0ae547d86).  Thanks!
>
>
> Armin

Great! I'll report back once I get around to compiling the thing.

Thanks,
Cheers,
Julien

From julien.rioux at gmail.com  Tue Oct  9 16:14:38 2012
From: julien.rioux at gmail.com (Julien Rioux)
Date: Tue, 9 Oct 2012 16:14:38 +0200
Subject: [pypy-dev] failure in SymPy's test suite,
	isprime gives RPython AssertionError
In-Reply-To: <CAG4vuP0PSicVodhvLLmN1ticxSa+B-HOzQQ7m_Bj-x+ciG0Lvw@mail.gmail.com>
References: <CAG4vuP2e-c---93eOnLPGQKMvCjw217iS1DSV_BRm7ortpff=w@mail.gmail.com>
	<CAMSv6X2H-08geswVEpozyb+1C0+Yb-v2CWFO+10exe6w6qLdfg@mail.gmail.com>
	<CAG4vuP0PSicVodhvLLmN1ticxSa+B-HOzQQ7m_Bj-x+ciG0Lvw@mail.gmail.com>
Message-ID: <CAG4vuP0-mCeXdyVH0g0ZkPZMbVAQBchdOf3NZyZBsrYOEEjZdA@mail.gmail.com>

On Mon, Oct 8, 2012 at 4:14 PM, Julien Rioux <julien.rioux at gmail.com> wrote:
> Great! I'll report back once I get around to compiling the thing.

With sympy 0.7.2.rc1 and pypy-2fd0ae547d86, all tests pass.
Congrats!

--
Julien

From stefan_ml at behnel.de  Thu Oct 11 07:41:01 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Oct 2012 07:41:01 +0200
Subject: [pypy-dev] Weird C code in crashing code line
Message-ID: <k55m5b$7mk$1@ger.gmane.org>

Hi,

I get a crash in lxml's test suite on 64bit Linux, right with the first
test. I couldn't investigate it in depth so far, but at least I found that
it crashes in line 606 of obmalloc.c, which reads as follows:

    583    block *bp;
    [...]
    604         bp = pool->freeblock;
    605         assert(bp != NULL);
    606         if ((pool->freeblock = *(block **)bp) != NULL) {

The cast looks a bit smelly to me. Could someone who knows that code please
take a look to see if it makes any sense? This kind of code tends to hint
at potential aliasing problems to me etc. OTOH, "bp" holds the value 0xc
here when it crashes, so there's clearly something going wrong.

Here is the stack trace I get:

#0  PyObject_Malloc (nbytes=16) at .../pypy/translator/c/src/obmalloc.c:606
#1  0x0000000000eeb938 in
pypy_g__ll_malloc_varsize_no_length_zero__Signed_Signed
(l_length_1494=<value optimized out>, l_v1845914=<value optimized out>,
l_itemsize_9=<value optimized out>)
    at rpython_memory_gctransform_transform.c:249
#2  0x0000000000cad374 in pypy_g_BaseCpyTypedescr_allocate
(l_self_67=0x1aafeb8, l_w_type_11=0x1702040, l_itemcount_7=0) at
module_cpyext_pyobject.c:3190
#3  0x0000000000cadc92 in pypy_g_create_ref (l_w_obj_2087=0x7f0d25d49ac0,
l_itemcount_1=<value optimized out>) at module_cpyext_pyobject.c:1275
#4  0x0000000000cace27 in pypy_g_make_ref (l_w_obj_2086=0x168de00) at
module_cpyext_pyobject.c:509
#5  0x0000000001040796 in PyWeakref_NewRef (l_v2078649=<value optimized
out>, l_v2078623=0x0) at module_cpyext_api_2.c:38330
#6  0x00007f0d2576df72 in __pyx_f_4lxml_5etree__registerProxy () from
.../lxml-3.0/src/lxml/etree.pypy-19.so
#7  0x00007f0d257a361f in __pyx_f_4lxml_5etree__elementFactory () from
.../lxml-3.0/src/lxml/etree.pypy-19.so
#8  0x00007f0d2578f2cf in __pyx_f_4lxml_5etree_9_Document_getroot () from
.../lxml-3.0/src/lxml/etree.pypy-19.so
#9  0x00007f0d257c2990 in __pyx_pf_4lxml_5etree_22fromstring () from
.../lxml-3.0/src/lxml/etree.pypy-19.so
#10 0x00007f0d257c2847 in __pyx_pw_4lxml_5etree_23fromstring () from
.../lxml-3.0/src/lxml/etree.pypy-19.so

Note that this isn't using the latest nightly PyPy from the buildbot
(58001) but I quickly tested it with that as well and it seems to crash at
the same place.

When I skip the first test, the test suite runs without crashes, so if
there isn't something obviously wrong in the code above, I'd see when I
find the time to figure out why this particular set of doctests (in
test_autolink.txt) crashes but the rest doesn't, even though the etree part
of the stack trace above should be the same in a pretty large part of the
test suite.

Stefan


From arigo at tunes.org  Thu Oct 11 09:25:17 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Oct 2012 09:25:17 +0200
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <k55m5b$7mk$1@ger.gmane.org>
References: <k55m5b$7mk$1@ger.gmane.org>
Message-ID: <CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>

Hi Stefan,

On Thu, Oct 11, 2012 at 7:41 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> it crashes in line 606 of obmalloc.c, which reads as follows:
>
>     583    block *bp;
>     [...]
>     604         bp = pool->freeblock;
>     605         assert(bp != NULL);
>     606         if ((pool->freeblock = *(block **)bp) != NULL) {
>
> The cast looks a bit smelly to me. Could someone who knows that code please
> take a look to see if it makes any sense?

This obmalloc.c is straight from CPython.  It's very unlikely at this
point to contain problems.


A bient?t,

Armin.

From alex.gaynor at gmail.com  Thu Oct 11 11:53:57 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 11 Oct 2012 02:53:57 -0700
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>
References: <k55m5b$7mk$1@ger.gmane.org>
	<CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>
Message-ID: <CAFRnB2VArLH1k7fCcXdBLwvUg6aBb-Lyx5X1wrMGuVf6KcT0+w@mail.gmail.com>

On Thu, Oct 11, 2012 at 12:25 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Stefan,
>
> On Thu, Oct 11, 2012 at 7:41 AM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
> > it crashes in line 606 of obmalloc.c, which reads as follows:
> >
> >     583    block *bp;
> >     [...]
> >     604         bp = pool->freeblock;
> >     605         assert(bp != NULL);
> >     606         if ((pool->freeblock = *(block **)bp) != NULL) {
> >
> > The cast looks a bit smelly to me. Could someone who knows that code
> please
> > take a look to see if it makes any sense?
>
> This obmalloc.c is straight from CPython.  It's very unlikely at this
> point to contain problems.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

It was copied from CPython many years ago though, perhaps they've fixed
bugs since then?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121011/ac8584d9/attachment.html>

From stefan_ml at behnel.de  Thu Oct 11 12:01:38 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Oct 2012 12:01:38 +0200
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <CAFRnB2VArLH1k7fCcXdBLwvUg6aBb-Lyx5X1wrMGuVf6KcT0+w@mail.gmail.com>
References: <k55m5b$7mk$1@ger.gmane.org>
	<CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>
	<CAFRnB2VArLH1k7fCcXdBLwvUg6aBb-Lyx5X1wrMGuVf6KcT0+w@mail.gmail.com>
Message-ID: <k565e1$4ng$1@ger.gmane.org>

Alex Gaynor, 11.10.2012 11:53:
> On Thu, Oct 11, 2012 at 12:25 AM, Armin Rigo wrote:
>> On Thu, Oct 11, 2012 at 7:41 AM, Stefan Behnel wrote:
>>> it crashes in line 606 of obmalloc.c, which reads as follows:
>>>
>>>     583    block *bp;
>>>     [...]
>>>     604         bp = pool->freeblock;
>>>     605         assert(bp != NULL);
>>>     606         if ((pool->freeblock = *(block **)bp) != NULL) {
>>>
>>> The cast looks a bit smelly to me. Could someone who knows that code
>> please
>>> take a look to see if it makes any sense?
>>
>> This obmalloc.c is straight from CPython.  It's very unlikely at this
>> point to contain problems.
> 
> It was copied from CPython many years ago though, perhaps they've fixed
> bugs since then?

No, I just checked - the same code is in latest CPython hg.

Thanks for the hints.

Stefan



From amauryfa at gmail.com  Thu Oct 11 13:54:05 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 11 Oct 2012 13:54:05 +0200
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>
References: <k55m5b$7mk$1@ger.gmane.org>
	<CAMSv6X05BScS1ZHqKJGGW9=bQAk7dmn1Y+T=dB47qD6CRfinLA@mail.gmail.com>
Message-ID: <CAGmFidZFnofACcUSTpej337Dw3-itvAbsfH5hqEeikPo+WbppA@mail.gmail.com>

2012/10/11 Armin Rigo <arigo at tunes.org>:
> On Thu, Oct 11, 2012 at 7:41 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> it crashes in line 606 of obmalloc.c, which reads as follows:
>>
>>     583    block *bp;
>>     [...]
>>     604         bp = pool->freeblock;
>>     605         assert(bp != NULL);
>>     606         if ((pool->freeblock = *(block **)bp) != NULL) {
>>
>> The cast looks a bit smelly to me. Could someone who knows that code please
>> take a look to see if it makes any sense?
>
> This obmalloc.c is straight from CPython.  It's very unlikely at this
> point to contain problems.

It looks like the typical hack of a "free list" implementation.
Freed blocks form a linked list, and the address of the next free
block is stored in the block itself.

So the line above effectively pops the first item in the list:
pool->freeblock is the address of the first free block,
PyObject_Malloc() will return it,
and pool->freeblock is changed to point to the next free block.

Everything looks normal to me, even if a bit obscure.

-- 
Amaury Forgeot d'Arc

From joehillen at gmail.com  Thu Oct 11 19:13:15 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Thu, 11 Oct 2012 10:13:15 -0700
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <k55m5b$7mk$1@ger.gmane.org>
References: <k55m5b$7mk$1@ger.gmane.org>
Message-ID: <CANJQ5C4PLhQfJxSanvtufFVbAi_azH17EzeLyWvLJtPPjrL1xw@mail.gmail.com>

On Wed, Oct 10, 2012 at 10:41 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I get a crash in lxml's test suite on 64bit Linux, right with the first
> test.

Is this part of an effort to make Cython code work with PyPy? lxml is
written in Cython, and AFAIK Cython doesn't work with PyPy.

Even if it did, it wouldn't much help right now because if I remember
correctly, Cython compiles to CPython extensions which are slower in
PyPy than in CPython, but that may have changed since I last looked
into it.

-Joe Hillenbrand

From stefan_ml at behnel.de  Thu Oct 11 19:20:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Oct 2012 19:20:46 +0200
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <CANJQ5C4PLhQfJxSanvtufFVbAi_azH17EzeLyWvLJtPPjrL1xw@mail.gmail.com>
References: <k55m5b$7mk$1@ger.gmane.org>
	<CANJQ5C4PLhQfJxSanvtufFVbAi_azH17EzeLyWvLJtPPjrL1xw@mail.gmail.com>
Message-ID: <k56v5c$pdn$1@ger.gmane.org>

Joe Hillenbrand, 11.10.2012 19:13:
> On Wed, Oct 10, 2012 at 10:41 PM, Stefan Behnel wrote:
>> I get a crash in lxml's test suite on 64bit Linux, right with the first
>> test.
> 
> Is this part of an effort to make Cython code work with PyPy?

Yes.


> lxml is written in Cython, and AFAIK Cython doesn't work with PyPy.

Yes, it does.


> Even if it did, it wouldn't much help right now because if I remember
> correctly, Cython compiles to CPython extensions which are slower in
> PyPy than in CPython

Yes, they are. But they still work. You should run your own benchmarks, as
usual, and report any problems to the appropriate projects.

Stefan



From fijall at gmail.com  Fri Oct 12 09:42:24 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 12 Oct 2012 09:42:24 +0200
Subject: [pypy-dev] Weird C code in crashing code line
In-Reply-To: <k55m5b$7mk$1@ger.gmane.org>
References: <k55m5b$7mk$1@ger.gmane.org>
Message-ID: <CAK5idxRf6ktmrVF7+o8Fvon-0b3ExqzGXnf2+5d+W=-D2h+5YQ@mail.gmail.com>

On Thu, Oct 11, 2012 at 7:41 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I get a crash in lxml's test suite on 64bit Linux, right with the first
> test. I couldn't investigate it in depth so far, but at least I found that
> it crashes in line 606 of obmalloc.c, which reads as follows:
>
>     583    block *bp;
>     [...]
>     604         bp = pool->freeblock;
>     605         assert(bp != NULL);
>     606         if ((pool->freeblock = *(block **)bp) != NULL) {
>
> The cast looks a bit smelly to me. Could someone who knows that code please
> take a look to see if it makes any sense? This kind of code tends to hint
> at potential aliasing problems to me etc. OTOH, "bp" holds the value 0xc
> here when it crashes, so there's clearly something going wrong.

That's very likely a refcounting problem (not that I can help much
more than that though :(

Cheers,
fijal

From fijall at gmail.com  Sat Oct 13 23:29:53 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 13 Oct 2012 23:29:53 +0200
Subject: [pypy-dev] [pypy-commit] pypy default: fix for None in kwargs
In-Reply-To: <20121013192653.B15CE1C1C5B@cobra.cs.uni-duesseldorf.de>
References: <20121013192653.B15CE1C1C5B@cobra.cs.uni-duesseldorf.de>
Message-ID: <CAK5idxT98KRGhAWvSGg9L3H7XQRz7J-JCWDA1b=++e23AXRk-Q@mail.gmail.com>

Did this fail any test? If not, it should come with one.

On Sat, Oct 13, 2012 at 9:26 PM, mattip <noreply at buildbot.pypy.org> wrote:
> Author: mattip <matti.picus at gmail.com>
> Branch:
> Changeset: r58111:e537e0093563
> Date: 2012-10-13 21:26 +0200
> http://bitbucket.org/pypy/pypy/changeset/e537e0093563/
>
> Log:    fix for None in kwargs
>
> diff --git a/pypy/module/micronumpy/interp_numarray.py b/pypy/module/micronumpy/interp_numarray.py
> --- a/pypy/module/micronumpy/interp_numarray.py
> +++ b/pypy/module/micronumpy/interp_numarray.py
> @@ -661,7 +661,7 @@
>              raise operationerrfmt(space.w_ValueError, "Unknown order: %s",
>                                    order)
>      if isinstance(w_object, W_NDimArray):
> -        if (not space.is_w(w_dtype, space.w_None) and
> +        if (not space.is_none(w_dtype) and
>              w_object.get_dtype() is not w_dtype):
>              raise OperationError(space.w_NotImplementedError, space.wrap(
>                                    "copying over different dtypes unsupported"))
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> http://mail.python.org/mailman/listinfo/pypy-commit

From stefan_ml at behnel.de  Sun Oct 14 19:59:33 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 19:59:33 +0200
Subject: [pypy-dev] [cpyext] how to make use of the weakref callback?
Message-ID: <k5eui4$47l$1@ger.gmane.org>

Hi,

when creating a new weakref using PyWeakref_NewRef(obj, callback), a Python
callback function can be registered that is supposed to get called when the
weakly referenced object dies. The callback receives a single argument, the
weakref object.

Now, the weak reference object itself is mostly useless, what is
interesting is the object it originally referenced. I tried getting at that
object by calling PyWeakref_LockObject(), but that already returns None at
that point. PyWeakref_GetObject() is basically just a wrapper around that
function and behaves the same.

Is there any way to figure out what that object was that has just died when
the callback gets called?

Stefan


From amauryfa at gmail.com  Sun Oct 14 20:11:58 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 14 Oct 2012 20:11:58 +0200
Subject: [pypy-dev] [cpyext] how to make use of the weakref callback?
In-Reply-To: <k5eui4$47l$1@ger.gmane.org>
References: <k5eui4$47l$1@ger.gmane.org>
Message-ID: <CAGmFidYDu-Xyr+HeNanbA6i0dze8O6R3qrv2O5+1OGMyituC1Q@mail.gmail.com>

2012/10/14 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> when creating a new weakref using PyWeakref_NewRef(obj, callback), a Python
> callback function can be registered that is supposed to get called when the
> weakly referenced object dies. The callback receives a single argument, the
> weakref object.
>
> Now, the weak reference object itself is mostly useless, what is
> interesting is the object it originally referenced. I tried getting at that
> object by calling PyWeakref_LockObject(), but that already returns None at
> that point. PyWeakref_GetObject() is basically just a wrapper around that
> function and behaves the same.
>
> Is there any way to figure out what that object was that has just died when
> the callback gets called?

I don't think so:
http://docs.python.org/library/weakref.html#weakref.ref
"""the weak reference object will be passed as the only parameter to
the callback; the referent will no longer be available."""


-- 
Amaury Forgeot d'Arc

From stefan_ml at behnel.de  Sun Oct 14 20:44:58 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 20:44:58 +0200
Subject: [pypy-dev] [cpyext] how to make use of the weakref callback?
In-Reply-To: <CAGmFidYDu-Xyr+HeNanbA6i0dze8O6R3qrv2O5+1OGMyituC1Q@mail.gmail.com>
References: <k5eui4$47l$1@ger.gmane.org>
	<CAGmFidYDu-Xyr+HeNanbA6i0dze8O6R3qrv2O5+1OGMyituC1Q@mail.gmail.com>
Message-ID: <k5f179$omr$1@ger.gmane.org>

Amaury Forgeot d'Arc, 14.10.2012 20:11:
> 2012/10/14 Stefan Behnel:
>> when creating a new weakref using PyWeakref_NewRef(obj, callback), a Python
>> callback function can be registered that is supposed to get called when the
>> weakly referenced object dies. The callback receives a single argument, the
>> weakref object.
>>
>> Now, the weak reference object itself is mostly useless, what is
>> interesting is the object it originally referenced. I tried getting at that
>> object by calling PyWeakref_LockObject(), but that already returns None at
>> that point. PyWeakref_GetObject() is basically just a wrapper around that
>> function and behaves the same.
>>
>> Is there any way to figure out what that object was that has just died when
>> the callback gets called?
> 
> I don't think so:
> http://docs.python.org/library/weakref.html#weakref.ref
> """the weak reference object will be passed as the only parameter to
> the callback; the referent will no longer be available."""

Ah, right, thanks! I see it now, CPython behaves the same way in its
weakrefobject.c. It clears the references before calling the callbacks. How
very unfortunate.

Guess I'll have to see if a closure is worth the hassle for me (I only care
about C pointer fields of the object).

Stefan



From fijall at gmail.com  Sun Oct 21 14:03:34 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 21 Oct 2012 14:03:34 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
Message-ID: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>

Hi

Due to impressive work by Ronan Lamy, we're now able to split RPython
and PyPy. Note that the fact of splitting this is not up to
discussion, however, how we go about it is. During discussions with
Armin we came up with the following plan:

* We make a copy of pypy repo called rpython. it'll still live under
pypy team on bitbucket.

* We'll rename toplevel package of pypy to rpython, for the rpython
part. Since we need to change ALL THE IMPORTS, we can do refactorings
of imports now. Proposed changes:

  * to be moved from pypy to rpython: annotation, translator, rlib
  * pypy.config must be split somehow, same for tool, bin and for doc
  * move pypy.rpython namespace to rpython.rtyper
  * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
  * pypy.rpython.memory becomes rpython.gc
  * pypy.objspace.flow becomes rpython.flowspace
  * testrunner and dotviewer can become independent packages
  * _pytest stays with pypy, however for rpython you can use whatever
version of py.test you have installed. rpython test suite should also
however be runnable under current contents of _pytest

* RPython will come with setup.py and be a "normal" python package

Cheers,
fijal & armin

From fijall at gmail.com  Sun Oct 21 14:09:55 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 21 Oct 2012 14:09:55 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
Message-ID: <CAK5idxSmb=8aX7Vmr_q1dk97mkW_JsBKFgtgaLP4tiVjVtpWkg@mail.gmail.com>

On Sun, Oct 21, 2012 at 2:03 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hi
>
> Due to impressive work by Ronan Lamy, we're now able to split RPython
> and PyPy. Note that the fact of splitting this is not up to
> discussion, however, how we go about it is. During discussions with
> Armin we came up with the following plan:
>
> * We make a copy of pypy repo called rpython. it'll still live under
> pypy team on bitbucket.
>
> * We'll rename toplevel package of pypy to rpython, for the rpython
> part. Since we need to change ALL THE IMPORTS, we can do refactorings
> of imports now. Proposed changes:
>
>   * to be moved from pypy to rpython: annotation, translator, rlib

and of course the jit directory.

>   * pypy.config must be split somehow, same for tool, bin and for doc
>   * move pypy.rpython namespace to rpython.rtyper
>   * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
>   * pypy.rpython.memory becomes rpython.gc
>   * pypy.objspace.flow becomes rpython.flowspace
>   * testrunner and dotviewer can become independent packages
>   * _pytest stays with pypy, however for rpython you can use whatever
> version of py.test you have installed. rpython test suite should also
> however be runnable under current contents of _pytest
>
> * RPython will come with setup.py and be a "normal" python package
>
> Cheers,
> fijal & armin

From samuele.pedroni at gmail.com  Sun Oct 21 14:40:43 2012
From: samuele.pedroni at gmail.com (Samuele Pedroni)
Date: Sun, 21 Oct 2012 14:40:43 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <CAK5idxSmb=8aX7Vmr_q1dk97mkW_JsBKFgtgaLP4tiVjVtpWkg@mail.gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<CAK5idxSmb=8aX7Vmr_q1dk97mkW_JsBKFgtgaLP4tiVjVtpWkg@mail.gmail.com>
Message-ID: <CAGwLUHYyc=sjuTqVaVySUvyAEQM_qWNRswW=3=uQtvnUEXdjBA@mail.gmail.com>

nice!

rpython is  a bit long for the top package name, the only other option
I can think of would be rpyc (with c = compiler or chain) though which
isn't that great;

in any case translate.py should become either rpythonc or rpyc, in the
tradition of cc and javac etc... I can even imagine a world/future
where an rpython command alone would be for an interpreter for
rpython.


>>   * move pypy.rpython namespace to rpython.rtyper

I know that has been the historical name but the "r" in "rtyper" for
restricted there is a bit of strange repetition with hindsight

From holger at merlinux.eu  Sun Oct 21 20:30:43 2012
From: holger at merlinux.eu (holger krekel)
Date: Sun, 21 Oct 2012 18:30:43 +0000
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
Message-ID: <20121021183043.GE27773@merlinux.eu>

Hi Maciej, Armin,

On Sun, Oct 21, 2012 at 14:03 +0200, Maciej Fijalkowski wrote:
> Hi
> 
> Due to impressive work by Ronan Lamy, we're now able to split RPython
> and PyPy. Note that the fact of splitting this is not up to
> discussion, however, how we go about it is. During discussions with
> Armin we came up with the following plan:
> 
> * We make a copy of pypy repo called rpython. it'll still live under
> pypy team on bitbucket.
> 
> * We'll rename toplevel package of pypy to rpython, for the rpython
> part. Since we need to change ALL THE IMPORTS, we can do refactorings
> of imports now. Proposed changes:
> 
>   * to be moved from pypy to rpython: annotation, translator, rlib
>   * pypy.config must be split somehow, same for tool, bin and for doc
>   * move pypy.rpython namespace to rpython.rtyper
>   * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
>   * pypy.rpython.memory becomes rpython.gc
>   * pypy.objspace.flow becomes rpython.flowspace
>   * testrunner and dotviewer can become independent packages
>   * _pytest stays with pypy, however for rpython you can use whatever
> version of py.test you have installed. rpython test suite should also
> however be runnable under current contents of _pytest
> 
> * RPython will come with setup.py and be a "normal" python package

great stuff!

Samuele's comments also make sense to me as well (rpython.typer etc.).

Will there be a setup.py with pypy to automatically install perequisites
(such as RPython, possibly other bits)?  FWIW, I think it could help to
eventually allow people more easily maintain modular rpython-level pypy
modules by depending on a specific pypy version and declaring an
entry-point in order to be automatically picked up by the pypy
translation process.  This way one could install multiple pypy-MODULE*
packages, then cause a pypy build and all those modules would become
part of the translated pypy-c.  (happy to help with this and relate my
experiences if wanted).  It's not binary compatibility but would be
a way to have a user choose modules without having to dive into
translation target scripts etc.

best,
holger

> Cheers,
> fijal & armin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 

From ronan.lamy at gmail.com  Sun Oct 21 21:28:50 2012
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Sun, 21 Oct 2012 20:28:50 +0100
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
Message-ID: <1350847730.2277.14.camel@ronan-desktop>

Le dimanche 21 octobre 2012 ? 14:03 +0200, Maciej Fijalkowski a ?crit :
> Hi
> 
> Due to impressive work by Ronan Lamy, we're now able to split RPython
> and PyPy. Note that the fact of splitting this is not up to
> discussion, however, how we go about it is. During discussions with
> Armin we came up with the following plan:
> 
> * We make a copy of pypy repo called rpython. it'll still live under
> pypy team on bitbucket.
> 
> * We'll rename toplevel package of pypy to rpython, for the rpython
> part. Since we need to change ALL THE IMPORTS, we can do refactorings
> of imports now. Proposed changes:
> 
>   * to be moved from pypy to rpython: annotation, translator, rlib
>   * pypy.config must be split somehow, same for tool, bin and for doc
>   * move pypy.rpython namespace to rpython.rtyper
>   * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
>   * pypy.rpython.memory becomes rpython.gc
>   * pypy.objspace.flow becomes rpython.flowspace

I think that the changes should go deeper. I see model.py and
operation.py as defining the flowgraph-based intermediate representation
on which the rest of the tool chain operates. In contrast, the rest of
pypy.objspace.flow only deals with translating Python callables to this
IR. So I would suggest to move model.py and operation.py to
rpython.flowgraph, and to also move there the various generic flow graph
manipulation functions that are currently scattered around (e.g. in
pypy.translator.backendopt.ssa).

>   * testrunner and dotviewer can become independent packages

+1 (well, I don't really have an opinion on testrunner). IMO, this
implies that FunctionGraph should lose its view/show method.

>   * _pytest stays with pypy, however for rpython you can use whatever
> version of py.test you have installed. rpython test suite should also
> however be runnable under current contents of _pytest

Why the last stipulation? I would think that fishing out _pytest from
pypy would require significant acrobatics once the repos are separated.
> 
> * RPython will come with setup.py and be a "normal" python package
> 
> Cheers,
> fijal & armin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev



From anto.cuni at gmail.com  Sun Oct 21 21:35:40 2012
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 21 Oct 2012 21:35:40 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <1350847730.2277.14.camel@ronan-desktop>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<1350847730.2277.14.camel@ronan-desktop>
Message-ID: <50844E8C.10601@gmail.com>

On 10/21/2012 09:28 PM, Ronan Lamy wrote:
>> >   * testrunner and dotviewer can become independent packages
> +1 (well, I don't really have an opinion on testrunner). IMO, this
> implies that FunctionGraph should lose its view/show method.

nothing stops us to have a .show() method which tries to lazily import
dotviewer and complain in case it's not there.
.show() is too damn useful to kill it, IMHO :-)

ciao,
Anto


From alex.gaynor at gmail.com  Sun Oct 21 21:41:09 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sun, 21 Oct 2012 12:41:09 -0700
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <50844E8C.10601@gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<1350847730.2277.14.camel@ronan-desktop> <50844E8C.10601@gmail.com>
Message-ID: <CAFRnB2UCeVVdW2gEcyZCiFgoAV=6fYa9uO5hJwqPrOC8VKthsA@mail.gmail.com>

On Sun, Oct 21, 2012 at 12:35 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> On 10/21/2012 09:28 PM, Ronan Lamy wrote:
> >> >   * testrunner and dotviewer can become independent packages
> > +1 (well, I don't really have an opinion on testrunner). IMO, this
> > implies that FunctionGraph should lose its view/show method.
>
> nothing stops us to have a .show() method which tries to lazily import
> dotviewer and complain in case it's not there.
> .show() is too damn useful to kill it, IMHO :-)
>
> ciao,
> Anto
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

We can always have a show() function which takes anything as a param, but
anyways these are small details. Once things are split into two
repositories it's easy to move stuff around within either one, it's just
moving *between* the two of them that's hard, so let's make sure we get
which repo everything lives in correctly, anything else is just a matter of
`mv` + `sed` in a repository.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121021/e5d2864e/attachment.html>

From fijall at gmail.com  Mon Oct 22 07:00:43 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 22 Oct 2012 07:00:43 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <20121021183043.GE27773@merlinux.eu>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<20121021183043.GE27773@merlinux.eu>
Message-ID: <CAK5idxS22S8f=Wc17GJGV_REKhZDAF6nOQwaYFRJ0ivJ6sJr-A@mail.gmail.com>

On Sun, Oct 21, 2012 at 8:30 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi Maciej, Armin,
>
> On Sun, Oct 21, 2012 at 14:03 +0200, Maciej Fijalkowski wrote:
>> Hi
>>
>> Due to impressive work by Ronan Lamy, we're now able to split RPython
>> and PyPy. Note that the fact of splitting this is not up to
>> discussion, however, how we go about it is. During discussions with
>> Armin we came up with the following plan:
>>
>> * We make a copy of pypy repo called rpython. it'll still live under
>> pypy team on bitbucket.
>>
>> * We'll rename toplevel package of pypy to rpython, for the rpython
>> part. Since we need to change ALL THE IMPORTS, we can do refactorings
>> of imports now. Proposed changes:
>>
>>   * to be moved from pypy to rpython: annotation, translator, rlib
>>   * pypy.config must be split somehow, same for tool, bin and for doc
>>   * move pypy.rpython namespace to rpython.rtyper
>>   * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
>>   * pypy.rpython.memory becomes rpython.gc
>>   * pypy.objspace.flow becomes rpython.flowspace
>>   * testrunner and dotviewer can become independent packages
>>   * _pytest stays with pypy, however for rpython you can use whatever
>> version of py.test you have installed. rpython test suite should also
>> however be runnable under current contents of _pytest
>>
>> * RPython will come with setup.py and be a "normal" python package
>
> great stuff!
>
> Samuele's comments also make sense to me as well (rpython.typer etc.).
>
> Will there be a setup.py with pypy to automatically install perequisites
> (such as RPython, possibly other bits)?  FWIW, I think it could help to
> eventually allow people more easily maintain modular rpython-level pypy
> modules by depending on a specific pypy version and declaring an
> entry-point in order to be automatically picked up by the pypy
> translation process.  This way one could install multiple pypy-MODULE*
> packages, then cause a pypy build and all those modules would become
> part of the translated pypy-c.  (happy to help with this and relate my
> experiences if wanted).  It's not binary compatibility but would be
> a way to have a user choose modules without having to dive into
> translation target scripts etc.
>
> best,
> holger

IMO PyPy should not come with setup.py (unlike RPython). The main
reason why is that PyPy is not a python package - after all python
setup.py install will not do what you think it'll do. I think RPython
will be the only dependency really.

From holger at merlinux.eu  Mon Oct 22 09:27:05 2012
From: holger at merlinux.eu (holger krekel)
Date: Mon, 22 Oct 2012 07:27:05 +0000
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <CAK5idxS22S8f=Wc17GJGV_REKhZDAF6nOQwaYFRJ0ivJ6sJr-A@mail.gmail.com>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<20121021183043.GE27773@merlinux.eu>
	<CAK5idxS22S8f=Wc17GJGV_REKhZDAF6nOQwaYFRJ0ivJ6sJr-A@mail.gmail.com>
Message-ID: <20121022072705.GG27773@merlinux.eu>

On Mon, Oct 22, 2012 at 07:00 +0200, Maciej Fijalkowski wrote:
> On Sun, Oct 21, 2012 at 8:30 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Maciej, Armin,
> >
> > On Sun, Oct 21, 2012 at 14:03 +0200, Maciej Fijalkowski wrote:
> >> Hi
> >>
> >> Due to impressive work by Ronan Lamy, we're now able to split RPython
> >> and PyPy. Note that the fact of splitting this is not up to
> >> discussion, however, how we go about it is. During discussions with
> >> Armin we came up with the following plan:
> >>
> >> * We make a copy of pypy repo called rpython. it'll still live under
> >> pypy team on bitbucket.
> >>
> >> * We'll rename toplevel package of pypy to rpython, for the rpython
> >> part. Since we need to change ALL THE IMPORTS, we can do refactorings
> >> of imports now. Proposed changes:
> >>
> >>   * to be moved from pypy to rpython: annotation, translator, rlib
> >>   * pypy.config must be split somehow, same for tool, bin and for doc
> >>   * move pypy.rpython namespace to rpython.rtyper
> >>   * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype
> >>   * pypy.rpython.memory becomes rpython.gc
> >>   * pypy.objspace.flow becomes rpython.flowspace
> >>   * testrunner and dotviewer can become independent packages
> >>   * _pytest stays with pypy, however for rpython you can use whatever
> >> version of py.test you have installed. rpython test suite should also
> >> however be runnable under current contents of _pytest
> >>
> >> * RPython will come with setup.py and be a "normal" python package
> >
> > great stuff!
> >
> > Samuele's comments also make sense to me as well (rpython.typer etc.).
> >
> > Will there be a setup.py with pypy to automatically install perequisites
> > (such as RPython, possibly other bits)?  FWIW, I think it could help to
> > eventually allow people more easily maintain modular rpython-level pypy
> > modules by depending on a specific pypy version and declaring an
> > entry-point in order to be automatically picked up by the pypy
> > translation process.  This way one could install multiple pypy-MODULE*
> > packages, then cause a pypy build and all those modules would become
> > part of the translated pypy-c.  (happy to help with this and relate my
> > experiences if wanted).  It's not binary compatibility but would be
> > a way to have a user choose modules without having to dive into
> > translation target scripts etc.
> >
> > best,
> > holger
> 
> IMO PyPy should not come with setup.py (unlike RPython). The main
> reason why is that PyPy is not a python package - after all python
> setup.py install will not do what you think it'll do. I think RPython
> will be the only dependency really.

Trying "python setup.py install" can be made to bail out pointing
to what one needs to do, though.  Or maybe just install a
"pypy_translate" script so one doesn't need to fish.  The other
orthogonal issue i raised is how to organise modularity of rpython-level
pypy modules without the need to hack at the repo.

But of course this could all be considered/done independently after the
repo split so it's not directly related.

best,
holger

From arigo at tunes.org  Mon Oct 22 10:23:21 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 22 Oct 2012 10:23:21 +0200
Subject: [pypy-dev] Splitting RPython and PyPy
In-Reply-To: <1350847730.2277.14.camel@ronan-desktop>
References: <CAK5idxSwnRBvBUg_PtEgBNkirUv4qUQ4CRaeOQHk1EpQJ7BB4A@mail.gmail.com>
	<1350847730.2277.14.camel@ronan-desktop>
Message-ID: <CAMSv6X21L2XeKsQzwHfBDJSTo4OoFx50XvuPkcBUmSYsbzf5QQ@mail.gmail.com>

Hi Ronan,

On Sun, Oct 21, 2012 at 9:28 PM, Ronan Lamy <ronan.lamy at gmail.com> wrote:
> I think that the changes should go deeper. I see model.py and
> operation.py as defining the flowgraph-based intermediate representation
> on which the rest of the tool chain operates. (...)

I believe you're putting too much focus on the flow object space; it's
only part 1 of many in the translation process (even though it was
that part that kept us from splitting RPython and PyPy so far).  But
anyway as others have pointed out we can discuss such additional moves
after the main split.

>>   * _pytest stays with pypy, however for rpython you can use whatever
>> version of py.test you have installed. rpython test suite should also
>> however be runnable under current contents of _pytest
>
> Why the last stipulation? I would think that fishing out _pytest from
> pypy would require significant acrobatics once the repos are separated.

That last stipulation was saying: if you install the PyPy and the
RPython repos, then you should be able to use the (older) version of
py.test that comes with PyPy in order to run tests in the RPython repo
as well.  In other words, RPython's tests should not suddenly become
dependent on newer features of py.test.


A bient?t,

Armin.

From fijall at gmail.com  Thu Oct 25 11:21:09 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 25 Oct 2012 11:21:09 +0200
Subject: [pypy-dev] [pypy-commit] pypy default: fix jit translation,
 don't lookinside this function until ptradd works with things
 besides ccharp
In-Reply-To: <20121025023845.EFB071C0F47@cobra.cs.uni-duesseldorf.de>
References: <20121025023845.EFB071C0F47@cobra.cs.uni-duesseldorf.de>
Message-ID: <CAK5idxQEBXpq0qNGn1HpZUCh=seB7ntqdP0y=JOwWvLHaxJK=g@mail.gmail.com>

On Thu, Oct 25, 2012 at 4:38 AM, alex_gaynor <noreply at buildbot.pypy.org> wrote:
> Author: Alex Gaynor <alex.gaynor at gmail.com>
> Branch:
> Changeset: r58413:88158af0627f
> Date: 2012-10-24 19:38 -0700
> http://bitbucket.org/pypy/pypy/changeset/88158af0627f/
>
> Log:    fix jit translation, don't lookinside this function until ptradd
>         works with things besides ccharp

We can use raw_load and raw_store (from pypy.rlib.rawstorage) that can
be understood by the JIT. making ptradd understood is also not a big
deal.

>
> diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py
> --- a/pypy/module/array/interp_array.py
> +++ b/pypy/module/array/interp_array.py
> @@ -9,6 +9,7 @@
>  from pypy.objspace.std.multimethod import FailedToImplement
>  from pypy.objspace.std.stdtypedef import SMM, StdTypeDef
>  from pypy.objspace.std.register_all import register_all
> +from pypy.rlib import jit
>  from pypy.rlib.rarithmetic import ovfcheck, widen
>  from pypy.rlib.unroll import unrolling_iterable
>  from pypy.rlib.objectmodel import specialize, keepalive_until_here
> @@ -466,6 +467,7 @@
>          self.setlen(0)
>          self.fromsequence(w_lst)
>
> +    @jit.dont_look_inside
>      def delslice__Array_ANY_ANY(space, self, w_i, w_j):
>          i = space.int_w(w_i)
>          if i < 0:
> @@ -495,13 +497,13 @@
>              rffi.c_memcpy(
>                  rffi.cast(rffi.VOIDP, rffi.ptradd(self.buffer, i)),
>                  rffi.cast(rffi.VOIDP, rffi.ptradd(oldbuffer, j)),
> -                (self.len - j) * mytype.bytes)
> +                (self.len - j) * mytype.bytes
> +            )
>          self.len -= j - i
>          self.allocated = self.len
>          if oldbuffer:
>              lltype.free(oldbuffer, flavor='raw')
>
> -
>      # Add and mul methods
>
>      def add__Array_Array(space, self, other):
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> http://mail.python.org/mailman/listinfo/pypy-commit

From mumrah at gmail.com  Fri Oct 26 19:44:54 2012
From: mumrah at gmail.com (David Arthur)
Date: Fri, 26 Oct 2012 13:44:54 -0400
Subject: [pypy-dev] Trouble translating llvm
Message-ID: <CA+0Ze6orjcXD5ncJkA855ZVwiLuiC8Mw_kMzu=w3pXtT1rOa2w@mail.gmail.com>

Running "python translate.py -Ojit --jit-backend=llvm" with pypy 1.9
(pypy-pypy-341e1e3821ff), I get

[translation:info] JIT compiler generation...
[Timer] Timings:
[Timer] annotate                       --- 1354.8 s
[Timer] rtype_lltype                   --- 1855.0 s
[Timer] pyjitpl_lltype                 ---    0.2 s
[Timer] ===========================================
[Timer] Total:                         --- 3210.1 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 308, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
line 791, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/tool/taskengine.py",
line 116, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
line 285, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
line 397, in task_pyjitpl_lltype
[translation:ERROR]
backend_name=self.config.translation.jit_backend, inline=True)
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/metainterp/warmspot.py",
line 35, in apply_jit
[translation:ERROR]     kwds['CPUClass'] = getcpuclass(backend_name)
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/detect_cpu.py",
line 82, in getcpuclass
[translation:ERROR]     mod = __import__(modname, {}, {}, clsname)
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/runner.py",
line 9, in <module>
[translation:ERROR]     from pypy.jit.backend.llvm import llvm_rffi
[translation:ERROR]    File
"/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/llvm_rffi.py",
line 4, in <module>
[translation:ERROR]     from pypy.translator.tool.cbuild import
ExternalCompilationInfo, log
[translation:ERROR]  ImportError: cannot import name log

Any insights?

-- 
David Arthur

From fijall at gmail.com  Fri Oct 26 19:50:27 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 26 Oct 2012 19:50:27 +0200
Subject: [pypy-dev] Trouble translating llvm
In-Reply-To: <CA+0Ze6orjcXD5ncJkA855ZVwiLuiC8Mw_kMzu=w3pXtT1rOa2w@mail.gmail.com>
References: <CA+0Ze6orjcXD5ncJkA855ZVwiLuiC8Mw_kMzu=w3pXtT1rOa2w@mail.gmail.com>
Message-ID: <CAK5idxSet7cqajOFvbg5gJud-AvKMrkHOCEpP4iUJ+r6MyV8ag@mail.gmail.com>

The LLVM backend does not work. If it works, it's on some branch or
some fork, no idea where.

It almost definitely does not support the JIT and there is very likely
little reason to use it.

Why did you try to use it?

Cheers,
fijal

On Fri, Oct 26, 2012 at 7:44 PM, David Arthur <mumrah at gmail.com> wrote:
> Running "python translate.py -Ojit --jit-backend=llvm" with pypy 1.9
> (pypy-pypy-341e1e3821ff), I get
>
> [translation:info] JIT compiler generation...
> [Timer] Timings:
> [Timer] annotate                       --- 1354.8 s
> [Timer] rtype_lltype                   --- 1855.0 s
> [Timer] pyjitpl_lltype                 ---    0.2 s
> [Timer] ===========================================
> [Timer] Total:                         --- 3210.1 s
> [translation:ERROR] Error:
> [translation:ERROR]  Traceback (most recent call last):
> [translation:ERROR]    File "translate.py", line 308, in main
> [translation:ERROR]     drv.proceed(goals)
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
> line 791, in proceed
> [translation:ERROR]     return self._execute(goals, task_skip =
> self._maybe_skip())
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/tool/taskengine.py",
> line 116, in _execute
> [translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
> line 285, in _do
> [translation:ERROR]     res = func()
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py",
> line 397, in task_pyjitpl_lltype
> [translation:ERROR]
> backend_name=self.config.translation.jit_backend, inline=True)
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/metainterp/warmspot.py",
> line 35, in apply_jit
> [translation:ERROR]     kwds['CPUClass'] = getcpuclass(backend_name)
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/detect_cpu.py",
> line 82, in getcpuclass
> [translation:ERROR]     mod = __import__(modname, {}, {}, clsname)
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/runner.py",
> line 9, in <module>
> [translation:ERROR]     from pypy.jit.backend.llvm import llvm_rffi
> [translation:ERROR]    File
> "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/llvm_rffi.py",
> line 4, in <module>
> [translation:ERROR]     from pypy.translator.tool.cbuild import
> ExternalCompilationInfo, log
> [translation:ERROR]  ImportError: cannot import name log
>
> Any insights?
>
> --
> David Arthur
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From wlavrijsen at lbl.gov  Fri Oct 26 20:04:14 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 26 Oct 2012 11:04:14 -0700 (PDT)
Subject: [pypy-dev] unable to translate due to isinstance call
Message-ID: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>

Hi,

my translation run fails with:

   -> assert s_value.contains(self.bindings[arg])

where s_value = SomeBool(knowntypedata={}) and self.bindings[arg] is
SomeBool(const=True, knowntypedata={}).

The cause is this:

   if isinstance(self.functions[0], CPPFunction):

where the s_value above is op.result coming the isinstance() call.

With moving things around, the problem comes and goes (hence I thought I
had it fixed already, but it came back ...).

Checking other calls to isinstance() in other modules, I can't see why it
would not be allowed here? Also, what would be the difference between a
const and a non-const bool?

Any hints to get around this one are much appreciated!

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From fijall at gmail.com  Fri Oct 26 20:27:30 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 26 Oct 2012 20:27:30 +0200
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
Message-ID: <CAK5idxR9FowO+oJSYWhogku=HfGiE9MWJyDk0E6LmiuFxw6GiQ@mail.gmail.com>

On Fri, Oct 26, 2012 at 8:04 PM,  <wlavrijsen at lbl.gov> wrote:
> Hi,
>
> my translation run fails with:
>
>   -> assert s_value.contains(self.bindings[arg])
>
> where s_value = SomeBool(knowntypedata={}) and self.bindings[arg] is
> SomeBool(const=True, knowntypedata={}).
>
> The cause is this:
>
>   if isinstance(self.functions[0], CPPFunction):
>
> where the s_value above is op.result coming the isinstance() call.
>
> With moving things around, the problem comes and goes (hence I thought I
> had it fixed already, but it came back ...).
>
> Checking other calls to isinstance() in other modules, I can't see why it
> would not be allowed here? Also, what would be the difference between a
> const and a non-const bool?
>
> Any hints to get around this one are much appreciated!
>
> Thanks,
>      Wim
> --
> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hey.

Can you give me the exact branch and revision so I can try? It depends
a bit on the full traceback.

From wlavrijsen at lbl.gov  Fri Oct 26 20:43:53 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 26 Oct 2012 11:43:53 -0700 (PDT)
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <CAK5idxR9FowO+oJSYWhogku=HfGiE9MWJyDk0E6LmiuFxw6GiQ@mail.gmail.com>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
	<CAK5idxR9FowO+oJSYWhogku=HfGiE9MWJyDk0E6LmiuFxw6GiQ@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1210261130340.16170@pc3-wlav.lbl.gov>

Hi Maciej,

> Can you give me the exact branch and revision so I can try? It depends
> a bit on the full traceback.

current reflex-support branch. Although as said, it comes and goes: I'll
re-run a few times and post the traceback when it occurs again.

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From wlavrijsen at lbl.gov  Fri Oct 26 21:41:33 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 26 Oct 2012 12:41:33 -0700 (PDT)
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <CAK5idxR9FowO+oJSYWhogku=HfGiE9MWJyDk0E6LmiuFxw6GiQ@mail.gmail.com>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
	<CAK5idxR9FowO+oJSYWhogku=HfGiE9MWJyDk0E6LmiuFxw6GiQ@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1210261239500.16170@pc3-wlav.lbl.gov>

Hi Majiec,

> Can you give me the exact branch and revision so I can try? It depends
> a bit on the full traceback.

full stack trace below. One way out seems to be to @jit.elidable_promote(),
which is fine, although seems superfluous (the functions[] list is already
declared to be immutable).

Thanks,
      Wim

[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 303, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/translator/driver.py", line 771, in proceed
[translation:ERROR]     return self._execute(goals, task_skip = self._maybe_skip())
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/translator/tool/taskengine.py", line 116, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/translator/driver.py", line 283, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/translator/driver.py", line 319, in task_annotate
[translation:ERROR]     s = annotator.build_types(self.entry_point, self.inputtypes)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 89, in build_types
[translation:ERROR]     return self.build_graph_types(flowgraph, inputcells, complete_now=complete_now)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 142, in build_graph_types
[translation:ERROR]     self.complete()
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 195, in complete
[translation:ERROR]     self.processblock(graph, block)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 340, in processblock
[translation:ERROR]     self.flowin(graph, block)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 399, in flowin
[translation:ERROR]     self.consider_op(block, i)
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 605, in consider_op
[translation:ERROR]     self.setbinding(op.result, resultcell)  # bind resultcell to op.result
[translation:ERROR]    File "/home/wlav/pypy-dev/pypy/pypy/annotation/annrpython.py", line 247, in setbinding
[translation:ERROR]     assert s_value.contains(self.bindings[arg])
[translation:ERROR]  AssertionError
[translation:ERROR] Processing block:
[translation:ERROR]  block at 6 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.cppyy.interp_cppyy:425)W_CPPOverload.is_static
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v585 = getattr(self_79, ('functions'))
[translation:ERROR]        v586 = getitem(v585, (0))
[translation:ERROR]        v587 = simple_call((builtin_function isinstance), v586, (type CPPFunction))
[translation:ERROR]        v588 = is_true(v587)
[translation:ERROR]  --end--

-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From arigo at tunes.org  Sat Oct 27 09:45:34 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Oct 2012 09:45:34 +0200
Subject: [pypy-dev] Trouble translating llvm
In-Reply-To: <CA+0Ze6orjcXD5ncJkA855ZVwiLuiC8Mw_kMzu=w3pXtT1rOa2w@mail.gmail.com>
References: <CA+0Ze6orjcXD5ncJkA855ZVwiLuiC8Mw_kMzu=w3pXtT1rOa2w@mail.gmail.com>
Message-ID: <CAMSv6X2DX-F6Ro_jfJ4VXmMeCPhqwYMxO-Xq+AqZeTcZvQPNJw@mail.gmail.com>

Hi David,

On Fri, Oct 26, 2012 at 7:44 PM, David Arthur <mumrah at gmail.com> wrote:
> Running "python translate.py -Ojit --jit-backend=llvm" with pypy 1.9

Oups, my fault for leaving the option "--jit-backend=llvm" around when
the llvm jit backend was removed long ago.  It never worked.  (Removed
the option in f0d036cd3b7b.)


A bient?t,

Armin.

From arigo at tunes.org  Sat Oct 27 10:30:02 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Oct 2012 10:30:02 +0200
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
Message-ID: <CAMSv6X1H_bgb_dccGD3AhahRRFXH=CTv259V1C3OLtMBuZtd0A@mail.gmail.com>

Hi Wim,

On Fri, Oct 26, 2012 at 8:04 PM,  <wlavrijsen at lbl.gov> wrote:
>   -> assert s_value.contains(self.bindings[arg])
>
> where s_value = SomeBool(knowntypedata={}) and self.bindings[arg] is
> SomeBool(const=True, knowntypedata={}).

Are you sure about that?  Isn't it the opposite: s_value has the
'const' but self.bindings[arg] doesn't?


A bient?t,

Armin.

From arigo at tunes.org  Sat Oct 27 10:56:28 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Oct 2012 10:56:28 +0200
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <CAMSv6X1H_bgb_dccGD3AhahRRFXH=CTv259V1C3OLtMBuZtd0A@mail.gmail.com>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
	<CAMSv6X1H_bgb_dccGD3AhahRRFXH=CTv259V1C3OLtMBuZtd0A@mail.gmail.com>
Message-ID: <CAMSv6X0WaG+pgM+SWYkKrSuMyEh07-sn89gExF_LtDXC_mMcqQ@mail.gmail.com>

Re-hi,

On Sat, Oct 27, 2012 at 10:30 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Wim,
>
> On Fri, Oct 26, 2012 at 8:04 PM,  <wlavrijsen at lbl.gov> wrote:
>>   -> assert s_value.contains(self.bindings[arg])
>>
>> where s_value = SomeBool(knowntypedata={}) and self.bindings[arg] is
>> SomeBool(const=True, knowntypedata={}).
>
> Are you sure about that?  Isn't it the opposite: s_value has the
> 'const' but self.bindings[arg] doesn't?

Indeed, that's how it crashes.  The issue comes from the fact that
"knowntypedata={}" should not be attached to SomeBools, a fact that we
overlooked when refactoring builtin_isinstance().  Fixed by
introducing a cleaner interface in 28ae0f0e0b79.


A bient?t,

Armin.

From wlavrijsen at lbl.gov  Sun Oct 28 21:39:25 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Sun, 28 Oct 2012 13:39:25 -0700 (PDT)
Subject: [pypy-dev] unable to translate due to isinstance call
In-Reply-To: <CAMSv6X0WaG+pgM+SWYkKrSuMyEh07-sn89gExF_LtDXC_mMcqQ@mail.gmail.com>
References: <alpine.LNX.2.00.1210261058310.16170@pc3-wlav.lbl.gov>
	<CAMSv6X1H_bgb_dccGD3AhahRRFXH=CTv259V1C3OLtMBuZtd0A@mail.gmail.com>
	<CAMSv6X0WaG+pgM+SWYkKrSuMyEh07-sn89gExF_LtDXC_mMcqQ@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1210281338580.26396@pc3-wlav.lbl.gov>

Hi Armin,

> Fixed by introducing a cleaner interface in 28ae0f0e0b79.

thanks! Works for me.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From wlavrijsen at lbl.gov  Mon Oct 29 23:19:23 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 29 Oct 2012 15:19:23 -0700 (PDT)
Subject: [pypy-dev] documentation of _immutable_?
Message-ID: <alpine.LNX.2.00.1210291514570.3558@pc3-wlav.lbl.gov>

Hi,

any place where I can find docs describing _immutable_ and _immutable_fields_
in some detail? As it happens, they don't quite do what I expected, which led
to bugs. I'd like to know what their scope is and what the wildcards do.

Right now I just removed them all to be safe (and allow some experimentation).
But that is costing performance (20% extra overhead).

Also, I see _attrs_ in quite a few places. What is it for?

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From fijall at gmail.com  Tue Oct 30 00:15:04 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 30 Oct 2012 01:15:04 +0200
Subject: [pypy-dev] documentation of _immutable_?
In-Reply-To: <alpine.LNX.2.00.1210291514570.3558@pc3-wlav.lbl.gov>
References: <alpine.LNX.2.00.1210291514570.3558@pc3-wlav.lbl.gov>
Message-ID: <CAK5idxRkTUyeOYzhePvs5vRUgmwAnwfkMx66V0L9ghpr6Dqhcw@mail.gmail.com>

On Tue, Oct 30, 2012 at 12:19 AM,  <wlavrijsen at lbl.gov> wrote:
> Hi,
>
> any place where I can find docs describing _immutable_ and
> _immutable_fields_
> in some detail? As it happens, they don't quite do what I expected, which
> led
> to bugs. I'd like to know what their scope is and what the wildcards do.
>
> Right now I just removed them all to be safe (and allow some
> experimentation).
> But that is costing performance (20% extra overhead).
>
> Also, I see _attrs_ in quite a few places. What is it for?
>
> Thanks,
>      Wim
> --
> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hi Wim.

I'll write some docs for them later this week, this is a bit embarrasing :)

From fijall at gmail.com  Tue Oct 30 00:17:49 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 30 Oct 2012 01:17:49 +0200
Subject: [pypy-dev] documentation of _immutable_?
In-Reply-To: <CAK5idxRkTUyeOYzhePvs5vRUgmwAnwfkMx66V0L9ghpr6Dqhcw@mail.gmail.com>
References: <alpine.LNX.2.00.1210291514570.3558@pc3-wlav.lbl.gov>
	<CAK5idxRkTUyeOYzhePvs5vRUgmwAnwfkMx66V0L9ghpr6Dqhcw@mail.gmail.com>
Message-ID: <CAK5idxQFwCV_yg03rVRcEgCQgiiJwhc21Jr6CLrbuHZmpw-aXg@mail.gmail.com>

On Tue, Oct 30, 2012 at 1:15 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Tue, Oct 30, 2012 at 12:19 AM,  <wlavrijsen at lbl.gov> wrote:
>> Hi,
>>
>> any place where I can find docs describing _immutable_ and
>> _immutable_fields_
>> in some detail? As it happens, they don't quite do what I expected, which
>> led
>> to bugs. I'd like to know what their scope is and what the wildcards do.
>>
>> Right now I just removed them all to be safe (and allow some
>> experimentation).
>> But that is costing performance (20% extra overhead).
>>
>> Also, I see _attrs_ in quite a few places. What is it for?
>>
>> Thanks,
>>      Wim
>> --
>> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
> Hi Wim.
>
> I'll write some docs for them later this week, this is a bit embarrasing :)

A very short answer is "never use _immutable_ it's very confusing".
_immutable_fields_ is better (for subclassing) Wildcards mean "array
on the instance is immutable" so a[*] means x.a[3] will be constant
folded, if x is a constant.

From wlavrijsen at lbl.gov  Tue Oct 30 01:51:21 2012
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 29 Oct 2012 17:51:21 -0700 (PDT)
Subject: [pypy-dev] documentation of _immutable_?
In-Reply-To: <CAK5idxQFwCV_yg03rVRcEgCQgiiJwhc21Jr6CLrbuHZmpw-aXg@mail.gmail.com>
References: <alpine.LNX.2.00.1210291514570.3558@pc3-wlav.lbl.gov>
	<CAK5idxRkTUyeOYzhePvs5vRUgmwAnwfkMx66V0L9ghpr6Dqhcw@mail.gmail.com>
	<CAK5idxQFwCV_yg03rVRcEgCQgiiJwhc21Jr6CLrbuHZmpw-aXg@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1210291736571.3558@pc3-wlav.lbl.gov>

Hi Maciej,

> A very short answer is "never use _immutable_ it's very confusing".

that I found out the hard way already. :)

> _immutable_fields_ is better (for subclassing) Wildcards mean "array
> on the instance is immutable" so a[*] means x.a[3] will be constant
> folded, if x is a constant.

I've also seen an '?' appended to field names? But it is the last set of
rules that I'm interested in (the "if" portion of the rule in particular).
Something like it seems to have bit me: there was a field that was constant
only by inheritance of _immutable_ = True, but it didn't hurt until the
container variable received a jit.promote(). With both in place, I got
wrong results. With the promote() removed, I got:

[bogus _immutable_field_ declaration: <FieldU pypy.module.cppyy.executor.BasicRefExecutor.inst_do_assign 12>]

which was very nice (as a diagnostic). Removing the _immutable_ = True,
however, did not resolve the issue like I expected, as (I think, still
investigating) the object was part of an object that also had _immutable_
set to True, making it immutable still.

My current guess is that I should selectively use _immutable_fields_ only,
and not promote whole objects, but only fields of objects.

The other issue that I have, is that it seems to be okay to make a field
immutable if it will be set once on first use (i.e. if guaranteed to be set
before the JIT kicks in)? If true, that'd be a nice feature.

Looking forward to those docs. :)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From sasikanth.ece at gmail.com  Tue Oct 30 16:19:51 2012
From: sasikanth.ece at gmail.com (Sasikanth Eda)
Date: Tue, 30 Oct 2012 20:49:51 +0530
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
Message-ID: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>

Hi All,

I have tried to perform a CPU intensive operation ( which typically
involved calculation of prime numbers from 1-100000 ) using Multi-process
and Multi-Thread models provided by
C, Python-2.6, PyPy-1.9.

Here are results that are obtained from this exercise;

*C Program results ( Number of processes = 500 )*

CPU utilization      = 99.81% (peak)
Memory utilization = 14.77 GB (peak)
Execution time      = *2m16s* (real) ( time taken between invocation to
termination )

*C Program results ( Number of Threads = 500 )*

CPU utilization      = 99.94% (peak)
Memory utilization = 14.79 GB (peak)
Execution time      = *2m10s* (real)

*{ Here I came to a conclusion that C gave a comparatively equal
good results in terms of both Multi-Threaded as well as Multi-Process }*

*Python Program results ( Number of processes = 500 )*

CPU utilization      = 99.62% (peak)
Memory utilization = 17.03 GB (peak)
Execution time      = *87m42.785s* (real)

*Python Program results ( Number of Threads = 500 )*

CPU utilization      = 18.23% (peak)
Memory utilization = 33.074 GB (peak)
Execution time      = *239m10.210s* (real)

*{ Here I came to a conclusion that Python is limited by the Global
Interpreter Look (GIL) when used in Multi-Thread mode (that is why it gave
poor results when compared with process model }*

*PyPy Program results ( Number of processes = 500 )*

CPU utilization      = 99.19% (peak)
Memory utilization = 22.92 GB (peak)
Execution time      = *6m4.970s* (real)

*PyPy Program results ( Number of Threads = 500 )*

CPU utilization      = 38.88% (peak)
Memory utilization = 21.06 GB (peak)
Execution time      = *59m29s* (real)

*{ Here I came to a conclusion that PyPy is better than Python }*
*
*
Now what am I struggling to understand is ;
*
*
1.  Has PyPy optimized / reduced the GIL limitation ? ( what is the
progress in PyPy version 1.9 in that when compared with Python's progress )
2.  If PyPy is also suffering from the same GIL limitations, what made the
program run faster than Python, is it because of
more warm-up time, optimization of loops  ?

3.  What are your suggestions for me if I wanted to go for
Multi-Thread application design ( in-terms of Python / PyPy )

I request your inputs/suggestion techniques towards
optimizing Multi threaded load loads.

Kindly share your experiences and feedback in this scenario.

-- 
Sasikanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121030/f6633640/attachment.html>

From dripton at ripton.net  Tue Oct 30 17:33:10 2012
From: dripton at ripton.net (David Ripton)
Date: Tue, 30 Oct 2012 12:33:10 -0400
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
Message-ID: <50900146.50903@ripton.net>

On 10/30/2012 11:19 AM, Sasikanth Eda wrote:

> I have tried to perform a CPU intensive operation ( which typically
> involved calculation of prime numbers from 1-100000 ) using
> Multi-process and Multi-Thread models provided by
> C, Python-2.6, PyPy-1.9.

Note that your code is very slow.  I can find all primes up to 100000 in 
less than 0.3s in CPython and less than 0.2s in PyPy, using a single 
process with a single thread.

> */{ Here I came to a conclusion that Python is limited by the Global
> Interpreter Look (GIL) when used in Multi-Thread mode (that is why it
> gave poor results when compared with process model }/*

Yes.

> *{ /Here I came to a conclusion that PyPy is better than Python }/*

Yes, PyPy is faster than CPython for most (not all!) Python programs.

> 1.  Has PyPy optimized / reduced the GIL limitation ? ( what is the
> progress in PyPy version 1.9 in that when compared with Python's progress )
> 2.  If PyPy is also suffering from the same GIL limitations, what made
> the program run faster than Python, is it because of
> more warm-up time, optimization of loops  ?

PyPy has a Just In Time (JIT) compiler that compiles hot loops to 
machine language.

> 3.  What are your suggestions for me if I wanted to go for
> Multi-Thread application design ( in-terms of Python / PyPy )

For now, you will not be happy with multiple CPU-bound threads in 
CPython or PyPy, because of the GIL.  Jython is free-threaded but is 
slow in other ways.  If you insist on using threads for CPU-bound work, 
you will probably be happier with another language.

However, Armin Rigo is working on Software Transactional Memory for 
PyPy, which may someday end up making multi-threaded code fast.  Web 
search for "pypy stm" to find details.

-- 
David Ripton    dripton at ripton.net

From arigo at tunes.org  Tue Oct 30 17:39:44 2012
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 30 Oct 2012 17:39:44 +0100
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
Message-ID: <CAMSv6X2KXi4Bz+kTWCKGcYQFd9VosRw=X+gRnQCyKEH6kf5qDg@mail.gmail.com>

Hi,

On Tue, Oct 30, 2012 at 4:19 PM, Sasikanth Eda <sasikanth.ece at gmail.com> wrote:
> 1.  Has PyPy optimized / reduced the GIL limitation ?

Not so far: PyPy has a GIL, similar to CPython.  (Note that I'm
answering with "CPython" here to make it clearer; "Python" means the
language which both CPython and PyPy implement.)

> 2.  If PyPy is also suffering from the same GIL limitations, what made the
> program run faster than Python

PyPy is faster than CPython in a lot of cases.  It has a rather good
just-in-time compiler.

> 3.  What are your suggestions for me if I wanted to go for Multi-Thread
> application design ( in-terms of Python / PyPy )

One of the researchy goals of PyPy is to use STM to make
multi-threaded programming easier and still scale to multiple CPUs:
http://morepypy.blogspot.ch/2012/08/multicore-programming-in-pypy-and.html
.  This is still in-development.

Note that the speed you'll get is unlikely to be better than using
multiple processes, so doing that is the best solution for a few
cases.  But there are also a few other cases where using multiple
processes would be very hard.  Moreover, the theory goes that in all
remaining intermediate cases, using multiple threads ---not directly
but hidden below some interface like thread.atomic--- is actually much
easier and cleaner than using multiple processes.


A bient?t,

Armin.

From drsalists at gmail.com  Tue Oct 30 23:53:25 2012
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 30 Oct 2012 15:53:25 -0700
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <50900146.50903@ripton.net>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
	<50900146.50903@ripton.net>
Message-ID: <CAGGBd_oO3Kfyz3KM76+d6u_93D5-0CtoczT=SijajZi8rZeSdw@mail.gmail.com>

On Tue, Oct 30, 2012 at 9:33 AM, David Ripton <dripton at ripton.net> wrote:

> On 10/30/2012 11:19 AM, Sasikanth Eda wrote:
>
>  I have tried to perform a CPU intensive operation ( which typically
>> involved calculation of prime numbers from 1-100000 ) using
>> Multi-process and Multi-Thread models provided by
>> C, Python-2.6, PyPy-1.9.
>>
>

> For now, you will not be happy with multiple CPU-bound threads in CPython
> or PyPy, because of the GIL.  Jython is free-threaded but is slow in other
> ways.  If you insist on using threads for CPU-bound work, you will probably
> be happier with another language.
>

I've actually found Jython rather slow to start up, but decent once it's
loaded.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121030/7559248e/attachment.html>

From balasrin at gmail.com  Wed Oct 31 08:04:40 2012
From: balasrin at gmail.com (Srinivasan Balasubramanian)
Date: Wed, 31 Oct 2012 12:34:40 +0530
Subject: [pypy-dev] Problems in compiling libpypy-c.so
Message-ID: <CA+8r7F01zhdkHpHsgynhpcxrc3oiUQN57LLnQLqSpK6Agi5f0w@mail.gmail.com>

Folks,
   I am trying to run a python uwsgi container using pypy. For doing that i
had to compile libpypy-c.so and i have been getting numerous errors while
doing so.  I was trying to build pypy-1.8 release. I have couple of
questions in this regard.

1)Does anybody know of an RPM containing libpypy-c.so for CentOS release
5.4?
2)Has anybody compiled libpypy-c.so,pypy-c for CentOS release 5.4 ?

I would be delighted If someone can share the libpypy-c.so and pypy-c.

-Srini
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121031/f03faa66/attachment-0001.html>

From amauryfa at gmail.com  Wed Oct 31 08:42:18 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 31 Oct 2012 08:42:18 +0100
Subject: [pypy-dev] Problems in compiling libpypy-c.so
In-Reply-To: <CA+8r7F01zhdkHpHsgynhpcxrc3oiUQN57LLnQLqSpK6Agi5f0w@mail.gmail.com>
References: <CA+8r7F01zhdkHpHsgynhpcxrc3oiUQN57LLnQLqSpK6Agi5f0w@mail.gmail.com>
Message-ID: <CAGmFidbU5fL+cO50Hc1QAVynUVvmq-E_s8JDz4TxdZfg6O78UA@mail.gmail.com>

2012/10/31 Srinivasan Balasubramanian <balasrin at gmail.com>:
> Folks,
>    I am trying to run a python uwsgi container using pypy. For doing that i
> had to compile libpypy-c.so and i have been getting numerous errors while
> doing so.  I was trying to build pypy-1.8 release. I have couple of
> questions in this regard.
>
> 1)Does anybody know of an RPM containing libpypy-c.so for CentOS release
> 5.4?
> 2)Has anybody compiled libpypy-c.so,pypy-c for CentOS release 5.4 ?
>
> I would be delighted If someone can share the libpypy-c.so and pypy-c.

Did you see the instructions on this page?
http://projects.unbit.it/uwsgi/wiki/PyPy

-- 
Amaury Forgeot d'Arc

From phyo.arkarlwin at gmail.com  Wed Oct 31 15:14:12 2012
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Wed, 31 Oct 2012 20:44:12 +0630
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <CAGGBd_oO3Kfyz3KM76+d6u_93D5-0CtoczT=SijajZi8rZeSdw@mail.gmail.com>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
	<50900146.50903@ripton.net>
	<CAGGBd_oO3Kfyz3KM76+d6u_93D5-0CtoczT=SijajZi8rZeSdw@mail.gmail.com>
Message-ID: <CA+HjJriBgP6-Z+7c14CCmYvupSxon_svdCfd73+utKLxzdXuYA@mail.gmail.com>

Jython is wayyy much slower than CPython in many cases. I can't even find
its use.
On Wed, Oct 31, 2012 at 5:23 AM, Dan Stromberg <drsalists at gmail.com> wrote:

>
> On Tue, Oct 30, 2012 at 9:33 AM, David Ripton <dripton at ripton.net> wrote:
>
>> On 10/30/2012 11:19 AM, Sasikanth Eda wrote:
>>
>>  I have tried to perform a CPU intensive operation ( which typically
>>> involved calculation of prime numbers from 1-100000 ) using
>>> Multi-process and Multi-Thread models provided by
>>> C, Python-2.6, PyPy-1.9.
>>>
>>
>
>> For now, you will not be happy with multiple CPU-bound threads in CPython
>> or PyPy, because of the GIL.  Jython is free-threaded but is slow in other
>> ways.  If you insist on using threads for CPU-bound work, you will probably
>> be happier with another language.
>>
>
> I've actually found Jython rather slow to start up, but decent once it's
> loaded.
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121031/761e9583/attachment.html>

From arigo at tunes.org  Wed Oct 31 15:52:08 2012
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 31 Oct 2012 15:52:08 +0100
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <CA+HjJriBgP6-Z+7c14CCmYvupSxon_svdCfd73+utKLxzdXuYA@mail.gmail.com>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
	<50900146.50903@ripton.net>
	<CAGGBd_oO3Kfyz3KM76+d6u_93D5-0CtoczT=SijajZi8rZeSdw@mail.gmail.com>
	<CA+HjJriBgP6-Z+7c14CCmYvupSxon_svdCfd73+utKLxzdXuYA@mail.gmail.com>
Message-ID: <CAMSv6X3x78pRL_bhu2MoTY+EMZ+6mSqFBtt+577JJenxDaosEQ@mail.gmail.com>

Hi,

On Wed, Oct 31, 2012 at 3:14 PM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> Jython is wayyy much slower than CPython in many cases. I can't even find
> its use.

If your problem is hard to divide in multiple processes but easy to
divide in multiple threads, and if you have a machine with enough
CPUs, then Jython wins.  Although dividing in multiple raw threads is
rarely easy.


A bient?t,

Armin.

From phyo.arkarlwin at gmail.com  Wed Oct 31 16:05:32 2012
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Wed, 31 Oct 2012 21:35:32 +0630
Subject: [pypy-dev] Need help in concluding Multi-Thread based model
 thoughts between Python and PyPy
In-Reply-To: <CAMSv6X3x78pRL_bhu2MoTY+EMZ+6mSqFBtt+577JJenxDaosEQ@mail.gmail.com>
References: <CAKw8Pp7hJ9N1R8QqhqnASO=sm=KBDOmGD9ct_OkM8CoFrtMjtQ@mail.gmail.com>
	<50900146.50903@ripton.net>
	<CAGGBd_oO3Kfyz3KM76+d6u_93D5-0CtoczT=SijajZi8rZeSdw@mail.gmail.com>
	<CA+HjJriBgP6-Z+7c14CCmYvupSxon_svdCfd73+utKLxzdXuYA@mail.gmail.com>
	<CAMSv6X3x78pRL_bhu2MoTY+EMZ+6mSqFBtt+577JJenxDaosEQ@mail.gmail.com>
Message-ID: <CA+HjJrgiV-tdwMST8wTPGNEtkaHJz5FfuryVrk7NyFH_SVYfQQ@mail.gmail.com>

I see that may be the case.
My program uses a lot of File IO and it is divided into multiple processess
, working good in python.Failing so far in pypy (due to requirement of so
many different document parsers) .
I tested with Jython and i frustrated (At-least , in jython i can use Tika
to parse documents) .

On Wed, Oct 31, 2012 at 9:22 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi,
>
> On Wed, Oct 31, 2012 at 3:14 PM, Phyo Arkar <phyo.arkarlwin at gmail.com>
> wrote:
> > Jython is wayyy much slower than CPython in many cases. I can't even find
> > its use.
>
> If your problem is hard to divide in multiple processes but easy to
> divide in multiple threads, and if you have a machine with enough
> CPUs, then Jython wins.  Although dividing in multiple raw threads is
> rarely easy.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121031/b772eac3/attachment.html>