From vincent.legoll at gmail.com  Sun Mar  1 09:44:45 2015
From: vincent.legoll at gmail.com (Vincent Legoll)
Date: Sun, 1 Mar 2015 09:44:45 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <CAMSv6X3Bn6UZH25QqRsDXqoEOJk581nHF4saFPuFaAcPvvgdRA@mail.gmail.com>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
 <54ED135C.3080405@oz.net>
 <CAEwRq=rjos2MZRmmeGx-gXaGeG2T11vv8h9GY01KWnYF9HGqDg@mail.gmail.com>
 <54EDDA3F.4040804@oz.net>
 <CAMSv6X3Bn6UZH25QqRsDXqoEOJk581nHF4saFPuFaAcPvvgdRA@mail.gmail.com>
Message-ID: <CAEwRq=o3Jge6oGNUPJi2=USbhAtvxUZk7RMVBY9zLpSvs0ro2A@mail.gmail.com>

I coincidentally read something related yesterday, it's not completely on
topic, but was an interesting bit of info about firefox OS JIT and the use
of fork() to amortize some costs.

You can start reading at :
http://www.realworldtech.com/forum/?threadid=146598&curpostid=147184

That's a digression in a (long but interesting) thread about the Mill CPU
architecture (currently non existent)


On Sat, Feb 28, 2015 at 6:23 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Bengt,
>
> On 25 February 2015 at 15:20, Bengt Richter <bokr at oz.net> wrote:
> > Maybe it's worth a re-think, if only to say "no, we really mean no" in
> the
> > FAQ ;-)
>
> It's unclear to me if you're suggesting something more than adding a
> checkpointing system to stop and restart the process.  It's a hack
> that might work in some cases and not in others.  For example,
> re-opening the files and re-seeking them to the same position as
> before --- it seems interesting but I somehow doubt it is very helpful
> in general.
>
> Another point of view on your suggestion is "just use os.fork()", as
> this solves all these hard issues out of the box.  Of course you can't
> save the forked process to disk, but that seems like a small price to
> pay and it works on today's PyPy.
>
>
> A bient?t,
>
> Armin.
>



-- 
Vincent Legoll
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150301/fd8bc10b/attachment.html>

From rich at pasra.at  Tue Mar  3 16:32:43 2015
From: rich at pasra.at (Richard Plangger)
Date: Tue, 03 Mar 2015 16:32:43 +0100
Subject: [pypy-dev] GSoC 15 vectorizing traces
Message-ID: <54F5D41B.8070108@pasra.at>

hi,

pypy participated in GSoC 14 and my thesis mentor had the idea that it
would benefit if I apply to GSoC 15 for my thesis. I'm currently
planning my thesis and I think the application would be very similar to
what is required to GSoC 15.
Do you think that it would be worth trying?

Best,
Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150303/d2cc1b85/attachment.sig>

From arigo at tunes.org  Tue Mar  3 19:27:07 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 3 Mar 2015 19:27:07 +0100
Subject: [pypy-dev] GSoC 15 vectorizing traces
In-Reply-To: <54F5D41B.8070108@pasra.at>
References: <54F5D41B.8070108@pasra.at>
Message-ID: <CAMSv6X2oCtCpKGmPHkOnwAhL9dS86fm=RzTQhUF=0kmtWHnUaw@mail.gmail.com>

Hi Richard,

In the past few years, we have found GSoC to be tricky to handle.  As
a result of this, we're likely to have a high bar for student
acceptance this year.  The main criteria will be whether you have
already contributed to PyPy in a significant way.  If you only come up
with a GSoC proposal, no matter how cool, it will likely be rejected.

The first step is to get involved with the community, which means showing
up on irc (#pypy on irc.freenode.net).  We can give you pointers to
possible ideas there, or you can find your own, or you can look at
http://bugs.pypy.org/.

This is what we say in general to people that want to contribute to
PyPy.  We can move on to discuss GSoC only after we have seen concrete
results from this first step.


Armin Rigo

From liik.joonas at gmail.com  Wed Mar  4 09:58:35 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Wed, 4 Mar 2015 10:58:35 +0200
Subject: [pypy-dev] trsnslating pypy to another language besides C
Message-ID: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>

Hey

If i wanted to translate PyPy in to another language..

Where would i start?
What would be the absolute minimum i had to do to get it working?

I cant run any C code.. at all  (obscure sandboxed platform that only runs
an obscure scripting language none of you has probably ever heard of) how
much of a problem will this be? how to mitigate?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/a4c4777f/attachment.html>

From rymg19 at gmail.com  Wed Mar  4 19:15:27 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 4 Mar 2015 12:15:27 -0600
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
Message-ID: <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>

I'd kind of like to know this, too. I've wanted to try to get PyPy to
translate to C++. I doubt it would be of any advantage, but I'm just too
curious...

On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com> wrote:

> Hey
>
> If i wanted to translate PyPy in to another language..
>
> Where would i start?
> What would be the absolute minimum i had to do to get it working?
>
> I cant run any C code.. at all  (obscure sandboxed platform that only runs
> an obscure scripting language none of you has probably ever heard of) how
> much of a problem will this be? how to mitigate?
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/de73ca99/attachment.html>

From fijall at gmail.com  Wed Mar  4 19:23:12 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Mar 2015 20:23:12 +0200
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
Message-ID: <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>

C is C++ right? or what precisely did you want to do?

On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> I'd kind of like to know this, too. I've wanted to try to get PyPy to
> translate to C++. I doubt it would be of any advantage, but I'm just too
> curious...
>
> On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com> wrote:
>>
>> Hey
>>
>> If i wanted to translate PyPy in to another language..
>>
>> Where would i start?
>> What would be the absolute minimum i had to do to get it working?
>>
>> I cant run any C code.. at all  (obscure sandboxed platform that only runs
>> an obscure scripting language none of you has probably ever heard of) how
>> much of a problem will this be? how to mitigate?
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> Ryan
> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
> nul-terminated."
> Personal reality distortion fields are immune to contradictory evidence. -
> srean
> Check out my website: http://kirbyfan64.github.io/
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From fijall at gmail.com  Wed Mar  4 19:24:14 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Mar 2015 20:24:14 +0200
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
Message-ID: <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>

you need to write a backend. Backends are in translator/xxx.
Historically we used to support C, LLVM (to varying degrees), JS, C#
(or it's bytecode). What are you targeting? is it low-level enough to
make sense? What about the JIT? can you emit assembler and run it?

On Wed, Mar 4, 2015 at 10:58 AM, Joonas Liik <liik.joonas at gmail.com> wrote:
> Hey
>
> If i wanted to translate PyPy in to another language..
>
> Where would i start?
> What would be the absolute minimum i had to do to get it working?
>
> I cant run any C code.. at all  (obscure sandboxed platform that only runs
> an obscure scripting language none of you has probably ever heard of) how
> much of a problem will this be? how to mitigate?
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From rymg19 at gmail.com  Wed Mar  4 19:48:32 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 4 Mar 2015 12:48:32 -0600
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
Message-ID: <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>

Not necessarily. I figured that a C++ target might look a tad nicer because
it has built-in objects and exception handling.

Of course, again, it'd likely have its own set of painful holes and would
be largely useless. But I still want to try it. ;)

On Wed, Mar 4, 2015 at 12:23 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> C is C++ right? or what precisely did you want to do?
>
> On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> > I'd kind of like to know this, too. I've wanted to try to get PyPy to
> > translate to C++. I doubt it would be of any advantage, but I'm just too
> > curious...
> >
> > On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com>
> wrote:
> >>
> >> Hey
> >>
> >> If i wanted to translate PyPy in to another language..
> >>
> >> Where would i start?
> >> What would be the absolute minimum i had to do to get it working?
> >>
> >> I cant run any C code.. at all  (obscure sandboxed platform that only
> runs
> >> an obscure scripting language none of you has probably ever heard of)
> how
> >> much of a problem will this be? how to mitigate?
> >>
> >>
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >
> >
> >
> > --
> > Ryan
> > If anybody ever asks me why I prefer C++ to C, my answer will be simple:
> > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
> > nul-terminated."
> > Personal reality distortion fields are immune to contradictory evidence.
> -
> > srean
> > Check out my website: http://kirbyfan64.github.io/
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/8ff42672/attachment.html>

From liik.joonas at gmail.com  Wed Mar  4 21:29:26 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Wed, 4 Mar 2015 22:29:26 +0200
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
Message-ID: <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>

the host language is called JASS2*

JASS2 is the underlying scripting language for game basically,(Blizzards
Warcraft 3 from 2002/2003 depending on where you're from)
It is
    -strongly typed,
    -not very fast,
    -supports a grand total of 0 bit-wise operations
    -it has obscure and arcane ways.. / limitations that need to be worked
around*
    -it has a type declaration, C-like.. a bit (type A extends B)
    -it has no type-casting facilities at all.. (except for some number
types and strings)**,
    -oh and it the end product is larger then 4 Megabytes.. it will have
significantly reduced usefulness(thx blizzard, you have awesome ideas..)
    -FileIO is.. well it IS possible, surprisingly but it is so haxy you
really wouldn't want to rely on it too much.. (oh.. u can write ur crap all
overs peoples hard drives
            but if you want to read it back you need to set a registry
flag.. just .. charming..)
    -oh.. if your snippet of code runs for more than 30 000 bytecode ops
(internally there's a bytecode interpreter, we can't access it in any
meaningful
            way but some people have hacked away enough to prove it is
there)***
    - numeric types include "integer" == int32 and "real" == float32
(except some funny NaN behaviour i've heard about but not verified)
    - it is multithreaded... except it isn't. u can start many threads
(using ExecuteFunc, TriggerExecute or.. what was that 3rd one..) but
             they will run sequentially, this is useful for circumventing
the OP limit.
    - it has arrays****
    - the arrays are sized (read: every array is exactly 8192 instances of
which the last one you cant
             really use because it is buggy (an off by 1 some place i
guess...))
    - classes? who needs em?! not us, no classes for you :P (there's a few
custom dialects that mend this and usually
             do a just good enough solution to be used by everyone and
loved by none)
    - It has a strange standard library that mainly consists of functions
that swap its arguments and pass it to an underlying native..
             or do a nested loop just to set a value in a table.. it is
absolutely awesome! trembling with excitement ;)

    - it doesn't seem to do any bounds checking ... anywhere (u can tell it
to create a unit in a strange location and the whole game crashes)
    - null is 0 .. almost. well it is when it matters and bites you in the
ass.
    - no introspection (the closest one can come is ExecuteFunc which
actually takes a functions name to execute...(sic!)
              but it is useless unless the function takes no parameters and
returns none)
    - I'm sure i missed a ton of other /great/ features ;)


i can't emit assembler. (well i kind of CAN write files but they will emit
so much extra garbage that there's very low (0) chance it will really be
executable.. well except that the way to read files is basically to execute
them("Preload"))



So to sum it up.. if i can even compile RPython and have it not implode i
would be rather happy.

* a simpler compile target might be Galaxy Script (the scripting language
for Starcraft 2) tho i probably wouldn't be motivated enough to finish this
without the first one.
** theres about 20 built in types.. that are mostly ints in reality.. but
you cant use that.
*** this can be circumvented in a number of ways but at some cost ..  its a
real pain tho if u want to calculate something expensive.
**** of all types except "code" (meaning function) this can be circumvented
by wrapping it in a boolexpr



On 4 March 2015 at 20:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> Not necessarily. I figured that a C++ target might look a tad nicer
> because it has built-in objects and exception handling.
>
> Of course, again, it'd likely have its own set of painful holes and would
> be largely useless. But I still want to try it. ;)
>
> On Wed, Mar 4, 2015 at 12:23 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>
>> C is C++ right? or what precisely did you want to do?
>>
>> On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>> > I'd kind of like to know this, too. I've wanted to try to get PyPy to
>> > translate to C++. I doubt it would be of any advantage, but I'm just too
>> > curious...
>> >
>> > On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com>
>> wrote:
>> >>
>> >> Hey
>> >>
>> >> If i wanted to translate PyPy in to another language..
>> >>
>> >> Where would i start?
>> >> What would be the absolute minimum i had to do to get it working?
>> >>
>> >> I cant run any C code.. at all  (obscure sandboxed platform that only
>> runs
>> >> an obscure scripting language none of you has probably ever heard of)
>> how
>> >> much of a problem will this be? how to mitigate?
>> >>
>> >>
>> >> _______________________________________________
>> >> pypy-dev mailing list
>> >> pypy-dev at python.org
>> >> https://mail.python.org/mailman/listinfo/pypy-dev
>> >>
>> >
>> >
>> >
>> > --
>> > Ryan
>> > If anybody ever asks me why I prefer C++ to C, my answer will be simple:
>> > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
>> > nul-terminated."
>> > Personal reality distortion fields are immune to contradictory
>> evidence. -
>> > srean
>> > Check out my website: http://kirbyfan64.github.io/
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >
>>
>
>
>
> --
> Ryan
> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
> nul-terminated."
> Personal reality distortion fields are immune to contradictory evidence. -
> srean
> Check out my website: http://kirbyfan64.github.io/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/4eeb3502/attachment-0001.html>

From liik.joonas at gmail.com  Wed Mar  4 21:58:35 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Wed, 4 Mar 2015 22:58:35 +0200
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
 <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
Message-ID: <CAB1GNpS0osPw4wH7jPZnidjCwtdmFPD2AgKakiUwkAKe6EETCQ@mail.gmail.com>

Oh yeah, forgot to mention
there are hashtables (in JASS2) that do have infinite storage so .. not
quite hopeless.. i hope. :)


On 4 March 2015 at 22:29, Joonas Liik <liik.joonas at gmail.com> wrote:

> the host language is called JASS2*
>
> JASS2 is the underlying scripting language for game basically,(Blizzards
> Warcraft 3 from 2002/2003 depending on where you're from)
> It is
>     -strongly typed,
>     -not very fast,
>     -supports a grand total of 0 bit-wise operations
>     -it has obscure and arcane ways.. / limitations that need to be worked
> around*
>     -it has a type declaration, C-like.. a bit (type A extends B)
>     -it has no type-casting facilities at all.. (except for some number
> types and strings)**,
>     -oh and it the end product is larger then 4 Megabytes.. it will have
> significantly reduced usefulness(thx blizzard, you have awesome ideas..)
>     -FileIO is.. well it IS possible, surprisingly but it is so haxy you
> really wouldn't want to rely on it too much.. (oh.. u can write ur crap all
> overs peoples hard drives
>             but if you want to read it back you need to set a registry
> flag.. just .. charming..)
>     -oh.. if your snippet of code runs for more than 30 000 bytecode ops
> (internally there's a bytecode interpreter, we can't access it in any
> meaningful
>             way but some people have hacked away enough to prove it is
> there)***
>     - numeric types include "integer" == int32 and "real" == float32
> (except some funny NaN behaviour i've heard about but not verified)
>     - it is multithreaded... except it isn't. u can start many threads
> (using ExecuteFunc, TriggerExecute or.. what was that 3rd one..) but
>              they will run sequentially, this is useful for circumventing
> the OP limit.
>     - it has arrays****
>     - the arrays are sized (read: every array is exactly 8192 instances of
> which the last one you cant
>              really use because it is buggy (an off by 1 some place i
> guess...))
>     - classes? who needs em?! not us, no classes for you :P (there's a few
> custom dialects that mend this and usually
>              do a just good enough solution to be used by everyone and
> loved by none)
>     - It has a strange standard library that mainly consists of functions
> that swap its arguments and pass it to an underlying native..
>              or do a nested loop just to set a value in a table.. it is
> absolutely awesome! trembling with excitement ;)
>
>     - it doesn't seem to do any bounds checking ... anywhere (u can tell
> it to create a unit in a strange location and the whole game crashes)
>     - null is 0 .. almost. well it is when it matters and bites you in the
> ass.
>     - no introspection (the closest one can come is ExecuteFunc which
> actually takes a functions name to execute...(sic!)
>               but it is useless unless the function takes no parameters
> and returns none)
>     - I'm sure i missed a ton of other /great/ features ;)
>
>
> i can't emit assembler. (well i kind of CAN write files but they will emit
> so much extra garbage that there's very low (0) chance it will really be
> executable.. well except that the way to read files is basically to execute
> them("Preload"))
>
>
>
> So to sum it up.. if i can even compile RPython and have it not implode i
> would be rather happy.
>
> * a simpler compile target might be Galaxy Script (the scripting language
> for Starcraft 2) tho i probably wouldn't be motivated enough to finish this
> without the first one.
> ** theres about 20 built in types.. that are mostly ints in reality.. but
> you cant use that.
> *** this can be circumvented in a number of ways but at some cost ..  its
> a real pain tho if u want to calculate something expensive.
> **** of all types except "code" (meaning function) this can be
> circumvented by wrapping it in a boolexpr
>
>
>
> On 4 March 2015 at 20:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> Not necessarily. I figured that a C++ target might look a tad nicer
>> because it has built-in objects and exception handling.
>>
>> Of course, again, it'd likely have its own set of painful holes and would
>> be largely useless. But I still want to try it. ;)
>>
>> On Wed, Mar 4, 2015 at 12:23 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>>
>>> C is C++ right? or what precisely did you want to do?
>>>
>>> On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>> > I'd kind of like to know this, too. I've wanted to try to get PyPy to
>>> > translate to C++. I doubt it would be of any advantage, but I'm just
>>> too
>>> > curious...
>>> >
>>> > On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com>
>>> wrote:
>>> >>
>>> >> Hey
>>> >>
>>> >> If i wanted to translate PyPy in to another language..
>>> >>
>>> >> Where would i start?
>>> >> What would be the absolute minimum i had to do to get it working?
>>> >>
>>> >> I cant run any C code.. at all  (obscure sandboxed platform that only
>>> runs
>>> >> an obscure scripting language none of you has probably ever heard of)
>>> how
>>> >> much of a problem will this be? how to mitigate?
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> pypy-dev mailing list
>>> >> pypy-dev at python.org
>>> >> https://mail.python.org/mailman/listinfo/pypy-dev
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Ryan
>>> > If anybody ever asks me why I prefer C++ to C, my answer will be
>>> simple:
>>> > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that
>>> was
>>> > nul-terminated."
>>> > Personal reality distortion fields are immune to contradictory
>>> evidence. -
>>> > srean
>>> > Check out my website: http://kirbyfan64.github.io/
>>> >
>>> > _______________________________________________
>>> > pypy-dev mailing list
>>> > pypy-dev at python.org
>>> > https://mail.python.org/mailman/listinfo/pypy-dev
>>> >
>>>
>>
>>
>>
>> --
>> Ryan
>> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
>> nul-terminated."
>> Personal reality distortion fields are immune to contradictory evidence.
>> - srean
>> Check out my website: http://kirbyfan64.github.io/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/25bd826f/attachment.html>

From rymg19 at gmail.com  Wed Mar  4 22:41:33 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 4 Mar 2015 15:41:33 -0600
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
 <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
Message-ID: <CAO41-mO26kS_tMqEfozhhHq6Fs6Fa5LVVUBqExSOcikP0n7JnQ@mail.gmail.com>

I seriously doubt PyPy is going to work well at all under JASS2; it'll
probably be insanely slow.

This <http://www.wc3c.net/showthread.php?t=61352> would probably be better.
It's a Python-to-Jass converter.

On Wed, Mar 4, 2015 at 2:29 PM, Joonas Liik <liik.joonas at gmail.com> wrote:

> the host language is called JASS2*
>
> JASS2 is the underlying scripting language for game basically,(Blizzards
> Warcraft 3 from 2002/2003 depending on where you're from)
> It is
>     -strongly typed,
>     -not very fast,
>     -supports a grand total of 0 bit-wise operations
>     -it has obscure and arcane ways.. / limitations that need to be worked
> around*
>     -it has a type declaration, C-like.. a bit (type A extends B)
>     -it has no type-casting facilities at all.. (except for some number
> types and strings)**,
>     -oh and it the end product is larger then 4 Megabytes.. it will have
> significantly reduced usefulness(thx blizzard, you have awesome ideas..)
>     -FileIO is.. well it IS possible, surprisingly but it is so haxy you
> really wouldn't want to rely on it too much.. (oh.. u can write ur crap all
> overs peoples hard drives
>             but if you want to read it back you need to set a registry
> flag.. just .. charming..)
>     -oh.. if your snippet of code runs for more than 30 000 bytecode ops
> (internally there's a bytecode interpreter, we can't access it in any
> meaningful
>             way but some people have hacked away enough to prove it is
> there)***
>     - numeric types include "integer" == int32 and "real" == float32
> (except some funny NaN behaviour i've heard about but not verified)
>     - it is multithreaded... except it isn't. u can start many threads
> (using ExecuteFunc, TriggerExecute or.. what was that 3rd one..) but
>              they will run sequentially, this is useful for circumventing
> the OP limit.
>     - it has arrays****
>     - the arrays are sized (read: every array is exactly 8192 instances of
> which the last one you cant
>              really use because it is buggy (an off by 1 some place i
> guess...))
>     - classes? who needs em?! not us, no classes for you :P (there's a few
> custom dialects that mend this and usually
>              do a just good enough solution to be used by everyone and
> loved by none)
>     - It has a strange standard library that mainly consists of functions
> that swap its arguments and pass it to an underlying native..
>              or do a nested loop just to set a value in a table.. it is
> absolutely awesome! trembling with excitement ;)
>
>     - it doesn't seem to do any bounds checking ... anywhere (u can tell
> it to create a unit in a strange location and the whole game crashes)
>     - null is 0 .. almost. well it is when it matters and bites you in the
> ass.
>     - no introspection (the closest one can come is ExecuteFunc which
> actually takes a functions name to execute...(sic!)
>               but it is useless unless the function takes no parameters
> and returns none)
>     - I'm sure i missed a ton of other /great/ features ;)
>
>
> i can't emit assembler. (well i kind of CAN write files but they will emit
> so much extra garbage that there's very low (0) chance it will really be
> executable.. well except that the way to read files is basically to execute
> them("Preload"))
>
>
>
> So to sum it up.. if i can even compile RPython and have it not implode i
> would be rather happy.
>
> * a simpler compile target might be Galaxy Script (the scripting language
> for Starcraft 2) tho i probably wouldn't be motivated enough to finish this
> without the first one.
> ** theres about 20 built in types.. that are mostly ints in reality.. but
> you cant use that.
> *** this can be circumvented in a number of ways but at some cost ..  its
> a real pain tho if u want to calculate something expensive.
> **** of all types except "code" (meaning function) this can be
> circumvented by wrapping it in a boolexpr
>
>
>
> On 4 March 2015 at 20:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> Not necessarily. I figured that a C++ target might look a tad nicer
>> because it has built-in objects and exception handling.
>>
>> Of course, again, it'd likely have its own set of painful holes and would
>> be largely useless. But I still want to try it. ;)
>>
>> On Wed, Mar 4, 2015 at 12:23 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>>
>>> C is C++ right? or what precisely did you want to do?
>>>
>>> On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>> > I'd kind of like to know this, too. I've wanted to try to get PyPy to
>>> > translate to C++. I doubt it would be of any advantage, but I'm just
>>> too
>>> > curious...
>>> >
>>> > On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com>
>>> wrote:
>>> >>
>>> >> Hey
>>> >>
>>> >> If i wanted to translate PyPy in to another language..
>>> >>
>>> >> Where would i start?
>>> >> What would be the absolute minimum i had to do to get it working?
>>> >>
>>> >> I cant run any C code.. at all  (obscure sandboxed platform that only
>>> runs
>>> >> an obscure scripting language none of you has probably ever heard of)
>>> how
>>> >> much of a problem will this be? how to mitigate?
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> pypy-dev mailing list
>>> >> pypy-dev at python.org
>>> >> https://mail.python.org/mailman/listinfo/pypy-dev
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Ryan
>>> > If anybody ever asks me why I prefer C++ to C, my answer will be
>>> simple:
>>> > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that
>>> was
>>> > nul-terminated."
>>> > Personal reality distortion fields are immune to contradictory
>>> evidence. -
>>> > srean
>>> > Check out my website: http://kirbyfan64.github.io/
>>> >
>>> > _______________________________________________
>>> > pypy-dev mailing list
>>> > pypy-dev at python.org
>>> > https://mail.python.org/mailman/listinfo/pypy-dev
>>> >
>>>
>>
>>
>>
>> --
>> Ryan
>> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
>> nul-terminated."
>> Personal reality distortion fields are immune to contradictory evidence.
>> - srean
>> Check out my website: http://kirbyfan64.github.io/
>>
>
>


-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/74087909/attachment-0001.html>

From rymg19 at gmail.com  Wed Mar  4 22:42:24 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 4 Mar 2015 15:42:24 -0600
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAO41-mO26kS_tMqEfozhhHq6Fs6Fa5LVVUBqExSOcikP0n7JnQ@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
 <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
 <CAO41-mO26kS_tMqEfozhhHq6Fs6Fa5LVVUBqExSOcikP0n7JnQ@mail.gmail.com>
Message-ID: <CAO41-mNUzkDgfyPSPe7fZdeRk+K7rPcgLxAZ=zfcbNOhnekEbw@mail.gmail.com>

Nevermind. The link seems dead.

On Wed, Mar 4, 2015 at 3:41 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> I seriously doubt PyPy is going to work well at all under JASS2; it'll
> probably be insanely slow.
>
> This <http://www.wc3c.net/showthread.php?t=61352> would probably be
> better. It's a Python-to-Jass converter.
>
> On Wed, Mar 4, 2015 at 2:29 PM, Joonas Liik <liik.joonas at gmail.com> wrote:
>
>> the host language is called JASS2*
>>
>> JASS2 is the underlying scripting language for game basically,(Blizzards
>> Warcraft 3 from 2002/2003 depending on where you're from)
>> It is
>>     -strongly typed,
>>     -not very fast,
>>     -supports a grand total of 0 bit-wise operations
>>     -it has obscure and arcane ways.. / limitations that need to be
>> worked around*
>>     -it has a type declaration, C-like.. a bit (type A extends B)
>>     -it has no type-casting facilities at all.. (except for some number
>> types and strings)**,
>>     -oh and it the end product is larger then 4 Megabytes.. it will have
>> significantly reduced usefulness(thx blizzard, you have awesome ideas..)
>>     -FileIO is.. well it IS possible, surprisingly but it is so haxy you
>> really wouldn't want to rely on it too much.. (oh.. u can write ur crap all
>> overs peoples hard drives
>>             but if you want to read it back you need to set a registry
>> flag.. just .. charming..)
>>     -oh.. if your snippet of code runs for more than 30 000 bytecode ops
>> (internally there's a bytecode interpreter, we can't access it in any
>> meaningful
>>             way but some people have hacked away enough to prove it is
>> there)***
>>     - numeric types include "integer" == int32 and "real" == float32
>> (except some funny NaN behaviour i've heard about but not verified)
>>     - it is multithreaded... except it isn't. u can start many threads
>> (using ExecuteFunc, TriggerExecute or.. what was that 3rd one..) but
>>              they will run sequentially, this is useful for circumventing
>> the OP limit.
>>     - it has arrays****
>>     - the arrays are sized (read: every array is exactly 8192 instances
>> of which the last one you cant
>>              really use because it is buggy (an off by 1 some place i
>> guess...))
>>     - classes? who needs em?! not us, no classes for you :P (there's a
>> few custom dialects that mend this and usually
>>              do a just good enough solution to be used by everyone and
>> loved by none)
>>     - It has a strange standard library that mainly consists of functions
>> that swap its arguments and pass it to an underlying native..
>>              or do a nested loop just to set a value in a table.. it is
>> absolutely awesome! trembling with excitement ;)
>>
>>     - it doesn't seem to do any bounds checking ... anywhere (u can tell
>> it to create a unit in a strange location and the whole game crashes)
>>     - null is 0 .. almost. well it is when it matters and bites you in
>> the ass.
>>     - no introspection (the closest one can come is ExecuteFunc which
>> actually takes a functions name to execute...(sic!)
>>               but it is useless unless the function takes no parameters
>> and returns none)
>>     - I'm sure i missed a ton of other /great/ features ;)
>>
>>
>> i can't emit assembler. (well i kind of CAN write files but they will
>> emit so much extra garbage that there's very low (0) chance it will really
>> be executable.. well except that the way to read files is basically to
>> execute them("Preload"))
>>
>>
>>
>> So to sum it up.. if i can even compile RPython and have it not implode i
>> would be rather happy.
>>
>> * a simpler compile target might be Galaxy Script (the scripting language
>> for Starcraft 2) tho i probably wouldn't be motivated enough to finish this
>> without the first one.
>> ** theres about 20 built in types.. that are mostly ints in reality.. but
>> you cant use that.
>> *** this can be circumvented in a number of ways but at some cost ..  its
>> a real pain tho if u want to calculate something expensive.
>> **** of all types except "code" (meaning function) this can be
>> circumvented by wrapping it in a boolexpr
>>
>>
>>
>> On 4 March 2015 at 20:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>>> Not necessarily. I figured that a C++ target might look a tad nicer
>>> because it has built-in objects and exception handling.
>>>
>>> Of course, again, it'd likely have its own set of painful holes and
>>> would be largely useless. But I still want to try it. ;)
>>>
>>> On Wed, Mar 4, 2015 at 12:23 PM, Maciej Fijalkowski <fijall at gmail.com>
>>> wrote:
>>>
>>>> C is C++ right? or what precisely did you want to do?
>>>>
>>>> On Wed, Mar 4, 2015 at 8:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>>> > I'd kind of like to know this, too. I've wanted to try to get PyPy to
>>>> > translate to C++. I doubt it would be of any advantage, but I'm just
>>>> too
>>>> > curious...
>>>> >
>>>> > On Wed, Mar 4, 2015 at 2:58 AM, Joonas Liik <liik.joonas at gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Hey
>>>> >>
>>>> >> If i wanted to translate PyPy in to another language..
>>>> >>
>>>> >> Where would i start?
>>>> >> What would be the absolute minimum i had to do to get it working?
>>>> >>
>>>> >> I cant run any C code.. at all  (obscure sandboxed platform that
>>>> only runs
>>>> >> an obscure scripting language none of you has probably ever heard
>>>> of) how
>>>> >> much of a problem will this be? how to mitigate?
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> pypy-dev mailing list
>>>> >> pypy-dev at python.org
>>>> >> https://mail.python.org/mailman/listinfo/pypy-dev
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Ryan
>>>> > If anybody ever asks me why I prefer C++ to C, my answer will be
>>>> simple:
>>>> > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that
>>>> was
>>>> > nul-terminated."
>>>> > Personal reality distortion fields are immune to contradictory
>>>> evidence. -
>>>> > srean
>>>> > Check out my website: http://kirbyfan64.github.io/
>>>> >
>>>> > _______________________________________________
>>>> > pypy-dev mailing list
>>>> > pypy-dev at python.org
>>>> > https://mail.python.org/mailman/listinfo/pypy-dev
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Ryan
>>> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
>>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
>>> nul-terminated."
>>> Personal reality distortion fields are immune to contradictory evidence.
>>> - srean
>>> Check out my website: http://kirbyfan64.github.io/
>>>
>>
>>
>
>
> --
> Ryan
> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
> nul-terminated."
> Personal reality distortion fields are immune to contradictory evidence. -
> srean
> Check out my website: http://kirbyfan64.github.io/
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150304/50ccb913/attachment.html>

From arigo at tunes.org  Thu Mar  5 05:51:32 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 5 Mar 2015 05:51:32 +0100
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAO41-mNUzkDgfyPSPe7fZdeRk+K7rPcgLxAZ=zfcbNOhnekEbw@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
 <CAB1GNpQkpzG4DhR-qFMR8QoBX1eq9j1fM6ciuLoU=ttih64uLA@mail.gmail.com>
 <CAO41-mO26kS_tMqEfozhhHq6Fs6Fa5LVVUBqExSOcikP0n7JnQ@mail.gmail.com>
 <CAO41-mNUzkDgfyPSPe7fZdeRk+K7rPcgLxAZ=zfcbNOhnekEbw@mail.gmail.com>
Message-ID: <CAMSv6X2pPcg3emAC_DOkbhn=XOEmSet-yXGHCgqSJHNC7a3gLQ@mail.gmail.com>

Hi Joonas,

To make sense out of it, you would need an "ootype" backend, as
opposed to a "lltype" backend which generates C-like code.  Google for
Antonio's thesis "High performance implementation of Python for
CLI/.NET with JIT compiler generation for dynamic languages".  But we
killed support for ootype backends soon afterward, because (despite
the thesis) it was not really going anywhere in practice.  Nowadays
PyPy is only focusing on its C backend.  If you really want you can
dig in the Mercurial history.  Be warned that a lot of work is needed
in any case...


A bient?t,

Armin.

From arigo at tunes.org  Thu Mar  5 06:02:22 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 5 Mar 2015 06:02:22 +0100
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
Message-ID: <CAMSv6X1cP5fPporsOZpSiDtDGTZ3BGv5wozOVL+7iS68ghvasg@mail.gmail.com>

Hi Ryan,

On 4 March 2015 at 19:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> Not necessarily. I figured that a C++ target might look a tad nicer because
> it has built-in objects and exception handling.

Yes, this might be true for the static backend.  However, the JIT
integration would be extremely painful.  To handle, let's say, the
exceptions you get from C++, you'd need to write custom assembler that
depends on the C++ compiler you used, full of non-standard binary data
like the "eh" sections produced by gcc.  Similarly, there is no
standard way at all (as far as I know) to learn how to build a new
object from scratch (like get its vtable pointer and know where it
must be stored).  You might start by learning how they manage to do
that in other VMs (like the various JVM), but my guess is that
although they are using C++ to write the VM, all Java-level objects
are implemented by controlling the exact layout of memory, not as C++
objects at all.


A bient?t,

Armin.

From rymg19 at gmail.com  Thu Mar  5 18:55:09 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 5 Mar 2015 11:55:09 -0600
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAMSv6X1cP5fPporsOZpSiDtDGTZ3BGv5wozOVL+7iS68ghvasg@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAO41-mM-j95kzuZbLLuznvsAxecePA-3JsvdNdP-mJtcF9dFVw@mail.gmail.com>
 <CAK5idxTbC839CMD39-qwiCtBs5D1k8x_5Fp9r03mY_NWuMNkWw@mail.gmail.com>
 <CAO41-mM=xJ9bAefX_ahFMj9Z0Q5t_=dLkwsBpqcWD4SEAbETDw@mail.gmail.com>
 <CAMSv6X1cP5fPporsOZpSiDtDGTZ3BGv5wozOVL+7iS68ghvasg@mail.gmail.com>
Message-ID: <CAO41-mPpJbifym3aGx6g6V1jrbBESCuLkCBummf9M8AkXoGJ1w@mail.gmail.com>

I didn't figure I'd even get that far...

It's mostly just for toying with the static backend, without the JIT.

On Wed, Mar 4, 2015 at 11:02 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Ryan,
>
> On 4 March 2015 at 19:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> > Not necessarily. I figured that a C++ target might look a tad nicer
> because
> > it has built-in objects and exception handling.
>
> Yes, this might be true for the static backend.  However, the JIT
> integration would be extremely painful.  To handle, let's say, the
> exceptions you get from C++, you'd need to write custom assembler that
> depends on the C++ compiler you used, full of non-standard binary data
> like the "eh" sections produced by gcc.  Similarly, there is no
> standard way at all (as far as I know) to learn how to build a new
> object from scratch (like get its vtable pointer and know where it
> must be stored).  You might start by learning how they manage to do
> that in other VMs (like the various JVM), but my guess is that
> although they are using C++ to write the VM, all Java-level objects
> are implemented by controlling the exact layout of memory, not as C++
> objects at all.
>
>
> A bient?t,
>
> Armin.
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150305/b16d64ff/attachment.html>

From techtonik at gmail.com  Fri Mar  6 13:24:58 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 6 Mar 2015 15:24:58 +0300
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
Message-ID: <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>

On Wed, Mar 4, 2015 at 9:24 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> you need to write a backend. Backends are in translator/xxx.

There are multiple dirs in rpython/translator:

  backendopt
  c
  goal
  platform
  sandbox
  test
  tool

Which of those are backends?

-- 
anatoly t.

From amauryfa at gmail.com  Fri Mar  6 14:48:21 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 6 Mar 2015 14:48:21 +0100
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
 <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>
Message-ID: <CAGmFidZC4QStQjMnDKJBAZ+e+qRqdgDbSkR4ED-hASxhSYQmYg@mail.gmail.com>

2015-03-06 13:24 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:

> On Wed, Mar 4, 2015 at 9:24 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> > you need to write a backend. Backends are in translator/xxx.
>
> There are multiple dirs in rpython/translator:
>
>   backendopt
>   c
>   goal
>   platform
>   sandbox
>   test
>   tool
>
> Which of those are backends?
>

Only one: c

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150306/61856316/attachment.html>

From techtonik at gmail.com  Fri Mar  6 15:23:49 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 6 Mar 2015 17:23:49 +0300
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAGmFidZC4QStQjMnDKJBAZ+e+qRqdgDbSkR4ED-hASxhSYQmYg@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
 <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>
 <CAGmFidZC4QStQjMnDKJBAZ+e+qRqdgDbSkR4ED-hASxhSYQmYg@mail.gmail.com>
Message-ID: <CAPkN8x+He5k-XPFRL3vyuxkqHNder6BvzyBJn70AK51zfmPcyQ@mail.gmail.com>

On Fri, Mar 6, 2015 at 4:48 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> 2015-03-06 13:24 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>
>> On Wed, Mar 4, 2015 at 9:24 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>> > you need to write a backend. Backends are in translator/xxx.
>>
>> There are multiple dirs in rpython/translator:
>>
>>   backendopt
>>   c
>>   goal
>>   platform
>>   sandbox
>>   test
>>   tool
>>
>> Which of those are backends?
>
>
> Only one: c

I c. =) Can I propose to rename the directory to back-c/ ? Kind of
compromise between brevity and clarity.
-- 
anatoly t.

From vm at klankschap.nl  Sat Mar  7 12:49:23 2015
From: vm at klankschap.nl (Floris van Manen)
Date: Sat, 7 Mar 2015 12:49:23 +0100
Subject: [pypy-dev] random numbers
Message-ID: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>

Are the random number generators from pypy and the standard python generating the exact same numbers from identical seeds?

.F




From cfbolz at gmx.de  Sat Mar  7 13:21:50 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sat, 07 Mar 2015 13:21:50 +0100
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <CAPkN8x+He5k-XPFRL3vyuxkqHNder6BvzyBJn70AK51zfmPcyQ@mail.gmail.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
 <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>
 <CAGmFidZC4QStQjMnDKJBAZ+e+qRqdgDbSkR4ED-hASxhSYQmYg@mail.gmail.com>
 <CAPkN8x+He5k-XPFRL3vyuxkqHNder6BvzyBJn70AK51zfmPcyQ@mail.gmail.com>
Message-ID: <f6728fba-7bc2-4d80-8a75-214af118cb6b@email.android.com>

That makes no sense, one cannot import packages with '-' in them. 

Carl Friedrich 

On March 6, 2015 3:23:49 PM GMT+01:00, anatoly techtonik <techtonik at gmail.com> wrote:
>On Fri, Mar 6, 2015 at 4:48 PM, Amaury Forgeot d'Arc
><amauryfa at gmail.com> wrote:
>> 2015-03-06 13:24 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>>
>>> On Wed, Mar 4, 2015 at 9:24 PM, Maciej Fijalkowski
><fijall at gmail.com>
>>> wrote:
>>> > you need to write a backend. Backends are in translator/xxx.
>>>
>>> There are multiple dirs in rpython/translator:
>>>
>>>   backendopt
>>>   c
>>>   goal
>>>   platform
>>>   sandbox
>>>   test
>>>   tool
>>>
>>> Which of those are backends?
>>
>>
>> Only one: c
>
>I c. =) Can I propose to rename the directory to back-c/ ? Kind of
>compromise between brevity and clarity.
>-- 
>anatoly t.
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150307/f10593b4/attachment.html>

From matti.picus at gmail.com  Sat Mar  7 13:28:52 2015
From: matti.picus at gmail.com (matti picus)
Date: Sat, 7 Mar 2015 14:28:52 +0200
Subject: [pypy-dev] random numbers
In-Reply-To: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
Message-ID: <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>

What did you try that led you to suspect they are different?
Matti
On Mar 7, 2015 2:18 PM, "Floris van Manen" <vm at klankschap.nl> wrote:

> Are the random number generators from pypy and the standard python
> generating the exact same numbers from identical seeds?
>
> .F
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150307/8529c3a5/attachment.html>

From matti.picus at gmail.com  Sat Mar  7 19:37:32 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 07 Mar 2015 20:37:32 +0200
Subject: [pypy-dev] random numbers
In-Reply-To: <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
Message-ID: <54FB456C.1020000@gmail.com>

Floris is seems you sent the response to only me.
The "right" behaviour is to not use jumpahead, which does not work well 
enough to be part of python 3.1 (note the use of "likely" in the docstring):
'''
jumpahead(self, n) method of random.Random instance
     Change the internal state to one that is likely far away
     from the current state.  This method will not be in Py3.x,
     so it is better to simply reseed.
'''

The implementation changed for python 2.4, in changeset
https://hg.python.org/cpython/rev/edd056c4a40d/
as part of the adaptations to the Mersenne Twister generator that 
replaced WichmannHill in python 2.3
I will try to make pypy return the same values as cpython, but you 
should consider reseeding for each thread instead
Matti

On 07/03/15 14:33, Floris van Manen wrote:
>
>> On 7 Mar 2015, at 13:28, matti picus <matti.picus at gmail.com 
>> <mailto:matti.picus at gmail.com>> wrote:
>>
>> What did you try that led you to suspect they are different?
>> Matti
>
> yes indeed.
> Using the jumpahead() fnction will lead to different results.
> I don't now which are the correct ones...
>
>
> import random
>
> random.seed(12345)
> for i in xrange(10):
>     print random.random()
> print
> randomState = random.getstate()
> for i in xrange(10):
>     random.setstate(randomState)
>     random.jumpahead(1)
>     randomState = random.getstate()
>     print random.random()
>
>
> '''
> standard python:
>
> 0.416619872545
> 0.0101691694571
> 0.825206509254
> 0.2986398552
> 0.368411689488
> 0.193661349045
> 0.566008168729
> 0.161687823929
> 0.124266884284
> 0.43293626801
>
> jumpahead()
> 0.94861718596
> 0.466389725591
> 0.440309106388
> 0.756595610966
> 0.329954203719
> 0.207243180479
> 0.982769966933
> 0.856141036147
> 0.746277878413
> 0.304000958341
>
>
> pypy
>
> 0.416619872545
> 0.0101691694571
> 0.825206509254
> 0.2986398552
> 0.368411689488
> 0.193661349045
> 0.566008168729
> 0.161687823929
> 0.124266884284
> 0.43293626801
>
> jumpahead()
> 0.41362858709
> 0.482594587067
> 0.363386428038
> 0.0122210322735
> 0.218894401377
> 0.786381037234
> 0.535611315209
> 0.545681381301
> 0.0153701629836
> 0.136216130724
> '''
>


From vm at klankschap.nl  Sat Mar  7 21:06:13 2015
From: vm at klankschap.nl (Floris van Manen)
Date: Sat, 7 Mar 2015 21:06:13 +0100
Subject: [pypy-dev] random numbers
In-Reply-To: <54FB456C.1020000@gmail.com>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
 <54FB456C.1020000@gmail.com>
Message-ID: <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>

Matti,

> On 7 Mar 2015, at 19:37, Matti Picus <matti.picus at gmail.com> wrote:
> 
> '''
> jumpahead(self, n) method of random.Random instance
>    Change the internal state to one that is likely far away
>    from the current state.  This method will not be in Py3.x,
>    so it is better to simply reseed.
> '''

Thanks for pointing out.

I'm using the jumpahead() to have multiple streams in parallel, derived from a single point.

def jumpRandomState(self):
    self.random.setstate(self.randomState)
    self.random.jumpahead(1)
    self.randomState = self.random.getstate()


Would this be a good alternative?

def jumpRandomState(self):
    self.random.setstate(self.randomState)
    self.random.seed(self.random.random())
    self.randomState = self.random.getstate()


.Floris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150307/c2346d07/attachment.html>

From matti.picus at gmail.com  Sat Mar  7 22:13:55 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 07 Mar 2015 23:13:55 +0200
Subject: [pypy-dev] random numbers
In-Reply-To: <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
 <54FB456C.1020000@gmail.com>
 <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
Message-ID: <54FB6A13.40704@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150307/be52688e/attachment-0001.html>

From steve at pearwood.info  Sun Mar  8 02:24:16 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 8 Mar 2015 12:24:16 +1100
Subject: [pypy-dev] random numbers
In-Reply-To: <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
 <54FB456C.1020000@gmail.com>
 <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
Message-ID: <20150308012415.GQ7655@ando.pearwood.info>

On Sat, Mar 07, 2015 at 09:06:13PM +0100, Floris van Manen wrote:

> I'm using the jumpahead() to have multiple streams in parallel, 
> derived from a single point.

I'm afraid I don't understand what that sentence means. Do you mean 
derived from a single seed?

If so, I think the easiest way is to just create multiple random 
instances. Suppose you want five streams of random numbers: create one 
extra "seeder" (or just use the pre-defined top-level random functions):

import random
seeder = random.Random(myseed)  # or just use random.seed(myseed)
streams = [random.Random(seeder.random()) for _ in range(5)]

If you want to reset them to the initial state, just 
reseed the seeder stream and recreate them. Or you can just grab all 
their internal states:

states = [s.getstate() for s in streams]


> def jumpRandomState(self):
>     self.random.setstate(self.randomState)
>     self.random.jumpahead(1)
>     self.randomState = self.random.getstate()

I'm afraid I don't understand the purpose of that method or why you are 
restoring the internal state before jumping. I tried putting it in a 
subclass of random.Random, but it raises an exception:


py> class MyRandom(random.Random):
...     def jumpRandomState(self):
...             self.random.setstate(self.randomState)
...             self.random.jumpahead(1)
...             self.randomState = self.random.getstate()
...
py> rnd = MyRandom(1000)
py> rnd.jumpRandomState()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in jumpRandomState
AttributeError: 'builtin_function_or_method' object has no attribute 
'setstate'

so your code doesn't work for me.

Based just on the name, I would do this:

class MyRandom(random.Random):
    def jumpRandomState(self):
        self.jumpahead(356789)


If you don't trust jumpahead to reliably jump to an independent part 
of the PRNG sequence, re-seed with an independent pseudo-random number:

class MyRandom(random.Random):
    _JUMPER = random.Random()
    def jumpRandomState(self):
        self.seed(self._JUMPER.random())


-- 
Steve

From vm at klankschap.nl  Sun Mar  8 00:02:28 2015
From: vm at klankschap.nl (Floris van Manen)
Date: Sun, 8 Mar 2015 00:02:28 +0100
Subject: [pypy-dev] random numbers
In-Reply-To: <54FB6A13.40704@gmail.com>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
 <54FB456C.1020000@gmail.com>
 <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
 <54FB6A13.40704@gmail.com>
Message-ID: <C0D83DC4-34D9-453A-B8B1-7D1A40F46AF3@klankschap.nl>


> On 7 Mar 2015, at 22:13, Matti Picus <matti.picus at gmail.com> wrote:
> 
> Looks OK to me, maybe someone else will chime in as well. 
> In any case jumpahead(n) should give identical results in both cpython and pypy if you use a pypy after changeset a2a352a3b535 (in other words, a nightly from tomorrow onwards)
> Matti

Thanks, i'll give it a try.

.F


From cfbolz at gmx.de  Sun Mar  8 11:20:29 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 08 Mar 2015 11:20:29 +0100
Subject: [pypy-dev] random numbers
In-Reply-To: <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
References: <9552842B-ED10-4887-96C1-62BA90AF0977@klankschap.nl>
 <CAO8bCgjOG=xyzXUP9bR7yacs9wghVRoc1xxj9+DC6y36Rd83oQ@mail.gmail.com>
 <89C40034-D128-4A50-8C05-8B221265C936@klankschap.nl>
 <54FB456C.1020000@gmail.com>
 <33A446F6-FF79-44AE-9008-CD6AA908B59B@klankschap.nl>
Message-ID: <3b7f4b1e-f689-47bb-84aa-f8ce6c40ff46@email.android.com>

I think for multiple streams it's best to make your own instances of random.Random. Most of the functions that the random module exposes are just methods of a single global random function, see the source here:

https://hg.python.org/cpython/file/2.7/Lib/random.py#l885

Cheers, 

Carl Friedrich 

On March 7, 2015 9:06:13 PM GMT+01:00, Floris van Manen <vm at klankschap.nl> wrote:
>Matti,
>
>> On 7 Mar 2015, at 19:37, Matti Picus <matti.picus at gmail.com> wrote:
>> 
>> '''
>> jumpahead(self, n) method of random.Random instance
>>    Change the internal state to one that is likely far away
>>    from the current state.  This method will not be in Py3.x,
>>    so it is better to simply reseed.
>> '''
>
>Thanks for pointing out.
>
>I'm using the jumpahead() to have multiple streams in parallel, derived
>from a single point.
>
>def jumpRandomState(self):
>    self.random.setstate(self.randomState)
>    self.random.jumpahead(1)
>    self.randomState = self.random.getstate()
>
>
>Would this be a good alternative?
>
>def jumpRandomState(self):
>    self.random.setstate(self.randomState)
>    self.random.seed(self.random.random())
>    self.randomState = self.random.getstate()
>
>
>.Floris
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150308/b89a3452/attachment.html>

From fijall at gmail.com  Mon Mar  9 09:21:03 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 9 Mar 2015 10:21:03 +0200
Subject: [pypy-dev] [pypy-commit] pypy vmprof: uh
In-Reply-To: <20150308170336.C0ECD1C11F1@cobra.cs.uni-duesseldorf.de>
References: <20150308170336.C0ECD1C11F1@cobra.cs.uni-duesseldorf.de>
Message-ID: <CAK5idxS0A2KqZG8=jzjX02E47hN590ZvVgUq+O_YD0YgqzM3FQ@mail.gmail.com>

uh, I really don't think this checkin is correct (it was meant to seek
to the end, not 2 bytes)

On Sun, Mar 8, 2015 at 7:03 PM, arigo <noreply at buildbot.pypy.org> wrote:
> Author: Armin Rigo <arigo at tunes.org>
> Branch: vmprof
> Changeset: r76280:4c0fa46303da
> Date: 2015-03-08 18:03 +0100
> http://bitbucket.org/pypy/pypy/changeset/4c0fa46303da/
>
> Log:    uh
>
> diff --git a/pypy/module/__pypy__/interp_magic.py b/pypy/module/__pypy__/interp_magic.py
> --- a/pypy/module/__pypy__/interp_magic.py
> +++ b/pypy/module/__pypy__/interp_magic.py
> @@ -141,4 +141,4 @@
>      try:
>          return space.wrap(ec.code_info_file.read())
>      finally:
> -        ec.code_info_file.seek(2, 0)
> +        ec.code_info_file.seek(0, 2)
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> https://mail.python.org/mailman/listinfo/pypy-commit

From techtonik at gmail.com  Mon Mar  9 12:56:03 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 9 Mar 2015 14:56:03 +0300
Subject: [pypy-dev] trsnslating pypy to another language besides C
In-Reply-To: <f6728fba-7bc2-4d80-8a75-214af118cb6b@email.android.com>
References: <CAB1GNpQsvFi2PSvbvAMiDHrMOSnXu9ukXDHHtpPFSU3axwTKbQ@mail.gmail.com>
 <CAK5idxRbirMw6GthW7rsOv=8KT-mBm1ukDZch1AXoZGfsNOnSA@mail.gmail.com>
 <CAPkN8x+88-g67m_GyCzHd8DK=3YVykaq21yayhhEuOxCt9AU3w@mail.gmail.com>
 <CAGmFidZC4QStQjMnDKJBAZ+e+qRqdgDbSkR4ED-hASxhSYQmYg@mail.gmail.com>
 <CAPkN8x+He5k-XPFRL3vyuxkqHNder6BvzyBJn70AK51zfmPcyQ@mail.gmail.com>
 <f6728fba-7bc2-4d80-8a75-214af118cb6b@email.android.com>
Message-ID: <CAPkN8x+Q_okfxy6abaGXOQJW+nMpH3j1dnS9cODjsmRrOoL8pw@mail.gmail.com>

I used to backends that are kind of service that is plugged into
application. Maybe a `backendc` would be a better name, but
then what `backendopt` is for?

On Sat, Mar 7, 2015 at 3:21 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> That makes no sense, one cannot import packages with '-' in them.
>
> Carl Friedrich
>
>
>
> On March 6, 2015 3:23:49 PM GMT+01:00, anatoly techtonik
> <techtonik at gmail.com> wrote:
>>
>> On Fri, Mar 6, 2015 at 4:48 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>
>> wrote:
>>>
>>>  2015-03-06 13:24 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>>>
>>>>
>>>>  On Wed, Mar 4, 2015 at 9:24 PM, Maciej Fijalkowski <fijall at gmail.com>
>>>>  wrote:
>>>>>
>>>>>  you need to write a backend. Backends are in translator/xxx.
>>>>
>>>>
>>>>  There are multiple dirs in rpython/translator:
>>>>
>>>>    backendopt
>>>>    c
>>>>    goal
>>>>    platform
>>>>    sandbox
>>>>    test
>>>>    tool
>>>>
>>>>  Which of those are backends?
>>>
>>>
>>>
>>>  Only one: c
>>
>>
>> I c. =) Can I propose to rename the directory to back-c/ ? Kind of
>> compromise between brevity and clarity.



-- 
anatoly t.

From arigo at tunes.org  Mon Mar  9 13:42:20 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 9 Mar 2015 13:42:20 +0100
Subject: [pypy-dev] [pypy-commit] pypy vmprof: uh
In-Reply-To: <CAK5idxS0A2KqZG8=jzjX02E47hN590ZvVgUq+O_YD0YgqzM3FQ@mail.gmail.com>
References: <20150308170336.C0ECD1C11F1@cobra.cs.uni-duesseldorf.de>
 <CAK5idxS0A2KqZG8=jzjX02E47hN590ZvVgUq+O_YD0YgqzM3FQ@mail.gmail.com>
Message-ID: <CAMSv6X22knfGLJfHeEaQHRVT_osfeJ1MGKz9A3D2cgD07Pdt=g@mail.gmail.com>

Hi Fijal,

On 9 March 2015 at 09:21, Maciej Fijalkowski <fijall at gmail.com> wrote:
> uh, I really don't think this checkin is correct (it was meant to seek
> to the end, not 2 bytes)

>> -        ec.code_info_file.seek(2, 0)
>> +        ec.code_info_file.seek(0, 2)

Sorry, I may be missing something, but it seems to me that "seek(0,
2)" is the order you need for "seek to the end"...


A bient?t,

Armin.

From alex.gaynor at gmail.com  Mon Mar  9 13:58:59 2015
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 9 Mar 2015 08:58:59 -0400
Subject: [pypy-dev] [pypy-commit] pypy vmprof: uh
In-Reply-To: <CAMSv6X22knfGLJfHeEaQHRVT_osfeJ1MGKz9A3D2cgD07Pdt=g@mail.gmail.com>
References: <20150308170336.C0ECD1C11F1@cobra.cs.uni-duesseldorf.de>
 <CAK5idxS0A2KqZG8=jzjX02E47hN590ZvVgUq+O_YD0YgqzM3FQ@mail.gmail.com>
 <CAMSv6X22knfGLJfHeEaQHRVT_osfeJ1MGKz9A3D2cgD07Pdt=g@mail.gmail.com>
Message-ID: <CAFRnB2VRHwt7o6B3h0npRnG1SqVN-hopW0Pd5BxHuJLG0SMOAw@mail.gmail.com>

May I suggest we use the `os.SEEK_*` constants for this; not all of us have
memorized POSIX :-)

Alex

On Mon, Mar 9, 2015 at 8:42 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Fijal,
>
> On 9 March 2015 at 09:21, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > uh, I really don't think this checkin is correct (it was meant to seek
> > to the end, not 2 bytes)
>
> >> -        ec.code_info_file.seek(2, 0)
> >> +        ec.code_info_file.seek(0, 2)
>
> Sorry, I may be missing something, but it seems to me that "seek(0,
> 2)" is the order you need for "seek to the end"...
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150309/769e3316/attachment.html>

From techtonik at gmail.com  Mon Mar  9 19:43:15 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 9 Mar 2015 21:43:15 +0300
Subject: [pypy-dev] Language Parser Ontology
Message-ID: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>

I'll start from afar, so that it will be easier to understand what I
am thinking about..

CFFI uses pycparser, which parses C files, but! uses C compiler
to strip comments from C files and process defines, but almost
all .c files contain comments, so pycparser is basically useless
as a parser, but maybe it has a good API for working with AST.

Anyway, I tried to see if I can teach pycparser to strip
comments itself, and in c_lexer.py I found a list of tokens,
among which there were no token representing the comment
start. Stripped list:

    ##
    ## All the tokens recognized by the lexer
    ##
    tokens = keywords + (
        # Identifiers
        'ID',

        # Type identifiers (identifiers previously defined as
        # types with typedef)
        'TYPEID',

        # constants
        'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX',
        'FLOAT_CONST', 'HEX_FLOAT_CONST',
        'CHAR_CONST',
        'WCHAR_CONST',
.    ...

So I thought that I need to add a name for a token
corresponding to comments start //, /* and end */
and it will be better if the token name would be somewhat
common among parsers, so that people looking at token
could immediately recognize that it is a comment related.
Apparently, properly naming is a little bit ambiguous for a
automated processing. Editors like Spyder could also
benefit information about token and their meaning in
different programming languages. The processing of text
comments that can be catched from the parsing stream is
same for any language and could be IDE independent.
Right now you can't just reuse the language definitions
(such as ASDL) to just feed the IDE so that it can
automatically figure out, what parts of text it can attach
its functions to.

I read the ontologies is way to express relations between
object in this automatic was as triples. Like;

  COMMENTSTART is a TOKEN
  COMMENTSTART starts a COMMENT

And I wonder, have anybody tried to apply this ontology
stuff to designing and analysing computer languages?
If yes, maybe there are some databases with such
information about parsers. I would like to query names of
all tokens that represent a program comment.

-- 
anatoly t.

From amauryfa at gmail.com  Tue Mar 10 11:32:54 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Mar 2015 11:32:54 +0100
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
Message-ID: <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>

Hi Anatoly,

2015-03-09 19:43 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:

> I'll start from afar, so that it will be easier to understand what I
> am thinking about..
>
> CFFI uses pycparser, which parses C files, but! uses C compiler
> to strip comments from C files and process defines, but almost
> all .c files contain comments, so pycparser is basically useless
> as a parser, but maybe it has a good API for working with AST.
>
> Anyway, I tried to see if I can teach pycparser to strip
> comments itself, and in c_lexer.py I found a list of tokens,
> among which there were no token representing the comment
> start.


This looks off-topic for PyPy or CFFI.
Did you intend to send this message to some pycparser forum?


> Stripped list:
>
>     ##
>     ## All the tokens recognized by the lexer
>     ##
>     tokens = keywords + (
>         # Identifiers
>         'ID',
>
>         # Type identifiers (identifiers previously defined as
>         # types with typedef)
>         'TYPEID',
>
>         # constants
>         'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX',
>         'FLOAT_CONST', 'HEX_FLOAT_CONST',
>         'CHAR_CONST',
>         'WCHAR_CONST',
> .    ...
>
> So I thought that I need to add a name for a token
> corresponding to comments start //, /* and end */
> and it will be better if the token name would be somewhat
> common among parsers, so that people looking at token
> could immediately recognize that it is a comment related.
> Apparently, properly naming is a little bit ambiguous for a
> automated processing. Editors like Spyder could also
> benefit information about token and their meaning in
> different programming languages. The processing of text
> comments that can be catched from the parsing stream is
> same for any language and could be IDE independent.
> Right now you can't just reuse the language definitions
> (such as ASDL) to just feed the IDE so that it can
> automatically figure out, what parts of text it can attach
> its functions to.
>
> I read the ontologies is way to express relations between
> object in this automatic was as triples. Like;
>
>   COMMENTSTART is a TOKEN
>   COMMENTSTART starts a COMMENT
>
> And I wonder, have anybody tried to apply this ontology
> stuff to designing and analysing computer languages?
> If yes, maybe there are some databases with such
> information about parsers. I would like to query names of
> all tokens that represent a program comment.
>
> --
> anatoly t.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150310/515366cd/attachment-0001.html>

From rymg19 at gmail.com  Tue Mar 10 16:28:01 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 10 Mar 2015 10:28:01 -0500
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
Message-ID: <CAO41-mNqAOYnEYQmi-CJYZGc+nLr5qaDtTOdFckiQ=GS7G+0BA@mail.gmail.com>

Pycparser doesn't use the C compiler to strip comments; it uses the C
*preprocessor*.

Even without comments, it'd probably need the C preprocessor anyway for
things like macros and includes.

On Mon, Mar 9, 2015 at 1:43 PM, anatoly techtonik <techtonik at gmail.com>
wrote:

> I'll start from afar, so that it will be easier to understand what I
> am thinking about..
>
> CFFI uses pycparser, which parses C files, but! uses C compiler
> to strip comments from C files and process defines, but almost
> all .c files contain comments, so pycparser is basically useless
> as a parser, but maybe it has a good API for working with AST.
>
> Anyway, I tried to see if I can teach pycparser to strip
> comments itself, and in c_lexer.py I found a list of tokens,
> among which there were no token representing the comment
> start. Stripped list:
>
>     ##
>     ## All the tokens recognized by the lexer
>     ##
>     tokens = keywords + (
>         # Identifiers
>         'ID',
>
>         # Type identifiers (identifiers previously defined as
>         # types with typedef)
>         'TYPEID',
>
>         # constants
>         'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX',
>         'FLOAT_CONST', 'HEX_FLOAT_CONST',
>         'CHAR_CONST',
>         'WCHAR_CONST',
> .    ...
>
> So I thought that I need to add a name for a token
> corresponding to comments start //, /* and end */
> and it will be better if the token name would be somewhat
> common among parsers, so that people looking at token
> could immediately recognize that it is a comment related.
> Apparently, properly naming is a little bit ambiguous for a
> automated processing. Editors like Spyder could also
> benefit information about token and their meaning in
> different programming languages. The processing of text
> comments that can be catched from the parsing stream is
> same for any language and could be IDE independent.
> Right now you can't just reuse the language definitions
> (such as ASDL) to just feed the IDE so that it can
> automatically figure out, what parts of text it can attach
> its functions to.
>
> I read the ontologies is way to express relations between
> object in this automatic was as triples. Like;
>
>   COMMENTSTART is a TOKEN
>   COMMENTSTART starts a COMMENT
>
> And I wonder, have anybody tried to apply this ontology
> stuff to designing and analysing computer languages?
> If yes, maybe there are some databases with such
> information about parsers. I would like to query names of
> all tokens that represent a program comment.
>
> --
> anatoly t.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Ryan
If I were in a 10-story building glass-sided building and forced to write
either Go or autotools scripts, I?d jump out a window.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150310/1f25dbbf/attachment.html>

From matti.picus at gmail.com  Wed Mar 11 21:19:15 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 11 Mar 2015 22:19:15 +0200
Subject: [pypy-dev] fdopen(fd, 'w') whe fd is opened 'r'
In-Reply-To: <CACw2m-ZwXM1oLu+H7uyo1GPtGHmvn_b41VcYmsq80Boo8A2K1g@mail.gmail.com>
References: <54E8FF9E.1060106@gmail.com>
 <CACw2m-ZwXM1oLu+H7uyo1GPtGHmvn_b41VcYmsq80Boo8A2K1g@mail.gmail.com>
Message-ID: <5500A343.7020907@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150311/402fb192/attachment.html>

From chris.peel at ieee.org  Thu Mar 12 04:10:48 2015
From: chris.peel at ieee.org (Christian Peel)
Date: Wed, 11 Mar 2015 20:10:48 -0700
Subject: [pypy-dev] Codespeed as on speed.pypy.org
Message-ID: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>

Hi all,

I'm working with the author of Codespeed (Miquel Torres) to determine
whether there would be interest in a hosted  service which provided
information similar to those from Codespeed as used at speed.pypy.org.

Our belief is that pypy.org would be interested in using a freemium hosted
service with features similar to (or better than!) those in codespeed and
which was as easy to set up as Travis CI; can you confirm this?  What
additional features would you want in such a service?    Are any of you
using Codespeed commercially?

My best

Chris

-- 
chris.peel at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150311/268b1d46/attachment.html>

From fijall at gmail.com  Thu Mar 12 08:37:04 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 12 Mar 2015 09:37:04 +0200
Subject: [pypy-dev] Codespeed as on speed.pypy.org
In-Reply-To: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
References: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
Message-ID: <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>

Hi Christian

We as PyPy are not generally interested in freemium services (we are
poor to start with), however I think there is some potential with
codespeed being notoriously hard to deploy. I've seen
http://speed.pyston.org/ and http://speed.twistedmatrix.org/ (which is
under some URL, not clue where) deployed already and I do know about a
few commercial deployments.

Cheers,
fijal

On Thu, Mar 12, 2015 at 5:10 AM, Christian Peel <chris.peel at ieee.org> wrote:
> Hi all,
>
> I'm working with the author of Codespeed (Miquel Torres) to determine
> whether there would be interest in a hosted  service which provided
> information similar to those from Codespeed as used at speed.pypy.org.
>
> Our belief is that pypy.org would be interested in using a freemium hosted
> service with features similar to (or better than!) those in codespeed and
> which was as easy to set up as Travis CI; can you confirm this?  What
> additional features would you want in such a service?    Are any of you
> using Codespeed commercially?
>
> My best
>
> Chris
>
> --
> chris.peel at ieee.org
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From mount.sarah at gmail.com  Thu Mar 12 10:29:55 2015
From: mount.sarah at gmail.com (Sarah Mount)
Date: Thu, 12 Mar 2015 09:29:55 +0000
Subject: [pypy-dev] Codespeed as on speed.pypy.org
In-Reply-To: <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
References: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
 <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
Message-ID: <CALG3WZnu9sBnFCTgCT8einjUmBksyZXP6+S2=MeggGO8h9tBRw@mail.gmail.com>

Hi there,

The twistedmatrix deployment is here BTW: http://speed.twistedmatrix.com/

This is an interesting idea (and Codespeed is ace!). I have a few things
that Codespeed could help with, although I'm, not sure I could justify
paying for them, but certainly this is something that would interest me.
One of my undergrads (Sam Giles) used it in his final year thesis and it
worked very well for him.

A few random thoughts for you:

- To be more useful for academics Codespeed would need to generate more
statistical information and much better quality charts. Statistical tests
on improvements between versions, confidence intervals, etc. would be
handy. This piece of work has some very nice ideas:
https://github.com/jamesbornholt/plotty This would move Codespeed from
being able to provide a basic "smoke test" for performance improvement, to
something much more rigorous. I guess not everyone would necessarily need
that, but I think it would not do anyone harm to have it.

- Table output in LaTeX would also be useful for academic types, as would
some way to make the charts more "configurable", or maybe just being able
to dump the Python code (+ data) that creates the charts would do, then
people could mess around offline. Another option would be to use
https://plot.ly/ which gives you the code to generate each chart you create
on its API (e.g. click on the "code" tab on the left of this:
https://plot.ly/~snim2/43/ ). You could go a step further and have
something like an "open my results in LaTeX / Overleaf [
https://www.overleaf.com/devs ] or ShareLaTeX. The big thing here is
workflow - how does the user go from pushing a changeset in their code to
generating high quality analysis and output for a blog or paper publication
or a business pitch? There are several ways to reduce friction there, you
can probably think of more ideas than I have.

- The style files could do with an update, which probably doesn't matter
much to people on this list, but may matter more if you want to
commercialise

- For any user ease of use is really the key thing, but also flexibility.
It is important to be able to get at the data in different formats, but
people will probably find uses for Codespeed that you hadn't intended (and
I guess not all future users would necessarily be full time software
developers), and there are cases where the PyPy way of doing things isn't
quite what you want. Here's a simple example use case: maybe instead of
tracking the progress of some software project, I want to compare
benchmarks for different pieces of hardware. When I push changes to the
benchmark code and re-run the analysis I would not necessarily expect the
results to change over time (although they might with updated firmware /
SDKs, etc) but I might want to add more hardware platforms to the data and
more benchmarks to the code. Codespeed doesn't quite match that use case at
the moment, but I don't see why it shouldn't be possible to reconfigure it
to be useful in that situation.

- If you decide not to go down the commercial route, just having a "deploy
to Heroku" button [ https://devcenter.heroku.com/articles/heroku-button ]
might bring in many more users.


Cheers,

Sarah



On Thu, Mar 12, 2015 at 7:37 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Christian
>
> We as PyPy are not generally interested in freemium services (we are
> poor to start with), however I think there is some potential with
> codespeed being notoriously hard to deploy. I've seen
> http://speed.pyston.org/ and http://speed.twistedmatrix.org/ (which is
> under some URL, not clue where) deployed already and I do know about a
> few commercial deployments.
>
> Cheers,
> fijal
>
> On Thu, Mar 12, 2015 at 5:10 AM, Christian Peel <chris.peel at ieee.org>
> wrote:
> > Hi all,
> >
> > I'm working with the author of Codespeed (Miquel Torres) to determine
> > whether there would be interest in a hosted  service which provided
> > information similar to those from Codespeed as used at speed.pypy.org.
> >
> > Our belief is that pypy.org would be interested in using a freemium
> hosted
> > service with features similar to (or better than!) those in codespeed and
> > which was as easy to set up as Travis CI; can you confirm this?  What
> > additional features would you want in such a service?    Are any of you
> > using Codespeed commercially?
> >
> > My best
> >
> > Chris
> >
> > --
> > chris.peel at ieee.org
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
website:  http://www.snim2.org/
twitter: @snim2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150312/7eca3014/attachment-0001.html>

From chris.peel at ieee.org  Thu Mar 12 20:30:53 2015
From: chris.peel at ieee.org (Christian Peel)
Date: Thu, 12 Mar 2015 12:30:53 -0700
Subject: [pypy-dev] Codespeed as on speed.pypy.org
In-Reply-To: <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
References: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
 <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
Message-ID: <CAHvgqq2mTfuHrwxPFZRxqF=dYsRpDMAQ1UkSitQOxpv7cPNbAQ@mail.gmail.com>

Fijal,

Thanks for your positive comments. To be clear, the service would be free
for open-source users; I guess in this case you'd be more interested; is
that right?

Could you point us to the commercial deployments?

Chris


On Thu, Mar 12, 2015 at 12:37 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Christian
>
> We as PyPy are not generally interested in freemium services (we are
> poor to start with), however I think there is some potential with
> codespeed being notoriously hard to deploy. I've seen
> http://speed.pyston.org/ and http://speed.twistedmatrix.org/ (which is
> under some URL, not clue where) deployed already and I do know about a
> few commercial deployments.
>
> Cheers,
> fijal
>
> On Thu, Mar 12, 2015 at 5:10 AM, Christian Peel <chris.peel at ieee.org>
> wrote:
> > Hi all,
> >
> > I'm working with the author of Codespeed (Miquel Torres) to determine
> > whether there would be interest in a hosted  service which provided
> > information similar to those from Codespeed as used at speed.pypy.org.
> >
> > Our belief is that pypy.org would be interested in using a freemium
> hosted
> > service with features similar to (or better than!) those in codespeed and
> > which was as easy to set up as Travis CI; can you confirm this?  What
> > additional features would you want in such a service?    Are any of you
> > using Codespeed commercially?
> >
> > My best
> >
> > Chris
> >
> > --
> > chris.peel at ieee.org
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
>



-- 
chris.peel at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150312/776cdc67/attachment.html>

From fijall at gmail.com  Thu Mar 12 20:35:20 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 12 Mar 2015 21:35:20 +0200
Subject: [pypy-dev] Codespeed as on speed.pypy.org
In-Reply-To: <CAHvgqq2mTfuHrwxPFZRxqF=dYsRpDMAQ1UkSitQOxpv7cPNbAQ@mail.gmail.com>
References: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
 <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
 <CAHvgqq2mTfuHrwxPFZRxqF=dYsRpDMAQ1UkSitQOxpv7cPNbAQ@mail.gmail.com>
Message-ID: <CAK5idxT5U_AJ-C5dw444meGRisPQ16RFdaUjNCS706x_9YnR2Q@mail.gmail.com>

On Thu, Mar 12, 2015 at 9:30 PM, Christian Peel <chris.peel at ieee.org> wrote:
> Fijal,
>
> Thanks for your positive comments. To be clear, the service would be free
> for open-source users; I guess in this case you'd be more interested; is
> that right?

Yes, I believe so, it all depends what exact benefits it gives us (but
as far as I know we don't mind hosting some stuff on outside systems).

>
> Could you point us to the commercial deployments?

No :-( as far as I know I'm not allowed to talk about any details
(none of it is public).

From chris.peel at ieee.org  Thu Mar 12 22:51:20 2015
From: chris.peel at ieee.org (Christian Peel)
Date: Thu, 12 Mar 2015 14:51:20 -0700
Subject: [pypy-dev] Codespeed as on speed.pypy.org
In-Reply-To: <CALG3WZnu9sBnFCTgCT8einjUmBksyZXP6+S2=MeggGO8h9tBRw@mail.gmail.com>
References: <CAHvgqq2m=WzQ_h_seAYAgVsW0J2d8qVRx70L9G3gX8iWwVgiqg@mail.gmail.com>
 <CAK5idxQoTQQbbg2_fdKnn8Lekm1qe5vV4AQOOn0aSvwB6mH7XA@mail.gmail.com>
 <CALG3WZnu9sBnFCTgCT8einjUmBksyZXP6+S2=MeggGO8h9tBRw@mail.gmail.com>
Message-ID: <CAHvgqq3fgeSCnTm-PzebRBuRnnmu7yT_1xTO_jhQvC5MwDJ+MA@mail.gmail.com>

Sarah,

Thanks for the positive comments;  and again I should have been more clear
that we're thinking that the hosted service would be be free for
open-source users.  I guess in this case that you'd be more inclined to use
it; is that right?

Also, thank you so much for the other ideas and suggestions.   I like all
the plotting suggestions and your thoughts about flexibility.  If we make a
hosted service, there indeed are many issues with hardware that would need
to be addressed; we have considered comparing hardware architectures.

And yes, thanks for the pointer to the "Deploy to Heroku" button;  I agree
that for an open-source project this would be very useful.

I'm happy to talk with any of you further, but we should do it off the pypy
list.

Thanks again!

Chris

On Thu, Mar 12, 2015 at 2:29 AM, Sarah Mount <mount.sarah at gmail.com> wrote:

> Hi there,
>
> The twistedmatrix deployment is here BTW: http://speed.twistedmatrix.com/
>
> This is an interesting idea (and Codespeed is ace!). I have a few things
> that Codespeed could help with, although I'm, not sure I could justify
> paying for them, but certainly this is something that would interest me.
> One of my undergrads (Sam Giles) used it in his final year thesis and it
> worked very well for him.
>
> A few random thoughts for you:
>
> - To be more useful for academics Codespeed would need to generate more
> statistical information and much better quality charts. Statistical tests
> on improvements between versions, confidence intervals, etc. would be
> handy. This piece of work has some very nice ideas:
> https://github.com/jamesbornholt/plotty This would move Codespeed from
> being able to provide a basic "smoke test" for performance improvement, to
> something much more rigorous. I guess not everyone would necessarily need
> that, but I think it would not do anyone harm to have it.
>
> - Table output in LaTeX would also be useful for academic types, as would
> some way to make the charts more "configurable", or maybe just being able
> to dump the Python code (+ data) that creates the charts would do, then
> people could mess around offline. Another option would be to use
> https://plot.ly/ which gives you the code to generate each chart you
> create on its API (e.g. click on the "code" tab on the left of this:
> https://plot.ly/~snim2/43/ ). You could go a step further and have
> something like an "open my results in LaTeX / Overleaf [
> https://www.overleaf.com/devs ] or ShareLaTeX. The big thing here is
> workflow - how does the user go from pushing a changeset in their code to
> generating high quality analysis and output for a blog or paper publication
> or a business pitch? There are several ways to reduce friction there, you
> can probably think of more ideas than I have.
>
> - The style files could do with an update, which probably doesn't matter
> much to people on this list, but may matter more if you want to
> commercialise
>
> - For any user ease of use is really the key thing, but also flexibility.
> It is important to be able to get at the data in different formats, but
> people will probably find uses for Codespeed that you hadn't intended (and
> I guess not all future users would necessarily be full time software
> developers), and there are cases where the PyPy way of doing things isn't
> quite what you want. Here's a simple example use case: maybe instead of
> tracking the progress of some software project, I want to compare
> benchmarks for different pieces of hardware. When I push changes to the
> benchmark code and re-run the analysis I would not necessarily expect the
> results to change over time (although they might with updated firmware /
> SDKs, etc) but I might want to add more hardware platforms to the data and
> more benchmarks to the code. Codespeed doesn't quite match that use case at
> the moment, but I don't see why it shouldn't be possible to reconfigure it
> to be useful in that situation.
>
> - If you decide not to go down the commercial route, just having a "deploy
> to Heroku" button [ https://devcenter.heroku.com/articles/heroku-button ]
> might bring in many more users.
>
>
> Cheers,
>
> Sarah
>
>
>
> On Thu, Mar 12, 2015 at 7:37 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>
>> Hi Christian
>>
>> We as PyPy are not generally interested in freemium services (we are
>> poor to start with), however I think there is some potential with
>> codespeed being notoriously hard to deploy. I've seen
>> http://speed.pyston.org/ and http://speed.twistedmatrix.org/ (which is
>> under some URL, not clue where) deployed already and I do know about a
>> few commercial deployments.
>>
>> Cheers,
>> fijal
>>
>> On Thu, Mar 12, 2015 at 5:10 AM, Christian Peel <chris.peel at ieee.org>
>> wrote:
>> > Hi all,
>> >
>> > I'm working with the author of Codespeed (Miquel Torres) to determine
>> > whether there would be interest in a hosted  service which provided
>> > information similar to those from Codespeed as used at speed.pypy.org.
>> >
>> > Our belief is that pypy.org would be interested in using a freemium
>> hosted
>> > service with features similar to (or better than!) those in codespeed
>> and
>> > which was as easy to set up as Travis CI; can you confirm this?  What
>> > additional features would you want in such a service?    Are any of you
>> > using Codespeed commercially?
>> >
>> > My best
>> >
>> > Chris
>> >
>> > --
>> > chris.peel at ieee.org
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
> website:  http://www.snim2.org/
> twitter: @snim2
>



-- 
chris.peel at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150312/950bdb5e/attachment-0001.html>

From matti.picus at gmail.com  Fri Mar 13 15:41:39 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 13 Mar 2015 16:41:39 +0200
Subject: [pypy-dev] culling non-functional build slaves
In-Reply-To: <A2F6C55C-6BAD-4D5B-B2A1-52AD904563B9@mutehq.net>
References: <A2F6C55C-6BAD-4D5B-B2A1-52AD904563B9@mutehq.net>
Message-ID: <5502F723.8010003@gmail.com>

I propose we remove the non-functioning build slaves. I have been trying 
to contact the owners, here is what I have so far:

http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-64-single-run
speed-python-64: offline since June

http://buildbot.pypy.org/builders/own-indiana-x86-32
jcea-openindiana-32: no owner listed

http://buildbot.pypy.org/builders/own-win-x86-64
snakepit64: offline since at least October

http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-8-x86-64
ananke: offline since at least Nov

http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64 headless:
The buildslave is connected but non-functional. I contacted the owner, 
they cannot continue to run the buildslave at this time [0]

http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64
xerxes: The buildslave is connected but non-functional. The owner cannot 
diagnose the build failures, but is willing to work through the problems 
if anyone versed in an old version of MacOS can help him. In the mean 
time I have paused the slave as we have an alternative slave available

I suggest we remove speed-python-64, jcea-openindiana-32, snakepit64, 
anake, and headless, and leave xerxes in "Paused" state. Any objections?
Matti


[0] here is what the headless buildslave owner replied, I encouraged him 
to join us again in the future:
Hi Matti,
I still have the FreeBSD machine, but at some point the memory 
requirements for a 64 bits build went over 4GB and the machine couldn?t 
complete the builds in a reasonable time. Unfortunately, the machine is 
getting old and I cannot add more memory to it. I remember trying the 
use of the following environment variables to attempt reducing the build 
time memory usage without success:
PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300

I initially provided this buildslave as I had some time to fix PyPy on 
FreeBSD a few years ago for my master degree, I haven?t had the chance 
to work with Python for a long time.

Regards,
Gabriel


From fijall at gmail.com  Fri Mar 13 15:44:48 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 13 Mar 2015 16:44:48 +0200
Subject: [pypy-dev] culling non-functional build slaves
In-Reply-To: <5502F723.8010003@gmail.com>
References: <A2F6C55C-6BAD-4D5B-B2A1-52AD904563B9@mutehq.net>
 <5502F723.8010003@gmail.com>
Message-ID: <CAK5idxSbBe8CnbAxOMPuZMQaaidVR8aBqnzAaNzLxmkP11+rzQ@mail.gmail.com>

+1 on all you propose

On Fri, Mar 13, 2015 at 4:41 PM, Matti Picus <matti.picus at gmail.com> wrote:
> I propose we remove the non-functioning build slaves. I have been trying to
> contact the owners, here is what I have so far:
>
> http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-64-single-run
> speed-python-64: offline since June
>
> http://buildbot.pypy.org/builders/own-indiana-x86-32
> jcea-openindiana-32: no owner listed
>
> http://buildbot.pypy.org/builders/own-win-x86-64
> snakepit64: offline since at least October
>
> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-8-x86-64
> ananke: offline since at least Nov
>
> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64 headless:
> The buildslave is connected but non-functional. I contacted the owner, they
> cannot continue to run the buildslave at this time [0]
>
> http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64
> xerxes: The buildslave is connected but non-functional. The owner cannot
> diagnose the build failures, but is willing to work through the problems if
> anyone versed in an old version of MacOS can help him. In the mean time I
> have paused the slave as we have an alternative slave available
>
> I suggest we remove speed-python-64, jcea-openindiana-32, snakepit64, anake,
> and headless, and leave xerxes in "Paused" state. Any objections?
> Matti
>
>
> [0] here is what the headless buildslave owner replied, I encouraged him to
> join us again in the future:
> Hi Matti,
> I still have the FreeBSD machine, but at some point the memory requirements
> for a 64 bits build went over 4GB and the machine couldn?t complete the
> builds in a reasonable time. Unfortunately, the machine is getting old and I
> cannot add more memory to it. I remember trying the use of the following
> environment variables to attempt reducing the build time memory usage
> without success:
> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300
>
> I initially provided this buildslave as I had some time to fix PyPy on
> FreeBSD a few years ago for my master degree, I haven?t had the chance to
> work with Python for a long time.
>
> Regards,
> Gabriel
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From yorik.sar at gmail.com  Fri Mar 13 19:19:03 2015
From: yorik.sar at gmail.com (Yuriy Taraday)
Date: Fri, 13 Mar 2015 18:19:03 +0000
Subject: [pypy-dev] culling non-functional build slaves
In-Reply-To: <5502F723.8010003@gmail.com>
References: <A2F6C55C-6BAD-4D5B-B2A1-52AD904563B9@mutehq.net>
 <5502F723.8010003@gmail.com>
Message-ID: <CABocrW7fqN-qub25GFuyckKrC1Xij40u-XFPpChGarMeTWFS8g@mail.gmail.com>

Hello, Matti.

On Fri, Mar 13, 2015 at 5:40 PM Matti Picus <matti.picus at gmail.com> wrote:

> I propose we remove the non-functioning build slaves. I have been trying
> to contact the owners, here is what I have so far:
>
> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-8-x86-64
> ananke: offline since at least Nov
>
> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64 headless:
> The buildslave is connected but non-functional. I contacted the owner,
> they cannot continue to run the buildslave at this time [0]
>
> [0] here is what the headless buildslave owner replied, I encouraged him
> to join us again in the future:
> Hi Matti,
> I still have the FreeBSD machine, but at some point the memory
> requirements for a 64 bits build went over 4GB and the machine couldn?t
> complete the builds in a reasonable time. Unfortunately, the machine is
> getting old and I cannot add more memory to it. I remember trying the
> use of the following environment variables to attempt reducing the build
> time memory usage without success:
> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300
>
> I initially provided this buildslave as I had some time to fix PyPy on
> FreeBSD a few years ago for my master degree, I haven?t had the chance
> to work with Python for a long time.
>

I'm in (rather slow) process of finishing fresh FreeBSD install on my home
server after long awaited and surprisingly significant hardware upgrade.
So I would be glad to donate chunk of RAM and one CPU to PyPy for a slave
(or even two). I have couple of questions though:
1. Do builds on FreeBSD provide any value to PyPy? It seems that all
FreeBSD slaves are failing and there's no mention of FreeBSD support on
download page...
2. How often are builds run on these slaves? Should it be occupied 100%
time it might face problems when I'd want to reboot or upgrade it.
3. Who administers them? Will I have to provide root access to this jail to
someone or will I need to watch after them myself?

Kind regards,
Yuriy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150313/2ddd3c65/attachment.html>

From arigo at tunes.org  Sat Mar 14 09:57:07 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 14 Mar 2015 09:57:07 +0100
Subject: [pypy-dev] culling non-functional build slaves
In-Reply-To: <CABocrW7fqN-qub25GFuyckKrC1Xij40u-XFPpChGarMeTWFS8g@mail.gmail.com>
References: <A2F6C55C-6BAD-4D5B-B2A1-52AD904563B9@mutehq.net>
 <5502F723.8010003@gmail.com>
 <CABocrW7fqN-qub25GFuyckKrC1Xij40u-XFPpChGarMeTWFS8g@mail.gmail.com>
Message-ID: <CAMSv6X0kzkTaQMA6KF_gq092Pj7GvEFeSNyL2U9ymWLwGjvvEA@mail.gmail.com>

Hi Yuriy,

Looking at the situation Matti describes where some buildslaves have
not been running for months, it seems that nobody was really
interested in looking at the results enough to care about sending a
mail to their owners...  So while we do appreciate some buildslaves,
and they are not a burden to set up for us, maybe you want to consider
the wider picture: would your buildslave be looked at by anyone?  In
more details:

On 13 March 2015 at 19:19, Yuriy Taraday <yorik.sar at gmail.com> wrote:
> 1. Do builds on FreeBSD provide any value to PyPy? It seems that all FreeBSD
> slaves are failing and there's no mention of FreeBSD support on download
> page...

So far, a few people have contributed FreeBSD buildslaves and
occasional fixes.  This makes FreeBSD somewhat supported (possibly
with some time lag), unofficially.  So whether additional buildslaves
provide value is up to the FreeBSD-interested people to decide.  I
would recommend to look at what FreeBSD buildslaves are working, and
if there is a gap you're interested in filling.

> 2. How often are builds run on these slaves? Should it be occupied 100% time
> it might face problems when I'd want to reboot or upgrade it.

By default, there is one or two builds every 24 hours.  A build should
finish in 2-3 hours as long as there is no swapping.  We can also give
the slave a custom configuration, like have it run a build weekly
instead of daily.  (It seems a bit pointless to run daily builds if
no-one looks at the result every day).

> 3. Who administers them? Will I have to provide root access to this jail to
> someone or will I need to watch after them myself?

Likely the second case.  We provide installation instructions.  It's
mostly up to you to make sure it runs, although usually it requires
little intervention.


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Mar 15 20:41:47 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 15 Mar 2015 21:41:47 +0200
Subject: [pypy-dev] Release 2.5.1 is close, please help test
Message-ID: <5505E07B.3000003@gmail.com>

The 2.5.1 release is almost ready. The release notice is here
http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
Note it has a broken link, the whatsnew is here instead
http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
This will be fixed when I merge the release branch back into default 
together with the official release

The files ready for repackaging are here
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux.tar.bz2
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux64.tar.bz2
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-win32.zip
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-osx64.tar.bz2
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raspbian.tar.bz2
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raring.tar.bz2
http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raring.tar.bz2
https://bitbucket.org/pypy/pypy/get/release-2.5.1.zip
https://bitbucket.org/pypy/pypy/get/release-2.5.1.tar.gz

Could people please take a few minutes to test them out on their 
favorite platform?
Also, any corrections, additions, or edits to the release notice would 
be welcome, especially a name instead of XXXXXX
Perhaps from http://xkcd.com,  something to do with bromeliads 
http://www.bsi.org/brom_info/what.html, maybe we can draw a parallel 
between the spread of them and the spreading popularity of rpython and cffi:

"Within the last hundred years, bromeliads have become more widely used 
as ornamental plants. Originally only found in royal botanical gardens 
or the private greenhouses of wealthy Europeans, their popularity has 
spread to the masses. Today bromeliads are more available to the 
enthusiast than ever before. New species are still being discovered and 
plant breeders are developing ever more stunning hybrids to choose from."

Matti

From techtonik at gmail.com  Mon Mar 16 11:49:28 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 13:49:28 +0300
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
 <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>
Message-ID: <CAPkN8xKChTPoUqf62Jy4NwXVryLjnmq7_bUjWykusQWMa8yNHQ@mail.gmail.com>

On Tue, Mar 10, 2015 at 1:32 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2015-03-09 19:43 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>
>> I'll start from afar, so that it will be easier to understand what I
>> am thinking about..
>>
>> CFFI uses pycparser, which parses C files, but! uses C compiler
>> to strip comments from C files and process defines, but almost
>> all .c files contain comments, so pycparser is basically useless
>> as a parser, but maybe it has a good API for working with AST.
>>
>> Anyway, I tried to see if I can teach pycparser to strip
>> comments itself, and in c_lexer.py I found a list of tokens,
>> among which there were no token representing the comment
>> start.
>
>
> This looks off-topic for PyPy or CFFI.
> Did you intend to send this message to some pycparser forum?

I'll keep in mind that 10 lines is a limit for a topic start. Let me quote
from the bottom:

--cut--
I read the ontologies is way to express relations between
object in this automatic was as triples. Like;

  COMMENTSTART is a TOKEN
  COMMENTSTART starts a COMMENT

And I wonder, have anybody tried to apply this ontology
stuff to designing and analysing computer languages?
If yes, maybe there are some databases with such
information about parsers. I would like to query names of
all tokens that represent a program comment.
--cut--

Is the discussion about language parser strategy appropriate for
pypy-dev?

-- 
anatoly t.

From fijall at gmail.com  Mon Mar 16 11:58:19 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 16 Mar 2015 12:58:19 +0200
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAPkN8xKChTPoUqf62Jy4NwXVryLjnmq7_bUjWykusQWMa8yNHQ@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
 <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>
 <CAPkN8xKChTPoUqf62Jy4NwXVryLjnmq7_bUjWykusQWMa8yNHQ@mail.gmail.com>
Message-ID: <CAK5idxQ8kwucwbnoPcCmi6nibuSoZ3ZxV10ihHbcb-PrBshjLA@mail.gmail.com>

On Mon, Mar 16, 2015 at 12:49 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Tue, Mar 10, 2015 at 1:32 PM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
>> 2015-03-09 19:43 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>>
>>> I'll start from afar, so that it will be easier to understand what I
>>> am thinking about..
>>>
>>> CFFI uses pycparser, which parses C files, but! uses C compiler
>>> to strip comments from C files and process defines, but almost
>>> all .c files contain comments, so pycparser is basically useless
>>> as a parser, but maybe it has a good API for working with AST.
>>>
>>> Anyway, I tried to see if I can teach pycparser to strip
>>> comments itself, and in c_lexer.py I found a list of tokens,
>>> among which there were no token representing the comment
>>> start.
>>
>>
>> This looks off-topic for PyPy or CFFI.
>> Did you intend to send this message to some pycparser forum?
>
> I'll keep in mind that 10 lines is a limit for a topic start. Let me quote
> from the bottom:
>
> --cut--
> I read the ontologies is way to express relations between
> object in this automatic was as triples. Like;
>
>   COMMENTSTART is a TOKEN
>   COMMENTSTART starts a COMMENT
>
> And I wonder, have anybody tried to apply this ontology
> stuff to designing and analysing computer languages?
> If yes, maybe there are some databases with such
> information about parsers. I would like to query names of
> all tokens that represent a program comment.
> --cut--
>
> Is the discussion about language parser strategy appropriate for
> pypy-dev?
>

No, this is not an appropriate topic. pypy-dev is about development of
PyPy and associated technologies, not about discussing theoretical
properties of parsers.

Cheers,
fijal

From techtonik at gmail.com  Mon Mar 16 12:01:22 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 14:01:22 +0300
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAK5idxQ8kwucwbnoPcCmi6nibuSoZ3ZxV10ihHbcb-PrBshjLA@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
 <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>
 <CAPkN8xKChTPoUqf62Jy4NwXVryLjnmq7_bUjWykusQWMa8yNHQ@mail.gmail.com>
 <CAK5idxQ8kwucwbnoPcCmi6nibuSoZ3ZxV10ihHbcb-PrBshjLA@mail.gmail.com>
Message-ID: <CAPkN8x+QwmtE_9nBCBQ6t1rkPZJYgV6c7Z=RDiqhQkxF8_kxVw@mail.gmail.com>

On Mon, Mar 16, 2015 at 1:58 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Mar 16, 2015 at 12:49 PM, anatoly techtonik <techtonik at gmail.com> wrote:
>> On Tue, Mar 10, 2015 at 1:32 PM, Amaury Forgeot d'Arc
>> <amauryfa at gmail.com> wrote:
>>> 2015-03-09 19:43 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>>>
>>>> I'll start from afar, so that it will be easier to understand what I
>>>> am thinking about..
>>>>
>>>> CFFI uses pycparser, which parses C files, but! uses C compiler
>>>> to strip comments from C files and process defines, but almost
>>>> all .c files contain comments, so pycparser is basically useless
>>>> as a parser, but maybe it has a good API for working with AST.
>>>>
>>>> Anyway, I tried to see if I can teach pycparser to strip
>>>> comments itself, and in c_lexer.py I found a list of tokens,
>>>> among which there were no token representing the comment
>>>> start.
>>>
>>>
>>> This looks off-topic for PyPy or CFFI.
>>> Did you intend to send this message to some pycparser forum?
>>
>> I'll keep in mind that 10 lines is a limit for a topic start. Let me quote
>> from the bottom:
>>
>> --cut--
>> I read the ontologies is way to express relations between
>> object in this automatic was as triples. Like;
>>
>>   COMMENTSTART is a TOKEN
>>   COMMENTSTART starts a COMMENT
>>
>> And I wonder, have anybody tried to apply this ontology
>> stuff to designing and analysing computer languages?
>> If yes, maybe there are some databases with such
>> information about parsers. I would like to query names of
>> all tokens that represent a program comment.
>> --cut--
>>
>> Is the discussion about language parser strategy appropriate for
>> pypy-dev?
>>
>
> No, this is not an appropriate topic. pypy-dev is about development of
> PyPy and associated technologies, not about discussing theoretical
> properties of parsers.

Yes, but developing a new language in RPython requires choosing a
parser first, no?

-- 
anatoly t.

From arigo at tunes.org  Mon Mar 16 12:35:19 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 16 Mar 2015 12:35:19 +0100
Subject: [pypy-dev] Language Parser Ontology
In-Reply-To: <CAPkN8x+QwmtE_9nBCBQ6t1rkPZJYgV6c7Z=RDiqhQkxF8_kxVw@mail.gmail.com>
References: <CAPkN8xL90wth8pYFTmGU3iDUjmajVz29M+0O9dD9v=Ztcq_GRA@mail.gmail.com>
 <CAGmFidY+eQwxvjpsTKLHYRzXVHB9763dbcjzAxFDpkJoAd5sBg@mail.gmail.com>
 <CAPkN8xKChTPoUqf62Jy4NwXVryLjnmq7_bUjWykusQWMa8yNHQ@mail.gmail.com>
 <CAK5idxQ8kwucwbnoPcCmi6nibuSoZ3ZxV10ihHbcb-PrBshjLA@mail.gmail.com>
 <CAPkN8x+QwmtE_9nBCBQ6t1rkPZJYgV6c7Z=RDiqhQkxF8_kxVw@mail.gmail.com>
Message-ID: <CAMSv6X28vtBSoihG=uz42eU7G9MDbAjrVwCVJ8epBwAiH7BHtg@mail.gmail.com>

Hi,

On 16 March 2015 at 12:01, anatoly techtonik <techtonik at gmail.com> wrote:
> Yes, but developing a new language in RPython requires choosing a
> parser first, no?

No.  Some of RPython's interpreters are only interpreters, without any
parser.  I'm not saying yours must be too, but in general we tend not
to be too interested in parsers at all.  Maybe look at
rpython.rlib.parsing for a pre-made example.  You are free to discuss
parsing more theoretically here or elsewhere, but I would guess it is
more likely you'd get feedback elsewhere...


A bient?t,

Armin.

From me at manueljacob.de  Mon Mar 16 18:42:49 2015
From: me at manueljacob.de (Manuel Jacob)
Date: Mon, 16 Mar 2015 18:42:49 +0100
Subject: [pypy-dev] =?utf-8?q?GSoC_ideas_--_AArch64_JIT_backend=3F?=
Message-ID: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>

Hi,

I decided that this year is a good time to apply for GSoC.  Because I 
never worked on anything JIT-related, this could be a chance to get 
started with it.  There was some discussion about possible improvements 
for the ARM backend.  Also, some newer ARM processors support a 64-bit 
execution mode called AArch64.  I think it makes sense to implement a 
JIT backend for AArch64 sooner or later.  There are just a few 
affordable AArch64 development boards available at the moment, but I was 
able to cross-compile a non-jitted version of PyPy and run it in QEMU.

Do you think that implementing an AArch64 JIT backend is a good GSoC 
project?  If not, can you think of other JIT-related projects that fit 
better in the scope of GSoC?

-Manuel

From estama at gmail.com  Tue Mar 17 18:27:06 2015
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Tue, 17 Mar 2015 19:27:06 +0200
Subject: [pypy-dev] UTF8 string passing in cffi and PyPy internal string
	optimizations
In-Reply-To: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
References: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
Message-ID: <550863EA.2060203@gmail.com>

Hello,

I'm sending the following here as they involve both cffi and PyPy.

For the last few years i have been trying to find the most efficient way 
to pass UTF8 strings between PyPy and C code using cffi.

Right now when PyPy receives a utf8 string (from a C function) it has to 
do 2 copies:

1. convert the cdata string to a pypy byte string via ffi.string
2. convert ffi.string to a unicode string

When pypy sends a utf8 string it also does 2 copies:

1. convert pypy unicode string to utf8-encoded byte string
2. copy the byte string into a cdata string.

 From what i understand, there is a cffi optimization dealing with 
windows unicode (via set_unicode) where on windows platforms and when 
using the native windows unicode strings, cffi avoids doing one of the 
copies in both of above cases.

On linux where the default unicode format for C libraries nowadays is 
UTF8, there is no such optimization, so we have to do the two copies in 
all string passing.

PyPy at some point was going towards using utf8 string internally, but i 
don't know if this is still the plan or not. Using utf8 strings would 
optimize away one of the two copies on the linux platform (utf8 
encoding/decoding would become a nop operator).

All of the above is the current status of cffi and pypy string handling 
as i understand it. So my proposal to reduce the string copies to a 
minimum is this:

1. If PyPy doesn't go towards using utf8 strings internally, maybe we 
need some special C type that denotes that the string is utf8 and 
pypy/cffi should do the conversion from-to it automatically. Something 
like "wchar_t" in windows but denoting a utf8 string. CFFI can define a 
special type ("__utf8char_t"?) for these strings.

Alternatively, an encoding parameter could be added in ffi.string, so 
that it'll do both the cdata and encoding conversions in one step.

2. If PyPy does go towards using utf8 string internally. Then it could 
call C functions that do not mutate the pypy strings and do not store 
pointers to them, by passing the strings directly. This could be 
accomplished by using a cffi annotation for these kind of 
non-string-mutating C functions.

Above ideas are based on my understanding of the current status and the 
future directions of PyPy. If i have misunderstood something i would be 
glad to be set right :).

Kind regards,

l.


From tushart91 at gmail.com  Wed Mar 18 00:26:15 2015
From: tushart91 at gmail.com (Tushar Tiwari)
Date: Tue, 17 Mar 2015 16:26:15 -0700
Subject: [pypy-dev] GSoC 2015: Implement copy-on-write list slicing
Message-ID: <0BCEA231-D97B-4007-A573-D66549117D22@gmail.com>

Hi All,
I am a masters student at the University of Southern California and was hoping to participate in this year?s Google Summer of Code on Implementing copy-on-write list slicing. I would love some documentation on the issue and also, I noticed there is no mentor for the projects so that info would be nice.
Thank You.
?
Tushar Tiwari
tushart91 at gmail.com
http://www.github.com/tushart91
University of Southern California

From arigo at tunes.org  Wed Mar 18 09:20:39 2015
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 18 Mar 2015 09:20:39 +0100
Subject: [pypy-dev] GSoC 2015: Implement copy-on-write list slicing
In-Reply-To: <0BCEA231-D97B-4007-A573-D66549117D22@gmail.com>
References: <0BCEA231-D97B-4007-A573-D66549117D22@gmail.com>
Message-ID: <CAMSv6X1=--NV9Y1U0pRp0qHT3UDyNZ3zuzXzW=Rn0R_=1s8X5w@mail.gmail.com>

Hi Tushar,

In the past few years, we have found GSoC to be tricky to handle.  As
a result of this, we're likely to have a high bar for student
acceptance this year.  The main criteria will be whether you have
already contributed to PyPy in a significant way.  If you only come up
with a GSoC proposal, no matter how cool, it will likely be rejected.

The first step is to get involved with the community, which means showing
up on irc (#pypy on irc.freenode.net).  This is what we say in general
to people that want to contribute to PyPy.  We can move on to discuss
GSoC only after we have seen concrete results from this first step.  I
fear that it might be late for getting involved in a significant way,
though...


A bient?t,

Armin.

From u5157779 at uds.anu.edu.au  Wed Mar 18 01:01:31 2015
From: u5157779 at uds.anu.edu.au (John Zhang)
Date: Wed, 18 Mar 2015 11:01:31 +1100
Subject: [pypy-dev] Compiling PyPy interpreter without GC
Message-ID: <5508C05B.6080609@uds.anu.edu.au>

Hi all,
     I'm working on developing a MicroVM backend for PyPy. It's a 
virtual machine under active research and development by my colleagues 
in ANU. It aims to capture GC, threading and JIT in the virtual machine, 
and frees up the burden of the language implementers.

     Since MicroVM provides GC, I need to remove GC from the PyPy 
interpreter. As I was trying to compile it with the following command:
     pypy $PYPY/rpython/bin/rpython \
           -O0 \
           --gc=none \
           --no-translation-rweakref \
           --annotate \
           --rtype \
           --translation-backendopt-none \
           $PYPY/pypy/goal/targetpypystandalone.py
     It gives off an error during annotation stage, saying that it's not 
able to find a module called '_rweakref'.
     Does anyone know what the problem might be, and how one might go 
and solve it?

     Appreciate greatly,
     John Zhang

From fijall at gmail.com  Wed Mar 18 09:57:15 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 18 Mar 2015 10:57:15 +0200
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <5508C05B.6080609@uds.anu.edu.au>
References: <5508C05B.6080609@uds.anu.edu.au>
Message-ID: <CAK5idxTcBaxaam_0Uo3iH=5Cppjxs8QCt69CCY1jV-GsNaEXpg@mail.gmail.com>

Hi John.

Can you describe the microVM and it's capabilities? Chances are it
captures things at the wrong level (I have a longer response in mind,
but I'll wait for you to describe it, in case I'm plain wrong)

What do you mean by "provides a GC"? Does it mean you just call malloc
and you never have to call free?

Generally speaking we don't suggest you translate pypy as a first
step, but instead write tests (equivalent to what's in
translator/c/test) and check aspects of translation one bit at a time.
That said, dependency on rweakref even when disabled is a bug, can you
post a full traceback?

Cheers,
fijal





On Wed, Mar 18, 2015 at 2:01 AM, John Zhang <u5157779 at uds.anu.edu.au> wrote:
> Hi all,
>     I'm working on developing a MicroVM backend for PyPy. It's a virtual
> machine under active research and development by my colleagues in ANU. It
> aims to capture GC, threading and JIT in the virtual machine, and frees up
> the burden of the language implementers.
>
>     Since MicroVM provides GC, I need to remove GC from the PyPy
> interpreter. As I was trying to compile it with the following command:
>     pypy $PYPY/rpython/bin/rpython \
>           -O0 \
>           --gc=none \
>           --no-translation-rweakref \
>           --annotate \
>           --rtype \
>           --translation-backendopt-none \
>           $PYPY/pypy/goal/targetpypystandalone.py
>     It gives off an error during annotation stage, saying that it's not able
> to find a module called '_rweakref'.
>     Does anyone know what the problem might be, and how one might go and
> solve it?
>
>     Appreciate greatly,
>     John Zhang
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From dimaqq at gmail.com  Wed Mar 18 14:29:22 2015
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 18 Mar 2015 14:29:22 +0100
Subject: [pypy-dev] GSoC ideas -- AArch64 JIT backend?
In-Reply-To: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
References: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
Message-ID: <CAGGBzXJOkmqcKSE78t1dCcm3BHS507RTeJxS4sJQneub1oifvQ@mail.gmail.com>

Please take with a grain of salt, as I'm not a pypy dev.

In general:
yes it's a great idea!
arm64 definitely fits the bill in terms of hardware and arm64 devices
could really use pypy.
I have some reservations in terms how far this can be pushed in a
single summer, but hey, it's good to be ambitious!

In terms of development and testing:
Given enough host ram, you could try to translate pypy under arm64
qemu directly.
It's going to be slow, by about factor of 10, but set up is much easier.

Other thoughts:
ARM memory [synchronisation] model is different than x86, I bet the
difference is present in 64-bit version too.
Given that most popular arm hardware is now multicore, the issue
cannot be avoided.
You'd need a core dev to provide guidance here.

d.

On 16 March 2015 at 18:42, Manuel Jacob <me at manueljacob.de> wrote:
> Hi,
>
> I decided that this year is a good time to apply for GSoC.  Because I never
> worked on anything JIT-related, this could be a chance to get started with
> it.  There was some discussion about possible improvements for the ARM
> backend.  Also, some newer ARM processors support a 64-bit execution mode
> called AArch64.  I think it makes sense to implement a JIT backend for
> AArch64 sooner or later.  There are just a few affordable AArch64
> development boards available at the moment, but I was able to cross-compile
> a non-jitted version of PyPy and run it in QEMU.
>
> Do you think that implementing an AArch64 JIT backend is a good GSoC
> project?  If not, can you think of other JIT-related projects that fit
> better in the scope of GSoC?
>
> -Manuel
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From amauryfa at gmail.com  Wed Mar 18 15:49:22 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 18 Mar 2015 15:49:22 +0100
Subject: [pypy-dev] UTF8 string passing in cffi and PyPy internal string
	optimizations
In-Reply-To: <550863EA.2060203@gmail.com>
References: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
 <550863EA.2060203@gmail.com>
Message-ID: <CAGmFidZ4k8wKFwS5VRzvgqSnf_X5zycaPKG7Fvwdb14C8JTSVQ@mail.gmail.com>

Hi,

2015-03-17 18:27 GMT+01:00 Eleytherios Stamatogiannakis <estama at gmail.com>:

> Hello,
>
> I'm sending the following here as they involve both cffi and PyPy.
>
> For the last few years i have been trying to find the most efficient way
> to pass UTF8 strings between PyPy and C code using cffi.
>
> Right now when PyPy receives a utf8 string (from a C function) it has to
> do 2 copies:
>
> 1. convert the cdata string to a pypy byte string via ffi.string
> 2. convert ffi.string to a unicode string
>
> When pypy sends a utf8 string it also does 2 copies:
>
> 1. convert pypy unicode string to utf8-encoded byte string
> 2. copy the byte string into a cdata string.
>
> From what i understand, there is a cffi optimization dealing with windows
> unicode (via set_unicode) where on windows platforms and when using the
> native windows unicode strings, cffi avoids doing one of the copies in both
> of above cases.
>
> On linux where the default unicode format for C libraries nowadays is
> UTF8, there is no such optimization, so we have to do the two copies in all
> string passing.
>
> PyPy at some point was going towards using utf8 string internally, but i
> don't know if this is still the plan or not. Using utf8 strings would
> optimize away one of the two copies on the linux platform (utf8
> encoding/decoding would become a nop operator).
>
> All of the above is the current status of cffi and pypy string handling as
> i understand it. So my proposal to reduce the string copies to a minimum is
> this:
>
> 1. If PyPy doesn't go towards using utf8 strings internally, maybe we need
> some special C type that denotes that the string is utf8 and pypy/cffi
> should do the conversion from-to it automatically. Something like "wchar_t"
> in windows but denoting a utf8 string. CFFI can define a special type
> ("__utf8char_t"?) for these strings.
>

This is a first step towards SWIG's typemaps:
http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_nn4

That's also something I wanted to have in another projects: automatic
conversion to PYTHON_HANDLE, for example.

But typemaps are a tough thing, and they would likely differ between
CPython and PyPy.
Armin, what do you think?


Alternatively, an encoding parameter could be added in ffi.string, so that
> it'll do both the cdata and encoding conversions in one step.
>
> 2. If PyPy does go towards using utf8 string internally. Then it could
> call C functions that do not mutate the pypy strings and do not store
> pointers to them, by passing the strings directly. This could be
> accomplished by using a cffi annotation for these kind of
> non-string-mutating C functions.
>

Even utf8 is used internally (it is the case already in the py3k branch, as
a cached attribute), I'm not sure I would like fuctions like strlen() to
silently accept unicode strings...


>
> Above ideas are based on my understanding of the current status and the
> future directions of PyPy. If i have misunderstood something i would be
> glad to be set right :).
>
> Kind regards,
>
> l.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150318/3235036d/attachment.html>

From hrc706 at gmail.com  Wed Mar 18 16:49:33 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Thu, 19 Mar 2015 00:49:33 +0900
Subject: [pypy-dev] Some summary and questions about the 'small function'
	problem
Message-ID: <81697ED9-D32F-4E6B-8805-B65BAB56A619@gmail.com>

Hi Fijal, 

   This is Ruochen Huang, I want to begin to write my proposal and I think actually there is not so much time left. I tried to make a summary of what I have understood until now and the questions I want to know. Please feel free to point out any incorrect things in my summary, and for the questions, if you think the question is meaningless, you can just skip it, or provide some possible document link or source code path if you think it will take too much time to explain it. 

   As far as I understood,
The ?small function? problem occurred when one trace try to call another trace. In source code level, it should be the situation that, inside one loop, there is a function invocation to a function which has another loop.
Let me take the small example we discussed before, function g() tried to call function f(a,b,c,d) in a big loop, and there is another loop inside f(a,b,c,d). So in current version of PyPy, the result is that, two traces were generated:
the trace for the loop in g(), let me call it T1, actually, g() tried to inline f(a,b,c,d), but since there is a loop in f, so the result is that T1 will inline only the first iteration of the loop in f, let?s say f was taken apart into f1(for the first iteration) and f?(for the rest iterations), so what T1 exactly does is start the loop in g -> do f1 -> do some allocations of PyFrame (in order to call f?) -> call_assembler for f?.
the trace for the loop in f?. let me call it T2. T2 firstly unpack the PyFrame prepared by T1, then do a preamble work, which means f? is once again taken apart into f2 (for the 1st iteration in f?, and it actually is also the 2nd iteration in original f), and f??(the iterations from 3rd iteration to the last iteration), for f2 and f??, there is a label in the head of them, respectively. So finally we can say T2 consist of 3 parts: T2a (for PyFrame unpacking), T2b(with label1, do f2), T2c(with label2, do f??).
As mentioned above, we have T1 -> T2a -> T2b -> T3c, from the viewpoint of the loop in f, f is distributed into: T1(f1) -> T2a -> T2b(f2) -> T2c(f??), which means the loop in f was peeled twice, so T2b might be not needed, further more, the work for PyFrame before call_assembler in T1, and the unpacking work in T2a is a waste. I can?t understand why it?s a waste very well, but I guess it?s because T2c(f?') actually do the similar thing as f1 in T1, (or, T2c is already *inside* the loop) Anyway, T2b is also not needed, so we want to have T1 -> T2c, and since the work in PyFrame in T2a is eliminated, the allocation for PyFrame in T1 can also be eliminated. So ideally we want to have T1? (without PyFrame allocation) -> T2c.

Some questions until now:
What?s the bridge you mentioned? To be honest I have only a very slight understand of bridge, I know it is executed when some guard failed, but as far as I knew, in normal trace JIT compiler, only one path of a loop will be traced, any guard failure will make the execution escape from the native code and return to VM, but I guess the bridge is a special kind of trace (native code), is it right?
Could you please explain more about why T2b is not needed? I guess the answer may be related to the ?virtualizable? optimization for PyFrame, so what if PyFrame is not virtualizable? I mean, if in that situation, does the problem disappear? or become easier to solve?
What?s the difficulties in solving this problem? I?m sorry I?m not so familiar with the details of RPython JIT, but in my opinion, we need just to make the JIT know that, 
when tries to inline a function, and encounter a loop so the inline work has to stop, it?s time to do optimization O.
what O does is to delete the allocation instructions about PyFrame before call_assembler, and them tell call_assembler to jump to 2rd label of target trace. (In our example is T2c).
       So It may seem not so difficult to solve.

Best Regards,
Ruochen Huang


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150319/d7ebce28/attachment-0001.html>

From u5157779 at anu.edu.au  Wed Mar 18 23:42:59 2015
From: u5157779 at anu.edu.au (John Zhang)
Date: Wed, 18 Mar 2015 22:42:59 +0000
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <5509B35B.5050102@gmx.de>
References: <5508C05B.6080609@uds.anu.edu.au> <5509B35B.5050102@gmx.de>
Message-ID: <B89A6C9B-0E2B-4DD0-BBCB-CE2F3C5A507C@uds.anu.edu.au>

Hi Carl,
	Great! It worked!
	So the option disables all modules, and IO as well?
	
	Cheers,
	John Zhang
> On 19 Mar 2015, at 4:18 am, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> 
> On 18/03/15 01:01, John Zhang wrote:
>> Hi all,
>>     I'm working on developing a MicroVM backend for PyPy. It's a
>> virtual machine under active research and development by my colleagues
>> in ANU. It aims to capture GC, threading and JIT in the virtual machine,
>> and frees up the burden of the language implementers.
>> 
>>     Since MicroVM provides GC, I need to remove GC from the PyPy
>> interpreter. As I was trying to compile it with the following command:
>>     pypy $PYPY/rpython/bin/rpython \
>>           -O0 \
>>           --gc=none \
>>           --no-translation-rweakref \
>>           --annotate \
>>           --rtype \
>>           --translation-backendopt-none \
>>           $PYPY/pypy/goal/targetpypystandalone.py
> 
> Hey John,
> 
> Try the following:
> rpython -O0 --gc=none --no-translation-rweakref --annotate --rtype --translation-backendopt-none targetpypystandalone.py --no-allworkingmodules --withoutmod-_io
> 
> Cheers,
> 
> Carl


From cfbolz at gmx.de  Thu Mar 19 11:17:09 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 19 Mar 2015 11:17:09 +0100
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <B89A6C9B-0E2B-4DD0-BBCB-CE2F3C5A507C@uds.anu.edu.au>
References: <5508C05B.6080609@uds.anu.edu.au> <5509B35B.5050102@gmx.de>
 <B89A6C9B-0E2B-4DD0-BBCB-CE2F3C5A507C@uds.anu.edu.au>
Message-ID: <550AA225.4050704@gmx.de>

On 18/03/15 23:42, John Zhang wrote:
> Hi Carl,
> 	Great! It worked!
> 	So the option disables all modules, and IO as well?

They enable a lot of built-in modules, and the _io module in
particular. Those need weak references and can be fixed later.

CF


From kunshan.wang at anu.edu.au  Fri Mar 20 04:16:40 2015
From: kunshan.wang at anu.edu.au (Kunshan Wang)
Date: Fri, 20 Mar 2015 14:16:40 +1100
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <CAK5idxTcBaxaam_0Uo3iH=5Cppjxs8QCt69CCY1jV-GsNaEXpg@mail.gmail.com>
References: <5508C05B.6080609@uds.anu.edu.au>
 <CAK5idxTcBaxaam_0Uo3iH=5Cppjxs8QCt69CCY1jV-GsNaEXpg@mail.gmail.com>
Message-ID: <550B9118.6050800@anu.edu.au>

Hi Maciej,

I am a PhD student at the Australian National University. I am a
colleague of John Zhang and the chief designer of the Mu project, a
micro virtual machine. (http://microvm.org) I can introduce this project
to this mailing list.


TL;DR: Implementing a managed language is hard. Existing VMs are either
too big or too monolithic. Mu, the micro VM, only handles concurrency,
JIT and GC, which an average language designer probably don't want to
(and is unlikely to have the expertise to) work with. Then a "client"
program can use Mu to implement its language.


Our group at the ANU coined the concept "micro virtual machines" or
"micro VM", which is a parody the concept of "micro kernel" in the OS
literature. Mu is a concrete micro VM (the same way "seL4 is a concrete
microkernel". Mu was previously called MicroVM or ?VM, but we changed
the name to distinguish between "our particular micro VM" and "the
general concept of micro virtual machines".

Our motivation: Many managed programming language implementations suck,
such as CPython which uses GIL, naive reference counting GC and no JIT.
People may want to create alternative implementations. Currently they
either base on another (macro) VM or build a new VM from scratch.
Existing VMs (like JVM, CLR, etc.) usually provide too much abstraction.
They cause semantic gaps and a lot of unnecessary dependencies, while
still do not work well with languages they are not designed for (See how
PyPy outperforms Jython). Others (including PyPy) build a whole new VM,
doing everything from scratch. There are many "high-performance VM"
projects like PyPy (LuaJIT, v8, JavaScriptCore, HHVM to name a few), but
the most important low-level parts (JIT, GC, ...) are not reused.

We proposed a third option. A "micro virtual machine" is minimal,
providing only the abstractions of concurrency, execution (JIT) and
garbage collection, which we identify as the three major concerns that
make language implementation hard. Another program which we call a
"client" sits above the micro VM and implements the concrete language.

Here are some facts about Mu, our concrete micro VM:

1. It has a specification which defines the behaviour, and multiple
implementations are allowed. We also have a reference implementation.

2. The architecture has two layers: a micro VM at the bottom, and a
"client" on top of it. The client interacts with the micro VM through an
API.

3. Its type system is similar with the level of C, but with object
references. The level is similar to the LL type system in PyPy. It has
fixed integers, FP numbers, references, structs, arrays, ..., but no
Java-like object hierarchy. (The micro VM is minimal and does not know
OOP. The client implements its own types using structs, arrays, ...)

4. Its instruction set is LLVM-like, using the SSA form. But it is
designed for managed languages.

5. Heap memory allocation is a primitive instruction. Heap memory is
garbage-collected.

6. Mu has a garbage collector. All references in Mu can be identified,
whether they are in the heap, stack, global variables or held for the
client (in which case the reference is exposed as opaque handles, like
JNI). With all references identified, Mu can perform precise (accurate,
exact) garbage collection.

7. Mu has threads which can run simultaneously (supposed to be
implemented as OS threads, but the spec does not force it). Mu also has
a C++11-like memory model. (Yes. Mu has RELAXED, CONSUME, ACQUIRE,
RELEASE, ACQ_REL, SEQ_CST to scare your children.)

8. Mu's code loading unit is "bundle" (like the ".class" file in Java).
The format is called "Mu IR" (like LLVM IR) which contain codes and
top-level definitions. The client can define new codes at run time. (You
can JIT-compile high-level programs to Mu IR at run time.)

9. Mu has the "TRAP" instruction which pauses the execution of a Mu IR
program and execute a "handler" in the client. The "handler" can
introspect the states of the execution (including local variables) and
perform OSR (on-stack replacement, removing a stack frame and pushing a
new frame of a probably newly-compiled function). After the handler, the
client decide where the thread should continue.

10. Mu has a simple exception handling mechanism. It does not rely on
system libraries.


John Zhang is working on making RPython a front-end language of Mu (in
other words, making Mu a back-end of RPython). We think the LL type
system is roughly at the same level as Mu, but since Mu already has
reference types, heap allocation instruction and an internal garbage
collector, RPython no longer need to insert low-level implementation
details for GC (like read/write barriers, GC-safe points, stack maps and
so on). However, Mu does not expose the memory directly as bytes. For
this reason, some implementation strategies are no longer applicable in
Mu. For example, array copy must be done element by element, and memcpy
cannot be used.


Regards,
Kunshan Wang


On 18/03/2015 7:57 pm, Maciej Fijalkowski wrote:
> Hi John.
> 
> Can you describe the microVM and it's capabilities? Chances are it
> captures things at the wrong level (I have a longer response in mind,
> but I'll wait for you to describe it, in case I'm plain wrong)
> 
> What do you mean by "provides a GC"? Does it mean you just call malloc
> and you never have to call free?
> 
> Generally speaking we don't suggest you translate pypy as a first
> step, but instead write tests (equivalent to what's in
> translator/c/test) and check aspects of translation one bit at a time.
> That said, dependency on rweakref even when disabled is a bug, can you
> post a full traceback?
> 
> Cheers,
> fijal
> 
> 
> 
> 
> 
> On Wed, Mar 18, 2015 at 2:01 AM, John Zhang <u5157779 at uds.anu.edu.au> wrote:
>> Hi all,
>>     I'm working on developing a MicroVM backend for PyPy. It's a virtual
>> machine under active research and development by my colleagues in ANU. It
>> aims to capture GC, threading and JIT in the virtual machine, and frees up
>> the burden of the language implementers.
>>
>>     Since MicroVM provides GC, I need to remove GC from the PyPy
>> interpreter. As I was trying to compile it with the following command:
>>     pypy $PYPY/rpython/bin/rpython \
>>           -O0 \
>>           --gc=none \
>>           --no-translation-rweakref \
>>           --annotate \
>>           --rtype \
>>           --translation-backendopt-none \
>>           $PYPY/pypy/goal/targetpypystandalone.py
>>     It gives off an error during annotation stage, saying that it's not able
>> to find a module called '_rweakref'.
>>     Does anyone know what the problem might be, and how one might go and
>> solve it?
>>
>>     Appreciate greatly,
>>     John Zhang
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150320/0877a940/attachment.sig>

From tbaldridge at gmail.com  Sat Mar 21 01:43:21 2015
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 20 Mar 2015 18:43:21 -0600
Subject: [pypy-dev] Specialization for app level types
Message-ID: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>

I'd like to add some optimization to app level types in Pixie. What I'm
thinking of is something like this (in app level PyPy code):


class Foo(object):
   def __init__(self, some_val):
     self._some_val = some_val
   def set_value(self, val):
     self._some_val = val

In a perfect world the JIT should be able to recognize that ._some_val is
only ever an int, and therefore store it unboxed in the instance of the
type, hopefully this would decrease pressure on the GC if ._some_val is
modified often. Also in a perfect world, the value of _some_val should be
auto promoted to an object if someone ever decides to set it to something
besides an int.

How would I go about coding this up in RPython? I can't seem to figure out
a way to do this without either bloating each instance of the type with an
array of object, an array of ints and an array of floats.

Currently app level objects in Pixie are just a wrapper around a object
array. The type then holds the lookups for name->slot_idx.

Thanks in advance.

Timothy Baldridge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150320/308fae83/attachment.html>

From arigo at tunes.org  Sat Mar 21 09:10:35 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 21 Mar 2015 09:10:35 +0100
Subject: [pypy-dev] Specialization for app level types
In-Reply-To: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
References: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
Message-ID: <CAMSv6X0joq4tn=LU5JEgQDHgKr01rvDp28YaMoiganS4GB3c9w@mail.gmail.com>

Hi Timothy,

On 21 March 2015 at 01:43, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> I'd like to add some optimization to app level types in Pixie. What I'm
> thinking of is something like this (in app level PyPy code):

There was some experiment in the PyPy branch
'type-specialized-instances' (in 2011-2013).  The idea there was to
use tagged pointers: every item in the array is either a regular
reference to a W_Root object, or a 31/63-bit integer stored with the
lowest bit set.  This must be specially supported by the GC, but
support is mostly there, and JIT support was in the branch
'int-tag-untag-as-operations'.

Nowadays, a better alternative is to store data in single array (say,
declared as array of integers), and have some items in there be GC
references while others are not.  A custom tracing hook needs to be
written.  This is similar to what we use for JITFRAME objects.  See
jitframe_trace() in rpython.jit.backend.llsupport.jitframe, which
enumerates items in the inlined jf_frame array based on
meta-information from the jf_gcmap field.

Note that although JITFRAME is obviously there for the JIT, it is
unclear how well the JIT itself supports RPython code doing this kind
of thing.  If you only access this kind of object from
@jit.dont_look_inside functions, then no problem, but this can't be
the case if you plan to use it for core language features.  The main
problem is that some operations that seem to be atomic in RPython
might not be any more while JITting.  We'd need to test exactly in
which order you need to update the information and the
meta-information, knowing that the GC may be randomly invoked in the
middle when tracing; and we'd need to check that instruction
reordering as done by the JIT optimizer don't create issues (they
probably do, e.g. if the update to the meta-information field is
delayed past a malloc).


A bient?t,

Armin.

From arigo at tunes.org  Sat Mar 21 09:39:08 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 21 Mar 2015 09:39:08 +0100
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <550B9118.6050800@anu.edu.au>
References: <5508C05B.6080609@uds.anu.edu.au>
 <CAK5idxTcBaxaam_0Uo3iH=5Cppjxs8QCt69CCY1jV-GsNaEXpg@mail.gmail.com>
 <550B9118.6050800@anu.edu.au>
Message-ID: <CAMSv6X3=N8SWSbuK+mYkDd3XbN3SsNqxAHYhqfG=UYdYwpXfEg@mail.gmail.com>

Hi Kunshan,

On 20 March 2015 at 04:16, Kunshan Wang <kunshan.wang at anu.edu.au> wrote:
> (...) Others (including PyPy) build a whole new VM,
> doing everything from scratch. There are many "high-performance VM"
> projects like PyPy (LuaJIT, v8, JavaScriptCore, HHVM to name a few), but
> the most important low-level parts (JIT, GC, ...) are not reused.

Nice to see a project that is similar to PyPy...  even though you seem
to have missed that point :-)  PyPy is based on RPython, whose goal is
also to provide a generic implementation substrate for any
interpreter-based language, by providing a general GC and a
meta-tracing JIT.  For that reason I would not put PyPy in the same
bag as LuaJIT, V8 and HHVM.  There are a number of very different
languages that already exist on top of the RPython architecture; PyPy
(the Python implementation) is only the most well-known.

Unlike Mu, the RPython toolchain was co-developed with PyPy, and so is
a bit more specifically suited for languages that share some
characteristics with Python (for example, multithreading is quite
limited).  Also, the separation between the two layers does not come
with (e.g.) a nice intermediate file format produced from the source
interpreter and fed to the RPython toolchain.

It is certainly interesting to see what kind of results you'd get by
feeding the "rtyped but not GC-transformed" instructions from RPython
to the Mu platform.  Again, I would recommend to start with smaller
things than the whole PyPy.  You can go about it in a test-driven way
by starting from, say, the simple examples in
rpython/translator/c/test/test_typed.py, and then pick the Prolog
interpreter at https://bitbucket.org/cfbolz/pyrolog/ as a much-smaller
first complete example.


A bient?t,

Armin.

From kunshan.wang at anu.edu.au  Sat Mar 21 11:43:50 2015
From: kunshan.wang at anu.edu.au (Kunshan Wang)
Date: Sat, 21 Mar 2015 21:43:50 +1100
Subject: [pypy-dev] Compiling PyPy interpreter without GC
In-Reply-To: <CAMSv6X3=N8SWSbuK+mYkDd3XbN3SsNqxAHYhqfG=UYdYwpXfEg@mail.gmail.com>
References: <5508C05B.6080609@uds.anu.edu.au>
 <CAK5idxTcBaxaam_0Uo3iH=5Cppjxs8QCt69CCY1jV-GsNaEXpg@mail.gmail.com>
 <550B9118.6050800@anu.edu.au>
 <CAMSv6X3=N8SWSbuK+mYkDd3XbN3SsNqxAHYhqfG=UYdYwpXfEg@mail.gmail.com>
Message-ID: <550D4B66.4070207@anu.edu.au>

Hi Armin,

Thank you for your interest.

On 21/03/15 19:39, Armin Rigo wrote:
> Hi Kunshan,
> 
> On 20 March 2015 at 04:16, Kunshan Wang <kunshan.wang at anu.edu.au> wrote:
>> (...) Others (including PyPy) build a whole new VM,
>> doing everything from scratch. There are many "high-performance VM"
>> projects like PyPy (LuaJIT, v8, JavaScriptCore, HHVM to name a few), but
>> the most important low-level parts (JIT, GC, ...) are not reused.
> 
> Nice to see a project that is similar to PyPy...  even though you seem
> to have missed that point :-)  PyPy is based on RPython, whose goal is
> also to provide a generic implementation substrate for any
> interpreter-based language, by providing a general GC and a
> meta-tracing JIT.  For that reason I would not put PyPy in the same
> bag as LuaJIT, V8 and HHVM.  There are a number of very different
> languages that already exist on top of the RPython architecture; PyPy
> (the Python implementation) is only the most well-known.

Indeed. The RPython framework has already hosted several languages. In
this sense, RPython is much more reuseable.

> 
> Unlike Mu, the RPython toolchain was co-developed with PyPy, and so is
> a bit more specifically suited for languages that share some
> characteristics with Python (for example, multithreading is quite
> limited).  Also, the separation between the two layers does not come
> with (e.g.) a nice intermediate file format produced from the source
> interpreter and fed to the RPython toolchain.

Mu is explicitly minimal, language-neutral, and has a much lower level,
although slightly higher than LLVM in term of supporting managed
languages. For example, Mu does not understand "interpreter" or
"meta-tracing". We already realised that it will be very hard to build a
full language implementation directly on top of such a low-level micro
VM, in which case the implementer will have to generate SSA-based CFG
with low-level types (though there are still reference types).

We think there should be several kinds of "libraries" for different
kinds of languages. Libraries can be pre-written Mu IR code snippets or
programs running in the client doing various transformations, but
libraries are *not* part of the micro VM (the micro VM is minimal to the
extent that if anything can be done outside the micro VM without
breaking the abstrctions, it will be). For example, a library that
supports concurrent languages should provide implementations of
locking/synchronisation primitives written in Mu IR (the Mu IR only
provides atomic memory access and a futex-like waiting mechanism). A
library that supports OOP should provide some abstractions of classes,
inheritence and virtual methods. There can also be libraries that
perform Mu IR to Mu IR optimisations, but the language implementer
should also perform other optimisations on a higher level.

From this point of view, RPython can play two roles: as a language
implemented on Mu, and a library to implement other languages.

John is working on getting RPython working on Mu. If this approach
eventually works (unlikely to happen soon since we don't have a lot of
man power), other languages implemented in RPython will work on Mu, too.

> 
> It is certainly interesting to see what kind of results you'd get by
> feeding the "rtyped but not GC-transformed" instructions from RPython
> to the Mu platform.

This is the first step of our plan. The interpreter written in RPython
will be ahead-of-time compiled to the Mu IR (rather than translated to C
as RPython currently does). Then the interpreter will run on Mu as other
ordinary Mu IR programs. After this step, there will be a Python
implementation as an interpreter running on Mu.

The next step is letting the PyPy-level tracing JIT compiler work. The
generated JIT compiler can be ahead-of-time compiled to Mu IR, too,
together with the interpreter. The JIT compiler, instead of generating
native machine code, will generate Mu IR code. (That is "Mu IR code
generating other Mu IR code", something meta-circular.)

In this Mu-based RPython implementation, two things will be different.

1. Mu has its own garbage collector and exception handling. This is why
we work just below the RTyper but before inserting GC and exception
codes. RPython no longer needs to implement the concrete garbage
collector by itself.

2. Both the RPython back-end compiler and the JIT compiler will
generates Mu IR instead. It reduces some platform dependency. The
foreign function interface (FFI) will still be platform-dependent.

Some researchers in the University of Massachusetts are planning to add
transactional memory to Mu. This will provide some low-level support of
the STM-based PyPy.

However, we are still far from having a "high-performance" Mu
implementation. Only until then could we know how exactly our approach
performs.

> Again, I would recommend to start with smaller
> things than the whole PyPy.  You can go about it in a test-driven way
> by starting from, say, the simple examples in
> rpython/translator/c/test/test_typed.py, and then pick the Prolog
> interpreter at https://bitbucket.org/cfbolz/pyrolog/ as a much-smaller
> first complete example.

Thank you for the advice.

> 
> 
> A bient?t,
> 
> Armin.
> 

Regards,
Kunshan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150321/e7f63f88/attachment-0001.sig>

From arigo at tunes.org  Sat Mar 21 18:08:14 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 21 Mar 2015 18:08:14 +0100
Subject: [pypy-dev] UTF8 string passing in cffi and PyPy internal string
	optimizations
In-Reply-To: <CAGmFidZ4k8wKFwS5VRzvgqSnf_X5zycaPKG7Fvwdb14C8JTSVQ@mail.gmail.com>
References: <7a7c329c37d052158073977b9d218c81@indus.uberspace.de>
 <550863EA.2060203@gmail.com>
 <CAGmFidZ4k8wKFwS5VRzvgqSnf_X5zycaPKG7Fvwdb14C8JTSVQ@mail.gmail.com>
Message-ID: <CAMSv6X30NctDdo_VTn=B10Tr5tVow4QyEdzNepcgL=4RJU8xHw@mail.gmail.com>

Hi,

On 18 March 2015 at 15:49, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> 2015-03-17 18:27 GMT+01:00 Eleytherios Stamatogiannakis <estama at gmail.com>:
>> Right now when PyPy receives a utf8 string (from a C function) it has to
>> do 2 copies:
>>
>> 1. convert the cdata string to a pypy byte string via ffi.string
>> 2. convert ffi.string to a unicode string
>>
>> When pypy sends a utf8 string it also does 2 copies:
>>
>> 1. convert pypy unicode string to utf8-encoded byte string
>> 2. copy the byte string into a cdata string.

The "easy" solution to reduce the number of copies is to have one
custom function that does both steps.  The more involved solution that
you suggest is, imho, breaking the way CFFI is supposed to work; see
below.

>> From what i understand, there is a cffi optimization dealing with windows
>> unicode (via set_unicode) where on windows platforms and when using the
>> native windows unicode strings, cffi avoids doing one of the copies in both
>> of above cases.
>>
>> On linux where the default unicode format for C libraries nowadays is
>> UTF8, there is no such optimization, so we have to do the two copies in all
>> string passing.

I think you're misunderstanding set_unicode() (or else I'm
misunderstanding what you say).  It's just a way to declare some
Windows-specific unicode types, like TCHAR, to be either "char" or
"wchar_t".  It doesn't enable or disable any optimization.

>> PyPy at some point was going towards using utf8 string internally, but i
>> don't know if this is still the plan or not.

PyPy might go there, at some point, but clearly not CPython.  We still
need a way to avoid the double copies there.

>> 1. If PyPy doesn't go towards using utf8 strings internally, maybe we need
>> some special C type that denotes that the string is utf8 and pypy/cffi
>> should do the conversion from-to it automatically. Something like "wchar_t"
>> in windows but denoting a utf8 string. CFFI can define a special type
>> ("__utf8char_t"?) for these strings.

What we really want is simply a variant of ffi.string() that accepts a
"char *" pointer, interprets it as utf-8, and returns a unicode
object; as well as another function that does the opposite.  If you're
interested in supporting the Windows case specially, then you want a
variant that would copy from/to a "TCHAR *" pointer on Windows.  This
is doable without any CFFI special types.

> This is a first step towards SWIG's typemaps:
> http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_nn4
>
> That's also something I wanted to have in another projects: automatic
> conversion to PYTHON_HANDLE, for example.
>
> But typemaps are a tough thing, and they would likely differ between CPython
> and PyPy.
> Armin, what do you think?

I think that typemaps are not the right solution to this problem :-)


A bient?t,

Armin.

From techtonik at gmail.com  Sun Mar 22 13:31:03 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 22 Mar 2015 15:31:03 +0300
Subject: [pypy-dev] Release 2.5.1 is close, please help test
In-Reply-To: <5505E07B.3000003@gmail.com>
References: <5505E07B.3000003@gmail.com>
Message-ID: <CAPkN8xKFYP91M3J9ueDjLLSLwRU2DQnu3XLKK087T=tp2hb9BQ@mail.gmail.com>

Looks like there is a regression in pypy-2.5.0 since 2.4.0 in virtualenv
https://github.com/pypa/virtualenv/issues/725

On Sun, Mar 15, 2015 at 10:41 PM, Matti Picus <matti.picus at gmail.com> wrote:
> The 2.5.1 release is almost ready. The release notice is here
> http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
> Note it has a broken link, the whatsnew is here instead
> http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
> This will be fixed when I merge the release branch back into default
> together with the official release
>
> The files ready for repackaging are here
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux.tar.bz2
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux64.tar.bz2
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-win32.zip
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-osx64.tar.bz2
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raspbian.tar.bz2
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raring.tar.bz2
> http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e2-linux-armhf-raring.tar.bz2
> https://bitbucket.org/pypy/pypy/get/release-2.5.1.zip
> https://bitbucket.org/pypy/pypy/get/release-2.5.1.tar.gz
>
> Could people please take a few minutes to test them out on their favorite
> platform?
> Also, any corrections, additions, or edits to the release notice would be
> welcome, especially a name instead of XXXXXX
> Perhaps from http://xkcd.com,  something to do with bromeliads
> http://www.bsi.org/brom_info/what.html, maybe we can draw a parallel between
> the spread of them and the spreading popularity of rpython and cffi:
>
> "Within the last hundred years, bromeliads have become more widely used as
> ornamental plants. Originally only found in royal botanical gardens or the
> private greenhouses of wealthy Europeans, their popularity has spread to the
> masses. Today bromeliads are more available to the enthusiast than ever
> before. New species are still being discovered and plant breeders are
> developing ever more stunning hybrids to choose from."
>
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev



-- 
anatoly t.

From gmludo at gmail.com  Sun Mar 22 14:45:25 2015
From: gmludo at gmail.com (Ludovic Gasc)
Date: Sun, 22 Mar 2015 14:45:25 +0100
Subject: [pypy-dev] How to help PyPy 3 ?
Message-ID: <CAON-fpGCXOdfn-uzBW-kf=H2bfk20O-ZbktRRRcxDBh3cVcwyg@mail.gmail.com>

Hi,

I want to try to help PyPy 3, especially to run AsyncIO on PyPy 3.

For now, I've tried to compile PyPy from 3.3 branch, and install AsyncIO.
I'd an issue with time.monotonic() and time.get_clock_info() in AsyncIO,
because it isn't implemented in PyPy 3.

I've replaced that by time.time() and hardcoded value of
time.get_clock_info() return, only to see until AsyncIO should work on PyPy
3.

I've launched several examples from AsyncIO documentation, it works, except
when I launch a TCP client:
https://docs.python.org/3/library/asyncio-protocol.html#tcp-echo-client-protocol
I've the same issue with aiotest: https://bitbucket.org/haypo/aiotest All
tests pass, except:
$ ./pypy-c -Wd ../tulip/run_aiotest.py
Run tests in debug mode
...............E
======================================================================
ERROR: test_tcp_hello (aiotest.test_network.NetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/lg/Documents/IDEA/pypy/site-packages/aiotest/test_network.py", line
78, in test_tcp_hello
    self.loop.run_until_complete(coro)
  File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py",
line 317, in run_until_complete
    return future.result()
  File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/futures.py",
line 275, in result
    raise self._exception
  File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/tasks.py", line
238, in _step
    result = next(coro)
  File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/coroutines.py",
line 79, in __next__
    return next(self.gen)
  File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py",
line 644, in create_connection
    sock, protocol_factory, ssl, server_hostname)
TypeError: '_SelectorSocketTransport' object is not iterable

----------------------------------------------------------------------
Ran 16 tests in 0.337s

FAILED (errors=1)

Finally, I've tested to launch the aiohttp server example:
https://github.com/KeepSafe/aiohttp/blob/master/examples/srv.py
The servers starts, but when I launch a HTTP request, I've a strange
stackstrace:
Error handling request
Traceback (most recent call last):
  File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py", line
240, in start
    yield from handler
  File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in
handle_request
    message.method, message.path, message.version))
AttributeError: 'str' object has no attribute 'method'
Traceback (most recent call last):
  File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py", line
240, in start
    yield from handler
  File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in
handle_request
    message.method, message.path, message.version))
AttributeError: 'str' object has no attribute 'method'

After debugging a little bit with pdb, I see that aiohttp.HttpRequestParser
parses correctly the request, but "message = yield from httpstream.read()"
in aiohttp.server at line 226 returns "GET" string instead of a
RawRequestMessage object.

I'm a total newbie about PyPy internals, implement a monotonic timer seems
to be over-complicated to me.
But, I should maybe help with tests and fix small issues like with
aiohttp.server if I've a clue to fix that.
Are you interested if I create issues on PyPy project for each problem, or
I will add only noise in PyPy project ?

BTW, to help PyPy project not only with my brain time, I've did a small
recurring donation for PyPy 3.

Regards.
--
Ludovic Gasc (GMLudo)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150322/04845fe6/attachment.html>

From w0mTea at 163.com  Sun Mar 22 15:31:31 2015
From: w0mTea at 163.com (w0mTea)
Date: Sun, 22 Mar 2015 22:31:31 +0800
Subject: [pypy-dev] GSoC 2015
Message-ID: <550ED243.80605@163.com>

Dear developers,

I'm a student interested in the idea of copy-on-write list slicing.

I noticed that on the PSF's GSoC wiki, students are suggested to fix a 
beginner-friendly bug, but after some searching, I eventually failed in 
finding some appropriate ones. May you help me about this?

Another question is about building pypy. I'm novice in pypy's 
implementation, so I decide to modify it's source code to help me 
understand it clearly. But it's really slow to build pypy. According to 
the pypy doc, I use this command to build it:

pypy rpython/bin/rpython --opt=jit pypy/goal/targetpypystandalone.py

It takes about an hour to complete on my computer. Then I modify a file, 
adding only one line. But rebuilding through this command also takes an 
hour. Is there any faster way to rebuild pypy with little modification?

Your answer and advice will be highly appreciated.

Thanks,
Yule Zhao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150322/96dd9a00/attachment.html>

From cfbolz at gmx.de  Sun Mar 22 16:21:40 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 22 Mar 2015 16:21:40 +0100
Subject: [pypy-dev] Specialization for app level types
In-Reply-To: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
References: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
Message-ID: <b28c4af8-f8a3-4438-bcc7-1959bd88c11a@email.android.com>

Hey Timothy, 

PyPy has no great solution for this yet. We have played with various approaches, Armin has described some of them. The simplest one is one that I've been trying recently on the typed-cells branch:

The basic idea is that for an attribute that stores an int, the attribute stores a special MutableIntCell instead. When that value is overwritten with another int, the value in the int cell is changed instead, thus there won't be an allocation. If another kind of object is stored in the attribute, the cell is replaced by a reference to that object. 

On reading an attribute it is then necessary to check whether the attribute value is such a cell. If it is, the value of the cell needs to be boxed into a regular W_IntObject.

Here's the relevant code on the branch (with some extra complication):

https://bitbucket.org/pypy/pypy/src/b193ed03acc0c93af48e70f77c6660a4c79aa3ae/pypy/objspace/std/mapdict.py?at=typed-cells#cl-305

This approach has the advantage that it's really simple to implement. It won't make instances that store integers smaller in memory, since the MutableIntCell needs as much memory as a regular W_IntObject. However, the GC pressure is reduced a lot when the field is written to often. 

Cheers, 

Carl Friedrich 


On March 21, 2015 1:43:21 AM GMT+01:00, Timothy Baldridge <tbaldridge at gmail.com> wrote:
>I'd like to add some optimization to app level types in Pixie. What I'm
>thinking of is something like this (in app level PyPy code):
>
>
>class Foo(object):
>   def __init__(self, some_val):
>     self._some_val = some_val
>   def set_value(self, val):
>     self._some_val = val
>
>In a perfect world the JIT should be able to recognize that ._some_val
>is
>only ever an int, and therefore store it unboxed in the instance of the
>type, hopefully this would decrease pressure on the GC if ._some_val is
>modified often. Also in a perfect world, the value of _some_val should
>be
>auto promoted to an object if someone ever decides to set it to
>something
>besides an int.
>
>How would I go about coding this up in RPython? I can't seem to figure
>out
>a way to do this without either bloating each instance of the type with
>an
>array of object, an array of ints and an array of floats.
>
>Currently app level objects in Pixie are just a wrapper around a object
>array. The type then holds the lookups for name->slot_idx.
>
>Thanks in advance.
>
>Timothy Baldridge
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150322/51a0e71f/attachment-0001.html>

From n210241048576 at gmail.com  Sun Mar 22 17:56:22 2015
From: n210241048576 at gmail.com (Robert Grosse)
Date: Sun, 22 Mar 2015 09:56:22 -0700
Subject: [pypy-dev] Specialization for app level types
In-Reply-To: <b28c4af8-f8a3-4438-bcc7-1959bd88c11a@email.android.com>
References: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
 <b28c4af8-f8a3-4438-bcc7-1959bd88c11a@email.android.com>
Message-ID: <CALJjGisXy+TS18CP2uPXGJ+6-oKkO7Gy-G4+9JvErLcLnDJjQA@mail.gmail.com>

Since lists are specialized for storing primative ints, would it be
possible to do something similar with user defined types? Assuming that
most instances of the same type will either store or not store an int in
the same attribute, you could optimistically specialize it initially and
fallback to the normal code if this invariant is broken.

On Sun, Mar 22, 2015 at 8:21 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> Hey Timothy,
>
> PyPy has no great solution for this yet. We have played with various
> approaches, Armin has described some of them. The simplest one is one that
> I've been trying recently on the typed-cells branch:
>
> The basic idea is that for an attribute that stores an int, the attribute
> stores a special MutableIntCell instead. When that value is overwritten
> with another int, the value in the int cell is changed instead, thus there
> won't be an allocation. If another kind of object is stored in the
> attribute, the cell is replaced by a reference to that object.
>
> On reading an attribute it is then necessary to check whether the
> attribute value is such a cell. If it is, the value of the cell needs to be
> boxed into a regular W_IntObject.
>
> Here's the relevant code on the branch (with some extra complication):
>
>
> https://bitbucket.org/pypy/pypy/src/b193ed03acc0c93af48e70f77c6660a4c79aa3ae/pypy/objspace/std/mapdict.py?at=typed-cells#cl-305
>
> This approach has the advantage that it's really simple to implement. It
> won't make instances that store integers smaller in memory, since the
> MutableIntCell needs as much memory as a regular W_IntObject. However, the
> GC pressure is reduced a lot when the field is written to often.
>
> Cheers,
>
> Carl Friedrich
>
>
> On March 21, 2015 1:43:21 AM GMT+01:00, Timothy Baldridge <
> tbaldridge at gmail.com> wrote:
>
>> I'd like to add some optimization to app level types in Pixie. What I'm
>> thinking of is something like this (in app level PyPy code):
>>
>>
>> class Foo(object):
>>    def __init__(self, some_val):
>>      self._some_val = some_val
>>    def set_value(self, val):
>>      self._some_val = val
>>
>> In a perfect world the JIT should be able to recognize that ._some_val is
>> only ever an int, and therefore store it unboxed in the instance of the
>> type, hopefully this would decrease pressure on the GC if ._some_val is
>> modified often. Also in a perfect world, the value of _some_val should be
>> auto promoted to an object if someone ever decides to set it to something
>> besides an int.
>>
>> How would I go about coding this up in RPython? I can't seem to figure
>> out a way to do this without either bloating each instance of the type with
>> an array of object, an array of ints and an array of floats.
>>
>> Currently app level objects in Pixie are just a wrapper around a object
>> array. The type then holds the lookups for name->slot_idx.
>>
>> Thanks in advance.
>>
>> Timothy Baldridge
>>
>>
>> ------------------------------
>>
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150322/eb426e7c/attachment.html>

From arigo at tunes.org  Sun Mar 22 17:57:56 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 22 Mar 2015 17:57:56 +0100
Subject: [pypy-dev] Release 2.5.1 is close, please help test
In-Reply-To: <CAPkN8xKFYP91M3J9ueDjLLSLwRU2DQnu3XLKK087T=tp2hb9BQ@mail.gmail.com>
References: <5505E07B.3000003@gmail.com>
 <CAPkN8xKFYP91M3J9ueDjLLSLwRU2DQnu3XLKK087T=tp2hb9BQ@mail.gmail.com>
Message-ID: <CAMSv6X2UfsgxnaLJVydtEWz7mKkPpOxACsb4fyXpGQE5MmRbcg@mail.gmail.com>

Hi,

On 22 March 2015 at 13:31, anatoly techtonik <techtonik at gmail.com> wrote:
> Looks like there is a regression in pypy-2.5.0 since 2.4.0 in virtualenv
> https://github.com/pypa/virtualenv/issues/725

That's because pypy comes with a shared library since 2.5.  I'll leave
it to virtualenv people to decide if they need to fix the test or if
there is a more serious problem for them.  At least virtualenv seems
to work in practice.


Armin

From fijall at gmail.com  Sun Mar 22 18:32:21 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 22 Mar 2015 19:32:21 +0200
Subject: [pypy-dev] GSoC 2015
In-Reply-To: <550ED243.80605@163.com>
References: <550ED243.80605@163.com>
Message-ID: <CAK5idxSpQ5okpGer5=sOOy0S+9_yncG4F0oxbsOzOmRpRSWt1g@mail.gmail.com>

On Sun, Mar 22, 2015 at 4:31 PM, w0mTea <w0mTea at 163.com> wrote:
> Dear developers,
>
> I'm a student interested in the idea of copy-on-write list slicing.
>
> I noticed that on the PSF's GSoC wiki, students are suggested to fix a
> beginner-friendly bug, but after some searching, I eventually failed in
> finding some appropriate ones. May you help me about this?
>
> Another question is about building pypy. I'm novice in pypy's
> implementation, so I decide to modify it's source code to help me understand
> it clearly. But it's really slow to build pypy. According to the pypy doc, I
> use this command to build it:
>
> pypy rpython/bin/rpython --opt=jit pypy/goal/targetpypystandalone.py
>
> It takes about an hour to complete on my computer. Then I modify a file,
> adding only one line. But rebuilding through this command also takes an
> hour. Is there any faster way to rebuild pypy with little modification?

You don't rebuild pypy for each change. Instead you write tests (which
we have in abundance) that are in various test/ subdirectories that
can be run untranslated.

Cheers,
fijal

From cfbolz at gmx.de  Sun Mar 22 18:51:52 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 22 Mar 2015 18:51:52 +0100
Subject: [pypy-dev] Specialization for app level types
In-Reply-To: <CALJjGisXy+TS18CP2uPXGJ+6-oKkO7Gy-G4+9JvErLcLnDJjQA@mail.gmail.com>
References: <CAL36E+sPd_-oVDAmeA-yyA34MFWo+6TzHvhxPEqnJ=pENZOzAA@mail.gmail.com>
 <b28c4af8-f8a3-4438-bcc7-1959bd88c11a@email.android.com>
 <CALJjGisXy+TS18CP2uPXGJ+6-oKkO7Gy-G4+9JvErLcLnDJjQA@mail.gmail.com>
Message-ID: <07c0f5ba-73b0-44fb-92c6-3b5c8824df25@email.android.com>



On March 22, 2015 5:56:22 PM GMT+01:00, Robert Grosse <n210241048576 at gmail.com> wrote:
>Since lists are specialized for storing primative ints, would it be
>possible to do something similar with user defined types? Assuming that
>most instances of the same type will either store or not store an int
>in
>the same attribute, you could optimistically specialize it initially
>and
>fallback to the normal code if this invariant is broken.

Hi Robert, 

Note that the whole thread *is* about user defined types. The specialization you describe is possible, but entirely non-trivial, with various trade offs involved. This paper describes an approach to solve it:

http://www.chrisseaton.com/rubytruffle/pppj14-om/pppj14-om.pdf

Cheers, 

Carl Friedrich 

From heng at cantab.net  Sun Mar 22 21:02:08 2015
From: heng at cantab.net (Henry Gomersall)
Date: Sun, 22 Mar 2015 20:02:08 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
Message-ID: <550F1FC0.4000502@cantab.net>

I'm looking at using PyPy's flow object space for an experimental 
converter for MyHDL (http://www.myhdl.org/), a Python library for 
representing HDL (i.e. hardware) models. By conversion, I mean 
converting the MyHDL model that represents the hardware into either 
Verilog or VHDL that downstream tools will support. Currently, there is 
a converter that works very well in many situations, but there are 
substantial limitations on the code that can be converted (much greater 
restrictions than RPython imposes), as well as somewhat frustrating 
corner cases.

It strikes me that much of the heavy lifting of the conversion problem 
can be handled by the PyPy stack.

My question then regards the following. MyHDL represents certain low 
level structures as python objects. For example, there is a notion of a 
signal, represented by a Signal object, that has a one to one mapping to 
the target HDL language. All the attributes of the Signal object 
describe how it should be converted. So, during annotation, the Signal 
object should be maintained as a base type, rather than burying deeper 
into the object to try and infer more about its type (which invariably 
breaks things due to RPython non-conformity). There are probably a few 
other types (though not many) that should be handled similarly.

How does one instruct the translator to do this? Is it a case of writing 
a custom TranslationDriver to handle the custom types?

Thanks for any help,

Henry

From ronan.lamy at gmail.com  Sun Mar 22 22:45:54 2015
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Sun, 22 Mar 2015 21:45:54 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <550F1FC0.4000502@cantab.net>
References: <550F1FC0.4000502@cantab.net>
Message-ID: <550F3812.8020509@gmail.com>

Le 22/03/15 20:02, Henry Gomersall a ?crit :
> I'm looking at using PyPy's flow object space for an experimental
> converter for MyHDL (http://www.myhdl.org/), a Python library for
> representing HDL (i.e. hardware) models. By conversion, I mean
> converting the MyHDL model that represents the hardware into either
> Verilog or VHDL that downstream tools will support. Currently, there is
> a converter that works very well in many situations, but there are
> substantial limitations on the code that can be converted (much greater
> restrictions than RPython imposes), as well as somewhat frustrating
> corner cases.
>
> It strikes me that much of the heavy lifting of the conversion problem
> can be handled by the PyPy stack.
>
> My question then regards the following. MyHDL represents certain low
> level structures as python objects. For example, there is a notion of a
> signal, represented by a Signal object, that has a one to one mapping to
> the target HDL language. All the attributes of the Signal object
> describe how it should be converted. So, during annotation, the Signal
> object should be maintained as a base type, rather than burying deeper
> into the object to try and infer more about its type (which invariably
> breaks things due to RPython non-conformity). There are probably a few
> other types (though not many) that should be handled similarly.
>
> How does one instruct the translator to do this? Is it a case of writing
> a custom TranslationDriver to handle the custom types?

No, you simply need to register your Python types with the translator 
and describe their properties by subclassing some internal classes.
The interface is a bit complicated and requires some understanding of 
RPython's inner workings, but it works. rply has a good example of that: 
https://github.com/alex/rply/blob/master/rply/lexergenerator.py 
(specifically, the part that is guarded by 'if rpython:')

Here's how it works:
The ExtRegistryEntry tells the annotator to special-case Rule objects by 
annotating them as SomeRule(). SomeRule describes the annotator-level 
properties of Rule objects while RuleRepr tells the rtyper how to 
translate them to low-level (C-like) objects.

In your case, you definitely need an ExtRegistryEntry and a SomeSignal, 
but what to put in SignalRepr and whether you even need it depends on 
how you intend the Verilog/VHDL backend to work.

From fijall at gmail.com  Mon Mar 23 09:33:20 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 23 Mar 2015 10:33:20 +0200
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <550F1FC0.4000502@cantab.net>
References: <550F1FC0.4000502@cantab.net>
Message-ID: <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>

Hi Henry.

I must say we had quite a bit of a discussion and it seems we did not
understand what are you trying to achieve. What is the goal of what
you're doing? Translating MyHDL (or verilog) to rpython and compiling
it? Something else? Writing the converter itself in RPython?

Please explain.

Cheers,
fijal


On Sun, Mar 22, 2015 at 10:02 PM, Henry Gomersall <heng at cantab.net> wrote:
> I'm looking at using PyPy's flow object space for an experimental converter
> for MyHDL (http://www.myhdl.org/), a Python library for representing HDL
> (i.e. hardware) models. By conversion, I mean converting the MyHDL model
> that represents the hardware into either Verilog or VHDL that downstream
> tools will support. Currently, there is a converter that works very well in
> many situations, but there are substantial limitations on the code that can
> be converted (much greater restrictions than RPython imposes), as well as
> somewhat frustrating corner cases.
>
> It strikes me that much of the heavy lifting of the conversion problem can
> be handled by the PyPy stack.
>
> My question then regards the following. MyHDL represents certain low level
> structures as python objects. For example, there is a notion of a signal,
> represented by a Signal object, that has a one to one mapping to the target
> HDL language. All the attributes of the Signal object describe how it should
> be converted. So, during annotation, the Signal object should be maintained
> as a base type, rather than burying deeper into the object to try and infer
> more about its type (which invariably breaks things due to RPython
> non-conformity). There are probably a few other types (though not many) that
> should be handled similarly.
>
> How does one instruct the translator to do this? Is it a case of writing a
> custom TranslationDriver to handle the custom types?
>
> Thanks for any help,
>
> Henry
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From heng at cantab.net  Mon Mar 23 09:36:53 2015
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 23 Mar 2015 08:36:53 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <550F3812.8020509@gmail.com>
References: <550F1FC0.4000502@cantab.net> <550F3812.8020509@gmail.com>
Message-ID: <550FD0A5.3030102@cantab.net>

On 22/03/15 21:45, Ronan Lamy wrote:
>>
>> My question then regards the following. MyHDL represents certain low
>> level structures as python objects. For example, there is a notion of a
>> signal, represented by a Signal object, that has a one to one mapping to
>> the target HDL language. All the attributes of the Signal object
>> describe how it should be converted. So, during annotation, the Signal
>> object should be maintained as a base type, rather than burying deeper
>> into the object to try and infer more about its type (which invariably
>> breaks things due to RPython non-conformity). There are probably a few
>> other types (though not many) that should be handled similarly.
>>
>> How does one instruct the translator to do this? Is it a case of writing
>> a custom TranslationDriver to handle the custom types?
>
> No, you simply need to register your Python types with the translator 
> and describe their properties by subclassing some internal classes.
> The interface is a bit complicated and requires some understanding of 
> RPython's inner workings, but it works. rply has a good example of 
> that: https://github.com/alex/rply/blob/master/rply/lexergenerator.py 
> (specifically, the part that is guarded by 'if rpython:')
>
> Here's how it works:
> The ExtRegistryEntry tells the annotator to special-case Rule objects 
> by annotating them as SomeRule(). SomeRule describes the 
> annotator-level properties of Rule objects while RuleRepr tells the 
> rtyper how to translate them to low-level (C-like) objects.
>
> In your case, you definitely need an ExtRegistryEntry and a 
> SomeSignal, but what to put in SignalRepr and whether you even need it 
> depends on how you intend the Verilog/VHDL backend to work. 

Ah, that's fantastic :)

So as I understand it, ExtRegistryEntry provides some metaclass magic 
that auto registers the relevant type as a special case, with the 
decision made by the object returned from compute_annotation() ?

Is there some documentation on this aspect of the internals from 
RPython? There are obviously lots of questions that would be much easier 
to get with some docs (ordering? default actions? what the default 
methods are doing?).

Thanks!
Henry

From heng at cantab.net  Mon Mar 23 09:53:27 2015
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 23 Mar 2015 08:53:27 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
Message-ID: <550FD487.6030804@cantab.net>

On 23/03/15 08:33, Maciej Fijalkowski wrote:
> I must say we had quite a bit of a discussion and it seems we did not
> understand what are you trying to achieve. What is the goal of what
> you're doing? Translating MyHDL (or verilog) to rpython and compiling
> it? Something else? Writing the converter itself in RPython?
Ah, my apologies for not being clear. The goal is to translate RPython 
(MyHDL) to VHDL or Verilog.

Essentially, the valid _convertible_ MyHDL would have the constraint of 
being restricted to RPython. This is essentially a mix of hardware 
specific types, some simple types and control flow (it's inside a simple 
generator, and the function is wrapped in a decorator to provide some 
hardware specific semantics, but I suspect this is not helpful detail).

So, we have a series of generators in which the generator code, and all 
types referenced inside the generator are either written in RPython, or 
are conceptually a base type (that is, a type that should not be 
burrowed into and is a direct representation of a hardware feature), 
that describes the activity of hardware.

The intention would be to use PyPy to translate these RPython blocks 
into VHDL or Verilog.

There is support code that is normal python that surrounds these 
generators which is normal Python, but that is just essentially to set 
up the name space and attributes of the generator, which is then 
returned as the active object.

Does that all make more sense?

Cheers,

Henry

From w0mTea at 163.com  Mon Mar 23 10:17:40 2015
From: w0mTea at 163.com (w0mTea)
Date: Mon, 23 Mar 2015 17:17:40 +0800
Subject: [pypy-dev] GSoC 2015
In-Reply-To: <CAK5idxSpQ5okpGer5=sOOy0S+9_yncG4F0oxbsOzOmRpRSWt1g@mail.gmail.com>
References: <550ED243.80605@163.com>
 <CAK5idxSpQ5okpGer5=sOOy0S+9_yncG4F0oxbsOzOmRpRSWt1g@mail.gmail.com>
Message-ID: <550FDA34.5030601@163.com>

Thank you very much, Fijal . I read docs of pypy today and I found there 
is detailed introduction about test. Sorry for that careless question.

And how about the bug? I don't know which one is beginner-friendly. Or 
may I submit a little piece of code about the idea instead of  a bug-fixing?

Also, I find that a developer has already written a COW list slice and 
pulled it, so maybe it's better for me to choose another idea. After 
some consideration, I decide to take the Unicode Optimization. So, how 
can I contact the potential mentor of this idea?

Regards,
Yule

On 2015, Mar 23 01:32, Maciej Fijalkowski wrote:
> On Sun, Mar 22, 2015 at 4:31 PM, w0mTea <w0mTea at 163.com> wrote:
>> Dear developers,
>>
>> I'm a student interested in the idea of copy-on-write list slicing.
>>
>> I noticed that on the PSF's GSoC wiki, students are suggested to fix a
>> beginner-friendly bug, but after some searching, I eventually failed in
>> finding some appropriate ones. May you help me about this?
>>
>> Another question is about building pypy. I'm novice in pypy's
>> implementation, so I decide to modify it's source code to help me understand
>> it clearly. But it's really slow to build pypy. According to the pypy doc, I
>> use this command to build it:
>>
>> pypy rpython/bin/rpython --opt=jit pypy/goal/targetpypystandalone.py
>>
>> It takes about an hour to complete on my computer. Then I modify a file,
>> adding only one line. But rebuilding through this command also takes an
>> hour. Is there any faster way to rebuild pypy with little modification?
> You don't rebuild pypy for each change. Instead you write tests (which
> we have in abundance) that are in various test/ subdirectories that
> can be run untranslated.
>
> Cheers,
> fijal



From fijall at gmail.com  Mon Mar 23 10:37:10 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 23 Mar 2015 11:37:10 +0200
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <550FD487.6030804@cantab.net>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
Message-ID: <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>

On Mon, Mar 23, 2015 at 10:53 AM, Henry Gomersall <heng at cantab.net> wrote:
> On 23/03/15 08:33, Maciej Fijalkowski wrote:
>>
>> I must say we had quite a bit of a discussion and it seems we did not
>> understand what are you trying to achieve. What is the goal of what
>> you're doing? Translating MyHDL (or verilog) to rpython and compiling
>> it? Something else? Writing the converter itself in RPython?
>
> Ah, my apologies for not being clear. The goal is to translate RPython
> (MyHDL) to VHDL or Verilog.
>
> Essentially, the valid _convertible_ MyHDL would have the constraint of
> being restricted to RPython. This is essentially a mix of hardware specific
> types, some simple types and control flow (it's inside a simple generator,
> and the function is wrapped in a decorator to provide some hardware specific
> semantics, but I suspect this is not helpful detail).
>
> So, we have a series of generators in which the generator code, and all
> types referenced inside the generator are either written in RPython, or are
> conceptually a base type (that is, a type that should not be burrowed into
> and is a direct representation of a hardware feature), that describes the
> activity of hardware.
>
> The intention would be to use PyPy to translate these RPython blocks into
> VHDL or Verilog.
>
> There is support code that is normal python that surrounds these generators
> which is normal Python, but that is just essentially to set up the name
> space and attributes of the generator, which is then returned as the active
> object.
>
> Does that all make more sense?
>
> Cheers,
>
> Henry

RPython does not emit verilog or vhdl. How are you going to address
that problem? Also, what precisely are you achieving by using RPython
toolchain?

From heng at cantab.net  Mon Mar 23 11:38:54 2015
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 23 Mar 2015 10:38:54 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
Message-ID: <550FED3E.5060405@cantab.net>

On 23/03/15 09:37, Maciej Fijalkowski wrote:
> On Mon, Mar 23, 2015 at 10:53 AM, Henry Gomersall<heng at cantab.net>  wrote:
>> >On 23/03/15 08:33, Maciej Fijalkowski wrote:
>>> >>
>>> >>I must say we had quite a bit of a discussion and it seems we did not
>>> >>understand what are you trying to achieve. What is the goal of what
>>> >>you're doing? Translating MyHDL (or verilog) to rpython and compiling
>>> >>it? Something else? Writing the converter itself in RPython?
>> >
>> >Ah, my apologies for not being clear. The goal is to translate RPython
>> >(MyHDL) to VHDL or Verilog.
>> >
>> >Essentially, the valid_convertible_  MyHDL would have the constraint of
>> >being restricted to RPython. This is essentially a mix of hardware specific
>> >types, some simple types and control flow (it's inside a simple generator,
>> >and the function is wrapped in a decorator to provide some hardware specific
>> >semantics, but I suspect this is not helpful detail).
>> >
>> >So, we have a series of generators in which the generator code, and all
>> >types referenced inside the generator are either written in RPython, or are
>> >conceptually a base type (that is, a type that should not be burrowed into
>> >and is a direct representation of a hardware feature), that describes the
>> >activity of hardware.
>> >
>> >The intention would be to use PyPy to translate these RPython blocks into
>> >VHDL or Verilog.
>> >
>> >There is support code that is normal python that surrounds these generators
>> >which is normal Python, but that is just essentially to set up the name
>> >space and attributes of the generator, which is then returned as the active
>> >object.
>> >
>> >Does that all make more sense?
>> >
>> >Cheers,
>> >
>> >Henry
> RPython does not emit verilog or vhdl. How are you going to address
> that problem? Also, what precisely are you achieving by using RPython
> toolchain?

No it doesn't. The VHDL/Verilog parts need to be written.

There is a task of inferring flow from a restricted subset of Python, 
from which V* can be generated. Any solution to the problem looks 
remarkably like what I understand is being done by RPython. Would you 
suggest a better strategy?

The current solution is built on traversing the AST and special casing 
certain inferred structures. The IR is never explicit and so it's hard 
to do much with the flow graph. The concept is that by building on the 
RPython flow object space as an intermediate representation, much more 
flexibility can be added to the convertible code (e.g. inferring 
translated structures from the IR, rather than from the code).

Cheers,

Henry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150323/c4fe51c1/attachment.html>

From mount.sarah at gmail.com  Mon Mar 23 13:10:42 2015
From: mount.sarah at gmail.com (Sarah Mount)
Date: Mon, 23 Mar 2015 12:10:42 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <550FED3E.5060405@cantab.net>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
Message-ID: <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>

As it happens this is of interest to be, because I may have a use for MyHDL
or something similar in a year or so. However, from this thread I'm a
little confused about where the RPython toolchain would fit in. Sorry if
I'm completely off-base here...

Presumably you want to keep your Python (2?) front end -- you wouldn't want
application programmers using RPython directly, it is really tricky to
write in as it's designed for a very specific use case which is very
different from generic Python programming.

I would guess that the reason for porting MyHDL to an RPython toolchain is
that you don't just produce a flat Verilog file, but you also have a
runtime system which simulates the Verilog (or the higher level Python
description)? This sounds very interesting, I'm not sure how helpful the
RPython GCs and JIT would be in your case, but it would be interesting to
hear where you think a performance improvement can be made.

You might have seen this paper on the PyPy status blog, on instruction set
simulators:
http://csl.cornell.edu/~cbatten/pdfs/lockhart-pydgin-ispass2015.pdf I don't
think it is quite the same as your use case, but there are some interesting
ideas in there.

Cheers,

Sarah

On Mon, Mar 23, 2015 at 10:38 AM, Henry Gomersall <heng at cantab.net> wrote:

>  On 23/03/15 09:37, Maciej Fijalkowski wrote:
>
>  On Mon, Mar 23, 2015 at 10:53 AM, Henry Gomersall <heng at cantab.net> <heng at cantab.net> wrote:
>
>  > On 23/03/15 08:33, Maciej Fijalkowski wrote:
>
>  >>>> I must say we had quite a bit of a discussion and it seems we did not>> understand what are you trying to achieve. What is the goal of what>> you're doing? Translating MyHDL (or verilog) to rpython and compiling>> it? Something else? Writing the converter itself in RPython?
>
>  >> Ah, my apologies for not being clear. The goal is to translate RPython> (MyHDL) to VHDL or Verilog.>> Essentially, the valid _convertible_ MyHDL would have the constraint of> being restricted to RPython. This is essentially a mix of hardware specific> types, some simple types and control flow (it's inside a simple generator,> and the function is wrapped in a decorator to provide some hardware specific> semantics, but I suspect this is not helpful detail).>> So, we have a series of generators in which the generator code, and all> types referenced inside the generator are either written in RPython, or are> conceptually a base type (that is, a type that should not be burrowed into> and is a direct representation of a hardware feature), that describes the> activity of hardware.>> The intention would be to use PyPy to translate these RPython blocks into> VHDL or Verilog.>> There is support code that is normal python that surrounds these generators> which is normal Python, but that is just essentially to set up the name> space and attributes of the generator, which is then returned as the active> object.>> Does that all make more sense?>> Cheers,>> Henry
>
>  RPython does not emit verilog or vhdl. How are you going to address
> that problem? Also, what precisely are you achieving by using RPython
> toolchain?
>
>
> No it doesn't. The VHDL/Verilog parts need to be written.
>
> There is a task of inferring flow from a restricted subset of Python, from
> which V* can be generated. Any solution to the problem looks remarkably
> like what I understand is being done by RPython. Would you suggest a better
> strategy?
>
> The current solution is built on traversing the AST and special casing
> certain inferred structures. The IR is never explicit and so it's hard to
> do much with the flow graph. The concept is that by building on the RPython
> flow object space as an intermediate representation, much more flexibility
> can be added to the convertible code (e.g. inferring translated structures
> from the IR, rather than from the code).
>
> Cheers,
>
> Henry
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
website:  http://www.snim2.org/
twitter: @snim2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150323/6a5829ab/attachment.html>

From heng at cantab.net  Mon Mar 23 13:41:41 2015
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 23 Mar 2015 12:41:41 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
Message-ID: <55100A05.3020503@cantab.net>

On 23/03/15 12:10, Sarah Mount wrote:
> As it happens this is of interest to be, because I may have a use for 
> MyHDL or something similar in a year or so. However, from this thread 
> I'm a little confused about where the RPython toolchain would fit in. 
> Sorry if I'm completely off-base here...
>
> Presumably you want to keep your Python (2?) front end -- you wouldn't 
> want application programmers using RPython directly, it is really 
> tricky to write in as it's designed for a very specific use case which 
> is very different from generic Python programming.

Well, users currently _are_ restricted to a small subset of 
"convertible" python. I would suggest this is actually more restrictive 
than RPython. I suppose the question I have is what happens at the 
boundary with RPython - can one construct an object that is crosses the 
boundary, with some RPython only methods and some general Python methods?

It might be that code that is RPython compliant cannot also use 
necessary MyHDL constructs, though there may well be workarounds (it 
would almost certainly not be the case that the code is too flexible).

It's worth noting the MyHDL is several overlapping things. It's first 
and foremost a library for representing hardware concurrency. If a 
developer is restricted to only a small subset of valid Python and 
types, the blocks that are described using MyHDL constructs can be 
converted into VHDL or Verilog. It allows very neat workflow - one has 
the full power of Python for defining behaviour, and then one writes the 
convertible code (which is verified against the behavioural model), and 
then one cosimulates the converted code, verifying that is doing the 
write thing (using a Verilog/VHDL simulator).

>
> I would guess that the reason for porting MyHDL to an RPython 
> toolchain is that you don't just produce a flat Verilog file, but you 
> also have a runtime system which simulates the Verilog (or the higher 
> level Python description)? This sounds very interesting, I'm not sure 
> how helpful the RPython GCs and JIT would be in your case, but it 
> would be interesting to hear where you think a performance improvement 
> can be made.
>
Well, potentially, but the big win is in being allowed a broader range 
of convertible constructs. For example, there is currently no way to 
handle general iterables (only loops of the form `for i in range(N):` 
are allowed). Clearly, this is very restrictive for writing nice, 
expressive code.

Stepping back a minute. The ultimate goal IMO would be a tool that takes 
a MyHDL instance block (that is, that represents the function of a 
hardware block), along with the associated static namespace, and 
converts into something that downstream tools can understand (VHDL or 
Verilog), with as much expressive power in the code as makes sense given 
the target restrictions.

All these things are possible in the current approach, but I've been 
wondering if the similar functionality of similar tools can be used to 
do some of the heavy lifting. The current approach is essentially trying 
to implement a "MyHDL restricted" python to VHDL/Verilog translator.

> You might have seen this paper on the PyPy status blog, on instruction 
> set simulators: 
> http://csl.cornell.edu/~cbatten/pdfs/lockhart-pydgin-ispass2015.pdf 
> <http://csl.cornell.edu/%7Ecbatten/pdfs/lockhart-pydgin-ispass2015.pdf> I 
> don't think it is quite the same as your use case, but there are some 
> interesting ideas in there.

That's interesting. I had seen it but not really noticed it's relevance.

Cheers,

Henry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150323/9f7e84b0/attachment.html>

From mount.sarah at gmail.com  Mon Mar 23 13:50:59 2015
From: mount.sarah at gmail.com (Sarah Mount)
Date: Mon, 23 Mar 2015 12:50:59 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <55100A05.3020503@cantab.net>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
Message-ID: <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>

Hi,

On Mon, Mar 23, 2015 at 12:41 PM, Henry Gomersall <heng at cantab.net> wrote:
> On 23/03/15 12:10, Sarah Mount wrote:
>

<snip>

> Well, users currently _are_ restricted to a small subset of "convertible"
> python. I would suggest this is actually more restrictive than RPython. I
> suppose the question I have is what happens at the boundary with RPython -
> can one construct an object that is crosses the boundary, with some RPython
> only methods and some general Python methods?

Ah, my mistake.

> It's worth noting the MyHDL is several overlapping things. It's first and
> foremost a library for representing hardware concurrency. If a developer is
> restricted to only a small subset of valid Python and types, the blocks that
> are described using MyHDL constructs can be converted into VHDL or Verilog.
> It allows very neat workflow - one has the full power of Python for defining
> behaviour, and then one writes the convertible code (which is verified
> against the behavioural model), and then one cosimulates the converted code,
> verifying that is doing the write thing (using a Verilog/VHDL simulator).
>

Thanks, that has certainly clarified things for me.

> > I would guess that the reason for porting MyHDL to an RPython toolchain is
> > that you don't just produce a flat Verilog file, but you also have a runtime
> > system which simulates the Verilog (or the higher level Python description)?
> > This sounds very interesting, I'm not sure how helpful the RPython GCs and
> > JIT would be in your case, but it would be interesting to hear where you
> > think a performance improvement can be made.
>
> Well, potentially, but the big win is in being allowed a broader range of
> convertible constructs. For example, there is currently no way to handle
> general iterables (only loops of the form `for i in range(N):` are allowed).
> Clearly, this is very restrictive for writing nice, expressive code.
>
> Stepping back a minute. The ultimate goal IMO would be a tool that takes a
> MyHDL instance block (that is, that represents the function of a hardware
> block), along with the associated static namespace, and converts into
> something that downstream tools can understand (VHDL or Verilog), with as
> much expressive power in the code as makes sense given the target
> restrictions.
>

Hmm. So, what I understand from this is that your current front-end
implements a very small subset of Python, you have noticed that
RPython implements a slightly larger subset of Python, so you want to
replace your front-end and maybe some internals with the RPython
equivalents? I'm not sure how well this will work. For one thing,
RPython is intended to be translated to a native format (i.e. you take
a description of an interpreter or VM in RPython and after a very long
compile you get a native executable that is your interpreter, with any
RPython internals, such as JITs and GCs, included). I'm not sure if
this is a win for you or not, because I'm not sure if an RPython
front-end can really be made to fit a MyHDL back-end, without just
re-writing the whole thing as an interpreter.

Cheers,

Sarah

-- 
Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
website:  http://www.snim2.org/
twitter: @snim2

From heng at cantab.net  Mon Mar 23 14:00:17 2015
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 23 Mar 2015 13:00:17 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
 <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
Message-ID: <55100E61.2000508@cantab.net>

On 23/03/15 12:50, Sarah Mount wrote:
<snip>
>> Well, potentially, but the big win is in being allowed a broader range of
>> >convertible constructs. For example, there is currently no way to handle
>> >general iterables (only loops of the form `for i in range(N):` are allowed).
>> >Clearly, this is very restrictive for writing nice, expressive code.
>> >
>> >Stepping back a minute. The ultimate goal IMO would be a tool that takes a
>> >MyHDL instance block (that is, that represents the function of a hardware
>> >block), along with the associated static namespace, and converts into
>> >something that downstream tools can understand (VHDL or Verilog), with as
>> >much expressive power in the code as makes sense given the target
>> >restrictions.
>> >
> Hmm. So, what I understand from this is that your current front-end
> implements a very small subset of Python, you have noticed that
> RPython implements a slightly larger subset of Python, so you want to
> replace your front-end and maybe some internals with the RPython
> equivalents? I'm not sure how well this will work. For one thing,
> RPython is intended to be translated to a native format (i.e. you take
> a description of an interpreter or VM in RPython and after a very long
> compile you get a native executable that is your interpreter, with any
> RPython internals, such as JITs and GCs, included). I'm not sure if
> this is a win for you or not, because I'm not sure if an RPython
> front-end can really be made to fit a MyHDL back-end, without just
> re-writing the whole thing as an interpreter.

So, the thinking initially, which I still think might be the route to 
go, is to tap in to the flow object space. Having a really good 
representation of the flow graph would be hugely useful in generating 
the relevant HDL code. If it's possible to also tap into some of the 
annotation code, this might be useful, but then again it might not :)

Cheers,

Henry

From amauryfa at gmail.com  Mon Mar 23 14:08:18 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 23 Mar 2015 14:08:18 +0100
Subject: [pypy-dev] How to help PyPy 3 ?
In-Reply-To: <CAON-fpGCXOdfn-uzBW-kf=H2bfk20O-ZbktRRRcxDBh3cVcwyg@mail.gmail.com>
References: <CAON-fpGCXOdfn-uzBW-kf=H2bfk20O-ZbktRRRcxDBh3cVcwyg@mail.gmail.com>
Message-ID: <CAGmFidZaNOZXafRbMgUEp1aAq7u3b_bScdWb0+oyQrORbMtxBA@mail.gmail.com>

Hi,

2015-03-22 14:45 GMT+01:00 Ludovic Gasc <gmludo at gmail.com>:

> Hi,
>
> I want to try to help PyPy 3, especially to run AsyncIO on PyPy 3.
>
> For now, I've tried to compile PyPy from 3.3 branch, and install AsyncIO.
> I'd an issue with time.monotonic() and time.get_clock_info() in AsyncIO,
> because it isn't implemented in PyPy 3.
>
> I've replaced that by time.time() and hardcoded value of
> time.get_clock_info() return, only to see until AsyncIO should work on PyPy
> 3.
>
> I've launched several examples from AsyncIO documentation, it works,
> except when I launch a TCP client:
> https://docs.python.org/3/library/asyncio-protocol.html#tcp-echo-client-protocol
> I've the same issue with aiotest: https://bitbucket.org/haypo/aiotest All
> tests pass, except:
> $ ./pypy-c -Wd ../tulip/run_aiotest.py
> Run tests in debug mode
> ...............E
> ======================================================================
> ERROR: test_tcp_hello (aiotest.test_network.NetworkTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/lg/Documents/IDEA/pypy/site-packages/aiotest/test_network.py", line
> 78, in test_tcp_hello
>     self.loop.run_until_complete(coro)
>   File
> "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py", line
> 317, in run_until_complete
>     return future.result()
>   File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/futures.py",
> line 275, in result
>     raise self._exception
>   File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/tasks.py", line
> 238, in _step
>     result = next(coro)
>   File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/coroutines.py",
> line 79, in __next__
>     return next(self.gen)
>   File
> "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py", line
> 644, in create_connection
>     sock, protocol_factory, ssl, server_hostname)
> TypeError: '_SelectorSocketTransport' object is not iterable
>

Please file a bug, this is an issue with "yield from" and return values.
The script below prints "(1, 2)" with CPython3, but "1" with pypy3.3

def f():
    yield 3
    return 1, 2

def g():
    x = yield from f()
    print(x)

list(g())



>
> ----------------------------------------------------------------------
> Ran 16 tests in 0.337s
>
> FAILED (errors=1)
>
> Finally, I've tested to launch the aiohttp server example:
> https://github.com/KeepSafe/aiohttp/blob/master/examples/srv.py
> The servers starts, but when I launch a HTTP request, I've a strange
> stackstrace:
> Error handling request
> Traceback (most recent call last):
>   File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py",
> line 240, in start
>     yield from handler
>   File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in
> handle_request
>     message.method, message.path, message.version))
> AttributeError: 'str' object has no attribute 'method'
> Traceback (most recent call last):
>   File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py",
> line 240, in start
>     yield from handler
>   File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in
> handle_request
>     message.method, message.path, message.version))
> AttributeError: 'str' object has no attribute 'method'
>
> After debugging a little bit with pdb, I see
> that aiohttp.HttpRequestParser parses correctly the request, but "message =
> yield from httpstream.read()" in aiohttp.server at line 226 returns "GET"
> string instead of a RawRequestMessage object.
>
> I'm a total newbie about PyPy internals, implement a monotonic timer seems
> to be over-complicated to me.
> But, I should maybe help with tests and fix small issues like with
> aiohttp.server if I've a clue to fix that.
> Are you interested if I create issues on PyPy project for each problem, or
> I will add only noise in PyPy project ?
>
> BTW, to help PyPy project not only with my brain time, I've did a small
> recurring donation for PyPy 3.
>
> Regards.
> --
> Ludovic Gasc (GMLudo)
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150323/2daca908/attachment.html>

From mount.sarah at gmail.com  Mon Mar 23 14:02:28 2015
From: mount.sarah at gmail.com (Sarah Mount)
Date: Mon, 23 Mar 2015 13:02:28 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <55100E61.2000508@cantab.net>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
 <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
 <55100E61.2000508@cantab.net>
Message-ID: <CALG3WZm-XD=k=3VHVqRFtRdF2Bppm4Fr9hUHD+15HSOn3-fzQw@mail.gmail.com>

Hi,

On Mon, Mar 23, 2015 at 1:00 PM, Henry Gomersall <heng at cantab.net> wrote:
> On 23/03/15 12:50, Sarah Mount wrote:
> <snip>
>>>
>>> Well, potentially, but the big win is in being allowed a broader range of
>>> >convertible constructs. For example, there is currently no way to handle
>>> >general iterables (only loops of the form `for i in range(N):` are
>>> > allowed).
>>> >Clearly, this is very restrictive for writing nice, expressive code.
>>> >
>>> >Stepping back a minute. The ultimate goal IMO would be a tool that takes
>>> > a
>>> >MyHDL instance block (that is, that represents the function of a
>>> > hardware
>>> >block), along with the associated static namespace, and converts into
>>> >something that downstream tools can understand (VHDL or Verilog), with
>>> > as
>>> >much expressive power in the code as makes sense given the target
>>> >restrictions.
>>> >
>>
>> Hmm. So, what I understand from this is that your current front-end
>> implements a very small subset of Python, you have noticed that
>> RPython implements a slightly larger subset of Python, so you want to
>> replace your front-end and maybe some internals with the RPython
>> equivalents? I'm not sure how well this will work. For one thing,
>> RPython is intended to be translated to a native format (i.e. you take
>> a description of an interpreter or VM in RPython and after a very long
>> compile you get a native executable that is your interpreter, with any
>> RPython internals, such as JITs and GCs, included). I'm not sure if
>> this is a win for you or not, because I'm not sure if an RPython
>> front-end can really be made to fit a MyHDL back-end, without just
>> re-writing the whole thing as an interpreter.
>
>
> So, the thinking initially, which I still think might be the route to go, is
> to tap in to the flow object space. Having a really good representation of
> the flow graph would be hugely useful in generating the relevant HDL code.
> If it's possible to also tap into some of the annotation code, this might be
> useful, but then again it might not :)
>

I don't know enough about the internals of RPython etc. to speak to
this. It sounds like it would be more useful to you to try and extract
the modules and packages you need and move them over to MyHDL (if your
licenses match suitably) rather than using the full toolchain.

Perhaps I have the wrong end of the stick though.

Sarah

-- 
Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
website:  http://www.snim2.org/
twitter: @snim2

From lockhart at csl.cornell.edu  Mon Mar 23 16:26:10 2015
From: lockhart at csl.cornell.edu (Derek Lockhart)
Date: Mon, 23 Mar 2015 11:26:10 -0400
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CALG3WZm-XD=k=3VHVqRFtRdF2Bppm4Fr9hUHD+15HSOn3-fzQw@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
 <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
 <55100E61.2000508@cantab.net>
 <CALG3WZm-XD=k=3VHVqRFtRdF2Bppm4Fr9hUHD+15HSOn3-fzQw@mail.gmail.com>
Message-ID: <CADEfmBuQshtpi5f9MJ5SgAd-1dLo1SWJMuKSV05n3dMmu8R7ow@mail.gmail.com>

Hi all,

Thought I would chime in since this is also of interest to me.  I have two
distinct projects related to hardware simulation, one of which is very well
suited towards the RPython toolchain (Pydgin), and another which is much
more closely related to MyHDL (PyMTL).  Sarah kindly pointed out that
Pydgin is relevant here (thanks Sarah!), but I thought it might be useful
to briefly summarize both Pydgin and PyMTL to explain why one uses RPython
and the other does not.

[[ This turned out to be really long, so feel free to check out the TL;DR
at the end. ]]

== PyMTL==

PyMTL (https://github.com/cornell-brg/pymtl) is a framework for hardware
design. If you just use PyMTL for RTL-modeling it could very much be
considered an alternative to MyHDL: special Python types are provided to
model fixed-bitwidth logic that can be simulated directly in Python. Once
this behavioral logic is verified in Python simulation, and provided the
logic is described in a sufficiently restricted subset of Python, we can
then translate it into Verilog. We do not have a VHDL backend.

More generally, PyMTL supports multi-level simulation, so that
functional-level, cycle-level, and register-transfer level logic can be
described, composed, and simulated together. This, along with the way we
construct our models (we use Python classes like Verilog modules), makes us
a bit different than MyHDL.  However, I think the mechanism we use to
translate RTL logic into Verilog is quite similar: we have our own
"translator" to manually walk the AST of behavioral logic, infer types, and
generate Verilog source.

[[ Aside: for a clarification of the distinctions between functional-level,
cycle-level, and RTL modeling you can see a brief summary here:
http://morepypy.blogspot.com/2015/03/pydgin-using-rpython-to-generate-fast.html#simulators-designing-hardware-with-software
]]

== Pydgin ==

Pydgin (https://github.com/cornell-brg/pydgin) is a framework for
constructing fast **functional-level** simulators for processors, also
known as Instruction-Set Simulators (ISSs).  Pydgin uses the RPython
translation toolchain to take simple Python descriptions of a processor's
state and instructions and generate a JIT-enabled ISS.  This simulator is
capable of executing compiled ELF binaries (however, your compiler and your
Pydgin ISS must agree on what the target architecture is!).

Pydgin is a great match for RPython because writing a simple processor
simulator is very similar to writing a bytecode interpreter.  Instructions
are fetched, decoded, and then executed; the primary difference being that
instead of bytecodes we are executing binary instructions extracted from a
compiled executable.

However, this does not necessarily work for arbitrary hardware. Processors
specifically just happen to behave a lot like a language VM. I'm not sure
this approach could be used to create say, a fast functional-level model of
a router; or a specialized vision-processing accelerator. It certainly
wouldn't make sense for something simple like a ripple-carry adder (bad
example, but I think you catch what I mean).

== Using the RPython for HDL Generation ==

PyMTL certainly shares the same challenge Henry mentioned in terms of
translator/converter/compiler robustness: we only support a subset of
Python which is is many ways more restrictive than RPython. I've thought a
little bit about using the RPython translation toolchain to make Verilog
generation easier; its type analysis is certainly more sophisticated than
my simple translator.  To make this possible, support for fixed-bitwidth
types would be needed for the type annotator, and in our case we would
really need support for Python properties because we use these quite a bit.

However, I think the RPython translation toolchain wouldn't really address
the biggest challenge here, which is lowering the behavioral logic into a
representation translatable into synthesizable Verilog.  I've actually
found the the type annotation to be manageable (although my type inference
for local temporaries really could/should be improved), the hard part for
me has been transforming the AST into a representation that maps to valid
Verilog.  This often involves unrolling lists of ports into individual
ports, expanding attribute accesses into temporary wires and assignments,
determining which signals should be marked as type wire or reg, creating
wire lists so that indexed accesses map to the correct input/output ports.
More importantly, it would be really nice to know when it simply **is not
possible** to convert a Python construct into valid Verilog. My analysis is
currently not great at this.

I'm not sure the intermediate representation used by the RPython toolchain
would be a very good mapping for Verilog, so I'm not entirely sure there is
much else than the annotator we could really leverage. That combined with
the relative complexity of the RPython toolchain makes it seem like it
might not be the right way to go.  It will likely need to support much more
complex Python code than we could ever hope to convert into Verilog, which
results in both slower translation times and potentially complicates
maintainability. However, I'm not a compiler expert so I could be (and
hopefully I am!) wrong here.

== How Can RPython Help RTL Designers? ==

That being said, I think there are some neat opportunities for using
RPython to speed up hardware simulation. Pydgin was certainly one approach,
but like I mentioned earlier it only works well for models that behave like
an interpreter. However, both MyHDL and PyMTL use Python for hardware
simulation, and both have shown rather good speedups when using PyPy. I
think the addition of a few features could potentially improve our
simulation performance on PyPy even more:

=== Backend support for fixed bit-width datatypes ===

RPython already has this for 32-bit integers, but in RTL we are dealing
with arbitrary bitwidths. I currently model these in Python using a special
Bits class which handles all shifting, slicing, bitmasking, and conversions
when interacting with Python ints.  I suspect these operations are pretty
expensive.  Maybe PyPy already does a good job of optimizing these, but it
seems like there might be an opportunity here.

=== Support for tracing loop-less logic ===

RTL models have very data dependent control flow, however, this control
flow very rarely uses loops and instead consists of large if/else trees.
If/else trees model muxes which are common in hardware, whereas loops are
only sometimes synthesizable into real hardware and are therefore used less
frequently (loops in HDL descriptions are generally only synthesizable if
they represent logic that will ultimately be unrolled in a hardware
implementation).

In addition, PyMTL models combinational logic with many small functions and
an event queue: the order and frequency in which these small functions are
placed on the queue to execute is extremely data-dependent.  I suspect the
PyPy tracer has a very hard time optimizing this logic. However, we know
ahead of time these small functions will be executed very frequently and
should be JIT'd, we just don't know the order in which they will execute
since this will likely change each iteration through the simulator.

For these reasons, we think RTL models are very challenging for PyPy to
optimize; and we have data that supports this.  If you look at Figure 14 of
our PyMTL paper (
http://csl.cornell.edu/~cbatten/pdfs/lockhart-pymtl-micro2014.pdf), you'll
see that PyPy is able to give us 25x speedup on a functional-level model of
a simple mesh network (~1 primary logic function), a 12x speedup on a
cycle-level model (~3*64 primary logic functions), and only a 6x
improvement on an RTL model (~14*64 logic functions).  A 6x performance
improvement is nothing to sneeze at, but it seems like a more detailed
model should have an opportunity to get **higher** speedups than
less-detailed models because there are more operations to optimize.

=== Hooks for guaranteeing JITing of specific functions ===

To address the problems mentioned above, it would be really helpful to be
able to mark specific functions as being JIT'able to PyPy.  We know these
will be executed very frequently, so we don't even need/want a visit
counter, just JIT it as soon as you see it.  Or maybe set the trace
boundries to be the entry and exit of the decorated function.
Unfortunately, this is more of a method-based approach and I don't think
it's easy to implement in RPython/PyPy.

= TL;DR =

RPython is great for creating fast simulators for hardware models that act
like an interpreter (e.g. processors). I have doubts about how useful it
can be for generating Verilog for a Python HDL; although I hope I'm wrong
here and someone smarter than me can figure it out.  However, I think both
MyHDL and PyMTL simulations could greatly benefit from certain PyPy
optimizations.


Derek


On Mon, Mar 23, 2015 at 9:02 AM, Sarah Mount <mount.sarah at gmail.com> wrote:

> Hi,
>
> On Mon, Mar 23, 2015 at 1:00 PM, Henry Gomersall <heng at cantab.net> wrote:
> > On 23/03/15 12:50, Sarah Mount wrote:
> > <snip>
> >>>
> >>> Well, potentially, but the big win is in being allowed a broader range
> of
> >>> >convertible constructs. For example, there is currently no way to
> handle
> >>> >general iterables (only loops of the form `for i in range(N):` are
> >>> > allowed).
> >>> >Clearly, this is very restrictive for writing nice, expressive code.
> >>> >
> >>> >Stepping back a minute. The ultimate goal IMO would be a tool that
> takes
> >>> > a
> >>> >MyHDL instance block (that is, that represents the function of a
> >>> > hardware
> >>> >block), along with the associated static namespace, and converts into
> >>> >something that downstream tools can understand (VHDL or Verilog), with
> >>> > as
> >>> >much expressive power in the code as makes sense given the target
> >>> >restrictions.
> >>> >
> >>
> >> Hmm. So, what I understand from this is that your current front-end
> >> implements a very small subset of Python, you have noticed that
> >> RPython implements a slightly larger subset of Python, so you want to
> >> replace your front-end and maybe some internals with the RPython
> >> equivalents? I'm not sure how well this will work. For one thing,
> >> RPython is intended to be translated to a native format (i.e. you take
> >> a description of an interpreter or VM in RPython and after a very long
> >> compile you get a native executable that is your interpreter, with any
> >> RPython internals, such as JITs and GCs, included). I'm not sure if
> >> this is a win for you or not, because I'm not sure if an RPython
> >> front-end can really be made to fit a MyHDL back-end, without just
> >> re-writing the whole thing as an interpreter.
> >
> >
> > So, the thinking initially, which I still think might be the route to
> go, is
> > to tap in to the flow object space. Having a really good representation
> of
> > the flow graph would be hugely useful in generating the relevant HDL
> code.
> > If it's possible to also tap into some of the annotation code, this
> might be
> > useful, but then again it might not :)
> >
>
> I don't know enough about the internals of RPython etc. to speak to
> this. It sounds like it would be more useful to you to try and extract
> the modules and packages you need and move them over to MyHDL (if your
> licenses match suitably) rather than using the full toolchain.
>
> Perhaps I have the wrong end of the stick though.
>
> Sarah
>
> --
> Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
> website:  http://www.snim2.org/
> twitter: @snim2
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150323/5ac0c27d/attachment-0001.html>

From heng at cantab.net  Tue Mar 24 19:46:58 2015
From: heng at cantab.net (Henry Gomersall)
Date: Tue, 24 Mar 2015 18:46:58 +0000
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <CADEfmBuQshtpi5f9MJ5SgAd-1dLo1SWJMuKSV05n3dMmu8R7ow@mail.gmail.com>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
 <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
 <55100E61.2000508@cantab.net>
 <CALG3WZm-XD=k=3VHVqRFtRdF2Bppm4Fr9hUHD+15HSOn3-fzQw@mail.gmail.com>
 <CADEfmBuQshtpi5f9MJ5SgAd-1dLo1SWJMuKSV05n3dMmu8R7ow@mail.gmail.com>
Message-ID: <5511B122.8050603@cantab.net>

On 23/03/15 15:26, Derek Lockhart wrote:
> More generally, PyMTL supports multi-level simulation, so that 
> functional-level, cycle-level, and register-transfer level logic can 
> be described, composed, and simulated together. This, along with the 
> way we construct our models (we use Python classes like Verilog 
> modules), makes us a bit different than MyHDL.  However, I think the 
> mechanism we use to translate RTL logic into Verilog is quite similar: 
> we have our own "translator" to manually walk the AST of behavioral 
> logic, infer types, and generate Verilog source.
>

PyMTL looks interesting, though I don't quite understand what you mean 
by multi-level simulation.

On the topic at hand, I wonder if there is scope for defining a common 
intermediate representation (I mean, that both PyMTL and MyHDL can 
target). Certainly, that would allow substantially more capability for 
novel schemes for writing to V*, as well as sharing improvements in 
target generation. If so, perhaps this discussion should be taken off list.

My question in the context of PyPy is then, is building on the flow 
object space sensible here? This is quite a natural fit for MyHDL, as 
the logic is fully described (including all the signals) inside a single 
function.

Cheers,

Henry

From matti.picus at gmail.com  Tue Mar 24 22:09:46 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 24 Mar 2015 23:09:46 +0200
Subject: [pypy-dev] Release 2.5.1 binaries now available for download
In-Reply-To: <5505E07B.3000003@gmail.com>
References: <5505E07B.3000003@gmail.com>
Message-ID: <5511D29A.30405@gmail.com>

I have uploaded releases bundles to
https://bitbucket.org/pypy/pypy/downloads
Could people please take a few minutes to test them out on their 
favorite platform? Also, any corrections, additions, or edits to the 
release notice would be welcome, it is here
http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
Note it has a broken link, the whatsnew is here instead
http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
This will be fixed when I merge the release branch back into default

Barring any issues, the release notice should hit the blog by the 
weekend, and I will update the web sites appropriately before then.
Matti

From fijall at gmail.com  Wed Mar 25 08:58:11 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Mar 2015 09:58:11 +0200
Subject: [pypy-dev] Custom types for annotating a flow object space
In-Reply-To: <5511B122.8050603@cantab.net>
References: <550F1FC0.4000502@cantab.net>
 <CAK5idxTZZW5CnkNfRMKD4toFLrDR-RbbGszhGQevcZRc=HNYPQ@mail.gmail.com>
 <550FD487.6030804@cantab.net>
 <CAK5idxTzbQhJnkAM8D=c+5LzSJC1oQ91RpF-AmYh-FDOkj8ahw@mail.gmail.com>
 <550FED3E.5060405@cantab.net>
 <CALG3WZkxW8aSVz+0dqn1uakrTriWTQYx8iv_CmwX6ac=9hjrmA@mail.gmail.com>
 <55100A05.3020503@cantab.net>
 <CALG3WZmVA5_FpzuUJ22ti7bgrDKNjU-PPyvrVCtDE3cS=cgPLA@mail.gmail.com>
 <55100E61.2000508@cantab.net>
 <CALG3WZm-XD=k=3VHVqRFtRdF2Bppm4Fr9hUHD+15HSOn3-fzQw@mail.gmail.com>
 <CADEfmBuQshtpi5f9MJ5SgAd-1dLo1SWJMuKSV05n3dMmu8R7ow@mail.gmail.com>
 <5511B122.8050603@cantab.net>
Message-ID: <CAK5idxSqw4QWx2bQ2f9o8v6VrMvqn6CaajCJ-viNQJCeE2rgow@mail.gmail.com>

On Tue, Mar 24, 2015 at 8:46 PM, Henry Gomersall <heng at cantab.net> wrote:
> On 23/03/15 15:26, Derek Lockhart wrote:
>>
>> More generally, PyMTL supports multi-level simulation, so that
>> functional-level, cycle-level, and register-transfer level logic can be
>> described, composed, and simulated together. This, along with the way we
>> construct our models (we use Python classes like Verilog modules), makes us
>> a bit different than MyHDL.  However, I think the mechanism we use to
>> translate RTL logic into Verilog is quite similar: we have our own
>> "translator" to manually walk the AST of behavioral logic, infer types, and
>> generate Verilog source.
>>
>
> PyMTL looks interesting, though I don't quite understand what you mean by
> multi-level simulation.
>
> On the topic at hand, I wonder if there is scope for defining a common
> intermediate representation (I mean, that both PyMTL and MyHDL can target).
> Certainly, that would allow substantially more capability for novel schemes
> for writing to V*, as well as sharing improvements in target generation. If
> so, perhaps this discussion should be taken off list.
>
> My question in the context of PyPy is then, is building on the flow object
> space sensible here? This is quite a natural fit for MyHDL, as the logic is
> fully described (including all the signals) inside a single function.
>
> Cheers,
>
> Henry

I'm skeptical. Maybe you can abuse parts of it, but you'll soon
realize that some things are not fitting in your model. Additionally,
you would need to write verilog/vhdl backend which is a non-trivial
amount of work

From fijall at gmail.com  Wed Mar 25 08:59:08 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Mar 2015 09:59:08 +0200
Subject: [pypy-dev] Release 2.5.1 binaries now available for download
In-Reply-To: <5511D29A.30405@gmail.com>
References: <5505E07B.3000003@gmail.com> <5511D29A.30405@gmail.com>
Message-ID: <CAK5idxRvdTwpCUKMa7NqLJT1zj5mAN5Cz291gxjSaDBaQC6Yow@mail.gmail.com>

No releases on weekends please :-) (It's a PR disaster)

On Tue, Mar 24, 2015 at 11:09 PM, Matti Picus <matti.picus at gmail.com> wrote:
> I have uploaded releases bundles to
> https://bitbucket.org/pypy/pypy/downloads
> Could people please take a few minutes to test them out on their favorite
> platform? Also, any corrections, additions, or edits to the release notice
> would be welcome, it is here
> http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
> Note it has a broken link, the whatsnew is here instead
> http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
> This will be fixed when I merge the release branch back into default
>
> Barring any issues, the release notice should hit the blog by the weekend,
> and I will update the web sites appropriately before then.
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From fijall at gmail.com  Wed Mar 25 09:22:15 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Mar 2015 10:22:15 +0200
Subject: [pypy-dev] Release 2.5.1 binaries now available for download
In-Reply-To: <CAK5idxRvdTwpCUKMa7NqLJT1zj5mAN5Cz291gxjSaDBaQC6Yow@mail.gmail.com>
References: <5505E07B.3000003@gmail.com> <5511D29A.30405@gmail.com>
 <CAK5idxRvdTwpCUKMa7NqLJT1zj5mAN5Cz291gxjSaDBaQC6Yow@mail.gmail.com>
Message-ID: <CAK5idxTjZg1kQc0LoX1nDB_n3=oHR0P8KzpkXFGctKa63iDwcA@mail.gmail.com>

maybe we should actually write up (or copy-paste) what stdlib 2.7.9
means (it changes quite a bit)

On Wed, Mar 25, 2015 at 9:59 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> No releases on weekends please :-) (It's a PR disaster)
>
> On Tue, Mar 24, 2015 at 11:09 PM, Matti Picus <matti.picus at gmail.com> wrote:
>> I have uploaded releases bundles to
>> https://bitbucket.org/pypy/pypy/downloads
>> Could people please take a few minutes to test them out on their favorite
>> platform? Also, any corrections, additions, or edits to the release notice
>> would be welcome, it is here
>> http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
>> Note it has a broken link, the whatsnew is here instead
>> http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
>> This will be fixed when I merge the release branch back into default
>>
>> Barring any issues, the release notice should hit the blog by the weekend,
>> and I will update the web sites appropriately before then.
>> Matti
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

From amauryfa at gmail.com  Wed Mar 25 09:41:35 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 25 Mar 2015 09:41:35 +0100
Subject: [pypy-dev] Release 2.5.1 binaries now available for download
In-Reply-To: <CAK5idxTjZg1kQc0LoX1nDB_n3=oHR0P8KzpkXFGctKa63iDwcA@mail.gmail.com>
References: <5505E07B.3000003@gmail.com> <5511D29A.30405@gmail.com>
 <CAK5idxRvdTwpCUKMa7NqLJT1zj5mAN5Cz291gxjSaDBaQC6Yow@mail.gmail.com>
 <CAK5idxTjZg1kQc0LoX1nDB_n3=oHR0P8KzpkXFGctKa63iDwcA@mail.gmail.com>
Message-ID: <CAGmFidZZ7UkFceDFTromNd5V4=Q2D8gqygvpcmzrVfJS7LBZ_g@mail.gmail.com>

2015-03-25 9:22 GMT+01:00 Maciej Fijalkowski <fijall at gmail.com>:

> maybe we should actually write up (or copy-paste) what stdlib 2.7.9
> means (it changes quite a bit)
>

Agreed. Also the stdlib contains the test suite,
so the changes also impact the core interpreter.


>
> On Wed, Mar 25, 2015 at 9:59 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> > No releases on weekends please :-) (It's a PR disaster)
> >
> > On Tue, Mar 24, 2015 at 11:09 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
> >> I have uploaded releases bundles to
> >> https://bitbucket.org/pypy/pypy/downloads
> >> Could people please take a few minutes to test them out on their
> favorite
> >> platform? Also, any corrections, additions, or edits to the release
> notice
> >> would be welcome, it is here
> >> http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html
> >> Note it has a broken link, the whatsnew is here instead
> >> http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html
> >> This will be fixed when I merge the release branch back into default
> >>
> >> Barring any issues, the release notice should hit the blog by the
> weekend,
> >> and I will update the web sites appropriately before then.
> >> Matti
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150325/477adcd4/attachment-0001.html>

From matti.picus at gmail.com  Thu Mar 26 05:57:48 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 26 Mar 2015 06:57:48 +0200
Subject: [pypy-dev] PyPy 2.5.1 is live, please help get the word out
Message-ID: <551391CC.1010500@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150326/134cb498/attachment.html>

From john.m.camara at gmail.com  Thu Mar 26 17:29:18 2015
From: john.m.camara at gmail.com (John Camara)
Date: Thu, 26 Mar 2015 12:29:18 -0400
Subject: [pypy-dev] vmprof compression
Message-ID: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>

Hi Fijal,

To recap and continue the discussion from irc.

We already discussed that the stack id are based on a counter which is good
but I also want to confirm that the ids have locality associated with the
code.  That is similar areas of the code will have similar ids.  Just to
make sure are not random with respect to the code otherwise compression
will not be helpful.  If the ids are random that would need to be corrected
first.

Right now the stack traces are written to the file repeating the following
sequence

MARKER_STACKTRACE
count
depth
stack
stack
...
stack

In order to get a high compression ratio it would be better to combine
multiple stacktraces and rearrange the data as follows

MARKER_COMPRESSED_STACKTRACES
counts_compressed_length
counts_compressed
depths_compressed_length
depths_compressed
stacks_compressed_length
stacks_compressed

In order to build the compress data you will want to 3 pairs of 2 buffers.
A pair of buffers for counts, depths, and stacks.  Your profiller would be
writing to one set of buffers and another thread would be responsible for
compressing buffers that are full and writing them to the file.  Once a set
of buffers are full the profiller would start filling up the other set of
buffers.

For each set of buffers you need a variable to hold the previous count,
depth, and stack id.  They will be initialized to 0 before any data is
written to an empty buffer.  In stead of writing the actual count value
into the counts buffer you will write the difference between the current
count and the previous count.  The reason for doing this is that the delta
values will mostly be around 0 which will significantly improve the
compression ratio without adding much overhead.  Of course you would do the
same for depths and stack ids.

When you compress the data you compress each buffer individually to make
sure like data is being compressed.  Like data compresses better the unlike
data and by saving deltas very few bits will be required to represent the
data and you are likely to have long strings of 0s and 1s.

I'm sure now you can see why I don't want stack ids being random.  As if
they are random then the deltas will be all over the place so you wont end
up with long strings of 0s and 1s and random data itself does not compress.

To test this out I wouldn't bother modifying the c code but instead try it
out in Python to first make sure the compression is providing huge gains
and figure out how to tune the algorithm without having to mess with the
signal handlers and writing the code for the separate thread and dealing
issues such as making sure you don't start writing to a buffer before the
thread finished writing the data to the file, etc.  I would just read an
existing profile file and rewrite it to a different file by rearranging the
data and compressing the delta as I described.  You can get away with one
set of buffers as you wouldn't be profiling at the same time.

To tune this process you will need to determine the appropriate number of
stack traces that is small enough to keep memory down but large enough so
that the overhead associated with compression small.  Maybe start of with
about 8000 stack traces.  I would try gzip, bz2, and lzma and look at their
compression ratios and times.  Gzip is general faster than bz2 and lzma is
the slowest.  On the other hand lzma provides the best compression and gzip
the worse.  Since you will be compressing deltas you most likely can get
away with using the fastest compression options under each compressor and
not effect the compression ratio.  But I would test it to verify this as it
does depend on the data being compressed whether or not this is true.  Also
one option that is available in lzma is the ability to set the width of the
data to look at when looking for patterns.  Since you are saving 32 or 64
bit ints depending on the platform you can set the option to either 4 or 8
bytes based on the platform.  I don't believe qzip or bz2 have this
option.  By setting this option in lzma you will likely improve the
compression ratio.

You may find that counts and depths give similar compression, between the 3
compression types in which case just use the fastest which will likely be
gzip.  On the other hand maybe the stack ids will be better off using
lzma.  This is also another reason to separate out, like data, as it gives
you an option to use the fastest compressors for some data types while
using others to provide for better compression.

I would not be surprised if this approach achieves a compression ratio
better than 100x but that will be heavily dependent on how local the stack
ids are.  Also don't forget about simple things like not using 64 bit ints
when you can get away with smaller ones.

Also for a slight variation to the above.  If you find most of your deltas
are < 127 you could write them out as 1 byte and when greater than 127
write them out as a 4 byte int with the high bit set.  If you do this then
don't set the lzma option to 4 or 8 byte boundaries as now your data is a
mixture of 1 and 4 byte values.  This sometimes can provide huge reductions
in compression times without much effect on the overall compression ratio.

Hope you find this helpful.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150326/f84dec86/attachment.html>

From stuaxo2 at yahoo.com  Fri Mar 27 12:39:42 2015
From: stuaxo2 at yahoo.com (Stuart Axon)
Date: Fri, 27 Mar 2015 11:39:42 +0000 (UTC)
Subject: [pypy-dev] vmprof compression
In-Reply-To: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
References: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
Message-ID: <622779707.2783029.1427456382707.JavaMail.yahoo@mail.yahoo.com>

Hi,?? It's worth adding lzop to the list, of compressors to test, as it's built specifically to have a low CPU overhead, at the cost of some compression ratio.

http://www.lzop.org/
?S++ 


     On Thursday, March 26, 2015 11:29 PM, John Camara <john.m.camara at gmail.com> wrote:
   
 

 Hi Fijal,
To recap and continue the discussion from irc.
We already discussed that the stack id are based on a counter which is good but I also want to confirm that the ids have locality associated with the code.? That is similar areas of the code will have similar ids.? Just to make sure are not random with respect to the code otherwise compression will not be helpful.? If the ids are random that would need to be corrected first.
Right now the stack traces are written to the file repeating the following sequence
MARKER_STACKTRACE
countdepthstackstack...stack
In order to get a high compression ratio it would be better to combine multiple stacktraces and rearrange the data as follows
MARKER_COMPRESSED_STACKTRACEScounts_compressed_lengthcounts_compresseddepths_compressed_lengthdepths_compressedstacks_compressed_lengthstacks_compressed
In order to build the compress data you will want to 3 pairs of 2 buffers.? A pair of buffers for counts, depths, and stacks.? Your profiller would be writing to one set of buffers and another thread would be responsible for compressing buffers that are full and writing them to the file.? Once a set of buffers are full the profiller would start filling up the other set of buffers.
For each set of buffers you need a variable to hold the previous count, depth, and stack id.? They will be initialized to 0 before any data is written to an empty buffer.? In stead of writing the actual count value into the counts buffer you will write the difference between the current count and the previous count.? The reason for doing this is that the delta values will mostly be around 0 which will significantly improve the compression ratio without adding much overhead.? Of course you would do the same for depths and stack ids.
When you compress the data you compress each buffer individually to make sure like data is being compressed.? Like data compresses better the unlike data and by saving deltas very few bits will be required to represent the data and you are likely to have long strings of 0s and 1s.
I'm sure now you can see why I don't want stack ids being random.? As if they are random then the deltas will be all over the place so you wont end up with long strings of 0s and 1s and random data itself does not compress.
To test this out I wouldn't bother modifying the c code but instead try it out in Python to first make sure the compression is providing huge gains and figure out how to tune the algorithm without having to mess with the signal handlers and writing the code for the separate thread and dealing issues such as making sure you don't start writing to a buffer before the thread finished writing the data to the file, etc.? I would just read an existing profile file and rewrite it to a different file by rearranging the data and compressing the delta as I described.? You can get away with one set of buffers as you wouldn't be profiling at the same time.
To tune this process you will need to determine the appropriate number of stack traces that is small enough to keep memory down but large enough so that the overhead associated with compression small.? Maybe start of with about 8000 stack traces.? I would try gzip, bz2, and lzma and look at their compression ratios and times.? Gzip is general faster than bz2 and lzma is the slowest.? On the other hand lzma provides the best compression and gzip the worse.? Since you will be compressing deltas you most likely can get away with using the fastest compression options under each compressor and not effect the compression ratio.? But I would test it to verify this as it does depend on the data being compressed whether or not this is true.? Also one option that is available in lzma is the ability to set the width of the data to look at when looking for patterns.? Since you are saving 32 or 64 bit ints depending on the platform you can set the option to either 4 or 8 bytes based on the platform.? I don't believe qzip or bz2 have this option.? By setting this option in lzma you will likely improve the compression ratio.
You may find that counts and depths give similar compression, between the 3 compression types in which case just use the fastest which will likely be gzip.? On the other hand maybe the stack ids will be better off using lzma.? This is also another reason to separate out, like data, as it gives you an option to use the fastest compressors for some data types while using others to provide for better compression.
I would not be surprised if this approach achieves a compression ratio better than 100x but that will be heavily dependent on how local the stack ids are.? Also don't forget about simple things like not using 64 bit ints when you can get away with smaller ones.
Also for a slight variation to the above.? If you find most of your deltas are < 127 you could write them out as 1 byte and when greater than 127 write them out as a 4 byte int with the high bit set.? If you do this then don't set the lzma option to 4 or 8 byte boundaries as now your data is a mixture of 1 and 4 byte values.? This sometimes can provide huge reductions in compression times without much effect on the overall compression ratio.
Hope you find this helpful.
John
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


 
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150327/e23d64a9/attachment.html>

From fijall at gmail.com  Fri Mar 27 12:55:31 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 27 Mar 2015 13:55:31 +0200
Subject: [pypy-dev] vmprof compression
In-Reply-To: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
References: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
Message-ID: <CAK5idxSb_b1k7mXZ8XPZR1qd9QAU+nhyVmyNsy6mBMpwh4TSfA@mail.gmail.com>

yeah I think putting gzip or something in the loop is A LOT easier :-)

On Thu, Mar 26, 2015 at 6:29 PM, John Camara <john.m.camara at gmail.com> wrote:
> Hi Fijal,
>
> To recap and continue the discussion from irc.
>
> We already discussed that the stack id are based on a counter which is good
> but I also want to confirm that the ids have locality associated with the
> code.  That is similar areas of the code will have similar ids.  Just to
> make sure are not random with respect to the code otherwise compression will
> not be helpful.  If the ids are random that would need to be corrected
> first.
>
> Right now the stack traces are written to the file repeating the following
> sequence
>
> MARKER_STACKTRACE
> count
> depth
> stack
> stack
> ...
> stack
>
> In order to get a high compression ratio it would be better to combine
> multiple stacktraces and rearrange the data as follows
>
> MARKER_COMPRESSED_STACKTRACES
> counts_compressed_length
> counts_compressed
> depths_compressed_length
> depths_compressed
> stacks_compressed_length
> stacks_compressed
>
> In order to build the compress data you will want to 3 pairs of 2 buffers.
> A pair of buffers for counts, depths, and stacks.  Your profiller would be
> writing to one set of buffers and another thread would be responsible for
> compressing buffers that are full and writing them to the file.  Once a set
> of buffers are full the profiller would start filling up the other set of
> buffers.
>
> For each set of buffers you need a variable to hold the previous count,
> depth, and stack id.  They will be initialized to 0 before any data is
> written to an empty buffer.  In stead of writing the actual count value into
> the counts buffer you will write the difference between the current count
> and the previous count.  The reason for doing this is that the delta values
> will mostly be around 0 which will significantly improve the compression
> ratio without adding much overhead.  Of course you would do the same for
> depths and stack ids.
>
> When you compress the data you compress each buffer individually to make
> sure like data is being compressed.  Like data compresses better the unlike
> data and by saving deltas very few bits will be required to represent the
> data and you are likely to have long strings of 0s and 1s.
>
> I'm sure now you can see why I don't want stack ids being random.  As if
> they are random then the deltas will be all over the place so you wont end
> up with long strings of 0s and 1s and random data itself does not compress.
>
> To test this out I wouldn't bother modifying the c code but instead try it
> out in Python to first make sure the compression is providing huge gains and
> figure out how to tune the algorithm without having to mess with the signal
> handlers and writing the code for the separate thread and dealing issues
> such as making sure you don't start writing to a buffer before the thread
> finished writing the data to the file, etc.  I would just read an existing
> profile file and rewrite it to a different file by rearranging the data and
> compressing the delta as I described.  You can get away with one set of
> buffers as you wouldn't be profiling at the same time.
>
> To tune this process you will need to determine the appropriate number of
> stack traces that is small enough to keep memory down but large enough so
> that the overhead associated with compression small.  Maybe start of with
> about 8000 stack traces.  I would try gzip, bz2, and lzma and look at their
> compression ratios and times.  Gzip is general faster than bz2 and lzma is
> the slowest.  On the other hand lzma provides the best compression and gzip
> the worse.  Since you will be compressing deltas you most likely can get
> away with using the fastest compression options under each compressor and
> not effect the compression ratio.  But I would test it to verify this as it
> does depend on the data being compressed whether or not this is true.  Also
> one option that is available in lzma is the ability to set the width of the
> data to look at when looking for patterns.  Since you are saving 32 or 64
> bit ints depending on the platform you can set the option to either 4 or 8
> bytes based on the platform.  I don't believe qzip or bz2 have this option.
> By setting this option in lzma you will likely improve the compression
> ratio.
>
> You may find that counts and depths give similar compression, between the 3
> compression types in which case just use the fastest which will likely be
> gzip.  On the other hand maybe the stack ids will be better off using lzma.
> This is also another reason to separate out, like data, as it gives you an
> option to use the fastest compressors for some data types while using others
> to provide for better compression.
>
> I would not be surprised if this approach achieves a compression ratio
> better than 100x but that will be heavily dependent on how local the stack
> ids are.  Also don't forget about simple things like not using 64 bit ints
> when you can get away with smaller ones.
>
> Also for a slight variation to the above.  If you find most of your deltas
> are < 127 you could write them out as 1 byte and when greater than 127 write
> them out as a 4 byte int with the high bit set.  If you do this then don't
> set the lzma option to 4 or 8 byte boundaries as now your data is a mixture
> of 1 and 4 byte values.  This sometimes can provide huge reductions in
> compression times without much effect on the overall compression ratio.
>
> Hope you find this helpful.
>
> John
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From tbaldridge at gmail.com  Fri Mar 27 23:46:07 2015
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 27 Mar 2015 16:46:07 -0600
Subject: [pypy-dev] Mod typing error
Message-ID: <CAL36E+ti6j3wO37YZbLQuX6nzyamH89KFHOUW=E+KtJamR8S1Q@mail.gmail.com>

I have some RPython that does this:

return a % b

while in another function I'm calling the same, but the two functions
differ in the types of a and b (mix of ints and floats).

However, during translation I'm getting a blocked block exception.

translation:ERROR] AnnotatorError:
[translation:ERROR]
[translation:ERROR] Blocked block -- operation cannot succeed
[translation:ERROR]
[translation:ERROR]     v12 = mod(v10, v11)
[translation:ERROR]
[translation:ERROR] In <FunctionGraph of
(pixie.vm.numbers:1)_rem_Float_Integer at 0x105b80490>:
[translation:ERROR] no source!
[translation:ERROR] Known variable annotations:
[translation:ERROR]  v10 = SomeFloat()
[translation:ERROR]  v11 = SomeInteger(knowntype=int, nonneg=False,
unsigned=False)
[translation:ERROR]
[translation:ERROR] Blocked block -- operation cannot succeed
[translation:ERROR]
[translation:ERROR]     v15 = mod(v13, v14)
[translation:ERROR]
[translation:ERROR] In <FunctionGraph of
(pixie.vm.numbers:1)_rem_Integer_Float at 0x1059ac550>:
[translation:ERROR] no source!
[translation:ERROR] Known variable annotations:
[translation:ERROR]  v13 = SomeInteger(knowntype=int, nonneg=False,
unsigned=False)
[translation:ERROR]  v14 = SomeFloat()
[translation:ERROR]
[translation:ERROR] Blocked block -- operation cannot succeed
[translation:ERROR]
[translation:ERROR]     v18 = mod(v16, v17)
[translation:ERROR]
[translation:ERROR] In <FunctionGraph of
(pixie.vm.numbers:1)_rem_Float_Float at 0x10544a650>:
[translation:ERROR] no source!
[translation:ERROR] Known variable annotations:
[translation:ERROR]  v16 = SomeFloat()
[translation:ERROR]  v17 = SomeFloat()
[translation:ERROR]


Do we need another entry in rtyper? Looking at rtyper/rfloat.py I see
entries on how to type add, sub, etc, but nothing for mod.

Thanks,

Timothy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150327/c2a9b48d/attachment.html>

From rymg19 at gmail.com  Sat Mar 28 00:00:59 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 27 Mar 2015 18:00:59 -0500
Subject: [pypy-dev] Mod typing error
In-Reply-To: <CAL36E+ti6j3wO37YZbLQuX6nzyamH89KFHOUW=E+KtJamR8S1Q@mail.gmail.com>
References: <CAL36E+ti6j3wO37YZbLQuX6nzyamH89KFHOUW=E+KtJamR8S1Q@mail.gmail.com>
Message-ID: <CAO41-mOwVb4vFsMwCCznoQ_uyG6A1NiHtV_COeuk3bffCqSSPg@mail.gmail.com>

I don't know much about RPython internals, but PyPy
calls rpython.rtyper.lltypesystem.module.ll_math.math_fmod for modulus
operations on floats.

On Fri, Mar 27, 2015 at 5:46 PM, Timothy Baldridge <tbaldridge at gmail.com>
wrote:

> I have some RPython that does this:
>
> return a % b
>
> while in another function I'm calling the same, but the two functions
> differ in the types of a and b (mix of ints and floats).
>
> However, during translation I'm getting a blocked block exception.
>
> translation:ERROR] AnnotatorError:
> [translation:ERROR]
> [translation:ERROR] Blocked block -- operation cannot succeed
> [translation:ERROR]
> [translation:ERROR]     v12 = mod(v10, v11)
> [translation:ERROR]
> [translation:ERROR] In <FunctionGraph of
> (pixie.vm.numbers:1)_rem_Float_Integer at 0x105b80490>:
> [translation:ERROR] no source!
> [translation:ERROR] Known variable annotations:
> [translation:ERROR]  v10 = SomeFloat()
> [translation:ERROR]  v11 = SomeInteger(knowntype=int, nonneg=False,
> unsigned=False)
> [translation:ERROR]
> [translation:ERROR] Blocked block -- operation cannot succeed
> [translation:ERROR]
> [translation:ERROR]     v15 = mod(v13, v14)
> [translation:ERROR]
> [translation:ERROR] In <FunctionGraph of
> (pixie.vm.numbers:1)_rem_Integer_Float at 0x1059ac550>:
> [translation:ERROR] no source!
> [translation:ERROR] Known variable annotations:
> [translation:ERROR]  v13 = SomeInteger(knowntype=int, nonneg=False,
> unsigned=False)
> [translation:ERROR]  v14 = SomeFloat()
> [translation:ERROR]
> [translation:ERROR] Blocked block -- operation cannot succeed
> [translation:ERROR]
> [translation:ERROR]     v18 = mod(v16, v17)
> [translation:ERROR]
> [translation:ERROR] In <FunctionGraph of
> (pixie.vm.numbers:1)_rem_Float_Float at 0x10544a650>:
> [translation:ERROR] no source!
> [translation:ERROR] Known variable annotations:
> [translation:ERROR]  v16 = SomeFloat()
> [translation:ERROR]  v17 = SomeFloat()
> [translation:ERROR]
>
>
> Do we need another entry in rtyper? Looking at rtyper/rfloat.py I see
> entries on how to type add, sub, etc, but nothing for mod.
>
> Thanks,
>
> Timothy
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150327/1e168d97/attachment.html>

From santagada at gmail.com  Sat Mar 28 02:15:42 2015
From: santagada at gmail.com (Leonardo Santagada)
Date: Fri, 27 Mar 2015 22:15:42 -0300
Subject: [pypy-dev] vmprof compression
In-Reply-To: <CAK5idxSb_b1k7mXZ8XPZR1qd9QAU+nhyVmyNsy6mBMpwh4TSfA@mail.gmail.com>
References: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
 <CAK5idxSb_b1k7mXZ8XPZR1qd9QAU+nhyVmyNsy6mBMpwh4TSfA@mail.gmail.com>
Message-ID: <CA+HFc858QZgAf-skOB18teQiW29bm5Zh8WUQkf2NAubbRaOcmw@mail.gmail.com>

snappy and lz4 are good algos to try too.

On Fri, Mar 27, 2015 at 8:55 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> yeah I think putting gzip or something in the loop is A LOT easier :-)
>
> On Thu, Mar 26, 2015 at 6:29 PM, John Camara <john.m.camara at gmail.com>
> wrote:
> > Hi Fijal,
> >
> > To recap and continue the discussion from irc.
> >
> > We already discussed that the stack id are based on a counter which is
> good
> > but I also want to confirm that the ids have locality associated with the
> > code.  That is similar areas of the code will have similar ids.  Just to
> > make sure are not random with respect to the code otherwise compression
> will
> > not be helpful.  If the ids are random that would need to be corrected
> > first.
> >
> > Right now the stack traces are written to the file repeating the
> following
> > sequence
> >
> > MARKER_STACKTRACE
> > count
> > depth
> > stack
> > stack
> > ...
> > stack
> >
> > In order to get a high compression ratio it would be better to combine
> > multiple stacktraces and rearrange the data as follows
> >
> > MARKER_COMPRESSED_STACKTRACES
> > counts_compressed_length
> > counts_compressed
> > depths_compressed_length
> > depths_compressed
> > stacks_compressed_length
> > stacks_compressed
> >
> > In order to build the compress data you will want to 3 pairs of 2
> buffers.
> > A pair of buffers for counts, depths, and stacks.  Your profiller would
> be
> > writing to one set of buffers and another thread would be responsible for
> > compressing buffers that are full and writing them to the file.  Once a
> set
> > of buffers are full the profiller would start filling up the other set of
> > buffers.
> >
> > For each set of buffers you need a variable to hold the previous count,
> > depth, and stack id.  They will be initialized to 0 before any data is
> > written to an empty buffer.  In stead of writing the actual count value
> into
> > the counts buffer you will write the difference between the current count
> > and the previous count.  The reason for doing this is that the delta
> values
> > will mostly be around 0 which will significantly improve the compression
> > ratio without adding much overhead.  Of course you would do the same for
> > depths and stack ids.
> >
> > When you compress the data you compress each buffer individually to make
> > sure like data is being compressed.  Like data compresses better the
> unlike
> > data and by saving deltas very few bits will be required to represent the
> > data and you are likely to have long strings of 0s and 1s.
> >
> > I'm sure now you can see why I don't want stack ids being random.  As if
> > they are random then the deltas will be all over the place so you wont
> end
> > up with long strings of 0s and 1s and random data itself does not
> compress.
> >
> > To test this out I wouldn't bother modifying the c code but instead try
> it
> > out in Python to first make sure the compression is providing huge gains
> and
> > figure out how to tune the algorithm without having to mess with the
> signal
> > handlers and writing the code for the separate thread and dealing issues
> > such as making sure you don't start writing to a buffer before the thread
> > finished writing the data to the file, etc.  I would just read an
> existing
> > profile file and rewrite it to a different file by rearranging the data
> and
> > compressing the delta as I described.  You can get away with one set of
> > buffers as you wouldn't be profiling at the same time.
> >
> > To tune this process you will need to determine the appropriate number of
> > stack traces that is small enough to keep memory down but large enough so
> > that the overhead associated with compression small.  Maybe start of with
> > about 8000 stack traces.  I would try gzip, bz2, and lzma and look at
> their
> > compression ratios and times.  Gzip is general faster than bz2 and lzma
> is
> > the slowest.  On the other hand lzma provides the best compression and
> gzip
> > the worse.  Since you will be compressing deltas you most likely can get
> > away with using the fastest compression options under each compressor and
> > not effect the compression ratio.  But I would test it to verify this as
> it
> > does depend on the data being compressed whether or not this is true.
> Also
> > one option that is available in lzma is the ability to set the width of
> the
> > data to look at when looking for patterns.  Since you are saving 32 or 64
> > bit ints depending on the platform you can set the option to either 4 or
> 8
> > bytes based on the platform.  I don't believe qzip or bz2 have this
> option.
> > By setting this option in lzma you will likely improve the compression
> > ratio.
> >
> > You may find that counts and depths give similar compression, between
> the 3
> > compression types in which case just use the fastest which will likely be
> > gzip.  On the other hand maybe the stack ids will be better off using
> lzma.
> > This is also another reason to separate out, like data, as it gives you
> an
> > option to use the fastest compressors for some data types while using
> others
> > to provide for better compression.
> >
> > I would not be surprised if this approach achieves a compression ratio
> > better than 100x but that will be heavily dependent on how local the
> stack
> > ids are.  Also don't forget about simple things like not using 64 bit
> ints
> > when you can get away with smaller ones.
> >
> > Also for a slight variation to the above.  If you find most of your
> deltas
> > are < 127 you could write them out as 1 byte and when greater than 127
> write
> > them out as a 4 byte int with the high bit set.  If you do this then
> don't
> > set the lzma option to 4 or 8 byte boundaries as now your data is a
> mixture
> > of 1 and 4 byte values.  This sometimes can provide huge reductions in
> > compression times without much effect on the overall compression ratio.
> >
> > Hope you find this helpful.
> >
> > John
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150327/8d97d521/attachment-0001.html>

From arigo at tunes.org  Sat Mar 28 07:29:40 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 28 Mar 2015 07:29:40 +0100
Subject: [pypy-dev] Mod typing error
In-Reply-To: <CAO41-mOwVb4vFsMwCCznoQ_uyG6A1NiHtV_COeuk3bffCqSSPg@mail.gmail.com>
References: <CAL36E+ti6j3wO37YZbLQuX6nzyamH89KFHOUW=E+KtJamR8S1Q@mail.gmail.com>
 <CAO41-mOwVb4vFsMwCCznoQ_uyG6A1NiHtV_COeuk3bffCqSSPg@mail.gmail.com>
Message-ID: <CAMSv6X1ndoDVXFsvjJx9TiAm4OMoGzTWrzgJbWUmWNN8RhpT-A@mail.gmail.com>

Hi Timothy, hi Ryan,

On 28 March 2015 at 00:00, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> I don't know much about RPython internals, but PyPy calls
> rpython.rtyper.lltypesystem.module.ll_math.math_fmod for modulus operations
> on floats.

Yes, "%" on floats is not supported.  You should use math.fmod() in
RPython.  See also how PyPy implements "%" on floats in
pypy.objspace.std.floatobject: descr_mod().  Of course it would be
possible to add RPython support, but we figured out that the
particular rules that make "%" on floats different from math.fmod()
are specific to the Python language anyway, whereas math.fmod() is
directly the C fmod() function.


A bient?t,

Armin.

From john.m.camara at gmail.com  Sat Mar 28 15:22:51 2015
From: john.m.camara at gmail.com (John Camara)
Date: Sat, 28 Mar 2015 10:22:51 -0400
Subject: [pypy-dev] vmprof compression
In-Reply-To: <CA+HFc858QZgAf-skOB18teQiW29bm5Zh8WUQkf2NAubbRaOcmw@mail.gmail.com>
References: <CAL=Xk32_qOzzgSrW_Wi=q09cbF6vRe_Bq9=mEXLUyQiH3=FGOw@mail.gmail.com>
 <CAK5idxSb_b1k7mXZ8XPZR1qd9QAU+nhyVmyNsy6mBMpwh4TSfA@mail.gmail.com>
 <CA+HFc858QZgAf-skOB18teQiW29bm5Zh8WUQkf2NAubbRaOcmw@mail.gmail.com>
Message-ID: <CAL=Xk32+kKQhBgUCgs1GUKTZii4Tuw57rX9NDpYt0UQK+ZeLTA@mail.gmail.com>

I meant to mention them in my email as both of them are great options when
you don't mind sacrificing some compression for significant improvements in
compression and decompression speeds.  These libraries are I/O bound when
saving to a hard drive unless you are using a very low powered processor.
Generally the compression ratio is 0.5-0.75 of that achieved by gzip.
Compression speeds can approach 0.5 GB/s.

These libraries don't offer any advance compression techniques so anything
you do help create long strings of 0s and 1s like compressing the deltas
like I mention in the earlier email will go a long way at significantly
improving the compression ratio while also maintaining high performance.

On Fri, Mar 27, 2015 at 9:15 PM, Leonardo Santagada <santagada at gmail.com>
wrote:

> snappy and lz4 are good algos to try too.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150328/c687ab20/attachment.html>

From cfbolz at gmx.de  Mon Mar 30 15:04:48 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 30 Mar 2015 15:04:48 +0200
Subject: [pypy-dev] CfP ICOOOLPS Workshop: Implementation Compilation
	Optimization of Object Oriented Languages Programming and Systems
In-Reply-To: <8B3B0F9E-C1D3-40A5-8EBA-CB644A782610@labri.fr>
References: <8B3B0F9E-C1D3-40A5-8EBA-CB644A782610@labri.fr>
Message-ID: <828aca2f-3cfd-4265-8c42-44c268a83a6f@email.android.com>




CALL FOR PAPERS (DEADLINE EXTENSION)

		10th Workshop on
Implementation, Compilation, Optimization of
Object-Oriented Languages, Programs and Systems !
		(Collocated with Ecoop?15)
Prague, Czech Republic, July 6th, 2015

[The deadline has been extended to April, 13th 2015 ]

The ICOOOLPS workshop series brings together researchers and practitioners working in the field of OO languages implementation and optimization. ICOOOLPS key goal is to identify current and emerging issues relating to the efficient implementation, compilation and optimization of such languages, and outlining future challenges and research directions.

CALL FOR PAPERS

Topics of interest for ICOOOLPS include, but are not limited to:
	- implementation of fundamental OO and OO-like features (e.g. inheritance, parametric types, memory management, objects, prototypes),
	- runtime systems (e.g. compilers, linkers, virtual machines, garbage collectors),
	- optimizations (e.g. static or dynamic analyses, adaptive virtual machines),
	- resource constraints (e.g. time for real-time systems, space or low-power for embedded systems) and relevant choices and tradeoffs (e.g. constant time vs. non-constant time mechanisms,
	- separate compilation vs. global compilation,
	- dynamic loading vs. global linking, dynamic checking vs. proof-carrying code?).

SUBMISSION TYPES

ICOOOLPS is not a mini-conference; it is a workshop designed to facilitate discussion and the exchange of ideas between peers. ICOOOLPS therefore welcomes both position (1?4 pages) and research (max. 10 pages) papers.

Position papers should outline interesting or unconventional ideas, which need not be fully fleshed out.
Research papers are expected to contain more complete ideas, but these need not necessarily be fully complete as with a traditional conference.
Authors will be given the option to publish their papers (short or long) in the ACM Digital Library if they wish.
Submissions must be written in English, formatted according to ACM SIGPLAN Proceedings style. Please submit via the EasyChair submission site https://easychair.org/conferences/?conf=icooolps15.


ICOOOLPS PROGRAM  COMMITTEE

Chair: Flor?al Morandat (LaBRI/France) - Olivier Zendra (Inria, Loria/France)
Carl Friedrich Bolz (King's College London)
Eric Jul (Alcatel-Lucent Bell Labs)
Tobias Pape (Hasso-Plattner-Institute, Potsdam)
Jean Privat (Universit? du Qu?bec ? Montr?al)
Jeremy Singer (University of Glasgow)
Ga?l Thomas (Telecom SudParis)
Laurence Tratt (King's College London)
Jan Vitek (Northeastern University)
Mario Wolczko (Oracle Labs)

Carl Friedrich Bolz

From gaianeka at yahoo.com  Mon Mar 30 16:13:24 2015
From: gaianeka at yahoo.com (Heng Cheong)
Date: Mon, 30 Mar 2015 14:13:24 +0000 (UTC)
Subject: [pypy-dev] Pypy - Unable to find vcvarsall.bat
Message-ID: <199966669.1491982.1427724804004.JavaMail.yahoo@mail.yahoo.com>

I was reluctant to send this email but after more than a couple of thoughts, I decided what the heck and go for it.I am writing in relation to installing pypy.I was successful in installing numpy, pandas, and matplotlib, but not pypy.My problem is when I install it, I got this error "Unable to find vcvarsall.bat" statement.A simple Google found this problem to be widespread since many years ago, and I guess until today it is still unresolved.How come the pypy development community does not do anything about this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150330/8b43f7d3/attachment.html>

From yeomanyaacov at gmail.com  Mon Mar 30 16:37:46 2015
From: yeomanyaacov at gmail.com (Yaacov Finkelman)
Date: Mon, 30 Mar 2015 10:37:46 -0400
Subject: [pypy-dev] Pypy - Unable to find vcvarsall.bat
In-Reply-To: <199966669.1491982.1427724804004.JavaMail.yahoo@mail.yahoo.com>
References: <199966669.1491982.1427724804004.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CAGBU8t6HRPzkQq9Wd7MUHbdbMg+=X8gK00_YwBkQoP4TJOZu0w@mail.gmail.com>

HI,

Sorry you are having trouble. What operating system are you running?
What steps did you talk to encounter the problem? What compiler are
you trying to use?

Best.

P.S. I am not a dev so this is just some of the questions I think they
will need.

On Mon, Mar 30, 2015 at 10:13 AM, Heng Cheong
<gaianeka at yahoo.com.dmarc.invalid> wrote:
> I was reluctant to send this email but after more than a couple of thoughts,
> I decided what the heck and go for it.
> I am writing in relation to installing pypy.
> I was successful in installing numpy, pandas, and matplotlib, but not pypy.
> My problem is when I install it, I got this error "Unable to find
> vcvarsall.bat" statement.
> A simple Google found this problem to be widespread since many years ago,
> and I guess until today it is still unresolved.
> How come the pypy development community does not do anything about this?
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From gaianeka at yahoo.com  Mon Mar 30 18:41:10 2015
From: gaianeka at yahoo.com (Heng Cheong)
Date: Mon, 30 Mar 2015 16:41:10 +0000 (UTC)
Subject: [pypy-dev] Pypy - Unable to find vcvarsall.bat
In-Reply-To: <CAGBU8t6HRPzkQq9Wd7MUHbdbMg+=X8gK00_YwBkQoP4TJOZu0w@mail.gmail.com>
References: <CAGBU8t6HRPzkQq9Wd7MUHbdbMg+=X8gK00_YwBkQoP4TJOZu0w@mail.gmail.com>
Message-ID: <1267615595.1625982.1427733670977.JavaMail.yahoo@mail.yahoo.com>

Hi Yaacov,
I am using Python 3.4 on Windows 7.I did a lot of Goggling and found differing solutions. Like some people suggested to install Visual Studio 2008 (which is no longer available), or Visual C++ 2010 Express, or MinGW compiler (haven't try this one out yet), and at the same time some people said they have tried out such suggestions and while some got their problem solved, some others said a different kind of technical problem cropped up. So it's like the solutions to this "Unable to find vcvarsall.bat" isn't standardized, thus I have no idea what exactly should I do.


     On Monday, March 30, 2015 10:38 PM, Yaacov Finkelman <yeomanyaacov at gmail.com> wrote:
   

 HI,

Sorry you are having trouble. What operating system are you running?
What steps did you talk to encounter the problem? What compiler are
you trying to use?

Best.

P.S. I am not a dev so this is just some of the questions I think they
will need.

On Mon, Mar 30, 2015 at 10:13 AM, Heng Cheong
<gaianeka at yahoo.com.dmarc.invalid> wrote:
> I was reluctant to send this email but after more than a couple of thoughts,
> I decided what the heck and go for it.
> I am writing in relation to installing pypy.
> I was successful in installing numpy, pandas, and matplotlib, but not pypy.
> My problem is when I install it, I got this error "Unable to find
> vcvarsall.bat" statement.
> A simple Google found this problem to be widespread since many years ago,
> and I guess until today it is still unresolved.
> How come the pypy development community does not do anything about this?
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150330/8ee23f33/attachment.html>

From matti.picus at gmail.com  Mon Mar 30 19:03:25 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 30 Mar 2015 20:03:25 +0300
Subject: [pypy-dev] Pypy - Unable to find vcvarsall.bat
In-Reply-To: <1267615595.1625982.1427733670977.JavaMail.yahoo@mail.yahoo.com>
References: <CAGBU8t6HRPzkQq9Wd7MUHbdbMg+=X8gK00_YwBkQoP4TJOZu0w@mail.gmail.com>
 <1267615595.1625982.1427733670977.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <551981DD.3020505@gmail.com>


On 30/03/15 19:41, Heng Cheong wrote:
 > Hi Yaacov,
 > I am using Python 3.4 on Windows 7.
 > I did a lot of Goggling and found differing solutions. Like some 
people suggested to install Visual Studio 2008
 > (which is no longer available), or Visual C++ 2010 Express, or MinGW 
compiler (haven't try this one out yet),
 > and at the same time some people said they have tried out such 
suggestions and while some got their
 > problem solved, some others said a different kind of technical 
problem cropped up. So it's like the
 > solutions to this "Unable to find vcvarsall.bat" isn't standardized, 
thus I have no idea what exactly should I do.

PyPy is available as a binary zip file for windows from 
http://pypy.org/download.html, there is no reason for you to compile it. 
If you do choose to do so, please follow these  instructions carefully : 
http://doc.pypy.org/en/latest/windows.html

Note that by default PyPy is compatible with cpython 2.7.9, we do not 
have a cpython 3.4 version available yet.

As far as obtaining visual studio 2008, you might want to try 
http://www.microsoft.com/en-us/download/details.aspx?id=44266, which was 
the first Google hit for "microsoft python compiler" and provides enough 
of visual studio 2008 to build python packages for python 2.7. The 
standard compiler for python 3.x is visual 2010, also a free download 
from https://www.visualstudio.com/en-us/downloads#d-2010-express

Matti

From phyo.arkarlwin at gmail.com  Mon Mar 30 21:40:58 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Tue, 31 Mar 2015 02:10:58 +0630
Subject: [pypy-dev] PyPy 2.5.1 is live, please help get the word out
In-Reply-To: <551391CC.1010500@gmail.com>
References: <551391CC.1010500@gmail.com>
Message-ID: <CA+HjJrh93AqbBimGaPGJpJE6uNx4pibTTQWNEGwfsj1O93dDoA@mail.gmail.com>

For those linux pypy users who have trouble with dependencies to
openssl/etc :
use https://github.com/squeaky-pl/portable-pypy#
its most portable of all and developer updates it frequently.


On Thu, Mar 26, 2015 at 11:27 AM, Matti Picus <matti.picus at gmail.com> wrote:

>  Tell your friends, your employer, your employees. A Python 2.7.9
> compatible PyPy has been released.
> Thanks to all who contributed
> Release notice:
> http://morepypy.blogspot.com/2015/03/pypy-251-released.html
> Matti
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150331/5925f485/attachment.html>