From holger at merlinux.eu  Tue Mar  1 19:42:57 2011
From: holger at merlinux.eu (holger krekel)
Date: Tue, 1 Mar 2011 18:42:57 +0000
Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups
Message-ID: <20110301184257.GD16231@merlinux.eu>

Hi all,

I just turned

    http://codespeak.net/svn/pypy/extradoc

as well as the internal pypy-z repository READ-ONLY.

Thanks to Ronny's conversion work we now have these repositories at bitbucket.org:

    http://bitbucket.org/pypy/extradoc
        talks, sprintinfo, the source of pypy.org website

and

    http://bitbucket.org/pypy/z (PRIVATE)
        internal contracts/docs, available to long-time contributors

I created a new pypy group on bitbucket, the "pypy:committers" group
which has general read/write access to all public pypy repositories. It
contains the people formerly listed individually for the pypy repo.  New
committers should be added to that pypy:committers group so they get
access to all pypy repos.  There also is a pypy:z group which also gets
write/admin acccess to all repos but additionally can read/write the
pypy/z one. 

Next in line for conversion are:

    pyrepl -> pypy/pyrepl  (discussed with Michael Hudson, the primary author already)
    testrunner -> to be flatly inlined into pypy repo 
    and then the remaining parts of pypy/lang

After that there should be nothing left from PyPy that would 
require codespeak's svn.

best,
holger



From fijall at gmail.com  Wed Mar  2 07:45:12 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 2 Mar 2011 08:45:12 +0200
Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups
In-Reply-To: <20110301184257.GD16231@merlinux.eu>
References: <20110301184257.GD16231@merlinux.eu>
Message-ID: <AANLkTimN7A21OXvt36azQsZRmuXaaoq8u97MoDcJh6hF@mail.gmail.com>

On Tue, Mar 1, 2011 at 8:42 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi all,
>
> I just turned
>
> ? ?http://codespeak.net/svn/pypy/extradoc
>
> as well as the internal pypy-z repository READ-ONLY.
>
> Thanks to Ronny's conversion work we now have these repositories at bitbucket.org:
>
> ? ?http://bitbucket.org/pypy/extradoc
> ? ? ? ?talks, sprintinfo, the source of pypy.org website
>
> and
>
> ? ?http://bitbucket.org/pypy/z (PRIVATE)
> ? ? ? ?internal contracts/docs, available to long-time contributors
>
> I created a new pypy group on bitbucket, the "pypy:committers" group
> which has general read/write access to all public pypy repositories. It
> contains the people formerly listed individually for the pypy repo. ?New
> committers should be added to that pypy:committers group so they get
> access to all pypy repos. ?There also is a pypy:z group which also gets
> write/admin acccess to all repos but additionally can read/write the
> pypy/z one.
>
> Next in line for conversion are:
>
> ? ?pyrepl -> pypy/pyrepl ?(discussed with Michael Hudson, the primary author already)
> ? ?testrunner -> to be flatly inlined into pypy repo
> ? ?and then the remaining parts of pypy/lang

What about sqlite3?

>
> After that there should be nothing left from PyPy that would
> require codespeak's svn.
>
> best,
> holger
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From alex.gaynor at gmail.com  Wed Mar  2 07:47:51 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 2 Mar 2011 01:47:51 -0500
Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups
In-Reply-To: <AANLkTimN7A21OXvt36azQsZRmuXaaoq8u97MoDcJh6hF@mail.gmail.com>
References: <20110301184257.GD16231@merlinux.eu>
	<AANLkTimN7A21OXvt36azQsZRmuXaaoq8u97MoDcJh6hF@mail.gmail.com>
Message-ID: <AANLkTikKKjsZVoBFK3zMCis8H54_1tCBV-JYTE1=e74y@mail.gmail.com>

On Wed, Mar 2, 2011 at 1:45 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Tue, Mar 1, 2011 at 8:42 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi all,
> >
> > I just turned
> >
> >    http://codespeak.net/svn/pypy/extradoc
> >
> > as well as the internal pypy-z repository READ-ONLY.
> >
> > Thanks to Ronny's conversion work we now have these repositories at
> bitbucket.org:
> >
> >    http://bitbucket.org/pypy/extradoc
> >        talks, sprintinfo, the source of pypy.org website
> >
> > and
> >
> >    http://bitbucket.org/pypy/z (PRIVATE)
> >        internal contracts/docs, available to long-time contributors
> >
> > I created a new pypy group on bitbucket, the "pypy:committers" group
> > which has general read/write access to all public pypy repositories. It
> > contains the people formerly listed individually for the pypy repo.  New
> > committers should be added to that pypy:committers group so they get
> > access to all pypy repos.  There also is a pypy:z group which also gets
> > write/admin acccess to all repos but additionally can read/write the
> > pypy/z one.
> >
> > Next in line for conversion are:
> >
> >    pyrepl -> pypy/pyrepl  (discussed with Michael Hudson, the primary
> author already)
> >    testrunner -> to be flatly inlined into pypy repo
> >    and then the remaining parts of pypy/lang
>
> What about sqlite3?
>
> >
> > After that there should be nothing left from PyPy that would
> > require codespeak's svn.
> >
> > best,
> > holger
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Amaury already moved it into the main PyPy repo I think.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110302/6597762c/attachment.html>

From cfbolz at gmx.de  Wed Mar  2 11:25:52 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 02 Mar 2011 11:25:52 +0100
Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups
In-Reply-To: <20110301184257.GD16231@merlinux.eu>
References: <20110301184257.GD16231@merlinux.eu>
Message-ID: <4D6E1B30.6060504@gmx.de>

Hi Holger,

On 03/01/2011 07:42 PM, holger krekel wrote:
> I just turned
>
>      http://codespeak.net/svn/pypy/extradoc
>
> as well as the internal pypy-z repository READ-ONLY.
>
> Thanks to Ronny's conversion work we now have these repositories at bitbucket.org:
>
>      http://bitbucket.org/pypy/extradoc
>          talks, sprintinfo, the source of pypy.org website
>
> and
>
>      http://bitbucket.org/pypy/z (PRIVATE)
>          internal contracts/docs, available to long-time contributors
>
> I created a new pypy group on bitbucket, the "pypy:committers" group
> which has general read/write access to all public pypy repositories. It
> contains the people formerly listed individually for the pypy repo.  New
> committers should be added to that pypy:committers group so they get
> access to all pypy repos.  There also is a pypy:z group which also gets
> write/admin acccess to all repos but additionally can read/write the
> pypy/z one.

Makes perfect sense to me.

> Next in line for conversion are:
>
>      pyrepl ->  pypy/pyrepl  (discussed with Michael Hudson, the primary author already)
>      testrunner ->  to be flatly inlined into pypy repo
>      and then the remaining parts of pypy/lang
>
> After that there should be nothing left from PyPy that would
> require codespeak's svn.

There is also this:

http://codespeak.net/svn/pypy/benchmarks/

sorry for not thinking of it earlier.

Thanks for the good work to Ronny and you,

Carl Friedrich


From lac at openend.se  Thu Mar  3 17:34:31 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 3 Mar 2011 17:34:31 +0100
Subject: [pypy-dev] we need to fix the links on our blog posts.
Message-ID: <201103031634.p23GYVV6000421@theraft.openend.se>

http://morepypy.blogspot.com/2008/06/hi-all-some-news-from-jit-front.html
the link to the pyrolog paper is broken.

This works today, but needs to have its mimetype set
http://codespeak.net/svn/user/cfbolz/jitpl/paper/iclp08/jit_iclp08.pdf

But I don't know where Carl Friedrich's svn dir is migrating to from
codespeak, so this isn't a permanent fix.

This paper really should be in our papers, in extradoc, no?  I cannot find it there.

Laura


From lac at openend.se  Thu Mar  3 17:37:01 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 03 Mar 2011 17:37:01 +0100
Subject: [pypy-dev] we need to fix the links on our blog posts.
In-Reply-To: Message from Laura Creighton <lac@openend.se> of "Thu,
	03 Mar 2011 17:34:31 +0100."
	<201103031634.p23GYVV6000421@theraft.openend.se> 
References: <201103031634.p23GYVV6000421@theraft.openend.se> 
Message-ID: <201103031637.p23Gb1Fp000896@theraft.openend.se>

Do we need dcolish's bouncer to work for what used to be people's svn
directories on codespeak?

Laura


From arigo at tunes.org  Thu Mar  3 18:01:05 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 3 Mar 2011 09:01:05 -0800
Subject: [pypy-dev] we need to fix the links on our blog posts.
In-Reply-To: <201103031637.p23Gb1Fp000896@theraft.openend.se>
References: <lac@openend.se> <201103031634.p23GYVV6000421@theraft.openend.se>
	<201103031637.p23Gb1Fp000896@theraft.openend.se>
Message-ID: <AANLkTi=q36ZpO1SysZS8DSeRRJj3iAqtQWp98gQJ_EAL@mail.gmail.com>

Hi Laura,

On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton <lac at openend.se> wrote:
> Do we need dcolish's bouncer to work for what used to be people's svn
> directories on codespeak?

Actually, does bitbucket serve files straight from repositories, like
http://codespeak.net/svn/some/path did?  I tried a bit, but seem to
only get links to specific revisions of files, not "whatever the
latest revision is".


A bient?t,

Armin.


From anto.cuni at gmail.com  Thu Mar  3 18:11:08 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 3 Mar 2011 18:11:08 +0100
Subject: [pypy-dev] we need to fix the links on our blog posts.
In-Reply-To: <AANLkTi=q36ZpO1SysZS8DSeRRJj3iAqtQWp98gQJ_EAL@mail.gmail.com>
References: <lac@openend.se> <201103031634.p23GYVV6000421@theraft.openend.se>
	<201103031637.p23Gb1Fp000896@theraft.openend.se>
	<AANLkTi=q36ZpO1SysZS8DSeRRJj3iAqtQWp98gQJ_EAL@mail.gmail.com>
Message-ID: <AANLkTikb4SQwEURf7H6T8ZrQLaHmPA8LNrrf9sB=mxcv@mail.gmail.com>

you can get a link to "whatever the latest revision is" on bitbucket
by manually substituting the mercurial id with "default" inside the
link.

On 3/3/11, Armin Rigo <arigo at tunes.org> wrote:
> Hi Laura,
>
> On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton <lac at openend.se> wrote:
>> Do we need dcolish's bouncer to work for what used to be people's svn
>> directories on codespeak?
>
> Actually, does bitbucket serve files straight from repositories, like
> http://codespeak.net/svn/some/path did?  I tried a bit, but seem to
> only get links to specific revisions of files, not "whatever the
> latest revision is".
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From scott+pypy-dev at scottdial.com  Thu Mar  3 18:19:15 2011
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Thu, 03 Mar 2011 12:19:15 -0500
Subject: [pypy-dev] we need to fix the links on our blog posts.
In-Reply-To: <AANLkTi=q36ZpO1SysZS8DSeRRJj3iAqtQWp98gQJ_EAL@mail.gmail.com>
References: <lac@openend.se>
	<201103031634.p23GYVV6000421@theraft.openend.se>	<201103031637.p23Gb1Fp000896@theraft.openend.se>
	<AANLkTi=q36ZpO1SysZS8DSeRRJj3iAqtQWp98gQJ_EAL@mail.gmail.com>
Message-ID: <4D6FCD93.5050203@scottdial.com>

On 3/3/2011 12:01 PM, Armin Rigo wrote:
> On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton <lac at openend.se> wrote:
>> Do we need dcolish's bouncer to work for what used to be people's svn
>> directories on codespeak?
> 
> Actually, does bitbucket serve files straight from repositories, like
> http://codespeak.net/svn/some/path did?  I tried a bit, but seem to
> only get links to specific revisions of files, not "whatever the
> latest revision is".

AFAICT, you can substitute "tip" for the revision id in the URL and that
will track the latest. For example, you can browse to a file and copy
the raw link:

https://bitbucket.org/pypy/pypy/raw/b9887cd8aab0/README

And turn that into:

https://bitbucket.org/pypy/pypy/raw/tip/README

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu


From lac at openend.se  Fri Mar  4 19:45:30 2011
From: lac at openend.se (Laura Creighton)
Date: Fri, 04 Mar 2011 19:45:30 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Fri,
	04 Mar 2011 19:34:15 +0100."
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com> 
Message-ID: <201103041845.p24IjUDR015867@theraft.openend.se>

In a message of Fri, 04 Mar 2011 19:34:15 +0100, Miquel Torres writes:
>Hi,
>
>just a quick note to Laura and the rest to let you know that I am
>working on the issues you raised, and expect to have everything nailed
>down this weekend, just in time for PyCon.
>
>Miquel

Thank you very much.

Laura



From ademan555 at gmail.com  Fri Mar  4 03:02:20 2011
From: ademan555 at gmail.com (Dan Roberts)
Date: Thu, 3 Mar 2011 18:02:20 -0800
Subject: [pypy-dev] Yelp RSVP
In-Reply-To: <AANLkTi=q2m6pH8y1jWd4KOCmeqrUU5xy0tcaTMVBD4rc@mail.gmail.com>
References: <AANLkTi=q2m6pH8y1jWd4KOCmeqrUU5xy0tcaTMVBD4rc@mail.gmail.com>
Message-ID: <AANLkTinxhJ2o696daYDpE_9FF3LTiE3AUU1Dy6zjDJLg@mail.gmail.com>

Apparently I missed the memo about RSVPing to the Yelp talk, and we're
having trouble getting in the door, assuming the situation can be rectified,
can someone help out?

Cheers,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110303/7cf2fb8b/attachment.html>

From tbaldridge at gmail.com  Fri Mar  4 14:14:43 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 4 Mar 2011 07:14:43 -0600
Subject: [pypy-dev] EBNF translation error
Message-ID: <AANLkTi=8ixE6n7k5UTuwPoK7UnN=L2EXA=03H4EHt3bE@mail.gmail.com>

I hope this is the right place for this question. I'm trying to write
a simple JIT-enabled interpreter. However, I'm hitting a odd error
when it comes to translation. I've copied the JavaScript example
parser almost verbatim, but here's my issues:

 First of all, here's parser (almost 100% like the JS example):

from pypy.rlib.parsing.ebnfparse import parse_ebnf, make_parse_function
from pypy.rlib.parsing.parsing import ParseError, Rule
import py
import sys

GFILE = py.magic.autopath().dirpath().join("grammar.txt")

try:
	t = GFILE.read(mode="U")
	regexs, rules, ToAST = parse_ebnf(t)
except ParseError, e:
	print e.nice_error_message(filename=str(GFILE), source=t)
	raise

parsef = make_parse_function(regexs, rules, eof=True)

def parse(code):
	t = parsef(code)
	return ToAST().transform(t)

and my grammar:

STRING: "\\"[^\\\\"]*\\"";
SYMBOL: "[A-Za-z+-_*<>]+";
KEYWORD: ":[A-Za-z+-_*<>]+";
INTEGER: "\-?([0-9]+)";
DECIMAL: "\-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][\+\-]?[0-9]+)?";
IGNORE: " |\n|\t|,";
value: <KEYWORD> | <SYMBOL> | <STRING> | <DECIMAL> | <INTEGER> |
<hash> | <vector> | <sexps>;
hash: ["{"] (entry [","])* entry ["}"];
vector: ["["] value* ["]"];
entry: STRING [":"] value;
sexps: ["("] value+ [")"];

I'm doing the following to compile the code to c:

import parse
t = Translation(parse.parse)
t.annotate([str])
t.rtype()
t.compile_c()
>>>
<---snip---->
File "/home/tbaldridge/pypy/pypy/translator/c/genc.py", line 339, in
getentrypointptr
    self._wrapper = new_wrapper(self.entrypoint, self.translator)
  File "/home/tbaldridge/pypy/pypy/translator/llsupport/wrapper.py",
line 57, in new_wrapper
    r_to = pyobj_repr)
  File "/home/tbaldridge/pypy/pypy/rpython/rtyper.py", line 931, in convertvar
    (r_from, r_to))
pypy.rpython.error.TyperError: don't know how to convert from
<InstanceRepr for pypy.rlib.parsing.tree.Node> to <PyObjRepr *
PyObject>

What am I missing? This seemed so straight-forward....

Thanks,

Timothy


-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From arigo at tunes.org  Sat Mar  5 16:42:36 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 5 Mar 2011 07:42:36 -0800
Subject: [pypy-dev] Yelp RSVP
In-Reply-To: <AANLkTinxhJ2o696daYDpE_9FF3LTiE3AUU1Dy6zjDJLg@mail.gmail.com>
References: <AANLkTi=q2m6pH8y1jWd4KOCmeqrUU5xy0tcaTMVBD4rc@mail.gmail.com>
	<AANLkTinxhJ2o696daYDpE_9FF3LTiE3AUU1Dy6zjDJLg@mail.gmail.com>
Message-ID: <AANLkTimXqYewrU9-GfnDQYZ29gi5mb=t7=VHmf6am45o@mail.gmail.com>

Hi Dan,

On Thu, Mar 3, 2011 at 6:02 PM, Dan Roberts <ademan555 at gmail.com> wrote:
> Apparently I missed the memo about RSVPing to the Yelp talk, and we're
> having trouble getting in the door, assuming the situation can be rectified,
> can someone help out?

For (un?)interested readers of this mailing list, the situation was
eventually solved.


Armin


From arigo at tunes.org  Sat Mar  5 16:51:44 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 5 Mar 2011 07:51:44 -0800
Subject: [pypy-dev] EBNF translation error
In-Reply-To: <AANLkTi=8ixE6n7k5UTuwPoK7UnN=L2EXA=03H4EHt3bE@mail.gmail.com>
References: <AANLkTi=8ixE6n7k5UTuwPoK7UnN=L2EXA=03H4EHt3bE@mail.gmail.com>
Message-ID: <AANLkTimn0Y57TBmB4B9B07XZepytB+mGb2SDBsp3z_t5@mail.gmail.com>

Hi Timothy,

On Fri, Mar 4, 2011 at 5:14 AM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> I hope this is the right place for this question.
> <---snip---->
> pypy.rpython.error.TyperError: don't know how to convert from
> <InstanceRepr for pypy.rlib.parsing.tree.Node> to <PyObjRepr *
> PyObject>

Basically, it worked all right, except that you are trying to return
from the main translated function an instance of Node, which the
translator doesn't know how to convert back to Python level.  You can
only return simple types, like ints, strings, and so on.

In "real usage", i.e. translating a stand-alone program with
pypy/translator/goal/translate.py, the main translated function needs
anyway to have a specific signature, similar to the main() function in
C code: takes a list of strings (argv), and returns an integer (exit
code).


A bient?t,

Armin.


From tobami at googlemail.com  Fri Mar  4 19:34:15 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Fri, 4 Mar 2011 19:34:15 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201102280744.p1S7iFh7009804@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
Message-ID: <AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>

Hi,

just a quick note to Laura and the rest to let you know that I am
working on the issues you raised, and expect to have everything nailed
down this weekend, just in time for PyCon.

Miquel



2011/2/28 Laura Creighton <lac at openend.se>:
> Also, can you change every usage of the word 'less' on the axis of the
> graphs to 'smaller'. ?As far as I can tell, every time it is an
> incorrect and ungramatically usage of English. ?Sometimes the legend
> is 'seconds - less is better'. ?But seconds are inherantly countable,
> so it is incorrect English to say 'there are less seconds' -- rather
> that there are *fewer* seconds. ?Or you could change that legend to
> read 'time in seconds' -- because time is considered uncountable, so
> you have more or less time, not more or fewer time. ?Smaller would also
> work here.
>
> Ratios, however, such as on http://speed.pypy.org/comparison/
> cannot be 'more or less' or 'more or fewer'. ?They have to be greater
> (or larger) ?or smaller. ?Thus we need 'smaller' here.
>
> Laura
>
>


From holger at merlinux.eu  Sun Mar  6 20:00:04 2011
From: holger at merlinux.eu (holger krekel)
Date: Sun, 6 Mar 2011 19:00:04 +0000
Subject: [pypy-dev] pytest2 branch ready
Message-ID: <20110306190004.GU16231@merlinux.eu>

Hi all,

the pytest2 branch is ready to be merged.  The branch updates the
inlined pytest version to the pytest-2.0.2 dev5 release candidate.
I did some pypy test runs on buildbot and it looks good to me.  
Anything that speaks against merging now?

After merging one can install the "pytest" package indepedently and type
"py.test" or use the version that comes inlined with the PyPy source
tree and type "python pypy/test_all.py".

On a sidenote, i'd actually like to avoid shipping pytest and
py lib within the PyPy source tree especially now that there are
three extra items "pytest.py", "py" and "_pytest" cluttering
up the pypy root. (There are good reasons for this which
have not much to do with PyPy though).  FWIW I don't plan to
do many wild releases of pytest in the foreseeable future :)

best,
holger


From lac at openend.se  Mon Mar  7 00:43:38 2011
From: lac at openend.se (Laura Creighton)
Date: Mon, 07 Mar 2011 00:43:38 +0100
Subject: [pypy-dev] pytest2 branch ready
In-Reply-To: Message from holger krekel <holger@merlinux.eu> of "Sun,
	06 Mar 2011 19:00:04 GMT." <20110306190004.GU16231@merlinux.eu> 
References: <20110306190004.GU16231@merlinux.eu> 
Message-ID: <201103062343.p26Nhc0m006701@theraft.openend.se>

In a message of Sun, 06 Mar 2011 19:00:04 GMT, holger krekel writes:
<snip>

>On a sidenote, i'd actually like to avoid shipping pytest and
>py lib within the PyPy source tree especially now that there are
>three extra items "pytest.py", "py" and "_pytest" cluttering
>up the pypy root. (There are good reasons for this which
>have not much to do with PyPy though).  FWIW I don't plan to
>do many wild releases of pytest in the foreseeable future :)
>
>best,
>holger

I think I am confused.  If we begin shipping pypy without a py.test
that is known to work with it, then some year in the future some poor
soul (maybe me) who happens to want to run an old verion of pypy --
say a hypothetical pypy 1.5 that comes with no included py.test -- 
may find that your latest and greatest py.test does not run the tests.
And then it will be a matter of archeology to find out what version of
py.test does work with this release.  Is there something wrong with
this understanding?

Laura



From benjamin at python.org  Mon Mar  7 01:00:00 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 6 Mar 2011 18:00:00 -0600
Subject: [pypy-dev] pytest2 branch ready
In-Reply-To: <201103062343.p26Nhc0m006701@theraft.openend.se>
References: <holger@merlinux.eu> <20110306190004.GU16231@merlinux.eu>
	<201103062343.p26Nhc0m006701@theraft.openend.se>
Message-ID: <AANLkTimc-eXOUbCSqwR8CK8hrDGxZwaMVNrF-r7iWUBu@mail.gmail.com>

2011/3/6 Laura Creighton <lac at openend.se>:
> In a message of Sun, 06 Mar 2011 19:00:04 GMT, holger krekel writes:
> <snip>
>
>>On a sidenote, i'd actually like to avoid shipping pytest and
>>py lib within the PyPy source tree especially now that there are
>>three extra items "pytest.py", "py" and "_pytest" cluttering
>>up the pypy root. (There are good reasons for this which
>>have not much to do with PyPy though). ?FWIW I don't plan to
>>do many wild releases of pytest in the foreseeable future :)
>>
>>best,
>>holger
>
> I think I am confused. ?If we begin shipping pypy without a py.test
> that is known to work with it, then some year in the future some poor
> soul (maybe me) who happens to want to run an old verion of pypy --
> say a hypothetical pypy 1.5 that comes with no included py.test --
> may find that your latest and greatest py.test does not run the tests.
> And then it will be a matter of archeology to find out what version of
> py.test does work with this release. ?Is there something wrong with
> this understanding?

If you're using an old version of PyPy, there's probably going to be a
lot of archeology anyway. I don't see how py.test is any different
than another dep like gcc or boehm.



-- 
Regards,
Benjamin


From arigo at tunes.org  Mon Mar  7 01:36:17 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 6 Mar 2011 16:36:17 -0800
Subject: [pypy-dev] pytest2 branch ready
In-Reply-To: <AANLkTimc-eXOUbCSqwR8CK8hrDGxZwaMVNrF-r7iWUBu@mail.gmail.com>
References: <holger@merlinux.eu> <20110306190004.GU16231@merlinux.eu>
	<201103062343.p26Nhc0m006701@theraft.openend.se>
	<AANLkTimc-eXOUbCSqwR8CK8hrDGxZwaMVNrF-r7iWUBu@mail.gmail.com>
Message-ID: <AANLkTikObmR0=Xbhao27vdRBFqPShWZn80kq0aP9XOKd@mail.gmail.com>

Hi Benjamin,

On Sun, Mar 6, 2011 at 4:00 PM, Benjamin Peterson <benjamin at python.org> wrote:
> If you're using an old version of PyPy, there's probably going to be a
> lot of archeology anyway. I don't see how py.test is any different
> than another dep like gcc or boehm.

No, that's wrong.  Any version of PyPy supports any version of gcc
within reason, and any version of Boehm as long as it's not a too
buggy version.  But PyPy from, say, the month of January only works with the
version of the py lib that was the one included at that time.  Right
now I can say "hg up -r xyz" for some xyz from a few months ago and
just run py.test in it with no further installation, which is
essential from time to time (e.g. to run hg bisect).


A bient?t,

Armin.


From tbaldridge at gmail.com  Mon Mar  7 17:48:38 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Mon, 7 Mar 2011 10:48:38 -0600
Subject: [pypy-dev] GIL in non-Python languages
Message-ID: <AANLkTimTdq9VOHGpWnPSKKZ=kbf0x1o_E7kMx6h-Khk4@mail.gmail.com>

I'm in the process of writing a PyPy based Clojure interpreter/JIT.
One of the man tenants of Clojure is immutability, and one of the
other tenants is extreme co-currency. Starting soon I'd like to start
experimenting with co-currency in my interpreter. For those who aren't
familiar with Clojure, let me give an example:


(def my-agent (agent "Hello"))

(send my-agent #(str %1 " World"))

Send then takes the current state of my-agent and then calls the given
lambda function, passing it the current state of the agent (%1 points
to the first argument of the lambda). The lambda simply then concats
the two strings together.

So here's the deal. My entire interpreter is 100% thread safe. All my
variables are either immutable, or local to the given function. But
from what I'm hearing on IRC, it sounds like that a GIL will be
automatically generated in my interpreter after I run it through PyPy.
I know ripping the GIL out of Python is a major deal. But how about my
project? If I start with the concept of a GIL-less interpreter, will
PyPy get in my way? If so, how can I get it to "back-off" a bit?

Thanks,

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From benjamin at python.org  Mon Mar  7 17:50:50 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 7 Mar 2011 10:50:50 -0600
Subject: [pypy-dev] GIL in non-Python languages
In-Reply-To: <AANLkTimTdq9VOHGpWnPSKKZ=kbf0x1o_E7kMx6h-Khk4@mail.gmail.com>
References: <AANLkTimTdq9VOHGpWnPSKKZ=kbf0x1o_E7kMx6h-Khk4@mail.gmail.com>
Message-ID: <AANLkTi=At9B4Ex4+E2GoZMWqaHxYpnC9PBz3oS8WX1rs@mail.gmail.com>

2011/3/7 Timothy Baldridge <tbaldridge at gmail.com>:
> I'm in the process of writing a PyPy based Clojure interpreter/JIT.
> One of the man tenants of Clojure is immutability, and one of the
> other tenants is extreme co-currency. Starting soon I'd like to start
> experimenting with co-currency in my interpreter. For those who aren't
> familiar with Clojure, let me give an example:
>
>
> (def my-agent (agent "Hello"))
>
> (send my-agent #(str %1 " World"))
>
> Send then takes the current state of my-agent and then calls the given
> lambda function, passing it the current state of the agent (%1 points
> to the first argument of the lambda). The lambda simply then concats
> the two strings together.
>
> So here's the deal. My entire interpreter is 100% thread safe. All my
> variables are either immutable, or local to the given function. But
> from what I'm hearing on IRC, it sounds like that a GIL will be
> automatically generated in my interpreter after I run it through PyPy.
> I know ripping the GIL out of Python is a major deal. But how about my
> project? If I start with the concept of a GIL-less interpreter, will
> PyPy get in my way? If so, how can I get it to "back-off" a bit?

One major problem is that RPython its self is not really thread-safe.
For example, the gc is very non-concurrent. So, that would have to be
fixed.



-- 
Regards,
Benjamin


From tbaldridge at gmail.com  Mon Mar  7 18:08:20 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Mon, 7 Mar 2011 11:08:20 -0600
Subject: [pypy-dev] GIL in non-Python languages
In-Reply-To: <AANLkTi=At9B4Ex4+E2GoZMWqaHxYpnC9PBz3oS8WX1rs@mail.gmail.com>
References: <AANLkTimTdq9VOHGpWnPSKKZ=kbf0x1o_E7kMx6h-Khk4@mail.gmail.com>
	<AANLkTi=At9B4Ex4+E2GoZMWqaHxYpnC9PBz3oS8WX1rs@mail.gmail.com>
Message-ID: <AANLkTini_8u8Fw57cKH-2GHis5q5eKhJa8Hbg_rr3Q0O@mail.gmail.com>

Right, that would be an issue. In this case, I could probably start
with a reference counting GC....as I'm not exactly sure that it's
possible to get cyclic references with immutable objects. At least not
without allot of work...yeah, I'm pretty sure they are impossible to
code in Clojure. But someone may correct me there.

Anyway, reference counting would get rid of most of the parallel
issues with the GC, I could simply use a CAS or atomic instructions to
increment/decrement reference counts.

Timothy


> One major problem is that RPython its self is not really thread-safe.
> For example, the gc is very non-concurrent. So, that would have to be
> fixed.
>
>
>
> --
> Regards,
> Benjamin
>



-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From fijall at gmail.com  Mon Mar  7 18:39:43 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 7 Mar 2011 09:39:43 -0800
Subject: [pypy-dev] GIL in non-Python languages
In-Reply-To: <AANLkTini_8u8Fw57cKH-2GHis5q5eKhJa8Hbg_rr3Q0O@mail.gmail.com>
References: <AANLkTimTdq9VOHGpWnPSKKZ=kbf0x1o_E7kMx6h-Khk4@mail.gmail.com>
	<AANLkTi=At9B4Ex4+E2GoZMWqaHxYpnC9PBz3oS8WX1rs@mail.gmail.com>
	<AANLkTini_8u8Fw57cKH-2GHis5q5eKhJa8Hbg_rr3Q0O@mail.gmail.com>
Message-ID: <AANLkTikTA2Wz=pGakiikT-P1Y2CcMtmrk=uJHbOMoCDB@mail.gmail.com>

On Mon, Mar 7, 2011 at 9:08 AM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> Right, that would be an issue. In this case, I could probably start
> with a reference counting GC....as I'm not exactly sure that it's
> possible to get cyclic references with immutable objects. At least not
> without allot of work...yeah, I'm pretty sure they are impossible to
> code in Clojure. But someone may correct me there.

Hey.

Refcounting GC would be generated for you, have you chosen to use it,
but it'll be very inefficient and probably not thread safe.

What you can do *right now* is to use boehm GC which is thread safe,
but not very fast (much faster and much better than refcounting
though). I don't think there are any other issues that are not thread
safe in RPython than GC. Obviously it's like C - if you do something
wrong you'll segfault.

However, the true answer would be to work on a real GC that
thread-friendly and thread-safe. This is work, it's however fun :-)

>
> Anyway, reference counting would get rid of most of the parallel
> issues with the GC, I could simply use a CAS or atomic instructions to
> increment/decrement reference counts.
>
> Timothy
>
>
>> One major problem is that RPython its self is not really thread-safe.
>> For example, the gc is very non-concurrent. So, that would have to be
>> fixed.
>>
>>
>>
>> --
>> Regards,
>> Benjamin
>>
>
>
>
> --
> ?One of the main causes of the fall of the Roman Empire was
> that?lacking zero?they had no way to indicate successful termination
> of their C programs.?
> (Robert Firth)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From tobami at googlemail.com  Tue Mar  8 09:10:32 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Tue, 8 Mar 2011 09:10:32 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201103041845.p24IjUDR015867@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
Message-ID: <AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>

Hi,

I finished the changes to the speed.pypy.org home page last night, but
alas!, I didn't have time to deploy. I will do it later today and will
then ping you back.

The extra info provided is really nice as an overview, you will see ;-)


2011/3/4 Laura Creighton <lac at openend.se>:
> In a message of Fri, 04 Mar 2011 19:34:15 +0100, Miquel Torres writes:
>>Hi,
>>
>>just a quick note to Laura and the rest to let you know that I am
>>working on the issues you raised, and expect to have everything nailed
>>down this weekend, just in time for PyCon.
>>
>>Miquel
>
> Thank you very much.
>
> Laura
>
>


From tbaldridge at gmail.com  Tue Mar  8 17:12:51 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Tue, 8 Mar 2011 10:12:51 -0600
Subject: [pypy-dev] Basic math ops
Message-ID: <AANLkTi=MMoeZXr30NStxEFQhHsTsxNx_t4a2vst7KvSh@mail.gmail.com>

In my interpreter, I have several primitive wrapper objects:
IntObject, FloatObject, etc.

Currently I'm planning to create functions for every combination:

def add_int_float(i, f):
     return FloatObject(i.int_value() + f.float_value())

then I'd need some sort of dispatch code that would say

if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject):
    return add_int_float(arg1, arg2)

It seems like pypy should have something to do this already. I saw
some information on this but that seemed specific to the Python
interpreter.

What's the recommended procedure on this? Is there a way to tell pypy
"this object wraps a single value, so keep it unwrapped if you can",
or does pypy figure that out automagically?

Thanks,

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From lac at openend.se  Tue Mar  8 17:14:52 2011
From: lac at openend.se (Laura Creighton)
Date: Tue, 08 Mar 2011 17:14:52 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Tue,
	08 Mar 2011 09:10:32 +0100."
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com> 
Message-ID: <201103081614.p28GEqKT013585@theraft.openend.se>

In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes:
>Hi,
>
>I finished the changes to the speed.pypy.org home page last night, but
>alas!, I didn't have time to deploy. I will do it later today and will
>then ping you back.
>
>The extra info provided is really nice as an overview, you will see ;-)
>
>

Ah good.  Thank you very much.  We spent yesterday afternoon with
the Mozilla engineers, and I got to talk to the person who maintains
the benchmarks for tracemonkey.  He had timelines very much like ours.
There is one feature he has that I would like to have.  Take a look
at    the timeline for spectral.norm.  There are two spikes there.
Mozilla has lines like that too, though mostly it is because their
jit decides that the whole benchmark is bogus and optimises out all the
code.  So it takes 0 time.  oops.

At any rate, aside from knowing that something went horribly wrong with
that rev, you don't really need to know how wrong.  And by making the
graph display up to that point means that the dots where things really
do matter get crammed closer together than would otherwise be the case.
So he had a mode where things wehre displayed with an arbitrary value
at the bottom (in our coase it would be the top) which he could specify.
Then the graph would be replotted, with the outliers off the graph, but
making it easier to read the dots for the more normal cases.

Any chance we could do that too?

Laura


From fijall at gmail.com  Tue Mar  8 17:29:02 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 8 Mar 2011 08:29:02 -0800
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201103081614.p28GEqKT013585@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
Message-ID: <AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>

On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton <lac at openend.se> wrote:
> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes:
>>Hi,
>>
>>I finished the changes to the speed.pypy.org home page last night, but
>>alas!, I didn't have time to deploy. I will do it later today and will
>>then ping you back.
>>
>>The extra info provided is really nice as an overview, you will see ;-)
>>
>>
>
> Ah good. ?Thank you very much. ?We spent yesterday afternoon with
> the Mozilla engineers, and I got to talk to the person who maintains
> the benchmarks for tracemonkey. ?He had timelines very much like ours.
> There is one feature he has that I would like to have. ?Take a look
> at ? ?the timeline for spectral.norm. ?There are two spikes there.
> Mozilla has lines like that too, though mostly it is because their
> jit decides that the whole benchmark is bogus and optimises out all the
> code. ?So it takes 0 time. ?oops.
>
> At any rate, aside from knowing that something went horribly wrong with
> that rev, you don't really need to know how wrong. ?And by making the
> graph display up to that point means that the dots where things really
> do matter get crammed closer together than would otherwise be the case.
> So he had a mode where things wehre displayed with an arbitrary value
> at the bottom (in our coase it would be the top) which he could specify.
> Then the graph would be replotted, with the outliers off the graph, but
> making it easier to read the dots for the more normal cases.
>
> Any chance we could do that too?

Link maybe?

>
> Laura
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From benjamin at python.org  Tue Mar  8 18:14:27 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 8 Mar 2011 11:14:27 -0600
Subject: [pypy-dev] Basic math ops
In-Reply-To: <AANLkTi=MMoeZXr30NStxEFQhHsTsxNx_t4a2vst7KvSh@mail.gmail.com>
References: <AANLkTi=MMoeZXr30NStxEFQhHsTsxNx_t4a2vst7KvSh@mail.gmail.com>
Message-ID: <AANLkTimKbsSk__4ZKhwjJqT9b7ESutgOT2OcJ2FRVF5g@mail.gmail.com>

2011/3/8 Timothy Baldridge <tbaldridge at gmail.com>:
> In my interpreter, I have several primitive wrapper objects:
> IntObject, FloatObject, etc.
>
> Currently I'm planning to create functions for every combination:
>
> def add_int_float(i, f):
> ? ? return FloatObject(i.int_value() + f.float_value())
>
> then I'd need some sort of dispatch code that would say
>
> if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject):
> ? ?return add_int_float(arg1, arg2)
>
> It seems like pypy should have something to do this already. I saw
> some information on this but that seemed specific to the Python
> interpreter.

You could use multimethods. Look at the stuff in pypy/objspace/std/
for examples. (It's a bit messy.)

>
> What's the recommended procedure on this? Is there a way to tell pypy
> "this object wraps a single value, so keep it unwrapped if you can",
> or does pypy figure that out automagically?

The JIT will unwrap things automatically.



-- 
Regards,
Benjamin


From tobami at googlemail.com  Tue Mar  8 18:17:17 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Tue, 8 Mar 2011 18:17:17 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
Message-ID: <AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>

you mean this timeline, right?:
http://speed.pypy.org/timeline/?ben=spectral-norm

Because the December 22 result is so high, the yaxis maximum goes up
to 2.5, thus having less space for the more interesting < 1 range,
right?

Regarding mozilla, do you mean this site?: http://arewefastyet.com/
I can see their timelines have some holes, probably failed runs...

I see a problem with the approach you suggest. Entering an arbitrary
maximum yaxis number is not a good thing. I think the onus is there on
the benchmark infrastructure to not send results that aren't
statistically significant. See Javastats
(http://www.elis.ugent.be/en/JavaStats), or ReBench
(https://github.com/smarr/ReBench).

Something that can be done on the Codespeed side is to treat
differently points that have a too high stddev. In the aforementioned
spectral-norm timeline, the stddev "floor" is around 0.0050, while the
spike has a 0.30 stddev, much higher. A "strict" mode could be
implemented that invalidates or hides statistically unsound data.

Btw., I had written to the arewefastyet guys about the possibility of
configuring a Codespeed instance for them. We may yet see
collaboration there ;-)

Miquel


2011/3/8 Maciej Fijalkowski <fijall at gmail.com>:
> On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton <lac at openend.se> wrote:
>> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes:
>>>Hi,
>>>
>>>I finished the changes to the speed.pypy.org home page last night, but
>>>alas!, I didn't have time to deploy. I will do it later today and will
>>>then ping you back.
>>>
>>>The extra info provided is really nice as an overview, you will see ;-)
>>>
>>>
>>
>> Ah good. ?Thank you very much. ?We spent yesterday afternoon with
>> the Mozilla engineers, and I got to talk to the person who maintains
>> the benchmarks for tracemonkey. ?He had timelines very much like ours.
>> There is one feature he has that I would like to have. ?Take a look
>> at ? ?the timeline for spectral.norm. ?There are two spikes there.
>> Mozilla has lines like that too, though mostly it is because their
>> jit decides that the whole benchmark is bogus and optimises out all the
>> code. ?So it takes 0 time. ?oops.
>>
>> At any rate, aside from knowing that something went horribly wrong with
>> that rev, you don't really need to know how wrong. ?And by making the
>> graph display up to that point means that the dots where things really
>> do matter get crammed closer together than would otherwise be the case.
>> So he had a mode where things wehre displayed with an arbitrary value
>> at the bottom (in our coase it would be the top) which he could specify.
>> Then the graph would be replotted, with the outliers off the graph, but
>> making it easier to read the dots for the more normal cases.
>>
>> Any chance we could do that too?
>
> Link maybe?
>
>>
>> Laura
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>


From tobami at googlemail.com  Tue Mar  8 20:20:06 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Tue, 8 Mar 2011 20:20:06 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
Message-ID: <AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>

Ok, I just committed the changes.

They address two general cases:
- You want to know how fast PyPy is *now* compared to CPython in
different benchmark scenarios, or tasks.
- You want to know how PyPy has been *improving* overall over the last releases

That is now answered on the front page, and the reports are now much
less prominent (I didn't change the logic because it is something I
want to do properly, not just as a hack for speed.pypy).
- I have not yet addressed the "smaller is better" point.

I am aware that the wording of the "faster on average" needs to be
improved (I am discussing it with Holger even now ;). Please chime in
so that we can have a good paragraph that is informative and short
enough while at the same time not being misleading.

Miquel


2011/3/8 Miquel Torres <tobami at googlemail.com>:
> you mean this timeline, right?:
> http://speed.pypy.org/timeline/?ben=spectral-norm
>
> Because the December 22 result is so high, the yaxis maximum goes up
> to 2.5, thus having less space for the more interesting < 1 range,
> right?
>
> Regarding mozilla, do you mean this site?: http://arewefastyet.com/
> I can see their timelines have some holes, probably failed runs...
>
> I see a problem with the approach you suggest. Entering an arbitrary
> maximum yaxis number is not a good thing. I think the onus is there on
> the benchmark infrastructure to not send results that aren't
> statistically significant. See Javastats
> (http://www.elis.ugent.be/en/JavaStats), or ReBench
> (https://github.com/smarr/ReBench).
>
> Something that can be done on the Codespeed side is to treat
> differently points that have a too high stddev. In the aforementioned
> spectral-norm timeline, the stddev "floor" is around 0.0050, while the
> spike has a 0.30 stddev, much higher. A "strict" mode could be
> implemented that invalidates or hides statistically unsound data.
>
> Btw., I had written to the arewefastyet guys about the possibility of
> configuring a Codespeed instance for them. We may yet see
> collaboration there ;-)
>
> Miquel
>
>
> 2011/3/8 Maciej Fijalkowski <fijall at gmail.com>:
>> On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton <lac at openend.se> wrote:
>>> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes:
>>>>Hi,
>>>>
>>>>I finished the changes to the speed.pypy.org home page last night, but
>>>>alas!, I didn't have time to deploy. I will do it later today and will
>>>>then ping you back.
>>>>
>>>>The extra info provided is really nice as an overview, you will see ;-)
>>>>
>>>>
>>>
>>> Ah good. ?Thank you very much. ?We spent yesterday afternoon with
>>> the Mozilla engineers, and I got to talk to the person who maintains
>>> the benchmarks for tracemonkey. ?He had timelines very much like ours.
>>> There is one feature he has that I would like to have. ?Take a look
>>> at ? ?the timeline for spectral.norm. ?There are two spikes there.
>>> Mozilla has lines like that too, though mostly it is because their
>>> jit decides that the whole benchmark is bogus and optimises out all the
>>> code. ?So it takes 0 time. ?oops.
>>>
>>> At any rate, aside from knowing that something went horribly wrong with
>>> that rev, you don't really need to know how wrong. ?And by making the
>>> graph display up to that point means that the dots where things really
>>> do matter get crammed closer together than would otherwise be the case.
>>> So he had a mode where things wehre displayed with an arbitrary value
>>> at the bottom (in our coase it would be the top) which he could specify.
>>> Then the graph would be replotted, with the outliers off the graph, but
>>> making it easier to read the dots for the more normal cases.
>>>
>>> Any chance we could do that too?
>>
>> Link maybe?
>>
>>>
>>> Laura
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>
>


From holger at merlinux.eu  Tue Mar  8 20:27:31 2011
From: holger at merlinux.eu (holger krekel)
Date: Tue, 8 Mar 2011 19:27:31 +0000
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
References: <AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
Message-ID: <20110308192731.GD16231@merlinux.eu>

On Tue, Mar 08, 2011 at 20:20 +0100, Miquel Torres wrote:
> Ok, I just committed the changes.
> 
> They address two general cases:
> - You want to know how fast PyPy is *now* compared to CPython in
> different benchmark scenarios, or tasks.
> - You want to know how PyPy has been *improving* overall over the last releases
> 
> That is now answered on the front page, and the reports are now much
> less prominent (I didn't change the logic because it is something I
> want to do properly, not just as a hack for speed.pypy).
> - I have not yet addressed the "smaller is better" point.

Great work, Miquel.  I like it!

> I am aware that the wording of the "faster on average" needs to be
> improved (I am discussing it with Holger even now ;). Please chime in
> so that we can have a good paragraph that is informative and short
> enough while at the same time not being misleading.

Maybe something that reads like this::

    On average, PyPy trunk runs the benchmarks 3.3 times faster than CPython.
    The average is computed as the geometric mean of all benchmark timings.

Idea is to avoid the notion that "pypy is 3.3. times faster than cpython"
at this point because the benchmarks are not yet a balanced selection of
real life use cases.

best,
holger

> Miquel
> 
> 
> 2011/3/8 Miquel Torres <tobami at googlemail.com>:
> > you mean this timeline, right?:
> > http://speed.pypy.org/timeline/?ben=spectral-norm
> >
> > Because the December 22 result is so high, the yaxis maximum goes up
> > to 2.5, thus having less space for the more interesting < 1 range,
> > right?
> >
> > Regarding mozilla, do you mean this site?: http://arewefastyet.com/
> > I can see their timelines have some holes, probably failed runs...
> >
> > I see a problem with the approach you suggest. Entering an arbitrary
> > maximum yaxis number is not a good thing. I think the onus is there on
> > the benchmark infrastructure to not send results that aren't
> > statistically significant. See Javastats
> > (http://www.elis.ugent.be/en/JavaStats), or ReBench
> > (https://github.com/smarr/ReBench).
> >
> > Something that can be done on the Codespeed side is to treat
> > differently points that have a too high stddev. In the aforementioned
> > spectral-norm timeline, the stddev "floor" is around 0.0050, while the
> > spike has a 0.30 stddev, much higher. A "strict" mode could be
> > implemented that invalidates or hides statistically unsound data.
> >
> > Btw., I had written to the arewefastyet guys about the possibility of
> > configuring a Codespeed instance for them. We may yet see
> > collaboration there ;-)
> >
> > Miquel
> >
> >
> > 2011/3/8 Maciej Fijalkowski <fijall at gmail.com>:
> >> On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton <lac at openend.se> wrote:
> >>> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes:
> >>>>Hi,
> >>>>
> >>>>I finished the changes to the speed.pypy.org home page last night, but
> >>>>alas!, I didn't have time to deploy. I will do it later today and will
> >>>>then ping you back.
> >>>>
> >>>>The extra info provided is really nice as an overview, you will see ;-)
> >>>>
> >>>>
> >>>
> >>> Ah good. ?Thank you very much. ?We spent yesterday afternoon with
> >>> the Mozilla engineers, and I got to talk to the person who maintains
> >>> the benchmarks for tracemonkey. ?He had timelines very much like ours.
> >>> There is one feature he has that I would like to have. ?Take a look
> >>> at ? ?the timeline for spectral.norm. ?There are two spikes there.
> >>> Mozilla has lines like that too, though mostly it is because their
> >>> jit decides that the whole benchmark is bogus and optimises out all the
> >>> code. ?So it takes 0 time. ?oops.
> >>>
> >>> At any rate, aside from knowing that something went horribly wrong with
> >>> that rev, you don't really need to know how wrong. ?And by making the
> >>> graph display up to that point means that the dots where things really
> >>> do matter get crammed closer together than would otherwise be the case.
> >>> So he had a mode where things wehre displayed with an arbitrary value
> >>> at the bottom (in our coase it would be the top) which he could specify.
> >>> Then the graph would be replotted, with the outliers off the graph, but
> >>> making it easier to read the dots for the more normal cases.
> >>>
> >>> Any chance we could do that too?
> >>
> >> Link maybe?
> >>
> >>>
> >>> Laura
> >>> _______________________________________________
> >>> pypy-dev at codespeak.net
> >>> http://codespeak.net/mailman/listinfo/pypy-dev
> >>>
> >>
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From chef at ghum.de  Tue Mar  8 20:28:47 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Tue, 8 Mar 2011 20:28:47 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
Message-ID: <AANLkTi=pHuVN5XrVQf8whaODwkps9ERGKbxVff94BDwh@mail.gmail.com>

Miquel,

that looks great! Could you please add an additional PHB-Explanation
of "orange = cPython (xxx), blue = PyPy" ?

best wishes,

Harald


-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare


From drsalists at gmail.com  Tue Mar  8 20:39:41 2011
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 8 Mar 2011 11:39:41 -0800
Subject: [pypy-dev] Amusing graph
Message-ID: <AANLkTinbwQBu1Aj9cKpa9if9=yB6BCeScGyHd5_tivRC@mail.gmail.com>

I thought there might be some people here who'd be interested in this graph:

http://stromberg.dnsalias.org/~dstromberg/backshift/

It shows pypy performing very well when compared to a number of other Python
runtime systems, including cpython+cython (2 and 3) and cpython+psyco (just
2 of course).  It's just one part (the critical section) of one application,
but still...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110308/3c5b0937/attachment.html>

From janzert at janzert.com  Tue Mar  8 23:11:17 2011
From: janzert at janzert.com (Janzert)
Date: Tue, 08 Mar 2011 17:11:17 -0500
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
References: <lac@openend.se>
	<201102271359.p1RDxkpX012065@theraft.openend.se>	<201102271538.p1RFc3rG020721@theraft.openend.se>	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>	<201102280744.p1S7iFh7009804@theraft.openend.se>	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>	<201103041845.p24IjUDR015867@theraft.openend.se>	<tobami@googlemail.com>	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>	<201103081614.p28GEqKT013585@theraft.openend.se>	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
Message-ID: <il69i2$jen$1@dough.gmane.org>

On 3/8/2011 12:17 PM, Miquel Torres wrote:
>
> I see a problem with the approach you suggest. Entering an arbitrary
> maximum yaxis number is not a good thing. I think the onus is there on
> the benchmark infrastructure to not send results that aren't
> statistically significant. See Javastats
> (http://www.elis.ugent.be/en/JavaStats), or ReBench
> (https://github.com/smarr/ReBench).
>

I think the problem is that even a statistically significant result can 
be arbitrarily bad (e.g. if a major regression is checked in). It does 
seem like it would be really nice to have the option of viewing with a 
scale such that the maximum displayed is something like say 2 standard 
deviations above the mean of displayed results.

Janzert



From lac at openend.se  Wed Mar  9 05:12:56 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 09 Mar 2011 05:12:56 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Tue,
	08 Mar 2011 18:17:17 +0100."
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com> 
Message-ID: <201103090412.p294CuMS011453@theraft.openend.se>

In a message of Tue, 08 Mar 2011 18:17:17 +0100, Miquel Torres writes:
>you mean this timeline, right?:
>http://speed.pypy.org/timeline/?ben=3Dspectral-norm
>
>Because the December 22 result is so high, the yaxis maximum goes up
>to 2.5, thus having less space for the more interesting < 1 range,
>right?

yes

>
>Regarding mozilla, do you mean this site?: http://arewefastyet.com/
>I can see their timelines have some holes, probably failed runs...

I was seeing something else, and I don't have a url. I think that what
I was seeing is what they use to make the arewefastyet.com site.

>I see a problem with the approach you suggest. Entering an arbitrary
>maximum yaxis number is not a good thing. I think the onus is there on
>the benchmark infrastructure to not send results that aren't
>statistically significant. See Javastats
>(http://www.elis.ugent.be/en/JavaStats), or ReBench
>(https://github.com/smarr/ReBench).

I don't think you understand what I want. Sorry if I was unclear.
I am fine with the way that the benchmarks are displayed right now,
but I want a way to dynamically do there and say, I want to throw
away all data that is higher than a certain figure, or lower than
a certain one, because right now I am onoy interested in results
in a certain range.

I'm not looking to change what the benchmark says for everybody
who looks at it, or change how it is presented in general.  I just
want a way to zoom in and only see results in the range that 
interests me.  You and anybody else might have a different
range that interests you, and you should be free to get this as well.

>Something that can be done on the Codespeed side is to treat
>differently points that have a too high stddev. In the aforementioned
>spectral-norm timeline, the stddev "floor" is around 0.0050, while the
>spike has a 0.30 stddev, much higher. A "strict" mode could be
>implemented that invalidates or hides statistically unsound data.

The problem is that I want to throw away arbitrary amounts of data
regardless of whether they are statistically significant or not,
on the basis of I know what I want to see, and this other stuff
is getting in the way or being distracting?.

>Btw., I had written to the arewefastyet guys about the possibility of
>configuring a Codespeed instance for them. We may yet see
>collaboration there ;-)
>
>Miquel

Laura


From lac at openend.se  Wed Mar  9 06:17:41 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 09 Mar 2011 06:17:41 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Tue,
	08 Mar 2011 20:20:06 +0100."
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com> 
Message-ID: <201103090517.p295HfDa017396@theraft.openend.se>

In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
>Ok, I just committed the changes.
>
>They address two general cases:
>- You want to know how fast PyPy is *now* compared to CPython in
>different benchmark scenarios, or tasks.
>- You want to know how PyPy has been *improving* overall over the last re
>le=
>ases
>
>That is now answered on the front page, and the reports are now much
>less prominent (I didn't change the logic because it is something I
>want to do properly, not just as a hack for speed.pypy).
>- I have not yet addressed the "smaller is better" point.
>
>I am aware that the wording of the "faster on average" needs to be
>improved (I am discussing it with Holger even now ;). Please chime in
>so that we can have a good paragraph that is informative and short
>enough while at the same time not being misleading.
>
>Miquel

The graphic is lovely.

you have a s?pelling error s/taks/task/.

Many of us are at PyCon now, so working on wording may not be
something we have time for now.  I am not sure that the geometric
mean of all benchmarks give you anything meaningful, so I would
have avoided saying anything like that.  More specifically, I think
that there is a division between some highly mathematical programs,
where you might get a speedup of 20 to 30 times CPython, and the
benchmarks whcih I find much more meaningful, those that represent
actualy python programs -- where I think we are typically only between 
2 and 3 times faster.

The only reason to have some of the benchmarks is because they are
well known.  So people expect them.   But doing very well on them is
not actually all that significant -- it would be easy to write something
that is great and running these contrived, synthetic benchmarks, but
really lousy at running real python code.

And I don't think that you can use the geometric mean to prove a thing
with this.  So I think talking about it makes us look bad -- we are
making claims based on either bad science, or pseudo-science.

And I want the 'lestest results' stuff gone from the front page.
It's as misleading as ever.  And I have done a poll of developers.  it's
not just me.  Nobody finds it valuable.  Because things stay red forever,
we all ignore it all the time and go directly to the raw results of
runs, which is what we are interested in.  This also tells us of
improvements, which we are also interested in, because unexpected
improvements often mean something is very wrong.  

The whole thing has the same problem as those popup windows 'do you
really want to delete that file? confirm y/n'.  You get used to typing
y.  Then you do it when you meant not to save the file.  The red pages
get ignored for precisely the same reason.  We're all used to all the
red, which generally refer to problems which were already fixed, and
thus are things that we _should_ ignore.  So we end up ignoring it all,
all of the time.  So it therefore doesn't serve its purpose of reminding
developers of anything.  

The general public, however, reads the thing and thinks -- latest
results, pypy has just gone to hell, look at all those slowdowns.
Explaining that they aren't current, or latest results, but instead
'sometime in the past when we were bad once' is getting irritating.
Can you please move it someplace else so I don't have to have this
conversation with pycon attendees any more?

Later, when pycon is over, we can discuss and work out a better design
for informing developers that a given build may have broken things.  This
way is not working.

Laura


From tobami at googlemail.com  Wed Mar  9 09:14:53 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 09:14:53 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTi=pHuVN5XrVQf8whaODwkps9ERGKbxVff94BDwh@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<AANLkTi=pHuVN5XrVQf8whaODwkps9ERGKbxVff94BDwh@mail.gmail.com>
Message-ID: <AANLkTinwOy9D-CgRO0SmrjfyBUTGF1o7q6ysaNJxXaCc@mail.gmail.com>

I added a legend to the first plot


2011/3/8 Massa, Harald Armin <chef at ghum.de>:
> Miquel,
>
> that looks great! Could you please add an additional PHB-Explanation
> of "orange = cPython (xxx), blue = PyPy" ?
>
> best wishes,
>
> Harald
>
>
> --
> GHUM GmbH
> Harald Armin Massa
> Spielberger Stra?e 49
> 70435 Stuttgart
> 0173/9409607
>
> Amtsgericht Stuttgart, HRB 734971
> -
> persuadere.
> et programmare
>


From tobami at googlemail.com  Wed Mar  9 09:26:29 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 09:26:29 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201103090412.p294CuMS011453@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<201103090412.p294CuMS011453@theraft.openend.se>
Message-ID: <AANLkTi=Y2v3Mq0ge4PLZ4093av=s_fpgV2yGuN4Crbay@mail.gmail.com>

I see.

There is an easy solution for that, at least for the moment: enabling
zooming. I just did that, and you can now use zooming in a timeline
plot to select a narrower yaxis range or just view a particular area
in detail. A single click resets the zoom level.

If that is not enough, we can discuss a better solution when you have more time.



2011/3/9 Laura Creighton <lac at openend.se>:
> In a message of Tue, 08 Mar 2011 18:17:17 +0100, Miquel Torres writes:
>>you mean this timeline, right?:
>>http://speed.pypy.org/timeline/?ben=3Dspectral-norm
>>
>>Because the December 22 result is so high, the yaxis maximum goes up
>>to 2.5, thus having less space for the more interesting < 1 range,
>>right?
>
> yes
>
>>
>>Regarding mozilla, do you mean this site?: http://arewefastyet.com/
>>I can see their timelines have some holes, probably failed runs...
>
> I was seeing something else, and I don't have a url. I think that what
> I was seeing is what they use to make the arewefastyet.com site.
>
>>I see a problem with the approach you suggest. Entering an arbitrary
>>maximum yaxis number is not a good thing. I think the onus is there on
>>the benchmark infrastructure to not send results that aren't
>>statistically significant. See Javastats
>>(http://www.elis.ugent.be/en/JavaStats), or ReBench
>>(https://github.com/smarr/ReBench).
>
> I don't think you understand what I want. Sorry if I was unclear.
> I am fine with the way that the benchmarks are displayed right now,
> but I want a way to dynamically do there and say, I want to throw
> away all data that is higher than a certain figure, or lower than
> a certain one, because right now I am onoy interested in results
> in a certain range.
>
> I'm not looking to change what the benchmark says for everybody
> who looks at it, or change how it is presented in general. ?I just
> want a way to zoom in and only see results in the range that
> interests me. ?You and anybody else might have a different
> range that interests you, and you should be free to get this as well.
>
>>Something that can be done on the Codespeed side is to treat
>>differently points that have a too high stddev. In the aforementioned
>>spectral-norm timeline, the stddev "floor" is around 0.0050, while the
>>spike has a 0.30 stddev, much higher. A "strict" mode could be
>>implemented that invalidates or hides statistically unsound data.
>
> The problem is that I want to throw away arbitrary amounts of data
> regardless of whether they are statistically significant or not,
> on the basis of I know what I want to see, and this other stuff
> is getting in the way or being distracting?.
>
>>Btw., I had written to the arewefastyet guys about the possibility of
>>configuring a Codespeed instance for them. We may yet see
>>collaboration there ;-)
>>
>>Miquel
>
> Laura
>


From tobami at googlemail.com  Wed Mar  9 09:46:14 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 09:46:14 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201103090517.p295HfDa017396@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
Message-ID: <AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>

> And I want the 'lestest results' stuff gone from the front page.
> It's as misleading as ever.  And I have done a poll of developers.  it's
> not just me.  Nobody finds it valuable.  Because things stay red forever,
> we all ignore it all the time and go directly to the raw results of
> runs, which is what we are interested in.  This also tells us of
> improvements, which we are also interested in, because unexpected
> improvements often mean something is very wrong.
Ok, that is pretty clear.

> Explaining that they aren't current, or latest results, but instead
> 'sometime in the past when we were bad once' is getting irritating.
> Can you please move it someplace else so I don't have to have this
> conversation with pycon attendees any more?
Sorry about that.
I have removed it from the site.

> Later, when pycon is over, we can discuss and work out a better design
> for informing developers that a given build may have broken things.  This
> way is not working.

Yes, we can figure out a better approach after PyCon.

> And I don't think that you can use the geometric mean to prove a thing
> with this.  So I think talking about it makes us look bad -- we are
> making claims based on either bad science, or pseudo-science.
I agree that it wasn't the best explanation. I have removed most text,
so that it doesn't explicitly say that it means PyPy *is* X times
faster. It now just states that the geometric mean is X times faster.
If it is still too much, we can remove all text. But then some people
won't understand the numbers.

We can also remove the second plot.

The mean of a set of benchmarks that does not represent a balanced
real-world task mix is of course not very good. But then again the
world is complicated. And, as a normal Python developer, I find the
second plot extremely interesting, because it gives me a ballpark idea
of where the PyPy project is heading to. Before I decide to use PyPy
in production instead of CPython, I will do my own testing for *my*
application, but I assure you that not having a ballpark average won't
be a plus in considering to invest the time to test and adopt PyPy.
But that is my opinion of course ;-)

Have fun at PyCon!
Miquel


2011/3/9 Laura Creighton <lac at openend.se>:
> In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
>>Ok, I just committed the changes.
>>
>>They address two general cases:
>>- You want to know how fast PyPy is *now* compared to CPython in
>>different benchmark scenarios, or tasks.
>>- You want to know how PyPy has been *improving* overall over the last re
>>le=
>>ases
>>
>>That is now answered on the front page, and the reports are now much
>>less prominent (I didn't change the logic because it is something I
>>want to do properly, not just as a hack for speed.pypy).
>>- I have not yet addressed the "smaller is better" point.
>>
>>I am aware that the wording of the "faster on average" needs to be
>>improved (I am discussing it with Holger even now ;). Please chime in
>>so that we can have a good paragraph that is informative and short
>>enough while at the same time not being misleading.
>>
>>Miquel
>
> The graphic is lovely.
>
> you have a s?pelling error s/taks/task/.
>
> Many of us are at PyCon now, so working on wording may not be
> something we have time for now. ?I am not sure that the geometric
> mean of all benchmarks give you anything meaningful, so I would
> have avoided saying anything like that. ?More specifically, I think
> that there is a division between some highly mathematical programs,
> where you might get a speedup of 20 to 30 times CPython, and the
> benchmarks whcih I find much more meaningful, those that represent
> actualy python programs -- where I think we are typically only between
> 2 and 3 times faster.
>
> The only reason to have some of the benchmarks is because they are
> well known. ?So people expect them. ? But doing very well on them is
> not actually all that significant -- it would be easy to write something
> that is great and running these contrived, synthetic benchmarks, but
> really lousy at running real python code.
>
> And I don't think that you can use the geometric mean to prove a thing
> with this. ?So I think talking about it makes us look bad -- we are
> making claims based on either bad science, or pseudo-science.
>
> And I want the 'lestest results' stuff gone from the front page.
> It's as misleading as ever. ?And I have done a poll of developers. ?it's
> not just me. ?Nobody finds it valuable. ?Because things stay red forever,
> we all ignore it all the time and go directly to the raw results of
> runs, which is what we are interested in. ?This also tells us of
> improvements, which we are also interested in, because unexpected
> improvements often mean something is very wrong.
>
> The whole thing has the same problem as those popup windows 'do you
> really want to delete that file? confirm y/n'. ?You get used to typing
> y. ?Then you do it when you meant not to save the file. ?The red pages
> get ignored for precisely the same reason. ?We're all used to all the
> red, which generally refer to problems which were already fixed, and
> thus are things that we _should_ ignore. ?So we end up ignoring it all,
> all of the time. ?So it therefore doesn't serve its purpose of reminding
> developers of anything.
>
> The general public, however, reads the thing and thinks -- latest
> results, pypy has just gone to hell, look at all those slowdowns.
> Explaining that they aren't current, or latest results, but instead
> 'sometime in the past when we were bad once' is getting irritating.
> Can you please move it someplace else so I don't have to have this
> conversation with pycon attendees any more?
>
> Later, when pycon is over, we can discuss and work out a better design
> for informing developers that a given build may have broken things. ?This
> way is not working.
>
> Laura
>


From greg at quora.com  Wed Mar  9 13:29:15 2011
From: greg at quora.com (Greg Price)
Date: Wed, 9 Mar 2011 04:29:15 -0800
Subject: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty
	list-assignments
Message-ID: <AANLkTinrk9tJbBdkup40zP09-f_4Cs3x5DOOKqvaUs2z@mail.gmail.com>

This same patch is on bitbucket at https://bitbucket.org/price/pypy,
where I've sent a pull request. Holger Krekel suggested on IRC that I
send mail here. If others have different preferences for how to submit
a patch, let me know.

Before this patch, "[] = []" would abort the interpreter, with a
segmentation fault if in pypy-c. A segmentation fault is always bad,
but in this case further the code is valid Python, if not very useful.
(In my commit message on bitbucket, I incorrectly said it only affects
invalid Python, like "[] += []".)

Greg


diff -r eb44d135f334 -r 0db4ac049ea2 pypy/interpreter/astcompiler/asthelpers.py
--- a/pypy/interpreter/astcompiler/asthelpers.py Tue Mar 08 11:14:36 2011 -0800
+++ b/pypy/interpreter/astcompiler/asthelpers.py Wed Mar 09 03:26:54 2011 -0800
@@ -40,9 +40,10 @@
?? ? ? ? return self.elts

?? ? def set_context(self, ctx):
- ? ? ? ?for elt in self.elts:
- ? ? ? ? ? ?elt.set_context(ctx)
- ? ? ? ?self.ctx = ctx
+ ? ? ? ?if self.elts:
+ ? ? ? ? ? ?for elt in self.elts:
+ ? ? ? ? ? ? ? ?elt.set_context(ctx)
+ ? ? ? ? ? ?self.ctx = ctx


?class __extend__(ast.Attribute):
diff -r eb44d135f334 -r 0db4ac049ea2
pypy/interpreter/astcompiler/test/test_compiler.py
--- a/pypy/interpreter/astcompiler/test/test_compiler.py Tue Mar 08
11:14:36 2011 -0800
+++ b/pypy/interpreter/astcompiler/test/test_compiler.py Wed Mar 09
03:26:54 2011 -0800
@@ -70,6 +70,9 @@

?? ? st = simple_test

+ ? ?def error_test(self, source, exc_type):
+ ? ? ? ?py.test.raises(exc_type, self.simple_test, source, None, None)
+
?? ? def test_long_jump(self):
?? ? ? ? func = """def f(x):
?? ? y = 0
@@ -98,11 +101,13 @@
?? ? ? ? self.simple_test(stmt, "type(x)", int)

?? ? def test_tuple_assign(self):
+ ? ? ? ?yield self.error_test, "() = 1", SyntaxError
?? ? ? ? yield self.simple_test, "x,= 1,", "x", 1
?? ? ? ? yield self.simple_test, "x,y = 1,2", "x,y", (1, 2)
?? ? ? ? yield self.simple_test, "x,y,z = 1,2,3", "x,y,z", (1, 2, 3)
?? ? ? ? yield self.simple_test, "x,y,z,t = 1,2,3,4", "x,y,z,t", (1, 2, 3, 4)
?? ? ? ? yield self.simple_test, "x,y,x,t = 1,2,3,4", "x,y,t", (3, 2, 4)
+ ? ? ? ?yield self.simple_test, "[] = []", "1", 1
?? ? ? ? yield self.simple_test, "[x]= 1,", "x", 1
?? ? ? ? yield self.simple_test, "[x,y] = [1,2]", "x,y", (1, 2)
?? ? ? ? yield self.simple_test, "[x,y,z] = 1,2,3", "x,y,z", (1, 2, 3)


From fijall at gmail.com  Wed Mar  9 13:36:26 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Mar 2011 04:36:26 -0800
Subject: [pypy-dev] Basic math ops
In-Reply-To: <AANLkTimKbsSk__4ZKhwjJqT9b7ESutgOT2OcJ2FRVF5g@mail.gmail.com>
References: <AANLkTi=MMoeZXr30NStxEFQhHsTsxNx_t4a2vst7KvSh@mail.gmail.com>
	<AANLkTimKbsSk__4ZKhwjJqT9b7ESutgOT2OcJ2FRVF5g@mail.gmail.com>
Message-ID: <AANLkTikr_TQCrUGqYJNbtF+yA3rdDg7FcPEYdHoV=dsi@mail.gmail.com>

On Tue, Mar 8, 2011 at 9:14 AM, Benjamin Peterson <benjamin at python.org> wrote:
> 2011/3/8 Timothy Baldridge <tbaldridge at gmail.com>:
>> In my interpreter, I have several primitive wrapper objects:
>> IntObject, FloatObject, etc.
>>
>> Currently I'm planning to create functions for every combination:
>>
>> def add_int_float(i, f):
>> ? ? return FloatObject(i.int_value() + f.float_value())
>>
>> then I'd need some sort of dispatch code that would say
>>
>> if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject):
>> ? ?return add_int_float(arg1, arg2)
>>
>> It seems like pypy should have something to do this already. I saw
>> some information on this but that seemed specific to the Python
>> interpreter.
>
> You could use multimethods. Look at the stuff in pypy/objspace/std/
> for examples. (It's a bit messy.)
>
>>
>> What's the recommended procedure on this? Is there a way to tell pypy
>> "this object wraps a single value, so keep it unwrapped if you can",
>> or does pypy figure that out automagically?
>
> The JIT will unwrap things automatically.
>

For JIT to be happier you can however say:

class A(object):
  _immutable_fields_ = ['value']

so it'll know that fields don't change once the object is constructed

>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From fijall at gmail.com  Wed Mar  9 13:39:59 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Mar 2011 04:39:59 -0800
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
Message-ID: <AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>

Hey.

Awesome job Miquel, I love it. This is exactly what we wanted


From tobami at googlemail.com  Wed Mar  9 14:10:29 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 14:10:29 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
Message-ID: <AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>

Cool! ;-)

PS: Currently, only 20 benchmarks are shown (and averaged!) because
older pypy revisions (mostly PyPy 1.3) don't have numbers for the
newest benchmarks. If Maciej reruns those revisions the rest will be
displayed ;-)


2011/3/9 Maciej Fijalkowski <fijall at gmail.com>:
> Hey.
>
> Awesome job Miquel, I love it. This is exactly what we wanted
>


From chef at ghum.de  Wed Mar  9 14:19:22 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Wed, 9 Mar 2011 14:19:22 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
Message-ID: <AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>

I really, really like the new display!

And it motivated me to dig into the data ... which is a great result on its own.

The first question for myself was "hey, why is it slow on
slowspitfire, and, btw, what is slowspitfire? Could that be something
that my application does, too?"

But I was unable to find out what slowspitfire is doing ... I found
spitfire, which does some HTML templating stuff, and deducted, that
slowspitfire will do some slow HTML templating stuff. Where did I
click wrong? Is there a path down to the slowspitfire.py file or an
explanation what slowspitfire is doing?

Harald


-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare


From fijall at gmail.com  Wed Mar  9 14:21:50 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Mar 2011 08:21:50 -0500
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
	<AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
Message-ID: <AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>

On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef at ghum.de> wrote:
> I really, really like the new display!
>
> And it motivated me to dig into the data ... which is a great result on its own.
>
> The first question for myself was "hey, why is it slow on
> slowspitfire, and, btw, what is slowspitfire? Could that be something
> that my application does, too?"
>
> But I was unable to find out what slowspitfire is doing ... I found
> spitfire, which does some HTML templating stuff, and deducted, that
> slowspitfire will do some slow HTML templating stuff. Where did I
> click wrong? Is there a path down to the slowspitfire.py file or an
> explanation what slowspitfire is doing?
>
> Harald
>

https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py

It's creating a very large template table (1000x1000 elements I think)

The explanation "why it's slow" is a bit longish. It's a combination
of factors, including very long lists with GC objects in it, using
''.join(list) instead of cStringIO (the latter is faster and yes, it
is a bug) and a bit of other factors.

>
> --
> GHUM GmbH
> Harald Armin Massa
> Spielberger Stra?e 49
> 70435 Stuttgart
> 0173/9409607
>
> Amtsgericht Stuttgart, HRB 734971
> -
> persuadere.
> et programmare
>


From fijall at gmail.com  Wed Mar  9 14:53:47 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Mar 2011 08:53:47 -0500
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
	<AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
	<AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>
Message-ID: <AANLkTimS2CpGwpXUYzp=RGX-oYb7=J5ydG7iAuYCf11=@mail.gmail.com>

Hey Miquel.

A small feature request ;-) Can we get favicon?

Cheers,
fijal


From lac at openend.se  Wed Mar  9 14:58:16 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 09 Mar 2011 14:58:16 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Wed,
	09 Mar 2011 09:26:29 +0100."
	<AANLkTi=Y2v3Mq0ge4PLZ4093av=s_fpgV2yGuN4Crbay@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<201103090412.p294CuMS011453@theraft.openend.se>
	<AANLkTi=Y2v3Mq0ge4PLZ4093av=s_fpgV2yGuN4Crbay@mail.gmail.com> 
Message-ID: <201103091358.p29DwGYw032370@theraft.openend.se>

In a message of Wed, 09 Mar 2011 09:26:29 +0100, Miquel Torres writes:
>I see.
>
>There is an easy solution for that, at least for the moment: enabling
>zooming. I just did that, and you can now use zooming in a timeline
>plot to select a narrower yaxis range or just view a particular area
>in detail. A single click resets the zoom level.
>
>If that is not enough, we can discuss a better solution when you have mor
>e time.

This is terrific.  Thank you very much.

Laura


From anto.cuni at gmail.com  Wed Mar  9 14:59:19 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 09 Mar 2011 14:59:19 +0100
Subject: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty
	list-assignments
In-Reply-To: <AANLkTinrk9tJbBdkup40zP09-f_4Cs3x5DOOKqvaUs2z@mail.gmail.com>
References: <AANLkTinrk9tJbBdkup40zP09-f_4Cs3x5DOOKqvaUs2z@mail.gmail.com>
Message-ID: <4D7787B7.70509@gmail.com>

On 09/03/11 13:29, Greg Price wrote:
> This same patch is on bitbucket at https://bitbucket.org/price/pypy,
> where I've sent a pull request. Holger Krekel suggested on IRC that I
> send mail here. If others have different preferences for how to submit
> a patch, let me know.

Hi Greg,
I have pushed your commit upstream, thanks for help!

ciao,
anto


From pgiarrusso at mathematik.uni-marburg.de  Wed Mar  9 15:15:37 2011
From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso)
Date: Wed, 9 Mar 2011 15:15:37 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
	<AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
	<AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>
Message-ID: <AANLkTiky6d=SA1NwRoDLXpcqe+19sisXwK9ZTyG65qV_@mail.gmail.com>

Hi all,

On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef at ghum.de> wrote:
> > I really, really like the new display!
Congratulations to Miquel for the great work!

A minor comment about the homepage: the answer to "How fast is PyPy?"
would better stay close to the question, i.e. above Plot 1 (at least
with the current wording).

> > But I was unable to find out what slowspitfire is doing ... I found
> > spitfire, which does some HTML templating stuff, and deducted, that
> > slowspitfire will do some slow HTML templating stuff. Where did I
> > click wrong?

> > Is there a path down to the slowspitfire.py file or an
> > explanation what slowspitfire is doing?

Is there a place the answer this information to the website? I propose
a link to the source in each benchmark page.
Additionally, on the frontpage the individual benchmark names could be
links to the benchmark page, like in the grid view.

The specific benchmark has no description:
http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200

Following the spitfire_cstringio template, I propose the following
rewording of the below answer (I'm not entirely happy but I guess
Maciej can fix it easily if it's too bad):
slowspitfire: Uses the Spitfire template system to build a
1000x1000-cell HTML table; it differs from spitfire which is slower on
PyPy: it uses .join(list) instead of cStringIO module, has very long
lists with GC objects in it, and some other smaller problems.

> https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py

> It's creating a very large template table (1000x1000 elements I think)
>
> The explanation "why it's slow" is a bit longish. It's a combination
> of factors, including very long lists with GC objects in it, using
> ''.join(list) instead of cStringIO (the latter is faster and yes, it
> is a bug) and a bit of other factors.

Another small problem I had with zooming (which is really cool, BTW):

>There is an easy solution for that, at least for the moment: enabling
>zooming. I just did that, and you can now use zooming in a timeline
>plot to select a narrower yaxis range or just view a particular area
>in detail. A single click resets the zoom level.

While trying this I clicked on a revision; I immediately clicked on
"back", but I was brought too much backwards, to the grid of all
benchmarks, which loads slow enough for one to notice. If you instead
click "Back" from a specific benchmark page, you are brought back to
the home.
Fixing this without loading a separate page for each plot seems hard;
however, it seems that e.g. Facebook handles this by modifying the URL
part after #, so that the page is not reloaded from scratch, but I'm
no web developer, so you probably know better than me.

Cheers,
--
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From lac at openend.se  Wed Mar  9 15:56:26 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 09 Mar 2011 15:56:26 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Wed,
	09 Mar 2011 09:46:14 +0100."
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com> 
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com> 
Message-ID: <201103091456.p29EuQqA006242@theraft.openend.se>

Thank you very, very much.  I am sorry if I was short with you last
night, I was very tired.  This is great.  I am fine with
saying that the geometric mean is X times faster.  I'd  like to come up
with a good way to say that we are faster on real programs, not just
contrived examples, but this will take work and thought.  Maybe some
of the people who are on this list but not at pycon can come up with
something.

Thank you very much.  I'm very happy now.

Laura


From tobami at googlemail.com  Wed Mar  9 21:00:33 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 21:00:33 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
	<AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
Message-ID: <AANLkTimP6dUvc_kMHUoAiL2w3Nr1yZfORo5jotAV3ph+@mail.gmail.com>

> But I was unable to find out what slowspitfire is doing ... I found
> spitfire, which does some HTML templating stuff, and deducted, that
> slowspitfire will do some slow HTML templating stuff. Where did I
> click wrong? Is there a path down to the slowspitfire.py file or an
> explanation what slowspitfire is doing?

You definitely have a point, and there are features planned that
should make benchmark-discovery more user-friendly.



2011/3/9 Massa, Harald Armin <chef at ghum.de>:
> I really, really like the new display!
>
> And it motivated me to dig into the data ... which is a great result on its own.
>
> The first question for myself was "hey, why is it slow on
> slowspitfire, and, btw, what is slowspitfire? Could that be something
> that my application does, too?"
>
> But I was unable to find out what slowspitfire is doing ... I found
> spitfire, which does some HTML templating stuff, and deducted, that
> slowspitfire will do some slow HTML templating stuff. Where did I
> click wrong? Is there a path down to the slowspitfire.py file or an
> explanation what slowspitfire is doing?
>
> Harald
>
>
> --
> GHUM GmbH
> Harald Armin Massa
> Spielberger Stra?e 49
> 70435 Stuttgart
> 0173/9409607
>
> Amtsgericht Stuttgart, HRB 734971
> -
> persuadere.
> et programmare
>


From tobami at googlemail.com  Wed Mar  9 21:11:50 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 21:11:50 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <AANLkTiky6d=SA1NwRoDLXpcqe+19sisXwK9ZTyG65qV_@mail.gmail.com>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<tobami@googlemail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<AANLkTi=YFDg0OuA_yZpDGX4iBxvawhKc_K=pvzhz6PcG@mail.gmail.com>
	<AANLkTimXj=WxMFean6QtqzMXdTruYc-k+siK35VsAMUk@mail.gmail.com>
	<AANLkTimu9eNKB7zdTvoqMUp9Mq5kU9NWE3hqXxrPeoEw@mail.gmail.com>
	<AANLkTinoiL-x1ujfYgdR=Gx53-xGbLpG=bFDYvK2dOmc@mail.gmail.com>
	<AANLkTiky6d=SA1NwRoDLXpcqe+19sisXwK9ZTyG65qV_@mail.gmail.com>
Message-ID: <AANLkTineFRiNfN9g2giJ4rovwbb9jK-U+dDuWCKfdtUf@mail.gmail.com>

> The specific benchmark has no description:
> http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200

Thanks, I have added your description.

> Is there a place the answer this information to the website? I propose
> a link to the source in each benchmark page.
Yes, I will try to find a way to put links to the code in each benchmark page.

> Additionally, on the frontpage the individual benchmark names could be
> links to the benchmark page, like in the grid view.
I first thought that it would be difficult because the benchmark names
themselves are tick labels for the axes, but I think I can actually
make the bars clickable. I will see.

> While trying this I clicked on a revision; I immediately clicked on
> "back", but I was brought too much backwards, to the grid of all
> benchmarks, which loads slow enough for one to notice. If you instead
> click "Back" from a specific benchmark page, you are brought back to
> the home.
> Fixing this without loading a separate page for each plot seems hard;
> however, it seems that e.g. Facebook handles this by modifying the URL
> part after #, so that the page is not reloaded from scratch, but I'm
> no web developer, so you probably know better than me.

Yes, we are looking at a solution there. See issue #17 :
https://github.com/tobami/codespeed/issues#issue/17
Stefan Marr has already implemented it. There were some issues at
first, and he uses a development un-minified version of a plugin, but
it works. So we should be able to integrate the feature soon.

Miquel


2011/3/9 Paolo Giarrusso <pgiarrusso at mathematik.uni-marburg.de>:
> Hi all,
>
> On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef at ghum.de> wrote:
>> > I really, really like the new display!
> Congratulations to Miquel for the great work!
>
> A minor comment about the homepage: the answer to "How fast is PyPy?"
> would better stay close to the question, i.e. above Plot 1 (at least
> with the current wording).
>
>> > But I was unable to find out what slowspitfire is doing ... I found
>> > spitfire, which does some HTML templating stuff, and deducted, that
>> > slowspitfire will do some slow HTML templating stuff. Where did I
>> > click wrong?
>
>> > Is there a path down to the slowspitfire.py file or an
>> > explanation what slowspitfire is doing?
>
> Is there a place the answer this information to the website? I propose
> a link to the source in each benchmark page.
> Additionally, on the frontpage the individual benchmark names could be
> links to the benchmark page, like in the grid view.
>
> The specific benchmark has no description:
> http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200
>
> Following the spitfire_cstringio template, I propose the following
> rewording of the below answer (I'm not entirely happy but I guess
> Maciej can fix it easily if it's too bad):
> slowspitfire: Uses the Spitfire template system to build a
> 1000x1000-cell HTML table; it differs from spitfire which is slower on
> PyPy: it uses .join(list) instead of cStringIO module, has very long
> lists with GC objects in it, and some other smaller problems.
>
>> https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py
>
>> It's creating a very large template table (1000x1000 elements I think)
>>
>> The explanation "why it's slow" is a bit longish. It's a combination
>> of factors, including very long lists with GC objects in it, using
>> ''.join(list) instead of cStringIO (the latter is faster and yes, it
>> is a bug) and a bit of other factors.
>
> Another small problem I had with zooming (which is really cool, BTW):
>
>>There is an easy solution for that, at least for the moment: enabling
>>zooming. I just did that, and you can now use zooming in a timeline
>>plot to select a narrower yaxis range or just view a particular area
>>in detail. A single click resets the zoom level.
>
> While trying this I clicked on a revision; I immediately clicked on
> "back", but I was brought too much backwards, to the grid of all
> benchmarks, which loads slow enough for one to notice. If you instead
> click "Back" from a specific benchmark page, you are brought back to
> the home.
> Fixing this without loading a separate page for each plot seems hard;
> however, it seems that e.g. Facebook handles this by modifying the URL
> part after #, so that the page is not reloaded from scratch, but I'm
> no web developer, so you probably know better than me.
>
> Cheers,
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From tobami at googlemail.com  Wed Mar  9 21:21:13 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 9 Mar 2011 21:21:13 +0100
Subject: [pypy-dev] Change to the frontpage of speed.pypy.org
In-Reply-To: <201103091456.p29EuQqA006242@theraft.openend.se>
References: <lac@openend.se> <201102271359.p1RDxkpX012065@theraft.openend.se>
	<201102271538.p1RFc3rG020721@theraft.openend.se>
	<AANLkTi=SG14co6Ny7+gajOHwPrieXtw6V25GRh4fT8SE@mail.gmail.com>
	<201102280744.p1S7iFh7009804@theraft.openend.se>
	<AANLkTimo0RN+MpmUqRkRPYgjPJzuEJ=3+p0Z2DKvC-K9@mail.gmail.com>
	<201103041845.p24IjUDR015867@theraft.openend.se>
	<AANLkTinUX_cfxFRkHyNrD+ayG_k0a=tOE5LMw_LhKP+3@mail.gmail.com>
	<201103081614.p28GEqKT013585@theraft.openend.se>
	<AANLkTin-ne9558LD62CpSR993g=6mcV=3Qofdq4U7-m6@mail.gmail.com>
	<AANLkTinJ1YPmXXB5EJZ46+LxH2+ZAfGwg0iBNeDt0Zi1@mail.gmail.com>
	<AANLkTikHST-xBekwk-=JT9c1W_OCDyFKeA-b7nrvtQe3@mail.gmail.com>
	<201103090517.p295HfDa017396@theraft.openend.se>
	<tobami@googlemail.com>
	<AANLkTikOJyP39HhKJhx60N20VC8C_Q8j54ke+f-5wZia@mail.gmail.com>
	<201103091456.p29EuQqA006242@theraft.openend.se>
Message-ID: <AANLkTimh49Wr+vQw+VEmMPE8+3ewarpbnEwy4TbrVSJP@mail.gmail.com>

> I am sorry if I was short with you last night, I was very tired.

No problem Laura, that only shows that you care ;-)
It would have also been much easier to discuss things if I had
finished the changes a couple of days earlier instead of in the middle
of your arrival in PyCon. But I didn't have the time. Life's like
that.

Criticism is sometimes hard to swallow, but very necessary to improve,
as a person and as a project.
As long as people use reasoned arguments and are respectful, it is not
only OK to do so: It is your duty! ;-)

So thanks to you too.

Cheers,
Miquel


2011/3/9 Laura Creighton <lac at openend.se>:
> Thank you very, very much. ?I am sorry if I was short with you last
> night, I was very tired. ?This is great. ?I am fine with
> saying that the geometric mean is X times faster. ?I'd ?like to come up
> with a good way to say that we are faster on real programs, not just
> contrived examples, but this will take work and thought. ?Maybe some
> of the people who are on this list but not at pycon can come up with
> something.
>
> Thank you very much. ?I'm very happy now.
>
> Laura
>


From tbaldridge at gmail.com  Wed Mar  9 23:30:16 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Wed, 9 Mar 2011 16:30:16 -0600
Subject: [pypy-dev] Regex for RPython
Message-ID: <AANLkTikcCsW+WzyQE+iiNi-kttTo-2jF69j5pHCAOziq@mail.gmail.com>

I know rlib has a regex module, but it seems to be limited to only
recognizing characters and not returning groups. Is there a full
featured regex parser somewhere that is supported by rpython?

Thanks,

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From greg at quora.com  Thu Mar 10 01:13:46 2011
From: greg at quora.com (Greg Price)
Date: Wed, 9 Mar 2011 16:13:46 -0800
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
Message-ID: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>

The following program works in CPython, but fails in PyPy:

  class C(object):
    def __radd__(self, other):
      other.append(1)
      return other

  z = []
  z += C()
  print z  # should be [1]

In PyPy, this fails with "TypeError: 'C' object is not iterable".

The issue is that PyPy is allowing the list's inplace_add behavior to
take precedence over the C object's __radd__ method, while CPython
does the reverse.

A similar issue occurs in the following program:

  class C(object):
      def __rmul__(self, other):
          other *= 2
          return other
      def __index__(self):
          return 3

  print [1] * C() # should be [1,1]

where PyPy instead prints [1,1,1].


In Python, if the LHS of a foo-augmented assignment has an in-place
foo method defined, then that method is given the first opportunity to
perform the operation, followed by the foo method and the RHS's
reverse-foo methods.
http://docs.python.org/reference/datamodel.html#object.__iadd__

But: CPython's 'list' type does not have any numeric methods defined
at the C level, in-place or otherwise. See PyList_Type in
listobject.c, where tp_as_number is null. So an in-place addition
falls through to the RHS's nb_add, if present, and for a class with
metaclass 'type' and an __radd__() method this is slot_nb_add() from
typeobject.c, which calls __radd__().

So when "z += [1,2]" runs in CPython, it works via a further wrinkle.
The meat of the implementation of INPLACE_ADD is PyNumber_InPlaceAdd()
in abstract.c. When all numeric methods fail to handle the addition,
this function falls back to sequence methods, looking for
sq_inplace_concat or sq_concat methods on the LHS. 'list' has these
methods, so its sq_inplace_concat method handles the operation.

Similarly, PyNumber_InPlaceMultiply() tries all numeric methods first,
before falling back to sq_inplace_repeat or sq_repeat.

In PyPy, by contrast, it doesn't look like there's any logic for
falling back to a concatenate method if numeric methods fail. Instead,
if I'm reading correctly, the inplace_add__List_ANY() method in
pypy.objspace.std.listobject runs *before* any numeric methods on the
RHS are tried.


For the narrow case at hand, a sufficient hack for e.g. addition would
be to teach inplace_add__List_ANY() to look for an __radd__() first.
At a quick grep through CPython, full compatibility by that approach
would require similar hacks for bytes, array.array, collections.deque,
str, unicode, buffer, and bytearray; plus the users of structseq.c,
including type(sys.float_info), pwd.struct_passwd, grp.struct_group,
posix.stat_result, time.struct_time, and several others. Those types
in CPython all have sq_concat and not nb_add or nb_inplace_add, so
they will all permit an RHS's __radd__ to take precedence, but then
fall back on concatenation if no such method exists.

A more comprehensive approach would teach the generic dispatch code
about sequence methods falling after numeric methods in the dispatch
sequence. Then each sequence type would need to identify its sequence
methods for that code.

Perhaps a hybrid approach is best. Assume that no type with a
sq_concat also has a nb_add, and no type with a sq_repeat also has a
nb_multiply. (I believe this is true for all built-in, stdlib, and
pure-Python types.) Then whenever we define a method
{inplace_,}{add,mul}__Foo_ANY, for a sequence type Foo, it's enough
for that method to check for __r{add,mul}__ on the RHS. So we can
write a generic helper function and use that in each such method.

Does that last approach sound reasonable? I'm happy to go and
implement it, but I'm open to other suggestions.


I've posted tests (which fail) at
  https://bitbucket.org/price/pypy-queue/changeset/9dd9c2a5116a

Greg


From alex.gaynor at gmail.com  Thu Mar 10 01:32:26 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 9 Mar 2011 19:32:26 -0500
Subject: [pypy-dev] Regex for RPython
In-Reply-To: <AANLkTikcCsW+WzyQE+iiNi-kttTo-2jF69j5pHCAOziq@mail.gmail.com>
References: <AANLkTikcCsW+WzyQE+iiNi-kttTo-2jF69j5pHCAOziq@mail.gmail.com>
Message-ID: <AANLkTinRd0eeroRLEKo6K+3PTrOPwVz_XR+Kuy_o8FHq@mail.gmail.com>

On Wed, Mar 9, 2011 at 5:30 PM, Timothy Baldridge <tbaldridge at gmail.com>wrote:

> I know rlib has a regex module, but it seems to be limited to only
> recognizing characters and not returning groups. Is there a full
> featured regex parser somewhere that is supported by rpython?
>
> Thanks,
>
> Timothy
>
> --
> ?One of the main causes of the fall of the Roman Empire was
> that?lacking zero?they had no way to indicate successful termination
> of their C programs.?
> (Robert Firth)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

The PyPy python interpret just uses the stdlib regex parser, so no I don't
think there is a regex parser.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110309/57c55060/attachment.html>

From cfbolz at gmx.de  Thu Mar 10 10:01:24 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 10 Mar 2011 10:01:24 +0100
Subject: [pypy-dev] Regex for RPython
In-Reply-To: <AANLkTikcCsW+WzyQE+iiNi-kttTo-2jF69j5pHCAOziq@mail.gmail.com>
References: <AANLkTikcCsW+WzyQE+iiNi-kttTo-2jF69j5pHCAOziq@mail.gmail.com>
Message-ID: <4D789364.5030206@gmx.de>

On 03/09/2011 11:30 PM, Timothy Baldridge wrote:
> I know rlib has a regex module, but it seems to be limited to only
> recognizing characters and not returning groups. Is there a full
> featured regex parser somewhere that is supported by rpython?

Are you just interested in the parser or also in a way to do regex 
matching? For the parser we indeed use the stdlib version, for the 
matching there is an RPython implementation taking as input the 
bytecodes that the stdlib regex parser produces.

Cheers,

Carl Friedrich


From alex.gaynor at gmail.com  Thu Mar 10 14:42:05 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 10 Mar 2011 08:42:05 -0500
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
Message-ID: <AANLkTinhi3_Co9BypUbZEN6ZkMxsv+=ypudVLZJKAivX@mail.gmail.com>

On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg at quora.com> wrote:

> The following program works in CPython, but fails in PyPy:
>
>  class C(object):
>    def __radd__(self, other):
>      other.append(1)
>      return other
>
>  z = []
>  z += C()
>  print z  # should be [1]
>
> In PyPy, this fails with "TypeError: 'C' object is not iterable".
>
> The issue is that PyPy is allowing the list's inplace_add behavior to
> take precedence over the C object's __radd__ method, while CPython
> does the reverse.
>
> A similar issue occurs in the following program:
>
>  class C(object):
>      def __rmul__(self, other):
>          other *= 2
>          return other
>      def __index__(self):
>          return 3
>
>  print [1] * C() # should be [1,1]
>
> where PyPy instead prints [1,1,1].
>
>
> In Python, if the LHS of a foo-augmented assignment has an in-place
> foo method defined, then that method is given the first opportunity to
> perform the operation, followed by the foo method and the RHS's
> reverse-foo methods.
> http://docs.python.org/reference/datamodel.html#object.__iadd__
>
> But: CPython's 'list' type does not have any numeric methods defined
> at the C level, in-place or otherwise. See PyList_Type in
> listobject.c, where tp_as_number is null. So an in-place addition
> falls through to the RHS's nb_add, if present, and for a class with
> metaclass 'type' and an __radd__() method this is slot_nb_add() from
> typeobject.c, which calls __radd__().
>
> So when "z += [1,2]" runs in CPython, it works via a further wrinkle.
> The meat of the implementation of INPLACE_ADD is PyNumber_InPlaceAdd()
> in abstract.c. When all numeric methods fail to handle the addition,
> this function falls back to sequence methods, looking for
> sq_inplace_concat or sq_concat methods on the LHS. 'list' has these
> methods, so its sq_inplace_concat method handles the operation.
>
> Similarly, PyNumber_InPlaceMultiply() tries all numeric methods first,
> before falling back to sq_inplace_repeat or sq_repeat.
>
> In PyPy, by contrast, it doesn't look like there's any logic for
> falling back to a concatenate method if numeric methods fail. Instead,
> if I'm reading correctly, the inplace_add__List_ANY() method in
> pypy.objspace.std.listobject runs *before* any numeric methods on the
> RHS are tried.
>
>
> For the narrow case at hand, a sufficient hack for e.g. addition would
> be to teach inplace_add__List_ANY() to look for an __radd__() first.
> At a quick grep through CPython, full compatibility by that approach
> would require similar hacks for bytes, array.array, collections.deque,
> str, unicode, buffer, and bytearray; plus the users of structseq.c,
> including type(sys.float_info), pwd.struct_passwd, grp.struct_group,
> posix.stat_result, time.struct_time, and several others. Those types
> in CPython all have sq_concat and not nb_add or nb_inplace_add, so
> they will all permit an RHS's __radd__ to take precedence, but then
> fall back on concatenation if no such method exists.
>
> A more comprehensive approach would teach the generic dispatch code
> about sequence methods falling after numeric methods in the dispatch
> sequence. Then each sequence type would need to identify its sequence
> methods for that code.
>
> Perhaps a hybrid approach is best. Assume that no type with a
> sq_concat also has a nb_add, and no type with a sq_repeat also has a
> nb_multiply. (I believe this is true for all built-in, stdlib, and
> pure-Python types.) Then whenever we define a method
> {inplace_,}{add,mul}__Foo_ANY, for a sequence type Foo, it's enough
> for that method to check for __r{add,mul}__ on the RHS. So we can
> write a generic helper function and use that in each such method.
>
> Does that last approach sound reasonable? I'm happy to go and
> implement it, but I'm open to other suggestions.
>
>
> I've posted tests (which fail) at
>  https://bitbucket.org/price/pypy-queue/changeset/9dd9c2a5116a
>
> Greg
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Greg,

Is this affecting some real world code?  I ask because right now the
semantics we implement basically treat everything with the same semantics as
a pure Python class on CPython would, and I think we'd like to keep it that
way, as it makes the source a good bit simpler.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110310/80430efd/attachment.html>

From pgiarrusso at mathematik.uni-marburg.de  Thu Mar 10 15:08:07 2011
From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso)
Date: Thu, 10 Mar 2011 15:08:07 +0100
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTinhi3_Co9BypUbZEN6ZkMxsv+=ypudVLZJKAivX@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTinhi3_Co9BypUbZEN6ZkMxsv+=ypudVLZJKAivX@mail.gmail.com>
Message-ID: <AANLkTikbBkf9Xcug7bO3aeXgGqd5FhXB_RCOm61HHHg6@mail.gmail.com>

On Thu, Mar 10, 2011 at 14:42, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Greg,
>
> Is this affecting some real world code?? I ask because right now the
> semantics we implement basically treat everything with the same semantics as
> a pure Python class on CPython would, and I think we'd like to keep it that
> way, as it makes the source a good bit simpler.
Hi,
wasn't very good compatibility with CPython one of PyPy's points,
without restrictions to real-world code?
For instance Armin (IIRC) made significant effort to support Python's
semantics for destructors (in particular, to call destructors in the
"right" order), unlike done in Jython and IronPython.
Cheers,
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From anto.cuni at gmail.com  Thu Mar 10 16:21:06 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 10 Mar 2011 16:21:06 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
Message-ID: <4D78EC62.8030101@gmail.com>

Hi Hakan, hi all,

as you might have noticed, there is test_pypy_c_new.test_f1 which is currently
failing:

http://buildbot.pypy.org/summary/longrepr?testname=TestPyPyCNew.%28%29.test_f1&builder=pypy-c-jit-linux-x86-32&build=699&mod=pypy.module.pypyjit.test_pypy_c.test_pypy_c_new

buildbot did not show it earlier because before yesterday it did not run the
test at all. As you can see from the traceback, the problem is that running f1
makes three loops instead of one as expected.

I did a bit of manual binary search and discovered that the test started
failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the
commit, it seems that the only one relevant to the problem is
r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state.

Do you have any idea why jit-virtual_state causes this problem? (if it's a
problem at all, but I think so because the three loops contain the very same
operations, so I don't see why we need to compile the more than once).

ciao,
Anto


From anto.cuni at gmail.com  Thu Mar 10 18:22:37 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 10 Mar 2011 18:22:37 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <4D78EC62.8030101@gmail.com>
References: <4D78EC62.8030101@gmail.com>
Message-ID: <4D7908DD.40503@gmail.com>

On 10/03/11 16:21, Antonio Cuni wrote:

> I did a bit of manual binary search and discovered that the test started
> failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the
> commit, it seems that the only one relevant to the problem is
> r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state.

I confirm that the problem is the merging of jit-virtual_state: I just tried
the revision just before the merge, and the test passes.

ciao,
Anto


From arigo at tunes.org  Thu Mar 10 21:53:49 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 10 Mar 2011 15:53:49 -0500
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
Message-ID: <AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>

Hi Greg,

On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg at quora.com> wrote:
> The following program works in CPython, but fails in PyPy:

This is (if we are positive) an internal implementation detail and (if
we are negative) a bug in CPython.  There is no way to define in pure
Python a list-like type that would have the exact same behavior.  We
already have one such special case for __add__/__radd__ on subclasses
of strings and unicodes, which appears to be the use case that people
rely on most often, but I don't see the point in duplicating this
strange behavior for lists, tuples, and so on, unless there are really
several programs out there that rely on it.


A bient?t,

Armin.


From greg at quora.com  Thu Mar 10 23:38:35 2011
From: greg at quora.com (Greg Price)
Date: Thu, 10 Mar 2011 14:38:35 -0800
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
Message-ID: <AANLkTim4u1r7rXU9Ump20Cb7vuzr2iJFZyRfacrb=F1T@mail.gmail.com>

Hi Armin and Alex,

> This is (if we are positive) an internal implementation detail and (if
> we are negative) a bug in CPython. ?There is no way to define in pure
> Python a list-like type that would have the exact same behavior.

I don't disagree with this -- indeed, I think a strict reading of the
language reference would clearly identify this as wrong behavior by
CPython.

Nevertheless, real-world programs do exist that rely on this behavior.
I don't have a broad survey of real programs (I don't think anyone
does), but this issue causes Quora's codebase to fail a ton of unit
tests and basically not work at all under PyPy without modification.
The reason is that we have this idiom in thousands of places:

def f():
  z = []
  z += Foo()
  z += Bar()
  for x in something():
    z += Baz(x)
  return z

When the classes Foo, Bar, Baz, etc were implemented, the author gave
them __radd__ methods to implement the above behavior -- after all,
addition is precisely what we're doing here, right? So they look like

class Foo(object):
  def __radd__(self, other):
    other.append("something")
    return other

as in my original mail.

Now, knowing how this all works, it's not too hard to fix my company's
code so that it works here -- I just add an __iter__ method like so:

class Foo(object):
  [...]
  def __iter__(self):
    yield "something"

And if the original author had been developing under PyPy, no doubt he
would have seen the TypeError, accepted the behavior as is, and
quickly figured out this approach. Programmers adapt to all kinds of
details in their development environments, whether those details are
justified well or poorly.

But for someone considering PyPy and bringing over an existing
codebase, it's awfully disconcerting to see a bunch of tests fail and
have to go debugging. Especially so when it's pure Python and doesn't
muck around with GC, or in sys, or use anything that a non-expert
would recognize as an implementation detail. I know I'm an early,
early adopter and so I don't mind, but for the broader class of users
that I hope PyPy acquires in the next few years, an issue like this
would be a pretty negative signal.


As I outlined at the end of my first email, I don't think this would
require a great deal of code. I appreciate that there's a cost to
making things more complex, but I think it's worth it to make the way
smooth for people moving from CPython.

Let me know what you think. I'm happy to write up a patch.

Greg


On Thu, Mar 10, 2011 at 12:53 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Greg,
>
> On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg at quora.com> wrote:
>> The following program works in CPython, but fails in PyPy:
>
> This is (if we are positive) an internal implementation detail and (if
> we are negative) a bug in CPython. ?There is no way to define in pure
> Python a list-like type that would have the exact same behavior. ?We
> already have one such special case for __add__/__radd__ on subclasses
> of strings and unicodes, which appears to be the use case that people
> rely on most often, but I don't see the point in duplicating this
> strange behavior for lists, tuples, and so on, unless there are really
> several programs out there that rely on it.
>
>
> A bient?t,
>
> Armin.
>


From holger at merlinux.eu  Fri Mar 11 19:46:54 2011
From: holger at merlinux.eu (holger krekel)
Date: Fri, 11 Mar 2011 18:46:54 +0000
Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan)
Message-ID: <20110311184654.GR16231@merlinux.eu>

Hi all, especially devs at Pycon,

FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai )
who works with the Google App Engine time, oldest to newest:

    At the Python VM panel. I want to believe alternate VMs are about
    more than just benchmarks. Question do not agree with me #pycon

    For instance, I want to see the tooling that can be built around
    alternative VMs. Profilers, heap analyzers, better debuggers #pycon
    
    Python VM implementors think sandboxing is not something that should
    be implemented at VM layer. Oh, well. #pycon

To me, this does not seem to be fitting for PyPy, does it?
Maybe somebody can discuss a bit with him.  I still believe that Google 
and GAE are a potentially very good application of PyPy tech.

best,
holger


From alex.gaynor at gmail.com  Fri Mar 11 19:48:48 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Fri, 11 Mar 2011 13:48:48 -0500
Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan)
In-Reply-To: <20110311184654.GR16231@merlinux.eu>
References: <20110311184654.GR16231@merlinux.eu>
Message-ID: <AANLkTi=mXFRN2d2jbVDSJwSNPQ3JaAUR3Pyqmiri-r0b@mail.gmail.com>

On Fri, Mar 11, 2011 at 1:46 PM, holger krekel <holger at merlinux.eu> wrote:

> Hi all, especially devs at Pycon,
>
> FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai )
> who works with the Google App Engine time, oldest to newest:
>
>    At the Python VM panel. I want to believe alternate VMs are about
>    more than just benchmarks. Question do not agree with me #pycon
>
>    For instance, I want to see the tooling that can be built around
>    alternative VMs. Profilers, heap analyzers, better debuggers #pycon
>
>    Python VM implementors think sandboxing is not something that should
>    be implemented at VM layer. Oh, well. #pycon
>
> To me, this does not seem to be fitting for PyPy, does it?
> Maybe somebody can discuss a bit with him.  I still believe that Google
> and GAE are a potentially very good application of PyPy tech.
>
> best,
> holger
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

That's a bit confusing IMO.  Maciej, Frank, and Dino all said that their VM
supports sandboxing, but it shouldn't be implemente din a python specific
way because Python is such a big language.  Not sure why someone would want
Python specific sandboxing.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110311/dc16d957/attachment.html>

From holger at merlinux.eu  Fri Mar 11 20:07:43 2011
From: holger at merlinux.eu (holger krekel)
Date: Fri, 11 Mar 2011 19:07:43 +0000
Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan)
In-Reply-To: <AANLkTi=mXFRN2d2jbVDSJwSNPQ3JaAUR3Pyqmiri-r0b@mail.gmail.com>
References: <20110311184654.GR16231@merlinux.eu>
	<AANLkTi=mXFRN2d2jbVDSJwSNPQ3JaAUR3Pyqmiri-r0b@mail.gmail.com>
Message-ID: <20110311190743.GS16231@merlinux.eu>

On Fri, Mar 11, 2011 at 13:48 -0500, Alex Gaynor wrote:
> On Fri, Mar 11, 2011 at 1:46 PM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi all, especially devs at Pycon,
> >
> > FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai )
> > who works with the Google App Engine time, oldest to newest:
> >
> >    At the Python VM panel. I want to believe alternate VMs are about
> >    more than just benchmarks. Question do not agree with me #pycon
> >
> >    For instance, I want to see the tooling that can be built around
> >    alternative VMs. Profilers, heap analyzers, better debuggers #pycon
> >
> >    Python VM implementors think sandboxing is not something that should
> >    be implemented at VM layer. Oh, well. #pycon
> >
> > To me, this does not seem to be fitting for PyPy, does it?
> > Maybe somebody can discuss a bit with him.  I still believe that Google
> > and GAE are a potentially very good application of PyPy tech.
> >
> 
> That's a bit confusing IMO.  Maciej, Frank, and Dino all said that their VM
> supports sandboxing, but it shouldn't be implemente din a python specific
> way because Python is such a big language.  Not sure why someone would want
> Python specific sandboxing.

Ah, indeed confusing.  I would describe PyPy's sandobxing to be at VM level
as opposed to approaches that rather try to limit access to attributes
or recursively control by proxies on top of a VM.

best,
holger



From arigo at tunes.org  Fri Mar 11 20:16:45 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 11 Mar 2011 14:16:45 -0500
Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan)
In-Reply-To: <20110311190743.GS16231@merlinux.eu>
References: <20110311184654.GR16231@merlinux.eu>
	<AANLkTi=mXFRN2d2jbVDSJwSNPQ3JaAUR3Pyqmiri-r0b@mail.gmail.com>
	<20110311190743.GS16231@merlinux.eu>
Message-ID: <AANLkTi=MjgMnBtPn74Heo3=X8KBWAk-w9nF2wZocwkbQ@mail.gmail.com>

Hi Holger,

Yes, that's what all three representatives of alternate VMs said at
the panel: sandboxing at the VM level is cool and supported, but we
don't want sandboxing at the language level.  As far as I can tell,
Ikan Lan just misunderstood it and got the opposite message.


Armin


From greg at quora.com  Sat Mar 12 02:55:33 2011
From: greg at quora.com (Greg Price)
Date: Fri, 11 Mar 2011 17:55:33 -0800
Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping, to fix a crash
Message-ID: <AANLkTinYx=AtPki-DStUuehwY7L-0d7vjh_UYR+5Y09G@mail.gmail.com>

(I've sent this also as a Bitbucket pull request. Sending it here too
because I'm not sure whose attention that reaches, and also for the
sake of developing in the open.)

We were increfing an object on the first time we borrow a reference to it,
then decrefing it on the first time we destroy a borrowed-from container.
So if we borrow from two containers, then destroy one, we'd take the
refcount back where it started -- possibly zero -- even though the C code
still has a legitimate reference.  This is demonstrated by test_borrow_destroy
in test_borrow.py, added last night.

Change the logic to incref the object the first time we borrow it from
each container, and decref when any borrowed-from container is destroyed.

Also fix a NameError in some code that runs when DEBUG_REFCOUNT is set.

 pypy/module/cpyext/pyobject.py         |   32 +++++++++-----------------------
 pypy/module/cpyext/test/test_borrow.py |   17 ++++++++++++++++-
 2 files changed, 25 insertions(+), 24 deletions(-)

diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py
index 99aabfb..9e42153 100644
--- a/pypy/module/cpyext/pyobject.py
+++ b/pypy/module/cpyext/pyobject.py
@@ -144,14 +144,11 @@ class RefcountState:
         # { w_container -> { w_containee -> None } }
         # the None entry manages references borrowed during a call to
         # generic_cpy_call()
-        self.borrowed_objects = {}
-        # { addr of containee -> None }

         # For tests
         self.non_heaptypes_w = []

     def _freeze_(self):
-        assert not self.borrowed_objects
         assert self.borrow_mapping == {None: {}}
         self.py_objects_r2w.clear() # is not valid anymore after translation
         return False
@@ -187,22 +184,19 @@ class RefcountState:
         """
         ref = make_ref(self.space, w_borrowed)
         obj_ptr = rffi.cast(ADDR, ref)
-        if obj_ptr not in self.borrowed_objects:
-            # borrowed_objects owns the reference
-            self.borrowed_objects[obj_ptr] = None
-        else:
-            Py_DecRef(self.space, ref) # already in borrowed list

         borrowees = self.borrow_mapping.setdefault(w_container, {})
-        borrowees[w_borrowed] = None
+        if w_borrowed in borrowees:
+            Py_DecRef(self.space, w_borrowed) # cancel incref from make_ref()
+        else:
+            borrowees[w_borrowed] = None
+
         return ref

     def reset_borrowed_references(self):
         "Used in tests"
-        while self.borrowed_objects:
-            addr, _ = self.borrowed_objects.popitem()
-            w_obj = self.py_objects_r2w[addr]
-            Py_DecRef(self.space, w_obj)
+        for w_container, w_borrowed in self.borrow_mapping.items():
+            Py_DecRef(self.space, w_borrowed)
         self.borrow_mapping = {None: {}}

     def delete_borrower(self, w_obj):
@@ -232,17 +226,10 @@ class RefcountState:
         ref = self.py_objects_w2r.get(w_obj, lltype.nullptr(PyObject.TO))
         if not ref:
             if DEBUG_REFCOUNT:
-                print >>sys.stderr, "Borrowed object is already gone:", \
-                      hex(containee)
+                print >>sys.stderr, "Borrowed object is already gone!"
             return

-        containee_ptr = rffi.cast(ADDR, ref)
-        try:
-            del self.borrowed_objects[containee_ptr]
-        except KeyError:
-            pass
-        else:
-            Py_DecRef(self.space, ref)
+        Py_DecRef(self.space, ref)

 class InvalidPointerException(Exception):
     pass
@@ -290,7 +277,6 @@ def track_reference(space, py_obj, w_obj, replace=False):
         if not replace:
             assert w_obj not in state.py_objects_w2r
         assert ptr not in state.py_objects_r2w
-        assert ptr not in state.borrowed_objects
     state.py_objects_w2r[w_obj] = py_obj
     if ptr: # init_typeobject() bootstraps with NULL references
         state.py_objects_r2w[ptr] = w_obj
diff --git a/pypy/module/cpyext/test/test_borrow.py
b/pypy/module/cpyext/test/test_borrow.py
index 25f5101..c903271 100644
--- a/pypy/module/cpyext/test/test_borrow.py
+++ b/pypy/module/cpyext/test/test_borrow.py
@@ -39,7 +39,6 @@ class AppTestBorrow(AppTestCpythonExtensionBase):
         assert module.test_borrowing() # the test should not leak

     def test_borrow_destroy(self):
-        skip("FIXME")
         module = self.import_extension('foo', [
             ("test_borrow_destroy", "METH_NOARGS",
              """
@@ -59,3 +58,19 @@ class AppTestBorrow(AppTestCpythonExtensionBase):
              """),
             ])
         assert module.test_borrow_destroy() == 42
+
+    def test_double_borrow(self):
+        module = self.import_extension('foo', [
+                ("run", "METH_NOARGS",
+                 """
+                    PyObject *t = PyTuple_New(1);
+                    PyObject *i = PyInt_FromLong(42);
+                    PyTuple_SetItem(t, 0, i);
+                    i = PyTuple_GetItem(t, 0);
+                    PyTuple_GetItem(t, 0);
+                    Py_DECREF(t);
+                    return PyInt_FromLong(PyInt_AsLong(i));
+                 """),
+                ])
+        # i.e., there was an InvalidPointerException
+        assert module.run() == 0


From amauryfa at gmail.com  Sat Mar 12 15:40:55 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 12 Mar 2011 15:40:55 +0100
Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping,
	to fix a crash
In-Reply-To: <AANLkTinYx=AtPki-DStUuehwY7L-0d7vjh_UYR+5Y09G@mail.gmail.com>
References: <AANLkTinYx=AtPki-DStUuehwY7L-0d7vjh_UYR+5Y09G@mail.gmail.com>
Message-ID: <AANLkTi=sUkN_hFxD1sCX2Cz=3x8CwkmWBmrQqrAbXpJq@mail.gmail.com>

Hi,

2011/3/12 Greg Price <greg at quora.com>:
> + ? ? ? ? ? ? ? ? """
> + ? ? ? ? ? ? ? ? ? ?PyObject *t = PyTuple_New(1);
> + ? ? ? ? ? ? ? ? ? ?PyObject *i = PyInt_FromLong(42);
> + ? ? ? ? ? ? ? ? ? ?PyTuple_SetItem(t, 0, i);
> + ? ? ? ? ? ? ? ? ? ?i = PyTuple_GetItem(t, 0);
> + ? ? ? ? ? ? ? ? ? ?PyTuple_GetItem(t, 0);
> + ? ? ? ? ? ? ? ? ? ?Py_DECREF(t);
> + ? ? ? ? ? ? ? ? ? ?return PyInt_FromLong(PyInt_AsLong(i));
> + ? ? ? ? ? ? ? ? """),

This example is wrong: you don't own the reference to int(42);
after the tuple is released, PyInt_AsLong(i) is illegal.

-- 
Amaury Forgeot d'Arc


From hakan at debian.org  Sat Mar 12 19:25:53 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sat, 12 Mar 2011 19:25:53 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <4D7908DD.40503@gmail.com>
References: <4D78EC62.8030101@gmail.com>
	<4D7908DD.40503@gmail.com>
Message-ID: <AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>

Yes, this is probably the VirtualState checking. It will retrace a
loop whenever the VirtualState at the end of a bridge differs from the
VirtualState at the beginning of the compiled trace (any of the
compiled traces). This might indeed produce an identical trace if we
are unlucky, but the idea is that this should only happen rarely.

This is because the VirtualState  at the beginning of a trace is the
state of all the OptValue of the inputargs produced during the
optimization of the trace. This does not have to be the most general
state for which the trace is usable (which would be hard to calculate
I'm afraid).

A few cases that would (most likely) result in identical traces are
salvaged in NotVirtualInfo._generate_guards by producing some extra
gurads at the end of a bridge to make the VirtualState there match the
VirtualState of a compiled trace. This is however only done if the
guards would (most likely) not fail for the traced iteration.

I'll look into what's happening in this particular test...

On Thu, Mar 10, 2011 at 6:22 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 10/03/11 16:21, Antonio Cuni wrote:
>
>> I did a bit of manual binary search and discovered that the test started
>> failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the
>> commit, it seems that the only one relevant to the problem is
>> r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state.
>
> I confirm that the problem is the merging of jit-virtual_state: I just tried
> the revision just before the merge, and the test passes.
>
> ciao,
> Anto
>



-- 
H?kan Ard?


From anto.cuni at gmail.com  Sat Mar 12 20:34:11 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sat, 12 Mar 2011 20:34:11 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
References: <4D78EC62.8030101@gmail.com>	<4D7908DD.40503@gmail.com>
	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
Message-ID: <4D7BCAB3.7020906@gmail.com>

Hi Hakan,

On 12/03/11 19:25, Hakan Ardo wrote:
> Yes, this is probably the VirtualState checking. It will retrace a
> loop whenever the VirtualState at the end of a bridge differs from the
> VirtualState at the beginning of the compiled trace (any of the
> compiled traces). This might indeed produce an identical trace if we
> are unlucky, but the idea is that this should only happen rarely.

ok, that's clear. So, hopefully this particular example looks a bit bad, but
in general it should not be an issue. It'd be nice to have a way to check this
thesis, but I agree that it's a bit hard.

> This is because the VirtualState  at the beginning of a trace is the
> state of all the OptValue of the inputargs produced during the
> optimization of the trace. This does not have to be the most general
> state for which the trace is usable (which would be hard to calculate
> I'm afraid).

so, if I understand correctly, this is what happens:

1. we trace, optimize and compile loop A

2. after a while, we trace, optimize a compile a bridge B which then jumps
back to A; by chance, the bridge looks the same as the loop

Am I right?

> A few cases that would (most likely) result in identical traces are
> salvaged in NotVirtualInfo._generate_guards by producing some extra
> gurads at the end of a bridge to make the VirtualState there match the
> VirtualState of a compiled trace. This is however only done if the
> guards would (most likely) not fail for the traced iteration.
> 
> I'll look into what's happening in this particular test...

I just did a quick check because I'm in a hurry, but from what I see we get
three actual *loops*, not bridges.

ciao,
Anto


From hakan at debian.org  Sat Mar 12 22:59:11 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sat, 12 Mar 2011 22:59:11 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <4D7BCAB3.7020906@gmail.com>
References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com>
	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
	<4D7BCAB3.7020906@gmail.com>
Message-ID: <AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>

On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Hi Hakan,
>
> On 12/03/11 19:25, Hakan Ardo wrote:
>> Yes, this is probably the VirtualState checking. It will retrace a
>> loop whenever the VirtualState at the end of a bridge differs from the
>> VirtualState at the beginning of the compiled trace (any of the
>> compiled traces). This might indeed produce an identical trace if we
>> are unlucky, but the idea is that this should only happen rarely.
>
> ok, that's clear. So, hopefully this particular example looks a bit bad, but
> in general it should not be an issue. It'd be nice to have a way to check this
> thesis, but I agree that it's a bit hard.

We should probably log the VirtualState together with the produced
loops and bridges. That would allow us to see how they differ when a
new version of a loop is traced. There are __repr__ methods I've been
using for that while debugging. They might need some rpythonizing to
translate though---

>
>> This is because the VirtualState ?at the beginning of a trace is the
>> state of all the OptValue of the inputargs produced during the
>> optimization of the trace. This does not have to be the most general
>> state for which the trace is usable (which would be hard to calculate
>> I'm afraid).
>
> so, if I understand correctly, this is what happens:
>
> 1. we trace, optimize and compile loop A
>
> 2. after a while, we trace, optimize a compile a bridge B which then jumps
> back to A; by chance, the bridge looks the same as the loop
>
> Am I right?

Maybe, I've not had the chance to look into any details yet. I'll do
that tomorrow...

>
>> A few cases that would (most likely) result in identical traces are
>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>> gurads at the end of a bridge to make the VirtualState there match the
>> VirtualState of a compiled trace. This is however only done if the
>> guards would (most likely) not fail for the traced iteration.
>>
>> I'll look into what's happening in this particular test...
>
> I just did a quick check because I'm in a hurry, but from what I see we get
> three actual *loops*, not bridges.

So if it's the same loop traced several times they should all have the
same preamble, and the preamble would have two bridges leading to the
two second versions of the loop. The preamble and it's two bridges
should end with different VirtualState's. The loops should  be
specialized to the different VirtualState's, but if the VirtualState's
are similar enough (but not equal) they might consist of the exact
same operations.

If there are 3 preambles for the same loop, there is a bug somewhere...

-- 
H?kan Ard?


From pjenvey at underboss.org  Sat Mar 12 23:21:01 2011
From: pjenvey at underboss.org (Philip Jenvey)
Date: Sat, 12 Mar 2011 17:21:01 -0500
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
Message-ID: <D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>


On Mar 10, 2011, at 3:53 PM, Armin Rigo wrote:

> Hi Greg,
> 
> On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg at quora.com> wrote:
>> The following program works in CPython, but fails in PyPy:
> 
> This is (if we are positive) an internal implementation detail and (if
> we are negative) a bug in CPython. 

Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY be returning NotImplemented (or whatever the pypy equiv. would be, FailedToImplement?) instead of raising the exception? To allow the binop rules to continue.

We fail the 2nd example like PyPy does but that's a different problem.

--
Philip Jenvey



From greg at quora.com  Sat Mar 12 23:34:30 2011
From: greg at quora.com (Greg Price)
Date: Sat, 12 Mar 2011 14:34:30 -0800
Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping,
	to fix a crash
In-Reply-To: <AANLkTi=sUkN_hFxD1sCX2Cz=3x8CwkmWBmrQqrAbXpJq@mail.gmail.com>
References: <AANLkTinYx=AtPki-DStUuehwY7L-0d7vjh_UYR+5Y09G@mail.gmail.com>
	<AANLkTi=sUkN_hFxD1sCX2Cz=3x8CwkmWBmrQqrAbXpJq@mail.gmail.com>
Message-ID: <AANLkTikfce54fR9e2dk07bVJkciAtWVQ2N74yO5SQiW1@mail.gmail.com>

On Sat, Mar 12, 2011 at 6:40 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2011/3/12 Greg Price <greg at quora.com>:
>> + ? ? ? ? ? ? ? ? """
>> + ? ? ? ? ? ? ? ? ? ?PyObject *t = PyTuple_New(1);
>> + ? ? ? ? ? ? ? ? ? ?PyObject *i = PyInt_FromLong(42);
>> + ? ? ? ? ? ? ? ? ? ?PyTuple_SetItem(t, 0, i);
>> + ? ? ? ? ? ? ? ? ? ?i = PyTuple_GetItem(t, 0);
>> + ? ? ? ? ? ? ? ? ? ?PyTuple_GetItem(t, 0);
>> + ? ? ? ? ? ? ? ? ? ?Py_DECREF(t);
>> + ? ? ? ? ? ? ? ? ? ?return PyInt_FromLong(PyInt_AsLong(i));
>> + ? ? ? ? ? ? ? ? """),
>
> This example is wrong: you don't own the reference to int(42);
> after the tuple is released, PyInt_AsLong(i) is illegal.

Yes, that's the intent of the test; see the comment just below. An
earlier version of my patch would have left 'i' alive, so I wanted to
check that in a test. This C code returns 42 if the int stays alive, 0
if it is destroyed when it should be.

This way of testing is a little ugly, though, so if you can suggest a
better way to check for the InvalidPointerException I'd be glad to
hear it.

Greg


From herve.coatanhay at gmail.com  Sun Mar 13 03:11:40 2011
From: herve.coatanhay at gmail.com (=?utf-8?Q?Herv=C3=A9_Coatanhay?=)
Date: Sat, 12 Mar 2011 21:11:40 -0500
Subject: [pypy-dev] translation with --stackless option
Message-ID: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com>

After a clone of the repository, whereas i can do the following translation with just some warnings:
python translate.py --opt=jit targetpypystandalone.py

I got an exception with that one:
python translate.py --stackless targetpypystandalone.py

As the --stackless option is not detailed in documentation maybe I should had something along with it ?

I tried this on both Mac OS X and a CentOS Linux box, the attach file contains the traceback of the translation.

Any idea on what i am doing wrong ?
--
Herv? Coatanhay

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110312/893747be/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy_traceback.txt
Type: application/octet-stream
Size: 3544 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110312/893747be/attachment.obj>

From fijall at gmail.com  Sun Mar 13 03:26:21 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 12 Mar 2011 21:26:21 -0500
Subject: [pypy-dev] translation with --stackless option
In-Reply-To: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com>
References: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com>
Message-ID: <AANLkTimiDHT7Bo+qK3aXJZYcWGBviDnDL6TLkp78PYUf@mail.gmail.com>

On Sat, Mar 12, 2011 at 9:11 PM, Herv? Coatanhay
<herve.coatanhay at gmail.com> wrote:
> After a clone of the repository,?whereas i can do the following translation
> with just some warnings:
> python translate.py --opt=jit targetpypystandalone.py
> I got an exception with that one:
> python translate.py --stackless targetpypystandalone.py
> As the --stackless option is not detailed in documentation maybe I should
> had something along with it ?
> I tried this on both Mac OS X and a CentOS Linux box, the attach file
> contains the traceback of the translation.
> Any idea on what i am doing wrong ?

It's broken.

We should fix it some time, preferably soon. That said, there is a
feature missing which is stackless working with JIT. We're not
actively working on it but this is something that would be a useful
contribution to the project and not too much work. But yes, we should
fix the failure first,.

> --
> Herv? Coatanhay
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From herve.coatanhay at gmail.com  Sun Mar 13 04:30:34 2011
From: herve.coatanhay at gmail.com (=?UTF-8?Q?Herv=C3=A9_Coatanhay?=)
Date: Sat, 12 Mar 2011 22:30:34 -0500
Subject: [pypy-dev] translation with --stackless option
In-Reply-To: <AANLkTimiDHT7Bo+qK3aXJZYcWGBviDnDL6TLkp78PYUf@mail.gmail.com>
References: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com>
	<AANLkTimiDHT7Bo+qK3aXJZYcWGBviDnDL6TLkp78PYUf@mail.gmail.com>
Message-ID: <AANLkTim+B3MpmR4_D0RibKKqwf2MkQ8ew=LmHeQ9p6rc@mail.gmail.com>

On Sat, Mar 12, 2011 at 9:26 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Sat, Mar 12, 2011 at 9:11 PM, Herv? Coatanhay
> <herve.coatanhay at gmail.com> wrote:
>> After a clone of the repository,?whereas i can do the following translation
>> with just some warnings:
>> python translate.py --opt=jit targetpypystandalone.py
>> I got an exception with that one:
>> python translate.py --stackless targetpypystandalone.py
>> As the --stackless option is not detailed in documentation maybe I should
>> had something along with it ?
>> I tried this on both Mac OS X and a CentOS Linux box, the attach file
>> contains the traceback of the translation.
>> Any idea on what i am doing wrong ?
>
> It's broken.
>
> We should fix it some time, preferably soon. That said, there is a
> feature missing which is stackless working with JIT. We're not
> actively working on it but this is something that would be a useful
> contribution to the project and not too much work. But yes, we should
> fix the failure first,.
Thanks for the answer.
I guess I'll have to wait a bit before giving a try at pypy as a
stackless implementation.

Maybe while waiting, I should look at documentation about JIT anyway.
-- 
Herv? Coatanhay


From lac at openend.se  Sun Mar 13 05:08:00 2011
From: lac at openend.se (Laura Creighton)
Date: Sun, 13 Mar 2011 05:08:00 +0100
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: Message from Philip Jenvey <pjenvey@underboss.org> of "Sat,
	12 Mar 2011 17:21:01 EST."
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org> 
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org> 
Message-ID: <201103130408.p2D480Zd013368@theraft.openend.se>

In a message of Sat, 12 Mar 2011 17:21:01 EST, Philip Jenvey writes:
>
>On Mar 10, 2011, at 3:53 PM, Armin Rigo wrote:
>
>> Hi Greg,
>> 
>> On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg at quora.com> wrote:
>>> The following program works in CPython, but fails in PyPy:
>> 
>> This is (if we are positive) an internal implementation detail and (if
>> we are negative) a bug in CPython. 
>
>Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY b
>e returning NotImplemented (or whatever the pypy equiv. would be, FailedT
>oImplement?) instead of raising the exception? To allow the binop rules t
>o continue.
>
>We fail the 2nd example like PyPy does but that's a different problem.
>
>--
>Philip Jenvey

I posted a note about this to python-dev
http://mail.python.org/pipermail/python-dev/2011-March/109130.html
and the reaction on python-dev seems to be unanimous that Cpython is
broken, and that pypy is doing the correct thing.  Current discussion
is about whether to just fix it, as a bug in Cpython or whether to
deprecate it and then stop it, as a kindness to those who relied on
it, or to backport the fix to 2.7 as well.

Laura


From hakan at debian.org  Sun Mar 13 11:12:56 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sun, 13 Mar 2011 11:12:56 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>
References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com>
	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
	<4D7BCAB3.7020906@gmail.com>
	<AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>
Message-ID: <AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>

Hi,
this is what happens here:

1. The inner loop is traced and Loop0 is produced with preamble Loop1

2. A bridge from Guard3 (the test in the while) back to Loop0 is
traced (i.e the remaining parts of the outer loop)

3. At the end of this bridge the VirtualState does not match the
VirtualState of Loop0, so the loop is retraced

4. The VirtualState of the newly traced version of the loop does not
match the VirtualState at the end of the bridge so the bridge has to
jump to the preamble instead of jumping to the new specialized version
of the loop.

5. A bridge from Guard6 (signal/thread counter) is traced and the same
thing happens for this bridge.

This means that the additional two versions of the loop will never be
used and should hopefully be reomved by the gc...

So there are two issues:

A. The additional specialized versions created does not become usable.
This is the issue I'm working on in the jit-usable_retrace branch. The
idea there is to have the retrace inherit the OptValue's of the
jumpargs at the end of the bridge. This will become a fairly large
change functionality wise...

B. The VirtualStates' s differs in the first place forcing a retrace.
This is probably fixable by introducing some more cases in
NotVirtualInfo._generate_guards(). The jit-usable_retrace branch
contains more cases than trunk, don't know if those are enough for
this test though...

Note however that
jit/metainterp/test/test_nested_loops_discovered_by_bridge in
test_loop_unroll.py, which conatins the same loop for a simple
interpreter, does work nicely, wihtout the issues above.

On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo <hakan at debian.org> wrote:
> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> Hi Hakan,
>>
>> On 12/03/11 19:25, Hakan Ardo wrote:
>>> Yes, this is probably the VirtualState checking. It will retrace a
>>> loop whenever the VirtualState at the end of a bridge differs from the
>>> VirtualState at the beginning of the compiled trace (any of the
>>> compiled traces). This might indeed produce an identical trace if we
>>> are unlucky, but the idea is that this should only happen rarely.
>>
>> ok, that's clear. So, hopefully this particular example looks a bit bad, but
>> in general it should not be an issue. It'd be nice to have a way to check this
>> thesis, but I agree that it's a bit hard.
>
> We should probably log the VirtualState together with the produced
> loops and bridges. That would allow us to see how they differ when a
> new version of a loop is traced. There are __repr__ methods I've been
> using for that while debugging. They might need some rpythonizing to
> translate though---
>
>>
>>> This is because the VirtualState ?at the beginning of a trace is the
>>> state of all the OptValue of the inputargs produced during the
>>> optimization of the trace. This does not have to be the most general
>>> state for which the trace is usable (which would be hard to calculate
>>> I'm afraid).
>>
>> so, if I understand correctly, this is what happens:
>>
>> 1. we trace, optimize and compile loop A
>>
>> 2. after a while, we trace, optimize a compile a bridge B which then jumps
>> back to A; by chance, the bridge looks the same as the loop
>>
>> Am I right?
>
> Maybe, I've not had the chance to look into any details yet. I'll do
> that tomorrow...
>
>>
>>> A few cases that would (most likely) result in identical traces are
>>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>>> gurads at the end of a bridge to make the VirtualState there match the
>>> VirtualState of a compiled trace. This is however only done if the
>>> guards would (most likely) not fail for the traced iteration.
>>>
>>> I'll look into what's happening in this particular test...
>>
>> I just did a quick check because I'm in a hurry, but from what I see we get
>> three actual *loops*, not bridges.
>
> So if it's the same loop traced several times they should all have the
> same preamble, and the preamble would have two bridges leading to the
> two second versions of the loop. The preamble and it's two bridges
> should end with different VirtualState's. The loops should ?be
> specialized to the different VirtualState's, but if the VirtualState's
> are similar enough (but not equal) they might consist of the exact
> same operations.
>
> If there are 3 preambles for the same loop, there is a bug somewhere...
>
> --
> H?kan Ard?
>



-- 
H?kan Ard?


From arigo at tunes.org  Sun Mar 13 11:19:30 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 13 Mar 2011 06:19:30 -0400
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>
Message-ID: <AANLkTin1MnVNKefsEHu4O3bvdO+y8v2vaGMv5Yaq1O7G@mail.gmail.com>

Hi Philip,

On Sat, Mar 12, 2011 at 5:21 PM, Philip Jenvey <pjenvey at underboss.org> wrote:
> Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY be returning NotImplemented (or whatever the pypy equiv. would be, FailedToImplement?) instead of raising the exception? To allow the binop rules to continue.

Good idea.  Right now, both on CPython and on PyPy,  [].__iadd__(5)
raises TypeError (instead of returning NotImplemented), but
[].__add__(5)  already behaves differently: it raises TypeError on
CPython and returns NotImplemented on PyPy, with the result that
[]+Bar()  does call the __radd__() method on Bar().  So having the
same difference in the __iadd__() method looks like a useful
compromise.  This is particularly true given that it already works on
all sequence types different from lists, because these don't have an
__iadd__() method at all.


Armin


From arigo at tunes.org  Sun Mar 13 12:05:42 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 13 Mar 2011 07:05:42 -0400
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <AANLkTin1MnVNKefsEHu4O3bvdO+y8v2vaGMv5Yaq1O7G@mail.gmail.com>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>
	<AANLkTin1MnVNKefsEHu4O3bvdO+y8v2vaGMv5Yaq1O7G@mail.gmail.com>
Message-ID: <AANLkTintAcKLpHfBDu-cM3KX7s=chQ_vtuHfHxzMC8Ny@mail.gmail.com>

Hi,

Checked in 72ce40f4803c.

Greg, about the second example you reported:

class C(object):
     def __rmul__(self, other):
         other *= 2
         return other
     def __index__(self):
         return 3
print [1] * C()   # -> CPython: [1,1]  PyPy: [1,1,1]

This is obscure.  It works that way because in that particular case,
CPython first tries __mul__/__rmul__ and then tries the internal
sq_repeat slot of the list (which calls C.__index__()).  I think there
is just no way to reproduce this behavior while remaining sane :-/


Armin


From lac at openend.se  Sun Mar 13 13:24:27 2011
From: lac at openend.se (Laura Creighton)
Date: Sun, 13 Mar 2011 13:24:27 +0100
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: Message from Armin Rigo <arigo@tunes.org> of "Sun,
	13 Mar 2011 07:05:42 -0400."
	<AANLkTintAcKLpHfBDu-cM3KX7s=chQ_vtuHfHxzMC8Ny@mail.gmail.com> 
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>
	<AANLkTin1MnVNKefsEHu4O3bvdO+y8v2vaGMv5Yaq1O7G@mail.gmail.com>
	<AANLkTintAcKLpHfBDu-cM3KX7s=chQ_vtuHfHxzMC8Ny@mail.gmail.com> 
Message-ID: <201103131224.p2DCORkq025186@theraft.openend.se>

Note: I told python-dev about this behaviour 
http://mail.python.org/pipermail/python-dev/2011-March/109130.html
thinking to get a 'NotPortableWarning'.  Now python-dev is hot for making 
this a bug in Cpython as well.

Laura


From hakan at debian.org  Sun Mar 13 13:35:27 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sun, 13 Mar 2011 13:35:27 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>
References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com>
	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
	<4D7BCAB3.7020906@gmail.com>
	<AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>
	<AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>
Message-ID: <AANLkTindOXKO0qeNUmxAOzp31WXJpVgVhrBSQpZVWn=x@mail.gmail.com>

Hi agian,
I digged a bit more, and it turns out that the issue B below is
actually a special case of issue A :) The first version of the loop
becomes specialized for the variables j and x being virtual integers.
After the bridge passing through the the outer loop once, the variable
i also becomes a vritual integer. At this point a new specialized
version of the loop is generated now for the case of i, j and x being
virtual integers. The new version will contain the exact same
operations since the trace does not touch the variable i. This is
actually the intended behavior. Even though i is not touched in the
loop it might be used in some bridge from the loop and this would
allow it to stay virtual within that bridge.

The third version of the loop I think is generated because the
information that n is nonnull is not inherited by the bridge from the
loop. This is worked around in the  jit-usable_retrace branch by
producing an extra guard_nonnull() at the end of the bridge instead of
retracing. That fix could be easily cheery-picked into default as
well.

On Sun, Mar 13, 2011 at 11:12 AM, Hakan Ardo <hakan at debian.org> wrote:
> Hi,
> this is what happens here:
>
> 1. The inner loop is traced and Loop0 is produced with preamble Loop1
>
> 2. A bridge from Guard3 (the test in the while) back to Loop0 is
> traced (i.e the remaining parts of the outer loop)
>
> 3. At the end of this bridge the VirtualState does not match the
> VirtualState of Loop0, so the loop is retraced
>
> 4. The VirtualState of the newly traced version of the loop does not
> match the VirtualState at the end of the bridge so the bridge has to
> jump to the preamble instead of jumping to the new specialized version
> of the loop.
>
> 5. A bridge from Guard6 (signal/thread counter) is traced and the same
> thing happens for this bridge.
>
> This means that the additional two versions of the loop will never be
> used and should hopefully be reomved by the gc...
>
> So there are two issues:
>
> A. The additional specialized versions created does not become usable.
> This is the issue I'm working on in the jit-usable_retrace branch. The
> idea there is to have the retrace inherit the OptValue's of the
> jumpargs at the end of the bridge. This will become a fairly large
> change functionality wise...
>
> B. The VirtualStates' s differs in the first place forcing a retrace.
> This is probably fixable by introducing some more cases in
> NotVirtualInfo._generate_guards(). The jit-usable_retrace branch
> contains more cases than trunk, don't know if those are enough for
> this test though...
>
> Note however that
> jit/metainterp/test/test_nested_loops_discovered_by_bridge in
> test_loop_unroll.py, which conatins the same loop for a simple
> interpreter, does work nicely, wihtout the issues above.
>
> On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo <hakan at debian.org> wrote:
>> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>>> Hi Hakan,
>>>
>>> On 12/03/11 19:25, Hakan Ardo wrote:
>>>> Yes, this is probably the VirtualState checking. It will retrace a
>>>> loop whenever the VirtualState at the end of a bridge differs from the
>>>> VirtualState at the beginning of the compiled trace (any of the
>>>> compiled traces). This might indeed produce an identical trace if we
>>>> are unlucky, but the idea is that this should only happen rarely.
>>>
>>> ok, that's clear. So, hopefully this particular example looks a bit bad, but
>>> in general it should not be an issue. It'd be nice to have a way to check this
>>> thesis, but I agree that it's a bit hard.
>>
>> We should probably log the VirtualState together with the produced
>> loops and bridges. That would allow us to see how they differ when a
>> new version of a loop is traced. There are __repr__ methods I've been
>> using for that while debugging. They might need some rpythonizing to
>> translate though---
>>
>>>
>>>> This is because the VirtualState ?at the beginning of a trace is the
>>>> state of all the OptValue of the inputargs produced during the
>>>> optimization of the trace. This does not have to be the most general
>>>> state for which the trace is usable (which would be hard to calculate
>>>> I'm afraid).
>>>
>>> so, if I understand correctly, this is what happens:
>>>
>>> 1. we trace, optimize and compile loop A
>>>
>>> 2. after a while, we trace, optimize a compile a bridge B which then jumps
>>> back to A; by chance, the bridge looks the same as the loop
>>>
>>> Am I right?
>>
>> Maybe, I've not had the chance to look into any details yet. I'll do
>> that tomorrow...
>>
>>>
>>>> A few cases that would (most likely) result in identical traces are
>>>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>>>> gurads at the end of a bridge to make the VirtualState there match the
>>>> VirtualState of a compiled trace. This is however only done if the
>>>> guards would (most likely) not fail for the traced iteration.
>>>>
>>>> I'll look into what's happening in this particular test...
>>>
>>> I just did a quick check because I'm in a hurry, but from what I see we get
>>> three actual *loops*, not bridges.
>>
>> So if it's the same loop traced several times they should all have the
>> same preamble, and the preamble would have two bridges leading to the
>> two second versions of the loop. The preamble and it's two bridges
>> should end with different VirtualState's. The loops should ?be
>> specialized to the different VirtualState's, but if the VirtualState's
>> are similar enough (but not equal) they might consist of the exact
>> same operations.
>>
>> If there are 3 preambles for the same loop, there is a bug somewhere...
>>
>> --
>> H?kan Ard?
>>
>
>
>
> --
> H?kan Ard?
>



-- 
H?kan Ard?


From arigo at tunes.org  Sun Mar 13 14:59:42 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 13 Mar 2011 09:59:42 -0400
Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
In-Reply-To: <201103131224.p2DCORkq025186@theraft.openend.se>
References: <AANLkTikvQpnDzzwqTddg72hOAwh7aeRX=jdV_GFcjjmp@mail.gmail.com>
	<AANLkTiktdGBcLk10KWSG1zn-CJf87fjFLoRrHwj64j8a@mail.gmail.com>
	<D7B65A0B-312C-42DA-A94E-4E8182176D1C@underboss.org>
	<AANLkTin1MnVNKefsEHu4O3bvdO+y8v2vaGMv5Yaq1O7G@mail.gmail.com>
	<arigo@tunes.org>
	<AANLkTintAcKLpHfBDu-cM3KX7s=chQ_vtuHfHxzMC8Ny@mail.gmail.com>
	<201103131224.p2DCORkq025186@theraft.openend.se>
Message-ID: <AANLkTi=iWDpjqRQegkhSEZFM4DOk7XK5BZQBUaAjbtgA@mail.gmail.com>

Hi Laura,

On Sun, Mar 13, 2011 at 8:24 AM, Laura Creighton <lac at openend.se> wrote:
> Note: I told python-dev about this behaviour

Yes, I replied on the CPython bug tracker.  The fix I checked into
PyPy is for one particular case, but there are other cases that remain
as differences of behavior where CPython's is arguably wrong and
PyPy's cannot easily match it.


Armin.


From lac at openend.se  Sun Mar 13 20:04:21 2011
From: lac at openend.se (Laura Creighton)
Date: Sun, 13 Mar 2011 20:04:21 +0100
Subject: [pypy-dev] possibly of use for our documentation
Message-ID: <201103131904.p2DJ4LTi028359@theraft.openend.se>

I went and saw the pybookbuilder project's poster.
https://launchpad.net/pybookbuilder

This is Jeff Elkner and his extremely smart high school students.
They have a neat plugin for doing Latex inside sphinx, and to combine
it with ReST to make, well, books of documentation.

I don't want to lose this idea.

Laura


From anto.cuni at gmail.com  Mon Mar 14 11:49:25 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 14 Mar 2011 11:49:25 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>
References: <4D78EC62.8030101@gmail.com>	<4D7908DD.40503@gmail.com>	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>	<4D7BCAB3.7020906@gmail.com>	<AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>
	<AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>
Message-ID: <4D7DF2B5.50506@gmail.com>

Hi Hakan,
thank you for the deep explanation. Now I understand what's going on :-)

So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number),
so that the bridge from the signal/thread counter is never created and we can
forget about it.

Now, if I understand correctly, the two remaining loops are one for the case
"i non virtual" and the other for the case "i virtual", although both lead to
the same operations. I think this is the expected behavior in this case, so
are you ok if I just fix test_f1 to expect two loops?

ciao,
Anto

On 13/03/11 11:12, Hakan Ardo wrote:
> Hi,
> this is what happens here:
> 
> 1. The inner loop is traced and Loop0 is produced with preamble Loop1
> 
> 2. A bridge from Guard3 (the test in the while) back to Loop0 is
> traced (i.e the remaining parts of the outer loop)
> 
> 3. At the end of this bridge the VirtualState does not match the
> VirtualState of Loop0, so the loop is retraced
> 
> 4. The VirtualState of the newly traced version of the loop does not
> match the VirtualState at the end of the bridge so the bridge has to
> jump to the preamble instead of jumping to the new specialized version
> of the loop.
> 
> 5. A bridge from Guard6 (signal/thread counter) is traced and the same
> thing happens for this bridge.
> 
> This means that the additional two versions of the loop will never be
> used and should hopefully be reomved by the gc...
> 
> So there are two issues:
> 
> A. The additional specialized versions created does not become usable.
> This is the issue I'm working on in the jit-usable_retrace branch. The
> idea there is to have the retrace inherit the OptValue's of the
> jumpargs at the end of the bridge. This will become a fairly large
> change functionality wise...
> 
> B. The VirtualStates' s differs in the first place forcing a retrace.
> This is probably fixable by introducing some more cases in
> NotVirtualInfo._generate_guards(). The jit-usable_retrace branch
> contains more cases than trunk, don't know if those are enough for
> this test though...
> 
> Note however that
> jit/metainterp/test/test_nested_loops_discovered_by_bridge in
> test_loop_unroll.py, which conatins the same loop for a simple
> interpreter, does work nicely, wihtout the issues above.
> 
> On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo <hakan at debian.org> wrote:
>> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>>> Hi Hakan,
>>>
>>> On 12/03/11 19:25, Hakan Ardo wrote:
>>>> Yes, this is probably the VirtualState checking. It will retrace a
>>>> loop whenever the VirtualState at the end of a bridge differs from the
>>>> VirtualState at the beginning of the compiled trace (any of the
>>>> compiled traces). This might indeed produce an identical trace if we
>>>> are unlucky, but the idea is that this should only happen rarely.
>>>
>>> ok, that's clear. So, hopefully this particular example looks a bit bad, but
>>> in general it should not be an issue. It'd be nice to have a way to check this
>>> thesis, but I agree that it's a bit hard.
>>
>> We should probably log the VirtualState together with the produced
>> loops and bridges. That would allow us to see how they differ when a
>> new version of a loop is traced. There are __repr__ methods I've been
>> using for that while debugging. They might need some rpythonizing to
>> translate though---
>>
>>>
>>>> This is because the VirtualState  at the beginning of a trace is the
>>>> state of all the OptValue of the inputargs produced during the
>>>> optimization of the trace. This does not have to be the most general
>>>> state for which the trace is usable (which would be hard to calculate
>>>> I'm afraid).
>>>
>>> so, if I understand correctly, this is what happens:
>>>
>>> 1. we trace, optimize and compile loop A
>>>
>>> 2. after a while, we trace, optimize a compile a bridge B which then jumps
>>> back to A; by chance, the bridge looks the same as the loop
>>>
>>> Am I right?
>>
>> Maybe, I've not had the chance to look into any details yet. I'll do
>> that tomorrow...
>>
>>>
>>>> A few cases that would (most likely) result in identical traces are
>>>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>>>> gurads at the end of a bridge to make the VirtualState there match the
>>>> VirtualState of a compiled trace. This is however only done if the
>>>> guards would (most likely) not fail for the traced iteration.
>>>>
>>>> I'll look into what's happening in this particular test...
>>>
>>> I just did a quick check because I'm in a hurry, but from what I see we get
>>> three actual *loops*, not bridges.
>>
>> So if it's the same loop traced several times they should all have the
>> same preamble, and the preamble would have two bridges leading to the
>> two second versions of the loop. The preamble and it's two bridges
>> should end with different VirtualState's. The loops should  be
>> specialized to the different VirtualState's, but if the VirtualState's
>> are similar enough (but not equal) they might consist of the exact
>> same operations.
>>
>> If there are 3 preambles for the same loop, there is a bug somewhere...
>>
>> --
>> H?kan Ard?
>>
> 
> 
> 



From hakan at debian.org  Mon Mar 14 11:52:55 2011
From: hakan at debian.org (Hakan Ardo)
Date: Mon, 14 Mar 2011 11:52:55 +0100
Subject: [pypy-dev] problem after merging of jit-virtual_state
In-Reply-To: <4D7DF2B5.50506@gmail.com>
References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com>
	<AANLkTimHkEJeXWkw10Y=OwiU9ie49s-UB+Vhahrt=Hbu@mail.gmail.com>
	<4D7BCAB3.7020906@gmail.com>
	<AANLkTinUsrWBWnKLQpoKUbfh=UDjCd4HfT29b+hXpkNB@mail.gmail.com>
	<AANLkTinVGg=opEtVD9m=EPM3DsqB7t502ijT2CFENW-B@mail.gmail.com>
	<4D7DF2B5.50506@gmail.com>
Message-ID: <AANLkTimxUC4VoRuBRyjrr5Jcn-YiyVLDJgQYE254r07F@mail.gmail.com>

On Mon, Mar 14, 2011 at 11:49 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Hi Hakan,
> thank you for the deep explanation. Now I understand what's going on :-)
>
> So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number),
> so that the bridge from the signal/thread counter is never created and we can
> forget about it.
>
> Now, if I understand correctly, the two remaining loops are one for the case
> "i non virtual" and the other for the case "i virtual", although both lead to
> the same operations. I think this is the expected behavior in this case, so
> are you ok if I just fix test_f1 to expect two loops?

Yes

>
> ciao,
> Anto
>
> On 13/03/11 11:12, Hakan Ardo wrote:
>> Hi,
>> this is what happens here:
>>
>> 1. The inner loop is traced and Loop0 is produced with preamble Loop1
>>
>> 2. A bridge from Guard3 (the test in the while) back to Loop0 is
>> traced (i.e the remaining parts of the outer loop)
>>
>> 3. At the end of this bridge the VirtualState does not match the
>> VirtualState of Loop0, so the loop is retraced
>>
>> 4. The VirtualState of the newly traced version of the loop does not
>> match the VirtualState at the end of the bridge so the bridge has to
>> jump to the preamble instead of jumping to the new specialized version
>> of the loop.
>>
>> 5. A bridge from Guard6 (signal/thread counter) is traced and the same
>> thing happens for this bridge.
>>
>> This means that the additional two versions of the loop will never be
>> used and should hopefully be reomved by the gc...
>>
>> So there are two issues:
>>
>> A. The additional specialized versions created does not become usable.
>> This is the issue I'm working on in the jit-usable_retrace branch. The
>> idea there is to have the retrace inherit the OptValue's of the
>> jumpargs at the end of the bridge. This will become a fairly large
>> change functionality wise...
>>
>> B. The VirtualStates' s differs in the first place forcing a retrace.
>> This is probably fixable by introducing some more cases in
>> NotVirtualInfo._generate_guards(). The jit-usable_retrace branch
>> contains more cases than trunk, don't know if those are enough for
>> this test though...
>>
>> Note however that
>> jit/metainterp/test/test_nested_loops_discovered_by_bridge in
>> test_loop_unroll.py, which conatins the same loop for a simple
>> interpreter, does work nicely, wihtout the issues above.
>>
>> On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo <hakan at debian.org> wrote:
>>> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>>>> Hi Hakan,
>>>>
>>>> On 12/03/11 19:25, Hakan Ardo wrote:
>>>>> Yes, this is probably the VirtualState checking. It will retrace a
>>>>> loop whenever the VirtualState at the end of a bridge differs from the
>>>>> VirtualState at the beginning of the compiled trace (any of the
>>>>> compiled traces). This might indeed produce an identical trace if we
>>>>> are unlucky, but the idea is that this should only happen rarely.
>>>>
>>>> ok, that's clear. So, hopefully this particular example looks a bit bad, but
>>>> in general it should not be an issue. It'd be nice to have a way to check this
>>>> thesis, but I agree that it's a bit hard.
>>>
>>> We should probably log the VirtualState together with the produced
>>> loops and bridges. That would allow us to see how they differ when a
>>> new version of a loop is traced. There are __repr__ methods I've been
>>> using for that while debugging. They might need some rpythonizing to
>>> translate though---
>>>
>>>>
>>>>> This is because the VirtualState ?at the beginning of a trace is the
>>>>> state of all the OptValue of the inputargs produced during the
>>>>> optimization of the trace. This does not have to be the most general
>>>>> state for which the trace is usable (which would be hard to calculate
>>>>> I'm afraid).
>>>>
>>>> so, if I understand correctly, this is what happens:
>>>>
>>>> 1. we trace, optimize and compile loop A
>>>>
>>>> 2. after a while, we trace, optimize a compile a bridge B which then jumps
>>>> back to A; by chance, the bridge looks the same as the loop
>>>>
>>>> Am I right?
>>>
>>> Maybe, I've not had the chance to look into any details yet. I'll do
>>> that tomorrow...
>>>
>>>>
>>>>> A few cases that would (most likely) result in identical traces are
>>>>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>>>>> gurads at the end of a bridge to make the VirtualState there match the
>>>>> VirtualState of a compiled trace. This is however only done if the
>>>>> guards would (most likely) not fail for the traced iteration.
>>>>>
>>>>> I'll look into what's happening in this particular test...
>>>>
>>>> I just did a quick check because I'm in a hurry, but from what I see we get
>>>> three actual *loops*, not bridges.
>>>
>>> So if it's the same loop traced several times they should all have the
>>> same preamble, and the preamble would have two bridges leading to the
>>> two second versions of the loop. The preamble and it's two bridges
>>> should end with different VirtualState's. The loops should ?be
>>> specialized to the different VirtualState's, but if the VirtualState's
>>> are similar enough (but not equal) they might consist of the exact
>>> same operations.
>>>
>>> If there are 3 preambles for the same loop, there is a bug somewhere...
>>>
>>> --
>>> H?kan Ard?
>>>
>>
>>
>>
>
>



-- 
H?kan Ard?


From lac at openend.se  Mon Mar 14 19:13:34 2011
From: lac at openend.se (Laura Creighton)
Date: Mon, 14 Mar 2011 19:13:34 +0100
Subject: [pypy-dev] Thinking about the GIL
Message-ID: <201103141813.p2EIDY9C027808@theraft.openend.se>


Robert Hancock hosted a bof at pycon about concurrency and multiprocessing.
I went there looking to find out how other people were doing things,
especially looking for information about how other languages handled
things.  It would be nice to kill the GIL, if only we knew of a
brilliant way to do this.

Unfortunately, I was one year to late for this discussion.  This is
what Robert Hancock David Beazley, Peter Portante and others discussed at Pycon
_last year_.  So I asked Robert Hancock for the notes he took then.
(I continue after this  forwarded message)

------- Forwarded Message

Return-Path: hancock.robert at gmail.com
Delivery-Date: Mon Mar 14 17:09:51 2011
Return-Path: <hancock.robert at gmail.com>
Subject: Re: please send me the notes you took last year
To: Laura Creighton <lac at openend.se>

These are the books that I mentioned:
Machine Learning: An Algorithmic Perspective
http://www.amazon.com/gp/product/1420067184
 <http://www.amazon.com/gp/product/1420067184>
I found this more approachable than the Bishop  and a number of 
examples are in Python.

Introduction to Data Mining
http://www.amazon.com/gp/product/0321321367
<http://www.amazon.com/gp/product/0321321367>
I've only started this, but it is nice with David Mease's Google Tech Talk 
series. http://www.youtube.com/watch?v=zRsMEl6PHhM

1.  Make all IO non-blocking and mediate the processes like greenlets.  This
does not allow you take advantage of the OS level thread scheduler which is
far more sophisticated than greenlets.  See the Linux kernel specifications
for the details of the multi-level feedback queue.

2. Construct a multi-level feedback queue within Python.  This is
extraordinarily complicated and complex to implement.  Why duplicate what
already exists?

3. Do we need to maintain compatibility with being able to call out to C
functions?  The primary complaint about the GIL is that it does not
efficiently handle CPU bound processes and multi-cores.  Running sequential 
processes in threads on multi-cores can actually slow down the processes.

4. Who has already solved this problem as part of the language?
    -   Erlang (No one knew the nitty gritty details.)

    -   Go - based on Tony Hoare's CSP and the work done on Plan 9 at Bell
Labs.  Uses the system scheduler and creates its own mini-threads (4k).
Need to investiagate the source code on line.  Goroutines do not have OS thread
affinity; they can multiplex over multiple threads.

    -   Java - Early on Java used several versions of Greenlets, but now
uses system threads.  The JVM punts to the OS.

Conclusions
--------------
1.  Do not reinvent the wheel!  Many people have worked decades on this
problem.  Leverage thier expertise.

2.  Coroutines are frequently better than threads, but do scale and each
coroutine must me restarted in the thread where it was spawned.  See
greenlet.c.  Greenlets are also chained and have mutual dependencies.  
The order of execution is arbitrary with not method for priorities.

3.  Investigate if there is an alternative to the current method of calling
external C objects.

4. Dave did a POC on priorities:
http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html

5.  Everyone agreed that some type of priority mechanism is a good idea, but
wanted to see what Unladen Swallow does. (As of March 2011 Google is no
longer actively developing this project.)


References
- -----------
Dave Beazley - GIL Wars
Dave Beazley - Yieldable Threads http://www.dabeaz.com/blog.html
Linux Kernel http://goo.gl/RkxVs
Erlang
Go - golang.org
CSP - Tony Hoare http://www.usingcsp.com/cspbook.pdf

I spoke with Peter Portante yesterday, and he would be very interested in
participating even though he has very little free time.  Peter works at HP
and worked on their OS threading model.  Also, see his Pycon 2010 talk on
non-blocking IO and the 2011 talk on co-routines.

Let me know if you have any questions.

Bob Hancock

Blog - www.bobhancock.org
Twitter - bob_hancock and nycgtug
------- End of Forwarded Message

And, indeed, Peter Portante is very interested in thinking about doing
without the GIL.  He's already sent me this:

Date:    Sun, 13 Mar 2011 16:42:00 -0400
To:      Laura Creighton <lac at openend.se>
From:    Peter Portante <peter.a.portante at gmail.com>
Subject: Re: [pypy-dev] possibly of use for our documentation

Return-Path: peter.a.portante at gmail.com
Delivery-Date: Sun Mar 13 21:42:21 2011
Return-Path: <peter.a.portante at gmail.com>
Hi Laura,

Just left pycon and heard about talks of pypy removing the gil.

I work on tru64 unix's thread library for 8 years. If there is any thing I 
can do to help with this effort, please let me know.

Thanks,

-peter
-------------------------

Note: I have never promised anybody anything.  This was a 'please
educate me appeal'.  But Bob Hancock is coming back this afternoon
to talk with us.

Anybody got any questions they want to make sure I ask him?

Laura


From tbaldridge at gmail.com  Mon Mar 14 20:08:20 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Mon, 14 Mar 2011 14:08:20 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <201103141813.p2EIDY9C027808@theraft.openend.se>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
Message-ID: <AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>

I guess I'm missing something, but what's wrong with simply ripping
out the GIL? In C# we have threads, C FFI (via PInvoke), and never
have any major issues with threading. It seems to me that the GIL is
only needed because assumptions were made when writing the
interpreter. I don't know if it's full of global variables or
something, but can anyone explain why a GIL is needed at all? I've
done quite a bit of multi-threading programming, and I fail to see the
need for a GIL.

>Why duplicate what
> already exists?

Timothy


From benjamin at python.org  Mon Mar 14 20:12:35 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 14 Mar 2011 14:12:35 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
Message-ID: <AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>

2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> I guess I'm missing something, but what's wrong with simply ripping
> out the GIL? In C# we have threads, C FFI (via PInvoke), and never
> have any major issues with threading. It seems to me that the GIL is
> only needed because assumptions were made when writing the
> interpreter. I don't know if it's full of global variables or
> something, but can anyone explain why a GIL is needed at all? I've
> done quite a bit of multi-threading programming, and I fail to see the
> need for a GIL.

Many reasons. The biggest being python data structures aren't thread-safe.



-- 
Regards,
Benjamin


From amauryfa at gmail.com  Mon Mar 14 20:19:53 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 14 Mar 2011 20:19:53 +0100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
Message-ID: <AANLkTinG18peOMkz3ubQcpdPA=3R72Y5nyKS2o2R+9uc@mail.gmail.com>

Hi,

2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> I guess I'm missing something, but what's wrong with simply ripping
> out the GIL? In C# we have threads, C FFI (via PInvoke), and never
> have any major issues with threading. It seems to me that the GIL is
> only needed because assumptions were made when writing the
> interpreter. I don't know if it's full of global variables or
> something, but can anyone explain why a GIL is needed at all? I've
> done quite a bit of multi-threading programming, and I fail to see the
> need for a GIL.

The GIL greatly simplifies multi-threaded programming in Python.
For example, you are guaranteed that someList.append(x) won't run into
some race condition and crash the program.

In CPython, simply incrementing the reference count of an object needs some
synchronization between threads. And in PyPy, garbage collectors are
not (yet) thread-safe.

See better explanations about the GIL in CPython:
http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
http://effbot.org/pyfaq/can-t-we-get-rid-of-the-global-interpreter-lock.htm

-- 
Amaury Forgeot d'Arc


From benjamin at python.org  Mon Mar 14 20:28:39 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 14 Mar 2011 14:28:39 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
Message-ID: <AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>

2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> They may not be thread-safe, but as far as a program goes, do we
> really care? If I have two threads adding items to the same list,
> should I really be expecting the interpreter to keep things straight?
> What's wrong with forcing the user to lock structures before editing
> them? This is something that Java, and C#, and C++ all require.

Yes, the Python interpreter should never crash because of user mistakes.



-- 
Regards,
Benjamin


From Ben.Young at sungard.com  Tue Mar 15 09:20:11 2011
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Tue, 15 Mar 2011 08:20:11 +0000
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se><AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com><AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com><AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
Message-ID: <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>



> -----Original Message-----
> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
> bounces at codespeak.net] On Behalf Of Benjamin Peterson
> Sent: 14 March 2011 19:29
> To: Timothy Baldridge
> Cc: PyPy Dev
> Subject: Re: [pypy-dev] Thinking about the GIL
> 
> 2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> > They may not be thread-safe, but as far as a program goes, do we
> > really care? If I have two threads adding items to the same list,
> > should I really be expecting the interpreter to keep things
straight?
> > What's wrong with forcing the user to lock structures before editing
> > them? This is something that Java, and C#, and C++ all require.
> 
> Yes, the Python interpreter should never crash because of user
> mistakes.
> 

C# structures aren't thread-safe, but you can't crash the CLR by using
them in a multithreaded manner

Ben

> 
> 
> --
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev




From benjamin at python.org  Tue Mar 15 15:17:58 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 15 Mar 2011 09:17:58 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
Message-ID: <AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>

2011/3/15  <Ben.Young at sungard.com>:
>
>
>> -----Original Message-----
>> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
>> bounces at codespeak.net] On Behalf Of Benjamin Peterson
>> Sent: 14 March 2011 19:29
>> To: Timothy Baldridge
>> Cc: PyPy Dev
>> Subject: Re: [pypy-dev] Thinking about the GIL
>>
>> 2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
>> > They may not be thread-safe, but as far as a program goes, do we
>> > really care? If I have two threads adding items to the same list,
>> > should I really be expecting the interpreter to keep things
> straight?
>> > What's wrong with forcing the user to lock structures before editing
>> > them? This is something that Java, and C#, and C++ all require.
>>
>> Yes, the Python interpreter should never crash because of user
>> mistakes.
>>
>
> C# structures aren't thread-safe, but you can't crash the CLR by using
> them in a multithreaded manner

The primitive data structures must be thread-safe, though.



-- 
Regards,
Benjamin


From Ben.Young at sungard.com  Tue Mar 15 15:36:50 2011
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Tue, 15 Mar 2011 14:36:50 +0000
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se><AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com><AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com><AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com><AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com><01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
Message-ID: <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp>



> -----Original Message-----
> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
> bounces at codespeak.net] On Behalf Of Benjamin Peterson
> Sent: 15 March 2011 14:18
> To: Young, Ben
> Cc: pypy-dev at codespeak.net
> Subject: Re: [pypy-dev] Thinking about the GIL
> 
> 2011/3/15  <Ben.Young at sungard.com>:
> >
> >
> >> -----Original Message-----
> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson
> >> Sent: 14 March 2011 19:29
> >> To: Timothy Baldridge
> >> Cc: PyPy Dev
> >> Subject: Re: [pypy-dev] Thinking about the GIL
> >>
> >> 2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> >> > They may not be thread-safe, but as far as a program goes, do we
> >> > really care? If I have two threads adding items to the same list,
> >> > should I really be expecting the interpreter to keep things
> > straight?
> >> > What's wrong with forcing the user to lock structures before
> editing
> >> > them? This is something that Java, and C#, and C++ all require.
> >>
> >> Yes, the Python interpreter should never crash because of user
> >> mistakes.
> >>
> >
> > C# structures aren't thread-safe, but you can't crash the CLR by
> using
> > them in a multithreaded manner
> 
> The primitive data structures must be thread-safe, though.
> 

No, just cleverly designed with the memory model in mind I believe (at
least there's no locking or even atomic synchronisation), and you can
get errors, it just wont bring down the VM

Cheers,
Ben

> 
> 
> --
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev




From tbaldridge at gmail.com  Tue Mar 15 16:04:04 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Tue, 15 Mar 2011 10:04:04 -0500
Subject: [pypy-dev] Any Multithreading Primitives?
Message-ID: <AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com>

Does PyPy have any sort of multithreading primitives that are
available in rpython programs? I'm specifically looking for CAS. I can
probably simulate a mutex with CAS. Or do I have to drop down to some
sort of assembly generation to atomically update a object reference?

Thanks,

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From lac at openend.se  Tue Mar 15 16:56:23 2011
From: lac at openend.se (Laura Creighton)
Date: Tue, 15 Mar 2011 16:56:23 +0100
Subject: [pypy-dev] Any Multithreading Primitives?
In-Reply-To: Message from Timothy Baldridge <tbaldridge@gmail.com> of "Tue,
	15 Mar 2011 10:04:04 EST."
	<AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com> 
References: <AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com> 
Message-ID: <201103151556.p2FFuNqE017090@theraft.openend.se>

In a message of Tue, 15 Mar 2011 10:04:04 EST, Timothy Baldridge writes:
>Does PyPy have any sort of multithreading primitives that are
>available in rpython programs? I'm specifically looking for CAS. I can
>probably simulate a mutex with CAS. Or do I have to drop down to some
>sort of assembly generation to atomically update a object reference?
>
>Thanks,
>
>Timothy


Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html
be of any use to you?  Note, this documentation isquite old and needs
attention, and right now the stackless transform does not work with the JIT.
changing things so that it would work doesn't seem to be that much work,
though, we just haven't gotten around to it.

Laura



From tbaldridge at gmail.com  Tue Mar 15 17:04:30 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Tue, 15 Mar 2011 11:04:30 -0500
Subject: [pypy-dev] Any Multithreading Primitives?
In-Reply-To: <201103151556.p2FFuNqE017090@theraft.openend.se>
References: <tbaldridge@gmail.com>
	<AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com>
	<201103151556.p2FFuNqE017090@theraft.openend.se>
Message-ID: <AANLkTim5eW3EYVuxL5GGYgj4rVxro6bfL+MX8eG6ELXm@mail.gmail.com>

> Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html
> be of any use to you?  Note, this documentation isquite old and needs
> attention, and right now the stackless transform does not work with the JIT.
> changing things so that it would work doesn't seem to be that much work,
> though, we just haven't gotten around to it.


Not really. What I need are CAS operations so I can do the following:

while(True):
   oldval = var
   newval = oldval + 1
   if cas(oldval, newval, var):
       break

Basically cas takes the value you think var currently has, the value
you want var to be, and the var. Cas then updates var if and only if
var's current value == oldvar. But cas is atomic. So basically it
returns false if the cas failed, and true if it worked. Using cas you
can build mutexes, shared objects, etc.

http://en.wikipedia.org/wiki/Compare-and-swap

Timothy


From amauryfa at gmail.com  Tue Mar 15 17:15:41 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 15 Mar 2011 17:15:41 +0100
Subject: [pypy-dev] Any Multithreading Primitives?
In-Reply-To: <AANLkTim5eW3EYVuxL5GGYgj4rVxro6bfL+MX8eG6ELXm@mail.gmail.com>
References: <tbaldridge@gmail.com>
	<AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com>
	<201103151556.p2FFuNqE017090@theraft.openend.se>
	<AANLkTim5eW3EYVuxL5GGYgj4rVxro6bfL+MX8eG6ELXm@mail.gmail.com>
Message-ID: <AANLkTimTapiHg4E+BRgsHbVG_pfhZpfLvie-4xdVUgaR@mail.gmail.com>

Hi,

2011/3/15 Timothy Baldridge <tbaldridge at gmail.com>:
> Basically cas takes the value you think var currently has, the value
> you want var to be, and the var. Cas then updates var if and only if
> var's current value == oldvar. But cas is atomic. So basically it
> returns false if the cas failed, and true if it worked. Using cas you
> can build mutexes, shared objects, etc.

In pypy/module/thread/ll_thread.py, allocate_lock() creates a Lock object,
similar to the Python's thread.Lock.

Careful though with threads: most of PyPy's garbage collectors are not
thread-safe.

-- 
Amaury Forgeot d'Arc


From santagada at gmail.com  Tue Mar 15 18:25:23 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Tue, 15 Mar 2011 14:25:23 -0300
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp>
Message-ID: <AANLkTi=f2dQim-rUi_JeheLN4fmtXtqVQPF=N5wJWYkO@mail.gmail.com>

On Tue, Mar 15, 2011 at 11:36 AM,  <Ben.Young at sungard.com> wrote:
>
>
>> -----Original Message-----
>> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
>> bounces at codespeak.net] On Behalf Of Benjamin Peterson
>> Sent: 15 March 2011 14:18
>> To: Young, Ben
>> Cc: pypy-dev at codespeak.net
>> Subject: Re: [pypy-dev] Thinking about the GIL
>>
>> 2011/3/15 ?<Ben.Young at sungard.com>:
>> >
>> >
>> >> -----Original Message-----
>> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
>> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson
>> >> Sent: 14 March 2011 19:29
>> >> To: Timothy Baldridge
>> >> Cc: PyPy Dev
>> >> Subject: Re: [pypy-dev] Thinking about the GIL
>> >>
>> >> 2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
>> >> > They may not be thread-safe, but as far as a program goes, do we
>> >> > really care? If I have two threads adding items to the same list,
>> >> > should I really be expecting the interpreter to keep things
>> > straight?
>> >> > What's wrong with forcing the user to lock structures before
>> editing
>> >> > them? This is something that Java, and C#, and C++ all require.
>> >>
>> >> Yes, the Python interpreter should never crash because of user
>> >> mistakes.
>> >>
>> >
>> > C# structures aren't thread-safe, but you can't crash the CLR by
>> using
>> > them in a multithreaded manner
>>
>> The primitive data structures must be thread-safe, though.
>>
>
> No, just cleverly designed with the memory model in mind I believe (at
> least there's no locking or even atomic synchronisation), and you can
> get errors, it just wont bring down the VM

That is what I think he meant. The only way that I know for a vm to be
thread-safe is its internal structures to be thread-safe. The exposed
C# structures might not be, but the VM ones have to.

-- 
Leonardo Santagada


From Ben.Young at sungard.com  Wed Mar 16 09:24:17 2011
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Wed, 16 Mar 2011 08:24:17 +0000
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTi=f2dQim-rUi_JeheLN4fmtXtqVQPF=N5wJWYkO@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTi=f2dQim-rUi_JeheLN4fmtXtqVQPF=N5wJWYkO@mail.gmail.com>
Message-ID: <01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp>


> -----Original Message-----
> From: Leonardo Santagada [mailto:santagada at gmail.com]
> Sent: 15 March 2011 17:25
> To: Young, Ben
> Cc: benjamin at python.org; pypy-dev at codespeak.net
> Subject: Re: [pypy-dev] Thinking about the GIL
> 
> On Tue, Mar 15, 2011 at 11:36 AM,  <Ben.Young at sungard.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson
> >> Sent: 15 March 2011 14:18
> >> To: Young, Ben
> >> Cc: pypy-dev at codespeak.net
> >> Subject: Re: [pypy-dev] Thinking about the GIL
> >>
> >> 2011/3/15 ?<Ben.Young at sungard.com>:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev-
> >> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson
> >> >> Sent: 14 March 2011 19:29
> >> >> To: Timothy Baldridge
> >> >> Cc: PyPy Dev
> >> >> Subject: Re: [pypy-dev] Thinking about the GIL
> >> >>
> >> >> 2011/3/14 Timothy Baldridge <tbaldridge at gmail.com>:
> >> >> > They may not be thread-safe, but as far as a program goes, do
> we
> >> >> > really care? If I have two threads adding items to the same
> list,
> >> >> > should I really be expecting the interpreter to keep things
> >> > straight?
> >> >> > What's wrong with forcing the user to lock structures before
> >> editing
> >> >> > them? This is something that Java, and C#, and C++ all require.
> >> >>
> >> >> Yes, the Python interpreter should never crash because of user
> >> >> mistakes.
> >> >>
> >> >
> >> > C# structures aren't thread-safe, but you can't crash the CLR by
> >> using
> >> > them in a multithreaded manner
> >>
> >> The primitive data structures must be thread-safe, though.
> >>
> >
> > No, just cleverly designed with the memory model in mind I believe
> (at
> > least there's no locking or even atomic synchronisation), and you can
> > get errors, it just wont bring down the VM
> 
> That is what I think he meant. The only way that I know for a vm to be
> thread-safe is its internal structures to be thread-safe. The exposed
> C# structures might not be, but the VM ones have to.
> 

Yes, I guess so. I just meant that you won't run into scaling issues in C# because the core VM is locking all over the place or anything like that. In Python (and PyPy) the problem is worse because the VM is implemented using many of the same structures as are exposed at app level, whereas the CLR is much lower level and so is probably easier to write in a thread safe manner. 

Having said that, if the PyPy GCs were thread-safe, how many other structures would need to be changed to allow a free threaded python that's "use at own risk" e.g. one that's allowed to crash if you do silly things as I guess most people would be happy with that. The import machinery would need to be thread safe but how much more?

Cheers,
Ben

> --
> Leonardo Santagada


From amauryfa at gmail.com  Wed Mar 16 09:59:35 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 16 Mar 2011 09:59:35 +0100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTi=f2dQim-rUi_JeheLN4fmtXtqVQPF=N5wJWYkO@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp>
Message-ID: <AANLkTi=+bJMxkssb31cOLouLCoTt+0c6KaVQp7SuyJOm@mail.gmail.com>

Hi

2011/3/16  <Ben.Young at sungard.com>:
> Having said that, if the PyPy GCs were thread-safe, how many other
> structures would need to be changed to allow a free threaded python that's
> "use at own risk" e.g. one that's allowed to crash if you do silly things as
> I guess most people would be happy with that. The import machinery would
> need to be thread safe but how much more?

All low-level structures: rlist, rdict &co.
For example these functions would need some kind of synchronization:
https://bitbucket.org/pypy/pypy/src/6dabfe362323/pypy/rpython/lltypesystem/rdict.py#cl-712
And probably many more...

-- 
Amaury Forgeot d'Arc


From renesd at gmail.com  Wed Mar 16 10:52:26 2011
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Wed, 16 Mar 2011 09:52:26 +0000
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
Message-ID: <AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>

Hi,

one alternative approach is to have a separate VM in each thread.  Then pass
messages between them.  Works well, and no GIL in each VM.  You have to have
clean code that allows you to have a separate VMs in a process.  However,
it's easier to make your code be able to run in separate VMs, than to recode
it to allow concurrent thread access to all data structures.  This way is
easier to implement than removing a GIL.

eg,
>>> vms = [VM() for i in range(8)]
>>> vms[0].send(["going in"])
>>> vms[0].get()
["coming out!"]

I did this with tinypy vms in a cpython host, and it worked kind of ok.  It
still used the GIL when I wanted to communicate with the VMs.  Only one vm
could be talked to at a time.  I used it with the SDL "fastevent" event
queue, so each vm posted into the queue could be read from cpython.  The
other communication that worked well was using mmap'd data structures.  This
means you do not need to serialise the data when sharing messages between
threads.

(As a side note... It turned out to be less code just using CPython separate
processes communicating via shared mmap buffers for this particular task.
Since the data was all numpy/pygame.Surface they could be shared easily.  If
it was pure python code, it probably would have been a different matter.)


Apart from easier implementation on the VM level, this approach also has
other advantages.  One is that it makes message sharing more explicit
between threads.The other is that GC pressure is made smaller.  Each VM has
its own GC heap, rather than all of the objects in one big heap shared by
all the threads.

(another aside, CPython can also use multiple vms in one process, and that's
how some webservers have embedded python... one python vm per thread).

cya.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110316/671ff567/attachment.html>

From Ben.Young at sungard.com  Wed Mar 16 11:06:36 2011
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Wed, 16 Mar 2011 10:06:36 +0000
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se><AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com><AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com><AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com><AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com><01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp><AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
Message-ID: <01781CA2CC22B145B230504679ECF48C030D3081@EMEA-EXCHANGE03.internal.sungard.corp>

It's a nice solution for lots of problems, but unfortunately not the areas I'm interested in, where there's a shared constant memory data set of 20-100GB and 32-64 threads all performing calculations on it. As soon as you do any data copying you blow the memory requirements and kill the performance. Of course, I can't say I'm working in a "standard" environment, but for some things nothing but true threading will do

 

Note this is in C# which does scale quite well to 64 threads, as long as you don't do any allocation!

 

Cheers,

Ben

 

From: Ren? Dudfield [mailto:renesd at gmail.com] 
Sent: 16 March 2011 09:52
To: Benjamin Peterson
Cc: Young, Ben; pypy-dev at codespeak.net
Subject: Re: [pypy-dev] Thinking about the GIL

 

Hi,

one alternative approach is to have a separate VM in each thread.  Then pass messages between them.  Works well, and no GIL in each VM.  You have to have clean code that allows you to have a separate VMs in a process.  However, it's easier to make your code be able to run in separate VMs, than to recode it to allow concurrent thread access to all data structures.  This way is easier to implement than removing a GIL.

eg,
>>> vms = [VM() for i in range(8)]
>>> vms[0].send(["going in"])
>>> vms[0].get()
["coming out!"]

I did this with tinypy vms in a cpython host, and it worked kind of ok.  It still used the GIL when I wanted to communicate with the VMs.  Only one vm could be talked to at a time.  I used it with the SDL "fastevent" event queue, so each vm posted into the queue could be read from cpython.  The other communication that worked well was using mmap'd data structures.  This means you do not need to serialise the data when sharing messages between threads.

(As a side note... It turned out to be less code just using CPython separate processes communicating via shared mmap buffers for this particular task.  Since the data was all numpy/pygame.Surface they could be shared easily.  If it was pure python code, it probably would have been a different matter.)


Apart from easier implementation on the VM level, this approach also has other advantages.  One is that it makes message sharing more explicit between threads.The other is that GC pressure is made smaller.  Each VM has its own GC heap, rather than all of the objects in one big heap shared by all the threads.

(another aside, CPython can also use multiple vms in one process, and that's how some webservers have embedded python... one python vm per thread).

cya.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110316/70ce1c57/attachment.html>

From tbaldridge at gmail.com  Wed Mar 16 14:13:06 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Wed, 16 Mar 2011 08:13:06 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
Message-ID: <AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>

> one alternative approach is to have a separate VM in each thread.? Then pass
> messages between them.? Works well, and no GIL in each VM.? You have to have
> clean code that allows you to have a separate VMs in a process.? However,
> it's easier to make your code be able to run in separate VMs, than to recode
> it to allow concurrent thread access to all data structures.? This way is
> easier to implement than removing a GIL.
>

This is what Erlang does, but unfortunately it scales very poorly when
 you have large datasets (as mentioned above). If you want 4 threads
pounding on 1GB of data, either they need to ask a 5th thread for each
data item, or each thread needs its own copy of that 1GB dataset. Even
embarrassingly parallel things such as raytracing are very hard in
this model.

Timothy


From bob at redivi.com  Wed Mar 16 18:30:29 2011
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 16 Mar 2011 13:30:29 -0400
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
	<AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
Message-ID: <AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>

On Wed, Mar 16, 2011 at 9:13 AM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
>> one alternative approach is to have a separate VM in each thread.? Then pass
>> messages between them.? Works well, and no GIL in each VM.? You have to have
>> clean code that allows you to have a separate VMs in a process.? However,
>> it's easier to make your code be able to run in separate VMs, than to recode
>> it to allow concurrent thread access to all data structures.? This way is
>> easier to implement than removing a GIL.
>>
>
> This is what Erlang does, but unfortunately it scales very poorly when
> ?you have large datasets (as mentioned above). If you want 4 threads
> pounding on 1GB of data, either they need to ask a 5th thread for each
> data item, or each thread needs its own copy of that 1GB dataset. Even
> embarrassingly parallel things such as raytracing are very hard in
> this model.

If the data is constant you can share it in Erlang by generating a
module for it and putting it in the code server (a hack, but it works)
or simply using a binary. These are zero-copy in Erlang.

-bob


From tbaldridge at gmail.com  Wed Mar 16 18:51:37 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Wed, 16 Mar 2011 12:51:37 -0500
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
	<AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
	<AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>
Message-ID: <AANLkTikqnP0hJBRbLEa1Ka3T8ih_hwaZ=Y-DR-WqWZ-s@mail.gmail.com>

> If the data is constant you can share it in Erlang by generating a
> module for it and putting it in the code server (a hack, but it works)
> or simply using a binary. These are zero-copy in Erlang.
>
> -bob

True, but that doesn't really solve the problem of writes; you can't
have 5 threads hammering the same data structure. In essence this
model doesn't really solve any more of the same basic issues that
multiprocessing does. In the erlang model you can have thousands of
"threads", and currently with multiprocessing you're limited to <
100-ish. But they still both require you to pass messages.

We should probably mention a somewhat new method of co-currency. This
is my favorite, and is used by Clojure. First of all, I must preface
this with the statement that almost all structures in Clojure are
immutable:

d = {}
d = d.assoc(key, value)

l = []
l = l.append("foo")

So basically every time you add or remove something from a collection
it returns a new collection. Behind the scenes the collections
carefully manipulate the structures as to a) not modify the old
structure and b) maintain reasonable performance (adding a item to a
hashset does not copy the entire hashset).

That alone gets rid of 90% of your co-currency issues. From there you
have refs for Software Transactional Memory. Here's an example of how
it would be used in a Python-esque syntax:

val = 0
account1 = Ref(100)
account2 = Ref(0)

transaction:
   account1.value = account1.value - 50
   account2.value = account2.value + 50

The transaction block insures that either the entire block succeeds,
or fails as an atomic unit.

Now the nice thing is, most of this can be done without any special
syntax. It's possible to use the Clojure-CLR STM libs within C#, and I
have, and it works like a dream. It's a tad ugly, but it works. Adding
a bit of syntax sugar to python (like the transaction block) should
make this quite elegant.

At any rate, here's a video from Rich Hickey the creator of Clojure on
how modern lock based systems and even perhaps the Erlang model is
quite flawed: http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey

Just some more points to think over,

Timothy


From tbaldridge at gmail.com  Wed Mar 16 18:53:44 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Wed, 16 Mar 2011 12:53:44 -0500
Subject: [pypy-dev] *Rant* No Reply-To?
Message-ID: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>

Why is there no "Reply-To" in the headers on this list? Whenever, in
gmail, I hit "reply" it automatically sets the to address to the
"from" field in the mailing list e-mail. So I end up e-mailing the
person directly instead of the list. This is highly frustrating.

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From lac at openend.se  Wed Mar 16 19:34:18 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 16 Mar 2011 19:34:18 +0100
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: Message from Timothy Baldridge <tbaldridge@gmail.com> of "Wed,
	16 Mar 2011 12:53:44 EST."
	<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com> 
References: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com> 
Message-ID: <201103161834.p2GIYI52006709@theraft.openend.se>

In a message of Wed, 16 Mar 2011 12:53:44 EST, Timothy Baldridge writes:
>Why is there no "Reply-To" in the headers on this list? Whenever, in
>gmail, I hit "reply" it automatically sets the to address to the
>"from" field in the mailing list e-mail. So I end up e-mailing the
>person directly instead of the list. This is highly frustrating.
>
>Timothy
>

http://woozle.org/~neale/papers/reply-to-still-harmful.html

Laura


From victorgarcianet at gmail.com  Wed Mar 16 19:33:11 2011
From: victorgarcianet at gmail.com (Victor Garcia)
Date: Wed, 16 Mar 2011 15:33:11 -0300
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
References: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
Message-ID: <AANLkTinD0nOZUNGc_o4q=HytNCi1tmMr8MUReM_7cCXF@mail.gmail.com>

Timothy Baldridge wrote:
> Why is there no "Reply-To" in the headers on this list? Whenever, in
> gmail, I hit "reply" it automatically sets the to address to the
> "from" field in the mailing list e-mail. So I end up e-mailing the
> person directly instead of the list. This is highly frustrating.

A quick workaround is to enable "Default 'Reply to all'" in GMail's labs.

Victor


From gorla.patricia at gmail.com  Wed Mar 16 21:02:31 2011
From: gorla.patricia at gmail.com (gorla.patricia at gmail.com)
Date: Wed, 16 Mar 2011 20:02:31 +0000
Subject: [pypy-dev] *Rant* No Reply-To?
Message-ID: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry>

I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter.

New weekend project?
------Original Message------
From: Victor Garcia
Sender: pypy-dev-bounces at codespeak.net
To: Timothy Baldridge
Cc: pypy-dev at codespeak.net
Subject: Re: [pypy-dev] *Rant* No Reply-To?
Sent: Mar 16, 2011 14:33

Timothy Baldridge wrote:
> Why is there no "Reply-To" in the headers on this list? Whenever, in
> gmail, I hit "reply" it automatically sets the to address to the
> "from" field in the mailing list e-mail. So I end up e-mailing the
> person directly instead of the list. This is highly frustrating.

A quick workaround is to enable "Default 'Reply to all'" in GMail's labs.

Victor
_______________________________________________
pypy-dev at codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev


Sent on the Sprint? Now Network from my BlackBerry?

From adam.jtm30 at gmail.com  Wed Mar 16 21:06:00 2011
From: adam.jtm30 at gmail.com (Adam Bark)
Date: Wed, 16 Mar 2011 20:06:00 +0000
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry>
References: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry>
Message-ID: <4D811828.90109@gmail.com>

I believe you can just set it for certain email addresses, ie. mailing 
lists.

On 16/03/11 20:02, gorla.patricia at gmail.com wrote:
> I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter.
>
> New weekend project?
> ------Original Message------
> From: Victor Garcia
> Sender: pypy-dev-bounces at codespeak.net
> To: Timothy Baldridge
> Cc: pypy-dev at codespeak.net
> Subject: Re: [pypy-dev] *Rant* No Reply-To?
> Sent: Mar 16, 2011 14:33
>
> Timothy Baldridge wrote:
>> Why is there no "Reply-To" in the headers on this list? Whenever, in
>> gmail, I hit "reply" it automatically sets the to address to the
>> "from" field in the mailing list e-mail. So I end up e-mailing the
>> person directly instead of the list. This is highly frustrating.
> A quick workaround is to enable "Default 'Reply to all'" in GMail's labs.
>
> Victor
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
> Sent on the Sprint? Now Network from my BlackBerry?
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



From william.leslie.ttg at gmail.com  Thu Mar 17 03:18:07 2011
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Thu, 17 Mar 2011 13:18:07 +1100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <AANLkTikqnP0hJBRbLEa1Ka3T8ih_hwaZ=Y-DR-WqWZ-s@mail.gmail.com>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
	<AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
	<AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>
	<AANLkTikqnP0hJBRbLEa1Ka3T8ih_hwaZ=Y-DR-WqWZ-s@mail.gmail.com>
Message-ID: <AANLkTiki_=6Q7jhmDA8ujcBpgz7DJ7OOCFEiFS8mrQbg@mail.gmail.com>

Where did you want this discussion to go, Laura?  It looks like you
wanted to talk about the specific problems that need to be dealt with
while removing the GIL, but it seems to have disintegrated into the
same "concurrency model X is better than concurrency model Y" free for
all that regularly seems to happen on this list.  Regardless of the
API that runtimes written with the translation toolkit may provide,
getting rid of the GIL is a precursor to the implementations of most
of these models.

-- 
William Leslie


From dimaqq at gmail.com  Thu Mar 17 04:51:59 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 16 Mar 2011 20:51:59 -0700
Subject: [pypy-dev] Any Multithreading Primitives?
In-Reply-To: <AANLkTim5eW3EYVuxL5GGYgj4rVxro6bfL+MX8eG6ELXm@mail.gmail.com>
References: <tbaldridge@gmail.com>
	<AANLkTikDOiavxY3bEQcoESo8pqxgyC6NH7DjcKxEvH=c@mail.gmail.com>
	<201103151556.p2FFuNqE017090@theraft.openend.se>
	<AANLkTim5eW3EYVuxL5GGYgj4rVxro6bfL+MX8eG6ELXm@mail.gmail.com>
Message-ID: <AANLkTik3kw5Fh246Qiw6paM2h3v=iWrWGKFG3-N_1VpZ@mail.gmail.com>

how about dict.setdefault or dict.popitem as a quick hack?

unless you are trying to get rid of gil, in which case, sorry I missed
your point

On 15 March 2011 09:04, Timothy Baldridge <tbaldridge at gmail.com> wrote:
>> Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html
>> be of any use to you? ?Note, this documentation isquite old and needs
>> attention, and right now the stackless transform does not work with the JIT.
>> changing things so that it would work doesn't seem to be that much work,
>> though, we just haven't gotten around to it.
>
>
> Not really. What I need are CAS operations so I can do the following:
>
> while(True):
> ? oldval = var
> ? newval = oldval + 1
> ? if cas(oldval, newval, var):
> ? ? ? break
>
> Basically cas takes the value you think var currently has, the value
> you want var to be, and the var. Cas then updates var if and only if
> var's current value == oldvar. But cas is atomic. So basically it
> returns false if the cas failed, and true if it worked. Using cas you
> can build mutexes, shared objects, etc.
>
> http://en.wikipedia.org/wiki/Compare-and-swap
>
> Timothy
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From dimaqq at gmail.com  Thu Mar 17 05:04:01 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 16 Mar 2011 21:04:01 -0700
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <4D811828.90109@gmail.com>
References: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry>
	<4D811828.90109@gmail.com>
Message-ID: <AANLkTimvY3=j7XJgeULJME2Vhg64AzuGgz5Hwjq6TY5-@mail.gmail.com>

+1 for reply-to from me.

the munging flame is from last century! it's a different world now.

alternatively convince google to use list-post header, see when that happens

On 16 March 2011 13:06, Adam Bark <adam.jtm30 at gmail.com> wrote:
> I believe you can just set it for certain email addresses, ie. mailing
> lists.
>
> On 16/03/11 20:02, gorla.patricia at gmail.com wrote:
>> I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter.
>>
>> New weekend project?
>> ------Original Message------
>> From: Victor Garcia
>> Sender: pypy-dev-bounces at codespeak.net
>> To: Timothy Baldridge
>> Cc: pypy-dev at codespeak.net
>> Subject: Re: [pypy-dev] *Rant* No Reply-To?
>> Sent: Mar 16, 2011 14:33
>>
>> Timothy Baldridge wrote:
>>> Why is there no "Reply-To" in the headers on this list? Whenever, in
>>> gmail, I hit "reply" it automatically sets the to address to the
>>> "from" field in the mailing list e-mail. So I end up e-mailing the
>>> person directly instead of the list. This is highly frustrating.
>> A quick workaround is to enable "Default 'Reply to all'" in GMail's labs.
>>
>> Victor
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>>
>> Sent on the Sprint? Now Network from my BlackBerry?
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From lac at openend.se  Thu Mar 17 11:30:26 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 17 Mar 2011 11:30:26 +0100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: Message from William ML Leslie <william.leslie.ttg@gmail.com> 
	of "Thu, 17 Mar 2011 13:18:07 +1100."
	<AANLkTiki_=6Q7jhmDA8ujcBpgz7DJ7OOCFEiFS8mrQbg@mail.gmail.com> 
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
	<AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
	<AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>
	<AANLkTikqnP0hJBRbLEa1Ka3T8ih_hwaZ=Y-DR-WqWZ-s@mail.gmail.com>
	<AANLkTiki_=6Q7jhmDA8ujcBpgz7DJ7OOCFEiFS8mrQbg@mail.gmail.com> 
Message-ID: <201103171030.p2HAUQCT028822@theraft.openend.se>

In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes:
>Where did you want this discussion to go, Laura?  It looks like you
>wanted to talk about the specific problems that need to be dealt with
>while removing the GIL, but it seems to have disintegrated into the
>same "concurrency model X is better than concurrency model Y" free for
>all that regularly seems to happen on this list.  Regardless of the
>API that runtimes written with the translation toolkit may provide,
>getting rid of the GIL is a precursor to the implementations of most
>of these models.
>
>-- 
>William Leslie

I'm at a Sprint at PyCON, as are many of the people I think would be
best at answering these specific questions.   So it is not surprising
that they are not answering them now.  I, myself, am personally interested
in finding out how languages I have never looked at do these things,
because I expect it to influence how one gets rid of the GIL.  I was
hoping to have an insight as to how one could avoid going the route
of reimplementing fine grained locks everywhere, pervasively, all
through the codebase.  But all I am seeing now is more evidence that
this is impossible.

Laura


From santagada at gmail.com  Thu Mar 17 12:50:20 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 17 Mar 2011 08:50:20 -0300
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <201103171030.p2HAUQCT028822@theraft.openend.se>
References: <201103141813.p2EIDY9C027808@theraft.openend.se>
	<AANLkTinfLcXi+m=JdjoAJ5VC+_ihxvgcZvyPx4M370d0@mail.gmail.com>
	<AANLkTinyAnDjMkxv70+J24Ckz3TRq6d8wmvXDS_1YVah@mail.gmail.com>
	<AANLkTikbUDXe+0EOeB7t3SPdmMW_mPewC5F_32JwJmPd@mail.gmail.com>
	<AANLkTimwQgHTQu83ke=FUHaF6b0p2fMK+B7xt910kY-E@mail.gmail.com>
	<01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp>
	<AANLkTik=0uwtqS8vs0BtYUzpEfJH3fzUVfPo3MRwMiFT@mail.gmail.com>
	<AANLkTik3qdcsB2ncFjnW7bsOHgU_zvLTirBiO_bVTdFW@mail.gmail.com>
	<AANLkTi=GWwYBstEaaBPRGiQaA2-M9-OST7WksRXXhqKC@mail.gmail.com>
	<AANLkTinEnbWNUaWf8K=HQ3CfzK3H2CGgg_CU1BZEkpp9@mail.gmail.com>
	<AANLkTikqnP0hJBRbLEa1Ka3T8ih_hwaZ=Y-DR-WqWZ-s@mail.gmail.com>
	<william.leslie.ttg@gmail.com>
	<AANLkTiki_=6Q7jhmDA8ujcBpgz7DJ7OOCFEiFS8mrQbg@mail.gmail.com>
	<201103171030.p2HAUQCT028822@theraft.openend.se>
Message-ID: <AANLkTikC+CqZMqrXYdQNRmUxw-7pG=V-r95ciaHUKwY6@mail.gmail.com>

On Thu, Mar 17, 2011 at 7:30 AM, Laura Creighton <lac at openend.se> wrote:
> In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes:
>>Where did you want this discussion to go, Laura? ?It looks like you
>>wanted to talk about the specific problems that need to be dealt with
>>while removing the GIL, but it seems to have disintegrated into the
>>same "concurrency model X is better than concurrency model Y" free for
>>all that regularly seems to happen on this list. ?Regardless of the
>>API that runtimes written with the translation toolkit may provide,
>>getting rid of the GIL is a precursor to the implementations of most
>>of these models.
>>
>>--
>>William Leslie
>
> I'm at a Sprint at PyCON, as are many of the people I think would be
> best at answering these specific questions. ? So it is not surprising
> that they are not answering them now. ?I, myself, am personally interested
> in finding out how languages I have never looked at do these things,
> because I expect it to influence how one gets rid of the GIL. ?I was
> hoping to have an insight as to how one could avoid going the route
> of reimplementing fine grained locks everywhere, pervasively, all
> through the codebase. ?But all I am seeing now is more evidence that
> this is impossible.
>
> Laura

I think it would be cool to have something to point people to in the
docs, a page describing why pypy has a gil and what would it take to
remove it. I would like to see a clear separation on what steps is
needed to make RPython threadsafe (ie. fixing gc choosing and
implementing a concurrency model) and what would it take to make the
pypy-python interpreter not need a gil (choosing a maybe even
different concurrency model and memory semantics, etc).

-- 
Leonardo Santagada


From anto.cuni at gmail.com  Thu Mar 17 17:09:19 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 17 Mar 2011 17:09:19 +0100
Subject: [pypy-dev] ideas for Google Summer of code
Message-ID: <4D82322F.7070700@gmail.com>

Hi all,

the deadlines for GSoc are approaching, and at some point we should probably
make a blog post about that.

But first, we need to 1) collect ideas for possible tasks and 2) find
potential mentors.

Two ideas that just came to my mind:

- "general work" on speed.pypy.org (we need to define better what we want, of
course)

- improving the jitviewer, maybe integrating it with the profiler (when we'll
have one :-))

- <insert-your-idea-here> :-)

ciao,
Anto


From lac at openend.se  Thu Mar 17 17:34:20 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 17 Mar 2011 17:34:20 +0100
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: Message from Antonio Cuni <anto.cuni@gmail.com> 
	of "Thu, 17 Mar 2011 17:09:19 +0100." <4D82322F.7070700@gmail.com> 
References: <4D82322F.7070700@gmail.com> 
Message-ID: <201103171634.p2HGYKVf029380@theraft.openend.se>

In a message of Thu, 17 Mar 2011 17:09:19 +0100, Antonio Cuni writes:
>Hi all,
>
>the deadlines for GSoc are approaching, and at some point we should proba
>bly
>make a blog post about that.
>
>But first, we need to 1) collect ideas for possible tasks and 2) find
>potential mentors.
>
>Two ideas that just came to my mind:
>
>- "general work" on speed.pypy.org (we need to define better what we want
>, of
>course)
>
>- improving the jitviewer, maybe integrating it with the profiler (when w
>e'll
>have one :-))
>
>- <insert-your-idea-here> :-)
>
>ciao,
>Anto

3.x conversions -- a) write an interpreter
                   b) do the fiddly bits needed to integrate the new
                      interpreter with our codebase
                   c) get the 3.whatever tests to pass

I think this is too much work for one SoC student, but maybe not if it
was set up as 2 projects, one of which stared after the other did.  I
am not sure how SoC is being handles for people who live in the 
Southern hemisphere and who go to classes in June, July, etc.

>_______________________________________________
>pypy-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/pypy-dev


From chef at ghum.de  Thu Mar 17 17:48:43 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Thu, 17 Mar 2011 17:48:43 +0100
Subject: [pypy-dev] phb impression from speed.pypy.org
Message-ID: <AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com>

Hello,

I really love the improvements to the clearity of speed.pypy.org

In the last days I have dropped some times on that page, aaaand...
although I read the description and understand it, what my mind FIRST
processes from the

How has PyPy performance evolved over time?

picture is that "performance got worse"

Yes; right after that I read the small print of "smaller is better".
But still... my eyes tell me "it got worse".

Would it be possible to think of a presentation that allows to GROW
those elements? Having a measurement comparable to transaction per
second; or whatever-stones.

Or just project the 1/x of every measurment ... just the subconcious
thinks "something shrinking is not sexy". There is allways the
experience with "wow, you have grown so much" ... nobody tells their
(shrinking) elders "wow, you have shrunken so much"...

Another small point: The headline is "...over time", but the x-axis
shows "over versions". If I call correctly, the delta-t between
"release of cpython 2.6.2" to "pypy1.3" is different form dt(1.3,1.4)
is different from dt(1.4,trunk)

As much as I remember, the speedup is speeding up ...

Would such changes be possible? Does someone else think they may be positive?

harald
-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare


From bokr at oz.net  Thu Mar 17 07:04:09 2011
From: bokr at oz.net (Bengt Richter)
Date: Wed, 16 Mar 2011 23:04:09 -0700
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
References: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
Message-ID: <4D81A459.6050007@oz.net>

On 03/16/2011 10:53 AM Timothy Baldridge wrote:
> Why is there no "Reply-To" in the headers on this list? Whenever, in
> gmail, I hit "reply" it automatically sets the to address to the
> "from" field in the mailing list e-mail. So I end up e-mailing the
> person directly instead of the list. This is highly frustrating.
>
> Timothy
>
I am using Thunderbird on Linux, and this is a reply-all to your post
received as a plain email (as opposed to using the newsgroup interface
of Thunderbird, which is also available, e.g. via gmane).

There is also a plain reply and a reply-list option. I will save this
and the result of the other options as drafts, and extract the headers,
so you can see what it did. First this email:
_______________________________________________________________________

 From - Wed Mar 16 22:42:08 2011
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
FCC: mailbox://bokr at mail.oz.net/Sent
X-Identity-Key: id1
Message-ID:<4D819F30.1090203 at oz.net>
Date: Wed, 16 Mar 2011 22:42:08 -0700
From: Bengt Richter<bokr at oz.net>
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5
MIME-Version: 1.0
To: Timothy Baldridge<tbaldridge at gmail.com>
CC: pypy-dev at codespeak.net
Subject: Re: [pypy-dev] *Rant* No Reply-To?
References:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
In-Reply-To:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
_____________________________________________________________________________
Next, headers for plain plain reply:
 From - Wed Mar 16 22:46:45 2011
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
FCC: mailbox://bokr at mail.oz.net/Sent
X-Identity-Key: id1
Message-ID:<4D81A045.9080306 at oz.net>
Date: Wed, 16 Mar 2011 22:46:45 -0700
From: Bengt Richter<bokr at oz.net>
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5
MIME-Version: 1.0
To: Timothy Baldridge<tbaldridge at gmail.com>
Subject: Re: [pypy-dev] *Rant* No Reply-To?
References:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
In-Reply-To:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
___________________________________________________________________________
Next, headers resulting from reply-list:
 From - Wed Mar 16 22:49:10 2011
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
FCC: mailbox://bokr at mail.oz.net/Sent
X-Identity-Key: id1
Message-ID:<4D81A0D6.7090102 at oz.net>
Date: Wed, 16 Mar 2011 22:49:10 -0700
From: Bengt Richter<bokr at oz.net>
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5
MIME-Version: 1.0
To: pypy-dev at codespeak.net
Subject: Re: [pypy-dev] *Rant* No Reply-To?
References:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
In-Reply-To:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
___________________________________________________________________________

Does gmail not have similar reply options?

PS. The source of your incoming post as email, so you know what
Thunderbird used to generate the reply options:

###########################################################################

 From - Wed Mar 16 10:58:14 2011
X-Account-Key: account2
X-UIDL: 0C777EF600002BB7
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
X-Eon-Dm: dm34
Return-Path:<pypy-dev-bounces at codespeak.net>
Received: from codespeak.net (88.198.193.90 [88.198.193.90])
	by dm34.mta.everyone.net (EON-INBOUND) with ESMTP id dm34.4d751061.1bddd2b
	for<bokr at oz.net>; Wed, 16 Mar 2011 10:53:51 -0700
Received: from codespeak.net (localhost [127.0.0.1])
	by codespeak.net (Postfix) with ESMTP id 198A7282BD6;
	Wed, 16 Mar 2011 18:53:48 +0100 (CET)
X-Original-To: pypy-dev at codespeak.net
Delivered-To: pypy-dev at codespeak.net
Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com
	[209.85.214.182]) (using TLSv1 with cipher RC4-SHA (128/128 bits))
	(No client certificate requested)
	by codespeak.net (Postfix) with ESMTP id 81F28282B9D
	for<pypy-dev at codespeak.net>; Wed, 16 Mar 2011 18:53:45 +0100 (CET)
Received: by iwn33 with SMTP id 33so3148560iwn.27
	for<pypy-dev at codespeak.net>; Wed, 16 Mar 2011 10:53:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.43.51.65 with SMTP id vh1mr374043icb.435.1300298024183; Wed,
	16 Mar 2011 10:53:44 -0700 (PDT)
Received: by 10.42.223.9 with HTTP; Wed, 16 Mar 2011 10:53:44 -0700 (PDT)
Date: Wed, 16 Mar 2011 12:53:44 -0500
Message-ID:<AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W at mail.gmail.com>
From: Timothy Baldridge<tbaldridge at gmail.com>
To: pypy-dev at codespeak.net
Subject: [pypy-dev] *Rant* No Reply-To?
X-BeenThere: pypy-dev at codespeak.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: development of a Python Implementation mostly developed within Python
	<pypy-dev.codespeak.net>
List-Unsubscribe:<http://codespeak.net/mailman/listinfo/pypy-dev>,
	<mailto:pypy-dev-request at codespeak.net?subject=unsubscribe>
List-Archive:<http://codespeak.net/pipermail/pypy-dev>
List-Post:<mailto:pypy-dev at codespeak.net>
List-Help:<mailto:pypy-dev-request at codespeak.net?subject=help>
List-Subscribe:<http://codespeak.net/mailman/listinfo/pypy-dev>,
	<mailto:pypy-dev-request at codespeak.net?subject=subscribe>
Content-Type: text/plain; charset="windows-1252"
Content-Transfer-Encoding: quoted-printable
Sender: pypy-dev-bounces at codespeak.net
Errors-To: pypy-dev-bounces at codespeak.net

Why is there no "Reply-To" in the headers on this list? Whenever, in
gmail, I hit "reply" it automatically sets the to address to the
"from" field in the mailing list e-mail. So I end up e-mailing the
person directly instead of the list. This is highly frustrating.

Timothy

-- =

=93One of the main causes of the fall of the Roman Empire was
that=96lacking zero=96they had no way to indicate successful termination
of their C programs.=94
(Robert Firth)

###########################################################################

Regards,
Bengt Richter
Munging does not seem necessary? (Thanks for the link, Laura)
<relurk/>



From santagada at gmail.com  Thu Mar 17 18:11:32 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 17 Mar 2011 14:11:32 -0300
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <4D81A459.6050007@oz.net>
References: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
	<4D81A459.6050007@oz.net>
Message-ID: <AANLkTik+bqOxOoF2OLJWJ+dqcAVOi+Q_DmFxrQsxXe5g@mail.gmail.com>

On Thu, Mar 17, 2011 at 3:04 AM, Bengt Richter <bokr at oz.net> wrote:
> Does gmail not have similar reply options?

Nope, also Mail.app doesn't have reply-list also.

> PS. The source of your incoming post as email, so you know what
> Thunderbird used to generate the reply options:

Which doesn't contain the List-to header that wooz talked about.


-- 
Leonardo Santagada


From lac at openend.se  Thu Mar 17 18:22:39 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 17 Mar 2011 18:22:39 +0100
Subject: [pypy-dev] phb impression from speed.pypy.org
In-Reply-To: Message from "Massa, Harald Armin" <chef@ghum.de> of "Thu,
	17 Mar 2011 17:48:43 +0100."
	<AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com> 
References: <AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com> 
Message-ID: <201103171722.p2HHMdIs001438@theraft.openend.se>


Smaller is better when you are dieting.  Or when you are racing.
Given that there is talk that we will measure memory size as well, and
turn into performance.pypy.org then I think that the 'smaller is
better' idea will be well understood.  As a practical matter, making
'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which
you want to be the first to finish.

Laura



From santagada at gmail.com  Thu Mar 17 18:23:59 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 17 Mar 2011 14:23:59 -0300
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <201103171634.p2HGYKVf029380@theraft.openend.se>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
Message-ID: <AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>

My ideas, take the ones you guys like and don't bother if some are too
hard or the pypy team is not interested:

1 - Some pypy compatibility site, like the one brett made for python 3
2 - Rework pypy website to make it pretty and more organized, maybe
migrate or integrate bitbucket wiki with the rest of the docs
3 - make a killer app for pypy: either make mercurial/bzr or django
faster on pypy to get more users
4 - Finish faster ctypes, make a cython/swig backend
5 - stackless on pypy with jit
6 - make an embeding api (for mod_wsgi and uWSGI)
7 - make pypy on .net jit work (or on java)
8 - better mac os or windows support (or strange unix like aix and
[net|open|free]bsd)
9 - Port ZODB or other less complex c extension to pypy/ctypes (pycripto)
10 - ultra fast pickle (nice for everyone specially to incentive zodb
to be ported)
11 - make the 64bit jit better (IIRC it was not as great as it could be)
12 - powerpc jit
13 - better GC

On Thu, Mar 17, 2011 at 1:34 PM, Laura Creighton <lac at openend.se> wrote:
> In a message of Thu, 17 Mar 2011 17:09:19 +0100, Antonio Cuni writes:
>>Hi all,
>>
>>the deadlines for GSoc are approaching, and at some point we should proba
>>bly
>>make a blog post about that.
>>
>>But first, we need to 1) collect ideas for possible tasks and 2) find
>>potential mentors.
>>
>>Two ideas that just came to my mind:
>>
>>- "general work" on speed.pypy.org (we need to define better what we want
>>, of
>>course)
>>
>>- improving the jitviewer, maybe integrating it with the profiler (when w
>>e'll
>>have one :-))
>>
>>- <insert-your-idea-here> :-)
>>
>>ciao,
>>Anto
>
> 3.x conversions -- a) write an interpreter
> ? ? ? ? ? ? ? ? ? b) do the fiddly bits needed to integrate the new
> ? ? ? ? ? ? ? ? ? ? ?interpreter with our codebase
> ? ? ? ? ? ? ? ? ? c) get the 3.whatever tests to pass
>
> I think this is too much work for one SoC student, but maybe not if it
> was set up as 2 projects, one of which stared after the other did. ?I
> am not sure how SoC is being handles for people who live in the
> Southern hemisphere and who go to classes in June, July, etc.
>
>>_______________________________________________
>>pypy-dev at codespeak.net
>>http://codespeak.net/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
Leonardo Santagada


From baiju.m.mail at gmail.com  Thu Mar 17 18:37:51 2011
From: baiju.m.mail at gmail.com (Baiju M)
Date: Thu, 17 Mar 2011 13:37:51 -0400
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
Message-ID: <AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>

On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada <santagada at gmail.com> wrote:
> My ideas, take the ones you guys like and don't bother if some are too
> hard or the pypy team is not interested:
>
> 1 - Some pypy compatibility site, like the one brett made for python 3

I am interested to set up a site for PyPY.  I have already created a similar
site for Python 3:  http://getpython3.net/

If this idea sound good, you can add one DNS "A record" pointing to this IP:
184.106.69.139  A sub-domain like http://compatibility.pypy.org/ would be fine.
Otherwise you can provide me a hosting place also.  BTW, the code is here:
https://github.com/baijum/getpython3  (Flask app)

Regards,
Baiju M


From greg at quora.com  Thu Mar 17 18:38:56 2011
From: greg at quora.com (Greg Price)
Date: Thu, 17 Mar 2011 10:38:56 -0700
Subject: [pypy-dev] *Rant* No Reply-To?
In-Reply-To: <AANLkTik+bqOxOoF2OLJWJ+dqcAVOi+Q_DmFxrQsxXe5g@mail.gmail.com>
References: <AANLkTinz_k90Jk08u2B=-4N5ykDqa6sPwhL=Ev0hB5_W@mail.gmail.com>
	<4D81A459.6050007@oz.net>
	<AANLkTik+bqOxOoF2OLJWJ+dqcAVOi+Q_DmFxrQsxXe5g@mail.gmail.com>
Message-ID: <AANLkTikcfoKNRsueP07eUf5WRi1p+o7dt3mx1RyvSqEW@mail.gmail.com>

On Thu, Mar 17, 2011 at 10:11 AM, Leonardo Santagada
<santagada at gmail.com> wrote:
> On Thu, Mar 17, 2011 at 3:04 AM, Bengt Richter <bokr at oz.net> wrote:
>> Does gmail not have similar reply options?
>
> Nope, also Mail.app doesn't have reply-list also.

Gmail has "Reply to all". Use that. I do every day.

Last I looked over someone's shoulder using Mail.app, it did too. I'd
be surprised to hear of a mail client that did not.

As Bengt demonstrated by his choice to use it, reply-all is most often
what you want instead of reply-list in any case.

Greg


From dsurviver at gmail.com  Thu Mar 17 19:00:58 2011
From: dsurviver at gmail.com (Danilo Freitas)
Date: Thu, 17 Mar 2011 15:00:58 -0300
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
Message-ID: <AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>

Hi, all.

I'm interested in applying for GSoC this year.
I'm talking to Miquel Torres about some stuff in Codespeed, but I
don't know if it could be considered as PyPy project for GSoC.
We're trying to allow Codespeed branch comparing, to check if a
feature branch is getting faster than trunk. So, we'll see if a
feature is really evolving.
This would also affect speed.pypy.org. After that, we shall work on more stuff.
So, could Codespeed improvements be considered as PyPy GSoC projects?

Laura, I'm from Brazil and was a GSoC student in 2009 for Cython. I
had about only 1 month free of college (June~July), but I completed
what I promised without problems caused by college. So, I guess people
from south hemisphere can handle with it, if they dedicate themselves
:)

2011/3/17 Baiju M <baiju.m.mail at gmail.com>:
> On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>> My ideas, take the ones you guys like and don't bother if some are too
>> hard or the pypy team is not interested:
>>
>> 1 - Some pypy compatibility site, like the one brett made for python 3
>
> I am interested to set up a site for PyPY. ?I have already created a similar
> site for Python 3: ?http://getpython3.net/
>
> If this idea sound good, you can add one DNS "A record" pointing to this IP:
> 184.106.69.139 ?A sub-domain like http://compatibility.pypy.org/ would be fine.
> Otherwise you can provide me a hosting place also. ?BTW, the code is here:
> https://github.com/baijum/getpython3 ?(Flask app)
>
> Regards,
> Baiju M
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
- Danilo Freitas


From garyrob at me.com  Thu Mar 17 18:38:36 2011
From: garyrob at me.com (Gary Robinson)
Date: Thu, 17 Mar 2011 13:38:36 -0400
Subject: [pypy-dev] ideas for Google Summer of code
Message-ID: <6393F570-E4D1-44CE-8AC6-CA55234DF287@me.com>

Work on numpy/scipy integration. That will really help pypy be more useful to people in the scientific and A.I. communities..

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From wlavrijsen at lbl.gov  Thu Mar 17 19:33:56 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 17 Mar 2011 11:33:56 -0700 (PDT)
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <4D82322F.7070700@gmail.com>
References: <4D82322F.7070700@gmail.com>
Message-ID: <alpine.LNX.2.00.1103171132080.10215@pc3-wlav.lbl.gov>

Hi Anto,

> - <insert-your-idea-here> :-)

the Reflex work has at one time before been proposed as a GSoC. Now that
a prototype is in place, there are several minor tasks that can be done,
which can lead to bigger/more research tasks as desired/appropriate.

How does this work anyway, do you need to come with your own student?

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net


From dmalcolm at redhat.com  Thu Mar 17 19:20:20 2011
From: dmalcolm at redhat.com (David Malcolm)
Date: Thu, 17 Mar 2011 14:20:20 -0400
Subject: [pypy-dev] Documentation sprint at PyCon
Message-ID: <1300386020.16889.16.camel@surprise>

Laura, me, and others sprinted on documentation cleanups at PyCon, using
  https://bitbucket.org/dmalcolm/pypy-dmalcolm
as a development branch.

Laura and Armin just merged the changes from that repo

Significant changes are:
  - the Sphinxification of the docs
  - the renaming of the sources from .txt to .rst, to play better with
text editors with smarts for rst
  - organizing the documentation, to try to stress the high-important
up-to-date material, whilst moving the more out-of-date materials to a
clearly-marked annex (see cleanup.rst).

Running:
   make html
within pypy/docs should now generate the documentation part of the site.

You can also use:
   make linkcheck
to check links; we believe we've fixed all internal links, but some
external links are still broken.

There's a nasty hack for pointing at sources which you can see in:
  https://bitbucket.org/pypy/pypy/changeset/c6f7ecf2dc01
but fixing this properly looks like we would need to write a new sphinx
plugin; as it is, it works, but is ugly at the rst level.

Going forward, our idea is that http://docs.pypy.org ought to point at
the sphinx-generated html, and the "development" link on pypy.org should
point to docs.pypy.org (moving another thing off of codespeak).

The folks at readthedocs.org have offered hosting space, and can keep
  http://pypy.readthedocs.org
up-to-date with a nightly build of the documentation.

They have some docs on setting up a CNAME for this:
http://read-the-docs.readthedocs.org/en/latest/alternate_domains.html#cname-support
so that docs.pypy.org can be set up to point there.

Currently that site has an out-of-date build of my bitbucket branch I
mentioned at the top of this mail (and there've been a lot of changes
since then).

Someone will need to sync up with the readthedocs folks to make their
site points at the correct bitbucket repo, and to ensure that it's
getting regularly rebuilt

Hope this makes sense and is helpful.
Dave



From drsalists at gmail.com  Thu Mar 17 21:25:38 2011
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 17 Mar 2011 13:25:38 -0700
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
Message-ID: <AANLkTikQ-AfaPk7vjtXytoJu-TV-Or18hEPw7ReHq_K5@mail.gmail.com>

On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada <santagada at gmail.com>wrote:

> 7 - make pypy on .net jit work (or on java)
>

This reminds me: it might be good to make the JVM PyPy be able to call
native Java code - on a typical JRE, and on Android.  Last I heard, on
Android people were using a CPython port, which reportedly requires a stub
for the various Android library calls that're written in
what-is-essentially-Java.  What I was told is that the Ruby port to Android
gives much better API access, because they started with a Ruby that runs on
a JVM.

Also, on the matter of performance testing, coming up with a bunch of tests
that run unmodified on a large number of Python interpreters might be
included - though perhaps that goes without saying.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110317/4bf01b11/attachment.html>

From santagada at gmail.com  Thu Mar 17 21:32:48 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 17 Mar 2011 17:32:48 -0300
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTikQ-AfaPk7vjtXytoJu-TV-Or18hEPw7ReHq_K5@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTikQ-AfaPk7vjtXytoJu-TV-Or18hEPw7ReHq_K5@mail.gmail.com>
Message-ID: <AANLkTi=zV-uvFU=8zacjOK5wZ=4k9BGQrJvA+3zmTB_s@mail.gmail.com>

On Thu, Mar 17, 2011 at 5:25 PM, Dan Stromberg <drsalists at gmail.com> wrote:
>
> On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada <santagada at gmail.com>
> wrote:
>>
>> 7 - make pypy on .net jit work (or on java)
>
> This reminds me: it might be good to make the JVM PyPy be able to call
> native Java code - on a typical JRE, and on Android.? Last I heard, on
> Android people were using a CPython port, which reportedly requires a stub
> for the various Android library calls that're written in
> what-is-essentially-Java.? What I was told is that the Ruby port to Android
> gives much better API access, because they started with a Ruby that runs on
> a JVM.

I think dalvik and jme don't support compiling during runs, so no jit,
then I think making jython (which also needs compilation at runtime)
work on dalvik seems like a better idea.


> Also, on the matter of performance testing, coming up with a bunch of tests
> that run unmodified on a large number of Python interpreters might be
> included - though perhaps that goes without saying.

This is part of the gsoc to implement a speed.python.org (so it is a
great idea, but it is not a pypy gsoc).


-- 
Leonardo Santagada


From chris.mulligan at gmail.com  Thu Mar 17 21:34:56 2011
From: chris.mulligan at gmail.com (Chris Mulligan)
Date: Thu, 17 Mar 2011 16:34:56 -0400
Subject: [pypy-dev] GC Tuning script values
Message-ID: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com>

Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very easy!

Here's a link to my tuning values http://a.libpa.st/hMGxQ. In short with a dual cure i5 w/ 3MB L3 the following were fastest:
1 CPU: 2M
2 CPU: 768K
3 CPU: 768K
4 CPU: 512K

Congrats on the hard work, everyone. I'm very excited to start testing some of our own code with pypy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110317/980795ee/attachment.html>

From anto.cuni at gmail.com  Thu Mar 17 22:26:55 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 17 Mar 2011 22:26:55 +0100
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTikQ-AfaPk7vjtXytoJu-TV-Or18hEPw7ReHq_K5@mail.gmail.com>
References: <anto.cuni@gmail.com>
	<4D82322F.7070700@gmail.com>	<201103171634.p2HGYKVf029380@theraft.openend.se>	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTikQ-AfaPk7vjtXytoJu-TV-Or18hEPw7ReHq_K5@mail.gmail.com>
Message-ID: <4D827C9F.9090506@gmail.com>

On 17/03/11 21:25, Dan Stromberg wrote:
> 
> On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada <santagada at gmail.com
> <mailto:santagada at gmail.com>> wrote:
> 
>     7 - make pypy on .net jit work (or on java)

this is probably a too large task for GSoc. For one, before working on the JIT
it is necessary to make normal translation working, because right now ootype
backends are broken.  IIRC, it's not too hard because it consists of porting
virtualrefs to ootype, which should be simple. But it is possible that there
are other issues after that, since ootype translations have not run since a
long time (more than 1 year, I think).

About the JIT: JIT on .NET is not going to be any fast.  I did it for my
thesis when the JIT was more .NET friendly (no virtualrefs) and results were
interesting as a research project, but not good enough to be used in production.

The JIT for pypy-jvm is an open topic: the JVM has the potential to do a much
better job than the CLI, but we cannot know until we really try.

Having a working pypy-jvm-jit is a lot of work, though. It consists of:

1) make pypy-jvm (without jit) working again

2) design and implement a way to use Java objects at RPython level: this is
needed to write the backend

3) port the JIT to ootype again. Should not be too hard, but there are going
to be issues, because the codewriter has been heavily refactored since last
year, when the JIT on ootype worked well

4) write the backend

All together, it's probably huge for GSoc. But e.g 1+2 could fit it; we
already had a GSoc on this topic few years ago, but it didn't work well.

> 
> This reminds me: it might be good to make the JVM PyPy be able to call native
> Java code - on a typical JRE, and on Android.  Last

yes, that's another interesting topic. It requires both points 1 and 2 above,
though. Once we have that, it should not be too hard, as it has already been
implemented for .NET and could use the same techniques (and probably reuse
also most of the code).

ciao,
Anto


From anto.cuni at gmail.com  Thu Mar 17 22:30:05 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 17 Mar 2011 22:30:05 +0100
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <alpine.LNX.2.00.1103171132080.10215@pc3-wlav.lbl.gov>
References: <4D82322F.7070700@gmail.com>
	<alpine.LNX.2.00.1103171132080.10215@pc3-wlav.lbl.gov>
Message-ID: <4D827D5D.9040703@gmail.com>

Hi Wim,

On 17/03/11 19:33, wlavrijsen at lbl.gov wrote:
> Hi Anto,
> 
>> - <insert-your-idea-here> :-)
> 
> the Reflex work has at one time before been proposed as a GSoC. Now that
> a prototype is in place, there are several minor tasks that can be done,
> which can lead to bigger/more research tasks as desired/appropriate.

good idea!

> How does this work anyway, do you need to come with your own student?

not necessarily. At this stage, the goal is to collect ideas which are
reasonable, then publish it and see which students are interested in them.
However, nothing stops you to suggest a student of course (especially if you
already know him and you are confident that can do the job well).

ciao,
Anto


From fijall at gmail.com  Thu Mar 17 22:32:56 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 17 Mar 2011 15:32:56 -0600
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
	<AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>
Message-ID: <AANLkTi=Jie1Agh50Ynr2UzSb9bd_Jis64H3qU51YpqEi@mail.gmail.com>

On Thu, Mar 17, 2011 at 12:00 PM, Danilo Freitas <dsurviver at gmail.com> wrote:
> Hi, all.
>
> I'm interested in applying for GSoC this year.
> I'm talking to Miquel Torres about some stuff in Codespeed, but I
> don't know if it could be considered as PyPy project for GSoC.
> We're trying to allow Codespeed branch comparing, to check if a
> feature branch is getting faster than trunk. So, we'll see if a
> feature is really evolving.
> This would also affect speed.pypy.org. After that, we shall work on more stuff.
> So, could Codespeed improvements be considered as PyPy GSoC projects?
>
> Laura, I'm from Brazil and was a GSoC student in 2009 for Cython. I
> had about only 1 month free of college (June~July), but I completed
> what I promised without problems caused by college. So, I guess people
> from south hemisphere can handle with it, if they dedicate themselves
> :)

Hi.

That would definitely be considered PyPy project. One ideas that I
have in mind is to create speed.python.org - a place where a whole lot
of different implementations can be run. This requires improvements to
both our benchmark infrastructure and codespeed itself.

>
> 2011/3/17 Baiju M <baiju.m.mail at gmail.com>:
>> On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>>> My ideas, take the ones you guys like and don't bother if some are too
>>> hard or the pypy team is not interested:
>>>
>>> 1 - Some pypy compatibility site, like the one brett made for python 3
>>
>> I am interested to set up a site for PyPY. ?I have already created a similar
>> site for Python 3: ?http://getpython3.net/
>>
>> If this idea sound good, you can add one DNS "A record" pointing to this IP:
>> 184.106.69.139 ?A sub-domain like http://compatibility.pypy.org/ would be fine.
>> Otherwise you can provide me a hosting place also. ?BTW, the code is here:
>> https://github.com/baijum/getpython3 ?(Flask app)
>>
>> Regards,
>> Baiju M
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> - Danilo Freitas
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From jacob at openend.se  Thu Mar 17 23:30:15 2011
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Thu, 17 Mar 2011 23:30:15 +0100
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <4D827D5D.9040703@gmail.com>
References: <4D82322F.7070700@gmail.com>
	<alpine.LNX.2.00.1103171132080.10215@pc3-wlav.lbl.gov>
	<4D827D5D.9040703@gmail.com>
Message-ID: <201103172330.15939.jacob@openend.se>

torsdag 17 mars 2011 22.30.05 skrev  Antonio Cuni:
> Hi Wim,
> 
> On 17/03/11 19:33, wlavrijsen at lbl.gov wrote:
> > Hi Anto,
> > 
> >> - <insert-your-idea-here> :-)
> > 
> > the Reflex work has at one time before been proposed as a GSoC. Now that
> > a prototype is in place, there are several minor tasks that can be done,
> > which can lead to bigger/more research tasks as desired/appropriate.
> 
> good idea!
> 
> > How does this work anyway, do you need to come with your own student?
> 
> not necessarily. At this stage, the goal is to collect ideas which are
> reasonable, then publish it and see which students are interested in them.
> However, nothing stops you to suggest a student of course (especially if
> you already know him and you are confident that can do the job well).

We should also note that mentors are the really scarce resource. If yoiu are 
willing to mentor a student, the chances of this being a GSoC project 
increases dramatically.

Jacob Hall?n


From wlavrijsen at lbl.gov  Thu Mar 17 23:28:11 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 17 Mar 2011 15:28:11 -0700 (PDT)
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <201103172330.15939.jacob@openend.se>
References: <4D82322F.7070700@gmail.com>
	<alpine.LNX.2.00.1103171132080.10215@pc3-wlav.lbl.gov>
	<4D827D5D.9040703@gmail.com> <201103172330.15939.jacob@openend.se>
Message-ID: <alpine.LNX.2.00.1103171527270.10215@pc3-wlav.lbl.gov>

Hi Jacob,

> If yoiu are willing to mentor a student

most definitely.

> the chances of this being a GSoC project increases dramatically.

Great!

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net


From lac at openend.se  Fri Mar 18 01:10:30 2011
From: lac at openend.se (Laura Creighton)
Date: Fri, 18 Mar 2011 01:10:30 +0100
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: Message from Maciej Fijalkowski <fijall@gmail.com> of "Thu,
	17 Mar 2011 15:32:56 CST."
	<AANLkTi=Jie1Agh50Ynr2UzSb9bd_Jis64H3qU51YpqEi@mail.gmail.com> 
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
	<AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>
	<AANLkTi=Jie1Agh50Ynr2UzSb9bd_Jis64H3qU51YpqEi@mail.gmail.com> 
Message-ID: <201103180010.p2I0AVLY006544@theraft.openend.se>

I'd like speed.python.org to become performance.python.org so we can
measure memoryt consumption too.

Laura


From hyarion at iinet.net.au  Fri Mar 18 01:44:41 2011
From: hyarion at iinet.net.au (hyarion at iinet.net.au)
Date: Fri, 18 Mar 2011 08:44:41 +0800
Subject: [pypy-dev] Thinking about the GIL
Message-ID: <41662.1300409081@iinet.net.au>


On Thu Mar 17 21:30 , Laura Creighton  sent:

>In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes:
>>Where did you want this discussion to go, Laura?  It looks like you
>>wanted to talk about the specific problems that need to be dealt with
>>while removing the GIL, but it seems to have disintegrated into the
>>same "concurrency model X is better than concurrency model Y" free for
>>all that regularly seems to happen on this list.  Regardless of the
>>API that runtimes written with the translation toolkit may provide,
>>getting rid of the GIL is a precursor to the implementations of most
>>of these models.
>>
>>-- 
>>William Leslie
>
>I'm at a Sprint at PyCON, as are many of the people I think would be
>best at answering these specific questions.   So it is not surprising
>that they are not answering them now.  I, myself, am personally interested
>in finding out how languages I have never looked at do these things,
>because I expect it to influence how one gets rid of the GIL.  I was
>hoping to have an insight as to how one could avoid going the route
>of reimplementing fine grained locks everywhere, pervasively, all
>through the codebase.  But all I am seeing now is more evidence that
>this is impossible.


This may be a dumb question, but has anyone considered software transactional memory 
for this sort of thing? My experience comes from the implementation side of STM in a 
very different language to (R)Python, but it seems like this might be a reasonable 
fir for STM's strengths. The GIL gives you a serial execution order of bytecodes, 
with no particular guarantees about which order (because if it matters there should 
be application level concurrency control). Wrapping each bytecode in an STM 
transaction would give you an as-if-serial execution order, again with no guarantees 
about which order. You get transaction overheads instead of lock/unlock overheads, 
but some STM systems can be quite efficient for short transactions that rarely 
conflict.

There'd be a lot of problems to solve, of course. But has anyone already considered 
this and figured out that it's impossible or impractical?

-- Ben


From william.leslie.ttg at gmail.com  Fri Mar 18 02:01:00 2011
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Fri, 18 Mar 2011 12:01:00 +1100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <41662.1300409081@iinet.net.au>
References: <41662.1300409081@iinet.net.au>
Message-ID: <AANLkTimnyDu7K=11h8RYS9NaUbxW8nCSFriUV-kiFkhm@mail.gmail.com>

On 18 March 2011 11:44, hyarion at iinet.net.au <hyarion at iinet.net.au> wrote:
> There'd be a lot of problems to solve, of course. But has anyone already considered
> this and figured out that it's impossible or impractical?

My own fork of pypy will eventually use STM as an implementation
technique, more or less, so we will get to find out first hand if it
is practical, at least within an environment with a fine-grain effect
system.

-- 
William Leslie


From asenchi at gmail.com  Fri Mar 18 04:35:25 2011
From: asenchi at gmail.com (Curt Micol)
Date: Thu, 17 Mar 2011 23:35:25 -0400
Subject: [pypy-dev] gcbench run
Message-ID: <AANLkTi=s22GtR0ikz3yFuJ5uBeULtSxwB3jmK2yTHnsr@mail.gmail.com>

Hello,

In his post today, Bob Ippolito said the devs were interested in
results from the gcbench run. Here's a gist with all of the data. Hope
this is helpful.

https://gist.github.com/874910

Thanks for the great work on PyPy,

-- 
# Curt Micol


From arigo at tunes.org  Fri Mar 18 04:43:34 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 17 Mar 2011 23:43:34 -0400
Subject: [pypy-dev] GC Tuning script values
In-Reply-To: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com>
References: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com>
Message-ID: <AANLkTinEcdGFgYw_fb2A8gj6Zd9WTFMeE=xVqq3GMmLK@mail.gmail.com>

Hi Chris,

On Thu, Mar 17, 2011 at 4:34 PM, Chris Mulligan
<chris.mulligan at gmail.com> wrote:
> Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very
> easy!

Thanks for the feedback.  The results make it clear that we should
somehow tune the number according to the load of the machine --
picking up the right number for the load can easily make a 20% speed
difference (at least on Mac OS X, but I strongly suspect the same is
true on other platforms).

Ideally, it should dynamically adapt its nursery size in order to
minimize the cache misses.  If anyone has a suggestion on how to
implement that, preferably in a non-OS-specific way (e.g. by reading
some x86 CPU counters), I'd welcome it :-)


A bient?t,

Armin.


From bob at redivi.com  Fri Mar 18 05:52:27 2011
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 18 Mar 2011 00:52:27 -0400
Subject: [pypy-dev] GC Tuning script values
In-Reply-To: <AANLkTinEcdGFgYw_fb2A8gj6Zd9WTFMeE=xVqq3GMmLK@mail.gmail.com>
References: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com>
	<AANLkTinEcdGFgYw_fb2A8gj6Zd9WTFMeE=xVqq3GMmLK@mail.gmail.com>
Message-ID: <AANLkTikrz0tSLrVE34dYv7k0s=zY5YbWKoqYoWzLJnk6@mail.gmail.com>

It also seems that about L3//4 is a pretty good number on both his
machine and mine. Not optimal in the single core case but works well
as load increases. Of course, single or four core machines could be
wildly different.

On Thursday, March 17, 2011, Armin Rigo <arigo at tunes.org> wrote:
> Hi Chris,
>
> On Thu, Mar 17, 2011 at 4:34 PM, Chris Mulligan
> <chris.mulligan at gmail.com> wrote:
>> Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very
>> easy!
>
> Thanks for the feedback. ?The results make it clear that we should
> somehow tune the number according to the load of the machine --
> picking up the right number for the load can easily make a 20% speed
> difference (at least on Mac OS X, but I strongly suspect the same is
> true on other platforms).
>
> Ideally, it should dynamically adapt its nursery size in order to
> minimize the cache misses. ?If anyone has a suggestion on how to
> implement that, preferably in a non-OS-specific way (e.g. by reading
> some x86 CPU counters), I'd welcome it :-)
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From greg at quora.com  Fri Mar 18 05:53:28 2011
From: greg at quora.com (Greg Price)
Date: Thu, 17 Mar 2011 21:53:28 -0700
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <201103180010.p2I0AVLY006544@theraft.openend.se>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
	<AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>
	<fijall@gmail.com>
	<AANLkTi=Jie1Agh50Ynr2UzSb9bd_Jis64H3qU51YpqEi@mail.gmail.com>
	<201103180010.p2I0AVLY006544@theraft.openend.se>
Message-ID: <AANLkTi=hAA_BUAYQ1F_QjtkLmEqohPWeYOmo2R3R3aDa@mail.gmail.com>

Hi Laura,

On Mar 17, 2011 5:10 PM, "Laura Creighton" <lac at openend.se> wrote:
>
> I'd like speed.python.org to become performance.python.org so we can
> measure memoryt consumption too.

We should definitely measure memory consumption too, but "speed.python.org"
has a nice rhythm to it - few syllables, quick to say - such that I think it
might make sense to keep the name even so. To me, "speed" also suggests that
it *is* fast, more so than "performance" suggests that the performance
actually is good, as opposed to neutrally measuring it, whatever it is. I
always thought "speed.pypy.org" quite a bold name, saying "we are fast"
right there in the domain so nobody could miss the point. Maybe that last is
just me, though.

Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110317/16c2bf53/attachment.html>

From tobami at googlemail.com  Fri Mar 18 08:13:15 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Fri, 18 Mar 2011 08:13:15 +0100
Subject: [pypy-dev] phb impression from speed.pypy.org
In-Reply-To: <201103171722.p2HHMdIs001438@theraft.openend.se>
References: <chef@ghum.de>
	<AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com>
	<201103171722.p2HHMdIs001438@theraft.openend.se>
Message-ID: <AANLkTi=3O3_7=hSV2b=72z4=rXHH-5JerGC-anp7B_ZR@mail.gmail.com>

I understand what you say, and it is certainly possible to turn
benchmarks into "bigger is better". As Laura wrote though, it is only
natural to measure time, because that is what you actually what to do:
reduce the time it takes to complete some task.

Other projects do the same. See for example jQuery announcing a newer,
faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/

(scroll down for lots of performance plots)

*However*, it seems that for the last release they have done what you
propose: http://blog.jquery.com/2011/01/31/jquery-15-released/

They have changed the units from "seconds" to number of "iterations per second".

In any case that is something for the PyPy developers to decide.

Cheers,
Miquel


2011/3/17 Laura Creighton <lac at openend.se>:
>
> Smaller is better when you are dieting. ?Or when you are racing.
> Given that there is talk that we will measure memory size as well, and
> turn into performance.pypy.org then I think that the 'smaller is
> better' idea will be well understood. ?As a practical matter, making
> 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which
> you want to be the first to finish.
>
> Laura
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From tobami at googlemail.com  Fri Mar 18 08:23:51 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Fri, 18 Mar 2011 08:23:51 +0100
Subject: [pypy-dev] phb impression from speed.pypy.org
In-Reply-To: <AANLkTi=3O3_7=hSV2b=72z4=rXHH-5JerGC-anp7B_ZR@mail.gmail.com>
References: <chef@ghum.de>
	<AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com>
	<201103171722.p2HHMdIs001438@theraft.openend.se>
	<AANLkTi=3O3_7=hSV2b=72z4=rXHH-5JerGC-anp7B_ZR@mail.gmail.com>
Message-ID: <AANLkTi=1QC4k+X3xcLNCyZ==vhR1s_8Xee3mg6Zp8Zs_@mail.gmail.com>

Oh! it just downed on me that there is a very easy way to do a "bigger
is better" plot without changing any of the current views of plots.

The "How has PyPy performance evolved over time?" plot, with
represents the geometric average of all normalized benchmarks. You may
have noticed the number inside parenthesis, which it is precisely the
inverse.

That one plot can thus be easily inverted, to be able to show "times
faster than", without changing the rest of the site.



2011/3/18 Miquel Torres <tobami at googlemail.com>:
> I understand what you say, and it is certainly possible to turn
> benchmarks into "bigger is better". As Laura wrote though, it is only
> natural to measure time, because that is what you actually what to do:
> reduce the time it takes to complete some task.
>
> Other projects do the same. See for example jQuery announcing a newer,
> faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/
>
> (scroll down for lots of performance plots)
>
> *However*, it seems that for the last release they have done what you
> propose: http://blog.jquery.com/2011/01/31/jquery-15-released/
>
> They have changed the units from "seconds" to number of "iterations per second".
>
> In any case that is something for the PyPy developers to decide.
>
> Cheers,
> Miquel
>
>
> 2011/3/17 Laura Creighton <lac at openend.se>:
>>
>> Smaller is better when you are dieting. ?Or when you are racing.
>> Given that there is talk that we will measure memory size as well, and
>> turn into performance.pypy.org then I think that the 'smaller is
>> better' idea will be well understood. ?As a practical matter, making
>> 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which
>> you want to be the first to finish.
>>
>> Laura
>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>


From holger at merlinux.eu  Fri Mar 18 10:25:19 2011
From: holger at merlinux.eu (holger krekel)
Date: Fri, 18 Mar 2011 09:25:19 +0000
Subject: [pypy-dev] ideas for Google Summer of code
In-Reply-To: <AANLkTi=hAA_BUAYQ1F_QjtkLmEqohPWeYOmo2R3R3aDa@mail.gmail.com>
References: <anto.cuni@gmail.com> <4D82322F.7070700@gmail.com>
	<201103171634.p2HGYKVf029380@theraft.openend.se>
	<AANLkTikZTV1piYFwLqAibrvAT1hUS_=fq2bQZ285x-X_@mail.gmail.com>
	<AANLkTimM8pqMwnvvUs67UMdkdNyzTcctmdCKq=zeFB+2@mail.gmail.com>
	<AANLkTimWHuk_gD1uQ3eLfRV9sVBQq=0959MmXVGBZhqT@mail.gmail.com>
	<fijall@gmail.com>
	<AANLkTi=Jie1Agh50Ynr2UzSb9bd_Jis64H3qU51YpqEi@mail.gmail.com>
	<201103180010.p2I0AVLY006544@theraft.openend.se>
	<AANLkTi=hAA_BUAYQ1F_QjtkLmEqohPWeYOmo2R3R3aDa@mail.gmail.com>
Message-ID: <20110318092519.GA16231@merlinux.eu>

Hi Greg, all,

On Thu, Mar 17, 2011 at 21:53 -0700, Greg Price wrote:
> Hi Laura,
> 
> On Mar 17, 2011 5:10 PM, "Laura Creighton" <lac at openend.se> wrote:
> >
> > I'd like speed.python.org to become performance.python.org so we can
> > measure memoryt consumption too.
> 
> We should definitely measure memory consumption too, but "speed.python.org"
> has a nice rhythm to it - few syllables, quick to say - such that I think it
> might make sense to keep the name even so. To me, "speed" also suggests that
> it *is* fast, more so than "performance" suggests that the performance
> actually is good, as opposed to neutrally measuring it, whatever it is. I
> always thought "speed.pypy.org" quite a bold name, saying "we are fast"
> right there in the domain so nobody could miss the point. Maybe that last is
> just me, though.

Nope, that was actually the idea when we came up with the domain name and
i think it makes sense to keep it such :)

But maybe we can postpone further discussion until when we have
nice memory measurement.
cheers,

holger


From holger at merlinux.eu  Fri Mar 18 14:32:19 2011
From: holger at merlinux.eu (holger krekel)
Date: Fri, 18 Mar 2011 13:32:19 +0000
Subject: [pypy-dev] Documentation sprint at PyCon
In-Reply-To: <1300386020.16889.16.camel@surprise>
References: <1300386020.16889.16.camel@surprise>
Message-ID: <20110318133219.GB16231@merlinux.eu>

Hey David, Laura, Armin,

nice work!

I just removed the documentation tests now that sphinx performs link-checking.

I noticed that the :config:`OPTNAME` was not ported to sphinx. It generated
a link to the appropriate config option.  Maybe some one can port
it or we can just manually replace the few places where it is used.
The docutils role-registering code is in pypy/config/makerestdoc.py
and is a bit involved - maybe Carl Friedrich can tell if this is
still neccessary.

If someone takes care for setting up readthedocs or some other
place to generate the docs i can point the doc.pypy.org domain
name to it. 

cheers,
holger

On Thu, Mar 17, 2011 at 14:20 -0400, David Malcolm wrote:
> Laura, me, and others sprinted on documentation cleanups at PyCon, using
>   https://bitbucket.org/dmalcolm/pypy-dmalcolm
> as a development branch.
> 
> Laura and Armin just merged the changes from that repo
> 
> Significant changes are:
>   - the Sphinxification of the docs
>   - the renaming of the sources from .txt to .rst, to play better with
> text editors with smarts for rst
>   - organizing the documentation, to try to stress the high-important
> up-to-date material, whilst moving the more out-of-date materials to a
> clearly-marked annex (see cleanup.rst).
> 
> Running:
>    make html
> within pypy/docs should now generate the documentation part of the site.
> 
> You can also use:
>    make linkcheck
> to check links; we believe we've fixed all internal links, but some
> external links are still broken.
> 
> There's a nasty hack for pointing at sources which you can see in:
>   https://bitbucket.org/pypy/pypy/changeset/c6f7ecf2dc01
> but fixing this properly looks like we would need to write a new sphinx
> plugin; as it is, it works, but is ugly at the rst level.
> 
> Going forward, our idea is that http://docs.pypy.org ought to point at
> the sphinx-generated html, and the "development" link on pypy.org should
> point to docs.pypy.org (moving another thing off of codespeak).
> 
> The folks at readthedocs.org have offered hosting space, and can keep
>   http://pypy.readthedocs.org
> up-to-date with a nightly build of the documentation.
> 
> They have some docs on setting up a CNAME for this:
> http://read-the-docs.readthedocs.org/en/latest/alternate_domains.html#cname-support
> so that docs.pypy.org can be set up to point there.
> 
> Currently that site has an out-of-date build of my bitbucket branch I
> mentioned at the top of this mail (and there've been a lot of changes
> since then).
> 
> Someone will need to sync up with the readthedocs folks to make their
> site points at the correct bitbucket repo, and to ensure that it's
> getting regularly rebuilt
> 
> Hope this makes sense and is helpful.
> Dave
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From lac at openend.se  Fri Mar 18 17:48:46 2011
From: lac at openend.se (Laura Creighton)
Date: Fri, 18 Mar 2011 17:48:46 +0100
Subject: [pypy-dev] European sprints?
In-Reply-To: Message from Laura Creighton <lac@openend.se> of "Wed,
	16 Feb 2011 13:08:07 +0100."
	<201102161208.p1GC875O027115@theraft.openend.se> 
References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com>
	<201102151054.p1FAsZWb018955@theraft.openend.se>
	<4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com>
	<4D5BAE92.2080306@changemaker.nu>
	<201102161208.p1GC875O027115@theraft.openend.se> 
Message-ID: <201103181648.p2IGmktc030478@theraft.openend.se>

In a message of Wed, 16 Feb 2011 13:08:07 +0100, Laura Creighton writes:
>In a message of Wed, 16 Feb 2011 12:01:38 +0100, Bea During writes:
>>Here is a suggestion of places and dates based on Lauras, Carl Friedrich
> 
>>and Antos
>>input:
>>
>>- Gothenburg sprint: 25th of April to 1st of May
>>- Europython sprint/Florence: 25th of June to 26th of June (EP2011 
>>official sprint dates)
>>- D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of 
>>the funded PyJIT project which ends end August)
>>
>>What do you think about these dates - would they work?
>>
>>Cheers
>>
>>Bea

There is now a reason to like the week before Florence EuroPython June 20-26
-- so June 12 - 20 or so -- EuroDjangoCon is June 6-10 in Amsterdam
 http://djangocon.eu/

So people fron out of Europe could plan a Python summer, if they were
coming to EuroDjangocon anyway.

Also, I have just got mail from the original author of this thread.  He'd
like to book tickets to come to G?teborg and sprint with us.  Can we
confirm the dates?

Laura



From anto.cuni at gmail.com  Sat Mar 19 10:07:21 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sat, 19 Mar 2011 10:07:21 +0100
Subject: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect
 my optimizeopt's decision	to emit int_sub(iX, -x) when x < 0
In-Reply-To: <20110319034511.8510A282BAA@codespeak.net>
References: <20110319034511.8510A282BAA@codespeak.net>
Message-ID: <4D847249.8000401@gmail.com>

Hi Daniel,

On 19/03/11 04:45, ademan wrote:
> Author: Daniel Roberts <Ademan555 at gmail.com>
> Branch: fold_intadd
> Changeset: r42802:386510d0fb45
> Date: 2011-03-18 20:41 -0700
> http://bitbucket.org/pypy/pypy/changeset/386510d0fb45/
> 
> Log:	Changed test to reflect my optimizeopt's decision to emit
> 	int_sub(iX, -x) when x < 0

what happens when you are on 32 bit and see int_add(i0, -2147483648)?

ciao,
Anto


From dimaqq at gmail.com  Sat Mar 19 17:19:52 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Sat, 19 Mar 2011 09:19:52 -0700
Subject: [pypy-dev] phb impression from speed.pypy.org
In-Reply-To: <AANLkTi=1QC4k+X3xcLNCyZ==vhR1s_8Xee3mg6Zp8Zs_@mail.gmail.com>
References: <chef@ghum.de>
	<AANLkTikwfF1wQpLmjSbv1whRqV+-LKd6s4quO8E-duTa@mail.gmail.com>
	<201103171722.p2HHMdIs001438@theraft.openend.se>
	<AANLkTi=3O3_7=hSV2b=72z4=rXHH-5JerGC-anp7B_ZR@mail.gmail.com>
	<AANLkTi=1QC4k+X3xcLNCyZ==vhR1s_8Xee3mg6Zp8Zs_@mail.gmail.com>
Message-ID: <AANLkTi=qzveHi=jUwN1ovpUCWdNywj8Qy58Rn-rxBBTE@mail.gmail.com>

I'm all for smaller is better in the evolution plot

Seems more rational

except you can't call it "performance" in the header above the plot then.

On 18 March 2011 00:23, Miquel Torres <tobami at googlemail.com> wrote:
> Oh! it just downed on me that there is a very easy way to do a "bigger
> is better" plot without changing any of the current views of plots.
>
> The "How has PyPy performance evolved over time?" plot, with
> represents the geometric average of all normalized benchmarks. You may
> have noticed the number inside parenthesis, which it is precisely the
> inverse.
>
> That one plot can thus be easily inverted, to be able to show "times
> faster than", without changing the rest of the site.
>
>
>
> 2011/3/18 Miquel Torres <tobami at googlemail.com>:
>> I understand what you say, and it is certainly possible to turn
>> benchmarks into "bigger is better". As Laura wrote though, it is only
>> natural to measure time, because that is what you actually what to do:
>> reduce the time it takes to complete some task.
>>
>> Other projects do the same. See for example jQuery announcing a newer,
>> faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/
>>
>> (scroll down for lots of performance plots)
>>
>> *However*, it seems that for the last release they have done what you
>> propose: http://blog.jquery.com/2011/01/31/jquery-15-released/
>>
>> They have changed the units from "seconds" to number of "iterations per second".
>>
>> In any case that is something for the PyPy developers to decide.
>>
>> Cheers,
>> Miquel
>>
>>
>> 2011/3/17 Laura Creighton <lac at openend.se>:
>>>
>>> Smaller is better when you are dieting. ?Or when you are racing.
>>> Given that there is talk that we will measure memory size as well, and
>>> turn into performance.pypy.org then I think that the 'smaller is
>>> better' idea will be well understood. ?As a practical matter, making
>>> 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which
>>> you want to be the first to finish.
>>>
>>> Laura
>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Sun Mar 20 14:14:46 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 20 Mar 2011 14:14:46 +0100
Subject: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect
 my optimizeopt's decision to emit int_sub(iX, -x) when x < 0
In-Reply-To: <4D847249.8000401@gmail.com>
References: <20110319034511.8510A282BAA@codespeak.net>
	<4D847249.8000401@gmail.com>
Message-ID: <AANLkTikmeJqdayReQOAw5Zmby_cQJQPFwUuDxSQ8v4Nt@mail.gmail.com>

Hi Anto, hi Ademan,

On Sat, Mar 19, 2011 at 10:07 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> what happens when you are on 32 bit and see int_add(i0, -2147483648)?

It probably works, because -(-2147483648) == -2147483648, and
int_add(i0, -2147483648) and int_sub(i0, -2147483648) do the same
thing.  However I have no clue why this replacement is giving you
anything.  If anything at all I would say that replacing int_add(i0,
-128) with int_sub(i0, 128) is a (very marginally) bad idea on x86
because the constant -128 fits in 8 bits but the constant 128 doesn't.
 Well, I suppose that "it makes it easier for me, the human, to
understand it" is a valid reason after all.


A bient?t,

Armin.


From brownan at gmail.com  Tue Mar 22 20:44:36 2011
From: brownan at gmail.com (Andrew Brown)
Date: Tue, 22 Mar 2011 15:44:36 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
Message-ID: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>

Hello PyPy team,

So I became interested in the translation toolchain plus JIT compiler
generator, and in attempt to learn more about it, I set out to write a very
simple interpreter for the language BF (brainf***). Simple enough, 8
opcodes, each with no arguments, and most have a one line implementation.
Also plenty of examples out there to run, like a mandelbrot generator =)

So I wrote up an interpreter for it in RPython. This worked great until I
tried to enable the JIT option, at which point it would produce incorrect
results. Strange, I thought I may have been using the hints incorrectly, I
couldn't find too many details on exactly what the red and green vars should
be.

Here's the strange part though: I finally fixed it by changing an
implementation detail that shouldn't have changed semantics at all.

My implementation creates an instance of a Tape object which has two
attributes: a list of integers representing the state of the machine, and a
single integer identifying the current active cell on the tape.

The implementation of each opcode was a method of this class, and the state
of the program (what I passed as the "red" variable) was the instance of
this class.

After I manually factored the functionality of the class directly into the
main dispatch loop and got rid of the class entirely, the JIT compiler
started producing correct results.

Can anyone help me figure out why my first attempt didn't work? Do red
variables that are in class instances need to be handled different somehow?

Here's the initial version that runs incorrectly when translated with JIT:
https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py
Here's the modified version that seems to work just fine:
https://bitbucket.org/brownan/bf-interpreter/src/8095853278e9/targetbf.py

In particular, note the elimination of the Tape object in the second
version, and the differences in the mainloop function as well as the
differences in the "red" variables.

I've also included a few example BF programs if someone wants to try it out.
The hanoi example crashes almost immediately with the first version
translated with JIT.

By the way, I've been translating it with the latest version of PyPy off of
bitbucket. (latest as of a few weeks ago, that is)

Thanks and great work on this project!
-Andrew Brown
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110322/11394033/attachment.html>

From tbaldridge at gmail.com  Tue Mar 22 21:41:19 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Tue, 22 Mar 2011 15:41:19 -0500
Subject: [pypy-dev] Implementing CAS
Message-ID: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>

After my last discussion about getting some multi-threading
primitives, I took a look at how locks are implemented in PyPy. PyPy
currently uses OS level mutexes...which is okay for most uses, but in
my case, I need super fast spinlocks, and for that I need a CAS
operation. What I'm looking for is a bit of direction on how to go
about implementing this function (shown in C syntax):

int cas(*expected, *new, **location)

This function looks at the contents of the data pointed at by
**location. If the contents == *expected, the contents are replaced
with *new and the function returns true. Otherwise the function
returns false.

There are several issues I see with implementing this in python:

1) **location must be pointer to a object pointer, is this even
possible in PyPy?
2) I'd rather not call an FFI C function for this, when this entire
function could be inlined with just a few lines of assembly (CAS is a
single instruction on x86). So could this be created as a C function
that is simply inserted into the generated C code, so that GCC could
inline it at will?

I'm just brainstorming here. No matter how you dice it, CAS is pretty
much a prerequisite to any sort of multi-threaded programming these
days. That is unless you want to spend thousands of clock cycles in
context switches.

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From benjamin at python.org  Tue Mar 22 22:14:09 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 22 Mar 2011 16:14:09 -0500
Subject: [pypy-dev] Implementing CAS
In-Reply-To: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>
References: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>
Message-ID: <AANLkTikaMPt0hP7V3ck6f_EphpvXAUjumacJAaR8ym_y@mail.gmail.com>

2011/3/22 Timothy Baldridge <tbaldridge at gmail.com>:
> After my last discussion about getting some multi-threading
> primitives, I took a look at how locks are implemented in PyPy. PyPy
> currently uses OS level mutexes...which is okay for most uses, but in
> my case, I need super fast spinlocks, and for that I need a CAS
> operation. What I'm looking for is a bit of direction on how to go
> about implementing this function (shown in C syntax):
>
> int cas(*expected, *new, **location)

I suggest you create a new low-level operation. Then the C backend can
translate it into asm.


-- 
Regards,
Benjamin


From fijall at gmail.com  Tue Mar 22 22:18:48 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 22 Mar 2011 15:18:48 -0600
Subject: [pypy-dev] Implementing CAS
In-Reply-To: <AANLkTikaMPt0hP7V3ck6f_EphpvXAUjumacJAaR8ym_y@mail.gmail.com>
References: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>
	<AANLkTikaMPt0hP7V3ck6f_EphpvXAUjumacJAaR8ym_y@mail.gmail.com>
Message-ID: <AANLkTin975A0ofMVzH=vqY2r6sD6WrS2pEg4uhtX4dqF@mail.gmail.com>

On Tue, Mar 22, 2011 at 3:14 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2011/3/22 Timothy Baldridge <tbaldridge at gmail.com>:
>> After my last discussion about getting some multi-threading
>> primitives, I took a look at how locks are implemented in PyPy. PyPy
>> currently uses OS level mutexes...which is okay for most uses, but in
>> my case, I need super fast spinlocks, and for that I need a CAS
>> operation. What I'm looking for is a bit of direction on how to go
>> about implementing this function (shown in C syntax):
>>
>> int cas(*expected, *new, **location)
>
> I suggest you create a new low-level operation. Then the C backend can
> translate it into asm.
>

Which requires touching multiple places, but it's relatively easy.
Start with writing a test and then add it to
pypy/rpython/lltypesystem/lloperation.py

>
> --
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Tue Mar 22 22:54:33 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 22 Mar 2011 22:54:33 +0100
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
Message-ID: <AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>

Hi Andrew,

On Tue, Mar 22, 2011 at 8:44 PM, Andrew Brown <brownan at gmail.com> wrote:
> https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py

can_enter_jit() is not correct.  For it to work, it must be called
just before jit_merge_point().  It's wrong that there are two
intermediate instructions here: "pc+=1" and the "pc < len(program)"
condition.

As a first attempt, you should just not call can_enter_jit() at all.
Nowadays, if can_enter_jit is never called, it's done automatically
for you; moreover, a misplaced can_enter_jit can give nonsensical
results, as opposed to many other hints, which cannot give a result
worst than terribly bad performance (like the greens/reds variable
separation --- which seems correct in your example).


A bient?t,

Armin.


From lac at openend.se  Wed Mar 23 02:17:21 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 23 Mar 2011 02:17:21 +0100
Subject: [pypy-dev] status of the graphviz server on codespeak?
Message-ID: <201103230117.p2N1HLfS016404@theraft.openend.se>

getting-started-dev says:

Download and install `Dot Graphviz`_ (optional if you have an internet
    connection: the flowgraph viewer then connects to
    codespeak.net and lets it convert the flowgraph by a graphviz server).

What's going to happen to that when codespeak goes away?

Laura


From fijall at gmail.com  Wed Mar 23 02:36:02 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 22 Mar 2011 19:36:02 -0600
Subject: [pypy-dev] status of the graphviz server on codespeak?
In-Reply-To: <201103230117.p2N1HLfS016404@theraft.openend.se>
References: <201103230117.p2N1HLfS016404@theraft.openend.se>
Message-ID: <AANLkTinwc1ioj-fu5UR+CzzRVf-TV7n=FvfzbPn85QKt@mail.gmail.com>

On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton <lac at openend.se> wrote:
> getting-started-dev says:
>
> Download and install `Dot Graphviz`_ (optional if you have an internet
> ? ?connection: the flowgraph viewer then connects to
> ? ?codespeak.net and lets it convert the flowgraph by a graphviz server).
>
> What's going to happen to that when codespeak goes away?

This one is defunct for ages now I think.

>
> Laura
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From pypy at pocketnix.org  Wed Mar 23 05:34:53 2011
From: pypy at pocketnix.org (pypy at pocketnix.org)
Date: Wed, 23 Mar 2011 04:34:53 +0000
Subject: [pypy-dev] Implementing CAS
In-Reply-To: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>
References: <AANLkTi=_vGw0jc8SDHXG230iEgYshr+MaoEvhqcNVGoK@mail.gmail.com>
Message-ID: <20110323043453.GA14527@pocketnix.org>

On Tue, Mar 22, 2011 at 03:41:19PM -0500, Timothy Baldridge wrote:
> After my last discussion about getting some multi-threading
> primitives, I took a look at how locks are implemented in PyPy. PyPy
> currently uses OS level mutexes...which is okay for most uses, but in
> my case, I need super fast spinlocks, and for that I need a CAS
> operation. What I'm looking for is a bit of direction on how to go
> about implementing this function (shown in C syntax):
> 
> int cas(*expected, *new, **location)
> 
> This function looks at the contents of the data pointed at by
> **location. If the contents == *expected, the contents are replaced
> with *new and the function returns true. Otherwise the function
> returns false.
> 
> There are several issues I see with implementing this in python:
> 
> 1) **location must be pointer to a object pointer, is this even
> possible in PyPy?
> 2) I'd rather not call an FFI C function for this, when this entire
> function could be inlined with just a few lines of assembly (CAS is a
> single instruction on x86). So could this be created as a C function
> that is simply inserted into the generated C code, so that GCC could
> inline it at will?
> 
> I'm just brainstorming here. No matter how you dice it, CAS is pretty
> much a prerequisite to any sort of multi-threaded programming these
> days. That is unless you want to spend thousands of clock cycles in
> context switches.
> 
> Timothy
> 
> -- 
> ?One of the main causes of the fall of the Roman Empire was
> that?lacking zero?they had no way to indicate successful termination
> of their C programs.?
> (Robert Firth)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

Dont know if this would be helpful at all but i have been wondering if 
RCU's would be useful at all 
http://en.wikipedia.org/wiki/Read-copy-update (userspace libary at 
http://lttng.org/urcu ) from what i ahve read up on them i thought 
they would be a nice match to python and pypy


From anto.cuni at gmail.com  Wed Mar 23 09:02:25 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 23 Mar 2011 09:02:25 +0100
Subject: [pypy-dev] status of the graphviz server on codespeak?
In-Reply-To: <AANLkTinwc1ioj-fu5UR+CzzRVf-TV7n=FvfzbPn85QKt@mail.gmail.com>
References: <201103230117.p2N1HLfS016404@theraft.openend.se>
	<AANLkTinwc1ioj-fu5UR+CzzRVf-TV7n=FvfzbPn85QKt@mail.gmail.com>
Message-ID: <4D89A911.3050004@gmail.com>

On 23/03/11 02:36, Maciej Fijalkowski wrote:
> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton <lac at openend.se> wrote:
>> getting-started-dev says:
>>
>> Download and install `Dot Graphviz`_ (optional if you have an internet
>>    connection: the flowgraph viewer then connects to
>>    codespeak.net and lets it convert the flowgraph by a graphviz server).
>>
>> What's going to happen to that when codespeak goes away?
> 
> This one is defunct for ages now I think.

no, I think it's still up:
http://codespeak.net/pypy/convertdot.cgi

we can either migrate it to some other machine (tannit?) or discontinue the
service. Nowadays, it's only useful for us developers, and (I think) we all
have graphviz installed locally.

ciao,
Anto


From lac at openend.se  Wed Mar 23 09:36:40 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 23 Mar 2011 09:36:40 +0100
Subject: [pypy-dev] status of the graphviz server on codespeak?
In-Reply-To: Message from Antonio Cuni <anto.cuni@gmail.com> 
	of "Wed, 23 Mar 2011 09:02:25 +0100." <4D89A911.3050004@gmail.com> 
References: <201103230117.p2N1HLfS016404@theraft.openend.se>
	<AANLkTinwc1ioj-fu5UR+CzzRVf-TV7n=FvfzbPn85QKt@mail.gmail.com>
	<4D89A911.3050004@gmail.com> 
Message-ID: <201103230836.p2N8aeg6023861@theraft.openend.se>

In a message of Wed, 23 Mar 2011 09:02:25 +0100, Antonio Cuni writes:
>On 23/03/11 02:36, Maciej Fijalkowski wrote:
>> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton <lac at openend.se> wrote
>:
>>> getting-started-dev says:
>>>
>>> Download and install `Dot Graphviz`_ (optional if you have an internet
>>>    connection: the flowgraph viewer then connects to
>>>    codespeak.net and lets it convert the flowgraph by a graphviz serve
>r).
>>>
>>> What's going to happen to that when codespeak goes away?
>> 
>> This one is defunct for ages now I think.
>
>no, I think it's still up:
>http://codespeak.net/pypy/convertdot.cgi
>
>we can either migrate it to some other machine (tannit?) or discontinue t
>he
>service. Nowadays, it's only useful for us developers, and (I think) we a
>ll
>have graphviz installed locally.

I think discontinuation is the way to go.  Running on tannit is a bad
idea, since it would impact our benchmarks, but running someplace else
is a possibility.  I don't think it's worth it.

Laura

>
>ciao,
>Anto


From fijall at gmail.com  Wed Mar 23 15:16:40 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 23 Mar 2011 08:16:40 -0600
Subject: [pypy-dev] status of the graphviz server on codespeak?
In-Reply-To: <201103230836.p2N8aeg6023861@theraft.openend.se>
References: <201103230117.p2N1HLfS016404@theraft.openend.se>
	<AANLkTinwc1ioj-fu5UR+CzzRVf-TV7n=FvfzbPn85QKt@mail.gmail.com>
	<anto.cuni@gmail.com> <4D89A911.3050004@gmail.com>
	<201103230836.p2N8aeg6023861@theraft.openend.se>
Message-ID: <AANLkTik7dpQQB51gu+ss73BukaLxXOj6WjhXx3PmFrJ_@mail.gmail.com>

On Wed, Mar 23, 2011 at 2:36 AM, Laura Creighton <lac at openend.se> wrote:
> In a message of Wed, 23 Mar 2011 09:02:25 +0100, Antonio Cuni writes:
>>On 23/03/11 02:36, Maciej Fijalkowski wrote:
>>> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton <lac at openend.se> wrote
>>:
>>>> getting-started-dev says:
>>>>
>>>> Download and install `Dot Graphviz`_ (optional if you have an internet
>>>> ? ?connection: the flowgraph viewer then connects to
>>>> ? ?codespeak.net and lets it convert the flowgraph by a graphviz serve
>>r).
>>>>
>>>> What's going to happen to that when codespeak goes away?
>>>
>>> This one is defunct for ages now I think.
>>
>>no, I think it's still up:
>>http://codespeak.net/pypy/convertdot.cgi
>>
>>we can either migrate it to some other machine (tannit?) or discontinue t
>>he
>>service. Nowadays, it's only useful for us developers, and (I think) we a
>>ll
>>have graphviz installed locally.
>
> I think discontinuation is the way to go. ?Running on tannit is a bad
> idea, since it would impact our benchmarks, but running someplace else
> is a possibility. ?I don't think it's worth it.

I assumed it doesn't work because it never worked for me :) Crash on
dot also crashed later on trying to run on codespeak. I would say just
remove

>
> Laura
>
>>
>>ciao,
>>Anto
>


From akg2136 at columbia.edu  Wed Mar 23 16:48:40 2011
From: akg2136 at columbia.edu (Alexander Golec)
Date: Wed, 23 Mar 2011 11:48:40 -0400
Subject: [pypy-dev] When translating pypy,
	how do I notify it of library locations?
Message-ID: <E97982F1-DF23-45FD-BCAF-102B986F00D8@columbia.edu>

I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers. 

I've tried setting the following options :

--cflags=COMPILERFLAGS
--ldflags=LINKERFLAGS

but to no avail... 

Alex



From anto.cuni at gmail.com  Wed Mar 23 17:09:32 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 23 Mar 2011 17:09:32 +0100
Subject: [pypy-dev] When translating pypy,
 how do I notify it of library locations?
In-Reply-To: <E97982F1-DF23-45FD-BCAF-102B986F00D8@columbia.edu>
References: <E97982F1-DF23-45FD-BCAF-102B986F00D8@columbia.edu>
Message-ID: <4D8A1B3C.1040008@gmail.com>

Hi Alexander,

On 23/03/11 16:48, Alexander Golec wrote:
> I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers. 

which libraries are you talking about? Pure python modules/packages (i.e., the
ones contained in lib-python and lib_pypy) or shared libraries?

ciao,
Anto


From lac at openend.se  Wed Mar 23 19:01:09 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 23 Mar 2011 19:01:09 +0100
Subject: [pypy-dev] way to refer to our online doc
Message-ID: <201103231801.p2NI19Y4012616@theraft.openend.se>

over on bitbucket I am looking at

https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/

and I assume that the 04*44 is just some revision number.  How do
we want to refer to the docs in general now that we have moved?

Laura



From amauryfa at gmail.com  Wed Mar 23 19:12:57 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 23 Mar 2011 19:12:57 +0100
Subject: [pypy-dev] way to refer to our online doc
In-Reply-To: <201103231801.p2NI19Y4012616@theraft.openend.se>
References: <201103231801.p2NI19Y4012616@theraft.openend.se>
Message-ID: <AANLkTikkY6g9AQiCYXw0eQKkrnXfO_xR6RyYpDibgwR7@mail.gmail.com>

Hi,
2011/3/23 Laura Creighton <lac at openend.se>:
> over on bitbucket I am looking at
>
> https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/
>
> and I assume that the 04*44 is just some revision number. ?How do
> we want to refer to the docs in general now that we have moved?

I just replaced the rev number with "default":
https://bitbucket.org/pypy/pypy/src/default/pypy/doc/

Another branch name should work equally.

-- 
Amaury Forgeot d'Arc


From lac at openend.se  Wed Mar 23 19:18:58 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 23 Mar 2011 19:18:58 +0100
Subject: [pypy-dev] way to refer to our online doc
In-Reply-To: Message from "Amaury Forgeot d'Arc" <amauryfa@gmail.com> of "Wed,
	23 Mar 2011 19:12:57 +0100."
	<AANLkTikkY6g9AQiCYXw0eQKkrnXfO_xR6RyYpDibgwR7@mail.gmail.com> 
References: <201103231801.p2NI19Y4012616@theraft.openend.se>
	<AANLkTikkY6g9AQiCYXw0eQKkrnXfO_xR6RyYpDibgwR7@mail.gmail.com> 
Message-ID: <201103231818.p2NIIwC7014407@theraft.openend.se>

In a message of Wed, 23 Mar 2011 19:12:57 +0100, "Amaury Forgeot d'Arc" writes:
>Hi,
>2011/3/23 Laura Creighton <lac at openend.se>:
>> over on bitbucket I am looking at
>>
>> https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/
>>
>> and I assume that the 04*44 is just some revision number. =A0How do
>> we want to refer to the docs in general now that we have moved?
>
>I just replaced the rev number with "default":
>https://bitbucket.org/pypy/pypy/src/default/pypy/doc/
>
>Another branch name should work equally.
>
>Amaury Forgeot d'Arc

I guess the question is, do we want to refer to 'default' or to 'tip' ?

Laura



From amauryfa at gmail.com  Wed Mar 23 19:23:24 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 23 Mar 2011 19:23:24 +0100
Subject: [pypy-dev] way to refer to our online doc
In-Reply-To: <201103231818.p2NIIwC7014407@theraft.openend.se>
References: <201103231801.p2NI19Y4012616@theraft.openend.se>
	<amauryfa@gmail.com>
	<AANLkTikkY6g9AQiCYXw0eQKkrnXfO_xR6RyYpDibgwR7@mail.gmail.com>
	<201103231818.p2NIIwC7014407@theraft.openend.se>
Message-ID: <AANLkTikU4bpBCpLUx07YuQgKunjvcicKqosYJYaUY9mn@mail.gmail.com>

2011/3/23 Laura Creighton <lac at openend.se>:
> I guess the question is, do we want to refer to 'default' or to 'tip' ?

"tip" refers to the last committed change, and may belongs to any
development branch

-- 
Amaury Forgeot d'Arc


From brownan at gmail.com  Wed Mar 23 20:27:40 2011
From: brownan at gmail.com (Andrew Brown)
Date: Wed, 23 Mar 2011 15:27:40 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
Message-ID: <AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>

On Tue, Mar 22, 2011 at 5:54 PM, Armin Rigo <arigo at tunes.org> wrote:

> can_enter_jit() is not correct.  For it to work, it must be called
> just before jit_merge_point().  It's wrong that there are two
> intermediate instructions here: "pc+=1" and the "pc < len(program)"
> condition.
>

Okay, I think I understand. I'm still learning how all this stuff works.
Regardless...

>
> As a first attempt, you should just not call can_enter_jit() at all.
> Nowadays, if can_enter_jit is never called, it's done automatically
> for you;


I did not know this. Good to know!

I've removed the can_enter_jit() call from the two versions of my
interpreter. However, the version that didn't work before still does not run
correctly. It seems like I'm still left with the same problem as before.

This works (version without Tape class and with can_enter_jit call removed)
https://bitbucket.org/brownan/bf-interpreter/src/6c6c80397554/targetbf.py
Incidentally, I think it may run slightly slower now, but I'm not sure.

This version still does not (version *with* Tape class, and with
can_enter_jit removed)
https://bitbucket.org/brownan/bf-interpreter/src/1d16c3eed7e2/targetbf.py

Any other ideas? I'm still at a loss. Thanks for taking a look!
-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110323/74980c11/attachment.html>

From fijall at gmail.com  Wed Mar 23 20:43:33 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 23 Mar 2011 13:43:33 -0600
Subject: [pypy-dev] When translating pypy,
	how do I notify it of library locations?
In-Reply-To: <E97982F1-DF23-45FD-BCAF-102B986F00D8@columbia.edu>
References: <E97982F1-DF23-45FD-BCAF-102B986F00D8@columbia.edu>
Message-ID: <AANLkTimZAbMLOENb3jj350bki32Ju=iZRVLx9teLMwAh@mail.gmail.com>

On Wed, Mar 23, 2011 at 9:48 AM, Alexander Golec <akg2136 at columbia.edu> wrote:
> I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers.
>
> I've tried setting the following options :
>
> --cflags=COMPILERFLAGS
> --ldflags=LINKERFLAGS
>
> but to no avail...

Can you explain exactly what didn't work? I think we honor those options.

>
> Alex
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From alex.perry at pamurray.com  Thu Mar 24 01:50:01 2011
From: alex.perry at pamurray.com (Alex Perry)
Date: Wed, 23 Mar 2011 17:50:01 -0700
Subject: [pypy-dev] Is the built-in re.py supposed to translate?
In-Reply-To: <AANLkTinNGW4tV+huAqL3dRUrEZSY5e5n7Xr2mAThyQ0k@mail.gmail.com>
References: <AANLkTinNGW4tV+huAqL3dRUrEZSY5e5n7Xr2mAThyQ0k@mail.gmail.com>
Message-ID: <AANLkTikkPjSr0p_Ppmz=TV2AupDYYh6pAR-MgW4vNbTv@mail.gmail.com>

$ ./pypy-c translate.py target.py
[...]
?AnnotatorError: annotation of <pypy.annotation.dictdef.DictValue
object at 0x0000000004cd7a98> degenerated to SomeObject()
?Simple call of incompatible family:
? ? ? (KeyError getting at the binding!)

?Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232

? ? ? ? cachekey = (type(key[0]),) + key
?==> ? ? p = _cache.get(cachekey)
? ? ? ? if p is not None:

?Previous annotation:
? (none)
?Processing block:
?block at 9 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
?in (re:229)_compile__star_2
?containing the following operations:
? ? ? ?v0 = getitem(key_0, (0))
? ? ? ?v1 = type(v0)
? ? ? ?v2 = newtuple(v1)
? ? ? ?v3 = add(v2, key_0)
? ? ? ?v4 = simple_call((method get), v3)
? ? ? ?v5 = is_(v4, (None))
? ? ? ?v6 = is_true(v5)
?--end--

$ cat target.py
#! /usr/bin/python2.6

def entry_point(argv):
?import re
?s = re.sub(r'[^a-fA-F\d]', '', str(argv[1]))
?print s.decode('hex')

def target(*args):
?return entry_point, None

if __name__ == '__main__':
?import sys
?entry_point(sys.argv)


From alex.gaynor at gmail.com  Thu Mar 24 01:51:24 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 23 Mar 2011 20:51:24 -0400
Subject: [pypy-dev] Is the built-in re.py supposed to translate?
In-Reply-To: <AANLkTikkPjSr0p_Ppmz=TV2AupDYYh6pAR-MgW4vNbTv@mail.gmail.com>
References: <AANLkTinNGW4tV+huAqL3dRUrEZSY5e5n7Xr2mAThyQ0k@mail.gmail.com>
	<AANLkTikkPjSr0p_Ppmz=TV2AupDYYh6pAR-MgW4vNbTv@mail.gmail.com>
Message-ID: <AANLkTik5YRhcwBkm2eY8169D06hnggyU23OHyqALz-6M@mail.gmail.com>

On Wed, Mar 23, 2011 at 8:50 PM, Alex Perry <alex.perry at pamurray.com> wrote:

> $ ./pypy-c translate.py target.py
> [...]
>  AnnotatorError: annotation of <pypy.annotation.dictdef.DictValue
> object at 0x0000000004cd7a98> degenerated to SomeObject()
>  Simple call of incompatible family:
>       (KeyError getting at the binding!)
>
>  Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232
>
>         cachekey = (type(key[0]),) + key
>  ==>     p = _cache.get(cachekey)
>         if p is not None:
>
>  Previous annotation:
>   (none)
>  Processing block:
>  block at 9 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
>  in (re:229)_compile__star_2
>  containing the following operations:
>        v0 = getitem(key_0, (0))
>        v1 = type(v0)
>        v2 = newtuple(v1)
>        v3 = add(v2, key_0)
>        v4 = simple_call((method get), v3)
>        v5 = is_(v4, (None))
>        v6 = is_true(v5)
>  --end--
>
> $ cat target.py
> #! /usr/bin/python2.6
>
> def entry_point(argv):
>  import re
>  s = re.sub(r'[^a-fA-F\d]', '', str(argv[1]))
>  print s.decode('hex')
>
> def target(*args):
>  return entry_point, None
>
> if __name__ == '__main__':
>  import sys
>  entry_point(sys.argv)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Nope, basically nothing in the stdlib is RPython.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110323/5c77f1ee/attachment.html>

From alex.gaynor at gmail.com  Thu Mar 24 02:06:59 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 23 Mar 2011 21:06:59 -0400
Subject: [pypy-dev] Is the built-in re.py supposed to translate?
In-Reply-To: <AANLkTinP=H13QVRTGtX90Vk5K1eiABr+W_nb4=_DZKJL@mail.gmail.com>
References: <AANLkTinNGW4tV+huAqL3dRUrEZSY5e5n7Xr2mAThyQ0k@mail.gmail.com>
	<AANLkTikkPjSr0p_Ppmz=TV2AupDYYh6pAR-MgW4vNbTv@mail.gmail.com>
	<AANLkTik5YRhcwBkm2eY8169D06hnggyU23OHyqALz-6M@mail.gmail.com>
	<AANLkTinP=H13QVRTGtX90Vk5K1eiABr+W_nb4=_DZKJL@mail.gmail.com>
Message-ID: <AANLkTikVeGRWkn6tX-XP=CDLxGJuPOkuyMocLEsF2qP8@mail.gmail.com>

On Wed, Mar 23, 2011 at 9:01 PM, Alex Perry <alex.perry at pamurray.com> wrote:

> I'm confused then.  I thought all imports were supposed to go through
> the type emulation before they hit rpython and the nitty gritty of
> translation.  Do I have to get my target.py to load the type emulator
> and then import re.py on top of it?
>
> On Wed, Mar 23, 2011 at 5:51 PM, Alex Gaynor <alex.gaynor at gmail.com>
> wrote:
> >
> >
> > On Wed, Mar 23, 2011 at 8:50 PM, Alex Perry <alex.perry at pamurray.com>
> wrote:
> >>
> >> $ ./pypy-c translate.py target.py
> >> [...]
> >>  AnnotatorError: annotation of <pypy.annotation.dictdef.DictValue
> >> object at 0x0000000004cd7a98> degenerated to SomeObject()
> >>  Simple call of incompatible family:
> >>       (KeyError getting at the binding!)
> >>
> >>  Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232
> >>
> >>         cachekey = (type(key[0]),) + key
> >>  ==>     p = _cache.get(cachekey)
> >>         if p is not None:
> >>
> >>  Previous annotation:
> >>   (none)
> >>  Processing block:
> >>  block at 9 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
> >>  in (re:229)_compile__star_2
> >>  containing the following operations:
> >>        v0 = getitem(key_0, (0))
> >>        v1 = type(v0)
> >>        v2 = newtuple(v1)
> >>        v3 = add(v2, key_0)
> >>        v4 = simple_call((method get), v3)
> >>        v5 = is_(v4, (None))
> >>        v6 = is_true(v5)
> >>  --end--
> >>
> >> $ cat target.py
> >> #! /usr/bin/python2.6
> >>
> >> def entry_point(argv):
> >>  import re
> >>  s = re.sub(r'[^a-fA-F\d]', '', str(argv[1]))
> >>  print s.decode('hex')
> >>
> >> def target(*args):
> >>  return entry_point, None
> >>
> >> if __name__ == '__main__':
> >>  import sys
> >>  entry_point(sys.argv)
> >> _______________________________________________
> >> pypy-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/pypy-dev
> >
> > Nope, basically nothing in the stdlib is RPython.
> >
> > Alex
> >
> > --
> > "I disapprove of what you say, but I will defend to the death your right
> to
> > say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> > "The people's good is the highest law." -- Cicero
> >
> >
>

I'm not sure I follow, what is the type emulator?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110323/a624fc22/attachment.html>

From arigo at tunes.org  Thu Mar 24 17:11:18 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 24 Mar 2011 17:11:18 +0100
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
Message-ID: <AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>

Hi Andrew,

On Wed, Mar 23, 2011 at 8:27 PM, Andrew Brown <brownan at gmail.com> wrote:
> However, the version that didn't work before still does not run
> correctly. It seems like I'm still left with the same problem as before.

Ah.  Looking more closely, it turns out to be a bug in the
optimization step of the JIT which just never showed up so far :-/

Working on it, by cleaning up optimizeopt/heap.py...


Armin


From arigo at tunes.org  Thu Mar 24 17:56:29 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 24 Mar 2011 17:56:29 +0100
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
Message-ID: <AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>

Re-hi,

On Thu, Mar 24, 2011 at 5:11 PM, Armin Rigo <arigo at tunes.org> wrote:
> Working on it, by cleaning up optimizeopt/heap.py...

Done, at least as far as fixing the bug is concerned.  Now your
original version (with can_enter_jit removed) works.

Armin


From alex.perry at pamurray.com  Thu Mar 24 22:14:56 2011
From: alex.perry at pamurray.com (Alex Perry)
Date: Thu, 24 Mar 2011 14:14:56 -0700
Subject: [pypy-dev] Ctypes module written in RPython
Message-ID: <AANLkTi=EL=f2ZnU+aNWNRUyyzVD6ztM+2qrf2b-EtOOY@mail.gmail.com>

I can't find it in the docs, but it has been alluded to in the past.
How far is the project from being able to compile a rpython module?
I'd expect that to emit a wrapper that imports ctypes and declares
calls into a shared library, and a directory of C code which
implements the internals and can compile into the expected shared
library.


From fijall at gmail.com  Fri Mar 25 01:25:40 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 24 Mar 2011 18:25:40 -0600
Subject: [pypy-dev] Ctypes module written in RPython
In-Reply-To: <AANLkTi=EL=f2ZnU+aNWNRUyyzVD6ztM+2qrf2b-EtOOY@mail.gmail.com>
References: <AANLkTi=EL=f2ZnU+aNWNRUyyzVD6ztM+2qrf2b-EtOOY@mail.gmail.com>
Message-ID: <AANLkTimEnH7hc77PjfJqch8Kq6dKAPh+A9TLeWUROzTH@mail.gmail.com>

On Thu, Mar 24, 2011 at 3:14 PM, Alex Perry <alex.perry at pamurray.com> wrote:
> I can't find it in the docs, but it has been alluded to in the past.
> How far is the project from being able to compile a rpython module?
> I'd expect that to emit a wrapper that imports ctypes and declares
> calls into a shared library, and a directory of C code which
> implements the internals and can compile into the expected shared
> library.

Hi.

I'm a bit confused what you're trying to achieve. re compiler is not
RPython, however the sre module (which runs the regular expression
is). Regular expressions are kind of fast and they can be improved in
places where JIT don't run, but in general re module is faster than
the equivalent written in RPython, because it's jitted.

I completely don't follow the second part of your mail. Again: what
are you trying to achieve?

> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From cfbolz at gmx.de  Fri Mar 25 10:39:37 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 25 Mar 2011 10:39:37 +0100
Subject: [pypy-dev] Ctypes module written in RPython
In-Reply-To: <AANLkTi=EL=f2ZnU+aNWNRUyyzVD6ztM+2qrf2b-EtOOY@mail.gmail.com>
References: <AANLkTi=EL=f2ZnU+aNWNRUyyzVD6ztM+2qrf2b-EtOOY@mail.gmail.com>
Message-ID: <4D8C62D9.2030407@gmx.de>

On 03/24/2011 10:14 PM, Alex Perry wrote:
> I can't find it in the docs, but it has been alluded to in the past.
> How far is the project from being able to compile a rpython module?
> I'd expect that to emit a wrapper that imports ctypes and declares
> calls into a shared library, and a directory of C code which
> implements the internals and can compile into the expected shared
> library.

I think the crucial part of your question that you don't state is that 
you want to be able to use these RPython modules compiled to C from 
CPython and Jython or really anything that supports ctypes, right?

Carl Friedrich


From arigo at tunes.org  Fri Mar 25 11:41:16 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 25 Mar 2011 11:41:16 +0100
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <41662.1300409081@iinet.net.au>
References: <41662.1300409081@iinet.net.au>
Message-ID: <AANLkTim-Q8hq=TsVp+2BU+BdP=y_YTBZUdyqMh5o__2g@mail.gmail.com>

Hi,

On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au
<hyarion at iinet.net.au> wrote:
> (...) Wrapping each bytecode in an STM
> transaction would give you an as-if-serial execution order, again with no guarantees
> about which order. You get transaction overheads instead of lock/unlock overheads,
> but some STM systems can be quite efficient for short transactions that rarely
> conflict.

Yes, I also thought about this as one of the solutions that would "fit
the model" of PyPy by not needing changes all over the place.
However, I am unsure that the performance of STM is good enough for
that application so far.  Maybe I'm wrong, but I fear (a priori, with
no precise experience) that it would be really too slow to wrap *all*
memory reads and writes with the STM machinery.

I would be interested in learning if I'm wrong, or if there are
hardware solutions around the corner ready to be tried.


A bient?t,

Armin.


From tbaldridge at gmail.com  Fri Mar 25 16:00:22 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 25 Mar 2011 10:00:22 -0500
Subject: [pypy-dev] Runtime module loading
Message-ID: <AANLkTim=3CmsBVg5MV_NoH_kHXRaGAJaeT=6TKhDCvYB@mail.gmail.com>

I have a program written in RPython. Is there a built-in way to import
other RPython modules at runtime? Or is that some sort of upcoming
feature? Basically I'm writing a language using RPython and I'm trying
to figure out how to allow the user to load modules without making
them create an FFI.

Timothy

-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From brownan at gmail.com  Fri Mar 25 17:47:20 2011
From: brownan at gmail.com (Andrew Brown)
Date: Fri, 25 Mar 2011 12:47:20 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
Message-ID: <AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>

Thanks, it does indeed work now!
-Andrew

On Thu, Mar 24, 2011 at 12:56 PM, Armin Rigo <arigo at tunes.org> wrote:

> Re-hi,
>
> On Thu, Mar 24, 2011 at 5:11 PM, Armin Rigo <arigo at tunes.org> wrote:
> > Working on it, by cleaning up optimizeopt/heap.py...
>
> Done, at least as far as fixing the bug is concerned.  Now your
> original version (with can_enter_jit removed) works.
>
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110325/b629dbc2/attachment.html>

From arigo at tunes.org  Fri Mar 25 19:18:10 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 25 Mar 2011 19:18:10 +0100
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
Message-ID: <AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>

Hi Andrew,

On Fri, Mar 25, 2011 at 5:47 PM, Andrew Brown <brownan at gmail.com> wrote:
> Thanks, it does indeed work now!

The next step is to have a look at the traces produced (run with
PYPYLOG=jit-log-opt:logfile), and spot the obvious missing
optimizations.  The biggest issue seems to be the fact that the
dictionary 'bracket_map' is green, but it is not enough to ensure that
it is a constant dict (it could be mutated behind the JIT's back); so
in the end, every trace contains reads from it.  You could fix it by
moving the line

            newpc = bracket_map[pc]

to a new function to which you apply the decorator @pypy.rlib.jit.pure_function.


A bient?t,

Armin.


From benjamin at python.org  Fri Mar 25 19:41:46 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 25 Mar 2011 13:41:46 -0500
Subject: [pypy-dev] Runtime module loading
In-Reply-To: <AANLkTim=3CmsBVg5MV_NoH_kHXRaGAJaeT=6TKhDCvYB@mail.gmail.com>
References: <AANLkTim=3CmsBVg5MV_NoH_kHXRaGAJaeT=6TKhDCvYB@mail.gmail.com>
Message-ID: <AANLkTi=Q2Po2=bBE=zG-O4oO0c+wrCMry5aCQ1+_25wK@mail.gmail.com>

2011/3/25 Timothy Baldridge <tbaldridge at gmail.com>:
> I have a program written in RPython. Is there a built-in way to import
> other RPython modules at runtime? Or is that some sort of upcoming
> feature? Basically I'm writing a language using RPython and I'm trying
> to figure out how to allow the user to load modules without making
> them create an FFI.
No.


-- 
Regards,
Benjamin


From dimaqq at gmail.com  Sat Mar 26 05:53:30 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Fri, 25 Mar 2011 21:53:30 -0700
Subject: [pypy-dev] cpyext reference counting and other gc's
Message-ID: <AANLkTikN2Zf=2gGwXDxvrjQAw0myt1rMffd7UeM-Ga8D@mail.gmail.com>

Hey, I had a look at cpyext recently and saw that reference counting
is emulated with, err, reference counting, seeminlgy in the referenced
object itself.

Does this mean that cpyext would not work with other gc's or is there
some wrapping going on behind the scenes?


From amauryfa at gmail.com  Sat Mar 26 09:33:45 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 26 Mar 2011 09:33:45 +0100
Subject: [pypy-dev] cpyext reference counting and other gc's
In-Reply-To: <AANLkTikN2Zf=2gGwXDxvrjQAw0myt1rMffd7UeM-Ga8D@mail.gmail.com>
References: <AANLkTikN2Zf=2gGwXDxvrjQAw0myt1rMffd7UeM-Ga8D@mail.gmail.com>
Message-ID: <AANLkTimE1csSr7RCBa53v2iT_wZbQ5iPzcWj3ZdyLqzc@mail.gmail.com>

Hi,

2011/3/26 Dima Tisnek <dimaqq at gmail.com>:
> Hey, I had a look at cpyext recently and saw that reference counting
> is emulated with, err, reference counting, seeminlgy in the referenced
> object itself.
>
> Does this mean that cpyext would not work with other gc's or is there
> some wrapping going on behind the scenes?

Cpyext works with all pypy gc's. The PyObject* exposed to C code is actually
a proxy to the "real" interpreter object; a dict lookup is necessary each time a
reference crosses the C/pypy boundary. Yes, this is slow.

This is implemented in pypy/module/cpyext/pyobject.py; the main functions are
create_ref() and from_ref().

-- 
Amaury Forgeot d'Arc


From arigo at tunes.org  Sat Mar 26 11:50:36 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 26 Mar 2011 11:50:36 +0100
Subject: [pypy-dev] Runtime module loading
In-Reply-To: <AANLkTim=3CmsBVg5MV_NoH_kHXRaGAJaeT=6TKhDCvYB@mail.gmail.com>
References: <AANLkTim=3CmsBVg5MV_NoH_kHXRaGAJaeT=6TKhDCvYB@mail.gmail.com>
Message-ID: <AANLkTim7RhMQ-18V1UQ3xWomZC6EAXw4kcMp2vgW+L-Q@mail.gmail.com>

Hi Timothy,

On Fri, Mar 25, 2011 at 4:00 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> I have a program written in RPython. Is there a built-in way to import
> other RPython modules at runtime?

No, and that's not obviously fixed; for example, the GC relies on
having a table of all the RPython types in the program.  In order to
load dynamically new RPython extension modules, we would need to do
something about that.  Nothing unsolvable, but there are quite a few
such places around.


A bient?t,

Armin.


From dimaqq at gmail.com  Sat Mar 26 18:03:47 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Sat, 26 Mar 2011 10:03:47 -0700
Subject: [pypy-dev] cpyext reference counting and other gc's
In-Reply-To: <AANLkTimE1csSr7RCBa53v2iT_wZbQ5iPzcWj3ZdyLqzc@mail.gmail.com>
References: <AANLkTikN2Zf=2gGwXDxvrjQAw0myt1rMffd7UeM-Ga8D@mail.gmail.com>
	<AANLkTimE1csSr7RCBa53v2iT_wZbQ5iPzcWj3ZdyLqzc@mail.gmail.com>
Message-ID: <AANLkTimLEVBT9ZX=XOPBH8N4f_P-2R6RLaicNoeFda=z@mail.gmail.com>

I have an alternative idea in mind

I'll write up a doc, stick it on github and share with you guys in a
couple of days

thanks for a clear answer, I just couldn't figure that out form code easily :P

d.

On 26 March 2011 01:33, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Hi,
>
> 2011/3/26 Dima Tisnek <dimaqq at gmail.com>:
>> Hey, I had a look at cpyext recently and saw that reference counting
>> is emulated with, err, reference counting, seeminlgy in the referenced
>> object itself.
>>
>> Does this mean that cpyext would not work with other gc's or is there
>> some wrapping going on behind the scenes?
>
> Cpyext works with all pypy gc's. The PyObject* exposed to C code is actually
> a proxy to the "real" interpreter object; a dict lookup is necessary each time a
> reference crosses the C/pypy boundary. Yes, this is slow.
>
> This is implemented in pypy/module/cpyext/pyobject.py; the main functions are
> create_ref() and from_ref().
>
> --
> Amaury Forgeot d'Arc
>


From arigo at tunes.org  Sun Mar 27 19:49:00 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 27 Mar 2011 19:49:00 +0200
Subject: [pypy-dev] Next sprint
Message-ID: <AANLkTikP8DptiZ4nrv2u9D71QWCRUtkDM-6d1+yHZn9f@mail.gmail.com>

Hi all,

The next sprint will most probably occur in Gothenburg, April 25th to
May 1st.  A proper sprint announcement should follow shortly.


A bient?t,

Armin.


From romain.py at gmail.com  Sun Mar 27 22:07:04 2011
From: romain.py at gmail.com (Romain Guillebert)
Date: Sun, 27 Mar 2011 21:07:04 +0100
Subject: [pypy-dev] Google Summer of Code Idea Proposal
Message-ID: <20110327200703.GA12448@ubuntu>

Hi

I'm interested in doing the Summer of code on PyPy, from what I saw on
the mailing list and on irc, I would like to work on 2 things which
might interest you (there is no order of preference).

* Python backend for Cython (to interface with C code, this could use
  use ctypes or an API exposed by PyPy), the backend would either
  produce python code or python bytecode and will allow PyPy to JIT the
  Cython extension (I talked about it with Alex Gaynor yesterday).

* Improve the JVM backend, making it translate on the trunk and
  interface (R)Python code with Java. (As proposed by Antonio Cuni).

I'm available from the end of May (around the 25th) to the beginning of
September so this shouldn't be a problem.

If this is OK, which one would you prefer ?

Regards
Romain


From ademan555 at gmail.com  Sun Mar 27 22:27:48 2011
From: ademan555 at gmail.com (Dan Roberts)
Date: Sun, 27 Mar 2011 13:27:48 -0700
Subject: [pypy-dev] Google Summer of Code Idea Proposal
Message-ID: <BANLkTinCeuFMVRwq9-zvaXP7Ar_zU-bwTQ@mail.gmail.com>

Hi Romain,
    I just wanted to chime in that I think a Python backend for Cython would
be awesome. You wouldn't want to generate bytecode for compatibility
reasons: PyPy doesn't use exactly CPython's bytecode, and afaik IronPython
and Jython both just use their host VM bytecode directly. Generating
ctypes+pure python sources would be the most generally useful, and I think
that'd be the best GSoC project.
    Some great follow-up work would be to skip ctypes on PyPy and generate
different calls, as Antonio has found that even with his awesome
optimizations, ctypes is significantly (I think it was 5x) slower than the
alternative, rawffi I think, but don't quote me on that or the number.
    As for reviving the jvm backend, I think that's a very *interesting*
project, I actually have looked at doing that myself, there's a tiny bit of
work in the ootype-virtualrefs that I've done. You'll need to learn a ton
about how RPython works, the translator, and so on. I'm not sure that one
would get approval though, especially pitted against awesome projects like
your ctypes project. But don't let me discourage you from applying for
*both*, I just wouldn't put all my eggs in the jvm-backend basket.

Cheers,
Dan
On Mar 27, 2011 1:07 PM, "Romain Guillebert" <romain.py at gmail.com> wrote:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110327/6fdd2eb1/attachment.html>

From hyarion at iinet.net.au  Mon Mar 28 02:53:55 2011
From: hyarion at iinet.net.au (hyarion at iinet.net.au)
Date: Mon, 28 Mar 2011 00:53:55 +0000
Subject: [pypy-dev] Thinking about the GIL
Message-ID: <33870.1301273635@iinet.net.au>


On Fri Mar 25 21:41 , Armin Rigo  sent:

>Hi,
>
>On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au
>hyarion at iinet.net.au> wrote:
>> (...) Wrapping each bytecode in an STM
>> transaction would give you an as-if-serial execution order, again with no 
guarantees
>> about which order. You get transaction overheads instead of lock/unlock 
overheads,
>> but some STM systems can be quite efficient for short transactions that rarely
>> conflict.
>
>Yes, I also thought about this as one of the solutions that would "fit
>the model" of PyPy by not needing changes all over the place.
>However, I am unsure that the performance of STM is good enough for
>that application so far.  Maybe I'm wrong, but I fear (a priori, with
>no precise experience) that it would be really too slow to wrap *all*
>memory reads and writes with the STM machinery.

Yeah, I suppose to get the performance I'm thinking of you probably need to know 
what you're sharing (and for it not to be everything). You wouldn't need all memory 
reads and writes to be transactional, just ones to interpreter-level values that are 
mutable and shared between threads, and ones to things that represent app-level 
shareable values.

The STM system I worked on was for a declarative language with immutable data. In 
that context it turns out the system only had to be careful when retrieving 
references from STM; what the reference points to could still be used as normal non-
shared data. In a language like Python I guess almost any bit of memory could 
potentially be (or later become) a shared value that another thread could be using.

What's the "success-rate" of the JIT's malloc-removal like? You could omit the 
transactional overhead for those values. Would that get close enough for the 
important 20% of code? You could probably do some other jiggery-pokery under the 
hood to minimise the amount of data protected by STM overhead too.

If I had more time I'd love to look at this sort of thing.

-- Ben


From alex.gaynor at gmail.com  Mon Mar 28 03:01:28 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sun, 27 Mar 2011 21:01:28 -0400
Subject: [pypy-dev] Thinking about the GIL
In-Reply-To: <33870.1301273635@iinet.net.au>
References: <33870.1301273635@iinet.net.au>
Message-ID: <AANLkTinWj5AT-7fx3bmNaPDa0LcPyydpZtc6GvHVR_XQ@mail.gmail.com>

On Sun, Mar 27, 2011 at 8:53 PM, hyarion at iinet.net.au
<hyarion at iinet.net.au>wrote:

>
> On Fri Mar 25 21:41 , Armin Rigo  sent:
>
> >Hi,
> >
> >On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au
> >hyarion at iinet.net.au> wrote:
> >> (...) Wrapping each bytecode in an STM
> >> transaction would give you an as-if-serial execution order, again with
> no
> guarantees
> >> about which order. You get transaction overheads instead of lock/unlock
> overheads,
> >> but some STM systems can be quite efficient for short transactions that
> rarely
> >> conflict.
> >
> >Yes, I also thought about this as one of the solutions that would "fit
> >the model" of PyPy by not needing changes all over the place.
> >However, I am unsure that the performance of STM is good enough for
> >that application so far.  Maybe I'm wrong, but I fear (a priori, with
> >no precise experience) that it would be really too slow to wrap *all*
> >memory reads and writes with the STM machinery.
>
> Yeah, I suppose to get the performance I'm thinking of you probably need to
> know
> what you're sharing (and for it not to be everything). You wouldn't need
> all memory
> reads and writes to be transactional, just ones to interpreter-level values
> that are
> mutable and shared between threads, and ones to things that represent
> app-level
> shareable values.
>
> The STM system I worked on was for a declarative language with immutable
> data. In
> that context it turns out the system only had to be careful when retrieving
> references from STM; what the reference points to could still be used as
> normal non-
> shared data. In a language like Python I guess almost any bit of memory
> could
> potentially be (or later become) a shared value that another thread could
> be using.
>
> What's the "success-rate" of the JIT's malloc-removal like? You could omit
> the
> transactional overhead for those values. Would that get close enough for
> the
> important 20% of code? You could probably do some other jiggery-pokery
> under the
> hood to minimise the amount of data protected by STM overhead too.
>
> If I had more time I'd love to look at this sort of thing.
>
> -- Ben
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

https://bitbucket.org/pypy/extradoc/raw/63e4617062b2/talk/pepm2011/bolz-allocation-removal.pdfhas
info on the success rate of allocation removal.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110327/bea74927/attachment.html>

From brownan at gmail.com  Mon Mar 28 19:21:01 2011
From: brownan at gmail.com (Andrew Brown)
Date: Mon, 28 Mar 2011 13:21:01 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
Message-ID: <AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>

I tried that logging option once, but I didn't know how to read the logs.
They're not exactly self explanatory. Is there a resource somewhere that
explains how to read those logs?

Regardless, I've implemented your suggestion and moved reads from that
dictionary to a function decorated with @purefunction. Indeed, performance
is greatly improved! Thanks!

Current version:
https://bitbucket.org/brownan/bf-interpreter/src/d3394345272e/targetbf.py

A few questions:

When the optimizer encounters a "pure" function, it must compare the objects
passed in to previous invocations... does it consider the contents of
container or other mutatible objects? or just the object identity, to be
part of the function's input?
It looks like, from logs of my new version, it's not reading from the
dictionary at all during the trace, so I would guess it's not considering
the actual contents of the dictionary as part of the function's input. This
isn't surprising, but I just want to know for sure.

Second, I noticed in jit.py the function hint() which has a parameter:
"promote - promote the argument from a variable into a constant". Could this
be an appropriate alternate to the @purefunction solution? Or, I'm guessing,
does it just mean the name bracket_map won't change bindings, but does not
impose a restriction on mutating the dictionary?

-Andrew

On Fri, Mar 25, 2011 at 2:18 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Andrew,
>
> On Fri, Mar 25, 2011 at 5:47 PM, Andrew Brown <brownan at gmail.com> wrote:
> > Thanks, it does indeed work now!
>
> The next step is to have a look at the traces produced (run with
> PYPYLOG=jit-log-opt:logfile), and spot the obvious missing
> optimizations.  The biggest issue seems to be the fact that the
> dictionary 'bracket_map' is green, but it is not enough to ensure that
> it is a constant dict (it could be mutated behind the JIT's back); so
> in the end, every trace contains reads from it.  You could fix it by
> moving the line
>
>            newpc = bracket_map[pc]
>
> to a new function to which you apply the decorator
> @pypy.rlib.jit.pure_function.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110328/e58c14de/attachment.html>

From cfbolz at gmx.de  Mon Mar 28 19:25:22 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 28 Mar 2011 19:25:22 +0200
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
Message-ID: <4D90C482.7070608@gmx.de>

On 03/28/2011 07:21 PM, Andrew Brown wrote:
> I tried that logging option once, but I didn't know how to read the
> logs. They're not exactly self explanatory. Is there a resource
> somewhere that explains how to read those logs?

Not really, no :-(

> Regardless, I've implemented your suggestion and moved reads from that
> dictionary to a function decorated with @purefunction. Indeed,
> performance is greatly improved! Thanks!
>
> Current version:
> https://bitbucket.org/brownan/bf-interpreter/src/d3394345272e/targetbf.py
>
> A few questions:
>
> When the optimizer encounters a "pure" function, it must compare the
> objects passed in to previous invocations... does it consider the
> contents of container or other mutatible objects? or just the object
> identity, to be part of the function's input?

Just the object's identity.

> It looks like, from logs of my new version, it's not reading from the
> dictionary at all during the trace, so I would guess it's not
> considering the actual contents of the dictionary as part of the
> function's input. This isn't surprising, but I just want to know for sure.
>
> Second, I noticed in jit.py the function hint() which has a parameter:
> "promote - promote the argument from a variable into a constant". Could
> this be an appropriate alternate to the @purefunction solution? Or, I'm
> guessing, does it just mean the name bracket_map won't change bindings,
> but does not impose a restriction on mutating the dictionary?
>

If you are interested, this blog series explains the usage of hints:

http://bit.ly/bundles/cfbolz/1

The logs there are a bit niceified though.

Carl Friedrich


From fijall at gmail.com  Tue Mar 29 00:24:33 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 28 Mar 2011 16:24:33 -0600
Subject: [pypy-dev] Google Summer of Code Idea Proposal
In-Reply-To: <20110327200703.GA12448@ubuntu>
References: <20110327200703.GA12448@ubuntu>
Message-ID: <AANLkTi=n3akDy-M7yh353hxLJxkmZAcJC3+ipnvKL_G=@mail.gmail.com>

On Sun, Mar 27, 2011 at 2:07 PM, Romain Guillebert <romain.py at gmail.com> wrote:
> Hi
>
> I'm interested in doing the Summer of code on PyPy, from what I saw on
> the mailing list and on irc, I would like to work on 2 things which
> might interest you (there is no order of preference).
>
> * Python backend for Cython (to interface with C code, this could use
> ?use ctypes or an API exposed by PyPy), the backend would either
> ?produce python code or python bytecode and will allow PyPy to JIT the
> ?Cython extension (I talked about it with Alex Gaynor yesterday).
>
> * Improve the JVM backend, making it translate on the trunk and
> ?interface (R)Python code with Java. (As proposed by Antonio Cuni).
>
> I'm available from the end of May (around the 25th) to the beginning of
> September so this shouldn't be a problem.
>
> If this is OK, which one would you prefer ?

Hey.

Personally I would vastly prefer the first one. It's more immediately usable.

We do require people submitting proposals to contribute some code
(like fixing a small issue) first before being accepted. Anywhere you
would like to contribute something? Feel free to pop up on IRC and ask
people around what's interesting.

Cheers,
fijal

>
> Regards
> Romain
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From anto.cuni at gmail.com  Tue Mar 29 09:31:19 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 29 Mar 2011 09:31:19 +0200
Subject: [pypy-dev] Hacking dropbox
Message-ID: <4D918AC7.7070604@gmail.com>

Hi,

I don't have any news from the dropbox side, but as an aside note, I found a
project which is open source and implements the very same technique I used for
dropbox, and additionally it also decompiles back to the source code:

http://code.google.com/p/pyretic/

ciao,
Anto


From arigo at tunes.org  Tue Mar 29 11:33:52 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 29 Mar 2011 11:33:52 +0200
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
Message-ID: <AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>

Hi Andrew,

On Mon, Mar 28, 2011 at 7:21 PM, Andrew Brown <brownan at gmail.com> wrote:
> When the optimizer encounters a "pure" function, it must compare the objects
> "promote - promote the argument from a variable into a constant". Could this
> be an appropriate alternate to the @purefunction solution? Or, I'm guessing,
> does it just mean the name bracket_map won't change bindings, but does not
> impose a restriction on mutating the dictionary?

One point of view on 'promote' is to mean "this variable was red, but
now turn it green (i.e. make it constant)".  It has no effect on a
variable that is already green (= a constant).

We have no support for considering that a dict is immutable, so it
needs to be done with @purefunction.  But to be effective,
@purefunction must receive constant arguments; so in one or two places
in the source code of PyPy you will find a construction like:

   x = hint(x, promote=True)   # turn x into a constant
   some_pure_function(x)     # call this pure function on x

Indeed, Carl Friedrich's blog series explains it nicely, but it should
also mention that when the hints described in the blog are applied not
to integer but to pointers, they apply only to the pointers
themselves, not on the fields of the objects they point to.


A bient?t,

Armin.


From fwierzbicki at gmail.com  Wed Mar 30 04:40:10 2011
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Tue, 29 Mar 2011 19:40:10 -0700
Subject: [pypy-dev] The JVM backend and Jython
Message-ID: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>

Hi all,

It was nice meeting up with many of you at PyCon!

I've been thinking about the first steps towards collaboration between
the Jython project and the PyPy project. It looks like it isn't going
to be too long before we are all (CPython, PyPy, IronPython, Jython,
etc) working on a single shared repository for all of our standard
library .py code. In my ideal world there would come a day when there
is also no standalone Java code in the Jython project: that is the
shared standard library would contain all of Jython's .py files, and
all of the Java would be generated from PyPy and Jython as a
standalone project would disappear. It is possible that this is too
ambitious, but big goals are more fun, right? In reality even if this
where to get going, I imagine it would be a 10+ year plan :)

So to my question - just how broken is the JVM backend? Are there
workarounds that would allow the Java code to get generated? I ask
because I would like to evaluate the generated Java parser as a
potential replacement for our current ANTLR based parser. It seems
like a nice baby step towards real collaboration since it seems like a
relatively easy place to start. Clearly it would need adjustments to
actually work for Jython - but I'd be able to look into that part. I
don't think I have the time to try to unbreak the translation
though...

-Frank


From anto.cuni at gmail.com  Wed Mar 30 09:18:57 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 30 Mar 2011 09:18:57 +0200
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
Message-ID: <4D92D961.3070309@gmail.com>

Hi Frank,

On 30/03/11 04:40, fwierzbicki at gmail.com wrote:
> Hi all,
> 
> It was nice meeting up with many of you at PyCon!
> 
> I've been thinking about the first steps towards collaboration between
> the Jython project and the PyPy project. It looks like it isn't going
> to be too long before we are all (CPython, PyPy, IronPython, Jython,
> etc) working on a single shared repository for all of our standard
> library .py code. In my ideal world there would come a day when there
> is also no standalone Java code in the Jython project: that is the
> shared standard library would contain all of Jython's .py files, and
> all of the Java would be generated from PyPy and Jython as a
> standalone project would disappear. It is possible that this is too
> ambitious, but big goals are more fun, right? In reality even if this
> where to get going, I imagine it would be a 10+ year plan :)


wow, that's definitely a nice (and big) plan :-)

> So to my question - just how broken is the JVM backend? Are there
> workarounds that would allow the Java code to get generated? 

"not much broken".
Last time I tried, the only broken thing was "virtualrefs", which is something
needed for the jit, but that at the moment is not supported at all by ootype
and thus it blocks the translation.

However, I think that fixing it is probably very easy: there is a branch for
this which was started by Ademan:
https://bitbucket.org/pypy/pypy/src/ootype-virtualrefs

Dan, do you plan to finish the work on it? Else, I can just do it probably.

> I ask
> because I would like to evaluate the generated Java parser as a
> potential replacement for our current ANTLR based parser. It seems
> like a nice baby step towards real collaboration since it seems like a
> relatively easy place to start. Clearly it would need adjustments to
> actually work for Jython - but I'd be able to look into that part. I
> don't think I have the time to try to unbreak the translation
> though...

I have to warn you that at the moment, you cannot invoke any Java code from
RPython.  Implementing it has been on my todo list for years now :-(, but I
never managed to find the time and the motivation to do it.  However, for
using the PyPy parser inside Jython it should be enough to do the other way
around, i.e. call RPython code from Java, which should be possible.

ciao,
Anto


From fwierzbicki at gmail.com  Wed Mar 30 19:37:02 2011
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Wed, 30 Mar 2011 10:37:02 -0700
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <4D92D961.3070309@gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
	<4D92D961.3070309@gmail.com>
Message-ID: <AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>

On Wed, Mar 30, 2011 at 12:18 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> I have to warn you that at the moment, you cannot invoke any Java code from
> RPython. ?Implementing it has been on my todo list for years now :-(, but I
> never managed to find the time and the motivation to do it. ?However, for
> using the PyPy parser inside Jython it should be enough to do the other way
> around, i.e. call RPython code from Java, which should be possible.
My thoughts here are taking a very primitive step - that is run the
JVM translation and look at the generated Java - then see what needs
to be modified so that I could use the generated Java parser from
Jython. At this stage I would be using PyPy exactly the way I use
ANTLR now - as a parser generator. There wouldn't be any need at all
for calling into Java code (as far as I can think of). I think if we
Jython developers get some experience with PyPY - we might be able to
help with the task of calling into Java from PyPy - since we know a
bit about that :)

-Frank


From santagada at gmail.com  Wed Mar 30 19:57:30 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Wed, 30 Mar 2011 14:57:30 -0300
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
	<4D92D961.3070309@gmail.com>
	<AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
Message-ID: <AANLkTim-Pqr1X2zbcvj1Za7WROhbN0yt-hrtZQiaDbaj@mail.gmail.com>

On Wed, Mar 30, 2011 at 2:37 PM, fwierzbicki at gmail.com
<fwierzbicki at gmail.com> wrote:
> On Wed, Mar 30, 2011 at 12:18 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> I have to warn you that at the moment, you cannot invoke any Java code from
>> RPython. ?Implementing it has been on my todo list for years now :-(, but I
>> never managed to find the time and the motivation to do it. ?However, for
>> using the PyPy parser inside Jython it should be enough to do the other way
>> around, i.e. call RPython code from Java, which should be possible.
> My thoughts here are taking a very primitive step - that is run the
> JVM translation and look at the generated Java - then see what needs
> to be modified so that I could use the generated Java parser from
> Jython. At this stage I would be using PyPy exactly the way I use
> ANTLR now - as a parser generator. There wouldn't be any need at all
> for calling into Java code (as far as I can think of). I think if we
> Jython developers get some experience with PyPY - we might be able to
> help with the task of calling into Java from PyPy - since we know a
> bit about that :)

IIRC the jvm backend generates java bytecode directly in text form for
a java assembler (I forgot the name of it), maybe a step would be to
see if there is any way to import the .class back in a java program.

-- 
Leonardo Santagada


From anto.cuni at gmail.com  Wed Mar 30 20:11:36 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 30 Mar 2011 20:11:36 +0200
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>	<4D92D961.3070309@gmail.com>
	<AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
Message-ID: <4D937258.6090500@gmail.com>

On 30/03/11 19:37, fwierzbicki at gmail.com wrote:

> My thoughts here are taking a very primitive step - that is run the
> JVM translation and look at the generated Java - then see what needs
> to be modified so that I could use the generated Java parser from
> Jython. At this stage I would be using PyPy exactly the way I use
> ANTLR now - as a parser generator. There wouldn't be any need at all
> for calling into Java code (as far as I can think of). 

yes, I think it makes sense.
Actually, as Leonardo says we don't generate java code but assembler which is
converted to .class by jasmin. However, it should not change anything.

> I think if we
> Jython developers get some experience with PyPY - we might be able to
> help with the task of calling into Java from PyPy - since we know a
> bit about that :)

that would be extremely cool :-)

Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref
branch, I'll try to finish the work so you can start playing with it.

ciao,
Anto


From pjenvey at underboss.org  Wed Mar 30 21:59:39 2011
From: pjenvey at underboss.org (Philip Jenvey)
Date: Wed, 30 Mar 2011 12:59:39 -0700
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <4D92D961.3070309@gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
	<4D92D961.3070309@gmail.com>
Message-ID: <F603FB1F-9F9B-4F79-9465-DC928ACD991F@underboss.org>


On Mar 30, 2011, at 12:18 AM, Antonio Cuni wrote:
> 
>> I ask
>> because I would like to evaluate the generated Java parser as a
>> potential replacement for our current ANTLR based parser. It seems
>> like a nice baby step towards real collaboration since it seems like a
>> relatively easy place to start. Clearly it would need adjustments to
>> actually work for Jython - but I'd be able to look into that part. I
>> don't think I have the time to try to unbreak the translation
>> though...
> 
> I have to warn you that at the moment, you cannot invoke any Java code from
> RPython.  Implementing it has been on my todo list for years now :-(, but I
> never managed to find the time and the motivation to do it.  However, for
> using the PyPy parser inside Jython it should be enough to do the other way
> around, i.e. call RPython code from Java, which should be possible.

FYI there's an interesting solution on how to call into arbitrary Java code from an invokedynamic enabled language via Atilla Szegedi's somewhat experimental Meta Object Protocol:

https://github.com/szegedi/dynalink

Basically on Java 7 invokedynamic a dynamic language invocation instruction is something along the lines of:

obj.someattr = someobj
->
invokedynamic "__setattr__"(Ljava/lang/String;Ljava/lang/Object;I)V;

Which might dispatch to a PyObject.__setattr__(String name, PyObject value) method (or easily something completely different in invokedynamic land).

The MOP ads another layer in between the invocation and the call site. So as the language implementor you'd use the MOP library to 'relink' your __setattr__ call site to the meta object protocol's more generic version (it only supports a few features but one of them is generic property access). Then you can do the invocation via the MOP, something like:

invokedynamic "dyn:setProp:someattr"(Ljava/lang/Object;I)V;

The point being that other JVM languages will eventually support the MOP protocol and then you'd get property access, invocation, etc. to those languages for free. More importantly, out of the box the MOP lib implements the protocol for plane old Java objects.

If you're already using invokedynamic the library seems simple to hook into and there's basically no call overhead added.

I'm not sure this would even be applicable to RPython as it's more static in nature. But it will certainly help in calling Java from regular Python.

--
Philip Jenvey

From fwierzbicki at gmail.com  Thu Mar 31 00:35:42 2011
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Wed, 30 Mar 2011 15:35:42 -0700
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <4D937258.6090500@gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
	<4D92D961.3070309@gmail.com>
	<AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
	<4D937258.6090500@gmail.com>
Message-ID: <AANLkTi=eKXNxx4_Z53TbzZwT6NHjFPb2bnUq6O14t79W@mail.gmail.com>

On Wed, Mar 30, 2011 at 11:11 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 30/03/11 19:37, fwierzbicki at gmail.com wrote:
>
>> My thoughts here are taking a very primitive step - that is run the
>> JVM translation and look at the generated Java - then see what needs
>> to be modified so that I could use the generated Java parser from
>> Jython. At this stage I would be using PyPy exactly the way I use
>> ANTLR now - as a parser generator. There wouldn't be any need at all
>> for calling into Java code (as far as I can think of).
>
> yes, I think it makes sense.
> Actually, as Leonardo says we don't generate java code but assembler which is
> converted to .class by jasmin. However, it should not change anything.
Assember/.class files shouldn't be a problem.

>> I think if we
>> Jython developers get some experience with PyPY - we might be able to
>> help with the task of calling into Java from PyPy - since we know a
>> bit about that :)
>
> that would be extremely cool :-)
>
> Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref
> branch, I'll try to finish the work so you can start playing with it.
Great thanks!

-Frank


From brownan at gmail.com  Thu Mar 31 14:28:48 2011
From: brownan at gmail.com (Andrew Brown)
Date: Thu, 31 Mar 2011 08:28:48 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
Message-ID: <AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>

Thanks for the info!
That's all the questions I have, for now at least. Feel free to reply with
any more tips if you think of any.

I read over your posts you linked, Carl. They were certainly informative and
helpful, thanks. I'll keep thinking of ways to improve the performance of my
test interpreter, but it's so simple, I don't think there's much more that
can be done. The shared attribute maps described by that link don't really
apply here.

In any case, I'm satisfied with the speed. It's still beaten by a BF to C
translator combined with gcc -O2 though, that'd be a tough case to beat. =)

-Andrew

On Tue, Mar 29, 2011 at 5:33 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Andrew,
>
> On Mon, Mar 28, 2011 at 7:21 PM, Andrew Brown <brownan at gmail.com> wrote:
> > When the optimizer encounters a "pure" function, it must compare the
> objects
> > "promote - promote the argument from a variable into a constant". Could
> this
> > be an appropriate alternate to the @purefunction solution? Or, I'm
> guessing,
> > does it just mean the name bracket_map won't change bindings, but does
> not
> > impose a restriction on mutating the dictionary?
>
> One point of view on 'promote' is to mean "this variable was red, but
> now turn it green (i.e. make it constant)".  It has no effect on a
> variable that is already green (= a constant).
>
> We have no support for considering that a dict is immutable, so it
> needs to be done with @purefunction.  But to be effective,
> @purefunction must receive constant arguments; so in one or two places
> in the source code of PyPy you will find a construction like:
>
>   x = hint(x, promote=True)   # turn x into a constant
>   some_pure_function(x)     # call this pure function on x
>
> Indeed, Carl Friedrich's blog series explains it nicely, but it should
> also mention that when the hints described in the blog are applied not
> to integer but to pointers, they apply only to the pointers
> themselves, not on the fields of the objects they point to.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110331/8c564280/attachment.html>

From anto.cuni at gmail.com  Thu Mar 31 14:33:40 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 31 Mar 2011 14:33:40 +0200
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
Message-ID: <4D9474A4.3020909@gmail.com>

On 31/03/11 14:28, Andrew Brown wrote:
> In any case, I'm satisfied with the speed. It's still beaten by a BF to C
> translator combined with gcc -O2 though, that'd be a tough case to beat. =)

what happens if you combine the BF to C with gcc -O0 or -O1?

Anyway, I think that if you feel like writing a post explaining your
experience with using pypy and its jit for writing an interpreter, we could
publish it on our blog.  I suppose it would be useful/interesting for other
people as well.

What do the others think?


From lac at openend.se  Thu Mar 31 14:38:24 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 31 Mar 2011 14:38:24 +0200
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: Message from Antonio Cuni <anto.cuni@gmail.com> 
	of "Thu, 31 Mar 2011 14:33:40 +0200." <4D9474A4.3020909@gmail.com> 
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
	<4D9474A4.3020909@gmail.com> 
Message-ID: <201103311238.p2VCcOuk024603@theraft.openend.se>

In a message of Thu, 31 Mar 2011 14:33:40 +0200, Antonio Cuni writes:
>On 31/03/11 14:28, Andrew Brown wrote:
>> In any case, I'm satisfied with the speed. It's still beaten by a BF to
> C
>> translator combined with gcc -O2 though, that'd be a tough case to beat
>. =)
>
>what happens if you combine the BF to C with gcc -O0 or -O1?
>
>Anyway, I think that if you feel like writing a post explaining your
>experience with using pypy and its jit for writing an interpreter, we cou
>ld
>publish it on our blog.  I suppose it would be useful/interesting for oth
>er
>people as well.
>
>What do the others think?

I'd look forward to reading it.

Laura



From brownan at gmail.com  Thu Mar 31 14:40:09 2011
From: brownan at gmail.com (Andrew Brown)
Date: Thu, 31 Mar 2011 08:40:09 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <4D9474A4.3020909@gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
	<4D9474A4.3020909@gmail.com>
Message-ID: <AANLkTikgFDOKgRwZ4MM0gvN=tgCDHFoc=YPgzryDw+Lb@mail.gmail.com>

Compiling with -O0 is really quick, but the runtime is fairly slow. I
haven't tried with -O1. -O2 takes a few seconds to compile, but that plus
runtime is still faster than the pypy version with jit, but not by too much
(I'm recalling the tests I did with the mandelbrot program specifically). I
can get some actual numbers later today.

Sure I'll write up a post. This was a lot of fun, and I think it's a great
way to teach people how pypy works.

-Andrew

On Thu, Mar 31, 2011 at 8:33 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> On 31/03/11 14:28, Andrew Brown wrote:
> > In any case, I'm satisfied with the speed. It's still beaten by a BF to C
> > translator combined with gcc -O2 though, that'd be a tough case to beat.
> =)
>
> what happens if you combine the BF to C with gcc -O0 or -O1?
>
> Anyway, I think that if you feel like writing a post explaining your
> experience with using pypy and its jit for writing an interpreter, we could
> publish it on our blog.  I suppose it would be useful/interesting for other
> people as well.
>
> What do the others think?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110331/eaa0ee18/attachment.html>

From tbaldridge at gmail.com  Thu Mar 31 16:19:53 2011
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Thu, 31 Mar 2011 09:19:53 -0500
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTikgFDOKgRwZ4MM0gvN=tgCDHFoc=YPgzryDw+Lb@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
	<4D9474A4.3020909@gmail.com>
	<AANLkTikgFDOKgRwZ4MM0gvN=tgCDHFoc=YPgzryDw+Lb@mail.gmail.com>
Message-ID: <AANLkTinO-kBAs4DYjCUPOGXP01VKRyOLHZnrZJXAnh5k@mail.gmail.com>

> Sure I'll write up a post. This was a lot of fun, and I think it's a great
> way to teach people how pypy works.

I'd love to read a post on this. Perhaps I'll get a few pointers that
I can use in my Clojure-pypy port.

Timothy
-- 
?One of the main causes of the fall of the Roman Empire was
that?lacking zero?they had no way to indicate successful termination
of their C programs.?
(Robert Firth)


From brownan at gmail.com  Thu Mar 31 17:44:32 2011
From: brownan at gmail.com (Andrew Brown)
Date: Thu, 31 Mar 2011 11:44:32 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTikJ9Zgk2FKfHg-CZhwSL+qj7j2c4MT_7tq8SfGB@mail.gmail.com>
References: <AANLkTikJ9Zgk2FKfHg-CZhwSL+qj7j2c4MT_7tq8SfGB@mail.gmail.com>
Message-ID: <AANLkTi==Sesg-U4B4mKDqOp2982BsDR-XmRq7QfD+3Uu@mail.gmail.com>

Here's three emails that accidentally didn't get sent to the list.

On Thu, Mar 31, 2011 at 10:41 AM, Dan Roberts <ademan555 at gmail.com> wrote:

> Hi Andrew,
>     Did you ever try your interpreter with the 99 bottles program? I got my
> interpreter down faster than the beef interpreter ~4s vs ~15s on mandelbrot,
> however even with that speed, both 'bf' and 'beef' trounced my interpreter
> by an absurd amount. It seems like it probably was a problem with my code
> base, when I first saw you were working on this I meant to ask you to try
> 99bottles.bf and see if you had similar problems. I haven't had a chance
> to examine where the problem is coming from though.
>
> Cheers,
> Dan
>

On Thu, Mar 31, 2011 at 11:02 AM, Andrew Brown <brownan at gmail.com> wrote:

> Hi Dan,
> Did you mean to send this to the list as well? I only ask because it's easy
> to hit "Reply" instead of "Reply All".
>
> Regardless, I have a 99 bottles program, in the comments it says it's
> written by Andrew Paczkowski. I haven't mentioned it just because it runs
> absurdly fast: 0.02 seconds or so (compared with 0.2s for running the py
> code on cpython), so I didn't consider it a good test. I wanted something
> that took a bit longer.
>
> I just searched for another and found one by Raphael Bois, but that runs in
> 0.04 seconds.
>
> Perhaps you're using a different version of this program that's less
> efficient and runs faster? (or maybe this really is just that fast?)
>
> Also, the mandelbrot program that I included in my repo takes 8.4 seconds
> to run on my computer. Not quite the 4 second time you're getting (have you
> published your interpreter anywhere? I'd like to look at it) I have a
> feeling I've taken this interpreter as far as it will go without doing any
> more intelligent inspection of the bf code directly.
>
> -Andrew
>

On Thu, Mar 31, 2011 at 11:20 AM, Dan Roberts <ademan555 at gmail.com> wrote:

> Hey,
>     Yeah, that's the second or third time I didn't reply to all lately :-/
> And my interpreter is on paste.pocoo.org somewhere, I can paste it again
> when I go home today. I suspect there's something wrong with it though,
> considering you're getting proper performance on 99bottles, it takes about 3
> minutes here! (On the same system where it wins on mandelbrot by >66%
> against 'beef' or bf whichever one is faster) One immediately obvious
> difference was your use of the bracket map, which I think is an awesome
> idea. I may adopt it, currently I calculate how far backwards/forwards to
> travel at "runtime" instead of preprocessing it. I made it pure so that it
> would be constant folded by the JIT, but I suppose the 1000 iterations
> before the JIT kicks in (per loop) could explain a large performance
> difference. I could probably combine both techniques and cache the results
> at runtime, by the second run, it'll be a dict lookup, so it'll be jitted
> essentially the same, and I won't have to think about parsing to find
> matching braces :-)
>
> If you can think of a good way to bring this discussion back on the mailing
> list that'd be fine.
>
> Cheers,
> Dan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110331/7a51cb94/attachment.html>

From dimaqq at gmail.com  Thu Mar 31 20:09:36 2011
From: dimaqq at gmail.com (Dima Tisnek)
Date: Thu, 31 Mar 2011 11:09:36 -0700
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <4D9474A4.3020909@gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
	<4D9474A4.3020909@gmail.com>
Message-ID: <AANLkTim=FoR7sixG_bMbKq4NvAG=bRPs7xkMNWu9o0i+@mail.gmail.com>

On 31 March 2011 05:33, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 31/03/11 14:28, Andrew Brown wrote:
>> In any case, I'm satisfied with the speed. It's still beaten by a BF to C
>> translator combined with gcc -O2 though, that'd be a tough case to beat. =)

What if bf code was really really large?

bf to c then gcc could take a hit as it might thrash cpu cache, as
single pass gcc doesn't know what a given program would actually do at
runtime.

jit'd rpy would only have 1 hotspot, always in cache, and might be a
little smarter too.

I suppose it's hard to beat 2-pass (profile driven optimized) compiled c though.

>
> what happens if you combine the BF to C with gcc -O0 or -O1?
>
> Anyway, I think that if you feel like writing a post explaining your
> experience with using pypy and its jit for writing an interpreter, we could
> publish it on our blog. ?I suppose it would be useful/interesting for other
> people as well.
>
> What do the others think?

I think it can be a great example. It's very educational ;-)

It could go into official docs/howto too.


From fijall at gmail.com  Thu Mar 31 21:57:20 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 31 Mar 2011 13:57:20 -0600
Subject: [pypy-dev] The JVM backend and Jython
In-Reply-To: <4D937258.6090500@gmail.com>
References: <AANLkTimRqcp8wpJZjZoPCgPc-ZUQHWs_SpcWh_f97+Yb@mail.gmail.com>
	<4D92D961.3070309@gmail.com>
	<AANLkTik8w8_t5MbwyDqcBp4XtOmfAMgk9MZ-6_fvaNAj@mail.gmail.com>
	<4D937258.6090500@gmail.com>
Message-ID: <AANLkTikX799KkPprxuYNH5Zp26-Z05mryMMYN31cGjgx@mail.gmail.com>

On Wed, Mar 30, 2011 at 12:11 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 30/03/11 19:37, fwierzbicki at gmail.com wrote:
>
>> My thoughts here are taking a very primitive step - that is run the
>> JVM translation and look at the generated Java - then see what needs
>> to be modified so that I could use the generated Java parser from
>> Jython. At this stage I would be using PyPy exactly the way I use
>> ANTLR now - as a parser generator. There wouldn't be any need at all
>> for calling into Java code (as far as I can think of).
>
> yes, I think it makes sense.
> Actually, as Leonardo says we don't generate java code but assembler which is
> converted to .class by jasmin. However, it should not change anything.
>
>> I think if we
>> Jython developers get some experience with PyPY - we might be able to
>> help with the task of calling into Java from PyPy - since we know a
>> bit about that :)
>
> that would be extremely cool :-)
>
> Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref
> branch, I'll try to finish the work so you can start playing with it.

Note to frank: this is kind of cool but only needed for the JIT,
otherwise it's a normal reference.

>
> ciao,
> Anto
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From brownan at gmail.com  Thu Mar 31 22:05:49 2011
From: brownan at gmail.com (Andrew Brown)
Date: Thu, 31 Mar 2011 16:05:49 -0400
Subject: [pypy-dev] Pypy custom interpreter JIT question
In-Reply-To: <AANLkTim=FoR7sixG_bMbKq4NvAG=bRPs7xkMNWu9o0i+@mail.gmail.com>
References: <AANLkTi=tiWJs3PokOLpi+Hn_+44g2fcuaXJg4yGL+P23@mail.gmail.com>
	<AANLkTi=wGaogrSxh5M=d2r2p919QNUP2nM4KTSdO6-9Q@mail.gmail.com>
	<AANLkTinEAzy4uzy9jYupbgU-F9cjc=FEu7_LWpr1s41o@mail.gmail.com>
	<AANLkTi=RpVAeBffnKOG3_7CedwgXx5EwkL3cPwS9P7sz@mail.gmail.com>
	<AANLkTimAdYHDpGTk7jy+dNshLOYAXS+B6iF6KE5R8FT_@mail.gmail.com>
	<AANLkTimyRiE6=f=v0Dek6T00=_ONd9K6oC+LiGr-Wyfg@mail.gmail.com>
	<AANLkTin5H8U7ao=5CMmW3R0snx+jbTE_wDYEqAr0VUPi@mail.gmail.com>
	<AANLkTi=sTNpQq+dfG9ZgwRMDkbRYZE4R7C6TBTPES-1-@mail.gmail.com>
	<AANLkTimukCD7LTy0xo1anXM0Q34gJGGnmaB+JAmB9zHc@mail.gmail.com>
	<AANLkTinpinGrZ29nPqvticLxi6rpqfsk=9EGCdCAKMo2@mail.gmail.com>
	<4D9474A4.3020909@gmail.com>
	<AANLkTim=FoR7sixG_bMbKq4NvAG=bRPs7xkMNWu9o0i+@mail.gmail.com>
Message-ID: <AANLkTimL2e2+b9a2zHf0JPvzDaNQgrcUY3ox1vea52Tu@mail.gmail.com>

On Thu, Mar 31, 2011 at 2:09 PM, Dima Tisnek <dimaqq at gmail.com> wrote:

> What if bf code was really really large?
>

I've only been testing with the examples at hand included in my repo. The
mandelbrot and towers of hanoi examples are pretty big though. If you can
find some larger examples, I'd like to try them.

I think it can be a great example. It's very educational ;-)


> It could go into official docs/howto too.
>

Awesome! I'm working on writing up everything, it's turning out to be pretty
long. I'm assuming no prior PyPy knowledge in the readers though =)

Here are a few numbers from tests I just did.

python double-interpreted: > 78m (did not finish)
pypy-c (with jit) double-interpreted: 41m 34.528s
translated interpreter no jit: 45s
translated interpreter jit: 7.5s
 translated direct to C, gcc -O0
  translate: 0.2s
  compile: 0.4s
  run: 18.5s
translated direct to C, gcc -O1
  translate: 0.2s
  compile: 0.85s
  run: 1.28s
translated direct to C, gcc -O2
  translate: 0.2s
  compile: 2.0s
  run: 1.34s

These were all running the mandelbrot program.

-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110331/66012db5/attachment.html>