From kenji.yano at gmail.com  Mon Aug  1 07:25:26 2011
From: kenji.yano at gmail.com (Kenji Yano)
Date: Mon, 1 Aug 2011 14:25:26 +0900
Subject: [pypy-dev] =?windows-1252?q?html=5Ffibo=2Epy=92s_problem=2E?=
Message-ID: <CAMLJXURbGCH7wsZ-bNh4okcGGO-XF9+=SidU=8WhQu2HNBBccw@mail.gmail.com>

Hi

I?found the cause of html_fibo.py at the EuroPython Hands-on.
This case, HtmlTag destructor isn?t call when the object didn?t use.
Is this garbage collection?s problem?
(pypy-trunk(1.6.0-dev) same too.)

===
def html_fibo(f):
    f.write('<ul>\n')
    try:
        for n in fibo():
            tag = HtmlTag(f, 4, 'li')
            yield n
            tag = None       # doesn?t call destructor here.
    finally:
        tag = None           # here, too.
        f.write('</ul>\n')

def write_file():
    f = open('fibo.txt', 'w')   # f.close does?nt call, too.
    for n in html_fibo(f):
        f.write('%d' % n)
        if n > 100:
            break

runing code and patch are the following link.(not pypy source code patch.)
https://gist.github.com/1117550

and is this problem in bugs.pypy.org? I can?nt found it.

Cheers.
Kenji

From eric at vanrietpaap.nl  Mon Aug  1 08:07:38 2011
From: eric at vanrietpaap.nl (Eric van Riet Paap)
Date: Mon, 1 Aug 2011 08:07:38 +0200
Subject: [pypy-dev] Status of ARM backend
Message-ID: <CAFKq6r+P+mUsSN4g59cTVtLi5y3jJKrG-jX8-nmJNFuWD+gxOA@mail.gmail.com>

Hi,

Can someone please give us an update on the status of the ARM backend.
Is it time already to get our iPhones and Androids out of our pockets
and start creating that killer app using PyPy?

cheers,
Eric

From anto.cuni at gmail.com  Mon Aug  1 09:59:00 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 01 Aug 2011 09:59:00 +0200
Subject: [pypy-dev] newbie needs pypy setup help
In-Reply-To: <878vreduux.fsf@pobox.com>
References: <j14dan$h8r$1@dough.gmane.org>	<CAK5idxQsz9Wk6fedaG1jR9kT0X6z1ha+pzvngUcRnvAjy1-=hQ@mail.gmail.com>	<CAK5idxTZEstim-aWjCpJ5VJDujei_FHLuH9=DWsEiAuYkOOmag@mail.gmail.com>	<CAGGBd_oJhoRs4VOK9f4AT5jc1zDDt5i_-CJ9H1DNfMJKaWstmw@mail.gmail.com>	<874o23fm69.fsf@pobox.com>
	<871ux7ffwt.fsf@pobox.com>	<20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain>	<CAGGBd_q1SX4xbyqmT6Rw1-q3gEe8XsBqReEkSQuFBbZZRNwxJA@mail.gmail.com>	<CAK5idxTXaESa9vy8OOYGGWLYvXxxFFLpzS6i1aJEqw6B96=keQ@mail.gmail.com>	<87mxfue0hl.fsf@pobox.com>	<CAK5idxSJtmDbApKLfYoRPmQPcXhLY=SwxGJ49rV_uqWTL8KjBQ@mail.gmail.com>	<87hb62dyko.fsf@pobox.com>	<CAK5idxQAUwc=ggDZF7yQnmCaH5K-a7O5OAKk__7gdQAmA6JZuQ@mail.gmail.com>	<87ei16dxq7.fsf@pobox.com>
	<878vreduux.fsf@pobox.com>
Message-ID: <4E365CC4.2060506@gmail.com>

Hi Tom,

On 31/07/11 23:05, Tom Roche wrote:
>> https://bugs.pypy.org/issue772
>
>> so the next release should resolve that problem for system wide installs.
>
> By "next release," do you mean 1.6? (BTW, to show bona fides, and since it needed done, I updated

yes, but it's also possible to download the nightly builds right now:
http://buildbot.pypy.org/nightly/trunk/

could you please check that the permission issue is actually fixed in the 
nightly tarballs?

ciao,
Anto

From anto.cuni at gmail.com  Mon Aug  1 10:05:09 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 01 Aug 2011 10:05:09 +0200
Subject: [pypy-dev] =?utf-8?b?aHRtbF9maWJvLnB54oCZcyBwcm9ibGVtLg==?=
In-Reply-To: <CAMLJXURbGCH7wsZ-bNh4okcGGO-XF9+=SidU=8WhQu2HNBBccw@mail.gmail.com>
References: <CAMLJXURbGCH7wsZ-bNh4okcGGO-XF9+=SidU=8WhQu2HNBBccw@mail.gmail.com>
Message-ID: <4E365E35.6020505@gmail.com>

On 01/08/11 07:25, Kenji Yano wrote:
> Hi
>
> I?found the cause of html_fibo.py at the EuroPython Hands-on.
> This case, HtmlTag destructor isn?t call when the object didn?t use.
> Is this garbage collection?s problem?
> (pypy-trunk(1.6.0-dev) same too.)

yes.  The difference with CPython is that CPython destroys the object as soon 
as it forgets the last reference to the object (and thus it is deterministic), 
while on PyPy the objects are destroyed only the the GC runs (which is not 
deterministic).

> runing code and patch are the following link.(not pypy source code patch.)
> https://gist.github.com/1117550

your patch works but it's suboptimal and slower than it need. The proper 
solution would be to add the appropriate calls to .close() to the file and to 
the generator (possibily using the with statement) and also to provide a 
method to HtmlTag to explicitly close the tag instead of relying on the __del__.

> and is this problem in bugs.pypy.org? I can?nt found it.

it works for me:
https://bugs.pypy.org/

you need to explicitly allow our custom SSL certificate, but that's all.

ciao,
Anto

From kenji.yano at gmail.com  Mon Aug  1 13:03:46 2011
From: kenji.yano at gmail.com (Kenji Yano)
Date: Mon, 1 Aug 2011 20:03:46 +0900
Subject: [pypy-dev] =?windows-1252?q?html=5Ffibo=2Epy=92s_problem=2E?=
In-Reply-To: <4E365E35.6020505@gmail.com>
References: <CAMLJXURbGCH7wsZ-bNh4okcGGO-XF9+=SidU=8WhQu2HNBBccw@mail.gmail.com>
	<4E365E35.6020505@gmail.com>
Message-ID: <CAMLJXUQea4O8jek5aY_p62UCfnoqGO_ZQtVzHWr=cuAaigS41Q@mail.gmail.com>

Hi, Anto

Thanks for your reply.

2011/8/1 Antonio Cuni <anto.cuni at gmail.com>:
> On 01/08/11 07:25, Kenji Yano wrote:
>>
>> Hi
>>
>> I?found the cause of html_fibo.py at the EuroPython Hands-on.
>> This case, HtmlTag destructor isn?t call when the object didn?t use.
>> Is this garbage collection?s problem?
>> (pypy-trunk(1.6.0-dev) same too.)
>
> yes.  The difference with CPython is that CPython destroys the object as
> soon as it forgets the last reference to the object (and thus it is
> deterministic), while on PyPy the objects are destroyed only the the GC runs
> (which is not deterministic).

OK, I understand.

>
>> runing code and patch are the following link.(not pypy source code patch.)
>> https://gist.github.com/1117550
>
> your patch works but it's suboptimal and slower than it need. The proper
> solution would be to add the appropriate calls to .close() to the file and
> to the generator (possibily using the with statement) and also to provide a
> method to HtmlTag to explicitly close the tag instead of relying on the
> __del__.

yes, I know. I wanted to say gc doesn't work. That's the only reason.

>> and is this problem in bugs.pypy.org? I can?nt found it.
>
> it works for me:
> https://bugs.pypy.org/
>
> you need to explicitly allow our custom SSL certificate, but that's all.
> ciao,
> Anto
>

I created new issue, because pypy have compatibility problem.
but, I learned that pypy has different garbage collection strategies.
so I remove that issue.

Cheers
Kenji

From Tom_Roche at pobox.com  Mon Aug  1 15:56:45 2011
From: Tom_Roche at pobox.com (Tom Roche)
Date: Mon, 01 Aug 2011 09:56:45 -0400
Subject: [pypy-dev] issue772 install permissions fix,
 was:  newbie needs pypy setup help
In-Reply-To: <4E365CC4.2060506@gmail.com>
References: <4E365CC4.2060506@gmail.com>
Message-ID: <87tya1fd5u.fsf@pobox.com>


Janzert Sun, 31 Jul 2011 16:18:32 -0400
>> https://bugs.pypy.org/issue772

Antonio Cuni Mon, 01 Aug 2011 09:59:00 +0200
> could you please check that the permission issue is actually fixed in
> the nightly tarballs?

It is currently: the tail of the output from

BUILD_NAME="pypy-c-jit-latest-linux64"
BUILD_TAG="pypy-c-jit-46097-54515dec1d20-linux64"
URI="http://buildbot.pypy.org/nightly/trunk/${BUILD_NAME}.tar.bz2"
TMP_DIR_ROOT="/tmp/pypy/${BUILD_TAG}"
TARGET_DIR_ROOT="/opt/${BUILD_NAME}"
TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy"
USR_PYPY_EXEC="/usr/local/bin/pypy"
#  "sudo chmod -R go+rX ${TARGET_DIR_ROOT}/" \
for CMD in \
  "rm -fr $(dirname ${TMP_DIR_ROOT})" \
  "mkdir -p ${TMP_DIR_ROOT}" \
  "pushd $(dirname ${TMP_DIR_ROOT})" \
  "wget -O - ${URI} | tar xvjf -" \
  "ls -alh" \
  "popd" \
  "sudo rm -fr ${TARGET_DIR_ROOT}" \
  "sudo mkdir -p ${TARGET_DIR_ROOT}" \
  "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \
  "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \
  "which pypy" \
  "sudo ls -al ${USR_PYPY_EXEC}" \
  "${USR_PYPY_EXEC} --version" \
  "sudo ls -al ${TARGET_DIR_ROOT}" \
  "pypy --version" \
  "pypy" \
  "popd" \
; do
  echo -e "$ ${CMD}"
  eval "${CMD}"
done

is

> $ sudo ls -al /opt/pypy-c-jit-latest-linux64
> total 40
> drwxr-xr-x 7 root root 4096 2011-08-01 09:47 .
> drwxr-xr-x 9 root root 4096 2011-08-01 09:47 ..
> drwxr-xr-x 2 root root 4096 2011-08-01 09:47 bin
> drwxr-xr-x 2 root root 4096 2011-08-01 09:47 include
> drwxr-xr-x 7 root root 4096 2011-08-01 09:47 lib_pypy
> drwxr-xr-x 4 root root 4096 2011-08-01 09:47 lib-python
> -rw-r--r-- 1 root root 7462 2011-08-01 09:47 LICENSE
> -rw-r--r-- 1 root root  735 2011-08-01 09:47 README
> drwxr-xr-x 2 root root 4096 2011-08-01 09:47 site-packages
> $ pypy --version
> Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34)
> [PyPy 1.5.0-alpha0 with GCC 4.4.3]
> $ pypy
> Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34)
> [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> And now for something completely different: ``don't trip on the power cord''
> >>>>

HTH, Tom Roche <Tom_Roche at pobox.com>

From cfbolz at gmx.de  Tue Aug  2 10:20:31 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 2 Aug 2011 10:20:31 +0200
Subject: [pypy-dev] unroll if
Message-ID: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>

hi Alex, hi all,

I don't have much time for things, about to leave on vacation. Just so
that it doesn't get lost: on the unrolling-if branch, if you use the
decorator, the JIT should *not* look inside the function if the
condition if false. even if the function doesn't contain a loop. so
maybe the decorators should be renamed to unroll_iff (too
mathematical?) or unroll_exactly_if.

Cheers,

Carl Friedrich

From arigo at tunes.org  Tue Aug  2 11:41:23 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 2 Aug 2011 11:41:23 +0200
Subject: [pypy-dev] unroll if
In-Reply-To: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>
References: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>
Message-ID: <CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>

Hi Carl,

On Tue, Aug 2, 2011 at 10:20 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> I don't have much time for things, about to leave on vacation. Just so
> that it doesn't get lost: on the unrolling-if branch, if you use the
> decorator, the JIT should *not* look inside the function if the
> condition if false. even if the function doesn't contain a loop. so
> maybe the decorators should be renamed to unroll_iff (too
> mathematical?) or unroll_exactly_if.

Then it has nothing to do with unrolling: it should be renamed to
something like "look_inside_if".  It just happens to mean, as a
side-effect, "if the condition is true, then look inside even in case
there is a loop, thus unrolling it".


A bient?t,

Armin.

From anto.cuni at gmail.com  Tue Aug  2 12:05:34 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 02 Aug 2011 12:05:34 +0200
Subject: [pypy-dev] unroll if
In-Reply-To: <CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>
References: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>
	<CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>
Message-ID: <4E37CBEE.5070307@gmail.com>

On 02/08/11 11:41, Armin Rigo wrote:
> Then it has nothing to do with unrolling: it should be renamed to
> something like "look_inside_if".  It just happens to mean, as a
> side-effect, "if the condition is true, then look inside even in case
> there is a loop, thus unrolling it".

look_inside_and_maybe_unroll_if?

From cfbolz at gmx.de  Tue Aug  2 12:00:20 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 02 Aug 2011 12:00:20 +0200
Subject: [pypy-dev] unroll if
In-Reply-To: <CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>
References: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>
	<CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>
Message-ID: <4E37CAB4.20304@gmx.de>

On 08/02/2011 11:41 AM, Armin Rigo wrote:
> Hi Carl,
>
> On Tue, Aug 2, 2011 at 10:20 AM, Carl Friedrich Bolz<cfbolz at gmx.de>  wrote:
>> I don't have much time for things, about to leave on vacation. Just so
>> that it doesn't get lost: on the unrolling-if branch, if you use the
>> decorator, the JIT should *not* look inside the function if the
>> condition if false. even if the function doesn't contain a loop. so
>> maybe the decorators should be renamed to unroll_iff (too
>> mathematical?) or unroll_exactly_if.
>
> Then it has nothing to do with unrolling: it should be renamed to
> something like "look_inside_if".  It just happens to mean, as a
> side-effect, "if the condition is true, then look inside even in case
> there is a loop, thus unrolling it".

look_inside_iff then? :-)

Carl Friedrich

From alex.gaynor at gmail.com  Tue Aug  2 19:35:52 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 2 Aug 2011 10:35:52 -0700
Subject: [pypy-dev] unroll if
In-Reply-To: <4E37CAB4.20304@gmx.de>
References: <CADLBHoSODFdALqUfqmQ7F0OZ17mcs-BbWjYAeLBGbsmua3wtyg@mail.gmail.com>
	<CAMSv6X2zOTrqf-Qq1XT1CsXQzUN1-6oJXnfPn=VdyF+9RKvc7A@mail.gmail.com>
	<4E37CAB4.20304@gmx.de>
Message-ID: <CAFRnB2XOaL1AJswZer--oLPL33L7iWGp4t8TMkXWZiA561uKCw@mail.gmail.com>

On Tue, Aug 2, 2011 at 3:00 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> On 08/02/2011 11:41 AM, Armin Rigo wrote:
>
>> Hi Carl,
>>
>> On Tue, Aug 2, 2011 at 10:20 AM, Carl Friedrich Bolz<cfbolz at gmx.de>
>>  wrote:
>>
>>> I don't have much time for things, about to leave on vacation. Just so
>>> that it doesn't get lost: on the unrolling-if branch, if you use the
>>> decorator, the JIT should *not* look inside the function if the
>>> condition if false. even if the function doesn't contain a loop. so
>>> maybe the decorators should be renamed to unroll_iff (too
>>> mathematical?) or unroll_exactly_if.
>>>
>>
>> Then it has nothing to do with unrolling: it should be renamed to
>> something like "look_inside_if".  It just happens to mean, as a
>> side-effect, "if the condition is true, then look inside even in case
>> there is a loop, thus unrolling it".
>>
>
> look_inside_iff then? :-)
>
> Carl Friedrich
>
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>

Any of the above are fine with me :), I'll rename it when I next work on it.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110802/14c3fa47/attachment.html>

From laurie at tratt.net  Tue Aug  2 20:14:40 2011
From: laurie at tratt.net (Laurence Tratt)
Date: Tue, 2 Aug 2011 19:14:40 +0100
Subject: [pypy-dev] Porting PyPy to OpenBSD/amd64
Message-ID: <20110802181440.GB31466@phase.tratt.net>

Hello all,

I'm looking to port PyPy to OpenBSD (specifically amd64/x86_64, as I no
longer have any i386 machines; that might be the cause of some of my woes).
Getting much of PyPy to run on OpenBSD/amd64 is simple - partly basing things
on the FreeBSD parts - and compilation on my laptop merrily steams ahead for
half an hour or more. It then stumbles over time types which, in terms of
RPython / PyPy, I must admit ignorance. Any help is much appreciated!

My current problem centres around pypy/module/rctime/interp_time.py. The
offending code is here:

  t = (((c_time(lltype.nullptr(rffi.TIME_TP.TO))) / YEAR) * YEAR)
  # we cannot have reference to stack variable, put it on the heap
  t_ref = lltype.malloc(rffi.TIME_TP.TO, 1, flavor='raw')
  t_ref[0] = t
  p = c_localtime(t_ref)

That last line (#210) then causes this error message:

  E         TypeError: <Array of INT > items:
  E         expect <INT>
  E            got <Signed>

This is surprising because, naively, all the types in interp_time.py seem to
match up with what I'd expected. On line 320 of that module there is
something which gave me an idea. So if I change the first line of that
excerpt to:

  t = rffi.r_time_t((((c_time(lltype.nullptr(rffi.TIME_TP.TO))) / YEAR) * YEAR))

I don't get the error. Is it the right patch? Dunno. But, if it is, I doubt
it's OpenBSD specific (is it a 64 bit thing perhaps?).

However, things then die a little later:

  [translation:ERROR]  TyperError: don't know how to convert from <FloatRepr Float> to <IntegerRepr INT>
  [translation:ERROR] .. (pypy.module.rctime.interp_time:287)_get_inttime
  [translation:ERROR] .. block at 121 with 1 exits
  [translation:ERROR] .. v1 = simple_call((type r_INT), v0)

So I'm not sure if this is a one-off problem; or if the wrong types are
being used in other places too; or if PyPy is getting confused by some of
OpenBSD's types; or... who knows. Unfortunately the backtraces I get are in
the guts of RPython, and only seem to tell me the point of the error, and
not the full backtrace that led to it - I'm not sure what function called
_get_inttime.

If anyone has any suggestions, I'm all ears!


Laurie
-- 
http://tratt.net/laurie/ -- Personal
http://fetegeo.org/      -- Free text geocoding
http://convergepl.org/   -- The Converge programming language

From arigo at tunes.org  Tue Aug  2 20:51:50 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 2 Aug 2011 20:51:50 +0200
Subject: [pypy-dev] Porting PyPy to OpenBSD/amd64
In-Reply-To: <20110802181440.GB31466@phase.tratt.net>
References: <20110802181440.GB31466@phase.tratt.net>
Message-ID: <CAMSv6X3QGdOtRxU-1a0L4beSagFud7YHj5vStHpjLTJVDRvuAg@mail.gmail.com>

Hi Laurie,

On Tue, Aug 2, 2011 at 8:14 PM, Laurence Tratt <laurie at tratt.net> wrote:
> I'm looking to port PyPy to OpenBSD (specifically amd64/x86_64, as I no
> longer have any i386 machines; that might be the cause of some of my woes).

I have fixed the problems you reported (see 109f80dac1c1).  I cannot
say if there will be more, though.  These are caused by a different
combination of 32- versus 64-bit integer types than on Linux64 and
MacOSX64.  Feel free to report again here or on our irc channel, #pypy
at irc.freenode.net.


A bient?t,

Armin.

From Tom_Roche at pobox.com  Wed Aug  3 03:32:49 2011
From: Tom_Roche at pobox.com (Tom Roche)
Date: Tue, 02 Aug 2011 21:32:49 -0400
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
References: <CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
Message-ID: <8762mfffem.fsf@pobox.com>


summary: Maciej: please see link to patch for download.html rework (below). David: please see link to page (below) and critique the (very brief) section on installing from a package manager.

details:

Maciej Fijalkowski Sun, 31 Jul 2011 23:13:17 +0200
> I guess [the PyPy wiki situation] is a bit half-hearted.

Personally, I suspect you'd get more contributions from a wiki: HTML feels kinda heavyweight these days. E.g., I had to `tidy` download.html to work with it, then, after reworking the content, spent an astonishing amount of time twiddling div's in download.html to unbreak the "sidepic" buttons. (This may be mostly due to thickheadedness on my part :-)

> Ideally we would like a diff from pypy.org download page to what you
> want to put info there.

OK, the patch.txt is @

https://docs.google.com/leaf?id=0BzDAFHgIxRzKZjBkZDZkNjEtZTJhMi00MzlkLThiMjItMGFmNjY3ZmFkOGI2&hl=en_US&authkey=CNCjt7QK

and my version of the HTML is @

https://docs.google.com/leaf?id=0BzDAFHgIxRzKMjM0ZDNlNTEtZmYwMi00YmYyLTg5NmItMDc5MDlkOWFmODgw&hl=en_US

Of course the HTML won't format correctly standalone (notably without the CSS), but the content is readable (which is still what counts :-)

HTH, Tom Roche <Tom_Roche at pobox.com>

From arigo at tunes.org  Wed Aug  3 08:21:55 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 3 Aug 2011 08:21:55 +0200
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <8762mfffem.fsf@pobox.com>
References: <CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
Message-ID: <CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>

Hi Tom,

On Wed, Aug 3, 2011 at 3:32 AM, Tom Roche <Tom_Roche at pobox.com> wrote:
> Personally, I suspect you'd get more contributions from a wiki: HTML feels kinda heavyweight these days.

Did you actually edit the file download.html?  Bah, it looks like some
communication problem.  If you understood "you have to edit the .html
file by hand" then fijal was unclear :-(  We don't do that.  We edit
download.rst, which is a ReST file out of which the .html is
generated...


Armin

From fijall at gmail.com  Wed Aug  3 09:24:16 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 3 Aug 2011 09:24:16 +0200
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
References: <CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
Message-ID: <CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>

On Wed, Aug 3, 2011 at 8:21 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Tom,
>
> On Wed, Aug 3, 2011 at 3:32 AM, Tom Roche <Tom_Roche at pobox.com> wrote:
>> Personally, I suspect you'd get more contributions from a wiki: HTML feels kinda heavyweight these days.
>
> Did you actually edit the file download.html? ?Bah, it looks like some
> communication problem. ?If you understood "you have to edit the .html
> file by hand" then fijal was unclear :-( ?We don't do that. ?We edit
> download.rst, which is a ReST file out of which the .html is
> generated...
>

I think it's download.txt

>
> Armin
>

From anto.cuni at gmail.com  Wed Aug  3 10:06:23 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 03 Aug 2011 10:06:23 +0200
Subject: [pypy-dev] [pypy-commit] pypy unroll-if-alt: a) Renamed
 @unroll_if to @look_inside_iff at carl's request
In-Reply-To: <20110803001617.8E45A822AB@wyvern.cs.uni-duesseldorf.de>
References: <20110803001617.8E45A822AB@wyvern.cs.uni-duesseldorf.de>
Message-ID: <4E39017F.70305@gmail.com>

On 03/08/11 02:16, alex_gaynor wrote:
> Author: Alex Gaynor<alex.gaynor at gmail.com>
> Branch: unroll-if-alt
> Changeset: r46219:524483d98407
> Date: 2011-08-02 17:17 -0700
> http://bitbucket.org/pypy/pypy/changeset/524483d98407/
>
> Log:	a) Renamed @unroll_if to @look_inside_iff at carl's request b)
> 	Unroll Argument.unpack if one should. c) Leave a comment about a
> 	small level mess with keyword argumengs and the decorator
>
> diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
> --- a/pypy/interpreter/argument.py
> +++ b/pypy/interpreter/argument.py
> @@ -125,6 +125,7 @@
>
>       ###  Manipulation  ###
>
> +    @jit.look_inside_iff(lambda self: not self._dont_jit)
>       def unpack(self): # slowish
>           "Return a ([w1,w2...], {'kw':w3...}) pair."
>           kwds_w = {}
> @@ -245,6 +246,8 @@

I think that this should have a test_pypy_c test.

ciao,
Anto

From Tom_Roche at pobox.com  Wed Aug  3 16:50:39 2011
From: Tom_Roche at pobox.com (Tom Roche)
Date: Wed, 03 Aug 2011 10:50:39 -0400
Subject: [pypy-dev] maintaining pypy.org docs,
	was: PyPy download/install page
In-Reply-To: <CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
References: <CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
	<CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
Message-ID: <87vcuepn0g.fsf@pobox.com>


Tom Roche Wed, Aug 3, 2011 at 3:32 AM
>>> I suspect you'd get more contributions from a wiki: HTML feels kinda
>>> heavyweight these days.

and there are so many fewer gotchas !-(

Armin Rigo Wed, Aug 3, 2011 at 8:21 AM
>> Did you actually edit the file download.html? Bah, it looks like some
>> communication problem. If you understood "you have to edit the .html
>> file by hand" then fijal was unclear

"Unclear"? How 'bout "unduly terse"? Or, "did not mention *anything* about the following:"

>> We edit download.rst, which is a ReST file out of which the .html is
>> generated...

Maciej Fijalkowski Wed, 3 Aug 2011 09:24:16 +0200
> I think it's download.txt

It's

pypy.org/source/download.txt

about which there is exactly *nothing* (am I missing something?) in

pypy.org/

!!! So here's my suggestion: while I get yatiblog (see below--or has that part of the process changed?) and edit

pypy.org/source/download.txt

, how 'bout youse do 2 things:

1 edit/commit the attached

pypy.org/README
> DO NOT EDIT any of

> pypy.org/*.html

> !!! Those files are all generated from

> pypy.org/source/*.txt

> so edit those. For more details about the process, see

> pypy.org/source/README

  for the benefit of the next person who tries to help with the doc. Please add whatever a reasonably-skilled person new to your process would need to know--and then, maintain it when your process changes! </sermonette>

2 (unless I'm missing something) fix

pypy.org/source/README
> You can get necessary software by doing:

> 	pip install yatiblog

> (tested with version 1.0) you can do:

>     yatiblog -o ..

> you'll get html output in the parent directory. you need an account
> Then you can check it in, login to pypy at pypy.org
> and go to pypy.org/htdocs/ and type "hg pull -u".

  I say "fix" because, minimally, there is no

pypy.org/htdocs/

  that I can see @

https://bitbucket.org/pypy/pypy.org/src

  (The only subdirectories are {css, image, js, source}--am I missing something?) Maximally, I don't know what else has changed since pypy.org/source/README was committed: just make it accurate enough.

FWIW, Tom Roche <Tom_Roche at pobox.com>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: README
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110803/127926cc/attachment.ksh>

From alex.gaynor at gmail.com  Wed Aug  3 19:12:28 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 3 Aug 2011 10:12:28 -0700
Subject: [pypy-dev] [pypy-commit] pypy unroll-if-alt: a) Renamed
 @unroll_if to @look_inside_iff at carl's request
In-Reply-To: <4E39017F.70305@gmail.com>
References: <20110803001617.8E45A822AB@wyvern.cs.uni-duesseldorf.de>
	<4E39017F.70305@gmail.com>
Message-ID: <CAFRnB2ULJQ4deyt+z7YE5w9+QiTVGYXWepgOmaoJ4R01mrtyow@mail.gmail.com>

On Wed, Aug 3, 2011 at 1:06 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> On 03/08/11 02:16, alex_gaynor wrote:
>
>> Author: Alex Gaynor<alex.gaynor at gmail.com>
>> Branch: unroll-if-alt
>> Changeset: r46219:524483d98407
>> Date: 2011-08-02 17:17 -0700
>> http://bitbucket.org/pypy/**pypy/changeset/524483d98407/<http://bitbucket.org/pypy/pypy/changeset/524483d98407/>
>>
>> Log:    a) Renamed @unroll_if to @look_inside_iff at carl's request b)
>>        Unroll Argument.unpack if one should. c) Leave a comment about a
>>        small level mess with keyword argumengs and the decorator
>>
>> diff --git a/pypy/interpreter/argument.py b/pypy/interpreter/argument.py
>> --- a/pypy/interpreter/argument.py
>> +++ b/pypy/interpreter/argument.py
>> @@ -125,6 +125,7 @@
>>
>>      ###  Manipulation  ###
>>
>> +    @jit.look_inside_iff(lambda self: not self._dont_jit)
>>      def unpack(self): # slowish
>>          "Return a ([w1,w2...], {'kw':w3...}) pair."
>>          kwds_w = {}
>> @@ -245,6 +246,8 @@
>>
>
> I think that this should have a test_pypy_c test.
>
> ciao,
> Anto
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>

Probably, or I can revert it.  Currently in pyjitpl a call wipes all heap
caches in the frontend, so I wanted to remove that call so that something
would be properly constant early enough, unfortunately I forgot creating a
dict is a call of it's own, so it doesn't do anything useful ATM.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110803/0e7a3db0/attachment.html>

From Tom_Roche at pobox.com  Sat Aug  6 14:40:28 2011
From: Tom_Roche at pobox.com (Tom Roche)
Date: Sat, 06 Aug 2011 08:40:28 -0400
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
References: <87vcuepn0g.fsf@pobox.com>
	<CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
	<CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
Message-ID: <874o1u1znn.fsf@pobox.com>


Armin Rigo Wed, Aug 3, 2011 at 8:21 AM
> We edit [pypy.org/source/download.txt]

Done! The new'n'improved download.txt file is @

https://docs.google.com/leaf?id=0BzDAFHgIxRzKYzQzNzUwYTktNzBjMS00ZmNmLTk3YzItYThhNTdmY2Q3OWM1&hl=en_US

and the patch file is @

https://docs.google.com/leaf?id=0BzDAFHgIxRzKZmQzZWZkNDYtODE2YS00MjUwLTgxMDYtZDdiYzVmMmE5ODJh&hl=en_US

HTH, Tom Roche <Tom_Roche at pobox.com>

From tobami at googlemail.com  Sun Aug  7 13:35:19 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Sun, 7 Aug 2011 13:35:19 +0200
Subject: [pypy-dev] PyPy at last infinitely fast
In-Reply-To: <4E313989.3030907@gmail.com>
References: <CAGf+9Vw8W1jn61TMnwXNrgB5=xFRfenaStshB57-jQ7pqbX4zw@mail.gmail.com>
	<4E248CCC.6090800@gmail.com>
	<CAGf+9VwO+w2gP7MXPapHJUWTvq_8rcUykG5_BJ-HZkrVyPVTEA@mail.gmail.com>
	<4E2532FB.4030006@gmail.com>
	<CAGf+9VyYo2HKzC2LOC31XaKh1SN3EACd2Shbgbq_3Z+SmDcX5w@mail.gmail.com>
	<CAK5idxQJD+AVKK3b6LAkVtgBvtX=+_D-CptFhf8DuoywG70AxA@mail.gmail.com>
	<CAGf+9Vy8zEozk-7_g3Ah8FpczFqMdwWpUW8zJD8+1b4aTVawHg@mail.gmail.com>
	<4E27D09B.8040204@gmail.com>
	<CAK5idxQeqbw7F4sgKWVDVUK79ZQWARU=JK=XVwFZAtZLw_NWOA@mail.gmail.com>
	<4E27D447.3070807@gmail.com>
	<CAGf+9VztHgbYkhJmFn94zkOOMy0MxRNMHrV=sQy5F1w+UnJoVw@mail.gmail.com>
	<4E2812FE.8000407@gmail.com>
	<CAK5idxT9ZOHjPWjrr-189713cL-+OB2VPKniBbQE8yHdSDKZzg@mail.gmail.com>
	<CAGf+9VxCQswhm+e8afq51PehonPi41rfVYMwmzWaxu+_is36bQ@mail.gmail.com>
	<CAGf+9VzpYaFv3hOvWmSnAm5Zv7unwytTvYW0N9G1ozMxnj776g@mail.gmail.com>
	<4E2FDDE4.3050109@gmail.com>
	<CAGf+9VwEXH00Mue1qSHOSVp0b5-XwUwiq2phKoOvmMQq4_BNEQ@mail.gmail.com>
	<4E313989.3030907@gmail.com>
Message-ID: <CAGf+9Vx61vcb3n-0vU6WTxKMWB1H3RQ-xOA3==B7qXeGvyXSnA@mail.gmail.com>

Hi Antonio,

I didn't answer before because I was on holidays.

> I can see that the function finds the last_results to compare with simply by
> doing last_revs[1].  What about changing this logic to pick the latest rev
> which actually contains at least one result for the current environment?

That would be a quick workaround, yes, but we would quickly get into
trouble. How would you then know which revision to use to compute the
trend? A naive approach would be:
trend_rev = last_rev_with_data - 10 * number_of_revs_without_data (in
the current case 1 for 32 bit, 1 for 64 bit, which means 1)

Another would be
trend_rev = last_rev_with_data - 10 - number_of_revs_until_data_is_found

Those two revisions may or may not be the same.

So if there is no guarantee that every revision saved has data for
every executable in the project, there will be a random number of
executable groups (grouped by revision that is benchmarked). It *is*
solvable, but it can quickly become a mess. And we may want to show
other data or statistics in the future other than the trend.

That is the reason I suggest using different projects when different
executables will be using different revisions to do the benchmarking.
I can be proven wrong, however, and the workaround can be implemented
for changes, trend, and rev list without a performance penalty.

> uhm, I don't see any option to select a different project from within
> speed.pypy.org.  Is it simply because there is only one? Or having another
> project would mean to visit a complete different webpage?

First, speed.pypy.org only has a single project: PyPy (cpython exes
are under a CPython project, but they are not shown in the changes or
timeline view).
Second, even with several projects you will currently only see a list
of executables. Improving that is a theme for the GSoC

I will be able to work on this next week, so I maybe we can meet on
IRC and have a go at it? (and close this huge email thread ;-)


2011/7/28 Antonio Cuni <anto.cuni at gmail.com>:
> Hi Miquel,
>
> On 28/07/11 11:04, Miquel Torres wrote:
>>
>> Hi Antonio,
>>
>> I'm afraid you are right, the solution I proposed makes no sense.
>> Sorry I gave you a wrong answer. A revision is unique for a project
>> (well, now to a branch of a project), and thus they are not separated
>> by environment. Codespeed was not really designed with revisions in
>> mind that sometimes have results, and sometimes not. To solve that,
>> revisions would need to depend on an executable as well, or introduce
>> a check that so that the revision list is tailored to a particular
>> exe, but it would be ugly.
>
> I can think of a semi-ugly workaround, I don't know how close it is to a
> working solution.
>
> If I understand correctly, the numbers which are displayed in the "changes"
> page are precomputed and saved into "Reports" by
> create_report_if_enough_data.
>
> I can see that the function finds the last_results to compare with simply by
> doing last_revs[1]. ?What about changing this logic to pick the latest rev
> which actually contains at least one result for the current environment?
>
>> There is a way though to "solve" it right now. Separate executables
>> into two different projects. pypy32, pypy64, instead of different
>> environements. The revision list shown does change on-the-fly
>> depending on the project the selected exe belongs to.
>
> uhm, I don't see any option to select a different project from within
> speed.pypy.org. ?Is it simply because there is only one? Or having another
> project would mean to visit a complete different webpage?
>
> Moreover, I wonder how this problem relates to the upcoming
> speed.python.org: will every interpreter (cpython, pypy, jython, etc.) be a
> separate project? Will it be possible to compare results of different
> interpreters?
>
> ciao,
> Anto
>

From josh.ayers at gmail.com  Tue Aug  9 05:07:30 2011
From: josh.ayers at gmail.com (Josh Ayers)
Date: Mon, 8 Aug 2011 20:07:30 -0700
Subject: [pypy-dev] file reading/writing speed
Message-ID: <CACOB4aNUx9qN3EBfcGoLrGbUnRkx7ti3yr6ka2PBmUqszhORbg@mail.gmail.com>

Sorry if this is a dumb question - I'm very new to PyPy, but also very
interested in using it in several applications.

I currently use PyTables and NumPy in a particular application, and I was
playing around with the idea of migrating it to PyPy.  I don't really need
the querying or compression capabilities of PyTables, and most processing of
the data is done in pure Python code, so I was hoping to see a speed-up with
PyPy.

I ran some tests yesterday on storing data on disk in flat binary files, and
PyPy was significantly slower than CPython.  I haven't done any comparisons
with PyTables yet.  Note that I'm running CPython 2.7.2 and PyPy 1.5 on a
rather old Windows XP machine.

I generated a 1 million element array using the built-in array module, wrote
it to disk, and then read it back in.  See http://pastie.org/2342676 for the
code.

Each operation was slower with PyPy than with CPython.

 * Array creation - CPython: 0.16s - PyPy: 0.47s
 * Writing file - CPython: 0.05s - PyPy: 0.11s
 * Reading file: CPython: 0.02s - PyPy: 0.08s

This method won't quite work for me in any case - I need to store 64 bit
integers, and the built-in array module doesn't support them.  To get around
that, I modified the pure-python array.py that comes in the pypy\lib_pypy
directory.  I added a "q" to the end of the line "TYPECODES = ..." which
represents a 64 bit signed integer within the struct module.  I saved that
modified file as array2.py and imported it in place of the built-in array.
See http://pastie.org/2342721 for the code.

That allowed me to use a 64 bit integer, but the array creation step was
again much slower on PyPy than it was on CPython.  The disk accessing steps
were more similar, and are probably at about the limit of the hard disk
anyway, but creating the array takes much longer under PyPy.

 * Array creation - CPython: 0.31s - PyPy: 1.42s
 * Writing file - CPython: 0.16s - PyPy: 0.25s
 * Reading file: CPython: 0.83s - PyPy: 0.13s

Any ideas on what could be causing this speed difference?  Am I doing
anything egregiously stupid in my code?  Any ideas on better methods for
efficiently storing and retrieving binary data from disk under PyPy?

Thanks in advance for your help.

Sincerely,
Josh Ayers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110808/1551d0e1/attachment.html>

From hakan at debian.org  Tue Aug  9 08:22:19 2011
From: hakan at debian.org (Hakan Ardo)
Date: Tue, 9 Aug 2011 08:22:19 +0200
Subject: [pypy-dev] file reading/writing speed
In-Reply-To: <CACOB4aNUx9qN3EBfcGoLrGbUnRkx7ti3yr6ka2PBmUqszhORbg@mail.gmail.com>
References: <CACOB4aNUx9qN3EBfcGoLrGbUnRkx7ti3yr6ka2PBmUqszhORbg@mail.gmail.com>
Message-ID: <CAEkL+RjGKQYEBb2R0mmJ-e_BrnfaAEe1GB2DX4bOA_qNhtHTxw@mail.gmail.com>

On Tue, Aug 9, 2011 at 5:07 AM, Josh Ayers <josh.ayers at gmail.com> wrote:
>
> I generated a 1 million element array using the built-in array module, wrote
> it to disk, and then read it back in.? See http://pastie.org/2342676 for the
> code.
>
> Each operation was slower with PyPy than with CPython.
>
> ?* Array creation - CPython: 0.16s - PyPy: 0.47s

Whats taking time here is to iterate over the range-list and
unwrapping all the integers. If all you want is to allocate an array
it's significantly faster (both on pypy and on cpython) to do:

    a = array.array(outputDataType,[0]) * dataSize

> ?* Writing file - CPython: 0.05s - PyPy: 0.11s
> ?* Reading file: CPython: 0.02s - PyPy: 0.08s

The builtin array module uses space.call_method(w_f, 'write') and
space.call_method(w_f, 'read') to implement fromfile and tofile. For
fromfile that means copying the data atleast once, and maybe that's
whats going on with tofile too. I dont know how hard it would be to
add some fast path for common cases that reads/writes data directly
into the array buffer?

>
> This method won't quite work for me in any case - I need to store 64 bit
> integers, and the built-in array module doesn't support them.? To get around
> that, I modified the pure-python array.py that comes in the pypy\lib_pypy
> directory.? I added a "q" to the end of the line "TYPECODES = ..." which
> represents a 64 bit signed integer within the struct module.? I saved that
> modified file as array2.py and imported it in place of the built-in array.
> See http://pastie.org/2342721 for the code.
>
> That allowed me to use a 64 bit integer, but the array creation step was
> again much slower on PyPy than it was on CPython.? The disk accessing steps
> were more similar, and are probably at about the limit of the hard disk
> anyway, but creating the array takes much longer under PyPy.

Why do you think this is limited by the harddisk? I would imagine this
approach to be slower than using the builtin module. Did you try this
approach with a datatype supported by the built in module to compare
the performance of the two approaches?

-- 
H?kan Ard?

From dje.gcc at gmail.com  Tue Aug  9 15:46:31 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Tue, 9 Aug 2011 09:46:31 -0400
Subject: [pypy-dev] PowerPC 64 bit support
Message-ID: <CAGWvnynceRZ0Wd+nNPtYpb9V-NHpqxt6gwcgrVUea5xwchoAxw@mail.gmail.com>

Sven,

Attached is a patch that adds 64 bit PowerPC instructions to ppcgen.
The backend builds with this patch.

I need to understand how to add tests that only run in 64 bit mode
because the instruction will fail on a 32 bit PowerPC system as
illegal instructions.

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff
Type: application/octet-stream
Size: 6861 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110809/c04f484f/attachment.obj>

From dje.gcc at gmail.com  Wed Aug 10 02:04:40 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Tue, 9 Aug 2011 20:04:40 -0400
Subject: [pypy-dev] PPC backend design questions
Message-ID: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>

The nop mnemonic is defined twice in ppc_assmbler.py:

In class PPCAssembler

    # F.9 Recommended Simplified Mnemonics

    nop = BA.ori(rS=0, rA=0, UIMM=0)

and in class PPCBuilder

    def nop(self):
        self.ori(0, 0, 0)

Which one should be removed?  I assume the one in PPCBuilder.


Some PPC64 instructions have complicated encodings that split operands
among multiple fields, such as shift immediate instructions, or
truncates fields, such as load and store displacements.  I implemented
the instructions exposing all of the fields, but that does not match
the way instructions are written in assembly language.  These aren't
simplified mnemonics.

For example,

    sradi 3, 4, 33

splits the shift count of the third operand between two fields in the
instruction representation.  Or

    ld 3, 0x8000(9)

does not place the displacement 0x8000 in the instruction, but shifts
the value by 2 bits -- displacements always are word aligned.  The
base address can be anything, so addresses themselves are not strictly
aligned, but the displacement is word aligned.  However, the
programmer does not write the shifted address.

My inclination is to implement the instructions the way they are
written by a programmer in PPCAssembler and change the names in
BasicPPCAssembler.  Is that reasonable and is there a precedent I
should use for the names?

Thanks, David

From sungsuha at gmail.com  Wed Aug 10 09:16:13 2011
From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha)
Date: Wed, 10 Aug 2011 07:16:13 +0000 (UTC)
Subject: [pypy-dev] Testing pypy
Message-ID: <loom.20110810T091341-451@post.gmane.org>

http://docs.python.org/devguide/runtests.html

Can I run tests on pypy as above?

I downloaded the linux binary version for the pypy website
(https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2)

running
$./pypy -m test

prints 
"
 No module named test.__main__; 'test' is a package and cannot be directly 
executed
"

(without a new line at the end!)

Is this expected? Am I doing something wrong here?


From arigo at tunes.org  Wed Aug 10 10:21:48 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 10 Aug 2011 10:21:48 +0200
Subject: [pypy-dev] Testing pypy
In-Reply-To: <loom.20110810T091341-451@post.gmane.org>
References: <loom.20110810T091341-451@post.gmane.org>
Message-ID: <CAMSv6X0mG4AgooPVdUqaF2rOcqqWGC1f6wKDOyr9gRAqjf13AA@mail.gmail.com>

Hi,

On Wed, Aug 10, 2011 at 9:16 AM, Seung Soo, <sungsuha at gmail.com> wrote:
> ?No module named test.__main__; 'test' is a package and cannot be directly
> executed
>
> (without a new line at the end!)

(I fixed the missing new line.)

But PyPy behaves like Python 2.7.  I don't know if you are reading the
docs for Python 3.x, but it doesn't work this way in Python 2.7
either.


A bient?t,

Armin.

From micahel at gmail.com  Wed Aug 10 10:28:18 2011
From: micahel at gmail.com (Michael Hudson-Doyle)
Date: Wed, 10 Aug 2011 09:28:18 +0100
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
Message-ID: <CAFXevfe9Uud94JipHO2MiZ66DdAEmTDanHSyvEPKuwo4=a+pxw@mail.gmail.com>

(sorry, forgot reply to all on the first version of this)

I guess a useful thing to understand here is that the powerpc
assembler was mostly written by me on a whim many many years ago --
there never was a really strong design sense and if there was, I've
probably forgotten it :)

On 10 August 2011 01:04, David Edelsohn <dje.gcc at gmail.com> wrote:
> The nop mnemonic is defined twice in ppc_assmbler.py:
>
> In class PPCAssembler
>
> ? ?# F.9 Recommended Simplified Mnemonics
>
> ? ?nop = BA.ori(rS=0, rA=0, UIMM=0)
>
> and in class PPCBuilder
>
> ? ?def nop(self):
> ? ? ? ?self.ori(0, 0, 0)
>
> Which one should be removed? ?I assume the one in PPCBuilder.

However, I do remember that there was a reason I did this, although
not what.  Take out the version in PPCBuilder and see if anything
breaks, I guess.

> Some PPC64 instructions have complicated encodings that split operands
> among multiple fields, such as shift immediate instructions, or
> truncates fields, such as load and store displacements. ?I implemented
> the instructions exposing all of the fields, but that does not match
> the way instructions are written in assembly language. ?These aren't
> simplified mnemonics.
>
> For example,
>
> ? ?sradi 3, 4, 33
>
> splits the shift count of the third operand between two fields in the
> instruction representation. ?Or
>
> ? ?ld 3, 0x8000(9)
>
> does not place the displacement 0x8000 in the instruction, but shifts
> the value by 2 bits -- displacements always are word aligned. ?The
> base address can be anything, so addresses themselves are not strictly
> aligned, but the displacement is word aligned. ?However, the
> programmer does not write the shifted address.
>
> My inclination is to implement the instructions the way they are
> written by a programmer in PPCAssembler and change the names in
> BasicPPCAssembler. ?Is that reasonable and is there a precedent I
> should use for the names?

There are already fields that behave in unusual ways -- see

https://bitbucket.org/pypy/pypy/src/f8c92c646275/pypy/jit/backend/ppc/ppcgen/ppc_field.py#cl-55

I think you should be able to add your fields in this fashion -- in
fact, the IField looks a teensy bit like how the displacement field
should be already.  The split encodings should be fairly easy too,
although there might be some paranoid code checking that fields don't
overlap that you need to change.

Cheers,
mwh

From sungsuha at gmail.com  Wed Aug 10 11:35:36 2011
From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha)
Date: Wed, 10 Aug 2011 09:35:36 +0000 (UTC)
Subject: [pypy-dev] Testing pypy
References: <loom.20110810T091341-451@post.gmane.org>
	<CAMSv6X0mG4AgooPVdUqaF2rOcqqWGC1f6wKDOyr9gRAqjf13AA@mail.gmail.com>
Message-ID: <loom.20110810T113354-170@post.gmane.org>

Armin Rigo <arigo <at> tunes.org> writes:

> 
> Hi,
> But PyPy behaves like Python 2.7.  I don't know if you are reading the
> docs for Python 3.x, but it doesn't work this way in Python 2.7
> either.
> 
> A bient?t,
> 
> Armin.
> 


Sorry. I was reading the 3.x docs.

Anyway I'd like to know if there is any single way to test both 
python and pypy(2.7), where the results should be the same.




From arigo at tunes.org  Wed Aug 10 13:49:03 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 10 Aug 2011 13:49:03 +0200
Subject: [pypy-dev] Testing pypy
In-Reply-To: <loom.20110810T113354-170@post.gmane.org>
References: <loom.20110810T091341-451@post.gmane.org>
	<CAMSv6X0mG4AgooPVdUqaF2rOcqqWGC1f6wKDOyr9gRAqjf13AA@mail.gmail.com>
	<loom.20110810T113354-170@post.gmane.org>
Message-ID: <CAMSv6X3Ac3C9mD2yMQikc0e1LWWuynS9+=7pzFt8E0GqdoiAaQ@mail.gmail.com>

Hi Seung,

On Wed, Aug 10, 2011 at 11:35 AM, Seung Soo, <sungsuha at gmail.com> wrote:
> Anyway I'd like to know if there is any single way to test both
> python and pypy(2.7), where the results should be the same.

We use our own custom runner, which you run differently than
CPython's.  Well, you can also try running "python -m test.regrtest"
and "pypy -m test.regrtest", which works in both cases -- or at least
"almost works" on top of PyPy.  The issue is that this will try to run
all CPython tests, including the few ones that we normally skip; in
particular there is test_capi that tests the C API, which is not
complete on top of PyPy yet, so you get a fatal AssertionError.


A bient?t,

Armin.

From dje.gcc at gmail.com  Wed Aug 10 15:23:29 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Wed, 10 Aug 2011 09:23:29 -0400
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
Message-ID: <CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>

On Wed, Aug 10, 2011 at 4:26 AM, Michael Hudson-Doyle <micahel at gmail.com> wrote:

> There are already fields that behave in unusual ways -- see
>
> https://bitbucket.org/pypy/pypy/src/f8c92c646275/pypy/jit/backend/ppc/ppcgen/ppc_field.py#cl-55
>
> I think you should be able to add your fields in this fashion -- in
> fact, the IField looks a teensy bit like how the displacement field
> should be already. ?The split encodings should be fairly easy too,
> although there might be some paranoid code checking that fields don't
> overlap that you need to change.

Yes and no.  The ds displacement field for 64 bit loads is similar to
the LI field of branches.  But the shift fields are not like the spr
field -- the value is split between multiple fields with different bit
ranges, not just permuted within the field.

Thanks, David

From micahel at gmail.com  Wed Aug 10 15:34:17 2011
From: micahel at gmail.com (Michael Hudson-Doyle)
Date: Wed, 10 Aug 2011 14:34:17 +0100
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
Message-ID: <CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>

On Aug 10, 2011 2:23 PM, "David Edelsohn" <dje.gcc at gmail.com> wrote:
>
> On Wed, Aug 10, 2011 at 4:26 AM, Michael Hudson-Doyle <micahel at gmail.com> wrote:
>
> > There are already fields that behave in unusual ways -- see
> >
> > https://bitbucket.org/pypy/pypy/src/f8c92c646275/pypy/jit/backend/ppc/ppcgen/ppc_field.py#cl-55
> >
> > I think you should be able to add your fields in this fashion -- in
> > fact, the IField looks a teensy bit like how the displacement field
> > should be already. ?The split encodings should be fairly easy too,
> > although there might be some paranoid code checking that fields don't
> > overlap that you need to change.
>
> Yes and no. ?The ds displacement field for 64 bit loads is similar to
> the LI field of branches. ?But the shift fields are not like the spr
> field -- the value is split between multiple fields with different bit
> ranges, not just permuted within the field.

Well sure, but at the end of the day the instruction is assembled by
doing roughly this:

insn = 0
for field in fields:
    insn |= field.encode()

-- nothing prevents a field returning a value that is split between
multiple bit ranges.  There is a bit of scaffolding that assumes
contiguous bit ranges but nothing fundamental (and as I said, I think
this is mostly for checking that fields don't overlap, and there would
be other ways of doing that).

Cheers,
mwh

From chris at kateandchris.net  Wed Aug 10 15:47:46 2011
From: chris at kateandchris.net (Chris Lambacher)
Date: Wed, 10 Aug 2011 09:47:46 -0400
Subject: [pypy-dev] Testing pypy
In-Reply-To: <CAMSv6X3Ac3C9mD2yMQikc0e1LWWuynS9+=7pzFt8E0GqdoiAaQ@mail.gmail.com>
References: <loom.20110810T091341-451@post.gmane.org>
	<CAMSv6X0mG4AgooPVdUqaF2rOcqqWGC1f6wKDOyr9gRAqjf13AA@mail.gmail.com>
	<loom.20110810T113354-170@post.gmane.org>
	<CAMSv6X3Ac3C9mD2yMQikc0e1LWWuynS9+=7pzFt8E0GqdoiAaQ@mail.gmail.com>
Message-ID: <CAAXXHgKFi14w8_r0Mq_nGvbfYH25T3v6WV2RkN=itidE=ryo3w@mail.gmail.com>

Hi Seung,

Are you talking about the interpreter tests or tests for your own packages?

If you want to run tests, I think the latest versions of Tox support
PyPy. You can have it run tests against multiple versions of python:
http://tox.testrun.org/en/latest/

-Chris


On Wed, Aug 10, 2011 at 7:49 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Seung,
>
> On Wed, Aug 10, 2011 at 11:35 AM, Seung Soo, <sungsuha at gmail.com> wrote:
>> Anyway I'd like to know if there is any single way to test both
>> python and pypy(2.7), where the results should be the same.
>
> We use our own custom runner, which you run differently than
> CPython's. ?Well, you can also try running "python -m test.regrtest"
> and "pypy -m test.regrtest", which works in both cases -- or at least
> "almost works" on top of PyPy. ?The issue is that this will try to run
> all CPython tests, including the few ones that we normally skip; in
> particular there is test_capi that tests the C API, which is not
> complete on top of PyPy yet, so you get a fatal AssertionError.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Christopher Lambacher
chris at kateandchris.net

From hakan at debian.org  Wed Aug 10 16:43:06 2011
From: hakan at debian.org (Hakan Ardo)
Date: Wed, 10 Aug 2011 16:43:06 +0200
Subject: [pypy-dev] jit-short_from_state
Message-ID: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>

Hi,
what I had in mind for jit-short_from_state is now in place and seem
to be working. So if anyone feels like reviewing now is a good time :)
Benchmarks show ai 15% faster, spectral-norm 47% faster and bm_mako
11% slower as compared to trunk. The others are not affected much:

    http://paste.pocoo.org/show/455974/

There are still 3 test_pypy_c tests failing due worse optimization:

    http://buildbot.pypy.org/summary?branch=jit-short_from_state

Addressing those would need some though though...

-- 
H?kan Ard?

From fijall at gmail.com  Wed Aug 10 17:06:03 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 10 Aug 2011 17:06:03 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
Message-ID: <CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>

On Wed, Aug 10, 2011 at 4:43 PM, Hakan Ardo <hakan at debian.org> wrote:
> Hi,
> what I had in mind for jit-short_from_state is now in place and seem
> to be working. So if anyone feels like reviewing now is a good time :)
> Benchmarks show ai 15% faster, spectral-norm 47% faster and bm_mako
> 11% slower as compared to trunk. The others are not affected much:

go is 35% faster (just reading the paste :)

>
> ? ?http://paste.pocoo.org/show/455974/
>
> There are still 3 test_pypy_c tests failing due worse optimization:
>
> ? ?http://buildbot.pypy.org/summary?branch=jit-short_from_state
>
> Addressing those would need some though though...
>
> --
> H?kan Ard?
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From fijall at gmail.com  Wed Aug 10 17:25:42 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 10 Aug 2011 17:25:42 +0200
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <874o1u1znn.fsf@pobox.com>
References: <87vcuepn0g.fsf@pobox.com>
	<CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
	<CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
	<874o1u1znn.fsf@pobox.com>
Message-ID: <CAK5idxQDzQcQU8tG-5My-fkOhzE9siK0ngwZRxnCFvku4VLXBQ@mail.gmail.com>

On Sat, Aug 6, 2011 at 2:40 PM, Tom Roche <Tom_Roche at pobox.com> wrote:
>
> Armin Rigo Wed, Aug 3, 2011 at 8:21 AM
>> We edit [pypy.org/source/download.txt]
>
> Done! The new'n'improved download.txt file is @
>
> https://docs.google.com/leaf?id=0BzDAFHgIxRzKYzQzNzUwYTktNzBjMS00ZmNmLTk3YzItYThhNTdmY2Q3OWM1&hl=en_US
>
> and the patch file is @
>
> https://docs.google.com/leaf?id=0BzDAFHgIxRzKZmQzZWZkNDYtODE2YS00MjUwLTgxMDYtZDdiYzVmMmE5ODJh&hl=en_US
>
> HTH, Tom Roche <Tom_Roche at pobox.com>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Hi

This diff does look a bit weird - it contains a lot of stuff and some
docutils help among other things. What did you try to diff from? Also
I'm not completely sure the idea is to put a copy-pasted script to do
something there. Maybe a link would be better?

Cheers,
fijal

From fijall at gmail.com  Wed Aug 10 17:26:39 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 10 Aug 2011 17:26:39 +0200
Subject: [pypy-dev] PowerPC 64 bit support
In-Reply-To: <CAGWvnynceRZ0Wd+nNPtYpb9V-NHpqxt6gwcgrVUea5xwchoAxw@mail.gmail.com>
References: <CAGWvnynceRZ0Wd+nNPtYpb9V-NHpqxt6gwcgrVUea5xwchoAxw@mail.gmail.com>
Message-ID: <CAK5idxT6Ri8r1s5DbYtRt9yJy7M9JUgeWAf2Sd7X2thDVS4kVg@mail.gmail.com>

> I need to understand how to add tests that only run in 64 bit mode
> because the instruction will fail on a 32 bit PowerPC system as
> illegal instructions.
>
> Thanks, David
>

Look how skip-on-32bit-intel is implemented in tests.

From alex.gaynor at gmail.com  Wed Aug 10 17:27:18 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 10 Aug 2011 08:27:18 -0700
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
Message-ID: <CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>

On Wed, Aug 10, 2011 at 8:06 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Wed, Aug 10, 2011 at 4:43 PM, Hakan Ardo <hakan at debian.org> wrote:
> > Hi,
> > what I had in mind for jit-short_from_state is now in place and seem
> > to be working. So if anyone feels like reviewing now is a good time :)
> > Benchmarks show ai 15% faster, spectral-norm 47% faster and bm_mako
> > 11% slower as compared to trunk. The others are not affected much:
>
> go is 35% faster (just reading the paste :)
>
> >
> >    http://paste.pocoo.org/show/455974/
> >
> > There are still 3 test_pypy_c tests failing due worse optimization:
> >
> >    http://buildbot.pypy.org/summary?branch=jit-short_from_state
> >
> > Addressing those would need some though though...
> >
> > --
> > H?kan Ard?
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

I remember you also showed me it made my HTML escaper about 20% faster on
some benchmarks :)  Of the 3 test_pypy_  benchmarks, only the test_instance
one shows a really noticable regression (test_strings looks better
actually...), can you explain why it got worse?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110810/637fda2b/attachment.html>

From anto.cuni at gmail.com  Wed Aug 10 18:15:12 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 10 Aug 2011 18:15:12 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
	<CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
Message-ID: <4E42AE90.8070003@gmail.com>

On 10/08/11 17:27, Alex Gaynor wrote:
> I remember you also showed me it made my HTML escaper about 20% faster on some
> benchmarks :)  Of the 3 test_pypy_  benchmarks, only the test_instance one
> shows a really noticable regression (test_strings looks better actually...),
> can you explain why it got worse?

while you are at it, could you please also explain what is the branch about 
again? :-)

ciao,
Anto

From hakan at debian.org  Wed Aug 10 18:42:55 2011
From: hakan at debian.org (Hakan Ardo)
Date: Wed, 10 Aug 2011 18:42:55 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
	<CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
Message-ID: <CAEkL+RhOtK+89mjg2RypU0mXMZ5bQrWqkyo8XY9tkTG+hOBFRA@mail.gmail.com>

On Wed, Aug 10, 2011 at 5:27 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>
> On Wed, Aug 10, 2011 at 8:06 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>>
>> On Wed, Aug 10, 2011 at 4:43 PM, Hakan Ardo <hakan at debian.org> wrote:
>> > Hi,
>> > what I had in mind for jit-short_from_state is now in place and seem
>> > to be working. So if anyone feels like reviewing now is a good time :)
>> > Benchmarks show ai 15% faster, spectral-norm 47% faster and bm_mako
>> > 11% slower as compared to trunk. The others are not affected much:
>>
>> go is 35% faster (just reading the paste :)
>>
>> >
>> > ? ?http://paste.pocoo.org/show/455974/
>> >
>> > There are still 3 test_pypy_c tests failing due worse optimization:
>> >
>> > ? ?http://buildbot.pypy.org/summary?branch=jit-short_from_state
>> >
>> > Addressing those would need some though though...
>> >
>> > --
>> > H?kan Ard?
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > http://mail.python.org/mailman/listinfo/pypy-dev
>> >
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
> I remember you also showed me it made my HTML escaper about 20% faster on
> some benchmarks :) ?Of the 3 test_pypy_ ?benchmarks, only the test_instance
> one shows a really noticable regression (test_strings looks better
> actually...), can you explain why it got worse?

In test_string, there is an extra int_eq, guard_false, probably
because intbounds now are generalized at the end of the preamble if
they are outside the range  pluss minus sys.maxint/2 in order to not
create a lot of identical version of the same loop that are
specialized to silly intbounds.

I'm not sure why test_instance works on trunk actually, but the way to
get it to work on this branch I think is to make the heap optimizer
aware of the setfield_gc's produced by forcing p22 and then have the
heap optimizer remove the getfield_gc_pure. Using the pure operations
optimizer here is not straight forwars as there is no
getfield_gc_pure(p22).

In test_misc i17 is passes at two places of the jumpargs. This
indicates that I have refrained from specializing this loop to the
fact that these two boxes are equal (which might not be the case at
the end of some bridge). Instead the single box is duplicated at the
end of the loop. To do that a new box has to be introduced and the
pure operation removal algorithm will only link the int_add operations
to one of them. This would be fixable by recursivly duplicating all
potential operations in the short preamble that have arguments that
have been duplicated. But I'm not sure how good an idea that is...

-- 
H?kan Ard?

From hakan at debian.org  Wed Aug 10 18:55:11 2011
From: hakan at debian.org (Hakan Ardo)
Date: Wed, 10 Aug 2011 18:55:11 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <4E42AE90.8070003@gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
	<CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
	<4E42AE90.8070003@gmail.com>
Message-ID: <CAEkL+Rh_b0nwdq2nJ=Khh5TFuiWZmz4TEsoYCsfCn2Fo7VMn2A@mail.gmail.com>

On Wed, Aug 10, 2011 at 6:15 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 10/08/11 17:27, Alex Gaynor wrote:
>>
>> I remember you also showed me it made my HTML escaper about 20% faster on
>> some
>> benchmarks :) ?Of the 3 test_pypy_ ?benchmarks, only the test_instance one
>> shows a really noticable regression (test_strings looks better
>> actually...),
>> can you explain why it got worse?
>
> while you are at it, could you please also explain what is the branch about
> again? :-)

For a branch to be able to jump directly to the peeled loop and not to
the preamble, a short preamble has to be inlined at the end of the
bridge. On trunk we try to create such short preambles by reordering
the operations of  the preamble. That very often fails because we are
unable to prove that reordering the operations as needed is safe. In
that case no short preamble is constructed and bridges have to jump to
the preamble.

In jit-short_from_state, only a subset of the optimizer state is
allowed to survive the recreation of the optimizer chain at the end of
the preamble. That way we can ensure that the state subset surviving
is always reconstructable by a short preamble and this short preamble
is created directly from the optimizer state. Thereby no reordering of
resops are needed and we will always get a short preamble.

Note that bridges might still end up jumping to the preamble. This
will happen if a guard in the short preamble is proven to always fail
during optimization. Also if a guard from the short preamble fails
often enough for a new bridge to be generated, this will be a bridge
to the preamble. A possible future improvement here would be to issue
a retrace instead.

-- 
H?kan Ard?

From naylor.b.david at gmail.com  Wed Aug 10 21:27:10 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Wed, 10 Aug 2011 21:27:10 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
Message-ID: <201108102127.13752.naylor.b.david@gmail.com>

Hi,

I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache.  For comparison I put together 
the following test:

<file="iforkey.py">
import datetime
import random
import timeit

ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)]

def ifdict():
    cache = {}
    b = []
    for i in ranges:
        key = i.day
        if key in cache:
            b.append(cache[key])
        else:
            date = i.date()
            cache[key] = date
            b.append(date)

def keydict():
    cache = {}
    b = []
    for i in ranges:
        key = i.day
        try:
            b.append(cache[key])
        except KeyError:
            date = i.date()
            cache[key] = date
            b.append(date)

def defaultdict():
    cache = {}
    b= []
    for i in ranges:
        b.append(cache.setdefault(i, i.date()))

print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000)
print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000)
print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000)
</file>

# python iforfile.py
ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379]
keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542]
defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844]

# pypy iforfile.py	(pypy 1.5)
ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047]
keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078]
defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909]

Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup.  To check what is the cause I replaced i.date() 
with i.day and found no major difference in times.  It appears dict.setdefault (or it's interaction with jit) is causing a slow down.  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110810/f833eb4f/attachment-0001.pgp>

From Tom_Roche at pobox.com  Wed Aug 10 21:37:50 2011
From: Tom_Roche at pobox.com (Tom Roche)
Date: Wed, 10 Aug 2011 15:37:50 -0400
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <CAK5idxQDzQcQU8tG-5My-fkOhzE9siK0ngwZRxnCFvku4VLXBQ@mail.gmail.com>
References: <87vcuepn0g.fsf@pobox.com>
	<CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
	<CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
	<874o1u1znn.fsf@pobox.com>
	<CAK5idxQDzQcQU8tG-5My-fkOhzE9siK0ngwZRxnCFvku4VLXBQ@mail.gmail.com>
Message-ID: <CAPRwwRFHs=edD35XLOR3gXseq1822m5PLi-Tc=DnKZJP+=7WMg@mail.gmail.com>

Tom Roche Sat, Aug 6, 2011 at 2:40 PM
>> The new'n'improved download.txt file is @

>> https://docs.google.com/leaf?id=0BzDAFHgIxRzKYzQzNzUwYTktNzBjMS00ZmNmLTk3YzItYThhNTdmY2Q3OWM1&hl=en_US

>> and the patch file is @

>> https://docs.google.com/leaf?id=0BzDAFHgIxRzKZmQzZWZkNDYtODE2YS00MjUwLTgxMDYtZDdiYzVmMmE5ODJh&hl=en_US

Maciej Fijalkowski Wed, 10 Aug 2011 17:25:42 +0200
> This diff does look a bit weird

Read the download.txt. I reorganized it (e.g., to actually explain
things :-), and `diff` doesn't handle that well. If you really believe
the previous download.txt only needed marginal tweaking, I can't help
you.

> I'm not completely sure the idea is to put a copy-pasted script to
> do something there.

Example scripts in doc are far from unknown. But if it bothers you, take it out.

From fijall at gmail.com  Wed Aug 10 21:45:58 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 10 Aug 2011 21:45:58 +0200
Subject: [pypy-dev] PyPy download/install page
In-Reply-To: <CAPRwwRFHs=edD35XLOR3gXseq1822m5PLi-Tc=DnKZJP+=7WMg@mail.gmail.com>
References: <87vcuepn0g.fsf@pobox.com>
	<CAK5idxS8g-xc7MCQ2XOUuWAYJUY9v=zxN2XBdHy6GBy6s+9byA@mail.gmail.com>
	<8762mfffem.fsf@pobox.com>
	<CAMSv6X0Rs5O15p4rCKVg3sJg8ieysV2xQ=5Ctn4BfW0Bp87T7A@mail.gmail.com>
	<CAK5idxSt809fxJEghMuK+kTAGpPdPqUL8Y7OuUOTNyBjc_9tug@mail.gmail.com>
	<874o1u1znn.fsf@pobox.com>
	<CAK5idxQDzQcQU8tG-5My-fkOhzE9siK0ngwZRxnCFvku4VLXBQ@mail.gmail.com>
	<CAPRwwRFHs=edD35XLOR3gXseq1822m5PLi-Tc=DnKZJP+=7WMg@mail.gmail.com>
Message-ID: <CAK5idxToPb-h0bDTpp57rGfr_t+Ww2bEih11-0Kj80is7BAV8A@mail.gmail.com>

On Wed, Aug 10, 2011 at 9:37 PM, Tom Roche <Tom_Roche at pobox.com> wrote:
> Tom Roche Sat, Aug 6, 2011 at 2:40 PM
>>> The new'n'improved download.txt file is @
>
>>> https://docs.google.com/leaf?id=0BzDAFHgIxRzKYzQzNzUwYTktNzBjMS00ZmNmLTk3YzItYThhNTdmY2Q3OWM1&hl=en_US
>
>>> and the patch file is @
>
>>> https://docs.google.com/leaf?id=0BzDAFHgIxRzKZmQzZWZkNDYtODE2YS00MjUwLTgxMDYtZDdiYzVmMmE5ODJh&hl=en_US
>
> Maciej Fijalkowski Wed, 10 Aug 2011 17:25:42 +0200
>> This diff does look a bit weird
>
> Read the download.txt. I reorganized it (e.g., to actually explain
> things :-), and `diff` doesn't handle that well. If you really believe
> the previous download.txt only needed marginal tweaking, I can't help
> you.

Ok. I think homebrew, gentoo and arch are providing pypy packages
[citation needed though]

>
>> I'm not completely sure the idea is to put a copy-pasted script to
>> do something there.
>
> Example scripts in doc are far from unknown. But if it bothers you, take it out.

ok.

> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Thu Aug 11 09:37:03 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Aug 2011 09:37:03 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
Message-ID: <CAMSv6X3H9oBeqPiH7HUMYGxPXUzCMxtvSJbrR40fieAT14+dhQ@mail.gmail.com>

Hi Hakan,

On Wed, Aug 10, 2011 at 4:43 PM, Hakan Ardo <hakan at debian.org> wrote:
> what I had in mind for jit-short_from_state is now in place and seem
> to be working.

Great work!

> So if anyone feels like reviewing now is a good time :)

I might give it a serious look, but right now I cannot promise to.  I
suppose it's fine if you want to merge at any time (now that we have a
release branch already), and we can still look.


A bient?t,

Armin.

From arigo at tunes.org  Thu Aug 11 10:46:36 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Aug 2011 10:46:36 +0200
Subject: [pypy-dev] Stacklets
Message-ID: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>

Hi all,

Some progress report from my side: I am working on "stacklets", which
is, as the name implies, similar to Stackless.  It is a rewrite of
tealets, which was itself a greenlet-like replacement for Stackless
Python.  (Yes, "tealet" because of "greenlet": green tea... last time
I can do the joke, though, as tealets go away.)

Basically a "stacklet" is just a one-shot continuation.  The name
"stacklet" is going to be internal; at app-level it's just called a
continuation.  It's the most basic primitive I could eventually arrive
at, which is not surprising at all, given that it *is* well known to
be the most basic primitive in functional programming.  See for
example http://en.wikipedia.org/wiki/Call-with-current-continuation.
(Of course it is possible to emulate it using other primitives, e.g.
http://sigusr2.net/2011/Aug/09/call-cc-for-python.html, but that's
abstraction inversion, really.)

In the stacklet branch you get a new built-in module called
"_continuation" which has only one function: "new", also exposed under
the name "callcc", for the reference to various functional programming
languages.  Doing "_continuation.new(func, args...)" calls "func(c,
args...)" with 'c' being a new Continuation object.  Calling
"c.switch()" will switch back to the point where 'c' was created, and
this point will return another continuation 'c2' that can be used to
come back again.  After a number of switches the function func() can
return; it should return yet another Continuation, the one to jump to
next.

- Continuations are objects with a "switch()" method instead of
directly callables, because that seemed more Pythonic

- One shot: once you resume a continuation, the object cannot be
switched to any more

- Garbage collectable: if a pending continuation dies, then every
resource reachable only by resuming it dies too

- Support for exceptions: if func() terminates by raising an
exception, the exception is propagated in the parent (see below)

- Implemented like greenlets by moving around part of the C stack,
which is easily compatible with the JIT

- See http://en.wikipedia.org/wiki/Call-with-current-continuation for
a list of what can easily be built on top of them.

Once I'm done refactoring the GC to support them (in particular with
shadowstack, which requires a bit of care), I will write a greenlet.py
module at app-level implementing full greenlets (which should be less
than 100 lines of code).  It should be useful as a demo of how
higher-level abstractions can be implemented on top of continuations.
Ah, I should also write a CPython wrapper around the core, which is
already written in C, to expose the same continuations for CPython.


One (longish, sorry) word about propagating exceptions: this is not
usually a concern of functional programming languages, so I had to
come up with a way to define where an exception goes.  It goes to the
'parent', which is defined in a greenlet-like way: the parent of a
'func' is the piece of C stack that originally started 'func'.  I
think that this way is the most correct one.  Let me explain.  One
goal of stacklets is to give a "composable" concept, where
"composable" is defined as: if two pieces of code independently use
continuations, and if both pieces work fine when run in isolation,
then they should work fine when run together too.  Stackless Python's
tasklets are composable: you don't have actually references to
tasklets, but only to channel objects, which does the trick (I can
give more details if anyone is interested).  Greenlets on the other
hand are not, and neither are coroutines.  Basic continuations
(without exceptions) are composable again.

Internally, even with pypy+stacklets, continuations form a tree
similar to greenlets, but this tree is not exposed to app-level,
because doing so would break composability.  For example, you could
ask the question "what is my 'parent' piece of stack" and jump there;
to see how this can be wrong, consider the following call chain:

    e() -> _continuation.new() -> f() -> g() -> h()

where g() is in some other module but calls back your own h().  In
this case the 'parent' piece of stack is not well-defined: h() would
expect that it is e(), but maybe g() internally did another
_continuation.new() too.  However, in case of exceptions, then it is
*still* the correct thing to do to propagate the exception into the
parent.  If h() raises, it is propagated back to g(), back to f(), and
back to e(), even if g() internally did another _continuation.new().
Every level is free to capture the exception or let it propagate if
it's of an unknown class, just like normally.  So h() might raise an
exception and expect e() to catch it, which will work just like it
does in the normal case: it will work as long as g(), which is in
another module, doesn't explicitly catch exceptions of the same class
as well.  I suppose this is already well-known, but I couldn't find an
obvious reference.  (For example, looking at Ruby doesn't help much
because its callcc seems slightly different.)


A bient?t,

Armin.

From arigo at tunes.org  Thu Aug 11 10:59:52 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Aug 2011 10:59:52 +0200
Subject: [pypy-dev] Stacklets
In-Reply-To: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>
References: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>
Message-ID: <CAMSv6X256kE-Qk3F8Z0Kp7HCLWv6CDqBtFBphzndX7ux=iXYoQ@mail.gmail.com>

Re-hi,

On Thu, Aug 11, 2011 at 10:46 AM, Armin Rigo <arigo at tunes.org> wrote:
> I suppose this is already well-known, but I couldn't find an obvious reference.

Ah, I think it's what you get if you assume that your language has
callcc but no exceptions, and explicitly implement exceptions on top
of callcc.  At every call you would pass an extra argument, which is
the continuation to switch to in case of an exception.  It usually
just comes from the parent, but is changed in situations where we'd do
a "try:".  I need to check that (theoretically) doing this would give
the exact same behavior as what I'm writing right now to support
(real) exceptions with continuations.


A bient?t,

Armin.

From fijall at gmail.com  Thu Aug 11 11:08:18 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 11 Aug 2011 11:08:18 +0200
Subject: [pypy-dev] 1.6 status report
Message-ID: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>

Hi

A little status update from my side.

I'm trying to get 1.6 out of the door and there are a few things that
stand out before we can do it.

* Windows situation (a lot of crashes with missing liveness). Also
lib-python tests take 10h which is a bit too much IMO (not a new
regression though)
* OS X situation (missing buildbots, being addressed now)
* A bunch of issues that either should be decided we don't care or we
do care and fix it.
* Missing jitviewer release/demo that I'm doing right now
* Alex reported sqlite crashes
* random segfault when running trackgcroot, armin, did you have a
look? I can do if you don't

Anything else that comes to mind?

Cheers,
fijal

From arigo at tunes.org  Thu Aug 11 11:41:48 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Aug 2011 11:41:48 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
Message-ID: <CAMSv6X0exjMUTYs5FYKomi9Nx3Hz1mSNFt3OvCD-S2hMER+Nbg@mail.gmail.com>

Hi Fijal,

On Thu, Aug 11, 2011 at 11:08 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> * Windows situation (a lot of crashes with missing liveness).

Still?...  Indeed.  I thought we fixed that at some point, but no.  I
can look again...

> * random segfault when running trackgcroot, armin, did you have a
> look? I can do if you don't

Please do.  I cannot reproduce it at all; I've run trackgcroot
repeatedly for 9 hours on 4 cores, on all .s files of a build, both
with a debug and a release pypy.  Worked fine.  If it's even more rare
I can try running it again for several days, but I fear that somehow
these pypy's on these particular .s files won't show the bug at all.


A bient?t,

Armin.

From lac at openend.se  Thu Aug 11 12:22:40 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 11 Aug 2011 12:22:40 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: Message from Armin Rigo <arigo@tunes.org> of "Thu,
	11 Aug 2011 11:41:48 +0200."
	<CAMSv6X0exjMUTYs5FYKomi9Nx3Hz1mSNFt3OvCD-S2hMER+Nbg@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com><CAMSv6X0exjMUTYs5FYKomi9Nx3Hz1mSNFt3OvCD-S2hMER+Nbg@mail.gmail.com>
Message-ID: <201108111022.p7BAMeml007964@theraft.openend.se>

Do we want to run our speed comparisons against 2.7 not 2.6 for the
release?

Laura


From anto.cuni at gmail.com  Thu Aug 11 12:30:44 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 11 Aug 2011 12:30:44 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108111022.p7BAMeml007964@theraft.openend.se>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com><CAMSv6X0exjMUTYs5FYKomi9Nx3Hz1mSNFt3OvCD-S2hMER+Nbg@mail.gmail.com>
	<201108111022.p7BAMeml007964@theraft.openend.se>
Message-ID: <4E43AF54.4070607@gmail.com>

On 11/08/11 12:22, Laura Creighton wrote:
> Do we want to run our speed comparisons against 2.7 not 2.6 for the
> release?

I think we want CPython 2.7.  However, tannit runs ubuntu 10.4 TLS, which 
still gives only 2.6.

I'm going to manually compile CPython 2.7 on tannit, then I'll try to make it 
the default used by buildslave.

ciao,
Anto

From cfbolz at gmx.de  Thu Aug 11 12:55:11 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 11 Aug 2011 12:55:11 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
Message-ID: <CADLBHoSR-MaURF=eB5_c_X7RuXV3ZstqUkGXX7hnvJgnFeUqbA@mail.gmail.com>

On 08/10/2011 04:43 PM, Hakan Ardo wrote:
> Hi,
> what I had in mind for jit-short_from_state is now in place and seem
> to be working. So if anyone feels like reviewing now is a good time :)
> Benchmarks show ai 15% faster, spectral-norm 47% faster and bm_mako
> 11% slower as compared to trunk. The others are not affected much:
>
>      http://paste.pocoo.org/show/455974/
>
> There are still 3 test_pypy_c tests failing due worse optimization:
>
>      http://buildbot.pypy.org/summary?branch=jit-short_from_state
>
> Addressing those would need some though though...
>

Great work (as usual)! I won't look in detail at the moment (still on
vacation) but did you write the unit-tests for the virtual state
classes that we talked about during the Gothenburg sprint?

Cheers,

Carl Friedrich

From dje.gcc at gmail.com  Thu Aug 11 17:32:43 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 11 Aug 2011 11:32:43 -0400
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
Message-ID: <CAGWvnyk0ZnHWG7iLVRTw6EVhJDVdHaaYwM3SXuCG1gzQrymmaw@mail.gmail.com>

On Wed, Aug 10, 2011 at 9:34 AM, Michael Hudson-Doyle <micahel at gmail.com> wrote:

> -- nothing prevents a field returning a value that is split between
> multiple bit ranges. ?There is a bit of scaffolding that assumes
> contiguous bit ranges but nothing fundamental (and as I said, I think
> this is mostly for checking that fields don't overlap, and there would
> be other ways of doing that).

I would prefer to get more things working and then return to a better
way to encode the instruction than performing surgery on Field for one
instruction with four variants as a first project.

- David

From arigo at tunes.org  Thu Aug 11 18:46:45 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Aug 2011 18:46:45 +0200
Subject: [pypy-dev] Stacklets
In-Reply-To: <CAMSv6X256kE-Qk3F8Z0Kp7HCLWv6CDqBtFBphzndX7ux=iXYoQ@mail.gmail.com>
References: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>
	<CAMSv6X256kE-Qk3F8Z0Kp7HCLWv6CDqBtFBphzndX7ux=iXYoQ@mail.gmail.com>
Message-ID: <CAMSv6X08ssPtxnY4f6HjvWuzBDkTrjHmBa9-a3w-gPWrNedSZw@mail.gmail.com>

Re-hi,

On Thu, Aug 11, 2011 at 10:59 AM, Armin Rigo <arigo at tunes.org> wrote:
> Ah, I think it's what you get if you assume that your language has
> callcc but no exceptions, and explicitly implement exceptions on top
> of callcc.

Ok, after a fruitful discussion on IRC (thanks Da_Blitz), here's yet
another slightly different concept, almost as primitive as the
previous one, but capable of doing the obvious right thing with
exceptions.  It also has all the good properties of Python generators,
so let's call it "genlets" for now.  (For example, you can raise in a
Python generator, or resume a suspended generator in a different
thread, and the right thing occurs; this also occurs with genlets.)

Moreover, we can give it an interface very similar to generators.  A
genlet object has a primitive method send(x), a wrapper next() that
just calls send(None), and is an iterator.  Example:

@genlet
def f(gen, a, b):
    for i in range(a, b):
        gen.send(i)

for x in f(10, 20):
    print x

Of course this example is designed to show the similarity with
generators.  The 'gen' received by the generator as a first argument
is the same object as the one iterated over in the 'for' loop below;
this means more symmetrical usages are possible, too, using another
method gen1.switch(gen2, x) to do a "symmetrical switch" from gen1 to
gen2, as opposed to send() which does a going-into or getting-out-of
operation.

This looks like a "Pythonic" enough interface, just because it is very
similar to the existing one of generators.  It is also rather
primitive, although good enough to directly use like the example
above.  I can give a more precise definition of what send() and
switch() do, but I think I'll leave that for the next e-mail :-)  Does
the API presented here makes sense?


A bient?t,

Armin

From hakan at debian.org  Thu Aug 11 19:00:11 2011
From: hakan at debian.org (Hakan Ardo)
Date: Thu, 11 Aug 2011 19:00:11 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CADLBHoSR-MaURF=eB5_c_X7RuXV3ZstqUkGXX7hnvJgnFeUqbA@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CADLBHoSR-MaURF=eB5_c_X7RuXV3ZstqUkGXX7hnvJgnFeUqbA@mail.gmail.com>
Message-ID: <CAEkL+Rj2h9etq7-+cSn=7eKvTsh-6hktV1hC5rt9+0WJ6TkVuA@mail.gmail.com>

On Thu, Aug 11, 2011 at 12:55 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>>
>
> Great work (as usual)! I won't look in detail at the moment (still on
> vacation) but did you write the unit-tests for the virtual state
> classes that we talked about during the Gothenburg sprint?

Well, there is test_virtualstate.py with 86% coverage of
virtualstate.py. It contains some tests of the type we talked about
and some that I'm afraid you would call integration tests. I could
look into write a few more of the first kind. A complete testrun in
jit/metainterp has 95% coverage of virtualstate.py.

-- 
H?kan Ard?

From fijall at gmail.com  Thu Aug 11 20:04:58 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 11 Aug 2011 20:04:58 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <4E43AF54.4070607@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<CAMSv6X0exjMUTYs5FYKomi9Nx3Hz1mSNFt3OvCD-S2hMER+Nbg@mail.gmail.com>
	<201108111022.p7BAMeml007964@theraft.openend.se>
	<4E43AF54.4070607@gmail.com>
Message-ID: <CAK5idxT-eVYAxNjzc7W50DJXa82R1Y5kz=eLOpU7h35ZsFUxXg@mail.gmail.com>

On Thu, Aug 11, 2011 at 12:30 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 11/08/11 12:22, Laura Creighton wrote:
>>
>> Do we want to run our speed comparisons against 2.7 not 2.6 for the
>> release?
>
> I think we want CPython 2.7. ?However, tannit runs ubuntu 10.4 TLS, which
> still gives only 2.6.
>
> I'm going to manually compile CPython 2.7 on tannit, then I'll try to make
> it the default used by buildslave.

That's completely irrelevant, we don't run cpython while running
benchmarks repeatedly. Run benchmarks by hand using 2.7 and give me
the .json file, I'll care about uploading it.

>
> ciao,
> Anto
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From naylor.b.david at gmail.com  Thu Aug 11 20:17:58 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Thu, 11 Aug 2011 20:17:58 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
Message-ID: <201108112018.01642.naylor.b.david@gmail.com>

On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> Hi
> 
> A little status update from my side.
> 
> I'm trying to get 1.6 out of the door and there are a few things that
> stand out before we can do it.
> 
> * Windows situation (a lot of crashes with missing liveness). Also
> lib-python tests take 10h which is a bit too much IMO (not a new
> regression though)
> * OS X situation (missing buildbots, being addressed now)
> * A bunch of issues that either should be decided we don't care or we
> do care and fix it.
> * Missing jitviewer release/demo that I'm doing right now
> * Alex reported sqlite crashes
> * random segfault when running trackgcroot, armin, did you have a
> look? I can do if you don't

Would anyone be interested in the FreeBSD situation?  If so what support do 
you need?  I am currently updating my repository and will report back...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110811/4413a1d0/attachment.pgp>

From alex.gaynor at gmail.com  Thu Aug 11 21:31:36 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 11 Aug 2011 12:31:36 -0700
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108112018.01642.naylor.b.david@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108112018.01642.naylor.b.david@gmail.com>
Message-ID: <CAFRnB2X+WyhRMZwWa2BVLogOXQhJecM-Dtw=9Kqe3QAYHgcLSg@mail.gmail.com>

On Thu, Aug 11, 2011 at 11:17 AM, David Naylor <naylor.b.david at gmail.com>wrote:

> On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > Hi
> >
> > A little status update from my side.
> >
> > I'm trying to get 1.6 out of the door and there are a few things that
> > stand out before we can do it.
> >
> > * Windows situation (a lot of crashes with missing liveness). Also
> > lib-python tests take 10h which is a bit too much IMO (not a new
> > regression though)
> > * OS X situation (missing buildbots, being addressed now)
> > * A bunch of issues that either should be decided we don't care or we
> > do care and fix it.
> > * Missing jitviewer release/demo that I'm doing right now
> > * Alex reported sqlite crashes
> > * random segfault when running trackgcroot, armin, did you have a
> > look? I can do if you don't
>
> Would anyone be interested in the FreeBSD situation?  If so what support do
> you need?  I am currently updating my repository and will report back...
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
FWIW the sqlite issue is visible randomly on the buildbots, it showed up
today for example:
http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-c-jit-linux-x86-32&build=936&mod=lib-python.2.7.test.test_sqlite
.
 However, I've found that it shows up *always* when running the django
tests.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110811/61dfab2b/attachment.html>

From dje.gcc at gmail.com  Thu Aug 11 21:42:25 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 11 Aug 2011 15:42:25 -0400
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
Message-ID: <CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>

Attached is a new patch that corrects the PPC fields.

The only simple way I can think of handling the 64 bit immediate shift
field is by adding a keyword argument to Field to ignore overlap.
Expressing the details of the field splitting will be complicated and
doesn't seem worth it.

Is there a preference for the current patch that explicitly performs
the conversion or a new argument to ignore overlap?

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff2
Type: application/octet-stream
Size: 3109 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110811/44ac480a/attachment.obj>

From Sven.Hager at uni-duesseldorf.de  Thu Aug 11 22:32:21 2011
From: Sven.Hager at uni-duesseldorf.de (Sven Hager)
Date: Thu, 11 Aug 2011 22:32:21 +0200
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
	<CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>
Message-ID: <4E443C55.3070800@uni-duesseldorf.de>

On 08/11/2011 09:42 PM, David Edelsohn wrote:
> Attached is a new patch that corrects the PPC fields.
>
> The only simple way I can think of handling the 64 bit immediate shift
> field is by adding a keyword argument to Field to ignore overlap.
> Expressing the details of the field splitting will be complicated and
> doesn't seem worth it.
>
> Is there a preference for the current patch that explicitly performs
> the conversion or a new argument to ignore overlap?
>
> Thanks, David
Hello David,

I think it is okay to add an attribute to class Field which
explicitly allows overlapping bits. It is very easy to adapt
the existing code in form.py to this.

I will add your changes tomorrow.

Best regards,
Sven

From dje.gcc at gmail.com  Fri Aug 12 02:24:20 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 11 Aug 2011 20:24:20 -0400
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <4E443C55.3070800@uni-duesseldorf.de>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
	<CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>
	<4E443C55.3070800@uni-duesseldorf.de>
Message-ID: <CAGWvnykFQVcWUCWj3dj_=4xqguPOgNyTMMaLQemNc5fQtp5Ykw@mail.gmail.com>

Sven,

If we use the overlapping bits approach, let me update that part of the patch.

You can add the changes to fields, but don't add the rldicXXX changes
in ppc_assembler.py

Thanks, David

On Thu, Aug 11, 2011 at 4:32 PM, Sven Hager
<Sven.Hager at uni-duesseldorf.de> wrote:
> On 08/11/2011 09:42 PM, David Edelsohn wrote:
>>
>> Attached is a new patch that corrects the PPC fields.
>>
>> The only simple way I can think of handling the 64 bit immediate shift
>> field is by adding a keyword argument to Field to ignore overlap.
>> Expressing the details of the field splitting will be complicated and
>> doesn't seem worth it.
>>
>> Is there a preference for the current patch that explicitly performs
>> the conversion or a new argument to ignore overlap?
>>
>> Thanks, David
>
> Hello David,
>
> I think it is okay to add an attribute to class Field which
> explicitly allows overlapping bits. It is very easy to adapt
> the existing code in form.py to this.
>
> I will add your changes tomorrow.
>
> Best regards,
> Sven
>

From hakan at debian.org  Fri Aug 12 07:41:45 2011
From: hakan at debian.org (Hakan Ardo)
Date: Fri, 12 Aug 2011 07:41:45 +0200
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAEkL+RhOtK+89mjg2RypU0mXMZ5bQrWqkyo8XY9tkTG+hOBFRA@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
	<CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
	<CAEkL+RhOtK+89mjg2RypU0mXMZ5bQrWqkyo8XY9tkTG+hOBFRA@mail.gmail.com>
Message-ID: <CAEkL+RiU6o_DXXYfxwMV5itRu1EixmmM-VqjWYyM=K0zphGoZg@mail.gmail.com>

On Wed, Aug 10, 2011 at 6:42 PM, Hakan Ardo <hakan at debian.org> wrote:
>
> I'm not sure why test_instance works on trunk actually, but the way to
> get it to work on this branch I think is to make the heap optimizer
> aware of the setfield_gc's produced by forcing p22 and then have the
> heap optimizer remove the getfield_gc_pure. Using the pure operations
> optimizer here is not straight forwars as there is no
> getfield_gc_pure(p22).

This issues does not show up on trunk because of is_forced_virtual()
keeps track of forced virtuals and treats them separately. I dont
think we want specialize loops to inputargs beeing forced virtuals and
guard for this condition at the end of bridges...

-- 
H?kan Ard?

From alex.gaynor at gmail.com  Fri Aug 12 07:43:40 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 11 Aug 2011 22:43:40 -0700
Subject: [pypy-dev] jit-short_from_state
In-Reply-To: <CAEkL+RiU6o_DXXYfxwMV5itRu1EixmmM-VqjWYyM=K0zphGoZg@mail.gmail.com>
References: <CAEkL+Riwo864F5XM49N1_NTYwP69XJKLj7M-mUTqQtQziMTHpQ@mail.gmail.com>
	<CAK5idxT-_R4i1OZuGrfLR_G2EpcyBGXGkj1eX_iwu0OardD2BQ@mail.gmail.com>
	<CAFRnB2XCevtHH5r6+0vGEaJ4St9z5K-i50BuJZU_GdUaY9EwHQ@mail.gmail.com>
	<CAEkL+RhOtK+89mjg2RypU0mXMZ5bQrWqkyo8XY9tkTG+hOBFRA@mail.gmail.com>
	<CAEkL+RiU6o_DXXYfxwMV5itRu1EixmmM-VqjWYyM=K0zphGoZg@mail.gmail.com>
Message-ID: <CAFRnB2UOEKVG9B9xd+2510=TOkbcn8OotW=4w8Cf9=-7a8xTRQ@mail.gmail.com>

On Thu, Aug 11, 2011 at 10:41 PM, Hakan Ardo <hakan at debian.org> wrote:

> On Wed, Aug 10, 2011 at 6:42 PM, Hakan Ardo <hakan at debian.org> wrote:
> >
> > I'm not sure why test_instance works on trunk actually, but the way to
> > get it to work on this branch I think is to make the heap optimizer
> > aware of the setfield_gc's produced by forcing p22 and then have the
> > heap optimizer remove the getfield_gc_pure. Using the pure operations
> > optimizer here is not straight forwars as there is no
> > getfield_gc_pure(p22).
>
> This issues does not show up on trunk because of is_forced_virtual()
> keeps track of forced virtuals and treats them separately. I dont
> think we want specialize loops to inputargs beeing forced virtuals and
> guard for this condition at the end of bridges...
>
> --
> H?kan Ard?
>

The only motivation for `is_forced_virtual` was to have fields that are pure
be kept as known values (to prevent re-reading), since when you force a
virtual the setfields don't go through the heap cache (which they probably
should, TBH, since we know exactly what fields a call can force).

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110811/d69904a7/attachment.html>

From vincent at shishkabab.net  Fri Aug 12 11:32:46 2011
From: vincent at shishkabab.net (Vincent den Boer)
Date: Fri, 12 Aug 2011 11:32:46 +0200
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
Message-ID: <201108121132.46897.vincent@shishkabab.net>

Hello evryone,

I've just launched a project on GitHub called Wrappyr that may be of interest to 
the PyPy project. It is a collection of tools that can generate Ctypes-based 
APIs to C and C++ code. As I understand Ctypes is currently the cleanest way to 
communicate with C libraries in PyPy, so this may help replacing CPython 
extensions that are needed for more widespread usage of PyPy.

It's available at: https://github.com/ShishKabab/Wrappyr/

It works by:
* Using Clang to export data from a C/C++ header file to XML (I tried GCCXML, but
  it would ignore certain things)
* Using that data to:
- Generate a C API to C++ code if necessary.
- Generate another XML file describing the Python API (which calls should be made
  where, etc).
* And using that XML file to generate Ctypes-based Python code.

It is not finished yet and still needs some basic features to make most libraries 
(like the MySQL C API) useful, but I think it's shaping up nicely. For example, 
the Hello World example of the Box2D C++ library runs without problems. For more 
information on what needs to be done, see TODO.rst in the root of the project.

Kind regards,
Vincent den Boer

From anto.cuni at gmail.com  Fri Aug 12 14:22:28 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 12 Aug 2011 14:22:28 +0200
Subject: [pypy-dev] Stacklets
In-Reply-To: <CAMSv6X08ssPtxnY4f6HjvWuzBDkTrjHmBa9-a3w-gPWrNedSZw@mail.gmail.com>
References: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>	<CAMSv6X256kE-Qk3F8Z0Kp7HCLWv6CDqBtFBphzndX7ux=iXYoQ@mail.gmail.com>
	<CAMSv6X08ssPtxnY4f6HjvWuzBDkTrjHmBa9-a3w-gPWrNedSZw@mail.gmail.com>
Message-ID: <4E451B04.6050104@gmail.com>

Hi Armin,

On 11/08/11 18:46, Armin Rigo wrote:

> Moreover, we can give it an interface very similar to generators.  A
> genlet object has a primitive method send(x), a wrapper next() that
> just calls send(None), and is an iterator.  Example:
>
> @genlet
> def f(gen, a, b):
>      for i in range(a, b):
>          gen.send(i)
>
> for x in f(10, 20):
>      print x
>

I'm not sure to fully understand how it would work. Let's rewrite it this way:

def f_inner(gen, a, b):
     ...
f = genlen(f_inner)
...

When I call f(10, 20), it returns a genlet object, which is an iterator with a 
next() method.

When the for loop calls gen.next(), the execution of f_inner is resumed until 
it calls gen.send(i), and then "i" is used as a return value for next(). Is it 
correct so far?

What I don't get is how all these calls interact with the C stack.  I suppose 
that when I call "gen.send(i)" the C stack for f_inner is saved inside and 
moved away (and "gen" has a reference to it), and then it resumed later when I 
call gen.next().

But, what happens to the C stack of the code which executes "for x in f(10, 
20)"?  Is it saved as soon as I switch to the genlet?  Are the caller/callee 
genlets symmetrical? Is there a concept of "main genlet" or something like that?

thanks,
Anto

From arigo at tunes.org  Fri Aug 12 14:49:47 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 12 Aug 2011 14:49:47 +0200
Subject: [pypy-dev] Stacklets
In-Reply-To: <4E451B04.6050104@gmail.com>
References: <CAMSv6X2+A51ubsC5-79kQhEUpPi1AMcJVkymu2ccTT=qR5rJOA@mail.gmail.com>
	<CAMSv6X256kE-Qk3F8Z0Kp7HCLWv6CDqBtFBphzndX7ux=iXYoQ@mail.gmail.com>
	<CAMSv6X08ssPtxnY4f6HjvWuzBDkTrjHmBa9-a3w-gPWrNedSZw@mail.gmail.com>
	<4E451B04.6050104@gmail.com>
Message-ID: <CAMSv6X2RYr1DkiUtEva8c9qLqvqLbXbLGREn4rtsSK-767v+uw@mail.gmail.com>

Hi Anto,

On Fri, Aug 12, 2011 at 2:22 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Are the caller/callee genlets symmetrical?

Yes; gen.next() is exactly like gen.send(None), just like with
generators.  Well it's maybe just rambling and the interface will
change again next hour, but the idea is as follows.

If you try to build "local" properties that define how the stack of
frames is supposed to look like, you end up with (assuming only one
thread):

- there is one bottommost frame, and one topmost frame;

- every frame apart the bottommost one has one predecessor (f_back),
and every frame apart the topmost one has one successor.

Looks completely obvious, but actually it's not.  With the properties
described above, you can't forbid a situation where you have one
normal-looking stack plus N detached "cycles" of frames, where
following f_back or the other way around ends up in a circle.  (Here I
get my math logic hat: actually you can't give so-called "first-order
properties" that would prevent this case; you can only prevent it by
adding properties that talk about sets of frames, as opposed to single
frames.)

So, from this strange observation comes "gen.send()".  What it does is
change the frames in such a way that the two properties described
above are preserved.  More precisely, "gen.send()" takes the frame f
that "gen" references (which is always the bottommost frame in the
generator), and exchanges f.f_back with the topmost frame.  After
doing that, f.f_back points to what used to be the top frame, and the
new top frame is the old f.f_back.  It's a "local" transformation that
respects the two properties above.  That's how calling "gen.send(i)"
in the generator causes the generator to be suspended: we jump to
f.f_back.  Then the generator frame has a cyclic f_back chain, i.e.
f.f_back points to the topmost frame of the generator itself.  Then
when later we call again "gen.send()", the same exchange is done
again, and we jump to f.f_back, which is this time the topmost frame
of the generator.

If you compare that with Python's normal generators, the only
difference (apart from supporting multiple-frames generators) is that
in a suspended generator frame, the f_back is not set to None, but to
a cycle.

I agree that this is arguably completely obscure; well it was nice
from the logician me :-)

At least it has good properties, e.g. it generalizes without problem
to multiple threads --- just say that when there are N threads
running, there are N frames without an f_back and N frames which are
not the f_back of another.  It clearly points out (in theory) the fact
that the detached circles of frames don't belong to any particular
thread.


A bient?t,

Armin.

From anto.cuni at gmail.com  Fri Aug 12 14:51:36 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 12 Aug 2011 14:51:36 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
 [dict.setdefault]
In-Reply-To: <201108102127.13752.naylor.b.david@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
Message-ID: <4E4521D8.1050507@gmail.com>

Hello David,

On 10/08/11 21:27, David Naylor wrote:
> Hi,
>
> I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache.  For comparison I put together
> the following test:
>
[cut]
> Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup.  To check what is the cause I replaced i.date()
> with i.day and found no major difference in times.  It appears dict.setdefault (or it's interaction with jit) is causing a slow down.

I don't think that setdefault is the culprit here, as shown by this benchmark:

@bench.bench
def setdef():
     d = {}
     for i in range(10000000):
         d.setdefault(i, i)
     return d

@bench.bench
def tryexcept():
     d = {}
     for i in range(10000000):
         try:
             d[i]
         except KeyError:
             d[i] = i
     return d

setdef()
tryexcept()

$ python dictbench.py
setdef: 2.03 seconds
tryexcept: 8.54 seconds

tmp $ pypy-c dictbench.py
setdef: 1.31 seconds
tryexcept: 1.37 seconds

as you can see, in PyPy there is almost no difference between using a 
try/except or using setdefault.


What is very slow on PyPy seems to be hashing datetime objects:

import datetime

@bench.bench
def hashdate():
     res = 0
     for i in range(100000):
         now = datetime.datetime.now()
         res ^= hash(now)
     return res

hashdate()

$ pypy-c dictbench.py
hashdate: 0.83 seconds

$ python dictbench.py
hashdate: 0.22 seconds

I had a quick look at the code (which in PyPy is written at applevel) and it 
does a lot of nonsense.  In particular, __hash__ calls __getstate which 
formats a dynamically created string, just to call hash() on it.  I suppose 
that this code can (and should) be optimized a lot.  I may try to look at it 
but it's unclear when, since I'm about to go on vacation.

ciao,
Anto

From anto.cuni at gmail.com  Fri Aug 12 14:57:55 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 12 Aug 2011 14:57:55 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
 [dict.setdefault]
In-Reply-To: <4E4521D8.1050507@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<4E4521D8.1050507@gmail.com>
Message-ID: <4E452353.7060807@gmail.com>

On 12/08/11 14:51, Antonio Cuni wrote:
> @bench.bench

for reference, here is the implementation of the "bench" decorator:
https://bitbucket.org/antocuni/env/src/1b11491fab79/pypath/bench.py

From dje.gcc at gmail.com  Fri Aug 12 15:41:17 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Fri, 12 Aug 2011 09:41:17 -0400
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <4E443C55.3070800@uni-duesseldorf.de>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
	<CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>
	<4E443C55.3070800@uni-duesseldorf.de>
Message-ID: <CAGWvnykOsQ0L6AxoJ4AHbMAM17hvC21ygEbRjc3sekVqtVjbcQ@mail.gmail.com>

On Thu, Aug 11, 2011 at 4:32 PM, Sven Hager
<Sven.Hager at uni-duesseldorf.de> wrote:

> I think it is okay to add an attribute to class Field which
> explicitly allows overlapping bits. It is very easy to adapt
> the existing code in form.py to this.

Sven,

Attached is a patch that fixes the rldicXXX shift by adding an
"overlap" argument to Field.

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff3
Type: application/octet-stream
Size: 3410 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110812/24b779ac/attachment-0001.obj>

From Sven.Hager at uni-duesseldorf.de  Fri Aug 12 15:51:15 2011
From: Sven.Hager at uni-duesseldorf.de (Sven Hager)
Date: Fri, 12 Aug 2011 15:51:15 +0200
Subject: [pypy-dev] PPC backend design questions
In-Reply-To: <CAGWvnykOsQ0L6AxoJ4AHbMAM17hvC21ygEbRjc3sekVqtVjbcQ@mail.gmail.com>
References: <CAGWvnykOApQmAed88r4Nz+t3-mFnNpMkn_FEzY2sjJ2ivti+5Q@mail.gmail.com>
	<CAFXevfeo1g2Y9rroeuNehN+sGhGdkZ6ijS8mSgUBaLy_VfSp3w@mail.gmail.com>
	<CAGWvnykYw-1FPk2tsO3Q7z0vV7gqc5ebAH5s1VkAq-Fqye78Ew@mail.gmail.com>
	<CAFXevffbB_hSUFvROfEuiyZG6pHjJZhFOgaPwLdReKHj-9w4ng@mail.gmail.com>
	<CAGWvnyks0FxtjaUH9vM5859ENN_GJbxb8j0C-KiWKkx4trKe4w@mail.gmail.com>
	<4E443C55.3070800@uni-duesseldorf.de>
	<CAGWvnykOsQ0L6AxoJ4AHbMAM17hvC21ygEbRjc3sekVqtVjbcQ@mail.gmail.com>
Message-ID: <4E452FD3.2040703@uni-duesseldorf.de>

On 08/12/2011 03:41 PM, David Edelsohn wrote:
> On Thu, Aug 11, 2011 at 4:32 PM, Sven Hager
> <Sven.Hager at uni-duesseldorf.de>  wrote:
>
>> I think it is okay to add an attribute to class Field which
>> explicitly allows overlapping bits. It is very easy to adapt
>> the existing code in form.py to this.
> Sven,
>
> Attached is a patch that fixes the rldicXXX shift by adding an
> "overlap" argument to Field.
>
> Thanks, David
Hi David,

thank you.
I will add your patches this evening.

Best regards,
Sven

From amauryfa at gmail.com  Fri Aug 12 17:02:44 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 12 Aug 2011 17:02:44 +0200
Subject: [pypy-dev] Testing pypy
In-Reply-To: <loom.20110810T113354-170@post.gmane.org>
References: <loom.20110810T091341-451@post.gmane.org>
	<CAMSv6X0mG4AgooPVdUqaF2rOcqqWGC1f6wKDOyr9gRAqjf13AA@mail.gmail.com>
	<loom.20110810T113354-170@post.gmane.org>
Message-ID: <CAGmFidZf9GG7BPcYLcotjS=PpwB2pByMDADembcYcSDvxeus0g@mail.gmail.com>

2011/8/10 Seung Soo, <sungsuha at gmail.com>

> Anyway I'd like to know if there is any single way to test both
> python and pypy(2.7), where the results should be the same.


I routinely run tests like this:

pypy -m test.regrtest -v test_pickle

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110812/862f5a4e/attachment.html>

From naylor.b.david at gmail.com  Fri Aug 12 17:49:54 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Fri, 12 Aug 2011 17:49:54 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <4E4521D8.1050507@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<4E4521D8.1050507@gmail.com>
Message-ID: <201108121749.57954.naylor.b.david@gmail.com>

On Friday, 12 August 2011 14:51:36 Antonio Cuni wrote:
> Hello David,
> 
> On 10/08/11 21:27, David Naylor wrote:
> > Hi,
> > 
> > I needed to create a cache of date and time objects and I wondered what
> > was the best way to handle the cache.  For comparison I put together
> 
> > the following test:
> [cut]
> 
> > Pypy displays significant slowdown in the defaultdict function, otherwise
> > displays its usual speedup.  To check what is the cause I replaced
> > i.date() with i.day and found no major difference in times.  It appears
> > dict.setdefault (or it's interaction with jit) is causing a slow down.
> 
> I don't think that setdefault is the culprit here, as shown by this
> benchmark:

I made a mistake in the script, I intended to use the day number as the key 
(as was in the case of the other tests).  Changing the critical line to 
"b.append(cache.setdefault(i.day, i.date()))" gives comparable results to 
python (factoring in the speed up).  

<snip/>

> as you can see, in PyPy there is almost no difference between using a
> try/except or using setdefault.

I would argue that with an expensive value operation the try/except will be 
much faster than setdefault (where cache hits are above 0).  Using my fixed 
script I get:

keydict: [0.3298788070678711, 0.28450703620910645, 0.28931379318237305]
defaultdict: [0.47180604934692383, 0.4183311462402344, 0.4172670841217041]
 
indicating setdefault is about 40% slower (1.4 times slower), which I 
attribute to the cost of i.date().  

> I had a quick look at the code (which in PyPy is written at applevel) and
> it does a lot of nonsense.  In particular, __hash__ calls __getstate which
> formats a dynamically created string, just to call hash() on it.  I
> suppose that this code can (and should) be optimized a lot.  I may try to
> look at it but it's unclear when, since I'm about to go on vacation.

Would it not be a simple matter of changing the __(get|set)state method to use 
a tuple or even an int(long)?  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110812/af09aae4/attachment.pgp>

From wlavrijsen at lbl.gov  Fri Aug 12 18:10:31 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 12 Aug 2011 09:10:31 -0700 (PDT)
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
In-Reply-To: <201108121132.46897.vincent@shishkabab.net>
References: <201108121132.46897.vincent@shishkabab.net>
Message-ID: <alpine.LNX.2.00.1108120902380.7325@pc3-wlav.lbl.gov>

Hi Vincent,

> * Using Clang to export data from a C/C++ header file to XML (I tried
>   GCCXML, but it would ignore certain things)

several years ago, I did the something similar (more limited) based on gccxml,
so I'm wondering what are the parts that you found ignored by that tool?

An alternative to using clang to produce XML, is to use its PCH directly. This
is the approach taken by cling (a C++ interpreter).

> - Generate a C API to C++ code if necessary.

In cppyy, we keep the generated reflection information available, rather
than wrapping the C++ into C. The advantage of that approach is that the
complex C++ types can than be "unboxed" for the JIT. The disadvantage is,
of course, that it's much more work to deliver and that is taking a long
time, but we're getting there.

> - Generate another XML file describing the Python API (which calls should
>   be made where, etc).

This is like the SWIG .i files? Or am I misunderstanding? An alternative is
to add rules to your XML generator for common mappings (eg. C++ STL iterator
can be mapped on Python's iter protocol).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From arigo at tunes.org  Sat Aug 13 18:25:47 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 13 Aug 2011 18:25:47 +0200
Subject: [pypy-dev] Status of ARM backend
In-Reply-To: <CAFKq6r+P+mUsSN4g59cTVtLi5y3jJKrG-jX8-nmJNFuWD+gxOA@mail.gmail.com>
References: <CAFKq6r+P+mUsSN4g59cTVtLi5y3jJKrG-jX8-nmJNFuWD+gxOA@mail.gmail.com>
Message-ID: <CAMSv6X28-yHGeWy_gF3+dRgt9dRsc10roGmOro-PTDzmMw_7Mg@mail.gmail.com>

Hi Eric,

On Mon, Aug 1, 2011 at 8:07 AM, Eric van Riet Paap <eric at vanrietpaap.nl> wrote:
> Can someone please give us an update on the status of the ARM backend.

I will answer even though I'm not the most up-to-date person.  It is
kind of ready, but it needs a champion.  That means that we need one
of two things to occur: either someone donates an ARM buildslave for
buildbot, and takes care of setting it up so that translation works
(so it must have either 2 GB of RAM, or be entirely emulated on
another machine); or someone that cares a lot about ARM steps in and
does the necessary maintenance, even if setting up a 2GB ARM
buildslave is too messy.

Without any of these things happening, I fear that it will just rot,
until someone that cares shows up.


A bient?t,

Armin.

From anto.cuni at gmail.com  Sat Aug 13 18:32:58 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sat, 13 Aug 2011 18:32:58 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
 [dict.setdefault]
In-Reply-To: <201108121749.57954.naylor.b.david@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<4E4521D8.1050507@gmail.com>
	<201108121749.57954.naylor.b.david@gmail.com>
Message-ID: <4E46A73A.9000405@gmail.com>

On 12/08/11 17:49, David Naylor wrote:

> Would it not be a simple matter of changing the __(get|set)state method to use
> a tuple or even an int(long)?

yes, I think it should be enough. I'm going on vacation soon and I won't have 
a look at it right now, so if anybody wants to work on it, he's very welcome 
(hint, hint :-)).

ciao,
Anto

From naylor.b.david at gmail.com  Sat Aug 13 20:14:20 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Sat, 13 Aug 2011 20:14:20 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <4E46A73A.9000405@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108121749.57954.naylor.b.david@gmail.com>
	<4E46A73A.9000405@gmail.com>
Message-ID: <201108132014.25161.naylor.b.david@gmail.com>

On Saturday, 13 August 2011 18:32:58 Antonio Cuni wrote:
> On 12/08/11 17:49, David Naylor wrote:
> > Would it not be a simple matter of changing the __(get|set)state method
> > to use a tuple or even an int(long)?
> 
> yes, I think it should be enough. I'm going on vacation soon and I won't
> have a look at it right now, so if anybody wants to work on it, he's very
> welcome (hint, hint :-)).

See attached for my naive attempt (and I did not run any unit tests on the code).  It provides between 4.5x to 13.4x improvement in hash speed.  If 
method 1 is acceptable I could properly implement it.  

If you look at the __hash__ method for datetime you will notice three return statements.  The performance of those statements are as follows, 
based on:

@bench.bench
def hashdate():
     res = 0
     for i in range(10000000):
         now = datetime.datetime(i // 10000 + 1, (i % 10000) % 12 + 1, (i % 100) % 28 + 1)
         res ^= hash(now)
     return res

hashdate()

Method 1 (direct integer compute):
hashdate: 0.70 seconds

Method 2 (hash of __getstate()):
hashdate: 2.39 seconds

Method 3 (unity):
hashdate: 0.68 seconds

Method 4 (original):
hashdate: 10.93 seconds (python: 12.60 seconds)

And back to my original "benchmark" with the change of `key = i`:

# python iforkey.py
ifdict: [2.8676719665527344, 2.872897148132324, 2.8396730422973633]
keydict: [2.3266799449920654, 2.3431849479675293, 2.3421859741210938]
defaultdict: [3.706634044647217, 3.6940698623657227, 3.7520179748535156]

# pypy iforkey.py (original)
ifdict: [29.201794147491455, 29.047310829162598, 29.34461998939514]
keydict: [14.939809083938599, 15.250468015670776, 15.542209148406982]
defaultdict: [15.11891484260559, 15.064191102981567, 14.94817304611206]

# pypy iforkey (method 1)
ifdict: [7.455403804779053, 7.376722097396851, 7.447360038757324]
keydict: [3.9056499004364014, 3.833178997039795, 3.8482401371002197]
defaultdict: [3.9568910598754883, 3.8757669925689697, 3.88435697555542]

# pypy iforkey.py (method 2)
ifdict: [11.993246078491211, 11.865861892700195, 11.916783094406128]
keydict: [6.141685962677002, 6.092236042022705, 6.082683086395264]
defaultdict: [6.376708030700684, 6.337490081787109, 6.361854791641235]

So, it appears pypy is failing to speed up this contrived example...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: datetime.diff
Type: text/x-patch
Size: 5791 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110813/ed6fff4c/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110813/ed6fff4c/attachment.pgp>

From anto.cuni at gmail.com  Sun Aug 14 10:50:14 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 14 Aug 2011 10:50:14 +0200
Subject: [pypy-dev] [pypy-commit] pypy default: Fix a bug in
 _sqlite3.py: memory is freed too early
In-Reply-To: <20110813201523.B39A8829F6@wyvern.cs.uni-duesseldorf.de>
References: <20110813201523.B39A8829F6@wyvern.cs.uni-duesseldorf.de>
Message-ID: <4E478C46.4050403@gmail.com>

On 13/08/11 22:15, arigo wrote:

> Log:	Fix a bug in _sqlite3.py: memory is freed too early
>
> diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py
> --- a/lib_pypy/_sqlite3.py
> +++ b/lib_pypy/_sqlite3.py
> @@ -891,7 +891,8 @@
>
>           self.statement = c_void_p()
>           next_char = c_char_p()
> -        ret = sqlite.sqlite3_prepare_v2(self.con.db, sql, -1, byref(self.statement), byref(next_char))
> +        sql_char = c_char_p(sql)
> +        ret = sqlite.sqlite3_prepare_v2(self.con.db, sql_char, -1, byref(self.statement), byref(next_char))
>           if ret == SQLITE_OK and self.statement.value is None:
>               # an empty statement, we work around that, as it's the least trouble
>               ret = sqlite.sqlite3_prepare_v2(self.con.db, "select 42", -1, byref(self.statement), byref(next_char))
> @@ -903,6 +904,7 @@
>           if _check_remaining_sql(next_char.value):
>               raise Warning, "One and only one statement required: %r" % (
>                   next_char.value,)
> +        # sql_char should remain alive until here
>
>           self._build_row_cast_map()

why is this needed? Does the sqlite API requires that the SQL buffer stays 
alive even after the call to sqlite3_prepare?

ciao,
Anto

From timonator at perpetuum-immobile.de  Sun Aug 14 18:48:25 2011
From: timonator at perpetuum-immobile.de (Timo Paulssen)
Date: Sun, 14 Aug 2011 18:48:25 +0200
Subject: [pypy-dev] Efficiently piecing together callables with hooks?
Message-ID: <4E47FC59.9050207@perpetuum-immobile.de>

Hello List,

I'm currently experimenting with a piece of software I've recently started
maintaining, which is a cellular automaton simulation software. It's supposed to
make experimenting easy, but still be fast at simulating. As you might know, all
cellular automatons operate in steps, where usually the whole "configuration
space" is iterated over and all fields are updated depending on their
neighbourhood and the "local transition function". Currently, the step functions
are implemented in C and inlined with weave.inline.

The experiment I just tried out was generating step functions by piecing
together strings of C code. It was quite successful, but I would like to take
that one step further and use python code instead of having the user write C
code that has to be written with weave in mind.

The first attempt at making it work with python code was to put python callables
into lists and running them somewhat like this:

def step_pure_py(self):
    runhooks("init")
    loop_iter = self.loop.get_iter()
    for pos in loop_iter:
        runhooks("pre_compute")
        runhooks("compute")
        runhooks("post_compute")
    runhooks("after_step")

This is utterly slow, because - even though i turned the lists that contain the
hooks into tuples - i have no way to tell pypy, that the lists are not going to
be changed and all loops that handle them can safely be unrolled.

Unrolling the lists by hand gives me insane speedups. These are timings for
10.000 steps on a 1000 field array - that is calling step_pure_py 1000 times and
having the loop_iter iterate over 1000 fields each time.

hand-unrolled python code:
2.15user 0.01system 0:02.18elapsed 99%CPU (0avgtext+0avgdata 103024maxresident)k

hook based solution like above:
62.57user 0.05system 1:02.79elapsed 99%CPU (0avgtext+0avgdata 184576maxresident)k

So what would you suggest for making this much faster? I read on the IRC, that
in pypy itself, string concatenation is actually used to create python
functions, so maybe that would be way to go in this case? Or could pypy offer
decorators or other hints to cause pypy to unroll and constant-fold different
things for me?

Thank you
  - Timo Paulssen

From arigo at tunes.org  Mon Aug 15 08:49:26 2011
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 15 Aug 2011 08:49:26 +0200
Subject: [pypy-dev] [pypy-commit] pypy default: Fix a bug in
 _sqlite3.py: memory is freed too early
In-Reply-To: <4E478C46.4050403@gmail.com>
References: <20110813201523.B39A8829F6@wyvern.cs.uni-duesseldorf.de>
	<4E478C46.4050403@gmail.com>
Message-ID: <CAMSv6X3rVGvRtez4fE4rcnrFUG_NA6ixQaeHtjLeoMwJ4g3swA@mail.gmail.com>

Hi Anto,

On Sun, Aug 14, 2011 at 10:50 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> why is this needed? Does the sqlite API requires that the SQL buffer stays
> alive even after the call to sqlite3_prepare?

Yes, because the output argument 'next_value' receives a pointer to
the middle of the same buffer that was passed in as 'sql'.  So passing
'sql' as a Python string doesn't work: it gets converted to a C string
when passed to the function, and discarded after the call.  It works
anyway on CPython because the C string pointer is actually inside the
PyStringObject, which happens to stay alive.


A bient?t,

Armin.

From arigo at tunes.org  Mon Aug 15 09:05:22 2011
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 15 Aug 2011 09:05:22 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <201108132014.25161.naylor.b.david@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108121749.57954.naylor.b.david@gmail.com>
	<4E46A73A.9000405@gmail.com>
	<201108132014.25161.naylor.b.david@gmail.com>
Message-ID: <CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>

Hi David,

On Sat, Aug 13, 2011 at 8:14 PM, David Naylor <naylor.b.david at gmail.com> wrote:
> So, it appears pypy is failing to speed up this contrived example...

I think that it is expected, because the hash is computed entirely as
pure Python code in the case of PyPy (doing integer arithmetic with
overflow checks) whereas it is written in C in the case of RPython.  I
find it rather good that we get overall close to the speed of CPython,
given that we just have datetime.py...  Try using datetime.py on top
of CPython too :-)

More seriously, if we want to be really better with datetime objects,
as opposed to "good enough for now", then I fear we need to rewrite
datetime.py in RPython, like CPython did rewrite it in C.  Or some
intermediate solution is possible too, having an internal class which
implements only basic operations (like hash) and which is subclassed
in pure Python.  But I would happily leave that task to someone with
an absolutely real need for high-performance datetime objects; for me
"good enough for now" is good enough, for now...


A bient?t,

Armin.

From neppord at gmail.com  Mon Aug 15 10:18:26 2011
From: neppord at gmail.com (Samuel Ytterbrink)
Date: Mon, 15 Aug 2011 10:18:26 +0200
Subject: [pypy-dev] Efficiently piecing together callables with hooks?
In-Reply-To: <4E47FC59.9050207@perpetuum-immobile.de>
References: <4E47FC59.9050207@perpetuum-immobile.de>
Message-ID: <CABpX6DBVGThdi7+85_=7TYKf_abFHSmmyNAdAGo5f1zK=yTMLQ@mail.gmail.com>

Hi, i only have some questions about your example code. How does the
runhooks function work? And what are you using the pos loop variable for?
Den 14 aug 2011 18.58 skrev "Timo Paulssen" <timonator at perpetuum-immobile.de
>:
> Hello List,
>
> I'm currently experimenting with a piece of software I've recently started
> maintaining, which is a cellular automaton simulation software. It's
supposed to
> make experimenting easy, but still be fast at simulating. As you might
know, all
> cellular automatons operate in steps, where usually the whole
"configuration
> space" is iterated over and all fields are updated depending on their
> neighbourhood and the "local transition function". Currently, the step
functions
> are implemented in C and inlined with weave.inline.
>
> The experiment I just tried out was generating step functions by piecing
> together strings of C code. It was quite successful, but I would like to
take
> that one step further and use python code instead of having the user write
C
> code that has to be written with weave in mind.
>
> The first attempt at making it work with python code was to put python
callables
> into lists and running them somewhat like this:
>
> def step_pure_py(self):
> runhooks("init")
> loop_iter = self.loop.get_iter()
> for pos in loop_iter:
> runhooks("pre_compute")
> runhooks("compute")
> runhooks("post_compute")
> runhooks("after_step")
>
> This is utterly slow, because - even though i turned the lists that
contain the
> hooks into tuples - i have no way to tell pypy, that the lists are not
going to
> be changed and all loops that handle them can safely be unrolled.
>
> Unrolling the lists by hand gives me insane speedups. These are timings
for
> 10.000 steps on a 1000 field array - that is calling step_pure_py 1000
times and
> having the loop_iter iterate over 1000 fields each time.
>
> hand-unrolled python code:
> 2.15user 0.01system 0:02.18elapsed 99%CPU (0avgtext+0avgdata
103024maxresident)k
>
> hook based solution like above:
> 62.57user 0.05system 1:02.79elapsed 99%CPU (0avgtext+0avgdata
184576maxresident)k
>
> So what would you suggest for making this much faster? I read on the IRC,
that
> in pypy itself, string concatenation is actually used to create python
> functions, so maybe that would be way to go in this case? Or could pypy
offer
> decorators or other hints to cause pypy to unroll and constant-fold
different
> things for me?
>
> Thank you
> - Timo Paulssen
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/1641dbcc/attachment.html>

From vincent at shishkabab.net  Mon Aug 15 12:03:17 2011
From: vincent at shishkabab.net (Vincent den Boer)
Date: Mon, 15 Aug 2011 12:03:17 +0200
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
In-Reply-To: <alpine.LNX.2.00.1108120902380.7325@pc3-wlav.lbl.gov>
References: <201108121132.46897.vincent@shishkabab.net>
	<alpine.LNX.2.00.1108120902380.7325@pc3-wlav.lbl.gov>
Message-ID: <201108151203.18385.vincent@shishkabab.net>

Hello Wim,
>
> several years ago, I did the something similar (more limited) based on
> gccxml, so I'm wondering what are the parts that you found ignored by that
> tool?
>

Somehow, for some classes their copy constructor was exported, but their 
'normal' constructors were not. I couldn't find the cause, but when I parsed the 
.cpp implementation files individually, the missing constructors showed up again. 
Because it wouldn't have been very practical to require the user to have access 
to the .cpp files, and because one big XML was easier for me to process, I 
decided to create the Clang plugin.

> An alternative to using clang to produce XML, is to use its PCH directly.
> This is the approach taken by cling (a C++ interpreter).

That would be nice, because that would mean less C++ code to maintain and users 
wouldn't need to go through the trouble of getting the Clang plugin running. 
However, it would require me to write code to parse those files. Maybe I could 
also use this:
https://llvm.org/svn/llvm-project/cfe/trunk/bindings/python/clang/cindex.py
which is the ctypes-based API to
http://clang.llvm.org/doxygen/Index_8h_source.html

> In cppyy, we keep the generated reflection information available, rather
> than wrapping the C++ into C. The advantage of that approach is that the
> complex C++ types can than be "unboxed" for the JIT. The disadvantage is,
> of course, that it's much more work to deliver and that is taking a long
> time, but we're getting there.

I've read about it in the PyPy blog, but since I didn't hear anything about it 
for a long time I forgot about it. It seems like we have two different goals. 
While cppyy purely focuses on C++, Wrappyr tries to make the path from C or C++ 
to a working Python API (compatible with any Python implementation with a 
working ctypes module) as short and simple as possible.

> This is like the SWIG .i files? Or am I misunderstanding? An alternative is
> to add rules to your XML generator for common mappings (eg. C++ STL
> iterator can be mapped on Python's iter protocol).

Those XML file are just helpers for working with ctypes. See:
https://github.com/ShishKabab/Wrappyr/blob/master/ctypes_builder/README.rst

While SWIG knows what C and C++ is, this XML file allows you to group arbitrary C 
calls into classes and describe how the resulting modules and packages should be 
organized. This has advantages over generating the ctypes code directly from the 
Clang data conversion script. The first is that it is easier to generate since 
you don't have to worry about the code itself. The second is that you can write 
scripts to preprocess the data in the XML programmatically, so you can do things 
like grouping functions into classes (which helps when the XML was generated 
from C code).

As for mapping C/C++ idioms to Python (like the __iter__ protocol), the problem 
is that many libraries have their own way of doing things. Many libraries for 
example have their own string classes/functions, iterators, hash tables, etc. It 
would be nice have a way to indicate those relationships. However, I consider 
this a lower priority for now.

Kind regards,
Vincent

From arigo at tunes.org  Mon Aug 15 12:48:09 2011
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 15 Aug 2011 12:48:09 +0200
Subject: [pypy-dev] Efficiently piecing together callables with hooks?
In-Reply-To: <4E47FC59.9050207@perpetuum-immobile.de>
References: <4E47FC59.9050207@perpetuum-immobile.de>
Message-ID: <CAMSv6X3Fig-rAh53v6goEG1MpGEcGt0myeX8MPMNRoHUnSEPXQ@mail.gmail.com>

Hi Timo,

On Sun, Aug 14, 2011 at 6:48 PM, Timo Paulssen
<timonator at perpetuum-immobile.de> wrote:
>
> So what would you suggest for making this much faster? I read on the IRC, that
> in pypy itself, string concatenation is actually used to create python
> functions, so maybe that would be way to go in this case?

Sorry, I must correct myself.  Using strings to make a function's
source code is used as the "last resort" method; what we usually do is
just make and return local functions, with "global" variables actually
coming from the enclosing scope.  However, at the level of pure
Python, this will not have the effect that you want, because there is
still only one code object.

> Or could pypy offer decorators or other hints to cause pypy
> to unroll and constant-fold different things for me?

Carl Friedrich has played a bit with exposing hints in order to write
interpreters at app-level, but it's not ready for widespread usage.
Right now building new custom code objects is the only solution I can
think of (either by string concatenation, or directly using the ast
module).


A bient?t,

Armin.

From ian at ianozsvald.com  Mon Aug 15 15:03:53 2011
From: ian at ianozsvald.com (Ian Ozsvald)
Date: Mon, 15 Aug 2011 14:03:53 +0100
Subject: [pypy-dev] bounties for pypy
In-Reply-To: <BANLkTikaEmhen8dgRpwXsB_Pzr7kvRXDrg@mail.gmail.com>
References: <BANLkTi=40=RB_J4-mh-Jg4BDf3BTohTW+Q@mail.gmail.com>
	<201106281751.p5SHpixC014361@theraft.openend.se> <chef@ghum.de>
	<BANLkTikUcsjnS1MjX6jfS4x64Ug0ZYut1Q@mail.gmail.com>
	<201106290753.p5T7rMKh002901@theraft.openend.se>
	<BANLkTikaEmhen8dgRpwXsB_Pzr7kvRXDrg@mail.gmail.com>
Message-ID: <CAPvwAND90j9YWW13=htsdhPDfT8=HFcj8iHG2CcYmnUE1SE2eg@mail.gmail.com>

I'll re-open this earlier thread...Didrik Pinte of Enthought is
offering to match my suggested ?600 donation towards a numpy-pypy
project. He's offering it in a personal capacity (i.e. not as an
Enthought activity) out of his own pocket (and my donation would come
out of my Consultancy). His offer came out of a discussion on pypy's
merits at the London Financial Python Usergroup meet a few weeks back.

So, you've got ?1,200 via two individuals ready as a gift towards
numpy integration, as/when someone can use the money. I won't promise
that my gift will always be available as my personal situation may be
changing in the next 6 weeks (so please - if you can use it - ask for
it soon!).

Regards,
Ian (UK)

On 29 June 2011 14:16, Ian Ozsvald <ian at ianozsvald.com> wrote:
> I'm glad this thread is up. Laura - I'm the chap from Armin's talk who
> offered a monthly retainer for a year towards numpy integration (in my
> mind I'm offering ?50/month for 12 months). I spoke to you later and
> you mentioned the flattr site but having to do it each month is a bit
> of a pain (I know it is simple but I don't want to think about it...).
>
> So, for the record, I have ?600 sitting here with someone's name on
> it, I'll account for it as a marketing expense or something out of my
> company. I'm a one man consultancy, PyPy doesn't directly help me as
> I'm an A.I./science researcher (so I need numpy, IPython, matplotlib
> etc) but I believe strongly that it will help all of Python (and me in
> part) over time, so it is worth pledging some of my earnings towards
> the goal of eventual numpy integration.
>
> If I can pledge it to someone or a project then that's cool, if I
> should just move the money to someone's account then that's cool too.
> I'm quite happy to have my name down as ContributorNumber1ForNumpy if
> it helps you spread the word.
>
> Ian.
> ps. I posted the v0.1 PDF of my High Performance Python tutorial this
> morning (it is based on my EuroPython training session). It has a
> section on PyPy and I'd happily accept input if that section should be
> expanded: http://ianozsvald.com/2011/06/29/high-performance-pyethon-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
>
> On 29 June 2011 08:53, Laura Creighton <lac at openend.se> wrote:
>>
>>>The idea was also to possibly attract new developers ... for example, if
>>>there would be "10 days in money" for adapting py2exe, I am sure many wou
>>>ld
>>>jump to solve this puzzle.
>>
>> This is sort of a bad example. ?Because py2exe embeds CPython, and
>> we wouldn't want to do that. ?So what we would probably want to do is
>> to make some general tool that willmake a windows binary, or a
>> mac one, and get rid of the need for bzfreeze and friends. ?So now
>> you are looking at a general embedding solution, and that is more
>> than 10 days worth of work.
>>
>> But I get ?the idea.
>>
>> <snip>
>>>> my dream was of a trustee service: after somebody commits to do the wor
>>>k,
>>>the pledgers have to pay to a trustee. then the work is done. then the
>>>trustee pays the worker.
>>
>> This is one of the things I want to talk with fundedbyme about. ?But
>> having an explicit trustee is a new idea. ?I think the pypy core
>> developers are already rather well trusted in this community, but
>> this may be important to new developers who aren't as well known.
>> And it handles the problem' of 'I got sick and cannot do this ?any
>> more' more gracefully than other solutions.
>>
>>>Hmmm.... a structure could be:
>>>
>>>- service provider does the technical stuff, as in:
>>> ? # website
>>> ? # collect pledges
>>> ? # handle project description
>>> ? # collect money
>>> ? # distribute money after feature completion
>>
>> fundedbyme has sort of indicated an interst in doing this (except
>> they were talking about distribution before, and I was leaving
>> project description to the project, not outsiders). ?I will follow
>> up on this when I get back home to Sweden.
>>
>>>- PSF / pypy-foundation / whateverfoundation provides the trust
>>>
>>>Thanks for confirming the need for such a thing!
>>>
>>>Harald
>>
>> Thanks once again for seeing a marketing solution that nerds like
>> us often miss.
>>
>> Laura
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> Ian Ozsvald (A.I. researcher, screencaster)
> ian at IanOzsvald.com
>
> http://IanOzsvald.com
> http://SocialTiesApp.com/
> http://MorConsulting.com/
> http://blog.AICookbook.com/
> http://TheScreencastingHandbook.com
> http://FivePoundApp.com/
> http://twitter.com/IanOzsvald
>



-- 
Ian Ozsvald (A.I. researcher, screencaster)
ian at IanOzsvald.com

http://IanOzsvald.com
http://SocialTiesApp.com/
http://MorConsulting.com/
http://blog.AICookbook.com/
http://TheScreencastingHandbook.com
http://FivePoundApp.com/
http://twitter.com/IanOzsvald

From alex.gaynor at gmail.com  Mon Aug 15 15:36:20 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 15 Aug 2011 08:36:20 -0500
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108121749.57954.naylor.b.david@gmail.com>
	<4E46A73A.9000405@gmail.com>
	<201108132014.25161.naylor.b.david@gmail.com>
	<CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
Message-ID: <CAFRnB2VN7BOn+vL__=sz3-MLEV+pPKDEB4pS0mD6kmNocQix_A@mail.gmail.com>

On Mon, Aug 15, 2011 at 2:05 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi David,
>
> On Sat, Aug 13, 2011 at 8:14 PM, David Naylor <naylor.b.david at gmail.com>
> wrote:
> > So, it appears pypy is failing to speed up this contrived example...
>
> I think that it is expected, because the hash is computed entirely as
> pure Python code in the case of PyPy (doing integer arithmetic with
> overflow checks) whereas it is written in C in the case of RPython.  I
> find it rather good that we get overall close to the speed of CPython,
> given that we just have datetime.py...  Try using datetime.py on top
> of CPython too :-)
>
> More seriously, if we want to be really better with datetime objects,
> as opposed to "good enough for now", then I fear we need to rewrite
> datetime.py in RPython, like CPython did rewrite it in C.  Or some
> intermediate solution is possible too, having an internal class which
> implements only basic operations (like hash) and which is subclassed
> in pure Python.  But I would happily leave that task to someone with
> an absolutely real need for high-performance datetime objects; for me
> "good enough for now" is good enough, for now...
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

I'd like to express that, unless we have a very compelling reason, we should
try to keep more stuff in pure python, as opposed to RPython.  Mostly
because it speeds up translation ;) (also easier to test, easier to write,
etc.).

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/f1301a73/attachment-0001.html>

From arigo at tunes.org  Mon Aug 15 15:42:50 2011
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 15 Aug 2011 15:42:50 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <CAFRnB2VN7BOn+vL__=sz3-MLEV+pPKDEB4pS0mD6kmNocQix_A@mail.gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108121749.57954.naylor.b.david@gmail.com>
	<4E46A73A.9000405@gmail.com>
	<201108132014.25161.naylor.b.david@gmail.com>
	<CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
	<CAFRnB2VN7BOn+vL__=sz3-MLEV+pPKDEB4pS0mD6kmNocQix_A@mail.gmail.com>
Message-ID: <CAMSv6X2ndBNgpck4ndtoWyZo8iFvQkAWdmL5_-+ozzqu9MyEGQ@mail.gmail.com>

Hi Alex,

On Mon, Aug 15, 2011 at 3:36 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> I'd like to express that, unless we have a very compelling reason, we should
> try to keep more stuff in pure python, as opposed to RPython. ?Mostly
> because it speeds up translation ;) (also easier to test, easier to write,
> etc.).

I agree.  I was really talking about someone needing absolutely best
performance for datetime objects.  I'm sure that getting "only
average" performance is more than enough for all use cases apart from
microbenchmarks.


A bient?t,

Armin.

From glavoie at gmail.com  Mon Aug 15 17:30:33 2011
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Mon, 15 Aug 2011 11:30:33 -0400
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108112018.01642.naylor.b.david@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108112018.01642.naylor.b.david@gmail.com>
Message-ID: <CAKp1Nw6e6UA-jLmxLf=iY_u28OK8HE+HmdDSZMiiE3eStGza_w@mail.gmail.com>

Hi David,
    for your information, I'm currently trying to make my FreeBSD buildbot
back online. It broke when the requirement for the "pypy" binary to build
PyPy appeared. It seems there are still some issues to make PyPy build again
under FreeBSD. I'll keep you updated.

Gabriel

2011/8/11 David Naylor <naylor.b.david at gmail.com>

> On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > Hi
> >
> > A little status update from my side.
> >
> > I'm trying to get 1.6 out of the door and there are a few things that
> > stand out before we can do it.
> >
> > * Windows situation (a lot of crashes with missing liveness). Also
> > lib-python tests take 10h which is a bit too much IMO (not a new
> > regression though)
> > * OS X situation (missing buildbots, being addressed now)
> > * A bunch of issues that either should be decided we don't care or we
> > do care and fix it.
> > * Missing jitviewer release/demo that I'm doing right now
> > * Alex reported sqlite crashes
> > * random segfault when running trackgcroot, armin, did you have a
> > look? I can do if you don't
>
> Would anyone be interested in the FreeBSD situation?  If so what support do
> you need?  I am currently updating my repository and will report back...
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/a455690f/attachment.html>

From naylor.b.david at gmail.com  Mon Aug 15 18:20:59 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Mon, 15 Aug 2011 18:20:59 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108132014.25161.naylor.b.david@gmail.com>
	<CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
Message-ID: <201108151821.02922.naylor.b.david@gmail.com>

On Monday, 15 August 2011 09:05:22 Armin Rigo wrote:
> Hi David,
> 
> On Sat, Aug 13, 2011 at 8:14 PM, David Naylor <naylor.b.david at gmail.com> 
wrote:
> > So, it appears pypy is failing to speed up this contrived example...
> 
> I think that it is expected, because the hash is computed entirely as
> pure Python code in the case of PyPy (doing integer arithmetic with
> overflow checks) whereas it is written in C in the case of RPython.  I
> find it rather good that we get overall close to the speed of CPython,
> given that we just have datetime.py...  Try using datetime.py on top
> of CPython too :-)
> 
> More seriously, if we want to be really better with datetime objects,
> as opposed to "good enough for now", then I fear we need to rewrite
> datetime.py in RPython, like CPython did rewrite it in C.  Or some
> intermediate solution is possible too, having an internal class which
> implements only basic operations (like hash) and which is subclassed
> in pure Python.  But I would happily leave that task to someone with
> an absolutely real need for high-performance datetime objects; for me
> "good enough for now" is good enough, for now...

For me the performance of datetime object's hashing is sufficient but I think 
the python code could use some performance improvements.  Is my approach using 
a direct computation to type long acceptable (in principle).  If so I can 
refine it and submit a patch.  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/0ca0bea2/attachment.pgp>

From wlavrijsen at lbl.gov  Mon Aug 15 19:02:57 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 15 Aug 2011 10:02:57 -0700 (PDT)
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
In-Reply-To: <201108151203.18385.vincent@shishkabab.net>
References: <201108121132.46897.vincent@shishkabab.net>
	<alpine.LNX.2.00.1108120902380.7325@pc3-wlav.lbl.gov>
	<201108151203.18385.vincent@shishkabab.net>
Message-ID: <alpine.LNX.2.00.1108150943270.5987@pc3-wlav.lbl.gov>

Hi Vincent,

> Somehow, for some classes their copy constructor was exported, but their
> 'normal' constructors were not.

hmm; I can't say that has been a problem for us (if I understand things
correctly). If it's not too much work (since you already departed from
gccxml), do you have an explicit class that exhibits the problem?

> That would be nice, because that would mean less C++ code to maintain and
> users wouldn't need to go through the trouble of getting the Clang plugin
> running. However, it would require me to write code to parse those files.

Yes, we don't parse the PCH ourselves either. That'd be too much. We use the
ASTs directly from clang, through clang/AST.

> I've read about it in the PyPy blog, but since I didn't hear anything about
> it for a long time I forgot about it.

Yes, we really need to give a status update. :} Especially since we recently
had a sprint in Dusseldorf.

> It seems like we have two different goals. While cppyy purely focuses on
> C++, Wrappyr tries to make the path from C or C++ to a working Python API
> (compatible with any Python implementation with a working ctypes module)
> as short and simple as possible.

I don't see those as differences per se: cppyy works with C as well, of
course, and the CPython implementation is already available (albeit packaged
in the ROOT class library we use here in HEP and several other places, but
ROOT is part of a handful of Linux distro's already, so it's not too hard to
get, and it is supposed to move to llvm by the Summer of next year as well).

As for easy access, yes that is the main goal. For the CPython side, we have
managed to get it down to a one-liner for end-users, either within the build
system or within the interactive environment. It doesn't get easier than
that. :} On the cppyy side, it's a work in progress, but the same techniques
apply, and the end-result should be one-liners again.

> Those XML file are just helpers for working with ctypes. See:
> https://github.com/ShishKabab/Wrappyr/blob/master/ctypes_builder/README.rst

That's looking nice. In our case, the same information is scattered over
several files (b/c of historic reasons and different use cases).

> As for mapping C/C++ idioms to Python (like the __iter__ protocol), the
> problem is that many libraries have their own way of doing things.

They do, but there are not that many different ways of doing the same thing
that are in actual use. Not even in C++. :) And it is here where a lot of the
ease-of-use can be gained, precisely since all information is available to
a parser of headers. For example, if a class has methods called begin() and
end() which return types that have dereference and increment or decrement
operators, it's clear what is going on. The same with the examples you name.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From dje.gcc at gmail.com  Mon Aug 15 19:12:11 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Mon, 15 Aug 2011 13:12:11 -0400
Subject: [pypy-dev] PPC64 JIT progress
Message-ID: <CAGWvnykpGnB8n6WyudteXussnafTGpPwQhOEGytMUg_aNJBh9Q@mail.gmail.com>

Attached is the next patch for the PPC JIT backend.

The patch adds shift-left simplified mnemonic (sldi) and a load
doubleword (load_dword) function that uses it.

I also discovered some typos in my previous patch to ppc_field.py.

Currently every PPC JIT testcase fails on PPC64 Linux because the PPC
backend makes an incorrect assumption about function pointers.  PPC64
Linux function pointer point to function descriptors, not code
addresses.  PPC64 Linux libffi knows this, which makes

    def get_function(self):
        i = self.code.materialize(AsmMemoryManager(), [])
        t = lltype.FuncType([], lltype.Signed)
        return rffi.cast(lltype.Ptr(t), i)

wrong for PPC64.

A function descriptor is a three-word data vector containing the
function address, data addressibility pointer and static chain.  For
some of the tests to start working on PPC64 Linux, the backend needs
to create another data object that contains the function address and
that pointer is coerced into a function pointer.

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff4
Type: application/octet-stream
Size: 2243 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/61d4c2fa/attachment-0001.obj>

From timonator at perpetuum-immobile.de  Mon Aug 15 19:43:50 2011
From: timonator at perpetuum-immobile.de (Timo Paulssen)
Date: Mon, 15 Aug 2011 19:43:50 +0200
Subject: [pypy-dev] Efficiently piecing together callables with hooks?
In-Reply-To: <CABpX6DBVGThdi7+85_=7TYKf_abFHSmmyNAdAGo5f1zK=yTMLQ@mail.gmail.com>
References: <4E47FC59.9050207@perpetuum-immobile.de>
	<CABpX6DBVGThdi7+85_=7TYKf_abFHSmmyNAdAGo5f1zK=yTMLQ@mail.gmail.com>
Message-ID: <4E495AD6.8020509@perpetuum-immobile.de>

On 15.08.2011 10:18, Samuel Ytterbrink wrote:
> Hi, i only have some questions about your example code. How does the
> runhooks function work? And what are you using the pos loop variable for?

Hi,

here's all the code you'll need to understand what's going on:


    def step_pure_py(self):
        state = self.consts.copy()
        # self.attrs usually includes nconf and cconf, which are the
        # arrays that hold the cellular automaton configuration that's currently
        # being computed (nconf) and the last one (cconf)
        state.update(dict((k, getattr(self.target, k)) for k in self.attrs))

        def runhooks(hook):
            for hook in self.pycode[hook]:
                upd = hook(state)
                if isinstance(upd, dict):
                    state.update(upd)

        runhooks("init")
        loop_iter = self.loop.get_iter()
        for pos in loop_iter:
            state["pos"] = pos
            state.update(self.neigh.get_neighbourhood(pos))
            runhooks("pre_compute")
            runhooks("compute")
            runhooks("post_compute")
        runhooks("after_step")
        # flip current and next config for the next step
        self.target.nconf, self.target.cconf = \
            self.target.cconf, self.target.nconf


    # one example for a hook:
    # rule is an array that defines when to set the next cell state to 0 or 1
    # with this, you can create all elementary cellular automatons.
    binRuleTestCode.add_py_hook("compute",
            lambda state: dict(result=state["rule"][int(state["l"] * 4 +
                                      state["m"] * 2 + state["r"])]))


the "l", "m" and "r" variables get set by the LinearNeighbourhood class, which
adds the following hook:

    self.code.add_py_hook("pre_compute",
            lambda state, code=self.code: dict(zip(self.names,
                [self.code.acc.read_from(state["pos"] + offset)
                    for offset in self.offsets])))

where names would be list("lmr") and offsets would be (-1, 0, 1). self.code.acc
is the class that handles how access to the array is supposed to work.

I think I am going to kick out the lambda expressions that are supposed to
return a dict that would update the state dict and just use complete callables
instead, that directly manipulate it.

I hope this was enlightening.
  - Timo Paulssen

PS: i wonder how big the impact of defining the runhooks function inside the
step function?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/24aa67c4/attachment.pgp>

From vincent at shishkabab.net  Mon Aug 15 20:02:51 2011
From: vincent at shishkabab.net (Vincent den Boer)
Date: Mon, 15 Aug 2011 20:02:51 +0200
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
In-Reply-To: <alpine.LNX.2.00.1108150943270.5987@pc3-wlav.lbl.gov>
References: <201108121132.46897.vincent@shishkabab.net>
	<201108151203.18385.vincent@shishkabab.net>
	<alpine.LNX.2.00.1108150943270.5987@pc3-wlav.lbl.gov>
Message-ID: <201108152002.52876.vincent@shishkabab.net>

Hello Wim,

> 
> hmm; I can't say that has been a problem for us (if I understand things
> correctly). If it's not too much work (since you already departed from
> gccxml), do you have an explicit class that exhibits the problem?
> 

b2World from Box2D/Dynamics/b2World.h has the problem. I exported the header file 
using this command:
gccxml -o box2d_gccxml.xml /usr/local/include/Box2D/Box2D.h

> 
> I don't see those as differences per se: cppyy works with C as well, of
> course, and the CPython implementation is already available (albeit
> packaged in the ROOT class library we use here in HEP and several other
> places, but ROOT is part of a handful of Linux distro's already, so it's
> not too hard to get, and it is supposed to move to llvm by the Summer of
> next year as well).
> 

Then I think it is useful to compare cppyy and Wrappyr. It'd be unfortunate if 
the two projects really have exactly the same goals and usa cases, because it'd 
mean that while I have learned from this project, I've just written a lot of 
obsolete code ;). Where can I find documentation of cppyy? Google doesn't reveal 
anything useful to me.

Kind regards,
Vincent

From hakan at debian.org  Mon Aug 15 20:07:02 2011
From: hakan at debian.org (Hakan Ardo)
Date: Mon, 15 Aug 2011 20:07:02 +0200
Subject: [pypy-dev] go benchmark
Message-ID: <CAEkL+Rgtx6bRVdjG60J_WiMk5KiJsEyf7A7tq25PvpOknfatiA@mail.gmail.com>

Hi,
I've been looking a bit at the go benchmark, which sometimes is
significantly slower on jit-short_from_state.

What's happening with default trace_limit=12000 on trunk is that the
loop in random_playout is compiled into a loop containing 209 guards
and the loop in computer_move is compiled into a loop with 390 guards.
After that a lot of bridges are created. So I'm guessing we have some
large loops here with a lot of different paths that are common.

I patched go.py to print the number of loops, bridges and guards that
was created each iteration using a pypyjit.set_compile_hook. It also
prints when a loop with more than 200 guards are produced:

    http://paste.pocoo.org/show/458824/

Could this be fixed by...
... stop producing bridges after a fixed number of have been produced
with the same root loop?
... never produce a bridge if it's parent trace have a large number of
guards that have failed more than trace_eagerness/2 times?
... falling back on using a method jit in the above cases ;)

Is this something I should investigate on jit-short_from_state, or
could we consider it a separate problem and merge jit-short_from_state
anyway? The only connection to jit-short_from_state I can think of is
that bridges probably have become somewhat more expensive to produce.

-- 
H?kan Ard?

From wlavrijsen at lbl.gov  Mon Aug 15 20:37:17 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Mon, 15 Aug 2011 11:37:17 -0700 (PDT)
Subject: [pypy-dev] PyPy, C/C++ and Wrappyr
In-Reply-To: <201108152002.52876.vincent@shishkabab.net>
References: <201108121132.46897.vincent@shishkabab.net>
	<201108151203.18385.vincent@shishkabab.net>
	<alpine.LNX.2.00.1108150943270.5987@pc3-wlav.lbl.gov>
	<201108152002.52876.vincent@shishkabab.net>
Message-ID: <alpine.LNX.2.00.1108151117370.5987@pc3-wlav.lbl.gov>

Hi Vincent,

> b2World from Box2D/Dynamics/b2World.h has the problem. I exported the
> header file using this command:
> gccxml -o box2d_gccxml.xml /usr/local/include/Box2D/Box2D.h

which version of gccxml do you use? There haven't been any actual releases
of the tool. This is with trunk:

   <Constructor id="_1615" name="b2World" explicit="1" context="_600" access="public" mangled="_ZN7b2WorldC1ERK6b2Vec2b *INTERNAL* " demangled="b2World::b2World(b2Vec2 const&amp;, bool)" location="f19:45" file="f19" line="45" extern="1">
     <Argument name="gravity" type="_1245" location="f19:45" file="f19" line="45"/>
     <Argument name="doSleep" type="_1185" location="f19:45" file="f19" line="45"/>
   </Constructor>

Of course, the generated copy constructor is there as well.

> Where can I find documentation of cppyy?

There's no documentation yet, as it's not yet production quality. I've made
a small write-up for internal purposes (i.e. a few very friendly users :) ).
I still want to be able to change what is exposed to the user until it is
production quality.

That said, here's the gist of it: install ROOT, then use genreflex to
generate the descriptions:

   http://root.cern.ch/drupal/content/generating-reflex-dictionaries

Build pypy-c with cppyy support (use the reflex-support branch):

   $ pypy translate.py -O jit --gcrootfinder=shadowstack targetpypystandalone.py --withmod-cppyy

Then use the information like so (assuming myClassDict.so contains the info
for a class MyClass):

   >>>> import cppyy
   >>>> cppyy.load_reflection_info( "myClassDict.so" )
   >>>> m = cppyy.gbl.MyClass()
   >>>> result = m.CallMyMethod()
   >>>>

If you want to enable the fast path, patch-up genreflex with the patch that
is included in pypy/module/cppyy, but that's only needed for benchmarks, not
for just showing the functionality.

On the CPython side and within our build environment the genreflex portions
are hidden behind the build system and class loader, so our users only do:

   >>> import PyCintex
   >>> m = PyCintex.gbl.MyClass()
   >>> result = m.CallMyMethod()
   >>>

Again, this isn't production code. The b2World example should be fine AFAICS,
but things like std::basic_ostream (std::cout) are a different matter (I'm
working on that right now, as a user's code choked on it, due to a deadly
cocktail of virtual inheritance and public data members: they require an
actual instance for offset calculation and the instantiation of a temp
basic_ostream for that purpose went "plink" ... ).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From naylor.b.david at gmail.com  Mon Aug 15 21:03:37 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Mon, 15 Aug 2011 21:03:37 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAKp1Nw6e6UA-jLmxLf=iY_u28OK8HE+HmdDSZMiiE3eStGza_w@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108112018.01642.naylor.b.david@gmail.com>
	<CAKp1Nw6e6UA-jLmxLf=iY_u28OK8HE+HmdDSZMiiE3eStGza_w@mail.gmail.com>
Message-ID: <201108152103.41416.naylor.b.david@gmail.com>

On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
> Hi David,
>     for your information, I'm currently trying to make my FreeBSD buildbot
> back online. It broke when the requirement for the "pypy" binary to build
> PyPy appeared. It seems there are still some issues to make PyPy build
> again under FreeBSD. I'll keep you updated.

Thanks for the update.  I have been working on a port for pypy.  The code is 
at github.com/DragonSA/pypy.  It may be able to help you compile pypy under 
FreeBSD.  There is a branch called 1.6 that is compatible with trunk.  To 
compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:

# make install WRKSRC=/path/to/hg/checkout/of/pypy

The master branch uses the 1.5 sources.  

> 2011/8/11 David Naylor <naylor.b.david at gmail.com>
> 
> > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > > Hi
> > > 
> > > A little status update from my side.
> > > 
> > > I'm trying to get 1.6 out of the door and there are a few things that
> > > stand out before we can do it.
> > > 
> > > * Windows situation (a lot of crashes with missing liveness). Also
> > > lib-python tests take 10h which is a bit too much IMO (not a new
> > > regression though)
> > > * OS X situation (missing buildbots, being addressed now)
> > > * A bunch of issues that either should be decided we don't care or we
> > > do care and fix it.
> > > * Missing jitviewer release/demo that I'm doing right now
> > > * Alex reported sqlite crashes
> > > * random segfault when running trackgcroot, armin, did you have a
> > > look? I can do if you don't
> > 
> > Would anyone be interested in the FreeBSD situation?  If so what support
> > do you need?  I am currently updating my repository and will report
> > back...
> > 
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/108f74fe/attachment-0001.pgp>

From joe at goldthwaites.com  Mon Aug 15 23:48:35 2011
From: joe at goldthwaites.com (Joe Goldthwaite)
Date: Mon, 15 Aug 2011 14:48:35 -0700
Subject: [pypy-dev] A basic question.
Message-ID: <CA+FoDzojOGkx8nx4DwvJcJQhFoz4S=Tv0TeUgwpBgjiQ4LXZbw@mail.gmail.com>

I apologize if this is the wrong place for this. In looking at the archives
this list seems to be more geared to the developers than users.  I couldn't
find a list for end users though so I'm going to post it here.

I'd like to try pypy for a simple project.  I'm trying to use SUDS to
retrieve data from a web service.  SUDS uses a setup.py file to do the
install.  When I try that under pypy I get a message;

 ImportError: No module named setuptools

I know I need setup tools but how do you install that?  I have a version for
Python 2.7 but it thinks that 2.7 isn't installed.  I've searched the web
for pypy setuptools information but can't seem to find anything.

Is it too early to try and use pypy for production?  Is there a version of
setup tools available for it?

Thanks,

Joe Goldthwaite
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/05963dc7/attachment.html>

From anto.cuni at gmail.com  Tue Aug 16 00:10:55 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 16 Aug 2011 00:10:55 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
 [dict.setdefault]
In-Reply-To: <CAFRnB2VN7BOn+vL__=sz3-MLEV+pPKDEB4pS0mD6kmNocQix_A@mail.gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>	<201108121749.57954.naylor.b.david@gmail.com>	<4E46A73A.9000405@gmail.com>	<201108132014.25161.naylor.b.david@gmail.com>	<CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
	<CAFRnB2VN7BOn+vL__=sz3-MLEV+pPKDEB4pS0mD6kmNocQix_A@mail.gmail.com>
Message-ID: <4E49996F.90302@gmail.com>

On 15/08/11 15:36, Alex Gaynor wrote:
> I'd like to express that, unless we have a very compelling reason, we should
> try to keep more stuff in pure python, as opposed to RPython.  Mostly because
> it speeds up translation ;) (also easier to test, easier to write, etc.).

or, on the other hand, we should try to improve the JIT so that pure python 
code can be as fast as rpython :-)

From anto.cuni at gmail.com  Tue Aug 16 00:52:21 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 16 Aug 2011 00:52:21 +0200
Subject: [pypy-dev] A basic question.
In-Reply-To: <CA+FoDzojOGkx8nx4DwvJcJQhFoz4S=Tv0TeUgwpBgjiQ4LXZbw@mail.gmail.com>
References: <CA+FoDzojOGkx8nx4DwvJcJQhFoz4S=Tv0TeUgwpBgjiQ4LXZbw@mail.gmail.com>
Message-ID: <4E49A325.7000305@gmail.com>

Hello Joe,

On 15/08/11 23:48, Joe Goldthwaite wrote:
> I apologize if this is the wrong place for this. In looking at the archives
> this list seems to be more geared to the developers than users.  I couldn't
> find a list for end users though so I'm going to post it here.
>
> I'd like to try pypy for a simple project.  I'm trying to use SUDS to retrieve
> data from a web service.  SUDS uses a setup.py file to do the install.  When I
> try that under pypy I get a message;
>
>   ImportError: No module named setuptools
>
> I know I need setup tools but how do you install that?

look here:
http://doc.pypy.org/en/latest/getting-started.html#installing-pypy

in particular, you need to download and execute distribute_setup.py.

ciao,
Anto

From glavoie at gmail.com  Tue Aug 16 05:50:18 2011
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Mon, 15 Aug 2011 23:50:18 -0400
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108152103.41416.naylor.b.david@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108112018.01642.naylor.b.david@gmail.com>
	<CAKp1Nw6e6UA-jLmxLf=iY_u28OK8HE+HmdDSZMiiE3eStGza_w@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
Message-ID: <CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>

Hi David,
     I took a look at your port and it looks good! The patch you made for
SANDBOX should probably not be needed, with a fix upstream. It seems PyPy
still have a few "hardcoded path" issues that need to be looked at for
FreeBSD (and probably for Darwin too for tools like Darwin Ports). I felt on
one of those issues this morning with "expat.h" and "libexpat.so". The
translation process didn't look at /usr/local/include and /usr/local/lib to
find them. Actually, there are a few places where the
"pypy/translator/platform" module should be use to find include/lib paths. I
made the initial "freebsd.py" file in there that contains "general"
references to include/lib directories in "/usr/local".

About my current issue, maybe you've seen it. When I try to build PyPy using
CPython, I get a "Broken Pipe" error and the process fails (using the
standard "python translate.py -O2" command). Have you seen this problem
recently? I'm trying with the latest trunk. It seems the "python" process
runs out of memory as I have this in my kernel log:

swap_pager_getswapspace(16): failed
pid 15107 (cc1), uid 1001, was killed: out of swap space
swap_pager: out of swap space
swap_pager_getswapspace(2): failed
pid 37981 (python), uid 1001, was killed: out of swap space

PyPy used to build correctly but I haven't tried to build it in the last 6
months...

Gabriel

2011/8/15 David Naylor <naylor.b.david at gmail.com>

> On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
> > Hi David,
> >     for your information, I'm currently trying to make my FreeBSD
> buildbot
> > back online. It broke when the requirement for the "pypy" binary to build
> > PyPy appeared. It seems there are still some issues to make PyPy build
> > again under FreeBSD. I'll keep you updated.
>
> Thanks for the update.  I have been working on a port for pypy.  The code
> is
> at github.com/DragonSA/pypy.  It may be able to help you compile pypy
> under
> FreeBSD.  There is a branch called 1.6 that is compatible with trunk.  To
> compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:
>
> # make install WRKSRC=/path/to/hg/checkout/of/pypy
>
> The master branch uses the 1.5 sources.
>
> > 2011/8/11 David Naylor <naylor.b.david at gmail.com>
> >
> > > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > > > Hi
> > > >
> > > > A little status update from my side.
> > > >
> > > > I'm trying to get 1.6 out of the door and there are a few things that
> > > > stand out before we can do it.
> > > >
> > > > * Windows situation (a lot of crashes with missing liveness). Also
> > > > lib-python tests take 10h which is a bit too much IMO (not a new
> > > > regression though)
> > > > * OS X situation (missing buildbots, being addressed now)
> > > > * A bunch of issues that either should be decided we don't care or we
> > > > do care and fix it.
> > > > * Missing jitviewer release/demo that I'm doing right now
> > > > * Alex reported sqlite crashes
> > > > * random segfault when running trackgcroot, armin, did you have a
> > > > look? I can do if you don't
> > >
> > > Would anyone be interested in the FreeBSD situation?  If so what
> support
> > > do you need?  I am currently updating my repository and will report
> > > back...
> > >
> > > _______________________________________________
> > > pypy-dev mailing list
> > > pypy-dev at python.org
> > > http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110815/b0cac6aa/attachment.html>

From naylor.b.david at gmail.com  Tue Aug 16 06:28:52 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Tue, 16 Aug 2011 06:28:52 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
	<CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
Message-ID: <201108160628.57358.naylor.b.david@gmail.com>

On Tuesday, 16 August 2011 05:50:18 Gabriel Lavoie wrote:
> Hi David,
>      I took a look at your port and it looks good! The patch you made for
> SANDBOX should probably not be needed, with a fix upstream. It seems PyPy
> still have a few "hardcoded path" issues that need to be looked at for
> FreeBSD (and probably for Darwin too for tools like Darwin Ports). I felt
> on one of those issues this morning with "expat.h" and "libexpat.so". The
> translation process didn't look at /usr/local/include and /usr/local/lib
> to find them. Actually, there are a few places where the
> "pypy/translator/platform" module should be use to find include/lib paths.
> I made the initial "freebsd.py" file in there that contains "general"
> references to include/lib directories in "/usr/local".

Thanks, sandbox has not been tested as I do not know how to.  An example 
script that makes sandbox behave like a normal python/pypy instance would be 
nice...

On the expat issue:
# ldd `which pypy1.5`
/usr/local/bin/pypy1.5:
        libm.so.5 => /lib/libm.so.5 (0x801664000)
        libintl.so.9 => /usr/local/lib/libintl.so.9 (0x801885000)
        libssl.so.6 => /usr/lib/libssl.so.6 (0x801a8e000)
        libcrypto.so.6 => /lib/libcrypto.so.6 (0x801ce1000)
        libexpat.so.6 => /usr/local/lib/libexpat.so.6 (0x802082000)
        libbz2.so.4 => /usr/lib/libbz2.so.4 (0x8022a5000)
        libz.so.6 => /lib/libz.so.6 (0x8024b5000)
        librt.so.1 => /usr/lib/librt.so.1 (0x8026cb000)
        libutil.so.9 => /lib/libutil.so.9 (0x8028d0000)
        libffi.so.5 => /usr/local/lib/libffi.so.5 (0x802ae1000)
        libncurses.so.8 => /lib/libncurses.so.8 (0x802ce7000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x802f34000)
        libthr.so.3 => /lib/libthr.so.3 (0x803154000)
        libc.so.7 => /lib/libc.so.7 (0x803377000)
        libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x8036bf000)

I also see those error yet it appears expat is picked up somehow, maybe as a 
depenendency?  

> About my current issue, maybe you've seen it. When I try to build PyPy
> using CPython, I get a "Broken Pipe" error and the process fails (using
> the standard "python translate.py -O2" command). Have you seen this
> problem recently? I'm trying with the latest trunk. It seems the "python"
> process runs out of memory as I have this in my kernel log:

I have not encountered that problem before (except I suffer from ENOMEM).

> swap_pager_getswapspace(16): failed
> pid 15107 (cc1), uid 1001, was killed: out of swap space
> swap_pager: out of swap space
> swap_pager_getswapspace(2): failed
> pid 37981 (python), uid 1001, was killed: out of swap space

I have seen this problem before.  FreeBSD hard codes the size of swap it can 
access, and does not adapt it based on RAM or active swap.  To fix that try 
the following in /boot/loader.conf:

# Increase max swap zone to allow for ~64GB of swap                                                                                   
kern.maxswzone="268435456"  # 256MB

> PyPy used to build correctly but I haven't tried to build it in the last 6
> months...

I have successfully built pypy-1.5 and trunk (~1 week ago), so it is possible.  
If you are still having problems please send me the installed physical memory, 
and the output of `swapinfo` and `uname -a`.  

> 2011/8/15 David Naylor <naylor.b.david at gmail.com>
> 
> > On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
> > > Hi David,
> > > 
> > >     for your information, I'm currently trying to make my FreeBSD
> > 
> > buildbot
> > 
> > > back online. It broke when the requirement for the "pypy" binary to
> > > build PyPy appeared. It seems there are still some issues to make PyPy
> > > build again under FreeBSD. I'll keep you updated.
> > 
> > Thanks for the update.  I have been working on a port for pypy.  The code
> > is
> > at github.com/DragonSA/pypy.  It may be able to help you compile pypy
> > under
> > FreeBSD.  There is a branch called 1.6 that is compatible with trunk.  To
> > compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:
> > 
> > # make install WRKSRC=/path/to/hg/checkout/of/pypy
> > 
> > The master branch uses the 1.5 sources.
> > 
> > > 2011/8/11 David Naylor <naylor.b.david at gmail.com>
> > > 
> > > > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > > > > Hi
> > > > > 
> > > > > A little status update from my side.
> > > > > 
> > > > > I'm trying to get 1.6 out of the door and there are a few things
> > > > > that stand out before we can do it.
> > > > > 
> > > > > * Windows situation (a lot of crashes with missing liveness). Also
> > > > > lib-python tests take 10h which is a bit too much IMO (not a new
> > > > > regression though)
> > > > > * OS X situation (missing buildbots, being addressed now)
> > > > > * A bunch of issues that either should be decided we don't care or
> > > > > we do care and fix it.
> > > > > * Missing jitviewer release/demo that I'm doing right now
> > > > > * Alex reported sqlite crashes
> > > > > * random segfault when running trackgcroot, armin, did you have a
> > > > > look? I can do if you don't
> > > > 
> > > > Would anyone be interested in the FreeBSD situation?  If so what
> > 
> > support
> > 
> > > > do you need?  I am currently updating my repository and will report
> > > > back...
> > > > 
> > > > _______________________________________________
> > > > pypy-dev mailing list
> > > > pypy-dev at python.org
> > > > http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/ad50d6ec/attachment-0001.pgp>

From glavoie at gmail.com  Tue Aug 16 08:10:49 2011
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Tue, 16 Aug 2011 02:10:49 -0400
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108160628.57358.naylor.b.david@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
	<CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
	<201108160628.57358.naylor.b.david@gmail.com>
Message-ID: <CAKp1Nw4-4Uf7YrgHTBvJ4fXcL=osq7cZWEBULwAddEGFsqaLBQ@mail.gmail.com>

Okay! It seems I've hit the GCC 4.2 memory usage bug:
https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391
http://comments.gmane.org/gmane.comp.python.pypy/7634

I'm now trying to build with GCC 4.6 (a port is available) and I'll report
back.

Gabriel

2011/8/16 David Naylor <naylor.b.david at gmail.com>

> On Tuesday, 16 August 2011 05:50:18 Gabriel Lavoie wrote:
> > Hi David,
> >      I took a look at your port and it looks good! The patch you made for
> > SANDBOX should probably not be needed, with a fix upstream. It seems PyPy
> > still have a few "hardcoded path" issues that need to be looked at for
> > FreeBSD (and probably for Darwin too for tools like Darwin Ports). I felt
> > on one of those issues this morning with "expat.h" and "libexpat.so". The
> > translation process didn't look at /usr/local/include and /usr/local/lib
> > to find them. Actually, there are a few places where the
> > "pypy/translator/platform" module should be use to find include/lib
> paths.
> > I made the initial "freebsd.py" file in there that contains "general"
> > references to include/lib directories in "/usr/local".
>
> Thanks, sandbox has not been tested as I do not know how to.  An example
> script that makes sandbox behave like a normal python/pypy instance would
> be
> nice...
>
> On the expat issue:
> # ldd `which pypy1.5`
> /usr/local/bin/pypy1.5:
>        libm.so.5 => /lib/libm.so.5 (0x801664000)
>        libintl.so.9 => /usr/local/lib/libintl.so.9 (0x801885000)
>        libssl.so.6 => /usr/lib/libssl.so.6 (0x801a8e000)
>        libcrypto.so.6 => /lib/libcrypto.so.6 (0x801ce1000)
>        libexpat.so.6 => /usr/local/lib/libexpat.so.6 (0x802082000)
>        libbz2.so.4 => /usr/lib/libbz2.so.4 (0x8022a5000)
>        libz.so.6 => /lib/libz.so.6 (0x8024b5000)
>        librt.so.1 => /usr/lib/librt.so.1 (0x8026cb000)
>        libutil.so.9 => /lib/libutil.so.9 (0x8028d0000)
>        libffi.so.5 => /usr/local/lib/libffi.so.5 (0x802ae1000)
>        libncurses.so.8 => /lib/libncurses.so.8 (0x802ce7000)
>        libcrypt.so.5 => /lib/libcrypt.so.5 (0x802f34000)
>        libthr.so.3 => /lib/libthr.so.3 (0x803154000)
>        libc.so.7 => /lib/libc.so.7 (0x803377000)
>        libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x8036bf000)
>
> I also see those error yet it appears expat is picked up somehow, maybe as
> a
> depenendency?
>
> > About my current issue, maybe you've seen it. When I try to build PyPy
> > using CPython, I get a "Broken Pipe" error and the process fails (using
> > the standard "python translate.py -O2" command). Have you seen this
> > problem recently? I'm trying with the latest trunk. It seems the "python"
> > process runs out of memory as I have this in my kernel log:
>
> I have not encountered that problem before (except I suffer from ENOMEM).
>
> > swap_pager_getswapspace(16): failed
> > pid 15107 (cc1), uid 1001, was killed: out of swap space
> > swap_pager: out of swap space
> > swap_pager_getswapspace(2): failed
> > pid 37981 (python), uid 1001, was killed: out of swap space
>
> I have seen this problem before.  FreeBSD hard codes the size of swap it
> can
> access, and does not adapt it based on RAM or active swap.  To fix that try
> the following in /boot/loader.conf:
>
> # Increase max swap zone to allow for ~64GB of swap
> kern.maxswzone="268435456"  # 256MB
>
> > PyPy used to build correctly but I haven't tried to build it in the last
> 6
> > months...
>
> I have successfully built pypy-1.5 and trunk (~1 week ago), so it is
> possible.
> If you are still having problems please send me the installed physical
> memory,
> and the output of `swapinfo` and `uname -a`.
>
> > 2011/8/15 David Naylor <naylor.b.david at gmail.com>
> >
> > > On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
> > > > Hi David,
> > > >
> > > >     for your information, I'm currently trying to make my FreeBSD
> > >
> > > buildbot
> > >
> > > > back online. It broke when the requirement for the "pypy" binary to
> > > > build PyPy appeared. It seems there are still some issues to make
> PyPy
> > > > build again under FreeBSD. I'll keep you updated.
> > >
> > > Thanks for the update.  I have been working on a port for pypy.  The
> code
> > > is
> > > at github.com/DragonSA/pypy.  It may be able to help you compile pypy
> > > under
> > > FreeBSD.  There is a branch called 1.6 that is compatible with trunk.
>  To
> > > compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:
> > >
> > > # make install WRKSRC=/path/to/hg/checkout/of/pypy
> > >
> > > The master branch uses the 1.5 sources.
> > >
> > > > 2011/8/11 David Naylor <naylor.b.david at gmail.com>
> > > >
> > > > > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> > > > > > Hi
> > > > > >
> > > > > > A little status update from my side.
> > > > > >
> > > > > > I'm trying to get 1.6 out of the door and there are a few things
> > > > > > that stand out before we can do it.
> > > > > >
> > > > > > * Windows situation (a lot of crashes with missing liveness).
> Also
> > > > > > lib-python tests take 10h which is a bit too much IMO (not a new
> > > > > > regression though)
> > > > > > * OS X situation (missing buildbots, being addressed now)
> > > > > > * A bunch of issues that either should be decided we don't care
> or
> > > > > > we do care and fix it.
> > > > > > * Missing jitviewer release/demo that I'm doing right now
> > > > > > * Alex reported sqlite crashes
> > > > > > * random segfault when running trackgcroot, armin, did you have a
> > > > > > look? I can do if you don't
> > > > >
> > > > > Would anyone be interested in the FreeBSD situation?  If so what
> > >
> > > support
> > >
> > > > > do you need?  I am currently updating my repository and will report
> > > > > back...
> > > > >
> > > > > _______________________________________________
> > > > > pypy-dev mailing list
> > > > > pypy-dev at python.org
> > > > > http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/4193e317/attachment.html>

From fijall at gmail.com  Tue Aug 16 08:43:52 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 16 Aug 2011 08:43:52 +0200
Subject: [pypy-dev] go benchmark
In-Reply-To: <CAEkL+Rgtx6bRVdjG60J_WiMk5KiJsEyf7A7tq25PvpOknfatiA@mail.gmail.com>
References: <CAEkL+Rgtx6bRVdjG60J_WiMk5KiJsEyf7A7tq25PvpOknfatiA@mail.gmail.com>
Message-ID: <CAK5idxTnAEFcW73C6TnrkvZ2NcUciG+vS8GtuZjTHFmDxGr_SA@mail.gmail.com>

On Mon, Aug 15, 2011 at 8:07 PM, Hakan Ardo <hakan at debian.org> wrote:
> Hi,
> I've been looking a bit at the go benchmark, which sometimes is
> significantly slower on jit-short_from_state.
>
> What's happening with default trace_limit=12000 on trunk is that the
> loop in random_playout is compiled into a loop containing 209 guards
> and the loop in computer_move is compiled into a loop with 390 guards.
> After that a lot of bridges are created. So I'm guessing we have some
> large loops here with a lot of different paths that are common.
>
> I patched go.py to print the number of loops, bridges and guards that
> was created each iteration using a pypyjit.set_compile_hook. It also
> prints when a loop with more than 200 guards are produced:
>
> ? ?http://paste.pocoo.org/show/458824/
>
> Could this be fixed by...
> ... stop producing bridges after a fixed number of have been produced
> with the same root loop?
> ... never produce a bridge if it's parent trace have a large number of
> guards that have failed more than trace_eagerness/2 times?
> ... falling back on using a method jit in the above cases ;)
>
> Is this something I should investigate on jit-short_from_state, or
> could we consider it a separate problem and merge jit-short_from_state
> anyway? The only connection to jit-short_from_state I can think of is
> that bridges probably have become somewhat more expensive to produce.
>

Personally I would consider this a separate problem.

From chef at ghum.de  Tue Aug 16 11:59:59 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Tue, 16 Aug 2011 11:59:59 +0200
Subject: [pypy-dev] behaviour difference int-subclasses pypy 1.5 to cpython
	2.6.6.
Message-ID: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>

cPython 2.6.6 is able to initialize subclassed int-objects from strings:

Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]
on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class fisch(int):
...     def __init__(self, value):
...         int.__init__(value)
...
>>> a=fisch("12")
>>>


pypy 1.5.0 is not able to do this:

Python 2.7.1 (aefc70438132+, Apr 29 2011, 12:45:42)
[PyPy 1.5.0-alpha0 with MSC v.1600 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``topics are for the feeble
minded''
>>>> class fisch(int):
....     def __init__(self, value):
....         int.__init__(value)
....
>>>> a=fisch("12")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<console>", line 3, in __init__
TypeError: unbound method __init__() must be called with int instance as
first a
rgument (got str instance instead)


(detected this behavior difference when trying to pypy pyPdf)

why is this?

Harald
-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/c90ae48e/attachment.html>

From arigo at tunes.org  Tue Aug 16 13:16:14 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 16 Aug 2011 13:16:14 +0200
Subject: [pypy-dev] behaviour difference int-subclasses pypy 1.5 to
 cpython 2.6.6.
In-Reply-To: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>
References: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>
Message-ID: <CAMSv6X3sZUBiGVi6MQF2OM7pZgrFUOOOdke697PA40exRycs6A@mail.gmail.com>

Hi,

On Tue, Aug 16, 2011 at 11:59 AM, Massa, Harald Armin <chef at ghum.de> wrote:
>>>> class fisch(int):
> ... ? ? def __init__(self, value):
> ... ? ? ? ? int.__init__(value)

You probably mean "int.__init__(self, value)" here.  Your code is buggy :-)

It works on CPython anyway, which is a bit strange in my opinion, but
probably comes from the fact that the 'int' class has no attribute
'__init__', and "int.__init__" just returns the exact same object as
"object.__init__".  On PyPy instead, you get a regular unbound method,
which checks that it is passed a first argument of the correct class.


A bient?t,

Armin.

From arigo at tunes.org  Tue Aug 16 13:24:19 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 16 Aug 2011 13:24:19 +0200
Subject: [pypy-dev] go benchmark
In-Reply-To: <CAK5idxTnAEFcW73C6TnrkvZ2NcUciG+vS8GtuZjTHFmDxGr_SA@mail.gmail.com>
References: <CAEkL+Rgtx6bRVdjG60J_WiMk5KiJsEyf7A7tq25PvpOknfatiA@mail.gmail.com>
	<CAK5idxTnAEFcW73C6TnrkvZ2NcUciG+vS8GtuZjTHFmDxGr_SA@mail.gmail.com>
Message-ID: <CAMSv6X2TZhPUzQgYygqRLe8oJfp-ms9VjRHQP7vdi5xBX+X4TQ@mail.gmail.com>

Hi,

On Tue, Aug 16, 2011 at 8:43 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Is this something I should investigate on jit-short_from_state, or
>> could we consider it a separate problem and merge jit-short_from_state
>> anyway?
>
> Personally I would consider this a separate problem.

I agree.  You can merge the jit-short_from_state branch into trunk if
you consider it to be basically ready; and then we can think about
ways to increase the stability of the go benchmark, which has always
been an issue.


A bient?t,

Armin.

From chef at ghum.de  Tue Aug 16 13:32:26 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Tue, 16 Aug 2011 13:32:26 +0200
Subject: [pypy-dev] behaviour difference int-subclasses pypy 1.5 to
 cpython 2.6.6.
In-Reply-To: <CAMSv6X3sZUBiGVi6MQF2OM7pZgrFUOOOdke697PA40exRycs6A@mail.gmail.com>
References: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>
	<CAMSv6X3sZUBiGVi6MQF2OM7pZgrFUOOOdke697PA40exRycs6A@mail.gmail.com>
Message-ID: <CAMSP2L6Fb+0NOA=09J-qqxTstZJLWe3Qq+_rhoFzXr_OheQ4PA@mail.gmail.com>

>
> On Tue, Aug 16, 2011 at 11:59 AM, Massa, Harald Armin <chef at ghum.de>
> wrote:
> >>>> class fisch(int):
> > ...     def __init__(self, value):
> > ...         int.__init__(value)
>
> You probably mean "int.__init__(self, value)" here.  Your code is buggy :-)
>

just submitted the fix to the pyPdf author :)  my workaround was

def __init__(self, value):
       int.__init__(int(value))
.. so that I could run it under PyPy :)

so, I understand that the cPython behaviour is rather buggy. Good to have
another Python implementation :)

Thanks for the fast response,

Harald

-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/97ee74e2/attachment.html>

From Sven.Hager at uni-duesseldorf.de  Tue Aug 16 13:57:40 2011
From: Sven.Hager at uni-duesseldorf.de (Sven Hager)
Date: Tue, 16 Aug 2011 13:57:40 +0200
Subject: [pypy-dev] PPC64 JIT progress
In-Reply-To: <CAGWvnykpGnB8n6WyudteXussnafTGpPwQhOEGytMUg_aNJBh9Q@mail.gmail.com>
References: <CAGWvnykpGnB8n6WyudteXussnafTGpPwQhOEGytMUg_aNJBh9Q@mail.gmail.com>
Message-ID: <4E4A5B34.9010607@uni-duesseldorf.de>

On 08/15/2011 07:12 PM, David Edelsohn wrote:
> Attached is the next patch for the PPC JIT backend.
>
> The patch adds shift-left simplified mnemonic (sldi) and a load
> doubleword (load_dword) function that uses it.
>
> I also discovered some typos in my previous patch to ppc_field.py.
>
> Currently every PPC JIT testcase fails on PPC64 Linux because the PPC
> backend makes an incorrect assumption about function pointers.  PPC64
> Linux function pointer point to function descriptors, not code
> addresses.  PPC64 Linux libffi knows this, which makes
>
>      def get_function(self):
>          i = self.code.materialize(AsmMemoryManager(), [])
>          t = lltype.FuncType([], lltype.Signed)
>          return rffi.cast(lltype.Ptr(t), i)
>
> wrong for PPC64.
>
> A function descriptor is a three-word data vector containing the
> function address, data addressibility pointer and static chain.  For
> some of the tests to start working on PPC64 Linux, the backend needs
> to create another data object that contains the function address and
> that pointer is coerced into a function pointer.
>
> Thanks, David
Hello David,

I added your latest patch to the codebase.

I talked with Armin about the problem with the failing tests, and we
agreed that for now it's the best approach to implemented stuff for
a 32 bit machine until there is a possibility to test on a 64 bit machine.

The problem is that until I can't test the implemented 64 bit 
functionality I heavily doubt
that I am able to write code that does what it should. ;)

Best regards,
Sven

From fuzzyman at gmail.com  Tue Aug 16 14:06:23 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Tue, 16 Aug 2011 13:06:23 +0100
Subject: [pypy-dev] behaviour difference int-subclasses pypy 1.5 to
 cpython 2.6.6.
In-Reply-To: <CAMSP2L6Fb+0NOA=09J-qqxTstZJLWe3Qq+_rhoFzXr_OheQ4PA@mail.gmail.com>
References: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>
	<CAMSv6X3sZUBiGVi6MQF2OM7pZgrFUOOOdke697PA40exRycs6A@mail.gmail.com>
	<CAMSP2L6Fb+0NOA=09J-qqxTstZJLWe3Qq+_rhoFzXr_OheQ4PA@mail.gmail.com>
Message-ID: <CAKCKLWwx6acQw3=P+AftA9UEeiCMGafFHitt6z+Vzr2vUYNHiQ@mail.gmail.com>

On 16 August 2011 12:32, Massa, Harald Armin <chef at ghum.de> wrote:

> On Tue, Aug 16, 2011 at 11:59 AM, Massa, Harald Armin <chef at ghum.de>
>> wrote:
>> >>>> class fisch(int):
>> > ...     def __init__(self, value):
>> > ...         int.__init__(value)
>>
>> You probably mean "int.__init__(self, value)" here.  Your code is buggy
>> :-)
>>
>
> just submitted the fix to the pyPdf author :)  my workaround was
>
> def __init__(self, value):
>        int.__init__(int(value))
> .. so that I could run it under PyPy :)
>
>
Given that int doesn't *have* an __init__ method, just omitting that call
altogether is a better fix. (int objects use __new__ for construction and
have no post construction initialisation.)



> so, I understand that the cPython behaviour is rather buggy. Good to have
> another Python implementation :)
>

Well, it isn't buggy. The call is equivalent to:

   object.__init__("12")

This just does nothing as "12" is a valid instance of object...

I assume the difference is that int has an __init__ method on pypy.

Michael


> Thanks for the fast response,
>
> Harald
>
> --
> GHUM GmbH
> Harald Armin Massa
> Spielberger Stra?e 49
> 70435 Stuttgart
> 0173/9409607
>
> Amtsgericht Stuttgart, HRB 734971
> -
> persuadere.
> et programmare
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/928c0af4/attachment-0001.html>

From arigo at tunes.org  Tue Aug 16 15:27:30 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 16 Aug 2011 15:27:30 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <201108151821.02922.naylor.b.david@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108132014.25161.naylor.b.david@gmail.com>
	<CAMSv6X3J07RVAgpm4P__t6eoQo6vQxtykvpbMwJs1YNfMmZOaw@mail.gmail.com>
	<201108151821.02922.naylor.b.david@gmail.com>
Message-ID: <CAMSv6X1gJyFKRCn7j_YUz1s=remMWLx1=tg9AQBw=sHMaT82ug@mail.gmail.com>

Hi David,

On Mon, Aug 15, 2011 at 6:20 PM, David Naylor <naylor.b.david at gmail.com> wrote:
> For me the performance of datetime object's hashing is sufficient but I think
> the python code could use some performance improvements. ?Is my approach using
> a direct computation to type long acceptable (in principle). ?If so I can
> refine it and submit a patch.

Yes, replacing the hash with a faster-to-compute one is fine.  It's
best performance-wise if you can avoid using Python longs.  As far as
I know it just needs some random-looking xor-ing and shifting of the
fields.  Note, of course, that you must carefully satisfy the property
that for any objects x and y, if "x == y" then "hash(x) == hash(y)".


A bient?t,

Armin.

From arigo at tunes.org  Tue Aug 16 15:44:20 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 16 Aug 2011 15:44:20 +0200
Subject: [pypy-dev] behaviour difference int-subclasses pypy 1.5 to
 cpython 2.6.6.
In-Reply-To: <CAKCKLWwx6acQw3=P+AftA9UEeiCMGafFHitt6z+Vzr2vUYNHiQ@mail.gmail.com>
References: <CAMSP2L75M-qGrrqu8uaQjvgeqt_0jg6VWJ4b6UNhXG0kEPNhmg@mail.gmail.com>
	<CAMSv6X3sZUBiGVi6MQF2OM7pZgrFUOOOdke697PA40exRycs6A@mail.gmail.com>
	<CAMSP2L6Fb+0NOA=09J-qqxTstZJLWe3Qq+_rhoFzXr_OheQ4PA@mail.gmail.com>
	<CAKCKLWwx6acQw3=P+AftA9UEeiCMGafFHitt6z+Vzr2vUYNHiQ@mail.gmail.com>
Message-ID: <CAMSv6X2gXkgW_EHH04Ci5gX_R+wdbCLK_W69WCtJ4+WjUO78-A@mail.gmail.com>

Hi Michael,

On Tue, Aug 16, 2011 at 2:06 PM, Michael Foord <fuzzyman at gmail.com> wrote:
> I assume the difference is that int has an __init__ method on pypy.

No, it doesn't; the issue is that the call int.__init__("12") is not
strictly equivalent to object.__init__("12").  It is on CPython, but
that's a bit by chance.  If you define your own Python classes like
this:

class MyObject(object):
    def __init__(self):
        pass
class MyInt(MyObject):
     pass

Then for an instance "o=MyObject()", the call MyInt.__init__(o) is not
equivalent to the call MyObject.__init__(o): indeed, the former fails
on CPython too, because MyInt.__init__ returns an unbound method
object attached specifically to MyInt.  In PyPy the same occurs even
with built-in types (because there is no type "built-in method" in
PyPy).


A bient?t,

Armin.

From glavoie at gmail.com  Tue Aug 16 16:15:32 2011
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Tue, 16 Aug 2011 10:15:32 -0400
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAMQxofkLMVXwc2D6vQoj97=zdUiqv03nNGGPOKfD_7rBRfD+QA@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
	<CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
	<201108160628.57358.naylor.b.david@gmail.com>
	<CAKp1Nw4-4Uf7YrgHTBvJ4fXcL=osq7cZWEBULwAddEGFsqaLBQ@mail.gmail.com>
	<CAMQxofkLMVXwc2D6vQoj97=zdUiqv03nNGGPOKfD_7rBRfD+QA@mail.gmail.com>
Message-ID: <CAKp1Nw5jCJuCHigCsG_U5kL9uFVz4tWvL2dCkpS904dY2Pn=fQ@mail.gmail.com>

I'm on 7.4-STABLE. Anyway, as I understand, FreeBSD will never upgrade its
core GCC version higher than 4.2.2 due to GPL version switch to GPLv3...
Knowing there is an issue with GCC 4.2, I'll finish my testing with the
latest GCC available from ports and I'll report. It might be a good idea to
make a newer GCC version a requirement in your PyPy port. A better idea
would be to create "officials" PyPy build (probably just with the jit
default setup, 32bits and 64bits) and make a port that would install the
binary package instead of translating the source code, since translation has
a lot of requirements (including hardware) and takes a lot of time.

Gabriel

2011/8/16 David Naylor <naylor.b.david at gmail.com>

> I was not aware of that.  If you are on FreeBSD-current then clang is
> also available (and is also in ports).  clang however takes much longer
> to compile than gcc but can easily be run in parallel whereas gcc
> used too much memory to allow that.
>
> On 16 August 2011 08:10, Gabriel Lavoie <glavoie at gmail.com> wrote:
> > Okay! It seems I've hit the GCC 4.2 memory usage bug:
> > https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391
> > http://comments.gmane.org/gmane.comp.python.pypy/7634
> >
> > I'm now trying to build with GCC 4.6 (a port is available) and I'll
> report
> > back.
> >
> > Gabriel
> >
> > 2011/8/16 David Naylor <naylor.b.david at gmail.com>
> >>
> >> On Tuesday, 16 August 2011 05:50:18 Gabriel Lavoie wrote:
> >> > Hi David,
> >> >      I took a look at your port and it looks good! The patch you made
> >> > for
> >> > SANDBOX should probably not be needed, with a fix upstream. It seems
> >> > PyPy
> >> > still have a few "hardcoded path" issues that need to be looked at for
> >> > FreeBSD (and probably for Darwin too for tools like Darwin Ports). I
> >> > felt
> >> > on one of those issues this morning with "expat.h" and "libexpat.so".
> >> > The
> >> > translation process didn't look at /usr/local/include and
> /usr/local/lib
> >> > to find them. Actually, there are a few places where the
> >> > "pypy/translator/platform" module should be use to find include/lib
> >> > paths.
> >> > I made the initial "freebsd.py" file in there that contains "general"
> >> > references to include/lib directories in "/usr/local".
> >>
> >> Thanks, sandbox has not been tested as I do not know how to.  An example
> >> script that makes sandbox behave like a normal python/pypy instance
> would
> >> be
> >> nice...
> >>
> >> On the expat issue:
> >> # ldd `which pypy1.5`
> >> /usr/local/bin/pypy1.5:
> >>        libm.so.5 => /lib/libm.so.5 (0x801664000)
> >>        libintl.so.9 => /usr/local/lib/libintl.so.9 (0x801885000)
> >>        libssl.so.6 => /usr/lib/libssl.so.6 (0x801a8e000)
> >>        libcrypto.so.6 => /lib/libcrypto.so.6 (0x801ce1000)
> >>        libexpat.so.6 => /usr/local/lib/libexpat.so.6 (0x802082000)
> >>        libbz2.so.4 => /usr/lib/libbz2.so.4 (0x8022a5000)
> >>        libz.so.6 => /lib/libz.so.6 (0x8024b5000)
> >>        librt.so.1 => /usr/lib/librt.so.1 (0x8026cb000)
> >>        libutil.so.9 => /lib/libutil.so.9 (0x8028d0000)
> >>        libffi.so.5 => /usr/local/lib/libffi.so.5 (0x802ae1000)
> >>        libncurses.so.8 => /lib/libncurses.so.8 (0x802ce7000)
> >>        libcrypt.so.5 => /lib/libcrypt.so.5 (0x802f34000)
> >>        libthr.so.3 => /lib/libthr.so.3 (0x803154000)
> >>        libc.so.7 => /lib/libc.so.7 (0x803377000)
> >>        libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x8036bf000)
> >>
> >> I also see those error yet it appears expat is picked up somehow, maybe
> as
> >> a
> >> depenendency?
> >>
> >> > About my current issue, maybe you've seen it. When I try to build PyPy
> >> > using CPython, I get a "Broken Pipe" error and the process fails
> (using
> >> > the standard "python translate.py -O2" command). Have you seen this
> >> > problem recently? I'm trying with the latest trunk. It seems the
> >> > "python"
> >> > process runs out of memory as I have this in my kernel log:
> >>
> >> I have not encountered that problem before (except I suffer from
> ENOMEM).
> >>
> >> > swap_pager_getswapspace(16): failed
> >> > pid 15107 (cc1), uid 1001, was killed: out of swap space
> >> > swap_pager: out of swap space
> >> > swap_pager_getswapspace(2): failed
> >> > pid 37981 (python), uid 1001, was killed: out of swap space
> >>
> >> I have seen this problem before.  FreeBSD hard codes the size of swap it
> >> can
> >> access, and does not adapt it based on RAM or active swap.  To fix that
> >> try
> >> the following in /boot/loader.conf:
> >>
> >> # Increase max swap zone to allow for ~64GB of swap
> >> kern.maxswzone="268435456"  # 256MB
> >>
> >> > PyPy used to build correctly but I haven't tried to build it in the
> last
> >> > 6
> >> > months...
> >>
> >> I have successfully built pypy-1.5 and trunk (~1 week ago), so it is
> >> possible.
> >> If you are still having problems please send me the installed physical
> >> memory,
> >> and the output of `swapinfo` and `uname -a`.
> >>
> >> > 2011/8/15 David Naylor <naylor.b.david at gmail.com>
> >> >
> >> > > On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
> >> > > > Hi David,
> >> > > >
> >> > > >     for your information, I'm currently trying to make my FreeBSD
> >> > >
> >> > > buildbot
> >> > >
> >> > > > back online. It broke when the requirement for the "pypy" binary
> to
> >> > > > build PyPy appeared. It seems there are still some issues to make
> >> > > > PyPy
> >> > > > build again under FreeBSD. I'll keep you updated.
> >> > >
> >> > > Thanks for the update.  I have been working on a port for pypy.  The
> >> > > code
> >> > > is
> >> > > at github.com/DragonSA/pypy.  It may be able to help you compile
> pypy
> >> > > under
> >> > > FreeBSD.  There is a branch called 1.6 that is compatible with
> trunk.
> >> > >  To
> >> > > compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:
> >> > >
> >> > > # make install WRKSRC=/path/to/hg/checkout/of/pypy
> >> > >
> >> > > The master branch uses the 1.5 sources.
> >> > >
> >> > > > 2011/8/11 David Naylor <naylor.b.david at gmail.com>
> >> > > >
> >> > > > > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
> >> > > > > > Hi
> >> > > > > >
> >> > > > > > A little status update from my side.
> >> > > > > >
> >> > > > > > I'm trying to get 1.6 out of the door and there are a few
> things
> >> > > > > > that stand out before we can do it.
> >> > > > > >
> >> > > > > > * Windows situation (a lot of crashes with missing liveness).
> >> > > > > > Also
> >> > > > > > lib-python tests take 10h which is a bit too much IMO (not a
> new
> >> > > > > > regression though)
> >> > > > > > * OS X situation (missing buildbots, being addressed now)
> >> > > > > > * A bunch of issues that either should be decided we don't
> care
> >> > > > > > or
> >> > > > > > we do care and fix it.
> >> > > > > > * Missing jitviewer release/demo that I'm doing right now
> >> > > > > > * Alex reported sqlite crashes
> >> > > > > > * random segfault when running trackgcroot, armin, did you
> have
> >> > > > > > a
> >> > > > > > look? I can do if you don't
> >> > > > >
> >> > > > > Would anyone be interested in the FreeBSD situation?  If so what
> >> > >
> >> > > support
> >> > >
> >> > > > > do you need?  I am currently updating my repository and will
> >> > > > > report
> >> > > > > back...
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > pypy-dev mailing list
> >> > > > > pypy-dev at python.org
> >> > > > > http://mail.python.org/mailman/listinfo/pypy-dev
> >
> >
> >
> > --
> > Gabriel Lavoie
> > glavoie at gmail.com
> >
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/77db2349/attachment-0001.html>

From timur.tkachev at gmail.com  Tue Aug 16 16:19:11 2011
From: timur.tkachev at gmail.com (Timur Tkachev)
Date: Tue, 16 Aug 2011 10:19:11 -0400
Subject: [pypy-dev] python 3
Message-ID: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>

Hello,

Guys, maybe my question had been asked numerous times, but I couldn't google
even a remote answer to it.  What are the plans of python 3 support?  Please
shed some light on this topic.  Last poll in your blog regarding what's
holding off the pypy usage & acceptance showed that people do have an
interest in this matter.  Maybe there are people that can sponsor this work,
maybe there is a branch with initial support implemented, maybe there are
people eager to start helping you - we, python users and developers don't
know, because there is a strange veto talking about it.  It is very
important for python to adopt the 3rd version.  Please share your thoughts.

Thank you, and please keep doing the awesome work!

-
Timur
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/92231c0f/attachment.html>

From dje.gcc at gmail.com  Tue Aug 16 17:39:35 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Tue, 16 Aug 2011 11:39:35 -0400
Subject: [pypy-dev] Linux 3.0 kernel fallout
Message-ID: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>

It appears that the tests for 'linux2' in Python and PyPy libraries
are broken by the appearance of Linux 3.0 kernels.

- David

From lac at openend.se  Tue Aug 16 17:56:45 2011
From: lac at openend.se (Laura Creighton)
Date: Tue, 16 Aug 2011 17:56:45 +0200
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: Message from David Edelsohn <dje.gcc@gmail.com> of "Tue,
	16 Aug 2011 11:39:35 EDT."
	<CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
Message-ID: <201108161556.p7GFujtb022682@theraft.openend.se>

In a message of Tue, 16 Aug 2011 11:39:35 EDT, David Edelsohn writes:
>It appears that the tests for 'linux2' in Python and PyPy libraries
>are broken by the appearance of Linux 3.0 kernels.
>
>- David

Can you make a bug report about this?  Things sent to the mailing list
tend to get forgotten.

Laura

From amauryfa at gmail.com  Tue Aug 16 17:57:41 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 16 Aug 2011 17:57:41 +0200
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
Message-ID: <CAGmFidbsmx9LWpZu7E0So-cTpZFBi2y0Z23bmyk7XZyYpt3Dgg@mail.gmail.com>

2011/8/16 David Edelsohn <dje.gcc at gmail.com>

> It appears that the tests for 'linux2' in Python and PyPy libraries
> are broken by the appearance of Linux 3.0 kernels.
>

Yes, PyPy is consistent with CPython2.7 in this aspect :-)

By the way, does CPython work at all on linux 3.0?
what do you get with this command?

python -c 'import sys; print sys.platform'

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/93f7bf75/attachment.html>

From joe at goldthwaites.com  Tue Aug 16 18:06:06 2011
From: joe at goldthwaites.com (Joe Goldthwaite)
Date: Tue, 16 Aug 2011 09:06:06 -0700
Subject: [pypy-dev] A basic question.
In-Reply-To: <4E49A325.7000305@gmail.com>
References: <CA+FoDzojOGkx8nx4DwvJcJQhFoz4S=Tv0TeUgwpBgjiQ4LXZbw@mail.gmail.com>
	<4E49A325.7000305@gmail.com>
Message-ID: <CA+FoDzrarD=_Da7djY2+_Q-a2E8jd2U0YKDLcBP3Ztj=kzHHyw@mail.gmail.com>

Thanks Anto!  Or I should say molti grazie!

I feel a little stupid. I saw that line in the documentation but I the "curl
-O' confused me. I'm an old Windows programmer so when I see a bunch of
Linux install instructions I tend to skim over them looking for something
that makes sense.  In this case I didn't see anything.  Now that you've
clarified it it seems pretty obvious.

Thanks again!

On Mon, Aug 15, 2011 at 3:52 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> Hello Joe,
>
>
> On 15/08/11 23:48, Joe Goldthwaite wrote:
>
>> I apologize if this is the wrong place for this. In looking at the
>> archives
>> this list seems to be more geared to the developers than users.  I
>> couldn't
>> find a list for end users though so I'm going to post it here.
>>
>> I'd like to try pypy for a simple project.  I'm trying to use SUDS to
>> retrieve
>> data from a web service.  SUDS uses a setup.py file to do the install.
>>  When I
>> try that under pypy I get a message;
>>
>>  ImportError: No module named setuptools
>>
>> I know I need setup tools but how do you install that?
>>
>
> look here:
> http://doc.pypy.org/en/latest/**getting-started.html#**installing-pypy<http://doc.pypy.org/en/latest/getting-started.html#installing-pypy>
>
> in particular, you need to download and execute distribute_setup.py.
>
> ciao,
> Anto
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/b93720f9/attachment.html>

From drsalists at gmail.com  Tue Aug 16 18:45:30 2011
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 16 Aug 2011 09:45:30 -0700
Subject: [pypy-dev] python 3
In-Reply-To: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
Message-ID: <CAGGBd_ocNaOzxCRXxUh8E_nvZ_OP8K7DdD5SxnZ1G6z=297hMw@mail.gmail.com>

I'm inclined to agree: Python 3.x is important for PyPy's future.

On Tue, Aug 16, 2011 at 7:19 AM, Timur Tkachev <timur.tkachev at gmail.com>wrote:

> Hello,
>
> Guys, maybe my question had been asked numerous times, but I couldn't
> google even a remote answer to it.  What are the plans of python 3 support?
>  Please shed some light on this topic.  Last poll in your blog regarding
> what's holding off the pypy usage & acceptance showed that people do have an
> interest in this matter.  Maybe there are people that can sponsor this work,
> maybe there is a branch with initial support implemented, maybe there are
> people eager to start helping you - we, python users and developers don't
> know, because there is a strange veto talking about it.  It is very
> important for python to adopt the 3rd version.  Please share your thoughts.
>
> Thank you, and please keep doing the awesome work!
>
> -
> Timur
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/d3fde432/attachment.html>

From benjamin at python.org  Tue Aug 16 18:51:07 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 16 Aug 2011 11:51:07 -0500
Subject: [pypy-dev] python 3
In-Reply-To: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
Message-ID: <CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>

2011/8/16 Timur Tkachev <timur.tkachev at gmail.com>:
> Hello,
> Guys, maybe my question had been asked numerous times, but I couldn't google
> even a remote answer to it. ?What are the plans of python 3 support? ?Please
> shed some light on this topic. ?Last poll in your blog regarding what's
> holding off the pypy usage & acceptance showed that people do have an
> interest in this matter. ?Maybe there are people that can sponsor this work,
> maybe there is a branch with initial support implemented, maybe there are
> people eager to start helping you - we, python users and developers don't
> know, because there is a strange veto talking about it. ?It is very
> important for python to adopt the 3rd version. ?Please share your thoughts.

Essentially, there are no current plans. There's no opposition; it's
simply not a high priority. This could change with a sponsorship,
however.


-- 
Regards,
Benjamin

From amauryfa at gmail.com  Tue Aug 16 20:36:18 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 16 Aug 2011 20:36:18 +0200
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: <CAAV8QkYcqvrzat-3ADV70vO5nFW1bkFjD4ugJW5RdtiOVX0EOA@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
	<CAGmFidbsmx9LWpZu7E0So-cTpZFBi2y0Z23bmyk7XZyYpt3Dgg@mail.gmail.com>
	<CAAV8QkYcqvrzat-3ADV70vO5nFW1bkFjD4ugJW5RdtiOVX0EOA@mail.gmail.com>
Message-ID: <CAGmFidbyCHzjQrfqbFxukeTjZpPb8JYGcZEL6tEz3v5u=EfkuA@mail.gmail.com>

2011/8/16 Reiner | chocri <reiner at chocri.de>

> $ python -c 'import sys; print sys.platform'
> linux2
>
> $ uname -r
> 3.0.0-1-686-pae
>

So, where is the issue?

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/08e6aa14/attachment.html>

From yselivanov.ml at gmail.com  Tue Aug 16 21:05:59 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 16 Aug 2011 15:05:59 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
Message-ID: <B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>

Is it possible for pypy core developers to create a high-level roadmap with what needs to be done and where?  Should python3 be another translation target?  Will it be required to touch rpython spec?  What data structures need to be introduced?  etc.  I don't think this planning will take weeks of work, but it will help everyone to understand how much time and money should be invested in the matter.

I myself would like to start working on porting, but I simply don't know where to begin. Perhaps my company will support the implementation.

On 2011-08-16, at 12:51 PM, Benjamin Peterson wrote:

> 2011/8/16 Timur Tkachev <timur.tkachev at gmail.com>:
>> Hello,
>> Guys, maybe my question had been asked numerous times, but I couldn't google
>> even a remote answer to it.  What are the plans of python 3 support?  Please
>> shed some light on this topic.  Last poll in your blog regarding what's
>> holding off the pypy usage & acceptance showed that people do have an
>> interest in this matter.  Maybe there are people that can sponsor this work,
>> maybe there is a branch with initial support implemented, maybe there are
>> people eager to start helping you - we, python users and developers don't
>> know, because there is a strange veto talking about it.  It is very
>> important for python to adopt the 3rd version.  Please share your thoughts.
> 
> Essentially, there are no current plans. There's no opposition; it's
> simply not a high priority. This could change with a sponsorship,
> however.
> 
> 
> -- 
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From santagada at gmail.com  Tue Aug 16 22:20:49 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Tue, 16 Aug 2011 17:20:49 -0300
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: <CAGmFidbyCHzjQrfqbFxukeTjZpPb8JYGcZEL6tEz3v5u=EfkuA@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
	<CAGmFidbsmx9LWpZu7E0So-cTpZFBi2y0Z23bmyk7XZyYpt3Dgg@mail.gmail.com>
	<CAAV8QkYcqvrzat-3ADV70vO5nFW1bkFjD4ugJW5RdtiOVX0EOA@mail.gmail.com>
	<CAGmFidbyCHzjQrfqbFxukeTjZpPb8JYGcZEL6tEz3v5u=EfkuA@mail.gmail.com>
Message-ID: <CA+HFc86oaxvQv3mf_qkBK_ujZhL+aEhuJefk6FGJdMcDAO=cTg@mail.gmail.com>

On Tue, Aug 16, 2011 at 3:36 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2011/8/16 Reiner | chocri <reiner at chocri.de>
>>
>> $ python -c 'import sys; print sys.platform'
>> linux2
>> $ uname -r
>> 3.0.0-1-686-pae
>
> So, where is the issue?

I saw that some distros are putting in place patches to make linux 3.0
to appear to be 2.6.40 to applications, maybe this is the case here,
that you are looking at a patched python but vanilla would not work.


-- 
Leonardo Santagada

From benjamin at python.org  Wed Aug 17 00:39:04 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 16 Aug 2011 17:39:04 -0500
Subject: [pypy-dev] python 3
In-Reply-To: <B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
Message-ID: <CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>

2011/8/16 Yury Selivanov <yselivanov.ml at gmail.com>:
> Is it possible for pypy core developers to create a high-level roadmap with what needs to be done and where? ?Should python3 be another translation target? ?Will it be required to touch rpython spec? ?What data structures need to be introduced? ?etc. ?I don't think this planning will take weeks of work, but it will help everyone to understand how much time and money should be invested in the matter.

First of all, there are some rather large decisions to be made:

1. Port everything (Python interpreter, RPython) over to Python 3 and
only support Python 3. This would probably be the cleanest and easiest
in the longterm solution, but I doubt many are willing to accept it
quite yet.

2. Somehow maintain Python 2 and 3 in the same codebase. It sounds
like a hideous mess to me. (I'm happy to be proven wrong.)

3. Maintain a Python 3 interpreter in a separate repo or branch. This
is probably the best compromise, but it requires the constant
maintenance of someone merging the current head work.

Then someone has to buckle down and do the actual porting. Depending
on the option selected above, the amount of work will vary from huge
to colossal. If you pick option 2, you have to figure out how to test
both versions. I imagine there will be quite a tangled mess with
unicode.

At any rate, some of the initial steps which are compatible with
Python 2 such as removing tuple unpacking and normalizing raise
statements can now be taken. They might even make the codebase a bit
cleaner.



-- 
Regards,
Benjamin

From dje.gcc at gmail.com  Wed Aug 17 01:13:43 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Tue, 16 Aug 2011 19:13:43 -0400
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: <CA+HFc86oaxvQv3mf_qkBK_ujZhL+aEhuJefk6FGJdMcDAO=cTg@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
	<CAGmFidbsmx9LWpZu7E0So-cTpZFBi2y0Z23bmyk7XZyYpt3Dgg@mail.gmail.com>
	<CAAV8QkYcqvrzat-3ADV70vO5nFW1bkFjD4ugJW5RdtiOVX0EOA@mail.gmail.com>
	<CAGmFidbyCHzjQrfqbFxukeTjZpPb8JYGcZEL6tEz3v5u=EfkuA@mail.gmail.com>
	<CA+HFc86oaxvQv3mf_qkBK_ujZhL+aEhuJefk6FGJdMcDAO=cTg@mail.gmail.com>
Message-ID: <CAGWvny=cokRSP3d+hASRgCDr6z6mhwC_z9nGie8BvAtzseBCVw@mail.gmail.com>

On Tue, Aug 16, 2011 at 4:20 PM, Leonardo Santagada <santagada at gmail.com> wrote:
> On Tue, Aug 16, 2011 at 3:36 PM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
>> 2011/8/16 Reiner | chocri <reiner at chocri.de>
>>>
>>> $ python -c 'import sys; print sys.platform'
>>> linux2
>>> $ uname -r
>>> 3.0.0-1-686-pae
>>
>> So, where is the issue?

Did you *build* Python on the system with a Linux 3.0 kernel?

$ uname -r
3.0.1
$ /usr/local/bin/python -c 'import sys; print sys.platform'
linux3

From piotr.skamruk at gmail.com  Wed Aug 17 01:27:51 2011
From: piotr.skamruk at gmail.com (Piotr Skamruk)
Date: Wed, 17 Aug 2011 01:27:51 +0200
Subject: [pypy-dev] Linux 3.0 kernel fallout
In-Reply-To: <CAGWvny=cokRSP3d+hASRgCDr6z6mhwC_z9nGie8BvAtzseBCVw@mail.gmail.com>
References: <CAGWvnynJgt5=vz0A+3i=QyFNsP36Np=YG-rgry1AmYLEZXbL2w@mail.gmail.com>
	<CAGmFidbsmx9LWpZu7E0So-cTpZFBi2y0Z23bmyk7XZyYpt3Dgg@mail.gmail.com>
	<CAAV8QkYcqvrzat-3ADV70vO5nFW1bkFjD4ugJW5RdtiOVX0EOA@mail.gmail.com>
	<CAGmFidbyCHzjQrfqbFxukeTjZpPb8JYGcZEL6tEz3v5u=EfkuA@mail.gmail.com>
	<CA+HFc86oaxvQv3mf_qkBK_ujZhL+aEhuJefk6FGJdMcDAO=cTg@mail.gmail.com>
	<CAGWvny=cokRSP3d+hASRgCDr6z6mhwC_z9nGie8BvAtzseBCVw@mail.gmail.com>
Message-ID: <CAO_YSZNJPpuBEUE3KO2t1vLNe=SbDpZ5knw7OHn2-3j4O_Wt0w@mail.gmail.com>

strange, my pypy build was also done on uname -r == 3.0.0-8-generic
and i have 'linux2' as sys.platform

cpython is from distribution (ubuntu) and probably could be build with
kernel earlier than 3.0.0

2011/8/17 David Edelsohn <dje.gcc at gmail.com>:
> On Tue, Aug 16, 2011 at 4:20 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>> On Tue, Aug 16, 2011 at 3:36 PM, Amaury Forgeot d'Arc
>> <amauryfa at gmail.com> wrote:
>>> 2011/8/16 Reiner | chocri <reiner at chocri.de>
>>>>
>>>> $ python -c 'import sys; print sys.platform'
>>>> linux2
>>>> $ uname -r
>>>> 3.0.0-1-686-pae
>>>
>>> So, where is the issue?
>
> Did you *build* Python on the system with a Linux 3.0 kernel?
>
> $ uname -r
> 3.0.1
> $ /usr/local/bin/python -c 'import sys; print sys.platform'
> linux3
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From alex.gaynor at gmail.com  Wed Aug 17 02:25:37 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 16 Aug 2011 19:25:37 -0500
Subject: [pypy-dev] python 3
In-Reply-To: <CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
Message-ID: <CAFRnB2XpyMR5ZPO8aFjJ5AnJ_OG3dH3_zVDU91eX3yOxSrV_YQ@mail.gmail.com>

On Tue, Aug 16, 2011 at 5:39 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2011/8/16 Yury Selivanov <yselivanov.ml at gmail.com>:
> > Is it possible for pypy core developers to create a high-level roadmap
> with what needs to be done and where?  Should python3 be another translation
> target?  Will it be required to touch rpython spec?  What data structures
> need to be introduced?  etc.  I don't think this planning will take weeks of
> work, but it will help everyone to understand how much time and money should
> be invested in the matter.
>
> First of all, there are some rather large decisions to be made:
>
> 1. Port everything (Python interpreter, RPython) over to Python 3 and
> only support Python 3. This would probably be the cleanest and easiest
> in the longterm solution, but I doubt many are willing to accept it
> quite yet.
>
> 2. Somehow maintain Python 2 and 3 in the same codebase. It sounds
> like a hideous mess to me. (I'm happy to be proven wrong.)
>
> 3. Maintain a Python 3 interpreter in a separate repo or branch. This
> is probably the best compromise, but it requires the constant
> maintenance of someone merging the current head work.
>
> Then someone has to buckle down and do the actual porting. Depending
> on the option selected above, the amount of work will vary from huge
> to colossal. If you pick option 2, you have to figure out how to test
> both versions. I imagine there will be quite a tangled mess with
> unicode.
>
> At any rate, some of the initial steps which are compatible with
> Python 2 such as removing tuple unpacking and normalizing raise
> statements can now be taken. They might even make the codebase a bit
> cleaner.
>
>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Personally I think #3 is the only sane path.  We *need* a Python 2 VM for
the forseeable future.  We're pretty lucky in that the JIT, GC, and all the
honest to god complex code is totally seperate from the VM, so just
supporting 2 Python VMs is kind of easy (compared to maintaing 2 JITs or
something).

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110816/47eb3a67/attachment-0001.html>

From yselivanov.ml at gmail.com  Wed Aug 17 04:39:05 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 16 Aug 2011 22:39:05 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
Message-ID: <8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>

Re option #1, just trying to start a discussion:

I know it's a hard topic, but why not to adapt python 3?  Python 3 is the future, everybody understands and accepts that.  Pypy doesn't have substantially good support of c-extenstions, so, let's say, numpy has to be rewritten anyways.  RDB drivers are also poorly supported, while python 3 has an excellent pypostgresql written entirely in python.  Django, twisted and even zope will support python 3 eventually, it is a matter of time.  Why not to start the move now, and do all the heavy work of rewriting numpy & other libs in python 3 to save time later?

On 2011-08-16, at 6:39 PM, Benjamin Peterson wrote:

> 2011/8/16 Yury Selivanov <yselivanov.ml at gmail.com>:
>> Is it possible for pypy core developers to create a high-level roadmap with what needs to be done and where?  Should python3 be another translation target?  Will it be required to touch rpython spec?  What data structures need to be introduced?  etc.  I don't think this planning will take weeks of work, but it will help everyone to understand how much time and money should be invested in the matter.
> 
> First of all, there are some rather large decisions to be made:
> 
> 1. Port everything (Python interpreter, RPython) over to Python 3 and
> only support Python 3. This would probably be the cleanest and easiest
> in the longterm solution, but I doubt many are willing to accept it
> quite yet.
> 
> 2. Somehow maintain Python 2 and 3 in the same codebase. It sounds
> like a hideous mess to me. (I'm happy to be proven wrong.)
> 
> 3. Maintain a Python 3 interpreter in a separate repo or branch. This
> is probably the best compromise, but it requires the constant
> maintenance of someone merging the current head work.
> 
> Then someone has to buckle down and do the actual porting. Depending
> on the option selected above, the amount of work will vary from huge
> to colossal. If you pick option 2, you have to figure out how to test
> both versions. I imagine there will be quite a tangled mess with
> unicode.
> 
> At any rate, some of the initial steps which are compatible with
> Python 2 such as removing tuple unpacking and normalizing raise
> statements can now be taken. They might even make the codebase a bit
> cleaner.
> 
> 
> 
> -- 
> Regards,
> Benjamin


From benjamin at python.org  Wed Aug 17 04:42:24 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 16 Aug 2011 21:42:24 -0500
Subject: [pypy-dev] python 3
In-Reply-To: <8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
Message-ID: <CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>

2011/8/16 Yury Selivanov <yselivanov.ml at gmail.com>:
> Re option #1, just trying to start a discussion:
>
> I know it's a hard topic, but why not to adapt python 3? ?Python 3 is the future, everybody understands and accepts that. ?Pypy doesn't have substantially good support of c-extenstions, so, let's say, numpy has to be rewritten anyways. ?RDB drivers are also poorly supported, while python 3 has an excellent pypostgresql written entirely in python. ?Django, twisted and even zope will support python 3 eventually, it is a matter of time. ?Why not to start the move now, and do all the heavy work of rewriting numpy & other libs in python 3 to save time later?

Likely the usual argument about Python 3: Most libraries and code are
Python 2. People are interested in using PyPy now and now is Python 2.

(I personally don't care but this is because I mostly work on Python
implementations, and my income doesn't depend on Python 2. :))



-- 
Regards,
Benjamin

From yselivanov.ml at gmail.com  Wed Aug 17 04:54:55 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 16 Aug 2011 22:54:55 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
Message-ID: <8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>

Yes, but that is kind of a weak argument, since the situation with python 3 changes quickly.  More and more libraries are being ported each month.  Supporting python 2 obviously just harms the python ecosystem, as nobody interested in having two languages ;)  And pypy could be a very strong argument, and not only for a python community.

On 2011-08-16, at 10:42 PM, Benjamin Peterson wrote:

> 2011/8/16 Yury Selivanov <yselivanov.ml at gmail.com>:
>> Re option #1, just trying to start a discussion:
>> 
>> I know it's a hard topic, but why not to adapt python 3?  Python 3 is the future, everybody understands and accepts that.  Pypy doesn't have substantially good support of c-extenstions, so, let's say, numpy has to be rewritten anyways.  RDB drivers are also poorly supported, while python 3 has an excellent pypostgresql written entirely in python.  Django, twisted and even zope will support python 3 eventually, it is a matter of time.  Why not to start the move now, and do all the heavy work of rewriting numpy & other libs in python 3 to save time later?
> 
> Likely the usual argument about Python 3: Most libraries and code are
> Python 2. People are interested in using PyPy now and now is Python 2.
> 
> (I personally don't care but this is because I mostly work on Python
> implementations, and my income doesn't depend on Python 2. :))
> 
> 
> 
> -- 
> Regards,
> Benjamin


From yselivanov.ml at gmail.com  Wed Aug 17 05:01:24 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 16 Aug 2011 23:01:24 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <CAFRnB2XpyMR5ZPO8aFjJ5AnJ_OG3dH3_zVDU91eX3yOxSrV_YQ@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<CAFRnB2XpyMR5ZPO8aFjJ5AnJ_OG3dH3_zVDU91eX3yOxSrV_YQ@mail.gmail.com>
Message-ID: <D768C553-7648-4C74-97E7-118D60F08455@gmail.com>

On 2011-08-16, at 8:25 PM, Alex Gaynor wrote:
> Personally I think #3 is the only sane path.  We *need* a Python 2 VM for the forseeable future.  We're pretty lucky in that the JIT, GC, and all the honest to god complex code is totally seperate from the VM, so just supporting 2 Python VMs is kind of easy (compared to maintaing 2 JITs or something).

Well, if everybody agrees on the 3rd option, then can we have at least the process of porting outlined and reviewed by core devs?  No need for a super-detailed PEP, though.  A simple guideline would help a lot.

From arigo at tunes.org  Wed Aug 17 11:41:25 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 17 Aug 2011 11:41:25 +0200
Subject: [pypy-dev] python 3
In-Reply-To: <D768C553-7648-4C74-97E7-118D60F08455@gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<CAFRnB2XpyMR5ZPO8aFjJ5AnJ_OG3dH3_zVDU91eX3yOxSrV_YQ@mail.gmail.com>
	<D768C553-7648-4C74-97E7-118D60F08455@gmail.com>
Message-ID: <CAMSv6X0ikO4Qp4xMcWpNLw166HVqLsDsAuvkLhe3HpCfUKEHDA@mail.gmail.com>

Hi Yury,

On Wed, Aug 17, 2011 at 5:01 AM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> Well, if everybody agrees on the 3rd option, then can we have at least the process of porting outlined and reviewed by core devs? ?No need for a super-detailed PEP, though. ?A simple guideline would help a lot.

Yes, I think it's the only workable solution.  I suppose that what
will happen is that some sub-group of people starts the python3 branch
and keeps working on making a Python 3 interpreter (written in Python
2.x, for x in (6, 7)).  With regular merges from the trunk it should
be doable.  It is exactly how the 2.7 branch was done, at a time where
we supported only 2.5.

This would remain as a branch for the foreseeable future though,
because we still need a Python 2 interpreter, if only to run our own
translation toolchain on (and not suffer the 2.5x slow-down of running
it on CPython 2.x).  But at some later point we can think about
switching the whole code base of the translation toolchain to Python
3, which I also regard as quite some work --- but it will likely have
to be done some day.  Once this is done we can drop Python 2 entirely
and use the Python 3 branch as trunk.


A bient?t,

Armin.

From exarkun at twistedmatrix.com  Wed Aug 17 15:04:41 2011
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Wed, 17 Aug 2011 13:04:41 -0000
Subject: [pypy-dev] python 3
In-Reply-To: <8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
Message-ID: <20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>

On 02:54 am, yselivanov.ml at gmail.com wrote:
>Yes, but that is kind of a weak argument, since the situation with 
>python 3 changes quickly.  More and more libraries are being ported 
>each month.  Supporting python 2 obviously just harms the python 
>ecosystem, as nobody interested in having two languages ;)

I don't understand what the wink is for.  Hopefully it means that's just 
a joke?  Because that's what it sounds like - support for Python 2 in 
PyPy harms nothing.

Jean-Paul

From bmbouter at gmail.com  Wed Aug 17 15:11:00 2011
From: bmbouter at gmail.com (Brian Bouterse)
Date: Wed, 17 Aug 2011 09:11:00 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
Message-ID: <CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>

I'm assuming it was a joke.

A huge amount of people today and likely over the next few years will
continue to rely on python 2.x where x (6,7).  Let's not downplay the
importance of PyPy supporting those communities.  I agree with you
Jean-Paul, Python 2 support in PyPy harms nothing.

Brian

On Wed, Aug 17, 2011 at 9:04 AM, <exarkun at twistedmatrix.com> wrote:

> On 02:54 am, yselivanov.ml at gmail.com wrote:
>
>> Yes, but that is kind of a weak argument, since the situation with python
>> 3 changes quickly.  More and more libraries are being ported each month.
>>  Supporting python 2 obviously just harms the python ecosystem, as nobody
>> interested in having two languages ;)
>>
>
> I don't understand what the wink is for.  Hopefully it means that's just a
> joke?  Because that's what it sounds like - support for Python 2 in PyPy
> harms nothing.
>
> Jean-Paul
>
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>



-- 
Brian Bouterse
ITng Services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110817/01bfea8e/attachment.html>

From tobami at googlemail.com  Wed Aug 17 20:30:34 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 17 Aug 2011 20:30:34 +0200
Subject: [pypy-dev] python 3
In-Reply-To: <CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
Message-ID: <CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>

@Armin
> This would remain as a branch for the foreseeable future though,
> because we still need a Python 2 interpreter, if only to run our own
> translation toolchain on (and not suffer the 2.5x slow-down of running
> it on CPython 2.x).
I don't quite follow. Switching to Python 3 (I am not saying that
would be a good idea, just clarifying) and release as, let's say, PyPy
2.0, doesn't mean that the impending 1.6 release would go away. Python
3 users would use PyPy 2.0+, while users of Python 2.7 would use PyPy
1.6. You would still be able to compile PyPy and its Python 2.7
toolchain with PyPy 1.6, thus getting the 2.5x speed up.

Python 2.7 users would only miss on newer, faster PyPy releases, which
is not the black and white picture some where describing. The only
condition needed for this scenario to come about are continued
availability of PyPy 1.6 packages and important bug fixes.

That said, 1.6 is probably not the right point for Python 2.x
end-of-line, but maybe after 1.7 things will look differently...

Cheers,
Miquel


2011/8/17 Brian Bouterse <bmbouter at gmail.com>:
> I'm assuming it was a joke.
> A huge amount of people today and likely over the next few years will
> continue to rely on python 2.x where x (6,7). ?Let's not downplay the
> importance of PyPy supporting those communities. ?I agree with you
> Jean-Paul, Python 2 support in PyPy harms nothing.
> Brian
>
> On Wed, Aug 17, 2011 at 9:04 AM, <exarkun at twistedmatrix.com> wrote:
>>
>> On 02:54 am, yselivanov.ml at gmail.com wrote:
>>>
>>> Yes, but that is kind of a weak argument, since the situation with python
>>> 3 changes quickly. ?More and more libraries are being ported each month.
>>> ?Supporting python 2 obviously just harms the python ecosystem, as nobody
>>> interested in having two languages ;)
>>
>> I don't understand what the wink is for. ?Hopefully it means that's just a
>> joke? ?Because that's what it sounds like - support for Python 2 in PyPy
>> harms nothing.
>>
>> Jean-Paul
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
> --
> Brian Bouterse
> ITng Services
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>

From glavoie at gmail.com  Wed Aug 17 22:42:25 2011
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Wed, 17 Aug 2011 16:42:25 -0400
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAKp1Nw5jCJuCHigCsG_U5kL9uFVz4tWvL2dCkpS904dY2Pn=fQ@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
	<CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
	<201108160628.57358.naylor.b.david@gmail.com>
	<CAKp1Nw4-4Uf7YrgHTBvJ4fXcL=osq7cZWEBULwAddEGFsqaLBQ@mail.gmail.com>
	<CAMQxofkLMVXwc2D6vQoj97=zdUiqv03nNGGPOKfD_7rBRfD+QA@mail.gmail.com>
	<CAKp1Nw5jCJuCHigCsG_U5kL9uFVz4tWvL2dCkpS904dY2Pn=fQ@mail.gmail.com>
Message-ID: <CAKp1Nw7H1=M4fQQSRxGG+f2rWSP0GGGLFu06Y_LcEx4GYozs9w@mail.gmail.com>

Hum... It seems it's the end of my FreeBSD buildslave. :( I don't know when
it time the memory requirement jumped to over 4GB (Armin told me it was now
around 4.5GB for 64bits). I've been able to build a JIT enabled pypy binary
using Python and GCC 4.6 but it took over 10 hours. I'm unable to do the
same process using pypy to translate as its only swapping...

Gabriel

2011/8/16 Gabriel Lavoie <glavoie at gmail.com>

> I'm on 7.4-STABLE. Anyway, as I understand, FreeBSD will never upgrade its
> core GCC version higher than 4.2.2 due to GPL version switch to GPLv3...
> Knowing there is an issue with GCC 4.2, I'll finish my testing with the
> latest GCC available from ports and I'll report. It might be a good idea to
> make a newer GCC version a requirement in your PyPy port. A better idea
> would be to create "officials" PyPy build (probably just with the jit
> default setup, 32bits and 64bits) and make a port that would install the
> binary package instead of translating the source code, since translation has
> a lot of requirements (including hardware) and takes a lot of time.
>
>
> Gabriel
>
> 2011/8/16 David Naylor <naylor.b.david at gmail.com>
>
>> I was not aware of that.  If you are on FreeBSD-current then clang is
>> also available (and is also in ports).  clang however takes much longer
>> to compile than gcc but can easily be run in parallel whereas gcc
>> used too much memory to allow that.
>>
>> On 16 August 2011 08:10, Gabriel Lavoie <glavoie at gmail.com> wrote:
>> > Okay! It seems I've hit the GCC 4.2 memory usage bug:
>> > https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391
>> > http://comments.gmane.org/gmane.comp.python.pypy/7634
>> >
>> > I'm now trying to build with GCC 4.6 (a port is available) and I'll
>> report
>> > back.
>> >
>> > Gabriel
>> >
>> > 2011/8/16 David Naylor <naylor.b.david at gmail.com>
>> >>
>> >> On Tuesday, 16 August 2011 05:50:18 Gabriel Lavoie wrote:
>> >> > Hi David,
>> >> >      I took a look at your port and it looks good! The patch you made
>> >> > for
>> >> > SANDBOX should probably not be needed, with a fix upstream. It seems
>> >> > PyPy
>> >> > still have a few "hardcoded path" issues that need to be looked at
>> for
>> >> > FreeBSD (and probably for Darwin too for tools like Darwin Ports). I
>> >> > felt
>> >> > on one of those issues this morning with "expat.h" and "libexpat.so".
>> >> > The
>> >> > translation process didn't look at /usr/local/include and
>> /usr/local/lib
>> >> > to find them. Actually, there are a few places where the
>> >> > "pypy/translator/platform" module should be use to find include/lib
>> >> > paths.
>> >> > I made the initial "freebsd.py" file in there that contains "general"
>> >> > references to include/lib directories in "/usr/local".
>> >>
>> >> Thanks, sandbox has not been tested as I do not know how to.  An
>> example
>> >> script that makes sandbox behave like a normal python/pypy instance
>> would
>> >> be
>> >> nice...
>> >>
>> >> On the expat issue:
>> >> # ldd `which pypy1.5`
>> >> /usr/local/bin/pypy1.5:
>> >>        libm.so.5 => /lib/libm.so.5 (0x801664000)
>> >>        libintl.so.9 => /usr/local/lib/libintl.so.9 (0x801885000)
>> >>        libssl.so.6 => /usr/lib/libssl.so.6 (0x801a8e000)
>> >>        libcrypto.so.6 => /lib/libcrypto.so.6 (0x801ce1000)
>> >>        libexpat.so.6 => /usr/local/lib/libexpat.so.6 (0x802082000)
>> >>        libbz2.so.4 => /usr/lib/libbz2.so.4 (0x8022a5000)
>> >>        libz.so.6 => /lib/libz.so.6 (0x8024b5000)
>> >>        librt.so.1 => /usr/lib/librt.so.1 (0x8026cb000)
>> >>        libutil.so.9 => /lib/libutil.so.9 (0x8028d0000)
>> >>        libffi.so.5 => /usr/local/lib/libffi.so.5 (0x802ae1000)
>> >>        libncurses.so.8 => /lib/libncurses.so.8 (0x802ce7000)
>> >>        libcrypt.so.5 => /lib/libcrypt.so.5 (0x802f34000)
>> >>        libthr.so.3 => /lib/libthr.so.3 (0x803154000)
>> >>        libc.so.7 => /lib/libc.so.7 (0x803377000)
>> >>        libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x8036bf000)
>> >>
>> >> I also see those error yet it appears expat is picked up somehow, maybe
>> as
>> >> a
>> >> depenendency?
>> >>
>> >> > About my current issue, maybe you've seen it. When I try to build
>> PyPy
>> >> > using CPython, I get a "Broken Pipe" error and the process fails
>> (using
>> >> > the standard "python translate.py -O2" command). Have you seen this
>> >> > problem recently? I'm trying with the latest trunk. It seems the
>> >> > "python"
>> >> > process runs out of memory as I have this in my kernel log:
>> >>
>> >> I have not encountered that problem before (except I suffer from
>> ENOMEM).
>> >>
>> >> > swap_pager_getswapspace(16): failed
>> >> > pid 15107 (cc1), uid 1001, was killed: out of swap space
>> >> > swap_pager: out of swap space
>> >> > swap_pager_getswapspace(2): failed
>> >> > pid 37981 (python), uid 1001, was killed: out of swap space
>> >>
>> >> I have seen this problem before.  FreeBSD hard codes the size of swap
>> it
>> >> can
>> >> access, and does not adapt it based on RAM or active swap.  To fix that
>> >> try
>> >> the following in /boot/loader.conf:
>> >>
>> >> # Increase max swap zone to allow for ~64GB of swap
>> >> kern.maxswzone="268435456"  # 256MB
>> >>
>> >> > PyPy used to build correctly but I haven't tried to build it in the
>> last
>> >> > 6
>> >> > months...
>> >>
>> >> I have successfully built pypy-1.5 and trunk (~1 week ago), so it is
>> >> possible.
>> >> If you are still having problems please send me the installed physical
>> >> memory,
>> >> and the output of `swapinfo` and `uname -a`.
>> >>
>> >> > 2011/8/15 David Naylor <naylor.b.david at gmail.com>
>> >> >
>> >> > > On Monday, 15 August 2011 17:30:33 Gabriel Lavoie wrote:
>> >> > > > Hi David,
>> >> > > >
>> >> > > >     for your information, I'm currently trying to make my FreeBSD
>> >> > >
>> >> > > buildbot
>> >> > >
>> >> > > > back online. It broke when the requirement for the "pypy" binary
>> to
>> >> > > > build PyPy appeared. It seems there are still some issues to make
>> >> > > > PyPy
>> >> > > > build again under FreeBSD. I'll keep you updated.
>> >> > >
>> >> > > Thanks for the update.  I have been working on a port for pypy.
>>  The
>> >> > > code
>> >> > > is
>> >> > > at github.com/DragonSA/pypy.  It may be able to help you compile
>> pypy
>> >> > > under
>> >> > > FreeBSD.  There is a branch called 1.6 that is compatible with
>> trunk.
>> >> > >  To
>> >> > > compile from 1.6 set WRKSRC to point to a fresh checkout of pypy:
>> >> > >
>> >> > > # make install WRKSRC=/path/to/hg/checkout/of/pypy
>> >> > >
>> >> > > The master branch uses the 1.5 sources.
>> >> > >
>> >> > > > 2011/8/11 David Naylor <naylor.b.david at gmail.com>
>> >> > > >
>> >> > > > > On Thursday, 11 August 2011 11:08:18 Maciej Fijalkowski wrote:
>> >> > > > > > Hi
>> >> > > > > >
>> >> > > > > > A little status update from my side.
>> >> > > > > >
>> >> > > > > > I'm trying to get 1.6 out of the door and there are a few
>> things
>> >> > > > > > that stand out before we can do it.
>> >> > > > > >
>> >> > > > > > * Windows situation (a lot of crashes with missing liveness).
>> >> > > > > > Also
>> >> > > > > > lib-python tests take 10h which is a bit too much IMO (not a
>> new
>> >> > > > > > regression though)
>> >> > > > > > * OS X situation (missing buildbots, being addressed now)
>> >> > > > > > * A bunch of issues that either should be decided we don't
>> care
>> >> > > > > > or
>> >> > > > > > we do care and fix it.
>> >> > > > > > * Missing jitviewer release/demo that I'm doing right now
>> >> > > > > > * Alex reported sqlite crashes
>> >> > > > > > * random segfault when running trackgcroot, armin, did you
>> have
>> >> > > > > > a
>> >> > > > > > look? I can do if you don't
>> >> > > > >
>> >> > > > > Would anyone be interested in the FreeBSD situation?  If so
>> what
>> >> > >
>> >> > > support
>> >> > >
>> >> > > > > do you need?  I am currently updating my repository and will
>> >> > > > > report
>> >> > > > > back...
>> >> > > > >
>> >> > > > > _______________________________________________
>> >> > > > > pypy-dev mailing list
>> >> > > > > pypy-dev at python.org
>> >> > > > > http://mail.python.org/mailman/listinfo/pypy-dev
>> >
>> >
>> >
>> > --
>> > Gabriel Lavoie
>> > glavoie at gmail.com
>> >
>>
>
>
>
> --
> Gabriel Lavoie
> glavoie at gmail.com
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110817/556a8834/attachment-0001.html>

From yselivanov.ml at gmail.com  Thu Aug 18 01:01:17 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 17 Aug 2011 19:01:17 -0400
Subject: [pypy-dev] python 3
In-Reply-To: <CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
Message-ID: <BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>

+1 to the question.  Why can't it be that way?

On 2011-08-17, at 2:30 PM, Miquel Torres wrote:

> @Armin
>> This would remain as a branch for the foreseeable future though,
>> because we still need a Python 2 interpreter, if only to run our own
>> translation toolchain on (and not suffer the 2.5x slow-down of running
>> it on CPython 2.x).
> I don't quite follow. Switching to Python 3 (I am not saying that
> would be a good idea, just clarifying) and release as, let's say, PyPy
> 2.0, doesn't mean that the impending 1.6 release would go away. Python
> 3 users would use PyPy 2.0+, while users of Python 2.7 would use PyPy
> 1.6. You would still be able to compile PyPy and its Python 2.7
> toolchain with PyPy 1.6, thus getting the 2.5x speed up.
> 
> Python 2.7 users would only miss on newer, faster PyPy releases, which
> is not the black and white picture some where describing. The only
> condition needed for this scenario to come about are continued
> availability of PyPy 1.6 packages and important bug fixes.
> 
> That said, 1.6 is probably not the right point for Python 2.x
> end-of-line, but maybe after 1.7 things will look differently...
> 
> Cheers,
> Miquel
> 
> 
> 2011/8/17 Brian Bouterse <bmbouter at gmail.com>:
>> I'm assuming it was a joke.
>> A huge amount of people today and likely over the next few years will
>> continue to rely on python 2.x where x (6,7).  Let's not downplay the
>> importance of PyPy supporting those communities.  I agree with you
>> Jean-Paul, Python 2 support in PyPy harms nothing.
>> Brian
>> 
>> On Wed, Aug 17, 2011 at 9:04 AM, <exarkun at twistedmatrix.com> wrote:
>>> 
>>> On 02:54 am, yselivanov.ml at gmail.com wrote:
>>>> 
>>>> Yes, but that is kind of a weak argument, since the situation with python
>>>> 3 changes quickly.  More and more libraries are being ported each month.
>>>>  Supporting python 2 obviously just harms the python ecosystem, as nobody
>>>> interested in having two languages ;)
>>> 
>>> I don't understand what the wink is for.  Hopefully it means that's just a
>>> joke?  Because that's what it sounds like - support for Python 2 in PyPy
>>> harms nothing.
>>> 
>>> Jean-Paul
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>> 
>> 
>> 
>> --
>> Brian Bouterse
>> ITng Services
>> 
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>> 
>> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From wickedgrey at gmail.com  Thu Aug 18 02:13:56 2011
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Wed, 17 Aug 2011 17:13:56 -0700
Subject: [pypy-dev] python 3
In-Reply-To: <BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
	<BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>
Message-ID: <CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>

On Wed, Aug 17, 2011 at 4:01 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> +1 to the question. ?Why can't it be that way?

If by "that way" you mean "leave python 2.x behind post 1.6" I'd like
to note that IMO pypy has been under-acknowledged by the wider python
community for a very long time.  That's finally starting to change
(pypy production releases, cpython devs devoting resources to make
alternate implementations not second-class citizens, etc.), but by
abandoning the segment of the language with the largest userbase, the
project would go back to niche status again.  Yeah, doing so might
position pypy well to become the default python 3 implementation, but
I find it hard to imagine that tacking on another N years until pypy
is a significant percentage of python deployments is going to be good
for the project.

My $0.02,
Eli

From drsalists at gmail.com  Thu Aug 18 02:38:56 2011
From: drsalists at gmail.com (Dan Stromberg)
Date: Wed, 17 Aug 2011 17:38:56 -0700
Subject: [pypy-dev] python 3
In-Reply-To: <CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
	<BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>
	<CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>
Message-ID: <CAGGBd_o_-ibaNfqg49LdNBYnbdq6_SEAAH3hRUbh0Jgm5Y+J8w@mail.gmail.com>

On Wed, Aug 17, 2011 at 5:13 PM, Eli Stevens (Gmail)
<wickedgrey at gmail.com>wrote:

> On Wed, Aug 17, 2011 at 4:01 PM, Yury Selivanov <yselivanov.ml at gmail.com>
> wrote:
> > +1 to the question.  Why can't it be that way?
>
> If by "that way" you mean "leave python 2.x behind post 1.6" I'd like
> to note that IMO pypy has been under-acknowledged by the wider python
> community for a very long time.  That's finally starting to change
> (pypy production releases, cpython devs devoting resources to make
> alternate implementations not second-class citizens, etc.), but by
> abandoning the segment of the language with the largest userbase, the
> project would go back to niche status again.  Yeah, doing so might
> position pypy well to become the default python 3 implementation, but
> I find it hard to imagine that tacking on another N years until pypy
> is a significant percentage of python deployments is going to be good
> for the project.
>

There's a large difference between "Abandoning 2.x" and "Starting the ball
rolling toward 3.x in a timely manner".  If anything, not having a plan for
the move to 3.x is more likely to sideline the PyPy project.

BTW, a lot of sites use software that'll ask you which branches you want to
check each of your changesets into - IOW, click two buttons and your checkin
could go to 2.x and 3.x.  I don't know if there's anything free for
Mercurial like that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110817/55c80aaf/attachment.html>

From fijall at gmail.com  Thu Aug 18 10:10:00 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 18 Aug 2011 10:10:00 +0200
Subject: [pypy-dev] python 3
In-Reply-To: <CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
	<BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>
	<CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>
Message-ID: <CAK5idxRzeWdpiq3xLPH+NvFmHE=9+Axf7nMKgwf9y-jJQmjtoA@mail.gmail.com>

On Thu, Aug 18, 2011 at 2:13 AM, Eli Stevens (Gmail)
<wickedgrey at gmail.com> wrote:
> On Wed, Aug 17, 2011 at 4:01 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> +1 to the question. ?Why can't it be that way?
>
> If by "that way" you mean "leave python 2.x behind post 1.6" I'd like
> to note that IMO pypy has been under-acknowledged by the wider python
> community for a very long time. ?That's finally starting to change
> (pypy production releases, cpython devs devoting resources to make
> alternate implementations not second-class citizens, etc.), but by
> abandoning the segment of the language with the largest userbase, the
> project would go back to niche status again. ?Yeah, doing so might
> position pypy well to become the default python 3 implementation, but
> I find it hard to imagine that tacking on another N years until pypy
> is a significant percentage of python deployments is going to be good
> for the project.
>
> My $0.02,
> Eli

Just to answer some questions:

There is no way we're leaving python 2 support for the forseeable
future. It's all constructed precisely for the reason so JIT
improvements will benefit every interpreter written in RPython, not
just a specific one. In either of scenarios presented above, new
releases will include both py 3.x and 2.7 as release targets.

Cheers,
fijal

From arigo at tunes.org  Thu Aug 18 10:09:51 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 18 Aug 2011 10:09:51 +0200
Subject: [pypy-dev] python 3
In-Reply-To: <CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
Message-ID: <CAMSv6X3DGPbU=Q7uYd0VGYfmBycfCG0CprhwCc05-MPOmyfXdw@mail.gmail.com>

Hi Miquel,

On Wed, Aug 17, 2011 at 8:30 PM, Miquel Torres <tobami at googlemail.com> wrote:
>> This would remain as a branch for the foreseeable future though,
>> because we still need a Python 2 interpreter, if only to run our own
>> translation toolchain on (and not suffer the 2.5x slow-down of running
>> it on CPython 2.x).
> I don't quite follow. Switching to Python 3 (I am not saying that
> would be a good idea, just clarifying) and release as, let's say, PyPy
> 2.0, doesn't mean that the impending 1.6 release would go away. Python
> 3 users would use PyPy 2.0+, while users of Python 2.7 would use PyPy
> 1.6. You would still be able to compile PyPy and its Python 2.7
> toolchain with PyPy 1.6, thus getting the 2.5x speed up.

Yes, I think we are basically saying the same thing.  Internally we
need two branches and continuous merges, for everything done in the
translation toolchain part (and the JIT).  It doesn't really matter
which branch is called "a branch" or "the trunk" :-)  We can discuss
when the time comes how to give version numbers to the result.  Should
it be PyPy 1.6.1, 1.6.2, 1.6.3 and in parallel PyPy 2.0.1, 2.0.2,
2.0.3?...  The point I made originally was that I don't think we can
or should, for the foreseeable future, stop releasing newer versions
of PyPy supporting 2.7.


A bient?t,

Armin.

From tobami at googlemail.com  Thu Aug 18 10:32:50 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Thu, 18 Aug 2011 10:32:50 +0200
Subject: [pypy-dev] python 3
In-Reply-To: <CAK5idxRzeWdpiq3xLPH+NvFmHE=9+Axf7nMKgwf9y-jJQmjtoA@mail.gmail.com>
References: <CAF_QGARE6pDx2MFuqZM0nYQHL54+YtoOZoJQsno1rqAVR=04fQ@mail.gmail.com>
	<CAPZV6o_9gDNc9ZrWLggPPQ1-3H0x+WSyFJvDQEaUEr+MPSmYWA@mail.gmail.com>
	<B96972DD-7D20-4CBE-9962-B9EA411EA7B3@gmail.com>
	<CAPZV6o9dq4g+RFND5mDYfswrs83am+pT_jsadJLCYa+4ZibyMg@mail.gmail.com>
	<8AFCE0C3-A2D9-41D7-99B9-0BBFBE73DC75@gmail.com>
	<CAPZV6o8Sm=20Q99u5dbpTxTHN5ngLYNVxRV4c3KzvW65jMRqcg@mail.gmail.com>
	<8F80CD28-3DEF-4573-8A4C-0D36E2912DE6@gmail.com>
	<20110817130441.13284.1210301337.divmod.xquotient.152@localhost.localdomain>
	<CAKBMpNbsQFHU19_C212YP-mT3vG-84eAHnACWKiyoWS+0M6Dig@mail.gmail.com>
	<CAGf+9VyQNWJLwj8kw8UCq=oZYSCCMzyCVWpy2yKnVvcet+ToMQ@mail.gmail.com>
	<BBE1204A-3737-405F-B495-111E8B9008E9@gmail.com>
	<CADa34LC45=tozbqu-Ed-JsFPBgMXathMJ_tj5K=GG0tm3Em0Qg@mail.gmail.com>
	<CAK5idxRzeWdpiq3xLPH+NvFmHE=9+Axf7nMKgwf9y-jJQmjtoA@mail.gmail.com>
Message-ID: <CAGf+9VwiPgzzoYcUqE+Q83oVXiEoHLOpcFXB3Yo7mPCG-Nx+Bw@mail.gmail.com>

so the situation is even better: even if the PyPy Python 2.x
implementation was not updated any more after 1.6 (which won't be the
case), future versions of PyPy could have *both* a 2.x and a 3.x
interpreter (separately packaged), and *both* would leverage the newer
JIT versions. Is that correct Maciej?

In that case, the question for PyPy as a project would be when it
makes sense to invest the time to implement a 3.x interpreter if no
one steps up and just does it ;-)

Miquel


2011/8/18 Maciej Fijalkowski <fijall at gmail.com>:
> On Thu, Aug 18, 2011 at 2:13 AM, Eli Stevens (Gmail)
> <wickedgrey at gmail.com> wrote:
>> On Wed, Aug 17, 2011 at 4:01 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>>> +1 to the question. ?Why can't it be that way?
>>
>> If by "that way" you mean "leave python 2.x behind post 1.6" I'd like
>> to note that IMO pypy has been under-acknowledged by the wider python
>> community for a very long time. ?That's finally starting to change
>> (pypy production releases, cpython devs devoting resources to make
>> alternate implementations not second-class citizens, etc.), but by
>> abandoning the segment of the language with the largest userbase, the
>> project would go back to niche status again. ?Yeah, doing so might
>> position pypy well to become the default python 3 implementation, but
>> I find it hard to imagine that tacking on another N years until pypy
>> is a significant percentage of python deployments is going to be good
>> for the project.
>>
>> My $0.02,
>> Eli
>
> Just to answer some questions:
>
> There is no way we're leaving python 2 support for the forseeable
> future. It's all constructed precisely for the reason so JIT
> improvements will benefit every interpreter written in RPython, not
> just a specific one. In either of scenarios presented above, new
> releases will include both py 3.x and 2.7 as release targets.
>
> Cheers,
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Thu Aug 18 14:45:16 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 18 Aug 2011 14:45:16 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAKp1Nw7H1=M4fQQSRxGG+f2rWSP0GGGLFu06Y_LcEx4GYozs9w@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<201108152103.41416.naylor.b.david@gmail.com>
	<CAKp1Nw7vBZPoj8kMbBut=1jjvoYSAk+_wik2HWy6HQd6A+_nWA@mail.gmail.com>
	<201108160628.57358.naylor.b.david@gmail.com>
	<CAKp1Nw4-4Uf7YrgHTBvJ4fXcL=osq7cZWEBULwAddEGFsqaLBQ@mail.gmail.com>
	<CAMQxofkLMVXwc2D6vQoj97=zdUiqv03nNGGPOKfD_7rBRfD+QA@mail.gmail.com>
	<CAKp1Nw5jCJuCHigCsG_U5kL9uFVz4tWvL2dCkpS904dY2Pn=fQ@mail.gmail.com>
	<CAKp1Nw7H1=M4fQQSRxGG+f2rWSP0GGGLFu06Y_LcEx4GYozs9w@mail.gmail.com>
Message-ID: <CAMSv6X0t6d0xYMrVM_68ZKnhdsFEp7CZTSOCRc-s0DVQ_j-rjw@mail.gmail.com>

Hi Gabriel,

On Wed, Aug 17, 2011 at 10:42 PM, Gabriel Lavoie <glavoie at gmail.com> wrote:
> Hum... It seems it's the end of my FreeBSD buildslave. :( I don't know when
> it time the memory requirement jumped to over 4GB (Armin told me it was now
> around 4.5GB for 64bits).

Actually I found out that with proper settings I can run a 64-bit
"translate.py -Ojit" in 3.0GB.  I didn't measure the impact on time
--- I expect it to take "a bit longer" to run, but of course much less
than if it's swapping.  Maybe we need to tweak a bit our default
parameters...

PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ./translate.py -Ojit

I think that tweaks to this environment variable have a direct
predictable effect: e.g. with PYPY_GC_MAX_DELTA=400MB it would take
3.2GB (an extra 200MB), but do half as many major collections,
decreasing the total time.

For reference, running "python2.7 ./translate.py -Ojit" takes 3.4GB.
So it means that *with some tweaks* we run not only faster, but
consuming less memory than CPython, actually.


A bient?t,

Armin.

From dje.gcc at gmail.com  Thu Aug 18 16:19:56 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 18 Aug 2011 10:19:56 -0400
Subject: [pypy-dev] PPC JIT test_load_and_store()
Message-ID: <CAGWvnymj=fr-ULWtSxzF2Q6_-Wyp6DW0T-HgsaDBdLPOxprXZQ@mail.gmail.com>

Sven,

Does test_ppc.py:test_load_and_store() succeed for you?  The code
stores the values in registers r10 and r11 to the memory addressed by
registers r8 and r9, but r8 and r9 never are initialized, so it stores
to an arbitrary location in memory that may not be valid and causes a
segfault.

I understand the intent of the test, but I don't understand how it is
suppose to work as currently written.

Thanks, David

From dje.gcc at gmail.com  Thu Aug 18 16:50:43 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 18 Aug 2011 10:50:43 -0400
Subject: [pypy-dev] PPC JIT test_load_and_store()
In-Reply-To: <CAGWvnymj=fr-ULWtSxzF2Q6_-Wyp6DW0T-HgsaDBdLPOxprXZQ@mail.gmail.com>
References: <CAGWvnymj=fr-ULWtSxzF2Q6_-Wyp6DW0T-HgsaDBdLPOxprXZQ@mail.gmail.com>
Message-ID: <CAGWvnykQ0BCFv5iEP0Wh5JnJFS8LB7m1Dk8GvyYXFywYk0Nwaw@mail.gmail.com>

To be more concrete, the current test code is

    def test_load_and_store(self):
        a = PPCBuilder()
        word1 = 1000
        word2 = 2000
        a.load_word(10, word1)   # load constant 1000 into r10
        a.load_word(11, word2)   # load constant 2000 into r11
        a.stw(10, 8, 0)                # store r10 into memory at address in r8
        a.stw(11, 9, 0)                # store r11 into memory at address in r9
        a.lwz(4, 8, 0)                  # load r4 from memory at address in r8
        a.lwz(5, 9, 0)                  # load r5 from memory at address in r9
        a.add(3, 4, 5)                 # r3 = r4 + r5
        a.blr()
        f = a.assemble()
        assert f() == word1 + word2

r8 and r9 are not initialized and point to whatever locations those
registers happen to hold. In libffi for PPC, r8 happens to hold the
arg pointer, but there are no arguments and r9 is arbitrary.

In both PPC32 and PPC64 libffi, r7 points to the function address, so
a version that overwrites the beginning of the code produced by the
JIT and already executed is

        a.stw(10, 7, 0)
        a.stw(11, 7, 4)
        a.lwz(4, 7, 0)
        a.lwz(5, 7, 4)

Thanks, David

From dje.gcc at gmail.com  Thu Aug 18 17:25:50 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 18 Aug 2011 11:25:50 -0400
Subject: [pypy-dev] Update import path of PPC JIT tests
Message-ID: <CAGWvnymzt7msMUM9zf_hC-jRejL=vXgGH4f05hidEJ+C3YcQMA@mail.gmail.com>

Many of the files in ppc/ppcgen/test (and
ppc/ppcgen/rassemblermaker.py) reference the old path of
pypy.jit.codegen.XXX.  Updating the paths to pypy.jit.backend.XXX
allows most of the tests to run and pass.

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff6
Type: application/octet-stream
Size: 4415 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/6bd2fd94/attachment.obj>

From arigo at tunes.org  Thu Aug 18 17:28:26 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 18 Aug 2011 17:28:26 +0200
Subject: [pypy-dev] PPC JIT test_load_and_store()
In-Reply-To: <CAGWvnykQ0BCFv5iEP0Wh5JnJFS8LB7m1Dk8GvyYXFywYk0Nwaw@mail.gmail.com>
References: <CAGWvnymj=fr-ULWtSxzF2Q6_-Wyp6DW0T-HgsaDBdLPOxprXZQ@mail.gmail.com>
	<CAGWvnykQ0BCFv5iEP0Wh5JnJFS8LB7m1Dk8GvyYXFywYk0Nwaw@mail.gmail.com>
Message-ID: <CAMSv6X14RYrLLgpQwHv--dVdsJwQ3fHyZvDa1LvKaKba+S3MfQ@mail.gmail.com>

Hi David,

On Thu, Aug 18, 2011 at 4:50 PM, David Edelsohn <dje.gcc at gmail.com> wrote:
> r8 and r9 are not initialized and point to whatever locations those
> registers happen to hold.

I think that the test is just wrong...  You need to load explicitly
the address of some CArray in r8 and r9, for example with:

p = lltype.malloc(rffi.CArray(lltype.Signed), 2)
a.load_word(8, rffi.cast(lltype.Signed, p))
a.load_word(9, rffi.cast(lltype.Signed, p) + (4-or-8))


A bient?t,

Armin.

From Sven.Hager at uni-duesseldorf.de  Thu Aug 18 18:43:49 2011
From: Sven.Hager at uni-duesseldorf.de (Sven Hager)
Date: Thu, 18 Aug 2011 18:43:49 +0200
Subject: [pypy-dev] PPC JIT test_load_and_store()
In-Reply-To: <CAGWvnykQ0BCFv5iEP0Wh5JnJFS8LB7m1Dk8GvyYXFywYk0Nwaw@mail.gmail.com>
References: <CAGWvnymj=fr-ULWtSxzF2Q6_-Wyp6DW0T-HgsaDBdLPOxprXZQ@mail.gmail.com>
	<CAGWvnykQ0BCFv5iEP0Wh5JnJFS8LB7m1Dk8GvyYXFywYk0Nwaw@mail.gmail.com>
Message-ID: <4E4D4145.7020306@uni-duesseldorf.de>

On 08/18/2011 04:50 PM, David Edelsohn wrote:
> To be more concrete, the current test code is
>
>      def test_load_and_store(self):
>          a = PPCBuilder()
>          word1 = 1000
>          word2 = 2000
>          a.load_word(10, word1)   # load constant 1000 into r10
>          a.load_word(11, word2)   # load constant 2000 into r11
>          a.stw(10, 8, 0)                # store r10 into memory at address in r8
>          a.stw(11, 9, 0)                # store r11 into memory at address in r9
>          a.lwz(4, 8, 0)                  # load r4 from memory at address in r8
>          a.lwz(5, 9, 0)                  # load r5 from memory at address in r9
>          a.add(3, 4, 5)                 # r3 = r4 + r5
>          a.blr()
>          f = a.assemble()
>          assert f() == word1 + word2
>
> r8 and r9 are not initialized and point to whatever locations those
> registers happen to hold. In libffi for PPC, r8 happens to hold the
> arg pointer, but there are no arguments and r9 is arbitrary.
>
> In both PPC32 and PPC64 libffi, r7 points to the function address, so
> a version that overwrites the beginning of the code produced by the
> JIT and already executed is
>
>          a.stw(10, 7, 0)
>          a.stw(11, 7, 4)
>          a.lwz(4, 7, 0)
>          a.lwz(5, 7, 4)
>
> Thanks, David
Hi David,

thank you for telling me about the broken test.
I fixed the problem, as Armin recommended, and also added your
pathh no. 5 to the code.

Tomorrow I will fix the broken import paths.

Best regards,
Sven

From fijall at gmail.com  Thu Aug 18 19:30:44 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 18 Aug 2011 19:30:44 +0200
Subject: [pypy-dev] PyPy 1.6 released
Message-ID: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>

========================
PyPy 1.6 - kickass panda
========================

We're pleased to announce the 1.6 release of PyPy. This release brings a lot
of bugfixes and performance improvements over 1.5, and improves support for
Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
has beta level support for loading CPython C extensions.  You can download it
here:

    http://pypy.org/download.html

What is PyPy?
=============

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison)
due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64 or Mac OS X.  Windows 32
is beta (it roughly works but a lot of small issues have not been fixed so
far).  Windows 64 is not yet supported.

The main topics of this release are speed and stability: on average on
our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy 1.5,
which was already much faster than CPython on our set of benchmarks.

The speed improvements have been made possible by optimizing many of the
layers which compose PyPy.  In particular, we improved: the Garbage Collector,
the JIT warmup time, the optimizations performed by the JIT, the quality of
the generated machine code and the implementation of our Python interpreter.

.. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org


Highlights
==========

* Numerous performance improvements, overall giving considerable speedups:

  - better GC behavior when dealing with very large objects and arrays

  - **fast ctypes:** now calls to ctypes functions are seen and optimized
    by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times
    faster than CPython

  - improved generators(1): simple generators now are inlined into the caller
    loop, making performance up to 3.5 times faster than PyPy 1.5.

  - improved generators(2): thanks to other optimizations, even generators
    that are not inlined are between 10% and 20% faster than PyPy 1.5.

  - faster warmup time for the JIT

  - JIT support for single floats (e.g., for ``array('f')``)

  - optimized dictionaries: the internal representation of dictionaries is now
    dynamically selected depending on the type of stored objects, resulting in
    faster code and smaller memory footprint.  For example, dictionaries whose
    keys are all strings, or all integers. Other dictionaries are also smaller
    due to bugfixes.

* JitViewer: this is the first official release which includes the JitViewer,
  a web-based tool which helps you to see which parts of your Python code have
  been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1 has
  already been release and works well with PyPy 1.6.

* The CPython extension module API has been improved and now supports many
  more extensions. For information on which one are supported, please refer to
  our `compatibility wiki`_.

* Multibyte encoding support: this was of of the last areas in which we were
  still behind CPython, but now we fully support them.

* Preliminary support for NumPy: this release includes a preview of a very
  fast NumPy module integrated with the PyPy JIT.  Unfortunately, this does
  not mean that you can expect to take an existing NumPy program and run it on
  PyPy, because the module is still unfinished and supports only some of the
  numpy API. However, barring some details, what works should be
  blazingly fast :-)

* Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_, not
  counting the numerous bugs that were found and reported through other
  channels than the bug tracker.

Cheers,

Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
Armin Rigo and the PyPy team

.. _`jitviewer`:
http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
.. _`bug tracker`: https://bugs.pypy.org
.. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home

From drsalists at gmail.com  Thu Aug 18 20:16:10 2011
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 18 Aug 2011 11:16:10 -0700
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
Message-ID: <CAGGBd_rU11pmW9u=tobFKobP=Bter0qFPeMECdh6iD6aXTgNyQ@mail.gmail.com>

Sweet.  ^_^  I see the permissions bits look better now too.

I miss os.stat().st_rdev though - was it removed for a reason?

On Thu, Aug 18, 2011 at 10:30 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> ========================
> PyPy 1.6 - kickass panda
> ========================
>
> We're pleased to announce the 1.6 release of PyPy. This release brings a
> lot
> of bugfixes and performance improvements over 1.5, and improves support for
> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1
> and
> has beta level support for loading CPython C extensions.  You can download
> it
> here:
>
>    http://pypy.org/download.html
>
> What is PyPy?
> =============
>
> PyPy is a very compliant Python interpreter, almost a drop-in replacement
> for
> CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance
> comparison)
> due to its integrated tracing JIT compiler.
>
> This release supports x86 machines running Linux 32/64 or Mac OS X.
>  Windows 32
> is beta (it roughly works but a lot of small issues have not been fixed so
> far).  Windows 64 is not yet supported.
>
> The main topics of this release are speed and stability: on average on
> our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy
> 1.5,
> which was already much faster than CPython on our set of benchmarks.
>
> The speed improvements have been made possible by optimizing many of the
> layers which compose PyPy.  In particular, we improved: the Garbage
> Collector,
> the JIT warmup time, the optimizations performed by the JIT, the quality of
> the generated machine code and the implementation of our Python
> interpreter.
>
> .. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org
>
>
> Highlights
> ==========
>
> * Numerous performance improvements, overall giving considerable speedups:
>
>  - better GC behavior when dealing with very large objects and arrays
>
>  - **fast ctypes:** now calls to ctypes functions are seen and optimized
>    by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10
> times
>    faster than CPython
>
>  - improved generators(1): simple generators now are inlined into the
> caller
>    loop, making performance up to 3.5 times faster than PyPy 1.5.
>
>  - improved generators(2): thanks to other optimizations, even generators
>    that are not inlined are between 10% and 20% faster than PyPy 1.5.
>
>  - faster warmup time for the JIT
>
>  - JIT support for single floats (e.g., for ``array('f')``)
>
>  - optimized dictionaries: the internal representation of dictionaries is
> now
>    dynamically selected depending on the type of stored objects, resulting
> in
>    faster code and smaller memory footprint.  For example, dictionaries
> whose
>    keys are all strings, or all integers. Other dictionaries are also
> smaller
>    due to bugfixes.
>
> * JitViewer: this is the first official release which includes the
> JitViewer,
>  a web-based tool which helps you to see which parts of your Python code
> have
>  been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1
> has
>  already been release and works well with PyPy 1.6.
>
> * The CPython extension module API has been improved and now supports many
>  more extensions. For information on which one are supported, please refer
> to
>  our `compatibility wiki`_.
>
> * Multibyte encoding support: this was of of the last areas in which we
> were
>  still behind CPython, but now we fully support them.
>
> * Preliminary support for NumPy: this release includes a preview of a very
>  fast NumPy module integrated with the PyPy JIT.  Unfortunately, this does
>  not mean that you can expect to take an existing NumPy program and run it
> on
>  PyPy, because the module is still unfinished and supports only some of the
>  numpy API. However, barring some details, what works should be
>  blazingly fast :-)
>
> * Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_,
> not
>  counting the numerous bugs that were found and reported through other
>  channels than the bug tracker.
>
> Cheers,
>
> Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
> Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
> Armin Rigo and the PyPy team
>
> .. _`jitviewer`:
> http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
> .. _`bug tracker`: https://bugs.pypy.org
> .. _`compatibility wiki`:
> https://bitbucket.org/pypy/compatibility/wiki/Home
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/2ca8e6c5/attachment.html>

From dje.gcc at gmail.com  Thu Aug 18 21:51:17 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Thu, 18 Aug 2011 15:51:17 -0400
Subject: [pypy-dev] PPC64 JIT test_ppc.py tests fixed and working
Message-ID: <CAGWvnynohF8hGt0-2HsrcTWDiadt0Lq0bZe7dYj7AsyZyoMSVg@mail.gmail.com>

Sven,

With the attached patch, all of the ppc/ppcgen/test/test_ppc.py tests
pass except for test_call_function that I skip while I debug it.

I updated ppc_assembler.py:load_from() to load a full 64 bit value on
64 bit systems.  It is not exactly right because I need to fix the
signed offset compensation code, but it allows the test to pass.

The multiply tests were failing because the tests perform a signed
comparison and the result of PPC 32 bit multiply instructions are not
implicitly sign extended.  I sign extend the results on PPC64 and they
now pass.

test_load_and_store needs to use load_dword on PPC64 to load the
addresses into r8 and r9.

test_ld needs to allocate a 64 bit long, not a 32 bit int, and the
address needs to be loaded with load_dword.

Thanks, David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppcjit-diff7
Type: application/octet-stream
Size: 3915 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/d2215d85/attachment-0001.obj>

From jfcgauss at gmail.com  Thu Aug 18 22:26:22 2011
From: jfcgauss at gmail.com (Serhat Sevki Dincer)
Date: Thu, 18 Aug 2011 23:26:22 +0300
Subject: [pypy-dev] PyPy 1.6 released
Message-ID: <CAPqC6xSZpTCQmMHW+EzcsopBMYmH=zWHzL7b76mmwNXX0cX-sw@mail.gmail.com>

Great news! Thanks..

> From:?Maciej Fijalkowski <fijall at gmail.com>
> To:?PyPy Developer Mailing List <pypy-dev at python.org>
> Date:?Thu, 18 Aug 2011 19:30:44 +0200
> Subject:?[pypy-dev] PyPy 1.6 released
> ========================
> PyPy 1.6 - kickass panda
> ========================
>
> We're pleased to announce the 1.6 release of PyPy. This release brings a lot
> of bugfixes and performance improvements over 1.5, and improves support for
> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
> has beta level support for loading CPython C extensions. ?You can download it

From jake.biesinger at gmail.com  Thu Aug 18 23:35:12 2011
From: jake.biesinger at gmail.com (Jacob Biesinger)
Date: Thu, 18 Aug 2011 14:35:12 -0700
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
Message-ID: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>

Hi all,

New to the list and fairly new to pypy.  First of all, congrats on the new
1.6 release-- the growing support for numpy is very exciting (go, fight,
win, take state!).

So I snagged the 1.6 release to test if it would be faster on the kind of
code I often write: Bioinformatics.  In this snippet, the point is to check
the mappability of the genome-- if a particular substring appears more than
once in the genome, the region is called unmappable.

Machine Specs:
64bit Ubuntu 11.04
119048 CPython 2.7.1 pystones/second
416667 pypy1.6 pystones/second

The CPython version of the following code takes a bit more than a minute to
run on the 21st chromosome of the human reference genome, but the pypy
version has been going for 27+ minutes and hasn't yet finished the first
step of loading the genome as a dict of strings.

Am I using some construct that's particularly difficult for pypy?  Am I
missing something?

hg18 chrom 21 is available at
http://hgdownload.cse.ucsc.edu/goldenPath/hg18/chromosomes/chr21.fa.gz


import sys

def slide_dna(dna, windowsize):
    for i in xrange(len(dna) - windowsize):
        slice = dna[i:i+windowsize]
        if 'N' not in slice:
            yield slice

fasta_file = sys.argv[1]  # should be *.fa
print 'loading dna from', fasta_file
chroms = {}
dna = None
for l in open(fasta_file):
    if l.startswith('>'):  # new chromosome
        if dna is not None:
            chroms[chrom] = dna
        chrom = l.strip().replace('>', '')
        dna = ''
    else:
        dna += l.rstrip()
if dna is not None:
    chroms[chrom] = dna

for length in [15]:#, 25, 35, 45, 55, 65, 75]:
    print 'now on', length
    mappable = 0
    repeat = 0
    s = {}
    for dna in chroms.itervalues():
        for slice in slide_dna(dna, length):
            try:
                s[slice] += 1
            except KeyError:
                s[slice] = 1
    print 'built, now counting'
    for dna in chroms.itervalues():
        for slice in slide_dna(dna, length):
            if s[slice] == 1:
                mappable += 1
            else:
                repeat += 1
    print 'for substring length %s, mappable: %s, repeat: %s' % (length,
mappable, repeat)


--
Jake Biesinger
Graduate Student
Xie Lab, UC Irvine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/32fd96a6/attachment.html>

From jake.biesinger at gmail.com  Thu Aug 18 23:48:51 2011
From: jake.biesinger at gmail.com (Jacob Biesinger)
Date: Thu, 18 Aug 2011 14:48:51 -0700
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
Message-ID: <CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>

A better breakdown...

fasta_file = sys.argv[1]  # should be *.fa
> print 'loading dna from', fasta_file
> chroms = {}
> dna = None
> for l in open(fasta_file):
>     if l.startswith('>'):  # new chromosome
>         if dna is not None:
>             chroms[chrom] = dna
>         chrom = l.strip().replace('>', '')
>          dna = ''
>     else:
>         dna += l.rstrip()
> if dna is not None:
>     chroms[chrom] = dna
>

CPython takes about 1.5 seconds to get to this point in the code, while pypy
hasn't passed this point in 30 minutes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/51de3347/attachment.html>

From vincent.legoll at gmail.com  Fri Aug 19 00:24:53 2011
From: vincent.legoll at gmail.com (Vincent Legoll)
Date: Fri, 19 Aug 2011 00:24:53 +0200
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
Message-ID: <CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>

Hello,

Try this:

import sys

fasta_file = sys.argv[1]  # should be *.fa
print 'loading dna from', fasta_file
chroms = {}
dna = []
for l in open(fasta_file):
    if l.startswith('>'):  # new chromosome
        if len(dna) > 0:
            chroms[chrom] = ''.join(dna)
        chrom = l.strip().replace('>', '')
        dna = []
    else:
        dna.append(l.rstrip())
if len(dna) > 0:
    chroms[chrom] = ''.join(dna)

-- 
Vincent Legoll

From peelpy at gmail.com  Fri Aug 19 00:31:57 2011
From: peelpy at gmail.com (Justin Peel)
Date: Thu, 18 Aug 2011 16:31:57 -0600
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
Message-ID: <CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>

Yes, Vincent's way is the better way to go. To elaborate more on the
problem, string appending is O(N^2) while appending to a list and then
joining is an O(N) operation. Why CPython is faster than Pypy at doing
the less efficient way is something that I'm not fully sure about, but
I believe that it might have to do with the differing memory
management strategies.

On Thu, Aug 18, 2011 at 4:24 PM, Vincent Legoll
<vincent.legoll at gmail.com> wrote:
> Hello,
>
> Try this:
>
> import sys
>
> fasta_file = sys.argv[1] ?# should be *.fa
> print 'loading dna from', fasta_file
> chroms = {}
> dna = []
> for l in open(fasta_file):
> ? ?if l.startswith('>'): ?# new chromosome
> ? ? ? ?if len(dna) > 0:
> ? ? ? ? ? ?chroms[chrom] = ''.join(dna)
> ? ? ? ?chrom = l.strip().replace('>', '')
> ? ? ? ?dna = []
> ? ?else:
> ? ? ? ?dna.append(l.rstrip())
> if len(dna) > 0:
> ? ?chroms[chrom] = ''.join(dna)
>
> --
> Vincent Legoll
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From aaron.devore at gmail.com  Fri Aug 19 00:50:00 2011
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Thu, 18 Aug 2011 15:50:00 -0700
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
Message-ID: <CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>

Python 2.4 introduced a change that helps improve performance of
string concatenation, according to its release notes. I don't know
anything beyond that.

-Aaron DeVore

On Thu, Aug 18, 2011 at 3:31 PM, Justin Peel <peelpy at gmail.com> wrote:
> Yes, Vincent's way is the better way to go. To elaborate more on the
> problem, string appending is O(N^2) while appending to a list and then
> joining is an O(N) operation. Why CPython is faster than Pypy at doing
> the less efficient way is something that I'm not fully sure about, but
> I believe that it might have to do with the differing memory
> management strategies.
>
> On Thu, Aug 18, 2011 at 4:24 PM, Vincent Legoll
> <vincent.legoll at gmail.com> wrote:
>> Hello,
>>
>> Try this:
>>
>> import sys
>>
>> fasta_file = sys.argv[1] ?# should be *.fa
>> print 'loading dna from', fasta_file
>> chroms = {}
>> dna = []
>> for l in open(fasta_file):
>> ? ?if l.startswith('>'): ?# new chromosome
>> ? ? ? ?if len(dna) > 0:
>> ? ? ? ? ? ?chroms[chrom] = ''.join(dna)
>> ? ? ? ?chrom = l.strip().replace('>', '')
>> ? ? ? ?dna = []
>> ? ?else:
>> ? ? ? ?dna.append(l.rstrip())
>> if len(dna) > 0:
>> ? ?chroms[chrom] = ''.join(dna)
>>
>> --
>> Vincent Legoll
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From peelpy at gmail.com  Fri Aug 19 01:01:52 2011
From: peelpy at gmail.com (Justin Peel)
Date: Thu, 18 Aug 2011 17:01:52 -0600
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
Message-ID: <CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>

Yes, I just looked at it. For cases like this where there is
effectively only one reference to the string being appended to, it
just resizes the string in-place and copies in the string being
appended which gives it O(N) performance. It is a hack that is
available only because of the reference counting that CPython employs
for memory management.

For reference, the hack is in Python/ceval.c in the string_concatenate function.

On Thu, Aug 18, 2011 at 4:50 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> Python 2.4 introduced a change that helps improve performance of
> string concatenation, according to its release notes. I don't know
> anything beyond that.
>
> -Aaron DeVore
>
> On Thu, Aug 18, 2011 at 3:31 PM, Justin Peel <peelpy at gmail.com> wrote:
>> Yes, Vincent's way is the better way to go. To elaborate more on the
>> problem, string appending is O(N^2) while appending to a list and then
>> joining is an O(N) operation. Why CPython is faster than Pypy at doing
>> the less efficient way is something that I'm not fully sure about, but
>> I believe that it might have to do with the differing memory
>> management strategies.
>>
>> On Thu, Aug 18, 2011 at 4:24 PM, Vincent Legoll
>> <vincent.legoll at gmail.com> wrote:
>>> Hello,
>>>
>>> Try this:
>>>
>>> import sys
>>>
>>> fasta_file = sys.argv[1] ?# should be *.fa
>>> print 'loading dna from', fasta_file
>>> chroms = {}
>>> dna = []
>>> for l in open(fasta_file):
>>> ? ?if l.startswith('>'): ?# new chromosome
>>> ? ? ? ?if len(dna) > 0:
>>> ? ? ? ? ? ?chroms[chrom] = ''.join(dna)
>>> ? ? ? ?chrom = l.strip().replace('>', '')
>>> ? ? ? ?dna = []
>>> ? ?else:
>>> ? ? ? ?dna.append(l.rstrip())
>>> if len(dna) > 0:
>>> ? ?chroms[chrom] = ''.join(dna)
>>>
>>> --
>>> Vincent Legoll
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>

From jake.biesinger at gmail.com  Fri Aug 19 02:10:29 2011
From: jake.biesinger at gmail.com (Jacob Biesinger)
Date: Thu, 18 Aug 2011 17:10:29 -0700
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
Message-ID: <CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>

Wow thanks for the quick response.  The performance is *much, much* better
with the suggested list-join.  CPython still beats Pypy, but only by a
narrow margin:

pypy1.6:             1m33.142s
CPython 2.7.1:       1m12.092s

Thanks for the advice-- I had forgotten about string immutability and its
associated costs.  And keep up the good work on pypy!  I look forward to the
day I can replace CPython with pypy in more interesting scientific workflows
</end plug for scipy integration>

A bit OT:  The recent release of ipython added some powerful multiprocessing
features using ZeroMQ.  I've only glanced at pypy's extensive threading
optimizations (e.g., greenlets).  Does pypy jit across thread/process
boundaries?
--
Jake Biesinger
Graduate Student
Xie Lab, UC Irvine


On Thu, Aug 18, 2011 at 4:01 PM, Justin Peel <peelpy at gmail.com> wrote:

> Yes, I just looked at it. For cases like this where there is
> effectively only one reference to the string being appended to, it
> just resizes the string in-place and copies in the string being
> appended which gives it O(N) performance. It is a hack that is
> available only because of the reference counting that CPython employs
> for memory management.
>
> For reference, the hack is in Python/ceval.c in the string_concatenate
> function.
>
> On Thu, Aug 18, 2011 at 4:50 PM, Aaron DeVore <aaron.devore at gmail.com>
> wrote:
> > Python 2.4 introduced a change that helps improve performance of
> > string concatenation, according to its release notes. I don't know
> > anything beyond that.
> >
> > -Aaron DeVore
> >
> > On Thu, Aug 18, 2011 at 3:31 PM, Justin Peel <peelpy at gmail.com> wrote:
> >> Yes, Vincent's way is the better way to go. To elaborate more on the
> >> problem, string appending is O(N^2) while appending to a list and then
> >> joining is an O(N) operation. Why CPython is faster than Pypy at doing
> >> the less efficient way is something that I'm not fully sure about, but
> >> I believe that it might have to do with the differing memory
> >> management strategies.
> >>
> >> On Thu, Aug 18, 2011 at 4:24 PM, Vincent Legoll
> >> <vincent.legoll at gmail.com> wrote:
> >>> Hello,
> >>>
> >>> Try this:
> >>>
> >>> import sys
> >>>
> >>> fasta_file = sys.argv[1]  # should be *.fa
> >>> print 'loading dna from', fasta_file
> >>> chroms = {}
> >>> dna = []
> >>> for l in open(fasta_file):
> >>>    if l.startswith('>'):  # new chromosome
> >>>        if len(dna) > 0:
> >>>            chroms[chrom] = ''.join(dna)
> >>>        chrom = l.strip().replace('>', '')
> >>>        dna = []
> >>>    else:
> >>>        dna.append(l.rstrip())
> >>> if len(dna) > 0:
> >>>    chroms[chrom] = ''.join(dna)
> >>>
> >>> --
> >>> Vincent Legoll
> >>> _______________________________________________
> >>> pypy-dev mailing list
> >>> pypy-dev at python.org
> >>> http://mail.python.org/mailman/listinfo/pypy-dev
> >>>
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110818/5895807c/attachment.html>

From arigo at tunes.org  Fri Aug 19 11:21:57 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 19 Aug 2011 11:21:57 +0200
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
Message-ID: <CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>

Hi Jacob,

On Fri, Aug 19, 2011 at 2:10 AM, Jacob Biesinger
<jake.biesinger at gmail.com> wrote:
> A bit OT: ?The recent release of ipython added some powerful multiprocessing
> features using ZeroMQ. ?I've only glanced at pypy's extensive threading
> optimizations (e.g., greenlets). ?Does pypy jit across thread/process
> boundaries?

Greenlets are unrelated to multithreading or multiprocessing.  They
are related to Stackless Python.

So far PyPy cannot even *have* a JIT together with Stackless Python
features like greenlets, but this is going to change soon; in this
next model, PyPy's JIT will not work across greenlets, but will
happily JIT every single greenlet individually.


A bient?t,

Armin.

From Sven.Hager at uni-duesseldorf.de  Fri Aug 19 11:39:24 2011
From: Sven.Hager at uni-duesseldorf.de (Sven Hager)
Date: Fri, 19 Aug 2011 11:39:24 +0200
Subject: [pypy-dev] PPC64 JIT test_ppc.py tests fixed and working
In-Reply-To: <CAGWvnynohF8hGt0-2HsrcTWDiadt0Lq0bZe7dYj7AsyZyoMSVg@mail.gmail.com>
References: <CAGWvnynohF8hGt0-2HsrcTWDiadt0Lq0bZe7dYj7AsyZyoMSVg@mail.gmail.com>
Message-ID: <4E4E2F4C.8000807@uni-duesseldorf.de>

On 08/18/2011 09:51 PM, David Edelsohn wrote:
> Sven,
>
> With the attached patch, all of the ppc/ppcgen/test/test_ppc.py tests
> pass except for test_call_function that I skip while I debug it.
>
> I updated ppc_assembler.py:load_from() to load a full 64 bit value on
> 64 bit systems.  It is not exactly right because I need to fix the
> signed offset compensation code, but it allows the test to pass.
>
> The multiply tests were failing because the tests perform a signed
> comparison and the result of PPC 32 bit multiply instructions are not
> implicitly sign extended.  I sign extend the results on PPC64 and they
> now pass.
>
> test_load_and_store needs to use load_dword on PPC64 to load the
> addresses into r8 and r9.
>
> test_ld needs to allocate a 64 bit long, not a 32 bit int, and the
> address needs to be loaded with load_dword.
>
> Thanks, David
Hello David,

your recent changes have been added.
Great to see so many tests in ppcgen/test pass. =)

Best regards,
Sven

From roberto at unbit.it  Fri Aug 19 12:54:25 2011
From: roberto at unbit.it (Roberto De Ioris)
Date: Fri, 19 Aug 2011 12:54:25 +0200
Subject: [pypy-dev] work on libpypy.so
In-Reply-To: <CAMSv6X0TUT7P9+O06nWJwOtkWzfneyut9uXv=8P1vuStM61mVg@mail.gmail.com>
References: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it>
	<CAMSv6X0TUT7P9+O06nWJwOtkWzfneyut9uXv=8P1vuStM61mVg@mail.gmail.com>
Message-ID: <4CDEBDC0-B709-4952-B489-E6534D3568BA@unbit.it>


Il giorno 25/lug/2011, alle ore 14:31, Armin Rigo ha scritto:

> 
> Indeed, there is nothing like that so far.  We need to think out the
> kind of C-level API which makes sense for libpypy.so to expose.  There
> are two options: either we tweak a little bit cpyext to expose the
> CPython C API, or we design some higher-performance PyPy-only C API...
> 
> For the latter case the first issue is how to store a reference to a
> PyPy object in C, given that it moves around --- maybe with an
> indirection, which would be a bit similar to what cpyext does but can
> be done with much better performance if we are free to design the API.
> 
> Feel free to come to our IRC channel to discuss things (#pypy on
> irc.freenode.net).
> 
> 

Sorry for the big late/silence but i spent last days working/hacking on cpyext.

I have managed to have a working uWSGI+pypy inverting the initial concept
(not very useful, but this is a base for benchmark) :

1) i build the whole uWSGI stack core (so, without cpython plugin) as a shared library
2) i created a uwsgi_pypy c extension:

http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi_pypy.c

This extension export a bunch of functions, with the only useful one being uwsgi_pypy.init()
This function calls the original main() of the uWSGI server, so effectively pypy lose the process control.

3) i have developed a very raw/minimal plugin for uWSGI that inspect a symbol in the process space.

http://projects.unbit.it/uwsgi/browser/plugins/symcall/symcall_plugin.c

This symbol is defined in the uwsgi_pypy.c module. So, this is the way uWSGI give control back to pypy at each request.

This is an example WSGI-handler implementation (incomplete, it misses post handling, but can already execute werkzeug test app)

http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi-pypy.py

Well, running it and benchmarking showed a 10% improvement in response time, so i tried with a more complex app 
(fibonacci's sequence and some prime number generator) and ?damn it destroyed cpython :) it served the double of the connections in half of the time?.

So, cpyext looks really in good shape, so (probably) the best way would be tweek it to have a libpypy.so.

i will work on the current setup a bit more to have a full WSGI environment, then i will start investigating the original idea (but this time
base on cpyext)

Thanks for the amazing work

--
Roberto De Ioris
http://unbit.it
JID: roberto at jabber.unbit.it


From davidf at sjsoft.com  Fri Aug 19 13:56:43 2011
From: davidf at sjsoft.com (David Fraser)
Date: Fri, 19 Aug 2011 06:56:43 -0500 (CDT)
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
Message-ID: <5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>

Hi

This is a fairly naive question from a newbie...

The pypy JIT takes a while to work out which parts of python code need optimization etc, and only after that phase do the speedups become relevant. Have there been any efforts (indeed, is it a feasible idea at all) that look at saving these optimizations for future runs of the same codebase? The advantages would be for:
 * Small programs that get run frequently - you could pre-tune the JIT but running a longer batch and then save the results and take advantage of them from startup
 * Application restarts not causing a slowdown (my case is a web application server - it would be nice if the first N page views weren't terribly slow as they are now)

Cheers
David

-- 
David Fraser
St James Software

From tloramus at gmail.com  Fri Aug 19 16:47:16 2011
From: tloramus at gmail.com (Miha Marolt)
Date: Fri, 19 Aug 2011 16:47:16 +0200
Subject: [pypy-dev] Broken link in Getting Started
Message-ID: <4E4E7774.2070104@gmail.com>

Hi,

in the Getting Started section of the documentation 
(http://doc.pypy.org/en/latest/getting-started.html), the command to 
download pip should be

$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py

In browser, the old link 
(https://github.com/pypa/pip/raw/master/contrib/get-pip.py 
<https://github.com/pypa/pip/raw/master/contrib/get-pip.py>) is 
redirected to the new one, but the curl downloads the page with the 
redirection note.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110819/f8f10d6a/attachment.html>

From caleb.hattingh at gmail.com  Fri Aug 19 14:48:36 2011
From: caleb.hattingh at gmail.com (Caleb Hattingh)
Date: Fri, 19 Aug 2011 14:48:36 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find ffi.h
Message-ID: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>

Hi

This is a really silly thing, but I am hoping someone who has done
this before can just give me the one-liner answer.

I am trying to build pypy 1.6 on Windows XP using MinGW (in an MSYS
environment).   The build gets started, and I see a bit of Mandelbrot,
but when the time comes for JIT, I get an error which I will reproduce
below.  I am following the too-terse instructions at the bottom of
this page:

http://codespeak.net/pypy/dist/pypy/doc/windows.html

I have made sure that my newly-built libffi-5.dll is on the system
path, but I doubt that matters for translation.   The instruction that
I run is

$ pypy.exe translate.py -Ojit --cc=mingw32

(Note that the help page only uses a single hyphen as in
"-cc=mingw32", which doesn't work, that should be fixed on the page).
The error that I see is the following:

================================================================================
...
[rtyper] specializing: 156100 / 156162 blocks   (99%)
[rtyper] -=- specialized 2519 more blocks -=-
[rtyper] -=- specialized 0 more blocks -=-
#%%
[rtyper] -=- specialized 10 more blocks -=-
[translation:info] JIT compiler generation...
[platform:execute] gcc -c -O3
c:\docume~1\admini~1\locals~1\temp\usession-default-6\platcheck_54.c
-o c:\docum
e~1\admini~1\locals~1\temp\usession-default-6\platcheck_54.o
[Timer] Timings:
[Timer] annotate                       ---  639.2 s
[Timer] rtype_lltype                   ---  388.4 s
[Timer] pyjitpl_lltype                 ---    2.7 s
[Timer] ===========================================
[Timer] Total:                         --- 1030.2 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 308, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 810, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"j:\tools\pypy\pypy\translator\tool\taskengine.py", line 116, in
_execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 286, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 395, in task_pyjitpl_lltype
[translation:ERROR]     from pypy.jit.metainterp.warmspot import apply_jit
[translation:ERROR]    File
"j:\tools\pypy\pypy\jit\metainterp\warmspot.py", line 17, in <module>
[translation:ERROR]     from pypy.jit.metainterp import history,
pyjitpl, gc, memmgr
[translation:ERROR]    File
"j:\tools\pypy\pypy\jit\metainterp\pyjitpl.py", line 23, in <module>
[translation:ERROR]     from pypy.jit.metainterp.optimizeopt.util
import args_dict_box
[translation:ERROR]    File
"j:\tools\pypy\pypy\jit\metainterp\optimizeopt\__init__.py", line 8,
in <module>
[translation:ERROR]     from pypy.jit.metainterp.optimizeopt.fficall
import OptFfiCall
[translation:ERROR]    File
"j:\tools\pypy\pypy\jit\metainterp\optimizeopt\fficall.py", line 3, in
<module>
[translation:ERROR]     from pypy.rlib.libffi import Func
[translation:ERROR]    File "j:\tools\pypy\pypy\rlib\libffi.py", line
7, in <module>
[translation:ERROR]     from pypy.rlib import clibffi
[translation:ERROR]    File "j:\tools\pypy\pypy\rlib\clibffi.py", line
115, in <module>
[translation:ERROR]     include_dir='include', library_dir='.libs'),
[translation:ERROR]    File
"j:\tools\pypy\pypy\rpython\tool\rffi_platform.py", line 778, in
configure_externa
l_library
[translation:ERROR]     raise last_error
[translation:ERROR]  CompilationError: CompilationError(out="""
[translation:ERROR]     CompilationError(err="""
[translation:ERROR]
c:\docume~1\admini~1\locals~1\temp\usession-default-6\platcheck_54.c:20:17:
fa
tal error: ffi.h: No such file or directory
[translation:ERROR]             compilation terminated.
[translation:ERROR]             """)""")
[translation] start debugger...
> j:\tools\pypy\pypy\rpython\tool\rffi_platform.py(778)configure_external_library()
-> raise last_error
(Pdb+)
...
================================================================================

I have tried placing ffi.h (and ffi_common.h and ffitarget.h) in
various strategic places, but to no avail.   I could figure it out on
my own eventually, but perhaps someone could save me a little time?
In exchange, I would be happy to write a more detailed doc section for
building pypy on windows from source using mingw.

kind regards
Caleb Hattingh

From arigo at tunes.org  Fri Aug 19 15:32:07 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 19 Aug 2011 15:32:07 +0200
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
Message-ID: <CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>

Hi David,

On Fri, Aug 19, 2011 at 1:56 PM, David Fraser <davidf at sjsoft.com> wrote:
> The pypy JIT takes a while to work out which parts of python code need optimization etc, and only after that phase do the speedups become relevant. Have there been any efforts (indeed, is it a feasible idea at all) that look at saving these optimizations for future runs of the same codebase?

No, this is not really doable.  The JIT writes explicitly in the
assembler the address of a ton of constants.  We have no clue what
these constants become when we are in a different process.  Think even
just about Python classes: there is no way at all to know that a class
at address 0x1234567 is "the same" as a previous class in a previous
process at address 0x7654321, let alone defining what exactly "the
same" means.

Instead, we can work on lowering the warm-up time of the JIT, notably
by lowering the (so far very large) overhead it takes for the JIT to
trace a loop.


A bient?t,

Armin.

From chef at ghum.de  Fri Aug 19 15:40:24 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Fri, 19 Aug 2011 15:40:24 +0200
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
Message-ID: <CAMSP2L65W7Y-qJfwuL7Y1L+YchdC2c25f=1mmUcr8JAqc6RV0g@mail.gmail.com>

>> The pypy JIT takes a while to work out which parts of python code need optimization etc,

> No, this is not really doable. ?The JIT writes explicitly in the
> assembler the address of a ton of constants.

could a special "logging area" be of use? as in

JIT decides foo_do_something_often() is worthy to be compiled into assembler
-> this decision gets added to the logging area

on next startup the JIT can read the logging area, for which functions
compiling is a good idea?

Harald


-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare

From davidf at sjsoft.com  Fri Aug 19 15:48:46 2011
From: davidf at sjsoft.com (David Fraser)
Date: Fri, 19 Aug 2011 08:48:46 -0500 (CDT)
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
Message-ID: <59dea87d-65ed-468e-a895-21710fb92de3@jackdaw.local>



On Friday, August 19, 2011 at 3:32:07 PM, "Armin Rigo" <arigo at tunes.org> wrote:
> Hi David,
> 
> On Fri, Aug 19, 2011 at 1:56 PM, David Fraser <davidf at sjsoft.com>
> wrote:
> > The pypy JIT takes a while to work out which parts of python code
> > need optimization etc, and only after that phase do the speedups
> > become relevant. Have there been any efforts (indeed, is it a
> > feasible idea at all) that look at saving these optimizations for
> > future runs of the same codebase?
> 
> No, this is not really doable.  The JIT writes explicitly in the
> assembler the address of a ton of constants.  We have no clue what
> these constants become when we are in a different process.  Think even
> just about Python classes: there is no way at all to know that a class
> at address 0x1234567 is "the same" as a previous class in a previous
> process at address 0x7654321, let alone defining what exactly "the
> same" means.

OK, thanks for the explanation

> Instead, we can work on lowering the warm-up time of the JIT, notably
> by lowering the (so far very large) overhead it takes for the JIT to
> trace a loop.

That makes a lot of sense.
Also in my web-server case, I guess I could work around the current situation by deliberately doing a bunch of requests after startup that will prime the JIT

Cheers
David

From caleb.hattingh at gmail.com  Fri Aug 19 15:49:24 2011
From: caleb.hattingh at gmail.com (Caleb Hattingh)
Date: Fri, 19 Aug 2011 15:49:24 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
Message-ID: <CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>

On 19 August 2011 14:48, Caleb Hattingh <caleb.hattingh at gmail.com> wrote:
> I have tried placing ffi.h (and ffi_common.h and ffitarget.h) in
> various strategic places, but to no avail.

It seems this particular problem (libffi for mingw) was discussed in
IRC by Varriount:

http://www.tismer.com/pypy/irc-logs/pypy/pypy.2011-07-21.log.html

Unfortunately, the chat logs shed no light.

regards
Caleb

From davidf at sjsoft.com  Fri Aug 19 15:50:02 2011
From: davidf at sjsoft.com (David Fraser)
Date: Fri, 19 Aug 2011 08:50:02 -0500 (CDT)
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CAMSP2L65W7Y-qJfwuL7Y1L+YchdC2c25f=1mmUcr8JAqc6RV0g@mail.gmail.com>
Message-ID: <635fca56-9b4e-4064-ae96-467d77b8e4ba@jackdaw.local>



On Friday, August 19, 2011 at 3:40:24 PM, "Harald Armin Massa" <chef at ghum.de> wrote:
> >> The pypy JIT takes a while to work out which parts of python code
> >> need optimization etc,
>
> > No, this is not really doable. ?The JIT writes explicitly in the
> > assembler the address of a ton of constants.
>
> could a special "logging area" be of use? as in
>
> JIT decides foo_do_something_often() is worthy to be compiled into assembler
> -> this decision gets added to the logging area
>
> on next startup the JIT can read the logging area, for which functions
> compiling is a good idea?

That sounds interesting. But a lot of the JIT advantage seems to be cross-function, which might be more complicated right?
If something like this were to be done perhaps rather than being done on function names it could be done by hashing the compiled bytecode it's executing...

David

From amauryfa at gmail.com  Fri Aug 19 16:04:45 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 19 Aug 2011 16:04:45 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
	<CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
Message-ID: <CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>

2011/8/19 Caleb Hattingh <caleb.hattingh at gmail.com>

> On 19 August 2011 14:48, Caleb Hattingh <caleb.hattingh at gmail.com> wrote:
> > I have tried placing ffi.h (and ffi_common.h and ffitarget.h) in
> > various strategic places, but to no avail.
>
> It seems this particular problem (libffi for mingw) was discussed in
> IRC by Varriount:
>
> http://www.tismer.com/pypy/irc-logs/pypy/pypy.2011-07-21.log.html
>
> Unfortunately, the chat logs shed no light.
>

I had the same issue with a machine where libffi-dev is not available.

The first fix is to modify pypy/jit/metainterp/optimizeopt/fficall.py
and move the line "from pypy.rlib.libffi import Func" shortly before
the place where "Func" is used.

Then, I don't remember whether the options "--withoutmod-_ffi
--withoutmod-_rawffi"
were necessary. These modules (i.e ctypes)  won't be available anyway.


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110819/b808bf6f/attachment.html>

From caleb.hattingh at gmail.com  Fri Aug 19 16:16:13 2011
From: caleb.hattingh at gmail.com (Caleb Hattingh)
Date: Fri, 19 Aug 2011 16:16:13 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
	<CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
	<CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>
Message-ID: <CA+NzkJ4S-=2oag_c6k+B8G-PywsNE7jP7GX1CxSayBySKwL6QQ@mail.gmail.com>

On 19 August 2011 16:04, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> 2011/8/19 Caleb Hattingh <caleb.hattingh at gmail.com>
> I had the same issue with a machine where libffi-dev is not available.
> The first fix is to modify?pypy/jit/metainterp/optimizeopt/fficall.py
> and move the line "from pypy.rlib.libffi import Func" shortly before
> the place where "Func" is used.

Thanks very much for your reply.  I have put lots of copies of
[libffi.a, libffi-5.a], [libffi.dll.a, libffi-5.dll.a] and
libffi-5.dll in a bunch of places, and it now appears that I get past
the JIT stage (where [] refers to the same file but copied with a
different name).  The build is still running, 1.3GB and counting.

So it appears that one of them has the right name in the right
location.  I will backtrack to see which of the combinations was
correct.

Best way to contribute to the documentation for building pypy on
Windows using Mingw?

regards
Caleb

From zooko at zooko.com  Fri Aug 19 17:38:40 2011
From: zooko at zooko.com (Zooko O'Whielacronx)
Date: Fri, 19 Aug 2011 09:38:40 -0600
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
Message-ID: <CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>

On Fri, Aug 19, 2011 at 1:56 PM, David Fraser <davidf at sjsoft.com> wrote:
> The pypy JIT takes a while to work out which parts of python code need optimization etc, and only after that phase do the speedups become relevant. Have there been any efforts (indeed, is it a feasible idea at all) that look at saving these optimizations for future runs of the same codebase?

If it were possible, it would be a tremendous improvement. Not only
for reducing the time before JITting, like you mentioned, but also for
deploying compiled code to a different machine. Increasingly important
are small machines that can hardly afford to run the JIT itself (due
to e.g. RAM limitations) but which would benefit greatly from the
results of JITting.

Such a feature, if it turned out to be possible, would in some sense
bring the advantages of a traditional *ahead-of-time* compiler (and
better advantages).

On Fri, Aug 19, 2011 at 7:32 AM, Armin Rigo <arigo at tunes.org> wrote:
>
> No, this is not really doable.

Maciej and you have both said this in my hearing, and you are both
brilliant experts who know a lot more than I do, but...

Let's not give up so easily!

> The JIT writes explicitly in the assembler the address of a ton of constants. ?We have no clue what these constants become when we are in a different process. ?Think even just about Python classes: there is no way at all to know that a class at address 0x1234567 is "the same" as a previous class in a previous process at address 0x7654321, let alone defining what exactly "the same" means.

As for remembering what those constants mean when you are in a
different process, can't PyPy write down a table mapping from some
symbol to a constant, and then when loading a ahead-of-time-JITted
program, swap in the values of those constants in this process?

(I know this is a very na?ve question. I don't even know what these
constants are that you are talking about.)

As for defining what "the same" means, we require only that it never
considers two things the same when they should not behave
identically?for correctness?plus that it *usually* considers two
things the same when they should?for performance, so we can use a
conservative method of determining if two classes are the same. For
example: if they have identical Python source code and their
superclasses are the same (in this sense).

> Instead, we can work on lowering the warm-up time of the JIT, notably by lowering the (so far very large) overhead it takes for the JIT to trace a loop.

Well, that's certainly an improvement I would be happy to benefit
from. I'm not volunteering to work on the "ahead-of-time-JIT" project
right now (I would hardly know where to begin), but I wanted to speak
up and encourage any ambitious JIT experts to think that it could
actually be doable.

PyPy is already a state-of-the-art JITted interpreter, plus has other
very cool features. But if this "ahead-of-time-JIT" hack could work
then it would become a cool new kind of language implementation that
has never existed before.

Regards,

Zooko

From angelflow at yahoo.com  Fri Aug 19 17:48:27 2011
From: angelflow at yahoo.com (Andy)
Date: Fri, 19 Aug 2011 08:48:27 -0700 (PDT)
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
Message-ID: <1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>

Can you elaborate about this new model? I remember reading in this list that PyPy-JIT would not work with greenlet because of how greentlet manipulated the C stack and there wasn't any easy solution. What has changed?

So will PyPy JIT work with gevent which is based on libevent (and soon to be libev)? Any target release date?


________________________________

So far PyPy cannot even *have* a JIT together with Stackless Python
features like greenlets, but this is going to change soon; in this
next model, PyPy's JIT will not work across greenlets, but will
happily JIT every single greenlet individually.


A bient?t,

Armin.
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110819/d0a7ee35/attachment.html>

From holger at merlinux.eu  Fri Aug 19 19:35:32 2011
From: holger at merlinux.eu (holger krekel)
Date: Fri, 19 Aug 2011 17:35:32 +0000
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
Message-ID: <20110819173532.GD12151@merlinux.eu>

Congrats also from here!

On the plus side pytest's own test suite passes all tests, even some which
are marked as "expected-to-fail" with cpython-2.7.  On the minus side,
the whole test run is still 2-3 times slower compared to cpython which is slightly worse than with pypy-1.5. of course there is not too much to JIT but
still a bit of a dissappointing result.

best,
holger

On Thu, Aug 18, 2011 at 19:30 +0200, Maciej Fijalkowski wrote:
> ========================
> PyPy 1.6 - kickass panda
> ========================
> 
> We're pleased to announce the 1.6 release of PyPy. This release brings a lot
> of bugfixes and performance improvements over 1.5, and improves support for
> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
> has beta level support for loading CPython C extensions.  You can download it
> here:
> 
>     http://pypy.org/download.html
> 
> What is PyPy?
> =============
> 
> PyPy is a very compliant Python interpreter, almost a drop-in replacement for
> CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison)
> due to its integrated tracing JIT compiler.
> 
> This release supports x86 machines running Linux 32/64 or Mac OS X.  Windows 32
> is beta (it roughly works but a lot of small issues have not been fixed so
> far).  Windows 64 is not yet supported.
> 
> The main topics of this release are speed and stability: on average on
> our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy 1.5,
> which was already much faster than CPython on our set of benchmarks.
> 
> The speed improvements have been made possible by optimizing many of the
> layers which compose PyPy.  In particular, we improved: the Garbage Collector,
> the JIT warmup time, the optimizations performed by the JIT, the quality of
> the generated machine code and the implementation of our Python interpreter.
> 
> .. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org
> 
> 
> Highlights
> ==========
> 
> * Numerous performance improvements, overall giving considerable speedups:
> 
>   - better GC behavior when dealing with very large objects and arrays
> 
>   - **fast ctypes:** now calls to ctypes functions are seen and optimized
>     by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times
>     faster than CPython
> 
>   - improved generators(1): simple generators now are inlined into the caller
>     loop, making performance up to 3.5 times faster than PyPy 1.5.
> 
>   - improved generators(2): thanks to other optimizations, even generators
>     that are not inlined are between 10% and 20% faster than PyPy 1.5.
> 
>   - faster warmup time for the JIT
> 
>   - JIT support for single floats (e.g., for ``array('f')``)
> 
>   - optimized dictionaries: the internal representation of dictionaries is now
>     dynamically selected depending on the type of stored objects, resulting in
>     faster code and smaller memory footprint.  For example, dictionaries whose
>     keys are all strings, or all integers. Other dictionaries are also smaller
>     due to bugfixes.
> 
> * JitViewer: this is the first official release which includes the JitViewer,
>   a web-based tool which helps you to see which parts of your Python code have
>   been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1 has
>   already been release and works well with PyPy 1.6.
> 
> * The CPython extension module API has been improved and now supports many
>   more extensions. For information on which one are supported, please refer to
>   our `compatibility wiki`_.
> 
> * Multibyte encoding support: this was of of the last areas in which we were
>   still behind CPython, but now we fully support them.
> 
> * Preliminary support for NumPy: this release includes a preview of a very
>   fast NumPy module integrated with the PyPy JIT.  Unfortunately, this does
>   not mean that you can expect to take an existing NumPy program and run it on
>   PyPy, because the module is still unfinished and supports only some of the
>   numpy API. However, barring some details, what works should be
>   blazingly fast :-)
> 
> * Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_, not
>   counting the numerous bugs that were found and reported through other
>   channels than the bug tracker.
> 
> Cheers,
> 
> Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
> Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
> Armin Rigo and the PyPy team
> 
> .. _`jitviewer`:
> http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
> .. _`bug tracker`: https://bugs.pypy.org
> .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 

From caleb.hattingh at gmail.com  Fri Aug 19 21:20:46 2011
From: caleb.hattingh at gmail.com (Caleb Hattingh)
Date: Fri, 19 Aug 2011 21:20:46 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CA+NzkJ7A-ms_7fyJU2Ay6XGq-gVdFjE875OjnYDxEyZF3iG6-g@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
	<CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
	<CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>
	<CA+NzkJ4S-=2oag_c6k+B8G-PywsNE7jP7GX1CxSayBySKwL6QQ@mail.gmail.com>
	<CAGmFidavF43TyBTdG+LzwM+2SdXaeJ27X4TB5Jffm9gV8x-5gQ@mail.gmail.com>
	<CA+NzkJ7A-ms_7fyJU2Ay6XGq-gVdFjE875OjnYDxEyZF3iG6-g@mail.gmail.com>
Message-ID: <CA+NzkJ62hwvHqe0gM8fH33+JNQNgOxYK9NtDjcrY1zUiXTkHag@mail.gmail.com>

On 19 August 2011 18:55, Caleb Hattingh <caleb.hattingh at gmail.com> wrote:
> I ran out of memory some time after the JIT had successfully compiled,
> so now I am trying
>
> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ./translate.py -Ojit
>
> as suggested in the help page. ?Will post here again if I succeed.

Unfortunately, another error, and it looks like something possibly to
do with multibyte encoding:

    j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
error: conflicti
    ng types for 'ssize_t'

Here is the last bit of error log:

========================================================================

[c:writing] objspace_std_proxy_helpers.c
[c:writing] rpython_memory_gctransform_transform.c
[c:writing] objspace_std_unicodetype.c
[c:writing] module_pypyjit_interp_resop.c
[c:writing] objspace_std_complextype.c
[c:writing] module__random_interp_random.c
[c:writing] module__file___init__.c
[c:writing] module_micronumpy_interp_ufuncs.c
[c:writing] jit_metainterp_optimizeopt___init__.c
[translation:info] written:
c:\docume~1\admini~1\locals~1\temp\usession-default-12\testing_1\testing_1.c
[translation:info] Compiling c source...
[platform:execute] make  in
c:\docume~1\admini~1\locals~1\temp\usession-default-12\testing_1
[platform:Error] In file included from common_header.h:35:0,
[platform:Error]                  from testing_1.c:1:
[platform:Error] j:\tools\pypy\pypy\translator\c/src/stack.h:47:1:
warning: 'thread' attribute directive ignor
ed
[platform:Error] In file included from common_header.h:46:0,
[platform:Error]                  from testing_1.c:1:
[platform:Error]
j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
error: conflicting type
s for 'ssize_t'
[platform:Error]
c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/types.h:118:18:
note: previous
declaration of 'ssize_t' was here
[platform:Error] make: *** [testing_1.o] Error 1
[platform:Error] make: Entering directory `/tmp/usession-default-12/testing_1'
[platform:Error] gcc -O3 -DPy_BUILD_CORE -o testing_1.o -c testing_1.c
-Ij:\\tools\\pypy\\pypy\\translator\\c
-Ij:\\tools\\pypy\\pypy\\module\\cpyext\\include -I..
[platform:Error] make: Leaving directory `/tmp/usession-default-12/testing_1'
[Timer] Timings:
[Timer] annotate                       ---  656.2 s
[Timer] rtype_lltype                   ---  443.8 s
[Timer] pyjitpl_lltype                 ---  600.7 s
[Timer] backendopt_lltype              ---  240.1 s
[Timer] stackcheckinsertion_lltype     ---   52.7 s
[Timer] database_c                     ---  332.4 s
[Timer] source_c                       ---  458.4 s
[Timer] compile_c                      ---    7.2 s
[Timer] ===========================================
[Timer] Total:                         --- 2791.4 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 308, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 810, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"j:\tools\pypy\pypy\translator\tool\taskengine.py", line 116, in
_execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 286, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\driver.py",
line 573, in task_compile_c
[translation:ERROR]     cbuilder.compile(**kwds)
[translation:ERROR]    File "j:\tools\pypy\pypy\translator\c\genc.py",
line 513, in compile
[translation:ERROR]     extra_opts)
[translation:ERROR]    File
"j:\tools\pypy\pypy\translator\platform\posix.py", line 188, in
execute_makefile
[translation:ERROR]     self._handle_error(returncode, stdout, stderr,
path.join('make'))
[translation:ERROR]    File
"j:\tools\pypy\pypy\translator\platform\windows.py", line 372, in
_handle_error
[translation:ERROR]     returncode, '', stderr + stdout, outname)
[translation:ERROR]    File
"j:\tools\pypy\pypy\translator\platform\__init__.py", line 130, in
_handle_error
[translation:ERROR]     raise CompilationError(stdout, stderr)
[translation:ERROR]  CompilationError: CompilationError(err="""
[translation:ERROR]     In file included from common_header.h:35:0,
[translation:ERROR]                      from testing_1.c:1:
[translation:ERROR]
j:\tools\pypy\pypy\translator\c/src/stack.h:47:1: warning: 'thread'
attribute directiv
e ignored
[translation:ERROR]     In file included from common_header.h:46:0,
[translation:ERROR]                      from testing_1.c:1:
[translation:ERROR]
j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
error: conflicti
ng types for 'ssize_t'
[translation:ERROR]
c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/types.h:118:18:
note: pr
evious declaration of 'ssize_t' was here
[translation:ERROR]     make: *** [testing_1.o] Error 1
[translation:ERROR]     make: Entering directory
`/tmp/usession-default-12/testing_1'
[translation:ERROR]     gcc -O3 -DPy_BUILD_CORE -o testing_1.o -c
testing_1.c -Ij:\\tools\\pypy\\pypy\\transla
tor\\c -Ij:\\tools\\pypy\\pypy\\module\\cpyext\\include -I..
[translation:ERROR]     make: Leaving directory
`/tmp/usession-default-12/testing_1'
[translation:ERROR]     """)
[translation] start debugger...
> j:\tools\pypy\pypy\translator\platform\__init__.py(130)_handle_error()
-> raise CompilationError(stdout, stderr)
(Pdb+)

========================================================================

I am going to try to build with MSVC, see if that works.

regards
Caleb

From amauryfa at gmail.com  Fri Aug 19 21:29:54 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 19 Aug 2011 21:29:54 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CA+NzkJ62hwvHqe0gM8fH33+JNQNgOxYK9NtDjcrY1zUiXTkHag@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
	<CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
	<CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>
	<CA+NzkJ4S-=2oag_c6k+B8G-PywsNE7jP7GX1CxSayBySKwL6QQ@mail.gmail.com>
	<CAGmFidavF43TyBTdG+LzwM+2SdXaeJ27X4TB5Jffm9gV8x-5gQ@mail.gmail.com>
	<CA+NzkJ7A-ms_7fyJU2Ay6XGq-gVdFjE875OjnYDxEyZF3iG6-g@mail.gmail.com>
	<CA+NzkJ62hwvHqe0gM8fH33+JNQNgOxYK9NtDjcrY1zUiXTkHag@mail.gmail.com>
Message-ID: <CAGmFidae109c27gKnJD6=iAfzV2WRFsaTA9yE08mb8weJGGg0w@mail.gmail.com>

Actually it's enough to change the declaration of ssize_t in
multibytecodec.h: long instead of int.

Amaury Forgeot d'Arc
Le 19 ao?t 2011 21:20, "Caleb Hattingh" <caleb.hattingh at gmail.com> a ?crit :
> On 19 August 2011 18:55, Caleb Hattingh <caleb.hattingh at gmail.com> wrote:
>> I ran out of memory some time after the JIT had successfully compiled,
>> so now I am trying
>>
>> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ./translate.py
-Ojit
>>
>> as suggested in the help page.  Will post here again if I succeed.
>
> Unfortunately, another error, and it looks like something possibly to
> do with multibyte encoding:
>
> j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
> error: conflicti
> ng types for 'ssize_t'
>
> Here is the last bit of error log:
>
> ========================================================================
>
> [c:writing] objspace_std_proxy_helpers.c
> [c:writing] rpython_memory_gctransform_transform.c
> [c:writing] objspace_std_unicodetype.c
> [c:writing] module_pypyjit_interp_resop.c
> [c:writing] objspace_std_complextype.c
> [c:writing] module__random_interp_random.c
> [c:writing] module__file___init__.c
> [c:writing] module_micronumpy_interp_ufuncs.c
> [c:writing] jit_metainterp_optimizeopt___init__.c
> [translation:info] written:
>
c:\docume~1\admini~1\locals~1\temp\usession-default-12\testing_1\testing_1.c
> [translation:info] Compiling c source...
> [platform:execute] make in
> c:\docume~1\admini~1\locals~1\temp\usession-default-12\testing_1
> [platform:Error] In file included from common_header.h:35:0,
> [platform:Error] from testing_1.c:1:
> [platform:Error] j:\tools\pypy\pypy\translator\c/src/stack.h:47:1:
> warning: 'thread' attribute directive ignor
> ed
> [platform:Error] In file included from common_header.h:46:0,
> [platform:Error] from testing_1.c:1:
> [platform:Error]
> j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
> error: conflicting type
> s for 'ssize_t'
> [platform:Error]
>
c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/types.h:118:18:
> note: previous
> declaration of 'ssize_t' was here
> [platform:Error] make: *** [testing_1.o] Error 1
> [platform:Error] make: Entering directory
`/tmp/usession-default-12/testing_1'
> [platform:Error] gcc -O3 -DPy_BUILD_CORE -o testing_1.o -c testing_1.c
> -Ij:\\tools\\pypy\\pypy\\translator\\c
> -Ij:\\tools\\pypy\\pypy\\module\\cpyext\\include -I..
> [platform:Error] make: Leaving directory
`/tmp/usession-default-12/testing_1'
> [Timer] Timings:
> [Timer] annotate --- 656.2 s
> [Timer] rtype_lltype --- 443.8 s
> [Timer] pyjitpl_lltype --- 600.7 s
> [Timer] backendopt_lltype --- 240.1 s
> [Timer] stackcheckinsertion_lltype --- 52.7 s
> [Timer] database_c --- 332.4 s
> [Timer] source_c --- 458.4 s
> [Timer] compile_c --- 7.2 s
> [Timer] ===========================================
> [Timer] Total: --- 2791.4 s
> [translation:ERROR] Error:
> [translation:ERROR] Traceback (most recent call last):
> [translation:ERROR] File "translate.py", line 308, in main
> [translation:ERROR] drv.proceed(goals)
> [translation:ERROR] File "j:\tools\pypy\pypy\translator\driver.py",
> line 810, in proceed
> [translation:ERROR] return self._execute(goals, task_skip =
> self._maybe_skip())
> [translation:ERROR] File
> "j:\tools\pypy\pypy\translator\tool\taskengine.py", line 116, in
> _execute
> [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds)
> [translation:ERROR] File "j:\tools\pypy\pypy\translator\driver.py",
> line 286, in _do
> [translation:ERROR] res = func()
> [translation:ERROR] File "j:\tools\pypy\pypy\translator\driver.py",
> line 573, in task_compile_c
> [translation:ERROR] cbuilder.compile(**kwds)
> [translation:ERROR] File "j:\tools\pypy\pypy\translator\c\genc.py",
> line 513, in compile
> [translation:ERROR] extra_opts)
> [translation:ERROR] File
> "j:\tools\pypy\pypy\translator\platform\posix.py", line 188, in
> execute_makefile
> [translation:ERROR] self._handle_error(returncode, stdout, stderr,
> path.join('make'))
> [translation:ERROR] File
> "j:\tools\pypy\pypy\translator\platform\windows.py", line 372, in
> _handle_error
> [translation:ERROR] returncode, '', stderr + stdout, outname)
> [translation:ERROR] File
> "j:\tools\pypy\pypy\translator\platform\__init__.py", line 130, in
> _handle_error
> [translation:ERROR] raise CompilationError(stdout, stderr)
> [translation:ERROR] CompilationError: CompilationError(err="""
> [translation:ERROR] In file included from common_header.h:35:0,
> [translation:ERROR] from testing_1.c:1:
> [translation:ERROR]
> j:\tools\pypy\pypy\translator\c/src/stack.h:47:1: warning: 'thread'
> attribute directiv
> e ignored
> [translation:ERROR] In file included from common_header.h:46:0,
> [translation:ERROR] from testing_1.c:1:
> [translation:ERROR]
> j:\tools\pypy\pypy\translator\c/src/cjkcodecs/multibytecodec.h:12:13:
> error: conflicti
> ng types for 'ssize_t'
> [translation:ERROR]
>
c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/types.h:118:18:
> note: pr
> evious declaration of 'ssize_t' was here
> [translation:ERROR] make: *** [testing_1.o] Error 1
> [translation:ERROR] make: Entering directory
> `/tmp/usession-default-12/testing_1'
> [translation:ERROR] gcc -O3 -DPy_BUILD_CORE -o testing_1.o -c
> testing_1.c -Ij:\\tools\\pypy\\pypy\\transla
> tor\\c -Ij:\\tools\\pypy\\pypy\\module\\cpyext\\include -I..
> [translation:ERROR] make: Leaving directory
> `/tmp/usession-default-12/testing_1'
> [translation:ERROR] """)
> [translation] start debugger...
>> j:\tools\pypy\pypy\translator\platform\__init__.py(130)_handle_error()
> -> raise CompilationError(stdout, stderr)
> (Pdb+)
>
> ========================================================================
>
> I am going to try to build with MSVC, see if that works.
>
> regards
> Caleb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110819/3df0f0af/attachment-0001.html>

From jfcgauss at gmail.com  Fri Aug 19 22:07:34 2011
From: jfcgauss at gmail.com (Serhat Sevki Dincer)
Date: Fri, 19 Aug 2011 23:07:34 +0300
Subject: [pypy-dev] regex (new re) and 1.6
Message-ID: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>

http://pypi.python.org/pypi/regex installs nicely on pypy-1.6 on my
Ubuntu 11.04 i386 but

>>>> from regex import sub
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/var/Home/serhat/Program/pypy-1.6/site-packages/regex.py",
line 343, in <module>
    from _regex_core import *
  File "/var/Home/serhat/Program/pypy-1.6/site-packages/_regex_core.py",
line 21, in <module>
    import _regex
ImportError: unable to load extension module
'/var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so':
/var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so:
undefined symbol: Py_UNICODE_TOTITLE

I dont see any (in)compatibility info on
https://bitbucket.org/pypy/compatibility/wiki/regex.

Also
pypy-1.6$ grep -r 'Py_UNICODE_TO' include/
include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOLOWER(wchar_t arg0);
include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOUPPER(wchar_t arg0);

TOTITLE is missing?
http://docs.python.org/release/2.7.2/library/stdtypes.html#str.title
is working on (unicode) strings in pypy. I dont get it, what do u
think is up?

From caleb.hattingh at gmail.com  Fri Aug 19 22:22:17 2011
From: caleb.hattingh at gmail.com (Caleb Hattingh)
Date: Fri, 19 Aug 2011 22:22:17 +0200
Subject: [pypy-dev] Build pypy 1.6 on Windows XP with MinGW: can't find
	ffi.h
In-Reply-To: <CAGmFidae109c27gKnJD6=iAfzV2WRFsaTA9yE08mb8weJGGg0w@mail.gmail.com>
References: <CA+NzkJ5DNcrRRoOEK257cXXhXJtczf16N_cmJVEFNZp2WJd6tw@mail.gmail.com>
	<CA+NzkJ6E1WO+s9+pS-BtkyRONYPs6hN=kK8v+rTWagja0S_wqA@mail.gmail.com>
	<CAGmFidZs6DWNJTr9gVUDY=xm7qp3JF6op9B+-G=fj5Ra6Cp+xQ@mail.gmail.com>
	<CA+NzkJ4S-=2oag_c6k+B8G-PywsNE7jP7GX1CxSayBySKwL6QQ@mail.gmail.com>
	<CAGmFidavF43TyBTdG+LzwM+2SdXaeJ27X4TB5Jffm9gV8x-5gQ@mail.gmail.com>
	<CA+NzkJ7A-ms_7fyJU2Ay6XGq-gVdFjE875OjnYDxEyZF3iG6-g@mail.gmail.com>
	<CA+NzkJ62hwvHqe0gM8fH33+JNQNgOxYK9NtDjcrY1zUiXTkHag@mail.gmail.com>
	<CAGmFidae109c27gKnJD6=iAfzV2WRFsaTA9yE08mb8weJGGg0w@mail.gmail.com>
Message-ID: <CA+NzkJ5pnsk1zF===UR9B+p=zNe0+Bqm7bTg-WiKwHPamq8Nkw@mail.gmail.com>

On 19 August 2011 21:29, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Actually it's enough to change the declaration of ssize_t in
> multibytecodec.h: long instead of int.

Ok, I am going to give that a shot.  MSVC also failed (with a very
long list of errors).   Back to mingw it is.

regards
Caleb

From fijall at gmail.com  Fri Aug 19 23:52:00 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 19 Aug 2011 23:52:00 +0200
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <20110819173532.GD12151@merlinux.eu>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
	<20110819173532.GD12151@merlinux.eu>
Message-ID: <CAK5idxTUj+e4wuAKE8jBXdE5kq_P6L1BsEVgdA-Nw7jKGeWPVA@mail.gmail.com>

On Fri, Aug 19, 2011 at 7:35 PM, holger krekel <holger at merlinux.eu> wrote:
> Congrats also from here!
>
> On the plus side pytest's own test suite passes all tests, even some which
> are marked as "expected-to-fail" with cpython-2.7. ?On the minus side,
> the whole test run is still 2-3 times slower compared to cpython which is slightly worse than with pypy-1.5. of course there is not too much to JIT but
> still a bit of a dissappointing result.

Well, yes, but also we have no good answer to this. I think the reason
why it got slower is because we compile functions now as well (which
takes extra time). Test suites are hard I fear :(

>
> best,
> holger
>
> On Thu, Aug 18, 2011 at 19:30 +0200, Maciej Fijalkowski wrote:
>> ========================
>> PyPy 1.6 - kickass panda
>> ========================
>>
>> We're pleased to announce the 1.6 release of PyPy. This release brings a lot
>> of bugfixes and performance improvements over 1.5, and improves support for
>> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
>> has beta level support for loading CPython C extensions. ?You can download it
>> here:
>>
>> ? ? http://pypy.org/download.html
>>
>> What is PyPy?
>> =============
>>
>> PyPy is a very compliant Python interpreter, almost a drop-in replacement for
>> CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison)
>> due to its integrated tracing JIT compiler.
>>
>> This release supports x86 machines running Linux 32/64 or Mac OS X. ?Windows 32
>> is beta (it roughly works but a lot of small issues have not been fixed so
>> far). ?Windows 64 is not yet supported.
>>
>> The main topics of this release are speed and stability: on average on
>> our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy 1.5,
>> which was already much faster than CPython on our set of benchmarks.
>>
>> The speed improvements have been made possible by optimizing many of the
>> layers which compose PyPy. ?In particular, we improved: the Garbage Collector,
>> the JIT warmup time, the optimizations performed by the JIT, the quality of
>> the generated machine code and the implementation of our Python interpreter.
>>
>> .. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org
>>
>>
>> Highlights
>> ==========
>>
>> * Numerous performance improvements, overall giving considerable speedups:
>>
>> ? - better GC behavior when dealing with very large objects and arrays
>>
>> ? - **fast ctypes:** now calls to ctypes functions are seen and optimized
>> ? ? by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times
>> ? ? faster than CPython
>>
>> ? - improved generators(1): simple generators now are inlined into the caller
>> ? ? loop, making performance up to 3.5 times faster than PyPy 1.5.
>>
>> ? - improved generators(2): thanks to other optimizations, even generators
>> ? ? that are not inlined are between 10% and 20% faster than PyPy 1.5.
>>
>> ? - faster warmup time for the JIT
>>
>> ? - JIT support for single floats (e.g., for ``array('f')``)
>>
>> ? - optimized dictionaries: the internal representation of dictionaries is now
>> ? ? dynamically selected depending on the type of stored objects, resulting in
>> ? ? faster code and smaller memory footprint. ?For example, dictionaries whose
>> ? ? keys are all strings, or all integers. Other dictionaries are also smaller
>> ? ? due to bugfixes.
>>
>> * JitViewer: this is the first official release which includes the JitViewer,
>> ? a web-based tool which helps you to see which parts of your Python code have
>> ? been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1 has
>> ? already been release and works well with PyPy 1.6.
>>
>> * The CPython extension module API has been improved and now supports many
>> ? more extensions. For information on which one are supported, please refer to
>> ? our `compatibility wiki`_.
>>
>> * Multibyte encoding support: this was of of the last areas in which we were
>> ? still behind CPython, but now we fully support them.
>>
>> * Preliminary support for NumPy: this release includes a preview of a very
>> ? fast NumPy module integrated with the PyPy JIT. ?Unfortunately, this does
>> ? not mean that you can expect to take an existing NumPy program and run it on
>> ? PyPy, because the module is still unfinished and supports only some of the
>> ? numpy API. However, barring some details, what works should be
>> ? blazingly fast :-)
>>
>> * Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_, not
>> ? counting the numerous bugs that were found and reported through other
>> ? channels than the bug tracker.
>>
>> Cheers,
>>
>> Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
>> Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
>> Armin Rigo and the PyPy team
>>
>> .. _`jitviewer`:
>> http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
>> .. _`bug tracker`: https://bugs.pypy.org
>> .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>

From holger at merlinux.eu  Sat Aug 20 08:15:08 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 20 Aug 2011 06:15:08 +0000
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <CAK5idxTUj+e4wuAKE8jBXdE5kq_P6L1BsEVgdA-Nw7jKGeWPVA@mail.gmail.com>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
	<20110819173532.GD12151@merlinux.eu>
	<CAK5idxTUj+e4wuAKE8jBXdE5kq_P6L1BsEVgdA-Nw7jKGeWPVA@mail.gmail.com>
Message-ID: <20110820061508.GE12151@merlinux.eu>

On Fri, Aug 19, 2011 at 23:52 +0200, Maciej Fijalkowski wrote:
> On Fri, Aug 19, 2011 at 7:35 PM, holger krekel <holger at merlinux.eu> wrote:
> > Congrats also from here!
> >
> > On the plus side pytest's own test suite passes all tests, even some which
> > are marked as "expected-to-fail" with cpython-2.7. ?On the minus side,
> > the whole test run is still 2-3 times slower compared to cpython which is slightly worse than with pypy-1.5. of course there is not too much to JIT but
> > still a bit of a dissappointing result.
> 
> Well, yes, but also we have no good answer to this. I think the reason
> why it got slower is because we compile functions now as well (which
> takes extra time). Test suites are hard I fear :(

Maybe worth a small note because i guess test suites are often run to test pypy.

FWIW if i run the test suite with --jit off it takes 117 seconds instead of
72 with JIT (cpython takes about 25 seconds).  So maybe one question is
why pypy-no-jit is already 4-5 times slower to begin with.

Holger

> >
> > best,
> > holger
> >
> > On Thu, Aug 18, 2011 at 19:30 +0200, Maciej Fijalkowski wrote:
> >> ========================
> >> PyPy 1.6 - kickass panda
> >> ========================
> >>
> >> We're pleased to announce the 1.6 release of PyPy. This release brings a lot
> >> of bugfixes and performance improvements over 1.5, and improves support for
> >> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
> >> has beta level support for loading CPython C extensions. ?You can download it
> >> here:
> >>
> >> ? ? http://pypy.org/download.html
> >>
> >> What is PyPy?
> >> =============
> >>
> >> PyPy is a very compliant Python interpreter, almost a drop-in replacement for
> >> CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison)
> >> due to its integrated tracing JIT compiler.
> >>
> >> This release supports x86 machines running Linux 32/64 or Mac OS X. ?Windows 32
> >> is beta (it roughly works but a lot of small issues have not been fixed so
> >> far). ?Windows 64 is not yet supported.
> >>
> >> The main topics of this release are speed and stability: on average on
> >> our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy 1.5,
> >> which was already much faster than CPython on our set of benchmarks.
> >>
> >> The speed improvements have been made possible by optimizing many of the
> >> layers which compose PyPy. ?In particular, we improved: the Garbage Collector,
> >> the JIT warmup time, the optimizations performed by the JIT, the quality of
> >> the generated machine code and the implementation of our Python interpreter.
> >>
> >> .. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org
> >>
> >>
> >> Highlights
> >> ==========
> >>
> >> * Numerous performance improvements, overall giving considerable speedups:
> >>
> >> ? - better GC behavior when dealing with very large objects and arrays
> >>
> >> ? - **fast ctypes:** now calls to ctypes functions are seen and optimized
> >> ? ? by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times
> >> ? ? faster than CPython
> >>
> >> ? - improved generators(1): simple generators now are inlined into the caller
> >> ? ? loop, making performance up to 3.5 times faster than PyPy 1.5.
> >>
> >> ? - improved generators(2): thanks to other optimizations, even generators
> >> ? ? that are not inlined are between 10% and 20% faster than PyPy 1.5.
> >>
> >> ? - faster warmup time for the JIT
> >>
> >> ? - JIT support for single floats (e.g., for ``array('f')``)
> >>
> >> ? - optimized dictionaries: the internal representation of dictionaries is now
> >> ? ? dynamically selected depending on the type of stored objects, resulting in
> >> ? ? faster code and smaller memory footprint. ?For example, dictionaries whose
> >> ? ? keys are all strings, or all integers. Other dictionaries are also smaller
> >> ? ? due to bugfixes.
> >>
> >> * JitViewer: this is the first official release which includes the JitViewer,
> >> ? a web-based tool which helps you to see which parts of your Python code have
> >> ? been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1 has
> >> ? already been release and works well with PyPy 1.6.
> >>
> >> * The CPython extension module API has been improved and now supports many
> >> ? more extensions. For information on which one are supported, please refer to
> >> ? our `compatibility wiki`_.
> >>
> >> * Multibyte encoding support: this was of of the last areas in which we were
> >> ? still behind CPython, but now we fully support them.
> >>
> >> * Preliminary support for NumPy: this release includes a preview of a very
> >> ? fast NumPy module integrated with the PyPy JIT. ?Unfortunately, this does
> >> ? not mean that you can expect to take an existing NumPy program and run it on
> >> ? PyPy, because the module is still unfinished and supports only some of the
> >> ? numpy API. However, barring some details, what works should be
> >> ? blazingly fast :-)
> >>
> >> * Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_, not
> >> ? counting the numerous bugs that were found and reported through other
> >> ? channels than the bug tracker.
> >>
> >> Cheers,
> >>
> >> Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
> >> Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
> >> Armin Rigo and the PyPy team
> >>
> >> .. _`jitviewer`:
> >> http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
> >> .. _`bug tracker`: https://bugs.pypy.org
> >> .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >
> 

From mitchell.hashimoto at gmail.com  Sat Aug 20 08:57:57 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sat, 20 Aug 2011 00:57:57 -0600
Subject: [pypy-dev] PyGame for translatorshell on Lion
Message-ID: <CACOXMX7B3vZf-_eF54oS7Xjfc3E25brdPoFc=B8swd=8OBD2_w@mail.gmail.com>

Hello,

I've been trying to play around with the PyPy source and work on fixing some
bugs. One of the things I wanted to look at was the graph files generated by
the shell. I installed PyGame and GraphViz and when I run a "t.view" on a
Translation object, I get the following crazy thing.

Has anyone see this? Is this something I should bring up to the PyGame guys?
Does anyone have this working on Lion (assuming yes)?

Best,
Mitchell

Aug 20 00:57:14 picochip.local python[58716] <Error>:
kCGErrorInvalidConnection: CGSGetCurrentCursorLocation: Invalid connection
Aug 20 00:57:14 picochip.local python[58716] <Error>: kCGErrorFailure: Set a
breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
Aug 20 00:57:14 picochip.local python[58716] <Error>:
kCGErrorInvalidConnection: CGSGetCurrentCursorLocation: Invalid connection
Aug 20 00:57:14 picochip.local python[58716] <Error>:
kCGErrorInvalidConnection: CGSNewWindowWithOpaqueShape: Invalid connection
2011-08-20 00:57:14.130 python[58716:707] *** Terminating app due to
uncaught exception 'NSInternalInconsistencyException', reason: 'Error (1002)
creating CGSWindow on line 262'
*** First throw call stack:
(
0   CoreFoundation                      0x00007fff81445986
__exceptionPreprocess + 198
1   libobjc.A.dylib                     0x00007fff88125d5e
objc_exception_throw + 43
 2   CoreFoundation                      0x00007fff814457ba +[NSException
raise:format:arguments:] + 106
3   CoreFoundation                      0x00007fff81445744 +[NSException
raise:format:] + 116
 4   AppKit                              0x00007fff8a686e78
_NSCreateWindowWithOpaqueShape2 + 788
5   AppKit                              0x00007fff8a5d4c80 -[NSWindow
_commonAwake] + 1875
 6   AppKit                              0x00007fff8a5d3894 -[NSWindow
_commonInitFrame:styleMask:backing:defer:] + 1871
7   AppKit                              0x00007fff8a5d28e0 -[NSWindow
_initContent:styleMask:backing:defer:contentView:] + 1502
 8   AppKit                              0x00007fff8a5d22b1 -[NSWindow
initWithContentRect:styleMask:backing:defer:] + 47
9   libSDL-1.2.0.dylib                  0x00000001081d5cfd
-[SDL_QuartzWindow initWithContentRect:styleMask:backing:defer:] + 285
 10  libSDL-1.2.0.dylib                  0x00000001081d30a4 QZ_SetVideoMode
+ 1076
11  libSDL-1.2.0.dylib                  0x00000001081c9e8a SDL_SetVideoMode
+ 602
 12  display.so                          0x000000010824f489 set_mode + 361
13  python                              0x0000000107d63bab
PyEval_EvalFrameEx + 18011
 14  python                              0x0000000107d66f43 fast_function +
179
15  python                              0x0000000107d63c5d
PyEval_EvalFrameEx + 18189
 16  python                              0x0000000107d66e07
PyEval_EvalCodeEx + 2103
17  python                              0x0000000107d66fb8 fast_function +
296
 18  python                              0x0000000107d63c5d
PyEval_EvalFrameEx + 18189
19  python                              0x0000000107d66e07 PyEval_EvalCodeEx
+ 2103
 20  python                              0x0000000107cf480b function_call +
347
21  python                              0x0000000107ccc141 PyObject_Call +
97
 22  python                              0x0000000107cdddf7
instancemethod_call + 503
23  python                              0x0000000107ccc141 PyObject_Call +
97
 24  python                              0x0000000107d2798a slot_tp_init +
106
25  python                              0x0000000107d203a5 type_call + 229
 26  python                              0x0000000107ccc141 PyObject_Call +
97
27  python                              0x0000000107d63e75
PyEval_EvalFrameEx + 18725
 28  python                              0x0000000107d66f43 fast_function +
179
29  python                              0x0000000107d63c5d
PyEval_EvalFrameEx + 18189
 30  python                              0x0000000107d66e07
PyEval_EvalCodeEx + 2103
31  python                              0x0000000107cf480b function_call +
347
 32  python                              0x0000000107ccc141 PyObject_Call +
97
33  python                              0x0000000107d6451a
PyEval_EvalFrameEx + 20426
 34  python                              0x0000000107d66f43 fast_function +
179
35  python                              0x0000000107d63c5d
PyEval_EvalFrameEx + 18189
 36  python                              0x0000000107d66e07
PyEval_EvalCodeEx + 2103
37  python                              0x0000000107d66fb8 fast_function +
296
 38  python                              0x0000000107d63c5d
PyEval_EvalFrameEx + 18189
39  python                              0x0000000107d66e07 PyEval_EvalCodeEx
+ 2103
 40  python                              0x0000000107d66e86 PyEval_EvalCode
+ 54
41  python                              0x0000000107d851a4 PyRun_FileExFlags
+ 164
 42  python                              0x0000000107d86458
PyRun_SimpleFileExFlags + 392
43  python                              0x0000000107d99729 Py_Main + 2969
 44  python                              0x0000000107cc5f84 start + 52
45  ???                                 0x0000000000000004 0x0 + 4
)
terminate called throwing an exceptionTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/Users/mitchellh/code/3rdparty/python/pypy/pypy/translator/interactive.py",
line 36, in view
    self.context.view()
  File
"/Users/mitchellh/code/3rdparty/python/pypy/pypy/translator/translator.py",
line 151, in view
    FlowGraphPage(self).display()
  File "/Users/mitchellh/code/3rdparty/python/pypy/dotviewer/graphpage.py",
line 32, in display
    graphclient.display_page(self, save_tmp_file=self.save_tmp_file)
  File
"/Users/mitchellh/code/3rdparty/python/pypy/dotviewer/graphclient.py", line
44, in display_page
    reload(0)
  File
"/Users/mitchellh/code/3rdparty/python/pypy/dotviewer/graphclient.py", line
40, in reload
    send_graph_messages(io, messages)
  File
"/Users/mitchellh/code/3rdparty/python/pypy/dotviewer/graphclient.py", line
98, in send_graph_messages
    raise ioerror
IOError: connection unexpectedly closed (graphserver crash?)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/27b5714d/attachment-0001.html>

From mitchell.hashimoto at gmail.com  Sat Aug 20 09:40:17 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sat, 20 Aug 2011 01:40:17 -0600
Subject: [pypy-dev] List index out of range for many tests
Message-ID: <CACOXMX4niX4aSPKGBS6HgvpaG5OHGGKFQOtYzmfdiXX4X_Egdg@mail.gmail.com>

Hello,

I'm taking a look at some failing tests in the buildbot and trying to mirror
them here on my local machine, but the tests are giving me the following
error:

  File "/Users/mitchellh/code/3rdparty/python/pypy/pypy/interpreter/pyframe.py",
line 193, in pushvalue

    self.locals_stack_w[depth] = w_object
IndexError: list assignment index out of range

The full stack trace: https://gist.github.com/e458477a6c60c3d15b9c

And the command I'm running: pypy> CC=gcc python pytest.py -s
lib-python/2.7/test/test_distutils.py

I'm running "pytest.py" with CPython 2.7.1 although the same result occurs
when I try with PyPy 1.6.0.

I know there must be something wrong on my system since the buildbot sees
something completely different. Can someone please point me in the right
direction?

Thanks,
Mitchell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/b97538c8/attachment.html>

From fijall at gmail.com  Sat Aug 20 09:48:22 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 20 Aug 2011 09:48:22 +0200
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <20110820061508.GE12151@merlinux.eu>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
	<20110819173532.GD12151@merlinux.eu>
	<CAK5idxTUj+e4wuAKE8jBXdE5kq_P6L1BsEVgdA-Nw7jKGeWPVA@mail.gmail.com>
	<20110820061508.GE12151@merlinux.eu>
Message-ID: <CAK5idxSahVm06N3WYb8TZ1_21gDd8Z85SUfpbrvZeE-nurU+Gw@mail.gmail.com>

On Sat, Aug 20, 2011 at 8:15 AM, holger krekel <holger at merlinux.eu> wrote:
> On Fri, Aug 19, 2011 at 23:52 +0200, Maciej Fijalkowski wrote:
>> On Fri, Aug 19, 2011 at 7:35 PM, holger krekel <holger at merlinux.eu> wrote:
>> > Congrats also from here!
>> >
>> > On the plus side pytest's own test suite passes all tests, even some which
>> > are marked as "expected-to-fail" with cpython-2.7. ?On the minus side,
>> > the whole test run is still 2-3 times slower compared to cpython which is slightly worse than with pypy-1.5. of course there is not too much to JIT but
>> > still a bit of a dissappointing result.
>>
>> Well, yes, but also we have no good answer to this. I think the reason
>> why it got slower is because we compile functions now as well (which
>> takes extra time). Test suites are hard I fear :(
>
> Maybe worth a small note because i guess test suites are often run to test pypy.

A small note where?

>
> FWIW if i run the test suite with --jit off it takes 117 seconds instead of
> 72 with JIT (cpython takes about 25 seconds). ?So maybe one question is
> why pypy-no-jit is already 4-5 times slower to begin with.

I guess a lot of things come into play - pypy's interpreter is
generally slower than CPython's interpreter. We can attribute this to
many things, but we were so far happy with "we pay 2x for abstraction
layer". Maybe it's not good enough any more and we can actually come
up with a better interpreter, who knows.

>
> Holger
>
>> >
>> > best,
>> > holger
>> >
>> > On Thu, Aug 18, 2011 at 19:30 +0200, Maciej Fijalkowski wrote:
>> >> ========================
>> >> PyPy 1.6 - kickass panda
>> >> ========================
>> >>
>> >> We're pleased to announce the 1.6 release of PyPy. This release brings a lot
>> >> of bugfixes and performance improvements over 1.5, and improves support for
>> >> Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and
>> >> has beta level support for loading CPython C extensions. ?You can download it
>> >> here:
>> >>
>> >> ? ? http://pypy.org/download.html
>> >>
>> >> What is PyPy?
>> >> =============
>> >>
>> >> PyPy is a very compliant Python interpreter, almost a drop-in replacement for
>> >> CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison)
>> >> due to its integrated tracing JIT compiler.
>> >>
>> >> This release supports x86 machines running Linux 32/64 or Mac OS X. ?Windows 32
>> >> is beta (it roughly works but a lot of small issues have not been fixed so
>> >> far). ?Windows 64 is not yet supported.
>> >>
>> >> The main topics of this release are speed and stability: on average on
>> >> our benchmark suite, PyPy 1.6 is between **20% and 30%** faster than PyPy 1.5,
>> >> which was already much faster than CPython on our set of benchmarks.
>> >>
>> >> The speed improvements have been made possible by optimizing many of the
>> >> layers which compose PyPy. ?In particular, we improved: the Garbage Collector,
>> >> the JIT warmup time, the optimizations performed by the JIT, the quality of
>> >> the generated machine code and the implementation of our Python interpreter.
>> >>
>> >> .. _`pypy 1.5 and cpython 2.6.2`: http://speed.pypy.org
>> >>
>> >>
>> >> Highlights
>> >> ==========
>> >>
>> >> * Numerous performance improvements, overall giving considerable speedups:
>> >>
>> >> ? - better GC behavior when dealing with very large objects and arrays
>> >>
>> >> ? - **fast ctypes:** now calls to ctypes functions are seen and optimized
>> >> ? ? by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times
>> >> ? ? faster than CPython
>> >>
>> >> ? - improved generators(1): simple generators now are inlined into the caller
>> >> ? ? loop, making performance up to 3.5 times faster than PyPy 1.5.
>> >>
>> >> ? - improved generators(2): thanks to other optimizations, even generators
>> >> ? ? that are not inlined are between 10% and 20% faster than PyPy 1.5.
>> >>
>> >> ? - faster warmup time for the JIT
>> >>
>> >> ? - JIT support for single floats (e.g., for ``array('f')``)
>> >>
>> >> ? - optimized dictionaries: the internal representation of dictionaries is now
>> >> ? ? dynamically selected depending on the type of stored objects, resulting in
>> >> ? ? faster code and smaller memory footprint. ?For example, dictionaries whose
>> >> ? ? keys are all strings, or all integers. Other dictionaries are also smaller
>> >> ? ? due to bugfixes.
>> >>
>> >> * JitViewer: this is the first official release which includes the JitViewer,
>> >> ? a web-based tool which helps you to see which parts of your Python code have
>> >> ? been compiled by the JIT, down until the assembler. The `jitviewer`_ 0.1 has
>> >> ? already been release and works well with PyPy 1.6.
>> >>
>> >> * The CPython extension module API has been improved and now supports many
>> >> ? more extensions. For information on which one are supported, please refer to
>> >> ? our `compatibility wiki`_.
>> >>
>> >> * Multibyte encoding support: this was of of the last areas in which we were
>> >> ? still behind CPython, but now we fully support them.
>> >>
>> >> * Preliminary support for NumPy: this release includes a preview of a very
>> >> ? fast NumPy module integrated with the PyPy JIT. ?Unfortunately, this does
>> >> ? not mean that you can expect to take an existing NumPy program and run it on
>> >> ? PyPy, because the module is still unfinished and supports only some of the
>> >> ? numpy API. However, barring some details, what works should be
>> >> ? blazingly fast :-)
>> >>
>> >> * Bugfixes: since the 1.5 release we fixed 53 bugs in our `bug tracker`_, not
>> >> ? counting the numerous bugs that were found and reported through other
>> >> ? channels than the bug tracker.
>> >>
>> >> Cheers,
>> >>
>> >> Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni,
>> >> Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor,
>> >> Armin Rigo and the PyPy team
>> >>
>> >> .. _`jitviewer`:
>> >> http://morepypy.blogspot.com/2011/08/visualization-of-jitted-code.html
>> >> .. _`bug tracker`: https://bugs.pypy.org
>> >> .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home
>> >> _______________________________________________
>> >> pypy-dev mailing list
>> >> pypy-dev at python.org
>> >> http://mail.python.org/mailman/listinfo/pypy-dev
>> >>
>> >
>>
>

From fijall at gmail.com  Sat Aug 20 09:51:32 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 20 Aug 2011 09:51:32 +0200
Subject: [pypy-dev] regex (new re) and 1.6
In-Reply-To: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>
References: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>
Message-ID: <CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>

On Fri, Aug 19, 2011 at 10:07 PM, Serhat Sevki Dincer
<jfcgauss at gmail.com> wrote:
> http://pypi.python.org/pypi/regex installs nicely on pypy-1.6 on my
> Ubuntu 11.04 i386 but
>
>>>>> from regex import sub
> Traceback (most recent call last):
> ?File "<console>", line 1, in <module>
> ?File "/var/Home/serhat/Program/pypy-1.6/site-packages/regex.py",
> line 343, in <module>
> ? ?from _regex_core import *
> ?File "/var/Home/serhat/Program/pypy-1.6/site-packages/_regex_core.py",
> line 21, in <module>
> ? ?import _regex
> ImportError: unable to load extension module
> '/var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so':
> /var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so:
> undefined symbol: Py_UNICODE_TOTITLE
>
> I dont see any (in)compatibility info on
> https://bitbucket.org/pypy/compatibility/wiki/regex.
>
> Also
> pypy-1.6$ grep -r 'Py_UNICODE_TO' include/
> include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOLOWER(wchar_t arg0);
> include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOUPPER(wchar_t arg0);
>
> TOTITLE is missing?
> http://docs.python.org/release/2.7.2/library/stdtypes.html#str.title
> is working on (unicode) strings in pypy. I dont get it, what do u
> think is up?

yes, TOTITLE is not implemented in cpyext, the cpython C compatibility
layer. Btw - what are you trying to achieve? If you want to compare
regex performance on pypy and cpython, it'll probably be much slower
on pypy. The C extensions layer is just a compatibility one (it's also
in beta).

Cheers,
fijal

> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From fijall at gmail.com  Sat Aug 20 09:55:32 2011
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 20 Aug 2011 09:55:32 +0200
Subject: [pypy-dev] work on libpypy.so
In-Reply-To: <4CDEBDC0-B709-4952-B489-E6534D3568BA@unbit.it>
References: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it>
	<CAMSv6X0TUT7P9+O06nWJwOtkWzfneyut9uXv=8P1vuStM61mVg@mail.gmail.com>
	<4CDEBDC0-B709-4952-B489-E6534D3568BA@unbit.it>
Message-ID: <CAK5idxR2-KsvjDATxYYHiZZU=2dZqkLVdV4QvLOoMQv8nMiQBA@mail.gmail.com>

On Fri, Aug 19, 2011 at 12:54 PM, Roberto De Ioris <roberto at unbit.it> wrote:
>
> Il giorno 25/lug/2011, alle ore 14:31, Armin Rigo ha scritto:
>
>>
>> Indeed, there is nothing like that so far. ?We need to think out the
>> kind of C-level API which makes sense for libpypy.so to expose. ?There
>> are two options: either we tweak a little bit cpyext to expose the
>> CPython C API, or we design some higher-performance PyPy-only C API...
>>
>> For the latter case the first issue is how to store a reference to a
>> PyPy object in C, given that it moves around --- maybe with an
>> indirection, which would be a bit similar to what cpyext does but can
>> be done with much better performance if we are free to design the API.
>>
>> Feel free to come to our IRC channel to discuss things (#pypy on
>> irc.freenode.net).
>>
>>
>
> Sorry for the big late/silence but i spent last days working/hacking on cpyext.
>
> I have managed to have a working uWSGI+pypy inverting the initial concept
> (not very useful, but this is a base for benchmark) :
>
> 1) i build the whole uWSGI stack core (so, without cpython plugin) as a shared library
> 2) i created a uwsgi_pypy c extension:
>
> http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi_pypy.c
>
> This extension export a bunch of functions, with the only useful one being uwsgi_pypy.init()
> This function calls the original main() of the uWSGI server, so effectively pypy lose the process control.
>
> 3) i have developed a very raw/minimal plugin for uWSGI that inspect a symbol in the process space.
>
> http://projects.unbit.it/uwsgi/browser/plugins/symcall/symcall_plugin.c
>
> This symbol is defined in the uwsgi_pypy.c module. So, this is the way uWSGI give control back to pypy at each request.
>
> This is an example WSGI-handler implementation (incomplete, it misses post handling, but can already execute werkzeug test app)
>
> http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi-pypy.py
>
> Well, running it and benchmarking showed a 10% improvement in response time, so i tried with a more complex app
> (fibonacci's sequence and some prime number generator) and ?damn it destroyed cpython :) it served the double of the connections in half of the time?.
>
> So, cpyext looks really in good shape, so (probably) the best way would be tweek it to have a libpypy.so.
>
> i will work on the current setup a bit more to have a full WSGI environment, then i will start investigating the original idea (but this time
> base on cpyext)
>
> Thanks for the amazing work

Cool stuff, thanks for letting us know :)

Cheers,
fijal

From hakan at debian.org  Sat Aug 20 10:32:11 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sat, 20 Aug 2011 10:32:11 +0200
Subject: [pypy-dev] Duplicated guards
Message-ID: <CAEkL+RiTBuwLX7U-xjgAUgY_BaTU5Gm=VPbxbaXN_gAC4FxqfA@mail.gmail.com>

Hi,
I've been experimenting a bit with heuristics such as "not retracing
loops with a lot of guards" or "not unrolling if the peel loop is not
much better than the preamble". In some situations is seem possible to
increase performace by such means (go: +37%, html5lib: +13%, rietveld:
+8%). However I think the real problem is that we duplicate guards and
that this leads to an explosion of bridges. I added
test_excessive_bridgeing to test_ajit on the jit-limit_peeling branch.
It is a simple example where the last part of the loop is traced 8
times producing 8 identical bridges.

Is there any chance of identifying copies of "the same guard" (in some
sens that considers both the address of the guard and the state of
virtuals) and when the second copy of the guards is about to be traced
we would just connect it to the already compiled bridge? Maybe by
producing a small header-bridge that moves things around and then
jumps to the already compiled bridge?

-- 
H?kan Ard?

From alex.gaynor at gmail.com  Sat Aug 20 10:37:34 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 20 Aug 2011 03:37:34 -0500
Subject: [pypy-dev] Duplicated guards
In-Reply-To: <CAEkL+RiTBuwLX7U-xjgAUgY_BaTU5Gm=VPbxbaXN_gAC4FxqfA@mail.gmail.com>
References: <CAEkL+RiTBuwLX7U-xjgAUgY_BaTU5Gm=VPbxbaXN_gAC4FxqfA@mail.gmail.com>
Message-ID: <CAFRnB2V2FbUDCZDMQpsCA+tjqPYXmF1X4fCvHcHqe_0zDPsrrQ@mail.gmail.com>

I believe mozilla worked on this issue.  if.you're interested in this
perhaps we should get you in contact with them.
On Aug 20, 2011 3:35 AM, "Hakan Ardo" <hakan at debian.org> wrote:
> Hi,
> I've been experimenting a bit with heuristics such as "not retracing
> loops with a lot of guards" or "not unrolling if the peel loop is not
> much better than the preamble". In some situations is seem possible to
> increase performace by such means (go: +37%, html5lib: +13%, rietveld:
> +8%). However I think the real problem is that we duplicate guards and
> that this leads to an explosion of bridges. I added
> test_excessive_bridgeing to test_ajit on the jit-limit_peeling branch.
> It is a simple example where the last part of the loop is traced 8
> times producing 8 identical bridges.
>
> Is there any chance of identifying copies of "the same guard" (in some
> sens that considers both the address of the guard and the state of
> virtuals) and when the second copy of the guards is about to be traced
> we would just connect it to the already compiled bridge? Maybe by
> producing a small header-bridge that moves things around and then
> jumps to the already compiled bridge?
>
> --
> H?kan Ard?
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/a063d202/attachment.html>

From holger at merlinux.eu  Sat Aug 20 11:14:21 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 20 Aug 2011 09:14:21 +0000
Subject: [pypy-dev] PyPy 1.6 released
In-Reply-To: <CAK5idxSahVm06N3WYb8TZ1_21gDd8Z85SUfpbrvZeE-nurU+Gw@mail.gmail.com>
References: <CAK5idxTOKR6u9KwSj3uBamZfq8i04ZJ_7sWHiTGZ3Y6uhNyhNg@mail.gmail.com>
	<20110819173532.GD12151@merlinux.eu>
	<CAK5idxTUj+e4wuAKE8jBXdE5kq_P6L1BsEVgdA-Nw7jKGeWPVA@mail.gmail.com>
	<20110820061508.GE12151@merlinux.eu>
	<CAK5idxSahVm06N3WYb8TZ1_21gDd8Z85SUfpbrvZeE-nurU+Gw@mail.gmail.com>
Message-ID: <20110820091421.GI12151@merlinux.eu>

On Sat, Aug 20, 2011 at 09:48 +0200, Maciej Fijalkowski wrote:
> On Sat, Aug 20, 2011 at 8:15 AM, holger krekel <holger at merlinux.eu> wrote:
> > On Fri, Aug 19, 2011 at 23:52 +0200, Maciej Fijalkowski wrote:
> >> On Fri, Aug 19, 2011 at 7:35 PM, holger krekel <holger at merlinux.eu> wrote:
> >> > Congrats also from here!
> >> >
> >> > On the plus side pytest's own test suite passes all tests, even some which
> >> > are marked as "expected-to-fail" with cpython-2.7. ?On the minus side,
> >> > the whole test run is still 2-3 times slower compared to cpython which is slightly worse than with pypy-1.5. of course there is not too much to JIT but
> >> > still a bit of a dissappointing result.
> >>
> >> Well, yes, but also we have no good answer to this. I think the reason
> >> why it got slower is because we compile functions now as well (which
> >> takes extra time). Test suites are hard I fear :(
> >
> > Maybe worth a small note because i guess test suites are often run to test pypy.
> 
> A small note where?

at places where people read it ... nevermind :)

> > FWIW if i run the test suite with --jit off it takes 117 seconds instead of
> > 72 with JIT (cpython takes about 25 seconds). ?So maybe one question is
> > why pypy-no-jit is already 4-5 times slower to begin with.
> 
> I guess a lot of things come into play - pypy's interpreter is
> generally slower than CPython's interpreter. We can attribute this to
> many things, but we were so far happy with "we pay 2x for abstraction
> layer". Maybe it's not good enough any more and we can actually come
> up with a better interpreter, who knows.

sure, i know about the 2x bit - it's not that i never developed core pypy 
or wrote various reports and bits about all these issues :)

I am talking of 4-5 times slower, though, and that seems a bit much, doesn't it?

holger

From arigo at tunes.org  Sat Aug 20 11:39:41 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 20 Aug 2011 11:39:41 +0200
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
	<CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>
Message-ID: <CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>

Hi Zooko,

On Fri, Aug 19, 2011 at 5:38 PM, Zooko O'Whielacronx <zooko at zooko.com> wrote:
> conservative method of determining if two classes are the same. For
> example: if they have identical Python source code and their
> superclasses are the same (in this sense).

This cannot be something you can rely on.  I can define Python classes
that have identical Python source code and superclasses, yet are
totally different.  For example,

    class A(object): pass
    A.foo = 42
    A1 = A
    class A(object): pass
    A.foo = 43
    A2 = A

So no, there is no way at all to make this idea work in general; you
can only do hacks that hopefully don't break too often.  I am not
interested :-)


A bient?t,

Armin.

From arigo at tunes.org  Sat Aug 20 11:43:54 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 20 Aug 2011 11:43:54 +0200
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
	<1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
Message-ID: <CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>

Hi Andy,

On Fri, Aug 19, 2011 at 5:48 PM, Andy <angelflow at yahoo.com> wrote:
> I remember reading in this list that PyPy-JIT would not work with greenlet
> because of how greentlet manipulated the C stack and there wasn't any easy solution.

No, I think you are confusing two topics.  The existing Stackless
PyPy, which does no C stack manipulation, doesn't work with the JIT
without a lot of careful work.  This solution is deprecated now.  The
new solution, which does C stack manipulation, works with the JIT out
of the box.  Both solutions are "implementation details" only; you get
the same tasklet/greenlet API to work with.

> So will PyPy JIT work with gevent which is based on libevent (and soon to be
> libev)? Any target release date?

Yes.  The next release is likely to be 1.6.1 soonish.


A bient?t,

Armin.

From arigo at tunes.org  Sat Aug 20 11:47:21 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 20 Aug 2011 11:47:21 +0200
Subject: [pypy-dev] PyGame for translatorshell on Lion
In-Reply-To: <CACOXMX7B3vZf-_eF54oS7Xjfc3E25brdPoFc=B8swd=8OBD2_w@mail.gmail.com>
References: <CACOXMX7B3vZf-_eF54oS7Xjfc3E25brdPoFc=B8swd=8OBD2_w@mail.gmail.com>
Message-ID: <CAMSv6X3+2BXzW4uMU+bd69k6381_7US4YuSVDd6f6JCZx4Urpw@mail.gmail.com>

Hi Mitchell,

On Sat, Aug 20, 2011 at 8:57 AM, Mitchell Hashimoto
<mitchell.hashimoto at gmail.com> wrote:
> One of the things I wanted to look at was the graph files generated by
> the shell. I installed PyGame and GraphViz and when I run a "t.view" on a
> Translation object, I get the following crazy thing.

I can only guess that it's a PyGame issue on OS/X Lion.  Try running
some PyGame demos and see if they work.


A bient?t,

Armin.

From arigo at tunes.org  Sat Aug 20 11:58:21 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 20 Aug 2011 11:58:21 +0200
Subject: [pypy-dev] List index out of range for many tests
In-Reply-To: <CACOXMX4niX4aSPKGBS6HgvpaG5OHGGKFQOtYzmfdiXX4X_Egdg@mail.gmail.com>
References: <CACOXMX4niX4aSPKGBS6HgvpaG5OHGGKFQOtYzmfdiXX4X_Egdg@mail.gmail.com>
Message-ID: <CAMSv6X0gFRK20MtAoD3HDiVQOoxvKY95XpOR1V39dZxo0dhHqA@mail.gmail.com>

Hi Mitchell,

On Sat, Aug 20, 2011 at 9:40 AM, Mitchell Hashimoto
<mitchell.hashimoto at gmail.com> wrote:
> I'm taking a look at some failing tests in the buildbot and trying to mirror
> them here on my local machine, but the tests are giving me the following
> error:

Note that you are running the lib-python tests on top of a
non-translated pypy, which is not what the buildbot does.  See
https://bitbucket.org/pypy/buildbot/raw/default/bot2/pypybuildbot --
in particular the file builds.py -- for more information about how
buildbot runs its processes.  Grep for "if lib_python:", for example.

About your error precisely: I don't know, I cannot reproduce it...  It
looks like .pyc file corruption, maybe?  Try to kill all .pyc files
and restart...


A bient?t,

Armin.

From vsapre80 at gmail.com  Sat Aug 20 15:53:25 2011
From: vsapre80 at gmail.com (Vishal)
Date: Sat, 20 Aug 2011 19:23:25 +0530
Subject: [pypy-dev] Anyone interested in a MIPS port...
Message-ID: <CACPguY_Oq4kTTcw6r3rqSP9vbLxQUL+hLvgYjEEPKvNPhKE=qg@mail.gmail.com>

Hello,

I would like to collect volunteers for a MIPS port of PyPy JIT.

MIPS is used across the embedded scene universally and a PyPy support for
MIPS would be wonderful.

So I have 2 questions (may seem like very nuby like...pardon me for that):

a) Does it make sense to have a MIPS port of the PyPy JIT.
b) How much hardware dependent is a JIT port?

Thanks and best regards,
Vishal Sapre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/078dc56d/attachment.html>

From santagada at gmail.com  Sat Aug 20 16:43:59 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 20 Aug 2011 11:43:59 -0300
Subject: [pypy-dev] regex (new re) and 1.6
In-Reply-To: <CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>
References: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>
	<CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>
Message-ID: <CA+HFc855-4Sk915eEFspPTWDf7vYkRLPVdqGtYZhbc8A24Zmwg@mail.gmail.com>

On Sat, Aug 20, 2011 at 4:51 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Aug 19, 2011 at 10:07 PM, Serhat Sevki Dincer
> <jfcgauss at gmail.com> wrote:
>> http://pypi.python.org/pypi/regex installs nicely on pypy-1.6 on my
>> Ubuntu 11.04 i386 but
>>
>>>>>> from regex import sub
>> Traceback (most recent call last):
>> ?File "<console>", line 1, in <module>
>> ?File "/var/Home/serhat/Program/pypy-1.6/site-packages/regex.py",
>> line 343, in <module>
>> ? ?from _regex_core import *
>> ?File "/var/Home/serhat/Program/pypy-1.6/site-packages/_regex_core.py",
>> line 21, in <module>
>> ? ?import _regex
>> ImportError: unable to load extension module
>> '/var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so':
>> /var/Home/serhat/Program/pypy-1.6/site-packages/_regex.pypy-16.so:
>> undefined symbol: Py_UNICODE_TOTITLE
>>
>> I dont see any (in)compatibility info on
>> https://bitbucket.org/pypy/compatibility/wiki/regex.
>>
>> Also
>> pypy-1.6$ grep -r 'Py_UNICODE_TO' include/
>> include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOLOWER(wchar_t arg0);
>> include/pypy_decl.h:PyAPI_FUNC(wchar_t) Py_UNICODE_TOUPPER(wchar_t arg0);
>>
>> TOTITLE is missing?
>> http://docs.python.org/release/2.7.2/library/stdtypes.html#str.title
>> is working on (unicode) strings in pypy. I dont get it, what do u
>> think is up?
>
> yes, TOTITLE is not implemented in cpyext, the cpython C compatibility
> layer. Btw - what are you trying to achieve? If you want to compare
> regex performance on pypy and cpython, it'll probably be much slower
> on pypy. The C extensions layer is just a compatibility one (it's also
> in beta).

This new regex module seems very interesting, seems a little to
bloated and complex, but the one interesting feature is that it
releases the gil. Does the one in pypy 1.6 does that?


-- 
Leonardo Santagada

From mitchell.hashimoto at gmail.com  Sat Aug 20 16:47:11 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sat, 20 Aug 2011 08:47:11 -0600
Subject: [pypy-dev] List index out of range for many tests
In-Reply-To: <CAMSv6X0gFRK20MtAoD3HDiVQOoxvKY95XpOR1V39dZxo0dhHqA@mail.gmail.com>
References: <CACOXMX4niX4aSPKGBS6HgvpaG5OHGGKFQOtYzmfdiXX4X_Egdg@mail.gmail.com>
	<CAMSv6X0gFRK20MtAoD3HDiVQOoxvKY95XpOR1V39dZxo0dhHqA@mail.gmail.com>
Message-ID: <CACOXMX6-SrwN=s1J2tQ_yLWBqk4Ja9sM2dcQuo62sva=GCJPkg@mail.gmail.com>

Armin,

On Sat, Aug 20, 2011 at 3:58 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Mitchell,
>
> On Sat, Aug 20, 2011 at 9:40 AM, Mitchell Hashimoto
> <mitchell.hashimoto at gmail.com> wrote:
> > I'm taking a look at some failing tests in the buildbot and trying to
> mirror
> > them here on my local machine, but the tests are giving me the following
> > error:
>
> Note that you are running the lib-python tests on top of a
> non-translated pypy, which is not what the buildbot does.  See
> https://bitbucket.org/pypy/buildbot/raw/default/bot2/pypybuildbot --
> in particular the file builds.py -- for more information about how
> buildbot runs its processes.  Grep for "if lib_python:", for example.
>
> About your error precisely: I don't know, I cannot reproduce it...  It
> looks like .pyc file corruption, maybe?  Try to kill all .pyc files
> and restart...
>
>
Thank you. I did get the tests properly running. I'm not sure whether it was
the pyc files or how I was running it, but I first removed all the previous
pyc files, looked at buildbot source, and ran the tests like so:

python pypy/test_all.py --pypy=pypy/translator/goal/pypy-c
lib-python/2.7/test/test_ctypes.py

This ran successfully and gave the same failures as the buildbot. Hurrah!

One question: The tests take quite awhile to run, even for a single file. Is
this normal?

And once again, thanks for helping with such a noob question, I really
appreciate it.

Best,
Mitchell


>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/e79e386d/attachment.html>

From hakan at debian.org  Sat Aug 20 18:04:43 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sat, 20 Aug 2011 18:04:43 +0200
Subject: [pypy-dev] Duplicated guards
In-Reply-To: <CAFRnB2V2FbUDCZDMQpsCA+tjqPYXmF1X4fCvHcHqe_0zDPsrrQ@mail.gmail.com>
References: <CAEkL+RiTBuwLX7U-xjgAUgY_BaTU5Gm=VPbxbaXN_gAC4FxqfA@mail.gmail.com>
	<CAFRnB2V2FbUDCZDMQpsCA+tjqPYXmF1X4fCvHcHqe_0zDPsrrQ@mail.gmail.com>
Message-ID: <CAEkL+RjdMQLLq3=oGS9rkHuVuXSjA5EABLoB20bP=zf4mFcyKw@mail.gmail.com>

On 8/20/11, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> I believe mozilla worked on this issue.  if.you're interested in this
> perhaps we should get you in contact with them.

Good idea. Do you have some contact?

Does the following approach make sens:

When a guard is traced we trace to the next jit_merge_point() only,
and construct a small bridge there. That would be to complete the
current python bytecode, right? From that point we then produce
something similar to our current loops. The difference would be that
it does not end with a jump to itself but to somewhere else. It would
probably not need a preamble at all (unsure) but it would have a
peeled loop with a virtual state that matches the state needed by the
bridge.

Later, when "the same" guard in a different location is traced, it
will end up at the same jit_merge_point() where it discovers a
compiled loop and if it's virtual state matches it will jump to it and
if not the retrace feature will produce a new version of it that
matches the current bridge.

This way the duplicated part of the bridges will be very small (less
than a bytecode). Hmm, maybe it's actually possible to unify the
notions of perambles, peeled loops and bridges into traces attached to
a greenkey with a specific virtual state.

> On Aug 20, 2011 3:35 AM, "Hakan Ardo" <hakan at debian.org> wrote:
>> Hi,
>> I've been experimenting a bit with heuristics such as "not retracing
>> loops with a lot of guards" or "not unrolling if the peel loop is not
>> much better than the preamble". In some situations is seem possible to
>> increase performace by such means (go: +37%, html5lib: +13%, rietveld:
>> +8%). However I think the real problem is that we duplicate guards and
>> that this leads to an explosion of bridges. I added
>> test_excessive_bridgeing to test_ajit on the jit-limit_peeling branch.
>> It is a simple example where the last part of the loop is traced 8
>> times producing 8 identical bridges.
>>
>> Is there any chance of identifying copies of "the same guard" (in some
>> sens that considers both the address of the guard and the state of
>> virtuals) and when the second copy of the guards is about to be traced
>> we would just connect it to the already compiled bridge? Maybe by
>> producing a small header-bridge that moves things around and then
>> jumps to the already compiled bridge?
>>
>> --
>> H?kan Ard?
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>


-- 
H?kan Ard?

From amauryfa at gmail.com  Sat Aug 20 21:40:25 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 20 Aug 2011 21:40:25 +0200
Subject: [pypy-dev] regex (new re) and 1.6
In-Reply-To: <CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>
References: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>
	<CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>
Message-ID: <CAGmFidba=dw7QYTK0WY3XCRWtuj=rw4HNpy-VJELEmCdYOKUow@mail.gmail.com>

2011/8/20 Maciej Fijalkowski <fijall at gmail.com>

> > TOTITLE is missing?
> > http://docs.python.org/release/2.7.2/library/stdtypes.html#str.title
> > is working on (unicode) strings in pypy. I dont get it, what do u
> > think is up?
>
> yes, TOTITLE is not implemented in cpyext, the cpython C compatibility
> layer. Btw - what are you trying to achieve? If you want to compare
> regex performance on pypy and cpython, it'll probably be much slower
> on pypy. The C extensions layer is just a compatibility one (it's also
> in beta).


Py_UNICODE_TOTITLE and another missing function (PySet_Contains)
are now implemented in "default" branch.
The regex module compiles and works at least for the basic samples I tried.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/28ec5c31/attachment-0001.html>

From jfcgauss at gmail.com  Sat Aug 20 23:42:24 2011
From: jfcgauss at gmail.com (Serhat Sevki Dincer)
Date: Sun, 21 Aug 2011 00:42:24 +0300
Subject: [pypy-dev] regex (new re) and 1.6
In-Reply-To: <CAGmFidba=dw7QYTK0WY3XCRWtuj=rw4HNpy-VJELEmCdYOKUow@mail.gmail.com>
References: <CAPqC6xTsZNPoxDw0TEuOpz=CJNB-hZBNB4jQBi8b+VgVeAYBAg@mail.gmail.com>
	<CAK5idxQVB-FwZVZ9f6ri+J7q9uJ_sUJy91edfr6-S-9a0gtyBg@mail.gmail.com>
	<CAGmFidba=dw7QYTK0WY3XCRWtuj=rw4HNpy-VJELEmCdYOKUow@mail.gmail.com>
Message-ID: <CAPqC6xR-kUb-Hgvz8cid+_OKEtwVt=L4Zf=XSzevpQGrJNT9xg@mail.gmail.com>

>> > TOTITLE is missing?
>> > http://docs.python.org/release/2.7.2/library/stdtypes.html#str.title
>> > is working on (unicode) strings in pypy. I dont get it, what do u
>> > think is up?
>>
>> yes, TOTITLE is not implemented in cpyext, the cpython C compatibility
>> layer. Btw - what are you trying to achieve? If you want to compare
>> regex performance on pypy and cpython, it'll probably be much slower
>> on pypy. The C extensions layer is just a compatibility one (it's also
>> in beta).
>
> Py_UNICODE_TOTITLE and another missing function (PySet_Contains)
> are now implemented in "default" branch.
> The regex module compiles and works at least for the basic samples I tried.

Thank you..

From angelflow at yahoo.com  Sun Aug 21 08:31:42 2011
From: angelflow at yahoo.com (Andy)
Date: Sat, 20 Aug 2011 23:31:42 -0700 (PDT)
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
	<1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
	<CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>
Message-ID: <1313908302.18401.YahooMailNeo@web111306.mail.gq1.yahoo.com>

Thanks Armin.

I dug around and found an email from you titled "Stacklets". Is that the new solution?

If I understand correctly, in this solution a greenlet could only be suspended and resumed once. But what if my greenlet needs to call multiple Web services or that it's a long running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that?


________________________________
From: Armin Rigo <arigo at tunes.org>
To: Andy <angelflow at yahoo.com>
Cc: Jacob Biesinger <jake.biesinger at gmail.com>; "pypy-dev at python.org" <pypy-dev at python.org>
Sent: Saturday, August 20, 2011 5:43 AM
Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops

Hi Andy,

On Fri, Aug 19, 2011 at 5:48 PM, Andy <angelflow at yahoo.com> wrote:
> I remember reading in this list that PyPy-JIT would not work with greenlet
> because of how greentlet manipulated the C stack and there wasn't any easy solution.

No, I think you are confusing two topics.? The existing Stackless
PyPy, which does no C stack manipulation, doesn't work with the JIT
without a lot of careful work.? This solution is deprecated now.? The
new solution, which does C stack manipulation, works with the JIT out
of the box.? Both solutions are "implementation details" only; you get
the same tasklet/greenlet API to work with.

> So will PyPy JIT work with gevent which is based on libevent (and soon to be
> libev)? Any target release date?

Yes.? The next release is likely to be 1.6.1 soonish.


A bient?t,

Armin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110820/683a4c5c/attachment.html>

From boemmels at web.de  Sun Aug 21 15:15:43 2011
From: boemmels at web.de (Juergen Boemmels)
Date: Sun, 21 Aug 2011 15:15:43 +0200 (CEST)
Subject: [pypy-dev] status of scheme implementation
Message-ID: <997342288.4432887.1313932543265.JavaMail.fmail@mwmweb085>

Hi,

I just found the scheme implementation
https://bitbucket.org/pypy/lang-scheme/
but there is no history besides the initial import.

Is this maintained? I.e. is it usful to send patches for improving this? And is this the right mailing list for asking
questions about lang-scheme?

Anyway, I played a little with it and found one annoyance: The translated parser does not report the position
of the parse error. There was something prepared, but commented out (I think because it didn't translate).
By adding an assertion to
pypy/rlib/parsing/makepackrat.py
I was able to get scheme translated including the error location. This greatly simplifies debugging ;-)

I attached the two patches (one to pypy/pypy and one to pypy/lang-scheme)

Any comments
j?rgen
___________________________________________________________
Schon geh?rt? WEB.DE hat einen genialen Phishing-Filter in die
Toolbar eingebaut! http://produkte.web.de/go/toolbar
-------------- next part --------------
A non-text attachment was scrubbed...
Name: get_line_column.diff
Type: application/octet-stream
Size: 694 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/a0735c27/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error_reporting.diff
Type: application/octet-stream
Size: 709 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/a0735c27/attachment-0001.obj>

From naylor.b.david at gmail.com  Sun Aug 21 15:57:47 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Sun, 21 Aug 2011 15:57:47 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <CAMSv6X0t6d0xYMrVM_68ZKnhdsFEp7CZTSOCRc-s0DVQ_j-rjw@mail.gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<CAKp1Nw7H1=M4fQQSRxGG+f2rWSP0GGGLFu06Y_LcEx4GYozs9w@mail.gmail.com>
	<CAMSv6X0t6d0xYMrVM_68ZKnhdsFEp7CZTSOCRc-s0DVQ_j-rjw@mail.gmail.com>
Message-ID: <201108211557.51162.naylor.b.david@gmail.com>

On Thursday, 18 August 2011 14:45:16 Armin Rigo wrote:
> Hi Gabriel,
> 
> On Wed, Aug 17, 2011 at 10:42 PM, Gabriel Lavoie <glavoie at gmail.com> wrote:
> > Hum... It seems it's the end of my FreeBSD buildslave. :( I don't know
> > when it time the memory requirement jumped to over 4GB (Armin told me it
> > was now around 4.5GB for 64bits).
> 
> Actually I found out that with proper settings I can run a 64-bit
> "translate.py -Ojit" in 3.0GB.  I didn't measure the impact on time
> --- I expect it to take "a bit longer" to run, but of course much less
> than if it's swapping.  Maybe we need to tweak a bit our default
> parameters...
> 
> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ./translate.py -Ojit

FYI, the above options increases the build time by ~30% (on my system at-
least).  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/c63a4d32/attachment.pgp>

From mitchell.hashimoto at gmail.com  Sun Aug 21 18:25:27 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sun, 21 Aug 2011 10:25:27 -0600
Subject: [pypy-dev] Question about "Completing 'os' module" (issue 833)
Message-ID: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>

Hi all,

I noticed the 'os' module is incomplete, and I'd like to help complete this.
CPython does this by simply having these methods available on "posix" "nt"
"os2" etc. and the "os" module imports those. It appears that PyPy does the
same thing. I was able to successfully add 'getlogin' as practice, but I
wanted to post here before going further. Some questions below:

1.) Should I mimic CPython and add the functionality to the OS-specific
modules?
2.) I don't have a Windows computer on hand. What is the standard practice
for implementing some stdlib for one OS but not the other? Would PyPy accept
this temporarily?
3.) There are many missing methods, to simplify implementation time and the
patch, would it be okay to submit a patch for each stdlib method, so that
this was built up over time?

I'd love to do this "right" the first time so I can contribute it back, so
this feedback would be greatly appreciated.

Best,
Mitchell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/d69c4a8d/attachment.html>

From zooko at zooko.com  Sun Aug 21 19:44:37 2011
From: zooko at zooko.com (Zooko O'Whielacronx)
Date: Sun, 21 Aug 2011 11:44:37 -0600
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
	<CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>
	<CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>
Message-ID: <CANdZDc7tZUfg1+WQty6y9xhfmQ4NfkRXiCeKQjdVSXnN+byJ1g@mail.gmail.com>

Dear Armin Rigo:

Thank you for your reply.

On Sat, Aug 20, 2011 at 3:39 AM, Armin Rigo <arigo at tunes.org> wrote:
>
> So no, there is no way at all to make this idea work in general; you
> can only do hacks that hopefully don't break too often. ?I am not
> interested :-)

I also wouldn't be interested in techniques that sometimes resulted in
incorrect behavior.

But I still don't understand why you say this can't work in general.
If you don't mind explaining it a little more I would appreciate it.

The behavior of a program is a deterministic function of its source
code, right? So if the source code (*all* of it -- everything that
gets imported and executed at any point during the run of the program)
is identical, then the behavior will be identical, and then re-using
compiled code from previous runs would always yield a correct result.
Am I still on track so far?

So then suppose you have a copy of the source code of the program
(*all* of it), and you make one change, such as changing a constant
from 42 to 43. Now can't you still safely re-use *some* of the
previously compiled code to run the new program? Or perhaps not. Since
we are using JIT techniques to take advantage of as much knowledge as
we can, the we may use the knowledge that the constant is 42 to
compile all of the rest of the program differently, resulting in
compiled code none of which can be safely reused.

That is an interesting prospect to consider. It underscores how
powerful (at least potentially) JITting can be.

But, I still think there are probably some boundaries where you can
know for sure that the change cannot have invalidated some of the
compiled code. For example, what about code that you compiled before
you imported the module that had the changed constant? That code must
surely still be valid. Are there are other such boundaries that allow
the safe re-use of compiled code?

Thank you for your time in discussing this idea, and also by the way
thank you for contributing to the PyPy project!

Regards,

Zooko

From zooko at zooko.com  Sun Aug 21 19:49:22 2011
From: zooko at zooko.com (Zooko O'Whielacronx)
Date: Sun, 21 Aug 2011 11:49:22 -0600
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CANdZDc7tZUfg1+WQty6y9xhfmQ4NfkRXiCeKQjdVSXnN+byJ1g@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>
	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>
	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>
	<CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>
	<CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>
	<CANdZDc7tZUfg1+WQty6y9xhfmQ4NfkRXiCeKQjdVSXnN+byJ1g@mail.gmail.com>
Message-ID: <CANdZDc5q0gh1ZqDMm913bsqaCuAsBuG7SYX++D8JQG0adoAOSQ@mail.gmail.com>

[following-up to my own post]

On Sun, Aug 21, 2011 at 11:44 AM, Zooko O'Whielacronx <zooko at zooko.com> wrote:
>
> The behavior of a program is a deterministic function of its source
> code, right? So if the source code (*all* of it -- everything that
> gets imported and executed at any point during the run of the program)
> is identical, then the behavior will be identical, and then re-using
> compiled code from previous runs would always yield a correct result.
> Am I still on track so far?

Uh-oh, I think I may have already gone off track here in the first
paragraph. What if you use information from the runtime *inputs* --
not imported code but inputted data -- to change how you compile the
code? Then even if *all* of the source code is exactly the same,
re-using the compiled code from the previous run may generate
incorrect results. Is that right?

Solving that one is still, in my opinion, theoretically possible, but
now in addition to the formidable challenge of finding safe boundaries
where the knock-on effects of changed source code don't cross, we
would *also* have to find boundaries of the effects of changed data
inputs. This is, of course, one of those grand challenges that the
programming languages researchers have been knocking their collective
heads against for at least thirty years.

Therefore I expect the PyPy project will have to work on it for at
least a year or two more before you solve it. ;-)

Regards,

Zooko

From cfbolz at gmx.de  Sun Aug 21 21:32:18 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 21 Aug 2011 21:32:18 +0200
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <CANdZDc5q0gh1ZqDMm913bsqaCuAsBuG7SYX++D8JQG0adoAOSQ@mail.gmail.com>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>	<CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>	<CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>	<CANdZDc7tZUfg1+WQty6y9xhfmQ4NfkRXiCeKQjdVSXnN+byJ1g@mail.gmail.com>
	<CANdZDc5q0gh1ZqDMm913bsqaCuAsBuG7SYX++D8JQG0adoAOSQ@mail.gmail.com>
Message-ID: <4E515D42.1050107@gmx.de>

On 08/21/2011 07:49 PM, Zooko O'Whielacronx wrote:
> [following-up to my own post]
>
> On Sun, Aug 21, 2011 at 11:44 AM, Zooko O'Whielacronx<zooko at zooko.com>  wrote:
>>
>> The behavior of a program is a deterministic function of its source
>> code, right? So if the source code (*all* of it -- everything that
>> gets imported and executed at any point during the run of the program)
>> is identical, then the behavior will be identical, and then re-using
>> compiled code from previous runs would always yield a correct result.
>> Am I still on track so far?
>
> Uh-oh, I think I may have already gone off track here in the first
> paragraph. What if you use information from the runtime *inputs* --
> not imported code but inputted data -- to change how you compile the
> code? Then even if *all* of the source code is exactly the same,
> re-using the compiled code from the previous run may generate
> incorrect results. Is that right?

class A(object):
     if time.time() % 2 == 0:
         def f(self):
             return 1
     else:
         def f(self):
             return 2

:-)


Solving the problem of persisting source code is *seriously* hard, and I 
agree with Armin, I'm not going to invest any time in it.


Carl Friedrich

From amauryfa at gmail.com  Sun Aug 21 22:20:03 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 21 Aug 2011 22:20:03 +0200
Subject: [pypy-dev] Question about "Completing 'os' module" (issue 833)
In-Reply-To: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>
References: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>
Message-ID: <CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>

Hello,

2011/8/21 Mitchell Hashimoto <mitchell.hashimoto at gmail.com>

> I noticed the 'os' module is incomplete, and I'd like to help complete
> this.
>

You are very welcome!


> CPython does this by simply having these methods available on "posix" "nt"
> "os2" etc. and the "os" module imports those. It appears that PyPy does the
> same thing. I was able to successfully add 'getlogin' as practice, but I
> wanted to post here before going further. Some questions below:
>
> 1.) Should I mimic CPython and add the functionality to the OS-specific
> modules?
>

Yes we should mimic CPython: fortunately these modules have different names
but share the same source file.
With CPython it's Modules/posixmodule.c, with PyPy it's in
pypy/module/posix.


> 2.) I don't have a Windows computer on hand. What is the standard practice
> for implementing some stdlib for one OS but not the other? Would PyPy accept
> this temporarily?
>

Yes, no problem. In this case, I think it's best to let the test fail on
Windows so that someone may notice and fix it.


> 3.) There are many missing methods, to simplify implementation time and the
> patch, would it be okay to submit a patch for each stdlib method, so that
> this was built up over time?
>

Yes, smaller patches are easier to read and merge. I'd be happy to review
and commit them.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/28ac646e/attachment.html>

From mitchell.hashimoto at gmail.com  Sun Aug 21 23:04:43 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sun, 21 Aug 2011 15:04:43 -0600
Subject: [pypy-dev] Question about "Completing 'os' module" (issue 833)
In-Reply-To: <CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>
References: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>
	<CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>
Message-ID: <CACOXMX7uGfCAVyCF8xqBqGugWkaUiVYXB-pr71iSQsNkdzwgZw@mail.gmail.com>

Amaury,

On Sun, Aug 21, 2011 at 2:20 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> Hello,
>
> 2011/8/21 Mitchell Hashimoto <mitchell.hashimoto at gmail.com>
>
>> I noticed the 'os' module is incomplete, and I'd like to help complete
>> this.
>>
>
> You are very welcome!
>
>
>> CPython does this by simply having these methods available on "posix" "nt"
>> "os2" etc. and the "os" module imports those. It appears that PyPy does the
>> same thing. I was able to successfully add 'getlogin' as practice, but I
>> wanted to post here before going further. Some questions below:
>>
>> 1.) Should I mimic CPython and add the functionality to the OS-specific
>> modules?
>>
>
> Yes we should mimic CPython: fortunately these modules have different names
> but share the same source file.
> With CPython it's Modules/posixmodule.c, with PyPy it's in
> pypy/module/posix.
>
>
>> 2.) I don't have a Windows computer on hand. What is the standard practice
>> for implementing some stdlib for one OS but not the other? Would PyPy accept
>> this temporarily?
>>
>
> Yes, no problem. In this case, I think it's best to let the test fail on
> Windows so that someone may notice and fix it.
>
>
>> 3.) There are many missing methods, to simplify implementation time and
>> the patch, would it be okay to submit a patch for each stdlib method, so
>> that this was built up over time?
>>
>
>  Yes, smaller patches are easier to read and merge. I'd be happy to review
> and commit them.
>

Great! Thanks for the response :)

I'll post the patch in an issue for each stdlib method as I make it, and
I'll just post here to ask for review.

Thanks,
Mitchell


>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/8466fa1e/attachment.html>

From mitchell.hashimoto at gmail.com  Mon Aug 22 07:02:38 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sun, 21 Aug 2011 23:02:38 -0600
Subject: [pypy-dev] How to test code in pypy.rpython?
Message-ID: <CACOXMX4=xEcJtpdAVxp-Bnc2R7hLvz2-bqkQr-NgJ_bSMAZbLw@mail.gmail.com>

Hello!

I've been working on some 'os' standard library methods, which requires some
interp-level code in pypy.rpython.module.ll_os.

I noticed that while in the "py.py" interpreter that the methods I've been
adding to "os" have been using CPython's library. I've written unit tests
which appear to actually test the ll_os code and it seems to work, and I can
also always drop back into the interp-level repl and do a
"extregistry.lookup(fn).lltypeimpl" trick to get the method and it seems to
work there as well.

But it would still be great to see the whole thing work without having to
translate all of PyPy. Is this possible?

I've tried "py.py --nofaking" and that doesn't work either for what I'm
doing.

Thanks,
Mitchell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110821/a6312afe/attachment.html>

From amauryfa at gmail.com  Mon Aug 22 08:13:37 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 22 Aug 2011 08:13:37 +0200
Subject: [pypy-dev] How to test code in pypy.rpython?
In-Reply-To: <CACOXMX4=xEcJtpdAVxp-Bnc2R7hLvz2-bqkQr-NgJ_bSMAZbLw@mail.gmail.com>
References: <CACOXMX4=xEcJtpdAVxp-Bnc2R7hLvz2-bqkQr-NgJ_bSMAZbLw@mail.gmail.com>
Message-ID: <CAGmFidYFuQ+1cHD4LGTBEY08AHQ79JF4GjjnrZ-KLwD-KQ7h0Q@mail.gmail.com>

Hi,

2011/8/22 Mitchell Hashimoto <mitchell.hashimoto at gmail.com>

> I've been working on some 'os' standard library methods, which requires
> some interp-level code in pypy.rpython.module.ll_os.
>
> I noticed that while in the "py.py" interpreter that the methods I've been
> adding to "os" have been using CPython's library. I've written unit tests
> which appear to actually test the ll_os code and it seems to work, and I can
> also always drop back into the interp-level repl and do a
> "extregistry.lookup(fn).lltypeimpl" trick to get the method and it seems to
> work there as well.
>
> But it would still be great to see the whole thing work without having to
> translate all of PyPy. Is this possible?
>

Sure, you must write unit tests anyway.
Add your ones in pypy/rpython/module/test/test_ll_os.py,
there are already many examples there.
(the magic function is getllimpl(), which returns the low-level function)

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110822/744d1a8e/attachment.html>

From arigo at tunes.org  Mon Aug 22 11:45:11 2011
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 22 Aug 2011 11:45:11 +0200
Subject: [pypy-dev] How to test code in pypy.rpython?
In-Reply-To: <CAGmFidYFuQ+1cHD4LGTBEY08AHQ79JF4GjjnrZ-KLwD-KQ7h0Q@mail.gmail.com>
References: <CACOXMX4=xEcJtpdAVxp-Bnc2R7hLvz2-bqkQr-NgJ_bSMAZbLw@mail.gmail.com>
	<CAGmFidYFuQ+1cHD4LGTBEY08AHQ79JF4GjjnrZ-KLwD-KQ7h0Q@mail.gmail.com>
Message-ID: <CAMSv6X3FHseUB4KKFk9uEEcg3KQ7DbR3iUzNZFB_aQ8zeTVtXQ@mail.gmail.com>

Hi Mitchell,

On Mon, Aug 22, 2011 at 8:13 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
>> But it would still be great to see the whole thing work without having to
>> translate all of PyPy. Is this possible?
>
> Sure, you must write unit tests anyway.

To complete this answer, what you should test is on the one hand
pypy/module/posix, and on the other hand pypy/rpython/module.  If both
are working, then the combination in a full PyPy will likely work too.
 It is not possible to write integration tests in this case: you need
a full translation.  But it is, as I said, likely to work if both
parts work, and it will be tested nightly by using the translated
pypy-c to run "pypy-c pytest.py pypy/module/posix/test -A".

The -A flag is used to run AppTests directly on top of the current
interpreter.  In addition to the use case above, you can also directly
check that CPython 2.7 behaves really like your tests say it does, by
running "python2.7 pytest.py pypy/module/posix/test -A".


A bient?t,

Armin.

From mitchell.hashimoto at gmail.com  Mon Aug 22 13:53:20 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Mon, 22 Aug 2011 05:53:20 -0600
Subject: [pypy-dev] Question about "Completing 'os' module" (issue 833)
In-Reply-To: <CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>
References: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>
	<CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>
Message-ID: <CACOXMX5WvcCAr6oF+a_gzBWkPmZaygTyYirx5ax0PNDE0uXFwg@mail.gmail.com>

Amaury,

I've implemented one method (getlogin) and have created an issue + patch for
it:

https://bugs.pypy.org/issue841

Best,
Mitchell

On Sun, Aug 21, 2011 at 2:20 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> Hello,
>
> 2011/8/21 Mitchell Hashimoto <mitchell.hashimoto at gmail.com>
>
>> I noticed the 'os' module is incomplete, and I'd like to help complete
>> this.
>>
>
> You are very welcome!
>
>
>> CPython does this by simply having these methods available on "posix" "nt"
>> "os2" etc. and the "os" module imports those. It appears that PyPy does the
>> same thing. I was able to successfully add 'getlogin' as practice, but I
>> wanted to post here before going further. Some questions below:
>>
>> 1.) Should I mimic CPython and add the functionality to the OS-specific
>> modules?
>>
>
> Yes we should mimic CPython: fortunately these modules have different names
> but share the same source file.
> With CPython it's Modules/posixmodule.c, with PyPy it's in
> pypy/module/posix.
>
>
>> 2.) I don't have a Windows computer on hand. What is the standard practice
>> for implementing some stdlib for one OS but not the other? Would PyPy accept
>> this temporarily?
>>
>
> Yes, no problem. In this case, I think it's best to let the test fail on
> Windows so that someone may notice and fix it.
>
>
>> 3.) There are many missing methods, to simplify implementation time and
>> the patch, would it be okay to submit a patch for each stdlib method, so
>> that this was built up over time?
>>
>
>  Yes, smaller patches are easier to read and merge. I'd be happy to review
> and commit them.
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110822/d99fb270/attachment.html>

From dje.gcc at gmail.com  Mon Aug 22 22:07:48 2011
From: dje.gcc at gmail.com (David Edelsohn)
Date: Mon, 22 Aug 2011 16:07:48 -0400
Subject: [pypy-dev] test_ppc successful on PPC64
Message-ID: <CAGWvny=198uokrV_f3RmB4pM+CnJUHVq-05zrqq2e49m5EnAag@mail.gmail.com>

I fixed test_call_function on PPC64 so all test_ppc.py tests succeed on PPC64.

Now on to the more general tests.

- David

From angelflow at yahoo.com  Tue Aug 23 05:40:25 2011
From: angelflow at yahoo.com (Andy)
Date: Mon, 22 Aug 2011 20:40:25 -0700 (PDT)
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <1313908302.18401.YahooMailNeo@web111306.mail.gq1.yahoo.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
	<1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
	<CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>
	<1313908302.18401.YahooMailNeo@web111306.mail.gq1.yahoo.com>
Message-ID: <1314070825.41978.YahooMailNeo@web111314.mail.gq1.yahoo.com>

Hi,

Anyone who can answer my previous question - "what if my greenlet needs to call multiple Web services or that it's a long running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that?"

Thanks.


________________________________
From: Andy <angelflow at yahoo.com>
To: Armin Rigo <arigo at tunes.org>
Cc: "pypy-dev at python.org" <pypy-dev at python.org>
Sent: Sunday, August 21, 2011 2:31 AM
Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops


Thanks Armin.

I dug around and found an email from you titled "Stacklets". Is that the new solution?

If I understand correctly, in this solution a greenlet could only be suspended and resumed once. But what if my greenlet needs to call multiple Web services or that it's a long running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that?


________________________________
From: Armin Rigo <arigo at tunes.org>
To: Andy <angelflow at yahoo.com>
Cc: Jacob Biesinger <jake.biesinger at gmail.com>; "pypy-dev at python.org" <pypy-dev at python.org>
Sent: Saturday, August 20, 2011 5:43 AM
Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops

Hi Andy,

On Fri, Aug 19, 2011 at 5:48 PM, Andy <angelflow at yahoo.com> wrote:
> I remember reading in this list that PyPy-JIT would not work with greenlet
> because of how greentlet manipulated the C stack and there wasn't any easy solution.

No, I think you are confusing two topics.? The existing Stackless
PyPy, which does no C stack manipulation, doesn't work with the
 JIT
without a lot of careful work.? This solution is deprecated now.? The
new solution, which does C stack manipulation, works with the JIT out
of the box.? Both solutions are "implementation details" only; you get
the same tasklet/greenlet API to work with.

> So will PyPy JIT work with gevent which is based on libevent (and soon to be
> libev)? Any target release date?

Yes.? The next release is likely to be 1.6.1 soonish.


A bient?t,

Armin.



_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110822/9f3ca488/attachment.html>

From arigo at tunes.org  Tue Aug 23 14:09:56 2011
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 23 Aug 2011 14:09:56 +0200
Subject: [pypy-dev] 1.6 status report
In-Reply-To: <201108211557.51162.naylor.b.david@gmail.com>
References: <CAK5idxTt0fCPZP02wO=e049SG3n6-WkgmCfvEKKxMxvsrmbswQ@mail.gmail.com>
	<CAKp1Nw7H1=M4fQQSRxGG+f2rWSP0GGGLFu06Y_LcEx4GYozs9w@mail.gmail.com>
	<CAMSv6X0t6d0xYMrVM_68ZKnhdsFEp7CZTSOCRc-s0DVQ_j-rjw@mail.gmail.com>
	<201108211557.51162.naylor.b.david@gmail.com>
Message-ID: <CAMSv6X1f1nN+dPxqAaAMsTGyBg31oNrvHyb-AzMbAvW+RY9UUQ@mail.gmail.com>

Hi David,

On Sun, Aug 21, 2011 at 3:57 PM, David Naylor <naylor.b.david at gmail.com> wrote:
>> PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ./translate.py -Ojit
>
> FYI, the above options increases the build time by ~30% (on my system at-
> least).

Thank you.  That's what I feared.  We need to carefully find a
balance...  Of course 30% slower is still incredibly faster than
constantly swapping, but if there is enough RAM we should go with the
default...


A bient?t,

Armin.

From yselivanov.ml at gmail.com  Tue Aug 23 19:28:56 2011
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 23 Aug 2011 13:28:56 -0400
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <1314070825.41978.YahooMailNeo@web111314.mail.gq1.yahoo.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
	<1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
	<CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>
	<1313908302.18401.YahooMailNeo@web111306.mail.gq1.yahoo.com>
	<1314070825.41978.YahooMailNeo@web111314.mail.gq1.yahoo.com>
Message-ID: <F8C84977-368E-4A41-BD5F-96E997CF6D97@gmail.com>

If you read that Armin's email carefully, you notice that he talks about a low-level primitive called "stacklets", which have some limitations, but are not intended for a regular use.  Greenlets will be implemented on top of them.  So anything that was possible to do before will be supported in the new version.

On 2011-08-22, at 11:40 PM, Andy wrote:

> Hi,
> 
> Anyone who can answer my previous question - "what if my greenlet needs to call multiple Web services or that it's a long running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that?"
> 
> Thanks.
> 
> From: Andy <angelflow at yahoo.com>
> To: Armin Rigo <arigo at tunes.org>
> Cc: "pypy-dev at python.org" <pypy-dev at python.org>
> Sent: Sunday, August 21, 2011 2:31 AM
> Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops
> 
> Thanks Armin.
> 
> I dug around and found an email from you titled "Stacklets". Is that the new solution?
> 
> If I understand correctly, in this solution a greenlet could only be suspended and resumed once. But what if my greenlet needs to call multiple Web services or that it's a long running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that?
> 
> From: Armin Rigo <arigo at tunes.org>
> To: Andy <angelflow at yahoo.com>
> Cc: Jacob Biesinger <jake.biesinger at gmail.com>; "pypy-dev at python.org" <pypy-dev at python.org>
> Sent: Saturday, August 20, 2011 5:43 AM
> Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops
> 
> Hi Andy,
> 
> On Fri, Aug 19, 2011 at 5:48 PM, Andy <angelflow at yahoo.com> wrote:
> > I remember reading in this list that PyPy-JIT would not work with greenlet
> > because of how greentlet manipulated the C stack and there wasn't any easy solution.
> 
> No, I think you are confusing two topics.  The existing Stackless
> PyPy, which does no C stack manipulation, doesn't work with the JIT
> without a lot of careful work.  This solution is deprecated now.  The
> new solution, which does C stack manipulation, works with the JIT out
> of the box.  Both solutions are "implementation details" only; you get
> the same tasklet/greenlet API to work with.
> 
> > So will PyPy JIT work with gevent which is based on libevent (and soon to be
> > libev)? Any target release date?
> 
> Yes.  The next release is likely to be 1.6.1 soonish.
> 
> 
> A bient?t,
> 
> Armin.
> 
> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From lac at openend.se  Wed Aug 24 11:14:21 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 24 Aug 2011 11:14:21 +0200
Subject: [pypy-dev] from the psf board public mailing list.
Message-ID: <201108240914.p7O9ELHo026537@theraft.openend.se>

My comments follow.

------- Forwarded Message

Return-Path: psf-board-public-bounces+lac=openend.se at python.org
Delivery-Date: Wed Aug 24 10:58:13 2011


Steve Holden wrote:
> Guido forward this to me, but I think it should be developers getting involved. 

> Anyway, does anyone on this list have any strong feelings about applying for 
> funding? I am not sure security is Python's forte, but many people are doing
> things with it that I am unaware of.
>
> regards
> Steve

> Begin forwarded message:

>> *From: *Guido van Rossum <guido at python.org <mailto:guido at python.org>>
>> *Date: *August 23, 2011 4:31:53 PM PDT
>> *To: *"Steve Holden, Chairman, PSF" <chairman at python.org =

>> <mailto:chairman at python.org>>
>> *Subject: **Fwd: Get funding for your OSS project from Rapid7*
>>
>> FYI. I don't know who this is, and he doesn't seem to know me either. :-)
>>
>> ---------- Forwarded message ----------
>> From: *Christian Kirsch* <Christian_Kirsch at rapid7.com =

>> <mailto:Christian_Kirsch at rapid7.com>>
>> Date: Tue, Aug 23, 2011 at 4:15 PM
>> Subject: Get funding for your OSS project from Rapid7
>> To: "guido at python.org <mailto:guido at python.org>" <guido at python.org>

>> Hi Guido van,
>>
>> __ __
>>
>> I hope you're doing great! I was referred to you as the main contact for Python.
>> Rapid7 is dedicating $100,000 to support open source projects in the security

>> space in 2012 in a program we're calling the Magnificent7. We are looking for 

>> open source projects that bring value to the information security ecosystem by

>> taking an innovative approach to addressing security challenges. Up to seven

>> chosen projects will receive funding in 3 phases: an initial outlay, and then 

>> two further sums tied to the delivery of pre-agreed milestones.
>>
>> __ __
>>
>> Submissions will be judged on several criteria, including a working 
>> demonstration of the software and the caliber of project roadmap. To be 
>> eligible, the person participating must have the ability to represent the
>> project in question, including updating the web site and doing outbound 
>> communication on behalf of the project.
>>
>> After the Rapid7 program committee reviews all initial proposals, promising
>> projects will be chosen to participate in the second round of application,
>> taking place at the UNITED Security Summit, where they will be granted a =

>> 45-minute presentation and in-person Q&A session with the Rapid7 committee.

>> Selected recipients for the first funding phase will be announced following
>> the UNITED Security Summit and the second round will open for submission=
s in 

>> Q1 2012.
>>
>> __ __
>>
>> To nominate Python for Magnificent7 funding consideration, please visit the
>> Rapid7 Community website for details (http://r-7.co/qjE3WG) and guidelines 
and
>> email the submission to oss at rapid7.com <mailto:oss at rapid7.com>. Also see our
>> press release on the subject (http://r-7.co/n0bVFG). All first round 
>> submissions are due by September 9, 2011.
>>
>> Kind regards,
>>
>> Chris
>> Christian Kirsch
>>
>> Product Marketing Manager

<I chopped a zillion footers here.  lac>

------- End of Forwarded Message

The United Security Conference is  Sept 19 and 20 in San Francisco.
http://www.net-security.org/conference.php?id=445  Sept 9 is very
close for having a proposal ready, and then somebody would have
to go to SF.  On the other hand, the sandboxing in PyPy is
definitely innovative, and if they are just announcing their
contest now, there may not _be_ many other people competing for
the money.  Of course we would have to find out something to do
with sandboxing.

Laura

From chef at ghum.de  Wed Aug 24 11:29:41 2011
From: chef at ghum.de (Massa, Harald Armin)
Date: Wed, 24 Aug 2011 11:29:41 +0200
Subject: [pypy-dev] from the psf board public mailing list.
In-Reply-To: <201108240914.p7O9ELHo026537@theraft.openend.se>
References: <201108240914.p7O9ELHo026537@theraft.openend.se>
Message-ID: <CAMSP2L6dJzkbTtq+UvMW76j2+znEyQtDrnT=RTByvtpcUErBYg@mail.gmail.com>

>
>
> The United Security Conference is  Sept 19 and 20 in San Francisco.
> http://www.net-security.org/conference.php?id=445  Sept 9 is very
> close for having a proposal ready, and then somebody would have
> to go to SF.  On the other hand, the sandboxing in PyPy is
> definitely innovative, and if they are just announcing their
> contest now, there may not _be_ many other people competing for
> the money.  Of course we would have to find out something to do
> with sandboxing.
>

I can contribute ideas for things to do with sandboxing:

- expose an analyzing framework ("matlab style") to the web, pypy the bling
out of some big data (some genome or whatever): Buzzword bonus "big data",
"analyzing framework", and "genome"

- package the sandboxing as an PostgreSQL trusted language (as in
plPyPySandbox), giving the possibility to write database procedures in
Python without the chance to harm something)
Buzzword bonus "database"

- prove the concept of an scalable sandboxed application hosting in the
cloud. That sandboxed application hosting is something relevant could be
confirmed by Wesley.
Buzzword bonus: "scalable", "application hosting", "the cloud"


Harald
-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110824/ffd599bb/attachment.html>

From pypy at pocketnix.org  Wed Aug 24 13:29:31 2011
From: pypy at pocketnix.org (Da_Blitz)
Date: Wed, 24 Aug 2011 11:29:31 +0000
Subject: [pypy-dev] from the psf board public mailing list.
In-Reply-To: <201108240914.p7O9ELHo026537@theraft.openend.se>
References: <201108240914.p7O9ELHo026537@theraft.openend.se>
Message-ID: <20110824112931.GC12140@pocketnix.org>

> The United Security Conference is  Sept 19 and 20 in San Francisco.
> http://www.net-security.org/conference.php?id=445  Sept 9 is very
> close for having a proposal ready, and then somebody would have
> to go to SF.  On the other hand, the sandboxing in PyPy is
> definitely innovative, and if they are just announcing their
> contest now, there may not _be_ many other people competing for
> the money.  Of course we would have to find out something to do
> with sandboxing.
> 

If anyone is looking for ideas i have been playing with 
containers/namespaces and seccomp support on linux and believe it can 
be applied to sandbox mode with a bit of effort. there are one or two 
issues in regards to memory related syscalls that may need a bit of 
thinking through but it should tighten the security of sandbox mode 
under linux a bit more

it would also be interesting to look into what other security 
primitives Mac OSX and windows provide as well as the BSD's and if 
they can be integrated and i would be interested in these extra 
features to expand my code

if anybody wants to play or see what its about hit 
http://code.pocketnix.org/asylum code is WIP and massive changes over 
the next couple of weeks are possible. at the moment the code is not 
pypy specific and can be used to isolate cpython in a jail/container

containers/namspaces are OS level vitalization. one kernel multiple 
instances of userspace (think similar to BSD jails or a more fine 
grain openVZ)

seccomp disables all syscalls except sigreturn, read/write and _exit 
to isolate an app and help prevent a malicious program from inflicting 
harm. it was originally conceived so users could rent out cpu cycles

if someone decides to go with this i am more than willing to help out 
or get anyone up to speed on it. or if anyone is interested in general 
let me know

From arigo at tunes.org  Wed Aug 24 14:58:37 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 24 Aug 2011 14:58:37 +0200
Subject: [pypy-dev] pypy1.6 slow on string-heavy ops
In-Reply-To: <F8C84977-368E-4A41-BD5F-96E997CF6D97@gmail.com>
References: <CAHYXj6dCmdSvUCo6Aq6R6f13JqWO7eykb6+GX-gNKw_nwZNx5A@mail.gmail.com>
	<CAHYXj6cbAYV_ocy4c6tYRQ_f298eS6=6DjNMsdphSovDb0BsPQ@mail.gmail.com>
	<CAEwRq=p339E9YrhUgrgoK7wcBCvVgcnjk1NGuocMF9-UpTPjxQ@mail.gmail.com>
	<CA+=c-xZxviyWomVwuFkcK5zi1r90BxszVeETF-14aGSFo+FHOA@mail.gmail.com>
	<CAL4sBwj6g4PJNi2zvz3AtDKq03wPqtbH84ebTP2Ymm97JSU2vA@mail.gmail.com>
	<CA+=c-xYdhLEKsCg3f+RGHizfM_85EsWJrE+b6GpM64LzzThX-A@mail.gmail.com>
	<CAHYXj6ctRBPaSs593PbTjVy6t9-9KAn3rc72A6_4ck7Ojt5-1Q@mail.gmail.com>
	<CAMSv6X0J-Qu=8hZPDyt39WT7U4nm6_SWTYt3gB-Rxu+iNTbsoQ@mail.gmail.com>
	<1313768907.71818.YahooMailNeo@web111304.mail.gq1.yahoo.com>
	<CAMSv6X06qxii0HD-+NtHx0_AfxQXyJuUC-EVffbJhtfHKi_fpg@mail.gmail.com>
	<1313908302.18401.YahooMailNeo@web111306.mail.gq1.yahoo.com>
	<1314070825.41978.YahooMailNeo@web111314.mail.gq1.yahoo.com>
	<F8C84977-368E-4A41-BD5F-96E997CF6D97@gmail.com>
Message-ID: <CAMSv6X30fU5i1TQbo0ufen63THaP6iE9fr9_DsyWD8xUQK+zmA@mail.gmail.com>

Hi,

On Tue, Aug 23, 2011 at 7:28 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> If you read that Armin's email carefully, you notice that he talks about a low-level primitive called "stacklets", which have some limitations, but are not intended for a regular use. ?Greenlets will be implemented on top of them. ?So anything that was possible to do before will be supported in the new version.

Thanks, yes, indeed.  In fact the final solution, implemented now, has
one-shot continuations called "stacklets" at the C level, then a still
rather primitive notion called "continulets" exposed to Python, and
finally "greenlet.py", written as pure Python code on top of
continulets, implements the usual greenlets.  (I think that
continulets are a great primitive worth a thousand words --- or a
research article --- but that's another topic.)

Btw, I checked that this works nicely with the JIT.  In an example
with greenlets calling each other without doing anything themselves,
it gets easily twice as fast as CPython with "greenlet.c".


A bient?t,

Armin.

From arigo at tunes.org  Wed Aug 24 15:14:44 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 24 Aug 2011 15:14:44 +0200
Subject: [pypy-dev] Anyone interested in a MIPS port...
In-Reply-To: <CACPguY_Oq4kTTcw6r3rqSP9vbLxQUL+hLvgYjEEPKvNPhKE=qg@mail.gmail.com>
References: <CACPguY_Oq4kTTcw6r3rqSP9vbLxQUL+hLvgYjEEPKvNPhKE=qg@mail.gmail.com>
Message-ID: <CAMSv6X1_9bADj_G2VkwoUX3amc33qjqibDGrd0O13h-j3padpA@mail.gmail.com>

Hi Vishal,

On Sat, Aug 20, 2011 at 3:53 PM, Vishal <vsapre80 at gmail.com> wrote:
> a) Does it make sense to have a MIPS port of the PyPy JIT.

Yes, it definitely makes sense.  I assume that the MIPS machines you
consider as final targets have *some* amount of RAM, like, say,
minimum 32MB or 64MB.  PyPy would have issues running on smaller
machines, let alone with the JIT.

> b) How much hardware dependent is a JIT port?

You need to write pypy/jit/backend/mips/, similar to the other
existing JIT backends: x86 (the only one nightly tested), ARM or
PowerPC.  This is the only hardware-dependent part (not e.g. the JIT
front-end): it receives a list of operations (generic operations
represented as nice objects, like "integer addition" and "read this
field from that pointer") and must turn it into machine code.

Sven and David are currently working on the PowerPC backend in the
branch "ppc-jit-backend", if you want to follow; it is still at an
early stage, which means that the amount of code so far should be
reasonable.

Sorry to answer this late, it seems that nobody is very much
interested in contributing...  All I can promise myself is to give you
some help, as I do right now with Sven. :-)


A bient?t,

Armin.

From angelflow at yahoo.com  Wed Aug 24 23:26:25 2011
From: angelflow at yahoo.com (Andy)
Date: Wed, 24 Aug 2011 14:26:25 -0700 (PDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
Message-ID: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>

Hi,

In the PyPy benchmark there's a Django test that shows tremendous speedup of Django when running PyPy-JIT. But that is just for Django templates. What about the other parts of Django?

1) For URL routing?Django uses the re module, which is a C extension. Would JIT work with that?

2) Other parts such as DB driver also contains C code, again would that work with JIT? I suppose I could use a pure Python driver, but would a JIT'd pure Python driver be much slower than a native driver?

In general are there any rules of thumbs regarding PyPy and C extensions - what works, what doesn't?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110824/b1aa101a/attachment.html>

From santagada at gmail.com  Wed Aug 24 23:59:24 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Wed, 24 Aug 2011 18:59:24 -0300
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
Message-ID: <CA+HFc864BmQctkL8Na7fPUdxzKsBRYdtv-zxnrw02jPK==NooA@mail.gmail.com>

On Wed, Aug 24, 2011 at 6:26 PM, Andy <angelflow at yahoo.com> wrote:
> Hi,
> In the PyPy benchmark there's a Django test that shows tremendous speedup of
> Django when running PyPy-JIT. But that is just for Django templates. What
> about the other parts of Django?
> 1) For URL routing?Django uses the re module, which is a C extension. Would
> JIT work with that?
the re module in pypy is not a c extension, so yes :)

> 2) Other parts such as DB driver also contains C code, again would that work
> with JIT? I suppose I could use a pure Python driver, but would a JIT'd pure
> Python driver be much slower than a native driver?

there is a ctypes driver for sqllite and mysql (this last one being
used in quora). There is a postgresql driver also but this one is a
little harder to get going (it needs to be translated with the pypy
interpreter) but there are pure python ones and they might be pretty
fast.

> In general are there any rules of thumbs regarding PyPy and C extensions -
> what works, what doesn't?

this question I will leave for amaury to respond.


-- 
Leonardo Santagada

From davidf at sjsoft.com  Thu Aug 25 07:12:54 2011
From: davidf at sjsoft.com (David Fraser)
Date: Thu, 25 Aug 2011 00:12:54 -0500 (CDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
Message-ID: <0dd6e2ef-c45b-4ddb-a816-353ee0c583b3@jackdaw.local>

I'm not using Django but another web platform; I've found some speedup using pypy after the JIT has kicked in, but it did require some profiling and adjusting of my code. Why not give pypy a go?

----- Original Message ----- 

From: "Andy" <angelflow at yahoo.com> 
To: pypy-dev at python.org 
Sent: Wednesday, August 24, 2011 11:26:25 PM 
Subject: [pypy-dev] Is PyPy appropriate for Django? 



Hi, 


In the PyPy benchmark there's a Django test that shows tremendous speedup of Django when running PyPy-JIT. But that is just for Django templates. What about the other parts of Django? 


1) For URL routing Django uses the re module, which is a C extension. Would JIT work with that? 


2) Other parts such as DB driver also contains C code, again would that work with JIT? I suppose I could use a pure Python driver, but would a JIT'd pure Python driver be much slower than a native driver? 


In general are there any rules of thumbs regarding PyPy and C extensions - what works, what doesn't? 


Thanks. 
_______________________________________________ 
pypy-dev mailing list 
pypy-dev at python.org 
http://mail.python.org/mailman/listinfo/pypy-dev 

From arigo at tunes.org  Thu Aug 25 10:22:29 2011
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 25 Aug 2011 10:22:29 +0200
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
Message-ID: <CAMSv6X15L25XD8d6Rnn0d9pxWXq=QFmm2yDwB=h9Pxk0PxbumA@mail.gmail.com>

Hi Andy,

On Wed, Aug 24, 2011 at 11:26 PM, Andy <angelflow at yahoo.com> wrote:
> 1) For URL routing?Django uses the re module, which is a C extension. Would
> JIT work with that?

Is this "re" the regular expression module?  If so, it's a standard
library module, so PyPy provides one too, rewritten in RPython.  In
fact it is expected to be much faster than CPython's in the long run,
because regular expressions also use the JIT to compile regular
expression objects (which are originally some kind of bytecode).


A bient?t,

Armin.

From anto.cuni at gmail.com  Thu Aug 25 10:51:59 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 25 Aug 2011 10:51:59 +0200
Subject: [pypy-dev] [pypy-commit] pypy reflex-support: Remove
 elidable_promote that made the translation choke.
In-Reply-To: <20110816174745.505FB82112@wyvern.cs.uni-duesseldorf.de>
References: <20110816174745.505FB82112@wyvern.cs.uni-duesseldorf.de>
Message-ID: <4E560D2F.2020603@gmail.com>

Hi Wim,

On 16/08/11 19:47, wlav wrote:
> Author: Wim Lavrijsen<WLavrijsen at lbl.gov>
> Branch: reflex-support
> Changeset: r46550:17b6f87c70bc
> Date: 2011-08-16 10:54 -0700
> http://bitbucket.org/pypy/pypy/changeset/17b6f87c70bc/
>
> Log:	Remove elidable_promote that made the translation choke.
>
> diff --git a/pypy/module/cppyy/interp_cppyy.py b/pypy/module/cppyy/interp_cppyy.py
> --- a/pypy/module/cppyy/interp_cppyy.py
> +++ b/pypy/module/cppyy/interp_cppyy.py
> @@ -296,7 +296,6 @@
>       def is_static(self):
>           return self.space.newbool(self._is_static)
>
> -    @jit.elidable_promote()
>       def _get_offset(self, w_cppinstance):
>           cppinstance = self.space.interp_w(W_CPPInstance, w_cppinstance, can_be_None=True)
>           if cppinstance:

are you sure that you really want to kill this promote? I suspect that it 
makes accessing to C++ fields much slower, because it has to recalculate the 
offset every time.

ciao,
Anto

From jfcgauss at gmail.com  Thu Aug 25 11:15:17 2011
From: jfcgauss at gmail.com (Serhat Sevki Dincer)
Date: Thu, 25 Aug 2011 12:15:17 +0300
Subject: [pypy-dev] Incompatibility in binary distribution of libraries?
Message-ID: <CAPqC6xSqPJPK87s066OyRO=DmopONJBci2NrgRGC0Df7noKtgA@mail.gmail.com>

Start with main.py and mylib.py
with python2.7 that comes with ubuntu 11.04 i386, run
python2.7 main.py
the program runs correctly and a file mylib.pyc is produced.
Then, if I remove or rename mylib.py file, pyc file is sufficient for
main.py to work
python2.7 main.py
still works.

With pypy 1.6, after I rename mylib.py, and try
pypy main.py
it complains with
Traceback (most recent call last):
  File "app_main.py", line 53, in run_toplevel
  File "main.py", line 1, in <module>
    from mylib import myfunc
ImportError: No module named mylib

Is this an incompatibility with CPython 2.7? Or is it an environment
variable/default setting problem specific to Ubuntu?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.py
Type: text/x-python
Size: 39 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/9f233708/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mylib.py
Type: text/x-python
Size: 43 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/9f233708/attachment-0001.py>

From jfcgauss at gmail.com  Thu Aug 25 11:26:50 2011
From: jfcgauss at gmail.com (Serhat Sevki Dincer)
Date: Thu, 25 Aug 2011 12:26:50 +0300
Subject: [pypy-dev] Removing dead emails from list
Message-ID: <CAPqC6xT4VeQ062c89ga6D8GBxUdqh84ObpESE6zhSLjGNM=oNA@mail.gmail.com>

Whenever I send an email to pypy-dev I get the below bounce back. Can
somebody remove that dead email from the list?

from	postmaster at paran.com
to	jfcgauss at gmail.com
date	Thu, Aug 25, 2011 at 12:18 PM
subject	[ERR] [pypy-dev] Incompatibility in binary distribution of libraries?
mailed-by	nrmail7.paran.com
hide details 12:18 PM (6 minutes ago)
Transmit Report:

 To: seojiwonn at paran.com, 2011/08/25 18:18:18, 452, This mailbox is
temporarily disabled. This mailbox is automatically ena

From wlavrijsen at lbl.gov  Thu Aug 25 16:34:42 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 25 Aug 2011 07:34:42 -0700 (PDT)
Subject: [pypy-dev] [pypy-commit] pypy reflex-support: Remove
 elidable_promote that made the translation choke.
In-Reply-To: <4E560D2F.2020603@gmail.com>
References: <20110816174745.505FB82112@wyvern.cs.uni-duesseldorf.de>
	<4E560D2F.2020603@gmail.com>
Message-ID: <alpine.LNX.2.00.1108250727110.10586@pc3-wlav.lbl.gov>

Hi Anto,

On Thu, 25 Aug 2011, Antonio Cuni wrote:
>> -    @jit.elidable_promote()
>>       def _get_offset(self, w_cppinstance):
>>           cppinstance = self.space.interp_w(W_CPPInstance, w_cppinstance, can_be_None=True)
>>           if cppinstance:
>
> are you sure that you really want to kill this promote? I suspect that it 
> makes accessing to C++ fields much slower, because it has to recalculate the 
> offset every time.

yes, it would make it a lot slower, since it adds a C++ call. However, the
translation failed and if it does succeed (as was the case for functions, so
I still need to figure out what the real difference is), then it doesn't do
anything at runtime. I have it more on my "look into this later" list.

What I did for now, is that I took one short-cut out of the C-API, which is
the most common case: if the handle to which the data member belongs and the
handle of "this" are the same, the offset is by definition zero. I put that
in python, so fully visible to the JIT, and only call the back-end if some
calculation is needed. Then I get the expected speeds again (for the most
common case).

This is the kind of thing that I'll want to use the jitviewer for, once I'm
beyond functionality and only need to care about performance. :)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From angelflow at yahoo.com  Thu Aug 25 17:01:25 2011
From: angelflow at yahoo.com (Andy)
Date: Thu, 25 Aug 2011 08:01:25 -0700 (PDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <0dd6e2ef-c45b-4ddb-a816-353ee0c583b3@jackdaw.local>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
	<0dd6e2ef-c45b-4ddb-a816-353ee0c583b3@jackdaw.local>
Message-ID: <1314284485.18963.YahooMailNeo@web111302.mail.gq1.yahoo.com>

Hi David,

Which platform are you using?

How big of a speedup did you get? Did the memory footprint increased significantly?

Thanks.


________________________________
From: David Fraser <davidf at sjsoft.com>
To: Andy <angelflow at yahoo.com>
Cc: pypy-dev at python.org
Sent: Thursday, August 25, 2011 1:12 AM
Subject: Re: [pypy-dev] Is PyPy appropriate for Django?

I'm not using Django but another web platform; I've found some speedup using pypy after the JIT has kicked in, but it did require some profiling and adjusting of my code. Why not give pypy a go?

----- Original Message ----- 

From: "Andy" <angelflow at yahoo.com> 
To: pypy-dev at python.org 
Sent: Wednesday, August 24, 2011 11:26:25 PM 
Subject: [pypy-dev] Is PyPy appropriate for Django? 



Hi, 


In the PyPy benchmark there's a Django test that shows tremendous speedup of Django when running PyPy-JIT. But that is just for Django templates. What about the other parts of Django? 


1) For URL routing Django uses the re module, which is a C extension. Would JIT work with that? 


2) Other parts such as DB driver also contains C code, again would that work with JIT? I suppose I could use a pure Python driver, but would a JIT'd pure Python driver be much slower than a native driver? 


In general are there any rules of thumbs regarding PyPy and C extensions - what works, what doesn't? 


Thanks. 
_______________________________________________ 
pypy-dev mailing list 
pypy-dev at python.org 
http://mail.python.org/mailman/listinfo/pypy-dev 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/ad2809e0/attachment.html>

From santagada at gmail.com  Thu Aug 25 17:14:28 2011
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 25 Aug 2011 12:14:28 -0300
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
Message-ID: <CA+HFc87YQOobNqGAhRFGzGWq14Sj7hXfDNJLLqxp2RH50h95RA@mail.gmail.com>

On Wed, Aug 24, 2011 at 6:26 PM, Andy <angelflow at yahoo.com> wrote:
> Hi,
> In the PyPy benchmark there's a Django test that shows tremendous speedup of
> Django when running PyPy-JIT. But that is just for Django templates. What
> about the other parts of Django?

Did you notice that there is also a benchmark of rietveld (which is a
django app)? You can see that both are faster than cpython and getting
faster at each release :)

http://speed.pypy.org/comparison/?exe=1%2B41%2C1%2B172%2C1%2B301%2C1%2BL%2Bdefault%2C4%2B35&ben=3%2C10&env=1&hor=false&bas=2%2B35&chart=normal+bars


-- 
Leonardo Santagada

From angelflow at yahoo.com  Thu Aug 25 17:13:48 2011
From: angelflow at yahoo.com (Andy)
Date: Thu, 25 Aug 2011 08:13:48 -0700 (PDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <CAMSv6X15L25XD8d6Rnn0d9pxWXq=QFmm2yDwB=h9Pxk0PxbumA@mail.gmail.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
	<CAMSv6X15L25XD8d6Rnn0d9pxWXq=QFmm2yDwB=h9Pxk0PxbumA@mail.gmail.com>
Message-ID: <1314285228.10111.YahooMailNeo@web111313.mail.gq1.yahoo.com>

Armin,

Yes it is the regular expression module. Great to know that it'll be faster in PyPy.

A large part of my application is creating and manipulating objects using Django's ORM and other modules. Would PyPy speed that up too? Or are those objects too short-lived (they only live within a request-response cycle) for JIT to be useful?


________________________________
From: Armin Rigo <arigo at tunes.org>
To: Andy <angelflow at yahoo.com>
Cc: "pypy-dev at python.org" <pypy-dev at python.org>
Sent: Thursday, August 25, 2011 4:22 AM
Subject: Re: [pypy-dev] Is PyPy appropriate for Django?

Hi Andy,

On Wed, Aug 24, 2011 at 11:26 PM, Andy <angelflow at yahoo.com> wrote:
> 1) For URL routing?Django uses the re module, which is a C extension. Would
> JIT work with that?

Is this "re" the regular expression module?? If so, it's a standard
library module, so PyPy provides one too, rewritten in RPython.? In
fact it is expected to be much faster than CPython's in the long run,
because regular expressions also use the JIT to compile regular
expression objects (which are originally some kind of bytecode).


A bient?t,

Armin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/87cc9dcd/attachment.html>

From angelflow at yahoo.com  Thu Aug 25 17:45:12 2011
From: angelflow at yahoo.com (Andy)
Date: Thu, 25 Aug 2011 08:45:12 -0700 (PDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <CA+HFc87YQOobNqGAhRFGzGWq14Sj7hXfDNJLLqxp2RH50h95RA@mail.gmail.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
	<CA+HFc87YQOobNqGAhRFGzGWq14Sj7hXfDNJLLqxp2RH50h95RA@mail.gmail.com>
Message-ID: <1314287112.48994.YahooMailNeo@web111306.mail.gq1.yahoo.com>

I didn't know about the rietveld ?benchmark.?

Thanks!


________________________________
From: Leonardo Santagada <santagada at gmail.com>
To: Andy <angelflow at yahoo.com>
Cc: "pypy-dev at python.org" <pypy-dev at python.org>
Sent: Thursday, August 25, 2011 11:14 AM
Subject: Re: [pypy-dev] Is PyPy appropriate for Django?

On Wed, Aug 24, 2011 at 6:26 PM, Andy <angelflow at yahoo.com> wrote:
> Hi,
> In the PyPy benchmark there's a Django test that shows tremendous speedup of
> Django when running PyPy-JIT. But that is just for Django templates. What
> about the other parts of Django?

Did you notice that there is also a benchmark of rietveld (which is a
django app)? You can see that both are faster than cpython and getting
faster at each release :)

http://speed.pypy.org/comparison/?exe=1%2B41%2C1%2B172%2C1%2B301%2C1%2BL%2Bdefault%2C4%2B35&ben=3%2C10&env=1&hor=false&bas=2%2B35&chart=normal+bars


-- 
Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/52c71829/attachment-0001.html>

From davidf at sjsoft.com  Thu Aug 25 18:11:32 2011
From: davidf at sjsoft.com (David Fraser)
Date: Thu, 25 Aug 2011 11:11:32 -0500 (CDT)
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <1314284485.18963.YahooMailNeo@web111302.mail.gq1.yahoo.com>
Message-ID: <75ecec99-83d8-4fa5-8c13-6c820286899c@jackdaw.local>

Hi Andy 


I'm using our own internal platform which uses CherryPy, sqlalchemy and Routes. Currently I get a 20% speedup once JIT has completed; most of this I think has to do with our own native code that's not very friendly to PyPy JIT (lots of generators and **kwargs passing), so I'm hoping to get that better. I haven't monitored memory usage specifically 


Cheers 
David 

----- Original Message -----

From: "Andy" <angelflow at yahoo.com> 
To: "David Fraser" <davidf at sjsoft.com> 
Cc: pypy-dev at python.org 
Sent: Thursday, August 25, 2011 5:01:25 PM 
Subject: Re: [pypy-dev] Is PyPy appropriate for Django? 



Hi David, 


Which platform are you using? 


How big of a speedup did you get? Did the memory footprint increased significantly? 


Thanks. 




From: David Fraser <davidf at sjsoft.com> 
To: Andy <angelflow at yahoo.com> 
Cc: pypy-dev at python.org 
Sent: Thursday, August 25, 2011 1:12 AM 
Subject: Re: [pypy-dev] Is PyPy appropriate for Django? 

I'm not using Django but another web platform; I've found some speedup using pypy after the JIT has kicked in, but it did require some profiling and adjusting of my code. Why not give pypy a go? 

----- Original Message ----- 

From: "Andy" < angelflow at yahoo.com > 
To: pypy-dev at python.org 
Sent: Wednesday, August 24, 2011 11:26:25 PM 
Subject: [pypy-dev] Is PyPy appropriate for Django? 



Hi, 


In the PyPy benchmark there's a Django test that shows tremendous speedup of Django when running PyPy-JIT. But that is just for Django templates. What about the other parts of Django? 


1) For URL routing Django uses the re module, which is a C extension. Would JIT work with that? 


2) Other parts such as DB driver also contains C code, again would that work with JIT? I suppose I could use a pure Python driver, but would a JIT'd pure Python driver be much slower than a native driver? 


In general are there any rules of thumbs regarding PyPy and C extensions - what works, what doesn't? 


Thanks. 
_______________________________________________ 
pypy-dev mailing list 
pypy-dev at python.org 
http://mail.python.org/mailman/listinfo/pypy-dev 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/a873fb63/attachment.html>

From wlavrijsen at lbl.gov  Thu Aug 25 18:14:09 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 25 Aug 2011 09:14:09 -0700 (PDT)
Subject: [pypy-dev] [pypy-commit] pypy reflex-support: Remove
 elidable_promote that made the translation choke.
In-Reply-To: <4E560D2F.2020603@gmail.com>
References: <20110816174745.505FB82112@wyvern.cs.uni-duesseldorf.de>
	<4E560D2F.2020603@gmail.com>
Message-ID: <alpine.LNX.2.00.1108250911160.10586@pc3-wlav.lbl.gov>

Hi Anto,

>> -    @jit.elidable_promote()
>>       def _get_offset(self, w_cppinstance):
>>           cppinstance = self.space.interp_w(W_CPPInstance, w_cppinstance, 
>> can_be_None=True)
>>           if cppinstance:
>
> are you sure that you really want to kill this promote? I suspect that it 
> makes accessing to C++ fields much slower, because it has to recalculate the 
> offset every time.

just as a follow-up, this was the translation problem I ran into:

[translation:ERROR]    File "/home/wlav/pypydev/pypy/pypy/jit/codewriter/jtransform.py", line 271, in rewrite_op_direct_call
[translation:ERROR]     return getattr(self, 'handle_%s_call' % kind)(op)
[translation:ERROR]    File "/home/wlav/pypydev/pypy/pypy/jit/codewriter/jtransform.py", line 319, in handle_residual_call
[translation:ERROR]     calldescr = self.callcontrol.getcalldescr(op)
[translation:ERROR]    File "/home/wlav/pypydev/pypy/pypy/jit/codewriter/call.py", line 247, in getcalldescr
[translation:ERROR]     assert effectinfo is not None
[translation:ERROR]  AssertionError

for functions, I have the interp_w in the calling function, so I'm guessing
that the problem is with the promote, not with the elidability. I'll change
that to be the same as for functions, which should then work.

Actually, I'm thinking that an assert would work just as well. But that would
really be the kind of optimization that I'm not yet in for. :)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From davide.setti at gmail.com  Thu Aug 25 19:27:24 2011
From: davide.setti at gmail.com (Davide Setti)
Date: Thu, 25 Aug 2011 19:27:24 +0200
Subject: [pypy-dev] Is PyPy appropriate for Django?
In-Reply-To: <CA+HFc864BmQctkL8Na7fPUdxzKsBRYdtv-zxnrw02jPK==NooA@mail.gmail.com>
References: <1314221185.5285.YahooMailNeo@web111301.mail.gq1.yahoo.com>
	<CA+HFc864BmQctkL8Na7fPUdxzKsBRYdtv-zxnrw02jPK==NooA@mail.gmail.com>
Message-ID: <CA+p5JUabccawa6M75n8oDNKDaVoK58PnPxN6F3R0HfKn9Ezkkg@mail.gmail.com>

On Wed, Aug 24, 2011 at 11:59 PM, Leonardo Santagada
<santagada at gmail.com> wrote:
> there is a ctypes driver for sqllite and mysql (this last one being
> used in quora). There is a postgresql driver also but this one is a
> little harder to get going (it needs to be translated with the pypy
> interpreter) but there are pure python ones and they might be pretty
> fast.

There is https://github.com/mvantellingen/psycopg2-ctypes for
postgres. I'm using it and it seems to work fine.

I did two things with django+pypy: i ran a management command and it
took 30s against the 10s of cpython+psycopg2-ctpyes, maybe it's too
short. And i did a benchmark with ab. With a small number of
connection it works fine (and as fast as cpython), but as the number
grows gunicorn starts to give errors. I need to investigate further.
I also tried djangobench but i found this bug (in pypy ctypes, i think):

https://bugs.pypy.org/issue844

Regards
--

Davide Setti
blog: http://blog.flatlandia.eu
code: http://github.com/vad

From amauryfa at gmail.com  Thu Aug 25 19:40:54 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 25 Aug 2011 19:40:54 +0200
Subject: [pypy-dev] Incompatibility in binary distribution of libraries?
In-Reply-To: <CAPqC6xSqPJPK87s066OyRO=DmopONJBci2NrgRGC0Df7noKtgA@mail.gmail.com>
References: <CAPqC6xSqPJPK87s066OyRO=DmopONJBci2NrgRGC0Df7noKtgA@mail.gmail.com>
Message-ID: <CAGmFidaykyGL6v5r1xE4ReigJqHs7uTMddMZ343yOJNiv-cW4A@mail.gmail.com>

Hi,

2011/8/25 Serhat Sevki Dincer <jfcgauss at gmail.com>

> Start with main.py and mylib.py
> with python2.7 that comes with ubuntu 11.04 i386, run
> python2.7 main.py
> the program runs correctly and a file mylib.pyc is produced.
> Then, if I remove or rename mylib.py file, pyc file is sufficient for
> main.py to work
> python2.7 main.py
> still works.
>
> With pypy 1.6, after I rename mylib.py, and try
> pypy main.py
> it complains with
> Traceback (most recent call last):
>  File "app_main.py", line 53, in run_toplevel
>  File "main.py", line 1, in <module>
>    from mylib import myfunc
> ImportError: No module named mylib
>
> Is this an incompatibility with CPython 2.7? Or is it an environment
> variable/default setting problem specific to Ubuntu?
>

This is expected with pypy, see discussion there:
http://mail.python.org/pipermail/pypy-dev/2011-May/007385.html

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/86923d9c/attachment.html>

From naylor.b.david at gmail.com  Thu Aug 25 21:44:06 2011
From: naylor.b.david at gmail.com (David Naylor)
Date: Thu, 25 Aug 2011 21:44:06 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <CAMSv6X1gJyFKRCn7j_YUz1s=remMWLx1=tg9AQBw=sHMaT82ug@mail.gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108151821.02922.naylor.b.david@gmail.com>
	<CAMSv6X1gJyFKRCn7j_YUz1s=remMWLx1=tg9AQBw=sHMaT82ug@mail.gmail.com>
Message-ID: <201108252144.09934.naylor.b.david@gmail.com>

On Tuesday, 16 August 2011 15:27:30 Armin Rigo wrote:
> Hi David,
> 
> On Mon, Aug 15, 2011 at 6:20 PM, David Naylor <naylor.b.david at gmail.com> 
wrote:
> > For me the performance of datetime object's hashing is sufficient but I
> > think the python code could use some performance improvements.  Is my
> > approach using a direct computation to type long acceptable (in
> > principle).  If so I can refine it and submit a patch.
> 
> Yes, replacing the hash with a faster-to-compute one is fine.  It's
> best performance-wise if you can avoid using Python longs.  As far as
> I know it just needs some random-looking xor-ing and shifting of the
> fields.  Note, of course, that you must carefully satisfy the property
> that for any objects x and y, if "x == y" then "hash(x) == hash(y)".

Below is the patch, and results, for my proposed hash methods for 
datetime.datetime (and easily adaptable to include tzinfo and the other 
datetime objects).  I tried to make the hash safe for both 32bit and 64bit 
systems, and beyond.  

The results are:
# python datetest.py (datetime.py)
hash_unity: 35.83 seconds
hash_unity: 44.60 seconds
hash_datetime: 65.58 seconds
hash_datetime: 53.95 seconds

# python datetest.py
hash_unity: 5.70 seconds
hash_unity: 5.69 seconds
hash_datetime: 4.88 seconds
hash_datetime: 4.90 seconds

# pypy datetest.py
hash_unity: 0.74 seconds
hash_unity: 0.63 seconds
hash_datetime: 11.74 seconds
hash_datetime: 11.47 seconds

# pypy datetest.py (patched datetime.py)
hash_unity: 0.73 seconds
hash_unity: 0.62 seconds
hash_datetime: 0.76 seconds
hash_datetime: 0.64 seconds

So, based on my patch there is a 7.7x improvement over python and a 17.9x 
improvement over the previous pypy implementation.  

If the above approach is acceptable I will complete the patch?  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: datetime.patch
Type: text/x-patch
Size: 1412 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/f71bd3ba/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: datetest.py
Type: text/x-python
Size: 1030 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/f71bd3ba/attachment-0001.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110825/f71bd3ba/attachment-0001.pgp>

From arigo at tunes.org  Fri Aug 26 05:43:30 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 26 Aug 2011 05:43:30 +0200
Subject: [pypy-dev] List index out of range for many tests
In-Reply-To: <CACOXMX6-SrwN=s1J2tQ_yLWBqk4Ja9sM2dcQuo62sva=GCJPkg@mail.gmail.com>
References: <CACOXMX4niX4aSPKGBS6HgvpaG5OHGGKFQOtYzmfdiXX4X_Egdg@mail.gmail.com>
	<CAMSv6X0gFRK20MtAoD3HDiVQOoxvKY95XpOR1V39dZxo0dhHqA@mail.gmail.com>
	<CACOXMX6-SrwN=s1J2tQ_yLWBqk4Ja9sM2dcQuo62sva=GCJPkg@mail.gmail.com>
Message-ID: <CAMSv6X0Wc9TGt34hBHh9tR6HjX3E7YcTboW9nXOaJjt=bYSqNA@mail.gmail.com>

Hi Mitchell,

On Sat, Aug 20, 2011 at 4:47 PM, Mitchell Hashimoto
<mitchell.hashimoto at gmail.com> wrote:
> One question: The tests take quite awhile to run, even for a single file. Is
> this normal?

Yes, kind of.  This is particular to test_ctypes.  For me it takes
about 30 seconds just for the command you gave, whereas CPython
completes in 1.5 seconds.  This is due to a combination of factors: it
takes much longer to run ctypes tests because, due to the nature of
tests, the JIT doesn't really work (every test being run once), and so
the result is bad on PyPy because ctypes is mostly written at
app-level (in pure Python).  I also think that in this case some tests
take a long time and fail, but succeed very quickly on CPython.


A bient?t,

Armin.

From chrish at pianocktail.org  Fri Aug 26 05:47:16 2011
From: chrish at pianocktail.org (Christian Hudon)
Date: Thu, 25 Aug 2011 23:47:16 -0400
Subject: [pypy-dev] Suggestions for small projects for getting started
	hacking on pypy?
Message-ID: <4E571744.70406@pianocktail.org>

Hello,

I'm interested in getting my feet wet hacking on pypy. I've successfully 
bootstrapped pypy from source, using the documentation on doc.pypy.org. 
What small projects are open right now that I could use to get started 
learning about the code?

Thanks,

   Christian


From arigo at tunes.org  Fri Aug 26 06:01:55 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 26 Aug 2011 06:01:55 +0200
Subject: [pypy-dev] status of scheme implementation
In-Reply-To: <997342288.4432887.1313932543265.JavaMail.fmail@mwmweb085>
References: <997342288.4432887.1313932543265.JavaMail.fmail@mwmweb085>
Message-ID: <CAMSv6X3TrHLEkbXHeLgCckSWHXQy6wTjzW+WMxFHRMb4hZ827g@mail.gmail.com>

Hi Juergen,

On Sun, Aug 21, 2011 at 3:15 PM, Juergen Boemmels <boemmels at web.de> wrote:
> I just found the scheme implementation
> https://bitbucket.org/pypy/lang-scheme/
> but there is no history besides the initial import.
>
> Is this maintained? I.e. is it usful to send patches for improving this?

As you may have noticed by now (sorry for the delay), no, this is not
really maintained any more...  You are welcome to get direct access,
if you want to improve it.  Just make an account on bitbucket (if
needed), and tell us your username.

> I attached the two patches (one to pypy/pypy and one to pypy/lang-scheme)

Thanks!  I can see how the patch to packrat is useful, indeed.
Checked in.  I will leave the patch to lang-scheme for you to check in
:-)


A bient?t,

Armin.

From arigo at tunes.org  Fri Aug 26 06:37:30 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 26 Aug 2011 06:37:30 +0200
Subject: [pypy-dev] Python vs pypy: interesting performance difference
	[dict.setdefault]
In-Reply-To: <201108252144.09934.naylor.b.david@gmail.com>
References: <201108102127.13752.naylor.b.david@gmail.com>
	<201108151821.02922.naylor.b.david@gmail.com>
	<CAMSv6X1gJyFKRCn7j_YUz1s=remMWLx1=tg9AQBw=sHMaT82ug@mail.gmail.com>
	<201108252144.09934.naylor.b.david@gmail.com>
Message-ID: <CAMSv6X3bxiq+Y4o+oMjEStT1J_zuhd35PWW+=YbpeCC1pSCaOg@mail.gmail.com>

Hi David,

On Thu, Aug 25, 2011 at 9:44 PM, David Naylor <naylor.b.david at gmail.com> wrote:
> Below is the patch, and results, for my proposed hash methods for
> datetime.datetime (and easily adaptable to include tzinfo and the other
> datetime objects). ?I tried to make the hash safe for both 32bit and 64bit
> systems, and beyond.

Yes, the patch looks good to me.  I can definitely see how it can be a
huge improvement in performance :-)

If you can also "fix" the other __hash__ methods in the same way, it
would be great.


A bient?t,

Armin.

From arigo at tunes.org  Fri Aug 26 09:09:09 2011
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 26 Aug 2011 09:09:09 +0200
Subject: [pypy-dev] STM on CPython
Message-ID: <CAMSv6X2jNMd6NM5OUUQnmK2xD7-RQzYqZ+Q8vw=SJwu+s5mzng@mail.gmail.com>

Hi,

A follow-up to the blog post about Software Transactional Memory (STM)
at http://morepypy.blogspot.com/ .

First, there is a growing number of papers out there that seems to
give solid evidence that STM is "better" than lock-based systems, for
various definitions of "better".  In fact Greg Wilson pointed to
http://www.neverworkintheory.org/?p=122 ; it's a study of students,
showing that it takes them less time to implement the exercise with
STM than fine-grained locks --- but most importantly, only 10% of the
programs produced contain errors, as opposed to 70% with fine-grained
locks.  There are also more abstract definitions of "better", e.g. for
my point of view I'd say that the limiting ingredient in locks is that
they don't compose:
http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations
.  But more generally it seems to be emerging as a consensus in the
last 5 years.

Brett C. made in 2003 a post on python-dev that was rejected (see
http://mail.python.org/pipermail/python-dev/2003-February/033259.html);
it was about a hack to extend CPython's GIL, giving "atomic sections"
--- which would be implemented very simply by not releasing the GIL
for a while.

This was rejected as a hack that only worked because we had a GIL to
start with, and that would no longer work as soon as somebody came up
with a great idea to remove it.  Me, of course, at that time, I was
sure that this was the correct decision as well.

In retrospect, I would argue that this was possibly one of the most
dreadful decision ever made in Python.  It would be exactly STM
(without actually running on more than on core, but that's an
implementation detail).  If we had it 8 years ago, we would have had
all this time to get experience with STM in a nice language and how
you don't need locks at all to do multithreading.  And that would seem
quite precisely in line with the original goal of Python, i.e. to make
programming easier at the expense of raw performance (in this case it
is even worse, because we would not have lost any performance, but
only lost a hypothetical future path in which to gain more
performance).

So, all this to say: 8 years later, I implemented that on CPython:
https://bitbucket.org/arigo/cpython-withatomic/overview (on the 2.7
branch).  It is sadly a fork of CPython instead of a C extension
module because it needs to access and change a few things in ceval.c.
The total patch is 190 lines, and that's because I need a new type
(very verbose).  It implements "thread.atomic", which you use in a
"with" statement.  So you write "with atomic:" and you get precisely
the simple case described here:
http://en.wikipedia.org/wiki/Software_transactional_memory#Proposed_language_support
.

Fwiw it's probably also a minimal patch to PyPy.  Sorry for being
mostly off-topic for once...


A bient?t,

Armin.

From wlavrijsen at lbl.gov  Fri Aug 26 19:02:18 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 26 Aug 2011 10:02:18 -0700 (PDT)
Subject: [pypy-dev] [pypy-commit] pypy reflex-support: Remove
 elidable_promote that made the translation choke.
In-Reply-To: <alpine.LNX.2.00.1108250911160.10586@pc3-wlav.lbl.gov>
References: <20110816174745.505FB82112@wyvern.cs.uni-duesseldorf.de>
	<4E560D2F.2020603@gmail.com>
	<alpine.LNX.2.00.1108250911160.10586@pc3-wlav.lbl.gov>
Message-ID: <alpine.LNX.2.00.1108260957230.5987@pc3-wlav.lbl.gov>

Hi,

On Thu, 25 Aug 2011, WLavrijsen at lbl.gov wrote:
> for functions, I have the interp_w in the calling function, so I'm guessing
> that the problem is with the promote, not with the elidability. I'll change
> that to be the same as for functions, which should then work.

so yes, that was the ticket.

The other thing that I was thinking of, is that since access to the table of
functions to calculate offsets is going through the reflection layer anyway,
it may be possible to bootstrap this and have the call go through inter_cppyy
itself. At that point, the whole thing will be open to the JIT (and be a
rather strict application of "eating one's own dogfood" :) ). It may even be
possible to do this for all functions in the C-API that are called during the
dispatch of functions, as Reflex can run on itself (we already have this in a
package called Relax).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From exarkun at twistedmatrix.com  Fri Aug 26 18:57:51 2011
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 26 Aug 2011 16:57:51 -0000
Subject: [pypy-dev] STM on CPython
In-Reply-To: <CAMSv6X2jNMd6NM5OUUQnmK2xD7-RQzYqZ+Q8vw=SJwu+s5mzng@mail.gmail.com>
References: <CAMSv6X2jNMd6NM5OUUQnmK2xD7-RQzYqZ+Q8vw=SJwu+s5mzng@mail.gmail.com>
Message-ID: <20110826165751.1808.784020944.divmod.xquotient.1@localhost.localdomain>

On 07:09 am, arigo at tunes.org wrote:
>Hi,
>
>A follow-up to the blog post about Software Transactional Memory (STM)
>at http://morepypy.blogspot.com/ .
>
>[snio[
>
>So, all this to say: 8 years later, I implemented that on CPython:
>https://bitbucket.org/arigo/cpython-withatomic/overview (on the 2.7
>branch).  It is sadly a fork of CPython instead of a C extension
>module because it needs to access and change a few things in ceval.c.
>The total patch is 190 lines, and that's because I need a new type
>(very verbose).  It implements "thread.atomic", which you use in a
>"with" statement.  So you write "with atomic:" and you get precisely
>the simple case described here:
>http://en.wikipedia.org/wiki/Software_transactional_memory#Proposed_language_support
>.
>
>Fwiw it's probably also a minimal patch to PyPy.  Sorry for being
>mostly off-topic for once...

Very cool.  Thanks for the summary.

Jean-Paul

From boemmels at web.de  Fri Aug 26 19:17:50 2011
From: boemmels at web.de (Juergen Boemmels)
Date: Fri, 26 Aug 2011 19:17:50 +0200 (CEST)
Subject: [pypy-dev] status of scheme implementation
Message-ID: <1303789578.8114371.1314379070491.JavaMail.fmail@mwmweb070>

Hi Armin,

>Hi Juergen,
>
>On Sun, Aug 21, 2011 at 3:15 PM, Juergen Boemmels <boemmels at web.de> wrote:
>> I just found the scheme implementation
>> https://bitbucket.org/pypy/lang-scheme/
>> but there is no history besides the initial import.
>>
>> Is this maintained? I.e. is it usful to send patches for improving this?
>
>As you may have noticed by now (sorry for the delay), no, this is not
>really maintained any more... You are welcome to get direct access,
>if you want to improve it. Just make an account on bitbucket (if
>needed), and tell us your username.

I just created an account on bitbucket. Username is "boemmels"

Let's see what I can do with it.

>> I attached the two patches (one to pypy/pypy and one to pypy/lang-scheme)
>
>Thanks! I can see how the patch to packrat is useful, indeed.
>Checked in. I will leave the patch to lang-scheme for you to check in
>:-)

Ok, will do that

Regards
juergen

>A bient?t,
>
>Armin.


___________________________________________________________
Schon geh?rt? WEB.DE hat einen genialen Phishing-Filter in die
Toolbar eingebaut! http://produkte.web.de/go/toolbar

From cfbolz at gmx.de  Fri Aug 26 19:32:33 2011
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 26 Aug 2011 19:32:33 +0200
Subject: [pypy-dev] status of scheme implementation
In-Reply-To: <1303789578.8114371.1314379070491.JavaMail.fmail@mwmweb070>
References: <1303789578.8114371.1314379070491.JavaMail.fmail@mwmweb070>
Message-ID: <4E57D8B1.90802@gmx.de>

On 08/26/2011 07:17 PM, Juergen Boemmels wrote:
>> On Sun, Aug 21, 2011 at 3:15 PM, Juergen Boemmels<boemmels at web.de>  wrote:
>>> I just found the scheme implementation
>>> https://bitbucket.org/pypy/lang-scheme/
>>> but there is no history besides the initial import.
>>>
>>> Is this maintained? I.e. is it usful to send patches for improving this?
>>
>> As you may have noticed by now (sorry for the delay), no, this is not
>> really maintained any more... You are welcome to get direct access,
>> if you want to improve it. Just make an account on bitbucket (if
>> needed), and tell us your username.
>
> I just created an account on bitbucket. Username is "boemmels"
>
> Let's see what I can do with it.

Ok, I gave you write access to the repo. You should be able to push to 
it now.

Carl Friedrich

From lac at openend.se  Fri Aug 26 21:29:50 2011
From: lac at openend.se (Laura Creighton)
Date: Fri, 26 Aug 2011 21:29:50 +0200
Subject: [pypy-dev] STM on CPython
In-Reply-To: Message from exarkun@twistedmatrix.com of "Fri,
	26 Aug 2011 16:57:51 -0000."
	<20110826165751.1808.784020944.divmod.xquotient.1@localhost.localdomain>
References: <CAMSv6X2jNMd6NM5OUUQnmK2xD7-RQzYqZ+Q8vw=SJwu+s5mzng@mail.gmail.com><20110826165751.1808.784020944.divmod.xquotient.1@localhost.localdomain>
Message-ID: <201108261929.p7QJToFD028177@theraft.openend.se>


Armin wrote:
>So, all this to say: 8 years later, I implemented that on CPython:
>https://bitbucket.org/arigo/cpython-withatomic/overview (on the 2.7
>branch).  It is sadly a fork of CPython instead of a C extension
>module because it needs to access and change a few things in ceval.c.
>The total patch is 190 lines, and that's because I need a new type
>(very verbose).  It implements "thread.atomic", which you use in a
>"with" statement.  So you write "with atomic:" and you get precisely
>the simple case described here:
>http://en.wikipedia.org/wiki/Software_transactional_memory#Proposed_language_support.
>
>Fwiw it's probably also a minimal patch to PyPy.  Sorry for being
>mostly off-topic for once...

Have you told python-dev about this?

Laura




From hakan at debian.org  Sat Aug 27 08:41:37 2011
From: hakan at debian.org (Hakan Ardo)
Date: Sat, 27 Aug 2011 08:41:37 +0200
Subject: [pypy-dev] [pypy-commit] pypy faster-nested-scopes: fix tests.
 no clue where the extra arraylen comes from?
In-Reply-To: <20110826211914.3B1E9820D0@wyvern.cs.uni-duesseldorf.de>
References: <20110826211914.3B1E9820D0@wyvern.cs.uni-duesseldorf.de>
Message-ID: <CAEkL+RgrQ4Xu9VJukT8oC2AOn2oEZQHWc0A_pm9e+W8aviAhqg@mail.gmail.com>

If a getarrayitem is moved into the short_preamble we also need guards
on the array len as it might be out of bound at the end of some future
bridge. After jit-short_from_state the jump of the loop is produced by
inlining the short preamble there (that approach does not crash if the
optimizers introduces new boxes). It is typically optimized out
entierly. In case of an array len check the guard is optimized out
while the arraylen remains. It should be killed by the backend though
as it's result is never used.

On Fri, Aug 26, 2011 at 11:19 PM, cfbolz <noreply at buildbot.pypy.org> wrote:
> Author: Carl Friedrich Bolz <cfbolz at gmx.de>
> Branch: faster-nested-scopes
> Changeset: r46806:77b0daebad60
> Date: 2011-08-26 23:25 +0200
> http://bitbucket.org/pypy/pypy/changeset/77b0daebad60/
>
> Log: ? ?fix tests. no clue where the extra arraylen comes from?
>
> diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py
> --- a/pypy/module/pypyjit/test_pypy_c/test_call.py
> +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py
> @@ -67,24 +67,17 @@
> ? ? ? ? assert log.opnames(ops) == ["guard_value",
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "getfield_gc", "guard_value",
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "getfield_gc", "guard_value",
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"getfield_gc", "guard_nonnull_class"]
> - ? ? ? ?# LOAD_GLOBAL of OFFSET but in different function partially folded
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"guard_not_invalidated"]
> + ? ? ? ?# LOAD_GLOBAL of OFFSET but in different function folded
> ? ? ? ? # away
> - ? ? ? ?# XXX could be improved
> ? ? ? ? ops = entry_bridge.ops_by_id('add', opcode='LOAD_GLOBAL')
> - ? ? ? ?assert log.opnames(ops) == ["guard_value", "getfield_gc", "guard_value"]
> + ? ? ? ?assert log.opnames(ops) == ["guard_not_invalidated"]
> ? ? ? ? #
> ? ? ? ? # two LOAD_GLOBAL of f, the second is folded away
> ? ? ? ? ops = entry_bridge.ops_by_id('call', opcode='LOAD_GLOBAL')
> - ? ? ? ?assert log.opnames(ops) == ["getfield_gc", "guard_nonnull_class"]
> + ? ? ? ?assert log.opnames(ops) == []
> ? ? ? ? #
> ? ? ? ? assert entry_bridge.match_by_id('call', """
> - ? ? ? ? ? ?p29 = getfield_gc(ConstPtr(ptr28), descr=<GcPtrFieldDescr pypy.objspace.std.celldict.ModuleCell.inst_w_value .*>)
> - ? ? ? ? ? ?guard_nonnull_class(p29, ConstClass(Function), descr=...)
> - ? ? ? ? ? ?p33 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_code .*>)
> - ? ? ? ? ? ?guard_value(p33, ConstPtr(ptr34), descr=...)
> - ? ? ? ? ? ?p35 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_w_func_globals .*>)
> - ? ? ? ? ? ?p36 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_closure .*>)
> ? ? ? ? ? ? p38 = call(ConstClass(getexecutioncontext), descr=<GcPtrCallDescr>)
> ? ? ? ? ? ? p39 = getfield_gc(p38, descr=<GcPtrFieldDescr pypy.interpreter.executioncontext.ExecutionContext.inst_topframeref .*>)
> ? ? ? ? ? ? i40 = force_token()
> @@ -100,19 +93,16 @@
> ? ? ? ? # -----------------------------
> ? ? ? ? loop, = log.loops_by_id('call')
> ? ? ? ? assert loop.match("""
> - ? ? ? ? ? ?i12 = int_lt(i5, i6)
> - ? ? ? ? ? ?guard_true(i12, descr=...)
> + ? ? ? ? ? ?guard_not_invalidated(descr=...)
> + ? ? ? ? ? ?i9 = int_lt(i5, i6)
> + ? ? ? ? ? ?guard_true(i9, descr=...)
> + ? ? ? ? ? ?i10 = force_token()
> + ? ? ? ? ? ?i12 = int_add(i5, 1)
> ? ? ? ? ? ? i13 = force_token()
> - ? ? ? ? ? ?i15 = int_add(i5, 1)
> - ? ? ? ? ? ?i16 = int_add_ovf(i15, i7)
> - ? ? ? ? ? ?guard_no_overflow(descr=...)
> - ? ? ? ? ? ?i18 = force_token()
> - ? ? ? ? ? ?i20 = int_add_ovf(i16, 1)
> - ? ? ? ? ? ?guard_no_overflow(descr=...)
> - ? ? ? ? ? ?i21 = int_add_ovf(i20, i7)
> + ? ? ? ? ? ?i15 = int_add_ovf(i12, 1)
> ? ? ? ? ? ? guard_no_overflow(descr=...)
> ? ? ? ? ? ? --TICK--
> - ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, i21, i6, i7, p8, p9, p10, p11, descr=<Loop0>)
> + ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, i15, i6, i7, p8, descr=<Loop0>)
> ? ? ? ? """)
>
> ? ? def test_method_call(self):
> diff --git a/pypy/module/pypyjit/test_pypy_c/test_globals.py b/pypy/module/pypyjit/test_pypy_c/test_globals.py
> --- a/pypy/module/pypyjit/test_pypy_c/test_globals.py
> +++ b/pypy/module/pypyjit/test_pypy_c/test_globals.py
> @@ -20,11 +20,9 @@
> ? ? ? ? ? ? guard_value(p10, ConstPtr(ptr11), descr=...)
> ? ? ? ? ? ? p12 = getfield_gc(p10, descr=<GcPtrFieldDescr .*W_DictMultiObject.inst_strategy .*>)
> ? ? ? ? ? ? guard_value(p12, ConstPtr(ptr13), descr=...)
> - ? ? ? ? ? ?p15 = getfield_gc(ConstPtr(ptr14), descr=<GcPtrFieldDescr .*ModuleCell.inst_w_value .*>)
> - ? ? ? ? ? ?guard_isnull(p15, descr=...)
> ? ? ? ? ? ? guard_not_invalidated(descr=...)
> ? ? ? ? ? ? p19 = getfield_gc(ConstPtr(p17), descr=<GcPtrFieldDescr .*W_DictMultiObject.inst_strategy .*>)
> ? ? ? ? ? ? guard_value(p19, ConstPtr(ptr20), descr=...)
> ? ? ? ? ? ? p22 = getfield_gc(ConstPtr(ptr21), descr=<GcPtrFieldDescr .*ModuleCell.inst_w_value .*>)
> ? ? ? ? ? ? guard_nonnull(p22, descr=...)
> - ? ? ? ?""")
> \ No newline at end of file
> + ? ? ? ?""")
> diff --git a/pypy/module/pypyjit/test_pypy_c/test_instance.py b/pypy/module/pypyjit/test_pypy_c/test_instance.py
> --- a/pypy/module/pypyjit/test_pypy_c/test_instance.py
> +++ b/pypy/module/pypyjit/test_pypy_c/test_instance.py
> @@ -52,7 +52,7 @@
> ? ? ? ? ? ? i10 = int_add_ovf(i5, i7)
> ? ? ? ? ? ? guard_no_overflow(descr=...)
> ? ? ? ? ? ? --TICK--
> - ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, i10, i6, p7, i7, p8, descr=<Loop0>)
> + ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, i10, i6, i7, p8, descr=<Loop0>)
> ? ? ? ? """)
>
> ? ? def test_getattr_with_dynamic_attribute(self):
> @@ -151,6 +151,7 @@
> ? ? ? ? assert loop.match_by_id('loadattr',
> ? ? ? ? '''
> ? ? ? ? guard_not_invalidated(descr=...)
> + ? ? ? ?i16 = arraylen_gc(p10, descr=<GcPtrArrayDescr>)
> ? ? ? ? i19 = call(ConstClass(ll_dict_lookup), _, _, _, descr=...)
> ? ? ? ? guard_no_exception(descr=...)
> ? ? ? ? i21 = int_and(i19, _)
> diff --git a/pypy/module/pypyjit/test_pypy_c/test_math.py b/pypy/module/pypyjit/test_pypy_c/test_math.py
> --- a/pypy/module/pypyjit/test_pypy_c/test_math.py
> +++ b/pypy/module/pypyjit/test_pypy_c/test_math.py
> @@ -47,6 +47,7 @@
> ? ? ? ? assert loop.match("""
> ? ? ? ? ? ? i2 = int_lt(i0, i1)
> ? ? ? ? ? ? guard_true(i2, descr=...)
> + ? ? ? ? ? ?guard_not_invalidated(descr=...)
> ? ? ? ? ? ? f1 = cast_int_to_float(i0)
> ? ? ? ? ? ? i3 = float_eq(f1, inf)
> ? ? ? ? ? ? i4 = float_eq(f1, -inf)
> @@ -60,4 +61,4 @@
> ? ? ? ? ? ? i7 = int_add(i0, f1)
> ? ? ? ? ? ? --TICK--
> ? ? ? ? ? ? jump(..., descr=)
> - ? ? ? ?""")
> \ No newline at end of file
> + ? ? ? ?""")
> diff --git a/pypy/module/pypyjit/test_pypy_c/test_string.py b/pypy/module/pypyjit/test_pypy_c/test_string.py
> --- a/pypy/module/pypyjit/test_pypy_c/test_string.py
> +++ b/pypy/module/pypyjit/test_pypy_c/test_string.py
> @@ -105,5 +105,5 @@
> ? ? ? ? ? ? i58 = int_add_ovf(i6, i57)
> ? ? ? ? ? ? guard_no_overflow(descr=...)
> ? ? ? ? ? ? --TICK--
> - ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, p5, i58, i7, i8, p9, p10, descr=<Loop4>)
> + ? ? ? ? ? ?jump(p0, p1, p2, p3, p4, p5, i58, i7, descr=<Loop4>)
> ? ? ? ? """)
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> http://mail.python.org/mailman/listinfo/pypy-commit
>



-- 
H?kan Ard?

From mitchell.hashimoto at gmail.com  Sat Aug 27 09:00:52 2011
From: mitchell.hashimoto at gmail.com (Mitchell Hashimoto)
Date: Sat, 27 Aug 2011 00:00:52 -0700
Subject: [pypy-dev] Question about "Completing 'os' module" (issue 833)
In-Reply-To: <CACOXMX5WvcCAr6oF+a_gzBWkPmZaygTyYirx5ax0PNDE0uXFwg@mail.gmail.com>
References: <CACOXMX6Na4o=fdzxSZ4Mamma579Z+MtScZ97Jh4gCVOcq6fh_Q@mail.gmail.com>
	<CAGmFidaJRCahuwB2L+9wTzTiHMshCQP=Agvk4d96FhMFQjfZoA@mail.gmail.com>
	<CACOXMX5WvcCAr6oF+a_gzBWkPmZaygTyYirx5ax0PNDE0uXFwg@mail.gmail.com>
Message-ID: <CACOXMX7Duez_oPbFbfzhMjzR44bN8YXNQ_nM_NESam66vwC1Kw@mail.gmail.com>

Sorry to ping the list again, but I've addressed the issues raised in the
issue to complete the "os.getlogin" feature. Is there any way I can get
another review to get this merged please?

Best,
Mitchell

On Mon, Aug 22, 2011 at 4:53 AM, Mitchell Hashimoto <
mitchell.hashimoto at gmail.com> wrote:

> Amaury,
>
> I've implemented one method (getlogin) and have created an issue + patch
> for it:
>
> https://bugs.pypy.org/issue841
>
> Best,
> Mitchell
>
> On Sun, Aug 21, 2011 at 2:20 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:
>
>> Hello,
>>
>> 2011/8/21 Mitchell Hashimoto <mitchell.hashimoto at gmail.com>
>>
>>> I noticed the 'os' module is incomplete, and I'd like to help complete
>>> this.
>>>
>>
>> You are very welcome!
>>
>>
>>> CPython does this by simply having these methods available on "posix"
>>> "nt" "os2" etc. and the "os" module imports those. It appears that PyPy does
>>> the same thing. I was able to successfully add 'getlogin' as practice, but I
>>> wanted to post here before going further. Some questions below:
>>>
>>> 1.) Should I mimic CPython and add the functionality to the OS-specific
>>> modules?
>>>
>>
>> Yes we should mimic CPython: fortunately these modules have different
>> names but share the same source file.
>> With CPython it's Modules/posixmodule.c, with PyPy it's in
>> pypy/module/posix.
>>
>>
>>> 2.) I don't have a Windows computer on hand. What is the standard
>>> practice for implementing some stdlib for one OS but not the other? Would
>>> PyPy accept this temporarily?
>>>
>>
>> Yes, no problem. In this case, I think it's best to let the test fail on
>> Windows so that someone may notice and fix it.
>>
>>
>>> 3.) There are many missing methods, to simplify implementation time and
>>> the patch, would it be okay to submit a patch for each stdlib method, so
>>> that this was built up over time?
>>>
>>
>>  Yes, smaller patches are easier to read and merge. I'd be happy to review
>> and commit them.
>>
>> --
>> Amaury Forgeot d'Arc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110827/e81b41ff/attachment.html>

From arigo at tunes.org  Sat Aug 27 12:45:57 2011
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Aug 2011 12:45:57 +0200
Subject: [pypy-dev] STM on CPython
In-Reply-To: <201108261929.p7QJToFD028177@theraft.openend.se>
References: <CAMSv6X2jNMd6NM5OUUQnmK2xD7-RQzYqZ+Q8vw=SJwu+s5mzng@mail.gmail.com>
	<20110826165751.1808.784020944.divmod.xquotient.1@localhost.localdomain>
	<201108261929.p7QJToFD028177@theraft.openend.se>
Message-ID: <CAMSv6X0870TgRhCTg63hvnmpLmPSHiB28q=yBhvt3oB7aBOPzQ@mail.gmail.com>

Hi,

On Fri, Aug 26, 2011 at 9:29 PM, Laura Creighton <lac at openend.se> wrote:
> Have you told python-dev about this?

Thanks for pushing me to :-)  Yes.

Armin

From neppord at gmail.com  Sat Aug 27 15:10:12 2011
From: neppord at gmail.com (Samuel Ytterbrink)
Date: Sat, 27 Aug 2011 15:10:12 +0200
Subject: [pypy-dev] Suggestions for small projects for getting started
 hacking on pypy?
In-Reply-To: <4E571744.70406@pianocktail.org>
References: <4E571744.70406@pianocktail.org>
Message-ID: <CABpX6DAm8jBabfNEw5YAC+k9k2HQba7xyvcSOufDC52ZC6xAyA@mail.gmail.com>

What part? The Interpreter or the tool chain? or usage of the Interpreter?

2011/8/26 Christian Hudon <chrish at pianocktail.org>

> Hello,
>
> I'm interested in getting my feet wet hacking on pypy. I've successfully
> bootstrapped pypy from source, using the documentation on doc.pypy.org.
> What small projects are open right now that I could use to get started
> learning about the code?
>
> Thanks,
>
>  Christian
>
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>



-- 
//Samuel Ytterbrink
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110827/fd9f54be/attachment.html>

From chrish at pianocktail.org  Sun Aug 28 00:29:23 2011
From: chrish at pianocktail.org (Christian Hudon)
Date: Sat, 27 Aug 2011 18:29:23 -0400
Subject: [pypy-dev] Suggestions for small projects for getting started
 hacking on pypy?
In-Reply-To: <CABpX6DAm8jBabfNEw5YAC+k9k2HQba7xyvcSOufDC52ZC6xAyA@mail.gmail.com>
References: <4E571744.70406@pianocktail.org>
	<CABpX6DAm8jBabfNEw5YAC+k9k2HQba7xyvcSOufDC52ZC6xAyA@mail.gmail.com>
Message-ID: <4E596FC3.7030706@pianocktail.org>

Le Sat Aug 27 09:10:12 2011, Samuel Ytterbrink a ?crit :
> What part? The Interpreter or the tool chain? or usage of the Interpreter?

Hmm. A bit of the first two, I guess. I'm not clear how "usage of the 
interpreter" would be any different from using CPython (except for 
things executing faster). Actually, I liked the suggestion from Wim 
(off list?) to start by improving bits of the cppyy C++ bridge. I'd 
like to take a look at moving this along a bit further, unless other 
people chime in saying that it's not something relevant to work on.

The only information I found about cppyy is a blog post from last 
Summer (CERN Sprint Report - Wrapping C++ Libraries). Is the 
information in that blog post still relevant? Which branch to I have to 
get? (I didn't find cppyy on the trunk). I assume if I have more 
specific questions about a specific piece of code while I'm coding, the 
best way to proceed is to ask on the IRC channel?

Thanks,

  Christian


From wlavrijsen at lbl.gov  Sun Aug 28 07:30:31 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Sat, 27 Aug 2011 22:30:31 -0700 (PDT)
Subject: [pypy-dev] Suggestions for small projects for getting started
 hacking on pypy?
In-Reply-To: <4E596FC3.7030706@pianocktail.org>
References: <4E571744.70406@pianocktail.org>
	<CABpX6DAm8jBabfNEw5YAC+k9k2HQba7xyvcSOufDC52ZC6xAyA@mail.gmail.com>
	<4E596FC3.7030706@pianocktail.org>
Message-ID: <alpine.LNX.2.00.1108272216520.31065@pc3-wlav.lbl.gov>

Hi Christian,

> Actually, I liked the suggestion from Wim (off list?) to start by improving
> bits of the cppyy C++ bridge.

yes; that was sent off the list.

> The only information I found about cppyy is a blog post from last Summer 
> (CERN Sprint Report - Wrapping C++ Libraries). Is the information in that 
> blog post still relevant?

Yes, except the loading of the reflection library (load_reflection_info
instead of load_lib). A status update posting is forthcoming. C++ is a large
language and I'm not working on this full-time. Progress is steady, though.

More sample code is available in the tests of cppyy/test.

> Which branch to I have to get? (I didn't find cppyy on the trunk).

It's in the reflex-support branch. I'm keeping the branch close to default
by regularly merging. With a few exceptions (the use of arrays), the only
real difference between default and this branch is the cppyy module.

Another thing that is necessary (for now) is ROOT (for either Reflex or
the CINT backend). See root.cern.ch. In the long run, llvm pre-compiled
headers should become the main backend (this is true for ROOT as well, see
http://root.cern.ch/drupal/content/cling).

To get the best results, the reflex patch found under module cppyy needs
to be applied to genreflex. See the Makefile in test in how the generation
of reflection info works.

> I assume if I have more specific questions about a specific 
> piece of code while I'm coding, the best way to proceed is to ask on the
> IRC channel?

That will very likely give you most answers, but myself I'm not on IRC.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From lac at openend.se  Sun Aug 28 09:50:33 2011
From: lac at openend.se (Laura Creighton)
Date: Sun, 28 Aug 2011 09:50:33 +0200
Subject: [pypy-dev] [COMMENT] Pull request #8 for pypy/pypy: added
	numpy.sort, tests, and docstring for numpy.array.sort
In-Reply-To: Message from Bitbucket <notifications-noreply@bitbucket.org>
	of "Sun, 28 Aug 2011 06:32:52 -0000."
	<20110828063252.18999.23378@bitbucket01.managed.contegix.com>
References: <20110828063252.18999.23378@bitbucket01.managed.contegix.com>
Message-ID: <201108280750.p7S7oX5p023466@theraft.openend.se>

I'd like it if the comments on pull requests came with the name
of the person making the request, in this case yasirs.  Is this
something we can do on our end, (I don't think so) or something
we need to ask Bitbucket to change?

Laura

In a message of Sun, 28 Aug 2011 06:32:52 -0000, Bitbucket writes:
>New comment on pull request:
>
>https://bitbucket.org/pypy/pypy/pull-request/8/added-numpysort-tests-and-
>do=
>cstring-for#comment-35
>
>Alex Gaynor (alex_gaynor) said:
>
>Sorry, here are the issues I saw:
>
>* It would need to be rewritten after the merge of the dtypes branch (whi
>ch=
> rewrites most of everything :P)
>* It doesn't work with any of the virtual arrays (I'm surprised it even t
>ra=
>nslates)
>* It looks like a pretty simple quicksort, which means it can have perfor
>ma=
>nce issues (I'd like to try using the TimSort we already have)
>* There are stylistic issues, semicolons, and excessive parens mostly.
>
>--
>This is a pull request comment notification from bitbucket.org.
>You are receiving this either because you are participating
>in a pull request, or you are following it.

From jan at jandecaluwe.com  Sun Aug 28 20:01:21 2011
From: jan at jandecaluwe.com (Jan Decaluwe)
Date: Sun, 28 Aug 2011 20:01:21 +0200
Subject: [pypy-dev] Great results for MyHDL simulations
In-Reply-To: <4DEE9AE7.6020005@jandecaluwe.com>
References: <4DEE3D94.1020408@jandecaluwe.com> <4DEE4267.3020304@gmail.com>
	<4DEE9AE7.6020005@jandecaluwe.com>
Message-ID: <4E5A8271.7030002@jandecaluwe.com>

On 06/07/2011 11:40 PM, Jan Decaluwe wrote:
> On 06/07/2011 05:23 PM, Antonio Cuni wrote:
>> On 07/06/11 17:02, Jan Decaluwe wrote:
>>> I am seeing great improvements for MyHDL simulations
>>> by using PyPy, and I have written a page about it:
>>>
>>> http://www.myhdl.org/doku.php/performance
>>
>> Hello Jan,
>>
>> this is really nice to hear :-)
>>
>> Did you try to run the benchmarks with a more recent version of PyPy?
>> According to this chart, we are up to 30% faster than 1.5 on some benchmarks,
>> so you might get even better results:
>
> Not yet, I like to leave some further excitement for later :-)
>
>  From now on, I plan to track the evolution of my benchmarks
> with official PyPy releases.

Getting better all the time :-)

With PyPy 1.6, I see additional significant improvements (probably
also thanks to the generator-specific speedup). Speedup compared
to cPython is now 8-20x (was 6-12x).

http://www.myhdl.org/doku.php/performance

Thanks to all!

-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
     Python as a HDL: http://www.myhdl.org
     VHDL development, the modern way: http://www.sigasi.com
     World-class digital design: http://www.easics.com

From anto.cuni at gmail.com  Mon Aug 29 00:49:44 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 29 Aug 2011 00:49:44 +0200
Subject: [pypy-dev] [COMMENT] Pull request #8 for pypy/pypy: added
 numpy.sort, tests, and docstring for numpy.array.sort
In-Reply-To: <201108280750.p7S7oX5p023466@theraft.openend.se>
References: <20110828063252.18999.23378@bitbucket01.managed.contegix.com>
	<201108280750.p7S7oX5p023466@theraft.openend.se>
Message-ID: <4E5AC608.4020004@gmail.com>

On 28/08/11 09:50, Laura Creighton wrote:
> I'd like it if the comments on pull requests came with the name
> of the person making the request, in this case yasirs.  Is this
> something we can do on our end, (I don't think so) or something
> we need to ask Bitbucket to change?

I think it's something that we need to ask to Bitbucket.

From dirkjan at ochtman.nl  Mon Aug 29 10:21:03 2011
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Mon, 29 Aug 2011 10:21:03 +0200
Subject: [pypy-dev] Duplicated guards
In-Reply-To: <CAEkL+RjdMQLLq3=oGS9rkHuVuXSjA5EABLoB20bP=zf4mFcyKw@mail.gmail.com>
References: <CAEkL+RiTBuwLX7U-xjgAUgY_BaTU5Gm=VPbxbaXN_gAC4FxqfA@mail.gmail.com>
	<CAFRnB2V2FbUDCZDMQpsCA+tjqPYXmF1X4fCvHcHqe_0zDPsrrQ@mail.gmail.com>
	<CAEkL+RjdMQLLq3=oGS9rkHuVuXSjA5EABLoB20bP=zf4mFcyKw@mail.gmail.com>
Message-ID: <CAKmKYaC2HTUD2RRb0_tBbkDmojL-Qbeao1nJze28W4qLcpsTzQ@mail.gmail.com>

On Sat, Aug 20, 2011 at 18:04, Hakan Ardo <hakan at debian.org> wrote:
> On 8/20/11, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>> I believe mozilla worked on this issue. ?if.you're interested in this
>> perhaps we should get you in contact with them.
>
> Good idea. Do you have some contact?

You might just drop into #jsapi on irc.mozilla.org and ask around
(most of them are in US/Pacific).

Cheers,

Dirkjan

From van.lindberg at gmail.com  Tue Aug 30 18:23:29 2011
From: van.lindberg at gmail.com (VanL)
Date: Tue, 30 Aug 2011 11:23:29 -0500
Subject: [pypy-dev] Saving and reloading JIT optimizations
In-Reply-To: <4E515D42.1050107@gmx.de>
References: <4aca052c-7594-471e-816e-157d4f1b1d72@jackdaw.local>	<5d5c1b67-9bcd-4858-8d02-413db5acc17e@jackdaw.local>	<CAMSv6X0Z25DwHssQOxRZUQMk5yAb815rrA5ujOfP+Wi2c_cgpA@mail.gmail.com>	<CANdZDc7+5ENO6H184Sx=33E15RquqHP9Gu+4fxZFoQyu08TNBw@mail.gmail.com>	<CAMSv6X2ZgZ+W2RZx1XH9u2HmbgVRTv3TM6cyM6xKD92jq60djQ@mail.gmail.com>	<CANdZDc7tZUfg1+WQty6y9xhfmQ4NfkRXiCeKQjdVSXnN+byJ1g@mail.gmail.com>
	<CANdZDc5q0gh1ZqDMm913bsqaCuAsBuG7SYX++D8JQG0adoAOSQ@mail.gmail.com>
	<4E515D42.1050107@gmx.de>
Message-ID: <4E5D0E81.5030004@gmail.com>

On 8/21/2011 2:32 PM, Carl Friedrich Bolz wrote:
>
> Solving the problem of persisting source code is *seriously* hard, and 
> I agree with Armin, I'm not going to invest any time in it.

One interesting side note: The implementors of v8 decided that the best 
and most compact representation of source code was the code itself. 
Accordingly, they keep a representation of it around for reference and 
use in different parts of the interpreter loop.

Thanks,

Van
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110830/98f85189/attachment.html>

From aaron.devore at gmail.com  Tue Aug 30 23:12:52 2011
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Tue, 30 Aug 2011 14:12:52 -0700
Subject: [pypy-dev] Policy on Python 3 and Python 2 executable names
Message-ID: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>

>From what I can tell, PyPy doesn't have a policy on how package
maintainers should differentiate between Python 2 and Python 3
executables. I brought up adding "pypy2" on #pypy in March, but the
conversation quickly died.

CPython ran into the no-policy problem when Arch Linux decided to
switch /usr/bin/python to be Python 3. There wasn't a naming policy
for /usr/bin/python at that point. The PEP that represents the
relevant python-dev mailing list thread is PEP 394[1]. PEP 394 isn't
finished yet, but should be soon.

Possible solution [who uses it for CPython]:
1) pypy arbitrary, pypy2 for Python 2, pypy3 for Python 3 [PEP 394, Gentoo]
2) pypy for Python 2, pypy3 for Python 3, no pypy2 [Debian family]
3) pypy for Python 2, pypy2 for Python 2, pypy3 for Python 3 [Red Hat family]
4) pypy for Python 3, pypy2 for Python 2, pypy3 for Python 3 [Arch Linux]

I prefer #1 because it sticks with PEP 394 and is future-proof. #2 and
#3 take away future flexibility. #4 is a horrible, horrible idea at
present.

-Aaron DeVore

[1] http://www.python.org/dev/peps/pep-0394/

From gertjanvanzwieten at gmail.com  Tue Aug 30 23:14:22 2011
From: gertjanvanzwieten at gmail.com (Gertjan van Zwieten)
Date: Tue, 30 Aug 2011 23:14:22 +0200
Subject: [pypy-dev] your thoughts on low level optimizations
In-Reply-To: <CAMGQJJWYHuVqMXHLZOnV0hM4yerQHA5pMEULucB22ciiCSR0HQ@mail.gmail.com>
References: <CAMGQJJWYHuVqMXHLZOnV0hM4yerQHA5pMEULucB22ciiCSR0HQ@mail.gmail.com>
Message-ID: <CAMGQJJVVZtTcfY-eKNnKuqmU9rD3MRbj8RsMY+yxKTWTXGbAOA@mail.gmail.com>

Dear all,

Following today's blog post [1] about wrapping C++ libraries, I would like
to take the opportunity to get some opinions from the PyPy community on a
related topic. I hope this mailing list is the appropriate place for this
sort of discussion.

My name is Gertjan van Zwieten and I have been a Python enthusiast since
long. I have used it extensively in the context of my PhD project, which was
related to Finite Element modelling, and I have recently taken steps
together with a friend of mine towards continuing on this path commercially
[2]. We intend to design our own Finite Element framework, which we aim to
be open source, written largely in Python. It is the 'largely' that I would
like to discuss here.

I have always argued to people that, yes, it is possible to do
computationally intensive things in an interpreted language, as long as the
majority of work is done by compiled components. For me the obvious example
of that is Numpy, which provides a rather specific data type and a set of
vectorized operations, as well as an interface with optimized linear algebra
libraries. This leaves Python as a convenient and very powerful glue
language to connect these optimized components into something purposeful.
With that conceptualization in mind, my preferred route for optimizing code
was to identify critical components of generic nature, and re-implement them
as a plain C module using the Python API.

So this is how I used to look at things. I would say that things were nice
and clear this way, until PyPy started throwing stones in the water.

I have not so far used PyPy in any actual application, mainly (if not only)
because of lacking Numpy support. But I can see it holds great promise,
especially for the computational community (I am also eagerly following the
transactional memory developments) and I anticipate making the switch after
it matures a little bit further. However, that does put me in the situation
that, currently, I am not sure anymore what things to implement in C, if
any, and in what form.

Right now I am prototyping our framework using only Numpy and pure Python. I
avoid premature optimization as much as possible, and in fact at times find
myself sacrificing speed for elegance, arguing that I will be able to bring
back efficiency later on in a deeper layer. For example, rather than
adopting a numbering scheme for the many computational elements such as is
common in Finite Element implementations, I have moved to a more object
oriented approach that is more flexible and allows for more error checking,
but that forces me to put in dictionaries what used to be Numpy arrays.
Obviously dictionaries were not meant to be numeric data structures, and
it's a typical component that I would try to implement efficiently in C
eventually.

With PyPy at the horizon, I am not so sure anymore. For one I'm not sure if
PyPy will ever be able to use the C module, or use it efficiently; I could
understand if support exists merely for sake of compatibility. I am also not
certain if I should maybe forget about C altogether and rely on the JIT to
compile the Python for loops that I have always tried to avoid. Would you go
as far as to say that there will be no more reason for low level programming
whatsoever? Or would you advise to write the component in RPython and use
the translator to compile it? With my poor overview of these things, there
are very few arguments that I can formulate in favour or against any of
these options.

Regarding today's blog post, I have the feeling that this is meant more for
wrapping existing C++ libraries than for starting new ones, is that correct?
Or if not, and it is, in fact, an option to consider, will this be able to
work in CPython, too? That would make the transition a bit easier,
obviously.

I am very interested what your views are on this topic of optimizations.

Best regards and thanks a lot for working on PyPy!

Gertjan

[1]
http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.html
[2] http://hvzengineering.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110830/c4b92adf/attachment.html>

From amauryfa at gmail.com  Wed Aug 31 00:19:44 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 31 Aug 2011 00:19:44 +0200
Subject: [pypy-dev] Policy on Python 3 and Python 2 executable names
In-Reply-To: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>
References: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>
Message-ID: <CAGmFidYb7gdp=kCJbyk6WfV51WroWCa6ADO=QxEqBNq6jper=g@mail.gmail.com>

2011/8/30 Aaron DeVore <aaron.devore at gmail.com>

> Possible solution [who uses it for CPython]:
> 1) pypy arbitrary, pypy2 for Python 2, pypy3 for Python 3 [PEP 394, Gentoo]
> 2) pypy for Python 2, pypy3 for Python 3, no pypy2 [Debian family]
> 3) pypy for Python 2, pypy2 for Python 2, pypy3 for Python 3 [Red Hat
> family]
> 4) pypy for Python 3, pypy2 for Python 2, pypy3 for Python 3 [Arch Linux]
>
> I prefer #1 because it sticks with PEP 394 and is future-proof. #2 and
> #3 take away future flexibility. #4 is a horrible, horrible idea at
> present.
>

As far as I care, #1 also looks better to me, because it follows CPython
conventions.

But the ultimate solution is certainly:
0) python for arbitrary PyPy, python2 for PyPy 2, and python3 for PyPy 3!

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110831/ef317aa4/attachment.html>

From jnoller at gmail.com  Wed Aug 31 02:28:03 2011
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 30 Aug 2011 20:28:03 -0400
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
Message-ID: <CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>

(Re-sending intentionally - I wasn't subbed to pypy-dev and got rejected)

On Tue, Aug 30, 2011 at 8:22 PM, Jesse Noller <jnoller at gmail.com> wrote:
> Here's a summary (nicely put together by Nick Coghlan):
>
> - OSU/OSL have set up the machine itself (details in the July list archives)
> - I am the machine admin (I have root)
> - Tennessee and Miquel are working on getting codespeed up and running
>
> - I and Maciej are already too busy to handle the coordination/admin
> side of things
>
> I just added the "speedracer" account - a common account to setup
> things as needed. Currently, I don't see activity from Miquel on
> setting up the codespeed instance. I know Tennessee has been looking
> at setting up the runner.
>
> What we need is this, very simply:
>
> 1 - A working, running codespeed instance installed
> 2 - Benchmark runner(s) setup and going - for PyPy and CPython 2.x
> since the benchmarks don't run on Python 3 yet.
>
>
> Other needs/notes:
> - a project issue tracker doesn't appear to exist yet (needs to be done)
> - a better 'placeholder/about' page would be useful that offered links to:
> ?- the speed at python.org mailing list signup page
> ?- the issue tracker (once it exists)
> ?- the codespeed repo
>
> Ideally, the front landing page would end up looking like:
> http://speed.pypy.org/
>
> - the aim is to eventually have the web front end running on different
> (but still OSU/OSL hosted) hardware, although that likely isn't
> critical right now
>
> - the current 5 GB /tmp size may be a problem for PyPy
>
> Where do we go from here - I can not be the one to keep pestering the
> group to make forward progress. On behalf of the PSF, and the CPython
> and PyPy teams, I got us a donated platform for running these shared,
> common benchmarks. Right now we have a 10,000$+ paper weight taking
> rack space in OSU/OSL.
>
> I need someone to take lead on this and get the project moving forward
> - as a reminder, here was the original proposal I made to the board,
> after PyCon 2011:
>
> """
> Coming out of the PyCon VM and language summits, it was commonly
> agreed that PyPy, CPython, IronPython and Jython should strive to move
> to a common set of benchmarks and a single performance-oriented site.
> This was agreed upon by the maintainers present at the conference, and
> included leads from all of the major implementations. This project
> would be lead by the various VM development teams, and based around
> the Unladen Swallow/PyPy benchmark suite. The site would be derived
> from:
>
> http://speed.pypy.org/
>
> There are already GSOC students potentially lined up to work on
> porting the test suite to Python 3!
>
> There, of course, is the requirement that we have:
>
> 1> A machine
> 2> Hosting for that machine
>
> And I took the job of finding both. As the OSU/OSL [1] has come up
> several times in discussions about free, monitored hosting, I felt
> that now would be the time to float the trial balloon and start a
> continuing relationship with the OSU/OSL team. I got to speak with
> some of the team at PyCon, and I was quite impressed with their
> enthusiasm and willingness to help out.
> """
>
> I need help from everyone to get this up and running: the PyPy team is
> most familiar with the speed.pypy.org/codespeed system. If we could
> get a dump of what needs to be done to get things setup, that may help
> get more hands involved.
>
> I am feeling personally responsible that this has not been moving
> forward - I need your help to get this up and running for the
> betterment of CPython and PyPy. This project benefits both projects,
> and Python in general immensely.
>
> Once up and running, this system will be prominently placed on the
> Python.org home page, and be used by potential users to help select
> and evaluate their Python runtime.
>
> Jesse
>

From alex.gaynor at gmail.com  Wed Aug 31 02:35:13 2011
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 30 Aug 2011 20:35:13 -0400
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>
Message-ID: <CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>

I've gotten Noah Kantrowitz on here, he's doing some other infrastructure
stuff for the PSF, and would like to include speed.python.org in the proper
organization of the machines, rather than ad-hoc "everyone installs what
they think it needs" :)

Alex

On Tue, Aug 30, 2011 at 8:28 PM, Jesse Noller <jnoller at gmail.com> wrote:

> (Re-sending intentionally - I wasn't subbed to pypy-dev and got rejected)
>
> On Tue, Aug 30, 2011 at 8:22 PM, Jesse Noller <jnoller at gmail.com> wrote:
> > Here's a summary (nicely put together by Nick Coghlan):
> >
> > - OSU/OSL have set up the machine itself (details in the July list
> archives)
> > - I am the machine admin (I have root)
> > - Tennessee and Miquel are working on getting codespeed up and running
> >
> > - I and Maciej are already too busy to handle the coordination/admin
> > side of things
> >
> > I just added the "speedracer" account - a common account to setup
> > things as needed. Currently, I don't see activity from Miquel on
> > setting up the codespeed instance. I know Tennessee has been looking
> > at setting up the runner.
> >
> > What we need is this, very simply:
> >
> > 1 - A working, running codespeed instance installed
> > 2 - Benchmark runner(s) setup and going - for PyPy and CPython 2.x
> > since the benchmarks don't run on Python 3 yet.
> >
> >
> > Other needs/notes:
> > - a project issue tracker doesn't appear to exist yet (needs to be done)
> > - a better 'placeholder/about' page would be useful that offered links
> to:
> >  - the speed at python.org mailing list signup page
> >  - the issue tracker (once it exists)
> >  - the codespeed repo
> >
> > Ideally, the front landing page would end up looking like:
> > http://speed.pypy.org/
> >
> > - the aim is to eventually have the web front end running on different
> > (but still OSU/OSL hosted) hardware, although that likely isn't
> > critical right now
> >
> > - the current 5 GB /tmp size may be a problem for PyPy
> >
> > Where do we go from here - I can not be the one to keep pestering the
> > group to make forward progress. On behalf of the PSF, and the CPython
> > and PyPy teams, I got us a donated platform for running these shared,
> > common benchmarks. Right now we have a 10,000$+ paper weight taking
> > rack space in OSU/OSL.
> >
> > I need someone to take lead on this and get the project moving forward
> > - as a reminder, here was the original proposal I made to the board,
> > after PyCon 2011:
> >
> > """
> > Coming out of the PyCon VM and language summits, it was commonly
> > agreed that PyPy, CPython, IronPython and Jython should strive to move
> > to a common set of benchmarks and a single performance-oriented site.
> > This was agreed upon by the maintainers present at the conference, and
> > included leads from all of the major implementations. This project
> > would be lead by the various VM development teams, and based around
> > the Unladen Swallow/PyPy benchmark suite. The site would be derived
> > from:
> >
> > http://speed.pypy.org/
> >
> > There are already GSOC students potentially lined up to work on
> > porting the test suite to Python 3!
> >
> > There, of course, is the requirement that we have:
> >
> > 1> A machine
> > 2> Hosting for that machine
> >
> > And I took the job of finding both. As the OSU/OSL [1] has come up
> > several times in discussions about free, monitored hosting, I felt
> > that now would be the time to float the trial balloon and start a
> > continuing relationship with the OSU/OSL team. I got to speak with
> > some of the team at PyCon, and I was quite impressed with their
> > enthusiasm and willingness to help out.
> > """
> >
> > I need help from everyone to get this up and running: the PyPy team is
> > most familiar with the speed.pypy.org/codespeed system. If we could
> > get a dump of what needs to be done to get things setup, that may help
> > get more hands involved.
> >
> > I am feeling personally responsible that this has not been moving
> > forward - I need your help to get this up and running for the
> > betterment of CPython and PyPy. This project benefits both projects,
> > and Python in general immensely.
> >
> > Once up and running, this system will be prominently placed on the
> > Python.org home page, and be used by potential users to help select
> > and evaluate their Python runtime.
> >
> > Jesse
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110830/7bf8a7b5/attachment.html>

From jnoller at gmail.com  Wed Aug 31 02:37:05 2011
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 30 Aug 2011 20:37:05 -0400
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>
	<CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
Message-ID: <CACQrdOn78zrvHJ=+Z-Od+TKjzfbW_XJDVPcZ7aOW7VmMFE2Fwg@mail.gmail.com>

Yes, I should have looped Noah in sooner.

I have all the keys / passwords. Right now we need:

1. codespeed
2. benchmark runners

I've installed a very basic system with Django, apache, mod_wsgi

On Tue, Aug 30, 2011 at 8:35 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> I've gotten Noah Kantrowitz on here, he's doing some other infrastructure
> stuff for the PSF, and would like to include speed.python.org in the proper
> organization of the machines, rather than ad-hoc "everyone installs what
> they think it needs" :)
> Alex
>
> On Tue, Aug 30, 2011 at 8:28 PM, Jesse Noller <jnoller at gmail.com> wrote:
>>
>> (Re-sending intentionally - I wasn't subbed to pypy-dev and got rejected)
>>
>> On Tue, Aug 30, 2011 at 8:22 PM, Jesse Noller <jnoller at gmail.com> wrote:
>> > Here's a summary (nicely put together by Nick Coghlan):
>> >
>> > - OSU/OSL have set up the machine itself (details in the July list
>> > archives)
>> > - I am the machine admin (I have root)
>> > - Tennessee and Miquel are working on getting codespeed up and running
>> >
>> > - I and Maciej are already too busy to handle the coordination/admin
>> > side of things
>> >
>> > I just added the "speedracer" account - a common account to setup
>> > things as needed. Currently, I don't see activity from Miquel on
>> > setting up the codespeed instance. I know Tennessee has been looking
>> > at setting up the runner.
>> >
>> > What we need is this, very simply:
>> >
>> > 1 - A working, running codespeed instance installed
>> > 2 - Benchmark runner(s) setup and going - for PyPy and CPython 2.x
>> > since the benchmarks don't run on Python 3 yet.
>> >
>> >
>> > Other needs/notes:
>> > - a project issue tracker doesn't appear to exist yet (needs to be done)
>> > - a better 'placeholder/about' page would be useful that offered links
>> > to:
>> > ?- the speed at python.org mailing list signup page
>> > ?- the issue tracker (once it exists)
>> > ?- the codespeed repo
>> >
>> > Ideally, the front landing page would end up looking like:
>> > http://speed.pypy.org/
>> >
>> > - the aim is to eventually have the web front end running on different
>> > (but still OSU/OSL hosted) hardware, although that likely isn't
>> > critical right now
>> >
>> > - the current 5 GB /tmp size may be a problem for PyPy
>> >
>> > Where do we go from here - I can not be the one to keep pestering the
>> > group to make forward progress. On behalf of the PSF, and the CPython
>> > and PyPy teams, I got us a donated platform for running these shared,
>> > common benchmarks. Right now we have a 10,000$+ paper weight taking
>> > rack space in OSU/OSL.
>> >
>> > I need someone to take lead on this and get the project moving forward
>> > - as a reminder, here was the original proposal I made to the board,
>> > after PyCon 2011:
>> >
>> > """
>> > Coming out of the PyCon VM and language summits, it was commonly
>> > agreed that PyPy, CPython, IronPython and Jython should strive to move
>> > to a common set of benchmarks and a single performance-oriented site.
>> > This was agreed upon by the maintainers present at the conference, and
>> > included leads from all of the major implementations. This project
>> > would be lead by the various VM development teams, and based around
>> > the Unladen Swallow/PyPy benchmark suite. The site would be derived
>> > from:
>> >
>> > http://speed.pypy.org/
>> >
>> > There are already GSOC students potentially lined up to work on
>> > porting the test suite to Python 3!
>> >
>> > There, of course, is the requirement that we have:
>> >
>> > 1> A machine
>> > 2> Hosting for that machine
>> >
>> > And I took the job of finding both. As the OSU/OSL [1] has come up
>> > several times in discussions about free, monitored hosting, I felt
>> > that now would be the time to float the trial balloon and start a
>> > continuing relationship with the OSU/OSL team. I got to speak with
>> > some of the team at PyCon, and I was quite impressed with their
>> > enthusiasm and willingness to help out.
>> > """
>> >
>> > I need help from everyone to get this up and running: the PyPy team is
>> > most familiar with the speed.pypy.org/codespeed system. If we could
>> > get a dump of what needs to be done to get things setup, that may help
>> > get more hands involved.
>> >
>> > I am feeling personally responsible that this has not been moving
>> > forward - I need your help to get this up and running for the
>> > betterment of CPython and PyPy. This project benefits both projects,
>> > and Python in general immensely.
>> >
>> > Once up and running, this system will be prominently placed on the
>> > Python.org home page, and be used by potential users to help select
>> > and evaluate their Python runtime.
>> >
>> > Jesse
>> >
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>
>

From wlavrijsen at lbl.gov  Wed Aug 31 03:04:47 2011
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Tue, 30 Aug 2011 18:04:47 -0700 (PDT)
Subject: [pypy-dev] your thoughts on low level optimizations
In-Reply-To: <CAMGQJJVVZtTcfY-eKNnKuqmU9rD3MRbj8RsMY+yxKTWTXGbAOA@mail.gmail.com>
References: <CAMGQJJWYHuVqMXHLZOnV0hM4yerQHA5pMEULucB22ciiCSR0HQ@mail.gmail.com>
	<CAMGQJJVVZtTcfY-eKNnKuqmU9rD3MRbj8RsMY+yxKTWTXGbAOA@mail.gmail.com>
Message-ID: <alpine.LNX.2.00.1108301804400.28083@pc3-wlav.lbl.gov>

Hi Gertjan,

> With PyPy at the horizon, I am not so sure anymore. For one I'm not sure if
> PyPy will ever be able to use the C module, or use it efficiently

it's only the crossing into and out of a C extension module through cpyext
that is less than optimal. If that crossing does not happen often, the penalty
of the cross-language call overhead does not matter. Another option that you
have is to write an "extension" module in Python or RPython that just calls
into the C code through ctypes or libffi. That will be efficient.

Just keep the C module clean and simple: clear memory ownership rules, no use
of CPython internals, callbacks only on the Python side, etc. That way, any
extra code needed for glue is simple to write and maintain.

> I am also not certain if I should maybe forget about C altogether and rely
> on the JIT to compile the Python for loops that I have always tried to avoid.
> Would you go as far as to say that there will be no more reason for low level
> programming whatsoever? Or would you advise to write the component in RPython
> and use the translator to compile it?

Whatever works best, they all have their pro's and cons. With C, you can share
between PyPy and CPython and true low-level coding is by far the easiest in it.
RPython allows you to provide hints to the JIT which may make a big difference
in performance, but it isn't modular: RPython components become part of pypy-c
after the translation. Programming in Python is the cleanest, shares easily,
but performance behavior may be surprising, especially in tight loops.

I realize that that isn't an answer, but I don't think there is a single one
answer that is valid in all cases. Your original approach is most likely to
still be best: stick to Python until you find a performance problem, only then
look for and evaluate your other options.

> Regarding today's blog post, I have the feeling that this is meant more for
> wrapping existing C++ libraries than for starting new ones, is that correct?

Well, of course you can still write a new C++ library and wrap that too. And
in the process make it Python-bindings-friendly. :) But since your original
code was C, ctypes is likely easier to use since you can code in the needed
reflection info (you can also run Reflex over the C code and use that to
generate the Python-side annotations in a build process).

> Or if not, and it is, in fact, an option to consider, will this be able to
> work in CPython, too?

The code is based on an equivalent that exists for CPython. However, that is
not a standalone package, but part of ROOT (http://root.cern.ch). It could be
standalone, but I never saw the need: if you do not already have reflection
information libraries ("dictionaries") as we did for I/O, then SWIG or an
equivalent tool works just as well and is already available.

Best regards,
            Wim

P.S. I noticed on your web site that you're working on a project for Oce. I
      used to work there in times gone by. :)
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From lac at openend.se  Wed Aug 31 03:10:37 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 31 Aug 2011 03:10:37 +0200
Subject: [pypy-dev] can we change the mailing list
Message-ID: <201108310110.p7V1Abhw032013@theraft.openend.se>

so that it doesn't automatically bounce mail from non-subscribers
but instead holds them?  Now that we are on python.org we won't
drown in spam as python.org is most excellent at catching it before
it reaches the mailing list.  If you make me a list admin, I promise
to do all the work in approving things.

We just missed a nice note from John Pinner who wants somebody to
come talk at PyCON UK.

Laura


From arigo at tunes.org  Wed Aug 31 09:13:19 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 31 Aug 2011 09:13:19 +0200
Subject: [pypy-dev] Policy on Python 3 and Python 2 executable names
In-Reply-To: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>
References: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>
Message-ID: <CAMSv6X12XWqawf3zvcgRqOCP5J=j+ZMf0RaBrD7ejQ2vB7kkGg@mail.gmail.com>

Hi Aaron,

On Tue, Aug 30, 2011 at 11:12 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> From what I can tell, PyPy doesn't have a policy on how package
> maintainers should differentiate between Python 2 and Python 3
> executables. I brought up adding "pypy2" on #pypy in March, but the
> conversation quickly died.

I suppose that we need to *have* a pypy3 first, before any
conversation like that really makes sense.  Last March this wasn't
even being considered.  Now it is maybe in some very draftish early
planning stage.


Armin

From boemmels at web.de  Wed Aug 31 09:54:13 2011
From: boemmels at web.de (Juergen Boemmels)
Date: Wed, 31 Aug 2011 09:54:13 +0200 (CEST)
Subject: [pypy-dev] Failing App-Level-Test test_posix2.py
Message-ID: <189483600.102.1314777253718.JavaMail.fmail@mwmweb068>

Hi,

Since some weeks in one Applevel-test is failing consitently on the
buildbot:
module/posix/test/test_posix2.py
with the failure:
??????? if sys.platform.startswith('linux'):
>?????????? assert hasattr(st, 'st_rdev')
E?????????? assert hasattr(posix.stat_result(st_mode=33152,st_ino=1588275L, st_dev=64256L, st_nlink=1,s...integer_atime=1314759734, _integer_mtime=1314759734,_integer_ctime=1314759734), 'st_rdev')
module/posix/test/test_posix2.py:135: AssertionError ================ 1 failed, 77 passed, 6 skipped in 4.80 seconds ================
So it seems that st_rdev is not available

I tracked this problem down to a chicken-egg problem. The relevant
portion is here in
pypy/rpython/module/ll_os_stat.py:
# for now, check the host Python to know which st_xxx fields exist
STAT_FIELDS = [(_name, _TYPE) for (_name, _TYPE) in ALL_STAT_FIELDS
????????????????????????????? if hasattr(os.stat_result, _name)]

As pypy is meanwhile build with pypy, relying on the host python
is not a good idea. A pypy without st_rdev will again build only
a new pypy without st_rdev, even if the platform supports st_rdev.

As an experiment I bootstrapped pypy again with python an the
error disappered
# python translate.py -O2 targetpypystandalone.py
# ./pypy-c ../../../pytest.py ../../module/posix/test/test_posix2.py -A
========================= test session starts ==========================
platform linux2 -- Python 2.7.1[pypy-1.6.0] -- pytest-2.1.0.dev4
pytest-2.1.0.dev4 from /home/boemmels/src/pypy/pytest.pyc
collected 80 items

../../module/posix/test/test_posix2.py
...........................................................s............
........

================= 79 passed, 1 skipped in 5.07 seconds =================
This bootstrapped pypy-c is also cappable of rebuilding an pypy with
st_rdev enabled. But this is not a clean solution.

I think the way to go is to no longer rely on the Host-Python, but
use the appropiate configure magic. Something like
#ifdef HAVE_STRUCT_STAT_ST_RDEV
but in a pythonic way. Unfortunatly I'm not familar with pypy's
configuration system so I got stuck here.

Can anybody tell me how to test for available struct members like
AC_CHECK_MEMBERS([struct stat.st_rdev]) in autoconf?


Regards
juergen
___________________________________________________________
Schon geh?rt? WEB.DE hat einen genialen Phishing-Filter in die
Toolbar eingebaut! http://produkte.web.de/go/toolbar

From anto.cuni at gmail.com  Wed Aug 31 10:12:45 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 31 Aug 2011 10:12:45 +0200
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CACQrdOn78zrvHJ=+Z-Od+TKjzfbW_XJDVPcZ7aOW7VmMFE2Fwg@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>	<CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
	<CACQrdOn78zrvHJ=+Z-Od+TKjzfbW_XJDVPcZ7aOW7VmMFE2Fwg@mail.gmail.com>
Message-ID: <4E5DECFD.7020203@gmail.com>

Hi Jesse, hi all,

On 31/08/11 02:37, Jesse Noller wrote:
> Yes, I should have looped Noah in sooner.
>
> I have all the keys / passwords. Right now we need:
>
> 1. codespeed
> 2. benchmark runners

The second item is divided into two sub-items: the runner itself, and 
something that triggers a run nightly/on request.

In PyPy, we currently use a modified version of the Unladen Swallow benchmark 
runner (look at runner.py):
https://bitbucket.org/pypy/benchmarks/src

in its current state, it is sub-optimal, because it was designed to compare 
two pythons side-by-side, and not to save the results and then upload them to 
a website.  As a consequence, it always runs the benchmarks twice.  With PyPy, 
it's lesser of a problem because for each binary we always run the benchmarks 
with the default options and with "--jit off" anyway, but for CPython running 
them twice would result in a waste of time.

I suppose that the current runner is good enough for now and to get started, 
however at some point it will need some care.


Then, we need something which triggers the actual benchmark run.  In PyPy, we 
use buildbot, and each benchmark run first do a full translation, then 
executes the benchmarks.

I can setup a buildbot instance on speed.pypy.org, if you give me access to 
the machine.  I propose that as a very first step, we just make speed.pypy.org 
a buildslave which depends on pypy's own buildmaster.   This makes it very 
easy and fast to setup it, so we can have something running soon.

ciao,
Anto

From anto.cuni at gmail.com  Wed Aug 31 10:19:10 2011
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 31 Aug 2011 10:19:10 +0200
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <4E5DECFD.7020203@gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>	<CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
	<CACQrdOn78zrvHJ=+Z-Od+TKjzfbW_XJDVPcZ7aOW7VmMFE2Fwg@mail.gmail.com>
	<4E5DECFD.7020203@gmail.com>
Message-ID: <4E5DEE7E.3080801@gmail.com>

On 31/08/11 10:12, Antonio Cuni wrote:
> I can setup a buildbot instance on speed.pypy.org, if you give me access to
> the machine. I propose that as a very first step, we just make speed.pypy.org
> a buildslave which depends on pypy's own buildmaster. This makes it very easy
> and fast to setup it, so we can have something running soon.

sorry, of course I meant speed.python.org in the paragraph above!

So, to recapitulate: as a first and fast solution, I propose to setup 
speed.python.org as a buildslave for PyPy's buildmaster.

ciao,
Anto

From lac at openend.se  Wed Aug 31 11:06:21 2011
From: lac at openend.se (Laura Creighton)
Date: Wed, 31 Aug 2011 11:06:21 +0200
Subject: [pypy-dev] PyCon UK wants pypy
Message-ID: <201108310906.p7V96L03015271@theraft.openend.se>


This is the message that got bounced.  Since I now am admin
this shouldn't happen any more.

------- Forwarded Message

Return-Path: funthyme at gmail.com
Delivery-Date: Wed Aug 31 10:47:05 2011
Return-Path: <funthyme at gmail.com>
Message-ID: <CAB-vKOQJdybOeCWzo9MFxMMOuZcOXjRRBbz2kezR7xmbbrLUDw at mail.gmail.com>
Subject: Fwd: PyPy at PyCon UK
From: John Pinner <funthyme at gmail.com>
To: Laura Creighton <lac at openend.se>

The bounced post...


- ---------- Forwarded message ----------
From: John Pinner <funthyme at gmail.com>
Date: 30 August 2011 23:45
Subject: PyPy at PyCon UK
To: pypy-dev at python.org


Hello Guys,

You should have heard that we are holding PyCon UK 2011 on 24th-25th
September 2011 at Coventry, UK (about 25km from Birmingham where we
held EP2009/EP2010).

Details are on the wiki at http://pyconuk.net.

It would be good to have a PyPy presence, at least a talk, maybe a
Workshop, or better still a sprint.

The new venue has excellent facilities, including really good internet
access (typically 35 Mbits up and down).

If you'd like to take part, please put things up on the wiki.

If you want to hold a sprint, we could extend it either side of the
conference days, to include Friday 23rd and/or Monday 26th September,
but we need to know if you want to do this, and numbers of sprinters,
this week to make sure we have the facilities booked and finance
available.

Best wishes,

John
- --

------- End of Forwarded Message


From asmodai at in-nomine.org  Wed Aug 31 09:43:12 2011
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Wed, 31 Aug 2011 09:43:12 +0200
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <32A28CE2-DA63-4B47-9938-1DDCE702168E@coderanger.net>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>
	<CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
	<32A28CE2-DA63-4B47-9938-1DDCE702168E@coderanger.net>
Message-ID: <20110831074312.GJ99611@nexus.in-nomine.org>

-On [20110831 02:57], Noah Kantrowitz (noah at coderanger.net) wrote:
>Yahr, I be here. I would really like to see this done under a config management system (I prefer Chef and thats been the plan so far unless there are heavy objections). In general no one should ever be changing things on any PSF server by hand if at all possible in the interests of disaster recovery, reproducibility, and some modicum of enforced documentation (even if that doc is just a Chef recipe).

If Noah's going to pull this, being an Opscode guy, does it make sense for
me to help out. I mean, he's the Chef guru here and I doubt there's little I
can contribute then?

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | GPG: 2EAC625B
Under this standard shalt thou conquer...

From jnoller at gmail.com  Wed Aug 31 11:52:39 2011
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 31 Aug 2011 05:52:39 -0400
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <20110831074312.GJ99611@nexus.in-nomine.org>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOku0V4pAEfAvoVZvjeH6G=+zmvvCHWGZNFfHR+9bJGzSQ@mail.gmail.com>
	<CAFRnB2X9PbHqB7omvS7rtQKhv07e+MhhGtW5OijJ6VwzO_1_aQ@mail.gmail.com>
	<32A28CE2-DA63-4B47-9938-1DDCE702168E@coderanger.net>
	<20110831074312.GJ99611@nexus.in-nomine.org>
Message-ID: <CE8FE9FB-78BD-4EF1-9664-ED6C7B7E0637@gmail.com>

Noah is one person, yes you should help

On Aug 31, 2011, at 3:43 AM, Jeroen Ruigrok van der Werven <asmodai at in-nomine.org> wrote:

> -On [20110831 02:57], Noah Kantrowitz (noah at coderanger.net) wrote:
>> Yahr, I be here. I would really like to see this done under a config management system (I prefer Chef and thats been the plan so far unless there are heavy objections). In general no one should ever be changing things on any PSF server by hand if at all possible in the interests of disaster recovery, reproducibility, and some modicum of enforced documentation (even if that doc is just a Chef recipe).
> 
> If Noah's going to pull this, being an Opscode guy, does it make sense for
> me to help out. I mean, he's the Chef guru here and I doubt there's little I
> can contribute then?
> 
> -- 
> Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
> ????? ?????? ??? ?? ??????
> http://www.in-nomine.org/ | GPG: 2EAC625B
> Under this standard shalt thou conquer...

From funthyme at gmail.com  Wed Aug 31 11:54:13 2011
From: funthyme at gmail.com (John Pinner)
Date: Wed, 31 Aug 2011 10:54:13 +0100
Subject: [pypy-dev] PyCon UK and PyPy
Message-ID: <CAB-vKOTSFta1_ns3kQ=KfjahHi=Tb50YY0oGFSBv_wxdB86WjQ@mail.gmail.com>

Hello Guys,

You should have heard that we are holding PyCon UK 2011 on 24th-25th
September 2011 at Coventry, UK (about 25km from Birmingham where we
held EP2009/EP2010).

Details are on the wiki at http://pyconuk.net.

It would be good to have a PyPy presence, at least a talk, maybe a
Workshop, or better still a sprint.

The new venue has excellent facilities, including really good internet
access (typically 35 Mbits up and down).

If youd like to take part, please put things up on the wiki.

If you want to hold a sprint, we could extend it either side of the
conference days, to include Friday 23rd and/or Monday 26th September,
but we need to know if you want to do this, and numbers of sprinters,
this week to make sure we have the facilities booked and finance
available.

Best wishes,

John
--

From arigo at tunes.org  Wed Aug 31 13:35:12 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 31 Aug 2011 13:35:12 +0200
Subject: [pypy-dev] Hacking at tarfile.py and gzip.py
Message-ID: <CAMSv6X02nOJ8skvgByzaLJK4UjtmbGXftWL01GciZ=Fs8vv7pw@mail.gmail.com>

Hi Justin,

I see that you're hacking at CPython's tarfile.py and gzip.py, for
performance reason.  I should warn you that it's probably a bad idea.
For example, when we moved from supporting 2.5 to 2.7, we didn't
manually convert all the stuff in modified-2.5; we just mostly ignored
it and took the few still-relevant changes (the rest was fixed in
CPython itself in the meantime).  In other words, "modified-*" is only
supposed to be there to fix real issues that appear on PyPy and not on
CPython.

We have no place to put specifically performance improvements, and
that's good, because we don't want to diverge from CPython's stdlib.
If you do helpful performance fixes, they are probably helpful for
CPython, too, and so they should go to the CPython issue tracker.

Also, a check-in to pypy/module/bz2/interp_bz2.py has the message "fix
bz2. tests didn't find this."  But it doesn't add any test.  This is
also a bad idea.  If it really fixes an issue, it must come either
with a test that shows it, or a lengthy explanation of why you
couldn't design a test.


A bient?t,

Armin.

From arigo at tunes.org  Wed Aug 31 13:39:23 2011
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 31 Aug 2011 13:39:23 +0200
Subject: [pypy-dev] Hacking at tarfile.py and gzip.py
In-Reply-To: <CAMSv6X02nOJ8skvgByzaLJK4UjtmbGXftWL01GciZ=Fs8vv7pw@mail.gmail.com>
References: <CAMSv6X02nOJ8skvgByzaLJK4UjtmbGXftWL01GciZ=Fs8vv7pw@mail.gmail.com>
Message-ID: <CAMSv6X0D74h7JOy8SOjP9D11cGKtB3UN0CS9B77a8H8KEPJ_Ww@mail.gmail.com>

Re-hi,

On Wed, Aug 31, 2011 at 1:35 PM, Armin Rigo <arigo at tunes.org> wrote:
> If you do helpful performance fixes, they are probably helpful for
> CPython, too, and so they should go to the CPython issue tracker.

...or, I just saw this kind of check-in: "pypy doesn't like adding
empty strings."  Instead of "fixing" it in tarfile.py, this is really
an issue in PyPy: maybe the addition of an empty string and another
string is not special-cased to just return the other string, as it
would be in CPython.  If so, then tarfile.py is just a real-life
example of why it's a good idea, and we should fix PyPy's strings
instead.


A bient?t,

Armin.

From aaron.devore at gmail.com  Wed Aug 31 16:56:49 2011
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Wed, 31 Aug 2011 07:56:49 -0700
Subject: [pypy-dev] Policy on Python 3 and Python 2 executable names
In-Reply-To: <CAMSv6X12XWqawf3zvcgRqOCP5J=j+ZMf0RaBrD7ejQ2vB7kkGg@mail.gmail.com>
References: <CAL4sBwgXMVSRg7=UF3xXGUwaP__La2K_WCjB=iOrq6fqkqejTQ@mail.gmail.com>
	<CAMSv6X12XWqawf3zvcgRqOCP5J=j+ZMf0RaBrD7ejQ2vB7kkGg@mail.gmail.com>
Message-ID: <CAL4sBwivGUr4dEL2pJ0Bo8tOa6nrFHTzCbdfd+Vv9nWh7rTjmQ@mail.gmail.com>

On Wed, Aug 31, 2011 at 12:13 AM, Armin Rigo <arigo at tunes.org> wrote:
>
> I suppose that we need to *have* a pypy3 first, before any
> conversation like that really makes sense. ?Last March this wasn't
> even being considered. ?Now it is maybe in some very draftish early
> planning stage.

The idea is to be proactive in policy making instead of allowing the
problem to be acute like with CPython.

-Aaron DeVore

From jnoller at gmail.com  Wed Aug 31 17:00:44 2011
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 31 Aug 2011 11:00:44 -0400
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
Message-ID: <CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>

I've put up a splash page for the project this AM:

http://speed.python.org/

jesse

From tobami at googlemail.com  Wed Aug 31 20:34:49 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 31 Aug 2011 20:34:49 +0200
Subject: [pypy-dev] Moving the project forward
In-Reply-To: <CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
Message-ID: <CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>

Hi all,

though I took up on the task of installing a Codespeed instance
myself, I didn't have time until now. This weekend I will definitely
have  a *lot* of time to work on this, so count on that task being
done by then.

The bitbucket issue tracker is a start (though a organization account
would be better) and the splash page is great. So let's get started
organizing things.

Regarding the deployment strategy, it turns out I use Chef at work, so
I am in full agreement with Noah here (yey!). Actually, I am the
author of LittleChef (which we can use as a tool to execute Chef on
the node).

So, Configuration Management. I would propose that Noah starts the
repo with the Chef cookbooks (preferably a complete LittleChef
kitchen, but that is not a must :), and gets the main recipes (apache,
django) going, while I create a cookbook for Codespeed. What do you
think?

The benchmark runner question is still open. We need to clarify that.
Use the pypy runner? Tennessee's work?

Regarding repositories and issues, we could maybe have a "speed"
organization account (not sure on Bitbucket, you can do that in
Github), where we have a wiki, issues, and runner + config management
repo + other stuff.

Cheers,
Miquel

2011/8/31 Jesse Noller <jnoller at gmail.com>:
> I've put up a splash page for the project this AM:
>
> http://speed.python.org/
>
> jesse
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From tobami at googlemail.com  Wed Aug 31 22:12:58 2011
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 31 Aug 2011 22:12:58 +0200
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <6F0743E2-3C52-4463-B790-B995B02D4ED1@coderanger.net>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
	<CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
	<6F0743E2-3C52-4463-B790-B995B02D4ED1@coderanger.net>
Message-ID: <CAGf+9Vza+3ZcOp57ga1=jGnnZ=xktADc+UGtH9LNjJfr5WQ5LA@mail.gmail.com>

Oh, cool, so there will be an Opscode hosted account for the PSF,
right? Then the Chef repo should be for the PSF. Maybe in a current
account somewhere? What do you propose?

Miquel


2011/8/31 Noah Kantrowitz <noah at coderanger.net>:
> Opscode has already agreed to donate a Hosted account as long we keep it under ~20 clients :-) I can hand out the info for it to anyone that wants. As for setting up the Chef repo, just remember we are trying to not manage this system in isolation and that it will be part of a bigger PSF infrastructure management effort.
>
> --Noah
>
> On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:
>
>> Hi all,
>>
>> though I took up on the task of installing a Codespeed instance
>> myself, I didn't have time until now. This weekend I will definitely
>> have ?a *lot* of time to work on this, so count on that task being
>> done by then.
>>
>> The bitbucket issue tracker is a start (though a organization account
>> would be better) and the splash page is great. So let's get started
>> organizing things.
>>
>> Regarding the deployment strategy, it turns out I use Chef at work, so
>> I am in full agreement with Noah here (yey!). Actually, I am the
>> author of LittleChef (which we can use as a tool to execute Chef on
>> the node).
>>
>> So, Configuration Management. I would propose that Noah starts the
>> repo with the Chef cookbooks (preferably a complete LittleChef
>> kitchen, but that is not a must :), and gets the main recipes (apache,
>> django) going, while I create a cookbook for Codespeed. What do you
>> think?
>>
>> The benchmark runner question is still open. We need to clarify that.
>> Use the pypy runner? Tennessee's work?
>>
>> Regarding repositories and issues, we could maybe have a "speed"
>> organization account (not sure on Bitbucket, you can do that in
>> Github), where we have a wiki, issues, and runner + config management
>> repo + other stuff.
>>
>> Cheers,
>> Miquel
>>
>> 2011/8/31 Jesse Noller <jnoller at gmail.com>:
>>> I've put up a splash page for the project this AM:
>>>
>>> http://speed.python.org/
>>>
>>> jesse
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>
>> _______________________________________________
>> Speed mailing list
>> Speed at python.org
>> http://mail.python.org/mailman/listinfo/speed
>
>

From noah at coderanger.net  Wed Aug 31 22:06:45 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Wed, 31 Aug 2011 13:06:45 -0700
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
	<CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
Message-ID: <6F0743E2-3C52-4463-B790-B995B02D4ED1@coderanger.net>

Opscode has already agreed to donate a Hosted account as long we keep it under ~20 clients :-) I can hand out the info for it to anyone that wants. As for setting up the Chef repo, just remember we are trying to not manage this system in isolation and that it will be part of a bigger PSF infrastructure management effort.

--Noah

On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:

> Hi all,
> 
> though I took up on the task of installing a Codespeed instance
> myself, I didn't have time until now. This weekend I will definitely
> have  a *lot* of time to work on this, so count on that task being
> done by then.
> 
> The bitbucket issue tracker is a start (though a organization account
> would be better) and the splash page is great. So let's get started
> organizing things.
> 
> Regarding the deployment strategy, it turns out I use Chef at work, so
> I am in full agreement with Noah here (yey!). Actually, I am the
> author of LittleChef (which we can use as a tool to execute Chef on
> the node).
> 
> So, Configuration Management. I would propose that Noah starts the
> repo with the Chef cookbooks (preferably a complete LittleChef
> kitchen, but that is not a must :), and gets the main recipes (apache,
> django) going, while I create a cookbook for Codespeed. What do you
> think?
> 
> The benchmark runner question is still open. We need to clarify that.
> Use the pypy runner? Tennessee's work?
> 
> Regarding repositories and issues, we could maybe have a "speed"
> organization account (not sure on Bitbucket, you can do that in
> Github), where we have a wiki, issues, and runner + config management
> repo + other stuff.
> 
> Cheers,
> Miquel
> 
> 2011/8/31 Jesse Noller <jnoller at gmail.com>:
>> I've put up a splash page for the project this AM:
>> 
>> http://speed.python.org/
>> 
>> jesse
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>> 
> _______________________________________________
> Speed mailing list
> Speed at python.org
> http://mail.python.org/mailman/listinfo/speed

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4789 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110831/57caf879/attachment-0001.bin>

From brett at python.org  Wed Aug 31 22:11:51 2011
From: brett at python.org (Brett Cannon)
Date: Wed, 31 Aug 2011 13:11:51 -0700
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
	<CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
Message-ID: <CAP1=2W6MOwwRukW2c3OOY8N6mF_qsLvcOoT7Lbc55-bFHE0dNw@mail.gmail.com>

On Wed, Aug 31, 2011 at 11:34, Miquel Torres <tobami at googlemail.com> wrote:
> Hi all,
>
> though I took up on the task of installing a Codespeed instance
> myself, I didn't have time until now. This weekend I will definitely
> have ?a *lot* of time to work on this, so count on that task being
> done by then.
>
> The bitbucket issue tracker is a start (though a organization account
> would be better) and the splash page is great. So let's get started
> organizing things.
[SNIP]
> Regarding repositories and issues, we could maybe have a "speed"
> organization account (not sure on Bitbucket, you can do that in
> Github), where we have a wiki, issues, and runner + config management
> repo + other stuff.

The PyPy folk could answer this as they have their repo on bitbucket
already. Else I guess we can just create a standalone account that
represents the official speed.python.org account.

From noah at coderanger.net  Wed Aug 31 22:28:22 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Wed, 31 Aug 2011 13:28:22 -0700
Subject: [pypy-dev] [Speed]  Moving the project forward
In-Reply-To: <CAGf+9Vza+3ZcOp57ga1=jGnnZ=xktADc+UGtH9LNjJfr5WQ5LA@mail.gmail.com>
References: <CACQrdOmx-Ujp9RVb7kQ5E-xPLm-=2cutvFbbPksg1iMJ31pz2w@mail.gmail.com>
	<CACQrdOmQLbmwfHtZC91Ps2NAovZJpBjq3gCs8GqHJAoaBYtr6Q@mail.gmail.com>
	<CAGf+9Vx=_-9YWLGf3Q5i6dx=Hnf=cML=WC_T++22fGehRJ-C6g@mail.gmail.com>
	<6F0743E2-3C52-4463-B790-B995B02D4ED1@coderanger.net>
	<CAGf+9Vza+3ZcOp57ga1=jGnnZ=xktADc+UGtH9LNjJfr5WQ5LA@mail.gmail.com>
Message-ID: <7D8FD0F4-2107-4556-A8A8-139CFCA831AD@coderanger.net>

Its all branches all the way down, so we can start work anywhere and push it to an "official" PSF bin later I think. I'm sure we will want to host a mirror of it on the python.org hg server too, just for discoverability.

--Noah

On Aug 31, 2011, at 1:12 PM, Miquel Torres wrote:

> Oh, cool, so there will be an Opscode hosted account for the PSF,
> right? Then the Chef repo should be for the PSF. Maybe in a current
> account somewhere? What do you propose?
> 
> Miquel
> 
> 
> 2011/8/31 Noah Kantrowitz <noah at coderanger.net>:
>> Opscode has already agreed to donate a Hosted account as long we keep it under ~20 clients :-) I can hand out the info for it to anyone that wants. As for setting up the Chef repo, just remember we are trying to not manage this system in isolation and that it will be part of a bigger PSF infrastructure management effort.
>> 
>> --Noah
>> 
>> On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:
>> 
>>> Hi all,
>>> 
>>> though I took up on the task of installing a Codespeed instance
>>> myself, I didn't have time until now. This weekend I will definitely
>>> have  a *lot* of time to work on this, so count on that task being
>>> done by then.
>>> 
>>> The bitbucket issue tracker is a start (though a organization account
>>> would be better) and the splash page is great. So let's get started
>>> organizing things.
>>> 
>>> Regarding the deployment strategy, it turns out I use Chef at work, so
>>> I am in full agreement with Noah here (yey!). Actually, I am the
>>> author of LittleChef (which we can use as a tool to execute Chef on
>>> the node).
>>> 
>>> So, Configuration Management. I would propose that Noah starts the
>>> repo with the Chef cookbooks (preferably a complete LittleChef
>>> kitchen, but that is not a must :), and gets the main recipes (apache,
>>> django) going, while I create a cookbook for Codespeed. What do you
>>> think?
>>> 
>>> The benchmark runner question is still open. We need to clarify that.
>>> Use the pypy runner? Tennessee's work?
>>> 
>>> Regarding repositories and issues, we could maybe have a "speed"
>>> organization account (not sure on Bitbucket, you can do that in
>>> Github), where we have a wiki, issues, and runner + config management
>>> repo + other stuff.
>>> 
>>> Cheers,
>>> Miquel
>>> 
>>> 2011/8/31 Jesse Noller <jnoller at gmail.com>:
>>>> I've put up a splash page for the project this AM:
>>>> 
>>>> http://speed.python.org/
>>>> 
>>>> jesse
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>> 
>>> _______________________________________________
>>> Speed mailing list
>>> Speed at python.org
>>> http://mail.python.org/mailman/listinfo/speed
>> 
>> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4789 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110831/b7d9dafb/attachment.bin>