From p.giarrusso at gmail.com  Wed Dec  1 00:34:54 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Wed, 1 Dec 2010 00:34:54 +0100
Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released)
In-Reply-To: <AANLkTi=tNGbymFodLQgyB04zBf3FZj1k0dW3GhNCu9HM@mail.gmail.com>
References: <AANLkTikdiugoQSDDSZOSoYL4jERTL1i_MPCuBT6gkTQx@mail.gmail.com>
	<AANLkTik-n2amKz=z9fgFtqxXNPDJMNh0fKOg7vnwGcTm@mail.gmail.com>
	<AANLkTimFFHWSwYYVNGouYc=_SdH235Q60YX-vvcmH6o6@mail.gmail.com>
	<id0ah2$2pr$1@dough.gmane.org>
	<AANLkTimkE12+bGDR7fjzjgeAVA43h07-ZLwFSWmSLuvC@mail.gmail.com>
	<AANLkTi=+bWFGZiaccZX8SGVCTzNGfVSbCVf5fqhGYN08@mail.gmail.com>
	<AANLkTinZ8+0ona4nmHUBCSw-R1n4tawS+VrnSB+RFD_c@mail.gmail.com>
	<AANLkTi=AfnW9HagT2PaCPmCmrao0O=dNhq+XHvJEDat+@mail.gmail.com>
	<AANLkTi=QPWdXrCJGMRNA-1NMnNT9m22cPBDmoBEH8eWX@mail.gmail.com>
	<AANLkTi=tNGbymFodLQgyB04zBf3FZj1k0dW3GhNCu9HM@mail.gmail.com>
Message-ID: <AANLkTikHuE+1-auGs_dqq7JuCOGXwzarz6f_ajuZiYw9@mail.gmail.com>

On Tue, Nov 30, 2010 at 08:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Tue, Nov 30, 2010 at 1:45 AM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
>> 2010/11/30 Paolo Giarrusso <p.giarrusso at gmail.com>
>>>
>>> As a matter of fact, however, pyexpat is not involved here for PyPy,
>>> and here (v1.4) it is still implemented through ctypes (in
>>> lib_pypy/pyexpat.py), and not in RPython in pypy/rlib/.
>
> It's also module/pyexpat and not rlib (rlib is for RPython libraries)
>
>>
>> Did you compile pypy yourself?
>> if the expat development files are present, the translation should build the
>> pyexpat module:
>> Python 2.5.2 (79656, Nov 29 2010, 21:05:28)
>> [PyPy 1.4.0] on linux2
>>>>>> import pyexpat
>>>>>> pyexpat
>> <module 'pyexpat' (built-in)>

My apologies, I self-compiled PyPy and I get the output you describe indeed.
Therefore I guess that the ctypes implementation I come across in
lib_pypy/pyexpat.py is probably a fallback - in case only the library,
but not the headers, are present.

Anyway, this does not interact with benchmarks above - Stefan, I still
don't get why you complained that pyexpat is slow by showing
benchmarks for another module, I guess I do not understand your email,
but it asks "reasonable?" after Amaury talks about pyexpat.

I'll try to benchmark it soon; a reasonable way to call pyexpat would
make it simpler since I have limited time and mental energy to devote,
and figuring out a non-stupid way to use it might be non-trivial
without learning to use the library.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From stefan_ml at behnel.de  Wed Dec  1 08:48:41 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 01 Dec 2010 08:48:41 +0100
Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released)
In-Reply-To: <AANLkTikHuE+1-auGs_dqq7JuCOGXwzarz6f_ajuZiYw9@mail.gmail.com>
References: <AANLkTikdiugoQSDDSZOSoYL4jERTL1i_MPCuBT6gkTQx@mail.gmail.com>	<AANLkTik-n2amKz=z9fgFtqxXNPDJMNh0fKOg7vnwGcTm@mail.gmail.com>	<AANLkTimFFHWSwYYVNGouYc=_SdH235Q60YX-vvcmH6o6@mail.gmail.com>	<id0ah2$2pr$1@dough.gmane.org>	<AANLkTimkE12+bGDR7fjzjgeAVA43h07-ZLwFSWmSLuvC@mail.gmail.com>	<AANLkTi=+bWFGZiaccZX8SGVCTzNGfVSbCVf5fqhGYN08@mail.gmail.com>	<AANLkTinZ8+0ona4nmHUBCSw-R1n4tawS+VrnSB+RFD_c@mail.gmail.com>	<AANLkTi=AfnW9HagT2PaCPmCmrao0O=dNhq+XHvJEDat+@mail.gmail.com>	<AANLkTi=QPWdXrCJGMRNA-1NMnNT9m22cPBDmoBEH8eWX@mail.gmail.com>	<AANLkTi=tNGbymFodLQgyB04zBf3FZj1k0dW3GhNCu9HM@mail.gmail.com>
	<AANLkTikHuE+1-auGs_dqq7JuCOGXwzarz6f_ajuZiYw9@mail.gmail.com>
Message-ID: <id4ukq$cgf$1@dough.gmane.org>

Paolo Giarrusso, 01.12.2010 00:34:
> Anyway, this does not interact with benchmarks above - Stefan, I still
> don't get why you complained that pyexpat is slow by showing
> benchmarks for another module, I guess I do not understand your email,
> but it asks "reasonable?" after Amaury talks about pyexpat.

Well, in CPython, I can see little to no reason why anyone would go as 
low-level as pyexpat when you can use cElementTree. So IMHO the level to 
compare is what people would normally use rather than what people could 
potentially use if they were beaten hard enough.

Stefan



From fijall at gmail.com  Wed Dec  1 09:00:12 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 1 Dec 2010 10:00:12 +0200
Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released)
In-Reply-To: <id4ukq$cgf$1@dough.gmane.org>
References: <AANLkTikdiugoQSDDSZOSoYL4jERTL1i_MPCuBT6gkTQx@mail.gmail.com>
	<AANLkTik-n2amKz=z9fgFtqxXNPDJMNh0fKOg7vnwGcTm@mail.gmail.com>
	<AANLkTimFFHWSwYYVNGouYc=_SdH235Q60YX-vvcmH6o6@mail.gmail.com>
	<id0ah2$2pr$1@dough.gmane.org>
	<AANLkTimkE12+bGDR7fjzjgeAVA43h07-ZLwFSWmSLuvC@mail.gmail.com>
	<AANLkTi=+bWFGZiaccZX8SGVCTzNGfVSbCVf5fqhGYN08@mail.gmail.com>
	<AANLkTinZ8+0ona4nmHUBCSw-R1n4tawS+VrnSB+RFD_c@mail.gmail.com>
	<AANLkTi=AfnW9HagT2PaCPmCmrao0O=dNhq+XHvJEDat+@mail.gmail.com>
	<AANLkTi=QPWdXrCJGMRNA-1NMnNT9m22cPBDmoBEH8eWX@mail.gmail.com>
	<AANLkTi=tNGbymFodLQgyB04zBf3FZj1k0dW3GhNCu9HM@mail.gmail.com>
	<AANLkTikHuE+1-auGs_dqq7JuCOGXwzarz6f_ajuZiYw9@mail.gmail.com>
	<id4ukq$cgf$1@dough.gmane.org>
Message-ID: <AANLkTi=oO5_XWifcecF6Ea8n3WUk5cjQP9cT8_x6RV=B@mail.gmail.com>

On Wed, Dec 1, 2010 at 9:48 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Paolo Giarrusso, 01.12.2010 00:34:
>> Anyway, this does not interact with benchmarks above - Stefan, I still
>> don't get why you complained that pyexpat is slow by showing
>> benchmarks for another module, I guess I do not understand your email,
>> but it asks "reasonable?" after Amaury talks about pyexpat.
>
> Well, in CPython, I can see little to no reason why anyone would go as
> low-level as pyexpat when you can use cElementTree. So IMHO the level to
> compare is what people would normally use rather than what people could
> potentially use if they were beaten hard enough.
>
> Stefan
>

Hey.

Sure, makes sense :-) I think one of the reasons for some slowdown is
that calls from C are not jitted if they don't contain loops
themselves. This doesn't explain the whole thing obviously, because
there is something really wrong going on looking at numbers.


From alex.nanou at gmail.com  Wed Dec  1 11:23:08 2010
From: alex.nanou at gmail.com (Alex A. Naanou)
Date: Wed, 1 Dec 2010 13:23:08 +0300
Subject: [pypy-dev] a possible leak in the object namespace...
In-Reply-To: <4CF53579.6080504@gmx.de>
References: <AANLkTi=+oQzRR=gnvGhHij8+RJkybD2MwgV6NPJd54rJ@mail.gmail.com>
	<4CF3F4F3.9020204@gmx.de>
	<AANLkTin57JyAv1ihYa_FdCnwZqvDfDXbbuMDwvvp-a8f@mail.gmail.com>
	<4CF53579.6080504@gmx.de>
Message-ID: <AANLkTikOsyAKQMXOWNx=jOjhGco4-ASSRi2m5XtiHRBb@mail.gmail.com>

On Tue, Nov 30, 2010 at 20:33, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Hi Alex,
>
> On 11/29/2010 09:02 PM, Alex A. Naanou wrote:
>>
>> On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz<cfbolz at gmx.de> ?wrote:
>
> [snip]
>>>
>>> ?a) lots of attributes, which we expect to be rare
>>> ?b) access them with setattr, which is a lot slower than a fixed
>>> attribute
>>> ?c) access a different attribute every loop iteration, which means the
>>> compiler has to produce one bit of code for every attribute
>>
>> This is intentional (all three points), I did not want the jit to
>> factor out the loop -- I wanted to time the initial attribute
>> creation...
>
> Yes, I fear initial attribute creation is never going to be very efficient.
>
> [snip]
>>
>> But quadratic behaviour + a three orders of magnitude increase in time
>> it takes to create an attr is scarry... but you are right, how often
>> does that usecase happen? :)
>
> I improved some things so that setattr/getattr is a bit faster. Should now
> only be one or two orders of magnitude :-). If you run it tomorrow with the
> nightly build from tonight, you should see an improvement.

That's good news!
It is not uncommon, at least in my work, to write generic code that
does not know of attribute names it uses, thus getattr/setattr
functions are unavoidable.


> The quadraticness is not easily fixable without giving up all optimizations
> with instances of more than X (maybe 1000?) attributes. Again, I don't think
> this is common. And I don't want to chose an X.

I'd even argue that most of the cases were the number of attributes
exceeds a certain "sane" N would and should be better implemented via
a container...
(in spite of the how I wrote the code in my first mail ;) )

I've been planning of moving most of my commercial projects to PyPy as
soon as it is stable enough -- I'm tired of fighting the CPython
implementation for most of the time -- we will see how it goes... too
bad some of the code will be PyPy specific due to the way CPython
works.


Thanks!


>
> Cheers,
>
> Carl Friedrich
>



-- 
Alex.


From rndblnch at gmail.com  Wed Dec  1 14:18:01 2010
From: rndblnch at gmail.com (renaud blanch)
Date: Wed, 1 Dec 2010 11:18:01 -0200
Subject: [pypy-dev] (no subject)
Message-ID: <AANLkTi=Q-k=1tgr8rsSNkJEZOaUkzPqKDfFCOxkdHVrh@mail.gmail.com>

hi,

i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
this is partially successful, but i need some advice to progress further.
pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
works out of the box for simple functions (those that do not takes
c-pointer to buffers of data as arguments).
for the rest, the first issue is that pyopengl use two functions from
the ctypes.pythonapi lib, namely PyString_AsString and
PyBuffer_FromMemory.
any advice on how to replace those functions to make them compatible with pypy?

Mike Fletcher (pyopengl author) gave me some hints about that point:
> For the first issue, those are going to require some reworking, in
> essence those are "C" implemented code that happens to use Python/ctypes
> as the implementation language and makes assumptions about the
> data-storage for the objects (e.g. that a string is internally a
> contiguous series of bytes, which is *not necessarily* true in PyPy).
> We'd need to find a mechanism in PyPy that would give us that direct
> memory-pointer access to be able to use it.  Note: a compacting garbage
> collector (or anything else that can move memory locations) will cause
> problems there, so we may need to find a way to signal PyPy not to move
> a given object, and to use contiguous data-arrays for their storage.

thanks a lot for any advice,

renaud

0. PyOpenGL 3.x / The Python OpenGL? Binding
<http://pyopengl.sourceforge.net/>

1. opengl-programmable / a short step by step tutorial to OpenGL
programmable pipeline
<http://bitbucket.org/rndblnch/opengl-programmable/>


From rndblnch at gmail.com  Wed Dec  1 14:19:31 2010
From: rndblnch at gmail.com (renaud blanch)
Date: Wed, 1 Dec 2010 11:19:31 -0200
Subject: [pypy-dev] pyopengl on pypy
Message-ID: <AANLkTinh71HKkNtiF_z5hFZi-9W0F2gDq+f3PhF3gpo5@mail.gmail.com>

sorry, i forgot to put a subject.
renaud


From rndblnch at gmail.com  Wed Dec  1 14:21:21 2010
From: rndblnch at gmail.com (renaud blanch)
Date: Wed, 1 Dec 2010 11:21:21 -0200
Subject: [pypy-dev] pyopengl on pypy
Message-ID: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>

this time with a subject, the message, and my apologies for the noise,
renaud

On 12/1/10, renaud blanch <rndblnch at gmail.com> wrote:
> hi,
>
> i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> this is partially successful, but i need some advice to progress further.
> pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> works out of the box for simple functions (those that do not takes
> c-pointer to buffers of data as arguments).
> for the rest, the first issue is that pyopengl use two functions from
> the ctypes.pythonapi lib, namely PyString_AsString and
> PyBuffer_FromMemory.
> any advice on how to replace those functions to make them compatible with
> pypy?
>
> Mike Fletcher (pyopengl author) gave me some hints about that point:
>> For the first issue, those are going to require some reworking, in
>> essence those are "C" implemented code that happens to use Python/ctypes
>> as the implementation language and makes assumptions about the
>> data-storage for the objects (e.g. that a string is internally a
>> contiguous series of bytes, which is *not necessarily* true in PyPy).
>> We'd need to find a mechanism in PyPy that would give us that direct
>> memory-pointer access to be able to use it.  Note: a compacting garbage
>> collector (or anything else that can move memory locations) will cause
>> problems there, so we may need to find a way to signal PyPy not to move
>> a given object, and to use contiguous data-arrays for their storage.
>
> thanks a lot for any advice,
>
> renaud
>
> 0. PyOpenGL 3.x / The Python OpenGL? Binding
> <http://pyopengl.sourceforge.net/>
>
> 1. opengl-programmable / a short step by step tutorial to OpenGL
> programmable pipeline
> <http://bitbucket.org/rndblnch/opengl-programmable/>
>


From fijall at gmail.com  Wed Dec  1 14:52:39 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 1 Dec 2010 15:52:39 +0200
Subject: [pypy-dev] (no subject)
In-Reply-To: <AANLkTi=Q-k=1tgr8rsSNkJEZOaUkzPqKDfFCOxkdHVrh@mail.gmail.com>
References: <AANLkTi=Q-k=1tgr8rsSNkJEZOaUkzPqKDfFCOxkdHVrh@mail.gmail.com>
Message-ID: <AANLkTinijPONOEFQyTkQqPRo9SwO_63kou6APRW=4p=7@mail.gmail.com>

Hey.

I would suggest using array module. It allocates memory in a
non-moving location and it's address can be found using buffer_info
method.

Cheers,
fijal

On Wed, Dec 1, 2010 at 3:18 PM, renaud blanch <rndblnch at gmail.com> wrote:
> hi,
>
> i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> this is partially successful, but i need some advice to progress further.
> pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> works out of the box for simple functions (those that do not takes
> c-pointer to buffers of data as arguments).
> for the rest, the first issue is that pyopengl use two functions from
> the ctypes.pythonapi lib, namely PyString_AsString and
> PyBuffer_FromMemory.
> any advice on how to replace those functions to make them compatible with pypy?
>
> Mike Fletcher (pyopengl author) gave me some hints about that point:
>> For the first issue, those are going to require some reworking, in
>> essence those are "C" implemented code that happens to use Python/ctypes
>> as the implementation language and makes assumptions about the
>> data-storage for the objects (e.g. that a string is internally a
>> contiguous series of bytes, which is *not necessarily* true in PyPy).
>> We'd need to find a mechanism in PyPy that would give us that direct
>> memory-pointer access to be able to use it. ?Note: a compacting garbage
>> collector (or anything else that can move memory locations) will cause
>> problems there, so we may need to find a way to signal PyPy not to move
>> a given object, and to use contiguous data-arrays for their storage.
>
> thanks a lot for any advice,
>
> renaud
>
> 0. PyOpenGL 3.x / The Python OpenGL? Binding
> <http://pyopengl.sourceforge.net/>
>
> 1. opengl-programmable / a short step by step tutorial to OpenGL
> programmable pipeline
> <http://bitbucket.org/rndblnch/opengl-programmable/>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Wed Dec  1 15:23:01 2010
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 1 Dec 2010 15:23:01 +0100
Subject: [pypy-dev] pyopengl on pypy
In-Reply-To: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>
References: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>
Message-ID: <AANLkTikRdsWg1VNUr+YRX_GQjrYCh=737Fn81p9TA_vN@mail.gmail.com>

Hi Renaud,

On Wed, Dec 1, 2010 at 2:21 PM, renaud blanch <rndblnch at gmail.com> wrote:
>>> We'd need to find a mechanism in PyPy that would give us that direct
>>> memory-pointer access to be able to use it. ?Note: a compacting garbage
>>> collector (or anything else that can move memory locations) will cause
>>> problems there, so we may need to find a way to signal PyPy not to move
>>> a given object, and to use contiguous data-arrays for their storage.

Indeed, this part from Mike Fletcher got the cause right, but the
solution we recommend is different.  It's kind of impossible in PyPy
to mark objects as non-moving ("pin" them, as the garbage collection
language goes).  Instead you need to just use the official ctypes API,
create_string_buffer().  It's equivalent, but requires one extra copy
of the data.


A bient?t,

Armin.


From amauryfa at gmail.com  Wed Dec  1 15:33:47 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 1 Dec 2010 15:33:47 +0100
Subject: [pypy-dev] pyopengl on pypy
In-Reply-To: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>
References: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>
Message-ID: <AANLkTi=mUFPt_j1gyVeu6NGQKaP8A1yH3Ce78eTZ=LC_@mail.gmail.com>

Hi,

2010/12/1 renaud blanch <rndblnch at gmail.com>

> > i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> > this is partially successful, but i need some advice to progress further.
> > pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> > works out of the box for simple functions (those that do not takes
> > c-pointer to buffers of data as arguments).
> > for the rest, the first issue is that pyopengl use two functions from
> > the ctypes.pythonapi lib, namely PyString_AsString and
> > PyBuffer_FromMemory.
> > any advice on how to replace those functions to make them compatible with
> > pypy?
> >
> > Mike Fletcher (pyopengl author) gave me some hints about that point:
> >> For the first issue, those are going to require some reworking, in
> >> essence those are "C" implemented code that happens to use Python/ctypes
> >> as the implementation language and makes assumptions about the
> >> data-storage for the objects (e.g. that a string is internally a
> >> contiguous series of bytes, which is *not necessarily* true in PyPy).
> >> We'd need to find a mechanism in PyPy that would give us that direct
> >> memory-pointer access to be able to use it.  Note: a compacting garbage
> >> collector (or anything else that can move memory locations) will cause
> >> problems there, so we may need to find a way to signal PyPy not to move
> >> a given object, and to use contiguous data-arrays for their storage.
>

Don't worry, with PyPy's ctypes you'll never get the address of a moving
object.
c_char_p makes a non-moving copy of the string.

But IMO the call to pythonapi.PyString_AsString could be removed in
OpenGL/arrays/strings.py:

def dataPointer(value):
    return ctypes.cast(ctypes.c_char_p(value), ctypes.c_void_p).value

I don't know about the other function, though.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101201/def6fba1/attachment.html>

From rndblnch at gmail.com  Wed Dec  1 18:56:54 2010
From: rndblnch at gmail.com (rndblnch)
Date: Wed, 1 Dec 2010 17:56:54 +0000 (UTC)
Subject: [pypy-dev] pyopengl on pypy
References: <AANLkTin2B9UrPxHTtTCiK4rH1PBdR2y893Sw5f813pR5@mail.gmail.com>
	<AANLkTi=mUFPt_j1gyVeu6NGQKaP8A1yH3Ce78eTZ=LC_@mail.gmail.com>
Message-ID: <loom.20101201T183927-169@post.gmane.org>

hi,

Amaury Forgeot d'Arc <amauryfa <at> gmail.com> writes:
> But IMO the call to?pythonapi.PyString_AsString could be removed in
> OpenGL/arrays/strings.py:
> 
> def?dataPointer(value):
> ?? ?return ctypes.cast(ctypes.c_char_p(value),?ctypes.c_void_p).value

great, this replacement (and also the one suggested by Armin with
create_string_buffer instead of c_char_p) seams to works (i.e., runs fine 
on cpython).

> I don't know about the other function, though.
this one is only in experimental code of pyopengl, so i won't bother much 
for now.


the next issue is now that the array data types used by pyopengl are not
compatible with pypy ctypes: they are lacking (at least) _CData_input as shown
in the stack trace below.
any advice on that one?
should i try to fix PyOpenGL ArrayDatatype, or try to patch pypy's ctypes
function.py to work around that one?

thanks again,
renaud

% pypy 01-direct.py
Traceback (most recent call last):
  File "app_main.py", line 33, in run_toplevel
  File "01-direct.py", line 268, in <module>
    sys.exit(main())
  File "01-direct.py", line 262, in main
    init_texture()
  File "01-direct.py", line 48, in init_texture
    "".join(pixel(i, j, k) for i in range(width)
  File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/latebind.py", line
45, in __call__
    return self._finalCall( *args, **named )
  File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/wrapper.py", line
784, in wrapperCall
    result = self.wrappedOperation( *cArguments )
  File
"/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/platform/baseplatform.py",
line 335, in __call__
    return self( *args, **named )
  File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 166,
in __call__
    argtypes, argsandobjs = self._wrap_args(argtypes, args)
  File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 281,
in _wrap_args
    wrapped = argtype._CData_input(arg)
AttributeError: type object 'ArrayDatatype' has no attribute '_CData_input'




From anto.cuni at gmail.com  Wed Dec  1 23:48:23 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 01 Dec 2010 23:48:23 +0100
Subject: [pypy-dev] [PyPy Status Blog] New comment on PyPy 1.4:
	Ouroboros in practice.
In-Reply-To: <1291242102925.4c68f818-d73b-492b-a923-4edf7d1a74f7@google.com>
References: <1291242102925.4c68f818-d73b-492b-a923-4edf7d1a74f7@google.com>
Message-ID: <4CF6D0B7.9080900@gmail.com>

On 01/12/10 23:21, Amaury Forgeot d'Arc wrote:
[cut]
> There is already an ongoing effort to port PyPy to Python 2.7.
> 
> But we need some help! It's a good way to become a PyPy developer.
> And no, you don't have to be a JIT expert to implement itertools.combinations
> or asian codecs.

Nice comment Amaury :-).

What about writing a blog post explicitly asking for help? I also agree that
the fast-forward branch is a good way to enter pypy, moreover we should
exploit the hype we seem to have at the moment :-).

ciao,
Anto


From rndblnch at gmail.com  Thu Dec  2 17:21:47 2010
From: rndblnch at gmail.com (rndblnch)
Date: Thu, 2 Dec 2010 16:21:47 +0000 (UTC)
Subject: [pypy-dev] pyopengl on pypy, take 2
Message-ID: <loom.20101202T160111-272@post.gmane.org>

hi,


with your help, i have been able to patch pyopengl to make non-trivial 
glut/opengl interactive programs (involving shaders, vertex and pixel buffers)
run on pypy.
the programs:
http://bitbucket.org/rndblnch/opengl-programmable/

some details about pyopengl modifications here:
http://thread.gmane.org/gmane.comp.python.opengl.devel/1004/focus=1007


the only thing i add to change that is not already in the pypy svn is a minor 
check in _ctypes._Pointer.setcontents to make it raise a TypeError rather than 
an AttributeError if the value passed do not have a _ensure_objects method:

 class _Pointer(_CData):
     def setcontents(self, value):
         if not isinstance(value, self._type_):
             raise TypeError("expected %s instead of %s" % (
                 self._type_.__name__, type(value).__name__))
         self._objects = {keepalive_key(1):value}
+        if not hasattr(value, "_ensure_objects"):
+            raise TypeError
         if value._ensure_objects() is not None:
             self._objects[keepalive_key(0)] = value._objects
         value = value._buffer
         self._buffer[0] = value

a better check might be "if not isinstance(value, _CData)". i hope you will
consider this minor change for inclusion into pypy's ctypes implementation.


i still have other issues, but at least all my programs run and are interactive.
i have not looked at the performance, but i have a question on this aspect: 
since most of the python code is executed as c callbacks passed to glut, should
i except any gain at all? is pypy able to jit those functions wrapped by ctypes
and passed down as callbacks to a c library?

thanks again,

renaud






From renesd at gmail.com  Fri Dec  3 00:28:22 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Thu, 2 Dec 2010 23:28:22 +0000
Subject: [pypy-dev] =?koi8-r?b?9sXO08vB0SDMz8fJy8E=?=
In-Reply-To: <4cf7e88f.5989cc0a.4c3c.34f2@mx.google.com>
References: <4cf7e88f.5989cc0a.4c3c.34f2@mx.google.com>
Message-ID: <AANLkTimQm+U2Ci41FCrS0bFtrC=u7OD6ec_aruqRiFNP@mail.gmail.com>

>>>> a="""Want to understand women? Know the real their thoughts when they
try to hide them in a completely abstract words? On the practical training
of the Academy of Dating in the real world you learn to be attentive and
understanding man for women.

Do not believe me? Try it, you'll enjoy. I wish you a great scientific
success."""
>>>> spam(a)
True

>>>> block(a.sender)
...



2010/11/25 Vovan <sasmekoll at gmail.com>

>  ?????? ???????? ??????? ????? ???????? ?? ?????, ????? ??? ????????
> ???????? ?? ?? ?????????? ???????????? ??????? ?? ???????????? ?????????
> ???????? ????????? ?? ? ???????? ???????? ????????? ???? ????????????????? ?
> ?????????? ???????? ??? ??????.
>
> http://samec.org.ua/?p=202
>
> ?? ??????? ??????????, ??? ??????????. ????? ??? ??????? ??????? ???????.
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101202/c26d96a3/attachment.html>

From sungsuha at gmail.com  Fri Dec  3 11:19:53 2010
From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha)
Date: Fri, 3 Dec 2010 10:19:53 +0000 (UTC)
Subject: [pypy-dev] Windows source link
Message-ID: <loom.20101203T111928-339@post.gmane.org>

http://pypy.org/download/pypy-1.4-src.zip

Down for everyone or is it just me?



From arigo at tunes.org  Fri Dec  3 11:32:40 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 3 Dec 2010 11:32:40 +0100
Subject: [pypy-dev] Windows source link
In-Reply-To: <loom.20101203T111928-339@post.gmane.org>
References: <loom.20101203T111928-339@post.gmane.org>
Message-ID: <AANLkTimCSB_-3QBMcUokiVueSdAGZcqbfi5051w_8v7W@mail.gmail.com>

Hi Seung,

On Fri, Dec 3, 2010 at 11:19 AM, Seung Soo, <sungsuha at gmail.com> wrote:
> http://pypy.org/download/pypy-1.4-src.zip
>
> Down for everyone or is it just me?

Works for me.  Try again?


A bient?t,

Armin.


From fijall at gmail.com  Fri Dec  3 12:02:37 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 3 Dec 2010 13:02:37 +0200
Subject: [pypy-dev] Windows source link
In-Reply-To: <AANLkTimCSB_-3QBMcUokiVueSdAGZcqbfi5051w_8v7W@mail.gmail.com>
References: <loom.20101203T111928-339@post.gmane.org>
	<AANLkTimCSB_-3QBMcUokiVueSdAGZcqbfi5051w_8v7W@mail.gmail.com>
Message-ID: <AANLkTinL8QfG=pWKHqN2m=LEFQDE4uODb-isK7nk+jL1@mail.gmail.com>

I fixed it yesterday

On Fri, Dec 3, 2010 at 12:32 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Seung,
>
> On Fri, Dec 3, 2010 at 11:19 AM, Seung Soo, <sungsuha at gmail.com> wrote:
>> http://pypy.org/download/pypy-1.4-src.zip
>>
>> Down for everyone or is it just me?
>
> Works for me. ?Try again?
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From drsalists at gmail.com  Fri Dec  3 20:04:23 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 3 Dec 2010 11:04:23 -0800
Subject: [pypy-dev] Testing if a function exists?
Message-ID: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>

How does one test if a function exists in pypy?

In CPython 2.x and 3.x, it appears to be sufficient to use:

   'funcname' in __bultins__

...but that doesn't appear to work in pypy 1.4:

>>>> print 'platform_version' in platform
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: 'module' object is not iterable
>>>> print 'bytes' in __builtins__
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: 'module' object is not iterable
>>>>

TIA!


From alex.gaynor at gmail.com  Fri Dec  3 20:07:55 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Fri, 3 Dec 2010 14:07:55 -0500
Subject: [pypy-dev] Testing if a function exists?
In-Reply-To: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
References: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
Message-ID: <AANLkTik1X9zp8HbT9BHioUgn-90C2VyQ98YZneDzpu9C@mail.gmail.com>

On Fri, Dec 3, 2010 at 2:04 PM, Dan Stromberg <drsalists at gmail.com> wrote:

> How does one test if a function exists in pypy?
>
> In CPython 2.x and 3.x, it appears to be sufficient to use:
>
>   'funcname' in __bultins__
>
> ...but that doesn't appear to work in pypy 1.4:
>
> >>>> print 'platform_version' in platform
> Traceback (most recent call last):
>  File "<console>", line 1, in <module>
> TypeError: 'module' object is not iterable
> >>>> print 'bytes' in __builtins__
> Traceback (most recent call last):
>  File "<console>", line 1, in <module>
> TypeError: 'module' object is not iterable
> >>>>
>
> TIA!
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

That doesn't work for me on CPython:

alex at alex-laptop:~$ python
Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> "bytes" in __builtins__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument of type 'module' is not iterable


You can use `hasattr` to check if a module has an attribute (function).

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101203/caee9603/attachment.html>

From santagada at gmail.com  Fri Dec  3 20:13:09 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Fri, 3 Dec 2010 17:13:09 -0200
Subject: [pypy-dev] Testing if a function exists?
In-Reply-To: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
References: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
Message-ID: <AANLkTikZAKg-UXzZZbq6Jqgjp2cPLAeODwceAjZg2f-L@mail.gmail.com>

(I'm guessing but) In python 2.5 modules are not iterable, you can use
getattr for the same effect.

On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg <drsalists at gmail.com> wrote:
> How does one test if a function exists in pypy?
>
> In CPython 2.x and 3.x, it appears to be sufficient to use:
>
> ? 'funcname' in __bultins__
>
> ...but that doesn't appear to work in pypy 1.4:
>
>>>>> print 'platform_version' in platform
> Traceback (most recent call last):
> ?File "<console>", line 1, in <module>
> TypeError: 'module' object is not iterable
>>>>> print 'bytes' in __builtins__
> Traceback (most recent call last):
> ?File "<console>", line 1, in <module>
> TypeError: 'module' object is not iterable
>>>>>



-- 
Leonardo Santagada


From fijall at gmail.com  Fri Dec  3 20:37:44 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 3 Dec 2010 21:37:44 +0200
Subject: [pypy-dev] Testing if a function exists?
In-Reply-To: <AANLkTikZAKg-UXzZZbq6Jqgjp2cPLAeODwceAjZg2f-L@mail.gmail.com>
References: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
	<AANLkTikZAKg-UXzZZbq6Jqgjp2cPLAeODwceAjZg2f-L@mail.gmail.com>
Message-ID: <AANLkTimihE+MXKNyFEMfkNzDZQDzt_GjxgdLkY1T4F-t@mail.gmail.com>

Use:

import __builtin__

and treat it as module. __builtins__ is an ugly hack that is sometimes
a dict and sometimes a module and pypy has different corner cases.
__builtin__ will always work

On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada <santagada at gmail.com> wrote:
> (I'm guessing but) In python 2.5 modules are not iterable, you can use
> getattr for the same effect.
>
> On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg <drsalists at gmail.com> wrote:
>> How does one test if a function exists in pypy?
>>
>> In CPython 2.x and 3.x, it appears to be sufficient to use:
>>
>> ? 'funcname' in __bultins__
>>
>> ...but that doesn't appear to work in pypy 1.4:
>>
>>>>>> print 'platform_version' in platform
>> Traceback (most recent call last):
>> ?File "<console>", line 1, in <module>
>> TypeError: 'module' object is not iterable
>>>>>> print 'bytes' in __builtins__
>> Traceback (most recent call last):
>> ?File "<console>", line 1, in <module>
>> TypeError: 'module' object is not iterable
>>>>>>
>
>
>
> --
> Leonardo Santagada
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From jacob at openend.se  Sat Dec  4 00:42:59 2010
From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=)
Date: Sat, 4 Dec 2010 00:42:59 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
Message-ID: <201012040043.04547.jacob@openend.se>

Something that would be nice on the speed.pypy.org site would be that each 
benchmark links to a page that describes the benchmark

- What the benchmark is designed to measure
- Why PyPy performs better/worse than CPYthon (if known)
- What might be done to further improve performance (if there are ideas)
- Link to the source code of the benchmark

Any takers?

Jacob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101204/3ba9958f/attachment.pgp>

From exarkun at twistedmatrix.com  Sat Dec  4 04:14:55 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sat, 04 Dec 2010 03:14:55 -0000
Subject: [pypy-dev] Idea for speed.pypy.org
Message-ID: <20101204031455.2109.1197734270.divmod.xquotient.1807@localhost.localdomain>

On 3 Dec, 11:42 pm, jacob at openend.se wrote:
>Something that would be nice on the speed.pypy.org site would be that 
>each
>benchmark links to a page that describes the benchmark
>
>- What the benchmark is designed to measure
>- Why PyPy performs better/worse than CPYthon (if known)
>- What might be done to further improve performance (if there are 
>ideas)
>- Link to the source code of the benchmark
>
>Any takers?

There's a column in the database where descriptions can be added - this 
could either be the full description you're talking about, or a link to 
a page with more information.  The description is rendered on hover, and 
below the large version of the graph (if you have a recent enough 
codespeed).

So it may just be a matter of entering some text for each benchmark (as 
opposed to making changes to the codespeed software).

Jean-Paul


From drsalists at gmail.com  Sat Dec  4 06:42:33 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 3 Dec 2010 21:42:33 -0800
Subject: [pypy-dev] Testing if a function exists?
In-Reply-To: <AANLkTimihE+MXKNyFEMfkNzDZQDzt_GjxgdLkY1T4F-t@mail.gmail.com>
References: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
	<AANLkTikZAKg-UXzZZbq6Jqgjp2cPLAeODwceAjZg2f-L@mail.gmail.com>
	<AANLkTimihE+MXKNyFEMfkNzDZQDzt_GjxgdLkY1T4F-t@mail.gmail.com>
Message-ID: <AANLkTimhM5JoA2fSVnwBXXOToWc-60-w3qNjkQou+46k@mail.gmail.com>

This sounds promising, though sadly, it doesn't appear to work in Python 3.x:

$ /usr/local/cpython-3.1/bin/python3 -c 'import __builtin__'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named __builtin__
benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
22275 - above cmd done 2010 Fri Dec 03 09:40 PM

$ /usr/local/cpython-3.0/bin/python3.0 -c 'import __builtin__'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named __builtin__
benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
22275 - above cmd done 2010 Fri Dec 03 09:40 PM

$ /usr/local/cpython-2.7/bin/python -c 'import __builtin__'
benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
22275 - above cmd done 2010 Fri Dec 03 09:41 PM

$ /usr/local/cpython-2.6/bin/python -c 'import __builtin__'
benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
22275 - above cmd done 2010 Fri Dec 03 09:41 PM

$ /usr/local/cpython-2.5/bin/python -c 'import __builtin__'
benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
22275 - above cmd done 2010 Fri Dec 03 09:41 PM

$


On Fri, Dec 3, 2010 at 11:37 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Use:
>
> import __builtin__
>
> and treat it as module. __builtins__ is an ugly hack that is sometimes
> a dict and sometimes a module and pypy has different corner cases.
> __builtin__ will always work
>
> On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>> (I'm guessing but) In python 2.5 modules are not iterable, you can use
>> getattr for the same effect.
>>
>> On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg <drsalists at gmail.com> wrote:
>>> How does one test if a function exists in pypy?
>>>
>>> In CPython 2.x and 3.x, it appears to be sufficient to use:
>>>
>>> ? 'funcname' in __bultins__
>>>
>>> ...but that doesn't appear to work in pypy 1.4:
>>>
>>>>>>> print 'platform_version' in platform
>>> Traceback (most recent call last):
>>> ?File "<console>", line 1, in <module>
>>> TypeError: 'module' object is not iterable
>>>>>>> print 'bytes' in __builtins__
>>> Traceback (most recent call last):
>>> ?File "<console>", line 1, in <module>
>>> TypeError: 'module' object is not iterable
>>>>>>>
>>
>>
>>
>> --
>> Leonardo Santagada
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>


From fijall at gmail.com  Sat Dec  4 07:15:58 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 4 Dec 2010 08:15:58 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <201012040043.04547.jacob@openend.se>
References: <201012040043.04547.jacob@openend.se>
Message-ID: <AANLkTikRCsFsYKiR0yZyCtSsbs2oVHNjSyHRBWn6WJmk@mail.gmail.com>

> - Why PyPy performs better/worse than CPYthon (if known)

I'm a bit worried about keeping this up to date


From fijall at gmail.com  Sat Dec  4 07:17:29 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 4 Dec 2010 08:17:29 +0200
Subject: [pypy-dev] Testing if a function exists?
In-Reply-To: <AANLkTimhM5JoA2fSVnwBXXOToWc-60-w3qNjkQou+46k@mail.gmail.com>
References: <AANLkTinguVO7kTNL=OB=RReghtv5Byw-gr+VPG+dAfEX@mail.gmail.com>
	<AANLkTikZAKg-UXzZZbq6Jqgjp2cPLAeODwceAjZg2f-L@mail.gmail.com>
	<AANLkTimihE+MXKNyFEMfkNzDZQDzt_GjxgdLkY1T4F-t@mail.gmail.com>
	<AANLkTimhM5JoA2fSVnwBXXOToWc-60-w3qNjkQou+46k@mail.gmail.com>
Message-ID: <AANLkTikVRsPm-W3-VwdcT4GfK8ZhSVvJaBvknNHOAdeN@mail.gmail.com>

According to this: http://docs.python.org/py3k/whatsnew/3.0.html they
renamed __builtin__ to builtins (with removed underscores). So
probably a bunch of ifs would do

On Sat, Dec 4, 2010 at 7:42 AM, Dan Stromberg <drsalists at gmail.com> wrote:
> This sounds promising, though sadly, it doesn't appear to work in Python 3.x:
>
> $ /usr/local/cpython-3.1/bin/python3 -c 'import __builtin__'
> Traceback (most recent call last):
> ?File "<string>", line 1, in <module>
> ImportError: No module named __builtin__
> benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
> 22275 - above cmd done 2010 Fri Dec 03 09:40 PM
>
> $ /usr/local/cpython-3.0/bin/python3.0 -c 'import __builtin__'
> Traceback (most recent call last):
> ?File "<string>", line 1, in <module>
> ImportError: No module named __builtin__
> benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
> 22275 - above cmd done 2010 Fri Dec 03 09:40 PM
>
> $ /usr/local/cpython-2.7/bin/python -c 'import __builtin__'
> benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
> 22275 - above cmd done 2010 Fri Dec 03 09:41 PM
>
> $ /usr/local/cpython-2.6/bin/python -c 'import __builtin__'
> benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
> 22275 - above cmd done 2010 Fri Dec 03 09:41 PM
>
> $ /usr/local/cpython-2.5/bin/python -c 'import __builtin__'
> benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu
> 22275 - above cmd done 2010 Fri Dec 03 09:41 PM
>
> $
>
>
> On Fri, Dec 3, 2010 at 11:37 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Use:
>>
>> import __builtin__
>>
>> and treat it as module. __builtins__ is an ugly hack that is sometimes
>> a dict and sometimes a module and pypy has different corner cases.
>> __builtin__ will always work
>>
>> On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>>> (I'm guessing but) In python 2.5 modules are not iterable, you can use
>>> getattr for the same effect.
>>>
>>> On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg <drsalists at gmail.com> wrote:
>>>> How does one test if a function exists in pypy?
>>>>
>>>> In CPython 2.x and 3.x, it appears to be sufficient to use:
>>>>
>>>> ? 'funcname' in __bultins__
>>>>
>>>> ...but that doesn't appear to work in pypy 1.4:
>>>>
>>>>>>>> print 'platform_version' in platform
>>>> Traceback (most recent call last):
>>>> ?File "<console>", line 1, in <module>
>>>> TypeError: 'module' object is not iterable
>>>>>>>> print 'bytes' in __builtins__
>>>> Traceback (most recent call last):
>>>> ?File "<console>", line 1, in <module>
>>>> TypeError: 'module' object is not iterable
>>>>>>>>
>>>
>>>
>>>
>>> --
>>> Leonardo Santagada
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>


From angelflow at yahoo.com  Sat Dec  4 08:21:02 2010
From: angelflow at yahoo.com (Andy)
Date: Fri, 3 Dec 2010 23:21:02 -0800 (PST)
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <201012040043.04547.jacob@openend.se>
Message-ID: <240465.94379.qm@web111314.mail.gq1.yahoo.com>

The one thing that would be the most helpful is that for each test shows the memory usage comparison between PyPy-JIT & CPython.


--- On Fri, 12/3/10, Jacob Hall?n <jacob at openend.se> wrote:

> From: Jacob Hall?n <jacob at openend.se>
> Subject: [pypy-dev] Idea for speed.pypy.org
> To: pypy-dev at codespeak.net
> Date: Friday, December 3, 2010, 6:42 PM
> Something that would be nice on the
> speed.pypy.org site would be that each 
> benchmark links to a page that describes the benchmark
> 
> - What the benchmark is designed to measure
> - Why PyPy performs better/worse than CPYthon (if known)
> - What might be done to further improve performance (if
> there are ideas)
> - Link to the source code of the benchmark
> 
> Any takers?
> 
> Jacob
> 
> -----Inline Attachment Follows-----
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


      


From tobami at googlemail.com  Sat Dec  4 08:57:51 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Sat, 4 Dec 2010 08:57:51 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <240465.94379.qm@web111314.mail.gq1.yahoo.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
Message-ID: <AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>

Hi Jacob,

benchmark descriptions is certainly something that could be improved upon.

As exarkun said, a quick way to get *something* is to just fill out
the description fields for each benchmark. I can happily do that, I
just need a mail containing a list with brief descriptions for all the
benchmarks ;-)

I can even imaging improving it further so that there is a link to
source code (your point 4) or even an info page dedicated to each
benchmark.

Points 2 and 3, as Maciej said, is problematic. Reasons why it
performs better or worse than cpython and how it could be improved is
something that may change fairly often for some benchmarks, which
would be a pain to maintain. And speed.pypy.org doesn't have 3 or 4
benchmarks, it has 25 right now!

@Andy: memory consumption is difficult to measure. I think someone was
looking at how to best measure it?

Cheers,
Miquel


2010/12/4 Andy <angelflow at yahoo.com>:
> The one thing that would be the most helpful is that for each test shows the memory usage comparison between PyPy-JIT & CPython.
>
>
> --- On Fri, 12/3/10, Jacob Hall?n <jacob at openend.se> wrote:
>
>> From: Jacob Hall?n <jacob at openend.se>
>> Subject: [pypy-dev] Idea for speed.pypy.org
>> To: pypy-dev at codespeak.net
>> Date: Friday, December 3, 2010, 6:42 PM
>> Something that would be nice on the
>> speed.pypy.org site would be that each
>> benchmark links to a page that describes the benchmark
>>
>> - What the benchmark is designed to measure
>> - Why PyPy performs better/worse than CPYthon (if known)
>> - What might be done to further improve performance (if
>> there are ideas)
>> - Link to the source code of the benchmark
>>
>> Any takers?
>>
>> Jacob
>>
>> -----Inline Attachment Follows-----
>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From lac at openend.se  Sat Dec  4 12:05:56 2010
From: lac at openend.se (Laura Creighton)
Date: Sat, 04 Dec 2010 12:05:56 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Sat,
	04 Dec 2010 08:57:51 +0100."
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com> 
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com> 
Message-ID: <201012041105.oB4B5uo1019532@theraft.openend.se>

re: keeping the 'why we are slower/what we could do to fix it' info up
to date -- one possibility is to make a 'why we were/what we are for
release 1.4.'  Then every time you make a major release, you update
those fields as needed.  And if major changes happen between 'what
was in the last major release' vs 'what's on trunk now' you can even
make a note of it -- 'fixed in rev whatever it was, when we merged in
whoever it was' brank, see blog post over here'.  And if you forget,
well, you will catch it when the next major release comes out.

Just an idea.

Laura



From fijall at gmail.com  Sat Dec  4 13:35:18 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 4 Dec 2010 14:35:18 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <201012041105.oB4B5uo1019532@theraft.openend.se>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
Message-ID: <AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>

On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton <lac at openend.se> wrote:
> re: keeping the 'why we are slower/what we could do to fix it' info up
> to date -- one possibility is to make a 'why we were/what we are for
> release 1.4.' ?Then every time you make a major release, you update
> those fields as needed. ?And if major changes happen between 'what
> was in the last major release' vs 'what's on trunk now' you can even
> make a note of it -- 'fixed in rev whatever it was, when we merged in
> whoever it was' brank, see blog post over here'. ?And if you forget,
> well, you will catch it when the next major release comes out.
>
> Just an idea.
>
> Laura
>

I think the general idea "we know what's wrong" vs "we have no clue"
is good. However, I would like to maybe decide what we do with tons of
docs that we already have first :)


From tobami at googlemail.com  Wed Dec  8 19:41:51 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 8 Dec 2010 19:41:51 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
Message-ID: <AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>

Hey, I was serious when I said I would improve on benchmarks info!

Anyone want to help in creating that benchmark description list? Just
reply to this mail so that everyone can "review" what the benchmarks
do.


2010/12/4 Maciej Fijalkowski <fijall at gmail.com>:
> On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton <lac at openend.se> wrote:
>> re: keeping the 'why we are slower/what we could do to fix it' info up
>> to date -- one possibility is to make a 'why we were/what we are for
>> release 1.4.' ?Then every time you make a major release, you update
>> those fields as needed. ?And if major changes happen between 'what
>> was in the last major release' vs 'what's on trunk now' you can even
>> make a note of it -- 'fixed in rev whatever it was, when we merged in
>> whoever it was' brank, see blog post over here'. ?And if you forget,
>> well, you will catch it when the next major release comes out.
>>
>> Just an idea.
>>
>> Laura
>>
>
> I think the general idea "we know what's wrong" vs "we have no clue"
> is good. However, I would like to maybe decide what we do with tons of
> docs that we already have first :)
>


From fijall at gmail.com  Wed Dec  8 20:03:36 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 8 Dec 2010 21:03:36 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
Message-ID: <AANLkTik4Vghj+KexHndsQfGpeQwaJA3n8PuRtPqwFQ=L@mail.gmail.com>

On Wed, Dec 8, 2010 at 8:41 PM, Miquel Torres <tobami at googlemail.com> wrote:
> Hey, I was serious when I said I would improve on benchmarks info!
>
> Anyone want to help in creating that benchmark description list? Just
> reply to this mail so that everyone can "review" what the benchmarks
> do.
>

Hey.

I can do some stuff. I was serious about documenting why we're
slow/fast on benchmarks - that maybe we should bring down our docs to
a manageable number first :) Benchmarks descriptions are however
unlikely to change

>
> 2010/12/4 Maciej Fijalkowski <fijall at gmail.com>:
>> On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton <lac at openend.se> wrote:
>>> re: keeping the 'why we are slower/what we could do to fix it' info up
>>> to date -- one possibility is to make a 'why we were/what we are for
>>> release 1.4.' ?Then every time you make a major release, you update
>>> those fields as needed. ?And if major changes happen between 'what
>>> was in the last major release' vs 'what's on trunk now' you can even
>>> make a note of it -- 'fixed in rev whatever it was, when we merged in
>>> whoever it was' brank, see blog post over here'. ?And if you forget,
>>> well, you will catch it when the next major release comes out.
>>>
>>> Just an idea.
>>>
>>> Laura
>>>
>>
>> I think the general idea "we know what's wrong" vs "we have no clue"
>> is good. However, I would like to maybe decide what we do with tons of
>> docs that we already have first :)
>>
>


From santagada at gmail.com  Thu Dec  9 14:14:44 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 9 Dec 2010 11:14:44 -0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
Message-ID: <AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>

Here is a incomplete draft list:

ai:

chaos: create chaosgame-like fractals

fannkuch: http://shootout.alioth.debian.org/u64/performance.php?test=fannkuchredux

float: (this is just from looking at it) creates an array of points
using circular projection and then normalizes and maximizes them.

html5:

meteor-contest:
http://shootout.alioth.debian.org/u64/performance.php?test=meteor

nbody_modified: http://shootout.alioth.debian.org/u64/performance.php?test=nbody

richards: Martin Richards benchmark, implemented in many languages

rietveld: A Django application benchmark

[slow]spitfire[cstringio]: Spitfire is a template language, the
cstringio version uses a modified engine (that uses cstringio)

spambayes: Spambayes is a bayesian spam filter

telco:

go: A go (chess like game) computer player AI

pyflate-fast: Stand-alone pure-Python DEFLATE (gzip) and bzip2
decoder/decompressor.

raytrace: A raytracer renderer

crypto_pyaes: A pure python implementation of AES

waf: Waf is a Python-based framework for configuring, compiling and
installing applications. It derives from the concepts of other build
tools such as Scons, Autotools, CMake or Ant.

On Wed, Dec 8, 2010 at 4:41 PM, Miquel Torres <tobami at googlemail.com> wrote:
> Anyone want to help in creating that benchmark description list? Just
> reply to this mail so that everyone can "review" what the benchmarks
> do.




-- 
Leonardo Santagada


From jacob at openend.se  Thu Dec  9 14:15:21 2010
From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=)
Date: Thu, 9 Dec 2010 14:15:21 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTik4Vghj+KexHndsQfGpeQwaJA3n8PuRtPqwFQ=L@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTik4Vghj+KexHndsQfGpeQwaJA3n8PuRtPqwFQ=L@mail.gmail.com>
Message-ID: <201012091415.27316.jacob@openend.se>

Wednesday 08 December 2010 you wrote:
> On Wed, Dec 8, 2010 at 8:41 PM, Miquel Torres <tobami at googlemail.com> wrote:
> > Hey, I was serious when I said I would improve on benchmarks info!
> > 
> > Anyone want to help in creating that benchmark description list? Just
> > reply to this mail so that everyone can "review" what the benchmarks
> > do.
> 
> Hey.
> 
> I can do some stuff. I was serious about documenting why we're
> slow/fast on benchmarks - that maybe we should bring down our docs to
> a manageable number first :) Benchmarks descriptions are however
> unlikely to change

Just having the descriptions would be great. Why we are slow/fast was a 
wishlist item, and I can see that it is more difficult to provide this 
information and keep it up to date.

Providing Miguel with the information can be done in the simplest way, for 
instance by mailing texts to this list (so that people can see which ones have 
been produced)..

Jacob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101209/3a221a57/attachment.pgp>

From jacob at openend.se  Thu Dec  9 15:30:35 2010
From: jacob at openend.se (Jacob =?iso-8859-15?q?Hall=E9n?=)
Date: Thu, 9 Dec 2010 15:30:35 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <201012091415.27316.jacob@openend.se>
References: <201012040043.04547.jacob@openend.se>
	<AANLkTik4Vghj+KexHndsQfGpeQwaJA3n8PuRtPqwFQ=L@mail.gmail.com>
	<201012091415.27316.jacob@openend.se>
Message-ID: <201012091530.41121.jacob@openend.se>

Extracted from what exarkun said on the IRC channel.

twisted-tcp:

Connects one Twised client to one Twisted server over TCP (on the loopback 
interface) and then writes bytes as fast as it can.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101209/2865cc1b/attachment.pgp>

From renesd at gmail.com  Thu Dec  9 15:50:28 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Thu, 9 Dec 2010 14:50:28 +0000
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <201012091530.41121.jacob@openend.se>
References: <201012040043.04547.jacob@openend.se>
	<AANLkTik4Vghj+KexHndsQfGpeQwaJA3n8PuRtPqwFQ=L@mail.gmail.com>
	<201012091415.27316.jacob@openend.se>
	<201012091530.41121.jacob@openend.se>
Message-ID: <AANLkTim7fCkQepkMBqxTKm=-iE55rvBfpC78iOE1-fG9@mail.gmail.com>

Hello,

Why is this twisted tcp benchmark slow in pypy?  Does memory management play
a role?  Or perhaps the interface to the system calls?  eg, is there no
event io being used by the twisted version?

I think in CPython, it would be using a c binding, and not allocating any
memory as it would all be coming from memory pools.  It's probably using the
same pieces of memory too.

cu.

On Thu, Dec 9, 2010 at 2:30 PM, Jacob Hall?n <jacob at openend.se> wrote:

> Extracted from what exarkun said on the IRC channel.
>
> twisted-tcp:
>
> Connects one Twised client to one Twisted server over TCP (on the loopback
> interface) and then writes bytes as fast as it can.
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101209/4de8a1f6/attachment.html>

From p.giarrusso at gmail.com  Thu Dec  9 16:13:09 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 9 Dec 2010 16:13:09 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
Message-ID: <AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>

On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com> wrote:
> Here is a incomplete draft list:

> [slow]spitfire[cstringio]: Spitfire is a template language, the
> cstringio version uses a modified engine (that uses cstringio)

> spambayes: Spambayes is a bayesian spam filter

Why is [slow]spitfire slower with PyPy? Is it regex-related? I
remember when, because of this, spambayes was slower (including
release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
and the head (for slowspitfire it's the opposite).

For the rest, I see no significant case of slowdown of PyPy over time.
http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From arigo at tunes.org  Thu Dec  9 16:14:15 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Dec 2010 16:14:15 +0100
Subject: [pypy-dev] Leysin Winter sprint: Jan 16-22
Message-ID: <AANLkTikxwXvgRd+yHScU8u+U83bz9Z__6PDa+MTapZw=@mail.gmail.com>

=====================================================================
         PyPy Leysin Winter Sprint (16-22nd January 2011)
=====================================================================

As announced on http://morepypy.blogspot.com/2010/12/leysin-winter-sprint.html,
the next PyPy sprint will be in Leysin, Switzerland, for the
seventh time.  This is a fully public sprint: newcomers and topics
other than those proposed below are welcome.

------------------------------
Goals and topics of the sprint
------------------------------

* Now that we have released 1.4, and plan to release 1.4.1 soon
  (possibly before the sprint), the sprint itself is going to be
  mainly working on fixing issues reported by various users. Of
  course this does not prevent people from showing up with a more
  precise interest in mind.  If there are newcomers, we will gladly
  give introduction talks.

* We will also work on polishing and merging the long-standing
  branches that are around, which could eventually lead to the
  next PyPy release.  These branches are notably:

  - fast-forward (Python 2.7 support, by Benjamin, Amaury, and others)
  - jit-unroll-loops (improve JITting of smaller loops, by Hakan)
  - arm-backend (a JIT backend for ARM, by David)
  - jitypes2 (fast ctypes calls with the JIT, by Antonio).

* And as usual, the main side goal is to have fun in winter sports :-)
  We can take a day off for ski.

-----------
Exact times
-----------

The work days should be 16-22 January 2011.  People may arrive on
the 15th already and/or leave on the 23rd.

-----------------------
Location & Accomodation
-----------------------

Leysin, Switzerland, "same place as before".  Let me refresh your
memory: both the sprint venue and the lodging will be in a very spacious
pair of chalets built specifically for bed & breakfast:
http://www.ermina.ch/.  The place has a good ADSL Internet connexion
with wireless installed.  You can of course arrange your own
lodging anywhere (as long as you are in Leysin, you cannot be more than a
15 minutes walk away from the sprint venue), but I definitely recommend
lodging there too -- you won't find a better view anywhere else (though you
probably won't get much worse ones easily, either :-)

Please *confirm* that you are coming so that we can adjust the reservations
as appropriate.  The rate so far has been around 60 CHF a night all included
in 2-person rooms, with breakfast.  There are larger rooms too (less
expensive) and maybe the possibility to get a single room if you really want
to.

Please register by svn:

  http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2011/people.txt

or on the pypy-sprint mailing list if you do not yet have check-in rights:

  http://codespeak.net/mailman/listinfo/pypy-sprint

You need a Swiss-to-(insert country here) power adapter.  There will be
some Swiss-to-EU adapters around -- bring a EU-format power strip if you
have one.

=====================================================================
Armin Rigo


From drsalists at gmail.com  Thu Dec  9 17:25:40 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 9 Dec 2010 08:25:40 -0800
Subject: [pypy-dev] os.major? os.minor?
Message-ID: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>

What would be the best way to add os.major and os.minor to the os module?

They appear to be C macros on some systems, so I'm guessing most FFI's
aren't going to see them without a wrapper.

On Linux, there's a C function that a macro is #ifdef'd to, but that
seems less than amazingly portable.


From arigo at tunes.org  Thu Dec  9 17:56:17 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Dec 2010 17:56:17 +0100
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
Message-ID: <AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>

Hi Dan,

On Thu, Dec 9, 2010 at 5:25 PM, Dan Stromberg <drsalists at gmail.com> wrote:
> What would be the best way to add os.major and os.minor to the os module?

It's amazing the kind of stuff there is in the 'os' module that I've
never heard about before. :-)

> They appear to be C macros on some systems, so I'm guessing most FFI's
> aren't going to see them without a wrapper.

Indeed.  See for example pypy.rlib._rsocket_rffi.FD_ZERO, which is
also typically a macro.  There is a bit of workaround code to write,
I'm afraid.  This is mostly to support direct testing; if the goal is
only to have it work after translation, then it does not matter that
it is actually a macro.


A bient?t,

Armin.


From drsalists at gmail.com  Thu Dec  9 18:20:04 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 9 Dec 2010 09:20:04 -0800
Subject: [pypy-dev] gdbm and dbm merge - ctypes bug?
Message-ID: <AANLkTimpHpTjBcPehZJGNBnxOJv83PKQyaEA7neZxD7J@mail.gmail.com>

I'm finding that my gdbm module (based on ctypes) is infrequently (but
it might be consistent) giving gibberish when run on pypy 1.4, but not
when run on cpython 2.6 or cpython 3.1.

More specifically, sometimes the keys come up looking like undefined
memory.  The values associated with those keys always seem OK.

I believe I am comparing apples to apples - I'm using gdbm_ctypes with
the cpython's too, even though they come with a module for some doing
the same thing - for the sake of comparison.

Is there a known problem with pypy's ctypes?

The code is at http://stromberg.dnsalias.org/cgi-bin/viewvc.cgi/gdbm-ctypes/trunk/?root=svn

The tests are at
http://stromberg.dnsalias.org/cgi-bin/viewvc.cgi/backshift/trunk/?root=svn
, and are a bit involved right now, but I could look into finding a
minimal code snippet that shows the same problem.


From arigo at tunes.org  Thu Dec  9 19:04:36 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Dec 2010 19:04:36 +0100
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
Message-ID: <AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>

Re-hi,

I have opened an issue about the missing functions from the 'os' module:
https://codespeak.net/issue/pypy-dev/issue605

I will probably get around to implement a part of them in the next few
days.  If you plan to do more than os.major() and os.minor(), let's
coordinate there.


A bient?t,

Armin.


From amauryfa at gmail.com  Thu Dec  9 19:05:56 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 9 Dec 2010 19:05:56 +0100
Subject: [pypy-dev] gdbm and dbm merge - ctypes bug?
In-Reply-To: <AANLkTimpHpTjBcPehZJGNBnxOJv83PKQyaEA7neZxD7J@mail.gmail.com>
References: <AANLkTimpHpTjBcPehZJGNBnxOJv83PKQyaEA7neZxD7J@mail.gmail.com>
Message-ID: <AANLkTikBQKP8gGq-hBZP+xSDD61aDwqfgcO9dcSeNn=p@mail.gmail.com>

Hi

2010/12/9 Dan Stromberg <drsalists at gmail.com>

> Is there a known problem with pypy's ctypes?


Yes, several of them have already been fixed in trunk, especially
about memory chunks being freed too early.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101209/e0d9cc32/attachment.html>

From arigo at tunes.org  Thu Dec  9 19:33:35 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Dec 2010 19:33:35 +0100
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
Message-ID: <AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>

Hi,

Sorry for the series of mails.  I just wanted to add that after
looking more at the code, it's fine if you implement major() and
minor() the same way as the other posix functions (see e.g. r79941
adding os.fchdir()), calling the major() and minor() "functions" in C.
 If you add tests in the same places I did in r79941, then it should
work even if the functions are actually macros.  One place where it
would not work is in translated code running on the llinterp; the
other place might be the JIT trying to get the address of the
"function".  (If that's the case, we should fix that a bit more
generally.)


A bient?t,

Armin.


From fijall at gmail.com  Thu Dec  9 21:39:52 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 9 Dec 2010 22:39:52 +0200
Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix
 module/posix/test rpython/module translator/c/test
In-Reply-To: <20101209173035.299A0282B9D@codespeak.net>
References: <20101209173035.299A0282B9D@codespeak.net>
Message-ID: <AANLkTinW6AtakRUwWCnkT1woKpzSh6T5Rn4ZUnMEgf57@mail.gmail.com>

Do we *really* need those functions available on RPython level?

On Thu, Dec 9, 2010 at 7:30 PM,  <arigo at codespeak.net> wrote:
> Author: arigo
> Date: Thu Dec ?9 18:30:33 2010
> New Revision: 79938
>
> Modified:
> ? pypy/trunk/pypy/module/posix/__init__.py
> ? pypy/trunk/pypy/module/posix/interp_posix.py
> ? pypy/trunk/pypy/module/posix/test/test_posix2.py
> ? pypy/trunk/pypy/rpython/module/ll_os.py
> ? pypy/trunk/pypy/translator/c/test/test_extfunc.py
> Log:
> Implement os.getloadavg(). ?(Phew, we really need to edit files in all
> corners of the world for this...)
>
>
> Modified: pypy/trunk/pypy/module/posix/__init__.py
> ==============================================================================
> --- pypy/trunk/pypy/module/posix/__init__.py ? ?(original)
> +++ pypy/trunk/pypy/module/posix/__init__.py ? ?Thu Dec ?9 18:30:33 2010
> @@ -111,6 +111,8 @@
> ? ? ? ? interpleveldefs['sysconf_names'] = 'space.wrap(os.sysconf_names)'
> ? ? if hasattr(os, 'ttyname'):
> ? ? ? ? interpleveldefs['ttyname'] = 'interp_posix.ttyname'
> + ? ?if hasattr(os, 'getloadavg'):
> + ? ? ? ?interpleveldefs['getloadavg'] = 'interp_posix.getloadavg'
>
> ? ? for name in ['setsid', 'getuid', 'geteuid', 'getgid', 'getegid', 'setuid',
> ? ? ? ? ? ? ? ? ?'seteuid', 'setgid', 'setegid', 'getpgrp', 'setpgrp',
>
> Modified: pypy/trunk/pypy/module/posix/interp_posix.py
> ==============================================================================
> --- pypy/trunk/pypy/module/posix/interp_posix.py ? ? ? ?(original)
> +++ pypy/trunk/pypy/module/posix/interp_posix.py ? ? ? ?Thu Dec ?9 18:30:33 2010
> @@ -963,6 +963,17 @@
> ? ? return space.w_None
> ?chown.unwrap_spec = [ObjSpace, str, "c_nonnegint", "c_nonnegint"]
>
> +def getloadavg(space):
> + ? ?try:
> + ? ? ? ?load = os.getloadavg()
> + ? ?except OSError, e:
> + ? ? ? ?raise OperationError(space.w_OSError,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap("Load averages are unobtainable"))
> + ? ?return space.newtuple([space.wrap(load[0]),
> + ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap(load[1]),
> + ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap(load[2])])
> +getloadavg.unwrap_spec = [ObjSpace]
> +
> ?if _WIN:
> ? ? from pypy.rlib import rwin32
>
>
> Modified: pypy/trunk/pypy/module/posix/test/test_posix2.py
> ==============================================================================
> --- pypy/trunk/pypy/module/posix/test/test_posix2.py ? ?(original)
> +++ pypy/trunk/pypy/module/posix/test/test_posix2.py ? ?Thu Dec ?9 18:30:33 2010
> @@ -521,6 +521,14 @@
> ? ? ? ? ? ? ? ? assert os.WIFEXITED(status)
> ? ? ? ? ? ? ? ? assert os.WEXITSTATUS(status) == exit_status
>
> + ? ?if hasattr(os, 'getloadavg'):
> + ? ? ? ?def test_os_getloadavg(self):
> + ? ? ? ? ? ?os = self.posix
> + ? ? ? ? ? ?l0, l1, l2 = os.getloadavg()
> + ? ? ? ? ? ?assert type(l0) is float and l0 >= 0.0
> + ? ? ? ? ? ?assert type(l1) is float and l0 >= 0.0
> + ? ? ? ? ? ?assert type(l2) is float and l0 >= 0.0
> +
> ? ? if hasattr(os, 'fsync'):
> ? ? ? ? def test_fsync(self):
> ? ? ? ? ? ? os = self.posix
>
> Modified: pypy/trunk/pypy/rpython/module/ll_os.py
> ==============================================================================
> --- pypy/trunk/pypy/rpython/module/ll_os.py ? ? (original)
> +++ pypy/trunk/pypy/rpython/module/ll_os.py ? ? Thu Dec ?9 18:30:33 2010
> @@ -730,6 +730,22 @@
> ? ? ? ? return extdef([traits.str, int, int], int, traits.ll_os_name('open'),
> ? ? ? ? ? ? ? ? ? ? ? llimpl=os_open_llimpl, oofakeimpl=os_open_oofakeimpl)
>
> + ? ?@registering_if(os, 'getloadavg')
> + ? ?def register_os_getloadavg(self):
> + ? ? ? ?AP = rffi.CArrayPtr(lltype.Float)
> + ? ? ? ?c_getloadavg = self.llexternal('getloadavg', [AP, rffi.INT], rffi.INT)
> +
> + ? ? ? ?def getloadavg_llimpl():
> + ? ? ? ? ? ?load = lltype.malloc(AP.TO, 3, flavor='raw')
> + ? ? ? ? ? ?r = c_getloadavg(load, 3)
> + ? ? ? ? ? ?result_tuple = load[0], load[1], load[2]
> + ? ? ? ? ? ?lltype.free(load, flavor='raw')
> + ? ? ? ? ? ?if r != 3:
> + ? ? ? ? ? ? ? ?raise OSError
> + ? ? ? ? ? ?return result_tuple
> + ? ? ? ?return extdef([], (float, float, float),
> + ? ? ? ? ? ? ? ? ? ? ?"ll_os.ll_getloadavg", llimpl=getloadavg_llimpl)
> +
> ?# ------------------------------- os.read -------------------------------
>
> ? ? @registering(os.read)
>
> Modified: pypy/trunk/pypy/translator/c/test/test_extfunc.py
> ==============================================================================
> --- pypy/trunk/pypy/translator/c/test/test_extfunc.py ? (original)
> +++ pypy/trunk/pypy/translator/c/test/test_extfunc.py ? Thu Dec ?9 18:30:33 2010
> @@ -755,3 +755,13 @@
> ? ? ? ? for i in range(5):
> ? ? ? ? ? ? res = func(i)
> ? ? ? ? ? ? assert res == os.uname()[i]
> +
> +if hasattr(os, 'getloadavg'):
> + ? ?def test_os_getloadavg():
> + ? ? ? ?def does_stuff():
> + ? ? ? ? ? ?a, b, c = os.getloadavg()
> + ? ? ? ? ? ?print a, b, c
> + ? ? ? ? ? ?return a + b + c
> + ? ? ? ?f = compile(does_stuff, [])
> + ? ? ? ?res = f()
> + ? ? ? ?assert type(res) is float and res >= 0.0
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>

From benjamin at python.org  Fri Dec 10 02:45:46 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 9 Dec 2010 19:45:46 -0600
Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix
 module/posix/test rpython/module translator/c/test
In-Reply-To: <AANLkTinW6AtakRUwWCnkT1woKpzSh6T5Rn4ZUnMEgf57@mail.gmail.com>
References: <20101209173035.299A0282B9D@codespeak.net>
	<AANLkTinW6AtakRUwWCnkT1woKpzSh6T5Rn4ZUnMEgf57@mail.gmail.com>
Message-ID: <AANLkTimQLrGbP40BqCv-++8kKBGTC6eLDr7jRCCWAj4u@mail.gmail.com>

2010/12/9 Maciej Fijalkowski <fijall at gmail.com>:
> Do we *really* need those functions available on RPython level?

Even if we don't, I'd say the consistency is nice, rather than having
ll implementations all over creation.



-- 
Regards,
Benjamin


From arigo at tunes.org  Fri Dec 10 10:27:19 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 10 Dec 2010 10:27:19 +0100
Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix
 module/posix/test rpython/module translator/c/test
In-Reply-To: <AANLkTimQLrGbP40BqCv-++8kKBGTC6eLDr7jRCCWAj4u@mail.gmail.com>
References: <20101209173035.299A0282B9D@codespeak.net>
	<AANLkTinW6AtakRUwWCnkT1woKpzSh6T5Rn4ZUnMEgf57@mail.gmail.com>
	<AANLkTimQLrGbP40BqCv-++8kKBGTC6eLDr7jRCCWAj4u@mail.gmail.com>
Message-ID: <AANLkTinNc2f9uBu4zDyjq16c12St1K7otozrN_p3Hzwc@mail.gmail.com>

Hi,

On Fri, Dec 10, 2010 at 2:45 AM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/12/9 Maciej Fijalkowski <fijall at gmail.com>:
>> Do we *really* need those functions available on RPython level?
>
> Even if we don't, I'd say the consistency is nice, rather than having
> ll implementations all over creation.

Agreed.  Consistency is probably better than having pypy/module/posix
be half written in the way it is now and half written in a
lltype-specific way.


A bient?t,

Armin.


From arigo at tunes.org  Fri Dec 10 17:01:58 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 10 Dec 2010 17:01:58 +0100
Subject: [pypy-dev] Leysin Winter sprint: Jan 16-22
In-Reply-To: <AANLkTikxwXvgRd+yHScU8u+U83bz9Z__6PDa+MTapZw=@mail.gmail.com>
References: <AANLkTikxwXvgRd+yHScU8u+U83bz9Z__6PDa+MTapZw=@mail.gmail.com>
Message-ID: <AANLkTinn_NEJnbF2u1bjMjeGiVtfkAfvJXyR6S3V9POS@mail.gmail.com>

Hi,

An additional comment (thanks Michael Foord for pointing this out to
me): the Python Software Foundation (PSF) now has some money to use to
help non-core-Python sprints.  This means that if anyone is interested
in getting some financial support for the travel to or lodging at
Leysin, it might be possible from the PSF.  Please write to me or to
pypy-sprint at codespeak.net for more details.


A bient?t,

Armin.


From drsalists at gmail.com  Fri Dec 10 20:49:11 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 10 Dec 2010 11:49:11 -0800
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
	<AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
Message-ID: <AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>

On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> Sorry for the series of mails. ?I just wanted to add that after
> looking more at the code, it's fine if you implement major() and
> minor() the same way as the other posix functions (see e.g. r79941
> adding os.fchdir()), calling the major() and minor() "functions" in C.
> ?If you add tests in the same places I did in r79941, then it should
> work even if the functions are actually macros. ?One place where it
> would not work is in translated code running on the llinterp; the
> other place might be the JIT trying to get the address of the
> "function". ?(If that's the case, we should fix that a bit more
> generally.)
>
>
> A bient?t,
>
> Armin.

I attempted to add os.major and os.minor using fchdir as an example,
but...  I have to admit: I really don't know what to make of this
error.  Comments?

[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 290, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py",
line 812, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/tool/taskengine.py",
line 116, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py",
line 288, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py",
line 325, in task_annotate
[translation:ERROR]     s = annotator.build_types(self.entry_point,
self.inputtypes)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 103, in build_types
[translation:ERROR]     return self.build_graph_types(flowgraph,
inputcells, complete_now=complete_now)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 194, in build_graph_types
[translation:ERROR]     self.complete()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 250, in complete
[translation:ERROR]     self.processblock(graph, block)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 473, in processblock
[translation:ERROR]     self.flowin(graph, block)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 533, in flowin
[translation:ERROR]     self.consider_op(block.operations[i])
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 716, in consider_op
[translation:ERROR]     argcells = [self.binding(a) for a in op.args]
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py",
line 293, in binding
[translation:ERROR]     return self.bookkeeper.immutableconstant(arg)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py",
line 314, in immutableconstant
[translation:ERROR]     return self.immutablevalue(const.value)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py",
line 464, in immutablevalue
[translation:ERROR]     result = SomePBC([self.getdesc(x)])
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py",
line 530, in getdesc
[translation:ERROR]     pyobj,))
[translation:ERROR]  Exception: unexpected prebuilt constant:
<built-in function minor>
[translation:ERROR] Processing block:
[translation:ERROR]  block at 12 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.posix.interp_posix:179)minor
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v775 =
simple_call((builtin_function_or_method minor), raw_device_no_0)
[translation:ERROR]  --end--
[translation] start debugger...


From benjamin at python.org  Fri Dec 10 20:50:36 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 10 Dec 2010 13:50:36 -0600
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
	<AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
	<AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>
Message-ID: <AANLkTi=HgRVrAHJtX-hYuuQxBVOsSrHmOs6hAh+8sXFA@mail.gmail.com>

2010/12/10 Dan Stromberg <drsalists at gmail.com>:
> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo <arigo at tunes.org> wrote:
>> Hi,
>>
>> Sorry for the series of mails. ?I just wanted to add that after
>> looking more at the code, it's fine if you implement major() and
>> minor() the same way as the other posix functions (see e.g. r79941
>> adding os.fchdir()), calling the major() and minor() "functions" in C.
>> ?If you add tests in the same places I did in r79941, then it should
>> work even if the functions are actually macros. ?One place where it
>> would not work is in translated code running on the llinterp; the
>> other place might be the JIT trying to get the address of the
>> "function". ?(If that's the case, we should fix that a bit more
>> generally.)
>>
>>
>> A bient?t,
>>
>> Armin.
>
> I attempted to add os.major and os.minor using fchdir as an example,
> but... ?I have to admit: I really don't know what to make of this
> error. ?Comments?

You probably failed to add an ll implementation in
pypy/rpython/lltypesystem/module/ll_os.py


-- 
Regards,
Benjamin


From drsalists at gmail.com  Fri Dec 10 22:04:31 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 10 Dec 2010 13:04:31 -0800
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTi=HgRVrAHJtX-hYuuQxBVOsSrHmOs6hAh+8sXFA@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
	<AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
	<AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>
	<AANLkTi=HgRVrAHJtX-hYuuQxBVOsSrHmOs6hAh+8sXFA@mail.gmail.com>
Message-ID: <AANLkTimBtv4mqe0vv1vrLKH=3LZ2ZFhNYGn-ivzzwDy+@mail.gmail.com>

On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/12/10 Dan Stromberg <drsalists at gmail.com>:
>> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo <arigo at tunes.org> wrote:
>>> Hi,
>>>
>>> Sorry for the series of mails. ?I just wanted to add that after
>>> looking more at the code, it's fine if you implement major() and
>>> minor() the same way as the other posix functions (see e.g. r79941
>>> adding os.fchdir()), calling the major() and minor() "functions" in C.
>>> ?If you add tests in the same places I did in r79941, then it should
>>> work even if the functions are actually macros. ?One place where it
>>> would not work is in translated code running on the llinterp; the
>>> other place might be the JIT trying to get the address of the
>>> "function". ?(If that's the case, we should fix that a bit more
>>> generally.)
>>>
>>>
>>> A bient?t,
>>>
>>> Armin.
>>
>> I attempted to add os.major and os.minor using fchdir as an example,
>> but... ?I have to admit: I really don't know what to make of this
>> error. ?Comments?
>
> You probably failed to add an ll implementation in
> pypy/rpython/lltypesystem/module/ll_os.py

Do you mean pypy/rpython/module/ll_os.py ?  I have no
pypy/rpython/lltypesystem/module/ll_os.py .


From benjamin at python.org  Sat Dec 11 02:30:25 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 10 Dec 2010 19:30:25 -0600
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTimBtv4mqe0vv1vrLKH=3LZ2ZFhNYGn-ivzzwDy+@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
	<AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
	<AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>
	<AANLkTi=HgRVrAHJtX-hYuuQxBVOsSrHmOs6hAh+8sXFA@mail.gmail.com>
	<AANLkTimBtv4mqe0vv1vrLKH=3LZ2ZFhNYGn-ivzzwDy+@mail.gmail.com>
Message-ID: <AANLkTim4PHZWdbtFL4K599emJ_ya+jvoVmhSfmxvqFr8@mail.gmail.com>

2010/12/10 Dan Stromberg <drsalists at gmail.com>:
> On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2010/12/10 Dan Stromberg <drsalists at gmail.com>:
>>> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo <arigo at tunes.org> wrote:
>>>> Hi,
>>>>
>>>> Sorry for the series of mails. ?I just wanted to add that after
>>>> looking more at the code, it's fine if you implement major() and
>>>> minor() the same way as the other posix functions (see e.g. r79941
>>>> adding os.fchdir()), calling the major() and minor() "functions" in C.
>>>> ?If you add tests in the same places I did in r79941, then it should
>>>> work even if the functions are actually macros. ?One place where it
>>>> would not work is in translated code running on the llinterp; the
>>>> other place might be the JIT trying to get the address of the
>>>> "function". ?(If that's the case, we should fix that a bit more
>>>> generally.)
>>>>
>>>>
>>>> A bient?t,
>>>>
>>>> Armin.
>>>
>>> I attempted to add os.major and os.minor using fchdir as an example,
>>> but... ?I have to admit: I really don't know what to make of this
>>> error. ?Comments?
>>
>> You probably failed to add an ll implementation in
>> pypy/rpython/lltypesystem/module/ll_os.py
>
> Do you mean pypy/rpython/module/ll_os.py ? ?I have no
> pypy/rpython/lltypesystem/module/ll_os.py .

Yes, sorry. It's easy to get those rpython/ subdirs mixed up . :)



-- 
Regards,
Benjamin


From drsalists at gmail.com  Sat Dec 11 05:30:32 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 10 Dec 2010 20:30:32 -0800
Subject: [pypy-dev] Trunk failing to build?
Message-ID: <AANLkTikWiW=MPq7HSjuERBxGRb0muDKMvx73TcF-fPTP@mail.gmail.com>

Does today's trunk fail to build?

I'm getting the following on what I believe to be an unmodified trunk:

[Timer] Timings:
[Timer] annotate                       --- 1178.9 s
[Timer] rtype_lltype                   ---  691.9 s
[Timer] pyjitpl_lltype                 --- 1287.3 s
[Timer] backendopt_lltype              ---  621.4 s
[Timer] stackcheckinsertion_lltype     ---  101.2 s
[Timer] database_c                     ---   11.8 s
[Timer] ===========================================
[Timer] Total:                         --- 3892.4 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 290, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py",
line 812, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/tool/taskengine.py",
line 116, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py",
line 288, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py",
line 508, in task_database_c
[translation:ERROR]     database = cbuilder.build_database()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/c/genc.py",
line 157, in build_database
[translation:ERROR]     sandbox=self.config.translation.sandbox)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/c/database.py",
line 65, in __init__
[translation:ERROR]     self.gctransformer =
self.gcpolicy.transformerclass(translator)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/memory/gctransform/framework.py",
line 494, in __init__
[translation:ERROR]     annhelper.finish()   # at this point, annotate
all mix-level helpers
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/annlowlevel.py",
line 240, in finish
[translation:ERROR]     self.finish_annotate()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/annlowlevel.py",
line 259, in finish_annotate
[translation:ERROR]     ann.complete_helpers(self.policy)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 176, in complete_helpers
[translation:ERROR]     self.complete()
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 250, in complete
[translation:ERROR]     self.processblock(graph, block)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 473, in processblock
[translation:ERROR]     self.flowin(graph, block)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 533, in flowin
[translation:ERROR]     self.consider_op(block.operations[i])
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 735, in consider_op
[translation:ERROR]     raise_nicer_exception(op, str(graph))
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py",
line 732, in consider_op
[translation:ERROR]     resultcell = consider_meth(*argcells)
[translation:ERROR]    File "<3972-codegen
/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py:770>",
line 3, in consider_op_simple_call
[translation:ERROR]     return arg.simple_call(*args)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/unaryop.py",
line 174, in simple_call
[translation:ERROR]     return
obj.call(getbookkeeper().build_args("simple_call", args_s))
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/unaryop.py",
line 677, in call
[translation:ERROR]     return bookkeeper.pbc_call(pbc, args)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/bookkeeper.py",
line 662, in pbc_call
[translation:ERROR]     results.append(desc.pycall(schedule, args,
s_previous_result))
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py",
line 275, in pycall
[translation:ERROR]     inputcells = self.parse_arguments(args)
[translation:ERROR]    File
"/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py",
line 255, in parse_arguments
[translation:ERROR]     raise TypeError, "signature mismatch: %s" %
e.getmsg(self.name)
[translation:ERROR]  TypeError': signature mismatch:
get_L2cache_linux2() takes exactly 1 argument (0 given)
[translation:ERROR]     .. v1047 = simple_call((function get_L2cache_linux2))
[translation:ERROR]     ..
'(pypy.rpython.memory.gc.env:244)estimate_best_nursery_size'
[translation:ERROR] Processing block:
[translation:ERROR]  block at 3 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in
(pypy.rpython.memory.gc.env:244)estimate_best_nursery_size
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v1047 = simple_call((function get_L2cache_linux2))
[translation:ERROR]        v1048 = simple_call((function
best_nursery_size_for_L2cache), v1047)
[translation:ERROR]  --end--
[translation] start debugger...
> /home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py(255)parse_arguments()
-> raise TypeError, "signature mismatch: %s" % e.getmsg(self.name)
(Pdb+)


From benjamin at python.org  Sat Dec 11 05:46:54 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 10 Dec 2010 22:46:54 -0600
Subject: [pypy-dev] Trunk failing to build?
In-Reply-To: <AANLkTikWiW=MPq7HSjuERBxGRb0muDKMvx73TcF-fPTP@mail.gmail.com>
References: <AANLkTikWiW=MPq7HSjuERBxGRb0muDKMvx73TcF-fPTP@mail.gmail.com>
Message-ID: <AANLkTi=QC14sXHUNyrOyxgS87d6=ozAtkagyR=2WQ2Pf@mail.gmail.com>

2010/12/10 Dan Stromberg <drsalists at gmail.com>:
> Does today's trunk fail to build?
>
> I'm getting the following on what I believe to be an unmodified trunk:

Yep. Hopefully fixed in r79965.


-- 
Regards,
Benjamin


From drsalists at gmail.com  Sat Dec 11 05:29:14 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 10 Dec 2010 20:29:14 -0800
Subject: [pypy-dev] os.major? os.minor?
In-Reply-To: <AANLkTim4PHZWdbtFL4K599emJ_ya+jvoVmhSfmxvqFr8@mail.gmail.com>
References: <AANLkTim_qfEiCD_6ki0P07uyta5tAJ-yT2to1NQ-M8sX@mail.gmail.com>
	<AANLkTik-dWpPBY7zpV4wt+qubx+Vu_FKx+GS3YScUrBi@mail.gmail.com>
	<AANLkTinyz1HJ=U-o8qUbWjt=Vpo5KK-k5Vsut6X5rt1X@mail.gmail.com>
	<AANLkTinT_4eNYzHkAuqj1suanZV+WWtUGBuWhK0uR8k5@mail.gmail.com>
	<AANLkTiknHyjytyLtsHWvGVLKEj50_6Tp7zvynEB6KY7S@mail.gmail.com>
	<AANLkTi=HgRVrAHJtX-hYuuQxBVOsSrHmOs6hAh+8sXFA@mail.gmail.com>
	<AANLkTimBtv4mqe0vv1vrLKH=3LZ2ZFhNYGn-ivzzwDy+@mail.gmail.com>
	<AANLkTim4PHZWdbtFL4K599emJ_ya+jvoVmhSfmxvqFr8@mail.gmail.com>
Message-ID: <AANLkTinXqy++qefP20YHfSYohL2jot1cOc-fem5Gs2gh@mail.gmail.com>

On Fri, Dec 10, 2010 at 5:30 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/12/10 Dan Stromberg <drsalists at gmail.com>:
>> On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2010/12/10 Dan Stromberg <drsalists at gmail.com>:
>>>> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo <arigo at tunes.org> wrote:
>>>>> Hi,
>>>>>
>>>>> Sorry for the series of mails. ?I just wanted to add that after
>>>>> looking more at the code, it's fine if you implement major() and
>>>>> minor() the same way as the other posix functions (see e.g. r79941
>>>>> adding os.fchdir()), calling the major() and minor() "functions" in C.
>>>>> ?If you add tests in the same places I did in r79941, then it should
>>>>> work even if the functions are actually macros. ?One place where it
>>>>> would not work is in translated code running on the llinterp; the
>>>>> other place might be the JIT trying to get the address of the
>>>>> "function". ?(If that's the case, we should fix that a bit more
>>>>> generally.)
>>>>>
>>>>>
>>>>> A bient?t,
>>>>>
>>>>> Armin.
>>>>
>>>> I attempted to add os.major and os.minor using fchdir as an example,
>>>> but... ?I have to admit: I really don't know what to make of this
>>>> error. ?Comments?
>>>
>>> You probably failed to add an ll implementation in
>>> pypy/rpython/lltypesystem/module/ll_os.py
>>
>> Do you mean pypy/rpython/module/ll_os.py ? ?I have no
>> pypy/rpython/lltypesystem/module/ll_os.py .
>
> Yes, sorry. It's easy to get those rpython/ subdirs mixed up . :)

No problem.


From arigo at tunes.org  Sat Dec 11 12:05:13 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Dec 2010 12:05:13 +0100
Subject: [pypy-dev] Trunk failing to build?
In-Reply-To: <AANLkTi=QC14sXHUNyrOyxgS87d6=ozAtkagyR=2WQ2Pf@mail.gmail.com>
References: <AANLkTikWiW=MPq7HSjuERBxGRb0muDKMvx73TcF-fPTP@mail.gmail.com>
	<AANLkTi=QC14sXHUNyrOyxgS87d6=ozAtkagyR=2WQ2Pf@mail.gmail.com>
Message-ID: <AANLkTiku7K9dJMAKt4ZyCGTH1tm=7xGtPSNaBcrhEGpq@mail.gmail.com>

Hi,

On Sat, Dec 11, 2010 at 5:46 AM, Benjamin Peterson <benjamin at python.org> wrote:
> Yep. Hopefully fixed in r79965.

Sorry, my fault.


Armin


From drsalists at gmail.com  Sat Dec 11 17:40:42 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 11 Dec 2010 08:40:42 -0800
Subject: [pypy-dev] gdbm and dbm merge - ctypes bug?
In-Reply-To: <AANLkTikBQKP8gGq-hBZP+xSDD61aDwqfgcO9dcSeNn=p@mail.gmail.com>
References: <AANLkTimpHpTjBcPehZJGNBnxOJv83PKQyaEA7neZxD7J@mail.gmail.com>
	<AANLkTikBQKP8gGq-hBZP+xSDD61aDwqfgcO9dcSeNn=p@mail.gmail.com>
Message-ID: <AANLkTik3+uTPDjeKA8Pt==HZa+Bp0CJwgY+A1sXRN3V1@mail.gmail.com>

On Thu, Dec 9, 2010 at 10:05 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> Hi
> 2010/12/9 Dan Stromberg <drsalists at gmail.com>
>>
>> Is there a known problem with pypy's ctypes?
>
> Yes, several of them have already been fixed in trunk, especially
> about memory chunks being freed too early.
> --
> Amaury Forgeot d'Arc
>

Using a trunk from late in the day yesterday, my gdbm_ctypes module is
still giving gibberish keys sometimes, but only with pypy:

find /usr/share/jpilot -print0 | (export dir=$(pwd)/save-dir-pypy; cd
../..; ./this-pypy ./backshift --save-directory "$dir" --backup)
Using gdbm_ctypes
/home/dstromberg/src/home-svn/backshift/trunk/treap.py:40:
DeprecationWarning: setrecursionlimit deprecated
  sys.setrecursionlimit(min_heap_size)
Creating backup id
1292083099.51-benchbox-Sat_Dec_11_07_58_19_2010-7bada40ed14b282b
Inhaling filenames, readline0.__file__: /home/dstromberg/lib/readline0.py
processing directory /usr/share/jpilot
processing regular file /usr/share/jpilot/CalendarDB-PDat.pdb
  5.88% done, 3.61 files/second, ETA Sat Dec 11 07:58:24 2010,
processing regular file /usr/share/jpilot/Ma?anaDB.pdb
 11.76% done, 5.55 files/second, ETA Sat Dec 11 07:58:22 2010,
processing regular file /usr/share/jpilot/jpilotrc.green
 17.65% done, 6.17 files/second, ETA Sat Dec 11 07:58:22 2010,
processing regular file /usr/share/jpilot/jpilotrc.purple
 23.53% done, 6.63 files/second, ETA Sat Dec 11 07:58:22 2010,
processing regular file /usr/share/jpilot/ContactsDB-PAdd.pdb
 29.41% done, 7.37 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/MemoDB.pdb
 35.29% done, 8.03 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/jpilotrc.blue
 41.18% done, 8.59 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/ExpenseDB.pdb
 47.06% done, 8.74 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/jpilotrc.default
 52.94% done, 8.86 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/AddressDB.pdb
 58.82% done, 8.89 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/ToDoDB.pdb
 64.71% done, 9.23 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/TasksDB-PTod.pdb
 70.59% done, 9.53 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/DatebookDB.pdb
 76.47% done, 9.80 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/Memo32DB.pdb
 82.35% done, 9.99 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/MemosDB-PMem.pdb
 88.24% done, 10.21 files/second, ETA Sat Dec 11 07:58:21 2010,
processing regular file /usr/share/jpilot/jpilotrc.steel
 94.12% done, 10.36 files/second, ETA Sat Dec 11 07:58:21 2010,
! ../../dumpdb $(ls -td save-dir-pypy/files/1* | head
-1)/dir-usr/dir-share/dir-jpilot/files | egrep --binary-files=text --
'->' | awk ' { print $2 }' | tr -d '[A-Za-z0-9\._\-]' | sed '/^$/d' |
egrep --binary-files=text .
`
?
`
??@?
make[2]: *** [go] Error 1


From drsalists at gmail.com  Sat Dec 11 17:41:51 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 11 Dec 2010 08:41:51 -0800
Subject: [pypy-dev] setrecursionlimit deprecation?
Message-ID: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>

BTW, what's going on with the deprecation of sys.setrecursionlimit?
What should I use instead?

/home/dstromberg/src/home-svn/backshift/trunk/treap.py:40:
DeprecationWarning: setrecursionlimit deprecated
  sys.setrecursionlimit(min_heap_size)


From santagada at gmail.com  Sat Dec 11 18:10:08 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 11 Dec 2010 15:10:08 -0200
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
Message-ID: <AANLkTinVkROE5rGx8Zx52msm-5mAtgveQxyvcbPZrrUG@mail.gmail.com>

IIRC nothing, pypy knows how big the stack is (if you are using it
limit stack size and not get a core dump).

On Sat, Dec 11, 2010 at 2:41 PM, Dan Stromberg <drsalists at gmail.com> wrote:
> BTW, what's going on with the deprecation of sys.setrecursionlimit?
> What should I use instead?
>
> /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40:
> DeprecationWarning: setrecursionlimit deprecated
> ?sys.setrecursionlimit(min_heap_size)
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
Leonardo Santagada


From benjamin at python.org  Sat Dec 11 18:10:25 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 11 Dec 2010 11:10:25 -0600
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
Message-ID: <AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>

2010/12/11 Dan Stromberg <drsalists at gmail.com>:
> BTW, what's going on with the deprecation of sys.setrecursionlimit?
> What should I use instead?

You shouldn't use anything. It has no affect on pypy.

FWIW, I disagree with the deprecation. I think should just be a no-op.



-- 
Regards,
Benjamin


From arigo at tunes.org  Sat Dec 11 18:32:26 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Dec 2010 18:32:26 +0100
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
Message-ID: <AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>

Hi Benjamin,

On Sat, Dec 11, 2010 at 6:10 PM, Benjamin Peterson <benjamin at python.org> wrote:
> FWIW, I disagree with the deprecation. I think should just be a no-op.

But then people are going to complain that their app seems to hang up
on pypy, and neither they nor we have any clue what is going on ---
until we figure out that they used sys.setrecursionlimit() to put a
bound on recursion and catch the RuntimeError.  That's at least the
reason for which I suggested that calling sys.setrecursionlimit()
should at least print a warning.  Now I agree that maybe the message
of the warning is not the clearest one.


A bient?t,

Armin.


From anto.cuni at gmail.com  Sat Dec 11 19:16:32 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sat, 11 Dec 2010 19:16:32 +0100
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>	<AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
	<AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>
Message-ID: <4D03C000.6090206@gmail.com>

On 11/12/10 18:32, Armin Rigo wrote:

> But then people are going to complain that their app seems to hang up
> on pypy, and neither they nor we have any clue what is going on ---
> until we figure out that they used sys.setrecursionlimit() to put a
> bound on recursion and catch the RuntimeError.  That's at least the
> reason for which I suggested that calling sys.setrecursionlimit()
> should at least print a warning.  Now I agree that maybe the message
> of the warning is not the clearest one.

I was about to propose to change the message into something like
"sys.setrecursionlimit is ignored by PyPy".  I think this is already an
improvement over the current message, but has the drawback than then people
will complain that pypy might run out of stack (which is false, but not
apparent by the warning message).

Also, we should maybe change the class of the Warning. IIRC,
DeprecationWarning is going to be ignored by default on python2.7, which means
that most people won't see the warning at all once we merge fast-forward.

ciao,
Anto


From alex.gaynor at gmail.com  Sat Dec 11 19:46:35 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 11 Dec 2010 13:46:35 -0500
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <4D03C000.6090206@gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
	<AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>
	<4D03C000.6090206@gmail.com>
Message-ID: <AANLkTi=xhBZnJumQypjaP=68VQ7-dmKew_dpWOyF7AfJ@mail.gmail.com>

On Sat, Dec 11, 2010 at 1:16 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> On 11/12/10 18:32, Armin Rigo wrote:
>
> > But then people are going to complain that their app seems to hang up
> > on pypy, and neither they nor we have any clue what is going on ---
> > until we figure out that they used sys.setrecursionlimit() to put a
> > bound on recursion and catch the RuntimeError.  That's at least the
> > reason for which I suggested that calling sys.setrecursionlimit()
> > should at least print a warning.  Now I agree that maybe the message
> > of the warning is not the clearest one.
>
> I was about to propose to change the message into something like
> "sys.setrecursionlimit is ignored by PyPy".  I think this is already an
> improvement over the current message, but has the drawback than then people
> will complain that pypy might run out of stack (which is false, but not
> apparent by the warning message).
>
> Also, we should maybe change the class of the Warning. IIRC,
> DeprecationWarning is going to be ignored by default on python2.7, which
> means
> that most people won't see the warning at all once we merge fast-forward.
>
> ciao,
> Anto
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


Indeed, perhaps RuntimeWarning is more approrpiate?

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101211/dc31592e/attachment.html>

From drsalists at gmail.com  Sat Dec 11 19:59:59 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 11 Dec 2010 10:59:59 -0800
Subject: [pypy-dev] Trunk failing to build?
In-Reply-To: <AANLkTiku7K9dJMAKt4ZyCGTH1tm=7xGtPSNaBcrhEGpq@mail.gmail.com>
References: <AANLkTikWiW=MPq7HSjuERBxGRb0muDKMvx73TcF-fPTP@mail.gmail.com>
	<AANLkTi=QC14sXHUNyrOyxgS87d6=ozAtkagyR=2WQ2Pf@mail.gmail.com>
	<AANLkTiku7K9dJMAKt4ZyCGTH1tm=7xGtPSNaBcrhEGpq@mail.gmail.com>
Message-ID: <AANLkTin7CvZzPSTv4qdLm4qf99kRAyO4aYZ=fVOMOVOH@mail.gmail.com>

On Sat, Dec 11, 2010 at 3:05 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Sat, Dec 11, 2010 at 5:46 AM, Benjamin Peterson <benjamin at python.org> wrote:
>> Yep. Hopefully fixed in r79965.
>
> Sorry, my fault.
>
>
> Armin
>

Looks fixed.  Thanks.


From drsalists at gmail.com  Sat Dec 11 20:17:17 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 11 Dec 2010 11:17:17 -0800
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTinVkROE5rGx8Zx52msm-5mAtgveQxyvcbPZrrUG@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinVkROE5rGx8Zx52msm-5mAtgveQxyvcbPZrrUG@mail.gmail.com>
Message-ID: <AANLkTinMF7Nnd7VhaL3-dMFnGy_fY9H0mnL8Fv=P_G1m@mail.gmail.com>

I originally wrote the code (
http://stromberg.dnsalias.org/~strombrg/treap/ ) for CPython, where
the recursion depth was too low out of the box.  It sounds like on
pypy, I might actually be decreasing the recursion depth.

There probably should be a "set recursion depth to at least..."
function.  Or I should getrecursiondepth and inspect before
"increasing".

On Sat, Dec 11, 2010 at 9:10 AM, Leonardo Santagada <santagada at gmail.com> wrote:
> IIRC nothing, pypy knows how big the stack is (if you are using it
> limit stack size and not get a core dump).
>
> On Sat, Dec 11, 2010 at 2:41 PM, Dan Stromberg <drsalists at gmail.com> wrote:
>> BTW, what's going on with the deprecation of sys.setrecursionlimit?
>> What should I use instead?
>>
>> /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40:
>> DeprecationWarning: setrecursionlimit deprecated
>> ?sys.setrecursionlimit(min_heap_size)
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> Leonardo Santagada
>


From p.giarrusso at gmail.com  Sat Dec 11 21:49:29 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 11 Dec 2010 21:49:29 +0100
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <4D03C000.6090206@gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
	<AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>
	<4D03C000.6090206@gmail.com>
Message-ID: <AANLkTim9V-oOkd_4vftM5hLDqRHfSrfokHqLzU8voySn@mail.gmail.com>

On Sat, Dec 11, 2010 at 19:16, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 11/12/10 18:32, Armin Rigo wrote:
>
>> But then people are going to complain that their app seems to hang up
>> on pypy, and neither they nor we have any clue what is going on ---
>> until we figure out that they used sys.setrecursionlimit() to put a
>> bound on recursion and catch the RuntimeError. ?That's at least the
>> reason for which I suggested that calling sys.setrecursionlimit()
>> should at least print a warning. ?Now I agree that maybe the message
>> of the warning is not the clearest one.

> I was about to propose to change the message into something like
> "sys.setrecursionlimit is ignored by PyPy". ?I think this is already an
> improvement over the current message, but has the drawback than then people
> will complain that pypy might run out of stack (which is false, but not
> apparent by the warning message).

Do you mean that the stack size on Python is not limited (which has
its problems for infinite-recursion bugs), or just that stack overflow
is handled gracefully by throwing an exception?

In the first case, I'd propose "stack size on PyPy cannot be limited"
(and maybe additionally "sys.setrecursionlimit is ignored by PyPy").
In the second case, I'd propose "recursion depth is only limited by
stack size". Maybe the call should try adjusting the stack size?

However, reading os_thread.py:stack_size() [1] suggests that the stack
size _is_ limited for a thread, so I guess you're in the second case.

[1] codespeak.net/pypy/dist/pypy/module/thread/os_thread.py

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From benjamin at python.org  Sat Dec 11 21:52:11 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 11 Dec 2010 14:52:11 -0600
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTim9V-oOkd_4vftM5hLDqRHfSrfokHqLzU8voySn@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinfKeDgg4fssSa5yn9CJ4Qn-9F2iB6FEmAn8-C_@mail.gmail.com>
	<AANLkTi=zJMKQx=SBRVq4ixj4VnAZFYypMO6BV8jkRwy+@mail.gmail.com>
	<4D03C000.6090206@gmail.com>
	<AANLkTim9V-oOkd_4vftM5hLDqRHfSrfokHqLzU8voySn@mail.gmail.com>
Message-ID: <AANLkTinckjnsuf76uKdtQAqp7VyPT3Nwu3-y5JwK6eNR@mail.gmail.com>

2010/12/11 Paolo Giarrusso <p.giarrusso at gmail.com>:
> On Sat, Dec 11, 2010 at 19:16, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> On 11/12/10 18:32, Armin Rigo wrote:
>>
>>> But then people are going to complain that their app seems to hang up
>>> on pypy, and neither they nor we have any clue what is going on ---
>>> until we figure out that they used sys.setrecursionlimit() to put a
>>> bound on recursion and catch the RuntimeError. ?That's at least the
>>> reason for which I suggested that calling sys.setrecursionlimit()
>>> should at least print a warning. ?Now I agree that maybe the message
>>> of the warning is not the clearest one.
>
>> I was about to propose to change the message into something like
>> "sys.setrecursionlimit is ignored by PyPy". ?I think this is already an
>> improvement over the current message, but has the drawback than then people
>> will complain that pypy might run out of stack (which is false, but not
>> apparent by the warning message).
>
> Do you mean that the stack size on Python is not limited (which has
> its problems for infinite-recursion bugs), or just that stack overflow
> is handled gracefully by throwing an exception?

The latter.


-- 
Regards,
Benjamin


From arigo at tunes.org  Sun Dec 12 15:14:05 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 12 Dec 2010 15:14:05 +0100
Subject: [pypy-dev] setrecursionlimit deprecation?
In-Reply-To: <AANLkTinMF7Nnd7VhaL3-dMFnGy_fY9H0mnL8Fv=P_G1m@mail.gmail.com>
References: <AANLkTimDo8k=z2DX7C4_C3VmxJvyX8H-f-JEs8G0Yi5P@mail.gmail.com>
	<AANLkTinVkROE5rGx8Zx52msm-5mAtgveQxyvcbPZrrUG@mail.gmail.com>
	<AANLkTinMF7Nnd7VhaL3-dMFnGy_fY9H0mnL8Fv=P_G1m@mail.gmail.com>
Message-ID: <AANLkTimAumeNtX2uABHX1=1CKZZutZ52sqJL9o8826oQ@mail.gmail.com>

Hi Dan,

On Sat, Dec 11, 2010 at 8:17 PM, Dan Stromberg <drsalists at gmail.com> wrote:
> I originally wrote the code (
> http://stromberg.dnsalias.org/~strombrg/treap/ ) for CPython, where
> the recursion depth was too low out of the box. ?It sounds like on
> pypy, I might actually be decreasing the recursion depth.
>
> There probably should be a "set recursion depth to at least..."
> function. ?Or I should getrecursiondepth and inspect before
> "increasing".

PyPy throws a RuntimeError anyway when the stack is getting too big,
i.e. when it uses more than N kilobytes.  On the other hand,
setrecursiondepth() on CPython (and previously on PyPy) is based on
counting the recursion depth, which is a different measure which only
approximates the real stack size.  We decided that it doesn't make
sense to provide both on PyPy, so we killed the one that is an
approximation (and makes the JIT a little bit slower).  That's all.


A bient?t,

Armin.


From tobami at googlemail.com  Mon Dec 13 09:20:14 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 13 Dec 2010 09:20:14 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
Message-ID: <AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>

Thanks all for the input.
I've compiled a list based on your mails, the Unladen benchmarks page
(http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
alioth descriptions. Here is an extract of the current speed.pypy.org
admin:

ai	
chaos	Creates chaosgame-like fractals
crypto_pyaes	A pure python implementation of AES
django		Uses the Django template system to build a 150x150-cell HTML table

fannkuch		Indexed-access to tiny integer-sequence. The fannkuch
benchmark is defined by programs in Performing Lisp Analysis of the
FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.

float		Creates an array of points using circular projection and then
normalizes and maximizes them. Floating-point heavy.
go		A go (chess like game) computer player AI.
html5lib	Parses the HTML 5 spec using html5lib
meteor-contest	Searchs for solutions to shape packing puzzle.	
nbody_modified		Double-precision N-body simulation. It models the
orbits of Jovian planets, using a simple symplectic-integrator.
pyflate-fast		Stand-alone pure-Python DEFLATE (gzip) and bzip2
decoder/decompressor.
raytrace-simple	A raytracer renderer
richards		Medium-sized language benchmark that simulates the task
dispatcher in the kernel of an operating system.
rietveld	A Django application benchmark.
slowspitfire
spambayes	Runs a canned mailbox through a SpamBayes ham/spam classifier
spectral-norm	
spitfire	Uses the Spitfire template system to build a 1000x1000-cell HTML table.
spitfire_cstringio	Uses the Spitfire template system to build a
1000x1000-cell HTML table, using the cstringio module.
telco
twisted_iteration
twisted_names
twisted_pb
twisted_tcp	Connects one Twised client to one Twisted server over TCP
(on the loopback interface) and then writes bytes as fast as it can.
waf	Python-based framework for configuring, compiling and installing
applications. It derives from the concepts of other build tools such
as Scons, Autotools, CMake or Ant.


So the remaining descriptions are
ai
slowspitfire (what is the exact difference between the three spitfire benches?)
spectral-norm
telco
twisted (most of them)

Are the descriptions all right so far?. They can be made much longer
if you deem it desirable.

on speed.pypy.org you will currently see the descriptions in 3 places:
- Changes view: A tooltip on hover over each benchmark
- Timeline: a description box beneath each plot
- Comparison: A tooltip over each benchmark when hovering the
selection menu on the left side.

Any suggestions on how to improve it further are welcome ;-)

Miquel


2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com> wrote:
>> Here is a incomplete draft list:
>
>> [slow]spitfire[cstringio]: Spitfire is a template language, the
>> cstringio version uses a modified engine (that uses cstringio)
>
>> spambayes: Spambayes is a bayesian spam filter
>
> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
> remember when, because of this, spambayes was slower (including
> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
> and the head (for slowspitfire it's the opposite).
>
> For the rest, I see no significant case of slowdown of PyPy over time.
> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
>


From tobami at googlemail.com  Mon Dec 13 09:31:04 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 13 Dec 2010 09:31:04 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
Message-ID: <AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>

Oh, btw., the "normalized" stacked bars now display a warning note
about its correctness, and how it must be viewed as giving results a
weighting instead of them being normalized. It even includes a link to
the proper paper. I hope that is enough for the strict statisticians
among us ;-)

See:
http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars

PS: there is a bug in the jqPlot plotting library when null values are
present. Trying to display PyPy 1.3 results for the newer go, pyflake
or  raytrace will create some nasty js loops. It also has problems
with autoscaling the axis sometimes.



2010/12/13 Miquel Torres <tobami at googlemail.com>:
> Thanks all for the input.
> I've compiled a list based on your mails, the Unladen benchmarks page
> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
> alioth descriptions. Here is an extract of the current speed.pypy.org
> admin:
>
> ai
> chaos ? Creates chaosgame-like fractals
> crypto_pyaes ? ?A pure python implementation of AES
> django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table
>
> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch
> benchmark is defined by programs in Performing Lisp Analysis of the
> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.
>
> float ? ? ? ? ? Creates an array of points using circular projection and then
> normalizes and maximizes them. Floating-point heavy.
> go ? ? ? ? ? ? ?A go (chess like game) computer player AI.
> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib
> meteor-contest ?Searchs for solutions to shape packing puzzle.
> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the
> orbits of Jovian planets, using a simple symplectic-integrator.
> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2
> decoder/decompressor.
> raytrace-simple A raytracer renderer
> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task
> dispatcher in the kernel of an operating system.
> rietveld ? ? ? ?A Django application benchmark.
> slowspitfire
> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier
> spectral-norm
> spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table.
> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a
> 1000x1000-cell HTML table, using the cstringio module.
> telco
> twisted_iteration
> twisted_names
> twisted_pb
> twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP
> (on the loopback interface) and then writes bytes as fast as it can.
> waf ? ? Python-based framework for configuring, compiling and installing
> applications. It derives from the concepts of other build tools such
> as Scons, Autotools, CMake or Ant.
>
>
> So the remaining descriptions are
> ai
> slowspitfire (what is the exact difference between the three spitfire benches?)
> spectral-norm
> telco
> twisted (most of them)
>
> Are the descriptions all right so far?. They can be made much longer
> if you deem it desirable.
>
> on speed.pypy.org you will currently see the descriptions in 3 places:
> - Changes view: A tooltip on hover over each benchmark
> - Timeline: a description box beneath each plot
> - Comparison: A tooltip over each benchmark when hovering the
> selection menu on the left side.
>
> Any suggestions on how to improve it further are welcome ;-)
>
> Miquel
>
>
> 2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com> wrote:
>>> Here is a incomplete draft list:
>>
>>> [slow]spitfire[cstringio]: Spitfire is a template language, the
>>> cstringio version uses a modified engine (that uses cstringio)
>>
>>> spambayes: Spambayes is a bayesian spam filter
>>
>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
>> remember when, because of this, spambayes was slower (including
>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
>> and the head (for slowspitfire it's the opposite).
>>
>> For the rest, I see no significant case of slowdown of PyPy over time.
>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
>> --
>> Paolo Giarrusso - Ph.D. Student
>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>
>


From cfbolz at gmx.de  Mon Dec 13 10:23:07 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 13 Dec 2010 10:23:07 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>	<240465.94379.qm@web111314.mail.gq1.yahoo.com>	<tobami@googlemail.com>	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>	<201012041105.oB4B5uo1019532@theraft.openend.se>	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
Message-ID: <4D05E5FB.80103@gmx.de>

Hi Miquel,

On 12/13/2010 09:20 AM, Miquel Torres wrote:
> So the remaining descriptions are
> ai
> slowspitfire (what is the exact difference between the three spitfire benches?)

The difference is the way the final string is built, using cstringio, or 
other means. I guess Maciek knows the details.

> spectral-norm
> telco

Telco covers "the essence of a telephone company billing application" 
using decimal arithmetic: http://speleotrove.com/decimal/telco.html

Thanks for doing this,

Carl Friedrich


From exarkun at twistedmatrix.com  Mon Dec 13 15:51:39 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Mon, 13 Dec 2010 14:51:39 -0000
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
Message-ID: <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>

On 08:20 am, tobami at googlemail.com wrote:
>Thanks all for the input.
>I've compiled a list based on your mails, the Unladen benchmarks page
>(http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>alioth descriptions. Here is an extract of the current speed.pypy.org
>admin:
>
>[snip]
>twisted_iteration

Iterates a Twisted reactor as quickly as possible without doing any 
work.
>twisted_names

Runs a DNS server with Twisted Names and then issues requests to it over 
loopback UDP.
>twisted_pb

Runs a Perspective Broker server with a no-op method and invokes that 
method over loopback TCP with some strings, dictionaries, and tuples as 
arguments.


From tobami at googlemail.com  Mon Dec 13 16:03:24 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 13 Dec 2010 16:03:24 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
Message-ID: <AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>

@Carl Friedrich & exarkun: thanks, I've added those.

only spectral-norm, slowspitfire and ai to go.

slowspitfire is described at the Unladen page as using psyco, but it
doesn't make sense in our case?



2010/12/13  <exarkun at twistedmatrix.com>:
> On 08:20 am, tobami at googlemail.com wrote:
>>
>> Thanks all for the input.
>> I've compiled a list based on your mails, the Unladen benchmarks page
>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>> alioth descriptions. Here is an extract of the current speed.pypy.org
>> admin:
>>
>> [snip]
>> twisted_iteration
>
> Iterates a Twisted reactor as quickly as possible without doing any work.
>>
>> twisted_names
>
> Runs a DNS server with Twisted Names and then issues requests to it over
> loopback UDP.
>>
>> twisted_pb
>
> Runs a Perspective Broker server with a no-op method and invokes that method
> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>


From tobami at googlemail.com  Mon Dec 13 16:08:42 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 13 Dec 2010 16:08:42 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
Message-ID: <AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>

sorry, I meant the opposite. To recap, according to
http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
spitfire: psyco
slowspitfire: pure python

in addition we have spitfire_cstringio, which uses a c module (so it
is even faster).

what is vanilla spitfire in our case?


2010/12/13 Miquel Torres <tobami at googlemail.com>:
> @Carl Friedrich & exarkun: thanks, I've added those.
>
> only spectral-norm, slowspitfire and ai to go.
>
> slowspitfire is described at the Unladen page as using psyco, but it
> doesn't make sense in our case?
>
>
>
> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>> On 08:20 am, tobami at googlemail.com wrote:
>>>
>>> Thanks all for the input.
>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>> admin:
>>>
>>> [snip]
>>> twisted_iteration
>>
>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>
>>> twisted_names
>>
>> Runs a DNS server with Twisted Names and then issues requests to it over
>> loopback UDP.
>>>
>>> twisted_pb
>>
>> Runs a Perspective Broker server with a no-op method and invokes that method
>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>
>


From fijall at gmail.com  Mon Dec 13 17:43:27 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 13 Dec 2010 18:43:27 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
	<AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
Message-ID: <AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>

Hi.

spitfires are confusing.

slowspitfire and spitfire use ''.join(list-of-strings) where
spitfire_cstringio uses cStringIO instead.

spitfire and spitfire_cstringio use smaller table to render (100x100 I
think) which was the default on original benchmarks

slowspitfire uses 1000x1000 (which is why it used to be slower than
spitfire) and was chosen by US guys to let the JIT warm up. We should
remove _slow these days.

On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres <tobami at googlemail.com> wrote:
> sorry, I meant the opposite. To recap, according to
> http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
> spitfire: psyco
> slowspitfire: pure python
>
> in addition we have spitfire_cstringio, which uses a c module (so it
> is even faster).
>
> what is vanilla spitfire in our case?
>
>
> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>> @Carl Friedrich & exarkun: thanks, I've added those.
>>
>> only spectral-norm, slowspitfire and ai to go.
>>
>> slowspitfire is described at the Unladen page as using psyco, but it
>> doesn't make sense in our case?
>>
>>
>>
>> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>>> On 08:20 am, tobami at googlemail.com wrote:
>>>>
>>>> Thanks all for the input.
>>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>>> admin:
>>>>
>>>> [snip]
>>>> twisted_iteration
>>>
>>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>>
>>>> twisted_names
>>>
>>> Runs a DNS server with Twisted Names and then issues requests to it over
>>> loopback UDP.
>>>>
>>>> twisted_pb
>>>
>>> Runs a Perspective Broker server with a no-op method and invokes that method
>>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>>
>>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From santagada at gmail.com  Mon Dec 13 19:10:45 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Mon, 13 Dec 2010 16:10:45 -0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
	<AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
	<AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>
Message-ID: <AANLkTimfgnanYRZt9oP=d3c5dZS-WP=KRGZxaD+ty6gh@mail.gmail.com>

why not have only 2 versions, both with the same size table and name
one spitfire_cstringio and the other spitfire_strjoin? I think it
would make things clearer.


On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hi.
>
> spitfires are confusing.
>
> slowspitfire and spitfire use ''.join(list-of-strings) where
> spitfire_cstringio uses cStringIO instead.
>
> spitfire and spitfire_cstringio use smaller table to render (100x100 I
> think) which was the default on original benchmarks
>
> slowspitfire uses 1000x1000 (which is why it used to be slower than
> spitfire) and was chosen by US guys to let the JIT warm up. We should
> remove _slow these days.
>
> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres <tobami at googlemail.com> wrote:
>> sorry, I meant the opposite. To recap, according to
>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
>> spitfire: psyco
>> slowspitfire: pure python
>>
>> in addition we have spitfire_cstringio, which uses a c module (so it
>> is even faster).
>>
>> what is vanilla spitfire in our case?
>>
>>
>> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>>> @Carl Friedrich & exarkun: thanks, I've added those.
>>>
>>> only spectral-norm, slowspitfire and ai to go.
>>>
>>> slowspitfire is described at the Unladen page as using psyco, but it
>>> doesn't make sense in our case?
>>>
>>>
>>>
>>> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>>>> On 08:20 am, tobami at googlemail.com wrote:
>>>>>
>>>>> Thanks all for the input.
>>>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>>>> admin:
>>>>>
>>>>> [snip]
>>>>> twisted_iteration
>>>>
>>>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>>>
>>>>> twisted_names
>>>>
>>>> Runs a DNS server with Twisted Names and then issues requests to it over
>>>> loopback UDP.
>>>>>
>>>>> twisted_pb
>>>>
>>>> Runs a Perspective Broker server with a no-op method and invokes that method
>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>>>
>>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



-- 
Leonardo Santagada


From sasmekoll at gmail.com  Mon Dec 13 03:37:19 2010
From: sasmekoll at gmail.com (Vovan)
Date: Mon, 13 Dec 2010 05:37:19 +0300
Subject: [pypy-dev] =?utf-8?b?0J3QvtCy0L7QtSDRgdC+0L7QsdGJ0LXQvdC40LU=?=
Message-ID: <4d06522c.4b0fcc0a.5274.1a17@mx.google.com>

http://samec.org.ua/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101213/fef1f181/attachment.html>

From tobami at googlemail.com  Mon Dec 13 21:51:59 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 13 Dec 2010 21:51:59 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTimfgnanYRZt9oP=d3c5dZS-WP=KRGZxaD+ty6gh@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
	<AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
	<AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>
	<AANLkTimfgnanYRZt9oP=d3c5dZS-WP=KRGZxaD+ty6gh@mail.gmail.com>
Message-ID: <AANLkTimOke_ZXe+V2UjfuDCw0H3Qk7dDwkwfV8BVJC5z@mail.gmail.com>

@Maciej: it doesn't make a lot of sense. Looking at this graph:
http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars

slowspitfire is much faster than the other two. Is that because it
performs more iterations?

Also, how come pypy-c-jit is faster than cpython or psyco precisely in
cstringio, where performance should be dependent on cstringIO and thus
be more similar across interpreters?


2010/12/13 Leonardo Santagada <santagada at gmail.com>:
> why not have only 2 versions, both with the same size table and name
> one spitfire_cstringio and the other spitfire_strjoin? I think it
> would make things clearer.
>
>
> On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Hi.
>>
>> spitfires are confusing.
>>
>> slowspitfire and spitfire use ''.join(list-of-strings) where
>> spitfire_cstringio uses cStringIO instead.
>>
>> spitfire and spitfire_cstringio use smaller table to render (100x100 I
>> think) which was the default on original benchmarks
>>
>> slowspitfire uses 1000x1000 (which is why it used to be slower than
>> spitfire) and was chosen by US guys to let the JIT warm up. We should
>> remove _slow these days.
>>
>> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres <tobami at googlemail.com> wrote:
>>> sorry, I meant the opposite. To recap, according to
>>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
>>> spitfire: psyco
>>> slowspitfire: pure python
>>>
>>> in addition we have spitfire_cstringio, which uses a c module (so it
>>> is even faster).
>>>
>>> what is vanilla spitfire in our case?
>>>
>>>
>>> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>>>> @Carl Friedrich & exarkun: thanks, I've added those.
>>>>
>>>> only spectral-norm, slowspitfire and ai to go.
>>>>
>>>> slowspitfire is described at the Unladen page as using psyco, but it
>>>> doesn't make sense in our case?
>>>>
>>>>
>>>>
>>>> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>>>>> On 08:20 am, tobami at googlemail.com wrote:
>>>>>>
>>>>>> Thanks all for the input.
>>>>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>>>>> admin:
>>>>>>
>>>>>> [snip]
>>>>>> twisted_iteration
>>>>>
>>>>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>>>>
>>>>>> twisted_names
>>>>>
>>>>> Runs a DNS server with Twisted Names and then issues requests to it over
>>>>> loopback UDP.
>>>>>>
>>>>>> twisted_pb
>>>>>
>>>>> Runs a Perspective Broker server with a no-op method and invokes that method
>>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>>>>
>>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
> --
> Leonardo Santagada
>


From anto.cuni at gmail.com  Tue Dec 14 00:02:57 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 14 Dec 2010 00:02:57 +0100
Subject: [pypy-dev] Migration to mercurial
Message-ID: <4D06A621.4060300@gmail.com>

Hi all,

finally, it's happening :-).
Thanks to Ronny's work, we are going to complete the migration to mercurial
very soon.

The pypy buildbot is already configured to pull from the bitbucket repository:
http://bitbucket.org/pypy/pypy

I have tried to run a test build, and it seems to work ok. We will see
tomorrow morning how the nightly runs went.

If everything goes right, tomorrow we will declare the hg repo as the
"official" one: the svn repo will be made read-only (as soon as armin does it
:-)), the mercurial one will be synchronized to include the latest svn
commits, and from that point everyone should start to commit to mercurial.  If
you don't have write access to the bitbucket repo, please ask me or anyone
else who is an admin.

A note for people working on active branches: before you can work on those,
you need to perform two simple manual steps, as detailed in this howto:
http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt

ciao,
Anto


From benjamin at python.org  Tue Dec 14 01:57:26 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 13 Dec 2010 18:57:26 -0600
Subject: [pypy-dev] Migration to mercurial
In-Reply-To: <4D06A621.4060300@gmail.com>
References: <4D06A621.4060300@gmail.com>
Message-ID: <AANLkTi=40kjEAncrZt9-LMBBz0FM_Ha3pfEnV7A1dSX6@mail.gmail.com>

2010/12/13 Antonio Cuni <anto.cuni at gmail.com>:
> Hi all,
>
> finally, it's happening :-).
> Thanks to Ronny's work, we are going to complete the migration to mercurial
> very soon.

Excellent! Thanks for your work.

>
> The pypy buildbot is already configured to pull from the bitbucket repository:
> http://bitbucket.org/pypy/pypy
>
> I have tried to run a test build, and it seems to work ok. We will see
> tomorrow morning how the nightly runs went.
>
> If everything goes right, tomorrow we will declare the hg repo as the
> "official" one: the svn repo will be made read-only (as soon as armin does it
> :-)), the mercurial one will be synchronized to include the latest svn
> commits, and from that point everyone should start to commit to mercurial. ?If
> you don't have write access to the bitbucket repo, please ask me or anyone
> else who is an admin.
>
> A note for people working on active branches: before you can work on those,
> you need to perform two simple manual steps, as detailed in this howto:
> http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt

What will be happening to the other repos? ie. extradoc


-- 
Regards,
Benjamin


From dmalcolm at redhat.com  Tue Dec 14 01:57:04 2010
From: dmalcolm at redhat.com (David Malcolm)
Date: Mon, 13 Dec 2010 19:57:04 -0500
Subject: [pypy-dev] [PATCH] Improving readability of generated .c code
Message-ID: <1292288224.24137.22.camel@radiator.bos.redhat.com>

The attached patch is an attempt at extending the C code generator so
that it annotates the generated .c code with file/line information for
the corresponding Python sources.

Method:
I followed the instructions in pypy/doc/getting-started-dev.txt:
    cd pypy
    python bin/translatorshell.py
    >>> t = Translation(snippet.is_perfect_number)
    >>> t.annotate([int])
    >>> f = t.compile_c()
then examine the generated .c code.

With this patch, the top of a generated .c function contains an inline
copy of the python source, like this:
bool_t pypy_g_ll_issubclass__object_vtablePtr_object_vtablePtr(struct pypy_object_vtable0 *l_subcls_0, struct pypy_object_vtable0 *l_cls_0) {
	bool_t l_v11; long l_v12; long l_v13; long l_v14;
	/* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rclass.py'
	 *  665 : def ll_issubclass(subcls, cls): 
	 *  666 :     return llop.int_between(Bool, cls.subclassrange_min,
	 *  667 :                                   subcls.subclassrange_min,
	 *  668 :                                   cls.subclassrange_max)
	 */
	goto block0;


...and for every operation that has an offset, it calculates the
linenumber (using the "dis" module on the code object).  For every
operation that changes the line number, it emits a comment containing
that Python source, so that you get code like this:

    block0:
	/*  666 :     return llop.int_between(Bool, cls.subclassrange_min, */
	l_v12 = RPyField(l_cls_0, ov_subclassrange_min);
	/*  667 :                                   subcls.subclassrange_min, */
	l_v13 = RPyField(l_subcls_0, ov_subclassrange_min);
	/*  668 :                                   cls.subclassrange_max) */
	l_v14 = RPyField(l_cls_0, ov_subclassrange_max);
	OP_INT_BETWEEN(l_v12, l_v13, l_v14, l_v11);
	goto block1;

Hopefully this makes the generated .c much more readable.  (I hope to
work on the names of locals also; ideally they ought to embed the python
indentifier)

This is on a Fedora 13 x86_64 box, with cpython 2.6.5

Caveats:
  - this is my first patch to PyPy (please be gentle!): I hope I've
correctly understood the various levels, but I suspect I may still be
missing things at the conceptual level
  - haven't completed the full unit tests yet.
  - the patch attempts to propagate the "offset" of flowspace
SpaceOperation instances in a couple of places where the offset was
being discarded:
      - in BaseInliner.copy_operation
      - when generating operations in rtyper: set each of the new
operations' "offset" to that of the high-level operation
  - I added the offset to SpaceOperation.__repr__() to help track the
above down.
  - I also pass the reference to the func object from the flowspace
FunctionGraph to the rtyper's copy (I wonder if this affect memory usage
during translation?)
  - most functions appear correct, but there are some where I'm not at
all convinced by the output (see below)
  - the patch is assuming that within any given generated C function,
there is a single python function, and that the offsets are indexes into
the bytecode for that function's code object.  I suspect that this is an
oversimplification, and why I'm seeing the errors that I'm seeing.
  - I haven't stripped some of my debugging python comments from
funcgen.py in the patch (this is a work in progress).
  - I haven't tried doing this on a full build of the interpreter
(having highly readable generated .c for this is my objective [1]).

One other idea I had was to sort the blocks in the function by the
bytecode offset; currently the c blocks seem to "jump around" a fair bit
relative to the corresponding rpython code.  Has any tuning been done on
the ordering of the blocks within the generated .c code?  (or is it
assumed that the .c compiler will "flatten" these, for the case when a
node has a single successor node?)

(I wonder if similar work could be done on the JVM and CLI code
generators?)

As mentioned above, I think my patch is getting it wrong in a few
places.  Here's an example:
struct pypy_rpy_string0 *pypy_g_mallocstr__Signed(long l_length_3) {

   ...snip...

	/* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rstr.py'
	 *   35 :     def mallocstr(length): 
	 *   36 :         ll_assert(length >= 0, "negative string length")
	 *   37 :         r = malloc(TP, length)
	 *   38 :         if not we_are_translated() or not malloc_zero_filled: 
	 *   39 :             r.hash = 0
	 *   40 :         return r
	 */
  
   ...snip...

    block15:
	/*   37 :         r = malloc(TP, length) */
	OP_ADR_SUB(l_v202, 0, l_v234);
	l_v235 = (struct pypy_header0 *)l_v234;
	l_v236 = RPyField(l_v235, h_refcount);
	/*   38 :         if not we_are_translated() or not malloc_zero_filled: */
	OP_INT_ADD(l_v236, 1L, l_v237);
	RPyField(l_v235, h_refcount) = l_v237;
	goto block5;

The C code doesn't look anything like the .py code that my patch is
inserting in the above.  (My current guess is that I need to be smarter
about inlined operations, though that's a guess at this stage)



Hope this is helpful
Dave

[1] http://fedoraproject.org/wiki/Features/PyPyStack is what I'm aiming
at
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-add-py-source-as-comments-to-generated-c.patch
Type: text/x-patch
Size: 6498 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101213/77ef01d8/attachment.bin>

From benjamin at python.org  Tue Dec 14 02:14:04 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 13 Dec 2010 19:14:04 -0600
Subject: [pypy-dev] [PATCH] Improving readability of generated .c code
In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com>
References: <1292288224.24137.22.camel@radiator.bos.redhat.com>
Message-ID: <AANLkTimG82CAj_5n+z3WxKBoJOK=Swu565WhnRw5T0u_@mail.gmail.com>

2010/12/13 David Malcolm <dmalcolm at redhat.com>:
> One other idea I had was to sort the blocks in the function by the
> bytecode offset; currently the c blocks seem to "jump around" a fair bit
> relative to the corresponding rpython code. ?Has any tuning been done on
> the ordering of the blocks within the generated .c code? ?(or is it
> assumed that the .c compiler will "flatten" these, for the case when a
> node has a single successor node?)

GCC puts everything in SSA form, so it should eliminate all pointless jumps.



-- 
Regards,
Benjamin


From william.leslie.ttg at gmail.com  Tue Dec 14 04:22:45 2010
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Tue, 14 Dec 2010 14:22:45 +1100
Subject: [pypy-dev] [PATCH] Improving readability of generated .c code
In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com>
References: <1292288224.24137.22.camel@radiator.bos.redhat.com>
Message-ID: <AANLkTim4hzOksQLMZJ798CJJDmTdN2wOm-e5OyWW0O54@mail.gmail.com>

On 14 December 2010 11:57, David Malcolm <dmalcolm at redhat.com> wrote:
> One other idea I had was to sort the blocks in the function by the
> bytecode offset; currently the c blocks seem to "jump around" a fair bit
> relative to the corresponding rpython code. ?Has any tuning been done on
> the ordering of the blocks within the generated .c code? ?(or is it
> assumed that the .c compiler will "flatten" these, for the case when a
> node has a single successor node?)
>
> (I wonder if similar work could be done on the JVM and CLI code
> generators?)

An unfortunate consequence of abstract interpretation is that you
loose a fair bit of the structure of the code as it appears in the
source. There are situations, for example, where you can't know which
blocks (or which *part* of which blocks, even) are part of an inner
and which are part of an outer loop.

The existing code (at least when writing jitcode, but I imagine it's
the same for the backend) writes the startblock, and then immediately
writes its first dependency - essentially a depth-first traversal, a
method as good as any in the absence of structure, but I would be
interested to see what difference it makes for what cost in extra
translation time.

-- 
William Leslie


From fijall at gmail.com  Tue Dec 14 07:32:26 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Dec 2010 08:32:26 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTimOke_ZXe+V2UjfuDCw0H3Qk7dDwkwfV8BVJC5z@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
	<AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
	<AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>
	<AANLkTimfgnanYRZt9oP=d3c5dZS-WP=KRGZxaD+ty6gh@mail.gmail.com>
	<AANLkTimOke_ZXe+V2UjfuDCw0H3Qk7dDwkwfV8BVJC5z@mail.gmail.com>
Message-ID: <AANLkTiksWZ31dVKuDcfxe5iUFh5F36RvLQAAuFQXjEKt@mail.gmail.com>

On Mon, Dec 13, 2010 at 10:51 PM, Miquel Torres <tobami at googlemail.com> wrote:
> @Maciej: it doesn't make a lot of sense. Looking at this graph:
> http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars
>
> slowspitfire is much faster than the other two. Is that because it
> performs more iterations?

I think it's apples to oranges (they have different table sizes and
different number of iterations)

>
> Also, how come pypy-c-jit is faster than cpython or psyco precisely in
> cstringio, where performance should be dependent on cstringIO and thus
> be more similar across interpreters?

because having a list of small strings means you have a large (old)
object referencing a lot of young objects, hence GC cost. It's not the
case with cstringio where you have a single chunk of memory which does
not contain GC pointers.

>
>
> 2010/12/13 Leonardo Santagada <santagada at gmail.com>:
>> why not have only 2 versions, both with the same size table and name
>> one spitfire_cstringio and the other spitfire_strjoin? I think it
>> would make things clearer.
>>
>>
>> On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> Hi.
>>>
>>> spitfires are confusing.
>>>
>>> slowspitfire and spitfire use ''.join(list-of-strings) where
>>> spitfire_cstringio uses cStringIO instead.
>>>
>>> spitfire and spitfire_cstringio use smaller table to render (100x100 I
>>> think) which was the default on original benchmarks
>>>
>>> slowspitfire uses 1000x1000 (which is why it used to be slower than
>>> spitfire) and was chosen by US guys to let the JIT warm up. We should
>>> remove _slow these days.
>>>
>>> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres <tobami at googlemail.com> wrote:
>>>> sorry, I meant the opposite. To recap, according to
>>>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
>>>> spitfire: psyco
>>>> slowspitfire: pure python
>>>>
>>>> in addition we have spitfire_cstringio, which uses a c module (so it
>>>> is even faster).
>>>>
>>>> what is vanilla spitfire in our case?
>>>>
>>>>
>>>> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>>>>> @Carl Friedrich & exarkun: thanks, I've added those.
>>>>>
>>>>> only spectral-norm, slowspitfire and ai to go.
>>>>>
>>>>> slowspitfire is described at the Unladen page as using psyco, but it
>>>>> doesn't make sense in our case?
>>>>>
>>>>>
>>>>>
>>>>> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>>>>>> On 08:20 am, tobami at googlemail.com wrote:
>>>>>>>
>>>>>>> Thanks all for the input.
>>>>>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>>>>>> admin:
>>>>>>>
>>>>>>> [snip]
>>>>>>> twisted_iteration
>>>>>>
>>>>>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>>>>>
>>>>>>> twisted_names
>>>>>>
>>>>>> Runs a DNS server with Twisted Names and then issues requests to it over
>>>>>> loopback UDP.
>>>>>>>
>>>>>>> twisted_pb
>>>>>>
>>>>>> Runs a Perspective Broker server with a no-op method and invokes that method
>>>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> pypy-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>>
>>
>> --
>> Leonardo Santagada
>>
>


From anto.cuni at gmail.com  Tue Dec 14 08:23:41 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 14 Dec 2010 08:23:41 +0100
Subject: [pypy-dev] Migration to mercurial
In-Reply-To: <AANLkTi=40kjEAncrZt9-LMBBz0FM_Ha3pfEnV7A1dSX6@mail.gmail.com>
References: <4D06A621.4060300@gmail.com>
	<AANLkTi=40kjEAncrZt9-LMBBz0FM_Ha3pfEnV7A1dSX6@mail.gmail.com>
Message-ID: <4D071B7D.1030708@gmail.com>

On 14/12/10 01:57, Benjamin Peterson wrote:

> What will be happening to the other repos? ie. extradoc

it's open to discussion, but I suppose that it's fine to migrate them as well.
However, there is no hurry on this, we can do it one by one.

ciao,
Anto


From sungsuha at gmail.com  Tue Dec 14 08:24:23 2010
From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha)
Date: Tue, 14 Dec 2010 07:24:23 +0000 (UTC)
Subject: [pypy-dev] Migration to mercurial
References: <4D06A621.4060300@gmail.com>
Message-ID: <loom.20101214T081925-667@post.gmane.org>

Antonio Cuni <anto.cuni <at> gmail.com> writes:

> 
> Hi all,
> 
> finally, it's happening .
> Thanks to Ronny's work, we are going to complete the migration to mercurial
> very soon.

+1
Cool. 
Providing binaries through "bitbucket.org/pypy/pypy/downloads" would be nice too.

Are the bugtracking/code review facilities of bitbucket going to be used too?
Or is the current https://codespeak.net/issue/pypy-dev/ here to stay?



From fijall at gmail.com  Tue Dec 14 08:29:44 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Dec 2010 09:29:44 +0200
Subject: [pypy-dev] Migration to mercurial
In-Reply-To: <loom.20101214T081925-667@post.gmane.org>
References: <4D06A621.4060300@gmail.com>
	<loom.20101214T081925-667@post.gmane.org>
Message-ID: <AANLkTi=P=dz47DH6v669Bah_99AP1U0tn7dkVnakntrd@mail.gmail.com>

On Tue, Dec 14, 2010 at 9:24 AM, Seung Soo, <sungsuha at gmail.com> wrote:
> Antonio Cuni <anto.cuni <at> gmail.com> writes:
>
>>
>> Hi all,
>>
>> finally, it's happening .
>> Thanks to Ronny's work, we are going to complete the migration to mercurial
>> very soon.
>
> +1
> Cool.
> Providing binaries through "bitbucket.org/pypy/pypy/downloads" would be nice too.
>
> Are the bugtracking/code review facilities of bitbucket going to be used too?
> Or is the current https://codespeak.net/issue/pypy-dev/ here to stay?
>

All of this will happen, it's just a one-by-one process (since they
don't depend on each other). Don't worry, we'll get there :)


From fijall at gmail.com  Tue Dec 14 08:57:47 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Dec 2010 09:57:47 +0200
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTiksWZ31dVKuDcfxe5iUFh5F36RvLQAAuFQXjEKt@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain>
	<AANLkTimhWvRqq-s8hysFUBq21R8ZgFa2BMwfz=v6SeFy@mail.gmail.com>
	<AANLkTikGN4Y7k7AX9H864s6ASg2gH9swvskWvZFj8fAc@mail.gmail.com>
	<AANLkTin_NcDxrjxnPerAPytKXtviDg_aFM4nX_kvGexW@mail.gmail.com>
	<AANLkTimfgnanYRZt9oP=d3c5dZS-WP=KRGZxaD+ty6gh@mail.gmail.com>
	<AANLkTimOke_ZXe+V2UjfuDCw0H3Qk7dDwkwfV8BVJC5z@mail.gmail.com>
	<AANLkTiksWZ31dVKuDcfxe5iUFh5F36RvLQAAuFQXjEKt@mail.gmail.com>
Message-ID: <AANLkTik1p=_DtWuA3rBwCtH1KwVa=0QcB3Ya9jAsPb8r@mail.gmail.com>

Hey miquel, didn't we loose colors somehow?

On Tue, Dec 14, 2010 at 8:32 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Dec 13, 2010 at 10:51 PM, Miquel Torres <tobami at googlemail.com> wrote:
>> @Maciej: it doesn't make a lot of sense. Looking at this graph:
>> http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars
>>
>> slowspitfire is much faster than the other two. Is that because it
>> performs more iterations?
>
> I think it's apples to oranges (they have different table sizes and
> different number of iterations)
>
>>
>> Also, how come pypy-c-jit is faster than cpython or psyco precisely in
>> cstringio, where performance should be dependent on cstringIO and thus
>> be more similar across interpreters?
>
> because having a list of small strings means you have a large (old)
> object referencing a lot of young objects, hence GC cost. It's not the
> case with cstringio where you have a single chunk of memory which does
> not contain GC pointers.
>
>>
>>
>> 2010/12/13 Leonardo Santagada <santagada at gmail.com>:
>>> why not have only 2 versions, both with the same size table and name
>>> one spitfire_cstringio and the other spitfire_strjoin? I think it
>>> would make things clearer.
>>>
>>>
>>> On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>>> Hi.
>>>>
>>>> spitfires are confusing.
>>>>
>>>> slowspitfire and spitfire use ''.join(list-of-strings) where
>>>> spitfire_cstringio uses cStringIO instead.
>>>>
>>>> spitfire and spitfire_cstringio use smaller table to render (100x100 I
>>>> think) which was the default on original benchmarks
>>>>
>>>> slowspitfire uses 1000x1000 (which is why it used to be slower than
>>>> spitfire) and was chosen by US guys to let the JIT warm up. We should
>>>> remove _slow these days.
>>>>
>>>> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres <tobami at googlemail.com> wrote:
>>>>> sorry, I meant the opposite. To recap, according to
>>>>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks,
>>>>> spitfire: psyco
>>>>> slowspitfire: pure python
>>>>>
>>>>> in addition we have spitfire_cstringio, which uses a c module (so it
>>>>> is even faster).
>>>>>
>>>>> what is vanilla spitfire in our case?
>>>>>
>>>>>
>>>>> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>>>>>> @Carl Friedrich & exarkun: thanks, I've added those.
>>>>>>
>>>>>> only spectral-norm, slowspitfire and ai to go.
>>>>>>
>>>>>> slowspitfire is described at the Unladen page as using psyco, but it
>>>>>> doesn't make sense in our case?
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2010/12/13 ?<exarkun at twistedmatrix.com>:
>>>>>>> On 08:20 am, tobami at googlemail.com wrote:
>>>>>>>>
>>>>>>>> Thanks all for the input.
>>>>>>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>>>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>>>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>>>>>>> admin:
>>>>>>>>
>>>>>>>> [snip]
>>>>>>>> twisted_iteration
>>>>>>>
>>>>>>> Iterates a Twisted reactor as quickly as possible without doing any work.
>>>>>>>>
>>>>>>>> twisted_names
>>>>>>>
>>>>>>> Runs a DNS server with Twisted Names and then issues requests to it over
>>>>>>> loopback UDP.
>>>>>>>>
>>>>>>>> twisted_pb
>>>>>>>
>>>>>>> Runs a Perspective Broker server with a no-op method and invokes that method
>>>>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments.
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> pypy-dev at codespeak.net
>>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>>>
>>>> _______________________________________________
>>>> pypy-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>>
>>>
>>> --
>>> Leonardo Santagada
>>>
>>
>


From kevin at movieextras.ie  Tue Dec 14 11:13:46 2010
From: kevin at movieextras.ie (Kevin Gill)
Date: Tue, 14 Dec 2010 10:13:46 +0000
Subject: [pypy-dev] Problem building C Extension
Message-ID: <4D07435A.80608@movieextras.ie>

Hi,

I am new to PyPy.

I built pypy using...

$ python translate.py --opt=jit targetpypystandalone.py

When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build
uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS.

For example:

$ pip -v install zope.interface==3.6.1

shows this output

cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o
src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?:
src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast
cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so
/usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value
collect2: ld returned 1 exit status

I did some digging through the source. I found that the compiler configuration is normally loaded
via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override
using environment variables for example because the customisation function is as follows:

def customize_compiler(options):
     """Dummy method to let some easy_install packages that have
     optional C speedup components.
     """
     pass

I see from the mailing list that people are compiling C extensions. What am I doing wrong?

Thanks

Kevin

------------------------------------------------------------
Version of pypy that I am using.

$ svn info
Path: .
URL: http://codespeak.net/svn/pypy/trunk
Repository Root: http://codespeak.net/svn
Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada
Revision: 79796
Node Kind: directory
Schedule: normal
Last Changed Author: afa
Last Changed Rev: 79794
Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010)







From anto.cuni at gmail.com  Tue Dec 14 11:53:05 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 14 Dec 2010 11:53:05 +0100
Subject: [pypy-dev] Migration to mercurial
In-Reply-To: <4D06A621.4060300@gmail.com>
References: <4D06A621.4060300@gmail.com>
Message-ID: <4D074C91.7020005@gmail.com>

On 14/12/10 00:02, Antonio Cuni wrote:
> Hi all,
> 
> finally, it's happening :-).
> Thanks to Ronny's work, we are going to complete the migration to mercurial
> very soon.

so, the migration is done!
The svn repo is readonly, and from now the official pypy repo is this one:
http://bitbucket.org/pypy/pypy

There are still some rough edges to fix on buildbot, but for the rest
everything seems to work fine.

I would like to thank everyone who was involved in the migration, in
particular Ronny who did most of the dirty work :-)

ciao,
Anto


From fijall at gmail.com  Tue Dec 14 12:49:25 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Dec 2010 13:49:25 +0200
Subject: [pypy-dev] Problem building C Extension
In-Reply-To: <4D07435A.80608@movieextras.ie>
References: <4D07435A.80608@movieextras.ie>
Message-ID: <AANLkTik0Lj4mQq+TWzTGOiE6Lx4W0LZJAks_Ukmyu8GF@mail.gmail.com>

Not to comment on issue, but it's actually pretty good that
zope.interface C speedups don't work. They won't be speedups in pypy
case even if they did work.

On Tue, Dec 14, 2010 at 12:13 PM, Kevin Gill <kevin at movieextras.ie> wrote:
> Hi,
>
> I am new to PyPy.
>
> I built pypy using...
>
> $ python translate.py --opt=jit targetpypystandalone.py
>
> When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build
> uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS.
>
> For example:
>
> $ pip -v install zope.interface==3.6.1
>
> shows this output
>
> cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o
> src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?:
> src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast
> cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so
> /usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
> build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value
> collect2: ld returned 1 exit status
>
> I did some digging through the source. I found that the compiler configuration is normally loaded
> via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override
> using environment variables for example because the customisation function is as follows:
>
> def customize_compiler(options):
> ? ? """Dummy method to let some easy_install packages that have
> ? ? optional C speedup components.
> ? ? """
> ? ? pass
>
> I see from the mailing list that people are compiling C extensions. What am I doing wrong?
>
> Thanks
>
> Kevin
>
> ------------------------------------------------------------
> Version of pypy that I am using.
>
> $ svn info
> Path: .
> URL: http://codespeak.net/svn/pypy/trunk
> Repository Root: http://codespeak.net/svn
> Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada
> Revision: 79796
> Node Kind: directory
> Schedule: normal
> Last Changed Author: afa
> Last Changed Rev: 79794
> Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010)
>
>
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From cfbolz at gmx.de  Tue Dec 14 15:14:13 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 14 Dec 2010 15:14:13 +0100
Subject: [pypy-dev] [PATCH] Improving readability of generated .c code
In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com>
References: <1292288224.24137.22.camel@radiator.bos.redhat.com>
Message-ID: <4D077BB5.7030102@gmx.de>

Hi David,

On 12/14/2010 01:57 AM, David Malcolm wrote:
> The attached patch is an attempt at extending the C code generator so
> that it annotates the generated .c code with file/line information for
> the corresponding Python sources.
>
> Method:
> I followed the instructions in pypy/doc/getting-started-dev.txt:
>      cd pypy
>      python bin/translatorshell.py
>      >>>  t = Translation(snippet.is_perfect_number)
>      >>>  t.annotate([int])
>      >>>  f = t.compile_c()
> then examine the generated .c code.
>
> With this patch, the top of a generated .c function contains an inline
> copy of the python source, like this:
> bool_t pypy_g_ll_issubclass__object_vtablePtr_object_vtablePtr(struct pypy_object_vtable0 *l_subcls_0, struct pypy_object_vtable0 *l_cls_0) {
> 	bool_t l_v11; long l_v12; long l_v13; long l_v14;
> 	/* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rclass.py'
> 	 *  665 : def ll_issubclass(subcls, cls):
> 	 *  666 :     return llop.int_between(Bool, cls.subclassrange_min,
> 	 *  667 :                                   subcls.subclassrange_min,
> 	 *  668 :                                   cls.subclassrange_max)
> 	 */
> 	goto block0;
>
>
> ...and for every operation that has an offset, it calculates the
> linenumber (using the "dis" module on the code object).  For every
> operation that changes the line number, it emits a comment containing
> that Python source, so that you get code like this:
>
>      block0:
> 	/*  666 :     return llop.int_between(Bool, cls.subclassrange_min, */
> 	l_v12 = RPyField(l_cls_0, ov_subclassrange_min);
> 	/*  667 :                                   subcls.subclassrange_min, */
> 	l_v13 = RPyField(l_subcls_0, ov_subclassrange_min);
> 	/*  668 :                                   cls.subclassrange_max) */
> 	l_v14 = RPyField(l_cls_0, ov_subclassrange_max);
> 	OP_INT_BETWEEN(l_v12, l_v13, l_v14, l_v11);
> 	goto block1;
>
> Hopefully this makes the generated .c much more readable.

I like the general approach, though I have the vague fear that it will 
make the generated C code much bigger. Could you check that this is not 
the case?

> (I hope to
> work on the names of locals also; ideally they ought to embed the python
> indentifier)

This should already work in theory, see e.g. "l_cls_0" above, which 
embeds the Python variable name "cls". I guess that the information is 
simply lost in some of the transformation steps. If you feel like 
hunting down where, you can probably get better naming in some cases.

> This is on a Fedora 13 x86_64 box, with cpython 2.6.5
>
> Caveats:
>    - this is my first patch to PyPy (please be gentle!): I hope I've
> correctly understood the various levels, but I suspect I may still be
> missing things at the conceptual level

I think you are doing great. You might have to grep around a bit for 
more places that create SpaceOperations, or did you look at all of them?

>    - haven't completed the full unit tests yet.

This is the biggest problem of the patch, of course. I think that you 
should at least try to write tests for each of the places where you 
propagate the offset down one level. In addition, you probably need an 
integration test where you check the generated C code for the presence 
of the correct comments.

>    - the patch attempts to propagate the "offset" of flowspace
> SpaceOperation instances in a couple of places where the offset was
> being discarded:
>        - in BaseInliner.copy_operation

This is wrong. If you inline one function into another, it doesn't make 
sense to put the offset of the inner function into the outer function. 
To solve this problem, you have two options:
  1) when inlining, set the offset to -1, and just don't get source code 
then
  2) make every *operation* contain a reference to the func it comes 
from. The memory implications of this should be checked, but are 
probably not too bad.

>        - when generating operations in rtyper: set each of the new
> operations' "offset" to that of the high-level operation

That makes sense.

>    - I added the offset to SpaceOperation.__repr__() to help track the
> above down.
>    - I also pass the reference to the func object from the flowspace
> FunctionGraph to the rtyper's copy (I wonder if this affect memory usage
> during translation?)

Probably not enough to care.

>    - most functions appear correct, but there are some where I'm not at
> all convinced by the output (see below)
>    - the patch is assuming that within any given generated C function,
> there is a single python function, and that the offsets are indexes into
> the bytecode for that function's code object.  I suspect that this is an
> oversimplification, and why I'm seeing the errors that I'm seeing.
>    - I haven't stripped some of my debugging python comments from
> funcgen.py in the patch (this is a work in progress).

Now that we switched to mercurial, you can probably simply make a branch 
and work there: http://bitbucket.org/pypy/pypy/overview

>    - I haven't tried doing this on a full build of the interpreter
> (having highly readable generated .c for this is my objective [1]).

I fear it will never be "highly readable", but I think your work 
improves things already.

> One other idea I had was to sort the blocks in the function by the
> bytecode offset; currently the c blocks seem to "jump around" a fair bit
> relative to the corresponding rpython code.  Has any tuning been done on
> the ordering of the blocks within the generated .c code?  (or is it
> assumed that the .c compiler will "flatten" these, for the case when a
> node has a single successor node?)

The C compiler should not care about the order of blocks. If it does, I 
am prepared to call it insane.

> (I wonder if similar work could be done on the JVM and CLI code
> generators?)
>
> As mentioned above, I think my patch is getting it wrong in a few
> places.  Here's an example:
> struct pypy_rpy_string0 *pypy_g_mallocstr__Signed(long l_length_3) {
>
>     ...snip...
>
> 	/* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rstr.py'
> 	 *   35 :     def mallocstr(length):
> 	 *   36 :         ll_assert(length>= 0, "negative string length")
> 	 *   37 :         r = malloc(TP, length)
> 	 *   38 :         if not we_are_translated() or not malloc_zero_filled:
> 	 *   39 :             r.hash = 0
> 	 *   40 :         return r
> 	 */
>
>     ...snip...
>
>      block15:
> 	/*   37 :         r = malloc(TP, length) */
> 	OP_ADR_SUB(l_v202, 0, l_v234);
> 	l_v235 = (struct pypy_header0 *)l_v234;
> 	l_v236 = RPyField(l_v235, h_refcount);
> 	/*   38 :         if not we_are_translated() or not malloc_zero_filled: */
> 	OP_INT_ADD(l_v236, 1L, l_v237);
> 	RPyField(l_v235, h_refcount) = l_v237;
> 	goto block5;
>
> The C code doesn't look anything like the .py code that my patch is
> inserting in the above.  (My current guess is that I need to be smarter
> about inlined operations, though that's a guess at this stage)

Sounds like a correct guess, malloc is transformed into something else, 
and the offset of the inlined operations is basically just random, if 
you interpret it in the context of the outer function.

Cheers,

Carl Friedrich


From kevin at movieextras.ie  Tue Dec 14 16:33:42 2010
From: kevin at movieextras.ie (Kevin Gill)
Date: Tue, 14 Dec 2010 15:33:42 +0000
Subject: [pypy-dev] Problem building C Extension
Message-ID: <1292340822.28537.1410292989@webmail.messagingengine.com>

On Tue, 14 Dec 2010 13:49 +0200, "Maciej Fijalkowski" <fijall at gmail.com>
wrote:
> Not to comment on issue, but it's actually pretty good that
> zope.interface C speedups don't work. They won't be speedups in pypy
> case even if they did work.
> 

I accept that they may not be speedups. However, I cannot be certain
whether C code  in an egg represents C alternative to python
functionality, or whether the functionality is implemented in C. Perhaps
this information is in distutils (setup.py) - I haven't noticed it.

> On Tue, Dec 14, 2010 at 12:13 PM, Kevin Gill <kevin at movieextras.ie>
> wrote:
> > Hi,
> >
> > I am new to PyPy.
> >
> > I built pypy using...
> >
> > $ python translate.py --opt=jit targetpypystandalone.py
> >
> > When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build
> > uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS.
> >
> > For example:
> >
> > $ pip -v install zope.interface==3.6.1
> >
> > shows this output
> >
> > cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o
> > src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?:
> > src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast
> > cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so
> > /usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
> > build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value
> > collect2: ld returned 1 exit status
> >
> > I did some digging through the source. I found that the compiler configuration is normally loaded
> > via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override
> > using environment variables for example because the customisation function is as follows:
> >
> > def customize_compiler(options):
> > ? ? """Dummy method to let some easy_install packages that have
> > ? ? optional C speedup components.
> > ? ? """
> > ? ? pass
> >
> > I see from the mailing list that people are compiling C extensions. What am I doing wrong?
> >
> > Thanks
> >
> > Kevin
> >
> > ------------------------------------------------------------
> > Version of pypy that I am using.
> >
> > $ svn info
> > Path: .
> > URL: http://codespeak.net/svn/pypy/trunk
> > Repository Root: http://codespeak.net/svn
> > Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada
> > Revision: 79796
> > Node Kind: directory
> > Schedule: normal
> > Last Changed Author: afa
> > Last Changed Rev: 79794
> > Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010)
> >
> >
> >
> >
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> 



From jacob at openend.se  Tue Dec 14 18:04:06 2010
From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=)
Date: Tue, 14 Dec 2010 18:04:06 +0100
Subject: [pypy-dev] Mercurial workflow
Message-ID: <201012141804.06449.jacob@openend.se>

Hi folks,

now that the switch to Mercurial has happened, people are discovering that 
their Subversion workflow habits don't quite work. This is because Mercurial 
has a distributed philosophy, inlike svn which has the concept of the holy 
central server where all operations take place.

In Mercurial, you do all the important operations in your local repository. If 
you have a central node (which is not strictly necessary), you just push ready 
made change sets to it. Any conflict resolving has already been done on your 
local machine.

This means that it is very convenient to have at least 2 repositories locally 
(this takes hadly any extra space, since unchanged files are hard linked to be 
the same file in each repository).

You keep one repository as a staging area to which you pull changes from the 
central server (or other peoples repositories) and to which you pull changes 
from your other local repositories. It is here you resolve any merge conflicts 
and it is from here you do pushes of new versions to the central pypy server 
on bitbucket.

For every idea you have, you clone your repository and do the work for the 
idea in the clone. If you need to collaborate with someone else you 
synchronize your repo with their repo without going through the central 
server. Branching is a concept that is separated from the repository clones 
and it is just as good an idea to make branches for new features as it was in 
svn.

When you work on an idea you should keep in mind that it is an extremely fast 
operation to commit your work. You can do it for a number of various reasons - 
whenever you have made a set of changes that you feel belong together, if your 
machine is flakey and you want to have a backup copy somewhere else or if you 
want to continue the work on a different machine. I sometimes program in bed, 
and it is very simple to do a pull from my workstation to my laptop.

The new workflow will make the detailed changes less visible to others, since 
you commit work to the central server in fewer and bigger chunks. This means 
that there will be fewer "oops, I misspelled this" and more of a higher level 
view of what is being developed.

I hope this helps getting your minds around how to work with Mercurial. I hope 
you will discover that the way it is different from svn will improve your work 
environment and your work habits. It has done so for me.

Jacob


From adimasci at gmail.com  Tue Dec 14 18:42:04 2010
From: adimasci at gmail.com (Adrien Di Mascio)
Date: Tue, 14 Dec 2010 18:42:04 +0100
Subject: [pypy-dev] Mercurial workflow
In-Reply-To: <201012141804.06449.jacob@openend.se>
References: <201012141804.06449.jacob@openend.se>
Message-ID: <AANLkTikg=NH+85aX99sT3RUWF5fmRzZKqT9-tVCmL7BD@mail.gmail.com>

Hi Jacob,

On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n <jacob at openend.se> wrote:
> now that the switch to Mercurial has happened, people are discovering that
> their Subversion workflow habits don't quite work. This is because Mercurial
> has a distributed philosophy, inlike svn which has the concept of the holy
> central server where all operations take place.

It might be relevant for you to consider the mq extension :

- http://mercurial.selenic.com/wiki/MqExtension
- http://mercurial.selenic.com/wiki/MqTutorial
- http://mercurial.selenic.com/wiki/MqExtension

I find it very convenient on a daily-basis usage. When you want to
implement or fix something, you just create a new patch :

$ hg qnew my_new_idea

You then start to modify code and update your patch whenever you feel
the current version of your code is better than the previous one :

$ hq qrefresh

Then, when you're happy with it, all tests pass, etc., unapply your
patch, pull the new changesets from the "central" repository, and then
reapply your patch :

$ hg qpop                           # pop your patch
$ hg pull central_repo_url     # pull new changesets
$ hg up
$ hg qpush                         # reapply your patch
$ <make sure tests still pass>
$ hg qfinish -a                     # finalize and promote your patch
to a changeset tracked in the repository
$ hg push central_repo_url   # push your changeset

You can work on several patches at the same time, you can even track
your patches in a mercurial repository. In my experience, it helps
keeping a revision tree clean and avoid quite a few unnecessary
merges. Of course there is also the "rebase" command and some other
extensions I haven't tried such as pbranch but the standard "mq"
extension is itself very useful.

Just my two cents.
Regards,
Adrien.


From fijall at gmail.com  Tue Dec 14 20:01:39 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 14 Dec 2010 21:01:39 +0200
Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output
 on standard error when invoking Mercurial.
In-Reply-To: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com>
References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com>
Message-ID: <AANLkTi=nfOPtFZ=Duufng0bVj84RsYHq_MTSsBQe5=+j@mail.gmail.com>

This commit contains no tests whatsoever, it would be cooler if it did.

On Tue, Dec 14, 2010 at 8:19 PM, Bitbucket
<commits-noreply at bitbucket.org> wrote:
> # HG changeset patch -- Bitbucket.org
> # Project pypy
> # URL http://bitbucket.org/pypy/pypy/overview
> # User Dan Villiom Podlaski Christiansen <danchr at gmail.com>
> # Date 1292349796 -3600
> # Node ID 7b8aa74da1fb06bd5af8b37e086538e4e7d5d0a1
> # Parent ?1f81ed99bfbc17bbb3cc0920ac612218b4194f8c
> Handle output on standard error when invoking Mercurial.
>
> Instead of just piped to the process standard error, we use a logger
> to warn if 'hg' wrote anything on stderr.
>
> --- a/pypy/tool/version.py
> +++ b/pypy/tool/version.py
> @@ -13,24 +13,39 @@ def get_mercurial_info():
> ? ? hgexe = py.path.local.sysfind('hg')
>
> ? ? if hgexe and os.path.isdir(os.path.join(pypyroot, '.hg')):
> + ? ? ? ?def maywarn(err):
> + ? ? ? ? ? ?if not err:
> + ? ? ? ? ? ? ? ?return
> +
> + ? ? ? ? ? ?from pypy.tool.ansi_print import ansi_log
> + ? ? ? ? ? ?log = py.log.Producer("version")
> + ? ? ? ? ? ?py.log.setconsumer("version", ansi_log)
> + ? ? ? ? ? ?log.WARNING('Errors getting Mercurial information:' + err)
> +
> ? ? ? ? env = dict(os.environ)
> ? ? ? ? # get Mercurial into scripting mode
> ? ? ? ? env['HGPLAIN'] = '1'
> ? ? ? ? # disable user configuration, extensions, etc.
> ? ? ? ? env['HGRCPATH'] = os.devnull
>
> - ? ? ? ?p = Popen([str(hgexe), 'id', '-i', pypyroot], stdout=PIPE, env=env)
> + ? ? ? ?p = Popen([str(hgexe), 'id', '-i', pypyroot],
> + ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env)
> ? ? ? ? hgid = p.stdout.read().strip()
> + ? ? ? ?maywarn(p.stderr.read())
>
> - ? ? ? ?p = Popen([str(hgexe), 'id', '-t', pypyroot], stdout=PIPE, env=env)
> + ? ? ? ?p = Popen([str(hgexe), 'id', '-t', pypyroot],
> + ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env)
> ? ? ? ? hgtags = [t for t in p.stdout.read().strip().split() if t != 'tip']
> + ? ? ? ?maywarn(p.stderr.read())
>
> ? ? ? ? if hgtags:
> ? ? ? ? ? ? return 'PyPy', hgtags[0], hgid
> ? ? ? ? else:
> ? ? ? ? ? ? # use the branch instead
> - ? ? ? ? ? ?p = Popen([str(hgexe), 'id', '-b', pypyroot], stdout=PIPE, env=env)
> + ? ? ? ? ? ?p = Popen([str(hgexe), 'id', '-b', pypyroot],
> + ? ? ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env)
> ? ? ? ? ? ? hgbranch = p.stdout.read().strip()
> + ? ? ? ? ? ?maywarn(p.stderr.read())
>
> ? ? ? ? ? ? return 'PyPy', hgbranch, hgid
> ? ? else:
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>


From danchr at gmail.com  Tue Dec 14 20:28:34 2010
From: danchr at gmail.com (Dan Villiom Podlaski Christiansen)
Date: Tue, 14 Dec 2010 20:28:34 +0100
Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output
	on standard error when invoking Mercurial.
In-Reply-To: <AANLkTi=nfOPtFZ=Duufng0bVj84RsYHq_MTSsBQe5=+j@mail.gmail.com>
References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com>
	<AANLkTi=nfOPtFZ=Duufng0bVj84RsYHq_MTSsBQe5=+j@mail.gmail.com>
Message-ID: <32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com>

On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote:

> This commit contains no tests whatsoever, it would be cooler if it  
> did.

Antonio kindly wrote a test for the functionality in an earlier  
changeset. Ironically, I broke it that change, but that's fixed now :)

--

Dan Villiom Podlaski Christiansen
danchr at gmail.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1943 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101214/bb3b33b0/attachment.bin>

From anto.cuni at gmail.com  Tue Dec 14 23:49:32 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 14 Dec 2010 23:49:32 +0100
Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output
 on standard error when invoking Mercurial.
In-Reply-To: <32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com>
References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com>	<AANLkTi=nfOPtFZ=Duufng0bVj84RsYHq_MTSsBQe5=+j@mail.gmail.com>
	<32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com>
Message-ID: <4D07F47C.3060304@gmail.com>

On 14/12/10 20:28, Dan Villiom Podlaski Christiansen wrote:
> On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote:
> 
>> This commit contains no tests whatsoever, it would be cooler if it did.
> 
> Antonio kindly wrote a test for the functionality in an earlier changeset.
> Ironically, I broke it that change, but that's fixed now :)

anyway, maciek on the channel mentioned a way to test it without messing with
the hg repo: you can provide a custom hgexe script which returns whatever you
want.

Or even, restructure the code in a way that the running of the hg command is
decoupled by the parsing of its results, so that you can at least test the latter.

Sorry, I should have pointed this out when you asked me to review the patches,
but today I was fried and had thousands of things to look at :-/

ciao,
Anto


From qbproger at gmail.com  Tue Dec 14 23:53:41 2010
From: qbproger at gmail.com (Joe)
Date: Tue, 14 Dec 2010 17:53:41 -0500
Subject: [pypy-dev] Mercurial workflow
In-Reply-To: <AANLkTikg=NH+85aX99sT3RUWF5fmRzZKqT9-tVCmL7BD@mail.gmail.com>
References: <201012141804.06449.jacob@openend.se>
	<AANLkTikg=NH+85aX99sT3RUWF5fmRzZKqT9-tVCmL7BD@mail.gmail.com>
Message-ID: <AANLkTimKa=-1iOeQ0G4xfaT=R+SVZwio-LxmW0d2v-NT@mail.gmail.com>

In reference to MqExtention, I've used Stacked Git (a git equivalent)
in the past, and would generally advise against this.  I like to
commit after I get a piece of something working, so I know I can go
back to that point if I try something and it doesn't work out.
Stacked Git makes this hard.  In my opinion, you're essentially
by-passing a benefit of a scm tool.

I wouldn't advise people to use MqExtension unless they understand
that if they make a mistake they must undo it manually.

Joe

On Tue, Dec 14, 2010 at 12:42 PM, Adrien Di Mascio <adimasci at gmail.com> wrote:
> Hi Jacob,
>
> On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n <jacob at openend.se> wrote:
>> now that the switch to Mercurial has happened, people are discovering that
>> their Subversion workflow habits don't quite work. This is because Mercurial
>> has a distributed philosophy, inlike svn which has the concept of the holy
>> central server where all operations take place.
>
> It might be relevant for you to consider the mq extension :
>
> - http://mercurial.selenic.com/wiki/MqExtension
> - http://mercurial.selenic.com/wiki/MqTutorial
> - http://mercurial.selenic.com/wiki/MqExtension
>
> I find it very convenient on a daily-basis usage. When you want to
> implement or fix something, you just create a new patch :
>
> $ hg qnew my_new_idea
>
> You then start to modify code and update your patch whenever you feel
> the current version of your code is better than the previous one :
>
> $ hq qrefresh
>
> Then, when you're happy with it, all tests pass, etc., unapply your
> patch, pull the new changesets from the "central" repository, and then
> reapply your patch :
>
> $ hg qpop ? ? ? ? ? ? ? ? ? ? ? ? ? # pop your patch
> $ hg pull central_repo_url ? ? # pull new changesets
> $ hg up
> $ hg qpush ? ? ? ? ? ? ? ? ? ? ? ? # reapply your patch
> $ <make sure tests still pass>
> $ hg qfinish -a ? ? ? ? ? ? ? ? ? ? # finalize and promote your patch
> to a changeset tracked in the repository
> $ hg push central_repo_url ? # push your changeset
>
> You can work on several patches at the same time, you can even track
> your patches in a mercurial repository. In my experience, it helps
> keeping a revision tree clean and avoid quite a few unnecessary
> merges. Of course there is also the "rebase" command and some other
> extensions I haven't tried such as pbranch but the standard "mq"
> extension is itself very useful.
>
> Just my two cents.
> Regards,
> Adrien.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From santagada at gmail.com  Wed Dec 15 00:38:14 2010
From: santagada at gmail.com (Leonardo Santagada)
Date: Tue, 14 Dec 2010 21:38:14 -0200
Subject: [pypy-dev] Mercurial workflow
In-Reply-To: <AANLkTimKa=-1iOeQ0G4xfaT=R+SVZwio-LxmW0d2v-NT@mail.gmail.com>
References: <201012141804.06449.jacob@openend.se>
	<AANLkTikg=NH+85aX99sT3RUWF5fmRzZKqT9-tVCmL7BD@mail.gmail.com>
	<AANLkTimKa=-1iOeQ0G4xfaT=R+SVZwio-LxmW0d2v-NT@mail.gmail.com>
Message-ID: <AANLkTin9ONHf+1nY=KwbRAfXcwUACXiibGhFSEnmN20+@mail.gmail.com>

I would also advise against using mq for the same reason plus it is a
whole set of commands to learn.

What I would advise is to try to use rebase more instead of merging
most of the time. Here is a link to some docs about rebase
http://mercurial.selenic.com/wiki/RebaseProject

On Tue, Dec 14, 2010 at 8:53 PM, Joe <qbproger at gmail.com> wrote:
> In reference to MqExtention, I've used Stacked Git (a git equivalent)
> in the past, and would generally advise against this. ?I like to
> commit after I get a piece of something working, so I know I can go
> back to that point if I try something and it doesn't work out.
> Stacked Git makes this hard. ?In my opinion, you're essentially
> by-passing a benefit of a scm tool.
>
> I wouldn't advise people to use MqExtension unless they understand
> that if they make a mistake they must undo it manually.
>
> Joe
>
> On Tue, Dec 14, 2010 at 12:42 PM, Adrien Di Mascio <adimasci at gmail.com> wrote:
>> Hi Jacob,
>>
>> On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n <jacob at openend.se> wrote:
>>> now that the switch to Mercurial has happened, people are discovering that
>>> their Subversion workflow habits don't quite work. This is because Mercurial
>>> has a distributed philosophy, inlike svn which has the concept of the holy
>>> central server where all operations take place.
>>
>> It might be relevant for you to consider the mq extension :
>>
>> - http://mercurial.selenic.com/wiki/MqExtension
>> - http://mercurial.selenic.com/wiki/MqTutorial
>> - http://mercurial.selenic.com/wiki/MqExtension
>>
>> I find it very convenient on a daily-basis usage. When you want to
>> implement or fix something, you just create a new patch :
>>
>> $ hg qnew my_new_idea
>>
>> You then start to modify code and update your patch whenever you feel
>> the current version of your code is better than the previous one :
>>
>> $ hq qrefresh
>>
>> Then, when you're happy with it, all tests pass, etc., unapply your
>> patch, pull the new changesets from the "central" repository, and then
>> reapply your patch :
>>
>> $ hg qpop ? ? ? ? ? ? ? ? ? ? ? ? ? # pop your patch
>> $ hg pull central_repo_url ? ? # pull new changesets
>> $ hg up
>> $ hg qpush ? ? ? ? ? ? ? ? ? ? ? ? # reapply your patch
>> $ <make sure tests still pass>
>> $ hg qfinish -a ? ? ? ? ? ? ? ? ? ? # finalize and promote your patch
>> to a changeset tracked in the repository
>> $ hg push central_repo_url ? # push your changeset
>>
>> You can work on several patches at the same time, you can even track
>> your patches in a mercurial repository. In my experience, it helps
>> keeping a revision tree clean and avoid quite a few unnecessary
>> merges. Of course there is also the "rebase" command and some other
>> extensions I haven't tried such as pbranch but the standard "mq"
>> extension is itself very useful.
>>
>> Just my two cents.
>> Regards,
>> Adrien.
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
Leonardo Santagada


From p.giarrusso at gmail.com  Wed Dec 15 02:09:36 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Wed, 15 Dec 2010 02:09:36 +0100
Subject: [pypy-dev] Mercurial workflow
In-Reply-To: <201012141804.06449.jacob@openend.se>
References: <201012141804.06449.jacob@openend.se>
Message-ID: <AANLkTimEaBYeqqFG-E4mYT32LVdvjYKWWPWdW37mfc=z@mail.gmail.com>

Hi,
as a longer time DVCS (Git/Mercurial) user, here's some comments -
just my two cents.

On Tue, Dec 14, 2010 at 18:04, Jacob Hall?n <jacob at openend.se> wrote:
> Hi folks,

> In Mercurial, you do all the important operations in your local repository. If
> you have a central node (which is not strictly necessary), you just push ready
> made change sets to it. Any conflict resolving has already been done on your
> local machine.
>
> This means that it is very convenient to have at least 2 repositories locally
> (this takes hadly any extra space, since unchanged files are hard linked to be
> the same file in each repository).

If you use hg clone, only files under .hg are hard linked - and I
guess they stay so only until they're touched by a commit, when the
_whole_ previous content is duplicated.

If you can do that also for the checked-out copy, you need to use cp
-al, but it's not done by default,
and you need to have advanced Unixy editors like Vim/emacs, and to set
them up correctly. E.g. For Vim, you need to set
backupcopy=breakhardlink (+ any other settings) or =no, otherwise any
editing will affect _both_ repos (if you actually use Vim, read :help
'backupcopy' to understand your options).
patch(1) is also fine, as many tools used by Linux kernel hackers.

I hardly believe many other editors give you control such options. If
you used a Java-based editor (say jEdit), it would even be challenging
to code such a feature.

> You keep one repository as a staging area to which you pull changes from the
> central server (or other peoples repositories) and to which you pull changes
> from your other local repositories. It is here you resolve any merge conflicts
> and it is from here you do pushes of new versions to the central pypy server
> on bitbucket.

One could have two local heads (or something alike) and switch between
them - if you need to recompile after merging and before pushing,
however, things are different of course.

And you might want to have a bitbucket fork for your work which you're
not pushing yet, to have patches reviewed before merging.

> For every idea you have, you clone your repository and do the work for the
> idea in the clone. If you need to collaborate with someone else you
> synchronize your repo with their repo without going through the central
> server.

If "your repo" means a Bitbucket fork (or any sort of publicly
accessible clone, but Bitbucket has some goodies) that's fine and
good. Otherwise, it might make harder for people to have a casual look
at what you're working on - they need to be motivated enough to ask
you to join.

> When you work on an idea you should keep in mind that it is an extremely fast
> operation to commit your work. You can do it for a number of various reasons -
> whenever you have made a set of changes that you feel belong together, if your
> machine is flakey and you want to have a backup copy somewhere else or if you
> want to continue the work on a different machine. I sometimes program in bed,
> and it is very simple to do a pull from my workstation to my laptop.

> The new workflow will make the detailed changes less visible to others, since
> you commit work to the central server in fewer and bigger chunks. This means
> that there will be fewer "oops, I misspelled this" and more of a higher level
> view of what is being developed.

Actually, the ease of doing local commits might lead to many smaller
ones. If you use hg mq and push only finished patches, or you
extensively rewrite commits, then what you say is true.
But there are some disadvantages because history is lost, and the use
of git/hg rebase is not uncontroversial. See below for one
high-profile opinion against git rebase (I don't think the git vs hg
difference has any significance here):

http://lwn.net/Articles/359207/

The bottom line is that there are more possible workflows, and you
will maybe need to discuss which one you'll want to use, and set some
policy.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From p.giarrusso at gmail.com  Wed Dec 15 02:26:09 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Wed, 15 Dec 2010 02:26:09 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>
Message-ID: <AANLkTinG9VyPcFk05qitjHgKCDhd2xf=UuKyg=FH4Z4g@mail.gmail.com>

On Mon, Dec 13, 2010 at 09:31, Miquel Torres <tobami at googlemail.com> wrote:
> Oh, btw., the "normalized" stacked bars now display a warning note
> about its correctness, and how it must be viewed as giving results a
> weighting instead of them being normalized. It even includes a link to
> the proper paper. I hope that is enough for the strict statisticians
> among us ;-)

I see. Thanks!

> See:
> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars
>
> PS: there is a bug in the jqPlot plotting library when null values are
> present. Trying to display PyPy 1.3 results for the newer go, pyflake
> or ?raytrace will create some nasty js loops. It also has problems
> with autoscaling the axis sometimes.
>
>
>
> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>> Thanks all for the input.
>> I've compiled a list based on your mails, the Unladen benchmarks page
>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>> alioth descriptions. Here is an extract of the current speed.pypy.org
>> admin:
>>
>> ai
>> chaos ? Creates chaosgame-like fractals
>> crypto_pyaes ? ?A pure python implementation of AES
>> django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table
>>
>> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch
>> benchmark is defined by programs in Performing Lisp Analysis of the
>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.
>>
>> float ? ? ? ? ? Creates an array of points using circular projection and then
>> normalizes and maximizes them. Floating-point heavy.
>> go ? ? ? ? ? ? ?A go (chess like game) computer player AI.
>> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib
>> meteor-contest ?Searchs for solutions to shape packing puzzle.
>> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the
>> orbits of Jovian planets, using a simple symplectic-integrator.
>> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2
>> decoder/decompressor.
>> raytrace-simple A raytracer renderer
>> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task
>> dispatcher in the kernel of an operating system.
>> rietveld ? ? ? ?A Django application benchmark.
>> slowspitfire
>> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier
>> spectral-norm
>> spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table.
>> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a
>> 1000x1000-cell HTML table, using the cstringio module.
>> telco
>> twisted_iteration
>> twisted_names
>> twisted_pb
>> twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP
>> (on the loopback interface) and then writes bytes as fast as it can.
>> waf ? ? Python-based framework for configuring, compiling and installing
>> applications. It derives from the concepts of other build tools such
>> as Scons, Autotools, CMake or Ant.
>>
>>
>> So the remaining descriptions are
>> ai
>> slowspitfire (what is the exact difference between the three spitfire benches?)
>> spectral-norm
>> telco
>> twisted (most of them)
>>
>> Are the descriptions all right so far?. They can be made much longer
>> if you deem it desirable.
>>
>> on speed.pypy.org you will currently see the descriptions in 3 places:
>> - Changes view: A tooltip on hover over each benchmark
>> - Timeline: a description box beneath each plot
>> - Comparison: A tooltip over each benchmark when hovering the
>> selection menu on the left side.
>>
>> Any suggestions on how to improve it further are welcome ;-)
>>
>> Miquel
>>
>>
>> 2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com> wrote:
>>>> Here is a incomplete draft list:
>>>
>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the
>>>> cstringio version uses a modified engine (that uses cstringio)
>>>
>>>> spambayes: Spambayes is a bayesian spam filter
>>>
>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
>>> remember when, because of this, spambayes was slower (including
>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
>>> and the head (for slowspitfire it's the opposite).
>>>
>>> For the rest, I see no significant case of slowdown of PyPy over time.
>>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
>>> --
>>> Paolo Giarrusso - Ph.D. Student
>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>>
>>
>



-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From wing1127aishi at gmail.com  Wed Dec 15 18:11:28 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Wed, 15 Dec 2010 11:11:28 -0600
Subject: [pypy-dev] Interpretor for vectorized langugage
Message-ID: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>

Hi all

Is pypy suitable for writing interpretor of vectorized language like Matlab,
R etc which vector and matrix are first class objects? This includes matrix
shape inference, and efficient linear algebra code generation.

Any thoughts?

Leon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101215/2fbae5f4/attachment.html>

From renesd at gmail.com  Wed Dec 15 18:21:38 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Wed, 15 Dec 2010 17:21:38 +0000
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
Message-ID: <AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>

Hi,

have you seen numpy/scipy?

Also see these:
    http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html
    http://ademan.wordpress.com/category/code/pypy/


cheers,


On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit <wing1127aishi at gmail.com> wrote:

> Hi all
>
> Is pypy suitable for writing interpretor of vectorized language like
> Matlab, R etc which vector and matrix are first class objects? This includes
> matrix shape inference, and efficient linear algebra code generation.
>
> Any thoughts?
>
> Leon
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment.html>

From wing1127aishi at gmail.com  Wed Dec 15 18:52:55 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Wed, 15 Dec 2010 11:52:55 -0600
Subject: [pypy-dev] pypy-dev Digest, Vol 365, Issue 14
In-Reply-To: <mailman.5910.1292435139.6189.pypy-dev@codespeak.net>
References: <mailman.5910.1292435139.6189.pypy-dev@codespeak.net>
Message-ID: <AANLkTi=q+_OLByn7WHCPseqOowWmuopX2TJG6iTADxkM@mail.gmail.com>

Yes I have. However I want to write an interpretor for matlab/R codebase,
not using python to write numerical code.

Leon

Best Regards,

Leon Sit
Graduate Student in Computer Science at University of Chicago

Message: 7

> Date: Wed, 15 Dec 2010 17:21:38 +0000
> From: Ren? Dudfield <renesd at gmail.com>
> Subject: Re: [pypy-dev] Interpretor for vectorized langugage
> To: PyPy Dev <pypy-dev at codespeak.net>
> Message-ID:
>        <AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi,
>
> have you seen numpy/scipy?
>
> Also see these:
>    http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html
>    http://ademan.wordpress.com/category/code/pypy/
>
>
> cheers,
>
>
> On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit <wing1127aishi at gmail.com> wrote:
>
> > Hi all
> >
> > Is pypy suitable for writing interpretor of vectorized language like
> > Matlab, R etc which vector and matrix are first class objects? This
> includes
> > matrix shape inference, and efficient linear algebra code generation.
> >
> > Any thoughts?
> >
> > Leon
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://codespeak.net/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
> End of pypy-dev Digest, Vol 365, Issue 14
> *****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101215/e56113fd/attachment.html>

From tobami at googlemail.com  Wed Dec 15 19:52:29 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 15 Dec 2010 19:52:29 +0100
Subject: [pypy-dev] pypy-dev Digest, Vol 365, Issue 14
In-Reply-To: <AANLkTi=q+_OLByn7WHCPseqOowWmuopX2TJG6iTADxkM@mail.gmail.com>
References: <mailman.5910.1292435139.6189.pypy-dev@codespeak.net>
	<AANLkTi=q+_OLByn7WHCPseqOowWmuopX2TJG6iTADxkM@mail.gmail.com>
Message-ID: <AANLkTimxv_uB-ZkR-nWRc05z3D8-29pzxNXCwPSeESW9@mail.gmail.com>

I think what Ren? means is that
http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html
is a good example of how a vectorized library can be implemented,
which relates to your question of whether a vectorized language could
be implemented.

Regards,
Miquel


2010/12/15 Leon Sit <wing1127aishi at gmail.com>:
> Yes I have. However I want to write an interpretor for matlab/R codebase,
> not using python to write numerical code.
> Leon
> Best Regards,
>
> Leon Sit
> Graduate Student in Computer Science at University of Chicago
>
> Message: 7
>>
>> Date: Wed, 15 Dec 2010 17:21:38 +0000
>> From: Ren? Dudfield <renesd at gmail.com>
>> Subject: Re: [pypy-dev] Interpretor for vectorized langugage
>> To: PyPy Dev <pypy-dev at codespeak.net>
>> Message-ID:
>> ? ? ? ?<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi,
>>
>> have you seen numpy/scipy?
>>
>> Also see these:
>> ? ?http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html
>> ? ?http://ademan.wordpress.com/category/code/pypy/
>>
>>
>> cheers,
>>
>>
>> On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit <wing1127aishi at gmail.com> wrote:
>>
>> > Hi all
>> >
>> > Is pypy suitable for writing interpretor of vectorized language like
>> > Matlab, R etc which vector and matrix are first class objects? This
>> > includes
>> > matrix shape inference, and efficient linear algebra code generation.
>> >
>> > Any thoughts?
>> >
>> > Leon
>> >
>> > _______________________________________________
>> > pypy-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/pypy-dev
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> http://codespeak.net/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment.htm
>>
>> ------------------------------
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>>
>> End of pypy-dev Digest, Vol 365, Issue 14
>> *****************************************
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From anto.cuni at gmail.com  Thu Dec 16 08:25:32 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 16 Dec 2010 08:25:32 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
Message-ID: <4D09BEEC.8030204@gmail.com>

Hi Miquel, hi all,

as you probably have noticed, we have recently migrated the main repo to
mercurial.  Now speed.pypy.org receives a revision number in the form
"40046:2088ce763fc2", but of course it can no longer fetches the commit logs
from the svn server.

Would it be possible to fetch the commits from the bitbucket repo, please?

ciao,
Anto


From alexandre.fayolle at logilab.fr  Thu Dec 16 09:12:10 2010
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Thu, 16 Dec 2010 09:12:10 +0100
Subject: [pypy-dev] Mercurial workflow
In-Reply-To: <AANLkTimKa=-1iOeQ0G4xfaT=R+SVZwio-LxmW0d2v-NT@mail.gmail.com>
References: <201012141804.06449.jacob@openend.se>
	<AANLkTikg=NH+85aX99sT3RUWF5fmRzZKqT9-tVCmL7BD@mail.gmail.com>
	<AANLkTimKa=-1iOeQ0G4xfaT=R+SVZwio-LxmW0d2v-NT@mail.gmail.com>
Message-ID: <201012160912.11005.alexandre.fayolle@logilab.fr>

On Tuesday 14 December 2010 23:53:41 Joe wrote:
> In reference to MqExtention, I've used Stacked Git (a git equivalent)
> in the past, and would generally advise against this.  I like to
> commit after I get a piece of something working, so I know I can go
> back to that point if I try something and it doesn't work out.
> Stacked Git makes this hard.  In my opinion, you're essentially
> by-passing a benefit of a scm tool.
> 
> I wouldn't advise people to use MqExtension unless they understand
> that if they make a mistake they must undo it manually.

Hi there, my 2 cents

As Adrien said you can manage the patch repository itself with mercurial, so 
that you can quite easily undo changes that don't work for you. 

Also, the hg qimport command is a very useful one which enables to convert 
changesets to patches, which can be qfold'ed in a single larger but cleaner 
patch. 

In the end, all boils down to the style of changesets you want to see in the 
official repositories : lots of very small changes, with broken intermediate 
steps, trial and errors etc. or larger cleaner commits, which deal with a 
single idea and are much easier to peer review. This is not for me to decide 
for pypy, and both approach have benefits and drawbacks. It is however imho a 
worthwhile question. 

Kind regards,

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science


From tobami at googlemail.com  Thu Dec 16 09:29:10 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Thu, 16 Dec 2010 09:29:10 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <4D09BEEC.8030204@gmail.com>
References: <4D09BEEC.8030204@gmail.com>
Message-ID: <AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>

Hi Anto,

yes, that is expected, but no problem. Codespeed is designed in such a
way that it can support different version control systems. It is just
that there is only support for svn now ;-)

So for mercurial support, I need to implement a function that takes a
changeset hash, connects to the hg repo and returns a dict with
'date', 'user' and 'description' or 'summary' info. Anybody knows what
library I could use for that? or do I have to parse local 'hg log'
output?


2010/12/16 Antonio Cuni <anto.cuni at gmail.com>:
> Hi Miquel, hi all,
>
> as you probably have noticed, we have recently migrated the main repo to
> mercurial. ?Now speed.pypy.org receives a revision number in the form
> "40046:2088ce763fc2", but of course it can no longer fetches the commit logs
> from the svn server.
>
> Would it be possible to fetch the commits from the bitbucket repo, please?
>
> ciao,
> Anto
>


From arigo at tunes.org  Thu Dec 16 10:16:36 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Dec 2010 10:16:36 +0100
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
Message-ID: <AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>

Hi,

On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield <renesd at gmail.com> wrote:
>> Is pypy suitable for writing interpretor of vectorized language like
>> Matlab, R etc which vector and matrix are first class objects? This includes
>> matrix shape inference, and efficient linear algebra code generation.
>
> have you seen numpy/scipy?

The first aspect is simply if RPython would be suitable for writing an
interpreter for, say, Matlab.  The answer is "probably yes": PyPy
would be suitable for such dynamic languages, giving you a JIT
compiler for free.  I don't really know how complex the core of these
languages are, but I suspect not too much.

Of course you are then going to hit the same problems that Ademan
tries to solve for numpy/scipy, notably how to implement at least the
basic linear algebra operations in such a way that the JIT can improve
them.  There are various goals there, e.g. to turn Python (or Matlab)
code like A+B+C, adding three matrices together, into one matrix
operation instead of two (as it is now: (A+B)+C).  This is all a bit
experimental so far.


A bient?t,

Armin.


From anto.cuni at gmail.com  Thu Dec 16 10:33:09 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 16 Dec 2010 10:33:09 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
Message-ID: <4D09DCD5.3000602@gmail.com>

Hi Miquel,

On 16/12/10 09:29, Miquel Torres wrote:
> Hi Anto,
> 
> yes, that is expected, but no problem. Codespeed is designed in such a
> way that it can support different version control systems. It is just
> that there is only support for svn now ;-)
> 
> So for mercurial support, I need to implement a function that takes a
> changeset hash, connects to the hg repo and returns a dict with
> 'date', 'user' and 'description' or 'summary' info. Anybody knows what
> library I could use for that? or do I have to parse local 'hg log'
> output?

I think you have several options to do that. For the particular pypy case,
maybe the simplest is to directly ask bitbucket for the changeset:
https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/

this has the advantage that you don't need any library but just an http
request, but of course it works only for bitbucket repos.

Alternatively, if you install mercurial you can then "import
mercurial.commands" and use its public API from Python. Or as you said you can
just execute hg log and parse the output: in this case you might be interested
in the --template option, which allows you to format things exactly as you
want, e.g.:

$ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n'
Armin Rigo <arigo at tunes.org>
2010-12-16 10:01 +0100
Document this publicly.

see this page for more details:
http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html

ciao,
Anto


From tobami at googlemail.com  Thu Dec 16 10:38:04 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Thu, 16 Dec 2010 10:38:04 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <4D09DCD5.3000602@gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
Message-ID: <AANLkTikCAiH+rsPt=RLF12UL59v_5Crzgx_boJ1CW35S@mail.gmail.com>

Ok, thanks Anto, I'll have a look at it this weekend!

PS: If somebody has coded such a function, or feels like doing it that
is also welcome ;-)


2010/12/16 Antonio Cuni <anto.cuni at gmail.com>:
> Hi Miquel,
>
> On 16/12/10 09:29, Miquel Torres wrote:
>> Hi Anto,
>>
>> yes, that is expected, but no problem. Codespeed is designed in such a
>> way that it can support different version control systems. It is just
>> that there is only support for svn now ;-)
>>
>> So for mercurial support, I need to implement a function that takes a
>> changeset hash, connects to the hg repo and returns a dict with
>> 'date', 'user' and 'description' or 'summary' info. Anybody knows what
>> library I could use for that? or do I have to parse local 'hg log'
>> output?
>
> I think you have several options to do that. For the particular pypy case,
> maybe the simplest is to directly ask bitbucket for the changeset:
> https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/
>
> this has the advantage that you don't need any library but just an http
> request, but of course it works only for bitbucket repos.
>
> Alternatively, if you install mercurial you can then "import
> mercurial.commands" and use its public API from Python. Or as you said you can
> just execute hg log and parse the output: in this case you might be interested
> in the --template option, which allows you to format things exactly as you
> want, e.g.:
>
> $ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n'
> Armin Rigo <arigo at tunes.org>
> 2010-12-16 10:01 +0100
> Document this publicly.
>
> see this page for more details:
> http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html
>
> ciao,
> Anto
>


From danchr at gmail.com  Thu Dec 16 11:14:14 2010
From: danchr at gmail.com (Dan Villiom Podlaski Christiansen)
Date: Thu, 16 Dec 2010 11:14:14 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <4D09DCD5.3000602@gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
Message-ID: <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>

On 16 Dec 2010, at 10:33, Antonio Cuni wrote:

> Alternatively, if you install mercurial you can then "import
> mercurial.commands" and use its public API from Python. Or as you  
> said you can
> just execute hg log and parse the output: in this case you might be  
> interested
> in the --template option, which allows you to format things exactly  
> as you
> want, e.g.:
<snip>

I'd suggest parsing the command line output of Mercurial, as it's the  
officially sanctioned stable API for Mercurial. The Python API, while  
faster, provides no stability guarantees. That being said, the  
following will create a repository instance and access the context for  
a particular changeset:

>>> from mercurial import hg, ui
>>>
>>> repo = hg.repository(ui.ui(), 'path/to/repo')
>>> ctx = repo['tip']
>>> log = ctx.description()

The typical accessors for a changeset context are defined at <http://hg.intevation.org/mercurial/file/7397a53219c9/mercurial/context.py#l97 
 >.

Please note that any use of the Python API means that the entire  
application is covered by the GPL.

--

Dan Villiom Podlaski Christiansen
danchr at gmail.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1943 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/1fd877d1/attachment.bin>

From Arnd.Rechenburg at tomtom.com  Thu Dec 16 12:58:31 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Thu, 16 Dec 2010 12:58:31 +0100
Subject: [pypy-dev] import errors when calling pypy
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703F35920@BESRVWC-EXC01.ta.global>

Hi,

 

I have problems to start pypy.

 

I got the following errors:

 

$ pypy-1.4/bin/pypy

pypy-1.4/bin/pypy: /usr/lib64/libcrypto.so.0.9.8: no version information
available (required by pypy-1.4/bin/pypy)

pypy-1.4/bin/pypy: /usr/lib64/libssl.so.0.9.8: no version information
available (required by pypy-1.4/bin/pypy)

'import site' failed

Python 2.5.2 (79529, Nov 25 2010, 23:04:06)

[PyPy 1.4.0] on linux2

Type "help", "copyright", "credits" or "license" for more information.

ImportError: No module named atexit

debug: OperationError:

debug:  operror-type: ImportError

debug:  operror-value: No module named _pypy_interact

 

Could someone tell me how to set the environment for the libraries?

I tried update PATH etc. but without success.

 

On the system (linux SLES10, AMD 64-bit) itself python 2.5.5 is used.

 

Any help would be appreciated.

 

Thx in advance,

Arnd Rechenburg

 

Arnd Rechenburg * Senior Software Engineer * Tele Atlas Deutschland GmbH
& Co. KG * Am Neuen Horizont 1, 31177 Harsum, Deutschland * HRA 200613 *
VAT ID DE811953885 * Steuernummer 30/210/15702 * e-mail: 
arnd.rechenburg at teleatlas.com <mailto:arnd.rechenburg at teleatlas.com>  *
direct +49 5127 40 8364 * reception +49 5127 40 80 * fax +49 5127 40
8348 * www.teleatlas.com <http://www.teleatlas.com/> 

 

  <http://www.teleatlas.com/> 

 

P Please consider the environment before printing this e-mail.

 

E-mail Confidentiality: This e-mail and any attachments thereto may
contain information which is confidential and/or protected by
intellectual property rights and are intended for the sole use of the
recipient(s) named above. Any use of the information contained herein
(including, but not limited to, total or partial reproduction,
communication or distribution in any form) by persons other than the
designated recipient(s) is prohibited. TELE ATLAS DOES NOT AMEND
EXISTING AGREEMENTS OR CREATE ANY BINDING AGREEMENTS OR OBLIGATIONS VIA
E-MAIL. ANY STATEMENTS OR ATTACHMENTS WITHIN THIS MESSAGE WILL NOT BE
CONSIDERED BINDING. If you have received this e-mail in error, please
notify the sender either by telephone or by e-mail and delete the
material from any computer. Thank you for your cooperation. Tele Atlas
B.V. and its affiliates. www.teleatlas.com <http://www.teleatlas.com/> 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/c5923628/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 9550 bytes
Desc: image001.gif
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/c5923628/attachment.gif>

From arigo at tunes.org  Thu Dec 16 13:08:09 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Dec 2010 13:08:09 +0100
Subject: [pypy-dev] import errors when calling pypy
In-Reply-To: <FA92FB0BD1FD6349BD520B84A62DFAC703F35920@BESRVWC-EXC01.ta.global>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F35920@BESRVWC-EXC01.ta.global>
Message-ID: <AANLkTikBi88NWpT0c-ZgSU1GJLTsUM+J8dEh7uYNUMsW@mail.gmail.com>

Hi Arnd,

On Thu, Dec 16, 2010 at 12:58 PM, Arnd Rechenburg <
Arnd.Rechenburg at tomtom.com> wrote:

>  I have problems to start pypy.
>

It is explained (or rather, at least hinted) here:
http://pypy.org/download.html#installing

In more words, though, the important point is that you need to *not* split
up the zip or tbz2 archive that you are getting.  PyPy looks for its
libraries starting from the location of the "pypy" executable.  You can put
the uncompressed constant of the archive anywhere, but don't move "pypy"
outside it.  (It's fine to install a symlink to it from somewhere else.)

We may "rectify" this situation when we have more of a clue about the
directory layout that people would really expect, possibly including a
Makefile to perform installation, for example.  The current situation was
deemed sufficient so far.


A bient?t,

Armin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/f5aed236/attachment.html>

From Arnd.Rechenburg at tomtom.com  Thu Dec 16 13:35:39 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Thu, 16 Dec 2010 13:35:39 +0100
Subject: [pypy-dev] error from function.py
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703F3597A@BESRVWC-EXC01.ta.global>

Hi,

 

I got the following error message:

 

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib-python/2.5.2/calendar.py",
line 11, in <module>

    import locale as _locale

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/local
e.py", line 30, in <module>

    from _locale import *

  File "/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_locale.py",
line 8, in <module>

    from ctypes_support import standard_c_lib as libc

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/ctypes_support.py",
line 24, in <module>

    standard_c_lib.__errno_location.restype =
ctypes.POINTER(ctypes.c_int)

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/ctype
s/__init__.py", line 363, in __getattr__

    func = self.__getitem__(name)

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/ctype
s/__init__.py", line 368, in __getitem__

    func = self._FuncPtr((name_or_ordinal, self))

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_ctypes/function.py",
line 112, in __init__

    ptr = self._getfuncptr([], ctypes.c_int)

  File
"/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_ctypes/function.py",
line 197, in _getfuncptr

    return cdll.ptr(self.name, argshapes, resshape, self._flags_)

AttributeError: 'NoneType' object has no attribute 'ptr'

 

Someone knows more about it?

 

Regards,

Arnd

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/1679905d/attachment.html>

From arigo at tunes.org  Thu Dec 16 16:07:35 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Dec 2010 16:07:35 +0100
Subject: [pypy-dev] error from function.py
In-Reply-To: <FA92FB0BD1FD6349BD520B84A62DFAC703F3597A@BESRVWC-EXC01.ta.global>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F3597A@BESRVWC-EXC01.ta.global>
Message-ID: <AANLkTikEQpa2dXnwU7TBxx2R+_7bSwQsO5VghP-tDUTk@mail.gmail.com>

Hi,

On Thu, Dec 16, 2010 at 1:35 PM, Arnd Rechenburg
<Arnd.Rechenburg at tomtom.com> wrote:
> ??? return cdll.ptr(self.name, argshapes, resshape, self._flags_)
>
> AttributeError: 'NoneType' object has no attribute 'ptr'

That's a very indirect way of crashing because
ctypes.util.find_library('c') returned None, i.e. it did not find your
C library.  I agree that this could be improved by at least reporting
a clearer error.

To understand the error, first note that it occurs in standard code
from CPython's "ctypes.util" module.  Look in
lib-python/modified-2.5.2/ctypes/util.py.  On Linux it seems to call
"/sbin/ldconfig -p" and look in the result for "/libc.*".  On my
machine it would first find "/lib/libc.so.6" in this way, which is
correct.  In general it all looks pretty hackish, but it's not clear
if any particular piece of code is to blame, given the way that the
system of libraries is designed, which is not really adapted for fully
dynamic lookups like needed here...


A bient?t,

Armin.


From fijall at gmail.com  Thu Dec 16 20:22:27 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 16 Dec 2010 21:22:27 +0200
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
Message-ID: <AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>

On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen
<danchr at gmail.com> wrote:
> On 16 Dec 2010, at 10:33, Antonio Cuni wrote:
>
>> Alternatively, if you install mercurial you can then "import
>> mercurial.commands" and use its public API from Python. Or as you said you
>> can
>> just execute hg log and parse the output: in this case you might be
>> interested
>> in the --template option, which allows you to format things exactly as you
>> want, e.g.:
>
> <snip>
>
> I'd suggest parsing the command line output of Mercurial, as it's the
> officially sanctioned stable API for Mercurial. The Python API, while
> faster, provides no stability guarantees. That being said, the following
> will create a repository instance and access the context for a particular
> changeset:
>
>>>> from mercurial import hg, ui
>>>>
>>>> repo = hg.repository(ui.ui(), 'path/to/repo')
>>>> ctx = repo['tip']
>>>> log = ctx.description()
>
> The typical accessors for a changeset context are defined at
> <http://hg.intevation.org/mercurial/file/7397a53219c9/mercurial/context.py#l97>.
>
> Please note that any use of the Python API means that the entire application
> is covered by the GPL.
>

How is it even technically possible? It does not link against
mercurial and GPL specifically excludes anything about *running* your
software.


From fijall at gmail.com  Thu Dec 16 20:20:46 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 16 Dec 2010 21:20:46 +0200
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
Message-ID: <AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>

On Thu, Dec 16, 2010 at 11:16 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield <renesd at gmail.com> wrote:
>>> Is pypy suitable for writing interpretor of vectorized language like
>>> Matlab, R etc which vector and matrix are first class objects? This includes
>>> matrix shape inference, and efficient linear algebra code generation.
>>
>> have you seen numpy/scipy?
>
> The first aspect is simply if RPython would be suitable for writing an
> interpreter for, say, Matlab. ?The answer is "probably yes": PyPy
> would be suitable for such dynamic languages, giving you a JIT
> compiler for free. ?I don't really know how complex the core of these
> languages are, but I suspect not too much.
>
> Of course you are then going to hit the same problems that Ademan
> tries to solve for numpy/scipy, notably how to implement at least the
> basic linear algebra operations in such a way that the JIT can improve
> them. ?There are various goals there, e.g. to turn Python (or Matlab)
> code like A+B+C, adding three matrices together, into one matrix
> operation instead of two (as it is now: (A+B)+C). ?This is all a bit
> experimental so far.
>
>
> A bient?t,
>
> Armin.

Regarding this - I was thinking about haveing a + b - c create a
bytecode that would be executed using small interpreter with a
jit-merge-point and a hint "can be vectorized".


From alex.gaynor at gmail.com  Thu Dec 16 20:27:18 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 16 Dec 2010 14:27:18 -0500
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
Message-ID: <AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>

On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen
> <danchr at gmail.com> wrote:
> > On 16 Dec 2010, at 10:33, Antonio Cuni wrote:
> >
> >> Alternatively, if you install mercurial you can then "import
> >> mercurial.commands" and use its public API from Python. Or as you said
> you
> >> can
> >> just execute hg log and parse the output: in this case you might be
> >> interested
> >> in the --template option, which allows you to format things exactly as
> you
> >> want, e.g.:
> >
> > <snip>
> >
> > I'd suggest parsing the command line output of Mercurial, as it's the
> > officially sanctioned stable API for Mercurial. The Python API, while
> > faster, provides no stability guarantees. That being said, the following
> > will create a repository instance and access the context for a particular
> > changeset:
> >
> >>>> from mercurial import hg, ui
> >>>>
> >>>> repo = hg.repository(ui.ui(), 'path/to/repo')
> >>>> ctx = repo['tip']
> >>>> log = ctx.description()
> >
> > The typical accessors for a changeset context are defined at
> > <
> http://hg.intevation.org/mercurial/file/7397a53219c9/mercurial/context.py#l97
> >.
> >
> > Please note that any use of the Python API means that the entire
> application
> > is covered by the GPL.
> >
>
> How is it even technically possible? It does not link against
> mercurial and GPL specifically excludes anything about *running* your
> software.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Because the term "link" is totally meaningless in the context of Python
code.  Some people say this means it's impossible for Python to invoke that
clause of the GPL, others say it means `import`ing something invokes it.
 There's no correct answer.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/ccb3206a/attachment.html>

From alex.gaynor at gmail.com  Thu Dec 16 20:28:08 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 16 Dec 2010 14:28:08 -0500
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>
Message-ID: <AANLkTinCPOZo2ArPKOpXvGMo+q6Q3GK24A0d=oe-YUMj@mail.gmail.com>

On Thu, Dec 16, 2010 at 2:20 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Thu, Dec 16, 2010 at 11:16 AM, Armin Rigo <arigo at tunes.org> wrote:
> > Hi,
> >
> > On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield <renesd at gmail.com> wrote:
> >>> Is pypy suitable for writing interpretor of vectorized language like
> >>> Matlab, R etc which vector and matrix are first class objects? This
> includes
> >>> matrix shape inference, and efficient linear algebra code generation.
> >>
> >> have you seen numpy/scipy?
> >
> > The first aspect is simply if RPython would be suitable for writing an
> > interpreter for, say, Matlab.  The answer is "probably yes": PyPy
> > would be suitable for such dynamic languages, giving you a JIT
> > compiler for free.  I don't really know how complex the core of these
> > languages are, but I suspect not too much.
> >
> > Of course you are then going to hit the same problems that Ademan
> > tries to solve for numpy/scipy, notably how to implement at least the
> > basic linear algebra operations in such a way that the JIT can improve
> > them.  There are various goals there, e.g. to turn Python (or Matlab)
> > code like A+B+C, adding three matrices together, into one matrix
> > operation instead of two (as it is now: (A+B)+C).  This is all a bit
> > experimental so far.
> >
> >
> > A bient?t,
> >
> > Armin.
>
> Regarding this - I was thinking about haveing a + b - c create a
> bytecode that would be executed using small interpreter with a
> jit-merge-point and a hint "can be vectorized".
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

That seems like a pretty big special case, why not work at the larger idea
of cross-loop optimizations?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/a5a62eff/attachment.html>

From danchr at gmail.com  Thu Dec 16 21:05:05 2010
From: danchr at gmail.com (Dan Villiom Podlaski Christiansen)
Date: Thu, 16 Dec 2010 21:05:05 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
Message-ID: <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>

On 16 Dec 2010, at 20:27, Alex Gaynor wrote:

> On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski  
> <fijall at gmail.com>wrote:
>
>> On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen
>> <danchr at gmail.com> wrote:
>>
>>> Please note that any use of the Python API means that the entire
>>> application is covered by the GPL.
>>
>> How is it even technically possible? It does not link against
>> mercurial and GPL specifically excludes anything about *running* your
>> software.
>
> Because the term "link" is totally meaningless in the context of  
> Python
> code.  Some people say this means it's impossible for Python to  
> invoke that
> clause of the GPL, others say it means `import`ing something invokes  
> it.
> There's no correct answer.

Well, some people believe that any use of a GPL API constitutes a  
derivative work, and is thus covered by it. Mercurial even went  
through a licence change from GPLv2 to GPLv2+ to get rid of this  
ambiguity. See the links below; otherwise the convert extension (which  
is distributed with Mercurial) might be in violation of its own licence.

<http://mercurial.selenic.com/wiki/MercurialApi>
<http://mercurial.selenic.com/wiki/License>

<http://mercurial.selenic.com/wiki/Relicensing>
<http://selenic.com/pipermail/mercurial/2009-September/027740.html>
<http://selenic.com/pipermail/mercurial-devel/2009-October/015963.html>

Anyway, as long as your code is open, people are unlikely to care  
enough to do anything about it?

--

Dan Villiom Podlaski Christiansen
danchr at gmail.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1943 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/a6ccf788/attachment.bin>

From renesd at gmail.com  Thu Dec 16 23:43:19 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Thu, 16 Dec 2010 22:43:19 +0000
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
Message-ID: <AANLkTinKgX1xWoKoKkvFhgujGZKksQFtj77abZUNgkAG@mail.gmail.com>

On Thu, Dec 16, 2010 at 9:16 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi,
>
> On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield <renesd at gmail.com> wrote:
> >> Is pypy suitable for writing interpretor of vectorized language like
> >> Matlab, R etc which vector and matrix are first class objects? This
> includes
> >> matrix shape inference, and efficient linear algebra code generation.
> >
> > have you seen numpy/scipy?
>
> The first aspect is simply if RPython would be suitable for writing an
> interpreter for, say, Matlab.  The answer is "probably yes": PyPy
> would be suitable for such dynamic languages, giving you a JIT
> compiler for free.  I don't really know how complex the core of these
> languages are, but I suspect not too much.
>
> Of course you are then going to hit the same problems that Ademan
> tries to solve for numpy/scipy, notably how to implement at least the
> basic linear algebra operations in such a way that the JIT can improve
> them.  There are various goals there, e.g. to turn Python (or Matlab)
> code like A+B+C, adding three matrices together, into one matrix
> operation instead of two (as it is now: (A+B)+C).  This is all a bit
> experimental so far.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Hi,

Numexpr is really worth looking at:
    http://code.google.com/p/numexpr/

Especially as it combines expressions together, and does things like
chunking (called tiling in the graphics world).

>>> timeit a**2 + b**2 + 2*a*b
10 loops, best of 3: 35.9 ms per loop
>>> ne.set_num_threads(1)  # using 1 threads (on a 6-core machine)
>>> timeit ne.evaluate("a**2 + b**2 + 2*a*b")
100 loops, best of 3: 9.28 ms per loop   # 3.9x faster than NumPy
>>> ne.set_num_threads(4)  # using 4 threads (on a 6-core machine)
>>> timeit ne.evaluate("a**2 + b**2 + 2*a*b")
100 loops, best of 3: 4.17 ms per loop   # 8.6x faster than NumPy


This approach, could be applied in the pypy jit I guess.

For a vectorised xplatform run time assembler see:
    http://code.entropywave.com/git?p=orc.git;a=summary

This allows you to write asm code in the one language, and have it compile
to ARM, SSE, mmx, PPC, NEON, DSP, and even into C.  Instructions that are
not in one set, are emulated by multiple instructions.  It's mainly used for
video, sound and image processing - which are big arrays.


Just some interesting vectorised thingies to look at.

cya.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/9fc0908f/attachment.html>

From p.giarrusso at gmail.com  Fri Dec 17 00:12:10 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Fri, 17 Dec 2010 00:12:10 +0100
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
Message-ID: <AANLkTi=1nL8m2N-s0DcGY=Nq-3WM-LJrGemkrc2h5e8a@mail.gmail.com>

On Thu, Dec 16, 2010 at 10:16, Armin Rigo <arigo at tunes.org> wrote:
> Hi,

>> have you seen numpy/scipy?

> Of course you are then going to hit the same problems that Ademan
> tries to solve for numpy/scipy, notably how to implement at least the
> basic linear algebra operations in such a way that the JIT can improve
> them. ?There are various goals there, e.g. to turn Python (or Matlab)
> code like A+B+C, adding three matrices together, into one matrix
> operation instead of two (as it is now: (A+B)+C). ?This is all a bit
> experimental so far.

[More about numpy/scipy]:
How about the ideas from C++ expression templates? Or even better,
some sort of lazy evaluation?

With C++ expression templates [1], (A+B) returns something like
PlusMatrix(A, B). When finally assigning to a variable, the copy
constructor converts this into a real matrix; in the following line:
D = (A+B) + C
PlusMatrix(PlusMatrix(A, B), C) is built, and D gets the result.

What they could further do, but they don't (see FAQ of uBlas [1]), is
to cache the result of evaluating a template when requested; that
could be cached (as in lazy evaluation - expressions like PlusMatrix
are not very different from thunks, in that case) or maybe JIT
compilation can recognize the constantness of this.

Best regards

[1] One library using them is Boost::uBlas, but the idea is much older:
http://www.boost.org/doc/libs/1_45_0/libs/numeric/ublas/doc/index.htm
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From william.leslie.ttg at gmail.com  Fri Dec 17 00:21:37 2010
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Fri, 17 Dec 2010 10:21:37 +1100
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTinKgX1xWoKoKkvFhgujGZKksQFtj77abZUNgkAG@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTinKgX1xWoKoKkvFhgujGZKksQFtj77abZUNgkAG@mail.gmail.com>
Message-ID: <AANLkTimG8H-PmOj8RmFYfngDYJEV7T56pDmYPBYHcCrA@mail.gmail.com>

On 17 December 2010 09:43, Ren? Dudfield <renesd at gmail.com> wrote:
> [Numexpr] approach, could be applied in the pypy jit I guess.

There was a lot of discussion in this direction during micronumpy
experiments, but I'm not sure how well ideas were documented.

A similar class of optimisation is polyhedral loop refactoring;
implementing it within the framework of a tracing JIT would be new and
exciting research if anyone is looking for a topic.

-- 
William Leslie


From wing1127aishi at gmail.com  Fri Dec 17 02:08:00 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Thu, 16 Dec 2010 19:08:00 -0600
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTimG8H-PmOj8RmFYfngDYJEV7T56pDmYPBYHcCrA@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTinKgX1xWoKoKkvFhgujGZKksQFtj77abZUNgkAG@mail.gmail.com>
	<AANLkTimG8H-PmOj8RmFYfngDYJEV7T56pDmYPBYHcCrA@mail.gmail.com>
Message-ID: <AANLkTi=oOk4kj-tM+2QPVseLwbihbMKi4W6tfZvi9OQK@mail.gmail.com>

Actually.. my original idea is to parse matlab/r code and generate legal C++
code with this library
http://eigen.tuxfamily.org/index.php?title=Main_Page#Documentation

<http://eigen.tuxfamily.org/index.php?title=Main_Page#Documentation>and use
pypy to do some kind of type/shape inference on matrix object


On Thu, Dec 16, 2010 at 5:21 PM, William ML Leslie <
william.leslie.ttg at gmail.com> wrote:

> On 17 December 2010 09:43, Ren? Dudfield <renesd at gmail.com> wrote:
> > [Numexpr] approach, could be applied in the pypy jit I guess.
>
> There was a lot of discussion in this direction during micronumpy
> experiments, but I'm not sure how well ideas were documented.
>
> A similar class of optimisation is polyhedral loop refactoring;
> implementing it within the framework of a tracing JIT would be new and
> exciting research if anyone is looking for a topic.
>
> --
> William Leslie
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101216/96b428d6/attachment.html>

From fijall at gmail.com  Fri Dec 17 07:27:31 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 17 Dec 2010 08:27:31 +0200
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTi=1nL8m2N-s0DcGY=Nq-3WM-LJrGemkrc2h5e8a@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTi=1nL8m2N-s0DcGY=Nq-3WM-LJrGemkrc2h5e8a@mail.gmail.com>
Message-ID: <AANLkTimp9DqYrbDTE3+X+xr3po=s_Bm6aN-wgudy6tX6@mail.gmail.com>

On Fri, Dec 17, 2010 at 1:12 AM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Thu, Dec 16, 2010 at 10:16, Armin Rigo <arigo at tunes.org> wrote:
>> Hi,
>
>>> have you seen numpy/scipy?
>
>> Of course you are then going to hit the same problems that Ademan
>> tries to solve for numpy/scipy, notably how to implement at least the
>> basic linear algebra operations in such a way that the JIT can improve
>> them. ?There are various goals there, e.g. to turn Python (or Matlab)
>> code like A+B+C, adding three matrices together, into one matrix
>> operation instead of two (as it is now: (A+B)+C). ?This is all a bit
>> experimental so far.
>
> [More about numpy/scipy]:
> How about the ideas from C++ expression templates? Or even better,
> some sort of lazy evaluation?
>
> With C++ expression templates [1], (A+B) returns something like
> PlusMatrix(A, B). When finally assigning to a variable, the copy
> constructor converts this into a real matrix; in the following line:
> D = (A+B) + C
> PlusMatrix(PlusMatrix(A, B), C) is built, and D gets the result.
>
> What they could further do, but they don't (see FAQ of uBlas [1]), is
> to cache the result of evaluating a template when requested; that
> could be cached (as in lazy evaluation - expressions like PlusMatrix
> are not very different from thunks, in that case) or maybe JIT
> compilation can recognize the constantness of this.
>
> Best regards
>
> [1] One library using them is Boost::uBlas, but the idea is much older:
> http://www.boost.org/doc/libs/1_45_0/libs/numeric/ublas/doc/index.htm
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


Hi.

I did that (it's not published anywhere though) and it gave quite huge
speedups compared to numexpr or numpy. I'll look into implementing it
"properly" at some point (instead of doing that in pure python).


From tobami at googlemail.com  Fri Dec 17 09:27:26 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Fri, 17 Dec 2010 09:27:26 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
Message-ID: <AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>

Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ anyway...



2010/12/16 Dan Villiom Podlaski Christiansen <danchr at gmail.com>:
> On 16 Dec 2010, at 20:27, Alex Gaynor wrote:
>
>> On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski
>> <fijall at gmail.com>wrote:
>>
>>> On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen
>>> <danchr at gmail.com> wrote:
>>>
>>>> Please note that any use of the Python API means that the entire
>>>> application is covered by the GPL.
>>>
>>> How is it even technically possible? It does not link against
>>> mercurial and GPL specifically excludes anything about *running* your
>>> software.
>>
>> Because the term "link" is totally meaningless in the context of Python
>> code. ?Some people say this means it's impossible for Python to invoke
>> that
>> clause of the GPL, others say it means `import`ing something invokes it.
>> There's no correct answer.
>
> Well, some people believe that any use of a GPL API constitutes a derivative
> work, and is thus covered by it. Mercurial even went through a licence
> change from GPLv2 to GPLv2+ to get rid of this ambiguity. See the links
> below; otherwise the convert extension (which is distributed with Mercurial)
> might be in violation of its own licence.
>
> <http://mercurial.selenic.com/wiki/MercurialApi>
> <http://mercurial.selenic.com/wiki/License>
>
> <http://mercurial.selenic.com/wiki/Relicensing>
> <http://selenic.com/pipermail/mercurial/2009-September/027740.html>
> <http://selenic.com/pipermail/mercurial-devel/2009-October/015963.html>
>
> Anyway, as long as your code is open, people are unlikely to care enough to
> do anything about it?
>
> --
>
> Dan Villiom Podlaski Christiansen
> danchr at gmail.com
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Fri Dec 17 13:44:56 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 17 Dec 2010 13:44:56 +0100
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTinCPOZo2ArPKOpXvGMo+q6Q3GK24A0d=oe-YUMj@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>
	<AANLkTinCPOZo2ArPKOpXvGMo+q6Q3GK24A0d=oe-YUMj@mail.gmail.com>
Message-ID: <AANLkTikLdniRpYuxbEZNS7uh=LAc-pW-rG2rtc3D0oyy@mail.gmail.com>

Hi Alex,

On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>> Regarding this - I was thinking about haveing a + b - c create a
>> bytecode that would be executed using small interpreter with a
>> jit-merge-point and a hint "can be vectorized".
>
> That seems like a pretty big special case, why not work at the larger idea
> of cross-loop optimizations?

We have cross-loop optimizations.  Unless I'm misunderstanding you, we
have already done it.  That's how for example map(f, somelist) gets
JITted: it is a small loop that contains a call to the "normal" loop
of the interpreter for f().  This is JITted as the loop from map()
containing inline code from the interpreter loop.

I am a bit unsure however what is being discussed here, because
fijal's comment is very terse and contains hidden issues, like when to
create the bytecode (at runtime? but how can it be a green constant
then?  And what is the hint "can be vectorized"?).


A bient?t,

Armin.


From exarkun at twistedmatrix.com  Fri Dec 17 15:44:07 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 17 Dec 2010 14:44:07 -0000
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
	<AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>
Message-ID: <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>

On 08:27 am, tobami at googlemail.com wrote:
>Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ 
>anyway...

>:(

Jean-Paul


From tobami at googlemail.com  Fri Dec 17 15:48:59 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Fri, 17 Dec 2010 15:48:59 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
	<AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>
	<20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>
Message-ID: <AANLkTim4Ho5-QOV98r=NirZLyzbYGPEB4aQc+aYYEE2C@mail.gmail.com>

@Jean-Paul:  I thought it would only be a problem if the license was GPLv3?

But in any case, what is the conclusion, if any? Can the Mercurial API
be used from LGPL code or not?


2010/12/17  <exarkun at twistedmatrix.com>:
> On 08:27 am, tobami at googlemail.com wrote:
>>
>> Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ anyway...
>
>> :(
>
> Jean-Paul
>


From exarkun at twistedmatrix.com  Fri Dec 17 16:06:17 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 17 Dec 2010 15:06:17 -0000
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTim4Ho5-QOV98r=NirZLyzbYGPEB4aQc+aYYEE2C@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
	<AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>
	<20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>
	<AANLkTim4Ho5-QOV98r=NirZLyzbYGPEB4aQc+aYYEE2C@mail.gmail.com>
Message-ID: <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain>

On 02:48 pm, tobami at googlemail.com wrote:
>@Jean-Paul:  I thought it would only be a problem if the license was 
>GPLv3?

Yes, that's right.  But I got sad anyway.
>But in any case, what is the conclusion, if any? Can the Mercurial API
>be used from LGPL code or not?

This seems unanswerable.  If you believe some people, yes.  If you 
believe other people, no.  The only way to resolve the conflict is to do 
it, and either get sued by the copyright holder or not get sued by the 
copyright holder (you could argue that the outcome of the lawsuit must 
also be considered, but by the time you've gotten sued you probably 
don't care anymore).

Jean-Paul


From garyrob at me.com  Fri Dec 17 17:08:58 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 17 Dec 2010 11:08:58 -0500
Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
Message-ID: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>

When I try to start PyPy, I get:

  bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory

This is on a SuSE Linux Enterprise Server 10.3 SP2.

This system does have:

  /lib/libbz2.so.1.0.0

As a quick experiment, I tried temporarily renaming it to   /lib/libbz2.so.1.0 but that didn't change the result. I also tried installing the bzip2 package from the rpm, but it said it was already installed.

Can anyone suggest a solution? 



-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From arigo at tunes.org  Fri Dec 17 18:16:29 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 17 Dec 2010 18:16:29 +0100
Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
In-Reply-To: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
References: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
Message-ID: <AANLkTi=993RN8VKbibcOr7uiP5LsG=K_45tkUpGMgks2@mail.gmail.com>

Hi Gary,

On Fri, Dec 17, 2010 at 5:08 PM, Gary Robinson <garyrob at me.com> wrote:
> As a quick experiment, I tried temporarily renaming it to ? /lib/libbz2.so.1.0 but that didn't change the result.

It's strange, what I would expect on a system is that there is a
symlink from libbz2.so.1.0 to libbz2.so.1.0.0.  Adding it should be
enough -- did you run ldconfig after trying to rename?  It is needed
to refresh Linux's caches.


A bient?t,

Armin.


From wlavrijsen at lbl.gov  Fri Dec 17 18:32:11 2010
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 17 Dec 2010 09:32:11 -0800 (PST)
Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
In-Reply-To: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
References: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
Message-ID: <alpine.LNX.2.00.1012170929030.12126@pc3-wlav.lbl.gov>

Hi Gary,

>  bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open
> shared object file: No such file or directory
>
> This is on a SuSE Linux Enterprise Server 10.3 SP2.
>
> This system does have:
>
>  /lib/libbz2.so.1.0.0
>
> As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0
> but that didn't change the result. I also tried installing the bzip2 package
> from the rpm, but it said it was already installed.
>
> Can anyone suggest a solution?

Just guessing ... is this perhaps a 32/64b mix? I.e. are you running 64b pypy-c
and is a libbz2 expected from /lib64? That's what bit me in a similar situation
(albeit on a RH-clone linux install).

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net


From garyrob at me.com  Fri Dec 17 18:36:18 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 17 Dec 2010 12:36:18 -0500
Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
In-Reply-To: <AANLkTi=993RN8VKbibcOr7uiP5LsG=K_45tkUpGMgks2@mail.gmail.com>
References: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
	<AANLkTi=993RN8VKbibcOr7uiP5LsG=K_45tkUpGMgks2@mail.gmail.com>
Message-ID: <597BA6F7-6079-4FDD-B033-26EC39F8AB3C@me.com>

I did do ldconfig.

I tried a symlink, copying the /lib/libbz2.so.1.0.0, and the temporary rename I mentioned. An example:

dev01:~/pypy-1.4 # ln -s /lib/libbz2.so.1.0.0 /lib/libbz2.so.1.0
dev01:~/pypy-1.4 # ldconfig
dev01:~/pypy-1.4 # bin/pypy
bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory
dev01:~/pypy-1.4 # ls -ld /lib/libbz*
lrwxrwxrwx 1 root root    20 Dec 17 12:33 /lib/libbz2.so.1.0 -> /lib/libbz2.so.1.0.0
-rwxr-xr-x 1 root root 66864 Jun 16  2006 /lib/libbz2.so.1.0.0

I keep on thinking I must be doing something stupid, but it continues to elude me. (I also tried "ldconfig /lib" with no difference)


-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Dec 17, 2010, at 12:16 PM, Armin Rigo wrote:

> Hi Gary,
> 
> On Fri, Dec 17, 2010 at 5:08 PM, Gary Robinson <garyrob at me.com> wrote:
>> As a quick experiment, I tried temporarily renaming it to   /lib/libbz2.so.1.0 but that didn't change the result.
> 
> It's strange, what I would expect on a system is that there is a
> symlink from libbz2.so.1.0 to libbz2.so.1.0.0.  Adding it should be
> enough -- did you run ldconfig after trying to rename?  It is needed
> to refresh Linux's caches.
> 
> 
> A bient?t,
> 
> Armin.



From garyrob at me.com  Fri Dec 17 20:28:06 2010
From: garyrob at me.com (Gary Robinson)
Date: Fri, 17 Dec 2010 14:28:06 -0500
Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
In-Reply-To: <alpine.LNX.2.00.1012170929030.12126@pc3-wlav.lbl.gov>
References: <CE7AB051-7C97-4582-B353-FAC0912E61E8@me.com>
	<alpine.LNX.2.00.1012170929030.12126@pc3-wlav.lbl.gov>
Message-ID: <EB11498B-34DA-4763-9DC9-55C08893CCC4@me.com>

> are you running 64b pypy-c
> and is a libbz2 expected from /lib64

That was it!

I added a symbolic link with

  ln -s /lib64/libbz2.so.1.0.0 /lib64/libbz2.so.1.0

and now it works. 

Many thanks for the help!!!

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Dec 17, 2010, at 12:32 PM, wlavrijsen at lbl.gov wrote:

> Hi Gary,
> 
>> bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open
>> shared object file: No such file or directory
>> 
>> This is on a SuSE Linux Enterprise Server 10.3 SP2.
>> 
>> This system does have:
>> 
>> /lib/libbz2.so.1.0.0
>> 
>> As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0
>> but that didn't change the result. I also tried installing the bzip2 package
>> from the rpm, but it said it was already installed.
>> 
>> Can anyone suggest a solution?
> 
> Just guessing ... is this perhaps a 32/64b mix? I.e. are you running 64b pypy-c
> and is a libbz2 expected from /lib64? That's what bit me in a similar situation
> (albeit on a RH-clone linux install).
> 
> Best regards,
>           Wim
> -- 
> WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net



From p.giarrusso at gmail.com  Sat Dec 18 10:11:14 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 18 Dec 2010 10:11:14 +0100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain>
References: <4D09BEEC.8030204@gmail.com>
	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>
	<4D09DCD5.3000602@gmail.com>
	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>
	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>
	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>
	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>
	<AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>
	<20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>
	<AANLkTim4Ho5-QOV98r=NirZLyzbYGPEB4aQc+aYYEE2C@mail.gmail.com>
	<20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain>
Message-ID: <AANLkTikLNLLLtaJz3SocraCeocuXo2Do+_PDQomPBVgA@mail.gmail.com>

On Fri, Dec 17, 2010 at 16:06,  <exarkun at twistedmatrix.com> wrote:
> On 02:48 pm, tobami at googlemail.com wrote:
>>@Jean-Paul: ?I thought it would only be a problem if the license was
>>GPLv3?
>
> Yes, that's right. ?But I got sad anyway.
>>But in any case, what is the conclusion, if any? Can the Mercurial API
>>be used from LGPL code or not?
>
> This seems unanswerable. ?If you believe some people, yes. ?If you
> believe other people, no. ?The only way to resolve the conflict is to do
> it, and either get sued by the copyright holder or not get sued by the
> copyright holder (you could argue that the outcome of the lawsuit must
> also be considered, but by the time you've gotten sued you probably
> don't care anymore).

Well, one point is that the matter is complex enough to make you
suable. Moreover, the point that you're technically not linking seems
extremely technical and specific to Python, and even to CPython - a
JIT compiler links together the code, even more than it happens for C
libraries. Therefore, explaining it to a judge would take an extremely
expensive lawyer.

However, would the Mercurial developers sue you (through some
FSF-sponsorship or the like)? You can ask them if they're bothered by
your license. If they're not, they might still change their mind, but
at least they'll (likely) ask you to change license before suing you
(so they save the money for the lawyer).

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From p.giarrusso at gmail.com  Sat Dec 18 10:48:00 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 18 Dec 2010 10:48:00 +0100
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTikLdniRpYuxbEZNS7uh=LAc-pW-rG2rtc3D0oyy@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>
	<AANLkTinCPOZo2ArPKOpXvGMo+q6Q3GK24A0d=oe-YUMj@mail.gmail.com>
	<AANLkTikLdniRpYuxbEZNS7uh=LAc-pW-rG2rtc3D0oyy@mail.gmail.com>
Message-ID: <AANLkTimORRqEHJtHgj4Kt62VnqJJEqVZquRUDp7nphpL@mail.gmail.com>

Hi Armin, hi all,
below I give my 2 cents, giving some background on advanced
optimizations like cross-loop optimization and automatic
vectorization, which the others are mentioning without reference.
These optimizations are used for numerical code like the one discussed
(and mostly specific to that) in compiled languages like
C/C++/Fortran.

On Fri, Dec 17, 2010 at 13:44, Armin Rigo <arigo at tunes.org> wrote:
> Hi Alex,
>
> On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>>> Regarding this - I was thinking about haveing a + b - c create a
>>> bytecode that would be executed using small interpreter with a
>>> jit-merge-point and a hint "can be vectorized".
>>
>> That seems like a pretty big special case, why not work at the larger idea
>> of cross-loop optimizations?
>
> We have cross-loop optimizations. ?Unless I'm misunderstanding you, we
> have already done it. ?That's how for example map(f, somelist) gets
> JITted: it is a small loop that contains a call to the "normal" loop
> of the interpreter for f(). ?This is JITted as the loop from map()
> containing inline code from the interpreter loop.

Cross-loop optimization are useful to optimize C/Fortran numerical
code. For code like a + b - c, you want your compiler to optimize
together different loops from the application (the loop implementing
a+b and the one subtracting c from the result).
For instance, if a, b, c are vectors, a + b - c is equivalent (after
inlining) to the following code, which should be transformed into a
single loop, to reduce loop overhead but more importantly to perform
only one pass on d and thus improve cache locality for huge vectors,
not entirely fitting in CPU caches:

// a, b, c, d are vectors
for i in range(1, length):
  d[i] = a[i] + b[i]
for i in range(1, length):
  d[i] = d[i] - c[i]

optimized result:

for i in range(1, length):
  d[i] = a[i] + b[i] + c[i]

> I am a bit unsure however what is being discussed here, because
> fijal's comment is very terse and contains hidden issues, like when to
> create the bytecode (at runtime? but how can it be a green constant
> then?

I think he wants to do something similar to regexp compilation +
JITting as described on PyPy's blog.
One can compile a regexp to bytecode (at runtime), but afterwards the
result is constant for the regexp interpreter.

In this case, I read the statement as fijal wants to compile not a
regexp but the equivalent of a C++ expression template (see my other
email): for a + b - c, you'd get Minus(Plus(a, b), c), and hopefully
compile it directly to the efficient code above.

> And what is the hint "can be vectorized"?).

I can explain what "vectorized" means (it is a specific loop
optimization), but I'm not sure about why you want the annotation. A
reasonable possibility is to run this optimization pass only on the
code with such an annotation, because most code doesn't need it.

Most likely in this context, he refers to (automatic) loop
vectorization, i.e., automatically converting the code to use SIMD
instructions. On current processor, the following matrix code can be
implemented in one SIMD sum instruction, but the compiler has to
figure it out:

for i in range(1, 4):
  c[i] = a[i] + b[i]

In general, a SIMD operation operates in parallel on two short
operands of vectors, rather than on two single operands. The Intel
C/Fortran compilers, and more recently also GCC, can perform this
rewriting without annotations from the user.

See for instance:
http://en.wikipedia.org/wiki/Vectorization_(parallel_computing)
Introduction of:
http://software.intel.com/en-us/articles/vectorization-with-the-intel-compilers-part-i/
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From hyarion at iinet.net.au  Sat Dec 18 12:10:11 2010
From: hyarion at iinet.net.au (Ben)
Date: Sat, 18 Dec 2010 22:10:11 +1100
Subject: [pypy-dev] speed.pypy.org and mercurial
In-Reply-To: <AANLkTikLNLLLtaJz3SocraCeocuXo2Do+_PDQomPBVgA@mail.gmail.com>
References: <4D09BEEC.8030204@gmail.com>	<AANLkTiniAK4BAEtb-+_TbL9ebj4j1ow8s5Awcmq7B7HV@mail.gmail.com>	<4D09DCD5.3000602@gmail.com>	<0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com>	<AANLkTimpDv_wRdBkoZ-U1E3FahPrsru3aFcd5of7EGmB@mail.gmail.com>	<AANLkTincVXGmVqPowr0GuESPc=wobNVG+0=BA7kahKJ2@mail.gmail.com>	<7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com>	<AANLkTin_AtEOWs2Vidk9Pnio=k3KtpLaiLWTvu9MMPyV@mail.gmail.com>	<20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain>	<AANLkTim4Ho5-QOV98r=NirZLyzbYGPEB4aQc+aYYEE2C@mail.gmail.com>	<20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain>
	<AANLkTikLNLLLtaJz3SocraCeocuXo2Do+_PDQomPBVgA@mail.gmail.com>
Message-ID: <4D0C9693.2030307@iinet.net.au>

On 18/12/10 8:11 PM, Paolo Giarrusso wrote:
> On Fri, Dec 17, 2010 at 16:06,<exarkun at twistedmatrix.com>  wrote:
>> On 02:48 pm,tobami at googlemail.com  wrote:
>>> @Jean-Paul:  I thought it would only be a problem if the license was
>>> GPLv3?
>> Yes, that's right.  But I got sad anyway.
>>> But in any case, what is the conclusion, if any? Can the Mercurial API
>>> be used from LGPL code or not?
>> This seems unanswerable.  If you believe some people, yes.  If you
>> believe other people, no.  The only way to resolve the conflict is to do
>> it, and either get sued by the copyright holder or not get sued by the
>> copyright holder (you could argue that the outcome of the lawsuit must
>> also be considered, but by the time you've gotten sued you probably
>> don't care anymore).
> Well, one point is that the matter is complex enough to make you
> suable. Moreover, the point that you're technically not linking seems
> extremely technical and specific to Python, and even to CPython - a
> JIT compiler links together the code, even more than it happens for C
> libraries. Therefore, explaining it to a judge would take an extremely
> expensive lawyer.

The GPL doesn't mention linking at all. It applies to "the Program or 
any derivative work under copyright law: that is to say, a work 
containing the Program or a portion of it, either verbatim or with 
modifications and/or translated into another language". Linking is only 
relevant because traditional static linking causes parts of the library 
to be included in compiled binaries, which automatically makes them 
derivative works.

Downstream recipients of your code are also perfectly within their 
rights to combine GPL code with code released under an incompatible 
license; provided they never distribute such code, the GPL imposes no 
restrictions. So the fact that the JIT compiler links code together on 
the end user's system is not relevant. The GPL terms only come into play 
if the CodeSpeed source can be considered a "derivative work under 
copyright law" of Mercurial.

If CodeSpeed is designed to be support several version control systems 
via independently installable modules, then I don't see how it could be 
considered a derivative work of Mercurial. The integration module might 
be, but to accept that the existance of such a module makes CodeSpeed a 
derivative work means that CodeSpeed is also a derivative work of every 
version control system, or at least will be as soon as a module is 
released for each system (which could be done by other people). I find 
that rather ludicrous.

That said, I'm not a lawyer, and I'm applying common sense reasoning 
here, not knowledge of how the legal system actually treats things.

-- Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101218/d5e1e1a7/attachment.html>

From wing1127aishi at gmail.com  Sat Dec 18 17:14:08 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Sat, 18 Dec 2010 10:14:08 -0600
Subject: [pypy-dev] Interpretor for vectorized langugage
In-Reply-To: <AANLkTimORRqEHJtHgj4Kt62VnqJJEqVZquRUDp7nphpL@mail.gmail.com>
References: <AANLkTimkkGt9gXTGaY63gvG1M_-AgixKyJKuRut_g3Sg@mail.gmail.com>
	<AANLkTin_Q8C2DgEm06tnFi-NWT_p+6kg665EbcG=YPKj@mail.gmail.com>
	<AANLkTinBC+Qg7fGkx5kcnb35vjMk_pOPEByRAU2NpkB1@mail.gmail.com>
	<AANLkTikaT9JW47=tEsjc7T44PgpR6miE=QbEm4+GbXmY@mail.gmail.com>
	<AANLkTinCPOZo2ArPKOpXvGMo+q6Q3GK24A0d=oe-YUMj@mail.gmail.com>
	<AANLkTikLdniRpYuxbEZNS7uh=LAc-pW-rG2rtc3D0oyy@mail.gmail.com>
	<AANLkTimORRqEHJtHgj4Kt62VnqJJEqVZquRUDp7nphpL@mail.gmail.com>
Message-ID: <AANLkTim6YLQequ6nid3t2e=yGQxxquvX9tPPKdYTX_F6@mail.gmail.com>

My original intent is very much describe by Paolo. I want to use the idea of
template expression in c++ to apply to linear algebra code. The whole idea
of template expression is that all the overloaded operation does not trigger
immediate evaluation of the expression, but to return an expression instead
to next operator call. After chaining the entire formula, then the loop
unrolling starts to happen when caller invokes the operator[].

Sample

def fun(a,b,c,d): //say they are all matrix, and the formula might not make
sense but for sake of argument only
    return a*a+b*c*b.T/ norm(d.T*d)

into low level C++

expr& fun(const expr& a, const expr& b, const expr&c, const expr& d){
return sum(prod(a*a), prod(b, prod(c, transpose(b))))/
norm(prod(transpose(d), d));
}

note that the type is "expression" in both parameters and input. The reason
is twofold
1) no eager evaluation on the subexpression, only evaluate the expression
when the entire expression is known
2) support lazy evaluation on the expression like

fun(x+y,y*y+y,x**y,y)[a,b]

where only the necessary computation for the entry [a,b] are calculated. For
better demo on this idea, say I have

def sum3(a,b,c):
   return a+b+c

then sum3(x+y, y^2, 2*y)[1,10] should only trigger the evaluation of

return (x[1,10] + y[1,10]) + y[1,10]^2 + 2 * y[1, 10]

which itself can be optimised further and nothing else should be evaluated.
boost::ublas is rather a slow implementation of template expression idea and
the Eigen library has way more mature and efficient implementation. Without
this technique, numerical C library was never as fast as hand coded C and
Fortran code because hidden temporaries were everywhere in the formula. Only
after this template expressin technique, C++ matrix library are on par with
fortran and C. When the expression is long enough, C++ template expression
library gives much better than Lapack/atlas code because the compiler will
optimise the full expression.

I hand port an optimisation library from homebrew matrix library to ublas
and get 20x speed up, then moving to Eigen library get another 8x speed from
ublas (without using idiomatic Eigen style, only switch the type). Main
reason why Eigen is so much faster is that when unrolling the loop in Eigen,
SSE code are explicitly generated instead of depending compiler's auto
vectorisation which is not mature and only happen to simple code, openMP are
applied to matrix multiplication at the most outer loop, and data are
aligned in memory for less cache misses.


On Sat, Dec 18, 2010 at 3:48 AM, Paolo Giarrusso <p.giarrusso at gmail.com>wrote:

> Hi Armin, hi all,
> below I give my 2 cents, giving some background on advanced
> optimizations like cross-loop optimization and automatic
> vectorization, which the others are mentioning without reference.
> These optimizations are used for numerical code like the one discussed
> (and mostly specific to that) in compiled languages like
> C/C++/Fortran.
>
> On Fri, Dec 17, 2010 at 13:44, Armin Rigo <arigo at tunes.org> wrote:
> > Hi Alex,
> >
> > On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor <alex.gaynor at gmail.com>
> wrote:
> >>> Regarding this - I was thinking about haveing a + b - c create a
> >>> bytecode that would be executed using small interpreter with a
> >>> jit-merge-point and a hint "can be vectorized".
> >>
> >> That seems like a pretty big special case, why not work at the larger
> idea
> >> of cross-loop optimizations?
> >
> > We have cross-loop optimizations.  Unless I'm misunderstanding you, we
> > have already done it.  That's how for example map(f, somelist) gets
> > JITted: it is a small loop that contains a call to the "normal" loop
> > of the interpreter for f().  This is JITted as the loop from map()
> > containing inline code from the interpreter loop.
>
> Cross-loop optimization are useful to optimize C/Fortran numerical
> code. For code like a + b - c, you want your compiler to optimize
> together different loops from the application (the loop implementing
> a+b and the one subtracting c from the result).
> For instance, if a, b, c are vectors, a + b - c is equivalent (after
> inlining) to the following code, which should be transformed into a
> single loop, to reduce loop overhead but more importantly to perform
> only one pass on d and thus improve cache locality for huge vectors,
> not entirely fitting in CPU caches:
>
> // a, b, c, d are vectors
> for i in range(1, length):
>  d[i] = a[i] + b[i]
> for i in range(1, length):
>  d[i] = d[i] - c[i]
>
> optimized result:
>
> for i in range(1, length):
>  d[i] = a[i] + b[i] + c[i]
>
> > I am a bit unsure however what is being discussed here, because
> > fijal's comment is very terse and contains hidden issues, like when to
> > create the bytecode (at runtime? but how can it be a green constant
> > then?
>
> I think he wants to do something similar to regexp compilation +
> JITting as described on PyPy's blog.
> One can compile a regexp to bytecode (at runtime), but afterwards the
> result is constant for the regexp interpreter.
>
> In this case, I read the statement as fijal wants to compile not a
> regexp but the equivalent of a C++ expression template (see my other
> email): for a + b - c, you'd get Minus(Plus(a, b), c), and hopefully
> compile it directly to the efficient code above.
>
> > And what is the hint "can be vectorized"?).
>
> I can explain what "vectorized" means (it is a specific loop
> optimization), but I'm not sure about why you want the annotation. A
> reasonable possibility is to run this optimization pass only on the
> code with such an annotation, because most code doesn't need it.
>
> Most likely in this context, he refers to (automatic) loop
> vectorization, i.e., automatically converting the code to use SIMD
> instructions. On current processor, the following matrix code can be
> implemented in one SIMD sum instruction, but the compiler has to
> figure it out:
>
> for i in range(1, 4):
>  c[i] = a[i] + b[i]
>
> In general, a SIMD operation operates in parallel on two short
> operands of vectors, rather than on two single operands. The Intel
> C/Fortran compilers, and more recently also GCC, can perform this
> rewriting without annotations from the user.
>
> See for instance:
> http://en.wikipedia.org/wiki/Vectorization_(parallel_computing)
> Introduction of:
>
> http://software.intel.com/en-us/articles/vectorization-with-the-intel-compilers-part-i/
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101218/e4a4e31b/attachment.html>

From garyrob at me.com  Sat Dec 18 23:58:15 2010
From: garyrob at me.com (Gary Robinson)
Date: Sat, 18 Dec 2010 17:58:15 -0500
Subject: [pypy-dev] playing with fast-forward
Message-ID: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>

I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.)

I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit.

The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. 

I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. 

I was able to run:

  python translate.py -Ojit

successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!?

Any suggestions how to run fast-forward with jit?


** I've used text browsers w3m and elinks on SuSE, and Safari on OS X (two different machines in different cities). Twice, the Safari download has stopped at < 500K. (It LOOKS like a successful download, it's just incomplete.) The SuSE browsers usually get 16 or 18MB and stop, and tar gets "gzip: stdin: unexpected end of file" when I try to uncompress them.


-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From alex.gaynor at gmail.com  Sun Dec 19 00:00:32 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 18 Dec 2010 18:00:32 -0500
Subject: [pypy-dev] playing with fast-forward
In-Reply-To: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>
References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>
Message-ID: <AANLkTimkO-7XW70ZKOHHDrT_xyG+6tQ4iXfan1mZvd4Z@mail.gmail.com>

On Sat, Dec 18, 2010 at 5:58 PM, Gary Robinson <garyrob at me.com> wrote:

> I'm experimenting with the fast-forward branch. I'm actually not sure about
> the proper way to get it. (I have the main branch working fine.)
>
> I downloaded a nojit version from
> http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see
> any jit versions listed there. I was happy to see that the method I care
> most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want
> to test it with the jit.
>
> The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be
> for the branch, but the mercurial URL shown on that page appears to be for
> the main project.
>
> I tried the downloads under the Source pop-up menu. For some reason, I only
> sporadically am able to get a complete .gz file.** But I did get one.
>
> I was able to run:
>
>  python translate.py -Ojit
>
> successfully -- or at least it appeared so. But I couldn't find a bin/pypy
> to run!?
>
> Any suggestions how to run fast-forward with jit?
>
>
> ** I've used text browsers w3m and elinks on SuSE, and Safari on OS X (two
> different machines in different cities). Twice, the Safari download has
> stopped at < 500K. (It LOOKS like a successful download, it's just
> incomplete.) The SuSE browsers usually get 16 or 18MB and stop, and tar gets
> "gzip: stdin: unexpected end of file" when I try to uncompress them.
>
>
> --
>
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog:    http://www.garyrobinson.net
>
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

a) If you want to work with the code it's probably easier to just use a
mercurial checkout, rather than downloading a tarball.
b) non-release builds (aka what you built) don't generate bin/pypy, it
generates pypy-c in the directory you invoked the translator from.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101218/ad302245/attachment.html>

From anto.cuni at gmail.com  Sun Dec 19 12:35:05 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 19 Dec 2010 12:35:05 +0100
Subject: [pypy-dev] playing with fast-forward
In-Reply-To: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>
References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>
Message-ID: <4D0DEDE9.8050300@gmail.com>

On 18/12/10 23:58, Gary Robinson wrote:
> I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.)
> 
> I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit.

consider that we don't have automatic nightly builds for fast-forward, so the
ones you find on that page are manually triggered, and possibly outdated.

> The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. 

yes, if you clone the mercurial repo you get all the branches together.

> I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. 
> 
> I was able to run:
> 
>   python translate.py -Ojit
> 
> successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!?

as suggested by Alex, you probably have a pypy-c binary in
pypy/translator/goal (I think that the translate script even says so, but I
agree that it produces so much output that it might be hard to stop the
message :-))

> Any suggestions how to run fast-forward with jit?

I don't know how the "download source" button of bitbucket works, so I'm not
even sure that you downloaded the fast-forward branch instead of the default one.

The best way is to do this:

$ hg clone http://bitbucket.org/pypy/pypy
$ cd pypy
$ hg up -r fast-forward

Now you can go to pypy/translator/goal and run translate.py again.

ciao,
Anto


From Arnd.Rechenburg at tomtom.com  Mon Dec 20 19:16:26 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Mon, 20 Dec 2010 19:16:26 +0100
Subject: [pypy-dev] ImportError: No module named _sqlite3
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703F75FDF@BESRVWC-EXC01.ta.global>

Hi,

 

I tried to use sqlite3 with pypy but I got the following error:

 

>>>> import sqlite

Traceback (most recent call last):

  File "<console>", line 1, in <module>

ImportError: No module named sqlite

>>>> import sqlite3

Traceback (most recent call last):

  File "<console>", line 1, in <module>

  File "sqlite3/__init__.py", line 24, in <module>

    from dbapi2 import *

  File "sqlite3/dbapi2.py", line 27, in <module>

    from _sqlite3 import *

ImportError: No module named _sqlite3

 

Someone knows where to put the _sqlite3.so library?

 

Any help would be appreciated.

 

Thx in advance,

Arnd Rechenburg

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101220/c144aa13/attachment.html>

From garyrob at me.com  Mon Dec 20 20:42:21 2010
From: garyrob at me.com (Gary Robinson)
Date: Mon, 20 Dec 2010 14:42:21 -0500
Subject: [pypy-dev] playing with fast-forward
In-Reply-To: <4D0DEDE9.8050300@gmail.com>
References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com>
	<4D0DEDE9.8050300@gmail.com>
Message-ID: <5388F994-B794-43F5-B9DE-3AE417FB2090@me.com>

Many thanks for this detailed help. Since I hadn't used Mercurial before, the notes about Mercurial were particularly helpful.

I did some testing on fast-forward, and it looks like even now, it basically has what I need for serious use. multiprocessing.Pool.imap_unordered() is the method I tend to use all the time for spreading a task across processors, and it works. (At least it has so far in my testing.) And of course 64-bit was implemented a couple of months ago, which I need because of the size of the data structures I often use.

So this is really great news.

The main thing I'm looking for now is SciPy support, but I know that's a ways off, and I can do a lot without it.

Thanks again to everyone for the great work.

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net




On Dec 19, 2010, at 6:35 AM, Antonio Cuni wrote:

> On 18/12/10 23:58, Gary Robinson wrote:
>> I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.)
>> 
>> I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit.
> 
> consider that we don't have automatic nightly builds for fast-forward, so the
> ones you find on that page are manually triggered, and possibly outdated.
> 
>> The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. 
> 
> yes, if you clone the mercurial repo you get all the branches together.
> 
>> I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. 
>> 
>> I was able to run:
>> 
>>  python translate.py -Ojit
>> 
>> successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!?
> 
> as suggested by Alex, you probably have a pypy-c binary in
> pypy/translator/goal (I think that the translate script even says so, but I
> agree that it produces so much output that it might be hard to stop the
> message :-))
> 
>> Any suggestions how to run fast-forward with jit?
> 
> I don't know how the "download source" button of bitbucket works, so I'm not
> even sure that you downloaded the fast-forward branch instead of the default one.
> 
> The best way is to do this:
> 
> $ hg clone http://bitbucket.org/pypy/pypy
> $ cd pypy
> $ hg up -r fast-forward
> 
> Now you can go to pypy/translator/goal and run translate.py again.
> 
> ciao,
> Anto



From dimaqq at gmail.com  Tue Dec 21 01:59:50 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Mon, 20 Dec 2010 17:59:50 -0700
Subject: [pypy-dev] funding/popularity?
Message-ID: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>

First of all, congratulation on getting Eurostars

Would be nice if companies that use Python extensively chipped in [more].
By [out of] my [@%%] calculations, each percent of perfromance
improvement that pypy achieves potentially saves google many millions
a year in data centre costs.
What are other companies use/love python and how much?


Popularity, I for one would like python to be used much more widely;
it is of course a double-edges sword, on one hand more users and
developers means more contributors, libraries and understanding that
python is important, in terms of research, funding, etc; on the other
more bad code and mess. Anyhow I see at least these problems for now:

in general: 2.x/3.x dichtomy; some ext's are not ported, pypy doesn't
yet run 3k (I'd be willing to work on this btw), lack of authoritative
information on the subject; Since pypy is a forward-looking project, I
think a push to include 3.2 (and perhaps 2.7 which would come
naturally) would be very good here, at least clear statement that pypy
is going there. More visibility for performance achievements would do
good too.

in classic apps: hard to bundle python with downloadable apps, e.g.
win builds still recommend 2.5; perhaps stackless single-exe builds
are the best here? A bit better on linux where dependency tracking is
the responsibility of the disrto.

on the web: bad support in apache, little advice for users what to use
and how frameworks are structured; I reckon some pypy side projects
could be great here, e.g. isolated programs within same interpreter or
multiple interpreters in same process.

in schools/unis: I really can't comprehend why!


Anyhow I think 1.4 release is great!
Though I must contend I am still to try it with most of my code, too
many ext deps and very little of my python code is cpu-bound.


Sidetracking... one day when pypy jit/gc/etc are all worked out, how
hard would it be to use same techniques and most of backends for some
unrelated language that doesn't have jit yet, e.g. php? And how hard
would it be to marry two dynamic languages, so that modules from one
could be used in the other? Or that modules written in rpython could
be used in several langs?


From william.leslie.ttg at gmail.com  Tue Dec 21 03:21:49 2010
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Tue, 21 Dec 2010 13:21:49 +1100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
Message-ID: <AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>

On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
> More visibility for performance achievements would do
> good too.

Where are pypy's performance achievements *not* visible, but should be?

> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
> hard would it be to use same techniques and most of backends for some
> unrelated language that doesn't have jit yet, e.g. php?

You know that pypy already has implementations of other languages,
right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
integrated with the translated pypy-c, but they show that it is not
too difficult to write a runtime for any dynamic language you choose.

> And how hard
> would it be to marry two dynamic languages, so that modules from one
> could be used in the other? Or that modules written in rpython could
> be used in several langs?

It's in the "interesting problems" bucket, and the effort required
depends on the level of integration between languages you want.  There
are several projects already attempting to do decent integration
between several languages, besides the approach used on the JVM, there
are also GNU Guile, Racket, and Parrot, among others.  It might be
worth waiting to see how these different projects pan out, before
spending a bunch of effort just to be an also-ran in the
multi-language runtime market.

However, implementing more languages in rpython has the side-effect of
propagating the l * o * p problem: it introduces more and more
implementations that then have to be maintained, so good
cross-language integration probably belongs /outside/ pypy itself, so
existing runtimes can hook into it.

But it would be an interesting experiment (to marry the various
interpreters pypy ships with), if you wanted to try it.

My two cents.

-- 
William Leslie


From dimaqq at gmail.com  Tue Dec 21 07:57:52 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Mon, 20 Dec 2010 23:57:52 -0700
Subject: [pypy-dev]  funding/popularity?
In-Reply-To: <AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
Message-ID: <AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>

On 20 December 2010 19:21, William ML Leslie
<william.leslie.ttg at gmail.com> wrote:
> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>> More visibility for performance achievements would do
>> good too.
>
> Where are pypy's performance achievements *not* visible, but should be?

for example http://shootout.alioth.debian.org/
doesn't say which pypy version is used, what options, doesn't have
performance figures for multithreaded/multicore

also benchmarks are kinda small, most of them are not docuemented, and
I couldn't find any info if the same python code was used for cpython
and pypy (both shootout and speed pypy) or interpreter-specific
verions were used, that is each tweaked for best performance given the
known tradeoffs for each implementation.further the most standard
benchmarks, pystone, etc. completely ignore the fact that real world
programs are large and only a few small paths are execured often.

another temp problem with speed pypy is that it's terrubly slow in ff
beta. it also occasionally grinds in stable ff and opera, but I guess
this can be forgiven for the sake of simplicity / developer effort.

if you google for 'python performance' you don't get a single link to
pypy on the first page, as a matter of fact, codespeak is poorly
indexed, it took me quite some time to get some of my questions
answered with a search. also if you look up 'pypy gc' you get a page
on codespeak, but to interpret what the data actually means is so far
beyond me.

a good overview is found in the mainling list
http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
slowspitfire and spambayes bits are outdated by now.

the definitive good thing about pypy is that it's much easier to find
out about its inner workings than that of cpython!

hopefully a bit more of end-user stuff get known.
let's call it pypy public outreach (feature request)

>
>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>> hard would it be to use same techniques and most of backends for some
>> unrelated language that doesn't have jit yet, e.g. php?
>
> You know that pypy already has implementations of other languages,
> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
> integrated with the translated pypy-c, but they show that it is not
> too difficult to write a runtime for any dynamic language you choose.

Oh I didn't know there were so many, and I mistakenly though that js
was a target, not implmented langauge. In any case I read somewhere
that js support was retired...

>
>> And how hard
>> would it be to marry two dynamic languages, so that modules from one
>> could be used in the other? Or that modules written in rpython could
>> be used in several langs?
>
> It's in the "interesting problems" bucket, and the effort required
> depends on the level of integration between languages you want. ?There
> are several projects already attempting to do decent integration
> between several languages, besides the approach used on the JVM, there
> are also GNU Guile, Racket, and Parrot, among others. ?It might be
> worth waiting to see how these different projects pan out, before
> spending a bunch of effort just to be an also-ran in the
> multi-language runtime market.
>
> However, implementing more languages in rpython has the side-effect of
> propagating the l * o * p problem: it introduces more and more
> implementations that then have to be maintained, so good
> cross-language integration probably belongs /outside/ pypy itself, so
> existing runtimes can hook into it.

Makes perfect sense, after all any given other language hardly has the
same data model as python.

>
> But it would be an interesting experiment (to marry the various
> interpreters pypy ships with), if you wanted to try it.
>
> My two cents.
>
> --
> William Leslie
>


From tobami at googlemail.com  Tue Dec 21 09:06:22 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Tue, 21 Dec 2010 09:06:22 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
Message-ID: <AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>

Hi Dima,

> another temp problem with speed pypy is that it's terrubly slow in ff
> beta. it also occasionally grinds in stable ff and opera, but I guess
> this can be forgiven for the sake of simplicity / developer effort.

Well, speed.pypy is actually fast in all modern browsers. The problem
you are referring to is probably caused by a bug in the javascript
plotting library (jqPplot) that is triggered in the comparison view
when there are some results with 0 values. It only appears for some
plot types, but it is very annoying because it grinds the browser to a
halt like you say. Is that what you meant?

We are looking into it, and will fix that library if necessary.

Cheers,
Miquel


2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
> On 20 December 2010 19:21, William ML Leslie
> <william.leslie.ttg at gmail.com> wrote:
>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> More visibility for performance achievements would do
>>> good too.
>>
>> Where are pypy's performance achievements *not* visible, but should be?
>
> for example http://shootout.alioth.debian.org/
> doesn't say which pypy version is used, what options, doesn't have
> performance figures for multithreaded/multicore
>
> also benchmarks are kinda small, most of them are not docuemented, and
> I couldn't find any info if the same python code was used for cpython
> and pypy (both shootout and speed pypy) or interpreter-specific
> verions were used, that is each tweaked for best performance given the
> known tradeoffs for each implementation.further the most standard
> benchmarks, pystone, etc. completely ignore the fact that real world
> programs are large and only a few small paths are execured often.
>
> another temp problem with speed pypy is that it's terrubly slow in ff
> beta. it also occasionally grinds in stable ff and opera, but I guess
> this can be forgiven for the sake of simplicity / developer effort.
>
> if you google for 'python performance' you don't get a single link to
> pypy on the first page, as a matter of fact, codespeak is poorly
> indexed, it took me quite some time to get some of my questions
> answered with a search. also if you look up 'pypy gc' you get a page
> on codespeak, but to interpret what the data actually means is so far
> beyond me.
>
> a good overview is found in the mainling list
> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
> slowspitfire and spambayes bits are outdated by now.
>
> the definitive good thing about pypy is that it's much easier to find
> out about its inner workings than that of cpython!
>
> hopefully a bit more of end-user stuff get known.
> let's call it pypy public outreach (feature request)
>
>>
>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>> hard would it be to use same techniques and most of backends for some
>>> unrelated language that doesn't have jit yet, e.g. php?
>>
>> You know that pypy already has implementations of other languages,
>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>> integrated with the translated pypy-c, but they show that it is not
>> too difficult to write a runtime for any dynamic language you choose.
>
> Oh I didn't know there were so many, and I mistakenly though that js
> was a target, not implmented langauge. In any case I read somewhere
> that js support was retired...
>
>>
>>> And how hard
>>> would it be to marry two dynamic languages, so that modules from one
>>> could be used in the other? Or that modules written in rpython could
>>> be used in several langs?
>>
>> It's in the "interesting problems" bucket, and the effort required
>> depends on the level of integration between languages you want. ?There
>> are several projects already attempting to do decent integration
>> between several languages, besides the approach used on the JVM, there
>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>> worth waiting to see how these different projects pan out, before
>> spending a bunch of effort just to be an also-ran in the
>> multi-language runtime market.
>>
>> However, implementing more languages in rpython has the side-effect of
>> propagating the l * o * p problem: it introduces more and more
>> implementations that then have to be maintained, so good
>> cross-language integration probably belongs /outside/ pypy itself, so
>> existing runtimes can hook into it.
>
> Makes perfect sense, after all any given other language hardly has the
> same data model as python.
>
>>
>> But it would be an interesting experiment (to marry the various
>> interpreters pypy ships with), if you wanted to try it.
>>
>> My two cents.
>>
>> --
>> William Leslie
>>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From Arnd.Rechenburg at tomtom.com  Tue Dec 21 09:18:47 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Tue, 21 Dec 2010 09:18:47 +0100
Subject: [pypy-dev] ImportError: No module named _sqlite3
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703F760FB@BESRVWC-EXC01.ta.global>

Hi,

 

I saw my mistake already by myself.

 

Regards,

Arnd

________________________________

From: Arnd Rechenburg 
Sent: Montag, 20. Dezember 2010 19:16
To: 'pypy-dev at codespeak.net'
Subject: ImportError: No module named _sqlite3

 

Hi,

 

I tried to use sqlite3 with pypy but I got the following error:

 

>>>> import sqlite

Traceback (most recent call last):

  File "<console>", line 1, in <module>

ImportError: No module named sqlite

>>>> import sqlite3

Traceback (most recent call last):

  File "<console>", line 1, in <module>

  File "sqlite3/__init__.py", line 24, in <module>

    from dbapi2 import *

  File "sqlite3/dbapi2.py", line 27, in <module>

    from _sqlite3 import *

ImportError: No module named _sqlite3

 

Someone knows where to put the _sqlite3.so library?

 

Any help would be appreciated.

 

Thx in advance,

Arnd Rechenburg

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101221/e9e062c6/attachment.html>

From Arnd.Rechenburg at tomtom.com  Tue Dec 21 09:58:52 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Tue, 21 Dec 2010 09:58:52 +0100
Subject: [pypy-dev] long.__itemsize__
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>

Hi,

 

In my code I need something like

long.__itemsize__

in Python.

 

Any suggestions to do it with pypy?

 

Regards,

Arnd

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101221/d09914ca/attachment.html>

From fijall at gmail.com  Tue Dec 21 10:36:07 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 21 Dec 2010 11:36:07 +0200
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
Message-ID: <AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>

On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg
<Arnd.Rechenburg at tomtom.com> wrote:
> Hi,
>
>
>
> In my code I need something like
>
> long.__itemsize__
>
> in Python.
>
>
>
> Any suggestions to do it with pypy?
>

Hey.

This attribute is undocumented and untested as far as I can tell. What
does it do?

Cheers,
fijal


>
>
> Regards,
>
> Arnd
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From renesd at gmail.com  Tue Dec 21 12:02:29 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Tue, 21 Dec 2010 11:02:29 +0000
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
	<AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>
Message-ID: <AANLkTi=bne7g66nyr11SxKCTtKW=aArhgwZ_3XhVRP9z@mail.gmail.com>

Hi,

__itemsize__ - in bytes, corresponds to item size field in the types
definition structure.

It's a field for types.
See:
    http://docs.python.org/c-api/typeobj.html#tp_itemsize



On Tue, Dec 21, 2010 at 9:36 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
> On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg
> <Arnd.Rechenburg at tomtom.com> wrote:
> > Hi,
> >
> >
> >
> > In my code I need something like
> >
> > long.__itemsize__
> >
> > in Python.
> >
> >
> >
> > Any suggestions to do it with pypy?
> >
>
> Hey.
>
> This attribute is undocumented and untested as far as I can tell. What
> does it do?
>
> Cheers,
> fijal
>
>
> >
> >
> > Regards,
> >
> > Arnd
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From fijall at gmail.com  Tue Dec 21 12:05:40 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 21 Dec 2010 13:05:40 +0200
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <AANLkTi=bne7g66nyr11SxKCTtKW=aArhgwZ_3XhVRP9z@mail.gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
	<AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>
	<AANLkTi=bne7g66nyr11SxKCTtKW=aArhgwZ_3XhVRP9z@mail.gmail.com>
Message-ID: <AANLkTikfzzTz-Q=KF+qT=p_fHiCQnvoNGsBGT8CG8SVm@mail.gmail.com>

On Tue, Dec 21, 2010 at 1:02 PM, Ren? Dudfield <renesd at gmail.com> wrote:
> Hi,
>
> __itemsize__ - in bytes, corresponds to item size field in the types
> definition structure.
>
> It's a field for types.
> See:
> ? ?http://docs.python.org/c-api/typeobj.html#tp_itemsize
>

Well... Those are docs for C API. It doesn't say it's exposed at
applevel nor since which version. (Also, to be precise, C API is known
to be implementation specific)


From anto.cuni at gmail.com  Tue Dec 21 12:31:14 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 21 Dec 2010 12:31:14 +0100
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <AANLkTikfzzTz-Q=KF+qT=p_fHiCQnvoNGsBGT8CG8SVm@mail.gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>	<AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>	<AANLkTi=bne7g66nyr11SxKCTtKW=aArhgwZ_3XhVRP9z@mail.gmail.com>
	<AANLkTikfzzTz-Q=KF+qT=p_fHiCQnvoNGsBGT8CG8SVm@mail.gmail.com>
Message-ID: <4D109002.2050505@gmail.com>

On 21/12/10 12:05, Maciej Fijalkowski wrote:

>> __itemsize__ - in bytes, corresponds to item size field in the types
>> definition structure.
>>
>> It's a field for types.
>> See:
>>    http://docs.python.org/c-api/typeobj.html#tp_itemsize
>>
> 
> Well... Those are docs for C API. It doesn't say it's exposed at
> applevel nor since which version. (Also, to be precise, C API is known
> to be implementation specific)

Moreover, I don't think we could give it a sane semantics on PyPy, given that
the same applevel type can be potentially implemented by many different low
level structures with different sizes.

ciao,
Anto


From fijall at gmail.com  Tue Dec 21 12:31:59 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 21 Dec 2010 13:31:59 +0200
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <4D109002.2050505@gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
	<AANLkTikoPKe2CyseVqKYavWqHyy02+hBq5Z9qLqAjBe_@mail.gmail.com>
	<AANLkTi=bne7g66nyr11SxKCTtKW=aArhgwZ_3XhVRP9z@mail.gmail.com>
	<AANLkTikfzzTz-Q=KF+qT=p_fHiCQnvoNGsBGT8CG8SVm@mail.gmail.com>
	<4D109002.2050505@gmail.com>
Message-ID: <AANLkTikwq2Vfg3p_Z280XjqMFa8DA5NdRkmBY3LTgFh_@mail.gmail.com>

On Tue, Dec 21, 2010 at 1:31 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> On 21/12/10 12:05, Maciej Fijalkowski wrote:
>
>>> __itemsize__ - in bytes, corresponds to item size field in the types
>>> definition structure.
>>>
>>> It's a field for types.
>>> See:
>>> ? ?http://docs.python.org/c-api/typeobj.html#tp_itemsize
>>>
>>
>> Well... Those are docs for C API. It doesn't say it's exposed at
>> applevel nor since which version. (Also, to be precise, C API is known
>> to be implementation specific)
>
> Moreover, I don't think we could give it a sane semantics on PyPy, given that
> the same applevel type can be potentially implemented by many different low
> level structures with different sizes.
>

Not even "potentially", it actually is in some places.


From arigo at tunes.org  Tue Dec 21 12:37:00 2010
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 21 Dec 2010 12:37:00 +0100
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
Message-ID: <AANLkTi=4-R6JQ69tG7_yYfyEJaEbVXrsHDmgxKq77OoF@mail.gmail.com>

Hi,

On Tue, Dec 21, 2010 at 9:58 AM, Arnd Rechenburg
<Arnd.Rechenburg at tomtom.com> wrote:
> In my code I need something like
>
> long.__itemsize__
>
> in Python.

I suppose the question is "why"?  This is supposed to be the size in
bytes occupied by one "element" of the type, and one element of "long"
happens to be 15 bits, so that's why long.__itemsize__ is 2.  It has
no other meaning.  If you are interested in an estimate of how many
bytes some actual object takes, we have some functions in the 'gc'
module, but even that is incomplete -- there are cases were it's hard
to come up with a definitive answer, like objects based on "mapdicts".


A bient?t,

Armin.


From renesd at gmail.com  Tue Dec 21 13:24:37 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Tue, 21 Dec 2010 12:24:37 +0000
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <AANLkTi=4-R6JQ69tG7_yYfyEJaEbVXrsHDmgxKq77OoF@mail.gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
	<AANLkTi=4-R6JQ69tG7_yYfyEJaEbVXrsHDmgxKq77OoF@mail.gmail.com>
Message-ID: <AANLkTimAU1Zot4VcKuLcey3dpJfxFbpSYHkk2vz73CTN@mail.gmail.com>

sys.getsizeof(obj)


From arigo at tunes.org  Tue Dec 21 13:29:28 2010
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 21 Dec 2010 13:29:28 +0100
Subject: [pypy-dev] long.__itemsize__
In-Reply-To: <AANLkTimAU1Zot4VcKuLcey3dpJfxFbpSYHkk2vz73CTN@mail.gmail.com>
References: <FA92FB0BD1FD6349BD520B84A62DFAC703F7619B@BESRVWC-EXC01.ta.global>
	<AANLkTi=4-R6JQ69tG7_yYfyEJaEbVXrsHDmgxKq77OoF@mail.gmail.com>
	<AANLkTimAU1Zot4VcKuLcey3dpJfxFbpSYHkk2vz73CTN@mail.gmail.com>
Message-ID: <AANLkTinsJToxdJV+gbxd3ewE+a6EorH6RMuRX-iAwANe@mail.gmail.com>

Hi Ren?,

On Tue, Dec 21, 2010 at 1:24 PM, Ren? Dudfield <renesd at gmail.com> wrote:
> sys.getsizeof(obj)

Ah, thank you, didn't know about it.  It's a 2.6 feature only though.


A bient?t,

Armin.


From fuzzyman at gmail.com  Tue Dec 21 13:59:02 2010
From: fuzzyman at gmail.com (Michael Foord)
Date: Tue, 21 Dec 2010 12:59:02 +0000
Subject: [pypy-dev] Fwd: [IronPython] SciPy
In-Reply-To: <AANLkTi=Ua==-0BXFSd9jK-kw9nXV7KxP2qTP8HDcyhEf@mail.gmail.com>
References: <8036074B72EB5B49B0BC19ED6F4E402047CAEB@ws-be-exchange.completegenomics.com>
	<6C7ABA8B4E309440B857D74348836F2E0108A168@TK5EX14MBXC135.redmond.corp.microsoft.com>
	<AANLkTi=Ua==-0BXFSd9jK-kw9nXV7KxP2qTP8HDcyhEf@mail.gmail.com>
Message-ID: <AANLkTikPz3GOYgZV=r17MUCFjJH3A5qjBFj8HDaZcfm6@mail.gmail.com>

I thought this email could be relevant for those interested in SciPy / Numpy
on pypy. With enthought implementing a smaller core and using compatibility
layers for alternative platforms, it would seem to be a good basis for a
pypy port.

---------- Forwarded message ----------
From: Jason McCampbell <jmccampbell at enthought.com>
Date: 20 December 2010 15:13
Subject: Re: [IronPython] SciPy
To: Discussion of IronPython <users at lists.ironpython.com>


Hi Mark,

As Dino mentioned we (Enthought) are working on refactoring Numpy into a
pure "C" core with CPython and IronPython interface layers.  This is largely
complete and available at github (https://github.com/numpy/numpy-refactor),
though the core layer is largely undocumented thus far.  This is the
multi-dimensional array.

SciPy is in progress and we are updating it to work with the refactored
numpy core and to add an IronPython interface.

I assume you are looking for IronPython interfaces to SciPy as opposed to a
C interface, correct?

Regards,
Jason


On Thu, Dec 16, 2010 at 1:56 PM, Dino Viehland <dinov at microsoft.com> wrote:

>  Enthought has been working on getting numpy/scipy ported over to work w/
> IronPython.  I believe numpy is working but I?m not sure of how far along
> SciPy is.  There?s a separate mailing list for this at:
>
>
>
> https://mail.enthought.com/mailman/listinfo/scipy4dotnet
>
>
>
> It?s very low traffic ? it?s usually just working through issues Enthought
> has run into and either workarounds or suggested changes to IronPython.  I?d
> suggest sending a mail there ? they might have something you can try.
>
>
>
>
>
> *From:* users-bounces at lists.ironpython.com [mailto:
> users-bounces at lists.ironpython.com] *On Behalf Of *Mark Senko
> *Sent:* Thursday, December 16, 2010 11:49 AM
> *To:* users at lists.ironpython.com
> *Subject:* [IronPython] SciPy
>
>
>
> I?ve been searching for the current state of support for ?C? based
> libraries, specifically SciPy (I?m just looking for a decent numerical
> analysis package).  The responses I?ve seen on various websites are somewhat
> dated.
>
> What is the latest status, or is there no effort towards accommodating the
> C API? Is IronClad still the best option? Any info, suggestions and warnings
> would be appreciated before I start to invest a lot of time into installing
> and learning these packages.
>
>
>
> *Mark Senko*
>
> Complete Genomics, Inc.
>
> 2071 Stierlin Court
>
> Mountain View, CA 94043
>
>
>
>
>
>
>
> ____
>
>
>
> The contents of this e-mail and any attachments are confidential and only for
>
> use by the intended recipient. Any unauthorized use, distribution or copying
>
> of this message is strictly prohibited. If you are not the intended recipient
>
> please inform the sender immediately by reply e-mail and delete this message
>
> from your system. Thank you for your co-operation.
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>

_______________________________________________
Users mailing list
Users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com




-- 
http://www.voidspace.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101221/567d69ed/attachment.html>

From garyrob at me.com  Tue Dec 21 17:15:44 2010
From: garyrob at me.com (Gary Robinson)
Date: Tue, 21 Dec 2010 11:15:44 -0500
Subject: [pypy-dev] Fwd: [IronPython] SciPy
Message-ID: <B14D3730-456C-41BB-AD9F-A1019050FBB5@me.com>

> I thought this email could be relevant for those interested in SciPy / Numpy
> on pypy. With enthought implementing a smaller core and using compatibility
> layers for alternative platforms, it would seem to be a good basis for a
> pypy port.

All I can do is give a BIG +1 to anything that can get numpy/scipy up more quickly.

PyPy is starting to give us the speed we need for statistical/scientific work on python (the speed it still lacks compared to C or Java is made up for, for my purposes, by the ease of writing python compared to those languages). The recent 64-bit functionality lets us process a lot of data. The fast-forward branch is giving us the multiprocessing we need. (I recognize that there are other solutions, but for simple things we need to write quickly, the multiprocessing module is really sweet.)

The main thing missing now is numpy/scipy. The addition of that will make PyPy a huge win in the scientific community, IMO.

Anyway, I mention it in case the opinion of one person who is using Python in production for statistical processing is of interest.

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From arigo at tunes.org  Tue Dec 21 17:44:30 2010
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 21 Dec 2010 17:44:30 +0100
Subject: [pypy-dev] PyPy 1.4.1
Message-ID: <AANLkTikGa76VeouGoKjiHOr0-cjySg0-abCTeqaydX2V@mail.gmail.com>

Hi all,

Here is PyPy 1.4.1 :-)

Note that the Windows and the Mac OS X 64 builds are still missing; we
are waiting for contributions here (amaury, mvt? :-).

A bient?t,
Armin.

===============================
PyPy 1.4.1
===============================

We're pleased to announce the 1.4.1 release of PyPy.  This
release consolidates all the bug fixes that occurred since the
previous release.  To everyone that took the trouble to report
them, we want to say thank you.

    http://pypy.org/download.html

What is PyPy
============

PyPy is a very compliant Python interpreter, almost a drop-in
replacement for CPython.  Note that it still only emulates Python
2.5 by default; the ``fast-forward`` branch with Python 2.7
support is slowly getting ready but will only be integrated in
the next release.

In two words, the advantage of trying out PyPy instead of CPython
(the default implementation of Python) is, for now, the
performance.  Not all programs are faster in PyPy, but we are
confident that any CPU-intensive task will be much faster, at
least if it runs for long enough (the JIT has a slow warm-up
phase, which can take several seconds or even one minute on the
largest programs).

Note again that we do support compiling and using C extension
modules from CPython (``pypy setup.py install``).  However, this
is still an alpha feature, and the most complex modules typically
fail for various reasons; others work (e.g. ``PIL``) but take a
serious performance hit.  Also, for Mac OS X see below.

Please note also that PyPy's performance was optimized almost
exclusively on Linux.  It seems from some reports that on Windows
as well as Mac OS X (probably for different reasons) the
performance might be lower.  We did not investigate much so far.


More highlights
===============

* We migrated to Mercurial (thanks to Ronny Pfannschmidt and
  Antonio Cuni) for the effort) and moved to bitbucket.  The new
  command to check out a copy of PyPy is::

        hg clone http://bitbucket.org/pypy/pypy

* In long-running processes, the assembler generated by old
  JIT-compilations is now freed.  There should be no more leak,
  however long the process runs.

* Improve a lot the performance of the ``binascii`` module, and
  of ``hashlib.md5`` and ``hashlib.sha``.

* Made sys.setrecursionlimit() a no-op.  Instead, we rely purely
  on the built-in stack overflow detection mechanism, which also
  gives you a RuntimeError -- just not at some exact recursion
  level.

* Fix argument processing (now e.g. ``pypy -OScpass`` works like
  it does on CPython --- if you have a clue what it does there
  ``:-)`` )

* cpyext on Mac OS X: it still does not seem to work.  I get
  systematically a segfault in dlopen().  Contributions welcome.

* Fix two corner cases in the GC (one in minimark, one in
  asmgcc+JIT).  This notably prevented "pypy translate.py -Ojit"
  from working on Windows, leading to crashes.

* Fixed a corner case in the JIT's optimizer, leading to "Fatal
  RPython error: AssertionError".

* Added some missing built-in functions into the 'os' module.

* Fix ctypes (it was not propagating keepalive information from
  c_void_p).


Cheers,

Armin Rigo, for the rest of the team


From dimaqq at gmail.com  Tue Dec 21 19:07:14 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Tue, 21 Dec 2010 11:07:14 -0700
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
Message-ID: <AANLkTik8dLiPCVuXSgW5n0qD4Msk55MqERzpdk-k3uzQ@mail.gmail.com>

Yes it does grind ff 4b7 to an almost halt, little cpu usage,
reasonable memory size and constant disk activity for several minutes,
very weird...
So far, the difference between browsers is so large that I doubt it's
dependent on data.
Also it seems to tirgger as I remove columns progressively, thus new
zero values should not appear, right?

I'll invesitage some more...

On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
> Hi Dima,
>
>> another temp problem with speed pypy is that it's terrubly slow in ff
>> beta. it also occasionally grinds in stable ff and opera, but I guess
>> this can be forgiven for the sake of simplicity / developer effort.
>
> Well, speed.pypy is actually fast in all modern browsers. The problem
> you are referring to is probably caused by a bug in the javascript
> plotting library (jqPplot) that is triggered in the comparison view
> when there are some results with 0 values. It only appears for some
> plot types, but it is very annoying because it grinds the browser to a
> halt like you say. Is that what you meant?
>
> We are looking into it, and will fix that library if necessary.
>
> Cheers,
> Miquel
>
>
> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>> On 20 December 2010 19:21, William ML Leslie
>> <william.leslie.ttg at gmail.com> wrote:
>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>> More visibility for performance achievements would do
>>>> good too.
>>>
>>> Where are pypy's performance achievements *not* visible, but should be?
>>
>> for example http://shootout.alioth.debian.org/
>> doesn't say which pypy version is used, what options, doesn't have
>> performance figures for multithreaded/multicore
>>
>> also benchmarks are kinda small, most of them are not docuemented, and
>> I couldn't find any info if the same python code was used for cpython
>> and pypy (both shootout and speed pypy) or interpreter-specific
>> verions were used, that is each tweaked for best performance given the
>> known tradeoffs for each implementation.further the most standard
>> benchmarks, pystone, etc. completely ignore the fact that real world
>> programs are large and only a few small paths are execured often.
>>
>> another temp problem with speed pypy is that it's terrubly slow in ff
>> beta. it also occasionally grinds in stable ff and opera, but I guess
>> this can be forgiven for the sake of simplicity / developer effort.
>>
>> if you google for 'python performance' you don't get a single link to
>> pypy on the first page, as a matter of fact, codespeak is poorly
>> indexed, it took me quite some time to get some of my questions
>> answered with a search. also if you look up 'pypy gc' you get a page
>> on codespeak, but to interpret what the data actually means is so far
>> beyond me.
>>
>> a good overview is found in the mainling list
>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>> slowspitfire and spambayes bits are outdated by now.
>>
>> the definitive good thing about pypy is that it's much easier to find
>> out about its inner workings than that of cpython!
>>
>> hopefully a bit more of end-user stuff get known.
>> let's call it pypy public outreach (feature request)
>>
>>>
>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>> hard would it be to use same techniques and most of backends for some
>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>
>>> You know that pypy already has implementations of other languages,
>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>> integrated with the translated pypy-c, but they show that it is not
>>> too difficult to write a runtime for any dynamic language you choose.
>>
>> Oh I didn't know there were so many, and I mistakenly though that js
>> was a target, not implmented langauge. In any case I read somewhere
>> that js support was retired...
>>
>>>
>>>> And how hard
>>>> would it be to marry two dynamic languages, so that modules from one
>>>> could be used in the other? Or that modules written in rpython could
>>>> be used in several langs?
>>>
>>> It's in the "interesting problems" bucket, and the effort required
>>> depends on the level of integration between languages you want. ?There
>>> are several projects already attempting to do decent integration
>>> between several languages, besides the approach used on the JVM, there
>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>> worth waiting to see how these different projects pan out, before
>>> spending a bunch of effort just to be an also-ran in the
>>> multi-language runtime market.
>>>
>>> However, implementing more languages in rpython has the side-effect of
>>> propagating the l * o * p problem: it introduces more and more
>>> implementations that then have to be maintained, so good
>>> cross-language integration probably belongs /outside/ pypy itself, so
>>> existing runtimes can hook into it.
>>
>> Makes perfect sense, after all any given other language hardly has the
>> same data model as python.
>>
>>>
>>> But it would be an interesting experiment (to marry the various
>>> interpreters pypy ships with), if you wanted to try it.
>>>
>>> My two cents.
>>>
>>> --
>>> William Leslie
>>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>


From dmalcolm at redhat.com  Tue Dec 21 19:54:36 2010
From: dmalcolm at redhat.com (David Malcolm)
Date: Tue, 21 Dec 2010 13:54:36 -0500
Subject: [pypy-dev] [PATCH] Improving readability of generated .c code
In-Reply-To: <4D077BB5.7030102@gmx.de>
References: <1292288224.24137.22.camel@radiator.bos.redhat.com>
	<4D077BB5.7030102@gmx.de>
Message-ID: <1292957676.20268.110.camel@surprise>

On Tue, 2010-12-14 at 15:14 +0100, Carl Friedrich Bolz wrote:
> Hi David,
> 
> On 12/14/2010 01:57 AM, David Malcolm wrote:

Thanks for looking at this; sorry for the belated reply.

I'm attaching the latest work-in-progress version of the patch,
addressing some of the areas discussed.

(snip)

> > Hopefully this makes the generated .c much more readable.
> 
> I like the general approach, though I have the vague fear that it will 
> make the generated C code much bigger. Could you check that this is not 
> the case?

The .c code seems to get roughly 15% bigger (which I don't think is an
issue), but the translation may be getting significantly longer (40%
more time; it's a real pain to benchmark this, though).

I'm also not getting as many comments as I'd like for inlining (see
below), and I expect the code to grow when I fix that.

Method:
  Control:
  - clean pypy hg checkout of 40149:cd083843b67a 
  - cd pypy/translator/goal
  - python translate.py  (i.e. without any flags)
  - wait for the build to complete
  - locate the generated code in the "testing_1" subdirectory:
    - invoke David A. Wheeler's "SLOCCount" tool on the "testing_1" dir
(http://www.dwheeler.com/sloccount/)
    - run "du -h" on the "testing_1" dir

  Experiment:
  - as above, but with the attached changes to the source tree

Actually, the first time I built the clean version, I was hacking on the
debug tree, and lost my copy of the relevant /tmp/usession-N directory,
so I set PYPY_USESSION_DIR=/tmp/clean-build and redid it.  (I was also
checking mail, IRC, and other stuff on the machine, but I don't think
this significantly affects the timings).

Size of generated source::
  Output of "wc *.c"::
    (trimmed to final summary, giving newline, word, and byte counts)
    Unpatched::
      5166080  15803859 186007458 total

    With the patch::
      5544105  18458780 213205652 total

  Disk usage::
    This is:
       du -h -c *.c
    omitting the individual counts

    Unpatched::
      178M total

    With the patch::
      204M total

  "sloccount"::
    Unpatched::
      SLOC    Directory       SLOC-by-Language (Sorted)
      4728922 testing_1       ansic=4728922
  
    With the patch::
      SLOC    Directory       SLOC-by-Language (Sorted)
      4730181 testing_1       ansic=4730181

I haven't measured memory usage during conversion.

The above looks fine to me (I'll happily take a 15% increase in on-disk
size of files that are used at runtime only for debugging, in exchange
for easier debugging).


However, conversion time seems to increase roughly 40%.  

Unpatched:
[Timer] Timings:
[Timer] annotate                       --- 1180.5 s
[Timer] rtype_lltype                   ---  730.7 s
[Timer] backendopt_lltype              ---  468.5 s
[Timer] stackcheckinsertion_lltype     ---   68.9 s
[Timer] database_c                     ---  546.4 s
[Timer] source_c                       --- 1323.5 s
[Timer] compile_c                      ---  347.4 s
[Timer] ===========================================
[Timer] Total:                         --- 4665.9 s


Patched:
[Timer] Timings:
[Timer] annotate                       --- 1428.2 s
[Timer] rtype_lltype                   ---  913.6 s
[Timer] backendopt_lltype              ---  831.9 s
[Timer] stackcheckinsertion_lltype     ---  106.6 s
[Timer] database_c                     ---  991.1 s
[Timer] source_c                       --- 2385.8 s
[Timer] compile_c                      ---  400.2 s
[Timer] ===========================================
[Timer] Total:                         --- 7057.3 s



> > (I hope to
> > work on the names of locals also; ideally they ought to embed the python
> > indentifier)
> 
> This should already work in theory, see e.g. "l_cls_0" above, which 
> embeds the Python variable name "cls". I guess that the information is 
> simply lost in some of the transformation steps. If you feel like 
> hunting down where, you can probably get better naming in some cases.

I think I was wrong about this: the variables that are getting
uninformative names are the temporaries; .c locals that "are" RPython
locals are being named informatively.

> 
> > This is on a Fedora 13 x86_64 box, with cpython 2.6.5
> >
> > Caveats:
> >    - this is my first patch to PyPy (please be gentle!): I hope I've
> > correctly understood the various levels, but I suspect I may still be
> > missing things at the conceptual level
> 
> I think you are doing great. You might have to grep around a bit for 
> more places that create SpaceOperations, or did you look at all of them?

There are numerous places where ops are created, (in llops.genops iirc),
and somewhere in that process, many of the resulting operations are only
getting a partial "path" of CodeLoc instances (see below).  I plan to
look into that next.


> >    - haven't completed the full unit tests yet.
> 
> This is the biggest problem of the patch, of course. I think that you 
> should at least try to write tests for each of the places where you 
> propagate the offset down one level. In addition, you probably need an 
> integration test where you check the generated C code for the presence 
> of the correct comments.

The latest version of the patch adds two integration tests into
  pypy/translator/c/test/test_genc.py

I haven't yet added unit tests for the location propagation (see below).


> >    - the patch attempts to propagate the "offset" of flowspace
> > SpaceOperation instances in a couple of places where the offset was
> > being discarded:
> >        - in BaseInliner.copy_operation
> 
> This is wrong. If you inline one function into another, it doesn't make 
> sense to put the offset of the inner function into the outer function. 
> To solve this problem, you have two options:
>   1) when inlining, set the offset to -1, and just don't get source code 
> then
>   2) make every *operation* contain a reference to the func it comes 
> from. The memory implications of this should be checked, but are 
> probably not too bad.

What I've done is (2): introduce a CodeLoc class representing a (code,
offset) pair, and to replace the "offset" of a flowspace SpaceOperation
with a new OperationLoc class, which consists of a list of 1 or more
CodeLoc instances.  In the simple case we have a single CodeLoc, but in
the case of operations within an inlined function we have the CodeLoc of
the callsite, followed by that of the operation within the inlined body.
For the case of nested inlining, I'm assuming we could have an arbitrary
long list of CodeLoc instances describing the OperationLoc.  

Having said that, within inlining most operations seem not to get part
of the "path", and I'm using None for these.  This is probably the area
most needing further unit tests.


Other changes in the patch:
  - I've attempted to reorder blocks within the generated C to reflect
the source-level ordering of operations within the RPython.  This breaks
down somewhat in the presence of inlined ops with only partial paths,
but it makes a best attempt.
  - I had a go at renaming block labels in the C, so that e.g. a return
block might be "return_block3", and a raise block might be
"raise_block42".
  - I'm now cleaning out C-style comments from the .py code when writing
them into comments in the generated C code, to avoid syntax errors
  - two new self tests, as mentioned above.  I made interactive
Translation.source_c() return the .c filename to help write the tests


(...snip...)


> >    - I haven't tried doing this on a full build of the interpreter
> > (having highly readable generated .c for this is my objective [1]).
> 
> I fear it will never be "highly readable", but I think your work 
> improves things already.

Thanks!


> > One other idea I had was to sort the blocks in the function by the
> > bytecode offset; currently the c blocks seem to "jump around" a fair bit
> > relative to the corresponding rpython code.  Has any tuning been done on
> > the ordering of the blocks within the generated .c code?  (or is it
> > assumed that the .c compiler will "flatten" these, for the case when a
> > node has a single successor node?)
> 
> The C compiler should not care about the order of blocks. If it does, I 
> am prepared to call it insane.

As mentioned above, I attempted this.


(..snip...)

> > The C code doesn't look anything like the .py code that my patch is
> > inserting in the above.  (My current guess is that I need to be smarter
> > about inlined operations, though that's a guess at this stage)
> 
> Sounds like a correct guess, malloc is transformed into something else, 
> and the offset of the inlined operations is basically just random, if 
> you interpret it in the context of the outer function.

I'd hoped with the OperationLoc/CodeLoc changes to be able to propagate
the full hierarchy of RPython source location at an inlined callsite,
but I'm not getting that yet.  I plan to look at that next.

However, the comments do at least now reflect true locations throughout.


One other idea that occurred to me: currently in
pypy/translator/c/genc.py the code spits all of the functions into a
series of numbered implement.c files.

I thought it might be nice to split them up somehow by namespace, so for
example, all "PyUnicode_*" and "pypy_g_PyUnicode_*" functions could end
up in "PyUnicode.c", "PyUnicode_1.c", etc, and implement*.c could then
only contain the ones that don't match such a pattern.

Does this sound reasonable, and any pointers on minimal ways of testing
this?  (I'm guessing the unit tests for the C database are the best
guidance here).


Hope this is helpful
Dave
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-add-py-source-as-comments-to-generated-c-2010-12-21-001.patch
Type: text/x-patch
Size: 26643 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101221/8218cf33/attachment.bin>

From tobami at googlemail.com  Tue Dec 21 20:30:58 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Tue, 21 Dec 2010 20:30:58 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTik8dLiPCVuXSgW5n0qD4Msk55MqERzpdk-k3uzQ@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTik8dLiPCVuXSgW5n0qD4Msk55MqERzpdk-k3uzQ@mail.gmail.com>
Message-ID: <AANLkTikob+QeBHfmErJEbXhKnKsajkYfgUAntPBF52A6@mail.gmail.com>

It is weird that it happens as you de-select benchmarks.

Does it happen with non-beta browsers?


2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
> Yes it does grind ff 4b7 to an almost halt, little cpu usage,
> reasonable memory size and constant disk activity for several minutes,
> very weird...
> So far, the difference between browsers is so large that I doubt it's
> dependent on data.
> Also it seems to tirgger as I remove columns progressively, thus new
> zero values should not appear, right?
>
> I'll invesitage some more...
>
> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>> Hi Dima,
>>
>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>> this can be forgiven for the sake of simplicity / developer effort.
>>
>> Well, speed.pypy is actually fast in all modern browsers. The problem
>> you are referring to is probably caused by a bug in the javascript
>> plotting library (jqPplot) that is triggered in the comparison view
>> when there are some results with 0 values. It only appears for some
>> plot types, but it is very annoying because it grinds the browser to a
>> halt like you say. Is that what you meant?
>>
>> We are looking into it, and will fix that library if necessary.
>>
>> Cheers,
>> Miquel
>>
>>
>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>> On 20 December 2010 19:21, William ML Leslie
>>> <william.leslie.ttg at gmail.com> wrote:
>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>> More visibility for performance achievements would do
>>>>> good too.
>>>>
>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>
>>> for example http://shootout.alioth.debian.org/
>>> doesn't say which pypy version is used, what options, doesn't have
>>> performance figures for multithreaded/multicore
>>>
>>> also benchmarks are kinda small, most of them are not docuemented, and
>>> I couldn't find any info if the same python code was used for cpython
>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>> verions were used, that is each tweaked for best performance given the
>>> known tradeoffs for each implementation.further the most standard
>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>> programs are large and only a few small paths are execured often.
>>>
>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>> this can be forgiven for the sake of simplicity / developer effort.
>>>
>>> if you google for 'python performance' you don't get a single link to
>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>> indexed, it took me quite some time to get some of my questions
>>> answered with a search. also if you look up 'pypy gc' you get a page
>>> on codespeak, but to interpret what the data actually means is so far
>>> beyond me.
>>>
>>> a good overview is found in the mainling list
>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>> slowspitfire and spambayes bits are outdated by now.
>>>
>>> the definitive good thing about pypy is that it's much easier to find
>>> out about its inner workings than that of cpython!
>>>
>>> hopefully a bit more of end-user stuff get known.
>>> let's call it pypy public outreach (feature request)
>>>
>>>>
>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>> hard would it be to use same techniques and most of backends for some
>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>
>>>> You know that pypy already has implementations of other languages,
>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>> integrated with the translated pypy-c, but they show that it is not
>>>> too difficult to write a runtime for any dynamic language you choose.
>>>
>>> Oh I didn't know there were so many, and I mistakenly though that js
>>> was a target, not implmented langauge. In any case I read somewhere
>>> that js support was retired...
>>>
>>>>
>>>>> And how hard
>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>> could be used in the other? Or that modules written in rpython could
>>>>> be used in several langs?
>>>>
>>>> It's in the "interesting problems" bucket, and the effort required
>>>> depends on the level of integration between languages you want. ?There
>>>> are several projects already attempting to do decent integration
>>>> between several languages, besides the approach used on the JVM, there
>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>>> worth waiting to see how these different projects pan out, before
>>>> spending a bunch of effort just to be an also-ran in the
>>>> multi-language runtime market.
>>>>
>>>> However, implementing more languages in rpython has the side-effect of
>>>> propagating the l * o * p problem: it introduces more and more
>>>> implementations that then have to be maintained, so good
>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>> existing runtimes can hook into it.
>>>
>>> Makes perfect sense, after all any given other language hardly has the
>>> same data model as python.
>>>
>>>>
>>>> But it would be an interesting experiment (to marry the various
>>>> interpreters pypy ships with), if you wanted to try it.
>>>>
>>>> My two cents.
>>>>
>>>> --
>>>> William Leslie
>>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>


From dimaqq at gmail.com  Tue Dec 21 22:22:15 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Tue, 21 Dec 2010 14:22:15 -0700
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTikob+QeBHfmErJEbXhKnKsajkYfgUAntPBF52A6@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTik8dLiPCVuXSgW5n0qD4Msk55MqERzpdk-k3uzQ@mail.gmail.com>
	<AANLkTikob+QeBHfmErJEbXhKnKsajkYfgUAntPBF52A6@mail.gmail.com>
Message-ID: <AANLkTi=BokNmsYuZ-uMRaRDYgW=7CQasVOp-rQWPSa40@mail.gmail.com>

yes, same problem, but the grind is not so aweful, surviveable even,
tried with opera and normal ff

On 21 December 2010 12:30, Miquel Torres <tobami at googlemail.com> wrote:
> It is weird that it happens as you de-select benchmarks.
>
> Does it happen with non-beta browsers?
>
>
> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>> Yes it does grind ff 4b7 to an almost halt, little cpu usage,
>> reasonable memory size and constant disk activity for several minutes,
>> very weird...
>> So far, the difference between browsers is so large that I doubt it's
>> dependent on data.
>> Also it seems to tirgger as I remove columns progressively, thus new
>> zero values should not appear, right?
>>
>> I'll invesitage some more...
>>
>> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>>> Hi Dima,
>>>
>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>
>>> Well, speed.pypy is actually fast in all modern browsers. The problem
>>> you are referring to is probably caused by a bug in the javascript
>>> plotting library (jqPplot) that is triggered in the comparison view
>>> when there are some results with 0 values. It only appears for some
>>> plot types, but it is very annoying because it grinds the browser to a
>>> halt like you say. Is that what you meant?
>>>
>>> We are looking into it, and will fix that library if necessary.
>>>
>>> Cheers,
>>> Miquel
>>>
>>>
>>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>>> On 20 December 2010 19:21, William ML Leslie
>>>> <william.leslie.ttg at gmail.com> wrote:
>>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>>> More visibility for performance achievements would do
>>>>>> good too.
>>>>>
>>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>>
>>>> for example http://shootout.alioth.debian.org/
>>>> doesn't say which pypy version is used, what options, doesn't have
>>>> performance figures for multithreaded/multicore
>>>>
>>>> also benchmarks are kinda small, most of them are not docuemented, and
>>>> I couldn't find any info if the same python code was used for cpython
>>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>>> verions were used, that is each tweaked for best performance given the
>>>> known tradeoffs for each implementation.further the most standard
>>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>>> programs are large and only a few small paths are execured often.
>>>>
>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>
>>>> if you google for 'python performance' you don't get a single link to
>>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>>> indexed, it took me quite some time to get some of my questions
>>>> answered with a search. also if you look up 'pypy gc' you get a page
>>>> on codespeak, but to interpret what the data actually means is so far
>>>> beyond me.
>>>>
>>>> a good overview is found in the mainling list
>>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>>> slowspitfire and spambayes bits are outdated by now.
>>>>
>>>> the definitive good thing about pypy is that it's much easier to find
>>>> out about its inner workings than that of cpython!
>>>>
>>>> hopefully a bit more of end-user stuff get known.
>>>> let's call it pypy public outreach (feature request)
>>>>
>>>>>
>>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>>> hard would it be to use same techniques and most of backends for some
>>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>>
>>>>> You know that pypy already has implementations of other languages,
>>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>>> integrated with the translated pypy-c, but they show that it is not
>>>>> too difficult to write a runtime for any dynamic language you choose.
>>>>
>>>> Oh I didn't know there were so many, and I mistakenly though that js
>>>> was a target, not implmented langauge. In any case I read somewhere
>>>> that js support was retired...
>>>>
>>>>>
>>>>>> And how hard
>>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>>> could be used in the other? Or that modules written in rpython could
>>>>>> be used in several langs?
>>>>>
>>>>> It's in the "interesting problems" bucket, and the effort required
>>>>> depends on the level of integration between languages you want. ?There
>>>>> are several projects already attempting to do decent integration
>>>>> between several languages, besides the approach used on the JVM, there
>>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>>>> worth waiting to see how these different projects pan out, before
>>>>> spending a bunch of effort just to be an also-ran in the
>>>>> multi-language runtime market.
>>>>>
>>>>> However, implementing more languages in rpython has the side-effect of
>>>>> propagating the l * o * p problem: it introduces more and more
>>>>> implementations that then have to be maintained, so good
>>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>>> existing runtimes can hook into it.
>>>>
>>>> Makes perfect sense, after all any given other language hardly has the
>>>> same data model as python.
>>>>
>>>>>
>>>>> But it would be an interesting experiment (to marry the various
>>>>> interpreters pypy ships with), if you wanted to try it.
>>>>>
>>>>> My two cents.
>>>>>
>>>>> --
>>>>> William Leslie
>>>>>
>>>> _______________________________________________
>>>> pypy-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>
>


From dimaqq at gmail.com  Wed Dec 22 06:51:39 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Tue, 21 Dec 2010 22:51:39 -0700
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
Message-ID: <AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>

I sort of figured it out, although I don't have a ready solution.

This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8.
Basically memory consumption of the plot appears to be proportional to
plot area.
Normal plot that you get by default at
http://speed.pypy.org/comparison/ costs about 100M of browser memory
consumption:
opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050;
opera  80M, stable ff 55M, beta ff 70M at window size 1024x600;
Switching to "horizontal" produces a tall plot of same width and costs
about 300~700M of browser memory:
opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050;
opera 350M, stable ff 370M, beta ff 370M at window size 1024x600;

Suprisingly window size only matters while javascript produces the
plot, and not when window is resized, even though plot it resized with
the window correctly.

This alone is pretty heavy, but doesn't grind the browser.
What really grinds is that every time you change a tickbox on the
left, a plot is redrawn and another 200M of browser memory is wasted.
This is not double buffering, as next change adds yet another 200M or
so and so on, it appears that either js doesn't free something, or
browser caches or saves the previous page state.

As memory consumption grows, at some point browser hits the wall,
causes heavy swapping for some time, and I think garbage collection,
because practical (but not virtual) memory usage first drops to
something like 20~50M and then returns to "normal" 300M.
opera ~30 seconds, stable ff about a minute, beta ff several minutes
(total system mem 1G, cpu Atom @1.6GHz)

Perhaps OS also plays a role in the grind, as it is clearly swapping
and swaps out too much? or triggers gc too late and gc has to pull the
pages back from disk to perform collection?

ie8 doesn't use that much memory, as a matter of fact memory
consumption starts little (40M) and changes very little (only +10M) if
you go to horizonatal view; the price is very slow rendering, more
than 10 seconds per column change.

I'll post this on firefox bugzilla too, let's see if someone has a solution.

Meanwhile perhaps pypy speed center could start with a smaller plot
area (or fewer columns as that makes horizontal plot smaller) to
accomodate varying hardware and system mem usage that users might
have?
The simplest would be a warning next to "horizontal" checkbox.

On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
> Hi Dima,
>
>> another temp problem with speed pypy is that it's terrubly slow in ff
>> beta. it also occasionally grinds in stable ff and opera, but I guess
>> this can be forgiven for the sake of simplicity / developer effort.
>
> Well, speed.pypy is actually fast in all modern browsers. The problem
> you are referring to is probably caused by a bug in the javascript
> plotting library (jqPplot) that is triggered in the comparison view
> when there are some results with 0 values. It only appears for some
> plot types, but it is very annoying because it grinds the browser to a
> halt like you say. Is that what you meant?
>
> We are looking into it, and will fix that library if necessary.
>
> Cheers,
> Miquel
>
>
> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>> On 20 December 2010 19:21, William ML Leslie
>> <william.leslie.ttg at gmail.com> wrote:
>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>> More visibility for performance achievements would do
>>>> good too.
>>>
>>> Where are pypy's performance achievements *not* visible, but should be?
>>
>> for example http://shootout.alioth.debian.org/
>> doesn't say which pypy version is used, what options, doesn't have
>> performance figures for multithreaded/multicore
>>
>> also benchmarks are kinda small, most of them are not docuemented, and
>> I couldn't find any info if the same python code was used for cpython
>> and pypy (both shootout and speed pypy) or interpreter-specific
>> verions were used, that is each tweaked for best performance given the
>> known tradeoffs for each implementation.further the most standard
>> benchmarks, pystone, etc. completely ignore the fact that real world
>> programs are large and only a few small paths are execured often.
>>
>> another temp problem with speed pypy is that it's terrubly slow in ff
>> beta. it also occasionally grinds in stable ff and opera, but I guess
>> this can be forgiven for the sake of simplicity / developer effort.
>>
>> if you google for 'python performance' you don't get a single link to
>> pypy on the first page, as a matter of fact, codespeak is poorly
>> indexed, it took me quite some time to get some of my questions
>> answered with a search. also if you look up 'pypy gc' you get a page
>> on codespeak, but to interpret what the data actually means is so far
>> beyond me.
>>
>> a good overview is found in the mainling list
>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>> slowspitfire and spambayes bits are outdated by now.
>>
>> the definitive good thing about pypy is that it's much easier to find
>> out about its inner workings than that of cpython!
>>
>> hopefully a bit more of end-user stuff get known.
>> let's call it pypy public outreach (feature request)
>>
>>>
>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>> hard would it be to use same techniques and most of backends for some
>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>
>>> You know that pypy already has implementations of other languages,
>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>> integrated with the translated pypy-c, but they show that it is not
>>> too difficult to write a runtime for any dynamic language you choose.
>>
>> Oh I didn't know there were so many, and I mistakenly though that js
>> was a target, not implmented langauge. In any case I read somewhere
>> that js support was retired...
>>
>>>
>>>> And how hard
>>>> would it be to marry two dynamic languages, so that modules from one
>>>> could be used in the other? Or that modules written in rpython could
>>>> be used in several langs?
>>>
>>> It's in the "interesting problems" bucket, and the effort required
>>> depends on the level of integration between languages you want. ?There
>>> are several projects already attempting to do decent integration
>>> between several languages, besides the approach used on the JVM, there
>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>> worth waiting to see how these different projects pan out, before
>>> spending a bunch of effort just to be an also-ran in the
>>> multi-language runtime market.
>>>
>>> However, implementing more languages in rpython has the side-effect of
>>> propagating the l * o * p problem: it introduces more and more
>>> implementations that then have to be maintained, so good
>>> cross-language integration probably belongs /outside/ pypy itself, so
>>> existing runtimes can hook into it.
>>
>> Makes perfect sense, after all any given other language hardly has the
>> same data model as python.
>>
>>>
>>> But it would be an interesting experiment (to marry the various
>>> interpreters pypy ships with), if you wanted to try it.
>>>
>>> My two cents.
>>>
>>> --
>>> William Leslie
>>>
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>


From fijall at gmail.com  Wed Dec 22 07:51:57 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 22 Dec 2010 08:51:57 +0200
Subject: [pypy-dev] Fwd: [IronPython] SciPy
In-Reply-To: <B14D3730-456C-41BB-AD9F-A1019050FBB5@me.com>
References: <B14D3730-456C-41BB-AD9F-A1019050FBB5@me.com>
Message-ID: <AANLkTikjbEVmec4PLndP1J4n5Sy7KRpMkG0-Fk6izpNT@mail.gmail.com>

Oh wow, that's really cool.

On Tue, Dec 21, 2010 at 6:15 PM, Gary Robinson <garyrob at me.com> wrote:
>> I thought this email could be relevant for those interested in SciPy / Numpy
>> on pypy. With enthought implementing a smaller core and using compatibility
>> layers for alternative platforms, it would seem to be a good basis for a
>> pypy port.
>
> All I can do is give a BIG +1 to anything that can get numpy/scipy up more quickly.
>
> PyPy is starting to give us the speed we need for statistical/scientific work on python (the speed it still lacks compared to C or Java is made up for, for my purposes, by the ease of writing python compared to those languages). The recent 64-bit functionality lets us process a lot of data. The fast-forward branch is giving us the multiprocessing we need. (I recognize that there are other solutions, but for simple things we need to write quickly, the multiprocessing module is really sweet.)
>
> The main thing missing now is numpy/scipy. The addition of that will make PyPy a huge win in the scientific community, IMO.
>
> Anyway, I mention it in case the opinion of one person who is using Python in production for statistical processing is of interest.
>
> --
>
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog: ? ?http://www.garyrobinson.net
>
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From tobami at googlemail.com  Wed Dec 22 09:24:28 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 22 Dec 2010 09:24:28 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
Message-ID: <AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>

Mmm, yes, it appears that the memory is not properly freed. I can try
experimenting by destroying the dom element and recreating it each
time that the plot changes...

Btw., instead of continuing here "polluting" the pypy-dev mailing
list, we can move to http://groups.google.com/group/codespeed if
needed.

Thanks for reporting this.

Cheers,
Miquel


2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
> I sort of figured it out, although I don't have a ready solution.
>
> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8.
> Basically memory consumption of the plot appears to be proportional to
> plot area.
> Normal plot that you get by default at
> http://speed.pypy.org/comparison/ costs about 100M of browser memory
> consumption:
> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050;
> opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600;
> Switching to "horizontal" produces a tall plot of same width and costs
> about 300~700M of browser memory:
> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050;
> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600;
>
> Suprisingly window size only matters while javascript produces the
> plot, and not when window is resized, even though plot it resized with
> the window correctly.
>
> This alone is pretty heavy, but doesn't grind the browser.
> What really grinds is that every time you change a tickbox on the
> left, a plot is redrawn and another 200M of browser memory is wasted.
> This is not double buffering, as next change adds yet another 200M or
> so and so on, it appears that either js doesn't free something, or
> browser caches or saves the previous page state.
>
> As memory consumption grows, at some point browser hits the wall,
> causes heavy swapping for some time, and I think garbage collection,
> because practical (but not virtual) memory usage first drops to
> something like 20~50M and then returns to "normal" 300M.
> opera ~30 seconds, stable ff about a minute, beta ff several minutes
> (total system mem 1G, cpu Atom @1.6GHz)
>
> Perhaps OS also plays a role in the grind, as it is clearly swapping
> and swaps out too much? or triggers gc too late and gc has to pull the
> pages back from disk to perform collection?
>
> ie8 doesn't use that much memory, as a matter of fact memory
> consumption starts little (40M) and changes very little (only +10M) if
> you go to horizonatal view; the price is very slow rendering, more
> than 10 seconds per column change.
>
> I'll post this on firefox bugzilla too, let's see if someone has a solution.
>
> Meanwhile perhaps pypy speed center could start with a smaller plot
> area (or fewer columns as that makes horizontal plot smaller) to
> accomodate varying hardware and system mem usage that users might
> have?
> The simplest would be a warning next to "horizontal" checkbox.
>
> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>> Hi Dima,
>>
>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>> this can be forgiven for the sake of simplicity / developer effort.
>>
>> Well, speed.pypy is actually fast in all modern browsers. The problem
>> you are referring to is probably caused by a bug in the javascript
>> plotting library (jqPplot) that is triggered in the comparison view
>> when there are some results with 0 values. It only appears for some
>> plot types, but it is very annoying because it grinds the browser to a
>> halt like you say. Is that what you meant?
>>
>> We are looking into it, and will fix that library if necessary.
>>
>> Cheers,
>> Miquel
>>
>>
>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>> On 20 December 2010 19:21, William ML Leslie
>>> <william.leslie.ttg at gmail.com> wrote:
>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>> More visibility for performance achievements would do
>>>>> good too.
>>>>
>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>
>>> for example http://shootout.alioth.debian.org/
>>> doesn't say which pypy version is used, what options, doesn't have
>>> performance figures for multithreaded/multicore
>>>
>>> also benchmarks are kinda small, most of them are not docuemented, and
>>> I couldn't find any info if the same python code was used for cpython
>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>> verions were used, that is each tweaked for best performance given the
>>> known tradeoffs for each implementation.further the most standard
>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>> programs are large and only a few small paths are execured often.
>>>
>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>> this can be forgiven for the sake of simplicity / developer effort.
>>>
>>> if you google for 'python performance' you don't get a single link to
>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>> indexed, it took me quite some time to get some of my questions
>>> answered with a search. also if you look up 'pypy gc' you get a page
>>> on codespeak, but to interpret what the data actually means is so far
>>> beyond me.
>>>
>>> a good overview is found in the mainling list
>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>> slowspitfire and spambayes bits are outdated by now.
>>>
>>> the definitive good thing about pypy is that it's much easier to find
>>> out about its inner workings than that of cpython!
>>>
>>> hopefully a bit more of end-user stuff get known.
>>> let's call it pypy public outreach (feature request)
>>>
>>>>
>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>> hard would it be to use same techniques and most of backends for some
>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>
>>>> You know that pypy already has implementations of other languages,
>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>> integrated with the translated pypy-c, but they show that it is not
>>>> too difficult to write a runtime for any dynamic language you choose.
>>>
>>> Oh I didn't know there were so many, and I mistakenly though that js
>>> was a target, not implmented langauge. In any case I read somewhere
>>> that js support was retired...
>>>
>>>>
>>>>> And how hard
>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>> could be used in the other? Or that modules written in rpython could
>>>>> be used in several langs?
>>>>
>>>> It's in the "interesting problems" bucket, and the effort required
>>>> depends on the level of integration between languages you want. ?There
>>>> are several projects already attempting to do decent integration
>>>> between several languages, besides the approach used on the JVM, there
>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>>> worth waiting to see how these different projects pan out, before
>>>> spending a bunch of effort just to be an also-ran in the
>>>> multi-language runtime market.
>>>>
>>>> However, implementing more languages in rpython has the side-effect of
>>>> propagating the l * o * p problem: it introduces more and more
>>>> implementations that then have to be maintained, so good
>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>> existing runtimes can hook into it.
>>>
>>> Makes perfect sense, after all any given other language hardly has the
>>> same data model as python.
>>>
>>>>
>>>> But it would be an interesting experiment (to marry the various
>>>> interpreters pypy ships with), if you wanted to try it.
>>>>
>>>> My two cents.
>>>>
>>>> --
>>>> William Leslie
>>>>
>>> _______________________________________________
>>> pypy-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>


From tobami at googlemail.com  Wed Dec 22 09:26:26 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 22 Dec 2010 09:26:26 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTinG9VyPcFk05qitjHgKCDhd2xf=UuKyg=FH4Z4g@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>
	<AANLkTinG9VyPcFk05qitjHgKCDhd2xf=UuKyg=FH4Z4g@mail.gmail.com>
Message-ID: <AANLkTinn70yxvFy3DQNpk9Hu7rDB7HoiiQK_W+7oDWPw@mail.gmail.com>

so, what about the ai and spectral-norm benchmarks. Anybody can come
up with a description for them?


2010/12/15 Paolo Giarrusso <p.giarrusso at gmail.com>:
> On Mon, Dec 13, 2010 at 09:31, Miquel Torres <tobami at googlemail.com> wrote:
>> Oh, btw., the "normalized" stacked bars now display a warning note
>> about its correctness, and how it must be viewed as giving results a
>> weighting instead of them being normalized. It even includes a link to
>> the proper paper. I hope that is enough for the strict statisticians
>> among us ;-)
>
> I see. Thanks!
>
>> See:
>> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars
>>
>> PS: there is a bug in the jqPlot plotting library when null values are
>> present. Trying to display PyPy 1.3 results for the newer go, pyflake
>> or ?raytrace will create some nasty js loops. It also has problems
>> with autoscaling the axis sometimes.
>>
>>
>>
>> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>>> Thanks all for the input.
>>> I've compiled a list based on your mails, the Unladen benchmarks page
>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>>> alioth descriptions. Here is an extract of the current speed.pypy.org
>>> admin:
>>>
>>> ai
>>> chaos ? Creates chaosgame-like fractals
>>> crypto_pyaes ? ?A pure python implementation of AES
>>> django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table
>>>
>>> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch
>>> benchmark is defined by programs in Performing Lisp Analysis of the
>>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.
>>>
>>> float ? ? ? ? ? Creates an array of points using circular projection and then
>>> normalizes and maximizes them. Floating-point heavy.
>>> go ? ? ? ? ? ? ?A go (chess like game) computer player AI.
>>> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib
>>> meteor-contest ?Searchs for solutions to shape packing puzzle.
>>> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the
>>> orbits of Jovian planets, using a simple symplectic-integrator.
>>> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2
>>> decoder/decompressor.
>>> raytrace-simple A raytracer renderer
>>> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task
>>> dispatcher in the kernel of an operating system.
>>> rietveld ? ? ? ?A Django application benchmark.
>>> slowspitfire
>>> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier
>>> spectral-norm
>>> spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table.
>>> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a
>>> 1000x1000-cell HTML table, using the cstringio module.
>>> telco
>>> twisted_iteration
>>> twisted_names
>>> twisted_pb
>>> twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP
>>> (on the loopback interface) and then writes bytes as fast as it can.
>>> waf ? ? Python-based framework for configuring, compiling and installing
>>> applications. It derives from the concepts of other build tools such
>>> as Scons, Autotools, CMake or Ant.
>>>
>>>
>>> So the remaining descriptions are
>>> ai
>>> slowspitfire (what is the exact difference between the three spitfire benches?)
>>> spectral-norm
>>> telco
>>> twisted (most of them)
>>>
>>> Are the descriptions all right so far?. They can be made much longer
>>> if you deem it desirable.
>>>
>>> on speed.pypy.org you will currently see the descriptions in 3 places:
>>> - Changes view: A tooltip on hover over each benchmark
>>> - Timeline: a description box beneath each plot
>>> - Comparison: A tooltip over each benchmark when hovering the
>>> selection menu on the left side.
>>>
>>> Any suggestions on how to improve it further are welcome ;-)
>>>
>>> Miquel
>>>
>>>
>>> 2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
>>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com> wrote:
>>>>> Here is a incomplete draft list:
>>>>
>>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the
>>>>> cstringio version uses a modified engine (that uses cstringio)
>>>>
>>>>> spambayes: Spambayes is a bayesian spam filter
>>>>
>>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
>>>> remember when, because of this, spambayes was slower (including
>>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
>>>> and the head (for slowspitfire it's the opposite).
>>>>
>>>> For the rest, I see no significant case of slowdown of PyPy over time.
>>>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
>>>> --
>>>> Paolo Giarrusso - Ph.D. Student
>>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>>>
>>>
>>
>
>
>
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
>


From alex.gaynor at gmail.com  Wed Dec 22 09:33:08 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 22 Dec 2010 02:33:08 -0600
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTinn70yxvFy3DQNpk9Hu7rDB7HoiiQK_W+7oDWPw@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>
	<AANLkTinG9VyPcFk05qitjHgKCDhd2xf=UuKyg=FH4Z4g@mail.gmail.com>
	<AANLkTinn70yxvFy3DQNpk9Hu7rDB7HoiiQK_W+7oDWPw@mail.gmail.com>
Message-ID: <AANLkTiki_xPZgZvffzcZzcgMN2qOVyvcLO8opwEv5ZU5@mail.gmail.com>

ai: runs a bruce force n-queens solver

Alex

On Wed, Dec 22, 2010 at 2:26 AM, Miquel Torres <tobami at googlemail.com>wrote:

> so, what about the ai and spectral-norm benchmarks. Anybody can come
> up with a description for them?
>
>
> 2010/12/15 Paolo Giarrusso <p.giarrusso at gmail.com>:
> > On Mon, Dec 13, 2010 at 09:31, Miquel Torres <tobami at googlemail.com>
> wrote:
> >> Oh, btw., the "normalized" stacked bars now display a warning note
> >> about its correctness, and how it must be viewed as giving results a
> >> weighting instead of them being normalized. It even includes a link to
> >> the proper paper. I hope that is enough for the strict statisticians
> >> among us ;-)
> >
> > I see. Thanks!
> >
> >> See:
> >>
> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars
> >>
> >> PS: there is a bug in the jqPlot plotting library when null values are
> >> present. Trying to display PyPy 1.3 results for the newer go, pyflake
> >> or  raytrace will create some nasty js loops. It also has problems
> >> with autoscaling the axis sometimes.
> >>
> >>
> >>
> >> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
> >>> Thanks all for the input.
> >>> I've compiled a list based on your mails, the Unladen benchmarks page
> >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
> >>> alioth descriptions. Here is an extract of the current speed.pypy.org
> >>> admin:
> >>>
> >>> ai
> >>> chaos   Creates chaosgame-like fractals
> >>> crypto_pyaes    A pure python implementation of AES
> >>> django          Uses the Django template system to build a 150x150-cell
> HTML table
> >>>
> >>> fannkuch                Indexed-access to tiny integer-sequence. The
> fannkuch
> >>> benchmark is defined by programs in Performing Lisp Analysis of the
> >>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.
> >>>
> >>> float           Creates an array of points using circular projection
> and then
> >>> normalizes and maximizes them. Floating-point heavy.
> >>> go              A go (chess like game) computer player AI.
> >>> html5lib        Parses the HTML 5 spec using html5lib
> >>> meteor-contest  Searchs for solutions to shape packing puzzle.
> >>> nbody_modified          Double-precision N-body simulation. It models
> the
> >>> orbits of Jovian planets, using a simple symplectic-integrator.
> >>> pyflate-fast            Stand-alone pure-Python DEFLATE (gzip) and
> bzip2
> >>> decoder/decompressor.
> >>> raytrace-simple A raytracer renderer
> >>> richards                Medium-sized language benchmark that simulates
> the task
> >>> dispatcher in the kernel of an operating system.
> >>> rietveld        A Django application benchmark.
> >>> slowspitfire
> >>> spambayes       Runs a canned mailbox through a SpamBayes ham/spam
> classifier
> >>> spectral-norm
> >>> spitfire        Uses the Spitfire template system to build a
> 1000x1000-cell HTML table.
> >>> spitfire_cstringio      Uses the Spitfire template system to build a
> >>> 1000x1000-cell HTML table, using the cstringio module.
> >>> telco
> >>> twisted_iteration
> >>> twisted_names
> >>> twisted_pb
> >>> twisted_tcp     Connects one Twised client to one Twisted server over
> TCP
> >>> (on the loopback interface) and then writes bytes as fast as it can.
> >>> waf     Python-based framework for configuring, compiling and
> installing
> >>> applications. It derives from the concepts of other build tools such
> >>> as Scons, Autotools, CMake or Ant.
> >>>
> >>>
> >>> So the remaining descriptions are
> >>> ai
> >>> slowspitfire (what is the exact difference between the three spitfire
> benches?)
> >>> spectral-norm
> >>> telco
> >>> twisted (most of them)
> >>>
> >>> Are the descriptions all right so far?. They can be made much longer
> >>> if you deem it desirable.
> >>>
> >>> on speed.pypy.org you will currently see the descriptions in 3 places:
> >>> - Changes view: A tooltip on hover over each benchmark
> >>> - Timeline: a description box beneath each plot
> >>> - Comparison: A tooltip over each benchmark when hovering the
> >>> selection menu on the left side.
> >>>
> >>> Any suggestions on how to improve it further are welcome ;-)
> >>>
> >>> Miquel
> >>>
> >>>
> >>> 2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
> >>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada <santagada at gmail.com>
> wrote:
> >>>>> Here is a incomplete draft list:
> >>>>
> >>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the
> >>>>> cstringio version uses a modified engine (that uses cstringio)
> >>>>
> >>>>> spambayes: Spambayes is a bayesian spam filter
> >>>>
> >>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
> >>>> remember when, because of this, spambayes was slower (including
> >>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
> >>>> and the head (for slowspitfire it's the opposite).
> >>>>
> >>>> For the rest, I see no significant case of slowdown of PyPy over time.
> >>>>
> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
> >>>> --
> >>>> Paolo Giarrusso - Ph.D. Student
> >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/<http://www.informatik.uni-marburg.de/%7Epgiarrusso/>
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Paolo Giarrusso - Ph.D. Student
> > http://www.informatik.uni-marburg.de/~pgiarrusso/<http://www.informatik.uni-marburg.de/%7Epgiarrusso/>
> >
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/7f377bee/attachment.html>

From tobami at googlemail.com  Wed Dec 22 11:19:29 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Wed, 22 Dec 2010 11:19:29 +0100
Subject: [pypy-dev] Idea for speed.pypy.org
In-Reply-To: <AANLkTiki_xPZgZvffzcZzcgMN2qOVyvcLO8opwEv5ZU5@mail.gmail.com>
References: <201012040043.04547.jacob@openend.se>
	<240465.94379.qm@web111314.mail.gq1.yahoo.com>
	<tobami@googlemail.com>
	<AANLkTim3igMNQpOURng3PyyeGjJAMigNkH9sNXSx_m15@mail.gmail.com>
	<201012041105.oB4B5uo1019532@theraft.openend.se>
	<AANLkTi=6ueCB7iduVpGheE4QGQW0GZhqKAphAMpeAtUL@mail.gmail.com>
	<AANLkTi=p_wsbehZHUsdjCfztMKJ3O+ZPVhp0i5zgSkFy@mail.gmail.com>
	<AANLkTiku2L7Ykb39=ySiDzCWSHQErTwoyyDrvgJF_PO0@mail.gmail.com>
	<AANLkTinjBJAc35DM7zC8RbG+G2sfFWzr0Sy2=oAE0jmY@mail.gmail.com>
	<AANLkTik_dRyLtyBY8bg09iogCjT4kHnLsc0LXy+g7e9v@mail.gmail.com>
	<AANLkTikBbihivcM8RX12dy76dfeD797wcQLGfVh4gTmB@mail.gmail.com>
	<AANLkTinG9VyPcFk05qitjHgKCDhd2xf=UuKyg=FH4Z4g@mail.gmail.com>
	<AANLkTinn70yxvFy3DQNpk9Hu7rDB7HoiiQK_W+7oDWPw@mail.gmail.com>
	<AANLkTiki_xPZgZvffzcZzcgMN2qOVyvcLO8opwEv5ZU5@mail.gmail.com>
Message-ID: <AANLkTi==xwFLuKOOMsThh9vvYNqgK_LXpeJUWpEWtGYn@mail.gmail.com>

@Alex: added!



2010/12/22 Alex Gaynor <alex.gaynor at gmail.com>:
> ai: runs a bruce force n-queens solver
>
> Alex
>
> On Wed, Dec 22, 2010 at 2:26 AM, Miquel Torres <tobami at googlemail.com>
> wrote:
>>
>> so, what about the ai and spectral-norm benchmarks. Anybody can come
>> up with a description for them?
>>
>>
>> 2010/12/15 Paolo Giarrusso <p.giarrusso at gmail.com>:
>> > On Mon, Dec 13, 2010 at 09:31, Miquel Torres <tobami at googlemail.com>
>> > wrote:
>> >> Oh, btw., the "normalized" stacked bars now display a warning note
>> >> about its correctness, and how it must be viewed as giving results a
>> >> weighting instead of them being normalized. It even includes a link to
>> >> the proper paper. I hope that is enough for the strict statisticians
>> >> among us ;-)
>> >
>> > I see. Thanks!
>> >
>> >> See:
>> >>
>> >> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars
>> >>
>> >> PS: there is a bug in the jqPlot plotting library when null values are
>> >> present. Trying to display PyPy 1.3 results for the newer go, pyflake
>> >> or ?raytrace will create some nasty js loops. It also has problems
>> >> with autoscaling the axis sometimes.
>> >>
>> >>
>> >>
>> >> 2010/12/13 Miquel Torres <tobami at googlemail.com>:
>> >>> Thanks all for the input.
>> >>> I've compiled a list based on your mails, the Unladen benchmarks page
>> >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the
>> >>> alioth descriptions. Here is an extract of the current speed.pypy.org
>> >>> admin:
>> >>>
>> >>> ai
>> >>> chaos ? Creates chaosgame-like fractals
>> >>> crypto_pyaes ? ?A pure python implementation of AES
>> >>> django ? ? ? ? ?Uses the Django template system to build a
>> >>> 150x150-cell HTML table
>> >>>
>> >>> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The
>> >>> fannkuch
>> >>> benchmark is defined by programs in Performing Lisp Analysis of the
>> >>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig.
>> >>>
>> >>> float ? ? ? ? ? Creates an array of points using circular projection
>> >>> and then
>> >>> normalizes and maximizes them. Floating-point heavy.
>> >>> go ? ? ? ? ? ? ?A go (chess like game) computer player AI.
>> >>> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib
>> >>> meteor-contest ?Searchs for solutions to shape packing puzzle.
>> >>> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models
>> >>> the
>> >>> orbits of Jovian planets, using a simple symplectic-integrator.
>> >>> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and
>> >>> bzip2
>> >>> decoder/decompressor.
>> >>> raytrace-simple A raytracer renderer
>> >>> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates
>> >>> the task
>> >>> dispatcher in the kernel of an operating system.
>> >>> rietveld ? ? ? ?A Django application benchmark.
>> >>> slowspitfire
>> >>> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam
>> >>> classifier
>> >>> spectral-norm
>> >>> spitfire ? ? ? ?Uses the Spitfire template system to build a
>> >>> 1000x1000-cell HTML table.
>> >>> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a
>> >>> 1000x1000-cell HTML table, using the cstringio module.
>> >>> telco
>> >>> twisted_iteration
>> >>> twisted_names
>> >>> twisted_pb
>> >>> twisted_tcp ? ? Connects one Twised client to one Twisted server over
>> >>> TCP
>> >>> (on the loopback interface) and then writes bytes as fast as it can.
>> >>> waf ? ? Python-based framework for configuring, compiling and
>> >>> installing
>> >>> applications. It derives from the concepts of other build tools such
>> >>> as Scons, Autotools, CMake or Ant.
>> >>>
>> >>>
>> >>> So the remaining descriptions are
>> >>> ai
>> >>> slowspitfire (what is the exact difference between the three spitfire
>> >>> benches?)
>> >>> spectral-norm
>> >>> telco
>> >>> twisted (most of them)
>> >>>
>> >>> Are the descriptions all right so far?. They can be made much longer
>> >>> if you deem it desirable.
>> >>>
>> >>> on speed.pypy.org you will currently see the descriptions in 3 places:
>> >>> - Changes view: A tooltip on hover over each benchmark
>> >>> - Timeline: a description box beneath each plot
>> >>> - Comparison: A tooltip over each benchmark when hovering the
>> >>> selection menu on the left side.
>> >>>
>> >>> Any suggestions on how to improve it further are welcome ;-)
>> >>>
>> >>> Miquel
>> >>>
>> >>>
>> >>> 2010/12/9 Paolo Giarrusso <p.giarrusso at gmail.com>:
>> >>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada
>> >>>> <santagada at gmail.com> wrote:
>> >>>>> Here is a incomplete draft list:
>> >>>>
>> >>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the
>> >>>>> cstringio version uses a modified engine (that uses cstringio)
>> >>>>
>> >>>>> spambayes: Spambayes is a bayesian spam filter
>> >>>>
>> >>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I
>> >>>> remember when, because of this, spambayes was slower (including
>> >>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4
>> >>>> and the head (for slowspitfire it's the opposite).
>> >>>>
>> >>>> For the rest, I see no significant case of slowdown of PyPy over
>> >>>> time.
>> >>>>
>> >>>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars
>> >>>> --
>> >>>> Paolo Giarrusso - Ph.D. Student
>> >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>> >>>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Paolo Giarrusso - Ph.D. Student
>> > http://www.informatik.uni-marburg.de/~pgiarrusso/
>> >
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> "Code can always be simpler than you think, but never as simple as you want"
> -- Me
>


From lac at openend.se  Wed Dec 22 12:10:07 2010
From: lac at openend.se (Laura Creighton)
Date: Wed, 22 Dec 2010 12:10:07 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: Message from Miquel Torres <tobami@googlemail.com> of "Wed,
	22 Dec 2010 09:24:28 +0100."
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com> 
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com> 
Message-ID: <201012221110.oBMBA7Q8021121@theraft.openend.se>

In a message of Wed, 22 Dec 2010 09:24:28 +0100, Miquel Torres writes:

>Btw., instead of continuing here "polluting" the pypy-dev mailing
>list, we can move to http://groups.google.com/group/codespeed if
>needed.

I'd rather it stayed here, myself.
Laura


From chef at ghum.de  Wed Dec 22 12:15:32 2010
From: chef at ghum.de (Massa, Harald Armin)
Date: Wed, 22 Dec 2010 12:15:32 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
Message-ID: <AANLkTi=JGaQSYQc4fsa_CLgZ9vO1nihpaqsujZYnynd0@mail.gmail.com>

>
> Mmm, yes, it appears that the memory is not properly freed. I can
> try experimenting by destroying the dom element and recreating it each time
> that the plot changes...
>
> Btw., instead of continuing here "polluting" the pypy-dev mailing
> list, we can move to http://groups.google.com/group/codespeed if needed.
>
> I recommend to keep it here ... if some of the most important browser
vendors have challenges with memory, everybody with memory...challenges can
learn from the discussion.

Harald



-- 
GHUM GmbH
Harald Armin Massa
Spielberger Stra?e 49
70435 Stuttgart
0173/9409607

Amtsgericht Stuttgart, HRB 734971
-
persuadere.
et programmare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/f63819e6/attachment.html>

From Arnd.Rechenburg at tomtom.com  Wed Dec 22 18:00:10 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Wed, 22 Dec 2010 18:00:10 +0100
Subject: [pypy-dev] Fatal RPython error: AssertionError
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703FC14AD@BESRVWC-EXC01.ta.global>

Hi,

 

I have the following problem:

 

Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48)

[PyPy 1.4.1] on linux2

 

RPython traceback:

  File "implement_54.c", line 21229, in send_bridge_to_backend

  File "implement_61.c", line 14547, in Assembler386_assemble_bridge

  File "implement_63.c", line 44979, in RegAlloc_prepare_bridge

  File "implement_64.c", line 15487, in RegAlloc__update_bindings

  File "implement_38.c", line 60625, in
RegisterManager__check_invariants

Fatal RPython error: AssertionError

 

Any help would be appreciated.

 

Regards,

Arnd

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/dba27e35/attachment.html>

From Arnd.Rechenburg at tomtom.com  Wed Dec 22 18:24:31 2010
From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg)
Date: Wed, 22 Dec 2010 18:24:31 +0100
Subject: [pypy-dev] Fatal RPython error: AssertionError
Message-ID: <FA92FB0BD1FD6349BD520B84A62DFAC703FC14BC@BESRVWC-EXC01.ta.global>

Hi again,

 

It was my mistake.

The problem was in 1.4.0 and is solved in 1.4.1

 

Regards,

Arnd

________________________________

From: Arnd Rechenburg 
Sent: Mittwoch, 22. Dezember 2010 18:00
To: 'pypy-dev at codespeak.net'
Subject: Fatal RPython error: AssertionError

 

Hi,

 

I have the following problem:

 

Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48)

[PyPy 1.4.1] on linux2

 

RPython traceback:

  File "implement_54.c", line 21229, in send_bridge_to_backend

  File "implement_61.c", line 14547, in Assembler386_assemble_bridge

  File "implement_63.c", line 44979, in RegAlloc_prepare_bridge

  File "implement_64.c", line 15487, in RegAlloc__update_bindings

  File "implement_38.c", line 60625, in
RegisterManager__check_invariants

Fatal RPython error: AssertionError

 

Any help would be appreciated.

 

Regards,

Arnd

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/db3949b3/attachment.html>

From garyrob at me.com  Wed Dec 22 22:34:12 2010
From: garyrob at me.com (Gary Robinson)
Date: Wed, 22 Dec 2010 16:34:12 -0500
Subject: [pypy-dev] dlopen NotImplementedError
Message-ID: <9C59B64A-15B8-4965-86B5-EC0DA57183A2@me.com>

So, as of today, I'm starting to do as much of my day-to-day work in pypy as I can.

The missing numpy is a problem, but one I can work around a enough amount of the time to make this effort worth my while.

I don't know how, or even whether, I should report certain kinds of problems I might run into. I would appreciate some feedback on that. I want to help by giving feedback in whatever form, and to what degree, the pypy devs desire.

For example, I can run some scripts in pypy 1.4 but not in pypy fast-forward because of an error involving dlopen(). Other scripts are unaffected by this problem, depending on what needs to be imported.

When I try to run fast-forward in interactive mode, I get the same error, which is pasted below.

Python 2.7.0 (8c2427fa1a01, Dec 19 2010, 15:58:12)
[PyPy 1.4.0 with GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "app_main.py", line 53, in run_toplevel
  File "app_main.py", line 522, in run_it
  File "/etc/pythonstart", line 7, in <module>
    import readline
  File "/root/pypy/lib_pypy/readline.py", line 9, in <module>
  File "/root/pypy/lib_pypy/pyrepl/readline.py", line 33, in <module>
  File "/root/pypy/lib_pypy/pyrepl/unix_console.py", line 25, in <module>
  File "/root/pypy/lib_pypy/pyrepl/curses.py", line 30, in <module>
  File "/root/pypy/lib_pypy/_minimal_curses.py", line 12, in <module>
  File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 441, in <module>
  File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 353, in __init__
  File "/root/pypy/lib_pypy/_ctypes/dll.py", line 7, in dlopen
NotImplementedError: dlopen(None)
And now for something completely different: ``- do you know berlin? - all of
it? - well, just berlin''
Traceback (most recent call last):
  File "app_main.py", line 53, in run_toplevel
  File "/root/pypy/lib_pypy/_pypy_interact.py", line 27, in interactive_console
  File "/root/pypy/lib_pypy/pyrepl/simple_interact.py", line 27, in <module>
  File "/root/pypy/lib_pypy/pyrepl/readline.py", line 33, in <module>
  File "/root/pypy/lib_pypy/pyrepl/unix_console.py", line 25, in <module>
  File "/root/pypy/lib_pypy/pyrepl/curses.py", line 30, in <module>
  File "/root/pypy/lib_pypy/_minimal_curses.py", line 12, in <module>
  File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 441, in <module>
  File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 353, in __init__
  File "/root/pypy/lib_pypy/_ctypes/dll.py", line 7, in dlopen
NotImplementedError: dlopen(None)


-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From garyrob at me.com  Wed Dec 22 22:52:32 2010
From: garyrob at me.com (Gary Robinson)
Date: Wed, 22 Dec 2010 16:52:32 -0500
Subject: [pypy-dev] dlopen NotImplementedError
Message-ID: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>

Hmm, I'm thinking the dlopen error might be my problem. I think I built python at /root/pypy, and then moved it to a more proper location. But  the error message is referring to /root/pypy, so I'm guessing it has that location hardcoded because that's where I built it?

I'm going to rebuild but I wanted to post this message so that you guys don't worry about it (unless rebuilding doesn't help). 

-- 

Gary Robinson
CTO
Emergent Discovery, LLC
personal email: garyrob at me.com
work email: grobinson at emergentdiscovery.com
Company: http://www.emergentdiscovery.com
Blog:    http://www.garyrobinson.net






From alex.gaynor at gmail.com  Wed Dec 22 22:54:37 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 22 Dec 2010 15:54:37 -0600
Subject: [pypy-dev] dlopen NotImplementedError
In-Reply-To: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
References: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
Message-ID: <AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>

On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson <garyrob at me.com> wrote:

> Hmm, I'm thinking the dlopen error might be my problem. I think I built
> python at /root/pypy, and then moved it to a more proper location. But  the
> error message is referring to /root/pypy, so I'm guessing it has that
> location hardcoded because that's where I built it?
>
> I'm going to rebuild but I wanted to post this message so that you guys
> don't worry about it (unless rebuilding doesn't help).
>
> --
>
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog:    http://www.garyrobinson.net
>
>
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Nope, it's a known thing, anything trying to import ctypes on fast-forward
blows up ATM.  If anyone knows what dlopen(None) is supposed to mean, can
they let me know and I'll try to take a pass at fixing this?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/fa40c797/attachment.html>

From dimaqq at gmail.com  Wed Dec 22 23:21:52 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 22 Dec 2010 15:21:52 -0700
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
Message-ID: <AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>

Firefox people point the blame squarely at jqplot:

--- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
13:43:23 PST ---
So what I see this page do, in horizontal mode, is create 17 canvases each of
which is width="816" height="3587".  That means that each of them has a backing
store of 816*3587*4 = 11,707,968 bytes.  So that's about 200MB of memory usage
right there.

I have no idea why they feel a need for 17 huge canvases, but if they want
them, that's how much memory they'll take...

( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 )

On 22 December 2010 01:24, Miquel Torres <tobami at googlemail.com> wrote:
> Mmm, yes, it appears that the memory is not properly freed. I can try
> experimenting by destroying the dom element and recreating it each
> time that the plot changes...
>
> Btw., instead of continuing here "polluting" the pypy-dev mailing
> list, we can move to http://groups.google.com/group/codespeed if
> needed.
>
> Thanks for reporting this.
>
> Cheers,
> Miquel
>
>
> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>> I sort of figured it out, although I don't have a ready solution.
>>
>> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8.
>> Basically memory consumption of the plot appears to be proportional to
>> plot area.
>> Normal plot that you get by default at
>> http://speed.pypy.org/comparison/ costs about 100M of browser memory
>> consumption:
>> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050;
>> opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600;
>> Switching to "horizontal" produces a tall plot of same width and costs
>> about 300~700M of browser memory:
>> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050;
>> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600;
>>
>> Suprisingly window size only matters while javascript produces the
>> plot, and not when window is resized, even though plot it resized with
>> the window correctly.
>>
>> This alone is pretty heavy, but doesn't grind the browser.
>> What really grinds is that every time you change a tickbox on the
>> left, a plot is redrawn and another 200M of browser memory is wasted.
>> This is not double buffering, as next change adds yet another 200M or
>> so and so on, it appears that either js doesn't free something, or
>> browser caches or saves the previous page state.
>>
>> As memory consumption grows, at some point browser hits the wall,
>> causes heavy swapping for some time, and I think garbage collection,
>> because practical (but not virtual) memory usage first drops to
>> something like 20~50M and then returns to "normal" 300M.
>> opera ~30 seconds, stable ff about a minute, beta ff several minutes
>> (total system mem 1G, cpu Atom @1.6GHz)
>>
>> Perhaps OS also plays a role in the grind, as it is clearly swapping
>> and swaps out too much? or triggers gc too late and gc has to pull the
>> pages back from disk to perform collection?
>>
>> ie8 doesn't use that much memory, as a matter of fact memory
>> consumption starts little (40M) and changes very little (only +10M) if
>> you go to horizonatal view; the price is very slow rendering, more
>> than 10 seconds per column change.
>>
>> I'll post this on firefox bugzilla too, let's see if someone has a solution.
>>
>> Meanwhile perhaps pypy speed center could start with a smaller plot
>> area (or fewer columns as that makes horizontal plot smaller) to
>> accomodate varying hardware and system mem usage that users might
>> have?
>> The simplest would be a warning next to "horizontal" checkbox.
>>
>> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>>> Hi Dima,
>>>
>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>
>>> Well, speed.pypy is actually fast in all modern browsers. The problem
>>> you are referring to is probably caused by a bug in the javascript
>>> plotting library (jqPplot) that is triggered in the comparison view
>>> when there are some results with 0 values. It only appears for some
>>> plot types, but it is very annoying because it grinds the browser to a
>>> halt like you say. Is that what you meant?
>>>
>>> We are looking into it, and will fix that library if necessary.
>>>
>>> Cheers,
>>> Miquel
>>>
>>>
>>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>>> On 20 December 2010 19:21, William ML Leslie
>>>> <william.leslie.ttg at gmail.com> wrote:
>>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>>> More visibility for performance achievements would do
>>>>>> good too.
>>>>>
>>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>>
>>>> for example http://shootout.alioth.debian.org/
>>>> doesn't say which pypy version is used, what options, doesn't have
>>>> performance figures for multithreaded/multicore
>>>>
>>>> also benchmarks are kinda small, most of them are not docuemented, and
>>>> I couldn't find any info if the same python code was used for cpython
>>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>>> verions were used, that is each tweaked for best performance given the
>>>> known tradeoffs for each implementation.further the most standard
>>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>>> programs are large and only a few small paths are execured often.
>>>>
>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>
>>>> if you google for 'python performance' you don't get a single link to
>>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>>> indexed, it took me quite some time to get some of my questions
>>>> answered with a search. also if you look up 'pypy gc' you get a page
>>>> on codespeak, but to interpret what the data actually means is so far
>>>> beyond me.
>>>>
>>>> a good overview is found in the mainling list
>>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>>> slowspitfire and spambayes bits are outdated by now.
>>>>
>>>> the definitive good thing about pypy is that it's much easier to find
>>>> out about its inner workings than that of cpython!
>>>>
>>>> hopefully a bit more of end-user stuff get known.
>>>> let's call it pypy public outreach (feature request)
>>>>
>>>>>
>>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>>> hard would it be to use same techniques and most of backends for some
>>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>>
>>>>> You know that pypy already has implementations of other languages,
>>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>>> integrated with the translated pypy-c, but they show that it is not
>>>>> too difficult to write a runtime for any dynamic language you choose.
>>>>
>>>> Oh I didn't know there were so many, and I mistakenly though that js
>>>> was a target, not implmented langauge. In any case I read somewhere
>>>> that js support was retired...
>>>>
>>>>>
>>>>>> And how hard
>>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>>> could be used in the other? Or that modules written in rpython could
>>>>>> be used in several langs?
>>>>>
>>>>> It's in the "interesting problems" bucket, and the effort required
>>>>> depends on the level of integration between languages you want. ?There
>>>>> are several projects already attempting to do decent integration
>>>>> between several languages, besides the approach used on the JVM, there
>>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>>>> worth waiting to see how these different projects pan out, before
>>>>> spending a bunch of effort just to be an also-ran in the
>>>>> multi-language runtime market.
>>>>>
>>>>> However, implementing more languages in rpython has the side-effect of
>>>>> propagating the l * o * p problem: it introduces more and more
>>>>> implementations that then have to be maintained, so good
>>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>>> existing runtimes can hook into it.
>>>>
>>>> Makes perfect sense, after all any given other language hardly has the
>>>> same data model as python.
>>>>
>>>>>
>>>>> But it would be an interesting experiment (to marry the various
>>>>> interpreters pypy ships with), if you wanted to try it.
>>>>>
>>>>> My two cents.
>>>>>
>>>>> --
>>>>> William Leslie
>>>>>
>>>> _______________________________________________
>>>> pypy-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>
>>
>


From exarkun at twistedmatrix.com  Wed Dec 22 23:41:27 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Wed, 22 Dec 2010 22:41:27 -0000
Subject: [pypy-dev] dlopen NotImplementedError
In-Reply-To: <AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>
References: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
	<AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>
Message-ID: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain>

On 09:54 pm, alex.gaynor at gmail.com wrote:
>On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson <garyrob at me.com> wrote:
>>Hmm, I'm thinking the dlopen error might be my problem. I think I 
>>built
>>python at /root/pypy, and then moved it to a more proper location. But 
>>the
>>error message is referring to /root/pypy, so I'm guessing it has that
>>location hardcoded because that's where I built it?
>>
>>I'm going to rebuild but I wanted to post this message so that you 
>>guys
>>don't worry about it (unless rebuilding doesn't help).
>Nope, it's a known thing, anything trying to import ctypes on fast- 
>forward
>blows up ATM.  If anyone knows what dlopen(None) is supposed to mean, 
>can
>they let me know and I'll try to take a pass at fixing this?

 From the dlopen man page:

  If filename is NULL, then the returned handle is for the main program.

I imagine that's the case someone is trying to trigger with 
ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in 
this context).  On the other hand, maybe it's just a screw up somewhere 
else that causes None to show up instead of the right library name, I 
haven't looked at the code in question much.

Jean-Paul


From alex.gaynor at gmail.com  Wed Dec 22 23:45:03 2010
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 22 Dec 2010 16:45:03 -0600
Subject: [pypy-dev] dlopen NotImplementedError
In-Reply-To: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain>
References: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
	<AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>
	<20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain>
Message-ID: <AANLkTinbhDFxhvwuLt8HBCMjHjp4q-wijTriyfJ4zrz9@mail.gmail.com>

On Wed, Dec 22, 2010 at 4:41 PM, <exarkun at twistedmatrix.com> wrote:

> On 09:54 pm, alex.gaynor at gmail.com wrote:
> >On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson <garyrob at me.com> wrote:
> >>Hmm, I'm thinking the dlopen error might be my problem. I think I
> >>built
> >>python at /root/pypy, and then moved it to a more proper location. But
> >>the
> >>error message is referring to /root/pypy, so I'm guessing it has that
> >>location hardcoded because that's where I built it?
> >>
> >>I'm going to rebuild but I wanted to post this message so that you
> >>guys
> >>don't worry about it (unless rebuilding doesn't help).
> >Nope, it's a known thing, anything trying to import ctypes on fast-
> >forward
> >blows up ATM.  If anyone knows what dlopen(None) is supposed to mean,
> >can
> >they let me know and I'll try to take a pass at fixing this?
>
>  From the dlopen man page:
>
>  If filename is NULL, then the returned handle is for the main program.
>
> I imagine that's the case someone is trying to trigger with
> ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in
> this context).  On the other hand, maybe it's just a screw up somewhere
> else that causes None to show up instead of the right library name, I
> haven't looked at the code in question much.
>
> Jean-Paul
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Ok I see the issue I think.  It's coming from PyDLL(None).  Not sure what
the right solution is there, I know Amaury said he was working on the
cpythonapi stuff.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want"
-- Me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101222/883b2ae5/attachment.html>

From dimaqq at gmail.com  Thu Dec 23 03:36:59 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 22 Dec 2010 19:36:59 -0700
Subject: [pypy-dev] Fwd: pypy-dev Re: funding/popularity?
In-Reply-To: <765473.56989.qm@web65612.mail.ac4.yahoo.com>
References: <765473.56989.qm@web65612.mail.ac4.yahoo.com>
Message-ID: <AANLkTimjHGE3EJhqjPmRwh6kADSOZZ_CSmX-UCzv4fGq@mail.gmail.com>

---------- Forwarded message ----------
From: Isaac Gouy <igouy2 at yahoo.com>

Dima Tisnek <dimaqq <at> gmail.com> writes:

>
> On 20 December 2010 19:21, William ML Leslie
> <william.leslie.ttg <at> gmail.com> wrote:
> > On 21 December 2010 11:59, Dima Tisnek <dimaqq <at> gmail.com> wrote:
> >> More visibility for performance achievements would do
> >> good too.
> >
> > Where are pypy's performance achievements *not* visible, but should be?
>
> for example http://shootout.alioth.debian.org/
> doesn't say which pypy version is used


Oh yes it does -

http://shootout.alioth.debian.org/u32/compare.php?lang=pypy#about


> what options, doesn't have
> performance figures for multithreaded/multicore


All the Python multiprocessing programs "Failed" with PyPy -

http://shootout.alioth.debian.org/u32/measurements.php?lang=pypy


> also benchmarks are kinda small, most of them are not docuemented, and
> I couldn't find any info if the same python code was used for cpython
> and pypy (both shootout and speed pypy) or interpreter-specific
> verions were used, that is each tweaked for best performance given the
> known tradeoffs for each implementation.


The full source code is shown for all the programs -

http://shootout.alioth.debian.org/u32/program.php?test=spectralnorm?=pypy&id=6

http://shootout.alioth.debian.org/u32/program.php?test=spectralnorm?=python&id=6

The GZip byte size is shown for each program, in this case 341 for
both programs.


Maybe the benchmarks game shows more information than you realize.


From dimaqq at gmail.com  Thu Dec 23 03:46:19 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 22 Dec 2010 19:46:19 -0700
Subject: [pypy-dev] 1.4.1 threading
Message-ID: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>

Hi,

Somehow I'm getting exactly same performance in threaded and
non-threaded code (I would expect at least some speedup),
Is threading not there in this pypy release in practice or am I
missing some argument to pypy?

the test case is cpu-bound in python, and possibly gc-bound too


From benjamin at python.org  Thu Dec 23 03:49:57 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 22 Dec 2010 20:49:57 -0600
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
Message-ID: <AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>

2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
> Hi,
>
> Somehow I'm getting exactly same performance in threaded and
> non-threaded code (I would expect at least some speedup),
> Is threading not there in this pypy release in practice or am I
> missing some argument to pypy?

PyPy has a GIL if that's what you mean.



-- 
Regards,
Benjamin


From dimaqq at gmail.com  Thu Dec 23 04:01:14 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 22 Dec 2010 20:01:14 -0700
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
Message-ID: <AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>

Oh, I can't yet use alternative gc that obviates GIL then?

Or am I totally confused and pypy still uses GIL for other reasons,
e.g. globals dict safety?

On 22 December 2010 19:49, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>> Hi,
>>
>> Somehow I'm getting exactly same performance in threaded and
>> non-threaded code (I would expect at least some speedup),
>> Is threading not there in this pypy release in practice or am I
>> missing some argument to pypy?
>
> PyPy has a GIL if that's what you mean.
>
>
>
> --
> Regards,
> Benjamin
>


From benjamin at python.org  Thu Dec 23 04:05:26 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 22 Dec 2010 21:05:26 -0600
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
Message-ID: <AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>

2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
> Oh, I can't yet use alternative gc that obviates GIL then?
>
> Or am I totally confused and pypy still uses GIL for other reasons,
> e.g. globals dict safety?

All of the above.



-- 
Regards,
Benjamin


From sungsuha at gmail.com  Thu Dec 23 06:51:26 2010
From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha)
Date: Thu, 23 Dec 2010 05:51:26 +0000 (UTC)
Subject: [pypy-dev] Building PyPy on Windows
Message-ID: <loom.20101223T064823-607@post.gmane.org>

An interesting article came up on python planet:
http://www.redmountainsw.com/wordpress/archives/building-pypy-on-windows

Maybe with the consent of the original author and some cleaning up, 
we could add this to official documentation?

A simple link on the getting started/download pages 
won't hurt windows users either.



From drsalists at gmail.com  Thu Dec 23 06:58:55 2010
From: drsalists at gmail.com (Dan Stromberg)
Date: Wed, 22 Dec 2010 21:58:55 -0800
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
Message-ID: <AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>

On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>> Oh, I can't yet use alternative gc that obviates GIL then?
>>
>> Or am I totally confused and pypy still uses GIL for other reasons,
>> e.g. globals dict safety?
>
> All of the above.

On the topic of parallel dict safety...  I've not played around with
parallel dictionaries at all, but I've heard that treaps can be
parallelized nicely, and they can be (and have been) set up to look
much like a python dictionary.  I admit, I coded one and put it at
http://stromberg.dnsalias.org/~strombrg/treap/ - however, the
implementation is not (yet?) suitable for parallel use.

It's coded in such a way that it'll work as pure python or cython,
depending on which of two m4 symbols you define.  I've used the pure
python version on pypy.

Treaps tend to make just about everything O(log(n)), have good average
performance, and are not as even performers as red-black trees (that
is, they give a higher standard deviation, and a better average
performance than red-black trees).  And they don't play that nicely
with locality of reference (caches) - things are intentionally
randomized.  But if you throw enough cores at them, they might still
be a win compared to a big hash table with a big lock wrapped around
it.


From fijall at gmail.com  Thu Dec 23 07:50:58 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Dec 2010 08:50:58 +0200
Subject: [pypy-dev] dlopen NotImplementedError
In-Reply-To: <AANLkTinbhDFxhvwuLt8HBCMjHjp4q-wijTriyfJ4zrz9@mail.gmail.com>
References: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
	<AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>
	<20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain>
	<AANLkTinbhDFxhvwuLt8HBCMjHjp4q-wijTriyfJ4zrz9@mail.gmail.com>
Message-ID: <AANLkTi=Ej3StS7Z_caT4Dgc_ErpYThmNZjxiiVtHRC+6@mail.gmail.com>

CDLL(None) (or LoadLibrary(None)) is supposed to return to you the
whole namespace (all loaded libraries). It's unsupported as far as I
can tell.

On Thu, Dec 23, 2010 at 12:45 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>
> On Wed, Dec 22, 2010 at 4:41 PM, <exarkun at twistedmatrix.com> wrote:
>>
>> On 09:54 pm, alex.gaynor at gmail.com wrote:
>> >On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson <garyrob at me.com> wrote:
>> >>Hmm, I'm thinking the dlopen error might be my problem. I think I
>> >>built
>> >>python at /root/pypy, and then moved it to a more proper location. But
>> >>the
>> >>error message is referring to /root/pypy, so I'm guessing it has that
>> >>location hardcoded because that's where I built it?
>> >>
>> >>I'm going to rebuild but I wanted to post this message so that you
>> >>guys
>> >>don't worry about it (unless rebuilding doesn't help).
>> >Nope, it's a known thing, anything trying to import ctypes on fast-
>> >forward
>> >blows up ATM. ?If anyone knows what dlopen(None) is supposed to mean,
>> >can
>> >they let me know and I'll try to take a pass at fixing this?
>>
>> ?From the dlopen man page:
>>
>> ?If filename is NULL, then the returned handle is for the main program.
>>
>> I imagine that's the case someone is trying to trigger with
>> ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in
>> this context). ?On the other hand, maybe it's just a screw up somewhere
>> else that causes None to show up instead of the right library name, I
>> haven't looked at the code in question much.
>>
>> Jean-Paul
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>
> Ok I see the issue I think.? It's coming from PyDLL(None).? Not sure what
> the right solution is there, I know Amaury said he was working on the
> cpythonapi stuff.
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> "Code can always be simpler than you think, but never as simple as you want"
> -- Me
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From tobami at googlemail.com  Thu Dec 23 08:29:40 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Thu, 23 Dec 2010 08:29:40 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
	<AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
Message-ID: <AANLkTimViNwV_kBvqJcwNOWCzZZjhPxyZgvC-jx=10S1@mail.gmail.com>

Ok, so it seems from reading comments further down that bug report
that FF's garbage collection does not immediately free memory for old
jqplot canvases.

As I said, as soon as I find the time I will experiment with more
aggressive DOM element destruction/creation and see if it improves how
quickly memory is freed. I do think it will at least partly circumvent
jqPlot's bug.


2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
> Firefox people point the blame squarely at jqplot:
>
> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
> 13:43:23 PST ---
> So what I see this page do, in horizontal mode, is create 17 canvases each of
> which is width="816" height="3587". ?That means that each of them has a backing
> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage
> right there.
>
> I have no idea why they feel a need for 17 huge canvases, but if they want
> them, that's how much memory they'll take...
>
> ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 )
>
> On 22 December 2010 01:24, Miquel Torres <tobami at googlemail.com> wrote:
>> Mmm, yes, it appears that the memory is not properly freed. I can try
>> experimenting by destroying the dom element and recreating it each
>> time that the plot changes...
>>
>> Btw., instead of continuing here "polluting" the pypy-dev mailing
>> list, we can move to http://groups.google.com/group/codespeed if
>> needed.
>>
>> Thanks for reporting this.
>>
>> Cheers,
>> Miquel
>>
>>
>> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>>> I sort of figured it out, although I don't have a ready solution.
>>>
>>> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8.
>>> Basically memory consumption of the plot appears to be proportional to
>>> plot area.
>>> Normal plot that you get by default at
>>> http://speed.pypy.org/comparison/ costs about 100M of browser memory
>>> consumption:
>>> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050;
>>> opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600;
>>> Switching to "horizontal" produces a tall plot of same width and costs
>>> about 300~700M of browser memory:
>>> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050;
>>> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600;
>>>
>>> Suprisingly window size only matters while javascript produces the
>>> plot, and not when window is resized, even though plot it resized with
>>> the window correctly.
>>>
>>> This alone is pretty heavy, but doesn't grind the browser.
>>> What really grinds is that every time you change a tickbox on the
>>> left, a plot is redrawn and another 200M of browser memory is wasted.
>>> This is not double buffering, as next change adds yet another 200M or
>>> so and so on, it appears that either js doesn't free something, or
>>> browser caches or saves the previous page state.
>>>
>>> As memory consumption grows, at some point browser hits the wall,
>>> causes heavy swapping for some time, and I think garbage collection,
>>> because practical (but not virtual) memory usage first drops to
>>> something like 20~50M and then returns to "normal" 300M.
>>> opera ~30 seconds, stable ff about a minute, beta ff several minutes
>>> (total system mem 1G, cpu Atom @1.6GHz)
>>>
>>> Perhaps OS also plays a role in the grind, as it is clearly swapping
>>> and swaps out too much? or triggers gc too late and gc has to pull the
>>> pages back from disk to perform collection?
>>>
>>> ie8 doesn't use that much memory, as a matter of fact memory
>>> consumption starts little (40M) and changes very little (only +10M) if
>>> you go to horizonatal view; the price is very slow rendering, more
>>> than 10 seconds per column change.
>>>
>>> I'll post this on firefox bugzilla too, let's see if someone has a solution.
>>>
>>> Meanwhile perhaps pypy speed center could start with a smaller plot
>>> area (or fewer columns as that makes horizontal plot smaller) to
>>> accomodate varying hardware and system mem usage that users might
>>> have?
>>> The simplest would be a warning next to "horizontal" checkbox.
>>>
>>> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>>>> Hi Dima,
>>>>
>>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>
>>>> Well, speed.pypy is actually fast in all modern browsers. The problem
>>>> you are referring to is probably caused by a bug in the javascript
>>>> plotting library (jqPplot) that is triggered in the comparison view
>>>> when there are some results with 0 values. It only appears for some
>>>> plot types, but it is very annoying because it grinds the browser to a
>>>> halt like you say. Is that what you meant?
>>>>
>>>> We are looking into it, and will fix that library if necessary.
>>>>
>>>> Cheers,
>>>> Miquel
>>>>
>>>>
>>>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>>>> On 20 December 2010 19:21, William ML Leslie
>>>>> <william.leslie.ttg at gmail.com> wrote:
>>>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>>>> More visibility for performance achievements would do
>>>>>>> good too.
>>>>>>
>>>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>>>
>>>>> for example http://shootout.alioth.debian.org/
>>>>> doesn't say which pypy version is used, what options, doesn't have
>>>>> performance figures for multithreaded/multicore
>>>>>
>>>>> also benchmarks are kinda small, most of them are not docuemented, and
>>>>> I couldn't find any info if the same python code was used for cpython
>>>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>>>> verions were used, that is each tweaked for best performance given the
>>>>> known tradeoffs for each implementation.further the most standard
>>>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>>>> programs are large and only a few small paths are execured often.
>>>>>
>>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>>
>>>>> if you google for 'python performance' you don't get a single link to
>>>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>>>> indexed, it took me quite some time to get some of my questions
>>>>> answered with a search. also if you look up 'pypy gc' you get a page
>>>>> on codespeak, but to interpret what the data actually means is so far
>>>>> beyond me.
>>>>>
>>>>> a good overview is found in the mainling list
>>>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>>>> slowspitfire and spambayes bits are outdated by now.
>>>>>
>>>>> the definitive good thing about pypy is that it's much easier to find
>>>>> out about its inner workings than that of cpython!
>>>>>
>>>>> hopefully a bit more of end-user stuff get known.
>>>>> let's call it pypy public outreach (feature request)
>>>>>
>>>>>>
>>>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>>>> hard would it be to use same techniques and most of backends for some
>>>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>>>
>>>>>> You know that pypy already has implementations of other languages,
>>>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>>>> integrated with the translated pypy-c, but they show that it is not
>>>>>> too difficult to write a runtime for any dynamic language you choose.
>>>>>
>>>>> Oh I didn't know there were so many, and I mistakenly though that js
>>>>> was a target, not implmented langauge. In any case I read somewhere
>>>>> that js support was retired...
>>>>>
>>>>>>
>>>>>>> And how hard
>>>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>>>> could be used in the other? Or that modules written in rpython could
>>>>>>> be used in several langs?
>>>>>>
>>>>>> It's in the "interesting problems" bucket, and the effort required
>>>>>> depends on the level of integration between languages you want. ?There
>>>>>> are several projects already attempting to do decent integration
>>>>>> between several languages, besides the approach used on the JVM, there
>>>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be
>>>>>> worth waiting to see how these different projects pan out, before
>>>>>> spending a bunch of effort just to be an also-ran in the
>>>>>> multi-language runtime market.
>>>>>>
>>>>>> However, implementing more languages in rpython has the side-effect of
>>>>>> propagating the l * o * p problem: it introduces more and more
>>>>>> implementations that then have to be maintained, so good
>>>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>>>> existing runtimes can hook into it.
>>>>>
>>>>> Makes perfect sense, after all any given other language hardly has the
>>>>> same data model as python.
>>>>>
>>>>>>
>>>>>> But it would be an interesting experiment (to marry the various
>>>>>> interpreters pypy ships with), if you wanted to try it.
>>>>>>
>>>>>> My two cents.
>>>>>>
>>>>>> --
>>>>>> William Leslie
>>>>>>
>>>>> _______________________________________________
>>>>> pypy-dev at codespeak.net
>>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>>
>>>
>>
>


From arigo at tunes.org  Thu Dec 23 12:04:32 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 23 Dec 2010 12:04:32 +0100
Subject: [pypy-dev] Building PyPy on Windows
In-Reply-To: <loom.20101223T064823-607@post.gmane.org>
References: <loom.20101223T064823-607@post.gmane.org>
Message-ID: <AANLkTimdKwaT6__mkmpqdh7cTDshpMZTLGVx=E0gvF2F@mail.gmail.com>

Hi Seung,

On Thu, Dec 23, 2010 at 6:51 AM, Seung Soo, <sungsuha at gmail.com> wrote:
> An interesting article came up on python planet:
> http://www.redmountainsw.com/wordpress/archives/building-pypy-on-windows

I think the official documentation needs to state also how to build it
with the missing modules (expat etc.).
http://codespeak.net/pypy/dist/pypy/doc/windows.html has some hints,
but is not complete as far as I can tell (e.g. it doesn't say how to
arrange for the .h files to be found).


Armin


From arigo at tunes.org  Thu Dec 23 12:14:00 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 23 Dec 2010 12:14:00 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
	<AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
Message-ID: <AANLkTi=5ZjJYKN9WV7dk9J3LfTyoc0XiMYzWpxRAnoYR@mail.gmail.com>

Hi Dima,

On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
> 13:43:23 PST ---
> So what I see this page do, in horizontal mode, is create 17 canvases each of
> which is width="816" height="3587". ?That means that each of them has a backing
> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage
> right there.
>
> I have no idea why they feel a need for 17 huge canvases, but if they want
> them, that's how much memory they'll take...

That looks very similar to an issue with PyPy's own GC, in which
ctypes.create_string_buffer() returns objects which tend to be GC'ed
late.  That's because the string buffer object in ctypes appears (to
PyPy's GC) to be just a small object, even though it actually
references a potentially large piece of raw memory.  Similarly, my
vague guess about the above is that the 17*11MB of memory are hold by
17 small objects which firefox's GC think don't need to be collected
particularly aggressively.


A bient?t,

Armin.


From renesd at gmail.com  Thu Dec 23 14:33:50 2010
From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=)
Date: Thu, 23 Dec 2010 13:33:50 +0000
Subject: [pypy-dev] pypy GC on large objects Re:  funding/popularity?
Message-ID: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>

Hello,

I think this is a case where the object returned by
ctypes.create_string_buffer() could use a correct __sizeof__ method
return value.  If pypy supported that, then the GC's could support
extensions, and 'opaque' data structures in C too a little more
nicely.

I think ctypes was started before __sizeof__ became available... so it
seems many of it's methods are not updated yet.  __sizeof__ is not
mandatory, so many extensions have not been updated to support it yet.

cheers,


On Thu, Dec 23, 2010 at 11:14 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Dima,
>
> On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
>> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
>> 13:43:23 PST ---
>> So what I see this page do, in horizontal mode, is create 17 canvases each of
>> which is width="816" height="3587". ?That means that each of them has a backing
>> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage
>> right there.
>>
>> I have no idea why they feel a need for 17 huge canvases, but if they want
>> them, that's how much memory they'll take...
>
> That looks very similar to an issue with PyPy's own GC, in which
> ctypes.create_string_buffer() returns objects which tend to be GC'ed
> late. ?That's because the string buffer object in ctypes appears (to
> PyPy's GC) to be just a small object, even though it actually
> references a potentially large piece of raw memory. ?Similarly, my
> vague guess about the above is that the 17*11MB of memory are hold by
> 17 small objects which firefox's GC think don't need to be collected
> particularly aggressively.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Thu Dec 23 14:38:59 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 23 Dec 2010 14:38:59 +0100
Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity?
In-Reply-To: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>
References: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>
Message-ID: <AANLkTimf7Vo7znvicz0fp1nBbwsRH_nPyNipggzoW7Sz@mail.gmail.com>

Hi Ren?,

On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield <renesd at gmail.com> wrote:
> I think this is a case where the object returned by
> ctypes.create_string_buffer() could use a correct __sizeof__ method
> return value. ?If pypy supported that, then the GC's could support
> extensions, and 'opaque' data structures in C too a little more
> nicely.

I think you are confusing levels.  There is no way the GC can call
some app-level Python method to get information about the objects it
frees (and when would it even call it?).  Remember that our GC is
written at a level where it works for any interpreter for any
language, not just Python.


A bient?t,

Armin.


From p.giarrusso at gmail.com  Thu Dec 23 15:09:47 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 23 Dec 2010 15:09:47 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTi=5ZjJYKN9WV7dk9J3LfTyoc0XiMYzWpxRAnoYR@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
	<AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
	<AANLkTi=5ZjJYKN9WV7dk9J3LfTyoc0XiMYzWpxRAnoYR@mail.gmail.com>
Message-ID: <AANLkTimajMibR8RrJgF-Roc9kdwt_cOfOpVK0gUBB7jC@mail.gmail.com>

On Thu, Dec 23, 2010 at 12:14, Armin Rigo <arigo at tunes.org> wrote:
> Hi Dima,
>
> On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
>> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
>> 13:43:23 PST ---
>> So what I see this page do, in horizontal mode, is create 17 canvases each of
>> which is width="816" height="3587". ?That means that each of them has a backing
>> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage
>> right there.
>>
>> I have no idea why they feel a need for 17 huge canvases, but if they want
>> them, that's how much memory they'll take...
>
> That looks very similar to an issue with PyPy's own GC, in which
> ctypes.create_string_buffer() returns objects which tend to be GC'ed
> late. ?That's because the string buffer object in ctypes appears (to
> PyPy's GC) to be just a small object, even though it actually
> references a potentially large piece of raw memory.

Bigger objects are not collected sooner by typical GC algorithms* -
they might just fill the heap more quickly and trigger earlier a GC
invocation.
>From your description, I guess that what you describe as "raw memory"
is not accounted in the stats triggering GC. That would explain the
behavior you describe, and suggest an easy fix. Indeed, one could wrap
the raw-memory allocator to create and update such stats; then the
heap-overflow check could consider them to decide whether to trigger
GC. The details of the modified heap-overflow check would probably not
be entirely trivial, but still doable.

* "Typical" GC algorithms here includes copying, mark-sweep, and
mark-compact collectors, including generational variants; even
collectors having a large object space (typically reclaimed through
mark-sweep) should still typically collect small objects often (by
using the heuristic described above to trigger GC).

Best regards

>?Similarly, my
> vague guess about the above is that the 17*11MB of memory are hold by
> 17 small objects which firefox's GC think don't need to be collected
> particularly aggressively.

-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From arigo at tunes.org  Thu Dec 23 15:17:58 2010
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 23 Dec 2010 15:17:58 +0100
Subject: [pypy-dev] funding/popularity?
In-Reply-To: <AANLkTimajMibR8RrJgF-Roc9kdwt_cOfOpVK0gUBB7jC@mail.gmail.com>
References: <AANLkTikGquQh8YisE=gtehWMc88q+D_Rem=yAq7-wNZM@mail.gmail.com>
	<AANLkTik5nhaFL77vcRTeMEtj_SLzt4GMzwjU8tFKHZRn@mail.gmail.com>
	<AANLkTin2iYMu5ZbW9LqV8A7BrERbwMagFdtrD_zvzc8Q@mail.gmail.com>
	<AANLkTi=WCSqn4N_ML3YYu4hqRsSBhztP9FaPCzaKDTAd@mail.gmail.com>
	<AANLkTimJyZyo3Z8X841W2B1BKkD37_RDyMbHZvA20-cD@mail.gmail.com>
	<AANLkTikc02WKABotOnjn5NwfDiXzmccwkE+1oGLJA-kT@mail.gmail.com>
	<AANLkTiksK2EKgfpR5WZDZuJMFiLvj_4SbWJqKLx1gbfr@mail.gmail.com>
	<AANLkTikPADtmZZez=VSY-Y=HWUBn9fEjtktT4BQfU8p=@mail.gmail.com>
	<AANLkTi=5ZjJYKN9WV7dk9J3LfTyoc0XiMYzWpxRAnoYR@mail.gmail.com>
	<AANLkTimajMibR8RrJgF-Roc9kdwt_cOfOpVK0gUBB7jC@mail.gmail.com>
Message-ID: <AANLkTinSUxyLBNFVqRarApYLXJgP71HMMxwarbbA=+3i@mail.gmail.com>

Hi Paolo,

On Thu, Dec 23, 2010 at 3:09 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> From your description, I guess that what you describe as "raw memory"
> is not accounted in the stats triggering GC. That would explain the
> behavior you describe, and suggest an easy fix. Indeed, one could wrap
> the raw-memory allocator to create and update such stats; then the
> heap-overflow check could consider them to decide whether to trigger
> GC. The details of the modified heap-overflow check would probably not
> be entirely trivial, but still doable.

Absolutely.  I will write it down in extradoc/planning/ for now.


Armin


From p.giarrusso at gmail.com  Thu Dec 23 15:31:44 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 23 Dec 2010 15:31:44 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
Message-ID: <AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>

On Thu, Dec 23, 2010 at 06:58, Dan Stromberg <drsalists at gmail.com> wrote:
> On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>>> Oh, I can't yet use alternative gc that obviates GIL then?
>>>
>>> Or am I totally confused and pypy still uses GIL for other reasons,
>>> e.g. globals dict safety?
>>
>> All of the above.
>
> On the topic of parallel dict safety... ?I've not played around with
> parallel dictionaries at all, but I've heard that treaps can be
> parallelized nicely, and they can be (and have been) set up to look
> much like a python dictionary. ?I admit, I coded one and put it at
> http://stromberg.dnsalias.org/~strombrg/treap/ - however, the
> implementation is not (yet?) suitable for parallel use.
>
> It's coded in such a way that it'll work as pure python or cython,
> depending on which of two m4 symbols you define. ?I've used the pure
> python version on pypy.
>
> Treaps tend to make just about everything O(log(n)), have good average
> performance, and are not as even performers as red-black trees (that
> is, they give a higher standard deviation, and a better average
> performance than red-black trees). ?And they don't play that nicely
> with locality of reference (caches) - things are intentionally
> randomized. ?But if you throw enough cores at them, they might still
> be a win compared to a big hash table with a big lock wrapped around
> it.

That might be interesting; however, the state of the art includes many
concurrent hash map alternatives which are way smarter than "a lock
wrapped around". Java >=1.5 has one, by Doug Lea*. An even better (and
more complex) one, by Cliff Click*, used in production on >500 cores,
and presented to JavaOne, can be found by googling:

nonblockinghashmap Cliff Click

A C implementation can be found at:
http://code.google.com/p/nbds/

Link found through this blog post of Cliff Click:
http://www.azulsystems.com/blog/cliff-click/2008-11-13-some-source-forge-threads-nbhm

However, I don't think you can code them in nowadays Python, because
it offers neither a Java-equivalent volatile attribute nor
compare-and-swap nor memory barriers, and these structures can't be
implemented with plain locks.

Best regards

* Doug Lea wrote the memory allocator used on Linux, is behind the 1.5
java.util.concurrent package, and much more. Cliff Click is a
high-performance JVM hacker.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From dimaqq at gmail.com  Thu Dec 23 20:30:20 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Thu, 23 Dec 2010 12:30:20 -0700
Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity?
In-Reply-To: <AANLkTimf7Vo7znvicz0fp1nBbwsRH_nPyNipggzoW7Sz@mail.gmail.com>
References: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>
	<AANLkTimf7Vo7znvicz0fp1nBbwsRH_nPyNipggzoW7Sz@mail.gmail.com>
Message-ID: <AANLkTim7tKTiRSOYOjQNWZ5oKBLBgEdfGoo-Fncs=DQW@mail.gmail.com>

Basically collecting this is hard:

dict(a=range(9**9))

large list is referenced, the object that holds the only reference is
small no matter how you look at it.

I guess it gets harder still if there are many small live objects, as
getting to this dict takes a while
(easier in this simple case with generataional collector, O(n) in general case)


On 23 December 2010 06:38, Armin Rigo <arigo at tunes.org> wrote:
> Hi Ren?,
>
> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield <renesd at gmail.com> wrote:
>> I think this is a case where the object returned by
>> ctypes.create_string_buffer() could use a correct __sizeof__ method
>> return value. ?If pypy supported that, then the GC's could support
>> extensions, and 'opaque' data structures in C too a little more
>> nicely.
>
> I think you are confusing levels. ?There is no way the GC can call
> some app-level Python method to get information about the objects it
> frees (and when would it even call it?). ?Remember that our GC is
> written at a level where it works for any interpreter for any
> language, not just Python.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From p.giarrusso at gmail.com  Fri Dec 24 12:39:29 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Fri, 24 Dec 2010 12:39:29 +0100
Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity?
In-Reply-To: <AANLkTim7tKTiRSOYOjQNWZ5oKBLBgEdfGoo-Fncs=DQW@mail.gmail.com>
References: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>
	<AANLkTimf7Vo7znvicz0fp1nBbwsRH_nPyNipggzoW7Sz@mail.gmail.com>
	<AANLkTim7tKTiRSOYOjQNWZ5oKBLBgEdfGoo-Fncs=DQW@mail.gmail.com>
Message-ID: <AANLkTinCnvzkGbRv=10JsWcBy4oVaHMtspAp5ZYKUCCr@mail.gmail.com>

On Thu, Dec 23, 2010 at 20:30, Dima Tisnek <dimaqq at gmail.com> wrote:
> Basically collecting this is hard:
>
> dict(a=range(9**9))
>
> large list is referenced, the object that holds the only reference is
> small no matter how you look at it.
First, usually (in most GC-ed languages) you can collect the list
before the dict. In PyPy, if finalizers are involved (is this the case
here? That'd be surprising), this is no more true.

However, object size is not the point. For standard algorithms, the
size of an object does not matter at all in deciding when it's
collected - I already discussed this in my other email in this thread,
and I noted what actually could happen in the examples described by
Armin, and your examples show that it is a good property. A large
object in the same heap can fill it up and trigger an earlier garbage
collection.

In general, if GC ran in the background (but it usually doesn't, and
not in PyPy) it could make sense to free objects sooner or later,
depending not on object size, but on "how much memory would be
'indirectly freed' by freeing this object". However, because of
sharing, answering this question is too complex (it requires
collecting data from the whole heap). Moreover, the whole thing makes
no sense at all with usual, stop-the-world collectors: the app is
stopped, then the whole young generation, or the whole heap, is
collected, then the app is resumed.

When separate heaps are involved (such as with ctypes, or with Large
Object Spaces, which avoid using a copy collector for large objects),
it is more complicated to ensure that the same property holds: you
need to consider stats of all heaps to decide whether to trigger GC.

> I guess it gets harder still if there are many small live objects, as
> getting to this dict takes a while
> (easier in this simple case with generataional collector, O(n) in general case)

Not sure what you mean; I can make sense of it (not fully) only with
an incremental collector, and they are still used seldom (especially,
not in PyPy).

Best regards

> On 23 December 2010 06:38, Armin Rigo <arigo at tunes.org> wrote:
>> Hi Ren?,
>>
>> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield <renesd at gmail.com> wrote:
>>> I think this is a case where the object returned by
>>> ctypes.create_string_buffer() could use a correct __sizeof__ method
>>> return value. ?If pypy supported that, then the GC's could support
>>> extensions, and 'opaque' data structures in C too a little more
>>> nicely.
>>
>> I think you are confusing levels. ?There is no way the GC can call
>> some app-level Python method to get information about the objects it
>> frees (and when would it even call it?). ?Remember that our GC is
>> written at a level where it works for any interpreter for any
>> language, not just Python.
>>
>>
>> A bient?t,
>>
>> Armin.
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev



-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From tobami at googlemail.com  Sat Dec 25 00:10:19 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Sat, 25 Dec 2010 00:10:19 +0100
Subject: [pypy-dev] Mercurial support for speed.pypy.org
Message-ID: <AANLkTikptfFdgXoviCtx0+EvV210+DGnu+4zER+-igV1@mail.gmail.com>

Hi all,

I finally added Mercurial support to Codespeed, following Antonio's
suggestion of parsing the (templated) command line output.Thanks to
everyone that helped!

I waited for today to give the awesome PyPy team a present.

Merry Christmas!

Miquel


From fijall at gmail.com  Sat Dec 25 10:07:20 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 25 Dec 2010 11:07:20 +0200
Subject: [pypy-dev] Mercurial support for speed.pypy.org
In-Reply-To: <AANLkTikptfFdgXoviCtx0+EvV210+DGnu+4zER+-igV1@mail.gmail.com>
References: <AANLkTikptfFdgXoviCtx0+EvV210+DGnu+4zER+-igV1@mail.gmail.com>
Message-ID: <AANLkTimLRJ=G8=Qst8kT_YP-gk985zsgwp0QAtvbgxBQ@mail.gmail.com>

On Sat, Dec 25, 2010 at 1:10 AM, Miquel Torres <tobami at googlemail.com> wrote:
> Hi all,
>
> I finally added Mercurial support to Codespeed, following Antonio's
> suggestion of parsing the (templated) command line output.Thanks to
> everyone that helped!
>
> I waited for today to give the awesome PyPy team a present.
>
> Merry Christmas!
>
> Miquel

That's a great present, thanks!

Merry Christmas!

Cheers,
fijal


From arigo at tunes.org  Sat Dec 25 19:41:08 2010
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 25 Dec 2010 19:41:08 +0100
Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity?
In-Reply-To: <AANLkTim7tKTiRSOYOjQNWZ5oKBLBgEdfGoo-Fncs=DQW@mail.gmail.com>
References: <AANLkTinbyZ+JMWcSw7PpTFdipczekysYs6NypasZPib7@mail.gmail.com>
	<AANLkTimf7Vo7znvicz0fp1nBbwsRH_nPyNipggzoW7Sz@mail.gmail.com>
	<AANLkTim7tKTiRSOYOjQNWZ5oKBLBgEdfGoo-Fncs=DQW@mail.gmail.com>
Message-ID: <AANLkTim=-3XZCcqRZr=go8Arvabry3952MMtmuZbRiL3@mail.gmail.com>

Hi Ren?,

On Thu, Dec 23, 2010 at 8:30 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
> Basically collecting this is hard:
>
> dict(a=range(9**9))

I think you missed the point of my original email.  I was talking
about GC-referenced objects that hold a reference to a large piece of
memory allocated outside the GC.  There is none here, and any GC
(including PyPy's) will do a correct job in collecting this.


A bient?t,

Armin.


From amauryfa at gmail.com  Sun Dec 26 21:44:19 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 26 Dec 2010 21:44:19 +0100
Subject: [pypy-dev] Fwd:  Mercurial support for speed.pypy.org
In-Reply-To: <AANLkTi=bj+fBF_fmWV4DRQGkJau4ubAa9XdvDthaA_nT@mail.gmail.com>
References: <AANLkTikptfFdgXoviCtx0+EvV210+DGnu+4zER+-igV1@mail.gmail.com>
	<AANLkTi=bj+fBF_fmWV4DRQGkJau4ubAa9XdvDthaA_nT@mail.gmail.com>
Message-ID: <AANLkTim8N3DB0cjJ4QkC7WmB2gpO+pqWxAZv0-CeiNdY@mail.gmail.com>

Hi,

2010/12/25 Miquel Torres <tobami at googlemail.com>

I finally added Mercurial support to Codespeed, following Antonio's
> suggestion of parsing the (templated) command line output.Thanks to
> everyone that helped!
>
> I waited for today to give the awesome PyPy team a present.
>

Thanks!

It seems that all changes are listed, even those who happen to be in another
branch.
Is it possible to filter the list and only display the "default" branch?

Cheers,

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101226/24e46f50/attachment.html>

From amauryfa at gmail.com  Sun Dec 26 23:03:29 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 26 Dec 2010 23:03:29 +0100
Subject: [pypy-dev] dlopen NotImplementedError
In-Reply-To: <AANLkTi=Ej3StS7Z_caT4Dgc_ErpYThmNZjxiiVtHRC+6@mail.gmail.com>
References: <AA8CF80E-7F27-4585-8E2F-93D90EF24F41@me.com>
	<AANLkTi=x0Qj1JT9OVjm+Z=wRKc_J3Ld_N8kgWjy5_eY8@mail.gmail.com>
	<20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain>
	<AANLkTinbhDFxhvwuLt8HBCMjHjp4q-wijTriyfJ4zrz9@mail.gmail.com>
	<AANLkTi=Ej3StS7Z_caT4Dgc_ErpYThmNZjxiiVtHRC+6@mail.gmail.com>
Message-ID: <AANLkTikF-RHVTziN1HWwUMJbrS449EvGFYNRSDPFaje5@mail.gmail.com>

Hi,

2010/12/23 Maciej Fijalkowski <fijall at gmail.com>

> CDLL(None) (or LoadLibrary(None)) is supposed to return to you the
> whole namespace (all loaded libraries). It's unsupported as far as I
> can tell.


It should work by now (on Unix platforms of course).

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20101226/fad9a321/attachment.html>

From dimaqq at gmail.com  Mon Dec 27 07:00:30 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Sun, 26 Dec 2010 23:00:30 -0700
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
Message-ID: <AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>

(dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
constant factor for locks, etc, might make it worthwhile

(paolo) non blocking hash maps: memory barriers can be quite costly too

different options will have to be implemented and tested when we get there,
speaking on which, is there a test dict load?
does anyone have some profiling data, what portion of runtime is spent
reading and writing attributes and globals anyways?

while we are on the subject, is there a plan to provide different
levels of sparseness for different levels of name lookup?
for example globals vs builtins, first needs to be quite sparce so
that builtins show through well, second hardly, because there's
nowhere else to look if builtins don't have the name.
then the tradeoff could be more dynamic, that is frequently accessed
dicts could be e.g. more sparse and rarely accessed more compact?
(obviousely more sparse is not always better, e.g. in terms of cpu cache)
of course "frequently accessed" is not as easy as frequently ran code,
e.g. here "class A: ...; while 1: A().func()", func lookup occurs in
different objects, yet it is same logical operation.

come to think of it, is there any point in polymorphic dicts, e.g.
attribute access could be imeplemented as a near-perfect compact hash
map if attribute names change rarely, while regular dict()s are
expected to change keys often.

Anyhow, back to parallel interpretation, Is there an issue or page
that tracks what's needed for parallel interpretation?
so far  mentioned: gc, dict, c api

Btw, I think that jit is more important at the moment, but time comes
when jit juice has been mostly squeezed out ;-)


d.

On 23 December 2010 07:31, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Thu, Dec 23, 2010 at 06:58, Dan Stromberg <drsalists at gmail.com> wrote:
>> On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>>>> Oh, I can't yet use alternative gc that obviates GIL then?
>>>>
>>>> Or am I totally confused and pypy still uses GIL for other reasons,
>>>> e.g. globals dict safety?
>>>
>>> All of the above.
>>
>> On the topic of parallel dict safety... ?I've not played around with
>> parallel dictionaries at all, but I've heard that treaps can be
>> parallelized nicely, and they can be (and have been) set up to look
>> much like a python dictionary. ?I admit, I coded one and put it at
>> http://stromberg.dnsalias.org/~strombrg/treap/ - however, the
>> implementation is not (yet?) suitable for parallel use.
>>
>> It's coded in such a way that it'll work as pure python or cython,
>> depending on which of two m4 symbols you define. ?I've used the pure
>> python version on pypy.
>>
>> Treaps tend to make just about everything O(log(n)), have good average
>> performance, and are not as even performers as red-black trees (that
>> is, they give a higher standard deviation, and a better average
>> performance than red-black trees). ?And they don't play that nicely
>> with locality of reference (caches) - things are intentionally
>> randomized. ?But if you throw enough cores at them, they might still
>> be a win compared to a big hash table with a big lock wrapped around
>> it.
>
> That might be interesting; however, the state of the art includes many
> concurrent hash map alternatives which are way smarter than "a lock
> wrapped around". Java >=1.5 has one, by Doug Lea*. An even better (and
> more complex) one, by Cliff Click*, used in production on >500 cores,
> and presented to JavaOne, can be found by googling:
>
> nonblockinghashmap Cliff Click
>
> A C implementation can be found at:
> http://code.google.com/p/nbds/
>
> Link found through this blog post of Cliff Click:
> http://www.azulsystems.com/blog/cliff-click/2008-11-13-some-source-forge-threads-nbhm
>
> However, I don't think you can code them in nowadays Python, because
> it offers neither a Java-equivalent volatile attribute nor
> compare-and-swap nor memory barriers, and these structures can't be
> implemented with plain locks.
>
> Best regards
>
> * Doug Lea wrote the memory allocator used on Linux, is behind the 1.5
> java.util.concurrent package, and much more. Cliff Click is a
> high-performance JVM hacker.
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From william.leslie.ttg at gmail.com  Mon Dec 27 07:30:35 2010
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Mon, 27 Dec 2010 17:30:35 +1100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
Message-ID: <AANLkTikP3qn7MFfK0cS4EZdy_7Rus4E7f7p_f0wHY9aV@mail.gmail.com>

On 27 December 2010 17:00, Dima Tisnek <dimaqq at gmail.com> wrote:
> while we are on the subject, is there a plan to provide different
> levels of sparseness for different levels of name lookup?
> for example globals vs builtins, first needs to be quite sparce so
> that builtins show through well, second hardly, because there's
> nowhere else to look if builtins don't have the name.
> then the tradeoff could be more dynamic, that is frequently accessed
> dicts could be e.g. more sparse and rarely accessed more compact?
> (obviousely more sparse is not always better, e.g. in terms of cpu cache)
> of course "frequently accessed" is not as easy as frequently ran code,
> e.g. here "class A: ...; while 1: A().func()", func lookup occurs in
> different objects, yet it is same logical operation.
>
> come to think of it, is there any point in polymorphic dicts, e.g.
> attribute access could be imeplemented as a near-perfect compact hash
> map if attribute names change rarely, while regular dict()s are
> expected to change keys often.

No, not really, but pypy already heavily optimises this case - see
mapdicts, celldicts, and sharing dicts.  Celldict is commonly used for
module dictionaries, and emulates a direct pointer to the value, which
can be cached along side the LOAD_GLOBAL instruction.  I implemented a
similar system (pre-computed array based on known module-global names)
a few years back in pypy (pre-jit) and got a 6% speedup over regular
dicts, but celldicts get a similar speedup and are more generally
applicable.

As for perfect hashing: our existing mechanism for hashing beats it,
hands down.  In cpython at least, I haven't checked the pypy source on
this topic, the hash of a string is cached on the string object
itself, which means in the case of identifiers no hash is ever
computed on global lookup.  The only thing that could really be faster
is something like a slot on the symbol itself.

Celldicts move the synchronisation point out of the hash table and
into the entry for common cases, which changes the synchronisation
question significantly.

> Btw, I think that jit is more important at the moment, but time comes
> when jit juice has been mostly squeezed out ;-)

There are occasional memory-model discussions, but at the end of the
day what will probably happen is the people who step up to do the work
to implement it will probably also get to do most of the design work.

-- 
William Leslie


From cfbolz at gmx.de  Mon Dec 27 09:31:51 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 27 Dec 2010 09:31:51 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
Message-ID: <4D184EF7.1040200@gmx.de>

On 12/27/2010 07:00 AM, Dima Tisnek wrote:
> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
> constant factor for locks, etc, might make it worthwhile
>
> (paolo) non blocking hash maps: memory barriers can be quite costly too
>
> different options will have to be implemented and tested when we get there,
> speaking on which, is there a test dict load?
> does anyone have some profiling data, what portion of runtime is spent
> reading and writing attributes and globals anyways?
>
> while we are on the subject, is there a plan to provide different
> levels of sparseness for different levels of name lookup?
> for example globals vs builtins, first needs to be quite sparce so
> that builtins show through well, second hardly, because there's
> nowhere else to look if builtins don't have the name.
> then the tradeoff could be more dynamic, that is frequently accessed
> dicts could be e.g. more sparse and rarely accessed more compact?
> (obviousely more sparse is not always better, e.g. in terms of cpu cache)
> of course "frequently accessed" is not as easy as frequently ran code,
> e.g. here "class A: ...; while 1: A().func()", func lookup occurs in
> different objects, yet it is same logical operation.
>
> come to think of it, is there any point in polymorphic dicts, e.g.
> attribute access could be imeplemented as a near-perfect compact hash
> map if attribute names change rarely, while regular dict()s are
> expected to change keys often.

All these thoughts go into the wrong direction, imo. The JIT removes 
nearly all dictionary accesses to global dicts, instance and class 
dicts. Even without the JIT, purely interpreting things, there are 
caches that bypass most dict lookups.

The only case where improving dicts would help is for user-defined 
dictionaries, i.e. when you write dicts with curly braces. that is 
basically very hard, because the usage patterns vary widely and a lot of 
work would be necessary to find common uses.

Cheers,

Carl Friedrich


From anto.cuni at gmail.com  Mon Dec 27 09:35:23 2010
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 27 Dec 2010 09:35:23 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <4D184EF7.1040200@gmx.de>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<4D184EF7.1040200@gmx.de>
Message-ID: <4D184FCB.8050404@gmail.com>

On 27/12/10 09:31, Carl Friedrich Bolz wrote:

> The only case where improving dicts would help is for user-defined 
> dictionaries, i.e. when you write dicts with curly braces.

then it's easy to optimize, you just write "dict()" instead of "{}".

(sorry, I could not resist :-))


From tobami at googlemail.com  Mon Dec 27 11:27:03 2010
From: tobami at googlemail.com (Miquel Torres)
Date: Mon, 27 Dec 2010 11:27:03 +0100
Subject: [pypy-dev] Fwd: Mercurial support for speed.pypy.org
In-Reply-To: <AANLkTim8N3DB0cjJ4QkC7WmB2gpO+pqWxAZv0-CeiNdY@mail.gmail.com>
References: <AANLkTikptfFdgXoviCtx0+EvV210+DGnu+4zER+-igV1@mail.gmail.com>
	<AANLkTi=bj+fBF_fmWV4DRQGkJau4ubAa9XdvDthaA_nT@mail.gmail.com>
	<AANLkTim8N3DB0cjJ4QkC7WmB2gpO+pqWxAZv0-CeiNdY@mail.gmail.com>
Message-ID: <AANLkTinQFXCqS9wXGTJHJuyLA_V=3m9SYRiqfnhMofSd@mail.gmail.com>

Oh, ok. Fixed now by adding "-b default " to the hg log command.

(still learning mercurial :)


2010/12/26 Amaury Forgeot d'Arc <amauryfa at gmail.com>:
> Hi,
>
> 2010/12/25 Miquel Torres <tobami at googlemail.com>
>>
>> I finally added Mercurial support to Codespeed, following Antonio's
>> suggestion of parsing the (templated) command line output.Thanks to
>> everyone that helped!
>>
>> I waited for today to give the awesome PyPy team a present.
>
> Thanks!
> It seems that all changes are listed, even those who happen to be in another
> branch.
> Is it possible to filter the list and only display the "default" branch?
> Cheers,
>
> --
> Amaury Forgeot d'Arc
>
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>


From p.giarrusso at gmail.com  Mon Dec 27 18:25:22 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 27 Dec 2010 18:25:22 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <4D184EF7.1040200@gmx.de>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<4D184EF7.1040200@gmx.de>
Message-ID: <AANLkTikPxrOeep_k+q2rKx-APv-69ezDXLHjjm81hcJm@mail.gmail.com>

On Mon, Dec 27, 2010 at 09:31, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> On 12/27/2010 07:00 AM, Dima Tisnek wrote:
>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
>> constant factor for locks, etc, might make it worthwhile
>>
>> (paolo) non blocking hash maps: memory barriers can be quite costly too
>>
>> different options will have to be implemented and tested when we get there,
>> speaking on which, is there a test dict load?
>> does anyone have some profiling data, what portion of runtime is spent
>> reading and writing attributes and globals anyways?
>>
>> while we are on the subject, is there a plan to provide different
>> levels of sparseness for different levels of name lookup?
>> for example globals vs builtins, first needs to be quite sparce so
>> that builtins show through well, second hardly, because there's
>> nowhere else to look if builtins don't have the name.
>> then the tradeoff could be more dynamic, that is frequently accessed
>> dicts could be e.g. more sparse and rarely accessed more compact?
>> (obviousely more sparse is not always better, e.g. in terms of cpu cache)
>> of course "frequently accessed" is not as easy as frequently ran code,
>> e.g. here "class A: ...; while 1: A().func()", func lookup occurs in
>> different objects, yet it is same logical operation.
>>
>> come to think of it, is there any point in polymorphic dicts, e.g.
>> attribute access could be imeplemented as a near-perfect compact hash
>> map if attribute names change rarely, while regular dict()s are
>> expected to change keys often.
>
> All these thoughts go into the wrong direction, imo. The JIT removes
> nearly all dictionary accesses to global dicts, instance and class
> dicts. Even without the JIT, purely interpreting things, there are
> caches that bypass most dict lookups.

That's very interesting. However, aren't such caches also hash maps in
the end (unless you do inline caching in your interpreter, like in
Ruby 1.9)? I remember reading so in PyPy's docs; moreover, that's a
standard optimization for method lookup.

Sharing such caches between different interpreter threads would be
potentially useful, if that implied no expensive synchronization for
readers - which is possible for instance on x86 (read barriers are for
free), by using persistent data structures. Writes to such caches
should (hopefully) be rare enough to make the extra synchronization
not too expensive, however this needs benchmarking.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From p.giarrusso at gmail.com  Mon Dec 27 18:57:55 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 27 Dec 2010 18:57:55 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
Message-ID: <AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>

On Mon, Dec 27, 2010 at 07:00, Dima Tisnek <dimaqq at gmail.com> wrote:
> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
> constant factor for locks, etc, might make it worthwhile

"7.2"? What is that? Do you mean 7.2=log2(no. of builtins)?

> (paolo) non blocking hash maps: memory barriers can be quite costly too

Well, you just _cannot_ avoid them, whatever is your fancy data structure.
You can only move most of the cost (or all of it) to write operations
- which is helpful if reads are most common, a case which you discuss
in your mail.

== Why memory barriers are required (in more detail) ==
If you use persistent data structures, also called copy-on-write or
purely functional (and that's the most obvious way to use a tree in a
parallel setting), the whole synchronization problem is reduced to
exchanging a pointer (to the root of the tree) between a writer and a
reader thread.
Let's consider this in the Java memory model (the state of the art).
In this model the writer thread has to use an expensive StoreLoad
barrier, i.e. mfence on x86 [1], costing more or less as much as a
cache miss, while the reader needs just a cheap Load* barrier (for
free on x86, cheap on other processor).
If you don't use a volatile field, there is very little guarantee
about what your program means, and it is almost always buggy; the
equivalent in the new C++ memory model (which is otherwise very
similar) gives completely undefined semantics (and even crashes are
possible in practice, for instance if the reader thread sees a
not-fully-initialized object because of the race and invokes a virtual
method on it).

[1] http://g.oswego.edu/dl/jmm/cookbook.html

== Other stuff ==
> different options will have to be implemented and tested when we get there,
> speaking on which, is there a test dict load?
> does anyone have some profiling data, what portion of runtime is spent
> reading and writing attributes and globals anyways?

I'd be interested as well in such data.

> Anyhow, back to parallel interpretation, Is there an issue or page
> that tracks what's needed for parallel interpretation?
> so far ?mentioned: gc, dict, c api

> Btw, I think that jit is more important at the moment, but time comes
> when jit juice has been mostly squeezed out ;-)

-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From dimaqq at gmail.com  Mon Dec 27 19:54:41 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Mon, 27 Dec 2010 11:54:41 -0700
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>
Message-ID: <AANLkTinL2qSieqD=juu1DzsSk34+AUkAQCj0W_WTPbwW@mail.gmail.com>

what do you think of liburcu? lttng.org/urcu

p.s. if/when I get around to profiling pypy, which is something I want
to do, I'll be sure to share the results here.

d.

On 27 December 2010 10:57, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek <dimaqq at gmail.com> wrote:
>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
>> constant factor for locks, etc, might make it worthwhile
>
> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)?
>
>> (paolo) non blocking hash maps: memory barriers can be quite costly too
>
> Well, you just _cannot_ avoid them, whatever is your fancy data structure.
> You can only move most of the cost (or all of it) to write operations
> - which is helpful if reads are most common, a case which you discuss
> in your mail.
>
> == Why memory barriers are required (in more detail) ==
> If you use persistent data structures, also called copy-on-write or
> purely functional (and that's the most obvious way to use a tree in a
> parallel setting), the whole synchronization problem is reduced to
> exchanging a pointer (to the root of the tree) between a writer and a
> reader thread.
> Let's consider this in the Java memory model (the state of the art).
> In this model the writer thread has to use an expensive StoreLoad
> barrier, i.e. mfence on x86 [1], costing more or less as much as a
> cache miss, while the reader needs just a cheap Load* barrier (for
> free on x86, cheap on other processor).
> If you don't use a volatile field, there is very little guarantee
> about what your program means, and it is almost always buggy; the
> equivalent in the new C++ memory model (which is otherwise very
> similar) gives completely undefined semantics (and even crashes are
> possible in practice, for instance if the reader thread sees a
> not-fully-initialized object because of the race and invokes a virtual
> method on it).
>
> [1] http://g.oswego.edu/dl/jmm/cookbook.html
>
> == Other stuff ==
>> different options will have to be implemented and tested when we get there,
>> speaking on which, is there a test dict load?
>> does anyone have some profiling data, what portion of runtime is spent
>> reading and writing attributes and globals anyways?
>
> I'd be interested as well in such data.
>
>> Anyhow, back to parallel interpretation, Is there an issue or page
>> that tracks what's needed for parallel interpretation?
>> so far ?mentioned: gc, dict, c api
>
>> Btw, I think that jit is more important at the moment, but time comes
>> when jit juice has been mostly squeezed out ;-)
>
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
>


From p.giarrusso at gmail.com  Mon Dec 27 20:50:08 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 27 Dec 2010 20:50:08 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTinL2qSieqD=juu1DzsSk34+AUkAQCj0W_WTPbwW@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>
	<AANLkTinL2qSieqD=juu1DzsSk34+AUkAQCj0W_WTPbwW@mail.gmail.com>
Message-ID: <AANLkTi=yzbDJ42ewzYX5ySrdM+vcaf38TXdh0GzOYzh_@mail.gmail.com>

On Mon, Dec 27, 2010 at 19:54, Dima Tisnek <dimaqq at gmail.com> wrote:
> what do you think of liburcu? lttng.org/urcu

I've studied RCU in the Linux kernel (from which URCU derives) and
thought for a long time of using it for this problem. In short, once
you have GC, RCU (as in the Linux kernel) becomes (almost) trivial,
because RCU is almost entirely about how to delay freeing objects.
The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt
in the Linux sources.

So, you need to just read about persistent data structures, and
remember that persistence is not the point - data structures which are
just almost persistent can also be OK, if you get them right. If you
replace a value in a dictionary, for instance, you don't necessarily
need to keep the old version around, as long as you the value fits in
a memory word so that it can be atomically replaced.
http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/

Speaking of libraries, you might want to look at this for the atomic
primitives you'll need:
www.hpl.hp.com/research/linux/atomic_ops/
I'm more used to the primitives of the Linux kernel, I remember I
didn't fully like that library, but IIRC it was a matter of taste -
the library is from Hans Boehm, which makes it trustworthy.

> p.s. if/when I get around to profiling pypy, which is something I want
> to do, I'll be sure to share the results here.

Great, thanks!

> d.
>
> On 27 December 2010 10:57, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
>>> constant factor for locks, etc, might make it worthwhile
>>
>> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)?
>>
>>> (paolo) non blocking hash maps: memory barriers can be quite costly too
>>
>> Well, you just _cannot_ avoid them, whatever is your fancy data structure.
>> You can only move most of the cost (or all of it) to write operations
>> - which is helpful if reads are most common, a case which you discuss
>> in your mail.
>>
>> == Why memory barriers are required (in more detail) ==
>> If you use persistent data structures, also called copy-on-write or
>> purely functional (and that's the most obvious way to use a tree in a
>> parallel setting), the whole synchronization problem is reduced to
>> exchanging a pointer (to the root of the tree) between a writer and a
>> reader thread.
>> Let's consider this in the Java memory model (the state of the art).
>> In this model the writer thread has to use an expensive StoreLoad
>> barrier, i.e. mfence on x86 [1], costing more or less as much as a
>> cache miss, while the reader needs just a cheap Load* barrier (for
>> free on x86, cheap on other processor).
>> If you don't use a volatile field, there is very little guarantee
>> about what your program means, and it is almost always buggy; the
>> equivalent in the new C++ memory model (which is otherwise very
>> similar) gives completely undefined semantics (and even crashes are
>> possible in practice, for instance if the reader thread sees a
>> not-fully-initialized object because of the race and invokes a virtual
>> method on it).
>>
>> [1] http://g.oswego.edu/dl/jmm/cookbook.html
>>
>> == Other stuff ==
>>> different options will have to be implemented and tested when we get there,
>>> speaking on which, is there a test dict load?
>>> does anyone have some profiling data, what portion of runtime is spent
>>> reading and writing attributes and globals anyways?
>>
>> I'd be interested as well in such data.
>>
>>> Anyhow, back to parallel interpretation, Is there an issue or page
>>> that tracks what's needed for parallel interpretation?
>>> so far ?mentioned: gc, dict, c api
>>
>>> Btw, I think that jit is more important at the moment, but time comes
>>> when jit juice has been mostly squeezed out ;-)
>>
>> --
>> Paolo Giarrusso - Ph.D. Student
>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>
>



-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/


From dimaqq at gmail.com  Tue Dec 28 02:47:14 2010
From: dimaqq at gmail.com (Dima Tisnek)
Date: Mon, 27 Dec 2010 18:47:14 -0700
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTi=yzbDJ42ewzYX5ySrdM+vcaf38TXdh0GzOYzh_@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>
	<AANLkTinL2qSieqD=juu1DzsSk34+AUkAQCj0W_WTPbwW@mail.gmail.com>
	<AANLkTi=yzbDJ42ewzYX5ySrdM+vcaf38TXdh0GzOYzh_@mail.gmail.com>
Message-ID: <AANLkTimfa_=BgHEH5WLT0jH_7XG6HLc3qd1HH_SG3JNx@mail.gmail.com>

Thanks for the RTFP pointer, the Triplett ref, hash tab via rt is
really interesting!

I think I'll have to read it again and again to get all out of it...

This brings me to a new question: how much is pypy python memory model
allowed to diverge from that of cpython?

For example, dict lookups, etc, could be relativistic in the absense
of explicit synchronization, that is, dict updates are not quaranteed
to propagate [fast] until [some] therad acquires a lock [or
something]. Of course the order of dict mutation has to be preserved.
Would this be a worthwhile departure from GIL-based monotonic world?

That is thread-safe code, that uses locks or some other mechanism
[known to pypy] would see operations timely and in order, while code
that doesn't synchronize its operations may see somewhat outdated
shared state, as if threads executed in slightly different global
order.


About profiling, how do I distinguish between attribute vs globals vs
builtins lookups in the profile?

d.


On 27 December 2010 12:50, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> On Mon, Dec 27, 2010 at 19:54, Dima Tisnek <dimaqq at gmail.com> wrote:
>> what do you think of liburcu? lttng.org/urcu
>
> I've studied RCU in the Linux kernel (from which URCU derives) and
> thought for a long time of using it for this problem. In short, once
> you have GC, RCU (as in the Linux kernel) becomes (almost) trivial,
> because RCU is almost entirely about how to delay freeing objects.
> The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt
> in the Linux sources.
>
> So, you need to just read about persistent data structures, and
> remember that persistence is not the point - data structures which are
> just almost persistent can also be OK, if you get them right. If you
> replace a value in a dictionary, for instance, you don't necessarily
> need to keep the old version around, as long as you the value fits in
> a memory word so that it can be atomically replaced.
> http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/
>
> Speaking of libraries, you might want to look at this for the atomic
> primitives you'll need:
> www.hpl.hp.com/research/linux/atomic_ops/
> I'm more used to the primitives of the Linux kernel, I remember I
> didn't fully like that library, but IIRC it was a matter of taste -
> the library is from Hans Boehm, which makes it trustworthy.
>
>> p.s. if/when I get around to profiling pypy, which is something I want
>> to do, I'll be sure to share the results here.
>
> Great, thanks!
>
>> d.
>>
>> On 27 December 2010 10:57, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>>> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
>>>> constant factor for locks, etc, might make it worthwhile
>>>
>>> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)?
>>>
>>>> (paolo) non blocking hash maps: memory barriers can be quite costly too
>>>
>>> Well, you just _cannot_ avoid them, whatever is your fancy data structure.
>>> You can only move most of the cost (or all of it) to write operations
>>> - which is helpful if reads are most common, a case which you discuss
>>> in your mail.
>>>
>>> == Why memory barriers are required (in more detail) ==
>>> If you use persistent data structures, also called copy-on-write or
>>> purely functional (and that's the most obvious way to use a tree in a
>>> parallel setting), the whole synchronization problem is reduced to
>>> exchanging a pointer (to the root of the tree) between a writer and a
>>> reader thread.
>>> Let's consider this in the Java memory model (the state of the art).
>>> In this model the writer thread has to use an expensive StoreLoad
>>> barrier, i.e. mfence on x86 [1], costing more or less as much as a
>>> cache miss, while the reader needs just a cheap Load* barrier (for
>>> free on x86, cheap on other processor).
>>> If you don't use a volatile field, there is very little guarantee
>>> about what your program means, and it is almost always buggy; the
>>> equivalent in the new C++ memory model (which is otherwise very
>>> similar) gives completely undefined semantics (and even crashes are
>>> possible in practice, for instance if the reader thread sees a
>>> not-fully-initialized object because of the race and invokes a virtual
>>> method on it).
>>>
>>> [1] http://g.oswego.edu/dl/jmm/cookbook.html
>>>
>>> == Other stuff ==
>>>> different options will have to be implemented and tested when we get there,
>>>> speaking on which, is there a test dict load?
>>>> does anyone have some profiling data, what portion of runtime is spent
>>>> reading and writing attributes and globals anyways?
>>>
>>> I'd be interested as well in such data.
>>>
>>>> Anyhow, back to parallel interpretation, Is there an issue or page
>>>> that tracks what's needed for parallel interpretation?
>>>> so far ?mentioned: gc, dict, c api
>>>
>>>> Btw, I think that jit is more important at the moment, but time comes
>>>> when jit juice has been mostly squeezed out ;-)
>>>
>>> --
>>> Paolo Giarrusso - Ph.D. Student
>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>>
>>
>
>
>
> --
> Paolo Giarrusso - Ph.D. Student
> http://www.informatik.uni-marburg.de/~pgiarrusso/
>


From cfbolz at gmx.de  Thu Dec 30 01:27:02 2010
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 30 Dec 2010 01:27:02 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTikPxrOeep_k+q2rKx-APv-69ezDXLHjjm81hcJm@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<4D184EF7.1040200@gmx.de>
	<AANLkTikPxrOeep_k+q2rKx-APv-69ezDXLHjjm81hcJm@mail.gmail.com>
Message-ID: <4D1BD1D6.7010805@gmx.de>

On 12/27/2010 06:25 PM, Paolo Giarrusso wrote:
>> All these thoughts go into the wrong direction, imo. The JIT removes
>> nearly all dictionary accesses to global dicts, instance and class
>> dicts. Even without the JIT, purely interpreting things, there are
>> caches that bypass most dict lookups.
>
> That's very interesting. However, aren't such caches also hash maps in
> the end (unless you do inline caching in your interpreter, like in
> Ruby 1.9)? I remember reading so in PyPy's docs; moreover, that's a
> standard optimization for method lookup.

Indeed, some of the caches are again hash maps, e.g. the method cache is 
a global hash map. However, since it is a cache of a fixed size, its 
implementation is a lot simpler than that of dictionaries.

Some of the caches (like that of attribute access) are indeed inline 
caches and thus don't need hash maps at all. At some point the method 
cache was an inline cache as well, but that didn't seem to actually help 
much.

In general it seems that eliminating single dict lookups is rarely worth 
it in PyPy. We also had an inline global dict lookup cache at some 
point, but it didn't give an interesting enough effect to keep it. I'm 
talking purely about the interpreter here, of course, the jit gets rid 
of all these lookups anyway.

> Sharing such caches between different interpreter threads would be
> potentially useful, if that implied no expensive synchronization for
> readers - which is possible for instance on x86 (read barriers are for
> free), by using persistent data structures. Writes to such caches
> should (hopefully) be rare enough to make the extra synchronization
> not too expensive, however this needs benchmarking.

Again, this is all idle speculation. If PyPy wants to move towards a 
GIL-less implementation, a *huge* amount of problems would need to be 
solved first before such optimizations become important. We would need 
to fix the GC. We would need to think about the memory model of RPython 
in the presence of multi-CPU threading, and that of Python. At the 
moment I don't see that work happening, because nobody in the current 
core contributor group is qualified or interested.

Yes, in a sense that's not very forward-looking, given the move to 
multi-core. Otoh it's not clear to me that shared-memory multithreading 
is really the approach that makes most sense to PyPy in its current state.

Cheers,

Carl Friedrich


From p.giarrusso at gmail.com  Fri Dec 31 17:59:05 2010
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Fri, 31 Dec 2010 17:59:05 +0100
Subject: [pypy-dev] 1.4.1 threading
In-Reply-To: <AANLkTimfa_=BgHEH5WLT0jH_7XG6HLc3qd1HH_SG3JNx@mail.gmail.com>
References: <AANLkTimFB-Kf5vmN4Yx0QSJ+A6bJAyQO_og6jfF-knvo@mail.gmail.com>
	<AANLkTinHrKank0mQN7v_kmGQR8Swj_hqjvGLxba0910Z@mail.gmail.com>
	<AANLkTim61vvtGUWcgUYaGSZBUVm0kRP3zZ+xfW0dUJLk@mail.gmail.com>
	<AANLkTi=-raQz6FUv1t_egfzcd439oKPHTmpL_K1hiQ8c@mail.gmail.com>
	<AANLkTi=EOU9=A2oBcGC08rtvCOv4qKgBJh+3AkFg_LSB@mail.gmail.com>
	<AANLkTimHmty4K0tMZa7=i7+RkRmYzhBHRA5kgkyLf_v+@mail.gmail.com>
	<AANLkTi=cg9gPouohaHOLzW6Nwa2hhXoPGmTz7Mg3P-=F@mail.gmail.com>
	<AANLkTi=3qj=8OA0pDFWt=k2GgNBtrGsddf5VMzWub53F@mail.gmail.com>
	<AANLkTinL2qSieqD=juu1DzsSk34+AUkAQCj0W_WTPbwW@mail.gmail.com>
	<AANLkTi=yzbDJ42ewzYX5ySrdM+vcaf38TXdh0GzOYzh_@mail.gmail.com>
	<AANLkTimfa_=BgHEH5WLT0jH_7XG6HLc3qd1HH_SG3JNx@mail.gmail.com>
Message-ID: <AANLkTi=HU7-1C2D=p5WaT8KL71GHAPHw_SsZ2WYEZOof@mail.gmail.com>

On Tue, Dec 28, 2010 at 02:47, Dima Tisnek <dimaqq at gmail.com> wrote:
> Thanks for the RTFP pointer, the Triplett ref, hash tab via rt is
> really interesting!
>
> I think I'll have to read it again and again to get all out of it...

Ah, I remember that! Never found enough time to read up that... also I
remember that I didn't like it, at the time, because it seemed to give
a new name to existing design practice in the RCU community. I didn't
realize that even if the article just described existing work, the
generalization alone might be an important contribution.

> This brings me to a new question: how much is pypy python memory model
> allowed to diverge from that of cpython?

In fact, Jython and IronPython already have different memory models:
code.google.com/p/unladen-swallow/wiki/MemoryModel

But the most important answer is that to avoid reinventing the wheel,
one needs to know the state of the art, i.e. the work by Sarita Adve,
Jeremy Manson, William Pugh and Hans Boehm on memory models for Java
and C++.

The simplest introduction I know of comes in this blog post by Jeremy Manson:
http://jeremymanson.blogspot.com/2008/11/what-volatile-means-in-java.html

As a matter of fact, this work is unsurprisingly often totally ignored
in many discussions about Python's memory model. I'm not surprised
because it's complex stuff, and understanding it doesn't matter for
today's Python programming.

The key concept of these models is that _properly synchronized_
programs get "sequential consistent" semantics: the programs behaves
as if all memory operations were interleaved in a total order on a
single coherent memory, without reorderings due to cache, compiler
optimizations and so on. Sequential consistency is the key property of
what you call "GIL-based monotonic world".
Exactly as you suggested for dictionaries, but more in general,
programs which are not properly synchronized (which exhibit so-called
data races) are more-or-less left on their own. On this topic, the
Java and C++ models are different, and that leaves a design space
open, but one which is already somewhat smaller, and which doesn't
affect the semantics of properly synchronized programs. Java tries to
define the semantics of data races, C++ just gives up.

* Unlike C++ and like Java, it is probably important to guarantee that
programs with data races (not properly synchronized memory accesses)
don't crash the interpreter or corrupt native memory.
You don't want native code in the interpreter to crash or to loop if
the programmer forgets synchronization - simply because that's very
difficult to debug, short of running a debugger on the interpreter
itself (or fixing the code without a debugger at all).

It is not clear, though, if one should give specific semantics to
programs with data races, other than ensure the interpreter doesn't
crash: basically it looks too complex. For whoever understands the
rest of the Java Memory Model, this blog post explains the semantics
they use to model this.
http://jeremymanson.blogspot.com/2007/08/causality-and-java-memory-model.html

* Another difference between Java and C++ is about security
guarantees: it is crucial in Java that untrusted code cannot read
sensitive data (e.g., a password) managed by trusted code, not even
through a data race. Some kinds of behavior would allow
"out-of-thin-air values", which in practice might be such sensitive
data, if memory is zeroed by the memory allocator but one thread still
sees the old value.

I believe the most up-to-date introduction, with a useful
bibliography, is by S. Adve and H. Boehm [1]. Then you will probably
want to read [2] and [3] about C++ and Java. Through Google Scholar,
you should find you copies of all of this on CiteseerX.
[1] Sarita V. Adve and Hans-J. Boehm. 2010. Memory models: a case for
rethinking parallel languages and hardware. Communications of the ACM.
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.158.4288
[2] Hans-J. Boehm and Sarita V. Adve. 2008. Foundations of the C++
concurrency memory model. PLDI '08
[3] Jeremy Manson, William Pugh, and Sarita V. Adve. 2005. The Java
memory model. POPL '05.

> For example, dict lookups, etc, could be relativistic in the absense
> of explicit synchronization, that is, dict updates are not quaranteed
> to propagate [fast] until [some] therad acquires a lock [or
> something].

Talking about dictionaries restricts the scope unnecessarily. You want
to relax these requirement for memory operations in general.

> Would this be a worthwhile departure from GIL-based monotonic world?

Years of research agree the answer is yes, no doubt. Otherwise several
fundamental optimizations become impossible - including the usage of
memory caches.
To make a simple example, releasing a lock requires flushing pending
writes (i.e. a cache miss, ~100 cycles nowadays). Implementing
sequential consistency (what you term "monotonic world") means doing
that after _each_ memory write.
If you read Adve et Boehm [1], they'll elaborate more on the topic.

> That is thread-safe code, that uses locks or some other mechanism
> [known to pypy] would see operations timely and in order, while code
> that doesn't synchronize its operations may see somewhat outdated
> shared state, as if threads executed in slightly different global
> order.

Yes. However, the programmer can't and shouldn't synchronize, say, field
lookup, so those dictionaries (and only those) need to be thread-safe
- Jython uses a
ConcurrentHashMap for this purpose, even too that's too thread-safe
(there needs to be no thread-safety guarantee on the values of
fields).

> About profiling, how do I distinguish between attribute vs globals vs
> builtins lookups in the profile?

Others can surely answer better, since I don't know the code; I guess
I'd add a "type" field to the dictionary to store such information, or
maybe to the call sites, depending on what's easier to annotate; but I
also guess this naive solution could require lots of work.

> d.
>
>
> On 27 December 2010 12:50, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>> On Mon, Dec 27, 2010 at 19:54, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> what do you think of liburcu? lttng.org/urcu
>>
>> I've studied RCU in the Linux kernel (from which URCU derives) and
>> thought for a long time of using it for this problem. In short, once
>> you have GC, RCU (as in the Linux kernel) becomes (almost) trivial,
>> because RCU is almost entirely about how to delay freeing objects.
>> The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt
>> in the Linux sources.
>>
>> So, you need to just read about persistent data structures, and
>> remember that persistence is not the point - data structures which are
>> just almost persistent can also be OK, if you get them right. If you
>> replace a value in a dictionary, for instance, you don't necessarily
>> need to keep the old version around, as long as you the value fits in
>> a memory word so that it can be atomically replaced.
>> http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/
>>
>> Speaking of libraries, you might want to look at this for the atomic
>> primitives you'll need:
>> www.hpl.hp.com/research/linux/atomic_ops/
>> I'm more used to the primitives of the Linux kernel, I remember I
>> didn't fully like that library, but IIRC it was a matter of taste -
>> the library is from Hans Boehm, which makes it trustworthy.
>>
>>> p.s. if/when I get around to profiling pypy, which is something I want
>>> to do, I'll be sure to share the results here.
>>
>> Great, thanks!
>>
>>> d.
>>>
>>> On 27 December 2010 10:57, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>>>> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the
>>>>> constant factor for locks, etc, might make it worthwhile
>>>>
>>>> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)?
>>>>
>>>>> (paolo) non blocking hash maps: memory barriers can be quite costly too
>>>>
>>>> Well, you just _cannot_ avoid them, whatever is your fancy data structure.
>>>> You can only move most of the cost (or all of it) to write operations
>>>> - which is helpful if reads are most common, a case which you discuss
>>>> in your mail.
>>>>
>>>> == Why memory barriers are required (in more detail) ==
>>>> If you use persistent data structures, also called copy-on-write or
>>>> purely functional (and that's the most obvious way to use a tree in a
>>>> parallel setting), the whole synchronization problem is reduced to
>>>> exchanging a pointer (to the root of the tree) between a writer and a
>>>> reader thread.
>>>> Let's consider this in the Java memory model (the state of the art).
>>>> In this model the writer thread has to use an expensive StoreLoad
>>>> barrier, i.e. mfence on x86 [1], costing more or less as much as a
>>>> cache miss, while the reader needs just a cheap Load* barrier (for
>>>> free on x86, cheap on other processor).
>>>> If you don't use a volatile field, there is very little guarantee
>>>> about what your program means, and it is almost always buggy; the
>>>> equivalent in the new C++ memory model (which is otherwise very
>>>> similar) gives completely undefined semantics (and even crashes are
>>>> possible in practice, for instance if the reader thread sees a
>>>> not-fully-initialized object because of the race and invokes a virtual
>>>> method on it).
>>>>
>>>> [1] http://g.oswego.edu/dl/jmm/cookbook.html
>>>>
>>>> == Other stuff ==
>>>>> different options will have to be implemented and tested when we get there,
>>>>> speaking on which, is there a test dict load?
>>>>> does anyone have some profiling data, what portion of runtime is spent
>>>>> reading and writing attributes and globals anyways?
>>>>
>>>> I'd be interested as well in such data.
>>>>
>>>>> Anyhow, back to parallel interpretation, Is there an issue or page
>>>>> that tracks what's needed for parallel interpretation?
>>>>> so far ?mentioned: gc, dict, c api
>>>>
>>>>> Btw, I think that jit is more important at the moment, but time comes
>>>>> when jit juice has been mostly squeezed out ;-)
>>>>
>>>> --
>>>> Paolo Giarrusso - Ph.D. Student
>>>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>>>
>>>
>>
>>
>>
>> --
>> Paolo Giarrusso - Ph.D. Student
>> http://www.informatik.uni-marburg.de/~pgiarrusso/
>>
>



-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/