From mailsik at gmail.com  Mon Feb  2 15:46:17 2015
From: mailsik at gmail.com (Sik)
Date: Mon, 2 Feb 2015 15:46:17 +0100
Subject: [pypy-dev] how to register to sprint in Leysin, Switzerland
Message-ID: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>

Hi,

A friend of mine and I would like to join during the sprint in Leysin. We
are researchers in computer vision (with a background in computer science)
who happen live to be close enough to join the sprint.

How shall we sign in?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150202/4ede321b/attachment.html>

From me at manueljacob.de  Mon Feb  2 16:02:18 2015
From: me at manueljacob.de (Manuel Jacob)
Date: Mon, 02 Feb 2015 16:02:18 +0100
Subject: [pypy-dev] how to register to sprint in Leysin, Switzerland
In-Reply-To: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
References: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
Message-ID: <feae0b9242244d63e4cd2d6e037a32d0@indus.uberspace.de>

On 2015-02-02 15:46, Sik wrote:
> Hi,
> 
> A friend of mine and I would like to join during the sprint in Leysin. 
> We
> are researchers in computer vision (with a background in computer 
> science)
> who happen live to be close enough to join the sprint.
> 
> How shall we sign in?
> 
> Thanks

Hi,

if you give me the full name of you and your friend, I'll add you to the 
list of people coming [1].

-Manuel

[1] 
https://bitbucket.org/pypy/extradoc/src/extradoc/sprintinfo/leysin-winter-2015/people.txt?at=extradoc

From arigo at tunes.org  Mon Feb  2 16:56:41 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 2 Feb 2015 16:56:41 +0100
Subject: [pypy-dev] how to register to sprint in Leysin, Switzerland
In-Reply-To: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
References: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
Message-ID: <CAMSv6X3EjozgZBBmPRH=AAM=u0T3ogXfqGdvuHRkL5oBny1j6g@mail.gmail.com>

Hi Sik,

On 2 February 2015 at 15:46, Sik <mailsik at gmail.com> wrote:
> A friend of mine and I would like to join during the sprint in Leysin. We
> are researchers in computer vision (with a background in computer science)
> who happen live to be close enough to join the sprint.

Welcome !  The sprint is turning out to be as many newcomers as
old-time participants, but it is fine this way too :-)

Manuel Jacob added you two to the list.  Just checking: will you be
staying at the Chalet overnight, or do you plan to go home every day?


A bient?t,

Armin.

From mailsik at gmail.com  Mon Feb  2 17:50:45 2015
From: mailsik at gmail.com (Sik)
Date: Mon, 2 Feb 2015 17:50:45 +0100
Subject: [pypy-dev] how to register to sprint in Leysin, Switzerland
In-Reply-To: <CAMSv6X3EjozgZBBmPRH=AAM=u0T3ogXfqGdvuHRkL5oBny1j6g@mail.gmail.com>
References: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
 <CAMSv6X3EjozgZBBmPRH=AAM=u0T3ogXfqGdvuHRkL5oBny1j6g@mail.gmail.com>
Message-ID: <CA+7QG_MpBM7TzbnC03YfotwM1+k8+o6vd5aJi9kC5zUmbYDWGg@mail.gmail.com>

I guess that everybody feels the movement of coders all around the glove,
and lots of us try to lose fear and manifest our support.

We live close, but not that much ;) I guess that we'll try to find a place
at the Chalet. (In this manner we can jump early out of bed to go for
skimo).
We were waiting to see if we could come. Now I think is time to book the
room ;) Otherwise the worst plan B is to stay in Lausanne and drive to
Leysin. Maybe somebody else if from Lausanne.

Anyhow I'm looking forward to 20th of Feb.

On Mon, Feb 2, 2015 at 4:56 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Sik,
>
> On 2 February 2015 at 15:46, Sik <mailsik at gmail.com> wrote:
> > A friend of mine and I would like to join during the sprint in Leysin. We
> > are researchers in computer vision (with a background in computer
> science)
> > who happen live to be close enough to join the sprint.
>
> Welcome !  The sprint is turning out to be as many newcomers as
> old-time participants, but it is fine this way too :-)
>
> Manuel Jacob added you two to the list.  Just checking: will you be
> staying at the Chalet overnight, or do you plan to go home every day?
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150202/41b84579/attachment.html>

From rich at pasra.at  Mon Feb  2 17:56:48 2015
From: rich at pasra.at (Richard Plangger)
Date: Mon, 02 Feb 2015 17:56:48 +0100
Subject: [pypy-dev] PyPy improving generated machine code
In-Reply-To: <CAMSv6X2RCXU3-GasNaxw+aauqGSth-Q78KHjquNEbtr1U6MgbQ@mail.gmail.com>
References: <54CCA591.6000806@pasra.at>
 <CAMSv6X2RCXU3-GasNaxw+aauqGSth-Q78KHjquNEbtr1U6MgbQ@mail.gmail.com>
Message-ID: <54CFAC50.9020304@pasra.at>



On 01/31/2015 03:40 PM, Armin Rigo wrote:
ce optimizations utilizing type information."
> 
> This doesn't mean the performance of PyPy is perfectly optimal today.
> There are certainly things to do and try.  One of the major ones (in
> terms of work involved) would be to add a method-JIT-like approach
> with a quick-and-dirty initial JIT, able to give not-too-bad
> performance but without the large warm-up times of our current
> meta-tracing JIT.  More about this or others in a later e-mail, if
> you're interested.
> 
> 
> A bient?t,
> 
> Armin.
> 

Hi,

Sorry to bother again. I did not get any response yet. The problem is
that I need a better picture about a topic I could work on for my thesis
and I really would like to contribute to pypy. In this week I would like
to decide what I'm aiming for (otherwise things might get shifted).

It would be nice to have the information you mentioned earlier in your
email about the method-JIT-like approach and others!

Best,
Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150202/141c054d/attachment.sig>

From matti.picus at gmail.com  Mon Feb  2 18:54:55 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 02 Feb 2015 19:54:55 +0200
Subject: [pypy-dev] PyPy improving generated machine code
In-Reply-To: <54CFAC50.9020304@pasra.at>
References: <54CCA591.6000806@pasra.at>
 <CAMSv6X2RCXU3-GasNaxw+aauqGSth-Q78KHjquNEbtr1U6MgbQ@mail.gmail.com>
 <54CFAC50.9020304@pasra.at>
Message-ID: <54CFB9EF.1060101@gmail.com>


On 02/02/15 18:56, Richard Plangger wrote:
>
> On 01/31/2015 03:40 PM, Armin Rigo wrote:
> ce optimizations utilizing type information."
>> This doesn't mean the performance of PyPy is perfectly optimal today.
>> There are certainly things to do and try.  One of the major ones (in
>> terms of work involved) would be to add a method-JIT-like approach
>> with a quick-and-dirty initial JIT, able to give not-too-bad
>> performance but without the large warm-up times of our current
>> meta-tracing JIT.  More about this or others in a later e-mail, if
>> you're interested.
>>
>>
>> A bient?t,
>>
>> Armin.
>>
> Hi,
>
> Sorry to bother again. I did not get any response yet. The problem is
> that I need a better picture about a topic I could work on for my thesis
> and I really would like to contribute to pypy. In this week I would like
> to decide what I'm aiming for (otherwise things might get shifted).
>
> It would be nice to have the information you mentioned earlier in your
> email about the method-JIT-like approach and others!
>
> Best,
> Richard
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
Just to throw my uneducated opinions into the ring. It would be nice to 
have someone study autovectorization and hardware acceleration in a JIT. 
There are many possible directions: identifying vectorizable actions via 
traces or user-supplied hints,  resuse of llvm or gcc's strategies, 
creating the proper guards, somehow modelling in costs of memory caching 
into the tradeoff of what to parallelize, ...
Matti

From arigo at tunes.org  Mon Feb  2 22:18:35 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 2 Feb 2015 22:18:35 +0100
Subject: [pypy-dev] how to register to sprint in Leysin, Switzerland
In-Reply-To: <CA+7QG_MpBM7TzbnC03YfotwM1+k8+o6vd5aJi9kC5zUmbYDWGg@mail.gmail.com>
References: <CA+7QG_N1vRUhf+9Ez6McDeBHqYcxizoSBPuo-43SDmP=_WtYQQ@mail.gmail.com>
 <CAMSv6X3EjozgZBBmPRH=AAM=u0T3ogXfqGdvuHRkL5oBny1j6g@mail.gmail.com>
 <CA+7QG_MpBM7TzbnC03YfotwM1+k8+o6vd5aJi9kC5zUmbYDWGg@mail.gmail.com>
Message-ID: <CAMSv6X2u_6WwZH3t5_hd+7uDTpCa=3=mEqOKMiLPeWfiRvOqqQ@mail.gmail.com>

Hi Sik,

On 2 February 2015 at 17:50, Sik <mailsik at gmail.com> wrote:
> We live close, but not that much ;) I guess that we'll try to find a place
> at the Chalet. (In this manner we can jump early out of bed to go for
> skimo).
> We were waiting to see if we could come. Now I think is time to book the
> room ;) Otherwise the worst plan B is to stay in Lausanne and drive to
> Leysin. Maybe somebody else if from Lausanne.

No, the closest person (apart from me) is from Zurich.  The best
option for you is to come to the Chalet.  So far we have pre-booked
two rooms of three people each, which are now full, but if more people
show up, there is a large backup room as well.


A bient?t,

Armin.

From matti.picus at gmail.com  Tue Feb  3 12:06:08 2015
From: matti.picus at gmail.com (matti picus)
Date: Tue, 3 Feb 2015 13:06:08 +0200
Subject: [pypy-dev] 2.5.0 downloads available, please test them out
Message-ID: <CAO8bCgjLLgdNkjcAwTBa4-CBCRurxGyWUUJ3ab1_gcBTAO1FkQ@mail.gmail.com>

https://bitbucket.org/pypy/pypy/downloads
Please make sure they work as expected, especially the osx one since I
cannot test it.
If all is OK, I will "release" (update the pypy.org website and publish the
release notice on the blog) in 6 hours or so
Matti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150203/2e1a9df5/attachment.html>

From arigo at tunes.org  Tue Feb  3 12:54:40 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 3 Feb 2015 12:54:40 +0100
Subject: [pypy-dev] PyPy improving generated machine code
In-Reply-To: <54CFAC50.9020304@pasra.at>
References: <54CCA591.6000806@pasra.at>
 <CAMSv6X2RCXU3-GasNaxw+aauqGSth-Q78KHjquNEbtr1U6MgbQ@mail.gmail.com>
 <54CFAC50.9020304@pasra.at>
Message-ID: <CAMSv6X0N5aJ4P95kOS51XkhfCL_aYr9n25hgLBwG=-m7ume4VQ@mail.gmail.com>

Hi Richard,

On 2 February 2015 at 17:56, Richard Plangger <rich at pasra.at> wrote:
> Sorry to bother again. I did not get any response yet. The problem is
> that I need a better picture about a topic I could work on for my thesis
> and I really would like to contribute to pypy. In this week I would like
> to decide what I'm aiming for (otherwise things might get shifted).
>
> It would be nice to have the information you mentioned earlier in your
> email about the method-JIT-like approach and others

Our general topic list is here:
http://pypy.readthedocs.org/en/latest/project-ideas.html

Others than me should chime in too, because I'm not the best person to
recommend what should be worked on in general.  Myself, I am
particularly interested in the STM work, where there is certainly
something to do, although it's a bit hard to plan in advance: it's
mostly of the kind: run stuff with pypy-stm, find conflicts, figure
out which ones are not inherent to the application but are simply
limitations of the interpreter, and then think about ways to avoid
them.

About the method JIT, we have only vague ideas.  I can't give any
estimate of how much work it will be, as it depends a lot on
details of the approach, like how much of the existing
infrastructure can be reused and how much must be redone from
scratch.

To be clear, we don't even have any clear indication that it
would be a good idea.  It seems to require more
interpreter-specific hints (say for PyPy's Python interpreter) to
drive the process, in order to control where it should stop
inlining and start emitting calls to the existing functions.

The prior work gives mixed results: if you consider for example
Jython running on a method-JIT-based Java, it would be
similar (minus possible hints we can add), but Jython is not
faster than CPython.  On the other hand, untyped Cython is
usually faster than CPython, but it benefits from gcc's slow
optimizations.  I would classify it as very much a research
project.


A bient?t,

Armin.

From fijall at gmail.com  Tue Feb  3 14:25:10 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 3 Feb 2015 15:25:10 +0200
Subject: [pypy-dev] PyPy improving generated machine code
In-Reply-To: <CAMSv6X0N5aJ4P95kOS51XkhfCL_aYr9n25hgLBwG=-m7ume4VQ@mail.gmail.com>
References: <54CCA591.6000806@pasra.at>
 <CAMSv6X2RCXU3-GasNaxw+aauqGSth-Q78KHjquNEbtr1U6MgbQ@mail.gmail.com>
 <54CFAC50.9020304@pasra.at>
 <CAMSv6X0N5aJ4P95kOS51XkhfCL_aYr9n25hgLBwG=-m7ume4VQ@mail.gmail.com>
Message-ID: <CAK5idxSVDVUgGoq4-Y8vgqudy5V8jnqJq=h-pGbpTRwkP_FVHA@mail.gmail.com>

Hi

One topic that captured my interest is a light-weight type-specilizing
JIT that does not emit any assembler, but instead rewrites the
bytecode into a more type-specific form. This has been done before in
luajit and gives good results there. I wonder if the same can be
applied in PyPy

PS. Feel free to pop in to IRC #pypy on freenode, I can explain in
more detail what I mean

Cheers,
fijal

On Tue, Feb 3, 2015 at 1:54 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Richard,
>
> On 2 February 2015 at 17:56, Richard Plangger <rich at pasra.at> wrote:
>> Sorry to bother again. I did not get any response yet. The problem is
>> that I need a better picture about a topic I could work on for my thesis
>> and I really would like to contribute to pypy. In this week I would like
>> to decide what I'm aiming for (otherwise things might get shifted).
>>
>> It would be nice to have the information you mentioned earlier in your
>> email about the method-JIT-like approach and others
>
> Our general topic list is here:
> http://pypy.readthedocs.org/en/latest/project-ideas.html
>
> Others than me should chime in too, because I'm not the best person to
> recommend what should be worked on in general.  Myself, I am
> particularly interested in the STM work, where there is certainly
> something to do, although it's a bit hard to plan in advance: it's
> mostly of the kind: run stuff with pypy-stm, find conflicts, figure
> out which ones are not inherent to the application but are simply
> limitations of the interpreter, and then think about ways to avoid
> them.
>
> About the method JIT, we have only vague ideas.  I can't give any
> estimate of how much work it will be, as it depends a lot on
> details of the approach, like how much of the existing
> infrastructure can be reused and how much must be redone from
> scratch.
>
> To be clear, we don't even have any clear indication that it
> would be a good idea.  It seems to require more
> interpreter-specific hints (say for PyPy's Python interpreter) to
> drive the process, in order to control where it should stop
> inlining and start emitting calls to the existing functions.
>
> The prior work gives mixed results: if you consider for example
> Jython running on a method-JIT-based Java, it would be
> similar (minus possible hints we can add), but Jython is not
> faster than CPython.  On the other hand, untyped Cython is
> usually faster than CPython, but it benefits from gcc's slow
> optimizations.  I would classify it as very much a research
> project.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From yeomanyaacov at gmail.com  Wed Feb  4 19:30:27 2015
From: yeomanyaacov at gmail.com (Yaacov Finkelman)
Date: Wed, 4 Feb 2015 13:30:27 -0500
Subject: [pypy-dev] Windows: pypyw.exe?
Message-ID: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>

Hi,

I was wondering what the plan is for fixing:
https://bitbucket.org/pypy/pypy/issue/1651/

It is just a hassle with each upgrade of pypy.

I'd be happy to fix, but I don't know where to start.

Best,

Jacob

From lac at openend.se  Wed Feb  4 19:42:37 2015
From: lac at openend.se (Laura Creighton)
Date: Wed, 04 Feb 2015 19:42:37 +0100
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: Message from Yaacov Finkelman <yeomanyaacov@gmail.com> of "Wed,
 04 Feb 2015 13:30:27 -0500."
 <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
Message-ID: <201502041842.t14IgbBl022355@fido.openend.se>

In a message of Wed, 04 Feb 2015 13:30:27 -0500, Yaacov Finkelman writes:
>Hi,
>
>I was wondering what the plan is for fixing:
>https://bitbucket.org/pypy/pypy/issue/1651/
>
>It is just a hassle with each upgrade of pypy.
>
>I'd be happy to fix, but I don't know where to start.
>
>Best,
>
>Jacob

Please excuse me for sending you a blank message.
I was trying to test my 'send mail about bitbucket feature'
and, well, i screwed it up and instead of sending me a
private message about that you made a bitbucket issue
mention ....

it sent me that and you an empty string, null mail.

very, very sorry, this will not happen again.  I was careless.

Laura Creighton

From matti.picus at gmail.com  Wed Feb  4 20:04:13 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 04 Feb 2015 21:04:13 +0200
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
Message-ID: <54D26D2D.6010603@gmail.com>

On 04/02/15 20:30, Yaacov Finkelman wrote:
> Hi,
>
> I was wondering what the plan is for fixing:
> https://bitbucket.org/pypy/pypy/issue/1651/
>
> It is just a hassle with each upgrade of pypy.
>
> I'd be happy to fix, but I don't know where to start.
>
> Best,
>
> Jacob
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
We would love to get more windows people on board.
The "where to start" could mean a number of things
1. Where to start with the pypy dev process - try reading
      http://pypy.readthedocs.org/en/latest/getting-started-dev.html
2. Where to start with pypy and windows - try reading
    http://pypy.readthedocs.org/en/latest/windows.html
3. Where to start fixing this issue once you understand those two - try 
looking at how pythonw.exe is different from python.exe in cpython, 
create a branch and try to think of how to write tests for that 
difference, then implement code that passes the tests, then ask us to 
merge the branch.

When you say "it is just a hassle" that leads me to believe you already 
know how to fix it (Hint: |cl /subsystem:windows|)

Matti

From Martin.Gfeller at swisscom.com  Thu Feb  5 08:14:36 2015
From: Martin.Gfeller at swisscom.com (Martin.Gfeller at swisscom.com)
Date: Thu, 5 Feb 2015 07:14:36 +0000
Subject: [pypy-dev] Post on Swisscom ICT Blog "Python Performance: A
	Recommendation"
Message-ID: <06594777939d40edaeaf6e763f0667d9@SG001745.corproot.net>

Dear all, I wrote a plug for PyPy on the Swisscom ICT Blog: http://ict.swisscom.ch/2015/02/pyperformance/?

Kind regards
Martin Gfeller
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150205/9b50159f/attachment.html>

From arigo at tunes.org  Thu Feb  5 16:24:46 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 5 Feb 2015 16:24:46 +0100
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <54D26D2D.6010603@gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
 <54D26D2D.6010603@gmail.com>
Message-ID: <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>

Hi,

On 4 February 2015 at 20:04, Matti Picus <matti.picus at gmail.com> wrote:
> When you say "it is just a hassle" that leads me to believe you already know
> how to fix it (Hint: |cl /subsystem:windows|)

It's probably as easy as tweaking the generated Makefile, to call once
"cl" and once "cl /subsystem:windows" to make "pypy.exe" and
"pypyw.exe".


A bient?t,

Armin.

From arigo at tunes.org  Thu Feb  5 17:02:27 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 5 Feb 2015 17:02:27 +0100
Subject: [pypy-dev] A question about RPython's JIT in scheduler
In-Reply-To: <01D41760-62D1-43DD-97B9-78197D8A84C6@gmail.com>
References: <01D41760-62D1-43DD-97B9-78197D8A84C6@gmail.com>
Message-ID: <CAMSv6X3qDmDjw=24JN5q_4h_AvP2hHkppFC+FzsUPLSEMd6heA@mail.gmail.com>

Hi,

For future reference: we discussed this question on IRC yesterday, Feb
4th (see logs).  Let me repeat here the main answer: the JIT works
well because you're using a scheme where some counter is decremented
(and the soft-thread interrupted when it reaches zero) only once in
each app-level loop.  The soft-thread switch is done by returning to
some scheduler, which will resume a different soft-thread by calling
it.  It means the JIT can still compile each of the loops as usual,
with the generated machine code containing the
decrease-and-check-for-zero operation which, when true, exits the
assembler.


A bient?t,

Armin.

From yeomanyaacov at gmail.com  Fri Feb  6 03:34:51 2015
From: yeomanyaacov at gmail.com (Yaacov Finkelman)
Date: Thu, 5 Feb 2015 21:34:51 -0500
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
 <54D26D2D.6010603@gmail.com>
 <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>
Message-ID: <CAGBU8t72EPadJcUbOhv7+pRY5=R+3XH5zdwC8YD8RRPZ=JVhJQ@mail.gmail.com>

HI,

I have "fix it" by copying Pypy.exe then using "EDITBIN.EXE
/SUBSYSTEM:WINDOWS C:\pypy\pypyw.exe" from Microsoft Visual Studio to
change the relevant byte. [http://stackoverflow.com/questions/574911]
So yes, don't give me too much credit for knowing what I'm doing. In
honor of learning on the job, let me barg ahead.

tweaking a Makefile sounds like a good solution. What part of the code
generates the Makefile?

The difference between pythonw.exe and python.exe is that one is a
"windows" application and the other is a "console" application.
Testing this defence...
We could run pypyw.exe and check whether a console appears. Slow and
requires running on windows.
We could check the binary file to see if the relevant byte is set to
the correct setting. Requires thorough understanding of the structure
of exe that I don't at this time have. As such the test would require
good testing.
Uew a third party module to check the exe for the correct SUBSYSTEM.
All the packages I can find are incredibly out of date, but maybe I
missed one, or maybe they still work.

Best,

Jacob

On Thu, Feb 5, 2015 at 10:24 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On 4 February 2015 at 20:04, Matti Picus <matti.picus at gmail.com> wrote:
>> When you say "it is just a hassle" that leads me to believe you already know
>> how to fix it (Hint: |cl /subsystem:windows|)
>
> It's probably as easy as tweaking the generated Makefile, to call once
> "cl" and once "cl /subsystem:windows" to make "pypy.exe" and
> "pypyw.exe".
>
>
> A bient?t,
>
> Armin.

From matti.picus at gmail.com  Fri Feb  6 08:48:42 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 06 Feb 2015 09:48:42 +0200
Subject: [pypy-dev] Faulty binary 2.5.0 package(s) on bitbucket download page
Message-ID: <54D471DA.3020107@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150206/3887eb56/attachment.html>

From matti.picus at gmail.com  Fri Feb  6 09:48:32 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 06 Feb 2015 10:48:32 +0200
Subject: [pypy-dev] Faulty binary 2.5.0 package(s) on bitbucket download
	page
In-Reply-To: <54D471DA.3020107@gmail.com>
References: <54D471DA.3020107@gmail.com>
Message-ID: <54D47FE0.7000606@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150206/bf5aa984/attachment.html>

From fijall at gmail.com  Fri Feb  6 09:54:55 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 6 Feb 2015 10:54:55 +0200
Subject: [pypy-dev] [pypy-commit] pypy default: Rename the --output
 option to --really-force-output, and document it
In-Reply-To: <20150205155202.D3E071C0459@cobra.cs.uni-duesseldorf.de>
References: <20150205155202.D3E071C0459@cobra.cs.uni-duesseldorf.de>
Message-ID: <CAK5idxQczTPtbja32GgNmSFqZRfvNeCskUcnedwmL4xt60HWWQ@mail.gmail.com>

This change breaks places that recommend you to use --output (e.g.
check that you're not translating with the same executable)

On Thu, Feb 5, 2015 at 5:52 PM, arigo <noreply at buildbot.pypy.org> wrote:
> Author: Armin Rigo <arigo at tunes.org>
> Branch:
> Changeset: r75726:babfa1dd27a5
> Date: 2015-02-05 16:51 +0100
> http://bitbucket.org/pypy/pypy/changeset/babfa1dd27a5/
>
> Log:    Rename the --output option to --really-force-output, and document it
>         as "don't use with PyPy".
>
> diff --git a/rpython/config/translationoption.py b/rpython/config/translationoption.py
> --- a/rpython/config/translationoption.py
> +++ b/rpython/config/translationoption.py
> @@ -151,7 +151,9 @@
>                 default=False, cmdline="--dont-write-c-files"),
>      ArbitraryOption("instrumentctl", "internal",
>                 default=None),
> -    StrOption("output", "Output file name", cmdline="--output"),
> +    StrOption("output", "Output file name (don't change for PyPy!"
> +              " doesn't work with virtualenv+shared: issue 1971)",
> +              cmdline="--really-force-output"),
>      StrOption("secondaryentrypoints",
>              "Comma separated list of keys choosing secondary entrypoints",
>              cmdline="--entrypoints", default="main"),
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> https://mail.python.org/mailman/listinfo/pypy-commit

From hrc706 at gmail.com  Fri Feb  6 04:01:38 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Fri, 6 Feb 2015 12:01:38 +0900
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
Message-ID: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>

Hi, everyone, 
I tried to wrote down the experience I have done recently about my project of Pyrlang, but I?m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog.
Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance.

Ruochen Huang
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experience in Pyrlang with RPython.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 528919 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150206/8a47889e/attachment-0001.docx>
-------------- next part --------------


From amauryfa at gmail.com  Fri Feb  6 11:15:48 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 6 Feb 2015 11:15:48 +0100
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <CAGBU8t72EPadJcUbOhv7+pRY5=R+3XH5zdwC8YD8RRPZ=JVhJQ@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
 <54D26D2D.6010603@gmail.com>
 <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>
 <CAGBU8t72EPadJcUbOhv7+pRY5=R+3XH5zdwC8YD8RRPZ=JVhJQ@mail.gmail.com>
Message-ID: <CAGmFidZ0T=OwQEzk+=1knS1td7yTQrETFM15Dm_9pH5T9tFTug@mail.gmail.com>

2015-02-06 3:34 GMT+01:00 Yaacov Finkelman <yeomanyaacov at gmail.com>:

> The difference between pythonw.exe and python.exe is that one is a
> "windows" application and the other is a "console" application.
>

This has other implications.
For example, sys.stdout points to an invalid file descriptor,
and I remember that old versions of pythonw.exe used to freeze after
printing 8192 characters.


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150206/93ff483a/attachment.html>

From arigo at tunes.org  Fri Feb  6 11:27:07 2015
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 6 Feb 2015 11:27:07 +0100
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <CAGmFidZ0T=OwQEzk+=1knS1td7yTQrETFM15Dm_9pH5T9tFTug@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
 <54D26D2D.6010603@gmail.com>
 <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>
 <CAGBU8t72EPadJcUbOhv7+pRY5=R+3XH5zdwC8YD8RRPZ=JVhJQ@mail.gmail.com>
 <CAGmFidZ0T=OwQEzk+=1knS1td7yTQrETFM15Dm_9pH5T9tFTug@mail.gmail.com>
Message-ID: <CAMSv6X1GkquJtEYk620taT9RWSnZcxjDqeeQX1d=rTgQ1Ldbdw@mail.gmail.com>

Hi,

On 6 February 2015 at 11:15, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> This has other implications.
> For example, sys.stdout points to an invalid file descriptor,
> and I remember that old versions of pythonw.exe used to freeze after
> printing 8192 characters.

I'm not finding anything special done with stdout if we're
pythonw.exe.  Can you be more specific?

Otherwise I suppose the way forward is to produce pypyw.exe (by
editing rpython/translator/platform/windows.py, gen_makefile(), rule
for $(DEFAULT_TARGET), to run $(CC_LINK) twice instead of once), and
then to wait until people report issues with it.


A bient?t,

Armin.

From arigo at tunes.org  Fri Feb  6 11:50:09 2015
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 6 Feb 2015 11:50:09 +0100
Subject: [pypy-dev] [pypy-commit] pypy default: Rename the --output
 option to --really-force-output, and document it
In-Reply-To: <CAK5idxQczTPtbja32GgNmSFqZRfvNeCskUcnedwmL4xt60HWWQ@mail.gmail.com>
References: <20150205155202.D3E071C0459@cobra.cs.uni-duesseldorf.de>
 <CAK5idxQczTPtbja32GgNmSFqZRfvNeCskUcnedwmL4xt60HWWQ@mail.gmail.com>
Message-ID: <CAMSv6X2XaLh=61ULXR1HixMQ-5F9_-No9z1e3aaOFqhGop+3FA@mail.gmail.com>

Hi Fijal,

On 6 February 2015 at 09:54, Maciej Fijalkowski <fijall at gmail.com> wrote:
> This change breaks places that recommend you to use --output (e.g.
> check that you're not translating with the same executable)

Grepping for "--output", I can find only translate.py that says "use
--output=..." so I'll guess you mean that place.  Are you fine if we
simply change this error message to "please move the executable (and
possibly its associated libpypy-c) somewhere else before you execute
it"?


Armin

From amauryfa at gmail.com  Fri Feb  6 12:14:56 2015
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 6 Feb 2015 12:14:56 +0100
Subject: [pypy-dev] Windows: pypyw.exe?
In-Reply-To: <CAMSv6X1GkquJtEYk620taT9RWSnZcxjDqeeQX1d=rTgQ1Ldbdw@mail.gmail.com>
References: <CAGBU8t7oz=0EQPP_nxMhx03QfMZrD3xx=_PFkPsXv5g3bz-Zew@mail.gmail.com>
 <54D26D2D.6010603@gmail.com>
 <CAMSv6X0fjYxqs-6LS1D=tfypaziOvNSW4j-dEXXCfffjCMrfgQ@mail.gmail.com>
 <CAGBU8t72EPadJcUbOhv7+pRY5=R+3XH5zdwC8YD8RRPZ=JVhJQ@mail.gmail.com>
 <CAGmFidZ0T=OwQEzk+=1knS1td7yTQrETFM15Dm_9pH5T9tFTug@mail.gmail.com>
 <CAMSv6X1GkquJtEYk620taT9RWSnZcxjDqeeQX1d=rTgQ1Ldbdw@mail.gmail.com>
Message-ID: <CAGmFidZ39-+bjfKgArDyRekXqWqgD+NTi0LK0gU1jxwF5HY8qg@mail.gmail.com>

2015-02-06 11:27 GMT+01:00 Armin Rigo <arigo at tunes.org>:

> Hi,
>
> On 6 February 2015 at 11:15, Amaury Forgeot d'Arc <amauryfa at gmail.com>
> wrote:
> > This has other implications.
> > For example, sys.stdout points to an invalid file descriptor,
> > and I remember that old versions of pythonw.exe used to freeze after
> > printing 8192 characters.
>
> I'm not finding anything special done with stdout if we're
> pythonw.exe.  Can you be more specific?
>

CPython3 has this code to set sys.stdout to None in this case:
https://hg.python.org/cpython/file/v3.3.4/Python/pythonrun.c#l1083

But that's probably only for python3, python2 chose to not change anything:
http://bugs.python.org/issue706263


> Otherwise I suppose the way forward is to produce pypyw.exe (by
> editing rpython/translator/platform/windows.py, gen_makefile(), rule
> for $(DEFAULT_TARGET), to run $(CC_LINK) twice instead of once), and
> then to wait until people report issues with it.
>
>
> A bient?t,
>
> Armin.
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150206/71b9d5a0/attachment.html>

From arigo at tunes.org  Fri Feb  6 16:57:06 2015
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 6 Feb 2015 16:57:06 +0100
Subject: [pypy-dev] Fwd: [pypy-commit] pypy default: Rename the --output
 option to --really-force-output, and document it
In-Reply-To: <CAMSv6X1y4F6_3u12kQ3GzbNL0eQqYARKKRsdiBvGLC7rgDHMbw@mail.gmail.com>
References: <20150205155202.D3E071C0459@cobra.cs.uni-duesseldorf.de>
 <CAK5idxQczTPtbja32GgNmSFqZRfvNeCskUcnedwmL4xt60HWWQ@mail.gmail.com>
 <CAMSv6X2XaLh=61ULXR1HixMQ-5F9_-No9z1e3aaOFqhGop+3FA@mail.gmail.com>
 <E3AD40E9-B943-4A6D-BC86-930A2C345F45@gmx.de>
 <CAMSv6X1y4F6_3u12kQ3GzbNL0eQqYARKKRsdiBvGLC7rgDHMbw@mail.gmail.com>
Message-ID: <CAMSv6X2--=VqzdPsbgMLA29FsJptKqecu3J3jOgLm9ryJA=kpg@mail.gmail.com>

Hi Tobias,

(answering an off-list-by-mistake mail)

On 6 February 2015 at 13:50, Tobias Pape <Das.Linux at gmx.de> wrote:
> So this is changed in rpython for a pypy issue?

Yes, someone else made the same point on IRC.  I have now reverted
this change, and added a check to "targetpypystandalone.py" that you
don't try to translate pypy with both "--shared" and "--output", which
should be enough to avoid the original issue.


A bient?t,

Armin.

From tinchester at gmail.com  Sat Feb  7 00:12:24 2015
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Sat, 07 Feb 2015 00:12:24 +0100
Subject: [pypy-dev] Playing with PyPy and Django
Message-ID: <54D54A58.9070201@gmail.com>

Hello, PyPy folks!

While trying to speed up one of my Django sites, I noticed a new version 
of PyPy
had just been released. So I grabbed a fresh download of PyPy 3 (since 
this is
a Python 3 codebase) and tried taking it out for a spin.

However, as far as I can see, whatever I try PyPy is consistently slower 
than
CPython for this.

Since this is a proprietary site, I've basically ripped out all the code 
except
my settings.py and my requirements; and am benchmarking the Django admin 
index.
The results are about the same.

I've set up a small repo that can be used to reproduce the environment:
https://github.com/Tinche/PyPy-Django-Playground. There's additional info in
the README there.

These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is the
system Python, 3.4. PyPy has been downloaded from the official site and
unzipped.

So what I basically do is set up an admin session, and use the Django 
main admin
page. 200 warmup requests, then 100 benchmarked requests, look at the mean
request time.

Some results:

Django's runserver, DEBUG mode:

PyPy3            485.389 [ms]
CPython 3.4      105.777 [ms]

Django's runserver, no debug:

PyPy3             44.661 [ms]
CPython 3.4       18.697 [ms]

Gunicorn, 1 worker, no debug:

PyPy3             28.615 [ms]
CPython 3.4       13.532 [ms]

I don't exactly claim to be an expert on benchmarking, but assuming my site
is similar to the Django admin, CPython's gonna be giving me better 
performance.
Also the debug runserver performance is kinda worrying. Nobody's going to be
running this in production, but it makes development a little slower and 
more
annoying than it should be.

Is there anything to make PyPy more competitive in these kinds of scenarios?

Kind regards,
Tin

From omer.drow at gmail.com  Sat Feb  7 07:56:32 2015
From: omer.drow at gmail.com (Omer Katz)
Date: Sat, 7 Feb 2015 08:56:32 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D54A58.9070201@gmail.com>
References: <54D54A58.9070201@gmail.com>
Message-ID: <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>

You need to warm up the JIT first.
Run the benchmark a 10,000 times on PyPy before measuring and you'll see
the real performance improvement.
Nevertheless, it does sound like you're hitting a performance bug(s)
somewhere. It's worth investigating.

2015-02-07 1:12 GMT+02:00 Tin Tvrtkovi? <tinchester at gmail.com>:

> Hello, PyPy folks!
>
> While trying to speed up one of my Django sites, I noticed a new version
> of PyPy
> had just been released. So I grabbed a fresh download of PyPy 3 (since
> this is
> a Python 3 codebase) and tried taking it out for a spin.
>
> However, as far as I can see, whatever I try PyPy is consistently slower
> than
> CPython for this.
>
> Since this is a proprietary site, I've basically ripped out all the code
> except
> my settings.py and my requirements; and am benchmarking the Django admin
> index.
> The results are about the same.
>
> I've set up a small repo that can be used to reproduce the environment:
> https://github.com/Tinche/PyPy-Django-Playground. There's additional info
> in
> the README there.
>
> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is
> the
> system Python, 3.4. PyPy has been downloaded from the official site and
> unzipped.
>
> So what I basically do is set up an admin session, and use the Django main
> admin
> page. 200 warmup requests, then 100 benchmarked requests, look at the mean
> request time.
>
> Some results:
>
> Django's runserver, DEBUG mode:
>
> PyPy3            485.389 [ms]
> CPython 3.4      105.777 [ms]
>
> Django's runserver, no debug:
>
> PyPy3             44.661 [ms]
> CPython 3.4       18.697 [ms]
>
> Gunicorn, 1 worker, no debug:
>
> PyPy3             28.615 [ms]
> CPython 3.4       13.532 [ms]
>
> I don't exactly claim to be an expert on benchmarking, but assuming my site
> is similar to the Django admin, CPython's gonna be giving me better
> performance.
> Also the debug runserver performance is kinda worrying. Nobody's going to
> be
> running this in production, but it makes development a little slower and
> more
> annoying than it should be.
>
> Is there anything to make PyPy more competitive in these kinds of
> scenarios?
>
> Kind regards,
> Tin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150207/425b31ae/attachment.html>

From fijall at gmail.com  Sat Feb  7 08:18:43 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 7 Feb 2015 09:18:43 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>
Message-ID: <CAK5idxQrWenbe0c3GchgLeiF94mK8p4TMSiyz9_5aBVi1o8cBQ@mail.gmail.com>

PyPy 3 is definitely slower than PyPy 2 btw, just so you know

On Sat, Feb 7, 2015 at 8:56 AM, Omer Katz <omer.drow at gmail.com> wrote:
> You need to warm up the JIT first.
> Run the benchmark a 10,000 times on PyPy before measuring and you'll see the
> real performance improvement.
> Nevertheless, it does sound like you're hitting a performance bug(s)
> somewhere. It's worth investigating.
>
> 2015-02-07 1:12 GMT+02:00 Tin Tvrtkovi? <tinchester at gmail.com>:
>>
>> Hello, PyPy folks!
>>
>> While trying to speed up one of my Django sites, I noticed a new version
>> of PyPy
>> had just been released. So I grabbed a fresh download of PyPy 3 (since
>> this is
>> a Python 3 codebase) and tried taking it out for a spin.
>>
>> However, as far as I can see, whatever I try PyPy is consistently slower
>> than
>> CPython for this.
>>
>> Since this is a proprietary site, I've basically ripped out all the code
>> except
>> my settings.py and my requirements; and am benchmarking the Django admin
>> index.
>> The results are about the same.
>>
>> I've set up a small repo that can be used to reproduce the environment:
>> https://github.com/Tinche/PyPy-Django-Playground. There's additional info
>> in
>> the README there.
>>
>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is
>> the
>> system Python, 3.4. PyPy has been downloaded from the official site and
>> unzipped.
>>
>> So what I basically do is set up an admin session, and use the Django main
>> admin
>> page. 200 warmup requests, then 100 benchmarked requests, look at the mean
>> request time.
>>
>> Some results:
>>
>> Django's runserver, DEBUG mode:
>>
>> PyPy3            485.389 [ms]
>> CPython 3.4      105.777 [ms]
>>
>> Django's runserver, no debug:
>>
>> PyPy3             44.661 [ms]
>> CPython 3.4       18.697 [ms]
>>
>> Gunicorn, 1 worker, no debug:
>>
>> PyPy3             28.615 [ms]
>> CPython 3.4       13.532 [ms]
>>
>> I don't exactly claim to be an expert on benchmarking, but assuming my
>> site
>> is similar to the Django admin, CPython's gonna be giving me better
>> performance.
>> Also the debug runserver performance is kinda worrying. Nobody's going to
>> be
>> running this in production, but it makes development a little slower and
>> more
>> annoying than it should be.
>>
>> Is there anything to make PyPy more competitive in these kinds of
>> scenarios?
>>
>> Kind regards,
>> Tin
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From tinchester at gmail.com  Sat Feb  7 16:25:53 2015
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Sat, 07 Feb 2015 16:25:53 +0100
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>
Message-ID: <54D62E81.2020401@gmail.com>

Alright, trying 10,000 warmup requests, and then 100 benchmark requests, 
"production" scenario (gunicorn, one worker, no debug):

PyPy3          26.956 [ms]
CPython 3.4    10.822 [ms]

So, not much improvement there. Also consider the slight inconvenience 
of mandatory manual warmup every deploy:

* would I need to warm up just the site, every application, or every 
template?
* I'd need a prepared session to warm up anything except the login page
* I'll be running more worker processes, so even more warmup requests, 
hopefully they'd get distributed evenly amongst the workers


On 07.02.2015 07:56, Omer Katz wrote:
> You need to warm up the JIT first.
> Run the benchmark a 10,000 times on PyPy before measuring and you'll 
> see the real performance improvement.
> Nevertheless, it does sound like you're hitting a performance bug(s) 
> somewhere. It's worth investigating.
>
> 2015-02-07 1:12 GMT+02:00 Tin Tvrtkovi? <tinchester at gmail.com 
> <mailto:tinchester at gmail.com>>:
>
>     Hello, PyPy folks!
>
>     While trying to speed up one of my Django sites, I noticed a new
>     version of PyPy
>     had just been released. So I grabbed a fresh download of PyPy 3
>     (since this is
>     a Python 3 codebase) and tried taking it out for a spin.
>
>     However, as far as I can see, whatever I try PyPy is consistently
>     slower than
>     CPython for this.
>
>     Since this is a proprietary site, I've basically ripped out all
>     the code except
>     my settings.py and my requirements; and am benchmarking the Django
>     admin index.
>     The results are about the same.
>
>     I've set up a small repo that can be used to reproduce the
>     environment:
>     https://github.com/Tinche/PyPy-Django-Playground. There's
>     additional info in
>     the README there.
>
>     These tests have been carried out on Ubuntu Trusty, 64-bit.
>     CPython 3 is the
>     system Python, 3.4. PyPy has been downloaded from the official
>     site and
>     unzipped.
>
>     So what I basically do is set up an admin session, and use the
>     Django main admin
>     page. 200 warmup requests, then 100 benchmarked requests, look at
>     the mean
>     request time.
>
>     Some results:
>
>     Django's runserver, DEBUG mode:
>
>     PyPy3            485.389 [ms]
>     CPython 3.4      105.777 [ms]
>
>     Django's runserver, no debug:
>
>     PyPy3             44.661 [ms]
>     CPython 3.4       18.697 [ms]
>
>     Gunicorn, 1 worker, no debug:
>
>     PyPy3             28.615 [ms]
>     CPython 3.4       13.532 [ms]
>
>     I don't exactly claim to be an expert on benchmarking, but
>     assuming my site
>     is similar to the Django admin, CPython's gonna be giving me
>     better performance.
>     Also the debug runserver performance is kinda worrying. Nobody's
>     going to be
>     running this in production, but it makes development a little
>     slower and more
>     annoying than it should be.
>
>     Is there anything to make PyPy more competitive in these kinds of
>     scenarios?
>
>     Kind regards,
>     Tin
>     _______________________________________________
>     pypy-dev mailing list
>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150207/5a6a9dd2/attachment.html>

From omer.drow at gmail.com  Sat Feb  7 17:45:09 2015
From: omer.drow at gmail.com (Omer Katz)
Date: Sat, 7 Feb 2015 18:45:09 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D62E81.2020401@gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAOZCFcCb8_i5u-869QEFtOyKUWL1uQvxu3vjwJc5Yr6=v+1h9Q@mail.gmail.com>
 <54D62E81.2020401@gmail.com>
Message-ID: <CAOZCFcA+L6w4RMaRZaOX9tO56OfiZ+VEUyzpLH2nV4ktjfZcEA@mail.gmail.com>

I use locust to warm up the the entire site after deployments.
As a bonus you'll be able to stress test your code.

I really recommend using uwsgi for production python sites. It takes some
time to learn it but it makes the site so much more responsive it's worth
it.

As mentioned PyPy3 is still slower then CPython.
I haven't seen such a performance difference with PyPy2.

2015-02-07 17:25 GMT+02:00 Tin Tvrtkovi? <tinchester at gmail.com>:

>  Alright, trying 10,000 warmup requests, and then 100 benchmark requests,
> "production" scenario (gunicorn, one worker, no debug):
>
> PyPy3          26.956 [ms]
> CPython 3.4    10.822 [ms]
>
> So, not much improvement there. Also consider the slight inconvenience of
> mandatory manual warmup every deploy:
>
> * would I need to warm up just the site, every application, or every
> template?
> * I'd need a prepared session to warm up anything except the login page
> * I'll be running more worker processes, so even more warmup requests,
> hopefully they'd get distributed evenly amongst the workers
>
>
>
> On 07.02.2015 07:56, Omer Katz wrote:
>
>  You need to warm up the JIT first.
>  Run the benchmark a 10,000 times on PyPy before measuring and you'll see
> the real performance improvement.
>  Nevertheless, it does sound like you're hitting a performance bug(s)
> somewhere. It's worth investigating.
>
>  2015-02-07 1:12 GMT+02:00 Tin Tvrtkovi? <tinchester at gmail.com>:
>
>> Hello, PyPy folks!
>>
>> While trying to speed up one of my Django sites, I noticed a new version
>> of PyPy
>> had just been released. So I grabbed a fresh download of PyPy 3 (since
>> this is
>> a Python 3 codebase) and tried taking it out for a spin.
>>
>> However, as far as I can see, whatever I try PyPy is consistently slower
>> than
>> CPython for this.
>>
>> Since this is a proprietary site, I've basically ripped out all the code
>> except
>> my settings.py and my requirements; and am benchmarking the Django admin
>> index.
>> The results are about the same.
>>
>> I've set up a small repo that can be used to reproduce the environment:
>> https://github.com/Tinche/PyPy-Django-Playground. There's additional
>> info in
>> the README there.
>>
>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is
>> the
>> system Python, 3.4. PyPy has been downloaded from the official site and
>> unzipped.
>>
>> So what I basically do is set up an admin session, and use the Django
>> main admin
>> page. 200 warmup requests, then 100 benchmarked requests, look at the mean
>> request time.
>>
>> Some results:
>>
>> Django's runserver, DEBUG mode:
>>
>> PyPy3            485.389 [ms]
>> CPython 3.4      105.777 [ms]
>>
>> Django's runserver, no debug:
>>
>> PyPy3             44.661 [ms]
>> CPython 3.4       18.697 [ms]
>>
>> Gunicorn, 1 worker, no debug:
>>
>> PyPy3             28.615 [ms]
>> CPython 3.4       13.532 [ms]
>>
>> I don't exactly claim to be an expert on benchmarking, but assuming my
>> site
>> is similar to the Django admin, CPython's gonna be giving me better
>> performance.
>> Also the debug runserver performance is kinda worrying. Nobody's going to
>> be
>> running this in production, but it makes development a little slower and
>> more
>> annoying than it should be.
>>
>> Is there anything to make PyPy more competitive in these kinds of
>> scenarios?
>>
>> Kind regards,
>> Tin
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150207/b182f77a/attachment.html>

From arigo at tunes.org  Sat Feb  7 18:16:15 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 7 Feb 2015 18:16:15 +0100
Subject: [pypy-dev] Another idea for improving warm-up times
Message-ID: <CAMSv6X1BHdW5-0ahBcty7jnfc3HnaQkUxGEurheHVTo+hs7Q5A@mail.gmail.com>

Hi all,

Here's an idea that came up today on irc (thanks the_drow_) while
discussing again saving the generated assembler into files and
reloading it on the next run.  As you know we categorize this idea as
"can never work", but now I think that there is a variant that could
work, unless I'm missing something.

What *cannot* work is storing in the files any information about
Python-level data, like "this piece of assembler assumes that there is
a module X with a class Y with a method Z".  I'm not going to repeat
again the multiple problems with that (see
http://pypy.readthedocs.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-already-compiled-machine-code).
However, it might be possible to save enough lower-level information
to avoid the problem completely.

The idea would be that when we're about the enter tracing mode and
there is a saved file, we use a "fast-path":

* we find a *likely* candidate from the saved file, based on some
explicitly-provided way to hash the Python code objects, for example
(it doesn't have to be a guaranteed match)

* this likely saved trace comes with a recording of all the *green*
guards that we originally did (both promotions and things that just
depend on purely green inputs).  (This means extra work for making
sure we save this information in the first place.)

* we run it like in the blackhole interpreter, but checking the result
of the green guards against the recorded ones

* we also take all green values that we get this way and pass them as
constants to the next step (see below, "**").

This means that we generalize (and lower-level-ize) the vague idea "a
module X with a class Y with a method Z" to be instead a series of
guards that need to give the same results as before.  They would
automatically check some subset of the new interpreter's state by
comparing it against the old's --- but only as much as the actual loop
happens to need.  For example, if we had in the (old) normal trace a
guard_value(p12, <constant pointer>), then of course it makes no sense
to record the old interpreter's constant pointer, which will change.
But it makes sense to record *what* was really deduced from this
constant pointer, i.e. all the green getfields and getarrayitems we
did.  And for example, if it was a PyCode object, we would record the
green switch that we did on the integer value that we got in the old
interpreter (which is the next opcode), even though that's all green.
That's the real condition: that we would follow the same path by
constant-folding the decisions on the green variables.

So, to finish the new interpreter's reloading: if the checking done
above passes, the next step is a fast-path through the assembler.  We
"just" need to reload the saved assembler as a sequence of bytes, and
fix all constants there.  To continue the example above, if a piece of
assembler was generated from the instruction guard_value(p12,
<constant pointer>), then the saved file must contain enough
information so that we know we must replace this old constant
pointer's value in the assembler with the new constant pointer's value
recorded above (at "**").

Overall, this would result in a much faster warm-up: faster tracing,
and no optimization nor regular assembler generation at all --- only a
very quick assembler reloading and fixing step.

There are of course complications from the fact that we don't simply
record loops, but bridges.  They might be seen in a different order in
the new process, so that when we are in the checking mode, we might
run the start of the loop, but then jump into the bridge --- even
though the loop was not fully seen so far.  This is not impossible to
implement by reloading the complete loop+bridge, but making the tail
of the loop invalid until we really run into it (with an extra
temporary guard).  And I'm sure that unrolling will somehow come with
its lot of funniness, as usual.

However, does it sound reasonable at all, or am I missing something else?


A bient?t,

Armin.

From arigo at tunes.org  Sat Feb  7 20:18:48 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 7 Feb 2015 20:18:48 +0100
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D54A58.9070201@gmail.com>
References: <54D54A58.9070201@gmail.com>
Message-ID: <CAMSv6X0+DtRmbirX5M6pqnOcqK5Qs1=wYKtPUaFFS8w8ynWWfw@mail.gmail.com>

Hi Tin,

Also, not an answer, but unlike what your mail implies we didn't
release a new version of PyPy 3 the other day --- only PyPy 2.  It's
not an answer because it should not be much slower.  The problem is
probably with some PyPy-3-specific optimizations that are lagging
behind PyPy 2's (although for a lot of them the code is shared and the
optimizations are done in the common RPython base).


A bient?t,

Armin.

From arigo at tunes.org  Sat Feb  7 20:30:32 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 7 Feb 2015 20:30:32 +0100
Subject: [pypy-dev] Another idea for improving warm-up times
In-Reply-To: <CAMSv6X1BHdW5-0ahBcty7jnfc3HnaQkUxGEurheHVTo+hs7Q5A@mail.gmail.com>
References: <CAMSv6X1BHdW5-0ahBcty7jnfc3HnaQkUxGEurheHVTo+hs7Q5A@mail.gmail.com>
Message-ID: <CAMSv6X2-BYypfDKS8iL3uCBK7kiDgHMBbYaF7VYhH8Rh-cz0eg@mail.gmail.com>

Re-hi,

In fact, after more discussion on irc, it seems that we can (almost or
completely) decide "we can reuse this saved trace" like this: "if we
trace it again, we follow the same path in the jitcodes".  Indeed,
this path implicitly encodes all the relevant information, like which
Python opcodes we see, the result of the other kinds of guards
(including what I called "green guards"), and so on.  What is left is
the issue that in the reused trace, we need to patch the constants to
be the ones seen during this new tracing.  (Well, and a lot of careful
review for anything that might have looked inside Consts between
tracing and assembler generation, notably in the optimizer.)


A bient?t,

Armin.

From njh at njhurst.com  Sat Feb  7 20:43:44 2015
From: njh at njhurst.com (Nathan Hurst)
Date: Sun, 8 Feb 2015 06:43:44 +1100
Subject: [pypy-dev] Another idea for improving warm-up times
In-Reply-To: <CAMSv6X2-BYypfDKS8iL3uCBK7kiDgHMBbYaF7VYhH8Rh-cz0eg@mail.gmail.com>
References: <CAMSv6X1BHdW5-0ahBcty7jnfc3HnaQkUxGEurheHVTo+hs7Q5A@mail.gmail.com>
 <CAMSv6X2-BYypfDKS8iL3uCBK7kiDgHMBbYaF7VYhH8Rh-cz0eg@mail.gmail.com>
Message-ID: <20150207194344.GB25755@ajhurst.org>

This sounds good to me, Armin.  I've always felt that a JIT that can't
learn from previous executions is never going to be able to make deep
optimisations.  And all mainstream jits have this problem, they get to
a certain level of optimisation technology and stagnate.  So I think
this is a really good direction to be thinking in.

My supportive 0.1 cent,
njh

On Sat, Feb 07, 2015 at 08:30:32PM +0100, Armin Rigo wrote:
> Re-hi,
> 
> In fact, after more discussion on irc, it seems that we can (almost or
> completely) decide "we can reuse this saved trace" like this: "if we
> trace it again, we follow the same path in the jitcodes".  Indeed,
> this path implicitly encodes all the relevant information, like which
> Python opcodes we see, the result of the other kinds of guards
> (including what I called "green guards"), and so on.  What is left is
> the issue that in the reused trace, we need to patch the constants to
> be the ones seen during this new tracing.  (Well, and a lot of careful
> review for anything that might have looked inside Consts between
> tracing and assembler generation, notably in the optimizer.)
> 
> 
> A bient?t,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From fijall at gmail.com  Sun Feb  8 11:17:44 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 8 Feb 2015 12:17:44 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D54A58.9070201@gmail.com>
References: <54D54A58.9070201@gmail.com>
Message-ID: <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>

Hi Tin

I have a bit sad news for you, but essentially from what I looked at
profling, the answer seems to be "don't use django, it has not been
written with performance in mind". For example here (which features in
django admin quite prominently, some stuff calls lazy() at each call):
https://github.com/django/django/blob/master/django/utils/functional.py#L48

What does it, per call that pypy does not like:

* redefines classes

* walks the class __mro__ and defines tons of new methods

* proxies everything through a not so efficietn mechanisms

Those things quite defy the purpose of the JIT - you're making tons of
very dynamic things that pypy generally assumes is "static enough", if
you wanted to do the same thing in C++, you would need to call gcc
with each request, that would not be so much fun.

If you really care about performance, consider swapping your web
framework to something more performance-oriented where JIT compilation
can help.

As a side note, we (me and my company, not to be confused with PyPy
open source project) do offer looking at proprietary code for
benchmarking purposes as a commercial service, look at
baroquesoftware.com

Cheers,
fijal



On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> Hello, PyPy folks!
>
> While trying to speed up one of my Django sites, I noticed a new version of
> PyPy
> had just been released. So I grabbed a fresh download of PyPy 3 (since this
> is
> a Python 3 codebase) and tried taking it out for a spin.
>
> However, as far as I can see, whatever I try PyPy is consistently slower
> than
> CPython for this.
>
> Since this is a proprietary site, I've basically ripped out all the code
> except
> my settings.py and my requirements; and am benchmarking the Django admin
> index.
> The results are about the same.
>
> I've set up a small repo that can be used to reproduce the environment:
> https://github.com/Tinche/PyPy-Django-Playground. There's additional info in
> the README there.
>
> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is the
> system Python, 3.4. PyPy has been downloaded from the official site and
> unzipped.
>
> So what I basically do is set up an admin session, and use the Django main
> admin
> page. 200 warmup requests, then 100 benchmarked requests, look at the mean
> request time.
>
> Some results:
>
> Django's runserver, DEBUG mode:
>
> PyPy3            485.389 [ms]
> CPython 3.4      105.777 [ms]
>
> Django's runserver, no debug:
>
> PyPy3             44.661 [ms]
> CPython 3.4       18.697 [ms]
>
> Gunicorn, 1 worker, no debug:
>
> PyPy3             28.615 [ms]
> CPython 3.4       13.532 [ms]
>
> I don't exactly claim to be an expert on benchmarking, but assuming my site
> is similar to the Django admin, CPython's gonna be giving me better
> performance.
> Also the debug runserver performance is kinda worrying. Nobody's going to be
> running this in production, but it makes development a little slower and
> more
> annoying than it should be.
>
> Is there anything to make PyPy more competitive in these kinds of scenarios?
>
> Kind regards,
> Tin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From omer.drow at gmail.com  Sun Feb  8 13:09:32 2015
From: omer.drow at gmail.com (Omer Katz)
Date: Sun, 8 Feb 2015 14:09:32 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
Message-ID: <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>

Isn't there anything we can do about it?
We can open issues at the Django issue tracker if some code slows down
execution in PyPy.

2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:

> Hi Tin
>
> I have a bit sad news for you, but essentially from what I looked at
> profling, the answer seems to be "don't use django, it has not been
> written with performance in mind". For example here (which features in
> django admin quite prominently, some stuff calls lazy() at each call):
> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>
> What does it, per call that pypy does not like:
>
> * redefines classes
>
> * walks the class __mro__ and defines tons of new methods
>
> * proxies everything through a not so efficietn mechanisms
>
> Those things quite defy the purpose of the JIT - you're making tons of
> very dynamic things that pypy generally assumes is "static enough", if
> you wanted to do the same thing in C++, you would need to call gcc
> with each request, that would not be so much fun.
>
> If you really care about performance, consider swapping your web
> framework to something more performance-oriented where JIT compilation
> can help.
>
> As a side note, we (me and my company, not to be confused with PyPy
> open source project) do offer looking at proprietary code for
> benchmarking purposes as a commercial service, look at
> baroquesoftware.com
>
> Cheers,
> fijal
>
>
>
> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
> wrote:
> > Hello, PyPy folks!
> >
> > While trying to speed up one of my Django sites, I noticed a new version
> of
> > PyPy
> > had just been released. So I grabbed a fresh download of PyPy 3 (since
> this
> > is
> > a Python 3 codebase) and tried taking it out for a spin.
> >
> > However, as far as I can see, whatever I try PyPy is consistently slower
> > than
> > CPython for this.
> >
> > Since this is a proprietary site, I've basically ripped out all the code
> > except
> > my settings.py and my requirements; and am benchmarking the Django admin
> > index.
> > The results are about the same.
> >
> > I've set up a small repo that can be used to reproduce the environment:
> > https://github.com/Tinche/PyPy-Django-Playground. There's additional
> info in
> > the README there.
> >
> > These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is
> the
> > system Python, 3.4. PyPy has been downloaded from the official site and
> > unzipped.
> >
> > So what I basically do is set up an admin session, and use the Django
> main
> > admin
> > page. 200 warmup requests, then 100 benchmarked requests, look at the
> mean
> > request time.
> >
> > Some results:
> >
> > Django's runserver, DEBUG mode:
> >
> > PyPy3            485.389 [ms]
> > CPython 3.4      105.777 [ms]
> >
> > Django's runserver, no debug:
> >
> > PyPy3             44.661 [ms]
> > CPython 3.4       18.697 [ms]
> >
> > Gunicorn, 1 worker, no debug:
> >
> > PyPy3             28.615 [ms]
> > CPython 3.4       13.532 [ms]
> >
> > I don't exactly claim to be an expert on benchmarking, but assuming my
> site
> > is similar to the Django admin, CPython's gonna be giving me better
> > performance.
> > Also the debug runserver performance is kinda worrying. Nobody's going
> to be
> > running this in production, but it makes development a little slower and
> > more
> > annoying than it should be.
> >
> > Is there anything to make PyPy more competitive in these kinds of
> scenarios?
> >
> > Kind regards,
> > Tin
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150208/b5536da4/attachment.html>

From fijall at gmail.com  Sun Feb  8 15:59:49 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 8 Feb 2015 16:59:49 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
Message-ID: <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>

I don't know :-( not sure if fixing those issues one by one is the way
to go, it's not like *this* particular code is a problem, it's the
culture that breeds those sort of things

On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
> Isn't there anything we can do about it?
> We can open issues at the Django issue tracker if some code slows down
> execution in PyPy.
>
> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>
>> Hi Tin
>>
>> I have a bit sad news for you, but essentially from what I looked at
>> profling, the answer seems to be "don't use django, it has not been
>> written with performance in mind". For example here (which features in
>> django admin quite prominently, some stuff calls lazy() at each call):
>>
>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>
>> What does it, per call that pypy does not like:
>>
>> * redefines classes
>>
>> * walks the class __mro__ and defines tons of new methods
>>
>> * proxies everything through a not so efficietn mechanisms
>>
>> Those things quite defy the purpose of the JIT - you're making tons of
>> very dynamic things that pypy generally assumes is "static enough", if
>> you wanted to do the same thing in C++, you would need to call gcc
>> with each request, that would not be so much fun.
>>
>> If you really care about performance, consider swapping your web
>> framework to something more performance-oriented where JIT compilation
>> can help.
>>
>> As a side note, we (me and my company, not to be confused with PyPy
>> open source project) do offer looking at proprietary code for
>> benchmarking purposes as a commercial service, look at
>> baroquesoftware.com
>>
>> Cheers,
>> fijal
>>
>>
>>
>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>> wrote:
>> > Hello, PyPy folks!
>> >
>> > While trying to speed up one of my Django sites, I noticed a new version
>> > of
>> > PyPy
>> > had just been released. So I grabbed a fresh download of PyPy 3 (since
>> > this
>> > is
>> > a Python 3 codebase) and tried taking it out for a spin.
>> >
>> > However, as far as I can see, whatever I try PyPy is consistently slower
>> > than
>> > CPython for this.
>> >
>> > Since this is a proprietary site, I've basically ripped out all the code
>> > except
>> > my settings.py and my requirements; and am benchmarking the Django admin
>> > index.
>> > The results are about the same.
>> >
>> > I've set up a small repo that can be used to reproduce the environment:
>> > https://github.com/Tinche/PyPy-Django-Playground. There's additional
>> > info in
>> > the README there.
>> >
>> > These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is
>> > the
>> > system Python, 3.4. PyPy has been downloaded from the official site and
>> > unzipped.
>> >
>> > So what I basically do is set up an admin session, and use the Django
>> > main
>> > admin
>> > page. 200 warmup requests, then 100 benchmarked requests, look at the
>> > mean
>> > request time.
>> >
>> > Some results:
>> >
>> > Django's runserver, DEBUG mode:
>> >
>> > PyPy3            485.389 [ms]
>> > CPython 3.4      105.777 [ms]
>> >
>> > Django's runserver, no debug:
>> >
>> > PyPy3             44.661 [ms]
>> > CPython 3.4       18.697 [ms]
>> >
>> > Gunicorn, 1 worker, no debug:
>> >
>> > PyPy3             28.615 [ms]
>> > CPython 3.4       13.532 [ms]
>> >
>> > I don't exactly claim to be an expert on benchmarking, but assuming my
>> > site
>> > is similar to the Django admin, CPython's gonna be giving me better
>> > performance.
>> > Also the debug runserver performance is kinda worrying. Nobody's going
>> > to be
>> > running this in production, but it makes development a little slower and
>> > more
>> > annoying than it should be.
>> >
>> > Is there anything to make PyPy more competitive in these kinds of
>> > scenarios?
>> >
>> > Kind regards,
>> > Tin
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>
>

From alex.gaynor at gmail.com  Sun Feb  8 16:08:28 2015
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sun, 8 Feb 2015 07:08:28 -0800
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
Message-ID: <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>

FWIW, I've definitely seen and worked on Django sites that got major
improvements out of PyPy -- both the templates and ORM are both sped up
substantially by it. I think we should try to fix issues as we see them,
before writing it off.

FWIW: lazy does not create a new class per call of a function, it's only
one class per decorated function.

Alex

On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> I don't know :-( not sure if fixing those issues one by one is the way
> to go, it's not like *this* particular code is a problem, it's the
> culture that breeds those sort of things
>
> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
> > Isn't there anything we can do about it?
> > We can open issues at the Django issue tracker if some code slows down
> > execution in PyPy.
> >
> > 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
> >>
> >> Hi Tin
> >>
> >> I have a bit sad news for you, but essentially from what I looked at
> >> profling, the answer seems to be "don't use django, it has not been
> >> written with performance in mind". For example here (which features in
> >> django admin quite prominently, some stuff calls lazy() at each call):
> >>
> >>
> https://github.com/django/django/blob/master/django/utils/functional.py#L48
> >>
> >> What does it, per call that pypy does not like:
> >>
> >> * redefines classes
> >>
> >> * walks the class __mro__ and defines tons of new methods
> >>
> >> * proxies everything through a not so efficietn mechanisms
> >>
> >> Those things quite defy the purpose of the JIT - you're making tons of
> >> very dynamic things that pypy generally assumes is "static enough", if
> >> you wanted to do the same thing in C++, you would need to call gcc
> >> with each request, that would not be so much fun.
> >>
> >> If you really care about performance, consider swapping your web
> >> framework to something more performance-oriented where JIT compilation
> >> can help.
> >>
> >> As a side note, we (me and my company, not to be confused with PyPy
> >> open source project) do offer looking at proprietary code for
> >> benchmarking purposes as a commercial service, look at
> >> baroquesoftware.com
> >>
> >> Cheers,
> >> fijal
> >>
> >>
> >>
> >> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
> >> wrote:
> >> > Hello, PyPy folks!
> >> >
> >> > While trying to speed up one of my Django sites, I noticed a new
> version
> >> > of
> >> > PyPy
> >> > had just been released. So I grabbed a fresh download of PyPy 3 (since
> >> > this
> >> > is
> >> > a Python 3 codebase) and tried taking it out for a spin.
> >> >
> >> > However, as far as I can see, whatever I try PyPy is consistently
> slower
> >> > than
> >> > CPython for this.
> >> >
> >> > Since this is a proprietary site, I've basically ripped out all the
> code
> >> > except
> >> > my settings.py and my requirements; and am benchmarking the Django
> admin
> >> > index.
> >> > The results are about the same.
> >> >
> >> > I've set up a small repo that can be used to reproduce the
> environment:
> >> > https://github.com/Tinche/PyPy-Django-Playground. There's additional
> >> > info in
> >> > the README there.
> >> >
> >> > These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
> is
> >> > the
> >> > system Python, 3.4. PyPy has been downloaded from the official site
> and
> >> > unzipped.
> >> >
> >> > So what I basically do is set up an admin session, and use the Django
> >> > main
> >> > admin
> >> > page. 200 warmup requests, then 100 benchmarked requests, look at the
> >> > mean
> >> > request time.
> >> >
> >> > Some results:
> >> >
> >> > Django's runserver, DEBUG mode:
> >> >
> >> > PyPy3            485.389 [ms]
> >> > CPython 3.4      105.777 [ms]
> >> >
> >> > Django's runserver, no debug:
> >> >
> >> > PyPy3             44.661 [ms]
> >> > CPython 3.4       18.697 [ms]
> >> >
> >> > Gunicorn, 1 worker, no debug:
> >> >
> >> > PyPy3             28.615 [ms]
> >> > CPython 3.4       13.532 [ms]
> >> >
> >> > I don't exactly claim to be an expert on benchmarking, but assuming my
> >> > site
> >> > is similar to the Django admin, CPython's gonna be giving me better
> >> > performance.
> >> > Also the debug runserver performance is kinda worrying. Nobody's going
> >> > to be
> >> > running this in production, but it makes development a little slower
> and
> >> > more
> >> > annoying than it should be.
> >> >
> >> > Is there anything to make PyPy more competitive in these kinds of
> >> > scenarios?
> >> >
> >> > Kind regards,
> >> > Tin
> >> > _______________________________________________
> >> > pypy-dev mailing list
> >> > pypy-dev at python.org
> >> > https://mail.python.org/mailman/listinfo/pypy-dev
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pypy-dev
> >
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150208/78973681/attachment-0001.html>

From fijall at gmail.com  Sun Feb  8 18:44:32 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 8 Feb 2015 19:44:32 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
Message-ID: <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>

then it's used in the wrong ways in say django admin, look at
invocations to lazy there (this is how it showed up in my profile)

On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> FWIW, I've definitely seen and worked on Django sites that got major
> improvements out of PyPy -- both the templates and ORM are both sped up
> substantially by it. I think we should try to fix issues as we see them,
> before writing it off.
>
> FWIW: lazy does not create a new class per call of a function, it's only one
> class per decorated function.
>
> Alex
>
> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>> I don't know :-( not sure if fixing those issues one by one is the way
>> to go, it's not like *this* particular code is a problem, it's the
>> culture that breeds those sort of things
>>
>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
>> > Isn't there anything we can do about it?
>> > We can open issues at the Django issue tracker if some code slows down
>> > execution in PyPy.
>> >
>> > 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>> >>
>> >> Hi Tin
>> >>
>> >> I have a bit sad news for you, but essentially from what I looked at
>> >> profling, the answer seems to be "don't use django, it has not been
>> >> written with performance in mind". For example here (which features in
>> >> django admin quite prominently, some stuff calls lazy() at each call):
>> >>
>> >>
>> >> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>> >>
>> >> What does it, per call that pypy does not like:
>> >>
>> >> * redefines classes
>> >>
>> >> * walks the class __mro__ and defines tons of new methods
>> >>
>> >> * proxies everything through a not so efficietn mechanisms
>> >>
>> >> Those things quite defy the purpose of the JIT - you're making tons of
>> >> very dynamic things that pypy generally assumes is "static enough", if
>> >> you wanted to do the same thing in C++, you would need to call gcc
>> >> with each request, that would not be so much fun.
>> >>
>> >> If you really care about performance, consider swapping your web
>> >> framework to something more performance-oriented where JIT compilation
>> >> can help.
>> >>
>> >> As a side note, we (me and my company, not to be confused with PyPy
>> >> open source project) do offer looking at proprietary code for
>> >> benchmarking purposes as a commercial service, look at
>> >> baroquesoftware.com
>> >>
>> >> Cheers,
>> >> fijal
>> >>
>> >>
>> >>
>> >> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>> >> wrote:
>> >> > Hello, PyPy folks!
>> >> >
>> >> > While trying to speed up one of my Django sites, I noticed a new
>> >> > version
>> >> > of
>> >> > PyPy
>> >> > had just been released. So I grabbed a fresh download of PyPy 3
>> >> > (since
>> >> > this
>> >> > is
>> >> > a Python 3 codebase) and tried taking it out for a spin.
>> >> >
>> >> > However, as far as I can see, whatever I try PyPy is consistently
>> >> > slower
>> >> > than
>> >> > CPython for this.
>> >> >
>> >> > Since this is a proprietary site, I've basically ripped out all the
>> >> > code
>> >> > except
>> >> > my settings.py and my requirements; and am benchmarking the Django
>> >> > admin
>> >> > index.
>> >> > The results are about the same.
>> >> >
>> >> > I've set up a small repo that can be used to reproduce the
>> >> > environment:
>> >> > https://github.com/Tinche/PyPy-Django-Playground. There's additional
>> >> > info in
>> >> > the README there.
>> >> >
>> >> > These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
>> >> > is
>> >> > the
>> >> > system Python, 3.4. PyPy has been downloaded from the official site
>> >> > and
>> >> > unzipped.
>> >> >
>> >> > So what I basically do is set up an admin session, and use the Django
>> >> > main
>> >> > admin
>> >> > page. 200 warmup requests, then 100 benchmarked requests, look at the
>> >> > mean
>> >> > request time.
>> >> >
>> >> > Some results:
>> >> >
>> >> > Django's runserver, DEBUG mode:
>> >> >
>> >> > PyPy3            485.389 [ms]
>> >> > CPython 3.4      105.777 [ms]
>> >> >
>> >> > Django's runserver, no debug:
>> >> >
>> >> > PyPy3             44.661 [ms]
>> >> > CPython 3.4       18.697 [ms]
>> >> >
>> >> > Gunicorn, 1 worker, no debug:
>> >> >
>> >> > PyPy3             28.615 [ms]
>> >> > CPython 3.4       13.532 [ms]
>> >> >
>> >> > I don't exactly claim to be an expert on benchmarking, but assuming
>> >> > my
>> >> > site
>> >> > is similar to the Django admin, CPython's gonna be giving me better
>> >> > performance.
>> >> > Also the debug runserver performance is kinda worrying. Nobody's
>> >> > going
>> >> > to be
>> >> > running this in production, but it makes development a little slower
>> >> > and
>> >> > more
>> >> > annoying than it should be.
>> >> >
>> >> > Is there anything to make PyPy more competitive in these kinds of
>> >> > scenarios?
>> >> >
>> >> > Kind regards,
>> >> > Tin
>> >> > _______________________________________________
>> >> > pypy-dev mailing list
>> >> > pypy-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >> _______________________________________________
>> >> pypy-dev mailing list
>> >> pypy-dev at python.org
>> >> https://mail.python.org/mailman/listinfo/pypy-dev
>> >
>> >
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084

From tinchester at gmail.com  Sun Feb  8 19:23:24 2015
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Sun, 08 Feb 2015 19:23:24 +0100
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
Message-ID: <54D7A99C.3060801@gmail.com>

Hello,

thanks to everyone for their input (especially Maciej).

Since I've ripped out all my code from the test project, it's not a 
Python 3 project anymore, so I did try PyPy 2 with it too. It's faster, 
yes; the last test looked something like:

PyPy 2         20.206 [ms] (mean)
PyPy 3         27.588 [ms] (mean)
CPython 3.4    10.865 [ms] (mean)

(tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu PPA - 
about the same.)

I don't know, maybe the admin page isn't a good test candidate since it 
might do more introspective/reflective/meta stuff. I've tried doing this 
same test (10k warmup requests, 100 benchmark requests) on the main page 
of the site I'm working on; the results are:

PyPy 3         15.863 [ms] (mean)
CPython 3.48.668 [ms] (mean)

and I can be certain there's nothing much going on there. Just some 
permission checks, grabbing some stuff from the database, checking some 
query parameters and a template render.

I suppose if folks have CPU intensive stuff in their view functions 
(nested for-loops etc) the optimizations PyPy can do to that far 
outweigh the non-optimized Django parts; my sites generally don't though.

On 08.02.2015 18:44, Maciej Fijalkowski wrote:
> then it's used in the wrong ways in say django admin, look at
> invocations to lazy there (this is how it showed up in my profile)
>
> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>> FWIW, I've definitely seen and worked on Django sites that got major
>> improvements out of PyPy -- both the templates and ORM are both sped up
>> substantially by it. I think we should try to fix issues as we see them,
>> before writing it off.
>>
>> FWIW: lazy does not create a new class per call of a function, it's only one
>> class per decorated function.
>>
>> Alex
>>
>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> I don't know :-( not sure if fixing those issues one by one is the way
>>> to go, it's not like *this* particular code is a problem, it's the
>>> culture that breeds those sort of things
>>>
>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
>>>> Isn't there anything we can do about it?
>>>> We can open issues at the Django issue tracker if some code slows down
>>>> execution in PyPy.
>>>>
>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>>>> Hi Tin
>>>>>
>>>>> I have a bit sad news for you, but essentially from what I looked at
>>>>> profling, the answer seems to be "don't use django, it has not been
>>>>> written with performance in mind". For example here (which features in
>>>>> django admin quite prominently, some stuff calls lazy() at each call):
>>>>>
>>>>>
>>>>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>>>>
>>>>> What does it, per call that pypy does not like:
>>>>>
>>>>> * redefines classes
>>>>>
>>>>> * walks the class __mro__ and defines tons of new methods
>>>>>
>>>>> * proxies everything through a not so efficietn mechanisms
>>>>>
>>>>> Those things quite defy the purpose of the JIT - you're making tons of
>>>>> very dynamic things that pypy generally assumes is "static enough", if
>>>>> you wanted to do the same thing in C++, you would need to call gcc
>>>>> with each request, that would not be so much fun.
>>>>>
>>>>> If you really care about performance, consider swapping your web
>>>>> framework to something more performance-oriented where JIT compilation
>>>>> can help.
>>>>>
>>>>> As a side note, we (me and my company, not to be confused with PyPy
>>>>> open source project) do offer looking at proprietary code for
>>>>> benchmarking purposes as a commercial service, look at
>>>>> baroquesoftware.com
>>>>>
>>>>> Cheers,
>>>>> fijal
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>>>>> wrote:
>>>>>> Hello, PyPy folks!
>>>>>>
>>>>>> While trying to speed up one of my Django sites, I noticed a new
>>>>>> version
>>>>>> of
>>>>>> PyPy
>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
>>>>>> (since
>>>>>> this
>>>>>> is
>>>>>> a Python 3 codebase) and tried taking it out for a spin.
>>>>>>
>>>>>> However, as far as I can see, whatever I try PyPy is consistently
>>>>>> slower
>>>>>> than
>>>>>> CPython for this.
>>>>>>
>>>>>> Since this is a proprietary site, I've basically ripped out all the
>>>>>> code
>>>>>> except
>>>>>> my settings.py and my requirements; and am benchmarking the Django
>>>>>> admin
>>>>>> index.
>>>>>> The results are about the same.
>>>>>>
>>>>>> I've set up a small repo that can be used to reproduce the
>>>>>> environment:
>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's additional
>>>>>> info in
>>>>>> the README there.
>>>>>>
>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
>>>>>> is
>>>>>> the
>>>>>> system Python, 3.4. PyPy has been downloaded from the official site
>>>>>> and
>>>>>> unzipped.
>>>>>>
>>>>>> So what I basically do is set up an admin session, and use the Django
>>>>>> main
>>>>>> admin
>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look at the
>>>>>> mean
>>>>>> request time.
>>>>>>
>>>>>> Some results:
>>>>>>
>>>>>> Django's runserver, DEBUG mode:
>>>>>>
>>>>>> PyPy3            485.389 [ms]
>>>>>> CPython 3.4      105.777 [ms]
>>>>>>
>>>>>> Django's runserver, no debug:
>>>>>>
>>>>>> PyPy3             44.661 [ms]
>>>>>> CPython 3.4       18.697 [ms]
>>>>>>
>>>>>> Gunicorn, 1 worker, no debug:
>>>>>>
>>>>>> PyPy3             28.615 [ms]
>>>>>> CPython 3.4       13.532 [ms]
>>>>>>
>>>>>> I don't exactly claim to be an expert on benchmarking, but assuming
>>>>>> my
>>>>>> site
>>>>>> is similar to the Django admin, CPython's gonna be giving me better
>>>>>> performance.
>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
>>>>>> going
>>>>>> to be
>>>>>> running this in production, but it makes development a little slower
>>>>>> and
>>>>>> more
>>>>>> annoying than it should be.
>>>>>>
>>>>>> Is there anything to make PyPy more competitive in these kinds of
>>>>>> scenarios?
>>>>>>
>>>>>> Kind regards,
>>>>>> Tin
>>>>>> _______________________________________________
>>>>>> pypy-dev mailing list
>>>>>> pypy-dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>> _______________________________________________
>>>>> pypy-dev mailing list
>>>>> pypy-dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>>
>> --
>> "I disapprove of what you say, but I will defend to the death your right to
>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>> "The people's good is the highest law." -- Cicero
>> GPG Key fingerprint: 125F 5C67 DFE9 4084


From fijall at gmail.com  Sun Feb  8 19:30:53 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 8 Feb 2015 20:30:53 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D7A99C.3060801@gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
 <54D7A99C.3060801@gmail.com>
Message-ID: <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>

Hey Tin.

We are in the process of two (trivial) pull requests to django that
drop the rendering time from 25ms -> 8ms for this case. I'm not a
farseer, however I suspect something like this can be done with your
site too (depending what it really does).

On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> Hello,
>
> thanks to everyone for their input (especially Maciej).
>
> Since I've ripped out all my code from the test project, it's not a Python 3
> project anymore, so I did try PyPy 2 with it too. It's faster, yes; the last
> test looked something like:
>
> PyPy 2         20.206 [ms] (mean)
> PyPy 3         27.588 [ms] (mean)
> CPython 3.4    10.865 [ms] (mean)
>
> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu PPA -
> about the same.)
>
> I don't know, maybe the admin page isn't a good test candidate since it
> might do more introspective/reflective/meta stuff. I've tried doing this
> same test (10k warmup requests, 100 benchmark requests) on the main page of
> the site I'm working on; the results are:
>
> PyPy 3         15.863 [ms] (mean)
> CPython 3.48.668 [ms] (mean)
>
> and I can be certain there's nothing much going on there. Just some
> permission checks, grabbing some stuff from the database, checking some
> query parameters and a template render.
>
> I suppose if folks have CPU intensive stuff in their view functions (nested
> for-loops etc) the optimizations PyPy can do to that far outweigh the
> non-optimized Django parts; my sites generally don't though.
>
>
> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
>>
>> then it's used in the wrong ways in say django admin, look at
>> invocations to lazy there (this is how it showed up in my profile)
>>
>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>>>
>>> FWIW, I've definitely seen and worked on Django sites that got major
>>> improvements out of PyPy -- both the templates and ORM are both sped up
>>> substantially by it. I think we should try to fix issues as we see them,
>>> before writing it off.
>>>
>>> FWIW: lazy does not create a new class per call of a function, it's only
>>> one
>>> class per decorated function.
>>>
>>> Alex
>>>
>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com>
>>> wrote:
>>>>
>>>> I don't know :-( not sure if fixing those issues one by one is the way
>>>> to go, it's not like *this* particular code is a problem, it's the
>>>> culture that breeds those sort of things
>>>>
>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
>>>>>
>>>>> Isn't there anything we can do about it?
>>>>> We can open issues at the Django issue tracker if some code slows down
>>>>> execution in PyPy.
>>>>>
>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>>>>>
>>>>>> Hi Tin
>>>>>>
>>>>>> I have a bit sad news for you, but essentially from what I looked at
>>>>>> profling, the answer seems to be "don't use django, it has not been
>>>>>> written with performance in mind". For example here (which features in
>>>>>> django admin quite prominently, some stuff calls lazy() at each call):
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>>>>>
>>>>>> What does it, per call that pypy does not like:
>>>>>>
>>>>>> * redefines classes
>>>>>>
>>>>>> * walks the class __mro__ and defines tons of new methods
>>>>>>
>>>>>> * proxies everything through a not so efficietn mechanisms
>>>>>>
>>>>>> Those things quite defy the purpose of the JIT - you're making tons of
>>>>>> very dynamic things that pypy generally assumes is "static enough", if
>>>>>> you wanted to do the same thing in C++, you would need to call gcc
>>>>>> with each request, that would not be so much fun.
>>>>>>
>>>>>> If you really care about performance, consider swapping your web
>>>>>> framework to something more performance-oriented where JIT compilation
>>>>>> can help.
>>>>>>
>>>>>> As a side note, we (me and my company, not to be confused with PyPy
>>>>>> open source project) do offer looking at proprietary code for
>>>>>> benchmarking purposes as a commercial service, look at
>>>>>> baroquesoftware.com
>>>>>>
>>>>>> Cheers,
>>>>>> fijal
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello, PyPy folks!
>>>>>>>
>>>>>>> While trying to speed up one of my Django sites, I noticed a new
>>>>>>> version
>>>>>>> of
>>>>>>> PyPy
>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
>>>>>>> (since
>>>>>>> this
>>>>>>> is
>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
>>>>>>>
>>>>>>> However, as far as I can see, whatever I try PyPy is consistently
>>>>>>> slower
>>>>>>> than
>>>>>>> CPython for this.
>>>>>>>
>>>>>>> Since this is a proprietary site, I've basically ripped out all the
>>>>>>> code
>>>>>>> except
>>>>>>> my settings.py and my requirements; and am benchmarking the Django
>>>>>>> admin
>>>>>>> index.
>>>>>>> The results are about the same.
>>>>>>>
>>>>>>> I've set up a small repo that can be used to reproduce the
>>>>>>> environment:
>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's additional
>>>>>>> info in
>>>>>>> the README there.
>>>>>>>
>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
>>>>>>> is
>>>>>>> the
>>>>>>> system Python, 3.4. PyPy has been downloaded from the official site
>>>>>>> and
>>>>>>> unzipped.
>>>>>>>
>>>>>>> So what I basically do is set up an admin session, and use the Django
>>>>>>> main
>>>>>>> admin
>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look at the
>>>>>>> mean
>>>>>>> request time.
>>>>>>>
>>>>>>> Some results:
>>>>>>>
>>>>>>> Django's runserver, DEBUG mode:
>>>>>>>
>>>>>>> PyPy3            485.389 [ms]
>>>>>>> CPython 3.4      105.777 [ms]
>>>>>>>
>>>>>>> Django's runserver, no debug:
>>>>>>>
>>>>>>> PyPy3             44.661 [ms]
>>>>>>> CPython 3.4       18.697 [ms]
>>>>>>>
>>>>>>> Gunicorn, 1 worker, no debug:
>>>>>>>
>>>>>>> PyPy3             28.615 [ms]
>>>>>>> CPython 3.4       13.532 [ms]
>>>>>>>
>>>>>>> I don't exactly claim to be an expert on benchmarking, but assuming
>>>>>>> my
>>>>>>> site
>>>>>>> is similar to the Django admin, CPython's gonna be giving me better
>>>>>>> performance.
>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
>>>>>>> going
>>>>>>> to be
>>>>>>> running this in production, but it makes development a little slower
>>>>>>> and
>>>>>>> more
>>>>>>> annoying than it should be.
>>>>>>>
>>>>>>> Is there anything to make PyPy more competitive in these kinds of
>>>>>>> scenarios?
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Tin
>>>>>>> _______________________________________________
>>>>>>> pypy-dev mailing list
>>>>>>> pypy-dev at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> pypy-dev mailing list
>>>>>> pypy-dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>
>>>>>
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>>>
>>>
>>> --
>>> "I disapprove of what you say, but I will defend to the death your right
>>> to
>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>>> "The people's good is the highest law." -- Cicero
>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>
>

From fijall at gmail.com  Sun Feb  8 23:09:02 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 9 Feb 2015 00:09:02 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
 <54D7A99C.3060801@gmail.com>
 <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>
Message-ID: <CAK5idxTz1wswJbR8u60wQ4czS+vdahyjWEFm6Rpd6tkmHZYewQ@mail.gmail.com>

It got merged into django, PyPy (2, didn't test the 3) is now faster
than cpython on django admin. It likely does not help your cause
though so you need to come up with a better benchmark :-)

On Sun, Feb 8, 2015 at 8:30 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hey Tin.
>
> We are in the process of two (trivial) pull requests to django that
> drop the rendering time from 25ms -> 8ms for this case. I'm not a
> farseer, however I suspect something like this can be done with your
> site too (depending what it really does).
>
> On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
>> Hello,
>>
>> thanks to everyone for their input (especially Maciej).
>>
>> Since I've ripped out all my code from the test project, it's not a Python 3
>> project anymore, so I did try PyPy 2 with it too. It's faster, yes; the last
>> test looked something like:
>>
>> PyPy 2         20.206 [ms] (mean)
>> PyPy 3         27.588 [ms] (mean)
>> CPython 3.4    10.865 [ms] (mean)
>>
>> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu PPA -
>> about the same.)
>>
>> I don't know, maybe the admin page isn't a good test candidate since it
>> might do more introspective/reflective/meta stuff. I've tried doing this
>> same test (10k warmup requests, 100 benchmark requests) on the main page of
>> the site I'm working on; the results are:
>>
>> PyPy 3         15.863 [ms] (mean)
>> CPython 3.48.668 [ms] (mean)
>>
>> and I can be certain there's nothing much going on there. Just some
>> permission checks, grabbing some stuff from the database, checking some
>> query parameters and a template render.
>>
>> I suppose if folks have CPU intensive stuff in their view functions (nested
>> for-loops etc) the optimizations PyPy can do to that far outweigh the
>> non-optimized Django parts; my sites generally don't though.
>>
>>
>> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
>>>
>>> then it's used in the wrong ways in say django admin, look at
>>> invocations to lazy there (this is how it showed up in my profile)
>>>
>>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>>>>
>>>> FWIW, I've definitely seen and worked on Django sites that got major
>>>> improvements out of PyPy -- both the templates and ORM are both sped up
>>>> substantially by it. I think we should try to fix issues as we see them,
>>>> before writing it off.
>>>>
>>>> FWIW: lazy does not create a new class per call of a function, it's only
>>>> one
>>>> class per decorated function.
>>>>
>>>> Alex
>>>>
>>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com>
>>>> wrote:
>>>>>
>>>>> I don't know :-( not sure if fixing those issues one by one is the way
>>>>> to go, it's not like *this* particular code is a problem, it's the
>>>>> culture that breeds those sort of things
>>>>>
>>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
>>>>>>
>>>>>> Isn't there anything we can do about it?
>>>>>> We can open issues at the Django issue tracker if some code slows down
>>>>>> execution in PyPy.
>>>>>>
>>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>>>>>>
>>>>>>> Hi Tin
>>>>>>>
>>>>>>> I have a bit sad news for you, but essentially from what I looked at
>>>>>>> profling, the answer seems to be "don't use django, it has not been
>>>>>>> written with performance in mind". For example here (which features in
>>>>>>> django admin quite prominently, some stuff calls lazy() at each call):
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>>>>>>
>>>>>>> What does it, per call that pypy does not like:
>>>>>>>
>>>>>>> * redefines classes
>>>>>>>
>>>>>>> * walks the class __mro__ and defines tons of new methods
>>>>>>>
>>>>>>> * proxies everything through a not so efficietn mechanisms
>>>>>>>
>>>>>>> Those things quite defy the purpose of the JIT - you're making tons of
>>>>>>> very dynamic things that pypy generally assumes is "static enough", if
>>>>>>> you wanted to do the same thing in C++, you would need to call gcc
>>>>>>> with each request, that would not be so much fun.
>>>>>>>
>>>>>>> If you really care about performance, consider swapping your web
>>>>>>> framework to something more performance-oriented where JIT compilation
>>>>>>> can help.
>>>>>>>
>>>>>>> As a side note, we (me and my company, not to be confused with PyPy
>>>>>>> open source project) do offer looking at proprietary code for
>>>>>>> benchmarking purposes as a commercial service, look at
>>>>>>> baroquesoftware.com
>>>>>>>
>>>>>>> Cheers,
>>>>>>> fijal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello, PyPy folks!
>>>>>>>>
>>>>>>>> While trying to speed up one of my Django sites, I noticed a new
>>>>>>>> version
>>>>>>>> of
>>>>>>>> PyPy
>>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
>>>>>>>> (since
>>>>>>>> this
>>>>>>>> is
>>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
>>>>>>>>
>>>>>>>> However, as far as I can see, whatever I try PyPy is consistently
>>>>>>>> slower
>>>>>>>> than
>>>>>>>> CPython for this.
>>>>>>>>
>>>>>>>> Since this is a proprietary site, I've basically ripped out all the
>>>>>>>> code
>>>>>>>> except
>>>>>>>> my settings.py and my requirements; and am benchmarking the Django
>>>>>>>> admin
>>>>>>>> index.
>>>>>>>> The results are about the same.
>>>>>>>>
>>>>>>>> I've set up a small repo that can be used to reproduce the
>>>>>>>> environment:
>>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's additional
>>>>>>>> info in
>>>>>>>> the README there.
>>>>>>>>
>>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
>>>>>>>> is
>>>>>>>> the
>>>>>>>> system Python, 3.4. PyPy has been downloaded from the official site
>>>>>>>> and
>>>>>>>> unzipped.
>>>>>>>>
>>>>>>>> So what I basically do is set up an admin session, and use the Django
>>>>>>>> main
>>>>>>>> admin
>>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look at the
>>>>>>>> mean
>>>>>>>> request time.
>>>>>>>>
>>>>>>>> Some results:
>>>>>>>>
>>>>>>>> Django's runserver, DEBUG mode:
>>>>>>>>
>>>>>>>> PyPy3            485.389 [ms]
>>>>>>>> CPython 3.4      105.777 [ms]
>>>>>>>>
>>>>>>>> Django's runserver, no debug:
>>>>>>>>
>>>>>>>> PyPy3             44.661 [ms]
>>>>>>>> CPython 3.4       18.697 [ms]
>>>>>>>>
>>>>>>>> Gunicorn, 1 worker, no debug:
>>>>>>>>
>>>>>>>> PyPy3             28.615 [ms]
>>>>>>>> CPython 3.4       13.532 [ms]
>>>>>>>>
>>>>>>>> I don't exactly claim to be an expert on benchmarking, but assuming
>>>>>>>> my
>>>>>>>> site
>>>>>>>> is similar to the Django admin, CPython's gonna be giving me better
>>>>>>>> performance.
>>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
>>>>>>>> going
>>>>>>>> to be
>>>>>>>> running this in production, but it makes development a little slower
>>>>>>>> and
>>>>>>>> more
>>>>>>>> annoying than it should be.
>>>>>>>>
>>>>>>>> Is there anything to make PyPy more competitive in these kinds of
>>>>>>>> scenarios?
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Tin
>>>>>>>> _______________________________________________
>>>>>>>> pypy-dev mailing list
>>>>>>>> pypy-dev at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> pypy-dev mailing list
>>>>>>> pypy-dev at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> pypy-dev mailing list
>>>>> pypy-dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> "I disapprove of what you say, but I will defend to the death your right
>>>> to
>>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>>>> "The people's good is the highest law." -- Cicero
>>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>>
>>

From tinchester at gmail.com  Mon Feb  9 00:02:11 2015
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Mon, 09 Feb 2015 00:02:11 +0100
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxTz1wswJbR8u60wQ4czS+vdahyjWEFm6Rpd6tkmHZYewQ@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
 <54D7A99C.3060801@gmail.com>
 <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>
 <CAK5idxTz1wswJbR8u60wQ4czS+vdahyjWEFm6Rpd6tkmHZYewQ@mail.gmail.com>
Message-ID: <54D7EAF3.2050301@gmail.com>

Wow, that's what I call customer support! :) Can confirm PyPy is better 
than CPython on the admin now, post warmup.

You're right, it doesn't help my site very much, but that's my fault for 
preparing the wrong benchmark fixture. :)

On the other hand, it doesn't really make sense for you PyPy devs to 
help each and every user with their Django site. Would it be worth it 
for PyPy to add a few benchmarks involving Django (and say, Flask) to 
the regular benchmark suite? I see there's a "django" benchmark but it 
only involves the templating subsystem.

For example, one benchmark might be serving the admin page; another 
might involve a dummy site with a custom app, 3-4 models from sqlite, 
and 1-2 templates? And the same for Flask with Flask-SQLAlchemy. Another 
option would be using a complex, existing free software Django app 
(Review Board, Seafile, ...)

This would probably need to be worked out further (i.e. which versions 
to use, when to bump them, etc) but there's probably a large number of 
users out there using Python for exactly these kinds of things.

In any case, thanks for looking into this, it sure does feel nice when 
upstream devs engage with users. :)

On 08.02.2015 23:09, Maciej Fijalkowski wrote:
> It got merged into django, PyPy (2, didn't test the 3) is now faster
> than cpython on django admin. It likely does not help your cause
> though so you need to come up with a better benchmark :-)
>
> On Sun, Feb 8, 2015 at 8:30 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Hey Tin.
>>
>> We are in the process of two (trivial) pull requests to django that
>> drop the rendering time from 25ms -> 8ms for this case. I'm not a
>> farseer, however I suspect something like this can be done with your
>> site too (depending what it really does).
>>
>> On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
>>> Hello,
>>>
>>> thanks to everyone for their input (especially Maciej).
>>>
>>> Since I've ripped out all my code from the test project, it's not a Python 3
>>> project anymore, so I did try PyPy 2 with it too. It's faster, yes; the last
>>> test looked something like:
>>>
>>> PyPy 2         20.206 [ms] (mean)
>>> PyPy 3         27.588 [ms] (mean)
>>> CPython 3.4    10.865 [ms] (mean)
>>>
>>> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu PPA -
>>> about the same.)
>>>
>>> I don't know, maybe the admin page isn't a good test candidate since it
>>> might do more introspective/reflective/meta stuff. I've tried doing this
>>> same test (10k warmup requests, 100 benchmark requests) on the main page of
>>> the site I'm working on; the results are:
>>>
>>> PyPy 3         15.863 [ms] (mean)
>>> CPython 3.48.668 [ms] (mean)
>>>
>>> and I can be certain there's nothing much going on there. Just some
>>> permission checks, grabbing some stuff from the database, checking some
>>> query parameters and a template render.
>>>
>>> I suppose if folks have CPU intensive stuff in their view functions (nested
>>> for-loops etc) the optimizations PyPy can do to that far outweigh the
>>> non-optimized Django parts; my sites generally don't though.
>>>
>>>
>>> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
>>>> then it's used in the wrong ways in say django admin, look at
>>>> invocations to lazy there (this is how it showed up in my profile)
>>>>
>>>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>>>>> FWIW, I've definitely seen and worked on Django sites that got major
>>>>> improvements out of PyPy -- both the templates and ORM are both sped up
>>>>> substantially by it. I think we should try to fix issues as we see them,
>>>>> before writing it off.
>>>>>
>>>>> FWIW: lazy does not create a new class per call of a function, it's only
>>>>> one
>>>>> class per decorated function.
>>>>>
>>>>> Alex
>>>>>
>>>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com>
>>>>> wrote:
>>>>>> I don't know :-( not sure if fixing those issues one by one is the way
>>>>>> to go, it's not like *this* particular code is a problem, it's the
>>>>>> culture that breeds those sort of things
>>>>>>
>>>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com> wrote:
>>>>>>> Isn't there anything we can do about it?
>>>>>>> We can open issues at the Django issue tracker if some code slows down
>>>>>>> execution in PyPy.
>>>>>>>
>>>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>>>>>>> Hi Tin
>>>>>>>>
>>>>>>>> I have a bit sad news for you, but essentially from what I looked at
>>>>>>>> profling, the answer seems to be "don't use django, it has not been
>>>>>>>> written with performance in mind". For example here (which features in
>>>>>>>> django admin quite prominently, some stuff calls lazy() at each call):
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>>>>>>>
>>>>>>>> What does it, per call that pypy does not like:
>>>>>>>>
>>>>>>>> * redefines classes
>>>>>>>>
>>>>>>>> * walks the class __mro__ and defines tons of new methods
>>>>>>>>
>>>>>>>> * proxies everything through a not so efficietn mechanisms
>>>>>>>>
>>>>>>>> Those things quite defy the purpose of the JIT - you're making tons of
>>>>>>>> very dynamic things that pypy generally assumes is "static enough", if
>>>>>>>> you wanted to do the same thing in C++, you would need to call gcc
>>>>>>>> with each request, that would not be so much fun.
>>>>>>>>
>>>>>>>> If you really care about performance, consider swapping your web
>>>>>>>> framework to something more performance-oriented where JIT compilation
>>>>>>>> can help.
>>>>>>>>
>>>>>>>> As a side note, we (me and my company, not to be confused with PyPy
>>>>>>>> open source project) do offer looking at proprietary code for
>>>>>>>> benchmarking purposes as a commercial service, look at
>>>>>>>> baroquesoftware.com
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> fijal
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi? <tinchester at gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Hello, PyPy folks!
>>>>>>>>>
>>>>>>>>> While trying to speed up one of my Django sites, I noticed a new
>>>>>>>>> version
>>>>>>>>> of
>>>>>>>>> PyPy
>>>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
>>>>>>>>> (since
>>>>>>>>> this
>>>>>>>>> is
>>>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
>>>>>>>>>
>>>>>>>>> However, as far as I can see, whatever I try PyPy is consistently
>>>>>>>>> slower
>>>>>>>>> than
>>>>>>>>> CPython for this.
>>>>>>>>>
>>>>>>>>> Since this is a proprietary site, I've basically ripped out all the
>>>>>>>>> code
>>>>>>>>> except
>>>>>>>>> my settings.py and my requirements; and am benchmarking the Django
>>>>>>>>> admin
>>>>>>>>> index.
>>>>>>>>> The results are about the same.
>>>>>>>>>
>>>>>>>>> I've set up a small repo that can be used to reproduce the
>>>>>>>>> environment:
>>>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's additional
>>>>>>>>> info in
>>>>>>>>> the README there.
>>>>>>>>>
>>>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3
>>>>>>>>> is
>>>>>>>>> the
>>>>>>>>> system Python, 3.4. PyPy has been downloaded from the official site
>>>>>>>>> and
>>>>>>>>> unzipped.
>>>>>>>>>
>>>>>>>>> So what I basically do is set up an admin session, and use the Django
>>>>>>>>> main
>>>>>>>>> admin
>>>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look at the
>>>>>>>>> mean
>>>>>>>>> request time.
>>>>>>>>>
>>>>>>>>> Some results:
>>>>>>>>>
>>>>>>>>> Django's runserver, DEBUG mode:
>>>>>>>>>
>>>>>>>>> PyPy3            485.389 [ms]
>>>>>>>>> CPython 3.4      105.777 [ms]
>>>>>>>>>
>>>>>>>>> Django's runserver, no debug:
>>>>>>>>>
>>>>>>>>> PyPy3             44.661 [ms]
>>>>>>>>> CPython 3.4       18.697 [ms]
>>>>>>>>>
>>>>>>>>> Gunicorn, 1 worker, no debug:
>>>>>>>>>
>>>>>>>>> PyPy3             28.615 [ms]
>>>>>>>>> CPython 3.4       13.532 [ms]
>>>>>>>>>
>>>>>>>>> I don't exactly claim to be an expert on benchmarking, but assuming
>>>>>>>>> my
>>>>>>>>> site
>>>>>>>>> is similar to the Django admin, CPython's gonna be giving me better
>>>>>>>>> performance.
>>>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
>>>>>>>>> going
>>>>>>>>> to be
>>>>>>>>> running this in production, but it makes development a little slower
>>>>>>>>> and
>>>>>>>>> more
>>>>>>>>> annoying than it should be.
>>>>>>>>>
>>>>>>>>> Is there anything to make PyPy more competitive in these kinds of
>>>>>>>>> scenarios?
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>> Tin
>>>>>>>>> _______________________________________________
>>>>>>>>> pypy-dev mailing list
>>>>>>>>> pypy-dev at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>>> _______________________________________________
>>>>>>>> pypy-dev mailing list
>>>>>>>> pypy-dev at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> pypy-dev mailing list
>>>>>> pypy-dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> "I disapprove of what you say, but I will defend to the death your right
>>>>> to
>>>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>>>>> "The people's good is the highest law." -- Cicero
>>>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>>>


From fijall at gmail.com  Mon Feb  9 07:41:21 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 9 Feb 2015 08:41:21 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <54D7EAF3.2050301@gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
 <54D7A99C.3060801@gmail.com>
 <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>
 <CAK5idxTz1wswJbR8u60wQ4czS+vdahyjWEFm6Rpd6tkmHZYewQ@mail.gmail.com>
 <54D7EAF3.2050301@gmail.com>
Message-ID: <CAK5idxRN2YPfviHB=SeeHNKRzJ9iBZzGc20URhBcVKQjEg=frw@mail.gmail.com>

The answer is - yes maybe, but there is no such thing as "django
site". Most of the time the bottlenecks are somewhere else (as you can
see, the admin does not help you) and is often something stupid (e.g.
like here people defining classes at runtime for no good reason) that
has to be looked on a case to case basis. Ideally we would not care
and just optimize everything away anyway, but it's difficult and the
sort of Python that PyPy improves is growing but still not covering
everything we would want.

Cheers,
fijal


On Mon, Feb 9, 2015 at 1:02 AM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> Wow, that's what I call customer support! :) Can confirm PyPy is better than
> CPython on the admin now, post warmup.
>
> You're right, it doesn't help my site very much, but that's my fault for
> preparing the wrong benchmark fixture. :)
>
> On the other hand, it doesn't really make sense for you PyPy devs to help
> each and every user with their Django site. Would it be worth it for PyPy to
> add a few benchmarks involving Django (and say, Flask) to the regular
> benchmark suite? I see there's a "django" benchmark but it only involves the
> templating subsystem.
>
> For example, one benchmark might be serving the admin page; another might
> involve a dummy site with a custom app, 3-4 models from sqlite, and 1-2
> templates? And the same for Flask with Flask-SQLAlchemy. Another option
> would be using a complex, existing free software Django app (Review Board,
> Seafile, ...)
>
> This would probably need to be worked out further (i.e. which versions to
> use, when to bump them, etc) but there's probably a large number of users
> out there using Python for exactly these kinds of things.
>
> In any case, thanks for looking into this, it sure does feel nice when
> upstream devs engage with users. :)
>
>
> On 08.02.2015 23:09, Maciej Fijalkowski wrote:
>>
>> It got merged into django, PyPy (2, didn't test the 3) is now faster
>> than cpython on django admin. It likely does not help your cause
>> though so you need to come up with a better benchmark :-)
>>
>> On Sun, Feb 8, 2015 at 8:30 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>>>
>>> Hey Tin.
>>>
>>> We are in the process of two (trivial) pull requests to django that
>>> drop the rendering time from 25ms -> 8ms for this case. I'm not a
>>> farseer, however I suspect something like this can be done with your
>>> site too (depending what it really does).
>>>
>>> On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtkovi? <tinchester at gmail.com>
>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> thanks to everyone for their input (especially Maciej).
>>>>
>>>> Since I've ripped out all my code from the test project, it's not a
>>>> Python 3
>>>> project anymore, so I did try PyPy 2 with it too. It's faster, yes; the
>>>> last
>>>> test looked something like:
>>>>
>>>> PyPy 2         20.206 [ms] (mean)
>>>> PyPy 3         27.588 [ms] (mean)
>>>> CPython 3.4    10.865 [ms] (mean)
>>>>
>>>> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu PPA -
>>>> about the same.)
>>>>
>>>> I don't know, maybe the admin page isn't a good test candidate since it
>>>> might do more introspective/reflective/meta stuff. I've tried doing this
>>>> same test (10k warmup requests, 100 benchmark requests) on the main page
>>>> of
>>>> the site I'm working on; the results are:
>>>>
>>>> PyPy 3         15.863 [ms] (mean)
>>>> CPython 3.48.668 [ms] (mean)
>>>>
>>>> and I can be certain there's nothing much going on there. Just some
>>>> permission checks, grabbing some stuff from the database, checking some
>>>> query parameters and a template render.
>>>>
>>>> I suppose if folks have CPU intensive stuff in their view functions
>>>> (nested
>>>> for-loops etc) the optimizations PyPy can do to that far outweigh the
>>>> non-optimized Django parts; my sites generally don't though.
>>>>
>>>>
>>>> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
>>>>>
>>>>> then it's used in the wrong ways in say django admin, look at
>>>>> invocations to lazy there (this is how it showed up in my profile)
>>>>>
>>>>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> FWIW, I've definitely seen and worked on Django sites that got major
>>>>>> improvements out of PyPy -- both the templates and ORM are both sped
>>>>>> up
>>>>>> substantially by it. I think we should try to fix issues as we see
>>>>>> them,
>>>>>> before writing it off.
>>>>>>
>>>>>> FWIW: lazy does not create a new class per call of a function, it's
>>>>>> only
>>>>>> one
>>>>>> class per decorated function.
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <fijall at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> I don't know :-( not sure if fixing those issues one by one is the
>>>>>>> way
>>>>>>> to go, it's not like *this* particular code is a problem, it's the
>>>>>>> culture that breeds those sort of things
>>>>>>>
>>>>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Isn't there anything we can do about it?
>>>>>>>> We can open issues at the Django issue tracker if some code slows
>>>>>>>> down
>>>>>>>> execution in PyPy.
>>>>>>>>
>>>>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
>>>>>>>>>
>>>>>>>>> Hi Tin
>>>>>>>>>
>>>>>>>>> I have a bit sad news for you, but essentially from what I looked
>>>>>>>>> at
>>>>>>>>> profling, the answer seems to be "don't use django, it has not been
>>>>>>>>> written with performance in mind". For example here (which features
>>>>>>>>> in
>>>>>>>>> django admin quite prominently, some stuff calls lazy() at each
>>>>>>>>> call):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/django/django/blob/master/django/utils/functional.py#L48
>>>>>>>>>
>>>>>>>>> What does it, per call that pypy does not like:
>>>>>>>>>
>>>>>>>>> * redefines classes
>>>>>>>>>
>>>>>>>>> * walks the class __mro__ and defines tons of new methods
>>>>>>>>>
>>>>>>>>> * proxies everything through a not so efficietn mechanisms
>>>>>>>>>
>>>>>>>>> Those things quite defy the purpose of the JIT - you're making tons
>>>>>>>>> of
>>>>>>>>> very dynamic things that pypy generally assumes is "static enough",
>>>>>>>>> if
>>>>>>>>> you wanted to do the same thing in C++, you would need to call gcc
>>>>>>>>> with each request, that would not be so much fun.
>>>>>>>>>
>>>>>>>>> If you really care about performance, consider swapping your web
>>>>>>>>> framework to something more performance-oriented where JIT
>>>>>>>>> compilation
>>>>>>>>> can help.
>>>>>>>>>
>>>>>>>>> As a side note, we (me and my company, not to be confused with PyPy
>>>>>>>>> open source project) do offer looking at proprietary code for
>>>>>>>>> benchmarking purposes as a commercial service, look at
>>>>>>>>> baroquesoftware.com
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> fijal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi?
>>>>>>>>> <tinchester at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello, PyPy folks!
>>>>>>>>>>
>>>>>>>>>> While trying to speed up one of my Django sites, I noticed a new
>>>>>>>>>> version
>>>>>>>>>> of
>>>>>>>>>> PyPy
>>>>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
>>>>>>>>>> (since
>>>>>>>>>> this
>>>>>>>>>> is
>>>>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
>>>>>>>>>>
>>>>>>>>>> However, as far as I can see, whatever I try PyPy is consistently
>>>>>>>>>> slower
>>>>>>>>>> than
>>>>>>>>>> CPython for this.
>>>>>>>>>>
>>>>>>>>>> Since this is a proprietary site, I've basically ripped out all
>>>>>>>>>> the
>>>>>>>>>> code
>>>>>>>>>> except
>>>>>>>>>> my settings.py and my requirements; and am benchmarking the Django
>>>>>>>>>> admin
>>>>>>>>>> index.
>>>>>>>>>> The results are about the same.
>>>>>>>>>>
>>>>>>>>>> I've set up a small repo that can be used to reproduce the
>>>>>>>>>> environment:
>>>>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's
>>>>>>>>>> additional
>>>>>>>>>> info in
>>>>>>>>>> the README there.
>>>>>>>>>>
>>>>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit.
>>>>>>>>>> CPython 3
>>>>>>>>>> is
>>>>>>>>>> the
>>>>>>>>>> system Python, 3.4. PyPy has been downloaded from the official
>>>>>>>>>> site
>>>>>>>>>> and
>>>>>>>>>> unzipped.
>>>>>>>>>>
>>>>>>>>>> So what I basically do is set up an admin session, and use the
>>>>>>>>>> Django
>>>>>>>>>> main
>>>>>>>>>> admin
>>>>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look at
>>>>>>>>>> the
>>>>>>>>>> mean
>>>>>>>>>> request time.
>>>>>>>>>>
>>>>>>>>>> Some results:
>>>>>>>>>>
>>>>>>>>>> Django's runserver, DEBUG mode:
>>>>>>>>>>
>>>>>>>>>> PyPy3            485.389 [ms]
>>>>>>>>>> CPython 3.4      105.777 [ms]
>>>>>>>>>>
>>>>>>>>>> Django's runserver, no debug:
>>>>>>>>>>
>>>>>>>>>> PyPy3             44.661 [ms]
>>>>>>>>>> CPython 3.4       18.697 [ms]
>>>>>>>>>>
>>>>>>>>>> Gunicorn, 1 worker, no debug:
>>>>>>>>>>
>>>>>>>>>> PyPy3             28.615 [ms]
>>>>>>>>>> CPython 3.4       13.532 [ms]
>>>>>>>>>>
>>>>>>>>>> I don't exactly claim to be an expert on benchmarking, but
>>>>>>>>>> assuming
>>>>>>>>>> my
>>>>>>>>>> site
>>>>>>>>>> is similar to the Django admin, CPython's gonna be giving me
>>>>>>>>>> better
>>>>>>>>>> performance.
>>>>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
>>>>>>>>>> going
>>>>>>>>>> to be
>>>>>>>>>> running this in production, but it makes development a little
>>>>>>>>>> slower
>>>>>>>>>> and
>>>>>>>>>> more
>>>>>>>>>> annoying than it should be.
>>>>>>>>>>
>>>>>>>>>> Is there anything to make PyPy more competitive in these kinds of
>>>>>>>>>> scenarios?
>>>>>>>>>>
>>>>>>>>>> Kind regards,
>>>>>>>>>> Tin
>>>>>>>>>> _______________________________________________
>>>>>>>>>> pypy-dev mailing list
>>>>>>>>>> pypy-dev at python.org
>>>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> pypy-dev mailing list
>>>>>>>>> pypy-dev at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> pypy-dev mailing list
>>>>>>> pypy-dev at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> "I disapprove of what you say, but I will defend to the death your
>>>>>> right
>>>>>> to
>>>>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>>>>>> "The people's good is the highest law." -- Cicero
>>>>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>>>>
>>>>
>

From omer.drow at gmail.com  Mon Feb  9 09:05:36 2015
From: omer.drow at gmail.com (Omer Katz)
Date: Mon, 9 Feb 2015 10:05:36 +0200
Subject: [pypy-dev] Playing with PyPy and Django
In-Reply-To: <CAK5idxRN2YPfviHB=SeeHNKRzJ9iBZzGc20URhBcVKQjEg=frw@mail.gmail.com>
References: <54D54A58.9070201@gmail.com>
 <CAK5idxR70L=Ypp4xSZaD4Wozj5zWCtgcjaoL_diXTP1EFyaQPQ@mail.gmail.com>
 <CAOZCFcB4BaHfEdynJ=Lc4oSJtkds7GvyaA3WWXSaO69jr4vPaw@mail.gmail.com>
 <CAK5idxRpYocsHn7vaVXkWvarkfVYTWx_LSf3+PvuSEw5sV5Qrw@mail.gmail.com>
 <CAFRnB2UgPA=Or+NEBnS=17rG39BREVs_AX=pWuhrj5mHq1tebw@mail.gmail.com>
 <CAK5idxTq7VKRcDj-0nmquieCafyijr-t=Lmg3VHgW2Rh8smeMw@mail.gmail.com>
 <54D7A99C.3060801@gmail.com>
 <CAK5idxQxnHT=_Zs42QgsnpewcA3U3SEaKNET6--RniBfgJaPKg@mail.gmail.com>
 <CAK5idxTz1wswJbR8u60wQ4czS+vdahyjWEFm6Rpd6tkmHZYewQ@mail.gmail.com>
 <54D7EAF3.2050301@gmail.com>
 <CAK5idxRN2YPfviHB=SeeHNKRzJ9iBZzGc20URhBcVKQjEg=frw@mail.gmail.com>
Message-ID: <CAOZCFcDToM9QnuLwY8x1v6+KMoO0iBkB8+ir-5WYkR2BYSC+eg@mail.gmail.com>

I have some Django benchmarks that I was too lazy to contribute yet.
I will do so soon.

2015-02-09 8:41 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:

> The answer is - yes maybe, but there is no such thing as "django
> site". Most of the time the bottlenecks are somewhere else (as you can
> see, the admin does not help you) and is often something stupid (e.g.
> like here people defining classes at runtime for no good reason) that
> has to be looked on a case to case basis. Ideally we would not care
> and just optimize everything away anyway, but it's difficult and the
> sort of Python that PyPy improves is growing but still not covering
> everything we would want.
>
> Cheers,
> fijal
>
>
> On Mon, Feb 9, 2015 at 1:02 AM, Tin Tvrtkovi? <tinchester at gmail.com>
> wrote:
> > Wow, that's what I call customer support! :) Can confirm PyPy is better
> than
> > CPython on the admin now, post warmup.
> >
> > You're right, it doesn't help my site very much, but that's my fault for
> > preparing the wrong benchmark fixture. :)
> >
> > On the other hand, it doesn't really make sense for you PyPy devs to help
> > each and every user with their Django site. Would it be worth it for
> PyPy to
> > add a few benchmarks involving Django (and say, Flask) to the regular
> > benchmark suite? I see there's a "django" benchmark but it only involves
> the
> > templating subsystem.
> >
> > For example, one benchmark might be serving the admin page; another might
> > involve a dummy site with a custom app, 3-4 models from sqlite, and 1-2
> > templates? And the same for Flask with Flask-SQLAlchemy. Another option
> > would be using a complex, existing free software Django app (Review
> Board,
> > Seafile, ...)
> >
> > This would probably need to be worked out further (i.e. which versions to
> > use, when to bump them, etc) but there's probably a large number of users
> > out there using Python for exactly these kinds of things.
> >
> > In any case, thanks for looking into this, it sure does feel nice when
> > upstream devs engage with users. :)
> >
> >
> > On 08.02.2015 23:09, Maciej Fijalkowski wrote:
> >>
> >> It got merged into django, PyPy (2, didn't test the 3) is now faster
> >> than cpython on django admin. It likely does not help your cause
> >> though so you need to come up with a better benchmark :-)
> >>
> >> On Sun, Feb 8, 2015 at 8:30 PM, Maciej Fijalkowski <fijall at gmail.com>
> >> wrote:
> >>>
> >>> Hey Tin.
> >>>
> >>> We are in the process of two (trivial) pull requests to django that
> >>> drop the rendering time from 25ms -> 8ms for this case. I'm not a
> >>> farseer, however I suspect something like this can be done with your
> >>> site too (depending what it really does).
> >>>
> >>> On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtkovi? <tinchester at gmail.com>
> >>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> thanks to everyone for their input (especially Maciej).
> >>>>
> >>>> Since I've ripped out all my code from the test project, it's not a
> >>>> Python 3
> >>>> project anymore, so I did try PyPy 2 with it too. It's faster, yes;
> the
> >>>> last
> >>>> test looked something like:
> >>>>
> >>>> PyPy 2         20.206 [ms] (mean)
> >>>> PyPy 3         27.588 [ms] (mean)
> >>>> CPython 3.4    10.865 [ms] (mean)
> >>>>
> >>>> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu
> PPA -
> >>>> about the same.)
> >>>>
> >>>> I don't know, maybe the admin page isn't a good test candidate since
> it
> >>>> might do more introspective/reflective/meta stuff. I've tried doing
> this
> >>>> same test (10k warmup requests, 100 benchmark requests) on the main
> page
> >>>> of
> >>>> the site I'm working on; the results are:
> >>>>
> >>>> PyPy 3         15.863 [ms] (mean)
> >>>> CPython 3.48.668 [ms] (mean)
> >>>>
> >>>> and I can be certain there's nothing much going on there. Just some
> >>>> permission checks, grabbing some stuff from the database, checking
> some
> >>>> query parameters and a template render.
> >>>>
> >>>> I suppose if folks have CPU intensive stuff in their view functions
> >>>> (nested
> >>>> for-loops etc) the optimizations PyPy can do to that far outweigh the
> >>>> non-optimized Django parts; my sites generally don't though.
> >>>>
> >>>>
> >>>> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
> >>>>>
> >>>>> then it's used in the wrong ways in say django admin, look at
> >>>>> invocations to lazy there (this is how it showed up in my profile)
> >>>>>
> >>>>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> FWIW, I've definitely seen and worked on Django sites that got major
> >>>>>> improvements out of PyPy -- both the templates and ORM are both sped
> >>>>>> up
> >>>>>> substantially by it. I think we should try to fix issues as we see
> >>>>>> them,
> >>>>>> before writing it off.
> >>>>>>
> >>>>>> FWIW: lazy does not create a new class per call of a function, it's
> >>>>>> only
> >>>>>> one
> >>>>>> class per decorated function.
> >>>>>>
> >>>>>> Alex
> >>>>>>
> >>>>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <
> fijall at gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I don't know :-( not sure if fixing those issues one by one is the
> >>>>>>> way
> >>>>>>> to go, it's not like *this* particular code is a problem, it's the
> >>>>>>> culture that breeds those sort of things
> >>>>>>>
> >>>>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Isn't there anything we can do about it?
> >>>>>>>> We can open issues at the Django issue tracker if some code slows
> >>>>>>>> down
> >>>>>>>> execution in PyPy.
> >>>>>>>>
> >>>>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
> >>>>>>>>>
> >>>>>>>>> Hi Tin
> >>>>>>>>>
> >>>>>>>>> I have a bit sad news for you, but essentially from what I looked
> >>>>>>>>> at
> >>>>>>>>> profling, the answer seems to be "don't use django, it has not
> been
> >>>>>>>>> written with performance in mind". For example here (which
> features
> >>>>>>>>> in
> >>>>>>>>> django admin quite prominently, some stuff calls lazy() at each
> >>>>>>>>> call):
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> https://github.com/django/django/blob/master/django/utils/functional.py#L48
> >>>>>>>>>
> >>>>>>>>> What does it, per call that pypy does not like:
> >>>>>>>>>
> >>>>>>>>> * redefines classes
> >>>>>>>>>
> >>>>>>>>> * walks the class __mro__ and defines tons of new methods
> >>>>>>>>>
> >>>>>>>>> * proxies everything through a not so efficietn mechanisms
> >>>>>>>>>
> >>>>>>>>> Those things quite defy the purpose of the JIT - you're making
> tons
> >>>>>>>>> of
> >>>>>>>>> very dynamic things that pypy generally assumes is "static
> enough",
> >>>>>>>>> if
> >>>>>>>>> you wanted to do the same thing in C++, you would need to call
> gcc
> >>>>>>>>> with each request, that would not be so much fun.
> >>>>>>>>>
> >>>>>>>>> If you really care about performance, consider swapping your web
> >>>>>>>>> framework to something more performance-oriented where JIT
> >>>>>>>>> compilation
> >>>>>>>>> can help.
> >>>>>>>>>
> >>>>>>>>> As a side note, we (me and my company, not to be confused with
> PyPy
> >>>>>>>>> open source project) do offer looking at proprietary code for
> >>>>>>>>> benchmarking purposes as a commercial service, look at
> >>>>>>>>> baroquesoftware.com
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> fijal
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtkovi?
> >>>>>>>>> <tinchester at gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello, PyPy folks!
> >>>>>>>>>>
> >>>>>>>>>> While trying to speed up one of my Django sites, I noticed a new
> >>>>>>>>>> version
> >>>>>>>>>> of
> >>>>>>>>>> PyPy
> >>>>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
> >>>>>>>>>> (since
> >>>>>>>>>> this
> >>>>>>>>>> is
> >>>>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
> >>>>>>>>>>
> >>>>>>>>>> However, as far as I can see, whatever I try PyPy is
> consistently
> >>>>>>>>>> slower
> >>>>>>>>>> than
> >>>>>>>>>> CPython for this.
> >>>>>>>>>>
> >>>>>>>>>> Since this is a proprietary site, I've basically ripped out all
> >>>>>>>>>> the
> >>>>>>>>>> code
> >>>>>>>>>> except
> >>>>>>>>>> my settings.py and my requirements; and am benchmarking the
> Django
> >>>>>>>>>> admin
> >>>>>>>>>> index.
> >>>>>>>>>> The results are about the same.
> >>>>>>>>>>
> >>>>>>>>>> I've set up a small repo that can be used to reproduce the
> >>>>>>>>>> environment:
> >>>>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's
> >>>>>>>>>> additional
> >>>>>>>>>> info in
> >>>>>>>>>> the README there.
> >>>>>>>>>>
> >>>>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit.
> >>>>>>>>>> CPython 3
> >>>>>>>>>> is
> >>>>>>>>>> the
> >>>>>>>>>> system Python, 3.4. PyPy has been downloaded from the official
> >>>>>>>>>> site
> >>>>>>>>>> and
> >>>>>>>>>> unzipped.
> >>>>>>>>>>
> >>>>>>>>>> So what I basically do is set up an admin session, and use the
> >>>>>>>>>> Django
> >>>>>>>>>> main
> >>>>>>>>>> admin
> >>>>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look
> at
> >>>>>>>>>> the
> >>>>>>>>>> mean
> >>>>>>>>>> request time.
> >>>>>>>>>>
> >>>>>>>>>> Some results:
> >>>>>>>>>>
> >>>>>>>>>> Django's runserver, DEBUG mode:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3            485.389 [ms]
> >>>>>>>>>> CPython 3.4      105.777 [ms]
> >>>>>>>>>>
> >>>>>>>>>> Django's runserver, no debug:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3             44.661 [ms]
> >>>>>>>>>> CPython 3.4       18.697 [ms]
> >>>>>>>>>>
> >>>>>>>>>> Gunicorn, 1 worker, no debug:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3             28.615 [ms]
> >>>>>>>>>> CPython 3.4       13.532 [ms]
> >>>>>>>>>>
> >>>>>>>>>> I don't exactly claim to be an expert on benchmarking, but
> >>>>>>>>>> assuming
> >>>>>>>>>> my
> >>>>>>>>>> site
> >>>>>>>>>> is similar to the Django admin, CPython's gonna be giving me
> >>>>>>>>>> better
> >>>>>>>>>> performance.
> >>>>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
> >>>>>>>>>> going
> >>>>>>>>>> to be
> >>>>>>>>>> running this in production, but it makes development a little
> >>>>>>>>>> slower
> >>>>>>>>>> and
> >>>>>>>>>> more
> >>>>>>>>>> annoying than it should be.
> >>>>>>>>>>
> >>>>>>>>>> Is there anything to make PyPy more competitive in these kinds
> of
> >>>>>>>>>> scenarios?
> >>>>>>>>>>
> >>>>>>>>>> Kind regards,
> >>>>>>>>>> Tin
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> pypy-dev mailing list
> >>>>>>>>>> pypy-dev at python.org
> >>>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> pypy-dev mailing list
> >>>>>>>>> pypy-dev at python.org
> >>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>>>
> >>>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> pypy-dev mailing list
> >>>>>>> pypy-dev at python.org
> >>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> "I disapprove of what you say, but I will defend to the death your
> >>>>>> right
> >>>>>> to
> >>>>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> >>>>>> "The people's good is the highest law." -- Cicero
> >>>>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
> >>>>
> >>>>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150209/3d803c16/attachment-0001.html>

From arigo at tunes.org  Mon Feb  9 17:16:48 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 9 Feb 2015 17:16:48 +0100
Subject: [pypy-dev] Leysin Sprint location
Message-ID: <CAMSv6X35AvhqatCzoFb+FwzuOxwmBitxKJgQkyPKaST-FB0rpA@mail.gmail.com>

Hi all,

If you're coming back to Leysin for the sprint at the end of February:
warning, the main entrance to the two chalets Ermina was moved from
the upper chalet to the lower chalet.  In case you need it, here's a
map showing how to go to the lower chalet (red marker) when you're
standing in front of the upper chalet:

https://www.google.ch/maps/dir/46.3488973,7.0201613/46.3486126,7.0210687/@46.3482881,7.0203089,17z/data=!4m2!4m1!3e2?hl=en


A bient?t,

Armin.

From fijall at gmail.com  Wed Feb 11 06:53:50 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Feb 2015 07:53:50 +0200
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
References: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
Message-ID: <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>

Hi

I like the blog post, but it requires some cleanups and improvements,
do you plan on being on IRC today?

Cheers,
fijal

On Fri, Feb 6, 2015 at 5:01 AM, ??? <hrc706 at gmail.com> wrote:
> Hi, everyone,
> I tried to wrote down the experience I have done recently about my project of Pyrlang, but I?m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog.
> Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance.
>
> Ruochen Huang
>
>
>

From fijall at gmail.com  Wed Feb 11 10:46:02 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Feb 2015 11:46:02 +0200
Subject: [pypy-dev] numpy installation
Message-ID: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>

Hey

There is STILL no clear installation path for PyPy + numpy, with no
instructions anywhere, can you please do something about it?

Ideally pip install numpy-pypy should work and we should be done (this
+ info on pypy.org will do)

I asked about this months ago, if noone picks it up, I'll deal with
it, but this is by far the biggest problem for PyPy+numpy users (they
can't install it).

Cheers,
fijal

From kostia.lopuhin at gmail.com  Wed Feb 11 12:37:46 2015
From: kostia.lopuhin at gmail.com (=?UTF-8?B?0JrQvtGB0YLRjyDQm9C+0L/Rg9GF0LjQvQ==?=)
Date: Wed, 11 Feb 2015 15:37:46 +0400
Subject: [pypy-dev] numpy installation
In-Reply-To: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
Message-ID: <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>

Outsider perspective - I think these
http://pypy.org/download.html#installing-numpy instructions are pretty
visible and used to work at least several months ago?

2015-02-11 12:46 GMT+03:00 Maciej Fijalkowski <fijall at gmail.com>:
> Hey
>
> There is STILL no clear installation path for PyPy + numpy, with no
> instructions anywhere, can you please do something about it?
>
> Ideally pip install numpy-pypy should work and we should be done (this
> + info on pypy.org will do)
>
> I asked about this months ago, if noone picks it up, I'll deal with
> it, but this is by far the biggest problem for PyPy+numpy users (they
> can't install it).
>
> Cheers,
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From yyc1992 at gmail.com  Wed Feb 11 14:09:14 2015
From: yyc1992 at gmail.com (Yichao Yu)
Date: Wed, 11 Feb 2015 05:09:14 -0800 (PST)
Subject: [pypy-dev] numpy installation
In-Reply-To: <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
Message-ID: <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>

On 2015?2?11? ??? 15:37:46, ????? ??????? <Maciej Fijalkowski 
<fijall at gmail.com>> wrote:
> Outsider perspective - I think these
> http://pypy.org/download.html#installing-numpy instructions are pretty
> visible and used to work at least several months ago?

Also from here[1]
Which includes the fix you have to do for system installation.

[1] https://bitbucket.org/pypy/numpy

> 
> 2015-02-11 12:46 GMT+03:00 Maciej Fijalkowski <fijall at gmail.com>:
> > Hey
> > 
> > There is STILL no clear installation path for PyPy + numpy, with no
> > instructions anywhere, can you please do something about it?
> > 
> > Ideally pip install numpy-pypy should work and we should be done (this
> > + info on pypy.org will do)
> > 
> > I asked about this months ago, if noone picks it up, I'll deal with
> > it, but this is by far the biggest problem for PyPy+numpy users (they
> > can't install it).
> > 
> > Cheers,
> > fijal
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150211/4cbd0e17/attachment.sig>

From fijall at gmail.com  Wed Feb 11 14:37:59 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Feb 2015 15:37:59 +0200
Subject: [pypy-dev] numpy installation
In-Reply-To: <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
 <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
Message-ID: <CAK5idxTZVUCkzUbeOSS42MHzHBFQ1sOSOiUwEuLR8G6r9Od1hg@mail.gmail.com>

Then indeed maybe some people are not managing as I've seen in reddit
comments. Maybe pip install numpy-pypy is a good idea nonetheless?

On Wed, Feb 11, 2015 at 3:09 PM, Yichao Yu <yyc1992 at gmail.com> wrote:
> On 2015?2?11? ??? 15:37:46, ????? ??????? <Maciej Fijalkowski
> <fijall at gmail.com>> wrote:
>> Outsider perspective - I think these
>> http://pypy.org/download.html#installing-numpy instructions are pretty
>> visible and used to work at least several months ago?
>
> Also from here[1]
> Which includes the fix you have to do for system installation.
>
> [1] https://bitbucket.org/pypy/numpy
>
>>
>> 2015-02-11 12:46 GMT+03:00 Maciej Fijalkowski <fijall at gmail.com>:
>> > Hey
>> >
>> > There is STILL no clear installation path for PyPy + numpy, with no
>> > instructions anywhere, can you please do something about it?
>> >
>> > Ideally pip install numpy-pypy should work and we should be done (this
>> > + info on pypy.org will do)
>> >
>> > I asked about this months ago, if noone picks it up, I'll deal with
>> > it, but this is by far the biggest problem for PyPy+numpy users (they
>> > can't install it).
>> >
>> > Cheers,
>> > fijal
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From romain.py at gmail.com  Wed Feb 11 14:53:45 2015
From: romain.py at gmail.com (Romain Guillebert)
Date: Wed, 11 Feb 2015 14:53:45 +0100
Subject: [pypy-dev] numpy installation
In-Reply-To: <CAK5idxTZVUCkzUbeOSS42MHzHBFQ1sOSOiUwEuLR8G6r9Od1hg@mail.gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
 <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
 <CAK5idxTZVUCkzUbeOSS42MHzHBFQ1sOSOiUwEuLR8G6r9Od1hg@mail.gmail.com>
Message-ID: <20150211135345.GA4878@Plop>

Hi Maciej

I think the current situation is "ok", but having a PyPI package is
certainly better.

Cheers
Romain

On 02/11, Maciej Fijalkowski wrote:
> Then indeed maybe some people are not managing as I've seen in reddit
> comments. Maybe pip install numpy-pypy is a good idea nonetheless?
> 
> On Wed, Feb 11, 2015 at 3:09 PM, Yichao Yu <yyc1992 at gmail.com> wrote:
> > On 2015?2?11? ??? 15:37:46, ????? ??????? <Maciej Fijalkowski
> > <fijall at gmail.com>> wrote:
> >> Outsider perspective - I think these
> >> http://pypy.org/download.html#installing-numpy instructions are pretty
> >> visible and used to work at least several months ago?
> >
> > Also from here[1]
> > Which includes the fix you have to do for system installation.
> >
> > [1] https://bitbucket.org/pypy/numpy
> >
> >>
> >> 2015-02-11 12:46 GMT+03:00 Maciej Fijalkowski <fijall at gmail.com>:
> >> > Hey
> >> >
> >> > There is STILL no clear installation path for PyPy + numpy, with no
> >> > instructions anywhere, can you please do something about it?
> >> >
> >> > Ideally pip install numpy-pypy should work and we should be done (this
> >> > + info on pypy.org will do)
> >> >
> >> > I asked about this months ago, if noone picks it up, I'll deal with
> >> > it, but this is by far the biggest problem for PyPy+numpy users (they
> >> > can't install it).
> >> >
> >> > Cheers,
> >> > fijal
> >> > _______________________________________________
> >> > pypy-dev mailing list
> >> > pypy-dev at python.org
> >> > https://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pypy-dev
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From arigo at tunes.org  Wed Feb 11 16:56:30 2015
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 11 Feb 2015 16:56:30 +0100
Subject: [pypy-dev] numpy installation
In-Reply-To: <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
 <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
Message-ID: <CAMSv6X1qMYhPMi2VU7OtJ6CBo4HMtrK-GskX7xCvMhUfR_Eygg@mail.gmail.com>

Hi Yichao,

On 11 February 2015 at 14:09, Yichao Yu <yyc1992 at gmail.com> wrote:
> Also from here[1]
> Which includes the fix you have to do for system installation.
>
> [1] https://bitbucket.org/pypy/numpy

Ah, thanks.  I'll add the mention of the fix to the
pypy.org/install.html page and clarify what you get by clicking on the
link "our own repository".


Armin

From matti.picus at gmail.com  Wed Feb 11 22:09:02 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 11 Feb 2015 23:09:02 +0200
Subject: [pypy-dev] numpy installation
In-Reply-To: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
Message-ID: <54DBC4EE.8020609@gmail.com>


On 11/02/15 11:46, Maciej Fijalkowski wrote:
> Hey
>
> There is STILL no clear installation path for PyPy + numpy, with no
> instructions anywhere, can you please do something about it?
>
> Ideally pip install numpy-pypy should work and we should be done (this
> + info on pypy.org will do)
>
> I asked about this months ago, if noone picks it up, I'll deal with
> it, but this is by far the biggest problem for PyPy+numpy users (they
> can't install it).
>
The current installation procedure ensures the users get the latest 
version, with no lag for an extra packaging step after pushing to bitbucket.

IMO we should not concentrate on packaging right now. We need to make 
pypy/numpy more mature, i.e. pass much more of the numpy test suite, 
before we make it too easy to install. Are we really ready for a deluge 
of users who file issues with incomplete or erroneous numpy compliance?

Anyway, the packaging problem can be solved by someone well versed in 
python + cffi but with no rpython knowlege. The limited time of the 
current numpy contributors, again IMO, would be better spent on 
finishing features, passing more numpy tests, or creating compelling 
reasons to use our version, such as lazy evaluation, automatic 
vectorization using specialized hardware, etc.

Matti

From yyc1992 at gmail.com  Wed Feb 11 22:16:03 2015
From: yyc1992 at gmail.com (Yichao Yu)
Date: Wed, 11 Feb 2015 16:16:03 -0500
Subject: [pypy-dev] numpy installation
In-Reply-To: <CAMSv6X1qMYhPMi2VU7OtJ6CBo4HMtrK-GskX7xCvMhUfR_Eygg@mail.gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <CAAvCB=Tm=v11cqG-DyqiQYQ0kqwX7ML5CFr3SFX6WFMh4vFAUA@mail.gmail.com>
 <9371775.JzjtHAhK6Z@yyc2.yyc-arch.org>
 <CAMSv6X1qMYhPMi2VU7OtJ6CBo4HMtrK-GskX7xCvMhUfR_Eygg@mail.gmail.com>
Message-ID: <CAMvDr+TGqgCk9+F3G9WvN=QSxfy5VY_OdbU6SukNv7JaKd01bA@mail.gmail.com>

> Ah, thanks.  I'll add the mention of the fix to the
> pypy.org/install.html page and clarify what you get by clicking on the
> link "our own repository".

and apparently you mean pypy.org/download.html

>
>
> Armin

Yichao Yu

From hrc706 at gmail.com  Thu Feb 12 06:23:53 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Thu, 12 Feb 2015 14:23:53 +0900
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>
References: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
 <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>
Message-ID: <E158D4F5-FABF-493D-AE0E-2B34B820E211@gmail.com>

Hi, Fijal,
I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE. 
I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge. :-)
The docx format seems has something wrong so I also attached the pdf one.

Cheers, 
Huang
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 546479 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150212/27b06a5f/attachment-0001.docx>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experience in Pyrlang with RPython.pdf
Type: application/pdf
Size: 84776 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150212/27b06a5f/attachment-0001.pdf>
-------------- next part --------------

> 2015/02/11 ??2:53?Maciej Fijalkowski <fijall at gmail.com> ?????
> 
> Hi
> 
> I like the blog post, but it requires some cleanups and improvements,
> do you plan on being on IRC today?
> 
> Cheers,
> fijal
> 
> On Fri, Feb 6, 2015 at 5:01 AM, ??? <hrc706 at gmail.com> wrote:
>> Hi, everyone,
>> I tried to wrote down the experience I have done recently about my project of Pyrlang, but I?m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog.
>> Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance.
>> 
>> Ruochen Huang
>> 
>> 
>> 


From anto.cuni at gmail.com  Thu Feb 12 15:53:32 2015
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 12 Feb 2015 15:53:32 +0100
Subject: [pypy-dev] numpy installation
In-Reply-To: <54DBC4EE.8020609@gmail.com>
References: <CAK5idxToMQTAs0yaqDH4K8FU4bo1Ua0z0c+fGZRTovDOkB-yGg@mail.gmail.com>
 <54DBC4EE.8020609@gmail.com>
Message-ID: <CAHPg78xTmK=83NMpqL-fFFx1_8WGfDykL94Onbz9_79SODk0pw@mail.gmail.com>

On Wed, Feb 11, 2015 at 10:09 PM, Matti Picus <matti.picus at gmail.com> wrote:


> The current installation procedure ensures the users get the latest
> version, with no lag for an extra packaging step after pushing to bitbucket.
>

?this has also the drawback that it can break at any time for pypy
releases. For example, the current HEAD of numpypy does not work with pypy
2.4 (and it didn't work for a while, even before we released pypy 2.5)
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150212/a973af75/attachment.html>

From fijall at gmail.com  Fri Feb 13 07:56:04 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 13 Feb 2015 08:56:04 +0200
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <E158D4F5-FABF-493D-AE0E-2B34B820E211@gmail.com>
References: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
 <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>
 <E158D4F5-FABF-493D-AE0E-2B34B820E211@gmail.com>
Message-ID: <CAK5idxRP7XRPnzj=HZmSQvbKTa8zNaEU1FmYLFc88TaruXL=Tw@mail.gmail.com>

I like the blog post. It's full of technical details and might need
some editing, but otherwise fine. I'm also happy to just send it as it
is (but please add numbers and explanations on axis of graphs :-)

On Thu, Feb 12, 2015 at 7:23 AM, ??? <hrc706 at gmail.com> wrote:
> Hi, Fijal,
> I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE.
> I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge. :-)
> The docx format seems has something wrong so I also attached the pdf one.
>
> Cheers,
> Huang
>
>
>> 2015/02/11 ??2:53?Maciej Fijalkowski <fijall at gmail.com> ?????
>>
>> Hi
>>
>> I like the blog post, but it requires some cleanups and improvements,
>> do you plan on being on IRC today?
>>
>> Cheers,
>> fijal
>>
>> On Fri, Feb 6, 2015 at 5:01 AM, ??? <hrc706 at gmail.com> wrote:
>>> Hi, everyone,
>>> I tried to wrote down the experience I have done recently about my project of Pyrlang, but I?m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog.
>>> Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance.
>>>
>>> Ruochen Huang
>>>
>>>
>>>
>
>

From matti.picus at gmail.com  Fri Feb 13 08:52:47 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 13 Feb 2015 09:52:47 +0200
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <CAK5idxRP7XRPnzj=HZmSQvbKTa8zNaEU1FmYLFc88TaruXL=Tw@mail.gmail.com>
References: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
 <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>
 <E158D4F5-FABF-493D-AE0E-2B34B820E211@gmail.com>
 <CAK5idxRP7XRPnzj=HZmSQvbKTa8zNaEU1FmYLFc88TaruXL=Tw@mail.gmail.com>
Message-ID: <54DDAD4F.9000806@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150213/dd73113d/attachment.html>

From hrc706 at gmail.com  Fri Feb 13 08:53:45 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Fri, 13 Feb 2015 16:53:45 +0900
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <54DDAD4F.9000806@gmail.com>
References: <300E4C76-16C7-4AAC-AC25-3AAF1E1D57F1@gmail.com>
 <CAK5idxTYNZZeyZysMOOqUF=-=ifeQwfnfA+TFOGVK_Qv+FPMJg@mail.gmail.com>
 <E158D4F5-FABF-493D-AE0E-2B34B820E211@gmail.com>
 <CAK5idxRP7XRPnzj=HZmSQvbKTa8zNaEU1FmYLFc88TaruXL=Tw@mail.gmail.com>
 <54DDAD4F.9000806@gmail.com>
Message-ID: <6220C744-E834-448A-9906-BC4FA20A0BC8@gmail.com>

got it, thank you in advance :-)

> 2015/02/13 ??4:52?Matti Picus <matti.picus at gmail.com> ?????
> 
> Give me till Sunday night to see if I can get a quick edit to this (text, not graphics).
> Matti
> 
> On 13/02/15 08:56, Maciej Fijalkowski wrote:
>> I like the blog post. It's full of technical details and might need
>> some editing, but otherwise fine. I'm also happy to just send it as it
>> is (but please add numbers and explanations on axis of graphs :-)
>> 
>> On Thu, Feb 12, 2015 at 7:23 AM, ??? <hrc706 at gmail.com> <mailto:hrc706 at gmail.com> wrote:
>>> Hi, Fijal,
>>> I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE.
>>> I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge. :-)
>>> The docx format seems has something wrong so I also attached the pdf one.
>>> 
>>> Cheers,
>>> Huang
>>> 
>>> 
>>>> 2015/02/11 ??2:53?Maciej Fijalkowski <fijall at gmail.com> <mailto:fijall at gmail.com> ?????
>>>> 
>>>> Hi
>>>> 
>>>> I like the blog post, but it requires some cleanups and improvements,
>>>> do you plan on being on IRC today?
>>>> 
>>>> Cheers,
>>>> fijal
>>>> 
>>>> On Fri, Feb 6, 2015 at 5:01 AM, ??? <hrc706 at gmail.com> <mailto:hrc706 at gmail.com> wrote:
>>>>> Hi, everyone,
>>>>> I tried to wrote down the experience I have done recently about my project of Pyrlang, but I?m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog.
>>>>> Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance.
>>>>> 
>>>>> Ruochen Huang
>>>>> 
>>>>> 
>>>>> 
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org <mailto:pypy-dev at python.org>
>> https://mail.python.org/mailman/listinfo/pypy-dev <https://mail.python.org/mailman/listinfo/pypy-dev>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150213/651c21af/attachment-0001.html>

From matti.picus at gmail.com  Fri Feb 13 12:36:21 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 13 Feb 2015 13:36:21 +0200
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <CAE_NQXPzj4VTHJs9XUWedMfT1MQUzwda0Zpcp71PXYBWqVUu3w@mail.gmail.com>
References: <CAE_NQXPzj4VTHJs9XUWedMfT1MQUzwda0Zpcp71PXYBWqVUu3w@mail.gmail.com>
Message-ID: <54DDE1B5.3070904@gmail.com>

Here's a quick edit job, again as both PDF and docx (with revision marks).
Graphs are still missing axis labels and some kind of captioning
Matti

  On Thu, Feb 12, 2015 at 7:23 AM, ???<hrc706 at gmail.com>  wrote:

> I like the blog post. It's full of technical details and might need
> some editing, but otherwise fine. I'm also happy to just send it as it
> is (but please add numbers and explanations on axis of graphs
>
> On Thu, Feb 12, 2015 at 7:23 AM, ???<hrc706 at gmail.com>  wrote:
> Hi, Fijal,
> I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE.
> I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge.
> The docx format seems has something wrong so I also attached the pdf one.
>
> Cheers,
> Huang

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 675010 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150213/5850d80f/attachment-0001.docx>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython.pdf
Type: application/pdf
Size: 280845 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150213/5850d80f/attachment-0001.pdf>

From matti.picus at gmail.com  Fri Feb 13 14:46:41 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 13 Feb 2015 15:46:41 +0200
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
Message-ID: <54DE0041.5030108@gmail.com>

I am close to releasing a blog post about numpy.linalg

https://gist.github.com/mattip/25680e68fe7e2856fe0c

Note the benchmark section, this is not a typo. Here's how I got to the 
conclusion that cpyext is faster than python+cffi.

I installed numpy from the pypy/numpy repo on a nightly after 2.5.0

pip install git+https://bitbucket.org/pypy/numpy.git

I then benchmarked the two pypy methods of calling numpy.linalg.inv() 
via this script which simply calls linalg.inv() for different sized ndarray.

https://gist.github.com/mattip/a7b22dc237fb09a46758

$pypy inverse.py 10
UserWarning: npy_clear_floatstatus, npy_set_floatstatus_invalid not found
   warn('npy_clear_floatstatus, npy_set_floatstatus_invalid not found')
after   999999, for n= 10, time=20.64,     20.64 msec/1000 loops

and

$USE_CPYEXT=yes pypy inverse.py 10
after   999999, for n= 10, time= 8.52,      8.52 msec/1000 loops

Any thoughts? Comments on the blog post?
Matti


From phyo.arkarlwin at gmail.com  Fri Feb 13 15:33:00 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Fri, 13 Feb 2015 21:03:00 +0630
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
In-Reply-To: <54DE0041.5030108@gmail.com>
References: <54DE0041.5030108@gmail.com>
Message-ID: <CA+HjJrjZvx62FNxWrR6fO1_XC=sozTxva_oWc5-HUBQgPzksHA@mail.gmail.com>

Very interesting.
If that really working it will be game changer.
Can it build Scipy, scikit-learn, matplotlib now?
Pandas already working i think.

On Fri, Feb 13, 2015 at 8:16 PM, Matti Picus <matti.picus at gmail.com> wrote:
> I am close to releasing a blog post about numpy.linalg
>
> https://gist.github.com/mattip/25680e68fe7e2856fe0c
>
> Note the benchmark section, this is not a typo. Here's how I got to the
> conclusion that cpyext is faster than python+cffi.
>
> I installed numpy from the pypy/numpy repo on a nightly after 2.5.0
>
> pip install git+https://bitbucket.org/pypy/numpy.git
>
> I then benchmarked the two pypy methods of calling numpy.linalg.inv() via
> this script which simply calls linalg.inv() for different sized ndarray.
>
> https://gist.github.com/mattip/a7b22dc237fb09a46758
>
> $pypy inverse.py 10
> UserWarning: npy_clear_floatstatus, npy_set_floatstatus_invalid not found
>   warn('npy_clear_floatstatus, npy_set_floatstatus_invalid not found')
> after   999999, for n= 10, time=20.64,     20.64 msec/1000 loops
>
> and
>
> $USE_CPYEXT=yes pypy inverse.py 10
> after   999999, for n= 10, time= 8.52,      8.52 msec/1000 loops
>
> Any thoughts? Comments on the blog post?
> Matti
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From arigo at tunes.org  Sat Feb 14 10:40:56 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 14 Feb 2015 10:40:56 +0100
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
In-Reply-To: <54DE0041.5030108@gmail.com>
References: <54DE0041.5030108@gmail.com>
Message-ID: <CAMSv6X0xJ6HMvEdYpsoJ1zZPob5i4YO58y98d+nOGHMrhjPPKA@mail.gmail.com>

Hi Matti,

On 13 February 2015 at 14:46, Matti Picus <matti.picus at gmail.com> wrote:
> I am close to releasing a blog post about numpy.linalg

A comment: saying "In order to test it out, download PyPy 2.5.0 or
later, and install the pypy version of numpy" in the conclusion is, in
my opinion, both too late and not precise enough.  Please put in the
introduction two lines that say clearly something like:

    In order to install the version of numpy this blog post is talking
about, run::

         pypy -m pip install git+https://bitbucket.org/pypy/numpy.git

> $pypy inverse.py 10
> $USE_CPYEXT=yes pypy inverse.py 10
> Any thoughts?

That's probably because the cffi version does more copying of the
data, and more slowly.  It's not really about cffi versus cpyext.
Here you are comparing a version that, with cpyext, calls just "inv()"
once (with the overhead of cpyext once), with a version that uses cffi
to do in Python a whole layer of manipulations that was done in C (the
@TYPE at _inv() function in _umath_linalg.c.src and its callees).  I
think it is not surprizing that the former is faster.

If you want to find a design that makes sense performance-wise, I
would try to look for one where this C layer of manipulations is still
C code.  You would still having a family of functions called
"@TYPE at _inv()" that are called only once --- you can select the
correct "@TYPE at _inv()" to call from Python code.

In other words, working C code shouldn't be rewritten in Python.  Only
the parts of C code with strong dependencies on the CPython C API
needs to be.

Note that I just had a quick look, I may be missing some issues.


A bient?t,

Armin.

From techtonik at gmail.com  Sat Feb 14 09:48:22 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 14 Feb 2015 11:48:22 +0300
Subject: [pypy-dev] mix-and-match approach to implementation decisions
Message-ID: <CAPkN8xLipMmZ1t+YuyGjKiP3wYctpV6CxY4u6OMOFDu5uHeK9g@mail.gmail.com>

Hi,

Reading http://rpython.readthedocs.org/en/latest/ to see if I can
understand the text without a formal CS education. What does
"mix-and-match approach to implementation decisions" mean?

-- 
anatoly t.

From william.leslie.ttg at gmail.com  Sat Feb 14 11:58:40 2015
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Sat, 14 Feb 2015 21:58:40 +1100
Subject: [pypy-dev] mix-and-match approach to implementation decisions
In-Reply-To: <CAPkN8xLipMmZ1t+YuyGjKiP3wYctpV6CxY4u6OMOFDu5uHeK9g@mail.gmail.com>
References: <CAPkN8xLipMmZ1t+YuyGjKiP3wYctpV6CxY4u6OMOFDu5uHeK9g@mail.gmail.com>
Message-ID: <CAHgd1hFmFUFbZcT6BsEMGkSMrknb_JSgo4viNJq3X3OO5+ufnw@mail.gmail.com>

On 14 February 2015 at 19:48, anatoly techtonik <techtonik at gmail.com> wrote:

> Hi,
>
> Reading http://rpython.readthedocs.org/en/latest/ to see if I can
> understand the text without a formal CS education. What does
> "mix-and-match approach to implementation decisions" mean?
>

It means that making one decision - like which GC to use - does not have a
great impact on other decisions, like whether you can have a JIT.

Pypy itself is a great representation of this philosophy, where there are a
large number of 'strategies' that optimise built-in objects for different
use cases, which are enabled or disabled at translation time.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered under
copyright law.  You absolutely MAY reproduce any part of it in accordance
with the copyright law of the nation you are reading this in.  Any attempt
to DENY YOU THOSE RIGHTS would be illegal without prior contractual
agreement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150214/b7b6bf46/attachment.html>

From william.leslie.ttg at gmail.com  Sat Feb 14 12:11:28 2015
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Sat, 14 Feb 2015 22:11:28 +1100
Subject: [pypy-dev] mix-and-match approach to implementation decisions
In-Reply-To: <CAHgd1hFmFUFbZcT6BsEMGkSMrknb_JSgo4viNJq3X3OO5+ufnw@mail.gmail.com>
References: <CAPkN8xLipMmZ1t+YuyGjKiP3wYctpV6CxY4u6OMOFDu5uHeK9g@mail.gmail.com>
 <CAHgd1hFmFUFbZcT6BsEMGkSMrknb_JSgo4viNJq3X3OO5+ufnw@mail.gmail.com>
Message-ID: <CAHgd1hFD9pP+ECHAZUtuBaH_tU3f4fHhL3PY57MRt2X0G-RXUA@mail.gmail.com>

?I guess it's reasonable to ask how true this is of rpython.  What options
are there, and how do they exclude / depend on one another?  In the l*o*p
problem, it now ignores all p but one, and rpython doesn't concern itself
with l.?  Maybe it's the gc * o problem now?

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered under
copyright law.  You absolutely MAY reproduce any part of it in accordance
with the copyright law of the nation you are reading this in.  Any attempt
to DENY YOU THOSE RIGHTS would be illegal without prior contractual
agreement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150214/de3bb0a5/attachment.html>

From hrc706 at gmail.com  Sat Feb 14 15:30:54 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Sat, 14 Feb 2015 23:30:54 +0900
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <54DDE1B5.3070904@gmail.com>
References: <CAE_NQXPzj4VTHJs9XUWedMfT1MQUzwda0Zpcp71PXYBWqVUu3w@mail.gmail.com>
 <54DDE1B5.3070904@gmail.com>
Message-ID: <32962656-D951-4A99-BA04-0176783D8BF8@gmail.com>

Thank you very much for the modification!
I added the axis labels then.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython v2.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 618833 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150214/073b2e83/attachment-0001.docx>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython v2.pdf
Type: application/pdf
Size: 115483 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150214/073b2e83/attachment-0001.pdf>
-------------- next part --------------

> 2015/02/13 20:36?Matti Picus <matti.picus at gmail.com> ?????
> 
> Here's a quick edit job, again as both PDF and docx (with revision marks).
> Graphs are still missing axis labels and some kind of captioning
> Matti
> 
> On Thu, Feb 12, 2015 at 7:23 AM, ???<hrc706 at gmail.com>  wrote:
> 
>> I like the blog post. It's full of technical details and might need
>> some editing, but otherwise fine. I'm also happy to just send it as it
>> is (but please add numbers and explanations on axis of graphs
>> 
>> On Thu, Feb 12, 2015 at 7:23 AM, ???<hrc706 at gmail.com>  wrote:
>> Hi, Fijal,
>> I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE.
>> I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge.
>> The docx format seems has something wrong so I also attached the pdf one.
>> 
>> Cheers,
>> Huang
> 
> <Some Experiments in Pyrlang with RPython.docx><Some Experiments in Pyrlang with RPython.pdf>


From matti.picus at gmail.com  Sat Feb 14 17:44:55 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 14 Feb 2015 18:44:55 +0200
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
In-Reply-To: <CAMSv6X0xJ6HMvEdYpsoJ1zZPob5i4YO58y98d+nOGHMrhjPPKA@mail.gmail.com>
References: <54DE0041.5030108@gmail.com>
 <CAMSv6X0xJ6HMvEdYpsoJ1zZPob5i4YO58y98d+nOGHMrhjPPKA@mail.gmail.com>
Message-ID: <54DF7B87.7060606@gmail.com>


On 14/02/15 11:40, Armin Rigo wrote:
> Hi Matti,
>
> On 13 February 2015 at 14:46, Matti Picus <matti.picus at gmail.com> wrote:
>> I am close to releasing a blog post about numpy.linalg
> A comment: saying "In order to test it out, download PyPy 2.5.0 or
> later, and install the pypy version of numpy" in the conclusion is, in
> my opinion, both too late and not precise enough.  Please put in the
> introduction two lines that say clearly something like:
>
>      In order to install the version of numpy this blog post is talking
> about, run::
>
>           pypy -m pip install git+https://bitbucket.org/pypy/numpy.git

Done, thanks.

>> $pypy inverse.py 10
>> $USE_CPYEXT=yes pypy inverse.py 10
>> Any thoughts?
> That's probably because the cffi version does more copying of the
> data, and more slowly.  It's not really about cffi versus cpyext.
> Here you are comparing a version that, with cpyext, calls just "inv()"
> once (with the overhead of cpyext once), with a version that uses cffi
> to do in Python a whole layer of manipulations that was done in C (the
> @TYPE at _inv() function in _umath_linalg.c.src and its callees).  I
> think it is not surprizing that the former is faster.
>
> If you want to find a design that makes sense performance-wise, I
> would try to look for one where this C layer of manipulations is still
> C code.  You would still having a family of functions called
> "@TYPE at _inv()" that are called only once --- you can select the
> correct "@TYPE at _inv()" to call from Python code.
>
> In other words, working C code shouldn't be rewritten in Python.  Only
> the parts of C code with strong dependencies on the CPython C API
> needs to be.
>
> Note that I just had a quick look, I may be missing some issues.
>
>
> A bient?t,
>
> Armin.
That makes sense, thanks. I will see if I can easily reuse the c code as 
a shared object rather than a module, and what effect that has on timing.
Matti

From arigo at tunes.org  Sun Feb 15 09:49:11 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 15 Feb 2015 09:49:11 +0100
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
In-Reply-To: <54DF7B87.7060606@gmail.com>
References: <54DE0041.5030108@gmail.com>
 <CAMSv6X0xJ6HMvEdYpsoJ1zZPob5i4YO58y98d+nOGHMrhjPPKA@mail.gmail.com>
 <54DF7B87.7060606@gmail.com>
Message-ID: <CAMSv6X1z2m-ik1YEpoMVeceUDWRJoKXrDKLfJyTdMWfL=EH9OA@mail.gmail.com>

Hi Matti,

On 14 February 2015 at 17:44, Matti Picus <matti.picus at gmail.com> wrote:
> That makes sense, thanks. I will see if I can easily reuse the c code as a
> shared object rather than a module, and what effect that has on timing.

Note that it's what ffi.verify() is for: you add any amount of custom
C code there.


Armin

From techtonik at gmail.com  Sun Feb 15 08:25:49 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 15 Feb 2015 10:25:49 +0300
Subject: [pypy-dev] mix-and-match approach to implementation decisions
In-Reply-To: <CAHgd1hFmFUFbZcT6BsEMGkSMrknb_JSgo4viNJq3X3OO5+ufnw@mail.gmail.com>
References: <CAPkN8xLipMmZ1t+YuyGjKiP3wYctpV6CxY4u6OMOFDu5uHeK9g@mail.gmail.com>
 <CAHgd1hFmFUFbZcT6BsEMGkSMrknb_JSgo4viNJq3X3OO5+ufnw@mail.gmail.com>
Message-ID: <CAPkN8x+Km7xUPsr4XnrW7chs=On9AvbuQhxVkHW471Ja8fxnvw@mail.gmail.com>

On Sat, Feb 14, 2015 at 1:58 PM, William ML Leslie
<william.leslie.ttg at gmail.com> wrote:
> On 14 February 2015 at 19:48, anatoly techtonik <techtonik at gmail.com> wrote:
>>
>> Reading http://rpython.readthedocs.org/en/latest/ to see if I can
>> understand the text without a formal CS education. What does
>> "mix-and-match approach to implementation decisions" mean?
>
>
> It means that making one decision - like which GC to use - does not have a
> great impact on other decisions, like whether you can have a JIT.

Does not have a great impact or is completely orthogonal? Is it possible to
get a graph of dependencies between various features?

> Pypy itself is a great representation of this philosophy, where there are a
> large number of 'strategies' that optimise built-in objects for different
> use cases, which are enabled or disabled at translation time.

Just to make it clear:
  - translation - creating pypy from rpython definition
  - interpretation - executing python code with pypy
  - JIT compilation - optimization of executed code at run-time

So, if the strategies are chosen at translation time, which use cases are
covered there? I thought that optimizations should be chosen for executed
python code, but at the time pypy is being translated, there is no python
code executed yet.

From matti.picus at gmail.com  Sun Feb 15 21:10:47 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 15 Feb 2015 22:10:47 +0200
Subject: [pypy-dev] could it be that cpyext is faster than python + cffi?
In-Reply-To: <CAMSv6X1z2m-ik1YEpoMVeceUDWRJoKXrDKLfJyTdMWfL=EH9OA@mail.gmail.com>
References: <54DE0041.5030108@gmail.com>
 <CAMSv6X0xJ6HMvEdYpsoJ1zZPob5i4YO58y98d+nOGHMrhjPPKA@mail.gmail.com>
 <54DF7B87.7060606@gmail.com>
 <CAMSv6X1z2m-ik1YEpoMVeceUDWRJoKXrDKLfJyTdMWfL=EH9OA@mail.gmail.com>
Message-ID: <54E0FD47.5040205@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150215/9ae249df/attachment.html>

From rohangoel0296 at gmail.com  Sun Feb 15 21:23:27 2015
From: rohangoel0296 at gmail.com (Rohan Goel)
Date: Mon, 16 Feb 2015 01:53:27 +0530
Subject: [pypy-dev] GSoC 2015
Message-ID: <CAH4cMHwnc2qW1M8nt7Q0O-rK7np6GVfojrpEiEKGOvVSJAD=WQ@mail.gmail.com>

Dear Developers,

I am Rohan Goel ,  Computer Science undergraduate at BITS Pilani , India
and am interested in working for your organization in GSoC 2015. As I am a
beginner in open source coding , it would be great help if you guide me
where to start from. My primary language of interest is Python.



*Thanking You,*



*Rohan Goel*

*Under Graduate, B.E. (Hons) Computer Science Birla Institute of Technology
and Science, Pilani*
*K.K Birla Goa Campus*
*+91 7722047225 **| rohangoel0296 at gmail.com <rohangoel0296 at gmail.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150216/e683d41b/attachment.html>

From walderchristian at gmail.com  Sun Feb 15 21:36:30 2015
From: walderchristian at gmail.com (Christian Walder)
Date: Sun, 15 Feb 2015 20:36:30 +0000
Subject: [pypy-dev] Fwd: pypy memory test
In-Reply-To: <CAOwaaSh1cYX_Lv2PN2KE8Yii3uWuR0frzgUs4jscjXqtmxGJ9g@mail.gmail.com>
References: <CAOwaaSh1cYX_Lv2PN2KE8Yii3uWuR0frzgUs4jscjXqtmxGJ9g@mail.gmail.com>
Message-ID: <CAOwaaShh0kuGWU9ft=QNc24zB0YzDj+7JZM_oTcjGXBsv7NngQ@mail.gmail.com>

Hi there,

I seem to be running into a problem with freeing memory.

Under OS X all is well, but under Linux there seems to be a problem in the
case of linked data structures.

I have attached a python program which demonstrates the problem. The output
of the script for both linux and OS X are below.

Thanks very much in advance, and for the amazing work on PyPy.

OS X:



2.7.8 (857f34cd4254, Oct 14 2014, 22:01:17)

[PyPy 2.5.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 6.0
(clang-600.0.51)]

Darwin 14.1.0 ('', '', '')



initial 20 MB



0 558 MB

0 30 MB



1 558 MB

1 31 MB



2 558 MB

2 32 MB





Linux:



2.7.8 (dfffd5d7cc7e, Feb 08 2015, 14:59:48)

[PyPy 2.5.0 with GCC 4.8.2]

Linux 3.13.0-44-generic ('debian', 'jessie/sid', '')



initial 82 MB



0 626 MB

0 626 MB



1 627 MB

1 627 MB



2 627 MB

2 627 MB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150215/d786e5a1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memtest.py
Type: text/x-python-script
Size: 683 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150215/d786e5a1/attachment.bin>

From cfbolz at gmx.de  Mon Feb 16 12:36:40 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 16 Feb 2015 12:36:40 +0100
Subject: [pypy-dev] report of some experience in Pyrlang and RPython
In-Reply-To: <32962656-D951-4A99-BA04-0176783D8BF8@gmail.com>
References: <CAE_NQXPzj4VTHJs9XUWedMfT1MQUzwda0Zpcp71PXYBWqVUu3w@mail.gmail.com>
 <54DDE1B5.3070904@gmail.com> <32962656-D951-4A99-BA04-0176783D8BF8@gmail.com>
Message-ID: <54E1D648.7080801@gmx.de>

On 14/02/15 15:30, ??? wrote:
> Thank you very much for the modification!
> I added the axis labels then.

Hello Huang,

Looks good! Would you be up to adding the repository link to Pyrlang
into the blog post?

Also, could you please convert this to an HTML file? Otherwise we can't
actually post it on the blog.

Thanks,

Carl Friedrich

From arigo at tunes.org  Tue Feb 17 13:58:51 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 17 Feb 2015 13:58:51 +0100
Subject: [pypy-dev] GSoC 2015
In-Reply-To: <CAH4cMHwnc2qW1M8nt7Q0O-rK7np6GVfojrpEiEKGOvVSJAD=WQ@mail.gmail.com>
References: <CAH4cMHwnc2qW1M8nt7Q0O-rK7np6GVfojrpEiEKGOvVSJAD=WQ@mail.gmail.com>
Message-ID: <CAMSv6X2G61cPCXKuHSffwuWJgiGm-uNhNhJzOp6vbQsNj1q3cw@mail.gmail.com>

Hi Rohan,

On 15 February 2015 at 21:23, Rohan Goel <rohangoel0296 at gmail.com> wrote:

> I am Rohan Goel ,  Computer Science undergraduate at BITS Pilani , India
> and am interested in working for your organization in GSoC 2015. As I am a
> beginner in open source coding , it would be great help if you guide me
> where to start from. My primary language of interest is Python.
>

Welcome!

In the past few years, we have found GSoC to be tricky to handle.  We're
likely to have a high bar for student acceptance this year.

The first step is to get involved with the community, which means showing
up on irc (#pypy on irc.freenode.net).  We can give you pointers to
possible ideas there, or you can find your own, for example by looking at
http://bugs.pypy.org/.

This is what we say in general to people that want to contribute to PyPy.
We can move on to discuss GSoC only after we have seen results from this
step.


A bient?t,

Armin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150217/29e16db9/attachment.html>

From arigo at tunes.org  Tue Feb 17 14:10:15 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 17 Feb 2015 14:10:15 +0100
Subject: [pypy-dev] Fwd: pypy memory test
In-Reply-To: <CAOwaaShh0kuGWU9ft=QNc24zB0YzDj+7JZM_oTcjGXBsv7NngQ@mail.gmail.com>
References: <CAOwaaSh1cYX_Lv2PN2KE8Yii3uWuR0frzgUs4jscjXqtmxGJ9g@mail.gmail.com>
 <CAOwaaShh0kuGWU9ft=QNc24zB0YzDj+7JZM_oTcjGXBsv7NngQ@mail.gmail.com>
Message-ID: <CAMSv6X10902oWH6wUsD4LFcXU9uHfnGLVizesdKemXAX45GTFQ@mail.gmail.com>

Hi Christian,

On 15 February 2015 at 21:36, Christian Walder
<walderchristian at gmail.com> wrote:
> I seem to be running into a problem with freeing memory.

The problem you have, as I understand it, is that the OS-visible size
of the process goes up but not down.  This is expected.  I don't
really know why OS/X versus Linux makes any difference.

Someone should collect a few links to existing answers into a FAQ entry...


A bient?t,

Armin

From hrc706 at gmail.com  Tue Feb 17 09:48:32 2015
From: hrc706 at gmail.com (=?utf-8?B?6buE6Iul5bCY?=)
Date: Tue, 17 Feb 2015 17:48:32 +0900
Subject: [pypy-dev] Pyrlang Post
Message-ID: <A5087323-FC2C-4806-8044-B7ADB095EAE4@gmail.com>

A non-text attachment was scrubbed...
Name: Some Experiments in Pyrlang with RPython.zip
Type: application/zip
Size: 686807 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150217/770a7247/attachment-0001.zip>

From ahmedabdalrahman0 at gmail.com  Tue Feb 17 17:16:09 2015
From: ahmedabdalrahman0 at gmail.com (Ahmed Abdalrahman)
Date: Tue, 17 Feb 2015 18:16:09 +0200
Subject: [pypy-dev] contribute in the project
Message-ID: <CADCmufq3t-2jYcCgMTcNu7VneM+aQdTLcdaqK77ykZe-jQb=qQ@mail.gmail.com>

hello ,
my name is Ahmed ,

                              i want to join u and i am interested in open
source what ever your will join google summer of code or not
i want to coding , trying to learn and help you if i can please tell me,

my experience ,
                      below average in python
thanks for reading
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150217/02e215c9/attachment.html>

From arigo at tunes.org  Tue Feb 17 23:01:36 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 17 Feb 2015 23:01:36 +0100
Subject: [pypy-dev] contribute in the project
In-Reply-To: <CADCmufq3t-2jYcCgMTcNu7VneM+aQdTLcdaqK77ykZe-jQb=qQ@mail.gmail.com>
References: <CADCmufq3t-2jYcCgMTcNu7VneM+aQdTLcdaqK77ykZe-jQb=qQ@mail.gmail.com>
Message-ID: <CAMSv6X1P85CZ1nSmjNPcDn6C7ad1AqNCmW4CBQdrjE+D7L0xQg@mail.gmail.com>

Hi!

On 17 February 2015 at 17:16, Ahmed Abdalrahman
<ahmedabdalrahman0 at gmail.com> wrote:
> my name is Ahmed ,
>                               i want to join u and i am interested in open
> source what ever your will join google summer of code or not
> i want to coding , trying to learn and help you if i can please tell me,

Welcome!

Please see my answer to another potential GSoC student, sent a few
hours before you joined :-)  It is here:

https://mail.python.org/pipermail/pypy-dev/2015-February/013130.html


Armin

From walderchristian at gmail.com  Tue Feb 17 23:47:53 2015
From: walderchristian at gmail.com (Christian Walder)
Date: Tue, 17 Feb 2015 22:47:53 +0000
Subject: [pypy-dev] Fwd: pypy memory test
In-Reply-To: <CAMSv6X10902oWH6wUsD4LFcXU9uHfnGLVizesdKemXAX45GTFQ@mail.gmail.com>
References: <CAOwaaSh1cYX_Lv2PN2KE8Yii3uWuR0frzgUs4jscjXqtmxGJ9g@mail.gmail.com>
 <CAOwaaShh0kuGWU9ft=QNc24zB0YzDj+7JZM_oTcjGXBsv7NngQ@mail.gmail.com>
 <CAMSv6X10902oWH6wUsD4LFcXU9uHfnGLVizesdKemXAX45GTFQ@mail.gmail.com>
Message-ID: <CAOwaaSitRSPkQVyxmrZT0sLzwPfP7iNXVSJaEHGwtGxGVi2xgQ@mail.gmail.com>

Thanks for the quick reply.

You're right. When I tried to boil it down to a minimal example, I
incorrectly attributed my original problem to linked structures. This is
because if I create a large tuple instead of a linked structure in my
example, then the OS visible size does go down and up in both Linux and
OS/X.

Thanks again,
Christian

On 17 February 2015 at 13:10, Armin Rigo <arigo at tunes.org> wrote:

> Hi Christian,
>
> On 15 February 2015 at 21:36, Christian Walder
> <walderchristian at gmail.com> wrote:
> > I seem to be running into a problem with freeing memory.
>
> The problem you have, as I understand it, is that the OS-visible size
> of the process goes up but not down.  This is expected.  I don't
> really know why OS/X versus Linux makes any difference.
>
> Someone should collect a few links to existing answers into a FAQ entry...
>
>
> A bient?t,
>
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150217/a9d45c67/attachment.html>

From fijall at gmail.com  Sat Feb 21 17:04:46 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 21 Feb 2015 18:04:46 +0200
Subject: [pypy-dev] freebsd 7 and 8 buildslave
Message-ID: <CAK5idxR65Do+AAttsWnXn+AzsgYF3QgE6Qb=oW=t1HBhhO_-ng@mail.gmail.com>

there is a freebsd 8 buildslave called "ananke" which is offline, who
is the admin?

Also freebsd 7 fails to compile for an idiotic reason, should we take
this one down?
http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64/builds/290/steps/translate/logs/stdio

From matti.picus at gmail.com  Sat Feb 21 22:42:41 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 21 Feb 2015 23:42:41 +0200
Subject: [pypy-dev] freebsd 7 and 8 buildslave
In-Reply-To: <CAK5idxR65Do+AAttsWnXn+AzsgYF3QgE6Qb=oW=t1HBhhO_-ng@mail.gmail.com>
References: <CAK5idxR65Do+AAttsWnXn+AzsgYF3QgE6Qb=oW=t1HBhhO_-ng@mail.gmail.com>
Message-ID: <54E8FBD1.6040804@gmail.com>


On 21/02/2015 6:04 PM, Maciej Fijalkowski wrote:
> there is a freebsd 8 buildslave called "ananke" which is offline, who
> is the admin?
>
> Also freebsd 7 fails to compile for an idiotic reason, should we take
> this one down?
> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64/builds/290/steps/translate/logs/stdio

+1 for removing non-functioning buildslaves with non-responsive owners
Matti


From matti.picus at gmail.com  Sat Feb 21 22:58:54 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 21 Feb 2015 23:58:54 +0200
Subject: [pypy-dev] fdopen(fd, 'w') whe fd is opened 'r'
Message-ID: <54E8FF9E.1060106@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150221/8ec6a879/attachment.html>

From bdkearns at gmail.com  Sat Feb 21 23:44:12 2015
From: bdkearns at gmail.com (Brian Kearns)
Date: Sat, 21 Feb 2015 14:44:12 -0800
Subject: [pypy-dev] fdopen(fd, 'w') whe fd is opened 'r'
In-Reply-To: <54E8FF9E.1060106@gmail.com>
References: <54E8FF9E.1060106@gmail.com>
Message-ID: <CACw2m-ZwXM1oLu+H7uyo1GPtGHmvn_b41VcYmsq80Boo8A2K1g@mail.gmail.com>

What version of CPython did you compare with? This behavior was
added/changed in 2.7.8. CPython has the exact same test and it doesn't seem
to be skipped on Windows. Comparing directly with C fdopen does not make
sense because os.fdopen isn't just a direct call to fdopen (includes
verification of mode, fd, etc).

http://bugs.python.org/issue21191

On Sat, Feb 21, 2015 at 1:58 PM, Matti Picus <matti.picus at gmail.com> wrote:

>  Looking at this test test_fdopen_keeps_fd_open_on_errors which fails on
> win32
>
> http://buildbot.pypy.org/summary/longrepr?testname=AppTestPosix.%28%29.test_fdopen_keeps_fd_open_on_errors&builder=own-win-x86-32&build=442&mod=module.posix.test.test_posix2
>
> It also fails when run with python -A, on cpython. It even fails in C when
> running the code from the MSDN example of fdopen,
> https://msdn.microsoft.com/en-us/library/dye30d82.aspx
> if you replace the mode flag "r" with "w", windows will happily repopen a
> read-only file in write mode.
>
> The MSDN documentation is quiet on the issue of mode conflict between fd
> and a call to fdopen, where posix explicitly will return an error
> http://linux.die.net/man/3/fdopen "The mode of the stream (one of the
> values "r", "r+", "w", "w+", "a", "a+") must be compatible with the mode of
> the file descriptor." I could find any google results for complaints about
> win32's reckless behaviour.
>
> Two courses of action are possible:
> - fix the test to reflect current cpython and pypy behaviour on win32
> - file a bug with cpython and fix our win32 implementation to pass the
> current test
>
> Any thoughts?
> Matti
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150221/33e2e6a0/attachment.html>

From fijall at gmail.com  Sun Feb 22 08:08:41 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 22 Feb 2015 09:08:41 +0200
Subject: [pypy-dev] freebsd 7 and 8 buildslave
In-Reply-To: <54E8FBD1.6040804@gmail.com>
References: <CAK5idxR65Do+AAttsWnXn+AzsgYF3QgE6Qb=oW=t1HBhhO_-ng@mail.gmail.com>
 <54E8FBD1.6040804@gmail.com>
Message-ID: <CAK5idxSsCAVEG8g2ma+aFJmH6RXi4HthnYTqeWTOWfpqq=VmTA@mail.gmail.com>

give them more than 6h though ;-)

On Sat, Feb 21, 2015 at 11:42 PM, Matti Picus <matti.picus at gmail.com> wrote:
>
> On 21/02/2015 6:04 PM, Maciej Fijalkowski wrote:
>>
>> there is a freebsd 8 buildslave called "ananke" which is offline, who
>> is the admin?
>>
>> Also freebsd 7 fails to compile for an idiotic reason, should we take
>> this one down?
>>
>> http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64/builds/290/steps/translate/logs/stdio
>
>
> +1 for removing non-functioning buildslaves with non-responsive owners
> Matti
>

From matti.picus at gmail.com  Sun Feb 22 17:18:02 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 22 Feb 2015 18:18:02 +0200
Subject: [pypy-dev] fdopen(fd, 'w') whe fd is opened 'r'
In-Reply-To: <CACw2m-ZwXM1oLu+H7uyo1GPtGHmvn_b41VcYmsq80Boo8A2K1g@mail.gmail.com>
References: <54E8FF9E.1060106@gmail.com>
 <CACw2m-ZwXM1oLu+H7uyo1GPtGHmvn_b41VcYmsq80Boo8A2K1g@mail.gmail.com>
Message-ID: <54EA013A.9030608@gmail.com>

I tested with win32 2.7.8
The issue addressed in 21191 was that posix.fdopen was not closing fd on 
error. On windows there is no posix module so test_posix.py is skipped 
entirely. While the skipped test was modified in issue 21191 to test 
mode compatibility, posixmodule.c does not actually compare the input fd 
mode (perhaps via a call to fstat(fd, stat)? ) to the requested mode, 
rahter it only verifies that the requested mode itself is a valid mode 
for a call to fdopen.
On windows posix.fdopen is exposed through os.fdopen, which AFAICT is 
only minimally tested outside of test_posix.py.


On 22/02/2015 12:44 AM, Brian Kearns wrote:
> What version of CPython did you compare with? This behavior was 
> added/changed in 2.7.8. CPython has the exact same test and it doesn't 
> seem to be skipped on Windows. Comparing directly with C fdopen does 
> not make sense because os.fdopen isn't just a direct call to fdopen 
> (includes verification of mode, fd, etc).
>
> http://bugs.python.org/issue21191
>
> On Sat, Feb 21, 2015 at 1:58 PM, Matti Picus <matti.picus at gmail.com 
> <mailto:matti.picus at gmail.com>> wrote:
>
>     Looking at this test test_fdopen_keeps_fd_open_on_errors which
>     fails on win32
>     http://buildbot.pypy.org/summary/longrepr?testname=AppTestPosix.%28%29.test_fdopen_keeps_fd_open_on_errors&builder=own-win-x86-32&build=442&mod=module.posix.test.test_posix2
>
>     It also fails when run with python -A, on cpython. It even fails
>     in C when running the code from the MSDN example of fdopen,
>     https://msdn.microsoft.com/en-us/library/dye30d82.aspx
>     if you replace the mode flag "r" with "w", windows will happily
>     repopen a read-only file in write mode.
>
>     The MSDN documentation is quiet on the issue of mode conflict
>     between fd and a call to fdopen, where posix explicitly will
>     return an error
>     http://linux.die.net/man/3/fdopen "The mode of the stream (one of
>     the values "r", "r+", "w", "w+", "a", "a+") must be compatible
>     with the mode of the file descriptor." I could find any google
>     results for complaints about win32's reckless behaviour.
>
>     Two courses of action are possible:
>     - fix the test to reflect current cpython and pypy behaviour on win32
>     - file a bug with cpython and fix our win32 implementation to pass
>     the current test
>
>     Any thoughts?
>     Matti
>
>     _______________________________________________
>     pypy-dev mailing list
>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>
>


From rich at pasra.at  Tue Feb 24 19:18:10 2015
From: rich at pasra.at (Richard Plangger)
Date: Tue, 24 Feb 2015 19:18:10 +0100
Subject: [pypy-dev] Vectorizing numpy traces
Message-ID: <54ECC062.7010209@pasra.at>

hi,

i'm currently trying to find a way to generate simd code for python
loops that operate on numpy arrays.
The IRC is quite busy so I think I'd rather post it as an email...

I have familiarized myself with pypy and read most of the docs and read
the code in the metainterp to understand how the traces are built.

I translated pypy with --withmod-micronumpy enabled (installed numpy
pypy fork in an virtual env) and gave the jitviewer a try.
Here is a sample program that does not really make sense, but I think it
would contain opportunity to generate SIMD instructions.

```
import numpy
def invoke():
    a = numpy.zeros(1024)
    b = numpy.zeros(1024)
    c = numpy.zeros(1024)
    # (2)
    for i in range(0,1024):
        c[i] = a[i] + b[i]
    # (3)
    c = a + b

if __name__ == '__main__':
    i = 0
    # (1)
    while i < 500:
        invoke()
        i += 1
```

Here is the trace of invoke visible in jitviewer (uploaded to
https://postimg.org/image/kbntluw55/full/).

Here are some questions I have that would really help me to get going:

(1) Is there a better way to get loops hot?

(2) I cannot really figure out what all those trace/loop parameters are.
obviously i can guess some but most of them do not really make sense to
me. Am I missing some settings to be able to display more information
for the trace?
    In addition to that I do not really see any chance to generate a
simd loop for (2). Every array access is (arguably) guarded by an array
index overflow and I think to skip that check would be invalid.

(3) There is no trace generated for this instruction? Does this
internally call a c function?

(4) What in our opinion makes sense to vectorize with simd instructions?
Could you provide some sample loops/code (ranging from simple to complex
loops)?

I would really appreciate any help and/or push into the right direction.

Best,
Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150224/42668554/attachment.sig>

From fijall at gmail.com  Tue Feb 24 19:36:07 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 24 Feb 2015 20:36:07 +0200
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <54ECC062.7010209@pasra.at>
References: <54ECC062.7010209@pasra.at>
Message-ID: <CAK5idxRYgQYK_Uo8Voode-gECg9pn_8WJqHALPrn0XM3DNNkfw@mail.gmail.com>

before I answer, any clue why I get:

Google Chrome's connection attempt to postimg.org was rejected. The
website may be down, or your network may not be properly configured.

On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
> hi,
>
> i'm currently trying to find a way to generate simd code for python
> loops that operate on numpy arrays.
> The IRC is quite busy so I think I'd rather post it as an email...
>
> I have familiarized myself with pypy and read most of the docs and read
> the code in the metainterp to understand how the traces are built.
>
> I translated pypy with --withmod-micronumpy enabled (installed numpy
> pypy fork in an virtual env) and gave the jitviewer a try.
> Here is a sample program that does not really make sense, but I think it
> would contain opportunity to generate SIMD instructions.
>
> ```
> import numpy
> def invoke():
>     a = numpy.zeros(1024)
>     b = numpy.zeros(1024)
>     c = numpy.zeros(1024)
>     # (2)
>     for i in range(0,1024):
>         c[i] = a[i] + b[i]
>     # (3)
>     c = a + b
>
> if __name__ == '__main__':
>     i = 0
>     # (1)
>     while i < 500:
>         invoke()
>         i += 1
> ```
>
> Here is the trace of invoke visible in jitviewer (uploaded to
> https://postimg.org/image/kbntluw55/full/).
>
> Here are some questions I have that would really help me to get going:
>
> (1) Is there a better way to get loops hot?
>
> (2) I cannot really figure out what all those trace/loop parameters are.
> obviously i can guess some but most of them do not really make sense to
> me. Am I missing some settings to be able to display more information
> for the trace?
>     In addition to that I do not really see any chance to generate a
> simd loop for (2). Every array access is (arguably) guarded by an array
> index overflow and I think to skip that check would be invalid.
>
> (3) There is no trace generated for this instruction? Does this
> internally call a c function?
>
> (4) What in our opinion makes sense to vectorize with simd instructions?
> Could you provide some sample loops/code (ranging from simple to complex
> loops)?
>
> I would really appreciate any help and/or push into the right direction.
>
> Best,
> Richard
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From lac at openend.se  Tue Feb 24 19:37:16 2015
From: lac at openend.se (Laura Creighton)
Date: Tue, 24 Feb 2015 19:37:16 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: Message from Richard Plangger <rich@pasra.at>
 of "Tue, 24 Feb 2015 19:18:10 +0100." <54ECC062.7010209@pasra.at>
References: <54ECC062.7010209@pasra.at>
Message-ID: <201502241837.t1OIbGaK031173@fido.openend.se>

The PyPy core developers are at a sprint in Leysin right now,
and they happen to be out for dinner right now.

So you, alas, are likely to need to wait.

Sorry for impatience ....

Laura

From rich at pasra.at  Tue Feb 24 19:39:15 2015
From: rich at pasra.at (Richard Plangger)
Date: Tue, 24 Feb 2015 19:39:15 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <CAK5idxRYgQYK_Uo8Voode-gECg9pn_8WJqHALPrn0XM3DNNkfw@mail.gmail.com>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxRYgQYK_Uo8Voode-gECg9pn_8WJqHALPrn0XM3DNNkfw@mail.gmail.com>
Message-ID: <54ECC553.3060303@pasra.at>

Sorry, I put https instead of http:

http://postimg.org/image/kbntluw55/full/

On 02/24/2015 07:36 PM, Maciej Fijalkowski wrote:
> before I answer, any clue why I get:
> 
> Google Chrome's connection attempt to postimg.org was rejected. The
> website may be down, or your network may not be properly configured.
> 
> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>> hi,
>>
>> i'm currently trying to find a way to generate simd code for python
>> loops that operate on numpy arrays.
>> The IRC is quite busy so I think I'd rather post it as an email...
>>
>> I have familiarized myself with pypy and read most of the docs and read
>> the code in the metainterp to understand how the traces are built.
>>
>> I translated pypy with --withmod-micronumpy enabled (installed numpy
>> pypy fork in an virtual env) and gave the jitviewer a try.
>> Here is a sample program that does not really make sense, but I think it
>> would contain opportunity to generate SIMD instructions.
>>
>> ```
>> import numpy
>> def invoke():
>>     a = numpy.zeros(1024)
>>     b = numpy.zeros(1024)
>>     c = numpy.zeros(1024)
>>     # (2)
>>     for i in range(0,1024):
>>         c[i] = a[i] + b[i]
>>     # (3)
>>     c = a + b
>>
>> if __name__ == '__main__':
>>     i = 0
>>     # (1)
>>     while i < 500:
>>         invoke()
>>         i += 1
>> ```
>>
>> Here is the trace of invoke visible in jitviewer (uploaded to
>> https://postimg.org/image/kbntluw55/full/).
>>
>> Here are some questions I have that would really help me to get going:
>>
>> (1) Is there a better way to get loops hot?
>>
>> (2) I cannot really figure out what all those trace/loop parameters are.
>> obviously i can guess some but most of them do not really make sense to
>> me. Am I missing some settings to be able to display more information
>> for the trace?
>>     In addition to that I do not really see any chance to generate a
>> simd loop for (2). Every array access is (arguably) guarded by an array
>> index overflow and I think to skip that check would be invalid.
>>
>> (3) There is no trace generated for this instruction? Does this
>> internally call a c function?
>>
>> (4) What in our opinion makes sense to vectorize with simd instructions?
>> Could you provide some sample loops/code (ranging from simple to complex
>> loops)?
>>
>> I would really appreciate any help and/or push into the right direction.
>>
>> Best,
>> Richard
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150224/efb61834/attachment.sig>

From fijall at gmail.com  Tue Feb 24 23:17:33 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Feb 2015 00:17:33 +0200
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <54ECC062.7010209@pasra.at>
References: <54ECC062.7010209@pasra.at>
Message-ID: <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>

Hi Richard.


I will respond inline

On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
> hi,
>
> i'm currently trying to find a way to generate simd code for python
> loops that operate on numpy arrays.
> The IRC is quite busy so I think I'd rather post it as an email...
>
> I have familiarized myself with pypy and read most of the docs and read
> the code in the metainterp to understand how the traces are built.
>
> I translated pypy with --withmod-micronumpy enabled (installed numpy
> pypy fork in an virtual env) and gave the jitviewer a try.
> Here is a sample program that does not really make sense, but I think it
> would contain opportunity to generate SIMD instructions.
>
> ```
> import numpy
> def invoke():
>     a = numpy.zeros(1024)
>     b = numpy.zeros(1024)
>     c = numpy.zeros(1024)
>     # (2)
>     for i in range(0,1024):
>         c[i] = a[i] + b[i]
>     # (3)
>     c = a + b
>
> if __name__ == '__main__':
>     i = 0
>     # (1)
>     while i < 500:
>         invoke()
>         i += 1
> ```
>
> Here is the trace of invoke visible in jitviewer (uploaded to
> https://postimg.org/image/kbntluw55/full/).
>
> Here are some questions I have that would really help me to get going:
>
> (1) Is there a better way to get loops hot?

no, not really (you can lower the threshold though, see pypy --jit
help for details, only global)

>
> (2) I cannot really figure out what all those trace/loop parameters are.
> obviously i can guess some but most of them do not really make sense to
> me. Am I missing some settings to be able to display more information
> for the trace?
>     In addition to that I do not really see any chance to generate a
> simd loop for (2). Every array access is (arguably) guarded by an array
> index overflow and I think to skip that check would be invalid.

Not really, no, you can't get a better info, but I can probably explain

Some of the values can be easily promoted to constants (array sizes
for multiplication for example). What you can do for array bound
checks is to unroll this loop e.g. 4 times and do ONE array bound
check instead of 4 (checking x + 4). Actually it's possible to have an
optimization that removes all counters and has ONE number to track the
iteration (it's not that hard if the arrays are similar, you can
precompute if they match).

>
> (3) There is no trace generated for this instruction? Does this
> internally call a c function?

No, the trace follows the loop and does not compile the + (but the +
has it's own loop anyway)

>
> (4) What in our opinion makes sense to vectorize with simd instructions?
> Could you provide some sample loops/code (ranging from simple to complex
> loops)?

I can rewrite this loop in a vectorizable way if you want

>
> I would really appreciate any help and/or push into the right direction.
>
> Best,
> Richard
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From bokr at oz.net  Wed Feb 25 01:12:12 2015
From: bokr at oz.net (Bengt Richter)
Date: Wed, 25 Feb 2015 01:12:12 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
Message-ID: <54ED135C.3080405@oz.net>

On 02/24/2015 11:17 PM Maciej Fijalkowski wrote:
> Hi Richard.
>
>
> I will respond inline
>
> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>> hi,
[...]
>>
>> (1) Is there a better way to get loops hot?
>
> no, not really (you can lower the threshold though, see pypy --jit
> help for details, only global)
>
>>
PMJI, but I am curious if it would be possible to change pypy to use
mmap'd files for all memory allocations for heaps and stacks
so that the state of hotness and jit state could be captured.

Files could be virtually large, since UIAM physical allocation does
not occur until write, or at least is controllable.

The idea then would be to write programs with a warm-up prelude function,
and then have a checkpointing module with a method that could write a
specially stubbed ELF file along with all the file data, so that the ELF
would be an executable whose _start would get back to the checkpoint module
where everything would be restored as it was checkpointed, and execution
would continue as if just returning from the call to the checkpointing method,
which would be after the forced warmup prelude.

Sorry if I am intruding.
Regards,
Bengt Richter




From vincent.legoll at gmail.com  Wed Feb 25 07:14:57 2015
From: vincent.legoll at gmail.com (Vincent Legoll)
Date: Wed, 25 Feb 2015 07:14:57 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <54ED135C.3080405@oz.net>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
 <54ED135C.3080405@oz.net>
Message-ID: <CAEwRq=rjos2MZRmmeGx-gXaGeG2T11vv8h9GY01KWnYF9HGqDg@mail.gmail.com>

Hello Bengt,

If I'm not mistaken I think this FAQ item should at least partly answer
your question :

http://pypy.readthedocs.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-already-compiled-machine-code

Regards

On Wed, Feb 25, 2015 at 1:12 AM, Bengt Richter <bokr at oz.net> wrote:

> On 02/24/2015 11:17 PM Maciej Fijalkowski wrote:
>
>> Hi Richard.
>>
>>
>> I will respond inline
>>
>> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>>
>>> hi,
>>>
>> [...]
>
>>
>>> (1) Is there a better way to get loops hot?
>>>
>>
>> no, not really (you can lower the threshold though, see pypy --jit
>> help for details, only global)
>>
>>
>>>  PMJI, but I am curious if it would be possible to change pypy to use
> mmap'd files for all memory allocations for heaps and stacks
> so that the state of hotness and jit state could be captured.
>
> Files could be virtually large, since UIAM physical allocation does
> not occur until write, or at least is controllable.
>
> The idea then would be to write programs with a warm-up prelude function,
> and then have a checkpointing module with a method that could write a
> specially stubbed ELF file along with all the file data, so that the ELF
> would be an executable whose _start would get back to the checkpoint module
> where everything would be restored as it was checkpointed, and execution
> would continue as if just returning from the call to the checkpointing
> method,
> which would be after the forced warmup prelude.
>
> Sorry if I am intruding.
> Regards,
> Bengt Richter
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
Vincent Legoll
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150225/b958e69c/attachment.html>

From rich at pasra.at  Wed Feb 25 10:28:06 2015
From: rich at pasra.at (Richard Plangger)
Date: Wed, 25 Feb 2015 10:28:06 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
Message-ID: <54ED95A6.4050104@pasra.at>

Thx for your response.

Yes I would be interested in a rewritten loop.

After browsing the jittracer longer I found the following.

```
label(p0, f16, p2, p3, p55, i54, i59, p20, i18, p6, i24, p36, i34, i40,
f32, i15, i23, i31, i39, i48, i58, i60, descr=TargetToken(41748336))

(numpy_call2: no get_printable_location)
f62 = raw_load(i15, i24, descr=<ArrayF 8>)
guard_not_invalidated(descr=<Guard0x246fe60>)
i63 = i24 + i23
f64 = raw_load(i31, i40, descr=<ArrayF 8>)
i65 = i40 + i39
f66 = f62 + f64
raw_store(i48, i59, f66, descr=<ArrayF 8>)
i67 = i54 + 1
i68 = i59 + i58
i69 = i67 >= i60
guard(i69 is false) show bridge  (run 2799 times, ~0%)
(numpy_call2: no get_printable_location)

jump(p0, f62, p2, p3, p55, i67, i68, p20, i18, p6, i63, p36, i34, i65,
f64, i15, i23, i31, i39, i48, i58, i60, descr=TargetToken(41748336))
```

Is this the trace that calculates a = b + c internally?
This looks much more like what I have had in mind.

Here some questions that I have:

1) That does the prefix p/i mean in the label arguments? I guess it is a
box of i -> integer, p -> pointer, but why does raw_load(i31, ...)
operate on an i31 instead of p31?

2) Why does every raw_load/raw_store has its own index calculation (e.g.
i63 = i40 + i39) instead of just using one common index?

3) Are the numpy arrays or python arrays aligned in memory? If not would
it be hard to change their alignment when they are allocated?

Best,
Richard

On 02/24/2015 11:17 PM, Maciej Fijalkowski wrote:
> Hi Richard.
> 
> 
> I will respond inline
> 
> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>> hi,
>>
...
>>
>> (1) Is there a better way to get loops hot?
> 
> no, not really (you can lower the threshold though, see pypy --jit
> help for details, only global)
> 
>>
>> (2) I cannot really figure out what all those trace/loop parameters are.
>> obviously i can guess some but most of them do not really make sense to
>> me. Am I missing some settings to be able to display more information
>> for the trace?
>>     In addition to that I do not really see any chance to generate a
>> simd loop for (2). Every array access is (arguably) guarded by an array
>> index overflow and I think to skip that check would be invalid.
> 
> Not really, no, you can't get a better info, but I can probably explain
> 
> Some of the values can be easily promoted to constants (array sizes
> for multiplication for example). What you can do for array bound
> checks is to unroll this loop e.g. 4 times and do ONE array bound
> check instead of 4 (checking x + 4). Actually it's possible to have an
> optimization that removes all counters and has ONE number to track the
> iteration (it's not that hard if the arrays are similar, you can
> precompute if they match).
> 
>>
>> (3) There is no trace generated for this instruction? Does this
>> internally call a c function?
> 
> No, the trace follows the loop and does not compile the + (but the +
> has it's own loop anyway)
> 
>>
>> (4) What in our opinion makes sense to vectorize with simd instructions?
>> Could you provide some sample loops/code (ranging from simple to complex
>> loops)?
> 
> I can rewrite this loop in a vectorizable way if you want
> 
>>
>> I would really appreciate any help and/or push into the right direction.
>>
>> Best,
>> Richard
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150225/14c285af/attachment.sig>

From fijall at gmail.com  Wed Feb 25 10:45:28 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Feb 2015 11:45:28 +0200
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <54ED95A6.4050104@pasra.at>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
 <54ED95A6.4050104@pasra.at>
Message-ID: <CAK5idxS31-JA3gXbnv_Nn5UA0X46cJG=jBH4LB=pWSxKF=f5aw@mail.gmail.com>

On Wed, Feb 25, 2015 at 11:28 AM, Richard Plangger <rich at pasra.at> wrote:
> Thx for your response.
>
> Yes I would be interested in a rewritten loop.
>
> After browsing the jittracer longer I found the following.
>
> ```
> label(p0, f16, p2, p3, p55, i54, i59, p20, i18, p6, i24, p36, i34, i40,
> f32, i15, i23, i31, i39, i48, i58, i60, descr=TargetToken(41748336))
>
> (numpy_call2: no get_printable_location)
> f62 = raw_load(i15, i24, descr=<ArrayF 8>)
> guard_not_invalidated(descr=<Guard0x246fe60>)
> i63 = i24 + i23
> f64 = raw_load(i31, i40, descr=<ArrayF 8>)
> i65 = i40 + i39
> f66 = f62 + f64
> raw_store(i48, i59, f66, descr=<ArrayF 8>)
> i67 = i54 + 1
> i68 = i59 + i58
> i69 = i67 >= i60
> guard(i69 is false) show bridge  (run 2799 times, ~0%)
> (numpy_call2: no get_printable_location)
>
> jump(p0, f62, p2, p3, p55, i67, i68, p20, i18, p6, i63, p36, i34, i65,
> f64, i15, i23, i31, i39, i48, i58, i60, descr=TargetToken(41748336))
> ```
>
> Is this the trace that calculates a = b + c internally?

yes

> This looks much more like what I have had in mind.
>
> Here some questions that I have:
>
> 1) That does the prefix p/i mean in the label arguments? I guess it is a
> box of i -> integer, p -> pointer, but why does raw_load(i31, ...)
> operate on an i31 instead of p31?

p is a gc pointer, i is an integer or raw pointer (we should invent
new name for it, it's biting us in the JIT)

>
> 2) Why does every raw_load/raw_store has its own index calculation (e.g.
> i63 = i40 + i39) instead of just using one common index?

because we did not optimize it correctly ;-) it's about specializing
the loop and detecting you can reuse the same indexing

>
> 3) Are the numpy arrays or python arrays aligned in memory? If not would
> it be hard to change their alignment when they are allocated?

we control the alignment (I think they're 8 byte aligned now, but can
be made 16). Remember that numpy arrays don't have to be contiguous so
you have to be a bit careful

>
> Best,
> Richard
>
> On 02/24/2015 11:17 PM, Maciej Fijalkowski wrote:
>> Hi Richard.
>>
>>
>> I will respond inline
>>
>> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>>> hi,
>>>
> ...
>>>
>>> (1) Is there a better way to get loops hot?
>>
>> no, not really (you can lower the threshold though, see pypy --jit
>> help for details, only global)
>>
>>>
>>> (2) I cannot really figure out what all those trace/loop parameters are.
>>> obviously i can guess some but most of them do not really make sense to
>>> me. Am I missing some settings to be able to display more information
>>> for the trace?
>>>     In addition to that I do not really see any chance to generate a
>>> simd loop for (2). Every array access is (arguably) guarded by an array
>>> index overflow and I think to skip that check would be invalid.
>>
>> Not really, no, you can't get a better info, but I can probably explain
>>
>> Some of the values can be easily promoted to constants (array sizes
>> for multiplication for example). What you can do for array bound
>> checks is to unroll this loop e.g. 4 times and do ONE array bound
>> check instead of 4 (checking x + 4). Actually it's possible to have an
>> optimization that removes all counters and has ONE number to track the
>> iteration (it's not that hard if the arrays are similar, you can
>> precompute if they match).
>>
>>>
>>> (3) There is no trace generated for this instruction? Does this
>>> internally call a c function?
>>
>> No, the trace follows the loop and does not compile the + (but the +
>> has it's own loop anyway)
>>
>>>
>>> (4) What in our opinion makes sense to vectorize with simd instructions?
>>> Could you provide some sample loops/code (ranging from simple to complex
>>> loops)?
>>
>> I can rewrite this loop in a vectorizable way if you want
>>
>>>
>>> I would really appreciate any help and/or push into the right direction.
>>>
>>> Best,
>>> Richard
>>>
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>

From bokr at oz.net  Wed Feb 25 15:20:47 2015
From: bokr at oz.net (Bengt Richter)
Date: Wed, 25 Feb 2015 15:20:47 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <CAEwRq=rjos2MZRmmeGx-gXaGeG2T11vv8h9GY01KWnYF9HGqDg@mail.gmail.com>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
 <54ED135C.3080405@oz.net>
 <CAEwRq=rjos2MZRmmeGx-gXaGeG2T11vv8h9GY01KWnYF9HGqDg@mail.gmail.com>
Message-ID: <54EDDA3F.4040804@oz.net>

Hi Vincent,
I was aware of the FAQ item (my similar question long ago may have helped put it in the FAQ ;-)
AIUI the main issue is re-establishing the memory mapping, which I think could be re-established
by mmap-ing the saved files, if those files were created through mmap in the first place (along
with what lsofs might show at checkpoint time.

But in order to capture memory, malloc and the gc would have to be reworked to operate on memory by appending
zeroes to mmap-ed files serving as memory pools. If the program is running single threaded at the time the
checkpoint method is called, there wouldn't be an issue of restoring blocked multiple threads.

I would guess the remaining things would be the state of open files, but IWT their state could be saved and
reopens and seeks could be done on resume start-up.

A question would be if the jit discards warmed-up code so that it would not be seen on resume, but how would
it know there wasn't going to be an ordinary return from the checkpointing call?

With multiple mmap-ed files serving as memory pools, maybe they could even be put in a hash-identified directory
and gzipped for re-setup by the ELF resumption stub.

The idea for an elf stub would be to write a c program with ELF data space such that you could copy it and append
resumption data to result in data space seen by the resuming program. Something on the idea of peCOFF boot stub
for the linux kernel, but just for the individual pypy-warm-resume (at a minimum the gzipped mmap files archive name if
the latter is not actually appended to the stup program).

My hunch is that between the stackless guy Christian and Armin, they could figure it out ;-)
Maybe it's worth a re-think, if only to say "no, we really mean no" in the FAQ ;-)

Regards,
Bengt Richter

On 02/25/2015 07:14 AM Vincent Legoll wrote:
> Hello Bengt,
>
> If I'm not mistaken I think this FAQ item should at least partly answer
> your question :
>
> http://pypy.readthedocs.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-already-compiled-machine-code
>
> Regards
>
> On Wed, Feb 25, 2015 at 1:12 AM, Bengt Richter <bokr at oz.net> wrote:
>
>> On 02/24/2015 11:17 PM Maciej Fijalkowski wrote:
>>
>>> Hi Richard.
>>>
>>>
>>> I will respond inline
>>>
>>> On Tue, Feb 24, 2015 at 8:18 PM, Richard Plangger <rich at pasra.at> wrote:
>>>
>>>> hi,
>>>>
>>> [...]
>>
>>>
>>>> (1) Is there a better way to get loops hot?
>>>>
>>>
>>> no, not really (you can lower the threshold though, see pypy --jit
>>> help for details, only global)
>>>
>>>
>>>>   PMJI, but I am curious if it would be possible to change pypy to use
>> mmap'd files for all memory allocations for heaps and stacks
>> so that the state of hotness and jit state could be captured.
>>
>> Files could be virtually large, since UIAM physical allocation does
>> not occur until write, or at least is controllable.
>>
>> The idea then would be to write programs with a warm-up prelude function,
>> and then have a checkpointing module with a method that could write a
>> specially stubbed ELF file along with all the file data, so that the ELF
>> would be an executable whose _start would get back to the checkpoint module
>> where everything would be restored as it was checkpointed, and execution
>> would continue as if just returning from the call to the checkpointing
>> method,
>> which would be after the forced warmup prelude.
>>
>> Sorry if I am intruding.
>> Regards,
>> Bengt Richter
>>
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>


From arigo at tunes.org  Sat Feb 28 18:23:13 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 28 Feb 2015 18:23:13 +0100
Subject: [pypy-dev] Vectorizing numpy traces
In-Reply-To: <54EDDA3F.4040804@oz.net>
References: <54ECC062.7010209@pasra.at>
 <CAK5idxT3+-DHg8ztOjwEf78CNpAn_CK=EXzMMoFHsvhKM6Qf6A@mail.gmail.com>
 <54ED135C.3080405@oz.net>
 <CAEwRq=rjos2MZRmmeGx-gXaGeG2T11vv8h9GY01KWnYF9HGqDg@mail.gmail.com>
 <54EDDA3F.4040804@oz.net>
Message-ID: <CAMSv6X3Bn6UZH25QqRsDXqoEOJk581nHF4saFPuFaAcPvvgdRA@mail.gmail.com>

Hi Bengt,

On 25 February 2015 at 15:20, Bengt Richter <bokr at oz.net> wrote:
> Maybe it's worth a re-think, if only to say "no, we really mean no" in the
> FAQ ;-)

It's unclear to me if you're suggesting something more than adding a
checkpointing system to stop and restart the process.  It's a hack
that might work in some cases and not in others.  For example,
re-opening the files and re-seeking them to the same position as
before --- it seems interesting but I somehow doubt it is very helpful
in general.

Another point of view on your suggestion is "just use os.fork()", as
this solves all these hard issues out of the box.  Of course you can't
save the forked process to disk, but that seems like a small price to
pay and it works on today's PyPy.


A bient?t,

Armin.