From joehillen at gmail.com  Mon Dec  3 01:09:39 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Sun, 2 Dec 2012 16:09:39 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
Message-ID: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>

Now that lxml works in PyPy, I've been excited to try Scrapy in PyPy 2.0

I've run into this issue. I'm not sure what could be happening here, but I
suspect it could be a twisted+pypy issue. I'm hoping it might look familiar
to someone.

ERROR: Error caught on signal handler: <bound method LogStats.spider_opened
of <scrapy.contrib.logstats.LogStats object at 0x0000000006f3a8e0>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 225,
in open_spider
        yield self.signals.send_catch_log_deferred(signals.spider_opened,
spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 53,
in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
134, in maybeDeferred
        result = f(*args, **kw)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: spider_opened() got 2 unexpected keyword arguments

ERROR: Error caught on signal handler: <bound method
LogStats.response_received of <scrapy.contrib.logstats.LogStats object at
0x0000000006f3a8e0>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
464, in _startRunCallbacks
        self._runCallbacks()
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 200,
in _on_success
        response=response, request=request, spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
19, in send_catch_log
        return signal.send_catch_log(*a, **kw)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 22,
in send_catch_log
        *arguments, **named)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: response_received() got 4 unexpected keyword
arguments

ERROR: Error caught on signal handler: <bound method
CoreStats.response_received of <scrapy.contrib.corestats.CoreStats object
at 0x00000000061b8d08>>
    Traceback (most recent call last):
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
464, in _startRunCallbacks
        self._runCallbacks()
      File "/usr/local/pypy/site-packages/twisted/internet/defer.py", line
551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/pypy/site-packages/scrapy/core/engine.py", line 200,
in _on_success
        response=response, request=request, spider=spider)
      File "/usr/local/pypy/site-packages/scrapy/signalmanager.py", line
19, in send_catch_log
        return signal.send_catch_log(*a, **kw)
    --- <exception caught here> ---
      File "/usr/local/pypy/site-packages/scrapy/utils/signal.py", line 22,
in send_catch_log
        *arguments, **named)
      File
"/usr/local/pypy/site-packages/scrapy/xlib/pydispatch/robustapply.py", line
47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: response_received() got 4 unexpected keyword
arguments

Here are the definitions for CoreStats and LogStats:

https://github.com/scrapy/scrapy/blob/0.16/scrapy/contrib/logstats.py
https://github.com/scrapy/scrapy/blob/0.16/scrapy/contrib/corestats.py

Let me know if this is a PyPy bug and I will turn it into a bug report.

Thanks
-Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121202/8656497c/attachment.html>

From amauryfa at gmail.com  Mon Dec  3 11:23:51 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 3 Dec 2012 11:23:51 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
Message-ID: <CAGmFidYGk9RAbf6mDwEu=wK0udi893i1c0_msd5EccrepzCbEw@mail.gmail.com>

Hi,

2012/12/3 Joe Hillenbrand <joehillen at gmail.com>

>     exceptions.TypeError: spider_opened() got 2 unexpected keyword
> arguments


Could you modify a bit this spider_opened function, so that it accepts a
**kwargs?
Something like (not tested!):

    def spider_opened(self, spider, **kwargs):
        if kwargs:
            raise TypeError("unexpected keywords", kwargs)
        ...


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121203/bcadb6e7/attachment.html>

From wiktor8010 at o2.pl  Mon Dec  3 16:24:28 2012
From: wiktor8010 at o2.pl (=?UTF-8?Q?Wiktor_Mizdal?=)
Date: Mon, 03 Dec 2012 16:24:28 +0100
Subject: [pypy-dev] =?utf-8?q?PPC?=
Message-ID: <4bd2730e.42915765.50bcc42c.930f5@o2.pl>


Which operating systems will support Pypy PowerPC backend?
When it will appear?

From dje.gcc at gmail.com  Mon Dec  3 17:29:50 2012
From: dje.gcc at gmail.com (David Edelsohn)
Date: Mon, 3 Dec 2012 11:29:50 -0500
Subject: [pypy-dev] PPC
In-Reply-To: <4bd2730e.42915765.50bcc42c.930f5@o2.pl>
References: <4bd2730e.42915765.50bcc42c.930f5@o2.pl>
Message-ID: <CAGWvnynJAZ1Rqe6tvLiHkebb=JiYbaBn2i1Hszws2xwbT1674g@mail.gmail.com>

On Mon, Dec 3, 2012 at 10:24 AM, Wiktor Mizdal <wiktor8010 at o2.pl> wrote:
>
> Which operating systems will support Pypy PowerPC backend?

PPC64 Linux.  It probably could run on PPC64 FreeBSD as well.  It
could run on AIX, because it does not rely on a lot of system
services, but the PyPy infrastructure currently does not have the
hooks for AIX.

> When it will appear?

As soon as the remaining bug triggered by GC can be analyzed and fixed.

Thanks, David

From fijall at gmail.com  Mon Dec  3 18:22:59 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 3 Dec 2012 09:22:59 -0800
Subject: [pypy-dev] PPC
In-Reply-To: <CAGWvnynJAZ1Rqe6tvLiHkebb=JiYbaBn2i1Hszws2xwbT1674g@mail.gmail.com>
References: <4bd2730e.42915765.50bcc42c.930f5@o2.pl>
	<CAGWvnynJAZ1Rqe6tvLiHkebb=JiYbaBn2i1Hszws2xwbT1674g@mail.gmail.com>
Message-ID: <CAK5idxR8uoaSDjO_VF2-qJiFygGhOQKcMpCeZXvFQOe3nz_n8w@mail.gmail.com>

On Mon, Dec 3, 2012 at 8:29 AM, David Edelsohn <dje.gcc at gmail.com> wrote:
> On Mon, Dec 3, 2012 at 10:24 AM, Wiktor Mizdal <wiktor8010 at o2.pl> wrote:
>>
>> Which operating systems will support Pypy PowerPC backend?
>
> PPC64 Linux.  It probably could run on PPC64 FreeBSD as well.  It
> could run on AIX, because it does not rely on a lot of system
> services, but the PyPy infrastructure currently does not have the
> hooks for AIX.

as far as freebsd support goes (it's not officially supported and
there are known failing tests)

>
>> When it will appear?
>
> As soon as the remaining bug triggered by GC can be analyzed and fixed.
>
> Thanks, David
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From arigo at tunes.org  Mon Dec  3 19:01:47 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Dec 2012 10:01:47 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
Message-ID: <CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>

Hi,

On Sun, Dec 2, 2012 at 4:09 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
> (...)
>         response=response, request=request, spider=spider)
> (...)
>         return receiver(*arguments, **named)
>     exceptions.TypeError: response_received() got 4 unexpected keyword arguments

No real clue, but it looks like keyword arguments are passed around as
keyword arguments on PyPy, whereas CPython converts them to positional
arguments somewhere along the call chain (which is long and goes via
deferreds).  For us to help more, please provide a step-by-step "how
to reproduce" list.  It's typically easy if we can reproduce the bug
locally, and very hard if not.


A bient?t,

Armin.

From joehillen at gmail.com  Tue Dec  4 00:52:23 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Mon, 3 Dec 2012 15:52:23 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
Message-ID: <CANJQ5C7wESLRONRH5j2BiL2-MTVKACUd=bOMdYFUD-m69hr-KQ@mail.gmail.com>

I've narrowed down the problem thanks to Amaury's suggestion.

It looks like it is caused by some black magic to figure out what arguments
a signal handler can accept.

https://github.com/scrapy/scrapy/blob/0.16/scrapy/xlib/pydispatch/robustapply.py

I haven't completely figured out what it is trying to do. It looks like it
is touching innards that PyPy might not have.

 It doesn't look like it is particularly well written anyway, so it will
probably need to be rewritten to be both CPython and PyPy compatible.


On Mon, Dec 3, 2012 at 10:01 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi,
>
> On Sun, Dec 2, 2012 at 4:09 PM, Joe Hillenbrand <joehillen at gmail.com>
> wrote:
> > (...)
> >         response=response, request=request, spider=spider)
> > (...)
> >         return receiver(*arguments, **named)
> >     exceptions.TypeError: response_received() got 4 unexpected keyword
> arguments
>
> No real clue, but it looks like keyword arguments are passed around as
> keyword arguments on PyPy, whereas CPython converts them to positional
> arguments somewhere along the call chain (which is long and goes via
> deferreds).  For us to help more, please provide a step-by-step "how
> to reproduce" list.  It's typically easy if we can reproduce the bug
> locally, and very hard if not.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121203/ce7f5a75/attachment.html>

From stefan_ml at behnel.de  Tue Dec  4 10:14:37 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 04 Dec 2012 10:14:37 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
Message-ID: <k9ketr$1bu$1@ger.gmane.org>

Armin Rigo, 03.12.2012 19:01:
> On Sun, Dec 2, 2012 at 4:09 PM, Joe Hillenbrand wrote:
>> (...)
>>         response=response, request=request, spider=spider)
>> (...)
>>         return receiver(*arguments, **named)
>>     exceptions.TypeError: response_received() got 4 unexpected keyword arguments
> 
> No real clue, but it looks like keyword arguments are passed around as
> keyword arguments on PyPy, whereas CPython converts them to positional
> arguments somewhere along the call chain (which is long and goes via
> deferreds).

I noticed that PyPy validates keyword arguments at a different stage than
CPython, at least at the cpyext level. PyPy does it at call time, whereas
CPython leaves it to the callee. Leads to this difference for the argument
unpacking code in Cython implemented functions:

https://github.com/cython/cython/blob/1cc172882ff754840f025ce81b2ed3183dfea93f/Cython/Utility/FunctionArguments.c#L123

Stefan



From grickert at coldstorage.com  Wed Dec  5 20:20:01 2012
From: grickert at coldstorage.com (Gerrat Rickert)
Date: Wed, 5 Dec 2012 19:20:01 +0000
Subject: [pypy-dev] A minor nit
Message-ID: <25CB87184063F24C9BDD8C5E3566EA2506AA82@kit-ex01.coldstorage.com>

The installation instructions on pypy.org/download.html for the Windows binary (32bit) say "you may also need the VS 2009 runtime library installer vcredist_x86.exe<http://www.microsoft.com/en-us/download/details.aspx?id=5582>)"
This link actually points to something called the "Microsoft Visual C++ 2008 SP1 Redistributable package"...which doesn't seem to help.
When pypy is run, it complains that its missing msvcr100.dll.

It looks like what's actually needed for this file is the "Microsoft Visual C++ 2010 Redistributable package", located here:
http://www.microsoft.com/en-us/download/details.aspx?id=5555


Regards,
                Gerrat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121205/2e2133bd/attachment.html>

From grickert at coldstorage.com  Wed Dec  5 20:21:42 2012
From: grickert at coldstorage.com (Gerrat Rickert)
Date: Wed, 5 Dec 2012 19:21:42 +0000
Subject: [pypy-dev] A minor nit
Message-ID: <25CB87184063F24C9BDD8C5E3566EA2506AA91@kit-ex01.coldstorage.com>

Sorry, I should have mentioned my last message is referring specifically to the 2.0 beta1 instructions.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121205/342e5b23/attachment.html>

From matti.picus at gmail.com  Wed Dec  5 20:29:05 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 05 Dec 2012 21:29:05 +0200
Subject: [pypy-dev] A minor nit
In-Reply-To: <25CB87184063F24C9BDD8C5E3566EA2506AA91@kit-ex01.coldstorage.com>
References: <25CB87184063F24C9BDD8C5E3566EA2506AA91@kit-ex01.coldstorage.com>
Message-ID: <50BFA081.108@gmail.com>

Hmm, that's actually two bugs:
- the link on the web site has the wrong label but the correct link
- the buildbot picked up the wrong compiler, it should not be using the 
visual stidio 10 compiler.
Thanks for pointing this out.
Matti

On 5/12/2012 9:21 PM, Gerrat Rickert wrote:
>
> Sorry, I should have mentioned my last message is referring 
> specifically to the 2.0 beta1 instructions.
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev


From amauryfa at gmail.com  Wed Dec  5 21:02:29 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 5 Dec 2012 21:02:29 +0100
Subject: [pypy-dev] A minor nit
In-Reply-To: <50BFA081.108@gmail.com>
References: <25CB87184063F24C9BDD8C5E3566EA2506AA91@kit-ex01.coldstorage.com>
	<50BFA081.108@gmail.com>
Message-ID: <CAGmFidbQaJbovh5GXX2dRQqZ_Vg-d-1OQYH_=+ap29MpcXS8ag@mail.gmail.com>

2012/12/5 Matti Picus <matti.picus at gmail.com>

> - the buildbot picked up the wrong compiler, it should not be using the
> visual stidio 10 compiler.


The windows build will pick the latest compiler available, picked in a list:
In pypy/translator/platform/windows.py:
    for vsver in (100, 90, 80, 71, 70): # All the versions I know

If vs2008 should be preferred vs2010, it's enough to swap the first two
entries.
And add a comment...

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121205/652417fc/attachment.html>

From joehillen at gmail.com  Thu Dec  6 00:40:41 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Wed, 5 Dec 2012 15:40:41 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <k9ketr$1bu$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
Message-ID: <CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>

I've found a place where PyPy and CPython disagree.

https://gist.github.com/4220533

This might not be the only issue, but it's the first thing I've found so
far.

-Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121205/44e8ecd2/attachment.html>

From fijall at gmail.com  Thu Dec  6 09:34:28 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 00:34:28 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
Message-ID: <CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>

On Wed, Dec 5, 2012 at 3:40 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
> I've found a place where PyPy and CPython disagree.
>
> https://gist.github.com/4220533
>
> This might not be the only issue, but it's the first thing I've found so
> far.
>
> -Joe

This is a well known issue - PyPy's methods and builtin methods are
not that different. We can probably fix it though

From jonathan at slenders.be  Thu Dec  6 10:58:39 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Thu, 6 Dec 2012 10:58:39 +0100
Subject: [pypy-dev] Javascript interpreter and r_uint
Message-ID: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>

Hi all,

Yesterday, I did some experiments with the existing javascript interpreter
that was written in RPython. [1]
It worked very well when interpreted by CPython, but failed to translate to
C because of unsigned int issues.

Quite a few times, unsigned and signed objects were compared or added, or
one asigned to the other. (I'm not entirely sure.)
But I had to replace r_uint32 by r_int32 and r_uint by r_int, about 25
times, before I could translate anything.

So, my question is. What is the reason for having unsigned integers in the
javascript interpreter?

Apart from that, it works great, and I translated the interpreter even with
the --sandbox option!

A related question. how mature is the interpreter? It seems to run pretty
stable, although there hasn't been any development done since june 2011.
I'd like to choose this instead of Google's V8, because that way I can keep
the same sandbox hypervisor for both Python and Javascript.

[1] https://bitbucket.org/pypy/lang-js/

Cheers,
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121206/d427aea7/attachment.html>

From fijall at gmail.com  Thu Dec  6 11:14:54 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 02:14:54 -0800
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
Message-ID: <CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>

On Thu, Dec 6, 2012 at 1:58 AM, Jonathan Slenders <jonathan at slenders.be> wrote:
> Hi all,
>
> Yesterday, I did some experiments with the existing javascript interpreter
> that was written in RPython. [1]
> It worked very well when interpreted by CPython, but failed to translate to
> C because of unsigned int issues.
>
> Quite a few times, unsigned and signed objects were compared or added, or
> one asigned to the other. (I'm not entirely sure.)
> But I had to replace r_uint32 by r_int32 and r_uint by r_int, about 25
> times, before I could translate anything.

Good question, I have no idea. Can't you use normal integers in those places?

>
> So, my question is. What is the reason for having unsigned integers in the
> javascript interpreter?
>
> Apart from that, it works great, and I translated the interpreter even with
> the --sandbox option!
>
> A related question. how mature is the interpreter? It seems to run pretty
> stable, although there hasn't been any development done since june 2011.
> I'd like to choose this instead of Google's V8, because that way I can keep
> the same sandbox hypervisor for both Python and Javascript.

The interpreter is relatively complete, however it does require fixing
a lot of corner cases from ECMA tests. There are quite a few missing
pieces:

* the correct semicolon insertion requires a different parser
(standard says LL(1))

* regular expressions are not supported

* xml literals are unsupported

Cheers,
fijal

From holger at merlinux.eu  Thu Dec  6 12:52:36 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 6 Dec 2012 11:52:36 +0000
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
	<CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
Message-ID: <20121206115235.GP26599@merlinux.eu>

On Thu, Dec 06, 2012 at 02:14 -0800, Maciej Fijalkowski wrote:
> On Thu, Dec 6, 2012 at 1:58 AM, Jonathan Slenders <jonathan at slenders.be> wrote:
> > Hi all,
> >
> > Yesterday, I did some experiments with the existing javascript interpreter
> > that was written in RPython. [1]
> > It worked very well when interpreted by CPython, but failed to translate to
> > C because of unsigned int issues.
> >
> > Quite a few times, unsigned and signed objects were compared or added, or
> > one asigned to the other. (I'm not entirely sure.)
> > But I had to replace r_uint32 by r_int32 and r_uint by r_int, about 25
> > times, before I could translate anything.
> 
> Good question, I have no idea. Can't you use normal integers in those places?
> 
> >
> > So, my question is. What is the reason for having unsigned integers in the
> > javascript interpreter?
> >
> > Apart from that, it works great, and I translated the interpreter even with
> > the --sandbox option!
> >
> > A related question. how mature is the interpreter? It seems to run pretty
> > stable, although there hasn't been any development done since june 2011.
> > I'd like to choose this instead of Google's V8, because that way I can keep
> > the same sandbox hypervisor for both Python and Javascript.
> 
> The interpreter is relatively complete, however it does require fixing
> a lot of corner cases from ECMA tests. There are quite a few missing
> pieces:
> 
> * the correct semicolon insertion requires a different parser
> (standard says LL(1))
> 
> * regular expressions are not supported
> 
> * xml literals are unsupported

just want add a little sidenote on the relevance of a Javascript/PyPy
interpreter.  As i recall, in previous years the perspective was that
it's not of much use if it's not tied into the DOM on browsers.  I think
that's not true anymore as Javascript is increasingly used on the server
via node.js and Rhino.  One major use case has people looking into how
to best perform e. g. js-templating on the server or client side exchangeably,
depending on capabilities of the client.  For that, there is no DOM
integration needed.  For a related discussion, see:
http://stackoverflow.com/questions/5494839/templating-language-for-both-client-side-and-server-side-rendering

Maybe the dreaded semicolon-insertion issue could even be ignored, not sure.

cheers,
holger

> Cheers,
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 

From fijall at gmail.com  Thu Dec  6 13:01:59 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 04:01:59 -0800
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <20121206115235.GP26599@merlinux.eu>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
	<CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
	<20121206115235.GP26599@merlinux.eu>
Message-ID: <CAK5idxRnV5+5cT86juUtM+xscQoQZuQbUMbZs=A7Mw9mK1HbdQ@mail.gmail.com>

On Thu, Dec 6, 2012 at 3:52 AM, holger krekel <holger at merlinux.eu> wrote:
> On Thu, Dec 06, 2012 at 02:14 -0800, Maciej Fijalkowski wrote:
>> On Thu, Dec 6, 2012 at 1:58 AM, Jonathan Slenders <jonathan at slenders.be> wrote:
>> > Hi all,
>> >
>> > Yesterday, I did some experiments with the existing javascript interpreter
>> > that was written in RPython. [1]
>> > It worked very well when interpreted by CPython, but failed to translate to
>> > C because of unsigned int issues.
>> >
>> > Quite a few times, unsigned and signed objects were compared or added, or
>> > one asigned to the other. (I'm not entirely sure.)
>> > But I had to replace r_uint32 by r_int32 and r_uint by r_int, about 25
>> > times, before I could translate anything.
>>
>> Good question, I have no idea. Can't you use normal integers in those places?
>>
>> >
>> > So, my question is. What is the reason for having unsigned integers in the
>> > javascript interpreter?
>> >
>> > Apart from that, it works great, and I translated the interpreter even with
>> > the --sandbox option!
>> >
>> > A related question. how mature is the interpreter? It seems to run pretty
>> > stable, although there hasn't been any development done since june 2011.
>> > I'd like to choose this instead of Google's V8, because that way I can keep
>> > the same sandbox hypervisor for both Python and Javascript.
>>
>> The interpreter is relatively complete, however it does require fixing
>> a lot of corner cases from ECMA tests. There are quite a few missing
>> pieces:
>>
>> * the correct semicolon insertion requires a different parser
>> (standard says LL(1))
>>
>> * regular expressions are not supported
>>
>> * xml literals are unsupported
>
> just want add a little sidenote on the relevance of a Javascript/PyPy
> interpreter.  As i recall, in previous years the perspective was that
> it's not of much use if it's not tied into the DOM on browsers.  I think
> that's not true anymore as Javascript is increasingly used on the server
> via node.js and Rhino.  One major use case has people looking into how
> to best perform e. g. js-templating on the server or client side exchangeably,
> depending on capabilities of the client.  For that, there is no DOM
> integration needed.  For a related discussion, see:
> http://stackoverflow.com/questions/5494839/templating-language-for-both-client-side-and-server-side-rendering
>
> Maybe the dreaded semicolon-insertion issue could even be ignored, not sure.

Judging from internet flamewars, no it cannot be. However, the rewrite
of the parser is not such a horrible idea, it's also necessary for the
descriptive error messages  I think.

The problem, *right now* is that it's a proof of concept that requires
some work to be finished and/or to use JIT and be fast. I won't do it,
also because I have other Open Source commitments at hand.

Cheers,
fijal

From santagada at gmail.com  Thu Dec  6 14:08:33 2012
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 6 Dec 2012 11:08:33 -0200
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
	<CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
Message-ID: <CA+HFc84P7kU3c+y6Nf-iQRSwCLyWioqN=WbL2UNBkM_gz2QptA@mail.gmail.com>

On Thu, Dec 6, 2012 at 8:14 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> * the correct semicolon insertion requires a different parser
> (standard says LL(1))
>

The apple javascript parser (and maybe V8) don't use a LL(1) parser and
still cope with ASI so maybe there are alternatives to this... but yep, for
better error messages someone should take better care of the parser.


> * regular expressions are not supported
>

would be nice to have python regexes as a start in the js interpreter...
but I don't know if the pypy-python interpreter code is reusable.


>
> * xml literals are unsupported


They are only supported in firefox and flash so I don't think too many
people use it, but then if needed you can try
https://github.com/laverdet/js-xml-literal

What I would work on is:
 - having more granular tests
 - jit support
 - benchmark suite
 - co-routine support, for having something like gevent for javascript
 - ???
 - profit

If you could have a fast javascript vm with gevent instead of pure callback
based programming could be a good alternative to node.js


-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121206/0105fb71/attachment.html>

From benjamin at python.org  Thu Dec  6 14:53:21 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 6 Dec 2012 08:53:21 -0500
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <CA+HFc84P7kU3c+y6Nf-iQRSwCLyWioqN=WbL2UNBkM_gz2QptA@mail.gmail.com>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
	<CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
	<CA+HFc84P7kU3c+y6Nf-iQRSwCLyWioqN=WbL2UNBkM_gz2QptA@mail.gmail.com>
Message-ID: <CAPZV6o_SxD5+9-gC6YBf76TJED_ZMX81MRL=f-7P0FEfk4085w@mail.gmail.com>

2012/12/6 Leonardo Santagada <santagada at gmail.com>:
> On Thu, Dec 6, 2012 at 8:14 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> * xml literals are unsupported
>
>
> They are only supported in firefox and flash so I don't think too many
> people use it, but then if needed you can try
> https://github.com/laverdet/js-xml-literal

Also, SpiderMonkey is going to remove support for XML soon because E4X is awful.


-- 
Regards,
Benjamin

From arigo at tunes.org  Thu Dec  6 15:44:09 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Dec 2012 15:44:09 +0100
Subject: [pypy-dev] Javascript interpreter and r_uint
In-Reply-To: <CAK5idxRnV5+5cT86juUtM+xscQoQZuQbUMbZs=A7Mw9mK1HbdQ@mail.gmail.com>
References: <CAKfyG3zyQES1=U4yZbiHoX5J_B_qb=0dLC_3o8-Mo=2twv8XCg@mail.gmail.com>
	<CAK5idxQO+KfvGsvKtGwUM0zg-UPOjrhvG1PP67xOCPT4oMNogw@mail.gmail.com>
	<20121206115235.GP26599@merlinux.eu>
	<CAK5idxRnV5+5cT86juUtM+xscQoQZuQbUMbZs=A7Mw9mK1HbdQ@mail.gmail.com>
Message-ID: <CAMSv6X0oZgw3OPxBf+GhRJF9EndWZ9TsnMwcn=ObeCX0PZwMcQ@mail.gmail.com>

Hi,

On Thu, Dec 6, 2012 at 1:01 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> The problem, *right now* is that it's a proof of concept that requires
> some work to be finished and/or to use JIT and be fast. I won't do it,
> also because I have other Open Source commitments at hand.

Same here.  Anyone is welcome to invest efforts, as long as he knows
there will be efforts to be invested --- i.e. it's not a one-week side
project.  It's maybe a 6-month or 1-year project.  And the end result
might compete in speed with V8, but will unlikely be universally
faster.  I'd rather guess that it would be generally a bit slower
overall, but more consistent, i.e. without any case of "oups, we
didn't implement this in the JIT so far, so if you do that, your
program's performance is dropping 10x".  And yes, it can use the same
sandboxing layer, and (if needed) the same STM layer, and so on, if
that's useful for your use case.  It is still 6-to-12 months of work.


A bient?t,

Armin.

From holger at merlinux.eu  Thu Dec  6 16:23:06 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 6 Dec 2012 15:23:06 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
Message-ID: <20121206152306.GQ26599@merlinux.eu>

Hi folks,

for the last several years i cared for hosting these pypy services:

- pypy.org (static generated html)
- bugs.pypy.org (roundup instance, postfix server)
- speed.python.org (django app)

These services are living on two rented servers which cost me per-month
and i am finally getting rid of them in January 2013.

IIRC Maciej and others had various ideas (last year) on where the
pypy services could better live.  Any progress on this?

ASFAIK the PSF is willing to offer a VM that we could use, at least
for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
has other contacts to try, go ahead and maybe CC me to avoid chaos.

Note that i actually don't want to drive these efforts but offer
my help in helping to migrate.  I'd be happy if the currently active
pypy devs determine a plan of action and drive it.
 
best,
holger


From fijall at gmail.com  Thu Dec  6 16:57:51 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 07:57:51 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121206152306.GQ26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
Message-ID: <CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>

On Thu, Dec 6, 2012 at 7:23 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi folks,
>
> for the last several years i cared for hosting these pypy services:
>
> - pypy.org (static generated html)
> - bugs.pypy.org (roundup instance, postfix server)
> - speed.python.org (django app)
>
> These services are living on two rented servers which cost me per-month
> and i am finally getting rid of them in January 2013.
>
> IIRC Maciej and others had various ideas (last year) on where the
> pypy services could better live.  Any progress on this?
>
> ASFAIK the PSF is willing to offer a VM that we could use, at least
> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
> has other contacts to try, go ahead and maybe CC me to avoid chaos.
>
> Note that i actually don't want to drive these efforts but offer
> my help in helping to migrate.  I'd be happy if the currently active
> pypy devs determine a plan of action and drive it.
>
> best,
> holger
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hi Holger.

I have servers and domains to move stuff there. The communication with
PSF has been erratic, so I would rather have it hosted on servers I
pay for.

Cheers,
fijal

From harry at pythonanywhere.com  Thu Dec  6 16:26:01 2012
From: harry at pythonanywhere.com (Harry Percival)
Date: Thu, 6 Dec 2012 15:26:01 -0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121206152306.GQ26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
Message-ID: <67F5C82632284B8881605160B7DA9D88@HarryDellWork>

What kind of traffic are you seeing?   perhaps we could offer some free 
hosting... Currently Python/WSGI only, but we should have support for static 
sites coming soon.

-- 
Harry Percival
Developer
harry at pythonanywhere.com

PythonAnywhere - a fully browser-based Python development and hosting 
environment
<http://www.pythonanywhere.com/>

PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK
-----Original Message----- 
From: holger krekel
Sent: Thursday, December 06, 2012 3:23 PM
To: pypy-dev at python.org
Subject: [pypy-dev] pypy services hosting / action needed!

Hi folks,

for the last several years i cared for hosting these pypy services:

- pypy.org (static generated html)
- bugs.pypy.org (roundup instance, postfix server)
- speed.python.org (django app)

These services are living on two rented servers which cost me per-month
and i am finally getting rid of them in January 2013.

IIRC Maciej and others had various ideas (last year) on where the
pypy services could better live.  Any progress on this?

ASFAIK the PSF is willing to offer a VM that we could use, at least
for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
has other contacts to try, go ahead and maybe CC me to avoid chaos.

Note that i actually don't want to drive these efforts but offer
my help in helping to migrate.  I'd be happy if the currently active
pypy devs determine a plan of action and drive it.

best,
holger

_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
http://mail.python.org/mailman/listinfo/pypy-dev 


From holger at merlinux.eu  Thu Dec  6 17:26:07 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 6 Dec 2012 16:26:07 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
Message-ID: <20121206162607.GR26599@merlinux.eu>

On Thu, Dec 06, 2012 at 07:57 -0800, Maciej Fijalkowski wrote:
> On Thu, Dec 6, 2012 at 7:23 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi folks,
> >
> > for the last several years i cared for hosting these pypy services:
> >
> > - pypy.org (static generated html)
> > - bugs.pypy.org (roundup instance, postfix server)
> > - speed.python.org (django app)
> >
> > These services are living on two rented servers which cost me per-month
> > and i am finally getting rid of them in January 2013.
> >
> > IIRC Maciej and others had various ideas (last year) on where the
> > pypy services could better live.  Any progress on this?
> >
> > ASFAIK the PSF is willing to offer a VM that we could use, at least
> > for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
> > Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
> > has other contacts to try, go ahead and maybe CC me to avoid chaos.
> >
> > Note that i actually don't want to drive these efforts but offer
> > my help in helping to migrate.  I'd be happy if the currently active
> > pypy devs determine a plan of action and drive it.
> >
> > best,
> > holger
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> 
> Hi Holger.
> 
> I have servers and domains to move stuff there. The communication with
> PSF has been erratic, so I would rather have it hosted on servers I
> pay for.

Two things:

- as said i'd like to act based on a joint decision from the group.  
  I can try to attend an IRC meeting to answer questions about the status quo.
- after that and if the group wants it this way, could you start
  with migrating speed.pypy.org which is the heaviest service and
  create a speed2.pypy.org (just mail me privately the target IP
  address)?

best,
holger

From fijall at gmail.com  Thu Dec  6 17:30:13 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 08:30:13 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121206162607.GR26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
Message-ID: <CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>

On Thu, Dec 6, 2012 at 8:26 AM, holger krekel <holger at merlinux.eu> wrote:
> On Thu, Dec 06, 2012 at 07:57 -0800, Maciej Fijalkowski wrote:
>> On Thu, Dec 6, 2012 at 7:23 AM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi folks,
>> >
>> > for the last several years i cared for hosting these pypy services:
>> >
>> > - pypy.org (static generated html)
>> > - bugs.pypy.org (roundup instance, postfix server)
>> > - speed.python.org (django app)
>> >
>> > These services are living on two rented servers which cost me per-month
>> > and i am finally getting rid of them in January 2013.
>> >
>> > IIRC Maciej and others had various ideas (last year) on where the
>> > pypy services could better live.  Any progress on this?
>> >
>> > ASFAIK the PSF is willing to offer a VM that we could use, at least
>> > for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
>> > Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
>> > has other contacts to try, go ahead and maybe CC me to avoid chaos.
>> >
>> > Note that i actually don't want to drive these efforts but offer
>> > my help in helping to migrate.  I'd be happy if the currently active
>> > pypy devs determine a plan of action and drive it.
>> >
>> > best,
>> > holger
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > http://mail.python.org/mailman/listinfo/pypy-dev
>>
>> Hi Holger.
>>
>> I have servers and domains to move stuff there. The communication with
>> PSF has been erratic, so I would rather have it hosted on servers I
>> pay for.
>
> Two things:
>
> - as said i'd like to act based on a joint decision from the group.
>   I can try to attend an IRC meeting to answer questions about the status quo.
> - after that and if the group wants it this way, could you start
>   with migrating speed.pypy.org which is the heaviest service and
>   create a speed2.pypy.org (just mail me privately the target IP
>   address)?
>
> best,
> holger

Hi Holger.

So let's wait for a group decision.

Cheers,
fijal

From arigo at tunes.org  Thu Dec  6 17:43:27 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Dec 2012 17:43:27 +0100
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
Message-ID: <CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>

Hi,

On Thu, Dec 6, 2012 at 5:30 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> So let's wait for a group decision.

Right now, we have extra servers sitting around not doing much.  I'm
for moving all three services there.  Fijal and me have access, and
I'm sure anyone else that needs it would have access too.


A bient?t,

Armin.

From anto.cuni at gmail.com  Thu Dec  6 18:11:52 2012
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 06 Dec 2012 18:11:52 +0100
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
Message-ID: <50C0D1D8.6000808@gmail.com>

Hi,

On 12/06/2012 05:43 PM, Armin Rigo wrote:

> Right now, we have extra servers sitting around not doing much.  I'm
> for moving all three services there.  Fijal and me have access, and
> I'm sure anyone else that needs it would have access too.

which servers?


From alex.gaynor at gmail.com  Thu Dec  6 19:47:43 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Thu, 6 Dec 2012 10:47:43 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <50C0D1D8.6000808@gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
Message-ID: <CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>

I have a slight preference for the PSF servers if it's possible, I'll get
in touch with Noah and see what I can do to help move this along.

Alex


On Thu, Dec 6, 2012 at 9:11 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:

> Hi,
>
> On 12/06/2012 05:43 PM, Armin Rigo wrote:
>
> > Right now, we have extra servers sitting around not doing much.  I'm
> > for moving all three services there.  Fijal and me have access, and
> > I'm sure anyone else that needs it would have access too.
>
> which servers?
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121206/8ccf5e9c/attachment.html>

From fijall at gmail.com  Fri Dec  7 08:58:48 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Dec 2012 23:58:48 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
	<CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
Message-ID: <CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>

On Thu, Dec 6, 2012 at 10:47 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> I have a slight preference for the PSF servers if it's possible, I'll get in
> touch with Noah and see what I can do to help move this along.
>
> Alex

I have serious preference for servers where people having physical
access can respond in a timely fashion, which are not the PSF servers.

Cheers,
fijal

>
>
> On Thu, Dec 6, 2012 at 9:11 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>>
>> Hi,
>>
>> On 12/06/2012 05:43 PM, Armin Rigo wrote:
>>
>> > Right now, we have extra servers sitting around not doing much.  I'm
>> > for moving all three services there.  Fijal and me have access, and
>> > I'm sure anyone else that needs it would have access too.
>>
>> which servers?
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From cfbolz at gmx.de  Fri Dec  7 14:35:24 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 07 Dec 2012 14:35:24 +0100
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
	<CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
	<CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>
Message-ID: <50C1F09C.9030508@gmx.de>

On 12/07/2012 08:58 AM, Maciej Fijalkowski wrote:
> On Thu, Dec 6, 2012 at 10:47 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>> I have a slight preference for the PSF servers if it's possible, I'll get in
>> touch with Noah and see what I can do to help move this along.
>>
>> Alex
>
> I have serious preference for servers where people having physical
> access can respond in a timely fashion, which are not the PSF servers.

I tend to agree with this point of view, but don't have a strong
preference.

Cheers,

Carl Friedrich


From fijall at gmail.com  Fri Dec  7 14:58:42 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 7 Dec 2012 05:58:42 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <50C1F09C.9030508@gmx.de>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
	<CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
	<CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>
	<50C1F09C.9030508@gmx.de>
Message-ID: <CAK5idxQuz=f+ZCEFEUzBep+49w4203iLxk9f+b8SFuoiygm5RA@mail.gmail.com>

On Fri, Dec 7, 2012 at 5:35 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> On 12/07/2012 08:58 AM, Maciej Fijalkowski wrote:
>>
>> On Thu, Dec 6, 2012 at 10:47 AM, Alex Gaynor <alex.gaynor at gmail.com>
>> wrote:
>>>
>>> I have a slight preference for the PSF servers if it's possible, I'll get
>>> in
>>> touch with Noah and see what I can do to help move this along.
>>>
>>> Alex
>>
>>
>> I have serious preference for servers where people having physical
>> access can respond in a timely fashion, which are not the PSF servers.
>
>
> I tend to agree with this point of view, but don't have a strong
> preference.
>
> Cheers,
>
> Carl Friedrich
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

for what is worth both pypy.org and bugs are down ATM and I cannot log
into machines.

From holger at merlinux.eu  Fri Dec  7 15:19:34 2012
From: holger at merlinux.eu (holger krekel)
Date: Fri, 7 Dec 2012 14:19:34 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxQuz=f+ZCEFEUzBep+49w4203iLxk9f+b8SFuoiygm5RA@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
	<CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
	<CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>
	<50C1F09C.9030508@gmx.de>
	<CAK5idxQuz=f+ZCEFEUzBep+49w4203iLxk9f+b8SFuoiygm5RA@mail.gmail.com>
Message-ID: <20121207141934.GU26599@merlinux.eu>

On Fri, Dec 07, 2012 at 05:58 -0800, Maciej Fijalkowski wrote:
> On Fri, Dec 7, 2012 at 5:35 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> > On 12/07/2012 08:58 AM, Maciej Fijalkowski wrote:
> >>
> >> On Thu, Dec 6, 2012 at 10:47 AM, Alex Gaynor <alex.gaynor at gmail.com>
> >> wrote:
> >>>
> >>> I have a slight preference for the PSF servers if it's possible, I'll get
> >>> in
> >>> touch with Noah and see what I can do to help move this along.
> >>>
> >>> Alex
> >>
> >>
> >> I have serious preference for servers where people having physical
> >> access can respond in a timely fashion, which are not the PSF servers.
> >
> >
> > I tend to agree with this point of view, but don't have a strong
> > preference.
> >
> > Cheers,
> >
> > Carl Friedrich
> >
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> 
> for what is worth both pypy.org and bugs are down ATM and I cannot log
> into machines.

Are you sure it's not something on your side?  I just logged in
and uptime is fine; I also had a ssh-session/screen connected to
it for the last couple of days and there was no disconnect.

holger

From fijall at gmail.com  Fri Dec  7 15:26:02 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 7 Dec 2012 06:26:02 -0800
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121207141934.GU26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<CAK5idxS_Z355Qe8+GD0w6Bz6SareBB5UmWki-YJgcFzAeMOYkg@mail.gmail.com>
	<20121206162607.GR26599@merlinux.eu>
	<CAK5idxQ-ZH5+r3dgO+f6jpgLAp-PyLhLZPDjW0pyQ+=un5aFpw@mail.gmail.com>
	<CAMSv6X3A6VCZ4EhEy0vaN8c_OSRu6wUhCqNwj_ApkrCoXKyUSw@mail.gmail.com>
	<50C0D1D8.6000808@gmail.com>
	<CAFRnB2XgiohBNeEfN+v92cu5BAtvZd8S=rE9+K_fdA1az+gvyw@mail.gmail.com>
	<CAK5idxTejhQSY=6qpCqd0O5W+xtmPsF1azaU8dYbK=8ZD2fNBA@mail.gmail.com>
	<50C1F09C.9030508@gmx.de>
	<CAK5idxQuz=f+ZCEFEUzBep+49w4203iLxk9f+b8SFuoiygm5RA@mail.gmail.com>
	<20121207141934.GU26599@merlinux.eu>
Message-ID: <CAK5idxScSX8PDgDX45PwQn6N6EidPb3CZJLA18cv40Df1U-Pcg@mail.gmail.com>

On Fri, Dec 7, 2012 at 6:19 AM, holger krekel <holger at merlinux.eu> wrote:
> On Fri, Dec 07, 2012 at 05:58 -0800, Maciej Fijalkowski wrote:
>> On Fri, Dec 7, 2012 at 5:35 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> > On 12/07/2012 08:58 AM, Maciej Fijalkowski wrote:
>> >>
>> >> On Thu, Dec 6, 2012 at 10:47 AM, Alex Gaynor <alex.gaynor at gmail.com>
>> >> wrote:
>> >>>
>> >>> I have a slight preference for the PSF servers if it's possible, I'll get
>> >>> in
>> >>> touch with Noah and see what I can do to help move this along.
>> >>>
>> >>> Alex
>> >>
>> >>
>> >> I have serious preference for servers where people having physical
>> >> access can respond in a timely fashion, which are not the PSF servers.
>> >
>> >
>> > I tend to agree with this point of view, but don't have a strong
>> > preference.
>> >
>> > Cheers,
>> >
>> > Carl Friedrich
>> >
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > http://mail.python.org/mailman/listinfo/pypy-dev
>>
>> for what is worth both pypy.org and bugs are down ATM and I cannot log
>> into machines.
>
> Are you sure it's not something on your side?  I just logged in
> and uptime is fine; I also had a ssh-session/screen connected to
> it for the last couple of days and there was no disconnect.
>
> holger

It seems there are some routing problems (it's on *my side* for some
definition of my side, but it's definitely not my computer or my home
network).

Cheers,
fijal

From sarabas at igf.fuw.edu.pl  Mon Dec 10 22:02:43 2012
From: sarabas at igf.fuw.edu.pl (Sylwester Arabas)
Date: Mon, 10 Dec 2012 22:02:43 +0100
Subject: [pypy-dev] FOSS for scientists devroom at FOSDEM 2013
Message-ID: <50C64DF3.30707@igf.fuw.edu.pl>

Dear PyPy Team,

A day-long session ("devroom") on Free/Libre and Open Source Software 
(FLOSS) for scientists will be held during the next FOSDEM conference, 
Brussels, 2-3 February 2013 (http://fosdem.org/2013).

We aim at having a dozen or two short talks introducing projects, 
advertising brand new features of established tools, discussing issues 
relevant to the development of software for scientific computing, and 
touching on the interdependence of FLOSS and open science.

You can find more info on the call for talks at:
http://slayoo.github.com/fosdem2013/

The deadline for sending talk proposals is December 16th 2012.

Please send your submissions or comments to:
foss4scientists-devroom at lists.fosdem.org

Please do forward this message to anyone potentially interested.  Please 
also let us know if you have any suggestions for what would you like to 
hear about in the devroom.

Looking forward to meeting you in Brussels.
Thanks in advance.

The conveners,
Sylwester Arabas, Juan Antonio A?el, Christos Siopis

-- 
http://www.igf.fuw.edu.pl/~slayoo/

From estama at gmail.com  Tue Dec 11 18:00:42 2012
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Tue, 11 Dec 2012 19:00:42 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
Message-ID: <50C766BA.3080901@gmail.com>

Hello,

We have been testing CFFI here for the purpose of speeding up madIS [*], 
and here are some preliminary results.

First of all, under pypy, CFFI is a *lot* faster than ctypes. In 
callback microbenchmarks (using quicksort to call the callbacks), 
pypy/CFFI had ~8-10 times less overhead than pypy/ctypes.

Using libsqlite3 as the C library calling the callbacks, we found that, 
compared to Python/APSW [**] the callback overhead of pypy/CFFI was 3-4 
times larger.

To be able to run madIS, we started from pypy's _sqlite3.py (which uses 
ctypes) and did a rough conversion of it to CFFI. We then changed it to 
be APSW compatible. In the rest of the email, we'll refer to this APSW 
compatible CFFI wrapper, as MSPW.

The end results were very encouraging. Using madIS and for SQL queries 
that didn't produce a lot of columns, we could get speedups up to 2 times.

For SQL queries that produce a lot of columns, we get slowdowns of 2 - 
2.5 times. Running a profiler on pypy/madIS, we found out that the main 
bottleneck is not callback performance any more, but it is regular 
pypy->C call speed.

Whereas Python/APSW spends most of its time on madIS' Python execution 
of functions. Pypy/MSPW spends most of its time (~45-50%) on calling a 
libsqlite3 function that passes a Python string back to libsqlite3, and 
10% in pypy's string.encode('utf-8') function.

So for pypy/MSPW most of the time (55-60%) is spend just to pass a 
string back to libsqlite3.

In Python/APSW the time spended to pass the string back to libsqlite3 is 
<1%.

The libsqlite3 function's header info is:

void sqlite3_result_text(sqlite3_context*, const char*, int, 
void(*)(void*));

In the main query that we've used in our tests, above libsqlite3 
function is called 1.1 Million times.

The times of this query, running under the different options are:

  Python/APSW: 40s
  pypy/MSPW: 2m 3s
  pypy/APSW: 2m 21s

Best regards,

l.

[*] https://code.google.com/p/madis/
[*] https://code.google.com/p/apsw/

From fijall at gmail.com  Tue Dec 11 18:48:19 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 11 Dec 2012 19:48:19 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50C766BA.3080901@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
Message-ID: <CAK5idxQmmjaett65isgzDt8wTUpzYW8kgLUWKYweAeRcGNrjoQ@mail.gmail.com>

On Tue, Dec 11, 2012 at 7:00 PM, Eleytherios Stamatogiannakis
<estama at gmail.com> wrote:
> Hello,
>
> We have been testing CFFI here for the purpose of speeding up madIS [*], and
> here are some preliminary results.
>
> First of all, under pypy, CFFI is a *lot* faster than ctypes. In callback
> microbenchmarks (using quicksort to call the callbacks), pypy/CFFI had ~8-10
> times less overhead than pypy/ctypes.
>
> Using libsqlite3 as the C library calling the callbacks, we found that,
> compared to Python/APSW [**] the callback overhead of pypy/CFFI was 3-4
> times larger.
>
> To be able to run madIS, we started from pypy's _sqlite3.py (which uses
> ctypes) and did a rough conversion of it to CFFI. We then changed it to be
> APSW compatible. In the rest of the email, we'll refer to this APSW
> compatible CFFI wrapper, as MSPW.
>
> The end results were very encouraging. Using madIS and for SQL queries that
> didn't produce a lot of columns, we could get speedups up to 2 times.
>
> For SQL queries that produce a lot of columns, we get slowdowns of 2 - 2.5
> times. Running a profiler on pypy/madIS, we found out that the main
> bottleneck is not callback performance any more, but it is regular pypy->C
> call speed.
>
> Whereas Python/APSW spends most of its time on madIS' Python execution of
> functions. Pypy/MSPW spends most of its time (~45-50%) on calling a
> libsqlite3 function that passes a Python string back to libsqlite3, and 10%
> in pypy's string.encode('utf-8') function.
>
> So for pypy/MSPW most of the time (55-60%) is spend just to pass a string
> back to libsqlite3.
>
> In Python/APSW the time spended to pass the string back to libsqlite3 is
> <1%.
>
> The libsqlite3 function's header info is:
>
> void sqlite3_result_text(sqlite3_context*, const char*, int,
> void(*)(void*));
>
> In the main query that we've used in our tests, above libsqlite3 function is
> called 1.1 Million times.
>
> The times of this query, running under the different options are:
>
>  Python/APSW: 40s
>  pypy/MSPW: 2m 3s
>  pypy/APSW: 2m 21s
>
> Best regards,
>
> l.
>
> [*] https://code.google.com/p/madis/
> [*] https://code.google.com/p/apsw/
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hi

Quick question - can you post your benchmarks somewhere so I can try
them? (I'll answer the rest of your mail separately)

Cheers,
fijal

From cfbolz at gmx.de  Wed Dec 12 14:03:04 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 12 Dec 2012 14:03:04 +0100
Subject: [pypy-dev] Raymond's proposed new dict layout
Message-ID: <50C88088.4010701@gmx.de>

Hi all,

I've been wondering whether the recipe that Raymond Hettinger posted
would also solve PyPy's performance problems with making huge
dictionaries:

http://code.activestate.com/recipes/578375/

The proposed layout makes the pointer-storing part of the dictionary
behave much more like a list. In particular, the write pattern is much
more regular, so the card-marking techniques we use for lists should be
applicable.

Any thoughts?

Cheers,

Carl Friedrich

From arigo at tunes.org  Wed Dec 12 14:26:14 2012
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 12 Dec 2012 14:26:14 +0100
Subject: [pypy-dev] Raymond's proposed new dict layout
In-Reply-To: <50C88088.4010701@gmx.de>
References: <50C88088.4010701@gmx.de>
Message-ID: <CAMSv6X2Tbjgm3suhDQuP41wCPAbnTQz6g3GjoNr1wf5ZOoMH-Q@mail.gmail.com>

Hi Carl Friedrich,

On Wed, Dec 12, 2012 at 2:03 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> I've been wondering whether the recipe that Raymond Hettinger posted
> would also solve PyPy's performance problems with making huge
> dictionaries:

Yes, we've been wondering the same thing on IRC :-)

Armin

From cfbolz at gmx.de  Wed Dec 12 15:00:50 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 12 Dec 2012 15:00:50 +0100
Subject: [pypy-dev] Raymond's proposed new dict layout
In-Reply-To: <CAMSv6X2Tbjgm3suhDQuP41wCPAbnTQz6g3GjoNr1wf5ZOoMH-Q@mail.gmail.com>
References: <50C88088.4010701@gmx.de>
	<CAMSv6X2Tbjgm3suhDQuP41wCPAbnTQz6g3GjoNr1wf5ZOoMH-Q@mail.gmail.com>
Message-ID: <50C88E12.1030408@gmx.de>

On 12/12/2012 02:26 PM, Armin Rigo wrote:
> Hi Carl Friedrich,
>
> On Wed, Dec 12, 2012 at 2:03 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> I've been wondering whether the recipe that Raymond Hettinger posted
>> would also solve PyPy's performance problems with making huge
>> dictionaries:
>
> Yes, we've been wondering the same thing on IRC :-)

Ah, cool. Guess I should join the channel again :-). When was that?
Couldn't find the logs.

Carl Friedrich



From arigo at tunes.org  Wed Dec 12 15:11:57 2012
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 12 Dec 2012 15:11:57 +0100
Subject: [pypy-dev] Raymond's proposed new dict layout
In-Reply-To: <50C88E12.1030408@gmx.de>
References: <50C88088.4010701@gmx.de>
	<CAMSv6X2Tbjgm3suhDQuP41wCPAbnTQz6g3GjoNr1wf5ZOoMH-Q@mail.gmail.com>
	<50C88E12.1030408@gmx.de>
Message-ID: <CAMSv6X1-+E6_UuA3nSKR4h7kCwSekNs-4vfQb5sthK+RNFCkOw@mail.gmail.com>

Hi Carl Friedrich,

On Wed, Dec 12, 2012 at 3:00 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Ah, cool. Guess I should join the channel again :-). When was that?
> Couldn't find the logs.

The 10th, 17:31 (http://www.tismer.com/pypy/irc-logs/pypy/pypy.2012-12-10.log).


A bient?t,

Armin.

From cfbolz at gmx.de  Wed Dec 12 15:27:52 2012
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 12 Dec 2012 15:27:52 +0100
Subject: [pypy-dev] Raymond's proposed new dict layout
In-Reply-To: <CAMSv6X1-+E6_UuA3nSKR4h7kCwSekNs-4vfQb5sthK+RNFCkOw@mail.gmail.com>
References: <50C88088.4010701@gmx.de>
	<CAMSv6X2Tbjgm3suhDQuP41wCPAbnTQz6g3GjoNr1wf5ZOoMH-Q@mail.gmail.com>
	<50C88E12.1030408@gmx.de>
	<CAMSv6X1-+E6_UuA3nSKR4h7kCwSekNs-4vfQb5sthK+RNFCkOw@mail.gmail.com>
Message-ID: <50C89468.6000306@gmx.de>

On 12/12/2012 03:11 PM, Armin Rigo wrote:
> Hi Carl Friedrich,
>
> On Wed, Dec 12, 2012 at 3:00 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> Ah, cool. Guess I should join the channel again :-). When was that?
>> Couldn't find the logs.
>
> The 10th, 17:31 (http://www.tismer.com/pypy/irc-logs/pypy/pypy.2012-12-10.log).

Thanks. I guess what would be needed is an implementation to be able to
measure the differences. I don't think it possible to find out by
arguing whether one is inherently better. Maybe over Christmas :-).

Cheers,

Carl Friedrich


From joehillen at gmail.com  Wed Dec 12 18:06:14 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Wed, 12 Dec 2012 09:06:14 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
Message-ID: <CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>

I was able to fix the issue with scrapy.

https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c

Unfortunately, scrapy takes twice as long in PyPy than in CPython. I
suspect this is because lxml is twice as slow in PyPy vs CPython, which I
found in lxml's benchmarks.

Should lxml be added to the set of speed tests?


On Thu, Dec 6, 2012 at 12:34 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Wed, Dec 5, 2012 at 3:40 PM, Joe Hillenbrand <joehillen at gmail.com>
> wrote:
> > I've found a place where PyPy and CPython disagree.
> >
> > https://gist.github.com/4220533
> >
> > This might not be the only issue, but it's the first thing I've found so
> > far.
> >
> > -Joe
>
> This is a well known issue - PyPy's methods and builtin methods are
> not that different. We can probably fix it though
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121212/44d42a97/attachment.html>

From fijall at gmail.com  Wed Dec 12 20:10:55 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 12 Dec 2012 21:10:55 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
Message-ID: <CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>

On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
> I was able to fix the issue with scrapy.
>
> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>
> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
> this is because lxml is twice as slow in PyPy vs CPython, which I found in
> lxml's benchmarks.
>
> Should lxml be added to the set of speed tests?

no. lxml uses cpyext (CPython extension compatibility) that is and
will forever be slow.

From estama at gmail.com  Wed Dec 12 22:00:41 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Wed, 12 Dec 2012 23:00:41 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxQmmjaett65isgzDt8wTUpzYW8kgLUWKYweAeRcGNrjoQ@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAK5idxQmmjaett65isgzDt8wTUpzYW8kgLUWKYweAeRcGNrjoQ@mail.gmail.com>
Message-ID: <50C8F079.3080307@gmail.com>

We cannot publish the benchmark that we used due to its usage of non 
public data.

We'll prepare a benchmark which uses synthetic data, nevertheless it'll 
have similar performance characteristics to the previous one, and we'll 
provide it to you.

Many many thanks,

l.

On 11/12/2012 7:48 ??, Maciej Fijalkowski wrote:
> Hi
>
> Quick question - can you post your benchmarks somewhere so I can try
> them? (I'll answer the rest of your mail separately)
>
> Cheers,
> fijal
>


From stefan_ml at behnel.de  Thu Dec 13 08:35:24 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 13 Dec 2012 08:35:24 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
Message-ID: <kac0fq$s7p$1@ger.gmane.org>

Maciej Fijalkowski, 12.12.2012 20:10:
> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
>> I was able to fix the issue with scrapy.
>>
>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>>
>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
>> this is because lxml is twice as slow in PyPy vs CPython, which I found in
>> lxml's benchmarks.
>>
>> Should lxml be added to the set of speed tests?
> 
> no. lxml uses cpyext (CPython extension compatibility) that is and
> will forever be slow.

Well, I don't think it would be hard for any PyPy core developer to make it
twice as fast. Shouldn't be more than a day's work.

Stefan



From alex.gaynor at gmail.com  Thu Dec 13 08:43:11 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 12 Dec 2012 23:43:11 -0800
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kac0fq$s7p$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
Message-ID: <CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>

Out of curiosity Stefan, if we had an alternate C-API with similar methods
(e.g. PyPyList_Append or so), but different signatures and memory model,
how hard do you think it would be to have Cython support this?

Alex


On Wed, Dec 12, 2012 at 11:35 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Maciej Fijalkowski, 12.12.2012 20:10:
> > On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
> >> I was able to fix the issue with scrapy.
> >>
> >>
> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
> >>
> >> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I
> suspect
> >> this is because lxml is twice as slow in PyPy vs CPython, which I found
> in
> >> lxml's benchmarks.
> >>
> >> Should lxml be added to the set of speed tests?
> >
> > no. lxml uses cpyext (CPython extension compatibility) that is and
> > will forever be slow.
>
> Well, I don't think it would be hard for any PyPy core developer to make it
> twice as fast. Shouldn't be more than a day's work.
>
> Stefan
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121212/e6e0f5d4/attachment.html>

From fijall at gmail.com  Thu Dec 13 09:11:14 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 13 Dec 2012 10:11:14 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50C8F079.3080307@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAK5idxQmmjaett65isgzDt8wTUpzYW8kgLUWKYweAeRcGNrjoQ@mail.gmail.com>
	<50C8F079.3080307@gmail.com>
Message-ID: <CAK5idxSG3+jzawcbUe7vQjNZ882pZ4yF76pSwXmz552c0BGk2A@mail.gmail.com>

On Wed, Dec 12, 2012 at 11:00 PM, Elefterios Stamatogiannakis
<estama at gmail.com> wrote:
> We cannot publish the benchmark that we used due to its usage of non public
> data.
>
> We'll prepare a benchmark which uses synthetic data, nevertheless it'll have
> similar performance characteristics to the previous one, and we'll provide
> it to you.
>
> Many many thanks,

Thanks!

>
> l.
>
>
> On 11/12/2012 7:48 ??, Maciej Fijalkowski wrote:
>>
>> Hi
>>
>> Quick question - can you post your benchmarks somewhere so I can try
>> them? (I'll answer the rest of your mail separately)
>>
>> Cheers,
>> fijal
>>
>

From fijall at gmail.com  Thu Dec 13 09:13:30 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 13 Dec 2012 10:13:30 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kac0fq$s7p$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
Message-ID: <CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>

On Thu, Dec 13, 2012 at 9:35 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Maciej Fijalkowski, 12.12.2012 20:10:
>> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
>>> I was able to fix the issue with scrapy.
>>>
>>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>>>
>>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
>>> this is because lxml is twice as slow in PyPy vs CPython, which I found in
>>> lxml's benchmarks.
>>>
>>> Should lxml be added to the set of speed tests?
>>
>> no. lxml uses cpyext (CPython extension compatibility) that is and
>> will forever be slow.
>
> Well, I don't think it would be hard for any PyPy core developer to make it
> twice as fast. Shouldn't be more than a day's work.
>
> Stefan

I'm not so sure, we wouldn't know until someone tries it. What
optimizations did you have in mind?

For what is worth, cpyext is not twice as slow, lxml is. cpyext is
likely 10-20x slower or so. I presume lowering the overhead would not
automatically make lxml twice as fast, since it's doing quite a lot of
other work.

Anyway, without trying we don't really know

Cheers,
fijal

From arigo at tunes.org  Thu Dec 13 11:54:19 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 13 Dec 2012 11:54:19 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50C766BA.3080901@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
Message-ID: <CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>

Hi,

On Tue, Dec 11, 2012 at 6:00 PM, Eleytherios Stamatogiannakis
<estama at gmail.com> wrote:
>  Python/APSW: 40s
>  pypy/MSPW: 2m 3s
>  pypy/APSW: 2m 21s

Not horribly bad, given that we're comparing with APSW, which is a
piece of C code: PyPy makes Python only 3 times slower than
hand-crafted C.

That was just a general comment; there might be more precise issues
that can still be addressed (passing strings around, and callbacks,
are two known sub-efficient things).


A bient?t,

Armin.

From stefan_ml at behnel.de  Thu Dec 13 18:21:55 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 13 Dec 2012 18:21:55 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
Message-ID: <kad2rh$vrg$1@ger.gmane.org>

Maciej Fijalkowski, 13.12.2012 09:13:
> On Thu, Dec 13, 2012 at 9:35 AM, Stefan Behnel wrote:
>> Maciej Fijalkowski, 12.12.2012 20:10:
>>> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
>>>> I was able to fix the issue with scrapy.
>>>>
>>>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>>>>
>>>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
>>>> this is because lxml is twice as slow in PyPy vs CPython, which I found in
>>>> lxml's benchmarks.
>>>>
>>>> Should lxml be added to the set of speed tests?
>>>
>>> no. lxml uses cpyext (CPython extension compatibility) that is and
>>> will forever be slow.
>>
>> Well, I don't think it would be hard for any PyPy core developer to make it
>> twice as fast. Shouldn't be more than a day's work.
> 
> I'm not so sure, we wouldn't know until someone tries it. What
> optimizations did you have in mind?

Anything that creates a proper fast-path in the ref-counting functions and
that generally takes pressure off them, e.g. by keeping PyObjects alive in
a weakref dict as long as the corresponding PyPy object lives, so that
useless re-allocation cycles are avoided. I'm sure that really simple
changes can bring a substantial improvement here.


> For what is worth, cpyext is not twice as slow, lxml is. cpyext is
> likely 10-20x slower or so. I presume lowering the overhead would not
> automatically make lxml twice as fast, since it's doing quite a lot of
> other work.

lxml's API performance suffers a lot from object/reference creation and
deallocation time, so making object deallocation faster and making it
happen only when necessary would certainly improve the overall performance.

Stefan



From fijall at gmail.com  Thu Dec 13 19:23:47 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 13 Dec 2012 20:23:47 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kad2rh$vrg$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
Message-ID: <CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>

On Thu, Dec 13, 2012 at 7:21 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Maciej Fijalkowski, 13.12.2012 09:13:
>> On Thu, Dec 13, 2012 at 9:35 AM, Stefan Behnel wrote:
>>> Maciej Fijalkowski, 12.12.2012 20:10:
>>>> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
>>>>> I was able to fix the issue with scrapy.
>>>>>
>>>>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>>>>>
>>>>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
>>>>> this is because lxml is twice as slow in PyPy vs CPython, which I found in
>>>>> lxml's benchmarks.
>>>>>
>>>>> Should lxml be added to the set of speed tests?
>>>>
>>>> no. lxml uses cpyext (CPython extension compatibility) that is and
>>>> will forever be slow.
>>>
>>> Well, I don't think it would be hard for any PyPy core developer to make it
>>> twice as fast. Shouldn't be more than a day's work.
>>
>> I'm not so sure, we wouldn't know until someone tries it. What
>> optimizations did you have in mind?
>
> Anything that creates a proper fast-path in the ref-counting functions and
> that generally takes pressure off them, e.g. by keeping PyObjects alive in
> a weakref dict as long as the corresponding PyPy object lives, so that
> useless re-allocation cycles are avoided. I'm sure that really simple
> changes can bring a substantial improvement here.

short term allocations are usually very cheap. dictionaries lookups
not necesarilly so. Do you have any specific optimizations in mind? I
don't see any easy way of doing it all.

>
>
>> For what is worth, cpyext is not twice as slow, lxml is. cpyext is
>> likely 10-20x slower or so. I presume lowering the overhead would not
>> automatically make lxml twice as fast, since it's doing quite a lot of
>> other work.
>
> lxml's API performance suffers a lot from object/reference creation and
> deallocation time, so making object deallocation faster and making it
> happen only when necessary would certainly improve the overall performance.
>
> Stefan
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From stefan_ml at behnel.de  Thu Dec 13 21:31:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 13 Dec 2012 21:31:51 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
	<CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
Message-ID: <kaddvl$cdf$1@ger.gmane.org>

Maciej Fijalkowski, 13.12.2012 19:23:
> On Thu, Dec 13, 2012 at 7:21 PM, Stefan Behnel wrote:
>> Maciej Fijalkowski, 13.12.2012 09:13:
>>> On Thu, Dec 13, 2012 at 9:35 AM, Stefan Behnel wrote:
>>>> Maciej Fijalkowski, 12.12.2012 20:10:
>>>>> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote:
>>>>>> I was able to fix the issue with scrapy.
>>>>>>
>>>>>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c
>>>>>>
>>>>>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I suspect
>>>>>> this is because lxml is twice as slow in PyPy vs CPython, which I found in
>>>>>> lxml's benchmarks.
>>>>>>
>>>>>> Should lxml be added to the set of speed tests?
>>>>>
>>>>> no. lxml uses cpyext (CPython extension compatibility) that is and
>>>>> will forever be slow.
>>>>
>>>> Well, I don't think it would be hard for any PyPy core developer to make it
>>>> twice as fast. Shouldn't be more than a day's work.
>>>
>>> I'm not so sure, we wouldn't know until someone tries it. What
>>> optimizations did you have in mind?
>>
>> Anything that creates a proper fast-path in the ref-counting functions and
>> that generally takes pressure off them, e.g. by keeping PyObjects alive in
>> a weakref dict as long as the corresponding PyPy object lives, so that
>> useless re-allocation cycles are avoided. I'm sure that really simple
>> changes can bring a substantial improvement here.
> 
> short term allocations are usually very cheap.

In the profile I posted the last time we discussed this, I think it was
pretty clear that most of the time is not currently being spent in the
allocation but in the deallocation. More than 50% of the overall runtime in
this case:

http://cython.org/callgrind-pypy-nbody.png

Plus, connecting the lifetime of PyObjects to that of PyPy objects would
fix the problem that PyObjects can die prematurely and take C state with them:

http://docs.cython.org/src/userguide/pypy.html

My intuition was to add a fastpath to Py_DECREF() that would do (close to)
nothing if the PyPy object is still alive. Either that, or move this whole
decision into C by somehow increasing the C level refcount during the
lifetime of the PyPy object and decreasing it when the PyPy object dies.
The latter approach (if doable) is obviously preferable from a C point of
view because it would improve the hit-count of the "common case" tests in
the INCREF/DECREF C macros, thus avoiding unnecessary calls into PyPy all
together by using inlined code. That would give it about the same speed as
in CPython for objects that are being reused in C code more than once for
which a PyPy object reference exists (certainly not an unusual case).

Stefan


From amauryfa at gmail.com  Thu Dec 13 21:47:08 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 13 Dec 2012 21:47:08 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kaddvl$cdf$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
	<CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
	<kaddvl$cdf$1@ger.gmane.org>
Message-ID: <CAGmFidboq+5E_3aUGCgnoya2xpk2vtaCeiX6+Vt7tdNCrFwEmQ@mail.gmail.com>

2012/12/13 Stefan Behnel <stefan_ml at behnel.de>

> My intuition was to add a fastpath to Py_DECREF() that would do (close to)
> nothing if the PyPy object is still alive. Either that, or move this whole
> decision into C by somehow increasing the C level refcount during the
> lifetime of the PyPy object and decreasing it when the PyPy object dies.
>

It may be difficult, because most standard types don't have a __del__,
and I'm not sure we can attach a weak reference.


> The latter approach (if doable) is obviously preferable from a C point of
> view because it would improve the hit-count of the "common case" tests in
> the INCREF/DECREF C macros, thus avoiding unnecessary calls into PyPy all
> together by using inlined code. That would give it about the same speed as
> in CPython for objects that are being reused in C code more than once for
> which a PyPy object reference exists (certainly not an unusual case).
>



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121213/451ec33f/attachment.html>

From fijall at gmail.com  Thu Dec 13 22:21:17 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 13 Dec 2012 23:21:17 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAGmFidboq+5E_3aUGCgnoya2xpk2vtaCeiX6+Vt7tdNCrFwEmQ@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
	<CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
	<kaddvl$cdf$1@ger.gmane.org>
	<CAGmFidboq+5E_3aUGCgnoya2xpk2vtaCeiX6+Vt7tdNCrFwEmQ@mail.gmail.com>
Message-ID: <CAK5idxQ2CcmLYUzuEq_sCn5mnPXE3fd_Q-54gsphZmUfjor5-A@mail.gmail.com>

On Thu, Dec 13, 2012 at 10:47 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2012/12/13 Stefan Behnel <stefan_ml at behnel.de>
>>
>> My intuition was to add a fastpath to Py_DECREF() that would do (close to)
>> nothing if the PyPy object is still alive. Either that, or move this whole
>> decision into C by somehow increasing the C level refcount during the
>> lifetime of the PyPy object and decreasing it when the PyPy object dies.
>
>
> It may be difficult, because most standard types don't have a __del__,
> and I'm not sure we can attach a weak reference.

having tons of weakrefs is also a bad idea (or tons of objects with __del__s)

>
>>
>> The latter approach (if doable) is obviously preferable from a C point of
>> view because it would improve the hit-count of the "common case" tests in
>> the INCREF/DECREF C macros, thus avoiding unnecessary calls into PyPy all
>> together by using inlined code. That would give it about the same speed as
>> in CPython for objects that are being reused in C code more than once for
>> which a PyPy object reference exists (certainly not an unusual case).
>
>
>
>
> --
> Amaury Forgeot d'Arc
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From stefan_ml at behnel.de  Fri Dec 14 06:48:59 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 14 Dec 2012 06:48:59 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxQ2CcmLYUzuEq_sCn5mnPXE3fd_Q-54gsphZmUfjor5-A@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
	<CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
	<kaddvl$cdf$1@ger.gmane.org>
	<CAGmFidboq+5E_3aUGCgnoya2xpk2vtaCeiX6+Vt7tdNCrFwEmQ@mail.gmail.com>
	<CAK5idxQ2CcmLYUzuEq_sCn5mnPXE3fd_Q-54gsphZmUfjor5-A@mail.gmail.com>
Message-ID: <kaeek9$c6n$1@ger.gmane.org>

Maciej Fijalkowski, 13.12.2012 22:21:
> On Thu, Dec 13, 2012 at 10:47 PM, Amaury Forgeot d'Arc wrote:
>> 2012/12/13 Stefan Behnel
>>>
>>> My intuition was to add a fastpath to Py_DECREF() that would do (close to)
>>> nothing if the PyPy object is still alive. Either that, or move this whole
>>> decision into C by somehow increasing the C level refcount during the
>>> lifetime of the PyPy object and decreasing it when the PyPy object dies.
>>
>> It may be difficult, because most standard types don't have a __del__,
>> and I'm not sure we can attach a weak reference.

If you can't attach a weakref to an object, that sounds more like a bug to
me, especially in PyPy.


> having tons of weakrefs is also a bad idea (or tons of objects with __del__s)

So, I take it that it's a tradeoff - not unusual for optimisation. Why not
just give it a try?

Stefan



From amauryfa at gmail.com  Fri Dec 14 13:37:02 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 14 Dec 2012 13:37:02 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kaeek9$c6n$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAK5idxTaHoeTg-KL8+m_Fb=aAtf_M3khhUaBvkQj4q_P8i8ORA@mail.gmail.com>
	<kad2rh$vrg$1@ger.gmane.org>
	<CAK5idxStXNzSoSa4KkXFkHL_u_NfX4nF+XuDEbotC38vXbqrQA@mail.gmail.com>
	<kaddvl$cdf$1@ger.gmane.org>
	<CAGmFidboq+5E_3aUGCgnoya2xpk2vtaCeiX6+Vt7tdNCrFwEmQ@mail.gmail.com>
	<CAK5idxQ2CcmLYUzuEq_sCn5mnPXE3fd_Q-54gsphZmUfjor5-A@mail.gmail.com>
	<kaeek9$c6n$1@ger.gmane.org>
Message-ID: <CAGmFidZZ+4HwmkbyjGwX0DrgnUeeFkf48E8KHX+kL-KCqNCnKA@mail.gmail.com>

2012/12/14 Stefan Behnel <stefan_ml at behnel.de>

> Maciej Fijalkowski, 13.12.2012 22:21:
> > On Thu, Dec 13, 2012 at 10:47 PM, Amaury Forgeot d'Arc wrote:
> >> 2012/12/13 Stefan Behnel
> >>>
> >>> My intuition was to add a fastpath to Py_DECREF() that would do (close
> to)
> >>> nothing if the PyPy object is still alive. Either that, or move this
> whole
> >>> decision into C by somehow increasing the C level refcount during the
> >>> lifetime of the PyPy object and decreasing it when the PyPy object
> dies.
> >>
> >> It may be difficult, because most standard types don't have a __del__,
> >> and I'm not sure we can attach a weak reference.
>
> If you can't attach a weakref to an object, that sounds more like a bug to
> me, especially in PyPy.
>

Try this on any Python:
>>> weakref.ref(3)

Supporting weak references is not cheap, especially for short lived objects.
I'm sure you don't want to slow down all strings and numbers that are *not*
passed to extension modules.


> having tons of weakrefs is also a bad idea (or tons of objects with
> __del__s)
>
> So, I take it that it's a tradeoff - not unusual for optimisation. Why not
> just give it a try?



-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121214/a4652b99/attachment.html>

From estama at gmail.com  Fri Dec 14 18:25:35 2012
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Fri, 14 Dec 2012 19:25:35 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
Message-ID: <50CB610F.1010800@gmail.com>

Here is the synthetic benchmark.

To run it you'll need latest madIS. You can clone it using:

hg clone https://code.google.com/p/madis/

For running the test you can use:

CPython 2.7 + APSW:

https://code.google.com/p/apsw/

Or pypy + APSW (you'll need to clone APSW for it to build on pypy):

hg clone https://code.google.com/p/apsw/

Or pypy + MSPW. You'll find attached a "mspw.py". To use it *rename* it 
to "apsw.py" and put it in pypy's "site-packages" directory. For MSPW to 
work in pypy, you'll also need CFFI and "libsqlite3" installed.

Here i should note, that MSPW is in a extremely rough state. Our main 
focus at this time is to find out how fast we can make it go. So right 
now MSPW doesn't do much error checking. Expect segmentation faults if 
you try a query in mterm and something is wrong with the query.

To run the test with pypy/python:

pypy mterm.py < mspw_bench.sql

or

python mterm.py < mspw_bench.sql

For some reason pypy + APSW throws an exception when it finishes running 
mspw_bench, but the timings should be reliable.

The timings of "mspw_bench" that we get are:

CPython 2.7 + APSW: ~ 1.5 sec
pypy + APSW: ~ 9 sec
pypy + MSPW: ~4.5 sec

There are two ways to adjust the processing load of mspw_bench.

One is to change the value in "range(xxxxx)". This will in essence 
create a bigger/smaller "synthetic text". This puts more pressure on 
CPython's/pypy's side.

The other way is to adjust the window size of textwindow(t, xx, xx). 
This puts more pressure on the wrapper (APSW/MSPW) infrastructure 
because it changes the number of columns that CPython/pypy have to send 
to SQLite (they are send on value at a time).

Both Ioannis Foufoulas and me are available to help with the benchmark.

Many many thanks,

lefteris.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpsw.py
Type: text/x-python
Size: 65446 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121214/fb7dcdf7/attachment-0001.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mspw_bench.sql
Type: text/x-sql
Size: 120 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121214/fb7dcdf7/attachment-0001.bin>

From joehillen at gmail.com  Fri Dec 14 19:24:33 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Fri, 14 Dec 2012 10:24:33 -0800
Subject: [pypy-dev] SegFault when running Scrapy tests.
Message-ID: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>

Hi all,

I'm starting a new thread because the last one seems to be on a new topic.

I was just trying out the scrapy unit tests in PyPy and I've run into
some segfaults.

To reproduce, download and install Scrapy:

https://github.com/scrapy/scrapy

Then run this in the tests directory:

/usr/local/pypy/bin/trial test_http_request.py

I haven't found the exact source of the issue yet, but I'm posting
this in case somebody can find it faster.

Thanks,

-Joe

From fijall at gmail.com  Fri Dec 14 20:38:08 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 14 Dec 2012 21:38:08 +0200
Subject: [pypy-dev] SegFault when running Scrapy tests.
In-Reply-To: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>
References: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>
Message-ID: <CAK5idxRKJCcvcVU9dda7zr5b_0yqFOULCFZ5e58t+oWDwkaU1A@mail.gmail.com>

On Fri, Dec 14, 2012 at 8:24 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
> Hi all,
>
> I'm starting a new thread because the last one seems to be on a new topic.
>
> I was just trying out the scrapy unit tests in PyPy and I've run into
> some segfaults.
>
> To reproduce, download and install Scrapy:
>
> https://github.com/scrapy/scrapy
>
> Then run this in the tests directory:
>
> /usr/local/pypy/bin/trial test_http_request.py
>
> I haven't found the exact source of the issue yet, but I'm posting
> this in case somebody can find it faster.
>
> Thanks,
>
> -Joe
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hi

Thanks for the report. I'm running some tests to see what's going on,
however, can you please submit it to bugs.pypy.org so it doesn't get
lost?

For what is worth, it might be related to cpyext and lxml.

From stefan_ml at behnel.de  Fri Dec 14 20:44:31 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 14 Dec 2012 20:44:31 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
Message-ID: <kafvit$u6l$1@ger.gmane.org>

Alex Gaynor, 13.12.2012 08:43:
> Out of curiosity Stefan, if we had an alternate C-API with similar methods
> (e.g. PyPyList_Append or so), but different signatures and memory model,
> how hard do you think it would be to have Cython support this?

Impossible to say in that generality. If it's only about exchanging C
functions, it should be doable, but if it has an impact on Cython's type
system, it might turn into a horrible mess to come up with something that
works in both CPython and PyPy.

Also note that Cython knows a lot about reference counting internally. If
that alternate C-API requires substantial changes to the way references are
maintained in the C code, that would mean some work.

Also note that the amount of Cython code out there that uses explicit C-API
calls for one reason or another is most likely rather large.

All in all, I'm not a fan of that one big revolution that will make
everything beautiful, fast and shiny, but that will never happen, really. I
prefer small steps that make things work.

Stefan



From fijall at gmail.com  Fri Dec 14 21:04:12 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 14 Dec 2012 22:04:12 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kafvit$u6l$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
	<kafvit$u6l$1@ger.gmane.org>
Message-ID: <CAK5idxRJVQE==1NJxyUkaEZBcGsBJ7Af8zr8ok7qCSh-WBsAMA@mail.gmail.com>

On Fri, Dec 14, 2012 at 9:44 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Alex Gaynor, 13.12.2012 08:43:
>> Out of curiosity Stefan, if we had an alternate C-API with similar methods
>> (e.g. PyPyList_Append or so), but different signatures and memory model,
>> how hard do you think it would be to have Cython support this?
>
> Impossible to say in that generality. If it's only about exchanging C
> functions, it should be doable, but if it has an impact on Cython's type
> system, it might turn into a horrible mess to come up with something that
> works in both CPython and PyPy.
>
> Also note that Cython knows a lot about reference counting internally. If
> that alternate C-API requires substantial changes to the way references are
> maintained in the C code, that would mean some work.
>
> Also note that the amount of Cython code out there that uses explicit C-API
> calls for one reason or another is most likely rather large.
>
> All in all, I'm not a fan of that one big revolution that will make
> everything beautiful, fast and shiny, but that will never happen, really. I
> prefer small steps that make things work.
>
> Stefan

I don't want to be a naysayer here, but supporting CPython C API is a
mess. I don't think there is a way to make it nice and shiny (no
matter what) or a way to make incremental improvements that lead
anywhere good.

That said, I agree that exposing a different C API is not solving
much, while adding burden to maintainers of both Cython and PyPy and
I'm generally against the idea.

What can be done is keeping refcounts on python objects and then
growing few fields for keeping the C stuff forever. I can even think
about a scheme that would do it with a bit of a mess. This would
require storing an extra field on all objects. I can think about a
scheme to have this done only when invoking cpyext for the first time.

If we have a special pointer, we can allocate an object in old
generation, that's tied to the original object. It has a refcount,
with 0 means it goes away. Since it's not movable, you can take a
pointer to it and pass it to C. It's also a root, but a special kind
of root where during collection refcount == 0 means it dies away. The
objects have references to each other, so:

* PyPy object keeps the C object alive for the entire lifetime of the
pypy object.

* C object keeps the PyPy object alive as long as it's refcount is not
0 during collection time.

What we get:

* Simple refcounting (can be implemented in C as macros even)

* Lack of dictionaries

What we loose:

* We need to implement it (so time) and it requires a little bit of GC
complications.

Cheers,
fijal

From fijall at gmail.com  Fri Dec 14 21:57:20 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 14 Dec 2012 22:57:20 +0200
Subject: [pypy-dev] SegFault when running Scrapy tests.
In-Reply-To: <CAK5idxRKJCcvcVU9dda7zr5b_0yqFOULCFZ5e58t+oWDwkaU1A@mail.gmail.com>
References: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>
	<CAK5idxRKJCcvcVU9dda7zr5b_0yqFOULCFZ5e58t+oWDwkaU1A@mail.gmail.com>
Message-ID: <CAK5idxSF8ymTpRQ=Ncauf3g5DsA1_9h-gZNq39uYdAC0E_UTaQ@mail.gmail.com>

On Fri, Dec 14, 2012 at 9:38 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Dec 14, 2012 at 8:24 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
>> Hi all,
>>
>> I'm starting a new thread because the last one seems to be on a new topic.
>>
>> I was just trying out the scrapy unit tests in PyPy and I've run into
>> some segfaults.
>>
>> To reproduce, download and install Scrapy:
>>
>> https://github.com/scrapy/scrapy
>>
>> Then run this in the tests directory:
>>
>> /usr/local/pypy/bin/trial test_http_request.py
>>
>> I haven't found the exact source of the issue yet, but I'm posting
>> this in case somebody can find it faster.
>>
>> Thanks,
>>
>> -Joe
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>
> Hi
>
> Thanks for the report. I'm running some tests to see what's going on,
> however, can you please submit it to bugs.pypy.org so it doesn't get
> lost?
>
> For what is worth, it might be related to cpyext and lxml.

Hi Joe

It crashes somewhere in the error recovering code of lxml. Chances are
either cython or lxml do something that pypy is unhappy about.

We're just discussing a medium-term plan how to fight with such issues
(it migth be a missing incref in a place where CPython doesn't care
for example), however for now fixing cython and/or lxml seems like the
only option.

Cheers,
fijal

From joehillen at gmail.com  Fri Dec 14 22:39:20 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Fri, 14 Dec 2012 13:39:20 -0800
Subject: [pypy-dev] SegFault when running Scrapy tests.
In-Reply-To: <CAK5idxSF8ymTpRQ=Ncauf3g5DsA1_9h-gZNq39uYdAC0E_UTaQ@mail.gmail.com>
References: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>
	<CAK5idxRKJCcvcVU9dda7zr5b_0yqFOULCFZ5e58t+oWDwkaU1A@mail.gmail.com>
	<CAK5idxSF8ymTpRQ=Ncauf3g5DsA1_9h-gZNq39uYdAC0E_UTaQ@mail.gmail.com>
Message-ID: <CANJQ5C6kgnD9Go8c++O2hJ_LfUtd2Agseb-NDdrANuC0oGT9Qw@mail.gmail.com>

>
> Hi Joe
>
> It crashes somewhere in the error recovering code of lxml. Chances are
> either cython or lxml do something that pypy is unhappy about.
>
> We're just discussing a medium-term plan how to fight with such issues
> (it migth be a missing incref in a place where CPython doesn't care
> for example), however for now fixing cython and/or lxml seems like the
> only option.
>
> Cheers,
> fijal

Thanks for the quick response on this. I'll definitely write a bug.

From fijall at gmail.com  Fri Dec 14 22:40:46 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 14 Dec 2012 23:40:46 +0200
Subject: [pypy-dev] SegFault when running Scrapy tests.
In-Reply-To: <CANJQ5C6kgnD9Go8c++O2hJ_LfUtd2Agseb-NDdrANuC0oGT9Qw@mail.gmail.com>
References: <CANJQ5C6j5dXt3a=RNxqCDxFt+re3=_FmsV+K2Qm=Pnkzo8ugog@mail.gmail.com>
	<CAK5idxRKJCcvcVU9dda7zr5b_0yqFOULCFZ5e58t+oWDwkaU1A@mail.gmail.com>
	<CAK5idxSF8ymTpRQ=Ncauf3g5DsA1_9h-gZNq39uYdAC0E_UTaQ@mail.gmail.com>
	<CANJQ5C6kgnD9Go8c++O2hJ_LfUtd2Agseb-NDdrANuC0oGT9Qw@mail.gmail.com>
Message-ID: <CAK5idxTgjFvrH81KN6BQ86OeeK=dh9EOy10GZDpGJu8sTpm=tA@mail.gmail.com>

On Fri, Dec 14, 2012 at 11:39 PM, Joe Hillenbrand <joehillen at gmail.com> wrote:
>>
>> Hi Joe
>>
>> It crashes somewhere in the error recovering code of lxml. Chances are
>> either cython or lxml do something that pypy is unhappy about.
>>
>> We're just discussing a medium-term plan how to fight with such issues
>> (it migth be a missing incref in a place where CPython doesn't care
>> for example), however for now fixing cython and/or lxml seems like the
>> only option.
>>
>> Cheers,
>> fijal
>
> Thanks for the quick response on this. I'll definitely write a bug.

That said, it might as well be a cpyext bug, you can't tell until you
identify it. It's just that cpyext has generally less priority than
other stuff (although see the other thread, maybe we can do something
about it).

From fijall at gmail.com  Fri Dec 14 23:00:34 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 Dec 2012 00:00:34 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50CB610F.1010800@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
Message-ID: <CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>

On Fri, Dec 14, 2012 at 7:25 PM, Eleytherios Stamatogiannakis
<estama at gmail.com> wrote:
> Here is the synthetic benchmark.
>
> To run it you'll need latest madIS. You can clone it using:
>
> hg clone https://code.google.com/p/madis/
>
> For running the test you can use:
>
> CPython 2.7 + APSW:
>
> https://code.google.com/p/apsw/
>
> Or pypy + APSW (you'll need to clone APSW for it to build on pypy):
>
> hg clone https://code.google.com/p/apsw/
>
> Or pypy + MSPW. You'll find attached a "mspw.py". To use it *rename* it to
> "apsw.py" and put it in pypy's "site-packages" directory. For MSPW to work
> in pypy, you'll also need CFFI and "libsqlite3" installed.
>
> Here i should note, that MSPW is in a extremely rough state. Our main focus
> at this time is to find out how fast we can make it go. So right now MSPW
> doesn't do much error checking. Expect segmentation faults if you try a
> query in mterm and something is wrong with the query.
>
> To run the test with pypy/python:
>
> pypy mterm.py < mspw_bench.sql
>
> or
>
> python mterm.py < mspw_bench.sql
>
> For some reason pypy + APSW throws an exception when it finishes running
> mspw_bench, but the timings should be reliable.
>
> The timings of "mspw_bench" that we get are:
>
> CPython 2.7 + APSW: ~ 1.5 sec
> pypy + APSW: ~ 9 sec
> pypy + MSPW: ~4.5 sec
>
> There are two ways to adjust the processing load of mspw_bench.
>
> One is to change the value in "range(xxxxx)". This will in essence create a
> bigger/smaller "synthetic text". This puts more pressure on CPython's/pypy's
> side.
>
> The other way is to adjust the window size of textwindow(t, xx, xx). This
> puts more pressure on the wrapper (APSW/MSPW) infrastructure because it
> changes the number of columns that CPython/pypy have to send to SQLite (they
> are send on value at a time).
>
> Both Ioannis Foufoulas and me are available to help with the benchmark.
>
> Many many thanks,
>
> lefteris.
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Hi.

For what is worth roughly 1/3 of the time is spent importing all the
things. This is done in the compilation step in the ASPW, so please
try running the select few times. Another slightly worrying thing is
that a lot of time is spent doing utf8 decoding. Can you explain what
in the SQL statement requires UTF8 conversion?

Cheers,
fijal

From estama at gmail.com  Sat Dec 15 01:56:03 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Sat, 15 Dec 2012 02:56:03 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
Message-ID: <50CBCAA3.7030705@gmail.com>

On 15/12/2012 12:00 ??, Maciej Fijalkowski wrote:
> Hi.
>
> For what is worth roughly 1/3 of the time is spent importing all the
> things. This is done in the compilation step in the ASPW, so please
> try running the select few times. Another slightly worrying thing is
> that a lot of time is spent doing utf8 decoding. Can you explain what
> in the SQL statement requires UTF8 conversion?
>
> Cheers,
> fijal
>

Concerning running the select many times, you are right. We'll try to 
pay attention to it in the future.

This is easily achieved by copying the query in mspw_bench many times, 
or increasing the "range" value so the processing load will be bigger 
compared to all the other things.

When MSPW sends something to SQLite it has to encode it to UTF-8 
(default SQLite character encoding). When it gets it back it has to 
convert it back to Python's unicode.

l.

From estama at gmail.com  Sat Dec 15 02:28:35 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Sat, 15 Dec 2012 03:28:35 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
Message-ID: <50CBD243.3060809@gmail.com>

On 15/12/2012 12:00 ??, Maciej Fijalkowski wrote:
> Hi.
>
> For what is worth roughly 1/3 of the time is spent importing all the
> things. This is done in the compilation step in the ASPW, so please
> try running the select few times. Another slightly worrying thing is
> that a lot of time is spent doing utf8 decoding. Can you explain what
> in the SQL statement requires UTF8 conversion?
>
> Cheers,
> fijal
>

Also i should point out that most of the time we run our tests from 
inside the mterm console (by copy pasting), so the timings are not 
affected by the module import.

And conserning the utf8 conversions, our data are multilingual so we 
need to support utf8 throughout madIS.

Regards,

l.

From fijall at gmail.com  Sat Dec 15 07:51:58 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 Dec 2012 08:51:58 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50CBCAA3.7030705@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
Message-ID: <CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>

On Sat, Dec 15, 2012 at 2:56 AM, Elefterios Stamatogiannakis
<estama at gmail.com> wrote:
> On 15/12/2012 12:00 ??, Maciej Fijalkowski wrote:
>>
>> Hi.
>>
>> For what is worth roughly 1/3 of the time is spent importing all the
>> things. This is done in the compilation step in the ASPW, so please
>> try running the select few times. Another slightly worrying thing is
>> that a lot of time is spent doing utf8 decoding. Can you explain what
>> in the SQL statement requires UTF8 conversion?
>>
>> Cheers,
>> fijal
>>
>
> Concerning running the select many times, you are right. We'll try to pay
> attention to it in the future.
>
> This is easily achieved by copying the query in mspw_bench many times, or
> increasing the "range" value so the processing load will be bigger compared
> to all the other things.
>
> When MSPW sends something to SQLite it has to encode it to UTF-8 (default
> SQLite character encoding). When it gets it back it has to convert it back
> to Python's unicode.
>
> l.

And ASPW does the same right? I understand the general need for UTF8,
I just didn't find it in this particular query.

From arigo at tunes.org  Sat Dec 15 11:27:53 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 15 Dec 2012 11:27:53 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
Message-ID: <CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>

Hi,

On Sat, Dec 15, 2012 at 7:51 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> And ASPW does the same right? I understand the general need for UTF8,
> I just didn't find it in this particular query.

Fwiw, I wonder again if we couldn't have all our unicode strings
internally be UTF8 instead of 2- or 4-bytes strings.  This would mean
a W_UTF8UnicodeObject class that has both a reference to the RPython
string and some optional extra data to make it faster to locate the
n'th character or the total unicode length.  (We discussed it on IRC
some time ago.)


A bient?t,

Armin.

From fijall at gmail.com  Sat Dec 15 12:00:11 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 Dec 2012 13:00:11 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
Message-ID: <CAK5idxTCewY50X5y2i9jOBqNRYbjhyTgCP2fQi3+Y5JVP+zHyw@mail.gmail.com>

On Sat, Dec 15, 2012 at 12:27 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Sat, Dec 15, 2012 at 7:51 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> And ASPW does the same right? I understand the general need for UTF8,
>> I just didn't find it in this particular query.
>
> Fwiw, I wonder again if we couldn't have all our unicode strings
> internally be UTF8 instead of 2- or 4-bytes strings.  This would mean
> a W_UTF8UnicodeObject class that has both a reference to the RPython
> string and some optional extra data to make it faster to locate the
> n'th character or the total unicode length.  (We discussed it on IRC
> some time ago.)
>
>
> A bient?t,
>
> Armin.

Some sort of string strategies? like "we know it's ascii" or so as well?

From arigo at tunes.org  Sat Dec 15 12:08:40 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 15 Dec 2012 12:08:40 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxTCewY50X5y2i9jOBqNRYbjhyTgCP2fQi3+Y5JVP+zHyw@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<CAK5idxTCewY50X5y2i9jOBqNRYbjhyTgCP2fQi3+Y5JVP+zHyw@mail.gmail.com>
Message-ID: <CAMSv6X1Diuc=8BF_dd=rCU=x52oQO0Wb-0jZvYWEEVBOcpNzwg@mail.gmail.com>

Hi,

On Sat, Dec 15, 2012 at 12:00 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Some sort of string strategies? like "we know it's ascii" or so as well?

Something like that, but that "strategy" would be the only one needed.
 We don't need ascii-only nor two-bytes-only strategies (nor, of
course, 4-bytes strategy) if we have the general utf8 strategy, and
even the latin1-only strategy is probably not worth it.  This utf8
strategy is useful in any program that handles unicodes by converting
it from/to utf8, which in the recent years has become the de facto
standard for new programs.


A bient?t,

Armin.

From bokr at oz.net  Sat Dec 15 12:14:04 2012
From: bokr at oz.net (Bengt Richter)
Date: Sat, 15 Dec 2012 12:14:04 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>	<50987D05.1040204@gmx.de>
	<509A999E.3080609@gmail.com>	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>	<50C766BA.3080901@gmail.com>	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>	<50CB610F.1010800@gmail.com>	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>	<50CBCAA3.7030705@gmail.com>	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
Message-ID: <50CC5B7C.4010604@oz.net>

On 12/15/2012 11:27 AM Armin Rigo wrote:
> Hi,
>
> On Sat, Dec 15, 2012 at 7:51 AM, Maciej Fijalkowski<fijall at gmail.com>  wrote:
>> And ASPW does the same right? I understand the general need for UTF8,
>> I just didn't find it in this particular query.
>
> Fwiw, I wonder again if we couldn't have all our unicode strings
> internally be UTF8 instead of 2- or 4-bytes strings.  This would mean
> a W_UTF8UnicodeObject class that has both a reference to the RPython
> string and some optional extra data to make it faster to locate the
> n'th character or the total unicode length.  (We discussed it on IRC
> some time ago.)
>
>
> A bient?t,
>
> Armin.

Since

     >>> for i in range(256): assert chr(i).decode('latin1') == unichr(i)

I wonder whether something could be gained by having an alternative
internal unicode representation in the form of latin1 8-bit byte strings.

ISTM a lot of English speaking and western European locales would hardly
ever need anything else, and generating code to tag and use/transform
alternative representations would be an internal optimization matter.

I suppose some apps could well result in 8, 16, and 32-bit unicodes and utf8
all coexisting under the hood, but only when actually needed.

Regards,
Bengt Richter


From arigo at tunes.org  Sat Dec 15 13:00:22 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 15 Dec 2012 13:00:22 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50CC5B7C.4010604@oz.net>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
Message-ID: <CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>

Hi Bengt,

On Sat, Dec 15, 2012 at 12:14 PM, Bengt Richter <bokr at oz.net> wrote:
> I wonder whether something could be gained by having an alternative
> internal unicode representation in the form of latin1 8-bit byte strings.

This has been already implemented in CPython 3.x, based on earlier
experiments on PyPy.  What has not been tried is to have utf-8.


A bient?t,

Armin.

From estama at gmail.com  Sat Dec 15 13:53:20 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Sat, 15 Dec 2012 14:53:20 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
Message-ID: <50CC72C0.8020704@gmail.com>

On 15/12/2012 8:51 ??, Maciej Fijalkowski wrote:
> On Sat, Dec 15, 2012 at 2:56 AM, Elefterios Stamatogiannakis
> <estama at gmail.com> wrote:
>> On 15/12/2012 12:00 ??, Maciej Fijalkowski wrote:
>>>
>>> Hi.
>>>
>>> For what is worth roughly 1/3 of the time is spent importing all the
>>> things. This is done in the compilation step in the ASPW, so please
>>> try running the select few times. Another slightly worrying thing is
>>> that a lot of time is spent doing utf8 decoding. Can you explain what
>>> in the SQL statement requires UTF8 conversion?
>>>
>>> Cheers,
>>> fijal
>>>
>>
>> Concerning running the select many times, you are right. We'll try to pay
>> attention to it in the future.
>>
>> This is easily achieved by copying the query in mspw_bench many times, or
>> increasing the "range" value so the processing load will be bigger compared
>> to all the other things.
>>
>> When MSPW sends something to SQLite it has to encode it to UTF-8 (default
>> SQLite character encoding). When it gets it back it has to convert it back
>> to Python's unicode.
>>
>> l.
>
> And ASPW does the same right? I understand the general need for UTF8,
> I just didn't find it in this particular query.
>

Yes regarding UTF8, ASPW does the same.

l.

From santagada at gmail.com  Mon Dec 17 14:20:39 2012
From: santagada at gmail.com (Leonardo Santagada)
Date: Mon, 17 Dec 2012 11:20:39 -0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <kafvit$u6l$1@ger.gmane.org>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
	<kafvit$u6l$1@ger.gmane.org>
Message-ID: <CA+HFc85Nk=m+iAL2-g9-fvyhUXTk6U1rMFr6OLyhLK5ac5u4Ww@mail.gmail.com>

On Fri, Dec 14, 2012 at 5:44 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> All in all, I'm not a fan of that one big revolution that will make
> everything beautiful, fast and shiny, but that will never happen, really. I
> prefer small steps that make things work.
>

This is the biggest impedance mismatch that I see in this discussion. Pypy
is all about revolution, changing completely how the interpreter work to
keep a clean language for the user (python). The same should happen in
cython, a completely different backend for pypy only keeping the nice
cython language. I think there needs to be a fork of cython that don't have
any idea of reference counting and uses a mixture of cffi and a clean
c-api. But first pypy needs to have the c-api right?


-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121217/a31d3542/attachment.html>

From estama at gmail.com  Mon Dec 17 14:46:09 2012
From: estama at gmail.com (Eleytherios Stamatogiannakis)
Date: Mon, 17 Dec 2012 15:46:09 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
Message-ID: <50CF2221.4070805@gmail.com>

We had a "bug" in our previous benchmark ("mspw_bench.sql"). The way it 
was written allowed SQLite to short-circuit column data retrieval, 
ending up with minimal exercising of the CFFI layer.

The attached query exercises CFFI as it should. We also checked its 
profiling characteristics, and it is a lot closer to what we are seen 
with our working query loads.

Below are timings when running it from inside mterm (so there is no 
import overhead):

CPython + APSW: 0 sec 956 msec
pypy + APSW: 8 sec 673 msec
pypy + MSPW: 2 sec 550 msec

Best regards.

lefteris.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mspw_bench.sql
Type: text/x-sql
Size: 122 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121217/74b64fdd/attachment.bin>

From arigo at tunes.org  Mon Dec 17 14:46:46 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Dec 2012 14:46:46 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxRJVQE==1NJxyUkaEZBcGsBJ7Af8zr8ok7qCSh-WBsAMA@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
	<kafvit$u6l$1@ger.gmane.org>
	<CAK5idxRJVQE==1NJxyUkaEZBcGsBJ7Af8zr8ok7qCSh-WBsAMA@mail.gmail.com>
Message-ID: <CAMSv6X3HU62m_NkNs3PW05DatSaeeS+VjKiAbwX4xGCzST_qJA@mail.gmail.com>

Hi Fijal,

On Fri, Dec 14, 2012 at 9:04 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> That said, I agree that exposing a different C API is not solving
> much, while adding burden to maintainers of both Cython and PyPy and
> I'm generally against the idea.

I tend to agree with this point (so I disagree with what Leonardo said
just now, on this front :-).

(skipped explanation...)

> What we get:
>
> * Simple refcounting (can be implemented in C as macros even)
>
> * Lack of dictionaries

It sounds like a good model to me.  If I'm right, refcounting is even
simpler than in CPython, because all it needs is to increment or
decrement the refcount, but not check for zero.  This check is done
only during collection.

About "Lack of dictionaries", it's unclear: as a first approximation
to your idea, we could simply keep a dictionary going from the PyPy
object to the C object.  (We don't need the opposite direction.)


A bient?t,

Armin.

From fijall at gmail.com  Mon Dec 17 15:46:44 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 17 Dec 2012 16:46:44 +0200
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAMSv6X3HU62m_NkNs3PW05DatSaeeS+VjKiAbwX4xGCzST_qJA@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
	<kafvit$u6l$1@ger.gmane.org>
	<CAK5idxRJVQE==1NJxyUkaEZBcGsBJ7Af8zr8ok7qCSh-WBsAMA@mail.gmail.com>
	<CAMSv6X3HU62m_NkNs3PW05DatSaeeS+VjKiAbwX4xGCzST_qJA@mail.gmail.com>
Message-ID: <CAK5idxREtTwAPNj23oD=q4iLJTYayxMA-gF5sRi83sU_eYOFpw@mail.gmail.com>

On Mon, Dec 17, 2012 at 3:46 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Fijal,
>
> On Fri, Dec 14, 2012 at 9:04 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> That said, I agree that exposing a different C API is not solving
>> much, while adding burden to maintainers of both Cython and PyPy and
>> I'm generally against the idea.
>
> I tend to agree with this point (so I disagree with what Leonardo said
> just now, on this front :-).
>
> (skipped explanation...)
>
>> What we get:
>>
>> * Simple refcounting (can be implemented in C as macros even)
>>
>> * Lack of dictionaries
>
> It sounds like a good model to me.  If I'm right, refcounting is even
> simpler than in CPython, because all it needs is to increment or
> decrement the refcount, but not check for zero.  This check is done
> only during collection.
>
> About "Lack of dictionaries", it's unclear: as a first approximation
> to your idea, we could simply keep a dictionary going from the PyPy
> object to the C object.  (We don't need the opposite direction.)

If you have a dictionary, then the pypy object does not keep the cpy
object alive (or the dictionary keeps both of them alive or some other
stuff). Maybe we can something like what we do now with hashes of
objects. Dictionary while in nursery and then add the link if it
survives?

From arigo at tunes.org  Mon Dec 17 20:09:19 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Dec 2012 20:09:19 +0100
Subject: [pypy-dev] Scrapy fails in PyPy
In-Reply-To: <CAK5idxREtTwAPNj23oD=q4iLJTYayxMA-gF5sRi83sU_eYOFpw@mail.gmail.com>
References: <CANJQ5C6nY5RQ=C+ZAfv7Z-Bp0uwAfz=10P+DyUhuXijwXkWgog@mail.gmail.com>
	<CAMSv6X3THjMgaXrVPXd0XKMPjqbf7EtZDwo0eBr0FuBJOD2zmQ@mail.gmail.com>
	<k9ketr$1bu$1@ger.gmane.org>
	<CANJQ5C52yTo02QciGZNd1=P5qpURfrUO3MXEceNc849=AHKvwQ@mail.gmail.com>
	<CAK5idxSd=X=_Jfd=ZQEa-Ha=uBC-sC9BV1pj4RcbLTrCC-Xx+A@mail.gmail.com>
	<CANJQ5C7fzgUR6nkh1epoGcmDqY9aNmRBiPb-YJ3o8a0yhrOS1Q@mail.gmail.com>
	<CAK5idxRzaJyXOL+TS5kj2q_pJVXC3RyssXPZfxJ6JAg9Tvc3Jg@mail.gmail.com>
	<kac0fq$s7p$1@ger.gmane.org>
	<CAFRnB2Uzc2HrnFLET3fP3cC+s+k2Z19sp1v1GshcR=Pq4rgQYQ@mail.gmail.com>
	<kafvit$u6l$1@ger.gmane.org>
	<CAK5idxRJVQE==1NJxyUkaEZBcGsBJ7Af8zr8ok7qCSh-WBsAMA@mail.gmail.com>
	<CAMSv6X3HU62m_NkNs3PW05DatSaeeS+VjKiAbwX4xGCzST_qJA@mail.gmail.com>
	<CAK5idxREtTwAPNj23oD=q4iLJTYayxMA-gF5sRi83sU_eYOFpw@mail.gmail.com>
Message-ID: <CAMSv6X284sfy=x7eQwzvbOmwcn8Wvn2M2XaKaY_RXHYdrxHW=Q@mail.gmail.com>

Hi Maciej,

On Mon, Dec 17, 2012 at 3:46 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> If you have a dictionary, then the pypy object does not keep the cpy
> object alive (or the dictionary keeps both of them alive or some other
> stuff).

Obviouly it should be a dictionary with weak keys.  The values can be
raw addresses anyway, as the CPyExt objects don't move.

> Maybe we can something like what we do now with hashes of
> objects. Dictionary while in nursery and then add the link if it
> survives?

Yes, maybe, but first we should do it without this optimization.  Also
note that if a PyPy object was already old when we first asked for its
CPyExt object, then we need the dictionary anyway.  It's unclear how
common the optimization case is: accessing a few times a young
object's CPyExt equivalent (but not too often), and then continuing to
access it (often) when the young object was made old.


A bient?t,

Armin.

From fijall at gmail.com  Mon Dec 17 21:42:13 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 17 Dec 2012 22:42:13 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50CF2221.4070805@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
Message-ID: <CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>

On Mon, Dec 17, 2012 at 3:46 PM, Eleytherios Stamatogiannakis
<estama at gmail.com> wrote:
> We had a "bug" in our previous benchmark ("mspw_bench.sql"). The way it was
> written allowed SQLite to short-circuit column data retrieval, ending up
> with minimal exercising of the CFFI layer.

I thought it was a feature :)

>
> The attached query exercises CFFI as it should. We also checked its
> profiling characteristics, and it is a lot closer to what we are seen with
> our working query loads.
>
> Below are timings when running it from inside mterm (so there is no import
> overhead):
>
> CPython + APSW: 0 sec 956 msec
> pypy + APSW: 8 sec 673 msec
> pypy + MSPW: 2 sec 550 msec
>
> Best regards.
>
> lefteris.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From thomas at doylend.com  Wed Dec 19 23:15:48 2012
From: thomas at doylend.com (Thomas)
Date: Wed, 19 Dec 2012 14:15:48 -0800
Subject: [pypy-dev] I have a question...
Message-ID: <50D23C94.8040404@doylend.com>

PyPy Team,

I tried to use PyPy on Win64. It crashed because I needed a file called 
'MSCVR100.DLL'.
Could you distribute that with PyPy?

Signed,
Thomas

From matti.picus at gmail.com  Thu Dec 20 05:43:06 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 20 Dec 2012 06:43:06 +0200
Subject: [pypy-dev] I have a question...
In-Reply-To: <50D23C94.8040404@doylend.com>
References: <50D23C94.8040404@doylend.com>
Message-ID: <2CF8C1A5-0272-41EA-A2A9-5A1166BB9494@gmail.com>

Thanks for trying pypy.
Some of the nightly builds need the runtime you saw, which can be found here
http://www.microsoft.com/en-us/download/details.aspx?id=5555
The official version here http://pypy.org/download.html
depends on a different runtime, and a link is provided on that page to download it. Where did you get the version you tried? 
Matti

On 2012-12-20, at 12:15 AM, Thomas <thomas at doylend.com> wrote:

> PyPy Team,
> 
> I tried to use PyPy on Win64. It crashed because I needed a file called 'MSCVR100.DLL'.
> Could you distribute that with PyPy?
> 
> Signed,
> Thomas
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From holger at merlinux.eu  Thu Dec 20 13:19:11 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 20 Dec 2012 12:19:11 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121206152306.GQ26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
Message-ID: <20121220121911.GP26599@merlinux.eu>

Hi again,

any progress on internal discussions regarding the hosting questions?

Note that speed.pypy.org is the most important from the list as it
is the most expensive, sitting on an otherwise mostly unused machine.

best,
holger


On Thu, Dec 06, 2012 at 15:23 +0000, holger krekel wrote:
> Hi folks,
> 
> for the last several years i cared for hosting these pypy services:
> 
> - pypy.org (static generated html)
> - bugs.pypy.org (roundup instance, postfix server)
> - speed.python.org (django app)
> 
> These services are living on two rented servers which cost me per-month
> and i am finally getting rid of them in January 2013.
> 
> IIRC Maciej and others had various ideas (last year) on where the
> pypy services could better live.  Any progress on this?
> 
> ASFAIK the PSF is willing to offer a VM that we could use, at least
> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
> has other contacts to try, go ahead and maybe CC me to avoid chaos.
> 
> Note that i actually don't want to drive these efforts but offer
> my help in helping to migrate.  I'd be happy if the currently active
> pypy devs determine a plan of action and drive it.
>  
> best,
> holger
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 

From fijall at gmail.com  Thu Dec 20 14:23:27 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 20 Dec 2012 15:23:27 +0200
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121220121911.GP26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
Message-ID: <CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>

On Thu, Dec 20, 2012 at 2:19 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi again,
>
> any progress on internal discussions regarding the hosting questions?
>
> Note that speed.pypy.org is the most important from the list as it
> is the most expensive, sitting on an otherwise mostly unused machine.
>
> best,
> holger

My offer is still valid. Should I just go ahead or is someone
coordinating with the PSF?

>
>
> On Thu, Dec 06, 2012 at 15:23 +0000, holger krekel wrote:
>> Hi folks,
>>
>> for the last several years i cared for hosting these pypy services:
>>
>> - pypy.org (static generated html)
>> - bugs.pypy.org (roundup instance, postfix server)
>> - speed.python.org (django app)
>>
>> These services are living on two rented servers which cost me per-month
>> and i am finally getting rid of them in January 2013.
>>
>> IIRC Maciej and others had various ideas (last year) on where the
>> pypy services could better live.  Any progress on this?
>>
>> ASFAIK the PSF is willing to offer a VM that we could use, at least
>> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
>> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
>> has other contacts to try, go ahead and maybe CC me to avoid chaos.
>>
>> Note that i actually don't want to drive these efforts but offer
>> my help in helping to migrate.  I'd be happy if the currently active
>> pypy devs determine a plan of action and drive it.
>>
>> best,
>> holger
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

From wiktor8010 at o2.pl  Thu Dec 20 14:28:17 2012
From: wiktor8010 at o2.pl (=?UTF-8?Q?Wiktor_Mizdal?=)
Date: Thu, 20 Dec 2012 14:28:17 +0100
Subject: [pypy-dev] =?utf-8?q?pypy_compatibility_page?=
Message-ID: <64be42ab.6be53d1c.50d31271.b3c1e@o2.pl>


I try use in Pypy 2 codes from Compatibility Page:

1) open("filename", "w").write("stuff")

2) with open("filename", "w") as f:
    f.write("stuff")

and it works.

Is Compatibility Page actual?

Wiktor

From fijall at gmail.com  Thu Dec 20 14:34:10 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 20 Dec 2012 15:34:10 +0200
Subject: [pypy-dev] pypy compatibility page
In-Reply-To: <64be42ab.6be53d1c.50d31271.b3c1e@o2.pl>
References: <64be42ab.6be53d1c.50d31271.b3c1e@o2.pl>
Message-ID: <CAK5idxTHRQ0pc0FmpRVGzzr4=A7RZ741m3K7LqsL0ViP5GbGUA@mail.gmail.com>

On Thu, Dec 20, 2012 at 3:28 PM, Wiktor Mizdal <wiktor8010 at o2.pl> wrote:
>
> I try use in Pypy 2 codes from Compatibility Page:
>
> 1) open("filename", "w").write("stuff")
>
> 2) with open("filename", "w") as f:
>     f.write("stuff")
>
> and it works.
>
> Is Compatibility Page actual?
>
> Wiktor

The first one *might* not work if the next thing you do is to read the
file. Notably:

open("filename", "w").write("stuff")
open("filename", "r").read() # might return empty string

Can you suggest a better wording?

Cheers,
fijal

From holger at merlinux.eu  Thu Dec 20 15:04:36 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 20 Dec 2012 14:04:36 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
Message-ID: <20121220140436.GQ26599@merlinux.eu>

On Thu, Dec 20, 2012 at 15:23 +0200, Maciej Fijalkowski wrote:
> On Thu, Dec 20, 2012 at 2:19 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi again,
> >
> > any progress on internal discussions regarding the hosting questions?
> >
> > Note that speed.pypy.org is the most important from the list as it
> > is the most expensive, sitting on an otherwise mostly unused machine.
> >
> > best,
> > holger
> 
> My offer is still valid. Should I just go ahead or is someone
> coordinating with the PSF?

That's a question whose answer should best be discussed and decided by
the pypy dev group if i may repeat myself.  Probably hosting is not the
only issue that is worth discussing.

FYI I just talked to the provider of the machine hosting speed.pypy.org.
the next switch off date is the 6th of January 2013 (including) and
i need to confirm now.

pypy.org/bugs.pypy.org are living independently and there is less hurry.

best,
holger

> >
> >
> > On Thu, Dec 06, 2012 at 15:23 +0000, holger krekel wrote:
> >> Hi folks,
> >>
> >> for the last several years i cared for hosting these pypy services:
> >>
> >> - pypy.org (static generated html)
> >> - bugs.pypy.org (roundup instance, postfix server)
> >> - speed.python.org (django app)
> >>
> >> These services are living on two rented servers which cost me per-month
> >> and i am finally getting rid of them in January 2013.
> >>
> >> IIRC Maciej and others had various ideas (last year) on where the
> >> pypy services could better live.  Any progress on this?
> >>
> >> ASFAIK the PSF is willing to offer a VM that we could use, at least
> >> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
> >> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
> >> has other contacts to try, go ahead and maybe CC me to avoid chaos.
> >>
> >> Note that i actually don't want to drive these efforts but offer
> >> my help in helping to migrate.  I'd be happy if the currently active
> >> pypy devs determine a plan of action and drive it.
> >>
> >> best,
> >> holger
> >>
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> 

From fijall at gmail.com  Thu Dec 20 17:11:27 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 20 Dec 2012 18:11:27 +0200
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121220140436.GQ26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
Message-ID: <CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>

On Thu, Dec 20, 2012 at 4:04 PM, holger krekel <holger at merlinux.eu> wrote:
> On Thu, Dec 20, 2012 at 15:23 +0200, Maciej Fijalkowski wrote:
>> On Thu, Dec 20, 2012 at 2:19 PM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi again,
>> >
>> > any progress on internal discussions regarding the hosting questions?
>> >
>> > Note that speed.pypy.org is the most important from the list as it
>> > is the most expensive, sitting on an otherwise mostly unused machine.
>> >
>> > best,
>> > holger
>>
>> My offer is still valid. Should I just go ahead or is someone
>> coordinating with the PSF?
>
> That's a question whose answer should best be discussed and decided by
> the pypy dev group if i may repeat myself.  Probably hosting is not the
> only issue that is worth discussing.
>
> FYI I just talked to the provider of the machine hosting speed.pypy.org.
> the next switch off date is the 6th of January 2013 (including) and
> i need to confirm now.

Hi Holger.

We will move the speed out of your machine until Jan 6.

Cheers,
fijal

>
> pypy.org/bugs.pypy.org are living independently and there is less hurry.
>
> best,
> holger
>
>> >
>> >
>> > On Thu, Dec 06, 2012 at 15:23 +0000, holger krekel wrote:
>> >> Hi folks,
>> >>
>> >> for the last several years i cared for hosting these pypy services:
>> >>
>> >> - pypy.org (static generated html)
>> >> - bugs.pypy.org (roundup instance, postfix server)
>> >> - speed.python.org (django app)
>> >>
>> >> These services are living on two rented servers which cost me per-month
>> >> and i am finally getting rid of them in January 2013.
>> >>
>> >> IIRC Maciej and others had various ideas (last year) on where the
>> >> pypy services could better live.  Any progress on this?
>> >>
>> >> ASFAIK the PSF is willing to offer a VM that we could use, at least
>> >> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
>> >> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
>> >> has other contacts to try, go ahead and maybe CC me to avoid chaos.
>> >>
>> >> Note that i actually don't want to drive these efforts but offer
>> >> my help in helping to migrate.  I'd be happy if the currently active
>> >> pypy devs determine a plan of action and drive it.
>> >>
>> >> best,
>> >> holger
>> >>
>> >> _______________________________________________
>> >> pypy-dev mailing list
>> >> pypy-dev at python.org
>> >> http://mail.python.org/mailman/listinfo/pypy-dev
>> >>
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > http://mail.python.org/mailman/listinfo/pypy-dev
>>

From jonathan at slenders.be  Thu Dec 20 23:09:57 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Thu, 20 Dec 2012 23:09:57 +0100
Subject: [pypy-dev] Await-keyword
Message-ID: <CAKfyG3z_eXjooGqgr-2joC6X5Desz7gzTTNfbW87xB6ZRK7twA@mail.gmail.com>

Dear all,

For my work, I needed a python sandbox which contained the 'await' keyword
for asynchronous programming, just like c# does.
I found it very easy to extend the 2.7 grammar and include this keyword.

https://bitbucket.org/jonathanslenders/pypy

Personally, I think this is a very clean solution for Twisted's
@defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in
other async frameworks.

Just sharing this information, but I'd also like to know whether Python
code developers would consider to implement this in the Python standard
language. (maybe Python 3000.) I really have no idea what steps are taken
before accepting new grammar, but I'm willing to defend this syntax or to
write some articles about it.

Cheers,
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121220/5d37856c/attachment.html>

From benjamin at python.org  Thu Dec 20 23:11:11 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 20 Dec 2012 16:11:11 -0600
Subject: [pypy-dev] Await-keyword
In-Reply-To: <CAKfyG3z_eXjooGqgr-2joC6X5Desz7gzTTNfbW87xB6ZRK7twA@mail.gmail.com>
References: <CAKfyG3z_eXjooGqgr-2joC6X5Desz7gzTTNfbW87xB6ZRK7twA@mail.gmail.com>
Message-ID: <CAPZV6o_wfbAaTCTsEV9P8ME_awD-YJC3PYZAtZo7TDzQFnK7=Q@mail.gmail.com>

2012/12/20 Jonathan Slenders <jonathan at slenders.be>:
> Personally, I think this is a very clean solution for Twisted's
> @defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in other
> async frameworks.
>
> Just sharing this information, but I'd also like to know whether Python code
> developers would consider to implement this in the Python standard language.
> (maybe Python 3000.) I really have no idea what steps are taken before
> accepting new grammar, but I'm willing to defend this syntax or to write
> some articles about it.

That would be an issue for the python-ideas mailing list.

-- 
Regards,
Benjamin

From jonathan at slenders.be  Thu Dec 20 23:21:27 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Thu, 20 Dec 2012 23:21:27 +0100
Subject: [pypy-dev] Twisted sandbox library.
Message-ID: <CAKfyG3yQkV5X62a-2zFyyH-AWR2p+0SLTr2NDRJvNJgVN3DEiQ@mail.gmail.com>

Hi all, me again :)

As contributions for a better sandbox hypervisor library were welcome, I
here contribute our Twisted Matrix hypervisor.

https://github.com/jonathanslenders/twisted-sandlib/blob/master/sandlib.py

If there is interest for adding this to Pypy's repository, surely I can add
some more documentation, tests and sample code.

Cheers,
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121220/f965c797/attachment.html>

From fijall at gmail.com  Thu Dec 20 23:30:13 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Dec 2012 00:30:13 +0200
Subject: [pypy-dev] Twisted sandbox library.
In-Reply-To: <CAKfyG3yQkV5X62a-2zFyyH-AWR2p+0SLTr2NDRJvNJgVN3DEiQ@mail.gmail.com>
References: <CAKfyG3yQkV5X62a-2zFyyH-AWR2p+0SLTr2NDRJvNJgVN3DEiQ@mail.gmail.com>
Message-ID: <CAK5idxT-p--qaBGrBsKxf5BWc9zhSBOogPcT09DyrMx1L9xiBA@mail.gmail.com>

On Fri, Dec 21, 2012 at 12:21 AM, Jonathan Slenders
<jonathan at slenders.be> wrote:
> Hi all, me again :)
>
> As contributions for a better sandbox hypervisor library were welcome, I
> here contribute our Twisted Matrix hypervisor.
>
> https://github.com/jonathanslenders/twisted-sandlib/blob/master/sandlib.py
>
> If there is interest for adding this to Pypy's repository, surely I can add
> some more documentation, tests and sample code.
>
> Cheers,
> Jonathan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

Hi Jonathan

Thanks for your contribution. I think this code belongs more to
twisted than to pypy (or maybe to it's own project, who knows).

If you want to contribute to twisted, I suggest reading the
guildelines - http://twistedmatrix.com/trac/wiki/BasicGuideToContributingCode

You can also ask twisted folks for support - they're incredibly
friendly for newcomers and you can receive any feedback you need.

Cheers,
fijal

From joehillen at gmail.com  Fri Dec 21 00:37:10 2012
From: joehillen at gmail.com (Joe Hillenbrand)
Date: Thu, 20 Dec 2012 15:37:10 -0800
Subject: [pypy-dev] Twisted sandbox library.
In-Reply-To: <CAKfyG3yQkV5X62a-2zFyyH-AWR2p+0SLTr2NDRJvNJgVN3DEiQ@mail.gmail.com>
References: <CAKfyG3yQkV5X62a-2zFyyH-AWR2p+0SLTr2NDRJvNJgVN3DEiQ@mail.gmail.com>
Message-ID: <CANJQ5C7Hq45oxTx7dkgYDu29PGVn8Tiu7iRtkK3q6TXP+LwVmQ@mail.gmail.com>

I notice there is no license posted with this. Remember anything posted
without a license is considered all rights reserved, so nobody can use this
as it is right now.
On Dec 20, 2012 2:23 PM, "Jonathan Slenders" <jonathan at slenders.be> wrote:

> Hi all, me again :)
>
> As contributions for a better sandbox hypervisor library were welcome, I
> here contribute our Twisted Matrix hypervisor.
>
> https://github.com/jonathanslenders/twisted-sandlib/blob/master/sandlib.py
>
> If there is interest for adding this to Pypy's repository, surely I can
> add some more documentation, tests and sample code.
>
> Cheers,
> Jonathan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121220/bcc5f1e5/attachment.html>

From thomas at doylend.com  Fri Dec 21 03:01:00 2012
From: thomas at doylend.com (Thomas)
Date: Thu, 20 Dec 2012 18:01:00 -0800
Subject: [pypy-dev] I would like it if...
Message-ID: <50D3C2DC.2040701@doylend.com>

PyPy Team:

Could you figure out how to make PyPy support Pygame (www.pygame.org)?
I am trying, but having no luck so far.

Signed,
Thomas

From matti.picus at gmail.com  Fri Dec 21 08:10:49 2012
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 21 Dec 2012 09:10:49 +0200
Subject: [pypy-dev] I have a question...
In-Reply-To: <50D38EA6.1000405@doylend.com>
References: <50D23C94.8040404@doylend.com>
	<2CF8C1A5-0272-41EA-A2A9-5A1166BB9494@gmail.com>
	<50D38EA6.1000405@doylend.com>
Message-ID: <50D40B79.2040402@gmail.com>

It seems I made a mess with the latest win32 beta release (again).
Over 2800 downloads and you were the first to notice, so thanks for the 
feedback. I wonder how
many of the others failed and never mentioned anything?
Some of the dlls we shipped with that pypy depend on the Visual 10 
runtime, and some on the Visual 2008.
Specifically, openssl (libeay32.dll and ssleay32.dll) are linked with 
Visual 10 where everything else is linked with Visual 2008.
I repackaged a zip to
http://picus.org.il/matti/pypy-c-jit-58888-7e4f0faa3d51-win32.zip
with the md5 of a34dcd53537aaba0bed69d47f492ac73

Could someone with superpowers please replace the package at
https://bitbucket.org/pypy/pypy/downloads/pypy-2.0-beta1-win32.zip
and fix the md5 checksum at http://www.pypy.org/download.html ?

Hopefully the new win32 buildbot will not suffer these problems.
Matti

On 21/12/2012 12:18 AM, Thomas wrote:
> PyPy Team:
>
> I downloaded PyPy from www.pypy.org/downloads.html - version 2.0beta1, 
> Windows
> 32-bit version. I also downloaded 'vcredist_x86.exe' from Microsoft 
> and ran it.
>
> I tried to use PyPy on a 32-bit Windows after failing with Win64, but 
> before running 'vcredist_x86.exe'.
> Should I try running 'vcredist_x86.exe' on Win32?
>
> Signed,
> Thomas
>
> On 12/19/2012 8:43 PM, Matti Picus wrote:
>> Thanks for trying pypy.
>> Some of the nightly builds need the runtime you saw, which can be 
>> found here
>> http://www.microsoft.com/en-us/download/details.aspx?id=5555
>> The official version here http://pypy.org/download.html
>> depends on a different runtime, and a link is provided on that page 
>> to download it. Where did you get the version you tried?
>> Matti
>>
>> On 2012-12-20, at 12:15 AM, Thomas <thomas at doylend.com> wrote:
>>
>>> PyPy Team,
>>>
>>> I tried to use PyPy on Win64. It crashed because I needed a file 
>>> called 'MSCVR100.DLL'.
>>> Could you distribute that with PyPy?
>>>
>>> Signed,
>>> Thomas
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>


From zhoucan1990 at gmail.com  Fri Dec 21 08:15:17 2012
From: zhoucan1990 at gmail.com (=?GB2312?B?1tyy0w==?=)
Date: Fri, 21 Dec 2012 15:15:17 +0800
Subject: [pypy-dev] a question about pypy-sandbox
Message-ID: <CAPNw4fas8LOO3U74aHU3sTmfjo5dzvxRVMj7SZNK2Mcw45XVAw@mail.gmail.com>

Hi,all

I want to creat a sandbox to run untrust code. And I choose pypy and I have
immplemented my RPC server.

But i feel puzzled pypy sandbox is not compatible with all the standard
libs(such as threading), since pypy

does not allow interperter  to load native C modules.

I want to know why pypy doesn't allow enve standard libs load C modules,
then untrust code can not use

some standard libs. And if it's proper that I support some modules such as
threading.py by allow interperter

to load c modules for threading.py?

Zhoucan with Chinese English
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121221/248fecca/attachment.html>

From amauryfa at gmail.com  Fri Dec 21 09:56:42 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 21 Dec 2012 09:56:42 +0100
Subject: [pypy-dev] I would like it if...
In-Reply-To: <50D3C2DC.2040701@doylend.com>
References: <50D3C2DC.2040701@doylend.com>
Message-ID: <CAGmFidY7MDyqbknw1_EbYqM5-rx99RcjT-axgMe4=G1XqbuVCQ@mail.gmail.com>

2012/12/21 Thomas <thomas at doylend.com>

> PyPy Team:
>
> Could you figure out how to make PyPy support Pygame (www.pygame.org)?
> I am trying, but having no luck so far.
>

At some point (one year ago?) I had pygame run on pypy for sure.
Did you try compiling pygame with pypy as the Python interpreter?

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121221/632b6f4c/attachment.html>

From arigo at tunes.org  Fri Dec 21 10:05:16 2012
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Dec 2012 10:05:16 +0100
Subject: [pypy-dev] Await-keyword
In-Reply-To: <CAPZV6o_wfbAaTCTsEV9P8ME_awD-YJC3PYZAtZo7TDzQFnK7=Q@mail.gmail.com>
References: <CAKfyG3z_eXjooGqgr-2joC6X5Desz7gzTTNfbW87xB6ZRK7twA@mail.gmail.com>
	<CAPZV6o_wfbAaTCTsEV9P8ME_awD-YJC3PYZAtZo7TDzQFnK7=Q@mail.gmail.com>
Message-ID: <CAMSv6X1R=AwRhmzbUgHFunEz3u5bryRzbY_nMkjNT2ke7a5wOg@mail.gmail.com>

Hi Jonathan,

On Thu, Dec 20, 2012 at 11:11 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2012/12/20 Jonathan Slenders <jonathan at slenders.be>:
>> Personally, I think this is a very clean solution for Twisted's
>> @defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in other
>> async frameworks.
>>
>> Just sharing this information, but I'd also like to know whether Python code
>> developers would consider to implement this in the Python standard language.
>> (maybe Python 3000.) I really have no idea what steps are taken before
>> accepting new grammar, but I'm willing to defend this syntax or to write
>> some articles about it.
>
> That would be an issue for the python-ideas mailing list.

To expand on Benjamin's answer: we do welcome people that use PyPy to
play with syntax extensions.  However we're not going to do small
language extensions in ways incompatible with Python as implemented by
CPython.  That's why you should discuss your idea starting from the
python-ideas mailing list.  It is a plus if you have a working
prototype already, discuss it on your own blog, even use it already in
medium-scale projects.  But I warn you, adding new keywords is tough.
:-)


A bient?t,

Armin.

From holger at merlinux.eu  Fri Dec 21 10:09:34 2012
From: holger at merlinux.eu (holger krekel)
Date: Fri, 21 Dec 2012 09:09:34 +0000
Subject: [pypy-dev] Await-keyword
In-Reply-To: <CAMSv6X1R=AwRhmzbUgHFunEz3u5bryRzbY_nMkjNT2ke7a5wOg@mail.gmail.com>
References: <CAKfyG3z_eXjooGqgr-2joC6X5Desz7gzTTNfbW87xB6ZRK7twA@mail.gmail.com>
	<CAPZV6o_wfbAaTCTsEV9P8ME_awD-YJC3PYZAtZo7TDzQFnK7=Q@mail.gmail.com>
	<CAMSv6X1R=AwRhmzbUgHFunEz3u5bryRzbY_nMkjNT2ke7a5wOg@mail.gmail.com>
Message-ID: <20121221090934.GV26599@merlinux.eu>

On Fri, Dec 21, 2012 at 10:05 +0100, Armin Rigo wrote:
> Hi Jonathan,
> 
> On Thu, Dec 20, 2012 at 11:11 PM, Benjamin Peterson <benjamin at python.org> wrote:
> > 2012/12/20 Jonathan Slenders <jonathan at slenders.be>:
> >> Personally, I think this is a very clean solution for Twisted's
> >> @defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in other
> >> async frameworks.
> >>
> >> Just sharing this information, but I'd also like to know whether Python code
> >> developers would consider to implement this in the Python standard language.
> >> (maybe Python 3000.) I really have no idea what steps are taken before
> >> accepting new grammar, but I'm willing to defend this syntax or to write
> >> some articles about it.
> >
> > That would be an issue for the python-ideas mailing list.
> 
> To expand on Benjamin's answer: we do welcome people that use PyPy to
> play with syntax extensions.  However we're not going to do small
> language extensions in ways incompatible with Python as implemented by
> CPython.  That's why you should discuss your idea starting from the
> python-ideas mailing list.  It is a plus if you have a working
> prototype already, discuss it on your own blog, even use it already in
> medium-scale projects.  But I warn you, adding new keywords is tough.
> :-)

adding to that, there is a recent maybe related PEP draft from Guido
about async IO, see here:

http://www.python.org/dev/peps/pep-3156/

holger

From hodgestar at gmail.com  Fri Dec 21 10:15:45 2012
From: hodgestar at gmail.com (Simon Cross)
Date: Fri, 21 Dec 2012 11:15:45 +0200
Subject: [pypy-dev] I would like it if...
In-Reply-To: <CAGmFidY7MDyqbknw1_EbYqM5-rx99RcjT-axgMe4=G1XqbuVCQ@mail.gmail.com>
References: <50D3C2DC.2040701@doylend.com>
	<CAGmFidY7MDyqbknw1_EbYqM5-rx99RcjT-axgMe4=G1XqbuVCQ@mail.gmail.com>
Message-ID: <CAD5NRCGHWoEBjc_76VbhfQRMtN0pA9BDDYj1RayDVhAWd7-tvA@mail.gmail.com>

On Fri, Dec 21, 2012 at 10:56 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> At some point (one year ago?) I had pygame run on pypy for sure.
> Did you try compiling pygame with pypy as the Python interpreter?

Are you sure? In October 2012 CTPUG sprinted on getting pygame running
under pypy [1] and encountered a number of issues [2]. We got some
bugs fixed and a pygame window to display eventually but it wasn't
exactly working (or fast).

Duangle has started to wrap SDL using cffi [3] which might be a saner
starting point for getting an SDL wrapper that runs under pypy with
good performance.

[1] http://ctpug.org.za/wiki/Meeting20111008
[2] See commits at https://bitbucket.org/stefanor/pygame-pypy
[3] https://bitbucket.org/duangle/pysdl-cffi

Schiavo
Simon

From arigo at tunes.org  Fri Dec 21 10:18:39 2012
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Dec 2012 10:18:39 +0100
Subject: [pypy-dev] I have a question...
In-Reply-To: <50D40B79.2040402@gmail.com>
References: <50D23C94.8040404@doylend.com>
	<2CF8C1A5-0272-41EA-A2A9-5A1166BB9494@gmail.com>
	<50D38EA6.1000405@doylend.com> <50D40B79.2040402@gmail.com>
Message-ID: <CAMSv6X16joXCipWo0+ozLha6_5K2WYm7pAiF+krUrDwHU8=xNQ@mail.gmail.com>

Hi all,

On Fri, Dec 21, 2012 at 8:10 AM, Matti Picus <matti.picus at gmail.com> wrote:
> Could someone with superpowers please replace the package at
> https://bitbucket.org/pypy/pypy/downloads/pypy-2.0-beta1-win32.zip
> and fix the md5 checksum at http://www.pypy.org/download.html ?

Done.  Thanks for your ongoing work on Win32 support!


A bient?t,

Armin.

From arigo at tunes.org  Fri Dec 21 10:27:32 2012
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Dec 2012 10:27:32 +0100
Subject: [pypy-dev] a question about pypy-sandbox
In-Reply-To: <CAPNw4fas8LOO3U74aHU3sTmfjo5dzvxRVMj7SZNK2Mcw45XVAw@mail.gmail.com>
References: <CAPNw4fas8LOO3U74aHU3sTmfjo5dzvxRVMj7SZNK2Mcw45XVAw@mail.gmail.com>
Message-ID: <CAMSv6X3sRoZzfXW_e=YfFKHU-RTMUp273jKao3265WEfDjU0JQ@mail.gmail.com>

Hi Zhoucan.

On Fri, Dec 21, 2012 at 8:15 AM, ?? <zhoucan1990 at gmail.com> wrote:
> But i feel puzzled pypy sandbox is not compatible with all the standard
> libs(such as threading), since pypy does not allow interperter  to load native C modules.

There is some confusion here.  The regular PyPy (no sandbox) comes with:

* all standard libraries, rewritten not a C code, but as internal RPython code;

* for third-party modules, a way to load the standard CPython C
extension modules (slowly).

The PyPy with sandbox comes with almost no such module.  If your
application requires a specific standard library module and you can
verify that it is safe to include it, you might consider adding it,
with the proper tweaks.  For example, the "thread" module might in
theory be added, saying "threadsafe=True" on all external C functions
that it calls; but this needs efforts to make sure that it is safe.

On the other hand, third-party CPython C extension modules are a no-no
for the sandboxed PyPy.  Maybe this could change, but it's even more
work.


A bient?t,

Armin.

From jonathan at slenders.be  Fri Dec 21 12:56:42 2012
From: jonathan at slenders.be (Jonathan Slenders)
Date: Fri, 21 Dec 2012 12:56:42 +0100
Subject: [pypy-dev] a question about pypy-sandbox
In-Reply-To: <CAMSv6X3sRoZzfXW_e=YfFKHU-RTMUp273jKao3265WEfDjU0JQ@mail.gmail.com>
References: <CAPNw4fas8LOO3U74aHU3sTmfjo5dzvxRVMj7SZNK2Mcw45XVAw@mail.gmail.com>
	<CAMSv6X3sRoZzfXW_e=YfFKHU-RTMUp273jKao3265WEfDjU0JQ@mail.gmail.com>
Message-ID: <CAKfyG3wnsj3L4tV+Zu-TLLXM+LgtnL32UV1Ka=MrTpsKJeUA9g@mail.gmail.com>

Jep, threading would be really hard to do. Especially given that we are
serializing system calls through the stdin/out. A blocking system call in
the hypervisor process would make it impossible to switch to another
thread. (unless you decide to still execute until the next system call in
the other thread.)

An alternative what is working pretty well for me is using one asyncronous
event loop in the sandbox. (like Twisted Matrix, tornado, etc...), and
instead of epoll or select, you use any blocking operation like open() or
read() in the sandbox.

Cheers,
Jonathan



2012/12/21 Armin Rigo <arigo at tunes.org>

> Hi Zhoucan.
>
> On Fri, Dec 21, 2012 at 8:15 AM, ?? <zhoucan1990 at gmail.com> wrote:
> > But i feel puzzled pypy sandbox is not compatible with all the standard
> > libs(such as threading), since pypy does not allow interperter  to load
> native C modules.
>
> There is some confusion here.  The regular PyPy (no sandbox) comes with:
>
> * all standard libraries, rewritten not a C code, but as internal RPython
> code;
>
> * for third-party modules, a way to load the standard CPython C
> extension modules (slowly).
>
> The PyPy with sandbox comes with almost no such module.  If your
> application requires a specific standard library module and you can
> verify that it is safe to include it, you might consider adding it,
> with the proper tweaks.  For example, the "thread" module might in
> theory be added, saying "threadsafe=True" on all external C functions
> that it calls; but this needs efforts to make sure that it is safe.
>
> On the other hand, third-party CPython C extension modules are a no-no
> for the sandboxed PyPy.  Maybe this could change, but it's even more
> work.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121221/e7d524c6/attachment.html>

From holger at merlinux.eu  Fri Dec 21 15:11:17 2012
From: holger at merlinux.eu (holger krekel)
Date: Fri, 21 Dec 2012 14:11:17 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
	<CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
Message-ID: <20121221141117.GD26599@merlinux.eu>

Hi all,

as to migrating speed.pypy.org, how is it related to speed.python.org?
just stumbled over a thread discussing the latter here:

    http://mail.python.org/pipermail/speed/2012-October/000224.html

There is even an (empty) instance running setup by Miquel Torres (the
author of codespeed, the software behind speed.pypy.org) at:

    http://speed.python.org:8080/

Maybe it's possible to use this or setup a sister "speed.pypy.org"
instance on the same machine?

best,
holger

On Thu, Dec 20, 2012 at 18:11 +0200, Maciej Fijalkowski wrote:
> On Thu, Dec 20, 2012 at 4:04 PM, holger krekel <holger at merlinux.eu> wrote:
> > On Thu, Dec 20, 2012 at 15:23 +0200, Maciej Fijalkowski wrote:
> >> On Thu, Dec 20, 2012 at 2:19 PM, holger krekel <holger at merlinux.eu> wrote:
> >> > Hi again,
> >> >
> >> > any progress on internal discussions regarding the hosting questions?
> >> >
> >> > Note that speed.pypy.org is the most important from the list as it
> >> > is the most expensive, sitting on an otherwise mostly unused machine.
> >> >
> >> > best,
> >> > holger
> >>
> >> My offer is still valid. Should I just go ahead or is someone
> >> coordinating with the PSF?
> >
> > That's a question whose answer should best be discussed and decided by
> > the pypy dev group if i may repeat myself.  Probably hosting is not the
> > only issue that is worth discussing.
> >
> > FYI I just talked to the provider of the machine hosting speed.pypy.org.
> > the next switch off date is the 6th of January 2013 (including) and
> > i need to confirm now.
> 
> Hi Holger.
> 
> We will move the speed out of your machine until Jan 6.
> 
> Cheers,
> fijal
> 
> >
> > pypy.org/bugs.pypy.org are living independently and there is less hurry.
> >
> > best,
> > holger
> >
> >> >
> >> >
> >> > On Thu, Dec 06, 2012 at 15:23 +0000, holger krekel wrote:
> >> >> Hi folks,
> >> >>
> >> >> for the last several years i cared for hosting these pypy services:
> >> >>
> >> >> - pypy.org (static generated html)
> >> >> - bugs.pypy.org (roundup instance, postfix server)
> >> >> - speed.python.org (django app)
> >> >>
> >> >> These services are living on two rented servers which cost me per-month
> >> >> and i am finally getting rid of them in January 2013.
> >> >>
> >> >> IIRC Maciej and others had various ideas (last year) on where the
> >> >> pypy services could better live.  Any progress on this?
> >> >>
> >> >> ASFAIK the PSF is willing to offer a VM that we could use, at least
> >> >> for pypy.org and bugs.pypy.org.  However, i haven't yet heart back from
> >> >> Noah Kantrowitz.  I contacted him twice in recent weeks.  If somebody
> >> >> has other contacts to try, go ahead and maybe CC me to avoid chaos.
> >> >>
> >> >> Note that i actually don't want to drive these efforts but offer
> >> >> my help in helping to migrate.  I'd be happy if the currently active
> >> >> pypy devs determine a plan of action and drive it.
> >> >>
> >> >> best,
> >> >> holger
> >> >>
> >> >> _______________________________________________
> >> >> pypy-dev mailing list
> >> >> pypy-dev at python.org
> >> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >> >>
> >> > _______________________________________________
> >> > pypy-dev mailing list
> >> > pypy-dev at python.org
> >> > http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> 

From fijall at gmail.com  Fri Dec 21 15:13:39 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Dec 2012 16:13:39 +0200
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121221141117.GD26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
	<CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
	<20121221141117.GD26599@merlinux.eu>
Message-ID: <CAK5idxTf_opMU13=PLbH=LBradk0aC-=T4UPx59PUWoSuSnEyQ@mail.gmail.com>

On Fri, Dec 21, 2012 at 4:11 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi all,
>
> as to migrating speed.pypy.org, how is it related to speed.python.org?
> just stumbled over a thread discussing the latter here:
>
>     http://mail.python.org/pipermail/speed/2012-October/000224.html
>
> There is even an (empty) instance running setup by Miquel Torres (the
> author of codespeed, the software behind speed.pypy.org) at:
>
>     http://speed.python.org:8080/
>
> Maybe it's possible to use this or setup a sister "speed.pypy.org"
> instance on the same machine?
>
> best,
> holger

The idea is that this machine is used for benchmarking. That means we
should not have any web front (or anything else that can use CPU at
strange times). So, -1 from me.

Cheers,
fijal

From holger at merlinux.eu  Fri Dec 21 15:56:40 2012
From: holger at merlinux.eu (holger krekel)
Date: Fri, 21 Dec 2012 14:56:40 +0000
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxTf_opMU13=PLbH=LBradk0aC-=T4UPx59PUWoSuSnEyQ@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
	<CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
	<20121221141117.GD26599@merlinux.eu>
	<CAK5idxTf_opMU13=PLbH=LBradk0aC-=T4UPx59PUWoSuSnEyQ@mail.gmail.com>
Message-ID: <20121221145640.GF26599@merlinux.eu>

On Fri, Dec 21, 2012 at 16:13 +0200, Maciej Fijalkowski wrote:
> On Fri, Dec 21, 2012 at 4:11 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi all,
> >
> > as to migrating speed.pypy.org, how is it related to speed.python.org?
> > just stumbled over a thread discussing the latter here:
> >
> >     http://mail.python.org/pipermail/speed/2012-October/000224.html
> >
> > There is even an (empty) instance running setup by Miquel Torres (the
> > author of codespeed, the software behind speed.pypy.org) at:
> >
> >     http://speed.python.org:8080/
> >
> > Maybe it's possible to use this or setup a sister "speed.pypy.org"
> > instance on the same machine?
> >
> > best,
> > holger
> 
> The idea is that this machine is used for benchmarking. That means we
> should not have any web front (or anything else that can use CPU at
> strange times). So, -1 from me.

Sure.  Let's hope it's ever used for benchmarking - ASFAIK it is quite a decent
machine but it's sitting around idly for a while now.

holger

From fijall at gmail.com  Fri Dec 21 16:03:54 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Dec 2012 17:03:54 +0200
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <20121221145640.GF26599@merlinux.eu>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
	<CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
	<20121221141117.GD26599@merlinux.eu>
	<CAK5idxTf_opMU13=PLbH=LBradk0aC-=T4UPx59PUWoSuSnEyQ@mail.gmail.com>
	<20121221145640.GF26599@merlinux.eu>
Message-ID: <CAK5idxTpJ6-Bwg8Y3HaR+Ydq3tKKUGsrP-D7_x3HnDp+fYADYQ@mail.gmail.com>

On Fri, Dec 21, 2012 at 4:56 PM, holger krekel <holger at merlinux.eu> wrote:
> On Fri, Dec 21, 2012 at 16:13 +0200, Maciej Fijalkowski wrote:
>> On Fri, Dec 21, 2012 at 4:11 PM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi all,
>> >
>> > as to migrating speed.pypy.org, how is it related to speed.python.org?
>> > just stumbled over a thread discussing the latter here:
>> >
>> >     http://mail.python.org/pipermail/speed/2012-October/000224.html
>> >
>> > There is even an (empty) instance running setup by Miquel Torres (the
>> > author of codespeed, the software behind speed.pypy.org) at:
>> >
>> >     http://speed.python.org:8080/
>> >
>> > Maybe it's possible to use this or setup a sister "speed.pypy.org"
>> > instance on the same machine?
>> >
>> > best,
>> > holger
>>
>> The idea is that this machine is used for benchmarking. That means we
>> should not have any web front (or anything else that can use CPU at
>> strange times). So, -1 from me.
>
> Sure.  Let's hope it's ever used for benchmarking - ASFAIK it is quite a decent
> machine but it's sitting around idly for a while now.
>
> holger

We've been using it a bit for STM benchmarking and other stuff. For a
few months there was a problem of very little disk space that could
not be regained (this got solved by now), but it made it disappear
from our "things we normally do". For what is worth also being hosted
in the states makes it unpleasant to do manual benchmarking, so we use
tannit more.

From tobami at gmail.com  Fri Dec 21 18:10:45 2012
From: tobami at gmail.com (Miquel Torres)
Date: Fri, 21 Dec 2012 18:10:45 +0100
Subject: [pypy-dev] pypy services hosting / action needed!
In-Reply-To: <CAK5idxTpJ6-Bwg8Y3HaR+Ydq3tKKUGsrP-D7_x3HnDp+fYADYQ@mail.gmail.com>
References: <20121206152306.GQ26599@merlinux.eu>
	<20121220121911.GP26599@merlinux.eu>
	<CAK5idxTbE_VQDyXrYPMNnbnaC3+_-Y0PBXkEgEoaP2Nhra1rHQ@mail.gmail.com>
	<20121220140436.GQ26599@merlinux.eu>
	<CAK5idxT0y5tYFVVnhb-ak5KHi9x5LLNixZMCstw2QmESUAB5pg@mail.gmail.com>
	<20121221141117.GD26599@merlinux.eu>
	<CAK5idxTf_opMU13=PLbH=LBradk0aC-=T4UPx59PUWoSuSnEyQ@mail.gmail.com>
	<20121221145640.GF26599@merlinux.eu>
	<CAK5idxTpJ6-Bwg8Y3HaR+Ydq3tKKUGsrP-D7_x3HnDp+fYADYQ@mail.gmail.com>
Message-ID: <CAGf+9Vz=ckbJ6g_UyvrucdfOvh=L=JkkB5T12VLgNWz5u=qeWA@mail.gmail.com>

Btw., count on me to help at least by deploying Codespeed to another
machine. Just ping me when a VM is ready...


2012/12/21 Maciej Fijalkowski <fijall at gmail.com>

> On Fri, Dec 21, 2012 at 4:56 PM, holger krekel <holger at merlinux.eu> wrote:
> > On Fri, Dec 21, 2012 at 16:13 +0200, Maciej Fijalkowski wrote:
> >> On Fri, Dec 21, 2012 at 4:11 PM, holger krekel <holger at merlinux.eu>
> wrote:
> >> > Hi all,
> >> >
> >> > as to migrating speed.pypy.org, how is it related to speed.python.org
> ?
> >> > just stumbled over a thread discussing the latter here:
> >> >
> >> >     http://mail.python.org/pipermail/speed/2012-October/000224.html
> >> >
> >> > There is even an (empty) instance running setup by Miquel Torres (the
> >> > author of codespeed, the software behind speed.pypy.org) at:
> >> >
> >> >     http://speed.python.org:8080/
> >> >
> >> > Maybe it's possible to use this or setup a sister "speed.pypy.org"
> >> > instance on the same machine?
> >> >
> >> > best,
> >> > holger
> >>
> >> The idea is that this machine is used for benchmarking. That means we
> >> should not have any web front (or anything else that can use CPU at
> >> strange times). So, -1 from me.
> >
> > Sure.  Let's hope it's ever used for benchmarking - ASFAIK it is quite a
> decent
> > machine but it's sitting around idly for a while now.
> >
> > holger
>
> We've been using it a bit for STM benchmarking and other stuff. For a
> few months there was a problem of very little disk space that could
> not be regained (this got solved by now), but it made it disappear
> from our "things we normally do". For what is worth also being hosted
> in the states makes it unpleasant to do manual benchmarking, so we use
> tannit more.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121221/4906ac03/attachment.html>

From techtonik at gmail.com  Mon Dec 24 09:03:56 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 11:03:56 +0300
Subject: [pypy-dev] PyPy in getting started
Message-ID: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>

Hi,

The PyPy description in getting started is still too scientific. I attach a
patch with accessibility fix. =) But it is still not completely clear if
PyPy's Python implementation is written in Python or in RPython?

'''And the second is one particular implementation that is so generated ?
an implementation of the Python programming language written in Python
itself.'''

If Python implementation is written in Python, and not in RPython then
there is the missing link between the framework and implementation, which
is confusing.

http://doc.pypy.org/en/latest/getting-started.html#what-is-pypy
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/60c99bc6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getting-started.diff
Type: application/octet-stream
Size: 758 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/60c99bc6/attachment.obj>

From william.leslie.ttg at gmail.com  Mon Dec 24 09:24:55 2012
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Mon, 24 Dec 2012 19:24:55 +1100
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
Message-ID: <CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>

Ugh, hit the wrong reply button.

On 24 December 2012 19:24, William ML Leslie
<william.leslie.ttg at gmail.com> wrote:
> On 24 December 2012 19:03, anatoly techtonik <techtonik at gmail.com> wrote:
>> Hi,
>>
>> The PyPy description in getting started is still too scientific. I attach a
>> patch with accessibility fix. =) But it is still not completely clear if
>> PyPy's Python implementation is written in Python or in RPython?
>
> They aren't mutually exclusive; all rpython is python.
>
>> '''And the second is one particular implementation that is so generated ? an
>> implementation of the Python programming language written in Python
>> itself.'''
>
> Maybe we could be even clearer - in my mind, implementations aren't
> generated, they do have to be written (:
>
>> If Python implementation is written in Python, and not in RPython then there
>> is the missing link between the framework and implementation, which is
>> confusing.
>>
>> http://doc.pypy.org/en/latest/getting-started.html#what-is-pypy
>> --
>> anatoly t.
>
> --
> William Leslie



-- 
William Leslie

From techtonik at gmail.com  Mon Dec 24 09:35:25 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 11:35:25 +0300
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
	<CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
Message-ID: <CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>

On Mon, Dec 24, 2012 at 11:24 AM, William ML Leslie <
william.leslie.ttg at gmail.com> wrote:

> Ugh, hit the wrong reply button.
>
> On 24 December 2012 19:24, William ML Leslie
> <william.leslie.ttg at gmail.com> wrote:
> > On 24 December 2012 19:03, anatoly techtonik <techtonik at gmail.com>
> wrote:
> >> Hi,
> >>
> >> The PyPy description in getting started is still too scientific. I
> attach a
> >> patch with accessibility fix. =) But it is still not completely clear if
> >> PyPy's Python implementation is written in Python or in RPython?
> >
> > They aren't mutually exclusive; all rpython is python.
>

But it is still confusing, because the RPython doesn't include generators,
which means Python implementation is still written in subset of Python and
not using the real full fat Python.


> >> '''And the second is one particular implementation that is so generated
> ? an
> >> implementation of the Python programming language written in Python
> >> itself.'''
> >
> > Maybe we could be even clearer - in my mind, implementations aren't
> > generated, they do have to be written (:
>

I looks like the documentation is missing one more important aspect -
RPython toolchain can be used to compile standalone programs written in
RPython (which is why I've finally got to it at the moment).


> >> If Python implementation is written in Python, and not in RPython then
> there
> >> is the missing link between the framework and implementation, which is
> >> confusing.
> >>
> >> http://doc.pypy.org/en/latest/getting-started.html#what-is-pypy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/2525ee40/attachment.html>

From arigo at tunes.org  Mon Dec 24 10:47:12 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Dec 2012 10:47:12 +0100
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
	<CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
	<CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>
Message-ID: <CAMSv6X3-9X+zNAon+CtXsV1hrMkpkw2xKNiw=C0WzwVzN4XzhA@mail.gmail.com>

Hi Anatoly,

On Mon, Dec 24, 2012 at 9:35 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> But it is still confusing, because the RPython doesn't include generators,

I see your point --- but RPython does include generators nowadays.  It
does not include tons of other things though.

> which means Python implementation is still written in subset of Python and
> not using the real full fat Python.

RPython is Python with restrictions about the allowed runtime behavior
(but not the import-time behavior).  The Python implementation of PyPy
is written in RPython, yes.

> I looks like the documentation is missing one more important aspect -
> RPython toolchain can be used to compile standalone programs written in
> RPython (which is why I've finally got to it at the moment).

That is theoretically correct but not mentioned on purpose.  We don't
support this.  RPython is not meant as a general-purpose programming
language, and it doesn't support a lot of things that would make it
more convenient for that purpose.  If you are using RPython because of
performance issues only, PyPy has a good JIT nowadays which can get
full Python close.  If you are using RPython because you don't like
dynamic languages, then pick Java or .NET instead.


A bient?t,

Armin.

From techtonik at gmail.com  Mon Dec 24 11:35:11 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 13:35:11 +0300
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAMSv6X3-9X+zNAon+CtXsV1hrMkpkw2xKNiw=C0WzwVzN4XzhA@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
	<CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
	<CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>
	<CAMSv6X3-9X+zNAon+CtXsV1hrMkpkw2xKNiw=C0WzwVzN4XzhA@mail.gmail.com>
Message-ID: <CAPkN8xJA2aQqWQy328DQa4ib9x24hV-JgJ3PqajC58kKnW3ezg@mail.gmail.com>

On Mon, Dec 24, 2012 at 12:47 PM, Armin Rigo <arigo at tunes.org> wrote:

>
>
The Python implementation of PyPy
> is written in RPython, yes.


That's exactly the answer I needed to hear. Thanks.


>  > I looks like the documentation is missing one more important aspect -
> > RPython toolchain can be used to compile standalone programs written in
> > RPython (which is why I've finally got to it at the moment).
>
> That is theoretically correct but not mentioned on purpose.  We don't
> support this.  RPython is not meant as a general-purpose programming
> language, and it doesn't support a lot of things that would make it
> more convenient for that purpose.  If you are using RPython because of
> performance issues only, PyPy has a good JIT nowadays which can get
> full Python close.  If you are using RPython because you don't like
> dynamic languages, then pick Java or .NET instead.
>

There is a bsdiff compression/decompression algorithm written in C. It is
ported to Linux, Windows and other platforms many times. It requires
several separate toolchains to compile and integrate and everything could
be fine if not occasional crashes. According to Colin Percival the problem
is in algorithm, but it is hard to debug in C after the Python.

Maintainability is the sole reason I want to rewrite it. And I don't want
to maintain C and Python code. It is not possible to bundle the whole 30+
megabytes of Python for this small task, so a translator that can put the
algorithm directly into CPU code for the target machine ignoring the whole
HARDWARE-BIOS-OS-C-OBJ-BIN (include/lib/flags/suffix) toolchain is highly
interesting. I don't know if RPython can do this, but so far it is the
closest thing. Maybe you know better alternatives for the experiment?

http://debian.2.n7.nabble.com/Bug-409664-bsdiff-is-extremely-slow-on-some-files-sometimes-hangs-td1738215.html
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/4676a443/attachment-0001.html>

From arigo at tunes.org  Mon Dec 24 11:49:50 2012
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Dec 2012 11:49:50 +0100
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAPkN8xJA2aQqWQy328DQa4ib9x24hV-JgJ3PqajC58kKnW3ezg@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
	<CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
	<CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>
	<CAMSv6X3-9X+zNAon+CtXsV1hrMkpkw2xKNiw=C0WzwVzN4XzhA@mail.gmail.com>
	<CAPkN8xJA2aQqWQy328DQa4ib9x24hV-JgJ3PqajC58kKnW3ezg@mail.gmail.com>
Message-ID: <CAMSv6X21JO2Lpa+r9V5b7f+O_fH_q7BsPSw2OAZ6CnbPMbWsdA@mail.gmail.com>

Hi,

On Mon, Dec 24, 2012 at 11:35 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> algorithm directly into CPU code for the target machine ignoring the whole
> HARDWARE-BIOS-OS-C-OBJ-BIN (include/lib/flags/suffix) toolchain is highly
> interesting. I don't know if RPython can do this, but so far it is the
> closest thing. Maybe you know better alternatives for the experiment?

RPython is translated to C code that is then compiled with (for
example) gcc using the standard toolchain.  If the problem is
compiling C code in the first place, looking at RPython is the wrong
place.  Well, that's just my opinion; feel free to play with RPython.
I just believe it's very, very much the wrong tool.


A bient?t,

Armin.

From techtonik at gmail.com  Mon Dec 24 12:10:56 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 14:10:56 +0300
Subject: [pypy-dev] PyPy in getting started
In-Reply-To: <CAMSv6X21JO2Lpa+r9V5b7f+O_fH_q7BsPSw2OAZ6CnbPMbWsdA@mail.gmail.com>
References: <CAPkN8x+zpQS6sPbn_h8os0yDuKAL_OOPS5OdfjeJSPKEzK1x5w@mail.gmail.com>
	<CAHgd1hG+gNjcjqotgRB1ZcsxTnqHD4eW3J7rrP+hpjczNV3iOw@mail.gmail.com>
	<CAHgd1hFfnTKB6ZrG4E9hH8_X0dh4xSo8zBOMX=NqEVsfdHQjkA@mail.gmail.com>
	<CAPkN8xJQn-Mq7MfXF3Bh0u9_VijrCOHK6Kdej2_NXJR_T87Q8w@mail.gmail.com>
	<CAMSv6X3-9X+zNAon+CtXsV1hrMkpkw2xKNiw=C0WzwVzN4XzhA@mail.gmail.com>
	<CAPkN8xJA2aQqWQy328DQa4ib9x24hV-JgJ3PqajC58kKnW3ezg@mail.gmail.com>
	<CAMSv6X21JO2Lpa+r9V5b7f+O_fH_q7BsPSw2OAZ6CnbPMbWsdA@mail.gmail.com>
Message-ID: <CAPkN8x+jebTxpT_tzatDLExOgAtvFAyub=k2NNrJi+c2vyXKWQ@mail.gmail.com>

On Mon, Dec 24, 2012 at 1:49 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi,
>
> On Mon, Dec 24, 2012 at 11:35 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > algorithm directly into CPU code for the target machine ignoring the
> whole
> > HARDWARE-BIOS-OS-C-OBJ-BIN (include/lib/flags/suffix) toolchain is highly
> > interesting. I don't know if RPython can do this, but so far it is the
> > closest thing. Maybe you know better alternatives for the experiment?
>
> RPython is translated to C code that is then compiled with (for
> example) gcc using the standard toolchain.  If the problem is
> compiling C code in the first place, looking at RPython is the wrong
> place.  Well, that's just my opinion; feel free to play with RPython.
> I just believe it's very, very much the wrong tool.
>

All right. Got it. Thanks for the explanation.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/0d14ef61/attachment.html>

From techtonik at gmail.com  Mon Dec 24 12:16:39 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 14:16:39 +0300
Subject: [pypy-dev] Config value owner
Message-ID: <CAPkN8x+5cHSoOxqMsmsBJWCjaxsniFzj+67rjDzR6BF5F67DKA@mail.gmail.com>

What is the meaning of value owner in Config class? It prevents useful
debugging output from displaying (like --help option, which doesn't work
for translator.py on my machine due to missing compiler traceback).

https://bitbucket.org/pypy/pypy/src/36abd0dd07d468456e1acb2565f0b59979ed1445/pypy/config/config.py?at=default#cl-189
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/d90cfbc8/attachment.html>

From fijall at gmail.com  Mon Dec 24 12:18:50 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 24 Dec 2012 13:18:50 +0200
Subject: [pypy-dev] Config value owner
In-Reply-To: <CAPkN8x+5cHSoOxqMsmsBJWCjaxsniFzj+67rjDzR6BF5F67DKA@mail.gmail.com>
References: <CAPkN8x+5cHSoOxqMsmsBJWCjaxsniFzj+67rjDzR6BF5F67DKA@mail.gmail.com>
Message-ID: <CAK5idxTmYOmRdyrax_Q8noaVy=eEV20m0E4Zrzmvq6v6j3Jymw@mail.gmail.com>

On Mon, Dec 24, 2012 at 1:16 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> What is the meaning of value owner in Config class? It prevents useful
> debugging output from displaying (like --help option, which doesn't work for
> translator.py on my machine due to missing compiler traceback).
>
> https://bitbucket.org/pypy/pypy/src/36abd0dd07d468456e1acb2565f0b59979ed1445/pypy/config/config.py?at=default#cl-189

Hi Anatoly.

If you want real time feedback on such questions, I suggest you come
to #pypy on IRC. I seriously don't understand the question though.

> --
> anatoly t.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From techtonik at gmail.com  Mon Dec 24 18:31:45 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Dec 2012 20:31:45 +0300
Subject: [pypy-dev] Config value owner
In-Reply-To: <CAK5idxTmYOmRdyrax_Q8noaVy=eEV20m0E4Zrzmvq6v6j3Jymw@mail.gmail.com>
References: <CAPkN8x+5cHSoOxqMsmsBJWCjaxsniFzj+67rjDzR6BF5F67DKA@mail.gmail.com>
	<CAK5idxTmYOmRdyrax_Q8noaVy=eEV20m0E4Zrzmvq6v6j3Jymw@mail.gmail.com>
Message-ID: <CAPkN8xKbbNt+6K6sNgAdkn6oEgrTTaHKQVsjpzXKntBGTsHRhw@mail.gmail.com>

On Mon, Dec 24, 2012 at 2:18 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:

On Mon, Dec 24, 2012 at 1:16 PM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > What is the meaning of value owner in Config class? It prevents useful
> > debugging output from displaying (like --help option, which doesn't work
> for
> > translator.py on my machine due to missing compiler traceback).
> >
> >
> https://bitbucket.org/pypy/pypy/src/36abd0dd07d468456e1acb2565f0b59979ed1445/pypy/config/config.py?at=default#cl-189
>
> Hi Anatoly.
>
> If you want real time feedback on such questions, I suggest you come

to #pypy on IRC. I seriously don't understand the question though.
>

I am trying to read help for rpython cli utility.

> pypy\translator\goal\translate.py --help
[platform:error] Could not find a Microsoft Compiler
[platform:error] Could not find a Microsoft Compiler
[platform:msg] Set platform with 'host' cc=None, using cc='cl.exe'
[translation:info] Translating target as defined by targetpypystandalone
[platform:execute] cl.exe /nologo /c /MD /O2
/Foc:\users\anatoli\appdata\local\temp\usession-default-31\gcctest.obj
c:\users\ana
toli\appdata\local\temp\usession-default-31\gcctest.c
Traceback (most recent call last):
  File "translate.py", line 326, in <module>
    main()
  File "translate.py", line 211, in main
    targetspec_dic, translateconfig, config, args =
parse_options_and_load_target()
...


I tried to debug why the --help option doesn't work and the utility starts
translating. The options stuff is interwined with pypy.config.config.Config
object. If you try to print the value of this object in
pypy\pypy\translator\goal\translate.py:parse_options_and_load_target you'll
get just one line:
[translate]

If you comment the block marked by the link above, which is:
            if self._cfgimpl_value_owners.get(name, None) == 'default':
                continue

Then the output is an indented tree that includes the value of help option
I was looking for. So, the question is what the check
with _cfgimpl_value_owners is for?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121224/059b7937/attachment.html>

From arigo at tunes.org  Wed Dec 26 10:34:59 2012
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 26 Dec 2012 10:34:59 +0100
Subject: [pypy-dev] Config value owner
In-Reply-To: <CAPkN8xKbbNt+6K6sNgAdkn6oEgrTTaHKQVsjpzXKntBGTsHRhw@mail.gmail.com>
References: <CAPkN8x+5cHSoOxqMsmsBJWCjaxsniFzj+67rjDzR6BF5F67DKA@mail.gmail.com>
	<CAK5idxTmYOmRdyrax_Q8noaVy=eEV20m0E4Zrzmvq6v6j3Jymw@mail.gmail.com>
	<CAPkN8xKbbNt+6K6sNgAdkn6oEgrTTaHKQVsjpzXKntBGTsHRhw@mail.gmail.com>
Message-ID: <CAMSv6X2SY+BD3EomeZNV7d09m36uWh0c+O_cQuxUP5KnHy826Q@mail.gmail.com>

Hi Anatoly,

On Mon, Dec 24, 2012 at 6:31 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> [platform:error] Could not find a Microsoft Compiler

That was a bug caused by not finding any compiler.  I moved the help
display to occur earlier, which fixes this bug (and lets --help run in
less than 15 seconds, too).


A bient?t,

Armin.

From fijall at gmail.com  Wed Dec 26 11:48:40 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 26 Dec 2012 12:48:40 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
Message-ID: <CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>

On Mon, Dec 17, 2012 at 10:42 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Dec 17, 2012 at 3:46 PM, Eleytherios Stamatogiannakis
> <estama at gmail.com> wrote:
>> We had a "bug" in our previous benchmark ("mspw_bench.sql"). The way it was
>> written allowed SQLite to short-circuit column data retrieval, ending up
>> with minimal exercising of the CFFI layer.
>
> I thought it was a feature :)

Hi Elefterios.

We've been working towards improving the situation of this benchmark.
There is a branch done by antonio cuni and there are a few smaller
things that will be improved in some time. Stay tuned.

Cheers,
fijal

From estama at gmail.com  Wed Dec 26 16:24:34 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Wed, 26 Dec 2012 17:24:34 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
Message-ID: <50DB16B2.8070301@gmail.com>

On 26/12/2012 12:48 ??, Maciej Fijalkowski wrote:
> On Mon, Dec 17, 2012 at 10:42 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Mon, Dec 17, 2012 at 3:46 PM, Eleytherios Stamatogiannakis
>> <estama at gmail.com> wrote:
>>> We had a "bug" in our previous benchmark ("mspw_bench.sql"). The way it was
>>> written allowed SQLite to short-circuit column data retrieval, ending up
>>> with minimal exercising of the CFFI layer.
>>
>> I thought it was a feature :)
>
> Hi Elefterios.
>
> We've been working towards improving the situation of this benchmark.
> There is a branch done by antonio cuni and there are a few smaller
> things that will be improved in some time. Stay tuned.
>
> Cheers,
> fijal
>

Thank you for looking into this part of pypy's performance. Whenever 
something reaches a testable state we would be glad to test/benchmark it.

On another front. We tried using SQLite's UTF-16 API to avoid doing the 
conversion to UTF-8 whenever we returned a string from pypy to SQLite. 
We used function "sqlite3_result_text16":

http://www.sqlite.org/c3ref/result_blob.html

defining it in CFFI as:

void sqlite3_result_text16(sqlite3_context*, wchar_t*, int, void(*)(void*));

The problem was that giving directly a pypy string to above function 
(using wchar_t type), only the first character of the string was passed 
to SQLite.

The only way to successfully pass a string to SQLite was by explicitly 
converting/encoding it to utf-16.

So the question i have is, does pypy keep its strings internally to utf-16?

Thanks again (and a happy new year) to all.

l.

From fijall at gmail.com  Wed Dec 26 17:19:12 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 26 Dec 2012 18:19:12 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50DB16B2.8070301@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
Message-ID: <CAK5idxSdC5vytNAh1Bt-xOFFzEcyCEWLtLOh7qYtMhUhtP1YEA@mail.gmail.com>

On Wed, Dec 26, 2012 at 5:24 PM, Elefterios Stamatogiannakis
<estama at gmail.com> wrote:
> On 26/12/2012 12:48 ??, Maciej Fijalkowski wrote:
>>
>> On Mon, Dec 17, 2012 at 10:42 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>>>
>>> On Mon, Dec 17, 2012 at 3:46 PM, Eleytherios Stamatogiannakis
>>> <estama at gmail.com> wrote:
>>>>
>>>> We had a "bug" in our previous benchmark ("mspw_bench.sql"). The way it
>>>> was
>>>> written allowed SQLite to short-circuit column data retrieval, ending up
>>>> with minimal exercising of the CFFI layer.
>>>
>>>
>>> I thought it was a feature :)
>>
>>
>> Hi Elefterios.
>>
>> We've been working towards improving the situation of this benchmark.
>> There is a branch done by antonio cuni and there are a few smaller
>> things that will be improved in some time. Stay tuned.
>>
>> Cheers,
>> fijal
>>
>
> Thank you for looking into this part of pypy's performance. Whenever
> something reaches a testable state we would be glad to test/benchmark it.
>
> On another front. We tried using SQLite's UTF-16 API to avoid doing the
> conversion to UTF-8 whenever we returned a string from pypy to SQLite. We
> used function "sqlite3_result_text16":
>
> http://www.sqlite.org/c3ref/result_blob.html
>
> defining it in CFFI as:
>
> void sqlite3_result_text16(sqlite3_context*, wchar_t*, int, void(*)(void*));
>
> The problem was that giving directly a pypy string to above function (using
> wchar_t type), only the first character of the string was passed to SQLite.
>
> The only way to successfully pass a string to SQLite was by explicitly
> converting/encoding it to utf-16.
>
> So the question i have is, does pypy keep its strings internally to utf-16?
>
> Thanks again (and a happy new year) to all.
>
> l.

Are you talking about strings or unicodes? that is, type str or unicode?

From estama at gmail.com  Wed Dec 26 21:45:37 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Wed, 26 Dec 2012 22:45:37 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAK5idxSdC5vytNAh1Bt-xOFFzEcyCEWLtLOh7qYtMhUhtP1YEA@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
	<CAK5idxSdC5vytNAh1Bt-xOFFzEcyCEWLtLOh7qYtMhUhtP1YEA@mail.gmail.com>
Message-ID: <50DB61F1.60700@gmail.com>

On 26/12/2012 6:19 ??, Maciej Fijalkowski wrote:
> On Wed, Dec 26, 2012 at 5:24 PM, Elefterios Stamatogiannakis
> <estama at gmail.com> wrote:
>>
>> Thank you for looking into this part of pypy's performance. Whenever
>> something reaches a testable state we would be glad to test/benchmark it.
>>
>> On another front. We tried using SQLite's UTF-16 API to avoid doing the
>> conversion to UTF-8 whenever we returned a string from pypy to SQLite. We
>> used function "sqlite3_result_text16":
>>
>> http://www.sqlite.org/c3ref/result_blob.html
>>
>> defining it in CFFI as:
>>
>> void sqlite3_result_text16(sqlite3_context*, wchar_t*, int, void(*)(void*));
>>
>> The problem was that giving directly a pypy string to above function (using
>> wchar_t type), only the first character of the string was passed to SQLite.
>>
>> The only way to successfully pass a string to SQLite was by explicitly
>> converting/encoding it to utf-16.
>>
>> So the question i have is, does pypy keep its strings internally to utf-16?
>>
>> Thanks again (and a happy new year) to all.
>>
>> l.
>
> Are you talking about strings or unicodes? that is, type str or unicode?
>

For unicode only. For regular type == str we use SQLite's 
sqlite3_result_text without doing any conversion/encoding at all.

l.

From arigo at tunes.org  Thu Dec 27 10:44:13 2012
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Dec 2012 10:44:13 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50DB16B2.8070301@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
Message-ID: <CAMSv6X3D1PCcs=XvNTCDQ_yRQZv5jXG4spFNQ+R4FgkXCRj+fg@mail.gmail.com>

Hi Elefterios,

On Wed, Dec 26, 2012 at 4:24 PM, Elefterios Stamatogiannakis
<estama at gmail.com> wrote:
> The problem was that giving directly a pypy string to above function (using
> wchar_t type), only the first character of the string was passed to SQLite.

Do you mean you unexpectedly got only the first character?  That would
be a bug.  Can you tell me what you did?

Normally, you should be able to pass a unicode object directly to a
"wchar_t *" argument and get a complete wchar_t-encoded string.  (Note
that wchar_t is 16-bit on Windows but 32-bit on Linux, for example.)


A bient?t,

Armin.

From estama at gmail.com  Thu Dec 27 14:33:34 2012
From: estama at gmail.com (Elefterios Stamatogiannakis)
Date: Thu, 27 Dec 2012 15:33:34 +0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <CAMSv6X3D1PCcs=XvNTCDQ_yRQZv5jXG4spFNQ+R4FgkXCRj+fg@mail.gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
	<CAMSv6X3D1PCcs=XvNTCDQ_yRQZv5jXG4spFNQ+R4FgkXCRj+fg@mail.gmail.com>
Message-ID: <50DC4E2E.4050009@gmail.com>

On 27/12/12 11:44, Armin Rigo wrote:
> Normally, you should be able to pass a unicode object directly to a
> "wchar_t *" argument and get a complete wchar_t-encoded string.  (Note
> that wchar_t is 16-bit on Windows but 32-bit on Linux, for example.)

Armin thanks for the clarification.

 From your note, i see what went wrong. We were directly passing pypy's 
UTF-32 (we are testing on linux) to SQLite's UTF-16 API. So this is why 
we only got the first character on the SQLite side.

Also linux's 32 bit wchars are unfortunate in our case, because SQLite 
only supports UTF-8 and UTF-16 in their API. So there is no way on linux 
to directly pass pypy's unicode strings to SQLite.

l.


From santagada at gmail.com  Thu Dec 27 23:36:00 2012
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 27 Dec 2012 20:36:00 -0200
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50DC4E2E.4050009@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
	<CAMSv6X3D1PCcs=XvNTCDQ_yRQZv5jXG4spFNQ+R4FgkXCRj+fg@mail.gmail.com>
	<50DC4E2E.4050009@gmail.com>
Message-ID: <CA+HFc878BJQ88WQMPH4UhfsqY1np1p+Z62v2gVU5hpzXaJpdzA@mail.gmail.com>

I might be wrong but cpython uses UCS internally on unicode, so even if
sqlite had UTF-32 it would not work.


On Thu, Dec 27, 2012 at 11:33 AM, Elefterios Stamatogiannakis <
estama at gmail.com> wrote:

> On 27/12/12 11:44, Armin Rigo wrote:
>
>> Normally, you should be able to pass a unicode object directly to a
>> "wchar_t *" argument and get a complete wchar_t-encoded string.  (Note
>> that wchar_t is 16-bit on Windows but 32-bit on Linux, for example.)
>>
>
> Armin thanks for the clarification.
>
> From your note, i see what went wrong. We were directly passing pypy's
> UTF-32 (we are testing on linux) to SQLite's UTF-16 API. So this is why we
> only got the first character on the SQLite side.
>
> Also linux's 32 bit wchars are unfortunate in our case, because SQLite
> only supports UTF-8 and UTF-16 in their API. So there is no way on linux to
> directly pass pypy's unicode strings to SQLite.
>
>
> l.
>
> ______________________________**_________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
>



-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121227/75ff67ac/attachment.html>

From contacto at hebabylon.com  Fri Dec 28 12:16:55 2012
From: contacto at hebabylon.com (Independent Social Network)
Date: Fri, 28 Dec 2012 12:16:55 +0100
Subject: [pypy-dev] Independen Spcoal Network
Message-ID: <0MPNP2-1Tk85R300e-004PV4@mrelayeu.kundenserver.de>

           HE Babylon* 
                                                             Independen Social Network
http://www.hebabylon.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121228/c0531005/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: elefante+rosa+icono_b.png
Type: application/octet-stream
Size: 8664 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20121228/c0531005/attachment.obj>

From arigo at tunes.org  Sat Dec 29 16:03:14 2012
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Dec 2012 16:03:14 +0100
Subject: [pypy-dev] CFFI speed results
In-Reply-To: <50DC4E2E.4050009@gmail.com>
References: <20121105224302.A04EA1C0558@cobra.cs.uni-duesseldorf.de>
	<50987D05.1040204@gmx.de> <509A999E.3080609@gmail.com>
	<CAMSv6X3-V5aQTa3mgCrt7XxAd3Hmt6CN_1ofGpfu08++hw7nNA@mail.gmail.com>
	<50C766BA.3080901@gmail.com>
	<CAMSv6X3U1hWCr=hERm6UTT2B=U_qUm7zdO9uORqY+ffr543mrQ@mail.gmail.com>
	<50CB610F.1010800@gmail.com>
	<CAK5idxRaEWMj=jWX4KskV12E-uG+G6xJo0VHKN33v948j-BZgw@mail.gmail.com>
	<50CBCAA3.7030705@gmail.com>
	<CAK5idxTe_w=M__vSgZ_erdr=X=r5HjQxvPbE5n6x9N8hAPh2ug@mail.gmail.com>
	<CAMSv6X2rbERiduptJ+9WKoFc4PnJoDL=TYLLJ_oRTHmM3QLNXQ@mail.gmail.com>
	<50CC5B7C.4010604@oz.net>
	<CAMSv6X3f6_UFtbhwi+8UsRuRQvV53ATw-5nfoLXgHk2bqRZFfg@mail.gmail.com>
	<50CF2221.4070805@gmail.com>
	<CAK5idxT7XO5Qp1yFNDpYvWo3fm1p_53q7REJ=YGAhmgGLPGuGg@mail.gmail.com>
	<CAK5idxT-Zx9ToBiL7kMmZqo_Q--ROpN9vGTTTHY_KjKO2k2cZg@mail.gmail.com>
	<50DB16B2.8070301@gmail.com>
	<CAMSv6X3D1PCcs=XvNTCDQ_yRQZv5jXG4spFNQ+R4FgkXCRj+fg@mail.gmail.com>
	<50DC4E2E.4050009@gmail.com>
Message-ID: <CAMSv6X3nXtdUuUxjThEdL96mPLxZ5hq_i6u3mDYcHjU72utm0Q@mail.gmail.com>

Hi,

On Thu, Dec 27, 2012 at 2:33 PM, Elefterios Stamatogiannakis
<estama at gmail.com> wrote:
> Also linux's 32 bit wchars are unfortunate in our case, because SQLite only
> supports UTF-8 and UTF-16 in their API. So there is no way on linux to
> directly pass pypy's unicode strings to SQLite.

Yes, given that pypy strings use UTF-32 on Linux, they are not
compatible.  Note that CPython normally also uses UTF-32 on Linux (at
least in most major distributions), but that can be configured at
compile-time.

Leonardo: UTF-32 and UCS-4 are the same thing; the difference is only
UCS-2 which is a subset of UTF-16.


A bient?t,

Armin.

From denis.bilenko at gmail.com  Sun Dec 30 13:24:37 2012
From: denis.bilenko at gmail.com (Denis Bilenko)
Date: Sun, 30 Dec 2012 13:24:37 +0100
Subject: [pypy-dev] (unexpectedly) bad performance of stackless and greenlet
	on PyPy
Message-ID: <CAGRiapMhTKOk1Xt1SkgBzN7+rw=yCTyaaqiK=2i9WP--f3W0xA@mail.gmail.com>

Hi!

I've micro-benchmarked stackless and greenlet module in PyPy and was
quite surprised with the results. According to this benchmark
stackless.channel is 100x times slower on PyPy (vs Stackless) and
greenlet is 20x times slower on PyPy (vs CPython).

https://gist.github.com/4412582

Is this a bug in benchmark (or PyPy) or is it the expected current
state of stackless features on PyPy?

Cheers,
Denis.

From fijall at gmail.com  Sun Dec 30 16:53:40 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 30 Dec 2012 17:53:40 +0200
Subject: [pypy-dev] (unexpectedly) bad performance of stackless and
 greenlet on PyPy
In-Reply-To: <CAGRiapMhTKOk1Xt1SkgBzN7+rw=yCTyaaqiK=2i9WP--f3W0xA@mail.gmail.com>
References: <CAGRiapMhTKOk1Xt1SkgBzN7+rw=yCTyaaqiK=2i9WP--f3W0xA@mail.gmail.com>
Message-ID: <CAK5idxRAQCFQ0f_kPbY5fuYmCFvgK7L4vAT_ar_Z2_qhxpRL_Q@mail.gmail.com>

On Sun, Dec 30, 2012 at 2:24 PM, Denis Bilenko <denis.bilenko at gmail.com> wrote:
> Hi!
>
> I've micro-benchmarked stackless and greenlet module in PyPy and was
> quite surprised with the results. According to this benchmark
> stackless.channel is 100x times slower on PyPy (vs Stackless) and
> greenlet is 20x times slower on PyPy (vs CPython).
>
> https://gist.github.com/4412582
>
> Is this a bug in benchmark (or PyPy) or is it the expected current
> state of stackless features on PyPy?
>
> Cheers,
> Denis.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

This is at least partly expected because stackless and/or greenlets
disable the JIT. It probably shouldn't be 100x though. There is an
ongoing work to make this reason go away and then we can look what's
left.

Cheers,
fijal